Intuition and Idea Behind Reshaping 4D Array to 2D Array in Numpy

Intuition and idea behind reshaping 4D array to 2D array in NumPy

General idea for nd to nd transformation

The idea with such nd to nd transformation is using just two things -

• Permute axes (with numpy.transpose or numpy.moveaxis or numpy.rollaxis if the needed permute order is a rolled one or numpy.swapaxes if just two axes need to be swapped) and

• Reshape.

Permute axes : To get the order such that the flattened version corresponds to the flattened version of output. So, if you somehow end up using it twice, look again because you shouldn't.

Reshape : To split the axes or bring the final output to the desired shape. Splitting axes is needed mostly at the start, when the input is of lower-dim and we are needed to split into blocks. Again, you shouldn't need this more than twice.

Hence, generally we would have three steps :

    [ Reshape ]      --->  [ Permute axes ]   --->  [ Reshape ]

 Create more axes             Bring axes             Merge axes
                          into correct order

Back-tracking method

The safest way to solve, given the input and output is through, what one could call as the back-tracking method, i.e. split the axes of the input (when going from smaller nd to bigger nd) or split the axes of the output (when going from bigger nd to smaller nd). The idea with the splitting is to bring the number of dims of the smaller nd one same as the bigger nd one. Then, study the strides of the output and match it up against the input to get the required permute order. Finally, a reshape (default way or C order) might be needed at the end, if the final one is a smaller nd one, to merge axes.

If both input and output are of same number of dims, then we would need to split both and break into blocks and study their strides against each other. In such cases, we should have the additional input parameter of block sizes, but that's probably off-topic.

Example

Let's use this specific case to demonstrate how to apply those strategies. In here, the input is 4D, while output is 2D. So, most probably, we won't need reshape to split. So, we need to start with permuting axes. Since, the final output is not 4D, but a 2D one, we would need a reshape at the end.

Now, the input here is :

In [270]: a
Out[270]: 
array([[[[ 0,  0],
         [ 0,  0]],

        [[ 5, 10],
         [15, 20]]],

       [[[ 6, 12],
         [18, 24]],

        [[ 7, 14],
         [21, 28]]]])

The expected output is :

In [271]: out
    Out[271]: 
    array([[ 0,  5,  0, 10],
           [ 6,  7, 12, 14],
           [ 0, 15,  0, 20],
           [18, 21, 24, 28]])

Also, this is a bigger nd to smaller nd transformation, so the back-tracking method would involve, splitting the output and studying its strides and matching up against the corresponding values in input :

                    axis = 3
                   ---      -->          
                                        
                    axis = 1                    
                   ------>           
axis=2|  axis=0|   [ 0,  5,  0, 10],        

               |   [ 6,  7, 12, 14],
               v  
      |            [ 0, 15,  0, 20],
      v
                   [18, 21, 24, 28]])

Hence, the permuted order needed is (2,0,3,1) :

In [275]: a.transpose((2, 0, 3, 1))
Out[275]: 
array([[[[ 0,  5],
         [ 0, 10]],

        [[ 6,  7],
         [12, 14]]],

       [[[ 0, 15],
         [ 0, 20]],

        [[18, 21],
         [24, 28]]]])

Then, simply reshape to the expected shape :

In [276]: a.transpose((2, 0, 3, 1)).reshape(4,4)
Out[276]: 
array([[ 0,  5,  0, 10],
       [ 6,  7, 12, 14],
       [ 0, 15,  0, 20],
       [18, 21, 24, 28]])

---

More examples

I dug up my history and found few Q&As based on nd to nd transformations. These could serve as other example cases, albeit with lesser explanation (mostly). As mentioned earlier, at most two reshapes and at most one swapaxes/transpose did the job everywhere. They are listed below :

• Python Reshape 3d array into 2d
• reshape an array using python/numpy
• Merging non-overlapping array blocks
• Conversion from a Numpy 3D array to a 2D array
• how to reshape an N length vector to a 3x(N/3) matrix in numpy using reshape
• Construct image from 4D list
• Reshaping/Combining several sub-matrices to one matrix in multi-dimensional space
• Interlace various small 2D matrices into a bigger one
• how to retrieve every section by 3X3?
• Reshaping 3D Numpy Array to a 2D array
• Iterate in submatrices through a bigger matrix
• Reorganizing a 2D numpy array into 3D
• Numpy change shape from (3, 512, 660, 4) to (3,2048,660,1)
• Numpy: rotate sub matrix m of M
• Split a 3D numpy array into 3D blocks
• Converting 3D matrix to cascaded 2D Matrices
• Rearranging numpy array
• Numpy: Reshape array along a specified axis
• How to construct 2d array from 2d arrays
• How to form a matrix from submatrices?
• Python: Reshape 3D image series to pixel series

Numpy: reshaping of 2D array

It took a bit of squinting at your data to find out that, in the output, the columns of A have been chunked by blocks of n*n and tiled into nxn squares. So, for example, the B[:2, :2] has the values of A[:4, 0].

So, the key is to try to do put those blocks into contiguous dimensions, then use a transpose of the appropriate axes, before a final reshape.

Long story short:

B = np.reshape(np.reshape(A.T, (m, m, n, n)).transpose(0, 2, 1, 3), (n*m, n*m))

Tips or patterns for reshaping 4D/5D arrays, (videos to frames)

Simply swap the second and third axes, and then merge the new second axis (old third one) with the first one with reshaping -

output = input_array.swapaxes(1,2).reshape(N*D,C,H,W)

We can also use transpose : input_array.transpose(0,2,1,3,4) to get the same swapping axes effect.

For a general intuitive method, please refer to Intuition and idea behind reshaping 4D array to 2D array in NumPy.

Numpy blocks reshaping

I don't think you can do this with reshape and transpose alone (although I'd love to be proven wrong). Using np.block works, but it's a bit messy:

np.block([list(i) for i in input_data.reshape( (2*h+1), (2*h+1), n, n )])

array([[ 0,  1,  4,  5,  8,  9, 12, 13, 16, 17],
       [ 2,  3,  6,  7, 10, 11, 14, 15, 18, 19],
       [20, 21, 24, 25, 28, 29, 32, 33, 36, 37],
       [22, 23, 26, 27, 30, 31, 34, 35, 38, 39],
       [40, 41, 44, 45, 48, 49, 52, 53, 56, 57],
       [42, 43, 46, 47, 50, 51, 54, 55, 58, 59],
       [60, 61, 64, 65, 68, 69, 72, 73, 76, 77],
       [62, 63, 66, 67, 70, 71, 74, 75, 78, 79],
       [80, 81, 84, 85, 88, 89, 92, 93, 96, 97],
       [82, 83, 86, 87, 90, 91, 94, 95, 98, 99]])

EDIT: Never mind, you can do without np.block:

input_data.reshape( (2*h+1), (2*h+1), n, n).transpose(0, 2, 1, 3).reshape(10, 10)

array([[ 0,  1,  4,  5,  8,  9, 12, 13, 16, 17],
       [ 2,  3,  6,  7, 10, 11, 14, 15, 18, 19],
       [20, 21, 24, 25, 28, 29, 32, 33, 36, 37],
       [22, 23, 26, 27, 30, 31, 34, 35, 38, 39],
       [40, 41, 44, 45, 48, 49, 52, 53, 56, 57],
       [42, 43, 46, 47, 50, 51, 54, 55, 58, 59],
       [60, 61, 64, 65, 68, 69, 72, 73, 76, 77],
       [62, 63, 66, 67, 70, 71, 74, 75, 78, 79],
       [80, 81, 84, 85, 88, 89, 92, 93, 96, 97],
       [82, 83, 86, 87, 90, 91, 94, 95, 98, 99]])

Reshaping 3D Numpy Array to a 2D array

Reshape to split the first axis into two, permute axes and one more reshape -

a.reshape(2,2,2,2).transpose(0,2,1,3).reshape(4,4)
a.reshape(2,2,2,2).swapaxes(1,2).reshape(4,4)

Making it generic, would become -

m,n,r = a.shape
out = a.reshape(m//2,2,n,r).swapaxes(1,2).reshape(-1,2*r)

Sample run -

In [20]: a
Out[20]: 
array([[[ 1,  2],
        [ 3,  4]],

       [[ 5,  6],
        [ 7,  8]],

       [[ 9, 10],
        [11, 12]],

       [[13, 14],
        [15, 16]]])

In [21]: a.reshape(2,2,2,2).swapaxes(1,2).reshape(4,4)
Out[21]: 
array([[ 1,  2,  5,  6],
       [ 3,  4,  7,  8],
       [ 9, 10, 13, 14],
       [11, 12, 15, 16]])

How can I reshape this multi-dimensional list to a 2D array?

The nested list returned by yeardatescalendar has the following hierarchy:

• Multi-month chunks
• Months
• Weeks
• Days

Therefore, for example, yr_11[2][1][3][2] will give the datetime representing the 3rd day in the 4th week of the 2nd month in the 3rd chunk. Remember that indexing goes from left (least nested) to right (most nested), but we normally read such elements in the opposite direction (most granular first)

To simplify our calculations, we can pass width=12 so that our result will consist of one chunk containing 12 months.

Next, it remains simply to flatten the list and pass the result to np.array:

import calendar

cal = calendar.Calendar()
yr_11 = cal.yeardatescalendar(2011, width=12)

flat = [day for month in yr_11[0] for week in month for day in week]
dates = np.array(flat)

print(dates)

Output:

[datetime.date(2010, 12, 27) datetime.date(2010, 12, 28)
 datetime.date(2010, 12, 29) datetime.date(2010, 12, 30)
 datetime.date(2010, 12, 31) datetime.date(2011, 1, 1)
 ...
 datetime.date(2011, 12, 28) datetime.date(2011, 12, 29)
 datetime.date(2011, 12, 30) datetime.date(2011, 12, 31)
 datetime.date(2012, 1, 1)]

Numpy dot product of a 4D array with its transpose fails

On your knowledge-driven wish of having a 2x2 array at the end, what about using xarray.dot for that kind of task. With your A in hand

>>> A.shape
(60, 64, 2, 2)

you would do

>>> xA   = xr.DataArray(A, dims=['d1','d2','d3','d4'])
>>> xA_t = xA.T
>>> xr.dot(xA_t, xA, dims=['d1','d2']).shape
(2, 2)

---

Related Topics