Splitting a Data Frame into Equal Parts

Splitting a data frame into equal parts

You could use split(), with rep() to create the groupings.

n <- 10
nr <- nrow(df)
split(df, rep(1:ceiling(nr/n), each=n, length.out=nr))

Split a large pandas dataframe

Use np.array_split:

Docstring:
Split an array into multiple sub-arrays.

Please refer to the ``split`` documentation.  The only difference
between these functions is that ``array_split`` allows
`indices_or_sections` to be an integer that does *not* equally
divide the axis.

In [1]: import pandas as pd

In [2]: df = pd.DataFrame({'A' : ['foo', 'bar', 'foo', 'bar',
   ...:                           'foo', 'bar', 'foo', 'foo'],
   ...:                    'B' : ['one', 'one', 'two', 'three',
   ...:                           'two', 'two', 'one', 'three'],
   ...:                    'C' : randn(8), 'D' : randn(8)})

In [3]: print df
     A      B         C         D
0  foo    one -0.174067 -0.608579
1  bar    one -0.860386 -1.210518
2  foo    two  0.614102  1.689837
3  bar  three -0.284792 -1.071160
4  foo    two  0.843610  0.803712
5  bar    two -1.514722  0.870861
6  foo    one  0.131529 -0.968151
7  foo  three -1.002946 -0.257468

In [4]: import numpy as np
In [5]: np.array_split(df, 3)
Out[5]: 
[     A    B         C         D
0  foo  one -0.174067 -0.608579
1  bar  one -0.860386 -1.210518
2  foo  two  0.614102  1.689837,
      A      B         C         D
3  bar  three -0.284792 -1.071160
4  foo    two  0.843610  0.803712
5  bar    two -1.514722  0.870861,
      A      B         C         D
6  foo    one  0.131529 -0.968151
7  foo  three -1.002946 -0.257468]

Split a data frame into six equal parts based on number of rows without knowing the number of rows - pandas

You can use np.array_split():

dfs = np.array_split(df, 6)

for index, df in enumerate(dfs):
    df.to_csv(f'df{index+1}.csv')

>>> print(dfs)

[   ID Job  Salary
 0   1   A     100
 1   2   B     200
 2   3   B      20,

    ID Job  Salary
 3   4   C     150
 4   5   A     500
 5   6   A     600,

    ID Job  Salary
 6   7   A     200
 7   8   B     150,

    ID Job  Salary
 8   9   C     110
 9  10   B     200,

     ID Job  Salary
 10  11   B     220
 11  12   A     150,

     ID Job  Salary
 12  13   C      20
 13  14   B      50]

Split dataframe into equal parts and store each part as a separate data frame

You want list2env:

list2env(setNames(x,paste0("df",1:10)),environment())
# df3
# x         y
# 21 21 0.4935413
# 22 22 0.1862176
# 23 23 0.8273733
# 24 24 0.6684667
# 25 25 0.7942399
# 26 26 0.1079436
# 27 27 0.7237109
# 28 28 0.4112744
# 29 29 0.8209463
# 30 30 0.6470602

Pandas - splitting dataframe into equal rows and assign number in new column as case_id in increasing order from 1 and so on

You can try this:

import pandas as pd

n = 4 # number of rows in each chunk
data = {"id": [0,1,2,3,4,5,6,7],
        "col1": ["a", "b", "c", "d", "e", "f", "g", "h"],
        "col2": ["a", "b", "c", "d", "e", "f", "g", "h"]
        }
df = pd.DataFrame.from_dict(data)
length = len(df)
df["new_col_case_id"] = df["id"].apply(lambda x: int(x/n) + 1)
df = df.set_index("id") #optional
print(df)

output:

   col1 col2  new_col_case_id
id                           
0     a    a                1
1     b    b                1
2     c    c                1
3     d    d                1
4     e    e                2
5     f    f                2
6     g    g                2
7     h    h                2

Split large Dataframe into smaller equal dataframes

I don't know from your description if you are aware that np.array_split outputs n objects. If it's only a few objects you could manually assign them, for example:

df1, df2, df3 = np.array_split(df, 3)

This would assign every subarray to these variables in order.
Otherwise you could assign the series of subarrays to a single variable;

split_df = np.array_split(df, 3)
len(split_df)
# 3

then loop over this one variable and do your analysis per subarray. I would personally choose the latter.

for object in split_df:
    print(type(object))

This prints <class 'pandas.core.frame.DataFrame'> three times.

Split dataframe into 3 equally sized new dataframes - Pandas

Try using numpy.array_split:

import numpy as np
df1, df2, df3 = np.array_split(df_seen, 3)

To save each DataFrame to a separate file, you could do:

for i, df in enumerate(np.array_split(df_seen, 3)):
    df.to_csv(f"data{i+1}.csv", index=False)

Randomly split a dataframe in n equal pieces

You can use cut with split to do this:

k <- 4
str(
    lapply(df_list, function(df) {
        n <- nrow(df)
        split(df, cut(sample(n), seq(0, (k+1)*ceiling(n/k), by=ceiling(n/k)), labels=FALSE))
    })
)

output:

List of 8
 $ AB_df:List of 4
  ..$ 1:'data.frame':   51 obs. of  2 variables:
  .. ..$ X1: int [1:51] 13 50 84 6 109 125 88 120 38 41 ...
  .. ..$ X2: int [1:51] 36 107 14 71 92 115 130 126 111 67 ...
  ..$ 2:'data.frame':   50 obs. of  2 variables:
  .. ..$ X1: int [1:50] 42 105 73 72 36 69 110 43 118 121 ...
  .. ..$ X2: int [1:50] 23 81 96 52 97 42 82 102 89 89 ...
  ..$ 3:'data.frame':   50 obs. of  2 variables:
  .. ..$ X1: int [1:50] 67 128 78 105 32 41 19 86 69 120 ...
  .. ..$ X2: int [1:50] 116 85 55 1 110 96 67 101 73 48 ...
  ..$ 4:'data.frame':   50 obs. of  2 variables:
  .. ..$ X1: int [1:50] 20 104 33 83 73 24 23 129 44 69 ...
  .. ..$ X2: int [1:50] 13 100 74 14 0 59 55 80 72 2 ...
 $ BC_df:List of 4
  ..$ 1:'data.frame':   51 obs. of  2 variables:
  .. ..$ X1: int [1:51] 58 85 40 68 30 32 111 96 35 51 ...
  .. ..$ X2: int [1:51] 71 24 12 50 87 61 17 65 11 43 ...
  ..$ 2:'data.frame':   50 obs. of  2 variables:
  .. ..$ X1: int [1:50] 10 54 91 105 65 39 26 78 123 12 ...
  .. ..$ X2: int [1:50] 117 31 6 114 73 11 58 93 106 21 ...
.........................................................

data:

set.seed(0L)    
AB_df = data.frame(replicate(2,sample(0:130,201,rep=TRUE)))
BC_df = data.frame(replicate(2,sample(0:130,200,rep=TRUE)))
DE_df = data.frame(replicate(2,sample(0:130,197,rep=TRUE)))
FG_df = data.frame(replicate(2,sample(0:130,203,rep=TRUE)))

AB_pc = data.frame(replicate(2,sample(0:130,201,rep=TRUE)))
BC_pc = data.frame(replicate(2,sample(0:130,200,rep=TRUE)))
DE_pc = data.frame(replicate(2,sample(0:130,197,rep=TRUE)))
FG_pc = data.frame(replicate(2,sample(0:130,203,rep=TRUE)))

df_list = list(AB_df, BC_df, DE_df, FG_df, AB_pc, BC_pc, DE_pc, FG_pc)
names(df_list) = c("AB_df", "BC_df", "DE_df", "FG_df", "AB_pc", "BC_pc", "DE_pc", "FG_pc")

How do I split a data frame into groups of a fixed size?

This will give you a list of DataFrames:

lst = [df.iloc[i:i+group_size] for i in range(0,len(df)-group_size+1,group_size)]

It just uses built-in indexing, so it should be pretty fast. The fidgeting with the stop index takes care of discarding the last frame if it's too small - you can also break it down with

lst = [df.iloc[i:i+group_size] for i in range(0,len(df),group_size)]
if len(lst[-1]) < group_size:
   lst.pop()

---

Related Topics