Pandas Dataframe Concat VS Append

Pandas DataFrame concat vs append

So what are you doing is with append and concat is almost equivalent. The difference is the empty DataFrame. For some reason this causes a big slowdown, not sure exactly why, will have to look at some point. Below is a recreation of basically what you did.

I almost always use concat (though in this case they are equivalent, except for the empty frame);
if you don't use the empty frame they will be the same speed.

In [17]: df1 = pd.DataFrame(dict(A = range(10000)),index=pd.date_range('20130101',periods=10000,freq='s'))

In [18]: df1
Out[18]: 
<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 10000 entries, 2013-01-01 00:00:00 to 2013-01-01 02:46:39
Freq: S
Data columns (total 1 columns):
A    10000  non-null values
dtypes: int64(1)

In [19]: df4 = pd.DataFrame()

The concat

In [20]: %timeit pd.concat([df1,df2,df3])
1000 loops, best of 3: 270 us per loop

This is equavalent of your append

In [21]: %timeit pd.concat([df4,df1,df2,df3])
10 loops, best of 

 3: 56.8 ms per loop

Why and when use append() instead of concat() in Pandas?

append is a convenience method which calls concat under the hood. If you look at the implementation of the append method, you will see that.

def append(...
    ...
    if isinstance(other, (list, tuple)):
        to_concat = [self, *other]
    else:
        to_concat = [self, other]
    return concat(
        to_concat,
        ignore_index=ignore_index,
        verify_integrity=verify_integrity,
        sort=sort,
    )

As for the performance. Both of these called over and over in a loop can be computationally expensive. You should just create a list and do one concatenation after you are done looping.

From docs:

iteratively appending rows to a DataFrame can be more computationally
intensive than a single concatenate. A better solution is to append
those rows to a list and then concatenate the list with the original
DataFrame all at once.

How to convert DataFrame.append() to pandas.concat()?

You can store the DataFrames generated in the loop in a list and concatenate them with features once you finish the loop.

In other words, replace the loop:

for count in range(num_samples):
    # .... code to produce `input_vars`
    features = features.append(input_vars)        # remove this `DataFrame.append`

with the one below:

tmp = []                                  # initialize list
for count in range(num_samples):
    # .... code to produce `input_vars`
    tmp.append(input_vars)                        # append to the list, (not DF)
features = pd.concat(tmp)                         # concatenate after loop

You can certainly concatenate in the loop but it's more efficient to do it only once.

Appending row to dataframe with concat()

You can transform your dict in pandas DataFrame

import pandas as pd
df = pd.DataFrame(columns=['Name', 'Weight', 'Sample'])
for key in my_dict:
  ...
  #transform your dic in DataFrame
  new_df = pd.DataFrame([row])
  df = pd.concat([df, new_df], axis=0, ignore_index=True)

Pandas Dataframe concat: is it correct to understand append as a simplified version of concat with few kwargs and can only operate on axis=0

Yes, pd.append simply calls pd.concat with the default arguments axis=0, join='outer' which you can see in the return statement. It also has limited functionality, so you can't use it to construct a hierarchical index.

pd.append source

    from pandas.core.reshape.concat import concat
    if isinstance(other, (list, tuple)):
        to_concat = [self] + other
    else:
        to_concat = [self, other]
    return concat(to_concat, ignore_index=ignore_index,
                  verify_integrity=verify_integrity,
                  sort=sort)

Is pd.append() the quickest way to join two dataframes?

When you have multiple appends in series, it is often more efficient to create a list of dataframes and to concatenate it at the end than using the pd.append function at each iteration since there is some overhead with the pandas functions.

For example,

%%timeit
dfs= []

for i in range(10000):
    tmp1 = pd.DataFrame([[1,2,3],[1,2,3],[1,2,3]])
    dfs.append(tmp1)
pd.concat(dfs)

gives 1.44 s ± 88.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
where the same implementation but using append at each iteration gives
2.81 s ± 126 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Good alternative to Pandas .append() method, now that it is being deprecated?

Create a list with your dictionaries, if they are needed, and then create a new dataframe with df = pd.DataFrame.from_records(your_list). List's "append" method are very efficient and won't be ever deprecated. Dataframes on the other hand, frequently have to be recreated and all data copied over on appends, due to their design - that is why they deprecated the method

---

Related Topics