How to Append Dataframes Inside a for Loop in Python

Append values of for loop to dataframe

You could build the dataframe directly, without any appending etc.:

def name():
    return pd.DataFrame(
        ([i.Name, i.SystemName] for i in conn.Win32_LogicalDisk()
         if i.Name is not None),
        columns=['NAME', 'SystemName']
    )

If you need to do additional tasks with the dataframe in the function:

def name():
    df = pd.DataFrame(
        ([i.Name, i.SystemName] for i in conn.Win32_LogicalDisk()
         if i.Name is not None),
        columns=['NAME', 'SystemName']
    )
    #  ... do the tasks ... 
    return df

Appending pandas Dataframe to each other inside loop

A general approach is to create a list of dataframes, then pd.concat them once at the end. Something like this:

dfs = []

tables = driver.find_elements_by_tag_name('table')
table = tables[1].get_attribute('outerHTML')
df_i = pd.read_html(table)
df = pd.concat(df_i)
dfs.append(df)

while True:
    try:
        driver.find_element_by_xpath('//a[@title="Next Page"]').click()
        time.sleep(3)
        tables = driver.find_elements_by_tag_name('table')
        table = tables[1].get_attribute('outerHTML')
        df_x = pd.read_html(table)
        df = pd.concat(df_x)
        dfs.append(df)
    except:
        break

df = pd.concat(dfs)
df.to_excel(f'Handloom/Handloom_{str(lofi)}.xlsx')

Python Append dataframe generated in nested loops

Try:

• Change this biglist.append(tem_list) to this: biglist.append(pd.concat(tem_list)).

• Remove this line: biglist1 = [item for sublist in biglist for item in sublist]

• Modify this one df = pd.concat(biglist1) to df = pd.concat(biglist)

---

If you have defined column names, you can also create an empty DataFrame outside your looping scope, and append the data directly on it from your inner loop:

# Before loop
colnames = ['y1', 'y2', 'y3']
df = pd.DataFrame(data=None, columns=colnames)

chaging your append lines to a single one inside your inner loop:

df = df.append(tem_df)

Not needed the use of biglist, tem_list or pd.concat.

---

Edit after user comments:

biglist = []
for i in range (x1,...,x8):
    for j in range ([y1,y2,y3],[y4,..]...[y22,y23,y24]):
        tem_df = pd.DataFrame({'y1':[value1],'y2':[value2],'y3':[value3]},index=i)
    biglist.append(pd.concat(tem_df),axis=1)
df = pd.concat(biglist)
print(df)

---

Related Topics