How to Use Append with Pickle in Python

How to use append with pickle in python?

Pickle streams are entirely self-contained, and so unpickling will unpickle one object at a time.

Therefore, to unpickle multiple streams, you should repeatedly unpickle the file until you get an EOFError:

>>> f=open('a.p', 'wb')
>>> pickle.dump({1:2}, f)
>>> pickle.dump({3:4}, f)
>>> f.close()
>>> 
>>> f=open('a.p', 'rb')
>>> pickle.load(f)
{1: 2}
>>> pickle.load(f)
{3: 4}
>>> pickle.load(f)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
EOFError

so your unpickle code might look like

import pickle
objs = []
while 1:
    try:
        objs.append(pickle.load(f))
    except EOFError:
        break

How to append new data to pickle file using python

file_path = client_dir + '\embeddings.pickle'
file = open(file_path, 'rb')
old_data = pickle.load(file)
new_embeddings = old_data['embeddings']
new_names = old_data['names']
new_embeddings.append(known_embeddings[0])
new_names.append(known_names[0])
data1 = {"embeddings": new_embeddings, "names": new_names}
with open(file_path, 'ab+') as fp:
    pickle.dump(data1, fp)
    fp.close()
log.error("[INFO] Data appended to embeddings.pickle ")

This looks pretty close to being correct to me. You succesfully load the pickled data and add new elements to it. The problem appears to be the with open(file_path, 'ab+') as fp: call. If you open the file in "a" mode, then the pickle data you write will get added to the end, after the old pickle data. Then, on subsequent executions of your program, pickle.load will only load the old pickle data.

Try overwriting the old pickle data completely with your new pickle data. You can do this by opening in "w" mode instead.

with open(file_path, 'wb') as fp:
    pickle.dump(data1, fp)

Incidentally, you don't need that fp.close() call. A with statement automatically closes the opened file at the end of the block.

Python - appending to a pickled list

You need to pull the list from your database (i.e. your pickle file) first before appending to it.

import pickle
import os

high_scores_filename = 'high_scores.dat'

scores = []

# first time you run this, "high_scores.dat" won't exist
#   so we need to check for its existence before we load 
#   our "database"
if os.path.exists(high_scores_filename):
    # "with" statements are very handy for opening files. 
    with open(high_scores_filename,'rb') as rfp: 
        scores = pickle.load(rfp)
    # Notice that there's no "rfp.close()"
    #   ... the "with" clause calls close() automatically! 

first_name = input("Please enter your name:")
score = input("Please enter your score:")

high_scores = first_name, score
scores.append(high_scores)

# Now we "sync" our database
with open(high_scores_filename,'wb') as wfp:
    pickle.dump(scores, wfp)

# Re-load our database
with open(high_scores_filename,'rb') as rfp:
    scores = pickle.load(rfp)

print(scores)

Appending data to pickle in python

import pickle

lst = [1,2,3]

with open("test.dat", "wb") as msg:
    pickle.dump(lst, msg)

with open("test.dat", "ab+") as msg:
    pickle.dump(lst, msg)

with open("test.dat", "rb") as msg:
    print (pickle.load(msg))

It seems using "ab+" makes it.

Output:

[1, 2, 3]
[1, 2, 3]

---

Related Topics