How to Save Training History on Every Epoch in Keras

keras: how to save the training history attribute of the history object

What I use is the following:

    with open('/trainHistoryDict', 'wb') as file_pi:
        pickle.dump(history.history, file_pi)

In this way I save the history as a dictionary in case I want to plot the loss or accuracy later on.

How to get history over one epoch after every batch

I assume you want to save the accuracy and loss at the end of each batch. To do that you need to create a custom callback as shown below

class BCP(keras.callbacks.Callback):
    batch_accuracy = [] # accuracy at given batch
    batch_loss = [] # loss at given batch    
    def __init__(self):
        super(BCP,self).__init__() 
    def on_train_batch_end(self, batch, logs=None):                
        BCP.batch_accuracy.append(logs.get('accuracy'))
        BCP.batch_loss.append(logs.get('loss'))

now in model.fit include

callbacks = [BCP()]

now train for 1 epoch. at the end of the epoch the values of the accuracy and loss for each batch is stored in BCP.batch_accuracy and BCP.batch_loss. You can print them out
as follows:

print('{0:^4s}{1:^22s}{2:^10s}'.format('Batch', 'Loss', 'Accuracy'))
for i in range (len(BCP.batch_accuracy)):
    print('{0:^4s}{1:15.5f}{2:15.2f}'.format(str(i), BCP.batch_loss[i], BCP.batch_accuracy[i]* 100))

Keras: How to store history after each epoch?

You can use a dictionary to store the history.

history_dict = dict()

for i in range(num_epochs):
    x, y = generate_data()
    history_dict['epoch_%i' % i] = model.fit(x, y, epochs=1, batch_size=64)

It will give something like that:

Out[4]: 
{'epoch_0': <your history 1>,
 'epoch_1': <your history 2>,
 'epoch_2': <your history 3>,

Save history of model.fit for different epochs

You can achieve this functionality by creating a class which sub-classes tf.keras.callbacks.Callback and use the object of that class as callback to model.fit.

import csv
import tensorflow.keras.backend as K
from tensorflow import keras
import os

model_directory='./xyz' # directory to save model history after every epoch 

class StoreModelHistory(keras.callbacks.Callback):

  def on_epoch_end(self,batch,logs=None):
    if ('lr' not in logs.keys()):
      logs.setdefault('lr',0)
      logs['lr'] = K.get_value(self.model.optimizer.lr)

    if not ('model_history.csv' in os.listdir(model_directory)):
      with open(model_directory+'model_history.csv','a') as f:
        y=csv.DictWriter(f,logs.keys())
        y.writeheader()

    with open(model_directory+'model_history.csv','a') as f:
      y=csv.DictWriter(f,logs.keys())
      y.writerow(logs)


model.fit(...,callbacks=[StoreModelHistory()])

Then you can load the csv file and plot model's loss, learning rate, metrics, etc.

import pandas as pd
import matplotlib.pyplot as plt

EPOCH = 10 # number of epochs the model has trained for

history_dataframe = pd.read_csv(model_directory+'model_history.csv',sep=',')


# Plot training & validation loss values
plt.style.use("ggplot")
plt.plot(range(1,EPOCH+1),
         history_dataframe['loss'])
plt.plot(range(1,EPOCH+1),
         history_dataframe['val_loss'],
         linestyle='--')
plt.title('Model loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'], loc='upper left')
plt.show()

How do I save my keras model after every epoch?

Solved:

ModelCheckpoint('models\\unet_seg_{epoch:04d}.model', monitor=observe_var, save_best_only=False, period = 2)

How to extract and save the model from Callback history in TensorFlow

I am assuming you want to save the model at the time the training early stops.

Refer to:

https://keras.io/api/models/model_saving_apis/

https://keras.io/api/callbacks/model_checkpoint/

A simple model.save() should work (you can do so after the model.fit()) or if you want to have checkpoints every time the loss reaches a minimum then use the modelCheckpoint callback.

Is there a way to save a model at a specified epoch in tf.keras?

You can implement a Custom callback like this:

class CustomModelCheckpoint(tf.keras.callbacks.Callback):
    def on_epoch_end(self, epoch, logs=None):
        # logs is a dictionary
        print(f"epoch: {epoch}, train_acc: {logs['acc']}, valid_acc: {logs['val_acc']}")
        if logs['val_acc'] > logs['acc']: # your custom condition
            self.model.save('model.h5', overwrite=True)

cbk = CustomModelCheckpoint()
model.fit(....callbacks=[cbk]...)

---

Related Topics