Which Seeds Have to Be Set Where to Realize 100% Reproducibility of Training Results in Tensorflow

Which seeds have to be set where to realize 100% reproducibility of training results in tensorflow?

The best solution which works as of today with GPU is to install tensorflow-determinism with the following:

pip install tensorflow-determinism

Then include the following code to your code

import tensorflow as tf
import os
os.environ['TF_DETERMINISTIC_OPS'] = '1'

source: https://github.com/NVIDIA/tensorflow-determinism

Seed for reproducible results is not working (Tensorflow)

I suggest, after

weights = {
    'conv1/conv2d': tf.get_variable('conv1/weights', shape=[3,3,512,1024], initializer=tf.contrib.layers.xavier_initializer()),
    # and more ...
}

store the weight externally in a file, for example, then next time you run, do not go through that previous line and load the weight from the external file.

TensorFlow: Non-repeatable results

You need to set operation level seed in addition to graph-level seed, ie

tf.reset_default_graph()
a = tf.constant([1, 1, 1, 1, 1], dtype=tf.float32)
graph_level_seed = 1
operation_level_seed = 1
tf.set_random_seed(graph_level_seed)
b = tf.nn.dropout(a, 0.5, seed=operation_level_seed)

Do I need to set seed in all modules where I import numpy or tensorflow?

First of all, use the tensorflow.keras instead of keras.

Usually, it suffices to use the seed in the main script in the following manner.

import random
random.seed(1)
import numpy as np
np.random.seed(1)
import tensorflow as tf
tf.random.set_seed(1)

But, if you have multiple modules and they have some randomized operation (such as weight initialization), then add these lines to each of your module.

Additionally, these only don't guarantee 100% reproducibility, if you are using a GPU, there maybe some randomness due to that too.

You can use https://github.com/NVIDIA/tensorflow-determinism

os.environ['TF_DETERMINISTIC_OPS'] = '1' For tensorflow==2.1.0

For tensorflow < 2.1

import tensorflow as tf
from tfdeterminism import patch
patch()

---

Related Topics