How to Use a Pre-Trained Neural Network With Grayscale Images

How can I use a pre-trained neural network with grayscale images?

The model's architecture cannot be changed because the weights have been trained for a specific input configuration. Replacing the first layer with your own would pretty much render the rest of the weights useless.

-- Edit: elaboration suggested by Prune--

CNNs are built so that as they go deeper, they can extract high-level features derived from the lower-level features that the previous layers extracted. By removing the initial layers of a CNN, you are destroying that hierarchy of features because the subsequent layers won't receive the features that they are supposed to as their input. In your case the second layer has been trained to expect the features of the first layer. By replacing your first layer with random weights, you are essentially throwing away any training that has been done on the subsequent layers, as they would need to be retrained. I doubt that they could retain any of the knowledge learned during the initial training.

--- end edit ---

There is an easy way, though, which you can make your model work with grayscale images. You just need to make the image to appear to be RGB. The easiest way to do so is to repeat the image array 3 times on a new dimension. Because you will have the same image over all 3 channels, the performance of the model should be the same as it was on RGB images.

In numpy this can be easily done like this:

print(grayscale_batch.shape)  # (64, 224, 224)
rgb_batch = np.repeat(grayscale_batch[..., np.newaxis], 3, -1)
print(rgb_batch.shape)  # (64, 224, 224, 3)

The way this works is that it first creates a new dimension (to place the channels) and then it repeats the existing array 3 times on this new dimension.

I'm also pretty sure that keras' ImageDataGenerator can load grayscale images as RGB.

Using InceptionV3 for greyscale images

Implementing Inceptionv3 from scratch is easy. So you can follow https://gist.github.com/neggert/f8b86d001a367aa7dde1ab6b587246b5 or any other inceptionv3 model which had been put as open source and the number of channels from 3 to 1. But note these models will not be pre-trained models. So if you have available HPC systems you can give this a try.

Or

Visit https://github.com/keras-team/keras-applications/blob/master/keras_applications/inception_v3.py where they have used weights of the pre-trained model and you can change the number of channels from 3 to 1 in the code. I am not sure about this one because the model was trained on 3 channel images so the weights might not be the ones that we want.

Related Topics