How to Merge Two Cnn That Are Trained Over Different Data Stream

Using cnn models trained on different datasets

I think it depends on the data.

You may use three different models and make three binary predictions on each image. So you get a vote (probability) for each x vs. normal. If binary classifications are accurate, this should deliver okay results. But you kind of get a cummulated missclassification or error in this case.

If you can afford, you can train a four class model and compare the test error to the series of binary classifications. I understand that you already have three models. So training another one may be not too expensive.

If ONLY one of the classes can occur, a four class model might be the way to go. If in fact two (or more) classes can occur jointly, a series of binary classifications would make sense.

Merge two different deep learning models in Keras

Sequential models are not suited for creating models with branches.

You can have the two independent models as Sequential models, as you did, but from the Concatenate on, you should start using the functional Model API.

The idea is to get the output tensors of the two models and feed them in other layers to get new output tensors.

So, considering you have model and extra:

mergedOutput = Concatenate()([model.output, extra.output])

This mergetOutput is a tensor. You can either create the last part of the model using this tensor, or create the last part independently, and call it on this tensor. The second approach may be good if you want to train each model separately (doesn't seem to be your case).

Now, creating the new model as a functional API model:

out = Dense(128, activation='relu')(mergetOutput)
out = Dropout(0.8)(out)
out = Dense(32, activation='sigmoid')(out)
out = Dense(num_classes, activation='softmax')(out)

new_model = Model(
    [model.input, extra.input], #model with two input tensors
    out                         #and one output tensor
) 

An easier approach is to take all three models you have already created and use them to create a combined model:

model = Sequential() #your first model
extra = Sequential() #your second model    
new_model = Sequential() #all these three exactly as you did   

#in this case, you just need to add an input shape to new_model, compatible with the concatenated output of the previous models. 
new_model.add(FirstNewModelLayer(...,input_shape=(someValue,)))

Join them like this:

mergedOutput = Concatenate()([model.output, extra.output])
finalOutput = new_model(mergedOutput)    

fullModel = Model([model.input,extra.input],finalOutput)

Related Topics