How to Concatenate Two Layers in Keras

How to concatenate two layers in keras?

You're getting the error because result defined as Sequential() is just a container for the model and you have not defined an input for it.

Given what you're trying to build set result to take the third input x3.

first = Sequential()
first.add(Dense(1, input_shape=(2,), activation='sigmoid'))

second = Sequential()
second.add(Dense(1, input_shape=(1,), activation='sigmoid'))

third = Sequential()
# of course you must provide the input to result which will be your x3
third.add(Dense(1, input_shape=(1,), activation='sigmoid'))

# lets say you add a few more layers to first and second.
# concatenate them
merged = Concatenate([first, second])

# then concatenate the two outputs

result = Concatenate([merged,  third])

ada_grad = Adagrad(lr=0.1, epsilon=1e-08, decay=0.0)

result.compile(optimizer=ada_grad, loss='binary_crossentropy',
               metrics=['accuracy'])

However, my preferred way of building a model that has this type of input structure would be to use the functional api.

Here is an implementation of your requirements to get you started:

from keras.models import Model
from keras.layers import Concatenate, Dense, LSTM, Input, concatenate
from keras.optimizers import Adagrad

first_input = Input(shape=(2, ))
first_dense = Dense(1, )(first_input)

second_input = Input(shape=(2, ))
second_dense = Dense(1, )(second_input)

merge_one = concatenate([first_dense, second_dense])

third_input = Input(shape=(1, ))
merge_two = concatenate([merge_one, third_input])

model = Model(inputs=[first_input, second_input, third_input], outputs=merge_two)
ada_grad = Adagrad(lr=0.1, epsilon=1e-08, decay=0.0)
model.compile(optimizer=ada_grad, loss='binary_crossentropy',
               metrics=['accuracy'])

To answer the question in the comments:

1. How are result and merged connected? Assuming you mean how are they concatenated.

Concatenation works like this:

  a        b         c
a b c   g h i    a b c g h i
d e f   j k l    d e f j k l

i.e rows are just joined.

2. Now, x1 is input to first, x2 is input into second and x3 input into third.

Concatenate two layers in keras, tensorflow

The best (most flexible,elegant) solution is to use the Functional API in Keras.

Here is a working solution. Notice I am using the Model() (Functional API) instantiation and not Sequential():

from tensorflow.keras import Model
image_input = keras.layers.Input((32, 32, 3))
output_cnn_1 = cnn1(image_input)
output_left = module1_left(output_cnn_1)
output_middle = module1_middle(output_cnn_1)
output_right = module1_right(output_cnn_1)
concatenated_output = keras.layers.Concatenate()([output_left,output_middle,output_right])
final_model = Model(inputs=image_input, outputs=concatenated_output)
final_model.summary()

Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_29 (InputLayer)           [(None, 32, 32, 3)]  0                                            
__________________________________________________________________________________________________
sequential_11 (Sequential)      (None, 28, 28, 32)   2432        input_29[0][0]                   
__________________________________________________________________________________________________
sequential_12 (Sequential)      (None, 28, 28, 32)   1056        sequential_11[8][0]              
__________________________________________________________________________________________________
sequential_13 (Sequential)      (None, 28, 28, 64)   19552       sequential_11[8][0]              
__________________________________________________________________________________________________
sequential_14 (Sequential)      (None, 28, 28, 32)   1056        sequential_11[8][0]              
__________________________________________________________________________________________________
concatenate_15 (Concatenate)    (None, 28, 28, 128)  0           sequential_12[8][0]              
                                                                 sequential_13[8][0]              
                                                                 sequential_14[8][0]              
==================================================================================================
Total params: 24,096
Trainable params: 24,096
Non-trainable params: 0
__________________________________________________________________________________________________
    

Where the definition has slightly changed (we do not declare the input for each of the modules, since we are using the output of the cnn1).

import tensorflow
from tensorflow import keras
from tensorflow.keras import layers

cnn1 = keras.Sequential([
    layers.Conv2D(32, (5, 5), activation='relu')
    ]
)

'''Module 1'''
module1_left = keras.Sequential([
    layers.Conv2D(32, (1, 1), activation='relu', padding='same')
    ]
)
module1_middle = keras.Sequential([
    layers.Conv2D(32, (1, 1), activation='relu', padding='same'),
    layers.Conv2D(64, (3, 3), activation='relu', padding='same')
    ]
)
module1_right = keras.Sequential([
    layers.MaxPooling2D((3, 3), padding='same', strides=(1, 1)),
    layers.Conv2D(32, (1, 1), activation='relu', padding='same')
    ]
)

Add vs Concatenate layer in Keras

As you said, both of them combine input, but they combine in a different way.
their name already suggest their usage

Add() inputs are added together,

For example (assume batch_size=1)

x1 = [[0, 1, 2]]
x2 = [[3, 4, 5]]
x = Add()([x1, x2])

then x should be [[3, 5, 7]], where each element is added

notice that the input shape is (1, 3) and (1, 3), the output is also (1, 3)

Concatenate() concatenates the output,

For example (assume batch_size=1)

x1 = [[0, 1, 2]]
x2 = [[3, 4, 5]]
x = Concatenate()([x1, x2])

then x should be [[0, 1, 2, 3, 4, 5]], where the inputs are horizontally stacked together,

notice that the input shape is (1, 3) and (1, 3), the output is also (1, 6),

even when the tensor has more dimensions, similar behaviors still apply.

Concatenate creates a bigger model for an obvious reason, the output size is simply the size of all inputs summed, while add has the same size with one of the inputs

---

For more information about add/concatenate, and other ways to combine multiple inputs, see this

Concatenate two layers

You are trying to concatenate 2 models but what you want is to concatenate 2 layers. Try the following code.

from tensorflow.keras.models import Sequential, Model
from tensorflow.keras.layers import Flatten, Input

def cnn_model_fn(learning_rate):
    """Model function for CNN."""
    input_layer=Input(shape=(410,1,3))

    x1 = (tf.keras.layers.Conv2D(
          filters=20,
          kernel_size=[10, 1],
          kernel_initializer='he_uniform',
          bias_initializer=keras.initializers.Constant(value=0),
          padding="same",
          activation=tf.nn.relu ))(input_layer)
    x1 = Flatten()(x1)

    x2 = (tf.keras.layers.Conv2D(
          filters=20,
          kernel_size=[10, 1],
          kernel_initializer='he_uniform',
          bias_initializer=keras.initializers.Constant(value=0),
          padding="same",
          activation=tf.nn.relu))(input_layer)
    x2 = Flatten()(x2)

    x = (keras.layers.Concatenate(axis=-1)([x1,x2]))

    model = Model(input_layer, x)
    optimizer = tf.train.AdamOptimizer(learning_rate)
    model.compile(loss='mean_squared_error',
                optimizer=optimizer,
                metrics=['mean_absolute_error', 'mean_squared_error'])

    return model

How do I fit the model of two concatenate LSTM in keras?

You need to use Functional API in Keras.

Your model with Functional API:

from tensorflow.keras.layers import Dense, Concatenate, LSTM, Input, Flatten
from tensorflow.keras.models import Model

look_back = 3200  # just for running the code I used this number

# Model architecture 

inputs = Input(shape=(1,look_back),name='Input_1')

lstm1 = LSTM(6, name='LSTM_1')(inputs)

lstm2 = LSTM(6, name='LSTM_2')(inputs)

concatenated = Concatenate( name='Concatenate_1')([lstm1,lstm2])

output1 = Dense(1, name='Dense_1')(concatenated)

model = Model(inputs=inputs, outputs=output1)

Now let's see the architecture:

model.summary()

output:

Model: "functional_13"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
Input_1 (InputLayer)            [(None, 1, 3200)]    0                                            
__________________________________________________________________________________________________
LSTM_1 (LSTM)                   (None, 6)            76968       Input_1[0][0]                    
__________________________________________________________________________________________________
LSTM_2 (LSTM)                   (None, 6)            76968       Input_1[0][0]                    
__________________________________________________________________________________________________
Concatenate_1 (Concatenate)     (None, 12)           0           LSTM_1[0][0]                     
                                                                 LSTM_2[0][0]                     
__________________________________________________________________________________________________
Dense_1 (Dense)                 (None, 1)            13          Concatenate_1[0][0]              
==================================================================================================
Total params: 153,949
Trainable params: 153,949
Non-trainable params: 0
__________________________________________________________________________________________________

Also, we can take a better look at the layers and their connections by checking the model plot

The model plot:

from tensorflow.keras.utils import plot_model

plot_model(model)

output:

Now let's train the model:

I created some dummy data with sklearn for training the model and everything works fine.

Training the model:

from sklearn.datasets import make_blobs
train_x , train_y = make_blobs(n_samples=1000, centers=2, n_features=look_back,random_state=0)

train_x = train_x.reshape(train_x.shape[0], 1, train_x.shape[1])

model.compile(loss='mean_squared_error', optimizer='adam', metrics = ['mse'])
model.fit(train_x, train_y, epochs=5, batch_size=1, verbose=1)

output:

Epoch 1/5
1000/1000 [==============================] - 2s 2ms/step - loss: 0.0133 - mse: 0.0133
Epoch 2/5
1000/1000 [==============================] - 2s 2ms/step - loss: 1.4628e-13 - mse: 1.4628e-13
Epoch 3/5
1000/1000 [==============================] - 2s 2ms/step - loss: 2.2808e-14 - mse: 2.2808e-14
Epoch 4/5
1000/1000 [==============================] - 2s 2ms/step - loss: 5.2458e-15 - mse: 5.2458e-15
Epoch 5/5
1000/1000 [==============================] - 2s 2ms/step - loss: 1.1384e-15 - mse: 1.1384e-15
<tensorflow.python.keras.callbacks.History at 0x7f5fe4ce9f28>

Keras: How to equalize the dimensions of 2 layers for Concatenation

For your problem, the reason why you cannot concatenate is of course the height and width of the feature maps(thanks for pointing that out in the comment section, I misunderstood you question at first).

The reason why you cannot concatenate is in this line of code:

x2 = AveragePooling2D(pool_size=(2,2))(x2)

1. If you remove this AveragePooling2D() then the dimension is not
   reduced and you can concatenate on axis=3 (or -1 for that
   matter). 37 comes from integer division by 2 (both on height and
   width) ==> 75 // 2 = 37.
2. There is another trick that you could do. You could use Upsampling2D((2,2)) to arrive back from (37,37) to (74,74). However this is not enough as, due to the remainder of an integer division by 2 from an odd number, we still need to process the width and height to arrive to (75,75) from (74,74). You can solve this by using a ZeroPadding2D layer, such as ZeroPadding2D((1,0),(0,0))

Another thing to consider(providing the same dimensions of the filters) is for number of filters reduction(read this if you are also interested in this aspect):

You can use 1x1 Convolution for reducing the dimension of the filters.

Here is an example:

# example of a 1x1 filter for dimensionality reduction
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D
# create model
model = Sequential()
model.add(Conv2D(512, (3,3), padding='same', activation='relu', input_shape=(256, 256, 3)))
model.add(Conv2D(64, (1,1), activation='relu'))
# summarize model
model.summary()

This is the output of the model.summary()

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d_1 (Conv2D)            (None, 256, 256, 512)     14336
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 256, 256, 64)      32832
=================================================================
Total params: 47,168
Trainable params: 47,168
Non-trainable params: 0
_________________________________________________________________

The example is taken from here: https://machinelearningmastery.com/introduction-to-1x1-convolutions-to-reduce-the-complexity-of-convolutional-neural-networks/. I wrote the code since the link may break in the future but the code is visible and can be left as it is.

Of course, the example is in the Sequential() format but the gist is this line of code(which you have to slightly modify to your purpose):

model.add(Conv2D(64, (1,1), activation='relu'))

---

Related Topics