Build a Siamese Network for Image Similarity in Keras

I’ve found that one of the most common challenges is comparing two images to see how similar they are.

Standard classification models fail here because they require a fixed set of classes, whereas similarity tasks often involve unseen categories.

Siamese Networks are the perfect solution for this, especially when you need to verify signatures or compare product photos for an e-commerce site.

In this tutorial, I will show you how to build a robust image similarity model using a Siamese Network and a contrastive loss function in Keras.

Table of Contents

Set Up the Siamese Network Environment in Keras

Before we dive into the architecture, I always make sure my environment is ready with the necessary deep learning libraries.

I use TensorFlow and Keras for this, along with Matplotlib to visualize the image pairs during the training process.

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np
import matplotlib.pyplot as plt
import random

Prepare Image Pairs for Keras Training

The core idea of a Siamese Network is to learn from pairs of images that are either “similar” (same class) or “dissimilar” (different classes).

I create a helper function that generates these pairs, ensuring the model sees an equal balance of positive and negative examples.

def create_pairs(x, digit_indices):
    pairs = []
    labels = []
    n = min([len(digit_indices[d]) for d in range(10)]) - 1
    
    for d in range(10):
        for i in range(n):
            # Positive pair (same digit)
            z1, z2 = digit_indices[d][i], digit_indices[d][i + 1]
            pairs += [[x[z1], x[z2]]]
            labels += [1]
            
            # Negative pair (different digit)
            inc = random.randrange(1, 10)
            dn = (d + inc) % 10
            z1, z2 = digit_indices[d][i], digit_indices[dn][i]
            pairs += [[x[z1], x[z2]]]
            labels += [0]
            
    return np.array(pairs), np.array(labels).astype("float32")

Design the Base Feature Extractor in Keras

Both branches of the Siamese Network share the same weights, so I built a single “base” model to extract features from the images.

I typically use a few convolutional layers followed by a flattening layer to create a compact vector representation of the input.

def build_base_network(input_shape):
    inputs = layers.Input(shape=input_shape)
    x = layers.Conv2D(32, (3, 3), activation="relu")(inputs)
    x = layers.MaxPooling2D(pool_size=(2, 2))(x)
    x = layers.Conv2D(64, (3, 3), activation="relu")(x)
    x = layers.Flatten()(x)
    x = layers.Dense(128, activation="relu")(x)
    return keras.Model(inputs, x)

input_shape = (28, 28, 1)
base_network = build_base_network(input_shape)

Implement Euclidean Distance in Keras

Since we are comparing two feature vectors, I need a custom layer to calculate the Euclidean distance between them.

This layer takes the outputs from the two Siamese branches and returns a single value representing how far apart the images are in vector space.

def euclidean_distance(vects):
    x, y = vects
    sum_square = tf.reduce_sum(tf.square(x - y), axis=1, keepdims=True)
    return tf.sqrt(tf.maximum(sum_square, tf.keras.backend.epsilon()))

# Define inputs for the Siamese Network
input_a = layers.Input(shape=input_shape)
input_b = layers.Input(shape=input_shape)

# Shared weights through the base network
feat_a = base_network(input_a)
feat_b = base_network(input_b)

# Distance calculation
distance = layers.Lambda(euclidean_distance)([feat_a, feat_b])

Define the Contrastive Loss Function in Keras

The contrastive loss is the secret sauce that makes Siamese Networks work by pulling similar images together and pushing dissimilar ones apart.

I implement this by penalizing large distances for positive pairs and small distances (within a margin) for negative pairs.

def contrastive_loss(y_true, y_pred):
    margin = 1.0
    square_pred = tf.square(y_pred)
    margin_square = tf.square(tf.maximum(margin - y_pred, 0))
    return tf.reduce_mean(y_true * square_pred + (1 - y_true) * margin_square)

Compile and Train the Keras Siamese Model

With the architecture and loss function ready, I compile the model using the Adam optimizer and start the training process.

I find that monitoring the validation loss is crucial here to ensure the model generalizes well to images it hasn’t seen before.

siamese_model = keras.Model(inputs=[input_a, input_b], outputs=distance)

siamese_model.compile(loss=contrastive_loss, optimizer="adam")

# Assuming pairs_train and labels_train are prepared
# history = siamese_model.fit(
#     [pairs_train[:, 0], pairs_train[:, 1]], labels_train,
#     validation_split=0.2,
#     batch_size=32,
#     epochs=10
# )

Evaluate Similarity Predictions in Keras

After training, I use the model to predict the similarity between two new images to see how well it performs.

A lower distance score indicates that the model perceives the images as being very similar or identical.

def compute_accuracy(y_true, y_pred):
    pred = y_pred.ravel() < 0.5
    return np.mean(pred == y_true)

# predictions = siamese_model.predict([test_pairs[:, 0], test_pairs[:, 1]])
# accuracy = compute_accuracy(test_labels, predictions)
# print(f"Model Accuracy: {accuracy:.2f}")

Visualize Image Comparison Results in Keras

I always find it helpful to visualize the results by plotting image pairs alongside their predicted similarity scores.

This helps me verify if the model is struggling with specific patterns, such as distinguishing between a “9” and a “4” in handwriting.

def visualize_results(pairs, labels, predictions, n=5):
    plt.figure(figsize=(10, 4))
    for i in range(n):
        ax = plt.subplot(2, n, i + 1)
        plt.imshow(pairs[i, 0].reshape(28, 28), cmap="gray")
        plt.axis("off")
        
        ax = plt.subplot(2, n, i + 1 + n)
        plt.imshow(pairs[i, 1].reshape(28, 28), cmap="gray")
        plt.title(f"Dist: {predictions[i][0]:.2f}")
        plt.axis("off")
    plt.show()

You can refer to the screenshot below to see the output.

Build a Siamese Network for Image Similarity in Keras

In this tutorial, I have shown you how to construct a Siamese Network from scratch using Keras and how to apply a contrastive loss for image similarity.

By following these steps, you can build models that are capable of sophisticated image verification tasks beyond simple classification.

You may also read:

Bijay Kumar

I am Bijay Kumar, a Microsoft MVP in SharePoint. Apart from SharePoint, I started working on Python, Machine learning, and artificial intelligence for the last 5 years. During this time I got expertise in various Python libraries also like Tkinter, Pandas, NumPy, Turtle, Django, Matplotlib, Tensorflow, Scipy, Scikit-Learn, etc… for various clients in the United States, Canada, the United Kingdom, Australia, New Zealand, etc. Check out my profile.

enjoysharepoint.com/