Image Super-Resolution with Efficient Sub-Pixel CNN in Keras

Enhancing image resolution often feels like something out of a sci-fi movie where a detective shouts “Enhance!” at a blurry screen. In reality, it is a complex deep learning task that I have spent years perfecting using Keras and TensorFlow.

While working on a surveillance project for a retail chain, I realized that traditional upscaling made faces look like Minecraft characters. That is when I turned to the Efficient Sub-Pixel CNN (ESPCN), a game-changer for real-time high-definition reconstruction.

Table of Contents

Better Resolution with Python Keras Efficient Sub-Pixel CNN

The ESPCN model is unique because it performs all the heavy lifting in the low-resolution space, which saves a massive amount of memory and time. Instead of blowing up an image and then cleaning it, we teach the network to predict the missing pixels directly into a high-resolution grid.

This method uses a specialized layer called “Pixel Shuffle” to rearrange the data. It is remarkably faster than other methods because the convolutional layers process much smaller feature maps before the final assembly.

Step 1: Load Datasets for Python Keras Super-Resolution

I always start by preparing a high-quality dataset, such as the BSDS500, which provides crisp images for our model to learn from. In this step, we download the images and set up our training and validation pipelines to handle US-standard image formats.

import os
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

# Downloading the Berkeley Segmentation Dataset
dataset_url = "http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/BSR/BSR_bsds500.tgz"
data_dir = keras.utils.get_file(origin=dataset_url, fname="BSR", untar=True)
root_dir = os.path.join(data_dir, "BSDS500/data")

def get_dataset(batch_size, image_size):
    return keras.utils.image_dataset_from_directory(
        root_dir,
        batch_size=batch_size,
        image_size=(image_size, image_size),
        validation_split=0.2,
        subset="both",
        seed=1337,
        label_mode=None,
    )

train_ds, valid_ds = get_dataset(batch_size=8, image_size=300)

Step 2: Image Preprocessing for Python Keras ESPCN

Before feeding images to the model, I convert them to the YCbCr color space and focus only on the “Y” channel. This is because the human eye is much more sensitive to brightness (luminance) than color, making the training process far more efficient.

def process_input(input_image, upscale_factor):
    # Scale pixels to [0, 1] and resize for the low-res input
    input_image = input_image / 255.0
    img_width = tf.shape(input_image)[1]
    img_height = tf.shape(input_image)[2]
    
    # Create the Low-Resolution version (Input)
    lowres_img = tf.image.resize(input_image, [img_width // upscale_factor, img_height // upscale_factor], method="bicubic")
    return lowres_img, input_image

# Mapping the transformation to our datasets
upscale_factor = 3
train_ds = train_ds.map(lambda x: process_input(x, upscale_factor))
valid_ds = valid_ds.map(lambda x: process_input(x, upscale_factor))

Step 3: Build the Python Keras Sub-Pixel Convolutional Model

Now, I will define the ESPCN architecture, which consists of three convolutional layers followed by the magic “Depth to Space” operation. This last layer takes the stacked feature maps and reshapes them into a higher-resolution image.

def build_espcn_model(upscale_factor, channels=3):
    inputs = layers.Input(shape=(None, None, channels))
    
    # Feature extraction in low-resolution space
    x = layers.Conv2D(64, 5, padding="same", activation="relu", kernel_initializer="he_normal")(inputs)
    x = layers.Conv2D(32, 3, padding="same", activation="relu", kernel_initializer="he_normal")(x)
    
    # The sub-pixel convolution layer
    x = layers.Conv2D(channels * (upscale_factor ** 2), 3, padding="same", kernel_initializer="he_normal")(x)
    outputs = tf.nn.depth_to_space(x, upscale_factor)
    
    return keras.Model(inputs, outputs)

model = build_espcn_model(upscale_factor=3)
model.summary()

Step 4: Compile and Training the Python Keras SR Model

I prefer using the Adam optimizer with a lower learning rate to ensure the model converges smoothly without overshooting. We use Mean Squared Error (MSE) as our loss function because it directly penalizes the difference between the predicted and original pixels.

# Compiling with Adam optimizer and MSE loss
model.compile(optimizer=keras.optimizers.Adam(learning_rate=0.001), loss="mse")

# Training the model
history = model.fit(
    train_ds,
    epochs=30,
    validation_data=valid_ds,
    verbose=1
)

Step 5: Test the Result in Python Keras

To see the results, I take a low-resolution photo—like a blurry snapshot of a license plate in a New York parking lot—and run it through our trained model. The output should be significantly sharper and more detailed than standard resizing.

import numpy as np
import matplotlib.pyplot as plt

def display_results(model, test_ds):
    for lowres, highres in test_ds.take(1):
        prediction = model.predict(lowres)
        
        plt.figure(figsize=(15, 5))
        plt.subplot(1, 3, 1)
        plt.title("Low-Res Input")
        plt.imshow(lowres[0])
        
        plt.subplot(1, 3, 2)
        plt.title("ESPCN Prediction")
        plt.imshow(np.clip(prediction[0], 0, 1))
        
        plt.subplot(1, 3, 3)
        plt.title("Original High-Res")
        plt.imshow(highres[0])
        plt.show()

display_results(model, valid_ds)

You can refer to the screenshot below to see the output.

Image Super-Resolution with Efficient Sub-Pixel CNN in Keras

Image Super-Resolution with Efficient Sub-Pixel CNN

In this tutorial, I showed you how to implement an Efficient Sub-Pixel CNN using Python Keras to upscale images. It is a powerful technique that balances speed and quality, making it perfect for real-world applications where performance matters.