Image Super-Resolution in Keras with ESPCN (Sub-Pixel CNN)

A reader running a small photo archive asked whether a tiny Keras model could beat plain resizing when you enlarge low-resolution scans.

It can. Image super-resolution with ESPCN (an efficient sub-pixel CNN) learns to upscale a picture 3x, and after about 129 seconds of CPU training it beat bicubic resizing on two photos it never saw.

Here’s what we’re building:

Matplotlib window comparing a low-res cat photo, bicubic x3 upscaling, Keras ESPCN x3 upscaling and the original, with PSNR in dB
The finished model on the scikit-image cat photo: ESPCN scores 31.88 dB against 31.52 dB for bicubic, a difference that’s subtle to the eye.

In this tutorial, I’ll build image super-resolution in Keras step by step: training patches, the ESPCN model with depth_to_space, a short training run, and an honest PSNR test against bicubic.

Before You Start: install TensorFlow, Keras and scikit-image

You need Python 3.10+ and pip install tensorflow scikit-image matplotlib; TensorFlow 2.16+ installs Keras 3 for you. My runs used Python 3.12.5, TensorFlow 2.21.0, Keras 3.15.1 and scikit-image 0.26.0, CPU only. Setup help is in how to install TensorFlow.

If neural networks are new to you, start with building a neural network in TensorFlow and come back.

I didn’t download any dataset. The training photos ship inside scikit-image, so the whole build works offline:

DataImagesUsed for
skimage.data photos16 (astronaut, coffee, rocket, camera, moon, retina …)2,400 random 96×96 training crops
data.chelsea()1 cat photo, 451×300Test image (never trained on)
data.stereo_motorcycle()[0]1 street photo, 741×500Second test image

Save the helper module below as espcn.py. Every script in this build imports it:

"""ESPCN x3 helpers: sample images, patches, model and PSNR."""
import os, logging
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3"        # hide TensorFlow C++ logs
os.environ["TF_ENABLE_ONEDNN_OPTS"] = "0"       # and the oneDNN notice
logging.getLogger("tensorflow").setLevel(logging.ERROR)
import numpy as np
import keras
from keras import layers, ops
from skimage import color, data

SCALE = 3
TRAIN = ["astronaut", "coffee", "rocket", "camera", "coins", "moon",
         "immunohistochemistry", "hubble_deep_field", "brick", "grass",
         "gravel", "cell", "page", "text", "colorwheel", "retina"]


def luma(img):
    """Y (brightness) channel in 0-1, cropped to a multiple of SCALE."""
    img = img.astype("float32") / 255.0
    if img.ndim == 3:
        img = color.rgb2ycbcr(img[..., :3])[..., 0] / 255.0
    h, w = (s - s % SCALE for s in img.shape[:2])
    return img[:h, :w]


def shrink(hr):
    """Bicubic downscale by SCALE: the low-res input the model sees."""
    size = (hr.shape[1] // SCALE, hr.shape[2] // SCALE)
    lr = ops.image.resize(hr, size, interpolation="bicubic", antialias=True)
    return np.clip(ops.convert_to_numpy(lr), 0, 1)


def make_patches(count=2400, size=96, seed=7):
    """Random 96x96 crops of the training images + their 32x32 inputs."""
    rng = np.random.default_rng(seed)
    images = [luma(getattr(data, name)()) for name in TRAIN]
    hr = np.empty((count, size, size, 1), "float32")
    for i in range(count):
        img = images[i % len(images)]
        y = rng.integers(0, img.shape[0] - size)
        x = rng.integers(0, img.shape[1] - size)
        hr[i, ..., 0] = img[y:y + size, x:x + size]
    return shrink(hr), hr


@keras.saving.register_keras_serializable(package="espcn")
class DepthToSpace(layers.Layer):
    """Sub-pixel shuffle: (h, w, c*r*r) -> (h*r, w*r, c)."""

    def __init__(self, scale, **kwargs):
        super().__init__(**kwargs)
        self.scale = scale

    def call(self, x):
        return ops.depth_to_space(x, self.scale)

    def get_config(self):
        return {**super().get_config(), "scale": self.scale}


def build_espcn(channels=1):
    inputs = keras.Input(shape=(None, None, channels))
    x = layers.Conv2D(64, 5, padding="same", activation="relu")(inputs)
    x = layers.Conv2D(64, 3, padding="same", activation="relu")(x)
    x = layers.Conv2D(32, 3, padding="same", activation="relu")(x)
    x = layers.Conv2D(channels * SCALE**2, 3, padding="same")(x)
    return keras.Model(inputs, DepthToSpace(SCALE)(x), name="espcn")


def psnr(a, b):
    """Peak signal-to-noise ratio in dB for images in the 0-1 range."""
    return float(10 * np.log10(1.0 / np.mean((a - b) ** 2)))

How does sub-pixel convolution (depth_to_space) work in Keras?

ESPCN keeps every convolution at the small input size, which is why it’s fast. The last layer outputs scale x scale channels per pixel, and depth_to_space rearranges those channels into a bigger image.

PyTorch calls the same operation PixelShuffle (more on the two frameworks in TensorFlow vs PyTorch). A 2×2 image with 4 channels makes it easy to see:

import numpy as np
import espcn                      # quiet TensorFlow logs
from keras import ops

# one 2x2 image with 4 channels = 2*2 sub-pixels per pixel
x = np.arange(16, dtype="float32").reshape(1, 2, 2, 4)
print("input shape: ", x.shape)
print("top-left pixel channels:", x[0, 0, 0].tolist())

y = ops.convert_to_numpy(ops.depth_to_space(x, 2))
print("output shape:", y.shape)
print(y[0, ..., 0])

Output:

input shape:  (1, 2, 2, 4)
top-left pixel channels: [0.0, 1.0, 2.0, 3.0]
output shape: (1, 4, 4, 1)
[[ 0.  1.  4.  5.]
 [ 2.  3.  6.  7.]
 [ 8.  9. 12. 13.]
 [10. 11. 14. 15.]]
VS Code running a Keras ops.depth_to_space demo that turns a 2x2 image with 4 channels into a 4x4 image
The four channels of the top-left pixel become the top-left 2×2 block of the output.

So no pixel is invented by resizing: the network learns what each sub-pixel should be. For a look at what conv layers learn, see visualizing convnet layers in Keras.

Step 1: Cut training patches from the sample images

Super-resolution training needs pairs: a sharp high-res crop as the target and its shrunken copy as the input. make_patches() crops 96×96 pieces of the brightness channel and shrinks them 3x with bicubic resizing.

import espcn

lr, hr = espcn.make_patches()
print("training images: ", len(espcn.TRAIN))
print("low-res inputs:  ", lr.shape, lr.dtype)
print("high-res targets:", hr.shape, hr.dtype)
print(f"pixel range: {hr.min():.3f} to {hr.max():.3f}")

Output:

training images:  16
low-res inputs:   (2400, 32, 32, 1) float32
high-res targets: (2400, 96, 96, 1) float32
pixel range: 0.000 to 1.000
VS Code output showing 2,400 low-res 32x32 input patches and 96x96 high-res target patches for ESPCN training
Every 32×32 input has a 96×96 target, all values between 0 and 1.

Checkpoint: you get (2400, 32, 32, 1) inputs and (2400, 96, 96, 1) targets.

I train on the Y (brightness) channel only, like the original paper; color comes back later from bicubic. Random crops act as light augmentation, a theme learning to resize in Keras takes further.

Stronger augmentation tricks such as MixUp in Keras are built for classifiers, so I kept plain crops here.

Step 2: Build the ESPCN model with depth_to_space

The model is three Conv2D layers that extract features, a fourth with 1 x 3 x 3 = 9 output channels, and the DepthToSpace layer from espcn.py. The input shape is (None, None, 1), so any image size works.

import espcn

model = espcn.build_espcn()
model.summary()

Output:

Model: "espcn"
┌─────────────────────────────────┬────────────────────────┬───────────────┐
│ Layer (type)                    │ Output Shape           │       Param # │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ input_layer (InputLayer)        │ (None, None, None, 1)  │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d (Conv2D)                 │ (None, None, None, 64) │         1,664 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_1 (Conv2D)               │ (None, None, None, 64) │        36,928 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_2 (Conv2D)               │ (None, None, None, 32) │        18,464 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_3 (Conv2D)               │ (None, None, None, 9)  │         2,601 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ depth_to_space (DepthToSpace)   │ (None, None, None, 1)  │             0 │
└─────────────────────────────────┴────────────────────────┴───────────────┘
 Total params: 59,657 (233.04 KB)
 Trainable params: 59,657 (233.04 KB)
 Non-trainable params: 0 (0.00 B)
VS Code showing model.summary() of a Keras ESPCN model with four Conv2D layers and a DepthToSpace layer
Only the last layer changes the spatial size; everything before it runs on the small image.

Checkpoint: the summary ends with Total params: 59,657. That’s tiny next to a simple MNIST convnet in Keras, which needs a Dense head.

Note: I use a registered custom layer, not a Lambda, so the saved model loads again later. The troubleshooting section shows why.

Step 3: Train ESPCN with model.fit()

Mean squared error is the loss, because PSNR (peak signal-to-noise ratio, the standard super-resolution score) comes straight from MSE. Ten epochs with Adam keeps this CPU-friendly:

import time
import keras
import espcn

keras.utils.set_random_seed(42)       # same start weights every run

lr, hr = espcn.make_patches()
model = espcn.build_espcn()
model.compile(optimizer=keras.optimizers.Adam(1e-3), loss="mse")

start = time.perf_counter()
model.fit(lr, hr, batch_size=32, epochs=10, validation_split=0.1,
          ver