A reader running a small photo archive asked whether a tiny Keras model could beat plain resizing when you enlarge low-resolution scans.
It can. Image super-resolution with ESPCN (an efficient sub-pixel CNN) learns to upscale a picture 3x, and after about 129 seconds of CPU training it beat bicubic resizing on two photos it never saw.
Here’s what we’re building:

In this tutorial, I’ll build image super-resolution in Keras step by step: training patches, the ESPCN model with depth_to_space, a short training run, and an honest PSNR test against bicubic.
Before You Start: install TensorFlow, Keras and scikit-image
You need Python 3.10+ and pip install tensorflow scikit-image matplotlib; TensorFlow 2.16+ installs Keras 3 for you. My runs used Python 3.12.5, TensorFlow 2.21.0, Keras 3.15.1 and scikit-image 0.26.0, CPU only. Setup help is in how to install TensorFlow.
If neural networks are new to you, start with building a neural network in TensorFlow and come back.
I didn’t download any dataset. The training photos ship inside scikit-image, so the whole build works offline:
| Data | Images | Used for |
|---|---|---|
skimage.data photos | 16 (astronaut, coffee, rocket, camera, moon, retina …) | 2,400 random 96×96 training crops |
data.chelsea() | 1 cat photo, 451×300 | Test image (never trained on) |
data.stereo_motorcycle()[0] | 1 street photo, 741×500 | Second test image |
Save the helper module below as espcn.py. Every script in this build imports it:
"""ESPCN x3 helpers: sample images, patches, model and PSNR."""
import os, logging
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3" # hide TensorFlow C++ logs
os.environ["TF_ENABLE_ONEDNN_OPTS"] = "0" # and the oneDNN notice
logging.getLogger("tensorflow").setLevel(logging.ERROR)
import numpy as np
import keras
from keras import layers, ops
from skimage import color, data
SCALE = 3
TRAIN = ["astronaut", "coffee", "rocket", "camera", "coins", "moon",
"immunohistochemistry", "hubble_deep_field", "brick", "grass",
"gravel", "cell", "page", "text", "colorwheel", "retina"]
def luma(img):
"""Y (brightness) channel in 0-1, cropped to a multiple of SCALE."""
img = img.astype("float32") / 255.0
if img.ndim == 3:
img = color.rgb2ycbcr(img[..., :3])[..., 0] / 255.0
h, w = (s - s % SCALE for s in img.shape[:2])
return img[:h, :w]
def shrink(hr):
"""Bicubic downscale by SCALE: the low-res input the model sees."""
size = (hr.shape[1] // SCALE, hr.shape[2] // SCALE)
lr = ops.image.resize(hr, size, interpolation="bicubic", antialias=True)
return np.clip(ops.convert_to_numpy(lr), 0, 1)
def make_patches(count=2400, size=96, seed=7):
"""Random 96x96 crops of the training images + their 32x32 inputs."""
rng = np.random.default_rng(seed)
images = [luma(getattr(data, name)()) for name in TRAIN]
hr = np.empty((count, size, size, 1), "float32")
for i in range(count):
img = images[i % len(images)]
y = rng.integers(0, img.shape[0] - size)
x = rng.integers(0, img.shape[1] - size)
hr[i, ..., 0] = img[y:y + size, x:x + size]
return shrink(hr), hr
@keras.saving.register_keras_serializable(package="espcn")
class DepthToSpace(layers.Layer):
"""Sub-pixel shuffle: (h, w, c*r*r) -> (h*r, w*r, c)."""
def __init__(self, scale, **kwargs):
super().__init__(**kwargs)
self.scale = scale
def call(self, x):
return ops.depth_to_space(x, self.scale)
def get_config(self):
return {**super().get_config(), "scale": self.scale}
def build_espcn(channels=1):
inputs = keras.Input(shape=(None, None, channels))
x = layers.Conv2D(64, 5, padding="same", activation="relu")(inputs)
x = layers.Conv2D(64, 3, padding="same", activation="relu")(x)
x = layers.Conv2D(32, 3, padding="same", activation="relu")(x)
x = layers.Conv2D(channels * SCALE**2, 3, padding="same")(x)
return keras.Model(inputs, DepthToSpace(SCALE)(x), name="espcn")
def psnr(a, b):
"""Peak signal-to-noise ratio in dB for images in the 0-1 range."""
return float(10 * np.log10(1.0 / np.mean((a - b) ** 2)))
How does sub-pixel convolution (depth_to_space) work in Keras?
ESPCN keeps every convolution at the small input size, which is why it’s fast. The last layer outputs scale x scale channels per pixel, and depth_to_space rearranges those channels into a bigger image.
PyTorch calls the same operation PixelShuffle (more on the two frameworks in TensorFlow vs PyTorch). A 2×2 image with 4 channels makes it easy to see:
import numpy as np
import espcn # quiet TensorFlow logs
from keras import ops
# one 2x2 image with 4 channels = 2*2 sub-pixels per pixel
x = np.arange(16, dtype="float32").reshape(1, 2, 2, 4)
print("input shape: ", x.shape)
print("top-left pixel channels:", x[0, 0, 0].tolist())
y = ops.convert_to_numpy(ops.depth_to_space(x, 2))
print("output shape:", y.shape)
print(y[0, ..., 0])
Output:
input shape: (1, 2, 2, 4)
top-left pixel channels: [0.0, 1.0, 2.0, 3.0]
output shape: (1, 4, 4, 1)
[[ 0. 1. 4. 5.]
[ 2. 3. 6. 7.]
[ 8. 9. 12. 13.]
[10. 11. 14. 15.]]

So no pixel is invented by resizing: the network learns what each sub-pixel should be. For a look at what conv layers learn, see visualizing convnet layers in Keras.
Step 1: Cut training patches from the sample images
Super-resolution training needs pairs: a sharp high-res crop as the target and its shrunken copy as the input. make_patches() crops 96×96 pieces of the brightness channel and shrinks them 3x with bicubic resizing.
import espcn
lr, hr = espcn.make_patches()
print("training images: ", len(espcn.TRAIN))
print("low-res inputs: ", lr.shape, lr.dtype)
print("high-res targets:", hr.shape, hr.dtype)
print(f"pixel range: {hr.min():.3f} to {hr.max():.3f}")
Output:
training images: 16
low-res inputs: (2400, 32, 32, 1) float32
high-res targets: (2400, 96, 96, 1) float32
pixel range: 0.000 to 1.000

Checkpoint: you get (2400, 32, 32, 1) inputs and (2400, 96, 96, 1) targets.
I train on the Y (brightness) channel only, like the original paper; color comes back later from bicubic. Random crops act as light augmentation, a theme learning to resize in Keras takes further.
Stronger augmentation tricks such as MixUp in Keras are built for classifiers, so I kept plain crops here.
Step 2: Build the ESPCN model with depth_to_space
The model is three Conv2D layers that extract features, a fourth with 1 x 3 x 3 = 9 output channels, and the DepthToSpace layer from espcn.py. The input shape is (None, None, 1), so any image size works.
import espcn
model = espcn.build_espcn()
model.summary()
Output:
Model: "espcn"
┌─────────────────────────────────┬────────────────────────┬───────────────┐
│ Layer (type) │ Output Shape │ Param # │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ input_layer (InputLayer) │ (None, None, None, 1) │ 0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d (Conv2D) │ (None, None, None, 64) │ 1,664 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_1 (Conv2D) │ (None, None, None, 64) │ 36,928 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_2 (Conv2D) │ (None, None, None, 32) │ 18,464 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_3 (Conv2D) │ (None, None, None, 9) │ 2,601 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ depth_to_space (DepthToSpace) │ (None, None, None, 1) │ 0 │
└─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 59,657 (233.04 KB)
Trainable params: 59,657 (233.04 KB)
Non-trainable params: 0 (0.00 B)

Checkpoint: the summary ends with Total params: 59,657. That’s tiny next to a simple MNIST convnet in Keras, which needs a Dense head.
Note: I use a registered custom layer, not a Lambda, so the saved model loads again later. The troubleshooting section shows why.
Step 3: Train ESPCN with model.fit()
Mean squared error is the loss, because PSNR (peak signal-to-noise ratio, the standard super-resolution score) comes straight from MSE. Ten epochs with Adam keeps this CPU-friendly:
import time
import keras
import espcn
keras.utils.set_random_seed(42) # same start weights every run
lr, hr = espcn.make_patches()
model = espcn.build_espcn()
model.compile(optimizer=keras.optimizers.Adam(1e-3), loss="mse")
start = time.perf_counter()
model.fit(lr, hr, batch_size=32, epochs=10, validation_split=0.1,
ver