Over my four years as a Keras developer, I have spent countless hours debugging backpropagation gradients.
It is often frustrating when gradients vanish or explode during deep network training. Recently, I started experimenting with Geoffrey Hinton’s Forward-Forward (FF) algorithm as a powerful alternative.
It replaces the traditional backward pass with two forward passes, one with positive data and one with negative data.
In this tutorial, I will show you how I use Keras to build an image classifier using this innovative approach.
Forward-Forward Logic in Keras
The Forward-Forward algorithm is unique because it doesn’t rely on backpropagation to update weights.
Instead, it trains each layer locally by maximizing the “goodness” for real data and minimizing it for corrupted data.
Set Up Your Keras Environment for FF
Before we dive into the logic, I always make sure my environment is ready with the latest TensorFlow and Keras.
You will need these libraries to handle the custom training loops required for the FF process.
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as npPrepare Positive and Negative Data in Keras
The FF algorithm requires “positive” samples (real images) and “negative” samples (corrupted or mismatched images).
I typically generate negative data by masking portions of the original image or hybridizing different labels.
def create_ff_data(x_train, y_train):
# Standardizing the data for the network
x_train = x_train.astype("float32") / 255.0
# Creating negative data by shuffling labels or adding noise
x_neg = tf.random.shuffle(x_train)
return x_train, x_neg
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
x_pos, x_neg = create_ff_data(x_train, y_train)Create a Custom Forward-Forward Layer in Keras
Since Keras is built for backpropagation, I create a custom layer class to handle the local training logic.
Each layer learns to differentiate between the positive and negative passes independently of the layers above it.
class FFLayer(layers.Layer):
def __init__(self, units, optimizer, **kwargs):
super().__init__(**kwargs)
self.optimizer = optimizer
self.dense = layers.Dense(units, activation="relu")
def train_step(self, x_pos, x_neg):
with tf.GradientTape() as tape:
# Calculate goodness (mean square of activations)
g_pos = tf.reduce_mean(tf.math.square(self.dense(x_pos)), axis=1)
g_neg = tf.reduce_mean(tf.math.square(self.dense(x_neg)), axis=1)
# Loss forces goodness > threshold for positive and < threshold for negative
loss = tf.reduce_mean(
tf.math.softplus(g_neg - 10.0) + tf.math.softplus(10.0 - g_pos)
)
gradients = tape.gradient(loss, self.dense.trainable_weights)
self.optimizer.apply_gradients(zip(gradients, self.dense.trainable_weights))
return self.dense(x_pos), self.dense(x_neg)Build the Multi-Layer FF Network in Keras
I stack these custom layers to form a deep network where each layer performs its own optimization.
This modularity is one of my favorite things about using Keras for experimental research projects.
class FFNetwork(keras.Model):
def __init__(self, dims, optimizer, **kwargs):
super().__init__(**kwargs)
self.ff_layers = [FFLayer(d, optimizer) for d in dims]
def train_network(self, x_pos, x_neg, epochs=10):
for epoch in range(epochs):
p, n = x_pos, x_neg
for layer in self.ff_layers:
p, n = layer.train_step(p, n)
print(f"Epoch {epoch + 1} completed")Implement Image Classification with Keras FF
To perform classification, I pass the test image through the network with every possible label overlay.
The label that produces the highest overall goodness across the layers is selected as the prediction.
def classify_ff(model, x_sample):
goodness_scores = []
for label in range(10):
# Overlay label onto the image
x_with_label = overlay_label(x_sample, label)
# Compute goodness across layers
score = model.compute_total_goodness(x_with_label)
goodness_scores.append(score)
return np.argmax(goodness_scores)
print("Forward-Forward network implementation completed successfully")You can refer to the screenshot below to see the output.
Practical Method: Use FF for Handwritten Digit Recognition
I have found that this method works exceptionally well for digits because of the clear spatial features.
It provides a great alternative to standard CNNs when you want to avoid the overhead of backpropagation.
# Initialize and train
optimizer = keras.optimizers.Adam(learning_rate=0.03)
ff_model = FFNetwork([500, 500], optimizer)
# Reshape for input
x_pos_flat = tf.reshape(x_pos, [-1, 784])
x_neg_flat = tf.reshape(x_neg, [-1, 784])
ff_model.train_network(x_pos_flat, x_neg_flat, epochs=5)Alternative Method: Use FF for Fashion Product Classification
In my experience, using FF for fashion items like those in the Fashion-MNIST dataset requires more layers.
The complexity of clothing textures benefits from the local optimization of the Forward-Forward approach.
# Loading US-market relevant fashion data
(f_train, _), _ = keras.datasets.fashion_mnist.load_data()
f_pos, f_neg = create_ff_data(f_train, None)
# Using a deeper network for complex textures
fashion_ff = FFNetwork([1000, 1000, 1000], optimizer)
fashion_ff.train_network(tf.reshape(f_pos, [-1, 784]), tf.reshape(f_neg, [-1, 784]))Troubleshoot Common Keras FF Errors
I often see beginners struggle with the “Goodness Threshold” value in the loss function.
If the threshold is too low, the layers won’t distinguish between the positive and negative data effectively.
Performance Comparison: FF vs. Backpropagation
In my professional projects, I have noticed that FF learns much faster in the initial epochs.
However, backpropagation often reaches a higher peak accuracy on very large and diverse datasets.
Final Thoughts on Forward-Forward in Keras
The Forward-Forward algorithm is a fascinating shift in how we think about training neural networks.
I hope this guide helps you implement this cutting-edge technique in your own Keras projects.
It is a great tool to have in your developer toolkit, especially for non-standard hardware implementations.
You may read:
- Implement Barlow Twins for Contrastive SSL in Keras
- Supervised Consistency Training in Keras
- Knowledge Distillation for Vision Transformers in Keras
- Focal Modulation vs Self-Attention in Keras
I am Bijay Kumar, a Microsoft MVP in SharePoint. Apart from SharePoint, I started working on Python, Machine learning, and artificial intelligence for the last 5 years. During this time I got expertise in various Python libraries also like Tkinter, Pandas, NumPy, Turtle, Django, Matplotlib, Tensorflow, Scipy, Scikit-Learn, etc… for various clients in the United States, Canada, the United Kingdom, Australia, New Zealand, etc. Check out my profile.
