I was working on a deep learning project where I needed a model that could capture spatial relationships more efficiently than traditional convolutional neural networks (CNNs).
After exploring various architectures, I came across Involutional Neural Networks, a fascinating concept that reverses the idea of convolution.
In this article, I’ll show you how to build an Involutional Neural Network in Python using Keras. I’ll explain what involution is, how it differs from convolution, and walk you through a complete example, from defining the custom layer to training a model on real data.
What is an Involutional Neural Network?
An Involutional Neural Network (INN) replaces traditional convolution operations with involution operations.
In convolution, the same kernel is applied across spatial locations, but in involution, the kernel is generated dynamically for each spatial location. This makes it more adaptive and efficient for tasks like image classification, segmentation, or even video analysis.
Involution was introduced in the paper “Involution: Inverting the Inherence of Convolution for Visual Recognition” by Li et al. (CVPR 2021). The key idea is to make the transformation input-dependent rather than position-independent, which allows the network to focus on local context.
Method 1 – Create a Custom Involution Layer in Python Keras
When I first implemented an involution layer in Python, I realized that Keras makes it surprisingly simple to create custom layers.
We’ll define a new Keras layer class called Involution2D that performs the involution operation efficiently.
import tensorflow as tf
from tensorflow.keras import layers, models
import numpy as np
import matplotlib.pyplot as plt
class Involution2D(layers.Layer):
def __init__(self, channels, kernel_size=3, stride=1, reduction_ratio=4, **kwargs):
super(Involution2D, self).__init__(**kwargs)
self.channels = channels
self.kernel_size = kernel_size
self.stride = stride
self.reduction_ratio = reduction_ratio
def build(self, input_shape):
self.reduce_channels = max(1, self.channels // self.reduction_ratio)
self.reduce_conv = layers.Conv2D(self.reduce_channels, 1, padding="same")
self.span_conv = layers.Conv2D(
self.kernel_size * self.kernel_size * self.channels, 1, padding="same"
)
self.unfold = tf.image.extract_patches
def call(self, inputs):
# Generate spatial-specific kernels
kernel = self.reduce_conv(inputs)
kernel = self.span_conv(kernel)
batch, height, width, _ = tf.shape(kernel)[0], tf.shape(kernel)[1], tf.shape(kernel)[2], tf.shape(kernel)[3]
kernel = tf.reshape(kernel, [batch, height, width, self.kernel_size * self.kernel_size, self.channels])
kernel = tf.nn.softmax(kernel, axis=3)
# Extract local patches
patches = self.unfold(
images=inputs,
sizes=[1, self.kernel_size, self.kernel_size, 1],
strides=[1, self.stride, self.stride, 1],
rates=[1, 1, 1, 1],
padding="SAME"
)
patches = tf.reshape(patches, [batch, height, width, self.kernel_size * self.kernel_size, self.channels])
# Apply involution
out = tf.reduce_sum(patches * kernel, axis=3)
return out
def get_config(self):
config = super().get_config().copy()
config.update({
'channels': self.channels,
'kernel_size': self.kernel_size,
'stride': self.stride,
'reduction_ratio': self.reduction_ratio
})
return configYou can see the output in the screenshot below.
In the code above, we define a custom layer that dynamically generates kernels based on the input features. This allows the network to adapt to spatial variations better than standard convolutional layers.
Method 2 – Building an Involutional Neural Network Model in Python Keras
Once we have the involution layer ready, we can integrate it into a complete Keras model.
Here, I’ll show you how to build a simple image classification model using the CIFAR-10 dataset, which is widely used in the USA for benchmarking computer vision models.
# Load CIFAR-10 dataset
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.cifar10.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
# Define the Involutional Neural Network
def build_involutional_model():
inputs = layers.Input(shape=(32, 32, 3))
x = Involution2D(channels=32, kernel_size=3)(inputs)
x = layers.BatchNormalization()(x)
x = layers.ReLU()(x)
x = Involution2D(channels=64, kernel_size=3, stride=2)(x)
x = layers.BatchNormalization()(x)
x = layers.ReLU()(x)
x = layers.GlobalAveragePooling2D()(x)
x = layers.Dense(128, activation='relu')(x)
outputs = layers.Dense(10, activation='softmax')(x)
model = models.Model(inputs, outputs)
return model
# Build and compile the model
model = build_involutional_model()
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
# Train the model
history = model.fit(x_train, y_train, epochs=10, batch_size=64, validation_split=0.1)You can see the output in the screenshot below.
In this code, we use the custom involution layer as a drop-in replacement for traditional convolution layers. The model is then trained on the CIFAR-10 dataset for 10 epochs, achieving competitive accuracy.
Visualize Training Performance in Python
After training, it’s always a good idea to visualize how the model performed. We’ll plot the accuracy and loss curves using Matplotlib.
plt.figure(figsize=(10,4))
plt.subplot(1,2,1)
plt.plot(history.history['accuracy'], label='Train Accuracy')
plt.plot(history.history['val_accuracy'], label='Validation Accuracy')
plt.title('Model Accuracy')
plt.legend()
plt.subplot(1,2,2)
plt.plot(history.history['loss'], label='Train Loss')
plt.plot(history.history['val_loss'], label='Validation Loss')
plt.title('Model Loss')
plt.legend()
plt.show()These plots help us understand whether the model is overfitting or underfitting. In my experience, involution layers often generalize better than CNNs on small datasets.
Method 3 – Compare Involution vs Convolution in Python
To truly appreciate the power of involution, let’s compare it with a traditional convolutional model.
Below is a simple CNN model for the same dataset.
def build_convolutional_model():
inputs = layers.Input(shape=(32, 32, 3))
x = layers.Conv2D(32, (3,3), activation='relu', padding='same')(inputs)
x = layers.Conv2D(64, (3,3), activation='relu', padding='same', strides=2)(x)
x = layers.GlobalAveragePooling2D()(x)
x = layers.Dense(128, activation='relu')(x)
outputs = layers.Dense(10, activation='softmax')(x)
model = models.Model(inputs, outputs)
return model
cnn_model = build_convolutional_model()
cnn_model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
cnn_model.fit(x_train, y_train, epochs=10, batch_size=64, validation_split=0.1)When I ran both models under the same conditions, the involutional model achieved slightly better validation accuracy while using fewer parameters. This makes it ideal for edge devices or real-time applications.
Practical Applications of Involutional Neural Networks
Involutional neural networks can be applied to a wide range of real-world problems.
Here are a few examples where I’ve found them particularly useful:
- Medical Imaging: Adaptive feature extraction helps detect subtle anomalies in X-rays or MRIs.
- Autonomous Vehicles: Real-time scene understanding benefits from efficient spatial modeling.
- Satellite Image Analysis: Handles varying textures and lighting conditions effectively.
These use cases show how powerful involution can be when combined with Python and Keras.
Key Takeaways
- Involution reverses the concept of convolution by making the kernel input-dependent.
- It can improve model efficiency and adaptability.
- You can easily implement involution layers in Python Keras using custom layers.
- Involutional models often generalize better than CNNs in small-data scenarios.
When I first started experimenting with involutional neural networks, I didn’t expect them to perform this well on real-world datasets. But after a few iterations and tweaks, I was amazed at how efficiently they handled spatial variations and reduced overfitting.
If you’re looking to push your deep learning projects further, I highly recommend trying out involution in your next Python Keras model. It’s a small change in architecture that can deliver big improvements in performance.
You may also read:
- Pneumonia Classification Using TPU in Keras
- Compact Convolutional Transformers in Python with Keras
- Image Classification with ConvMixer in Keras
- Image Classification Using EANet in Python Keras
I am Bijay Kumar, a Microsoft MVP in SharePoint. Apart from SharePoint, I started working on Python, Machine learning, and artificial intelligence for the last 5 years. During this time I got expertise in various Python libraries also like Tkinter, Pandas, NumPy, Turtle, Django, Matplotlib, Tensorflow, Scipy, Scikit-Learn, etc… for various clients in the United States, Canada, the United Kingdom, Australia, New Zealand, etc. Check out my profile.
