Emotion Classification using CNN in Python with Keras

Working on a Python project where I needed to classify human emotions from facial expressions. The goal was simple, build a deep learning model that could recognize emotions like happiness, sadness, anger, and surprise from images.

The challenge, however, was figuring out the best way to do this efficiently using Python. After some trial and error, I found that using Convolutional Neural Networks (CNNs) with Keras made the process much easier and more effective.

In this tutorial, I’ll walk you through how to build a Python emotion classification CNN using Keras, step by step. I’ll explain everything in a way that’s easy to follow, even if you’re new to deep learning.

Table of Contents

What is Emotion Classification in Python?

Emotion classification is the process of identifying the emotional state of a person based on data, usually images, text, or audio. In this tutorial, we’ll focus on facial emotion recognition using image data.

Using Python and Keras, we can train a CNN model that learns to detect patterns in facial features, like smiles, frowns, or raised eyebrows, and classify them into different emotions.

Use CNN with Keras for Emotion Detection

CNNs are particularly powerful for image-based tasks because they automatically capture spatial hierarchies in data. In simpler terms, they can detect edges, shapes, and patterns that help in recognizing emotions.

Keras, on the other hand, provides a simple and flexible Python API for building neural networks. It’s built on top of TensorFlow, making it both beginner-friendly and production-ready.

Dataset for Emotion Classification

For this tutorial, I’ll use the FER-2013 dataset, which is a popular open-source dataset for facial emotion recognition. It contains grayscale images of faces categorized into seven emotions:

Angry
Disgust
Fear
Happy
Sad
Surprise
Neutral

You can easily download it from Kaggle’s Facial Expression Recognition dataset.

Set Up the Python Environment

Before we start coding, make sure you have the following Python libraries installed:

pip install tensorflow keras numpy pandas matplotlib seaborn opencv-python

Once installed, we’re ready to build the CNN model.

Step 1 – Import Required Python Libraries

Let’s start by importing all the necessary modules for our CNN model.

We’ll use TensorFlow and Keras for the deep learning part, and Matplotlib and Seaborn for visualization.

import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import cv2
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D, Dropout, Flatten, Dense, BatchNormalization, Activation
from keras.optimizers import Adam
from keras.utils import to_categorical

This code sets up the foundation for our Python emotion classification project using Keras.

Step 2 – Load and Preprocessing the Dataset

Next, we’ll load the FER-2013 dataset and prepare it for training. The dataset is usually in CSV format, where each image is represented as pixel values.

data = pd.read_csv('fer2013.csv')

# Display first few rows
print(data.head())

# Extract features and labels
X = []
y = []

for i in range(len(data)):
    pixels = np.array(data['pixels'][i].split(), dtype='float32')
    image = pixels.reshape(48, 48, 1)
    X.append(image)
    y.append(data['emotion'][i])

X = np.array(X)
y = to_categorical(np.array(y))

# Normalize pixel values
X = X / 255.0

Here, we reshape each image to 48×48 pixels and normalize the values between 0 and 1 for better model performance.

Step 3 – Split the Dataset

We’ll now split the dataset into training and testing sets. This helps the model learn from one part of the data and then test its performance on unseen images.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

This ensures that 80% of the data is used for training and 20% for testing.

Step 4 – Data Augmentation

To make our model more robust, we’ll augment the training images. Data augmentation helps the CNN generalize better by creating slightly modified versions of existing images.

train_datagen = ImageDataGenerator(
    rotation_range=30,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True
)

train_generator = train_datagen.flow(X_train, y_train, batch_size=64)

This step increases the diversity of images without collecting new data, a common Python deep learning trick.

Step 5 – Build the CNN Model in Keras

Now comes the most exciting part, building our CNN architecture. We’ll use multiple convolutional and pooling layers, followed by dense layers for classification.

emotion_model = Sequential()

emotion_model.add(Conv2D(32, (3,3), padding='same', input_shape=(48,48,1)))
emotion_model.add(BatchNormalization())
emotion_model.add(Activation('relu'))
emotion_model.add(MaxPooling2D(pool_size=(2,2)))
emotion_model.add(Dropout(0.25))

emotion_model.add(Conv2D(64, (3,3), padding='same'))
emotion_model.add(BatchNormalization())
emotion_model.add(Activation('relu'))
emotion_model.add(MaxPooling2D(pool_size=(2,2)))
emotion_model.add(Dropout(0.25))

emotion_model.add(Conv2D(128, (3,3), padding='same'))
emotion_model.add(BatchNormalization())
emotion_model.add(Activation('relu'))
emotion_model.add(MaxPooling2D(pool_size=(2,2)))
emotion_model.add(Dropout(0.25))

emotion_model.add(Flatten())
emotion_model.add(Dense(128))
emotion_model.add(BatchNormalization())
emotion_model.add(Activation('relu'))
emotion_model.add(Dropout(0.5))
emotion_model.add(Dense(7, activation='softmax'))

emotion_model.compile(optimizer=Adam(learning_rate=0.0001), loss='categorical_crossentropy', metrics=['accuracy'])
emotion_model.summary()

This CNN architecture is efficient for emotion recognition tasks and performs well with the FER-2013 dataset.

Step 6 – Train the CNN Model

Now that our model is ready, let’s train it using the training data. We’ll run it for 50 epochs to allow the CNN to learn deep patterns.

history = emotion_model.fit(
    train_generator,
    validation_data=(X_test, y_test),
    epochs=50,
    batch_size=64
)

During training, the model adjusts its parameters to minimize the loss and improve accuracy.

Step 7 – Evaluate Model Performance

After training, we’ll evaluate how well our model performs on the test data. This helps us understand if the model can generalize to new, unseen images.

loss, accuracy = emotion_model.evaluate(X_test, y_test)
print(f"Test Accuracy: {accuracy*100:.2f}%")

You should expect accuracy between 65% to 75%, depending on your system and training time.

Step 8 – Visualize Training Results

To get a better sense of how our model performed, we can visualize the accuracy and loss over epochs.

plt.figure(figsize=(12,4))
plt.subplot(1,2,1)
plt.plot(history.history['accuracy'], label='Train Accuracy')
plt.plot(history.history['val_accuracy'], label='Validation Accuracy')
plt.legend()
plt.title('Model Accuracy')

plt.subplot(1,2,2)
plt.plot(history.history['loss'], label='Train Loss')
plt.plot(history.history['val_loss'], label='Validation Loss')
plt.legend()
plt.title('Model Loss')

plt.show()

This visualization helps identify if the model is overfitting or underfitting.

Step 9 – Make Predictions

Once trained, you can use the model to predict emotions from new images. Here’s how you can test it with any facial image using OpenCV.

def predict_emotion(image_path):
    img = cv2.imread(image_path, cv2.IMREAD_GRAYSCALE)
    img = cv2.resize(img, (48,48))
    img = np.expand_dims(img, axis=-1)
    img = np.expand_dims(img, axis=0)
    img = img / 255.0

    prediction = emotion_model.predict(img)
    emotion_labels = ['Angry','Disgust','Fear','Happy','Sad','Surprise','Neutral']
    return emotion_labels[np.argmax(prediction)]

print(predict_emotion('test_image.jpg'))

This small Python function reads an image, processes it, and outputs the predicted emotion label.

Step 10 – Save and Loading the Model

You can save your trained model and reuse it later without retraining.

emotion_model.save('emotion_model_keras.h5')

To load it again:

from keras.models import load_model
model = load_model('emotion_model_keras.h5')

You can refer to the screenshot below to see the output.

Emotion Classification using CNN in Python with Keras

This is useful when deploying your Python emotion recognition model to production.

Tips for Better Accuracy

Use more data or fine-tune pre-trained models like VGG16 or ResNet50.
Increase training epochs and use learning rate schedulers.
Experiment with different activation functions such as LeakyReLU.
Always normalize and augment your dataset properly.

Building an emotion classification CNN using Python and Keras is a rewarding project that combines computer vision and deep learning.

With just a few lines of Python code, you can create a powerful model capable of recognizing human emotions from facial expressions.

Once you understand the basics, you can extend this project to real-time emotion detection using a webcam, which can be useful in areas like mental health monitoring, customer feedback analysis, and even gaming.