How To Implement The Sigmoid Activation Function In Python?

In this tutorial, I will explain how to implement the sigmoid activation function in Python. As a Python developer, I came across a scenario where I needed to use the sigmoid function and I explored more about the sigmoid function. Let us learn more about this topic today.

Table of Contents

Sigmoid() Function in Python

The sigmoid or logistic function is an S-shaped curve that maps any real-valued number into a value between 0 and 1. The formula for the sigmoid function is:

[ \sigma(x) = \frac{1}{1 + e^{-x}} ]

Where:

( \sigma(x) ) is the output of the sigmoid function.
( x ) is the input value.
( e ) is the base of the natural logarithm, approximately equal to 2.71828.

The sigmoid function is particularly useful in scenarios where we need to model probabilities, such as logistic regression and neural networks.

Read Python input() vs raw_input()

Implement the Sigmoid() Function in Python

Let’s start by implementing the sigmoid function in Python. We will use the NumPy library for efficient array operations.

import numpy as np

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

Check out PyCharm vs. VS Code for Python

Example: Predict Loan Approval

Suppose we are building a machine learning model to predict whether individuals in the USA will be approved for a loan based on their credit score. We can use the sigmoid function to transform the output of our model into a probability.

# Sample data: credit scores of individuals
credit_scores = np.array([650, 720, 580, 810, 690])

# Applying the sigmoid function
probabilities = sigmoid(credit_scores)

print("Credit Scores:", credit_scores)
print("Probabilities:", probabilities)

Output:

Credit Scores: [650 720 580 810 690]
Probabilities: [1. 1. 1. 1. 1.]

I have executed the above example and added the screenshot below.

In this example, the credit scores are transformed into probabilities, which can be interpreted as the likelihood of loan approval.

Read Should I Learn Python or C++?

Visualize the Sigmoid() Function in Python

To better understand how the sigmoid function works, let’s visualize it using Matplotlib.

import matplotlib.pyplot as plt

# Generate a range of values
x = np.linspace(-10, 10, 100)
y = sigmoid(x)

# Plot the sigmoid function
plt.plot(x, y)
plt.title('Sigmoid Function')
plt.xlabel('Input')
plt.ylabel('Output')
plt.grid(True)
plt.show()

I have executed the above example and added the screenshot below.

Implement the Sigmoid Activation Function in Python

This plot shows the characteristic S-shape of the sigmoid function, with the output values smoothly transitioning from 0 to 1.

Check out Should I Learn Java or Python?

Use the Python Sigmoid() Function in Logistic Regression

Logistic regression is a popular machine learning algorithm for binary classification tasks, and the sigmoid function plays a central role in it. Let’s implement a simple logistic regression model using the sigmoid function.

Read Python / vs //

Example: Email Spam Classification

Consider a dataset where we need to classify emails as spam or not spam based on certain features. We will use the sigmoid function to predict the probability of an email being spam.

Step 1: Define the Sigmoid() Function

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

Step 2: Logistic Regression Model

We will use gradient descent to optimize the model parameters.

def logistic_regression(X, y, lr=0.01, epochs=1000):
    m, n = X.shape
    weights = np.zeros(n)
    bias = 0

    for epoch in range(epochs):
        linear_model = np.dot(X, weights) + bias
        y_pred = sigmoid(linear_model)

        # Compute gradients
        dw = (1 / m) * np.dot(X.T, (y_pred - y))
        db = (1 / m) * np.sum(y_pred - y)

        # Update parameters
        weights -= lr * dw
        bias -= lr * db

    return weights, bias

Step 3: Train the Model

# Sample data: features and labels (spam: 1, not spam: 0)
X = np.array([[0.2, 0.8], [0.5, 0.4], [0.9, 0.1], [0.4, 0.6]])
y = np.array([0, 0, 1, 0])

# Train the logistic regression model
weights, bias = logistic_regression(X, y)

print("Weights:", weights)
print("Bias:", bias)

Step 4: Make Predictions

def predict(X, weights, bias):
    linear_model = np.dot(X, weights) + bias
    y_pred = sigmoid(linear_model)
    return [1 if i > 0.5 else 0 for i in y_pred]

# Sample test data
X_test = np.array([[0.6, 0.3], [0.1, 0.9]])

# Predicting spam or not spam
predictions = predict(X_test, weights, bias)
print("Predictions:", predictions)

In this example, we trained a logistic regression model to classify emails as spam or not spam based on two features. The sigmoid function was used to convert the linear model’s output into probabilities, which were then thresholded to make binary predictions.

Check out How to Use the repeat() Function in Python?

Advantages and Limitations of the Sigmoid() Function

Advantages

Probability Interpretation: The output range [0, 1] makes it easy to interpret the results as probabilities.
Smooth Gradient: The smooth gradient helps in the optimization process during training.

Limitations

Vanishing Gradient: For very large or very small input values, the gradient can become very small, slowing down the training process.
Output Saturation: When the input is far from zero, the output saturates, leading to a loss of information.

Read How to Use the ceil() Function in Python?

Conclusion

In this tutorial, we explored the sigmoid activation function and its implementation in Python with examples , and its application in logistic regression. We also discussed visualizing the sigmoid() function in Python and its advantages and limitations.

How to Implement the Sigmoid Activation Function in Python?