Scikit learn Gradient Descent

In this Python tutorial, we will learn how Scikit learn gradient descent works in python, and we will also cover different examples related to Gradient descent. Moreover, we will cover these topics.

• Scikit learn gradient descent
• Scikit learn gradient descent regression
• Scikit learn stochastic gradient descent classifier
• Scikit learn batch gradient descent
• Scikit learn minibatch gradient descent
• Scikit learn stochastic gradient descent regression
• Scikit learn logistic regression gradient descent
• Scikit learn gradient descent linear regression

Scikit learn gradient descent

In this section, we will learn about how Scikit learn gradient descent works in python.

• Gradient descent is a backbone of machine learning and is used when training a model. It is also combined with each and every algorithm and easily understand.
• Scikit learn gradient descent is a very simple and effective approach for regressor and classifier.
• It also applied to large-scale and machine learning problems and also has experience in text classification, natural language processing.
• In scikit learn gradient descent the gradient of loss guess each and every sample at a time and after that our model is updated.

Code:

In the following code, we import some functions to calculate the loss, hypothesis, and also calculate the gradient.

• hypothesis = num.dot(X, theta) is used to calculate the hypotheses.
• loss = hypothesis – Y is used to calculate the loss.
• cst = num.sum(loss ** 2) / (2 * a) is used to calculate the cost.
• theta = theta – alpha * gradient is used to update the theta parameter.
• Y[i] = (i + bias) + rand.uniform(0, 2) * variance is used as a target variable.
• X,Y = genData(90, 20, 9) is used to generate 90 points with the basis of 20 and 10 variances as a bit of noise.
• print(theta) is used to print the value of theta.

import numpy as num
import random as rand


def gradientDescent(X, Y, theta, alpha, a, numIterations):
    Xtrans = X.transpose()
    for i in range(0, numIterations):
        hypothesis = num.dot(X, theta)
        loss = hypothesis - Y
 

        cst = num.sum(loss ** 2) / (2 * a)
        print("Iteration %d | Cost: %f" % (i, cst))
        gradient = num.dot(Xtrans, loss) / a
        theta = theta - alpha * gradient
    return theta


def genData(numPoints, bias, variance):
    X = num.zeros(shape=(numPoints, 3))
    Y = num.zeros(shape=numPoints)
    # basically a straight line
    for i in range(0, numPoints):
        X[i][0] = 2
        X[i][1] = i
        Y[i] = (i + bias) + rand.uniform(0, 2) * variance
    return X, Y
X,Y = genData(90, 20, 9)
a, b = num.shape(X)
numIterations= 10000
alpha = 0.0004
theta = num.ones(b)
theta = gradientDescent(X,Y, theta, alpha,a, numIterations)
print(theta)

Output:

After running the above code, we get the following output in which we can see that the number of points is generated and the value of theta is printed on the screen.

Read: Scikit-learn logistic regression

Scikit learn gradient descent regression

In this section, we will learn about how Scikit learn gradient descent regression works in python.

• Scikit learn gradient descent regressor is defined as a process that calculates the cost function and supports different loss functions to fit the regressor model.
• The gradient descend regression model is convenient for a large number of training data.

Code:

In the following code, we will import some libraries from which we predict the best-fit regression line.

• self.X = X is used to define the method of a class.
• y_pred = self.predict() function is used to predict the model.
• self.a[0] = self.a[0] – (learningrate * ((1/b) * np.sum(y_pred – y))) is used to update the coefficient of the model.
• J = (1 / 2*b) * (np.sum(y_pred – self.y)**2) is used to compute the cost of the model.
• f = plt.figure(fig) is used to plot the figure on the graph.
• regr.plot_best_fit(y_pred, ‘Initial best fit line’) is used to plot the best fit line.
• plot.plot(range(iterations), costs, color=’r’) is used to verify the decrease in cost function.
• regr.predict([i for i in range(10)]) is used to predict using regressor.

import numpy as num
import matplotlib.pyplot as plot
 
class Linear_Regression:
    def __init__(self, X, y):
        self.X = X
        self.y = y
        self.a = [0, 0]
     
    def update_coeffs(self, learningrate):
        y_pred = self.predict()
        y = self.y
        b = len(y)
        self.a[0] = self.a[0] - (learningrate * ((1/b) *
                                np.sum(y_pred - y)))
 
        self.a[1] = self.a[1] - (learningrate * ((1/b) *
                                np.sum((y_pred - y) * self.X)))
 
    def predict(self, X=[]):
        y_pred = np.array([])
        if not X: X = self.X
        a = self.a
        for x in X:
            y_pred = np.append(y_pred, a[0] + (a[1] * x))
 
        return y_pred
     
    def get_current_accuracy(self, y_pred):
        t, e = y_pred, self.y
        s = len(y_pred)
        return 1-sum(
            [
                abs(t[i]-e[i])/e[i]
                for i in range(s)
                if e[i] != 0]
        )/s

 
    def compute_cost(self, y_pred):
        b = len(self.y)
        J = (1 / 2*b) * (np.sum(y_pred - self.y)**2)
        return J
 
    def plot_best_fit(self, y_pred, fig):
                f = plt.figure(fig)
                plot.scatter(self.X, self.y, color='r')
                plot.plot(self.X, y_pred, color='y')
                f.show()
 
 
def main():
    X = np.array([i for i in range(11)])
    y = np.array([2*i for i in range(11)])
 
    regr = Linear_Regression(X, y)
 
    iterations = 0
    steps = 90
    learningrate = 0.01
    costs = []
   
    y_pred = regr.predict()
    regr.plot_best_fit(y_pred, 'Initial best fit line')
     
 
    while 1:
        y_pred = regr.predict()
        cst = regr.compute_cost(y_pred)
        costs.append(cst)
        regr.update_coeffs(learningrate)
         
        iterations += 1
        if iterations % steps == 0:
            print(iterations, "epochs elapsed")
            print("current accuracy is :",
                regr.get_current_accuracy(y_pred))
            break
 
    #final best-fit line
    regr.plot_best_fit(y_pred, 'Final Best Fit Line')
 
    h = plot.figure('Verification')
    plot.plot(range(iterations), costs, color='r')
    h.show()
 
    regr.predict([i for i in range(10)])
 
if __name__ == '__main__':
    main()

Output:

After running the above code, we get the following code output in which we can see that the current accuracy and regression best fit line is plotted on the screen.

Read: Scikit learn Decision Tree

Scikit learn stochastic gradient descent classifier

In this section, we will learn about how Scikit learn stochastic gradient descent classifier works in python.

Scikit learn stochastic gradient descent classifier is a process to find the value of the coefficient of function which can decrease the cost function.

Code:

In the following code, we will import SDGClassifier from sklearn.linear_model by which we can easily understand every algorithm.

• x = [[1., 1.], [2., 2.]] is used to hold the training sample.
• y = [1, 2] is used to hold the target sample.
• classifier.fit(x, y) is used to fit the classifier.
• classifier.predict([[3., 3.]]) is used to predict the classifier.
• classifier.coef_ is used to get the coefficient of the classifier that hold the model parameters.

from sklearn.linear_model import SGDClassifier
x = [[1., 1.], [2., 2.]]
y = [1, 2]
classifier = SGDClassifier(loss="hinge", penalty="l2", max_iter=7)
classifier.fit(x, y)
classifier.predict([[3., 3.]])
classifier.coef_

Output:

After running the above code, we get the following output in which we can see that the coefficient of the classifier is printed on the screen.

Read: Scikit learn Hierarchical Clustering

Scikit learn stochastic gradient descent regression

In this section, we will learn about how Scikit stochastic gradient descent regression works in python.

Scikit learn stochastic gradient descent regression calculates the cost function and supports the loss function.

Code:

In the following code, we will import SGCRegressor from sklearn.linear_model by which we can calculate the cost of the function and also support different log functions.

• range = num.random.RandomState(0) is used to create a random function.
• y = range.randn(nsamples) is used to hold the target sample.
• x = range.randn(nsamples, nfeatures) is used to hold the training sample.
• regressor = make_pipeline(StandardScaler(),SGDRegressor(max_iter=100, tol=1e-3)) pipeline is used as a setup with the fit and predict function.
• regressor.fit(x, y) is used to fit the model.

import numpy as num
from sklearn.linear_model import SGDRegressor
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
nsamples, nfeatures = 12,7
range = num.random.RandomState(0)
y = range.randn(nsamples)
x = range.randn(nsamples, nfeatures)
#Always scale the input.The most convenient way to use a pipeline.
regressor = make_pipeline(StandardScaler(),
                          SGDRegressor(max_iter=100, tol=1e-3))
regressor.fit(x, y)

Output:

After running the above code, we get the following output in which we can see that the regressor function is printed on the screen.

Read: Scikit learn Hidden Markov Model

Scikit learn batch gradient descent

In this section, we will learn about how Scikit learn batch gradient descent works in python.

• Gradient descent is a process that observes the value of functions parameter which minimize the function cost.
• In Batch gradient descent the entire dataset is used in each step while calculating the gradient. It also builds easy updates inside the model.

Code:

In the following code, we will import make_regression from sklearn.datasets by which the batch gradient descent works.

• s = X.shape[0] is used as a number of samples.
• loss = hypothesis – Y is used to calculate the loss.
• J = num.sum(loss ** 2) / (2 * s) is used to calculate the cost.
• theta = theta – alpha * gradient is used to update he model.
• X = num.c_[ num.ones(s), X] is used to insert the values in columns.
• Y_predict = theta[0] + theta[1]*X is used to predict the values.
• pylab.plot(X[:,1],Y,’r’) is used to plot the graph.

import numpy as num
import random as rand
from sklearn.datasets import make_regression 
import pylab
from scipy import stats

def gradient_descent_2(alpha, X, Y, numIterations):
    s = X.shape[0]
    theta = num.ones(2)
    X_transpose = X.transpose()
    for iter in range(0, numIterations):
        hypothesis = num.dot(X, theta)
        loss = hypothesis - Y
        J = num.sum(loss ** 2) / (2 * s)  
        print ("iter %s | J: %.3f" % (iter, J))    
        gradient = np.dot(X_transpose, loss) / s         
        theta = theta - alpha * gradient  
    return theta

if __name__ == '__main__':

    X, Y = make_regression(n_samples=100, n_features=1, n_informative=1, 
                        random_state=0, noise=35) 
    s, q = num.shape(X)
    X = num.c_[ num.ones(s), X] # insert column
    alpha = 0.01 # learning rate
    theta = gradient_descent_2(alpha, X, Y, 1000)

    for i in range(X.shape[1]):
        Y_predict = theta[0] + theta[1]*X 
    pylab.plot(X[:,1],Y,'r')
    pylab.plot(X,Y_predict,'b-')
    pylab.show()

Output:

After running the above code, we get the following output in which we can see that the batch gradient descent graph is printed on the screen.

Read: Scikit learn Ridge Regression

Scikit learn minibatch gradient descent

In this section, we will learn about how Scikit learn minibatch gradient descent works in python.

• Scikit learn minibatch gradient descent is an alternative of gradient descent it breaks the large dataset set into small batches and calculates the error of a model.
• Minibatch gradient descent can easily and quickly update the parameters. If the batch is large less noisy to update the model.

Code:

In the following code, we will import some libraries from which we can make a minibatch gradient descent graph.

• data = num.random.multivariate_normal(mean, cov, 8000) is used to create the data.
• plot.scatter(data[:600, 0], data[:600, 1], marker = ‘.’) is used to plot or visuialising of the data.
• data = num.hstack((num.ones((data.shape[0], 2)), data)) is used to stack the sequence of the input array columnwise.
• X_train = data[:split, :-1] is used to split the dataset into train data.
• X_test = data[split:, :-1] is used to split the dataset into test data.
• print(“Number of training set = % d”%(X_train.shape[0])) is used to print the number of training dataset.
• print(“Number of testing set = % d”%(X_test.shape[0])) is used to print the number of testing dataset.

import numpy as num
import matplotlib.pyplot as plot
  

mean = num.array([6.0, 7.0])
cov = num.array([[2.0, 0.97], [0.97, 1.3]])
data = num.random.multivariate_normal(mean, cov, 8000)
  

plot.scatter(data[:600, 0], data[:600, 1], marker = '.')
plot.show()
  

data = num.hstack((num.ones((data.shape[0], 2)), data))
  
split_factor = 0.95
split = int(split_factor * data.shape[0])
  
X_train = data[:split, :-1]
y_train = data[:split, -1].reshape((-1, 1))
X_test = data[split:, :-1]
y_test = data[split:, -1].reshape((-1, 1))
  
print("Number of training set = % d"%(X_train.shape[0]))
print("Number of testing set = % d"%(X_test.shape[0]))

Output:

After running the above code, we get the following output in which we can see that the number of training and testing set with the graph is plotted on the screen.

Read: Scikit learn Random Forest

Scikit learn logistic regression gradient descent

In this section, we will learn about how Scikit learn logistic regression gradient descent works in python.

• Scikit learn logistic regression gradient descent is a process to solve the classification problem and the discrete variable comes as an outcome.
• Gradient descent is defined as an optimization algorithm that minimizes the loss or error of the model.

Code:

In the following code, we will import some libraries from which we can make logistic regression gradient descent.

• Source of dataset – https://www.kaggle.com/rakeshrau/social-network-ads download the dataset from here.
• data = pds.read_csv(“dataa.csv”) is used to load the dataset.
• plot.scatter(data[‘Age’], data[‘Purchased’]) is used to plot or visualizing the dataset.
• X_train, X_test, y_train, y_test = train_test_split(data[‘Age’], data[‘Purchased’], test_size=0.20) is used to divide the dataset into train set and test set.
• y_pred = predict(X, b1, b2) is used to predict the model.
• d_b1 = -2 * sum((Y – y_pred) * y_pred * (1 – y_pred)) is used to derivation of loss with respect to b1.
• d_b2 = -2 * sum(X * (Y – y_pred) * y_pred * (1 – y_pred)) is used to derivation of loss with respect to b2.
• b0, b1 = logistic_regression(X_train, y_train) is used train the model.
• y_pred = predict(X_test_norm, b0, b1) is used to make prediction.
• plot.scatter(X_test, y_pred, c=”red”) is used to plot the graph.
• print(f”Accuracy = {accuracy / len(y_pred)}”) is used to print the accuracy.

import numpy as num
import pandas as pds
import matplotlib.pyplot as plot
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from math import exp
plot.rcParams["figure.figsize"] = (12, 8)

data = pds.read_csv("dataa.csv")
data.head()

plot.scatter(data['Age'], data['Purchased'])
plot.show()

X_train, X_test, y_train, y_test = train_test_split(data['Age'], data['Purchased'], test_size=0.20)

def normalize(X):
    return X - X.mean()

# Method to make predictions
def predict(X, b1, b2):
    return num.array([1 / (1 + exp(-1*b1 + -1*b2*x)) for x in X])

def logistic_regression(X, Y):

    X = normalize(X)

    # Initializing variables
    b1 = 0
    b2 = 0
    l = 0.001
    epochs = 350

    for epoch in range(epochs):
        y_pred = predict(X, b1, b2)
        d_b1 = -2 * sum((Y - y_pred) * y_pred * (1 - y_pred))  
        d_b2 = -2 * sum(X * (Y - y_pred) * y_pred * (1 - y_pred))  
        # Update b1 and b2
        b1 = b1 - l * d_b1
        b2 = b2 - l * d_b2
    
    return b0, b1
b0, b1 = logistic_regression(X_train, y_train)

X_test_norm = normalize(X_test)
y_pred = predict(X_test_norm, b0, b1)
y_pred = [1 if p >= 0.7 else 0 for p in y_pred]

plot.clf()
plot.scatter(X_test, y_test)
plot.scatter(X_test, y_pred, c="red")
plot.show()

accuracy = 0
for i in range(len(y_pred)):
    if y_pred[i] == y_test.iloc[i]:
        accuracy += 1
print(f"Accuracy = {accuracy / len(y_pred)}")

Output:

After running the above code, we get the following output in which we can see that the logistic regression graph and accuracy score is printed on the screen.

Read: Scikit learn Feature Selection

Scikit learn gradient descent linear regression

In this section, we will learn about the scikit learn gradient regression linear regression works in python.

• Linear regression is defined as a linear system and the coefficient of linear regression is calculated using linear algebra.
• Gradient descent linear regression is also an optimization algorithm that can minimize the loss and error of the model.

Code:

In the following code, we will import numpy as num to find the linear regression gradient descent model.

• a = 0 is the intercept of the line.
• m = 7 is the slope of the line.
• num.random.seed(45) is used to generate the random numbers.
• classifier.fit_model(x, y) is used to fit the model.
• plot.plot(x, classifier.predict_model(x)) is used to plot the graph.
• plot.title(“Gradient Descent Linear Regression”) is used to give the title to the graph.

import numpy as num

class GradientDescentLinearRegression:
    def __init__(self, learning_rate=0.02, iterations=1000):
        self.learning_rate, self.iterations = learning_rate, iterations
    
    def fit_model(self, x, y):
        a = 0
        m = 7
        n = x.shape[0]
        for _ in range(self.iterations):
            a_gradient = -2 * num.sum(y - m*x + a) / n
            m_gradient = -2 * num.sum(x*(y - (m*x + a))) / n
            a = a + (self.learning_rate * a_gradient)
            m = m - (self.learning_rate * m_gradient)
        self.m, self.a = m, a
        
    def predict_model(self, X):
        return self.m*X + self.a
num.random.seed(45)
x = num.array(sorted(list(range(5))*20)) + num.random.normal(size=100, scale=0.10)
y = num.array(sorted(list(range(5))*20)) + num.random.normal(size=100, scale=0.20)

classifier = GradientDescentLinearRegression()
classifier.fit_model(x, y)

import matplotlib.pyplot as polt

plot.scatter(x, y, color='red')
plot.plot(x, classifier.predict_model(x))
plot.title("Gradient Descent Linear Regression")

Output:

After running the above code, we get the following output in which we can see that the gradient descent linear regression graph is plotted on the screen.

You may also like to read the following tutorials on Scikit learn.

• Scikit learn Hyperparameter Tuning
• Scikit learn hidden_layer_sizes
• Scikit learn Gaussian
• Scikit learn KNN Tutorial

So, in this tutorial we discussed Scikit learn gradient descent and we have also covered different examples related to its implementation. Here is the list of examples that we have covered.

• Scikit learn gradient descent
• Scikit learn gradient descent regression
• Scikit learn stochastic gradient descent classifier
• Scikit learn batch gradient descent
• Scikit learn minibatch gradient descent
• Scikit learn stochastic gradient descent regression
• Scikit learn logistic regression gradient descent
• Scikit learn gradient descent linear regression