Scikit learn Hyperparameter Tuning

In this Python tutorial, we will learn Scikit learn hyperparameter tuning, and we will also cover different examples related to Hyperparameter tuning using Scikit learn. Moreover, we will cover these topics.

• Scikit learn hyperparameter tuning
• Scikit learn random forest hyperparameter
• Scikit learn logistic regression hyperparameter tuning
• Scikit learn neural network hyperparameter tuning
• Scikit learn linear regression hyperparameters
• Scikit learn SVM hyperparameter
• Scikit learn hyperparameter estimation
• Scikit learn hyperparameter optimization

Scikit learn Hyperparameter Tuning

In this section, we will learn about scikit learn hyperparameter tuning works in python.

Hyperparameter tuning is defined as a parameter that passed as an argument to the constructor of the estimator classes.

Code:

• In the following code, we will import loguniform from sklearn.utils.fixes by which we compare random search and grid search for hyperparameter estimation.
• load_digits(return_X_y=True, n_class=3) is used for load the data.
• clf = SGDClassifier(loss=”hinge”, penalty=”elasticnet”, fit_intercept=True) is used to build the classifier.
• random_search = RandomizedSearchCV(clf, param_distributions=param_dist, n_iter=n_iter_search) is used to run randomized sizes.
• grid_search = GridSearchCV(clf, param_grid=param_grid) is used to run the grid search.

import numpy as np
from time import time
import scipy.stats as stats
from sklearn.utils.fixes import loguniform

from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from sklearn.datasets import load_digits
from sklearn.linear_model import SGDClassifier


x, y = load_digits(return_X_y=True, n_class=3)


clf = SGDClassifier(loss="hinge", penalty="elasticnet", fit_intercept=True)


def report(results, n_top=3):
    for i in range(1, n_top + 1):
        candidate = np.flatnonzero(results["rank_test_score"] == i)
        for candidate in candidate:
            print("Model with rank: {0}".format(i))
            print(
                "Mean validation score: {0:.3f} (std: {1:.3f})".format(
                    results["mean_test_score"][candidate],
                    results["std_test_score"][candidate],
                )
            )
            print("Parameters: {0}".format(results["params"][candidate]))
            print("")


param_dist = {
    "average": [True, False],
    "l1_ratio": stats.uniform(0, 1),
    "alpha": loguniform(1e-2, 1e0),
}


n_iter_search = 15
random_search = RandomizedSearchCV(
    clf, param_distributions=param_dist, n_iter=n_iter_search
)

start = time()
random_search.fit(x, y)
print(
    "RandomizedSearchCV took %.2f seconds for %d candidates parameter settings."
    % ((time() - start), n_iter_search)
)
report(random_search.cv_results_)


param_grid = {
    "average": [True, False],
    "l1_ratio": np.linspace(0, 1, num=10),
    "alpha": np.power(10, np.arange(-2, 1, dtype=float)),
}


grid_search = GridSearchCV(clf, param_grid=param_grid)
start = time()
grid_search.fit(x, y)

print(
    "GridSearchCV took %.2f seconds for %d candidate parameter settings."
    % (time() - start, len(grid_search.cv_results_["params"]))
)
report(grid_search.cv_results_)

Output:

After running the above code, we get the following output in which we can see that the comparing randomize search and grid search for hyperparameter is done on the screen.

Also, read: Scikit-learn Vs Tensorflow

Scikit learn random forest hyperparameter

In this section, we will learn about scikit learn random forest hyperparameter works in python.

In scikit learn hyperparameter includes the number of decision trees and number of features considered by splitting each tree while the nodes are splitting.

Code:

In the following code, we will import RandomForestRegressor from sklearn.ensemble by which we can see the current use hyperparameter.

from sklearn.ensemble import RandomForestRegressor
rf = RandomForestRegressor(random_state = 42)
from pprint import pprint

print('Hyperparameters currently in use:\n')
pprint(rf.get_params())

Output:

After running the above code, we get the following output in which we can see that the hyperparameter currently in use is printed on the screen.

Read: Scikit-learn logistic regression

Scikit learn logistic regression hyperparameter tuning

In this section we will learn about scikit learn logistic regression hyperparameter tuning in python.

• Logistic regression is a predictive analysis that is used to describe the data. It is used to evaluate the metrics for model performance to decide the best hyperparameter.
• Code:
• In the following code, we will import linear_model from sklearn by which we can evaluate the metric for model performance to decide the best hyperparameter.
• dataset = datasets.load_wine() is used to load the dataset.
• pca = decomposition.PCA() Principal component analysis is used to reduce the dimension of feature.
• print(‘BestPenalty:’,clf_GS.best_estimator_.get_params([‘logistic_Regression__penalty’]) is used to print the best hyperparameter on the screen.

import numpy as np
from sklearn import linear_model, decomposition, datasets
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV
from sklearn.preprocessing import StandardScaler
dataset = datasets.load_wine()
X = dataset.data
y = dataset.target
pca = decomposition.PCA()
logistic_Regression = linear_model.LogisticRegression()
pipe = Pipeline(steps=[('std_slc', std_slc),
                       ('pca', pca),
                       ('logistic_Regression', logistic_Regression)])
n_components = list(range(1,X.shape[1]+1,1))
C = np.logspace(-4, 4, 50)
penalty = ['l1', 'l2']
parameters = dict(pca__n_components=n_components,
                      logistic_Regression__C=C,
                      logistic_Regression__penalty=penalty)
clf_GS = GridSearchCV(pipe, parameters)
clf_GS.fit(X, y)
print('Best Penalty:', clf_GS.best_estimator_.get_params()['logistic_Regression__penalty'])
print('Best C:', clf_GS.best_estimator_.get_params()['logistic_Regression__C'])
print('Best Number Of Components:', clf_GS.best_estimator_.get_params()['pca__n_components'])
print(); print(clf_GS.best_estimator_.get_params()['logistic_Regression'])

Output:

After running the above code, we get the following output in which we can see that the best value of hyperparameter is printed on the screen.

Read: Scikit learn Decision Tree

Scikit learn neural network hyperparameter tuning

In this section, we will learn about how scikit learn neural network hyperparameter tuning works in python.

Neural network hyperparameter tuning is defined as a number of hyperparameters such as a number of hidden neurons.

Code:

In the following code, we will import MLPClassifier from sklearn.neural_network by which we can create a neural network hyperparameter tuning.

from sklearn.neural_network import MLPClassifier
x = [[0., 0.], [1., 1.]]
y = [0, 1]
classifier = MLPClassifier(solver='lbfgs', alpha=1e-5,
                    hidden_layer_sizes=(5, 2), random_state=1)

classifier.fit(x, y)
classifier.predict([[2., 2.], [-1., -2.]])
[coef.shape for coef in clf.coefs_]
classifier.predict_proba([[2., 2.], [1., 2.]])

Output:

After running the above code, we get the following output in which we can see that the neural network hyperparameter tuning is printed on the screen.

Scikit learn linear regression hyperparameters

In this section, we will learn how scikit learn linear regression hyperparameter works in python.

The hyperparameter is a process of searching for the ideal model architecture. The scikit learn linear regression is a linear approach for modeling.

Code:

In the following code, we will import pandas as pd and numpy as np for creating the model.

• diabetes = datasets.load_diabetes() is used to create the data set.
• df.head() is used to print the first five rows.

import numpy as np
import pandas as pd
import matplotlib.pylab as plt
%matplotlib inline
import sklearn
from sklearn import datasets
diabetes = datasets.load_diabetes()
df=pd.DataFrame(diabetes.data)
df.columns= diabetes.feature_names

df['diabetes_measure']=diabetes.target
df.head()

After running the above code, we get the following output in which we can see that the first five-row of the dataset is printed on the screen.

• x=df.iloc[:,:-1] is used to creating the feature matrix x.
• y=df.iloc[:,-1] is used to creating the response vector y.
• b = x.iloc[:,2] is used to reshaping the array to two dimension.
• from sklearn.linear_model import LinearRegression importing the linearregression class for linear model.
• plt.figure(figsize=(10,6)) is used to plot the regression line on the scatter plot.

x=df.iloc[:,:-1]

y=df.iloc[:,-1]
b = x.iloc[:,2]
b=b[:,np.newaxis]

y= df.iloc[:,-1]
y=y[:,np.newaxis]

from sklearn.linear_model import LinearRegression

simple_lr = LinearRegression()

simple_lr = LinearRegression().fit(b,y)

predicted_y = simple_lr.predict(b)

plt.figure(figsize=(10,6))
plt.scatter(b, y)
plt.plot(b, predicted_y, c='r')
plt.title('Scatter plot and a Simple Linear Regression Model')
plt.ylabel("y")
plt.xlabel("b")
plt.show()

After running the above code, we get the following output in which we can see that the regression line is plotted on the screen.

• Here from sklearn.model_selection import cross_val_score is used to importing cross_val_score function from the model_selection.
• And cross_val_score(simple_lr,b,y,scoring=’neg_mean_squared_error’,cv=10) is used to store the ten score in an object.

from sklearn.model_selection import cross_val_score
mse= cross_val_score(simple_lr,b,y,scoring='neg_mean_squared_error',cv=10)
mse.mean()

After running the above code, we get the following output in which we can see that the mean value of the model that shows the linear regression as an ideal model.

Read: Scikit learn Hierarchical Clustering

Scikit learn SVM hyperparameter

I this section we will learn how Scikit learn SVM hyperparameter works in Python.

• Before moving forward we should have a piece of knowledge about SVM. SVM stands for support vector machine.
• SVM is used as coordinated of individual observation. It is used for both classification and regression.
• SVM Hyperparameter define as a parameter whose value is used to control by the learning process.

Code:

In the following code, we will import SVC from sklearn.svm which is used as a coordinate of individual observation.

Hyperparameter is defined as a parameter that passed as an argument to the constructor of the estimator classes.

base_estimator = SVC(gamma=’scale’) gamma is used for support vector classifier.

from sklearn.datasets import make_classification
from sklearn.svm import SVC
from sklearn.experimental import enable_halving_search_cv 
from sklearn.model_selection import HalvingGridSearchCV
import pandas as pd
parameter_grid= {'kernel': ('linear', 'rbf'),
             'C': [1, 10, 100]}
base_estimator = SVC(gamma='scale')
X, y = make_classification(n_samples=1000)
sh = HalvingGridSearchCV(base_estimator, parameter_grid, cv=5,
                         factor=2, min_resources=20).fit(X, y)
sh.n_resources_

After running the above code, we get the following output in which we can see that scikit learn SVM hyperparameter resources is printed on the screen.

Read: Scikit learn Random Forest

Scikit learn hyperparameter estimation

In this section, we will learn about scikit learn hyperparameter estimation in python.

• As we know hyperparameter is a parameter that passed as an argument to the constructor of the estimators.
• Hyperparameter is a model parameter that is estimated without using actual observed data.

Code:

In the following code, we will import the make_classification from sklearn.datasets by which a model parameter is estimated without using actual observed data.

Here sh = HalvingGridSearchCV(base_estimator, parameter_grid, cv=5, factor=2, resource=’n_estimators’,max_resources=30).fit(x, y) is used to make the estimation without using actual observed data.

from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.experimental import enable_halving_search_cv 
from sklearn.model_selection import HalvingGridSearchCV
import pandas as pd

parameter_grid = {'max_depth': [3, 5, 10],
                  'min_samples_split': [2, 5, 10]}
base_estimator = RandomForestClassifier(random_state=0)
x, y = make_classification(n_samples=1000, random_state=0)
sh = HalvingGridSearchCV(base_estimator, parameter_grid, cv=5,
                         factor=2, resource='n_estimators',
                         max_resources=30).fit(x, y)
sh.best_estimator_

Output:

After running the above code, we get the following output in which we can see that the best estimator of the model is printed on the screen.

Read: Scikit learn Hidden Markov Model

Scikit learn hyperparameter optimization

In this section, we will learn about scikit learn hyperparameter optimization in python.

Hyperparameter optimization is defined as a process or a problem of choosing a set of optimal hyperparameters.

Syntax:

In this syntax, an estimator is optimized to find the name and current value for all the parameters.

estimator.get_parameter()

Code:

In the following code, we will import read_csv from data to read the dataset and we use hyperparameter to find a well-performing model configuration.

• ‘https://raw.githubusercontent.com/jbrownlee/Datasets/master/sonar.csv’ is used to load the dataset.
• data = dataframe.values are used to split the input and output elements.

from pandas import read_csv
dataseturl = 'https://raw.githubusercontent.com/jbrownlee/Datasets/master/sonar.csv'
dataframe = read_csv(url, header=None)
data = dataframe.values
x, y = data[:, :-1], data[:, -1]
print(x.shape, y.shape)

Output:

After running the above code, we get the following output in which we can see that 208 rows of data with 60 variables.

You may also like to read the following Scikit learn tutorials.

• Scikit learn KNN Tutorial
• Scikit learn Ridge Regression
• Scikit learn Feature Selection
• Scikit learn Gradient Descent

So, in this tutorial we discussed Scikit learn Hyperparameter tuning and we have also covered different examples related to its implementation. Here is the list of examples that we have covered.

• Scikit learn hyperparameter tuning
• Scikit learn random forest hyperparameter
• Scikit learn logistic regression hyperparameter tuning
• Scikit learn neural network hyperparameter tuning
• Scikit learn linear regression hyperparameters
• Scikit learn SVM hyperparameter
• Scikit learn hyperparameter estimation
• Scikit learn hyperparameter optimization