Python Scipy Leastsq

This Python Scipy tutorial explains everything about the “Python Scipy Leastsq“. We will see how to use the leastsq method of Python Scipy and find the least square of the given equations by covering the following topics.

  • What is Least Square?
  • How to find the Least Squares of the equation using the method leastsq
  • Python Scipy Leastsq Vs Least_squares

What is Least Square?

The least squares approach, a type of mathematical regression analysis, is used to identify the line of best fit for a set of data, presenting a graphic representation of the relationship between the sample points.

Each data point depicts the correlation between a known independent variable and an unidentified dependent variable.

The least squares method is a statistical technique for determining which set of data points fits a set of data points the best by minimizing the total of the offsets or residuals of the data points from the plotted curve.

In order to forecast the behavior of dependent variables, least squares regression is performed. The location of the best-fitted line among the data points under study is explained generally by the least squares method.

An analyst who wants to assess the correlation between an organization’s stock returns and the returns of the index of which the stock is a part can use the least squares approach as an example. The analyst is attempting to determine whether stock returns are influenced by index returns in this case.

Read Python Scipy Special Module With Examples

Python Scipy Leastsq

The Python Scipy has a method leastsq() in a module scipy.optimize that reduce the squared sum of a group of equations.

The syntax is given below.

scipy.optimize.leastsq(func, x0, args=(), Dfun=None, full_output=0, col_deriv=0, ftol=1.49012e-08, xtol=1.49012e-08, gtol=0.0, maxfev=0, epsfcn=None, factor=100, diag=None)

Where parameters are:

  • func(callable): Should accept a minimum of one input, with a maximum length of N vectors, and return M floating point numbers. It cannot return NaNs otherwise the fitting will not work. M must exceed N or be equal to it.
  • x0(ndarray): The initial estimation for the minimization.
  • args(tuple): This tuple contains any extra arguments that func receives.
  • Dfun(callable): A formula or technique for calculating the Jacobian of a function having derivatives across the rows. The Jacobian will be estimated if this is None.
  • full_output(boolean): Return all optional outputs if non-zero.
  • col_deriv(boolean): For the Jacobian function to calculate derivatives down the columns, the value must be non-zero.
  • ftol(float): The sum of squares should have some relative inaccuracy.
  • xtol(float): The approximation solution should have some relative inaccuracy.
  • gtol(float): The function vector and the Jacobian columns should be orthogonal.
  • maxfev(int): The most calls that can be made to the function. The default maxfev is 200(N+1) unless Dfun is given, in which case it is 100(N+1), where N is the number of items in x0.
  • epsfcn(float): A factor that is used to choose an appropriate step size for the forward-difference Jacobian approximation (for Dfun=None). Typically, sqrt(epsfcn)*x represents the actual step length. It is presumed that the relative errors are on the order of the machine precision if epsfcn is less than the machine precision.
  • factor(float): A variable that controls the first step’s bound (factor * || diag * x||). Should be spaced out (0.1, 100).
  • diag(squence): The scale factors for the variables are N positive entries.
READ:  How to encrypt and decrypt password in Django

The method leastsq() returns solution, cov_x and info_dict.

Let’s take an example by following the below steps:

Import the required libraries or methods using the below python code.

from scipy import optimize

Create a function and minimize it using the below code.

def fun(x):
    return 2*(x-1)**3+2
optimize.leastsq(fun, 0)
Python Scipy Leastsq
Python Scipy Leastsq

This is how to reduce the squared sum of a group of equations using the method leastsq() of Python Scipy.

Python Scipy Leastsq Vs Least_squares

The method leastsq() minimize the squared sum of a group of equations that we have learned in the above subsection whereas least_squares() making use of bounds on the variables to solve a nonlinear least-squares issue. So in this section, we will only know about the least_squares().

The syntax is given below.

scipy.optimize.least_squares(fun, x0, xtol=1e-08, gtol=1e-08, x_scale=1.0, jac='2-point', bounds=(- inf, inf), method='trf', ftol=1e-08, loss='linear', f_scale=1.0, diff_step=None, tr_solver=None, tr_options={}, jac_sparsity=None, max_nfev=None, verbose=0, args=(), kwargs={})

Where parameters are:

  • fun(callable): Fun(x, *args, **kwargs) is a function that computes the residuals vector; the minimization is done with regard to the first argument. This function receives an ndarray with the shape (n,) as its parameter x (never a scalar, even for n=1). It must allocate and return either a scalar or a 1-D array-like of shape (m). It must be wrapped in a real function with real parameters if the argument x is complex or the function fun produces complex residuals.
  • x0: First-pass estimation of independent variables. When a value is a float, it is handled as a 1-D array with a single element.
  • jac(2-point, 3-point, ‘cs’): An m by n matrix with element I j) being the partial derivative of f[i] with respect to x[j] is used to compute the Jacobian matrix. The keywords decide on a finite difference estimate method. Although the “3-point” technique is more accurate than the “2-point,” it requires twice as many processes (default). Though it may be the most precise, the strategy “cs” only works when fun correctly handles complex inputs and can be extended analytically to the complex plane. The “2-point” scheme is always used by the method “lm.” If callable, it is used as jac(x, *args, **kwargs) and should return an accurate value for the Jacobian as a scipy.sparse.linalg.LinearOperator, a sparse matrix (csr matrix recommended for performance), or an array-like value (np.atleast 2d is applied).
  • bounds(two-tuple of array data): restrictions on the bottom and upper values of independent variables. By default, there are no limits. Each array needs to be the same size as x0 or it must be a scalar, in which case each variable’s bound will be the same. Use the appropriate sign with the np.inf command to disable bounds for all or selected variables.
  • method: An algorithm for minimising like lm, dogbox and trf.
  • ftol(None, float): Tolerance for cost function changes that could result in termination. The standard is 1e-8. When dF ftol * F and there was a sufficient agreement between a local quadratic model and the real model in the previous phase, the optimization process is terminated. The termination caused by this circumstance is blocked if neither “method” nor “None” are “lm”. This tolerance needs to be greater than machine epsilon if the value of “method” is “lm.”
  • xtol(None,float): Tolerance for the independent variables changing and causing a termination. 1e-8 is the default. Depending on the technique, the precise state may be.
  • gtol(None,float): Tolerance for termination according to the gradient’s average. 1e-8 is the default. The procedure employed determines the precise condition.
  • x_scale(array_data, jac): Each variable’s characteristic scale. The problem is equivalently reformulated as xs = x / x scale when x scale is set. An alternative perspective holds that x scale[j] is proportional to the size of a trust region along the jth dimension. By setting x scale so that any step along the scaled variables has a similar impact on the cost function, improved convergence can be attained. If “jac” is selected, the inverse norms of the columns in the Jacobian matrix are used to iteratively update the scale.
  • loss(string, callable): The loss function is determined.
  • f_scale(float): Value of soft margin between inlier and outlier residuals, default is 1.0. The loss function is evaluated as follows rho_(f**2) = C**2 * rho(f**2 / C**2), where C is f_scale, and rho is determined by loss parameter. This parameter has no effect with loss='linear', but for other loss values, it is of crucial importance.
  • max_nfev(int): Number of function evaluations that can be performed before stopping. If None (the default), an automatic choice is made for the value.
  • diff_step(number or array_data): Identifies the relative step size for the Jacobian’s finite difference approximation. x * diff step is used to calculate the actual step. If None is selected (the default), the usual “optimal” power of the machine epsilon for the chosen finite difference method is assumed to be diff_step.
  • tr_solver(): ‘trf’ and ‘dogbox’ techniques exclusively apply this method for addressing trust-region subproblems.
  • tr_options(dict): Passed keyword choices to the solver for trust regions. The tr options are ignored when the tr solver=’exact’. Options for scipy.sparse.linalg.lsmr are found in tr_solver=’lsmr’. The’regularize’ option (bool, default is True) is also supported by method=’trf,’ and it adds a regularisation term to the normal equation to improve convergence in cases when the Jacobian is rank-deficient.
  • jac_sparsity(array_data, sparse matrix): Specifies the Jacobian matrix’s sparsity structure for finite difference estimation; it must have the following shape: (m, n). The computations will be significantly sped up if the sparsity structure is provided and the Jacobian has only a few non-zero components in each row. A zero entry indicates that the Jacobian’s corresponding element is also zero. Forces the usage of the “lsmr” trust-region solver if it is present. When None is specified (the default), dense differencing is applied. has no impact on the “lm” approach.
  • verbose(0,1,2): Algorithm verbosity level: 0 (default), silent operation show a termination report in step one. 2: Show iteration progress (the “lm” method does not support this).
  • args, kwargs(dict, tuple): Fun and Jac received additional arguments. Both are initially empty. Fun(x, *args, **kwargs) is the calling signature for both jac and x.
READ:  Python Tkinter Animation

The method least_squares() returns result with the following fields defined, optimizeResult:

  • x(ndarray, shape(n,)): Found solution.
  • cost(float): Cost function value at the solution.
  • fun(ndarray, shape(m,): Residuals at the solution vector.
  • jac(sparse matrix, ndarray): Modified Jacobian matrix at the solution in that J^T J approximates the cost function’s Hessian using a Gauss-Newton method. The type matches the one that the algorithm uses.
  • grad(ndarray,shape(,m): The cost function’s gradient at the solution.
  • optimality(float): A measure of first-order optimality. It is always the uniform norm of the gradient in unconstrained problems. It is the quantity that was compared with gtol during iterations in limited problems.
  • active_mask(ndarray of int, shape(n,)): Each element indicates if the corresponding constraint is in effect (that is, whether a variable is at the bound). 0: No constraint is in effect. 1: a lower bound is in effect. 1: An active upper bound. Given that the “trf” approach generates a series of strictly feasible iterations and that the active mask is decided within a tolerance level, it might be fairly arbitrary.
  • nfev(int): Number of completed function evaluations. Unlike the ‘lm’ approach, the methods ‘trf’ and ‘dogbox’ do not count function calls for numerical Jacobian approximation.
  • njev(int): Number of completed Jacobian evaluations. If the “lm” approach employs numerical Jacobian approximation, it is set to None.
  • status(int): The cause for stopping the algorithm.
  • message(string): An explanation in writing of the termination cause.
  • success(boolean): True if status > 0 indicates that one of the convergence requirements has been met.

Let’s take an example by following the below steps:

Import the required methods or libraries using the below python code.

from scipy import optimize
import numpy as np

Create a function Rosebrock and an array of data, and pass both things to method least_squares() using the below code.

def func_rosenbrock(x):
    return np.array([10 * (x[1] - x[0]**2), (1 - x[0])])


x0_rosenbrock_ = np.array([2, 2])
res_1_ = optimize.least_squares(func_rosenbrock, x0_rosenbrock_)
print('solution',res_1_.x)
print('cost',res_1_.cost)
print('optimality',res_1_.optimality)


Python Scipy Leastsq Vs Least_squares
Python Scipy Leastsq Vs Least_squares

From the above output, we can conclude how Leastsq is different from the Least_squares method in Python Scipy.

READ:  Cross Entropy Loss PyTorch

We have learned about how to find the least squares of the given equations and how leastsq is different from Least_squares method with the following topics.

  • What is Least Square?
  • How to find the Least Squares of the equation using the method leastsq
  • Python Scipy Leastsq Vs Least_squares

You may like the following Python Scipy tutorials: