The method of iteratively reweighted least squares (IRLS) is used to solve certain optimization problems with objective functions of the form of a p-norm,
by an iterative method in which each step involves solving a weighted least squares problem of the form
IRLS is used to find the maximum likelihood estimates of a generalized linear model, and in robust regression to find an M-estimator, as a way of mitigating the influence of outliers in an otherwise normally distributed data set, for example, by minimizing the least absolute errors rather than the least square errors.
One of the advantages of IRLS over linear programming and convex programming is that it can be used with GaussâÂÂNewton and LevenbergâÂÂMarquardt numerical algorithms.
IRLS can be used for âÂÂ<sub>1</sub> minimization and smoothed âÂÂ<sub>p</sub> minimization, p < 1, in compressed sensing problems. It has been proved that the algorithm has a linear rate of convergence for âÂÂ<sub>1</sub> norm and superlinear for âÂÂ<sub>t</sub> with t < 1, under the restricted isometry property, which is generally a sufficient condition for sparse solutions.
To find the parameters ò = (ò<sub>1</sub>, â¦,ò<sub>k</sub>)<sup>T</sup> which minimize the L<sup>p</sup> norm for the linear regression problem,
the IRLS algorithm at step t + 1 involves solving the weighted linear least squares problem
where W<sup>(t)</sup> is the diagonal matrix of weights, usually with all elements set initially to
and updated after each iteration to
In the case p = 1, this corresponds to least absolute deviation regression (in this case, the problem would be better approached by use of linear programming methods, so the result would be exact) and the formula is
To avoid dividing by zero, regularization must be done, so in practice the formula is
where is some small value, like 0.0001. Note the use of in the weighting function is equivalent to the Huber loss function in robust estimation.