Backtracking line search

In (unconstrained) minimization, a backtracking line search, a search scheme based on the Armijo–Goldstein condition, is a line search method to determine the maximum amount to move along a given search direction. It involves starting with a relatively large estimate of the step size for movement along the search direction, and iteratively shrinking the step size (i.e., "backtracking") until a decrease of the objective function is observed that adequately corresponds to the decrease that is expected, based on the local gradient of the objective function.

Motivation

Given a starting position $\mathbf {x}$$\mathbf {x}$ and a search direction $\mathbf {p}$$\mathbf {p}$, the task of a line search is to determine a step size $\alpha$$\alpha$ that adequately reduces the objective function $f:\mathbb {R} ^{n}\to \mathbb {R}$$f:\mathbb {R} ^{n}\to \mathbb {R}$ (assumed smooth), i.e., to find a value of $\alpha$$\alpha$ that reduces $f(\mathbf {x} +\alpha \,\mathbf {p} )$$f({\mathbf {x}}+\alpha \,{\mathbf {p}})$ relative to $f(\mathbf {x} )$$f(\mathbf {x} )$. However, it is usually undesirable to devote substantial resources to finding a value of $\alpha$$\alpha$ to precisely minimize $f$$f$. This is because the computing resources needed to find a more precise minimum along one particular direction could instead be employed to identify a better search direction. Once an improved starting point has been identified by the line search, another subsequent line search will ordinarily be performed in a new direction. The goal, then, is just to identify a value of $\alpha$$\alpha$ that provides a reasonable amount of improvement in the objective function, rather than to find the actual minimizing value of $\alpha$$\alpha$.

The backtracking line search starts with a large estimate of $\alpha$$\alpha$ and iteratively shrinks it. The shrinking continues until a value is found that is small enough to provide a decrease in the objective function that adequately matches the decrease that is expected to be achieved, based on the local function gradient $\nabla f(\mathbf {x} )\,.$$\nabla f({\mathbf {x}})\,.$

Define the local slope of the function of $\alpha$$\alpha$ along the search direction $\mathbf {p}$$\mathbf {p}$ as $m=\mathbf {p} ^{\mathrm {T} }\,\nabla f(\mathbf {x} )\,.$$m={\mathbf {p}}^{{{\mathrm T}}}\,\nabla f({\mathbf {x}})\,.$ It is assumed that $\mathbf {p}$$\mathbf {p}$ is a unit vector in a direction in which some local decrease is possible, i.e., it is assumed that $m<0$$m<0$.

Based on a selected control parameter $c\,\in \,(0,1)$$c\,\in \,(0,1)$, the Armijo–Goldstein condition tests whether a step-wise movement from a current position $\mathbf {x}$$\mathbf {x}$ to a modified position $\mathbf {x} +\alpha \,\mathbf {p}$${\mathbf {x}}+\alpha \,{\mathbf {p}}$ achieves an adequately corresponding decrease in the objective function. The condition is fulfilled if $f(\mathbf {x} +\alpha \,\mathbf {p} )\leq f(\mathbf {x} )+\alpha \,c\,m\,.$$f({\mathbf {x}}+\alpha \,{\mathbf {p}})\leq f({\mathbf {x}})+\alpha \,c\,m\,.$

This condition, when used appropriately as part of a line search, can ensure that the step size is not excessively large. However, this condition is not sufficient on its own to ensure that the step size is nearly optimal, since any value of $\displaystyle \alpha$$\displaystyle \alpha$ that is sufficiently small will satisfy the condition.

Thus, the backtracking line search strategy starts with a relatively large step size, and repeatedly shrinks it by a factor $\tau \,\in \,(0,1)$$\tau \,\in \,(0,1)$ until the Armijo–Goldstein condition is fulfilled.

The search will terminate after a finite number of steps for any positive values of $c$$c$ and $\tau$$\tau$ that are less than 1. For example, Armijo used  ¹⁄₂ for both $c$$c$ and $\tau$$\tau$ in a paper he published in 1966.

Algorithm

Starting with a maximum candidate step size value $\alpha _{0}>0\,$$\alpha _{0}>0\,$, using search control parameters $\tau \,\in \,(0,1)$$\tau \,\in \,(0,1)$ and $c\,\in \,(0,1)$$c\,\in \,(0,1)$, the backtracking line search algorithm can be expressed as follows:

1. Set $t=-c\,m$$t=-c\,m$ and iteration counter $j\,=\,0$$j\,=\,0$.
2. Until the condition is satisfied that $f(\mathbf {x} )-f(\mathbf {x} +\alpha _{j}\,\mathbf {p} )\geq \alpha _{j}\,t,$$f({\mathbf {x}})-f({\mathbf {x}}+\alpha _{j}\,{\mathbf {p}})\geq \alpha _{j}\,t,$ repeatedly increment $j$$j$ and set $\alpha _{j}=\tau \,\alpha _{j-1}\,.$$\alpha _{j}=\tau \,\alpha _{{j-1}}\,.$
3. Return $\alpha _{j}$$\alpha _{j}$ as the solution.

In other words, reduce $\alpha _{0}$$\alpha _{0}$ by a factor of $\tau \,$$\tau \,$ in each iteration until the Armijo–Goldstein condition is fulfilled.

References

• Armijo, Larry (1966). "Minimization of functions having Lipschitz continuous first partial derivatives". Pacific J. Math. 16 (1): 1–3. doi:10.2140/pjm.1966.16.1. 
• Dennis, J. E.; Schnabel, R. B. (1996). Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Philadelphia: SIAM Publications. ISBN 978-0-898713-64-0. 
• Nocedal, Jorge; Wright, Stephen J. (2000), Numerical Optimization, Springer-Verlag, ISBN 0-387-98793-2