Optimization, a selection of best value for a set of parameters, is an important problem in different fields of science. When the best possible answer is subject to some constraints, then it is not an easy problem to deal with. However, if the constraint is an equality relationship, “Lagrange Multiplier” can be used to turn it into an unconstrained problem. This method is introduced byJoseph-Louis Lagrange about 200 years ago.

Let’s consider the problem of optimizing a function with no constraint. One would say take the derivative and find its roots to get the extrema of the function. Finding a closed form equation for the maxima or minima of the function is not trivial if the function should follow certain constraints. Instead, Lagrange introduced a method to turn the problem into an unconstrained one: Lagrange multipliers:

Goal: optimize f(x,y), subject to: g(x,y)-c=0

Define Lagrangian as:

Consider contours of f which have a fixed f value while g(x,y)=c.

When the contour line of g intersects with f or cross the contour lines of f means while moving along the contour line the value of  f can change so it is not an extremum. While if the contour line of g meets contour lines of  f tangentially, the value of f is not changing and that point is a potential extremum. Mathematically it is equivalent to:

Solving the Lagrange equation gives the optimized constrained values for f: