1
$\begingroup$

I have certain confusion regarding lagrange multiplier for inequality constraints. I was going through this book by Bishop (Pattern Recognition and Machine Learning) and it says that if I have a function f to be maximized and constraint $g \leq c$ . Then if the maxima is at the boundary that means $g=c$, then the gradient of $f$ at the maxima point at the boundary should point opposite to that of the gradient of $g$. $$ \text{gradient}(f) = -\lambda \times \text{gradient}(g)$$ where $\lambda \geq 0$.

However, while referring to this tutorial http://www.youtube.com/watch?v=3VQBVf6Tr3Y, it says that they should point to the same direction. I am a bit confused and need some clarification.

I am basically confused regarding the direction of gradient(f) and gradient(g). I am not sure why the direction of gradient matters in the inequality constraints except that they should be parallel

  • 0
    The sign of the multiplier **DOES** matter for inequality constraints. However, your formula above has the wrong sign. If the gradient of $f$ is opposite that of $ g$, then clearly you can increase $f$ and decrease $g$ (thus remaining feasible) at the same time, ie. it is not optimal. In fact, this illustrates why the sign (meaning direction) matters.2012-06-25

1 Answers 1