Let $T\subset\mathbb{R}^2$ be the (closed) triangle bounded by the lines $x+y=4$, $x\ge-1$ and $y\ge-1$. I want to find and classify all the extrema of the function $f(x,y)=-x^2y(x+y-2)$ on the triangle $T$.
I've done the following: Solving $\nabla f=0$ gives the points $(0,0), (0,1)$ and $0,2$. But when I compute the Hessian, I keep getting eigenvalue 0.
Is there someone who can show me how to solve this problem?