Given a cone with height $H$ and radius $R$ we want to find the optimal radius $r$ and height $h$ of a cylinder placed inside the cone to maximize the volume of the cylinder.
This can be written as: $$f(\lambda, r, h)=\pi r^2h-\lambda \left(h-(1-r/R)H\right)$$
where $h=(1-r/R)H$ is out constraint.
Using the optimization via Lagrange multipliers we find:
$$0=\partial_hf=\pi r^2-\lambda\Rightarrow\lambda=\pi r^2$$ this we can insert into $\partial_rf=0$: $$0=\partial_rf=2\pi rh-\pi r^2\frac{H}{R}\Rightarrow r=2h\frac{R}{H}$$ inserted into the constraint gives us $h$: $$h=(1-r/R)H=\frac{1}{3}H$$ which in turn gives us finally $r$: $$r=\frac{2}{3}R$$
So we should have a local maximum for $(r^*,h^*)=(\frac{2}{3}R,\frac{1}{3}H)$ and $V^*=\frac{4}{27}\pi R^2H$ (which I also get using mathematica).
If I now want to confirm this result using the second derivative test I calculate the bounded hessian matrix:
$$H_f=\begin{pmatrix} \frac{\partial^2f}{\partial\lambda^2} & \frac{\partial^2f}{\partial\lambda\partial h} & \frac{\partial^2f}{\partial\lambda\partial r} \\ \frac{\partial^2f}{\partial h\partial\lambda} & \frac{\partial^2f}{\partial r^2} & \frac{\partial^2f}{\partial r\partial h} \\ \frac{\partial^2f}{\partial r\partial\lambda} & \frac{\partial^2f}{\partial h\partial r} & \frac{\partial^2f}{\partial h^2} \end{pmatrix} =\begin{pmatrix} 0 & -1 & -\frac{H}{R} \\ -1 & 0 & 2\pi r \\ -\frac{H}{R} & 2\pi r & 2\pi h\\ \end{pmatrix}$$
which has $det H_f(r^*,h^*)=2\pi H >0$ but which is not positive definite (as $det\begin{pmatrix}0 & -1 \\ -1 & 0\end{pmatrix}=-1<0$).
Where is the problem?
Update: I just tested another geometrical optimization problem (cuboid inside ellipsoid) where I face the same problem so I conclude that my bounded hessian matrix should be wrong.