Given that $x, y, z$ are nonnegative real numbers such that :
$$x^2 + y^2 + z^2 + xyz = 4$$
Prove that $0 ≤ xy + yz + zx − xyz ≤ 2$
Given that $x, y, z$ are nonnegative real numbers such that :
$$x^2 + y^2 + z^2 + xyz = 4$$
Prove that $0 ≤ xy + yz + zx − xyz ≤ 2$
Here's a straight-forward way, which is not very elegant, but is on the other hand very general, and does not require problem-specific tricks.
We want to calculate bounds for the function $$f=x y + y z+ z x - x y z\ ,$$ under the constraint $$g=x^2 + y^2 + z^2 + x y z-4=0\ .$$
For this, we introduce the Lagrange multiplier $\lambda$, and look for points for which $$\nabla(f-(\lambda-1) g)=\left( \begin{array}{c} y+z-2 x (\lambda -1)-y z \lambda \\ x+z-2 y (\lambda -1)-x z \lambda \\ x+y-2 z (\lambda -1)-x y \lambda \\ \end{array} \right)=0$$ The choice $\lambda-1$ rather than $\lambda$ is arbitrary and will be convenient in the following. Solving the above equation for $x,y,z$ requires some work, and the result gives 2 families of solutions:
Solving for $\lambda$, the first solution satisfies $g$ only when $\lambda=\frac{3}{4}$, and then $x=y=z=1$ and $f=2$. Doing the same trick for the second one, one gets only negative results for $x,y$ or $z$, so we can forget about that (for example, the point $x=y=\frac{2}{3},\ z=-2$ satisfies $g$ but not the non-negativity constraint).
We see that for $x,y,z>0$, there is only one critical point of $f$ under the constraint $g$, at the point $x=y=z=1$. A simple check shows that this is a global maximum, since the point $x=y=0, z=2$ satisfies $g$ and has $f=0<2$. You now just have to check the boundaries, i.e. WLOG $x=0$, but this gives immediately $f=yz$ which is non-negative by assumption. therefore $0\le f\le 2$.
For the left part of the inequality:
By the AGM-inequality, we have: $$3(xyz)^{2/3}=3((xy)(xz)(yz))^{1/3}\leq xy+xz+yz$$ Suppose that $xy+xz+yx
use this $$\cos^2{A}+\cos^2{B}+\cos^2{C}+2\cos{A}\cos{B}\cos{C}=1$$ and $$x^2+y^2+z^2+xyz=4$$ then we set $$x=2\cos{A},y=2\cos{B},z=2\cos{C}$$ $$\Longleftrightarrow 0\le 4(\cos{A}\cos{B}+\cos{B}\cos{C}+\cos{A}\cos{C})-8\cos{A}\cos{B}\cos{C}\le 2$$ and since $$\cos{A}\cos{B}\cos{C}=\dfrac{s^2-(2R+r)^2}{4R^2},\cos{A}\cos{B}+\cos{B}\cos{C}+\cos{C}\cos{A}=\dfrac{s^2+r^2-4R^2}{4R^2}$$
then $$\Longleftrightarrow 0\le\dfrac{s^2+r^2-4R^2}{R^2}-\dfrac{2}{R^2}\left(s^2-(2R+r)^2\right)\le 2$$ and left hand it suffices that $$s^2+r^2-4R^2-2s^2+2(2R+r)^2\ge 0$$ $$\Longleftrightarrow 3r^2+8Rr+4R^2\ge s^2$$
use the Gerrseten inequality: $$r(16R-5r)\le s^2\le 4R^2+4Rr+3r^2$$ then $$\Longleftrightarrow 4R^2+4Rr+3r^2\le 3r^2+4R^2+8Rr$$ $$\Longleftrightarrow 4Rr\ge 0$$is obvious. and the Right Hand, $$\Longleftrightarrow s^2+r^2-4R^2-2(s^2-(2R+r)^2)\le 2R^2 $$ $$\Longleftrightarrow 2R^2+8Rr+3r^2\le s^2$$ since $$s^2\ge r(16R-5r)$$, it follow that $$r(16R-5r)\ge 2R^2+8Rr+3r^2$$ $$\Longleftrightarrow (R-2r)^2\ge 0$$ is obivous.
Let $x=\frac{2a}{\sqrt{(a+b)(a+c)}}$ and $y=\frac{2b}{\sqrt{(a+b)(b+c)}}$, where $a$, $b$ and $c$ be positives.
Hence, $z=\frac{2c}{\sqrt{(a+c)(b+c)}}$ and the left inequality it's $$4\sum_{cyc}\frac{ab}{(a+b)\sqrt{(a+c)(b+c)}}\geq\frac{8abc}{(a+b)(a+c)(b+c)}$$ or $$\sum_{cyc}ab\sqrt{(a+c)(b+c)}\geq2abc,$$ which is true because $$\sum_{cyc}ab\sqrt{(a+c)(b+c)}\geq\sum_{cyc}abc\geq2abc.$$ The right inequality.
We need to prove that $$4\sum_{cyc}\frac{ab}{(a+b)\sqrt{(a+c)(b+c)}}-\frac{8abc}{(a+b)(a+c)(b+c)}\leq2$$ or $$2\sum_{cyc}ab\sqrt{(a+c)(b+c)}\leq(a+b)(a+c)(b+c)+4abc,$$ which is AM-GM: $$2\sum_{cyc}ab\sqrt{(a+c)(b+c)}\leq\sum_{cyc}ab(a+c+b+c)=$$ $$=\sum_{cyc}(a^2b+a^2c+2abc)=(a+b)(a+c)(b+c)+4abc.$$ Done!