Let $(x,y)\in[0,1)\times[0,1)$ cum $x^2+y^2<1$. Are there any $\mu\geq\lambda>0$ such that
$\lambda\xi_1^2+\lambda\xi_2^2\leq(1-x^2)\xi_1^2+2xy\xi_1\xi_2+(1-y^2)\xi_2^2\leq\mu\xi_1^2+\mu\xi_2^2$
$\forall\xi=(\xi_1,\xi_2)^T\in\mathbb R^2$ ?
Let $(x,y)\in[0,1)\times[0,1)$ cum $x^2+y^2<1$. Are there any $\mu\geq\lambda>0$ such that
$\lambda\xi_1^2+\lambda\xi_2^2\leq(1-x^2)\xi_1^2+2xy\xi_1\xi_2+(1-y^2)\xi_2^2\leq\mu\xi_1^2+\mu\xi_2^2$
$\forall\xi=(\xi_1,\xi_2)^T\in\mathbb R^2$ ?
Let $A = \begin{bmatrix} 1-x^2 & xy \\ xy & 1-y^2\end{bmatrix}$. Then $(1-x^2)\xi_1^2+2xy\xi_1\xi_2+(1-y^2)\xi_2^2 = \langle \xi, A\xi \rangle$.
Choose $x = y = \frac{3}{4}$. Then $A = \frac{1}{16}\begin{bmatrix} 7 & 9 \\ 9 & 7\end{bmatrix}$, and choose $\xi = (1,-1)^T$. Then $\langle \xi,A \xi \rangle = -\frac{1}{4}$. So the answer to part of your question is no.
The other side is easier, since $\langle \xi,A \xi \rangle \leq (1-x^2)\xi_1^2+2xy(\xi_1^2+\xi_2^2)+(1-y^2)\xi_2^2 \leq (2-(x-y)^2) (\xi_1^2+\xi_2^2) \leq 2(\xi_1^2+\xi_2^2)$.
Answer to the modified question:
Again choose $\xi = (1,-1)^T$. Now choose $y = x$, and note that we can choose any $|x|<\frac{1}{\sqrt{2}}$. Then we have $\langle \xi,A \xi \rangle = 2(1-2 x^2)$. However, since $\inf_{|x|<\frac{1}{\sqrt{2}}} 2(1-2 x^2) = 0$, no $\lambda>0$ can exist satifying the lower bound $\langle \xi,A \xi \rangle \geq \lambda ||\xi||^2$.
Let $(x,y)\equiv (\cos\phi_0,\sin\phi_0) \;\; $ and $\; (\xi_1,\xi_2)\equiv \rho (\cos\phi,\sin\phi) \;\;$. Then the condition simplifies a lot $(\lambda < \cos^2(\phi-\phi_0)<\mu \;\;)$. It hints that $\lambda=0 \;$ and $\mu=1$. Cheers!
One asks that $\lambda\|\xi\|^2\leqslant\|\xi\|^2-\langle z,\xi\rangle^2\leqslant\mu\|\xi\|^2$ for every $\xi$, where $\|\ \|$ and $\langle\ ,\ \rangle$ are the usual Euclidean norm and the Euclidean scalar product, and $z=(x,y)$.
Since $\langle z,\xi\rangle^2\leqslant\|\xi\|^2\cdot\|z\|^2$ by Cauchy-Schwarz inequality, the first inequality holds with $\lambda=1-\|z\|^2=1-x^2-y^2$ hence $0\lt\lambda\leqslant1$. The second inequality holds with $\mu=1$.
One can check that these values are optimal, $\lambda$ when $\xi$ and $z$ are colinear and $\mu$ when $\xi$ and $z$ are orthogonal.