I do not understand of how to come to the following conclusion and I would really appreciate a more in-detail explanation:
Given a non-cooperative game where one player minimizes $xy$ w.r.t $x$ and the other player minimizes $-xy$ w.r.t. $y$. Gradient descent enters a stable orbit, rather than converging to the desired equilibrium point $x=y=0$.
I don't come from a game theory background, but if I understand the question correctly, the gradient descent step is given by:
$$x_{n+1} = x_n - \gamma_n \frac{\partial}{\partial x} (xy) = x_n - \gamma_n y_n$$ and $$y_{n+1} = y_n - \gamma_n \frac{\partial}{\partial y} (-xy) = y_n + \gamma_n x_{n}$$ Hence we have a transition matrix $$ A_n = \begin{bmatrix} 1 & -\gamma_n\\ \gamma_n & 1\end{bmatrix} = \sqrt{1+\gamma_n^2}\begin{bmatrix} \cos\theta_n & -\sin\theta_n\\ \sin\theta_n & \cos\theta_n\end{bmatrix} $$ where $\theta_n = \cos^{-1}\sqrt{\frac{1}{1+\gamma_n^2}}$.
Then each step rotates and moves away from the optimum by a small amount. I don't think this reaches a stable orbit in general as the question claims, but it does if steps were taken so that $\rho = \prod_{n=0}^\infty \sqrt{1+\gamma_n^2} < \infty$ then the orbit would approach the ring of radius $\rho \|(x_0,y_0)\|_2$.
EDIT: Just because I was interested in trying this out, I did a bit of further analysis and implemented the steps in Python.
First, the length of the generated spiral is given by $\sum_{n=0}^{\infty} \gamma_n$. In order for our spiral to not slow to a stop then the length clearly needs to be infinite, i.e., $\sum_{n=0}^{\infty} \gamma_n = \infty$.
If we want a stable orbit, then as previously mentioned we need $\prod_{n=0}^{\infty}\sqrt{1+\gamma_n^2} < \infty$. A simple example of such a choice of step size is $\gamma_n=\frac{1}{n+1}$. Then after a change of variables, $m=n+1$,
$$ \sum_{n=0}^{\infty}\gamma_n = \sum_{m=1}^{\infty} \frac{1}{m} = \infty$$ as it is a harmonic series, and
$$ \prod_{n=0}^{\infty} \sqrt{1+\gamma_n^2} = \sqrt{\prod_{m=1}^{\infty} \left(1 + \frac{1}{m^2}\right)} = \sqrt{\frac{\sin(i\pi)}{i \pi}} = \sqrt{\frac{e^{\pi}-e^{-\pi}}{2\pi}} \approx 1.92$$ which follows from manipulation of the infinite product for $\sin(z)$.
Hence if we start at $x=1$ and $y=0$ then $\|(x,y)\|_2 = 1$, so we would expect the spiral to approach a circle of radius $1.92$. This may then be shown experimentally (with $100,000$ steps):
