Obviously, this problem have solution if $\alpha\geq 0$. From now I assume it. Substitute formula for $y(t)$ from the first equation into the second. After simplifications we get $ x^{(4)}(t)+2x^{(2)}(t)=0 $ Using standard algorithm for solving linear differential equations with constant coefficients we get $ x(t)=C_1 \cos(t\sqrt{2})+C_2 \sin(t\sqrt{2})+C_3 t + C_4 $ So, $ y(t)=x^{(2)}(t)+x(t)=-C_1 \cos(t\sqrt{2})-C_2 \sin(t\sqrt{2})+C_3 t + C_4 $ Now we find $ x^2(t)+y^2(t)=2\left((C_3 t+C_4)^2+C_1^2\cos^2(t\sqrt{2})+C_2^2\sin^2(t\sqrt{2})+C_1 C_2\cos(t\sqrt{2})\sin(t\sqrt{2})\right) $ It is easy to see that, the condition $x^2(t)+y^2(t)=\alpha={\rm const}$ is satisfied iff $ C_1=C_2=C_3=0 $ So, $x(t)=y(t)=C_4=\sqrt{\frac{\alpha}{2}}$