Obviously, this problem have solution if $\alpha\geq 0$. From now I assume it. Substitute formula for $y(t)$ from the first equation into the second. After simplifications we get
$$
x^{(4)}(t)+2x^{(2)}(t)=0
$$
Using standard algorithm for solving linear differential equations with constant coefficients we get
$$
x(t)=C_1 \cos(t\sqrt{2})+C_2 \sin(t\sqrt{2})+C_3 t + C_4
$$
So,
$$
y(t)=x^{(2)}(t)+x(t)=-C_1 \cos(t\sqrt{2})-C_2 \sin(t\sqrt{2})+C_3 t + C_4
$$
Now we find
$$
x^2(t)+y^2(t)=2\left((C_3 t+C_4)^2+C_1^2\cos^2(t\sqrt{2})+C_2^2\sin^2(t\sqrt{2})+C_1 C_2\cos(t\sqrt{2})\sin(t\sqrt{2})\right)
$$
It is easy to see that, the condition $x^2(t)+y^2(t)=\alpha={\rm const}$ is satisfied iff
$$
C_1=C_2=C_3=0
$$
So, $x(t)=y(t)=C_4=\sqrt{\frac{\alpha}{2}}$