Using this, if we rotate the axes $\theta $ as the rotation of axes does not change the nature of the curve,
$x=X\cos\theta-Y\sin\theta$, $y=X\sin\theta+Y\cos\theta$,
$5(X\cos\theta-Y\sin\theta)^2+5(X\sin\theta+Y\cos\theta)^2+8(X\cos\theta-Y\sin\theta)(X\sin\theta+Y\cos\theta)=1$
$=>X^2(5+8\sin\theta\cos\theta)+Y^2(5-8\sin\theta\cos\theta)+8XY(\cos^2\theta-\sin^2\theta)=1$
$=>X^2(5+4\sin2\theta)+Y^2(5-4\sin2\theta)+8XY(\cos2\theta)=1$
If we make $\cos2\theta=0=>\sin2\theta=±1$
If $\sin2\theta=1$ i.e, $2\theta=2n\pi+\frac{\pi}{2}=>9X^2+Y^2=1$
If $\sin2\theta=-1$ i.e, $2\theta=2n\pi-\frac{\pi}{2}=>X^2+9Y^2=1$
Alternatively, according to 7.2 of this,
Here a=b=5, h=4, f=g=0
So,
$C=\begin{pmatrix} a & h \\ h & b \end{pmatrix}=5^2-4^2>0$
$Δ= \begin{pmatrix} a & h & g \\ h & b & f \\g & f & c \end{pmatrix}$ $=\begin{pmatrix} 5 & 4 & 0 \\ 4 & 5 & 0 \\0 & 0 & -1 \end{pmatrix}$ $=-(5^2-4^2)=-9<0$
and $aΔ=5(-9)<0$
So, the given curve represents an ellipse.