Some important translations:
$\tag 1\sin(x\pm 2 \pi) = \sin x $ $\tag {1'}\cos(x\pm 2 \pi) = \cos x $ $\tag 2\cot(x\pm \pi)= \cot x$ $\tag {2'}\tan(x\pm \pi)= \tan x$
$\tag 3 \sin \left(\frac \pi 2 -x \right)=\cos x$ $\tag 4 \cos \left(\frac \pi 2 -x \right)=\sin x$ $\tag 5 \sin(\pi-x)=\sin x$ and $\tag 6\cos(\pi-x)=-\cos x$
and $\tag 7 \sin(-x)=-\sin x$ $\tag 8 \cos (-x) = \cos x$
Although taking $\alpha =0$ reveals that the equality doesn't hold, assume that there is no typo, then, we could move on as follows.
You have that
$ \frac{\sin^2 \left(\frac{5\pi}{6} - \alpha \right)}{\cos^2(\alpha - 4\pi)} - \cot^2(\alpha - 11\pi)\sin^2 \left(-\alpha - \frac{13\pi}{2}\right) =\sin^2(\alpha)$
Using the above, we can write
$\eqalign{ & {\sin ^2}\left( {\frac{{5\pi }}{6} - \alpha } \right) = {\left[ { - \sin \left( {\alpha - \frac{{5\pi }}{6}} \right)} \right]^2} = {\sin ^2}\left( {\alpha - \frac{{5\pi }}{6}} \right) \cr & {\cot ^2}(\alpha - 11\pi ) = {\cot ^2}\left( {\alpha - 10\pi } \right) = \cdots = {\cot ^2}\alpha \cr & {\sin ^2}\left( { - \alpha - \frac{{13\pi }}{2}} \right) = {\left[ { - \sin \left( {\alpha + \frac{{13\pi }}{2}} \right)} \right]^2} = \sin {\left( {\alpha + \frac{{13\pi }}{2}} \right)^2} \cr & {\cos ^2}(\alpha - 4\pi ) = {\cos ^2}(\alpha - 2\pi ) = {\cos ^2}\alpha \cr} $
so that we have
$\frac{{{{\sin }^2}\left( {\alpha - \frac{{5\pi }}{6}} \right)}}{{{{\cos }^2}\alpha }} - {\cot ^2}\alpha {\sin ^2}\left( {\alpha + \frac{{13\pi }}{2}} \right) = {\sin ^2}\alpha $
Now
${\sin ^2}\left( {\alpha - \frac{{5\pi }}{6}} \right) = {\sin ^2}\left( {\alpha - \frac{{3\pi }}{6} - \frac{{2\pi }}{6}} \right) = {\sin ^2}\left( {\alpha - \frac{\pi }{3} - \frac{\pi }{2}} \right) = {\left( { - 1} \right)^2}{\sin ^2}\left( {\frac{\pi }{2} - \left( {\alpha - \frac{\pi }{3}} \right)} \right) = {\cos ^2}\left( {\alpha - \frac{\pi }{3}} \right)$
and
$\eqalign{ & {\sin ^2}\left( {\alpha + \frac{{13\pi }}{2}} \right) = {\sin ^2}\left( {\alpha + \frac{{12\pi }}{2} + \frac{\pi }{2}} \right) = {\sin ^2}\left( {\alpha + 6\pi + \frac{\pi }{2}} \right) = {\sin ^2}\left( {\alpha + \frac{\pi }{2}} \right) \cr & = {\sin ^2}\left( {\frac{\pi }{2} - \left( { - \alpha } \right)} \right) = {\cos ^2}\left( { - \alpha } \right) = {\cos ^2}\alpha \cr} $
so that you have
$\frac{{{{\cos }^2}\left( {\alpha - \frac{\pi }{3}} \right)}}{{{{\cos }^2}\alpha }} - {\cot ^2}\alpha {\cos ^2}\alpha = {\sin ^2}\alpha $
Now, solving for ${{{\cos }^2}\left( {\alpha - \frac{\pi }{3}} \right)}$
gives
${\cos ^2}\left( {\alpha - \frac{\pi }{3}} \right) = {\cos ^2}\alpha {\sin ^2}\alpha + {\cos ^6}\alpha \frac{1}{{{{\sin }^2}\alpha }}$
There is some typo in your excercise, since letting $\alpha =0$ gives $1/4$ on the LHS and is not defined for the RHS. When you discover what the typo is, move on with the listed translations.