The comment you added, that $a^2 + b^2 = a_0^2$, makes things considerably more reasonable. Note that already, this yields $$ \tan^2\theta = \frac{b^2}{a^2} = \frac{a_0^2 - a^2}{a^2} = \frac{a_0^2}{a^2} - 1 ~~, $$ and as $\tan^2\theta + 1 =\sec^2\theta$ , we have that $\sec\theta = a_0/a$ , or $a = a_0 \cos \theta$ .
As for the differential equation, we can reduce the order once using the trick of 'integrating to a square'; not sure what the / any formal name is. Multiply by $\dot\theta$ to find that $$ \dot\theta \ddot\theta = -k \dot\theta\sin(2\theta) ~~. $$ Noting that both sides are perfect differentials, integrate with respect to $\theta$, including the constant: $$ \frac{1}{2}\dot\theta^2 = \frac{k}{2}\cos(2\theta) + c ~~. $$ A double angle for cosine is $\cos(2\theta) = 2\cos^2\theta - 1 = 2a^2/a_0^2 - 1$. Hence, absorbing constants, $$ \dot\theta^2 = \frac{2k}{a_0^2}a^2 + c ~~. $$ Note that $\dot a = -a_0 \dot\theta \sin \theta$ , and so $\dot a^2 = \dot\theta^2 (a_0^2 - a^2)$. Now at this point I have to make the undue assumption that you've been given an additional piece of information that would allow us to force $c = 0$; otherwise the solution does not come out tractable whatsoever. Connecting these pieces, we now have that $$ \frac{a_0}{\sqrt{2k}}\frac{\dot a}{a\sqrt{a_0^2-a^2}} = 1 ~~. $$ Wolfram or a textbook will confirm that the LHS has the same form as the derivative of arc-hyperbolic secant (assuming $a\ge 0$), and integrating precisely we find that $$ \text{asech}\left(\frac{a}{a_0}\right) = \sqrt{2k} t + C ~~, $$ for some new constant $C$. Using the initial condition for $a$ yields $C = 0$, and so finally $$ a = a_0 ~\text{sech} (\sqrt{2k} t) ~~. $$