Starting from $$I= \int\frac{dx}{(a^2-\tan^2 x)\sqrt{b^2-\tan^2 x}}$$ and using, as you did, $$b^2-\tan^2 x= t^2\implies x=-\tan ^{-1}\left(\sqrt{b^2-t^2}\right)\implies dx=\frac{t}{\sqrt{b^2-t^2} \left(b^2-t^2+1\right)}$$ $$I=\int\frac{dx}{\sqrt{b^2-t^2} \left(b^2-t^2+1\right) \left(a^2-b^2+t^2\right)}$$ The integrand can be decomposed as $$\frac{1}{a^2 \sqrt{b^2-t^2}}+\frac{\sqrt{b^2-t^2}}{\left(-a^2-1\right)
\left(b^2-t^2+1\right)}-\frac{\sqrt{b^2-t^2}}{a^2 \left(a^2+1\right)
\left(-a^2+b^2-t^2\right)}$$ leading, as given in comments, to $$a(a^2+1)\,I=\frac{\tan ^{-1}\left(\frac{a t}{\sqrt{a^2-b^2}
\sqrt{b^2-t^2}}\right)}{\sqrt{a^2-b^2}}+\frac{a \tan ^{-1}\left(\frac{t}{\sqrt{b^2+1}
\sqrt{b^2-t^2}}\right)}{\sqrt{b^2+1}}$$
Edit
Continuing from Rohan's answer, partial fraction decomposition leads, for the integrand, to $$\frac 1 {a^2+1}\left(\frac 1 {a^2+(a^2-b^2)w^2}+\frac 1{1+(b^2+1)w^2} \right)$$ which is much easier to integrate. This would lead to $$(a^2+1) \,I=\frac{\tan ^{-1}\left(\frac{ \sqrt{a^2-b^2}}{a}w\right)}{a
\sqrt{a^2-b^2}}+\frac{\tan ^{-1}\left(\sqrt{b^2+1} w\right)}{\sqrt{b^2+1}}$$