You make the change of variable $z=e^{ix}$. Then
$$
dx=\frac{1}{i}\frac{dz}{z},
$$
$$
\frac{1}{1+\cos^2x}=\frac{1}{1+(z+z^{-1})^2}=\frac{4\,z^2}{z^4+6\,z^2+1},
$$
and
$$
\int_0^{2\pi}\frac{dx}{1+\cos^2x}=\frac{1}{i}\int_{|z|=1}\frac{4\,z}{z^4+6\,z^2+1}\,dz.
$$
To apply the residue theorem you need the poles inside the circle $\{|z|=1\}$, that is, the solutions of
$$
z^4+6\,z^2+1=0,\quad |z|<1.
$$
Solving for $z^2$ gives
$$
z^2=-3\pm2\,\sqrt2.
$$
There are no double poles. You are interested only on the poles in the unit disk. Since
$$
|-3-2\,\sqrt2|>1\text{ and }|-3+2\,\sqrt2|<1,
$$
you have to consider only
$$
z^2=2\,\sqrt2-3\ .
$$
This gives you two simple poles at
$$
z=\pm\sqrt{2\,\sqrt2-3\,}\ .
$$