You make the change of variable $z=e^{ix}$. Then $$ dx=\frac{1}{i}\frac{dz}{z}, $$ $$ \frac{1}{1+\cos^2x}=\frac{1}{1+(z+z^{-1})^2}=\frac{4\,z^2}{z^4+6\,z^2+1}, $$ and $$ \int_0^{2\pi}\frac{dx}{1+\cos^2x}=\frac{1}{i}\int_{|z|=1}\frac{4\,z}{z^4+6\,z^2+1}\,dz. $$ To apply the residue theorem you need the poles inside the circle $\{|z|=1\}$, that is, the solutions of $$ z^4+6\,z^2+1=0,\quad |z|<1. $$ Solving for $z^2$ gives $$ z^2=-3\pm2\,\sqrt2. $$
There are no double poles. You are interested only on the poles in the unit disk. Since $$ |-3-2\,\sqrt2|>1\text{ and }|-3+2\,\sqrt2|<1, $$ you have to consider only $$ z^2=2\,\sqrt2-3\ . $$ This gives you two simple poles at $$ z=\pm\sqrt{2\,\sqrt2-3\,}\ . $$