You make the change of variable $z=e^{ix}$. Then $ dx=\frac{1}{i}\frac{dz}{z}, $ $ \frac{1}{1+\cos^2x}=\frac{1}{1+(z+z^{-1})^2}=\frac{4\,z^2}{z^4+6\,z^2+1}, $ and $ \int_0^{2\pi}\frac{dx}{1+\cos^2x}=\frac{1}{i}\int_{|z|=1}\frac{4\,z}{z^4+6\,z^2+1}\,dz. $ To apply the residue theorem you need the poles inside the circle $\{|z|=1\}$, that is, the solutions of $ z^4+6\,z^2+1=0,\quad |z|<1. $ Solving for $z^2$ gives $ z^2=-3\pm2\,\sqrt2. $
There are no double poles. You are interested only on the poles in the unit disk. Since $ |-3-2\,\sqrt2|>1\text{ and }|-3+2\,\sqrt2|<1, $ you have to consider only $ z^2=2\,\sqrt2-3\ . $ This gives you two simple poles at $ z=\pm\sqrt{2\,\sqrt2-3\,}\ . $