The validity of the result holds as long as you take your branches correctly. For example, if $\epsilon < a < b$, you can take the branches (blue for $a$ and green for $b$) as

and if $a < \epsilon < b$, then you can take the branches as 
In the thick part, the branches cancel, the path is well defined and you can use Cauchy's integral formula without problems,
$ \oint_{|z| = \epsilon} \frac{\sqrt{(z-a)(z-b)}}{z} dz = 2\pi i \sqrt{(z-a)(z-b)}\Big|_{z = 0} = 2 \pi i \sqrt{ab} $
If both $a$ and $b$ are negative, the same argument applies.
Case $a < \epsilon$ and $b < \epsilon$
In this case, you have to take
$\hskip1.3in$
and then see what happens with the small branch going from $a$ to $b$ by taking the contour
$\hskip1.3in$
and make the gap of the external circle go to zero. Then the result will be the contribution of the branch plus the pole.