This is to prove the uniform convergence using Sasha's approach. Write $g(a,b,\gamma)$ for $(a-b)^2+4ab\sin^2(\gamma/2)$. Then $\lim_{(a,b)\to (0,0)}\frac{\cos^{-1}(f(a,b,\lambda,\gamma)}{\cos^{-1}(f(a,b,1,\gamma)}=|\lambda|\lim_{(a,b)\to (0,0)}\sqrt{\frac{g(a,b,\gamma)+o((a-b)^2)+o(ab)\sin^2(\gamma/2)} {g(a,b,\gamma)+o_1((a-b)^2)+o_1(ab)\sin^2(\gamma/2)}},$ where $o,o_1$ are independent of $\gamma$. Here we are using the fact that $\lim_{x\to 0}\frac{\cos^{-1}(1-x)}{\sqrt{2x}}=1.$ Now divide the numerator and the denominator of the fraction inside the square root sign by $g(a,b,\gamma)$ (which is $>0$ since $(a,b)\neq (0,0)$ and \gamma>0). Note that $o((a-b)^2)=0$ if $a=b,$ and $o((a-b)^2)/g(a,b,\gamma)\le o((a-b)^2)/(a-b)^2\to 0.$
Also $o(ab)\sin^2(\gamma/2)/g(a,b,\gamma)=\frac{o(ab)\sin^2(\gamma/2)}{(a-b)^2+4ab\sin^2(\gamma/2)}\le o(ab)/ab\to 0.$ Same for $o_1$. So the limit converges uniformly for $0<\gamma\le\pi$ to $|\lambda| $ as $(a,b)\to (0,0)$.
Actually I also need to show that the two limits $\lim_{(a,b)\to (0,0)}\frac{\cos^{-1}(f(a,b,\lambda,\gamma))}{\sqrt{2(1-f(a,b,\lambda,\gamma))}}$ and $\lim_{(a,b)\to (0,0)}\frac{\cos^{-1}(f(a,b,1,\gamma))}{\sqrt{2(1-f(a,b,1,\gamma))}}$ also converge to 1 as $(a,b)\to (0,0)$ uniformly for $0<\gamma\le\pi$. But this is easier since it is equivalent to both $f(a,b,\lambda,\gamma)$ and $f(a,b,1,\gamma)$ converging to 1 uniformly for $0<\gamma\le\pi$.