Taking the origin as the intersection of $a$ and $b$, with the equation in terms of the angle $\theta$ at that vertex:
Through the cosine law we have:
$$(a-2)^2=a^2+100-20a\cos\theta$$ $$-4a+4=100-20a\cos\theta$$ $$a=\frac {96}{20\cos\theta-4}$$
With our particular setup, $a$ is the radius. So this is a polar equation, where
$$r(\theta)=\frac{96}{20\cos\theta-4}=\frac{24}{5\cos\theta-1}$$
we can convert to cartesian coordinates if you want:
$$5r\cos\theta-r=24$$ $$(5r\cos\theta-24)^2=r^2$$ $$(5x-24)^2=x^2+y^2$$ $$y^2=24x^2-240x+576$$ $$y^2=24(x-5)^2-24$$ $$(x-5)^2-\frac{y^2}{24}=1$$
which is a hyperbola. You want the rightmost branch (so $a>0$), which is valid when $\cos\theta<\frac 1 5$.
EDIT: picture time!
