Taking the origin as the intersection of $a$ and $b$, with the equation in terms of the angle $\theta$ at that vertex:
Through the cosine law we have:
$(a-2)^2=a^2+100-20a\cos\theta$ $-4a+4=100-20a\cos\theta$ $a=\frac {96}{20\cos\theta-4}$
With our particular setup, $a$ is the radius. So this is a polar equation, where
$r(\theta)=\frac{96}{20\cos\theta-4}=\frac{24}{5\cos\theta-1}$
we can convert to cartesian coordinates if you want:
$5r\cos\theta-r=24$ $(5r\cos\theta-24)^2=r^2$ $(5x-24)^2=x^2+y^2$ $y^2=24x^2-240x+576$ $y^2=24(x-5)^2-24$ $(x-5)^2-\frac{y^2}{24}=1$
which is a hyperbola. You want the rightmost branch (so $a>0$), which is valid when $\cos\theta<\frac 1 5$.
EDIT: picture time!
