Let $P=(h,k)$, $A=(a \cos \theta, a\sin \theta)$ and $B=(a \cos \beta, a\sin \beta)$. Observe that the $\angle BOA=\pi - \alpha=\theta - \beta$. Therefore
$$
\cos(\theta -\beta) = -\cos\alpha .............. (1)
$$
Consider the right angled triangle $\triangle OBP$, since $OP$ is the bisector of $\angle BOA$, therefore
$$\cos^2\left(\frac{\theta -\beta}{2}\right)=\frac{OB^2}{OP^2}=\frac{a^2}{h^2+k^2}.$$
Using this in equation (1), we get
\begin{align*}
2\cos^2\left(\frac{\theta -\beta}{2}\right)-1 & = -\cos \alpha\\
\frac{2a^2}{h^2+k^2}-1 & =- \cos \alpha\\
\sec \alpha & = \frac{h^2+k^2}{h^2+k^2-2a^2}\\
\tan^2 \alpha +1 & = \left(\frac{h^2+k^2}{h^2+k^2-2a^2}\right)^2\\
\tan^2 \alpha (h^2+k^2-2a^2)^2 & = (h^2+k^2)^2-(h^2+k^2-2a^2)^2\\
\tan^2 \alpha (h^2+k^2-2a^2)^2 & = (4a^2)(h^2+k^2-a^2).
\end{align*}
Thus the locus is given by
$$\tan^2 \alpha (x^2+y^2-2a^2)^2 = (4a^2)(x^2+y^2-a^2).$$