You have two important directions in your setup. $(0,0,1)$ is the direction of the axis connecting the poles. Any vector perpendicular to that will lie in the equatorial plane of $S$. $(\alpha,\beta,\gamma)$ is the normal of the plane $P$. Any vector perpendicular to this will lie within the plane. Now you can use the cross product of these two vectors to obtain a vector which is perpendicular to both. It will therefore be a vector in the plane $P$ and in the equatorial plane of $S$. This points in the direction of the semi-axis $m$, so if you scale that vector to length $a$ you'll get a point on the intersection.
To obtain the other direction, you need the perpendicular to the direction of $m$ within the plane. You can compute another cross product, this time with that direction and the normal of $P$ (again). Doing the math:
$$v=\left(\begin{pmatrix}0\\0\\1\end{pmatrix}\times
\begin{pmatrix}\alpha\\\beta\\\gamma\end{pmatrix}\right)\times
\begin{pmatrix}\alpha\\\beta\\\gamma\end{pmatrix}=
\begin{pmatrix}-\beta\\\alpha\\0\end{pmatrix}\times
\begin{pmatrix}\alpha\\\beta\\\gamma\end{pmatrix}=
\begin{pmatrix}\alpha\gamma\\\beta\gamma\\-\alpha^2-\beta^2\end{pmatrix}
$$
Now plug $s$ times this into the equation for $S$:
\begin{align*}
\left(\frac{s\alpha\gamma}a\right)^2+
\left(\frac{s\beta\gamma}a\right)^2+
\left(\frac{-s(\alpha^2+\beta^2)}b\right)^2&=1
\\
s^2(\alpha^2+\beta^2)
\left(\frac{\gamma^2}{a^2}+\frac{\alpha^2+\beta^2}{b^2}\right)&=1
\end{align*}
Solve for $s$ picking the positive solution and you get $s\,\lVert v\rVert$ as the length of the semiaxis that does not lie in the equatorial plane.
$$
n^2=s^2\,\lVert v\rVert^2
=\frac{\alpha^2\gamma^2+\beta^2\gamma^2+(\alpha^2+\beta^2)^2}{(\alpha^2+\beta^2)\left(\frac{\gamma^2}{a^2}+\frac{\alpha^2+\beta^2}{b^2}\right)}
=\frac{a^2b^2(\alpha^2+\beta^2+\gamma^2)}{a^2(\alpha^2+\beta^2)+b^2\gamma^2}
$$
In a comment you indicate that you did a sanity check with the plane $y=0$, i.e. $\alpha=0,\beta=1,\gamma=0$. In that case you get $n^2=\frac{a^2b^2}{a^2}=b^2$ as expected. For $\beta=2$ you get the same, which is important as $2y=0$ is the same plane.
]1