This is the problem of finding the intersection of a straight line and a
circle, as commented by J.M.. The more elementary method
using analytical geometry, without rotate or translate the coordinate
axes (which would make the computation easier$^1$), although not being a compact one, is the following (see sketch).

The equation defined by points $
S(N,J)$ and $T(M,I)$ is given by
$$
y-J=m(x-N),\qquad m=\frac{I-J}{M-N}\tag{1}.
$$
The equation of the circle centered at $R$ with radius $d=\overline{RQ}$ is
$$
(x-L)^{2}+(y-H)^{2}=d^{2}.\tag{2}
$$
You need to solve the following system
$$
\left\{
\begin{array}{c}
y-J=m(x-N) \\
(x-L)^{2}+(y-H)^{2}=d^{2},
\end{array}\tag{3}
\right.
$$
which is equivalent to
$$
\left\{
\begin{array}{c}
x=\frac{y-J+mN}{m} \\
\left(\frac{y-J+mN}{m}-L\right)^{2}+(y-H)^{2}=d^{2}.\tag{4}
\end{array}
\right.
$$
Solving the quadratic equation yields (with the help of SWP):
$$
y=\frac{1}{ m^{2}+1 }\left( -mN+Lm+J+m^{2}H\pm \sqrt{\Delta}\right), \tag{5}
$$
where the discriminant is
$$\begin{eqnarray*}
\Delta &=&A+B, \\ \text{with }
A
&=&-m^{4}N^{2}+m^{4}d^{2}-m^{4}L^{2}-m^{2}J^{2}-m^{2}H^{2}+d^{2}m^{2}-2m^{3}NH,
\\
B &=&2Lm^{3}H+2Jm^{2}H+2m^{4}NL-2m^{3}JL+2m^{3}JN.\tag{6}
\end{eqnarray*}$$
The information $\overline{RT}<\overline{RQ}<\overline{RS}$ will define the
signal of the term $ \pm \sqrt{\Delta}$. The coordinates of $Q$ are $O=x,K=y$.
$^1$By making the translation $X=x-L$ and $Y=y-H$, and computing the new coordinates of the points in this $X,Y$ system, the above formulae simplify
(it is equivalent to set $L=H=0$ in them). In the end they should be convert back to the original $x,y$ system.