Let's call the given two points (vectors) as
$$
P_{\,1} = \left( {x_{\,1} ,y_{\,1} ,z_{\,1} } \right)\quad P_{\,2} = \left( {x_{\,1} ,y_{\,1} ,z_{\,1} } \right)
$$
then a generic point $P$ on the line connecting the two points will be given by
$$
P \in \overline {P_{\,1} P_{\,2} } \quad \Rightarrow \quad P = t\,P_{\,1} + \left( {1 - t} \right)P_{\,2}
$$
and when $0 \leqslant t \leqslant 1$ the point will be internal to segment, and otherwise external.
The squared modulus of $P$ represents the square of its distance from origin and is given by the dot product by itself:
$$
\left| P \right|^{\,2} = P \cdot P = t^{\,2} \,P_{\,1} \cdot P_{\,1} + \left( {1 - t} \right)^{\,2} P_{\,2} \cdot P_{\,2} + 2t\left( {1 - t} \right)P_{\,2} \cdot P_{\,1}
$$
The various dot products are scalars, and quite simple to calculate, and you have a simple quadratic equation.
From here you can follow two possible approaches:
- Find the points and then their distance
The point $P$ is constrained on the line, then to constained it to lie on the sphere, just equate $\left| P \right|^{\,2} $ to $R^2$, and solve for $t$.
The values obtained will tell you whether the points are internal/external to the segment.
The two intersection points will then be
$$
P_{a,\,b} = t_{a,\,b} \,P_{\,1} + \left( {1 - t_{a,\,b} } \right)P_{\,2}
$$
and distance will follow obviously.
- Find the distance and then the points
Since your objective is the length of the segment, then you can more quickly
derive the expression of $\left| P \right|^{\,2}$ with respect to $t$ and find the value $t_0$
that minimize it.
$$
\begin{gathered}
0 = \frac{d}
{{dt}}\left| P \right|^{\,2} = 2P_{\,1} \cdot P_{\,1} \;t - 2\left( {1 - t} \right)P_{\,2} \cdot P_{\,2} + \left( {2 - 4t} \right)P_{\,2} P_{\,1} \hfill \\
\Rightarrow \left( {P_{\,1} \cdot P_{\,1} + P_{\,2} \cdot P_{\,2} - 2P_{\,2} P_{\,1} } \right)\;t - P_{\,2} \cdot P_{\,2} + P_{\,2} P_{\,1} = 0 \hfill \\
\end{gathered}
$$
This will correspond to the point on the segment which is closest to the origin. Therefore
$$
R^{\,2} - \left| P \right|^{\,2} (t_{\,0} ) = \left( {L/2} \right)^{\,2}
$$
and you can bypass solving the quadratic equation.
Then to find the points of intersection, if you need them, consider that $t$ as defined above is the measure
of the relative distance (i.e. "with sign") of the point $P$ from $P1$ along the line, in the direction from $P_1$ to $P_2$,
and normalized vs the absolute distance between $P_2$ and $P_1$.
Therefore
$$
P_{a,\,b} = P\left( {t_{\,0} \pm \frac{{L/2}}
{{\left| {P_{\,2} - P_{\,1} } \right|}}} \right)
$$