Can we prove this conjecture concerning Pell equations?

Question

For every positive integer $n$, not being a perfect square, denote the fundamental solution of the Pell equation $$x^2-ny^2=1$$ with $(a,b)$. In other words, $b$ is the smallest positive integer such that $nb^2+1$ is a perfect square.

Define $f(n)=b$. Searching for $n$ with large $b$ , I came to the following conjecture :

"If $s>46$ is a composite non-square number, there is a prime $pf(s)."$

To formulate it in another way : If we write down every non-square $s$ for which we get a new record for the largest number $b$ (for all smaller numbers, we get a smaller $b$), the only composite numbers that are written down are $10$ and $46$. The other numbers written down are prime numbers.

The conjecture is true for $s\le 10^6$, but can we prove it for all $s>46$ ?