Since $\cos(2x)=2\cos^2x-1$, we find that (note we are viewing cosine as a complex function)
$2\cos\left(2\,\cos^{-1}\left(\frac{u}{2}\right)\right)=u^2-2.$
Repeated composition induces telescopy, which motivates the following (provable with induction):
$x_n=2\cos\left(2^{n-1}\cos^{-1}\left(\frac{x_1}{2}\right)\right).$
Using complex exponentials we see that ($\varphi=\cos^{-1}(x_1/2),~z=e^{i\varphi}$)
$2^n\cos\varphi~\cos2\varphi~\cdots~\cos 2^n\varphi=(z+z^{-1})(z^2+z^{-2})\cdots(z^{2^{n-1}}+z^{-2^{n-1}})$
$=\frac{z^{2^{n}}-z^{-2^n}}{z-z^{-1}}=\frac{\sin 2^n\varphi}{\sin\varphi}$
whence
$\gamma_n:=\frac{x_{n+1}}{x_1\cdots x_n}=2\cot\big(2^n\cos^{-1}(x_1/2)\big)\sqrt{1-(x_1/2)^2}.$
With $\lim\limits_{r\to+\infty}\cot(ir)=-i$, $x_1>2$, and $(\cos^{-1}v)/i$ a positive real for $v>1$, we find that
$\lim_{n\to\infty}\gamma_n=\sqrt{x_1^2-4}.$
In particular, the answer to our puzzle ($x_1=5$) is $\sqrt{21}$.
For numerical verification, we compare an approximation ($n=5$) to the intended answer:
$\color{Blue}{4.582575694955840006588047193728008488984456}\color{Red}{8357053937} \tag{a}$ $\color{Blue}{4.5825756949558400065880471937280084889844565767679719} \tag{b}$
a=5949389624883225721727/(5*23*527*277727*77132286527); b=Sqrt[21]