The conjecture is true.
Define, as in @S.C.B.'s comment, the sequence of Telephone numbers:
$$
T_0 = T_1 = 1, \quad T_{n+1} = T_{n} + nT_{n-1},\ n\ge 1, \tag{1}
$$
so that
$$
x_n = \frac{T_n}{T_{n-1}},\ n\ge 1.
$$
Then
$$
T_n = \sum_{k=0}^{\lfloor n/2\rfloor} {n\choose 2k}(2k-1)!!.
$$
I will prove first @S.C.B.'s claim:
Claim For each $n\ge 1$, $T_n$ and $T_{n+1}$ have no common odd prime divisor.
It relies on
Lemma For each $n\ge 1$, $T_n$ and $n$ have no common odd prime divisor.
Proof Let $p>2$ be an odd prime dividing $n$. Write
$$
T_n = 1 + \sum_{k=1}^{\lfloor n/2\rfloor} {n\choose 2k}(2k-1)!!.
$$
Note that $p\mid {n\choose 2k}$ for $1\le k< p/2$, and $p\mid (2k-1)!!$ for $k> p/2$. Therefore, $T_n = 1\pmod p$, yielding the statement.
Proof of the claim Let $p>2$ be a prime. Assume, by way of contradiction, that $n$ is the smallest positive integer such that $p$ divides both $T_{n+1}$ and $T_n$. Then it follows from $(1)$ that $p\mid n$, contradicting Lemma.
Therefore, the only possibility for $x_n$ to be an integer is $T_{n-1}$ being a power of $2$. However, as it is explained in the Wikipedia page, this is not possible for $n\ge 4$.