If the sequence should converge to $x$, then we must have $\tag1x(3x^2-1)=2x^3$ because the given recursion implies $x_{n+1}(3x_n^2-1)=2x_n^3$. Equation $(1)$ can be transformed to $x^3-x=0$, i.e. $x\in \{-1,0,1\}$.
First note that if $a<0$ then we obtain the same $x_2=\frac{-2a^3}{3a^2-1}$ as we would with $-a$ instead. Therefore these cases are symmetric to the positive case: $a$ produces a converging sequence iff $-a$ does.
Note that $\tag2\frac{x_{n+1}}{x_n}=\frac{2x_n^2}{3x_n^2-1}.$ The right hand side is $<1$ if $x_n>1$. Also note that $\tag3x_{n+1}-1=\frac{2x_n^3-3x_n^2+1}{3x_n^2-1}=\frac{(2x+1)(x-1)^2}{3x_n^2-1}$ and the right hand side is positive if $x>1$. Thus by induction $a>1$ implies that all $x_n$ are $>1$ and the sequence is strictly decreasing. It must therefore converge to a limit $\ge 1$, hence to $+1$.
The case $a=1$ is trivial as it leads to the constant sequence $x_n=1$.
If $\frac1{\sqrt 3}, then we read from $(3)$ that $x_2>1$, hence from then on, the sequence is again decreasing and converges $\to 1$.
If $a=\frac1{\sqrt 3}$, then the sequence is not defined because $x_2$ is not defined.
If $0<|x_n|<\frac1{\sqrt 5}$, then $2x_n^2<1-3x_n^2$ implies via $(2)$ that $\left\vert\frac{x_{n+1}}{x_n}\right\vert$ is strictly decreasing, hence $x_n\to 0$ if $0.
If $x_n^2=\frac15$, then $(2)$ shows that $x_{n+1}=-x_n$. Hence $a=\frac1{\sqrt 5}$ leads to the nonconverging sequence $x_n=(-1)^{n-1}a$.
If $\frac15, then $0<1-3x_n^2<2x_n^2$, hence $(2)$ implies $|x_{n+1}|>|x_n|$. Therfore, with $\frac1{\sqrt 5} two things can happen:
- The sequence $x_n^2$ remains bounded by $\frac 13$. Then it is increasing and hence converges to some $b$ with $b=\frac{4b^3}{(3b-1)^2}$, which implies $b=0$ or $b=1$ or $b=\frac15$. Since the limit $b$ must also be $>\frac15$ and $\le \frac13$, this cannot happen.
- For some $n$, we have $x_n^2>\frac 13$. From then on, the sequence converges as seen above.
- For some $n$, we have $x_n^2=\frac 13$. Then $x_{n+1}$ is not defined.
The third case does happen for all members of a sequence beginning $\tag 4 0.46560062143367758\ldots,\\0.4472135954999579\ldots,\\ 0.450201477782475\ldots,\\0.44770950581291\ldots$
Summary: The only cases where the sequence does not converge are given by $a=\pm\frac1{\sqrt 5}$, $a=\pm\frac1{\sqrt 3}$ and when $|a|$ occurs in the sequence (4).
Remark: The sequence (4) is obtained by letting $a_0=\frac1{\sqrt 3}$ and then $a_{n+1}$ the unique solution of $2a_{n+1}^3-a_n(3a_{n+1}^2-1)=0$ between $\frac1{\sqrt 5}$ and $\frac1{\sqrt 3}$.