Set $x_{n+1}=f(x_n)=\alpha x_n(3-\frac{x_n^2}{a})+\beta x_n(1+\frac{a^2}{x_n^2})$ and denote $z=\sqrt{a}$. Replace $x_n$ by $x$.
Using Taylor's development we find:
$$f(x)=f(z)+f'(z)(x-z)+\frac{f''(z)}{2}(x-z)^2+\frac{f'''(z)}{6}(x-z)^3+\frac{f''''(\xi)}{24}(x-z)^4$$
for some $\xi\in(x,z)$. Since $f(z)=z$ and dividing the previous equation by $|x-z|^3$ we have:
$$\frac{f(x)-f(z)}{|x-z|^3}=\frac{f'(z)}{|x-z|^2}+\frac{f''(z)}{2|x-z|}+\frac{f'''(z)}{6}+\frac{f''''(\xi)}{24}|x-z|$$
So in order to converge to the $3^{rd}$ order we have to impose both $f'(z)=0$ and $f''(z)=0$. This give raise to the following system of equations:
$$
\begin{align}
\beta(1-a)&=0\\
-3\alpha+\beta&=0
\end{align}
$$
and hence
$$
\begin{align}
\beta(1-a)&=0\\
\beta&=3\alpha&
\end{align}
$$
Looking at the proposed solutions we have to choose $\color{red}{\boxed{\color{black}{(2)}}}$
Remark:
solution $(4)$ seems to be a duplicate of $(2)$.