How to find the power series of rational functions of type, e.g. $$\frac{1}{1-6x+12x^2},\frac{6x}{1-6x+12x^2}$$
where denominator can't be factorized over the real domain.
Is there a way to use the method of undetermined coefficients (by completing the square), or is it necessary to use complex domain and trigonometry?
Here is an approach based upon the geometric series expansion (see the comment from @YvesDaoust).
We obtain \begin{align*} \frac{1}{1-6x+12x^2}&=\frac{1}{1-6x(1-2x)}\\ &=\sum_{k=0}^\infty (6x)^k(1-2x)^k\\ &=\sum_{k=0}^\infty (6x)^k\sum_{j=0}^k\binom{k}{j}(-2x)^j\\ &=1+6x+24x^2+72x^3+144x^4\\ &\qquad+\color{grey}{0}x^5-1728x^6-10368x^7-41472x^8-\cdots \end{align*}
It is convenient to use the coefficient of operator $[x^n]$ to extract the coefficient of $x^n$.
We get \begin{align*} [x^n]\frac{1}{1-6x+12x^2}&=[x^n]\sum_{k=0}^\infty (6x)^k\sum_{j=0}^k\binom{k}{j}(-2x)^j\\ &=\sum_{k=0}^n6^k[x^{n-k}]\sum_{j=0}^k\binom{k}{j}(-2x)^j\tag{1}\\ &=\sum_{k=0}^n6^k\binom{k}{n-k}(-2)^{n-k}\\ &=(-2)^n\sum_{k=0}^n\binom{k}{n-k}(-3)^k\tag{2} \end{align*} and find this way a summation formula for the coefficient of the power series.
Comment:
In (1) we use the linearity of the coefficient of operator and the rule $[x^p]x^qA(x)=[x^{p-q}]A(x)$.
In (2) we select the coefficient of $x^{n-k}$.
Note: Since the zeros of $f(x)=1-6x+12x^2$ are non-real, we don't expect to get a closed formula of the binomial sum formula in real numbers only. In fact the following is valid (see e.g. formula 1.70 in this paper by R. Sprugnoli). \begin{align*} \sum_{k=0}^n\binom{n-k}{k}z^k =\frac{1}{\sqrt{1+4z}} \left(\left(\frac{1+\sqrt{1+4z}}{2}\right)^{n+1}-\left(\frac{1-\sqrt{1+4z}}{2}\right)^{n+1}\right) \end{align*} Replacing the index $k$ with $n-k$ gives \begin{align*} \sum_{k=0}^n\binom{k}{n-k}z^{n-k} =\frac{1}{\sqrt{1+4z}} \left(\left(\frac{1+\sqrt{1+4z}}{2}\right)^{n+1}-\left(\frac{1-\sqrt{1+4z}}{2}\right)^{n+1}\right) \end{align*}
This expression evaluated at $z=-\frac{1}{3}$ and multiplied with $(-2)^n$ gives the coefficient of the power series and we obtain \begin{align*} (-2)^n\sum_{k=0}^n\binom{k}{n-k}(-3)^k =\frac{i\sqrt{3}}{2} \left(\left(1-\frac{i}{\sqrt{3}}\right)^{n+1}-\left(1+\frac{i}{\sqrt{3}}\right)^{n+1}\right)3^n \end{align*}