Your equation is $y'''-2y''-4y'+8y=6xe^{2x}$. Now change the $y'$ to $Dy$ form as follows. So $y'''\to D^3y,\\ y''\to D^2y, \; \; \text{and} \;\;y'\to Dy,$ So by new arranging respect to $D$ operator we get our equation as: $D^3y-2D^2y-4Dy+8y=6xe^{2x}$ or by factoring and expanding $(D^3-2D^2-4D+8)y=(D-2)^2(D+2)y=6e^{2x}$ which you got before. Note that considering the corresponding homogeneous equation $(D-2)^2(D+2)y=0$ we get $(D-2)^2=0,\;\; (D+2)=0$ which leads us to write the general solutions as $y_c(x) = c_1e^{2x} + c_2xe^{2x}+c_3e^{-2x}$ and you did it right above before. Now have a look at some facts:
- If $y=\text{constant}$ so $y'=0$ or $Dy=0$. Here, the operator $D$ annihilates $y$ which is just a constant.($Dc=0$)
- If $y=cx$ in which $c$ is a constant so $y''=0$ or $D^2y=0$. It means that the operator polynomial $P(D)=D^2$ annihilates $y=cx$. $(\text{or} \; P(D)=D^2(cx)=cD^2x=c(x'')=0)$. Generally, $D^{n+1}$ annihilates not only the function $y=cx^{n}$ but also all linear functions as $y=c_0+c_1x+c_2x^2+...+c_nx^n$ It means that $P(D)y=D^{n+1}y=0$.
- As the same the differential operator $(D-\alpha)^n$ annihilates each of the following functions and every linear combinations of them: $e^{\alpha x},xe^{\alpha x},x^2e^{\alpha x},...,x^{n-1}e^{\alpha x}$ Now look at the RHS of your original equation. I mean $=6xe^{2x}$. Can we guess of which proper differential polynomial annihilates it? As above it would be $(D-2)^2$. It means that $(D-2)^2 \left(6xe^{2x}\right)=0$. Don't respect to numeric coefficients like $6$ here at all.
Now we consider of what we have achieved at last: $(D-2)^2(D+2)y=6e^{2x}$ Put the operator $(D-2)^2$ before both sides of the above converted equation: $(D-2)^2\left((D-2)^2(D+2)y\right)=(D-2)^26e^{2x}=0$ or $(D-2)^4(D+2)y=0$ In fact, we have found a proper differential operator $P(D)=(D-2)^4(D+2)$ which if it effects to $y$, $y$ will be lost.
Now, for a while, forget our equation and look at $(D-2)^4(D+2)y=0$ and think someone gave this to us asking to guess which function $y$ may satisfy the equality above? We reply:
- Since we have $(D-2)$ so we have some forms as $e^{2x}$ in $y$.
Since $(D-2)$ has a power $4$ so we have the forms $Ae^{2x},\; Bxe^{2x}, \;Cx^2e^{2x}, \text{and} \; Ex^3e^{2x}$ in $y$. Note that you multiply $e^{2x}$, in the previous line, by $A,\; Bx,\; Cx^2,\; Ex^3$. (Exactly until the power of $x$ gets $4-1=3$).
And, since we have $(D+2)$, then $y$ has the term $Fe^{-2x}$.
So we are done. Our probable function which satisfy the original equation is $y=Ae^{2x}+ Bxe^{2x}+Cx^2e^{2x}+Ex^3e^{2x}+Fe^{-2x}$. Now put the terms which generate $y_c(x)$ aside and take the rest for what we have looked for. It is $y_p=Cx^2e^{2x}+Ex^3e^{2x}$ where $C,E$ are unknown constant.