Your equation is $y'''-2y''-4y'+8y=6xe^{2x}$. Now change the $y'$ to $Dy$ form as follows. So $$y'''\to D^3y,\\ y''\to D^2y, \; \; \text{and} \;\;y'\to Dy,$$ So by new arranging respect to $D$ operator we get our equation as: $$D^3y-2D^2y-4Dy+8y=6xe^{2x}$$ or by factoring and expanding $$(D^3-2D^2-4D+8)y=(D-2)^2(D+2)y=6e^{2x}$$ which you got before. Note that considering the corresponding homogeneous equation $$(D-2)^2(D+2)y=0$$ we get $(D-2)^2=0,\;\; (D+2)=0$ which leads us to write the general solutions as $$y_c(x) = c_1e^{2x} + c_2xe^{2x}+c_3e^{-2x}$$ and you did it right above before. Now have a look at some facts:
- If $y=\text{constant}$ so $y'=0$ or $Dy=0$. Here, the operator $D$ annihilates $y$ which is just a constant.($Dc=0$)
- If $y=cx$ in which $c$ is a constant so $y''=0$ or $D^2y=0$. It means that the operator polynomial $P(D)=D^2$ annihilates $y=cx$. $(\text{or} \; P(D)=D^2(cx)=cD^2x=c(x'')=0)$. Generally, $D^{n+1}$ annihilates not only the function $y=cx^{n}$ but also all linear functions as $$y=c_0+c_1x+c_2x^2+...+c_nx^n$$ It means that $$P(D)y=D^{n+1}y=0$$.
- As the same the differential operator $(D-\alpha)^n$ annihilates each of the following functions and every linear combinations of them: $$e^{\alpha x},xe^{\alpha x},x^2e^{\alpha x},...,x^{n-1}e^{\alpha x}$$ Now look at the RHS of your original equation. I mean $=6xe^{2x}$. Can we guess of which proper differential polynomial annihilates it? As above it would be $(D-2)^2$. It means that $(D-2)^2 \left(6xe^{2x}\right)=0$. Don't respect to numeric coefficients like $6$ here at all.
Now we consider of what we have achieved at last: $$(D-2)^2(D+2)y=6e^{2x}$$ Put the operator $(D-2)^2$ before both sides of the above converted equation: $$(D-2)^2\left((D-2)^2(D+2)y\right)=(D-2)^26e^{2x}=0$$ or $$(D-2)^4(D+2)y=0$$ In fact, we have found a proper differential operator $P(D)=(D-2)^4(D+2)$ which if it effects to $y$, $y$ will be lost.
Now, for a while, forget our equation and look at $(D-2)^4(D+2)y=0$ and think someone gave this to us asking to guess which function $y$ may satisfy the equality above? We reply:
- Since we have $(D-2)$ so we have some forms as $e^{2x}$ in $y$.
Since $(D-2)$ has a power $4$ so we have the forms $Ae^{2x},\; Bxe^{2x}, \;Cx^2e^{2x}, \text{and} \; Ex^3e^{2x}$ in $y$. Note that you multiply $e^{2x}$, in the previous line, by $A,\; Bx,\; Cx^2,\; Ex^3$. (Exactly until the power of $x$ gets $4-1=3$).
And, since we have $(D+2)$, then $y$ has the term $Fe^{-2x}$.
So we are done. Our probable function which satisfy the original equation is $$y=Ae^{2x}+ Bxe^{2x}+Cx^2e^{2x}+Ex^3e^{2x}+Fe^{-2x}$$. Now put the terms which generate $y_c(x)$ aside and take the rest for what we have looked for. It is $$y_p=Cx^2e^{2x}+Ex^3e^{2x}$$ where $C,E$ are unknown constant.