solve $$y''-6y'-25y=2sinx+3cosx$$
I know that we first have to solve the homogeneous equation:
$$y''-6y'-25y=0$$
$$\lambda^2-6\lambda-25=0$$
$$\lambda_{1,2}\frac{6\pm\sqrt{36-100}}{2}=\frac{6 \pm 8i}{2}={3 \pm 4i}$$
So $y=c_{1}e^{3x}cos4x+c_{2}e^{3x}sin4x$
How should I continue by variation of variables? undetermined coefficients?
I think that you made a small mistake solving $$\lambda^2-6\lambda-25=0$$ since $\Delta=6^2\color{red}{+}4\times25=136$ which makes $$\lambda_{1,2}=3\pm\sqrt{34}$$ and the solution of the homogeneous equation is rather $$y=c_1 e^{\left(3-\sqrt{34}\right) x}+c_2 e^{\left(3+\sqrt{34}\right) x}$$ Now, for the particular equation, assume $$y=A\sin(x)+B\cos(x)$$ $$y'=A\cos(x)-B\sin(x)$$ $$y''=-A\sin(x)-B\cos(x)$$ and replace. After simplification, you would get $$(6B-26A)\sin(x)-(6A+26B)\cos(x)=2\sin(x)+3\cos(x)$$ Identify $$6B-26A=2$$ $$-6A-26B=3$$ Just solve for $A,B$.
If you've solved for y, you're done. If you're trying to determine the coefficients, then you need some initial condition(s) to do that.