I'm trying to use the method of undetermined coefficients, but I'm not sure what I should guess for $\cos^2(x)$. I know that you generally want to take the first few derivatives, so:
$f(x) = \cos^2(x)$
$f'(x) = -2\cos(x)\sin(x)$
$f''(x) = -2\cos(2x)$
$f^{(3)}(x) = 4\sin(2x)$
(now repeats between cosine and sine)
Solving the characteristic equation $5r^2+8r+8 = 0$ gives complementary solution $y_g = c_1e^{-4x/5}\sin(\frac{2\sqrt{6}x}{5})+c_2e^{-4x/5}\cos(\frac{2\sqrt{6}x}{5})$. So a reasonable guess may be:
$y_p = A\cos^2(x)+B\cos(x)\sin(x)+C\cos(2x)+D\sin(2x)$
Is this correct? If not, what am I doing wrong?