See below for the details. Your equation is
$\left( {5{D^2} + 8D + 8} \right)y = {\cos ^2}\left( x \right)$
Make that into
$\left( {5{D^2} + 8D + 8} \right)y = \frac{{1 + \cos \left( {2x} \right)}}{2}$
and apply $D$
$\eqalign{ & \left( {5{D^2} + 8D + 8} \right)y = \frac{{1 + \cos \left( {2x} \right)}}{2} \cr & \left( {5{D^2} + 8D + 8} \right)y' = - \sin \left( {2x} \right) \cr & \left( { - 5{D^2} - 8D - 8} \right)y' = \sin \left( {2x} \right) \cr} $
Thus, we have $\eqalign{ & - 5{D^2} - 8D - 8 = \left( { - 5{D^2} - 8} \right) - 8D \cr & \varphi \left( D \right) = - 5D - 8 \cr & \xi \left( D \right) = - 8 \cr} $
and then
$y' = \frac{{3\sin 2x + 4\cos 2x}}{{100}}{\text{ }}$
and you solution is $y = \frac{{ - 3\cos 2x + 4\sin 2x}}{{200}} + C{ _1}$
I leave it to you to find $C_1$.
I know the following is not completely rigorous, but it has its justifications.
As a generalization of Tapu's solution, suppose we're given
$\phi(D)=\sum_{k=0}^m a_kD^k$
where $D=\dfrac d {dx}$. Since $D^2\sin(ax)=-a^2\sin (ax)$, we have that
$\phi(D^2)\sin(ax)=\phi(-a^2)\sin(ax)$
But we may write
$\phi (D) = \sum\limits_{k = 0}^{m'} {{a_{2k}}{D^{2k}}} + \sum\limits_{k = 0}^{m'} {{a_{2k + 1}}{D^{2k + 1}}} $
whence
$\phi (D) = \varphi \left( {{D^2}} \right) + D\xi \left( {{D^2}} \right)$
thus an equation of the form
$\phi (D)y = \sin ax$
becomes $\left[ {\varphi \left( {{D^2}} \right) + D\xi \left( {{D^2}} \right)} \right]y = \sin ax$
We apply the "conjugate" of the operator to the equation, to get $\eqalign{ & \left[ {{\varphi ^2}\left( {{D^2}} \right) - {D^2}{\xi ^2}\left( {{D^2}} \right)} \right]y = \left[ {\varphi \left( {{D^2}} \right) - D\xi \left( {{D^2}} \right)} \right]\sin ax \cr & \left[ {{\varphi ^2}\left( {{D^2}} \right) - {D^2}{\xi ^2}\left( {{D^2}} \right)} \right]y = \varphi \left( { - {a^2}} \right)\sin ax - a\xi \left( { - {a^2}} \right)\cos ax \cr & \left[ {{\varphi ^2}\left( {{D^2}} \right) - {D^2}{\xi ^2}\left( {{D^2}} \right)} \right]y = \alpha \sin ax + \beta \cos ax \cr & \Phi \left( {{D^2}} \right)y = \alpha \sin ax + \beta \cos ax \cr} $
Assuming $\Phi \left( { - {a^2}} \right) \ne 0$ we may write $\frac{{\sin ax}}{{\Phi \left( { - {a^2}} \right)}} = \frac{{\sin ax}}{{\Phi \left( {{D^2}} \right)}}$ so that
$\eqalign{ & y = \frac{{\alpha \sin ax + \beta \cos ax}}{{\Phi \left( { - {a^2}} \right)}} \cr & y = \frac{{\varphi \left( { - {a^2}} \right)\sin ax - a\xi \left( { - {a^2}} \right)\cos ax}}{{{\varphi ^2}\left( { - {a^2}} \right) + {a^2}{\xi ^2}\left( { - {a^2}} \right)}} \cr} $
For the cosine, you get $y = \frac{{\varphi \left( { - {a^2}} \right)\cos ax + a\xi \left( { - {a^2}} \right)\sin ax}}{{{\varphi ^2}\left( { - {a^2}} \right) + {a^2}{\xi ^2}\left( { - {a^2}} \right)}}$
As an example, consider
$(D^2-3D+2)y=\sin 3x$
Then $\phi(D)=D^2-3D+2=D^2+2-3D=\varphi(D^2)-D \xi(D^2)$ where $\varphi(D)=D+2$ and $\xi(D)=-3$. Thus $y = \frac{{ - 7\sin 3x - 3\left( { - 3} \right)\cos 3x}}{{49 + 9{{\left( { - 3} \right)}^2}}} = \frac{{9\cos 3x - 7\sin 3x}}{{130}}$