The method of undetermined coefficients is a tricky one, but I refer you to Morris and Tenenbaum's Ordinary Differential Equations. It tells you when and how to make a good ansatz to a problem like that. There are conditions like (I don't remember off the top of my head) if the R.H.S. $f(x)$ does not appear in the complementary function, then you make.... but if it does, then you need to consider a linear combination like say $(ax+b)(\cos(x) + \sin(x))$ and all its linearly independent derivatives. However, a sure fire way to get an answer is using the operator method (Please see the textbook).
In operator notation, your problem reads:
$(D^2 + s^2)[y] = b\cos(sx).$
You have already told us that the solution to the homogeneous problem is $y = A\cos(sx ) + B\sin(sx)$. To find a particular solution to the inhomogeneous problem, we solve the related problem
$(D^2 + s^2)[y] = be^{isx}$
and take its real part. For details on this operator method, please see the textbook I mentioned above.
So there according to the reference, your solution can be found by writing
$y = \frac{1}{D^2 + s^2}[be^{isx}],$
and then use the so called shift formula given in the reference above, namely you can take the exponential out of the operator, but must replace $D$ with $(D + is)$. The operator here is $\displaystyle\frac{1}{D^2 + s^2}$ that operates on the right hand side $be^{isx}$. So
$\begin{align*} y &= e^{isx}\frac{1}{(D + is)^2 + s^2}[b] \\ &= e^{isx}\frac{1}{D^2 + 2iD}[b] \end{align*}$
I'll leave the rest for you to think about. With this method, you can treat an operator as if it were a "number", so that
$\frac{1}{1-D} = 1 + D + D^2 + \cdots.$
The only difference is that one does not talk of $|D|\lt1$ here (I know nothing about functional analysis, so I don't know what it means for a linear operator to be bounded).
Next step: Notice that the operator is operating on a constant, so when an infinite series like the above operates on $b$, it must terminate.
To answer your question why sometimes solutions to the homogeneous problem are included inside the particular solution, its because when constructing the one-sided green's function (this is the same as the method of variation of parameters here) you exclude the arbitrary constants in front of the two linearly independent solutions to the homogeneous problem.
Attention Noteventhetutorknows:
The differential equation in self-adjoint form
L[y] = -(p(x)y')'+ q(x)y = f(x)
with initial conditions y(x_0) = y'(x_0) = 0 has particular solution
$y(x) = \int_{t_0}^{t} G(x,\tau) f(\tau) d\tau$,
where $G(x, \tau) $ is the one sided green's function given by
$G(x, \tau) = \begin{cases} \frac{y_1(x)y_2(\tau) - y_2(x)y_1(\tau)}{W(\tau)}, &\text{if} \quad x \geq \tau \\ 0,& \text{if} \quad x < \tau. \end{cases}$