The first problem with your approach is that the problem is not in a compact domain, so we should not expect there to be a discrete countable set of eigenvalues. Therefore the solution has to be an integral instead of a sum. It is also easier to use exponentials than trig. functions for this case.
Separating variables, you have the correct equations, but let's write the separation constant as $\lambda=k^2$, and then we have solutions
$$ X_k(x) = A(k) e^{ikx} + B(k) e^{-ikx}, \quad Y_k(y) = C(k) e^{ky} + D(k) e^{-ky}, $$
where the dependence on the eigenvalue has been made explicit. Now, to keep $u$ bounded as $y \to \infty$, we need to also take the $Y_k$ bounded as $y \to \infty$: i.e., $e^{-\lvert k \rvert y}$; we can discard the coefficient since it plays no further rôle that cannot be taken up by those in $X_k$.
The total solution is then
$$ u(x,y) = \int_{-\infty}^{\infty} e^{ikx-\lvert k \rvert y} A(k) \, dk. $$
Of course, now we have to find $A(k)$ using the boundary conditions $u(x,0)=f(x)$, or
$$ f(x) = \int_{-\infty}^{\infty} e^{ikx} A(k) \, dk. $$
This is a Fourier transform, so we proceed in the same way as Fourier series, by multiplying by $e^{-imx}$ and integrating with respect to $x$ (we assume that this integral makes sense, or this approach doesn't work; the final answer can still be obtained this way, but by a more complicated limiting process):
$$ \int_{-\infty}^{\infty} e^{-imx} f(x)\, dx = \int_{-\infty}^{\infty} e^{-imx} \int_{-\infty}^{\infty} e^{ikx} A(k) \, dk \, dx. $$
Changing the order of integration on the right,
$$ \int_{-\infty}^{\infty} e^{-imx} f(x) \, dx = \int_{-\infty}^{\infty} \left( \int_{-\infty}^{\infty} e^{ix(k-m)} \, dx \right) A(k) \, dk. $$
At this point we use the orthogonality relation for the Fourier transform, which is
$$ \int_{-\infty}^{\infty} e^{ix(k-m)} \, dx = 2\pi \delta(k-m), $$
which may be shown in a number of different ways, such as using a cutoff to make the integral finite and taking the limit. Hence we find, using the property $ \int_{-\infty}^{\infty} \delta(k-m) A(k) \, dk = A(m) $,
$$ A(m) = \frac{1}{2\pi} \int_{-\infty}^{\infty} e^{-imx} f(x) \, dx, $$
and so we have
$$ u(x,y) = \frac{1}{2\pi} \int_{-\infty}^{\infty} e^{ikx-\lvert k \rvert y} \left( \int_{-\infty}^{\infty} e^{-ikx'} f(x') \, dx' \right) dk. $$
In this case, we can simplify this further: change the order of integration again to get
$$ u(x,y) = \frac{1}{2\pi} \int_{-\infty}^{\infty} f(x') \left( \int_{-\infty}^{\infty} e^{ik(x-x')-\lvert k \rvert y} \, dk \right) dx'. $$
The inside integral is straightforward, and can be done by splitting it at zero: we find that
$$ \int_{-\infty}^{\infty} e^{ik(x-x')-\lvert k \rvert y} \, dk = \frac{2y}{(x-x')^2+y^2}, $$
so the neatest answer is
$$ u(x,y) = \frac{1}{\pi} \int_{-\infty}^{\infty} \frac{y}{(x-x')^2+y^2} f(x') dx'. $$
Another way to derive this involves calculating the Green's function for Laplace's equation on the upper half-space, and using Green's third identity to get this integral more directly; you haven't given enough context to determine which is the intended method.