If I have solution basis , how can I find out if there is a real constant coefficient homogeneous linear ODE to that solution or not?
For example, I have the solution basis $e^x, xe^x, x^2e^x,\cos(x), \sin(x)$.
I know this site is not for homework, but I can't figure it out from the book I have.
For each one, you have to find a constant-coefficient differential operator that eliminates it, and then you can stack them together (i.e. compose them), because if the coefficients are constant, the operators commute. For example, $\cos{x}$ and $\sin{x}$ are annihilated by $(d/dx)^2+1$. The other three can be eliminated by the same operator, $(d/dx-1)^3$: $$ \left(\frac{d}{dx}-1\right) x^2e^x = 2xe^x + x^2e^x-x^2e^x = 2xe^x, \\ \left(\frac{d}{dx}-1\right) xe^x = e^x + xe^x-xe^x = e^x \\ \left(\frac{d}{dx}-1\right) e^x = e^x-e^x = 0 $$ Hence all five are eliminated by $ \left(\frac{d}{dx}-1\right)^3 \left( \frac{d^2}{dx^2}+1 \right) $, and you can find the actual form of the differential equation by applying this to a function $y$ and expanding out.
Each homogenous ODE with constant coefficients can be solved by finding the roots of the characteristic polynomial, e.g. $$y'' - 3y' + 2y = 0$$ has the characteristic polynomial $r^2-3r+2=(r-1)(r-2)$ and is solved by $y = c_1e^t+c_2e^{2t}$. If you had the functions $e^t$ and $e^{2t}$ as the basis functions for the solutions, then you would know that the characteristic polynomial had roots $1$ and $2$. Thus we can build an ODE which has the given solutions if we have the basis functions (as long as they are consistent).
So if we have $$e^{st}, \\ te^{st},\\ \cdots, \\ t^{k-1}e^{st}$$ as basis functions, we would know that our characteristic polynomial has a root of $s$ with multiplicity $k$, i.e., $p(r) =(s-r)^kq(r)$ where $p(r)$ is our characteristic polynomial and $q(r)$ is another polynomial. If we have $$e^{\alpha t}\cos(\beta t), e^{\alpha t}\sin(\beta t), \\ e^{\alpha t}t\cos(\beta t), e^{\alpha t}t\sin(\beta t), \\ \cdots, \\ e^{\alpha t} t^{k-1}\cos(\beta t), e^{\alpha t}t^{k-1}\sin(\beta t)$$ as basis functions, the characteristic polynomial would have a root $\alpha + \beta i$ and $\alpha - \beta i$ of multiplicity $k$.
To take your example, $e^x,xe^x,x^2e^x$ correspond to a root of $1$ with multiplicity $3$ and $\cos(x), \sin(x)$ correspond to roots $i$ and $-i$. A candidate characteristic polynomial would be $$(r-1)^3(r-i)(r+i) = r^5-3r^4+4r^3-4r^2+3r -1.$$ And so our ODE would be $$y^{(5)}-3y^{(4)}+4y^{(3)}-4y''+3y' -y=0.$$
Note that this would not work for any choice of basis functions. For instance, there isn't a constant coefficient homogenous ODE with $te^t$ as its only basis function (we need $e^t$ as well). We also can't have just $\cos(t)$ as a basis function (we need $\sin(t)$ as well).