It's often good to start with finding the solution to the homogenous solution y_h'' + y_h' =0 which, as you know, is $y_h = c_1 + c_2e^{-x}$. Thus you don't need to have a $e^{-x}$-term when you look for the particular solution, since this is already included in the homogenous (if you include it in your particular solution, it will cancel out).
Second, if you are prone to small misstakes, you can take terms in your particular solution one by one, since differentiation is linear. Start with $Axe^{-x}$: $\frac{d^2}{dx^2} Axe^{-x} + \frac{d}{dx} Axe^{-x} = -Ae^{-x}$ then $Bx^2e^{-x}$: $\frac{d^2}{dx^2} Bx^2e^{-x} + \frac{d}{dx} Bx^2e^{-x} = 2Be^{-x} - 2Bxe^{-x}$
You can see that if you have $y_p = Bx^2e^{-x} + Axe^{x}$ you will get y_p'' + y_p' = (2B - A)e^{-x} - 2Bxe^{-x} thus $B = -\frac{1}{2}$ and $2B - A = -1 -A=0$, which gives $A = -1$, which gives you your answer.
A rule which is convenient when making these kinds of calculations is $\frac{d^n}{dx^n} \left( f(x) e^{ax} \right) = e^{ax} \left( \frac{d}{dx} + a \right)^n f(x)$ where $\left(\frac{d}{dx}\right)^n$ is interpreted as $\frac{d^n}{dx^n}$.
For example, using this on $Axe^{-x}$ (writing $D$ for $\frac{d}{dx}$): $\begin{align} \frac{d^2}{dx^2} Axe^{-x} + \frac{d}{dx} Axe^{-x} &= Ae^{-x}( (D-1)^2 x + (D-1) x )= Ae^{-x} (D-1)((D-1)x + x)) = \\ &= Ae^{-x} (D-1)(1) = -Ae^{-x} \end{align}$ You can quickly see that the term $Axe^{-x}$ is not enough.