The marginal density of $Y$ is
$$\int_{-y}^y c(y^2-x^2) e^{-y} \mathop{dx}=ce^{-y}(2y^3 - (2/3)y^3 ) = \frac{4c}{3} y^3 e^{-y}.$$
Integrating this over $y$ gives
$$1=\int_0^\infty \frac{4c}{3} y^3 e^{-y} \mathop{dy} = \frac{4c}{3} \cdot 3! = 8c \implies c=1/8.$$
So, the marginal density of $Y$ is $\frac{1}{6} y^3 e^{-y}$ as you obtained.
The marginal density of $X$ is
$$\int_{|x|}^\infty c(y^2-x^2) e^{-y} \mathop{dy}
.$$
The tricky part is the limits of integration, which comes from the condition $|x| \le y$.
First, integration by parts twice gives
\begin{align}
\int_{|x|}^\infty y^2 e^{-y} \mathop{dy}
&=[-y^2 e^{-y}]_{y=|x|}^\infty + 2\int_{|x|}^\infty ye^{-y} \mathop{dy}\\
&= x^2 e^{-|x|} + 2[-ye^{-y}]_{y=|x|}^\infty + 2\int_{|x|}^\infty e^{-y}\mathop{dy}\\
&= x^2 e^{-|x|} + 2|x|e^{-|x|} + 2e^{-|x|}\\
&= e^{-|x|}(|x|^2+2|x|+2).
\end{align}
So, the marginal density of $X$ is
$$\int_{|x|}^\infty c(y^2-x^2) e^{-y} \mathop{dy}
= c(e^{-|x|}(|x|^2+2|x|+2) - x^2 e^{-|x|}) = \frac{1}{4} e^{-|x|}(|x|+1).$$
Edit (explanation for why the lower limit of integration is $|x|$):
In general, the marginal density of $X$ is
$$f_X(x) = \int_{-\infty}^\infty f(x,y) \mathop{dy}.$$
For a specific $x$, note that by definition $f(x,y)$ is zero if $|x|>y$; otherwise $f(x,y) = c(y^2-x^2)e^{-y}$. So
$$f_X(x) = \int_{-\infty}^\infty f(x,y) \mathop{dy} = \int_{-\infty}^{|x|} f(x,y) \mathop{dy} + \int_{|x|}^\infty f(x,y) \mathop{dy}
= 0 + \int_{|x|}^\infty c(y^2-x^2)e^{-y} \mathop{dy}.$$