The advantage of not being Mathematica is that I can use simpler symbols.
Suppose that our function maps $(x,y)$ to $(s,t)$. We want to recover $x$ and $y$ given $s$ and $t$. It turns out that we don't really need Mathematica.
We know that $\frac{1}{2}x^2+x(y-1)^3=s \qquad\text{and}\qquad xy-x=t.$ The second equation looks more tractable. Trusting that $x=0$ is not going to give us a problem, we rewrite the equation as $y-1 =\frac{t}{x}.$ Substituting in the first equation, we see that perhaps we are doing something right, since we get $\frac{1}{2}x^2 + \frac{t^3}{x^2}=s.$ Solve for $x$. The rest will be straightforward, since if we have $x$ in terms of $s$ and $t$, then from $y=1 +\frac{t}{x},$ we also have $y$ in terms of $s$ and $t$.
Rewrite the equation for $x$ as $x^4 -2sx^2 +2t^3=0.$ This is a quadratic in $x^2$, readily solved for $x^2$, and hence for $x$. We get $x=\pm\sqrt{s\pm \sqrt{s^2-2t^3}}.$