The advantage of not being Mathematica is that I can use simpler symbols.
Suppose that our function maps $(x,y)$ to $(s,t)$. We want to recover $x$ and $y$ given $s$ and $t$. It turns out that we don't really need Mathematica.
We know that
$$\frac{1}{2}x^2+x(y-1)^3=s \qquad\text{and}\qquad xy-x=t.$$
The second equation looks more tractable. Trusting that $x=0$ is not going to give us a problem, we rewrite the equation as
$$y-1 =\frac{t}{x}.$$
Substituting in the first equation, we see that perhaps we are doing something right, since we get
$$\frac{1}{2}x^2 + \frac{t^3}{x^2}=s.$$
Solve for $x$. The rest will be straightforward, since if we have $x$ in terms of $s$ and $t$, then from
$$y=1 +\frac{t}{x},$$
we also have $y$ in terms of $s$ and $t$.
Rewrite the equation for $x$ as
$$x^4 -2sx^2 +2t^3=0.$$
This is a quadratic in $x^2$, readily solved for $x^2$, and hence for $x$. We get
$$x=\pm\sqrt{s\pm \sqrt{s^2-2t^3}}.$$