I need the opposite direction, I was not quite seeing it. I then decided to try and find a transformation from square to triangle in a hope to invert it.
I am trying to find an invertible transformation between the unit square $x,y \in[0,1] \times [0,1]$ and the triangle $T$ bounded by the coordinate axes and the line $y = -\frac{b}{a}x + b$, ($\xi,\eta \in T$) .
My way of thinking: If we fix an $x$ then run though $y$, we want the $\eta $ ordinate to end at $b$ and the $\xi$ ordinate to end at 0. $$\xi = ax(1-y)$$ $$\eta = by$$
After a few diagrams I am confident this is correct. I am going to try to use it to simplify an integral so I would like to be sure that this transformation is good before moving on.
EDIT
Putting this in a different way. I would like an invertible transformation between the unit square defined above and a right-angled triangle bounded by the lines $y = 0$, $x = 0$ and $$y = -\frac{b}{a}x + b$$
This triangle will have vertices $(0,0),(0,b)$ and $(a,0)$.
From the constructive comments, I might have to have the restriction that the area of the triangle and the square must be equal. It is not important that I am mapping from the unit square it just seemed the most simple to begin with.
The transformation that I found is not invertible either. If we consider the determinant of the Jacobian of the transformation we find that
$$|J| = ab(1-y)$$ which is $0$ when $y = 1$. So I have failed in that respect.

