In my understanding, a polar coordinate system is usually used to describe points in the plane. You appear to be looking for something describing points on the (unit) sphere. So I guess you're actually referring to a geographic coordinate system.
The geographic coordinate system has a mostly rectangular structure: you have latitudes from $-\frac\pi2$ to $\frac\pi2$ and for every latitude you have longitudes from $-\pi$ to $\pi$. You could get your square into this shape using
$\begin{pmatrix}a\\b\end{pmatrix}\mapsto \begin{pmatrix}\frac{a}{1-\lvert b\rvert}\pi\\ b\frac\pi2\end{pmatrix}$
There is a slight problem, however: At the poles, the longitude is undefined. So there are many different coordinates representing the same point. In the above formula, this undefinedness manifests itself as a division by zero. You could make a case distinction and define that in those cases where you would have to divide by zero, you simply assume the latitude to be zero, as it doesn't really matter.
You also have an overlap at the $\pm\pi$ meridian, where the two edges of your input triangle represent the same points on the sphere. If this matters, you'd have to remove two edges from your input square, except for the corners corresponding to the poles.
On the whole, I doubt that any of the above will be particularly useful for practical applications, as the transformation defined in this way probably won't fulfill any of the requirements you'd expect from a projection. Except perhaps being simple to calculate.