Since $2j$ depends quadratically on $n$, you basically need the square root of $2j$ to get $n$. To get the limits exactly right, I'd try $n=\lfloor \sqrt{2j+a}+b\rfloor$ and then determine $a$ and $b$ so that everything comes out right. Once you have $n$, it's easy to get $m$.
Update: You can systematically determine $a$ and $b$ by looking at what happens at the boundaries. You need
$$\lfloor \sqrt{n(n+2)-n+a}+b\rfloor=n$$
but
$$\lfloor \sqrt{(n-1)((n-1)+2)+(n-1)+a}+b\rfloor=n-1\;.$$
The difference between the two arguments of the square root is $2$. To avoid rounding problems you can choose $a$ and $b$ such that the argument of the floor function is the integer $n$ when the argument of the square root is in the middle between the two:
$$\sqrt{n^2+n-1 + a} + b = n\;,$$
$$n^2+n-1+a=(n-b)^2\;,$$
$$n-1+a=-2bn+b^2\;.$$
To make this true for all $n$, you can equate the constant terms and the coefficients of $n$:
$$1=-2b\;,$$ $$-1+a=b^2\;.$$
That yields $a=\frac{5}{4}$ and $b=-\frac{1}{2}$, so the formula for $n$ is
$$n=\lfloor \sqrt{2j+\frac{5}{4}}-\frac{1}{2}\rfloor\;.$$