Since $2j$ depends quadratically on $n$, you basically need the square root of $2j$ to get $n$. To get the limits exactly right, I'd try $n=\lfloor \sqrt{2j+a}+b\rfloor$ and then determine $a$ and $b$ so that everything comes out right. Once you have $n$, it's easy to get $m$.
Update: You can systematically determine $a$ and $b$ by looking at what happens at the boundaries. You need
$\lfloor \sqrt{n(n+2)-n+a}+b\rfloor=n$
but
$\lfloor \sqrt{(n-1)((n-1)+2)+(n-1)+a}+b\rfloor=n-1\;.$
The difference between the two arguments of the square root is $2$. To avoid rounding problems you can choose $a$ and $b$ such that the argument of the floor function is the integer $n$ when the argument of the square root is in the middle between the two:
$\sqrt{n^2+n-1 + a} + b = n\;,$
$n^2+n-1+a=(n-b)^2\;,$
$n-1+a=-2bn+b^2\;.$
To make this true for all $n$, you can equate the constant terms and the coefficients of $n$:
$1=-2b\;,$ $-1+a=b^2\;.$
That yields $a=\frac{5}{4}$ and $b=-\frac{1}{2}$, so the formula for $n$ is
$n=\lfloor \sqrt{2j+\frac{5}{4}}-\frac{1}{2}\rfloor\;.$