You write $N(1,1)=2$, but also $N(1,1)=4$, so there is something wrong with your calculations. I'll just ignore the first two lines of the table.
Note we have
$i+|j|=2$ for $3\le N(i,j)\le4$
$i+|j|=3$ for $5\le N(i,j)\le8$
$i+|j|=4$ for $9\le N(i,j)\le14$
$i+|j|=5$ for $15\le N(i,j)\le22$
and so on. I'll rewrite these as
$i+|j|=2$ for $1\le 4N(i,j)-11\le5$
$i+|j|=3$ for $9\le 4N(i,j)-11\le21$
$i+|j|=4$ for $25\le 4N(i,j)-11\le45$
$i+|j|=5$ for $49\le 4N(i,j)-11\le77$
and then as
$i+|j|=2$ for $1\le \sqrt{4N(i,j)-11}\lt3$
$i+|j|=3$ for $3\le \sqrt{4N(i,j)-11}\lt5$
$i+|j|=4$ for $5\le \sqrt{4N(i,j)-11}\lt7$
$i+|j|=5$ for $7\le \sqrt{4N(i,j)-11}\lt9$
and finally as
$i+|j|=2$ for $2\le (\sqrt{4N(i,j)-11}+3)/2\lt3$
$i+|j|=3$ for $3\le (\sqrt{4N(i,j)-11}+3)/2\lt4$
$i+|j|=4$ for $4\le (\sqrt{4N(i,j)-11}+3)/2\lt5$
$i+|j|=5$ for $5\le (\sqrt{4N(i,j)-11}+3)/2\lt6$
So we see that $i+|j|=\left[{\sqrt{4N(i,j)-11}+3\over2}\right]$ where I use $[x]$ to indicate the greatest integer not exceeding $x$. So, for example, if $N(i,j)=12345$ then we compute $4N(i,j)-11=49369$, $\sqrt{49369}=222.19\dots$, add $3$ to get $225.19\dots$, divide by $2$ to get $112.59\dots$, and round down to get $i+|j|=112$.
Now let's do some computations on the last 10 rows of your table:
$\matrix{i&j&N&4N-11&A&B&C\cr5&-1&23&81&0&0&5\cr5&1&24&85&4&0&5\cr4&-2&25&89&8&1&4\cr\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots\cr1&5&32&117&36&4&1\cr}$ where I am using $Q=\sqrt{4N-11},\quad z=2\left[{Q+1\over2}\right]-1=9,\quad A=4N-11-z^2,\quad B=\left[{A\over8}\right],\quad C=\left[{Q+1\over2}\right]-B$ The first and last columns are identical, so we have a formula for $i$. If you unravel everything, it comes down to $i={[M]^2+[M]\over2}-\left[{N-3\over2}\right]$ where $M=(\sqrt{4N-11}+1)/2$ (although it's probably a good idea to check over my algebra). For $N=12345$, this works out to $i=45$.
Now that we have formulas for $i$ and for $i+|j|$, we subtract to find $|j|$. Then it's just a question of deciding whether $j$ is positive or negative, and it's positive when $N$ is even, negative when $N$ is odd, so the formula is $j=(-1)^N|j|$. In our example, this gives $j=-67$.