It is probably good to also keep track of the displacement angle. Let's say we call it $α_n$ and it starts with $α_1 = 0$. Then the orthogonal line added is rotated from the up
direction by $α_{n-1}$. So if $(x_0, y_0) = (0, 0)$ and $(x_1, y_1) = (1, 0)$, the coordinates change as
$ x_n = x_{n-1} - n\cdot \sin α_{n-1} \\ y_n = y_{n-1} + n\cdot \cos α_{n-1}$
Then you can calculate the new angle as
$ α_n = α_{n-1} + \arctan\left(\frac n {\sqrt{x_{n-1}^2+y_{n-1}^2}}\right)$
Of course this is the most simplistic approach and you could probably avoid using $\arctan$, $\sin$ and $\cos$ with a little more calculation.
As you are always adding the length $n$, the radius will be
$r_n := \sqrt{x_n^2+y_n^2} = \sqrt{\sum_{k=1}^n{k^2}} = \sqrt{\frac{n}{6}(n+1)(2n+1)}$
(Compare this formula.)
So then you could write either of
$ α_n - α_{n-1} = \arccos\left(\frac{r_{n-1}}{r_n}\right) = \arccos\sqrt{\frac{(n-1)(2n-1)}{(n+1)(2n+1)}}\\ = \arctan\left(\frac{n}{r_{n-1}}\right) = \arctan \sqrt{\frac{6n}{(n-1)(2n-1)}}\\ = \arcsin\left(\frac{n}{r_n}\right) = \arcsin\sqrt{\frac{6n}{(n+1)(2n+1)}}$
This way the calculation can be improved by defining
$ s_n := \sin α_n, \quad c_n := \cos α_n $
Then you will get
$s_{n+1} = \sin(α_n + (α_{n+1} - α_n)) = \sin α_n \cos (α_{n+1} - α_n) + \cos α_n \sin (α_{n+1} - α_n) = \frac{s_nr_n}{r_{n+1}}+\frac{c_n(n+1)}{r_{n+1}}\\ c_{n+1} = \cos(α_n + (α_{n+1} - α_n)) = \cos(α_n)\cos(α_{n+1}-α_n)-\sin{α_n}\sin(α_{n+1}-α_n) = \frac{c_nr_n}{r_{n+1}} - \frac{s_n(n+1)}{r_{n+1}}$
So you can put
$x_n = x_{n-1} - ns_{n-1}\\y_n = y_{n-1} + nc_{n-1}$
Wikipedia gives a formula for the Spiral of Theodorus, like meantioned by pritam. Probably either of the there referenced solutions can be adapted to this problem. The complex function equation would here take the form
$f(x+1) = \left(1+i\frac{x+1}{r_x}\right)f(x) = \left(1+i\sqrt{\frac{6(x+1)}{x(2x+1)}}\right)f(x)$