We have\begin{align*} f'(1) &= \lim_{z\to z_0}\frac{|z|^2-1}{z-1} = \lim_{(x,y)\to(1,0)}\frac{x^2+y^2-1}{(x-1) + iy} = \lim_{(x,y)\to (1,0)}\frac{(x^2+y^2-1)((x-1)-iy)}{(x-1)^2+y^2} \end{align*} Now let $g(x,y) =\frac{(x^2+y^2-1)((x-1)-iy)}{(x-1)^2+y^2}$.
$g(x,y)\rightarrow x+1$ as $(x,y)\rightarrow (1,0)$ along x-axis.
$g(x,y)\rightarrow (y+1)-(y+1)i$ as $(x,y)\rightarrow (1,0)$ along $x=y+1$ axis.
Is that enough for me to conclude $f'(1)$ does not exist.