Because the inverse of an element of $G/N$ is a subset of $G$, not an element of $G$; specifically, it’s a coset of $N$. In this context $f$ doesn’t actually have an inverse in the usual sense (unless $N$ is trivial); its ‘inverse’ is actually a set-valued mapping.
You see the same thing in a more familiar setting with the function $f:\Bbb R\to[0,\to):x\mapsto x^2$: it has no inverse in the usual sense, but it does have a set-valued inverse that takes $x\in[0,\to)$ to $\{\sqrt{x},-\sqrt{x}\}$.
In most such cases, if $f:X\to Y$, I prefer to look at $f^{-1}$ as a function from $\wp(Y)$ to $\wp(X)$ that takes $A\subseteq Y$ to $f^{-1}[A]=\{x\in X:f(x)\in A\}$; the square brackets are the signal that this is how $f^{-1}$ is being used. (Similarly, for $A\subseteq X$ one can write $f[A]$ for $\{f(x):x\in A\}$, the square brackets indicating that one is thinking of $f$ as a function from $\wp(X)$ to $\wp(Y)$.)