Suppose $b\neq 0$. Then $f(0)=-1/b$ and if $f$ is self inverse
$$0=f(-1/b)=\frac{a/b-1}{b+1/b}=\frac{a-b}{b^2+1}.$$
It follows that $a=b$. Then
$$f(f(x))=\frac{a\frac{ax+1}{x-a}+1}{\frac{ax+1}{x-a}-a}=\frac{a^2x+a+(x-a)}{ax+1-a(x-a)}=\frac{a^2x+x}{1+a^2}=x$$
and $f$ is self-inverse. It suffices now to check the remaining conditions. We have that
$$f\left(\frac{1+a}{1-a}\right)=\frac{a\frac{1+a}{1-a}+1}{\frac{1+a}{1-a}-a}=1$$
so it must be that
$$\frac{\frac{4a+1}{4-a}}{4}=\frac{\frac{12a+1}{12-a}}{12}=1$$
But both equations cannot be satisfied simultaneously.
Now, if $b=0$, then $f(x)=a+\frac{1}{x}$ and hence
$$f(f(x))=a+\frac{1}{a+\frac{1}{x}}=a+\frac{x}{ax+1}=\frac{a^2x+x+a}{ax+1}$$
And if $f$ is self inverse
$$a^2x+x+a=ax^2+x\implies ax^2-a^2x-a=0$$
And unless $a=0$, this can be true for at most two distinct values of $x$, so the only option for $f$ to be self inverse is $f(x)=1/x$. Howevr, the condition on $f(4)$ and $f(12)$ is once again not satisfied.
Perhaps it was wrongly transcribed? The following relations do hold in this case:
$$4\cdot f(4)=12\cdot f(12)=f\left(\frac{1+b}{1-a}\right)$$