You are close. If the determinant of the matrix is zero, then the matrix does not have full rank which means it is not invertible. If the determinant is non-zero, then $A^{-1}$ exists and some $x = A^{-1}y$ is the unique solution. Thus it admits a unique solution when $b \neq 2$, assuming you computed the determinant correctly.
On the other hand, consider the augmented matrix when $b=2$,
$$ \left( \begin{array}{cccc}
1 & 2 & -3 & | & a \\
4 & 2 & 0 & | & a+4 \\
-1 & -2 & 3 & | & a-1
\end{array} \right) $$
If you did everything correctly, when you row-reduce this matrix using Gauss-Jordan elimination, you will end up with a row of all zeroes. The corresponding vector on the right-hand side of $|$ must have a zero in that component in order for the system to be consistent. Choosing $a$ so that this is the case will give you a system that has infinitely many solutions. You have too many variables (3) and too few equations (2). On the other hand, every other value $a$ will mean the system has no solutions, since you will have a requirement that one equation be equal to two different constants, which can never hold.