I will give here a matrix explanation.
Let us consider for example the case $4 \times 4$, which is in fact illustrative of the general case $n \times n$.
You can write $A$ as the sum:
$$\pmatrix{1&1&1&1\\2&2&2&2\\3&3&3&3\\4&4&4&4}+\pmatrix{1&2&3&4\\1&2&3&4\\1&2&3&4\\1&2&3&4}=\pmatrix{1\\2\\3\\4}\pmatrix{1&1&1&1}+\pmatrix{1\\1\\1\\1}\pmatrix{1&2&3&4}$$
which the sum of 2 rank-1 matrices, thus has rank $\leq 2$, and in fact effectively two.
Thus, due to the rank-nullity theorem, $\dim \ker A= 4 -2>0$ (and more generally $n-2$). Thus $\det(A)=0$.
It explains the "barrier" beyond $n=2$.