Exchanging two rows of $A$ amounts to multiplying $A$ by an elementary matrix on the left, $B=EA$; so the inverse of $B$ is $A^{-1}E^{-1}$, and the inverse of the elementary matrix corresponding to exchanging two rows is itself. Multiplying on the right by $E$ corresponds to permuting two columns of $A^{-1}$. Thus, the inverse of the matrix we get form $A$ by exchanging two rows is the inverse of $A$ with two columns exchanged. Exchanging two columns of a matrix $M$ does not change the value of $\lVert M\rVert_{\infty}$; thus, $\lVert (EA)^{-1}\rVert_{\infty} = \lVert A^{-1}\rVert_{\infty}$.
Since any permutation of the rows of $A$ can be obtained as a sequence of row exchanges, the conclusion follows.