In order to have a handy name, let us call "admissible" an invertible matrix $A$ with nonnegative entries such that $A^{-1}$ has also nonnegative entries.
There is a well known set of admissible matrices, the set of permutation matrices.
Let us take the case $n=3$, in order to keep simple notations (generalization is straightforward).
Consider the set of matrices:
$$\tag{0}\left(\begin{smallmatrix}1&0&0\\0&1&0\\0&0&1\end{smallmatrix}\right),
\left(\begin{smallmatrix}0&1&0\\1&0&0\\0&0&1\end{smallmatrix}\right),
\left(\begin{smallmatrix}0&0&1\\0&1&0\\1&0&0\end{smallmatrix}\right),
\left(\begin{smallmatrix}0&1&0\\0&0&1\\1&0&0\end{smallmatrix}\right),
\left(\begin{smallmatrix}0&0&1\\1&0&0\\0&1&0\end{smallmatrix}\right),
\left(\begin{smallmatrix}1&0&0\\0&0&1\\0&1&0\end{smallmatrix}\right)$$
Left-multiplying all these matrices by $diag(a,b,c)$ with $a>0,b>0,c>0$ gives:
$$\tag{1}\left(\begin{smallmatrix}a&0&0\\0&b&0\\0&0&c\end{smallmatrix}\right),
\left(\begin{smallmatrix}0&a&0\\b&0&0\\0&0&c\end{smallmatrix}\right),
\left(\begin{smallmatrix}0&0&a\\0&b&0\\c&0&0\end{smallmatrix}\right),
\left(\begin{smallmatrix}0&a&0\\0&0&b\\c&0&0\end{smallmatrix}\right),
\left(\begin{smallmatrix}0&0&a\\b&0&0\\0&c&0\end{smallmatrix}\right),
\left(\begin{smallmatrix}a&0&0\\0&0&b\\0&c&0\end{smallmatrix}\right)$$
whereas right-multiplying them by $diag(a,b,c)$ gives:
$$\tag{2}\left(\begin{smallmatrix}a&0&0\\0&b&0\\0&0c\end{smallmatrix}\right),
\left(\begin{smallmatrix}0&b&0\\a&0&0\\0&0&c\end{smallmatrix}\right),
\left(\begin{smallmatrix}0&0&c\\0&b&0\\a&0&0\end{smallmatrix}\right),
\left(\begin{smallmatrix}0&b&0\\0&0&c\\a&0&0\end{smallmatrix}\right),
\left(\begin{smallmatrix}0&0&c\\a&0&0\\0&b&0\end{smallmatrix}\right),
\left(\begin{smallmatrix}a&0&0\\0&0&c\\0&b&0\end{smallmatrix}\right)$$
In the former case, it is the lines that are multiplied by $a,b,c$ ; in the latter case it is the columns; in both cases, the same set of matrices is generated.
The point is that this set of matrices is exactly the set of admissible matrices for the case $n=3$, with an immediate generalization.
Here is why. It has been proven in (Does $A$ is nonsingular, nonnegative and $A^{ - 1}$ nonnegative imply that $A$ has exactly one nonzero entry in each column?) that an admissible matrix has at most a non zero entry on each of its lines (and, by transposition, on each of its columns).
We can go further. In fact, each line and each column of an admissible matrix must have exactly one nonzero element, otherwise it would have a line or a column of zeros, thus wouldn't be invertible.
This necessary condition is in fact also sufficient, because a matrix that can be written:
$$\tag{3}\begin{cases}diag(a_1,a_2,\cdots a_n) \times P \ \ \text{with all} \ \ a_k>0 \ \text{or}\\P \times diag(a_1,a_2,\cdots a_n)\end{cases}$$
with $P$ a permutation matrix, is invertible, with resp. inverse:
$$\tag{4}\begin{cases}P^{-1} \times diag(a_1^{-1},a_2^{-1},\cdots a_n^{-1}) \ \ \text{or}\\
diag(a_1^{-1},a_2^{-1},\cdots a_n^{-1}) \times P^{-1}\end{cases}$$
which has the same structure.
Thus (3) (in either of its two equivalent forms), represents the general structure of all admissible matrices.
Remark : 2 1/2 years later, I discovered that a similar question had been already asked with a (rather concise) answer in (Find all non-singular $3 \times 3$ matrices, such as $A$ and $A^{-1}$ elements are non-negative).