Assuming $A$ is non-singular and further assuming that $A^{-1}$ is also the adjacency matrix of some graph, we can show that each vertex of the graph must have precisely one in-degree and one out-degree. Let $G(A)$ denote the graph of $A$ and similarly for $G(A^{-1})$.
Each vertex of $G(A)$ must have at least one in-degree and out-degree. Otherwise the the corresponding column/row would be all zero and the matrix would be singular.
We interpret the entries of the adjacency matrix as the number of walks. $AA^{-1}$ would be interpreted as concatenated walks with the first edge taken in $G(A)$ and the second in the graph of $G(A^{-1})$. In particular this means that for every edge in the graph of $A$, there must be an edge of the reverse direction in $G(A^{-1})$. If this was not the case, then suppose $u$ is connected to $v$ in $G(A)$ and $v$ is not connected to $u$ in $G(A^{-1})$. Then there exists an edge from $v$ to some $w\neq v$ in $G(A^{-1})$. But then the entry $(u,\ w)$ will be non-zero in $AA^{-1}$ contradicting the fact that it's the identity. By symmetry, this means that the edges of $G(A)$ and $G(A^{-1})$ are identical, except for a reversal in direction.
If some $u$ in $G(A)$ had out-degree greater than two, then there would exist $2$ or more walks back to $u$ in $AA^{-1}$. A symmetrical argument shows the result for in-degrees. It follows that the only graphs which have such adjacency matrices are directed cycle graphs (or unions of such).