Edit. This addresses the question as originally asked, without the invertibility condition.
$\left(\begin{array}{ccc} 0 & 0 & 0\\ 0 & 1 & 0\\ 0 & 0 & 0 \end{array}\right)$ is of the desired form, and is not invertible.
For the invertible case: note that your set is closed under transposing matrices, and under multiplication by (nonzero) scalars. Since the inverse of $A$ can be computed with the adjugate, which is a scalar multiple of the transpose of the matrix of cofactors, your question comes down to determining whether the $(i,j)$-cofactor of a matrix of this form is equal to the $(N+1-i,N+1-j)$-cofactor.
Let $B_{ij}$ be the matrix obtained from $A$ by removing the $i$th row and the $j$th column. That is, we are trying to compare $\det(B_{ij})$ with $\det(B_{N+1-i,N+1-j})$.
I claim that the $(r,s)$ entry of $B_{ij}$ equals the $(N-r,N-s)$ entry of $B_{N+1-i,N+1-j}$.
What is the $(r,s)$ entry of $B_{ij}$?
- If $r\lt i$ and $s\lt j$, then it's $a_{ij}$;
- If $r\lt i$ and $s\geq j$, then it's $a_{i,j+1}$;
- If $r\geq i$ and $s\lt j$, then it's $a_{i+1,j}$;
- If $r\geq i$ and $s\geq j$, then it's $a_{i+1,j+1}$.
If $r\lt i$ and $s\lt j$, then the $(r,s)$ entry of $B_{ij}$ is $a_{rs}= a_{N+1-r,N+1-s}$. Since $(r,s)$ is to the left and above of $(i,j)$, then $(N+1-r,N+1-s)$ is to the right and below $(N+1-i,N+1-j)$, and hence the $(N-r,N-s)$ entry of $B_{N+1-i,N+1-j}$ is $a_{N+1-r,N+1-s}=a_{rs}$, as desired.
If $r\lt i$ and $s\geq j$, then the $(r,s)$ entry of $B_{ij}$ is $a_{r,s+1} = a_{N+1-r,N-s}$. Since $(r,s+1)$ is the left and at below of $(i,j)$, then $(N+1-r,N-s)$ is to the right and above $(N+1-i,N+1-j)$, so $a_{N+1-r,N-s}$ becomes the $(N-r,N-s)$ entry of $B_{N+1-i,N+1-j}$, as desired. A symmetric argument holds if $r\geq i$ and $s\lt j$.
If $r\geq i$, $s\geq j$, then the $(r,s)$ entry of $B_{ij}$ is $a_{r+1,s+1} = a_{N-r,N-s}$. Since $(r+1,s+1)$ is to the right and below $(i,j)$, then $(N-r,N-s)$ is to the left and above $(N+1-i,N+1-j)$, so the $(N-r,N-s)$ entry of $B_{N+1-i,N+1-j}$ is $a_{N-r,N-s}$, as desired.
So the question comes down to whether the determinant of an $N\times N$ matrix is invariant under the transformation that maps the $(i,j)$ entry to the $(N+1-i,N+1-j)$th entry. This is achieved through a series of row and column exchanges: exchange first row with last row; second row with penultimate row; third row with antepenultimate row; etc.; and then exchange first column with last column; second column with penultimate column; third column with ante-penultimate column; etc. In the end, we have performed an even number of row/column exchanges, each of which multiplies the determinant by $-1$. So the two matrices have the same determinant.
Therefore, if $A\in\mathcal{H}$, then the $(i,j)$-cofactor of $A$ equals the $(N+1-i,N+1-j)$-cofactor of $A$; the cofactor matrix of $A$ lies in $\mathcal{H}$; hence the adjugate matrix of $A$ lies in $\mathcal{H}$; hence the inverse of $A$ lies in $\mathcal{H}$. Thus, $\mathcal{H}$ forms a group.