Gerry Myerson gave a recipe for finding a symmetric solution, also providing a solution in a comment. I wanted to implement that while preserving the structure of the Don Giles matrix for order 2 (Fano plane). That is, I wanted the first four rows and first four columns to be the analogous pattern to the 7 by 7. Gerry's method amounts to this: take all 13 lines in $\mathbb F_3^3,$ where $\mathbb F_3$ is the field with three elements. The 13 lines can be parametrized by triples of $1,0,-1,$ where the first nonzero entry is $+1.$ There is a $1$ in the matrix for a pair of vectors out of the 13 if their dot product is $0 \pmod 3.$ I decided to label the 13 points using capital letters. Notice that $A \cdot A \equiv G \cdot G \equiv J \cdot J \equiv M \cdot M \equiv 0 \pmod 3.$
LABELS: $ \begin{array}{rrrr} 0 & 0 & 1 & K \\ 0 & 1 & -1 & B \\ 0 & 1 & 0 & H \\ 0 & 1 & 1 & F \\ 1 & -1 & -1 & G \\ 1 & -1 & 0 & D \\ 1 & -1 & 1 & J \\ 1 & 0 & -1 & C \\ 1 & 0 & 0 & E \\ 1 & 0 & 1 & I \\ 1 & 1 & -1 & M \\ 1 & 1 & 0 & L \\ 1 & 1 & 1 & A \end{array} $
The order of both the rows and the columns of the matrix is alphabetical, $ABCDEFGHIJKLM.$ As I said, there is a $1$ in row $X,$ column $Y$ (and the reverse) when $X \cdot Y \equiv 0 \pmod 3.$
MATRIX: $ P \; = \; \left( \begin{array}{cccc|ccc|ccc|ccc} 1 & 1 & 1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 1 & 0 & 0 & 0 & 1 & 1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 \\ 1 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 & 1 & 0 & 0 & 0 \\ 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 & 1 \\ \hline 0 & 1 & 0 & 0 & 0 & 1 & 0 & 1 & 0 & 0 & 1 & 0 & 0 \\ 0 & 1 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 1 \\ 0 & 1 & 0 & 0 & 0 & 0 & 1 & 0 & 1 & 0 & 0 & 1 & 0 \\ \hline 0 & 0 & 1 & 0 & 1 & 0 & 0 & 0 & 1 & 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 & 0 & 0 & 1 & 1 & 0 & 0 & 0 & 0 & 1 \\ 0 & 0 & 1 & 0 & 0 & 1 & 0 & 0 & 0 & 1 & 0 & 1 & 0 \\ \hline 0 & 0 & 0 & 1 & 1 & 0 & 0 & 1 & 0 & 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 & 0 & 0 & 1 & 0 & 0 & 1 & 1 & 0 & 0 \\ 0 & 0 & 0 & 1 & 0 & 1 & 0 & 0 & 1 & 0 & 0 & 0 & 1 \end{array} \right). $
The result is that $P$ is symmetric and $P^2$ is the matrix with all 4's on the main diagonal and 1's everywhere else.