It is the dot product of the two matrices $\mathbf{A}$ and $\mathbf{B}$.
For $\mathbf{A} = (a_{i,j})_{1 \leq i,j \leq n}$ and $\mathbf{B} = (b_{i,j})_{1 \leq i,j \leq n}$ two real matrices, the dot product of $\mathbf{A}$ and $\mathbf{B}$ is :
$$ \left\langle \mathbf{A},\mathbf{B} \right\rangle = \mathrm{tr}\big( \mathbf{A}^{\top} \mathbf{B} \big) = \sum_{i=1}^{n} \sum_{l=1}^{n} a_{li}b_{li}. $$
It is related to the Euclidean inner product on $\mathbb{R}^{n^2}$ as follows :
$$ \left\langle \mathbf{A}, \mathbf{B} \right\rangle = \mathrm{vec}(\mathbf{A})^{\top} \mathrm{vec}(\mathbf{B}) $$
where $\mathrm{vec}$ denotes the vectorization operator.