I dont really understand what is going on in this question but I gave it a go anyway. I calculated $A^tB$ and got it's trace by adding the numbers on the diagonal. The trace was 0.
I then used the defintion of the dot product as $\langle a,b\rangle= ||a||||b||cos\theta$ I figured that as $\langle A,B\rangle = trace(A^tB)=0$, then $\theta$ must equal $\pi/2$.
This shows up as correct. However I dont know how to get the norm of A & B. I didnt even know a matrix had such a thing as a norm/length.
I also dont understand the wording of the question where they are saying $\langle A,B\rangle = trace(A^tB)$, how can you dot one matrix with another, I never heard of such a thing, we never covered it in class anyway.