For b) we need only find $A$ and $B$ non-zero so that the inner product is $0$, i.e.,
$$ \langle A, B \rangle = \left\langle \begin{bmatrix} a_{11} & a_{12}\\ 0 & a_{22} \end{bmatrix},
\begin{bmatrix} b_{11} & b_{12}\\ 0 & b_{22} \end{bmatrix}\right\rangle
= a_{11}b_{11} + a_{12}b_{12} + a_{22}b_{22} = 0$$
So one easy example would be taking $a_{11}=a_{12}=b_{22}=0$. This would give a sum of zeros, even if $A$ or $B$ were non-trivial. For instance,
$$ A= \begin{bmatrix} 0 & 0\\ 0 & 1 \end{bmatrix}, \quad
B= \begin{bmatrix} 3 & -17\\ 0 & 0 \end{bmatrix},\qquad \text{then $\langle A,B\rangle = 0$}$$
As for c), you are correct in recognizing the Cauchy-Schwartz inequality. As we have an inner product, it must satisfy
$$\langle A, B\rangle^2 \leq \langle A, A\rangle \langle B,B\rangle$$
where
$$\langle A, B\rangle^2 = (a_{11}b_{11}+a_{12}b_{12}+a_{22}b_{22})^2$$
$$\langle A, A\rangle \langle B,B\rangle
= (a_{11}a_{11}+a_{12}a_{12}+a_{22}a_{22})(b_{11}b_{11}+b_{12}b_{12}+b_{22}b_{22})
= (a_{11}^2+a_{12}^2+a_{22}^2)(b_{11}^2+b_{12}^2+b_{22}^2)$$