0
$\begingroup$

I have a scalar-valued function $y$ and a vector-valued function $\textbf{x}$:

\begin{equation} y = \underset{(1 \times D_h)}{\textbf{w}^T}\underset{(D_h \times 1)}{\textbf{x}} \end{equation} \begin{equation} \underset{(D_h \times 1)}{\textbf{x}} = \underset{(D_h \times D_z)}{\textbf{B}}\underset{(D_z \times 1)}{\textbf{z}} \end{equation}

My problem is to compute $\nabla_{\textbf{B}}y$. These are the steps I understand so far:

\begin{equation} \nabla_{\textbf{B}}y = \frac{\partial{y}}{\partial\textbf{x}}\frac{\partial\textbf{x}}{\partial\textbf{B}} = \textbf{w}^T\frac{\partial\textbf{x}}{\partial\textbf{B}} \end{equation}

$\frac{\partial\textbf{x}}{\partial\textbf{B}}$ seems to be 3-dimensional.

I took a wild leap at finishing this derivation and got:

\begin{equation} = \underset{(D_z \times 1)}{\textbf{z}}\underset{(1\times D_h)}{\textbf{w}^T} \end{equation}

I computed some two-point approximations of the gradient and this solution seems to be working.

My question is, why does this last step work? What identity is this? It seems very strange to me that the $\textbf{z}$ would show up on the left. I'm probably skipping some steps..

1 Answers 1

0

When working with matrices, it's better to use differentials than the chain rule. The intermediate quantities required by the chain rule are often 3rd and 4th order tensors, whereas differentials are ordinary matrices.

The chain rule is also fussy about how the intermediates are multiplied. Both the in type of products (single-dot vs double-dot), and in the ordering (and possibly transpositioning) of terms in the product.

The differential is much less fussy.

For your problem $$\eqalign{ y &= w:x \cr &= w:Bz \cr &= wz^T:B \cr \cr dy &= wz^T:dB \cr \cr \frac{\partial y}{\partial B} &= wz^T \cr \cr }$$ where colon represents the inner/Frobenius product, which is really just an infix notation for the matrix trace, i.e. $$A:B = {\rm tr}(A^TB)$$ Note that each term in a Frobenius product has the same shape, a feature that it shares with the elementwise (Hadamard) product. In fact, the Frobenius product is just the sum over all the elements in a Hadamard product.