1
$\begingroup$

Say you have

$x^T\Sigma{x} - y^T\Sigma{y}$

where $\Sigma$ is a symmetric $n\times{n}$ covariance matrix and $x$ and $y$ are $n\times{1}$ vectors. What are the steps to simplify this to

$(x + y)^T\Sigma(x - y)$

Say I didn't know that the simplified version is $(x + y)^T\Sigma(x - y)$. What is the brainwave required to simplify?

2 Answers 2

2

Equality is transitive and then reverse the argument: $$ \begin{align} (x + y)^T\Sigma (x - y) &= x^T\Sigma (x - y) + y^T\Sigma (x - y)\\ &= x^T\Sigma x - x^T\Sigma y + y^T\Sigma x - y^T \Sigma y\\ &= x^T\Sigma x - ((x^T\Sigma y)^T)^T + y^T\Sigma x - y^T \Sigma y\\ &= x^T\Sigma x - (y^T\Sigma^T x)^T + y^T\Sigma x - y^T \Sigma y \\ &= x^T\Sigma x - (y^T\Sigma x)^T + y^T\Sigma x - y^T \Sigma y\\ &= x^T\Sigma x - y^T\Sigma x + y^T\Sigma x - y^T \Sigma y\\ &= x^T\Sigma x - y^T \Sigma y. \end{align} $$ So why does this work? We are using (in order):

  1. (Left) distributivity of + and matrix multiplication.
  2. (Right) distributivity of + and matrix multiplication.
  3. Transpose of the transpose is the original matrix.
  4. Expand transpose: reverse order and add transposes.
  5. $\Sigma$ is symmetric so does not change by taking transpose.
  6. Actually, we are taking the transpose of a number! So it does nothing.
  7. +/- cancel.

Note that another way is to show that a symmetric $\Sigma$ corresponds to a symmetric bilinear form $B$ and then:

$$ \begin{align} (x + y)^T \Sigma (x - y) &= B(x + y, x - y)\\ &= B(x, x - y) + B(y, x - y)\\ &= B(x, x) - B(x, y) + B(y, x) - B(y, x)\\ &= B(x, x) - B(y, y)\\ &= x^T \Sigma x - y^T \Sigma y. \end{align} $$ This is a more general result: it holds even if $B$ did not come from a symmetric matrix, for example on the vector space of continuous real-valued functions on $[0, 1]$ there is a symmetric bilinear form $B$ given by $$ B(f, g) = \int_0^1 f(t)g(t)dt. $$ There cannot be a matrix in this case, because the space has infinite dimension.

Edit due to change in question (understanding).

How do you get this brainwave? If you don't know where you are going in this case? Pattern matching and heuristics.

This questions evokes feelings of the identity $$ x^2 - y^2 = x x - y y = (x + y)(x - y) $$ in me, in fact, this comes from the result that inspired this question on the matrix $(1)$ in vector space $\mathbb{R}^1$ and associated the symmetric bilinear form given by $B(x, y) = xy$.

The following are often useful:

  • Adding 0, written as $c−c$.
  • Multiplying by 1, written as $\tfrac{c}{c}$.
  • Some other operation, like taking the double transpose, that does nothing but can be expanded.

All can lead to simplified expressions, but you need to know if and how to use them. For example, the multiplying by 1 is used for rationalizing denominators. This is something you get good at with experience.

  • 0
    Thanks, I see how to get from the result back to what I started with, but the real question I'm asking is how to get from what I started with to the result (as if I didn't know it is $x^T\Sigma{x} - y^T\Sigma{y}$. Say I didn't have the brainwave to add and subtract $y^T\Sigma{x}$, and then go from there? – 2017-02-23
  • 0
    @NeoCortex64, there isn't really one answer to that. In many cases, adding 0, written as $c - c$, multiplying by 1, written as $c / c$ or some other operation (like taking the double transpose, or the transpose of a scalar) get you to a simplified expression. This is something you get good at with experience. Often it is simpler to go from the complicated form the simplified form, though here it was not immediately obvious I grant you. – 2017-02-23
  • 0
    Thanks. I see this is something I would easily get if I interact with linear algebra all the time. I suppose it's similar to factorising $(a^2-b^2)$, you would add and subtract $ab$ and then go from there. – 2017-02-23
  • 0
    @NeoCortex64, exactly! In fact, that result is simply a specialization of the general result. Often mathematics goes both ways: sometimes you've got to make an expression bigger, to simplify it in the end, sometimes you've got to go and specialize a theorem, sometimes you've got to and generalize a theorem. – 2017-02-23
0

We have $(x+y)^T\Sigma(x-y) = (x^T+y^T)\Sigma(x-y)=x^T\Sigma x-x^T\Sigma y+y^T\Sigma x-y^T\Sigma y$

Now as $\Sigma$ is symmetric, it is a self-adjoint operator, so we have $x^T\Sigma y = = <\Sigma y,x>= = y^T \Sigma x$, where $ = a^T b$ is the standard inner product.

So the expression simplifies to $x^T \Sigma x - y^T \Sigma y$.