1
$\begingroup$

I am new to vector derivatives and trying to figure out a lot for my Machine Learning course. I have given the following:

$x \in \mathbb{R}^n$, $y \in \mathbb{R}^d$, $A \in \mathbb{R}^{d \times n}$,

Let $B$ be symmetric (and pos.def.). What is the minimum of

$$(Ax -y)^T (Ax -y) + x^T Bx$$

with respect to $x$?

How can I approach this problem? I have no clue yet, however, I figured out, that $(Ax -y)^T (Ax -y) + x^T Bx \in \mathbb{R}^1$, thus a real number, isn't it? Hence $B \in \mathbb{R}^{n \times n}$

I looked up some derivation rules and I've got the following:

$$\frac{\partial}{\partial x} (Ax -y)^T (Ax -y) + x^T Bx = 2A^T (Ax - y) + (B + B^T) \cdot x$$

Setting $2A^T (Ax - y) + (B + B^T) \cdot x$ to $0$:

\begin{align*} 0 & = 2A^T (Ax - y) + (B + B^T) \cdot x\\ 0 & = 2A^T A x - A^T y + (B + B^T) \cdot x\\ A^T y & = 2A^T A x + (B + B^T) \cdot x\\ (B + B^T)^{-1} \cdot A^T y & = 2A^T A x + x\\ \end{align*}

and here I am some what stuck, however, I don't know if this is the right approach

  • 0
    Minimum with respect to which argument?2012-04-15
  • 0
    @Didier I have no clue, its just the minimum of the whole term.2012-04-15
  • 0
    Most likely it's not the minimum with respect to both $x$ and $y$; since both terms are non-negative, that would trivially be at $x=y=0$. It seems more likely that this is a sum of two cost functions, one measuring the error in solving $Ax=y$ and the other an additional quadratic cost; if so, the task would be to find the minimum with respect to $x$. (The minimum with respect to $y$ is also trivial, $y=Ax$.)2012-04-15
  • 0
    But what does this have to do with vector derivatives? I thought this has at least to be applied.2012-04-15
  • 1
    *No clue*?? So you want people to tell you *what is the question that you are asking*? (I am wondering: is this the exact formulation of your homework?)2012-04-15
  • 0
    @Didier Sadly, yes. Guess I have to ask my professor what he means.2012-04-15
  • 0
    Sounds like a good idea. Unless some similar problems solved during the lectures could indicate what is going on? (Anyway, if I had to guess, I would follow @joriki's suggestion.)2012-04-15

1 Answers 1