0
$\begingroup$

Having gained an informal appreciation for optimization a while back, but only now starting to get into the stepping stones to formally understanding it, I have some contextualization issues I'm hoping someone can help me resolve.

I'm reading a book on dynamical systems as well as taking a statistics class. Now I look back at mean squared error and gradient descent in a dynamical systems as well as a statistical perspective.

  1. We update the weight matrix $\mathbf{W}$ according to: $$\frac{d}{dt}\mathbf{W} = -\frac{\partial}{dW} E(\mathbf{W})$$ or $$\mathbf{\dot{W}} = -E'(\mathbf{W})$$ which is symbolically equivalent to the dynamical system $\dot x = f(x)$
  2. However, something about the process is probabalistic, although I cannot formally point my finger at something in particular with certainty. I see a bunch of elements from statistics going on but I am having trouble making a bijection between the statements being made by the learning equations and concepts elsewhere in statistics. For example, the definition of mean squared error looks suspiciously like a variance: $$E(W) = \frac{1}{n}\sum^n_i (\hat{Y} - Y_i)^2$$ I mean I can even imagine myself writing something like $Y_i = E[\hat{Y_i}]$, i.e. the expectation.

I'm having trouble figuring out to which distribution the variance belongs? The weights, the outputs?

  • 0
    Sure there is a huge overlap between optimisation and statistics, I'm not sure taxonomies diving maths up into pure, applied, statistics, machine and so on are ever really that useful, sometimes it makes sense to put the bats with the birds and sometimes not. It's probably better to think of them sharing broader similarities though, rather than right down at the distributional, moment level. – 2017-02-22
  • 0
    So for example that dynamical system $\dot{w} = - f^{\prime}(w)$ is just gradient descent of the function $f$, and that is such a ubiquitous notion in so many areas it wouldn't really be right to talk about it describing a bijection between any two. In the same way the idea of least squares as an orthogonal projection in a Hilbert space certainly does not belong to the statistical theory of linear regression – 2017-02-22

0 Answers 0