2
$\begingroup$

So here's the question:

Given a collection of points $(x_1,y_1), (x_2,y_2),\ldots,(x_n,y_n)$, let $x=(x_1,x_2,\ldots,x_n)^T$, $y=(y_1,y_2,\ldots,y_n)^T$, $\bar{x}=\frac{1}{n} \sum\limits_{i=1}^n x_i$, $\bar{y}=\frac{1}{n} \sum\limits_{i=1}^n y_i$.
Let $y=c_0+c_1x$ be the linear function that gives the best least squares fit to the points. Show that if $\bar{x}=0$, then $c_0=\bar{y}$ and $c_1=\frac{x^Ty}{x^Tx}$.

I've managed to do all the problems in this least squares chapter but this one has me completely and utterly stumped. I'm not entirely sure what the question is even telling me in terms of information nor do I get what it's asking. Any ideas on where to start?

  • 0
    Start by representing mathematically what it means that this linear function is the best least squares fit. Best in what sense? what is it minimizing? etc...2012-10-14
  • 0
    @Bitwise is it the error?2012-10-14
  • 0
    yes, the regression line is minimizing the squared error right? meaning that for each data point, there is a matching point on the line, and the squared distance between them should be minimal. Try writing that down explicitly.2012-10-14
  • 0
    @Bitwise okay I understand but i'm not really following on how to write that down explicitly2012-10-14
  • 0
    @CharlieYabben You sure the formula for $\hat c_1$ in the question is correct?2013-04-01
  • 0
    @TongZhang Your edit to replace the wrong denominators by $n$ is allright but the other edit replacing $c_0$ and $c_1$ by $\bar c_0$ and $\bar c_1$ is just unneeded.2013-04-01
  • 0
    @TongZhang Don't. Leave the post as it is, unless something is mathematically wrong in it.2013-04-01

2 Answers 2

1

You want to minimize:

$||\hat{y}-y||^2 = ||(c_0+c_1x)-y||^2$

where $\hat{y}$ are given by the linear regression. Intuitively, the best line minimizes the squared error between the "observed" ys and the "predicted" ys.

So, you need to find that $c_0,c_1$ that minimize this expression and show they are equal to those specified above.

Can you go on from there?

  • 0
    is $\hat{y}$ the same thing as $\bar{y}$? If so I would plug in $0$ for $x$ and that would leave me with $||\hat{y}-y||^2=||c_0-y||^2$. Then I sqrt root both sides and have $\hat{y}-y=c_0-y$ which then leads me to $\hat{y}=c_0$. Therefore I just proved one of them. Is that right?2012-10-14
  • 0
    Unfortunately, no. $\hat{y}$ and $y$ are vectors, while $\bar{y}$ is a scalar (number) which is actually just the mean of the vector $y$. $\hat{y}$ is the vector created by taking the vector $x$ and multiplying each coordinate by a number ($c_1$) and then adding to each coordinate a number ($c_0$). In other words, these are the values "predicted" by the regression line for $x$.2012-10-14
1

The OLS estimators for $c_0$ and $c_1$ under the general setting(with no $\bar x=0$) are: $$ c_0=\bar y- c_1\bar x, c_1= \frac{(x-\bar x\boldsymbol{1})^T(y-\bar y\boldsymbol{1})}{(x-\bar x\boldsymbol{1})^T(x-\bar x\boldsymbol{1})}, $$ where $\boldsymbol{1}=(1,...,1)^T$ is a vector of one's and of length $n$. With the additional assumption $\bar x=0$, this reduces to $$ c_0=\bar y, c_1= \frac{x^T(y-\bar y\boldsymbol{1})}{x^Tx} $$