0
$\begingroup$

I have a very simple question on the LASSO estimator (I am a beginner). This is the LASSO problem

$$ \hat{\beta}_n:=\operatorname*{argmin}_{\beta \in B} ||Y-X\beta||^2_2+\lambda||\beta||_1 $$

where $Y$ is an $n\times 1$ vector, $X$ is an $n\times k$ matrix, $\beta$ is a $k\times 1$ vector, $n$ is the sample size and $k$ is the number of regressors.

I was trying to understand why the LASSO estimator does not have a closed form when $X$ is non-orthogonal. One way to see it is by taking F.O.C. as explained here.

My question is: when taking F.O.C. we find the term$\frac{\partial |\beta_j|}{\partial \beta_j}|_{\hat{\beta}_{n,j}}$ for $j=1,...,k$. Writing that requires assuming that $\beta_j\neq 0$ for $j=1,...,k$ because otherwise the absolute value function is non differentiable. However, the econometrics motivation of the LASSO estimator is that, for some $j=1,...,k$, $\beta_j=0$ at the population level. How do these two things reconcile? Is it because $\beta_j\neq 0$ at the sample level as a consequence of sampling error?

  • 0
    You say you want to understand the result for non-orthogonal $X$, but your final question is about the orthogonal case. Anyways, the $\beta$ that you divide by is the non-LASSO estimator. – 2017-01-15
  • 0
    @LinAlg Thanks a lot. However, I don't understand: when I say "orthogonal case" I refer to the case in which $X'X=I$. Why did you say that my question is referred to the orthogonal case? Where do I divide by $\beta$ and what is the non-LASSO estimator? Thanks – 2017-01-15
  • 0
    I find it interesting that one would ask "why is there no closed form solution". Such a question implies an expectation that most optimization problems *should* have a closed-form solution, and that it's odd that this one would not. On the contrary: closed-form solutions are the *exception*, not the rule; it would indeed be a *surprise* to me if one existed here. – 2017-01-15
  • 0
    Your question is about the solution with the sign-operator, which is the solution for the [orthogonal case](http://stats.stackexchange.com/questions/174003/why-is-there-no-closed-form-lasso-solution). The sign operator is taken of the non-LASSO estimator, that is the least-squares solution ($\lambda=0$). – 2017-01-15
  • 0
    @LinAlg Thanks. I have removed it the sign operator and left the derivative of the absolute value of $\beta_j$. My question is: how can I be sure that this derivative exists? – 2017-01-16
  • 0
    You cannot be sure, but the probability is $0$ that the least squares estimat**or** is $0$. – 2017-01-16

0 Answers 0