1
$\begingroup$

Consider the following:

$\gamma=\dfrac{1}{4Y}\log_{\beta}\dfrac{\int_{\mathbb{R}^{n}}\exp(-a\eta||\theta||_1-\eta\theta^{\top}(\sum_{t=1}^Tx_tx_t^{\top})\theta+2\eta\sum_{t=1}^{T-1}(y_tx_t^{\top}-Yx_T^{\top})\theta) \mathrm d\theta}{\int_{\mathbb{R}^{n}}\exp(-a\eta||\theta||_1-\eta\theta^{\top}(\sum_{t=1}^Tx_tx_t^{\top})\theta+2\eta\sum_{t=1}^{T-1}(y_tx_t^{\top}+Yx_T^{\top})\theta)\mathrm d\theta}\;\;\;(1)$

where $\beta=e^{-\eta}$, for $||\theta||_2^2$, solution is:

$\gamma = \dfrac{1}{4Y}\log_{\beta}\dfrac{\int_{\mathbb{R}^{n}}\exp(-a\eta\theta^{\top}(aI+\sum_{t=1}^Tx_tx_t^{\top})\theta+2\eta\sum_{t=1}^{T-1}(y_tx_t^{\top}-Yx_T^{\top})\theta)\mathrm d\theta}{\int_{\mathbb{R}^{n}}\exp(-a\eta\theta^{\top}(aI+\sum_{t=1}^Tx_tx_t^{\top})\theta+2\eta\sum_{t=1}^{T-1}(y_tx_t^{\top}+Yx_T^{\top})\theta)\mathrm d\theta}$

By using Harville, 1997, Theorem 19.1.1 which is as follows:

$F(A,b,x)=\inf_{\theta\in\mathbb{R}^n}(\theta^{\top}A\theta+b^\top\theta+x^\top\theta)-\inf_{\theta\in\mathbb{R}^n}(\theta^{\top}A\theta+b^\top\theta-x^\top\theta)$

$=-0.25(b+x)^\top A^{-1}(b+x)+0.25(b-x)^{\top} A^{-1}(b-x)=-b^\top A^{-1}x$

$A$ is a positive definite $n\times n$ matrix. $b$ and $x$ are vectors.

we can write

$\gamma=\frac{1}{4Y}\log_{\beta}e^{-\eta F(aI+\sum_{t=1}^Tx_tx_t^{\top},-2\sum_{t=1}^{T-1}y_tx_t^{\top},2Yx_T^{\top})}$

$=\frac{1}{4Y}F(aI+\sum_{t=1}^Tx_tx_t^{\top},-2\sum_{t=1}^{T-1}y_tx_t^{\top},2Yx_T^{\top})$

$=(\sum_{t=1}^{T-1}y_tx_t^{\top})(aI+\sum_{t=1}^Tx_tx_t^{\top})^{-1} x_T\;\;\;(2)$

$a >,\eta >0$, all the rest are vectors. $(2)$ is representation of ridge regression in dual form.

Is it possible for someone to clarify me if one can evaluate $(1)$ analytically?I am hoping to get LASSO in dual form, one can suggest the changes required in $(1)$ to obtain LASSO.

Notes

Maybe $(1)$, can be solved by writing it as:

$\gamma=\frac{1}{4Y}\log_{\beta}e^{-\eta F(\sum_{t=1}^Tx_tx_t^{\top},-2\sum_{t=1}^{T-1}y_tx_t^{\top},2Yx_T^{\top})}\dfrac{\int_{\mathbb{R}}e^{-\eta a ||\theta||_1}\mathrm d\theta}{\int_{\mathbb{R}}e^{-\eta a ||\theta||_1}\mathrm d\theta}$

applying Fubini's theorem can we solve it?

(Fubini's theorem does not seem like the right choice here. Also not sure if one can write the integral in such a way)

Main Problem

Following is the main issue:

$F(A,b,x,c)=\inf_{\theta\in\mathbb{R}^n}(\theta^{\top}A\theta+b^\top\theta+x^\top\theta+c||\theta||_1)-\inf_{\theta\in\mathbb{R}^n}(\theta^{\top}A\theta+b^\top\theta-x^\top\theta+c||\theta||_1)=?$

$x,b,\theta\in\mathbb{R}^n$ i.e. are vectors, $c>0$ scalar. $A$ is a positive definite matrix. $||\theta||_1=\sum_{r=1}^n |\theta_r|$. In order to solve the above one may require to use something like subdifferential since absolute values are not differentiable.

Link to Above explanation

$\int_{\mathbb{R}^n}e^{-f(\theta)\mathrm d\theta}=e^{f_0}\frac{\pi^{n/2}}{\sqrt{\det A}}$

where $f_0=\min_{\theta\in\mathbb{R}^n} f(\theta)$. So if I can find $f_0$, I know that $\beta=e^{-\eta}$ and I have $\log_{\beta}$, which will cancel.

  • 0
    @littleO Nop if you see above the same expression is written with the '-' with $x$, with L1 norm on $\theta$. I am just asking to prove that the problem is a Lasso problem i.e. show that this is indeed Lasso in dual form. – 2017-01-23
  • 0
    Oh I see, I didn't notice the difference! – 2017-01-23
  • 0
    Since the "main problem" has been edited maybe we should clean up our previous comments. I'm going to delete mine. – 2017-01-23
  • 0
    Do you have any solution or suggestions for my main problem? – 2017-01-23
  • 0
    When you say "I am hoping to get Lasso in dual form", how would evaluating (1) analytically help you to get Lasso in dual form? If you wanted to derive the dual problem for a standard Lasso problem, there are some standard ways to do that. – 2017-01-23
  • 0
    The thing is I am driving an algorithm for LASSO. This is the mathematical challange I'm faced with. The data I've is huge and ridge regression is not that accurate for my data.Other algorithms have distributional assumptions or no upper bound to it. – 2017-01-23
  • 0
    Oh ok. I think that extra context makes your question much more clear. Are you able to derive the dual problem for a standard Lasso problem? If not, there are much simpler ways to do that. – 2017-01-23
  • 0
    By the way, Lasso is an extremely well studied problem, it will be quite difficult to beat the state of the art by inventing your own new algorithm. Have you tried just using one of the popular Lasso algorithms that are available? – 2017-01-23
  • 0
    Yes beating it is possible in my case. But that is irrelevant at this site. That is more of a computational aspect of it. – 2017-01-23
  • 0
    I think this question will get more attention if it were easier to understand the context. The quantity in equation (1) is labeled "prediction" but what is it a prediction of? Why would it be helpful to evaluate (1) analytically? – 2017-01-23
  • 0
    It really won't affect even if I write just P. This issue is solving the algebraic problem. To explain everything I will require to give elaborate background, I don't see the value in it, but for clarity I will clean up. – 2017-01-23

0 Answers 0