1
$\begingroup$

I have implemented a program to solve primal and dual problem defined by:

A)

Primal is:

maximize $cx$ subject to $Ax=b$ and $x>=0$

corresponding dual is:

minimize $kb$ subject to $kA-z=c$

In the above $A$ is a matrix all all other variable is vector.

Now I want to solve the minimization problem:

B)

minimize $(norm(A*x-b)+norm(x,1))$

Where norm(.) is usual L2 norm and norm(x,1) means L1 norm

Can I solve B using the same solving method of A ? If so how can I change the variables of B similar to A? Any guidance would be appreciated.

  • 0
    The short answer is no. What you're describing is a L1 regularized linear least squares (LLS). Additionally, unlike L2 regularized LLS it is not differentiable when $x_i = 0$ for any $i$, it does not have a closed form solution. Fortunately, there is a multitude of literature on the subject. For example, check out https://www.cs.ubc.ca/~schmidtm/Documents/2005_Notes_Lasso.pdf – 2017-01-19
  • 0
    @David Thank you for your comment. I was reading that reference, but could not close the gap between the problem A and B – 2017-01-19
  • 1
    If you're willing to square the two-norm, you'd have a quadratic programming problem that is easy to solve by a primal-dual interior point method. You can also keep the 2-norm without squaring it and get a second order cone program that can be solved by a primal-dual method. – 2017-01-19
  • 0
    @BrianBorchers Would you please send me a reference to your suggestion. It should work for me, I can square the L1 norm if that helps. ( but norm must be L1) – 2017-01-19
  • 0
    See the convex optimization textbook by Vandenberghe and Boyd. These are both standard formulations. – 2017-01-19
  • 0
    The issue of nondifferentiability can easily be resolved by adding some extra variables and constraints. The only true issue here is that your solver is for linear optimization, and your actual problem is nonlinear due to the L2 norm. As Brian suggests, you can modify your algorithm. – 2017-01-19
  • 0
    @LinAlg I am trying to use Newton method once I can convert the problem to primal-dual from. I was thinking Interior method does the same . Newton method can deal with nonlinear function. Any comment please. – 2017-01-19
  • 0
    Introduce $y$, rewrite the objective to $||Ax-b||_2 + \lambda \sum_i y_i$, and add constraints $y_i \geq x_i$, $y_i \geq -x_i$. Now you have a smooth constrained optimization problem. – 2017-01-19

1 Answers 1

1

Here is a more complete answer about the problem dual (since that is what you asked about originally):

We start with the primal problem, an L1-regularized LLS

$\min ||Ax-b||_2^2 + \lambda||x||_1$

Now, let us define $z$ s.t. $z := Ax - b$. Thus, our problem can be written as

$\min z^Tz + \lambda ||x||_1$ s.t. $z = Ax-b$.

Associate $\mu_i$ with the $i$th constaint, thus we have the Langrangian is simply

$L(x,z,\mu) = z^Tz + \lambda||x||_1 + \mu^T(Ax-b-z)$.

Therefore, the dual function is then given by

$\displaystyle g(\mu) = \inf_{x,z}\left\{L(x,z,\mu)\right\}$.

Now, we want to find $\max_{\mu}g(\mu)$, and so first we note that if, for some $i$, $|\sum_j a_{ji}\mu_j| > \lambda_i$, then we have a direction of unboundedness this is because the only terms in the numerator with $x_i$ are

$\displaystyle \left(\sum_ja_{ji}\mu_j\right)x_i$ and $\displaystyle \lambda_i|x_i|$.

Thus for any such $\mu$ we have $g(\mu) = -\infty$. Therefore, we can restrict our attention to $\mu$ s.t. $|(A^T\mu)_i| \leq \lambda_i \forall i$. Now, for each such $\mu$ it can be seen that the $x$ that minimizes $g(\mu)$ is $x=0$.Thus, for all $\mu$ of interest, we have that

$\displaystyle g(\mu) = \inf_{x,z}\{L(x,z,\mu)\} = \inf_z\left\{z^Tz - \mu^Tz - \mu^Tb\right\}$.

Fortunately, we can differentiate this, set it to zero, to see that in optimality, $z_i = \mu_i/2.$

Hence, for all $\mu$ s.t. $|(A^T\mu)_i| \leq \lambda_i \forall i$ we have that

$\displaystyle g(\mu) = -\frac{1}{4}\mu^T\mu-\mu^Tb$.

Therefore, the Lagrangian Dual is given by

$\displaystyle \max_\mu -\frac{1}{4}\mu^T\mu-\mu^Tb$ s.t.$|(A^T\mu)_i| \leq \lambda_i \forall i$.

Therefore, the dual is a convex optimization problem in $\mu$ and because the primal problem satisfies Slater's condition, we have that the duality gap is zero.

  • 0
    I am still trying to understand your answer. In your comment you said "no" but now you have shown the dual. Was the comment "no" is wrong? – 2017-01-19
  • 0
    @Creator I think I misunderstood what you asked for originally. If you simply want the dual, then the answer is yes it exists and I've given it and it's derivation above. – 2017-01-19