3
$\begingroup$

Consider the simple optimization problem $$\min_x \|b-Ax\|_2 \ \ \ \ \ \ \text{subject to} \ \ \ \ \ \ \|x\| = 1$$ where $b\in\mathbb{R}^{N}$, $A\in\mathbb{R}^{N\times 3}$, $x\in\mathbb{R}^{3}$ with $N > 3$ (overdetermined system). I need to solve this millions of times, so computational efficiency is important.

I am aware of the following aspects:

  • The norm equality constraint $\|x\| = 1$ makes the problem non-convex.
  • A relaxation to $\|x\| \leq 1$ leads to a convex optimization problem.
  • I can employ a two-dimensional parametrization of $\mathbb{R}^{3}$ unit vector $x$, e.g. azimuth angle $\phi$ and polar angle $\theta$, and use a gradient search (or similar) to solve $\min_{\phi,\theta} \|b-Ax(\phi,\theta)\|_2$ instead.
  • The problem can be written in quadratic fashion $$\min_x \ x^\text{T}A^\text{T}Ax - 2b^\text{T}Ax \ \ \ \ \ \ \text{s.t.} \ \ \ \ \ \ x^\text{T}x = 1 \ .$$

My questions are:

  1. Is there a super smart and super efficient way to solve my problem? If yes, a minimal Matlab code example that does so would be amazing.
  2. Is "quadratic progamming" eligible?
  3. Does a relaxation to $\|x\| \geq 1$ also lead to a convex problem? Would it be meaningful to first solve s.t. $\|x\| \leq 1$ and then s.t. $\|x\| \geq 1$ in hopes that one of the two solutions fulfills $\|x\| = 1$?
  4. A minimal Matlab example for the relaxed problem s.t. $\|x\| \leq 1$ would be amazing (this should be easy, but I never used CVX or similar).
  5. I found that a normalized pseudo-inverse solution, i.e. $x \approx y\,/\,\|y\|$ with $y = (A^\text{T}A)^{-1}A^\text{T} b$, is a decent approximation quite often. This can be the case even if $\|y\|$ is far off $1$. Can you identify a condition for this approximation being accurate/meaningful?

Info: I never really studied the fine arts of convex optimization and linear/quadratic programming. So, when writing your response, please do not omit details that might be obvious to experts.

Thank you very much!

  • 0
    Have you considered doing this as a Lagrange multiplier problem? It would be a little easier to minimize the square of the 2 norm. An analytic solution should be possible I should think. – 2017-01-06
  • 0
    The problem with $||x|| \geq 1$ is not convex. For $||x|| \leq 1$ you can find x by solving a linear system (write down the KKT conditions of the quadratic form). – 2017-01-06
  • 0
    @Paul I just did and indeed got an analytical solution for the components of $\tilde{x}$ which is the representation of $x$ in the eigenspace of $A^\text{T}A$. However, it's two equations with multivariate fourth-order polynomials in $\tilde{x}_1^2$ and $\tilde{x}_2^2$ (losing sign information), so it's quite hopeless. The ugliness emerges because the constraint dictates $x_3 = \pm \sqrt{1 - x_1^2 - x_2^2}$, which has to be incorporated at some point. – 2017-01-06
  • 1
    Even though this is convex, it is well known that it can has a tractable solution. This is basically the so-called _trust-region subproblem_. – 2017-01-06
  • 0
    @MichaelGrant Thank you, that's very helpful. Seems like I opened an old can of worms, I am reading a thesis about it right now. Do you mean "Even though this is non-convex"? – 2017-01-06
  • 0
    It seems like my problem is covered thoroughly here: http://www.maths.tcd.ie/~mnl/store/ForsytheGolub1965a.pdf – 2017-01-06
  • 0
    Glad you found a good reference. Sorry about the previous typo! – 2017-01-06

0 Answers 0