4
$\begingroup$

$\DeclareMathOperator*{\argmin}{arg\,min}$ Suppose I have to jointly minimize two functions. The solution to the joint minimization does not necessarily minimize each function individually but sort of best one can get for both cases together.

e.g., $\argmin_p f(t; p)$ and $\argmin_p g(t; p)$ could be two minimization problems. Suppose though I am looking for a $p$ that minimizes them "jointly".

This seems to be rather vague though? Is there any way to make this more exact? e.g., what could "joint minimization" or "simultaneous minimization" mean? Remember, the goal is to find a p that sort of minimizes both individually but it might not work perfectly for each individual case.


The joint minimization should minimize each individual minimization problem "well".

Suppose $\min_p J(f(t; p), g(t;p))$ is the joint minimization with solution p and $\min_p f(t; p)$ and $\min_p g(t; p)$ have solutions $p1$ and $p2$ respectively.

Then we would want $p$ to be close to $p1$ and $p2$ in some sense. Obviously is it were equal then it will would have solved each individual minimization problem. This doesn't necessarily work well though as maybe there is another solution $p^*$ that is almost a solution to the joint problem BUT provides better results to each individual solution.

That is, I don't think joint minimization says much about how it's solution minimizes the individual problems. (maybe it would be way off)

for example, suppose $f(t) = t^2$ and $g(t) = 1/t^2$ then the joint distribution $|fg| = 1$ has the whole real line the solution. Which, if we were using numerical methods, maybe end up with t = 10^10 for a solution. But $10^{10}$ is a really bad solution for the minimization problem on $f$ while it is ok for g.

Therefor, there seems to be some other criteria that needs to be added to get a better overall solution. Something like:

$\argmin_{p1} f(t; p1)$ $\argmin_{p2} g(t; p2)$ $\argmin_p J(f(t; p), g(t;p))$ $\min_{p^*} |p1 - p| + |p2 - p|$

The above is a bit sloppy but the idea is that we make sure our solutions to the individual problems are "close" to the joint problem if possible). This, at the very least prevents joint problems of the form $|fg|$.

  • 0
    @ChristianBlatter The issue is not so much about which is best but which will absolutely not work. Some interpretations of "joint minimization" simply make know sense and others seem to be preferred for a more general, unbiased case. The whole point of a joint minimization, I believe, is to minimize the individual cases well but leaving "wiggle" room. But the the solution will be worthless if the "wiggle" room is too great since it's solution will not work at all for the individual cases. Therefore, it seems having such an interpretation would limit possible choices for a joint function.2012-09-27

1 Answers 1

2

There are many ways to make it more exact. However, there are many different ways to make it more exact, and they lead to different answers, and there is no abstract sense in which one of these answers is better than the others.

So, some examples: you could try to minimize $f^2+g^2$ (this is very popular, under the name, "least squares", or "ell-two norm"), or you could try to minimize $|f+g|$, or more generally $\root q\of{f^q+g^q}$ for some chosen number $q$, $1\le q\lt\infty$; you could try to minimize $\max(f,g)$, or $\sqrt{|fg|}$, or the harmonic mean $2fg/(f+g)$. Each of these has its uses. You have to decide which one gives results the closest to what you are trying to get.

The ones I've given have been symmetric in $f$ and $g$. In case it's more important to minimize one than the other, there are weighted versions of all of these, like minimizing $f^2+17g^2$, or $|39f+g|$, and so on, and so forth.

EDIT: Let's look at an example. Let $f(x)=|x|$, $g(x)=|4-2x|$, and try to jointly minimize on $[0,2]$. Here are some candidates: $\matrix{x&f(x)&g(x)&\max&f^2+g^2&f+g&{\rm features}\cr4/3&4/3&4/3&1.33&3.56&2.67&{\rm minimizes\ maximum}\cr8/5&8/5&4/5&1.6&3.2&2.4&{\rm minimizes\ }f^2+g^2\cr2&2&0&2&4&2&{\rm minimizes\ }f+g\cr}$

  • 0
    Have you had a look at the examples I added to my answer?2012-09-28