I can't understand why max min of a function is less than equal to min max of that function i.e
Why $$\underset{x}{\text{max}}\:\underset{y}{\text{min}} f(x,y) \leq \underset{y}{\text{min}}\:\underset{x}{\text{max}}f(x,y)$$
Here $x,y \in \mathbb{R}$ and $f(x,y)\in \mathbb{R}$
Moreover, I don't understand intuitively what is the effect of just changing the order of max and min.
Suppose $(\hat{x},\hat{y})$ is the solution of $\underset{x}{\text{max}}\:\underset{y}{\text{min}} f(x,y)$, then why this is not the same solution for $\underset{y}{\text{min}}\:\underset{x}{\text{max}}f(x,y)$.
One more thing I want to know is do we evaluate inner optimization first or outer ? i.e in $\underset{x}{\text{max}}\:\underset{y}{\text{min}} f(x,y)$, do we evaluate $\underset{y}{\text{min}}$ first or $\underset{x}{\text{max}}$ first ?
Max Min of function less than Min max of function
-
2It's a difficult matter. John Forbes Nash has won a Nobel prize for a general theorem about min-max-problems. – 2012-08-25
-
1$\max_x \min_y f(x,y) = \max_x \left( \min_y f(x,y) \right)$, i.e. the maximum (with respect to $x$) of the minimum (with respect to $y$) of $f(x,y)$. It's usually kind of hard to evaluate the outer optimization without first evaluating the inner, since you need to evaluate the inner one just to know what the outer one is supposed to be optimizing. Of course, sometimes one _can_ find clever ways to skip or delay the inner optimization, but those cases are the exception, not the rule. – 2012-08-26
-
4An intuitive explanation: Think about this as a two-player game. The goal for player $x$ is to make $f(x,y)$ as small as possible and player $y$ to make it big. The left-hand side is the case where $x$ gets to move first, the right-hand side is where $y$ moves first. The inequality shows that it is sometimes advantageous to move first in a two player game, depending on the nature of $f(x,y)$. I imagine this interpretation is what gave rise to the minimax theorem, which gives some conditions for the interchange of $\min$ and $\max$ with equality. – 2015-01-04
-
1The above explanation is backward. Max-min inequality shows that is better to play second, not first. On the left, $y$ plays second, in reaction to any possible $x$. On RHS, $x$ plays second, in reaction to any possible $y$. Max-min inequality shows that it's better to wait until your opponent plays first. Intuitively, once your opponent (the x-player) reveals their strategy $x \in X$, you can choose your best response $\min_y f(x,y)$. If you choose first (RHS), any choice that you make (even the best one, $\min_{y\in Y}$) is going to be worse for you (higher $f(x,y)$) against an adversary x. – 2018-06-16
8 Answers
The inequality should be using the infimum and supremum:
$$ \sup_{x} \inf_{y} f \left( x, y \right) \leq \inf_{y} \sup_{x} f \left( x, y \right) $$
Defining $ m \left( x \right) = \inf_{y} f \left( x, y \right) $ and $ M \left( y \right) = \sup_{x} f \left( x, y \right) $.
By using the definition of infimum and supremum:
$$ \exists \hat{x}, \hat{y} \epsilon >0 : \sup_{x} m \left( x \right) - \epsilon \leq m \left( \hat{x} \right), \; M \left( \hat{y} \right) \leq \inf_{y} M \left( y \right) + \epsilon $$
Now:
$$ \sup_{x} m \left( x \right) - \epsilon \leq m \left( \hat{x} \right) \leq f \left( \hat{x}, \hat{y} \right) \leq M \left( \hat{y} \right) \leq \inf_{y} M \left( y \right) + \epsilon $$
Hence clearly $ \sup_{x} m \left( x \right) \leq \inf_{y} M \left( y \right) $ which means:
$$ \sup_{x} \inf_{y} f \left( x, y \right) \leq \inf_{y} \sup_{x} f \left( x, y \right) $$
As requested.
Intuition? Well, think of two forces. The one want to minimize its degree of freedom. The other one want to maximize its degree of freedom.
It seems there is much more incentive to the force for minimization to work first. It guarantees the end result will be lower than the other way around (It is symmetrical from the point of view of the maximizing force).
One asks that $$ \max\limits_x\left(\min\limits_sf(x,s)\right)\leqslant\min\limits_y\left(\max\limits_tf(t,y)\right). $$ The assertion is equivalent to the fact that, for every $x$ and $y$, $$ \min\limits_sf(x,s)\leqslant\max\limits_tf(t,y). $$ Since $\min\limits_sf(x,s)\leqslant f(x,y)\leqslant\max\limits_tf(t,y)$ by definition, this holds.
-
0what is $\lim_y$? – 2012-08-25
-
0@Seyhmus: A typo. As you guessed, one should read $\min\limits_y$. – 2012-08-25
Perhaps a simple example will help. Let $f(x,y) = \sin(x+y)$. Then
$\underset{y}{\text{min}} f(x,y) = -1$ for all $x$; and
$\underset{x}{\text{max}} f(x,y) = +1$ for all $y$.
So $\underset{x}{\text{max}}\:\underset{y}{\text{min}} f(x,y) = \underset{x}{\text{max}} (-1) = -1$; but $\underset{y}{\text{min}}\:\underset{x}{\text{max}} f(x,y) = \underset{y}{\text{min}} (+1) = +1\,$.
I personally found this question in ML class of mine and went down to solve it myself at home as the teacher did not give any proof. Here it is:
Let $ f(x_{0}, y_{0}) = \max_x\min_y f(x, y)$ and $f(x_{1}, y_{1}) = \min_y\max_x f(x, y)$.
By this definition the problem is to prove that $f(x_{0}, y_{0}) \leq f(x_{1}, y_{1})$ provided that they exist.
By definition of min and max function we have:
$\min_yf(x, y) = f(x, y_{0}) \leq f(x, y) \forall y$. Here $\min_yf(x, y)$ would be a function only of x.
$\max_xf(x, y_{0})=f(x_{0}, y_{0}) \geq f(x, y_{0})\forall x$. Here $\max_xf(x, y_{0})$ is a scalar.
$\max_xf(x, y)=f(x_{1}, y) \geq f(x,y) \forall x$. Here $\max_xf(x_{1}, y)$ is a function of y.
$\min_yf(x_{1}, y)=f(x_{1},y_{1}) \leq f(x_{1}, y) \forall y$. Here $\min_yf(x_{1}, y)$ is a scalar.
From all this equation you get the following inequalities:
$f(x, y_{0}) \leq f(x_{0},y_{0}) \leq f(x,y) \leq f(x_{1}, y_{1}) \leq f(x_{1}, y)$
$\min_yf(x,y) \leq \max_x\min_yf(x,y) \leq f(x,y) \leq \min_y\max_xf(x,y) \leq \max_xf(x,y)$ $g(x) \leq Scalar \leq f(x,y) \leq Scalar \leq h(y)$
Hope the last thing as visualization helps you understand the problem.
-
0Well just to add a comment that might on the other hand looks really confusing, but the left part of the inequalities are varying on X and the right ones are varying on y... I just hope it helps cause now I realize it might actually confuse you even more. – 2012-09-03
-
0This is excellent thanks a lot - I have edited your formatting as I feel this answer would gain a lot from being seen more – 2018-06-27
Let $\hat x,\hat y$ be the arguments responsible for the value $\underset{x}{\text{max}}\:\underset{y}{\text{min}} f(x,y)$. Then $f(\hat x,y)\ge f(\hat x,\hat y)$ for all $y$. For every $y$, the maximization $\underset{x}{\text{max}}f(x,y)$ extends over one of these values, and thus $\underset{x}{\text{max}}f(x,y)\ge f(\hat x,\hat y)$ for all $y$, and thus also $\underset{y}{\text{min}}\:\underset{x}{\text{max}}f(x,y)\ge f(\hat x,\hat y)=\underset{x}{\text{max}}\:\underset{y}{\text{min}} f(x,y)$.
-
0Provided that $\hat{x}$ and $\hat{y}$ exist, of course. – 2012-08-25
-
1As Juan and Siminore rightly point out, this only works if the minima and maxima actually exist; else they need to be replaced by infima and suprema, respectively. – 2012-08-25
-
0@Joriki I understood first line, but in starting of third line, you wrote $\underset{x}{\text{max}}f(x,y) \geq f(\hat{x},\hat{y})$, isn't $\underset{x}{\text{max}}f(x,y)$ same as $f(\hat{x},y)$ ? – 2012-08-25
-
0@Happy: No, why should it be? $\hat x$ is the value of $x$ that maximizes $\underset{y}{\text{min}} f(x,y)$. I don't see why the same value of $x$ should maximize $f(x,y)$ for every $y$. In fact, since the values of $f$ for different values of $y$ can be chosen entirely independently of each other, it would be weird if it did. – 2012-08-25
-
0joriki: Sorry but I do not understand the definition of $\hat y$. Shouldn't this depend on $x$? – 2012-08-25
-
0@Joriki Sorry I didn't understand the second and third line of your answer. Actually I am confused about the order in which $\underset{x}{max}\:\underset{y}{min}f(x,y)$ is evaluated. Do we minimize first or maximize ? – 2012-08-25
-
0@did: Your answer is much clearer anyway; maybe I should just delete this. I was hoping that "responsible" would be self-explanatory, but apparently it isn't -- I meant that $\underset{x}{\text{max}}\:\underset{y}{\text{min}} f(x,y)$ is the value of $f$ at some values $\hat x$, $\hat y$; i.e. $\hat y=\operatorname{argmin}_y(f(\hat x,y))$. – 2012-08-25
-
0@Happy: There is only one way to evaluate that expression. $(\max (\min))(f)$ makes no sense, so it has to be $\max(\min(f))$. – 2012-08-25
-
0@joriki: Understood. Thanks. – 2012-08-25
The inequality should be $\sup_x\inf_yf(x,y)\leq \inf_y\sup_x f(x,y)$. Consider $f(x,y)= x^2y^2$.
-
0I downvoted here, because the inequality wasn't reversed. – 2012-08-26
-
0It's not reversed, I was just (a bit pedantically) pointing out the inequality is in terms of supremums and infimums instead of maxima and minima, the original phrasing implicitly assumed that the minima and maxima actualy exists. – 2012-08-26
-
0Oh you're right. I'm sorry. – 2012-08-26
-
0That's ok, no worries ^^. – 2012-08-27
I had to prove this for an assignment the other day and I came up with a fairly clean proof that I thought I should share:
We define a' and b' as follows:
$$a'\triangleq\underset{a\in A}{argmin}\left\{ \underset{b\in B}{\max}f(a,b)\right\} \\ b'\triangleq\underset{b\in B}{argmax}\left\{ \underset{a\in A}{\min}f(a,b)\right\} $$
a' is the point in A where the MinMax is reached, whereas b' is the point in B where the MaxMin is reached.
Consider f(a',b'). We can bound it from above and below as follows:
$$\underset{a\in A}{\min}f(a,b')\leq f(a',b')\leq\underset{b\in B}{\max}f(a',b)$$
Now we can plug in a' and b' according to their definitions and get:
$$\underset{b\in B}{max}\left\{ \underset{a\in A}{\min}f(a,b)\right\} =\underset{a\in A}{\min}f(a,b')\leq f(a',b')\leq\underset{b\in B}{\max}f(a',b)=\underset{a\in A}{min}\left\{ \underset{b\in B}{\max}f(a,b)\right\} $$
Q.E.D
Note that this proof works for discrete A,B and as well as for infinite yet compact A,B (with continuous f).