9
$\begingroup$

I'd like to know why inner products in Reproducing kernel Hilbert spaces are (linear) evaluation functionals.

I understand that inner products are linear functionals, and I know what an evaluation functional is, I just can't explain why an inner product (in a RKHS) is evaluation functional, and vise-versa.

1 Answers 1

12

On a Hilbert space, all continuous linear functionals are inner product functionals (Riesz), and conversely (Cauchy-Schwarz). In a RKHS, evaluation functionals are continuous, which is equivalent to being inner product functionals. The converse is usually not true. That is, inner product functionals on a RKHS need not be point evaluations.

Let $H$ be a Hilbert space consisting of complex-valued functions on a set $X$ such that for each $x\in X$, the evaluation functional $f\mapsto f(x)$ is continuous. Then for each $x\in X$ there is a $k_x\in H$ such that for all $f\in H$, $f(x)=\langle f,k_x\rangle$. (The function $K:X\times X\to \mathbb{C}$ defined by $K(x,y)=k_y(x)=\langle k_y,k_x\rangle$ is the reproducing kernel of the RKHS $H$, and some authors start with $K$ in defining a RKHS.) Only the inner products with the elements $k_x$ are evaluation functionals.

For example, consider the Hardy space $H^2$ of holomorphic functions on the open unit disk whose sequences of Maclaurin series coefficients are in $\ell^2$, with inner product $\displaystyle{\left\langle \sum_{k=0}^\infty a_kz^k,\sum_{k=0}^\infty b_kz^k\right\rangle=\sum_{k=0}^\infty a_k\overline{b_k}}$. Evaluations on the open disk are continuous, as can be seen directly by writing down the element $k_w$ of $H^2$ whose inner product functional is evaluation at $w$, $k_w(z)=\sum_{k\geq0} \overline{w}^kz^k=\frac{1}{1-\overline{w}z}$. So a necessary and sufficient condition for an inner product functional $f\mapsto\langle f,g\rangle$ to be an evaluation functional is the existence of a $w$ in the open unit disk such that $g=k_w$, a condition which typical $g\in H^2$ will not satisfy. Note that the set of evaluation functionals is not closed under scalar multiplication, nor addition. In fact, it is linearly independent.

A simpler but in some ways less interesting example is $\ell^2$ thought of as a space of functions on the nonnegative integers, where the evaluation functionals are just the inner products with elements of $\ell^2$ that have the value $1$ at one point and vanish elsewhere. An even simpler example would be a finite dimensional Hilbert space thought of as functions on a finite set. In these cases, cardinality is enough to see that most inner product functionals are not point evaluations.

The inner product functionals and evaluation functionals would be identical if you were considering a Hilbert space $H$ as a space of functions on its dual space, in the usual isomorphism of $H$ with its double dual.


Added: Here is some elaboration on the first 2 sentences. If $H$ is a Hilbert space and $g\in H$, then the function $T_g :H\to \mathbb{C}$ defined by $T_g(f)=\langle f,g\rangle$ is a linear functional on $H$ called an "inner product functional" above. Each inner product functional is continuous. The operator norm of $T_g$ is equal to $\|g\|$. The Cauchy-Schwarz inequality gives $|T_g(f)|\leq \|f\|\|g\|$ for all $f$, which means $\|T_g\|\leq\|g\|$. Plugging $g$ into $T_g$ gives $|T_g(g)|=\|g\|^2$, showing that $\|T_g\|\geq \|g\|$.

So inner product functionals are continuous, and this would be true in any inner product space. The Riesz representation theorem (for Hilbert space, sometimes also called Riesz's lemma) says that the converse is true for a Hilbert space. You can see this for example in the Wikipedia article, and in many textbooks including the basics of Hilbert spaces, such as Rudin's Real and complex analysis. That is, if $T:H\to\mathbb{C}$ is any continuous linear functional, then there is a $g\in H$ such that $T=T_g$.

Hopefully the first sentence is clearer now. As for the second sentence, it follows directly from the first sentence and the definition of RKHS, and the second paragraph elaborates on this. There is more than one way to characterize RKHS, and if continuity of point evaluations isn't clear from your definition, perhaps you could provide the definition to make it easier to answer your questions.

  • 0
    Thanks for your reply. You've obviously put a lot of thought into it. However, its the first two sentences that I'm struggling with i.e. that "On a Hilbert space, all continuous linear functionals are inner product functionals." And that "In a RKHS, evaluation functionals are continuous, which is equivalent to being inner product functionals." I just can't make these inferences from my current understanding of the Cauchy-Schwartz and Riesz theorems.2011-02-20
  • 0
    Also can I add that what I hope to do is to make an intuitive argument for RKHS'. Currently I've found several references that claim that the Hilbert space $\mathcal{L}_2$ is "too big" and contains too many non-smooth functions and also that its evaluation functionals (still don't know how you get those) are unbounded. I assume that RKHS solve both problems and would appreciate help showing that this is indeed the case.2011-02-20
  • 0
    After some more surfing the internet I found a [document](http://lovvge.wordpress.com/2010/02/16/9-520-lecture-3-reproducing-kernel-hilbert-space/) and [another](http://www.mit.edu/~9.520/scribe-notes/class03_gdurett.pdf) that explains the rationale for excluding discontinuous functions. Also, contrary to what I originally thought, a RKHS has the property that at each $x \in \mathcal{R}^n$ there exists an _evaluation functional_ $K_x$ called the representer such that $f(x) \; = \; \langle K_x \; , \; f \rangle$ for any function $f$.2011-02-20
  • 0
    ... In other words, $K_x$ is the evaluation functional not the inner product. Although it appears that, like a the dirac delta, the $K_x$ is defined in terms of its effect on other functions via the inner product. What remains is to prove that evaluation functionals indeed have this property. (Now I realize that your post covers much of I've just learned.)2011-02-20
  • 0
    @Olumide: I elaborated a little after seeing your first 2 comments, and before seeing your last 2. In your second comment, what is $\mathcal{L}_2$? If it is what I think it is, it is not even a space of functions, let alone a RKHS. In your third comment, yes, that is in my post. In your 4th comment, $K_x$ is neither an evaluation nor an inner product; it is an element of your space whose inner product functional is equal to evaluation at $x$. Proving that evaluations have this property (existence of such $K_x$) is direct from definition/Riesz.2011-02-20
  • 0
    @Olumide: I just noticed the question http://mathoverflow.net/questions/36599/, of which this appears to be a sequel.2011-02-20
  • 0
    @Jonas: yes this question is one of several that I've asked on the subject of Hilbert spaces (here's [another](http://mathoverflow.net/questions/35840/the-role-of-completeness-in-hilbert-spaces)). I thought I'd understood the proof of Riesz theorem, because I was able to follow the argument in a "Principles of Functional Analysis" by Martin Schechter. Perhaps I should take a closer took at that text.2011-02-20
  • 0
    Here's how far I've gotten since my last post. (Please be patient with my slow pace. I'm teaching myself all this.) A RKHS is a subspace of continuous or pointwise defined functions. Therefore an evaluation functional on such a space will also be continuous or pointwise defined. Now from Riesz we know that if $\mathcal{F}$ is a bounded linear functional on a Hiblbert space $\mathcal{H}$, then there is a vector $x$ in that space such that $\mathcal{F}_x(f) = \langle \; f \; , \; x \; \rangle$ for all $f \in \mathcal{H}$. What I still can't show is that $\mathcal{F}$ is an evaluation functional.2011-02-21
  • 0
    Martin Schechter, on page 29 of "Principles of Functional Analysis", states without proof that the inner product is the only bounded linear functional that exists, but does not prove this statement. And if a RKHS is the space of bounded, continuous (i.e. nice/smooth) functions with bounded, continuous evaluation functionals, and the only bounded continuous functional that exists is the inner product, then the evaluation functional must be an inner product. This means what is must be proved is that the inner product is the only bounded linear functional that exists.2011-02-21
  • 0
    Here's another thought: if the indeed the inner product is the only bounded linear functional that there is, can one also assert the reverse statement i.e. all bounded linear functionals are inner products, and prove that instead by Cauchy-Schwartz. Is this the correct way to go about it?2011-02-21
  • 0
    "A RKHS is a subspace of continuous or pointwise defined functions. Therefore an evaluation functional on such a space will also be continuous or pointwise defined.": No. A RKHS is a Hilbert space $H$ whose elements are complex-valued functions on a set $X$ such that for all $x\in X$, the function $E_x: H\to \mathbb{C}$ defined by $E_x(f)=f(x)$ is continuous. Note that this is the definition, so you do not have to prove that evaluations are continuous. Rather, with a specific example you may have to check that the conditions of the definition are satisfied. (to be continued)2011-02-21
  • 0
    "What I still can't show is that $\mathcal{F}$ is an evaluation functional." That is because it is false, as indicated in the last 2 sentences of the 1st paragraph above, the last sentence of the 2nd paragraph, and specifically for the the examples that follow. All continuous functionals are inner product functionals, and conversely; evaluations on a RKHS are only a *subset*, almost always proper, of the set of continuous functionals.2011-02-21
  • 0
    "prove that instead by Cauchy-Schwartz": No, the Cauchy-Schwarz inequality is not enough, because it holds in any inner product space, while Riesz's theorem requires completeness. "Martin...states without proof": There is no shortage of standard texts where you can look up the proof. See for example Theorem 4.12 of Rudin's *Real and complex analysis* 3rd Ed., mentioned above. In your question I linked to above, I thought you already linked to other proofs, so I am confused. "the inner product is the only bounded linear functional that exists": I find the wording confusing here.(to be cont.2011-02-21
  • 0
    The inner product functionals are the only bounded linear functionals on a Hilbert space. See the first 2 paragraphs of the "Added" section in my answer, and a reference such as Rudin for the proof.2011-02-21
  • 0
    @Jonas: thanks as always for your comments. Re your first comment, because I'm an "applied person" I feel the need to justify definitions. Surely there is some intuition behind the restriction that the definition imposes on Hilbert spaces. And from the little that I've read and your original post, the reason for introduction RKHS is to create a subset/subspace of functions that are pointwise defined. It seems to me that an evaluation functional is an operator (on the Hilbert space of functions) that serves this purpose.2011-02-21
  • 0
    @Olumide: It seems you are getting into philosophical questions on which it is hard to say anything definitive. There is always some reason behind having a particular definition of an object that is studied; one typical justification is that enough experience shows that the conditions of the definition generalize a number of "interesting" examples, while still preserving enough structure for some nontrivial theory to be developed. In the case of RKHS, there were a number of specific examples, in particular spaces of holomorphic functions, before the general definition/theory.2011-02-21
  • 0
    @Olumide:"the reason for introduction RKHS is to create a subset/subspace of functions that are pointwise defined"; I do not understand what you mean by that. A RKHS is by (part of the) definition a space of functions defined on some set. This guarantees that pointwise evaluations are defined; the fact that pointwise evaluations are also continuous is another part of the definition. Now, for a particular Hilbert space whose elements are functions, it may be possible that some of the point evaluations are not bounded; that just means that such a space is not a RKHS. (to be continued)2011-02-21
  • 0
    @Olumide: In case you're interested, there is an example of a Hilbert space of functions with an unbounded point evaluation in Halmos's *Hilbert space problem book* (I forget exactly where, but the index makes it easy to find I think). The axiom of choice is used, and in fact some form of AC is necessary to give such an example.2011-02-21