0
$\begingroup$

According to Mercer's theorem, for $\mathbf{x},\mathbf{y}\in\mathbb{R}^d$, the symmetric positive-definite kernel $\exp\left(-\|\mathbf{x}-\mathbf{y}\|^2/(2\sigma^2)\right)$ admits an orthogonal basis $\{\phi_i(\cdot)\}_i$ with countable vectors, such that $$ \exp\left(-\frac{\|\mathbf{x}-\mathbf{y}\|^2}{2\sigma^2}\right)=\sum_{i=1}^\infty \lambda_i\phi_i(\mathbf{x})\phi_i(\mathbf{y}) $$

But I can "find" another basis with uncountably many eigenfunctions as follows.

First, the Fourier transform of a Gaussian function is Gaussian. So if changing the variable $\mathbf{s}$ in the transform $$ \exp\left(-\frac{\|\mathbf{s}\|^2}{2\sigma^2}\right)=C\int_{\mathbb{R}^d}\exp\left(-\frac{\sigma^2\|\mathbf{t}\|^2}{2}\right)\exp\left(-i\,\mathbf{s}^\mathrm{T} \mathbf{t}\right)\,\mathrm{d}\mathbf{t} $$ to $\mathbf{x}-\mathbf{y}$, we can have $$ \exp\left(-\frac{\|\mathbf{x-y}\|^2}{2\sigma^2}\right)=C\int_{\mathbb{R}^d}\exp\left(-\frac{\sigma^2\|\mathbf{t}\|^2}{2}\right)\exp\left(-i\,\mathbf{x}^\mathrm{T} \mathbf{t}\right)\exp\left(i\,\mathbf{y}^\mathrm{T} \mathbf{t}\right)\,\mathrm{d}\mathbf{t} $$ which means $\exp\left(-\|\mathbf{x-y}\|^2/(2\sigma^2)\right)$ can be expressed as an inner product of $\exp\left(-i\,\mathbf{x}^\mathrm{T} \mathbf{t}\right)$ and $\exp\left(-i\,\mathbf{y}^\mathrm{T} \mathbf{t}\right)$, with weight $C\exp\left(-\sigma^2\|\mathbf{t}\|^2/2\right)$. So $\phi(\mathbf{x})=(\exp(-i\,\mathbf{x}^\mathrm{T}\mathbf{t}))_\mathbf{t}$, an infinite-dimensional vector indexed by $\mathbf{t}\in\mathbb{R}^d$, can be seen as a basis for the Hilbert space from kernel $\exp\left(-\|\mathbf{x}-\mathbf{y}\|^2/(2\sigma^2)\right)$ (since the inverse Fourier transform of $0$ is $0$, the vectors are linear-independent).

So is there anything wrong in the above "proof"? It seems to me that a vector space with countable basis vectors cannot hold uncountable linear-independent vectors.

1 Answers 1

2

The expression $$\tag{1}\exp\left(-\frac{\|\mathbf{x}-\mathbf{y}\|^2}{2\sigma^2}\right)=\sum_{i=1}^\infty \lambda_i\phi_i(\mathbf{x})\phi_i(\mathbf{y})$$ is an equality between functions. The functions $\phi_i$ are orthonormal. You never bother to mention and/or consider with respect to what inner product. The context of Mercer's theorem is that you are considering kernels of Hilbert Schmidt operators on $L^2(\mathbb R^d)$. So the $\phi_i$ are "orthonormal" in the sense that $$ \delta_{k,j}=\langle \phi_j,\phi_k\rangle:=\int_{\mathbb R^d}\phi_j(\mathbf t)\phi_j(\mathbf t)\,d\mathbf t. $$

In your second approach you want to consider a function as a vector, indexed by its variable. Nothing wrong with that. But you say that the kernel " can be expressed as an inner product": note that $(1)$ is not an inner product with the inner product that is considered in the Hilbert space where your Hilbert-Schmidt operator acts. Also, you say that $\{\exp(-i\mathbf x^T\mathbf t)\}_{\mathbf t}$ is a basis: of what? With what inner product? For a fixed $\mathbf x$?

In the end, I'm just guessing, but I think that your confusion arises from seen $(1)$ as an inner product (which you could say it is, in $\ell^2$, for each fixed $\mathbf x,\mathbf y$, but it is not relevant to the context), when it is not really related with the underlying inner product of your Hilbert space.

  • 0
    To clarify, by saying $\{\exp(-i\mathbf{x}^\mathrm{T}\mathbf{t})\}_\mathbf{t}$ is a basis, I mean that if letting $\phi_\mathbf{t}(\mathbf{x})=\exp(-i\mathbf{x}^\mathrm{T}\mathbf{t})$ and $\phi(\mathbf{x})=(\phi_\mathbf{t}(\mathbf{x}))_\mathbf{t}$ (an infinite-dimensional vector), we can have $\langle \phi(\mathbf{x}), \phi(\mathbf{y})\rangle_{\mathcal{H}}=\int C\exp(-\sigma^2\|\mathbf{t}\|^2/2)\phi(\mathbf{x})\overline{\phi(\mathbf{y})}\,\mathrm{d}\mathbf{t}=\exp(-\|\mathbf{x-y}\|^2/(2\sigma^2))$. – 2017-01-25
  • 0
    So, you are re-defining the term "basis". No wonder it doesn't agree with properties of the usual "basis". – 2017-01-25
  • 0
    I am not sure how I implicitly "redefine" the term "basis". I think for $\{\sqrt{\lambda_i}\phi_i(\mathbf{x})\}_i$ and $\{\exp(-i\,\mathbf{x}^\mathrm{T}\mathbf{t})\}_\mathbf{t}$ it looks to me that $\exp(-\|\mathbf{x-y}\|^2/(2\sigma^2))=\langle(\sqrt{\lambda_i}\phi_i(\mathbf{x}))_i,\sqrt{\lambda_i}\phi_i(\mathbf{y})_i\rangle_{\ell_2}=\langle(\exp(-i\,\mathbf{x}^\mathrm{T}\mathbf{t}))_\mathbf{t},(\exp(-i\,\mathbf{y}^\mathrm{T}\mathbf{t}))_\mathbf{t}\rangle_\mathcal{H}$ (defined by the above integral with weight) holds. So it looks to me that both sets can be basis. – 2017-01-25
  • 0
    "Basis" on its own means "linearly independent and spans the whole space". In the context of inner space products, most of the time "basis" means *orthonormal basis*, which is a set where the elements are pairwise orthogonal, of norm one, and total (the two meanings do not agree, by the way). Writing one function as an inner product of two other functions has nothing to do with being a basis. Like I said, you are using the term "basis" to mean what you want and not what all the rest of us use it for. – 2017-01-26
  • 0
    OK, so the "inner product" defined in the above integral form is just an identity and has nothing to do with the (structure of the) $\ell_2$ space spanned by $\{\sqrt{\lambda_i}\phi_i(\mathbf{x})\}_i$. – 2017-01-26
  • 0
    If so, can I say something about the space spanned by $\{\exp(-i\,\mathbf{x}^\mathrm{T}\mathbf{t})\}_\mathbf{t}$, with the inner product $\langle\exp(-i\,\mathbf{x}^\mathrm{T}\mathbf{t}),\exp(-i\,\mathbf{y}^\mathrm{T}\mathbf{t})\rangle=\exp(-\|\mathbf{x-y}\|^2/(2\sigma^2))$? Of course this is another question already, but keywords are welcome. Thank you. (I think $\{\exp(-i\,\mathbf{x}^\mathrm{T}\mathbf{t})\}_\mathbf{t}$ is the basis for Fourier transform but I am not sure about the structure when equipped with the inner product) – 2017-01-26
  • 0
    I think you need to leave your problem for a few minutes and read what an inner product is. An inner product on a vector space $H$ is a **bilinear** (sesquilinear if you use complex coefficients) form $H\times H\to\mathbb R$. This means that $$\langle f+\lambda g,h\rangle=\langle f,h\rangle+\lambda \langle g,h\rangle.$$ On top of this it has to be symmetric ($\langle f,g\rangle = \langle g,f\rangle$, or conjugate in the complex case), we need $\langle f,f\rangle\geq0$ for all $f$, and $\langle f,f\rangle=0$ implies $f=0$. Your "inner product" doesn't even make sense for arbitrary functions. – 2017-01-26