1
$\begingroup$

According to Bishop's book, if your input is a multidimensional vector $x$, then non kernel based basis function models suffer from the curse of dimensionality:

y(x,w) be the linear regression function

But the above picture is based on polynomial basis functions in one dimension, $\phi_i(x)=x^i$.

If we had different basis functions, like sigmoidal $\phi(x)=\sigma((x-\mu_j)/s)$ where $\sigma(a)=(1/(1+exp(-a))$, would the sum still be equal to the above picture or would the combinations between the different dimensions differ?

1 Answers 1

2

You will always have as many terms in your sum, as you have basis-functions in your function space. Since you can combine univariate basis functions, you will not get out of it.

For example: You have monomials in $x$, $y$ and $z$ for degree in every direction up to 5. You will than have $3^6$ basis functions (1, $x$, $y$, $z$, $x^2$, ..., $x^5y^5z^4$, $x^5y^5z^5$), by using every chance of combining these to get new basis functions.

However you could use just a part of these. For example you decide, that the total degree matters. By that you will get less basis functions, less dimensionlity and less accuracy.

  • 0
    Thank you so much flr your answer!2017-01-19