On mathoverflow, Terry Tao says the following:
For instance, one can view a high-dimensional vector space as a state space for a system with many degrees of freedom. A megapixel image, for instance, is a point in a million-dimensional vector space; by varying the image, one can explore the space, and various subsets of this space correspond to various classes of images.
One can similarly interpret sound waves, a box of gases, an ecosystem, a voting population, a stream of digital data, trials of random variables, the results of a statistical survey, a probabilistic strategy in a two-player game, and many other concrete objects as states in a high-dimensional vector space, and various basic concepts such as convexity, distance, linearity, change of variables, orthogonality, or inner product can have very natural meanings in some of these models (though not in all).
This paragraph is saying something that helps a little but I can't quite grasp it.
The picture as a million-dimensional space.. why would you consider it $10^6$-space and not simply 5-space (with x,y,R,G,B as the basis vectors). Shouldn't you go for the simpler representation?
I understand that you can define orthogonality this way, by the inner product of two $10^6$ pictures equalling 0. But what does it mean for 2 pictures to be orthogonal, and why would you want to define that?
Why do we want to use multidimensional space to handle something like the Fourier Transform for example. A soundwave. You can decompose it into 22050-dimensional space using a 22050 point FFT, and MATLAB seems to have a rocking good time with that, and filters work. But mathematically I'm unsure of the reason.