Briefly, RG refers to mathematical tools that allows systematic investigation of the changes of a physical system as viewed at different distance scales. These methods are very important in quantum field theory and statistical mechanics. I'm currently taking a statistical physics course so I'm very curious about this.
Sarada Rajeev, a physicist at the University of Rochester, says that 'a new integral calculus of functions of an infinite number of variables is needed' in order to give these methods a rigorous mathematical foundation.
Update: In a correspondence where I was discussing how this would be different from variational calculus, I got the following reply from Professor Rajeev,
Variational calculus is the differential calculus of an infinite number of variables: paths. The corresponding integral calculus is the "path integral" one pioneered by Weiner and Feynman. Although we have the theory of integration over paths ( the space of functions of one variable (good enough for Stochastic ODEs and Quantum Mechanics) we do not have it for for fields (functions of several variables). Renormalization theory gives us hints on when such a theory is likely to exist. Constructing it would be at least as important as the discovery of Lebesgue integration.
This statement by professor Rajeev is very interesting as it highlights the importance of such mathematical foundations(to him and many other mathematical physicists).