Regarding to physical interpretation of Greens functions, for elliptic operators it makes sense to think of the Poisson equation in electrostatics
$\nabla^2\phi(\vec x)=\rho(\vec x),$
where $\phi(\vec x)$ is the electric potential. Here $\rho(\vec x)$ is the charge density, which you can think of as being composed of point charges localized at positions $\vec y$ with densities described by $\delta(\vec x-\vec y)$. This thought can somewhat tautologically be represented by
$\rho(\vec x)=\int\rho(\vec y)\delta(\vec x-\vec y)d\vec y.$
Now by the superposition principle, which holds for the Mawell equations (or mathematically by the fact that your differential operator is linear), if you know the potential $G(\vec x)$ of a point particle
$\nabla^2G(\vec x)=\delta(\vec x),$
you already know the solution to the full problem. With
$\phi(\vec x)=\int G(\vec x-\vec y)\rho(\vec y)d\vec y,$
which resembles the summing up of all the point potentials, you find that Poissons equations is solved:
$\nabla^2\phi(\vec x)=\int \nabla^2G(\vec x-\vec y)\rho(\vec y)d\vec y =\int \delta(\vec x-\vec y)\rho(\vec y)d\vec y=\rho(\vec x).$
Now what is the potential of the point particle? It's usefull to think of $\nabla^2$ as the divergence of the gradient
$\text{div}(\vec \nabla G(\vec x))=\delta(\vec x).$
The gradient of the potential is the electric field, which is proportional to the force imposed on other point charges. Now what is the point particle force field which has zero divergence but is singular for $\vec x =0$? In three dimensions, the surface area $A$ of a sphere goes with $A\propto r^2$, so if the divergence should be zero, the radial solution should go as $\frac 1 {r^2}$, which just is Coulombs law. Integrating the remaining gradient, we find $G(\vec r-\vec r_0)=\frac{c}{|r-r_0|}.$ Similarly, if you're in two dimensions, then the surface goes with $r$, field must go inverse to that and the integral, i.e. the Greens function goes as $log(r)$.
In physics, very often you think of the delta peak as a source of a pertubation of some field. The differential operator comes from some Lagrangian density which encodes conservation laws and the associated Greens function describes how information porpagates away from the source. The field decays spatially and (in contrast to the Poisson equation with a change densitiy $\rho(\vec x)$) the remaining Laplace equation $\nabla^2\phi(\vec x)=0$ describes free dispersal/propagation of the potential/wave.
All this business is a mayor theme in Field Theories (or 'its applications' like signal processing), where the Operators involve time derivatives. At the source points, there is some interaction and the field gets perturbed and then the information travels away from there. In Quantum Theories, these are "just" propability waves. Basically, if you know your free propagators and how to knot them together using Feynman diagrams, you know the whole theory. A graphic and therefore illustrive such example is the Greens function of the heat equation, where you can literally watch then density dissolve.