I don't know much about the specific use of the Radon transform for tomography, but I can speak to ill-posed inverse problems for medical imaging and geophysics more generally.
Suppose you have spaces $X$ and $Y$ (usually Banach manifolds), and a map between then $F:X \rightarrow Y$. Then the problem is:
Given observed data $y$ and noise $\eta$, solve $y=F(x)+\eta$ for $x$.
For example in geophysics reconstruction, $y$ might be a vector of siesmic measurements, $x$ a spatially varying elasticity coeffieicnt, and $F$ the operator that 1) takes a given elasticity field $x$, solves the relevant elasticity PDE's with that coefficient $x$ and initial conditions given by an earthquake disturbance then 2) evaluates the resultant displacement field at measurement locations.
Three problems are immediately evident:
- What if $F$ is not onto?
- What if $F$ is not 1-1?
- What if $F^{-1}$ is not continuous?
Problem 1 (existence) says there may be no spatial field that fits with your data.
Problem 2 (uniqueness) says that there may be several spatial fields that exactly explain what you observed.
Problem 3 (well-posedness) says that small variations in your measurements (eg, experimental noise, computational roundoff error, inaccuracy of the physical model) could lead to large changes in the field you're inverting for.
Problems 1 and 2 are usually not that big a deal. For problem 1 (non-existence), one generally projects the data onto the range of $F$, then solves $x=F^{-1}(\Pi y)$. The open question here is what projection to use. Alternatively, one might try to find an $x$ that is least bad. The question then is to find a meaningful definition of "least bad". These two approaches are basically equivalent since a definition of "least bad" also defines a projection operator, and vice versa.
Problem 2 (non-uniqueness) is usually either overcome by inverting on an open set on which the solution is locally unique, or is dealt with as a side-effect of dealing with problem 3.
Problem 3 (ill-posedness) is hard and problem-dependent. The standard tool is regularization, and the open question is: what regularization should be used and how can it be justified?
The existence of noise ($\eta$) raises additional issues. In practical computation it is usually used to determine the size of the regularization, and then basically ignored, and a best guess $x$ is calculated deterministically. However there is a whole wonderful probabilistic theory wherein it is crucial. There, given random noise you look for a probability distribution over the space of spatial fields $X$ rather than one particular field. For a good book on the subject, I recommend Tarantola.
A variety of open mathematical questions center around connecting the two approaches (deterministic vs probabilistic). For example, it is known that when inverting for the coefficient in an elliptic PDE with gaussian noise, the deterministic result found when using Tikhonov regularization scaled by the noise is equal to the maximum likelihood point of the probabilistic approach (also, the covariance can be found from the hessian of the deterministic system). The results here are few and far between, and would be of great theoretical and practical importance. The full probabilistic approach is usually computationally intractable - are there deterministic methods that can recover certain key properties of the full probability distribution (eg, moments)?
Finally, there are no end of open problems based around finding fast numerical algorithms to solve the deterministic system computationally. (what I'm doing for my phd would fall under this category)