You want $P^{-1}$ to come as close as possible to being an inverse of $A$, but you also want $P^{-1} b$ to be easy to compute. These desires conflict with each other. Two extreme cases are $P = A$ and $P = I$. In the first case, the resulting system $x = P^{-1}b$ is trivial to solve, but computing $P^{-1} b$ is as difficult as solving the original system. In the other case ($P = I$), $P^{-1} b = b$ is trivial to compute, but the new system $P^{-1} A x = P^{-1} b$ is no easier to solve than the original system. What we want is something in between these two extreme cases.
Trefethen's really good book Numerical Linear Algebra has a survey of popular preconditioners at the end. They say that finding a good preconditioner is difficult, and usually depends on your specific application.