In some cases, we could take the supremum--though not in every case, consider a line in the plane, in which we can find points as far from a given point off of the line as we care to--but even when we can, we'd generally prefer to look at the infimum. Again, it is probably useful to consider a line in the plane. When we wish to know how far a given point is from a given line, we find the point on the line that is the least far from the given point. Looking at arbitrary sets, we can't always deal with minima, and instead deal with infima.
For example, consider the point $(2,0)$ in the plane, and the set $A=\{(x,y):x^2+y^2<1\}$. In this case, $A$ has no point closest to $(2,0)$, but the infimum of the distances between the points of $A$ and $(2,0)$ is $1$.
As for why we do this, I know it comes in handy in many proofs. We can further generalize to distances between two given sets, which allows us to separate sets (of positive distance from each other) by disjoint open sets--which, again, is very useful in many proofs.