For what it's worth, here's the outline using the "derivative approach". Since you tagged this as analytic geometry, I would think the splendid answers by Americo and Didier would be the better approach.
Rewriting your equation of the line for $B\ne0$ gives: $$ y={-Ax-C\over B}. $$ (If $B=0$, the line is vertical, and the problem is simple.)
If the point $(x, {-Ax-C\over B})$ is on the line, its distance to $(x_1,y_1)$ is $$ \tag{1}D=\sqrt{ (x_1-x)^2 + {\Bigl(y_1- { \textstyle{-Ax-C\over B}}\Bigr)^2 }}. $$
You want to find the smallest value of $D$. This is equivalent to finding the smallest value of $$ P(x)=D^2= (x_1-x)^2 + {\Bigl(y_1- { \textstyle{-Ax-C\over B}}\Bigr)^2 } $$
Now, it should be obvious that the distance from a point $p$ on the line to $(x_1,y_1)$ is big, if $p$ is "on an extreme end of the line". So, the minimum distance is attained at a point where $P'(x)=0$.
So, you need to find $P'(x)$, then solve $P'(x)=0$. I'll leave this for you...
You should only get one solution to $P'(x)=0$. The minimum of $P$, and hence the minimum of $D$, would then have to occur at this point . If you then plug this solution into (1), your formula will result after a bit of simplification.