If every vertex in $X$ had an edge to every vertex in $Y$, there would be $rs=r(n-r)$ edges total. That's the maximum possible, and so the actual number of edges, $m$, can be at most $r(n-r)$.
If you already know $n$ and $r$ that's a better bound than $n^2/4$ - for instance, if $r=1$, then we know $m\leq n-1$. The question is implicitly asking you to find a bound that isn't as good as $r(n-r)$, but one that's independent of $r$ (and thus, independent of the structure of $X$ and $Y$). That is, if I said "I have a bipartite graph with $n$ vertices and $m$ edges", but didn't tell you how large each partition was, what would you know about $m$? If you say $m\leq n-1$, I can construct a graph with $X$ large enough to break that bound. So what bound can you give so that, given $m$ and $n$, I cannot construct a bipartite graph that breaks your bound?
To answer this, you need to think about the worst possible case. We know that $m$ could be as large as $r(n-r)$, so what's the largest that $r(n-r)$ could be (given a fixed $n$)? This you can solve with derivatives and get $r=n/2$ (Note: $n/2$ might not be integer, but since $n/2$ is the maximum, any integer value of $r$ will still be less than this). Substituting gives $r(n-r)\leq (n/2)(n-n/2)=n^2/4$.