The approach proposed in the OP may be good, as it gives a waste $W $ of
$$W=(m \mod{a}) \cdot n + (n \mod { b}) \cdot m - (m \mod { a} ) ( n \mod {b}) $$
that in most cases is distributed along two adjacent sides. For example, let us take a $120\times 100$ rectangle to be filled with $11\times 7$ rectangles. Using the approach of the OP, the waste is
$$W=(120 \mod 11) \cdot 100 + (100 \mod 7) \cdot 120 - ( 120 \mod 11 )\cdot ( 100 \mod 7) \\ =10 \cdot 100 + 2 \cdot 120 - 10 \cdot 2=320$$
However, I believe that there could be a potentially better strategy. We could assess whether one between $m $ and $n $ can be expressed as $ax+by $, where $x$ and $y $ are integers. If this is the case, we can leave an incomplete region only along a single side of the large rectangle (instead of two). For instance, in the example above, we can note that $120$ can be expressed as $$120=99+21=9 \cdot 11 +3\cdot 7 \,\,\,\,$$ So, dividing the sides equal to $120$ in two parts measuring $99$ and $21$, we can divide the initial rectangle in two rectangles ( $99 \times 100$ and $21 \cdot 100$). The first one can be filled beginning from the side equal to $99$, placing a first sequence of $9$ small rectangles (oriented as $11 \cdot 7$) to cover a $99 \times 7 $ area, then another sequence to achieve a $99 \times 14 $ area, and so on, arriving to a $99 \times 98$ area. This leaves a partial waste of $2 \cdot 99=198$. The second one can be filled beginning from the side equal to $21$, placing a first sequence of $3$ small rectangles (oriented as $7 \cdot 11$) to cover a $21 \times 11$ area, then another sequence to achieve a $21 \times 22 $ area, and so on, arriving to a $21 \times 99 $ area. This leaves a partial waste of $1 \cdot 21=21$. The total waste is $198+21=219$. Generalizing, if we can set $m=ax+by $, the waste with this approach is $$W=ax \cdot (n \mod b) + by \cdot (n \mod a) $$
There are several points to be highligted in this solution. First, this approach should be attempted by considering all possibilities of expressing $m $ or $n $ as $ax+by $, and choosing the one that minimizes the waste. In the example used before, other two possibilities could be to set $$m=120=2 \cdot 11 + 14 \cdot 7\,\,$$ or $$n=100=4 \cdot 11 + 8 \cdot 7\,\, $$ Using the same procedure, it can be easily shown that these choices lead to wastes of $142$ and $604$, respectively. Among all three possibilities, that leaving $142$ gives the best solution. Second, even if this approach allows a nearly full packing along three sides, it does not always produce a minor waste than the solution suggested in the OP. The case $W=604$ described above is an example of this. Also note that the solution of the OP could be applied after rotating the initial rectangle by $90^o $ to get
$$W=(100 \mod 11) \cdot 120 + (120 \mod 7) \cdot 100 - ( 100 \mod 11 )\cdot ( 120 \mod 7) \\ =1 \cdot 120 + 1 \cdot 100 - 1 \cdot 10=210$$
which is a better result than that initially obtained. Lastly,
note that a "complete" packing (with no waste) can be obtained only in two scenarios, that is to say 1) $a $ divides $m $ and $b $ divides $n $ (or alternatively $a $ divides $n$ and $b $ divides $m $), 2) both $a $ and $b $ divide one (the same) of $m $ and $n $, for example $m $, and the other (in this example $n $) is of the form $n=ax+by \,\,\,$ with $x,y$ integers.