Partial solution . . .
I'll do the case $R(3,n)$.
Claim: $R(3,n) = 3 + n$, for all $n \ge 3$.
For simplicity of discussion, instead of cells with either red dots or "other" dots, we'll just use cells either containing a dot, or else empty.
First we show $R(3,n) \le 3 + n$, for all $n \ge 3$.
First, consider the case $n = 3$.
Suppose $R(3,3) > 6$.
Thus, suppose we have a rectangle-free $3\,{\times}\,3$ matrix containing at least $7$ dots. Since we have only $3$ rows, some row must have more than two dots, hence must have $3$ dots (i.e., it's "fully filled"). If any of the other rows contains at least two dots, those two dots together with the corresponding dots in the fully filled row would yield a rectangle. Thus, the two rows other than the fully filled row must each have at most one dot. But then the total number of dots is at most $5$, contradiction.
It follows that $R(3,3) \le 6$.
Next, consider the case $n = 4$.
Suppose $R(3,4) > 7$.
Thus, suppose we have a rectangle-free $3\,{\times}\,4$ matrix containing at least $8$ dots.
Note if a matrix with dots is rectangle free, it's still rectangle free if any of the rows are permuted, or if any of the columns are permuted.
Since we have only $3$ rows, some row must have more than two dots, hence must have at least $3$ dots. By an appropriate permutation of the rows and columns, we can assume the first row has $3$ dots in its first $3$ cells (left to right). The $6$ cells below those $3$ dots can have at most one dot in each row, hence must have one dot in each row, else you can't reach $8$. But then the right-most column must have $3$ dots, which yields two rectangles, contradiction.
It follows that $R(3,4) \le 7$.
Proceed by induction on $n$ . . .
Thus, suppose $R(3,n) \le 3 + n$, for some $n \ge 4$.
Our goal is to show $R(3,n+1) \le 3 + (n + 1) = 4 + n$.
Suppose $A$ is a $3\,{\times}\,(n+1)$ rectangle-free matrix with $d$ dots.
If we delete any column from $A$, the new matrix is a $3\,{\times}\,n$ rectangle-free matrix, hence, by the inductive hypothesis, contains at most $3 + n$ dots.
There are $n + 1$ choices for the deleted column, and in each case, the new matrix contains at most $3 + n$ dots. If we count the total number of dots in all of these new matrices, the count is at most $(n + 1)(3 + n)$. But for this total, each of the dots in the original matrix $A$ is counted exactly $n$ times, hence
$$d \le \frac{(n + 1)(3 + n)}{n} = \frac{3}{n} + 4 + n < 5 + n$$
But $d < 5 + n \implies d \le 4 + n$.
It follows that $R(3,n+1) \le 4 + n$, which completes the induction.
Thus, we've shown $R(3,n) \le 3 + n$, for all $n \ge 3$.
Next we show $R(3,n) \ge 3 + n$, for all $n \ge 3$.
It suffices to construct a $3\,{\times}\,n$ matrix with $3 + n$ dots which is rectangle-free (i.e., for which no $4$ dots form a rectangle with horizontal or vertical sides).
Thus, consider the $3\,{\times}\,n$ matrix (stars represent dots)
$$
\begin{bmatrix}
& *& *& *& \cdots& *\\
*& *& & & & \\
*& & *& & & \\
\end{bmatrix}
$$
where the first row has dots in all but the first column, and each of the other two rows contains two dots, as shown.
It's clear that the matrix has $3 + n$ dots, and is rectangle-free.
It follows that $R(3,n) \ge 3 + n$, for all $n \ge 3$.
Combining the results, we have $R(3,n) = 3 + n$, for all $n \ge 3$, as was to be shown.
Notes:
- Using a much simpler argument, one can show $R(2,n) = 1 + n$, for all $n \ge 2$.
- By arguments similar to those used to determine $R(3,4)$, one can show that $R(4,4) = 9$.
- Thus, it might seem reasonable to conjecture that $R(m,n) = (2m - 3) + n$, for all $n \ge m$.
- Unfortunately, based on the link http://oeis.org/A072567 posted by Matthew Conroy, it's clear that the conjecture I proposed above is false. Moreover, it seems likely that no closed form expression for $R(m,n)$ is currently known.
