An orthogonal matrix refers to a matrix whose rows and columns are orthonormal. This is a key property of orthogonal matrices, one which ultimately requires these matrices to be square.
Suppose $A$ is rectangular matrix ($n > m$) with row and column vectors which are [a] non-zero and [b] orthogonal to one another. We know,
- Orthogonal vectors are also linearly independent.
- The row rank of $A$ equals the column rank of $A$, $\textrm{rank}(A') = \textrm{rank}(A)$.
Then, (1) and (2) together suggest $\textrm{rank}(A') = \textrm{rank}(A)$, or $m = n$. But this a contradiction.
Restricting the rows or columns to be orthogonal and non-zero is a departure of sorts. A semi-orthogonal matrix $B$ is a non-square matrix with real entries having the property that either (1) $BB' = I_m$ or (2) $B'B = I_n$, with the respective true case representing an orthonormal basis.
The case you speak of, a matrix whose rows or columns are orthogonal (not orthonormal), could be described as a semi-orthogonal matrix under a scaling transformation.