I'm wondering if these matrices have a name? (I'm somehow tempted to call them subunitary but it seems to be reserved for something else.)
The matrix $M \in \mathbb{C}^{n\times n}$ is called ..., if all the singular values $\sigma_1,\dots, \sigma_n$ are strictly smaller than 1.
Note that if all the singular values are 1 then $M$ is a unitary matrix. That is why I think it should be called subunitary.
The matrix $M \in \mathbb{C}^{n\times n}$ is called ..., if it fulfills $M = \alpha U$ with $|\alpha|<1, \alpha \in \mathbb{C}$ and $U \in U(n)$ ($U(n)$ is the group of unitary matrices). (maybe it has a name if one allows $|\alpha|=1$) It seems like this space is closed under multiplication. But obviously it is not a group as the $0$ matrix is not invertible.