If $M$ and $N$ are $R$-modules, let me write $M\odot N$ for the $R$-module whose underlying vector space is just $M\otimes N$ and whose $R$-module structure is defined as in your question.
Notice that if $M$, $M'$ and $N$ are $R$-modules, the $R$-module $(M\oplus M')\odot N$ is isomorphic to $(M\odot N)\oplus(M'\odot N)$. It follows that to prove that
if $M$ is a free $R$-module and $N$ is any $R$-module, then $M\odot N$ is a free $R$-module
it is enough to show that
for all $R$-modules $N$, the $R$-module $R\odot N$ is free.
So fix an $R$-module $N$, let $\lambda_N:R\otimes N\to N$ be the map induced by the multiplication, and consider the $K$-linear map $\beta:R\otimes N\to R\otimes N$ which is the composition $R\otimes N\xrightarrow{\quad\alpha\otimes1_N\quad}R\otimes R\otimes N\xrightarrow{\quad1_R\otimes\lambda_N\quad}R\otimes N$
An easy verification will show that $\beta$ is an $R$-module homomorphism $R\odot N\to R\otimes N$, when the codomain is endowed with its structure of $R$-module coming from $R$, that is, such that $r\cdot s\otimes n=rs\otimes n$ for all $r$, $s\in R$ and all $n\in N$.
I claim that $\beta$ is an isomorphism. To show this, consider the unique homomorphism of rings $S:R\to R$ such that $S(x_i)=-x_i$ for all $i$, and let $\gamma:R\otimes N\to R\otimes N$ be the composition $R\otimes N\xrightarrow{\quad\alpha\otimes1_N\quad}R\otimes R\otimes N\xrightarrow{\quad1_R\otimes S\otimes 1_N\quad}R\otimes R\otimes N\xrightarrow{\quad1_R\otimes\lambda_N\quad}R\otimes N$ A verification, which I'll omit here, shows that $\gamma$ and betta are inverse isomorphisms.
N.B. All this probably looks a bit magical, but it is in fact part of a general theory. The ring $R$ is in fact a Hopf algebra whose comultication is your map $\alpha$ and whose antipode is the map $S$ I defined above. It is true, at this level of generality, that the diagonal tensor product of two $R$-modules (this is the usual name for the construction of what I wrote $\odot$ above) one of which is free is free. This is very easy to prove —see, for example, the classical book on Hopf Algebras by Sweedler, which either states and proves this or, after the first few chapters, puts you in a position where my above argument becomes natural for you :)