In general, given any sets $X$ and $Y$, a subset $S\subseteq X$, and a function $f:X\to Y$, the function $f|_S:S\to Y$ is the function $f$, where the domain is now considered to be $S$. The subset $S\subset R[x]$ consisting of constant polynomials can be identified with $R$ in the obvious manner, and in fact we usually write $R\subset R[x]$, leaving this identification implicit. So you are correct, $\phi|_R$ means exactly the restriction of $\phi:R[x]\to R$ to the subset of $R[x]$ consisting of constant polynomials.
Hint on how to proceed: show that, for any ring homomorphism $\psi:R[x]\to T$ where $T$ could be any ring whatsoever, the place where any $p\in R[x]$ is sent by $\psi$, namely $\psi(p)$, is determined completely by where elements of $R$ are sent, and where $x$ is sent. That is, if I tell you how $\psi$ acts on elements of $R$, and how $\psi$ acts on $x$, and tell you that $\psi$ is a ring homomorphism, you can figure out what $\psi$ does to any element of $R[x]$.
Note that we are assuming $\phi:R[x]\to R$ doesn't change elements of $R$, and then think about where $x$ is sent under the substitution homomorphism $\phi_a$...