As stated the assertion is incorrect. (See below)
Take $R=\mathbb{Z}$, $S=\mathbb{Z}/6\mathbb{Z}$ (a torsion module).
Note that for a rational prime $p$ and $x\in S$, $p^nx = 0$ if and only if $6|p^nx$. In particular, we must either have $x=0$, or $x=3$ and $p=2$; or $x=2$ or $x=4$ and $p=3$. So we have: $S_P = \left\{\begin{array}{ll} S & \text{if }P=\langle 0\rangle;\\ \{0,3\} &\text{if }P=\langle 2\rangle;\\ \{0,2,4\} & \text{if }P=\langle 3\rangle;\\ \{0\} &\text{otherwise.} \end{array}\right.$
Now, $\mathrm{Hom}_R(S,S)$ has six elements (one each for the possible images of the generator of $S$). And $|\mathrm{Hom}_R(S_P,S_P)| = \left\{\begin{array}{ll} 6 & \text{if }P=\langle 0\rangle;\\ 2 &\text{if }P=\langle 2\rangle;\\ 3 &\text{if }P=\langle 3\rangle;\\ 1 &\text{otherwise.} \end{array}\right.$
So $\left|\prod_{P\text{ prime}}\mathrm{Hom}_R(S_P,S_P)\right| = 6\times 2 \times 3 = 36\neq 6 = \left|\mathrm{Hom}_R(S,S)\right|,$ and I still don't see how they can be isomorphic, even if $S$ is torsion and $R$ is a PID.
The OP notes he's taken it from Rotman's book. The problem is from the section on modules over PIDs, so it is clear that $R$ is meant to be a PID, not merely a domain. In addition, the problem takes the product over all $P$-primary components of $S$, and the definition of $P$-primary explicitly excludes the zero prime. So, what we really have is:
Let $R$ be a PID. Let $S$ be a torsion $R$-module, and for each nonzero prime $P=(p)$ of $R$, let $S_P$ be the $P$-primary component of $S$, $S_P = \{x\in S\mid p^nx=0\text{ for some }n\in\mathbb{N}\}.$ Then $\mathrm{Hom}_R(S,S) = \prod_{P\text{ nonzero prime}} \mathrm{Hom}_R(S_P,S_P).$
To see this, simply use the fact that since $S$ is torsion, then $S$ is the sum of its $P$-primary components, $S = \bigoplus_{P\text{ nonzero prime}} S_P.$Take $R=\mathbb{Z}$, $S=\mathbb{Z}/4\mathbb{Z}$. Then $\mathrm{Hom}_R(S,S)$ has four elements. If $P=(0)$, then $S_P = S$; if $P=(2)$, then $S_P=\{0,2\}$. If $P$ is neither of these, then $S_P=\{0\}$. $\mathrm{Hom}_R(S_2,S_2)$ has two elements. So the product on the right has $8$ elements in it, while $\mathrm{Hom}_R(S,S)$ has $4$, so the two cannot be isomorphic. So in fact, any module homomorphism from $S_P$ to $S$ will have image in $S_P$.
A map form a direct sum $\oplus M_i$ to a module $M$ is equivalent to a family of maps $f_i\colon M_i\to M$.
To see this, simply use the universal property of the direct sum: if $\iota_j\colon M_j\to \oplus M_i$ is the canonical embedding of $M_j$ into the direct sum, then for any family of homomorphisms $\f_j\colon M_j\to M$ there exists a unique homomorphism $f\colon \oplus M_i\to M$ such that $f_j = f\circ \iota_j$; this shows that for every element of $\prod\mathrm{Hom}_R(M_j,M)$ there is a unique corresponding element of $\mathrm{Hom}_R(\oplus M_i,M)$. Conversely, given any $f\colon\oplus M_i\to M$, we obtain a family of maps $f_j\colon M_j\to M$ by letting $f_j = f\circ\iota_j$ (equivalently, thinking of $M_j$ as a submodule of $\oplus M_i$, then $f_j$ is simply the restriction of $f$ to $M_j$). This defines a map $\mathrm{Hom}_R(\oplus M_i,M)$ to $\prod\mathrm{Hom}_R(M_j,M)$. It is now easy to see that the uniqueness clause of the universal property ensures that the two constructions are inverses of each other, proving the isomorphism. (This isomorphism is in fact categorical: for any category $\mathcal{C}$, if $A_i$ are objects and $\coprod A_i$ is their coproduct, then the elements of $\mathcal{C}(\coprod A_i,B)$ corresponds to families $\{f_i\}_{i\in I}$ with $f_i\in\mathcal{C}(A_i,B)$).
So $\begin{align*} \mathrm{Hom}_R(S,S) &= \mathrm{Hom}_R(\oplus S_P, S)\\ &\cong \prod \mathrm{Hom}_R(S_P, S)\\ &=\prod\mathrm{Hom}_R(S_P,S_P), \end{align*}$ since, as discussed, any map from $S_P$ to $S$ actually has image in $S_P$.