Your notation is not completely clear to me (especially with regards to bounded number
).
Markov Decision Process (MDP) is a Markov process (MP) where (probabilistic) control is allowed, that name usually refers to discrete-time processes. Probabilistic control means that at each step you choose just a distribution of the next value from the class of admissible distributions. Again, $\text{MDP} = \text{MP} + \text{probabilistic control}.$
Talking about the reward $R:S\to\mathbb R$ the name for the model I met in the literature (Computer Science as well) is Markov Reward Model (MRM): that is MP + reward function, not necessary with a control in general. The underlying process for MRM can be just MP or may be MDP.
Utility function can be defined e.g. as $U=\sum\limits_{i=0}^nR(X_i)$ given that $X_0,X_1,...,X_n$ is a realization of the process. In the case when the underlying process is uncontrolled, one studies distribution of this function. In the case when control is allowed, optimal control problems are solved. On the other hand, nothing will prevent you from defining $U$ is another form and studying its properties.