Let $\{X_n, n\geq 1\}$ be a finite state homogenous Markov chain with states $i = 1, \ldots, N$ . Let $g$ denote a function which returns out a reward for any given state of the Markov chain.
Let $R_K(i)$ denote the total reward after $K$ transitions of the Markov chain assuming that the chain starts at $X_0 = i$ so that:
$R_K(i) = \sum_{k=1}^K g(\ X_k ) $ given that $X_0 =i$ (not otherwise sure how to incorporate this in the notation).
Clearly for $K \geq 2$
$R_K(i) = R_{k-1}(i) + g(X_k)$
I am wondering if we can say anything about the dependence of $R_{K-1}(i)$ and $g(X_K)$. I am fairly sure that these terms should not be independent since $R_{K-1}(i)$ depends on $X_{K-1}$, and the values of $X_{K-1}$ and $X_K$ are not independent for a generic Markov chain.
Ultimately, the reason why I am asking is because I need to calculate $\mathbb{E}[R_k(i) \ | \ X_0 = i]$ and $\text{Var}[R_k(i) | \ X_0 = i]$ using some kind of recursive formulation but cannot see how to proceed without assuming that these terms are independent / understanding how they depend on one another.