3
$\begingroup$

I just started learning about Markov processes and got the following homework question.

Classify all the states as recurrent or transient for the Markov chain below

$$\begin{matrix} {}&s_1&s_2&s_3&s_4&s_5\cr s_1&1/4 & 3/4 & 0 & 0 & 0\cr s_2&1/2 & 1/2 & 0 & 0 & 0\cr s_3&0 & 0 & 1 & 0 & 0\cr s_4&0 & 0 & 1/3 & 2/3 & 0\cr s_5&1 & 0 & 0 & 0 & 0 \end{matrix}$$

I worked through the first few powers of the matrix and found the following:

For n=2 $$\begin{matrix} {}&s_1&s_2&s_3&s_4&s_5\cr s_1&7/16 & 9/16 & 0 & 0 & 0\cr s_2&3/8 & 5/8 & 0 & 0 & 0\cr s_3&0 & 0 & 1 & 0 & 0\cr s_4&0 & 0 & 5/9 & 4/9 & 0\cr s_5&1/4 & 3/4 & 0 & 0 & 0 \end{matrix}$$

For n=3 $$\begin{matrix} {}&s_1&s_2&s_3&s_4&s_5\cr s_1&25/64 & 39/64 & 0 & 0 & 0\cr s_2&13/32 & 19/32 & 0 & 0 & 0\cr s_3&0 & 0 & 1 & 0 & 0\cr s_4&0 & 0 & 19/27 & 8/27 & 0\cr s_5&7/16 & 9/16 & 0 & 0 & 0 \end{matrix}$$

From the diagonals, I can figure out that state 3 is recurrent because $$\Sigma p_{33}(n)=\infty$$

By the same reasoning, states 4 and 5 are transient because $$\Sigma p_{44}(n)$$ and $$\Sigma p_{55}(n)$$ are convergent.

First of all, is this method of finding whether the states are recurrent or transient correct? Is there a simpler way to solve the problem? Second, I can't figure out whether states 1 and 2 are transient or recurrent using this method.

Would appreciate if someone can point me in the right direction. Many thanks!

  • 6
    Drawing a diagram of the Markov chain with vertices for the states and arrows between states x and y if p(x,y) is positive would yield the answer instantaneously.2011-06-28
  • 0
    Look at Didier Piau's comment. It fully answers the question.2011-06-28

1 Answers 1