3
$\begingroup$

I just started learning about Markov processes and got the following homework question.

Classify all the states as recurrent or transient for the Markov chain below

$\begin{matrix} {}&s_1&s_2&s_3&s_4&s_5\cr s_1&1/4 & 3/4 & 0 & 0 & 0\cr s_2&1/2 & 1/2 & 0 & 0 & 0\cr s_3&0 & 0 & 1 & 0 & 0\cr s_4&0 & 0 & 1/3 & 2/3 & 0\cr s_5&1 & 0 & 0 & 0 & 0 \end{matrix}$

I worked through the first few powers of the matrix and found the following:

For n=2 $\begin{matrix} {}&s_1&s_2&s_3&s_4&s_5\cr s_1&7/16 & 9/16 & 0 & 0 & 0\cr s_2&3/8 & 5/8 & 0 & 0 & 0\cr s_3&0 & 0 & 1 & 0 & 0\cr s_4&0 & 0 & 5/9 & 4/9 & 0\cr s_5&1/4 & 3/4 & 0 & 0 & 0 \end{matrix}$

For n=3 $\begin{matrix} {}&s_1&s_2&s_3&s_4&s_5\cr s_1&25/64 & 39/64 & 0 & 0 & 0\cr s_2&13/32 & 19/32 & 0 & 0 & 0\cr s_3&0 & 0 & 1 & 0 & 0\cr s_4&0 & 0 & 19/27 & 8/27 & 0\cr s_5&7/16 & 9/16 & 0 & 0 & 0 \end{matrix}$

From the diagonals, I can figure out that state 3 is recurrent because $\Sigma p_{33}(n)=\infty$

By the same reasoning, states 4 and 5 are transient because $\Sigma p_{44}(n)$ and $\Sigma p_{55}(n)$ are convergent.

First of all, is this method of finding whether the states are recurrent or transient correct? Is there a simpler way to solve the problem? Second, I can't figure out whether states 1 and 2 are transient or recurrent using this method.

Would appreciate if someone can point me in the right direction. Many thanks!

  • 0
    Look at Didier Piau's comment. It fully answers the question.2011-06-28

1 Answers 1

1

Drawing a diagram of the Markov chain with vertices for the states and arrows between states x and y if and only if p(x,y) is positive yields the answer instantaneously.