2
$\begingroup$

I am working though the book of J. Norris, "Markov Chains" as self-study and have difficulty with ex. 2.7.1, part a.

The exercise can be read through Google books. My understanding is that the probability is given by (0,i) matrix element of exp(t*Q). Setting up forward evolution equation leads to differential difference equation which I was hinted admits no closed form solution (see my question on mathoverflow).

I obviously need an nudge in a right direction, and the exercise must admit a simple solution. Any directions are much appreciated.

1 Answers 1

2

It seems you misread the question. One is not interested in the probability of the event $[X_t=i]$ for a given time $t$ but in the probability of the event that there exists a time $t$ such that $X_t=i$.

Here is a nudge...

One approach is to consider the sequence $(Y_n)_n$ of the successive different states visited by $(X_t)_t$. You could try to show that $(Y_n)_n$ is a discrete Markov chain, to compute its transition probabilities (these are very simple and, hint, they do not depend on the numbers $q_i$) and, finally, to compute the probability that there exists an integer time $n$ such that $Y_n=i$.

  • 0
    Thanks ! Following the book of Norris, the associate discrete time Markov process is 1D random walk with probabilities`$\lambda$ and $\mu$ of up-moves and down-moves, respectively, and is independent of $q_i$. Let $i=2k$, this state can be reached in $n=2m$ number of steps with $m>abs(k)$. The probability is $\mathbb{P}(Y_n=i) = \binom{n}{m+k} \lambda^{(m+k)}*\mu^{(m-k)}$. So if $i>0$ and $1>\lambda>0$ the probability of reaching $i$ in $n$ steps is non-zero for every $n>=abs(i)$.2011-05-18
  • 0
    Sure, but this is not an answer to question (a).2011-05-18
  • 0
    Am I correct in looking at this as a random variate which is the difference of independent geometric distributions ? Then the probability is $\lambda \mu^{i+1}/(1-\lambda \mu)$ for positive $i$, and $\mu \lambda^{1-i}/(1-\lambda \mu)$ for non-positive.2011-05-18
  • 0
    You could explain what object you think is *the difference of* [two] *independent geometric* [random variables], I see none.2011-05-19
  • 0
    @Didier I realized it was a mistake. I should compute $\sum_{n \ge 0} \mathbb{P}_0(T_i = n)$, i.e. sums of first time passage probabilities from $0$ to $i$. Let $P_{0,i}(s)$ is pgf for transitioning from $0$ to $i$ in $n$ steps, then what I am after is $lim_{s->1} (P_{0,i}(s)- \delta_{0,i})/P_{0,0}(s)$. Assuming $i$ is even and positive, I have computed my answer to be $\lambda^i F(i/2,(1+i)/2,i+1,4 \lambda \mu)$. Does this make sense ?2011-05-19
  • 1
    No this does not. OK, let us go back to my post, shall we? Did you compute the transition probabilities of $(Y_n)$? Next, fix $i$, for example $i\ge1$. Can you compute $P_0(T_i, for a given $n\le-1$? Next, what happens when $n\to-\infty$?2011-05-19
  • 0
    @Didier For all integer $i$ the non-zero transition probabilities are $\mathbb{P}(Y_n = i +1 | Y_{n-1} = i) = \lambda$ and $\mathbb{P}(Y_n = i-1 | Y_{n-1} = i) = \mu$. Then $\mathbb{P}_0( T_i < T_{-n}) = (\alpha^n - 1)/(\alpha^{n+i}-1)$, where $\alpha = 1/\lambda -1$. The limit of large $n$ is different depending on $\lambda<1/2$, $\lambda =1/2$ or $\lambda>1/2$. Is my analysis correct this time ? Sorry for being hopelessly lost, and many thanks for your help. I would have to think why my approach with first passage times did not work.2011-05-19
  • 1
    @Didier A-ha. Many thanks for your help. It turns out my previous analysis with first passage times is also correct, as the particular Gauss function simplifies so that $\lambda^i F(i/2,(i+1)/2,i+1,4 \lambda \mu)$ equals $(1/\lambda-1)^i$ for $0<\lambda <1/2$ and equals $1$ for $1>\lambda >1/2$. Hurray! And thanks again!2011-05-19
  • 0
    @Sasha how did you find the probability $P_0(T_i? I dont see how to do this.2014-12-14