4
$\begingroup$

Having a defined markov chain with a known transition matrix, rather than to calculate the steady state probabilities, I would like to simulate and estimate them.

Firstly, from my understanding there is a transient/warm-up time we have to cater for, therefore run our markov chain for sufficiently long time in order to BE in a steady state. Any ideas how to estimate that?

Secondly, even if we discard the initial k observations, how do we actually measure the probabilities? I have the following two approaches I would use:

  1. after the initial transient period observe which state we are and register that. Rerun the same simulation and register then which state we are. aftern N runs, take average of each occurences of each state to get approximations. Problems: Bloody too inefficient

  2. After the initial transient time, generate N state transitions and count occurences (WITHOUT RESTARTING). Take average. Problems: Each of N samples are not independent

I was wondering whether you guys could point me into the right direction and spot any flaws in my logic. Thank you in advance for your help.

3 Answers 3

1

If you know that your chain is ergodic then the second approach is the natural and efficient way.

0

I guess both techniques are OK, it's just the questions of how "accurate" you want your results to be - tradeoff between speed and accuracy.

0

Your first question is essentially "How long does it take for a Markov chain to get close to its stationary distribution?" Obviously the answer depends on the structure of the chain in question, and it isn't always easy to estimate just by looking at the chain. There is a lot of research on this question under the name of "mixing times".