Absorption Probability Continuous Time Markov Chain

In the mathematical theory of probability, an absorbing Markov chain is a Markov chain in which every state can reach an absorbing state. An absorbing state is a state that, once entered, cannot be left.

Like general Markov chains, there can be continuous-time absorbing Markov chains with an infinite state space. However, this article concentrates on the discrete-time discrete-state-space case.

Formal definition [edit]

A Markov chain is an absorbing chain if^{[1] [2]}

1. there is at least one absorbing state and
2. it is possible to go from any state to at least one absorbing state in a finite number of steps.

In an absorbing Markov chain, a state that is not absorbing is called transient.

Canonical form [edit]

Let an absorbing Markov chain with transition matrix P have t transient states and r absorbing states. Unlike a typical transition matrix, the rows of P represent sources, while columns represent destinations. Then

${\displaystyle P=\left({\begin{array}{cc}Q&R\\\mathbf {0} &I_{r}\end{array}}\right),}$

where Q is a t-by-t matrix, R is a nonzero t-by-r matrix, 0 is an r-by-t zero matrix, and I _r is the r-by-r identity matrix. Thus, Q describes the probability of transitioning from some transient state to another while R describes the probability of transitioning from some transient state to some absorbing state.

Fundamental matrix [edit]

A basic property about an absorbing Markov chain is the expected number of visits to a transient state j starting from a transient state i (before being absorbed). The probability of transitioning from i to j in exactly k steps is the (i,j)-entry of Q ^k. Summing this for all k (from 0 to ∞) yields the fundamental matrix, denoted by N. It can be proven that

${\displaystyle N=\sum _{k=0}^{\infty }Q^{k}=(I_{t}-Q)^{-1},}$

where I _t is the t-by-t identity matrix. The (i,j) entry of matrix N is the expected number of times the chain is in state j, given that the chain started in state i. With the matrix N in hand, other properties of the Markov chain are easy to obtain.^[2]

This fundamental matrix can be thought of as a matrix equivalent of the following geometric series:

${\displaystyle n=\sum _{k=0}^{\infty }q^{k}={\frac {1}{1-q}}=(1-q)^{-1}.}$

Variance on number of visits [edit]

The variance on the number of visits to a transient state j with starting at a transient state i (before being absorbed) is the (i,j)-entry of the matrix

${\displaystyle N_{2}=N(2N_{\operatorname {dg} }-I_{t})-N_{\operatorname {sq} },}$

where N _dg is the diagonal matrix with the same diagonal as N and N _sq is the Hadamard product of N with itself (i.e. each entry of N is squared).

Expected number of steps [edit]

The expected number of steps before being absorbed when starting in transient state i is the ith entry of the vector

${\displaystyle \mathbf {t} =N\mathbf {1} ,}$

where 1 is a length-t column vector whose entries are all 1.

Variance on number of steps [edit]

The variance on the number of steps before being absorbed when starting in transient state i is the ith entry of the vector

${\displaystyle (2N-I_{t})\mathbf {t} -\mathbf {t} _{\operatorname {sq} },}$

where t _sq is the Hadamard product of t with itself (i.e. each entry of t is squared).

Transient probabilities [edit]

The probability of visiting transient state j when starting at a transient state i is the (i,j)-entry of the matrix

${\displaystyle H=(N-I_{t})(N_{\operatorname {dg} })^{-1},}$

where N _dg is the diagonal matrix with the same diagonal as N.

Absorbing probabilities [edit]

Another property is the probability of being absorbed in the absorbing state j when starting from transient state i, which is the (i,j)-entry of the matrix

${\displaystyle B=NR.}$

Alternatively, this probability can also be directly obtained from the (i,j)-entry of ${\displaystyle P^{n}}$ for a large enough value of n. That is:

${\displaystyle (\lim _{n\to \infty }P^{n})_{i,j}=B_{i,j}}$

Examples [edit]

String generation [edit]

Consider the process of repeatedly flipping a fair coin until the sequence (heads, tails, heads) appears. This process is modeled by an absorbing Markov chain with transition matrix

${\displaystyle P={\begin{bmatrix}1/2&1/2&0&0\\0&1/2&1/2&0\\1/2&0&0&1/2\\0&0&0&1\end{bmatrix}}.}$

The first state represents the empty string, the second state the string "H", the third state the string "HT", and the fourth state the string "HTH". Although in reality, the coin flips cease after the string "HTH" is generated, the perspective of the absorbing Markov chain is that the process has transitioned into the absorbing state representing the string "HTH" and, therefore, cannot leave.

For this absorbing Markov chain, the fundamental matrix is

${\displaystyle {\begin{aligned}N&=(I-Q)^{-1}=\left({\begin{bmatrix}1&0&0\\0&1&0\\0&0&1\end{bmatrix}}-{\begin{bmatrix}1/2&1/2&0\\0&1/2&1/2\\1/2&0&0\end{bmatrix}}\right)^{-1}\\[4pt]&={\begin{bmatrix}1/2&-1/2&0\\0&1/2&-1/2\\-1/2&0&1\end{bmatrix}}^{-1}={\begin{bmatrix}4&4&2\\2&4&2\\2&2&2\end{bmatrix}}.\end{aligned}}}$

The expected number of steps starting from each of the transient states is

${\displaystyle \mathbf {t} =N\mathbf {1} ={\begin{bmatrix}4&4&2\\2&4&2\\2&2&2\end{bmatrix}}{\begin{bmatrix}1\\1\\1\end{bmatrix}}={\begin{bmatrix}10\\8\\6\end{bmatrix}}.}$

Therefore, the expected number of coin flips before observing the sequence (heads, tails, heads) is 10, the entry for the state representing the empty string.

The cumulative probability of finishing a game of Snakes and Ladders by turnN

Games of chance [edit]

Games based entirely on chance can be modeled by an absorbing Markov chain. A classic example of this is the ancient Indian board game Snakes and Ladders. The graph on the left^[3] plots the probability mass in the lone absorbing state that represents the final square as the transition matrix is raised to larger and larger powers. To determine the expected number of turns to complete the game, compute the vector t as described above and examine t _start, which is approximately 39.2.

Infectious disease clinic [edit]

The example of infectious disease testing, in either blood products or in medical clinics, is often taught as an example of an absorbing Markov chain.^[4] The public U.S. Centers for Disease Control and Prevention (CDC) model for HIV and for hepatitis B, for example,^[5] illustrates the property that absorbing Markov chains can lead to the detection of disease, versus the loss of detection through other means.

In the standard CDC model, the Markov chain has five states, a state in which the individual is uninfected, then a state with infected but undetectable virus, a state with detectable virus, and absorbing states of having quit/been lost from the clinic, or of having been detected (the goal). The typical rates of transition between the Markov states are the probability p per unit time of being infected with the virus, w for the rate of window period removal (time until virus is detectable), q for quit/loss rate from the system, and d for detection, assuming a typical rate ${\displaystyle \lambda }$ at which the health system administers tests of the blood product or patients in question.

Classical example of HIV or hepatitis virus screening model

It follows that we can "walk along" the Markov model to identify the overall probability of detection for a person starting as undetected, by multiplying the probabilities of transition to each next state of the model as:

${\displaystyle {\frac {p}{(p+q)}}{\frac {w}{(w+q)}}{\frac {d}{(d+q)}}}$ .

The subsequent total absolute number of false negative tests—the primary CDC concern—would then be the rate of tests, multiplied by the probability of reaching the infected but undetectable state, times the duration of staying in the infected undetectable state:

${\displaystyle {\frac {p}{(p+q)}}{\frac {1}{(w+q)}}\lambda }$ .

See also [edit]

• Discrete phase-type distribution
• Absorbing set (random dynamical systems)

References [edit]

1. ^ ^{a b} Grinstead, Charles M.; Snell, J. Laurie (July 1997). "Ch. 11: Markov Chains" (PDF). Introduction to Probability. American Mathematical Society. ISBN978-0-8218-0749-1.
2. ^ ^{a b} Kemeny, John G.; Snell, J. Laurie (July 1976) [1960]. "Ch. 3: Absorbing Markov Chains". In Gehring, F. W.; Halmos, P. R. (eds.). Finite Markov Chains (Second ed.). New York Berlin Heidelberg Tokyo: Springer-Verlag. pp. 224. ISBN978-0-387-90192-3.
3. ^ Based on the definition found in S. C. Althoen; L. King; K. Schilling (March 1993). "How Long Is a Game of Snakes and Ladders?". The Mathematical Gazette. The Mathematical Gazette, Vol. 77, No. 478. 78 (478): 71–76. doi:10.2307/3619261. JSTOR 3619261.
4. ^ results, search (1998-07-28). Markov Chains. Cambridge: Cambridge University Press. ISBN9780521633963.
5. ^ Sanders, Gillian D.; Anaya, Henry D.; Asch, Steven; Hoang, Tuyen; Golden, Joya F.; Bayoumi, Ahmed M.; Owens, Douglas K. (June 2010). "Cost-Effectiveness of Strategies to Improve HIV Testing and Receipt of Results: Economic Analysis of a Randomized Controlled Trial". Journal of General Internal Medicine. 25 (6): 556–563. doi:10.1007/s11606-010-1265-5. ISSN 0884-8734. PMC2869414. PMID 20204538.

External links [edit]

• Wolfram Demonstration Project: Absorbing Markov Chain
• Monopoly as a Markov chain

coleclowed54.blogspot.com

Source: https://en.wikipedia.org/wiki/Absorbing_Markov_chain

Share this post