6.4.1. Markov Models

Markov Model

A Markov Model is an abstract probabilistic model that represents a system evolving through time where the future depends only on the present, not on the past. This is known as the Markov property (or memoryless property), formally expressed as
$P(X_t|X_0,X_1,\dots ,X_{t-1})=P(X_t|X_{t-1}),$
such that if each state $X_t$ is directly observable (i.e., can be measured without uncertainty), the model reduces to a Markov Chain. In the space of probabilistic models, a Markov Model is best explained as a special case of a Bayes Net with a repeated, time-indexed structure.

Each variable $X_t$ captures the system’s state at time $t$. To fully specify the model, we need:

• An initial distribution $P(X_0)$, describing where the process begins.
• A transition model $P(X_{t+1}|X_t)$, describing how it evolves between timesteps.

Using these, the joint distribution across $n$ variables expands by the chain rule and the Markov assumption as
$P(X_0,X_1,\dots ,X_n)=P(X_0){\prod }_{i=0}^{n-1}P(X_{i+1}|X_i).$

To compute the distribution of the variable at a future timestep without expanding the entire joint distribution, we use the Mini-Forward Algorithm, which recursively advances the belief state through time:
$P(X_{t+1})={\sum }_{x_t}P(x_t,X_{t+1})={\sum }_{x_t}P(X_{t+1}\mid x_t)P(x_t).$

This propagation continues iteratively, starting from $P(X_0)$, to obtain the marginal at any future $t$. Over time, the model may converge to a stationary distribution, satisfying
$P(W_{t+1})=P(W_t),$
which represents the equilibrium probabilities of each state once the process stabilizes.