1.5 Total Probability Theorem

Total Probability Theorem

For a finite set of mutually exclusive events $A_1,\dots ,A_N\subseteq \Omega$ ($A_i\cap A_j=\varnothing$ for $i\ne j$) that form a partition of the sample space (i.e. every possible outcome $\omega \in \Omega$ is included in only one event $A_i$ for all $i\in [1,N]$), the total probability theorem states that if $\mathbb{P}[A_i]>0$ for all $i$, the probability of any event $B\subseteq \Omega$ can be expressed as the sum of the joint probability of $B$ and every $A_i$, so that
$\mathbb{P}[B]={\sum }_{i=1}^N\mathbb{P}[A_i\cap B],$
or as the sum of the conditional probabilities of $B$ given $A_i$, weighted by $\mathbb{P}[A_i]$, and vice-versa, such that
$\mathbb{P}[B]={\sum }_{i=1}^N\mathbb{P}[B\mid A_i]\mathbb{P}[A_i]={\sum }_{i=1}^N\mathbb{P}[A_i\mid B]\mathbb{P}[B].$

Intuition

Consider events $A_1,A_2$ that form a partition of the sample space, such that $A_1\cup A_2=\Omega$. Consider another event $B$, which intersects with parts of $A_1$ and $A_2$ in the following figure:

\usepackage{tikz}
 
\begin{document}
 
\begin{tikzpicture}
 
% Draw the probability space rectangle
\draw[thick] (0,0) rectangle (6,4) node[below left]{$\Omega$};
 
% Draw the random squiggly line to divide the rectangle
\draw[thick] 
    (0.5,0) .. controls (1,1) and (1.5,2) .. (2,1.5)
    .. controls (2.5,1) and (3,2.5) .. (3.5,3.2)
    .. controls (4,3.8) and (5,3) .. (5.5,4);
 
% Label A_1 and A_2
\node at (1,3) {$A_1$};
\node at (5,1) {$A_2$};
 
% Draw the event B as a circle
\draw[thick] (3,2) circle (1.5);
\node at (3,2) {$B$};
 
% Label intersections and shade them
% Intersection of A_1 and B
\begin{scope}
  \clip (0,0) rectangle (6,4);
  \clip 
      (0.5,0) .. controls (1,1) and (1.5,2) .. (2,1.5)
      .. controls (2.5,1) and (3,2.5) .. (3.5,3.2)
      .. controls (4,3.8) and (5,3) .. (5.5,4) -- (6,4) -- (6,0) -- cycle;
  \fill[gray, opacity=0.3] (3,2) circle (1.5);
\end{scope}
\node at (3.5,1.5) {$A_2 \cap B$};
 
% Intersection of A_2 and B
\begin{scope}
  \clip (0,0) rectangle (6,4);
  \clip 
      (0.5,0) .. controls (1,1) and (1.5,2) .. (2,1.5)
      .. controls (2.5,1) and (3,2.5) .. (3.5,3.2)
      .. controls (4,3.8) and (5,3) .. (5.5,4) -- (0,4) -- (0,0) -- cycle;
  \fill[gray, opacity=0.7] (3,2) circle (1.5);
\end{scope}
\node at (2.35,2.5) {$A_1 \cap B$};
 
 
\end{tikzpicture}
\end{document}

Since $A_1\cap A_2=\varnothing$ (i.e. $A_1$ and $A_2$ are mutually exclusive), so are $A_1\cap B$ and $A_2\cap B$. Furthermore, these two intersections form a partition of $B$, such that $B=(A_1\cap B)\cup (A_2\cap B)$. Therefore, by the additivity axiom, $\mathbb{P}[B]=\mathbb{P}[A_1\cap B]+\mathbb{P}[A_2\cap B]$. Likewise, since $\mathbb{P}[B\mid A_i]$ is contained within a sample space ${\Omega }_2=A_i$, we must weigh it by $\mathbb{P}[A_i]$, as seen in 1.3.

Examples

Candies in a Jar

You are taking out candies one by one from a jar that contains 10 red candies, 20 blue candies, and 30 green candies.

What is the probability that there are at least 1 blue candy and 1 green candy left in the jar when you have taken out all the red candies? $r_{\max }$, $g_{\max }$, and $b_{\max }$ be the positions of the last red, green and blue candy in the sequence of 60 candies. Then, we are solving for the problem of $\mathbb{P}(r_{\max }<g_{\max }\cap r_{\max }<b_{\max })=\mathbb{P}(r_{\max }<g_{\max }<b_{\max })+\mathbb{P}(r_{\max }<b_{\max }<g_{\max }),$ since the two events are mutually exclusive.

This problem requires a little bit more creative thinking. Let

We abstract from the 60-candy sequence, and take a look at the two events. Since taking out each of the 60 candies is equally likely, and the blue candy is the last one, we can transform the first factor into
$\mathbb{P}(r_{\max }<g_{\max }<b_{\max })=\mathbb{P}(b_{\max }=60)\cdot \mathbb{P}(r_{\max }<g_{\max }\mid b_{\max }=60)=\frac{20}{60}\cdot \frac{30}{40}.$
Note that here $\mathbb{P}(r_{\max }<g_{\max }\mid b_{\max }=60)=\mathbb{P}(r_{\max }<g_{\max })$, since knowing that a blue candy occupies slot 60 tells you nothing about how the red and green candies are interleaved among the remaining slots.

Similarly, the second factor can be written as
$\mathbb{P}(r_{\max }<b_{\max }<g_{\max })=\mathbb{P}(g_{\max }=60)\cdot \mathbb{P}(r_{\max }<b_{\max }\mid g_{\max }=60)=\frac{30}{60}\cdot \frac{20}{30}.$

Therefore, our final answer is
$\mathbb{P}(r_{\max }<g_{\max }\cap r_{\max }<b_{\max })=\frac{1}{4}+\frac{1}{3}=\frac{7}{12}.$

Galton-Watson Branching

You are starting your sourdough with only one cell of yeast. Every hour each cell of yeast undergoes a transformation, which could be any of the three following events: (a) with probability 25% the cell dies, (b) with probability 25% the cell does nothing, (c) with probability 50% the cell replicates itself (so there will be 2 cells).

What is the probability the yeast eventually dies out in the dough?

Rolling Dice