Bayes' Rule
For a finite set of mutually exclusive events ( for ) with nonzero probabilities ( for all ) that form a partition of the sample space, Bayes’ rule states that for any event with nonzero probability (),
Intuition
In its essence, Bayes’ rule is just a substitution. To see why, let’s start from the basics. By the law of conditional probability, we have:
Then:
- Again by conditional probability, we have .
- By total probability, we have .
Therefore,
\mathbb{P}[A_{i} \mid B] &= \frac{\mathbb{P}[A_{i} \cap B]}{\mathbb{P}[B]} \\ &= \frac{\mathbb{P}[B \mid A_i]\mathbb{P}[A_i]}{\mathbb{P}[B]} \\ &=\frac{\mathbb{P}[B \mid A_i]\mathbb{P}[A_i]}{\sum^{N}_{j=1}\mathbb{P}[B \mid A_j]\mathbb{P}[A_j]}. \end{align}$$ However, Bayes' theorem is **crucial for Bayesian inference**; a method of statistical inference with the goal of **belief updating**, or updating the probability of a hypothesis as more information is taken into account. For example, we may estimate the **prior probability** $\mathbb{P}[A]$ of an event $A$, and then update the probability to $\mathbb{P}[A \mid B]$, known as the **posterior probability**, once we know that event $B$ has occurred. Real-world data is often conditional rather than joint or marginal, and Bayes' rule helps bridge the gap between posterior and prior beliefs. # Examples >[!example] Pharmaceutical Company > >A pharmaceutical company is marketing a new test for a medical disorder. It has the following probabilities: >1. When applied to an affected person, the test is positive in 90% of the cases and negative in 10% (**false negatives**). >2. When applied to a healthy person, the test is negative 80% of the time and positive 20% of the time (**false positive**). > >Suppose that the incidence of the disease in the U.S. is 5%; this is our prior knowledge. When a person tests positive, what is the updated probability that they have the disease? > >Let A be the event that the person is affected, and let B be the event that the person tests positive. Then, >$$ >\begin{alignat}{0} >\mathbb{P}[A]=0.05 \\ >\mathbb{P}[B|A]=0.90 \\ >\mathbb{P}[B|\bar{A}]=0.20 \end{alignat} >$$ >Since 90% of affected people test positive, and so do 20% of healthy people ($B|\bar{A}$). > >We want to calculate $\mathbb{P}[A|B]$. We proceed as follows: >$$\mathbb{P}[A|B]=\frac{\mathbb{P}[A \cap B]}{\mathbb{P}[B]}=\frac{\mathbb{P}[B|A]\mathbb{P}[A]}{\mathbb{P}[B]}$$ >We only need to find $\mathbb{P}[B]$. How? Well, we can picture $\mathbb{P}[A|B]$ in the following graph of $\Upomega$: >```tikz >\begin{document} >\begin{tikzpicture}[domain=0:4] >\draw[] (0, 0) ellipse (4 and 2) node[left]{}; >\draw[] (-1, 0) ellipse (2 and 1) node[below]{$A$}; >\draw[] (1.25, 0) circle (1) node[below]{$B$}; >\draw[->] (0.6,0) -- (0.4,-1.25) node[below]{$A \cap B$}; >\draw[->] (1.25,0) -- (2,-1) node[below]{$\bar{A} \cap B$}; >\end{tikzpicture} >\end{document} >``` >This means that $\mathbb{P}[B]=\mathbb{P}[A \cap B]+\mathbb{P}[\bar{A} \cap B]$. Applying the rules of conditional probability, we have: >$$ >\begin{equation}{} >\begin{aligned} >\mathbb{P}[B] &= \mathbb{P}[A \cap B]+\mathbb{P}[\bar{A} \cap B]=\mathbb{P}[B|A]\mathbb{P}[A]+\mathbb{P}[B|\bar{A}]\mathbb{P}[\bar{A}]= \\ &= \mathbb{P}[B|A]\mathbb{P}[A]+\mathbb{P}[B|\bar{A}](1-\mathbb{P}[A]) >\end{aligned} >\end{equation} >$$ >Finally, combining the two equations, we have: >$$\mathbb{P}[A|B]=\frac{\mathbb{P}[B|A]\mathbb{P}[A]}{\mathbb{P}[B|A]\mathbb{P}[A]+\mathbb{P}[B|\bar{A}](1-\mathbb{P}[A])}$$ >[!example] Skittles > >Your roommate was having Skittles for lunch and they offer you some. There are five different colors in a bag of Skittles: red, orange, yellow, green, and purple, and there are 20 of each color. You know your roommate is a huge fan of the green Skittles. With probability $1/2$ they ate all of the green ones, with probability $1/4$ they ate half of them, and with probability $1/4$ they only ate 5 green ones. >>[!example]- If you take a Skittle from the bag, what is the probability that it is green? >>Let $A$ be the event that all green skittles are eaten, and let $H$ be half and $Q$ be five. Per total probability, it holds that >>$$\begin{align}P(G) &= P(G \mid A) P(A) + P(G \mid H) P(H) + P(G \mid Q) P(Q) \\ &= 0 \frac{1}{2} + \frac{10}{90} \frac{1}{4} + \frac{15}{90} \frac{1}{4} \\ &\approx 0.0673.\end{align}$$ > >>[!example]- If you take two Skittles from the bag, what is the probability that at least one is green? >>It holds that $P(G_{1} \cup G_{2}) = 1 - P(\bar{G_{1}} \cap \bar{G_2})$. Per total probability, >>$$\begin{align}P(\bar{G_{1}} \cap \bar{G_2}) &= P(\bar{G}_{1} \cap \bar{G}_{2} \mid A) P(A) + P(\bar{G}_{1} \cap \bar{G}_{2} \mid H) P(H) + P(\bar{G}_{1} \cap \bar{G}_{2} \mid Q) P(Q) \\ &= 1 \cdot \frac{1}{2} + \frac{80 \cdot 79}{90 \cdot 89} \cdot \frac{1}{4} + \frac{80 \cdot 79}{95 \cdot 94} \cdot \frac{1}{4} \\ &\approx 0.874. \end{align}$$ > >>[!example]- If you take three Skittles from the bag, what is the probability that they are all green? >>Again, total probability dictates that >>$$\begin{align} >>P(G_{1} \cap G_{2} \cap G_{3}) &= P(G_{1} \cap G_{2} \cap G_{3} | A) P(A) + \ldots \\ >>&= 0 \cdot \frac{1}{2} + \frac{10 \cdot 9 \cdot 8}{90 \cdot 89 \cdot 88} \cdot \frac{1}{4} + \frac{15 \cdot 14 \cdot 13}{95 \cdot 94 \cdot 93} \cdot \frac{1}{4} \\ >>&\approx 0.00108. >>\end{align}$$ > >>[!example]- If all three Skittles you took from the bag are green, what are the probabilities that your roommate had all of the green ones, half of the green ones, or only 5 green ones? >>Let $G_3$ be the event that all 3 Skittles are green. Per Bayes' Rule, it holds that >>$$P(A \mid G_{3}) = \frac{P(G_{3} \mid A) P(A)}{G_{3}} = \frac{0 \cdot 1/2}{0.00108} = 0,$$ >>$$P(H \mid G_{3}) = \frac{P(G_{3} \mid H) P(H)}{G_{3}} = \frac{\frac{10 \cdot 9 \cdot 8}{90 \cdot 89 \cdot 88} \cdot 1/4}{0.00108} \approx 0.237,$$ >>and finally, >>$$P(Q \mid G_{3}) = \frac{P(G_{3} \mid Q) P(Q)}{G_{3}} = \frac{\frac{15 \cdot 14 \cdot 13}{95 \cdot 94 \cdot 93} \cdot 1/4}{0.00108} \approx 0.763$$ > >>[!example]- If you take three Skittles from the bag, what is the probability that they are all the same color? >>By total probability, >>$$P(C_{3})$$ CORRECT FOR LAST SUBPOINT: - P(C_3\mid A)=4\cdot\frac{20}{80}\cdot\frac{19}{79}\cdot\frac{18}{78} (no greens left; 4 colors with 20 each). - P(C_3\mid H)=4\cdot\frac{20}{90}\cdot\frac{19}{89}\cdot\frac{18}{88}+\frac{10}{90}\cdot\frac{9}{89}\cdot\frac{8}{88} (four 20’s and one 10). - P(C_3\mid Q)=4\cdot\frac{20}{95}\cdot\frac{19}{94}\cdot\frac{18}{93}+\frac{15}{95}\cdot\frac{14}{94}\cdot\frac{13}{93} (four 20’s and one 15). To finish part (e), weight by the scenario priors P(A)=\tfrac12, P(H)=\tfrac14, P(Q)=\tfrac14: P(C_3) =\tfrac12\,P(C_3\mid A)+\tfrac14\,P(C_3\mid H)+\tfrac14\,P(C_3\mid Q) \approx 0.04677.