Edit on GitHub

Bayes' Law Primer

I'm currently writing a blog post that uses Bayes' Law but don't want to muddy the post with a review in layman's terms. So I have something to link, here is a short description and a chance to flex my teaching muscles before the school year starts.

Bayes' Law

For those who aren't sure, Bayes' Law tells us that the probability event $X$ occurs given we know that event $Y$ has occurred can easily be computed. It is written as $\text{Pr}(X \, | \, Y)$ vertical bar is meant like the word "given", in other words, the event $X$ is distinct from the event $(X \, | \, Y),$ i.e. $X$ given $Y$. Bayes' law, states that

$\text{Pr}(X \, | \, Y) = \frac{\text{Pr}(X \text{ and } Y \text{ both occur})}{\text{Pr}(Y)}.$

This effectively is a re-scaling of the events by the total probability of the given event: $\text{Pr}(Y)$.

For example, if $X$ is the event that a $3$ is rolled on a fair die and $Y$ is the event that the roll is odd. We know of course that $\text{Pr}(Y) = \frac{1}{2}$ since half of the rolls are odd. The event $X \text{ and } Y \text{ both occur}$ in this case is the same as $X$ since the roll can only be $3$ if the roll is already odd. Thus

$\text{Pr}(X \text{ and } Y \text{ both occur}) = \frac{1}{6}$

and we can compute the conditional probability

$\text{Pr}(X \, | \, Y) = \frac{1 / 6}{1 / 2} = \frac{1}{3}.$

As we expect, one out of every three odd rolls is a $3$.

Bayes' Law Extended Form

Instead of considering a single event $Y,$ we can consider a range of $n$ possible events $Y_1, Y_2, \ldots, Y_n$ occur. We require that one of these $Y$-events must occur and that they cover all possible events that could occur. For example $Y_1$ is the event that H2O is vapor, $Y_2$ is the event that H2O is water and $Y_3$ is the event that H2O is ice.

In such a case we know that since the $Y$-events are distinct

\begin{aligned} \text{Pr}(X) &= \text{Pr}(X \text{ and } Y_1 \text{ both occur}) \\ &+ \text{Pr}(X \text{ and } Y_2 \text{ both occur}) \\ &+ \text{Pr}(X \text{ and } Y_3 \text{ both occur}). \end{aligned}

Using Bayes' law, we can reinterpret as

$\text{Pr}(X \text{ and } Y_j \text{ both occur}) = \text{Pr}(X \, | \, Y_j) \cdot \text{Pr}(Y_j)$

and the above becomes

\begin{aligned} \text{Pr}(X) &= \text{Pr}(X \, | \, Y_1) \cdot \text{Pr}(Y_1) + \text{Pr}(X \, | \, Y_2) \cdot \text{Pr}(Y_2) \\ &+ \text{Pr}(X \, | \, Y_3) \cdot \text{Pr}(Y_3). \end{aligned}

The same is true if we replace $3$ with an arbitrary number of events $n$.