6.5. Naive Bayes

Naive Bayes Classifier

Naive Bayes is a probabilistic classification model that represents the relationship between a class label $Y$ and a set of features $F_1,\dots ,F_n$ using a Bayes net with a single parent node. It is designed for supervised classification tasks, where each data point is represented by extracted features and an associated label.

The defining modeling assumption of Naive Bayes is conditional independence: all features are assumed to be independent of one another given the class label. Graphically, this corresponds to a Bayes net in which $Y$ is the parent of every feature $F_i$, with no edges between features. By the rules of d-separation, this implies
$P(F_1,\dots ,F_n\mid Y)={\prod }_{i=1}^{n}P(F_i\mid Y).$

This assumption is strong—and often false in reality—but it drastically reduces the number of parameters needed to represent the joint distribution, making inference and learning computationally efficient.

For classification, the goal is to predict the most likely label given an observed feature vector $f=(f_1,\dots ,f_n)$. Using Bayes’ rule and dropping the normalization constant, Naive Bayes predicts:

$\hat{Y}(f)=\arg {\max }_{y}P(Y=y\mid F_1=f_1,\dots ,F_n=f_n)=\arg {\max }_{y}P(Y=y){\prod }_{i=1}^{n}P(F_i=f_i\mid Y=y).$

The class prior $P(Y)$ captures how frequently each label appears, while the conditional probabilities $P(F_i\mid Y)$ capture how each feature behaves within a class. This decomposition is what makes Naive Bayes scalable even when the number of features is large.

The parameters of a Naive Bayes model are typically learned using maximum likelihood estimation (MLE) under the assumption that training examples are independent and identically distributed (i.i.d.). For Bernoulli features (e.g., whether a word appears in an email), the MLE for a conditional probability is simply the empirical frequency:
$P(F_i=1\mid Y=y)=\frac{\text{count of samples with }{F}_i=1\text{ and }Y=y}{\text{count of samples with }Y=y}.$

This results in simple count-based parameter estimation that scales linearly with the size of the training data.

A key limitation of pure MLE in Naive Bayes is overfitting, especially when certain feature–class combinations never appear in the training data, leading to zero probabilities and thus zeroed-out posteriors. To address this, Laplace smoothing is commonly applied. With smoothing strength k, probabilities are estimated as:
$P_{\text{Lap}}(x\mid y)=\frac{\text{count}(x,y)+k}{\text{count}(y)+k|X|},$
which can be interpreted as assuming k additional pseudo-counts of each possible outcome. Smoothing prevents zero probabilities and improves generalization to unseen data.