Classification
Classification is the task of assigning an input vector to one of discrete classes , where . When , the problem is called binary classification, and when , it becomes multi-class classification.
The central goal is to learn a function that partitions the input space into decision regions, each corresponding to one class. The boundaries between these regions are -dimensional hyperplanes called decision boundaries or decision surfaces. Their form determines how the model distinguishes between different categories.
Geometrically, classification can be visualized in terms of these boundaries:
- In one-dimensional data, the decision boundary is a single point dividing two regions.
- In two dimensions, the boundary becomes a line.
- In three dimensions, it forms a plane.
Datasets whose classes can be separated by a linear decision surface are said to be linearly separable, meaning that a single linear function can perfectly divide the classes without overlap.
There are three main approaches to solving classification problems:
- Discriminant functions directly map an input to a class without explicitly modeling probabilities. Support Vector Machines (SVMs) are a common example of this approach; they find the separating boundary that maximizes the margin between classes.
⠀- Generative probabilistic models model the class-conditional distribution and the prior probability , then apply Bayes’ theorem to obtain the posterior probability . Naïve Bayes is a well-known example of this method.
⠀- Discriminative probabilistic modeling directly models without needing to describe how the data are generated. Logistic regression is a classic example, learning the parameters that best separate the classes in the feature space by optimizing likelihood or minimizing cross-entropy loss.