2.3. Linear Discriminants

Linear Discriminants

Linear discriminant methods classify data by defining linear decision boundaries in the feature space. For two-class problems, the model assumes a linear function of the input vector,
$y(x)=w^{T}x+w_0,$ where the sign or magnitude of $y(x)$ determines the class. The decision rule assigns $C_1$ if $y(x)\ge 0.5$, and $C_2$ otherwise.

The boundary itself occurs when $y(\mathbf{x})=0.5$, forming a ($D-1)$-dimensional surface that splits the input space into two regions.

For any two points $x_A$ and $x_B$ lying on the decision surface,
$y({\mathbf{x}}_A)=y({\mathbf{x}}_B)=0\Rightarrow {\mathbf{w}}^{\top }({\mathbf{x}}_A-{\mathbf{x}}_B)=0.$
This means that $w$ is orthogonal to every vector lying within the decision boundary. The vector $\mathbf{w}$ therefore gives the orientation of the surface, while the bias term $w_0$ gives its location relative to the origin. The perpendicular (normal) distance from the origin to the surface is
$\frac{|{\mathbf{w}}^{\top }\mathbf{x}+w_0|}{\Vert \mathbf{w}\Vert }=\frac{|w_0|}{\Vert \mathbf{w}\Vert }.$