Principal component analysis projects the data in the directions of maximum variance. However, in some cases, maximizing the variance of the data may not achieve highly discriminative features. Fisher’s Linear discriminant analysis (LDA) is also a projection-based method that seeks to find a vector that defines the projection direction of the data . Unlike principal component analysis which maximizes the variance of the data , linear discriminant analysis uses the labels of the data and projects the data in the direction that maximizes the separation of the classes as shown in Figure 1.
How do we find the projection that maximizes the class separation of the data? LDA states that maximum separation is achieved when the distance between the two clusters is maximum (or far away from each other). We consider the classification of two classes first, then generalize to more than two classes.
Binary Class LDA
Let and denote the means of data after projection which belong to class RED and BLUE respectively.
Furthermore, let and denote the scatters (variances) of data after projection which belong to class RED and BLUE respectively.
To ensure maximum class separation, we want the means to be far away from each other and the projected data in the clusters to be close to each other. In other words, we want the mean difference to be large and the summation of the scatter to be small. Therefore, the objective is to maximize
The numerator can be expressed as
where is between-class scatter matrix.
The denominator can be expressed as
where is within-class scatter matrix.
Therefore, the objective function is
Taking the derivative and setting it equal to zero.
Given that is a scalar, the expression becomes
where is the eigenvector of and is the corresponding eigenvalue.
When there are more than two classes, the between-class scatter matrix is defined as
where and is the global (overall) mean.
The within-class scatter matrix is defined as
The number of components is defined as follows:
where is the number of classes.
Solving for Eigenvector and Eigenvalue
Compute and .
where is an identity matrix.
Solve to obtain .
Substitute into to obtain .
Consider the following dataset
Find by solving where is an identity matrix.
Let , thus
 R. A. Fisher, “The Use of Multiple Measurements in Taxonomic Problems,” Annals of Eugenics, vol. 7, no. 2, pp. 179–188, 1936, doi: 10.1111/j.1469-1809.1936.tb02137.x.
 A. M. Martinez and A. C. Kak, “PCA versus LDA,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 23, no. 2, pp. 228–233, Feb. 2001, doi: 10.1109/34.908974.