Weight Vector Tuning and Asymptotic Analysis of Binary Linear Classifiers
KAUST DepartmentCenter of Excellence for NEOM Research
Communication Theory Lab
Computer, Electrical and Mathematical Science and Engineering (CEMSE) Division
Electrical and Computer Engineering Program
Office of the President
KAUST Grant NumberOSR-CRG2019-4041
Permanent link to this recordhttp://hdl.handle.net/10754/679976
MetadataShow full item record
AbstractUnlike its intercept, a linear classifier's weight vector cannot be tuned by a simple grid search. Hence, this paper proposes weight vector tuning of a generic binary linear classifier through the parameterization of a decomposition of the discriminant by a scalar which controls the trade-off between conflicting informative and noisy terms. By varying this parameter, the original weight vector is modified in a meaningful way. Applying this method to a number of linear classifiers under a variety of data dimensionality and sample size settings reveals that the classification performance loss due to non-optimal native hyperparameters can be compensated for by weight vector tuning. This yields computational savings as the proposed tuning method reduces to tuning a scalar compared to tuning the native hyperparameter, which may involve repeated weight vector generation along with its burden of optimization, dimensionality reduction, etc., depending on the classifier. It is also found that weight vector tuning significantly improves the performance of Linear Discriminant Analysis (LDA) under high estimation noise. Proceeding from this second finding, an asymptotic study of the misclassification probability of the parameterized LDA classifier in the growth regime where the data dimensionality and sample size are comparable is conducted. Using random matrix theory, the misclassification probability is shown to converge to a quantity that is a function of the true statistics of the data. Additionally, an estimator of the misclassification probability is derived. Finally, computationally efficient tuning of the parameter using this estimator is demonstrated on real data.
CitationNiyazi, L. B., Kammoun, A., Dahrouj, H., Alouini, M.-S., & Al-Naffouri, T. Y. (2022). Weight Vector Tuning and Asymptotic Analysis of Binary Linear Classifiers. IEEE Open Journal of Signal Processing, 1–15. https://doi.org/10.1109/ojsp.2022.3195150
SponsorsThis publication is based upon work supported by the King Abdullah University of Science and Technology (KAUST) Office of Sponsored Research (OSR) under Award No. OSR-CRG2019-4041.
Except where otherwise noted, this item's license is described as Archived with thanks to IEEE Open Journal of Signal Processing under a Creative Commons license, details at: https://creativecommons.org/licenses/by/4.0/