Towards an Efficient Artificial Neural Network Pruning and Feature Ranking Tool

Handle URI:
http://hdl.handle.net/10754/555862
Title:
Towards an Efficient Artificial Neural Network Pruning and Feature Ranking Tool
Authors:
AlShahrani, Mona ( 0000-0002-9848-8248 )
Abstract:
Artificial Neural Networks (ANNs) are known to be among the most effective and expressive machine learning models. Their impressive abilities to learn have been reflected in many broad application domains such as image recognition, medical diagnosis, online banking, robotics, dynamic systems, and many others. ANNs with multiple layers of complex non-linear transformations (a.k.a Deep ANNs) have shown recently successful results in the area of computer vision and speech recognition. ANNs are parametric models that approximate unknown functions in which parameter values (weights) are adapted during training. ANN’s weights can be large in number and thus render the trained model more complex with chances for “overfitting” training data. In this study, we explore the effects of network pruning on performance of ANNs and ranking of features that describe the data. Simplified ANN model results in fewer parameters, less computation and faster training. We investigate the use of Hessian-based pruning algorithms as well as simpler ones (i.e. non Hessian-based) on nine datasets with varying number of input features and ANN parameters. The Hessian-based Optimal Brain Surgeon algorithm (OBS) is robust but slow. Therefore a faster parallel Hessian- approximation is provided. An additional speedup is provided using a variant we name ‘Simple n Optimal Brain Surgeon’ (SNOBS), which represents a good compromise between robustness and time efficiency. For some of the datasets, the ANN pruning experiments show on average 91% reduction in the number of ANN parameters and about 60% - 90% in the number of ANN input features, while maintaining comparable or better accuracy to the case when no pruning is applied. Finally, we show through a comprehensive comparison with seven state-of-the art feature filtering methods that the feature selection and ranking obtained as a byproduct of the ANN pruning is comparable in accuracy to these methods.
Advisors:
Bajic, Vladimir B. ( 0000-0001-5435-4750 )
Committee Member:
Hoehndorf, Robert ( 0000-0001-8149-5890 ) ; Soloviev, Victor ( 0000-0001-8885-493X )
KAUST Department:
Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division
Program:
Computer Science
Issue Date:
24-May-2015
Type:
Thesis
Appears in Collections:
Theses; Computer Science Program; Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division

Full metadata record

DC FieldValue Language
dc.contributor.advisorBajic, Vladimir B.en
dc.contributor.authorAlShahrani, Monaen
dc.date.accessioned2015-05-27T08:34:20Zen
dc.date.available2015-05-27T08:34:20Zen
dc.date.issued2015-05-24en
dc.identifier.urihttp://hdl.handle.net/10754/555862en
dc.description.abstractArtificial Neural Networks (ANNs) are known to be among the most effective and expressive machine learning models. Their impressive abilities to learn have been reflected in many broad application domains such as image recognition, medical diagnosis, online banking, robotics, dynamic systems, and many others. ANNs with multiple layers of complex non-linear transformations (a.k.a Deep ANNs) have shown recently successful results in the area of computer vision and speech recognition. ANNs are parametric models that approximate unknown functions in which parameter values (weights) are adapted during training. ANN’s weights can be large in number and thus render the trained model more complex with chances for “overfitting” training data. In this study, we explore the effects of network pruning on performance of ANNs and ranking of features that describe the data. Simplified ANN model results in fewer parameters, less computation and faster training. We investigate the use of Hessian-based pruning algorithms as well as simpler ones (i.e. non Hessian-based) on nine datasets with varying number of input features and ANN parameters. The Hessian-based Optimal Brain Surgeon algorithm (OBS) is robust but slow. Therefore a faster parallel Hessian- approximation is provided. An additional speedup is provided using a variant we name ‘Simple n Optimal Brain Surgeon’ (SNOBS), which represents a good compromise between robustness and time efficiency. For some of the datasets, the ANN pruning experiments show on average 91% reduction in the number of ANN parameters and about 60% - 90% in the number of ANN input features, while maintaining comparable or better accuracy to the case when no pruning is applied. Finally, we show through a comprehensive comparison with seven state-of-the art feature filtering methods that the feature selection and ranking obtained as a byproduct of the ANN pruning is comparable in accuracy to these methods.en
dc.language.isoenen
dc.subjectArtificialen
dc.subjectPrunningen
dc.subjectNeuralen
dc.subjectNetworken
dc.subjectFeatureen
dc.subjectRankingen
dc.titleTowards an Efficient Artificial Neural Network Pruning and Feature Ranking Toolen
dc.typeThesisen
dc.contributor.departmentComputer, Electrical and Mathematical Sciences and Engineering (CEMSE) Divisionen
thesis.degree.grantorKing Abdullah University of Science and Technologyen_GB
dc.contributor.committeememberHoehndorf, Roberten
dc.contributor.committeememberSoloviev, Victoren
thesis.degree.disciplineComputer Scienceen
thesis.degree.nameMaster of Scienceen
dc.person.id124047en
All Items in KAUST are protected by copyright, with all rights reserved, unless otherwise indicated.