Cargando…
Accelerating Hyperparameter Optimization of Deep Neural Network via Progressive Multi-Fidelity Evaluation
Deep neural networks usually require careful tuning of hyperparameters to show their best performance. However, with the size of state-of-the-art neural networks growing larger, the evaluation cost of the traditional Bayesian optimization has become unacceptable in most cases. Moreover, most practic...
Autores principales: | , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
2020
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206157/ http://dx.doi.org/10.1007/978-3-030-47426-3_58 |
_version_ | 1783530358307815424 |
---|---|
author | Zhu, Guanghui Zhu, Ruancheng |
author_facet | Zhu, Guanghui Zhu, Ruancheng |
author_sort | Zhu, Guanghui |
collection | PubMed |
description | Deep neural networks usually require careful tuning of hyperparameters to show their best performance. However, with the size of state-of-the-art neural networks growing larger, the evaluation cost of the traditional Bayesian optimization has become unacceptable in most cases. Moreover, most practical problems usually require good hyperparameter configurations within a limited time budget. To speed up the hyperparameter optimization, the successive halving technique is used to stop poorly-performed configurations as early as possible. In this paper, we propose a novel hyperparameter optimization method FastHO, which combines the progressive multi-fidelity technique with successive halving under a multi-armed bandit framework. Furthermore, we employ Bayesian optimization to guide the selection of initial configurations and an efficient data subsampling based method to warm start the surrogate model of Bayesian optimization. Extensive empirical evaluation on a broad range of neural networks and datasets shows that FastHO is not only effective to speed up hyperparameter optimization but also can achieve better anytime performance and final performance than the state-of-the-art hyperparameter optimization methods. |
format | Online Article Text |
id | pubmed-7206157 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2020 |
record_format | MEDLINE/PubMed |
spelling | pubmed-72061572020-05-08 Accelerating Hyperparameter Optimization of Deep Neural Network via Progressive Multi-Fidelity Evaluation Zhu, Guanghui Zhu, Ruancheng Advances in Knowledge Discovery and Data Mining Article Deep neural networks usually require careful tuning of hyperparameters to show their best performance. However, with the size of state-of-the-art neural networks growing larger, the evaluation cost of the traditional Bayesian optimization has become unacceptable in most cases. Moreover, most practical problems usually require good hyperparameter configurations within a limited time budget. To speed up the hyperparameter optimization, the successive halving technique is used to stop poorly-performed configurations as early as possible. In this paper, we propose a novel hyperparameter optimization method FastHO, which combines the progressive multi-fidelity technique with successive halving under a multi-armed bandit framework. Furthermore, we employ Bayesian optimization to guide the selection of initial configurations and an efficient data subsampling based method to warm start the surrogate model of Bayesian optimization. Extensive empirical evaluation on a broad range of neural networks and datasets shows that FastHO is not only effective to speed up hyperparameter optimization but also can achieve better anytime performance and final performance than the state-of-the-art hyperparameter optimization methods. 2020-04-17 /pmc/articles/PMC7206157/ http://dx.doi.org/10.1007/978-3-030-47426-3_58 Text en © Springer Nature Switzerland AG 2020 This article is made available via the PMC Open Access Subset for unrestricted research re-use and secondary analysis in any form or by any means with acknowledgement of the original source. These permissions are granted for the duration of the World Health Organization (WHO) declaration of COVID-19 as a global pandemic. |
spellingShingle | Article Zhu, Guanghui Zhu, Ruancheng Accelerating Hyperparameter Optimization of Deep Neural Network via Progressive Multi-Fidelity Evaluation |
title | Accelerating Hyperparameter Optimization of Deep Neural Network via Progressive Multi-Fidelity Evaluation |
title_full | Accelerating Hyperparameter Optimization of Deep Neural Network via Progressive Multi-Fidelity Evaluation |
title_fullStr | Accelerating Hyperparameter Optimization of Deep Neural Network via Progressive Multi-Fidelity Evaluation |
title_full_unstemmed | Accelerating Hyperparameter Optimization of Deep Neural Network via Progressive Multi-Fidelity Evaluation |
title_short | Accelerating Hyperparameter Optimization of Deep Neural Network via Progressive Multi-Fidelity Evaluation |
title_sort | accelerating hyperparameter optimization of deep neural network via progressive multi-fidelity evaluation |
topic | Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7206157/ http://dx.doi.org/10.1007/978-3-030-47426-3_58 |
work_keys_str_mv | AT zhuguanghui acceleratinghyperparameteroptimizationofdeepneuralnetworkviaprogressivemultifidelityevaluation AT zhuruancheng acceleratinghyperparameteroptimizationofdeepneuralnetworkviaprogressivemultifidelityevaluation |