Cargando…
Particle swarm optimization-based automatic parameter selection for deep neural networks and its applications in large-scale and high-dimensional data
In this paper, we propose a new automatic hyperparameter selection approach for determining the optimal network configuration (network structure and hyperparameters) for deep neural networks using particle swarm optimization (PSO) in combination with a steepest gradient descent algorithm. In the pro...
Autor principal: | |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Public Library of Science
2017
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5728507/ https://www.ncbi.nlm.nih.gov/pubmed/29236718 http://dx.doi.org/10.1371/journal.pone.0188746 |
_version_ | 1783286035500761088 |
---|---|
author | Ye, Fei |
author_facet | Ye, Fei |
author_sort | Ye, Fei |
collection | PubMed |
description | In this paper, we propose a new automatic hyperparameter selection approach for determining the optimal network configuration (network structure and hyperparameters) for deep neural networks using particle swarm optimization (PSO) in combination with a steepest gradient descent algorithm. In the proposed approach, network configurations were coded as a set of real-number m-dimensional vectors as the individuals of the PSO algorithm in the search procedure. During the search procedure, the PSO algorithm is employed to search for optimal network configurations via the particles moving in a finite search space, and the steepest gradient descent algorithm is used to train the DNN classifier with a few training epochs (to find a local optimal solution) during the population evaluation of PSO. After the optimization scheme, the steepest gradient descent algorithm is performed with more epochs and the final solutions (pbest and gbest) of the PSO algorithm to train a final ensemble model and individual DNN classifiers, respectively. The local search ability of the steepest gradient descent algorithm and the global search capabilities of the PSO algorithm are exploited to determine an optimal solution that is close to the global optimum. We constructed several experiments on hand-written characters and biological activity prediction datasets to show that the DNN classifiers trained by the network configurations expressed by the final solutions of the PSO algorithm, employed to construct an ensemble model and individual classifier, outperform the random approach in terms of the generalization performance. Therefore, the proposed approach can be regarded an alternative tool for automatic network structure and parameter selection for deep neural networks. |
format | Online Article Text |
id | pubmed-5728507 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2017 |
publisher | Public Library of Science |
record_format | MEDLINE/PubMed |
spelling | pubmed-57285072017-12-22 Particle swarm optimization-based automatic parameter selection for deep neural networks and its applications in large-scale and high-dimensional data Ye, Fei PLoS One Research Article In this paper, we propose a new automatic hyperparameter selection approach for determining the optimal network configuration (network structure and hyperparameters) for deep neural networks using particle swarm optimization (PSO) in combination with a steepest gradient descent algorithm. In the proposed approach, network configurations were coded as a set of real-number m-dimensional vectors as the individuals of the PSO algorithm in the search procedure. During the search procedure, the PSO algorithm is employed to search for optimal network configurations via the particles moving in a finite search space, and the steepest gradient descent algorithm is used to train the DNN classifier with a few training epochs (to find a local optimal solution) during the population evaluation of PSO. After the optimization scheme, the steepest gradient descent algorithm is performed with more epochs and the final solutions (pbest and gbest) of the PSO algorithm to train a final ensemble model and individual DNN classifiers, respectively. The local search ability of the steepest gradient descent algorithm and the global search capabilities of the PSO algorithm are exploited to determine an optimal solution that is close to the global optimum. We constructed several experiments on hand-written characters and biological activity prediction datasets to show that the DNN classifiers trained by the network configurations expressed by the final solutions of the PSO algorithm, employed to construct an ensemble model and individual classifier, outperform the random approach in terms of the generalization performance. Therefore, the proposed approach can be regarded an alternative tool for automatic network structure and parameter selection for deep neural networks. Public Library of Science 2017-12-13 /pmc/articles/PMC5728507/ /pubmed/29236718 http://dx.doi.org/10.1371/journal.pone.0188746 Text en © 2017 Fei Ye http://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. |
spellingShingle | Research Article Ye, Fei Particle swarm optimization-based automatic parameter selection for deep neural networks and its applications in large-scale and high-dimensional data |
title | Particle swarm optimization-based automatic parameter selection for deep neural networks and its applications in large-scale and high-dimensional data |
title_full | Particle swarm optimization-based automatic parameter selection for deep neural networks and its applications in large-scale and high-dimensional data |
title_fullStr | Particle swarm optimization-based automatic parameter selection for deep neural networks and its applications in large-scale and high-dimensional data |
title_full_unstemmed | Particle swarm optimization-based automatic parameter selection for deep neural networks and its applications in large-scale and high-dimensional data |
title_short | Particle swarm optimization-based automatic parameter selection for deep neural networks and its applications in large-scale and high-dimensional data |
title_sort | particle swarm optimization-based automatic parameter selection for deep neural networks and its applications in large-scale and high-dimensional data |
topic | Research Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5728507/ https://www.ncbi.nlm.nih.gov/pubmed/29236718 http://dx.doi.org/10.1371/journal.pone.0188746 |
work_keys_str_mv | AT yefei particleswarmoptimizationbasedautomaticparameterselectionfordeepneuralnetworksanditsapplicationsinlargescaleandhighdimensionaldata |