Cargando…

Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices: Design Considerations

In recent years, deep neural networks (DNN) have demonstrated significant business impact in large scale analysis and classification tasks such as speech recognition, visual object detection, pattern extraction, etc. Training of large DNNs, however, is universally considered as time consuming and co...

Descripción completa

Detalles Bibliográficos
Autores principales: Gokmen, Tayfun, Vlasov, Yurii
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Frontiers Media S.A. 2016
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4954855/
https://www.ncbi.nlm.nih.gov/pubmed/27493624
http://dx.doi.org/10.3389/fnins.2016.00333
_version_ 1782443844088889344
author Gokmen, Tayfun
Vlasov, Yurii
author_facet Gokmen, Tayfun
Vlasov, Yurii
author_sort Gokmen, Tayfun
collection PubMed
description In recent years, deep neural networks (DNN) have demonstrated significant business impact in large scale analysis and classification tasks such as speech recognition, visual object detection, pattern extraction, etc. Training of large DNNs, however, is universally considered as time consuming and computationally intensive task that demands datacenter-scale computational resources recruited for many days. Here we propose a concept of resistive processing unit (RPU) devices that can potentially accelerate DNN training by orders of magnitude while using much less power. The proposed RPU device can store and update the weight values locally thus minimizing data movement during training and allowing to fully exploit the locality and the parallelism of the training algorithm. We evaluate the effect of various RPU device features/non-idealities and system parameters on performance in order to derive the device and system level specifications for implementation of an accelerator chip for DNN training in a realistic CMOS-compatible technology. For large DNNs with about 1 billion weights this massively parallel RPU architecture can achieve acceleration factors of 30, 000 × compared to state-of-the-art microprocessors while providing power efficiency of 84, 000 GigaOps∕s∕W. Problems that currently require days of training on a datacenter-size cluster with thousands of machines can be addressed within hours on a single RPU accelerator. A system consisting of a cluster of RPU accelerators will be able to tackle Big Data problems with trillions of parameters that is impossible to address today like, for example, natural speech recognition and translation between all world languages, real-time analytics on large streams of business and scientific data, integration, and analysis of multimodal sensory data flows from a massive number of IoT (Internet of Things) sensors.
format Online
Article
Text
id pubmed-4954855
institution National Center for Biotechnology Information
language English
publishDate 2016
publisher Frontiers Media S.A.
record_format MEDLINE/PubMed
spelling pubmed-49548552016-08-04 Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices: Design Considerations Gokmen, Tayfun Vlasov, Yurii Front Neurosci Neuroscience In recent years, deep neural networks (DNN) have demonstrated significant business impact in large scale analysis and classification tasks such as speech recognition, visual object detection, pattern extraction, etc. Training of large DNNs, however, is universally considered as time consuming and computationally intensive task that demands datacenter-scale computational resources recruited for many days. Here we propose a concept of resistive processing unit (RPU) devices that can potentially accelerate DNN training by orders of magnitude while using much less power. The proposed RPU device can store and update the weight values locally thus minimizing data movement during training and allowing to fully exploit the locality and the parallelism of the training algorithm. We evaluate the effect of various RPU device features/non-idealities and system parameters on performance in order to derive the device and system level specifications for implementation of an accelerator chip for DNN training in a realistic CMOS-compatible technology. For large DNNs with about 1 billion weights this massively parallel RPU architecture can achieve acceleration factors of 30, 000 × compared to state-of-the-art microprocessors while providing power efficiency of 84, 000 GigaOps∕s∕W. Problems that currently require days of training on a datacenter-size cluster with thousands of machines can be addressed within hours on a single RPU accelerator. A system consisting of a cluster of RPU accelerators will be able to tackle Big Data problems with trillions of parameters that is impossible to address today like, for example, natural speech recognition and translation between all world languages, real-time analytics on large streams of business and scientific data, integration, and analysis of multimodal sensory data flows from a massive number of IoT (Internet of Things) sensors. Frontiers Media S.A. 2016-07-21 /pmc/articles/PMC4954855/ /pubmed/27493624 http://dx.doi.org/10.3389/fnins.2016.00333 Text en Copyright © 2016 Gokmen and Vlasov. http://creativecommons.org/licenses/by/4.0/ This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
spellingShingle Neuroscience
Gokmen, Tayfun
Vlasov, Yurii
Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices: Design Considerations
title Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices: Design Considerations
title_full Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices: Design Considerations
title_fullStr Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices: Design Considerations
title_full_unstemmed Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices: Design Considerations
title_short Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices: Design Considerations
title_sort acceleration of deep neural network training with resistive cross-point devices: design considerations
topic Neuroscience
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4954855/
https://www.ncbi.nlm.nih.gov/pubmed/27493624
http://dx.doi.org/10.3389/fnins.2016.00333
work_keys_str_mv AT gokmentayfun accelerationofdeepneuralnetworktrainingwithresistivecrosspointdevicesdesignconsiderations
AT vlasovyurii accelerationofdeepneuralnetworktrainingwithresistivecrosspointdevicesdesignconsiderations