Cargando…
A High-Precision Implementation of the Sigmoid Activation Function for Computing-in-Memory Architecture
Computing-In-Memory (CIM), based on non-von Neumann architecture, has lately received significant attention due to its lower overhead in delay and higher energy efficiency in convolutional and fully-connected neural network computing. Growing works have given the priority to researching the array of...
Autores principales: | , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
MDPI
2021
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8540118/ https://www.ncbi.nlm.nih.gov/pubmed/34683234 http://dx.doi.org/10.3390/mi12101183 |
_version_ | 1784588910166278144 |
---|---|
author | Xu, Siqiu Li, Xi Xie, Chenchen Chen, Houpeng Chen, Cheng Song, Zhitang |
author_facet | Xu, Siqiu Li, Xi Xie, Chenchen Chen, Houpeng Chen, Cheng Song, Zhitang |
author_sort | Xu, Siqiu |
collection | PubMed |
description | Computing-In-Memory (CIM), based on non-von Neumann architecture, has lately received significant attention due to its lower overhead in delay and higher energy efficiency in convolutional and fully-connected neural network computing. Growing works have given the priority to researching the array of memory and peripheral circuits to achieve multiply-and-accumulate (MAC) operation, but not enough attention has been paid to the high-precision hardware implementation of non-linear layers up to now, which still causes time overhead and power consumption. Sigmoid is a widely used non-linear activation function and most of its studies provided an approximation of the function expression rather than totally matched, inevitably leading to considerable error. To address this issue, we propose a high-precision circuit implementation of the sigmoid, matching the expression exactly for the first time. The simulation results with the SMIC 40 nm process suggest that the proposed circuit implemented high-precision sigmoid perfectly achieves the properties of the ideal sigmoid, showing the maximum error and average error between the proposed simulated sigmoid and ideal sigmoid is 2.74% and 0.21%, respectively. In addition, a multi-layer convolutional neural network based on CIM architecture employing the simulated high-precision sigmoid activation function verifies the similar recognition accuracy on the test database of handwritten digits compared to utilize the ideal sigmoid in software, with online training achieving 97.06% and with offline training achieving 97.74%. |
format | Online Article Text |
id | pubmed-8540118 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2021 |
publisher | MDPI |
record_format | MEDLINE/PubMed |
spelling | pubmed-85401182021-10-24 A High-Precision Implementation of the Sigmoid Activation Function for Computing-in-Memory Architecture Xu, Siqiu Li, Xi Xie, Chenchen Chen, Houpeng Chen, Cheng Song, Zhitang Micromachines (Basel) Article Computing-In-Memory (CIM), based on non-von Neumann architecture, has lately received significant attention due to its lower overhead in delay and higher energy efficiency in convolutional and fully-connected neural network computing. Growing works have given the priority to researching the array of memory and peripheral circuits to achieve multiply-and-accumulate (MAC) operation, but not enough attention has been paid to the high-precision hardware implementation of non-linear layers up to now, which still causes time overhead and power consumption. Sigmoid is a widely used non-linear activation function and most of its studies provided an approximation of the function expression rather than totally matched, inevitably leading to considerable error. To address this issue, we propose a high-precision circuit implementation of the sigmoid, matching the expression exactly for the first time. The simulation results with the SMIC 40 nm process suggest that the proposed circuit implemented high-precision sigmoid perfectly achieves the properties of the ideal sigmoid, showing the maximum error and average error between the proposed simulated sigmoid and ideal sigmoid is 2.74% and 0.21%, respectively. In addition, a multi-layer convolutional neural network based on CIM architecture employing the simulated high-precision sigmoid activation function verifies the similar recognition accuracy on the test database of handwritten digits compared to utilize the ideal sigmoid in software, with online training achieving 97.06% and with offline training achieving 97.74%. MDPI 2021-09-29 /pmc/articles/PMC8540118/ /pubmed/34683234 http://dx.doi.org/10.3390/mi12101183 Text en © 2021 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). |
spellingShingle | Article Xu, Siqiu Li, Xi Xie, Chenchen Chen, Houpeng Chen, Cheng Song, Zhitang A High-Precision Implementation of the Sigmoid Activation Function for Computing-in-Memory Architecture |
title | A High-Precision Implementation of the Sigmoid Activation Function for Computing-in-Memory Architecture |
title_full | A High-Precision Implementation of the Sigmoid Activation Function for Computing-in-Memory Architecture |
title_fullStr | A High-Precision Implementation of the Sigmoid Activation Function for Computing-in-Memory Architecture |
title_full_unstemmed | A High-Precision Implementation of the Sigmoid Activation Function for Computing-in-Memory Architecture |
title_short | A High-Precision Implementation of the Sigmoid Activation Function for Computing-in-Memory Architecture |
title_sort | high-precision implementation of the sigmoid activation function for computing-in-memory architecture |
topic | Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8540118/ https://www.ncbi.nlm.nih.gov/pubmed/34683234 http://dx.doi.org/10.3390/mi12101183 |
work_keys_str_mv | AT xusiqiu ahighprecisionimplementationofthesigmoidactivationfunctionforcomputinginmemoryarchitecture AT lixi ahighprecisionimplementationofthesigmoidactivationfunctionforcomputinginmemoryarchitecture AT xiechenchen ahighprecisionimplementationofthesigmoidactivationfunctionforcomputinginmemoryarchitecture AT chenhoupeng ahighprecisionimplementationofthesigmoidactivationfunctionforcomputinginmemoryarchitecture AT chencheng ahighprecisionimplementationofthesigmoidactivationfunctionforcomputinginmemoryarchitecture AT songzhitang ahighprecisionimplementationofthesigmoidactivationfunctionforcomputinginmemoryarchitecture AT xusiqiu highprecisionimplementationofthesigmoidactivationfunctionforcomputinginmemoryarchitecture AT lixi highprecisionimplementationofthesigmoidactivationfunctionforcomputinginmemoryarchitecture AT xiechenchen highprecisionimplementationofthesigmoidactivationfunctionforcomputinginmemoryarchitecture AT chenhoupeng highprecisionimplementationofthesigmoidactivationfunctionforcomputinginmemoryarchitecture AT chencheng highprecisionimplementationofthesigmoidactivationfunctionforcomputinginmemoryarchitecture AT songzhitang highprecisionimplementationofthesigmoidactivationfunctionforcomputinginmemoryarchitecture |