Cargando…

A High-Precision Implementation of the Sigmoid Activation Function for Computing-in-Memory Architecture

Computing-In-Memory (CIM), based on non-von Neumann architecture, has lately received significant attention due to its lower overhead in delay and higher energy efficiency in convolutional and fully-connected neural network computing. Growing works have given the priority to researching the array of...

Descripción completa

Detalles Bibliográficos
Autores principales: Xu, Siqiu, Li, Xi, Xie, Chenchen, Chen, Houpeng, Chen, Cheng, Song, Zhitang
Formato: Online Artículo Texto
Lenguaje:English
Publicado: MDPI 2021
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8540118/
https://www.ncbi.nlm.nih.gov/pubmed/34683234
http://dx.doi.org/10.3390/mi12101183
_version_ 1784588910166278144
author Xu, Siqiu
Li, Xi
Xie, Chenchen
Chen, Houpeng
Chen, Cheng
Song, Zhitang
author_facet Xu, Siqiu
Li, Xi
Xie, Chenchen
Chen, Houpeng
Chen, Cheng
Song, Zhitang
author_sort Xu, Siqiu
collection PubMed
description Computing-In-Memory (CIM), based on non-von Neumann architecture, has lately received significant attention due to its lower overhead in delay and higher energy efficiency in convolutional and fully-connected neural network computing. Growing works have given the priority to researching the array of memory and peripheral circuits to achieve multiply-and-accumulate (MAC) operation, but not enough attention has been paid to the high-precision hardware implementation of non-linear layers up to now, which still causes time overhead and power consumption. Sigmoid is a widely used non-linear activation function and most of its studies provided an approximation of the function expression rather than totally matched, inevitably leading to considerable error. To address this issue, we propose a high-precision circuit implementation of the sigmoid, matching the expression exactly for the first time. The simulation results with the SMIC 40 nm process suggest that the proposed circuit implemented high-precision sigmoid perfectly achieves the properties of the ideal sigmoid, showing the maximum error and average error between the proposed simulated sigmoid and ideal sigmoid is 2.74% and 0.21%, respectively. In addition, a multi-layer convolutional neural network based on CIM architecture employing the simulated high-precision sigmoid activation function verifies the similar recognition accuracy on the test database of handwritten digits compared to utilize the ideal sigmoid in software, with online training achieving 97.06% and with offline training achieving 97.74%.
format Online
Article
Text
id pubmed-8540118
institution National Center for Biotechnology Information
language English
publishDate 2021
publisher MDPI
record_format MEDLINE/PubMed
spelling pubmed-85401182021-10-24 A High-Precision Implementation of the Sigmoid Activation Function for Computing-in-Memory Architecture Xu, Siqiu Li, Xi Xie, Chenchen Chen, Houpeng Chen, Cheng Song, Zhitang Micromachines (Basel) Article Computing-In-Memory (CIM), based on non-von Neumann architecture, has lately received significant attention due to its lower overhead in delay and higher energy efficiency in convolutional and fully-connected neural network computing. Growing works have given the priority to researching the array of memory and peripheral circuits to achieve multiply-and-accumulate (MAC) operation, but not enough attention has been paid to the high-precision hardware implementation of non-linear layers up to now, which still causes time overhead and power consumption. Sigmoid is a widely used non-linear activation function and most of its studies provided an approximation of the function expression rather than totally matched, inevitably leading to considerable error. To address this issue, we propose a high-precision circuit implementation of the sigmoid, matching the expression exactly for the first time. The simulation results with the SMIC 40 nm process suggest that the proposed circuit implemented high-precision sigmoid perfectly achieves the properties of the ideal sigmoid, showing the maximum error and average error between the proposed simulated sigmoid and ideal sigmoid is 2.74% and 0.21%, respectively. In addition, a multi-layer convolutional neural network based on CIM architecture employing the simulated high-precision sigmoid activation function verifies the similar recognition accuracy on the test database of handwritten digits compared to utilize the ideal sigmoid in software, with online training achieving 97.06% and with offline training achieving 97.74%. MDPI 2021-09-29 /pmc/articles/PMC8540118/ /pubmed/34683234 http://dx.doi.org/10.3390/mi12101183 Text en © 2021 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
spellingShingle Article
Xu, Siqiu
Li, Xi
Xie, Chenchen
Chen, Houpeng
Chen, Cheng
Song, Zhitang
A High-Precision Implementation of the Sigmoid Activation Function for Computing-in-Memory Architecture
title A High-Precision Implementation of the Sigmoid Activation Function for Computing-in-Memory Architecture
title_full A High-Precision Implementation of the Sigmoid Activation Function for Computing-in-Memory Architecture
title_fullStr A High-Precision Implementation of the Sigmoid Activation Function for Computing-in-Memory Architecture
title_full_unstemmed A High-Precision Implementation of the Sigmoid Activation Function for Computing-in-Memory Architecture
title_short A High-Precision Implementation of the Sigmoid Activation Function for Computing-in-Memory Architecture
title_sort high-precision implementation of the sigmoid activation function for computing-in-memory architecture
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8540118/
https://www.ncbi.nlm.nih.gov/pubmed/34683234
http://dx.doi.org/10.3390/mi12101183
work_keys_str_mv AT xusiqiu ahighprecisionimplementationofthesigmoidactivationfunctionforcomputinginmemoryarchitecture
AT lixi ahighprecisionimplementationofthesigmoidactivationfunctionforcomputinginmemoryarchitecture
AT xiechenchen ahighprecisionimplementationofthesigmoidactivationfunctionforcomputinginmemoryarchitecture
AT chenhoupeng ahighprecisionimplementationofthesigmoidactivationfunctionforcomputinginmemoryarchitecture
AT chencheng ahighprecisionimplementationofthesigmoidactivationfunctionforcomputinginmemoryarchitecture
AT songzhitang ahighprecisionimplementationofthesigmoidactivationfunctionforcomputinginmemoryarchitecture
AT xusiqiu highprecisionimplementationofthesigmoidactivationfunctionforcomputinginmemoryarchitecture
AT lixi highprecisionimplementationofthesigmoidactivationfunctionforcomputinginmemoryarchitecture
AT xiechenchen highprecisionimplementationofthesigmoidactivationfunctionforcomputinginmemoryarchitecture
AT chenhoupeng highprecisionimplementationofthesigmoidactivationfunctionforcomputinginmemoryarchitecture
AT chencheng highprecisionimplementationofthesigmoidactivationfunctionforcomputinginmemoryarchitecture
AT songzhitang highprecisionimplementationofthesigmoidactivationfunctionforcomputinginmemoryarchitecture