Cargando…

Gradient Statistics-Based Multi-Objective Optimization in Physics-Informed Neural Networks

Modeling and simulation of complex non-linear systems are essential in physics, engineering, and signal processing. Neural networks are widely regarded for such tasks due to their ability to learn complex representations from data. Training deep neural networks traditionally requires large amounts o...

Descripción completa

Detalles Bibliográficos
Autores principales: Vemuri, Sai Karthikeya, Denzler, Joachim
Formato: Online Artículo Texto
Lenguaje:English
Publicado: MDPI 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10650063/
https://www.ncbi.nlm.nih.gov/pubmed/37960365
http://dx.doi.org/10.3390/s23218665
_version_ 1785135693707608064
author Vemuri, Sai Karthikeya
Denzler, Joachim
author_facet Vemuri, Sai Karthikeya
Denzler, Joachim
author_sort Vemuri, Sai Karthikeya
collection PubMed
description Modeling and simulation of complex non-linear systems are essential in physics, engineering, and signal processing. Neural networks are widely regarded for such tasks due to their ability to learn complex representations from data. Training deep neural networks traditionally requires large amounts of data, which may not always be readily available for such systems. Contrarily, there is a large amount of domain knowledge in the form of mathematical models for the physics/behavior of such systems. A new class of neural networks called Physics-Informed Neural Networks (PINNs) has gained much attention recently as a paradigm for combining physics into neural networks. They have become a powerful tool for solving forward and inverse problems involving differential equations. A general framework of a PINN consists of a multi-layer perceptron that learns the solution of the partial differential equation (PDE) along with its boundary/initial conditions by minimizing a multi-objective loss function. This is formed by the sum of individual loss terms that penalize the output at different collocation points based on the differential equation and initial and boundary conditions. However, multiple loss terms arising from PDE residual and boundary conditions in PINNs pose a challenge in optimizing the overall loss function. This often leads to training failures and inaccurate results. We propose advanced gradient statistics-based weighting schemes for PINNs to address this challenge. These schemes utilize backpropagated gradient statistics of individual loss terms to appropriately scale and assign weights to each term, ensuring balanced training and meaningful solutions. In addition to the existing gradient statistics-based weighting schemes, we introduce kurtosis–standard deviation-based and combined mean and standard deviation-based schemes for approximating solutions of PDEs using PINNs. We provide a qualitative and quantitative comparison of these weighting schemes on 2D Poisson’s and Klein–Gordon’s equations, highlighting their effectiveness in improving PINN performance.
format Online
Article
Text
id pubmed-10650063
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher MDPI
record_format MEDLINE/PubMed
spelling pubmed-106500632023-10-24 Gradient Statistics-Based Multi-Objective Optimization in Physics-Informed Neural Networks Vemuri, Sai Karthikeya Denzler, Joachim Sensors (Basel) Article Modeling and simulation of complex non-linear systems are essential in physics, engineering, and signal processing. Neural networks are widely regarded for such tasks due to their ability to learn complex representations from data. Training deep neural networks traditionally requires large amounts of data, which may not always be readily available for such systems. Contrarily, there is a large amount of domain knowledge in the form of mathematical models for the physics/behavior of such systems. A new class of neural networks called Physics-Informed Neural Networks (PINNs) has gained much attention recently as a paradigm for combining physics into neural networks. They have become a powerful tool for solving forward and inverse problems involving differential equations. A general framework of a PINN consists of a multi-layer perceptron that learns the solution of the partial differential equation (PDE) along with its boundary/initial conditions by minimizing a multi-objective loss function. This is formed by the sum of individual loss terms that penalize the output at different collocation points based on the differential equation and initial and boundary conditions. However, multiple loss terms arising from PDE residual and boundary conditions in PINNs pose a challenge in optimizing the overall loss function. This often leads to training failures and inaccurate results. We propose advanced gradient statistics-based weighting schemes for PINNs to address this challenge. These schemes utilize backpropagated gradient statistics of individual loss terms to appropriately scale and assign weights to each term, ensuring balanced training and meaningful solutions. In addition to the existing gradient statistics-based weighting schemes, we introduce kurtosis–standard deviation-based and combined mean and standard deviation-based schemes for approximating solutions of PDEs using PINNs. We provide a qualitative and quantitative comparison of these weighting schemes on 2D Poisson’s and Klein–Gordon’s equations, highlighting their effectiveness in improving PINN performance. MDPI 2023-10-24 /pmc/articles/PMC10650063/ /pubmed/37960365 http://dx.doi.org/10.3390/s23218665 Text en © 2023 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
spellingShingle Article
Vemuri, Sai Karthikeya
Denzler, Joachim
Gradient Statistics-Based Multi-Objective Optimization in Physics-Informed Neural Networks
title Gradient Statistics-Based Multi-Objective Optimization in Physics-Informed Neural Networks
title_full Gradient Statistics-Based Multi-Objective Optimization in Physics-Informed Neural Networks
title_fullStr Gradient Statistics-Based Multi-Objective Optimization in Physics-Informed Neural Networks
title_full_unstemmed Gradient Statistics-Based Multi-Objective Optimization in Physics-Informed Neural Networks
title_short Gradient Statistics-Based Multi-Objective Optimization in Physics-Informed Neural Networks
title_sort gradient statistics-based multi-objective optimization in physics-informed neural networks
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10650063/
https://www.ncbi.nlm.nih.gov/pubmed/37960365
http://dx.doi.org/10.3390/s23218665
work_keys_str_mv AT vemurisaikarthikeya gradientstatisticsbasedmultiobjectiveoptimizationinphysicsinformedneuralnetworks
AT denzlerjoachim gradientstatisticsbasedmultiobjectiveoptimizationinphysicsinformedneuralnetworks