Cargando…

Quantifying the impact of non-stationarity in reinforcement learning-based traffic signal control

In reinforcement learning (RL), dealing with non-stationarity is a challenging issue. However, some domains such as traffic optimization are inherently non-stationary. Causes for and effects of this are manifold. In particular, when dealing with traffic signal controls, addressing non-stationarity i...

Descripción completa

Detalles Bibliográficos
Autores principales: Alegre, Lucas N., Bazzan, Ana L.C., da Silva, Bruno C.
Formato: Online Artículo Texto
Lenguaje:English
Publicado: PeerJ Inc. 2021
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8176548/
https://www.ncbi.nlm.nih.gov/pubmed/34141896
http://dx.doi.org/10.7717/peerj-cs.575
_version_ 1783703276868337664
author Alegre, Lucas N.
Bazzan, Ana L.C.
da Silva, Bruno C.
author_facet Alegre, Lucas N.
Bazzan, Ana L.C.
da Silva, Bruno C.
author_sort Alegre, Lucas N.
collection PubMed
description In reinforcement learning (RL), dealing with non-stationarity is a challenging issue. However, some domains such as traffic optimization are inherently non-stationary. Causes for and effects of this are manifold. In particular, when dealing with traffic signal controls, addressing non-stationarity is key since traffic conditions change over time and as a function of traffic control decisions taken in other parts of a network. In this paper we analyze the effects that different sources of non-stationarity have in a network of traffic signals, in which each signal is modeled as a learning agent. More precisely, we study both the effects of changing the context in which an agent learns (e.g., a change in flow rates experienced by it), as well as the effects of reducing agent observability of the true environment state. Partial observability may cause distinct states (in which distinct actions are optimal) to be seen as the same by the traffic signal agents. This, in turn, may lead to sub-optimal performance. We show that the lack of suitable sensors to provide a representative observation of the real state seems to affect the performance more drastically than the changes to the underlying traffic patterns.
format Online
Article
Text
id pubmed-8176548
institution National Center for Biotechnology Information
language English
publishDate 2021
publisher PeerJ Inc.
record_format MEDLINE/PubMed
spelling pubmed-81765482021-06-16 Quantifying the impact of non-stationarity in reinforcement learning-based traffic signal control Alegre, Lucas N. Bazzan, Ana L.C. da Silva, Bruno C. PeerJ Comput Sci Agents and Multi-Agent Systems In reinforcement learning (RL), dealing with non-stationarity is a challenging issue. However, some domains such as traffic optimization are inherently non-stationary. Causes for and effects of this are manifold. In particular, when dealing with traffic signal controls, addressing non-stationarity is key since traffic conditions change over time and as a function of traffic control decisions taken in other parts of a network. In this paper we analyze the effects that different sources of non-stationarity have in a network of traffic signals, in which each signal is modeled as a learning agent. More precisely, we study both the effects of changing the context in which an agent learns (e.g., a change in flow rates experienced by it), as well as the effects of reducing agent observability of the true environment state. Partial observability may cause distinct states (in which distinct actions are optimal) to be seen as the same by the traffic signal agents. This, in turn, may lead to sub-optimal performance. We show that the lack of suitable sensors to provide a representative observation of the real state seems to affect the performance more drastically than the changes to the underlying traffic patterns. PeerJ Inc. 2021-05-27 /pmc/articles/PMC8176548/ /pubmed/34141896 http://dx.doi.org/10.7717/peerj-cs.575 Text en © 2021 Alegre et al. https://creativecommons.org/licenses/by/4.0/This is an open access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use, distribution, reproduction and adaptation in any medium and for any purpose provided that it is properly attributed. For attribution, the original author(s), title, publication source (PeerJ Computer Science) and either DOI or URL of the article must be cited.
spellingShingle Agents and Multi-Agent Systems
Alegre, Lucas N.
Bazzan, Ana L.C.
da Silva, Bruno C.
Quantifying the impact of non-stationarity in reinforcement learning-based traffic signal control
title Quantifying the impact of non-stationarity in reinforcement learning-based traffic signal control
title_full Quantifying the impact of non-stationarity in reinforcement learning-based traffic signal control
title_fullStr Quantifying the impact of non-stationarity in reinforcement learning-based traffic signal control
title_full_unstemmed Quantifying the impact of non-stationarity in reinforcement learning-based traffic signal control
title_short Quantifying the impact of non-stationarity in reinforcement learning-based traffic signal control
title_sort quantifying the impact of non-stationarity in reinforcement learning-based traffic signal control
topic Agents and Multi-Agent Systems
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8176548/
https://www.ncbi.nlm.nih.gov/pubmed/34141896
http://dx.doi.org/10.7717/peerj-cs.575
work_keys_str_mv AT alegrelucasn quantifyingtheimpactofnonstationarityinreinforcementlearningbasedtrafficsignalcontrol
AT bazzananalc quantifyingtheimpactofnonstationarityinreinforcementlearningbasedtrafficsignalcontrol
AT dasilvabrunoc quantifyingtheimpactofnonstationarityinreinforcementlearningbasedtrafficsignalcontrol