Cargando…

Ecological forecasts reveal limitations of common model selection methods: predicting changes in beaver colony densities

Over the past two decades, there have been numerous calls to make ecology a more predictive science through direct empirical assessments of ecological models and predictions. While the widespread use of model selection using information criteria has pushed ecology toward placing a higher emphasis on...

Descripción completa

Detalles Bibliográficos
Autores principales: Johnson‐Bice, Sean M., Ferguson, Jake M., Erb, John D., Gable, Thomas D., Windels, Steve K.
Formato: Online Artículo Texto
Lenguaje:English
Publicado: John Wiley and Sons Inc. 2020
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7816246/
https://www.ncbi.nlm.nih.gov/pubmed/32583507
http://dx.doi.org/10.1002/eap.2198
_version_ 1783638403575709696
author Johnson‐Bice, Sean M.
Ferguson, Jake M.
Erb, John D.
Gable, Thomas D.
Windels, Steve K.
author_facet Johnson‐Bice, Sean M.
Ferguson, Jake M.
Erb, John D.
Gable, Thomas D.
Windels, Steve K.
author_sort Johnson‐Bice, Sean M.
collection PubMed
description Over the past two decades, there have been numerous calls to make ecology a more predictive science through direct empirical assessments of ecological models and predictions. While the widespread use of model selection using information criteria has pushed ecology toward placing a higher emphasis on prediction, few attempts have been made to validate the ability of information criteria to correctly identify the most parsimonious model with the greatest predictive accuracy. Here, we used an ecological forecasting framework to test the ability of information criteria to accurately predict the relative contribution of density dependence and density‐independent factors (forage availability, harvest, weather, wolf [Canis lupus] density) on inter‐annual fluctuations in beaver (Castor canadensis) colony densities. We modeled changes in colony densities using a discrete‐time Gompertz model, and assessed the performance of four models using information criteria values: density‐independent models with (1) and without (2) environmental covariates; and density‐dependent models with (3) and without (4) environmental covariates. We then evaluated the forecasting accuracy of each model by withholding the final one‐third of observations from each population and compared observed vs. predicted densities. Information criteria and our forecasting accuracy metrics both provided strong evidence of compensatory density dependence in the annual dynamics of beaver colony densities. However, despite strong within‐sample performance by the most complex model (density‐dependent with covariates) as determined using information criteria, hindcasts of colony densities revealed that the much simpler density‐dependent model without covariates performed nearly as well predicting out‐of‐sample colony densities. The hindcast results indicated that the complex model over‐fit our data, suggesting that parameters identified by information criteria as important predictor variables are only marginally valuable for predicting landscape‐scale beaver colony dynamics. Our study demonstrates the importance of evaluating ecological models and predictions with long‐term data and revealed how a known limitation of information criteria (over‐fitting of complex models) can affect our interpretation of ecological dynamics. While incorporating knowledge of the factors that influence animal population dynamics can improve population forecasts, we suggest that comparing forecast performance metrics can likewise improve our knowledge of the factors driving population dynamics.
format Online
Article
Text
id pubmed-7816246
institution National Center for Biotechnology Information
language English
publishDate 2020
publisher John Wiley and Sons Inc.
record_format MEDLINE/PubMed
spelling pubmed-78162462021-01-27 Ecological forecasts reveal limitations of common model selection methods: predicting changes in beaver colony densities Johnson‐Bice, Sean M. Ferguson, Jake M. Erb, John D. Gable, Thomas D. Windels, Steve K. Ecol Appl Articles Over the past two decades, there have been numerous calls to make ecology a more predictive science through direct empirical assessments of ecological models and predictions. While the widespread use of model selection using information criteria has pushed ecology toward placing a higher emphasis on prediction, few attempts have been made to validate the ability of information criteria to correctly identify the most parsimonious model with the greatest predictive accuracy. Here, we used an ecological forecasting framework to test the ability of information criteria to accurately predict the relative contribution of density dependence and density‐independent factors (forage availability, harvest, weather, wolf [Canis lupus] density) on inter‐annual fluctuations in beaver (Castor canadensis) colony densities. We modeled changes in colony densities using a discrete‐time Gompertz model, and assessed the performance of four models using information criteria values: density‐independent models with (1) and without (2) environmental covariates; and density‐dependent models with (3) and without (4) environmental covariates. We then evaluated the forecasting accuracy of each model by withholding the final one‐third of observations from each population and compared observed vs. predicted densities. Information criteria and our forecasting accuracy metrics both provided strong evidence of compensatory density dependence in the annual dynamics of beaver colony densities. However, despite strong within‐sample performance by the most complex model (density‐dependent with covariates) as determined using information criteria, hindcasts of colony densities revealed that the much simpler density‐dependent model without covariates performed nearly as well predicting out‐of‐sample colony densities. The hindcast results indicated that the complex model over‐fit our data, suggesting that parameters identified by information criteria as important predictor variables are only marginally valuable for predicting landscape‐scale beaver colony dynamics. Our study demonstrates the importance of evaluating ecological models and predictions with long‐term data and revealed how a known limitation of information criteria (over‐fitting of complex models) can affect our interpretation of ecological dynamics. While incorporating knowledge of the factors that influence animal population dynamics can improve population forecasts, we suggest that comparing forecast performance metrics can likewise improve our knowledge of the factors driving population dynamics. John Wiley and Sons Inc. 2020-07-21 2021-01 /pmc/articles/PMC7816246/ /pubmed/32583507 http://dx.doi.org/10.1002/eap.2198 Text en © 2020 The Authors. Ecological Applications published by Wiley Periodicals LLC on behalf of Ecological Society of America This is an open access article under the terms of the http://creativecommons.org/licenses/by-nc/4.0/ License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited and is not used for commercial purposes.
spellingShingle Articles
Johnson‐Bice, Sean M.
Ferguson, Jake M.
Erb, John D.
Gable, Thomas D.
Windels, Steve K.
Ecological forecasts reveal limitations of common model selection methods: predicting changes in beaver colony densities
title Ecological forecasts reveal limitations of common model selection methods: predicting changes in beaver colony densities
title_full Ecological forecasts reveal limitations of common model selection methods: predicting changes in beaver colony densities
title_fullStr Ecological forecasts reveal limitations of common model selection methods: predicting changes in beaver colony densities
title_full_unstemmed Ecological forecasts reveal limitations of common model selection methods: predicting changes in beaver colony densities
title_short Ecological forecasts reveal limitations of common model selection methods: predicting changes in beaver colony densities
title_sort ecological forecasts reveal limitations of common model selection methods: predicting changes in beaver colony densities
topic Articles
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7816246/
https://www.ncbi.nlm.nih.gov/pubmed/32583507
http://dx.doi.org/10.1002/eap.2198
work_keys_str_mv AT johnsonbiceseanm ecologicalforecastsreveallimitationsofcommonmodelselectionmethodspredictingchangesinbeavercolonydensities
AT fergusonjakem ecologicalforecastsreveallimitationsofcommonmodelselectionmethodspredictingchangesinbeavercolonydensities
AT erbjohnd ecologicalforecastsreveallimitationsofcommonmodelselectionmethodspredictingchangesinbeavercolonydensities
AT gablethomasd ecologicalforecastsreveallimitationsofcommonmodelselectionmethodspredictingchangesinbeavercolonydensities
AT windelsstevek ecologicalforecastsreveallimitationsofcommonmodelselectionmethodspredictingchangesinbeavercolonydensities