Cargando…

A generalized approach for producing, quantifying, and validating citizen science data from wildlife images

Citizen science has the potential to expand the scope and scale of research in ecology and conservation, but many professional researchers remain skeptical of data produced by nonexperts. We devised an approach for producing accurate, reliable data from untrained, nonexpert volunteers. On the citize...

Descripción completa

Detalles Bibliográficos
Autores principales: Swanson, Alexandra, Kosmala, Margaret, Lintott, Chris, Packer, Craig
Formato: Online Artículo Texto
Lenguaje:English
Publicado: John Wiley and Sons Inc. 2016
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4999033/
https://www.ncbi.nlm.nih.gov/pubmed/27111678
http://dx.doi.org/10.1111/cobi.12695
_version_ 1782450047260033024
author Swanson, Alexandra
Kosmala, Margaret
Lintott, Chris
Packer, Craig
author_facet Swanson, Alexandra
Kosmala, Margaret
Lintott, Chris
Packer, Craig
author_sort Swanson, Alexandra
collection PubMed
description Citizen science has the potential to expand the scope and scale of research in ecology and conservation, but many professional researchers remain skeptical of data produced by nonexperts. We devised an approach for producing accurate, reliable data from untrained, nonexpert volunteers. On the citizen science website www.snapshotserengeti.org, more than 28,000 volunteers classified 1.51 million images taken in a large‐scale camera‐trap survey in Serengeti National Park, Tanzania. Each image was circulated to, on average, 27 volunteers, and their classifications were aggregated using a simple plurality algorithm. We validated the aggregated answers against a data set of 3829 images verified by experts and calculated 3 certainty metrics—level of agreement among classifications (evenness), fraction of classifications supporting the aggregated answer (fraction support), and fraction of classifiers who reported “nothing here” for an image that was ultimately classified as containing an animal (fraction blank)—to measure confidence that an aggregated answer was correct. Overall, aggregated volunteer answers agreed with the expert‐verified data on 98% of images, but accuracy differed by species commonness such that rare species had higher rates of false positives and false negatives. Easily calculated analysis of variance and post‐hoc Tukey tests indicated that the certainty metrics were significant indicators of whether each image was correctly classified or classifiable. Thus, the certainty metrics can be used to identify images for expert review. Bootstrapping analyses further indicated that 90% of images were correctly classified with just 5 volunteers per image. Species classifications based on the plurality vote of multiple citizen scientists can provide a reliable foundation for large‐scale monitoring of African wildlife.
format Online
Article
Text
id pubmed-4999033
institution National Center for Biotechnology Information
language English
publishDate 2016
publisher John Wiley and Sons Inc.
record_format MEDLINE/PubMed
spelling pubmed-49990332016-09-13 A generalized approach for producing, quantifying, and validating citizen science data from wildlife images Swanson, Alexandra Kosmala, Margaret Lintott, Chris Packer, Craig Conserv Biol Special Section: Moving from Citizen to Civic Science to AddressWicked Conservation Problems Citizen science has the potential to expand the scope and scale of research in ecology and conservation, but many professional researchers remain skeptical of data produced by nonexperts. We devised an approach for producing accurate, reliable data from untrained, nonexpert volunteers. On the citizen science website www.snapshotserengeti.org, more than 28,000 volunteers classified 1.51 million images taken in a large‐scale camera‐trap survey in Serengeti National Park, Tanzania. Each image was circulated to, on average, 27 volunteers, and their classifications were aggregated using a simple plurality algorithm. We validated the aggregated answers against a data set of 3829 images verified by experts and calculated 3 certainty metrics—level of agreement among classifications (evenness), fraction of classifications supporting the aggregated answer (fraction support), and fraction of classifiers who reported “nothing here” for an image that was ultimately classified as containing an animal (fraction blank)—to measure confidence that an aggregated answer was correct. Overall, aggregated volunteer answers agreed with the expert‐verified data on 98% of images, but accuracy differed by species commonness such that rare species had higher rates of false positives and false negatives. Easily calculated analysis of variance and post‐hoc Tukey tests indicated that the certainty metrics were significant indicators of whether each image was correctly classified or classifiable. Thus, the certainty metrics can be used to identify images for expert review. Bootstrapping analyses further indicated that 90% of images were correctly classified with just 5 volunteers per image. Species classifications based on the plurality vote of multiple citizen scientists can provide a reliable foundation for large‐scale monitoring of African wildlife. John Wiley and Sons Inc. 2016-04-25 2016-06 /pmc/articles/PMC4999033/ /pubmed/27111678 http://dx.doi.org/10.1111/cobi.12695 Text en © 2016 The Authors. Conservation Biology published by Wiley Periodicals, Inc. on behalf of Society for Conservation Biology. This is an open access article under the terms of the Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/) License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.
spellingShingle Special Section: Moving from Citizen to Civic Science to AddressWicked Conservation Problems
Swanson, Alexandra
Kosmala, Margaret
Lintott, Chris
Packer, Craig
A generalized approach for producing, quantifying, and validating citizen science data from wildlife images
title A generalized approach for producing, quantifying, and validating citizen science data from wildlife images
title_full A generalized approach for producing, quantifying, and validating citizen science data from wildlife images
title_fullStr A generalized approach for producing, quantifying, and validating citizen science data from wildlife images
title_full_unstemmed A generalized approach for producing, quantifying, and validating citizen science data from wildlife images
title_short A generalized approach for producing, quantifying, and validating citizen science data from wildlife images
title_sort generalized approach for producing, quantifying, and validating citizen science data from wildlife images
topic Special Section: Moving from Citizen to Civic Science to AddressWicked Conservation Problems
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4999033/
https://www.ncbi.nlm.nih.gov/pubmed/27111678
http://dx.doi.org/10.1111/cobi.12695
work_keys_str_mv AT swansonalexandra ageneralizedapproachforproducingquantifyingandvalidatingcitizensciencedatafromwildlifeimages
AT kosmalamargaret ageneralizedapproachforproducingquantifyingandvalidatingcitizensciencedatafromwildlifeimages
AT lintottchris ageneralizedapproachforproducingquantifyingandvalidatingcitizensciencedatafromwildlifeimages
AT packercraig ageneralizedapproachforproducingquantifyingandvalidatingcitizensciencedatafromwildlifeimages
AT swansonalexandra generalizedapproachforproducingquantifyingandvalidatingcitizensciencedatafromwildlifeimages
AT kosmalamargaret generalizedapproachforproducingquantifyingandvalidatingcitizensciencedatafromwildlifeimages
AT lintottchris generalizedapproachforproducingquantifyingandvalidatingcitizensciencedatafromwildlifeimages
AT packercraig generalizedapproachforproducingquantifyingandvalidatingcitizensciencedatafromwildlifeimages