Cargando…

An experimental test of the effects of redacting grant applicant identifiers on peer review outcomes

BACKGROUND: Blinding reviewers to applicant identity has been proposed to reduce bias in peer review. METHODS: This experimental test used 1200 NIH grant applications, 400 from Black investigators, 400 matched applications from White investigators, and 400 randomly selected applications from White i...

Descripción completa

Detalles Bibliográficos
Autores principales: Nakamura, Richard K, Mann, Lee S, Lindner, Mark D, Braithwaite, Jeremy, Chen, Mei-Ching, Vancea, Adrian, Byrnes, Noni, Durrant, Valerie, Reed, Bruce
Formato: Online Artículo Texto
Lenguaje:English
Publicado: eLife Sciences Publications, Ltd 2021
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8612703/
https://www.ncbi.nlm.nih.gov/pubmed/34665132
http://dx.doi.org/10.7554/eLife.71368
_version_ 1784603497945104384
author Nakamura, Richard K
Mann, Lee S
Lindner, Mark D
Braithwaite, Jeremy
Chen, Mei-Ching
Vancea, Adrian
Byrnes, Noni
Durrant, Valerie
Reed, Bruce
author_facet Nakamura, Richard K
Mann, Lee S
Lindner, Mark D
Braithwaite, Jeremy
Chen, Mei-Ching
Vancea, Adrian
Byrnes, Noni
Durrant, Valerie
Reed, Bruce
author_sort Nakamura, Richard K
collection PubMed
description BACKGROUND: Blinding reviewers to applicant identity has been proposed to reduce bias in peer review. METHODS: This experimental test used 1200 NIH grant applications, 400 from Black investigators, 400 matched applications from White investigators, and 400 randomly selected applications from White investigators. Applications were reviewed by mail in standard and redacted formats. RESULTS: Redaction reduced, but did not eliminate, reviewers’ ability to correctly guess features of identity. The primary, preregistered analysis hypothesized a differential effect of redaction according to investigator race in the matched applications. A set of secondary analyses (not preregistered) used the randomly selected applications from White scientists and tested the same interaction. Both analyses revealed similar effects: Standard format applications from White investigators scored better than those from Black investigators. Redaction cut the size of the difference by about half (e.g. from a Cohen’s d of 0.20–0.10 in matched applications); redaction caused applications from White scientists to score worse but had no effect on scores for Black applications. CONCLUSIONS: Grant-writing considerations and halo effects are discussed as competing explanations for this pattern. The findings support further evaluation of peer review models that diminish the influence of applicant identity. FUNDING: Funding was provided by the NIH.
format Online
Article
Text
id pubmed-8612703
institution National Center for Biotechnology Information
language English
publishDate 2021
publisher eLife Sciences Publications, Ltd
record_format MEDLINE/PubMed
spelling pubmed-86127032021-11-26 An experimental test of the effects of redacting grant applicant identifiers on peer review outcomes Nakamura, Richard K Mann, Lee S Lindner, Mark D Braithwaite, Jeremy Chen, Mei-Ching Vancea, Adrian Byrnes, Noni Durrant, Valerie Reed, Bruce eLife Cell Biology BACKGROUND: Blinding reviewers to applicant identity has been proposed to reduce bias in peer review. METHODS: This experimental test used 1200 NIH grant applications, 400 from Black investigators, 400 matched applications from White investigators, and 400 randomly selected applications from White investigators. Applications were reviewed by mail in standard and redacted formats. RESULTS: Redaction reduced, but did not eliminate, reviewers’ ability to correctly guess features of identity. The primary, preregistered analysis hypothesized a differential effect of redaction according to investigator race in the matched applications. A set of secondary analyses (not preregistered) used the randomly selected applications from White scientists and tested the same interaction. Both analyses revealed similar effects: Standard format applications from White investigators scored better than those from Black investigators. Redaction cut the size of the difference by about half (e.g. from a Cohen’s d of 0.20–0.10 in matched applications); redaction caused applications from White scientists to score worse but had no effect on scores for Black applications. CONCLUSIONS: Grant-writing considerations and halo effects are discussed as competing explanations for this pattern. The findings support further evaluation of peer review models that diminish the influence of applicant identity. FUNDING: Funding was provided by the NIH. eLife Sciences Publications, Ltd 2021-10-19 /pmc/articles/PMC8612703/ /pubmed/34665132 http://dx.doi.org/10.7554/eLife.71368 Text en https://creativecommons.org/publicdomain/zero/1.0/This is an open-access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication (https://creativecommons.org/publicdomain/zero/1.0/) .
spellingShingle Cell Biology
Nakamura, Richard K
Mann, Lee S
Lindner, Mark D
Braithwaite, Jeremy
Chen, Mei-Ching
Vancea, Adrian
Byrnes, Noni
Durrant, Valerie
Reed, Bruce
An experimental test of the effects of redacting grant applicant identifiers on peer review outcomes
title An experimental test of the effects of redacting grant applicant identifiers on peer review outcomes
title_full An experimental test of the effects of redacting grant applicant identifiers on peer review outcomes
title_fullStr An experimental test of the effects of redacting grant applicant identifiers on peer review outcomes
title_full_unstemmed An experimental test of the effects of redacting grant applicant identifiers on peer review outcomes
title_short An experimental test of the effects of redacting grant applicant identifiers on peer review outcomes
title_sort experimental test of the effects of redacting grant applicant identifiers on peer review outcomes
topic Cell Biology
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8612703/
https://www.ncbi.nlm.nih.gov/pubmed/34665132
http://dx.doi.org/10.7554/eLife.71368
work_keys_str_mv AT nakamurarichardk anexperimentaltestoftheeffectsofredactinggrantapplicantidentifiersonpeerreviewoutcomes
AT mannlees anexperimentaltestoftheeffectsofredactinggrantapplicantidentifiersonpeerreviewoutcomes
AT lindnermarkd anexperimentaltestoftheeffectsofredactinggrantapplicantidentifiersonpeerreviewoutcomes
AT braithwaitejeremy anexperimentaltestoftheeffectsofredactinggrantapplicantidentifiersonpeerreviewoutcomes
AT chenmeiching anexperimentaltestoftheeffectsofredactinggrantapplicantidentifiersonpeerreviewoutcomes
AT vanceaadrian anexperimentaltestoftheeffectsofredactinggrantapplicantidentifiersonpeerreviewoutcomes
AT byrnesnoni anexperimentaltestoftheeffectsofredactinggrantapplicantidentifiersonpeerreviewoutcomes
AT durrantvalerie anexperimentaltestoftheeffectsofredactinggrantapplicantidentifiersonpeerreviewoutcomes
AT reedbruce anexperimentaltestoftheeffectsofredactinggrantapplicantidentifiersonpeerreviewoutcomes
AT nakamurarichardk experimentaltestoftheeffectsofredactinggrantapplicantidentifiersonpeerreviewoutcomes
AT mannlees experimentaltestoftheeffectsofredactinggrantapplicantidentifiersonpeerreviewoutcomes
AT lindnermarkd experimentaltestoftheeffectsofredactinggrantapplicantidentifiersonpeerreviewoutcomes
AT braithwaitejeremy experimentaltestoftheeffectsofredactinggrantapplicantidentifiersonpeerreviewoutcomes
AT chenmeiching experimentaltestoftheeffectsofredactinggrantapplicantidentifiersonpeerreviewoutcomes
AT vanceaadrian experimentaltestoftheeffectsofredactinggrantapplicantidentifiersonpeerreviewoutcomes
AT byrnesnoni experimentaltestoftheeffectsofredactinggrantapplicantidentifiersonpeerreviewoutcomes
AT durrantvalerie experimentaltestoftheeffectsofredactinggrantapplicantidentifiersonpeerreviewoutcomes
AT reedbruce experimentaltestoftheeffectsofredactinggrantapplicantidentifiersonpeerreviewoutcomes