Cargando…
Peer Review of Grant Applications: Criteria Used and Qualitative Study of Reviewer Practices
BACKGROUND: Peer review of grant applications has been criticized as lacking reliability. Studies showing poor agreement among reviewers supported this possibility but usually focused on reviewers’ scores and failed to investigate reasons for disagreement. Here, our goal was to determine how reviewe...
Autores principales: | , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Public Library of Science
2012
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3460995/ https://www.ncbi.nlm.nih.gov/pubmed/23029386 http://dx.doi.org/10.1371/journal.pone.0046054 |
_version_ | 1782245027056975872 |
---|---|
author | Abdoul, Hendy Perrey, Christophe Amiel, Philippe Tubach, Florence Gottot, Serge Durand-Zaleski, Isabelle Alberti, Corinne |
author_facet | Abdoul, Hendy Perrey, Christophe Amiel, Philippe Tubach, Florence Gottot, Serge Durand-Zaleski, Isabelle Alberti, Corinne |
author_sort | Abdoul, Hendy |
collection | PubMed |
description | BACKGROUND: Peer review of grant applications has been criticized as lacking reliability. Studies showing poor agreement among reviewers supported this possibility but usually focused on reviewers’ scores and failed to investigate reasons for disagreement. Here, our goal was to determine how reviewers rate applications, by investigating reviewer practices and grant assessment criteria. METHODS AND FINDINGS: We first collected and analyzed a convenience sample of French and international calls for proposals and assessment guidelines, from which we created an overall typology of assessment criteria comprising nine domains relevance to the call for proposals, usefulness, originality, innovativeness, methodology, feasibility, funding, ethical aspects, and writing of the grant application. We then performed a qualitative study of reviewer practices, particularly regarding the use of assessment criteria, among reviewers of the French Academic Hospital Research Grant Agencies (Programmes Hospitaliers de Recherche Clinique, PHRCs). Semi-structured interviews and observation sessions were conducted. Both the time spent assessing each grant application and the assessment methods varied across reviewers. The assessment criteria recommended by the PHRCs were listed by all reviewers as frequently evaluated and useful. However, use of the PHRC criteria was subjective and varied across reviewers. Some reviewers gave the same weight to each assessment criterion, whereas others considered originality to be the most important criterion (12/34), followed by methodology (10/34) and feasibility (4/34). Conceivably, this variability might adversely affect the reliability of the review process, and studies evaluating this hypothesis would be of interest. CONCLUSIONS: Variability across reviewers may result in mistrust among grant applicants about the review process. Consequently, ensuring transparency is of the utmost importance. Consistency in the review process could also be improved by providing common definitions for each assessment criterion and uniform requirements for grant application submissions. Further research is needed to assess the feasibility and acceptability of these measures. |
format | Online Article Text |
id | pubmed-3460995 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2012 |
publisher | Public Library of Science |
record_format | MEDLINE/PubMed |
spelling | pubmed-34609952012-10-01 Peer Review of Grant Applications: Criteria Used and Qualitative Study of Reviewer Practices Abdoul, Hendy Perrey, Christophe Amiel, Philippe Tubach, Florence Gottot, Serge Durand-Zaleski, Isabelle Alberti, Corinne PLoS One Research Article BACKGROUND: Peer review of grant applications has been criticized as lacking reliability. Studies showing poor agreement among reviewers supported this possibility but usually focused on reviewers’ scores and failed to investigate reasons for disagreement. Here, our goal was to determine how reviewers rate applications, by investigating reviewer practices and grant assessment criteria. METHODS AND FINDINGS: We first collected and analyzed a convenience sample of French and international calls for proposals and assessment guidelines, from which we created an overall typology of assessment criteria comprising nine domains relevance to the call for proposals, usefulness, originality, innovativeness, methodology, feasibility, funding, ethical aspects, and writing of the grant application. We then performed a qualitative study of reviewer practices, particularly regarding the use of assessment criteria, among reviewers of the French Academic Hospital Research Grant Agencies (Programmes Hospitaliers de Recherche Clinique, PHRCs). Semi-structured interviews and observation sessions were conducted. Both the time spent assessing each grant application and the assessment methods varied across reviewers. The assessment criteria recommended by the PHRCs were listed by all reviewers as frequently evaluated and useful. However, use of the PHRC criteria was subjective and varied across reviewers. Some reviewers gave the same weight to each assessment criterion, whereas others considered originality to be the most important criterion (12/34), followed by methodology (10/34) and feasibility (4/34). Conceivably, this variability might adversely affect the reliability of the review process, and studies evaluating this hypothesis would be of interest. CONCLUSIONS: Variability across reviewers may result in mistrust among grant applicants about the review process. Consequently, ensuring transparency is of the utmost importance. Consistency in the review process could also be improved by providing common definitions for each assessment criterion and uniform requirements for grant application submissions. Further research is needed to assess the feasibility and acceptability of these measures. Public Library of Science 2012-09-28 /pmc/articles/PMC3460995/ /pubmed/23029386 http://dx.doi.org/10.1371/journal.pone.0046054 Text en © 2012 Abdoul et al http://creativecommons.org/licenses/by/4.0/ This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are properly credited. |
spellingShingle | Research Article Abdoul, Hendy Perrey, Christophe Amiel, Philippe Tubach, Florence Gottot, Serge Durand-Zaleski, Isabelle Alberti, Corinne Peer Review of Grant Applications: Criteria Used and Qualitative Study of Reviewer Practices |
title | Peer Review of Grant Applications: Criteria Used and Qualitative Study of Reviewer Practices |
title_full | Peer Review of Grant Applications: Criteria Used and Qualitative Study of Reviewer Practices |
title_fullStr | Peer Review of Grant Applications: Criteria Used and Qualitative Study of Reviewer Practices |
title_full_unstemmed | Peer Review of Grant Applications: Criteria Used and Qualitative Study of Reviewer Practices |
title_short | Peer Review of Grant Applications: Criteria Used and Qualitative Study of Reviewer Practices |
title_sort | peer review of grant applications: criteria used and qualitative study of reviewer practices |
topic | Research Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3460995/ https://www.ncbi.nlm.nih.gov/pubmed/23029386 http://dx.doi.org/10.1371/journal.pone.0046054 |
work_keys_str_mv | AT abdoulhendy peerreviewofgrantapplicationscriteriausedandqualitativestudyofreviewerpractices AT perreychristophe peerreviewofgrantapplicationscriteriausedandqualitativestudyofreviewerpractices AT amielphilippe peerreviewofgrantapplicationscriteriausedandqualitativestudyofreviewerpractices AT tubachflorence peerreviewofgrantapplicationscriteriausedandqualitativestudyofreviewerpractices AT gottotserge peerreviewofgrantapplicationscriteriausedandqualitativestudyofreviewerpractices AT durandzaleskiisabelle peerreviewofgrantapplicationscriteriausedandqualitativestudyofreviewerpractices AT alberticorinne peerreviewofgrantapplicationscriteriausedandqualitativestudyofreviewerpractices |