Cargando…

Ranking versus rating in peer review of research grant applications

The allocation of public funds for research has been predominantly based on peer review where reviewers are asked to rate an application on some form of ordinal scale from poor to excellent. Poor reliability and bias of peer review rating has led funding agencies to experiment with different approac...

Descripción completa

Detalles Bibliográficos
Autores principales: Tamblyn, Robyn, Girard, Nadyne, Hanley, James, Habib, Bettina, Mota, Adrian, Khan, Karim M., Ardern, Clare L.
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Public Library of Science 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10553257/
https://www.ncbi.nlm.nih.gov/pubmed/37796852
http://dx.doi.org/10.1371/journal.pone.0292306
_version_ 1785116126740480000
author Tamblyn, Robyn
Girard, Nadyne
Hanley, James
Habib, Bettina
Mota, Adrian
Khan, Karim M.
Ardern, Clare L.
author_facet Tamblyn, Robyn
Girard, Nadyne
Hanley, James
Habib, Bettina
Mota, Adrian
Khan, Karim M.
Ardern, Clare L.
author_sort Tamblyn, Robyn
collection PubMed
description The allocation of public funds for research has been predominantly based on peer review where reviewers are asked to rate an application on some form of ordinal scale from poor to excellent. Poor reliability and bias of peer review rating has led funding agencies to experiment with different approaches to assess applications. In this study, we compared the reliability and potential sources of bias associated with application rating with those of application ranking in 3,156 applications to the Canadian Institutes of Health Research. Ranking was more reliable than rating and less susceptible to the characteristics of the review panel, such as level of expertise and experience, for both reliability and potential sources of bias. However, both rating and ranking penalized early career investigators and favoured older applicants. Sex bias was only evident for rating and only when the applicant’s H-index was at the lower end of the H-index distribution. We conclude that when compared to rating, ranking provides a more reliable assessment of the quality of research applications, is not as influenced by reviewer expertise or experience, and is associated with fewer sources of bias. Research funding agencies should consider adopting ranking methods to improve the quality of funding decisions in health research.
format Online
Article
Text
id pubmed-10553257
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher Public Library of Science
record_format MEDLINE/PubMed
spelling pubmed-105532572023-10-06 Ranking versus rating in peer review of research grant applications Tamblyn, Robyn Girard, Nadyne Hanley, James Habib, Bettina Mota, Adrian Khan, Karim M. Ardern, Clare L. PLoS One Research Article The allocation of public funds for research has been predominantly based on peer review where reviewers are asked to rate an application on some form of ordinal scale from poor to excellent. Poor reliability and bias of peer review rating has led funding agencies to experiment with different approaches to assess applications. In this study, we compared the reliability and potential sources of bias associated with application rating with those of application ranking in 3,156 applications to the Canadian Institutes of Health Research. Ranking was more reliable than rating and less susceptible to the characteristics of the review panel, such as level of expertise and experience, for both reliability and potential sources of bias. However, both rating and ranking penalized early career investigators and favoured older applicants. Sex bias was only evident for rating and only when the applicant’s H-index was at the lower end of the H-index distribution. We conclude that when compared to rating, ranking provides a more reliable assessment of the quality of research applications, is not as influenced by reviewer expertise or experience, and is associated with fewer sources of bias. Research funding agencies should consider adopting ranking methods to improve the quality of funding decisions in health research. Public Library of Science 2023-10-05 /pmc/articles/PMC10553257/ /pubmed/37796852 http://dx.doi.org/10.1371/journal.pone.0292306 Text en © 2023 Tamblyn et al https://creativecommons.org/licenses/by/4.0/This is an open access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
spellingShingle Research Article
Tamblyn, Robyn
Girard, Nadyne
Hanley, James
Habib, Bettina
Mota, Adrian
Khan, Karim M.
Ardern, Clare L.
Ranking versus rating in peer review of research grant applications
title Ranking versus rating in peer review of research grant applications
title_full Ranking versus rating in peer review of research grant applications
title_fullStr Ranking versus rating in peer review of research grant applications
title_full_unstemmed Ranking versus rating in peer review of research grant applications
title_short Ranking versus rating in peer review of research grant applications
title_sort ranking versus rating in peer review of research grant applications
topic Research Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10553257/
https://www.ncbi.nlm.nih.gov/pubmed/37796852
http://dx.doi.org/10.1371/journal.pone.0292306
work_keys_str_mv AT tamblynrobyn rankingversusratinginpeerreviewofresearchgrantapplications
AT girardnadyne rankingversusratinginpeerreviewofresearchgrantapplications
AT hanleyjames rankingversusratinginpeerreviewofresearchgrantapplications
AT habibbettina rankingversusratinginpeerreviewofresearchgrantapplications
AT motaadrian rankingversusratinginpeerreviewofresearchgrantapplications
AT khankarimm rankingversusratinginpeerreviewofresearchgrantapplications
AT ardernclarel rankingversusratinginpeerreviewofresearchgrantapplications