Cargando…
Grant Peer Review: Improving Inter-Rater Reliability with Training
This study developed and evaluated a brief training program for grant reviewers that aimed to increase inter-rater reliability, rating scale knowledge, and effort to read the grant review criteria. Enhancing reviewer training may improve the reliability and accuracy of research grant proposal scorin...
Autores principales: | , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Public Library of Science
2015
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4468126/ https://www.ncbi.nlm.nih.gov/pubmed/26075884 http://dx.doi.org/10.1371/journal.pone.0130450 |
_version_ | 1782376445334519808 |
---|---|
author | Sattler, David N. McKnight, Patrick E. Naney, Linda Mathis, Randy |
author_facet | Sattler, David N. McKnight, Patrick E. Naney, Linda Mathis, Randy |
author_sort | Sattler, David N. |
collection | PubMed |
description | This study developed and evaluated a brief training program for grant reviewers that aimed to increase inter-rater reliability, rating scale knowledge, and effort to read the grant review criteria. Enhancing reviewer training may improve the reliability and accuracy of research grant proposal scoring and funding recommendations. Seventy-five Public Health professors from U.S. research universities watched the training video we produced and assigned scores to the National Institutes of Health scoring criteria proposal summary descriptions. For both novice and experienced reviewers, the training video increased scoring accuracy (the percentage of scores that reflect the true rating scale values), inter-rater reliability, and the amount of time reading the review criteria compared to the no video condition. The increase in reliability for experienced reviewers is notable because it is commonly assumed that reviewers—especially those with experience—have good understanding of the grant review rating scale. The findings suggest that both experienced and novice reviewers who had not received the type of training developed in this study may not have appropriate understanding of the definitions and meaning for each value of the rating scale and that experienced reviewers may overestimate their knowledge of the rating scale. The results underscore the benefits of and need for specialized peer reviewer training. |
format | Online Article Text |
id | pubmed-4468126 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2015 |
publisher | Public Library of Science |
record_format | MEDLINE/PubMed |
spelling | pubmed-44681262015-06-25 Grant Peer Review: Improving Inter-Rater Reliability with Training Sattler, David N. McKnight, Patrick E. Naney, Linda Mathis, Randy PLoS One Research Article This study developed and evaluated a brief training program for grant reviewers that aimed to increase inter-rater reliability, rating scale knowledge, and effort to read the grant review criteria. Enhancing reviewer training may improve the reliability and accuracy of research grant proposal scoring and funding recommendations. Seventy-five Public Health professors from U.S. research universities watched the training video we produced and assigned scores to the National Institutes of Health scoring criteria proposal summary descriptions. For both novice and experienced reviewers, the training video increased scoring accuracy (the percentage of scores that reflect the true rating scale values), inter-rater reliability, and the amount of time reading the review criteria compared to the no video condition. The increase in reliability for experienced reviewers is notable because it is commonly assumed that reviewers—especially those with experience—have good understanding of the grant review rating scale. The findings suggest that both experienced and novice reviewers who had not received the type of training developed in this study may not have appropriate understanding of the definitions and meaning for each value of the rating scale and that experienced reviewers may overestimate their knowledge of the rating scale. The results underscore the benefits of and need for specialized peer reviewer training. Public Library of Science 2015-06-15 /pmc/articles/PMC4468126/ /pubmed/26075884 http://dx.doi.org/10.1371/journal.pone.0130450 Text en © 2015 Sattler et al http://creativecommons.org/licenses/by/4.0/ This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are properly credited. |
spellingShingle | Research Article Sattler, David N. McKnight, Patrick E. Naney, Linda Mathis, Randy Grant Peer Review: Improving Inter-Rater Reliability with Training |
title | Grant Peer Review: Improving Inter-Rater Reliability with Training |
title_full | Grant Peer Review: Improving Inter-Rater Reliability with Training |
title_fullStr | Grant Peer Review: Improving Inter-Rater Reliability with Training |
title_full_unstemmed | Grant Peer Review: Improving Inter-Rater Reliability with Training |
title_short | Grant Peer Review: Improving Inter-Rater Reliability with Training |
title_sort | grant peer review: improving inter-rater reliability with training |
topic | Research Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4468126/ https://www.ncbi.nlm.nih.gov/pubmed/26075884 http://dx.doi.org/10.1371/journal.pone.0130450 |
work_keys_str_mv | AT sattlerdavidn grantpeerreviewimprovinginterraterreliabilitywithtraining AT mcknightpatricke grantpeerreviewimprovinginterraterreliabilitywithtraining AT naneylinda grantpeerreviewimprovinginterraterreliabilitywithtraining AT mathisrandy grantpeerreviewimprovinginterraterreliabilitywithtraining |