Cargando…
Assessing the Assessment in Emergency Care Training
OBJECTIVE: Each year over 1.5 million health care professionals attend emergency care courses. Despite high stakes for patients and extensive resources involved, little evidence exists on the quality of assessment. The aim of this study was to evaluate the validity and reliability of commonly used f...
Autores principales: | , , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Public Library of Science
2014
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4270684/ https://www.ncbi.nlm.nih.gov/pubmed/25521702 http://dx.doi.org/10.1371/journal.pone.0114663 |
_version_ | 1782349520666886144 |
---|---|
author | Dankbaar, Mary E. W. Stegers-Jager, Karen M. Baarveld, Frank van Merrienboer, Jeroen J. G. Norman, Geoff R. Rutten, Frans L. van Saase, Jan L. C. M. Schuit, Stephanie C. E. |
author_facet | Dankbaar, Mary E. W. Stegers-Jager, Karen M. Baarveld, Frank van Merrienboer, Jeroen J. G. Norman, Geoff R. Rutten, Frans L. van Saase, Jan L. C. M. Schuit, Stephanie C. E. |
author_sort | Dankbaar, Mary E. W. |
collection | PubMed |
description | OBJECTIVE: Each year over 1.5 million health care professionals attend emergency care courses. Despite high stakes for patients and extensive resources involved, little evidence exists on the quality of assessment. The aim of this study was to evaluate the validity and reliability of commonly used formats in assessing emergency care skills. METHODS: Residents were assessed at the end of a 2-week emergency course; a subgroup was videotaped. Psychometric analyses were conducted to assess the validity and inter-rater reliability of the assessment instrument, which included a checklist, a 9-item competency scale and a global performance scale. RESULTS: A group of 144 residents and 12 raters participated in the study; 22 residents were videotaped and re-assessed by 8 raters. The checklists showed limited validity and poor inter-rater reliability for the dimensions “correct” and “timely” (ICC = .30 and.39 resp.). The competency scale had good construct validity, consisting of a clinical and a communication subscale. The internal consistency of the (sub)scales was high (α = .93/.91/.86). The inter-rater reliability was moderate for the clinical competency subscale (.49) and the global performance scale (.50), but poor for the communication subscale (.27). A generalizability study showed that for a reliable assessment 5–13 raters are needed when using checklists, and four when using the clinical competency scale or the global performance scale. CONCLUSIONS: This study shows poor validity and reliability for assessing emergency skills with checklists but good validity and moderate reliability with clinical competency or global performance scales. Involving more raters can improve the reliability substantially. Recommendations are made to improve this high stakes skill assessment. |
format | Online Article Text |
id | pubmed-4270684 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2014 |
publisher | Public Library of Science |
record_format | MEDLINE/PubMed |
spelling | pubmed-42706842014-12-26 Assessing the Assessment in Emergency Care Training Dankbaar, Mary E. W. Stegers-Jager, Karen M. Baarveld, Frank van Merrienboer, Jeroen J. G. Norman, Geoff R. Rutten, Frans L. van Saase, Jan L. C. M. Schuit, Stephanie C. E. PLoS One Research Article OBJECTIVE: Each year over 1.5 million health care professionals attend emergency care courses. Despite high stakes for patients and extensive resources involved, little evidence exists on the quality of assessment. The aim of this study was to evaluate the validity and reliability of commonly used formats in assessing emergency care skills. METHODS: Residents were assessed at the end of a 2-week emergency course; a subgroup was videotaped. Psychometric analyses were conducted to assess the validity and inter-rater reliability of the assessment instrument, which included a checklist, a 9-item competency scale and a global performance scale. RESULTS: A group of 144 residents and 12 raters participated in the study; 22 residents were videotaped and re-assessed by 8 raters. The checklists showed limited validity and poor inter-rater reliability for the dimensions “correct” and “timely” (ICC = .30 and.39 resp.). The competency scale had good construct validity, consisting of a clinical and a communication subscale. The internal consistency of the (sub)scales was high (α = .93/.91/.86). The inter-rater reliability was moderate for the clinical competency subscale (.49) and the global performance scale (.50), but poor for the communication subscale (.27). A generalizability study showed that for a reliable assessment 5–13 raters are needed when using checklists, and four when using the clinical competency scale or the global performance scale. CONCLUSIONS: This study shows poor validity and reliability for assessing emergency skills with checklists but good validity and moderate reliability with clinical competency or global performance scales. Involving more raters can improve the reliability substantially. Recommendations are made to improve this high stakes skill assessment. Public Library of Science 2014-12-18 /pmc/articles/PMC4270684/ /pubmed/25521702 http://dx.doi.org/10.1371/journal.pone.0114663 Text en © 2014 Dankbaar et al http://creativecommons.org/licenses/by/4.0/ This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are properly credited. |
spellingShingle | Research Article Dankbaar, Mary E. W. Stegers-Jager, Karen M. Baarveld, Frank van Merrienboer, Jeroen J. G. Norman, Geoff R. Rutten, Frans L. van Saase, Jan L. C. M. Schuit, Stephanie C. E. Assessing the Assessment in Emergency Care Training |
title | Assessing the Assessment in Emergency Care Training |
title_full | Assessing the Assessment in Emergency Care Training |
title_fullStr | Assessing the Assessment in Emergency Care Training |
title_full_unstemmed | Assessing the Assessment in Emergency Care Training |
title_short | Assessing the Assessment in Emergency Care Training |
title_sort | assessing the assessment in emergency care training |
topic | Research Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4270684/ https://www.ncbi.nlm.nih.gov/pubmed/25521702 http://dx.doi.org/10.1371/journal.pone.0114663 |
work_keys_str_mv | AT dankbaarmaryew assessingtheassessmentinemergencycaretraining AT stegersjagerkarenm assessingtheassessmentinemergencycaretraining AT baarveldfrank assessingtheassessmentinemergencycaretraining AT vanmerrienboerjeroenjg assessingtheassessmentinemergencycaretraining AT normangeoffr assessingtheassessmentinemergencycaretraining AT ruttenfransl assessingtheassessmentinemergencycaretraining AT vansaasejanlcm assessingtheassessmentinemergencycaretraining AT schuitstephaniece assessingtheassessmentinemergencycaretraining |