Cargando…

Reliability testing of a portfolio assessment tool for postgraduate family medicine training in South Africa

BACKGROUND: Competency-based education and the validity and reliability of workplace-based assessment of postgraduate trainees have received increasing attention worldwide. Family medicine was recognised as a speciality in South Africa six years ago and a satisfactory portfolio of learning is a prer...

Descripción completa

Detalles Bibliográficos
Autores principales: Jenkins, Louis, Mash, Bob, Derese, Anselme
Formato: Online Artículo Texto
Lenguaje:English
Publicado: AOSIS OpenJournals 2013
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4502840/
http://dx.doi.org/10.4102/phcfm.v5i1.577
_version_ 1782381260847448064
author Jenkins, Louis
Mash, Bob
Derese, Anselme
author_facet Jenkins, Louis
Mash, Bob
Derese, Anselme
author_sort Jenkins, Louis
collection PubMed
description BACKGROUND: Competency-based education and the validity and reliability of workplace-based assessment of postgraduate trainees have received increasing attention worldwide. Family medicine was recognised as a speciality in South Africa six years ago and a satisfactory portfolio of learning is a prerequisite to sit the national exit exam. A massive scaling up of the number of family physicians is needed in order to meet the health needs of the country. AIM: The aim of this study was to develop a reliable, robust and feasible portfolio assessment tool (PAT) for South Africa. METHODS: Six raters each rated nine portfolios from the Stellenbosch University programme, using the PAT, to test for inter-rater reliability. This rating was repeated three months later to determine test–retest reliability. Following initial analysis and feedback the PAT was modified and the inter-rater reliability again assessed on nine new portfolios. An acceptable intra-class correlation was considered to be > 0.80. RESULTS: The total score was found to be reliable, with a coefficient of 0.92. For test–retest reliability, the difference in mean total score was 1.7%, which was not statistically significant. Amongst the subsections, only assessment of the educational meetings and the logbook showed reliability coefficients > 0.80. CONCLUSION: This was the first attempt to develop a reliable, robust and feasible national portfolio assessment tool to assess postgraduate family medicine training in the South African context. The tool was reliable for the total score, but the low reliability of several sections in the PAT helped us to develop 12 recommendations regarding the use of the portfolio, the design of the PAT and the training of raters.
format Online
Article
Text
id pubmed-4502840
institution National Center for Biotechnology Information
language English
publishDate 2013
publisher AOSIS OpenJournals
record_format MEDLINE/PubMed
spelling pubmed-45028402016-02-03 Reliability testing of a portfolio assessment tool for postgraduate family medicine training in South Africa Jenkins, Louis Mash, Bob Derese, Anselme Afr J Prim Health Care Fam Med Original Research BACKGROUND: Competency-based education and the validity and reliability of workplace-based assessment of postgraduate trainees have received increasing attention worldwide. Family medicine was recognised as a speciality in South Africa six years ago and a satisfactory portfolio of learning is a prerequisite to sit the national exit exam. A massive scaling up of the number of family physicians is needed in order to meet the health needs of the country. AIM: The aim of this study was to develop a reliable, robust and feasible portfolio assessment tool (PAT) for South Africa. METHODS: Six raters each rated nine portfolios from the Stellenbosch University programme, using the PAT, to test for inter-rater reliability. This rating was repeated three months later to determine test–retest reliability. Following initial analysis and feedback the PAT was modified and the inter-rater reliability again assessed on nine new portfolios. An acceptable intra-class correlation was considered to be > 0.80. RESULTS: The total score was found to be reliable, with a coefficient of 0.92. For test–retest reliability, the difference in mean total score was 1.7%, which was not statistically significant. Amongst the subsections, only assessment of the educational meetings and the logbook showed reliability coefficients > 0.80. CONCLUSION: This was the first attempt to develop a reliable, robust and feasible national portfolio assessment tool to assess postgraduate family medicine training in the South African context. The tool was reliable for the total score, but the low reliability of several sections in the PAT helped us to develop 12 recommendations regarding the use of the portfolio, the design of the PAT and the training of raters. AOSIS OpenJournals 2013-11-14 /pmc/articles/PMC4502840/ http://dx.doi.org/10.4102/phcfm.v5i1.577 Text en © 2013. The Authors http://creativecommons.org/licenses/by/2.0/ AOSIS OpenJournals. This work is licensed under the Creative Commons Attribution License.
spellingShingle Original Research
Jenkins, Louis
Mash, Bob
Derese, Anselme
Reliability testing of a portfolio assessment tool for postgraduate family medicine training in South Africa
title Reliability testing of a portfolio assessment tool for postgraduate family medicine training in South Africa
title_full Reliability testing of a portfolio assessment tool for postgraduate family medicine training in South Africa
title_fullStr Reliability testing of a portfolio assessment tool for postgraduate family medicine training in South Africa
title_full_unstemmed Reliability testing of a portfolio assessment tool for postgraduate family medicine training in South Africa
title_short Reliability testing of a portfolio assessment tool for postgraduate family medicine training in South Africa
title_sort reliability testing of a portfolio assessment tool for postgraduate family medicine training in south africa
topic Original Research
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4502840/
http://dx.doi.org/10.4102/phcfm.v5i1.577
work_keys_str_mv AT jenkinslouis reliabilitytestingofaportfolioassessmenttoolforpostgraduatefamilymedicinetraininginsouthafrica
AT mashbob reliabilitytestingofaportfolioassessmenttoolforpostgraduatefamilymedicinetraininginsouthafrica
AT dereseanselme reliabilitytestingofaportfolioassessmenttoolforpostgraduatefamilymedicinetraininginsouthafrica