Cargando…

Pitfalls and Important Issues in Testing Reliability Using Intraclass Correlation Coefficients in Orthopaedic Research

BACKGROUND: Intra-class correlation coefficients (ICCs) provide a statistical means of testing the reliability. However, their interpretation is not well documented in the orthopedic field. The purpose of this study was to investigate the use of ICCs in the orthopedic literature and to demonstrate p...

Descripción completa

Detalles Bibliográficos
Autores principales: Lee, Kyoung Min, Lee, Jaebong, Chung, Chin Youb, Ahn, Soyeon, Sung, Ki Hyuk, Kim, Tae Won, Lee, Hui Jong, Park, Moon Seok
Formato: Online Artículo Texto
Lenguaje:English
Publicado: The Korean Orthopaedic Association 2012
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3360188/
https://www.ncbi.nlm.nih.gov/pubmed/22662301
http://dx.doi.org/10.4055/cios.2012.4.2.149
_version_ 1782233958562398208
author Lee, Kyoung Min
Lee, Jaebong
Chung, Chin Youb
Ahn, Soyeon
Sung, Ki Hyuk
Kim, Tae Won
Lee, Hui Jong
Park, Moon Seok
author_facet Lee, Kyoung Min
Lee, Jaebong
Chung, Chin Youb
Ahn, Soyeon
Sung, Ki Hyuk
Kim, Tae Won
Lee, Hui Jong
Park, Moon Seok
author_sort Lee, Kyoung Min
collection PubMed
description BACKGROUND: Intra-class correlation coefficients (ICCs) provide a statistical means of testing the reliability. However, their interpretation is not well documented in the orthopedic field. The purpose of this study was to investigate the use of ICCs in the orthopedic literature and to demonstrate pitfalls regarding their use. METHODS: First, orthopedic articles that used ICCs were retrieved from the Pubmed database, and journal demography, ICC models and concurrent statistics used were evaluated. Second, reliability test was performed on three common physical examinations in cerebral palsy, namely, the Thomas test, the Staheli test, and popliteal angle measurement. Thirty patients were assessed by three orthopedic surgeons to explore the statistical methods testing reliability. Third, the factors affecting the ICC values were examined by simulating the data sets based on the physical examination data where the ranges, slopes, and interobserver variability were modified. RESULTS: Of the 92 orthopedic articles identified, 58 articles (63%) did not clarify the ICC model used, and only 5 articles (5%) described all models, types, and measures. In reliability testing, although the popliteal angle showed a larger mean absolute difference than the Thomas test and the Staheli test, the ICC of popliteal angle was higher, which was believed to be contrary to the context of measurement. In addition, the ICC values were affected by the model, type, and measures used. In simulated data sets, the ICC showed higher values when the range of data sets were larger, the slopes of the data sets were parallel, and the interobserver variability was smaller. CONCLUSIONS: Care should be taken when interpreting the absolute ICC values, i.e., a higher ICC does not necessarily mean less variability because the ICC values can also be affected by various factors. The authors recommend that researchers clarify ICC models used and ICC values are interpreted in the context of measurement.
format Online
Article
Text
id pubmed-3360188
institution National Center for Biotechnology Information
language English
publishDate 2012
publisher The Korean Orthopaedic Association
record_format MEDLINE/PubMed
spelling pubmed-33601882012-06-02 Pitfalls and Important Issues in Testing Reliability Using Intraclass Correlation Coefficients in Orthopaedic Research Lee, Kyoung Min Lee, Jaebong Chung, Chin Youb Ahn, Soyeon Sung, Ki Hyuk Kim, Tae Won Lee, Hui Jong Park, Moon Seok Clin Orthop Surg Original Article BACKGROUND: Intra-class correlation coefficients (ICCs) provide a statistical means of testing the reliability. However, their interpretation is not well documented in the orthopedic field. The purpose of this study was to investigate the use of ICCs in the orthopedic literature and to demonstrate pitfalls regarding their use. METHODS: First, orthopedic articles that used ICCs were retrieved from the Pubmed database, and journal demography, ICC models and concurrent statistics used were evaluated. Second, reliability test was performed on three common physical examinations in cerebral palsy, namely, the Thomas test, the Staheli test, and popliteal angle measurement. Thirty patients were assessed by three orthopedic surgeons to explore the statistical methods testing reliability. Third, the factors affecting the ICC values were examined by simulating the data sets based on the physical examination data where the ranges, slopes, and interobserver variability were modified. RESULTS: Of the 92 orthopedic articles identified, 58 articles (63%) did not clarify the ICC model used, and only 5 articles (5%) described all models, types, and measures. In reliability testing, although the popliteal angle showed a larger mean absolute difference than the Thomas test and the Staheli test, the ICC of popliteal angle was higher, which was believed to be contrary to the context of measurement. In addition, the ICC values were affected by the model, type, and measures used. In simulated data sets, the ICC showed higher values when the range of data sets were larger, the slopes of the data sets were parallel, and the interobserver variability was smaller. CONCLUSIONS: Care should be taken when interpreting the absolute ICC values, i.e., a higher ICC does not necessarily mean less variability because the ICC values can also be affected by various factors. The authors recommend that researchers clarify ICC models used and ICC values are interpreted in the context of measurement. The Korean Orthopaedic Association 2012-06 2012-05-17 /pmc/articles/PMC3360188/ /pubmed/22662301 http://dx.doi.org/10.4055/cios.2012.4.2.149 Text en Copyright © 2012 by The Korean Orthopaedic Association http://creativecommons.org/licenses/by-nc/3.0 This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.
spellingShingle Original Article
Lee, Kyoung Min
Lee, Jaebong
Chung, Chin Youb
Ahn, Soyeon
Sung, Ki Hyuk
Kim, Tae Won
Lee, Hui Jong
Park, Moon Seok
Pitfalls and Important Issues in Testing Reliability Using Intraclass Correlation Coefficients in Orthopaedic Research
title Pitfalls and Important Issues in Testing Reliability Using Intraclass Correlation Coefficients in Orthopaedic Research
title_full Pitfalls and Important Issues in Testing Reliability Using Intraclass Correlation Coefficients in Orthopaedic Research
title_fullStr Pitfalls and Important Issues in Testing Reliability Using Intraclass Correlation Coefficients in Orthopaedic Research
title_full_unstemmed Pitfalls and Important Issues in Testing Reliability Using Intraclass Correlation Coefficients in Orthopaedic Research
title_short Pitfalls and Important Issues in Testing Reliability Using Intraclass Correlation Coefficients in Orthopaedic Research
title_sort pitfalls and important issues in testing reliability using intraclass correlation coefficients in orthopaedic research
topic Original Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3360188/
https://www.ncbi.nlm.nih.gov/pubmed/22662301
http://dx.doi.org/10.4055/cios.2012.4.2.149
work_keys_str_mv AT leekyoungmin pitfallsandimportantissuesintestingreliabilityusingintraclasscorrelationcoefficientsinorthopaedicresearch
AT leejaebong pitfallsandimportantissuesintestingreliabilityusingintraclasscorrelationcoefficientsinorthopaedicresearch
AT chungchinyoub pitfallsandimportantissuesintestingreliabilityusingintraclasscorrelationcoefficientsinorthopaedicresearch
AT ahnsoyeon pitfallsandimportantissuesintestingreliabilityusingintraclasscorrelationcoefficientsinorthopaedicresearch
AT sungkihyuk pitfallsandimportantissuesintestingreliabilityusingintraclasscorrelationcoefficientsinorthopaedicresearch
AT kimtaewon pitfallsandimportantissuesintestingreliabilityusingintraclasscorrelationcoefficientsinorthopaedicresearch
AT leehuijong pitfallsandimportantissuesintestingreliabilityusingintraclasscorrelationcoefficientsinorthopaedicresearch
AT parkmoonseok pitfallsandimportantissuesintestingreliabilityusingintraclasscorrelationcoefficientsinorthopaedicresearch