Cargando…
Inter-rater reliability of functional MRI data quality control assessments: A standardised protocol and practical guide using pyfMRIqc
Quality control is a critical step in the processing and analysis of functional magnetic resonance imaging data. Its purpose is to remove problematic data that could otherwise lead to downstream errors in the analysis and reporting of results. The manual inspection of data can be a laborious and err...
Autores principales: | , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Frontiers Media S.A.
2023
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9936142/ https://www.ncbi.nlm.nih.gov/pubmed/36816136 http://dx.doi.org/10.3389/fnins.2023.1070413 |
_version_ | 1784890174456463360 |
---|---|
author | Williams, Brendan Hedger, Nicholas McNabb, Carolyn B. Rossetti, Gabriella M. K. Christakou, Anastasia |
author_facet | Williams, Brendan Hedger, Nicholas McNabb, Carolyn B. Rossetti, Gabriella M. K. Christakou, Anastasia |
author_sort | Williams, Brendan |
collection | PubMed |
description | Quality control is a critical step in the processing and analysis of functional magnetic resonance imaging data. Its purpose is to remove problematic data that could otherwise lead to downstream errors in the analysis and reporting of results. The manual inspection of data can be a laborious and error-prone process that is susceptible to human error. The development of automated tools aims to mitigate these issues. One such tool is pyfMRIqc, which we previously developed as a user-friendly method for assessing data quality. Yet, these methods still generate output that requires subjective interpretations about whether the quality of a given dataset meets an acceptable standard for further analysis. Here we present a quality control protocol using pyfMRIqc and assess the inter-rater reliability of four independent raters using this protocol for data from the fMRI Open QC project (https://osf.io/qaesm/). Data were classified by raters as either “include,” “uncertain,” or “exclude.” There was moderate to substantial agreement between raters for “include” and “exclude,” but little to no agreement for “uncertain.” In most cases only a single rater used the “uncertain” classification for a given participant’s data, with the remaining raters showing agreement for “include”/“exclude” decisions in all but one case. We suggest several approaches to increase rater agreement and reduce disagreement for “uncertain” cases, aiding classification consistency. |
format | Online Article Text |
id | pubmed-9936142 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2023 |
publisher | Frontiers Media S.A. |
record_format | MEDLINE/PubMed |
spelling | pubmed-99361422023-02-18 Inter-rater reliability of functional MRI data quality control assessments: A standardised protocol and practical guide using pyfMRIqc Williams, Brendan Hedger, Nicholas McNabb, Carolyn B. Rossetti, Gabriella M. K. Christakou, Anastasia Front Neurosci Neuroscience Quality control is a critical step in the processing and analysis of functional magnetic resonance imaging data. Its purpose is to remove problematic data that could otherwise lead to downstream errors in the analysis and reporting of results. The manual inspection of data can be a laborious and error-prone process that is susceptible to human error. The development of automated tools aims to mitigate these issues. One such tool is pyfMRIqc, which we previously developed as a user-friendly method for assessing data quality. Yet, these methods still generate output that requires subjective interpretations about whether the quality of a given dataset meets an acceptable standard for further analysis. Here we present a quality control protocol using pyfMRIqc and assess the inter-rater reliability of four independent raters using this protocol for data from the fMRI Open QC project (https://osf.io/qaesm/). Data were classified by raters as either “include,” “uncertain,” or “exclude.” There was moderate to substantial agreement between raters for “include” and “exclude,” but little to no agreement for “uncertain.” In most cases only a single rater used the “uncertain” classification for a given participant’s data, with the remaining raters showing agreement for “include”/“exclude” decisions in all but one case. We suggest several approaches to increase rater agreement and reduce disagreement for “uncertain” cases, aiding classification consistency. Frontiers Media S.A. 2023-02-03 /pmc/articles/PMC9936142/ /pubmed/36816136 http://dx.doi.org/10.3389/fnins.2023.1070413 Text en Copyright © 2023 Williams, Hedger, McNabb, Rossetti and Christakou. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms. |
spellingShingle | Neuroscience Williams, Brendan Hedger, Nicholas McNabb, Carolyn B. Rossetti, Gabriella M. K. Christakou, Anastasia Inter-rater reliability of functional MRI data quality control assessments: A standardised protocol and practical guide using pyfMRIqc |
title | Inter-rater reliability of functional MRI data quality control assessments: A standardised protocol and practical guide using pyfMRIqc |
title_full | Inter-rater reliability of functional MRI data quality control assessments: A standardised protocol and practical guide using pyfMRIqc |
title_fullStr | Inter-rater reliability of functional MRI data quality control assessments: A standardised protocol and practical guide using pyfMRIqc |
title_full_unstemmed | Inter-rater reliability of functional MRI data quality control assessments: A standardised protocol and practical guide using pyfMRIqc |
title_short | Inter-rater reliability of functional MRI data quality control assessments: A standardised protocol and practical guide using pyfMRIqc |
title_sort | inter-rater reliability of functional mri data quality control assessments: a standardised protocol and practical guide using pyfmriqc |
topic | Neuroscience |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9936142/ https://www.ncbi.nlm.nih.gov/pubmed/36816136 http://dx.doi.org/10.3389/fnins.2023.1070413 |
work_keys_str_mv | AT williamsbrendan interraterreliabilityoffunctionalmridataqualitycontrolassessmentsastandardisedprotocolandpracticalguideusingpyfmriqc AT hedgernicholas interraterreliabilityoffunctionalmridataqualitycontrolassessmentsastandardisedprotocolandpracticalguideusingpyfmriqc AT mcnabbcarolynb interraterreliabilityoffunctionalmridataqualitycontrolassessmentsastandardisedprotocolandpracticalguideusingpyfmriqc AT rossettigabriellamk interraterreliabilityoffunctionalmridataqualitycontrolassessmentsastandardisedprotocolandpracticalguideusingpyfmriqc AT christakouanastasia interraterreliabilityoffunctionalmridataqualitycontrolassessmentsastandardisedprotocolandpracticalguideusingpyfmriqc |