Cargando…

Inter-rater reliability of functional MRI data quality control assessments: A standardised protocol and practical guide using pyfMRIqc

Quality control is a critical step in the processing and analysis of functional magnetic resonance imaging data. Its purpose is to remove problematic data that could otherwise lead to downstream errors in the analysis and reporting of results. The manual inspection of data can be a laborious and err...

Descripción completa

Detalles Bibliográficos
Autores principales: Williams, Brendan, Hedger, Nicholas, McNabb, Carolyn B., Rossetti, Gabriella M. K., Christakou, Anastasia
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Frontiers Media S.A. 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9936142/
https://www.ncbi.nlm.nih.gov/pubmed/36816136
http://dx.doi.org/10.3389/fnins.2023.1070413
_version_ 1784890174456463360
author Williams, Brendan
Hedger, Nicholas
McNabb, Carolyn B.
Rossetti, Gabriella M. K.
Christakou, Anastasia
author_facet Williams, Brendan
Hedger, Nicholas
McNabb, Carolyn B.
Rossetti, Gabriella M. K.
Christakou, Anastasia
author_sort Williams, Brendan
collection PubMed
description Quality control is a critical step in the processing and analysis of functional magnetic resonance imaging data. Its purpose is to remove problematic data that could otherwise lead to downstream errors in the analysis and reporting of results. The manual inspection of data can be a laborious and error-prone process that is susceptible to human error. The development of automated tools aims to mitigate these issues. One such tool is pyfMRIqc, which we previously developed as a user-friendly method for assessing data quality. Yet, these methods still generate output that requires subjective interpretations about whether the quality of a given dataset meets an acceptable standard for further analysis. Here we present a quality control protocol using pyfMRIqc and assess the inter-rater reliability of four independent raters using this protocol for data from the fMRI Open QC project (https://osf.io/qaesm/). Data were classified by raters as either “include,” “uncertain,” or “exclude.” There was moderate to substantial agreement between raters for “include” and “exclude,” but little to no agreement for “uncertain.” In most cases only a single rater used the “uncertain” classification for a given participant’s data, with the remaining raters showing agreement for “include”/“exclude” decisions in all but one case. We suggest several approaches to increase rater agreement and reduce disagreement for “uncertain” cases, aiding classification consistency.
format Online
Article
Text
id pubmed-9936142
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher Frontiers Media S.A.
record_format MEDLINE/PubMed
spelling pubmed-99361422023-02-18 Inter-rater reliability of functional MRI data quality control assessments: A standardised protocol and practical guide using pyfMRIqc Williams, Brendan Hedger, Nicholas McNabb, Carolyn B. Rossetti, Gabriella M. K. Christakou, Anastasia Front Neurosci Neuroscience Quality control is a critical step in the processing and analysis of functional magnetic resonance imaging data. Its purpose is to remove problematic data that could otherwise lead to downstream errors in the analysis and reporting of results. The manual inspection of data can be a laborious and error-prone process that is susceptible to human error. The development of automated tools aims to mitigate these issues. One such tool is pyfMRIqc, which we previously developed as a user-friendly method for assessing data quality. Yet, these methods still generate output that requires subjective interpretations about whether the quality of a given dataset meets an acceptable standard for further analysis. Here we present a quality control protocol using pyfMRIqc and assess the inter-rater reliability of four independent raters using this protocol for data from the fMRI Open QC project (https://osf.io/qaesm/). Data were classified by raters as either “include,” “uncertain,” or “exclude.” There was moderate to substantial agreement between raters for “include” and “exclude,” but little to no agreement for “uncertain.” In most cases only a single rater used the “uncertain” classification for a given participant’s data, with the remaining raters showing agreement for “include”/“exclude” decisions in all but one case. We suggest several approaches to increase rater agreement and reduce disagreement for “uncertain” cases, aiding classification consistency. Frontiers Media S.A. 2023-02-03 /pmc/articles/PMC9936142/ /pubmed/36816136 http://dx.doi.org/10.3389/fnins.2023.1070413 Text en Copyright © 2023 Williams, Hedger, McNabb, Rossetti and Christakou. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
spellingShingle Neuroscience
Williams, Brendan
Hedger, Nicholas
McNabb, Carolyn B.
Rossetti, Gabriella M. K.
Christakou, Anastasia
Inter-rater reliability of functional MRI data quality control assessments: A standardised protocol and practical guide using pyfMRIqc
title Inter-rater reliability of functional MRI data quality control assessments: A standardised protocol and practical guide using pyfMRIqc
title_full Inter-rater reliability of functional MRI data quality control assessments: A standardised protocol and practical guide using pyfMRIqc
title_fullStr Inter-rater reliability of functional MRI data quality control assessments: A standardised protocol and practical guide using pyfMRIqc
title_full_unstemmed Inter-rater reliability of functional MRI data quality control assessments: A standardised protocol and practical guide using pyfMRIqc
title_short Inter-rater reliability of functional MRI data quality control assessments: A standardised protocol and practical guide using pyfMRIqc
title_sort inter-rater reliability of functional mri data quality control assessments: a standardised protocol and practical guide using pyfmriqc
topic Neuroscience
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9936142/
https://www.ncbi.nlm.nih.gov/pubmed/36816136
http://dx.doi.org/10.3389/fnins.2023.1070413
work_keys_str_mv AT williamsbrendan interraterreliabilityoffunctionalmridataqualitycontrolassessmentsastandardisedprotocolandpracticalguideusingpyfmriqc
AT hedgernicholas interraterreliabilityoffunctionalmridataqualitycontrolassessmentsastandardisedprotocolandpracticalguideusingpyfmriqc
AT mcnabbcarolynb interraterreliabilityoffunctionalmridataqualitycontrolassessmentsastandardisedprotocolandpracticalguideusingpyfmriqc
AT rossettigabriellamk interraterreliabilityoffunctionalmridataqualitycontrolassessmentsastandardisedprotocolandpracticalguideusingpyfmriqc
AT christakouanastasia interraterreliabilityoffunctionalmridataqualitycontrolassessmentsastandardisedprotocolandpracticalguideusingpyfmriqc