The Child Mind Institute has released a paper detailing their pioneering study in the journal Nature Human Behaviour titled, "Moving Beyond Processing and Analysis-Related Variation in Resting State Functional Brain Imaging." The research identifies significant challenges in the reproducibility and standardization of functional magnetic resonance imaging (fMRI) used to understand brain function and behavior — and proposes concrete solutions to move the field towards results that translate into real world impact.
Along with a diverse team of international collaborators, the study was led by Michael P. Milham, MD, PhD, Child Mind Institute chief science officer, and Gregory Kiar, PhD, research scientist and director of the Center for Data Analytics, Innovation, and Rigor at the Child Mind Institute. The paper critically evaluates preprocessing pipelines in fMRI data analysis, widely used software tools that have increased participation in the field of fMRI research by automating basic analytical tasks.
The study found only moderate agreement between five popular pipelines — Adolescent Brain Cognitive Development fMRI Pipeline (ABCD-BIDS); Connectome Computational System (CCS); Configurable Pipeline for the Analysis of Connectomes default pipeline (C-PAC:Default), developed by the Child Mind Institute; Data Processing Assistant for Resting-State fMRI (DPARSF); and fMRIPrep Long-Term Support version (fMRIPrep-LTS) — when given identical data. This variability significantly affects the reliability of findings in brain-wide association studies, which are essential for understanding individual differences in brain function and behavior. This is particularly concerning since researchers often assume these pipelines are interchangeable.
"Critically, we found that what we learn about the brain can change significantly based on which pipeline we choose," says Dr. Kiar. "The most poignant finding was that low agreement between pipelines also compromises the conclusions we make with this preprocessed data. If we want our work to translate to understanding and care in the real world, we must recognize this and take action."
Key Findings:
- Moderate agreement across pipelines: The study demonstrated that different preprocessing pipelines produce varying outcomes, even when the same data is used. This is a significant challenge in achieving reproducibility in fMRI studies.
- Impact on scientific insights: Differences in preprocessing steps can lead to inconsistent results in identifying brain-behavior associations. This inconsistency is particularly concerning, as the neuroscience community aims to translate these findings into clinical applications.
- Data quality and pipeline differences: Analytical differences are more apparent with high-quality data. Pipelines will be the next hurdle towards generating findings, which can be reproduced across studies as data collection is optimized.
- Recommendations for standardization: The field must adopt standardized methods and transparent reporting practices. This involves detailing all data processing steps, including software versions and specific parameters used.
"This study underscores the importance of reliability and standardization in data processing," said Dr. Milham. "Reproducibility is critical to scientific progress, and it has been fiendishly difficult to achieve in brain imaging. This study represents a huge amount of time and effort to reveal common threats to reproducibility, and reproduce the common tools used in the field. This will lower the barrier for everyone and push us to the next step."
Dr. Milham, Dr. Kiar, and their team propose that improving inter-pipeline agreement (IPA) through standardization and clear methodological reporting is crucial for the field's advancement. They advocate for the documentation of all pipeline-related decisions, settings, and versions in every publication, as well as using alternative pipelines in analyses to determine if inter-pipeline variability is affecting research results. The Child Mind Institute's C-PAC configurable pipeline allows multiple pipelines to be compared together in one package and can be used for this purpose.
Creating a robust consensus and taking proactive action now on this "next frontier" in the reproducibility crisis for the fMRI field will make researchers better prepared to harness the power of improved data quality — and usher in an era of robust and continuous insights about the brain and human behavior.
ABOUT THE CHILD MIND INSTITUTE
We are the leading independent nonprofit in children's mental health providing gold-standard, evidence-based care, delivering educational resources to millions of families each year, training educators in underserved communities, and developing open science initiatives and tomorrow's breakthrough treatments.
