Imagine creating a comprehensive, searchable index for a library containing hundreds of millions of books - a task that would take a single person a lifetime to complete. Now imagine discovering that many of these books contain mislabeled pages or paragraphs written by different authors than those credited.
This dual challenge - creating an efficient system to quickly locate information while ensuring its accuracy - mirrors what Lawrence Livermore National Laboratory (LLNL) researchers faced when working with the National Center for Biotechnology Information's (NCBI) Nucleotide (nt) database, a vast repository of DNA sequences from across all known species.
Nucleotide databases like NCBI nt have a broad range of applications, from diagnosing infection and tracking disease to monitoring environmental health, studying microbiomes and developing bioengineered solutions. While NCBI nt contains an incredible amount of information - trillions of nucleotides - it has grown so large that it's become difficult for scientists to implement effectively, according to LLNL Microbiology/Immunology Group Leader Nicholas Be.
Be and his team identified two major problems with existing resources. First, the version of the nt database compatible with Centrifuge - a popular tool that helps classify DNA sequences quickly and accurately - hadn't been updated since 2018. Second, they discovered the nt database contained significant errors, inconsistencies and "contaminations" - in this context, contamination refers to genetic sequences incorrectly labeled or containing material from organisms other than those they're supposed to represent. These contaminated sequences can mislead scientists into mistakenly identifying pathogens or drawing incorrect conclusions about the microbes present in their samples.
In a new study published in the journal mSystems of the ASM (American Society for Microbiology), LLNL researchers have addressed this problem by creating new, optimized indices of the nt database that simplify how scientists classify microorganisms found in various samples, from soil to human bodies, significantly improving the ability to identify and understand the myriad microorganisms that inhabit our world. The researchers leveraged advanced computing technologies to build cleaner, curated databases optimized for Centrifuge, making it easier to determine which microorganisms are present in a sample.
"By resolving contamination, filtering errors and updating content, our new nucleotide-based reference database dramatically improves metagenomic classification accuracy and reliability," said Be, principal investigator on the project. "Our database dramatically reduces such errors, resulting in robust, reliable identification of unknown DNA sequences. Its implementation will facilitate a more complete understanding of the microbial world, regardless of the specimen source."
One of the key features of this new database is its use of rigorous quality control measures. The researchers implemented a range of techniques designed to filter out contaminants and improve the accuracy of classifications - in short, they cleaned up the data, ensuring that only relevant and trustworthy sequences made it into the database.
As the team demonstrated in their paper, using the new database significantly reduced the number of misleading classifications, particularly for the genus Plasmodium, a type of parasite responsible for malaria. In studies involving mice, previous analyses had incorrectly flagged certain species of Plasmodium as significant, leading to possible misinterpretations of the data.
The scientists conducted re-analyses of existing metagenomic data to illustrate the effectiveness of their new database. They found that when they used their newly constructed Centrifuge-compatible database, there was a dramatic decrease in false-positive results, which can lead to incorrect assumptions about the presence of harmful pathogens.
The work is valuable because researchers from various fields rely on accurate microbial identification to draw valid conclusions. In medicine, determining the presence of specific bacteria or viruses can guide treatment decisions. In environmental science, understanding microbial communities is vital for assessing ecosystem health or bioremediation efforts. Similarly, in forensics, accurate identification can be crucial in criminal investigations.
"We hope this new database will raise awareness of the extensive computational resources needed to regularly update searchable databases, ensuring comprehensive organism coverage and accuracy as new sequences are screened for errors," said bioinformatics scientist and co-author Jonathan Allen.
Beyond merely providing a reference database, the researchers emphasized the importance of treating such resources as dynamic entities - expanding and improving over time, much like software that needs regular updates to remain effective. This approach mirrors best practices from software development, where developers continuously refine and validate their products to ensure they are serving their users reliably.
"Given the exponential growth in genomic data and the continuous changes in the taxonomic database, there's a clear need for regular updates to serve the scientific community," echoed researcher and first author Jose Manuel Marti, adding that the team has already received numerous requests to continue releasing this invaluable resource for the field. This high demand is understandable given the significant computational challenges involved, researchers said.
The most demanding step - the indexing process - takes the equivalent of more than five years of CPU (computer processing unit) time on a single core, though parallel processing on the Lab's large, high-memory density high-performance computing clusters reduces this to a few weeks. This extraordinary computational requirement underscores why many researchers simply don't have the resources to create such databases themselves, making LLNL's contribution particularly valuable to the scientific community.
For scientists and researchers looking to utilize the resource, the new decontaminated databases can be freely downloaded from Amazon Web Service storage following the instructions at the Langmead Lab Centrifuge indexes webpage, thus providing the scientific community with the tools to conduct accurate and reliable metagenomic analyses. But the work is just beginning.
Marti said the team is transitioning to the NCBI core_nt database, a subset, yet still challenging-to-index version of nt, supported by a sustainable framework for regular updates and public releases of new indexes. Their documented pipeline ensures consistent quality control with each update.
Beyond Centrifuge, the team is working to generalize their database construction methodology for other classification engines and apply their decontamination, filtering and validation steps to specialized databases, such as those for viral or fungal identification, Marti said. Their goal is to create a dynamic, community-driven resource that evolves with advancements in genomic sequencing and taxonomy, providing researchers with the most accurate reference data for metagenomics analysis.
With the immense growth of data, the team is also interested in developing innovative strategies, such as using distributed computing, to help manage the growing computational demands of processing and analyzing these databases. This could involve breaking down the classification problem into more manageable parts, using multiple classifiers that focus on different levels of the taxonomic tree, rather than relying on a single, comprehensive classifier.
Additional co-authors on the paper include LLNL scientists and researchers Car Reen Kok, James Thissen, Nisha Mulakken, Aram Avila-Herrera and Crystal Jaing.
