Tag: Databases

Today, I will be giving a talk at the EDAR8 conference in Brisbane, which will partially cover something that I has been cooking in the research group for some time. It started off with a recurring need to see if a particular antibiotic resistance gene (ARG) was truly new or if it had been detected in previous studies. And after having done the same procedures in several studies, we started to think that maybe it would make sense to build a resource that could be used to keep track of both established and latent ARGs (1) and that could be easily updated over time. And the result is CLEVER – a scheme for Classification of Latent and Established Variants of Extant Antibiotic Resistance Genes.

CLEVER is two things. First, it is a set of definitions for ARGs, making it easier to assign them to categories, for example for risk assessment. Second, it is a database based on these criteria, and to the best of our knowledge the first resource to organize both established and latent ARGs into the same coherent database structure, while still keeping them categorized. To achieve this, CLEVER integrates data from ResFinder (2), CARD (3), ResFinderFG (4), as well as ARGs predicted by the fARGene (5) and MUSTARD (6) tools, gathered from published literature (6-13),

Very briefly, CLEVER is built on the following definitions:

• Established ARG (E): An ARG that is experimentally verified to confer antibiotic resistance and is present in human pathogens.
• Latent ARG (L): An ARG that confers a resistance function (or is predicted to do so), but does not exist in pathogens
• Mobile ARG (M): An ARG which is present on a mobile genetic element, which could be plasmids, integrons, transposons or integrative conjugative elements.
• Chromosomal ARG (C): Any ARG that does not meet the criteria for a mobile ARG above.
• Validated ARG (V): An ARG for which the resistance function has been verified in laboratory experiments, by showing that the presence of the gene increases the MIC of the host compared to an otherwise isogenic strain that does not carry the gene, or alternatively that over-expression of the gene induces a higher MIC compared to an isogenic reference strain.
• Predicted ARG (P): An ARG for which its function has not been verified experimentally (see above), but has been predicted to be an ARG by fARGene
• Structure-predicted ARG (S): An ARG for which its function has not been verified experimentally (see above), but has been predicted to be an ARG based on 3D-structure similarity by MUSTARD

Importantly, what CLEVER also does is to give names (CLEVER IDs) to latent ARGs that currently does not have any consistent way to refer to them. This makes it possible to unambiguously reference a particular ARG family across studies, which is crucial for risk assessment and understanding the spread of AMR.

Finally, by classifying ARGs as established or latent, mobile or chromosomal and verified or predicted makes it possible to identity particular latent ARGs that until now has gone under the radar. Particularly, latent ARGs that already appear on mobile genetic elements and have several mobile variants, are likely to be imminent AMR threats to human health, making them “most-wanted” ARGs that should be targeted for phenotypic evaluation and potential inclusion into AMR surveillance efforts.

By specifically looking into potential ARGs against the last-resort antibiotics carbapenems, colistin, tigecycline and plazomicin, and pulling out latent ARGs that belong to classes potentially conferring resistance to these antibiotics that are already present on plasmids, we can already present a list of five ARGs of imminent concern, which we label the first CLEVER list of most-wanted ARGs: ~blaA-145, ~blaD2-129, ~aac6p-147, ~aph2b-14, and ~aph6-89.

Naturally, there is still a lot work to be done on CLEVER, but I am very proud of what we have already, and I think that the current version (2.0) already has potential to be very useful for AMR studies. We have already started using this internally in the lab and within SEARCHER quite extensively in the last few months, so I am happy to see the use of this resource in the larger AMR community!

References:

1. Inda-Díaz JS, Lund D, Parras-Moltó M, Johnning A, Bengtsson-Palme J, Kristiansson E. Latent antibiotic resistance genes are abundant, diverse, and mobile in human, animal, and environmental microbiomes. Microbiome 2023;11:44. https://doi.org/10.1186/s40168-023-01479-0.
2. Bortolaia V, Kaas RS, Ruppe E, Roberts MC, Schwarz S, Cattoir V, et al. ResFinder 4.0 for predictions of phenotypes from genotypes. Journal of Antimicrobial Chemotherapy 2020;75:3491–500. https://doi.org/10.1093/jac/dkaa345.
3. Jia B, Raphenya AR, Alcock B, Waglechner N, Guo P, Tsang KK, et al. CARD 2017: expansion and model-centric curation of the comprehensive antibiotic resistance database. Nucleic Acids Research 2016:gkw1004. https://doi.org/10.1093/nar/gkw1004.
4. Gschwind R, Ugarcina Perovic S, Weiss M, Petitjean M, Lao J, Coelho LP, et al. ResFinderFG v2.0: a database of antibiotic resistance genes obtained by functional metagenomics. Nucleic Acids Research 2023:gkad384. https://doi.org/10.1093/nar/gkad384.
5. Berglund F, Österlund T, Boulund F, Marathe NP, Larsson DGJ, Kristiansson E. Identification and reconstruction of novel antibiotic resistance genes from metagenomes. Microbiome 2019;7, 52. https://doi.org/10.1186/s40168-019-0670-1
6. Ruppé E, Ghozlane A, Tap J, Pons N, Alvarez A-S, Maziers N, et al. Prediction of the intestinal resistome by a three-dimensional structure-based method. Nature Microbiology 2019;4:112–23. https://doi.org/10.1038/s41564-018-0292-6.
7. Inda-Díaz JS, Lund D, Parras-Moltó M, Johnning A, Bengtsson-Palme J, Kristiansson E. Latent antibiotic resistance genes are abundant, diverse, and mobile in human, animal, and environmental microbiomes. Microbiome 2023;11:44. https://doi.org/10.1186/s40168-023-01479-0.
8. Victor MP, Radisic V, Grevskott DH, Marathe NP. Hospital effluent in a low-resistance setting is responsible for dissemination of novel antibiotic resistance genes into the marine environment. Ecotoxicology and Environmental Safety 2025;301:118390. https://doi.org/10.1016/j.ecoenv.2025.118390.
9. Victor MP, Øvreås L, Marathe NP. Characterization of known and novel clinically important antibiotic resistance genes and novel microbes from wastewater-impacted high Arctic fjord sediments. Science of The Total Environment 2025;985:179699. https://doi.org/10.1016/j.scitotenv.2025.179699.
10. Li B, Jiang L, Johnson T, Wang G, Sun W, Wei G, et al. Global health risks lurking in livestock resistome. Sci Adv 2025;11:eadt8073. https://doi.org/10.1126/sciadv.adt8073.
11. Somerville V, Meola M, Nunes-Richards A, Bengtsson-Palme J, Neukamm J, Majander K, et al. Microbial community dynamics in a traditional Swiss mountain cheese over 142 years of cheesemaking 2026. https://doi.org/10.64898/2026.02.26.708305.
12. Coche‐Miranda J, Arros P, Canales N, Berríos‐Pastén C, Azziz G, Lagos R, et al. Antarctic soil microbiomes encode structurally conserved and phylogenetically diverse beta‐lactamases. iMetaOmics 2026;e70118. https://doi.org/10.1002/imo2.70118.
13. Wang K, Xu J, Li X, Zhu P, Suo R, Lu X, et al. Evolutionary selection of trimethoprim-resistant dfrA genes in lytic phages affects phage and host fitness during infection. Sci Adv 2025;11:eadt4817. https://doi.org/10.1126/sciadv.adt4817.

Together with Joakim Larsson‘s lab, we now have an open two-year postdoc position in bioinformatics on antibiotic resistance and biocide resistance. The development of antibiotic resistance has been driven by use of antibiotics, but antibacterial biocides also have the potential to select for antibiotic resistance. However, knowledge of which genes that contribute to biocide resistance and could be associated with antibiotic resistance is sparse. To some extent, such genes are documented in the BacMet database which we have developed, but this collection of resistance genes is only scratching the surface of all biocide resistance that exists among bacteria in the environment.

We are now looking for a postdoctoral fellow to continue the important work on bioinformatic analysis of biocide and antibiotic resistance to answer the question whether increasing biocide resistance would be a threat to human health. The postdoc will be working with the development of the BacMet database to make it more targeted towards biocidal substances and products in addition to resistance genes. The tasks include bioinformatic sequence analysis, literature studies and database and web programming. The work will also include investigations of the prevalence of the identified resistance genes in genomes and metagenomes.

The recruited person will work closely with both my group and the group of Prof. Joakim Larsson, and will participate in the JPIAMR-funded BIOCIDE project. You can apply to the postdoc position at the University of Gothenburg application portal: https://web103.reachmee.com/ext/I005/1035/job?site=7&lang=UK&validator=9b89bead79bb7258ad55c8d75228e5b7&job_id=25122

The deadline is May 4, 2022. Come work with us on this exciting topic in the intersect between two great research environments (if I may say it myself!) We look forward to your application!

I am happy to share the news that the paper describing out software tool Mumame is now out in its final form! (1) The paper got published today in the journal Metabarcoding and Metagenomics after being available as a preprint (2) since last autumn. This version has not changed a whole lot since the preprint, but it is more polished and better argued (thanks to a great review process). The software is virtually the same, but is not also available via Conda.

In the paper, we describe the Mumame software, which can be used to distinguish between wildtype and mutated sequences in shotgun metagenomic sequencing data and quantify their relative abundances. We further demonstrate the utility of the tool by quantifying antibiotic resistance mutations in several publicly available metagenomic data sets (3-6), and find that the tool is useful but that sequencing depth is a key factor to detect rare mutations. Therefore, much larger numbers of sequences may be required for reliable detection of mutations than is needed for most other applications of shotgun metagenomics. Since the preprint was published, Mumame has also found use in our recently published paper on selection for antibiotic resistance in a Croatian macrolide production wastewater treatment plant, unfortunately with inconclusive results (7). Mumame is freely available here.

I again want to stress the fantastic work that Shruthi Magesh did last year as a summer student at WID in the evaluation of this tool. As I have pointed out earlier, I did write the code for the software (with a lot of input from Viktor Jonsson), but Shruthi did the software testing and evaluations. Thanks and congratulations Shruthi, and good luck in pursuing your PhD program!

References

1. Magesh S, Jonsson V, Bengtsson-Palme J: Mumame: A software tool for quantifying gene-specific point-mutations in shotgun metagenomic data. Metabarcoding and Metagenomics, 3: 59–67 (2019). doi: 10.3897/mbmg.3.36236
2. Magesh S, Jonsson V, Bengtsson-Palme J: Quantifying point-mutations in metagenomic data. bioRxiv, 438572 (2018). doi: 10.1101/438572
3. Bengtsson-Palme J, Boulund F, Fick J, Kristiansson E, Larsson DGJ: Shotgun metagenomics reveals a wide array of antibiotic resistance genes and mobile elements in a polluted lake in India. Frontiers in Microbiology, 5, 648 (2014). doi: 10.3389/fmicb.2014.00648
4. Lundström S, Östman M, Bengtsson-Palme J, Rutgersson C, Thoudal M, Sircar T, Blanck H, Eriksson KM, Tysklind M, Flach C-F, Larsson DGJ: Minimal selective concentrations of tetracycline in complex aquatic bacterial biofilms. Science of the Total Environment, 553, 587–595 (2016). doi: 10.1016/j.scitotenv.2016.02.103
5. Pal C, Bengtsson-Palme J, Kristiansson E, Larsson DGJ: The structure and diversity of human, animal and environmental resistomes. Microbiome, 4, 54 (2016). doi: 10.1186/s40168-016-0199-5
6. Kraupner N, Ebmeyer S, Bengtsson-Palme J, Fick J, Kristiansson E, Flach C-F, Larsson DGJ: Selective concentration for ciprofloxacin in Escherichia coli grown in complex aquatic bacterial biofilms. Environment International, 116, 255–268 (2018). doi: 10.1016/j.envint.2018.04.029
7. Bengtsson-Palme J, Milakovic M, Švecová H, Ganjto M, Jonsson V, Grabic R, Udiković Kolić N: Pharmaceutical wastewater treatment plant enriches resistance genes and alter the structure of microbial communities. Water Research, 162, 437-445 (2019). doi: 10.1016/j.watres.2019.06.073

Since F1000Research uses a somewhat different publication scheme than most journals, I still haven’t understood if this paper is formally published after peer review, but I start to assume it is. There have been very little changes since the last version, so hence I will be lazy and basically repost what I wrote in April when the first version (the “preprint”) was posted online. The paper (1) is the result of a workshop arranged by the JRC in Italy in 2017. It describes various challenges arising from the process of designing a benchmark strategy for bioinformatics pipelines in the identification of antimicrobial resistance genes in next generation sequencing data.

The paper discusses issues about the benchmarking datasets used, testing samples, evaluation criteria for the performance of different tools, and how the benchmarking dataset should be created and distributed. Specially, we address the following questions:

• How should a benchmark strategy handle the current and expanding universe of NGS platforms?
• What should be the quality profile (in terms of read length, error rate, etc.) of in silico reference materials?
• Should different sets of reference materials be produced for each platform? In that case, how to ensure no bias is introduced in the process?
• Should in silico reference material be composed of the output of real experiments, or simulated read sets? If a combination is used, what is the optimal ratio?
• How is it possible to ensure that the simulated output has been simulated “correctly”?
• For real experiment datasets, how to avoid the presence of sensitive information?
• Regarding the quality metrics in the benchmark datasets (e.g. error rate, read quality), should these values be fixed for all datasets, or fall within specific ranges? How wide can/should these ranges be?
• How should the benchmark manage the different mechanisms by which bacteria acquire resistance?
• What is the set of resistance genes/mechanisms that need to be included in the benchmark? How should this set be agreed upon?
• Should datasets representing different sample types (e.g. isolated clones, environmental samples) be included in the same benchmark?
• Is a correct representation of different bacterial species (host genomes) important?
• How can the “true” value of the samples, against which the pipelines will be evaluated, be guaranteed?
• What is needed to demonstrate that the original sample has been correctly characterised, in case real experiments are used?
• How should the target performance thresholds (e.g. specificity, sensitivity, accuracy) for the benchmark suite be set?
• What is the impact of these performance thresholds on the required size of the sample set?
• How can the benchmark stay relevant when new resistance mechanisms are regularly characterized?
• How is the continued quality of the benchmark dataset ensured?
• Who should generate the benchmark resource?
• How can the benchmark resource be efficiently shared?

Of course, we have not answered all these questions, but I think we have come down to a decent description of the problems, which we see as an important foundation for solving these issues and implementing the benchmarking standard. Some of these issues were tackled in our review paper from last year on using metagenomics to study resistance genes in microbial communities (2). The paper also somewhat connects to the database curation paper we published in 2016 (3), although this time the strategies deal with the testing datasets rather than the actual databases. The paper is the first outcome of the workshop arranged by the JRC on “Next-generation sequencing technologies and antimicrobial resistance” held October 4-5 2017 in Ispra, Italy. You can find the paper here (it’s open access).

On another note, the new paper describing the UNITE database (4) has now got a formal issue assigned to it, as has the paper on tandem repeat barcoding in fungi published in Molecular Ecology Resources last year (5).

References and notes

1. Angers-Loustau A, Petrillo M, Bengtsson-Palme J, Berendonk T, Blais B, Chan KG, Coque TM, Hammer P, Heß S, Kagkli DM, Krumbiegel C, Lanza VF, Madec J-Y, Naas T, O’Grady J, Paracchini V, Rossen JWA, Ruppé E, Vamathevan J, Venturi V, Van den Eede G: The challenges of designing a benchmark strategy for bioinformatics pipelines in the identification of antimicrobial resistance determinants using next generation sequencing technologies. F1000Research, 7, 459 (2018). doi: 10.12688/f1000research.14509.1
2. Bengtsson-Palme J, Larsson DGJ, Kristiansson E: Using metagenomics to investigate human and environmental resistomes. Journal of Antimicrobial Chemotherapy, 72, 2690–2703 (2017). doi: 10.1093/jac/dkx199
3. Bengtsson-Palme J, Boulund F, Edström R, Feizi A, Johnning A, Jonsson VA, Karlsson FH, Pal C, Pereira MB, Rehammar A, Sánchez J, Sanli K, Thorell K: Strategies to improve usability and preserve accuracy in biological sequence databases. Proteomics, 16, 18, 2454–2460 (2016). doi: 10.1002/pmic.201600034
4. Nilsson RH, Larsson K-H, Taylor AFS, Bengtsson-Palme J, Jeppesen TS, Schigel D, Kennedy P, Picard K, Glöckner FO, Tedersoo L, Saar I, Kõljalg U, Abarenkov K: The UNITE database for molecular identification of fungi: handling dark taxa and parallel taxonomic classifications. Nucleic Acids Research, 47, D1, D259–D264 (2019). doi: 10.1093/nar/gky1022
5. Wurzbacher C, Larsson E, Bengtsson-Palme J, Van den Wyngaert S, Svantesson S, Kristiansson E, Kagami M, Nilsson RH: Introducing ribosomal tandem repeat barcoding for fungi. Molecular Ecology Resources, 19, 1, 118–127 (2019). doi: 10.1111/1755-0998.12944

A few days ago, my attention was turned to a duplicate in the COI database bundled with Metaxa2 2.2. While this duplicate sequence should not cause any troubles for Metaxa2 itself, it has created issues for people using the database itself together with, e.g., QIIME. Therefore, I have today issued a very very minor update to the Metaxa2 2.2 package as well as the entry in the Metaxa2 Database Repository, removing the duplicate sequence. I deemed that this was not significant enough to issue a new version, particularly as no code was changed and it did not cause issues for the software itself, so the version will stay at 2.2 for the time being. Happy barcoding!

In the 2019 database issue, Nucleic Acids Research will include a new paper on the UNITE database for molecular identification of fungi (1). I have been involved in the development of UNITE in different ways since 2012, most prominently via the ITSx (2) and Atosh software which are ticking under the hood of the database.

In this update paper, we introduce a redesigned handling of unclassifiable species hypotheses, integration with the taxonomic backbone of the Global Biodiversity Information Facility, and support for an unlimited number of parallel taxonomic classification systems. The database now contains around one million fungal ITS sequences that can be used for reference, which are clustered into roughly 459,000 species hypotheses (3). Each species hypothesis is assigned a digital object identifier (DOI), which enables unambiguous reference across studies. The paper is available as open access and the UNITE database is available open source from here.

References

1. Nilsson RH, Larsson K-H, Taylor AFS, Bengtsson-Palme J, Jeppesen TS, Schigel D, Kennedy P, Picard K, Glöckner FO, Tedersoo L, Saar I, Kõljalg U, Abarenkov K: The UNITE database for molecular identification of fungi: handling dark taxa and parallel taxonomic classifications. Nucleic Acids Research, Advance article, gky1022 (2018). doi: 10.1093/nar/gky1022
2. Bengtsson-Palme J, Ryberg M, Hartmann M, Branco S, Wang Z, Godhe A, De Wit P, Sánchez-García M, Ebersberger I, de Souza F, Amend AS, Jumpponen A, Unterseher M, Kristiansson E, Abarenkov K, Bertrand YJK, Sanli K, Eriksson KM, Vik U, Veldre V, Nilsson RH: Improved software detection and extraction of ITS1 and ITS2 from ribosomal ITS sequences of fungi and other eukaryotes for use in environmental sequencing. Methods in Ecology and Evolution, 4, 10, 914–919 (2013). doi: 10.1111/2041-210X.12073
3. Kõljalg U, Nilsson RH, Abarenkov K, Tedersoo L, Taylor AFS, Bahram M, Bates ST, Bruns TT, Bengtsson-Palme J, Callaghan TM, Douglas B, Drenkhan T, Eberhardt U, Dueñas M, Grebenc T, Griffith GW, Hartmann M, Kirk PM, Kohout P, Larsson E, Lindahl BD, Lücking R, Martín MP, Matheny PB, Nguyen NH, Niskanen T, Oja J, Peay KG, Peintner U, Peterson M, Põldmaa K, Saag L, Saar I, Schüßler A, Senés C, Smith ME, Suija A, Taylor DE, Telleria MT, Weiß M, Larsson KH: Towards a unified paradigm for sequence-based identification of Fungi. Molecular Ecology, 22, 21, 5271–5277 (2013). doi: 10.1111/mec.12481

Let me get straight to something somewhat besides the point here: summer students can achieve amazing things! One such student I had the pleasure to work with this summer is Shruthi Magesh, and a preprint based on work she did with me at the Wisconsin Institute for Discovery this summer just got published on bioRxiv (1). The preprint describes a software tool called Mumame, which uses database information on mutations in DNA or protein sequences to search metagenomic datasets and quantifies the relative proportion of resistance mutations over wild type sequences.

In the preprint (1), we first of all show that Mumame works on amplicon data where we already knew the true outcome (2). Second, we show that we can detect differences in mutation frequencies in controlled experiments (2,3). Lastly, we use the tool to gain some further information about resistance patterns in sediments from polluted environments in India (4,5). Together these analyses show that one of the most central aspects for Mumame to be able to find mutations is having a very high number of sequenced reads in all libraries (preferably more than 50 million per library), because these mutations are generally rare – even in polluted environments and microcosms exposed to antibiotics. We expect Mumame to be a useful addition to metagenomic studies of e.g. antibiotic resistance, and to increase the detail by which metagenomes can be screened for phenotypically important differences.

While I did write the code for the software (with a lot of input from Viktor Jonsson, who also is a coauthor on the preprint, on the statistical analysis), Shruthi did the software testing and evaluations, and the paper would not have been possible hadn’t she wanted a bioinformatic summer project related to metagenomics, aside from her laboratory work. The resulting preprint is available from bioRxiv and the Mumame software is freely available from this site.

References

1. Magesh S, Jonsson V, Bengtsson-Palme J: Quantifying point-mutations in metagenomic data. bioRxiv, 438572 (2018). doi: 10.1101/438572 [Link]
2. Kraupner N, Ebmeyer S, Bengtsson-Palme J, Fick J, Kristiansson E, Flach C-F, Larsson DGJ: Selective concentration for ciprofloxacin in Escherichia coli grown in complex aquatic bacterial biofilms. Environment International, 116, 255–268 (2018). doi: 10.1016/j.envint.2018.04.029 [Paper link]
3. Lundström S, Östman M, Bengtsson-Palme J, Rutgersson C, Thoudal M, Sircar T, Blanck H, Eriksson KM, Tysklind M, Flach C-F, Larsson DGJ: Minimal selective concentrations of tetracycline in complex aquatic bacterial biofilms. Science of the Total Environment, 553, 587–595 (2016). doi: 10.1016/j.scitotenv.2016.02.103 [Paper link]
4. Bengtsson-Palme J, Boulund F, Fick J, Kristiansson E, Larsson DGJ: Shotgun metagenomics reveals a wide array of antibiotic resistance genes and mobile elements in a polluted lake in India. Frontiers in Microbiology, 5, 648 (2014). doi: 10.3389/fmicb.2014.00648 [Paper link]
5. Kristiansson E, Fick J, Janzon A, Grabic R, Rutgersson C, Weijdegård B, Söderström H, Larsson DGJ: Pyrosequencing of antibiotic-contaminated river sediments reveals high levels of resistance and gene transfer elements. PLoS ONE, Volume 6, e17038 (2011). doi:10.1371/journal.pone.0017038.

Last week, I uploaded a new database to the Metaxa2 Database Repository, called DAIRYdb. DAIRYdb (1) is a manually curated reference database for 16S rRNA amplicon sequences from dairy products. Significant efforts have been put into improving annotation algorithms, such as Metaxa2 (2), while less attention has been put into curation of reliable and consistent databases (3). Previous studies have shown that databases restricted to the studied environment improve unambiguous taxonomy annotation to the species level, thanks to consistent taxonomy, lack of blanks and reduced competition between different reference taxonomies (4-5). The usage of DAIRYdb in combination with different classification tools allows taxonomy annotation accuracy of over 90% at species level for microbiome samples from dairy products, where species identification is mandatory due to the affiliation to few closely related genera of most dominant lactic acid bacteria.

The database can be added to your Metaxa2 (version 2.2 or later) installation by using the following command:

metaxa2_install_database -g SSU_DAIRYdb_v1.1.2

Further adaptations of the DAIRYdb can be found on GitHub and the preprint has been deposited in BioRxiv (1). DAIRYdb was developed by Marco Meola, Etienne Rifa and their collaborators, who also provided most of the text for this post. Thanks Marco for this excellent addition to the database collection!

References

1. Meola M, Rifa E, Shani N, Delbes C, Berthoud H, Chassard C: DAIRYdb: A manually curated gold standard reference database for improved taxonomy annotation of 16S rRNA gene sequences from dairy products. bioRxiv, 386151 (2018). doi: 10.1101/386151
2. Bengtsson-Palme J, Hartmann M, Eriksson KM, Pal C, Thorell K, Larsson DGJ, Nilsson RH: Metaxa2: Improved identification and taxonomic classification of small and large subunit rRNA in metagenomic data. Molecular Ecology Resources, 15, 6, 1403–1414 (2015). doi: 10.1111/1755-0998.12399
3. Edgar RC: Accuracy of taxonomy prediction for 16S rRNA and fungal ITS sequences. PeerJ, 6, e4652 (2018). doi: 10.7717/peerj.4652
4. Ritari J, Salojärvi J, Last L, de Vos WM: Improved taxonomic assignment of human intestinal 16S rRNA sequences by a dedicated reference database. BMC Genomics, 16, 1, 1056 (2015). doi: 10.1186/s12864-015-2265-y
5. Newton ILG, Roeselers G: The effect of training set on the classification of honey bee gut microbiota using the naïve bayesian classifier. BMC Microbiology, 12, 1, 221 (2012). doi: 10.1186/1471-2180-12-221

A few days ago I posted about that Bioinformatics had published our paper on the Metaxa2 Database Builder (1). Today, I am happy to report that PeerJ has published the first paper in which the database builder is used to create a new Metaxa2 (2) database! My colleagues at Ohio State University has used the software to build a database for the COI gene (3), which is commonly used in arthropod barcoding. The used region was extracted from COI sequences from arthropod whole mitochondrion genomes, and employed to create a database containing sequences from all major arthropod clades, including all insect orders, all arthropod classes and the Onychophora, Tardigrada and Mollusca outgroups.

Similar to what we did in our evaluation of taxonomic classifiers used on non-rRNA barcoding regions (4), we performed a cross-validation analysis to characterize the relationship between the Metaxa2 reliability score, an estimate of classification confidence, and classification error probability. We used this analysis to select a reliability score threshold which minimized error. We then estimated classification sensitivity, false discovery rate and overclassification, the propensity to classify sequences from taxa not represented in the reference database.

Since the database builder was still in its early inception stages when we started doing this work, the software itself saw several improvements because of this project. We believe that our work on the COI database, as well as on the recently released database builder software, will help researchers in designing and evaluating classification databases for metabarcoding on arthropods and beyond. The database is included in the new Metaxa2 2.2 release, and is also downloadable from the Metaxa2 Database Repository (1). The open access paper can be found here.

References

1. Bengtsson-Palme J, Richardson RT, Meola M, Wurzbacher C, Tremblay ED, Thorell K, Kanger K, Eriksson KM, Bilodeau GJ, Johnson RM, Hartmann M, Nilsson RH: Metaxa2 Database Builder: Enabling taxonomic identification from metagenomic and metabarcoding data using any genetic marker. Bioinformatics, advance article (2018). doi: 10.1093/bioinformatics/bty482
2. Bengtsson-Palme J, Hartmann M, Eriksson KM, Pal C, Thorell K, Larsson DGJ, Nilsson RH: Metaxa2: Improved identification and taxonomic classification of small and large subunit rRNA in metagenomic data. Molecular Ecology Resources, 15, 6, 1403–1414 (2015). doi: 10.1111/1755-0998.12399
3. Richardson RT, Bengtsson-Palme J, Gardiner MM, Johnson RM: A reference cytochrome c oxidase subunit I database curated for hierarchical classification of arthropod metabarcoding data. PeerJ, 6, e5126 (2018). doi: 10.7717/peerj.5126
4. Richardson RT, Bengtsson-Palme J, Johnson RM: Evaluating and Optimizing the Performance of Software Commonly Used for the Taxonomic Classification of DNA Sequence Data. Molecular Ecology Resources, 17, 4, 760–769 (2017). doi: 10.1111/1755-0998.12628

This weekend, F1000Research put online the non-peer-reviewed version of the paper resulting from a workshop arranged by the JRC in Italy last year (1). (I will refer to this as a preprint, but at F1000Research the line is quite blurry between preprint and published paper.) The paper describes various challenges arising from the process of designing a benchmark strategy for bioinformatics pipelines (2) in the identification of antimicrobial resistance genes in next generation sequencing data.

The paper discusses issues about the benchmarking datasets used, testing samples, evaluation criteria for the performance of different tools, and how the benchmarking dataset should be created and distributed. Specially, we address the following questions:

• How should a benchmark strategy handle the current and expanding universe of NGS platforms?
• What should be the quality profile (in terms of read length, error rate, etc.) of in silico reference materials?
• Should different sets of reference materials be produced for each platform? In that case, how to ensure no bias is introduced in the process?
• Should in silico reference material be composed of the output of real experiments, or simulated read sets? If a combination is used, what is the optimal ratio?
• How is it possible to ensure that the simulated output has been simulated “correctly”?
• For real experiment datasets, how to avoid the presence of sensitive information?
• Regarding the quality metrics in the benchmark datasets (e.g. error rate, read quality), should these values be fixed for all datasets, or fall within specific ranges? How wide can/should these ranges be?
• How should the benchmark manage the different mechanisms by which bacteria acquire resistance?
• What is the set of resistance genes/mechanisms that need to be included in the benchmark? How should this set be agreed upon?
• Should datasets representing different sample types (e.g. isolated clones, environmental samples) be included in the same benchmark?
• Is a correct representation of different bacterial species (host genomes) important?
• How can the “true” value of the samples, against which the pipelines will be evaluated, be guaranteed?
• What is needed to demonstrate that the original sample has been correctly characterised, in case real experiments are used?
• How should the target performance thresholds (e.g. specificity, sensitivity, accuracy) for the benchmark suite be set?
• What is the impact of these performance thresholds on the required size of the sample set?
• How can the benchmark stay relevant when new resistance mechanisms are regularly characterized?
• How is the continued quality of the benchmark dataset ensured?
• Who should generate the benchmark resource?
• How can the benchmark resource be efficiently shared?

Of course, we have not answered all these questions, but I think we have come down to a decent description of the problems, which we see as an important foundation for solving these issues and implementing the benchmarking standard. Some of these issues were tackled in our review paper from last year on using metagenomics to study resistance genes in microbial communities (3). The paper also somewhat connects to the database curation paper we published in 2016 (4), although this time the strategies deal with the testing datasets rather than the actual databases. The paper is the first outcome of the workshop arranged by the JRC on “Next-generation sequencing technologies and antimicrobial resistance” held October 4-5 last year in Ispra, Italy. You can find the paper here (it’s open access).

References and notes

1. Angers-Loustau A, Petrillo M, Bengtsson-Palme J, Berendonk T, Blais B, Chan KG, Coque TM, Hammer P, Heß S, Kagkli DM, Krumbiegel C, Lanza VF, Madec J-Y, Naas T, O’Grady J, Paracchini V, Rossen JWA, Ruppé E, Vamathevan J, Venturi V, Van den Eede G: The challenges of designing a benchmark strategy for bioinformatics pipelines in the identification of antimicrobial resistance determinants using next generation sequencing technologies. F1000Research, 7, 459 (2018). doi: 10.12688/f1000research.14509.1
2. You may remember that I hate the term “pipeline” for bioinformatics protocols. I would have preferred if it was called workflows or similar, but the term “pipeline” has taken hold and I guess this is a battle where I have essentially lost. The bioinformatics workflows will be known as pipelines, for better and worse.
3. Bengtsson-Palme J, Larsson DGJ, Kristiansson E: Using metagenomics to investigate human and environmental resistomes. Journal of Antimicrobial Chemotherapy, 72, 2690–2703 (2017). doi: 10.1093/jac/dkx199
4. Bengtsson-Palme J, Boulund F, Edström R, Feizi A, Johnning A, Jonsson VA, Karlsson FH, Pal C, Pereira MB, Rehammar A, Sánchez J, Sanli K, Thorell K: Strategies to improve usability and preserve accuracy in biological sequence databases. Proteomics, 16, 18, 2454–2460 (2016). doi: 10.1002/pmic.201600034