New scientific software is constantly created and published in journals. These publications usually illustrate how the software improves over the existing state of the field. These may take the form of comparing the authors' software against others using benchmarks, test data or both.
The problem with these published comparisons is that they are subjective and not standardised across the community. Researchers are subject to authorship bias and likely select the results that portray their software in the best way. How can you compare two different pieces of scientific software when their corresponding publications test them on two different data? The current state of software publication in bioinformatics makes it difficult for a researcher to objectively evaluate which software works best for their own data.
Previous Assemblathon 1, Assemblathon 2, and Genome Assembly Gold-Standard Evaluations (GAGE) projects aimed to resolve this by objectively evaluating the current state of genome assembly. The approach of the Assemblathon was to release a set of read data and ask the genomics community to submit their best genome assembly. The GAGE approach took several different genome assemblers and ran them against their own test datasets. In both cases the quality and accuracy of genome assemblies was evaluated for accuracy and performance.
This project aims to improve on these approaches in two ways:
Assembly benchmarks will be run on a regular basis. This allows the latest developments and publications in genome assembly to constantly be evaluated against the current corpus of assemblers. This means new breakthroughs in assembly can be quickly evaluated and thereby shared with the wider bioinformatics community.
The genome assemblers and pipelines will be submitted by the bioinformatics community itself. This will allow large numbers of assemblers to be evaluated simultaneously and without requiring manual installation or setting of parameters. This effectively crowd-sources genome assemblers from anyone who wishes to participate.
These two goals are made possible using Linux Containers via Docker. All genome assemblers and associated pipeline should be built within a docker image and then hosted on Docker Hub. Our benchmarking pipeline will then pull the image and run it against an array of reference data sets. The produced assembly is evaluated against the reference sequence using Quast. The assembly metrics and results are then posted on this site.