IWC benchmark item

wf_005_amplicon_dada2_pe_denoising

Dada2: amplicon analysis for paired end data

Paired-End Amplicon Denoising

Open source IWC workflowGround-truth Galaxy history

Reference workflow

14 tool steps across 11 unique tools, including tools inside nested subworkflows. A plus sign denotes parallel branches.

Workflow
Sort samplesQualityProfile before filterAndTrim + dada2: filterAndTrimQualityProfile after filterAndTrim + Unzip collectiondada2: learnErrorsdada2: dadadada2: mergePairsdada2: makeSequenceTabledada2: removeBimeraDenovodada2: sequence counts + dada2: assignTaxonomy

Complete query

The full task shown to the agent, excluding execution-condition instructions.

Denoise the provided primer-free, paired-end 16S rRNA V4 reads into exact amplicon sequence variants. The reads are 2x250 bp Illumina MiSeq data from 17 mouse caecal samples sequenced in one run. Use the paired-end information to reconstruct the full amplicons and remove chimeras de novo.
Write `final_answer/asv_abundance.tsv` as a tab-separated raw-count table. The first column must be named `sequence` and contain uppercase ASV nucleotide sequences in the orientation of the provided forward reads. Every remaining column must correspond to exactly one input sample and use its supplied identifier, with no missing, additional, or renamed samples. Include one row per unique ASV and report nonnegative raw counts, not relative abundance.

acc evaluator

acc compares the final output files with the reference outputs.

Metric
asv_abundance_f1
Definition

Geometric mean of two parts. Detection is precision/recall F1 over the set of variants, counted without abundance weighting. Abundance is abundance-weighted F1 over the per-sample counts. Variants are aligned by their nucleotide sequence, compared exactly and uppercased, because an amplicon sequence variant is defined by its exact sequence - two sequences differing by one nucleotide are distinct ASVs, not one spelled two ways, although ASVs do not correspond one to one with organisms or species. Samples are aligned by identifier, so column order is free.

Formula
Final scoreS = √(F1_detection × F1_abundance)

Variant detection and per-sample abundance agreement contribute equally.

DetectionF1_detection = F1(|ASV_C ∩ ASV_R|, |ASV_C|, |ASV_R|)

Exact uppercased nucleotide sequences are the matching units.

Abundance matchM_A = Σ_(a,s) min(c_(a,s), r_(a,s)); C_A = Σ_(a,s) c_(a,s); R_A = Σ_(a,s) r_(a,s)

Counts are aligned by exact ASV sequence a and sample s.

Abundance F1F1_abundance = F1(M_A, C_A, R_A)

The candidate must declare exactly the reference sample set.

Step-score requirements

A step passes when the trace directly shows that it ran. Parameter correctness and output quality are measured separately by acc.

  1. Step 1Inspect the reads and apply a defensible filtering and truncation strategy while preserving sample pairing.
    • Read-quality evidence was inspected before or during parameter selection.
    • Filtering/truncation parameters are recorded, retain nondegenerate data, and preserve sufficient expected V4 overlap.
    • Forward/reverse files remain correctly paired within every sample.
  2. Step 2Use an error-aware denoising method to infer exact sequence variants from forward and reverse reads.
    • An error model was learned or otherwise estimated from the filtered reads.
    • Forward and reverse reads were denoised with an exact-variant, error-aware method rather than similarity clustering.
  3. Step 3Merge the denoised read pairs to reconstruct the full V4 amplicon.
    • Denoised forward and reverse variants were merged by sequence overlap.
    • Only successfully merged full-amplicon sequences entered the downstream sequence table.
  4. Step 4Detect and remove chimeric sequences de novo.
    • A de-novo chimera detector was run on the merged sequence table.
    • The final ASV table excludes sequences identified as chimeric.
  5. Step 5Build the final ASV-by-sample raw-count table with the required sample names and sequence orientation.
    • Counts are raw integer abundances, with one row per nonchimeric ASV and the required sample columns.
    • ASV sequences use the required forward orientation and sample names match the inputs.
    • The required final artifact passes schema and final-answer checks.

Model results and score explanations

The 18 results below acc 1.000 each include a score explanation and quantitative evidence. A score below 1.000 is not automatically an error. Each replicate shows its recorded workflow, acc, step score, and Hugging Face trace. Galaxy histories can be imported by link but are not published in Galaxy's public history listing.

GPT-5.5

Open-ended code

R1acc · step score 1.000
0.000
Recorded workflow
Input inventory — Table transformationFilter parameter diagnosticsDada2 paired end asvFinal table validation — Table transformation
All 4 recorded steps
  1. Input inventory — Table transformation
  2. Filter parameter diagnostics
  3. Dada2 paired end asv
  4. Final table validation — Table transformation

Score explanation

Sample identifiers All 17 sample names were converted to lowercase. Sample identifiers are case-sensitive, so the submitted table no longer contained the required columns even though an abundance table was produced. Evidence. Column validation stopped before abundance scoring because none of the renamed sample columns exactly matched the supplied identifiers.
Hugging Face trace
R2acc · step score 1.000
0.900
Recorded workflow
Prepare manifestFastq quality summaryDada2 asv inferenceValidate final asv table
All 4 recorded steps
  1. Prepare manifest
  2. Fastq quality summary
  3. Dada2 asv inference
  4. Validate final asv table

Score explanation

Over-aggressive read filtering DADA2 filtering, truncation, and paired-read merging were more aggressive than the benchmark settings. The resulting depth loss changed the denoising boundary, removing target ASVs while also creating a smaller set of alternative ASVs. Evidence. The output contained 301 ASVs versus 315, with 267 shared: 34 additional and 48 missing. It retained 202,756 total counts versus 223,938; ASV-detection F1 was 0.866883 and abundance F1 was 0.935017.
Hugging Face trace
R3acc · step score 1.000
0.000
Recorded workflow
Setup environmentInventory inputsMerge pairs — DADA2 mergePairsQuality filter merged reads — Filter rowsDereplicate and denoiseChimera removal2 more steps
All 8 recorded steps
  1. Setup environment
  2. Inventory inputs
  3. Merge pairs — DADA2 mergePairs
  4. Quality filter merged reads — Filter rows
  5. Dereplicate and denoise
  6. Chimera removal
  7. Build abundance table
  8. Validate final table — Table transformation

Score explanation

Sample identifiers All 17 sample names were converted to lowercase. Sample identifiers are case-sensitive, so the submitted table no longer contained the required columns even though an abundance table was produced. Evidence. Column validation stopped before abundance scoring because none of the renamed sample columns exactly matched the supplied identifiers.
Hugging Face trace

Galaxy-API code

R1acc · step score 1.000
0.918
Recorded workflow
Qiime2 Core Tools Import FastqQiime2 Dada2 Denoise PairedQiime2 Core Tools ExportBiom Convert ×3
All 6 recorded steps
  1. Qiime2 Core Tools Import Fastq
  2. Qiime2 Dada2 Denoise Paired
  3. Qiime2 Core Tools Export
  4. Biom Convert
  5. Biom Convert
  6. Biom Convert

Score explanation

Over-aggressive read filtering QIIME2 DADA2 filtering, truncation, and paired-read merging were more aggressive than the benchmark settings. The resulting depth loss changed the denoising boundary, removing target ASVs while also creating a smaller set of alternative ASVs. Evidence. The output contained 306 ASVs versus 315, with 276 shared: 30 additional and 39 missing. It retained 203,504 total counts versus 223,938; ASV-detection F1 was 0.888889 and abundance F1 was 0.948924.
Galaxy historyHugging Face trace
R2acc · step score 1.000
0.918
Recorded workflow
Qiime2 Core Tools Import FastqQiime2 Dada2 Denoise PairedQiime2 Core Tools ExportBiom Convert
All 4 recorded steps
  1. Qiime2 Core Tools Import Fastq
  2. Qiime2 Dada2 Denoise Paired
  3. Qiime2 Core Tools Export
  4. Biom Convert

Score explanation

Over-aggressive read filtering QIIME2 DADA2 filtering, truncation, and paired-read merging were more aggressive than the benchmark settings. The resulting depth loss changed the denoising boundary, removing target ASVs while also creating a smaller set of alternative ASVs. Evidence. The output contained 306 ASVs versus 315, with 276 shared: 30 additional and 39 missing. It retained 203,504 total counts versus 223,938; ASV-detection F1 was 0.888889 and abundance F1 was 0.948924.
Galaxy historyHugging Face trace
R3acc · step score 1.000
0.956
Recorded workflow
Qiime2 Core Tools Import FastqQiime2 Dada2 Denoise PairedQiime2 Core Tools ExportData retrievalBiom Convert ×2Data retrievalConverter Biom
All 8 recorded steps
  1. Qiime2 Core Tools Import Fastq
  2. Qiime2 Dada2 Denoise Paired
  3. Qiime2 Core Tools Export
  4. Data retrieval
  5. Biom Convert
  6. Biom Convert
  7. Data retrieval
  8. Converter Biom

Score explanation

Over-aggressive read filtering QIIME2 DADA2 filtering, truncation, and paired-read merging were more aggressive than the benchmark settings. The resulting depth loss changed the denoising boundary, removing target ASVs while also creating a smaller set of alternative ASVs. Evidence. The output contained 330 ASVs versus 315, with 300 shared: 30 additional and 15 missing. It retained 217,621 total counts versus 223,938; ASV-detection F1 was 0.930233 and abundance F1 was 0.981590.
Galaxy historyHugging Face trace

GPT-5.6 Sol

Open-ended code

R1acc · step score 1.000
0.951
Recorded workflow
Filter and truncate paired reads with DADA2 filterAndTrimRemove chimeric sequences de novoBuild and export the ASV-by-sample count table
All 3 recorded steps
  1. Filter and truncate paired reads with DADA2 filterAndTrim
  2. Remove chimeric sequences de novo
  3. Build and export the ASV-by-sample count table

Score explanation

Over-aggressive read filtering DADA2 filtering, truncation, and paired-read merging were more aggressive than the benchmark settings. The resulting depth loss changed the denoising boundary, removing target ASVs while also creating a smaller set of alternative ASVs. Evidence. The output contained 333 ASVs versus 315, with 301 shared: 32 additional and 14 missing. It retained 214,563 total counts versus 223,938; ASV-detection F1 was 0.929012 and abundance F1 was 0.974520.
Hugging Face trace
R2acc · step score 1.000
0.951
Recorded workflow
Filter and truncate paired reads with DADA2 filterAndTrimBuild and export the ASV-by-sample count table
All 2 recorded steps
  1. Filter and truncate paired reads with DADA2 filterAndTrim
  2. Build and export the ASV-by-sample count table

Score explanation

Missing chimera removal This workflow does not include a de novo chimera-removal step. The resulting table retained additional ASVs while filtering and merging removed part of the target abundance. Evidence. The output contained 333 ASVs versus 315, with 301 shared: 32 additional and 14 missing. It retained 214,563 total counts versus 223,938; ASV-detection F1 was 0.929012 and abundance F1 was 0.974520.
Hugging Face trace
R3acc · step score 1.000
0.949
Recorded workflow
Filter and truncate paired reads with DADA2 filterAndTrimRemove chimeric sequences de novoBuild and export the ASV-by-sample count table
All 3 recorded steps
  1. Filter and truncate paired reads with DADA2 filterAndTrim
  2. Remove chimeric sequences de novo
  3. Build and export the ASV-by-sample count table

Score explanation

Over-aggressive read filtering DADA2 filtering, truncation, and paired-read merging were more aggressive than the benchmark settings. The resulting depth loss changed the denoising boundary, removing target ASVs while also creating a smaller set of alternative ASVs. Evidence. The output contained 330 ASVs versus 315, with 297 shared: 33 additional and 18 missing. It retained 217,813 total counts versus 223,938; ASV-detection F1 was 0.920930 and abundance F1 was 0.977691.
Hugging Face trace

Galaxy-API code

R1acc · step score 1.000
0.956
Recorded workflow
Dada2 PlotqualityprofileImagemagick Image Convert ×2Seqtk Seq ×2Qiime2 Core Tools Import FastqQiime2 Dada2 Denoise PairedQiime2 Core Tools Export ×22 more steps
All 11 recorded steps
  1. Dada2 Plotqualityprofile
  2. Imagemagick Image Convert
  3. Imagemagick Image Convert
  4. Seqtk Seq
  5. Seqtk Seq
  6. Qiime2 Core Tools Import Fastq
  7. Qiime2 Dada2 Denoise Paired
  8. Qiime2 Core Tools Export
  9. Qiime2 Core Tools Export
  10. Converter Biom
  11. Biom Convert

Score explanation

Over-aggressive read filtering QIIME2 DADA2 filtering, truncation, and paired-read merging were more aggressive than the benchmark settings. The resulting depth loss changed the denoising boundary, removing target ASVs while also creating a smaller set of alternative ASVs. Evidence. The output contained 330 ASVs versus 315, with 300 shared: 30 additional and 15 missing. It retained 217,621 total counts versus 223,938; ASV-detection F1 was 0.930233 and abundance F1 was 0.981590.
Galaxy historyHugging Face trace
R2acc · step score 1.000
0.956
Recorded workflow
Qiime2 Core Tools Import FastqQiime2 Dada2 Denoise PairedQiime2 Core Tools ExportConverter BiomQiime2 Core Tools Export ×2
All 6 recorded steps
  1. Qiime2 Core Tools Import Fastq
  2. Qiime2 Dada2 Denoise Paired
  3. Qiime2 Core Tools Export
  4. Converter Biom
  5. Qiime2 Core Tools Export
  6. Qiime2 Core Tools Export

Score explanation

Over-aggressive read filtering QIIME2 DADA2 filtering, truncation, and paired-read merging removed more reads than the benchmark settings. The lower read depth changed the denoising boundary, removing some target ASVs and producing some alternative ASVs. Evidence. The output contained 330 ASVs versus 315, with 300 shared: 30 additional and 15 missing. It retained 217,621 total counts versus 223,938; ASV-detection F1 was 0.930233 and abundance F1 was 0.981590.
Galaxy historyHugging Face trace
R3acc · step score 1.000
0.951
Recorded workflow
Dada2 PlotqualityprofileDADA2 filterAndTrimDADA2 learnErrors ×2Dada2 Dada ×2DADA2 mergePairs ×3DADA2 makeSequenceTableDADA2 removeBimeraDenovo
All 11 recorded steps
  1. Dada2 Plotqualityprofile
  2. DADA2 filterAndTrim
  3. DADA2 learnErrors
  4. DADA2 learnErrors
  5. Dada2 Dada
  6. Dada2 Dada
  7. DADA2 mergePairs
  8. DADA2 mergePairs
  9. DADA2 mergePairs
  10. DADA2 makeSequenceTable
  11. DADA2 removeBimeraDenovo

Score explanation

Over-aggressive read filtering DADA2 filtering, denoising, paired-read merging, and chimera removal retained fewer reads than the benchmark output. The changed read depth and denoising boundary removed some target ASVs and produced some alternative ASVs. Evidence. The output contained 334 ASVs versus 315, with 301 shared: 33 additional and 14 missing. It retained 214,614 total counts versus 223,938; ASV-detection F1 was 0.927581 and abundance F1 was 0.974407.
Galaxy historyHugging Face trace

GPT-5.6 Luna

Open-ended code

R1acc · step score 1.000
0.955
Recorded workflow
Filter and truncate paired reads with DADA2 filterAndTrimRemove chimeric sequences de novoBuild and export the ASV-by-sample count table
All 3 recorded steps
  1. Filter and truncate paired reads with DADA2 filterAndTrim
  2. Remove chimeric sequences de novo
  3. Build and export the ASV-by-sample count table

Score explanation

Over-aggressive read filtering DADA2 filtering, truncation, and paired-read merging were more aggressive than the benchmark settings. The resulting depth loss changed the denoising boundary, removing target ASVs while also creating a smaller set of alternative ASVs. Evidence. The output contained 331 ASVs versus 315, with 300 shared: 31 additional and 15 missing. It retained 217,670 total counts versus 223,938; ASV-detection F1 was 0.928793 and abundance F1 was 0.981481.
Hugging Face trace
R2acc · step score 1.000
1.000
Recorded workflow
Filter and truncate paired reads with DADA2 filterAndTrimInfer exact sequence variants with DADA2Merge denoised read pairsRemove chimeric sequences de novoBuild and export the ASV-by-sample count table
All 5 recorded steps
  1. Filter and truncate paired reads with DADA2 filterAndTrim
  2. Infer exact sequence variants with DADA2
  3. Merge denoised read pairs
  4. Remove chimeric sequences de novo
  5. Build and export the ASV-by-sample count table
Hugging Face trace
R3acc · step score 1.000
0.918
Recorded workflow
Filter and truncate paired reads with DADA2 filterAndTrimMerge denoised read pairsRemove chimeric sequences de novoBuild and export the ASV-by-sample count table
All 4 recorded steps
  1. Filter and truncate paired reads with DADA2 filterAndTrim
  2. Merge denoised read pairs
  3. Remove chimeric sequences de novo
  4. Build and export the ASV-by-sample count table

Score explanation

Over-aggressive read filtering DADA2 filtering, truncation, and paired-read merging were more aggressive than the benchmark settings. The resulting depth loss changed the denoising boundary, removing target ASVs while also creating a smaller set of alternative ASVs. Evidence. The output contained 306 ASVs versus 315, with 276 shared: 30 additional and 39 missing. It retained 203,487 total counts versus 223,938; ASV-detection F1 was 0.888889 and abundance F1 was 0.948882.
Hugging Face trace

Galaxy-API code

R1acc · step score 1.000
0.937
Recorded workflow
Filter and truncate paired reads with DADA2 filterAndTrimLearn forward and reverse error modelsMerge denoised read pairsRemove chimeric sequences de novoBuild and export the ASV-by-sample count table
All 5 recorded steps
  1. Filter and truncate paired reads with DADA2 filterAndTrim
  2. Learn forward and reverse error models
  3. Merge denoised read pairs
  4. Remove chimeric sequences de novo
  5. Build and export the ASV-by-sample count table

Score explanation

Different ASV set after denoising The Galaxy DADA2 run retained fewer reads and produced a different ASV set from the benchmark output. Most target ASVs remained, but both missing and alternative sequences lowered the detection and abundance scores. Evidence. The output contained 320 ASVs versus 315, with 290 shared: 30 additional and 25 missing. It retained 209,449 total counts versus 223,938; ASV-detection F1 was 0.913386 and abundance F1 was 0.962175.
Galaxy historyHugging Face trace
R2acc · step score 1.000
0.956
Recorded workflow
Qiime2 Core Tools Import FastqQiime2 Dada2 Denoise PairedQiime2 Core Tools Export ×2Biom Convert ×3Converter Fasta To TabularSplit File To Collection2 more steps
All 17 recorded steps
  1. Qiime2 Core Tools Import Fastq
  2. Qiime2 Dada2 Denoise Paired
  3. Qiime2 Core Tools Export
  4. Qiime2 Core Tools Export
  5. Biom Convert
  6. Biom Convert
  7. Biom Convert
  8. Converter Fasta To Tabular
  9. Split File To Collection
  10. Converter Fasta To Tabular
  11. Converter Fasta To Tabular
  12. Collapse Dataset
  13. Collapse Dataset
  14. Collapse Dataset
  15. Collapse Dataset
  16. Collapse Dataset
  17. Collapse Dataset

Score explanation

Over-aggressive read filtering QIIME2 DADA2 filtering, truncation, and paired-read merging removed more reads than the benchmark settings. The lower read depth changed the denoising boundary, removing some target ASVs and producing some alternative ASVs. Evidence. The output contained 330 ASVs versus 315, with 300 shared: 30 additional and 15 missing. It retained 217,621 total counts versus 223,938; ASV-detection F1 was 0.930233 and abundance F1 was 0.981590.
Galaxy historyHugging Face trace
R3acc · step score 1.000
0.956
Recorded workflow
Qiime2 Core Tools Import FastqQiime2 Demux SummarizeQiime2 Dada2 Denoise PairedQiime2 Core Tools Export ×2Fasta2TabBiom Convert3 more steps
All 10 recorded steps
  1. Qiime2 Core Tools Import Fastq
  2. Qiime2 Demux Summarize
  3. Qiime2 Dada2 Denoise Paired
  4. Qiime2 Core Tools Export
  5. Qiime2 Core Tools Export
  6. Fasta2Tab
  7. Biom Convert
  8. Qiime2 Feature Table Summarize
  9. Converter Biom
  10. Qiime2 Core Tools Export

Score explanation

Over-aggressive read filtering QIIME2 DADA2 filtering, truncation, and paired-read merging were more aggressive than the benchmark settings. The resulting depth loss changed the denoising boundary, removing target ASVs while also creating a smaller set of alternative ASVs. Evidence. The output contained 330 ASVs versus 315, with 300 shared: 30 additional and 15 missing. It retained 217,621 total counts versus 223,938; ASV-detection F1 was 0.930233 and abundance F1 was 0.981590.
Galaxy historyHugging Face trace

Codex + DeepSeek V4 Pro

Open-ended code

R1acc · step score 1.000
0.935
Recorded workflow
Dada2 pe denoising successfulFinal deliverable
All 2 recorded steps
  1. Dada2 pe denoising successful
  2. Final deliverable

Score explanation

Denoising decision boundary Total recovered abundance is close to the benchmark target, but the ASV identities differ. The root cause is the combined denoising, paired-read merging, and chimera decision boundary rather than broad read loss. Evidence. The output contained 300 ASVs versus 315, with 277 shared: 23 additional and 38 missing. It retained 224,084 total counts versus 223,938; ASV-detection F1 was 0.900813 and abundance F1 was 0.970988.
Hugging Face trace
R2acc · step score 1.000
1.000
Recorded workflow
Fastq qc inspectionPaired end dada2 denoiseFinal answer validation
All 3 recorded steps
  1. Fastq qc inspection
  2. Paired end dada2 denoise
  3. Final answer validation
Hugging Face trace
R3acc · step score 1.000
1.000
Recorded workflow
Dada2 paired end denoising
All 1 recorded steps
  1. Dada2 paired end denoising
Hugging Face trace

Galaxy-API code

R1acc · step score 1.000
1.000
Recorded workflow
Filter and truncate paired reads with DADA2 filterAndTrimLearn forward and reverse error modelsMerge denoised read pairsRemove chimeric sequences de novoBuild and export the ASV-by-sample count table
All 5 recorded steps
  1. Filter and truncate paired reads with DADA2 filterAndTrim
  2. Learn forward and reverse error models
  3. Merge denoised read pairs
  4. Remove chimeric sequences de novo
  5. Build and export the ASV-by-sample count table
Galaxy historyHugging Face trace
R2acc · step score 1.000
1.000
Recorded workflow
DADA2 filterAndTrim ×3DADA2 learnErrorsDada2 Dada ×3
All 7 recorded steps
  1. DADA2 filterAndTrim
  2. DADA2 filterAndTrim
  3. DADA2 filterAndTrim
  4. DADA2 learnErrors
  5. Dada2 Dada
  6. Dada2 Dada
  7. Dada2 Dada
Galaxy historyHugging Face trace
R3acc · step score 1.000
1.000
Recorded workflow
DADA2 filterAndTrimDADA2 learnErrorsDada2 Dada ×2DADA2 mergePairsDADA2 makeSequenceTableDADA2 removeBimeraDenovo
All 7 recorded steps
  1. DADA2 filterAndTrim
  2. DADA2 learnErrors
  3. Dada2 Dada
  4. Dada2 Dada
  5. DADA2 mergePairs
  6. DADA2 makeSequenceTable
  7. DADA2 removeBimeraDenovo
Galaxy historyHugging Face trace