IWC benchmark item

wf_007_vgp_mitogenome_assembly

Mitogenome Assembly VGP0

VGP Mitochondrial Genome Assembly

Open source IWC workflowGround-truth Galaxy history

Reference workflow

5 tool steps across 3 unique tools, including tools inside nested subworkflows. A plus sign denotes parallel branches.

Workflow
Compose text parameter value + MitoHiFiMitoHiFiCompress file(s)

Complete query

The full task shown to the agent, excluding execution-condition instructions.

Assemble the complete mitochondrial genome of `Agrius convolvuli` from the provided PacBio HiFi reads.
Write the assembled mitochondrial nucleotide sequence to `final_answer/mitogenome.fasta.gz` as a gzip-compressed FASTA containing exactly one non-empty, nonredundant sequence record.

acc evaluator

acc compares the final output files with the reference outputs.

Metric
canonical_kmer_f1
Definition

Precision/recall F1 over canonical 31-mer multiplicities. Canonical k-mers make the score strand-invariant, and nearly rotation-invariant for a circular genome: an assembly reported on the opposite strand is not penalised at all, while changing the reported origin replaces only the k-mers that spanned the old linear boundary and therefore incurs a small penalty - measured at 0.9981 for a 5 kb rotation. Missing, unsupported and duplicated sequence all reduce the score.

Formula
Final scoreS = F1(M, C, R)

Canonical 31-mer multiplicities are the matching units.

Multiset matchM = Σ_k min(n_C(k), n_R(k)); C = Σ_k n_C(k); R = Σ_k n_R(k)

Each k is the lexicographically canonical form of a 31-mer and its reverse complement.

Record constraintnumber of nonempty FASTA records = 1

This is a validity requirement evaluated before the k-mer score.

Step-score requirements

A step passes when the trace directly shows that it ran. Parameter correctness and output quality are measured separately by acc.

  1. Step 1Use the PacBio HiFi reads to produce one or more mitochondrial assembly candidates.
    • A successful assembly or equivalent read-derived reconstruction used the provided PacBio HiFi reads.
    • One or more mitochondrial candidate contigs/sequences were produced from that execution.
  2. Step 2Identify and resolve one complete, nonredundant mitochondrial sequence, including checks for competing contigs, redundant ends, and completeness.
    • Competing mitochondrial candidate contigs were compared or ruled out.
    • Circular/redundant terminal overlap was checked and removed without duplicating sequence.
    • Completeness was assessed using length, expected mitochondrial features, read support, or an equivalently defensible check.
    • Exactly one complete nonredundant sequence was selected from the read-derived candidates.
  3. Step 3Write exactly one nonempty mitochondrial FASTA record and gzip it.
    • The gzip file is readable and contains exactly one nonempty FASTA record.
    • The record is the sequence selected in step 2 and passes final-answer checks.

Model results and score explanations

The 21 results below acc 1.000 each include a score explanation and quantitative evidence. A score below 1.000 is not automatically an error. Each replicate shows its recorded workflow, acc, step score, and Hugging Face trace. Galaxy histories can be imported by link but are not published in Galaxy's public history listing.

GPT-5.5

Open-ended code

R1acc · step score 1.000
0.000
Recorded workflow
Input fastq stats — Table transformationKmer count k31Kmer histogram k31Dump highcopy kmers ci1000Compile highcopy read selectorDump highcopy kmers ci50007 more steps
All 13 recorded steps
  1. Input fastq stats — Table transformation
  2. Kmer count k31
  3. Kmer histogram k31
  4. Dump highcopy kmers ci1000
  5. Compile highcopy read selector
  6. Dump highcopy kmers ci5000
  7. Select highcopy reads ci5000 min5000
  8. Coverage window unitig sweep
  9. Anchor unitig1 raw read mapping
  10. Raven anchor unitig1 polished disabled
  11. Extract circular candidate from raven utg210
  12. Polish candidate with anchor reads — SAMtools
  13. Write final deliverable

Score explanation

Wrong assembly candidate The selected assembly candidate is not the benchmark mitochondrial contig. The file satisfies the one-record FASTA structure, but its sequence has no shared 31-mer evidence with the target. Evidence. Zero of 16,249 submitted 31-mers matched any of the 15,419 target 31-mers; the sequence had no ambiguous 31-mer windows.
Hugging Face trace
R2acc · step score 1.000
0.942
Recorded workflow
Input read statsKmer tool setupKmer count k31Kmer histogram k31Deterministic read subsample p10 s42Deterministic read subsample p01 s4310 more steps
All 16 recorded steps
  1. Input read stats
  2. Kmer tool setup
  3. Kmer count k31
  4. Kmer histogram k31
  5. Deterministic read subsample p10 s42
  6. Deterministic read subsample p01 s43
  7. Alternate assembler setup
  8. Consensus tool setup
  9. Full reads to flye draft paf
  10. Flye draft coverage candidate selection
  11. Extract mitochondrial unit
  12. Full reads to mito unit mapping — SAMtools
  13. Polish mito unit consensus
  14. Circularity and nonredundancy validation
  15. Final deliverable write and validate
  16. Final base composition validation

Score explanation

Incomplete consensus polishing The selected mitochondrial unit was valid and near the expected length, but consensus polishing did not recover the sample-specific sequence. The remaining base differences alter hundreds of overlapping 31-mers. Evidence. Of 15,411 submitted and 15,419 target 31-mers, 14,521 matched; 890 were submitted-only and 898 were target-only. The file contained no ambiguous 31-mer windows.
Hugging Face trace
R3acc · step score 1.000
0.996
Recorded workflow
Read statisticsKmer count ci80 k31 m4Kmer histogram ci80 k31Extract highcopy reads ci1000 frac02 — Filter rowsExtract highcopy reads ci1000 frac05 — Filter rowsExtract highcopy reads ci1000 frac08 — Filter rows5 more steps
All 11 recorded steps
  1. Read statistics
  2. Kmer count ci80 k31 m4
  3. Kmer histogram ci80 k31
  4. Extract highcopy reads ci1000 frac02 — Filter rows
  5. Extract highcopy reads ci1000 frac05 — Filter rows
  6. Extract highcopy reads ci1000 frac08 — Filter rows
  7. Extract low gc highcopy reads — Table transformation
  8. Assembler env create
  9. Derive and polish circular unit — SAMtools
  10. Validate polished mitogenome all reads — SAMtools
  11. Write final deliverable

Score explanation

Assembly consensus difference The assembly selected the correct mitochondrial contig, but its polished consensus still differs at a small number of bases from the benchmark target. Those local sequence differences propagate across overlapping 31-mers. Evidence. Of 15,418 submitted and 15,419 target 31-mers, 15,358 matched; 60 were submitted-only and 61 were target-only. The file contained no ambiguous 31-mer windows.
Hugging Face trace

Galaxy-API code

R1acc · step score 1.000
0.000
Recorded workflow
HifiasmGfa To FaData retrieval
All 3 recorded steps
  1. Hifiasm
  2. Gfa To Fa
  3. Data retrieval

Score explanation

Wrong assembly candidate The selected assembly candidate is not the benchmark mitochondrial contig. The file satisfies the one-record FASTA structure, but its sequence has no shared 31-mer evidence with the target. Evidence. Zero of 14,419 submitted 31-mers matched any of the 15,419 target 31-mers; the sequence had no ambiguous 31-mer windows.
Galaxy historyHugging Face trace
R3acc · step score 1.000
0.994
Recorded workflow
HifiasmMitoHiFi ×2
All 3 recorded steps
  1. Hifiasm
  2. MitoHiFi
  3. MitoHiFi

Score explanation

Assembly consensus difference The assembly selected the correct mitochondrial contig, but its polished consensus still differs at a small number of bases from the benchmark target. Those local sequence differences propagate across overlapping 31-mers. Evidence. Of 15,418 submitted and 15,419 target 31-mers, 15,330 matched; 88 were submitted-only and 89 were target-only. The file contained no ambiguous 31-mer windows.
Galaxy historyHugging Face trace

GPT-5.6 Sol

Open-ended code

R1acc · step score 1.000
0.996
Recorded workflow
Assemble candidate contigs with FlyeIdentify and resolve the complete mitochondrial contigCheck redundant ends and competing contigsWrite exactly one gzipped mitochondrial FASTA record
All 4 recorded steps
  1. Assemble candidate contigs with Flye
  2. Identify and resolve the complete mitochondrial contig
  3. Check redundant ends and competing contigs
  4. Write exactly one gzipped mitochondrial FASTA record

Score explanation

Assembly consensus difference The assembly selected the correct mitochondrial contig, but its polished consensus still differs at a small number of bases from the benchmark target. Those local sequence differences propagate across overlapping 31-mers. Evidence. Of 15,418 submitted and 15,419 target 31-mers, 15,358 matched; 60 were submitted-only and 61 were target-only. The file contained no ambiguous 31-mer windows.
Hugging Face trace
R2acc · step score 1.000
0.996
Recorded workflow
Assemble candidate contigs with hifiasmIdentify and resolve the complete mitochondrial contigCheck redundant ends and competing contigsWrite exactly one gzipped mitochondrial FASTA record
All 4 recorded steps
  1. Assemble candidate contigs with hifiasm
  2. Identify and resolve the complete mitochondrial contig
  3. Check redundant ends and competing contigs
  4. Write exactly one gzipped mitochondrial FASTA record

Score explanation

Assembly consensus difference The assembly selected the correct mitochondrial contig, but its polished consensus still differs at a small number of bases from the benchmark target. Those local sequence differences propagate across overlapping 31-mers. Evidence. Of 15,418 submitted and 15,419 target 31-mers, 15,358 matched; 60 were submitted-only and 61 were target-only. The file contained no ambiguous 31-mer windows.
Hugging Face trace
R3acc · step score 1.000
0.996
Recorded workflow
Assemble candidate contigs with hifiasmAssemble candidate contigs with FlyeIdentify and resolve the complete mitochondrial contigCheck redundant ends and competing contigsWrite exactly one gzipped mitochondrial FASTA record
All 5 recorded steps
  1. Assemble candidate contigs with hifiasm
  2. Assemble candidate contigs with Flye
  3. Identify and resolve the complete mitochondrial contig
  4. Check redundant ends and competing contigs
  5. Write exactly one gzipped mitochondrial FASTA record

Score explanation

Assembly consensus difference The assembly selected the correct mitochondrial contig, but its polished consensus still differs at a small number of bases from the benchmark target. Those local sequence differences propagate across overlapping 31-mers. Evidence. Of 15,418 submitted and 15,419 target 31-mers, 15,358 matched; 60 were submitted-only and 61 were target-only. The file contained no ambiguous 31-mer windows.
Hugging Face trace

Galaxy-API code

R1acc · step score 1.000
0.996
Recorded workflow
MitoHiFiMinimap2SAMtoolsBedtools BamtofastqFlyeSeqkit Grep ×22 more steps
All 9 recorded steps
  1. MitoHiFi
  2. Minimap2
  3. SAMtools
  4. Bedtools Bamtofastq
  5. Flye
  6. Seqkit Grep
  7. Seqkit Grep
  8. Filter rows
  9. Ncbi Blastn Wrapper

Score explanation

Assembly consensus difference The assembly selected the correct mitochondrial contig, but its polished consensus still differs at a small number of bases from the benchmark target. Those local sequence differences propagate across overlapping 31-mers. Evidence. Of 15,418 submitted and 15,419 target 31-mers, 15,358 matched; 60 were submitted-only and 61 were target-only. The file contained no ambiguous 31-mer windows.
Galaxy historyHugging Face trace
R2acc · step score 1.000
0.996
Recorded workflow
Flye ×2Data retrievalSeqtk SubseqData retrievalSeqtk SubseqMitoHiFi ×2
All 8 recorded steps
  1. Flye
  2. Flye
  3. Data retrieval
  4. Seqtk Subseq
  5. Data retrieval
  6. Seqtk Subseq
  7. MitoHiFi
  8. MitoHiFi

Score explanation

Assembly consensus difference The assembly selected the correct mitochondrial contig, but its polished consensus still differs at a small number of bases from the benchmark target. Those local sequence differences propagate across overlapping 31-mers. Evidence. Of 15,419 submitted and 15,419 target 31-mers, 15,361 matched; 58 were submitted-only and 58 were target-only. The file contained no ambiguous 31-mer windows.
Galaxy historyHugging Face trace

GPT-5.6 Luna

Open-ended code

R1acc · step score 1.000
0.996
Recorded workflow
Assemble candidate contigs with hifiasmAssemble candidate contigs with FlyeIdentify and resolve the complete mitochondrial contigCheck redundant ends and competing contigsWrite exactly one gzipped mitochondrial FASTA record
All 5 recorded steps
  1. Assemble candidate contigs with hifiasm
  2. Assemble candidate contigs with Flye
  3. Identify and resolve the complete mitochondrial contig
  4. Check redundant ends and competing contigs
  5. Write exactly one gzipped mitochondrial FASTA record

Score explanation

Assembly consensus difference The assembly selected the correct mitochondrial contig, but its polished consensus still differs at a small number of bases from the benchmark target. Those local sequence differences propagate across overlapping 31-mers. Evidence. Of 15,418 submitted and 15,419 target 31-mers, 15,358 matched; 60 were submitted-only and 61 were target-only. The file contained no ambiguous 31-mer windows.
Hugging Face trace
R2acc · step score 1.000
0.996
Recorded workflow
Assemble candidate contigs with hifiasmAssemble candidate contigs with FlyeIdentify and resolve the complete mitochondrial contigWrite exactly one gzipped mitochondrial FASTA record
All 4 recorded steps
  1. Assemble candidate contigs with hifiasm
  2. Assemble candidate contigs with Flye
  3. Identify and resolve the complete mitochondrial contig
  4. Write exactly one gzipped mitochondrial FASTA record

Score explanation

Assembly consensus difference The assembly selected the correct mitochondrial contig, but its polished consensus still differs at a small number of bases from the benchmark target. Those local sequence differences propagate across overlapping 31-mers. Evidence. Of 15,418 submitted and 15,419 target 31-mers, 15,358 matched; 60 were submitted-only and 61 were target-only. The file contained no ambiguous 31-mer windows.
Hugging Face trace
R3acc · step score 1.000
0.996
Recorded workflow
Assemble candidate contigs with hifiasmAssemble candidate contigs with FlyeIdentify and resolve the complete mitochondrial contigWrite exactly one gzipped mitochondrial FASTA record
All 4 recorded steps
  1. Assemble candidate contigs with hifiasm
  2. Assemble candidate contigs with Flye
  3. Identify and resolve the complete mitochondrial contig
  4. Write exactly one gzipped mitochondrial FASTA record

Score explanation

Assembly consensus difference The assembly selected the correct mitochondrial contig, but its polished consensus still differs at a small number of bases from the benchmark target. Those local sequence differences propagate across overlapping 31-mers. Evidence. Of 15,418 submitted and 15,419 target 31-mers, 15,358 matched; 60 were submitted-only and 61 were target-only. The file contained no ambiguous 31-mer windows.
Hugging Face trace

Galaxy-API code

R1acc · step score 1.000
0.995
Recorded workflow
Seqtk FqchkFlyeMitoHiFi ×2Data retrievalSeqtk Subseq
All 6 recorded steps
  1. Seqtk Fqchk
  2. Flye
  3. MitoHiFi
  4. MitoHiFi
  5. Data retrieval
  6. Seqtk Subseq

Score explanation

Public-record substitution The output is an exact public-record sequence rather than the sample-derived benchmark consensus. It is biologically close, but it omits sample-specific consensus differences. Evidence. Of 15,415 submitted and 15,419 target 31-mers, 15,347 matched; 68 were submitted-only and 72 were target-only.
Galaxy historyHugging Face trace
R2acc · step score 1.000
0.995
Recorded workflow
MitoHiFi ×2
All 2 recorded steps
  1. MitoHiFi
  2. MitoHiFi

Score explanation

Public-record substitution The output is an exact public-record sequence rather than the sample-derived benchmark consensus. It is biologically close, but it omits sample-specific consensus differences. Evidence. Of 15,415 submitted and 15,419 target 31-mers, 15,347 matched; 68 were submitted-only and 72 were target-only.
Galaxy historyHugging Face trace
R3acc · step score 1.000
0.995
Recorded workflow
MitoHiFi ×2Compress FileFasta Stats
All 4 recorded steps
  1. MitoHiFi
  2. MitoHiFi
  3. Compress File
  4. Fasta Stats

Score explanation

Public-record substitution The output is an exact public-record sequence rather than the sample-derived benchmark consensus. It is biologically close, but it omits sample-specific consensus differences. Evidence. Of 15,415 submitted and 15,419 target 31-mers, 15,347 matched; 68 were submitted-only and 72 were target-only.
Galaxy historyHugging Face trace

Codex + DeepSeek V4 Pro

Open-ended code

R1acc · step score 1.000
0.996
Recorded workflow
Input validationRead profilingRandom sample 30000Hifiasm 30000 lowmemRandom sample 100000Hifiasm 100000 lowmem2 more steps
All 8 recorded steps
  1. Input validation
  2. Read profiling
  3. Random sample 30000
  4. Hifiasm 30000 lowmem
  5. Random sample 100000
  6. Hifiasm 100000 lowmem
  7. Mito consensus validation — SAMtools
  8. Final packaging

Score explanation

Assembly consensus difference The assembly selected the correct mitochondrial contig, but its polished consensus still differs at a small number of bases from the benchmark target. Those local sequence differences propagate across overlapping 31-mers. Evidence. Of 15,418 submitted and 15,419 target 31-mers, 15,358 matched; 60 were submitted-only and 61 were target-only. The file contained no ambiguous 31-mer windows.
Hugging Face trace
R2acc · step score 1.000
0.000
Recorded workflow
Input statsHifiasm pilot 20k no bloomCandidate assembly 50kCandidate coverage screen — SAMtoolsFull read mapping to seedExtract mito reads5 more steps
All 11 recorded steps
  1. Input stats
  2. Hifiasm pilot 20k no bloom
  3. Candidate assembly 50k
  4. Candidate coverage screen — SAMtools
  5. Full read mapping to seed
  6. Extract mito reads
  7. Hifiasm mito reads
  8. Polish align mito reads — SAMtools
  9. Consensus polish — SAMtools
  10. Final validation — SAMtools
  11. Write final deliverable

Score explanation

Wrong assembly candidate The selected assembly candidate is not the benchmark mitochondrial contig. The file satisfies the one-record FASTA structure, but its sequence has no shared 31-mer evidence with the target. Evidence. Zero of 15,319 submitted 31-mers matched any of the 15,419 target 31-mers; the sequence had no ambiguous 31-mer windows.
Hugging Face trace
R3acc · step score 1.000
0.995
Recorded workflow
Inspect inputsDownload mitochondrial reference for read selection and closureMap hicreads to mito reference — SAMtoolsExtract and downsample mito reads — SAMtoolsAssemble mito hifiasm subsample 151 bloom disabledAssemble mito hifiasm subsample 500 bloom disabled2 more steps
All 8 recorded steps
  1. Inspect inputs
  2. Download mitochondrial reference for read selection and closure
  3. Map hicreads to mito reference — SAMtools
  4. Extract and downsample mito reads — SAMtools
  5. Assemble mito hifiasm subsample 151 bloom disabled
  6. Assemble mito hifiasm subsample 500 bloom disabled
  7. Finalize nonredundant mitogenome
  8. Validate and compress final

Score explanation

Public-record substitution The output is an exact public-record sequence rather than the sample-derived benchmark consensus. It is biologically close, but it omits sample-specific consensus differences. Evidence. Of 15,415 submitted and 15,419 target 31-mers, 15,347 matched; 68 were submitted-only and 72 were target-only.
Hugging Face trace

Galaxy-API code

R2acc · step score 1.000
0.995
Recorded workflow
MitoHiFi ×2
All 2 recorded steps
  1. MitoHiFi
  2. MitoHiFi

Score explanation

Public-record substitution The output is an exact public-record sequence rather than the sample-derived benchmark consensus. It is biologically close, but it omits sample-specific consensus differences. Evidence. Of 15,415 submitted and 15,419 target 31-mers, 15,347 matched; 68 were submitted-only and 72 were target-only.
Galaxy historyHugging Face trace
R3acc · step score 1.000
0.991
Recorded workflow
MitoHiFiMinimap2SAMtoolsMinimap2Ivar ConsensusBcftools Mpileup8 more steps
All 14 recorded steps
  1. MitoHiFi
  2. Minimap2
  3. SAMtools
  4. Minimap2
  5. Ivar Consensus
  6. Bcftools Mpileup
  7. Bcftools Call
  8. Bcftools Consensus
  9. SAMtools
  10. Bcftools Call
  11. Bcftools Mpileup
  12. Bcftools Call
  13. Data retrieval
  14. Bcftools Consensus

Score explanation

Consensus over-polishing Additional read-mapping and variant-consensus passes changed the assembled sequence away from the benchmark consensus. The output remains a valid complete contig, but the extra polishing introduced more base-level disagreement. Evidence. Of 15,426 submitted and 15,419 target 31-mers, 15,280 matched; 146 were submitted-only and 139 were target-only. The file contained no ambiguous 31-mer windows.
Galaxy historyHugging Face trace