IWC benchmark item

wf_008_amr_gene_detection

AMR Gene Detection

Antimicrobial Resistance Determinant Detection

Open source IWC workflowGround-truth Galaxy history

Reference workflow

7 tool steps across 7 unique tools, including tools inside nested subworkflows. A plus sign denotes parallel branches.

Workflow
Staramr AMR genes detection + AMRFinderPlus ARG and point mutation detection + ABRicate virulence genes detectionColumn Regex Find And Replace + ToolDistillator AMR DetectionMultiQC + ToolDistillator summarize AMR Detection

Complete query

The full task shown to the agent, excluding execution-condition instructions.

Identify the antimicrobial resistance determinants carried by the provided complete `Escherichia coli` genome. Report both acquired resistance genes and resistance-conferring chromosomal point mutations.
Identify acquired genes using ResFinder release `d1e607b` and point mutations using PointFinder release `694919f`, with `Escherichia coli` as the PointFinder organism. Require at least 90 percent nucleotide identity, at least 60 percent coverage for acquired-gene hits, and at least 95 percent coverage for point-mutation hits.
Write `final_answer/resistance_determinants.tsv` as a tab-separated table with a header and a first column named `determinant`, one determinant per row. Use the database names for acquired genes. Represent a point mutation as the gene followed by the substitution, for example `gyrA (S83L)`. Do not include plasmid replicons or multilocus sequence types.

acc evaluator

acc compares the final output files with the reference outputs.

Metric
resistance_determinant_f1
Definition

Precision/recall F1 over the set of resistance determinants. Names are normalised for formatting only - case, whitespace and the parentheses in forms like tet(B) - so that tetB and tet(B) are one determinant. Allele numbers are preserved, so blaCTX-M-15 and blaCTX-M-65 remain distinct determinants, because they are different enzymes with different substrate profiles rather than two spellings of one thing.

Formula
Final scoreS = F1(|D_C ∩ D_R|, |D_C|, |D_R|)

D_C and D_R are the normalized sets of resistance determinants.

Normalizationtet(B) ≡ tetB; case and whitespace ignored; allele numbers preserved

Formatting aliases collapse, but distinct alleles such as CTX-M-15 and CTX-M-65 remain distinct.

Step-score requirements

A step passes when the trace directly shows that it ran. Parameter correctness and output quality are measured separately by acc.

  1. Step 1Detect acquired resistance genes using the specified ResFinder release and thresholds.
    • A successful ResFinder execution used release d1e607b with identity at least 90% and coverage at least 60%.
    • Acquired resistance-gene hits from that execution entered the normalization chain.
  2. Step 2Detect chromosomal resistance mutations using the specified PointFinder release, organism setting, and thresholds.
    • A successful PointFinder execution used release 694919f, the E. coli organism setting, identity at least 95%, and coverage at least 60%.
    • Chromosomal resistance-mutation hits from that execution entered the normalization chain.
  3. Step 3Normalize and combine both result types into the final determinant list, remove duplicates, and exclude plasmid replicons and sequence types.
    • Acquired genes and chromosomal mutations were normalized into the required determinant representation and combined.
    • Duplicate determinants, plasmid replicons, and sequence types were excluded.
    • The required final determinant artifact passes schema and final-answer checks.

Model results and score explanations

All 24 results reached acc 1.000, so no score explanation is needed. Each replicate shows its recorded workflow, acc, step score, and Hugging Face trace. Galaxy histories can be imported by link but are not published in Galaxy's public history listing.

GPT-5.5

Open-ended code

R1acc · step score 1.000
1.000
Recorded workflow
Environment inspectionDatabase checkout bitbucketAmr detectionAmr detection refinedFinal validation
All 5 recorded steps
  1. Environment inspection
  2. Database checkout bitbucket
  3. Amr detection
  4. Amr detection refined
  5. Final validation
Hugging Face trace
R2acc · step score 1.000
1.000
Recorded workflow
Database checkoutOfficial resfinder pointfinder analysisFinal table extraction
All 3 recorded steps
  1. Database checkout
  2. Official resfinder pointfinder analysis
  3. Final table extraction
Hugging Face trace
R3acc · step score 1.000
1.000
Recorded workflow
Database fetchInput prepareResfinder pointfinder analysisFinal table buildResfinder tool fetch
All 5 recorded steps
  1. Database fetch
  2. Input prepare
  3. Resfinder pointfinder analysis
  4. Final table build
  5. Resfinder tool fetch
Hugging Face trace

Galaxy-API code

R3acc · step score 1.000
1.000
Recorded workflow
Data retrievalStarAMR ×3
All 4 recorded steps
  1. Data retrieval
  2. StarAMR
  3. StarAMR
  4. StarAMR
Galaxy historyHugging Face trace

GPT-5.6 Sol

Open-ended code

R1acc · step score 1.000
1.000
Recorded workflow
Detect acquired resistance genes with ResFinder/StarAMRDetect chromosomal resistance mutations with PointFinderNormalize, combine, de-duplicate, and filter determinants
All 3 recorded steps
  1. Detect acquired resistance genes with ResFinder/StarAMR
  2. Detect chromosomal resistance mutations with PointFinder
  3. Normalize, combine, de-duplicate, and filter determinants
Hugging Face trace
R2acc · step score 1.000
1.000
Recorded workflow
Detect acquired resistance genes with ResFinder/StarAMRDetect chromosomal resistance mutations with PointFinderNormalize, combine, de-duplicate, and filter determinants
All 3 recorded steps
  1. Detect acquired resistance genes with ResFinder/StarAMR
  2. Detect chromosomal resistance mutations with PointFinder
  3. Normalize, combine, de-duplicate, and filter determinants
Hugging Face trace
R3acc · step score 1.000
1.000
Recorded workflow
Detect acquired resistance genes with ResFinder/StarAMRDetect chromosomal resistance mutations with PointFinderNormalize, combine, de-duplicate, and filter determinants
All 3 recorded steps
  1. Detect acquired resistance genes with ResFinder/StarAMR
  2. Detect chromosomal resistance mutations with PointFinder
  3. Normalize, combine, de-duplicate, and filter determinants
Hugging Face trace

Galaxy-API code

R2acc · step score 1.000
1.000
Recorded workflow
Converter Gz To UncompressedStarAMR ×2
All 3 recorded steps
  1. Converter Gz To Uncompressed
  2. StarAMR
  3. StarAMR
Galaxy historyHugging Face trace
R3acc · step score 1.000
1.000
Recorded workflow
Data retrievalConverter Gz To UncompressedStarAMR ×2
All 4 recorded steps
  1. Data retrieval
  2. Converter Gz To Uncompressed
  3. StarAMR
  4. StarAMR
Galaxy historyHugging Face trace

GPT-5.6 Luna

Open-ended code

R1acc · step score 1.000
1.000
Recorded workflow
Detect acquired resistance genes with ResFinder/StarAMRDetect chromosomal resistance mutations with PointFinderDetect resistance determinants with AMRFinderPlusNormalize, combine, de-duplicate, and filter determinants
All 4 recorded steps
  1. Detect acquired resistance genes with ResFinder/StarAMR
  2. Detect chromosomal resistance mutations with PointFinder
  3. Detect resistance determinants with AMRFinderPlus
  4. Normalize, combine, de-duplicate, and filter determinants
Hugging Face trace
R2acc · step score 1.000
1.000
Recorded workflow
Detect acquired resistance genes with ResFinder/StarAMRDetect chromosomal resistance mutations with PointFinderDetect resistance determinants with AMRFinderPlusNormalize, combine, de-duplicate, and filter determinants
All 4 recorded steps
  1. Detect acquired resistance genes with ResFinder/StarAMR
  2. Detect chromosomal resistance mutations with PointFinder
  3. Detect resistance determinants with AMRFinderPlus
  4. Normalize, combine, de-duplicate, and filter determinants
Hugging Face trace
R3acc · step score 1.000
1.000
Recorded workflow
Detect acquired resistance genes with ResFinder/StarAMRDetect chromosomal resistance mutations with PointFinderDetect resistance determinants with AMRFinderPlusNormalize, combine, de-duplicate, and filter determinants
All 4 recorded steps
  1. Detect acquired resistance genes with ResFinder/StarAMR
  2. Detect chromosomal resistance mutations with PointFinder
  3. Detect resistance determinants with AMRFinderPlus
  4. Normalize, combine, de-duplicate, and filter determinants
Hugging Face trace

Galaxy-API code

Codex + DeepSeek V4 Pro

Open-ended code

R1acc · step score 1.000
1.000
Recorded workflow
Input preparationStaramr searchFinal table generationStaramr search no exclude validation
All 4 recorded steps
  1. Input preparation
  2. Staramr search
  3. Final table generation
  4. Staramr search no exclude validation
Hugging Face trace
R2acc · step score 1.000
1.000
Recorded workflow
Prepare databasesAmr detectionCompile final table
All 3 recorded steps
  1. Prepare databases
  2. Amr detection
  3. Compile final table
Hugging Face trace
R3acc · step score 1.000
1.000
Recorded workflow
Fetch resfinder dbFetch pointfinder dbDecompress inputRun resfinder pointfinderParse final results
All 5 recorded steps
  1. Fetch resfinder db
  2. Fetch pointfinder db
  3. Decompress input
  4. Run resfinder pointfinder
  5. Parse final results
Hugging Face trace

Galaxy-API code

R1acc · step score 1.000
1.000
Recorded workflow
Detect acquired resistance genes with ResFinder/StarAMRDetect chromosomal resistance mutations with PointFinderNormalize, combine, de-duplicate, and filter determinants
All 3 recorded steps
  1. Detect acquired resistance genes with ResFinder/StarAMR
  2. Detect chromosomal resistance mutations with PointFinder
  3. Normalize, combine, de-duplicate, and filter determinants
Galaxy historyHugging Face trace
R3acc · step score 1.000
1.000
Recorded workflow
Converter Gz To UncompressedStarAMR ×2
All 3 recorded steps
  1. Converter Gz To Uncompressed
  2. StarAMR
  3. StarAMR
Galaxy historyHugging Face trace