IWC benchmark item

wf_011_bioproject_metadata_sequence_retrieval

Metadata and Sequences from BioProject IDs

BioProject Metadata and Sequence Retrieval

Open source IWC workflowGround-truth Galaxy history

Reference workflow

10 tool steps across 8 unique tools, including tools inside nested subworkflows. A plus sign denotes parallel branches.

Workflow
Separate BioProject IDsRelabel the separated BioProject IDsAdd BioProject IDs as parametersMetadata From BioProject IDsGet run_acccession per projectGenerate a single listParallel Accession Download

Complete query

The full task shown to the agent, excluding execution-condition instructions.

Retrieve the run-level metadata and biological sequence reads for every currently public SRA run associated with the BioProject accessions listed in the input file. Exclude technical reads, preserve each run's `SINGLE` or `PAIRED` layout, and do not alter the biological read sequences or quality strings after retrieval.
Under `final_answer/metadata/`, write exactly one table for each input BioProject at `final_answer/metadata/<ACCESSION>.tsv`. Each table must contain exactly one row for every run of that BioProject, and every row's `bioproject` value must equal `<ACCESSION>`. Each table must contain at least the following columns: `bioproject`, `run_accession`, `study_accession`, `experiment_accession`, `sample_accession`, `biosample`, `organism_taxid`, `organism_name`, `library_strategy`, `library_source`, `library_selection`, `library_layout`, `instrument_model`, `run_total_spots`, and `run_total_bases`. Additional columns are allowed.
Write `final_answer/run_manifest.tsv` with at least the columns `bioproject`, `run_accession`, `library_layout`, `forward_path`, and `reverse_path`. Column order does not matter and additional columns are allowed. Include every run exactly once, and describe the same run set as the metadata tables.
For paired-end runs, set `library_layout` to `PAIRED` and provide synchronized forward and reverse FASTQ paths. For single-end runs, set `library_layout` to `SINGLE`, provide the FASTQ path in `forward_path`, and leave `reverse_path` empty. All paths must be relative to `final_answer/`.
Store all retrieved FASTQ files under `final_answer/reads/` and reference them from the manifest.

acc evaluator

acc compares the final output files with the reference outputs.

Metric
retrieval_composite
Definition

Geometric mean of three independently reported components.
`project_run_layout_f1` is an F1 over (bioproject, run_accession, library_layout) triples, so filing a run under the wrong project costs the same as missing it.
`metadata_f1` is an F1 over (run_accession, field, normalized_value) triples restricted to the fifteen stable fields listed below and stated in the query.
`exact_run_content_f1` is an F1 over runs, not over reads. Each expected run is one binary unit: its whole (sequence, quality) multiset must match the stored fingerprint, and there is no partial credit inside a run. The name says so deliberately - an earlier draft described this as read-level F1, which it is not. A run only half retrieved is a failed retrieval, not a differently shaped scientific answer, and a truncated run therefore costs exactly what a wholly wrong one does.

Formula
Final scoreS = (F1_layout × F1_metadata × F1_content)^(1/3)

Three retrieval components are combined by geometric mean.

LayoutF1_layout = F1(|L_C ∩ L_R|, |L_C|, |L_R|)

L is the set of (BioProject, run accession, library layout) triples.

MetadataF1_metadata = F1(|T_C ∩ T_R|, |T_C|, |T_R|)

T is the set of normalized (run accession, scored field, value) triples over the 15 declared fields.

Read contentF1_content = F1(N_exact, N_candidate_runs, N_reference_runs)

A run matches only when its complete sequence-and-quality multiset fingerprint matches exactly; there is no within-run partial credit.

F1F1(M,C,R) = 2M/(C+R)

Equivalent to the harmonic mean of precision M/C and recall M/R.

Step-score requirements

A step passes when the trace directly shows that it ran. Parameter correctness and output quality are measured separately by acc.

  1. Step 1Read the BioProject accessions and retrieve run-level metadata for every input project.
    • Every input BioProject accession was read and queried against a public run-level metadata source.
    • Retrieved metadata covers every public run selected for each project with required fields.
  2. Step 2Build the complete, deduplicated set of public run accessions while preserving project and layout information.
    • The selected run-accession set is complete for the retrieved public metadata snapshot and contains no duplicates.
    • Every selected run retains its BioProject and SINGLE/PAIRED layout association.
  3. Step 3Retrieve the biological reads for every run and exclude technical reads.
    • Biological FASTQ reads were successfully retrieved for every selected public run.
    • Technical reads were excluded by the retrieval mode or an explicit technical-read filter.
    • Retrieved FASTQ files are readable and consistent with run-level spots/bases or checksums when available.
  4. Step 4Separate paired-end and single-end outputs and keep paired mates synchronized.
    • SINGLE runs have one forward file and no reverse file; PAIRED runs have both mate files.
    • Every paired FASTQ has matching record counts and synchronized mate identifiers.
  5. Step 5Produce one metadata table per BioProject and a manifest linking each project, run, and layout to the correct FASTQ paths, with metadata and manifest describing the same runs.
    • Exactly one required metadata table exists per input BioProject with the required schema.
    • The manifest paths resolve to the correct SINGLE or PAIRED FASTQ files for every run.
    • Metadata and manifest contain exactly the same project-run set and pass final-answer checks.

Model results and score explanations

All 24 results reached acc 1.000, so no score explanation is needed. Each replicate shows its recorded workflow, acc, step score, and Hugging Face trace. Galaxy histories can be imported by link but are not published in Galaxy's public history listing.

GPT-5.5

Open-ended code

R1acc · step score 1.000
1.000
Recorded workflow
Inspect inputsRetrieve runinfoBuild metadata and manifest skeletonRetrieve all reads successful chainValidate final deliverablesCrosscheck run counts4 more steps
All 10 recorded steps
  1. Inspect inputs
  2. Retrieve runinfo
  3. Build metadata and manifest skeleton
  4. Retrieve all reads successful chain
  5. Validate final deliverables
  6. Crosscheck run counts
  7. Build metadata and manifest skeleton
  8. Validate final deliverables
  9. Manifest line ending shell validation after correction — Table transformation
  10. Validate final deliverables
Hugging Face trace
R2acc · step score 1.000
1.000
Recorded workflow
Initialize workspaceRetrieve ncbi runinfoRetrieve ena filereportCreate retrieval scriptExecute retrieval scriptCreate validation script3 more steps
All 9 recorded steps
  1. Initialize workspace
  2. Retrieve ncbi runinfo
  3. Retrieve ena filereport
  4. Create retrieval script
  5. Execute retrieval script
  6. Create validation script
  7. Fix validation script
  8. Validate deliverables
  9. Final inventory check — Table transformation
Hugging Face trace
R3acc · step score 1.000
1.000
Recorded workflow
Initial inspectionRuninfo sra db be endpointEntrez sra search fetchMetadata fetch and runset validationMetadata tables and download planMetadata fetch and runset validation5 more steps
All 11 recorded steps
  1. Initial inspection
  2. Runinfo sra db be endpoint
  3. Entrez sra search fetch
  4. Metadata fetch and runset validation
  5. Metadata tables and download plan
  6. Metadata fetch and runset validation
  7. Metadata tables and download plan
  8. Fastq download
  9. Final deliverable validation
  10. Cleanup artifact
  11. Independent final check — Table transformation
Hugging Face trace

Galaxy-API code

R1acc · step score 1.000
1.000
Recorded workflow
Data retrievalpysradb metadata ×6Data retrievalfasterq-dump
All 9 recorded steps
  1. Data retrieval
  2. pysradb metadata
  3. pysradb metadata
  4. pysradb metadata
  5. pysradb metadata
  6. pysradb metadata
  7. pysradb metadata
  8. Data retrieval
  9. fasterq-dump
Galaxy historyHugging Face trace
R2acc · step score 1.000
1.000
Recorded workflow
pysradb metadata ×2Data retrievalfasterq-dump
All 4 recorded steps
  1. pysradb metadata
  2. pysradb metadata
  3. Data retrieval
  4. fasterq-dump
Galaxy historyHugging Face trace
R3acc · step score 1.000
1.000
Recorded workflow
pysradb metadata ×6Data retrievalfastq-dump
All 8 recorded steps
  1. pysradb metadata
  2. pysradb metadata
  3. pysradb metadata
  4. pysradb metadata
  5. pysradb metadata
  6. pysradb metadata
  7. Data retrieval
  8. fastq-dump
Galaxy historyHugging Face trace

GPT-5.6 Sol

Open-ended code

R1acc · step score 1.000
1.000
Recorded workflow
Input and tool inspectionMetadata inventory ×2Fastq retrieval and validationMetadata inventoryFastq retrieval and validationDeliverable constructionFinal deliverable validation
All 8 recorded steps
  1. Input and tool inspection
  2. Metadata inventory
  3. Metadata inventory
  4. Fastq retrieval and validation
  5. Metadata inventory
  6. Fastq retrieval and validation
  7. Deliverable construction
  8. Final deliverable validation
Hugging Face trace
R2acc · step score 1.000
1.000
Recorded workflow
Input and environment inspectionPreflight empty scaffoldRetrieve public run inventoriesBuild and crosscheck metadataDownload biological fastqValidate fastq integrity and content3 more steps
All 9 recorded steps
  1. Input and environment inspection
  2. Preflight empty scaffold
  3. Retrieve public run inventories
  4. Build and crosscheck metadata
  5. Download biological fastq
  6. Validate fastq integrity and content
  7. Validate deliverable consistency
  8. Publish final deliverables
  9. Postpublication read only audit — Table transformation
Hugging Face trace
R3acc · step score 1.000
1.000
Recorded workflow
Retrieve run-level metadata for each BioProjectBuild the deduplicated run-accession setRetrieve biological reads for every runSeparate layouts, exclude technical reads, and preserve mate pairingWrite per-project metadata tables and the FASTQ manifest
All 5 recorded steps
  1. Retrieve run-level metadata for each BioProject
  2. Build the deduplicated run-accession set
  3. Retrieve biological reads for every run
  4. Separate layouts, exclude technical reads, and preserve mate pairing
  5. Write per-project metadata tables and the FASTQ manifest
Hugging Face trace

Galaxy-API code

R1acc · step score 1.000
1.000
Recorded workflow
pysradb metadata ×4Data retrievalfasterq-dump
All 6 recorded steps
  1. pysradb metadata
  2. pysradb metadata
  3. pysradb metadata
  4. pysradb metadata
  5. Data retrieval
  6. fasterq-dump
Galaxy historyHugging Face trace
R2acc · step score 1.000
1.000
Recorded workflow
pysradb metadata ×4Data retrievalfasterq-dump
All 6 recorded steps
  1. pysradb metadata
  2. pysradb metadata
  3. pysradb metadata
  4. pysradb metadata
  5. Data retrieval
  6. fasterq-dump
Galaxy historyHugging Face trace
R3acc · step score 1.000
1.000
Recorded workflow
pysradb metadata ×2fasterq-dump
All 3 recorded steps
  1. pysradb metadata
  2. pysradb metadata
  3. fasterq-dump
Galaxy historyHugging Face trace

GPT-5.6 Luna

Open-ended code

R1acc · step score 1.000
1.000
Recorded workflow
Public run discoveryEna fastq retrieval and output generationIndependent deliverable validationTechnical read and layout validation
All 4 recorded steps
  1. Public run discovery
  2. Ena fastq retrieval and output generation
  3. Independent deliverable validation
  4. Technical read and layout validation
Hugging Face trace
R2acc · step score 1.000
1.000
Recorded workflow
Retrieve run-level metadata for each BioProjectBuild the deduplicated run-accession setRetrieve biological reads for every runSeparate layouts, exclude technical reads, and preserve mate pairingWrite per-project metadata tables and the FASTQ manifest
All 5 recorded steps
  1. Retrieve run-level metadata for each BioProject
  2. Build the deduplicated run-accession set
  3. Retrieve biological reads for every run
  4. Separate layouts, exclude technical reads, and preserve mate pairing
  5. Write per-project metadata tables and the FASTQ manifest
Hugging Face trace
R3acc · step score 1.000
1.000
Recorded workflow
Retrieve run-level metadata for each BioProjectBuild the deduplicated run-accession setRetrieve biological reads for every runSeparate layouts, exclude technical reads, and preserve mate pairingWrite per-project metadata tables and the FASTQ manifest
All 5 recorded steps
  1. Retrieve run-level metadata for each BioProject
  2. Build the deduplicated run-accession set
  3. Retrieve biological reads for every run
  4. Separate layouts, exclude technical reads, and preserve mate pairing
  5. Write per-project metadata tables and the FASTQ manifest
Hugging Face trace

Galaxy-API code

R1acc · step score 1.000
1.000
Recorded workflow
Data retrievalpysradb metadata ×4Data retrieval ×2fasterq-dump ×2
All 9 recorded steps
  1. Data retrieval
  2. pysradb metadata
  3. pysradb metadata
  4. pysradb metadata
  5. pysradb metadata
  6. Data retrieval
  7. Data retrieval
  8. fasterq-dump
  9. fasterq-dump
Galaxy historyHugging Face trace
R2acc · step score 1.000
1.000
Recorded workflow
pysradb metadata ×2Data retrieval ×2fasterq-dump ×2
All 6 recorded steps
  1. pysradb metadata
  2. pysradb metadata
  3. Data retrieval
  4. Data retrieval
  5. fasterq-dump
  6. fasterq-dump
Galaxy historyHugging Face trace
R3acc · step score 1.000
1.000
Recorded workflow
pysradb metadata ×3Data retrievalfasterq-dump
All 5 recorded steps
  1. pysradb metadata
  2. pysradb metadata
  3. pysradb metadata
  4. Data retrieval
  5. fasterq-dump
Galaxy historyHugging Face trace

Codex + DeepSeek V4 Pro

Open-ended code

R1acc · step score 1.000
1.000
Recorded workflow
Query sra run idsFetch sra runinfo prjna1354216Fetch sra runinfo prjna1339024Fetch ena filereportsDownload biological fastqBuild outputsValidate outputs
All 7 recorded steps
  1. Query sra run ids
  2. Fetch sra runinfo prjna1354216
  3. Fetch sra runinfo prjna1339024
  4. Fetch ena filereports
  5. Download biological fastq
  6. Build outputs
  7. Validate outputs
Hugging Face trace
R2acc · step score 1.000
1.000
Recorded workflow
Sra metadata retrievalEna fastq metadata retrievalTechnical read filteringFastq download and verificationDeliverable generationDeliverable validation
All 6 recorded steps
  1. Sra metadata retrieval
  2. Ena fastq metadata retrieval
  3. Technical read filtering
  4. Fastq download and verification
  5. Deliverable generation
  6. Deliverable validation
Hugging Face trace
R3acc · step score 1.000
1.000
Recorded workflow
Fetch sra metadataDownload sra fastqsBuild deliverablesValidate deliverables
All 4 recorded steps
  1. Fetch sra metadata
  2. Download sra fastqs
  3. Build deliverables
  4. Validate deliverables
Hugging Face trace

Galaxy-API code

R1acc · step score 1.000
1.000
Recorded workflow
pysradb metadata ×3Fastq Dl
All 4 recorded steps
  1. pysradb metadata
  2. pysradb metadata
  3. pysradb metadata
  4. Fastq Dl
Galaxy historyHugging Face trace
R3acc · step score 1.000
1.000
Recorded workflow
pysradb metadata ×4Data retrievalfasterq-dump
All 6 recorded steps
  1. pysradb metadata
  2. pysradb metadata
  3. pysradb metadata
  4. pysradb metadata
  5. Data retrieval
  6. fasterq-dump
Galaxy historyHugging Face trace