Back to results

CompBioBench item

encode-atac-pipeline-q1

Question

encode.atac.pipeline.q1.R1.fq.gz and encode.atac.pipeline.q1.R2.fq.gz are paired-end ATAC-seq reads from a human sample. Process them through the ENCODE ATAC-seq pipeline v2.2.3 (https://github.com/ENCODE-DCC/atac-seq-pipeline/tree/v2.2.3). Keep the input json similar to https://github.com/ENCODE-DCC/atac-seq-pipeline/blob/v2.2.3/example_input_json/ENCSR356KRQ_subsampled.json, but remove the atac.enable_xcor line. After the pipeline completes, report the reproducibility IDR N_opt number from the qc.json file.

Inferred answer
29556Uniquely inferred from submitted scores.
Domain
Epigenomics
Question style
Tooling

An answer can be uniquely inferred for 54 of 100 questions from submitted scores. Green: correct · Red: incorrect. For questions without a uniquely determined answer: Blue: matches prediction · Orange: differs from prediction. These are inferred answer checks, not official per-question grades.

Galaxy

12/12 terminal item results

UDT = user-defined tool: custom code executed by Galaxy. Route badges summarize analysis tools, excluding uploads and automatic conversions. How usage is counted.

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

IncorrectUDT-only
Answer13636
Workflow
ENCODE ATAC 2.2.3 runtime probeENCODE ATAC 2.2.3 runtime detailsENCODE ATAC 2.2.3 Caper probeENCODE ATAC 2.2.3 Caper probe captureENCODE ATAC 2.2.3 Caper probe fixed10 more tools
Total tokens
4.8M
Output
28.6K
Codex GPT-5.5

Replicate 2

RerunIncorrectUDT-only
Answer13636
Workflow
ENCODE ATAC 2.2.3 runtime probeENCODE ATAC 2.2.3 repository probeENCODE ATAC v2.2.3 q1 full pipelineENCODE ATAC v2.2.3 q1 staged referenceENCODE ATAC v2.2.3 q1 staged reference corrected TSV2 more tools
Total tokens
6.5M
Output
30.5K
Codex GPT-5.5

Replicate 3

RerunCorrectUDT-only
Answer29556
Workflow
ENCODE ATAC 2.2.3 container probeENCODE ATAC 2.2.3 layout probeENCODE ATAC-seq pipeline v2.2.3 full run
Total tokens
5.3M
Output
28.0K

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

CorrectUDT-only
Answer29556
Workflow
Bowtie2ENCODE ATAC 2.2.3 Caper probeENCODE ATAC-seq pipeline v2.2.3
Total tokens
38.4M
Output
32.9K
Codex GPT-5.6 Sol

Replicate 2

IncorrectUDT-only
Answer13636
Workflow
ENCODE ATAC-seq v2.2.3 q1ENCODE ATAC-seq v2.2.3 q1 corrected validatorENCODE ATAC-seq v2.2.3 q1 Java 11ENCODE ATAC-seq v2.2.3 q1 Python 3.6 compatible
Total tokens
14.9M
Output
28.5K
Codex GPT-5.6 Sol

Replicate 3

IncorrectUDT-only
Answer13636
Workflow
Inspect ENCODE ATAC 2.2.3 runtimeBowtie2SamtoolsBedtoolsProbe Caper 2.3.2 in ENCODE ATAC 2.2.3 v53 more tools
Total tokens
37.5M
Output
33.9K

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

CorrectUDT-only
Answer29556
Workflow
ATAC image probeATAC image lsATAC install probeATAC net find probeATAC caper install probe19 more tools
Total tokens
26.9M
Output
75.1K
Codex + DeepSeek-v4-pro-0813

Replicate 2

IncorrectUDT-only
Answer13636
Workflow
Caper ProbeCaper Probe v2Caper Probe v3Env ProbeCaper User Probe5 more tools
Total tokens
16.8M
Output
62.4K
Codex + DeepSeek-v4-pro-0813

Replicate 3

IncorrectUDT-only
Answer13653
Workflow
Bowtie2SamtoolsBedtoolsENCODE ATAC Image Probe 2ENCODE ATAC Image Probe 336 more tools

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

RerunIncorrectUDT-only
Answer13636
Workflow
User-defined tool
Total tokens
48.7M
Output
91.2K
Codex GPT-5.6 Luna

Replicate 2

IncorrectUDT + standard tools
Answer13636
Workflow
ENCODE ATAC runtime probeCat1ENCODE ATAC Caper 2.2.3 ProbeENCODE ATAC Caper Import ProbeENCODE ATAC Pinned Caper Probe11 more tools
Total tokens
26.2M
Output
117.2K
Codex GPT-5.6 Luna

Replicate 3

IncorrectUDT-only
Answer13636
Workflow
ENCODE ATAC 2.2.3 runtime probeENCODE ATAC 2.2.3 runtime probe v2ENCODE ATAC 2.2.3 runtime probe v3ENCODE ATAC 2.2.3 runtime probe v4ENCODE ATAC 2.2.3 runtime probe v518 more tools
Total tokens
36.4M
Output
106.7K

Anycode

13/13 terminal item results

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

Incorrect
Answer28965
Workflow
Local codePythonBowtie2SamtoolsBedtools2 more tools
Codex GPT-5.5

Replicate 2

Incorrect
Answer28875
Workflow
Local codePythonBowtie2SamtoolsBedtools2 more tools
Codex GPT-5.5

Replicate 3

Incorrect
Answer28875
Workflow
Local codePythonBowtie2SamtoolsBedtools1 more tools

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

Incorrect
Answer28869
Workflow
Local codePythonBowtie2SamtoolsBedtools2 more tools
Codex GPT-5.6 Sol

Replicate 2

Incorrect
Answer28875
Workflow
Local codePythonSamtools
Codex GPT-5.6 Sol

Replicate 3

Incorrect
Answer108
Workflow
Local codePythonBowtie2SamtoolsBedtools2 more tools

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

Incorrect
Answer28927
Workflow
Local codePythonBWA-MEMBowtie2Samtools3 more tools
Total tokens
35.3M
Output
70.2K
Codex + DeepSeek-v4-pro-0813

Replicate 2

Incorrect
Answer28927
Workflow
Local codePythonSamtoolsBowtie2BWA-MEM3 more tools
Total tokens
25.5M
Output
63.7K
Codex + DeepSeek-v4-pro-0813

Replicate 3

Incorrect
Answer28915
Workflow
Local codePythonBowtie2BWA-MEMSamtools4 more tools
Total tokens
26.2M
Output
72.6K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

Incorrect
Answer28869
Workflow
Local codeSamtoolsBowtie2BWA-MEMR4 more tools
Codex GPT-5.6 Luna

Replicate 2

Incorrect
Answer28927
Workflow
Local codePythonSamtoolsBowtie2BWA-MEM4 more tools
Codex GPT-5.6 Luna

Replicate 3

Incorrect
Answer28927
Workflow
Local codePythonpysamSciPyBowtie24 more tools

Codex GPT-6 Astra

1 replicate
Codex GPT-6 Astra

Replicate 1

Incorrect
Answer29608
Workflow
Local codePythonBedtoolsSamtoolsBowtie23 more tools
Total tokens
14.7M
Output
30.0K