Back to results

CompBioBench item

pooled-infer-donors-q1

Question

pooled.infer.donors.q1.bam is an aligned scRNA-seq BAM from a pooled human sample. The file includes cell barcode tags. Determine the number of distinct donors contributing cells to the pool.

Inferred answer
4Uniquely inferred from submitted scores.
Domain
Single-cell
Question style
Metadata Recovery

An answer can be uniquely inferred for 54 of 100 questions from submitted scores. Green: correct · Red: incorrect. For questions without a uniquely determined answer: Blue: matches prediction · Orange: differs from prediction. These are inferred answer checks, not official per-question grades.

Galaxy

12/12 terminal item results

UDT = user-defined tool: custom code executed by Galaxy. Route badges summarize analysis tools, excluding uploads and automatic conversions. How usage is counted.

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

CorrectUDT-only
Answer4
Workflow
Infer donor count with cellsnp-lite and vireoInfer donor count with cellsnp-lite and vireo v2Infer donor count with cellsnp-lite and vireo v3
Galaxy historyAgent trace
Codex GPT-5.5

Replicate 2

CorrectUDT-only
Answer4
Workflow
BAM diagnostic for donor inferenceSelect cell barcodes for donor inferencePilot donor inference with cellsnp-lite and vireoCheck BAM and SNP panel reference conventionsPilot donor inference with hg19 SNP panel3 more tools
Total tokens
2.4M
Output
19.4K
Galaxy historyAgent trace
Codex GPT-5.5

Replicate 3

CorrectUDT-only
Answer4
Workflow
Bounded donor count pilotBounded donor count pilot v2Bounded donor count pilot v3UDT render probeUDT render probe v25 more tools
Total tokens
3.4M
Output
33.1K
Galaxy historyAgent trace

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

CorrectUDT + standard tools
Answer4
Workflow
Samtools ViewGalaxy UDTdonor_inferDonor inferenceInspect single-cell BAM3 more tools
Total tokens
13.1M
Output
28.5K
Galaxy historyAgent trace
Codex GPT-5.6 Sol

Replicate 2

CorrectUDT-only
Answer4
Workflow
Diagnose pooled scRNA BAM for donor inferenceUpload1Pooled scRNA barcode QC for donor inferenceInfer pooled scRNA donor count with cellSNP and Vireo model grid
Total tokens
6.5M
Output
19.5K
Galaxy historyAgent trace
Codex GPT-5.6 Sol

Replicate 3

CorrectUDT + standard tools
Answer4
Workflow
Inspect scRNA BAM tags and barcodesInspect scRNA BAM tags and barcodes v2UDT render probeSamtools ViewSamtools Mpileup
Total tokens
35.3M
Output
64.6K
Galaxy historyAgent trace

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

CorrectStandard tools only
Answer4
Workflow
No Galaxy tool job recorded
Total tokens
1.4M
Output
11.1K
Galaxy historyAgent trace
Codex + DeepSeek-v4-pro-0813

Replicate 2

CorrectUDT-only
Answer4
Workflow
SamtoolsRender probeResource probeTgz probeSlot probe7 more tools
Total tokens
4.6M
Output
32.1K
Galaxy historyAgent trace
Codex + DeepSeek-v4-pro-0813

Replicate 3

CorrectStandard tools only
Answer4
Workflow
Samtools ViewTp Head ToolSelect columnsTp Find And ReplaceBamtools Split Tag
Total tokens
4.6M
Output
29.3K
Galaxy historyAgent trace

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

CorrectUDT + standard tools
Answer4
Workflow
BAM tag diagnosticPooled scRNA donor deconvolutionSamtoolsDonor runtime probe
Total tokens
21.3M
Output
53.2K
Galaxy historyAgent trace
Codex GPT-5.6 Luna

Replicate 2

CorrectUDT + standard tools
Answer4
Workflow
Bam To SamTp Head ToolSamtools ViewSamtoolsDonor deconvolution runtime probe14 more tools
Total tokens
26.7M
Output
67.5K
Galaxy historyAgent trace
Codex GPT-5.6 Luna

Replicate 3

CorrectUDT + standard tools
Answer4
Workflow
Donor BAM runtime probeDonor UDT render probeUDT no-container probeUDT scalar probeGalaxyTool render probe13 more tools
Total tokens
26.4M
Output
87.5K
Galaxy historyAgent trace

Anycode

13/13 terminal item results

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

Correct
Answer4
Workflow
Local codeSamtoolsBCFtoolsPythonpysam1 more tools
Agent trace
Codex GPT-5.5

Replicate 2

Correct
Answer4
Workflow
Local codeSamtoolsPythonpysamSciPy1 more tools
Agent trace
Codex GPT-5.5

Replicate 3

Correct
Answer4
Workflow
Local codeBCFtoolsSamtoolsPythonpysam1 more tools
Agent trace

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

Correct
Answer4
Workflow
Local codeSamtoolsBCFtoolspysamPython1 more tools
Agent trace
Codex GPT-5.6 Sol

Replicate 2

Correct
Answer4
Workflow
Local codeSamtoolsBCFtoolsPython
Agent trace
Codex GPT-5.6 Sol

Replicate 3

Correct
Answer4
Workflow
Local codeSamtoolsPythonpysamBCFtools
Agent trace

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

Correct
Answer4
Workflow
Local codeSamtoolsPythonpysamBCFtools
Total tokens
2.0M
Output
22.4K
Agent trace
Codex + DeepSeek-v4-pro-0813

Replicate 2

Correct
Answer4
Workflow
Local codeSamtoolsPython
Total tokens
1.3M
Output
16.5K
Agent trace
Codex + DeepSeek-v4-pro-0813

Replicate 3

Correct
Answer4
Workflow
Local codeSamtoolsPythonpysam
Total tokens
568.7K
Output
10.0K
Agent trace

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

Correct
Answer4
Workflow
Local codeSamtoolsBCFtoolsMinimap2Python2 more tools
Agent trace
Codex GPT-5.6 Luna

Replicate 2

Correct
Answer4
Workflow
Local codeSamtoolspysamPythonBCFtools1 more tools
Agent trace
Codex GPT-5.6 Luna

Replicate 3

Correct
Answer4
Workflow
Local codeSamtoolsBCFtoolspysamPython2 more tools
Agent trace

Codex GPT-6 Astra

1 replicate
Codex GPT-6 Astra

Replicate 1

Correct
Answer4
Workflow
Local codeSamtoolsPythonBCFtoolspysam1 more tools
Total tokens
620.9K
Output
8.1K
Agent trace