Back to results

CompBioBench item

covid-patient-q1

Question

covid.patients.q1.h5ad contains PBMC scRNA-seq data from 33 donors, who are a mix of healthy and COVID patients. However, the patient metadata is missing. Identify which donors are healthy and which have COVID based on the gene expression data. Provide a single string containing comma-separated donor IDs for the healthy patients (numerically sorted), e.g."1,2,5".

Inferred answer
1,5,11,12,17,21,24,27,29,32Uniquely inferred from submitted scores.
Domain
Single-cell
Question style
Metadata Recovery

An answer can be uniquely inferred for 54 of 100 questions from submitted scores. Green: correct · Red: incorrect. For questions without a uniquely determined answer: Blue: matches prediction · Orange: differs from prediction. These are inferred answer checks, not official per-question grades.

Galaxy

12/12 terminal item results

UDT = user-defined tool: custom code executed by Galaxy. Route badges summarize analysis tools, excluding uploads and automatic conversions. How usage is counted.

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

CorrectUDT-only
Answer1,5,11,12,17,21,24,27,29,32
Workflow
COVID H5AD donor diagnosticCOVID H5AD direct donor diagnosticCOVID H5AD normalized signaturesCOVID H5AD inventoryCOVID pseudobulk donor cluster2 more tools
Total tokens
1.6M
Output
21.8K
Galaxy historyAgent trace
Codex GPT-5.5

Replicate 2

CorrectUDT-only
Answer1,5,11,12,17,21,24,27,29,32
Workflow
COVID H5AD diagnosticCOVID H5AD diagnostic v2COVID H5AD h5py diagnosticCOVID H5AD signature analysisCOVID H5AD signature analysis v21 more tools
Total tokens
912.6K
Output
16.4K
Galaxy historyAgent trace
Codex GPT-5.5

Replicate 3

CorrectUDT + standard tools
Answer1,5,11,12,17,21,24,27,29,32
Workflow
COVID h5ad donor diagnosticCOVID h5ad donor diagnostic v2COVID h5ad donor diagnostic v3COVID h5ad donor diagnostic v4Anndata Inspect3 more tools
Total tokens
3.9M
Output
22.6K
Galaxy historyAgent trace

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

CorrectUDT-only
Answer1,5,11,12,17,21,24,27,29,32
Workflow
COVID donor structural and marker diagnosticAnnData
Total tokens
6.6M
Output
20.7K
Galaxy historyAgent trace
Codex GPT-5.6 Sol

Replicate 2

CorrectUDT + standard tools
Answer1,5,11,12,17,21,24,27,29,32
Workflow
Anndata InspectCOVID donor expression diagnosticCOVID donor cell-stratified validationCOVID donor final expression call
Total tokens
5.5M
Output
22.0K
Galaxy historyAgent trace
Codex GPT-5.6 Sol

Replicate 3

CorrectUDT + standard tools
Answer1,5,11,12,17,21,24,27,29,32
Workflow
Anndata InspectCOVID donor expression diagnosticSelect columnsCOVID donor classification validationExtract validated healthy donor IDs
Total tokens
3.8M
Output
15.8K
Galaxy historyAgent trace

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

RerunIncorrectUDT + standard tools
Answer1,2,4,5,6,7,8,9,10,11,12,15,16,17,19,20,21,22,23,24,25,27,28,29,32
Workflow
anndata inspect → anndata export → User-defined tool
Total tokens
23.1M
Output
58.6K
Galaxy historyAgent trace
Codex + DeepSeek-v4-pro-0813

Replicate 2

RerunIncorrectUDT + standard tools
Answer1,5,6,8,10,11,12,15,17,19,21,22,23,24,25,27,29,32
Workflow
anndata inspect → scanpy inspect → scanpy normalize → decoupler pseudobulk → datamash ops → sklearn numeric clustering → sklearn pca → User-defined tool
Total tokens
18.3M
Output
67.5K
Galaxy historyAgent trace
Codex + DeepSeek-v4-pro-0813

Replicate 3

IncorrectUDT + standard tools
Answer1,2,4,5,6,7,9,10,11,12,15,16,17,20,21,22,23,24,25,27,28,29,32
Workflow
Anndata InspectAnndata ExportCOVID donor metricsUDT smokeUDT smoke scanpy7 more tools
Total tokens
74.7M
Output
173.7K
Galaxy historyAgent trace

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

IncorrectUDT + standard tools
Answer1,2,4,5,6,7,8,9,10,11,12,15,16,17,19,20,21,22,23,24,25,27,28,29,32
Workflow
Anndata InspectFilter TabularH5AD Runtime ProbeCOVID Donor Expression ExplorationH5AD Metadata Probe4 more tools
Total tokens
13.1M
Output
48.3K
Galaxy historyAgent trace
Codex GPT-5.6 Luna

Replicate 2

CorrectUDT + standard tools
Answer1,5,11,12,17,21,24,27,29,32
Workflow
Anndata InspectCOVID donor H5AD runtime probeCOVID H5AD structure probeCOVID donor expression diagnosticCOVID donor expression validation1 more tools
Total tokens
3.9M
Output
32.5K
Galaxy historyAgent trace
Codex GPT-5.6 Luna

Replicate 3

IncorrectUDT + standard tools
Answer1,2,4,5,6,7,8,9,10,11,12,15,16,17,19,20,21,22,23,24,25,27,28,29,32
Workflow
COVID donor response diagnosticAnnData
Total tokens
7.3M
Output
49.8K
Galaxy historyAgent trace

Anycode

13/13 terminal item results

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

Correct
Answer1,5,11,12,17,21,24,27,29,32
Workflow
Local codePythonAnnDataScanpySciPy
Agent trace
Codex GPT-5.5

Replicate 2

Correct
Answer1,5,11,12,17,21,24,27,29,32
Workflow
Local codePythonAnnDataScanpySciPy
Agent trace
Codex GPT-5.5

Replicate 3

Correct
Answer1,5,11,12,17,21,24,27,29,32
Workflow
Local codePythonAnnDataScanpySciPy
Agent trace

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

RerunCorrect
Answer1,5,11,12,17,21,24,27,29,32
Workflow
Open-ended code
Total tokens
997.3K
Output
11.1K
Agent trace
Codex GPT-5.6 Sol

Replicate 2

Correct
Answer1,5,11,12,17,21,24,27,29,32
Workflow
Local codePythonAnnDataScanpySciPy
Agent trace
Codex GPT-5.6 Sol

Replicate 3

Correct
Answer1,5,11,12,17,21,24,27,29,32
Workflow
Local codePythonAnnDataScanpySciPy
Agent trace

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

Incorrect
Answer1,5,6,10,11,12,16,17,21,23,24,27,29,32
Workflow
Local codePythonAnnDataScanpySciPy1 more tools
Total tokens
57.5M
Output
231.1K
Agent trace
Codex + DeepSeek-v4-pro-0813

Replicate 2

RerunCorrect
Answer1,5,11,12,17,21,24,27,29,32
Workflow
Open-ended code
Total tokens
11.1M
Output
52.1K
Agent trace
Codex + DeepSeek-v4-pro-0813

Replicate 3

Correct
Answer1,5,11,12,17,21,24,27,29,32
Workflow
Local codePythonAnnDataScanpySciPy
Total tokens
21.4M
Output
90.3K
Agent trace

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

RerunCorrect
Answer1,5,11,12,17,21,24,27,29,32
Workflow
Open-ended code
Total tokens
1.2M
Output
19.5K
Agent trace
Codex GPT-5.6 Luna

Replicate 2

Correct
Answer1,5,11,12,17,21,24,27,29,32
Workflow
Local codePythonAnnDataScanpySciPy
Agent trace
Codex GPT-5.6 Luna

Replicate 3

Incorrect
Answer1,2,4,5,6,7,8,9,10,11,12,13,15,16,17,19,20,21,22,23,24,25,27,28,29,30,31,32
Workflow
Local codePythonAnnDataScanpySciPy
Agent trace

Codex GPT-6 Astra

1 replicate
Codex GPT-6 Astra

Replicate 1

Correct
Answer1,5,11,12,17,21,24,27,29,32
Workflow
Local codePythonAnnDataScanpySciPy
Total tokens
801.7K
Output
7.3K
Agent trace