Back to results

CompBioBench item

finding-geo-q1

Question

finding.geo.q1.h5ad contains a scRNA-seq dataset that is derived from a Supplementary file on GEO, and processed to filter out barcodes with low counts. Find the GSE ID of the corresponding record. Return the GSE ID at the SuperSeries level, e.g. GSE242424.

Inferred answer
GSE114176Uniquely inferred from submitted scores.
Domain
Single-cell
Question style
Metadata Recovery

An answer can be uniquely inferred for 54 of 100 questions from submitted scores. Green: correct · Red: incorrect. For questions without a uniquely determined answer: Blue: matches prediction · Orange: differs from prediction. These are inferred answer checks, not official per-question grades.

Galaxy

12/12 terminal item results

UDT = user-defined tool: custom code executed by Galaxy. Route badges summarize analysis tools, excluding uploads and automatic conversions. How usage is counted.

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

IncorrectUDT-only
AnswerGSE117542
Workflow
H5AD accession diagnosticH5AD index signatureH5AD GEO supplement comparisonH5AD GEO supplement comparison grepH5AD expression signature15 more tools
Total tokens
11.3M
Output
62.0K
Codex GPT-5.5

Replicate 2

RerunIncorrectUDT-only
AnswerGSE132730
Workflow
Inspect H5AD GEO metadata q1Inspect H5AD GEO metadata q1 v2Fingerprint H5AD GEO q1Marker H5AD GEO q1Compare GSE118614 barcodes q15 more tools
Total tokens
12.7M
Output
46.7K
Codex GPT-5.5

Replicate 3

RerunIncorrectUDT-only
AnswerGSE153164
Workflow
H5AD metadata h5dumpH5AD index samplesH5AD count fingerprintH5AD marker panelCompare GSE112846 supplement16 more tools
Total tokens
30.3M
Output
99.4K

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

CorrectUDT + standard tools
AnswerGSE114176
Workflow
Anndata InspectInspect H5AD fingerprintInspect H5AD fingerprint extendedCompare H5AD to GEO inventoryNeural marker fingerprint5 more tools
Total tokens
9.2M
Output
25.5K
Codex GPT-5.6 Sol

Replicate 2

CorrectUDT-only
AnswerGSE114176
Workflow
Inspect H5AD identity metadataFingerprint H5AD countsProfile H5AD tissue markersVerify exact GEO H5 derivationVerify GEO H5 by Ensembl IDs
Total tokens
3.1M
Output
14.7K
Codex GPT-5.6 Sol

Replicate 3

CorrectUDT + standard tools
AnswerGSE114176
Workflow
Anndata InspectSciPyH5AD source fingerprint q1H5AD tissue marker fingerprint q1Stage GSE130803 RAW archive17 more tools
Total tokens
38.6M
Output
73.6K

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

RerunCorrectUDT-only
AnswerGSE114176
Workflow
GEO h5ad raw H5 match
Total tokens
4.5M
Output
43.5K
Codex + DeepSeek-v4-pro-0813

Replicate 2

IncorrectStandard tools only
AnswerGSE148115
Workflow
Anndata InspectScanpy InspectAnndata ExportTp Sort Header ToolComp1
Total tokens
76.4M
Output
189.5K
Codex + DeepSeek-v4-pro-0813

Replicate 3

IncorrectUDT + standard tools
Answerna
Workflow
Anndata InspectAnndata ExportGalaxy UDTH5AD metadata dumpx10 more tools

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

CorrectUDT + standard tools
AnswerGSE114176
Workflow
Anndata InspectScanpy InspectH5AD Matrix SignatureH5AD Layout DiagnosticH5AD Layout Diagnostic Binary62 more tools
Total tokens
67.1M
Output
152.7K
Codex GPT-5.6 Luna

Replicate 2

CorrectUDT + standard tools
AnswerGSE114176
Workflow
Anndata InspectAnndata ExportGalaxy Matrix FingerprintGalaxy Matrix Fingerprint HDF5Upload134 more tools
Total tokens
121.8M
Output
369.9K
Codex GPT-5.6 Luna

Replicate 3

CorrectUDT + standard tools
AnswerGSE114176
Workflow
Anndata InspectAnndata ExportH5AD Runtime ProbeProfile H5AD MatrixCompare H5AD to Cell Ranger H528 more tools
Total tokens
101.2M
Output
290.1K

Anycode

13/13 terminal item results

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

Correct
AnswerGSE114176
Workflow
Local codePython
Codex GPT-5.5

Replicate 2

Correct
AnswerGSE114176
Workflow
Local codePythonSciPy
Codex GPT-5.5

Replicate 3

Incorrect
AnswerGSE93421
Workflow
Local codePythonSciPyR

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

Correct
AnswerGSE114176
Workflow
Local codePythonSciPy
Codex GPT-5.6 Sol

Replicate 2

Correct
AnswerGSE114176
Workflow
Local codePythonAnnDataScanpySciPy
Codex GPT-5.6 Sol

Replicate 3

Correct
AnswerGSE114176
Workflow
Local codePythonSciPy

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

Correct
AnswerGSE114176
Workflow
Local codePythonAnnDataScanpySciPy
Total tokens
868.7K
Output
11.5K
Codex + DeepSeek-v4-pro-0813

Replicate 2

Incorrect
AnswerGSE102934
Workflow
Local codePythonScanpyAnnDataSciPy1 more tools
Total tokens
3.0M
Output
28.9K
Codex + DeepSeek-v4-pro-0813

Replicate 3

Correct
AnswerGSE114176
Workflow
Local codePython
Total tokens
15.7M
Output
46.5K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

Correct
AnswerGSE114176
Workflow
Local codePythonAnnDataScanpy
Codex GPT-5.6 Luna

Replicate 2

RerunCorrect
AnswerGSE114176
Workflow
Open-ended code
Total tokens
8.4M
Output
38.3K
Codex GPT-5.6 Luna

Replicate 3

Correct
AnswerGSE114176
Workflow
Local codePythonSciPyAnnDataR

Codex GPT-6 Astra

1 replicate
Codex GPT-6 Astra

Replicate 1

Correct
AnswerGSE114176
Workflow
Local codePythonSciPy
Total tokens
535.3K
Output
4.3K