Back to results

CompBioBench item

perturb-seq-align-q1

Question

perturb.seq.align.q1.ref.h5ad contains a scRNA-seq matrix from a small scale Perturb-seq experiment in a cell type. perturb.seq.align.q1.query.h5ad contains a scRNA-seq matrix from a Perturb-seq experiment using the same guides in a different cell type. However, the targets of the guides are missing. Can you find the guides corresponding to the following target genes: PABPC1, NUDT21, LEO1. Return the answer as a comma separate list in the format: "guide1,guide2,guide3".

Inferred answer
guide18,guide13,guide14Uniquely inferred from submitted scores.
Domain
Single-cell
Question style
Metadata Recovery

An answer can be uniquely inferred for 54 of 100 questions from submitted scores. Green: correct · Red: incorrect. For questions without a uniquely determined answer: Blue: matches prediction · Orange: differs from prediction. These are inferred answer checks, not official per-question grades.

Galaxy

12/12 terminal item results

UDT = user-defined tool: custom code executed by Galaxy. Route badges summarize analysis tools, excluding uploads and automatic conversions. How usage is counted.

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

CorrectUDT + standard tools
Answerguide18,guide13,guide14
Workflow
Anndata ExportH5AD metadata diagnosticPerturb-seq guide alignmentH5AD matrix scale diagnosticH5AD structure diagnostic
Codex GPT-5.5

Replicate 2

RerunCorrectUDT-only
Answerguide18,guide13,guide14
Workflow
Perturb-seq guide target lookupPerturb-seq guide target lookup metadataPerturb-seq guide lookup diagnosticsPerturb-seq signature guide alignmentPerturb-seq signature guide alignment report1 more tools
Total tokens
920.0K
Output
17.3K
Codex GPT-5.5

Replicate 3

RerunIncorrectUDT-only
Answerguide16,guide15,guide14
Workflow
Perturb guide target metadata map q1Perturb guide target metadata map q1 v2Perturb signature guide alignment q1Perturb signature guide alignment q1 v2Perturb global guide alignment q13 more tools
Total tokens
1.7M
Output
25.8K

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

CorrectUDT + standard tools
Answerguide18,guide13,guide14
Workflow
Anndata InspectPerturb-seq alignment diagnosticPerturb-seq signature alignment
Total tokens
2.4M
Output
17.5K
Codex GPT-5.6 Sol

Replicate 2

CorrectUDT + standard tools
Answerguide18,guide13,guide14
Workflow
Anndata InspectSciPyAnnDataPerturb-seq target knockdown diagnosticGrep5 more tools
Total tokens
7.7M
Output
28.9K
Codex GPT-5.6 Sol

Replicate 3

CorrectUDT + standard tools
Answerguide18,guide13,guide14
Workflow
Anndata InspectPerturb-seq cross-cell-type alignment diagnosticPerturb-seq validated guide assignment
Total tokens
3.5M
Output
15.1K

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

CorrectUDT + standard tools
Answerguide18,guide13,guide14
Workflow
Anndata InspectAnndata ExportProbe Python 3.11AnnDataProbe Path3 more tools
Total tokens
6.3M
Output
49.0K
Codex + DeepSeek-v4-pro-0813

Replicate 2

CorrectUDT + standard tools
Answerguide18,guide13,guide14
Workflow
Anndata InspectSciPyPerturb Probe MinimalImport ProbePerturb Seq Guide Alignment1 more tools
Total tokens
11.4M
Output
54.2K
Codex + DeepSeek-v4-pro-0813

Replicate 3

CorrectUDT + standard tools
Answerguide18,guide13,guide14
Workflow
Anndata InspectAnndata ExportGroup and aggregateSciPyAnnData1 more tools
Total tokens
7.5M
Output
60.0K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

CorrectUDT + standard tools
Answerguide18,guide13,guide14
Workflow
Anndata InspectFilter TabularGroup and aggregateGrepScanpy Find Markers3 more tools
Total tokens
10.2M
Output
48.0K
Codex GPT-5.6 Luna

Replicate 2

CorrectUDT + standard tools
Answerguide18,guide13,guide14
Workflow
Anndata InspectFilter TabularFilter rowsGroup and aggregateSelect columns11 more tools
Total tokens
12.5M
Output
77.1K
Codex GPT-5.6 Luna

Replicate 3

CorrectUDT + standard tools
Answerguide18,guide13,guide14
Workflow
Anndata InspectGuide match runtime probeGuide match H5AD runtime probeGuide match dependency probeH5AD structure probe7 more tools

Anycode

13/13 terminal item results

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

Correct
Answerguide18,guide13,guide14
Workflow
Local codePythonAnnDataSciPy
Codex GPT-5.5

Replicate 2

Correct
Answerguide18,guide13,guide14
Workflow
Local codePythonAnnDataScanpySciPy
Codex GPT-5.5

Replicate 3

Correct
Answerguide18,guide13,guide14
Workflow
Local codePythonAnnDataScanpySciPy

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

Correct
Answerguide18,guide13,guide14
Workflow
Local codePythonAnnDataScanpySciPy
Codex GPT-5.6 Sol

Replicate 2

Correct
Answerguide18,guide13,guide14
Workflow
Local codePythonAnnDataSciPy
Codex GPT-5.6 Sol

Replicate 3

Correct
Answerguide18,guide13,guide14
Workflow
Local codePythonAnnDataScanpyR1 more tools

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

RerunCorrect
Answerguide18,guide13,guide14
Workflow
Open-ended code
Total tokens
9.3M
Output
58.2K
Codex + DeepSeek-v4-pro-0813

Replicate 2

Correct
Answerguide18,guide13,guide14
Workflow
Local codePythonAnnDataScanpySciPy
Total tokens
5.6M
Output
58.9K
Codex + DeepSeek-v4-pro-0813

Replicate 3

RerunCorrect
Answerguide18,guide13,guide14
Workflow
Open-ended code
Total tokens
2.2M
Output
44.6K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

Correct
Answerguide18,guide13,guide14
Workflow
Local codePythonAnnDataScanpySciPy
Codex GPT-5.6 Luna

Replicate 2

Correct
Answerguide18,guide13,guide14
Workflow
Local codePythonAnnDataScanpySciPy
Codex GPT-5.6 Luna

Replicate 3

Correct
Answerguide18,guide13,guide14
Workflow
Local codePythonAnnDataScanpySciPy

Codex GPT-6 Astra

1 replicate
Codex GPT-6 Astra

Replicate 1

Correct
Answerguide18,guide13,guide14
Workflow
Local codePythonAnnDataScanpySciPy
Total tokens
366.9K
Output
4.1K