Back to results

CompBioBench item

cell-proportions-q1

Question

I have a counts matrix of single cell transcriptomic data, cell.proportions.q1.mtx.gz. The data contains measurements from 5350 cells, each with 1000 measured genes. That data was generated by measuring data from a co-culture of a few unique cell types. Can you tell me the proportions of those cells in descending order to the nearest 5 percent? Drop any cells with questionable QC. Expected output: 35;35;20;10 if there are 4 cell types.

Predicted answer
50;35;15Most likely answer for this question; not uniquely determined from submitted scores.
Domain
Single-cell
Question style
Synthetic/Augmented Data

An answer can be uniquely inferred for 54 of 100 questions from submitted scores. Green: correct · Red: incorrect. For questions without a uniquely determined answer: Blue: matches prediction · Orange: differs from prediction. These are inferred answer checks, not official per-question grades.

Galaxy

12/12 terminal item results

UDT = user-defined tool: custom code executed by Galaxy. Route badges summarize analysis tools, excluding uploads and automatic conversions. How usage is counted.

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

Matches predictionUDT-only
Answer50;35;15
Workflow
SC Cell Type ProportionsSC Cell Type Proportions v2SC Cell Type Proportions v3SC Cell Type Proportions v4
Total tokens
577.6K
Output
11.3K
Codex GPT-5.5

Replicate 2

Matches predictionUDT + standard tools
Answer50;35;15
Workflow
Single-cell proportions diagnosticSingle-cell proportions diagnostic v2UDT command field probeUDT shell probeUDT shell probe v21 more tools
Total tokens
6.4M
Output
33.2K
Codex GPT-5.5

Replicate 3

Matches predictionUDT + standard tools
Answer50;35;15
Workflow
Single-cell proportions diagnosticSingle-cell proportions diagnostic v2UDT JS probeUDT JS probe v2Anndata Import6 more tools
Total tokens
5.7M
Output
22.2K

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

Matches predictionStandard tools only
Answer50;35;15
Workflow
Anndata ImportAnndata InspectAnndata ManipulateScanpy InspectDatamash Ops5 more tools
Codex GPT-5.6 Sol

Replicate 2

Matches predictionUDT-only
Answer50;35;15
Workflow
Single-cell matrix QC diagnosticSingle-cell QC and cluster explorationFinal QC-filtered single-cell proportions
Total tokens
670.5K
Output
8.8K
Codex GPT-5.6 Sol

Replicate 3

Matches predictionUDT-only
Answer50;35;15
Workflow
Single-cell proportions diagnosticValidated single-cell type proportions
Total tokens
2.6M
Output
11.8K

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

RerunMatches predictionUDT + standard tools
Answer50;35;15
Workflow
User-defined tool → Summary Statistics1 → anndata import → scanpy filter cells
Total tokens
5.1M
Output
43.5K
Codex + DeepSeek-v4-pro-0813

Replicate 2

Matches predictionUDT + standard tools
Answer50;35;15
Workflow
No Galaxy tool job recorded
Total tokens
64.2M
Output
167.0K
Codex + DeepSeek-v4-pro-0813

Replicate 3

RerunMatches predictionUDT + standard tools
Answer50;35;15
Workflow
anndata import → anndata inspect → anndata manipulate → scanpy inspect → seurat create → anndata export → User-defined tool
Total tokens
8.4M
Output
47.9K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

Matches predictionUDT-only
Answer50;35;15
Workflow
Single-cell Proportion DiagnosticSingle-cell Proportion ValidationSingle-cell Proportions Final
Total tokens
1.1M
Output
23.0K
Codex GPT-5.6 Luna

Replicate 2

Matches predictionUDT-only
Answer50;35;15
Workflow
Matrix runtime probeExplore single-cell mixtureQuick matrix statsProbe single-cell mixture clustersValidate single-cell mixture clusters3 more tools
Total tokens
1.5M
Output
33.1K
Codex GPT-5.6 Luna

Replicate 3

Matches predictionStandard tools only
Answer50;35;15
Workflow
Anndata ImportAnndata InspectAnndata ManipulateScanpy InspectDatamash Ops5 more tools
Total tokens
20.5M
Output
35.8K

Anycode

13/13 terminal item results

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

Matches prediction
Answer50;35;15
Workflow
Local codePythonSciPyScanpyAnnData
Codex GPT-5.5

Replicate 2

Matches prediction
Answer50;35;15
Workflow
Local codePythonSciPyScanpyAnnData
Codex GPT-5.5

Replicate 3

Matches prediction
Answer50;35;15
Workflow
Local codePythonSciPy

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

Matches prediction
Answer50;35;15
Workflow
Local codePythonSciPyScanpyAnnData
Codex GPT-5.6 Sol

Replicate 2

Matches prediction
Answer50;35;15
Workflow
Local codePythonSciPyScanpy
Codex GPT-5.6 Sol

Replicate 3

Matches prediction
Answer50;35;15
Workflow
Local codePythonSciPyScanpyAnnData

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

Matches prediction
Answer50;35;15
Workflow
Local codePythonSciPyAnnDataScanpy
Total tokens
563.8K
Output
14.2K
Codex + DeepSeek-v4-pro-0813

Replicate 2

Matches prediction
Answer50;35;15
Workflow
Local codePythonSciPyAnnDataScanpy
Total tokens
1.1M
Output
26.0K
Codex + DeepSeek-v4-pro-0813

Replicate 3

Matches prediction
Answer50;35;15
Workflow
Local codePythonSciPyAnnDataScanpy
Total tokens
1.9M
Output
23.0K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

Matches prediction
Answer50;35;15
Workflow
Local codePythonSciPyScanpy
Codex GPT-5.6 Luna

Replicate 2

Matches prediction
Answer50;35;15
Workflow
Local codePythonSciPyScanpy
Codex GPT-5.6 Luna

Replicate 3

Matches prediction
Answer50;35;15
Workflow
Local codePythonSciPyScanpyAnnData

Codex GPT-6 Astra

1 replicate
Codex GPT-6 Astra

Replicate 1

Matches prediction
Answer50;35;15
Workflow
Local codePythonSciPyScanpy
Total tokens
235.4K
Output
2.5K