Back to results

CompBioBench item

spatial-sim-q1

Question

You are given a set of single cell transcriptomic data and Visium spatial transcriptomic data (spatial.sim.tar.gz). The single cell counts matrix is found at spatial_q_sc_counts.csv, and accompanying cell type metadata is found at spatial_q_sc_metadata.csv. Spatial count data is found at spatial_q_matrix.mtx.gz, gene counts are found at spatial_q_features.tsv.gz, and are found at spot barcodes spatial_q_barcodes.tsv.gz. Finally scale factors are found at spatial_q_scalefactors_json.json, high and low resolution tissue images are found at spatial_q_tissue_hires_image.png and patial_q_tissue_hires_image.png respectively, and tissue positions are found at spatial_q_tissue_positions.csv. The spot-based data is comprised entirely of cell types found in the single cell data. Can you tell me what cell type or cell types are included in spot ID Spot_710-1? Please list these separated by commas in alphabetical order. Expected answer format: Endothelial,Macrophage.

Predicted answer
B_Cell,Endothelial,MacrophageMost likely answer for this question; not uniquely determined from submitted scores.
Domain
Spatial
Question style
Synthetic/Augmented Data

An answer can be uniquely inferred for 54 of 100 questions from submitted scores. Green: correct · Red: incorrect. For questions without a uniquely determined answer: Blue: matches prediction · Orange: differs from prediction. These are inferred answer checks, not official per-question grades.

Galaxy

12/12 terminal item results

UDT = user-defined tool: custom code executed by Galaxy. Route badges summarize analysis tools, excluding uploads and automatic conversions. How usage is counted.

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

Matches predictionUDT-only
AnswerB_Cell,Endothelial,Macrophage
Workflow
Spot Cell Type NNLSSpot Cell Type NNLS v2Spot Cell Type NNLS v3 subset check
Codex GPT-5.5

Replicate 2

Matches predictionUDT-only
AnswerB_Cell,Endothelial,Macrophage
Workflow
Galaxy dataset inspection
Total tokens
2.4M
Output
20.2K
Codex GPT-5.5

Replicate 3

RerunMatches predictionUDT-only
AnswerB_Cell,Endothelial,Macrophage
Workflow
Spot 710 cell type deconvolution
Total tokens
291.2K
Output
9.1K

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

Matches predictionUDT-only
AnswerB_Cell,Endothelial,Macrophage
Workflow
SciPy
Total tokens
1.6M
Output
9.0K
Codex GPT-5.6 Sol

Replicate 2

Matches predictionUDT-only
AnswerB_Cell,Endothelial,Macrophage
Workflow
Spatial archive diagnosticTarget spot deconvolution diagnostic
Total tokens
480.3K
Output
8.0K
Codex GPT-5.6 Sol

Replicate 3

Matches predictionUDT-only
AnswerB_Cell,Endothelial,Macrophage
Workflow
Spatial archive diagnosticSpatial reference deconvolution
Total tokens
408.4K
Output
10.6K

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

Matches predictionUDT + standard tools
AnswerB_Cell,Endothelial,Macrophage
Workflow
Dropletutils Read 10XSceasy ConvertDatamash TransposeCsv To TabularMusic Construct Eset6 more tools
Total tokens
10.1M
Output
34.3K
Codex + DeepSeek-v4-pro-0813

Replicate 2

Matches predictionUDT + standard tools
AnswerB_Cell,Endothelial,Macrophage
Workflow
Scanpy Read 10XAnndata ExportAnndata InspectDatamash TransposeMatrixMarket to tabular3 more tools
Total tokens
7.5M
Output
34.0K
Codex + DeepSeek-v4-pro-0813

Replicate 3

Matches predictionStandard tools only
AnswerB_Cell,Endothelial,Macrophage
Workflow
Datamash TransposeCsv To TabularMusic Construct EsetAnndata ImportAnndata Export3 more tools
Total tokens
7.6M
Output
31.6K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

Matches predictionUDT + standard tools
AnswerB_Cell,Endothelial,Macrophage
Workflow
Converter Archive To DirectorySpot input diagnosticUDT mechanics probeUDT mechanics probe twoCsv To Tabular9 more tools
Total tokens
29.2M
Output
69.8K
Codex GPT-5.6 Luna

Replicate 2

Matches predictionUDT-only
AnswerB_Cell,Endothelial,Macrophage
Workflow
Spatial archive diagnosticSpatial target probeSpatial spot mixture probeFinal Spot_710-1 cell-type analysis
Total tokens
1.2M
Output
20.3K
Codex GPT-5.6 Luna

Replicate 3

Matches predictionUDT-only
AnswerB_Cell,Endothelial,Macrophage
Workflow
Spatial data inspectionSpatial deconvolution probeSpot cell-type identification
Total tokens
953.9K
Output
14.6K

Anycode

13/13 terminal item results

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

Matches prediction
AnswerB_Cell,Endothelial,Macrophage
Workflow
Local codePythonSciPy
Codex GPT-5.5

Replicate 2

Matches prediction
AnswerB_Cell,Endothelial,Macrophage
Workflow
Local codePythonSciPy
Codex GPT-5.5

Replicate 3

Matches prediction
AnswerB_Cell,Endothelial,Macrophage
Workflow
Local codePythonSciPy

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

Matches prediction
AnswerB_Cell,Endothelial,Macrophage
Workflow
Local codePythonSciPy
Codex GPT-5.6 Sol

Replicate 2

Matches prediction
AnswerB_Cell,Endothelial,Macrophage
Workflow
Local codePythonSciPy
Codex GPT-5.6 Sol

Replicate 3

Matches prediction
AnswerB_Cell,Endothelial,Macrophage
Workflow
Local codePythonSciPy

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

Matches prediction
AnswerB_Cell,Endothelial,Macrophage
Workflow
Local codePythonSciPy
Total tokens
506.2K
Output
14.4K
Codex + DeepSeek-v4-pro-0813

Replicate 2

Matches prediction
AnswerB_Cell,Endothelial,Macrophage
Workflow
Local codePythonSciPy
Total tokens
707.9K
Output
19.0K
Codex + DeepSeek-v4-pro-0813

Replicate 3

Matches prediction
AnswerB_Cell,Endothelial,Macrophage
Workflow
Local codePythonSciPyScanpyAnnData
Total tokens
651.0K
Output
12.6K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

Matches prediction
AnswerB_Cell,Endothelial,Macrophage
Workflow
Local codePythonSciPy
Codex GPT-5.6 Luna

Replicate 2

Matches prediction
AnswerB_Cell,Endothelial,Macrophage
Workflow
Local codePythonSciPy
Codex GPT-5.6 Luna

Replicate 3

Matches prediction
AnswerB_Cell,Endothelial,Macrophage
Workflow
Local codePythonSciPy

Codex GPT-6 Astra

1 replicate
Codex GPT-6 Astra

Replicate 1

Matches prediction
AnswerB_Cell,Endothelial,Macrophage
Workflow
Local codePythonSciPy
Total tokens
140.2K
Output
1.2K