Back to results

CompBioBench item

lung-cancer-sc-q1

Question

You are given a single cell transcriptomic counts data generated from primary human tumor samples from patients with lung cancer (lung.cancer.sc.h5ad). This file has been cleaned for doublets and low quality cells already. Each sample is from a patient with lung adenocarcinoma (LUAD) or lung squamous cell carcinoma (LUSC). Out of samples Patient_005, Patient_006, Patient_007, Patient_018, and Patient_040, identify the LUSC sample with the highest proportion of dendritic cells out of all cells in that sample? Secondly, what percentage, to the nearest 20 percent, of all non-immune cells across all samples are malignant basal cells from LUSC patients? Expected answer format: Patient_006;20.

Inferred answer
Patient_018;40Uniquely inferred from submitted scores.
Domain
Single-cell
Question style
Routine Analysis

An answer can be uniquely inferred for 54 of 100 questions from submitted scores. Green: correct · Red: incorrect. For questions without a uniquely determined answer: Blue: matches prediction · Orange: differs from prediction. These are inferred answer checks, not official per-question grades.

Galaxy

12/12 terminal item results

UDT = user-defined tool: custom code executed by Galaxy. Route badges summarize analysis tools, excluding uploads and automatic conversions. How usage is counted.

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

IncorrectUDT-only
AnswerPatient_018;20
Workflow
Galaxy dataset inspection
Total tokens
7.1M
Output
33.2K
Codex GPT-5.5

Replicate 2

CorrectUDT + standard tools
AnswerPatient_018;40
Workflow
Anndata InspectH5AD DiagnosticH5AD Diagnostic QuayH5AD Diagnostic Quay SimpleH5AD Diagnostic No Shell Vars6 more tools
Total tokens
6.3M
Output
22.6K
Codex GPT-5.5

Replicate 3

Rerun ยท standard service (no fast)CorrectUDT-only
AnswerPatient_018;40
Workflow
H5AD obs diagnosticH5AD structure diagnosticLung marker diagnosticH5AD index previewLung final marker analysis
Total tokens
2.0M
Output
20.7K

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

CorrectUDT + standard tools
AnswerPatient_018;40
Workflow
Anndata InspectXlsx2TsvLung source barcode mapping diagnosticLung source barcode mapping diagnostic v2Lung source barcode mapping diagnostic v31 more tools
Total tokens
8.8M
Output
17.9K
Codex GPT-5.6 Sol

Replicate 2

CorrectUDT + standard tools
AnswerPatient_018;40
Workflow
Anndata InspectLung cancer AnnData marker diagnosticCellTypist lung reference predictionCellTypist lung airway reference predictionAggregate lung annotation evidence1 more tools
Total tokens
5.1M
Output
20.6K
Codex GPT-5.6 Sol

Replicate 3

CorrectUDT + standard tools
AnswerPatient_018;40
Workflow
Anndata InspectLung cancer barcode mapping diagnosticLung cancer final cell composition counts
Total tokens
4.0M
Output
14.4K

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

CorrectUDT + standard tools
AnswerPatient_018;40
Workflow
Anndata InspectRemove Beginning1Group and aggregateLung cancer diagnosticLung cancer cluster analysis1 more tools
Total tokens
5.5M
Output
44.1K
Codex + DeepSeek-v4-pro-0813

Replicate 2

IncorrectUDT + standard tools
AnswerPatient_018;20
Workflow
Anndata InspectAnndata ExportAnnDataScanpySciPy
Total tokens
18.6M
Output
59.3K
Codex + DeepSeek-v4-pro-0813

Replicate 3

RerunCorrectUDT + standard tools
AnswerPatient_018;40
Workflow
AnnData inspectionSample and marker countsCell-type signature scoringMalignant basal-cell analysis
Total tokens
8.2M
Output
47.1K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

CorrectUDT-only
AnswerPatient_018;40
Workflow
Lung cancer H5AD runtime probeLung cancer expression diagnosticLung cancer cluster diagnosticLung cancer cluster marker mappingLung cancer final proportion analysis1 more tools
Total tokens
6.3M
Output
46.6K
Codex GPT-5.6 Luna

Replicate 2

CorrectUDT + standard tools
AnswerPatient_018;40
Workflow
Anndata InspectLung runtime probeLung H5AD structureLung marker diagnosticLung marker summary7 more tools
Total tokens
14.6M
Output
59.4K
Codex GPT-5.6 Luna

Replicate 3

CorrectUDT + standard tools
AnswerPatient_018;40
Workflow
H5AD inspectionMarker score diagnosticFilter TabularMatch lung cancer metadataMatch anonymized cell barcodes to source metadata6 more tools
Total tokens
11.3M
Output
61.9K

Anycode

13/13 terminal item results

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

Correct
AnswerPatient_018;40
Workflow
Local codePythonAnnDataScanpySciPy
Codex GPT-5.5

Replicate 2

Correct
AnswerPatient_018;40
Workflow
Local codePythonAnnDataScanpy
Codex GPT-5.5

Replicate 3

Incorrect
AnswerPatient_018;20
Workflow
Local codePythonAnnDataScanpy

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

Incorrect
AnswerPatient_018;20
Workflow
Local codePythonAnnDataScanpy
Codex GPT-5.6 Sol

Replicate 2

Incorrect
AnswerPatient_018;20
Workflow
Local codePythonAnnDataScanpy
Codex GPT-5.6 Sol

Replicate 3

Correct
AnswerPatient_018;40
Workflow
Local codePythonAnnData

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

Correct
AnswerPatient_018;40
Workflow
Local codePythonAnnDataScanpySciPy
Total tokens
3.3M
Output
41.7K
Codex + DeepSeek-v4-pro-0813

Replicate 2

Correct
AnswerPatient_018;40
Workflow
Local codePythonAnnDataScanpySciPy1 more tools
Total tokens
3.9M
Output
30.4K
Codex + DeepSeek-v4-pro-0813

Replicate 3

Incorrect
AnswerPatient_018;20
Workflow
Local codePythonAnnDataScanpySciPy
Total tokens
36.0M
Output
158.9K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

Incorrect
AnswerPatient_018;20
Workflow
Local codePythonAnnDataScanpySciPy
Codex GPT-5.6 Luna

Replicate 2

Incorrect
AnswerPatient_018;20
Workflow
Local codePythonAnnDataScanpy
Codex GPT-5.6 Luna

Replicate 3

Correct
AnswerPatient_018;40
Workflow
Local codePythonAnnDataScanpySciPy

Codex GPT-6 Astra

1 replicate
Codex GPT-6 Astra

Replicate 1

Incorrect
AnswerPatient_005;40
Workflow
Local codePythonAnnDataScanpySciPy
Total tokens
7.2M
Output
31.2K