Back to results

CompBioBench item

perturb-seq-effect-q1

Question

You are given the cellranger output of a CRISPR perturb-seq experiment in perturb.seq.effect.q1.tar.gz where cell features are at filtered_feature_bc_matrix/features.tsv.gz, counts are at filtered_feature_bc_matrix/matrix.mtx.gz, barcodes are at filtered_feature_bc_matrix/barcodes.tsv.gz, guide assignment per cell is at perturb_protospacer_calls_per_cell.csv, and guide reference at feature_reference.csv. I am interested in finding guides which robustly knockdown the interferon gamma pathway. We need to decide which target gene from this screen to explore in a follow up screen - which gene target should be selected? Respond with a single gene target name (i.e. CCL6).

Inferred answer
ISG15Uniquely inferred from submitted scores.
Domain
Single-cell
Question style
Synthetic/Augmented Data

An answer can be uniquely inferred for 54 of 100 questions from submitted scores. Green: correct · Red: incorrect. For questions without a uniquely determined answer: Blue: matches prediction · Orange: differs from prediction. These are inferred answer checks, not official per-question grades.

Galaxy

12/12 terminal item results

UDT = user-defined tool: custom code executed by Galaxy. Route badges summarize analysis tools, excluding uploads and automatic conversions. How usage is counted.

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

CorrectUDT-only
AnswerISG15
Workflow
Perturb-seq DiagnosticPerturb-seq IFNG pathway analysisPerturb-seq IFNG pathway robust analysis
Total tokens
711.9K
Output
12.2K
Codex GPT-5.5

Replicate 2

CorrectUDT-only
AnswerISG15
Workflow
Perturb-seq archive diagnosticPerturb-seq IFNG pathway scoringPerturb-seq robust IFNG pathway scoring
Total tokens
592.7K
Output
12.6K
Codex GPT-5.5

Replicate 3

CorrectUDT + standard tools
AnswerISG15
Workflow
Perturb-seq diagnosticUDT Python probeUDT Busybox probeUnzip
Total tokens
2.3M
Output
19.7K

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

CorrectUDT + standard tools
AnswerISG15
Workflow
Perturb-seq archive QCConverter Archive To DirectoryScanpy Read 10XAnndata InspectCsv To Tabular8 more tools
Total tokens
20.6M
Output
41.1K
Codex GPT-5.6 Sol

Replicate 2

CorrectUDT-only
AnswerISG15
Workflow
Perturb-seq archive diagnosticsPerturb-seq metadata and QCIFNG Perturb-seq robust target screen
Total tokens
1.5M
Output
12.2K
Codex GPT-5.6 Sol

Replicate 3

CorrectUDT-only
AnswerISG15
Workflow
Perturb-seq archive diagnosticPerturb-seq IFNG robust target rankingPerturb-seq IFNG top-target validation
Total tokens
1.6M
Output
20.6K

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

IncorrectStandard tools only
AnswerSTAT1
Workflow
Converter Archive To DirectoryCsv To TabularScanpy Read 10XScanpy NormalizeScanpy Inspect2 more tools
Total tokens
14.0M
Output
47.2K
Codex + DeepSeek-v4-pro-0813

Replicate 2

CorrectStandard tools only
AnswerISG15
Workflow
Converter Archive To DirectoryConverter Gz To UncompressedScanpy Read 10XCsv To TabularTp Sort Header Tool12 more tools
Total tokens
11.5M
Output
37.5K
Codex + DeepSeek-v4-pro-0813

Replicate 3

CorrectUDT + standard tools
AnswerISG15
Workflow
Perturb-seq diagnosticCat1UDT probeConverter Archive To DirectoryConverter Gz To Uncompressed11 more tools
Total tokens
30.0M
Output
56.8K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

CorrectUDT-only
AnswerISG15
Workflow
Converter Archive To DirectoryPerturb-seq archive diagnosticPerturb-seq metadata exportPerturb-seq IFN-gamma guide screenPerturb-seq curated IFN-gamma sensitivity1 more tools
Total tokens
4.1M
Output
39.3K
Codex GPT-5.6 Luna

Replicate 2

CorrectUDT-only
AnswerISG15
Workflow
Converter Archive To DirectoryPerturb-seq input inspectionPerturb-seq metadata diagnosticPerturb-seq interferon-gamma scoringPerturb-seq hit validation
Total tokens
1.5M
Output
24.3K
Codex GPT-5.6 Luna

Replicate 3

CorrectUDT-only
AnswerISG15
Workflow
Perturb archive diagnosticPerturb design diagnosticPilot IFNG pathway scoreRobust IFNG guide analysis
Total tokens
985.1K
Output
24.5K

Anycode

13/13 terminal item results

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

Incorrect
AnswerSTAT1
Workflow
Local codePythonSciPy
Codex GPT-5.5

Replicate 2

Correct
AnswerISG15
Workflow
Local codePythonSciPyAnnDataScanpy
Codex GPT-5.5

Replicate 3

Correct
AnswerISG15
Workflow
Local codePythonSciPy

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

Correct
AnswerISG15
Workflow
Local codePythonSciPy
Codex GPT-5.6 Sol

Replicate 2

Correct
AnswerISG15
Workflow
Local codePython
Codex GPT-5.6 Sol

Replicate 3

Correct
AnswerISG15
Workflow
Local codePython

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

Correct
AnswerISG15
Workflow
Local codePythonSciPyScanpy
Total tokens
1.3M
Output
21.8K
Codex + DeepSeek-v4-pro-0813

Replicate 2

Correct
AnswerISG15
Workflow
Local codePythonSciPyScanpyAnnData1 more tools
Total tokens
2.0M
Output
40.5K
Codex + DeepSeek-v4-pro-0813

Replicate 3

Correct
AnswerISG15
Workflow
Local codePythonSciPy
Total tokens
881.9K
Output
21.5K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

Correct
AnswerISG15
Workflow
Local codePythonSciPyScanpy
Codex GPT-5.6 Luna

Replicate 2

Incorrect
AnswerSTAT1
Workflow
Local codePythonSciPyAnnDataScanpy
Codex GPT-5.6 Luna

Replicate 3

Incorrect
AnswerSTAT1
Workflow
Local codePythonSciPy

Codex GPT-6 Astra

1 replicate
Codex GPT-6 Astra

Replicate 1

Correct
AnswerISG15
Workflow
Local codePythonSciPyScanpyAnnData
Total tokens
167.8K
Output
1.8K