Back to results

CompBioBench item

sample-swap-rna-q1

Question

I have a scRNA-seq pseudobulk lying around from amphioxus tissue (sample.swap.rna.q1.tsv.gz). I'm not sure, but I suspect there is a sample swap. Can you check if that is the case? If there isn't, respond "None". Else respond with the names of the two cell types that have been swapped in case-sensitive lexicographic order, e.g. "endoderm,epidermis".

Inferred answer
epidermal_neural,tailbud_neuralUniquely inferred from submitted scores.
Domain
Single-cell
Question style
Metadata Recovery

An answer can be uniquely inferred for 54 of 100 questions from submitted scores. Green: correct · Red: incorrect. For questions without a uniquely determined answer: Blue: matches prediction · Orange: differs from prediction. These are inferred answer checks, not official per-question grades.

Galaxy

12/12 terminal item results

UDT = user-defined tool: custom code executed by Galaxy. Route badges summarize analysis tools, excluding uploads and automatic conversions. How usage is counted.

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

CorrectUDT-only
Answerepidermal_neural,tailbud_neural
Workflow
Amphioxus Reference Layout ProbeDetect Amphioxus Pseudobulk Sample SwapSummarize Amphioxus Reference LabelsDetect Amphioxus Pseudobulk Sample Swap v2Detect Amphioxus Swap From Single Cell Counts3 more tools
Codex GPT-5.5

Replicate 2

CorrectUDT-only
Answerepidermal_neural,tailbud_neural
Workflow
Amphioxus swap checkAmphioxus swap check v2Amphioxus swap check v3
Total tokens
3.2M
Output
13.6K
Codex GPT-5.5

Replicate 3

RerunIncorrectUDT + standard tools
AnswerNone
Workflow
Select firstSwap diagnostic for pseudobulk matrixAnnotate BLAG marker genes from BraLan3 GFFExtract selected marker expression from pseudobulk
Total tokens
2.4M
Output
18.2K

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

CorrectUDT-only
Answerepidermal_neural,tailbud_neural
Workflow
Pseudobulk geometry and specificityInspect BLAG annotation overlapSummarize amphioxus cell annotationsReconstruct and match amphioxus pseudobulkSummarize pseudobulk assignment1 more tools
Total tokens
36.5M
Output
28.6K
Codex GPT-5.6 Sol

Replicate 2

CorrectUDT-only
Answerepidermal_neural,tailbud_neural
Workflow
Amphioxus pseudobulk annotation diagnosticAmphioxus pseudobulk swap reference match
Total tokens
4.0M
Output
15.8K
Codex GPT-5.6 Sol

Replicate 3

CorrectUDT-only
Answerepidermal_neural,tailbud_neural
Workflow
Amphioxus pseudobulk profile diagnosticAmphioxus authoritative reference swap testAmphioxus authoritative fine reference swap testAmphioxus exhaustive 12-profile swap validation
Total tokens
8.4M
Output
16.7K

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

CorrectUDT + standard tools
Answerepidermal_neural,tailbud_neural
Workflow
No Galaxy tool job recorded
Total tokens
49.9M
Output
135.6K
Codex + DeepSeek-v4-pro-0813

Replicate 2

CorrectUDT-only
Answerepidermal_neural,tailbud_neural
Workflow
Sample Swap CompareSample Swap Compare v2
Total tokens
1.4M
Output
17.4K
Codex + DeepSeek-v4-pro-0813

Replicate 3

CorrectUDT + standard tools
Answerepidermal_neural,tailbud_neural
Workflow
Check Pseudobulk Sample SwapUDT ProbeUDT Probe BusyboxDatamash TransposeTable Compute10 more tools
Total tokens
38.6M
Output
147.1K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

CorrectUDT + standard tools
Answerepidermal_neural,tailbud_neural
Workflow
Group and aggregateTp Head ToolAmphioxus source matching diagnosticUDT render probeUDT exact documentation probe18 more tools
Total tokens
63.5M
Output
140.5K
Codex GPT-5.6 Luna

Replicate 2

CorrectUDT + standard tools
Answerepidermal_neural,tailbud_neural
Workflow
Amphioxus swap diagnosticUDT render probeUDT input render probeUDT command probeUDT legacy probe18 more tools
Total tokens
48.5M
Output
138.0K
Codex GPT-5.6 Luna

Replicate 3

CorrectUDT + standard tools
Answerepidermal_neural,tailbud_neural
Workflow
Amphioxus sample swap diagnosticUDT copy probeUDT shell probeTp Head ToolUDT resource probe19 more tools
Total tokens
70.3M
Output
182.8K

Anycode

13/13 terminal item results

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

Correct
Answerepidermal_neural,tailbud_neural
Workflow
Local codePython
Codex GPT-5.5

Replicate 2

Correct
Answerepidermal_neural,tailbud_neural
Workflow
Local codePython
Codex GPT-5.5

Replicate 3

Correct
Answerepidermal_neural,tailbud_neural
Workflow
Local codePythonSciPy

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

Correct
Answerepidermal_neural,tailbud_neural
Workflow
Local codePython
Codex GPT-5.6 Sol

Replicate 2

Correct
Answerepidermal_neural,tailbud_neural
Workflow
Local codePython
Codex GPT-5.6 Sol

Replicate 3

Correct
Answerepidermal_neural,tailbud_neural
Workflow
Local codePythonSciPy

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

Correct
Answerepidermal_neural,tailbud_neural
Workflow
Local codePythonSciPyRScanpy1 more tools
Total tokens
27.2M
Output
73.8K
Codex + DeepSeek-v4-pro-0813

Replicate 2

Correct
Answerepidermal_neural,tailbud_neural
Workflow
Local codePythonSciPy
Total tokens
3.8M
Output
29.2K
Codex + DeepSeek-v4-pro-0813

Replicate 3

Incorrect
Answerepidermal_neural,neural_like
Workflow
Local codePython
Total tokens
13.8M
Output
53.6K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

Correct
Answerepidermal_neural,tailbud_neural
Workflow
Local codePythonSciPy
Codex GPT-5.6 Luna

Replicate 2

Correct
Answerepidermal_neural,tailbud_neural
Workflow
Local codePython
Codex GPT-5.6 Luna

Replicate 3

Correct
Answerepidermal_neural,tailbud_neural
Workflow
Local codePython

Codex GPT-6 Astra

1 replicate
Codex GPT-6 Astra

Replicate 1

Correct
Answerepidermal_neural,tailbud_neural
Workflow
Local codePython
Total tokens
884.5K
Output
4.7K