Back to results

CompBioBench item

protein-shape-q1

Question

Consider the PDB protein in the file protein.shape.q1.pdb. Which of the following uppercase letters does it resemble most closely: B, D, F, H, J, L, N, P, R, T, V, X, Z? Consider all possible projections and pick the best fit.

Inferred answer
FUniquely inferred from submitted scores.
Domain
Structure
Question style
Synthetic/Augmented Data

An answer can be uniquely inferred for 54 of 100 questions from submitted scores. Green: correct · Red: incorrect. For questions without a uniquely determined answer: Blue: matches prediction · Orange: differs from prediction. These are inferred answer checks, not official per-question grades.

Galaxy

12/12 terminal item results

UDT = user-defined tool: custom code executed by Galaxy. Route badges summarize analysis tools, excluding uploads and automatic conversions. How usage is counted.

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 2

IncorrectUDT-only
AnswerN
Workflow
PDB Shape DiagnosticPDB Shape Diagnostic RedirectedPDB Shape Letter Score
Total tokens
455.9K
Output
12.0K
Codex GPT-5.5

Replicate 3

IncorrectUDT-only
AnswerT
Workflow
Protein Letter Projection ScoreProtein Letter Projection Score v2Protein Letter Projection Score v3UDT Static ProbeUDT Static Probe v21 more tools
Total tokens
1.7M
Output
14.6K

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

IncorrectUDT-only
AnswerT
Workflow
Protein projection explorationProtein axis rotation scanProtein letter silhouette scoreProtein projection montage
Total tokens
1.4M
Output
15.1K
Codex GPT-5.6 Sol

Replicate 2

IncorrectUDT + standard tools
AnswerT
Workflow
Protein projection diagnosticProtein projection diagnostic v2Protein projection diagnostic v3Cat projection probeGrep3 more tools
Total tokens
3.1M
Output
14.0K
Codex GPT-5.6 Sol

Replicate 3

IncorrectUDT + standard tools
AnswerT
Workflow
PDB coordinate diagnostic Q1PDB coordinate diagnostic Q1 v2PDB coordinate diagnostic Q1 v3PDB coordinate diagnostic Q1 v4PDB coordinate diagnostic Q1 v56 more tools
Total tokens
8.6M
Output
18.8K

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

CorrectUDT-only
AnswerF
Workflow
Inspect PDBPCA PDBProject ASCIIProject PNGCompare PDB1 more tools
Total tokens
2.1M
Output
27.0K
Codex + DeepSeek-v4-pro-0813

Replicate 2

CorrectUDT-only
AnswerF
Workflow
SciPyProtein shape letter match
Total tokens
3.9M
Output
41.5K
Codex + DeepSeek-v4-pro-0813

Replicate 3

CorrectUDT + standard tools
AnswerF
Workflow
SciPyProbe MatplotlibShape ASCII ProjectionsShape Connected Lines ASCIIShape Render PGM2 more tools
Total tokens
3.4M
Output
30.1K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

IncorrectUDT + standard tools
AnswerZ
Workflow
Protein shape diagnosticProtein shape diagnostic v2Protein shape diagnostic v3UDT render probeApi Udt Probe V19 more tools
Total tokens
53.4M
Output
107.1K
Codex GPT-5.6 Luna

Replicate 2

RerunIncorrectUDT-only
AnswerN
Workflow
PDB Shape StatisticsPDB Projection DiagnosticPDB Letter Projection ScorePDB Letter Projection Score RefinedPDB Best Projection ASCII4 more tools
Total tokens
5.3M
Output
41.9K
Codex GPT-5.6 Luna

Replicate 3

CorrectUDT-only
AnswerF
Workflow
PDB Shape DiagnosticPDB Shape Projection RenderPDB Shape Letter MatchGalaxy UDTPDB Specific Projection Views
Total tokens
5.8M
Output
43.3K

Anycode

13/13 terminal item results

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

Incorrect
AnswerT
Workflow
Local codePython
Codex GPT-5.5

Replicate 2

Incorrect
AnswerT
Workflow
Local codePythonSciPy
Codex GPT-5.5

Replicate 3

Incorrect
AnswerT
Workflow
Local codePythonSciPy

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

Incorrect
AnswerT
Workflow
Local codePython
Codex GPT-5.6 Sol

Replicate 2

Incorrect
AnswerT
Workflow
Local codePython
Codex GPT-5.6 Sol

Replicate 3

Incorrect
AnswerT
Workflow
Local codePython

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

Correct
AnswerF
Workflow
Local codePython
Total tokens
477.2K
Output
9.3K
Codex + DeepSeek-v4-pro-0813

Replicate 2

Correct
AnswerF
Workflow
Local codePythonSciPy
Total tokens
2.8M
Output
42.7K
Codex + DeepSeek-v4-pro-0813

Replicate 3

Correct
AnswerF
Workflow
Local codePythonSciPy
Total tokens
2.2M
Output
36.8K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

Incorrect
AnswerP
Workflow
Local codePythonSciPy
Codex GPT-5.6 Luna

Replicate 2

Incorrect
AnswerJ
Workflow
Local codePythonSciPy
Codex GPT-5.6 Luna

Replicate 3

Incorrect
AnswerT
Workflow
Local codePythonSciPy

Codex GPT-6 Astra

1 replicate
Codex GPT-6 Astra

Replicate 1

Incorrect
AnswerT
Workflow
Local codePython
Total tokens
147.4K
Output
1.3K