Back to results

CompBioBench item

annotate-variant-gene-impact-q1

Question

What is the clinical significance reported in the ClinVar database for the SPI1 variant represented by the HGVS notation NM_003120.3(SPI1):c.143-2A>C? Respond with only the correct cateogry. Answer in all lowercase.

Predicted answer
pathogenicMost likely answer for this question; not uniquely determined from submitted scores.
Domain
Genomics
Question style
Retrieval

An answer can be uniquely inferred for 54 of 100 questions from submitted scores. Green: correct · Red: incorrect. For questions without a uniquely determined answer: Blue: matches prediction · Orange: differs from prediction. These are inferred answer checks, not official per-question grades.

Galaxy

12/12 terminal item results

UDT = user-defined tool: custom code executed by Galaxy. Route badges summarize analysis tools, excluding uploads and automatic conversions. How usage is counted.

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

Matches predictionStandard tools only
Answerpathogenic
Workflow
Upload1Grep
Total tokens
1.0M
Output
5.9K
Codex GPT-5.5

Replicate 2

Matches predictionUDT-only
Answerpathogenic
Workflow
ClinVar HGVS lookup SPI1 c143m2ClinVar UID diagnostic SPI1 c143m2ClinVar HGVS lookup SPI1 c143m2 v2
Total tokens
617.6K
Output
5.5K
Codex GPT-5.5

Replicate 3

Matches predictionUDT-only
Answerpathogenic
Workflow
ClinVar HGVS lookup for SPI1 variantClinVar HGVS lookup for SPI1 variant v2Upload1
Total tokens
1.3M
Output
11.1K

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

Matches predictionStandard tools only
Answerpathogenic
Workflow
Tp Replace In LineGrepTp Grep Tool
Codex GPT-5.6 Sol

Replicate 2

Matches predictionUDT-only
Answerpathogenic
Workflow
ClinVar exact HGVS lookupClinVar exact HGVS lookup with NCBI rate limitingClinVar exact HGVS summary
Total tokens
616.6K
Output
5.0K
Codex GPT-5.6 Sol

Replicate 3

Matches predictionUDT-only
Answerpathogenic
Workflow
ClinVar exact HGVS lookup for SPI1
Total tokens
3.0M
Output
7.4K

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

Matches predictionStandard tools only
Answerpathogenic
Workflow
Tp Grep Tool
Total tokens
419.5K
Output
4.7K
Codex + DeepSeek-v4-pro-0813

Replicate 2

Matches predictionStandard tools only
Answerpathogenic
Workflow
Tp Grep Tool
Total tokens
337.6K
Output
3.4K
Codex + DeepSeek-v4-pro-0813

Replicate 3

Matches predictionStandard tools only
Answerpathogenic
Workflow
Ctb Online Data FetchBcftools Query
Total tokens
1.2M
Output
11.2K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

Matches predictionStandard tools only
Answerpathogenic
Workflow
Data FetchTp Find And ReplaceTp Grep ToolTp Head Tool
Total tokens
4.5M
Output
22.0K
Codex GPT-5.6 Luna

Replicate 2

Matches predictionStandard tools only
Answerpathogenic
Workflow
Data FetchTp Grep Tool
Total tokens
3.8M
Output
11.9K
Codex GPT-5.6 Luna

Replicate 3

Matches predictionUDT + standard tools
Answerpathogenic
Workflow
ClinVar HGVS exact lookupFilter TabularTp Grep Tool
Total tokens
3.3M
Output
19.8K

Anycode

13/13 terminal item results

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

Matches prediction
Answerpathogenic
Workflow
Local code
Codex GPT-5.5

Replicate 2

Matches prediction
Answerpathogenic
Workflow
Local code
Codex GPT-5.5

Replicate 3

Matches prediction
Answerpathogenic
Workflow
No command recorded

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

Matches prediction
Answerpathogenic
Workflow
Local code
Codex GPT-5.6 Sol

Replicate 2

Matches prediction
Answerpathogenic
Workflow
Local code
Codex GPT-5.6 Sol

Replicate 3

Matches prediction
Answerpathogenic
Workflow
Local code

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

Matches prediction
Answerpathogenic
Workflow
No command recorded
Total tokens
21.2K
Output
157
Codex + DeepSeek-v4-pro-0813

Replicate 2

Matches prediction
Answerpathogenic
Workflow
No command recorded
Total tokens
34.1K
Output
258
Codex + DeepSeek-v4-pro-0813

Replicate 3

Matches prediction
Answerpathogenic
Workflow
No command recorded
Total tokens
34.0K
Output
268

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

Matches prediction
Answerpathogenic
Workflow
Local code
Codex GPT-5.6 Luna

Replicate 2

Matches prediction
Answerpathogenic
Workflow
Local codePython
Codex GPT-5.6 Luna

Replicate 3

Matches prediction
Answerpathogenic
Workflow
Local code

Codex GPT-6 Astra

1 replicate
Codex GPT-6 Astra

Replicate 1

Matches prediction
Answerpathogenic
Workflow
Local codePython
Total tokens
36.2K
Output
210