Back to results

CompBioBench item

annotate-variant-gene-impact-q2

Question

Using the Ensembl VEP API, determine the consequence for the following variant rs2136714166. Respond with a single word.

Predicted answer
upstream_gene_variantMost likely answer for this question; not uniquely determined from submitted scores.
Domain
Genomics
Question style
Retrieval

An answer can be uniquely inferred for 54 of 100 questions from submitted scores. Green: correct · Red: incorrect. For questions without a uniquely determined answer: Blue: matches prediction · Orange: differs from prediction. These are inferred answer checks, not official per-question grades.

Galaxy

12/12 terminal item results

UDT = user-defined tool: custom code executed by Galaxy. Route badges summarize analysis tools, excluding uploads and automatic conversions. How usage is counted.

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

Matches predictionNo analysis tools recorded
Answerupstream_gene_variant
Workflow
Upload1
Total tokens
622.6K
Output
6.4K
Codex GPT-5.5

Replicate 2

Matches predictionUDT-only
Answerupstream_gene_variant
Workflow
VEP rs2136714166
Total tokens
280.6K
Output
2.0K
Codex GPT-5.5

Replicate 3

Matches predictionUDT-only
Answerupstream_gene_variant
Workflow
Ensembl VEP rsID REST lookup rs2136714166
Total tokens
763.0K
Output
8.2K

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

Matches predictionUDT-only
Answerupstream_gene_variant
Workflow
Ensembl VEP API rsID consequenceVerify Ensembl VEP most severe consequenceExtract Ensembl VEP most severe consequence
Codex GPT-5.6 Sol

Replicate 2

Matches predictionUDT-only
Answerupstream_gene_variant
Workflow
Ensembl VEP API rs2136714166
Total tokens
254.3K
Output
2.0K
Codex GPT-5.6 Sol

Replicate 3

Matches predictionUDT-only
Answerupstream_gene_variant
Workflow
Ensembl VEP REST annotation rs2136714166Ensembl VEP REST annotation rs2136714166 retry
Total tokens
6.0M
Output
12.2K

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

Matches predictionUDT-only
Answerupstream_gene_variant
Workflow
Ensembl VEP rs lookup
Total tokens
344.6K
Output
5.0K
Codex + DeepSeek-v4-pro-0813

Replicate 2

Matches predictionUDT + standard tools
Answerupstream_gene_variant
Workflow
No Galaxy tool job recorded
Total tokens
4.1M
Output
22.1K
Codex + DeepSeek-v4-pro-0813

Replicate 3

Matches predictionUDT-only
Answerupstream_gene_variant
Workflow
VEP rsID consequence lookupVEP rsID response diagnostic
Total tokens
541.1K
Output
10.0K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

Matches predictionUDT-only
Answerupstream_gene_variant
Workflow
Ensembl VEP API query for rs2136714166Ensembl VEP API query for rs2136714166 (retry)Ensembl VEP API diagnostic for rs2136714166Ensembl VEP API host diagnosticEnsembl VEP API assembly check for rs21367141661 more tools
Total tokens
1.1M
Output
13.7K
Codex GPT-5.6 Luna

Replicate 2

Matches predictionNo analysis tools recorded
Answerupstream_gene_variant
Workflow
Upload1
Total tokens
1.7M
Output
9.7K
Codex GPT-5.6 Luna

Replicate 3

Matches predictionUDT-only
Answerupstream_gene_variant
Workflow
Ensembl VEP rs queryEnsembl VEP rs query consensusEnsembl VEP rs query with retry
Total tokens
1.6M
Output
10.3K

Anycode

13/13 terminal item results

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

Matches prediction
Answerupstream_gene_variant
Workflow
Local codePython
Codex GPT-5.5

Replicate 2

Matches prediction
Answerupstream_gene_variant
Workflow
Local code
Codex GPT-5.5

Replicate 3

Matches prediction
Answerupstream_gene_variant
Workflow
Local code

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

Matches prediction
Answerupstream_gene_variant
Workflow
Local code
Codex GPT-5.6 Sol

Replicate 2

Matches prediction
Answerupstream_gene_variant
Workflow
Local code
Codex GPT-5.6 Sol

Replicate 3

Matches prediction
Answerupstream_gene_variant
Workflow
Local code

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

Matches prediction
Answerupstream_gene_variant
Workflow
Local code
Total tokens
34.1K
Output
720
Codex + DeepSeek-v4-pro-0813

Replicate 2

Matches prediction
Answerupstream_gene_variant
Workflow
Local codePython
Total tokens
93.6K
Output
2.9K
Codex + DeepSeek-v4-pro-0813

Replicate 3

Matches prediction
Answerupstream_gene_variant
Workflow
Local code
Total tokens
63.3K
Output
1.3K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

Matches prediction
Answerupstream_gene_variant
Workflow
Local code
Codex GPT-5.6 Luna

Replicate 2

Matches prediction
Answerupstream_gene_variant
Workflow
Local code
Codex GPT-5.6 Luna

Replicate 3

Matches prediction
Answerupstream_gene_variant
Workflow
Local code

Codex GPT-6 Astra

1 replicate
Codex GPT-6 Astra

Replicate 1

Matches prediction
Answerupstream_gene_variant
Workflow
Local codePython
Total tokens
36.1K
Output
158