BixBench current item

bix-14-q1

Question

In the BLM mutation carrier cohort, what fraction of coding variants with a variant allele frequency (VAF) below 0.3 are annotated as synonymous?

Accepted answer
(0.7, 0.8)
Domain
variant callset

Comparison

Pass counts and token totals across the same three replicates.

AgentGalaxy-API codeOpen-ended code
Codex GPT-5.53/3
replicate 1 replicate 2 replicate 3
8.50M total		3/3
replicate 1 replicate 2 replicate 3
1.90M total
Codex GPT-5.6 Sol3/3
replicate 1 replicate 2 replicate 3
10.95M total		3/3
replicate 1 replicate 2 replicate 3
1.50M total
Codex GPT-5.6 Luna2/3
replicate 1 replicate 2 replicate 3
49.95M total		3/3
replicate 1 replicate 2 replicate 3
4.63M total
DeepSeek V4 Pro via Codex1/3
replicate 1 replicate 2 replicate 3
30.42M total		0/3
replicate 1 replicate 2 replicate 3
10.09M total
DeepSeek V4 Pro via Claude Codesuperseded3/3
replicate 1 replicate 2 replicate 3
12.23M total		3/3
replicate 1 replicate 2 replicate 3
2.76M total

Execution condition

Galaxy-API code with skills

Codex GPT-5.5

3 primary replicates
Codex GPT-5.5

replicate 1

passUDT-only
Answer0.731707317073
Workflow
Galaxy UDTvariant consequence filtering
Total tokens
3.32M
Output
26.6K
Galaxy historyRun trace
Codex GPT-5.5

replicate 2

passUDT-only
Answer0.731707317073
Workflow
Galaxy UDTvariant consequence filtering
Total tokens
1.10M
Output
19.6K
Galaxy historyRun trace
Codex GPT-5.5

replicate 3

passUDT-only
Answer0.731707317073
Workflow
Galaxy UDTvariant consequence filtering
Total tokens
4.08M
Output
37.6K
Galaxy historyRun trace

Codex GPT-5.6 Sol

3 primary replicates
Codex GPT-5.6 Sol

replicate 1

passUDT-only
Answer0.731707317073
Workflow
Galaxy UDTvariant consequence filtering
Total tokens
2.33M
Output
24.3K
Galaxy historyRun trace
Codex GPT-5.6 Sol

replicate 2

pass
Answer0.7317073170731707
Workflow
Excel to tabularFilter TabularMatch rowsConcatenate datasetsWc Gnuvariant consequence filtering
Total tokens
5.18M
Output
16.3K
Galaxy historyRun trace
Codex GPT-5.6 Sol

replicate 3

pass
Answer0.7317073170731707
Workflow
Excel to tabularTp CatFilter rowsGroup and aggregatevariant consequence filtering
Total tokens
3.43M
Output
14.3K
Galaxy historyRun trace

Codex GPT-5.6 Luna

3 primary replicates
Codex GPT-5.6 Luna

replicate 1

pass
Answer0.7317073170731707
Workflow
Excel to tabularFilter TabularTable Computemore stepsGroup and aggregatevariant consequence filtering
Total tokens
31.97M
Output
59.2K
Galaxy historyRun trace
Codex GPT-5.6 Luna

replicate 2

fail

Why it missed

Cohort scope collapsed to one workbookThis run converted and analyzed one carrier workbook from an 88-dataset cohort. That branch contained one low-VAF coding missense variant and no synonymous variant, so it returned 0 instead of the cohort-level fraction between 0.7 and 0.8.
Answer0
Workflow
Excel to tabularFilter TabularMatch rowsVAF filteringvariant consequence filtering
Total tokens
8.58M
Output
30.4K
Galaxy historyRun trace
Codex GPT-5.6 Luna

replicate 3

pass
Answer0.7317073170731707
Workflow
Excel to tabularConcatenate datasetsBuild Listmore stepsCount rowsvariant consequence filtering
Total tokens
9.40M
Output
34.8K
Galaxy historyRun trace

DeepSeek V4 Pro via Codex

3 primary replicates
DeepSeek V4 Pro via Codex

replicate 1

fail

Evaluation status

Manual review pendingThe evaluator marked this answer incorrect. The answer, workflow summary, usage, and available run records are linked, but a manual root-cause audit has not yet been added.
Answer0.6382978723404256
Workflow
Galaxy historyvariant consequence filtering
Total tokens
14.82M
Output
59.7K
Galaxy historyRun trace
DeepSeek V4 Pro via Codex

replicate 2

fail

Evaluation status

Manual review pendingThe evaluator marked this answer incorrect. The answer, workflow summary, usage, and available run records are linked, but a manual root-cause audit has not yet been added.
Answer0.6382978723404256
Workflow
Galaxy via BioBlendvariant consequence filtering
Total tokens
5.61M
Output
27.8K
Galaxy historyRun trace
DeepSeek V4 Pro via Codex

replicate 3

pass
Answer0.7317073170731707
Workflow
Galaxy via BioBlendvariant consequence filtering
Total tokens
9.99M
Output
35.8K
Galaxy historyRun trace

DeepSeek V4 Pro via Claude CodeSuperseded

3 primary replicates
DeepSeek V4 Pro via Claude Code

replicate 1

pass
Answer0.7794117647
Workflow
Excel to tabularConcatenate datasetsvariant consequence filtering
Total tokens
3.25M
Output
42.5K
Galaxy historyRun trace
DeepSeek V4 Pro via Claude Code

replicate 2

pass
Answer0.783333
Workflow
Excel to tabularvariant consequence filtering
Total tokens
4.45M
Output
44.1K
Galaxy historyRun trace
DeepSeek V4 Pro via Claude Code

replicate 3

pass
Answer0.7833
Workflow
Excel to tabularvariant consequence filteringSciPyBiopython
Total tokens
4.53M
Output
50.6K
Galaxy historyRun trace

Execution condition

Open-ended code with skills

Codex GPT-5.5

3 primary replicates
Codex GPT-5.5

replicate 1

pass
Answer0.7317073170731707
Workflow
Local analysisvariant consequence filtering
Total tokens
411.1K
Output
10.5K
Run trace
Codex GPT-5.5

replicate 2

pass
Answer0.7317073170731707
Workflow
Local analysisvariant consequence filtering
Total tokens
797.9K
Output
9.4K
Run trace
Codex GPT-5.5

replicate 3

pass
Answer0.7317073170731707
Workflow
Local analysisvariant consequence filtering
Total tokens
692.2K
Output
7.1K
Run trace

Codex GPT-5.6 Sol

3 primary replicates
Codex GPT-5.6 Sol

replicate 1

pass
Answer0.7317073170731707
Workflow
Local analysis
Total tokens
642.9K
Output
7.6K
Run trace
Codex GPT-5.6 Sol

replicate 2

pass
Answer0.7317073170731707
Workflow
Local analysis
Total tokens
435.0K
Output
5.7K
Run trace
Codex GPT-5.6 Sol

replicate 3

pass
Answer0.7317073170731707
Workflow
Local analysis
Total tokens
418.5K
Output
6.2K
Run trace

Codex GPT-5.6 Luna

3 primary replicates
Codex GPT-5.6 Luna

replicate 1

pass
Answer0.7317073171
Workflow
Local analysis
Total tokens
1.52M
Output
21.2K
Run trace
Codex GPT-5.6 Luna

replicate 2

pass
Answer0.7317073170731707
Workflow
Local analysis
Total tokens
1.27M
Output
16.5K
Run trace
Codex GPT-5.6 Luna

replicate 3

pass
Answer0.7317073170731707
Workflow
Local analysis
Total tokens
1.84M
Output
22.5K
Run trace

DeepSeek V4 Pro via Codex

3 primary replicates
DeepSeek V4 Pro via Codex

replicate 1

fail

Evaluation status

Manual review pendingThe evaluator marked this answer incorrect. The answer, workflow summary, usage, and available run records are linked, but a manual root-cause audit has not yet been added.
Answer0.6382978723404256
Workflow
Local analysisvariant consequence filtering
Total tokens
1.02M
Output
13.7K
Run trace
DeepSeek V4 Pro via Codex

replicate 2

fail

Evaluation status

Manual review pendingThe evaluator marked this answer incorrect. The answer, workflow summary, usage, and available run records are linked, but a manual root-cause audit has not yet been added.
Answer0.6382978723404256
Workflow
Local analysisSciPyvariant consequence filteringVAF filtering
Total tokens
4.85M
Output
30.9K
Run trace
DeepSeek V4 Pro via Codex

replicate 3

fail

Evaluation status

Manual review pendingThe evaluator marked this answer incorrect. The answer, workflow summary, usage, and available run records are linked, but a manual root-cause audit has not yet been added.
Answer0.6382978723404256
Workflow
Local analysisvariant consequence filteringmedian summarymean summary
Total tokens
4.23M
Output
24.5K
Run trace

DeepSeek V4 Pro via Claude CodeSuperseded

3 primary replicates
DeepSeek V4 Pro via Claude Code

replicate 1

pass
Answer0.7317073170731707
Workflow
Local analysisvariant consequence filtering
Total tokens
784.8K
Output
13.5K
Run trace
DeepSeek V4 Pro via Claude Code

replicate 2

pass
Answer0.7317073170731707
Workflow
Local analysisvariant consequence filtering
Total tokens
1.10M
Output
15.6K
Run trace
DeepSeek V4 Pro via Claude Code

replicate 3

pass
Answer0.731707
Workflow
Local analysisvariant consequence filtering
Total tokens
871.1K
Output
12.0K
Run trace

bix-14-q1 · October 2026 · three additional replicates

GPT-5.6 Sol · token optimization

Galaxy-API code, high reasoning, Fast service. Previous runs are retained above.

Open-ended code averaged 0.50M tokens; previous website Galaxy averaged 3.65M; the new Galaxy runs averaged 2.40M, or 4.81× Open-ended code. Each mean covers three replicates of this same task with GPT-5.6 Sol.

Cached input is already included in input tokens. Full nested tool/job evidence is in each HF run's galaxy_execution_records.tar.gz, with an accompanying member index.

Optimization overview and all 150 runs