bix-17-q2 | Current BixBench results

New token-optimization experiment

Three additional GPT-5.6 Sol Galaxy replicates are available separately. Previous results remain unchanged.

View the October 2026 batch and its run records

Question

Among patients classified as BLM mutation carriers, what is the median number of somatic CHIP variants - defined as variants with a Variant Allele Frequency (VAF) < 0.3? Prior to analysis, exclude all variants that are Intronic, Intergenic, in UTR regions, or have Reference zygosity.

Accepted answer: 2
Domain: variant callset

Comparison

Pass counts and token totals across the same three replicates.

Agent	Galaxy-API code	Open-ended code
Codex GPT-5.5	3/3 replicate 1 replicate 2 replicate 3 4.73M total	3/3 replicate 1 replicate 2 replicate 3 1.40M total
Codex GPT-5.6 Sol	3/3 replicate 1 replicate 2 replicate 3 36.31M total	3/3 replicate 1 replicate 2 replicate 3 1.07M total
Codex GPT-5.6 Luna	3/3 replicate 1 replicate 2 replicate 3 46.41M total	3/3 replicate 1 replicate 2 replicate 3 2.01M total
DeepSeek V4 Pro via Codex	3/3 replicate 1 replicate 2 replicate 3 53.35M total	3/3 replicate 1 replicate 2 replicate 3 4.52M total
DeepSeek V4 Pro via Claude Codesuperseded	3/3 replicate 1 replicate 2 replicate 3 16.54M total	3/3 replicate 1 replicate 2 replicate 3 3.01M total

Execution condition

Galaxy-API code with skills

Codex GPT-5.5

3 primary replicates

Codex GPT-5.5

replicate 1

passUDT-only

Answer2

Workflow

Galaxy UDTmedian summary

Total tokens: 1.65M
Output: 19.2K

Galaxy history Run trace

Codex GPT-5.5

replicate 2

passUDT-only

Answer2

Workflow

Galaxy UDT

Total tokens: 1.08M
Output: 12.0K

Galaxy history Run trace

Codex GPT-5.5

replicate 3

passUDT-only

Answer2

Workflow

Galaxy UDTBiopythonmedian summary

Total tokens: 2.00M
Output: 19.0K

Galaxy history Run trace

Codex GPT-5.6 Sol

3 primary replicates

Codex GPT-5.6 Sol

replicate 1

passUDT-only

Answer2

Workflow

Galaxy UDTmedian summary

Total tokens: 3.09M
Output: 18.5K

Galaxy history Run trace

Codex GPT-5.6 Sol

replicate 2

passUDT + standard tools

Answer2

Workflow

Excel to tabularGalaxy UDT

Total tokens: 30.17M
Output: 53.0K

Galaxy history Run trace

Codex GPT-5.6 Sol

replicate 3

passUDT-only

Answer2

Workflow

Galaxy UDTmedian summary

Total tokens: 3.06M
Output: 22.5K

Galaxy history Run trace

Codex GPT-5.6 Luna

3 primary replicates

Codex GPT-5.6 Luna

replicate 1

passUDT + standard tools

Answer2

Workflow

Excel to tabularGalaxy UDTmedian summary

Total tokens: 9.79M
Output: 37.8K

Galaxy history Run trace

Codex GPT-5.6 Luna

replicate 2

passUDT + standard tools

Answer2

Workflow

Excel to tabularAddvalueRemove headermore stepsDatamash summaryGroup and aggregate

Total tokens: 30.57M
Output: 72.5K

Galaxy history Run trace

Codex GPT-5.6 Luna

replicate 3

passUDT-only

Answer2

Workflow

Galaxy UDTmedian summary

Total tokens: 6.05M
Output: 33.2K

Galaxy history Run trace

DeepSeek V4 Pro via Codex

3 primary replicates

DeepSeek V4 Pro via Codex

replicate 1

pass

Answer2

Workflow

Galaxy history

Total tokens: 26.40M
Output: 76.4K

Galaxy history Run trace

DeepSeek V4 Pro via Codex

replicate 2

pass

Answer2

Workflow

Galaxy via BioBlendmedian summarymean summary

Total tokens: 8.92M
Output: 37.8K

Galaxy history Run trace

DeepSeek V4 Pro via Codex

replicate 3

pass

Answer2

Workflow

Galaxy via BioBlend

Total tokens: 18.02M
Output: 55.3K

Galaxy history Run trace

DeepSeek V4 Pro via Claude CodeSuperseded

3 primary replicates

DeepSeek V4 Pro via Claude Code

replicate 1

passUDT + standard tools

Answer2

Workflow

Excel to tabularmedian summary

Total tokens: 7.42M
Output: 81.6K

Galaxy history Run trace

DeepSeek V4 Pro via Claude Code

replicate 2

pass

Answer2

Workflow

Excel to tabularFilter rows

Total tokens: 4.27M
Output: 41.1K

Galaxy history Run trace

DeepSeek V4 Pro via Claude Code

replicate 3

passUDT + standard tools

Answer2

Workflow

Galaxy UDT

Total tokens: 4.85M
Output: 63.3K

Galaxy history Run trace

Execution condition

Open-ended code with skills

Codex GPT-5.5

3 primary replicates

Codex GPT-5.5

replicate 1

pass

Answer2

Workflow

Local analysismedian summary

Total tokens: 474.3K
Output: 5.0K

Run trace

Codex GPT-5.5

replicate 2

pass

Answer2

Workflow

Local analysisSciPymedian summary

Total tokens: 514.7K
Output: 6.8K

Run trace

Codex GPT-5.5

replicate 3

pass

Answer2

Workflow

Local analysisSciPymedian summary

Total tokens: 414.8K
Output: 7.6K

Run trace

Codex GPT-5.6 Sol

3 primary replicates

Codex GPT-5.6 Sol

replicate 1

pass

Answer2

Workflow

Local analysis

Total tokens: 347.0K
Output: 4.3K

Run trace

Codex GPT-5.6 Sol

replicate 2

pass

Answer2

Workflow

Local analysis

Total tokens: 308.0K
Output: 4.7K

Run trace

Codex GPT-5.6 Sol

replicate 3

pass

Answer2

Workflow

Local analysis

Total tokens: 412.1K
Output: 4.8K

Run trace

Codex GPT-5.6 Luna

3 primary replicates

Codex GPT-5.6 Luna

replicate 1

pass

Answer2

Workflow

Local analysis

Total tokens: 934.3K
Output: 13.3K

Run trace

Codex GPT-5.6 Luna

replicate 2

pass

Answer2

Workflow

Local analysis

Total tokens: 341.9K
Output: 7.4K

Run trace

Codex GPT-5.6 Luna

replicate 3

pass

Answer2

Workflow

Local analysis

Total tokens: 738.1K
Output: 12.1K

Run trace

DeepSeek V4 Pro via Codex

3 primary replicates

DeepSeek V4 Pro via Codex

replicate 1

pass

Answer2

Workflow

Local analysismedian summary

Total tokens: 1.89M
Output: 29.6K

Run trace

DeepSeek V4 Pro via Codex

replicate 2

pass

Answer2

Workflow

Local analysismedian summary

Total tokens: 1.80M
Output: 16.7K

Run trace

DeepSeek V4 Pro via Codex

replicate 3

pass

Answer2

Workflow

Local analysismedian summarymean summary

Total tokens: 830.5K
Output: 18.7K

Run trace

DeepSeek V4 Pro via Claude CodeSuperseded

3 primary replicates

DeepSeek V4 Pro via Claude Code

replicate 1

pass

Answer2

Workflow

Local analysismedian summarymean summary

Total tokens: 1.18M
Output: 16.2K

Run trace

DeepSeek V4 Pro via Claude Code

replicate 2

pass

Answer2

Workflow

Local analysismean summarymedian summary

Total tokens: 959.0K
Output: 17.1K

Run trace

DeepSeek V4 Pro via Claude Code

replicate 3

pass

Answer2

Workflow

Local analysismedian summarymean summary

Total tokens: 872.4K
Output: 17.8K

Run trace

bix-17-q2 · October 2026 · three additional replicates

GPT-5.6 Sol · token optimization

Galaxy-API code, high reasoning, Fast service. Previous runs are retained above.

Galaxy-API code · October 2026

replicate 1

Correct

Answer2

Total tokens: 1.33M
Input: 1.31M
Cached input: 1.24M
Output: 19.6K

Run trace Final answer Galaxy history

Galaxy-API code · October 2026

replicate 2

Correct

Answer2

Total tokens: 1.15M
Input: 1.14M
Cached input: 1.06M
Output: 12.1K

Run trace Final answer Galaxy history

Galaxy-API code · October 2026

replicate 3

Correct

Answer2

Total tokens: 1.78M
Input: 1.77M
Cached input: 1.70M
Output: 17.6K

Run trace Final answer Galaxy history

Open-ended code averaged 0.36M tokens; previous website Galaxy averaged 12.10M; the new Galaxy runs averaged 1.42M, or 4.00× Open-ended code. Each mean covers three replicates of this same task with GPT-5.6 Sol.

Cached input is already included in input tokens. Full nested tool/job evidence is in each HF run's galaxy_execution_records.tar.gz, with an accompanying member index.

Optimization overview and all 150 runs