bix-20-q3 | Current BixBench results

New token-optimization experiment

Three additional GPT-5.6 Sol Galaxy replicates are available separately. Previous results remain unchanged.

View the October 2026 batch and its run records

Question

Among samples with a 'Carrier' BLM mutation status, what proportion of somatic CHIP variants (VAF < 0.3) with ClinVar classifications are classified as Benign or Likely Benign? The proportion of somatic CHIP variants should be calculated after filtering out intronic, intergenic, UTR regions, and reference (non-variant) calls.

Accepted answer: (0.80, 1.00)
Domain: variant callset

Comparison

Pass counts and token totals across the same three replicates.

Agent	Galaxy-API code	Open-ended code
Codex GPT-5.5	3/3 replicate 1 replicate 2 replicate 3 4.93M total	3/3 replicate 1 replicate 2 replicate 3 1.26M total
Codex GPT-5.6 Sol	3/3 replicate 1 replicate 2 replicate 3 10.62M total	3/3 replicate 1 replicate 2 replicate 3 1.16M total
Codex GPT-5.6 Luna	3/3 replicate 1 replicate 2 replicate 3 41.18M total	3/3 replicate 1 replicate 2 replicate 3 2.89M total
DeepSeek V4 Pro via Codex	3/3 replicate 1 replicate 2 replicate 3 41.15M total	3/3 replicate 1 replicate 2 replicate 3 5.45M total
DeepSeek V4 Pro via Claude Codesuperseded	3/3 replicate 1 replicate 2 replicate 3 9.93M total	3/3 replicate 1 replicate 2 replicate 3 2.61M total

Execution condition

Galaxy-API code with skills

Codex GPT-5.5

3 primary replicates

Codex GPT-5.5

replicate 1

passUDT-only

Answer1

Workflow

Galaxy UDTClinVar filtering

Total tokens: 1.39M
Output: 20.9K

Galaxy history Run trace

Codex GPT-5.5

replicate 2

passUDT-only

Answer1

Workflow

Galaxy UDT

Total tokens: 2.10M
Output: 29.2K

Galaxy history Run trace

Codex GPT-5.5

replicate 3

passUDT-only

Answer1

Workflow

Galaxy UDT

Total tokens: 1.44M
Output: 22.1K

Galaxy history Run trace

Codex GPT-5.6 Sol

3 primary replicates

Codex GPT-5.6 Sol

replicate 1

pass

Answer1.0

Workflow

Excel to tabularFilter TabularFilter rowsConcatenate datasetsWc GnuClinVar filtering

Total tokens: 4.96M
Output: 19.9K

Galaxy history Run trace

Codex GPT-5.6 Sol

replicate 2

passUDT-only

Answer1.0

Workflow

Galaxy UDTClinVar filtering

Total tokens: 3.79M
Output: 18.6K

Galaxy history Run trace

Codex GPT-5.6 Sol

replicate 3

passUDT-only

Answer1

Workflow

Galaxy UDTClinVar filtering

Total tokens: 1.88M
Output: 14.4K

Galaxy history Run trace

Codex GPT-5.6 Luna

3 primary replicates

Codex GPT-5.6 Luna

replicate 1

pass

Answer1

Workflow

Excel to tabularConcatenate datasetsCat Multiplemore stepsCalculate Numeric ParamClinVar filtering

Total tokens: 21.49M
Output: 46.1K

Galaxy history Run trace

Codex GPT-5.6 Luna

replicate 2

passUDT-only

Answer1

Workflow

Galaxy UDTClinVar filtering

Total tokens: 5.46M
Output: 39.7K

Galaxy history Run trace

Codex GPT-5.6 Luna

replicate 3

pass

Answer1

Workflow

Excel to tabularFilter TabularSelect columnsmore stepsWc GnuClinVar filtering

Total tokens: 14.23M
Output: 41.2K

Galaxy history Run trace

DeepSeek V4 Pro via Codex

3 primary replicates

DeepSeek V4 Pro via Codex

replicate 1

pass

Answer1.0

Workflow

Galaxy via BioBlendClinVar filtering

Total tokens: 6.48M
Output: 41.1K

Galaxy history Run trace

DeepSeek V4 Pro via Codex

replicate 2

pass

Answer1

Workflow

Galaxy via BioBlend

Total tokens: 17.19M
Output: 43.7K

Galaxy history Run trace

DeepSeek V4 Pro via Codex

replicate 3

pass

Answer1.0

Workflow

Galaxy via BioBlend

Total tokens: 17.48M
Output: 42.6K

Galaxy history Run trace

DeepSeek V4 Pro via Claude CodeSuperseded

3 primary replicates

DeepSeek V4 Pro via Claude Code

replicate 1

pass

Answer1

Workflow

Excel to tabularConcatenate datasetsClinVar filtering

Total tokens: 2.26M
Output: 27.5K

Galaxy history Run trace

DeepSeek V4 Pro via Claude Code

replicate 2

pass

Answer1

Workflow

Excel to tabularConcatenate datasetsFilter TabularGroup and aggregateClinVar filtering

Total tokens: 4.92M
Output: 43.8K

Galaxy history Run trace

DeepSeek V4 Pro via Claude Code

replicate 3

pass

Answer1.0

Workflow

Excel to tabularClinVar filtering

Total tokens: 2.76M
Output: 31.6K

Galaxy history Run trace

Execution condition

Open-ended code with skills

Codex GPT-5.5

3 primary replicates

Codex GPT-5.5

replicate 1

pass

Answer1.0

Workflow

Local analysisClinVar filteringvariant consequence filtering

Total tokens: 505.2K
Output: 9.3K

Run trace

Codex GPT-5.5

replicate 2

pass

Answer1.0

Workflow

Local analysis

Total tokens: 432.0K
Output: 5.9K

Run trace

Codex GPT-5.5

replicate 3

pass

Answer1

Workflow

Local analysisClinVar filtering

Total tokens: 321.4K
Output: 6.4K

Run trace

Codex GPT-5.6 Sol

3 primary replicates

Codex GPT-5.6 Sol

replicate 1

pass

Answer1.0

Workflow

Local analysis

Total tokens: 359.3K
Output: 6.2K

Run trace

Codex GPT-5.6 Sol

replicate 2

pass

Answer1.0

Workflow

Local analysis

Total tokens: 394.0K
Output: 4.7K

Run trace

Codex GPT-5.6 Sol

replicate 3

pass

Answer1

Workflow

Local analysis

Total tokens: 408.7K
Output: 6.0K

Run trace

Codex GPT-5.6 Luna

3 primary replicates

Codex GPT-5.6 Luna

replicate 1

pass

Answer1

Workflow

Local analysis

Total tokens: 1.78M
Output: 18.0K

Run trace

Codex GPT-5.6 Luna

replicate 2

pass

Answer1

Workflow

Local analysis

Total tokens: 714.9K
Output: 13.7K

Run trace

Codex GPT-5.6 Luna

replicate 3

pass

Answer1

Workflow

Local analysis

Total tokens: 393.7K
Output: 11.5K

Run trace

DeepSeek V4 Pro via Codex

3 primary replicates

DeepSeek V4 Pro via Codex

replicate 1

pass

Answer1.0

Workflow

Local analysisClinVar filtering

Total tokens: 1.53M
Output: 23.8K

Run trace

DeepSeek V4 Pro via Codex

replicate 2

pass

Answer1.0

Workflow

Local analysisClinVar filtering

Total tokens: 970.1K
Output: 20.8K

Run trace

DeepSeek V4 Pro via Codex

replicate 3

pass

Answer1.0

Workflow

Local analysisvariant consequence filteringClinVar filtering

Total tokens: 2.95M
Output: 29.5K

Run trace

DeepSeek V4 Pro via Claude CodeSuperseded

3 primary replicates

DeepSeek V4 Pro via Claude Code

replicate 1

pass

Answer1.0

Workflow

Local analysisClinVar filtering

Total tokens: 1.05M
Output: 16.9K

Run trace

DeepSeek V4 Pro via Claude Code

replicate 2

pass

Answer1

Workflow

Local analysisClinVar filtering

Total tokens: 799.4K
Output: 14.3K

Run trace

DeepSeek V4 Pro via Claude Code

replicate 3

pass

Answer1

Workflow

Local analysisClinVar filtering

Total tokens: 759.6K
Output: 13.3K

Run trace

bix-20-q3 · October 2026 · three additional replicates

GPT-5.6 Sol · token optimization

Galaxy-API code, high reasoning, Fast service. Previous runs are retained above.

Galaxy-API code · October 2026

replicate 1

Correct

Answer1

Total tokens: 1.19M
Input: 1.17M
Cached input: 1.10M
Output: 14.5K

Run trace Final answer Galaxy history

Galaxy-API code · October 2026

replicate 2

Correct

Answer1

Total tokens: 1.87M
Input: 1.85M
Cached input: 1.76M
Output: 21.4K

Run trace Final answer Galaxy history

Galaxy-API code · October 2026

replicate 3

Correct

Answer1

Total tokens: 1.67M
Input: 1.65M
Cached input: 1.59M
Output: 14.1K

Run trace Final answer Galaxy history

Open-ended code averaged 0.39M tokens; previous website Galaxy averaged 3.54M; the new Galaxy runs averaged 1.57M, or 4.06× Open-ended code. Each mean covers three replicates of this same task with GPT-5.6 Sol.

Cached input is already included in input tokens. Full nested tool/job evidence is in each HF run's galaxy_execution_records.tar.gz, with an accompanying member index.

Optimization overview and all 150 runs