Back to dashboard

BixBench July 6 Codex experiment

Tokens Usage

Condition totals

Total tokens are computed as input tokens plus output tokens from each replicate's usage record. Percent changes are relative to Galaxy-API code with skills.

Galaxy-API code with skills 446.7M 150 replicates · base vs Galaxy-API code with skills
Input
443.4M base
Cached input
423.4M base
Uncached input
20.0M base
Output
3.3M base
Reasoning output
813.1K base
Galaxy-API code without skills 502.8M 150 replicates · +12.6% vs Galaxy-API code with skills
Input
499.6M +12.7%
Cached input
481.2M +13.7%
Uncached input
18.4M -8.4%
Output
3.3M -2.2%
Reasoning output
855.0K +5.1%
Open-ended code with skills 81.5M 150 replicates · -81.8% vs Galaxy-API code with skills
Input
80.4M -81.9%
Cached input
72.4M -82.9%
Uncached input
8.1M -59.8%
Output
1.1M -66.8%
Reasoning output
329.0K -59.5%
Open-ended code without skills 68.2M 150 replicates · -84.7% vs Galaxy-API code with skills
Input
67.3M -84.8%
Cached input
60.0M -85.8%
Uncached input
7.3M -63.7%
Output
941.0K -71.9%
Reasoning output
272.9K -66.4%

Why token use differs

The differences mostly come from platform context, not from answer length.

Galaxy conditions are much larger than anycode

Galaxy-API code includes Galaxy API interaction, history inspection, job/tool evidence, UDT or tool-selection context, and longer trace reasoning. Open-ended code does local file/code work and avoids that Galaxy evidence chain, so both anycode conditions use more than 80% fewer total tokens than Galaxy-API code with skills.

Galaxy without skills is higher than the base

Galaxy-API code without skills uses 12.3% more total tokens than Galaxy-API code with skills. Almost all of that increase is input context: cached input is 13.3% higher, while uncached input is 7.6% lower and output is essentially flat. The with-skills condition includes `galaxy-token-efficient-operations`, which tells the agent to save bulky Galaxy evidence to trace files and keep transcript output compact. Without that token-saving Galaxy skill, the agent repeats more Galaxy/tool/history context in the conversation.

Open-ended code with skills is still far below the base

Open-ended code with skills uses 81.4% fewer total tokens than the Galaxy base because it does not carry Galaxy histories, API schemas, or tool/job evidence. It is higher than anycode without skills because skills add setup guidance and domain checks, which increase input and reasoning tokens even without Galaxy.

Open-ended code without skills is the smallest

Open-ended code without skills has the lowest token usage: 84.4% below the Galaxy base. It has the shortest context stack: no Galaxy context and no skill instructions. That saves tokens, but the accuracy page shows it also has the lowest accuracy among the four conditions.

Replicate-level usage

Each replicate column is compared to the same replicate in Galaxy-API code with skills.

Condition Total replicate 1 replicate 2 replicate 3 Cached input Uncached input Output Reasoning output
Galaxy-API code with skills 446.7M base 144.7M base 50 items 142.8M base 50 items 159.2M base 50 items 423.4M base 20.0M base 3.3M base 813.1K base
Galaxy-API code without skills 502.8M +12.6% 177.0M +22.3% 50 items 179.3M +25.5% 50 items 146.6M -7.9% 50 items 481.2M +13.7% 18.4M -8.4% 3.3M -2.2% 855.0K +5.1%
Open-ended code with skills 81.5M -81.8% 24.7M -83.0% 50 items 28.2M -80.2% 50 items 28.6M -82.0% 50 items 72.4M -82.9% 8.1M -59.8% 1.1M -66.8% 329.0K -59.5%
Open-ended code without skills 68.2M -84.7% 23.4M -83.8% 50 items 21.1M -85.2% 50 items 23.7M -85.1% 50 items 60.0M -85.8% 7.3M -63.7% 941.0K -71.9% 272.9K -66.4%