Three models, two execution conditions, ten multi-step tasks
Each task is a full analysis, from raw data to final results, based on a reviewed Galaxy workflow from the Intergalactic Workflow Commission (IWC). The task text gives the scientific goal, the input data, the settings that change the result and the files to return. It names no software, so the agent has to recognize each method from its description and decide how to run it. Each model runs every task three times in each condition: through Galaxy only, or with any software in a container.
180 runs scored10 tasks3 replicates
Oct 19-21, 2026
Codex GPT-5.6 Luna
Galaxy-API code
0.495
BixBench3 acc (share of results that pass), averaged over 10 tasks (30 runs)
continuous score 0.618
All results pass0.067share of runs in which every scored result passes, averaged over the 10 tasks (2 of 30 runs)
replicate 10.49310 tasksreplicate 20.42810 tasksreplicate 30.56310 tasks
Open-ended code
0.339
BixBench3 acc (share of results that pass), averaged over 10 tasks (30 runs)
continuous score 0.418
All results pass0.067share of runs in which every scored result passes, averaged over the 10 tasks (2 of 30 runs)
replicate 10.34110 tasksreplicate 20.28710 tasksreplicate 30.38810 tasks
Oct 19-22, 2026
Codex GPT-5.6 Sol
Galaxy-API code
0.899
BixBench3 acc (share of results that pass), averaged over 10 tasks (30 runs)
continuous score 0.935
All results pass0.533share of runs in which every scored result passes, averaged over the 10 tasks (16 of 30 runs)
replicate 10.94210 tasksreplicate 20.91810 tasksreplicate 30.83810 tasks
Open-ended code
0.880
BixBench3 acc (share of results that pass), averaged over 10 tasks (30 runs)
continuous score 0.927
All results pass0.600share of runs in which every scored result passes, averaged over the 10 tasks (18 of 30 runs)
replicate 10.85910 tasksreplicate 20.88810 tasksreplicate 30.89110 tasks
Oct 19-26, 2026
DeepSeek V4 Pro via Codex
Galaxy-API code
0.631
BixBench3 acc (share of results that pass), averaged over 10 tasks (30 runs)
continuous score 0.730
All results pass0.133share of runs in which every scored result passes, averaged over the 10 tasks (4 of 30 runs)
replicate 10.65610 tasksreplicate 20.68210 tasksreplicate 30.55410 tasks
Open-ended code
0.543
BixBench3 acc (share of results that pass), averaged over 10 tasks (30 runs)
continuous score 0.616
All results pass0.033share of runs in which every scored result passes, averaged over the 10 tasks (1 of 30 runs)
replicate 10.52210 tasksreplicate 20.56710 tasksreplicate 30.54110 tasks
Three measures. Every required result gets a score from 0 to 1 by comparison with the reference. Passing requires a score of at least 0.80 and compliance with the required content, method and output instructions. BixBench3 acc is the share of results that pass. The continuous score is the plain average of all the result scores, with no pass mark. For example, run 1 of GPT-5.6 Luna with Galaxy-API code on task 1 scored 1.000, 1.000, 0.000, 1.000, 1.000, 1.000, 1.000 and 0.413 on its 8 results. 6 of them pass, so BixBench3 acc = 6/8 = 0.750; the average of all 8 scores is 0.802, the continuous score. A third measure, all results pass, counts a run only when every scored result passes, which is the share of runs that delivered the whole analysis correctly. A failing result still contributes its numerical score to the continuous average. A score above 0.80 does not excuse an incorrect method, missing member or an explicitly required output format. Each card averages acc and the continuous score over the runs of each task and then over tasks, so every task counts equally; a replicate averages run 1, 2 or 3 over the tasks.
Results by task
Each cell shows BixBench3 acc (the share of results that pass) averaged over the scored runs of one task, model and condition, then the continuous score and how many of the three runs are scored. Green cells have acc of 0.80 or above, amber cells 0.50 to 0.80. Open a task for its full text, how each result is scored, and the score of every result in every run.
| Task | GPT-5.6 Luna Galaxy-API code | GPT-5.6 Luna Open-ended code | GPT-5.6 Sol Galaxy-API code | GPT-5.6 Sol Open-ended code | DeepSeek V4 Pro Galaxy-API code | DeepSeek V4 Pro Open-ended code |
|---|---|---|---|---|---|---|
| 1. Genome profiling and HiFi assembly | 0.583continuous score 0.635 all results pass in 0 of 3 3 of 3 runs | 0.167continuous score 0.190 all results pass in 0 of 3 3 of 3 runs | 1.000continuous score 1.000 all results pass in 3 of 3 3 of 3 runs | 1.000continuous score 0.981 all results pass in 3 of 3 3 of 3 runs | 0.458continuous score 0.489 all results pass in 0 of 3 3 of 3 runs | 0.542continuous score 0.554 all results pass in 0 of 3 3 of 3 runs |
| 2. ATAC-seq peaks and normalized signal | 0.708continuous score 0.849 all results pass in 0 of 3 3 of 3 runs | 0.667continuous score 0.676 all results pass in 0 of 3 3 of 3 runs | 0.917continuous score 0.925 all results pass in 1 of 3 3 of 3 runs | 0.958continuous score 0.957 all results pass in 2 of 3 3 of 3 runs | 0.542continuous score 0.641 all results pass in 0 of 3 3 of 3 runs | 0.833continuous score 0.881 all results pass in 0 of 3 3 of 3 runs |
| 3. 16S amplicon sequence variants and taxonomy | 0.292continuous score 0.418 all results pass in 0 of 3 3 of 3 runs | 0.958continuous score 0.942 all results pass in 2 of 3 3 of 3 runs | 0.708continuous score 0.834 all results pass in 1 of 3 3 of 3 runs | 1.000continuous score 0.989 all results pass in 3 of 3 3 of 3 runs | 0.792continuous score 0.852 all results pass in 0 of 3 3 of 3 runs | 0.750continuous score 0.747 all results pass in 0 of 3 3 of 3 runs |
| 4. Paired RNA-seq effects and donor robustness | 0.300continuous score 0.425 all results pass in 0 of 3 3 of 3 runs | 0.333continuous score 0.594 all results pass in 0 of 3 3 of 3 runs | 0.933continuous score 0.931 all results pass in 2 of 3 3 of 3 runs | 0.800continuous score 0.880 all results pass in 0 of 3 3 of 3 runs | 0.600continuous score 0.878 all results pass in 0 of 3 3 of 3 runs | 0.600continuous score 0.849 all results pass in 0 of 3 3 of 3 runs |
| 5. Single-cell populations and marker genes | 0.528continuous score 0.576 all results pass in 1 of 3 3 of 3 runs | 0.639continuous score 0.804 all results pass in 0 of 3 3 of 3 runs | 0.972continuous score 0.977 all results pass in 2 of 3 3 of 3 runs | 0.944continuous score 0.968 all results pass in 1 of 3 3 of 3 runs | 0.611continuous score 0.758 all results pass in 0 of 3 3 of 3 runs | 0.833continuous score 0.925 all results pass in 0 of 3 3 of 3 runs |
| 6. Replicate-supported CUT&RUN consensus | 0.500continuous score 0.516 all results pass in 0 of 3 3 of 3 runs | 0.208continuous score 0.339 all results pass in 0 of 3 3 of 3 runs | 1.000continuous score 0.999 all results pass in 3 of 3 3 of 3 runs | 0.958continuous score 0.987 all results pass in 2 of 3 3 of 3 runs | 0.708continuous score 0.778 all results pass in 1 of 3 3 of 3 runs | 0.917continuous score 0.915 all results pass in 1 of 3 3 of 3 runs |
| 7. Carbon-isotope incorporation in a metaproteome | 0.708continuous score 0.790 all results pass in 0 of 3 3 of 3 runs | 0.125continuous score 0.125 all results pass in 0 of 3 3 of 3 runs | 0.875continuous score 0.863 all results pass in 0 of 3 3 of 3 runs | 1.000continuous score 0.989 all results pass in 3 of 3 3 of 3 runs | 0.625continuous score 0.753 all results pass in 0 of 3 3 of 3 runs | 0.417continuous score 0.533 all results pass in 0 of 3 3 of 3 runs |
| 8. Small-variant differences among haploid yeast strains | 0.167continuous score 0.694 all results pass in 0 of 3 3 of 3 runs | 0.167continuous score 0.384 all results pass in 0 of 3 3 of 3 runs | 0.917continuous score 0.909 all results pass in 1 of 3 3 of 3 runs | 0.917continuous score 0.912 all results pass in 2 of 3 3 of 3 runs | 0.833continuous score 0.918 all results pass in 1 of 3 3 of 3 runs | 0.417continuous score 0.615 all results pass in 0 of 3 3 of 3 runs |
| 9. Multiplex tissue imaging and cell neighborhoods | 0.704continuous score 0.740 all results pass in 1 of 3 3 of 3 runs | 0.000continuous score 0.000 all results pass in 0 of 3 3 of 3 runs | 1.000continuous score 1.000 all results pass in 3 of 3 3 of 3 runs | 0.259continuous score 0.625 all results pass in 0 of 3 3 of 3 runs | 0.889continuous score 0.936 all results pass in 2 of 3 3 of 3 runs | 0.000continuous score 0.002 all results pass in 0 of 3 3 of 3 runs |
| 10. LC-MS features and ion relationships | 0.458continuous score 0.539 all results pass in 0 of 3 3 of 3 runs | 0.125continuous score 0.127 all results pass in 0 of 3 3 of 3 runs | 0.667continuous score 0.917 all results pass in 0 of 3 3 of 3 runs | 0.958continuous score 0.978 all results pass in 2 of 3 3 of 3 runs | 0.250continuous score 0.292 all results pass in 0 of 3 3 of 3 runs | 0.125continuous score 0.140 all results pass in 0 of 3 3 of 3 runs |
Token use in the two conditions
Tokens per run are the input tokens, including cached input, plus the output tokens, averaged over the scored runs of a task. Each row compares the same task and model in the two conditions; a ratio above 1 means the Galaxy runs used more tokens. Tasks appear only where both conditions have scored runs.
GPT-5.6 Luna
| Task | Open-ended code tokens | Galaxy-API code tokens | Galaxy ÷ Open-ended | Open-ended BixBench3 acc | Galaxy BixBench3 acc |
|---|---|---|---|---|---|
| 1. Genome profiling and HiFi assembly | 53.0M | 15.0M | 0.28 | 0.167 | 0.583 |
| 2. ATAC-seq peaks and normalized signal | 14.3M | 16.7M | 1.16 | 0.667 | 0.708 |
| 3. 16S amplicon sequence variants and taxonomy | 6.3M | 35.1M | 5.56 | 0.958 | 0.292 |
| 4. Paired RNA-seq effects and donor robustness | 16.5M | 28.4M | 1.72 | 0.333 | 0.300 |
| 5. Single-cell populations and marker genes | 1.3M | 17.0M | 13.28 | 0.639 | 0.528 |
| 6. Replicate-supported CUT&RUN consensus | 39.3M | 27.7M | 0.71 | 0.208 | 0.500 |
| 7. Carbon-isotope incorporation in a metaproteome | 12.8M | 2.6M | 0.20 | 0.125 | 0.708 |
| 8. Small-variant differences among haploid yeast strains | 12.8M | 13.3M | 1.04 | 0.167 | 0.167 |
| 9. Multiplex tissue imaging and cell neighborhoods | 6.1M | 29.4M | 4.78 | 0.000 | 0.704 |
| 10. LC-MS features and ion relationships | 4.4M | 21.8M | 4.90 | 0.125 | 0.458 |
| All tasks above | 166.8M | 206.9M | 1.24 |
GPT-5.6 Sol
| Task | Open-ended code tokens | Galaxy-API code tokens | Galaxy ÷ Open-ended | Open-ended BixBench3 acc | Galaxy BixBench3 acc |
|---|---|---|---|---|---|
| 1. Genome profiling and HiFi assembly | 14.1M | 10.6M | 0.75 | 1.000 | 1.000 |
| 2. ATAC-seq peaks and normalized signal | 6.0M | 12.5M | 2.07 | 0.958 | 0.917 |
| 3. 16S amplicon sequence variants and taxonomy | 8.0M | 18.1M | 2.27 | 1.000 | 0.708 |
| 4. Paired RNA-seq effects and donor robustness | 5.2M | 15.9M | 3.06 | 0.800 | 0.933 |
| 5. Single-cell populations and marker genes | 1.7M | 9.7M | 5.81 | 0.944 | 0.972 |
| 6. Replicate-supported CUT&RUN consensus | 18.1M | 20.2M | 1.12 | 0.958 | 1.000 |
| 7. Carbon-isotope incorporation in a metaproteome | 6.4M | 3.3M | 0.51 | 1.000 | 0.875 |
| 8. Small-variant differences among haploid yeast strains | 6.4M | 10.1M | 1.58 | 0.917 | 0.917 |
| 9. Multiplex tissue imaging and cell neighborhoods | 13.9M | 9.7M | 0.70 | 0.259 | 1.000 |
| 10. LC-MS features and ion relationships | 3.1M | 11.6M | 3.78 | 0.958 | 0.667 |
| All tasks above | 82.7M | 121.6M | 1.47 |
DeepSeek V4 Pro
| Task | Open-ended code tokens | Galaxy-API code tokens | Galaxy ÷ Open-ended | Open-ended BixBench3 acc | Galaxy BixBench3 acc |
|---|---|---|---|---|---|
| 1. Genome profiling and HiFi assembly | 31.5M | 13.7M | 0.44 | 0.542 | 0.458 |
| 2. ATAC-seq peaks and normalized signal | 13.5M | 17.7M | 1.31 | 0.833 | 0.542 |
| 3. 16S amplicon sequence variants and taxonomy | 6.1M | 25.9M | 4.25 | 0.750 | 0.792 |
| 4. Paired RNA-seq effects and donor robustness | 8.6M | 27.7M | 3.22 | 0.600 | 0.600 |
| 5. Single-cell populations and marker genes | 6.1M | 12.6M | 2.05 | 0.833 | 0.611 |
| 6. Replicate-supported CUT&RUN consensus | 10.2M | 33.4M | 3.26 | 0.917 | 0.708 |
| 7. Carbon-isotope incorporation in a metaproteome | 10.7M | 9.1M | 0.85 | 0.417 | 0.625 |
| 8. Small-variant differences among haploid yeast strains | 11.1M | 20.5M | 1.85 | 0.417 | 0.833 |
| 9. Multiplex tissue imaging and cell neighborhoods | 11.7M | 14.7M | 1.25 | 0.000 | 0.889 |
| 10. LC-MS features and ion relationships | 12.8M | 43.6M | 3.42 | 0.125 | 0.250 |
| All tasks above | 122.4M | 218.9M | 1.79 |
How results are scored
Each task asks for a fixed list of results, and each result gets a score from 0 to 1 against a reference. The same rules apply to every run. Each task page lists the checks used for each of its results.
- Reference. Made by running the task's IWC workflow on usegalaxy.org with the task's settings. Scoring only reads the submitted files.
- Checks. A result's score is the product of its checks, each from 0 to 1. The checks come from
BixBench3:
- same rows as the reference, such as genes, reads, peaks, variants or cells (F1)
- required columns present (F1)
- values agree on matching rows (Lin's concordance correlation; counts on a log scale)
- category labels agree (macro F1)
- summary statistics within 1% of the reference
- alignments, signal tracks and images match record by record, base by base or pixel by pixel
- One copy per sample. When a result has one file per sample, each file is scored on its own and the result's score is their average. Each required member must also meet its checks. A missing or unreadable file makes the result 0. The same applies per model fit, strain, imaging cycle or tissue core.
- BixBench3 acc. A result must score at least 0.80 and meet the required member checks, method and output instructions. acc is the share of results that pass, for example 3/8 = 0.375.
- Continuous score. The average of the result scores, with no pass mark.
- All results pass. The share of runs in which every scored result passes. A multi-step analysis is usable only when all of its results are right, so this is the strictest of the three measures.
- Output instructions. Follow the named files, fields and encodings. Alternatives are accepted where the task permits them. In Galaxy, an allowed native dataset holding the required content also satisfies the requirement. Each failing result links to its reason, including format and instruction errors.
- Not scored. Quality plots, reports that repeat scored values, and written interpretations are required but not scored. Each task page lists them.
Conditions and runs
Both conditions receive the same task text. Only the instructions about where to run the work differ.
- Galaxy-API code
- The agent must run every analysis step as a Galaxy job on usegalaxy.org, in a private history that already holds the input data. It works from a container with 4 CPU cores and 8 GB of memory. Local code may move files, inspect results and organize tables, but may not carry out the analysis; Galaxy's own text and table tools can connect the steps.
- Open-ended code
- The agent works in an isolated container with 4 CPU cores, 64 GB of memory and internet access, and may install and run any software.
- Agent and models
- The Codex command-line agent, version 0.156.0, with reasoning effort set to high, running GPT-5.6 Luna, GPT-5.6 Sol or DeepSeek V4 Pro.
- Runs
- Three runs per task, model and condition. For tasks 1, 2, 4, 5, 6, 7, 9, 10, the last trial run with GPT-5.6 Luna counts as run 1 in both conditions, because the task text did not change after it. Runs interrupted by an account usage limit were repeated from scratch. A run that ended after a progress message without returning any results was repeated once. In both cases only the repeat counts.
General submission rules given with every task
- Each numbered path is a fixed deliverable name under final_answer/. A trailing slash denotes a bundle of the named files. Replace only placeholders appearing in this item with the identities defined by its input manifest or task; comma-separated braces list all required members.
- Submit every listed scientific result. Retain legitimate empty results, missing values and unresolved assignments.
- Include final_answer/method.md: software versions, chosen parameters, units/coordinates, sample identities, and a mapping from the required names to the actual outputs. Include any brief interpretation requested by the question unless a separate deliverable is named. Keep this concise; no extra scientific analysis or packaging program is required.
- Analyze all supplied samples and preserve their identities. Do not introduce additional subsampling.
- Follow the scientific method requirements and key parameters stated in the task. Report implementation choices not specified in the task.
Reading these results
The references were made with the same Galaxy tools that Galaxy runs can use. A Galaxy run that recognizes the method and applies the stated settings can match a reference exactly. An Open-ended code run has to find, install and run an equivalent implementation, and differences between implementations lower its score even when the analysis is sound. On tasks 3 and 5, Galaxy runs used 2.1 to 13.3 times as many tokens as Open-ended code runs with the same model, and they did not score consistently higher.
Data updated 27 October 2026, 08:48 EDT.