Project record
How the project grew from initial Anycode runs into a reproducible comparison across models and execution environments.
Initial Anycode evaluation
CompBioBench began with three Codex GPT-5.5 Anycode runs. Their public leaderboard scores were 74%, 84%, and 83%, revealing substantial variation between repeated evaluations of the same model.
Trace retention
Each item was retained with its final answer and agent trace so disagreements could be reviewed rather than reduced to one aggregate score. Development submissions during this period reached 88%, 89%, and 91% on the public leaderboard.
Experimental standardization
The project fixed the 100-item task set, execution conditions, reasoning settings, answer format, and retained artifacts. This created comparable formal replicates rather than a sequence of one-off runs.
Anycode replicate matrix
Codex GPT-5.5, Codex GPT-5.6 Sol, and Codex GPT-5.6 Luna each completed three independent 100-item Anycode replicates under matched settings.
Galaxy protocol
Galaxy runs used a fresh assigned history for every item. Agents could use standard Galaxy tools and create user-defined tools when a required operation was unavailable, while the scientific task and input data remained unchanged.
DeepSeek-v4-pro-0813 expansion
Three Anycode and three Galaxy replicates extended the comparison to a fourth model configuration, using high reasoning effort and the same 100-item benchmark.
Verification and publication
Interrupted runs, answer files, traces, and Galaxy histories were reconciled before publication. Each published replicate contains 100 answers, and every answer links to a retained item trace.
August release
The release included 2,400 item traces and 1,200 unlisted, link-accessible Galaxy histories, including Luna Galaxy results and token totals.
Astra results
One Astra replicate adds 100 item answers and complete review directories. Sol Galaxy Replicate 3 scored 92% after an equivalent regulatory-element label was normalized for submission; the original agent trace is preserved.