Back to results

CompBioBench item

saluki-setup-optimize-q1

Question

Set up the Saluki model detailed in https://doi.org/10.1186/s13059-022-02811-x and use it to optimize the 3' UTR of the mRNA sequence GCCGCCACCATGGTGAGCAAGGGCGAGTAGTGTACATAATAAGGACT. Specifically, perform 3 rounds of directed evolution, trying every possible substitution in the 3' UTR and choosing the best after each round. Always use the average of predictions across all folds and use only the human output. Make sure to use GPUs if available. Report the full sequence of the optimized mRNA.

Inferred answer
GCCGCCACCATGGTGAGCAAGGGCGAGTAGTATAAATAATAAGGGCTUniquely inferred from submitted scores.
Domain
Machine Learning
Question style
Tooling

An answer can be uniquely inferred for 54 of 100 questions from submitted scores. Green: correct · Red: incorrect. For questions without a uniquely determined answer: Blue: matches prediction · Orange: differs from prediction. These are inferred answer checks, not official per-question grades.

Galaxy

12/12 terminal item results

UDT = user-defined tool: custom code executed by Galaxy. Route badges summarize analysis tools, excluding uploads and automatic conversions. How usage is counted.

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

CorrectUDT-only
AnswerGCCGCCACCATGGTGAGCAAGGGCGAGTAGTATAAATAATAAGGGCT
Workflow
Saluki directed evolution human 3UTR
Codex GPT-5.5

Replicate 2

CorrectUDT-only
AnswerGCCGCCACCATGGTGAGCAAGGGCGAGTAGTATAAATAATAAGGGCT
Workflow
Saluki 3UTR Directed Evolution
Total tokens
1.9M
Output
12.3K
Codex GPT-5.5

Replicate 3

RerunCorrectUDT-only
AnswerGCCGCCACCATGGTGAGCAAGGGCGAGTAGTATAAATAATAAGGGCT
Workflow
Saluki human directed evolutionSaluki human directed evolution CPUSaluki human directed evolution 64GB
Total tokens
3.4M
Output
17.6K

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

CorrectUDT-only
AnswerGCCGCCACCATGGTGAGCAAGGGCGAGTAGTATAAATAATAAGGGCT
Workflow
Saluki directed evolution
Total tokens
3.6M
Output
11.9K
Codex GPT-5.6 Sol

Replicate 2

CorrectUDT-only
AnswerGCCGCCACCATGGTGAGCAAGGGCGAGTAGTATAAATAATAAGGGCT
Workflow
Saluki mRNA boundary diagnosticSaluki TensorFlow 2.6 GPU runtime probeSaluki published model restore probeSaluki three-round 3-prime UTR directed evolution
Total tokens
8.3M
Output
22.4K
Codex GPT-5.6 Sol

Replicate 3

CorrectUDT-only
AnswerGCCGCCACCATGGTGAGCAAGGGCGAGTAGTATAAATAATAAGGGCT
Workflow
Saluki checkpoint and annotation diagnostic (GPU preferred)Saluki checkpoint and annotation diagnosticSaluki 3-prime UTR directed evolution (3 rounds, all folds, human)
Total tokens
5.6M
Output
20.9K

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

CorrectUDT-only
AnswerGCCGCCACCATGGTGAGCAAGGGCGAGTAGTATAAATAATAAGGGCT
Workflow
No Galaxy tool job recorded
Total tokens
64.7M
Output
124.4K
Codex + DeepSeek-v4-pro-0813

Replicate 2

RerunCorrectUDT-only
AnswerGCCGCCACCATGGTGAGCAAGGGCGAGTAGTATAAATAATAAGGGCT
Workflow
Saluki ProbeSaluki 3-prime UTR Directed Evolution
Total tokens
10.1M
Output
49.2K
Codex + DeepSeek-v4-pro-0813

Replicate 3

CorrectUDT-only
AnswerGCCGCCACCATGGTGAGCAAGGGCGAGTAGTATAAATAATAAGGGCT
Workflow
Saluki TensorFlow ProbeSaluki TensorFlow Probe CPUSaluki TF Probe MinCat User Defined TestGalaxy UDT1 more tools
Total tokens
26.4M
Output
58.9K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

CorrectUDT + standard tools
AnswerGCCGCCACCATGGTGAGCAAGGGCGAGTAGTATAAATAATAAGGGCT
Workflow
Saluki diagnosticSaluki GPU diagnosticSaluki CPU diagnosticCompose Text ParamSaluki minimal diagnostic1 more tools
Total tokens
19.5M
Output
68.8K
Codex GPT-5.6 Luna

Replicate 2

CorrectUDT + standard tools
AnswerGCCGCCACCATGGTGAGCAAGGGCGAGTAGTATAAATAATAAGGGCT
Workflow
Saluki runtime probeSaluki Torch runtime probeSaluki TensorFlow runtime probeSaluki UDT minimal probeSaluki UDT command probe8 more tools
Total tokens
54.5M
Output
172.5K
Codex GPT-5.6 Luna

Replicate 3

RerunCorrectUDT-only
AnswerGCCGCCACCATGGTGAGCAAGGGCGAGTAGTATAAATAATAAGGGCT
Workflow
User-defined tool
Total tokens
18.8M
Output
40.6K

Anycode

13/13 terminal item results

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

Correct
AnswerGCCGCCACCATGGTGAGCAAGGGCGAGTAGTATAAATAATAAGGGCT
Workflow
Local codePython
Codex GPT-5.5

Replicate 2

Correct
AnswerGCCGCCACCATGGTGAGCAAGGGCGAGTAGTATAAATAATAAGGGCT
Workflow
Local codePython
Codex GPT-5.5

Replicate 3

Correct
AnswerGCCGCCACCATGGTGAGCAAGGGCGAGTAGTATAAATAATAAGGGCT
Workflow
Local codePythonSciPy

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

Correct
AnswerGCCGCCACCATGGTGAGCAAGGGCGAGTAGTATAAATAATAAGGGCT
Workflow
Local codePython
Codex GPT-5.6 Sol

Replicate 2

Correct
AnswerGCCGCCACCATGGTGAGCAAGGGCGAGTAGTATAAATAATAAGGGCT
Workflow
Local codePython
Codex GPT-5.6 Sol

Replicate 3

Correct
AnswerGCCGCCACCATGGTGAGCAAGGGCGAGTAGTATAAATAATAAGGGCT
Workflow
Local codePython

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

Correct
AnswerGCCGCCACCATGGTGAGCAAGGGCGAGTAGTATAAATAATAAGGGCT
Workflow
Local codePythonSciPy
Total tokens
25.0M
Output
106.4K
Codex + DeepSeek-v4-pro-0813

Replicate 2

Correct
AnswerGCCGCCACCATGGTGAGCAAGGGCGAGTAGTATAAATAATAAGGGCT
Workflow
Local codePythonSciPy
Total tokens
6.8M
Output
25.3K
Codex + DeepSeek-v4-pro-0813

Replicate 3

Incorrect
AnswerGCCGCCACCATGGTGAGCAAGGGCGAGTAGTGCCCGTAATAAGGACT
Workflow
Local codePython
Total tokens
15.6M
Output
62.5K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

Incorrect
AnswerGCCGCCACCATGGTGAGCAAGGGCGAGTAGCGTACATAATAAGTAGT
Workflow
Local codePythonSciPy
Codex GPT-5.6 Luna

Replicate 2

Correct
AnswerGCCGCCACCATGGTGAGCAAGGGCGAGTAGTATAAATAATAAGGGCT
Workflow
Local codePythonSciPy
Codex GPT-5.6 Luna

Replicate 3

Correct
AnswerGCCGCCACCATGGTGAGCAAGGGCGAGTAGTATAAATAATAAGGGCT
Workflow
Local codePython

Codex GPT-6 Astra

1 replicate
Codex GPT-6 Astra

Replicate 1

Correct
AnswerGCCGCCACCATGGTGAGCAAGGGCGAGTAGTATAAATAATAAGGGCT
Workflow
Local codePythonSciPy
Total tokens
2.6M
Output
14.4K