Back to results

CompBioBench item

huggingface-entropy-q1

Question

Run the DNA language model here: https://huggingface.co/kuleshov-group/caduceus-ph_seqlen-131k_d_model-256_n_layer-16 on the 1000 bp long DNA sequence at hg38 chr12:7792299-7793299. Specifically, mask each base one at a time, keeping all others unmasked, and compute the per position base 2 entropy of the predictions. Return the substring corresponding to the longest run with <0.5 entropy.

Inferred answer
GTCTCTACTAAAAATACAAAATTAGCCUniquely inferred from submitted scores.
Domain
Machine Learning
Question style
Tooling

An answer can be uniquely inferred for 54 of 100 questions from submitted scores. Green: correct · Red: incorrect. For questions without a uniquely determined answer: Blue: matches prediction · Orange: differs from prediction. These are inferred answer checks, not official per-question grades.

Galaxy

12/12 terminal item results

UDT = user-defined tool: custom code executed by Galaxy. Route badges summarize analysis tools, excluding uploads and automatic conversions. How usage is counted.

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

CorrectUDT-only
AnswerGTCTCTACTAAAAATACAAAATTAGCC
Workflow
Caduceus mask entropy chr12 intervalCaduceus env probeCaduceus import probeCaduceus mask entropy chr12 interval CPU shimCaduceus mask entropy chr12 interval unfused CPU1 more tools
Codex GPT-5.5

Replicate 2

RerunCorrectUDT-only
AnswerGTCTCTACTAAAAATACAAAATTAGCC
Workflow
Caduceus coordinate diagnosticCaduceus model tokenizer probeCaduceus model tokenizer probe high memoryCaduceus model tokenizer probe low memoryCaduceus import probe13 more tools
Total tokens
2.9M
Output
23.2K
Codex GPT-5.5

Replicate 3

RerunCorrectUDT-only
AnswerGTCTCTACTAAAAATACAAAATTAGCC
Workflow
hg38 interval diagnosticCaduceus model probeCaduceus environment probeCaduceus import detail probeCaduceus code probe5 more tools
Total tokens
2.0M
Output
22.5K

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

CorrectUDT-only
AnswerGTCTCTACTAAAAATACAAAATTAGCC
Workflow
Galaxy UDTcaduceus_entropyCaduceus entropyUDT container smoke testMamba runtime probe4 more tools
Total tokens
27.7M
Output
32.5K
Codex GPT-5.6 Sol

Replicate 2

CorrectUDT-only
AnswerGTCTCTACTAAAAATACAAAATTAGCC
Workflow
Validate UCSC hg38 interval and emit FASTACaduceus masked-base entropy preflightCaduceus mask-each-base entropy and longest low-entropy run
Total tokens
4.2M
Output
14.8K
Codex GPT-5.6 Sol

Replicate 3

CorrectUDT-only
AnswerGTCTCTACTAAAAATACAAAATTAGCC
Workflow
Validate UCSC hg38 interval and convert to FASTAProbe Caduceus masked-LM token alignment and probabilitiesInspect pinned Caduceus runtime layoutProbe pinned Caduceus Python importsProbe Caduceus masked-LM with CPU reference backend1 more tools
Total tokens
8.8M
Output
21.6K

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

CorrectUDT + standard tools
AnswerGTCTCTACTAAAAATACAAAATTAGCC
Workflow
No Galaxy tool job recorded
Total tokens
56.1M
Output
78.7K
Codex + DeepSeek-v4-pro-0813

Replicate 2

CorrectUDT + standard tools
AnswerGTCTCTACTAAAAATACAAAATTAGCC
Workflow
Extract Genomic Dna 1Caduceus ProbeCaduceus Probe2Caduceus Probe2 v2Caduceus Probe2 v36 more tools
Total tokens
7.3M
Output
48.7K
Codex + DeepSeek-v4-pro-0813

Replicate 3

CorrectUDT + standard tools
AnswerGTCTCTACTAAAAATACAAAATTAGCC
Workflow
Extract Genomic Dna 1Caduceus Runtime ProbeCaduceus Runtime Probe v2Caduceus Entropy SmokeCaduceus Entropy Smoke v21 more tools
Total tokens
4.5M
Output
34.0K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

CorrectUDT + standard tools
AnswerGTCTCTACTAAAAATACAAAATTAGCC
Workflow
Caduceus coordinate diagnosticCaduceus UDT render probeCaduceus UDT render probe 2Cat Multi DatasetsBusybox UDT render probe5 more tools
Total tokens
27.8M
Output
75.7K
Codex GPT-5.6 Luna

Replicate 2

CorrectUDT + standard tools
AnswerGTCTCTACTAAAAATACAAAATTAGCC
Workflow
CreateintervalExtract Genomic Dna 1Caduceus runtime probeUDT render probe44967277 Cb96 47F9 8633 C8Da7Cff93248 more tools
Total tokens
37.1M
Output
112.6K
Codex GPT-5.6 Luna

Replicate 3

CorrectUDT + standard tools
AnswerGTCTCTACTAAAAATACAAAATTAGCC
Workflow
Extract Genomic Dna 1Caduceus preflightCaduceus runtime probeCaduceus runtime probe v2Caduceus preflight with CPU scan8 more tools
Total tokens
19.5M
Output
49.3K

Anycode

13/13 terminal item results

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

Correct
AnswerGTCTCTACTAAAAATACAAAATTAGCC
Workflow
Local codePython
Codex GPT-5.5

Replicate 2

Correct
AnswerGTCTCTACTAAAAATACAAAATTAGCC
Workflow
Local codePython
Codex GPT-5.5

Replicate 3

Correct
AnswerGTCTCTACTAAAAATACAAAATTAGCC
Workflow
Local codePython

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

Incorrect
AnswerTT
Workflow
Local codePython
Codex GPT-5.6 Sol

Replicate 2

Correct
AnswerGTCTCTACTAAAAATACAAAATTAGCC
Workflow
Local codePython
Codex GPT-5.6 Sol

Replicate 3

Correct
AnswerGTCTCTACTAAAAATACAAAATTAGCC
Workflow
Local codePython

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

Correct
AnswerGTCTCTACTAAAAATACAAAATTAGCC
Workflow
Local codePython
Total tokens
11.6M
Output
59.2K
Codex + DeepSeek-v4-pro-0813

Replicate 2

Correct
AnswerGTCTCTACTAAAAATACAAAATTAGCC
Workflow
Local codePython
Total tokens
13.0M
Output
70.9K
Codex + DeepSeek-v4-pro-0813

Replicate 3

Correct
AnswerGTCTCTACTAAAAATACAAAATTAGCC
Workflow
Local codePython
Total tokens
11.8M
Output
64.4K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

Correct
AnswerGTCTCTACTAAAAATACAAAATTAGCC
Workflow
Local codePython
Codex GPT-5.6 Luna

Replicate 2

Correct
AnswerGTCTCTACTAAAAATACAAAATTAGCC
Workflow
Local codePython
Codex GPT-5.6 Luna

Replicate 3

Correct
AnswerGTCTCTACTAAAAATACAAAATTAGCC
Workflow
Local codePython

Codex GPT-6 Astra

1 replicate
Codex GPT-6 Astra

Replicate 1

Correct
AnswerGTCTCTACTAAAAATACAAAATTAGCC
Workflow
Local codePython
Total tokens
2.2M
Output
17.7K