Back to results

CompBioBench item

conservation-lookup-q1

Question

Using the human SHH MANE Select transcript, take the cDNA sequence from the annotated start codon through the end of exon 1. For each of the following species: Pan troglodytes, Macaca mulatta, Pan paniscus, Indri indri, Semnopithecus johnii, Ailuropoda melanoleuca, Delphinapterus leucas, obtain the orthologous cDNA segment corresponding to the same region (prefer a one-to-one ortholog; if multiple transcripts exist, use the canonical/longest protein-coding transcript). Compute the Levenshtein edit distance to the human sequence. Return results as a comma-separated list (no spaces) in the listed species order.

Inferred answer
0,7,0,20,5,24,27Uniquely inferred from submitted scores.
Domain
Genomics
Question style
Retrieval

An answer can be uniquely inferred for 54 of 100 questions from submitted scores. Green: correct · Red: incorrect. For questions without a uniquely determined answer: Blue: matches prediction · Orange: differs from prediction. These are inferred answer checks, not official per-question grades.

Galaxy

12/12 terminal item results

UDT = user-defined tool: custom code executed by Galaxy. Route badges summarize analysis tools, excluding uploads and automatic conversions. How usage is counted.

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

IncorrectUDT-only
Answer9,16,0,20,5,24,35
Workflow
SHH MANE ortholog diagnosticSHH parse initial Ensembl sourcesSHH final cDNA segment distancesSHH final cDNA segment distances v2
Total tokens
4.2M
Output
30.0K
Codex GPT-5.5

Replicate 2

CorrectUDT-only
Answer0,7,0,20,5,24,27
Workflow
Diagnose SHH Ensembl ortholog annotationsDiagnose Ensembl homology endpointsCompute SHH exon 1 ortholog edit distancesDiagnose SHH species homologyFind Ensembl species names11 more tools
Total tokens
3.5M
Output
31.7K
Codex GPT-5.5

Replicate 3

CorrectUDT + standard tools
Answer0,7,0,20,5,24,27
Workflow
SHH Ensembl cDNA exon1 diagnosticUDT input path probeUDT cat doc probeUDT constant command probe
Total tokens
6.7M
Output
34.8K

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

CorrectUDT + standard tools
Answer0,7,0,20,5,24,27
Workflow
SHH Ensembl orthology transcript diagnosticSHH Ensembl orthology transcript diagnostic v2SHH Ensembl orthology transcript diagnostic v3Datasets Download GeneSHH core transcript and exon diagnostic15 more tools
Total tokens
27.1M
Output
50.7K
Codex GPT-5.6 Sol

Replicate 2

CorrectUDT + standard tools
Answer0,7,0,20,5,24,27
Workflow
SHH human MANE exon-1 boundary diagnosticSHH ortholog and transcript diagnosticNCBI SHH vertebrate ortholog data packageInspect NCBI SHH target reportsInspect Ensembl Rapid Release target species8 more tools
Total tokens
15.2M
Output
46.1K
Codex GPT-5.6 Sol

Replicate 3

CorrectUDT + standard tools
Answer0,7,0,20,5,24,27
Workflow
SHH human MANE transcript diagnosticSHH UDT runtime probeDatasets Download GeneTp Grep ToolUpload110 more tools
Total tokens
33.1M
Output
41.3K

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

CorrectUDT + standard tools
Answer0,7,0,20,5,24,27
Workflow
No Galaxy tool job recorded
Total tokens
61.0M
Output
107.5K
Codex + DeepSeek-v4-pro-0813

Replicate 2

CorrectUDT + standard tools
Answer0,7,0,20,5,24,27
Workflow
Datasets Download GeneNcbi Blastn WrapperDatasets Download GenomeSHH Edit Distance AnalysisSHH Probe Minimal6 more tools
Total tokens
21.4M
Output
71.7K
Codex + DeepSeek-v4-pro-0813

Replicate 3

RerunIncorrectUDT + standard tools
Answer9,16,0,29,14,24,35
Workflow
NCBI Datasets GeneNCBI Accession DownloadNCBI Datasets GenomesNCBI BLAST+ makeblastdbNCBI BLAST+ blastn1 more tools
Total tokens
16.2M
Output
72.6K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

IncorrectUDT + standard tools
Answer9,16,9,20,5,24,36
Workflow
SHH cDNA segment extractionUDT render probeEmboss: Extractseq35Minimap2Bedtools Bamtobed7 more tools
Total tokens
41.9M
Output
86.6K
Codex GPT-5.6 Luna

Replicate 2

RerunIncorrectUDT-only
Answer9,16,0,20,5,24,35
Workflow
SHH Annotation DiagnosticMANE SHH FilterSHH First Exon Segment Edit DistanceFilter Indri SHH XenoRefGeneIndri SHH First Exon Segment Edit Distance5 more tools
Total tokens
19.8M
Output
63.7K
Codex GPT-5.6 Luna

Replicate 3

IncorrectUDT + standard tools
Answer9,16,0,20,5,24,36
Workflow
SHH transcript annotation diagnosticSHH UDT render probeSHH UDT format probeSHH UDT legacy probeSHH UDT busybox probe18 more tools
Total tokens
30.6M
Output
88.9K

Anycode

13/13 terminal item results

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

Correct
Answer0,7,0,20,5,24,27
Workflow
Local codePythonBLASTNMinimap2
Codex GPT-5.5

Replicate 2

Correct
Answer0,7,0,20,5,24,27
Workflow
Local codePythonBLASTNMinimap2
Codex GPT-5.5

Replicate 3

Correct
Answer0,7,0,20,5,24,27
Workflow
Local codePython

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

Correct
Answer0,7,0,20,5,24,27
Workflow
Local codePythonBLASTN
Codex GPT-5.6 Sol

Replicate 2

Correct
Answer0,7,0,20,5,24,27
Workflow
Local codePython
Codex GPT-5.6 Sol

Replicate 3

Correct
Answer0,7,0,20,5,24,27
Workflow
Local codePythonBLASTNMinimap2BWA-MEM

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

Incorrect
Answer9,16,9,20,5,24,35
Workflow
Local codePythonBLASTNSamtools
Total tokens
7.1M
Output
46.1K
Codex + DeepSeek-v4-pro-0813

Replicate 2

Incorrect
Answer9,16,0,29,5,24,35
Workflow
Local codePythonBLASTNMinimap2Samtools
Total tokens
17.0M
Output
87.3K
Codex + DeepSeek-v4-pro-0813

Replicate 3

Incorrect
Answer9,16,9,29,14,24,35
Workflow
Local codePythonBLASTNMinimap2Samtools2 more tools
Total tokens
24.1M
Output
92.0K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

RerunCorrect
Answer0,7,0,20,5,24,27
Workflow
Open-ended code
Total tokens
14.5M
Output
67.5K
Codex GPT-5.6 Luna

Replicate 2

Incorrect
Answer9,16,9,29,14,24,35
Workflow
Local codePythonMAFFTBLASTNMinimap22 more tools
Codex GPT-5.6 Luna

Replicate 3

RerunIncorrect
Answer9,16,9,29,14,24,35
Workflow
Open-ended code
Total tokens
25.7M
Output
79.6K

Codex GPT-6 Astra

1 replicate
Codex GPT-6 Astra

Replicate 1

Correct
Answer0,7,0,20,5,24,27
Workflow
Local codePython
Total tokens
896.9K
Output
6.1K