Back to results

CompBioBench item

retina-score-snps-q1

Question

https://zenodo.org/records/6330053 contains TensorFlow 2 models that are used for scoring the impact of SNPs on ATAC-seq in various cells types, as illustrated in the ScoreSNPs.ipynb notebook in https://zenodo.org/records/6796067. retina.score.snps.q1.tsv contains 500 SNPs (hg19 coordinates), of which half are known to be caQTLs. I would like to score the SNPs following the code in ScoreSNPs.ipynb, using the models corresponding to Rod cell type. After scoring the SNPs, report the AUPRC against the labels given in retina.score.snps.q1.tsv when using the absolute value of Rod_fold_avg_lfc as a predictor. Report answer by rounding down to two decimal places in the format: 0.80.

Inferred answer
0.67Uniquely inferred from submitted scores.
Domain
Machine Learning
Question style
Tooling

An answer can be uniquely inferred for 54 of 100 questions from submitted scores. Green: correct · Red: incorrect. For questions without a uniquely determined answer: Blue: matches prediction · Orange: differs from prediction. These are inferred answer checks, not official per-question grades.

Galaxy

12/12 terminal item results

UDT = user-defined tool: custom code executed by Galaxy. Route badges summarize analysis tools, excluding uploads and automatic conversions. How usage is counted.

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

CorrectUDT-only
Answer0.67
Workflow
SciPyRetina SNP build diagnosticStage retina models Zenodo 6330053Score retina SNPs with Rod BPNet models
Total tokens
981.6K
Output
11.0K
Codex GPT-5.5

Replicate 2

CorrectUDT-only
Answer0.67
Workflow
Coordinate diagnostic for retina SNPsCoordinate diagnostic for retina SNPs v2Probe Rod model shapeScore Rod SNPs and AUPRCValidate Rod PR metric
Total tokens
3.0M
Output
16.3K
Codex GPT-5.5

Replicate 3

RerunCorrectUDT-only
Answer0.67
Workflow
Retina coordinate diagnosticScore Rod SNPs
Total tokens
1.2M
Output
17.5K

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

CorrectUDT + standard tools
Answer0.67
Workflow
Prepare retina SNP coordinate diagnosticsExtract Genomic Dna 1Check retina SNP reference conventionsPrepare 2114 bp retina model windowsProbe Retina Rod TensorFlow model1 more tools
Total tokens
10.8M
Output
26.7K
Codex GPT-5.6 Sol

Replicate 2

CorrectUDT + standard tools
Answer0.67
Workflow
Retina SNP coordinate interpretationsLiftover1Retina SNP lift-over and allele diagnosticRetina SNP validated lift-overRod TensorFlow model compatibility probe2 more tools
Total tokens
21.7M
Output
32.2K
Codex GPT-5.6 Sol

Replicate 3

CorrectUDT + standard tools
Answer0.67
Workflow
Data FetchRetina coordinate diagnostic intervalsExtract Genomic Dna 1Retina reference match summaryInspect Retina Rod model2 more tools
Total tokens
16.2M
Output
26.6K

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

RerunCorrectUDT + standard tools
Answer0.67
Workflow
Extract Genomic DNAMake BED intervalsbedtools getfastaCompare SNP ref allelesConvert genome coordinates5 more tools
Total tokens
21.4M
Output
115.8K
Codex + DeepSeek-v4-pro-0813

Replicate 2

CorrectUDT + standard tools
Answer0.67
Workflow
Liftover1Crossmap BedBedtools GetfastabedModel ProbeModel Probe Data5 more tools
Total tokens
18.1M
Output
72.4K
Codex + DeepSeek-v4-pro-0813

Replicate 3

CorrectUDT + standard tools
Answer0.67
Workflow
Make SNP intervalsMake SNP intervals v2Select columnsMake SNP intervals v3UDT busybox probe5 more tools
Total tokens
74.0M
Output
136.7K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

CorrectUDT + standard tools
Answer0.67
Workflow
Upload1UnzipRetina Notebook InspectUDT Render ProbeUDT Canonical Probe10 more tools
Total tokens
32.0M
Output
78.9K
Codex GPT-5.6 Luna

Replicate 2

CorrectUDT + standard tools
Answer0.67
Workflow
Upload1Retina reference convention diagnosticPrepare SNP intervals for hg19 to hg38Liftover1Validate hg19 to hg38 SNP liftover11 more tools
Total tokens
14.2M
Output
54.4K
Codex GPT-5.6 Luna

Replicate 3

CorrectUDT + standard tools
Answer0.67
Workflow
Make SNP IntervalsUDT Render ProbeUDT Render Probe CommandUDT Render Probe TabularUDT Render Probe Legacy20 more tools
Total tokens
105.0M
Output
278.1K

Anycode

13/13 terminal item results

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

Correct
Answer0.67
Workflow
Local codePythonSciPy
Codex GPT-5.5

Replicate 2

Correct
Answer0.67
Workflow
Local codePython
Codex GPT-5.5

Replicate 3

Correct
Answer0.67
Workflow
Local codePython

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

Correct
Answer0.67
Workflow
Local codePython
Codex GPT-5.6 Sol

Replicate 2

Correct
Answer0.67
Workflow
Local codePython
Codex GPT-5.6 Sol

Replicate 3

Correct
Answer0.67
Workflow
Local codePython

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

Correct
Answer0.67
Workflow
Local codePythonSciPypysam
Total tokens
20.5M
Output
83.7K
Codex + DeepSeek-v4-pro-0813

Replicate 2

Correct
Answer0.67
Workflow
Local codePythonSciPy
Total tokens
8.5M
Output
37.4K
Codex + DeepSeek-v4-pro-0813

Replicate 3

Correct
Answer0.67
Workflow
Local codePythonSciPySamtoolspysam
Total tokens
19.0M
Output
100.0K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

Incorrect
Answer0.58
Workflow
Local codePythonSciPy
Codex GPT-5.6 Luna

Replicate 2

Correct
Answer0.67
Workflow
Local codePythonSciPy
Codex GPT-5.6 Luna

Replicate 3

Correct
Answer0.67
Workflow
Local codePythonSciPy

Codex GPT-6 Astra

1 replicate
Codex GPT-6 Astra

Replicate 1

Correct
Answer0.67
Workflow
Local codePython
Total tokens
1.1M
Output
8.6K