Back to results

CompBioBench item

reverse-search-gwas-q1

Question

Identify the PubMed ID of the study from which this summary statistics in reverse.search.gwas.q1.tsv.gz are derived. Print only the PMID, e.g. 31510655.

Inferred answer
41491094Uniquely inferred from submitted scores.
Domain
Population Genetics
Question style
Metadata Recovery

An answer can be uniquely inferred for 54 of 100 questions from submitted scores. Green: correct · Red: incorrect. For questions without a uniquely determined answer: Blue: matches prediction · Orange: differs from prediction. These are inferred answer checks, not official per-question grades.

Galaxy

12/12 terminal item results

UDT = user-defined tool: custom code executed by Galaxy. Route badges summarize analysis tools, excluding uploads and automatic conversions. How usage is counted.

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

IncorrectUDT-only
Answer29403010
Workflow
GWAS signature diagnosticGWAS loci diagnosticCompare Catalog target associationsCompare lead-position Catalog associationsSummarize lead-position comparison8 more tools
Total tokens
12.1M
Output
43.5K
Codex GPT-5.5

Replicate 2

IncorrectUDT + standard tools
Answer22558069
Workflow
GWAS diagnostic q1GWAS leads q1Confirm source GCST90241623Confirm source GCST90241623 diagnosticConfirm source GCST90241623 harmonised diagnostic
Total tokens
18.5M
Output
59.8K
Codex GPT-5.5

Replicate 3

IncorrectUDT + standard tools
Answer34648354
Workflow
Datamash OpsGWAS top hits extractorGWAS top hits extractor v2Tp Sort Header ToolTp Head Tool
Total tokens
5.5M
Output
26.5K

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

IncorrectUDT + standard tools
Answer29403010
Workflow
Tp Sort Header ToolExtract GWAS strongest signals and diagnosticsExtract GWAS independent locus minimaCompare significant loci to authoritative GWAS Catalog studies
Total tokens
5.7M
Output
15.2K
Codex GPT-5.6 Sol

Replicate 2

CorrectUDT + standard tools
Answer41491094
Workflow
Sort rowsShow Beginning1Select GWAS lead regionsUpload1Verify GWAS source and identify publication
Total tokens
25.3M
Output
29.4K
Codex GPT-5.6 Sol

Replicate 3

IncorrectUDT-only
Answer38798542
Workflow
GWAS fingerprint top signalsMatch GWAS fingerprint to authoritative MultiSuSiE resultsFinalize GWAS source PMID
Total tokens
3.0M
Output
9.9K

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

IncorrectUDT + standard tools
Answer32929287
Workflow
Tp Head ToolTp Tail ToolWc GnuTp Sort Header ToolProbe GWAS API14 more tools
Total tokens
67.1M
Output
186.6K
Codex + DeepSeek-v4-pro-0813

Replicate 2

RerunIncorrectUDT + standard tools
Answer33303764
Workflow
tp head tool → wc gnu → tp tail tool → Count1 → Grouping1 → tp sort header tool → secure hash message digest → User-defined tool → random lines1
Total tokens
103.7M
Output
125.5K
Codex + DeepSeek-v4-pro-0813

Replicate 3

IncorrectStandard tools only
Answer33710309
Workflow
Tp Sort Header ToolBcftools QueryData Fetch
Total tokens
128.8M
Output
320.2K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

RerunIncorrectStandard tools only
Answer32929287
Workflow
tp head tool → tp sort header tool → datamash ops → filter tabular → ctb online data fetch → Grep1
Total tokens
73.6M
Output
156.1K
Codex GPT-5.6 Luna

Replicate 2

IncorrectStandard tools only
Answer40593539
Workflow
Tp Sort Header ToolSort rowsTp Head ToolNlTp Tail Tool3 more tools
Total tokens
43.7M
Output
137.1K
Codex GPT-5.6 Luna

Replicate 3

IncorrectUDT + standard tools
Answer32128391
Workflow
Datamash OpsSummary Statistics1Filter TabularSelect columnsTp Sort Header Tool12 more tools
Total tokens
448.0M
Output
973.7K

Anycode

13/13 terminal item results

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

Incorrect
Answer32929287
Workflow
Local codePython
Codex GPT-5.5

Replicate 2

Incorrect
Answer37337107
Workflow
Local codePython
Codex GPT-5.5

Replicate 3

Incorrect
Answer33462484
Workflow
Local codePython

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

Incorrect
Answer38798542
Workflow
Local codePython
Codex GPT-5.6 Sol

Replicate 2

Correct
Answer41491094
Workflow
Local codePython
Codex GPT-5.6 Sol

Replicate 3

Correct
Answer41491094
Workflow
Local codePythonBCFtoolsPLINK

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

Incorrect
Answer40593539
Workflow
Local codePython
Total tokens
41.3M
Output
169.8K
Codex + DeepSeek-v4-pro-0813

Replicate 2

RerunIncorrect
Answer40593539
Workflow
Open-ended code
Total tokens
187.6M
Output
183.4K
Codex + DeepSeek-v4-pro-0813

Replicate 3

RerunIncorrect
Answer33710309
Workflow
Open-ended code
Total tokens
110.1M
Output
182.4K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

Incorrect
Answer40593539
Workflow
Local codePython
Codex GPT-5.6 Luna

Replicate 2

Incorrect
Answer40593539
Workflow
Local codePythonRBCFtoolspysam1 more tools
Codex GPT-5.6 Luna

Replicate 3

Correct
Answer41491094
Workflow
Local codePython

Codex GPT-6 Astra

1 replicate
Codex GPT-6 Astra

Replicate 1

Incorrect
Answer33462484
Workflow
Local codePythonpysamPLINK
Total tokens
46.3M
Output
180.5K