Back to results

CompBioBench item

variant-status-q1

Question

A collaborator sent me variant.status.q1.bam corresponding to a single-end RNA-seq experiment from an individual mapped to hg38. What is the likely variant status at position chrX:154,398,500 (1-based) for this individual? Respond in the format x/y, where x and y are one of A,C,G,T and x<y lexicographically, or x=y if homozygous.

Inferred answer
C/CUniquely inferred from submitted scores.
Domain
Transcriptomics
Question style
Routine Analysis

An answer can be uniquely inferred for 54 of 100 questions from submitted scores. Green: correct · Red: incorrect. For questions without a uniquely determined answer: Blue: matches prediction · Orange: differs from prediction. These are inferred answer checks, not official per-question grades.

Galaxy

12/12 terminal item results

UDT = user-defined tool: custom code executed by Galaxy. Route badges summarize analysis tools, excluding uploads and automatic conversions. How usage is counted.

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

IncorrectStandard tools only
AnswerC/T
Workflow
Galaxy dataset inspection
Total tokens
557.3K
Output
5.6K
Codex GPT-5.5

Replicate 2

IncorrectStandard tools only
AnswerC/T
Workflow
Samtools MpileupVarscan Mpileup
Total tokens
535.3K
Output
6.4K
Codex GPT-5.5

Replicate 3

IncorrectStandard tools only
AnswerC/T
Workflow
Galaxy dataset inspection
Total tokens
234.9K
Output
3.6K

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

IncorrectStandard tools only
AnswerC/T
Workflow
Samtools MpileupBcftools MpileupVarscan Mpileup
Total tokens
890.5K
Output
5.0K
Codex GPT-5.6 Sol

Replicate 2

CorrectUDT + standard tools
AnswerC/C
Workflow
Samtools MpileupRead-position allele audit chrX 154398500
Total tokens
838.9K
Output
8.2K
Codex GPT-5.6 Sol

Replicate 3

CorrectUDT + standard tools
AnswerC/C
Workflow
Samtools MpileupSamtools ViewBcftools MpileupTarget allele position-aware diagnostic
Total tokens
2.2M
Output
10.2K

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

IncorrectStandard tools only
AnswerC/T
Workflow
Samtools IdxstatsSamtools Mpileup
Total tokens
721.7K
Output
10.1K
Codex + DeepSeek-v4-pro-0813

Replicate 2

IncorrectStandard tools only
AnswerC/T
Workflow
Samtools ViewSamtools Slice BamExtract Genomic Dna 1Samtools MpileupBcftools Mpileup1 more tools
Total tokens
5.7M
Output
32.7K
Codex + DeepSeek-v4-pro-0813

Replicate 3

IncorrectStandard tools only
AnswerC/T
Workflow
Bcftools MpileupSamtools IdxstatsSamtools Mpileup
Total tokens
1.4M
Output
14.3K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

IncorrectStandard tools only
AnswerC/T
Workflow
Samtools MpileupBcftools MpileupSamtools Depth
Total tokens
2.4M
Output
14.1K
Codex GPT-5.6 Luna

Replicate 2

IncorrectStandard tools only
AnswerC/T
Workflow
Bam To SamSamtools FlagstatSamtools MpileupExtract Genomic Dna 1Fasta2Tab3 more tools
Total tokens
11.7M
Output
33.5K
Codex GPT-5.6 Luna

Replicate 3

IncorrectStandard tools only
AnswerC/T
Workflow
Samtools Mpileup
Total tokens
1.1M
Output
12.1K

Anycode

13/13 terminal item results

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

Correct
AnswerC/C
Workflow
Local codeSamtoolsPythonBCFtools
Codex GPT-5.5

Replicate 2

Incorrect
AnswerC/T
Workflow
Local codeSamtoolsBCFtoolsPython
Codex GPT-5.5

Replicate 3

Incorrect
AnswerC/T
Workflow
Local codeSamtools

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

Incorrect
AnswerC/T
Workflow
Local codeSamtoolsBCFtoolspysam
Codex GPT-5.6 Sol

Replicate 2

Incorrect
AnswerC/T
Workflow
Local codeSamtools
Codex GPT-5.6 Sol

Replicate 3

Incorrect
AnswerC/T
Workflow
Local codeSamtools

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

Incorrect
AnswerC/T
Workflow
Local codeSamtoolsPythonpysamBCFtools
Total tokens
249.7K
Output
7.0K
Codex + DeepSeek-v4-pro-0813

Replicate 2

Correct
AnswerC/C
Workflow
Local codeSamtoolsBCFtoolsPythonpysam2 more tools
Total tokens
7.0M
Output
64.5K
Codex + DeepSeek-v4-pro-0813

Replicate 3

Incorrect
AnswerC/T
Workflow
Local codeSamtoolsBCFtoolspysamPython
Total tokens
457.6K
Output
10.9K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

Incorrect
AnswerC/T
Workflow
Local codeSamtoolsBCFtoolspysamPython
Codex GPT-5.6 Luna

Replicate 2

Incorrect
AnswerC/T
Workflow
Local codeSamtoolsBCFtoolsBedtoolspysam
Codex GPT-5.6 Luna

Replicate 3

Incorrect
AnswerC/T
Workflow
Local codeSamtoolsBCFtoolsBedtoolspysam1 more tools

Codex GPT-6 Astra

1 replicate
Codex GPT-6 Astra

Replicate 1

Correct
AnswerC/C
Workflow
Local codeSamtoolsPython
Total tokens
79.4K
Output
604