Back to results

CompBioBench item

three-way-barnyard-q2

Question

Given paired-end 10x scRNA-seq FASTQs (three.way.barnyard.q2.R1.fq.gz, three.way.barnyard.q2.R2.fq.gz) from a 3-species barnyard (human, mouse, pig)—R1=28-bp cell barcode+UMI, R2≈91-bp cDNA. The barcodes are pre-thresholded for minimum read depth. Report the percentages among confidently assigned single-cell barcodes for human, mouse, pig (rounded to the nearest 10%, comma-separated, summing to 100) and also assign a tissue of origin for human, mouse, pig, respectively, chosen from: liver, heart, testis, kidney, bone_marrow, spleen, cortex, retina, lung, skin. Expected output format: 40;40;20;kidney;heart;heart.

Inferred answer
10;30;60;bone_marrow;lung;testisUniquely inferred from submitted scores.
Domain
Single-cell
Question style
Synthetic/Augmented Data

An answer can be uniquely inferred for 54 of 100 questions from submitted scores. Green: correct · Red: incorrect. For questions without a uniquely determined answer: Blue: matches prediction · Orange: differs from prediction. These are inferred answer checks, not official per-question grades.

Galaxy

12/12 terminal item results

UDT = user-defined tool: custom code executed by Galaxy. Route badges summarize analysis tools, excluding uploads and automatic conversions. How usage is counted.

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

CorrectUDT-only
Answer10;30;60;bone_marrow;lung;testis
Workflow
Minimap2SamtoolsThree species barnyard analysis
Total tokens
265.3K
Output
10.1K
Galaxy historyAgent trace
Codex GPT-5.5

Replicate 2

CorrectUDT-only
Answer10;30;60;bone_marrow;lung;testis
Workflow
Minimap2BWA-MEMBowtie2SamtoolsBedtools3 more tools
Total tokens
622.6K
Output
11.9K
Galaxy historyAgent trace
Codex GPT-5.5

Replicate 3

RerunIncorrectUDT-only
Answer10;40;50;bone_marrow;lung;testis
Workflow
Runtime probe sequence toolboxBarnyard species tissue diagnosticBarnyard species tissue fullBarnyard threshold scan
Total tokens
1.6M
Output
18.4K
Galaxy historyAgent trace

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

CorrectUDT + standard tools
Answer10;30;60;bone_marrow;lung;testis
Workflow
Umi Tools ExtractBarnyard preprocessBarnyard preprocess v2Minimap2Barnyard classify
Total tokens
9.5M
Output
22.2K
Galaxy historyAgent trace
Codex GPT-5.6 Sol

Replicate 2

CorrectUDT + standard tools
Answer10;30;60;bone_marrow;lung;testis
Workflow
FastqcBuild Ensembl 115 three-species kallisto indexUpload1Probe kallisto runtimeDiagnose three-species 10x barnyard1 more tools
Total tokens
15.6M
Output
48.3K
Galaxy historyAgent trace
Codex GPT-5.6 Sol

Replicate 3

CorrectUDT + standard tools
Answer10;30;60;bone_marrow;lung;testis
Workflow
Three-species FASTQ barcode diagnosticPrepare three-species barnyard mapping inputsMinimap2Summarize three-species barnyard alignmentsFinalize three-species barnyard answer
Total tokens
9.5M
Output
27.4K
Galaxy historyAgent trace

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

IncorrectUDT + standard tools
Answer20;30;50;bone_marrow;lung;testis
Workflow
AlevinAlevin diagnosticAlevin classificationFinal Alevin answerFinal Alevin answer v2
Total tokens
4.2M
Output
42.1K
Galaxy historyAgent trace
Codex + DeepSeek-v4-pro-0813

Replicate 2

CorrectUDT + standard tools
Answer10;30;60;bone_marrow;lung;testis
Workflow
Ensembl Gtf2Gene ListFasta Concatenate0Cat MultipleZip CollectionAlevin9 more tools
Total tokens
11.4M
Output
55.7K
Galaxy historyAgent trace
Codex + DeepSeek-v4-pro-0813

Replicate 3

IncorrectUDT + standard tools
Answer10;30;60;retina;heart;cortex
Workflow
Fastq InfoBarnyard q2 analysisProbeProbe2ProbeBusybox19 more tools
Total tokens
78.2M
Output
219.2K
Galaxy historyAgent trace

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

IncorrectUDT + standard tools
Answer20;30;50;bone_marrow;lung;testis
Workflow
Fastq StatsUpload1Barnyard marker diagnosticUDT render probeUDT render probe literal19 more tools
Total tokens
131.8M
Output
197.2K
Galaxy historyAgent trace
Codex GPT-5.6 Luna

Replicate 2

IncorrectUDT + standard tools
Answer20;30;50;bone_marrow;lung;testis
Workflow
Three-way barnyard FASTQ diagnosticUDT binding probeFastq StatsUpload1Ensembl Gtf2Gene List13 more tools
Total tokens
48.9M
Output
108.5K
Galaxy historyAgent trace
Codex GPT-5.6 Luna

Replicate 3

CorrectUDT + standard tools
Answer10;30;60;bone_marrow;lung;testis
Workflow
FastqcBarnyard sequence-toolbox runtime probeBarnyard species mapping diagnosticBarnyard species mapping diagnostic, score-voteThree-species barnyard full mapping2 more tools
Total tokens
7.4M
Output
52.5K
Galaxy historyAgent trace

Anycode

13/13 terminal item results

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

Correct
Answer10;30;60;bone_marrow;lung;testis
Workflow
Local codePythonMinimap2Bowtie2pysam1 more tools
Agent trace
Codex GPT-5.5

Replicate 2

Correct
Answer10;30;60;bone_marrow;lung;testis
Workflow
Local codePythonMinimap2Bowtie2BWA-MEM4 more tools
Agent trace
Codex GPT-5.5

Replicate 3

Incorrect
Answer20;30;50;bone_marrow;lung;skin
Workflow
Local codeMinimap2Bowtie2BLASTNPython
Agent trace

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

Correct
Answer10;30;60;bone_marrow;lung;testis
Workflow
Local codeMinimap2Bowtie2SamtoolsPython
Agent trace
Codex GPT-5.6 Sol

Replicate 2

Incorrect
Answer20;30;50;bone_marrow;lung;testis
Workflow
Local codeBowtie2Minimap2SamtoolsBLASTN1 more tools
Agent trace
Codex GPT-5.6 Sol

Replicate 3

Correct
Answer10;30;60;bone_marrow;lung;testis
Workflow
Local codeMinimap2SamtoolsPython
Agent trace

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

Correct
Answer10;30;60;bone_marrow;lung;testis
Workflow
Local codePythonSamtoolsBowtie2Minimap25 more tools
Total tokens
15.8M
Output
84.8K
Agent trace
Codex + DeepSeek-v4-pro-0813

Replicate 2

Incorrect
Answer20;30;50;bone_marrow;lung;testis
Workflow
Local codePythonMinimap2BWA-MEMBowtie23 more tools
Total tokens
20.5M
Output
83.9K
Agent trace
Codex + DeepSeek-v4-pro-0813

Replicate 3

Correct
Answer10;30;60;bone_marrow;lung;testis
Workflow
Local codePythonMinimap2Bowtie2Samtools1 more tools
Total tokens
3.8M
Output
33.8K
Agent trace

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

Correct
Answer10;30;60;bone_marrow;lung;testis
Workflow
Local codeMinimap2Bowtie2BWA-MEMSamtools1 more tools
Agent trace
Codex GPT-5.6 Luna

Replicate 2

Correct
Answer10;30;60;bone_marrow;lung;testis
Workflow
Local codeMinimap2Bowtie2BWA-MEMSamtools2 more tools
Agent trace
Codex GPT-5.6 Luna

Replicate 3

Incorrect
Answer20;30;50;bone_marrow;lung;testis
Workflow
Local codeBWA-MEMMinimap2Bowtie2Samtools3 more tools
Agent trace

Codex GPT-6 Astra

1 replicate
Codex GPT-6 Astra

Replicate 1

Correct
Answer10;30;60;bone_marrow;lung;testis
Workflow
Local codePythonMinimap2SciPypysam
Total tokens
1.2M
Output
11.1K
Agent trace