Back to results

CompBioBench item

three-way-barnyard-q1

Question

Given paired-end 10x scRNA-seq FASTQs (three.way.barnyard.q1.R1.fq.gz, three.way.barnyard.q1.R2.fq.gz) from a 3-species barnyard (human, mouse, pig)—R1 = 28-bp cell barcode+UMI, R2 = 91-bp cDNA—assume all barcodes are valid single cells (no empty droplets or doublets; no further barcode filtering required) and report the mixture as percentages rounded to the nearest 10%, semicolon-separated, summing to 100, for human, mouse, and pig, respectively. Example output: "50;20;30".

Predicted answer
10;30;60Most likely answer for this question; not uniquely determined from submitted scores.
Domain
Single-cell
Question style
Synthetic/Augmented Data

An answer can be uniquely inferred for 54 of 100 questions from submitted scores. Green: correct · Red: incorrect. For questions without a uniquely determined answer: Blue: matches prediction · Orange: differs from prediction. These are inferred answer checks, not official per-question grades.

Galaxy

12/12 terminal item results

UDT = user-defined tool: custom code executed by Galaxy. Route badges summarize analysis tools, excluding uploads and automatic conversions. How usage is counted.

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

Matches predictionUDT-only
Answer10;30;60
Workflow
FASTQ Structure DiagnosticThree Way Barnyard Kmer Classifier
Codex GPT-5.5

Replicate 2

Matches predictionUDT-only
Answer10;30;60
Workflow
Barnyard species kmer diagnosticBarnyard species kmer full
Total tokens
448.9K
Output
10.4K
Codex GPT-5.5

Replicate 3

Matches predictionUDT + standard tools
Answer10;30;60
Workflow
Barnyard Species Minimap DiagnosticUDT Render ProbeUDT Render Probe 2UDT Render Probe 3UDT Render Probe 4
Total tokens
5.6M
Output
29.9K

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

Matches predictionUDT + standard tools
Answer10;30;60
Workflow
Fastq StatsKraken2UDT environment checkMinimap2Aggregate three-species barcodes4 more tools
Total tokens
17.3M
Output
34.6K
Codex GPT-5.6 Sol

Replicate 2

Matches predictionUDT-only
Answer10;30;60
Workflow
10x paired FASTQ barcode diagnostic10x paired FASTQ barcode diagnostic correctedUpload1Minimap2BWA-MEM4 more tools
Total tokens
2.2M
Output
16.0K
Codex GPT-5.6 Sol

Replicate 3

Matches predictionUDT + standard tools
Answer10;30;60
Workflow
FastqcBarnyard barcode and mate diagnosticMinimap2
Total tokens
3.3M
Output
18.2K

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

Matches predictionUDT + standard tools
Answer10;30;60
Workflow
Rna StarHisat2SamtoolsBedtools
Total tokens
8.2M
Output
39.4K
Codex + DeepSeek-v4-pro-0813

Replicate 2

Matches predictionUDT + standard tools
Answer10;30;60
Workflow
Converter Gz To UncompressedZip CollectionAlevinUmi Tools WhitelistRna Starsolo1 more tools
Total tokens
10.3M
Output
41.0K
Codex + DeepSeek-v4-pro-0813

Replicate 3

Matches predictionStandard tools only
Answer10;30;60
Workflow
Fastq TrimmerCshl Fastq To FastaFasta2TabSelect columnsCount11 more tools
Total tokens
22.1M
Output
52.2K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

Matches predictionUDT + standard tools
Answer10;30;60
Workflow
Datasets Download GeneDatasets Download GenomeMinimap2Species mapping runtime probeUDT mechanics probe12 more tools
Total tokens
32.3M
Output
80.7K
Codex GPT-5.6 Luna

Replicate 2

Matches predictionUDT + standard tools
Answer10;30;60
Workflow
FastqcDatasets Download GenomeMinimap210x Three-Species Barnyard Classifier
Total tokens
5.3M
Output
33.8K
Codex GPT-5.6 Luna

Replicate 3

Matches predictionUDT + standard tools
Answer10;30;60
Workflow
Fastq StatsUmi Tools ExtractDatasets Download GenomeUpload1Barnyard Runtime Probe2 more tools
Total tokens
10.4M
Output
40.0K

Anycode

13/13 terminal item results

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

Matches prediction
Answer10;30;60
Workflow
Local codeMinimap2Bowtie2BWA-MEMPython1 more tools
Codex GPT-5.5

Replicate 2

Matches prediction
Answer10;30;60
Workflow
Local codeBowtie2Minimap2PythonKraken22 more tools
Codex GPT-5.5

Replicate 3

Matches prediction
Answer10;30;60
Workflow
Local codeMinimap2Kraken2

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

Matches prediction
Answer10;30;60
Workflow
Local codeMinimap2Bowtie2SamtoolsPython
Codex GPT-5.6 Sol

Replicate 2

Matches prediction
Answer10;30;60
Workflow
Local codeMinimap2BWA-MEMBowtie2Kraken2
Codex GPT-5.6 Sol

Replicate 3

Matches prediction
Answer10;30;60
Workflow
Local codeMinimap2BWA-MEMKraken2Samtools1 more tools

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

Matches prediction
Answer10;30;60
Workflow
Local codePythonSamtoolsMinimap2BWA-MEM1 more tools
Total tokens
4.9M
Output
26.1K
Codex + DeepSeek-v4-pro-0813

Replicate 2

RerunMatches prediction
Answer10;30;60
Workflow
Open-ended code
Total tokens
4.1M
Output
26.3K
Codex + DeepSeek-v4-pro-0813

Replicate 3

Matches prediction
Answer10;30;60
Workflow
Local codePythonSamtoolsMinimap2BWA-MEM1 more tools
Total tokens
2.3M
Output
32.1K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

Matches prediction
Answer10;30;60
Workflow
Local codeSamtoolsMinimap2Bowtie2BWA-MEM3 more tools
Codex GPT-5.6 Luna

Replicate 2

Matches prediction
Answer10;30;60
Workflow
Local codeMinimap2BWA-MEMBowtie2Kraken21 more tools
Codex GPT-5.6 Luna

Replicate 3

Matches prediction
Answer10;30;60
Workflow
Local codeMinimap2Bowtie2BWA-MEMKraken22 more tools

Codex GPT-6 Astra

1 replicate
Codex GPT-6 Astra

Replicate 1

Matches prediction
Answer10;30;60
Workflow
Local codePythonMinimap2BWA-MEMBowtie21 more tools
Total tokens
671.1K
Output
6.4K