IWC benchmark item

wf_002_rnaseq_de_visualization

RNA-Seq Differential Expression Analysis with Visualization

RNA-Seq Differential Expression Analysis

Open source IWC workflowGround-truth Galaxy history

Reference workflow

17 tool steps across 13 unique tools, including tools inside nested subworkflows. A plus sign denotes parallel branches.

Workflow
Create text file + Differential Analysis + Compose text parameter valueText transformation + Annotate DESeq2/DEXSeq output tables + Text reformattingAnnotate DESeq2 table + Parse parameter valueFilter with p-adj threshold + Generate Volcanot plot of DE genes + Compose text parameter valueFilter with log2 FC thresholdJoin two DatasetsCutGenerate Heatmap of counts + Generate Heatmap of Z-scores

Complete query

The full task shown to the agent, excluding execution-condition instructions.

Using the provided raw count tables and two-condition design, perform a standard differential-expression analysis for raw count data, comparing the changed condition against the reference condition.
Before testing, retain a gene only when it has at least 10 raw counts in at least two of the four samples. Test every retained gene and no others.
Report an unshrunken log2 fold change for each retained gene, positive when expression is higher in the changed condition, together with a raw p-value and a Benjamini-Hochberg adjusted p-value computed across the retained genes.
Write `final_answer/differential_expression.tsv` with exactly four tab-separated columns named `gene_id`, `log2_fold_change`, `p_value` and `fdr`. Include one row for every retained gene and no other rows, and do not remove rows by significance. Use the gene identifiers exactly as they appear in the input count tables. Every log2 fold change must be finite, and every p-value and adjusted p-value must lie between 0 and 1.
Write `final_answer/method.json` as a JSON object with a single key `method` whose value is the common name of the differential-expression method you used.

acc evaluator

acc compares the final output files with the reference outputs.

Metric
differential_expression_continuous
Definition

Continuous per-gene agreement over the fixed prefiltered gene universe. Reported as five separate components combined by geometric mean: gene-set coverage, fold-change direction on effect-bearing genes, fold-change magnitude by Lin's concordance correlation coefficient, fold-change ranking, and the ranking of statistical evidence. Magnitude deliberately is not a correlation - correlation is invariant to multiplying every fold change by a constant, and CCC is not.

Formula
Final scoreS = (G × D × M × R × E)^(1/5), or 0 if any component ≤ 0

Five continuous components are combined by geometric mean.

Gene coverageG = F1(|Genes_C ∩ Genes_R|, |Genes_C|, |Genes_R|)

Gene identifiers are compared over the fixed tested-gene universe.

DirectionD = max(0, 2A − 1)

A is sign agreement on reference genes with |log2FC| ≥ 0.5; random 50% agreement maps to zero.

MagnitudeM = max(0, 2 cov(x,y) / (var(x) + var(y) + (μ_x − μ_y)²))

Lin's concordance correlation coefficient over the complete reference gene universe; missing candidate genes contribute log2FC=0.

Fold-change rankR = max(0, Spearman(x_effect, y_effect))

Ranking is evaluated on reference effect-bearing genes.

Evidence rankE = max(0, Spearman(−log10 p_R, −log10 p_C))

Missing candidate genes contribute zero evidence.

BH checkmax_g |FDR_g − BH(p)_g| ≤ 10⁻⁶

The submitted adjusted p-values must be the Benjamini-Hochberg adjustment of the submitted raw p-values.

Step-score requirements

A step passes when the trace directly shows that it ran. Parameter correctness and output quality are measured separately by acc.

  1. Step 1Build the correct tested-gene universe and two-condition comparison from the raw count tables, including the fixed prefilter.
    • Raw counts and sample-condition assignments were loaded without substituting normalized values.
    • The fixed prefilter retained genes with count at least 10 in at least 2 of the 4 samples.
    • The tested comparison used the two requested conditions and the retained-gene universe.
  2. Step 2Perform differential-expression analysis for all retained genes and compute unshrunken log2 fold changes, raw p-values, and BH-adjusted p-values.
    • A successful count-based negative-binomial differential-expression analysis covered every retained gene.
    • Reported log2 fold changes are unshrunken and use the requested direction.
    • Raw p-values and Benjamini-Hochberg adjusted p-values were computed for the same tested universe.
  3. Step 3Produce the complete per-gene differential-expression table with the required gene identifiers and method declaration.
    • The final table has exactly the required columns and one row per retained gene with the required identifier.
    • The method declaration identifies the actual method and comparison used.
    • Required final artifacts pass their schema and final-answer checks.

Model results and score explanations

The 6 results below acc 1.000 each include a score explanation and quantitative evidence. A score below 1.000 is not automatically an error. Each replicate shows its recorded workflow, acc, step score, and Hugging Face trace. Galaxy histories can be imported by link but are not published in Galaxy's public history listing.

GPT-5.5

Open-ended code

R1acc · step score 1.000
0.999966
Recorded workflow
Run PyDESeq2 differential expressionRun DESeq2 differential expressionWrite the complete per-gene differential-expression table
All 3 recorded steps
  1. Run PyDESeq2 differential expression
  2. Run DESeq2 differential expression
  3. Write the complete per-gene differential-expression table

Score explanation

DESeq2 implementation difference The Python DESeq2 implementation reproduced the complete gene set and every effect direction, but its numerical fitting made one borderline significance decision differently. The remaining deficit comes from small p-value and ranking differences, not gene filtering. Evidence. All 2,526 genes matched, with 12 significant calls versus 13; p-value evidence was 0.999861, magnitude concordance 0.999996, and rank similarity 0.999973.
Hugging Face trace
R2acc · step score 1.000
1.000
Recorded workflow
Build the tested-gene universe and apply the count prefilterRun DESeq2 differential expressionCompute raw p-values and BH-adjusted p-valuesWrite the complete per-gene differential-expression table
All 4 recorded steps
  1. Build the tested-gene universe and apply the count prefilter
  2. Run DESeq2 differential expression
  3. Compute raw p-values and BH-adjusted p-values
  4. Write the complete per-gene differential-expression table
Hugging Face trace
R3acc · step score 1.000
0.999966
Recorded workflow
Run PyDESeq2 differential expressionRun DESeq2 differential expressionWrite the complete per-gene differential-expression table
All 3 recorded steps
  1. Run PyDESeq2 differential expression
  2. Run DESeq2 differential expression
  3. Write the complete per-gene differential-expression table

Score explanation

DESeq2 implementation difference The Python DESeq2 implementation reproduced the complete gene set and every effect direction, but its numerical fitting made one borderline significance decision differently. The remaining deficit comes from small p-value and ranking differences, not gene filtering. Evidence. All 2,526 genes matched, with 12 significant calls versus 13; p-value evidence was 0.999861, magnitude concordance 0.999996, and rank similarity 0.999973.
Hugging Face trace

Galaxy-API code

R1acc · step score 1.000
1.000
Recorded workflow
Data retrieval ×4DESeq2
All 5 recorded steps
  1. Data retrieval
  2. Data retrieval
  3. Data retrieval
  4. Data retrieval
  5. DESeq2
Galaxy historyHugging Face trace
R2acc · step score 1.000
1.000
Recorded workflow
Data retrieval ×4DESeq2
All 5 recorded steps
  1. Data retrieval
  2. Data retrieval
  3. Data retrieval
  4. Data retrieval
  5. DESeq2
Galaxy historyHugging Face trace
R3acc · step score 1.000
1.000000
Recorded workflow
Data retrieval ×4DESeq2
All 5 recorded steps
  1. Data retrieval
  2. Data retrieval
  3. Data retrieval
  4. Data retrieval
  5. DESeq2

Score explanation

Floating-point precision The DESeq2 result is scientifically identical to the benchmark target. The value below 1.000 comes only from floating-point serialization of otherwise matching statistics. Evidence. All 2,526 genes and all 13 significant calls matched; direction, coverage, and rank components are 1.000000, while p-value evidence is 0.999999996.
Galaxy historyHugging Face trace

GPT-5.6 Sol

Open-ended code

R1acc · step score 1.000
1.000
Recorded workflow
Build the tested-gene universe and apply the count prefilterRun DESeq2 differential expressionCompute raw p-values and BH-adjusted p-valuesWrite the complete per-gene differential-expression table
All 4 recorded steps
  1. Build the tested-gene universe and apply the count prefilter
  2. Run DESeq2 differential expression
  3. Compute raw p-values and BH-adjusted p-values
  4. Write the complete per-gene differential-expression table
Hugging Face trace
R2acc · step score 1.000
1.000
Recorded workflow
Input discovery — Table transformationSoftware setup micromamba — DESeq2Differential expression — DESeq2Output validation — DESeq2
All 4 recorded steps
  1. Input discovery — Table transformation
  2. Software setup micromamba — DESeq2
  3. Differential expression — DESeq2
  4. Output validation — DESeq2
Hugging Face trace
R3acc · step score 1.000
1.000
Recorded workflow
Build the tested-gene universe and apply the count prefilterRun DESeq2 differential expressionCompute raw p-values and BH-adjusted p-valuesWrite the complete per-gene differential-expression table
All 4 recorded steps
  1. Build the tested-gene universe and apply the count prefilter
  2. Run DESeq2 differential expression
  3. Compute raw p-values and BH-adjusted p-values
  4. Write the complete per-gene differential-expression table
Hugging Face trace

Galaxy-API code

R1acc · step score 1.000
1.000
Recorded workflow
Query TabularDESeq2 ×2Query Tabular
All 4 recorded steps
  1. Query Tabular
  2. DESeq2
  3. DESeq2
  4. Query Tabular
Galaxy historyHugging Face trace
R2acc · step score 1.000
1.000
Recorded workflow
Join1 ×3Filter rows ×2Select columns ×8DESeq2
All 14 recorded steps
  1. Join1
  2. Join1
  3. Join1
  4. Filter rows
  5. Filter rows
  6. Select columns
  7. Select columns
  8. Select columns
  9. Select columns
  10. Select columns
  11. Select columns
  12. Select columns
  13. Select columns
  14. DESeq2
Galaxy historyHugging Face trace
R3acc · step score 1.000
1.000
Recorded workflow
Paste1 ×3Filter rowsSelect columns ×4Filter rows ×4DESeq2Filter rows ×4Select columns ×4
All 21 recorded steps
  1. Paste1
  2. Paste1
  3. Paste1
  4. Filter rows
  5. Select columns
  6. Select columns
  7. Select columns
  8. Select columns
  9. Filter rows
  10. Filter rows
  11. Filter rows
  12. Filter rows
  13. DESeq2
  14. Filter rows
  15. Filter rows
  16. Filter rows
  17. Filter rows
  18. Select columns
  19. Select columns
  20. Select columns
  21. Select columns
Galaxy historyHugging Face trace

GPT-5.6 Luna

Open-ended code

R1acc · step score 1.000
1.000
Recorded workflow
Build the tested-gene universe and apply the count prefilterRun DESeq2 differential expressionCompute raw p-values and BH-adjusted p-valuesWrite the complete per-gene differential-expression table
All 4 recorded steps
  1. Build the tested-gene universe and apply the count prefilter
  2. Run DESeq2 differential expression
  3. Compute raw p-values and BH-adjusted p-values
  4. Write the complete per-gene differential-expression table
Hugging Face trace
R2acc · step score 1.000
1.000
Recorded workflow
Build the tested-gene universe and apply the count prefilterRun DESeq2 differential expressionWrite the complete per-gene differential-expression table
All 3 recorded steps
  1. Build the tested-gene universe and apply the count prefilter
  2. Run DESeq2 differential expression
  3. Write the complete per-gene differential-expression table
Hugging Face trace
R3acc · step score 1.000
1.000
Recorded workflow
Build the tested-gene universe and apply the count prefilterRun DESeq2 differential expressionCompute raw p-values and BH-adjusted p-valuesWrite the complete per-gene differential-expression table
All 4 recorded steps
  1. Build the tested-gene universe and apply the count prefilter
  2. Run DESeq2 differential expression
  3. Compute raw p-values and BH-adjusted p-values
  4. Write the complete per-gene differential-expression table
Hugging Face trace

Galaxy-API code

R1acc · step score 1.000
0.994
Recorded workflow
Tp Head ToolData retrievalPaste1 ×3Select columnsData retrievaledgeR13 more steps
All 23 recorded steps
  1. Tp Head Tool
  2. Data retrieval
  3. Paste1
  4. Paste1
  5. Paste1
  6. Select columns
  7. Data retrieval
  8. edgeR
  9. Tp Head Tool
  10. Tp Head Tool
  11. Tp Head Tool
  12. Remove Beginning1
  13. Tp Head Tool
  14. Remove Beginning1
  15. Tp Head Tool
  16. Remove Beginning1
  17. Tp Head Tool
  18. Remove Beginning1
  19. Tp Head Tool
  20. Remove Beginning1
  21. Tp Head Tool
  22. Remove Beginning1
  23. Tp Head Tool

Score explanation

Differential-expression method The edgeR model estimated dispersion and significance differently from the DESeq2 benchmark target. Gene coverage and effect direction are complete, so the deficit is statistical-method disagreement rather than a missing or filtered output. Evidence. All 2,526 genes and their effect directions matched, but edgeR called 37 significant genes versus 13; p-value evidence similarity was 0.981049 and rank similarity was 0.991610.
Galaxy historyHugging Face trace
R2acc · step score 1.000
1.000
Recorded workflow
Collection Column JoinFilter rowsSelect columns ×4DESeq2Select columns
All 8 recorded steps
  1. Collection Column Join
  2. Filter rows
  3. Select columns
  4. Select columns
  5. Select columns
  6. Select columns
  7. DESeq2
  8. Select columns
Galaxy historyHugging Face trace
R3acc · step score 1.000
1.000
Recorded workflow
Collection Column JoinTable Compute ×2Select columns ×8DESeq2Table transformationDESeq2Table transformation ×11
All 25 recorded steps
  1. Collection Column Join
  2. Table Compute
  3. Table Compute
  4. Select columns
  5. Select columns
  6. Select columns
  7. Select columns
  8. Select columns
  9. Select columns
  10. Select columns
  11. Select columns
  12. DESeq2
  13. Table transformation
  14. DESeq2
  15. Table transformation
  16. Table transformation
  17. Table transformation
  18. Table transformation
  19. Table transformation
  20. Table transformation
  21. Table transformation
  22. Table transformation
  23. Table transformation
  24. Table transformation
  25. Table transformation
Galaxy historyHugging Face trace

Codex + DeepSeek V4 Pro

Open-ended code

R1acc · step score 1.000
1.000
Recorded workflow
Build the tested-gene universe and apply the count prefilterRun PyDESeq2 differential expressionRun DESeq2 differential expressionCompute raw p-values and BH-adjusted p-valuesWrite the complete per-gene differential-expression table
All 5 recorded steps
  1. Build the tested-gene universe and apply the count prefilter
  2. Run PyDESeq2 differential expression
  3. Run DESeq2 differential expression
  4. Compute raw p-values and BH-adjusted p-values
  5. Write the complete per-gene differential-expression table
Hugging Face trace
R2acc · step score 1.000
0.999966
Recorded workflow
Build the tested-gene universe and apply the count prefilterRun PyDESeq2 differential expressionRun DESeq2 differential expressionCompute raw p-values and BH-adjusted p-valuesWrite the complete per-gene differential-expression table
All 5 recorded steps
  1. Build the tested-gene universe and apply the count prefilter
  2. Run PyDESeq2 differential expression
  3. Run DESeq2 differential expression
  4. Compute raw p-values and BH-adjusted p-values
  5. Write the complete per-gene differential-expression table

Score explanation

DESeq2 implementation difference The Python DESeq2 implementation reproduced the complete gene set and every effect direction, but its numerical fitting made one borderline significance decision differently. The remaining deficit comes from small p-value and ranking differences, not gene filtering. Evidence. All 2,526 genes matched, with 12 significant calls versus 13; p-value evidence was 0.999861, magnitude concordance 0.999996, and rank similarity 0.999973.
Hugging Face trace
R3acc · step score 1.000
0.999967
Recorded workflow
Build the tested-gene universe and apply the count prefilterRun PyDESeq2 differential expressionRun DESeq2 differential expressionCompute raw p-values and BH-adjusted p-valuesWrite the complete per-gene differential-expression table
All 5 recorded steps
  1. Build the tested-gene universe and apply the count prefilter
  2. Run PyDESeq2 differential expression
  3. Run DESeq2 differential expression
  4. Compute raw p-values and BH-adjusted p-values
  5. Write the complete per-gene differential-expression table

Score explanation

DESeq2 implementation difference The Python DESeq2 implementation reproduced the complete gene set and every effect direction, but its numerical fitting made one borderline significance decision differently. The remaining deficit comes from small p-value and ranking differences, not gene filtering. Evidence. All 2,526 genes matched, with 12 significant calls versus 13; p-value evidence was 0.999866, magnitude concordance 0.999996, and rank similarity 0.999974.
Hugging Face trace

Galaxy-API code

R1acc · step score 1.000
1.000
Recorded workflow
DESeq2Data retrieval ×4DESeq2
All 6 recorded steps
  1. DESeq2
  2. Data retrieval
  3. Data retrieval
  4. Data retrieval
  5. Data retrieval
  6. DESeq2
Galaxy historyHugging Face trace
R2acc · step score 1.000
1.000
Recorded workflow
Join1 ×3Select columnsTable ComputeSelect columns ×4Regexcolumn1 ×2DESeq2
All 12 recorded steps
  1. Join1
  2. Join1
  3. Join1
  4. Select columns
  5. Table Compute
  6. Select columns
  7. Select columns
  8. Select columns
  9. Select columns
  10. Regexcolumn1
  11. Regexcolumn1
  12. DESeq2
Galaxy historyHugging Face trace
R3acc · step score 1.000
1.000
Recorded workflow
DESeq2Data retrieval ×4
All 5 recorded steps
  1. DESeq2
  2. Data retrieval
  3. Data retrieval
  4. Data retrieval
  5. Data retrieval
Galaxy historyHugging Face trace