IWC multi-step task

Task 1: Genome profiling and HiFi assembly

What the task asks for

Estimate genome size and heterozygosity from k-mers, assemble primary and alternate contigs from the HiFi reads of a green alga, and check both assemblies with read k-mers and conserved genes.

8 scored results8 required18 runs scored

Codex GPT-5.6 Luna

6 of 6 runs scored
Galaxy-API code 0.583 BixBench3 acc (share of results that pass), averaged over 3 of 3 runs continuous score 0.635
All results pass0 of 3 runsruns in which every scored result passes
run 10.7506/8run 20.3753/8run 30.6255/8
Open-ended code 0.167 BixBench3 acc (share of results that pass), averaged over 3 of 3 runs continuous score 0.190
All results pass0 of 3 runsruns in which every scored result passes
run 10.2502/8run 20.1251/8run 30.1251/8

Codex GPT-5.6 Sol

6 of 6 runs scored
Galaxy-API code 1.000 BixBench3 acc (share of results that pass), averaged over 3 of 3 runs continuous score 1.000
All results pass3 of 3 runsruns in which every scored result passes
run 11.0008/8run 21.0008/8run 31.0008/8
Open-ended code 1.000 BixBench3 acc (share of results that pass), averaged over 3 of 3 runs continuous score 0.981
All results pass3 of 3 runsruns in which every scored result passes
run 11.0008/8run 21.0008/8run 31.0008/8

DeepSeek V4 Pro via Codex

6 of 6 runs scored
Galaxy-API code 0.458 BixBench3 acc (share of results that pass), averaged over 3 of 3 runs continuous score 0.489
All results pass0 of 3 runsruns in which every scored result passes
run 10.3753/8run 20.6255/8run 30.3753/8
Open-ended code 0.542 BixBench3 acc (share of results that pass), averaged over 3 of 3 runs continuous score 0.554
All results pass0 of 3 runsruns in which every scored result passes
run 10.6255/8run 20.8757/8run 30.1251/8

Three measures. Every required result gets a score from 0 to 1 by comparison with the reference. Passing requires a score of at least 0.80 and compliance with the required content, method and output instructions. BixBench3 acc is the share of results that pass. The continuous score is the plain average of all the result scores, with no pass mark. For example, run 1 of GPT-5.6 Luna with Galaxy-API code scored 1.000, 1.000, 0.000, 1.000, 1.000, 1.000, 1.000 and 0.413 on its 8 results. 6 of them pass, so BixBench3 acc = 6/8 = 0.750; the average of all 8 scores is 0.802, the continuous score. A third measure, all results pass, counts a run only when every scored result passes, which is the share of runs that delivered the whole analysis correctly. A failing result still contributes its numerical score to the continuous average. A score above 0.80 does not excuse an incorrect method, missing member or an explicitly required output format. Each card averages the scored runs of this task.

Error analysis

BixBench3 acc, Galaxy-API code vs Open-ended code: GPT-5.6 Luna 0.583 vs 0.167; GPT-5.6 Sol 1.000 vs 1.000; DeepSeek V4 Pro 0.458 vs 0.542.

What Galaxy added

Every step the task describes, from adapter filtering and the k-mer genome model to assembly statistics and the conserved-gene check, is a ready Galaxy tool. In Open-ended code, GPT-5.6 Luna and DeepSeek V4 Pro often could not find or install these tools and wrote their own versions. Their adapter counts, genome models and statistics drifted, and wrong genome-size estimates went into the assembler settings, so the assembly results failed too.

Why Galaxy did not do better

Five Galaxy runs checked conserved genes with a different method from the one the task describes, and three reported the uncorrected genome length from the genome model. The assembler tool also reported success with empty outputs when the genome size was written in a form the assembler rejects. Every Galaxy run hit this once; most recovered, but one DeepSeek V4 Pro run switched to an older assembler version and a different genome size.

Run-level details (4)
  • In the first Open-ended code run with GPT-5.6 Luna, the agent wrote its own adapter matching and k-mer checks and returned the conserved-gene table in a format of its own.
  • In Open-ended code, the second and third runs with GPT-5.6 Luna and the third with DeepSeek V4 Pro fitted the genome model with their own code. The coverage or genome size they passed to the assembler changed the assemblies, so results 4 to 7 failed.
  • The first two Galaxy runs with DeepSeek V4 Pro and all three with GPT-5.6 Luna checked conserved genes with a different method from the one the task describes; result 8 scored about 0.41.
  • The third Galaxy run with DeepSeek V4 Pro gave the assembler a genome size in a form it rejects. The tool still reported success, with empty outputs, and the agent moved to an older assembler version and a 15 Mb genome size. The first Galaxy run with DeepSeek V4 Pro set the assembler's k-mer length to 21, and the second with GPT-5.6 Luna chose a weaker purge level.
Instruction, format and method errors (1)
  • Required file is missing: 03_genome_profile/model_parameters.txtTask requirement: 3. final_answer/03_genome_profile/ native text reports model_parameters.txt: the six fitted point estimates as one tab-separated record, in order: repetitiveness (duplication) parameter, heterozygosity rate, per-copy k-mer coverage, overdispersion (bias) parameter, haploid length implied by the non-error k-mers at the fitted coverage, and model topology (0 for this diploid model); summary.txt: lower and upper bounds for homozygosity, heterozygosity, haploid/repeat/unique length, model fit and read error. Diagnostic plots may accompany the reports. Files: model_parameters.txt, summary.txt.

    GPT-5.6 Luna / Galaxy-API code / run 3 → result 3

Every run

The task asks the agent to return a numbered list of results, each a file or a folder of files. The columns on the right give the score of each required result in each run, from 0 to 1 against the reference, under the result's number and name; the task text and the scoring table below describe each one. Green marks a passing result. Failing cells link to the reasons below, including instruction and format errors even when the numerical score is at least 0.80. In each row, BixBench3 acc is the number of green cells divided by the number of results, and the continuous score is the average of all the result scores in the row. Tokens are input (including cached input) plus output tokens; minutes are the wall time of the run. Under Records, Trace opens the run's task text, agent event log, final message and returned files in the lab's run trace dataset on Hugging Face, and History opens the usegalaxy.org history that a Galaxy-API code run worked in.

Score of each required result
ModelConditionRunBixBench3 acc
share passing		Continuous score TokensMinutesRecords1read QC2k-mer histogram3genome profile4assembly graphs5assemblies6assembly statistics7k-mer validation8ortholog validation
GPT-5.6 LunaGalaxy-API code1trial run6/8 = 0.7500.80215.8M272Trace
History		1.00 Passed1.00 Passed0.00Content1.00 Passed1.00 Passed1.00 Passed1.00 Passed0.41Content
23/8 = 0.3750.42713.9M239Trace
History		1.00 Passed1.00 Passed1.00 Passed0.00Content0.00Content0.00Content0.00Content0.41Content
35/8 = 0.6250.67715.4M280Trace
History		0.00Content1.00 Passed0.00Instruction1.00 Passed1.00 Passed1.00 Passed1.00 Passed0.41Content
Open-ended code1trial run2/8 = 0.2500.25096.2M316Trace0.00Content1.00 Passed1.00 Passed0.00Content0.00Content0.00Content0.00Content0.00Content
21/8 = 0.1250.19634.1M155Trace0.00Content1.00 Passed0.00Content0.00Content0.00Content0.00Content0.15Content0.41Content
31/8 = 0.1250.12528.7M303Trace0.00Content1.00 Passed0.00Content0.00Content0.00Content0.00Content0.00Content0.00Content
GPT-5.6 SolGalaxy-API code18/8 = 1.0001.00011.0M139Trace
History		1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed0.99 Passed
28/8 = 1.0001.0008.3M217Trace
History		1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed0.99 Passed
38/8 = 1.0001.00012.6M158Trace
History		1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed0.99 Passed
Open-ended code18/8 = 1.0000.98114.3M148Trace1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed0.84 Passed0.99 Passed
28/8 = 1.0000.98218.5M151Trace1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed0.85 Passed0.99 Passed
38/8 = 1.0000.9819.3M144Trace1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed0.84 Passed0.99 Passed
DeepSeek V4 ProGalaxy-API code13/8 = 0.3750.44011.6M158Trace
History		1.00 Passed1.00 Passed1.00 Passed0.00Content0.00Content0.00Content0.11Content0.40Content
25/8 = 0.6250.67710.2M200Trace
History		1.00 Passed1.00 Passed0.00Content1.00 Passed1.00 Passed0.00Content1.00 Passed0.41Content
33/8 = 0.3750.35219.4M298Trace
History		1.00 Passed1.00 Passed0.00Content0.00Content0.00Content0.00Content0.00Content0.80 Passed
Open-ended code15/8 = 0.6250.66131.6M169Trace1.00 Passed1.00 Passed0.00Content1.00 Passed1.00 Passed0.00Content0.84 Passed0.44Content
27/8 = 0.8750.85637.2M152Trace1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed0.00Content0.85 Passed0.99 Passed
31/8 = 0.1250.14425.7M195Trace0.00Content1.00 Passed0.00Content0.00Content0.00Content0.00Content0.15Content0.00Content

Why results did not pass

  1. GPT-5.6 Luna / Galaxy-API code / run 1 → result 3 (genome profile)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Summary statistics within tolerance: 0
  2. GPT-5.6 Luna / Galaxy-API code / run 1 → result 8 (ortholog validation)
    • Content: Content score below 0.80. Value: 0.413931. Required minimum: 0.8.
      • Category agreement (F1): 0.938026 to 0.954505 (2 checks)
      • Matching records (F1): 0.994468 to 0.998757 (2 checks)
      • Numerical agreement (CCC): 0.713387 to 0.965533 (4 checks)
      • Present, finite data cells (F1): 0.993425 to 0.99858 (2 checks)
  3. GPT-5.6 Luna / Galaxy-API code / run 2 → result 4 (assembly graphs)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Summary statistics within tolerance: 0 (4 checks)
  4. GPT-5.6 Luna / Galaxy-API code / run 2 → result 5 (assemblies)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Summary statistics within tolerance: 0 (5 checks)
  5. GPT-5.6 Luna / Galaxy-API code / run 2 → result 6 (assembly statistics)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Numerical agreement (CCC): 0.992224 to 0.999865 (2 checks)
      • Summary statistics within tolerance: 0 (54 checks)
  6. GPT-5.6 Luna / Galaxy-API code / run 2 → result 7 (k-mer validation)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Category agreement (F1): 0.546918 to 0.608163 (2 checks)
      • Matching records (F1): 0.955224 to 0.997173 (4 checks)
      • Numerical agreement (CCC): 0.42266 to 0.999998 (6 checks)
      • Present, finite data cells (F1): 0.916667 to 0.997173 (4 checks)
      • Required content missing or unreadable: 0 (6 checks)
  7. GPT-5.6 Luna / Galaxy-API code / run 2 → result 8 (ortholog validation)
    • Content: Content score below 0.80. Value: 0.414702. Required minimum: 0.8.
      • Category agreement (F1): 0.938616 to 0.955404 (2 checks)
      • Matching records (F1): 0.994163 to 0.999067 (2 checks)
      • Numerical agreement (CCC): 0.713388 to 0.965495 (4 checks)
      • Present, finite data cells (F1): 0.993425 to 0.998934 (2 checks)
  8. GPT-5.6 Luna / Galaxy-API code / run 3 → result 1 (read QC)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Summary statistics within tolerance: 0
  9. GPT-5.6 Luna / Galaxy-API code / run 3 → result 3 (genome profile)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
    • Instruction: Required file is missing: 03_genome_profile/model_parameters.txtTask requirement: 3. final_answer/03_genome_profile/ native text reports model_parameters.txt: the six fitted point estimates as one tab-separated record, in order: repetitiveness (duplication) parameter, heterozygosity rate, per-copy k-mer coverage, overdispersion (bias) parameter, haploid length implied by the non-error k-mers at the fitted coverage, and model topology (0 for this diploid model); summary.txt: lower and upper bounds for homozygosity, heterozygosity, haploid/repeat/unique length, model fit and read error. Diagnostic plots may accompany the reports. Files: model_parameters.txt, summary.txt.
  10. GPT-5.6 Luna / Galaxy-API code / run 3 → result 8 (ortholog validation)
    • Content: Content score below 0.80. Value: 0.413931. Required minimum: 0.8.
      • Category agreement (F1): 0.938026 to 0.954505 (2 checks)
      • Matching records (F1): 0.994468 to 0.998757 (2 checks)
      • Numerical agreement (CCC): 0.713387 to 0.965533 (4 checks)
      • Present, finite data cells (F1): 0.993425 to 0.99858 (2 checks)
  11. GPT-5.6 Luna / Open-ended code / run 1 → result 1 (read QC)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Accounting: 0 (2 checks)
      • Matching records (F1): 0.999929
  12. GPT-5.6 Luna / Open-ended code / run 1 → result 4 (assembly graphs)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Summary statistics within tolerance: 0 (12 checks)
  13. GPT-5.6 Luna / Open-ended code / run 1 → result 5 (assemblies)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Summary statistics within tolerance: 0 (3 checks)
  14. GPT-5.6 Luna / Open-ended code / run 1 → result 6 (assembly statistics)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Numerical agreement (CCC): 0.986348 to 0.999973 (2 checks)
      • Summary statistics within tolerance: 0 (78 checks)
  15. GPT-5.6 Luna / Open-ended code / run 1 → result 7 (k-mer validation)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0 (12 checks)
  16. GPT-5.6 Luna / Open-ended code / run 1 → result 8 (ortholog validation)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Numerical agreement (CCC): 0.465314 to 0.506879 (2 checks)
      • Required content missing or unreadable: 0 (2 checks)
  17. GPT-5.6 Luna / Open-ended code / run 2 → result 1 (read QC)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Summary statistics within tolerance: 0
  18. GPT-5.6 Luna / Open-ended code / run 2 → result 3 (genome profile)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
      • Summary statistics within tolerance: 0 (5 checks)
  19. GPT-5.6 Luna / Open-ended code / run 2 → result 4 (assembly graphs)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0 (4 checks)
  20. GPT-5.6 Luna / Open-ended code / run 2 → result 5 (assemblies)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Summary statistics within tolerance: 0 (3 checks)
  21. GPT-5.6 Luna / Open-ended code / run 2 → result 6 (assembly statistics)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Numerical agreement (CCC): 0.993087 to 0.999985 (2 checks)
      • Summary statistics within tolerance: 0 (142 checks)
  22. GPT-5.6 Luna / Open-ended code / run 2 → result 7 (k-mer validation)
    • Content: Content score below 0.80. Value: 0.153511. Required minimum: 0.8.
      • Category agreement (F1): 0.653476 to 0.952935 (2 checks)
      • Matching records (F1): 0.936014 to 0.999084 (5 checks)
      • Numerical agreement (CCC): 0.449433 to 1 (12 checks)
      • Present, finite data cells (F1): 0.936014 to 0.999084 (6 checks)
  23. GPT-5.6 Luna / Open-ended code / run 2 → result 8 (ortholog validation)
    • Content: Content score below 0.80. Value: 0.410969. Required minimum: 0.8.
      • Category agreement (F1): 0.938026 to 0.951617 (2 checks)
      • Matching records (F1): 0.994468 to 0.997823 (2 checks)
      • Numerical agreement (CCC): 0.713359 to 0.9655 (4 checks)
      • Present, finite data cells (F1): 0.993425 to 0.997512 (2 checks)
  24. GPT-5.6 Luna / Open-ended code / run 3 → result 1 (read QC)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Accounting: 0 (2 checks)
      • Matching records (F1): 0.999966
      • Summary statistics within tolerance: 0
  25. GPT-5.6 Luna / Open-ended code / run 3 → result 3 (genome profile)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
      • Summary statistics within tolerance: 0 (5 checks)
  26. GPT-5.6 Luna / Open-ended code / run 3 → result 4 (assembly graphs)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Summary statistics within tolerance: 0 (11 checks)
  27. GPT-5.6 Luna / Open-ended code / run 3 → result 5 (assemblies)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Summary statistics within tolerance: 0 (4 checks)
  28. GPT-5.6 Luna / Open-ended code / run 3 → result 6 (assembly statistics)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Numerical agreement (CCC): 0.852687 to 0.999885 (2 checks)
      • Summary statistics within tolerance: 0 (123 checks)
  29. GPT-5.6 Luna / Open-ended code / run 3 → result 7 (k-mer validation)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0 (12 checks)
  30. GPT-5.6 Luna / Open-ended code / run 3 → result 8 (ortholog validation)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Numerical agreement (CCC): 0.967566 to 0.991412 (2 checks)
      • Required content missing or unreadable: 0 (2 checks)
  31. DeepSeek V4 Pro / Galaxy-API code / run 1 → result 4 (assembly graphs)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Summary statistics within tolerance: 0 (10 checks)
  32. DeepSeek V4 Pro / Galaxy-API code / run 1 → result 5 (assemblies)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Summary statistics within tolerance: 0
  33. DeepSeek V4 Pro / Galaxy-API code / run 1 → result 6 (assembly statistics)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Numerical agreement (CCC): 0.999557 to 0.999607 (2 checks)
      • Summary statistics within tolerance: 0 (29 checks)
  34. DeepSeek V4 Pro / Galaxy-API code / run 1 → result 7 (k-mer validation)
    • Content: Content score below 0.80. Value: 0.113227. Required minimum: 0.8.
      • Category agreement (F1): 0.732143 to 0.760264 (2 checks)
      • Matching records (F1): 0.942684 to 0.991098 (5 checks)
      • Numerical agreement (CCC): 0.657467 to 1 (12 checks)
      • Present, finite data cells (F1): 0.942684 to 0.986963 (6 checks)
  35. DeepSeek V4 Pro / Galaxy-API code / run 1 → result 8 (ortholog validation)
    • Content: Content score below 0.80. Value: 0.407149. Required minimum: 0.8.
      • Category agreement (F1): 0.938007 to 0.947313 (2 checks)
      • Matching records (F1): 0.993861 to 0.995012 (2 checks)
      • Numerical agreement (CCC): 0.713315 to 0.965423 (4 checks)
      • Present, finite data cells (F1): 0.993108 to 0.994298 (2 checks)
  36. DeepSeek V4 Pro / Galaxy-API code / run 2 → result 3 (genome profile)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Summary statistics within tolerance: 0
  37. DeepSeek V4 Pro / Galaxy-API code / run 2 → result 6 (assembly statistics)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Summary statistics within tolerance: 0 (6 checks)
  38. DeepSeek V4 Pro / Galaxy-API code / run 2 → result 8 (ortholog validation)
    • Content: Content score below 0.80. Value: 0.413931. Required minimum: 0.8.
      • Category agreement (F1): 0.938026 to 0.954505 (2 checks)
      • Matching records (F1): 0.994468 to 0.998757 (2 checks)
      • Numerical agreement (CCC): 0.713387 to 0.965533 (4 checks)
      • Present, finite data cells (F1): 0.993425 to 0.99858 (2 checks)
  39. DeepSeek V4 Pro / Galaxy-API code / run 3 → result 3 (genome profile)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Summary statistics within tolerance: 0
  40. DeepSeek V4 Pro / Galaxy-API code / run 3 → result 4 (assembly graphs)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Summary statistics within tolerance: 0 (8 checks)
  41. DeepSeek V4 Pro / Galaxy-API code / run 3 → result 5 (assemblies)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Summary statistics within tolerance: 0
  42. DeepSeek V4 Pro / Galaxy-API code / run 3 → result 6 (assembly statistics)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Numerical agreement (CCC): 0.988716 to 0.999631 (2 checks)
      • Summary statistics within tolerance: 0 (65 checks)
  43. DeepSeek V4 Pro / Galaxy-API code / run 3 → result 7 (k-mer validation)
    • Content: Content score below 0.80. Value: 0.00229672. Required minimum: 0.8.
      • Category agreement (F1): 0.444903 to 0.618056 (2 checks)
      • Matching records (F1): 0.927306 to 0.997067 (6 checks)
      • Numerical agreement (CCC): 0.33917 to 1 (12 checks)
      • Present, finite data cells (F1): 0.927306 to 0.987305 (6 checks)
  44. DeepSeek V4 Pro / Open-ended code / run 1 → result 3 (genome profile)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
      • Summary statistics within tolerance: 0 (3 checks)
  45. DeepSeek V4 Pro / Open-ended code / run 1 → result 6 (assembly statistics)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Summary statistics within tolerance: 0 (140 checks)
  46. DeepSeek V4 Pro / Open-ended code / run 1 → result 8 (ortholog validation)
    • Content: Content score below 0.80. Value: 0.441453. Required minimum: 0.8.
      • Category agreement (F1): 0.938026 to 0.954505 (2 checks)
      • Matching records (F1): 0.994468 to 0.998757 (2 checks)
      • Numerical agreement (CCC): 0.71336 to 0.989294 (4 checks)
      • Present, finite data cells (F1): 0.993425 to 0.99858 (2 checks)
  47. DeepSeek V4 Pro / Open-ended code / run 2 → result 6 (assembly statistics)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Summary statistics within tolerance: 0 (2 checks)
  48. DeepSeek V4 Pro / Open-ended code / run 3 → result 1 (read QC)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Summary statistics within tolerance: 0
  49. DeepSeek V4 Pro / Open-ended code / run 3 → result 3 (genome profile)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
      • Summary statistics within tolerance: 0 (4 checks)
  50. DeepSeek V4 Pro / Open-ended code / run 3 → result 4 (assembly graphs)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Summary statistics within tolerance: 0 (10 checks)
  51. DeepSeek V4 Pro / Open-ended code / run 3 → result 5 (assemblies)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Summary statistics within tolerance: 0 (3 checks)
  52. DeepSeek V4 Pro / Open-ended code / run 3 → result 6 (assembly statistics)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Numerical agreement (CCC): 0.993087 to 0.999985 (2 checks)
      • Summary statistics within tolerance: 0 (122 checks)
  53. DeepSeek V4 Pro / Open-ended code / run 3 → result 7 (k-mer validation)
    • Content: Content score below 0.80. Value: 0.153553. Required minimum: 0.8.
      • Category agreement (F1): 0.653476 to 0.952935 (2 checks)
      • Matching records (F1): 0.936014 to 0.999084 (5 checks)
      • Numerical agreement (CCC): 0.449554 to 1 (12 checks)
      • Present, finite data cells (F1): 0.936014 to 0.999084 (6 checks)
  54. DeepSeek V4 Pro / Open-ended code / run 3 → result 8 (ortholog validation)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Numerical agreement (CCC): 0.996593 to 0.999886 (2 checks)
      • Required content missing or unreadable: 0 (2 checks)

Task text

The scientific part of the task, identical in both conditions. The general submission rules and the instructions for each condition are on the overview page.

Full task text

Task

Characterize the supplied Picochlorum BPE23 HiFi library, construct primary and alternate assemblies, and assess their sequence, graph, k-mer and conserved-ortholog evidence. Explain contiguity, redundancy and completeness without calling internal consistency independent biological validation. Use k-mer multiplicity to model genome size and heterozygosity, and resolve high-accuracy long-read overlaps into primary and alternate assembly graphs. Assess allelic redundancy as well as contiguity. Use read k-mers and conserved orthologs as complementary evidence, distinguishing sequence accuracy from genome completeness.

Inputs

  • One complete HiFi read library and its file/identity manifest.
  • chlorophyta_odb10 ortholog resources from snapshot all-odb10-2026-03-19-213122.
  • The two residual-adapter definitions in adapters.fasta (protocol constants, supplied with the input manifest).
  • output-layouts.json lists required statistic labels only; it contains no measurements.

Key parameters and conditions

  • Use k=21 and fit a diploid four-component negative-binomial mixture to the k-mer multiplicity histogram, estimating heterozygous/homozygous and repeat contributions. Estimate coverage from the data without a maximum-coverage cutoff. The histogram and k-mer validation use the complete original library.
  • For assembly, discard an entire read if either supplied adapter matches anywhere in either orientation, with at least 35 nt overlap and error rate at most 0.10, allowing indels. Do not retain a trimmed remainder or add quality trimming. Retain the adapter-free reads and rejection counts.
  • Assemble using only the supplied HiFi reads with haplotype-aware error correction and phased overlap graphs, incorporating allelic-redundancy removal within assembly. Produce primary contigs plus alternate contigs, not a claim of two fully phased chromosomes; use the fitted homozygous coverage and genome-size estimates. Assess both assemblies by spliced protein-to-genome alignment with minimizer seeding, chaining and frameshift handling against the supplied lineage proteins. Distinguish single-copy, duplicated, fragmented and missing orthologs, and separately report genes whose complementary portions align at two positions.
  • Retain primary and alternate results, missing-ortholog records and zero-error QV values. No between-library comparison is required.
  • Perform three rounds of haplotype-aware read correction. Remove contained and other redundant haplotigs using the most aggressive purge regime, with a read-level similarity threshold of 0.55; do not disable allelic-redundancy removal.
  • Use canonical 21-mers without a multiplicity cap. Set the homozygous assembly coverage to floor(2 × the fitted per-copy coverage); use the upper fitted haploid-length bound as the expected genome size. Retained assembly reads must have at least one base. GFA segments must contain their nucleotide sequences, not only length placeholders.

Required results

  1. final_answer/01_read_qc/
    FASTQ + JSON + TSV
    Retained reads in retained.fastq.gz; adapter-rejection counts in adapter_rejection.json (read_counts and basepair_counts); input_statistics.tsv is a two-column metric/value table of the complete input read statistics with the input_statistics metric labels in output-layouts.json, where average per-base quality is the Phred-scaled mean per-base error probability over all bases rather than the mean of quality scores. Counts are not optional plots. Files: adapter_rejection.json, input_statistics.tsv, retained.fastq.gz.
  2. final_answer/02_kmer_histogram.tsv
    TSV
    Headerless two columns: multiplicity, count; the full histogram.
  3. final_answer/03_genome_profile/
    native text reports
    model_parameters.txt: the six fitted point estimates as one tab-separated record, in order: repetitiveness (duplication) parameter, heterozygosity rate, per-copy k-mer coverage, overdispersion (bias) parameter, haploid length implied by the non-error k-mers at the fitted coverage, and model topology (0 for this diploid model); summary.txt: lower and upper bounds for homozygosity, heterozygosity, haploid/repeat/unique length, model fit and read error. Diagnostic plots may accompany the reports. Files: model_parameters.txt, summary.txt.
  4. final_answer/04_assembly_graphs/
    GFA
    Raw and processed unitig graphs and primary/alternate contig graphs, each labeled. Files: alternate_contig_graph.gfa, primary_contig_graph.gfa, processed_unitigs.gfa, raw_unitigs.gfa.
  5. final_answer/05_assemblies/
    FASTA
    primary.fasta and alternate.fasta, including every retained contig. Files: alternate.fasta, primary.fasta.
  6. final_answer/06_assembly_statistics/
    TSV
    For each assembly, a complete metric/value statistics table and a two-column contig/length table. Include the expected genome size and its length-dependent contiguity statistics; do not substitute a FASTA index. Use the assembly_statistics metric labels in output-layouts.json. Files: alternate.lengths.tsv, alternate.statistics.tsv, primary.lengths.tsv, primary.statistics.tsv.
  7. final_answer/07_kmer_validation/
    standard k-mer report tables
    Complete copy-number and assembly spectra, auxiliary histograms, global and per-contig QV and completeness reports for primary and alternate assemblies. Preserve report content and contig identifiers. Files: alternate.only.tsv, alternate.qv.tsv, alternate.spectra-cn.tsv, completeness.tsv, dist_only.tsv, global.qv.tsv, only.tsv, primary.only.tsv, primary.qv.tsv, primary.spectra-cn.tsv, spectra-asm.tsv, spectra-cn.tsv.
  8. final_answer/08_ortholog_validation/
    native ortholog tables + FASTA
    Complete ortholog tables and translated proteins for both assemblies, with missing calls retained. Files: alternate.full_table.tsv, alternate.proteins.fasta, primary.full_table.tsv, primary.proteins.fasta.

How this task is scored

Each scored result is compared with the reference result, which the task authors made by running the IWC workflow behind this task on usegalaxy.org with the settings in the task text. A result's score is the product of the checks listed for it, each from 0 to 1. A passing result must score at least 0.80 and meet the required member checks, method and output instructions. A failing result retains its numerical score in the continuous average. The overview page describes the checks in more detail.

No.Required resultWhat the task asks for How it is scored
101_read_qc/
FASTQ + JSON + TSV		Retained reads in retained.fastq.gz; adapter-rejection counts in adapter_rejection.json (read_counts and basepair_counts); input_statistics.tsv is a two-column metric/value table of the complete input read statistics with the input_statistics metric labels in output-layouts.json, where average per-base quality is the Phred-scaled mean per-base error probability over all bases rather than the mean of quality scores. Counts are not optional plots. Files: adapter_rejection.json, input_statistics.tsv, retained.fastq.gz.
  • the same rows or records as the reference (F1 of their identifiers)
  • the required columns or fields are present (F1)
  • every reported statistic is within 1% of the reference value
202_kmer_histogram.tsv
TSV		Headerless two columns: multiplicity, count; the full histogram.
  • the same rows or records as the reference (F1 of their identifiers)
  • the required columns or fields are present (F1)
  • values are present in the same cells (F1)
  • counts or intensities agree on a log(1 + x) scale (concordance correlation)
303_genome_profile/
native text reports		model_parameters.txt: the six fitted point estimates as one tab-separated record, in order: repetitiveness (duplication) parameter, heterozygosity rate, per-copy k-mer coverage, overdispersion (bias) parameter, haploid length implied by the non-error k-mers at the fitted coverage, and model topology (0 for this diploid model); summary.txt: lower and upper bounds for homozygosity, heterozygosity, haploid/repeat/unique length, model fit and read error. Diagnostic plots may accompany the reports. Files: model_parameters.txt, summary.txt.
  • the required columns or fields are present (F1)
  • every reported statistic is within 1% of the reference value
404_assembly_graphs/
GFA		Raw and processed unitig graphs and primary/alternate contig graphs, each labeled. Files: alternate_contig_graph.gfa, primary_contig_graph.gfa, processed_unitigs.gfa, raw_unitigs.gfa.
  • the required columns or fields are present (F1)
  • every reported statistic is within 1% of the reference value
505_assemblies/
FASTA		primary.fasta and alternate.fasta, including every retained contig. Files: alternate.fasta, primary.fasta.
  • the required columns or fields are present (F1)
  • every reported statistic is within 1% of the reference value
606_assembly_statistics/
TSV		For each assembly, a complete metric/value statistics table and a two-column contig/length table. Include the expected genome size and its length-dependent contiguity statistics; do not substitute a FASTA index. Use the assembly_statistics metric labels in output-layouts.json. Files: alternate.lengths.tsv, alternate.statistics.tsv, primary.lengths.tsv, primary.statistics.tsv.
  • the same rows or records as the reference (F1 of their identifiers)
  • the required columns or fields are present (F1)
  • values are present in the same cells (F1)
  • counts or intensities agree on a log(1 + x) scale (concordance correlation)
  • every reported statistic is within 1% of the reference value
707_kmer_validation/
standard k-mer report tables		Complete copy-number and assembly spectra, auxiliary histograms, global and per-contig QV and completeness reports for primary and alternate assemblies. Preserve report content and contig identifiers. Files: alternate.only.tsv, alternate.qv.tsv, alternate.spectra-cn.tsv, completeness.tsv, dist_only.tsv, global.qv.tsv, only.tsv, primary.only.tsv, primary.qv.tsv, primary.spectra-cn.tsv, spectra-asm.tsv, spectra-cn.tsv.
  • the same rows or records as the reference (F1 of their identifiers)
  • the required columns or fields are present (F1)
  • values are present in the same cells (F1)
  • values agree (Lin's concordance correlation)
  • category labels agree (macro F1)
808_ortholog_validation/
native ortholog tables + FASTA		Complete ortholog tables and translated proteins for both assemblies, with missing calls retained. Files: alternate.full_table.tsv, alternate.proteins.fasta, primary.full_table.tsv, primary.proteins.fasta.
  • the same rows or records as the reference (F1 of their identifiers)
  • the required columns or fields are present (F1)
  • values are present in the same cells (F1)
  • values agree (Lin's concordance correlation)
  • category labels agree (macro F1)
  • every reported statistic is within 1% of the reference value

Data updated 27 October 2026, 08:48 EDT.