IWC multi-step task

Task 5: Single-cell populations and marker genes

What the task asks for

Filter, normalize and cluster a single-cell count matrix with the stated settings, and test marker genes for every cluster.

12 scored results12 required18 runs scored

Codex GPT-5.6 Luna

6 of 6 runs scored
Galaxy-API code 0.528 BixBench3 acc (share of results that pass), averaged over 3 of 3 runs continuous score 0.576
All results pass1 of 3 runsruns in which every scored result passes
run 10.3334/12run 20.2503/12run 31.00012/12
Open-ended code 0.639 BixBench3 acc (share of results that pass), averaged over 3 of 3 runs continuous score 0.804
All results pass0 of 3 runsruns in which every scored result passes
run 10.5837/12run 20.5006/12run 30.83310/12

Codex GPT-5.6 Sol

6 of 6 runs scored
Galaxy-API code 0.972 BixBench3 acc (share of results that pass), averaged over 3 of 3 runs continuous score 0.977
All results pass2 of 3 runsruns in which every scored result passes
run 10.91711/12run 21.00012/12run 31.00012/12
Open-ended code 0.944 BixBench3 acc (share of results that pass), averaged over 3 of 3 runs continuous score 0.968
All results pass1 of 3 runsruns in which every scored result passes
run 10.91711/12run 21.00012/12run 30.91711/12

DeepSeek V4 Pro via Codex

6 of 6 runs scored
Galaxy-API code 0.611 BixBench3 acc (share of results that pass), averaged over 3 of 3 runs continuous score 0.758
All results pass0 of 3 runsruns in which every scored result passes
run 10.3334/12run 20.7509/12run 30.7509/12
Open-ended code 0.833 BixBench3 acc (share of results that pass), averaged over 3 of 3 runs continuous score 0.925
All results pass0 of 3 runsruns in which every scored result passes
run 10.7509/12run 20.91711/12run 30.83310/12

Three measures. Every required result gets a score from 0 to 1 by comparison with the reference. Passing requires a score of at least 0.80 and compliance with the required content, method and output instructions. BixBench3 acc is the share of results that pass. The continuous score is the plain average of all the result scores, with no pass mark. For example, run 2 of GPT-5.6 Luna with Galaxy-API code scored 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.999, 0.854, 0.902 and 0.000 on its 12 results. 3 of them pass, so BixBench3 acc = 3/12 = 0.250; the average of all 12 scores is 0.230, the continuous score. A third measure, all results pass, counts a run only when every scored result passes, which is the share of runs that delivered the whole analysis correctly. A failing result still contributes its numerical score to the continuous average. A score above 0.80 does not excuse an incorrect method, missing member or an explicitly required output format. Each card averages the scored runs of this task.

Error analysis

BixBench3 acc, Galaxy-API code vs Open-ended code: GPT-5.6 Luna 0.528 vs 0.639; GPT-5.6 Sol 0.972 vs 0.944; DeepSeek V4 Pro 0.611 vs 0.833.

What Galaxy added

Galaxy's current single-cell tools reproduce the reference neighbor graph and clustering: the four Galaxy runs that used them throughout produced the same passing partition, so Galaxy passed the cluster result in 4 of 9 runs against 1 of 9 in Open-ended code. Open-ended runs either rebuilt the neighbor graph with general-purpose libraries or changed the clustering method, and their populations differed.

Why Galaxy did not do better

Galaxy offers two generations of single-cell tools. The older variable-gene tool pre-fills its form with the top 2,000 genes, which overrides the requested mean and dispersion cutoffs, and two runs kept that value; the older tools also crashed on data prepared by the newer ones. None of the three DeepSeek V4 Pro Galaxy runs set the option that stores the full expression matrix for marker testing, and GPT-5.6 Luna made single-run errors, such as naming genes by ID instead of symbol.

Run-level details (5)
  • The first Open-ended run with DeepSeek V4 Pro used the Leiden algorithm in place of the required Louvain method. Clustering also amplified small upstream differences in other runs: the first Galaxy run with GPT-5.6 Sol let the PCA tool choose its solver, while its second and third runs set ARPACK; their neighbor graphs and population partitions differed.
  • The first Galaxy runs with GPT-5.6 Luna and DeepSeek V4 Pro kept the variable-gene tool's default of the top 2,000 genes, which overrides the requested mean and dispersion cutoffs. The first two Open-ended runs with GPT-5.6 Luna divided only the 0.0125 to 3 mean window into the 20 bins, leaving the other genes out. This changes the gene domain used by scaling, PCA and neighbor construction.
  • The second Galaxy run with GPT-5.6 Luna kept Ensembl IDs as gene names, while the task asks for the supplied gene symbols. After several attempts to flag mitochondrial genes, the agent pointed the flag at the stored symbol column but left the gene names as IDs.
  • In all three Galaxy runs and the first Open-ended run with DeepSeek V4 Pro, the marker analysis tested the main matrix without filling the raw slot (raw/X with raw/var) that the task names for the tested gene domain.
  • The first Galaxy run with GPT-5.6 Luna returned marker statistics for only 50 of the 24,018 genes in each population, the marker tool's default, instead of the complete requested gene domain.
Instruction, format and method errors (4)

Every run

The task asks the agent to return a numbered list of results, each a file or a folder of files. The columns on the right give the score of each required result in each run, from 0 to 1 against the reference, under the result's number and name; the task text and the scoring table below describe each one. Green marks a passing result. Failing cells link to the reasons below, including instruction and format errors even when the numerical score is at least 0.80. In each row, BixBench3 acc is the number of green cells divided by the number of results, and the continuous score is the average of all the result scores in the row. Tokens are input (including cached input) plus output tokens; minutes are the wall time of the run. Under Records, Trace opens the run's task text, agent event log, final message and returned files in the lab's run trace dataset on Hugging Face, and History opens the usegalaxy.org history that a Galaxy-API code run worked in.

Score of each required result
ModelConditionRunBixBench3 acc
share passing		Continuous score TokensMinutesRecords1gene filtering2cell QC3normalized expression4log expression5variable genes6regressed expression7scaled expression8PCA9neighbors10clusters11population counts12markers
GPT-5.6 LunaGalaxy-API code1trial run4/12 = 0.3330.51915.4M53Trace
History		1.00 Passed1.00 Passed1.00 Passed1.00 Passed0.72Content0.09Content0.10Content0.00Content0.00Content0.59Content0.69Content0.00Instruction
23/12 = 0.2500.23021.2M85Trace
History		0.00Content0.00Content0.00Content0.00Content0.00Content0.00Content0.00Content0.00Content0.99 Passed0.85 Passed0.90 Passed0.00Content
312/12 = 1.0000.98014.4M46Trace
History		1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed0.99 Passed0.85 Passed0.90 Passed1.00 Passed
Open-ended code1trial run7/12 = 0.5830.7481.3M8Trace1.00 Passed1.00 Passed1.00 Passed1.00 Passed0.86 Passed0.67Content0.67Content0.00Content0.13Content0.77Content0.84 Passed1.00 Passed
26/12 = 0.5000.7141.3M8Trace1.00 Passed1.00 Passed1.00 Passed1.00 Passed0.86 Passed0.67Content0.67Content0.00Content0.13Content0.60Content0.62Content0.99 Passed
310/12 = 0.8330.9501.2M19Trace1.00 Passed1.00 Passed1.00 Passed1.00 Passed0.99 Passed1.00 Passed1.00 Passed0.99 Passed0.92 Passed0.72Content0.75Content1.00 Passed
GPT-5.6 SolGalaxy-API code111/12 = 0.9170.9719.8M50Trace
History		1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed0.99 Passed0.98 Passed0.78Content0.87 Passed1.00 Passed
212/12 = 1.0000.98010.4M44Trace
History		1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed0.99 Passed0.85 Passed0.90 Passed1.00 Passed
312/12 = 1.0000.9808.9M103Trace
History		1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed0.99 Passed0.85 Passed0.90 Passed1.00 Passed
Open-ended code111/12 = 0.9170.9623.0M8Trace1.00 Passed1.00 Passed1.00 Passed1.00 Passed0.99 Passed1.00 Passed1.00 Passed0.99 Passed0.92 Passed0.77Content0.84 Passed1.00 Passed
212/12 = 1.0000.9711.1M8Trace1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed0.92 Passed0.82 Passed0.90 Passed1.00 Passed
311/12 = 0.9170.9700.9M6Trace1.00 Passed1.00 Passed1.00 Passed1.00 Passed0.99 Passed1.00 Passed1.00 Passed0.99 Passed0.92 Passed0.79Content0.91 Passed1.00 Passed
DeepSeek V4 ProGalaxy-API code14/12 = 0.3330.5246.7M69Trace
History		1.00 Passed1.00 Passed1.00 Passed0.99 Passed0.72Content0.12Content0.12Content0.00Content0.00Content0.61Content0.70Content0.00Format
29/12 = 0.7500.86410.7M55Trace
History		1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed0.92 Passed0.70Content0.73Content0.00Format
39/12 = 0.7500.88720.3M85Trace
History		1.00 Passed0.99Method1.00 Passed1.00 Passed1.00 Passed0.99 Passed0.99 Passed0.99 Passed0.98 Passed0.78Content0.87 Passed0.00Format
Open-ended code19/12 = 0.7500.8543.8M15Trace1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed0.99 Passed0.63Content0.62Content0.00Format
211/12 = 0.9170.9634.8M16Trace0.99 Passed0.99 Passed0.99 Passed0.99 Passed0.99 Passed1.00 Passed1.00 Passed1.00 Passed0.99 Passed0.75Content0.81 Passed0.99 Passed
310/12 = 0.8330.9579.7M27Trace0.99 Passed0.99 Passed0.99 Passed0.99 Passed0.99 Passed1.00 Passed1.00 Passed1.00 Passed0.92 Passed0.75Content0.81 Passed0.99Method

Why results did not pass

  1. GPT-5.6 Luna / Galaxy-API code / run 1 → result 5 (variable genes)
    • Content: Content score below 0.80. Value: 0.723221. Required minimum: 0.8.
      • Required fields: 0.723221 to 1 (2 checks)
  2. GPT-5.6 Luna / Galaxy-API code / run 1 → result 6 (regressed expression)
    • Content: Content score below 0.80. Value: 0.0992413. Required minimum: 0.8.
      • Genes f1: 0.47204
      • Present, finite data cells (F1): 0.47204
      • Required fields: 0.445384
  3. GPT-5.6 Luna / Galaxy-API code / run 1 → result 7 (scaled expression)
    • Content: Content score below 0.80. Value: 0.105982. Required minimum: 0.8.
      • Genes f1: 0.47204
      • Present, finite data cells (F1): 0.47204
      • Required fields: 0.475635
  4. GPT-5.6 Luna / Galaxy-API code / run 1 → result 8 (PCA)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
  5. GPT-5.6 Luna / Galaxy-API code / run 1 → result 9 (neighbors)
    • Content: Content score below 0.80. Value: 0.00553923. Required minimum: 0.8.
      • Matching records (F1): 0.371062 to 0.390225 (2 checks)
      • Numerical agreement (CCC): 0.286753 to 0.921334 (2 checks)
      • Present, finite data cells (F1): 0.371062 to 0.390225 (2 checks)
  6. GPT-5.6 Luna / Galaxy-API code / run 1 → result 10 (clusters)
    • Content: Content score below 0.80. Value: 0.590598. Required minimum: 0.8.
      • Category agreement (F1): 0.590598
  7. GPT-5.6 Luna / Galaxy-API code / run 1 → result 11 (population counts)
    • Content: Content score below 0.80. Value: 0.699138. Required minimum: 0.8.
      • Matching records (F1): 0.888889
      • Numerical agreement (CCC): 0.884847
      • Present, finite data cells (F1): 0.888889
  8. GPT-5.6 Luna / Galaxy-API code / run 1 → result 12 (markers)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
    • Instruction: A marker group does not retain the submitted raw/var gene domain.Task requirement: 12. final_answer/12_markers/ H5AD Full marker names, statistics, effects, p-values and adjusted p-values for every population. Stored fields: uns/rank_genes_groups: names, scores (z), logfoldchanges, pvals, pvals_adj; params/groupby identifies the supplied cell partition. Files: markers.h5ad.
  9. GPT-5.6 Luna / Galaxy-API code / run 2 → result 1 (gene filtering)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Genes f1: 0
      • Matching records (F1): 0
      • Present, finite data cells (F1): 0 (2 checks)
      • Required fields: 0 (2 checks)
  10. GPT-5.6 Luna / Galaxy-API code / run 2 → result 2 (cell QC)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Genes f1: 0
      • Present, finite data cells (F1): 0
      • Required fields: 0
  11. GPT-5.6 Luna / Galaxy-API code / run 2 → result 3 (normalized expression)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Genes f1: 0
      • Present, finite data cells (F1): 0
      • Required fields: 0
  12. GPT-5.6 Luna / Galaxy-API code / run 2 → result 4 (log expression)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Genes f1: 0
      • Present, finite data cells (F1): 0
      • Required fields: 0
  13. GPT-5.6 Luna / Galaxy-API code / run 2 → result 5 (variable genes)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Matching records (F1): 0
      • Present, finite data cells (F1): 0
      • Required fields: 0 (2 checks)
  14. GPT-5.6 Luna / Galaxy-API code / run 2 → result 6 (regressed expression)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Genes f1: 0
      • Present, finite data cells (F1): 0
      • Required fields: 0
  15. GPT-5.6 Luna / Galaxy-API code / run 2 → result 7 (scaled expression)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Genes f1: 0
      • Present, finite data cells (F1): 0
      • Required fields: 0
  16. GPT-5.6 Luna / Galaxy-API code / run 2 → result 8 (PCA)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
  17. GPT-5.6 Luna / Galaxy-API code / run 2 → result 12 (markers)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Matching records (F1): 0
      • Numerical agreement (CCC): 0
      • Present, finite data cells (F1): 0
  18. GPT-5.6 Luna / Open-ended code / run 1 → result 6 (regressed expression)
    • Content: Content score below 0.80. Value: 0.67855. Required minimum: 0.8.
      • Genes f1: 0.899537
      • Present, finite data cells (F1): 0.899537
      • Required fields: 0.838579
  19. GPT-5.6 Luna / Open-ended code / run 1 → result 7 (scaled expression)
    • Content: Content score below 0.80. Value: 0.67855. Required minimum: 0.8.
      • Genes f1: 0.899537
      • Present, finite data cells (F1): 0.899537
      • Required fields: 0.838579
  20. GPT-5.6 Luna / Open-ended code / run 1 → result 8 (PCA)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
  21. GPT-5.6 Luna / Open-ended code / run 1 → result 9 (neighbors)
    • Content: Content score below 0.80. Value: 0.136251. Required minimum: 0.8.
      • Matching records (F1): 0.679374 to 0.696345 (2 checks)
      • Numerical agreement (CCC): 0.619047 to 0.983442 (2 checks)
      • Present, finite data cells (F1): 0.679374 to 0.696345 (2 checks)
  22. GPT-5.6 Luna / Open-ended code / run 1 → result 10 (clusters)
    • Content: Content score below 0.80. Value: 0.778361. Required minimum: 0.8.
      • Category agreement (F1): 0.778361
  23. GPT-5.6 Luna / Open-ended code / run 2 → result 6 (regressed expression)
    • Content: Content score below 0.80. Value: 0.67855. Required minimum: 0.8.
      • Genes f1: 0.899537
      • Present, finite data cells (F1): 0.899537
      • Required fields: 0.838579
  24. GPT-5.6 Luna / Open-ended code / run 2 → result 7 (scaled expression)
    • Content: Content score below 0.80. Value: 0.67855. Required minimum: 0.8.
      • Genes f1: 0.899537
      • Present, finite data cells (F1): 0.899537
      • Required fields: 0.838579
  25. GPT-5.6 Luna / Open-ended code / run 2 → result 8 (PCA)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
  26. GPT-5.6 Luna / Open-ended code / run 2 → result 9 (neighbors)
    • Content: Content score below 0.80. Value: 0.131159. Required minimum: 0.8.
      • Matching records (F1): 0.679371 to 0.69635 (2 checks)
      • Numerical agreement (CCC): 0.595911 to 0.983438 (2 checks)
      • Present, finite data cells (F1): 0.679371 to 0.69635 (2 checks)
  27. GPT-5.6 Luna / Open-ended code / run 2 → result 10 (clusters)
    • Content: Content score below 0.80. Value: 0.60061. Required minimum: 0.8.
      • Category agreement (F1): 0.60061
  28. GPT-5.6 Luna / Open-ended code / run 2 → result 11 (population counts)
    • Content: Content score below 0.80. Value: 0.623652. Required minimum: 0.8.
      • Matching records (F1): 0.857143
      • Numerical agreement (CCC): 0.848859
      • Present, finite data cells (F1): 0.857143
  29. GPT-5.6 Luna / Open-ended code / run 3 → result 10 (clusters)
    • Content: Content score below 0.80. Value: 0.726285. Required minimum: 0.8.
      • Category agreement (F1): 0.726285
  30. GPT-5.6 Luna / Open-ended code / run 3 → result 11 (population counts)
    • Content: Content score below 0.80. Value: 0.753362. Required minimum: 0.8.
      • Matching records (F1): 0.971429
      • Numerical agreement (CCC): 0.798329
      • Present, finite data cells (F1): 0.971429
  31. GPT-5.6 Sol / Galaxy-API code / run 1 → result 10 (clusters)
    • Content: Content score below 0.80. Value: 0.784849. Required minimum: 0.8.
      • Category agreement (F1): 0.784849
  32. GPT-5.6 Sol / Open-ended code / run 1 → result 10 (clusters)
    • Content: Content score below 0.80. Value: 0.774412. Required minimum: 0.8.
      • Category agreement (F1): 0.774412
  33. GPT-5.6 Sol / Open-ended code / run 3 → result 10 (clusters)
    • Content: Content score below 0.80. Value: 0.797694. Required minimum: 0.8.
      • Category agreement (F1): 0.797694
  34. DeepSeek V4 Pro / Galaxy-API code / run 1 → result 5 (variable genes)
    • Content: Content score below 0.80. Value: 0.723221. Required minimum: 0.8.
      • Required fields: 0.723221 to 1 (2 checks)
  35. DeepSeek V4 Pro / Galaxy-API code / run 1 → result 6 (regressed expression)
    • Content: Content score below 0.80. Value: 0.120142. Required minimum: 0.8.
      • Genes f1: 0.492191
      • Present, finite data cells (F1): 0.492191
      • Required fields: 0.495939
  36. DeepSeek V4 Pro / Galaxy-API code / run 1 → result 7 (scaled expression)
    • Content: Content score below 0.80. Value: 0.120142. Required minimum: 0.8.
      • Genes f1: 0.492191
      • Present, finite data cells (F1): 0.492191
      • Required fields: 0.495939
  37. DeepSeek V4 Pro / Galaxy-API code / run 1 → result 8 (PCA)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
  38. DeepSeek V4 Pro / Galaxy-API code / run 1 → result 9 (neighbors)
    • Content: Content score below 0.80. Value: 0.00419002. Required minimum: 0.8.
      • Matching records (F1): 0.347235 to 0.370042 (2 checks)
      • Numerical agreement (CCC): 0.279924 to 0.906621 (2 checks)
      • Present, finite data cells (F1): 0.347235 to 0.370042 (2 checks)
  39. DeepSeek V4 Pro / Galaxy-API code / run 1 → result 10 (clusters)
    • Content: Content score below 0.80. Value: 0.616386. Required minimum: 0.8.
      • Category agreement (F1): 0.616386
  40. DeepSeek V4 Pro / Galaxy-API code / run 1 → result 11 (population counts)
    • Content: Content score below 0.80. Value: 0.700263. Required minimum: 0.8.
      • Matching records (F1): 0.914286
      • Numerical agreement (CCC): 0.837717
      • Present, finite data cells (F1): 0.914286
  41. DeepSeek V4 Pro / Galaxy-API code / run 1 → result 12 (markers)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
    • Format: Named stored field is missing: raw/XTask requirement: 12. final_answer/12_markers/ H5AD Full marker names, statistics, effects, p-values and adjusted p-values for every population. Stored fields: uns/rank_genes_groups: names, scores (z), logfoldchanges, pvals, pvals_adj; params/groupby identifies the supplied cell partition. Files: markers.h5ad.
    • Format: Named stored field is missing: raw/varTask requirement: 12. final_answer/12_markers/ H5AD Full marker names, statistics, effects, p-values and adjusted p-values for every population. Stored fields: uns/rank_genes_groups: names, scores (z), logfoldchanges, pvals, pvals_adj; params/groupby identifies the supplied cell partition. Files: markers.h5ad.
  42. DeepSeek V4 Pro / Galaxy-API code / run 2 → result 10 (clusters)
    • Content: Content score below 0.80. Value: 0.708943. Required minimum: 0.8.
      • Category agreement (F1): 0.708943
  43. DeepSeek V4 Pro / Galaxy-API code / run 2 → result 11 (population counts)
    • Content: Content score below 0.80. Value: 0.738993. Required minimum: 0.8.
      • Matching records (F1): 0.918919
      • Numerical agreement (CCC): 0.875156
      • Present, finite data cells (F1): 0.918919
  44. DeepSeek V4 Pro / Galaxy-API code / run 2 → result 12 (markers)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
    • Format: Named stored field is missing: raw/XTask requirement: 12. final_answer/12_markers/ H5AD Full marker names, statistics, effects, p-values and adjusted p-values for every population. Stored fields: uns/rank_genes_groups: names, scores (z), logfoldchanges, pvals, pvals_adj; params/groupby identifies the supplied cell partition. Files: markers.h5ad.
    • Format: Named stored field is missing: raw/varTask requirement: 12. final_answer/12_markers/ H5AD Full marker names, statistics, effects, p-values and adjusted p-values for every population. Stored fields: uns/rank_genes_groups: names, scores (z), logfoldchanges, pvals, pvals_adj; params/groupby identifies the supplied cell partition. Files: markers.h5ad.
  45. DeepSeek V4 Pro / Galaxy-API code / run 3 → result 2 (cell QC)
    • Method: Per-cell QC values differ from those computed on the retained gene domain of result 01.Task requirement: Retain genes detected in at least three input cells, then cells with 200–6,000 detected genes inclusive and mitochondrial percentage<15% using MT- symbols. QC uses the retained full-gene domain before variable-gene selection.
  46. DeepSeek V4 Pro / Galaxy-API code / run 3 → result 10 (clusters)
    • Content: Content score below 0.80. Value: 0.785072. Required minimum: 0.8.
      • Category agreement (F1): 0.785072
  47. DeepSeek V4 Pro / Galaxy-API code / run 3 → result 12 (markers)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
    • Format: Named stored field is missing: raw/XTask requirement: 12. final_answer/12_markers/ H5AD Full marker names, statistics, effects, p-values and adjusted p-values for every population. Stored fields: uns/rank_genes_groups: names, scores (z), logfoldchanges, pvals, pvals_adj; params/groupby identifies the supplied cell partition. Files: markers.h5ad.
    • Format: Named stored field is missing: raw/varTask requirement: 12. final_answer/12_markers/ H5AD Full marker names, statistics, effects, p-values and adjusted p-values for every population. Stored fields: uns/rank_genes_groups: names, scores (z), logfoldchanges, pvals, pvals_adj; params/groupby identifies the supplied cell partition. Files: markers.h5ad.
  48. DeepSeek V4 Pro / Open-ended code / run 1 → result 10 (clusters)
    • Content: Content score below 0.80. Value: 0.631123. Required minimum: 0.8.
      • Category agreement (F1): 0.631123
  49. DeepSeek V4 Pro / Open-ended code / run 1 → result 11 (population counts)
    • Content: Content score below 0.80. Value: 0.628723. Required minimum: 0.8.
      • Matching records (F1): 0.888889
      • Numerical agreement (CCC): 0.795728
      • Present, finite data cells (F1): 0.888889
  50. DeepSeek V4 Pro / Open-ended code / run 1 → result 12 (markers)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
    • Format: Named stored field is missing: raw/XTask requirement: 12. final_answer/12_markers/ H5AD Full marker names, statistics, effects, p-values and adjusted p-values for every population. Stored fields: uns/rank_genes_groups: names, scores (z), logfoldchanges, pvals, pvals_adj; params/groupby identifies the supplied cell partition. Files: markers.h5ad.
    • Format: Named stored field is missing: raw/varTask requirement: 12. final_answer/12_markers/ H5AD Full marker names, statistics, effects, p-values and adjusted p-values for every population. Stored fields: uns/rank_genes_groups: names, scores (z), logfoldchanges, pvals, pvals_adj; params/groupby identifies the supplied cell partition. Files: markers.h5ad.
  51. DeepSeek V4 Pro / Open-ended code / run 2 → result 10 (clusters)
    • Content: Content score below 0.80. Value: 0.753504. Required minimum: 0.8.
      • Category agreement (F1): 0.753504
  52. DeepSeek V4 Pro / Open-ended code / run 3 → result 10 (clusters)
    • Content: Content score below 0.80. Value: 0.753504. Required minimum: 0.8.
      • Category agreement (F1): 0.753504
  53. DeepSeek V4 Pro / Open-ended code / run 3 → result 12 (markers)
    • Method: Marker effects differ from the stated log-scale-mean formula on the delivered expression and partition.Task requirement: Define the effect as the following log-scale-mean effect: log2((expm1(mean log-normalized expression in the population)+10^-9)/(expm1(mean log-normalized expression in the rest)+10^-9)), not a ratio of arithmetic mean raw counts.

Task text

The scientific part of the task, identical in both conditions. The general submission rules and the instructions for each condition are on the overview page.

Full task text

Task

Identify cell populations and their positive-expression markers from the supplied count matrix. Cell-type annotation is not requested. Identify populations through variable-gene expression structure after nuisance adjustment: use a low-dimensional expression representation and local-neighbor graph communities, not supervised cell-type labels. Assess population markers against the remaining cells using the full retained gene domain.

Inputs

  • The complete single-cell count matrix, cell identifiers and gene identifiers/symbols.

Key parameters and conditions

  • Retain genes detected in at least three input cells, then cells with 200–6,000 detected genes inclusive and mitochondrial percentage<15% using MT- symbols. QC uses the retained full-gene domain before variable-gene selection.
  • Normalize library size to 10,000 and use natural-log(1+x). Select variable genes by dispersion standardized within mean-expression bins: 20 bins, 0.0125<log(1+mean)<3 and normalized dispersion>0.5.
  • Regress total counts and mitochondrial percentage; center the selected genes, scale residuals to unit sample variance and clip at ±10. Use the first ten centered, unwhitened PCs, Euclidean distance and neighborhoods of 15 cells including self, with seed 0. Retain locally scaled fuzzy-simplicial-set union connectivity weights as graph evidence. For clustering, use the unweighted, undirected union of its nonzero edges without self-edges, with Louvain, the Reichardt–Bornholdt configuration-model objective, resolution 1.0 and seed 0. A two-dimensional embedding is not a substitute for the PCA coordinates or graph.
  • For each population, apply a two-sided Wilcoxon rank-sum test without tie correction to full-domain log-normalized expression, against all other retained cells. Apply BH within each population across the full retained gene domain. Define the effect as the following log-scale-mean effect: log2((expm1(mean log-normalized expression in the population)+10^-9)/(expm1(mean log-normalized expression in the rest)+10^-9)), not a ratio of arithmetic mean raw counts. Positive markers require effect>0 and adjusted p<0.05; retain nonsignificant genes and all test results.
  • Gene identities are the supplied gene symbols made unique in input order. For dispersion selection, recover linear expression with expm1, compute log(variance/mean), bin log1p(mean) into 20 equal-width bins and standardize dispersion within each bin. Report the standardized Wilcoxon z statistic, not the raw rank-sum U statistic.
  • Each H5AD uses observation names for cells and variable names for genes. A stage expression matrix is stored in X, not only in an unnamed layer. Keep the complete marker-test gene domain in raw/X with raw/var identities. The clustering column is obs/louvain.

Required results

  1. final_answer/01_gene_filtering/
    H5AD
    Filtered gene domain, full count matrix and detection evidence. Stored fields: var/n_cells. Files: counts.h5ad.
  2. final_answer/02_cell_qc/
    H5AD
    Per-cell QC and complete retained/excluded-cell evidence. Stored fields: all_cells.h5ad: obs/total_counts, obs/n_genes_by_counts, obs/pct_counts_mito for all cells; retained_counts.h5ad: retained count matrix. Files: all_cells.h5ad, retained_counts.h5ad.
  3. final_answer/03_normalized_expression/
    H5AD
    Library-size-normalized expression and scaling totals. Files: expression.h5ad.
  4. final_answer/04_log_expression/
    H5AD
    Full-domain log expression, including the domain used for marker tests. Files: expression.h5ad.
  5. final_answer/05_variable_genes/
    H5AD
    Mean/dispersion evidence and selection flags for the complete assessed gene domain. Stored fields: var/means, var/dispersions, var/dispersions_norm, var/highly_variable. Files: genes.h5ad.
  6. final_answer/06_regressed_expression/
    H5AD
    Regression results for the selected gene domain. Files: expression.h5ad.
  7. final_answer/07_scaled_expression/
    H5AD
    Centered/scaled/clipped expression with cell and gene identities. Files: expression.h5ad.
  8. final_answer/08_pca/
    H5AD
    Coordinates, loadings, explained variance and their identities. Stored fields: obsm/X_pca, varm/PCs, uns/pca/variance and uns/pca/variance_ratio, first ten components. Files: pca.h5ad.
  9. final_answer/09_neighbors/
    H5AD
    Distances, neighbor membership and the clustering graph. Stored fields: obsp/distances and obsp/connectivities. Files: graphs.h5ad.
  10. final_answer/10_clusters/
    H5AD
    Complete cell labels and the exact partition used for marker testing. Files: clusters.h5ad.
  11. final_answer/11_population_counts/
    TSV
    counts.tsv columns: population, cells; every population in the submitted partition. Files: counts.tsv.
  12. final_answer/12_markers/
    H5AD
    Full marker names, statistics, effects, p-values and adjusted p-values for every population. Stored fields: uns/rank_genes_groups: names, scores (z), logfoldchanges, pvals, pvals_adj; params/groupby identifies the supplied cell partition. Files: markers.h5ad.

How this task is scored

Each scored result is compared with the reference result, which the task authors made by running the IWC workflow behind this task on usegalaxy.org with the settings in the task text. A result's score is the product of the checks listed for it, each from 0 to 1. A passing result must score at least 0.80 and meet the required member checks, method and output instructions. A failing result retains its numerical score in the continuous average. The overview page describes the checks in more detail.

No.Required resultWhat the task asks for How it is scored
101_gene_filtering/
H5AD		Filtered gene domain, full count matrix and detection evidence. Stored fields: var/n_cells. Files: counts.h5ad.
  • the same rows or records as the reference (F1 of their identifiers)
  • the required columns or fields are present (F1)
  • values are present in the same cells (F1)
  • values agree (Lin's concordance correlation)
  • the same cells and genes, with values agreeing column by column
202_cell_qc/
H5AD		Per-cell QC and complete retained/excluded-cell evidence. Stored fields: all_cells.h5ad: obs/total_counts, obs/n_genes_by_counts, obs/pct_counts_mito for all cells; retained_counts.h5ad: retained count matrix. Files: all_cells.h5ad, retained_counts.h5ad.
  • the same rows or records as the reference (F1 of their identifiers)
  • the required columns or fields are present (F1)
  • values are present in the same cells (F1)
  • values agree (Lin's concordance correlation)
  • the same cells and genes, with values agreeing column by column
303_normalized_expression/
H5AD		Library-size-normalized expression and scaling totals. Files: expression.h5ad.
  • values are present in the same cells (F1)
  • the same cells and genes, with values agreeing column by column
404_log_expression/
H5AD		Full-domain log expression, including the domain used for marker tests. Files: expression.h5ad.
  • values are present in the same cells (F1)
  • the same cells and genes, with values agreeing column by column
505_variable_genes/
H5AD		Mean/dispersion evidence and selection flags for the complete assessed gene domain. Stored fields: var/means, var/dispersions, var/dispersions_norm, var/highly_variable. Files: genes.h5ad.
  • the same rows or records as the reference (F1 of their identifiers)
  • the required columns or fields are present (F1)
  • values are present in the same cells (F1)
  • values agree (Lin's concordance correlation)
  • category labels agree (macro F1)
606_regressed_expression/
H5AD		Regression results for the selected gene domain. Files: expression.h5ad.
  • values are present in the same cells (F1)
  • the same cells and genes, with values agreeing column by column
707_scaled_expression/
H5AD		Centered/scaled/clipped expression with cell and gene identities. Files: expression.h5ad.
  • values are present in the same cells (F1)
  • the same cells and genes, with values agreeing column by column
808_pca/
H5AD		Coordinates, loadings, explained variance and their identities. Stored fields: obsm/X_pca, varm/PCs, uns/pca/variance and uns/pca/variance_ratio, first ten components. Files: pca.h5ad.
  • the same rows or records as the reference (F1 of their identifiers)
  • the required columns or fields are present (F1)
  • values are present in the same cells (F1)
  • values agree (Lin's concordance correlation)
909_neighbors/
H5AD		Distances, neighbor membership and the clustering graph. Stored fields: obsp/distances and obsp/connectivities. Files: graphs.h5ad.
  • the same rows or records as the reference (F1 of their identifiers)
  • the required columns or fields are present (F1)
  • values are present in the same cells (F1)
  • values agree (Lin's concordance correlation)
1010_clusters/
H5AD		Complete cell labels and the exact partition used for marker testing. Files: clusters.h5ad.
  • the same rows or records as the reference (F1 of their identifiers)
  • the required columns or fields are present (F1)
  • category labels agree (macro F1)
1111_population_counts/
TSV		counts.tsv columns: population, cells; every population in the submitted partition. Files: counts.tsv.
  • the same rows or records as the reference (F1 of their identifiers)
  • the required columns or fields are present (F1)
  • values are present in the same cells (F1)
  • values agree (Lin's concordance correlation)
  • the population counts agree with the submitted cluster labels
1212_markers/
H5AD		Full marker names, statistics, effects, p-values and adjusted p-values for every population. Stored fields: uns/rank_genes_groups: names, scores (z), logfoldchanges, pvals, pvals_adj; params/groupby identifies the supplied cell partition. Files: markers.h5ad.
  • the same rows or records as the reference (F1 of their identifiers)
  • the required columns or fields are present (F1)
  • values are present in the same cells (F1)
  • values agree (Lin's concordance correlation)
  • category labels agree (macro F1)

Data updated 27 October 2026, 08:48 EDT.