IWC multi-step task

Task 7: Carbon-isotope incorporation in a metaproteome

What the task asks for

Identify peptides in one mass-spectrometry run of a microbial community and estimate how much labelled carbon each peptide took up.

8 scored results8 required18 runs scored

Codex GPT-5.6 Luna

6 of 6 runs scored
Galaxy-API code 0.708 BixBench3 acc (share of results that pass), averaged over 3 of 3 runs continuous score 0.790
All results pass0 of 3 runsruns in which every scored result passes
run 10.6255/8run 20.6255/8run 30.8757/8
Open-ended code 0.125 BixBench3 acc (share of results that pass), averaged over 3 of 3 runs continuous score 0.125
All results pass0 of 3 runsruns in which every scored result passes
run 10.1251/8run 20.1251/8run 30.1251/8

Codex GPT-5.6 Sol

6 of 6 runs scored
Galaxy-API code 0.875 BixBench3 acc (share of results that pass), averaged over 3 of 3 runs continuous score 0.863
All results pass0 of 3 runsruns in which every scored result passes
run 10.8757/8run 20.8757/8run 30.8757/8
Open-ended code 1.000 BixBench3 acc (share of results that pass), averaged over 3 of 3 runs continuous score 0.989
All results pass3 of 3 runsruns in which every scored result passes
run 11.0008/8run 21.0008/8run 31.0008/8

DeepSeek V4 Pro via Codex

6 of 6 runs scored
Galaxy-API code 0.625 BixBench3 acc (share of results that pass), averaged over 3 of 3 runs continuous score 0.753
All results pass0 of 3 runsruns in which every scored result passes
run 10.7506/8run 20.5004/8run 30.6255/8
Open-ended code 0.417 BixBench3 acc (share of results that pass), averaged over 3 of 3 runs continuous score 0.533
All results pass0 of 3 runsruns in which every scored result passes
run 10.5004/8run 20.5004/8run 30.2502/8

Three measures. Every required result gets a score from 0 to 1 by comparison with the reference. Passing requires a score of at least 0.80 and compliance with the required content, method and output instructions. BixBench3 acc is the share of results that pass. The continuous score is the plain average of all the result scores, with no pass mark. For example, run 1 of GPT-5.6 Luna with Galaxy-API code scored 0.733, 1.000, 0.001, 0.735, 0.853, 0.853, 1.000 and 1.000 on its 8 results. 5 of them pass, so BixBench3 acc = 5/8 = 0.625; the average of all 8 scores is 0.772, the continuous score. A third measure, all results pass, counts a run only when every scored result passes, which is the share of runs that delivered the whole analysis correctly. A failing result still contributes its numerical score to the continuous average. A score above 0.80 does not excuse an incorrect method, missing member or an explicitly required output format. Each card averages the scored runs of this task.

Error analysis

BixBench3 acc, Galaxy-API code vs Open-ended code: GPT-5.6 Luna 0.708 vs 0.125; GPT-5.6 Sol 0.875 vs 1.000; DeepSeek V4 Pro 0.625 vs 0.417.

What Galaxy added

Every Galaxy run used Galaxy's ready mass-spectrometry tools for feature detection, database search, FDR, mapping and isotope analysis. In Open-ended code, GPT-5.6 Luna never installed them and wrote its own version of every step, accepting at most 8 peptide matches where passing runs accepted about 159, and DeepSeek V4 Pro wrote its own FDR in all three runs. GPT-5.6 Sol installed the same tools locally and passed everything.

Why Galaxy did not do better

The search result failed in 8 of 9 Galaxy runs. To keep hits tied for first rank, agents asked the search for 10 or 20 matches per spectrum and returned all of them. The tool that trims hits back to the leading ones is not installed on usegalaxy.org, and the one Galaxy run that asked for a single match passed. Four Galaxy runs also turned on the isotope tool's grouping option, whose label does not say that it clusters peptides by isotope abundance, and two built decoys with a cleavage rule that cuts before proline.

Run-level details (5)
  • Most Galaxy runs requested 10 matches per spectrum, or 20 in the third run with GPT-5.6 Luna, to keep ties for first rank, and delivered all of them rather than just the leading hits; the tool that trims hits back to the leading ones is not installed on usegalaxy.org. The Open-ended runs with DeepSeek V4 Pro requested 10, 50 and 20 matches per spectrum. GPT-5.6 Sol also requested more in Open-ended code but trimmed back to the leading hits.
  • All three Open-ended code runs with GPT-5.6 Luna wrote their own feature detection, database search, FDR and isotope fitting, and some of their isotope reports contained only headers.
  • In the second Open-ended run with DeepSeek V4 Pro, the isotope analysis used the search E-value as each peptide's score. The first run wrote isotope reports with its own script without the specified isotope decomposition. The third detected features with a model-fitting feature finder instead of the specified isotope-pattern filtering and clustering.
  • All three Galaxy runs with DeepSeek V4 Pro and the second with GPT-5.6 Luna enabled grouping by relative isotope abundance, although the task asks for no such clustering. The second DeepSeek run also ran without grouping but delivered the grouped report.
  • The first Galaxy run with GPT-5.6 Luna and second with DeepSeek V4 Pro built decoys using a trypsin rule that cleaves before proline. The task requires the decoys to keep tryptic sites with no cleavage before proline.
Instruction, format and method errors (2)
  • Peptides carry RIA cluster groups other than 0, so incorporation estimates were clustered by relative isotope abundance.Task requirement: Collect incorporation estimates at correlation maxima from sequence-informed isotope-pattern decomposition, without clustering by relative isotope abundance.

    GPT-5.6 Luna / Galaxy-API code / run 2 → result 7; DeepSeek V4 Pro / Galaxy-API code / run 1 → result 7; DeepSeek V4 Pro / Galaxy-API code / run 3 → result 7

  • Delivered target leading-hit q-values do not follow the stated D/T rule on the main score the delivered search output declares.Task requirement: For PSM-level FDR, count one leading hit per spectrum; when first rank is tied, use the first hit in the original search-result order. At each score threshold, let D and T count decoy and target leading hits with scores at least as good as that threshold; a hit mapping to both target and decoy proteins counts as target. Estimate FDR as D/T, without a pseudocount, and obtain each target hit’s q-value as the minimum FDR over thresholds that include it. Pool charge states and accept target PSMs at q≤0.01. Retain all target leading-hit q-values and the accepted set; do not substitute peptide- or protein-level FDR.

    DeepSeek V4 Pro / Open-ended code / run 3 → result 4

Every run

The task asks the agent to return a numbered list of results, each a file or a folder of files. The columns on the right give the score of each required result in each run, from 0 to 1 against the reference, under the result's number and name; the task text and the scoring table below describe each one. Green marks a passing result. Failing cells link to the reasons below, including instruction and format errors even when the numerical score is at least 0.80. In each row, BixBench3 acc is the number of green cells divided by the number of results, and the continuous score is the average of all the result scores in the row. Tokens are input (including cached input) plus output tokens; minutes are the wall time of the run. Under Records, Trace opens the run's task text, agent event log, final message and returned files in the lab's run trace dataset on Hugging Face, and History opens the usegalaxy.org history that a Galaxy-API code run worked in.

Score of each required result
ModelConditionRunBixBench3 acc
share passing		Continuous score TokensMinutesRecords1search database2features3identifications4PSM q-values5PSM accepted6feature identification links7isotope peptides8isotope groups
GPT-5.6 LunaGalaxy-API code1trial run5/8 = 0.6250.7722.7M22Trace
History		0.73Content1.00 Passed0.00Content0.73Content0.85 Passed0.85 Passed1.00 Passed1.00 Passed
25/8 = 0.6250.7382.7M40Trace
History		1.00 Passed1.00 Passed0.00Content0.90 Passed1.00 Passed1.00 Passed1.00Method0.00Check
37/8 = 0.8750.8582.4M24Trace
History		1.00 Passed1.00 Passed0.00Content0.86 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed
Open-ended code1trial run1/8 = 0.1250.12513.0M61Trace0.99 Passed0.00Content0.00Content0.00Content0.00Content0.00Check0.00Content0.00Content
21/8 = 0.1250.12512.7M45Trace0.99 Passed0.00Content0.00Content0.00Content0.00Content0.00Content0.00Content0.00Content
31/8 = 0.1250.12512.6M72Trace0.99 Passed0.00Content0.00Content0.00Content0.00Content0.00Content0.00Content0.00Content
GPT-5.6 SolGalaxy-API code17/8 = 0.8750.8633.8M25Trace
History		1.00 Passed1.00 Passed0.00Content0.90 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed
27/8 = 0.8750.8633.3M20Trace
History		1.00 Passed1.00 Passed0.00Content0.90 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed
37/8 = 0.8750.8632.8M35Trace
History		1.00 Passed1.00 Passed0.00Content0.90 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed
Open-ended code18/8 = 1.0000.9836.8M46Trace1.00 Passed0.98 Passed0.99 Passed0.88 Passed1.00 Passed0.98 Passed1.00 Passed1.00 Passed
28/8 = 1.0000.9854.0M16Trace1.00 Passed0.99 Passed0.99 Passed0.90 Passed1.00 Passed0.99 Passed1.00 Passed1.00 Passed
38/8 = 1.0001.0008.4M14Trace1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed0.99 Passed1.00 Passed
DeepSeek V4 ProGalaxy-API code16/8 = 0.7500.87515.1M28Trace
History		1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00 Passed1.00Method0.00Check
24/8 = 0.5000.6477.6M26Trace
History		0.73Content1.00 Passed0.00Content0.73Content0.85 Passed0.85 Passed1.00 Passed0.00Check
35/8 = 0.6250.7384.7M22Trace
History		1.00 Passed1.00 Passed0.00Content0.90 Passed1.00 Passed1.00 Passed1.00Method0.00Check
Open-ended code14/8 = 0.5000.5659.6M29Trace0.99 Passed0.98 Passed0.05Content0.49Content1.00 Passed0.98 Passed0.00Content0.00Content
24/8 = 0.5000.66612.4M34Trace0.99 Passed0.98 Passed0.00Content0.00Content1.00 Passed0.98 Passed0.75Content0.60Check
32/8 = 0.2500.36910.1M37Trace0.99 Passed0.00Content0.00Content0.89Method1.00 Passed0.00Check0.05Content0.00Check

Why results did not pass

  1. GPT-5.6 Luna / Galaxy-API code / run 1 → result 1 (search database)
    • Content: Content score below 0.80. Value: 0.733023. Required minimum: 0.8.
      • Category agreement (F1): 0.733023
  2. GPT-5.6 Luna / Galaxy-API code / run 1 → result 3 (identifications)
    • Content: Content score below 0.80. Value: 0.00132098. Required minimum: 0.8.
      • Category agreement (F1): 0.801687 to 0.980666 (2 checks)
      • Matching records (F1): 0.202462 (2 checks)
      • Present, finite data cells (F1): 0.202462 (2 checks)
  3. GPT-5.6 Luna / Galaxy-API code / run 1 → result 4 (PSM q-values)
    • Content: Content score below 0.80. Value: 0.735247. Required minimum: 0.8.
      • Category agreement (F1): 0.969815
      • Matching records (F1): 0.878542
      • Numerical agreement (CCC): 0.982244
      • Present, finite data cells (F1): 0.878542
  4. GPT-5.6 Luna / Galaxy-API code / run 2 → result 3 (identifications)
    • Content: Content score below 0.80. Value: 0.00217067. Required minimum: 0.8.
      • Category agreement (F1): 0.804856
      • Matching records (F1): 0.227887 (2 checks)
      • Present, finite data cells (F1): 0.227887 (2 checks)
  5. GPT-5.6 Luna / Galaxy-API code / run 2 → result 7 (isotope peptides)
    • Method: Peptides carry RIA cluster groups other than 0, so incorporation estimates were clustered by relative isotope abundance.Task requirement: Collect incorporation estimates at correlation maxima from sequence-informed isotope-pattern decomposition, without clustering by relative isotope abundance.
  6. GPT-5.6 Luna / Galaxy-API code / run 2 → result 8 (isotope groups)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Matching records (F1): 0 (2 checks)
      • Numerical agreement (CCC): 0 (2 checks)
      • Present, finite data cells (F1): 0 (2 checks)
      • Required fields: 0 (2 checks)
    • Check: A required check is below its minimum. Required minimum: 0.995.
      • Matching records (F1): 0
      • Numerical agreement (CCC): 0
      • Present, finite data cells (F1): 0
      • Required fields: 0
  7. GPT-5.6 Luna / Galaxy-API code / run 3 → result 3 (identifications)
    • Content: Content score below 0.80. Value: 0.000221193. Required minimum: 0.8.
      • Matching records (F1): 0.121953 (2 checks)
      • Present, finite data cells (F1): 0.121953 (2 checks)
  8. GPT-5.6 Luna / Open-ended code / run 1 → result 2 (features)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Matching records (F1): 0.0747643 to 0.114522 (2 checks)
      • Numerical agreement (CCC): 0 to 0.99999 (3 checks)
      • Present, finite data cells (F1): 0.0747643 to 0.114522 (3 checks)
  9. GPT-5.6 Luna / Open-ended code / run 1 → result 3 (identifications)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Category agreement (F1): 0.5 (2 checks)
      • Matching records (F1): 0.0128636 (2 checks)
      • Numerical agreement (CCC): 0 (2 checks)
      • Present, finite data cells (F1): 0.0128636 (2 checks)
  10. GPT-5.6 Luna / Open-ended code / run 1 → result 4 (PSM q-values)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Category agreement (F1): 0.5
      • Matching records (F1): 0.0151184
      • Numerical agreement (CCC): 0
      • Present, finite data cells (F1): 0.0151184
  11. GPT-5.6 Luna / Open-ended code / run 1 → result 5 (PSM accepted)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Category agreement (F1): 0
      • Matching records (F1): 0
      • Numerical agreement (CCC): 0
      • Present, finite data cells (F1): 0
  12. GPT-5.6 Luna / Open-ended code / run 1 → result 6 (feature identification links)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Category agreement (F1): 0
      • Matching records (F1): 0 to 0.114522 (3 checks)
      • Numerical agreement (CCC): 0 to 0.99999 (4 checks)
      • Present, finite data cells (F1): 0 to 0.114522 (4 checks)
    • Check: A required check is below its minimum. Required minimum: 0.998.
      • Matching records (F1): 0
  13. GPT-5.6 Luna / Open-ended code / run 1 → result 7 (isotope peptides)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
  14. GPT-5.6 Luna / Open-ended code / run 1 → result 8 (isotope groups)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
  15. GPT-5.6 Luna / Open-ended code / run 2 → result 2 (features)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
  16. GPT-5.6 Luna / Open-ended code / run 2 → result 3 (identifications)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
  17. GPT-5.6 Luna / Open-ended code / run 2 → result 4 (PSM q-values)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
  18. GPT-5.6 Luna / Open-ended code / run 2 → result 5 (PSM accepted)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
  19. GPT-5.6 Luna / Open-ended code / run 2 → result 6 (feature identification links)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
  20. GPT-5.6 Luna / Open-ended code / run 2 → result 7 (isotope peptides)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
  21. GPT-5.6 Luna / Open-ended code / run 2 → result 8 (isotope groups)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
  22. GPT-5.6 Luna / Open-ended code / run 3 → result 2 (features)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
  23. GPT-5.6 Luna / Open-ended code / run 3 → result 3 (identifications)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
  24. GPT-5.6 Luna / Open-ended code / run 3 → result 4 (PSM q-values)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
  25. GPT-5.6 Luna / Open-ended code / run 3 → result 5 (PSM accepted)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
  26. GPT-5.6 Luna / Open-ended code / run 3 → result 6 (feature identification links)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
  27. GPT-5.6 Luna / Open-ended code / run 3 → result 7 (isotope peptides)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
  28. GPT-5.6 Luna / Open-ended code / run 3 → result 8 (isotope groups)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
  29. GPT-5.6 Sol / Galaxy-API code / run 1 → result 3 (identifications)
    • Content: Content score below 0.80. Value: 0.00269697. Required minimum: 0.8.
      • Matching records (F1): 0.227887 (2 checks)
      • Present, finite data cells (F1): 0.227887 (2 checks)
  30. GPT-5.6 Sol / Galaxy-API code / run 2 → result 3 (identifications)
    • Content: Content score below 0.80. Value: 0.00217067. Required minimum: 0.8.
      • Category agreement (F1): 0.804856
      • Matching records (F1): 0.227887 (2 checks)
      • Present, finite data cells (F1): 0.227887 (2 checks)
  31. GPT-5.6 Sol / Galaxy-API code / run 3 → result 3 (identifications)
    • Content: Content score below 0.80. Value: 0.00217067. Required minimum: 0.8.
      • Category agreement (F1): 0.804856
      • Matching records (F1): 0.227887 (2 checks)
      • Present, finite data cells (F1): 0.227887 (2 checks)
  32. DeepSeek V4 Pro / Galaxy-API code / run 1 → result 7 (isotope peptides)
    • Method: Peptides carry RIA cluster groups other than 0, so incorporation estimates were clustered by relative isotope abundance.Task requirement: Collect incorporation estimates at correlation maxima from sequence-informed isotope-pattern decomposition, without clustering by relative isotope abundance.
  33. DeepSeek V4 Pro / Galaxy-API code / run 1 → result 8 (isotope groups)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Matching records (F1): 0 (2 checks)
      • Numerical agreement (CCC): 0 (2 checks)
      • Present, finite data cells (F1): 0 (2 checks)
      • Required fields: 0 (2 checks)
    • Check: A required check is below its minimum. Required minimum: 0.995.
      • Matching records (F1): 0
      • Numerical agreement (CCC): 0
      • Present, finite data cells (F1): 0
      • Required fields: 0
  34. DeepSeek V4 Pro / Galaxy-API code / run 2 → result 1 (search database)
    • Content: Content score below 0.80. Value: 0.733023. Required minimum: 0.8.
      • Category agreement (F1): 0.733023
  35. DeepSeek V4 Pro / Galaxy-API code / run 2 → result 3 (identifications)
    • Content: Content score below 0.80. Value: 0.00132098. Required minimum: 0.8.
      • Category agreement (F1): 0.801687 to 0.980666 (2 checks)
      • Matching records (F1): 0.202462 (2 checks)
      • Present, finite data cells (F1): 0.202462 (2 checks)
  36. DeepSeek V4 Pro / Galaxy-API code / run 2 → result 4 (PSM q-values)
    • Content: Content score below 0.80. Value: 0.735247. Required minimum: 0.8.
      • Category agreement (F1): 0.969815
      • Matching records (F1): 0.878542
      • Numerical agreement (CCC): 0.982244
      • Present, finite data cells (F1): 0.878542
  37. DeepSeek V4 Pro / Galaxy-API code / run 2 → result 8 (isotope groups)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Matching records (F1): 0 (2 checks)
      • Numerical agreement (CCC): 0 (2 checks)
      • Present, finite data cells (F1): 0 (2 checks)
      • Required fields: 0 (2 checks)
    • Check: A required check is below its minimum. Required minimum: 0.995.
      • Matching records (F1): 0
      • Numerical agreement (CCC): 0
      • Present, finite data cells (F1): 0
      • Required fields: 0
  38. DeepSeek V4 Pro / Galaxy-API code / run 3 → result 3 (identifications)
    • Content: Content score below 0.80. Value: 0.00269697. Required minimum: 0.8.
      • Matching records (F1): 0.227887 (2 checks)
      • Present, finite data cells (F1): 0.227887 (2 checks)
  39. DeepSeek V4 Pro / Galaxy-API code / run 3 → result 7 (isotope peptides)
    • Method: Peptides carry RIA cluster groups other than 0, so incorporation estimates were clustered by relative isotope abundance.Task requirement: Collect incorporation estimates at correlation maxima from sequence-informed isotope-pattern decomposition, without clustering by relative isotope abundance.
  40. DeepSeek V4 Pro / Galaxy-API code / run 3 → result 8 (isotope groups)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Matching records (F1): 0 (2 checks)
      • Numerical agreement (CCC): 0 (2 checks)
      • Present, finite data cells (F1): 0 (2 checks)
      • Required fields: 0 (2 checks)
    • Check: A required check is below its minimum. Required minimum: 0.995.
      • Matching records (F1): 0
      • Numerical agreement (CCC): 0
      • Present, finite data cells (F1): 0
      • Required fields: 0
  41. DeepSeek V4 Pro / Open-ended code / run 1 → result 3 (identifications)
    • Content: Content score below 0.80. Value: 0.0519179. Required minimum: 0.8.
      • Matching records (F1): 0.227885 to 0.99987 (2 checks)
      • Present, finite data cells (F1): 0.227885 to 0.99987 (2 checks)
  42. DeepSeek V4 Pro / Open-ended code / run 1 → result 4 (PSM q-values)
    • Content: Content score below 0.80. Value: 0.492488. Required minimum: 0.8.
      • Category agreement (F1): 0.871693
      • Matching records (F1): 0.935262
      • Numerical agreement (CCC): 0.6459
      • Present, finite data cells (F1): 0.935262
  43. DeepSeek V4 Pro / Open-ended code / run 1 → result 7 (isotope peptides)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
  44. DeepSeek V4 Pro / Open-ended code / run 1 → result 8 (isotope groups)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Required content missing or unreadable: 0
  45. DeepSeek V4 Pro / Open-ended code / run 2 → result 3 (identifications)
    • Content: Content score below 0.80. Value: 7.11182e-06. Required minimum: 0.8.
      • Matching records (F1): 0.0516411 (2 checks)
      • Present, finite data cells (F1): 0.0516411 (2 checks)
  46. DeepSeek V4 Pro / Open-ended code / run 2 → result 4 (PSM q-values)
    • Content: Content score below 0.80. Value: 0.00118706. Required minimum: 0.8.
      • Category agreement (F1): 0.871292
      • Matching records (F1): 0.0471084
      • Numerical agreement (CCC): 0.613921
      • Present, finite data cells (F1): 0.0471084
  47. DeepSeek V4 Pro / Open-ended code / run 2 → result 7 (isotope peptides)
    • Content: Content score below 0.80. Value: 0.75. Required minimum: 0.8.
      • Numerical agreement (CCC): 0.75 to 1 (2 checks)
  48. DeepSeek V4 Pro / Open-ended code / run 2 → result 8 (isotope groups)
    • Content: Content score below 0.80. Value: 0.6. Required minimum: 0.8.
      • Numerical agreement (CCC): 1
      • Required fields: 0.75 to 0.8 (2 checks)
    • Check: A required check is below its minimum. Required minimum: 0.995.
      • Required fields: 0.75 to 0.8 (2 checks)
  49. DeepSeek V4 Pro / Open-ended code / run 3 → result 2 (features)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Matching records (F1): 0.250797 to 0.268813 (2 checks)
      • Numerical agreement (CCC): 0 to 0.999996 (3 checks)
      • Present, finite data cells (F1): 0.250797 to 0.268813 (3 checks)
  50. DeepSeek V4 Pro / Open-ended code / run 3 → result 3 (identifications)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Category agreement (F1): 0.5 (2 checks)
      • Matching records (F1): 0.121955 (2 checks)
      • Numerical agreement (CCC): 0 (2 checks)
      • Present, finite data cells (F1): 0.121955 (2 checks)
  51. DeepSeek V4 Pro / Open-ended code / run 3 → result 4 (PSM q-values)
    • Method: Delivered target leading-hit q-values do not follow the stated D/T rule on the main score the delivered search output declares.Task requirement: For PSM-level FDR, count one leading hit per spectrum; when first rank is tied, use the first hit in the original search-result order. At each score threshold, let D and T count decoy and target leading hits with scores at least as good as that threshold; a hit mapping to both target and decoy proteins counts as target. Estimate FDR as D/T, without a pseudocount, and obtain each target hit’s q-value as the minimum FDR over thresholds that include it. Pool charge states and accept target PSMs at q≤0.01. Retain all target leading-hit q-values and the accepted set; do not substitute peptide- or protein-level FDR.
  52. DeepSeek V4 Pro / Open-ended code / run 3 → result 6 (feature identification links)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Matching records (F1): 0.250797 to 0.735849 (3 checks)
      • Numerical agreement (CCC): 0 to 0.999996 (3 checks)
      • Present, finite data cells (F1): 0.250797 to 0.735849 (4 checks)
    • Check: A required check is below its minimum. Required minimum: 0.998.
      • Matching records (F1): 0.735849
  53. DeepSeek V4 Pro / Open-ended code / run 3 → result 7 (isotope peptides)
    • Content: Content score below 0.80. Value: 0.0545381. Required minimum: 0.8.
      • Matching records (F1): 0.679245 to 0.699972 (3 checks)
      • Numerical agreement (CCC): 0.75 to 1 (2 checks)
      • Present, finite data cells (F1): 0.679245 to 0.699972 (4 checks)
  54. DeepSeek V4 Pro / Open-ended code / run 3 → result 8 (isotope groups)
    • Content: Content score below 0.80. Value: 0. Required minimum: 0.8.
      • Matching records (F1): 0 to 0.666667 (7 checks)
      • Numerical agreement (CCC): 0 to 1 (3 checks)
      • Present, finite data cells (F1): 0 to 0.666667 (8 checks)
      • Required fields: 0 to 0.8 (4 checks)
    • Check: A required check is below its minimum. Required minimum: 0.995.
      • Matching records (F1): 0 to 0.666667 (4 checks)
      • Numerical agreement (CCC): 0
      • Present, finite data cells (F1): 0 to 0.666667 (5 checks)
      • Required fields: 0 to 0.8 (3 checks)

Task text

The scientific part of the task, identical in both conditions. The general submission rules and the instructions for each condition are on the overview page.

Full task text

Task

Identify peptides and quantify carbon-isotope incorporation from the supplied centroided liquid chromatography–mass spectrometry acquisition and metaproteome FASTA. This is one acquisition, not biological replication; do not infer differential abundance or organism-level activity. Use natural-abundance peptide isotope envelopes and chromatographic elution as MS1 feature anchors. Link them to database-supported MS/MS identifications, then infer carbon incorporation by sequence-informed isotope-pattern fitting and decomposition. Estimate a carbon-labeling distribution, not a fixed light/heavy amino-acid pair ratio.

Inputs

  • One complete centroided liquid chromatography–mass spectrometry acquisition and its acquisition metadata.
  • The metaproteome FASTA with protein accession identities.

Key parameters and conditions

  • Detect MS1 anchors without prior peptide identifications using mass-shift isotope-pattern filtering and m/z–RT clustering, in an unlabelled singlet mode rather than fixed light/heavy pairs. Require charges 1–4, 3–6 isotope peaks, m/z tolerance 6 ppm and intensity at least 1,000. The isotope intensities must correlate with peptide averagine by both Pearson and Spearman at ≥0.4. Use a typical elution scale of 40 s for clustering, not a maximum duration; retain features lasting at least 2 s.
  • Use spectral-generating-function database scoring, fully tryptic specificity allowing cleavage before proline, fixed cysteine carbamidomethylation and variable methionine oxidation (at most two variable modifications per peptide). Search lengths 6–40, isotope offsets 0/+1 and precursor tolerance 10 ppm. For spectra lacking charge information, consider charges 2–3; do not filter known charges by this range. Preserve all compatible protein accessions. Use the low-resolution instrument scoring model and the fragmentation method recorded in each spectrum. Do not impose a missed-cleavage-count cap. In both search-identification outputs, retain every hit tied for first rank within a spectrum, preserving their original order. Protein mapping distinguishes I from L, allows no residue mismatches and permits protein N-terminal methionine removal.
  • Use a target-decoy database with peptide-internal pseudoreversal: tryptic cleavage sites (no cleavage before proline) and each peptide’s terminal residues are preserved; decoy accessions have prefix DECOY_. For PSM-level FDR, count one leading hit per spectrum; when first rank is tied, use the first hit in the original search-result order. At each score threshold, let D and T count decoy and target leading hits with scores at least as good as that threshold; a hit mapping to both target and decoy proteins counts as target. Estimate FDR as D/T, without a pseudocount, and obtain each target hit’s q-value as the minimum FDR over thresholds that include it. Pool charge states and accept target PSMs at q≤0.01. Retain all target leading-hit q-values and the accepted set; do not substitute peptide- or protein-level FDR.
  • Map accepted identifications to features using 20 ppm m/z and 5 s RT tolerances with compatible charge. Use the same complete generated search database for identification and isotope inference. Only feature-linked identifications passing the PSM threshold enter incorporation inference; retain unassigned identifications separately in the mapping output. For matching, use feature centroid m/z and its chromatographic hull RT range expanded by 5 s; do not replace the RT range by a ±5 s window around its apex. Compare the calculated peptide-ion m/z with a ±20 ppm interval centered on the feature m/z.
  • Estimate carbon labeling using m/z tolerance 10 ppm, RT tolerance 30 s and minimum MS1 intensity 10. Set isotope-pattern correlation and extracted-ion-chromatogram correlation thresholds to 0.7, and decomposition fit R² threshold to 0.7. Collect incorporation estimates at correlation maxima from sequence-informed isotope-pattern decomposition, without clustering by relative isotope abundance.
  • Retain relative isotope abundance, labeling estimates, fit quality, ambiguity and group summaries. The peptide-level incorporation report must exclude purely natural-abundance peptides, while retaining unresolved fields actually reported within the included records.
  • During protein mapping, allow at most three ambiguous amino-acid codes. In isotope inference combine decomposition weights within ±5 percentage points, require at least two consecutive isotopic intensities and at least half of the expected isotope peaks to be observed; retain 95% of the theoretical carbon-isotope pattern intensity.

Required results

  1. final_answer/01_search_database.fasta
    FASTA
    Complete generated target/decoy search database, with construction recorded in method.md.
  2. final_answer/02_features/
    featureXML
    All detected features with identities, coordinates, intensities and isotope-envelope evidence. Include the stored feature quality score and complete isotope hulls. Files: features.featureXML.
  3. final_answer/03_identifications/
    idXML
    Both original search identifications and protein-mapped identifications, retaining spectrum–peptide identities, all tied first-rank hits in their original order, ranking evidence and all compatible accessions. Each hit in both files must carry its target, decoy or target-and-decoy classification based on those protein mappings. Files: indexed.idXML, search.idXML.
  4. final_answer/04_psm_qvalues/
    idXML
    Target leading-hit PSM q-values before the final 1% threshold and the target/decoy evidence supporting them. The supporting target/decoy evidence may be identified in the submitted original search output; it need not be recomputed into an extra table. Files: qvalues.idXML.
  5. final_answer/05_psm_accepted/
    idXML
    The thresholded PSM set with acceptance criteria. Files: accepted.idXML.
  6. final_answer/06_feature_identification_links/
    featureXML
    Feature-to-identification relationships and unassigned identifications, preserving multiplicity. Files: mapped.featureXML.
  7. final_answer/07_isotope_peptides/
    native peptide isotope TSV
    Native peptide-level abundance, incorporation, fit and ambiguity evidence for the incorporation report, excluding peptides explained solely by one natural-abundance component (relative isotope abundance below 5%). Preserve unresolved fields actually reported within the included records. Do not add natural-abundance-only or placeholder peptide rows. For identifications with no incorporation-report entry, retain the submitted identification/feature evidence; do not invent estimates or per-peptide failure reasons. Files: peptides.tsv.
  8. final_answer/08_isotope_groups/
    native group isotope TSV
    Complete native group and protein summaries covering every peptide with an accepted isotope-pattern fit and at least one supported relative-isotope-abundance estimate, including peptides with only natural-abundance isotope patterns, with each peptide's unique or shared protein assignments. Files: groups.tsv.

How this task is scored

Each scored result is compared with the reference result, which the task authors made by running the IWC workflow behind this task on usegalaxy.org with the settings in the task text. A result's score is the product of the checks listed for it, each from 0 to 1. A passing result must score at least 0.80 and meet the required member checks, method and output instructions. A failing result retains its numerical score in the continuous average. The overview page describes the checks in more detail.

No.Required resultWhat the task asks for How it is scored
101_search_database.fasta
FASTA		Complete generated target/decoy search database, with construction recorded in method.md.
  • the same rows or records as the reference (F1 of their identifiers)
  • the required columns or fields are present (F1)
  • values are present in the same cells (F1)
  • values agree (Lin's concordance correlation)
  • category labels agree (macro F1)
202_features/
featureXML		All detected features with identities, coordinates, intensities and isotope-envelope evidence. Include the stored feature quality score and complete isotope hulls. Files: features.featureXML.
  • the same rows or records as the reference (F1 of their identifiers)
  • the required columns or fields are present (F1)
  • values are present in the same cells (F1)
  • values agree (Lin's concordance correlation)
  • counts or intensities agree on a log(1 + x) scale (concordance correlation)
  • category labels agree (macro F1)
303_identifications/
idXML		Both original search identifications and protein-mapped identifications, retaining spectrum–peptide identities, all tied first-rank hits in their original order, ranking evidence and all compatible accessions. Each hit in both files must carry its target, decoy or target-and-decoy classification based on those protein mappings. Files: indexed.idXML, search.idXML.
  • the same rows or records as the reference (F1 of their identifiers)
  • the required columns or fields are present (F1)
  • values are present in the same cells (F1)
  • values agree (Lin's concordance correlation)
  • category labels agree (macro F1)
404_psm_qvalues/
idXML		Target leading-hit PSM q-values before the final 1% threshold and the target/decoy evidence supporting them. The supporting target/decoy evidence may be identified in the submitted original search output; it need not be recomputed into an extra table. Files: qvalues.idXML.
  • the same rows or records as the reference (F1 of their identifiers)
  • the required columns or fields are present (F1)
  • values are present in the same cells (F1)
  • values agree (Lin's concordance correlation)
  • category labels agree (macro F1)
505_psm_accepted/
idXML		The thresholded PSM set with acceptance criteria. Files: accepted.idXML.
  • the same rows or records as the reference (F1 of their identifiers)
  • the required columns or fields are present (F1)
  • values are present in the same cells (F1)
  • values agree (Lin's concordance correlation)
  • category labels agree (macro F1)
  • every accepted match is a target match with a q-value of at most 0.01
606_feature_identification_links/
featureXML		Feature-to-identification relationships and unassigned identifications, preserving multiplicity. Files: mapped.featureXML.
  • the same rows or records as the reference (F1 of their identifiers)
  • the required columns or fields are present (F1)
  • values are present in the same cells (F1)
  • values agree (Lin's concordance correlation)
  • counts or intensities agree on a log(1 + x) scale (concordance correlation)
  • category labels agree (macro F1)
707_isotope_peptides/
native peptide isotope TSV		Native peptide-level abundance, incorporation, fit and ambiguity evidence for the incorporation report, excluding peptides explained solely by one natural-abundance component (relative isotope abundance below 5%). Preserve unresolved fields actually reported within the included records. Do not add natural-abundance-only or placeholder peptide rows. For identifications with no incorporation-report entry, retain the submitted identification/feature evidence; do not invent estimates or per-peptide failure reasons. Files: peptides.tsv.
  • the same rows or records as the reference (F1 of their identifiers)
  • the required columns or fields are present (F1)
  • values are present in the same cells (F1)
  • values agree (Lin's concordance correlation)
  • counts or intensities agree on a log(1 + x) scale (concordance correlation)
  • category labels agree (macro F1)
808_isotope_groups/
native group isotope TSV		Complete native group and protein summaries covering every peptide with an accepted isotope-pattern fit and at least one supported relative-isotope-abundance estimate, including peptides with only natural-abundance isotope patterns, with each peptide's unique or shared protein assignments. Files: groups.tsv.
  • the same rows or records as the reference (F1 of their identifiers)
  • the required columns or fields are present (F1)
  • values are present in the same cells (F1)
  • values agree (Lin's concordance correlation)
  • counts or intensities agree on a log(1 + x) scale (concordance correlation)
  • category labels agree (macro F1)

Data updated 27 October 2026, 08:48 EDT.