Error Analysis

One replicate used the right input values but reported the Mann-Whitney U statistic for the opposite group order.

For this test, swapping animals and fungi changes the reported U value even when the underlying ranks are the same.

Unstable under Open-ended code + skills: at least one replicate passed and at least one replicate failed in the same condition.

This is an answer-format/setup failure for that replicate, not a dataset problem.

Some no-skills replicates ranked the raw CRISPR gene-effect correlation instead of the biological essentiality direction.

In this dataset, more negative CRISPR scores mean stronger dependency. Ranking the raw sign answers a different question.

The same condition did not flip between pass and fail; the failure pattern is consistent across replicates.

Count those replicates as wrong because they used the wrong score direction.

Failing replicates counted positive correlations on the raw CRISPR score.

The benchmark expects the dependency/essentiality direction. With that direction, three genes pass the threshold; with the raw sign, the replicate gets none.

Unstable under Galaxy no skills, Open-ended code no skills: at least one replicate passed and at least one replicate failed in the same condition.

This is the same score-direction mistake as bix-16-q1, applied to a threshold count.

One Galaxy no-skills replicate chose the smallest signed Pearson correlation.

The accepted answer treats weakest as closest to zero by absolute strength. That gives a different cell type.

Unstable under Galaxy no skills: at least one replicate passed and at least one replicate failed in the same condition.

This replicate failed because it used a different interpretation of 'weakest'.

Some replicates merged pathway results too early and missed one pathway that is significant only in the iron-depleted condition.

The item asks for a condition-specific comparison after enrichment, not a pooled pathway list.

The same condition did not flip between pass and fail; the failure pattern is consistent across replicates.

Failed replicates answered a simpler pooled comparison rather than the requested condition-specific count.

No-skills replicates reported about 56.47%, while the accepted calculation is 55.60%.

Small PCA setup choices such as matrix orientation, centering, and variance denominator change the percentage.

The same condition did not flip between pass and fail; the failure pattern is consistent across replicates.

This is a setup mismatch: the replicate computed a plausible PCA number, but not the benchmark's PCA number.

One replicate returned a single PhyKIT output value.

The question asks for the median across the target set, so a single output row is not enough.

Unstable under Galaxy no skills: at least one replicate passed and at least one replicate failed in the same condition.

This is a summarization failure: the tool output was not aggregated to the requested statistic.

Two no-skills anycode replicates used a t-test setup that found one Bonferroni-significant miRNA.

The accepted benchmark setup gives zero Bonferroni hits and zero BY hits.

Unstable under Open-ended code no skills: at least one replicate passed and at least one replicate failed in the same condition.

These replicates failed because the statistical test setup did not match the benchmark setup.

Some replicates found two or three common pathways instead of zero.

The KEGG enrichment background and direction filter changed which pathways passed in all three mutants.

Unstable under Open-ended code + skills, Open-ended code no skills: at least one replicate passed and at least one replicate failed in the same condition.

This item is unstable because small enrichment-setup changes change the final intersection.

One Galaxy no-skills replicate computed the median from a partial set of pairwise distances.

The accepted answer uses six pairwise distances for the requested taxa.

Unstable under Galaxy no skills: at least one replicate passed and at least one replicate failed in the same condition.

The failed replicate used the right kind of metric but the wrong row set.

Replicates used different Reactome/gseapy rows or foreground sizes for the p53 cell-cycle term.

The same human-readable pathway name can map to different rows or odds-ratio formulas across library snapshots.

Unstable under Galaxy + skills: at least one replicate passed and at least one replicate failed in the same condition.

Failed replicates are enrichment-version/scope mismatches, not arithmetic mistakes.

Replicates consistently computed a different p-value from the verifier.

The difference is likely the Mann-Whitney mode or tie/continuity convention, not random replicate behavior.

The same condition did not flip between pass and fail; the failure pattern is consistent across replicates.

Treat this as a verifier-convention mismatch when interpreting the failure count.

Replicates alternated among nearby counts such as 2118, 2117, 2119, and 2106.

Sample exclusions, shrinkage route, non-NA adjusted-p rows, or counted-row scope changed the final number.

Unstable under Galaxy + skills, Galaxy no skills, Open-ended code + skills, Open-ended code no skills: at least one replicate passed and at least one replicate failed in the same condition.

This is an unstable counting item: several routes are close, but only one matches the benchmark answer.

One replicate counted 19160 removed rows instead of 19159.

The most likely cause is including the header row or using the boundary condition differently.

Unstable under Galaxy no skills: at least one replicate passed and at least one replicate failed in the same condition.

This is a row-counting failure, not a model or Galaxy-tool failure.

Most replicates fit the mixture-only branch and returned about 180771 instead of the accepted 184k range.

The prompt leaves the endpoint scope easy to read in more than one way.

Unstable under Galaxy + skills: at least one replicate passed and at least one replicate failed in the same condition.

This failure should be read as scope ambiguity in the eval item.

One no-skills anycode replicate counted complete BUSCO markers including duplicated hits.

The question asks for single-copy orthologs, so duplicated complete markers should not be counted.

Unstable under Open-ended code no skills: at least one replicate passed and at least one replicate failed in the same condition.

This is a category-filter failure in the BUSCO output.

No-skills replicates used the raw VCF Ts/Tv ratio.

The accepted answer uses the filtered SNP scope, which gives a different Ts/Tv for the MDR sample.

The same condition did not flip between pass and fail; the failure pattern is consistent across replicates.

This is a variant-scope failure: raw variants and filtered SNPs answer different questions.