Back to results

CompBioBench item

compute-hvg-jaccard-q1

Question

Using scanpy and the two AnnData H5AD files below, compute the Jaccard index (similarity) between their HVG. First subset them to common genes using the intersection of all gene names. Use exact string values from adata.var_names to resolve genes. Use sc.pp.highly_variable_genes with seurat flavor for HVG identification and default parameters. A gene is considered HVG if var['highly_variable'] string-casts exactly to any of the following (deduplicate this list as needed in order): ["True", "True", "1", "TRUE"]. Do not apply any var-level filter beyond excluding rows with missing gene_symbols. Return NaN if thresholds are not met. The datasets are: pbmc3k.h5ad and 10x_pbmc68k_reduced.h5ad. Round the answer to two decimal places.

Predicted answer
0.22Most likely answer for this question; not uniquely determined from submitted scores.
Domain
Single-cell
Question style
Routine Analysis

An answer can be uniquely inferred for 54 of 100 questions from submitted scores. Green: correct · Red: incorrect. For questions without a uniquely determined answer: Blue: matches prediction · Orange: differs from prediction. These are inferred answer checks, not official per-question grades.

Galaxy

12/12 terminal item results

UDT = user-defined tool: custom code executed by Galaxy. Route badges summarize analysis tools, excluding uploads and automatic conversions. How usage is counted.

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 2

Matches predictionUDT-only
Answer0.22
Workflow
HVG Jaccard DiagnosticHVG Jaccard Final
Total tokens
451.1K
Output
5.1K
Codex GPT-5.5

Replicate 3

Matches predictionUDT + standard tools
Answer0.22
Workflow
HVG Jaccard DiagnosticUDT Render ProbeUDT Render Probe Scalar FormatUDT Echo ProbeAnndata Inspect9 more tools
Total tokens
7.1M
Output
22.2K

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 2

Matches predictionUDT-only
Answer0.22
Workflow
Inspect PBMC H5AD inputsPBMC HVG Jaccard with Scanpy Seurat defaults
Total tokens
655.6K
Output
5.9K
Codex GPT-5.6 Sol

Replicate 3

Matches predictionUDT-only
Answer0.22
Workflow
Inspect H5AD inputs for exact HVG comparisonExact Scanpy Seurat HVG Jaccard
Total tokens
6.9M
Output
10.7K

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

Matches predictionUDT-only
Answer0.22
Workflow
HVG Jaccard ScanpyScanpy Version Probe
Total tokens
5.2M
Output
34.8K
Codex + DeepSeek-v4-pro-0813

Replicate 2

Matches predictionUDT + standard tools
Answer0.22
Workflow
No Galaxy tool job recorded
Total tokens
38.6M
Output
64.4K
Codex + DeepSeek-v4-pro-0813

Replicate 3

Matches predictionUDT + standard tools
Answer0.22
Workflow
Inspect HVGsUDT ProbeAnndata ExportSelect columnsFilter Tabular5 more tools
Total tokens
18.1M
Output
58.1K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

Matches predictionUDT-only
Answer0.22
Workflow
Scanpy HVG ProbeScanpy HVG Jaccard
Total tokens
565.2K
Output
14.3K
Codex GPT-5.6 Luna

Replicate 2

Matches predictionUDT + standard tools
Answer0.22
Workflow
Scanpy HVG diagnosticScanpy runtime probeUDT render probeUdt Render Probe V1UDT format probe5 more tools
Total tokens
15.3M
Output
54.3K
Codex GPT-5.6 Luna

Replicate 3

Matches predictionUDT-only
Answer0.22
Workflow
Galaxy HVG Metadata DiagnosticGalaxy Common Gene Seurat HVG ProbeGalaxy Common Gene Seurat HVG Jaccard
Total tokens
1.3M
Output
19.5K

Anycode

13/13 terminal item results

Codex GPT-5.5

3 replicates
Codex GPT-5.5

Replicate 1

Matches prediction
Answer0.22
Workflow
Local codePythonScanpyAnnData
Codex GPT-5.5

Replicate 2

Matches prediction
Answer0.22
Workflow
Local codePythonScanpyAnnDataSciPy
Codex GPT-5.5

Replicate 3

Matches prediction
Answer0.22
Workflow
Local codePythonScanpyAnnData

Codex GPT-5.6 Sol

3 replicates
Codex GPT-5.6 Sol

Replicate 1

Matches prediction
Answer0.22
Workflow
Local codePythonScanpySciPyAnnData
Codex GPT-5.6 Sol

Replicate 2

Matches prediction
Answer0.22
Workflow
Local codePythonScanpySciPyAnnData
Codex GPT-5.6 Sol

Replicate 3

Matches prediction
Answer0.22
Workflow
Local codePythonScanpyAnnDataSciPy

Codex + DeepSeek-v4-pro-0813

3 replicates
Codex + DeepSeek-v4-pro-0813

Replicate 1

Matches prediction
Answer0.22
Workflow
Local codePythonScanpyAnnDataSciPy
Total tokens
565.4K
Output
12.2K
Codex + DeepSeek-v4-pro-0813

Replicate 2

Matches prediction
Answer0.22
Workflow
Local codePythonScanpyAnnDataSciPy
Total tokens
701.0K
Output
9.1K
Codex + DeepSeek-v4-pro-0813

Replicate 3

Matches prediction
Answer0.22
Workflow
Local codePythonScanpyAnnDataSciPy
Total tokens
658.7K
Output
16.0K

Codex GPT-5.6 Luna

3 replicates
Codex GPT-5.6 Luna

Replicate 1

Matches prediction
Answer0.22
Workflow
Local codePythonScanpyAnnDataSciPy
Codex GPT-5.6 Luna

Replicate 2

Matches prediction
Answer0.22
Workflow
Local codePythonScanpyAnnData
Codex GPT-5.6 Luna

Replicate 3

Matches prediction
Answer0.22
Workflow
Local codePythonScanpyAnnData

Codex GPT-6 Astra

1 replicate
Codex GPT-6 Astra

Replicate 1

Matches prediction
Answer0.22
Workflow
Local codePythonScanpy
Total tokens
299.8K
Output
899