Blog · arXiv Analysis · Last reviewed July 12, 2026

The Epigenomics Benchmark Becomes the Science Gate

EpiBench is valuable because it measures short, concrete scientific decisions instead of letting tool use masquerade as discovery.

The result is blunt: current agents often find files and compute intermediates, but still fail when the next step requires assay-specific judgment.

The Paper

The paper is EpiBench: Verifiable Evaluation of AI Agents on Epigenomics Analysis, arXiv:2606.13602 [cs.AI]. arXiv lists it as submitted on June 11, 2026, with DOI 10.48550/arXiv.2606.13602 and a CC BY 4.0 license.

The authors are Harihara Muralidharan, Reema Baskar, Soo Hee Lee, Tim Proctor, and Kenny Workman of LatchBio. The paper introduces a benchmark for short-horizon epigenomics analysis tasks with deterministic grading.

The Benchmark

EpiBench evaluates whether agents can make well-defined analysis decisions from realistic workflow states and return structured answers that can be graded deterministically. That framing is important. The benchmark does not ask whether an agent can write a broad literature review or generate plausible biological prose. It asks whether it can make the next assay-grounded decision correctly.

The benchmark includes 106 evaluations across CUT&Tag/CUT&RUN, ATAC-seq, ChIP-seq, and DNA methylation workflows. The inventory spans eight task categories: downstream analysis, QC, chromatin state analysis, peak calling, genomic annotation, differential methylation, alignment, and visualization.

The task sources are real workflow artifacts. CUT&Tag/CUT&RUN tasks come from zebrafish chromatin workflow snapshots. ATAC-seq tasks use pediatric B-ALL ATAC/RNA data from GSE161501 and related workflow artifacts. ChIP-seq tasks use B-ALL H3K27ac data. DNA methylation tasks use GSE149608 and GSE149609.

Scores

The paper evaluates 16 model-harness pairs, with each pair run three times on each of the 106 evaluations. That yields 318 attempts per pair and 5,088 valid trajectories total.

No model-harness pair reaches a 50 percent endpoint pass rate. GPT-5.5 / Pi leads at 45.0 percent, or 143/318 attempts, with 95 percent CI 36.3 to 53.7. GPT-5.5 / OpenAI Codex follows at 39.9 percent, or 127/318 attempts, with 95 percent CI 31.6 to 48.3. Claude Opus 4.8 Max / Pi and GPT-5.4 / Pi each pass 39.0 percent, or 124/318 attempts, with overlapping confidence intervals.

The overlap matters. EpiBench does not crown a decisive science-agent winner. It exposes a broad ceiling: the strongest current model-harness pairs still fail most endpoint-graded attempts.

Assay Difficulty

Aggregate pass rates vary by assay family. CUT&Tag/CUT&RUN has the highest reported pass rate at 34.0 percent, or 768/2,256 attempts. Methylation-seq follows at 33.3 percent, or 400/1,200 attempts. ChIP-seq is 30.6 percent, or 147/480 attempts. ATAC-seq is lowest at 22.8 percent, or 263/1,152 attempts.

The authors caution against reading those as clean intrinsic difficulty estimates. The assay groups differ in size and task composition: CUT&Tag/CUT&RUN contributes 47 evaluations, methylation-seq 25, ATAC-seq 24, and ChIP-seq 10. Downstream analysis and QC dominate the task inventory.

That is the right caveat. Benchmarks of scientific work can accidentally measure task mix, artifact familiarity, harness habits, or workflow defaults rather than domain competence.

Failure Shape

The most useful result is the gap between partial work and correct endpoints. The paper reports 68.2 percent component-field pass rate across 26,574 scored answer fields from 5,051 trajectories, compared with a 31.0 percent endpoint pass rate across 5,088 attempts. Agents often found the right files and computed useful intermediate values, but failed to assemble the scientifically correct final decision.

Manual review covers 25 of 106 evaluations and is diagnostic rather than exhaustive. The recurring pattern is not simple tool failure. Agents choose plausible defaults, follow generic bioinformatics expectations, or rely on familiar workflows when the specific assay artifacts point elsewhere.

That is exactly where science-agent evaluation should bite. If an agent can run commands but cannot tell which assay-specific evidence controls the answer, it is a productive assistant with a dangerous autopilot label.

Science-Agent Receipt

A science-agent receipt should name the assay, source dataset, workflow snapshot, available files, target decision, accepted answer schema, deterministic grader, harness, model, retry count, runtime, cost, intermediate artifacts, endpoint answer, failed fields, manual-review label, and scientific rationale.

It should also separate execution from judgment. File discovery, code execution, intermediate calculation, endpoint formatting, and assay interpretation are different skills. EpiBench is useful because it shows agents making progress on some of those skills while still failing the endpoint that matters.

This belongs beside AI Agents, AI Agent Observability, The Agent Environment Becomes the Discovery Lab, The Benchmark Becomes the Judge Agent, The Reanalysis Agent Becomes the Reproducibility Screen, and The Lab Notebook Becomes the Discovery Engine.

Limits

The benchmark is deliberately bounded. It is not a complete test of epigenomics reasoning, lab practice, biological discovery, or long-horizon scientific planning. It tests a measurable sample of short-horizon decisions where deterministic grading is feasible.

The task inventory is imbalanced across assay families and task categories. Related evaluations may repeat underlying decision patterns. Deterministic graders constrain the answer surface and may miss alternative scientifically valid paths. The authors explicitly frame component scores and failure labels as diagnostics rather than benchmark scores.

The practical conclusion is sober. Agents are already useful enough to navigate files and compute pieces of the answer. They are not yet reliable enough to own epigenomics decisions without assay-specific review, endpoint receipts, and human accountability.

Sources

Return to Blog