Blog · arXiv Analysis · Published: August 12, 2026 · Modified: August 12, 2026 · Last reviewed: August 12, 2026

The Relevant Skill Becomes the Wrong Procedure

An agent skill need not be hostile or obviously irrelevant to cause failure. Reusable guidance can fit the topic while distorting a required field, path, environment, or work budget.

Relevance is a retrieval result. Compatibility is a task-level evidence claim, and successful execution is a third claim again.

The Paper

The source is Gen Dong, Yanjie Gao, Liqun Li, Tianyin Xu, Yu Hua, and Fan Yang's Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents, arXiv:2608.11888v1 [cs.AI], submitted August 12, 2026. The twelve-page, CC BY 4.0 paper lists affiliations at Huazhong University of Science and Technology, Microsoft Research, Microsoft, and the University of Illinois Urbana-Champaign. It studies reusable instruction packages that can shape an agent's planning, tool use, edits, validation, and stopping behavior.

This is not another malicious-skill study. Its question is more ordinary and therefore more institutional: how can apparently applicable guidance make an otherwise solvable task fail, or make a successful run substantially more expensive?

The Differential Comparison

The study design uses SkillsBench, with 84 evaluated tasks across 11 domains, and SWE-Skills-Bench, with 490 repository tasks. The authors add public skills retrieved from Smithery and SkillsMP by metadata similarity, retaining as many as five candidates per curated skill at a cosine-similarity threshold of 0.7. Runs use OpenCode 1.15.1 and Claude Opus 4.6. Within a pair, the task, verifier, model, harness, data, and repository or container state are fixed; the skill setup changes.

A functional case requires the audited skill run to fail while a no-skill or semantically matched-skill run passes. An efficiency case requires both runs to pass, both token use and elapsed time to increase, and at least one to more than double. The reference is deliberately called a pseudo-oracle: it shows that the same task can be solved, or solved more cheaply, under the paired setup. It is not a universal ground-truth implementation.

Relevance Is Not Compatibility

The expanded design creates 20,664 potential pair comparisons. Executed evaluations produce 665 labeled candidates; after removing ambiguous, verifier-narrow, and duplicate cases, manual inspection and group consensus leave 307 analysis cases: 125 functional failures and 182 high-confidence efficiency regressions. This is a deliberately mined case set, not an estimate that skills fail at some population rate.

In the functional taxonomy, only 2 of the 125 failures are classified as applicability mismatch. By contrast, 86 are task-implementation faults, 24 are artifact misplacements, and 13 are environment mismatches. The result separates semantic proximity from operational fit. A skill can mention the right framework and workflow while carrying the wrong assumptions about the concrete contract.

When Examples Become Requirements

Within the 86 task-implementation faults, the authors assign 46 to incorrectly filled required elements and 36 to omitted required elements. Their cases include a reusable example that steers an agent toward a ratio when the task requires a percentage, guidance that covers retrieval parameters but leaves out a required model parameter, and package-layout advice that sends an otherwise plausible artifact to a path the verifier does not inspect.

The governance lesson is not to eliminate examples. It is to mark their authority. Defaults, templates, paths, libraries, and checklists should declare whether they are mandatory, optional, illustrative, or conditional. The task specification must win a conflict unless an authorized policy says otherwise.

When the Checklist Becomes the Spend

Of the 182 efficiency regressions, 114 are classified as excessive procedure, 46 as context bloat, and 22 as dependency resolution. Excessive verification alone accounts for 67 cases; heavy implementation pipelines account for 30. A skill can therefore cost more than its prompt length. It can induce extra searches, builds, tests, retries, simulations, or dependency repairs across the trajectory.

That finding complicates the slogan that more verification is always safer. Verification scope should respond to change risk, uncertainty, repository size, available budget, and the consequences of failure. A universal checklist can consume the time needed to finish the task or obscure which checks actually protect the outcome.

Triage After Confirmation

The paper's SkillTriage tool does not discover failures from an unaudited skill catalog. It receives a target/reference pair already confirmed as a failure or regression, extracts differential evidence, and assigns a taxonomy label. Using GPT-5.5 three times with a two-of-three vote, it matches the manual exact subcategory on 111 of 125 functional cases and 132 of 182 efficiency cases.

Those are agreement results on the same case sets used for the manual taxonomy, not a held-out deployment test. The useful artifact is the evidence bundle: which instruction, trajectory step, path, environment change, or cost-heavy phase supports the label. An automated label without that contrast is merely another verdict.

The Claim Boundary

The evidence comes from two benchmark ecosystems, one agent harness, and one execution model. A deterministic verifier judges each individual run, but model execution can still vary. Root-cause assignment requires manual judgment; the authors report excluding ambiguous cases and reaching group consensus. Results may depend on the tasks, harness, model, and skill ecosystem, as the paper's validity discussion acknowledges.

The reviewed version 1 source archive contains the manuscript and three figure files, but identifies no released run-level dataset, trajectory archive, or study code. The case counts are inspectable in the paper; independent replay is not supplied there.

The Skill-Compatibility Receipt

A skill-compatibility receipt should bind the task specification to the skill's publisher, version, hash, retrieval query, relevance score, loaded sections, declared assumptions, mandatory and optional steps, permitted paths, environment constraints, dependencies, token and time budget, validation scope, no-skill or alternate-skill comparison, observed trace, verifier result, cost delta, reviewer, and expiration. Compatibility should be reassessed when the task, skill, harness, model, environment, or verifier changes.

The Governance Standard

A registry scan can ask whether a skill is malicious. A router can ask whether it is relevant. Neither establishes that its procedure fits this task. Before reusable guidance becomes executable context, compare its assumptions with the task's required fields, paths, formats, environment, and budget; preserve a counterfactual where feasible; and log the divergence between planned and observed work. Helpful procedure becomes authority when the agent follows it. Authority needs a receipt.

Sources

Return to Blog