Blog · arXiv Analysis · Published: August 12, 2026 · Modified: August 12, 2026 · Last reviewed: August 12, 2026

The Language Switch Becomes the Policy Fork

Sourabrata Mukherjee, Kalika Bali, and Sunayana Sitaram measure whether a tool-using model preserves its action route when only the task language changes.

A policy-fork receipt records the paired tasks, action traces, same-language baseline, parser failures, and human authority behind a cross-lingual agent claim.

The Paper

The paper is Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents, arXiv:2608.11110v1 [cs.CL], by Sourabrata Mukherjee, Kalika Bali, and Sunayana Sitaram. arXiv records submission on August 11, 2026 and acceptance at COLM 2026. The 35-page version 1 PDF lists Microsoft Research India as the affiliation.

The question is narrower than multilingual capability: when two prompts express the same task in different languages, does an agent choose the same sequence of actions? The route can change cost, latency, failure points, and which operational safeguard applies even when both final answers look acceptable.

The Policy Fork

This is distinct from the site's PolyWorkBench essay, which examines long-horizon workplace artifacts, model-harness pairs, and grader disagreement. Here the measured object is a paired symbolic action trace. The fork is procedural: translate first or not, search before calculating or after, finish in three steps or consume the full iteration budget.

Different routes are not automatically wrong. A language may require a different operation. The assurance failure begins when an institution certifies the English route and silently assumes that certification travels with a non-English answer.

The Experiment

The authors assemble 5,776 aligned tasks from five adapted public benchmarks and one synthetic benchmark, covering 41 languages, 17 language families, 16 scripts, and 809 language pairs. Eight instruction-tuned models appear across the full study, which reports 2,382,875 rollouts in 505 experimental cells.

Every model receives the same five-action alphabet: Search, Calc, Translate, Summarize, and Finish. The calls are parsed but never executed, and no tool result returns to the model. This is a controlled study of induced policy under one scaffold, not a test of grounded agents changing external state.

The Missing Baseline

Raw trace similarity is not enough. The paper prices five confounds: shorter traces appear more similar, empty traces can score as perfect matches, unrelated traces agree by chance, each model's reproducibility caps the possible comparison, and repeated runs in one language can disagree before any language is switched.

The corrected design generates two runs per condition. Same-language agreement measures how consistently the model reproduces itself; cross-language agreement compares paired language variants under matched decoding. The paper defines normalized retention as cross-language agreement divided by same-language agreement, excludes empty pairs, matches trace lengths in both directions, and bootstraps at the task level.

What the Corrected Measure Finds

Under greedy decoding, the four protocol-adherent frontier models retain a reported 71–73 percent of their own action-policy consistency across languages. All 24 model-benchmark cells show a positive raw gap, while model identity explains 5.7 percent of the variance in the normalized measure. The result is temperature-specific and descriptive of those systems, not a universal constant.

Below roughly ten billion parameters, the reported regularity breaks, but the paper treats that boundary cautiously: only two smaller models provide clean retention measurements. It also shows that rankings from an uncorrected gap can reverse when decoding temperature changes.

The English Pivot

In the adapted benchmarks, Translate is the most-used action for every compliant model, and the recorded reasoning is about 99 percent ASCII even for tested prompts in Devanagari, Tamil, or Odia. The authors remove or mandate Translate while holding other experimental factors fixed. After length matching, removing it reduces cross-language agreement most consistently in models that relied on it more.

The intervention supports an English-pivot mechanism within this scaffold, but not a universal deployment prescription. A prompt demanding reasoning in the task language achieved under one percent compliance in two tested models, so the intended contrast never formed. The paper properly reports a failed manipulation rather than claiming that reasoning language does not matter.

When the Parser Fails the Model

One of the sharpest findings concerns the harness. GPT-OSS-120B produced no parseable trace in 76.4 percent of reported rollouts because it often named an action in prose instead of matching one regular expression. Adding two worked examples raised measured accuracy twenty-sixfold while accuracy among readable outputs barely changed. Aya-Expanse-8B also exceeded the paper's proposed unranked threshold, with a 31.2 percent parse-failure rate.

The evaluation can therefore manufacture a language failure by confusing format adherence with capability. Parse failure, task correctness, and route retention need separate columns.

The Governance Reading

Cross-lingual assurance should follow actions, not only answers. If permissions, rate limits, approvals, or escalation rules were tested on an English trace, a deployment record should show whether other languages invoke the same controls. Where routes legitimately differ, each route needs its own safety case.

This also changes procurement. A vendor's multilingual accuracy score does not establish action-policy parity, and high route similarity does not establish correctness. Institutions need both measurements, paired by language and grounded in the actual harness that will act.

The Policy-Fork Receipt

A policy-fork receipt should record the canonical task, each language rendering and alignment key, model and checkpoint, scaffold, tool alphabet, prompt, seed, temperature, token and iteration limits, raw trace, parsed trace, parser version, empty-trace rule, same-language replicates, length matching, chance floor, correctness score, confidence interval, and reviewer decision.

For deployed tools, add permission checks, tool outputs, state changes, cost, latency, rollback, and the owner authorized to accept a language-specific route. The receipt turns “multilingual support” from a product label into a testable operating claim.

Limits

The paper does not dispatch tools, feed observations back into the loop, or study real side effects. Its frontier band contains four clean models from three vendors; its below-10B comparison contains two clean models; one trace-similarity metric is used; and gold correctness exists for four of six benchmarks. The authors also distinguish route invariance from accuracy and do not treat one as a proxy for the other.

These limits preserve the useful claim: within a fixed symbolic scaffold, changing language changes measured action policy after several confounds are controlled. Whether the magnitude survives grounded tools, different parsers, alternative trace metrics, or production workflows remains open.

Source Discipline

Primary sources were the arXiv abstract, metadata API, HTML, and complete version 1 PDF, plus the two linked Hugging Face dataset repositories. The paper says code, prompts, results, and rollout traces are released, but its linked GitHub repository was empty when reviewed on August 12, 2026. This page therefore does not claim independent access to or reproduction of that package.

Sources

Return to Blog