The Session Scheduler Becomes the Agent Fairness Layer
SMetric treats agent-serving traffic as a session problem. The governance question is whether the scheduler can balance throughput, cache reuse, latency, and session fairness before an agent platform blames its delays on raw demand.
The Paper
The paper is SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling, arXiv:2607.08565 [cs.DC, cs.AI]. The arXiv API lists Jiahao Wang, Kaizhan Lin, Kaixi Zhang, Jinbo Han, Xingda Wei, Sijie Shen, Chenguang Fang, Wenyuan Yu, Rong Chen, and Haibo Chen as authors, with version 1 submitted on July 9, 2026. The PDF is 17 pages. The arXiv HTML lists Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University, and Alibaba Group affiliations.
This belongs near AI agents, agent harnesses, recursive research agents, agent registries, and AI audit trails. The fresh angle is infrastructure: agent behavior is shaped by how the serving cluster routes each session's requests.
The Session
SMetric starts from a difference between human chat and agentic serving. A human watches tokens stream in and experiences time-to-first-token and per-token delay directly. An agent usually waits for a complete response before it can call a tool, edit a file, or choose the next step. That shifts the objective toward cluster tokens per second while keeping latency constraints alive.
The second difference is cache reuse. The paper studies two BAILIAN production traces from May 29, 2026, both collected during 10:00-12:00 peak usage hours from large-scale clusters dedicated to a commercial coding-agent product. In those traces, reusable KV-cache tokens exceed 80 percent of request tokens, compared with 54-62 percent in chat workloads cited by the paper. More than 65 percent of reuse comes from follow-up requests reusing earlier turns of the same session, and roughly 90 percent of reuse recurs within about 100 seconds.
The Method
The failure mode is simple. Cache-aware schedulers route requests to the instance that already holds the relevant KV cache. In an agent workload, that can pin many sessions to a few instances because first-turn requests from the same agent framework share system prompts, and follow-up turns tend to return to the instance that served the first turn. A scheduler then has high cache reuse but bad load balance: a few instances are overloaded while others sit underused.
SMetric's policy is deliberately small. Route the first request of each session for load balance. Route follow-up requests in a cache-aware way, so they usually stay near their local KV cache. Add guards for overloaded instances and likely evicted sessions. The paper emphasizes that the routing hint is the session turn, which can be inferred from standard stateless LLM API request histories, so the router does not need to maintain a per-session table that grows, ages, and must be garbage-collected.
The paper reports an implementation on vLLM and LMCache, with a global KV-cache tier based on Mooncake and LMCache. It also says SMetric and traces will be open-sourced upon publication, so the current claim is a paper-level system report rather than an independently reproducible public artifact.
Evidence
The evaluation testbed has four GPU servers, each with eight NVIDIA H20 GPUs, 160 CPU cores, and 1280 GB of host DRAM, for up to 32 single-GPU instances. Servers are connected by 200 Gbps RDMA NICs. The evaluated models are Qwen3-Coder-30B-A3B and Qwen3-235B-A22B-FP8. The workload replay uses Trace 1, sampled down to fit the testbed.
Under prefill-decode colocation with a global store, the paper reports that SMetric serves 10-16 percent more TPS than the best-performing baseline across global-tier provisionings. In the 32-instance Qwen3-Coder-30B-A3B setting, SMetric preserves local reuse while improving load balance: with a fully provisioned global store it reports a local-tier hit ratio comparable to BAILIAN, 64 percent versus 65 percent, and a mean max-to-mean load ratio of 2.6x, lower than BAILIAN's 3.0x and LMetric's 3.2x. Median time-to-first-token is 1.1 seconds, 17 percent lower than BAILIAN in the cited setup.
For the larger Qwen3-235B-A22B-FP8 model, the paper reports a peak of 516 tokens per second within SLO, 8 percent higher than BAILIAN's peak over the sweep and 20 percent higher than LMetric at the same operating point. Under prefill-decode disaggregation, SMetric reports 2-34 percent higher prefill TPS than the best-performing baseline across global-tier settings, and a median TTFT of 1.1 seconds, 37 percent lower than BAILIAN.
Limits
This is not a universal scheduler certificate. The traces come from BAILIAN and a commercial coding-agent product, under a privacy policy that limits collection. The replay uses Trace 1, sampled to fit the testbed, and forces evaluated models to generate response lengths matching the recorded trace so cache reuse remains consistent. The authors also scope out inter-cluster scheduling.
The governance value is still real. SMetric makes visible that a platform can have enough GPUs and still deliver unfair or unstable service if routing turns session locality into instance overload. But the evidence has to travel with the workload: model size, cache tier, session mix, global-store capacity, and timeout policy all change the result.
The Receipt
An agent-serving receipt should record the model, serving architecture, number of instances, GPU type, KV-cache tiering, global-store capacity, RDMA assumptions, workload source, privacy filtering, session definition, turn-inference rule, first-turn routing policy, follow-up routing policy, overload threshold, eviction guard, baselines, request-rate sweep, TTFT and TPOT SLOs, TPS within SLO, local and global hit ratios, per-instance load imbalance, timeout behavior, sampled-trace policy, and artifact availability.
The audit question is not "how many accelerators are in the cluster?" It is "which sessions got pinned, which sessions waited, which cache tier was exhausted, and what scheduling rule made that distribution legitimate?"
Sources
- Jiahao Wang, Kaizhan Lin, Kaixi Zhang, Jinbo Han, Xingda Wei, Sijie Shen, Chenguang Fang, Wenyuan Yu, Rong Chen, and Haibo Chen, SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling, arXiv:2607.08565 [cs.DC, cs.AI], submitted July 9, 2026.
- arXiv experimental HTML for arXiv:2607.08565v1, checked for abstract, affiliations, scheduling design, implementation notes, result descriptions, and open-source-upon-publication statement.
- arXiv API record for arXiv:2607.08565, checked for exact title, authors, subject category, submission timestamp, and version metadata.
- arXiv PDF for arXiv:2607.08565, checked for page count, production-trace description, cache-reuse findings, evaluation testbed, model settings, TPS and latency results, and scope limits.