The Dispatch Weight Becomes the Marketplace Dial
The interesting move is not replacing dispatch optimization with reinforcement learning.
It is giving a learned policy a narrow, monitored dial in front of the optimizer, then measuring whether that dial improves a live three-sided marketplace.
The Paper
The paper is Multi-Agent Reinforcement Learning from Delayed Marketplace Feedback for Objective-Weight Adaptation in Three-Sided Dispatch, arXiv:2606.13604 [cs.AI, cs.LG, cs.MA]. arXiv lists it as submitted on June 11, 2026, with DOI 10.48550/arXiv.2606.13604 and a CC BY 4.0 license. The comments note acceptance at the ICML 2026 Workshop on Reinforcement Learning from World Feedback.
The arXiv metadata lists Haochen Wu, Yi Hou, and Shiguang Xie as authors. The work is a DoorDash production case study for adapting dispatch objective weights using delayed marketplace feedback.
The Interface
The key design choice is a constrained interface. The RL policy does not assign orders to couriers directly. DoorDash keeps the existing combinatorial assignment optimizer in charge of feasible courier-order assignments, constraints, and operational safeguards.
The learned outer policy acts at the store level. Every assignment cycle, each store observes local state and selects a discrete multiplier for the baseline delivery-speed objective weight. The action set is A = {0.8, 0.9, 1.0, 1.1, 1.2}. Lower weights make batching-compatible assignments more attractive. Higher weights emphasize faster delivery completion. The neutral action 1.0 recovers the static production baseline.
That makes the policy powerful enough to alter the speed-efficiency tradeoff but narrow enough to preserve production feasibility. In governance terms, the learned system changes the optimizer's preference weights, not the underlying dispatch law.
Learning From Delay
The paper formulates objective-weight adaptation as a decentralized multi-agent Markov decision process with centralized offline training. Stores execute the policy independently, but the value function is trained from pooled logged marketplace data and delayed regional rewards.
The state vector is deliberately small: outstanding delivery count, localized supply pressure, and median courier wait time. The reward joins dispatch decisions with delayed fulfillment outcomes. ASAP captures customer-facing delivery duration, CAT captures courier active time, and XCAT captures excess courier active time beyond direct route. The reward aggregates regionally to reflect network effects across nearby stores and couriers.
The training recipe is conservative offline RL. The system uses Double Q-learning targets and a Conservative Q-Learning regularizer to reduce unsupported-action value overestimation. The policy network is a two-layer multilayer perceptron with hidden dimension 16, trained offline from logged transition tuples before production deployment.
Switchback Evidence
The production evidence comes from a global switchback experiment against the DoorDash baseline. The randomization units are about 4,000 geographic regions. At each two-hour interval, regions are assigned to treatment or control over a two-week period. Control uses the static objective weight. Treatment uses the OWA-RL policy trained with alpha = 0.9.
The experiment fixes the optimizer, constraints, and serving infrastructure. Only objective-weight selection changes. The authors estimate treatment effects with CUPED variance reduction and clustered p-values at the region-hour switchback bucket. They monitor customer experience, cancellations, and carryover effects as guardrails.
Operational Impact
The reported all-day impact is small but directionally coherent. Compared with baseline, OWA-RL reduces CAT with ATE -1.261 seconds (p = 0.019), reduces courier wait time with ATE -0.856 seconds (p = 0.004), and increases batching by 0.495 percentage points (p < 0.001). ASAP changes by +0.972 seconds (p = 0.264), and 20-minute lateness changes by -0.012 percentage points (p = 0.237), both statistically unchanged in the authors' framing.
Dinner-hour results are similar: CAT -1.289 seconds (p = 0.042), courier wait time -1.030 seconds (p = 0.041), batching +0.600 percentage points (p = 0.010), ASAP +0.869 seconds (p = 0.633), and 20-minute lateness -0.037 percentage points (p = 0.040). The paper's production claim is therefore not dramatic automation. It is measured retuning of a live marketplace's speed-efficiency dial.
The paper also reports that production action distributions vary with backlog, supply pressure, and courier wait time, especially during a San Francisco-Bay Area Friday dinner peak. That matters because the policy is not just a new global static weight hidden behind an RL label.
Marketplace Receipt
A marketplace RL receipt should name the action interface, baseline optimizer, state features, reward definition, delayed attribution window, traffic eligibility, region randomization, switchback interval, CUPED model, guardrail metrics, clustered p-value rule, rollback criteria, distribution-shift monitors, and labor-impact review.
It should also distinguish optimization safety from social safety. A narrow objective-weight interface can preserve dispatch feasibility and serving reliability. It does not by itself settle whether customers, merchants, and couriers share gains fairly, whether regional heterogeneity hides harms, or whether a better batching rate means a better marketplace.
This belongs beside Reinforcement Learning, AI Governance, Recommender Systems, The Agent Environment Becomes the Discovery Lab, and The Reliability Scorecard Becomes the Agent Gate.
Limits
The evidence is strongest as a production integration pattern. It shows how offline RL can modulate an existing optimizer using delayed world feedback while keeping a constrained action space, conservative training, and online experimentation in the loop.
The generalization limits are substantial. The data, optimizer, marketplace, and guardrail system are DoorDash-specific. The paper does not prove that the same action interface, reward weighting, or treatment effect will transfer to other logistics markets. It also acknowledges noisy credit assignment from delayed regional rewards and dependence on marketplace stability after deployment.
The practical lesson is architectural. If RL is going into a high-throughput marketplace, give it a small dial first, prove the dial moves outcomes under controlled randomization, and keep the receipt close enough that operations, policy, and labor review can read it.
Sources
- Haochen Wu, Yi Hou, and Shiguang Xie, Multi-Agent Reinforcement Learning from Delayed Marketplace Feedback for Objective-Weight Adaptation in Three-Sided Dispatch, arXiv:2606.13604 [cs.AI, cs.LG, cs.MA], submitted June 11, 2026.
- arXiv HTML: Dispatch objective-weight adaptation HTML, reviewed for abstract, formulation, experiment setup, results, appendix parameters, and impact statement.
- arXiv PDF: Dispatch objective-weight adaptation PDF, reviewed for action space, state features, reward design, training recipe, switchback metrics, and limitations.
- arXiv license metadata: CC BY 4.0, linked from the arXiv abstract page as the article license.