Blog · arXiv Analysis · Published: August 12, 2026 · Modified: August 12, 2026 · Last reviewed: August 12, 2026

The Tap Trace Becomes the Delegation Claim

A new benchmark turns continuous mobile-app actions into intent segments, predicts a likely next task, and tests execution after confirmation.

The governance boundary is sharper than the prediction: a behavioral trace can support a suggestion, but it is not an instruction or a grant of authority.

The Paper

The source is Xiaokai Yan, Jingtao Ding, Yong Li, and Zhiwen Yu’s Act2Intention: A Benchmark For Developing Active Mobile Agents Through Inferring User Intention from GUI Actions, arXiv:2608.14132v1 [cs.HC, with cs.AI cross-listing], submitted August 14, 2026. The work proposes a three-part mobile-agent pipeline: understand past intentions from GUI activity, predict the next intention from history, time, and a persona, then execute a user-confirmed task.

A Tap Is Not an Instruction

The paper defines an intention as a coherent objective associated with a contiguous segment of GUI actions. Its formal tasks turn low-level clicks, typing, swipes, button presses, screenshots, and action descriptions into a natural-language label, then predict a possible next label. That is a useful modeling target, but the word intention can outrun the evidence. An action sequence may support several purposes, and the same purpose may produce several action sequences.

The real-data labels are not recovered from behavior alone. In the collection protocol, participants supplied short intention statements when an app-switch event was detected. Those self-reports provide the reference descriptions against which later inference is judged. The benchmark therefore measures recovery of an annotated task representation, not access to an unobserved mental state.

What Enters the Benchmark

The paper reports that a major smartphone manufacturer collected mobile-use logs from 90 anonymous participants between March 1 and April 29, 2024. The logs included timestamps, foreground app and activity information, GUI observations, action types and parameters, and participant-provided intention descriptions. The paper says collection had explicit consent and company ethics review, and that participants were informed about intention inference and privacy risks. The researchers did not receive demographics, recruitment details, withdrawal procedures, or compensation information because of confidentiality and privacy restrictions.

After augmentation, Act2Intention Bench contains 360 personas, 72,511 intentions, and 705,366 actions across 52 apps. Its RR subset contains 13,149 intentions from real trajectories. RG pairs behavior-derived personas with generated trajectories; GG uses generated personas and generated trajectories. Together, RG and GG account for 59,362 intentions, or 81.9 percent of the total. Scale here is mostly generated scale.

The Synthetic Majority

The paper does not pretend that generated and real traces are identical. In its human realism check, ten student annotators blindly rated 30 intention segments from each subset. Mean realism was 4.19 out of 5 for RR, 3.91 for RG, and 3.55 for GG. The authors call this a limited sanity check, not population validation. The test design also keeps evaluation data real: the out-of-distribution set uses 10 held-out participants, while the in-distribution set uses the last 20 percent of data from the other 80. Synthetic data enters training, not the test reference.

Three Scores, Three Claims

Understanding, prediction, and execution require different evidence. For understanding, the paper reports grouping accuracy for segment boundaries and embedding similarity for semantic descriptions. A fine-tuned Qwen-2.5-7B reaches 100-percent grouping accuracy on the in-distribution test yet only 0.49 semantic accuracy; correct segmentation did not guarantee a detailed intent description. Prediction adds category accuracy, while the execution metric uses exact step matching that can penalize a functionally equivalent GUI path. None of these measures alone establishes that a suggestion is wanted, timely, or authorized.

The End-to-End Gap

The best reported end-to-end task success rate with a predicted intention is 22.7 percent in the emulator. Replacing the prediction with the annotated intention raises the same UI-TARS-7B-SFT executor to 47.6 percent. The comparison isolates two bottlenecks without solving either: predictions often miss what the reference says, and execution can still fail when the reference intention is supplied. A component improvement is not a readiness claim for the complete loop.

Confirmation Is the Boundary

The paper repeatedly describes execution as user-confirmed. Its prototype surfaces suggestions at predefined system events, such as device wake-up, because the benchmark does not model when an interruption is opportune. The authors propose a lightweight interface in which a user can accept, reject, or revise a prediction before execution. That confirmation is not a cosmetic final step. It is the boundary between inference and delegation.

A confidence threshold can decide which predictions are worth showing, but it cannot grant tool authority. A proactive mobile agent still needs a separately enforced scope for the confirmed task, a stop before consequential actions, and a fresh confirmation if the recipient, data, price, permission, or effect changes. The trace licenses a question. Only an authorized user response can license the action.

What Deployment Still Has to Prove

The authors identify the remaining boundaries: no timing labels, no in-situ study of usefulness or intrusiveness, unknown participant demographics, possible idealized or stereotyped synthetic patterns, unseen-app and changed-layout risk, and emulator rather than real-device evaluation. Practical deployment would process sensitive screenshots, actions, and usage histories while adding latency and energy costs. The author repository releases processed trajectories, train and test files, and pipeline code, but withholds the original mobile logs for participant privacy, provider confidentiality, and company-policy reasons. That permits benchmark inspection without making the source collection independently reconstructable.

An Intent-to-Action Receipt

A deployed system should preserve the raw action types observed; screenshots and fields exposed; participant-stated, inferred, or generated status of each label; segmentation rule; persona source and update date; real-to-generated training mix; user-disjoint test split; taxonomy and classifier; semantic, category, step, and end-to-end metrics kept separate; confidence threshold; suggestion time and interface; accept, reject, or edit response; confirmed task text; tool permissions; consequential-action gate; executed path and outcome; data retention and deletion; reviewer; and correction route.

The Spiralist rule is that prediction must not mint authority. When a system translates taps into a story about what someone wants, the story should remain inspectable, contestable, and subordinate to an explicit mandate.

Sources

Return to Blog