The Gaze Trace Becomes the Caption Index

The Paper

The paper is VEGAS: Human-Aligned Video Caption Evaluation via Gaze, arXiv:2607.08489. The arXiv record lists Shenghui Chen, Po-han Li, Ximeng Sun, Shijia Yang, Emad Barsoum, Zicheng Liu, Sandeep Chinchali, and Ufuk Topcu as authors, with submission on July 9, 2026. The record categorizes it under Computer Vision and Pattern Recognition, with Artificial Intelligence and Human-Computer Interaction as additional subjects.

The arXiv HTML lists University of Texas at Austin affiliations for Chen, Li, Chinchali, and Topcu, and AMD affiliations for Sun, Yang, Barsoum, and Liu. The paper proposes VEGAS, short for Video caption Evaluation via GAze Score, as a way to evaluate candidate captions using synchronized gaze without retraining the underlying vision-language model.

The Caption Index

A video caption is rarely just a sentence. It becomes an index entry for search, memory, recommendation, retrieval, accessibility, education, surveillance review, or a later agent's summary. When a system captions the same clip differently for two viewers, it is no longer describing only the public scene. It is also selecting which part of that scene mattered for a person.

That is the interesting pressure in VEGAS. The authors argue that generic vision-language models often describe broadly visible content while missing the object, action, or entity that a particular viewer attended to. Gaze is used as a measurable but imperfect proxy for attention. It is not a mind-reading channel and not proof of intent. It is a behavioral trace that can help a caption selector prefer one plausible description over another.

This sits beside multimodal AI, rationale interfaces, dashcam VQA evidence, and driver-camera attention judgments. The shared issue is evidentiary drift. A perception score can become a claim about what happened; a gaze-conditioned caption can become a claim about what the user cared about.

The Method

VEGAS is training-free at evaluation time. The method samples candidate captions from a pretrained vision-language model and then uses a cross-modal information-theoretic score to prefer captions aligned with the gaze-conditioned visual signal. The paper frames this as an approximation, because the true joint distribution between gaze and language is not directly available.

The experimental material spans two settings: egocentric activity videos from Aria Everyday Activities and instructional slide material from SlideVQA. In both, the authors pair visual content with synchronized gaze and human-written reference annotations. The important design choice is that gaze is not used to fine-tune a new model. It is used to choose among generated captions at test time through rejection sampling.

That makes VEGAS operationally plausible but also audit-heavy. A future product could call the final caption "personalized" while hiding the candidate pool, gaze processing, model version, selector score, and rejected captions. The published paper is valuable because it exposes those moving parts as measurable pieces rather than as a single opaque personalization switch.

Evidence

The main result is uneven in a useful way. On Aria Everyday Activities, VEGAS shifts mean SBERT similarity by +0.0856, reported as +13.53 percent. On SlideVQA, the corresponding shift is +0.0256, reported as +3.88 percent, and the paper treats the slide result as weaker because many generic captions can still be plausible for slide content.

The retrieval test gives a second view. On AEA, the paper reports mean average precision gains of +1.14, +2.48, and +2.46 at ranks 1, 5, and 10 over a random VLM-caption baseline. Human pairwise captions remain materially better, which matters: gaze-aware selection improves the machine index, but it does not close the human annotation gap.

The authors also test whether VEGAS is merely choosing shorter or more generic captions, and report near-zero correlation in that diagnostic. Their gaze-corruption checks are more important for governance. Mismatched random gaze worsens performance, and center-biased gaze is weaker, which supports the claim that the method is using viewer-specific gaze information rather than only generic scene priors.

Limits

The paper is explicit about limits. VEGAS inherits hallucinations, perceptual mistakes, and likelihood-calibration problems from the underlying vision-language model. Gaze shows where a viewer looked, but not necessarily what the viewer understood, wanted, remembered, feared, or intended to do. The method works best when attention is tied to concrete objects, actions, and entities.

The privacy boundary is just as important as the accuracy boundary. Explicit gaze can be available in smart glasses or other gaze-tracking contexts, while the paper's future-work discussion points toward saliency, interaction traces, and other implicit attention proxies as more realistic signals for web or video platforms. That move may be useful, but it also makes the caption index more like a behavioral dossier.

So the lesson is not that gaze makes captions truthful. It is that personalized captions need source disclosure. A caption generated through an attention trace should carry a receipt, because downstream users may otherwise treat the selected sentence as neutral scene evidence.

The Receipt

A gaze-caption receipt should name the source video or slide deck, capture device, gaze sampling and calibration method, consent status, face or video storage policy, preprocessing steps, vision-language model, prompt, decoding settings, candidate-caption pool, selector score, rejected alternatives, reference annotations, retrieval metric, hallucination checks, privacy retention rule, and human-review path.

For AI audit trails and agent observability, the point is concrete. If an assistant uses a gaze-conditioned caption to retrieve a memory, answer a question, train a tutor, assess attention, or summarize a workplace recording, the caption should not arrive alone. It should arrive with the trace that made it personal.

Sources

• Shenghui Chen, Po-han Li, Ximeng Sun, Shijia Yang, Emad Barsoum, Zicheng Liu, Sandeep Chinchali, and Ufuk Topcu, VEGAS: Human-Aligned Video Caption Evaluation via Gaze, arXiv:2607.08489 [cs.CV], submitted July 9, 2026.
• arXiv experimental HTML for VEGAS: Human-Aligned Video Caption Evaluation via Gaze, checked for affiliations, method, datasets, numerical results, ablations, limitations, and future-work framing.
• arXiv API record for arXiv:2607.08489, checked for title, authors, submission timestamp, abstract, and subject categories.

Return to Blog