THE ESSENTIALS IN 30 SECONDS

  • StreamVibe-VL checks the video stream every second and chooses between staying silent and responding, with a reinforcement learning stage meant to curb unnecessary interruptions.
  • The accompanying benchmark contains about 4,100 evaluation samples drawn from 183.4 hours of video covering everyday tasks and computer use.
  • The model leads open-source baselines in the authors' tests, but Google's proprietary Gemini 3.8 Flash scores higher on several metrics, and performance falls on long videos.

Most video assistants wait for a question. A preprint posted on arXiv by researchers from Google, UC Santa Barbara, UC Santa Cruz, Stanford, the University of Oxford, Emory and Columbia describes StreamVibe, a system meant to watch someone at work and decide by itself when a tip would be useful, for example while cooking, assembling furniture or editing an image in software such as GIMP.

Three pieces: benchmark, model and memory

The work has three parts. StreamVibe-Bench holds about 4,100 evaluation samples split across explicit requests, vague or implicit goals and intents that change over time. According to the project page, the samples come from 183.4 hours of video, average 8.3 minutes and contain more than 20,000 annotated moments where help would be appropriate. Candidate moments were proposed by a Gemini model, and a sample was kept only if all four human reviewers approved it.

StreamVibe-VL is the model. At each one-second step it sees the video so far and its own earlier replies, then outputs either silence or a response. It was first fine-tuned on 70,000 task samples plus a larger pool of general streaming-video data, then trained with reinforcement learning (GRPO) on 15,000 tasks; the reward penalises excess responses, which The Decoder describes as teaching the model to interrupt less. The project page lists Qwen3.5-9B as the base model in its comparisons. A third component, StreamVibe-Harness, keeps track of pending requests, completed steps and past replies.

Results and limits

The authors report a 10.7% improvement in response timing and answer quality over baselines, plus a further 12.3% in proactive interaction from the harness. They say StreamVibe-VL gives the best open-source results at every intent level, though not on every single metric. Proprietary models remain ahead in places: in the paper's tables, Gemini 3.8 Flash reaches an implicit-intent F1 score of 81.00 against 72.01 for StreamVibe-VL.

The Decoder's reading of the paper points to weaker results when systems must process a full video and build on their own earlier replies, as errors compound. It reports that models rarely pick the right moment in videos over five minutes. The authors themselves say implicit and changing intents remain hard for all tested models and that real situations are likely more complex than their setup.

Code and model weights are not yet available. The project page marks them as coming soon, pending internal approval, so the results cannot yet be checked independently.