StreamVibe

StreamVibe is a research project on proactive streaming video agents. Rather than waiting for a request that names a specific event, the agent watches a video stream, infers the user's goal, tracks task progress, and decides when and how to offer assistance. It also handles requests that are revised, extended or interrupted mid-task.

The project has three parts: StreamVibe-Bench, a benchmark; StreamVibe-VL, a streaming vision-language model; and StreamVibe-Harness, which tracks requests, task progress and past replies and retrieves relevant history. The benchmark covers physical and digital task videos with 600 explicit, 2,000 implicit and 1,500 dynamic samples, scored on timing (P/R/F1/FPR) and answer quality. Samples are proposed by a Gemini model and kept only if four reviewers approve.

The model is trained in two stages: adaptive streaming supervised fine-tuning (70K + 768K samples) to balance silence and response, followed by localized trajectory reinforcement learning (GRPO, 15K tasks) rewarding timing, content and consistency. The authors compare it against 11 proprietary, offline and streaming models. A YouTube creator reported that code and models were promised within seven days; the provided pages do not show a release, so availability of code and weights is unconfirmed.