01The problem
Editors and small content teams need to find moments in their footage fast, but a similarity score is not proof that something happened. A video agent also needs to choose the right tool for a request and must never render or export anything without a person approving it.
02The approach
- Index uploads in bounded background jobs with FFmpeg and a frozen CLIP retriever, and return timestamped, playable frames so every match can be checked against the source. An optional filter keeps only sharper frames.
- Post-train SmolLM2-135M-Instruct with a rank-8 LoRA as a five-way tool router (search, library, count, export, inspect): three epochs of completion-only SFT, then 60 steps of finite-action GRPO with group size six, a clipped objective, an exact-route reward and a KL penalty to the SFT reference.
- Generate 560 synthetic requests split 480/40/40 by template family, pick the released stage on validation only, then score the held-out test once, with CPU latency and an explicitly assumed compute price per correct answer.
- Validate every typed tool call, cap call budgets, recover from tool errors and require explicit visitor approval before any export renders, whatever the model chooses. Visitors get private session workspaces whose uploads expire after 24 hours, plus DETR detection and a GrabCut mask for inspecting a frame.
03The evidence
- On the held-out test, routing accuracy went from 8/40 (base model) to 40/40 for both SFT and GRPO, with mean CPU routing latency of about 1.5 to 1.7 seconds.
- On validation, GRPO regressed slightly against SFT (37/40 vs 39/40, both from export requests), so the release gate shipped SFT. The report publishes both checkpoints and the RL result anyway.
- CLIP retrieval found the right moment as the top result for 5 of 6 annotated queries on a public Big Buck Bunny excerpt; the miss (“a dark hole in a tree”) is recorded.
- The workspace is live on a CPU server behind TLS. Browser tests run the released model and CLIP on real video and check the exported MP4 with FFprobe.
04Scope & status
The routing benchmark is small and fully synthetic, so it says nothing about real-user reliability or general reasoning. The router is a small language-model adapter, not a trained vision-language model, and its confidence scores are uncalibrated. Retrieval checks cover six illustrative queries on one film without independent review. No user or customer counts are claimed.
Public source is available for independent inspection.