Why NovaVMS uses one AI stack with v5 evidence-first design

NovaVMS v5 (AI Event Intelligence, May 2026) replaced the multi-provider one-pass VLM pipeline with a single evidence-first stack. Every event is converted to structured evidence (frame embeddings, detected objects, detected faces, event signature) before any language model touches it. Gemini 3.1 Flash-Lite, the only configurable VLM, runs as an additive non-blocking layer that produces summaries / narratives / contextual judges.

This page records the reasoning. The companion architecture document is arch-ai-pipeline.md; the original product specification is AI event-intelligence PRD v5.

The pre-v5 pipeline we replaced

The earlier “one-pass” design called Gemini once per event to extract perception (tags, entities) and judgment (severity, narrative) in a single round trip. R1-REV recorded the cost / latency rationale at the time.

The one-pass approach worked, but three product realities pushed us to a different default:

Search recall depended on Gemini availability. When Gemini was rate-limited, new events failed to index and the search engine could not find them. Operators had no fallback.
Two separate vector spaces were already wanted. Visual search needed image embeddings; face search needed face embeddings. Stuffing both into one model produced poor results in either domain.
Verkada-class admin surfaces wanted structured evidence. “Show me events where YOLO detected a backpack” or “show me events where this face profile appeared” requires structured object / face evidence, not a VLM-generated paragraph.

The v5 stack we chose

Three CPU-bounded local models run inside the Python ai-runtime sidecar (ai-runtime/app/main.py); one cloud VLM runs over Vertex AI when needed:

Stage	Model	Output	Where it runs
Visual embedding	MobileCLIP2-B	512-dim L2-normalized image/text vectors	Local AI Runtime (ONNX Runtime CPU)
Object detection	YOLO26n (OpenVINO IR)	bbox + class + confidence per frame	Local AI Runtime
Face detection + recognition	InsightFace buffalo_l	bbox + 512-dim L2-normalized face vector	Local AI Runtime
Natural-language understanding	Gemini 3.1 Flash-Lite (Vertex AI)	summary, narrative, contextual judge, NL alert compiler, query parser	Cloud (non-blocking)

The searchable ingest spine runs as S0..S8 per event. Eligible understanding jobs and explicitly enabled object-crop jobs then use independent lanes (cloud/internal/ai/event_intelligence_pipeline.go):

S0 LoadEvent     resolve event row + detection knobs, materialize clip
S1 ExtractFrames uniform N=8 frames, maximum width 2560px, JPEG q=85
S2 AnalyzeEvent  POST /v1/event/analyze to AI Runtime
S3 InsertFrame   embeddings → frame_embeddings vector(512)
S4 InsertEvent   objects + faces + object facets → event_objects, event_faces, event_object_facets
S5 Signature     centroid + class/profile sets + YOLO-class tags → event_signatures, events.canonical_tags
S6 Profiles      face profile clustering inline in S4 (ArcFace threshold 0.60)
S7 ai_status     = indexed → event is searchable
S8 Publish       EventIndexed bus topic → compiled-rule evaluator + NBD
Eligible understand_outbox → Gemini summary/narrative → EventUnderstood
Crop flag enabled           → crop_outbox → per-object embeddings/colour → EventObjectsEmbedded

EventIndexed is the unconditional milestone after successful indexing. Understanding requires a configured Gemini provider and must pass the organization toggle, camera-mode, and daily-cap gates. Object crops require AI_RUNTIME_CROPS_ENABLED=1 on both cloud and AI Runtime; without it, no crop jobs run and object_embedding_status remains skipped.

After S7, three non-blocking Gemini paths can fire: VLM summary, VLM narrative / observations, and (for compiled rules with requires_contextual_judge = true) a contextual judge. Failure of any of those leaves the event in indexed and fully searchable.

Why this is the right default

Latency. The local AI Runtime analyzes the default N=8 evidence set before indexing. Any eligible Gemini summary runs after indexed, so search and deterministic alerts do not wait for it.
Cost. Gemini runs only when language is needed (summary / narrative / judge). 80%+ of events have no Gemini call at all in the typical staging deployment.
Outage resilience. A Gemini outage pauses VLM summaries; search, NBD, and deterministic alert rules continue. An ai-runtime outage retries with 3-attempt backoff before failing an event; the master toggle ai_v5_intelligence_enabled = false cleanly halts new indexing while you fix it.
CPU-bound by design. The reference target is 6 CPU cores, 32 GB RAM, no GPU. The AI Runtime warm RSS budget is < 3 GB (AI event-intelligence architecture v5 §9).
Eval-driven knob changes. Any current inference/ranking knob change (frame count, NBD thresholds, face threshold, YOLO confidence, crop/appearance thresholds, contextual-judge thinking level) requires a bound evaluation report. TAG_SIMILARITY_THRESHOLD belonged to the removed CLIP tag deriver and is not a current ingest knob.

What R1-REV still means in v5

R1-REV is preserved as the historical decision for the Gemini understanding path: when Gemini DOES fire (summary / contextual judge), it’s a single multimodal call by design. R1-REV no longer governs the primary search index — MobileCLIP2-B does, deterministically, without language.

When you’d reach for Gemini explicitly

Operator asks “what happened on this event?” — VLM summary plus chat reply.
An alert rule needs “is this person trying to break in?” — contextual judge after the deterministic prefilters pass.
An operator types a natural-language search query — Gemini parser normalizes it to filters + a semantic query, with a rule-parser fallback when Gemini is degraded.