Skip to content
NovaVMS Manual

How visual event search works

Every query typed into the Events page runs ONE filtered HNSW pass over per-frame visual embeddings and ranks events by similarity to your intent. A natural-language parser sits in front of the engine, translating “white truck at the loading dock yesterday afternoon” into the structured fields the engine consumes — chip filters, camera or site references, a time window, an event category, and the semantic terms that drive the visual rank.

The Phase 4 rebuild (May 2026) replaced the older multi-lane RRF model with a single filtered HNSW path because the new MobileCLIP2-B embedding space is precise enough that fusing additional lanes added complexity without measurable recall. The retired three-lane design (Tag + Text + Vector with Reciprocal Rank Fusion) is documented as historical context in docs/arch-search.md §§1-9; the canonical implementation is in docs/arch-search.md §10.

Why this works this way

The Events page was originally split from a Search page, and the split confused users. R6 records the merge: one page, one input, with the search box treated as an optional filter on top of the chronological feed. When the input is empty you see a time-ordered stream. When you type, the engine re-sorts the feed by visual similarity to your query.

The visual ranking pipeline is:

  1. Parser. Gemini 3.1 Flash-Lite (default gemini-3.1-flash-lite, temperature 0.1, see cloud/config/search_config.json) translates the prose into structured filters and a semantic query. A rule-based parser is the fallback when Gemini is unavailable or the request opts out.
  2. Query embedding. The semantic query is encoded by MobileCLIP2-B via the local AI Runtime sidecar (cloud/cmd/server/main.go:2206). The 512-dim, L2-normalized vector shares the embedding space with frame_embeddings.embedding, which is the only reason the cosine search returns meaningful results.
  3. Filtered HNSW. ONE pgvector HNSW query against frame_embeddings with the chip filters compiled into the WHERE clause (camera, site, date range, event category, source, AI status, plus EXISTS event_objects for object chips and EXISTS event_faces for face profile chips). The query runs inside a read-only tx with SET LOCAL hnsw.iterative_scan = strict_order so the planner keeps exploring candidates if early ones get filtered out by the org-scoped predicate.
  4. Aggregation. Best-frame per event via MIN(distance); the per-frame index returns the best-matching frame and the engine groups by event_id.
  5. Go-side rerank. A bulk lookup (one query per Search() call) populates three boolean axes per result — object_match, canonical_tag_match, face_profile_match — by intersecting the rendered hits with the explicit filter values. The current production code applies a single +0.15 rerank bonus when either an object filter OR a face filter matched; tag-only matches do not add a bonus (the boost is reserved for explicit class/face filters per engine.go:300-321). The v5 search-ranking PRD §7 prescribes a granular +0.05 / +0.05 / +0.10 per-axis bonus — that’s a known PRD-vs-code divergence tracked in docs/audits/2026-05-21-manual-alignment/gaps-by-phase/phase-2.md GAP-2-12; the operator-visible behaviour today is binary.
  6. Fallback. When the planner estimates fewer than ~100 rows would match the EXISTS predicates (typical for very selective object/face filters), an ExplainGate (cloud/internal/search/explain_gate.go) drops the EXISTS predicates from the HNSW query, runs it with 4× oversample, and post-filters in Go via EventObjectLookup. This is the only fallback in EngineV2; the pre-Phase-4 “tag-strict relaxation” was retired alongside the multi-lane model.

The natural-language parser is what makes “white truck yesterday afternoon at the loading dock” work. A small LLM call (gemini-3.1-flash-lite, temperature 0.1) parses the user’s prose into the structured fields above. The parsed query then feeds the engine. If the parser fails or the LLM is offline, the rule parser still drives the visual lane on the raw query string — search degrades, it does not break.

The Ask AI cohort chat

The Ask AI sheet on the Events page (event list, NOT event detail — frontend/src/pages/EventsPage.tsx:880-889) is a separate surface on top of the same data. It POSTs the visible cohort’s event IDs to POST /api/v1/events/chat and the LLM returns:

  • a freeform answer,
  • a confidence score,
  • a sources list of the most-relevant event IDs,
  • a matching_event_ids array — the canonical set of events the LLM claims match the question. For count questions the backend (cloud/internal/search/count_grounding.go, post-FE2E-QA commit fb10d16b) rewrites any stated count in answer to len(matching_event_ids) so the user-visible count is always grounded in the enumeration.

Per-event chat (the POST /api/v1/events/{id}/chat endpoint described in older docs) was retired in L12; only the cohort endpoint remains.

What this means for you

  • An empty search box shows the chronological feed; typing turns it into a ranked result list.
  • Chip filters (object class, recognised face, canonical tag, camera, site, time, event category) are the explicit predicates the rerank bonus reads from. Without a chip filter, the rank is pure visual similarity.
  • Prose queries like “someone loitering near the back door last night” parse to a time window plus a semantic query; v5 cannot detect loitering directly (it’s on the unsupported-keyword list, see cloud/internal/events/object_categories.go:152-158), so the parser emits a warning and the query falls back to pure visual similarity.
  • Multilingual queries work through Gemini, but the launch gate is bilingual English + one CJK pair top-5 recall ≥ 80% (v5 search-ranking PRD §10). Other languages are not benchmarked.
  • The score panel on each hit (open it via the score chip) breaks the match into base_score plus three rerank axes — Object / Tag / Face. The final similarity_score is base_score + 0.15 when Object OR Face matched, otherwise equal to base_score.

Where to go next