Skip to content
NovaVMS Manual

Event has no AI evidence

You opened an event on the Events page and the AI evidence panel is empty — no frames, no canonical tags, no detected objects, no detected faces. Other recent events from the same camera have complete evidence. This page helps the admin who owns the v5 AI pipeline triage what went wrong.

Symptom

On the event detail page you see one of:

  • The AI evidence panel shows zero frames AND zero canonical tags AND ai_status = pending or processing beyond the expected ~5–30 second window for v5 indexing.
  • The event’s ai_status is failed or skipped.
  • The Events list shows the event with no canonical-tag chips next to the camera name.

Likely causes (v5)

  1. AI Runtime sidecar is degraded. The local Python AI Runtime process (ai-runtime/) is unreachable or returning HTTP 503. The cloud client retries 3 times with backoff (500ms then 5s, 3s per-attempt timeout); persistent failures mark the event AI evidence stage as failed.
  2. Master toggle is off. ai_v5_intelligence_enabled = false on /admin/ai short-circuits the AI pipeline — no event is enqueued. New events stay in pending.
  3. Master feature flag for the missing surface is off. If object_detection_enabled = false, the YOLO stage is skipped and the AI evidence panel’s “Objects” tab is empty; same for face_recognition_enabled and the “Faces” tab.
  4. Frame extraction failed. The pipeline pulls N=4 uniform frames from the event clip before calling the AI Runtime. If the clip is missing, corrupted, or uses a codec ffmpeg cannot decode on the cloud, S1 fails and the event reaches failed.
  5. Embedding-norm mismatch. A returned embedding’s L2 norm fell outside [0.99, 1.01]. The cloud client rejects the response and retries; a model artifact swap or corrupt ONNX export typically causes this.
  6. Org is in ai_maintenance mode. The AI queue is paused so admins can run a clean migration / backfill. New events sit in pending until maintenance finishes.
  7. Clip upload failed. When the gateway / cloud recorder cannot upload the clip, the event is skipped with clip_status = 'failed' and the AI pipeline never runs.

Fixes

If the AI Runtime is degraded

  1. Open /admin/ai and look at AI Runtime — Model health.
  2. If any of the three local cards (MobileCLIP2-B / YOLO26n / InsightFace) reads Unloaded, Degraded, or Unknown for more than a refresh cycle, the sidecar is not serving requests.
  3. Use the operator runbook in Emergency: handle Gemini and AI Runtime degraded modes to restart the sidecar or fall back to feature-flagged degradation.
  4. After the sidecar is healthy, trigger a bulk reprocess: /admin/ai → Maintenance wizard is the safer path for a backlog; POST /api/v1/ai/reprocess with {"scope": "failed"} works for a small failed-event count.

If the master or feature toggle is off

  1. Open /admin/ai and click the Scope matrix tab in AI features.
  2. Verify the master toggle is on and the per-feature flag for the missing surface (object detection, face recognition, etc.) is on.
  3. Toggle it back on. Audit row ai_feature_flags.updated is emitted automatically.
  4. New events will be analysed immediately; reprocess past events via POST /api/v1/ai/reprocess.

If frame extraction failed

  1. Open the event, check the Clip tab. If the clip will not play, the v5 frame extractor cannot decode it either.
  2. Check the camera’s codec on the Cameras detail page. H.264 is universal; H.265 requires a healthy cloud H.265 decoder.
  3. If many events from the same camera fail, the camera’s stream configuration is the common cause — see the live-view troubleshooting page for codec checks.

If the event is stuck because of ai_maintenance mode

  1. Open /admin/ai → Maintenance wizard. The wizard shows the current mode.
  2. If the org is in ai_maintenance and a backfill is in progress, wait for it to finish. The Finish button is gated server-side: POST /api/v1/admin/ai-maintenance/finish returns 409 while backfill is running.
  3. Once the wizard exits to normal mode, the AI queue drains and new events index.

If the embedding norm is out of range

This indicates a model artifact mismatch. Run the checksum preflight:

  1. Open /admin/ai → Maintenance wizard.
  2. Click Run checksum preflight (or GET /api/v1/admin/ai-maintenance/preflight/checksums).
  3. Any model whose actual SHA256 does not match ai-runtime/reference_checksums.json is degraded — the sidecar must be rebuilt with the canonical artifact before indexing can resume.

Verify the fix

  1. Trigger a new event from the affected camera — record a clip manually from the Cameras page.
  2. Wait up to 30 seconds.
  3. Open the new event’s AI evidence panel. You should see:
    • Overview tab: ai_status = indexed (or understood if the VLM summary already ran).
    • Frames tab: 4 frame thumbnails (signed URLs from object storage).
    • Tags tab: 1–8 canonical tags with similarity scores.
    • Objects tab: YOLO26n detections (when object_detection_enabled = true).
    • Faces tab: InsightFace faces with matched profile labels (when face_recognition_enabled = true).

Still stuck?