Skip to content
NovaVMS Manual

Emergency — handle Gemini and AI Runtime degraded modes

NovaVMS v5 ships a single VLM provider (Gemini 3.1 Flash-Lite via Vertex AI) plus a local AI Runtime sidecar running three CPU models. There is no second VLM provider to fail over to — the multi-provider AI_PROVIDER=openai toggle was retired in the L12 cutover. This runbook covers the two degraded modes operators DO have control over.

Mode 1: Gemini degraded

Symptom: gemini_judge_usage continues to climb but the “VLM understanding” tab on AI evidence panels stops populating; OR AIModelHealthCards Gemini card flips to a degraded badge; OR cloud logs show gemini AnalyzeEvent permanent failure for sustained periods.

Cloud behaviour:

  • Deterministic alert rules continue to fire (do not require Gemini).
  • Search falls back to the rule-based query parser (Phase 2 path).
  • Contextual alert judges return a “Gemini degraded” error code; the judge is skipped and the compiled rule treats it as a non-match by default.
  • New events still reach indexed because indexing does not call Gemini.

Operator action:

  1. Confirm the outage at https://status.cloud.google.com/ (Vertex AI). If it’s region-localised, you can flip VERTEX_AI_LOCATION to a different region in cloud/.env.prod:

    Terminal window
    ssh [email protected] 'cd /root/NovaVMS && \
      sed -i "s/^VERTEX_AI_LOCATION=.*/VERTEX_AI_LOCATION=us-east1/" cloud/.env.prod && \
      docker compose restart cloud'

    The cloud container restarts in < 30 seconds. Verify with docker logs novavms-cloud --tail=50 | grep gemini.

  2. If the outage is global to Vertex AI, announce a degraded mode in #novavms-incidents. There is no second VLM to fail over to. NovaVMS continues to function for indexing, search (rule parser), and deterministic alerts.

  3. If a partial org-level disable helps (for example, a customer’s contextual judges are spending too much during a Gemini brownout), set vlm_understanding_enabled = false on the org via PATCH /api/v1/ai/settings/features. The admin can do this from /admin/ai → AI features → Scope matrix.

  4. When Vertex AI recovers, flip the region back and restart the cloud container. The non-blocking VLM understanding layer drains automatically as the queue runs through pending summaries.

Mode 2: AI Runtime degraded

Symptom: /admin/ai → AI Runtime — Model health shows any of MobileCLIP2-B / YOLO26n / InsightFace as Degraded, Unloaded, or Unknown for more than a refresh cycle; OR cloud logs show ai-runtime retries exhausted for sustained periods; OR GET /api/v1/admin/ai-runtime/health returns {"ok": false, ...}.

Cloud behaviour:

  • Cloud client (cloud/internal/ai/runtime_client.go) retries 3 times with backoff (500ms then 5s, 3s per-attempt timeout).
  • Persistent failures mark per-event evidence stages failed; events do NOT reach indexed until the runtime recovers.
  • Search against already-indexed events continues — only new events stall.

Operator action:

  1. SSH to the cloud host and check the AI Runtime container:

    Terminal window
    ssh [email protected] 'docker compose ps ai-runtime'

    If it’s down or restart-looping, check the logs:

    Terminal window
    ssh [email protected] 'docker compose logs --tail=200 ai-runtime'

  2. Common AI Runtime failure modes:

    • Model checksum mismatch: run /admin/ai → Maintenance wizard → Run checksum preflight. Any non-matching SHA256 indicates a bad model artifact; rebuild the AI Runtime container with the canonical artifact from ai-runtime/reference_checksums.json.
    • OOM: the v5 RSS budget is < 3 GB warm. If the host is OOM-killing the container, free memory on the host or move to a larger instance.
    • Port collision: the sidecar binds 127.0.0.1:8091 by default. Confirm nothing else has claimed the port.

  3. If the AI Runtime is fully down and recovery will take more than a few minutes, set the org-level master toggle ai_v5_intelligence_enabled = false on every affected org via PATCH /api/v1/ai/settings. This stops events from piling up in pending and lets operators see the failure clearly in /admin/ai.

  4. After recovery, flip ai_v5_intelligence_enabled back on. Stalled events (those captured during the outage) do not auto-retry; trigger them via POST /api/v1/ai/reprocess with {"scope": "failed"} or {"scope": "date_range", "date_from": ..., "date_to": ...}.

Common variations

  • Tighter per-org scope. If only one org is affected, use the per-feature flags at /admin/ai → AI features → Scope matrix to disable specific stages (e.g. only object detection) without halting indexing.
  • AI Studio fallback for dev only. Setting GEMINI_API_KEY without VERTEX_AI_PROJECT routes Gemini calls to AI Studio’s public endpoint. Production HostHatch hosts are geo-blocked from that endpoint — keep Vertex AI on for prod.