Prometheus metrics¶
Outcome: Scrape GET /metrics for tier and boundary counters.
Steps¶
Enable exposition (see config observability / prometheus flags in reference). Scrape:
curl -s http://127.0.0.1:11435/metrics | head
When server.api_key is set, that scrape needs Authorization: Bearer …. For a
private scrape network, set observability.metrics_port (e.g. 9090) so daari
also binds 127.0.0.1:<port>/metrics without API key auth — keep that port
off public ingress. Main-port /metrics auth is unchanged.
Import Grafana dashboard: deploy/grafana/daari-dashboard.json, or enable Helm
grafanaDashboard.enabled so the chart ships the same JSON as a labeled
ConfigMap (see Capacity and Helm).
The dashboard includes a
TTFT p50 / p95 by tier panel on daari_ttft_ms for stream startup health,
a Soft warnings by kind panel on daari_soft_warnings_total for soft-band
pressure before hard 402/429, a Hard rejects by kind panel on
daari_rejects_total for cliffs after the soft band, a Rate-limit Redis
degraded panel on daari_rate_limit_degraded for shared-counter fallback,
a TTFT preference by from→to panel on daari_ttft_preference_total
for TTFT-aware local tier bias, a Backend pool health & outstanding
panel on daari_backend_up / daari_backend_outstanding (and request rate),
a Concurrency gate panel on daari_rate_limit_in_flight vs
_in_flight_max / _queued, a Scoped in-flight by key/team/model panel on
daari_in_flight / daari_in_flight_max (labels scope + id), alert-series panels for
daari_upstream_retries_total, daari_cache_false_hits_avoided_total, and
daari_budget_alerts_total, an MCP tool ingress by outcome panel on
daari_mcp_tool_calls_total, a Tier shadow agree/disagree panel on
daari_tier_shadow_samples_total, a Compact-to-fit applied panel on
daari_compact_to_fit_applied_total, a Compact-to-fit tokens dropped panel on
daari_compact_to_fit_tokens_dropped_total, an MCP grant denied panel on
daari_mcp_grant_denied_total, a Decision classifier by outcome panel on
daari_decision_classifier_total and a Decision classifier latency p50 / p95
panel on daari_decision_classifier_latency_ms, a Virtual keys by status panel on
daari_virtual_keys{status}, a Retired model rejects panel on
daari_rejects_total{kind="retired"}, and an Escalations & errors panel on
daari_escalations_total / daari_errors_total so operators see flake / false-hit / budget /
MCP deny / key-lifecycle / retired-model / routing-shadow / compact-to-fit pressure without raw PromQL).
Useful series: request latency histograms by tier (daari_request_latency_ms),
stream time-to-first-token histograms by tier (daari_ttft_ms — stream path
only; non-stream omits TTFT), daari_guardrail_trips_total,
daari_boundary_decisions_total.
Alert pack¶
The Helm chart ships an opt-in Prometheus Operator PrometheusRule pack
(deploy/helm/daari/templates/prometheusrule.yaml) gated on
prometheusRules.enabled (default false). Enable with scrape already
wired (serviceMonitor.enabled=true or an equivalent scrape of /metrics):
helm upgrade --install daari ./deploy/helm/daari \
--set serviceMonitor.enabled=true \
--set prometheusRules.enabled=true
Thresholds and for durations live under prometheusRules.* in
deploy/helm/daari/values.yaml (backend, budget.fraction,
errors.rateThreshold, queue.threshold, guardrails.rateThreshold,
keys, cacheInvalidate.rateThreshold). Each alert carries a severity
label and a runbook_url annotation pointing here.
| Alert | Metric(s) | Default trigger |
|---|---|---|
DaariBackendDown |
daari_backend_up |
== 0 for 5m |
DaariBudgetLow |
daari_team_budget_remaining_usd / daari_team_budget_limit_usd |
remaining/limit < 0.1 for 10m |
DaariErrorRateHigh |
daari_errors_total{status} |
sum(rate(...)) > 0.05 for 5m |
DaariAdmissionQueueDeep |
daari_rate_limit_queued |
> 10 for 5m |
DaariGuardrailTripSpike |
daari_guardrail_trips_total |
sum(rate(...)) > 1 for 5m |
DaariKeysExpiringSoon |
daari_virtual_keys{status="expiring_soon"} |
> 0 for 1h |
DaariCacheInvalidateErrors |
daari_cache_invalidate_total{result="error"} |
sum(rate(...)) > 0.05 for 5m |
Requires the monitoring.coreos.com CRDs (kube-prometheus /
prometheus-operator), same as ServiceMonitor.
Series worth watching¶
| Series | Reading it |
|---|---|
daari_upstream_retries_total |
Transient upstream failures absorbed by backoff. Rising here while errors stay flat means clients never saw the instability. Rising alongside errors means the failures are not retryable — check for 401s or malformed requests. |
daari_cache_false_hits_avoided_total |
L1 hits vetoed by verification. Steady growth is the cache working; a spike suggests the similarity threshold is too loose. |
daari_cache_invalidate_total{scope,result} |
Admin POST /v1/daari/cache/invalidate (and daemon CLI via that path). scope is model|hash|team|key|all; result is ok|error. Alert on surprise mass rate(...[5m]) spikes. No per-hash labels. |
daari_cache_invalidate_entries_removed_total{layer} |
Entries removed by those ops (layer="l0" or l1). Bounded cardinality companion to the op counter. |
daari_soft_warnings_total{kind} |
Soft-band pressure before hard rejects: kind="budget" (x-daari-budget-warning), kind="request_quota" (x-daari-quota-requests-warning), or kind="rate_limit" (x-daari-ratelimit-warning). Rising here while 402/429 stay flat means agents are hovering in the soft band — scale Ollama or widen caps before hard rejects start. Hard 402/429 do not increment this counter. Overview Grafana charts all kinds under Soft warnings by kind and isolates USD soft pressure under Soft USD budget warnings. |
daari_rejects_total{kind} |
Hard denials only: kind="budget" (USD 402), kind="request_quota" (request-cap 402), kind="rate_limit" (429), or kind="retired" (retired-model 410). Soft warning headers never increment this. Rising rejects after soft warnings means caps are undersized relative to demand. Overview Grafana isolates retired blocks under Retired model rejects; all kinds also chart under Hard rejects by kind. |
daari_virtual_keys{status} |
Scrape-time virtual-key lifecycle counts (active, expired, expiring_soon, revoked). Flat when no keys are stored. Rising expired / expiring_soon means credential rotation pressure. Stats counterpart: virtual_keys on GET /v1/daari/stats. Overview Grafana charts under Virtual keys by status. |
daari_ttft_preference_total{from,to} |
When routing.ttft_aware rewrites the heuristic local tier (e.g. L4→L3). Flat while TTFT bias is idle; rising means the preference path is active. Off / no-op picks do not increment. |
daari_rate_limit_degraded{mode} |
1 while Redis rate-limit counting is degraded (mode="sqlite_fallback" or mode="fail_open"); clears to 0 when Redis answers again. Page when any pod stays at 1 — fleet RPM/TPM is no longer shared. |
daari_budget_degraded{mode} |
1 while Postgres usage-ledger spend reads failed and budget pre-checks soft-allow (mode="soft_allow"). Responses also carry X-Daari-Budget-Degraded. See auth-and-keys. |
daari_mcp_tool_calls_total{tool,outcome} |
MCP ingress tools/call (and legacy /v1/mcp/query) by tool name and outcome (ok, deny, error, guardrail). No-op when observability.prometheus=false. Rising deny vs ok is policy pressure on agent traffic. |
daari_team_budget_remaining_usd{team,window} / daari_team_budget_limit_usd{team,window} / daari_team_budget_remaining_hours{team,window} |
Per-team USD budget windows from the virtual-key team store (scrape-time snapshot). Flat when no teams have max_usd windows. Alert when remaining approaches zero before hard 402s. Overview Grafana charts remaining vs limit under Team budget remaining (USD). |
daari_team_rate_limit_remaining{team,kind,scope} / daari_team_rate_limit_limit{team,kind,scope} |
Per-team RPM/TPM/RPD remaining and ceilings (scope="team", kind="rpm"|"tpm"|"rpd"). Flat when no teams have rpm/tpm/rpd set. Overview Grafana charts remaining vs limit under Team rate-limit remaining. |
daari_key_rate_limit_remaining{key,kind} / daari_key_rate_limit_limit{key,kind} |
Per-key RPD remaining for keys with rpd > 0. The key label is the key name (or key id if the name is empty), never the secret. A scrape does not consume the cap. Overview Grafana charts remaining vs limit under Key rate-limit remaining. |
daari_tier_shadow_samples_total{agreed} |
Local-tier answers replayed at a comparison tier (agreed="true\|false"). Rising false vs true means routing drift under routing.shadow_sample_rate. See routing-tiers.md shadow section. Overview Grafana charts agree/disagree under Tier shadow agree/disagree. |
daari_cancelled_requests_total{phase} |
Client hung up (phase is chat, anthropic, responses, stream, mcp, embed, asr, translation, or tts) and the upstream call was cancelled. Normal completions do not increment. Overview Grafana charts by phase under Cancelled requests by phase. |
daari_request_deadline_exceeded_total |
Wall-clock request budget spent (upstream.request_deadline_seconds or X-Daari-Deadline-Ms) on chat, audio, embeddings, and MCP tools/call — not chat-only. Escalation stops without a frontier call. Overview Grafana charts under Request deadline exceeded. |
daari_compact_to_fit_applied_total |
Chat histories actually trimmed before L6 (messages_before != messages_after). Unchanged or fail-closed history does not increment. No-op when observability.prometheus=false. Overview Grafana charts under Compact-to-fit applied. |
daari_compact_to_fit_tokens_dropped_total |
Estimated tokens removed by compact-to-fit before L6 (tokens_before - tokens_after via the shared estimator). Increments only when a trim changes message count; unchanged or fail-closed history does not. Zero when unused. No-op when observability.prometheus=false. Stats counterpart: compact_to_fit_tokens_dropped on GET /v1/daari/stats. Overview Grafana charts under Compact-to-fit tokens dropped. |
daari_mcp_grant_denied_total |
MCP initialize / tools/list / tools/call (and legacy /v1/mcp/query) when require_key_access_defined fail-closed applies because the virtual key has no MCP grant. Ordinary tool-name deny does not increment. No-op when observability.prometheus=false. Overview Grafana charts under MCP grant denied. |
daari_decision_classifier_total{outcome} |
Optional /v1/systemone difficulty hop (routing.decision_classifier.enabled) by outcome (success, heuristic_fallback, error, skipped). skipped is enabled-but-bypassed (agent turn, tier override, empty text); disabled classifier does not increment. No-op when observability.prometheus=false. Overview Grafana charts under Decision classifier by outcome. |
daari_decision_classifier_latency_ms |
Histogram of classifier hop wall time (ms) for success / heuristic_fallback / error — not recorded on skipped. No-op when observability.prometheus=false. Overview Grafana charts p50/p95 under Decision classifier latency p50 / p95. |
daari_escalations_total / daari_errors_total{status} |
Local→frontier (L6) escalations and gateway/router errors. status is the HTTP status returned to the client (503, 500, …); unlabeled call sites bucket as status="0". Alert pack uses sum(rate(daari_errors_total[…])) so labeled series stay compatible. Stats counterpart: errors_by_status on GET /v1/daari/stats. Overview Grafana charts both under Escalations & errors. |
The same soft_warnings / rejects maps are also on GET /v1/daari/stats (and the
local web-ui) so operators can see cliff pressure without scraping Prometheus.
That stats payload also includes backend_summary
(total / healthy / unhealthy / open_circuit) next to the per-backend
rows — see traces-stats.md.
Retries also appear per-request as upstream_retry trace steps with the status and
delay, so a slow request explains its own latency. See
traces and upstream.* in the
config reference.