Auth and virtual keys¶

Outcome: Lock down the gateway and issue per-client keys.

Master API key¶

server:
  api_key: "replace-me-with-a-unique-secret"

Clients send Authorization: Bearer <key> or x-api-key. /health stays open.

daari serve refuses to bind when server.api_key is unset/empty/whitespace or matches a small weak denylist (sk-1234, changeme, daari-local, …), and prints which source failed (#1320). For hermetic tests and local sandboxes set:

server:
  dangerously_permit_weak_or_unset_api_key: true

(or DAARI_SERVER__DANGEROUSLY_PERMIT_WEAK_OR_UNSET_API_KEY=true). daari doctor warns when the escape hatch is on, and when the master key is unset while virtual keys or mcp_oauth.local_as imply auth is expected.

The local watchdog's com.daari.serve launchd unit sets DAARI_SERVER__DANGEROUSLY_PERMIT_WEAK_OR_UNSET_API_KEY=true so sandbox serve survives the weak-key bind gate (#1328). After pulling that change, re-run scripts/autodev-local.sh --install so launchd reloads the plist; skipping reload leaves serve crash-looping and the watchdog filing connection-refused tickets.

Token introspection (RFC 7662)¶

POST /introspect lets an authenticated caller (master or virtual key) ask whether another token is active without leaking secrets. Accepts JSON {"token":"…"} or form token=…. Response: {"active": true, …} with client_id, username, optional exp, team_id / team_name, tier_cap, rpm / tpm / rpd (omitted when 0); or {"active": false} for unknown / revoked / expired tokens (HTTP 200). Unauthenticated callers get 401.

Tunnel setup (daari setup cursor --tunnel) auto-generates a key when unset.

Virtual keys¶

daari keys create --name alice --daily-budget 5
daari keys create --name ci --expires 30d
daari keys create --name shared-agent --user-daily-cap 2
daari keys list
daari keys rotate <key_id>            # new secret; old works for 24h
daari keys rotate <key_id> --grace 0  # immediate cutover
daari keys revoke <key_id>

Hashed storage; plaintext shown once. Supports daily/monthly budgets, RPM, TPM, tier caps, and an optional expiry (--expires 30d|12h|45m or an ISO-8601 timestamp). Existing keys with no expiry never expire. An expired key is a 401 key_expired (distinct from invalid/revoked) and writes an auth.key_expired audit row. daari keys list shows expires, grace_until (pending rotation), and a status of active / expired / revoked.

Proactive expiry (before that 401): alerts.key_expiry_warn_days (default 14) drives daari doctor warnings, the scrape gauge daari_virtual_keys{status}, and an optional alerts.key_expiry_webhook_url POST with event: key_expiry (same X-Daari-Signature scheme as budget webhooks). Recipe and payload: budgets and frontier.

daari keys rotate mints a new secret for the same key_id — budgets, team, policies, RPM/TPM, and usage history are untouched. The old secret keeps authenticating until the grace window ends (default 24h; --grace 0 cuts over immediately), then is rejected as expired. Both secrets share one budget and rate-limit bucket. Rotation writes a keys.rotate audit row (key_id + grace deadline; never the secret).

daari keys create --name ci --rpm 60 --tpm 40000

Rate limits¶

Defaults apply to every key (including the master key). A virtual key's --rpm / --tpm override the global defaults for that key. 0 means unlimited. Chat and embeddings charge TPM from request characters (len // 4, minimum 1). POST /v1/audio/transcriptions and POST /v1/audio/translations charge len(file_bytes) // 4 so a large upload is not counted as one token. Teams can also set aggregate ceilings with daari keys team-create --rpm/--tpm (or team-update); those counters use rpm:team:{team_id} / tpm:team:{team_id} and are checked after per-key limits so N keys on one team share one budget (#546).

Optional modality-family ceilings on a virtual key live in key metadata (rate_families) so chat cannot starve images (or the reverse) on a shared key (#1099). Families: chat, embeddings, images, audio, moderations, rerank, ocr, mcp, systemone, decisions, other. Each entry may set rpm and/or tpm (0 / omitted = no family ceiling). Counters are rpm:{key_id}:family:{name} / tpm:{key_id}:family:{name} — exhausting images leaves chat open; exhausting mcp (POST /mcp, /v1/mcp/*) leaves chat open (#1202); exhausting systemone (POST /v1/systemone) leaves chat open (#1443); exhausting decisions (POST /v1/decisions) leaves chat open. Unset rate_families preserves today's global key rpm/tpm only.

daari keys create --name studio --rate-family chat:120 --rate-family images:10
# optional tpm: --rate-family chat:60:40000
# typed judgments: --rate-family decisions:30

--rpd on daari keys create / keys update and daari keys team-create / team-update sets a calendar-day request cap (UTC, 86400-second window, 0 or omitted = unlimited). It is independent of rpm: a key can exhaust its day cap while the minute window is still open, and the reverse. Key and team caps both apply; the tighter remaining counter wins. Team rpd sums member traffic on rpd:team:{team_id}, the same way team rpm sums rpm:team:{team_id}. A hard deny is 429 with X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset (epoch when the UTC day rolls) and a body that names rpd. Retry-After on that 429 is the seconds until the UTC day resets, not the short rate_limit.retry_after_seconds used by rpm/tpm. Export/import round-trips the field.

rate_limit:
  rpm: 60
  tpm: 40000
  model_rpm: 0          # 0 = same as rpm, scoped per model
  model_tpm: 0
  max_in_flight: 8      # 0 = no concurrency gate
  queue_size: 32        # waiters before 503 + Retry-After
  retry_after_seconds: 1
  fail_open: false      # true = allow (uncounted) if Redis is down
  # Named per-model overrides (#1592). Exact id wins; else longest prefix.
  models:
    llama3-70b:
      rpm: 10
      tpm: 20000
      max_in_flight: 2

Per-key / per-team concurrency¶

Virtual keys and teams accept optional max_in_flight (0 = unlimited) via daari keys create --max-in-flight N / daari keys team-create --max-in-flight N (also on the store API). When set, admission tracks a scoped gauge (inflight:key:{id} / inflight:team:{id}) on the same Redis/SQLite counter backend as RPM/TPM. Overflow is 503 + Retry-After with X-RateLimit-Scope of key_concurrency or team_concurrency (global gate stays concurrency). daari_rejects_total uses those same kind labels so dashboards can separate key monopolization from fleet-wide admission pressure. Scrapes expose daari_in_flight{scope,id} / daari_in_flight_max{scope,id} for keys, teams, and named rate_limit.models with active caps.

Named per-model map¶

rate_limit.models maps a model id (or prefix) to rpm / tpm / max_in_flight. Matching is exact id first, then longest prefix. A positive map value overrides the global model_rpm / model_tpm defaults for that model; 0 falls through. Model-scoped max_in_flight uses gauge inflight:model:{model} with scope model_concurrency.

Counters live in Redis when cache.backend: redis (daari:rl: prefix); otherwise SQLite next to the virtual-key store. Responses include X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-RateLimit-Backend, and X-RateLimit-Scope (key / team / model / key_concurrency / team_concurrency / model_concurrency — which counter was tightest). When usage crosses frontier.soft_budget_ratio (default 0.8) but is still under the hard RPM/TPM cap, responses also carry x-daari-ratelimit-warning: soft and (with X-Daari-Meta) daari_meta.warning=rate_limit_warning so agents can back off before the hard 429 (#518). Hard denials stay 429 with the existing rate-limit headers unchanged. /metrics exposes the configured limits, current in-flight / queued gauges, and per-team RPM/TPM remaining when teams have ceilings.

Redis outage semantics (fleet)¶

With cache.backend: redis, every Redis client uses cache.redis_timeout_seconds (default 2s) for connect and command timeouts so a hung Redis cannot stall requests for the OS TCP timeout. If Redis counters raise or time out, the gateway logs one rate_limit.degraded event and switches rate-limit counting to the per-replica SQLite backend (or allows without counting when rate_limit.fail_open: true). Requests never 500 because Redis is down. When Redis answers again, counting returns to Redis without a restart. /ready includes a redis check; Redis down with fallback active is degraded but HTTP 200 (same pattern as a partial local-pool outage). L0/L1 cache misses already degrade gracefully on Redis errors.

Postgres outage semantics (control plane)¶

When any store uses observability.postgres_url (server.virtual_keys.backend, observability.backend, observability.request_log.backend, enterprise.audit_backend, or batches/files/responses backends), /ready exposes checks.postgres and daari doctor reports per-store reachability (postgres.virtual_keys, postgres.usage, …). Sharp stores fail closed; telemetry fails open:

Store	Redis down	Postgres down
Auth (virtual keys)	n/a (VK is SQLite/PG)	fail-closed — `503` with `error.code=postgres_unavailable` (never open, never raw 500)
Budgets (usage ledger)	n/a	soft-allow — spend reads return `0.0`, response carries `X-Daari-Budget-Degraded: soft_allow`, Prom `daari_budget_degraded{mode="soft_allow"}=1`
Rate limit	sqlite_fallback (default) or fail_open when `rate_limit.fail_open: true`; `/ready` degraded	n/a (counters are Redis/SQLite)
Audit	n/a	fail-open (writes swallowed; undocumented prior behavior, now explicit)
Request log	n/a	fail-open (writes swallowed; telemetry must not block inference)

SSO (admin)¶

OIDC/JWKS for admin surfaces — see ADR and enterprise settings (enterprise.sso). RSA (RS256/384/512) and EC (ES256/384/512) signing keys; use: "sig" is preferred when a JWKS also lists encryption keys. HMAC stub remains for local/dev when JWKS unset.

Multiple IdPs: set jwks_urls to a list (and optionally keep jwks_url as the first entry). Verification tries each JWKS until the token's kid matches; if every document misses, the request is rejected with the same 401 as a single unknown key.

Role matrix (when `enterprise.sso.enabled`)¶

Roles rank admin > analyst > user (daari/enterprise/rbac.py). The master server.api_key always counts as admin. Claim name defaults to role (enterprise.sso.role_claim).

SSO bearers are control-plane only. A verified IdP access token is accepted on /v1/daari/* and /v1/org-learning/* (role-gated admin surfaces). Raw SSO bearers on inference/data-plane routes (/v1/chat/completions, embeddings, images, …) return 401 with code: sso_key_required — mint a virtual key via POST /v1/daari/sso/session and send that dk_… key so budgets, allowlists, and attribution apply.

Surface	Minimum role
`GET /v1/daari/stats`, `traces`, `report`, `audit`, `config`	`analyst`
`GET/POST /v1/daari/route/preview`, `GET /v1/daari/cache/diversity`, `GET /v1/daari/learn/stats`, `POST /v1/daari/feedback`	`analyst`
`PATCH /v1/daari/config`	`enterprise.sso.admin_min_role` (default `admin`)
`POST /v1/daari/reload-caches`, `POST /v1/org-learning/sync`	`admin_min_role`

With SSO off (or no JWKS/secret configured) and a master key set, admin-level gates (PATCH /v1/daari/config, reload-caches, …) require the master key — virtual keys get 403. Analyst/read surfaces (GET /v1/daari/config, stats, traces) stay viewer-style for virtual keys. With no master key (sandbox hatch dangerously_permit_weak_or_unset_api_key), gates stay open; daari doctor warns that the config editor is ungoverned.

Scoped reads (`?team_id=`)¶

GET /v1/daari/traces, /v1/daari/stats, and /v1/daari/report accept an optional ?team_id= filter so analysts can mirror CLI daari spend export --team on the HTTP plane (#1677).

Caller	Behavior
Master key / SSO `admin`	May omit `team_id` (org-wide) or pass any team id
SSO `analyst` or virtual key bound to a team	Restricted to that team: omitting `team_id` auto-scopes; requesting another team's id returns 403 `team scope denied`
Virtual key with no team pin	Unfiltered (same as today) unless `?team_id=` is passed

Trace rows store team_id on write (from the request's VK/team attribution). SQLite and Postgres trace stores both honor the list filter.

IdP-minted virtual keys (MDM)¶

Alongside daari enterprise bootstrap, map an IdP claim to key policy so devices never need a hand-distributed dk_… secret. First verified POST /v1/daari/sso/session mints a key; later logins resync limits. When the mapped claim disappears, the key is revoked. Unmapped claims use default_policy or 403 if deny_unmapped: true.

enterprise:
  sso:
    enabled: true
    jwks_url: https://idp.corp/jwks
    # Or trust two IdPs:
    # jwks_urls:
    #   - https://okta.corp/oauth2/default/v1/keys
    #   - https://ci-idp.corp/jwks
    mint_virtual_key_on_login: true
    mapping_claim: groups
    key_mappings:
      eng:
        daily_budget_usd: 5
        rpm: 120
        tier_cap: L4
        boundary_profile: fintech
      contractors:
        tier_cap: L3
        key_ttl: 8h
    default_policy:
      tier_cap: L3
    deny_unmapped: false

Mint and revoke events land in the audit log with the claim that caused them.

Review and export¶

daari audit list --limit 50
daari audit list --actor alice --action budget. --since 7d --json
daari audit export --format jsonl --since 30d --out /tmp/daari-audit.jsonl
daari audit export --format jsonl --since 7d | jq -c 'select(.action|startswith("keys."))'
daari audit verify
daari audit verify --json

Rows are newest-first. Export streams JSONL (seq, ts, actor, role, action, detail, prev_hash, row_hash) without loading the whole table — suitable for SIEM ingestion. New rows are hash-chained (SHA-256 over a canonical serialization of seq/ts/actor/role/action/detail/prev_hash); the first chained row anchors on genesis 0×64. Pre-upgrade rows verify as legacy. daari audit verify walks oldest-first and exits non-zero on tamper (hash_mismatch or seq_gap). Retention prune re-anchors the chain head. The default store is SQLite at enterprise.audit_path (~/.daari/audit/audit.sqlite3). For a multi-replica fleet set enterprise.audit_backend: postgres (reuses observability.postgres_url) so every replica appends to one serialized hash chain; daari audit verify then covers the whole fleet. Helm sets DAARI_ENTERPRISE__AUDIT_BACKEND=postgres when postgres.enabled is true.

Audit event catalog¶

Action	When	Detail (never secrets)
`keys.create`	`daari keys create`	`key_id`, `name`, `prefix`
`keys.rotate`	`daari keys rotate`	`key_id`, `grace`, `grace_until`
`keys.revoke`	`daari keys revoke`	`key_id`
`teams.create`	`daari keys team-create` (new team)	`team_id`, `name`, `windows`
`teams.update`	`daari keys team-update`	`team_id`, `name`, `windows`
`auth.key_expired`	Expired virtual key 401	`key_id`, `expires_at`
`auth.sso_key_required`	Raw SSO bearer on inference route 401	`path`, `sub`
`auth.invalid_key`	Invalid/missing key 401	`prefix` (≤10 chars), `path`; identical `(prefix, path)` deduped for 60s
`auth.key_locked`	Same key prefix exceeded `auth.key_lockout.max_failures`	`prefix` (≤10), `failures`, `retry_after`, `scope=key`
`budget.exceeded`	Frontier/key budget or request-quota 402	`key_id`, `team_id`, `scope`, `window`, `remaining`, `limit`, `quota` (no prompts/secrets)
`rate_limit.exceeded`	RPM/TPM/RPD 429	`key_id`, `team_id`, `model`, `scope`, `bucket`, `remaining`, `limit`
`concurrency.exceeded`	In-flight concurrency 503	`key_id`, `team_id`, `model`, `scope`, `remaining`, `limit`
`tenancy.denied`	Cross-tenant file/batch/response access	`key_id`, `kind`, `id`
`sso.mint_virtual_key` / `sso.revoke_virtual_key`	SSO key sync	claim + key_id
`budget.alert`	Budget webhook fired	scope / threshold
`config.patch`	Admin config patch	changed keys
`admin.keys.list`	`GET /v1/daari/keys`	`limit`, `offset`, `total`, `returned`
`admin.teams.list`	`GET /v1/daari/teams`	`limit`, `offset`, `total`, `returned`
`cache.invalidate`	`POST /v1/daari/cache/invalidate`	`scope` (model\|hash\|team\|key\|all), `l0_removed`, `l1_removed` (no entry bodies)
`retention.prune`	Retention sweep	counts
`mcp.tools/call` / `mcp.guardrail`	MCP policy	tool / decision

Secret references (`secret://`)¶

Any secret-bearing config value (frontier provider keys, org tokens, Redis/Postgres URLs, OIDC client secrets) can be a secret:// URI instead of plaintext (issue #288). Resolution shells out at daemon startup — no vault SDK, no new dependency:

frontier:
  providers:
    - id: openai
      keys:
        - secret://keychain/daari-frontier/naveen      # macOS security / Linux secret-tool
enterprise:
  shared_cache_token: secret://env-file//etc/daari/secrets.env#ORG_TOKEN
cache:
  redis_url: secret://exec/op read op://infra/daari-redis/url
secrets:
  refresh_ttl_seconds: 300   # default; 0 = boot-only (legacy)

secret://env-file/<path>#<KEY> — KEY=VALUE line in a root-only file (quotes and export prefixes are stripped).
secret://exec/<command> — stdout of any operator command: op read, vault kv get -field=..., aws secretsmanager get-secret-value ....
secret://keychain/<service>/<account> — macOS Keychain via security find-generic-password, Linux secret-service via secret-tool.
secret://oauth/<token-url>?client_id=<id>&client_secret=<secret-ref>[&scope=…] — no static upstream key at all; see below.

A ref that fails to resolve is fatal at startup with a message naming the ref (never the value); daari doctor verifies every configured ref resolves. Resolved values are redacted from gateway logs. Plain string values keep working unchanged.

Refresh and rotation (#1204)¶

Non-oauth refs (env-file, exec, keychain) re-resolve at use time when secrets.refresh_ttl_seconds is greater than zero (default 300):

Source	Refresh trigger
`env-file`	TTL elapsed or source file mtime changed
`exec` / `keychain`	TTL elapsed

Call sites that already go through current_secret() — frontier key pick, org cache/learning clients — pick up the new value on the next request without a process restart. Set secrets.refresh_ttl_seconds: 0 (or DAARI_SECRETS__REFRESH_TTL_SECONDS=0) to keep the legacy boot-only behavior.

Rotation runbook (env-file / vault CLI):

Write the new value to the env file (or update the vault item the exec command reads).
For env-file, touch/save is enough — the next credential read notices the mtime change even inside the TTL window.
For exec/keychain, wait for the TTL (or temporarily set a short TTL) before the next request; no daemon restart.
Confirm with daari doctor (secret_refs row shows the refresh TTL) and a live request that exercises the frontier/org path.

OAuth tokens ignore this TTL — they re-mint near expires_in as below. daari doctor warns when refresh is disabled (ttl=0) and no oauth refs are configured, so operators know a rotation still needs a restart.

OAuth client credentials (`secret://oauth`)¶

Enterprises that front OpenAI/Anthropic with an internal token service (workload identity federation, OAuth client-credentials upstream auth) never want a long-lived provider key on a laptop. The oauth scheme performs an RFC 6749 client-credentials grant against your token endpoint and uses the returned access_token as the credential (#321):

frontier:
  providers:
    - id: openai
      base_url: https://llm-proxy.corp.example.com/v1
      keys:
        - secret://oauth/https://idp.corp.example.com/oauth2/token?client_id=daari-laptops&client_secret=secret://keychain/daari-idp/client-secret&scope=llm.invoke&audience=https://llm-proxy.corp.example.com

client_id (required) and client_secret (required) — the secret must be another secret:// ref (env-file, exec or keychain); a plaintext secret in the URL is rejected. Percent-encode a nested ref that contains &, + or spaces.
scope, audience, resource — optional, passed through verbatim for token services that require them.
auth=basic|post — how the client authenticates to the token endpoint. Default basic (HTTP Basic, which RFC 6749 requires every server to support); post sends client_id/client_secret in the form body for IdPs configured for client_secret_post.
refresh_margin=<seconds> — re-mint this long before expires_in (default 60). A response without expires_in is treated as a one-hour token.

The token is minted on the machine that uses it and cached in memory until expiry minus the margin. Refresh is lazy: the next place that reads the credential — frontier key rotation before each L6 attempt, org cache/learning clients before each call — picks up a fresh token, with concurrent readers sharing one token request. A refresh that fails (non-2xx, malformed body, network error) raises with the endpoint in the message — never the secret or token — and the L6 attempt fails over to the next provider rather than sending an expired credential.

Header policy (pre-auth)¶

server.header_policy (default off) screens request headers before require_api_key — useful for blocking known-bad User-Agents, requiring a custom client header, or allowlisting X-* values (Portkey-style start hooks).

server:
  header_policy:
    enabled: true
    required: ["X-Client-Id"]
    deny:
      - header: user-agent
        exact: "BadBot/1.0"
      - header: user-agent
        regex: "(?i)scrapy"
    allow:
      x-daari-env: ["prod", "staging"]

Knob	Behavior
`required`	Header must be present and non-empty → 400 `header_required`
`deny`	Exact and/or regex match → 403 `header_denied`
`allow`	When the header is present, value must be one of the listed strings → 403 `header_not_allowed`

Deny bodies use "type": "header_policy_error" with the code above. Open probes stay exempt: /health, /ready, /v1/messages/health, /metrics.

Env: DAARI_SERVER__HEADER_POLICY__ENABLED (and nested keys). See config reference. Fleet installs turn the same knobs on via Helm server.headerPolicy in capacity-helm (DAARI_SERVER__HEADER_POLICY__* env). /mcp is screened the same as chat; open probes stay exempt.

Verify¶

curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:11435/v1/daari/stats
# expect 401 when key required
curl -s -H "Authorization: Bearer $KEY" http://127.0.0.1:11435/v1/daari/stats

daari doctor includes a secret_refs row when any secret:// value is configured.

Next¶

→ Virtual keys feature · Security