Skip to content

Routing tiers

Outcome: Understand the ordered pipeline that turns a request into a response (and a tier label).

Pipeline

flowchart TD
  req[Client request] --> auth[Auth]
  auth --> adapt[Gateway adapter]
  adapt --> bound[Boundaries optional]
  bound --> guard[Guardrails optional]
  guard --> L0[L0 exact cache]
  L0 --> L1[L1 semantic cache]
  L1 --> CCS[CCS command context]
  CCS --> L2[L2 rules / L2-dev]
  L2 --> Lt[Lt tools + PolicyEngine]
  Lt --> L3[L3 small local]
  L3 --> L4[L4 medium local]
  L4 --> L5[L5 large local]
  L5 --> pool[Org pool optional]
  pool --> L6[L6 frontier]
Tier Role Marginal cost
L0 Exact cache (identical prompt) $0
L1 Semantic cache (similar meaning) $0
CCS Reuse command/tool output across turns $0
L2 Deterministic rules / transforms $0
Lt Shell/IDE tools (policy gated) $0 compute
L3–L5 Local models (Ollama or MLX) Electricity
L6 Frontier APIs (OpenAI, Anthropic, …) Paid

$0 tiers = L0, L1, L2, Lt (no frontier invoice).

Concurrent identical L0 cold misses coalesce in-process (singleflight): one upstream fill, waiters share the body, cache writes once. Stream and non-stream share the same in-flight map (#499 / #506). An upstream error clears the in-flight slot so the next attempt can retry.

Ask-path L1 cold misses with the same normalized embed key likewise share one nearest lookup and one upstream fill (#517); agent turns still skip L1 and keep exact-L0 singleflight only.

Escalation

Local models escalate on low confidence, latency budget miss, or capability gaps (tools/vision/json). Image blocks on user or tool-result messages require vision (#397); hops log modality_escalation. Caps:

  • Config: routing.max_tier_for_chat, routing.no_frontier (via project profile)
  • Headers: X-Daari-Tier-Cap, X-Daari-No-Frontier, X-Daari-Tier-Override
  • Body: OpenRouter cost_tier / plugins: [{id: "auto-router", cost_tier}] (low→L3, medium→L4, high→L5, xhigh/max→L6). Header wins.

Agent/tool_calls flows skip L1 and Lt/L2. Exact L0 is on for an identical full history + tools schema; changing the last tool result is a miss (ADR-0004 / G1).

Shadow evals for tier decisions

"Would a bigger model have answered differently?" is measurable without changing what users see. With routing.shadow_sample_rate > 0, that fraction of responses served by a local tier (L3–L5, not cache hits, not tool flows) is replayed in a background task at a comparison tier and the two answers are compared with the same embedder/cosine path the L1 shadow check uses. The served response is never delayed or altered; the replay is executor-only, so it writes nothing to the caches, adds no usage-ledger row, and is not a learned-routing outcome.

  • routing.shadow_compare_tier — empty (default) means the highest configured local tier, so measuring routing quality costs $0. L6 replays against the frontier and is honoured only while routing.shadow_daily_usd > 0; the in-memory daily estimate stops replays at the cap.
  • A replay against the same model as the one that served (e.g. a single-model install) is skipped — there is nothing to compare.

Results land per category in the feedback store next to the cache false-hit rate: daari learn stats prints a "Tier divergence" table (samples, agree%, diverge%, comparison tier), daari report / GET /v1/daari/report carry tier_divergence, GET /v1/daari/learn/stats carries tier_shadow, and Prometheus exports daari_tier_shadow_samples_total{agreed="true|false"}. Default off; tests set the rate explicitly so the suite stays deterministic.

Session affinity

routing.session_affinity (default off) keeps an agent loop on the model that planned the task. A continuation — tool results after an assistant tool-call, or the same user-turn prefix — replays that tier. A new human turn re-routes.

Session identity is X-Daari-Session or the OpenAI user field when present, otherwise a hash of the user/system prefix. Pins expire after routing.session_affinity_ttl_seconds (default 30 minutes). When cache.backend: redis, session pins, profile pins (category/complexity for tool continuations), and the session cost-avoided rollup are stored in Redis with that TTL so a multi-replica fleet shares them; Redis errors fail open to in-process memory and log session_affinity.degraded. Confidence failure, context-length failover, an open circuit, a down model, and X-Daari-Tier-Cap beat the pin. Hits log session_pin; overrides log session_pin_override.

Harness-aware profiling

routing.harness_aware_profile (default on) keeps Claude Code / Cursor / Codex catalogs from flipping complexity. System messages and recognized harness blocks are ignored for category and complexity:

  • <system-reminder>…</system-reminder> (Claude Code)
  • <environment_context>…</environment_context>
  • <recommended_plugins>…</recommended_plugins>
  • <user_instructions>…</user_instructions>
  • <environments_instructions>…</environments_instructions>
  • # agents.md instructions for …</instructions> (Codex repository envelope)

prompt_tokens_est still counts the full request so context-window escalation (#385) sees real size. A no-op when those markers are absent. Off restores the old sum-every-message heuristic. Trace/event: harness_profile with stripped_chars.

Agent User-Agents (cursor / claude-code / codex) also get the classify user turn UA shortcut so tool-result continuations reuse the stripped profile without re-scoring harness noise.

Classify user turn

routing.classify_user_turn (default off) skips re-running build_prompt_profile on tool-result continuations (assistant tool-call + role=tool / tool_result suffix, no new user text). The prior user-turn category/complexity are reused; prompt_tokens_est still reflects the full message list. Phase routing and stall escalation still inspect tool history. A new user message re-profiles. Trace/event: classify_user_turn with reused: true and source: config.

When the global flag is off, routing.classify_user_turn_agents (default on) still enables the same reuse for agent User-Agents matching cursor, claude-code, claude code, or codex (also via sniffed client_id). Event source is then ua. Set classify_user_turn_agents: false to disable the shortcut. Explicit classify_user_turn: true applies to every client.

Stall escalation

routing.stall_escalation.enabled (default off) looks only at the request's tool history. Three identical calls (same name and normalized arguments) in the last six tool calls, or three consecutive error tool results, bump the chosen tier by one. X-Daari-Tier-Cap and routing.max_tier_for_chat still win. The event is stall_escalation with the pattern and repeat count.

Phase routing

routing.phase_routing.enabled (default off) classifies the last routing.phase_routing.window (default 6) tool-call names as explore (read/search/list/fetch/grep), implement (edit/write/apply/replace/patch), or verify (test/run/build/lint). Unknown or empty history leaves the heuristic pick alone. Each phase maps to a relative delta or absolute tier (explore: -1, implement: 0, verify: 0 by default; floor L3).

Composition: heuristic → phase → tier cap → latency budget → TTFT preference (when routing.ttft_aware) → capability filter → stall → context-window escalation → cap again. Stall beats a phase downgrade. Session affinity pins the phase-adjusted served tier on continuations (no re-classify); a context-window bump is not pinned. Non-agent requests (no tools, no tool history) are unchanged. Trace step and event: phase_route (phase, signals, delta, from, to).

Decision-model complexity classifier

routing.decision_classifier.enabled (default off) calls Ollama /v1/systemone (model routing.decision_classifier.model, default nimble) with a difficulty choice before the heuristic local tier pick. choice/score answers map onto L3–L5; on timeout or Ollama failure the request continues with today's heuristics and logs decision_classifier_degraded. Ask turns only unless routing.decision_classifier.agent_turns is true. Trace step decision_classifier and daari_meta.decision record model, answer, complexity, and chosen tier. Prometheus exposes daari_decision_classifier_total{outcome} (success / heuristic_fallback / error / skipped) and daari_decision_classifier_latency_ms (see metrics-prometheus.md).

Compact-to-fit before L6

routing.compact_to_fit.enabled (default off) trims oldest unprotected chat turns before frontier escalate. System messages, tool-call payloads, and multimodal turns that carry images / audio are never dropped; estimate_tokens adds a per-image floor (85) and a per-audio heuristic (floor 50, plus base64 length / 100) so vision/audio turns are not treated as empty when content is blank. If the remainder still exceeds the cap, history stays oversized (fail closed). When a trim actually changes the message count, daari_meta.compact_to_fit records messages_before / messages_after plus tokens_before / tokens_after, /v1/daari/stats increments compact_to_fit_applied, and compact_to_fit_tokens_dropped adds tokens_before - tokens_after. Unchanged or fail-closed history omits the meta object and does not increment the counters.

TTFT-aware local preference

Dry-run the initial local pick without calling Ollama or frontier: POST /v1/daari/route/preview (body messages or prompt) or daari route preview "sample prompt". The response is {tier, reasons, policy, chain} where reasons has heuristic, phase, latency_budget, and ttft_preference (a later field is set only when that step changed the pick). policy is the resolved timeout/retry for the chosen local tier; chain lists every local tier and frontier slot with its effective timeout_s, retry_attempts, and retry_backoff_s.

routing.ttft_aware (default off) uses recent stream daari_ttft_ms samples to prefer a faster local tier (never escalate) when that tier has at least routing.ttft_min_samples (default 20) and a lower routing.ttft_percentile (default 0.95) than the heuristic pick. Trace/event: ttft_preference. Non-stream traffic does not record TTFT, so the bias stays quiet until streams have warmed the histograms.

Context-window escalation

routing.context_window_escalation (default on) compares prompt_tokens_est to routing.context_windows (defaults L3=8192, L4=32768, L5=131072) times context_window_escalation_buffer (0.95). A proven overflow picks the cheapest higher local tier with a known window that fits. Unknown windows are left alone. X-Daari-Tier-Cap still wins. Trace/event: context_window_escalation. Post-error context_length_failover stays as the safety net. The same table is advertised as context_length on local tier cards from GET /v1/models (#400); unknown windows are omitted. Frontier/L6 cards are never overwritten from this table. Capability long_context is no longer inferred from a 24k-char threshold (#401) — this escalation path is the single length hop.

Per-entry retry and timeout

upstream.retry and upstream.local_timeout_seconds / upstream.frontier_timeout_seconds remain the defaults. A single global budget is wrong for a chain that mixes slow local models with hosted APIs, so each hop can override:

models:
  l3: llama3.2:3b
  l4: llama3.1:8b
  timeout_s:
    L3: 45
    L5: 180          # L4 keeps upstream.local_timeout_seconds

frontier:
  enabled: true
  providers:
    - id: openai
      timeout_s: 30
      retry_attempts: 2
      retry_backoff_s: 0.5
    - id: anthropic
      timeout_s: 60
      retry_attempts: 0   # one attempt, then fail over

Retries live inside the hop. Exhausting a provider's attempts advances the failover chain; that hop's timeout does not shrink the next slot's budget. Unset fields inherit today's globals, so existing configs are unchanged.

Knobs

See Config overview and Config reference.

Next

→ Caching and trust · Headers