Benchmark: load

Generated by python scripts/bench_load.py against a hermetic daari serve and real Ollama. Three mixes: a warmed L0 replay (cache), unique no-cache generations with max_tokens capped (generate), and a tool-bearing agent replay (agent, G1 / #223). Reproduce: guide.

Hermetic micro-benchmarks (no Ollama)

In-process regression guards for TTFT-adjacent paths live under tests/benchmark/ (test_hermetic_paths.py, plus the existing L0-vs-mocked-L3 case). They are marked @pytest.mark.benchmark, so the default CI suite (pytest -m "not integration and not benchmark") skips them. Run locally when changing cache, budgets, rate limiting, or batches:

pytest -m benchmark -q

Each hermetic test asserts a loose absolute ceiling (roughly 10× typical local medians) so only order-of-magnitude regressions fail. They do not replace this page's live load numbers. The suite also guards L0 exact-cache singleflight: N concurrent identical cold misses must share exactly one upstream fill under a wall-clock ceiling (#520). The same pattern covers L1 embed-key singleflight: N concurrent same-normalized-embed cold misses (distinct L0 keys) must share exactly one embed before fill and one upstream under a wall ceiling (#528). Soft RPM soft-band bursts are also guarded: N concurrent allowed requests (memory rate limiter, frozen mid-window) must all return 200 with x-daari-ratelimit-warning: soft under a wall ceiling (#540). TTFT-aware preference rewrites are also guarded: N concurrent _choose_initial_tier calls (seeded TTFT histograms, ttft_aware=True) must all prefer the faster local tier with daari_ttft_preference_total ≥ N under a wall ceiling (#574). Hard RPM rejects are also guarded: N concurrent requests after the memory rate limiter is exhausted must all return 429 with daari_rejects_total{kind="rate_limit"} ≥ N under a wall ceiling (#585). Hard 402 budget and request-quota rejects are also guarded: N concurrent requests after USD budget or max_requests is exhausted must all return 402 with daari_rejects_total{kind="budget"|"request_quota"} ≥ N under a wall ceiling (#592). Soft request-quota soft-band bursts are also guarded: N concurrent allowed requests (VK ledger prefilled to frontier.soft_budget_ratio) must all return 200 with x-daari-quota-requests-warning: soft and daari_soft_warnings_total{kind="request_quota"} ≥ N under a wall ceiling, with no hard 402 (#620). Soft USD budget soft-band bursts are also guarded: N concurrent allowed requests (VK frontier spend prefilled to frontier.soft_budget_ratio) must all return 200 with x-daari-budget-warning: soft and daari_soft_warnings_total{kind="budget"} ≥ N under a wall ceiling, with no hard 402 (#626).

• Date: 2026-08-26
• Commit: c56999c
• Hardware: Apple M4 Pro, 48 GB RAM
• Ollama: 0.18.2

Mixes

Mix	Requests	Concurrency	RPS	p50 ms	p95 ms	L0 hit	Errors
cache	200	16	219.7	68	89	100%	0
generate	16	2	6.0	288	458	0%	0
agent	80	8	217.8	36	42	100%	0

Cache and agent RPS are the replay phase only (warm-up is excluded). The L0 hit column is daari_meta.cache_hit on those replay requests. Generate requests carry X-Daari-No-Cache and a small max_tokens so the run is a latency/throughput sample, not a quality eval.

The agent mix sends the same short prompts with an OpenAI tools schema attached — the Cursor/Claude Code shape. Implied frontier $ is gpt-4o list input rates on prompt tokens of L0 hits (priced, not billed).

Agent prefix (G1)

• Prompt tokens on L0 hits: 7280
• Implied frontier $ avoided: $0.0182 (gpt-4o input, priced not billed)