Skip to content

Ollama backend

Outcome: Local L3–L5 inference via Ollama.

Steps

ollama pull llama3.2:3b
# optional L4
ollama pull llama3.1:8b
daari serve
daari doctor

Compose stack pulls models for you (docker compose up).

Config

ollama:
  base_url: http://127.0.0.1:11434
models:
  l3: llama3.2:3b

Add more hosts with routing.local_pool (issue #170). Empty backends keeps the single ollama.base_url. A dead host is skipped; /ready is degraded when some hosts are down and not_ready (503) when none can serve. Chat requests return typed backend_unavailable (503) when the pool cannot serve the chosen tier. Set routing.local_pool.frontier_fallback: true (default off) to escalate those requests to L6 instead — no_frontier, model allowlists, budgets, and PII scrub still apply (#846).

routing:
  local_pool:
    strategy: least_outstanding   # or round_robin
    health_interval_seconds: 15
    frontier_fallback: false      # true → L6 when all local hosts are down
    backends:
      - id: gpu-a
        base_url: http://127.0.0.1:11434
        tiers: [L3, L4, L5]
      - id: gpu-b
        base_url: http://192.168.1.10:11434
        tiers: [L3, L4]

Verify

GET /ready succeeds; chat completions return local tiers. daari_meta.backend_id names the host that served.

Decision models (POST /v1/systemone / POST /v1/decisions)

Ollama 0.35+ exposes TypeSafe/Jev-style decision models at /v1/systemone. daari proxies the same shape through the gateway (and OpenAI-shaped POST /v1/decisions) so clients keep API-key auth, spend metering, and daari_meta.

Built-in local catalog: clef, clef-flash, nimble, tev1, drex-1.5, microsoft-decision-1 (vendor/:tag variants accepted). Extend via decisions.local_models without a daari release. When systemone.enabled (default true), those ids also appear on OpenAI GET /v1/models, Anthropic-shaped GET /v1/models (Accept / anthropic-version / x-api-key clients get capabilities.decision.supported: true), and Ollama /api/tags (and /api/show) with capability tag decision so IDE/agents can discover on-box judges without hardcoding. Virtual-key allowed_models still filters /v1/models.

decisions:
  local_models: [my-judge, org/custom-decision]

curl -s http://127.0.0.1:11435/v1/systemone -d '{
  "model": "nimble",
  "state": "Checkout returns 500 since 9am.",
  "questions": {
    "label": {
      "type": "choice",
      "instructions": "Which label fits?",
      "criteria": {"billing": "Payments", "bug": "Errors", "other": "Else"}
    }
  }
}'

Ollama 0.35.1+ Clef multimodal models also accept "images": ["<base64>", …] alongside state; daari forwards them verbatim when present.

Opt out with systemone.enabled: false (returns 501). Upstream is ollama.base_url; when Ollama is down the route returns typed backend_unavailable (503). Doctor tip decisions_local_models fires when systemone is on but none of the catalog models appear in Ollama tags.

Next

→ MLX · Docker Compose