Phase 2: Architecture¶

Status: approved 2026-11-10; Phase 3 corrections noted in 03-api/README.md · Inputs: 01-requirements.md, research.md, ADRs

1. Summary¶

• Processes: two (server and worker) from one Python image, with Postgres 18 as the only stateful service.
• Hot paths: the server handles ingestion, the proxy, outcomes, policy serving and the admin, review and report APIs.
• Background work: the worker runs everything in the background from a job table in Postgres: joins, refits, drift checks, alerts, shadow reports and retention.
• Apps: gate locally with a pure-Python SDK that polls for policies.

Phase 1 open questions, settled:

2. C4 level 1: system context¶

flowchart TB
  dev["App developer / ML engineer<br/>[person]"]
  appr["Approver<br/>[person]"]
  rev["Reviewer<br/>[person]"]
  ops["Operator<br/>[person]"]
  app["Customer app<br/>[software system]<br/>calls decision models"]
  ts["trimset<br/>[software system]<br/>captures decisions, joins outcomes,<br/>refits and serves policies, alerts"]
  up["Decision-model upstream<br/>[external]<br/>Jev, Ollaya, Ollama, OpenAI"]
  src["Outcome sources<br/>[external]<br/>help desk, payments, moderation tools"]
  ch["Alert channels<br/>[external]<br/>Slack, email, webhooks"]
  otel["Observability stack<br/>[external, optional]<br/>OTLP collector"]

  dev -- "builds" --> app
  app -- "decision calls through the proxy,<br/>or log() and gate()" --> ts
  ts -- "forwards decision calls" --> up
  app -. "direct calls in SDK-only mode" .-> up
  src -- "outcomes by webhook" --> ts
  ts -- "alerts" --> ch
  ts -- "traces, metrics" --> otel
  dev -- "CLI: calibrate, simulate, compile" --> ts
  appr -- "CLI: approve, roll back" --> ts
  rev -- "CLI: label review items" --> ts
  ops -- "compose, migrate, keys" --> ts

3. C4 level 2: containers¶

flowchart TB
  app["Customer app<br/>+ trimset Python SDK"]
  cli["trimset CLI<br/>[Python, on a laptop]"]
  up["Upstream model"]
  cand["Shadow candidate model"]
  src["Outcome sources"]
  ch["Alert channels"]

  subgraph sys["trimset deployment"]
    proxy["proxy<br/>[Rust, ADR 0015]<br/>front door: forwards, decision ids,<br/>passes the rest to the server"]
    server["server<br/>[Python, FastAPI]<br/>ingest · records from the proxy · outcomes ·<br/>policies · admin · review · reports"]
    worker["worker<br/>[Python]<br/>scheduler · joiner · refit and compiler ·<br/>drift and health · alerts · shadow reports · retention"]
    db[("Postgres 18<br/>decisions partitioned by month,<br/>outcomes, fits, policies, jobs")]
    dash["dashboard<br/>[v2, static web app]"]
  end

  app -- "proxy: /p/key/v1/systemone, /v1/decisions" --> proxy
  app -- "log(), outcome(), GET /v1/policies" --> proxy
  proxy -- "forward" --> up
  proxy -- "every other path; keys and exchanges (internal)" --> server
  server -. "sampled copy, after the response" .-> cand
  src -- "POST /v1/outcomes" --> server
  cli -- "admin and review API" --> server
  dash -. "same API (v2)" .-> server
  server -- "SQL" --> db
  worker -- "SQL, jobs with SKIP LOCKED" --> db
  worker -- "alerts" --> ch

The brief listed more containers. ADR 0004 shows which process each one runs in.

4. C4 level 3: components¶

Server¶

flowchart LR
  subgraph server["server process"]
    auth["Key auth<br/>scope check, 60 s cache"]
    ingest["Ingest handler<br/>validate · question_id · hash"]
    proxy["Proxy handler<br/>forward · parse · decision ids"]
    buf["Record buffer<br/>bounded, drop-oldest"]
    writer["Batch writer<br/>INSERT … ON CONFLICT DO NOTHING"]
    outc["Outcomes handler<br/>match or hold"]
    pol["Policy API<br/>bundle + ETag"]
    admin["Admin API<br/>projects · keys · settings · lifecycle · audit"]
    sim["Reports and simulator<br/>reads fit artifacts"]
    revq["Review API<br/>claim · label · skip"]
    shadow["Shadow dispatcher<br/>sampled, capped, after the response"]
    mv["Model-version watcher<br/>new served version → job"]
  end
  db[("Postgres")]
  up["Upstream"]
  auth --> ingest & proxy & outc & pol & admin & sim & revq
  ingest --> writer
  proxy --> up
  proxy --> buf --> writer --> db
  proxy --> shadow
  ingest & proxy --> mv
  mv & outc & pol & admin & sim & revq --> db

Worker¶

flowchart LR
  subgraph worker["worker process"]
    sched["Scheduler<br/>due schedules → jobs"]
    runner["Job runner<br/>SKIP LOCKED, retries"]
    join["Outcome joiner<br/>held → matched · rule labels · shadow labels"]
    refit["Refit<br/>dataset · splits · calibration"]
    comp["Policy compiler<br/>Learn-then-Test · cost · feasibility"]
    drift["Drift and health<br/>PSI · JS · permutation · live checks"]
    shrep["Shadow reports<br/>McNemar"]
    alert["Alert dispatcher<br/>de-duplicate · cool-down · channels"]
    maint["Maintenance<br/>partitions · retention · deletions · leases"]
  end
  stats["trimset-stats<br/>library, no I/O"]
  db[("Postgres")]
  ch["Alert channels"]
  sched --> db
  runner --> join & refit & drift & shrep & alert & maint
  refit --> comp
  refit & comp & drift & shrep --> stats
  join & refit & comp & drift & shrep & maint --> db
  alert --> ch

Where the Phase 1 components live¶

5. Sequence diagrams¶

(a) Proxy request with logging¶

sequenceDiagram
  autonumber
  participant App
  participant P as trimset server (proxy)
  participant U as Upstream (Jev / Ollaya)
  participant B as Record buffer
  participant DB as Postgres
  App->>P: POST /p/{proxy key}/v1/systemone (state, questions)
  P->>P: check key scope, find project and upstream
  P->>U: same request, the app's Authorization passed through
  U-->>P: 200 answers (probabilities, model version)
  P-->>App: 200 unchanged body + X-Trimset-Decision-Id
  P->>B: one record per question (state hashed, never stored)
  loop every 50 ms or 500 records
    B->>DB: INSERT batch ON CONFLICT DO NOTHING
  end
  alt Postgres down
    B->>B: hold up to the limit, then drop oldest and count
  end
  opt first time this served version is seen
    P->>DB: record model version, queue silent-upgrade check
  end

(b) Late outcome join¶

sequenceDiagram
  autonumber
  participant S as Outcome source (payments webhook)
  participant API as trimset server
  participant DB as Postgres
  participant W as worker
  S->>API: POST /v1/outcomes (external_ref rf-1001, refund_fraud, true, downstream)
  API->>DB: find decisions by project, external_ref, question
  alt decision found
    API->>DB: upsert current outcome, append history
    API-->>S: 200 matched
  else decision not logged yet
    API->>DB: hold as unmatched (expires in 30 days)
    API-->>S: 202 held
  end
  loop every minute
    W->>DB: match held outcomes to decisions that have arrived
  end
  loop daily
    W->>DB: rule labels for matured decisions with no outcome
    W->>DB: copy labels to linked shadow decisions
  end

(c) Scheduled refit → proposal → approval → SDK pickup¶

sequenceDiagram
  autonumber
  participant W as worker
  participant DB as Postgres
  participant Ch as Slack
  actor A as Approver (CLI)
  participant API as trimset server
  participant SDK as SDK in the app
  W->>DB: claim scheduled refit job
  W->>DB: read matured labeled decisions in the window
  W->>W: split fit / tune / eval, fit calibration, compile (Learn-then-Test)
  W->>DB: insert fit and proposed policy with a diff against the active one
  W->>Ch: proposal ready
  A->>API: policy show (diff, guarantee, metrics)
  A->>API: approve and activate (admin key, reason)
  API->>DB: proposed → approved → active, old active → retired, events and audit
  loop every 30 s ± jitter
    SDK->>API: GET /v1/policies (If-None-Match)
    API-->>SDK: 200 new bundle, or 304
  end
  SDK->>SDK: verify fingerprints, swap the cache
  Note over SDK: gate() now answers with the new version

(d) Drift detection → alert¶

sequenceDiagram
  autonumber
  participant W as worker
  participant DB as Postgres
  participant Ch as Alert channels
  W->>DB: claim hourly drift job
  W->>DB: probabilities for the current and reference windows
  W->>W: PSI, JS divergence, permutation p-value
  alt effect ≥ threshold and p below α and n ≥ minimum
    W->>DB: drift report (triggered), open or update the alert
    opt not in cool-down
      W->>Ch: drift alert with the evidence
    end
    Note over DB: the active policy's guarantee now reads "suspended"
  else no drift
    W->>DB: drift report (not triggered)
    opt an alert was open
      W->>DB: resolve the alert
      W->>Ch: resolved
    end
  end

(e) Shadow comparison¶

sequenceDiagram
  autonumber
  participant App
  participant P as trimset server
  participant U as Primary model
  participant C as Candidate model
  participant DB as Postgres
  actor E as ML engineer (CLI)
  App->>P: decision request
  P->>U: forward
  U-->>P: answers
  P-->>App: primary answers only
  opt sampled (10%) and under the daily cap
    P->>C: same request, candidate model
    C-->>P: answers
    P->>DB: shadow records linked to the primary decision
  end
  Note over DB: outcomes for the primary are copied to its shadows
  E->>P: shadow report
  P->>DB: paired decisions and labels
  P-->>E: agreement, flip rate, McNemar p, cost and latency deltas

6. Data model¶

erDiagram
  projects ||--o{ api_keys : has
  projects ||--o{ questions : has
  projects ||--o{ model_versions : sees
  questions ||--o{ decisions : answered_in
  decisions ||--o| outcomes : current_label
  decisions ||--o{ outcome_history : label_history
  decisions ||--o| decision_states : raw_state_if_opted_in
  projects ||--o{ unmatched_outcomes : holds
  questions ||--o{ fits : calibrated_by
  fits ||--o{ policies : compiled_into
  policies ||--o{ policy_events : transitions
  questions ||--o{ drift_reports : checked_by
  projects ||--o{ alert_rules : configures
  alert_rules ||--o{ alerts : raises
  decisions ||--o| review_items : queued_as
  projects ||--o{ audit_log : records
  projects ||--o{ schedules : runs
  schedules ||--o{ jobs : creates

  projects {
    uuid project_id PK
    text name
    jsonb settings "upstream, retention, privacy, env-var secret refs"
  }
  api_keys {
    uuid key_id PK
    uuid project_id FK
    text scope "proxy, ingest, review or admin"
    text key_hash
    timestamptz revoked_at
    timestamptz last_used_at
  }
  questions {
    text question_id PK
    uuid project_id FK
    text name "stable alias, e.g. refund_fraud"
    text type "noul, choice or score"
    jsonb definition "text, options or levels"
    jsonb settings "actions, targets, rules, audit share, link template"
  }
  model_versions {
    uuid project_id FK
    text model_requested
    text model_served
    timestamptz first_seen
  }
  decisions {
    uuid decision_id PK
    timestamptz ts PK "partition key, monthly"
    uuid project_id PK
    text question_id FK
    text model_requested
    text model_served
    real_array probs
    int latency_ms
    numeric cost_usd
    int policy_version
    text action_taken
    bool audit
    jsonb tags
    text external_ref
    text state_hash "HMAC-SHA-256"
    text role "primary or shadow"
    uuid shadow_of
  }
  decision_states {
    uuid decision_id PK
    timestamptz ts "partition key"
    jsonb state "only if the project opted in"
  }
  outcomes {
    uuid decision_id PK
    text label
    text source "human_review, downstream or rule"
    timestamptz observed_at
    uuid key_id
  }
  outcome_history {
    bigint id PK
    uuid decision_id FK
    text label
    text source
    timestamptz received_at
    uuid key_id
  }
  unmatched_outcomes {
    bigint id PK
    text decision_ref "decision_id or external_ref"
    text label
    timestamptz expires_at
  }
  fits {
    uuid fit_id PK
    text question_id FK
    text model_served
    text method
    jsonb params
    jsonb metrics "raw and calibrated, eval split"
    jsonb reliability
    bytea eval_artifact "scores and labels, no ids"
    jsonb provenance "window, split sizes, seed"
  }
  policies {
    uuid policy_id PK
    text question_id FK
    text model_served
    int version
    text state
    jsonb body "immutable: rules, calibration map, guarantee"
    text fingerprint "SHA-256 of canonical body"
    uuid fit_id FK
  }
  policy_events {
    bigint id PK
    uuid policy_id FK
    text from_state
    text to_state
    uuid key_id
    text reason
  }
  drift_reports {
    uuid report_id PK
    text question_id FK
    text model_served
    text kind
    jsonb stats "PSI, JS, p, window sizes"
    bool triggered
  }
  alert_rules {
    uuid rule_id PK
    text kind
    jsonb params
    jsonb channels "env-var refs only"
  }
  alerts {
    uuid alert_id PK
    uuid rule_id FK
    text status "open, acknowledged or resolved"
    text dedupe_key
    timestamptz last_sent_at
  }
  review_items {
    uuid item_id PK
    uuid decision_id FK
    text state
    real priority
    uuid claimed_by
    timestamptz lease_until
  }
  audit_log {
    bigint id PK
    uuid key_id
    text action
    jsonb before_value
    jsonb after_value
  }
  schedules {
    uuid schedule_id PK
    text kind
    text cron
    timestamptz next_run_at
  }
  jobs {
    bigint job_id PK
    text kind
    jsonb payload
    timestamptz run_at
    int attempts
    text last_error
  }

Shadow decisions point to their primary through decisions.shadow_of. Every table carries project_id, and every query filters by it (FR-51).

7. Lifecycles¶

Policy¶

stateDiagram-v2
  [*] --> draft: compiled by hand
  [*] --> proposed: scheduled refit
  draft --> proposed: submit
  proposed --> approved: approve (admin)
  proposed --> rejected: reject (admin, reason), or a newer refit's proposal
  approved --> active: activate (admin)
  active --> retired: retire (admin, reason), or another version activated
  retired --> active: rollback (admin, reason; the version active before)
  rejected --> [*]

Review item¶

stateDiagram-v2
  [*] --> queued: escalated or audited decision
  queued --> claimed: claim (15-minute lease)
  claimed --> queued: lease expired, released or skipped
  claimed --> labeled: label (becomes an outcome)
  queued --> closed: outcome arrived elsewhere, or data deleted
  labeled --> [*]
  closed --> [*]

8. Deployment¶

flowchart TB
  subgraph vm["One VM (2 vCPU, 4 GB): docker compose"]
    rp["reverse proxy (optional)<br/>Caddy or nginx, TLS"]
    px["proxy<br/>Rust, :8800"]
    s1["server<br/>uvicorn, one process per core<br/>:8810, internal"]
    w1["worker"]
    pg[("postgres:18<br/>volume pgdata")]
  end
  apps["Apps + SDKs"] --> rp --> px --> s1
  s1 --> pg
  w1 --> pg
  px --> up["Upstream: Jev on the internet,<br/>or Ollaya on the host at :11435"]
  w1 --> ch["Slack · SMTP · webhooks"]
  s1 & w1 -. OTLP .-> otel["OTel collector (optional)"]
  bk["Backups: nightly pg_dump<br/>or volume snapshots"] -.-> pg

Scaling, in order of need: 1. More servers: behind a load balancer. They are stateless apart from the record buffer. 2. A second worker: jobs use SKIP LOCKED, so two are safe. 3. Postgres: monthly partitions and retention keep it small. A read replica can serve reports and the simulator. 4. ClickHouse: past about 1 billion decisions, or for heavy analytics, fed by logical replication. This needs no API change.

9. Data flows and trust boundaries¶

flowchart LR
  subgraph z1["Zone 1: app network"]
    app["App + SDK"]
  end
  subgraph z2["Zone 2: trimset deployment"]
    srv["server"]
    wrk["worker"]
    db[("Postgres")]
  end
  subgraph z3["Zone 3: model providers"]
    up["Upstream or candidate model"]
  end
  subgraph z4["Zone 4: third-party systems"]
    src["Outcome sources"]
  end
  subgraph z5["Zone 5: alert channels"]
    ch["Slack · SMTP · webhooks"]
  end
  subgraph z6["Zone 6: people"]
    cli["CLI with admin or reviewer keys"]
  end
  app -- "F1 state + questions; proxy key" --> srv
  srv -- "F2 state + questions; the app's upstream key" --> up
  up -- "F3 probabilities" --> srv
  app -- "F4 records: probabilities, hashes, tags; ingest key" --> srv
  srv -- "F5 policy bundle + state-hash key" --> app
  src -- "F6 outcomes; ingest key" --> srv
  cli -- "F7 approvals, labels, settings; admin key" --> srv
  srv --> db
  wrk --> db
  wrk -- "F8 alerts: names and numbers, never state" --> ch

10. Threat model (STRIDE)¶

11. Changes to Phase 1 that come from this phase¶

• FR-01 / record: state_sha256 becomes state_hash, with state_hash_alg: "hmac-sha256/1" (ADR 0009).
• FR-07: X-Trimset-Decision-Id carries one id per question, comma-separated in question order (ADR 0011).
• FR-52: a third key scope, proxy (ADR 0011).
• Homepage: the adoption snippet needs the /p/<proxy key> path. This gets fixed with the other homepage claims before Phase 7.