Phase 1: Requirements¶

Status: approved 2026-11-10, with the Phase 2 refinements noted on FR-01, FR-07, FR-09 and FR-52 · Inputs: kickoff.md, 00-discovery.md

1. Vision¶

Decision models answer typed questions with probabilities in milliseconds. Teams turn those probabilities into actions with thresholds they guess once and never revisit. Meanwhile three things go wrong: - the model is upgraded behind an alias; - the traffic shifts; - the truth about each decision (a chargeback, a reassigned ticket, a moderator's call) arrives hours to months later in some other system, and is never joined back to the decision.

trimset closes that loop, self-hosted and open source. It: - captures every decision; - joins each one to its real outcome when that arrives; - refits calibration and thresholds; - proposes a new policy that says what it guarantees, and puts it live only after a human approves; - alerts when the model or the traffic drifts.

Adopting it takes one base-URL change.

In one line: Langfuse for decision models: thresholds that stay true.

Release scope¶

2. Glossary¶

3. Personas¶

4. Use cases: real-world examples¶

The ★ ones get runnable demos in examples/ (Phase 5) and pages on the site. All six are scenarios in the synthetic data generator (FR-60).

UC-1 ★ Refund fraud¶

An online shop pays refunds automatically. Before paying one, its refund service asks a decision model about the request.

• Question: "Is this refund request fraudulent?" (noul). external_ref is the refund id.
• Actions:
• p < lo → ACCEPT (pay out)
• lo ≤ p < hi → ESCALATE (risk team)
• p ≥ hi → BLOCK (hold the refund)
• Outcomes:
• a chargeback or a confirmed fraud case, 30–90 days later (downstream);
• the risk team's verdict on escalated refunds (human_review);
• no chargeback within 120 days → "not fraud" (rule).
• Target: false-pass rate ≤ 2% with 95% confidence, and at most 15% escalated.
• Why it's hard: labels arrive months late, and "no news" is itself a label. Held refunds almost never get a true label, so 1% audit sampling keeps the refits honest.
• Exercises: FR-04, FR-10, FR-13, FR-14, FR-21, FR-23, FR-33, FR-37.

UC-2 ★ Support ticket triage¶

An online shop routes incoming tickets to one of four teams.

• Question: "Which team should handle this ticket?" (choice: billing, shipping, returns, tech). external_ref is the ticket id.
• Actions:
• the top option's probability ≥ t → SELECT:<team>
• otherwise → ESCALATE (a human dispatcher)
• Outcomes:
• the team that finally closed the ticket, from the help desk's webhook, hours later (downstream);
• no reassignment within 72 hours → the first routing was right (rule).
• Target: misroutes ≤ 5% among auto-routed tickets, with 95% confidence.
• Why it's hard: jev-latest moving to a new version changes the probabilities overnight.
• Exercises: FR-05, FR-20, FR-31, FR-39.

UC-3 ★ Model routing¶

A chat product sends easy requests to a small model and hard ones to a large model.

• Question: "How hard is this request?" (score, levels 0–3).
• Actions:
• P(level ≥ 2) < t → ACCEPT (the small model answers)
• otherwise → ESCALATE (the large model answers)
• Outcomes:
• an LLM judge grades a 2% sample of answers with the true level (downstream);
• a thumbs-down or a retry within 10 minutes of a small-model answer means the request was harder than predicted (downstream).
• Target: cost model. A large-model call costs $0.012 and a bad small-model answer costs $0.20 (illustrative). Minimise the expected cost per 1,000 requests.
• Why it's hard: a cost target gives the cheapest expected point, not a guarantee, and the policy says so (FR-25).
• Exercises: FR-21, FR-25, FR-26, FR-45.

UC-4 PII guardrail¶

A company's internal chat assistant checks every prompt before it reaches an LLM.

• Question: "Does this text contain personal data?" (noul).
• Actions:
• p ≥ hi → BLOCK
• band → ESCALATE (redact, or send to a human)
• p < lo → ACCEPT
• Outcomes:
• the privacy team reviews a weekly sample of passed prompts (audit sampling, human_review);
• the company's data-loss scanner flags leaks (downstream).
• Target: false-pass rate ≤ 1% with 95% confidence.
• Why it's hard: the raw prompts are exactly what must not be stored. Reviewers open the prompt in the company's own tool through a link template (FR-50).
• Exercises: FR-09, FR-14, FR-50.

UC-5 Agent tool approval¶

An operations agent wants to run a database write or send an email.

• Question: "Is this tool call safe to run without a human?" (noul; p is the probability that it is safe).
• Actions:
• p ≥ hi → ACCEPT (run it)
• band → ESCALATE (ask a human)
• p < lo → BLOCK
• Outcomes:
• the human's approve or deny on escalated calls (human_review);
• incident reports on calls that ran, rare and late (downstream).
• Target: unsafe passes ≤ 0.5% with 95% confidence.
• Why it's hard: bad events are rare. Claiming ≤ 0.5% at 95% confidence needs about 600 labeled auto-run calls with no incident (exact binomial bound; the "rule of three" gives 3/n). Before that, trimset says "not enough labels: 412 more needed" instead of guessing (FR-23).
• Exercises: FR-20, FR-23, FR-35.

UC-6 Content moderation¶

A marketplace checks new listings before they go live.

• Question: "How serious is the problem with this listing?" (score: none, mild, serious, remove).
• Actions:
• "remove" → BLOCK
• mild or serious → ESCALATE (the moderator queue)
• none → ACCEPT

Cut points are set on the cumulative probabilities. - Outcomes: - the moderator's decision (human_review); - user reports on live listings, days later (downstream). - Target: at most 2,000 items a day in the review queue, with the fewest errors under that cap. - Why it's hard: the binding constraint is people's time, not an error rate. - Exercises: FR-21, FR-24, FR-47.

5. User journeys¶

J1 · Adopt (app developer) 1. docker compose up -d. 2. trimset admin create-key, then create a project, an ingest key and the upstream (for example a local Ollaya). 3. Point the app's TYPESAFE_BASE_URL at trimset. No code changes. 4. Watch decisions arrive: trimset decisions tail. 5. To link outcomes later, read the X-Trimset-Decision-Id response header, or send your own id in X-Trimset-External-Ref.

J2 · Close the loop (app developer) 1. Point the help desk's or payment processor's webhook at POST /v1/outcomes. 2. Add a default-by-rule ("no chargeback within 120 days → not fraud"). 3. Check coverage: trimset outcomes coverage refund_fraud shows the share of decisions with a label, for each action path.

J3 · First policy (ML engineer, then approver) 1. trimset calibrate refund_fraud shows the raw and calibrated metrics and a reliability table. 2. trimset simulate refund_fraud --lo 0.30 --hi 0.85 shows the trade-off. 3. trimset policy compile refund_fraud --false-pass 0.02 --confidence 0.95 --max-escalation 0.15 creates a proposal. 4. The approver reads the diff and the guarantee: trimset policy show. 5. The approver approves and activates it. 6. SDKs pick it up within 60 seconds, and the app calls gate().

J4 · Drift (ML engineer) 1. A Slack alert says the probability distribution shifted, with the evidence. 2. trimset drift show refund_fraud gives the details. 3. Then one of three things: wait for a refit proposal, roll back, or pin the model version upstream.

J5 · Silent upgrade (ML engineer) 1. An alert says jev-latest now answers as jev-1.14.0. 2. The active policy no longer applies (it is bound to jev-1.13.0), so the SDK returns its configured default. 3. Labels are collected for the new version. 4. A new policy is compiled for it and approved.

J6 · Try a cheaper model (ML engineer) 1. Set laya:latest on Ollaya as the shadow model for 20% of traffic. 2. Wait for labeled pairs. 3. trimset shadow report shows the comparison. 4. Switch the upstream, then compile and approve a policy for the new model.

J7 · Review (reviewer) 1. Run trimset review. 2. Claim an item and see its question, probabilities, tags and a link into the app. 3. Label it with one key, then move to the next.

J8 · Operate (operator) - Back up Postgres. - Upgrade: pull the new images, then migrate. - Rotate keys. - Handle a deletion request: trimset data delete --external-ref rf-1001.

6. Diagrams¶

Use cases¶

flowchart LR
  dev["App developer"]
  app["App"]
  ml["ML / platform engineer"]
  appr["Approver"]
  rev["Reviewer"]
  ops["Operator"]
  up[("Upstream model<br/>Jev · Ollaya · Ollama · OpenAI")]

  subgraph ts["trimset"]
    uc1(["Proxy a decision call"])
    uc2(["Log decisions through the SDK"])
    uc3(["Send outcomes"])
    uc4(["Gate locally with the active policy"])
    uc5(["Calibrate and simulate thresholds"])
    uc6(["Compile and propose a policy"])
    uc7(["Approve, activate, roll back"])
    uc8(["Label escalated items"])
    uc9(["Get drift and health alerts"])
    uc10(["Compare a shadow model"])
    uc11(["Install, upgrade, delete data"])
  end

  dev --> app
  app --> uc1 --> up
  app --> uc2
  app --> uc3
  app --> uc4
  ml --> uc5
  ml --> uc6
  ml --> uc9
  ml --> uc10
  appr --> uc7
  rev --> uc8
  ops --> uc11

The loop¶

flowchart LR
  A["Capture<br/>proxy or SDK"] --> B["Join outcomes<br/>hours to months later"]
  B --> C["Refit<br/>calibration and thresholds"]
  C --> D{"Human approves?"}
  D -- yes --> E["Enforce<br/>local SDK gate"]
  D -- no --> C
  E --> A
  A --> F["Monitor<br/>drift, upgrades, health"]
  F -- alert --> C

7. User stories¶

US-01 · App developer · Adopt with one base URL (FR-04, FR-07, FR-08) As an app developer, I want my decision calls to go through trimset by changing only the base URL, so I get logging without touching code. - Given a project whose upstream is a local Ollaya, when my app sends its unchanged POST /v1/systemone to trimset, then it gets Ollaya's status and body back unchanged, plus an X-Trimset-Decision-Id header. - Given that call, then one decision record per question is stored within 5 seconds. - Given Postgres is down, when my app calls the proxy, then the call still succeeds and the dropped-records counter goes up.

US-02 · App developer · Log without risk (FR-01, FR-36) As an app developer, I want log() to never slow down or break my service. - Given the trimset server is unreachable, when I call log(), then it returns immediately without raising, and the record is retried with backoff. - Given the buffer is full, when I call log(), then the oldest record is dropped and counted, and memory stays within its limit.

US-03 · App developer · Send an outcome months later (FR-10, FR-11) - Given a decision logged 45 days ago with external_ref rf-1001, when I post an outcome for rf-1001 with label true and source downstream, then it is joined to that decision and is used by the next refit. - Given an outcome arrives before its decision, when the decision arrives within 30 days, then the two are joined.

US-04 · App developer · Correct a label (FR-12) - Given an outcome false, when a later outcome says true, then the current label is true, and the history shows both with their sources and times.

US-05 · App developer · No news is a label (FR-13) - Given the rule "no chargeback within 120 days → false", when 120 days pass with no outcome, then the label false is recorded with source rule. - Given a real outcome arrives after that, then it overrides the rule's label.

US-06 · ML engineer · See calibration (FR-15 to FR-18) - Given 1,000 or more labeled decisions for refund_fraud on jev-1.13.0, when I run trimset calibrate refund_fraud, then I see log loss, Brier and ECE (equal-width and equal-mass), raw against calibrated, on the evaluation split, plus a reliability table. - And no fit is ever evaluated on the items it was fitted on.

US-07 · ML engineer · Too few labels (FR-19, FR-23) - Given 43 labeled decisions, when I calibrate or compile, then trimset declines with "43 labeled items; at least 200 needed" (the minimum is configurable).

US-08 · ML engineer · Policy with a guarantee (FR-20 to FR-23, FR-25) - Given a fit, when I ask for a false-pass rate ≤ 2% with 95% confidence and at most 15% escalated, then I get a proposed policy with its thresholds, fingerprint, metrics with exact 95% intervals on the evaluation split, and a guarantee statement that names its assumptions and sample sizes.

US-09 · ML engineer · Impossible targets (FR-24) - Given targets that can't all be met, when I compile, then the result says "infeasible", names the constraint that binds, shows the best achievable trade-off, and creates no policy.

US-10 · ML engineer · Try thresholds (FR-26) - When I run trimset simulate refund_fraud --lo 0.30 --hi 0.85, then I see the share automated, the error rate on the automated path with its 95% interval, the share escalated, and the expected cost.

US-11 · Approver · Approve with a trail (FR-27, FR-28, FR-54) - Given a proposed policy, when an admin key approves and activates it, then the previous active version is retired, and an event records who did it, when, and why. - Given an ingest-only key, when it tries to approve, then the request is refused (403).

US-12 · Approver · Roll back (FR-29, NFR-05) - Given v8 is active and v7 came before it, when I run trimset policy rollback refund_fraud, then v7 is active in every running SDK within 60 seconds.

US-13 · ML engineer · Refits only propose (FR-30) - Given a daily refit schedule, when a refit finds a better policy, then a proposal with a diff against the active policy exists and a "proposal ready" alert is sent. Nothing is activated.

US-14 · App developer · Gate locally (FR-32, FR-33, NFR-04) - Given an active policy for refund_fraud on jev-1.13.0 is cached, when I call gate("refund_fraud", p=0.83, model="jev-1.13.0"), then it returns ESCALATE with policy v7, in under 1 ms at p99, with no network call.

US-15 · App developer · Fail open (FR-35) - Given the server is unreachable and nothing is cached, when I call gate(), then it returns my configured default action with source default, and doesn't raise.

US-16 · App developer · Tampered policy (FR-34) - Given a fetched policy whose fingerprint doesn't match its content, then the SDK rejects it, keeps the last good policy, and reports an error.

US-17 · App developer · Wrong model (FR-31) - Given the active policy is for jev-1.13.0, when gate() is called with model_served = jev-1.14.0, then that policy is not applied, the default is returned, and the reason says "no policy for this model".

US-18 · ML engineer · Drift that matters (FR-38, FR-43) - Given a 7-day reference window, when the probability distribution shifts (PSI ≥ 0.2, permutation-test p < 0.01, at least 500 decisions in the window), then one Slack alert is sent, with the report. - Given a window of only 30 decisions with a similar PSI, then no alert is sent.

US-19 · ML engineer · Silent upgrade (FR-39) - Given jev-latest was answering as jev-1.13.0, when it starts answering as jev-1.14.0, then within 5 minutes an alert names both versions and says the active policy no longer applies.

US-20 · ML engineer · Compare a shadow model (FR-45, FR-46) - Given laya:latest shadows 20% of traffic, when there are 1,000 paired decisions of which 300 are labeled, then the report shows agreement, flip rate, the McNemar p-value, and cost and latency deltas. - And the app only ever received the primary model's answers, with its p99 latency unchanged (within 1 ms).

US-21 · Reviewer · Label without seeing private data (FR-47 to FR-50) - Given an escalated refund, in a project that stores no raw state, when I claim it, then I see the question, the probabilities, the tags and a link to the shop's own refund page built from external_ref. - When I label it "fraudulent", then an outcome is recorded with source human_review and my key's id.

US-22 · Operator · Install (FR-53, NFR-01) - Given a clean Linux VM with Docker, when I run docker compose up -d, then /ready answers 200 within 2 minutes of the images being pulled, and trimset admin create-key prints the first admin key. There are no default passwords.

US-23 · Operator · Upgrade (NFR-15) - Given v0.1.0 holding data, when I upgrade to v0.1.1, then migrations run once and all data is intact.

US-24 · Operator · Delete a customer's data (FR-57, NFR-14) - When I delete by external_ref, then within 1 hour its decisions, outcomes, review items and raw state are gone, and no fit or policy keeps item-level data from it.

US-25 · Operator · Revoke a leaked key (FR-52) - Given a revoked key, when it is used, then it is refused within 60 seconds of revocation.

8. Functional requirements¶

Each requirement lists its release and milestone. M6 (dashboard) moved to v2 in Phase 0.

A. Capture¶

• FR-01 Decision logging API (v0.1, M1). POST /v1/decisions:log takes one record or a batch (Phase 3 fixes the limits; the starting point is 1,000 records or 1 MB). The fields are as in the brief:
• decision_id, project_id, timestamp
• question_id, type, options or levels
• model_requested, model_served
• the raw probabilities, latency_ms, cost_usd
• policy_version, action_taken, tags, external_ref, state_hash (the brief's state_sha256, now keyed: ADR 0009)

Writes are idempotent by decision_id: sending the same record again stores it once. The reply gives a status for each record. - FR-02 Validation (v0.1, M1). Probabilities must lie in [0, 1]. Choice and score probabilities must sum to 1 within 1e-3, then are renormalised before use (Phase 3: vendors round to 4 decimals). Their count must match the question's options or levels. Valid records in a batch are stored; invalid ones come back with reasons. - FR-03 Questions (v0.1, M1). - The server computes question_id from the type, the exact wording, and the options or levels in order (canonical form and test vectors in Phase 3). - A client-sent id must match it, or the record is rejected. - A question is registered the first time it is seen. - FR-04 Proxy for /v1/systemone (v0.1, M1). - Forwards to the project's upstream: TypeSafe's Jev, Ollaya, or Ollama 0.35.0 and later (0.40.5 for cloud models; corrected in Phase 3). - Returns the upstream's status and body unchanged. - Logs one decision record per question asynchronously. A logging failure never fails the request. - FR-05 Proxy for /v1/decisions (v0.1, M1). The same behaviour for OpenAI's decisions format. The format follows the project's upstream setting, not the path: Ollaya serves the systemone format at /v1/decisions too (Phase 3). - FR-06 Upstream settings (v0.1, M1). - Each project sets its upstream base URL and timeout. - Upstream credentials are either passed through from the app's request (never stored) or held server-side. - The upstream can be any address the operator configures, including localhost (Ollaya listens on 11435). - FR-07 What the proxy records (v0.1, M1). - From the request and response: model_requested, model_served and latency_ms. - cost_usd, from token usage and a per-model price table in the configuration. Prices are configuration, never code; an unknown model gives an empty cost. - state_sha256. - Optional request headers set X-Trimset-External-Ref and X-Trimset-Tags. The response gains an X-Trimset-Decision-Id header, with one id per question, comma-separated in question order (ADR 0011). - FR-08 One-base-URL adoption (v0.1, M1). In proxy mode, trimset must know the project without the app adding headers or changing its body. Phase 2 picks the mechanism; it weighs that a key inside a URL can end up in logs. - FR-09 Private by default (v0.1, M1). - Raw state is never stored unless the project switches on "store raw state" (an audited setting, with its own shorter retention). - Only hashes and tags are kept. - Tags have size limits and an optional per-project allow-list of keys, so personal data can't slip in through tags. - state_hash is a keyed hash (HMAC-SHA-256, with the key derived from a server secret held outside the database): ADR 0009.

B. Outcomes¶

• FR-10 Outcomes API (v0.1, M1). POST /v1/outcomes takes one outcome or a batch.
• It names a decision by decision_id, or by external_ref plus question_id.
• The label is typed: true/false for noul, an option for choice, a level for score.
• It carries a source (human_review, downstream or rule) and observed_at.
• It is idempotent, and its flat JSON is easy to send from a webhook.
• FR-11 Late and early outcomes (v0.1, M1).
• An outcome can arrive at any time within the retention period.
• One that arrives before its decision is held as unmatched (30 days by default) and joined when the decision arrives.
• When held outcomes expire unmatched, they are counted.
• FR-12 Corrections (v0.1, M1). The last outcome received wins. The full history (label, source, key, time) is kept.
• FR-13 Default labels by rule (v0.1, M1).
• A question can set "if no outcome within D days, record label L with source rule".
• A real outcome that arrives later overrides it.
• FR-14 Label coverage and audit sampling (v0.1, M2 for coverage, M3 for sampling).
• trimset reports, for each group and action path, the share of decisions with a label and the median delay.
• It warns when an action path has almost no labels: for example, blocked refunds never get a chargeback.
• A policy can set an audit fraction (0 by default). The SDK sends that random share to ESCALATE whatever the probability, and flags it audit (FR-37).
• The compiler uses audited items to estimate errors on paths that otherwise lack labels. This addresses the selective-labels problem (research source to be verified in Phase 2).

C. Calibration¶

• FR-15 Never pooled (v0.1, M2). Fits are made separately for each group.
• FR-16 Methods (v0.1, M2).
• noul: temperature scaling, Platt scaling and isotonic regression.
• choice and score: at least temperature scaling. Phase 2 research decides the others.
• The method is chosen on the calibration data and reported on the evaluation split.
• FR-17 Metrics (v0.1, M2).
• Log loss, Brier score, and ECE both equal-width and equal-mass (15 bins by default, configurable).
• Reliability-diagram data: for each bin, the count, the mean predicted value, the observed frequency and its interval.
• All raw against calibrated.
• FR-18 Strict splits (v0.1, M2).
• Labeled items are split three ways (fit, tune, eval), assigned by a hash of decision_id.
• Every fit records which items it used.
• trimset refuses to tune or evaluate on items a fit was trained on.
• FR-19 Windows and minimums (v0.1, M2).
• Fits use a time window (the last 90 days of labeled items by default) and need a minimum number of labeled items (200 by default).
• Below the minimum, trimset says how many more are needed.

D. Policy compiler¶

• FR-20 Action mapping (v0.1, M2). Each question says how probabilities become actions:
• noul: which side means BLOCK and which means ACCEPT; the band [lo, hi) means ESCALATE.
• choice: SELECT:<option> when the top option's probability reaches t (optionally a separate t per option); otherwise ESCALATE. Options can map to BLOCK (for example "spam").
• score: cut points map levels to actions.
• FR-21 Targets (v0.1, M2). A target is one of:
• an error budget or a false-pass cap, with a confidence level (95% by default) and an optional escalation cap; or
• a cost model (the cost of an escalation and the cost of each kind of wrong action), optionally with an escalation cap.

Several budgets at once comes in v0.2. - FR-22 Output (v0.1, M2). A policy is versioned per group (v1, v2, …) and uses schema policy/1. It carries: - a SHA-256 fingerprint over its canonical JSON; - metrics on the tune and eval splits: share automated, share escalated, error or false-pass rate with its exact 95% interval, and expected cost; - where it came from: the fit, the time window, and the item count per split. - FR-23 Small samples (v0.1, M2). - Guarantees use exact bounds (Clopper-Pearson, or the method Phase 2 picks), never point estimates. - When there are too few labeled items to support the target, the compiler declines and says how many more it needs. For example, ≤ 0.5% at 95% needs about 600 error-free items. - FR-24 Impossible targets (v0.1, M2). trimset says which constraint binds and shows the best achievable values. It creates no policy. - FR-25 Honest guarantees (v0.1, M2). - Every policy carries guarantee: either the statement, its method, its assumptions and its item counts, or none with the reason. Reasons include a cost-only target, or drift detected since the tuning window. - Where assumptions are known to break, trimset uses the adaptive methods from Phase 2 research and labels the result "adaptive, no guarantee". - It never shows a guarantee it can't back. - FR-26 Simulator (v0.1, M2). - For a group and some candidate thresholds (or a grid of them), it returns the FR-22 metrics on the eval split. - It is used by trimset simulate, the homepage's trade-off table (with numbers from demo data) and the v2 dashboard.

E. Policy lifecycle¶

• FR-27 States (v0.1, M3). A policy moves draft → proposed → approved → active → retired, or proposed → rejected. Every transition is an event recording the key, the time and the reason.
• FR-28 Who approves (v0.1, M3).
• Only admin keys can approve, activate or roll back.
• Rejecting or rolling back requires a reason.
• FR-29 One active policy per group (v0.1, M3).
• Activating a version retires the previous one.
• Rollback re-activates the previous approved version in one command.
• FR-30 Scheduled refits only propose (v0.1, M3).
• Refits run on a schedule per project (daily by default).
• A refit produces a proposal with a diff against the active policy, plus a "proposal ready" alert.
• Refits never activate anything in v0.x.
• FR-31 Bound to its model (v0.1, M3). A policy applies only to its own (question_id, model_served). It is never applied to another model version or another wording.

F. Runtime gate (Python SDK)¶

• FR-32 Fetch and cache (v0.1, M3).
• The SDK pulls the project's active policies every 30 s by default, with jitter, and only downloads when they have changed (ETag).
• It keeps the last good set in memory, and optionally on disk so it survives restarts.
• FR-33 gate() (v0.1, M3).
• It takes the question, the probabilities, model_served and, optionally, a decision_id.
• It returns the action, the label (for SELECT), the policy version, the source (policy, default or audit) and the reason.
• It runs purely locally. Given a decision_id (in proxy mode, from the proxy's header), it reports the action it took through POST /v1/decisions:actions (Phase 3).
• FR-34 Fingerprint check (v0.1, M3). Before using a policy, the SDK recomputes its fingerprint. On a mismatch it rejects that policy, keeps the last good one, and logs and counts an error.
• FR-35 Fail open (v0.1, M3).
• When no policy applies (server unreachable, nothing cached, model mismatch), gate() returns the default action configured for that question.
• The library's own default when none is configured is set in the fail-open ADR (Phase 2).
• gate() never raises.
• FR-36 log() and outcome() (v0.1, M3). Both are non-blocking and batched on a background thread, and retry with exponential backoff and jitter. Their buffer is bounded: when full, the oldest record is dropped, and drop counts are reported to the server.
• instrument(model_callable, model=...) wraps a decision model that runs in the app's own process, such as a Hugging Face pipeline("decision"), which the proxy can't see. Every call is logged as log_exchange would, with the served model given by the caller as <repo>@<revision> (wire-formats.md §2).
• FR-37 Audit sampling in the gate (v0.1, M3). The SDK applies the policy's audit fraction, decided deterministically by a hash of decision_id. Audited decisions return ESCALATE with source audit.

G. Drift and health¶

• FR-38 Label-free drift (v0.1, M4).
• For each group, the current window (the last 24 hours by default) is compared with a reference window (7 days by default).
• The comparison uses the population stability index (PSI) and Jensen-Shannon divergence, with a permutation-test p-value.
• It alerts only when the effect is large enough (PSI ≥ 0.2 by default), significant (p < 0.01 by default), and the window holds at least 500 decisions by default.
• FR-39 Silent upgrades (v0.1, M4).
• A new model_served for the same model_requested creates an event and an alert within 5 minutes.
• The alert names both versions and says the active policy no longer applies (FR-31).
• Only possible where the upstream names the version that answered: fully on TypeSafe, partly on Ollaya, not on Ollama or OpenAI, which echo the requested name (wire-formats.md; Phase 3).
• FR-40 Health signals (v0.1, M4):
• the escalation rate moving away from what the policy expected;
• jumps in upstream errors (proxy);
• latency regressions at p50 and p99;
• calibration getting worse once labels arrive (recent ECE compared with ECE at fit time).
• FR-41 Drift reports (v0.1, M4). Drift reports are stored (schema drift_report/1) and can be read through the API and the CLI.

J. Alerts¶

• FR-42 Channels (v0.1, M4): webhook (the body signed with HMAC, with retries), Slack (incoming webhook), and email (SMTP).
• FR-43 Rules (v0.1, M4).
• Thresholds are set per project and can be overridden per question.
• A repeat of the same alert is held back during a cool-down (1 hour by default).
• A "resolved" notice is sent when the condition clears.
• FR-44 Alert types (v0.1, M4):
• drift; a silent upgrade; escalation-rate shift; upstream errors; latency; calibration degradation;
• proposal ready; policy activated or rolled back;
• records dropped; outcome join backlog; review queue over capacity.

H. Shadow comparison¶

• FR-45 Shadow traffic (v0.1, M5). Each question can name a candidate upstream or model, a sample share (10% by default) and a daily call cap. The proxy calls the candidate after the primary has answered. The candidate never changes the response or its latency, and its failures are only logged.
• FR-46 Paired comparison (v0.1, M5). For each question, trimset shadow report shows:
• agreement, and the flip rate of actions under the active policy;
• an exact McNemar test on labeled pairs;
• accuracy and log loss on labeled pairs;
• cost and latency deltas.

I. Review queue¶

• FR-47 What gets queued (v0.1, M5).
• Each question chooses whether escalated items, audited items or both are queued.
• The queue is ordered by age and by how close each item is to a threshold.
• An alert fires when the queue exceeds its capacity.
• FR-48 Working the queue (v0.1, M5).
• API and CLI commands list, claim (a 15-minute lease by default), label and skip.
• trimset review is a keyboard-driven labeling session in the terminal.
• The web screen comes with the v2 dashboard.
• FR-49 Labels become outcomes (v0.1, M5). A reviewer's label is recorded as an outcome with source human_review and the reviewer's key id.
• FR-50 Context without raw state (v0.1, M5).
• Reviewers see the question, options, probabilities, tags and external_ref.
• A per-question link template, such as https://shop.example/admin/refunds/{external_ref}, opens the item in the app's own tool.
• Raw state is shown only if the project stores it.

K. Projects, keys and audit¶

• FR-51 Projects (v0.1, M1). Every record, key, policy and setting belongs to one project. There are no reads across projects.
• FR-52 API keys (v0.1, M1).
• A proxy key can only call the proxy endpoints; it is the key that sits in a base URL (ADR 0011). A review key can only work the review queue (added in Phase 3). An ingest key can log, send outcomes, use the proxy and read active policies. An admin key can do all of that, plus approvals and settings.
• Keys are stored as hashes, shown once, and revocable, and their last use is recorded.
• A revocation takes effect within 60 seconds.
• FR-53 Bootstrap (v0.1, M1). The first admin key is created with a CLI command on the server. There are no default credentials.
• FR-54 Audit log (v0.1, M3). Records policy transitions, key creation and revocation, changes to settings and privacy, and deletions, each with the key, the time, and the before and after values.

M. CLI¶

• FR-55 CLI (v0.1, from M2, growing with each milestone). Everything the v2 dashboard will show is available in the CLI. Output is a readable table by default, or JSON with --json. Phase 3 names the commands.

N. Data management¶

• FR-56 Retention (v0.1, M5).
• Decisions and outcomes are kept 400 days by default; fraud labels need 120 days plus a 90-day fit window.
• Stored raw state is kept 30 days by default.
• Old time partitions are dropped.
• FR-57 Deletion (v0.1, M5).
• Data can be deleted by project, by external_ref or by time range.
• Deletion reaches the outcomes, review items and raw state.
• Fits and policies keep only aggregates, never item-level data.
• FR-58 Export (v0.1, M5). Decisions and their outcomes for a group and a time range can be exported as JSONL, for offline analysis and the research/ experiments.

O. Developer tooling¶

• FR-59 Mock upstream (v0.1, M1). It serves /v1/systemone and /v1/decisions with deterministic probabilities derived from a seed and the state's hash. Latency and errors can be injected, and the model version can be switched to simulate a silent upgrade.
• FR-60 Synthetic data generator (v0.1, M1).
• It has scenarios for all six use cases.
• It can plant miscalibration, drift, a silent model swap, outcome delays and missing labels.
• It uses fixed seeds and writes through the API or as JSONL.
• All tests and demos use it.

L. Dashboard (v2)¶

• FR-61 Web dashboard (v2). The screens are those in design/dashboard-prompt.md. Every screen's data is already available from the v0.1 API.

9. Non-functional requirements¶

10. Non-goals for v0.1¶

• Running models (inference) or training and fine-tuning them. Calibration maps sit after the model and never change it.
• A hosted SaaS, user accounts or SSO (API keys only).
• The web dashboard and the TypeScript SDK (both v2 / v0.2).
• Activating policies automatically.
• Several error budgets at once, and guarantees per slice (v0.2).
• Generic tracing of LLM text, and replacing LangGraph or decision-circuits.
• High availability across several nodes or regions. One node, with documented backups.
• Telemetry of any kind.

11. Success metrics¶

The product works, measured on synthetic data. These are the numbers allowed on the homepage.

Adoption, 90 days after the v0.1 launch. These are public signals only, because trimset has no telemetry. They are first guesses, to revisit at launch.

12. Requirements mind map¶

mindmap
  root((trimset v0.1))
    Capture
      Log API
      Proxy systemone
      Proxy OpenAI decisions
      Private by default
    Outcomes
      Late and early join
      Corrections with history
      Default labels by rule
      Coverage and audit sampling
    Calibration
      Temperature Platt isotonic
      Log loss Brier ECE
      Strict splits
    Policy compiler
      Error budget
      False-pass cap
      Escalation band and cap
      Cost model
      Honest guarantees
      Simulator
    Lifecycle
      Propose approve activate
      Rollback
      Refits only propose
      Bound to model version
    SDK gate
      Local under 1 ms
      Fingerprint check
      Fail open
      Audit sampling
    Monitoring
      PSI and JS drift
      Silent upgrades
      Health signals
      Alerts
    Shadow and review
      Paired comparison
      Review queue
    Platform
      Projects and keys
      Audit log
      Retention and deletion
      CLI
      Mock upstream and data generator

13. Traceability matrix¶

Components are provisional; Phase 2 fixes them. Test kinds:

14. Decisions made in this phase, and open questions¶

Decided here (please check): 1. Upstreams in v0.1: the /v1/systemone format (Jev, Ollaya, Ollama) and OpenAI's /v1/decisions. Azure OpenAI and others come once their APIs are verified. 2. Default labels by rule (FR-13): added, because the most common outcome in fraud and triage is "nothing happened". 3. Audit sampling (FR-14, FR-37): added. Without it, a blocked refund never gets a true label, and refits learn only from the paths that do get labels. 4. Review link template (FR-50): added, so reviewers get context without trimset storing raw state. 5. Review queue in v0.1 is API and CLI. The web screen comes with the v2 dashboard. 6. The proxy keeps forwarding when the database is down (NFR-06). Records are dropped and counted, never allowed to block the app. 7. A policy is bound to its exact model version (FR-31). A silent upgrade therefore falls back to the SDK default and raises a loud alert, rather than applying thresholds tuned for another model. 8. Escalation cap as a target (FR-21), because moderation and fraud review are limited by people's time. 9. No telemetry, so adoption is measured only from public signals.

Open for Phase 2 and 3 (I'll decide these, with ADRs): 1. How the proxy identifies the project from the base URL alone (a key in the URL path, one project per port, or a default project), weighing that URLs end up in logs. 2. A keyed hash for state_sha256 (HMAC with a per-project secret) versus plain SHA-256. A plain hash of a short state can be reversed by guessing. 3. The SDK's own default action when the app configures none. 4. The exact threshold forms for choice and score. 5. Which method backs each guarantee: exact bounds on a held-out split, Learn-then-Test, or conformal risk control. This gets settled in research.md. 6. The exact wire formats of /v1/systemone and /v1/decisions, from the official docs.