Task — engineering-spec@1

"CUO Phase 2 trace rows include prompt + model + temperature + seed for deterministic replay"

doneTASK-CUO-103
module cuo · class product · priority p0 · created 2026-05-17 · shipped null
depends on TASK-CUO-101, TASK-CUO-102 · blocks none

§1 — Description (BCP-14 normative)

The CUO service MUST ship trace replay rows at services/cuo/src/trace/ capturing prompt+model+temp+seed per AI call + replay test, 4 memory audit kinds.

  1. MUST validate trace_call_kind against closed enum per DEC-2331.
  1. MUST capture per call at writer.rs::write(call) per DEC-2330:
  1. MUST replay at replay.rs::replay(trace_id) per DEC-2333:
  1. MUST define table at migration 0003: ``sql CREATE TABLE cuo_trace_rows ( trace_id UUID PRIMARY KEY, tenant_id UUID NOT NULL, run_id UUID NOT NULL, checkpoint_id UUID, -- TASK-CUO-102 ref kind TEXT NOT NULL CHECK (kind IN ('router_decision','tool_call','response_generation','validation_check')), prompt_text TEXT NOT NULL, model TEXT NOT NULL, temperature NUMERIC(5,4) NOT NULL, seed BIGINT, response_text TEXT NOT NULL, correction_of UUID REFERENCES cuo_trace_rows(trace_id), created_at TIMESTAMPTZ NOT NULL DEFAULT now() ) PARTITION BY RANGE (created_at); CREATE INDEX trace_run_idx ON cuo_trace_rows(tenant_id, run_id, created_at); ALTER TABLE cuo_trace_rows ENABLE ROW LEVEL SECURITY; CREATE POLICY trace_rls ON cuo_trace_rows USING (tenant_id = current_setting('auth.tenant_id')::uuid) WITH CHECK (tenant_id = current_setting('auth.tenant_id')::uuid); REVOKE UPDATE, DELETE ON cuo_trace_rows FROM cyberos_app; ``
  1. MUST expose endpoints: ``text POST /v1/cuo/trace/{id}/replay (CDO triggers replay test) GET /v1/cuo/runs/{run_id}/trace (list trace rows) ``
  1. MUST emit 4 memory audit kinds per DEC-2334. PII per TASK-MEMORY-111: prompt + response SHA-256 hashed.
  1. MUST thread trace_id from supervisor → AI call → writer → audit.
  1. MUST NOT mutate prior trace per DEC-2332 (REVOKE UPDATE/DELETE).
  1. MUST NOT skip seed capture per DEC-2330 (even if null, document why).

§2 — Why this design

Why prompt + model + temp + seed (DEC-2330)? Reproducibility — these 4 fields are the AI call's inputs; with seed, output deterministic.

Why immutable (DEC-2332)? Audit lineage; without immutability, replay becomes circular.

Why drift detection (DEC-2333)? Vendor models version-drift; without drift alert, replay quietly produces different output.


§3 — API contract

Sample trace row:

{
  "trace_id": "uuid",
  "run_id": "uuid",
  "kind": "router_decision",
  "prompt_text": "Given user request: ... Choose tool from: ...",
  "model": "claude-sonnet-4-6",
  "temperature": 0.0,
  "seed": 42,
  "response_text": "calendar.list_events"
}

§4 — Acceptance criteria

  1. trace_call_kind enum cardinality 4. 2. prompt + model + temp + seed captured. 3. Immutable rows. 4. Replay test exists. 5. Match → audit emitted. 6. Drift → sev-2 audit. 7. 4 memory audit kinds emitted. 8. PII scrubbed (prompt + response SHA256). 9. RLS denies cross-tenant. 10. Trace_id preserved. 11. Append-only via REVOKE. 12. Monthly partitioning. 13. Replay async (LLM call expensive). 14. rust_decimal for temperature. 15. bigint for seed. 16. NULL seed allowed (vendor-dependent). 17. Correction_of for re-recorded traces. 18. CDO-only replay trigger. 19. Run-scoped query indexed. 20. Partition retention same as TASK-CUO-102 (7 years).

§5 — Verification

#[tokio::test]
async fn replay_byte_identical_with_seed() {
    let ctx = TestContext::with_trace_seed_42().await;
    let r = ctx.replay(ctx.trace_id).await;
    assert_eq!(r.status, "match");
}

#[tokio::test]
async fn drift_detected_when_response_differs() {
    let ctx = TestContext::with_trace().await;
    ctx.mock_ai_returns_different_response().await;
    let r = ctx.replay(ctx.trace_id).await;
    assert_eq!(r.status, "drift");
    let audits = ctx.fetch_memory_audits("cuo.trace_replay_drift").await;
    assert!(!audits.is_empty());
}

#[tokio::test]
async fn immutable_no_update() {
    let ctx = TestContext::with_trace().await;
    let r = ctx.try_update_trace(ctx.trace_id).await;
    assert!(r.is_err());
}

// 5.4..5.10

§7 — Dependencies

Upstream: TASK-CUO-102. Cross-module: TASK-AI-003 (LLM calls captured), TASK-AUTH-101 (CDO role), TASK-MEMORY-111 (PII).

§10 — Failure modes

FailureDetectionOutcomeRecovery
Mutation attemptREVOKEDB errorinherent
Replay driftcomparisonsev-2investigate vendor
Vendor model deprecatedreplay failsev-2model upgrade plan
Cross-tenant queryRLS0 rowsinherent
NULL seed (vendor)OKreplay marked non-deterministicinherent
Prompt > 100k charsvalidatereject; sev-2reduce
Replay rate limitinherentretryinherent
Concurrent traceappendinherentinherent
Decimal precisionrust_decimalinherentinherent
Partition fillmonthlyinherentinherent

§11 — Implementation notes


End of TASK-CUO-103 spec.