"Metering ai_tokens emit at cost_reconcile — WalQueue push from ai-gateway"
TASK-TEN-204: ai_tokens metering emit at cost_reconcile
Summary
Add the first live metering emit path: when ai-gateway cost_reconcile finalises a hold with token usage (Success or Cancelled with partial usage), push one MeteringEvent with axis = ai_tokens onto a process-local WalQueue (cyberos-metering), idempotent on hold_id.
Problem
TASK-TEN-004 shipped axes / recorder / WalQueue / SQL migration with no callers. Spec §1 #7 names the cost-ledger postcall hook; as-built that hook is services/ai-gateway/src/cost_reconcile.rs (not cost_ledger.rs). Without an emit path, period aggregates stay empty.
Proposed Solution
cyberos-ai-gatewaydepends oncyberos-metering.- Module
metering_emit.rs: process-localOnceLock<Mutex<WalQueue>>(or test-injectableRecorder);emit_ai_tokens(tenant_id, hold_id, provider, model, prompt, completion). - Call from
reconcileonCallOutcome::SuccessandCancelled { partial_usage: Some(_) }after usage is known; do not emit on ProviderError or Cancelled(None). - Event shape:
axis: AiTokensquantity: prompt_tokens + completion_tokens(skip if quantity == 0)idempotency_key: hold_id.to_string()source_service: "ai-gateway"extra: { provider, model_alias, input_tokens, output_tokens }
- WAL overflow: log + metric/counter; reconcile still commits (metering must not fail the call).
- Unit test: push Success path → queue depth 1; duplicate hold_id → still depth 1 if using InMemoryRecorder for the test surface, or WalQueue accepts duplicates until Pg drain (document: WAL may carry dupes; Pg UNIQUE is the idempotency floor — for this slice test InMemoryRecorder via a thin
emit_tohelper).
Alternatives Considered
- Auth middleware
api_callsfirst. Rejected for host-a: larger surface (AppState, every JWT route, overage). Operator chose ai_tokens as smallest emit. - Postgres insert inside reconcile TX. Rejected: metering migration not in ai-gateway DB; WAL decouples; Pg drain is a later slice.
- Block reconcile on WAL overflow. Rejected: TEN-004 hot-path doctrine — metering outage must not fail AI calls.
Success Metrics
- Primary: one Success reconcile with N tokens produces one queued/recorded AiTokens event.
- Guardrail: ProviderError reconcile does not enqueue; existing cost_reconcile tests pass.
Scope
In scope
- Dep + emit helper + two reconcile branches + unit tests.
Out of scope / Non-Goals
- Auth
api_callsmiddleware emit and overage 402. - Pg
Recorder/ sqlx migrate CI for metering / background drain. - Seats / storage snapshot jobs; period close; memory dual-write for metering.
- INV Wise host.
Dependencies
TASK-TEN-004done (library).- ai-gateway
cost_reconcile(TASK-AI-001 lineage).
AI Authorship Disclosure
Generated then reviewed against as-built metering crate + cost_reconcile (2026-07-26).
Acceptance Criteria
- Success emit — reconcile Success with prompt=10 completion=5 → event quantity 15, axis ai_tokens.
- Cancelled partial — Cancelled(Some(usage)) emits; Cancelled(None) does not.
- ProviderError — no metering event.
- Idempotency key — equals
hold_idUUID string. - extra fields — provider, model_alias, input_tokens, output_tokens present.
- Non-blocking — WalQueue overflow / emit error does not change ReconcileOutcome success.
- Zero tokens — quantity 0 skipped (no invalid quantity push).
- Dep —
Cargo.tomllistscyberos-metering.
Verification
cd services
cargo test -p cyberos-ai-gateway --test metering_ai_tokens_emit_test -- --test-threads=1
cargo test -p cyberos-metering -- --test-threads=1
bash .cyberos/cuo/gates/run-gates.sh
Failure Modes
| Failure | Detection | Outcome | Recovery |
|---|---|---|---|
| WAL overflow | WalError::Overflow | log; call OK | Drain / raise capacity later |
| qty out of range | validate_quantity | skip + log | Cap at provider |
| Missing hold fields | N/A (locked row) | — | — |
| Double reconcile | AlreadyFinalised | no second emit | Idempotent hold |
| Mutex poison | lock err | log; call OK | Restart process |
| Zero tokens | qty check | no emit | OK |
| Test isolation | reset_for_tests | clean queue | cfg(test) reset |
| Dep version skew | cargo | build fail | Workspace path |
| Extra JSON missing keys | test assert | fail CI | Fix emit |
| Emit after TX commit vs before | code review | prefer after successful apply, before commit OK if non-blocking | Document |
End of TASK-TEN-204.
Audit
§1 — Verdict summary
Smallest metering emit residual: cost_reconcile → ai_tokens WalQueue. 8 ACs, 10 failure modes, honest Out of scope for api_calls and Pg drain.
§2 — Findings (all resolved)
ISS-001 — Spec cited cost_ledger.rs
As-built is cost_reconcile.rs. Resolved: Problem + Proposed Solution cite real path.
ISS-002 — Emit must not fail AI calls
TEN-004 latency doctrine. Resolved: AC #6 + Alternatives + failure WAL overflow.
ISS-003 — Idempotency unclear
hold_id vs hold.idempotency_key. Resolved: AC #4 hold_id string.
ISS-004 — Zero-token Success
validate_quantity rejects 0 for ai_tokens. Resolved: AC #7 skip.
ISS-005 — ProviderError false billing
Resolved: AC #3 no emit.
ISS-006 — WAL vs Recorder for tests
Duplicates in WAL until Pg. Resolved: Proposed Solution #6 test via InMemoryRecorder helper.
§3 — Resolution
All 6 mechanical concerns addressed. Score = 10/10.
End of TASK-TEN-204 audit.