OBS is the shared telemetry plane: logs, metrics, traces, and AI-trace observability for every CyberOS module. Operationally, OpenTelemetry SDKs in every service ship to a single OTel collector that fans out - logs to Loki, metrics to Prometheus, traces to Tempo. Grafana renders dashboards (per-module SLO, per-tenant cost, per-region health). LangSmith captures full LLM call traces independently from the operational pipeline so AI debugging doesn't require correlating across three tools. Alert Manager fans critical alerts to PagerDuty, mid alerts to #cyberos-alerts, low signals into the CUO morning digest. The audit chain - owned by memory - is exposed via a separate read-only OBS surface for regulators (PDPL Art. 14, EU AI Act Art. 12). Tenant scoping is enforced at the query proxy so a member of tenant A cannot see tenant B's logs.

• Strategic role: observability spine - 3 pillars + AI traces + audit
• Status: planned - P0; slice 1 in build
• Stack: LGTM + LangSmith - Loki, Grafana, Tempo, Prometheus
• Trace exporter: OTel SDK - in every service binary
• Correlation key: trace_id x tenant_id - propagated via W3C TraceContext
• Retention: 7d / 90d / 1y - hot / warm / cold tiers
• Tenant isolation: proxy-enforced - no cross-tenant reads
• SLO targets: >= 99.5% - platform composite SLO
• Auto-runbook coverage: >= 60% by P1 - alerts -> CUO triage skill
• Compliance surfaces: EU AI Act, PDPL, SOC 2 - read-only regulator views
• Depends on: memory, plus AUTH (P0 slice 2) and AI Gateway (P0 slice 1)
• Used by: all 24 modules - every service emits OTel

The bigger picture - three strategic roles

OBS is one of the two earliest P0 modules to ship (P0 slice 1, alongside AI Gateway), because any debugging requires it. More than a Grafana install, it is the protocol-level guarantee that any incident - operational, AI-decision, or compliance - has exactly one investigation surface instead of three competing dashboards from three vendors.

Role 1 - three-pillars unified pane

Logs + metrics + traces + AI traces, one trace_id. Every module emits OpenTelemetry. The collector fans out to Loki (logs), Prometheus (metrics), Tempo (operational traces), LangSmith (AI traces). All four are correlated by the W3C TraceContext trace_id + tenant_id. One click in Grafana takes you from a slow API trace to the LLM call inside it to the memory audit row that recorded the decision. No more "the AI did something weird, but I can only see the HTTP request."

Role 2 - auto-runbook router

Alerts route through CUO before paging humans. An alert fires. Before PagerDuty pages someone, the alert hits CUO's obs.triage-alert@1 skill. CUO consults the runbook catalogue (in KB), suggests the first step, attaches the relevant traces + dashboard links, and either (a) creates a self-service ticket in CHAT if confidence >= 0.70 OR (b) escalates to PagerDuty with the suggested first step pre-attached. Target: >= 60% of alerts auto-runbookable by P1; the on-call schedule benefits, the runbook catalogue grows, the audit trail is complete.

Role 3 - compliance evidence surface

Read-only regulator view of the memory audit chain. Auditors don't need access to Postgres. They need a scoped, time-bounded, exportable view of the audit chain that proves "the AI made this decision at this time under this persona, and a human confirmed it before it executed." OBS surfaces this via the /compliance/{regulator} read-only endpoint per regulator (EU AI Act / PDPL / SOC 2 / ISO 27001). Each regulator sees only the chained audit rows + retention metadata they're entitled to.

OBS in the runtime - every module emits, OBS correlates

flowchart TB
  subgraph emitters["Emitters (all 24 modules)"]
    CUO["CUO"]
    memory["memory"]
    SKILL["Skill"]
    AI["AI Gateway"]
    AUTH["AUTH"]
    MCP["MCP"]
    PROJ["PROJ"]
    N["... 17 more"]
  end
  COLLECT["OTel Collector<br/>(W3C TraceContext)"]
  OBS["OBS<br/>LGTM + LangSmith"]
  subgraph stores["Stores (per pillar)"]
    LOKI["Loki<br/>logs"]
    PROM["Prometheus<br/>metrics"]
    TEMPO["Tempo<br/>traces"]
    LS["LangSmith<br/>AI traces"]
  end
  subgraph consumers["Consumers"]
    GRAF["Grafana dashboards"]
    ALERT["Alert Manager"]
    CUO_TRIAGE["CUO<br/>obs.triage-alert@1"]
    PAGER["PagerDuty"]
    CHAT["CHAT digest"]
    COMP["Compliance views<br/>(EU AI Act / PDPL / SOC 2)"]
  end
  CUO --> COLLECT
  memory --> COLLECT
  SKILL --> COLLECT
  AI --> COLLECT
  AUTH --> COLLECT
  MCP --> COLLECT
  PROJ --> COLLECT
  N --> COLLECT
  COLLECT --> OBS
  OBS --> LOKI
  OBS --> PROM
  OBS --> TEMPO
  OBS --> LS
  OBS --> GRAF
  OBS --> ALERT
  ALERT --> CUO_TRIAGE
  CUO_TRIAGE -.->|"conf >= 0.70"| CHAT
  CUO_TRIAGE -.->|"conf below 0.70 OR P1"| PAGER
  OBS --> COMP
  classDef hub fill:#ede9fe,stroke:#5b21b6,stroke-width:3px,color:#3b0764
  classDef mod fill:#e0e7ff,stroke:#3730a3
  classDef store fill:#f5ede6,stroke:#45210e
  classDef consumer fill:#fef6e0,stroke:#9c750a
  class OBS,COLLECT hub
  class CUO,memory,SKILL,AI,AUTH,MCP,PROJ,N mod
  class LOKI,PROM,TEMPO,LS store
  class GRAF,ALERT,CUO_TRIAGE,PAGER,CHAT,COMP consumer

The OTel Collector is the only fan-in; OBS is the only correlation point. Removing OBS means each module re-implements observability - and the implementations diverge on tenant scoping, retention, and trace correlation.

Auto vs human-in-loop operations matrix

Why OBS exists

Production observability is one of the line items that, if not centralised early, fragments quickly: one team picks Datadog, another picks Honeycomb, the AI team picks LangSmith, the compliance team asks for an audit-log dashboard that nobody owns. Centralise the platform, let every module emit OpenTelemetry, give compliance read-only audit views, and the question "is the platform healthy?" has one answer instead of five.

• LGTM is enough. Loki + Grafana + Tempo + Prometheus = the full operational picture. Self-hosted; runs on Fargate + S3.
• AI traces are different. LangSmith captures full prompt + completion + tool-call chains. Operational tracing alone won't tell you why an agent made a bad decision.
• Compliance is a first-class view. EU AI Act Art. 12 + PDPL Art. 14 demand decision logging that regulators can inspect - OBS owns the read-only audit surface.

The bet: pay the LGTM operational cost once, plug LangSmith in beside it, and you get incident response, SLO tracking, AI debugging, and compliance evidence from one plane. The alternative - three different SaaS tools, each with its own auth and bill - is a money-and-context drain that compounds with every new module.

What it does - 5W1H2C5M

Three-pillars unified pane - logs, metrics, traces, AI traces

The naive multi-tool stack - Datadog for metrics + Honeycomb for traces + Sentry for errors + LangSmith for AI - costs $50k/month at small scale and forces engineers to context-switch across vendors during incidents. The CyberOS stack is LGTM + LangSmith, all self-hosted, all correlated by trace_id x tenant_id x persona_version. One Grafana, one investigation surface.

Pillar x signal-type mapping

Cross-pillar correlation example - one investigation, four pillars

An engineer investigates "why did this Member's CHAT message take 8 s?". The investigation path:

1. Metric (Prom): chat_send_p95_ms{tenant=acme} = 8200 at 14:32 - spike confirmed.
2. Trace (Tempo): pick a slow trace from the spike window; see the request hit MCP Gateway, then CUO, then AI Gateway; AI Gateway took 7.4 s of the 8.2 s.
3. AI trace (LangSmith): click the AI Gateway span; see the LLM call; the primary provider returned 429 (rate limit), failover added 6.8 s.
4. Log (Loki): same trace_id; see the structured log {level: warn, msg: "provider rate limit", provider: bedrock, retry_after_s: 6}.
5. Memory audit row: same trace_id; confirm the ai.invocation audit row carries failover_path: fallback; root cause identified, the runbook says "increase Bedrock quota for this tenant".

5 minutes from "p95 alert" to "open the quota-increase ticket." Without correlation, the same investigation takes hours across separate tool consoles.

Tenant query proxy - the isolation guarantee

All four pillars (Loki, Prom, Tempo, LangSmith) are queried through a Rust proxy that:

• Validates the JWT (per AUTH §2.7) and extracts tenant_id + scope_grants.
• Injects a mandatory tenant=<id> label filter into every LogQL / PromQL / TraceQL query.
• Rejects queries that attempt to use the tenant label in their own filter (prevents bypass via a crafted query).
• Audits the query itself (compliance evidence that the engineer queried only their authorised tenants).
• Rate-limits at 100 QPS/Member to prevent runaway dashboards from DOS'ing the stack.

Cross-tenant query attempts return 403_TENANT_SCOPE_VIOLATION + emit an audit row + page the CSO. This is the protocol-level guarantee that an SRE on the engineering team cannot accidentally (or maliciously) see another tenant's logs.

Auto-runbook router - alerts that don't page humans

The default behaviour of every monitoring tool is "alert fires -> PagerDuty -> human pages." That model assumes the human knows what to do; in practice, 60-80% of alerts are repeats with known remediation. OBS inverts the default: alerts route through CUO's obs.triage-alert@1 skill first, which consults the runbook catalogue (in KB), suggests the first step, and either auto-creates a CHAT ticket OR escalates to PagerDuty. Humans only see the alerts that genuinely need human judgment.

The 6-step routing sequence

sequenceDiagram
  autonumber
  participant P as Prometheus / Loki rules
  participant AM as Alert Manager
  participant C as CUO obs.triage-alert@1
  participant KB as KB runbook catalogue
  participant CH as CHAT (#cyberos-alerts)
  participant PD as PagerDuty
  participant B as memory audit
  P->>AM: alert fires (severity, labels, value)
  AM->>C: invoke triage skill (alert payload + trace_id)
  C->>KB: query runbook by alert signature (similar past alerts)
  KB-->>C: top-3 runbook matches + confidence scores
  C->>C: confidence = max(matches)
  alt confidence >= 0.70 AND severity <= P2
    C->>CH: post ticket with runbook link + suggested first step
    C->>B: audit row (alert, runbook, routing=self-service)
  else confidence below 0.70 OR severity in {P0, P1}
    C->>PD: escalate with suggested first step pre-attached
    C->>B: audit row (alert, routing=pagerduty)
  end
  Note over CH,PD: on-call sees only alerts that need human judgment

Alert severity x routing matrix

Runbook catalogue grows by itself

When CUO can't auto-route an alert (confidence < 0.70), the resulting PagerDuty incident becomes a runbook-authoring trigger. Post-incident, the on-call adds a runbook entry; CUO's confidence on similar future alerts rises. The feedback loop:

1. Novel alert -> PagerDuty -> human handles -> resolves -> writes a runbook (or skill-author drafts one from the resolution).
2. Runbook indexed in KB with the alert-signature embedding.
3. Next similar alert -> CUO finds the runbook -> confidence >= 0.70 -> CHAT self-service.
4. On-call load decreases over time; the runbook catalogue grows.

Target: >= 60% of alerts auto-runbookable by P1 exit; >= 80% by P2. The KPI dashboard surfaces "% of alerts that never paged anyone" as the headline metric.

Compliance evidence surface - read-only regulator views

The standard pattern when an auditor arrives is "let me give you a Postgres readonly account and good luck." It's a security incident waiting to happen. OBS exposes per-regulator scoped read-only views over the memory audit chain so auditors see exactly what they need - chained, exportable, tamper-evident - without touching the underlying operational data.

Regulator view x audit scope matrix

Per-view scoping mechanism

compliance_views:
  eu_ai_act:
    audit_kinds: [ai.invocation, ai.failover_triggered, ai.degraded_mode]
    filter: { tenant.residency: eu-1 }
    fields_visible: [seq, ts_ns, op, extra.tenant_id, extra.agent_persona, extra.module,
                     extra.usage, extra.redaction_applied, extra.failover_path,
                     prev_chain, chain]
    fields_hidden: [extra.prompt_hash, extra.response_hash]   # hashes only proven, not exposed
    retention_min_days: 180
    export_format: signed_json
    auditor_account: requires_csr_signoff
  pdpl_dsar:
    audit_kinds: [*]   # all kinds - DSAR is data-subject-scoped, not kind-scoped
    filter: { subject_in_extra: $subject_id }
    fields_visible: all_except_pii_hashes
    retention_min_days: 365
    export_format: json_or_csv
    auditor_account: tenant_dpo_only
  soc2_cc7_2:
    audit_kinds: [auth.*, access.*, backup.*, incident.*]
    fields_visible: [seq, ts_ns, op, extra.actor, extra.action, extra.target]
    retention_min_days: 365
    export_format: csv_bundle
    auditor_account: external_auditor_only

Chain-of-custody manifest

Every regulator export includes a manifest that proves the export is a faithful slice of the memory chain. The manifest contains:

• Time window: ISO 8601 range; rows outside the window are excluded.
• Filter predicates: the exact compliance-view filter that was applied.
• Row count: N rows exported, sequence numbers N_min..N_max.
• Chain anchors: the memory chain hash at the start and end of the window (proves no rows were inserted or removed between those points).
• Export signer: Ed25519 signature by OBS with the export-request audit row hash.
• Auditor identity: JWT subject of the auditor requesting the export.

An auditor can verify the export independently: recompute the chain hashes from the exported rows + the manifest anchors, and confirm they match. This is the protocol-level guarantee that compliance evidence cannot be tampered with mid-export.

Architecture

Every CyberOS service ships the OTel SDK in-process. The collector receives all signals, applies PII redaction, tags with tenant_id, and fans out to Loki (logs), Tempo (traces), Prometheus (metrics). LangSmith receives AI-trace data directly from AI Gateway. Grafana renders dashboards via a Rust tenant-aware query proxy. Alert Manager evaluates Prometheus rules and routes.

graph TB
  subgraph SERVICES ["Every CyberOS service"]
    SVC1["AUTH"]
    SVC2["AI Gateway"]
    SVC3["MCP Gateway"]
    SVC4["CHAT"]
    SVCN["... 20 more modules"]
  end
  subgraph COLLECTOR ["OTel Collector (Fargate, per-region)"]
    REC["receivers<br/>OTLP grpc/http"]
    RED["redactor processor<br/>PII scrub, recall >= 99.5%"]
    TAG["tenant_tag processor<br/>JWT -> tenant_id label"]
    SAMP["sampler<br/>tail-based for traces"]
    EXP["exporters"]
  end
  subgraph LGTM ["LGTM backends (S3-backed)"]
    LOKI[("Loki<br/>logs, 7d hot / 90d warm")]
    TEMPO[("Tempo<br/>traces, 7d hot / 30d warm")]
    PROM[("Prometheus<br/>metrics, 15d local / 1y Mimir P1+")]
  end
  subgraph DASH ["Grafana + query proxy"]
    QP["tenant_query_proxy.rs<br/>Rust, enforces tenant scope on every query"]
    GRAF["Grafana 11.x<br/>dashboards"]
  end
  subgraph ALERT ["Alert Manager"]
    AM["alertmanager.yml<br/>routes by severity"]
    PD["PagerDuty<br/>critical"]
    CHAT["CHAT bot<br/>mid"]
    DIG["CUO digest<br/>low"]
  end
  subgraph AI ["AI trace plane"]
    LANG["LangSmith SaaS<br/>(zero-retention)"]
  end
  subgraph AUDIT ["Audit surface"]
    BR["memory<br/>binlog"]
    AS["audit_view.rs<br/>read-only, auditor-scoped"]
  end
  SVC1 --> REC
  SVC2 --> REC
  SVC3 --> REC
  SVC4 --> REC
  SVCN --> REC
  REC --> RED
  RED --> TAG
  TAG --> SAMP
  SAMP --> EXP
  EXP --> LOKI
  EXP --> TEMPO
  EXP --> PROM
  PROM --> AM
  AM --> PD
  AM --> CHAT
  AM --> DIG
  GRAF --> QP
  QP --> LOKI
  QP --> TEMPO
  QP --> PROM
  SVC2 -. LangSmith SDK .-> LANG
  BR --> AS
  AS --> GRAF
  classDef shipped fill:#f5ede6,stroke:#45210e
  classDef planned fill:#fef6e0,stroke:#7c3aed
  classDef store fill:#f5f3ff,stroke:#7c3aed
  class BR shipped
  class REC,RED,TAG,SAMP,EXP,QP,GRAF,AM,AS planned
  class LOKI,TEMPO,PROM,LANG store
  class PD,CHAT,DIG planned
  class SVC1,SVC2,SVC3,SVC4,SVCN planned

Internal components

Data model

OBS is mostly streaming - its "data model" is the schema of OTel signals plus SLO and alert configuration. Below are the entity relationships.

erDiagram
  TENANT ||--o{ LOG_STREAM: "owns"
  TENANT ||--o{ METRIC_SERIES: "owns"
  TENANT ||--o{ TRACE: "owns"
  TENANT ||--o{ SLO_TARGET: "defines"
  TENANT ||--o{ ALERT_RULE: "defines"
  SERVICE ||--o{ LOG_STREAM: "produces"
  SERVICE ||--o{ METRIC_SERIES: "exposes"
  SERVICE ||--o{ SPAN: "emits"
  TRACE ||--|{ SPAN: "contains"
  ALERT_RULE ||--o{ ALERT_INSTANCE: "fires"
  SLO_TARGET ||--o{ SLO_BURN: "calculated by"
  AI_TRACE ||--o{ LLM_SPAN: "contains"
  AI_TRACE ||--o{ TOOL_CALL_SPAN: "contains"
  LOG_STREAM {
    string label_set "service=auth,tenant=acme,env=prod"
    timestamp ts
    string level "DEBUG | INFO | WARN | ERROR"
    string message
    obj attributes
    string trace_id
  }
  METRIC_SERIES {
    string name "http_requests_total"
    string label_set
    string type "counter | gauge | histogram"
    float value
    timestamp ts
  }
  TRACE {
    string trace_id PK
    int64 duration_ns
    string root_service
    int span_count
    timestamp start_ts
    string tenant_id
  }
  SPAN {
    string span_id PK
    string trace_id FK
    string parent_span_id
    string service_name
    string operation
    int64 duration_ns
    obj attributes
    string status "ok | error"
    timestamp start_ts
  }
  AI_TRACE {
    string id PK "LangSmith run id"
    string trace_id "correlates with OTel"
    string persona
    string persona_version
    string model_id
    int tokens_in
    int tokens_out
    string outcome "ok | err"
    timestamp ts
  }
  LLM_SPAN {
    string id PK
    string ai_trace_id FK
    string messages_in_hash
    string completion_hash
    int tokens
  }
  TOOL_CALL_SPAN {
    string id PK
    string ai_trace_id FK
    string tool_name
    string args_hash
    string result_hash
  }
  SLO_TARGET {
    string id PK
    string service
    string indicator "availability | latency | error_rate"
    float target_pct
    string window "28d | 7d"
  }
  SLO_BURN {
    string slo_id FK
    float budget_remaining_pct
    float burn_rate_short
    float burn_rate_long
    timestamp ts
  }
  ALERT_RULE {
    string id PK
    string promql
    string severity "critical | warning | info"
    int for_seconds
    obj routing
  }
  ALERT_INSTANCE {
    string id PK
    string rule_id FK
    string state "pending | firing | resolved"
    timestamp started_at
    timestamp resolved_at
    obj labels
  }

Canonical OTel attribute schema

API surface

Query API (Grafana-compatible, tenant-scoped)

All queries flow through tenant_query_proxy.rs, which extracts tenant_id from the caller's JWT and rewrites the query to inject a {tenant_id="..."} label filter. Cross-tenant queries return 403.

GraphQL subgraph (federated)

extend schema
 @link(url: "https://specs.apollo.dev/federation/v2.5", import: ["@key", "@requiresScopes"])

type SLO @key(fields: "id") {
 id: ID!
 service: String!
 indicator: SLOIndicator!
 targetPct: Float!
 window: String!
 currentPct: Float!
 budgetRemainingPct: Float!
 burnRateShort: Float!
 burnRateLong: Float!
}

type Alert @key(fields: "id") {
 id: ID!
 ruleName: String!
 severity: Severity!
 state: AlertState!
 startedAt: DateTime!
 resolvedAt: DateTime
 labels: JSON!
}

type CostReport @key(fields: "tenantId month") {
 tenantId: ID!
 month: String!
 totalUsdCost: Float!
 infraUsdCost: Float!
 aiUsdCost: Float!
 storageUsdCost: Float!
 byService: [ServiceCost!]!
}

type ServiceCost {
 service: String!
 usdCost: Float!
}

enum SLOIndicator { AVAILABILITY LATENCY ERROR_RATE THROUGHPUT }
enum Severity { CRITICAL WARNING INFO }
enum AlertState { PENDING FIRING RESOLVED }

type Query {
 slos(service: String): [SLO!]! @requiresScopes(scopes: [["obs.read"]])
 alertsActive: [Alert!]! @requiresScopes(scopes: [["obs.read"]])
 costMTD: CostReport! @requiresScopes(scopes: [["obs.cost_read"]])
 trace(id: String!): Trace @requiresScopes(scopes: [["obs.read"]])
}

OTel ingest endpoints

Key flows

Flow 1 - log ingestion (PII-scrubbed, tenant-tagged)

sequenceDiagram
  autonumber
  participant SVC as AUTH service
  participant SDK as OTel SDK (Rust)
  participant COL as OTel Collector
  participant RED as redactor
  participant TAG as tenant_tag
  participant LOKI as Loki
  SVC->>SDK: log "login attempt for user={email} cccd={cccd}"
  SDK->>SDK: attach trace_id, span_id, service.name, tenant.id
  SDK->>COL: OTLP /v1/logs
  COL->>RED: scan attributes + message
  RED->>RED: find vn.cccd, en.email and replace with sentinels
  RED-->>COL: redacted record (email=[REDACTED], cccd=[REDACTED])
  COL->>TAG: ensure tenant_id label present
  TAG-->>COL: labelled
  COL->>LOKI: push (label_set, ts, message)
  LOKI-->>COL: 204 No Content

(task pending): PII recall >= 99.5%. Redaction at the collector is the last point at which PII can be stopped before it lands on S3.

Flow 2 - metric scrape + alert evaluation

sequenceDiagram
  autonumber
  participant SVC as AI Gateway
  participant PROM as Prometheus
  participant AM as Alert Manager
  participant PD as PagerDuty
  participant CHAT as cyberos-alerts channel
  PROM->>SVC: GET /metrics (every 15 s)
  SVC-->>PROM: ai_request_latency_p95_seconds 2.3<br/>ai_provider_error_total 47
  loop alert eval every 30 s
    PROM->>PROM: evaluate rule (ai_request_latency_p95_seconds above 2 for 5m)
    alt firing
      PROM->>AM: alert {severity=critical, service=ai-gateway}
      AM->>AM: route by labels
      AM->>PD: page on-call (severity=critical)
      AM->>CHAT: post message
    end
  end

Flow 3 - trace propagation across modules

sequenceDiagram
  autonumber
  participant U as User SPA
  participant AR as Apollo Router
  participant AUTH as AUTH RBAC
  participant CHAT as CHAT service
  participant AI as AI Gateway
  participant BR as memory
  participant TEMPO as Tempo
  U->>AR: POST /graphql (mutation sendMessage)
  AR->>AR: start trace (trace_id=t1)
  AR->>AUTH: RBAC.Check (carries trace_id)
  AUTH-->>AR: allow
  AR->>CHAT: forward request (carries trace_id)
  CHAT->>AI: summarise (carries trace_id)
  AI->>BR: get persona (carries trace_id)
  BR-->>AI: persona
  AI-->>CHAT: completion
  CHAT->>BR: append message row
  CHAT-->>AR: ok
  AR-->>U: ok
  Note over AR,TEMPO: each service emits its own span -<br/>all share trace_id=t1
  AR->>TEMPO: span (root)
  AUTH->>TEMPO: span
  CHAT->>TEMPO: span
  AI->>TEMPO: span
  BR->>TEMPO: span (x2)

(task pending): end-to-end trace continuity verified. W3C TraceContext propagation through every internal call. One trace_id stitches the whole transaction.

Flow 4 - alert escalation (severity-based routing)

sequenceDiagram
  autonumber
  participant R as Prometheus rule
  participant AM as Alert Manager
  participant PD as PagerDuty
  participant CHAT as CHAT bot
  participant CUO as CUO digest queue
  participant OPS as On-call engineer
  R->>AM: alert {severity, service, summary}
  alt severity=critical (SLO burn fast, error_budget below 5%)
    AM->>PD: trigger incident
    AM->>CHAT: post to incidents channel
    PD->>OPS: page
    OPS->>CHAT: acknowledge
    OPS->>OPS: investigate via Grafana
  else severity=warning (burn slow, budget below 30%)
    AM->>CHAT: post to cyberos-alerts channel
    Note over OPS: handled async, SLA 4h
  else severity=info (trend, advisory)
    AM->>CUO: enqueue for morning digest
    Note over CUO: surfaced in the CEO morning brief
  end

(task pending): PagerDuty for critical, CHAT for low, CUO digest for trends.

Flow 5 - audit-log query (compliance review)

sequenceDiagram
  autonumber
  participant AUD as Auditor
  participant G as Grafana
  participant QP as query_proxy
  participant AUTH as AUTH RBAC
  participant AV as audit_view.rs
  participant BR as memory binlog
  AUD->>G: open the "Audit 2026-Q1" dashboard
  G->>QP: GET /api/v1/audit/query?since=2026-01-01&actor=stephen@...
  QP->>AUTH: RBAC.Check(action="obs.audit_read", resource=...)
  AUTH-->>QP: allow (auditor scope)
  QP->>AV: query(filter)
  AV->>BR: walk binlog from seq=12000
  BR-->>AV: rows matching filter
  AV-->>QP: ChainedAuditRow
  QP-->>G: rows + inclusion proofs (optional)
  G-->>AUD: table view with a verify button per row

EU AI Act Art. 12: decision logs retained >= 6 months; PDPL Art. 14 DSAR; auditors get read-only access scoped by engagement.

Alert lifecycle

Alerts traverse a five-state lifecycle. Every state transition emits a metric for SLO compliance tracking.

stateDiagram-v2
  [*] --> Inactive: rule loaded
  Inactive --> Pending: condition met within for_seconds window
  Pending --> Inactive: condition cleared before window
  Pending --> Firing: condition held for for_seconds duration
  Firing --> Acknowledged: on-call ack
  Firing --> Silenced: operator silences
  Acknowledged --> Resolving: investigation in progress
  Resolving --> Resolved: condition cleared
  Silenced --> Firing: silence expires
  Resolved --> Inactive
  Resolved --> Postmortem: severity = critical
  Postmortem --> [*]

SLO catalogue (P0)

Functional requirements

The CyberOS task catalogue is being rebuilt one feature at a time via the open task-author Agent Skill.

Previous task enumerations were archived 2026-05-14 and are no longer reflected on this page. Specific tasks land here as they are re-authored.

Non-functional requirements

Dependencies

graph LR
  subgraph upstream ["OBS depends on"]
    AUTH["AUTH<br/>tenant + scope verification"]
    memory["memory<br/>audit-log surface"]
    S3["S3<br/>Loki/Tempo storage"]
    LANGSMITH["LangSmith SaaS<br/>AI traces"]
  end
  OBS["OBS"]
  subgraph emitters ["Every CyberOS service emits OTel"]
    AUTH2["AUTH"]
    AI["AI"]
    MCP["MCP"]
    CHAT["CHAT"]
    BR2["memory"]
    SK["Skill"]
    OTH["...all 24"]
  end
  subgraph consumers ["Consumers"]
    OPS["On-call ops"]
    COMP["Compliance"]
    AUDIT["External auditor"]
    CEO["CEO morning digest"]
  end
  AUTH --> OBS
  memory --> OBS
  S3 --> OBS
  LANGSMITH --> OBS
  AUTH2 --> OBS
  AI --> OBS
  MCP --> OBS
  CHAT --> OBS
  BR2 --> OBS
  SK --> OBS
  OTH --> OBS
  OBS --> OPS
  OBS --> COMP
  OBS --> AUDIT
  OBS --> CEO
  classDef shipped fill:#f5ede6,stroke:#45210e
  classDef planned fill:#fef6e0,stroke:#7c3aed
  class memory,SK shipped
  class OBS,AUTH,AI,MCP,CHAT,AUTH2,BR2,OTH planned
  class S3,LANGSMITH planned

Compliance scope

Risk entries

KPIs

RACI matrix

Planned CLI surface

Operator CLI cyberos-obs plus the standard Grafana + Loki + Prometheus CLIs.

1. Tail logs for a tenant

$ cyberos-obs logs tail --tenant acme --service auth --since 5m

2026-05-14T07:19:02Z INFO auth login_attempt subject=[REDACTED:email] trace=t_3ab9
2026-05-14T07:19:02Z INFO auth login_success aal=aal3 trace=t_3ab9
2026-05-14T07:19:03Z INFO rbac check action=memory.put decision=allow trace=t_3ab9
...

2. SLO status

$ cyberos-obs slo status

SERVICE INDICATOR TARGET CURRENT BUDGET BURN
platform availability 99.5% 99.94% 99% 0.06x (28d)
chat availability 99.9% 99.97% 71% 1.2x (warning)
auth availability 99.95% 100% 100% 0x
ai-gateway latency p95 2 s 1.4 s ok -
memory-search availability 99.5% 99.99% 99% 0x
mcp-gateway write p95 1 s 0.42 s ok -
graphql-router latency p95 400 ms 280 ms ok -

3. Active alerts

$ cyberos-obs alerts active

ALERT SEVERITY STARTED STATUS
ChatErrorBudgetBurnFast warning 5m ago firing
AIProviderLatencyHigh info 12m ago firing
S3LifecycleStaleConfig (cost-bucket) info 2h ago silenced

4. Per-tenant cost MTD

$ cyberos-obs cost mtd --tenant acme

TENANT: acme
MONTH: 2026-05
-------------------------------------
Infra: $182.40
 Fargate (chat) $52.10
 Fargate (auth) $48.20
 RDS Postgres $42.10
 S3 storage $24.00
 Other $16.00
AI: $97.42 (cap $150, 64.9%)
Storage: $24.00
-------------------------------------
TOTAL: $303.82

5. Trace lookup by id

$ cyberos-obs trace get t_3ab9c8d4

trace_id: t_3ab9c8d4
duration: 412 ms
spans:
 apollo-router sendMessage(graphql) 412 ms
 ├─ auth RBAC.Check 8 ms
 ├─ chat CreateMessage 286 ms
 │ ├─ memory put_message 12 ms
 │ └─ ai-gateway summariseSync 260 ms
 │ ├─ tenant_policy 3 ms
 │ ├─ redactor 2 ms
 │ └─ bedrock invoke 254 ms
 └─ chat FanoutMentions 14 ms

6. Audit-log query (compliance)

$ cyberos-obs audit query --since 2026-04-01 --action 'memory.delete' --format jsonl
{"seq":12031,"action":"memory.delete","actor":"stephen@...","mode":"tombstone","path":"memories/...","ts":"..."}
{"seq":12102,"action":"memory.delete","actor":"dpo@...","mode":"purge","reason":"DSAR-2026-014","path":"memories/...","ts":"..."}
...
[query] 47 rows, chain integrity verified

7. SLO definition (YAML)

# cyberos-obs/slo/ai-gateway-latency.yml
slo:
  id: ai-gateway-latency-p95
  service: ai-gateway
  indicator: latency_p95
  target: 2.0 # seconds
  window: 28d
  alerts:
    burn_rate_fast:
      severity: critical
      route: pagerduty
      threshold: 2.0 # 2x burn over 1h
    burn_rate_slow:
      severity: warning
      route: chat
      threshold: 1.0 # 1x burn over 6h

Phase status & estimates

• Status: planned - P0; slice 1 in build
• Est. LoC: ~3,000 - Rust query proxy + Go collector configs
• SLOs at P0: ~11 - platform + per-module
• P0 budget: ~$130/mo - LGTM hosting + LangSmith starter
• Decision-log retention: >= 180 d - EU AI Act Art. 12
• PII recall target: >= 99.5% - (task pending)

References

• Bigger picture (above): 3 strategic roles + emitter/consumer diagram + auto-vs-human matrix.
• Three-pillars unified pane (above): pillar x signal-type table + cross-pillar correlation example + the tenant query proxy guarantee.
• Auto-runbook router (above): 6-step routing sequence + severity x routing matrix + the runbook-catalogue growth loop.
• Compliance evidence surface (above): regulator view x audit scope matrix + per-view scoping YAML + the chain-of-custody manifest.
• Cross-module page links: memory.html, auth.html, ai.html, cuo.html, mcp.html, kb.html, chat.html, proj.html
• Build-readiness audit: archive/2026-05-14/AUDIT_AND_PLAN.md (archived; see cyberos/CHANGELOG.md) - OBS placed at P0 slice 1 alongside AI Gateway as the two earliest P0 modules.
• Research review: archive/2026-05-14/RESEARCH_REVIEW.md (archived; see cyberos/CHANGELOG.md) - OBS rated "Strong" (9/10); the auto-runbook-router framing flagged as the most differentiated operational play.
• Memory auto-sync vision: MEMORY_AUTOSYNC_DESIGN.md §8 - OBS reads from memory's audit chain for compliance exports; it never writes back (memory is upstream).
• task authoring discipline: modules/skill/task-audit/AUTHORING_DISCIPLINE.md.
• EU AI Act (Reg. 2024/1689) - Art. 12 logging, Art. 13 transparency, Art. 14 human oversight, Art. 26 deployer obligations.
• ISO/IEC 27001:2022 - A.8.15 logging, A.8.16 monitoring activities.
• ISO/IEC 42001 (AIMS) - § 9.1 performance evaluation.
• SOC 2 Type II - CC7.2 monitoring, CC7.3 evaluation, CC7.4 incident response evidence.
• Vietnam PDPL (Law 91/2025) - Art. 14 DSAR transparency, Art. 20 security obligations.
• Vietnam Decree 13/2023 - Art. 17 processing log requirement.
• GDPR (EU 2016/679) - Art. 30 Records of Processing Activities.
• OpenTelemetry - specification + Rust + Python SDKs; W3C TraceContext propagation.
• Grafana Loki + Tempo + Mimir - upstream LGTM stack.
• LangSmith - managed AI-trace observability (EU-residency tenants use a self-hosted equivalent).
• Sloth - SLO-as-code engine (Prometheus rule generator).
• Architecture context: infrastructure.html#obs.

Previous: MCP Gateway | Next module: CHAT

Changelog

History lives in the changelog; this page describes only the current state.

Generated from services/obs-collector/docs/index.md — edit the markdown source, not this file (TASK-DOCS-002).