§1 — Description (BCP-14 normative)

The OTel collector MUST apply tail-based sampling per the following policies (highest precedence first):

1. MUST sample 100% of traces with any span where status.code = ERROR.
2. MUST sample 100% of traces with any span carrying HTTP status ≥ 500 (http.status_code or http.response.status_code attribute, depending on OTel semantic-convention version).
3. MUST sample 100% of traces with tenant_id in the flagged-tenants list. The list is configured via deploy/obs/flagged_tenants.yaml (one tenant_id per line); operators add via cyberos-ai flag-tenant <id> --confirm (TASK-AI-021 subcommand). Hot-reload on file change (no collector restart).
4. MUST sample 100% of traces with end-to-end latency above per-route p99 budget. Default budgets per route (loaded from deploy/obs/route_latency_budgets.yaml):

• ai-gateway:/v1/chat/completions → 5000ms
• ai-gateway:/v1/embeddings → 500ms
• auth-service:/v1/auth/token → 250ms
• default for unspecified routes → 2000ms

1. MUST sample 10% of normal traces (no error, no 5xx, not flagged tenant, not slow). Probabilistic sampling on trace_id hash.
2. MUST evaluate policies in order; first-match wins. A trace matching error AND slow AND normal is sampled exactly once (counted as "error" in metrics).
3. MUST emit metric obs_sampled_traces_total{reason} per sampling decision (reason ∈ error | http_5xx | flagged_tenant | slow | normal_sample | dropped).
4. MUST wait decision_wait: 30s for trace completion before applying policies. Traces in flight at decision time are buffered; full-trace evaluation (all spans seen) is the policy input.
5. MUST size num_traces: 100000 (in-flight buffer cap). At slice-2 load (~25K spans/sec), this caps memory at ~1GB.
6. MUST route the sampling processor only on the TRACES pipeline (not logs, not metrics — those have their own retention controls).
7. MUST support hot-reload of flagged_tenants.yaml via collector's file-watch extension (no restart needed). Reload latency ≤ 30s.
8. MUST emit obs_sampling_buffer_depth gauge (current count of in-flight traces); sev-2 alarm when > 90% of num_traces.
9. SHOULD support per-tenant sampling rate override (e.g., one tenant requests 50% sampling for their own monitoring). Tenant-policy field obs_sampling_rate overrides the 10% default.

§2 — Why this design (rationale for humans)

Why tail-based not head-based (DEC-165)? Head-based sampling decides at trace start without knowing if the trace will error. Result: 90% of error traces (the high-value cases) get sampled out probabilistically. Tail-based waits until trace completes (all spans seen), then decides — error traces ALWAYS captured.

Why 100% on errors + 5xx (§1 #1, #2)? Errors are the high-value debugging cases. Probabilistic sampling on them defeats the purpose of having traces. The cost is small: error rate is typically <1% of traffic, so 100%×1% + 10%×99% ≈ 11% effective rate vs uniform 10%.

Why 100% on slow traces (§1 #4)? A slow call is a debugging case ("why was this 5x normal latency?"). Tempo with full span detail is the answer. Probabilistic sampling would lose the slow trace 90% of the time. Per-route p99 budgets adapt to each route's normal latency profile.

Why flagged-tenants override (§1 #3)? When a tenant complains "calls are slow today," ops needs ALL their traces for the investigation window — not 10%. Flagging captures the next ~hour at 100%; ops investigates; un-flag when done. Hot-reload makes this fast.

Why 30s decision_wait (§1 #8)? Long enough that slow traces (up to 30s) complete + are sampled by latency policy. Shorter (10s) would miss long traces — they'd be evaluated mid-flight and the slow status wouldn't trigger.

Why first-match policy ordering (§1 #6)? Avoids double-counting AND avoids ambiguous "is this an error or a slow?" — pick one based on order.

Why hot-reload via file-watch (§1 #11)? Collector restart loses 30s of in-flight traces (decision_wait window). Hot-reload preserves the buffer; sampling decisions just start using the new flagged_tenants list. Critical for ops responsiveness during incidents.

Why num_traces 100K (§1 #9)? Sizing math: 25K spans/sec × 30s decision_wait × ~3 spans/trace = 250K spans buffered. With 100K trace limit (each trace averaging 3 spans), buffer ≈ 1GB. Higher would over-provision memory; lower would drop in-flight traces during traffic bursts.

Why per-route latency budgets (§1 #4)? A 5s ai-gateway call is normal (LLM round-trip); a 5s auth call is catastrophic. One global threshold can't capture both. Per-route lets ops express "what's slow for THIS route."

Why metric per sampling decision (§1 #7)? Operators need to know "what fraction of error traces are we keeping?" The metric answers directly. Without it, sampling decisions are invisible.

§3 — API contract

Tail sampling config

# deploy/obs/tail_sampling_config.yaml
processors:
  tail_sampling:
    decision_wait: 30s
    num_traces: 100000
    expected_new_traces_per_sec: 1000
    policies:
      # Order matters — first match wins.
      - name: errors_100pct
        type: status_code
        status_code: { status_codes: [ERROR] }

      - name: http_5xx_100pct
        type: numeric_attribute
        numeric_attribute:
          key: http.response.status_code
          min_value: 500
          max_value: 599

      - name: flagged_tenants_100pct
        type: string_attribute
        string_attribute:
          key: tenant_id
          values: ${flagged_tenants}   # populated from flagged_tenants.yaml

      - name: slow_traces_100pct
        type: latency
        latency: { threshold_ms: 2000 }   # default; per-route via composite policy below

      - name: per_route_slow
        type: composite
        composite:
          composite_sub_policy:
            - name: chat_completion_slow
              type: latency
              latency: { threshold_ms: 5000 }
            - name: chat_completion_route
              type: string_attribute
              string_attribute: { key: route, values: [/v1/chat/completions] }
          rate_allocation:
            - { policy: chat_completion_slow, percent: 100 }

      - name: normal_sample_10pct
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

Modified collector config

# deploy/obs/otel-collector-config.yaml (modified)
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resource, attributes/pii_scrub, tail_sampling, batch]
      exporters: [otlp/tempo]

Flag-tenant CLI

// services/ai-gateway/src/cli/flag_tenant.rs (TASK-AI-021 subcommand)
pub async fn flag_tenant(tenant_id: Uuid, confirm: bool) -> Result<(), CliError> {
    require_confirm(confirm)?;
    let path = "deploy/obs/flagged_tenants.yaml";
    let mut current: Vec<String> = serde_yaml::from_str(&std::fs::read_to_string(path)?)?;
    if current.contains(&tenant_id.to_string()) {
        return Err(CliError::AlreadyFlagged);
    }
    current.push(tenant_id.to_string());
    std::fs::write(path, serde_yaml::to_string(&current)?)?;
    // Collector picks up via file-watch within 30s
    memory::emit(canonical::tenant_flagged_for_sampling(tenant_id, claims.subject_id, request_id)).await?;
    Ok(())
}

§4 — Acceptance criteria

1. 100 normal traces → ~10 sampled (10% probabilistic).
2. 1 error trace among 99 normal → error sampled + ~10 normal sampled (error 100%).
3. 1 5xx trace among 99 normal → 5xx sampled + ~10 normal.
4. 1 slow trace (3s, default route) among 99 normal → slow + ~10 normal.
5. chat-completions trace at 6s sampled (per-route 5000ms budget exceeded).
6. Flagged tenant: all traces sampled — flag tenant_id; emit 100 traces from that tenant; all 100 in Tempo.
7. Sampling metric per decision — obs_sampled_traces_total{reason="error"} increments by 1 per error trace sampled.
8. decision_wait blocks early decisions — trace completing in 5s evaluated only after decision_wait expires.
9. Hot-reload of flagged_tenants — modify file; within 30s, new tenant's traces sampled at 100% without collector restart.
10. First-match precedence — error+slow trace counted as reason="error" not reason="slow".
11. num_traces buffer cap — burst of 200K traces → oldest dropped; metric obs_sampled_traces_total{reason="dropped"} increments.
12. Buffer-depth metric — obs_sampling_buffer_depth reflects in-flight count.
13. Per-route latency budget loaded from yaml — change route_latency_budgets.yaml; policy uses new threshold.

§5 — Verification

# deploy/obs/tests/sampling_test.sh
#!/usr/bin/env bash
set -euo pipefail

echo "Sending 100 normal traces + 1 error + 1 slow + 1 5xx..."
for i in $(seq 1 100); do
    emit_trace --trace-id "normal_$i" --duration-ms 500 --status ok
done
emit_trace --trace-id "error_1" --duration-ms 500 --status error
emit_trace --trace-id "slow_1" --duration-ms 3000 --status ok
emit_trace --trace-id "5xx_1" --duration-ms 500 --status ok --http-status 503

echo "Waiting for decision_wait + processing..."
sleep 40

# Query Tempo for sampled traces
sampled_normal=$(tempo_query --filter "name=normal_*" | jq 'length')
sampled_error=$(tempo_query --filter "name=error_1" | jq 'length')
sampled_slow=$(tempo_query --filter "name=slow_1" | jq 'length')
sampled_5xx=$(tempo_query --filter "name=5xx_1" | jq 'length')

# Assert
[ "$sampled_normal" -ge 7 ] && [ "$sampled_normal" -le 13 ] || { echo "FAIL: normal count $sampled_normal not in [7,13]"; exit 1; }
[ "$sampled_error" -eq 1 ] || { echo "FAIL: error not sampled"; exit 1; }
[ "$sampled_slow" -eq 1 ] || { echo "FAIL: slow not sampled"; exit 1; }
[ "$sampled_5xx" -eq 1 ] || { echo "FAIL: 5xx not sampled"; exit 1; }
echo "✅ sampling_test passed"

# Hot-reload test
flagged_yaml="deploy/obs/flagged_tenants.yaml"
echo "- 550e8400-e29b-41d4-a716-446655440000" > "$flagged_yaml"
sleep 35   # wait for file-watch reload
for i in $(seq 1 50); do
    emit_trace --trace-id "flagged_$i" --tenant-id "550e8400-..." --duration-ms 500 --status ok
done
sleep 40
flagged_count=$(tempo_query --filter "tenant_id=550e8400" | jq 'length')
[ "$flagged_count" -eq 50 ] || { echo "FAIL: only $flagged_count/50 flagged traces sampled"; exit 1; }

§6 — Implementation skeleton

See §3.

§7 — Dependencies

• TASK-OBS-001 — OTel collector running.
• TASK-OBS-005 — trace_id propagation (sampling decision is per-trace, not per-span).
• TASK-AI-021 — operator CLI for flag-tenant.

§8 — Example payloads

Sampling metric

obs_sampled_traces_total{reason="error"} 423
obs_sampled_traces_total{reason="http_5xx"} 18
obs_sampled_traces_total{reason="flagged_tenant"} 1247
obs_sampled_traces_total{reason="slow"} 89
obs_sampled_traces_total{reason="normal_sample"} 9842
obs_sampled_traces_total{reason="dropped"} 0

Flagged tenants config

# deploy/obs/flagged_tenants.yaml
- 550e8400-e29b-41d4-a716-446655440000   # added 2026-05-15 — investigating customer complaint

obs.tenant_flagged_for_sampling audit row

{
  "kind": "obs.tenant_flagged_for_sampling",
  "payload": {
    "tenant_id": "550e8400-...",
    "flagged_by_subject_id": "...",
    "request_id": "cli_..."
  }
}

§9 — Open questions

All resolved. Deferred:

• Adaptive sampling (auto-adjust rate based on volume) — slice 5+.
• Per-tenant sampling override beyond flag — slice 4+ with TASK-AI-005 schema extension.
• Sampling for specific user_ids (not just tenants) — slice 5+.

§10 — Failure modes inventory

§11 — Notes

• The 30s decision_wait is the standard. Don't reduce below 20s (some slow traces won't complete in time).
• Tail-based sampling at the collector is the sole sampling decision point — services emit 100% (TASK-AI-022 §1 #10), the collector reduces.
• Flagged-tenants is the operational primitive for "investigate this tenant deeply." Flag → wait → un-flag pattern.
• num_traces 100K caps memory at ~1GB at slice-2 load. Higher would over-provision; lower would drop traces during bursts.
• Per-route latency budgets recognise that "slow" is route-specific. ai-gateway 5s is normal; auth 5s is incident.
• The CLI cyberos-ai flag-tenant is the operator surface; emits memory audit row; 24h auto-unflag is TASK-AI-021 enhancement (slice 4+).

End of TASK-OBS-006. Status: draft (10/10 target).