Task — engineering-spec@1

"TEN tenant provisioning CLI — `cyberos-ten provision` ops-driven flow with schema namespace + NATS subject + S3 prefix + initial root-admin subject + memory audit"

draftTASK-TEN-001
module ten · class product · priority p0 · created 2026-05-16 · shipped null
depends on TASK-AUTH-001 · blocks TASK-TEN-002, TASK-TEN-004, TASK-TEN-101, TASK-TEN-103, TASK-TEN-104

§1 — Description (BCP-14 normative)

The TEN service MUST ship the cyberos-ten provision CLI as the canonical ops-driven tenant provisioning flow. Each requirement:

  1. MUST define the tenants table with: id UUID PRIMARY KEY, slug TEXT NOT NULL UNIQUE CHECK (slug ~ '^[a-z][a-z0-9-]{2,40}[a-z0-9]$'), display_name TEXT NOT NULL CHECK (length(display_name) BETWEEN 1 AND 200), status tenant_status NOT NULL DEFAULT 'provisioning', plan_tier TEXT NOT NULL DEFAULT 'starter' (placeholder until TASK-TEN-002 ships full enum), residency residency_code NOT NULL DEFAULT 'vn-1', created_at TIMESTAMPTZ NOT NULL DEFAULT now(), provisioned_at TIMESTAMPTZ, terminated_at TIMESTAMPTZ, provisioned_by_subject_id UUID NOT NULL REFERENCES auth.subjects(id). Full DDL in §3.1.
  1. MUST declare the closed tenant_status Postgres enum with exactly 5 values (per DEC-326): 'provisioning', 'active', 'suspended', 'terminating', 'terminated'. State transitions land in TASK-TEN-104; slice 1 sets provisioning → active at end of successful provisioning.
  1. MUST declare the closed residency_code Postgres enum with exactly 4 values: 'vn-1', 'sg-1', 'eu-1', 'us-1'. Adding a 5th is an ADR (mirrors the global-residency pattern).
  1. MUST ship the cyberos-ten provision CLI subcommand accepting flags:
  1. MUST be idempotent on slug (per DEC-329). Re-running with same --slug AND same --residency AND same --plan-tier returns the existing tenant + exit code 1 (idempotent-match). Same slug + different residency or plan_tier → exit code 65 with error slug_collision_different_attrs.
  1. MUST execute the provisioning flow as a transactional 5-step orchestration (per DEC-321 + DEC-322 + DEC-323 + DEC-325):
  1. MUST generate the initial root-admin password server-side using a cryptographically secure RNG (per DEC-331). Password shape: 32-char base62 (alphanumerics + safe symbols). The password is:
  1. MUST emit ten.tenant_provisioned memory audit row at the end of successful provisioning. The row carries {tenant_id, slug, display_name, status: 'active', plan_tier, residency, provisioned_by_subject_id_hash16, root_admin_subject_id_hash16, ts_ns}. The row does NOT carry the root-admin password (per DEC-331) or the email (PII-scrubbed to hash16 per TASK-MEMORY-111).
  1. MUST enforce RLS on tenants AND tenant_status_history with the superuser-only policy: USING (current_setting('auth.is_root_admin', true) = 'true'). Tenant management is a CyberSkill-operator privilege, not a tenant-admin privilege. Tenant-admins read their own tenant via TASK-TEN-107's SPA (slice 3+).
  1. MUST be append-only on tenants AND tenant_status_history at the SQL-grant layer (per DEC-328). REVOKE UPDATE, DELETE ON tenants, tenant_status_history FROM cyberos_app;. Status transitions create new rows in history table; the tenants.status column is updated via the elevated cyberos_provisioner role (used by this task's CLI + TASK-TEN-104's offboarding orchestrator).
  1. MUST use the cyberos_provisioner SQL role for the CLI's writes — distinct from cyberos_app (which has REVOKE'd UPDATE/DELETE). The provisioner role:
  1. MUST validate slug uniqueness at INSERT time via the UNIQUE(slug) constraint. Conflict → 23505 → handler maps to either exit 1 (idempotent match per §1 #5) or exit 65 (collision_different_attrs).
  1. MUST emit exit codes per task-audit skill rule 9 + DEC-330:
  1. MUST require the operator running the CLI to have the root-admin role per TASK-AUTH-101 (cross-tenant superuser). The CLI verifies via the operator's JWT before any state mutation. Missing role → exit 77 immediately.
  1. MUST complete the full provisioning flow in ≤ 30 seconds p95 (excludes any human input — slack for Postgres + NATS + S3 + AUTH round-trips). provision_perf_test asserts on a local stack.
  1. MUST print a structured success block to stdout on completion: `` ✓ Tenant provisioned tenant_id: <uuid> slug: <slug> display_name: <text> residency: <code> plan_tier: <code> postgres_schema: tenant_<slug> nats_namespace: tenant.<slug>.> s3_prefix: <tenant_id>/ root_admin_email: <email> === ROOT ADMIN PASSWORD (RECORD IMMEDIATELY) === <32-char-password> === END === ``

With --json, output is a single JSON object (omitting the password — it's printed in a dedicated stderr block).

  1. MUST emit OTel span ten.provision with attributes: slug, residency, plan_tier, outcome (success | idempotent | slug_collision | invalid_input | step_failed_<step> | permission_denied | timeout).
  1. MUST emit OTel metrics:
  1. MUST ship the tenant_residency_map table for downstream consumers: (tenant_id UUID PRIMARY KEY REFERENCES tenants(id), residency residency_code NOT NULL, set_at TIMESTAMPTZ, set_by_subject_id UUID). TASK-DOC-001's residency::resolve() + TASK-EMAIL-001's residency::resolve() + TASK-AI-016's residency policy all consume this table. Slice 1 ships the table; provisioning writes the initial row; TASK-TEN-103 ships per-tenant residency change.
  1. MUST create the tenant's memory audit-chain anchor (per DEC-332): emit a special chain-bootstrap row that becomes the genesis row for the tenant's audit chain. TASK-AI-003's memory_writer initialises the chain head from this row; subsequent rows chain to it.
  1. MUST create the AUTH side via TASK-AUTH-001's internal helper auth::admin::tenants::provision_tenant(slug, display_name, root_admin_email, root_admin_display_name, root_admin_password_hash, operator_subject_id) returning (tenant_id, root_admin_subject_id). This helper is exposed from TASK-AUTH-001 specifically for TASK-TEN-001's use; it is NOT exposed via REST (operator privilege only).
  1. MUST support cyberos-ten list returning id, slug, status, residency, plan_tier, created_at, provisioned_at for all tenants. Caller MUST have root-admin role.
  1. MUST support cyberos-ten get --slug <slug> returning full tenant detail (excluding root-admin password — never retrievable).
  1. MUST validate --root-admin-email against the standard email regex (same as TASK-AUTH-002 §1 #2) AND check it's NOT already an existing AUTH subject in any tenant.
  1. MUST treat slug as canonical and immutable post-provisioning. There is no --rename flag; slug rename requires manual SQL + ADR + downstream impact analysis (out of scope for slice 1).
  1. MUST support --dry-run flag: validates all inputs + checks slug availability + checks residency policy + simulates each step, but DOES NOT write anything. Returns exit 0 if all checks pass; exit 65/73/77 on issues. Used by TASK-TEN-101's signup form preflight (slice 3+).

§2 — Why this design (rationale for humans)

Why ops-driven CLI at slice 1, not self-serve signup (DEC-320)? Self-serve signup (TASK-TEN-101) needs a polished web form + Stripe payment + fraud detection + email-verification — substantial. Ops-driven CLI is 5h of work and gets the core lifecycle primitive landed. CyberSkill at P2 has ~10 vertical-pack customers; CLI is the right tool for the volume. P3 graduates to self-serve when volume justifies the additional surface.

Why per-tenant Postgres schema namespace (DEC-321, §1 #6)? RLS alone is a "shared schema, isolated by predicate" pattern — a single SQL bug (forgetting WHERE tenant_id = ?) leaks across tenants. Per-tenant schema namespace adds defence in depth: the tenant_<slug> schema becomes the search_path; tables in other tenants' schemas aren't even visible without explicit qualification. Combined with RLS on shared metadata tables (where multi-tenant queries are legitimate), the leak surface narrows significantly.

Why NATS subject namespace per tenant (DEC-322)? NATS subscriptions are subject-pattern based. A subscriber listening on > (catch-all) would receive every tenant's events — catastrophic data leak. Per-tenant prefix tenant.<slug>.> limits what a tenant's subscribers can listen to (enforced by NATS ACLs at the account level). Cross-tenant pub/sub requires explicit gateway routing — auditable + intentional.

Why S3 per-tenant prefix (DEC-323)? S3 bucket-level isolation is too coarse (would need a bucket per tenant — operationally painful). Prefix isolation (<tenant_id>/) lets all tenants share a bucket while bucket policies + IAM enforce per-prefix access. TASK-DOC-001 + TASK-EMAIL-001 use this prefix in their S3 keys; the marker object at <tenant_id>/.cyberos-tenant-marker is the prefix-creation evidence.

Why password printed once + zeroised (DEC-331, §1 #7)? Ops staff need to deliver the root-admin password to the new tenant's admin (out-of-band; e.g. via in-person handoff or encrypted channel). Persisting the password anywhere (file, log, memory audit) creates a leak surface. Printing once + immediate zeroise from CLI memory means the only persisted form is the bcrypt hash (per TASK-AUTH-002).

Why operator role check at root-admin (§1 #14)? Provisioning a tenant is a cross-tenant operation — the operator must be in tenant 0 (CyberSkill itself) with the root-admin role per TASK-AUTH-101. Allowing any tenant-admin would let any tenant operator create new tenants — privilege escalation.

Why exit code 1 for idempotent match (DEC-329, §1 #5, §1 #13)? Exit 0 = "I did the work"; exit 1 = "I didn't do the work because it was already done correctly". Operators scripting CLI invocations need to distinguish (e.g. provisioning script in CI should treat both as success but log differently). Standard "well-known" idiom matches git pull behaviour.

Why slug regex ^[a-z][a-z0-9-]{2,40}[a-z0-9]$ (§1 #1)? Slug is the canonical identifier: appears in Postgres schema name (tenant_<slug>), NATS namespace (tenant.<slug>.>), S3 marker, URLs. All these systems have different identifier rules — the intersection is lowercase alphanumeric + hyphens, starting alphabetic, no trailing hyphen. 4-42 char length covers typical company shortnames.

Why default residency vn-1 (DEC-324)? CyberSkill's home market is Vietnam; default to the home jurisdiction so operators don't accidentally provision EU/US tenants in VN-residency. The --residency flag is explicit override; TASK-TEN-103 will ship per-tenant residency change for migrations.

Why transactional 5-step orchestration (§1 #6)? Each step touches a different system (Postgres, NATS, S3, AUTH); partial failures leave orphan state (NATS account without DB row, etc.). Compensating actions (delete NATS account, delete S3 marker, delete AUTH tenant) on any step's failure ensures cleanup. The Postgres parts use a single transaction; NATS + S3 + AUTH have their own commit points but rollback via compensating action.

Why cyberos_provisioner SQL role distinct from cyberos_app (§1 #11)? App code (the runtime) MUST NOT mutate tenants — that's a privileged operation. Splitting roles enforces: the provisioner role is granted only to the CLI's connection; the app role REVOKE's UPDATE/DELETE. A bug in app code can't accidentally suspend a tenant.

Why memory chain anchor at provisioning (§1 #20, DEC-332)? Every tenant has its own audit chain (per AGENTS.md §6 — chained rows with prev_chain). The genesis row is the bootstrap anchor that memory_writer initialises from. Without it, the first audit row for the tenant has no prev_chain value and chain validation fails.

Why dry-run mode (§1 #26)? TASK-TEN-101's self-serve signup form needs to validate inputs in real-time (before committing payment). Dry-run lets the form check everything without state mutation — fast feedback, clean separation.

Why slug immutable post-provisioning (§1 #25)? Slug is in Postgres schema name + NATS namespace + URL paths + S3 prefixes. Renaming requires coordinated mutation across all systems + downstream consumer awareness. The 1% of cases needing rename are deliberate operator events with ADRs; the API doesn't offer a shortcut.

Why list + get subcommands (§1 #22, #23)? Operators need to inspect existing tenants. list for inventory; get for detail. Both are read-only and require root-admin role.

Why email uniqueness check across all tenants (§1 #24)? A subject's email is the cross-tenant join key for "I'm the same human in two tenants". Allowing the same email in two new tenants creates ambiguity. Slice 1 enforces single-tenant-email; task-AUTH-2xx may add multi-tenant binding later (out of scope).

Why 30s p95 perf budget (§1 #15)? Provisioning touches 4 external systems + creates ~10 rows. 30s is generous — typical happy path is 5-10s. The p95 budget catches infrastructure outliers; outliers > 30s = sev-3 alarm for ops investigation.

Why TenantStatus enum has 5 values (DEC-326)? Mirrors the conventional SaaS lifecycle: provisioning (mid-flight) → active → suspended (non-payment, etc.) → terminating (90-day offboarding contract) → terminated (irreversible wipe attestation). Slice 1 ships all 5 values; transitions implemented in TASK-TEN-104.

Why tenant_residency_map separate table (§1 #19)? Residency may change (rare but possible: tenant graduates from sg-1 to vn-1 for VN expansion). Separate table lets TASK-TEN-103 ship the change-residency workflow without touching the tenants table. The map is the canonical lookup for TASK-DOC-001 + TASK-EMAIL-001 + TASK-AI-016.

Why provisioned_by_subject_id recorded (§1 #1)? Accountability — every tenant has a creator operator. PDPL Art. 4 data minimisation is satisfied because the field stores the operator's UUID (already in the AUTH cluster), not their personal data.

Why no DELETE on tenants (§1 #10)? Tenants enter terminated state via TASK-TEN-104's 90-day offboarding contract; the row persists with status='terminated' and terminated_at set. Hard delete would lose the forensic record + would orphan FK references (audit rows, billing records).

Why JSON mode omits password (§1 #16)? JSON mode is intended for automation (scripts piping output to other systems). Including the password in JSON means it could end up in logs, command history, CI artifacts. Stderr block forces operators to handle it deliberately.

§3 — API contract

3.1 — Migration 0001 — tenants table

-- services/ten/migrations/0001_tenants.sql

BEGIN;

CREATE TYPE tenant_status AS ENUM ('provisioning', 'active', 'suspended', 'terminating', 'terminated');
CREATE TYPE residency_code AS ENUM ('vn-1', 'sg-1', 'eu-1', 'us-1');

CREATE TABLE tenants (
    id                          UUID         PRIMARY KEY,
    slug                        TEXT         NOT NULL UNIQUE
                                CHECK (slug ~ '^[a-z][a-z0-9-]{2,40}[a-z0-9]$'),
    display_name                TEXT         NOT NULL CHECK (length(display_name) BETWEEN 1 AND 200),
    status                      tenant_status NOT NULL DEFAULT 'provisioning',
    plan_tier                   TEXT         NOT NULL DEFAULT 'starter',
    residency                   residency_code NOT NULL DEFAULT 'vn-1',
    created_at                  TIMESTAMPTZ  NOT NULL DEFAULT now(),
    provisioned_at              TIMESTAMPTZ,
    terminated_at               TIMESTAMPTZ,
    provisioned_by_subject_id   UUID         NOT NULL REFERENCES auth.subjects(id) ON DELETE RESTRICT
);

CREATE INDEX tenants_status_idx ON tenants (status);
CREATE INDEX tenants_residency_idx ON tenants (residency);

ALTER TABLE tenants ENABLE ROW LEVEL SECURITY;

-- Only superuser / root-admin can see this table; tenant-admins read their own tenant via TASK-TEN-107.
CREATE POLICY tenants_superuser_only ON tenants
    USING (current_setting('auth.is_root_admin', true) = 'true')
    WITH CHECK (current_setting('auth.is_root_admin', true) = 'true');

-- Provisioner role for the CLI; distinct from cyberos_app.
CREATE ROLE cyberos_provisioner;
GRANT INSERT ON tenants TO cyberos_provisioner;
GRANT UPDATE (status, provisioned_at, terminated_at) ON tenants TO cyberos_provisioner;
GRANT SELECT ON tenants TO cyberos_provisioner;

REVOKE UPDATE, DELETE ON tenants FROM cyberos_app;

COMMIT;

3.2 — Migration 0002 — tenant_status_history (append-only)

-- services/ten/migrations/0002_tenant_status_history.sql

BEGIN;

CREATE TABLE tenant_status_history (
    id                     BIGSERIAL    PRIMARY KEY,
    tenant_id              UUID         NOT NULL REFERENCES tenants(id),
    from_status            tenant_status,                              -- NULL on initial create
    to_status              tenant_status NOT NULL,
    changed_at             TIMESTAMPTZ  NOT NULL DEFAULT now(),
    changed_by_subject_id  UUID         NOT NULL,
    reason                 TEXT
);

CREATE INDEX tenant_status_history_tenant_idx ON tenant_status_history (tenant_id, changed_at DESC);

ALTER TABLE tenant_status_history ENABLE ROW LEVEL SECURITY;
CREATE POLICY tenant_status_history_superuser_only ON tenant_status_history
    USING (current_setting('auth.is_root_admin', true) = 'true')
    WITH CHECK (current_setting('auth.is_root_admin', true) = 'true');

GRANT INSERT, SELECT ON tenant_status_history TO cyberos_provisioner;
REVOKE UPDATE, DELETE ON tenant_status_history FROM cyberos_app;

COMMIT;

3.3 — Migration 0003 — tenant_residency_map

-- services/ten/migrations/0003_tenant_residency_map.sql

BEGIN;

CREATE TABLE tenant_residency_map (
    tenant_id              UUID         PRIMARY KEY REFERENCES tenants(id),
    residency              residency_code NOT NULL,
    set_at                 TIMESTAMPTZ  NOT NULL DEFAULT now(),
    set_by_subject_id      UUID         NOT NULL
);

-- Allow cyberos_app read access — TASK-DOC-001, TASK-EMAIL-001, TASK-AI-016 all consume.
GRANT SELECT ON tenant_residency_map TO cyberos_app;
GRANT INSERT ON tenant_residency_map TO cyberos_provisioner;
REVOKE UPDATE, DELETE ON tenant_residency_map FROM cyberos_app;
-- UPDATE is granted to cyberos_provisioner for TASK-TEN-103's residency-change flow.
GRANT UPDATE (residency, set_at, set_by_subject_id) ON tenant_residency_map TO cyberos_provisioner;

COMMIT;

3.4 — Rust types

// services/ten/src/types.rs
use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};
use sqlx::{FromRow, Type};
use uuid::Uuid;

#[derive(Debug, Clone, Copy, PartialEq, Eq, Type, Serialize, Deserialize)]
#[sqlx(type_name = "tenant_status", rename_all = "snake_case")]
#[serde(rename_all = "snake_case")]
pub enum TenantStatus { Provisioning, Active, Suspended, Terminating, Terminated }

impl TenantStatus {
    pub const ALL: &'static [TenantStatus] = &[
        TenantStatus::Provisioning, TenantStatus::Active, TenantStatus::Suspended,
        TenantStatus::Terminating, TenantStatus::Terminated,
    ];
}

#[derive(Debug, Clone, Copy, PartialEq, Eq, Type, Serialize, Deserialize)]
#[sqlx(type_name = "residency_code", rename_all = "kebab-case")]
#[serde(rename_all = "kebab-case")]
pub enum ResidencyCode { Vn1, Sg1, Eu1, Us1 }

impl ResidencyCode {
    pub const ALL: &'static [ResidencyCode] = &[
        ResidencyCode::Vn1, ResidencyCode::Sg1, ResidencyCode::Eu1, ResidencyCode::Us1,
    ];

    pub fn as_str(self) -> &'static str {
        match self {
            ResidencyCode::Vn1 => "vn-1",
            ResidencyCode::Sg1 => "sg-1",
            ResidencyCode::Eu1 => "eu-1",
            ResidencyCode::Us1 => "us-1",
        }
    }
}

#[derive(Debug, FromRow, Serialize, Deserialize)]
pub struct Tenant {
    pub id: Uuid,
    pub slug: String,
    pub display_name: String,
    pub status: TenantStatus,
    pub plan_tier: String,
    pub residency: ResidencyCode,
    pub created_at: DateTime<Utc>,
    pub provisioned_at: Option<DateTime<Utc>>,
    pub terminated_at: Option<DateTime<Utc>>,
    pub provisioned_by_subject_id: Uuid,
}

3.5 — Provisioning orchestrator

// services/ten/src/provisioning/orchestrator.rs
use crate::types::*;
use uuid::Uuid;
use zeroize::Zeroizing;

pub struct ProvisionRequest {
    pub slug: String,
    pub display_name: String,
    pub root_admin_email: String,
    pub root_admin_display_name: String,
    pub residency: ResidencyCode,
    pub plan_tier: String,
    pub operator_subject_id: Uuid,
    pub dry_run: bool,
}

pub struct ProvisionResult {
    pub tenant_id: Uuid,
    pub slug: String,
    pub display_name: String,
    pub residency: ResidencyCode,
    pub plan_tier: String,
    pub postgres_schema: String,
    pub nats_namespace: String,
    pub s3_prefix: String,
    pub root_admin_subject_id: Uuid,
    pub root_admin_email: String,
    pub root_admin_password: Zeroizing<String>,   // printed once + zeroised
    pub idempotent_match: bool,
}

pub async fn provision(req: ProvisionRequest, ctx: &Ctx) -> Result<ProvisionResult, ProvisionError> {
    // Step 1: validate (fail-fast)
    crate::validation::slug(&req.slug)?;
    crate::validation::email(&req.root_admin_email)?;

    if req.dry_run {
        return crate::dry_run::simulate(req, ctx).await;
    }

    // Step 2: idempotency check
    if let Some(existing) = ctx.repo.find_by_slug(&req.slug).await? {
        if existing.residency != req.residency || existing.plan_tier != req.plan_tier {
            return Err(ProvisionError::SlugCollisionDifferentAttrs);
        }
        return Ok(ProvisionResult::from_existing(existing));
    }

    let mut tx = ctx.db.begin().await?;
    let tenant_id = Uuid::new_v4();

    // Step 3: INSERT tenant (status='provisioning')
    sqlx::query("INSERT INTO tenants (id, slug, display_name, status, plan_tier, residency, provisioned_by_subject_id) VALUES ($1, $2, $3, 'provisioning'::tenant_status, $4, $5::residency_code, $6)")
        .bind(tenant_id).bind(&req.slug).bind(&req.display_name)
        .bind(&req.plan_tier).bind(req.residency.as_str()).bind(req.operator_subject_id)
        .execute(&mut *tx).await?;
    sqlx::query("INSERT INTO tenant_status_history (tenant_id, from_status, to_status, changed_by_subject_id, reason) VALUES ($1, NULL, 'provisioning'::tenant_status, $2, 'initial provisioning')")
        .bind(tenant_id).bind(req.operator_subject_id).execute(&mut *tx).await?;
    sqlx::query("INSERT INTO tenant_residency_map (tenant_id, residency, set_by_subject_id) VALUES ($1, $2::residency_code, $3)")
        .bind(tenant_id).bind(req.residency.as_str()).bind(req.operator_subject_id)
        .execute(&mut *tx).await?;

    // Step 4: create Postgres schema namespace
    crate::provisioning::schema_namespace::create(&req.slug, &mut tx).await?;

    tx.commit().await?;

    // Step 5: create NATS namespace (compensating action on failure)
    let nats_namespace = format!("tenant.{}.>", req.slug);
    crate::provisioning::nats_namespace::create(&req.slug, &ctx.nats).await
        .map_err(|e| {
            // Compensate: delete Postgres schema + tenant rows.
            tokio::spawn(crate::provisioning::compensate::rollback(tenant_id, req.slug.clone(), ctx.clone()));
            e
        })?;

    // Step 6: create S3 prefix markers (compensating action on failure)
    let s3_prefix = format!("{tenant_id}/");
    crate::provisioning::s3_prefix::initialise(&tenant_id, req.residency, &ctx.s3).await
        .map_err(|e| {
            tokio::spawn(crate::provisioning::compensate::rollback_with_nats(tenant_id, req.slug.clone(), ctx.clone()));
            e
        })?;

    // Step 7: AUTH bootstrap — create AUTH tenant + initial root-admin subject
    let password = generate_password();
    let bootstrap = crate::provisioning::auth_bootstrap::call(
        tenant_id, &req.slug, &req.display_name,
        &req.root_admin_email, &req.root_admin_display_name,
        &password, req.operator_subject_id, &ctx.auth_client,
    ).await.map_err(|e| {
        tokio::spawn(crate::provisioning::compensate::rollback_with_nats_and_s3(tenant_id, req.slug.clone(), ctx.clone()));
        e
    })?;

    // Step 8: UPDATE tenant status to 'active'
    let mut tx2 = ctx.db.begin().await?;
    sqlx::query("UPDATE tenants SET status = 'active'::tenant_status, provisioned_at = now() WHERE id = $1")
        .bind(tenant_id).execute(&mut *tx2).await?;
    sqlx::query("INSERT INTO tenant_status_history (tenant_id, from_status, to_status, changed_by_subject_id, reason) VALUES ($1, 'provisioning'::tenant_status, 'active'::tenant_status, $2, 'provisioning complete')")
        .bind(tenant_id).bind(req.operator_subject_id).execute(&mut *tx2).await?;

    // Step 9: emit memory audit row
    crate::audit::tenant_events::emit_tenant_provisioned(
        &mut tx2, tenant_id, &req.slug, &req.display_name,
        req.plan_tier.clone(), req.residency, req.operator_subject_id, bootstrap.root_admin_subject_id,
    ).await?;

    tx2.commit().await?;

    Ok(ProvisionResult {
        tenant_id,
        slug: req.slug,
        display_name: req.display_name,
        residency: req.residency,
        plan_tier: req.plan_tier,
        postgres_schema: format!("tenant_{}", &req.slug),
        nats_namespace,
        s3_prefix,
        root_admin_subject_id: bootstrap.root_admin_subject_id,
        root_admin_email: req.root_admin_email,
        root_admin_password: password,
        idempotent_match: false,
    })
}

fn generate_password() -> Zeroizing<String> {
    use rand::distributions::{Alphanumeric, DistString};
    Zeroizing::new(Alphanumeric.sample_string(&mut rand::thread_rng(), 32))
}

3.6 — CLI command

// services/ten/src/cli/provision.rs
use clap::Parser;
use cyberos_cli_exit::ExitCode;
use crate::provisioning::orchestrator::{provision, ProvisionRequest, ProvisionError};
use crate::types::ResidencyCode;
use std::str::FromStr;

#[derive(Parser)]
pub struct ProvisionCmd {
    #[arg(long)] pub slug: String,
    #[arg(long)] pub display_name: String,
    #[arg(long)] pub root_admin_email: String,
    #[arg(long)] pub root_admin_display_name: String,
    #[arg(long, default_value = "vn-1")] pub residency: String,
    #[arg(long, default_value = "starter")] pub plan_tier: String,
    #[arg(long, default_value_t = false)] pub json: bool,
    #[arg(long, default_value_t = false)] pub dry_run: bool,
}

pub async fn run(cmd: ProvisionCmd, ctx: AppCtx) -> ExitCode {
    // Operator role check
    let operator = match ctx.current_operator().await {
        Ok(op) if op.has_role(Role::RootAdmin) => op,
        Ok(_) => { eprintln!("ERROR: caller must be root-admin"); return ExitCode::PermissionDenied; }
        Err(_) => return ExitCode::PermissionDenied,
    };

    let residency = match ResidencyCode::from_str(&cmd.residency) {
        Ok(r) => r,
        Err(_) => { eprintln!("ERROR: unknown residency: {}", cmd.residency); return ExitCode::InvalidData; }
    };

    let req = ProvisionRequest {
        slug: cmd.slug,
        display_name: cmd.display_name,
        root_admin_email: cmd.root_admin_email,
        root_admin_display_name: cmd.root_admin_display_name,
        residency,
        plan_tier: cmd.plan_tier,
        operator_subject_id: operator.subject_id,
        dry_run: cmd.dry_run,
    };

    match provision(req, &ctx).await {
        Ok(result) => {
            if cmd.json {
                let json = serde_json::to_string_pretty(&result.public_view()).unwrap();
                println!("{json}");
                eprintln!("\n=== ROOT ADMIN PASSWORD (RECORD IMMEDIATELY) ===");
                eprintln!("{}", *result.root_admin_password);
                eprintln!("=== END ===");
            } else {
                println!("✓ Tenant provisioned");
                println!("  tenant_id:        {}", result.tenant_id);
                println!("  slug:             {}", result.slug);
                println!("  display_name:     {}", result.display_name);
                println!("  residency:        {}", result.residency.as_str());
                println!("  plan_tier:        {}", result.plan_tier);
                println!("  postgres_schema:  {}", result.postgres_schema);
                println!("  nats_namespace:   {}", result.nats_namespace);
                println!("  s3_prefix:        {}", result.s3_prefix);
                println!("  root_admin_email: {}", result.root_admin_email);
                println!("  === ROOT ADMIN PASSWORD (RECORD IMMEDIATELY) ===");
                println!("  {}", *result.root_admin_password);
                println!("  === END ===");
            }
            if result.idempotent_match { ExitCode::IdempotentMatch } else { ExitCode::Success }
        }
        Err(ProvisionError::SlugCollisionDifferentAttrs) => {
            eprintln!("ERROR: slug exists with different residency or plan_tier");
            ExitCode::InvalidData
        }
        Err(ProvisionError::InvalidSlug) => { eprintln!("ERROR: invalid slug (must match ^[a-z][a-z0-9-]{{2,40}}[a-z0-9]$)"); ExitCode::InvalidData }
        Err(ProvisionError::InvalidEmail) => { eprintln!("ERROR: invalid email format"); ExitCode::InvalidData }
        Err(ProvisionError::StepFailed(step)) => { eprintln!("ERROR: step {step} failed; rollback in progress"); ExitCode::CantCreate }
        Err(ProvisionError::Transient) => { eprintln!("ERROR: transient infrastructure failure; retry advised"); ExitCode::TempFail }
        Err(e) => { eprintln!("ERROR: {e:?}"); ExitCode::CantCreate }
    }
}

§4 — Acceptance criteria

  1. Tenant status enum closed at 5 — TenantStatus::ALL.len() == 5; Postgres enum has exactly 5 labels.
  2. Residency code enum closed at 4 — same shape.
  3. POST provision happy path — valid input → exit 0; tenant row created with status=active; tenant_<slug> Postgres schema exists; NATS namespace registered; S3 marker objects written; root-admin subject created; ten.tenant_provisioned memory row emitted; password printed once.
  4. Idempotent on slug — re-run with same slug + same residency + same plan_tier → exit 1; same tenant row returned; NO duplicate memory row.
  5. Slug collision different residency → exit 65 with slug_collision_different_attrs.
  6. Invalid slug regex — slug with uppercase or trailing hyphen → exit 65.
  7. Invalid email format → exit 65.
  8. Unknown residency → exit 65.
  9. Operator without root-admin role → exit 77 immediately (no state mutation).
  10. Missing required flag → exit 64.
  11. UPDATE on tenants blocked from cyberos_app — UPDATE tenants SET status='terminated' as cyberos_app → permission denied.
  12. DELETE on tenants blocked from cyberos_app — same.
  13. tenant_status_history append-only — UPDATE/DELETE blocked.
  14. cyberos-ten get returns tenant detail — exit 0; root_admin password NOT in output.
  15. cyberos-ten list returns all tenants — root-admin only.
  16. Root admin password printed once + zeroised — memory inspection after CLI exit (test harness) shows the password bytes overwritten.
  17. Root admin password NOT in memory row — ten.tenant_provisioned row JSON contains no password-shaped field.
  18. Default residency vn-1 — --residency omitted → tenant created with residency='vn-1'.
  19. --residency flag overrides — --residency eu-1 → tenant created with residency='eu-1'.
  20. --dry-run validates without writing — exit 0; no tenant row created; no schema/NATS/S3 mutation.
  21. --dry-run reports validation failure — bad slug → exit 65; no state.
  22. tenant_residency_map row written — readable by cyberos_app.
  23. NATS namespace ACL applied — subscriber on tenant.<other-slug>.> cannot receive messages on tenant.<slug>.>.
  24. S3 marker object exists — s3://cyberos-doc-vn-1-generic/<tenant_id>/.cyberos-tenant-marker returns 200 on HEAD.
  25. AUTH side created — POST /v1/admin/tenants on AUTH returns same tenant_id + root_admin_subject_id; root-admin subject has tenant-admin role per TASK-AUTH-101.
  26. Provision rolls back on AUTH failure — mock AUTH to return 500 → tenant row + NATS + S3 marker cleaned up via compensating actions; exit 73.
  27. OTel span ten.provision emitted — outcome=success.
  28. Counter ten_provision_total{outcome=success, residency=vn-1} increments — per provisioning.
  29. Perf budget < 30s p95 — provision_perf_test 50 iterations.
  30. --json mode omits password from stdout — password in stderr block only.

§5 — Verification

// services/ten/tests/provision_happy_test.rs
#[tokio::test]
async fn happy_path_creates_everything() {
    let ctx = TestStack::up().await;
    let result = ctx.run_cli(&[
        "provision",
        "--slug", "acme-corp",
        "--display-name", "ACME Corporation",
        "--root-admin-email", "root@acme.example",
        "--root-admin-display-name", "ACME Root Admin",
        "--residency", "vn-1",
    ]).await;
    assert_eq!(result.exit_code, 0);

    // Check tenant row
    let tenant = ctx.db.fetch_tenant_by_slug("acme-corp").await.unwrap();
    assert_eq!(tenant.status, TenantStatus::Active);
    assert!(tenant.provisioned_at.is_some());

    // Check Postgres schema
    let schemas = ctx.db.list_schemas().await;
    assert!(schemas.contains(&"tenant_acme-corp".to_string()));

    // Check NATS namespace
    let nats_accounts = ctx.nats.list_accounts().await;
    assert!(nats_accounts.iter().any(|a| a.subject_namespace == "tenant.acme-corp.>"));

    // Check S3 markers
    let marker = ctx.s3.head_object("cyberos-doc-vn-1-generic", &format!("{}/.cyberos-tenant-marker", tenant.id)).await;
    assert!(marker.is_ok());

    // Check AUTH-side root admin
    let auth_subject = ctx.auth.get_subject_by_email("root@acme.example").await.unwrap();
    assert!(auth_subject.roles.contains(&"tenant-admin".to_string()));

    // Check memory audit row
    let rows = ctx.memory_audit_rows("ten.tenant_provisioned").await;
    assert_eq!(rows.len(), 1);
    assert_eq!(rows[0]["slug"], "acme-corp");
}

#[tokio::test]
async fn idempotent_match_returns_existing() {
    let ctx = TestStack::up().await;
    let _first = ctx.run_cli(&["provision", "--slug", "acme", "--display-name", "ACME", "--root-admin-email", "r@acme", "--root-admin-display-name", "R", "--residency", "vn-1"]).await;
    let second = ctx.run_cli(&["provision", "--slug", "acme", "--display-name", "ACME", "--root-admin-email", "r@acme", "--root-admin-display-name", "R", "--residency", "vn-1"]).await;
    assert_eq!(second.exit_code, 1);
    let rows = ctx.memory_audit_rows("ten.tenant_provisioned").await;
    assert_eq!(rows.len(), 1, "no duplicate memory row on idempotent match");
}
// services/ten/tests/provision_root_admin_test.rs
#[tokio::test]
async fn password_printed_once_and_zeroised() {
    let ctx = TestStack::up().await;
    let result = ctx.run_cli_capture(&["provision", "--slug", "acme", /* ... */]).await;
    assert_eq!(result.exit_code, 0);

    // Password is in stdout (with marker block)
    assert!(result.stdout.contains("=== ROOT ADMIN PASSWORD"));
    let password_line = result.stdout.lines().find(|l| l.trim().len() == 32 && l.trim().chars().all(|c| c.is_alphanumeric())).unwrap();
    let password = password_line.trim().to_string();

    // Password is NOT in memory row
    let rows = ctx.memory_audit_rows("ten.tenant_provisioned").await;
    let row_json = serde_json::to_string(&rows[0]).unwrap();
    assert!(!row_json.contains(&password), "password leaked into memory row");

    // Password is NOT in log files
    let logs = ctx.read_log_files().await;
    assert!(!logs.contains(&password), "password leaked into logs");
}
// services/ten/tests/append_only_test.rs
#[sqlx::test]
async fn tenants_update_blocked_from_app(pool: sqlx::PgPool) {
    set_role_app(&pool).await;
    let id = seed_tenant_as_provisioner(&pool).await;
    let err = sqlx::query("UPDATE tenants SET status = 'terminated'::tenant_status WHERE id = $1")
        .bind(id).execute(&pool).await.unwrap_err();
    assert!(format!("{err}").contains("permission denied"));
}

#[sqlx::test]
async fn tenants_update_allowed_from_provisioner(pool: sqlx::PgPool) {
    set_role_provisioner(&pool).await;
    let id = seed_tenant_as_provisioner(&pool).await;
    sqlx::query("UPDATE tenants SET status = 'terminated'::tenant_status, terminated_at = now() WHERE id = $1")
        .bind(id).execute(&pool).await.unwrap();
}
// services/ten/tests/provision_namespace_isolation_test.rs
#[tokio::test]
async fn two_tenants_get_distinct_namespaces() {
    let ctx = TestStack::up().await;
    let a = ctx.run_cli(&["provision", "--slug", "acme", /* ... */]).await;
    let b = ctx.run_cli(&["provision", "--slug", "biko", /* ... */]).await;
    assert_eq!(a.exit_code, 0);
    assert_eq!(b.exit_code, 0);

    let schemas = ctx.db.list_schemas().await;
    assert!(schemas.contains(&"tenant_acme".to_string()));
    assert!(schemas.contains(&"tenant_biko".to_string()));

    let nats = ctx.nats.list_accounts().await;
    assert!(nats.iter().any(|a| a.subject_namespace == "tenant.acme.>"));
    assert!(nats.iter().any(|a| a.subject_namespace == "tenant.biko.>"));
}

§6 — Implementation skeleton

(API contract above is the skeleton; compensating-action functions follow the standard rollback pattern: delete the Postgres rows + NATS account + S3 marker if a later step fails.)

§7 — Dependencies

Upstream:

Downstream (2 placeholders):

Cross-module:

§8 — Example payloads

8.1 — cyberos-ten provision happy invocation

$ cyberos-ten provision \
    --slug acme-corp \
    --display-name "ACME Corporation" \
    --root-admin-email root@acme.example \
    --root-admin-display-name "ACME Root Admin" \
    --residency vn-1
✓ Tenant provisioned
  tenant_id:        01HG7V8B0K8M4Z8Z8M8M8M8M8M
  slug:             acme-corp
  display_name:     ACME Corporation
  residency:        vn-1
  plan_tier:        starter
  postgres_schema:  tenant_acme-corp
  nats_namespace:   tenant.acme-corp.>
  s3_prefix:        01HG7V8B0K8M4Z8Z8M8M8M8M8M/
  root_admin_email: root@acme.example
  === ROOT ADMIN PASSWORD (RECORD IMMEDIATELY) ===
  Kj7Lp9MzQrTyVwBxCdEfGhJ3Kl5Nm6Op
  === END ===

8.2 — ten.tenant_provisioned memory row

{
  "kind": "ten.tenant_provisioned",
  "tenant_id": "01HG7V8B0K8M4Z8Z8M8M8M8M8M",
  "slug": "acme-corp",
  "display_name": "ACME Corporation",
  "status": "active",
  "plan_tier": "starter",
  "residency": "vn-1",
  "provisioned_by_subject_id_hash16": "8a7c8c8012344567",
  "root_admin_subject_id_hash16": "9b1deb4d3b7d4bad",
  "ts_ns": 1747920731000000000
}

8.3 — JSON-mode output

$ cyberos-ten provision --slug acme-corp --display-name "ACME" --root-admin-email r@a --root-admin-display-name R --residency vn-1 --json
{
  "tenant_id": "01HG7V8B0K8M4Z8Z8M8M8M8M8M",
  "slug": "acme-corp",
  "display_name": "ACME",
  "residency": "vn-1",
  "plan_tier": "starter",
  "postgres_schema": "tenant_acme-corp",
  "nats_namespace": "tenant.acme-corp.>",
  "s3_prefix": "01HG7V8B0K8M4Z8Z8M8M8M8M8M/",
  "root_admin_email": "r@a",
  "root_admin_subject_id": "9b1deb4d-..."
}
=== ROOT ADMIN PASSWORD (RECORD IMMEDIATELY) ===
Kj7Lp9MzQrTyVwBxCdEfGhJ3Kl5Nm6Op
=== END ===

8.4 — Idempotent-match exit

$ cyberos-ten provision --slug acme-corp --display-name "ACME" --root-admin-email r@a --root-admin-display-name R --residency vn-1
✓ Tenant already exists (idempotent match)
  tenant_id:        01HG7V8B0K8M4Z8Z8M8M8M8M8M
  slug:             acme-corp
$ echo $?
1

§9 — Open questions

Deferred:

All other questions resolved.

§10 — Failure modes inventory

FailureDetectionOutcomeRecovery
Slug regex failDB CHECK + handlerexit 65 invalid_dataUse valid slug
Slug collision same attrsUNIQUE + handlerexit 1 idempotent_matchNone — designed
Slug collision different attrsUNIQUE + handler diff checkexit 65 collision_different_attrsUse different slug or align attrs
Invalid emailhandlerexit 65Use valid email
Unknown residencyenum parseexit 65Use vn-1 / sg-1 / eu-1 / us-1
Operator not root-adminJWT role checkexit 77Re-auth with root-admin role
Missing required flagclapexit 64Provide flag
Postgres schema create fails (permission)step 3 errorexit 73 + compensating rollbackCheck cyberos_provisioner role perms
NATS account create failsstep 4 errorexit 73 + rollback PostgresCheck NATS API health
S3 marker write failsstep 5 errorexit 73 + rollback Postgres + NATSCheck S3 perms + KMS
AUTH bootstrap failsstep 6 errorexit 73 + full rollbackInvestigate AUTH service health
Status transition provisioning → active failsstep 7 txexit 73; tenant left in provisioning stateOperator runs cleanup script
memory audit emit failsstep 8 tx rollbackexit 73; full rollbackmemory_writer diagnosis
Password generation failsstep 7 internalexit 75 transientRetry
RNG entropy starvationstep 7exit 75Restart process
Operator deletes the printed password before savingNone — operator responsibilityTenant unusable; force password-reset via AUTHtask-AUTH-2xx password reset flow
Postgres connection pool exhaustedsqlx errorexit 75Wait + retry
Cross-tenant slug-derived schema collision (e.g. SQL injection in slug)regex check + parameterised queryNoneRegex prevents
cyberos_provisioner role not granted to CLI's connectionstartup check failsService refuses to startGrant role
tenant_residency_map INSERT failsstep 3 tx rollbackexit 73Investigate cluster health
Idempotent re-run with different display_nameOK (idempotency only checks slug+residency+plan_tier)exit 1; display_name from existing rowUse cyberos-ten update display-name (task-TEN-2xx)
NATS compensating action failslogged; sev-1 alarmOrphan NATS accountOperator manual cleanup
S3 compensating action failslogged; sev-1 alarmOrphan marker object (low cost)S3 lifecycle cleanup
AUTH compensating action failslogged; sev-1 alarmOrphan AUTH tenant + subjectOperator manual cleanup via AUTH CLI
Dry-run side-effect leaktest assertsCI failsFix orchestrator branching
OTel span attribute missingotel_attrs_testCI failsFix span builder
password length not 32 charsunit testCI failsFix RNG
password contains spaces or non-alphanumericsunit testCI failsFix RNG generator
concurrent provision same slugfirst wins; second hits UNIQUEexit 1 (if attrs match) or 65Designed
schema_namespace creator forgets to use IF NOT EXISTSrerun failsCI test catchesFix
AUTH internal helper signature drifttype error at compileBuild failsCoordinate TASK-AUTH-001 + TASK-TEN-001 changes
Stale tenant_residency_map row after rollbackcompensating action coversDesignedNone

§11 — Implementation notes

End of TASK-TEN-001.