"4-residency provisioning — sg-1 / eu-1 / us-1 / vn-1 region pinning across Postgres + S3 + NATS + Stripe + KMS with cross-residency-write trip-wire"
§1 — Description (BCP-14 normative)
The TEN service MUST ship 4-residency provisioning (sg-1, eu-1, us-1, vn-1) with per-residency Postgres + S3 + NATS + KMS + Stripe + AUTH-issuer separation, defense-in-depth cross-residency-write trip-wires, atomic-at-residency provisioning, per-residency memory chain partitioning, residency-aware logging context, 6-component health check + CLI, and 8 memory audit kinds.
- MUST define the closed
residencyPostgres enum at migration0015:('sg-1','eu-1','us-1','vn-1'). CI cardinality test asserts exactly 4 per DEC-920. Adding a fifth requires schema migration + DEC entry.
- MUST change
tenant_residency_map.residencycolumn type from TEXT toresidencyenum via migration0003modification (single source of truth — no free-text residency anywhere). Existing rows are validated against the enum at migration; non-conforming rows reject migration (fail-fast).
- MUST enforce closed currency → residency mapping per DEC-922 (consistent with TASK-TEN-003 DEC-785 + TASK-TEN-101 DEC-825):
VND → vn-1SGD → sg-1EUR → eu-1GBP → eu-1USD → us-1The mapping is aconst fninservices/ten/src/residency/mod.rs::derive_residency(billing_currency); no other code path may compute the mapping (single source of truth).
- MUST lock
tenant.residencypost-provisioning per DEC-925. Migration0016includes a triggertrg_residency_immutable()that RAISEs on any UPDATE attempting to changetenants.residency(analogous to TASK-TEN-003trg_billing_currency_immutableper DEC-798). Mutation requires new tenant + manual data migration.
- MUST provision per-residency infrastructure via Terraform modules at
infra/terraform/residency/{residency}/main.tf. Each residency stands up:
- One Aurora PostgreSQL cluster in a private VPC (no peering per DEC-928).
- One S3 bucket pair:
cyberos-{residency}-tenants(data) +cyberos-{residency}-audit(audit archive); cross-region replication DISABLED per DEC-933. - One NATS cluster (3-node JetStream); subjects local per DEC-932.
- One AWS KMS key for envelope encryption (DEC-929: region-pinned).
- One OBS stack subset (Loki + Tempo + Prometheus regional shards) per DEC-934.
- One Stripe account binding per DEC-935 (TASK-TEN-003 DEC-801 already provisioned; this task consumes).
- One AUTH issuer URL per DEC-931 (
https://auth.<residency>.cyberos.world).
- MUST provide a residency router in
services/ten/src/residency/:
pool_router.rs:Map<Residency, PgPool>with per-residency connection strings loaded from KMS-encrypted secrets.s3_router.rs:Map<Residency, aws_sdk_s3::Client>configured to the correct AWS region per DEC-929.nats_router.rs:Map<Residency, async_nats::Client>connected to per-residency NATS clusters.kms_router.rs:Map<Residency, aws_sdk_kms::Client>for envelope decrypt operations.issuer_map.rs:Map<Residency, IssuerConfig>mapping residency → AUTH issuer URL + JWKS URL.
- MUST route every tenant-scoped operation through the residency router. Service-level handlers MUST:
- Receive
tenant_idfrom request context. - Lookup
tenants.residencyvia a per-residency-fanout query (one cross-residency lookup is permitted at the entry: the JWT carriesresidencyclaim per TASK-AUTH-004 + this task's modification, so the lookup is JWT-only in 99% of cases). - Select the correct pool/client from the router.
- Issue the operation against that pool only.
- Mis-route (e.g., handler uses sg-1 pool for an eu-1 tenant) is caught by the trip-wire trigger + emits
ten.residency_pool_misroutesev-1.
- MUST install a cross-residency-write trip-wire trigger on EVERY tenant-scoped Postgres table per DEC-924 + DEC-939 + task-audit skill rule 13 derivative. The trigger function
trg_cross_residency_write_block(): ``sql CREATE OR REPLACE FUNCTION trg_cross_residency_write_block() RETURNS trigger AS $$ DECLARE expected_residency TEXT := current_setting('auth.residency', true); BEGIN IF expected_residency IS NULL OR expected_residency = '' THEN RAISE EXCEPTION 'cross_residency_write_blocked: auth.residency session var not set'; END IF; IF EXISTS (SELECT 1 FROM tenants WHERE id = NEW.tenant_id AND residency::text != expected_residency) THEN RAISE EXCEPTION 'cross_residency_write_blocked: tenant residency != session residency'; END IF; RETURN NEW; END $$ LANGUAGE plpgsql;``
Applied via cursor loop in migration 0016 to all 28 tenant-scoped tables identified at migration time. New tables added in future tasks MUST include this trigger.
- MUST set the session-local
auth.residencyPostgres setting at handler entry from the validated JWT'sresidencyclaim. The TASK-AUTH-004 JWT mint is extended to include this claim; TASK-AUTH-004'sservices/auth/src/handlers/login.rsmodification covers the wiring.
- MUST enforce per-residency JWT issuer validation per DEC-931. The AUTH issuer URL embedded in the JWT (
issclaim) MUST match the residency-derived expected issuer; mismatch returns401 wrong_residency_token+ emitsten.cross_residency_access_attemptsev-1.
- MUST partition the memory audit chain per residency per DEC-926. Each residency's services append to that residency's chain only; the chain head is stored in that residency's Aurora
memory.chain_statetable. Cross-residency memory events are FORBIDDEN — attempting to append a chain row for a tenant in a different residency than the current handler's residency raisescross_residency_memory_event_blocked+ emits the corresponding memory row in the LOCAL chain.
- MUST provision atomically per residency per DEC-936. The
services/ten/src/provisioning/orchestrator.rs(modified per TASK-TEN-001 build envelope) runs: - Open tx in target residency's Aurora.
- Insert tenant row + tenant_residency_map row.
- Create S3 prefix in target residency's bucket (idempotent marker object).
- Register NATS subject in target residency's cluster.
- Verify AUTH bootstrap in target residency.
- Commit tx. On any step failure: ROLLBACK tx + manual cleanup of any partial S3/NATS state via
cyberos-ten residency-cleanup-orphan(slice 3 CLI; slice 2 = operator runs cleanup manually after sev-1 alert per DEC-944).
- MUST add
auth.residencyto AUTH JWT claims per #9 + DEC-931. JWT shape post-this-task:{ sub, tenant_id, residency, exp, iat, iss, aud, scope_grants, persona }. Tokens issued before this task ships (legacy) lack the claim; transitional handler accepts them only for 24h post-deploy then enforces presence (fail closed).
- MUST carry
residency=<rid>field on every log line viatracing::instrument(fields(residency = %ctx.residency))per DEC-942. Missing field on any log line emitted from a tenant-scoped handler raises an OBS alarm sev-3 (ten.residency_logging_context_missing— informational, ops sweep).
- MUST ship
cyberos-ten residency-statusCLI per DEC-943 — 6-component health check per residency: - Aurora connectivity (SELECT 1 + < 100ms).
- S3 reachability (PUT/DELETE marker object + < 500ms).
- NATS heartbeat (publish + subscribe round-trip + < 200ms).
- Stripe ping (GET /v1/account + < 1s — only stripe-rail residencies).
- KMS responsiveness (Encrypt 32 bytes + < 200ms).
- AUTH issuer responsiveness (GET JWKS + < 500ms). Output: per-residency score 0-6 with timing per component; exit code 0 if all 4 residencies score ≥ 5, exit code 73 otherwise.
- MUST emit 8 memory audit row kinds per DEC-941 (task-audit skill rule 6 namespace pattern):
ten.tenant_residency_assigned(sev-2 — material commercial event)ten.residency_provisioned(sev-1 — infrastructure event; one per residency standup)ten.residency_pool_misroute(sev-1 — silent-leak prevented)ten.cross_residency_access_attempt(sev-1 — security signal)ten.cross_residency_write_blocked(sev-1 — trip-wire fired)ten.cross_residency_memory_event_blocked(sev-1 — chain pollution prevented)ten.residency_health_degraded(sev-2 — one component failing)ten.residency_kms_unavailable(sev-1 — encryption broken in one region)
- MUST maintain
residency_health_logtable at migration0017:(id BIGSERIAL PRIMARY KEY, residency residency NOT NULL, component TEXT NOT NULL CHECK (component IN ('aurora','s3','nats','stripe','kms','auth_issuer')), status TEXT NOT NULL CHECK (status IN ('healthy','degraded','down')), latency_ms INT, checked_at TIMESTAMPTZ NOT NULL DEFAULT now()). Append-only via REVOKE per task-audit skill rule 12. Per-residency append + global read (no RLS — health is system-tenant scope).
- MUST map vn-1 to ap-southeast-1 physical region per DEC-930. The mapping is documented in
services/ten/src/residency/mod.rsconst + reflected ininfra/terraform/residency/vn-1/main.tfasprovider "aws" { region = "ap-southeast-1" }. PDPL Law 91/2025 §17 disclosure is presented at TASK-TEN-101 signup for VN-residency tenants.
- MUST require explicit
--residencyflag on TASK-TEN-001'scyberos-ten provisionCLI per DEC-945 (no auto-derive at slice 2). CCO process review checks consistency between provided--billing-currencyand--residencyagainst the DEC-922 mapping; mismatch is rejected with exit code 64 + message naming the expected residency.
- MUST PII-scrub per-residency memory rows via TASK-MEMORY-111 (task-audit skill rule 18). Cross-residency event payloads carry
tenant_id_hash16not the raw tenant_id (defense-in-depth: even if a cross-residency audit row leaks across residencies due to incident, no PII flows).
- MUST thread W3C
traceparentacross the residency-fanout entry (single-cross-residency lookup) + per-residency operation (task-audit skill rule 22 + 23 + 24). Trace_id present on every memory row + every log line.
- MUST NOT support automatic failover between residencies at slice 2 per DEC-940. Cross-region DR is out-of-scope; per-residency multi-AZ Aurora is the DR primitive at slice 2.
- MUST NOT share KMS keys across residencies per DEC-929. Per-residency keys are separate AWS KMS aliases; cross-region key replication NOT enabled (consistent with DEC-921 no-shared-infra).
- MUST NOT allow
tenant_idcollision across residencies per DEC-927. UUIDv7 generation is collision-free across regions via the residency-prefix nibble pattern: high nibble of byte 6 encodes residency (0=sg-1, 1=eu-1, 2=us-1, 3=vn-1). This is reserved encoding; TASK-TEN-001 modified_files includes the generator update.
- SHOULD observe per-residency p95 latency for tenant-scoped operations via OTel histogram
tenant_op_duration_seconds_by_residency. Alarms route to TASK-OBS-007 runbook router per residency.
§2 — Why this design (rationale for humans)
Why no-shared-infra per residency (§1 #5, DEC-921)? Compliance regulators (GDPR Art. 44, PDPA §26, PDPL §17) treat "shared" as "transferred." Sharing an Aurora cluster across residencies means EU customer data is technically in scope of US-jurisdiction queries if a US handler executes against it. Zero-sharing is the only defensible architecture for regulated markets.
Why defense-in-depth trip-wires (§1 #8, DEC-924)? Cross-residency leaks are silent failures — the data just moves to the wrong region and we don't know until audit (typically years later). Three layers (pool router + trigger + Postgres FDW absence) means catching the bug at the first of: bad code (router catches), bad config (trigger catches), bad infra (no FDW to traverse). Each layer can fail; three together approach zero leakage probability.
Why memory chain per-residency (§1 #11, DEC-926)? Audit chains are subpoena-able. A single global chain means subpoena for tenant A's audit history forces production of ALL tenants' chain rows (the chain is a Merkle structure — you can't redact). Per-residency chains scope the subpoena response to one residency's tenants; cross-residency tenants are out of scope.
Why per-residency KMS keys (§1 #5, DEC-929)? Compromise scoping. One leaked KMS key = one residency's data decryptable. Cross-residency key sharing turns one compromise into four residencies' data exposed. Aligns with NIST SP 800-57 key-scoping guidance.
Why vn-1 in ap-southeast-1 physical (§1 #18, DEC-930)? AWS has no Vietnam region (Q2 2026 unchanged). Self-hosting in VN is infeasible at our scale. The PDPL §17 contract-residency clause permits this with explicit customer disclosure — the TASK-TEN-101 signup consent flow handles the disclosure. When AWS opens a VN region, vn-1 migrates physically (data + Terraform); the residency identifier stays vn-1 so customers don't see a label change.
Why JWT-carries-residency (§1 #9, DEC-931)? The JWT is presented at every request; reading the residency from the JWT is free (already validated). Looking up tenants.residency per request would require a cross-residency lookup — defeats the whole architecture. JWT-carries-residency means the per-request residency is signed by the AUTH issuer + tamper-evident.
Why immutable residency (§1 #4, DEC-925)? Changing residency = physically moving data across regions = a migration project (Aurora dump + restore + verify + cutover + back-out plan). Allowing UPDATE on tenants.residency would silently corrupt — the row would say "eu-1" but the data is still in us-1's bucket/cluster. Lock at the schema level, force migration through ops review.
Why atomic provisioning per-residency (§1 #12, DEC-936)? Half-provisioned tenants accumulate (tenants table row but no S3 prefix). At scale, ops alerts about "missing S3 prefix" become noise. Atomic transitions ensure tenant either exists fully or not at all in each residency. Cross-residency atomicity is intentionally NOT attempted (would require 2-phase commit across AWS regions — operational nightmare).
Why session-local auth.residency Postgres setting (§1 #9)? RLS evaluates current_setting('auth.residency') per row at query time. Setting it per-request via SET LOCAL makes it visible to RLS predicates + trip-wire triggers without polluting connection state. This is standard Postgres RLS pattern.
Why explicit --residency flag (§1 #19, DEC-945)? Auto-derivation hides operator intent. If billing_currency is wrong, auto-derive propagates the error. Explicit flag forces operator to commit to a residency separately from currency; CCO process review catches mismatches before provisioning.
Why 8 audit kinds with heavy sev-1 weighting (§1 #16, DEC-941)? Every kind in this task is a cross-residency event — by definition unusual + forensically critical. Sev-1 routes to TASK-OBS-007's CHAT/PagerDuty path for immediate response. Cross-residency leakage is a regulatory-reportable event; sev-1 catches it before any breach-notification window closes.
§3 — API contract
3.1 Postgres schema
-- 0015_residency_enum.sql
CREATE TYPE residency AS ENUM ('sg-1','eu-1','us-1','vn-1');
-- Modification to 0003_tenant_residency_map.sql (referenced in modified_files):
ALTER TABLE tenant_residency_map ALTER COLUMN residency TYPE residency USING residency::residency;
ALTER TABLE tenants ADD COLUMN residency residency; -- nullable until backfill complete
UPDATE tenants t SET residency = (SELECT residency FROM tenant_residency_map WHERE tenant_id=t.id);
ALTER TABLE tenants ALTER COLUMN residency SET NOT NULL;
-- Immutability trigger (DEC-925)
CREATE OR REPLACE FUNCTION trg_residency_immutable() RETURNS trigger AS $$
BEGIN
IF OLD.residency IS DISTINCT FROM NEW.residency THEN
RAISE EXCEPTION 'residency_immutable: cannot change residency on existing tenant (DEC-925); create new tenant + manual data migration';
END IF;
RETURN NEW;
END $$ LANGUAGE plpgsql;
CREATE TRIGGER tenants_residency_immutable
BEFORE UPDATE ON tenants
FOR EACH ROW EXECUTE FUNCTION trg_residency_immutable();
-- 0016_residency_trip_wire.sql
CREATE OR REPLACE FUNCTION trg_cross_residency_write_block() RETURNS trigger AS $$
DECLARE expected_residency TEXT := current_setting('auth.residency', true);
DECLARE row_tenant_id UUID;
DECLARE row_residency TEXT;
BEGIN
IF expected_residency IS NULL OR expected_residency = '' THEN
RAISE EXCEPTION 'cross_residency_write_blocked: auth.residency session var not set';
END IF;
-- NEW.tenant_id may be null on system-tenant rows (e.g., audit log); skip check
IF NEW.tenant_id IS NULL THEN RETURN NEW; END IF;
SELECT residency::text INTO row_residency FROM tenants WHERE id = NEW.tenant_id;
IF row_residency IS NULL THEN
RAISE EXCEPTION 'cross_residency_write_blocked: tenant % not found in this residency', NEW.tenant_id;
END IF;
IF row_residency != expected_residency THEN
RAISE EXCEPTION 'cross_residency_write_blocked: tenant % residency=% but session residency=%', NEW.tenant_id, row_residency, expected_residency;
END IF;
RETURN NEW;
END $$ LANGUAGE plpgsql;
-- Apply to all tenant-scoped tables (cursor loop):
DO $$
DECLARE t RECORD;
BEGIN
FOR t IN
SELECT c.table_name
FROM information_schema.columns c
WHERE c.column_name='tenant_id' AND c.table_schema='public'
LOOP
EXECUTE format(
'CREATE TRIGGER cross_residency_write_trigger BEFORE INSERT OR UPDATE ON %I FOR EACH ROW EXECUTE FUNCTION trg_cross_residency_write_block()',
t.table_name
);
END LOOP;
END $$;
-- 0017_residency_health_log.sql
CREATE TABLE residency_health_log (
id BIGSERIAL PRIMARY KEY,
residency residency NOT NULL,
component TEXT NOT NULL CHECK (component IN ('aurora','s3','nats','stripe','kms','auth_issuer')),
status TEXT NOT NULL CHECK (status IN ('healthy','degraded','down')),
latency_ms INT,
checked_at TIMESTAMPTZ NOT NULL DEFAULT now(),
trace_id CHAR(32)
);
CREATE INDEX idx_residency_health_residency_checked ON residency_health_log(residency, checked_at DESC);
REVOKE UPDATE, DELETE ON residency_health_log FROM cyberos_app;
3.2 Rust types
// services/ten/src/residency/mod.rs
#[derive(Copy, Clone, Eq, PartialEq, Debug, sqlx::Type, serde::Serialize, serde::Deserialize)]
#[sqlx(type_name = "residency", rename_all = "lowercase")]
pub enum Residency { Sg1, Eu1, Us1, Vn1 }
impl Residency {
pub const ALL: [Residency; 4] = [Residency::Sg1, Residency::Eu1, Residency::Us1, Residency::Vn1];
pub const fn as_str(self) -> &'static str {
match self {
Residency::Sg1 => "sg-1",
Residency::Eu1 => "eu-1",
Residency::Us1 => "us-1",
Residency::Vn1 => "vn-1",
}
}
pub const fn aws_region(self) -> &'static str {
match self {
Residency::Sg1 => "ap-southeast-1",
Residency::Eu1 => "eu-west-1",
Residency::Us1 => "us-east-1",
Residency::Vn1 => "ap-southeast-1", // vn-1 physically in Singapore (DEC-930)
}
}
}
pub const fn derive_residency(currency: BillingCurrency) -> Residency {
match currency {
BillingCurrency::Vnd => Residency::Vn1,
BillingCurrency::Sgd => Residency::Sg1,
BillingCurrency::Eur => Residency::Eu1,
BillingCurrency::Gbp => Residency::Eu1,
BillingCurrency::Usd => Residency::Us1,
}
}
// services/ten/src/residency/pool_router.rs
pub struct PoolRouter {
pools: std::collections::HashMap<Residency, sqlx::PgPool>,
}
impl PoolRouter {
pub fn pool_for(&self, residency: Residency) -> Result<&sqlx::PgPool, ResidencyError> {
self.pools.get(&residency).ok_or(ResidencyError::ResidencyUnavailable(residency))
}
}
3.3 CLI
cyberos-ten residency-status [--residency sg-1|eu-1|us-1|vn-1] [--json]
cyberos-ten residency-cleanup-orphan <tenant_id> (slice 3)
§4 — Acceptance criteria
- Enum cardinality —
residency_enum_cardinality_testasserts enum = exactly{sg-1, eu-1, us-1, vn-1}. - Currency → residency mapping —
derive_residency(VND)=vn-1,(SGD)=sg-1,(EUR)=eu-1,(GBP)=eu-1,(USD)=us-1. - Residency immutable trigger —
UPDATE tenants SET residency='eu-1' WHERE id=<sg-1 tenant>raisesresidency_immutable. - Cross-residency-write trip-wire — handler with
SET LOCAL auth.residency='sg-1'attempting INSERT into a tenant-scoped table with tenant in eu-1 raisescross_residency_write_blocked. - Pool router selects correct pool —
pool_router.pool_for(Eu1)returns the eu-1 pool; mis-call returns ResidencyError. - Pool misroute detected — handler using sg-1 pool for eu-1 tenant operation triggers trip-wire → 500 +
ten.residency_pool_misroutesev-1 emitted. - JWT cross-residency rejection — request with JWT carrying
iss=https://auth.us-1.cyberos.worldagainst eu-1 endpoint returns 401 +ten.cross_residency_access_attempt. - Atomic provisioning — provisioning failure at S3 step rolls back Aurora insert; no half-provisioned state in any residency.
- 6-component health check —
cyberos-ten residency-statusreports all 6 components per residency with timing. - memory chain partitioning — appending a chain row for an eu-1 tenant from a sg-1 handler raises
cross_residency_memory_event_blocked. - KMS per-residency — eu-1 encrypt uses eu-1 KMS key; sg-1 encrypt uses sg-1 KMS key; cross-region attempts fail.
- vn-1 physical region — Terraform plan for vn-1 targets ap-southeast-1; AWS resources are tagged with
cyberos_residency=vn-1for clarity. - UUIDv7 residency-prefix nibble — tenant_id high nibble of byte 6 = residency index; collision-free across residencies.
- Explicit --residency flag required — provision CLI without
--residencyexits 64 (invalid arg). - Currency-residency mismatch rejected —
--billing-currency VND --residency us-1exits 64 with explicit error. - JWT carries residency claim — post-task JWT has
residencyclaim; pre-task JWTs accepted for 24h transitional then 401. - Logging carries
residency=<rid>field — log scrape of tenant handler shows field on every line. - AUTH issuer URLs per residency —
issuer_maphas 4 entries matchinghttps://auth.<residency>.cyberos.world. - No cross-residency NATS subscription — sg-1 NATS subscriber cannot subscribe to
tenant.<slug>.*on eu-1 cluster (separate clusters per DEC-932). residency_health_logappend-only — UPDATE on the table raises permission-denied (REVOKE enforced).
§5 — Verification
5.1 residency_enum_cardinality_test.rs
#[tokio::test]
async fn residency_enum_has_exactly_4_values() {
let ctx = TestContext::new().await;
let labels: Vec<String> = sqlx::query_scalar(
"SELECT unnest(enum_range(NULL::residency))::text"
).fetch_all(&ctx.pool).await.unwrap();
let mut labels = labels;
labels.sort();
assert_eq!(labels, vec!["eu-1","sg-1","us-1","vn-1"]);
}
5.2 residency_trip_wire_test.rs
#[tokio::test]
async fn cross_residency_insert_blocked() {
let ctx = TestContext::with_residency(Residency::Eu1).await;
let sg_tenant = ctx.provision_tenant_in(Residency::Sg1, "sg-tenant").await;
sqlx::query("SET LOCAL auth.residency = 'eu-1'").execute(&ctx.pool).await.unwrap();
let err = sqlx::query("INSERT INTO projects (id, tenant_id, name) VALUES ($1, $2, 'x')")
.bind(uuid::Uuid::new_v4()).bind(sg_tenant).execute(&ctx.pool).await.unwrap_err();
assert!(err.to_string().contains("cross_residency_write_blocked"));
let audit = ctx.memory_rows().await;
assert!(audit.iter().any(|r| r.kind == "ten.cross_residency_write_blocked"));
}
#[tokio::test]
async fn missing_session_residency_blocks_write() {
let ctx = TestContext::new().await;
let tenant = ctx.provision_tenant_in(Residency::Sg1, "sg-tenant").await;
// No SET LOCAL auth.residency
let err = sqlx::query("INSERT INTO projects (id, tenant_id, name) VALUES ($1, $2, 'x')")
.bind(uuid::Uuid::new_v4()).bind(tenant).execute(&ctx.pool).await.unwrap_err();
assert!(err.to_string().contains("auth.residency session var not set"));
}
5.3 residency_immutable_test.rs
#[tokio::test]
async fn residency_cannot_change_post_provision() {
let ctx = TestContext::new().await;
let tenant = ctx.provision_tenant_in(Residency::Sg1, "sg-tenant").await;
let err = sqlx::query("UPDATE tenants SET residency='eu-1' WHERE id=$1")
.bind(tenant).execute(&ctx.pool).await.unwrap_err();
assert!(err.to_string().contains("residency_immutable"));
}
5.4 residency_pool_routing_test.rs
#[tokio::test]
async fn pool_router_selects_correct_pool() {
let ctx = TestContext::with_all_residencies().await;
let sg_pool = ctx.pool_router.pool_for(Residency::Sg1).unwrap();
let eu_pool = ctx.pool_router.pool_for(Residency::Eu1).unwrap();
assert_ne!(sg_pool as *const _, eu_pool as *const _); // different pools
let sg_db: String = sqlx::query_scalar("SELECT current_database()").fetch_one(sg_pool).await.unwrap();
let eu_db: String = sqlx::query_scalar("SELECT current_database()").fetch_one(eu_pool).await.unwrap();
assert_ne!(sg_db, eu_db);
}
5.5 residency_jwt_cross_rejection_test.rs
#[tokio::test]
async fn wrong_residency_jwt_rejected() {
let ctx = TestContext::with_residency(Residency::Eu1).await;
let us_tenant = ctx.provision_tenant_in(Residency::Us1, "us-tenant").await;
let us_jwt = ctx.mint_jwt_in_residency(us_tenant, Residency::Us1).await;
let r = ctx.get_in_residency(Residency::Eu1, "/v1/projects").bearer_auth(us_jwt).send().await.unwrap();
assert_eq!(r.status(), 401);
let body: serde_json::Value = r.json().await.unwrap();
assert_eq!(body["error"], "wrong_residency_token");
}
5.6 residency_atomic_provisioning_test.rs
#[tokio::test]
async fn provisioning_rollback_on_s3_failure() {
let ctx = TestContext::with_residency(Residency::Sg1).await;
ctx.simulate_s3_failure();
let result = ctx.provision_tenant_in(Residency::Sg1, "atomic-test").await;
assert!(result.is_err());
let count: i64 = sqlx::query_scalar("SELECT count(*) FROM tenants WHERE slug='atomic-test'")
.fetch_one(ctx.pool_router.pool_for(Residency::Sg1).unwrap()).await.unwrap();
assert_eq!(count, 0);
}
5.7 residency_health_check_test.rs
#[tokio::test]
async fn all_residencies_healthy() {
let ctx = TestContext::with_all_residencies().await;
let report = run_residency_status(&ctx).await;
for residency in Residency::ALL {
let score = report.score_for(residency);
assert_eq!(score.aurora.status, "healthy");
assert!(score.aurora.latency_ms < 100);
assert_eq!(score.s3.status, "healthy");
assert_eq!(score.nats.status, "healthy");
assert_eq!(score.kms.status, "healthy");
assert_eq!(score.auth_issuer.status, "healthy");
}
}
5.8 residency_memory_chain_partitioning_test.rs
#[tokio::test]
async fn cross_residency_memory_append_blocked() {
let ctx = TestContext::with_all_residencies().await;
let eu_tenant = ctx.provision_tenant_in(Residency::Eu1, "eu-tenant").await;
let sg_handler_ctx = ctx.handler_ctx_in(Residency::Sg1);
let result = sg_handler_ctx.memory.append_row(
MemoryRow::new("test.event", eu_tenant, json!({})).build()
).await;
assert!(result.is_err());
assert!(result.unwrap_err().to_string().contains("cross_residency_memory_event_blocked"));
let local_audit = sg_handler_ctx.memory.recent_rows().await;
assert!(local_audit.iter().any(|r| r.kind == "ten.cross_residency_memory_event_blocked"));
}
5.9 residency_currency_mapping_test.rs
#[test]
fn currency_to_residency_mapping() {
assert_eq!(derive_residency(BillingCurrency::Vnd), Residency::Vn1);
assert_eq!(derive_residency(BillingCurrency::Sgd), Residency::Sg1);
assert_eq!(derive_residency(BillingCurrency::Eur), Residency::Eu1);
assert_eq!(derive_residency(BillingCurrency::Gbp), Residency::Eu1);
assert_eq!(derive_residency(BillingCurrency::Usd), Residency::Us1);
}
5.10 residency_pool_misroute_test.rs
#[tokio::test]
async fn pool_misroute_caught_by_trip_wire() {
let ctx = TestContext::with_all_residencies().await;
let eu_tenant = ctx.provision_tenant_in(Residency::Eu1, "eu-tenant").await;
// Simulate a buggy handler using the sg-1 pool for an eu-1 tenant
let sg_pool = ctx.pool_router.pool_for(Residency::Sg1).unwrap();
sqlx::query("SET LOCAL auth.residency = 'sg-1'").execute(sg_pool).await.unwrap();
let err = sqlx::query("INSERT INTO projects (id, tenant_id, name) VALUES ($1, $2, 'x')")
.bind(uuid::Uuid::new_v4()).bind(eu_tenant).execute(sg_pool).await.unwrap_err();
assert!(err.to_string().contains("cross_residency_write_blocked"));
let audit = ctx.memory_rows_in(Residency::Sg1).await;
assert!(audit.iter().any(|r| r.kind == "ten.residency_pool_misroute"));
}
§6 — Implementation skeleton
(API contract in §3 is the skeleton. Additional notes below.)
6.1 Handler-entry residency context setting
// At every tenant-scoped handler entry:
pub async fn require_residency(ctx: &AppCtx, jwt: &Jwt) -> Result<HandlerCtx, AuthError> {
let residency = jwt.claims.residency.parse::<Residency>()?;
let expected_issuer = ctx.residency_issuer_map.issuer_for(residency);
if jwt.claims.iss != expected_issuer {
emit_audit(&ctx, "ten.cross_residency_access_attempt", json!({
"jwt_iss": jwt.claims.iss, "expected_iss": expected_issuer,
})).await;
return Err(AuthError::WrongResidencyToken);
}
let pool = ctx.pool_router.pool_for(residency)?;
let mut conn = pool.acquire().await?;
sqlx::query("SET LOCAL auth.residency = $1").bind(residency.as_str()).execute(&mut *conn).await?;
sqlx::query("SET LOCAL auth.tenant_id = $1").bind(jwt.claims.tenant_id).execute(&mut *conn).await?;
Ok(HandlerCtx { conn, residency, tenant_id: jwt.claims.tenant_id })
}
6.2 Residency health check
pub async fn check_residency(ctx: &AppCtx, residency: Residency) -> ResidencyHealthReport {
let mut report = ResidencyHealthReport::new(residency);
report.aurora = check_aurora(ctx.pool_router.pool_for(residency)?).await;
report.s3 = check_s3(ctx.s3_router.client_for(residency)?, &format!("cyberos-{}-tenants", residency.as_str())).await;
report.nats = check_nats(ctx.nats_router.client_for(residency)?).await;
report.stripe = check_stripe(ctx.stripe_router.client_for(residency)?).await;
report.kms = check_kms(ctx.kms_router.client_for(residency)?).await;
report.auth_issuer = check_auth_issuer(ctx.residency_issuer_map.issuer_for(residency)).await;
persist_health_log(ctx, &report).await;
report
}
§7 — Dependencies
Upstream (depends_on):
- TASK-AI-016 Residency pinning — establishes residency concept at AI layer; TEN-103 extends to data layer.
- TASK-TEN-001 Provisioning CLI — TEN-103 modifies the orchestrator to consume residency router.
Cross-module (related_tasks):
- TASK-TEN-003 Stripe billing — per-residency Stripe account routing (DEC-801 consumed here).
- TASK-TEN-004 4-axis metering — emits per-residency metering events; trip-wire applies.
- TASK-TEN-101 Self-serve signup — residency derivation at signup hand-off.
- TASK-TEN-102 VND domestic rail — vn-1 residency consumes this rail.
- TASK-TEN-104 Lifecycle — tenant termination per-residency.
- TASK-AUTH-004 JWT mint —
residencyclaim added. - TASK-AI-003 memory audit-row bridge — 8 new kinds register; chain partitioned per residency.
- TASK-MEMORY-111 PII scrubbing — tenant_id hash16 in cross-residency events.
- TASK-DOC-001 Documents — S3 bucket per-residency consumed.
- TASK-EMAIL-001 Email — SES region per-residency.
- TASK-OBS-005 Trace correlation — residency tag on every span.
- TASK-OBS-007 Auto-runbook — sev-1 alerts route per-residency.
- TASK-OBS-008 Compliance views — per-residency scoping consumed.
Downstream (blocks): None (this task is bottom-of-stack for residency infrastructure; other tasks consume but don't block on TEN-103 at slice 2).
§8 — Example payloads
8.1 ten.cross_residency_write_blocked memory row
{
"kind": "ten.cross_residency_write_blocked",
"severity": 1,
"tenant_id": "00000000-0000-0000-0000-000000000001",
"actor_id": "system.ten.residency",
"trace_id": "0af7651916cd43dd8448eb211c80319c",
"occurred_at": "2026-05-17T09:14:32.847Z",
"payload": {
"expected_residency": "sg-1",
"blocked_tenant_residency": "eu-1",
"blocked_tenant_id_hash16": "f8a1b2c3d4e5f607",
"table_name": "projects",
"operation": "INSERT",
"service": "cyberos-projects-service",
"stack_trace_sha256": "9c4e7a8b..."
}
}
8.2 cyberos-ten residency-status --json output
{
"checked_at": "2026-05-17T09:14:32.847Z",
"overall_score": "22/24",
"residencies": {
"sg-1": {
"score": "6/6",
"components": {
"aurora": {"status": "healthy", "latency_ms": 12},
"s3": {"status": "healthy", "latency_ms": 84},
"nats": {"status": "healthy", "latency_ms": 7},
"stripe": {"status": "healthy", "latency_ms": 412},
"kms": {"status": "healthy", "latency_ms": 18},
"auth_issuer": {"status": "healthy", "latency_ms": 89}
}
},
"eu-1": {"score": "5/6", "components": { "nats": {"status": "degraded", "latency_ms": 380}, "...": "..." }},
"us-1": {"score": "6/6", "components": "..."},
"vn-1": {"score": "5/6", "components": "...", "note": "vn-1 physically hosted in ap-southeast-1 per DEC-930"}
}
}
8.3 ten.residency_provisioned memory row (one per residency standup)
{
"kind": "ten.residency_provisioned",
"severity": 1,
"tenant_id": "00000000-0000-0000-0000-000000000001",
"actor_id": "system.ten.residency",
"trace_id": "0af7651916cd43dd8448eb211c80319c",
"occurred_at": "2026-05-17T09:14:32.847Z",
"payload": {
"residency": "sg-1",
"aws_region": "ap-southeast-1",
"aurora_cluster_arn": "arn:aws:rds:ap-southeast-1:...",
"s3_bucket": "cyberos-sg-1-tenants",
"s3_audit_bucket": "cyberos-sg-1-audit",
"nats_endpoint": "nats.sg-1.cyberos.world:4222",
"kms_key_arn": "arn:aws:kms:ap-southeast-1:...",
"auth_issuer_url": "https://auth.sg-1.cyberos.world",
"terraform_apply_id": "tf_abc123..."
}
}
§9 — Open questions
All resolved for slice 2. Deferred:
- Deferred: Cross-region DR (failover from sg-1 → us-1 etc.) — slice 3, task-TEN-2xx (placeholder).
- Deferred: vn-1 physical migration to AWS Vietnam region when it opens — slice 3, ops project (no code change required; Terraform plan-only).
- Deferred: Per-residency self-serve via signup form — TASK-TEN-101 derives residency from billing_currency; explicit residency picker in UI is slice 3.
- Deferred: Cross-residency tenant migration (rare admin op) — slice 3, task-TEN-2xx.
- Deferred: Residency-aware backup/restore CLI (
cyberos-ten residency-backup --to s3://...) — slice 3. - Deferred:
cyberos-ten residency-cleanup-orphanCLI — slice 3 (slice 2 = manual ops cleanup on sev-1). - Deferred: Per-residency rate limiting policy (different per residency) — slice 3.
- Deferred: Multi-region OBS query proxy (TASK-OBS-002 enhancement for residency federation) — slice 3.
§10 — Failure modes inventory
| Failure | Detection | Outcome | Recovery |
|---|---|---|---|
| Cross-residency write attempt | trip-wire trigger raises | INSERT/UPDATE blocked + sev-1 ten.cross_residency_write_blocked | Operator investigates handler bug + fixes pool routing |
| Cross-residency read attempt | RLS USING predicate residency=current_setting('auth.residency') rejects | 0 rows returned (silent — RLS conformant); audited at handler if it expected ≥1 | Same: handler bug investigation |
| Pool misroute (handler uses wrong pool) | trip-wire catches OR rows-returned=0 | sev-1 ten.residency_pool_misroute + 500 to caller | Operator audits handler; fixes pool router consumption |
| JWT residency mismatch | issuer check at handler entry | 401 + wrong_residency_token + ten.cross_residency_access_attempt sev-1 | User signs in via correct-residency portal |
auth.residency session var missing | trip-wire raises | INSERT blocked + sev-1 alert; handler bug | Handler audited; missing SET LOCAL call added |
| Residency mutation attempt | trg_residency_immutable raises | UPDATE blocked + audit | Manual data migration via cross-region tooling (slice 3) |
| Atomic provisioning rolls back | orchestrator step failure | Aurora row rolled back; S3/NATS partial cleanup needed via manual ops | cyberos-ten residency-cleanup-orphan <slug> (slice 3) |
| KMS unavailable in one residency | health check 200ms timeout | sev-1 ten.residency_kms_unavailable + degraded encryption | AWS KMS incident; encrypt-fail-closed (no plaintext write) until recovery |
| NATS cluster down in one residency | health check fails | sev-1 + that residency's tenant events queue locally (TASK-TEN-004 WAL queue) | NATS-side recovery; queue drains |
| Aurora primary failover within residency | RDS Multi-AZ event | < 30s outage; handlers retry; sev-2 alert | Inherent — Multi-AZ DR primitive |
| Stripe US account credentials misconfig | Stripe API 401 | sev-1 + stripe-rail temporarily fails for that residency | Operator rotates per DEC-801 / TASK-TEN-003 ops |
| Per-residency Terraform drift | nightly tf plan drift detector | sev-2 alert with diff | Operator reviews + applies tf changes (with change-management) |
| Cross-residency memory append attempted | chain.append guard | sev-1 ten.cross_residency_memory_event_blocked + local-chain row | Inherent — chain-pollution prevented |
| Residency-aware logging missing field | OBS log-quality sweep | sev-3 informational | CI test enforces; PR rejected if missing |
| New tenant table added without trip-wire trigger | migration audit (CI) | CI fails the PR | PR adds trigger before merge |
| UUIDv7 high-nibble collision (astronomically improbable) | partial-unique constraint on (tenant_id) global | INSERT fails | Re-generate ID (1-in-2^60 chance) |
| Per-residency JWKS rotation mid-flight | TASK-AUTH-004 JWKS cache stale | Token validation fails briefly; cache refresh triggered | Inherent |
| Health check timeout false positive | network blip | One degraded-status row in residency_health_log; next 5-min poll recovers | Inherent — degraded ≠ down |
current_setting('auth.residency') typo in handler | dev mistake | trip-wire raises; sev-1; handler ships broken | CI integration test catches |
Migration 0016 cursor loop misses a table | check at CI | CI test asserts every tenant_id-bearing table has the trigger | CI catches; manual trigger add |
| Cross-residency NATS subject subscription attempt | NATS clusters are separate; connect to wrong cluster URL → connection refused | Connection-level failure; not a runtime risk | Inherent isolation |
| Per-residency KMS key revocation | AWS KMS audit | Encrypt operations fail in that residency; sev-1 alert | Operator restores key OR rotates to new key with re-encrypt sweep |
§11 — Implementation notes
§11.1 The 28-tenant-scoped-tables count (§1 #8) is a deploy-time inventory; CI test re-counts via information_schema.columns WHERE column_name='tenant_id' AND table_schema='public' and asserts each has the trip-wire trigger.
§11.2 The trip-wire trigger does ONE additional SELECT per write (SELECT residency FROM tenants WHERE id=NEW.tenant_id). At write-heavy workloads this is a ~5% overhead; acceptable cost for the defense.
§11.3 The residency_immutable trigger uses IS DISTINCT FROM to handle NULL-vs-NULL correctly (residency is NOT NULL post-backfill, but the IS-DISTINCT-FROM is future-proof if NULL ever becomes valid).
§11.4 UUIDv7 residency-nibble encoding: byte 6 high nibble = residency index (0/1/2/3). UUIDv7 already uses byte 6 for version field; we reserve the high nibble of byte 7 instead (compatible with RFC 4122 + RFC 9562 extension space). services/ten/src/residency/uuid_gen.rs implements + tests cross-residency collision-free.
§11.5 AWS region mapping is const fn (compile-time); changing it requires recompile, which is desired (region mapping is forensic-critical).
§11.6 Per-residency Terraform state is stored in per-residency S3 backend (each residency's tf state lives in that residency's S3) — no central tf state that becomes a cross-residency single point of failure.
§11.7 The handler-entry require_residency function (§6.1) is mandatory at every tenant-scoped handler; CI rule via clippy::custom_lint enforces presence.
§11.8 pool_router and friends are loaded at service startup from KMS-encrypted connection strings in AWS Secrets Manager. Boot fails fast if any residency's secrets unreachable; service does NOT serve partial (better: refuse to start than serve wrong residency).
§11.9 The cyberos-ten residency-status CLI is operator-only; runs under cyberos_ops IAM role with read-only KMS/Stripe/AUTH access.
§11.10 Health check latency thresholds are SLOs, not absolute caps; intermittent breach degrades the score but doesn't fire sev-1 (sustained breach via TASK-OBS-007 alarm definition does).
§11.11 The transitional 24h post-deploy window for residency-claim-missing JWTs uses a feature flag auth.residency_claim_required=false that flips to true via a scheduled job at deploy+24h.
§11.12 Per-residency memory chain Merkle Mountain Range proof (TASK-MEMORY-101 §6.4 PROPOSAL P2) is computed per-residency; cross-residency chain-of-custody verification is out-of-scope at slice 2.
§11.13 The Cross-Region-NATS-Sync is intentionally absent — DEC-932 forbids; future cross-region event federation (slice 3) would require explicit consent mechanism per regulation.
§11.14 Test fixtures for with_all_residencies() provision 4 in-memory Postgres instances + 4 LocalStack S3 buckets + 4 NATS clusters; integration tests against real AWS deploy nightly (not on every PR).
§11.15 The residency_health_log table's latency_ms INT is nullable for status='down' (no latency observable when down); other statuses have non-null latency.
§11.16 The 8 memory kinds (§1 #16) are all sev-1 or sev-2 because every kind by definition is unusual: routine residency operation does not emit (steady-state silent). TASK-AI-003 closed-set extension adds 8.
§11.17 Trip-wire trigger error messages include enough context to identify the buggy handler (table name, expected vs actual residency, tenant_id hash16). Operators can grep memory rows to find the offending service.
§11.18 Per-residency Terraform modules share helper modules at infra/terraform/residency/_shared/modules/ — DRY without sharing actual resources.
§11.19 The vn-1 residency's PDPL-disclosure copy is at services/ten/web/signup/consents/vi/pdpl-vn-residency-disclosure-v1.md — TASK-TEN-101 surfaces this at signup for VN-residency tenants.
§11.20 Cross-region IAM role policies — each residency's services use IAM roles scoped to that residency's resources only; no cross-region role assumption permitted.
§11.21 Per-residency Stripe account API keys (TASK-TEN-003 DEC-801) are stored in that residency's AWS Secrets Manager only; no shared key store.
§11.22 The residency_currency_mapping_test is a pure-Rust unit test (no DB); validates the const fn truth-table.
End of TASK-TEN-103 spec.