Task — engineering-spec@1

"VND domestic billing rail — VnPay + Momo + ZaloPay subscription, recurring-charge, refund, dunning + per-PSP webhook bridge for vn-1 tenants"

draftTASK-TEN-102
module ten · class product · priority p0 · created 2026-05-17 · shipped null
depends on TASK-TEN-003, TASK-INV-005 · blocks none

§1 — Description (BCP-14 normative)

The TEN service MUST ship the VND domestic billing rail at services/ten/src/billing/vnd/ — token-bind / subscription / overage / refund / dunning across VnPay + Momo + ZaloPay, NATS-bridged per-PSP webhook dispatch from TASK-INV-005 + new INV webhook handlers, Decree 123 hóa đơn issuance with eHĐĐT signing + annual gap-free numbering, and 12 memory audit kinds.

  1. MUST define the closed vnd_psp Postgres enum at migration 0018: ('vnpay','momo','zalopay'). CI cardinality test asserts 3. Adding a fourth PSP requires schema migration + DEC entry.
  1. MUST define vnd_payment_tokens table at migration 0018: (id BIGSERIAL PRIMARY KEY, tenant_id UUID NOT NULL, psp vnd_psp NOT NULL, payment_token_kms_blob BYTEA NOT NULL, kms_key_id TEXT NOT NULL, masked_account_hint TEXT, status TEXT NOT NULL CHECK (status IN ('active','revoked','expired')) DEFAULT 'active', bound_at TIMESTAMPTZ NOT NULL DEFAULT now(), revoked_at TIMESTAMPTZ, expires_at TIMESTAMPTZ). Partial unique (tenant_id) WHERE status='active' — one active token per tenant at slice 2.
  1. MUST define vnd_psp_credentials table at migration 0019: (id BIGSERIAL PRIMARY KEY, psp vnd_psp NOT NULL, residency residency NOT NULL CHECK (residency='vn-1'), credentials_kms_blob BYTEA NOT NULL, kms_key_id TEXT NOT NULL, status TEXT NOT NULL CHECK (status IN ('active','rotated','revoked')) DEFAULT 'active', created_at TIMESTAMPTZ NOT NULL DEFAULT now(), rotated_at TIMESTAMPTZ). Partial unique (psp) WHERE status='active' — one active credential per PSP. RLS scoped to vn-1 residency per DEC-980 + task-audit skill §8.1d.
  1. MUST define vnd_invoices table at migration 0020: (invoice_number TEXT PRIMARY KEY, tenant_id UUID NOT NULL, charge_ref TEXT NOT NULL, issued_at TIMESTAMPTZ NOT NULL DEFAULT now(), pre_tax_amount_vnd BIGINT NOT NULL, vat_amount_vnd BIGINT NOT NULL, total_amount_vnd BIGINT NOT NULL, line_items JSONB NOT NULL, signed_xml_kms_blob BYTEA, ehoadon_tax_authority_ref TEXT, pdf_s3_key TEXT, status TEXT NOT NULL CHECK (status IN ('issued','signed','cancelled')) DEFAULT 'issued'). Append-only per task-audit skill rule 12 (cancellation = new compensating invoice).
  1. MUST define vnd_invoice_sequence table at migration 0021: (year INT PRIMARY KEY, last_sequence INT NOT NULL DEFAULT 0) for annual gap-free numbering per DEC-983 + Decree 123 §10. Sequence allocation via SELECT ... FOR UPDATE to guarantee no gap, no duplicate.
  1. MUST define vnd_event_dispatch_log table at migration 0022: (id BIGSERIAL PRIMARY KEY, tenant_id UUID NOT NULL, psp vnd_psp NOT NULL, psp_event_id TEXT NOT NULL, event_type TEXT NOT NULL, dispatch_status TEXT NOT NULL CHECK (dispatch_status IN ('dispatched','duplicate','failed')), dispatched_at TIMESTAMPTZ NOT NULL DEFAULT now(), UNIQUE(psp, psp_event_id)). Inbound webhook idempotency per DEC-986. Append-only.
  1. MUST enforce RLS with both USING and WITH CHECK on all 5 VND tables, scoped to tenant_id = current_setting('auth.tenant_id')::uuid AND current_setting('auth.residency') = 'vn-1' (TASK-TEN-103 trip-wire integration).
  1. MUST define the VndPsp trait at billing/vnd/psp_trait.rs: ``rust #[async_trait] pub trait VndPsp: Send + Sync { async fn token_bind_url(&self, tenant_id: Uuid, return_url: &str, requested_at: DateTime<Utc>) -> Result<String, VndError>; async fn token_bind_callback(&self, callback_query: &CallbackQuery) -> Result<PaymentToken, VndError>; async fn charge(&self, payment_token: &PaymentToken, amount_vnd: i64, description: &str, idempotency_key: &str) -> Result<ChargeAck, VndError>; async fn refund(&self, original_charge_ref: &str, amount_vnd: i64, idempotency_key: &str) -> Result<RefundAck, VndError>; async fn revoke_token(&self, payment_token: &PaymentToken) -> Result<(), VndError>; fn verify_webhook_signature(&self, headers: &HeaderMap, body: &[u8]) -> Result<(), VndError>; } ``

Implementations: VnPayClient, MomoClient, ZaloPayClient.

  1. MUST support the token-bind flow at signup per DEC-963. Flow:
  2. UI invokes POST /v1/signup/vnd/token-bind-start with { signup_session_id, psp }.
  3. Handler resolves the chosen PSP adapter; calls token_bind_url(tenant_id, return_url, ...) to get the PSP-hosted authorisation URL.
  4. UI redirects user to PSP-hosted page.
  5. User authorises (SMS-OTP from billing_contact_phone) on PSP page.
  6. PSP redirects back to /v1/signup/vnd/token-bind-return?session_id=...&signed_params=....
  7. Handler calls psp.token_bind_callback(query) to extract + KMS-encrypt the payment_token.
  8. INSERT into vnd_payment_tokens with status='active'.
  9. Emit ten.vnd_token_bind_completed. Total signup-flow target: <30s same as TASK-TEN-101 SLO, accounting for ~10s PSP redirect+auth time.
  1. MUST charge monthly subscription at billing_cycle_anchor per DEC-964. The billing/vnd/subscription.rs::charge_monthly(tenant_id) job:
  1. MUST charge overage at period_close per DEC-967. The vnd/overage.rs::charge_for_period(tenant_id, period_end):
  1. MUST advance dunning per DEC-965. Mirrors TASK-TEN-003 §1 #11:
  1. MUST expose POST /v1/admin/tenants/{id}/vnd/refund for CFO-gated refunds per DEC-966. Body: { original_charge_ref, amount_vnd, reason }. Validations:
  1. MUST expose POST /v1/admin/tenants/{id}/vnd/token/revoke per DEC-985. Caller has tenant_admin role. Handler:
  1. MUST support per-PSP webhook ingestion via TASK-INV-005 extension (modified_files: services/inv/src/webhook/momo.rs + zalopay.rs + their signature verifiers; VnPay already covered by TASK-INV-005). The INV layer:
  1. MUST issue a Decree 123 hóa đơn for every successful charge per DEC-968 + DEC-972 + DEC-983 + DEC-984. The vnd/hoadon.rs::issue_invoice(tenant_id, charge_ref, total_vnd):
  1. MUST use annual gap-free invoice numbering per DEC-983 + Decree 123 §10. The hoadon_seq.rs::next_invoice_number(year):
  1. MUST thread idempotency keys per DEC-970 + task-audit skill §8.3b. Per-PSP key adapter at billing/vnd/idempotency.rs:
  1. MUST emit 12 memory audit row kinds per DEC-979 (task-audit skill rule 6 + §8 namespace):
  1. MUST NOT charge any tenant whose billing_currency != 'VND' via this rail per DEC-973 (symmetric to TASK-TEN-003 §1 #23). Guard at vnd::api_client::call() entry — cross-rail attempts return 400 + { error: "wrong_billing_rail", expected: "stripe", got: "vnd" }.
  1. MUST NOT charge the founder tenant per DEC-974. Guard mirrors TASK-TEN-003 §1 #22 — emits sev-3 ten.vnd_founder_skip informational row.
  1. MUST lock billing_currency immutable post-provisioning per DEC-977 (already enforced by TASK-TEN-003's trg_billing_currency_immutable trigger).
  1. MUST require billing_contact_phone at vn-1 signup per DEC-981. TASK-TEN-101's SignupCompleteReq body extended (in this task's modified_files) with optional billing_contact_phone: Option<String>; validation in vnd::token_bind::start requires it for VND tenants (returns 400 + phone_required_for_vnd if missing).
  1. MUST PII-scrub all audit rows per task-audit skill rule 18 + DEC-979. Phone hashed as phone_hash16 = HMAC(global_salt, phone_e164_form); masked account hint (last-4 of bank account) hashed.
  1. MUST thread W3C traceparent end-to-end per task-audit skill rule 22-24. Signup → token-bind → callback → INSERT → audit row chain MUST share trace_id.
  1. MUST support concurrent token-bind safely per task-audit skill §8.3. Per-tenant SELECT ... FOR UPDATE on vnd_payment_tokens during INSERT; concurrent binds for same tenant return 409 (active token already exists).
  1. SHOULD observe per-PSP charge latency p95 via OTel histogram vnd_charge_duration_seconds_by_psp per task-audit skill rule 22. Alarm sev-2 if p95 > 10s sustained 10 min (PSP-side issue).

§2 — Why this design (rationale for humans)

Why three PSPs (§1 #1, DEC-960)? Vietnamese consumer payment landscape is fragmented. VnPay dominates merchant-side (~60% market share); Momo dominates wallet-based payments (~40% of consumer wallets); ZaloPay is the Zalo-ecosystem default (~30% of social-app users). One PSP loses ~40-70% of potential customers depending on which one. Three covers >95% with reasonable integration cost.

Why one PSP per tenant at slice 2 (§1 #2 partial-unique, DEC-960)? Multi-PSP-per-tenant adds combinatorial complexity (which PSP for next monthly charge? failover order? per-PSP credentials?) without immediate commercial benefit (a tenant picks one PSP at signup and rarely switches). Slice 3 adds failover; slice 2 keeps the rail simple.

Why PSP-native subscription primitives over manual monthly redirects (§1 #10, DEC-962)? Manual monthly redirects mean the user has to authorise every single month — ~30% drop-off per month per industry benchmarks = catastrophic churn. Token-bind authorises once; subsequent charges are headless. All three PSPs support this primitive (VnPay TokenPay, Momo recurring-token, ZaloPay wallet-bind); leveraging them is the conventional path.

Why hóa đơn issuance mandatory per charge (§1 #16, DEC-968)? VN Decree 123 §15 requires electronic invoice for every business-to-business transaction. Non-compliance = tax-authority enforcement + commercial-licence revocation. This is not optional commercial UX; it's legal commercial requirement.

Why gap-free annual sequence (§1 #17, DEC-983)? Decree 123 §10 + Circular 78 §3 mandate gap-free invoice numbering for tax authority audit. Gaps indicate hidden transactions = fraud signal. The FOR UPDATE allocation pattern + the documented "skip = audit log" approach handles rollback semantics legally.

Why async PSP responses + Postgres LISTEN/NOTIFY (§1 #10, DEC-976)? VN PSPs return 200 + processing immediately; real outcome arrives via webhook within seconds-to-5min. Polling = wasted load; LISTEN/NOTIFY = real-time wake-up when the webhook handler INSERTs. Same pattern as TASK-INV-005's webhook dispatch.

Why per-PSP webhook signature variance (§1 #15, DEC-971)? Each PSP designed its signature scheme independently — VnPay HMAC-SHA512 over query params; Momo HMAC-SHA256 over JSON body; ZaloPay HMAC-SHA256 over key1. We can't normalise the upstream; the adapter pattern (VndPsp::verify_webhook_signature) hides the variance behind a uniform interface.

Why dunning state machine parallel to TASK-TEN-003 (§1 #12, DEC-965)? Operational simplicity. Two rails (Stripe + VND) with identical dunning state machines mean operators learn one model. Per-PSP dunning differences would explode the operator-mental-model surface.

Why per-PSP credentials with rotation (§1 #3, DEC-969)? Each PSP issues separate API keys. Rotation per PSP is independent (Momo might force-rotate quarterly; VnPay might do annually). Storing them KMS-wrapped + per-PSP table makes rotation independent + auditable.

Why billing_contact_phone required for VND (§1 #23, DEC-981)? VN PSPs require SMS-OTP for token authorisation (regulatory + anti-fraud). Without a phone, the token-bind redirect fails at the PSP side — user can't complete signup. Capturing at signup avoids the deadlock.

Why cross-rail block (§1 #20, DEC-973)? Tenant billing_currency is locked at provisioning (TASK-TEN-003 DEC-798). A VND tenant attempting Stripe rail = bug, either client-side or server-side. Fail closed at the rail entry; trip-wire catches at schema level too (TASK-TEN-103 §1 #8 cross-residency trigger).

Why VAT 10% inclusive (§1 #16, DEC-972)? VN consumer-facing prices conventionally show VAT-inclusive. Showing exclusive is unusual + breaks the user's mental model. Decree 123 mandates line-item breakdown (pre-tax + VAT + total) — the invoice carries the split; the price catalog uses inclusive amounts.

Why hóa đơn signing via tax authority eHĐĐT (§1 #16, DEC-984)? Decree 123 §13 requires either (a) sign with tax authority's central service OR (b) sign with our own digital cert pre-registered with tax authority. (a) is operationally simpler + always-up-to-date with regulatory changes; (b) requires our cert lifecycle management. Picked (a) for slice 2; (b) is slice 3 if we hit eHĐĐT availability issues.

§3 — API contract

3.1 Postgres schema (key migrations)

-- 0018_vnd_payment_tokens.sql
CREATE TYPE vnd_psp AS ENUM ('vnpay','momo','zalopay');

CREATE TABLE vnd_payment_tokens (
  id BIGSERIAL PRIMARY KEY,
  tenant_id UUID NOT NULL,
  psp vnd_psp NOT NULL,
  payment_token_kms_blob BYTEA NOT NULL,
  kms_key_id TEXT NOT NULL,
  masked_account_hint TEXT,
  status TEXT NOT NULL DEFAULT 'active'
    CHECK (status IN ('active','revoked','expired')),
  bound_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  revoked_at TIMESTAMPTZ,
  expires_at TIMESTAMPTZ
);
CREATE UNIQUE INDEX uniq_active_vnd_token ON vnd_payment_tokens(tenant_id) WHERE status='active';
ALTER TABLE vnd_payment_tokens ENABLE ROW LEVEL SECURITY;
CREATE POLICY vnd_payment_tokens_rls ON vnd_payment_tokens
  USING (tenant_id = current_setting('auth.tenant_id')::uuid
         AND current_setting('auth.residency') = 'vn-1')
  WITH CHECK (tenant_id = current_setting('auth.tenant_id')::uuid
              AND current_setting('auth.residency') = 'vn-1');
REVOKE UPDATE, DELETE ON vnd_payment_tokens FROM cyberos_app;
GRANT UPDATE (status, revoked_at, expires_at) ON vnd_payment_tokens TO cyberos_app;

CREATE TABLE vnd_idempotency_cache (
  canonical_key TEXT PRIMARY KEY,
  tenant_id UUID NOT NULL,
  psp vnd_psp NOT NULL,
  per_psp_key TEXT NOT NULL,
  request_sha256 CHAR(64) NOT NULL,
  response_status INT,
  response_body_sha256 CHAR(64),
  created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  ttl_until TIMESTAMPTZ NOT NULL DEFAULT now() + INTERVAL '7 days'
);

-- 0021_vnd_invoice_sequence.sql
CREATE TABLE vnd_invoice_sequence (
  year INT PRIMARY KEY,
  last_sequence INT NOT NULL DEFAULT 0,
  notes JSONB NOT NULL DEFAULT '[]'::jsonb  -- skipped numbers + reasons per Decree 123 §10
);

-- 0020_vnd_invoices.sql
CREATE TABLE vnd_invoices (
  invoice_number TEXT PRIMARY KEY,
  tenant_id UUID NOT NULL,
  charge_ref TEXT NOT NULL,
  issued_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  pre_tax_amount_vnd BIGINT NOT NULL,
  vat_amount_vnd BIGINT NOT NULL,
  total_amount_vnd BIGINT NOT NULL,
  line_items JSONB NOT NULL,
  signed_xml_kms_blob BYTEA,
  ehoadon_tax_authority_ref TEXT,
  pdf_s3_key TEXT,
  status TEXT NOT NULL DEFAULT 'issued'
    CHECK (status IN ('issued','signed','cancelled'))
);
ALTER TABLE vnd_invoices ENABLE ROW LEVEL SECURITY;
CREATE POLICY vnd_invoices_rls ON vnd_invoices
  USING (tenant_id = current_setting('auth.tenant_id')::uuid
         AND current_setting('auth.residency') = 'vn-1')
  WITH CHECK (tenant_id = current_setting('auth.tenant_id')::uuid
              AND current_setting('auth.residency') = 'vn-1');
REVOKE UPDATE, DELETE ON vnd_invoices FROM cyberos_app;
GRANT UPDATE (signed_xml_kms_blob, ehoadon_tax_authority_ref, pdf_s3_key, status)
  ON vnd_invoices TO cyberos_app;

3.2 Rust types (selected)

// services/ten/src/billing/vnd/psp_trait.rs
#[async_trait]
pub trait VndPsp: Send + Sync {
    fn psp_kind(&self) -> VndPspKind;
    async fn token_bind_url(&self, ctx: &TokenBindCtx) -> Result<String, VndError>;
    async fn token_bind_callback(&self, query: &CallbackQuery) -> Result<PaymentToken, VndError>;
    async fn charge(&self, token: &PaymentToken, amount_vnd: i64, description: &str, idempotency_key: &str) -> Result<ChargeAck, VndError>;
    async fn refund(&self, original_charge_ref: &str, amount_vnd: i64, idempotency_key: &str) -> Result<RefundAck, VndError>;
    async fn revoke_token(&self, token: &PaymentToken) -> Result<(), VndError>;
    fn verify_webhook_signature(&self, headers: &http::HeaderMap, body: &[u8]) -> Result<(), VndError>;
}

pub struct PaymentToken {
    pub tenant_id: Uuid,
    pub psp: VndPspKind,
    pub raw_token: SecretString,  // KMS-decrypted at use
    pub masked_account_hint: Option<String>,
    pub bound_at: DateTime<Utc>,
    pub expires_at: Option<DateTime<Utc>>,
}

pub enum ChargeAck {
    Processing { psp_request_id: String },  // async resolution via webhook
    Completed  { psp_charge_ref: String },  // synchronous (rare)
}

3.3 REST endpoints

POST   /v1/signup/vnd/token-bind-start              (public, signup-session-bound)
GET    /v1/signup/vnd/token-bind-return             (PSP redirect callback)
POST   /v1/admin/tenants/{id}/vnd/token/revoke      (tenant_admin)
POST   /v1/admin/tenants/{id}/vnd/refund            (cfo)
GET    /v1/admin/tenants/{id}/vnd/invoices          (tenant_admin or cfo — list)
GET    /v1/admin/tenants/{id}/vnd/invoices/{num}    (tenant_admin or cfo — single, PDF download)

§4 — Acceptance criteria

  1. vnd_psp cardinality — enum = exactly {vnpay, momo, zalopay}.
  2. Token-bind happy — start → PSP redirect → return → token persisted KMS-encrypted; ten.vnd_token_bind_completed emitted.
  3. Token-bind failure — PSP returns failure code → token NOT persisted + ten.vnd_token_bind_failed emitted.
  4. Monthly subscription charge — at billing_cycle_anchor, subscription::charge_monthly(tenant) invokes PSP charge; webhook resolves to success → ten.vnd_subscription_charged + hóa đơn issued.
  5. Async charge resolution — PSP responds processing; webhook arrives 30s later → handler unblocks via LISTEN/NOTIFY.
  6. Overage one-off — period_close with overage > 0 triggers single VND charge with idempotency_key matching vnd.<tid>.overage.<period_end>.
  7. Dunning state machine — three consecutive charge_failed → retry_3; fourth → suspended + TASK-TEN-104 suspension.
  8. Refund CFO-only — non-cfo refund attempt returns 403; cfo returns 201 + sev-1 audit.
  9. Refund amount cap — refund > original charge returns 400 refund_exceeds_charge.
  10. Token revoke — POST /vnd/token/revoke → token.status='revoked' + next charge fails.
  11. Cross-rail rejection — USD tenant invoking VND rail returns 400 wrong_billing_rail.
  12. Founder skip — founder tenant VND attempt is no-op + emits ten.vnd_founder_skip sev-3 (not in 12-kind core).
  13. Hóa đơn issuance — every successful charge produces a vnd_invoices row + signed XML.
  14. Hóa đơn sequence gap-free — 1000 concurrent issue requests produce 1000 unique invoice numbers with no duplicates (skipped on rollback IS allowed but logged in notes).
  15. Per-PSP webhook signature verify — VnPay HMAC-SHA512 + Momo HMAC-SHA256-body + ZaloPay HMAC-SHA256-key1 all verified correctly; bad sig → 401.
  16. Idempotency key per PSP — same canonical_key → same per-PSP key; replay request returns cached response.
  17. Dispatcher idempotency — same (psp, psp_event_id) twice → one dispatch + one duplicate log row.
  18. Residency isolation — non-vn-1 handler cannot read vnd_payment_tokens (RLS rejects).
  19. Billing_contact_phone required at vn-1 signup — VND signup without phone returns 400 phone_required_for_vnd.
  20. 12 memory audit kinds emitted — happy flow + failure flow produces all 12 kinds across scenarios.

§5 — Verification

5.1 vnd_psp_enum_cardinality_test.rs

#[tokio::test]
async fn vnd_psp_has_exactly_3_values() {
    let ctx = TestContext::vn1().await;
    let labels: Vec<String> = sqlx::query_scalar("SELECT unnest(enum_range(NULL::vnd_psp))::text")
        .fetch_all(&ctx.pool).await.unwrap();
    let mut labels = labels; labels.sort();
    assert_eq!(labels, vec!["momo","vnpay","zalopay"]);
}

5.2 vnd_token_bind_happy_test.rs

#[tokio::test]
async fn vnpay_token_bind_completes_and_persists_kms_wrapped() {
    let ctx = TestContext::vn1_with_psp(VndPspKind::Vnpay).await;
    let session = ctx.email_verified_signup_session("alice@vn.example", "+84901234567").await;
    let start = ctx.post("/v1/signup/vnd/token-bind-start").json(&json!({
        "signup_session_id": session, "psp": "vnpay"
    })).send().await.unwrap();
    let body: serde_json::Value = start.json().await.unwrap();
    let redirect_url = body["redirect_url"].as_str().unwrap();
    let return_query = ctx.simulate_vnpay_user_auth(redirect_url, "ok").await;
    let r = ctx.get(&format!("/v1/signup/vnd/token-bind-return?{}", return_query)).send().await.unwrap();
    assert_eq!(r.status(), 200);
    let row = sqlx::query("SELECT psp::text, status, payment_token_kms_blob FROM vnd_payment_tokens WHERE tenant_id=$1")
        .bind(ctx.tenant_for_session(session).await)
        .fetch_one(&ctx.pool).await.unwrap();
    let blob: Vec<u8> = row.get("payment_token_kms_blob");
    assert!(blob.len() > 0);
    assert!(!std::str::from_utf8(&blob).map(|s| s.contains("token=")).unwrap_or(false), "plaintext leak");
    assert_eq!(row.get::<String,_>("status"), "active");

    let audit = ctx.memory_rows().await;
    assert!(audit.iter().any(|r| r.kind == "ten.vnd_token_bind_completed"));
}

5.3 vnd_subscription_charge_test.rs

#[tokio::test]
async fn monthly_charge_emits_audit_and_invoice() {
    let ctx = TestContext::vn1_with_psp(VndPspKind::Momo).await;
    let tenant = ctx.provision_vnd_tenant_with_token("acme-vn", "+84901234567").await;
    let _ = ctx.run_subscription_charge(tenant, PlanTier::Team).await;
    ctx.simulate_psp_webhook_success(tenant, VndPspKind::Momo).await;

    let charges: Vec<String> = sqlx::query_scalar(
        "SELECT kind FROM memory_rows WHERE tenant_id=$1 AND kind LIKE 'ten.vnd_%'"
    ).bind(tenant).fetch_all(&ctx.pool).await.unwrap();
    assert!(charges.iter().any(|k| k == "ten.vnd_subscription_charged"));
    assert!(charges.iter().any(|k| k == "ten.vnd_invoice_issued"));

    let invoice_count: i64 = sqlx::query_scalar(
        "SELECT count(*) FROM vnd_invoices WHERE tenant_id=$1"
    ).bind(tenant).fetch_one(&ctx.pool).await.unwrap();
    assert_eq!(invoice_count, 1);
}

5.4 vnd_async_charge_resolution_test.rs

#[tokio::test]
async fn async_charge_resolves_via_listen_notify() {
    let ctx = TestContext::vn1_with_psp(VndPspKind::Zalopay).await;
    let tenant = ctx.provision_vnd_tenant_with_token("zlp-tenant", "+84901234567").await;
    let charge_fut = tokio::spawn(ctx.charge_with_wait(tenant));
    tokio::time::sleep(Duration::from_millis(500)).await;
    ctx.simulate_psp_webhook_success(tenant, VndPspKind::Zalopay).await;
    let result = charge_fut.await.unwrap();
    assert!(result.is_ok());
}

5.5 vnd_dunning_state_machine_test.rs

#[tokio::test]
async fn vnd_dunning_advances_then_suspends() {
    let ctx = TestContext::vn1_with_psp(VndPspKind::Vnpay).await;
    let tenant = ctx.provision_vnd_tenant_with_token("dunning-test", "+84901234567").await;
    for expected in &[DunningState::Retry1, DunningState::Retry2, DunningState::Retry3] {
        ctx.simulate_psp_webhook_failure(tenant, VndPspKind::Vnpay).await;
        assert_eq!(ctx.load_dunning(tenant).await, *expected);
    }
    ctx.simulate_psp_webhook_failure(tenant, VndPspKind::Vnpay).await;
    assert_eq!(ctx.load_dunning(tenant).await, DunningState::Suspended);

    let status: String = sqlx::query_scalar("SELECT status::text FROM tenants WHERE id=$1")
        .bind(tenant).fetch_one(&ctx.pool).await.unwrap();
    assert_eq!(status, "suspended");
}

5.6 vnd_hoadon_seq_gap_free_test.rs

#[tokio::test]
async fn invoice_sequence_no_duplicates_under_concurrency() {
    let ctx = TestContext::vn1_with_psp(VndPspKind::Vnpay).await;
    let tenants: Vec<_> = (0..100).map(|i| async move {
        ctx.provision_vnd_tenant(&format!("conc-{i}"), "+84901234567").await
    }).collect::<futures::stream::FuturesUnordered<_>>().collect::<Vec<_>>().await;
    let invoices = futures::future::join_all(tenants.into_iter().map(|t| ctx.issue_invoice(t, 100_000))).await;
    let unique: std::collections::HashSet<_> = invoices.iter().filter_map(|r| r.as_ref().ok()).cloned().collect();
    assert_eq!(unique.len(), invoices.iter().filter(|r| r.is_ok()).count(), "duplicate invoice number issued");
}

5.7 vnd_refund_cfo_only_test.rs

#[tokio::test]
async fn vnd_refund_requires_cfo_role() {
    let ctx = TestContext::vn1_with_psp(VndPspKind::Momo).await;
    let tenant = ctx.provision_vnd_tenant_with_charge("refund-test").await;
    let admin_token = ctx.mint_jwt(tenant, "tenant_admin");
    let r = ctx.post(&format!("/v1/admin/tenants/{tenant}/vnd/refund")).bearer_auth(admin_token)
        .json(&json!({"original_charge_ref": "ch_1", "amount_vnd": 100_000, "reason": "duplicate"}))
        .send().await.unwrap();
    assert_eq!(r.status(), 403);

    let cfo_token = ctx.mint_jwt(tenant, "chief-financial-officer");
    let r = ctx.post(&format!("/v1/admin/tenants/{tenant}/vnd/refund")).bearer_auth(cfo_token)
        .json(&json!({"original_charge_ref": "ch_1", "amount_vnd": 100_000, "reason": "duplicate"}))
        .send().await.unwrap();
    assert_eq!(r.status(), 201);
}

5.8 vnd_cross_rail_rejection_test.rs

#[tokio::test]
async fn usd_tenant_blocked_from_vnd_rail() {
    let ctx = TestContext::vn1().await;
    let us_tenant = ctx.provision_us_tenant("usd-tenant").await;
    let r = ctx.attempt_vnd_charge(us_tenant).await;
    assert!(matches!(r, Err(VndError::WrongBillingRail { .. })));
}

5.9 vnd_psp_signature_verify_test.rs

#[tokio::test]
async fn vnpay_signature_validates() {
    let secret = b"vnp_secret_key";
    let body = b"vnp_Amount=10000&vnp_TxnRef=abc";
    let mut mac = Hmac::<Sha512>::new_from_slice(secret).unwrap();
    mac.update(body);
    let sig = hex::encode(mac.finalize().into_bytes());
    let mut headers = HeaderMap::new();
    headers.insert("vnp-SecureHash", sig.parse().unwrap());
    let vnp = VnPayClient::with_secret(secret);
    assert!(vnp.verify_webhook_signature(&headers, body).is_ok());

    // tampered body
    let bad_body = b"vnp_Amount=999999&vnp_TxnRef=abc";
    assert!(vnp.verify_webhook_signature(&headers, bad_body).is_err());
}

5.10 vnd_residency_isolation_test.rs

#[tokio::test]
async fn non_vn1_handler_cannot_read_vnd_tables() {
    let ctx = TestContext::with_all_residencies().await;
    let vn_tenant = ctx.provision_vnd_tenant("acme-vn", "+84901234567").await;
    let eu_handler = ctx.handler_ctx_in(Residency::Eu1);
    let rows: Vec<(String,)> = sqlx::query_as("SELECT id::text FROM vnd_payment_tokens WHERE tenant_id=$1")
        .bind(vn_tenant).fetch_all(eu_handler.pool()).await.unwrap_or_default();
    assert_eq!(rows.len(), 0, "RLS should block cross-residency read");
}

§6 — Implementation skeleton

(API contract in §3 is the skeleton; selected helpers below.)

6.1 Hóa đơn sequence allocation (gap-free)

// services/ten/src/billing/vnd/hoadon_seq.rs
pub async fn next_invoice_number(tx: &mut PgTx<'_>, year: i32, occurred_on: NaiveDate) -> Result<String, HoaDonError> {
    let row = sqlx::query("SELECT last_sequence FROM vnd_invoice_sequence WHERE year=$1 FOR UPDATE")
        .bind(year).fetch_optional(&mut **tx).await?;
    let next = row.map(|r| r.get::<i32,_>(0) + 1).unwrap_or(1);
    sqlx::query(
        "INSERT INTO vnd_invoice_sequence (year, last_sequence) VALUES ($1, $2)
         ON CONFLICT (year) DO UPDATE SET last_sequence = $2"
    ).bind(year).bind(next).execute(&mut **tx).await?;
    Ok(format!(
        "CYBOS-{:02}{:02}{:02}-{:06}",
        year % 100, occurred_on.month(), occurred_on.day(), next
    ))
}

6.2 Async charge resolution via LISTEN/NOTIFY

pub async fn charge_with_wait(ctx: &AppCtx, tenant_id: Uuid, plan: PlanTier) -> Result<(), VndError> {
    let mut listener = sqlx::postgres::PgListener::connect_with(&ctx.pool).await?;
    listener.listen(&format!("vnd_charge_{tenant_id}")).await?;
    let psp = ctx.vnd_psp_for(tenant_id).await?;
    let ack = psp.charge(/* ... */).await?;
    match ack {
        ChargeAck::Completed { .. } => Ok(()),
        ChargeAck::Processing { .. } => {
            tokio::time::timeout(Duration::from_secs(300), listener.recv()).await
                .map_err(|_| VndError::PspTimeout)?
                .map_err(VndError::ListenerError)?;
            Ok(())
        }
    }
}

6.3 NATS dispatcher

pub async fn run_dispatcher(ctx: AppCtx) {
    let sub = ctx.nats.subscribe("tenant.*.ten.vnd.>").await.unwrap();
    while let Some(msg) = sub.next().await {
        let event: VndWebhookEvent = serde_json::from_slice(&msg.payload).unwrap();
        let dispatched = ctx.repo.vnd_dispatch_log.upsert_idempotent(event.psp, event.psp_event_id).await;
        if !dispatched.is_new { continue; }
        match event.event_type.as_str() {
            "subscription_charge_completed" => handle_charge_completed(&ctx, event).await,
            "subscription_charge_failed"    => handle_charge_failed(&ctx, event).await,
            "refund_completed"              => handle_refund_completed(&ctx, event).await,
            "token_expired"                 => handle_token_expired(&ctx, event).await,
            _ => {}
        }
    }
}

§7 — Dependencies

Upstream (depends_on):

Cross-module (related_tasks):

Downstream (blocks): None at slice 2.

§8 — Example payloads

8.1 ten.vnd_subscription_charged memory row

{
  "kind": "ten.vnd_subscription_charged",
  "severity": 2,
  "tenant_id": "8a2f...",
  "actor_id": "system.ten.vnd",
  "trace_id": "0af7651916cd43dd8448eb211c80319c",
  "occurred_at": "2026-05-17T03:14:32.847Z",
  "payload": {
    "psp": "momo",
    "amount_vnd": 1_290_000,
    "period_start": "2026-04-17T17:00:00Z",
    "period_end": "2026-05-17T17:00:00Z",
    "psp_charge_ref": "MOMO_TXN_3OabcXYZ",
    "invoice_number": "CYBOS-260517-000042",
    "billing_contact_phone_hash16": "9c4e7a8b6d2f1e3a"
  }
}

8.2 Token-bind start request/response

// POST /v1/signup/vnd/token-bind-start
{ "signup_session_id": "0190f7c0-8b3c-7a4f-aaaa-000000000001", "psp": "vnpay" }

// response
{ "redirect_url": "https://sandbox.vnpayment.vn/paymentv2/vpcpay.html?vnp_Amount=10000&vnp_TxnRef=...",
  "expires_at": "2026-05-17T03:24:32.847Z" }

8.3 Hóa đơn (Decree 123 line items)

{
  "invoice_number": "CYBOS-260517-000042",
  "tenant_id": "8a2f...",
  "issued_at": "2026-05-17T03:14:35.221Z",
  "taxpayer": { "tax_id": "0312345678", "name": "ACME Vietnam JSC", "address": "..." },
  "line_items": [
    { "description": "Cyberos Team plan - billing period 2026-04-17 → 2026-05-17",
      "quantity": 1, "pre_tax_amount_vnd": 1_172_727, "vat_rate": 0.10, "vat_amount_vnd": 117_273, "total_vnd": 1_290_000 }
  ],
  "totals": { "pre_tax_vnd": 1_172_727, "vat_vnd": 117_273, "total_vnd": 1_290_000 },
  "ehoadon_tax_authority_ref": "GDT-2026-000123456",
  "signed_xml_sha256": "9c4e7a8b6d2f1e3a..."
}

8.4 ten.vnd_refund_issued memory row

{
  "kind": "ten.vnd_refund_issued",
  "severity": 1,
  "tenant_id": "8a2f...",
  "actor_id": "user.cfo.456",
  "trace_id": "0af7651916cd43dd8448eb211c80319c",
  "occurred_at": "2026-05-17T09:14:32.847Z",
  "payload": {
    "psp": "vnpay",
    "original_charge_ref": "VNP_TXN_3OabcXYZ",
    "refund_amount_vnd": 645_000,
    "compensating_invoice_number": "CYBOS-260517-000043",
    "reason": "duplicate_charge"
  }
}

§9 — Open questions

All resolved for slice 2. Deferred:

§10 — Failure modes inventory

FailureDetectionOutcomeRecovery
PSP token-bind redirect timeout10-min TTL on token-bind sessionBind session expired; user re-clicks "Sign up with VnPay/Momo/ZaloPay"Re-initiate; idempotent
PSP returns failure at callback (user cancelled SMS-OTP)psp.token_bind_callback returns Err200 to UI with { status: "user_cancelled" }; signup session remains email_verified for retryUser retries token-bind
Webhook signature invalidHMAC verify fails401 + signature_invalid + inv.vnd_event_rejectedPSP-side rotated credential without our update; ops triages
Webhook replay (same psp_event_id)dispatcher (psp, event_id) UNIQUE200 + duplicate log row + no side effectInherent — idempotency
PSP charge API 5xxapi_client retries 3x with backoffCharge marked failed → dunning advancesStripe-style auto-retry; persistent → sev-2 alert
PSP charge timeout (>5min wait)LISTEN/NOTIFY timeout firesCharge marked processing_timeout; webhook may arrive later → reconciliationReconciliation sweep runs hourly; manual review on persistent timeout
Hóa đơn signing service down (eHĐĐT API unavailable)sign_invoice returns ErrInvoice persisted unsigned (status='issued', signed_xml NULL); retry job re-signs every 5mineHĐĐT recovery; max 24h before sev-1 alert (Decree 123 §13 grace)
Hóa đơn sequence number duplicate (concurrent INSERT)partial unique on invoice_numberSecond INSERT fails; caller retries with next numberInherent — FOR UPDATE + retry on conflict
Invoice sequence rollback (tx fails post-allocation)sequence number "skipped"Skipped number recorded in vnd_invoice_sequence.notes with reasonAuditable per Decree 123 §10
Cross-rail attempt (USD tenant invoking VND)api_client guard400 + wrong_billing_rail + ten.vnd_cross_rail_rejected sev-2Handler bug investigation
Founder tenant VND attemptapi_client guardNo-op + sev-3 ten.vnd_founder_skipInherent guard
Tenant has no active payment_token at monthly charge timecharge_monthly preflightcharge skipped + ten.vnd_subscription_charge_failed reason='no_token' + dunning advancesTenant re-binds token via signup-style flow OR support intervention
PSP credential rotation overlap expired before rolloutapi_client uses old credential401 from PSP; sev-1 alertTenant_admin re-rotates; usual fix
KMS unavailable when decrypting payment_tokenKMS timeoutcharge skipped + sev-1 ten.vnd_kms_unavailableAWS KMS recovery; charge retried
Webhook arrives before charge_monthly job completesLISTEN/NOTIFY raceWebhook handler runs first; charge_monthly's wait returns immediately on next NOTIFYInherent — atomic on (tenant, period)
Tenant phone number invalid / OTP undeliverablePSP returns at token-bind start400 + phone_invalid; user updates phoneUser edits phone in signup flow
Cancellation invoice references a non-existent originalhoadon.cancel preflight400 + original_invoice_not_foundManual review
RLS bypassed (residency drift detected)TASK-TEN-103 trip-wireINSERT/UPDATE blocked; sev-1 ten.cross_residency_write_blockedInherent — defense-in-depth
vnd_invoice_sequence for new year not yet seededyear row missingFirst-of-year INSERT triggers seed (Jan 1 02:00 UTC scheduled job)Inherent — seed job
eHĐĐT signing succeeds but DB write failsPostgres write error post-signSigned XML temporarily orphaned in eHĐĐT; sev-1 alert; reconciliation job re-linksReconciliation runs nightly + matches eHĐĐT refs
Per-PSP credential leaked (suspicious activity)OBS anomaly detectionSev-1 alert; tenant_admin rotates immediatelyStandard incident response
Hóa đơn line item amounts don't sum to totalpre_save validationReject + sev-2 log; charge proceeds without hóa đơn (illegal — operator must fix manually)Operator validates + reissues

§11 — Implementation notes

§11.1 PSP adapters are independently versioned crates inside services/ten/src/billing/vnd/ — bumping VnPay's API version doesn't touch Momo/ZaloPay.

§11.2 Per-PSP API endpoints are loaded from services/ten/src/billing/vnd/psp_endpoints.yaml (environment-pinned URLs); test/sandbox URLs distinct from prod.

§11.3 Hóa đơn line-item format follows Circular 78/2021/TT-BTC field-naming convention; XSD validation included in hoadon.rs before signing.

§11.4 The vnd_invoice_sequence.notes JSONB stores skipped-number reasons per Decree 123 §10 ("tx rolled back due to KMS unavailable at 2026-05-17T03:14"); auditor can reconstruct gap rationale.

§11.5 Per-PSP idempotency_key encoding constraints: VnPay vnp_TxnRef max 100 chars; Momo requestId max 50; ZaloPay app_trans_id format yyMMdd_<num> max 40. Adapter SHA-1-shortens canonical key when needed; collision risk negligible (~2^60 entropy).

§11.6 Webhook signature verification timing-safe via subtle::ConstantTimeEq to prevent timing attacks.

§11.7 Per-PSP test environments use sandbox accounts; integration smoke tests run nightly (not in PR CI) to validate signature + recurring-charge end-to-end.

§11.8 Token expiration handling: VnPay tokens are 365d default; Momo 180d; ZaloPay 365d. Expiry monitor job emails tenant_admin T-30 days; auto-suspends at T+1 if no rebind.

§11.9 SBV Circular 39/2014 PSP regulations: all 3 chosen PSPs are SBV-licensed e-payment providers; CyberSkill's merchant account requires SBV registration (Stephen handles this commercial workstream out-of-band).

§11.10 The dunning state machine reuses TASK-TEN-003's DunningState enum (Ok/Retry1/Retry2/Retry3/Suspended); same advancement semantics; same un-suspend on success.

§11.11 Per-PSP error mapping into our VndError enum: PSP-specific error codes documented in adapter source files; uniform VndError::PspError { code, message, retryable } shape consumed by upstream handlers.

§11.12 The trace_id thread: signup_session → token_bind → PSP redirect (state param carries trace_id) → callback → INSERT → audit row. PSP-side may not preserve trace_id across redirect; the callback handler RE-INSTATES the original trace_id from server-side session lookup.

§11.13 The vnd_invoice_sequence annual-reset is at Vietnam timezone (UTC+7) midnight, not UTC; matches tax authority convention.

§11.14 Cross-PSP charge attempts (e.g., tenant bound to VnPay token but operator tries Momo) are prevented by vnd_payment_tokens query — there's only one active token per tenant, and its psp column dictates which adapter to invoke.

§11.15 The 7-day idempotency cache TTL (mirror of TASK-TEN-003 DEC-807) — Stripe's idempotency window is 24h, VND PSPs vary (some don't enforce window). 7d is forensically generous + matches existing pattern.

§11.16 Phone number canonical form for hashing: E.164 (+84901234567); user inputs locale-form (0901 234 567) normalised at signup.

§11.17 The description field on PSP charge requests is user-visible on bank statement / wallet — "Cyberos Team plan - 2026-05" chosen for brand recognition + period clarity.

§11.18 Refund hóa đơn (compensating invoice per Decree 123) carries negative line items + references original invoice via original_invoice_ref column (added in slice 3); slice 2 stores via JSONB free-form note.

§11.19 OBS dashboard per-PSP success-rate watching alerts on sustained <95% (3 PSP outages within a week = sev-1 escalation per TASK-OBS-007).

§11.20 Cargo dependency on hmac + sha2 (for VnPay HMAC-SHA512); already in workspace from TASK-INV-005's VietQR pattern.

§11.21 Per-tenant PSP statistics (vnd_charges_total, vnd_charges_failed_total Prometheus counters) labelled by PSP for operator visibility.

§11.22 The vnd_event_dispatch_log mirrors TASK-TEN-003's stripe_event_dispatch_log table shape; consistent operational model across rails.

End of TASK-TEN-102 spec.