"KB FTS5 + PGroonga lexical search — VN bigram tokenisation + English stemming + per-tenant index with tier filter"

§1 — Description (BCP-14 normative)

The KB service MUST ship lexical search at services/kb/src/search/lexical.rs with PGroonga primary + FTS5 fallback + VN bigram + tier filter, 3 memory audit kinds.

1. MUST validate lexical_engine against closed enum per DEC-1911.

1. MUST index at migration 0004: ``sql CREATE EXTENSION IF NOT EXISTS pgroonga; ALTER TABLE kb_documents ADD COLUMN search_doc_tsv TSVECTOR; CREATE INDEX kb_docs_pgroonga_idx ON kb_documents USING pgroonga ((slug || ' ' || title || ' ' || rendered_plaintext)) WITH (tokenizer='TokenBigramSplitSymbolAlphaDigit'); CREATE INDEX kb_docs_fts5_idx ON kb_documents USING gin(to_tsvector('english', slug || ' ' || title || ' ' || rendered_plaintext)); ``

1. MUST tokenise per DEC-1912 — bigram for VN tenants, Snowball English for others. Detection per tenant.locale.

1. MUST query at lexical.rs::search(tenant, query, engine?, limit):

• Default engine = pgroonga
• Fallback to fts5 on PGroonga error
• Apply RLS tier filter (TASK-KB-003)
• Return ranked snippets

1. MUST update index synchronously on doc version commit per DEC-1913 — TRIGGER ON INSERT/UPDATE kb_documents.

1. MUST expose endpoint: ``text POST /v1/kb/search/lexical body: {query, engine?, limit?, filters?} ``

1. MUST emit 3 memory audit kinds per DEC-1915. PII per TASK-MEMORY-111: query text SHA-256 hashed.

1. MUST thread trace_id from query → engine → audit.

1. MUST NOT return docs the user lacks permission for per DEC-1914 (RLS + tier filter).

1. MUST NOT skip VN bigram for VN tenants per DEC-1912.

§2 — Why this design

Why PGroonga primary (DEC-1910)? Better tokenisation for VN/CJK languages; FTS5 weak on multi-byte UTF-8 word boundaries.

Why FTS5 fallback (DEC-1910)? PGroonga extension may not be available in all envs (e.g. dev SQLite); fallback maintains feature.

Why bigram for VN (DEC-1912)? Vietnamese has no spaces in compound words (e.g. "côngtyTNHH"); bigram catches partial matches.

Why sync index (DEC-1913)? Search-as-you-type UX needs fresh results; async lag = stale UX.

§3 — API contract

POST /v1/kb/search/lexical

Sample request:

{
  "query": "công ty TNHH thanh toán",
  "engine": "pgroonga",
  "limit": 10,
  "filters": {"category": "finance"}
}

Sample response:

{
  "results": [
    {
      "doc_id": "uuid",
      "title": "Quy trình thanh toán cho công ty TNHH",
      "snippet": "... <b>công ty TNHH</b> cần xuất hóa đơn cho mỗi <b>thanh toán</b>...",
      "rank": 0.92,
      "engine": "pgroonga"
    }
  ],
  "total": 1
}

§4 — Acceptance criteria

1. lexical_engine enum cardinality 2. 2. PGroonga primary. 3. FTS5 fallback on PGroonga error. 4. VN bigram for VN tenant. 5. English stemming for global tenant. 6. Tier filter applied (RLS + visibility_tier). 7. Synchronous index update via trigger. 8. 3 memory audit kinds emitted. 9. PII scrubbed (query text SHA256). 10. RLS denies cross-tenant. 11. Trace_id preserved. 12. Snippet highlighting. 13. Result rank score returned. 14. Pagination support. 15. filters parameter supported. 16. Empty result returns empty array (not 404). 17. Eventual consistency ≤ 1s post-write. 18. Search performance < 100ms p95. 19. Query length capped 500 chars. 20. Cross-language search per locale.

§5 — Verification

#[tokio::test]
async fn vn_bigram_matches() {
    let ctx = TestContext::vn_tenant_with_doc("Quy trình thanh toán cho công ty TNHH").await;
    let r = ctx.lexical_search("công ty TNHH").await;
    assert!(!r.results.is_empty());
}

#[tokio::test]
async fn english_stem_matches_plural() {
    let ctx = TestContext::en_tenant_with_doc("invoicing best practices").await;
    let r = ctx.lexical_search("invoice").await;
    assert!(!r.results.is_empty());
}

#[tokio::test]
async fn tier_filter_excludes_role_restricted() {
    let ctx = TestContext::with_public_and_role_restricted_doc().await;
    let r = ctx.lexical_search_as(ctx.am_user, "topic").await;
    let restricted_ids = ctx.role_restricted_doc_ids();
    assert!(r.results.iter().none(|d| restricted_ids.contains(&d.doc_id)));
}

// 5.4..5.10

§7 — Dependencies

Upstream: TASK-KB-001, TASK-KB-003. Downstream: TASK-KB-006 (rerank consumes lexical results). Cross-module: TASK-MEMORY-111 (PII).

§10 — Failure modes

§11 — Implementation notes

• §11.1 PGroonga TokenBigramSplitSymbolAlphaDigit handles VN + English mixed.
• §11.2 Trigger on kb_documents INSERT/UPDATE refreshes tsvector + reindex row.
• §11.3 Result snippet via PGroonga's snippet_html() function with <b> highlight.
• §11.4 memory audit body: tenant_id, engine, result_count; query SHA256.
• §11.5 Pagination via OFFSET/LIMIT; cursor-based for >1k results.

End of TASK-KB-004 spec.