Task — engineering-spec@1

Benchmark gates G1-G16 - checkers, risk register, BRAIN recording

doneTASK-IMP-140
module improvement · class product · priority p1 · created 2026-07-23 · shipped null
depends on TASK-MEMORY-303 · blocks none

TASK-IMP-140: Benchmark gates G1-G16 - checkers, risk register, BRAIN recording

Summary

The 2026-07-23 deep audit defined sixteen benchmark gates - drift-prevention criteria that make its findings re-checkable forever instead of a one-time snapshot. This task lands them as product: docs/verification/benchmark-gates.md carrying all sixteen definitions (embedded in full below, so this spec is self-contained); automated checkers for the six gates no sibling hardening task owns (G3 enum cross-check, G4 headline counts, G5 payload reference walker, G6 vendored-gate smoke, G13 stuck-WIP detector, G16 reinstall idempotency), wired into CI through the existing suite-discovery glob; the audit's seven risk entries appended to docs/reference/risk-register.md as R-EXT-* rows; and the audit + its decisions recorded into the BRAIN per §13 - a step that depends on TASK-MEMORY-303 unfreezing the live store first.

Problem

The audit's issue register is point-in-time: every finding it verified (fail-open gates, enum forks, payload gaps, schema drift...) regrows the moment attention moves, because nothing mechanical re-measures. The audit therefore defined G1-G16 with pass/fail criteria and automation tiers - but they exist only in a conversation transcript. Meanwhile docs/reference/risk-register.md predates the audit (its newest rows are project-context R-EXT entries, none covering self-approval, vacuous gates, or the config-wipe class the audit demonstrated), and the audit itself is recorded in no durable store: the BRAIN is FROZEN_RECOVERABLE (layout failure), so the repo's own doctrine - record decisions and audits into the BRAIN - is currently unsatisfiable.

Proposed Solution

Author docs/verification/benchmark-gates.md from the embedded definitions below: one section per gate (purpose, pass/fail, severity, test method, automation tier, checked files, owning checker), plus a status table the checkers can be diffed against. Implement scripts/tests/test_benchmark_gates.sh with one function per unowned gate - t_g03 (parse the status enum from STATUS-REFERENCE.md §1 and compare byte-for-byte against RUBRIC.md FM-104's list, task-lint.mjs STATUSES, render-status-hub.mjs STATUSES, and the vendored BACKLOG template's off-ramp vocabulary), t_g04 (recompute module/workflow/task counts and compare against README.md + docs/README.md headline claims), t_g05 (walk every path reference in vendored docs/skills against the built payload's file set - the skill-log.mjs class), t_g06 (scratch-install smoke: the vendored caf/awh gate entry points exit with semantic codes, never 127/not-found), t_g13 (report-only detector: in-flight statuses older than N=30 days emitted as a triage list; detection automated, decision human), t_g16 (install -> reinstall on a scratch repo diffs .cyberos/ clean modulo timestamped backups AND preserves a pre-set config.yaml override - the C1 wipe class). The suite auto-registers via run_all.sh's glob and therefore rides TASK-IMP-128's CI job and the TASK-IMP-136 workflow when those land. Append the seven audit risk entries to the risk register as R-EXT rows with the register's full field set. Finally - after TASK-MEMORY-303 lands - record the audit verdict, the sixteen gates, and the hardening-wave decisions into the BRAIN as memory files + chained audit rows per §13, closing the loop the frozen store left open.

Alternatives Considered

Success Metrics

Scope

In scope: benchmark-gates.md, the six-checker suite, the risk-register extension, the BRAIN recording step, CHANGELOG.

Out of scope / Non-Goals

Dependencies

depends_on: [TASK-MEMORY-303] - the BRAIN recording step (clause 1.6) requires the store un-frozen by 303's layout repair (303 carries the reciprocal blocks entry). The dependency gates only that clause: the doc, checkers, and register rows are implementable immediately, and the task may ship its non-BRAIN clauses while 303 is in flight per the slice plan, but final acceptance includes the recording. Soft references (no cycles): TASK-IMP-128 + TASK-IMP-136 carry the gate suite into CI; TASK-CUO-302/303/304, TASK-SKILL-202, TASK-IMP-137/138/139 own the mapped checkers.

AI Authorship Disclosure

Gate definitions (G1-G16) - embedded normative reference

The content contract for docs/verification/benchmark-gates.md. Severity: how bad a regression is. Tier: ci (fully automated), ci+human (automated floor, periodic human judgment), detect+human (automated detection, human decision). Owner: the checker that enforces it.

G1 - Gate-floor non-vacuous (severity: critical, tier: ci)

G2 - HITL mechanical lock (severity: critical, tier: ci; verdicts stay human)

G3 - Status-enum single source (severity: high, tier: ci)

G4 - Headline-count truth (severity: medium, tier: ci)

G5 - Payload completeness (severity: high, tier: ci)

G6 - Vendored-gate executability (severity: high, tier: ci)

G7 - Skill quality floor (severity: high, tier: ci+human)

G8 - Injection-discipline coverage (severity: high, tier: ci+human)

G9 - BRAIN health in gates (severity: high, tier: ci)

G10 - Schema copy consistency (severity: high, tier: ci)

G11 - Loop-bound single-sourcing (severity: high, tier: ci)

G12 - UNREVIEWED hygiene (severity: high, tier: ci)

G13 - Stuck-WIP detection (severity: medium, tier: detect+human)

G14 - CI parity & stub honesty (severity: high, tier: ci)

G15 - Entry-point consistency (severity: medium, tier: ci+human)

G16 - Idempotent reinstall (severity: high, tier: ci)

Risk-register extension (content contract for the seven R-EXT rows)

Each row carries the register's full field set (description, cause, impact, detection, prevention, recovery, automation tier), sourced from the audit: (1) Self-approval / skipped HITL - no mechanical lock; unreviewed work marked done; detect: audit chain lacks verdict rows; prevent: G2; recover: reconcile + done -> ready_to_review flip. (2) Vacuous green gates - fail-open floor + autodetect unknown; false confidence at both HITL gates; prevent: G1; recover: re-run with restored config. (3) Config wipe on reinstall - gates.env regeneration; silent loss of the only test command (observed); prevent: G16 + durable config.yaml; recover: .bak restore. (4) Prompt injection via repo files - repo-reading skills without untrusted discipline; steered workflows in consumer repos; prevent: G8; detect: injection-marker scan. (5) Payload/doc divergence - docs referencing unvendored files; consumer workflows halt on missing tools; prevent: G5. (6) Partial install window - rm/cp vendor step; broken .cyberos/ mid-install; prevent: staged swap (TASK-IMP-137) + G16; recover: re-run install. (7) BRAIN frozen-by-layout - store pollution; protocol-compliant agents must refuse writes, audit trail silently absent; prevent: G9; recover: operator-gated move (TASK-MEMORY-303).

1. Description (normative)

2. Acceptance criteria

3. Edge cases

Audit

§1 — Verdict summary

Seven §1 clauses, seven ACs, seven edge cases, plus two embedded content contracts (the full G1-G16 definitions and the seven risk-register rows) that make the spec self-contained per the coordinator's instruction. The audit's pressure points: one-owner-per-gate discipline (nine of sixteen checkers belong to sibling tasks and must not be re-implemented here), the green-at-HEAD paradox for checkers that measure unfixed defects, and protocol-legality of the BRAIN recording step.

§2 — Findings (all resolved)

ISS-001 — checker ownership was ambiguous in the first draft (sixteen gates, one suite)

Implementing all sixteen checkers here would duplicate the G1/G2/G7-G12/G14/G15 checkers that the sibling hardening tasks ship as their own test suites - two authorities per gate, guaranteed drift. Resolved: source_decisions records the one-gate-one-checker-one-owner rule; every embedded gate definition carries an Owner line; clause 1.2 scopes this task's suite to exactly the six unowned checkers; the doc's status table is the coordination surface.

ISS-002 — the suite could not be green at HEAD if its checkers enforce unshipped fixes

G3's enum fork and G6's vendored-CAF structural failure are real defects today, owned by sibling tasks; a suite that enforces them fails run_all.sh repo-wide the day it lands (and the pre-commit hook would block every commit). Resolved: clause 1.3 defines report-only mode with the doc's status table as the declared state and the flip-to-enforcing tied to the owning fix; AC 3 asserts green-at-HEAD with report-only gates declared.

ISS-003 — the BRAIN recording step was protocol-illegal as first drafted

Recording "as soon as possible" would write to a store that cyberos doctor reports below READY - §12 forbids it and §1's pre-write checklist halts on it; the audit's own record would violate the protocol it measured. Resolved: depends_on: [TASK-MEMORY-303] (with the reciprocal blocks entry on 303), and clause 1.6 carries an explicit READY-precondition so even a manual override cannot execute the clause against a frozen store without visibly breaking the spec.

ISS-004 — G13 could be read as an auto-triage

A detector that flips stale tasks to on_hold "helpfully" is a status mutation without a verdict - the exact class TASK-CUO-303 locks. Resolved: the G13 definition pins detect+human tier; clause 1.4 forbids status changes; AC 4 asserts byte-identical specs after a detector run.

ISS-005 — G5's walker would false-positive on illustrative paths

Vendored docs legitimately mention example paths that are not delivery promises; a naive walker fails on them and someone "fixes" it by weakening the walk. Resolved: edge case defines the inline exemption marker with the allowlist-with-reasons pattern (visible in diffs, greppable), keeping the walker strict for real references.

ISS-006 — G16's "modulo timestamps" could silently grow

An open-ended diff-exclusion list is how an idempotency check becomes a tautology. Resolved: edge case requires the exclusion list to live in the checker with a per-entry comment - growth is reviewable, not silent.

ISS-007 — transcript provenance vs §11 untrusted-content discipline

The gate definitions originate in a conversation transcript, which is untrusted-by-protocol for authorizing anything. Resolved: the definitions are embedded here as the operator-approved plan's Phase-3 content (the plan approval is the authorization; the transcript is provenance), normalized to testable shape - and the Alternatives entry records why verbatim transcript import was rejected.

§3 — TRACE-006 semantic sufficiency (per clause)

ClauseVerb demandCited test assertsVerdict
1.1 MUST carry 16 gates, 7 fields each, status table, matching this specsection census + field completeness + severity/tier matchAC 1: asserts all threesufficient
1.2 MUST implement six checkers, each failing on violation, glob-registeredsix negative fixtures + discoveryAC 2: asserts the negatives; AC 3/AC 6 cover discovery + modessufficient
1.3 report-only MUST be loud + declared; suite green at HEADgreen suite + report blocks + table declarationAC 3: asserts all threesufficient after revision (ISS-002)
1.4 MUST NOT change statusbyte-identical corpus after runAC 4: asserts it directlysufficient after revision (ISS-004)
1.5 MUST add seven complete R-EXT rows with G-referencesrow count + field completeness + G-reference presenceAC 5: asserts all threesufficient
1.6 MUST record post-303 through canonical writer, READY-gatedfixture-store demonstration in CI + live-store verification at final acceptanceAC 6: asserts the fixture half; the live half is the HITL gate's evidencesufficient after revision (ISS-003)
1.7 MUST record four deliverablesfour substrings in top entryAC 7: asserts all foursufficient

§4 — Resolution

Seven findings - two protocol-legality, five material - all resolved in the audited revision. Score = 10/10.

Status transition draft -> ready_to_implement is authorised by this verdict per STATUS-REFERENCE.md §1.1 - with the note that the depends_on: [TASK-MEMORY-303] edge gates clause 1.6 (and final acceptance), while clauses 1.1-1.5/1.7 are buildable immediately. The two human-acceptance gates in /ship-tasks are unchanged - this audit clears the spec-correctness gate only.


End of TASK-IMP-140 audit.