Task — engineering-spec@1

"memory capture daemon supervision — systemd + launchd units + /healthz + watchdog + crash-restart with exponential backoff + sweeper cron"

doneTASK-MEMORY-110
module memory · class product · priority p0 · created 2026-05-16 · shipped 2026-05-23
depends on TASK-MEMORY-105, TASK-MEMORY-107, TASK-MEMORY-109 · blocks none

§1 — Description (BCP-14 normative)

The capture daemon's runtime supervision MUST consist of: (a) an OS-native init unit, (b) a /healthz HTTP endpoint, (c) a periodic sweeper for ephemeral state, (d) signal-driven graceful shutdown. The contract:

  1. MUST ship a systemd unit cyberos-memory-capture.service (Linux) AND a launchd plist world.cyberos.memory-capture.plist (macOS). The installer (install-daemon.sh) detects the OS and copies the appropriate file to:
  1. MUST auto-restart on non-zero exit per OS-native semantics:
  1. MUST expose GET /healthz on 127.0.0.1:7777 (no external bind; loopback only). Response schema: ``jsonc // 200 OK { "status": "healthy", "version": "0.1.0", "uptime_seconds": 14392, "watched_folders": 12, "queue_depth": 47, "queue_capacity": 10000, "last_emit_ns": 1747407137483000000, "doctor_invariants": {"all_pass": true, "checked_at_ns": 1747407100000000000} } // 503 Service Unavailable { "status": "unhealthy", "reasons": ["doctor_invariant_failed: WatchedFolderResolvable", "queue_saturated"], "queue_depth": 9876, "queue_capacity": 10000, "doctor_invariants": {"all_pass": false, "first_failure": "WatchedFolderResolvable"} } ``
  2. MUST return 503 when ANY of the following are true:
  1. MUST run a sweeper task on a 60-second tokio interval that prunes:
  1. MUST install signal handlers per AGENTS.md operational conventions:
  1. MUST track crash count in /tmp/cyberos-memory-capture-crashes (single u32 LE; incremented at startup; reset to 0 after 5 minutes of stable uptime). The daemon uses this to compute its own startup-delay backoff on launchd.
  2. MUST emit OTel metrics:
  1. MUST emit OTel span memory.capture.sweeper.tick per sweep with attributes pruned_trace_cache, pruned_dedup, pruned_metric_snapshots, duration_ms.
  2. MUST emit a memory.capture_supervisor_event memory audit row when the daemon process starts (kind=started), reloads manifest (kind=reloaded), or exits (kind=exited, exit_code, uptime_seconds). This gives operators a first-class audit trail for "when did the daemon last go down?"
  3. MUST integrate with TASK-OBS-007 alert routing: 503 on /healthz for ≥ 60 seconds triggers a sev-2 alert; ≥ 5 minutes triggers sev-1 (capture is effectively offline; user actions not being recorded).
  4. MUST support cyberos memory capture status CLI which hits /healthz and pretty-prints the response (human-friendly format with colour-coded badges).
  5. MUST support cyberos memory capture logs [--follow] [--lines N] which tails the daemon log file (~/Library/Logs/cyberos-memory-capture.log on macOS, journalctl -u cyberos-memory-capture on Linux).
  6. MUST be installed by install-daemon.sh and uninstalled by uninstall-daemon.sh. Install is idempotent; uninstall is total (removes unit file, stops daemon, removes /tmp files).
  7. SHOULD publish cyberos-memory-capture.service to a Homebrew formula (macOS) and a deb/rpm package (Linux) — slice 3+.

§2 — Why this design (rationale for humans)

Why systemd + launchd (§1 #1)? They are the canonical OS init systems for Linux + macOS. They handle: process supervision, restart on crash, log rotation, signal delivery, dependency ordering. Building a custom supervisor would re-implement 30 years of init-system engineering. Per DEC-161 we do not.

Why exp backoff capped at 5 min (§1 #2)? A daemon that crashes immediately on startup (e.g. corrupted manifest) without backoff would consume parent process CPU and flood logs. Exp backoff is the standard pattern. 5-min cap prevents long outages: operators see "daemon down 5 min, sev-1 alarm" and act; without a cap, an undetected misconfiguration could leave the daemon restart-locked for hours.

Why /healthz on 127.0.0.1:7777 (§1 #3)? Loopback-only because: (a) we don't want external network exposure on capture daemon; (b) the consumers (monitoring agents, TASK-OBS-007 probes) run on the same machine. Port 7777 is unallocated by IANA + memorable. The endpoint is unauthenticated; loopback is the perimeter.

Why 503 on multiple conditions (§1 #4)? Operators need to distinguish "daemon healthy" from "daemon up but useless." Queue saturated = events being dropped silently. Doctor invariant failed = capture wrong / unsafe. Stale emit = likely deadlock. Each condition warrants restart; 503 makes the supervisor handle it (systemd Restart=on-failure with a healthcheck wrapper).

Why 60s sweep interval (§1 #5)? Faster sweeping wastes CPU on idle systems; slower sweeping accumulates /tmp clutter. 60s is the empirical sweet spot: low overhead, low clutter. Operators can SIGHUP to force a sweep early (debugging).

Why crash-count file in /tmp (§1 #7)? launchd doesn't expose crash counts natively (unlike systemd's Restart=on-failure which has RestartSteps built in). To implement exp backoff on macOS, we maintain our own counter. /tmp is the right scope (per-machine, ephemeral); resets to 0 after stable uptime so transient crashes don't lock us into 5-min backoff forever.

Why supervisor_event audit rows (§1 #10)? Operators investigating "memory had a gap between 14:00 and 14:23" need to answer "was the daemon down?" Without a first-class audit row, they have to cross-reference systemd journalctl with memory — friction. The row turns the supervisor's behaviour into queryable data.

Why sev-2 → sev-1 at 5 min (§1 #11)? 60s of capture downtime is recoverable (most editor saves get caught on next batch). 5 min is significant data loss (8+ files of meaningful work). The escalation gives ops a window to self-resolve before paging.

Why no --restart-now CLI command? The OS init system owns the lifecycle. systemctl restart cyberos-memory-capture (Linux) and launchctl kickstart -k gui/$UID/world.cyberos.memory-capture (macOS) are the canonical paths. Adding a Cyberos-level wrapper duplicates them; operators benefit from learning the OS commands once.

Why cyberos memory capture logs (§1 #13)? Cross-platform log access is a friction point: journalctl -u name --follow on Linux vs tailing a file on macOS. The CLI command abstracts the difference.


§3 — API contract

systemd unit

# install/systemd/cyberos-memory-capture.service
[Unit]
Description=CyberOS memory capture daemon
After=network.target
Documentation=https://docs.cyberos.world/runbooks/memory-capture-runbook.html

[Service]
Type=simple
ExecStart=/usr/local/bin/cyberos-memory-capture --foreground
Restart=on-failure
RestartSec=5s
# Exponential backoff (systemd 254+): 5s → 10s → 30s → 1m → 5m → 5m → ...
RestartSteps=5
RestartMaxDelaySec=5min
# Resource limits (sane defaults; operator may override)
MemoryMax=1G
TasksMax=512
# Graceful shutdown deadline (matches §1 #6)
TimeoutStopSec=10s
# Crash notification to journal (parsed by TASK-OBS-001 collector)
StandardOutput=journal
StandardError=journal
# Environment file (writer endpoint, manifest path, OTLP endpoint)
EnvironmentFile=-/etc/cyberos/memory-capture.env

[Install]
WantedBy=default.target

launchd plist

<!-- install/launchd/world.cyberos.memory-capture.plist -->
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>             <string>world.cyberos.memory-capture</string>
  <key>ProgramArguments</key>  <array><string>/usr/local/bin/cyberos-memory-capture</string><string>--foreground</string></array>
  <key>RunAtLoad</key>         <true/>
  <key>KeepAlive</key>         <true/>
  <!-- launchd's anti-flap; daemon's own backoff handles longer delays -->
  <key>ThrottleInterval</key>  <integer>60</integer>
  <key>ExitTimeOut</key>       <integer>10</integer>
  <key>StandardOutPath</key>   <string>/Users/USERNAME/Library/Logs/cyberos-memory-capture.log</string>
  <key>StandardErrorPath</key> <string>/Users/USERNAME/Library/Logs/cyberos-memory-capture.err</string>
  <key>WorkingDirectory</key>  <string>/tmp</string>
  <key>EnvironmentVariables</key>
  <dict>
    <key>RUST_LOG</key>        <string>info,cyberos_memory_capture=debug</string>
    <key>CYBEROS_MANIFEST</key><string>/Users/USERNAME/Library/Application Support/CyberOS/manifest.json</string>
  </dict>
</dict>
</plist>

healthz endpoint

// services/memory-capture/src/healthz.rs
use axum::{Router, routing::get, Json};
use axum::http::StatusCode;
use std::sync::Arc;
use std::time::{Duration, Instant};
use serde::Serialize;

pub struct HealthState {
    pub daemon_start:   Instant,
    pub queue_depth:    Arc<std::sync::atomic::AtomicU64>,
    pub queue_capacity: u64,
    pub last_emit_ns:   Arc<std::sync::atomic::AtomicU64>,
    pub last_doctor:    Arc<tokio::sync::RwLock<DoctorSnapshot>>,
    pub watched_count:  Arc<std::sync::atomic::AtomicU32>,
}

#[derive(Clone, Serialize)]
pub struct DoctorSnapshot {
    pub all_pass:      bool,
    pub first_failure: Option<String>,
    pub checked_at_ns: u64,
}

#[derive(Serialize)]
pub struct HealthReport {
    pub status:           &'static str,
    pub version:          &'static str,
    pub uptime_seconds:   u64,
    pub watched_folders:  u32,
    pub queue_depth:      u64,
    pub queue_capacity:   u64,
    pub last_emit_ns:     u64,
    pub doctor_invariants: DoctorSnapshot,
    #[serde(skip_serializing_if = "Vec::is_empty")] pub reasons: Vec<String>,
}

pub async fn handler(state: Arc<HealthState>) -> (StatusCode, Json<HealthReport>) {
    let now = Instant::now();
    let queue_depth = state.queue_depth.load(std::sync::atomic::Ordering::Relaxed);
    let last_emit_ns = state.last_emit_ns.load(std::sync::atomic::Ordering::Relaxed);
    let doctor = state.last_doctor.read().await.clone();
    let watched = state.watched_count.load(std::sync::atomic::Ordering::Relaxed);

    let mut reasons = Vec::new();
    if !doctor.all_pass {
        reasons.push(format!("doctor_invariant_failed: {}", doctor.first_failure.clone().unwrap_or_default()));
    }
    if queue_depth >= (state.queue_capacity * 95) / 100 {
        reasons.push("queue_saturated".into());
    }
    let stale_emit = watched > 0
        && (unix_ns() as u128).saturating_sub(last_emit_ns as u128) > Duration::from_secs(300).as_nanos();
    if stale_emit {
        reasons.push("stale_emit".into());
    }

    let status = if reasons.is_empty() { "healthy" } else { "unhealthy" };
    let code = if reasons.is_empty() { StatusCode::OK } else { StatusCode::SERVICE_UNAVAILABLE };
    metrics::counter!("memory_capture_health_endpoint_total", "status" => status).increment(1);

    let report = HealthReport {
        status, version: env!("CARGO_PKG_VERSION"),
        uptime_seconds: state.daemon_start.elapsed().as_secs(),
        watched_folders: watched, queue_depth, queue_capacity: state.queue_capacity, last_emit_ns,
        doctor_invariants: doctor,
        reasons,
    };
    (code, Json(report))
}

pub fn router(state: Arc<HealthState>) -> Router {
    Router::new().route("/healthz", get({
        let state = state.clone();
        move || handler(state.clone())
    }))
}

Sweeper

// services/memory-capture/src/sweeper.rs
use tokio::time::{interval, Duration};
use std::time::SystemTime;
use std::path::Path;

pub async fn run_sweeper_loop() {
    let mut ticker = interval(Duration::from_secs(60));
    loop {
        ticker.tick().await;
        let start = std::time::Instant::now();
        let pruned_traces  = prune_dir_older_than("/tmp/cyberos-memory-claude-traces", Duration::from_secs(3600));
        let pruned_metrics = prune_dir_older_than("/tmp/cyberos-memory-metrics",       Duration::from_secs(86400));
        let reset_crashes  = maybe_reset_crash_count("/tmp/cyberos-memory-capture-crashes", Duration::from_secs(300));
        tracing::debug!(
            pruned_trace_cache = pruned_traces,
            pruned_metric_snapshots = pruned_metrics,
            crash_count_reset = reset_crashes,
            duration_ms = start.elapsed().as_millis() as u64,
            "memory.capture.sweeper.tick"
        );
        metrics::counter!("memory_capture_sweeper_pruned_total", "kind" => "trace_cache").increment(pruned_traces as u64);
        metrics::counter!("memory_capture_sweeper_pruned_total", "kind" => "metric_snapshot").increment(pruned_metrics as u64);
        if reset_crashes {
            metrics::counter!("memory_capture_sweeper_pruned_total", "kind" => "crash_count_reset").increment(1);
        }
    }
}

fn prune_dir_older_than(dir: &str, age: Duration) -> u32 {
    let now = SystemTime::now();
    let mut pruned = 0u32;
    if let Ok(rd) = std::fs::read_dir(dir) {
        for entry in rd.flatten() {
            if let Ok(meta) = entry.metadata() {
                if let Ok(mtime) = meta.modified() {
                    if now.duration_since(mtime).map(|d| d > age).unwrap_or(false) {
                        let _ = std::fs::remove_file(entry.path());
                        pruned += 1;
                    }
                }
            }
        }
    }
    pruned
}

fn maybe_reset_crash_count(path: &str, age: Duration) -> bool {
    if let Ok(meta) = std::fs::metadata(path) {
        if let Ok(mtime) = meta.modified() {
            if SystemTime::now().duration_since(mtime).map(|d| d > age).unwrap_or(false) {
                let _ = std::fs::write(path, 0u32.to_le_bytes());
                return true;
            }
        }
    }
    false
}

Installer

#!/usr/bin/env bash
# install/install-daemon.sh
set -euo pipefail

case "$(uname -s)" in
  Linux)
    UNIT=cyberos-memory-capture.service
    DEST="${XDG_CONFIG_HOME:-$HOME/.config}/systemd/user/$UNIT"
    mkdir -p "$(dirname "$DEST")"
    install -m 644 "$(dirname "$0")/systemd/$UNIT" "$DEST"
    systemctl --user daemon-reload
    systemctl --user enable --now "$UNIT"
    echo "✓ installed and started → $DEST"
    ;;
  Darwin)
    PLIST="world.cyberos.memory-capture.plist"
    DEST="$HOME/Library/LaunchAgents/$PLIST"
    sed "s|/Users/USERNAME|$HOME|g" "$(dirname "$0")/launchd/$PLIST" > "$DEST"
    launchctl unload "$DEST" 2>/dev/null || true
    launchctl load -w "$DEST"
    echo "✓ installed and loaded → $DEST"
    ;;
  *)
    echo "Unsupported OS: $(uname -s)" >&2
    exit 1
    ;;
esac

§4 — Acceptance criteria

  1. systemd: daemon starts on systemctl --user start — fixture installs unit; systemctl --user is-active returns active.
  2. systemd: daemon restarts on non-zero exitkill -SEGV $(pidof cyberos-memory-capture) → process restarts within RestartSec=5s ± 2s; MainPID differs.
  3. systemd: exp backoff cap at 5min — crash 10× rapidly → 10th restart waits ~5 min (within ±10s).
  4. launchd: daemon starts on launchctl load — fixture installs plist; launchctl print gui/$UID/world.cyberos.memory-capture shows state=running.
  5. launchd: daemon restarts on crash — kill -9 the process → launchd restarts within ThrottleInterval=60s.
  6. launchd: daemon's own backoff — crash 5× rapidly → daemon-side sleep(min(2^crash_count, 300)) is observable in stderr.
  7. /healthz: 200 on happy — fresh daemon, no failures → curl 127.0.0.1:7777/healthz returns 200; body has status: "healthy".
  8. /healthz: 503 on doctor failure — inject a dangling-symlink fixture → doctor cache reports failure → next /healthz returns 503; reasons includes doctor_invariant_failed.
  9. /healthz: 503 on queue saturation — fill queue to 9500/10000 for ≥ 30 seconds → 503 with queue_saturated.
  10. /healthz: 503 on stale emit — block emitter for 6 minutes (test seam) → 503 with stale_emit.
  11. Sweeper prunes trace cache — write /tmp/cyberos-memory-claude-traces/<uuid> with mtime 2h ago → next tick removes it; metric memory_capture_sweeper_pruned_total{kind="trace_cache"} increments.
  12. Sweeper prunes metric snapshots — fixture file in /tmp/cyberos-memory-metrics/ with mtime 25h ago → pruned.
  13. Sweeper resets crash count after stable uptime — write crash-count file with mtime 6 min ago → sweeper resets to 0.
  14. SIGTERM graceful shutdown — send SIGTERM with full queue → daemon drains queue (≤ 5s), emits memory.capture_supervisor_event kind=exited, exits 0.
  15. SIGHUP reloads manifest — SIGHUP → daemon re-reads manifest; new folders watched; row kind=reloaded emitted.
  16. SIGUSR1 dumps health to stderr — SIGUSR1 → human-formatted health snapshot appears in stderr.
  17. supervisor_event row on start — fresh start → exactly one memory.capture_supervisor_event row with kind=started.
  18. supervisor_event row on exit — SIGTERM → row with kind=exited, exit_code=0, uptime_seconds present.
  19. install-daemon.sh idempotent — run twice → second run is a no-op; unit file byte-identical.
  20. uninstall-daemon.sh total cleanup — uninstall → unit/plist removed; daemon stopped; /tmp/cyberos-memory-capture-crashes removed.
  21. cyberos memory capture status — pretty-prints /healthz JSON with colour badges (PASS/WARN/FAIL per reason).
  22. cyberos memory capture logs --follow --lines 100 — tails the appropriate log source; works on both Linux + macOS.
  23. OTel metrics: uptime + restart count — Prometheus registry has both metrics; restart count survives a SIGSEGV→restart cycle.
  24. TASK-OBS-007 alert fires — 503 for 60s → sev-2 alert with service: memory-capture label.

§5 — Verification

// services/memory/tests/ingest_test.rs

#[tokio::test]
async fn returns_200_when_healthy() {
    let state = build_healthy_state();
    let router = healthz::router(state);
    let req = axum::http::Request::get("/healthz").body(axum::body::Body::empty()).unwrap();
    let resp = router.oneshot(req).await.unwrap();
    assert_eq!(resp.status(), 200);
    let body: HealthReport = parse_body(resp).await;
    assert_eq!(body.status, "healthy");
    assert!(body.reasons.is_empty());
}

#[tokio::test]
async fn returns_503_when_doctor_failed() {
    let state = build_healthy_state();
    *state.last_doctor.write().await = DoctorSnapshot {
        all_pass: false, first_failure: Some("WatchedFolderResolvable".into()),
        checked_at_ns: unix_ns(),
    };
    let resp = healthz::router(state).oneshot(get_healthz_req()).await.unwrap();
    assert_eq!(resp.status(), 503);
    let body: HealthReport = parse_body(resp).await;
    assert!(body.reasons.iter().any(|r| r.starts_with("doctor_invariant_failed")));
}

#[tokio::test]
async fn returns_503_when_queue_saturated() {
    let state = build_healthy_state();
    state.queue_depth.store(9600, std::sync::atomic::Ordering::Relaxed);
    let resp = healthz::router(state).oneshot(get_healthz_req()).await.unwrap();
    assert_eq!(resp.status(), 503);
    let body: HealthReport = parse_body(resp).await;
    assert!(body.reasons.contains(&"queue_saturated".to_string()));
}
// services/memory/tests/ingest_test.rs
#[test]
fn prunes_files_older_than_ttl() {
    let tmpdir = tempdir().unwrap();
    let dir = tmpdir.path().join("traces");
    std::fs::create_dir_all(&dir).unwrap();
    let path = dir.join("00000000-0000-0000-0000-000000000001");
    std::fs::write(&path, b"x").unwrap();
    // Backdate mtime by 2 hours
    set_mtime(&path, std::time::SystemTime::now() - std::time::Duration::from_secs(7200));

    let pruned = sweeper::prune_dir_older_than(dir.to_str().unwrap(), Duration::from_secs(3600));
    assert_eq!(pruned, 1);
    assert!(!path.exists());
}

#[test]
fn preserves_fresh_files() {
    let tmpdir = tempdir().unwrap();
    let dir = tmpdir.path().join("traces");
    std::fs::create_dir_all(&dir).unwrap();
    let path = dir.join("fresh");
    std::fs::write(&path, b"y").unwrap();
    let pruned = sweeper::prune_dir_older_than(dir.to_str().unwrap(), Duration::from_secs(3600));
    assert_eq!(pruned, 0);
    assert!(path.exists());
}
# services/memory-capture/tests/restart_e2e_test.sh
#!/usr/bin/env bash
set -euo pipefail

bash install/install-daemon.sh
sleep 2
[ "$(systemctl --user is-active cyberos-memory-capture)" = "active" ]

# Crash + restart
ORIG_PID=$(systemctl --user show -p MainPID --value cyberos-memory-capture)
kill -9 "$ORIG_PID"
sleep 7
NEW_PID=$(systemctl --user show -p MainPID --value cyberos-memory-capture)
[ "$ORIG_PID" != "$NEW_PID" ]
[ "$(systemctl --user is-active cyberos-memory-capture)" = "active" ]

# Crash 10× rapidly → 10th restart should be at 5 min cap (sample 3 to keep CI under 5 min)
for i in 1 2 3; do
  kill -9 $(systemctl --user show -p MainPID --value cyberos-memory-capture)
  sleep 1
done
# After 3 crashes, RestartSec should be 30s (per RestartSteps); verify within ±5s
START=$(date +%s)
while [ "$(systemctl --user is-active cyberos-memory-capture)" != "active" ]; do
  sleep 1
  [ $(($(date +%s) - START)) -lt 40 ] || { echo "did not restart within 40s"; exit 1; }
done

bash install/uninstall-daemon.sh

§6 — Implementation skeleton

(API contract above is the skeleton.)

// services/memory-capture/src/main.rs (excerpt — daemon entrypoint)

#[tokio::main]
async fn main() -> cyberos_cli_exit::ExitCode {
    use cyberos_cli_exit::ExitCode;
    let cli = Cli::parse();
    cyberos_obs_sdk::init("memory-capture");

    // Crash-count exp-backoff (launchd-side)
    if cfg!(target_os = "macos") {
        let count = read_crash_count();
        let delay_s = std::cmp::min(2u64.pow(count.min(10)), 300);
        if delay_s > 0 {
            tracing::warn!(count, delay_s, "delaying startup for exp backoff");
            tokio::time::sleep(Duration::from_secs(delay_s)).await;
        }
        increment_crash_count();
    }

    // Doctor gate (TASK-MEMORY-105 §1 #13)
    if let Err(e) = run_doctor_gate().await {
        tracing::error!(?e, "doctor invariant failure; refusing to start");
        return ExitCode::InternalError;
    }

    // Start the daemon (TASK-MEMORY-107) and supervisory pieces
    let daemon = cyberos_memory_capture::CaptureDaemon::start(cli.manifest, cli.dry_run).await
        .expect("daemon start failed");
    let health_state = daemon.health_state();

    // Bind /healthz on 127.0.0.1:7777
    let healthz_listener = tokio::net::TcpListener::bind("127.0.0.1:7777").await.unwrap();
    tokio::spawn(async move {
        axum::serve(healthz_listener, cyberos_memory_capture::healthz::router(health_state)).await.unwrap();
    });

    // Spawn sweeper
    tokio::spawn(cyberos_memory_capture::sweeper::run_sweeper_loop());

    // Emit supervisor_event:started
    daemon.emit_supervisor_event("started", None).await;

    // Signal-driven main loop
    let mut sighup  = tokio::signal::unix::signal(tokio::signal::unix::SignalKind::hangup()).unwrap();
    let mut sigusr1 = tokio::signal::unix::signal(tokio::signal::unix::SignalKind::user_defined1()).unwrap();
    let mut sigterm = tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate()).unwrap();
    let mut sigint  = tokio::signal::unix::signal(tokio::signal::unix::SignalKind::interrupt()).unwrap();
    let start = std::time::Instant::now();
    loop {
        tokio::select! {
            _ = sighup.recv()  => { let _ = daemon.reload().await; daemon.emit_supervisor_event("reloaded", None).await; }
            _ = sigusr1.recv() => { eprintln!("{:#?}", daemon.health_state().snapshot().await); }
            _ = sigterm.recv() => { daemon.stop().await; daemon.emit_supervisor_event("exited", Some(0)).await; break; }
            _ = sigint.recv()  => { daemon.stop().await; daemon.emit_supervisor_event("exited", Some(0)).await; break; }
        }
    }
    ExitCode::Ok
}

§7 — Dependencies


§8 — Example payloads

Happy /healthz

{
  "status": "healthy",
  "version": "0.1.0",
  "uptime_seconds": 14392,
  "watched_folders": 12,
  "queue_depth": 47,
  "queue_capacity": 10000,
  "last_emit_ns": 1747407137483000000,
  "doctor_invariants": {
    "all_pass": true,
    "first_failure": null,
    "checked_at_ns": 1747407100000000000
  }
}

Unhealthy /healthz

{
  "status": "unhealthy",
  "version": "0.1.0",
  "uptime_seconds": 432,
  "watched_folders": 12,
  "queue_depth": 9876,
  "queue_capacity": 10000,
  "last_emit_ns": 1747406500000000000,
  "doctor_invariants": {
    "all_pass": false,
    "first_failure": "WatchedFolderResolvable",
    "checked_at_ns": 1747407100000000000
  },
  "reasons": [
    "doctor_invariant_failed: WatchedFolderResolvable",
    "queue_saturated",
    "stale_emit"
  ]
}

memory.capture_supervisor_event (start)

{
  "kind": "memory.capture_supervisor_event",
  "payload": {
    "event":     "started",
    "version":   "0.1.0",
    "pid":       4823,
    "host_id":   "mac-stephen-001",
    "started_at_ns": 1747407137483000000
  }
}

§9 — Open questions

All resolved. Deferred:


§10 — Failure modes inventory

FailureDetectionOutcomeRecovery
Process SEGFAULTOS signalsystemd/launchd restart per backoffNone — automatic
OOM killOOM-killerRestart; sev-1 if MemoryMax repeatedly hitOperator investigates leak or raises limit
Manifest corrupted (bad JSON)Daemon exits 1 at bootBackoff escalates to 5min; sev-1Operator restores manifest
Doctor gate failsDaemon exits 7 at bootBackoff escalates; sev-2Operator runs cyberos doctor to identify failure
/healthz port 7777 in usebind() ErrDaemon exits 1 with stderr; sev-1Operator frees port or sets env CYBEROS_HEALTHZ_PORT
Queue saturated for 30shealth check 503systemd's healthcheck wrapper restarts (or sev-2 alarm)Operator investigates emitter
Stale emit > 5minhealth check 503RestartOperator investigates writer
Sweeper task panicstokio task watchdogProcess exits; restartOperator investigates
/tmp fullsweeper writes failsweeper logs WARN; continues with in-memOperator frees /tmp; auto-recovers
Crash count file unreadableread returns 0 (treated as fresh)No backoff; possible rapid-restartOperator restores file or accepts behaviour
Crash count file write fails (/tmp ENOSPC)write ErrBackoff math degrades to "no backoff"; restart loop tightOperator frees /tmp
Two daemon instances (operator error)second instance: bind() Err on healthz portSecond exits 1; first continuesOperator stops one
systemd unit file ENOENT after upgradesystemctl daemon-reload finds nothingDaemon dies; uninstall triggerOperator reinstalls
launchd plist syntax errorlaunchctl load ErrDaemon doesn't startOperator fixes plist
OTel exporter downmetric/span bufferingBuffer fills; oldest dropped; daemon survivesOperator restores TASK-OBS-001
cyberos memory capture status can't reach /healthzconnection refusedCLI prints error; exit 1Operator checks daemon is running
cyberos memory capture logs on macOS but log file path wrongtail ErrCLI prints errorOperator updates plist path or accepts default
RestartMaxDelaySec hit; daemon-down 5+ minTASK-OBS-007 sev-1 pageOperator investigates manuallyManual fix and restart
Network mount unavailable when daemon startsmanifest path unreadableexit 1; backoff; sev-2Operator remounts

§11 — Implementation notes


End of TASK-MEMORY-110.

As built (2026-07-02)

Shipped inside modules/memory (serve.py healthz/sweeper), not a services/memory-capture crate.