sync — refresh now
One idempotent pass: refresh crawler archives (e.g. slacrawl sync), ingest every available source, embed new rows, rebuild both search sidecars, then lake maintenance (flush inlined data, expire snapshots older than 7 days, compact, clean up). Safe to interrupt and re-run — every step is incremental.
Scoped (sync slack) skips the other sources and the maintenance pass but still ends searchable — embed and index always run.
sync on — stay synced
Three mechanisms, one switch (they compose safely — sync is debounced and single-flight locked):
The streaming daemon runs three lanes, so freshness isn’t tiered by source — it’s bounded only by physics:
duckbrain init runs sync on for you by default (--no-sync to opt out).
Tuning: sync on --every 12h, --debounce 30m, --events SessionEnd, --no-watch / --no-schedule / --no-hooks.
Headless Linux VM: run
loginctl enable-linger $USER once so the user units run without a login session.Why streaming matters
Multiple concurrent agent sessions need each other’s context. Streamed rows are searchable seconds after they land via the search tail leg — bronze rows the sidecar hasn’t indexed yet are fused into every query, so no per-message index rebuild is needed. Session B can ask “what did session A just decide?” while A is still running. Embedding stays out of the hot path: BM25 + recency are fresh instantly, vectors catch up on the next full sync.Concurrency & failure
- Single-flight pid lock (
~/.duckbrain/<instance>/logs/sync.lock) — overlapping triggers no-op instead of racing; stale locks from dead processes are reclaimed. --debounce 15m(used by hooks): exit instantly if a full sync completed recently.- The embed step runs as a subprocess with a hard 30-minute timeout and one retry — a wedged local model runtime can’t hang an unattended sync. If it fails twice, sync continues; vectors catch up next pass.