Drain-on-demand: SYNAPBUS_DREAM_PARALLEL (default 1) and
`synapbus memory dream-run --parallel N` fan out N concurrent
dream-agent k8s Jobs per (owner, job_type) in one shot. Set
high (e.g. 8) to drain backlog quickly, then back to 1 for normal
hourly operation.
Schema:
- migration 030_dream_parallelism: adds slot INTEGER NOT NULL DEFAULT 0
to memory_consolidation_jobs. Drops + recreates the partial unique
in-flight index as (owner, job_type, slot) so slots 0..N-1 each hold
one in-flight job independently.
Stores:
- JobsStore.CreateOnSlot + CreateNextAvailableSlot.
- ConsolidatorWorker.ForceRunN dispatches N parallel jobs through the
existing launchOne path (extracted from ForceRun).
- core_rewrite coerces to N=1 regardless of the knob — per-(owner,
agent) blob is wholesale-replace and concurrent rewrites would race.
Three bug fixes discovered while bringing the parallel path up on
kubic:
1. k8s Job names collided on rapid relaunch because runner.go used
"synapbus-<agent>-<msg_id>", and dream dispatches have msg_id=0.
Now appends a unique (timestamp%1e6, 4-byte random) suffix when
msg_id is zero; historical "synapbus-<agent>-<id>" prefix preserved.
2. memory_list_unprocessed didn't actually exclude already-refined
messages — the contract said it should, the implementation
returned the same oldest-50 every cycle. The dream agent kept
re-refining the same set: 221 refines links touched only 55
unique dst messages, so progress flat-lined. Added the
NOT IN (refines/duplicate_of/superseded_by) filter and a
from_agent NOT LIKE 'dream:%' clause so the agent never refines
its own reflections.
3. The k8sjob harness was constructed with nil Waiter in main.go,
so every dream dispatch failed instantly with "k8sjob: no Waiter
configured". Now builds a ClientsetWaiter from the in-cluster
clientset.
Plus admin/server.go gets DreamRunN closure + DefaultDreamParallel
(sourced from MemoryConfig.DreamParallel). admin/socket.go
handleMemoryDreamRun accepts `parallel` arg and returns job_ids[].
CLI admin command grows --parallel N flag.
Live evidence from kubic (image v0.21.0-amd64):
1 CLI call with --parallel 8 produced 8 job rows on slots 0..7,
spawned 8 distinct k8s Jobs with unique suffixes, retired ~86
unprocessed messages in <1 min (vs ~10/cycle for the buggy
serial version pre-fix-2).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
