27 Commits
Author SHA1 Message Date
QiuSWandClaude Opus 5.5 198b043186 docs: add worktree lifecycle and cleanup rules
Feature work happens in D:/OPC/synapbus-wt/issue-<N>; after the branch
is merged into opc/main (and the running binary replaced if needed) the
implementing agent removes the worktree with git worktree remove, after
restoring the skip-worktree go.mod/go.sum patch. Never --force, never
delete branches, never remove unmerged or dirty worktrees.

OPC local patch (opc/main only).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 14:21:44 +08:00
QiuSWandClaude Opus 5.5 92a2e034c0 merge: agent-key authenticated SSE stream /api/agent-events (#1)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:20:19 +08:00
QiuSWandClaude Sonnet 5.5 5da3a5d2a9 fix(#1): encode agent-events error bodies with encoding/json
The 429 body was invalid JSON because the agent name was interpolated with %q.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-05 16:19:41 +08:00
QiuSWandClaude Sonnet 5.5 def4bb55a8 fix(#1): keep out-of-order live agent events, add store tests
Replay dedup now uses a fixed replayedUpTo that live events never advance.
Add ListEventMetaAfter store tests, document current-membership replay,
and skip member/subject lookups when no agent stream is connected.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-05 16:15:13 +08:00
QiuSWandClaude Sonnet 5.5 2e8a49ca5a feat(#1): add agent-key authenticated SSE stream /api/agent-events
Per-agent SSE (metadata-only new_message events, id=message_id, 30s heartbeat,
Last-Event-ID replay capped at 200 with resync_required, max 5 connections per
agent, slow clients dropped). Mounted behind the agent API-key middleware.
/api/events is unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-05 16:12:39 +08:00
QiuSWandClaude Opus 5.5 352f9f6c61 docs: add OPC AGENTS.md with fork, run and secret rules
Document the Gitea fork workflow (origin/upstream, main vs opc/main),
localhost-only runtime, admin socket path, skip-worktree on the built
index.html, and that data/ and API keys must never be committed.

OPC local patch (opc/main only).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:01:39 +08:00
QiuSWandClaude Opus 5.5 cdae55e1d5 build(windows): pin renameio v0.1.0 for hnsw on Windows
github.com/TFMV/hnsw uses renameio.TempFile, which renameio v1 does not
provide on Windows, so the pristine tree fails to build there with
"undefined: renameio.TempFile". Pin v0.1.0 via a replace directive.

OPC local patch (opc/main only); not intended for upstream as-is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:00:53 +08:00
Algis DumbrisandClaude Opus 4.7 0d4a9539b5 fix(watchdog): add sqlite to runtime + aggregate today usage across owners
Two silent failures making the watchdog's circuit-breaker checks
toothless:

1) The Alpine runtime image had no sqlite3 binary, so every
   `kubectl exec -- sqlite3 …` call inside the watchdog returned empty.
   $JS/$TIN/$JOK/$JFL/$JCB all defaulted to 0 → every "soft cap" /
   "tokens > 30M" / "circuit broke but still firing" check trivially
   passed regardless of real state. apk add sqlite (≈700KB).

2) The "today usage" query filtered to owner_id='2' only, but dream
   jobs run for any owner (we just saw a clean owner_id=1 dispatch).
   Replace with SUM across all rows for date=date('now'); the caps
   are intentionally global, not per-owner.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 06:16:58 +03:00
Algis DumbrisandClaude Opus 4.7 8a70ba7440 fix(api): route analytics handlers through read pool
The four /api/analytics/* endpoints (timeline, summary, top-agents,
top-channels) all ran their SELECT queries on the write pool
(MaxOpenConns=1, serialized) and would time out at 125s with
context-canceled whenever a long writer (e.g. dream dispatch) held the
single connection. Summary swallows the error and returns {0,0,0}, so
the dashboard rendered an empty-cluster lie.

Plumb ReadDB through RouterConfig from main, fall back to DB if the
read pool is unset, and pass it to NewAnalyticsHandler. Same shape as
the /readyz fix in b03350f.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-18 15:01:07 +03:00
Algis DumbrisandClaude Opus 4.7 b03350ffc1 fix(health): route /readyz through read pool to survive long writers
Repeated production crash: pod ran 27min–3.5h then went ready=false,
restarts=0 (process alive but readiness probe failing). Watchdog
correctly scaled deploy to 0 each time.

Root cause: /readyz calls db.PingContext() on the write pool, which
has MaxOpenConns=1 (serialized writes). The consolidator's dream-job
dispatch (introduced in 020) holds that single connection for 30s+
during one tick: it creates a K8s Job, writes the job row, issues a
dispatch token, all sequentially. /readyz blocks waiting for the
connection through the entire dispatch. With probe period=5s,
failureThreshold=3, the pod flips to NotReady after ~15s — long
before the dispatch finishes.

The new diagnostic: rebuilt v0.17.0 (pre-020) on kubic — runs 5h+
clean, memory flat at 134Mi. v0.21.2 (with 020) dies within hours.
The dispatch path is the only ~30s write holding the conn.

Fix: pass db.QueryDB() to health.NewChecker. QueryDB returns the
read pool (MaxOpenConns=8) when available, write pool when not, so
the readiness probe can run concurrently with any writer.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-15 15:17:42 +03:00
Algis DumbrisandClaude Opus 4.7 c4d888082d fix(mcp): gate suggester on leading verb match
Production v0.18.1 logs showed read_message → "did you mean: send_message"
— two edits away by Levenshtein, but the opposite intent. An agent asking
to READ a single message getting nudged toward SEND is actively
misleading. Same trap was active for anything sharing a verb-less suffix
like _message, _channel, _task.

Constrain the Levenshtein candidates to those whose leading verb (token
before the first underscore) matches the input verb exactly. Substring
matching is unchanged. Drop the now-stale sned_message typo test case
(cross-verb typo correction is no longer in scope) and add a regression
test covering read_message, delete_message, fetch_channel.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 21:39:43 +03:00
Algis DumbrisandClaude Opus 4.7 9d0c5696cd fix: expiry pre-check on read pool + memory_rewrite_core hint
After v0.18.0 deploy two residual issues remained on kubic.

ExpireTasks: 6 "context deadline exceeded" errors in 2.5h despite the
v0.18.0 partial index. Root cause is connection-pool contention, not
SQL speed — the tasks table is empty (steady state) and the query
plan correctly uses idx_tasks_expiry, but the worker still queues
behind the serialized write connection (MaxOpenConns=1) when another
writer holds it for >30s. Fix: add an EXISTS pre-check on the read
pool. If nothing matches, return (0, nil) without touching the write
pool. Wired via SQLiteTaskStore.WithReadDB to avoid changing the
constructor signature and disrupting tests.

Bridge: bridgeTopLevelOnly only hinted on the misspelled
rewrite_core_memory. Agents have since learned and call the real name
memory_rewrite_core via call(), which fell through to plain "unknown
action: memory_rewrite_core". Add the real name to the hint map so
both spellings get the targeted "this is a top-level MCP tool"
message.

Tests:
- ExpireTasks_EmptyShortCircuits: 0-row table returns (0, nil)
- ExpireTasks_UsesReadPoolForPreCheck: pre-check runs on read pool,
  UPDATE still runs on write pool when work is present
- TopLevelToolHint: extended to cover memory_rewrite_core via bridge

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 10:19:11 +03:00
Algis DumbrisandClaude Opus 4.7 e59fb5a1e2 fix(mcp): alias common wrong action names + "did you mean" suggestions
Agents call() into the bridge with action names they guess from prior MCP
conventions or their training data — e.g. `read_channel`, `search`,
`my_status`, `read_dm`, `read_article`. Each produced a useless
"unknown action: X" WARN and no progress.

This change:
- Adds a small alias map (bridgeActionAliases) for observed wrong names
  that have a single unambiguous bridge equivalent:
    read_channel → get_channel_messages
    search       → search_messages
    read_dm      → read_inbox
    my_status    → read_inbox
    read_article → get_article
- Adds bridgeTopLevelOnly for wrong names whose real implementation lives
  as a top-level MCP tool, not a bridge action (rewrite_core_memory →
  memory_rewrite_core); the error now tells the agent to invoke the
  top-level tool instead of failing silently.
- For everything else, the default error includes a "did you mean"
  suggestion computed via substring match + Levenshtein (threshold 2-3)
  against the known bridge actions. Pure Go, no deps.

Tests cover each alias, the top-level-tool hint, "did you mean"
suggestions for close typos, no suggestion for distant strings, and the
Levenshtein helper.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 07:00:16 +03:00
Algis DumbrisandClaude Opus 4.7 a2ea7cc516 fix(expiry): partial composite index + batched UPDATE to stop "context deadline exceeded"
Root cause
----------
ExpireTasks in internal/channels/task_store.go ran a single unbounded
UPDATE filtered on (status='open' AND deadline IS NOT NULL AND deadline < now).
The only indexes on tasks were idx_tasks_status(status) and
idx_tasks_channel(channel_id). With status cardinality of ~4 and a growing
auction-tasks table on kubic, the planner used idx_tasks_status to enumerate
all open rows then evaluated deadline per row, holding a SQLite write
transaction the whole time. Under WAL contention with concurrent writers
(message inserts, consolidator) the worker's 30s context regularly expired,
producing the recurring expiry-worker log line.

Fix
---
1. New migration 031_tasks_expiry_index.sql: partial composite index
   idx_tasks_expiry(status, deadline) WHERE status='open' AND deadline IS NOT NULL.
   This is the exact predicate ExpireTasks uses, so the planner now seeks
   straight to eligible rows. The partial form keeps the index empty for the
   steady-state majority of rows (completed/cancelled), so writes elsewhere
   aren't penalized.

2. Batch the UPDATE in chunks of 500 (rowid IN subquery; UPDATE ... LIMIT
   isn't compiled into modernc.org/sqlite by default). Bounded write
   transactions stop the worker from starving other writers and let it
   observe context cancellation between batches.

Perf
----
New test exercises 2400 mixed rows (1200 expirable). With the index +
batching, expiry finishes in ~3ms inside a 5s context; without the index a
regression to full status-scan would be measurably worse and is also
guarded by an EXPLAIN QUERY PLAN test.

Operational notes
-----------------
- Migration is additive and idempotent (CREATE INDEX IF NOT EXISTS). No
  backfill needed; it will apply on next pod startup.
- After rollout, expiry-worker error logs should clear within one tick
  (default 1m).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 07:00:16 +03:00
Algis DumbrisandClaude Opus 4.7 0bb2b6500a fix(020): don't burn jobs_started on pre-dispatch failures
Root cause: ConsolidatorWorker.tryDispatch / launchOne incremented
the per-(owner, day) `jobs_started` counter immediately after
JobsStore.Create, BEFORE the dispatch flip succeeded. When the
subsequent steps fail (token issue, agent lookup, dispatch flip
race) the job row is Completed as `failed` but `jobs_started`
remains incremented — and the error paths never call
RecordCompletion, so `jobs_failed` stays flat while `jobs_started`
drifts upward.

Over hours/days, owners whose dispatches fail consistently (e.g.
owner_id=2 in the kubic deployment, hitting one of the harness
failure modes from commit bfb2551) accumulate phantom
`jobs_started` until the default DreamDailyJobLimit=100 trips. From
that point every tick logs `circuit broken … reason=jobs_exceeded`
for all four job types, even though no real jobs ran — and the
counter never decays until midnight UTC.

Fix: move `usage.RecordStart(...)` to AFTER a successful
`jobs.Dispatch(...)` in both tryDispatch (consolidator.go:518)
and launchOne (consolidator.go:455). Now only dispatches that
actually transitioned a row to `dispatched` count against the
daily-job-limit gate.

Test: TestConsolidator_PreDispatchFailureDoesNotBurnJobsStarted
seeds DreamDailyJobLimit=2, makes the agent lookup fail, calls
ForceRun three times, asserts jobs_started stays 0 and the gate
still allows. Verified to fail without the fix
(jobs_started=2 / reason=jobs_exceeded) and pass with it.
Counterpart TestConsolidator_DispatchSuccessIncrementsJobsStarted
asserts jobs_started=1 on a real successful dispatch so the
counter still feeds the gate correctly.

Operational note: this prevents future inflation. Existing stuck
rows for owner_id=2 in today's `memory_dream_usage` bucket need a
one-shot SQL fix —
  UPDATE memory_dream_usage
     SET jobs_started = jobs_succeeded + jobs_failed
   WHERE date = date('now')
     AND owner_id = '2';
or simply wait for the next UTC-midnight reset.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 06:59:28 +03:00
Algis DumbrisandClaude Opus 4.7 205093e654 merge: feature 020 — proactive memory injection + dream worker
Bundles 10 commits implementing owner-scoped memory + a background
dream worker that dispatches consolidation jobs to a Claude Code agent
through the existing harness seam.

Highlights:
- US1: Proactive injection on MCP tool responses (relevant_context
  block, owner-scoped retrieval via existing search.Service hybrid).
- US2: Per-(owner, agent) core memory blob, always included on
  session-start tools.
- US3: ConsolidatorWorker + 6 memory MCP tools + 1 SQL view, with
  dispatch tokens and an audit log. Worker dispatches via
  harness.Harness.Execute → k8sjob backend, NOT via system DMs (per
  saved feedback about cascading stalemate retries).
- Configurable parallelism (SYNAPBUS_DREAM_PARALLEL, default 1) +
  --parallel N CLI flag for backlog drains.
- Daily-token / daily-job UsageGate circuit breaker.
- 14d recency window (configurable; set to 99999d on kubic to process
  all history).
- Grafana dashboard (deploy/kubic/grafana/dream-dashboard.json) and
  hourly watchdog CronJob (deploy/kubic/watchdog/) that auto-stops
  synapbus on runaway-token-drain signal.

Operational evidence from the kubic drain:
- Backlog: 1411 unprocessed → 0, in 50 minutes via 8-parallel waves.
- 260 reflection memories written (ids 32111–32393).
- 5,577 typed links added (5,331 refines + 246 other).
- 4 agent core-memory blobs distilled by the dream agent.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 22:24:53 +03:00
Algis DumbrisandClaude Opus 4.7 73a1802155 feat(020): token accounting + watchdog CronJob
Token-usage wiring (closes the 0-tokens gap in memory_dream_usage):
- internal/harness/k8sjob/k8sjob.go: after extractResultJSON, parse
  tokens_in/tokens_out/tokens_cached/cost_usd from the final JSON
  envelope and stash into ExecResult.Usage so the dream worker's
  UsageGate circuit breaker actually counts consumption.
- dream-agent/dream_runner.py: Max20 OAuth sessions don't surface
  per-call tokens through the SDK's ResultMessage.usage. Falls back
  to a turn-based estimate so the gate has SOME signal:
    tokens_in_est = turns * 5000 + tool_calls * 2000
    tokens_out_est = turns * 300
  Calibrated against observed reflection runs.

Watchdog (deploy/kubic/watchdog/):
- watchdog.yaml: in-cluster CronJob runs every hour at :05 past UTC,
  with a dedicated ServiceAccount + Role granting (get/list/exec on
  pods, patch+update on deployments/scale) inside the synapbus
  namespace only.
- Health checks: pod readiness + restart count; last-1h job
  succ/fail/in_flight counts; today's jobs_started + tokens_in +
  circuit_broken.
- Red flags that auto-stop synapbus (scale to 0):
    * pod restart count > 3
    * failed dream jobs in last 1h > 20
    * jobs_started today > 200 OR tokens_in > 30M
    * circuit broke AND still firing (started >> completed)
- Dockerfile: slim alpine + kubectl v1.30.5 binary (synapbus-watchdog:v1).
  Built locally and imported into kubic's containerd because the
  public docker.io/bitnami/kubectl manifest was returning text/html
  from kubic's network egress.

Replaces the schedule-skill remote-agent approach because Anthropic
cloud agents can't reach kubic.home.arpa (LAN-only) and can't call
kubectl scale. The k8s CronJob is the right primitive for an
in-cluster safety watchdog.

Live evidence: first manual run on kubic reported
  pod=synapbus-... ready=true restarts=0
  last_1h jobs total=18 succ=18 fail=0 in_flight=0
  today: jobs_started=189 tokens_in=0 succeeded=169 failed=15
  HEALTHY — no action

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 22:22:13 +03:00
Algis DumbrisandClaude Opus 4.7 bfb2551b45 feat(020): configurable dream parallelism + 3 bug fixes from kubic drain
Drain-on-demand: SYNAPBUS_DREAM_PARALLEL (default 1) and
`synapbus memory dream-run --parallel N` fan out N concurrent
dream-agent k8s Jobs per (owner, job_type) in one shot. Set
high (e.g. 8) to drain backlog quickly, then back to 1 for normal
hourly operation.

Schema:
- migration 030_dream_parallelism: adds slot INTEGER NOT NULL DEFAULT 0
  to memory_consolidation_jobs. Drops + recreates the partial unique
  in-flight index as (owner, job_type, slot) so slots 0..N-1 each hold
  one in-flight job independently.

Stores:
- JobsStore.CreateOnSlot + CreateNextAvailableSlot.
- ConsolidatorWorker.ForceRunN dispatches N parallel jobs through the
  existing launchOne path (extracted from ForceRun).
- core_rewrite coerces to N=1 regardless of the knob — per-(owner,
  agent) blob is wholesale-replace and concurrent rewrites would race.

Three bug fixes discovered while bringing the parallel path up on
kubic:

1. k8s Job names collided on rapid relaunch because runner.go used
   "synapbus-<agent>-<msg_id>", and dream dispatches have msg_id=0.
   Now appends a unique (timestamp%1e6, 4-byte random) suffix when
   msg_id is zero; historical "synapbus-<agent>-<id>" prefix preserved.

2. memory_list_unprocessed didn't actually exclude already-refined
   messages — the contract said it should, the implementation
   returned the same oldest-50 every cycle. The dream agent kept
   re-refining the same set: 221 refines links touched only 55
   unique dst messages, so progress flat-lined. Added the
   NOT IN (refines/duplicate_of/superseded_by) filter and a
   from_agent NOT LIKE 'dream:%' clause so the agent never refines
   its own reflections.

3. The k8sjob harness was constructed with nil Waiter in main.go,
   so every dream dispatch failed instantly with "k8sjob: no Waiter
   configured". Now builds a ClientsetWaiter from the in-cluster
   clientset.

Plus admin/server.go gets DreamRunN closure + DefaultDreamParallel
(sourced from MemoryConfig.DreamParallel). admin/socket.go
handleMemoryDreamRun accepts `parallel` arg and returns job_ids[].
CLI admin command grows --parallel N flag.

Live evidence from kubic (image v0.21.0-amd64):
  1 CLI call with --parallel 8 produced 8 job rows on slots 0..7,
  spawned 8 distinct k8s Jobs with unique suffixes, retired ~86
  unprocessed messages in <1 min (vs ~10/cycle for the buggy
  serial version pre-fix-2).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 19:53:25 +03:00
Algis DumbrisandClaude Opus 4.7 1d894ec5aa fix(020): wire dispatch-token + claude config bridge
Two short fixes discovered while bringing the dream-claude agent live
on kubic against a Max20 subscription:

1. internal/mcp/server.go: HTTPContextFunc now reads
   X-Synapbus-Dispatch-Token from request headers and stuffs it into
   ctx via WithDispatchToken. The memory_* tools already expected it
   in context; the bridge was missing on the HTTP boundary. Without
   this, every memory_* call returned dispatch_token_missing — which
   is the failure the live dream-agent hit on first run.

2. Followed searcher's proven pattern for Max20 OAuth: the k8sjob
   harness already auto-mounts /home/user/.claude → /app/.claude;
   the agent record just needs CLAUDE_CONFIG_DIR=/app/.claude in
   k8s_env_json. Documented for future agents in the k8s-job
   template, no code change needed here.

Live evidence (kubic, image v0.20.7-amd64):
  Job 2015 reflection: status=succeeded, 3 actions, 2 new reflection
  memories (ids 32111, 32112) on reflections-mcpproxy, each
  synthesizing 14 source memories. End-to-end functional.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 14:16:21 +03:00
Algis DumbrisandClaude Opus 4.7 069a985af5 feat(020): 14d window + token-budget circuit breaker + dream-agent + dashboard
Backend (Go, in this commit):
- migration 029_memory_dream_usage: per (date, owner) counters for
  tokens_in/out, jobs_started/succeeded/failed/circuit_broken
- DreamUsageStore + UsageGate (internal/messaging/dream_usage.go).
  Gate inspects today's usage against new env knobs:
  - SYNAPBUS_DREAM_RECENT_WINDOW (default 336h / 14d)
  - SYNAPBUS_DREAM_DAILY_TOKEN_LIMIT_IN (default 1M)
  - SYNAPBUS_DREAM_DAILY_TOKEN_LIMIT_OUT (default 200k)
  - SYNAPBUS_DREAM_DAILY_JOB_LIMIT (default 100)
- Consolidator now bounds watermarks + core_rewrite eligibility by the
  recency window. core_rewrite skipped for owners with no in-window
  activity. ForceRun honors the breaker.
- Recency fallback in BuildContextPacket + memory_list_unprocessed now
  accept RecentWindowDays so injection and dream queries see the same
  14d slice.
- Prometheus metrics registered (internal/metrics/metrics.go):
  synapbus_dream_jobs_total{owner,job_type,status},
  synapbus_dream_tokens_total{owner,direction},
  synapbus_dream_job_duration_seconds{owner,job_type},
  synapbus_dream_circuit_broken_total{owner,reason},
  synapbus_injection_packets_total{tool},
  synapbus_injection_memories_per_packet{tool},
  synapbus_injection_packet_chars{tool},
  synapbus_injection_skipped_total{tool,reason}.
- deploy/kubic/deployment.yaml: liveness/readiness timeoutSeconds: 1→5
  (root-causes the "connection refused" mcpproxy errors at 13:02 today —
  /readyz occasionally exceeded 1s under dream-worker tick load, so the
  pod fell out of the Service endpoints intermittently).

Dream-claude agent (Python, in /dream-agent/):
- dream_runner.py uses claude-agent-sdk 0.1.48 to drive Claude Code
  against SynapBus's MCP server. MCP transport carries
  Authorization: Bearer <api_key> AND X-Synapbus-Dispatch-Token from env
  via the SDK's McpHttpServerConfig.headers field — confirmed supported.
- Tools restricted via allowed_tools to mcp__synapbus__memory_*.
- Final JSON envelope reports tokens_in/out so harness.Usage stays
  populated and the circuit breaker can count consumption.
- Dockerfile builds linux/amd64 at 189 MB, mirroring searcher's
  agents/universal recipe.
- k8s-job-template.yaml: backoffLimit 0, ttl 600s, 512Mi/1CPU,
  Anthropic credentials via secret-ref.

Grafana dashboard (deploy/kubic/grafana/):
- dream-dashboard.json — 14 panels across 5 rows (dream activity,
  token usage vs limit, circuit breaker, injection layer, MCP
  transport health), all templated to ${DS_PROMETHEUS}.
- import.sh: resolves the cluster's Prometheus DS uid and POSTs the
  dashboard via Grafana API.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 13:58:28 +03:00
Algis DumbrisandClaude Opus 4.7 5a5794beb1 fix(020): inject relevant_context on every eligible tool call
Two bugs discovered while verifying on kubic:

1. HTTP→MCP context propagation lost the full *agents.Agent struct
   (only the bare name was copied via ContextWithAgentName). The
   injection wrapper called agents.AgentFromContext and got nothing,
   silently skipping the packet. Now propagate both: full struct for
   middleware that needs the owner_id, name kept for backward-compat.

2. BuildContextPacket gated retrieval on `query != ""`, so my_status —
   the highest-value injection target — always returned a nil packet.
   FR-009 actually says "use recent owner activity as the implicit
   query" in that case. Added recentMemoriesForOwner: a direct SQL
   query over memory channels filtered by author owner, sorted by id
   DESC. New search_mode "recent" surfaces the fallback path in the
   packet so clients can tell it apart from semantic/fulltext.

Live verification on kubic (image v0.20.3-amd64):
- my_status as `claude-code` now returns relevant_context with 2
  memories from algis-owned agents, packet sized exactly at the
  500-token budget.
- memory_injections audit ring captures each packet (research-mcpproxy
  → search → 1 item; claude-code → my_status → 2 items).
- Cross-owner SC-008 holds: the recency query filters on
  agents.owner_id, so an unrelated owner's agent sees nothing.

Dream worker autonomously fired 31 consolidation jobs (4 types × 2
owners × periodic ticks); all failed at the harness step with
"k8sjob: no Waiter configured" — expected, the claude-code agent has
no k8s_image set. Dispatch chain itself works end-to-end (job row →
token → harness.Execute → audit).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 16:04:10 +03:00
Algis DumbrisandClaude Opus 4.7 2044b199b8 feat(020): US3 — dream worker + 6 MCP consolidation tools
Background ConsolidatorWorker dispatches consolidation work to a Claude
Code agent through harness.Harness.Execute (NOT via system DMs — per
feedback_system_dm_no_trigger.md) with a one-time 15m dispatch token.
The dispatched agent uses six new MCP tools, all token-gated and
recording every action to memory_consolidation_jobs.actions JSON.

Stores:
- memory_links.go (+ test): typed edges with actor-prefix reserved-type
  guard; AddConsolidationLink bypass for memory_mark_duplicate /
  memory_supersede (their contractual writers).
- memory_pins.go (+ test): owner pin overlay, bypasses score floor.
- memory_status.go: queries the memory_status view.
- consolidation_jobs.go: Create / Dispatch / Lease / AppendAction /
  Complete with ErrJobAlreadyInFlight via partial unique index.
- auto_links.go: MessageListener generating mention / reply_to /
  channel_cooccurrence links automatically on send.

Worker:
- consolidator.go (+ test): ticker pattern modeled on StalemateWorker.
  Watermark trigger for link_gen / dedup_contradiction; daily 03:00
  UTC for sleep-time core rewrite. Wallclock budget via harness Budget.
  Global semaphore gates concurrent owners. Mocked-harness test asserts
  no system DM is ever sent.
- consolidator_prompts.go: four job-type prompts passed via env to the
  dispatched agent.

MCP tools (internal/mcp/memory_tools.go + test):
- memory_list_unprocessed, memory_write_reflection, memory_rewrite_core,
  memory_mark_duplicate, memory_supersede, memory_add_link.
- Full error-code matrix tested per contracts/mcp-memory-tools.md.
- Registered only when SYNAPBUS_DREAM_ENABLED=1.

Injection extensions:
- search/injection.go: pin overlay applied after retrieval; status
  filter drops soft_deleted / superseded unless pinned. New
  PinProvider, StatusProvider, MessageLookup hooks on InjectionOpts.

Wiring:
- cmd/synapbus/main.go: stores constructed, AutoLinkListener attached
  to MessagingService, mcpSrv.SetDream wired, ConsolidatorWorker
  start/stop, admin DreamRun closure.
- cmd/synapbus/admin.go: synapbus memory dream-run --owner --job
  socket-RPC command (forces a single job bypassing trigger).

Cycle workarounds (documented in code):
- messaging.DreamAgent / HarnessDispatcher are local interfaces (the
  agents and harness packages import messaging, not the reverse).
  main.go wraps the real types via adapter structs.

Stubbed:
- Cron expression parsing (DreamDeepCron). Hardcoded daily 03:00 UTC.
  Adding robfig/cron deferred to keep no-new-deps.

Pre-existing reactor test failures (5) are unchanged; confirmed
pre-020 via stash check.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 15:40:49 +03:00
Algis DumbrisandClaude Opus 4.7 a52d68ed88 feat(020): US2 — per-agent core memory blob
Letta-style identity blob, one per (owner, agent), always included in
session-start (my_status) responses. Replaces wholesale on rewrite; size
capped at SYNAPBUS_CORE_MEMORY_MAX_BYTES (default 2048); owner-scoped.

Components:
- internal/messaging/memory_core.go (+ test): CoreMemoryStore with
  Get/Set/Delete/List, ErrCoreMemoryTooLarge, NewCoreProvider adapter
  for search.CoreMemoryProvider.
- internal/mcp/server.go SetInjection: wires the core provider into
  the my_status handler wrap.
- internal/mcp/injection_core_test.go: seed → wrapped my_status →
  relevant_context.core_memory matches; missing row → no field.
- internal/api/memory_core.go + router: GET/PUT/DELETE
  /api/owner/{ownerID}/agents/{agentName}/core-memory, session-auth,
  413 on oversize.
- internal/admin/socket.go: memory.core.{get,set,delete} dispatch
  handlers with username→user.id resolution.
- cmd/synapbus/admin.go: `synapbus memory core {get,set,delete}` cobra
  subtree.
- cmd/synapbus/main.go: wires ParseMemoryConfig, CoreMemoryStore,
  MemoryInjections; calls mcpSrv.SetInjection on startup.

Pin overlay still TODO (US3-T029).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 15:18:06 +03:00
Algis DumbrisandClaude Opus 4.7 8a5d5e1f59 feat(020): US1 — proactive injection on MCP tool responses
Wraps eligible MCP tool handlers (my_status, send_message, search,
execute; get_replies excluded as pure metadata) with a middleware that
appends relevant_context to the JSON response. Retrieval reuses the
existing search.Service hybrid pipeline; owner scoping filters out
memories from other owners' agents (SC-008). Pin overlay is a marked
TODO for US3.

Components:
- internal/search/injection.go (+ test): BuildContextPacket with token
  budget greedy fill, score floor, truncation flag, CoreMemoryProvider
  interface stubbed for US2.
- internal/mcp/injection_wrap.go (+ test): WrapInjection middleware,
  registered via SetInjection on the existing handler.
- internal/mcp/injection_e2e_test.go: adversarial cross-owner test
  asserts H1 cannot see H2's memories on any wrapped tool.
- internal/messaging/memory_injections.go (+ test): 24h audit ring,
  hourly cleanup tick wired into stalemate worker.

Discovery during impl: claim_messages/read_inbox/read_channel live as
actions inside the execute bridge, not as registered top-level MCP
tools. They inherit injection through the execute wrapper.

This commit also bundles pre-existing working-tree changes for the
027 "remove approval noise" cleanup (migration 027, design doc,
removal of reminder/escalate logic from stalemate worker, related
trims in goals_tools.go and tools_hybrid.go). The two changes touch
the same files (stalemate.go, tools_hybrid.go) and bundling them
keeps history readable.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 15:09:38 +03:00
Algis DumbrisandClaude Opus 4.7 da827c03b3 feat(020): foundational — migration 028, dispatch tokens, owner resolver
Adds the SQL substrate (6 tables + memory_status view) and the helpers
every user story depends on:
- migration 028_memory_consolidation.sql + smoke test
- internal/messaging/memory_config.go (env-flag plumbing)
- internal/messaging/dispatch_tokens.go (32-byte rand, 15m TTL, single-job-bound)
- internal/messaging/memory_channels.go (open-brain / reflections-* / is_memory flag)
- internal/agents/owner.go (OwnerFor with sentinel errors)

Deviations from spec, all documented in code:
- owner_id is stored as INTEGER FK to users; OwnerFor converts to the
  string scope-key the new tables use.
- MemoryChannel is a local struct to avoid an import cycle between
  internal/channels and internal/messaging.
- channels.metadata column does not exist yet; IsMemoryChannel honors
  it conditionally so MemoryChannelIDs can extend trivially when added.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 14:55:33 +03:00
Algis DumbrisandClaude Opus 4.7 84973ad665 specs(020): proactive memory injection + dream worker
Owner-scoped memory the system pushes onto MCP tool responses, plus a
background "dream" worker that dispatches consolidation jobs to a Claude
Code agent through the existing harness seam. Memory pool reuses the
messages table on memory-flagged channels; six new SQLite tables for
core blob, links, audit, pins, dispatch tokens, and a 24h injection
ring. Zero CGO, no new external deps.

Includes: spec.md (4 user stories), plan.md, research.md (10 decisions),
data-model.md, contracts/ (injection shape + 6 memory tools),
quickstart.md, tasks.md (47 tasks across foundational + 3 stories +
polish, with parallel-subagent cluster plan).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 14:49:38 +03:00
Algis DumbrisandClaude Opus 4.7 a3d8681715 chore(deploy): replace stale helm chart with plain kubic manifests
The helm release went into failed state in March 2026 after an out-of-band
kubectl set image broke server-side-apply ownership; every deploy since has
been a direct kubectl set image, leaving the chart values drifting against
live state.

Drop deploy/helm/ entirely. Add deploy/kubic/{namespace,pvc,service,
deployment,secret.example}.yaml mirroring what's actually running, plus
scripts/deploy-kubic.sh encoding the build → docker save → scp → microk8s
ctr image import → kubectl set image flow used for v0.13.x-reactive through
v0.17.0. README documents why no helm and how to back up /data before
schema-touching versions.

kubectl diff -f deploy/kubic/ is empty against the live cluster.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 12:08:26 +03:00
103 changed files with 13746 additions and 1817 deletions
+50
View File
@@ -0,0 +1,50 @@
# OPC SynapBus 开发规则
本仓库是 GitHub `synapbus/synapbus` 的 OPC fork,托管在 Gitea `https://git.ilapage.cn/OPC/synapbus`。技术栈、目录结构和编码约定以上游 `CLAUDE.md` 为准;本文件只补充 OPC 本地的分支、运行和安全规则。两者冲突时,安全规则以本文件为准。
## 1. 远程与分支
- `origin` = Gitea(OPC 改动推送到这里);`upstream` = GitHub(只拉取,推送地址设为 `DISABLED`,禁止向上游直接推送)。
- `main`:只同步上游,不放 OPC 改动。同步方式:`git fetch upstream` 后把 `upstream/main` 快进推到 `origin/main`。
- `opc/main`:上游 + OPC 补丁,是本机实际编译运行的版本。
- 新功能或修复从 `main` 拉分支(如 `feat/<工单号>-<简述>`),在分支上开发和测试,合入 `opc/main`。这样以后可以把同一分支直接提给上游。
- 本地专用、不打算提给上游的补丁(如 Windows 编译修复)直接提交到 `opc/main`,并在提交信息中注明 `OPC local patch`。
- 同步上游后把 `main` 合入 `opc/main`,重新编译并跑测试,确认本地补丁仍然需要、仍然有效。
### worktree 使用与清理
- 功能分支在独立 worktree 中开发,统一放在 `D:/OPC/synapbus-wt/issue-<N>`;`D:/OPC/synapbus` 是正在运行服务的检出,不在其中切分支开发。
- Windows 编译补丁(renameio)在功能 worktree 中以未提交 + `skip-worktree` 的方式放置,不提交到功能分支。
- 功能分支合入 `opc/main` 并推送后(需要替换运行中二进制的,在替换完成后),由实施该工单的 Agent 删除对应 worktree,并在工单记录。删除前确认:HEAD 已合入 `opc/main`;除 skip-worktree 文件外 `git status --porcelain` 为空;没有进程引用;目录内指向外部的链接先断开。
- 删除前先撤销 skip-worktree 标记并还原 `go.mod`/`go.sum`(`git update-index --no-skip-worktree go.mod go.sum` 后 `git checkout -- go.mod go.sum`),再执行 `git worktree remove <路径>`,不加 `--force`,不删除分支;之后 `git worktree prune`。
- 未合并或有未提交改动的 worktree 不得删除,先向用户报告。
## 2. 工单
- 功能、缺陷修复,以及鉴权、权限、接口、数据库迁移方面的改动,先在 Gitea `OPC/synapbus` 建工单,写清目标、非目标、方案、验收、风险和测试;方案经用户确认后再写代码。
- 涉及鉴权、权限边界、数据库迁移、删除数据的改动属于高风险:实施前和替换运行中的服务前都要等用户确认。
- 提交信息引用工单号,例如 `feat(#1): ...`。
## 3. 本机运行
- 运行命令:`D:/OPC/synapbus/bin/synapbus.exe serve --host 127.0.0.1 --port 8182 --data D:/OPC/synapbus/data`。
- **只监听 127.0.0.1**:不使用 `0.0.0.0` 或局域网地址,不设置 `SYNAPBUS_ALLOW_PRIVATE_NETWORKS=true`;需要对外开放时先建单评估。
- 管理命令要显式指定 socket:`bin/synapbus.exe --socket ./data/synapbus.sock <command>`(Windows 上默认的 `/tmp/synapbus.sock` 连不上)。
- 编译输出到 `bin/`,不要把 `synapbus.exe` 留在仓库根目录。替换 `bin/synapbus.exe` 意味着重启服务,会断开所有 Agent 的连接,需要先告知用户。
- `internal/web/dist/` 下只有 `index.html` 受版本控制,前端资源文件(`_app/`)被忽略。本机对 `index.html` 设置了 `skip-worktree`,不要还原或提交它,否则会和当前构建的资源文件名对不上。
## 4. 数据与密钥
- `data/`(`synapbus.db`、`secrets.key`、`vapid_keys.json`、附件)和 `bin/` 已被 `.gitignore` 忽略,**永远不要提交**,也不要复制到工单、日志或消息里。
- Agent API Key 只保存在 `C:\Users\ila20\synapbus.env`;创建或重新生成 key 时,用脚本把 key 直接写入该文件,不在屏幕或对话中显示。
- 测试使用临时数据目录,不读写正在运行的 `data/`。
## 5. 验证
- 改动后至少运行:`go build ./cmd/synapbus`(输出到 `bin/` 或临时目录)和受影响包的 `go test`;修改 Web 时按上游 `Makefile` 构建前端。
- 鉴权和权限相关改动必须有越权测试,例如某个 Agent 的 key 不能看到其他 Agent 的私信或事件。
- 测试结果必须如实记录;没跑的、跑不了的要写明。
## 6. 许可证
- 上游为 Apache-2.0:保留 `LICENSE`;修改过的文件在文件头或改动处注明 OPC 修改。
+2
View File
@@ -114,6 +114,8 @@ make lint # Run linters
- SQLite via `modernc.org/sqlite` — five new migrations (`021_goals_tasks.sql`, `022_agent_proposals.sql`, `023_agent_trust_model.sql`, `024_secrets.sql`, `025_harness_runs_task_id.sql`); existing content-addressable attachment store reused for encrypted secret blobs (018-dynamic-agent-spawning)
- Go 1.25+ (per go.mod) + `mark3labs/mcp-go` (MCP), `go-chi/chi` (HTTP), `spf13/cobra` (CLI), `modernc.org/sqlite` (storage), `jmoiron/sqlx` (query helpers), `cloudflare/tableflip` (graceful restart — NEW), `gopkg.in/yaml.v3` (config), `xeipuuv/gojsonschema` (config-schema validation) (019-plugin-system)
- SQLite via `modernc.org/sqlite` (pure Go, zero CGO). New core table `plugin_migrations`. Plugin tables namespaced `plugin_<name>_*`. (019-plugin-system)
- Go 1.25+ (per `go.mod`) + `mark3labs/mcp-go` (MCP tools), `go-chi/chi` (HTTP), `modernc.org/sqlite` (storage), `TFMV/hnsw` (vectors via existing `search.Service`), existing `internal/harness` package (dispatch seam). **No new external dependencies.** (020-proactive-memory-dream-worker)
- SQLite via `modernc.org/sqlite` — one new migration `028_memory_consolidation.sql`. Memory pool reuses the existing `messages` table on memory-flagged channels. (020-proactive-memory-dream-worker)
## Recent Changes
- 002-mcp-auth-ux-polish: Added Go 1.23+ + ory/fosite (OAuth 2.1), mark3labs/mcp-go (MCP server), go-chi/chi (HTTP), Svelte 5 + Tailwind (Web UI)
+1 -1
View File
@@ -19,7 +19,7 @@ RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w -X main.version=${VERSION}
# Stage 3: Runtime
FROM alpine:3.19
RUN apk add --no-cache ca-certificates tzdata && touch /.dockerenv
RUN apk add --no-cache ca-certificates tzdata sqlite && touch /.dockerenv
COPY --from=go-builder /synapbus /synapbus
EXPOSE 8080
VOLUME ["/data"]
+131 -5
View File
@@ -702,10 +702,10 @@ func addAdminCommands(rootCmd *cobra.Command) {
channelsJoinCmd.MarkFlagRequired("agent")
var (
channelsUpdateName string
channelsUpdateAutoApprove string
channelsUpdateStalemateRemind string
channelsUpdateStalemateEscalate string
channelsUpdateName string
channelsUpdateAutoApprove string
channelsUpdateStalemateRemind string
channelsUpdateStalemateEscalate string
)
channelsUpdateCmd := &cobra.Command{
Use: "update",
@@ -1341,10 +1341,136 @@ Examples:
harnessConfigCmd.AddCommand(harnessConfigGetCmd, harnessConfigSetCmd, harnessConfigEditCmd)
harnessCmd.AddCommand(harnessConfigCmd)
// ----- memory commands (feature 020 — proactive memory + dream worker) -----
memoryCmd := &cobra.Command{
Use: "memory",
Short: "Manage proactive memory (feature 020)",
}
memoryCoreCmd := &cobra.Command{
Use: "core",
Short: "Manage per-(owner, agent) core memory blobs",
}
var (
memCoreOwner string
memCoreAgent string
memCoreBlob string
memCoreBlobFile string
memCoreUpdater string
)
memoryCoreGetCmd := &cobra.Command{
Use: "get",
Short: "Print the current core memory blob for an (owner, agent)",
RunE: func(cmd *cobra.Command, args []string) error {
resp, err := adminRequest("memory.core.get", map[string]string{
"owner": memCoreOwner,
"agent": memCoreAgent,
})
if err != nil {
return err
}
printJSON(resp["data"])
return nil
},
}
memoryCoreGetCmd.Flags().StringVar(&memCoreOwner, "owner", "", "Owner username or numeric user ID")
memoryCoreGetCmd.Flags().StringVar(&memCoreAgent, "agent", "", "Agent name")
_ = memoryCoreGetCmd.MarkFlagRequired("owner")
_ = memoryCoreGetCmd.MarkFlagRequired("agent")
memoryCoreSetCmd := &cobra.Command{
Use: "set",
Short: "Replace the core memory blob (wholesale, no merge)",
RunE: func(cmd *cobra.Command, args []string) error {
blob := memCoreBlob
if memCoreBlobFile != "" {
data, err := os.ReadFile(memCoreBlobFile)
if err != nil {
return fmt.Errorf("read --blob-file: %w", err)
}
blob = string(data)
}
if blob == "" {
return fmt.Errorf("either --blob or --blob-file is required (non-empty)")
}
resp, err := adminRequest("memory.core.set", map[string]string{
"owner": memCoreOwner,
"agent": memCoreAgent,
"blob": blob,
"updated_by": memCoreUpdater,
})
if err != nil {
return err
}
printJSON(resp["data"])
return nil
},
}
memoryCoreSetCmd.Flags().StringVar(&memCoreOwner, "owner", "", "Owner username or numeric user ID")
memoryCoreSetCmd.Flags().StringVar(&memCoreAgent, "agent", "", "Agent name")
memoryCoreSetCmd.Flags().StringVar(&memCoreBlob, "blob", "", "Core memory blob (inline)")
memoryCoreSetCmd.Flags().StringVar(&memCoreBlobFile, "blob-file", "", "Read blob from file path (overrides --blob)")
memoryCoreSetCmd.Flags().StringVar(&memCoreUpdater, "updated-by", "human", "updated_by audit field (default: human)")
_ = memoryCoreSetCmd.MarkFlagRequired("owner")
_ = memoryCoreSetCmd.MarkFlagRequired("agent")
memoryCoreDeleteCmd := &cobra.Command{
Use: "delete",
Short: "Remove the core memory blob",
RunE: func(cmd *cobra.Command, args []string) error {
resp, err := adminRequest("memory.core.delete", map[string]string{
"owner": memCoreOwner,
"agent": memCoreAgent,
})
if err != nil {
return err
}
printJSON(resp["data"])
return nil
},
}
memoryCoreDeleteCmd.Flags().StringVar(&memCoreOwner, "owner", "", "Owner username or numeric user ID")
memoryCoreDeleteCmd.Flags().StringVar(&memCoreAgent, "agent", "", "Agent name")
_ = memoryCoreDeleteCmd.MarkFlagRequired("owner")
_ = memoryCoreDeleteCmd.MarkFlagRequired("agent")
memoryCoreCmd.AddCommand(memoryCoreGetCmd, memoryCoreSetCmd, memoryCoreDeleteCmd)
memoryCmd.AddCommand(memoryCoreCmd)
// ----- memory dream-run (feature 020 — manual dispatch) -----
var (
dreamRunOwner string
dreamRunJobType string
dreamRunParallel int
)
memoryDreamRunCmd := &cobra.Command{
Use: "dream-run",
Short: "Force a consolidation job dispatch (bypasses trigger checks). With --parallel N spawns N concurrent jobs.",
RunE: func(cmd *cobra.Command, args []string) error {
resp, err := adminRequest("memory.dream_run", map[string]any{
"owner": dreamRunOwner,
"job_type": dreamRunJobType,
"parallel": dreamRunParallel,
})
if err != nil {
return err
}
printJSON(resp["data"])
return nil
},
}
memoryDreamRunCmd.Flags().StringVar(&dreamRunOwner, "owner", "", "Owner username or numeric user ID")
memoryDreamRunCmd.Flags().StringVar(&dreamRunJobType, "job", "reflection", "Job type (reflection | core_rewrite | dedup_contradiction | link_gen)")
memoryDreamRunCmd.Flags().IntVar(&dreamRunParallel, "parallel", 0, "Spawn N concurrent jobs of this type (0 = use server SYNAPBUS_DREAM_PARALLEL; core_rewrite forces 1)")
_ = memoryDreamRunCmd.MarkFlagRequired("owner")
memoryCmd.AddCommand(memoryDreamRunCmd)
// ----- add persistent flag and commands to root -----
rootCmd.PersistentFlags().StringVar(&adminSocket, "socket", "/tmp/synapbus.sock", "Path to admin Unix socket")
rootCmd.AddCommand(userCmd, agentCmd, auditCmd, backupCmd, messagesCmd, channelsCmd, conversationsCmd, embeddingsCmd, dbCmd, retentionCmd, webhookCmd, k8sCmd, attachmentsCmd, harnessCmd)
rootCmd.AddCommand(userCmd, agentCmd, auditCmd, backupCmd, messagesCmd, channelsCmd, conversationsCmd, embeddingsCmd, dbCmd, retentionCmd, webhookCmd, k8sCmd, attachmentsCmd, harnessCmd, memoryCmd)
}
// toTableRows remaps []map[string]string using a header->key mapping.
+224 -50
View File
@@ -26,6 +26,7 @@ import (
"github.com/synapbus/synapbus/internal/a2a"
"github.com/synapbus/synapbus/internal/actions"
"github.com/synapbus/synapbus/internal/admin"
"github.com/synapbus/synapbus/internal/agentquery"
"github.com/synapbus/synapbus/internal/agents"
"github.com/synapbus/synapbus/internal/api"
"github.com/synapbus/synapbus/internal/apikeys"
@@ -34,51 +35,50 @@ import (
"github.com/synapbus/synapbus/internal/auth/idp"
"github.com/synapbus/synapbus/internal/channels"
"github.com/synapbus/synapbus/internal/console"
"github.com/synapbus/synapbus/internal/dispatcher"
"github.com/synapbus/synapbus/internal/goals"
"github.com/synapbus/synapbus/internal/goaltasks"
"github.com/synapbus/synapbus/internal/secrets"
"github.com/synapbus/synapbus/internal/dispatcher"
"github.com/synapbus/synapbus/internal/health"
"github.com/synapbus/synapbus/internal/jsruntime"
k8spkg "github.com/synapbus/synapbus/internal/k8s"
"github.com/synapbus/synapbus/internal/marketplace"
mcpserver "github.com/synapbus/synapbus/internal/mcp"
"github.com/synapbus/synapbus/internal/agentquery"
reactorpkg "github.com/synapbus/synapbus/internal/reactor"
"github.com/synapbus/synapbus/internal/messaging"
prommetrics "github.com/synapbus/synapbus/internal/metrics"
"github.com/synapbus/synapbus/internal/harness"
"github.com/synapbus/synapbus/internal/harness/docker"
"github.com/synapbus/synapbus/internal/harness/k8sjob"
"github.com/synapbus/synapbus/internal/harness/runs"
"github.com/synapbus/synapbus/internal/harness/subprocess"
"github.com/synapbus/synapbus/internal/harness/webhook"
"github.com/synapbus/synapbus/internal/health"
"github.com/synapbus/synapbus/internal/jsruntime"
k8spkg "github.com/synapbus/synapbus/internal/k8s"
"github.com/synapbus/synapbus/internal/marketplace"
mcpserver "github.com/synapbus/synapbus/internal/mcp"
"github.com/synapbus/synapbus/internal/messaging"
prommetrics "github.com/synapbus/synapbus/internal/metrics"
"github.com/synapbus/synapbus/internal/observability"
"github.com/synapbus/synapbus/internal/push"
"github.com/synapbus/synapbus/internal/reactions"
reactorpkg "github.com/synapbus/synapbus/internal/reactor"
"github.com/synapbus/synapbus/internal/search"
"github.com/synapbus/synapbus/internal/search/embedding"
"github.com/synapbus/synapbus/internal/secrets"
"github.com/synapbus/synapbus/internal/storage"
"github.com/synapbus/synapbus/internal/push"
"github.com/synapbus/synapbus/internal/trace"
"github.com/synapbus/synapbus/internal/trust"
"github.com/synapbus/synapbus/internal/web"
"github.com/synapbus/synapbus/internal/wiki"
"github.com/synapbus/synapbus/internal/webhooks"
"github.com/synapbus/synapbus/internal/wiki"
)
// version is set at build time via -ldflags "-X main.version=..."
var version = "dev"
var (
host string
port int
dataDir string
logLevel string
metricsEnabled bool
traceRetention string
adminSocketPath string
webhookWorkers int
messageRetention string
host string
port int
dataDir string
logLevel string
metricsEnabled bool
traceRetention string
adminSocketPath string
webhookWorkers int
messageRetention string
)
func main() {
@@ -305,7 +305,7 @@ func runServe(cmd *cobra.Command, args []string) error {
}
// Create swarm service (task auction + stigmergy)
taskStore := channels.NewSQLiteTaskStore(db.DB)
taskStore := channels.NewSQLiteTaskStore(db.DB).WithReadDB(db.ReadDB)
swarmService := channels.NewSwarmService(taskStore, channelStore, tracer)
// Create attachment service
@@ -512,7 +512,15 @@ func runServe(cmd *cobra.Command, args []string) error {
// going through the existing createJob + poller path; subprocess
// and webhook agents go through Registry.Execute.
harnessRegistry := harness.NewRegistry()
harnessRegistry.Register(k8sjob.New(k8sRunner, nil, slog.Default()))
// Build a ClientsetWaiter when we have a real in-cluster runner so
// the k8sjob backend can actually wait for Job completion. Without
// this the backend errors immediately with "no Waiter configured"
// (the failure mode dream-worker jobs were hitting pre-fix).
var k8sWaiter k8sjob.Waiter
if rr, ok := k8sRunner.(*k8spkg.K8sJobRunner); ok {
k8sWaiter = k8sjob.NewClientsetWaiter(rr.GetClientset(), 0)
}
harnessRegistry.Register(k8sjob.New(k8sRunner, k8sWaiter, slog.Default()))
// SYNAPBUS_KEEP_WORKDIR=1 preserves per-run workdirs after successful
// runs. Useful when debugging MCP tool traces, gemini stdout, or
// materialized config files. Default off to avoid disk growth.
@@ -582,6 +590,49 @@ func runServe(cmd *cobra.Command, args []string) error {
mcpSrv := mcpserver.NewMCPServer(msgService, agentService, channelService, swarmService, attachmentService, searchService, reactionService, trustService, wikiService, con, jsPool, actionRegistry, actionIndex, db.DB)
// Feature 020 — proactive memory injection.
//
// Parse the memory config from env, build the audit-ring store and
// the per-(owner, agent) core memory store, and wire both into the
// MCP hybrid tool surface. When SYNAPBUS_INJECTION_ENABLED=0 (the
// default), SetInjection still runs but WrapInjection returns each
// handler unchanged, so tool responses keep their pre-feature shape
// bit-for-bit (FR-012, SC-009).
memCfg := messaging.ParseMemoryConfig()
memoryInjectionStore := messaging.NewMemoryInjections(db.DB)
coreMemoryStore := messaging.NewCoreMemoryStore(db.DB, memCfg.CoreMemoryMaxBytes)
mcpSrv.SetInjection(memCfg, memoryInjectionStore, messaging.NewCoreProvider(coreMemoryStore))
slog.Info("proactive memory injection wired",
"enabled", memCfg.InjectionEnabled,
"budget_tokens", memCfg.InjectionBudgetTokens,
"core_max_bytes", memCfg.CoreMemoryMaxBytes,
)
// Feature 020 — dream worker (US3) stores. These are always
// constructed so admin CLI / future REST endpoints can read them
// even when SYNAPBUS_DREAM_ENABLED=0. The worker itself starts
// only when the flag is on.
memoryLinkStore := messaging.NewLinkStore(db.DB)
memoryPinStore := messaging.NewPinStore(db.DB)
memoryJobsStore := messaging.NewJobsStore(db.DB)
dispatchTokens := messaging.NewDispatchTokenStore(db.DB)
// Wire the auto-link emitter as a message listener (T035).
msgService.AddMessageListener(messaging.NewAutoLinkListener(db.DB, memoryLinkStore))
// Register the six memory_* MCP tools when SYNAPBUS_DREAM_ENABLED=1.
mcpSrv.SetDream(mcpserver.MemoryToolDeps{
DB: db.DB,
Msg: msgService,
Agents: agentService,
Core: coreMemoryStore,
Links: memoryLinkStore,
Pins: memoryPinStore,
Jobs: memoryJobsStore,
Tokens: dispatchTokens,
MemConfig: memCfg,
})
// Wire the agent marketplace (spec 016 MVP).
marketplaceStore := marketplace.NewStore(db.DB)
marketplaceSvc := marketplace.NewService(marketplaceStore, wikiService, swarmService, channelService, msgService, tracer)
@@ -599,7 +650,7 @@ func runServe(cmd *cobra.Command, args []string) error {
db.DB,
)
mcpSrv.WireGoalsTools(goalsToolReg)
slog.Info("spec-018 MCP tools wired (create_goal, propose_task_tree, propose_agent, claim_task, request_resource, list_resources)")
slog.Info("spec-018 MCP tools wired (create_goal, propose_task_tree, claim_task, request_resource, list_resources, complete_goal)")
// Set up SQL query executor for agents (uses read pool if available)
queryDB := db.QueryDB()
@@ -645,18 +696,51 @@ func runServe(cmd *cobra.Command, args []string) error {
slog.Info("stalemate worker disabled by SYNAPBUS_DISABLE_STALEMATE_WORKER=1")
} else {
stalemateConfig := messaging.ParseStalemateConfig()
stalemateWorker = messaging.NewStalemateWorker(db.DB, msgService, &channelLookupAdapter{channelService: channelService}, stalemateConfig)
stalemateWorker = messaging.NewStalemateWorker(db.DB, msgService, stalemateConfig)
stalemateWorker.SetMemoryInjections(memoryInjectionStore)
stalemateWorker.Start()
slog.Info("stalemate worker started",
"processing_timeout", stalemateConfig.ProcessingTimeout.String(),
"reminder_after", stalemateConfig.ReminderAfter.String(),
"escalate_after", stalemateConfig.EscalateAfter.String(),
"interval", stalemateConfig.Interval.String(),
)
}
// Create health checker
healthChecker := health.NewChecker(db.DB, version)
// Feature 020 — consolidator (dream) worker. Only starts when
// SYNAPBUS_DREAM_ENABLED=1.
var consolidator *messaging.ConsolidatorWorker
if memCfg.DreamEnabled {
consolidator = messaging.NewConsolidatorWorker(
db.DB,
memoryJobsStore,
dispatchTokens,
&harnessDispatcherAdapter{reg: harnessRegistry},
&agentLookupAdapter{svc: agentService},
memCfg,
)
// Wire per-(owner, day) circuit breaker. The gate skips
// dispatch (and records a circuit_broken job) once any of
// SYNAPBUS_DREAM_DAILY_{TOKEN_LIMIT_IN,TOKEN_LIMIT_OUT,JOB_LIMIT}
// is exceeded for the owner.
dreamUsageStore := messaging.NewDreamUsageStore(db.DB)
consolidator.SetUsageGate(dreamUsageStore, messaging.NewUsageGate(memCfg, dreamUsageStore))
consolidator.Start()
slog.Info("consolidator (dream) worker started",
"interval", memCfg.DreamInterval.String(),
"watermark", memCfg.DreamWatermark,
"max_concurrent", memCfg.DreamMaxConcurrent,
"agent", memCfg.DreamAgent,
)
} else {
slog.Info("consolidator (dream) worker disabled (SYNAPBUS_DREAM_ENABLED=0)")
}
// Create health checker. Use the read pool (8 concurrent conns) so /readyz
// can't be starved by a long-running writer holding the serialized write
// connection — most notably the consolidator's dream-job dispatch, which
// does a K8s Job create + DB writes that can run 30s+. With the write pool
// (MaxOpenConns=1) the probe blocked for the entire dispatch, the kubelet
// flipped the pod to not-ready, and the watchdog scaled the deploy to 0.
healthChecker := health.NewChecker(db.QueryDB(), version)
// Set up chi router
r := chi.NewRouter()
@@ -757,6 +841,14 @@ func runServe(cmd *cobra.Command, args []string) error {
// Register broadcaster as a message listener so SSE events fire
// for messages sent via MCP (agents) as well as the REST API.
msgService.AddMessageListener(sseBroadcaster)
sseBroadcaster.SetMessageService(msgService)
// Per-agent SSE stream (agent API key auth, not session auth).
sseHub.SetAgentEventBacklog(msgService)
r.Group(func(r chi.Router) {
r.Use(agents.RequiredAuthMiddlewareWithOAuth(agentService, apiKeyService, oauthProvider))
r.Get("/api/agent-events", sseHub.HandleAgentEvents)
})
// Initialize push notification service
pushStore := push.NewSQLiteStore(db.DB)
@@ -781,6 +873,7 @@ func runServe(cmd *cobra.Command, args []string) error {
Broadcaster: sseBroadcaster,
SessionMiddleware: sessionMiddleware,
DB: db.DB,
ReadDB: db.QueryDB(),
Version: version,
PushService: pushService,
TrustService: trustService,
@@ -791,6 +884,7 @@ func runServe(cmd *cobra.Command, args []string) error {
GoalTasksService: goalTasksService,
BaseURL: baseURL,
WikiService: wikiService,
CoreMemoryStore: coreMemoryStore,
})
r.Mount("/", apiRouter)
@@ -799,13 +893,13 @@ func runServe(cmd *cobra.Command, args []string) error {
// Start admin socket server
adminSvcs := &admin.Services{
Users: userStore,
Sessions: sessionStore,
Agents: agentService,
Messages: msgService,
Channels: channelService,
Traces: traceStore,
DataDir: dataDir,
Users: userStore,
Sessions: sessionStore,
Agents: agentService,
Messages: msgService,
Channels: channelService,
Traces: traceStore,
DataDir: dataDir,
}
// Wire optional services into admin (may be nil if not configured)
if searchCfg.IsEnabled() {
@@ -821,6 +915,19 @@ func runServe(cmd *cobra.Command, args []string) error {
}
adminSvcs.WebhookService = webhookService
adminSvcs.K8sService = k8sService
adminSvcs.CoreMemoryStore = coreMemoryStore
if consolidator != nil {
// Closure form keeps admin's import graph independent of
// messaging.ConsolidatorWorker's full surface.
c := consolidator
adminSvcs.DreamRun = func(ctx context.Context, ownerID, jobType string) (int64, error) {
return c.ForceRun(ctx, ownerID, jobType)
}
adminSvcs.DreamRunN = func(ctx context.Context, ownerID, jobType string, parallel int) ([]int64, error) {
return c.ForceRunN(ctx, ownerID, jobType, parallel)
}
adminSvcs.DefaultDreamParallel = memCfg.DreamParallel
}
adminServer := admin.NewServer(adminSocketPath, db.DB, adminSvcs, logger)
if err := adminServer.Start(); err != nil {
return fmt.Errorf("start admin socket: %w", err)
@@ -886,6 +993,11 @@ func runServe(cmd *cobra.Command, args []string) error {
stalemateWorker.Stop()
}
// Stop consolidator (dream) worker
if consolidator != nil {
consolidator.Stop()
}
// Stop embedding pipeline
if embPipeline != nil {
embPipeline.Stop()
@@ -1158,19 +1270,6 @@ func ensureDefaultMCPClient(ctx context.Context, db *sql.DB, bcryptCost int) {
)
}
// channelLookupAdapter adapts channels.Service to messaging.ChannelLookup.
type channelLookupAdapter struct {
channelService *channels.Service
}
func (a *channelLookupAdapter) GetChannelIDByName(ctx context.Context, name string) (int64, error) {
ch, err := a.channelService.GetChannelByName(ctx, name)
if err != nil {
return 0, err
}
return ch.ID, nil
}
// trustAdjusterAdapter adapts trust.Service to reactions.TrustAdjuster.
type trustAdjusterAdapter struct {
svc *trust.Service
@@ -1211,3 +1310,78 @@ func (a *messageAuthorResolverAdapter) GetMessageAuthor(ctx context.Context, mes
}
return msg.FromAgent, nil
}
// agentLookupAdapter adapts agents.AgentService to
// messaging.AgentLookup so the dream worker can resolve the dream-agent
// record without dragging the full *agents.AgentService into the
// messaging package. The returned messaging.DreamAgent is the raw
// *agents.Agent itself — DreamAgent's only required method
// (AgentName()) is satisfied by agents.Agent.Name via the
// agentNameMethod helper below.
type agentLookupAdapter struct {
svc *agents.AgentService
}
func (a *agentLookupAdapter) GetAgent(ctx context.Context, name string) (messaging.DreamAgent, error) {
ag, err := a.svc.GetAgent(ctx, name)
if err != nil {
return nil, err
}
return agentDreamWrap{ag: ag}, nil
}
// agentDreamWrap adapts *agents.Agent to messaging.DreamAgent.
type agentDreamWrap struct{ ag *agents.Agent }
func (w agentDreamWrap) AgentName() string {
if w.ag == nil {
return ""
}
return w.ag.Name
}
// harnessDispatcherAdapter adapts *harness.Registry to
// messaging.HarnessDispatcher so the consolidator worker can dispatch
// dream-agent runs without importing the harness package (which would
// create an import cycle — harness already imports messaging).
type harnessDispatcherAdapter struct {
reg *harness.Registry
}
func (a *harnessDispatcherAdapter) Execute(
ctx context.Context,
agent messaging.DreamAgent,
req *messaging.HarnessExecRequest,
) (*messaging.HarnessExecResult, error) {
// Unbox the agent record. The worker stores a DreamAgent
// interface; in production it's an agentDreamWrap holding the
// real *agents.Agent. Tests / admin force-runs may pass a bare
// DreamAgentNamed which has no underlying record — the harness
// fallback chain then resolves the backend by name alone.
var realAgent *agents.Agent
if wrap, ok := agent.(agentDreamWrap); ok {
realAgent = wrap.ag
}
hreq := &harness.ExecRequest{
RunID: req.RunID,
AgentName: req.AgentName,
Agent: realAgent,
Env: req.Env,
Budget: harness.Budget{MaxWallClock: req.MaxWallClock},
}
if req.Body != "" {
hreq.Message = &messaging.Message{Body: req.Body}
}
res, err := a.reg.Execute(ctx, realAgent, hreq)
if err != nil {
return nil, err
}
out := &messaging.HarnessExecResult{}
if res != nil {
out.ExitCode = res.ExitCode
out.Logs = res.Logs
out.TokensIn = res.Usage.TokensIn
out.TokensOut = res.Usage.TokensOut
}
return out, nil
}
-6
View File
@@ -1,6 +0,0 @@
apiVersion: v2
name: synapbus
description: Agent-to-agent messaging for AI swarms
type: application
version: 0.1.0
appVersion: "0.1.0"
@@ -1,51 +0,0 @@
{{/*
Expand the name of the chart.
*/}}
{{- define "synapbus.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "synapbus.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "synapbus.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Common labels
*/}}
{{- define "synapbus.labels" -}}
helm.sh/chart: {{ include "synapbus.chart" . }}
{{ include "synapbus.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}
{{/*
Selector labels
*/}}
{{- define "synapbus.selectorLabels" -}}
app.kubernetes.io/name: {{ include "synapbus.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
@@ -1,88 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "synapbus.fullname" . }}
labels:
{{- include "synapbus.labels" . | nindent 4 }}
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
{{- include "synapbus.selectorLabels" . | nindent 6 }}
template:
metadata:
labels:
{{- include "synapbus.selectorLabels" . | nindent 8 }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
containers:
- name: {{ .Chart.Name }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
args:
- serve
- --host
- "0.0.0.0"
- --port
- "8080"
- --data
- /data
ports:
- name: http
containerPort: 8080
protocol: TCP
env:
{{- range $key, $value := .Values.env }}
- name: {{ $key }}
value: {{ $value | quote }}
{{- end }}
{{- with .Values.envFrom }}
envFrom:
{{- toYaml . | nindent 12 }}
{{- end }}
livenessProbe:
httpGet:
path: /healthz
port: http
initialDelaySeconds: 5
periodSeconds: 10
readinessProbe:
httpGet:
path: /readyz
port: http
initialDelaySeconds: 3
periodSeconds: 5
resources:
{{- toYaml .Values.resources | nindent 12 }}
volumeMounts:
- name: data
mountPath: /data
- name: run
mountPath: /tmp
volumes:
- name: data
{{- if .Values.persistence.enabled }}
persistentVolumeClaim:
claimName: {{ include "synapbus.fullname" . }}
{{- else }}
emptyDir: {}
{{- end }}
- name: run
emptyDir:
medium: Memory
sizeLimit: 1Mi
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
@@ -1,41 +0,0 @@
{{- if .Values.ingress.enabled }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ include "synapbus.fullname" . }}
labels:
{{- include "synapbus.labels" . | nindent 4 }}
{{- with .Values.ingress.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
{{- if .Values.ingress.className }}
ingressClassName: {{ .Values.ingress.className }}
{{- end }}
{{- if .Values.ingress.tls }}
tls:
{{- range .Values.ingress.tls }}
- hosts:
{{- range .hosts }}
- {{ . | quote }}
{{- end }}
secretName: {{ .secretName }}
{{- end }}
{{- end }}
rules:
{{- range .Values.ingress.hosts }}
- host: {{ .host | quote }}
http:
paths:
{{- range .paths }}
- path: {{ .path }}
pathType: {{ .pathType }}
backend:
service:
name: {{ include "synapbus.fullname" $ }}
port:
name: http
{{- end }}
{{- end }}
{{- end }}
-17
View File
@@ -1,17 +0,0 @@
{{- if .Values.persistence.enabled }}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: {{ include "synapbus.fullname" . }}
labels:
{{- include "synapbus.labels" . | nindent 4 }}
spec:
accessModes:
{{- toYaml .Values.persistence.accessModes | nindent 4 }}
{{- if .Values.persistence.storageClass }}
storageClassName: {{ .Values.persistence.storageClass | quote }}
{{- end }}
resources:
requests:
storage: {{ .Values.persistence.size }}
{{- end }}
@@ -1,18 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: {{ include "synapbus.fullname" . }}
labels:
{{- include "synapbus.labels" . | nindent 4 }}
spec:
type: {{ .Values.service.type }}
ports:
- port: {{ .Values.service.port }}
targetPort: http
protocol: TCP
name: http
{{- if and (eq .Values.service.type "NodePort") .Values.service.nodePort }}
nodePort: {{ .Values.service.nodePort }}
{{- end }}
selector:
{{- include "synapbus.selectorLabels" . | nindent 4 }}
@@ -1,20 +0,0 @@
{{- if .Values.metrics.enabled }}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: {{ include "synapbus.fullname" . }}
labels:
{{- include "synapbus.labels" . | nindent 4 }}
{{- with .Values.metrics.serviceMonitor.additionalLabels }}
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
selector:
matchLabels:
{{- include "synapbus.selectorLabels" . | nindent 6 }}
endpoints:
- port: http
path: /metrics
interval: {{ .Values.metrics.serviceMonitor.interval }}
scrapeTimeout: {{ .Values.metrics.serviceMonitor.scrapeTimeout }}
{{- end }}
-66
View File
@@ -1,66 +0,0 @@
replicaCount: 1
image:
repository: ghcr.io/synapbus/synapbus
pullPolicy: IfNotPresent
tag: "latest"
imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""
serviceAccount:
create: false
name: ""
service:
type: ClusterIP
port: 8080
ingress:
enabled: false
className: ""
annotations: {}
# kubernetes.io/ingress.class: nginx
# cert-manager.io/cluster-issuer: letsencrypt-prod
hosts:
- host: synapbus.local
paths:
- path: /
pathType: Prefix
tls: []
# - secretName: synapbus-tls
# hosts:
# - synapbus.local
persistence:
enabled: true
storageClass: ""
accessModes:
- ReadWriteOnce
size: 1Gi
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 256Mi
env:
SYNAPBUS_LOG_LEVEL: info
SYNAPBUS_METRICS: "true"
metrics:
enabled: true
serviceMonitor:
interval: 30s
scrapeTimeout: 10s
additionalLabels: {}
nodeSelector: {}
tolerations: []
affinity: {}
+71
View File
@@ -0,0 +1,71 @@
# SynapBus on kubic
Plain Kubernetes manifests for the kubic single-node MicroK8s cluster
(`kubic.home.arpa`). No Helm — the image is built locally, imported directly
into MicroK8s containerd, and rolled with `kubectl set image`.
## Files
| File | Purpose |
|------|---------|
| `namespace.yaml` | `synapbus` namespace |
| `pvc.yaml` | 2 Gi PVC on `microk8s-hostpath` for `/data` (DB + WAL + attachments + HNSW index) |
| `secret.example.yaml` | Template for `synapbus-secrets` (OpenAI/Gemini keys, mounted via `envFrom`) |
| `deployment.yaml` | Single replica, `docker.io/library/synapbus:vX.Y.Z-amd64`, `imagePullPolicy: IfNotPresent` (image is pre-loaded into containerd) |
| `service.yaml` | NodePort 30088 on port 8080 |
| `otel-collector.yaml` | OpenTelemetry collector for traces/metrics |
## Initial install
```sh
kubectl apply -f deploy/kubic/namespace.yaml
kubectl apply -f deploy/kubic/pvc.yaml
# Edit secret.example.yaml first — never commit real keys.
kubectl apply -f deploy/kubic/secret.example.yaml
kubectl apply -f deploy/kubic/service.yaml
kubectl apply -f deploy/kubic/deployment.yaml
```
## Releasing a new version
```sh
scripts/deploy-kubic.sh v0.17.0
```
The script:
1. `docker buildx build --platform linux/amd64` with the version baked in.
2. `docker save` to a tarball.
3. `scp` to `kubic.home.arpa`.
4. `ssh kubic 'sudo microk8s ctr image import …'` (loads the image into the
in-cluster containerd registry — the image is *not* pushed to a remote
registry).
5. `kubectl set image deploy/synapbus synapbus=docker.io/library/synapbus:vX.Y.Z-amd64`.
6. `kubectl rollout status …` and a `/healthz` smoke test.
The `docker.io/library/` prefix is required because that's how containerd
resolves image references that don't specify a registry — `synapbus:v…`
written into the deployment is normalised to `docker.io/library/synapbus:v…`
on the node.
## Why no Helm?
The original chart under `deploy/helm/` (since deleted) was used for the very
first install (Mar 2026) and then went into a `failed` state when someone
ran `kubectl set image` for a hotfix; subsequent `helm upgrade` attempts hit
server-side-apply ownership conflicts. Rather than reconcile, we now own the
manifests directly. The deploy flow is simple enough that templating buys
nothing.
## Backups
Before any version that touches schema, snapshot `/data`:
```sh
kubectl exec -n synapbus deploy/synapbus -- \
tar -C /data -cf - synapbus.db synapbus.db-shm synapbus.db-wal vapid_keys.json \
| tar -xf - -C "$HOME/synapbus-backups/$(date -u +%Y%m%dT%H%M%SZ)/"
```
Then `sqlite3 synapbus.db 'PRAGMA wal_checkpoint(TRUNCATE); PRAGMA integrity_check;'`
to fold the WAL into the main file and verify integrity before archiving.
+77
View File
@@ -0,0 +1,77 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: synapbus
namespace: synapbus
labels:
app.kubernetes.io/name: synapbus
app.kubernetes.io/instance: synapbus
spec:
replicas: 1
revisionHistoryLimit: 10
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
selector:
matchLabels:
app.kubernetes.io/name: synapbus
app.kubernetes.io/instance: synapbus
template:
metadata:
labels:
app.kubernetes.io/name: synapbus
app.kubernetes.io/instance: synapbus
spec:
containers:
- name: synapbus
image: docker.io/library/synapbus:v0.17.0-amd64
imagePullPolicy: IfNotPresent
args: ["serve", "--host", "0.0.0.0", "--port", "8080", "--data", "/data"]
ports:
- name: http
containerPort: 8080
protocol: TCP
env:
- name: SYNAPBUS_BASE_URL
value: auto
- name: SYNAPBUS_EMBEDDING_PROVIDER
value: openai
- name: SYNAPBUS_LOG_LEVEL
value: info
- name: SYNAPBUS_MESSAGE_RETENTION
value: "0"
- name: SYNAPBUS_METRICS
value: "true"
envFrom:
- secretRef:
name: synapbus-secrets
livenessProbe:
httpGet:
path: /healthz
port: http
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /readyz
port: http
initialDelaySeconds: 3
periodSeconds: 5
timeoutSeconds: 5
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: "1"
memory: 512Mi
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: synapbus
+697
View File
@@ -0,0 +1,697 @@
{
"annotations": {
"list": [
{
"name": "Annotations & Alerts",
"datasource": {
"type": "grafana",
"uid": "-- Grafana --"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"type": "dashboard"
},
{
"name": "Circuit breaker trips",
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"enable": true,
"iconColor": "red",
"expr": "changes(synapbus_dream_circuit_broken_total[5m]) > 0",
"step": "60s",
"titleFormat": "Circuit broken: {{reason}}",
"tagKeys": "owner,reason",
"textFormat": "owner={{owner}} reason={{reason}}"
}
]
},
"description": "Visualizes dream worker activity, token budgets, circuit breaker trips, and proactive memory injection emitted by SynapBus feature 020.",
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"id": null,
"links": [
{
"title": "SynapBus Web UI",
"url": "http://kubic.home.arpa:30088",
"type": "link",
"icon": "external link",
"tooltip": "Open SynapBus Web UI",
"targetBlank": true,
"tags": []
}
],
"panels": [
{
"type": "row",
"id": 100,
"title": "Dream worker activity",
"collapsed": false,
"gridPos": {"h": 1, "w": 24, "x": 0, "y": 0},
"panels": []
},
{
"id": 1,
"type": "timeseries",
"title": "Jobs/hour by type",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"gridPos": {"h": 8, "w": 8, "x": 0, "y": 1},
"fieldConfig": {
"defaults": {
"unit": "short",
"custom": {
"drawStyle": "line",
"lineWidth": 2,
"fillOpacity": 10,
"showPoints": "never"
}
},
"overrides": []
},
"options": {
"legend": {"displayMode": "table", "placement": "bottom", "showLegend": true, "calcs": ["mean", "max"]},
"tooltip": {"mode": "multi", "sort": "desc"}
},
"targets": [
{
"refId": "A",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"expr": "sum by (job_type) (rate(synapbus_dream_jobs_total{owner=~\"$owner\",job_type=~\"$job_type\"}[5m]) * 3600)",
"legendFormat": "{{job_type}}"
}
]
},
{
"id": 2,
"type": "timeseries",
"title": "Jobs/hour by status (stacked)",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"gridPos": {"h": 8, "w": 8, "x": 8, "y": 1},
"fieldConfig": {
"defaults": {
"unit": "short",
"custom": {
"drawStyle": "line",
"lineWidth": 1,
"fillOpacity": 60,
"stacking": {"mode": "normal", "group": "A"},
"showPoints": "never"
},
"color": {"mode": "palette-classic"}
},
"overrides": [
{
"matcher": {"id": "byName", "options": "succeeded"},
"properties": [{"id": "color", "value": {"mode": "fixed", "fixedColor": "green"}}]
},
{
"matcher": {"id": "byName", "options": "failed"},
"properties": [{"id": "color", "value": {"mode": "fixed", "fixedColor": "red"}}]
},
{
"matcher": {"id": "byName", "options": "circuit_broken"},
"properties": [{"id": "color", "value": {"mode": "fixed", "fixedColor": "orange"}}]
}
]
},
"options": {
"legend": {"displayMode": "table", "placement": "bottom", "showLegend": true, "calcs": ["sum"]},
"tooltip": {"mode": "multi", "sort": "desc"}
},
"targets": [
{
"refId": "A",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"expr": "sum by (status) (rate(synapbus_dream_jobs_total{owner=~\"$owner\",job_type=~\"$job_type\"}[5m]) * 3600)",
"legendFormat": "{{status}}"
}
]
},
{
"id": 3,
"type": "timeseries",
"title": "Job duration p50 / p95 (s)",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"gridPos": {"h": 8, "w": 8, "x": 16, "y": 1},
"fieldConfig": {
"defaults": {
"unit": "s",
"custom": {
"drawStyle": "line",
"lineWidth": 2,
"fillOpacity": 5,
"showPoints": "never"
}
},
"overrides": []
},
"options": {
"legend": {"displayMode": "table", "placement": "bottom", "showLegend": true, "calcs": ["mean", "max"]},
"tooltip": {"mode": "multi", "sort": "desc"}
},
"targets": [
{
"refId": "A",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"expr": "histogram_quantile(0.5, sum by (job_type, le) (rate(synapbus_dream_job_duration_seconds_bucket{owner=~\"$owner\",job_type=~\"$job_type\"}[5m])))",
"legendFormat": "p50 {{job_type}}"
},
{
"refId": "B",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"expr": "histogram_quantile(0.95, sum by (job_type, le) (rate(synapbus_dream_job_duration_seconds_bucket{owner=~\"$owner\",job_type=~\"$job_type\"}[5m])))",
"legendFormat": "p95 {{job_type}}"
}
]
},
{
"type": "row",
"id": 101,
"title": "Token usage vs limit",
"collapsed": false,
"gridPos": {"h": 1, "w": 24, "x": 0, "y": 9},
"panels": []
},
{
"id": 4,
"type": "stat",
"title": "Daily tokens IN by owner (24h)",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"gridPos": {"h": 7, "w": 8, "x": 0, "y": 10},
"fieldConfig": {
"defaults": {
"unit": "short",
"decimals": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{"color": "green", "value": null},
{"color": "yellow", "value": 700000},
{"color": "red", "value": 1000000}
]
}
},
"overrides": []
},
"options": {
"reduceOptions": {"values": false, "calcs": ["lastNotNull"], "fields": ""},
"orientation": "auto",
"textMode": "value_and_name",
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"showPercentChange": false
},
"targets": [
{
"refId": "A",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"expr": "sum by (owner) (increase(synapbus_dream_tokens_total{direction=\"in\",owner=~\"$owner\"}[24h]))",
"legendFormat": "{{owner}}"
}
]
},
{
"id": 5,
"type": "stat",
"title": "Daily tokens OUT by owner (24h)",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"gridPos": {"h": 7, "w": 8, "x": 8, "y": 10},
"fieldConfig": {
"defaults": {
"unit": "short",
"decimals": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{"color": "green", "value": null},
{"color": "yellow", "value": 140000},
{"color": "red", "value": 200000}
]
}
},
"overrides": []
},
"options": {
"reduceOptions": {"values": false, "calcs": ["lastNotNull"], "fields": ""},
"orientation": "auto",
"textMode": "value_and_name",
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"showPercentChange": false
},
"targets": [
{
"refId": "A",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"expr": "sum by (owner) (increase(synapbus_dream_tokens_total{direction=\"out\",owner=~\"$owner\"}[24h]))",
"legendFormat": "{{owner}}"
}
]
},
{
"id": 6,
"type": "timeseries",
"title": "Token usage (15-min windows)",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"gridPos": {"h": 7, "w": 8, "x": 16, "y": 10},
"fieldConfig": {
"defaults": {
"unit": "short",
"custom": {
"drawStyle": "line",
"lineWidth": 2,
"fillOpacity": 10,
"showPoints": "never"
}
},
"overrides": []
},
"options": {
"legend": {"displayMode": "table", "placement": "bottom", "showLegend": true, "calcs": ["mean", "max"]},
"tooltip": {"mode": "multi", "sort": "desc"}
},
"targets": [
{
"refId": "A",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"expr": "sum by (owner, direction) (rate(synapbus_dream_tokens_total{owner=~\"$owner\"}[15m]) * 900)",
"legendFormat": "{{owner}} / {{direction}}"
}
]
},
{
"type": "row",
"id": 102,
"title": "Circuit breaker",
"collapsed": false,
"gridPos": {"h": 1, "w": 24, "x": 0, "y": 17},
"panels": []
},
{
"id": 7,
"type": "stat",
"title": "Circuit-breaker trips (24h)",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"gridPos": {"h": 7, "w": 8, "x": 0, "y": 18},
"fieldConfig": {
"defaults": {
"unit": "short",
"decimals": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{"color": "green", "value": null},
{"color": "orange", "value": 1},
{"color": "red", "value": 5}
]
}
},
"overrides": []
},
"options": {
"reduceOptions": {"values": false, "calcs": ["lastNotNull"], "fields": ""},
"orientation": "auto",
"textMode": "value_and_name",
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"showPercentChange": false
},
"targets": [
{
"refId": "A",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"expr": "sum by (reason) (increase(synapbus_dream_circuit_broken_total{owner=~\"$owner\"}[24h]))",
"legendFormat": "{{reason}}"
}
]
},
{
"id": 8,
"type": "state-timeline",
"title": "Circuit-breaker events timeline",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"gridPos": {"h": 7, "w": 16, "x": 8, "y": 18},
"fieldConfig": {
"defaults": {
"custom": {
"lineWidth": 0,
"fillOpacity": 70
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{"color": "green", "value": null},
{"color": "red", "value": 1}
]
}
},
"overrides": []
},
"options": {
"mergeValues": true,
"showValue": "auto",
"alignValue": "left",
"rowHeight": 0.9,
"legend": {"displayMode": "list", "placement": "bottom", "showLegend": true},
"tooltip": {"mode": "single", "sort": "none"}
},
"targets": [
{
"refId": "A",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"expr": "sum by (owner, reason) (rate(synapbus_dream_circuit_broken_total{owner=~\"$owner\"}[5m])) > 0",
"legendFormat": "{{owner}} / {{reason}}"
}
]
},
{
"type": "row",
"id": 103,
"title": "Injection layer",
"collapsed": false,
"gridPos": {"h": 1, "w": 24, "x": 0, "y": 25},
"panels": []
},
{
"id": 9,
"type": "timeseries",
"title": "Injection packets/hr by tool (stacked)",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 26},
"fieldConfig": {
"defaults": {
"unit": "short",
"custom": {
"drawStyle": "line",
"lineWidth": 1,
"fillOpacity": 60,
"stacking": {"mode": "normal", "group": "A"},
"showPoints": "never"
},
"color": {"mode": "palette-classic"}
},
"overrides": []
},
"options": {
"legend": {"displayMode": "table", "placement": "bottom", "showLegend": true, "calcs": ["mean", "sum"]},
"tooltip": {"mode": "multi", "sort": "desc"}
},
"targets": [
{
"refId": "A",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"expr": "sum by (tool) (rate(synapbus_injection_packets_total{tool=~\"$tool\"}[5m]) * 3600)",
"legendFormat": "{{tool}}"
}
]
},
{
"id": 10,
"type": "timeseries",
"title": "Memories per packet (p50 / p95)",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 26},
"fieldConfig": {
"defaults": {
"unit": "short",
"custom": {
"drawStyle": "line",
"lineWidth": 2,
"fillOpacity": 5,
"showPoints": "never"
}
},
"overrides": []
},
"options": {
"legend": {"displayMode": "table", "placement": "bottom", "showLegend": true, "calcs": ["mean", "max"]},
"tooltip": {"mode": "multi", "sort": "desc"}
},
"targets": [
{
"refId": "A",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"expr": "histogram_quantile(0.5, sum by (tool, le) (rate(synapbus_injection_memories_per_packet_bucket{tool=~\"$tool\"}[5m])))",
"legendFormat": "p50 {{tool}}"
},
{
"refId": "B",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"expr": "histogram_quantile(0.95, sum by (tool, le) (rate(synapbus_injection_memories_per_packet_bucket{tool=~\"$tool\"}[5m])))",
"legendFormat": "p95 {{tool}}"
}
]
},
{
"id": 11,
"type": "timeseries",
"title": "Packet size (chars) p95",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 34},
"fieldConfig": {
"defaults": {
"unit": "short",
"custom": {
"drawStyle": "line",
"lineWidth": 2,
"fillOpacity": 10,
"showPoints": "never"
}
},
"overrides": []
},
"options": {
"legend": {"displayMode": "table", "placement": "bottom", "showLegend": true, "calcs": ["mean", "max"]},
"tooltip": {"mode": "multi", "sort": "desc"}
},
"targets": [
{
"refId": "A",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"expr": "histogram_quantile(0.95, sum by (tool, le) (rate(synapbus_injection_packet_chars_bucket{tool=~\"$tool\"}[5m])))",
"legendFormat": "p95 {{tool}}"
},
{
"refId": "B",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"expr": "histogram_quantile(0.5, sum by (tool, le) (rate(synapbus_injection_packet_chars_bucket{tool=~\"$tool\"}[5m])))",
"legendFormat": "p50 {{tool}}"
}
]
},
{
"id": 12,
"type": "table",
"title": "Injection skipped reasons (24h)",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 34},
"fieldConfig": {
"defaults": {
"custom": {
"align": "auto",
"displayMode": "auto",
"inspect": false
},
"thresholds": {
"mode": "absolute",
"steps": [
{"color": "green", "value": null}
]
}
},
"overrides": [
{
"matcher": {"id": "byName", "options": "Value"},
"properties": [
{"id": "custom.displayMode", "value": "gradient-gauge"},
{"id": "custom.align", "value": "right"},
{"id": "displayName", "value": "skipped (24h)"}
]
}
]
},
"options": {
"showHeader": true,
"sortBy": [{"displayName": "skipped (24h)", "desc": true}]
},
"transformations": [
{
"id": "organize",
"options": {
"excludeByName": {"Time": true, "__name__": true, "job": true, "instance": true},
"indexByName": {},
"renameByName": {}
}
}
],
"targets": [
{
"refId": "A",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"expr": "sum by (tool, reason) (increase(synapbus_injection_skipped_total{tool=~\"$tool\"}[24h]))",
"format": "table",
"instant": true
}
]
},
{
"type": "row",
"id": 104,
"title": "MCP transport health",
"collapsed": false,
"gridPos": {"h": 1, "w": 24, "x": 0, "y": 42},
"panels": []
},
{
"id": 13,
"type": "timeseries",
"title": "MCP request rate (req/s)",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 43},
"fieldConfig": {
"defaults": {
"unit": "reqps",
"custom": {
"drawStyle": "line",
"lineWidth": 2,
"fillOpacity": 10,
"showPoints": "never"
}
},
"overrides": []
},
"options": {
"legend": {"displayMode": "table", "placement": "bottom", "showLegend": true, "calcs": ["mean", "max"]},
"tooltip": {"mode": "multi", "sort": "desc"}
},
"targets": [
{
"refId": "A",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"expr": "sum by (path,status) (rate(synapbus_http_requests_total{path=~\".*mcp.*\"}[5m]))",
"legendFormat": "{{path}} {{status}}"
}
]
},
{
"id": 14,
"type": "timeseries",
"title": "MCP latency p95 (s)",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 43},
"fieldConfig": {
"defaults": {
"unit": "s",
"custom": {
"drawStyle": "line",
"lineWidth": 2,
"fillOpacity": 5,
"showPoints": "never"
}
},
"overrides": []
},
"options": {
"legend": {"displayMode": "table", "placement": "bottom", "showLegend": true, "calcs": ["mean", "max"]},
"tooltip": {"mode": "multi", "sort": "desc"}
},
"targets": [
{
"refId": "A",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"expr": "histogram_quantile(0.95, sum by (le) (rate(synapbus_http_request_duration_seconds_bucket{path=~\".*mcp.*\"}[5m])))",
"legendFormat": "p95"
}
]
}
],
"refresh": "30s",
"schemaVersion": 39,
"tags": ["synapbus", "dream", "memory", "feature-020"],
"templating": {
"list": [
{
"name": "DS_PROMETHEUS",
"label": "Prometheus",
"type": "datasource",
"query": "prometheus",
"refresh": 1,
"current": {},
"hide": 0,
"includeAll": false,
"multi": false,
"options": [],
"regex": "",
"skipUrlSync": false
},
{
"name": "owner",
"label": "Owner",
"type": "query",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"definition": "label_values(synapbus_dream_jobs_total, owner)",
"query": {"query": "label_values(synapbus_dream_jobs_total, owner)", "refId": "StandardVariableQuery"},
"refresh": 2,
"regex": "",
"sort": 1,
"multi": true,
"includeAll": true,
"allValue": ".*",
"current": {"selected": true, "text": ["All"], "value": ["$__all"]},
"options": [],
"hide": 0,
"skipUrlSync": false
},
{
"name": "job_type",
"label": "Job type",
"type": "query",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"definition": "label_values(synapbus_dream_jobs_total, job_type)",
"query": {"query": "label_values(synapbus_dream_jobs_total, job_type)", "refId": "StandardVariableQuery"},
"refresh": 2,
"regex": "",
"sort": 1,
"multi": true,
"includeAll": true,
"allValue": ".*",
"current": {"selected": true, "text": ["All"], "value": ["$__all"]},
"options": [],
"hide": 0,
"skipUrlSync": false
},
{
"name": "tool",
"label": "Tool",
"type": "query",
"datasource": {"type": "prometheus", "uid": "${DS_PROMETHEUS}"},
"definition": "label_values(synapbus_injection_packets_total, tool)",
"query": {"query": "label_values(synapbus_injection_packets_total, tool)", "refId": "StandardVariableQuery"},
"refresh": 2,
"regex": "",
"sort": 1,
"multi": true,
"includeAll": true,
"allValue": ".*",
"current": {"selected": true, "text": ["All"], "value": ["$__all"]},
"options": [],
"hide": 0,
"skipUrlSync": false
}
]
},
"time": {"from": "now-6h", "to": "now"},
"timepicker": {},
"timezone": "",
"title": "SynapBus — Dream Worker & Memory Injection",
"uid": "synapbus-dream-memory-020",
"version": 1,
"weekStart": ""
}
+32
View File
@@ -0,0 +1,32 @@
#!/bin/bash
# Imports the SynapBus dream worker dashboard into Grafana.
# Usage:
# GRAFANA_PASS=... ./import.sh
# GRAFANA_URL=http://grafana.example:3000 GRAFANA_USER=admin GRAFANA_PASS=... ./import.sh
set -euo pipefail
GRAFANA_URL="${GRAFANA_URL:-http://kubic.home.arpa:30083}"
GRAFANA_USER="${GRAFANA_USER:-admin}"
GRAFANA_PASS="${GRAFANA_PASS:?need GRAFANA_PASS}"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
DASH_FILE="${SCRIPT_DIR}/dream-dashboard.json"
[ -f "$DASH_FILE" ] || { echo "dashboard JSON not found: $DASH_FILE" >&2; exit 1; }
DS_UID=$(curl -fsS -u "$GRAFANA_USER:$GRAFANA_PASS" "$GRAFANA_URL/api/datasources" \
| jq -r '.[] | select(.type=="prometheus") | .uid' | head -1)
[ -z "$DS_UID" ] && { echo "no prometheus datasource found in $GRAFANA_URL" >&2; exit 1; }
echo "Using Prometheus DS uid=$DS_UID" >&2
DASHBOARD=$(jq --arg uid "$DS_UID" '
(.. | objects | select(.type? == "prometheus") | .uid) |= $uid
| .id = null
| . as $dash | { dashboard: $dash, overwrite: true, message: "feat(020): dream worker + memory injection dashboard" }
' "$DASH_FILE")
curl -fsS -u "$GRAFANA_USER:$GRAFANA_PASS" \
-H "Content-Type: application/json" \
-X POST "$GRAFANA_URL/api/dashboards/db" \
-d "$DASHBOARD"
echo
+4
View File
@@ -0,0 +1,4 @@
apiVersion: v1
kind: Namespace
metadata:
name: synapbus
+15
View File
@@ -0,0 +1,15 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: synapbus
namespace: synapbus
labels:
app.kubernetes.io/name: synapbus
app.kubernetes.io/instance: synapbus
spec:
accessModes:
- ReadWriteOnce
storageClassName: microk8s-hostpath
resources:
requests:
storage: 2Gi
+9
View File
@@ -0,0 +1,9 @@
apiVersion: v1
kind: Secret
metadata:
name: synapbus-secrets
namespace: synapbus
type: Opaque
stringData:
OPENAI_API_KEY: "sk-..."
GEMINI_API_KEY: ""
+19
View File
@@ -0,0 +1,19 @@
apiVersion: v1
kind: Service
metadata:
name: synapbus
namespace: synapbus
labels:
app.kubernetes.io/name: synapbus
app.kubernetes.io/instance: synapbus
spec:
type: NodePort
selector:
app.kubernetes.io/name: synapbus
app.kubernetes.io/instance: synapbus
ports:
- name: http
port: 8080
targetPort: http
nodePort: 30088
protocol: TCP
+8
View File
@@ -0,0 +1,8 @@
FROM alpine:3.21
RUN apk add --no-cache bash curl ca-certificates
RUN curl -fsSL -o /usr/local/bin/kubectl \
https://dl.k8s.io/release/v1.30.5/bin/linux/amd64/kubectl \
&& chmod +x /usr/local/bin/kubectl \
&& kubectl version --client
WORKDIR /scripts
ENTRYPOINT ["/bin/bash"]
+172
View File
@@ -0,0 +1,172 @@
# synapbus-watchdog: hourly k8s CronJob that checks dream-worker health
# and scales synapbus/synapbus to 0 replicas if any red-flag trips.
# Goal: prevent runaway Claude Code token drain while Algis is AFK.
#
# Cadence: every hour at :05 past (covers the requested +2h and +4h
# horizons and keeps catching problems indefinitely until disabled).
#
# Disable with:
# microk8s kubectl -n synapbus patch cronjob synapbus-watchdog \
# -p '{"spec":{"suspend":true}}'
#
# Stop manually:
# microk8s kubectl -n synapbus delete cronjob synapbus-watchdog
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: synapbus-watchdog
namespace: synapbus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: synapbus-watchdog
namespace: synapbus
rules:
- apiGroups: [""]
resources: ["pods", "pods/exec"]
verbs: ["get", "list", "create"]
- apiGroups: ["apps"]
resources: ["deployments", "deployments/scale"]
verbs: ["get", "patch", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: synapbus-watchdog
namespace: synapbus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: synapbus-watchdog
subjects:
- kind: ServiceAccount
name: synapbus-watchdog
namespace: synapbus
---
apiVersion: v1
kind: ConfigMap
metadata:
name: synapbus-watchdog-script
namespace: synapbus
data:
watchdog.sh: |
#!/bin/bash
set -uo pipefail
NS=synapbus
DEPLOY=synapbus
LOG_PREFIX="[watchdog $(date -u +%FT%TZ)]"
log() { echo "$LOG_PREFIX $*"; }
fail() { log "RED-FLAG: $*"; STOP=1; STOP_REASON="$*"; }
STOP=0
STOP_REASON=""
# 1) Pod state
POD=$(kubectl -n $NS get pod -l app.kubernetes.io/name=synapbus \
-o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
if [ -z "$POD" ]; then fail "no synapbus pod"; else
READY=$(kubectl -n $NS get pod "$POD" \
-o jsonpath='{.status.containerStatuses[?(@.name=="synapbus")].ready}')
RESTARTS=$(kubectl -n $NS get pod "$POD" \
-o jsonpath='{.status.containerStatuses[?(@.name=="synapbus")].restartCount}')
log "pod=$POD ready=$READY restarts=$RESTARTS"
[ "$READY" = "true" ] || fail "pod not ready"
[ "${RESTARTS:-0}" -le 3 ] || fail "restart count $RESTARTS > 3"
fi
# 2) Dream-job hourly aggregate
if [ -n "$POD" ]; then
ROW=$(kubectl -n $NS exec "$POD" -- sqlite3 /data/synapbus.db \
"SELECT COALESCE(SUM(CASE WHEN status='succeeded' THEN 1 ELSE 0 END),0), \
COALESCE(SUM(CASE WHEN status='failed' THEN 1 ELSE 0 END),0), \
COALESCE(SUM(CASE WHEN status IN ('running','dispatched','pending') THEN 1 ELSE 0 END),0), \
COALESCE(COUNT(*),0) \
FROM memory_consolidation_jobs \
WHERE created_at > datetime('now','-1 hour');" 2>/dev/null \
| tr '|' ' ')
SUCC=$(echo "$ROW" | awk '{print $1}')
FAIL=$(echo "$ROW" | awk '{print $2}')
INFL=$(echo "$ROW" | awk '{print $3}')
TOTAL=$(echo "$ROW" | awk '{print $4}')
log "last_1h jobs total=$TOTAL succ=$SUCC fail=$FAIL in_flight=$INFL"
[ "${FAIL:-0}" -le 20 ] || fail "failed jobs in last 1h = $FAIL > 20"
fi
# 3) Today's usage — aggregate across all owners (caps are global,
# not per-owner; owner_id is just a partition key in the table)
if [ -n "$POD" ]; then
U=$(kubectl -n $NS exec "$POD" -- sqlite3 /data/synapbus.db \
"SELECT COALESCE(SUM(jobs_started),0), COALESCE(SUM(tokens_in),0), \
COALESCE(SUM(jobs_succeeded),0), COALESCE(SUM(jobs_failed),0), \
COALESCE(SUM(jobs_circuit_broken),0) \
FROM memory_dream_usage WHERE date=date('now');" 2>/dev/null \
| tr '|' ' ')
JS=$(echo "$U" | awk '{print $1}'); JS=${JS:-0}
TIN=$(echo "$U" | awk '{print $2}'); TIN=${TIN:-0}
JOK=$(echo "$U" | awk '{print $3}'); JOK=${JOK:-0}
JFL=$(echo "$U" | awk '{print $4}'); JFL=${JFL:-0}
JCB=$(echo "$U" | awk '{print $5}'); JCB=${JCB:-0}
log "today: jobs_started=$JS tokens_in=$TIN succeeded=$JOK failed=$JFL circuit_broken=$JCB"
[ "$JS" -le 200 ] || fail "jobs_started today $JS > 200 (soft cap)"
[ "$TIN" -le 30000000 ] || fail "tokens_in today $TIN > 30M (budget cliff)"
# "still firing despite breaker": more started than completed by >5
DELTA=$((JS - JOK - JFL - JCB))
if [ "$JCB" -gt 0 ] && [ "$DELTA" -gt 5 ]; then
fail "circuit broke but still firing (started=$JS, completed_or_broken=$((JOK+JFL+JCB)), delta=$DELTA)"
fi
fi
# Act
if [ "$STOP" = "1" ]; then
log "STOPPING synapbus: $STOP_REASON"
kubectl -n $NS scale deploy/$DEPLOY --replicas=0
log "synapbus scaled to 0 replicas. Re-enable with: kubectl -n $NS scale deploy/$DEPLOY --replicas=1"
exit 2
fi
log "HEALTHY — no action"
exit 0
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: synapbus-watchdog
namespace: synapbus
spec:
schedule: "5 * * * *" # every hour at :05 past (UTC)
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 6
failedJobsHistoryLimit: 6
startingDeadlineSeconds: 600
jobTemplate:
spec:
backoffLimit: 0
ttlSecondsAfterFinished: 86400
activeDeadlineSeconds: 180
template:
spec:
serviceAccountName: synapbus-watchdog
restartPolicy: Never
containers:
- name: watchdog
image: docker.io/library/synapbus-watchdog:v1
imagePullPolicy: Never
command: ["/bin/bash", "/scripts/watchdog.sh"]
volumeMounts:
- name: script
mountPath: /scripts
readOnly: true
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 200m
memory: 128Mi
volumes:
- name: script
configMap:
name: synapbus-watchdog-script
defaultMode: 0755
@@ -0,0 +1,102 @@
# Internal-only mode: remove approvals & escalations
**Date:** 2026-05-10
**Status:** Design
**Owner:** Algis
## Problem
SynapBus today assumes a human is in the loop: a stalemate worker DMs reminders after 4h, escalates to `#approvals` after 48h, and the dynamic-agent-spawning flow (spec 018) gates new agents and task trees on human approval. In practice the user is the only operator, the approval queue stalls, and the volume of reminder/escalation messages drowns out signal. The user is moving to a single daily summary (separate `#summary-daily` channel + summarizer agent already in progress) and treats SynapBus as an internal-only comms + data store — nothing publishes externally.
The goal is to remove the human-in-the-loop surfaces so the message stream stops generating noise the user will never read.
## Scope
### Removed
1. **Stalemate reminders** — `StalemateWorker.sendPendingReminders` and supporting helpers (`reminderExists`, the 4h ReminderAfter knob).
2. **Stalemate escalations** — `StalemateWorker.escalatePendingMessages` and `checkWorkflowStalemates`, plus the 48h EscalateAfter knob and `#approvals` lookup path.
3. **`propose_agent` MCP tool** (spec 018). It writes a `pending` row to `agent_proposals` for human approval via `#approvals` and there is no automated consumer of that table. Removing the tool leaves agent creation to the admin CLI, which matches the internal-only stance.
**Note:** `propose_task_tree` is intentionally KEPT despite its name — it is not an approval gate. It directly inserts tasks in `approved` status and auto-transitions the goal to `active`. Removing it would break the spec-018 goal/task flow.
4. *(Reactions service intentionally untouched — it's a generic workflow primitive that also drives trust adjustments. Once no upstream feature creates approval-bearing messages, the `approve` / `reject` reaction paths become dormant on their own.)*
### Kept
- **`StalemateWorker.ProcessingTimeout`** (24h auto-fail of claimed-but-abandoned messages). Protects the inbox from crashed agents; not human-facing.
- **The `#approvals` channel row** in the `channels` table. Cheaper to leave than to migrate; user can drop via admin CLI later.
- **Webhook / K8s runner approval gates** (spec 003). User confirmed these are out of scope.
- **Trust system** (spec 011). No approval surface, just delegation.
### One-shot DB cleanup
New migration `internal/storage/schema/027_remove_approval_noise.sql`:
```sql
-- Drop reminder and escalation system DMs.
DELETE FROM messages
WHERE subject LIKE 'stalemate-reminder:%'
OR subject LIKE 'stalemate-escalation:%';
-- Drop everything in the #approvals channel.
DELETE FROM messages
WHERE channel_id = (SELECT id FROM channels WHERE name = 'approvals');
-- Drop pending agent proposals (table itself stays for reversibility).
DELETE FROM agent_proposals;
```
`VACUUM` cannot run inside a migration transaction, so reclaiming disk is a separate `synapbus admin vacuum` command (or a manual `kubectl exec ... sqlite3 ... 'VACUUM;'`). Out of scope for this change unless trivial to wire up.
## Architecture impact
```
Before:
agent → MCP propose_agent → agent_proposals row → human reacts in #approvals
→ spawn or reject
message claimed → StalemateWorker (every 15m) → 4h reminder DM
→ 48h escalation to #approvals
→ 24h auto-fail (KEEP)
After:
agent → MCP create_agent (existing direct path) → agent registered
message claimed → StalemateWorker (every 15m) → 24h auto-fail
```
Net code deletion. No new components, no new config surface, no new dependencies.
## Components touched
| File | Change |
|------|--------|
| `internal/messaging/stalemate.go` | Delete `sendPendingReminders`, `escalatePendingMessages`, `checkWorkflowStalemates`, `reminderExists`, `escalationExists`. Trim `StalemateConfig` to `ProcessingTimeout` + `Interval`. Remove `ReminderAfter` / `EscalateAfter` env vars. |
| `internal/messaging/stalemate_test.go` | Delete tests for removed methods; keep ProcessingTimeout tests. |
| `internal/messaging/options.go` | Remove channelLookup wiring if it's only used by escalation. |
| `internal/messaging/service.go` | Remove escalation hooks if any. |
| `internal/mcp/goals_tools.go` (spec 018) | Delete `propose_agent` tool registration (`proposeAgentTool`) and its `handleProposeAgent` handler. Keep `propose_task_tree` and the rest of the registrar. |
| `internal/storage/schema/027_remove_approval_noise.sql` | New migration. |
| `cmd/synapbus/admin.go`, `cmd/synapbus/main.go` | Remove any escalation-related flags. |
| `CLAUDE.md` (project + user) | Update SynapBus protocol section to drop "#approvals" + "stalemate auto-fails after 24h" mention of escalation. Keep claim-process-done loop. |
| User's `~/.claude/CLAUDE.md` | Same — drop approval-channel references and the auto-report trigger for "Need approval → #approvals". |
## Testing
- Existing `stalemate_test.go` cases for `ProcessingTimeout` continue to pass.
- New test: confirm `StalemateWorker.tick()` no longer queries pending messages for reminder/escalation candidates (no rows touched, no DMs sent).
- New test: confirm `propose_agent` MCP tool returns "tool not found" / is unregistered.
- Migration test: apply `027_remove_approval_noise.sql` to a fixture DB containing stalemate DMs + an `#approvals` message + an `agent_proposals` row; assert all three are gone, other messages untouched.
- No UI testing required — Web UI just stops showing approval-channel content because the channel is empty.
## Risks & mitigations
- **An external agent calls `propose_agent` after deletion.** MCP returns an unknown-tool error; agent's runbook should tolerate this. Acceptable because the user controls all agents.
- **Hidden consumer of escalation messages.** Search confirms reminders/escalations are only produced by `StalemateWorker` and consumed by humans. Low risk.
- **Migration deletes too much.** The `LIKE 'stalemate-%'` pattern is narrow and the `#approvals` channel is internal-only; nothing user-authored lives there. Take a `data/synapbus.db` backup before applying in prod (kubic).
## Out of scope
- Webhook/K8s runner human gates (spec 003).
- Removing the `#approvals` channel row.
- Adding `SYNAPBUS_APPROVALS_DISABLED` env flag — code deletion is reversible via git revert.
- Daily summarizer agent + `#summary-daily` channel — already in progress in a separate effort.
- Reclaiming disk via `VACUUM` — separate admin command if needed.
+41
View File
@@ -0,0 +1,41 @@
# SynapBus dream-agent — slim Python container that runs Claude Code via
# claude-agent-sdk against SynapBus's MCP server. Built for linux/amd64.
#
# The proven recipe (per ~/repos/searcher/agents/universal/Dockerfile)
# is a single-stage image with `uv pip install --system`. Multi-stage
# saves little since claude-agent-sdk transitively pulls anyio/httpx,
# and the heavy bit (the `claude` CLI binary) ships inside the wheel as
# a JS bundle.
FROM python:3.12-slim
# Bring in `uv` from its official image. Pure binary, no apt.
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
# System deps: git (claude-agent-sdk shells out for some workspace ops),
# ca-certs + curl for TLS / health checks. Cleanup apt lists.
RUN apt-get update && \
apt-get install -y --no-install-recommends git ca-certificates curl && \
apt-get clean && rm -rf /var/lib/apt/lists/* && \
git config --global user.email "dream-agent@synapbus.dev" && \
git config --global user.name "SynapBus Dream Agent"
WORKDIR /app
# Pinned versions — keep aligned with pyproject.toml. claude-agent-sdk
# 0.1.48 bundles the `claude` CLI Node binary inside its wheel, so no
# separate `claude-code` install step is required.
RUN uv pip install --system --no-cache \
"claude-agent-sdk==0.1.48" \
"httpx>=0.27" \
"opentelemetry-api>=1.27" \
"opentelemetry-sdk>=1.27" \
"opentelemetry-exporter-otlp-proto-http>=1.27"
COPY dream_runner.py /app/dream_runner.py
# Non-root user (matches searcher convention)
RUN groupadd -g 1000 dream && useradd -u 1000 -g 1000 -m dream && \
chown -R dream:dream /app
USER dream
ENTRYPOINT ["python", "/app/dream_runner.py"]
+69
View File
@@ -0,0 +1,69 @@
# synapbus-dream-agent
A slim Python container that performs **memory consolidation** for
SynapBus, dispatched on demand by the in-server `ConsolidatorWorker`
via the `k8sjob` harness backend.
## What it does
1. Reads its job context from env vars (dispatch token, job id, job
type, owner id, prompt).
2. Connects to SynapBus's MCP endpoint over streamable-http, passing
the agent's API key (`Authorization: Bearer ...`) **and** the
dispatch token (`X-Synapbus-Dispatch-Token: ...`) on every request.
3. Runs Claude Code (via `claude-agent-sdk`) constrained to the six
`memory_*` MCP tools defined in
`specs/020-proactive-memory-dream-worker/contracts/mcp-memory-tools.md`.
4. Streams structured JSON logs to stdout (Loki-friendly) and emits a
final `{"final": true, ...}` envelope so the harness can parse Usage.
## How the worker invokes it
`internal/messaging/consolidator.go` builds an `HarnessExecRequest`
with:
| Env var | Set by |
|---------------------------------|----------------------|
| `SYNAPBUS_DISPATCH_TOKEN` | ConsolidatorWorker |
| `SYNAPBUS_CONSOLIDATION_JOB_ID` | ConsolidatorWorker |
| `SYNAPBUS_JOB_TYPE` | ConsolidatorWorker |
| `SYNAPBUS_OWNER_ID` | ConsolidatorWorker |
| `SYNAPBUS_DREAM_PROMPT` | ConsolidatorWorker |
| `SYNAPBUS_RUN_ID` | k8sjob harness |
| `SYNAPBUS_URL`, `SYNAPBUS_API_KEY`, `ANTHROPIC_API_KEY` | Pod spec / Secret |
## Build
```bash
docker buildx build --platform=linux/amd64 \
-t kubic.home.arpa:32000/synapbus-dream-agent:v0.1.0 \
--load /Users/user/repos/synapbus/dream-agent/
```
Push:
```bash
docker push kubic.home.arpa:32000/synapbus-dream-agent:v0.1.0
```
## Local smoke test
The `--mock` flag validates the env contract and exits without
invoking the SDK or hitting the network:
```bash
SYNAPBUS_URL=http://localhost:8080 \
SYNAPBUS_API_KEY=fake \
SYNAPBUS_DISPATCH_TOKEN=fake \
SYNAPBUS_CONSOLIDATION_JOB_ID=1 \
SYNAPBUS_JOB_TYPE=reflection \
SYNAPBUS_OWNER_ID=algis \
SYNAPBUS_DREAM_PROMPT="test" \
SYNAPBUS_RUN_ID=r-test \
python3 dream_runner.py --mock
```
## Deploy
See `k8s-job-template.yaml`. The harness clones the template and
overlays `req.Env` into `containers[0].env`.
+401
View File
@@ -0,0 +1,401 @@
#!/usr/bin/env python3
"""SynapBus dream-agent runner — memory consolidation worker.
Dispatched by SynapBus's ConsolidatorWorker via the k8sjob harness.
Runs Claude Code (via claude-agent-sdk) against SynapBus's MCP server,
using a one-time dispatch token to authorize the six memory_* tools.
Environment contract (set by ConsolidatorWorker.runJob + k8sjob harness):
SYNAPBUS_URL base URL, e.g. http://synapbus.synapbus.svc.cluster.local:8080
SYNAPBUS_API_KEY dream-claude agent API key (Bearer auth)
SYNAPBUS_DISPATCH_TOKEN one-shot token authorizing memory_* tools
SYNAPBUS_CONSOLIDATION_JOB_ID parent job id (audit anchor)
SYNAPBUS_JOB_TYPE reflection | core_rewrite | dedup_contradiction | link_gen
SYNAPBUS_OWNER_ID target owner id
SYNAPBUS_DREAM_PROMPT job-type prompt (PromptFor)
SYNAPBUS_RUN_ID harness-injected run id
Optional:
ANTHROPIC_API_KEY or CLAUDE_CONFIG_DIR Claude Code credentials
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT OTLP/HTTP traces endpoint
DREAM_MAX_TURNS override max_turns (default 20)
DREAM_MODEL override model (default claude-sonnet-4-6)
"""
from __future__ import annotations
import argparse
import asyncio
import json
import logging
import os
import shutil
import sys
import tempfile
import time
from typing import Any
# --- Structured JSON logging (one obj per line for Loki) -------------------
class _JsonFormatter(logging.Formatter):
def __init__(self) -> None:
super().__init__()
self.job_id = os.environ.get("SYNAPBUS_CONSOLIDATION_JOB_ID", "")
self.job_type = os.environ.get("SYNAPBUS_JOB_TYPE", "")
self.owner_id = os.environ.get("SYNAPBUS_OWNER_ID", "")
self.run_id = os.environ.get("SYNAPBUS_RUN_ID", "")
self.trace_id: str = ""
def format(self, record: logging.LogRecord) -> str:
entry: dict[str, Any] = {
"ts": self.formatTime(record, "%Y-%m-%dT%H:%M:%SZ"),
"level": record.levelname,
"logger": record.name,
"job_id": self.job_id,
"job_type": self.job_type,
"owner_id": self.owner_id,
"run_id": self.run_id,
}
if self.trace_id:
entry["traceID"] = self.trace_id
if isinstance(record.msg, dict):
entry.update(record.msg)
else:
entry["msg"] = record.getMessage()
return json.dumps(entry, default=str)
def _setup_logging() -> logging.Logger:
lg = logging.getLogger("dream-agent")
lg.setLevel(logging.INFO)
lg.handlers.clear()
lg.propagate = False
h = logging.StreamHandler(sys.stdout)
h.setFormatter(_JsonFormatter())
lg.addHandler(h)
root = logging.getLogger()
root.handlers.clear()
root.addHandler(h)
return lg
logger = logging.getLogger("dream-agent")
# --- OTEL tracing (best-effort) --------------------------------------------
_tracer = None
def _init_tracing() -> None:
global _tracer
ep = os.environ.get("OTEL_EXPORTER_OTLP_TRACES_ENDPOINT", "")
if not ep:
return
try:
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.resources import Resource
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
resource = Resource.create({
"service.name": "synapbus-dream-agent",
"service.version": "0.1.0",
"synapbus.job_id": os.environ.get("SYNAPBUS_CONSOLIDATION_JOB_ID", ""),
"synapbus.job_type": os.environ.get("SYNAPBUS_JOB_TYPE", ""),
"synapbus.owner_id": os.environ.get("SYNAPBUS_OWNER_ID", ""),
})
provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint=ep)))
trace.set_tracer_provider(provider)
_tracer = trace.get_tracer("synapbus-dream-agent", "0.1.0")
logger.info({"msg": "OTEL tracing enabled", "endpoint": ep})
except ImportError:
logger.info({"msg": "OTEL packages not installed; tracing disabled"})
except Exception as e: # noqa: BLE001
logger.warning({"msg": "OTEL init failed", "error": str(e)})
def _shutdown_tracing() -> None:
try:
from opentelemetry import trace
p = trace.get_tracer_provider()
if hasattr(p, "shutdown"):
p.shutdown()
except Exception: # noqa: BLE001
pass
# --- Claude Code creds: ensure config dir is writable ----------------------
def _ensure_writable_config() -> str:
src = os.environ.get("CLAUDE_CONFIG_DIR", os.path.expanduser("~/.claude"))
test = os.path.join(src, ".write_test")
try:
os.makedirs(src, exist_ok=True)
with open(test, "w") as f:
f.write("ok")
os.remove(test)
return src
except OSError:
pass
tmp = tempfile.mkdtemp(prefix="claude_config_")
for fn in (".credentials.json", "credentials.json", "settings.json"):
s = os.path.join(src, fn)
if os.path.exists(s):
shutil.copy2(s, os.path.join(tmp, fn))
logger.info({"msg": "Created writable Claude config dir", "path": tmp})
return tmp
# --- Required-env helper ---------------------------------------------------
_REQUIRED = (
"SYNAPBUS_URL",
"SYNAPBUS_API_KEY",
"SYNAPBUS_DISPATCH_TOKEN",
"SYNAPBUS_CONSOLIDATION_JOB_ID",
"SYNAPBUS_JOB_TYPE",
"SYNAPBUS_OWNER_ID",
"SYNAPBUS_DREAM_PROMPT",
)
def _read_env() -> dict[str, str]:
out: dict[str, str] = {}
missing: list[str] = []
for k in _REQUIRED:
v = os.environ.get(k, "")
if not v:
missing.append(k)
out[k] = v
if missing:
raise RuntimeError(f"missing required env vars: {','.join(missing)}")
out["SYNAPBUS_RUN_ID"] = os.environ.get("SYNAPBUS_RUN_ID", "")
return out
# --- Prompt builder --------------------------------------------------------
_ALLOWED_TOOLS = [
"mcp__synapbus__memory_list_unprocessed",
"mcp__synapbus__memory_write_reflection",
"mcp__synapbus__memory_rewrite_core",
"mcp__synapbus__memory_mark_duplicate",
"mcp__synapbus__memory_supersede",
"mcp__synapbus__memory_add_link",
]
def _build_prompt(env: dict[str, str]) -> str:
return (
f"{env['SYNAPBUS_DREAM_PROMPT']}\n\n"
"Context:\n"
f"- job_id: {env['SYNAPBUS_CONSOLIDATION_JOB_ID']}\n"
f"- job_type: {env['SYNAPBUS_JOB_TYPE']}\n"
f"- owner_id: {env['SYNAPBUS_OWNER_ID']}\n"
f"- run_id: {env['SYNAPBUS_RUN_ID']}\n"
"- The dispatch token is forwarded automatically on every MCP "
"request via the `X-Synapbus-Dispatch-Token` header. You do not "
"need to pass it as a tool argument.\n"
"- Pass `owner_id` from the context above on every memory_* call.\n"
"- Use ONLY the memory_* tools listed in `allowed_tools`. Do not "
"call send_message, execute, search, or any other tool.\n"
"- When you are done, output a one-line JSON summary and exit.\n"
)
# --- Session runner --------------------------------------------------------
async def run_session(env: dict[str, str], model: str, max_turns: int, config_dir: str) -> dict[str, Any]:
from claude_agent_sdk import (
AssistantMessage,
ClaudeAgentOptions,
ResultMessage,
TextBlock,
query,
)
try:
from claude_agent_sdk import ToolUseBlock, ToolResultBlock, ThinkingBlock, UserMessage
except ImportError:
ToolUseBlock = ToolResultBlock = ThinkingBlock = UserMessage = None
base = env["SYNAPBUS_URL"].rstrip("/")
mcp_servers: dict[str, Any] = {
"synapbus": {
"type": "http",
"url": f"{base}/mcp",
"headers": {
"Authorization": f"Bearer {env['SYNAPBUS_API_KEY']}",
"X-Synapbus-Dispatch-Token": env["SYNAPBUS_DISPATCH_TOKEN"],
},
},
}
prompt = _build_prompt(env)
def _on_stderr(line: str) -> None:
logger.warning({"msg": "sdk_stderr", "line": line.rstrip()})
tokens_in = 0
tokens_out = 0
tool_calls = 0
turn = 0
started = time.time()
status = "ok"
error_msg = ""
try:
async for message in query(
prompt=prompt,
options=ClaudeAgentOptions(
model=model,
max_turns=max_turns,
mcp_servers=mcp_servers,
permission_mode="bypassPermissions",
allowed_tools=_ALLOWED_TOOLS,
env={"CLAUDE_CONFIG_DIR": config_dir},
stderr=_on_stderr,
),
):
if isinstance(message, ResultMessage):
usage = getattr(message, "usage", None)
tokens_in = getattr(usage, "input_tokens", 0) if usage else 0
tokens_out = getattr(usage, "output_tokens", 0) if usage else 0
# Max20 OAuth sessions don't surface tokens through the
# SDK's ResultMessage.usage. Fall back to a turn-based
# estimate so the server-side UsageGate has *some* signal.
# Numbers calibrated from observed reflection runs:
# ~5K input + ~300 output per turn, plus ~2K per tool call
# (memory_list_unprocessed payloads dominate).
if tokens_in == 0:
num_turns = int(getattr(message, "num_turns", 0) or 0)
tokens_in = max(0, num_turns * 5000 + tool_calls * 2000)
if tokens_out == 0:
num_turns = int(getattr(message, "num_turns", 0) or 0)
tokens_out = max(0, num_turns * 300)
is_error = getattr(message, "is_error", False)
duration_s = round(time.time() - started, 1)
logger.info({
"type": "result",
"turns": getattr(message, "num_turns", 0),
"cost_usd": getattr(message, "cost_usd", 0) or 0,
"tokens_in": tokens_in,
"tokens_out": tokens_out,
"tool_calls": tool_calls,
"duration_s": duration_s,
"is_error": is_error,
})
if is_error:
status = "error"
error_msg = "result_message.is_error=true"
elif isinstance(message, AssistantMessage):
turn += 1
for block in message.content:
if isinstance(block, TextBlock):
logger.info({
"type": "text",
"turn": turn,
"text": block.text[:300].replace("\n", " "),
})
elif ToolUseBlock and isinstance(block, ToolUseBlock):
tool_calls += 1
logger.info({
"type": "tool_use",
"turn": turn,
"tool": getattr(block, "name", "unknown"),
"input": str(getattr(block, "input", ""))[:200],
})
elif ThinkingBlock and isinstance(block, ThinkingBlock):
logger.info({
"type": "thinking",
"turn": turn,
"text": getattr(block, "text", "")[:200].replace("\n", " "),
})
elif UserMessage and isinstance(message, UserMessage):
for block in message.content:
if ToolResultBlock and isinstance(block, ToolResultBlock):
is_err = getattr(block, "is_error", False)
logger.info({
"type": "tool_result",
"turn": turn,
"is_error": is_err,
"content": str(getattr(block, "content", ""))[:200],
})
except Exception as e: # noqa: BLE001
status = "error"
error_msg = f"{type(e).__name__}: {e}"
logger.error({"msg": "session failed", "error": error_msg})
return {
"final": True,
"tokens_in": tokens_in,
"tokens_out": tokens_out,
"tool_calls": tool_calls,
"turns": turn,
"status": status,
"error": error_msg,
}
# --- main ------------------------------------------------------------------
def main() -> int:
global logger
parser = argparse.ArgumentParser(description="SynapBus dream-agent runner")
parser.add_argument("--mock", action="store_true",
help="Log env contract and exit without invoking the SDK")
parser.add_argument("--max-turns", type=int,
default=int(os.environ.get("DREAM_MAX_TURNS", "20")))
parser.add_argument("--model", default=os.environ.get("DREAM_MODEL", "claude-sonnet-4-6"))
args = parser.parse_args()
logger = _setup_logging()
_init_tracing()
try:
env = _read_env()
except RuntimeError as e:
logger.error({"msg": "env validation failed", "error": str(e)})
print(json.dumps({
"final": True, "tokens_in": 0, "tokens_out": 0,
"tool_calls": 0, "status": "error", "error": str(e),
}))
return 1
logger.info({
"msg": "dream-agent starting",
"synapbus_url": env["SYNAPBUS_URL"],
"model": args.model,
"max_turns": args.max_turns,
})
if args.mock:
logger.info({"msg": "--mock; skipping SDK invocation"})
print(json.dumps({
"final": True, "tokens_in": 0, "tokens_out": 0,
"tool_calls": 0, "status": "ok", "error": "",
}))
return 0
config_dir = _ensure_writable_config()
try:
result = asyncio.run(run_session(env, args.model, args.max_turns, config_dir))
except Exception as e: # noqa: BLE001
logger.error({"msg": "fatal", "error": f"{type(e).__name__}: {e}"})
print(json.dumps({
"final": True, "tokens_in": 0, "tokens_out": 0,
"tool_calls": 0, "status": "error", "error": str(e),
}))
_shutdown_tracing()
return 1
# Final single-line envelope for harness Usage parsing.
print(json.dumps(result))
_shutdown_tracing()
return 0 if result.get("status") == "ok" else 1
if __name__ == "__main__":
sys.exit(main())
+82
View File
@@ -0,0 +1,82 @@
# Reference Job template that the SynapBus k8sjob harness instantiates
# per dispatched dream-agent run. The harness will:
# 1. Clone this template
# 2. Populate `metadata.name` with `dream-<job_id>-<run_id_short>`
# 3. Merge req.Env into `env:` (SYNAPBUS_DISPATCH_TOKEN,
# SYNAPBUS_CONSOLIDATION_JOB_ID, SYNAPBUS_JOB_TYPE,
# SYNAPBUS_OWNER_ID, SYNAPBUS_DREAM_PROMPT, SYNAPBUS_RUN_ID)
# 4. Tail container logs back to the worker
#
# Replace `kubic.home.arpa:32000/synapbus-dream-agent:v0.1.0` with the
# actual image tag your registry publishes.
apiVersion: batch/v1
kind: Job
metadata:
name: dream-agent-PLACEHOLDER
namespace: synapbus
labels:
app: synapbus-dream-agent
synapbus.io/role: memory-consolidator
spec:
backoffLimit: 0 # one-shot — server-side circuit breaker decides retries
ttlSecondsAfterFinished: 600
activeDeadlineSeconds: 900 # hard cap above DreamWallclockBudget (default 10m)
template:
metadata:
labels:
app: synapbus-dream-agent
spec:
restartPolicy: Never
serviceAccountName: default
containers:
- name: dream-agent
image: kubic.home.arpa:32000/synapbus-dream-agent:v0.1.0
imagePullPolicy: IfNotPresent
env:
# In-cluster SynapBus address (cluster DNS).
- name: SYNAPBUS_URL
value: "http://synapbus.synapbus.svc.cluster.local:8080"
- name: SYNAPBUS_API_KEY
valueFrom:
secretKeyRef:
name: dream-agent-secrets
key: SYNAPBUS_API_KEY
- name: ANTHROPIC_API_KEY
valueFrom:
secretKeyRef:
name: dream-agent-secrets
key: ANTHROPIC_API_KEY
- name: CLAUDE_CONFIG_DIR
value: "/home/dream/.claude"
# Optional OTLP/HTTP traces export to Tempo
- name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
value: "http://tempo.observability.svc.cluster.local:4318/v1/traces"
# ---- The harness Env map appends here at dispatch time ----
# SYNAPBUS_DISPATCH_TOKEN, SYNAPBUS_CONSOLIDATION_JOB_ID,
# SYNAPBUS_JOB_TYPE, SYNAPBUS_OWNER_ID, SYNAPBUS_DREAM_PROMPT,
# SYNAPBUS_RUN_ID
resources:
requests:
cpu: "200m"
memory: "256Mi"
limits:
cpu: "1"
memory: "512Mi"
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: false
runAsNonRoot: true
runAsUser: 1000
capabilities:
drop: ["ALL"]
---
# Secret skeleton — populate via kubectl + sealed-secrets / sops out of band.
apiVersion: v1
kind: Secret
metadata:
name: dream-agent-secrets
namespace: synapbus
type: Opaque
stringData:
SYNAPBUS_API_KEY: "REPLACE_ME" # dream-claude agent's SynapBus API key
ANTHROPIC_API_KEY: "REPLACE_ME" # Anthropic API key for Claude Code
+19
View File
@@ -0,0 +1,19 @@
[project]
name = "synapbus-dream-agent"
version = "0.1.0"
description = "SynapBus memory-consolidation dream-agent runner (claude-agent-sdk)"
requires-python = ">=3.12"
dependencies = [
"claude-agent-sdk==0.1.48",
"httpx>=0.27",
"opentelemetry-api>=1.27",
"opentelemetry-sdk>=1.27",
"opentelemetry-exporter-otlp-proto-http>=1.27",
]
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = []
+5
View File
@@ -142,3 +142,8 @@ require (
sigs.k8s.io/structured-merge-diff/v6 v6.3.0 // indirect
sigs.k8s.io/yaml v1.6.0 // indirect
)
// OPC local patch: github.com/TFMV/hnsw calls renameio.TempFile, which
// renameio v1 does not provide on Windows. Pin v0.1.0 so the binary builds
// on Windows. Drop this once upstream hnsw/renameio no longer needs it.
replace github.com/google/renameio => github.com/google/renameio v0.1.0
+1
View File
@@ -232,6 +232,7 @@ github.com/google/pprof v0.0.0-20201203190320-1bf35d6f28c2/go.mod h1:kpwsk12EmLe
github.com/google/pprof v0.0.0-20201218002935-b9804c9f04c2/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
github.com/google/pprof v0.0.0-20250403155104-27863c87afa6 h1:BHT72Gu3keYf3ZEu2J0b1vyeLSOYI8bm5wbJM/8yDe8=
github.com/google/pprof v0.0.0-20250403155104-27863c87afa6/go.mod h1:boTsfXsheKC2y+lKOCMpSfarhxDeIzfZG1jqGcPl3cA=
github.com/google/renameio v0.1.0 h1:GOZbcHa3HfsPKPlmyPyN2KEohoMXOhdMbHrvbpl2QaA=
github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
github.com/google/renameio v1.0.1 h1:Lh/jXZmvZxb0BBeSY5VKEfidcbcbenKjZFzM/q0fSeU=
github.com/google/renameio v1.0.1/go.mod h1:t/HQoYBZSsWSNK35C6CO/TpPLDVWvxOHboWUAweKUpk=
+34 -14
View File
@@ -34,20 +34,40 @@ type K8sServiceProvider interface {
// Services holds references to all services the admin socket can control.
type Services struct {
Users *auth.SQLiteUserStore
Sessions auth.SessionStore
Agents *agents.AgentService
Messages *messaging.MessagingService
Channels *channels.Service
Traces trace.TraceStore
EmbeddingStore *search.EmbeddingStore
VectorIndex *search.VectorIndex
SearchService *search.Service
AttachmentService *attachments.Service
WebhookService WebhookServiceProvider
K8sService K8sServiceProvider
DataDir string
RetentionWorker RetentionStatusProvider
Users *auth.SQLiteUserStore
Sessions auth.SessionStore
Agents *agents.AgentService
Messages *messaging.MessagingService
Channels *channels.Service
Traces trace.TraceStore
EmbeddingStore *search.EmbeddingStore
VectorIndex *search.VectorIndex
SearchService *search.Service
AttachmentService *attachments.Service
WebhookService WebhookServiceProvider
K8sService K8sServiceProvider
DataDir string
RetentionWorker RetentionStatusProvider
// CoreMemoryStore is the per-(owner, agent) core memory store wired
// in for feature 020 admin CLI commands (`synapbus memory core ...`).
// May be nil — handlers report "core memory store not configured".
CoreMemoryStore *messaging.CoreMemoryStore
// DreamRun, when non-nil, dispatches a single consolidation job
// bypassing the trigger check. Wired by main.go when the
// consolidator worker is enabled. Closure form keeps the worker
// internals out of the admin package's import graph.
DreamRun func(ctx context.Context, ownerID, jobType string) (jobID int64, err error)
// DreamRunN fans out N parallel consolidation jobs (via slot 0..N-1)
// for one (owner, job_type). Used by `synapbus memory dream-run
// --parallel N`. core_rewrite always coerces to N=1 server-side.
DreamRunN func(ctx context.Context, ownerID, jobType string, parallel int) (jobIDs []int64, err error)
// DefaultDreamParallel is consulted when the CLI request omits
// --parallel. Sourced from MemoryConfig.DreamParallel.
DefaultDreamParallel int
}
// RetentionStatusProvider provides retention status information.
+185 -2
View File
@@ -226,6 +226,16 @@ func (s *AdminServer) dispatch(req Request) Response {
case "harness.config_set":
return s.handleHarnessConfigSet(ctx, req.Args)
// --- memory core (feature 020 — proactive memory) ---
case "memory.core.get":
return s.handleMemoryCoreGet(ctx, req.Args)
case "memory.core.set":
return s.handleMemoryCoreSet(ctx, req.Args)
case "memory.core.delete":
return s.handleMemoryCoreDelete(ctx, req.Args)
case "memory.dream_run":
return s.handleMemoryDreamRun(ctx, req.Args)
default:
return Response{OK: false, Error: fmt.Sprintf("unknown command: %s", req.Command)}
}
@@ -545,7 +555,7 @@ func (s *AdminServer) handleAuditStats(ctx context.Context) Response {
}
return Response{OK: true, Data: map[string]interface{}{
"total_traces": totalTraces,
"total_traces": totalTraces,
"counts_by_action": counts,
}}
}
@@ -1694,7 +1704,7 @@ func (s *AdminServer) handleAttachmentsGC(ctx context.Context) Response {
}
return Response{OK: true, Data: map[string]interface{}{
"files_removed": result.FilesRemoved,
"files_removed": result.FilesRemoved,
"bytes_reclaimed": result.BytesReclaimed,
}}
}
@@ -1850,5 +1860,178 @@ func (s *AdminServer) handleHarnessConfigSet(ctx context.Context, args json.RawM
}}
}
// ---------- memory core handlers (feature 020) ----------
//
// owner_id wire format: callers pass the owner as a username; we resolve
// to `users.id` and pass the string form to CoreMemoryStore so it matches
// the proactive-memory tables' TEXT owner_id convention.
func (s *AdminServer) resolveOwnerString(ctx context.Context, ownerInput string) (string, error) {
if ownerInput == "" {
return "", fmt.Errorf("owner is required")
}
// First try numeric — admins may already know the user ID.
var id int64
if _, err := fmt.Sscanf(ownerInput, "%d", &id); err == nil && id > 0 {
return fmt.Sprintf("%d", id), nil
}
// Otherwise treat as username.
user, err := s.services.Users.GetUserByUsername(ctx, ownerInput)
if err != nil {
return "", fmt.Errorf("resolve owner %q: %w", ownerInput, err)
}
return fmt.Sprintf("%d", user.ID), nil
}
func (s *AdminServer) handleMemoryCoreGet(ctx context.Context, args json.RawMessage) Response {
var p struct {
Owner string `json:"owner"`
Agent string `json:"agent"`
}
if err := json.Unmarshal(args, &p); err != nil {
return Response{OK: false, Error: "invalid args: " + err.Error()}
}
if p.Agent == "" {
return Response{OK: false, Error: "agent is required"}
}
if s.services.CoreMemoryStore == nil {
return Response{OK: false, Error: "core memory store not configured"}
}
ownerStr, err := s.resolveOwnerString(ctx, p.Owner)
if err != nil {
return Response{OK: false, Error: err.Error()}
}
blob, updatedAt, ok, err := s.services.CoreMemoryStore.Get(ctx, ownerStr, p.Agent)
if err != nil {
return Response{OK: false, Error: err.Error()}
}
if !ok {
return Response{OK: true, Data: map[string]any{
"owner_id": ownerStr,
"agent_name": p.Agent,
"exists": false,
}}
}
return Response{OK: true, Data: map[string]any{
"owner_id": ownerStr,
"agent_name": p.Agent,
"exists": true,
"blob": blob,
"updated_at": updatedAt.Format(time.RFC3339),
}}
}
func (s *AdminServer) handleMemoryCoreSet(ctx context.Context, args json.RawMessage) Response {
var p struct {
Owner string `json:"owner"`
Agent string `json:"agent"`
Blob string `json:"blob"`
UpdatedBy string `json:"updated_by"`
}
if err := json.Unmarshal(args, &p); err != nil {
return Response{OK: false, Error: "invalid args: " + err.Error()}
}
if p.Agent == "" {
return Response{OK: false, Error: "agent is required"}
}
if s.services.CoreMemoryStore == nil {
return Response{OK: false, Error: "core memory store not configured"}
}
ownerStr, err := s.resolveOwnerString(ctx, p.Owner)
if err != nil {
return Response{OK: false, Error: err.Error()}
}
updatedBy := p.UpdatedBy
if updatedBy == "" {
updatedBy = "human"
}
if err := s.services.CoreMemoryStore.Set(ctx, ownerStr, p.Agent, p.Blob, updatedBy); err != nil {
if err == messaging.ErrCoreMemoryTooLarge {
return Response{OK: false, Error: fmt.Sprintf("core_memory_too_large: blob exceeds %d bytes", s.services.CoreMemoryStore.MaxBytes())}
}
return Response{OK: false, Error: err.Error()}
}
return Response{OK: true, Data: map[string]any{
"owner_id": ownerStr,
"agent_name": p.Agent,
"blob_chars": len(p.Blob),
"updated_by": updatedBy,
}}
}
func (s *AdminServer) handleMemoryCoreDelete(ctx context.Context, args json.RawMessage) Response {
var p struct {
Owner string `json:"owner"`
Agent string `json:"agent"`
}
if err := json.Unmarshal(args, &p); err != nil {
return Response{OK: false, Error: "invalid args: " + err.Error()}
}
if p.Agent == "" {
return Response{OK: false, Error: "agent is required"}
}
if s.services.CoreMemoryStore == nil {
return Response{OK: false, Error: "core memory store not configured"}
}
ownerStr, err := s.resolveOwnerString(ctx, p.Owner)
if err != nil {
return Response{OK: false, Error: err.Error()}
}
if err := s.services.CoreMemoryStore.Delete(ctx, ownerStr, p.Agent); err != nil {
return Response{OK: false, Error: err.Error()}
}
return Response{OK: true, Data: map[string]any{
"owner_id": ownerStr,
"agent_name": p.Agent,
"deleted": true,
}}
}
// handleMemoryDreamRun forces dream-job dispatch(es). With parallel=1
// (default) returns one job_id. With parallel>1, fans out N concurrent
// jobs across slots 0..N-1 and returns the list of created ids. The
// circuit breaker still applies.
func (s *AdminServer) handleMemoryDreamRun(ctx context.Context, args json.RawMessage) Response {
var p struct {
Owner string `json:"owner"`
JobType string `json:"job_type"`
Parallel int `json:"parallel"`
}
if err := json.Unmarshal(args, &p); err != nil {
return Response{OK: false, Error: "invalid args: " + err.Error()}
}
if p.JobType == "" {
return Response{OK: false, Error: "job_type is required"}
}
if s.services.DreamRunN == nil {
return Response{OK: false, Error: "dream worker not configured (SYNAPBUS_DREAM_ENABLED=0?)"}
}
ownerStr, err := s.resolveOwnerString(ctx, p.Owner)
if err != nil {
return Response{OK: false, Error: err.Error()}
}
parallel := p.Parallel
if parallel <= 0 {
parallel = s.services.DefaultDreamParallel
}
if parallel <= 0 {
parallel = 1
}
ids, err := s.services.DreamRunN(ctx, ownerStr, p.JobType, parallel)
if err != nil && len(ids) == 0 {
return Response{OK: false, Error: err.Error()}
}
out := map[string]any{
"job_ids": ids,
"owner_id": ownerStr,
"job_type": p.JobType,
"parallel": parallel,
}
if err != nil {
out["error"] = err.Error()
}
return Response{OK: true, Data: out}
}
// Ensure the messaging import is used.
var _ = messaging.StatusPending
+53
View File
@@ -0,0 +1,53 @@
package agents
import (
"context"
"database/sql"
"errors"
"fmt"
"strconv"
)
// Sentinel errors returned by OwnerFor.
//
// Callers in the proactive-memory / dream-worker path use these to
// distinguish "no such agent" from "agent exists but has no owner"
// without parsing error strings.
var (
// ErrAgentNotFound is returned when no row exists in `agents` for
// the requested name.
ErrAgentNotFound = errors.New("agent not found")
// ErrAgentUnowned is returned when an agent row exists but its
// owner_id is zero / empty. In the current schema owner_id is
// declared NOT NULL, so this is effectively an integrity guard for
// rows backfilled with 0.
ErrAgentUnowned = errors.New("agent has no owner")
)
// OwnerFor returns the string-encoded owner_id of the named agent.
//
// The schema stores `agents.owner_id` as INTEGER (FK to `users.id`), but
// the proactive-memory tables and the request-context `owner_id`
// (populated by auth middleware via `trace.ContextWithOwnerID`) carry it
// as a string. OwnerFor canonicalizes to that string form so call sites
// can compare without re-converting.
//
// Returns ("", ErrAgentNotFound) when no row matches; ("",
// ErrAgentUnowned) when a row exists but owner_id is 0.
func OwnerFor(ctx context.Context, db *sql.DB, agentName string) (string, error) {
var ownerID int64
err := db.QueryRowContext(ctx,
`SELECT owner_id FROM agents WHERE name = ?`, agentName,
).Scan(&ownerID)
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
return "", ErrAgentNotFound
}
return "", fmt.Errorf("query owner for agent %q: %w", agentName, err)
}
if ownerID == 0 {
return "", ErrAgentUnowned
}
return strconv.FormatInt(ownerID, 10), nil
}
+285
View File
@@ -0,0 +1,285 @@
package api
import (
"context"
"encoding/json"
"fmt"
"net/http"
"strconv"
"time"
"github.com/synapbus/synapbus/internal/agents"
"github.com/synapbus/synapbus/internal/messaging"
)
const (
// maxAgentConnections is the maximum number of simultaneous
// /api/agent-events connections per agent.
maxAgentConnections = 5
// maxAgentBacklog is the maximum number of missed events replayed on
// reconnect; beyond this a single resync_required event is sent instead.
maxAgentBacklog = 200
// agentSubBuffer is the per-connection live event buffer. A connection
// whose buffer is full is dropped rather than blocking the broadcaster.
agentSubBuffer = 64
// agentWriteTimeout bounds a single write to a slow client.
agentWriteTimeout = 10 * time.Second
)
// AgentEventBacklog loads body-free metadata for messages visible to an agent,
// used to replay events after a reconnect with Last-Event-ID.
type AgentEventBacklog interface {
ListEventMetaAfter(ctx context.Context, agentName string, afterID int64, limit int) ([]*messaging.EventMeta, error)
}
// AgentMessageEvent is the payload of a new_message event on the agent stream.
// It carries metadata only, never the message body.
type AgentMessageEvent struct {
MessageID int64 `json:"message_id"`
Channel string `json:"channel,omitempty"`
FromAgent string `json:"from_agent,omitempty"`
ToAgent string `json:"to_agent,omitempty"`
Subject string `json:"subject,omitempty"`
}
// agentSub is one connected /api/agent-events client.
type agentSub struct {
ch chan AgentMessageEvent
}
// SetAgentEventBacklog configures the source used to replay missed events.
func (h *SSEHub) SetAgentEventBacklog(b AgentEventBacklog) {
h.mu.Lock()
defer h.mu.Unlock()
h.agentBacklog = b
}
// hasAgentSubs reports whether any /api/agent-events connection is open.
func (h *SSEHub) hasAgentSubs() bool {
h.mu.RLock()
defer h.mu.RUnlock()
return len(h.agentSubs) > 0
}
// BroadcastAgentMessage delivers a new_message event to every connection of the
// named agents. Connections that cannot keep up are dropped.
func (h *SSEHub) BroadcastAgentMessage(agentNames []string, ev AgentMessageEvent) {
type slowSub struct {
name string
sub *agentSub
}
var slow []slowSub
h.mu.RLock()
for _, name := range agentNames {
for sub := range h.agentSubs[name] {
select {
case sub.ch <- ev:
default:
slow = append(slow, slowSub{name, sub})
}
}
}
h.mu.RUnlock()
for _, s := range slow {
h.logger.Warn("dropping slow agent SSE client", "agent", s.name)
h.removeAgentSub(s.name, s.sub)
}
}
// addAgentSub registers a connection for the agent. It returns nil when the
// agent already has maxAgentConnections open connections.
func (h *SSEHub) addAgentSub(agentName string) *agentSub {
h.mu.Lock()
defer h.mu.Unlock()
set := h.agentSubs[agentName]
if len(set) >= maxAgentConnections {
return nil
}
if set == nil {
set = make(map[*agentSub]struct{})
h.agentSubs[agentName] = set
}
sub := &agentSub{ch: make(chan AgentMessageEvent, agentSubBuffer)}
set[sub] = struct{}{}
h.logger.Info("agent SSE client connected", "agent", agentName)
return sub
}
func (h *SSEHub) removeAgentSub(agentName string, sub *agentSub) {
h.mu.Lock()
defer h.mu.Unlock()
set := h.agentSubs[agentName]
if _, ok := set[sub]; ok {
delete(set, sub)
close(sub.ch)
h.logger.Info("agent SSE client disconnected", "agent", agentName)
}
if len(set) == 0 {
delete(h.agentSubs, agentName)
}
}
// closeAgentSubsLocked disconnects all agent subscriptions. Caller holds h.mu.
func (h *SSEHub) closeAgentSubsLocked() {
for name, set := range h.agentSubs {
for sub := range set {
close(sub.ch)
}
delete(h.agentSubs, name)
}
}
// HandleAgentEvents handles GET /api/agent-events. It must be mounted behind
// the agent API-key middleware; the authenticated agent is read from the
// request context. Only metadata for messages visible to that agent is sent.
func (h *SSEHub) HandleAgentEvents(w http.ResponseWriter, r *http.Request) {
agent, ok := agents.AgentFromContext(r.Context())
if !ok || agent == nil {
writeAgentEventsError(w, http.StatusUnauthorized, "unauthorized", "An agent API key is required")
return
}
flusher, ok := w.(http.Flusher)
if !ok {
http.Error(w, "Streaming not supported", http.StatusInternalServerError)
return
}
var lastID int64
hasLast := false
if v := r.Header.Get("Last-Event-ID"); v != "" {
id, err := strconv.ParseInt(v, 10, 64)
if err != nil || id < 0 {
writeAgentEventsError(w, http.StatusBadRequest, "bad_request", "Last-Event-ID must be a message id")
return
}
lastID, hasLast = id, true
}
// Subscribe before replaying so no message falls between the replay query
// and the live stream; duplicates are removed by id below.
sub := h.addAgentSub(agent.Name)
if sub == nil {
writeAgentEventsError(w, http.StatusTooManyRequests, "too_many_connections",
fmt.Sprintf("Agent %s already has the maximum of %d event stream connections", agent.Name, maxAgentConnections))
return
}
defer h.removeAgentSub(agent.Name, sub)
w.Header().Set("Content-Type", "text/event-stream")
w.Header().Set("Cache-Control", "no-cache")
w.Header().Set("Connection", "keep-alive")
w.Header().Set("X-Accel-Buffering", "no")
rc := http.NewResponseController(w)
write := func(id int64, eventType string, data any) bool {
_ = rc.SetWriteDeadline(time.Now().Add(agentWriteTimeout))
return writeSSEEvent(w, flusher, id, eventType, data)
}
if !write(0, "connected", map[string]any{
"agent": agent.Name,
"timestamp": time.Now().Format(time.RFC3339),
}) {
return
}
// replayedUpTo is fixed once the replay ends: live events at or below it
// are duplicates of the replay. It is never advanced by live events, because
// concurrent senders may broadcast ids out of order (11 before 10).
replayedUpTo := lastID
if hasLast {
h.mu.RLock()
backlog := h.agentBacklog
h.mu.RUnlock()
if backlog != nil {
metas, err := backlog.ListEventMetaAfter(r.Context(), agent.Name, lastID, maxAgentBacklog+1)
if err != nil || len(metas) > maxAgentBacklog {
if err != nil {
h.logger.Warn("agent SSE backlog failed", "agent", agent.Name, "error", err)
}
if !write(0, "resync_required", map[string]any{
"after_id": lastID,
"limit": maxAgentBacklog,
}) {
return
}
} else {
for _, m := range metas {
ev := AgentMessageEvent{MessageID: m.MessageID, Subject: m.Subject}
if m.Channel != "" {
ev.Channel = m.Channel
} else {
ev.FromAgent, ev.ToAgent = m.FromAgent, m.ToAgent
}
if !write(m.MessageID, "new_message", ev) {
return
}
replayedUpTo = m.MessageID
}
}
}
}
interval := h.heartbeat
if interval <= 0 {
interval = 30 * time.Second
}
ticker := time.NewTicker(interval)
defer ticker.Stop()
ctx := r.Context()
for {
select {
case <-ctx.Done():
return
case ev, ok := <-sub.ch:
if !ok {
return
}
if ev.MessageID <= replayedUpTo {
continue
}
if !write(ev.MessageID, "new_message", ev) {
return
}
case <-ticker.C:
if !write(0, "heartbeat", map[string]any{
"timestamp": time.Now().Format(time.RFC3339),
}) {
return
}
}
}
}
// writeAgentEventsError writes a JSON error body encoded with encoding/json.
func writeAgentEventsError(w http.ResponseWriter, status int, code, message string) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
_ = json.NewEncoder(w).Encode(map[string]string{"error": code, "message": message})
}
// writeSSEEvent writes one SSE frame. A positive id is emitted as the "id:"
// line. It reports whether the write succeeded.
func writeSSEEvent(w http.ResponseWriter, flusher http.Flusher, id int64, eventType string, data any) bool {
jsonData, err := json.Marshal(data)
if err != nil {
return false
}
if id > 0 {
if _, err := fmt.Fprintf(w, "id: %d\n", id); err != nil {
return false
}
}
if _, err := fmt.Fprintf(w, "event: %s\ndata: %s\n\n", eventType, jsonData); err != nil {
return false
}
flusher.Flush()
return true
}
+546
View File
@@ -0,0 +1,546 @@
package api
import (
"bufio"
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"strconv"
"strings"
"testing"
"time"
"github.com/go-chi/chi/v5"
_ "modernc.org/sqlite"
"github.com/synapbus/synapbus/internal/agents"
"github.com/synapbus/synapbus/internal/channels"
"github.com/synapbus/synapbus/internal/messaging"
)
// agentEventsEnv wires real services over a temporary in-memory SQLite
// database behind the real agent API-key middleware.
type agentEventsEnv struct {
t *testing.T
srv *httptest.Server
hub *SSEHub
msgSvc *messaging.MessagingService
chanSvc *channels.Service
keys map[string]string // agent name -> API key
channels map[string]int64 // channel name -> id
}
func newAgentEventsEnv(t *testing.T, agentNames ...string) *agentEventsEnv {
t.Helper()
db := newTestDBFull(t)
agentSvc := agents.NewAgentService(agents.NewSQLiteAgentStore(db), nil)
msgSvc := messaging.NewMessagingService(messaging.NewSQLiteMessageStore(db), nil)
chanSvc := channels.NewService(channels.NewSQLiteChannelStore(db), msgSvc, nil)
hub := NewSSEHub()
hub.SetAgentEventBacklog(msgSvc)
b := NewSSEBroadcaster(hub, agentSvc, chanSvc)
b.SetMessageService(msgSvc)
msgSvc.AddMessageListener(b)
env := &agentEventsEnv{
t: t, hub: hub, msgSvc: msgSvc, chanSvc: chanSvc,
keys: map[string]string{}, channels: map[string]int64{},
}
ctx := context.Background()
for _, name := range agentNames {
_, key, err := agentSvc.Register(ctx, name, name, "ai", nil, 1)
if err != nil {
t.Fatalf("register %s: %v", name, err)
}
env.keys[name] = key
}
r := chi.NewRouter()
r.Group(func(r chi.Router) {
r.Use(agents.RequiredAuthMiddlewareWithOAuth(agentSvc, nil, nil))
r.Get("/api/agent-events", hub.HandleAgentEvents)
})
env.srv = httptest.NewServer(r)
t.Cleanup(env.srv.Close)
t.Cleanup(hub.Close) // runs before srv.Close so open streams end
return env
}
func (e *agentEventsEnv) makeChannel(name string, members ...string) {
e.t.Helper()
ctx := context.Background()
ch, err := e.chanSvc.CreateChannel(ctx, channels.CreateChannelRequest{Name: name, CreatedBy: members[0]})
if err != nil {
e.t.Fatalf("create channel: %v", err)
}
e.channels[name] = ch.ID
for _, m := range members {
if err := e.chanSvc.JoinChannel(ctx, ch.ID, m); err != nil {
e.t.Fatalf("join %s: %v", m, err)
}
}
}
func (e *agentEventsEnv) dm(from, to, body, subject string) int64 {
e.t.Helper()
m, err := e.msgSvc.SendMessage(context.Background(), from, to, body, messaging.SendOptions{Subject: subject})
if err != nil {
e.t.Fatalf("send dm: %v", err)
}
return m.ID
}
func (e *agentEventsEnv) post(from, channel, body string) int64 {
e.t.Helper()
id := e.channels[channel]
m, err := e.msgSvc.SendMessage(context.Background(), from, "", body, messaging.SendOptions{ChannelID: &id})
if err != nil {
e.t.Fatalf("post: %v", err)
}
return m.ID
}
// sseFrame is one parsed SSE frame.
type sseFrame struct {
ID string
Event string
Data string
}
type sseClient struct {
t *testing.T
frames chan sseFrame
cancel context.CancelFunc
status int
body string
}
// connect opens the stream. For non-200 statuses it records status/body only.
func (e *agentEventsEnv) connect(key, lastEventID string) *sseClient {
e.t.Helper()
ctx, cancel := context.WithCancel(context.Background())
req, _ := http.NewRequestWithContext(ctx, "GET", e.srv.URL+"/api/agent-events", nil)
if key != "" {
req.Header.Set("Authorization", "Bearer "+key)
}
if lastEventID != "" {
req.Header.Set("Last-Event-ID", lastEventID)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
cancel()
e.t.Fatalf("connect: %v", err)
}
c := &sseClient{t: e.t, frames: make(chan sseFrame, 512), cancel: cancel, status: resp.StatusCode}
e.t.Cleanup(cancel)
if resp.StatusCode != http.StatusOK {
var sb strings.Builder
sc := bufio.NewScanner(resp.Body)
for sc.Scan() {
sb.WriteString(sc.Text())
}
resp.Body.Close()
c.body = sb.String()
return c
}
go func() {
defer resp.Body.Close()
defer close(c.frames)
sc := bufio.NewScanner(resp.Body)
var f sseFrame
for sc.Scan() {
line := sc.Text()
switch {
case line == "":
if f.Event != "" {
c.frames <- f
}
f = sseFrame{}
case strings.HasPrefix(line, "id: "):
f.ID = line[4:]
case strings.HasPrefix(line, "event: "):
f.Event = line[7:]
case strings.HasPrefix(line, "data: "):
f.Data = line[6:]
}
}
}()
return c
}
func (c *sseClient) next() sseFrame {
c.t.Helper()
select {
case f, ok := <-c.frames:
if !ok {
c.t.Fatal("stream closed while waiting for event")
}
return f
case <-time.After(5 * time.Second):
c.t.Fatal("timed out waiting for SSE event")
}
return sseFrame{}
}
// nextMessage skips nothing: it expects the next frame to be new_message.
func (c *sseClient) nextMessage() (sseFrame, map[string]any) {
c.t.Helper()
f := c.next()
if f.Event != "new_message" {
c.t.Fatalf("event = %q (data %s), want new_message", f.Event, f.Data)
}
var m map[string]any
if err := json.Unmarshal([]byte(f.Data), &m); err != nil {
c.t.Fatalf("bad json %q: %v", f.Data, err)
}
return f, m
}
func (c *sseClient) expectConnected(agent string) {
c.t.Helper()
f := c.next()
if f.Event != "connected" || !strings.Contains(f.Data, `"agent":"`+agent+`"`) {
c.t.Fatalf("first frame = %+v, want connected for %s", f, agent)
}
}
func TestAgentEvents_Auth(t *testing.T) {
env := newAgentEventsEnv(t, "alice")
tests := []struct {
name string
key string
}{
{"missing key", ""},
{"invalid key", "not-a-real-key"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
c := env.connect(tt.key, "")
if c.status != http.StatusUnauthorized {
t.Fatalf("status = %d, want 401", c.status)
}
})
}
t.Run("valid key gets connected event", func(t *testing.T) {
c := env.connect(env.keys["alice"], "")
if c.status != http.StatusOK {
t.Fatalf("status = %d, want 200", c.status)
}
c.expectConnected("alice")
})
}
func TestAgentEvents_HandlerWithoutAgentContext(t *testing.T) {
hub := NewSSEHub()
rr := httptest.NewRecorder()
hub.HandleAgentEvents(rr, httptest.NewRequest("GET", "/api/agent-events", nil))
if rr.Code != http.StatusUnauthorized {
t.Fatalf("status = %d, want 401", rr.Code)
}
}
func TestAgentEvents_Delivery(t *testing.T) {
const secret = "TOP-SECRET-BODY-TEXT"
tests := []struct {
name string
// send performs the action; returns the message id to expect for "bob".
send func(env *agentEventsEnv) int64
want map[string]any // expected fields (besides message_id)
absent []string // fields that must not be present
viewers []string
}{
{
name: "dm to bob",
send: func(env *agentEventsEnv) int64 { return env.dm("alice", "bob", secret, "hello subject") },
want: map[string]any{"from_agent": "alice", "to_agent": "bob", "subject": "hello subject"},
absent: []string{"channel", "body"},
},
{
name: "channel message to member",
send: func(env *agentEventsEnv) int64 {
env.makeChannel("room", "alice", "bob")
return env.post("alice", "room", secret)
},
want: map[string]any{"channel": "room"},
absent: []string{"to_agent", "from_agent", "body"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
env := newAgentEventsEnv(t, "alice", "bob")
bob := env.connect(env.keys["bob"], "")
bob.expectConnected("bob")
alice := env.connect(env.keys["alice"], "")
alice.expectConnected("alice")
id := tt.send(env)
f, m := bob.nextMessage()
if f.ID != strconv.FormatInt(id, 10) {
t.Errorf("id line = %q, want %d", f.ID, id)
}
if int64(m["message_id"].(float64)) != id {
t.Errorf("message_id = %v, want %d", m["message_id"], id)
}
for k, v := range tt.want {
if m[k] != v {
t.Errorf("%s = %v, want %v", k, m[k], v)
}
}
for _, k := range tt.absent {
if _, ok := m[k]; ok {
t.Errorf("field %q must not be present: %s", k, f.Data)
}
}
if strings.Contains(f.Data, secret) {
t.Errorf("event leaks body: %s", f.Data)
}
// Sender must not get its own message: send a canary to alice
// and make sure it is the next thing she sees.
canary := env.dm("bob", "alice", "canary", "")
f2, _ := alice.nextMessage()
if f2.ID != strconv.FormatInt(canary, 10) {
t.Errorf("alice's next event id = %s, want canary %d (own message leaked?)", f2.ID, canary)
}
})
}
}
func TestAgentEvents_Isolation(t *testing.T) {
env := newAgentEventsEnv(t, "alice", "bob", "carol")
env.makeChannel("private-ish", "bob", "carol") // alice is NOT a member
env.makeChannel("shared", "alice", "bob")
alice := env.connect(env.keys["alice"], "")
alice.expectConnected("alice")
// None of these may reach alice.
env.dm("bob", "carol", "b->c", "") // DM between others
env.post("bob", "private-ish", "not for you") // channel alice has not joined
env.dm("alice", "bob", "alice->bob", "") // her own message
// Canary she is allowed to see; it must be the very next event.
canary := env.post("bob", "shared", "visible")
f, m := alice.nextMessage()
if f.ID != strconv.FormatInt(canary, 10) || m["channel"] != "shared" {
t.Fatalf("alice got %+v, want only the canary in 'shared'", f)
}
}
func TestAgentEvents_Resume(t *testing.T) {
t.Run("replays visible newer messages once, in order", func(t *testing.T) {
env := newAgentEventsEnv(t, "alice", "bob", "carol")
env.makeChannel("room", "alice", "bob")
env.makeChannel("other", "bob", "carol")
first := env.dm("bob", "alice", "old", "") // client already has this
m2 := env.dm("bob", "alice", "missed dm", "s")
env.dm("bob", "carol", "not alice's", "")
env.post("bob", "other", "not a member")
m5 := env.post("bob", "room", "missed channel")
env.dm("alice", "bob", "own", "") // own message excluded
c := env.connect(env.keys["alice"], strconv.FormatInt(first, 10))
c.expectConnected("alice")
f, m := c.nextMessage()
if f.ID != strconv.FormatInt(m2, 10) || m["subject"] != "s" {
t.Fatalf("first replay = %+v", f)
}
f, m = c.nextMessage()
if f.ID != strconv.FormatInt(m5, 10) || m["channel"] != "room" {
t.Fatalf("second replay = %+v", f)
}
// Live message after replay arrives exactly once.
live := env.dm("bob", "alice", "live", "")
f, _ = c.nextMessage()
if f.ID != strconv.FormatInt(live, 10) {
t.Fatalf("live event id = %s, want %d (duplicate or missing?)", f.ID, live)
}
})
t.Run("exactly 200 are replayed", func(t *testing.T) {
env := newAgentEventsEnv(t, "alice", "bob")
var ids []int64
for i := 0; i < maxAgentBacklog; i++ {
ids = append(ids, env.dm("bob", "alice", "x", ""))
}
c := env.connect(env.keys["alice"], "0")
c.expectConnected("alice")
for i, id := range ids {
f, _ := c.nextMessage()
if f.ID != strconv.FormatInt(id, 10) {
t.Fatalf("replay #%d id = %s, want %d", i, f.ID, id)
}
}
})
t.Run("more than 200 sends resync_required only", func(t *testing.T) {
env := newAgentEventsEnv(t, "alice", "bob")
for i := 0; i < maxAgentBacklog+1; i++ {
env.dm("bob", "alice", "x", "")
}
c := env.connect(env.keys["alice"], "0")
c.expectConnected("alice")
f := c.next()
if f.Event != "resync_required" {
t.Fatalf("event = %q, want resync_required", f.Event)
}
// No replay; the next event is the live one.
live := env.dm("bob", "alice", "live", "")
lf, _ := c.nextMessage()
if lf.ID != strconv.FormatInt(live, 10) {
t.Fatalf("after resync got id %s, want live %d", lf.ID, live)
}
})
t.Run("invalid Last-Event-ID is rejected", func(t *testing.T) {
env := newAgentEventsEnv(t, "alice")
c := env.connect(env.keys["alice"], "abc")
if c.status != http.StatusBadRequest {
t.Fatalf("status = %d, want 400", c.status)
}
})
}
func TestAgentEvents_ConnectionLimit(t *testing.T) {
env := newAgentEventsEnv(t, "alice", "bob")
var conns []*sseClient
for i := 0; i < maxAgentConnections; i++ {
c := env.connect(env.keys["alice"], "")
if c.status != http.StatusOK {
t.Fatalf("connection %d status = %d, want 200", i+1, c.status)
}
c.expectConnected("alice")
conns = append(conns, c)
}
over := env.connect(env.keys["alice"], "")
if over.status != http.StatusTooManyRequests {
t.Fatalf("6th connection status = %d, want 429", over.status)
}
var errBody map[string]string
if err := json.Unmarshal([]byte(over.body), &errBody); err != nil {
t.Fatalf("429 body is not valid JSON: %q: %v", over.body, err)
}
if errBody["error"] != "too_many_connections" || !strings.Contains(errBody["message"], "alice") {
t.Errorf("429 body = %v, want error=too_many_connections naming the agent", errBody)
}
// The limit is per agent.
other := env.connect(env.keys["bob"], "")
if other.status != http.StatusOK {
t.Fatalf("other agent status = %d, want 200", other.status)
}
// Closing a connection frees a slot.
conns[0].cancel()
deadline := time.Now().Add(5 * time.Second)
for {
c := env.connect(env.keys["alice"], "")
if c.status == http.StatusOK {
break
}
if time.Now().After(deadline) {
t.Fatal("slot was not released after disconnect")
}
time.Sleep(20 * time.Millisecond)
}
}
func TestAgentEvents_SlowClientDroppedWithoutBlocking(t *testing.T) {
hub := NewSSEHub()
sub := hub.addAgentSub("alice")
if sub == nil {
t.Fatal("addAgentSub returned nil")
}
done := make(chan struct{})
go func() {
defer close(done)
for i := 1; i <= agentSubBuffer+10; i++ { // nobody reads sub.ch
hub.BroadcastAgentMessage([]string{"alice"}, AgentMessageEvent{MessageID: int64(i)})
}
}()
select {
case <-done:
case <-time.After(2 * time.Second):
t.Fatal("broadcast blocked on a slow client")
}
hub.mu.RLock()
n := len(hub.agentSubs["alice"])
hub.mu.RUnlock()
if n != 0 {
t.Fatalf("slow subscription still registered (%d)", n)
}
}
func TestAgentEvents_Heartbeat(t *testing.T) {
env := newAgentEventsEnv(t, "alice")
env.hub.heartbeat = 50 * time.Millisecond
c := env.connect(env.keys["alice"], "")
c.expectConnected("alice")
if f := c.next(); f.Event != "heartbeat" {
t.Fatalf("event = %q, want heartbeat", f.Event)
}
}
func TestAgentEvents_OutOfOrderLiveEventsNotDropped(t *testing.T) {
env := newAgentEventsEnv(t, "alice")
c := env.connect(env.keys["alice"], "")
c.expectConnected("alice")
// Concurrent senders can broadcast ids out of order; both must arrive.
env.hub.BroadcastAgentMessage([]string{"alice"}, AgentMessageEvent{MessageID: 11, FromAgent: "x", ToAgent: "alice"})
env.hub.BroadcastAgentMessage([]string{"alice"}, AgentMessageEvent{MessageID: 10, FromAgent: "y", ToAgent: "alice"})
for _, want := range []string{"11", "10"} {
if f, _ := c.nextMessage(); f.ID != want {
t.Fatalf("got id %s, want %s", f.ID, want)
}
}
}
func TestAgentEvents_OutOfOrderLiveAfterReplay(t *testing.T) {
env := newAgentEventsEnv(t, "alice", "bob")
first := env.dm("bob", "alice", "old", "")
missed := env.dm("bob", "alice", "missed", "")
c := env.connect(env.keys["alice"], strconv.FormatInt(first, 10))
c.expectConnected("alice")
if f, _ := c.nextMessage(); f.ID != strconv.FormatInt(missed, 10) {
t.Fatalf("replay id = %s, want %d", f.ID, missed)
}
// A duplicate of the replayed message is skipped; later out-of-order ids pass.
env.hub.BroadcastAgentMessage([]string{"alice"}, AgentMessageEvent{MessageID: missed})
env.hub.BroadcastAgentMessage([]string{"alice"}, AgentMessageEvent{MessageID: missed + 20})
env.hub.BroadcastAgentMessage([]string{"alice"}, AgentMessageEvent{MessageID: missed + 10})
for _, want := range []int64{missed + 20, missed + 10} {
if f, _ := c.nextMessage(); f.ID != strconv.FormatInt(want, 10) {
t.Fatalf("got id %s, want %d", f.ID, want)
}
}
}
func TestSSEHub_HasAgentSubs(t *testing.T) {
hub := NewSSEHub()
if hub.hasAgentSubs() {
t.Fatal("empty hub reports subscriptions")
}
sub := hub.addAgentSub("alice")
if !hub.hasAgentSubs() {
t.Fatal("hub with a subscription reports none")
}
hub.removeAgentSub("alice", sub)
if hub.hasAgentSubs() {
t.Fatal("hub still reports subscriptions after removal")
}
}
+2
View File
@@ -21,6 +21,8 @@ type AnalyticsHandler struct {
}
// NewAnalyticsHandler creates a new analytics handler.
// db should be the read pool — every query in this handler is read-only and
// must not block on the serialized write pool.
func NewAnalyticsHandler(db *sql.DB, agentService *agents.AgentService, channelService *channels.Service) *AnalyticsHandler {
return &AnalyticsHandler{
db: db,
+56
View File
@@ -35,6 +35,7 @@ type SSEBroadcaster struct {
hub *SSEHub
agentService *agents.AgentService
channelService *channels.Service
msgService *messaging.MessagingService // optional: resolves conversation subjects
logger *slog.Logger
}
@@ -48,6 +49,12 @@ func NewSSEBroadcaster(hub *SSEHub, agentService *agents.AgentService, channelSe
}
}
// SetMessageService sets the messaging service used to resolve conversation
// subjects for agent events. Optional; without it events carry no subject.
func (b *SSEBroadcaster) SetMessageService(svc *messaging.MessagingService) {
b.msgService = svc
}
// BroadcastNewMessage sends a new_message event to the given owner.
func (b *SSEBroadcaster) BroadcastNewMessage(_ context.Context, ownerID int64, event NewMessageEvent) {
b.hub.Broadcast(ownerID, SSEEvent{
@@ -128,4 +135,53 @@ func (b *SSEBroadcaster) OnMessageSent(ctx context.Context, msg *messaging.Messa
} else {
b.BroadcastDM(ctx, event)
}
b.broadcastToAgents(ctx, msg)
}
// broadcastToAgents pushes a body-free new_message event to the per-agent
// stream (GET /api/agent-events): the DM recipient, or the members of the
// channel at send time. The sender is never notified of its own message.
func (b *SSEBroadcaster) broadcastToAgents(ctx context.Context, msg *messaging.Message) {
if !b.hub.hasAgentSubs() {
return // avoid member/subject DB lookups when nobody is listening
}
var recipients []string
if msg.ChannelID != nil {
if b.channelService == nil {
return
}
members, err := b.channelService.GetMembers(ctx, *msg.ChannelID)
if err != nil {
b.logger.Debug("could not get channel members for agent SSE broadcast",
"channel_id", *msg.ChannelID, "error", err)
return
}
for _, m := range members {
if m.AgentName != msg.FromAgent {
recipients = append(recipients, m.AgentName)
}
}
} else if msg.ToAgent != "" && msg.ToAgent != msg.FromAgent {
recipients = []string{msg.ToAgent}
}
if len(recipients) == 0 {
return
}
ev := AgentMessageEvent{
MessageID: msg.ID,
FromAgent: msg.FromAgent,
ToAgent: msg.ToAgent,
}
if msg.ChannelID != nil {
ev.FromAgent = ""
if ch, err := b.channelService.GetChannel(ctx, *msg.ChannelID); err == nil {
ev.Channel = ch.Name
}
}
if b.msgService != nil {
ev.Subject = b.msgService.GetConversationSubject(ctx, msg.ConversationID)
}
b.hub.BroadcastAgentMessage(recipients, ev)
}
+175
View File
@@ -0,0 +1,175 @@
// REST endpoints for per-(owner, agent) core memory (feature 020 — US2).
// Surfaces the underlying messaging.CoreMemoryStore to the Web UI under
// `/api/owner/{ownerID}/agents/{agentName}/core-memory`.
//
// Auth: every handler enforces that the session-bound owner matches the
// path's `ownerID`. Cross-owner access yields 403 to avoid leaking the
// existence of another owner's resources.
package api
import (
"encoding/json"
"errors"
"fmt"
"log/slog"
"net/http"
"strconv"
"github.com/go-chi/chi/v5"
"github.com/synapbus/synapbus/internal/messaging"
)
// MemoryCoreHandler exposes GET/PUT/DELETE for the `memory_core` table.
type MemoryCoreHandler struct {
store *messaging.CoreMemoryStore
logger *slog.Logger
}
// NewMemoryCoreHandler wires the handler. `store` must be non-nil.
func NewMemoryCoreHandler(store *messaging.CoreMemoryStore) *MemoryCoreHandler {
return &MemoryCoreHandler{
store: store,
logger: slog.Default().With("component", "api.memory-core"),
}
}
// authorize resolves the URL `ownerID` and confirms it matches the
// session-bound owner. Returns the resolved owner string (matching the
// memory_core.owner_id TEXT format) plus the agent name on success.
func (h *MemoryCoreHandler) authorize(w http.ResponseWriter, r *http.Request) (ownerStr string, agentName string, ok bool) {
sessionOwnerID, found := OwnerIDFromContext(r.Context())
if !found {
writeJSON(w, http.StatusUnauthorized, errorBody("unauthorized", "Authentication required"))
return "", "", false
}
pathOwner := chi.URLParam(r, "ownerID")
if pathOwner == "" {
writeJSON(w, http.StatusBadRequest, errorBody("bad_request", "ownerID is required"))
return "", "", false
}
pathOwnerID, err := strconv.ParseInt(pathOwner, 10, 64)
if err != nil || pathOwnerID <= 0 {
writeJSON(w, http.StatusBadRequest, errorBody("bad_request", "ownerID must be a positive integer"))
return "", "", false
}
if pathOwnerID != sessionOwnerID {
// Use 403, not 404, so the response shape matches other owner-scoped
// handlers in this package (see agents_handler.GetAgent).
writeJSON(w, http.StatusForbidden, errorBody("forbidden", "You do not have access to this owner"))
return "", "", false
}
agentName = chi.URLParam(r, "agentName")
if agentName == "" {
writeJSON(w, http.StatusBadRequest, errorBody("bad_request", "agentName is required"))
return "", "", false
}
return strconv.FormatInt(pathOwnerID, 10), agentName, true
}
// Get handles GET /api/owner/{ownerID}/agents/{agentName}/core-memory.
func (h *MemoryCoreHandler) Get(w http.ResponseWriter, r *http.Request) {
if h.store == nil {
writeJSON(w, http.StatusServiceUnavailable, errorBody("unavailable", "core memory store not configured"))
return
}
ownerStr, agentName, ok := h.authorize(w, r)
if !ok {
return
}
blob, updatedAt, exists, err := h.store.Get(r.Context(), ownerStr, agentName)
if err != nil {
h.logger.Error("memory_core get failed", "error", err, "owner", ownerStr, "agent", agentName)
writeJSON(w, http.StatusInternalServerError, errorBody("server_error", "Failed to read core memory"))
return
}
if !exists {
writeJSON(w, http.StatusNotFound, errorBody("not_found", "core memory not set for this agent"))
return
}
writeJSON(w, http.StatusOK, map[string]any{
"owner_id": ownerStr,
"agent_name": agentName,
"blob": blob,
"updated_at": updatedAt.Format("2006-01-02T15:04:05Z07:00"),
})
}
// Put handles PUT /api/owner/{ownerID}/agents/{agentName}/core-memory.
// Body: {"blob": "..."}. Returns 200 on success, 413 on
// core_memory_too_large, 400 on malformed body.
func (h *MemoryCoreHandler) Put(w http.ResponseWriter, r *http.Request) {
if h.store == nil {
writeJSON(w, http.StatusServiceUnavailable, errorBody("unavailable", "core memory store not configured"))
return
}
ownerStr, agentName, ok := h.authorize(w, r)
if !ok {
return
}
var body struct {
Blob string `json:"blob"`
UpdatedBy string `json:"updated_by,omitempty"`
}
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
writeJSON(w, http.StatusBadRequest, errorBody("bad_request", "Invalid JSON body"))
return
}
updatedBy := body.UpdatedBy
if updatedBy == "" {
updatedBy = "human"
}
if err := h.store.Set(r.Context(), ownerStr, agentName, body.Blob, updatedBy); err != nil {
if errors.Is(err, messaging.ErrCoreMemoryTooLarge) {
writeJSON(w, http.StatusRequestEntityTooLarge, errorBody(
"core_memory_too_large",
fmt.Sprintf("Blob exceeds %d bytes", h.store.MaxBytes()),
))
return
}
h.logger.Error("memory_core set failed", "error", err, "owner", ownerStr, "agent", agentName)
writeJSON(w, http.StatusInternalServerError, errorBody("server_error", "Failed to write core memory"))
return
}
writeJSON(w, http.StatusOK, map[string]any{
"owner_id": ownerStr,
"agent_name": agentName,
"blob_chars": len(body.Blob),
"updated_by": updatedBy,
})
}
// Delete handles DELETE /api/owner/{ownerID}/agents/{agentName}/core-memory.
// Returns 204 when a row existed and was removed, 404 when no row was
// present at the start of the call.
func (h *MemoryCoreHandler) Delete(w http.ResponseWriter, r *http.Request) {
if h.store == nil {
writeJSON(w, http.StatusServiceUnavailable, errorBody("unavailable", "core memory store not configured"))
return
}
ownerStr, agentName, ok := h.authorize(w, r)
if !ok {
return
}
// Check existence so we can return the canonical 204 vs 404. The
// store's Delete is idempotent — it never errors on missing rows.
_, _, exists, err := h.store.Get(r.Context(), ownerStr, agentName)
if err != nil {
h.logger.Error("memory_core get-before-delete failed", "error", err, "owner", ownerStr, "agent", agentName)
writeJSON(w, http.StatusInternalServerError, errorBody("server_error", "Failed to read core memory"))
return
}
if !exists {
writeJSON(w, http.StatusNotFound, errorBody("not_found", "core memory not set for this agent"))
return
}
if err := h.store.Delete(r.Context(), ownerStr, agentName); err != nil {
h.logger.Error("memory_core delete failed", "error", err, "owner", ownerStr, "agent", agentName)
writeJSON(w, http.StatusInternalServerError, errorBody("server_error", "Failed to delete core memory"))
return
}
w.WriteHeader(http.StatusNoContent)
}
+30 -5
View File
@@ -15,9 +15,9 @@ import (
"github.com/synapbus/synapbus/internal/harness/runs"
"github.com/synapbus/synapbus/internal/k8s"
"github.com/synapbus/synapbus/internal/messaging"
"github.com/synapbus/synapbus/internal/reactor"
"github.com/synapbus/synapbus/internal/push"
"github.com/synapbus/synapbus/internal/reactions"
"github.com/synapbus/synapbus/internal/reactor"
"github.com/synapbus/synapbus/internal/trace"
"github.com/synapbus/synapbus/internal/trust"
"github.com/synapbus/synapbus/internal/webhooks"
@@ -52,8 +52,16 @@ type RouterConfig struct {
Broadcaster *SSEBroadcaster
SessionMiddleware func(http.Handler) http.Handler
DB *sql.DB
Version string
BaseURL string
// ReadDB is the read-only connection pool (query_only=ON, MaxOpenConns>1).
// Read-only API handlers must use this to avoid blocking on the serialized
// write pool. If nil, handlers fall back to DB.
ReadDB *sql.DB
Version string
BaseURL string
// CoreMemoryStore (feature 020 — US2) wires the per-(owner, agent)
// core memory REST endpoints. Nil → routes not registered.
CoreMemoryStore *messaging.CoreMemoryStore
}
// NewRouter creates a chi router with all API routes configured.
@@ -320,9 +328,14 @@ func NewRouterWithConfig(cfg RouterConfig) chi.Router {
})
}
// Analytics (authenticated, requires DB)
// Analytics (authenticated, requires DB). Reads must use the read pool so
// dashboard queries don't serialize behind long-running writers.
if cfg.DB != nil {
analyticsHandler := NewAnalyticsHandler(cfg.DB, cfg.AgentService, cfg.ChannelService)
analyticsDB := cfg.ReadDB
if analyticsDB == nil {
analyticsDB = cfg.DB
}
analyticsHandler := NewAnalyticsHandler(analyticsDB, cfg.AgentService, cfg.ChannelService)
r.Group(func(r chi.Router) {
r.Use(authMiddleware)
@@ -333,6 +346,18 @@ func NewRouterWithConfig(cfg RouterConfig) chi.Router {
})
}
// Per-(owner, agent) core memory (feature 020 — US2)
if cfg.CoreMemoryStore != nil {
coreHandler := NewMemoryCoreHandler(cfg.CoreMemoryStore)
r.Group(func(r chi.Router) {
r.Use(authMiddleware)
r.Get("/api/owner/{ownerID}/agents/{agentName}/core-memory", coreHandler.Get)
r.Put("/api/owner/{ownerID}/agents/{agentName}/core-memory", coreHandler.Put)
r.Delete("/api/owner/{ownerID}/agents/{agentName}/core-memory", coreHandler.Delete)
})
}
// Version (unauthenticated)
if cfg.Version != "" {
versionHandler := NewVersionHandler(cfg.Version)
+9 -1
View File
@@ -19,6 +19,11 @@ type SSEEvent struct {
type SSEHub struct {
mu sync.RWMutex
clients map[int64]map[chan SSEEvent]struct{} // ownerID -> set of channels
// Per-agent subscriptions for GET /api/agent-events (see agent_events.go).
agentSubs map[string]map[*agentSub]struct{}
agentBacklog AgentEventBacklog
heartbeat time.Duration
nextID int64
logger *slog.Logger
}
@@ -26,7 +31,9 @@ type SSEHub struct {
// NewSSEHub creates a new SSE hub.
func NewSSEHub() *SSEHub {
return &SSEHub{
clients: make(map[int64]map[chan SSEEvent]struct{}),
clients: make(map[int64]map[chan SSEEvent]struct{}),
agentSubs: make(map[string]map[*agentSub]struct{}),
heartbeat: 30 * time.Second,
logger: slog.Default().With("component", "api.sse"),
}
}
@@ -76,6 +83,7 @@ func (h *SSEHub) Close() {
}
delete(h.clients, ownerID)
}
h.closeAgentSubsLocked()
h.logger.Info("all SSE clients disconnected")
}
+89 -10
View File
@@ -27,6 +27,7 @@ type TaskStore interface {
// SQLiteTaskStore implements TaskStore using SQLite.
type SQLiteTaskStore struct {
db *sql.DB
readDB *sql.DB
logger *slog.Logger
}
@@ -38,6 +39,23 @@ func NewSQLiteTaskStore(db *sql.DB) *SQLiteTaskStore {
}
}
// WithReadDB attaches a read-only connection pool used for cheap pre-checks
// (e.g. ExpireTasks asks "any rows to expire?" before contending for the
// serialized write connection). Returns the receiver for chaining. If never
// called, all queries run against the write pool.
func (s *SQLiteTaskStore) WithReadDB(readDB *sql.DB) *SQLiteTaskStore {
s.readDB = readDB
return s
}
// queryDB returns the read pool if attached, otherwise the write pool.
func (s *SQLiteTaskStore) queryDB() *sql.DB {
if s.readDB != nil {
return s.readDB
}
return s.db
}
// sqliteTimeFormat is the format used for storing timestamps consistently in SQLite.
const sqliteTimeFormat = "2006-01-02 15:04:05"
@@ -264,21 +282,82 @@ func (s *SQLiteTaskStore) UpdateBidStatus(ctx context.Context, bidID int64, stat
return nil
}
// expireTasksBatchSize bounds how many tasks a single UPDATE statement
// rewrites. A small batch keeps the SQLite write transaction short, so
// the expiry worker doesn't starve under WAL contention with concurrent
// writers (e.g. message inserts, consolidator) and respects the worker's
// context deadline even if the eligible set is large.
const expireTasksBatchSize = 500
// ExpireTasks marks all open tasks past their deadline as cancelled.
//
// The work is chunked into bounded UPDATEs (LIMIT expireTasksBatchSize)
// rather than a single unbounded UPDATE, for two reasons:
//
// 1. Bounded transactions: SQLite serializes writers, so a long UPDATE
// blocks every other writer until commit. Chunking caps the lock
// window per round-trip.
// 2. Context responsiveness: the expiry worker uses a 30s context. A
// single UPDATE doesn't observe ctx between rows, so a slow query
// would always run to completion and then return ctx error. Looping
// lets us bail between batches.
//
// Performance also depends on migration 031, which adds a partial
// composite index `idx_tasks_expiry(status, deadline) WHERE status='open'
// AND deadline IS NOT NULL`. The query below is shaped to match it.
func (s *SQLiteTaskStore) ExpireTasks(ctx context.Context) (int, error) {
// Use a string-formatted timestamp for consistent SQLite comparison
now := time.Now().UTC().Format("2006-01-02 15:04:05")
result, err := s.db.ExecContext(ctx,
`UPDATE tasks SET status = ?, updated_at = CURRENT_TIMESTAMP
WHERE status = ? AND deadline IS NOT NULL AND deadline < ?`,
TaskStatusCancelled, TaskStatusOpen, now,
)
if err != nil {
return 0, fmt.Errorf("expire tasks: %w", err)
now := time.Now().UTC().Format(sqliteTimeFormat)
// Fast pre-check on the read pool: if nothing is expirable right now,
// don't queue behind the serialized write connection at all. The write
// pool has MaxOpenConns=1 in production, so an idle ExpireTasks call
// that takes the slot while a long writer holds it would block until
// the context deadline — even though the work itself is empty. The
// EXISTS query is index-resident (idx_tasks_expiry) and runs on a
// concurrent read connection.
var hasWork int
if err := s.queryDB().QueryRowContext(ctx,
`SELECT EXISTS(SELECT 1 FROM tasks
WHERE status = ? AND deadline IS NOT NULL AND deadline < ?)`,
TaskStatusOpen, now,
).Scan(&hasWork); err != nil {
return 0, fmt.Errorf("expire tasks pre-check: %w", err)
}
if hasWork == 0 {
return 0, nil
}
rowsAffected, _ := result.RowsAffected()
return int(rowsAffected), nil
total := 0
for {
if err := ctx.Err(); err != nil {
return total, fmt.Errorf("expire tasks: %w", err)
}
// SQLite's UPDATE ... LIMIT is only enabled with the
// SQLITE_ENABLE_UPDATE_DELETE_LIMIT compile flag (not on by
// default in modernc.org/sqlite). Use a subquery on ROWID
// to portably bound the batch.
result, err := s.db.ExecContext(ctx,
`UPDATE tasks SET status = ?, updated_at = CURRENT_TIMESTAMP
WHERE rowid IN (
SELECT rowid FROM tasks
WHERE status = ? AND deadline IS NOT NULL AND deadline < ?
LIMIT ?
)`,
TaskStatusCancelled, TaskStatusOpen, now, expireTasksBatchSize,
)
if err != nil {
return total, fmt.Errorf("expire tasks: %w", err)
}
rowsAffected, _ := result.RowsAffected()
total += int(rowsAffected)
if rowsAffected < int64(expireTasksBatchSize) {
// Last (possibly empty) batch — nothing more to expire.
return total, nil
}
}
}
// CancelTasksByChannel cancels all open tasks for a channel (used before channel deletion).
+205
View File
@@ -2,7 +2,9 @@ package channels
import (
"context"
"database/sql"
"encoding/json"
"fmt"
"testing"
"time"
)
@@ -326,6 +328,209 @@ func TestSQLiteTaskStore_ExpireTasks(t *testing.T) {
}
}
// TestSQLiteTaskStore_ExpireTasks_LargeBatch exercises the expiry path at a
// scale comparable to a long-lived production instance: a few thousand mixed
// tasks (expired-open, future-open, no-deadline, cancelled), confirms the
// worker chunks past its internal batch size, and that the entire run fits
// inside a tight per-tick context — guarding the regression that the
// expiry-worker hit on kubic ("context deadline exceeded").
//
// The total row count is deliberately larger than expireTasksBatchSize (500)
// so the batching loop must iterate more than once.
func TestSQLiteTaskStore_ExpireTasks_LargeBatch(t *testing.T) {
taskStore, channelStore := newTestTaskStore(t)
ch := createAuctionChannel(t, channelStore)
ctx := context.Background()
const (
expiredOpen = 1200 // > 2 * batch size, forces multiple iterations
futureOpen = 400
noDeadline = 400
alreadyDone = 400
)
past := time.Now().Add(-1 * time.Hour)
future := time.Now().Add(1 * time.Hour)
for i := 0; i < expiredOpen; i++ {
if err := taskStore.CreateTask(ctx, &Task{
ChannelID: ch.ID, PostedBy: "poster-agent",
Title: "expired", Status: TaskStatusOpen,
Deadline: &past, Requirements: json.RawMessage(`{}`),
}); err != nil {
t.Fatalf("seed expired task %d: %v", i, err)
}
}
for i := 0; i < futureOpen; i++ {
taskStore.CreateTask(ctx, &Task{
ChannelID: ch.ID, PostedBy: "poster-agent",
Title: "future", Status: TaskStatusOpen,
Deadline: &future, Requirements: json.RawMessage(`{}`),
})
}
for i := 0; i < noDeadline; i++ {
taskStore.CreateTask(ctx, &Task{
ChannelID: ch.ID, PostedBy: "poster-agent",
Title: "no-deadline", Status: TaskStatusOpen,
Requirements: json.RawMessage(`{}`),
})
}
for i := 0; i < alreadyDone; i++ {
taskStore.CreateTask(ctx, &Task{
ChannelID: ch.ID, PostedBy: "poster-agent",
Title: "done", Status: TaskStatusCompleted,
Deadline: &past, Requirements: json.RawMessage(`{}`),
})
}
// Run with the same shape of context budget the worker uses, but tighter
// (5s) so a regression to the unbounded-scan plan would fail this test
// well before the worker's real 30s ceiling.
tightCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
start := time.Now()
count, err := taskStore.ExpireTasks(tightCtx)
elapsed := time.Since(start)
if err != nil {
t.Fatalf("ExpireTasks: %v (elapsed=%s)", err, elapsed)
}
if count != expiredOpen {
t.Errorf("expired count = %d, want %d", count, expiredOpen)
}
t.Logf("expired %d tasks in %s (over %d total rows)",
count, elapsed, expiredOpen+futureOpen+noDeadline+alreadyDone)
// Future / no-deadline tasks must remain open.
openTasks, _ := taskStore.ListTasks(ctx, ch.ID, TaskStatusOpen)
if len(openTasks) != futureOpen+noDeadline {
t.Errorf("open after expiry = %d, want %d",
len(openTasks), futureOpen+noDeadline)
}
// Second invocation on a clean set must be cheap and return 0.
count2, err := taskStore.ExpireTasks(ctx)
if err != nil {
t.Fatalf("ExpireTasks (idempotent run): %v", err)
}
if count2 != 0 {
t.Errorf("idempotent run expired = %d, want 0", count2)
}
}
// TestSQLiteTaskStore_ExpireTasks_UsesIndex confirms the partial composite
// index from migration 031 is the plan the query optimizer picks. If a
// future change drops the index or rewrites the query incompatibly, the
// planner will fall back to a SCAN and this test will fail loudly.
func TestSQLiteTaskStore_ExpireTasks_UsesIndex(t *testing.T) {
taskStore, _ := newTestTaskStore(t)
// Seed a few rows so the planner has stats to work with.
// (SQLite's planner is mostly schema-driven, but better safe.)
ctx := context.Background()
now := time.Now().UTC().Format(sqliteTimeFormat)
rows, err := taskStore.db.QueryContext(ctx,
`EXPLAIN QUERY PLAN
UPDATE tasks SET status = ?, updated_at = CURRENT_TIMESTAMP
WHERE rowid IN (
SELECT rowid FROM tasks
WHERE status = ? AND deadline IS NOT NULL AND deadline < ?
LIMIT ?
)`,
TaskStatusCancelled, TaskStatusOpen, now, expireTasksBatchSize,
)
if err != nil {
t.Fatalf("EXPLAIN QUERY PLAN: %v", err)
}
defer rows.Close()
sawIndex := false
var plan []string
for rows.Next() {
var id, parent, notused int
var detail string
if err := rows.Scan(&id, &parent, &notused, &detail); err != nil {
t.Fatalf("scan plan row: %v", err)
}
plan = append(plan, detail)
if contains(detail, "idx_tasks_expiry") {
sawIndex = true
}
}
if !sawIndex {
t.Errorf("expected query plan to use idx_tasks_expiry, got: %v", plan)
}
}
// TestSQLiteTaskStore_ExpireTasks_EmptyShortCircuits verifies that when the
// pre-check finds no expirable rows, ExpireTasks returns (0, nil) without
// running the UPDATE loop. In production this is the steady-state case
// (tasks is empty most of the time) and the pre-check is what keeps the
// worker off the serialized write connection.
func TestSQLiteTaskStore_ExpireTasks_EmptyShortCircuits(t *testing.T) {
taskStore, _ := newTestTaskStore(t)
ctx := context.Background()
count, err := taskStore.ExpireTasks(ctx)
if err != nil {
t.Fatalf("ExpireTasks on empty table: %v", err)
}
if count != 0 {
t.Errorf("count = %d, want 0", count)
}
}
// TestSQLiteTaskStore_ExpireTasks_UsesReadPoolForPreCheck verifies that when
// a read pool is attached via WithReadDB, the EXISTS pre-check runs on it
// (and the actual UPDATE still runs on the write pool when work is present).
func TestSQLiteTaskStore_ExpireTasks_UsesReadPoolForPreCheck(t *testing.T) {
taskStore, channelStore := newTestTaskStore(t)
// Open a separate handle to the same shared-cache memory DB.
dsn := fmt.Sprintf("file:%s?mode=memory&cache=shared", t.Name())
readDB, err := sql.Open("sqlite", dsn)
if err != nil {
t.Fatalf("open read pool: %v", err)
}
t.Cleanup(func() { readDB.Close() })
taskStore.WithReadDB(readDB)
ctx := context.Background()
// Empty: short-circuit on the read pool.
if n, err := taskStore.ExpireTasks(ctx); err != nil || n != 0 {
t.Fatalf("empty ExpireTasks with read pool: count=%d err=%v", n, err)
}
// Seed one expirable task and verify the UPDATE still runs.
ch := createAuctionChannel(t, channelStore)
past := time.Now().Add(-1 * time.Hour)
if err := taskStore.CreateTask(ctx, &Task{
ChannelID: ch.ID, PostedBy: "poster-agent", Title: "Expired",
Status: TaskStatusOpen, Deadline: &past, Requirements: json.RawMessage(`{}`),
}); err != nil {
t.Fatalf("create task: %v", err)
}
n, err := taskStore.ExpireTasks(ctx)
if err != nil {
t.Fatalf("ExpireTasks with read pool: %v", err)
}
if n != 1 {
t.Errorf("expired count = %d, want 1", n)
}
}
func contains(s, sub string) bool {
for i := 0; i+len(sub) <= len(s); i++ {
if s[i:i+len(sub)] == sub {
return true
}
}
return false
}
func TestSQLiteTaskStore_CancelTasksByChannel(t *testing.T) {
taskStore, channelStore := newTestTaskStore(t)
ch := createAuctionChannel(t, channelStore)
+16
View File
@@ -167,6 +167,22 @@ func (h *Harness) Execute(ctx context.Context, req *harness.ExecRequest) (*harne
// agents emitting a final result envelope). If it parses, stash it.
if rj := extractResultJSON(logs); rj != nil {
res.ResultJSON = rj
// Best-effort: pull token-usage fields when the agent emitted
// them on the final line (claude-agent-sdk / dream-runner do).
// Feeds into the dream worker's UsageGate circuit breaker so
// consumption actually counts.
var u struct {
TokensIn int64 `json:"tokens_in"`
TokensOut int64 `json:"tokens_out"`
TokensCached int64 `json:"tokens_cached"`
CostUSD float64 `json:"cost_usd"`
}
if err := json.Unmarshal(rj, &u); err == nil {
res.Usage.TokensIn = u.TokensIn
res.Usage.TokensOut = u.TokensOut
res.Usage.TokensCached = u.TokensCached
res.Usage.CostUSD = u.CostUSD
}
}
return res, nil
}
+14 -1
View File
@@ -2,11 +2,14 @@ package k8s
import (
"context"
"crypto/rand"
"encoding/hex"
"fmt"
"io"
"log/slog"
"os"
"strings"
"time"
batchv1 "k8s.io/api/batch/v1"
corev1 "k8s.io/api/core/v1"
@@ -94,7 +97,17 @@ func (r *K8sJobRunner) GetNamespace() string {
}
func (r *K8sJobRunner) CreateJob(ctx context.Context, handler *K8sHandler, msg *JobMessage) (string, error) {
jobName := sanitizeJobName(fmt.Sprintf("synapbus-%s-%d", handler.AgentName, msg.MessageID))
// When there is no triggering message (e.g. dream-worker dispatches
// where Message=nil → MessageID=0), fall back to a unique suffix so
// concurrent runs don't collide on the Job name. Format keeps the
// historic "synapbus-<agent>-<id>" prefix for log/grep continuity.
suffix := fmt.Sprintf("%d", msg.MessageID)
if msg.MessageID == 0 {
var b [4]byte
_, _ = rand.Read(b[:])
suffix = fmt.Sprintf("%d-%s", time.Now().UnixNano()%1_000_000, hex.EncodeToString(b[:]))
}
jobName := sanitizeJobName(fmt.Sprintf("synapbus-%s-%s", handler.AgentName, suffix))
namespace := handler.Namespace
if namespace == "" {
+173
View File
@@ -67,8 +67,45 @@ func NewServiceBridge(
}
}
// bridgeActionAliases maps observed wrong action names that agents have called
// to the real bridge action they probably meant. This is a small, hand-curated
// whitelist of guesses seen in production logs — not a fuzzy-match layer.
// Keep entries minimal and only add names that have a single unambiguous target.
var bridgeActionAliases = map[string]string{
// read_channel → fetch messages from a channel (requires channel_id/name)
"read_channel": "get_channel_messages",
// search → unified search over messages
"search": "search_messages",
// read_dm → DMs land in the inbox
"read_dm": "read_inbox",
// my_status → no exact bridge equivalent; agents probing for "what's new"
// most often want read_inbox. (The top-level MCP `my_status` tool is the
// canonical way; this alias keeps call() from failing.)
"my_status": "read_inbox",
// read_article → wiki retrieval
"read_article": "get_article",
}
// bridgeTopLevelOnly lists action names that exist as top-level MCP tools
// (registered on the MCP server), not as call() bridge actions. Agents that
// invoke these via call() / execute() get a targeted error pointing them at
// the real tool rather than a generic "unknown action". Keyed by what the
// agent typed; value is the canonical top-level tool name.
var bridgeTopLevelOnly = map[string]string{
"rewrite_core_memory": "memory_rewrite_core",
"memory_rewrite_core": "memory_rewrite_core",
}
// Call dispatches an action by name to the appropriate service method.
func (b *ServiceBridge) Call(ctx context.Context, actionName string, args map[string]any) (any, error) {
// Apply aliases for common wrong-name guesses observed in agent logs.
if real, ok := bridgeActionAliases[actionName]; ok {
actionName = real
}
// Catch wrong guesses that map to top-level MCP tools (not bridge actions).
if real, ok := bridgeTopLevelOnly[actionName]; ok {
return nil, fmt.Errorf("'%s' is a top-level MCP tool, not a call() action; invoke it as a tool directly (real name: %s)", actionName, real)
}
switch actionName {
// --- Messaging ---
case "read_inbox":
@@ -173,10 +210,146 @@ func (b *ServiceBridge) Call(ctx context.Context, actionName string, args map[st
return b.callQueryReputation(ctx, args)
default:
if suggestion := suggestBridgeAction(actionName); suggestion != "" {
return nil, fmt.Errorf("unknown action: %s (did you mean: %s)", actionName, suggestion)
}
return nil, fmt.Errorf("unknown action: %s", actionName)
}
}
// knownBridgeActions enumerates every action name routable through Call().
// Kept in sync with the switch in Call() by hand — there are not many. Used
// only to power "did you mean" suggestions when an unknown action arrives.
var knownBridgeActions = []string{
// Messaging
"read_inbox", "claim_messages", "mark_done", "search_messages",
"discover_agents", "send_message",
// Channels
"create_channel", "join_channel", "leave_channel", "list_channels",
"invite_to_channel", "kick_from_channel", "get_channel_messages",
"send_channel_message", "update_channel",
// Swarm
"post_task", "bid_task", "accept_bid", "complete_task", "list_tasks",
// Attachments
"upload_attachment", "download_attachment",
// Reactions
"react", "unreact", "get_reactions", "list_by_state",
// Threads
"get_replies",
// Trust
"get_trust",
// SQL Query
"query",
// Wiki
"create_article", "get_article", "update_article", "list_articles",
"get_backlinks",
// Marketplace
"post_auction", "bid", "award", "mark_task_done", "read_skill_card",
"query_reputation",
}
// suggestBridgeAction returns the closest known action name to `name`, or ""
// if nothing is plausibly close. Strategy: cheap prefix/substring check first,
// then Levenshtein within distance 3 — but Levenshtein matches must share the
// leading verb (the token before the first underscore) with the candidate.
// Without that constraint, e.g. `read_message` is two edits from `send_message`
// and gets suggested, which is the opposite intent and actively misleading.
// Distance threshold scales with the length of the input so very short strings
// don't false-match.
func suggestBridgeAction(name string) string {
if name == "" {
return ""
}
lower := strings.ToLower(name)
// 1. Substring match — common when agents drop or add a prefix/suffix.
for _, candidate := range knownBridgeActions {
if strings.Contains(candidate, lower) || strings.Contains(lower, candidate) {
return candidate
}
}
// 2. Levenshtein on full names, gated on a matching leading verb so
// `read_message` is not "fixed" by suggesting `send_message`.
threshold := 3
if len(name) <= 6 {
threshold = 2
}
inputVerb := leadingVerb(lower)
bestDist := threshold + 1
best := ""
for _, candidate := range knownBridgeActions {
if leadingVerb(candidate) != inputVerb {
continue
}
d := levenshtein(lower, candidate)
if d < bestDist {
bestDist = d
best = candidate
}
}
if bestDist <= threshold {
return best
}
return ""
}
// leadingVerb returns the substring before the first underscore, or the
// whole string if there is no underscore. Used to gate Levenshtein
// suggestions so a shared suffix (like `_message`) doesn't pair `read_…`
// with `send_…`.
func leadingVerb(s string) string {
if i := strings.IndexByte(s, '_'); i >= 0 {
return s[:i]
}
return s
}
// levenshtein computes the Levenshtein edit distance between a and b using
// a single rolling row of O(min(len)) space. Pure Go, no deps.
func levenshtein(a, b string) int {
if a == b {
return 0
}
if len(a) == 0 {
return len(b)
}
if len(b) == 0 {
return len(a)
}
// Ensure a is the shorter — minimises row width.
if len(a) > len(b) {
a, b = b, a
}
prev := make([]int, len(a)+1)
curr := make([]int, len(a)+1)
for i := 0; i <= len(a); i++ {
prev[i] = i
}
for j := 1; j <= len(b); j++ {
curr[0] = j
for i := 1; i <= len(a); i++ {
cost := 1
if a[i-1] == b[j-1] {
cost = 0
}
del := prev[i] + 1
ins := curr[i-1] + 1
sub := prev[i-1] + cost
m := del
if ins < m {
m = ins
}
if sub < m {
m = sub
}
curr[i] = m
}
prev, curr = curr, prev
}
return prev[len(a)]
}
// --- Messaging implementations ---
func (b *ServiceBridge) callSendMessage(ctx context.Context, args map[string]any) (any, error) {
+211
View File
@@ -3,6 +3,7 @@ package mcp
import (
"context"
"log/slog"
"strings"
"testing"
_ "modernc.org/sqlite"
@@ -486,3 +487,213 @@ func TestBridge_React_Toggle_Removes_WorkflowState(t *testing.T) {
}
var _ = storage.RunMigrations
// TestBridge_ActionAliases verifies that each observed wrong action name from
// production logs resolves to the correct real action. We don't validate the
// full result payload — only that the call succeeds (i.e. dispatch reached the
// real handler) rather than returning "unknown action".
func TestBridge_ActionAliases(t *testing.T) {
tests := []struct {
alias string
realName string
args map[string]any
setupCh string // optional: channel name to create+join before the call
expectErr string // optional: substring of expected error (when call reaches real handler but fails for unrelated reasons)
}{
{
alias: "search",
realName: "search_messages",
args: map[string]any{"query": "anything"},
},
{
alias: "my_status",
realName: "read_inbox",
args: map[string]any{"limit": 5},
},
{
alias: "read_dm",
realName: "read_inbox",
args: map[string]any{"limit": 5},
},
{
alias: "read_channel",
realName: "get_channel_messages",
args: map[string]any{"channel_name": "alias-ch"},
setupCh: "alias-ch",
},
{
alias: "read_article",
realName: "get_article",
args: map[string]any{"slug": "anything"},
expectErr: "wiki not available", // bridge has no wikiService → dispatch reached real handler
},
}
for _, tt := range tests {
t.Run(tt.alias, func(t *testing.T) {
bridge, _, _, channelService := newTestBridge(t)
ctx := context.Background()
if tt.setupCh != "" {
ch, err := channelService.CreateChannel(ctx, channels.CreateChannelRequest{
Name: tt.setupCh, Type: "standard", CreatedBy: "agent-a",
})
if err != nil {
t.Fatalf("create channel: %v", err)
}
if err := channelService.JoinChannel(ctx, ch.ID, "agent-a"); err != nil {
t.Fatalf("join channel: %v", err)
}
}
_, err := bridge.Call(ctx, tt.alias, tt.args)
if tt.expectErr != "" {
if err == nil {
t.Fatalf("expected error containing %q, got nil", tt.expectErr)
}
if !strings.Contains(err.Error(), tt.expectErr) {
t.Fatalf("error %q does not contain %q", err.Error(), tt.expectErr)
}
if strings.Contains(err.Error(), "unknown action") {
t.Fatalf("alias %q should have been resolved, got unknown-action error: %v", tt.alias, err)
}
return
}
if err != nil {
t.Fatalf("alias %q (real=%s) failed: %v", tt.alias, tt.realName, err)
}
})
}
}
// TestBridge_TopLevelToolHint verifies that wrong-guess names which map to
// top-level MCP tools (not bridge actions) produce a targeted hint pointing
// at the real tool, instead of a generic "unknown action" error.
func TestBridge_TopLevelToolHint(t *testing.T) {
bridge, _, _, _ := newTestBridge(t)
ctx := context.Background()
// The misspelled wrong guess hits the hint.
_, err := bridge.Call(ctx, "rewrite_core_memory", map[string]any{})
if err == nil {
t.Fatal("expected error for top-level-tool wrong-guess")
}
msg := err.Error()
if !strings.Contains(msg, "top-level MCP tool") {
t.Errorf("error should mention top-level MCP tool, got: %v", err)
}
if !strings.Contains(msg, "memory_rewrite_core") {
t.Errorf("error should name the real tool memory_rewrite_core, got: %v", err)
}
// The correct top-level tool name, when called via the bridge, also
// returns the hint instead of plain "unknown action". (Agents learn the
// real tool name from the first hint and then retry via call() — the
// hint must catch both spellings.)
_, err = bridge.Call(ctx, "memory_rewrite_core", map[string]any{})
if err == nil {
t.Fatal("expected error for memory_rewrite_core via bridge")
}
msg = err.Error()
if !strings.Contains(msg, "top-level MCP tool") {
t.Errorf("memory_rewrite_core via bridge should hint top-level MCP tool, got: %v", err)
}
}
// TestBridge_UnknownAction_Suggestion verifies that an unknown action that is
// close to a real one (Levenshtein-wise) returns a "did you mean" suggestion.
func TestBridge_UnknownAction_Suggestion(t *testing.T) {
bridge, _, _, _ := newTestBridge(t)
ctx := context.Background()
tests := []struct {
input string
wantContains string
}{
// substring hit: "send" is contained in "send_message"
{input: "send", wantContains: "send_message"},
// Levenshtein within a shared verb: "list_channel" → "list_channels" (distance 1)
{input: "list_channel", wantContains: "list_channels"},
// Levenshtein within a shared verb: "get_channel_message" → "get_channel_messages"
{input: "get_channel_message", wantContains: "get_channel_messages"},
}
for _, tt := range tests {
t.Run(tt.input, func(t *testing.T) {
_, err := bridge.Call(ctx, tt.input, map[string]any{})
if err == nil {
t.Fatalf("expected error for %q", tt.input)
}
if !strings.Contains(err.Error(), "did you mean") {
t.Errorf("error should contain 'did you mean', got: %v", err)
}
if !strings.Contains(err.Error(), tt.wantContains) {
t.Errorf("error should suggest %q, got: %v", tt.wantContains, err)
}
})
}
}
// TestBridge_UnknownAction_NoSuggestion verifies that a truly distant unknown
// action returns the plain "unknown action" error without a misleading
// suggestion.
func TestBridge_UnknownAction_NoSuggestion(t *testing.T) {
bridge, _, _, _ := newTestBridge(t)
ctx := context.Background()
_, err := bridge.Call(ctx, "xyzzy_quux_frobnicate", map[string]any{})
if err == nil {
t.Fatal("expected error")
}
if strings.Contains(err.Error(), "did you mean") {
t.Errorf("distant action should not get a suggestion, got: %v", err)
}
}
// TestBridge_UnknownAction_NoCrossVerbSuggestion guards against the suggester
// pairing actions that share a suffix but have opposite intent — e.g.
// `read_message` is two edits from `send_message`, but suggesting "send" to an
// agent that asked to read is actively misleading. The leading-verb gate in
// suggestBridgeAction must prevent this.
func TestBridge_UnknownAction_NoCrossVerbSuggestion(t *testing.T) {
bridge, _, _, _ := newTestBridge(t)
ctx := context.Background()
cases := []string{
"read_message", // would have suggested send_message (distance 2)
"delete_message", // would have suggested send_message (distance 3)
"fetch_channel", // unrelated verb; must not suggest send/list/get
}
for _, in := range cases {
t.Run(in, func(t *testing.T) {
_, err := bridge.Call(ctx, in, map[string]any{})
if err == nil {
t.Fatalf("expected error for %q", in)
}
if strings.Contains(err.Error(), "did you mean") {
t.Errorf("cross-verb suggestion leaked for %q: %v", in, err)
}
})
}
}
func TestLevenshtein(t *testing.T) {
tests := []struct {
a, b string
want int
}{
{"", "", 0},
{"abc", "abc", 0},
{"", "abc", 3},
{"abc", "", 3},
{"kitten", "sitting", 3},
{"send", "sned", 2},
{"list_channel", "list_channels", 1},
}
for _, tt := range tests {
got := levenshtein(tt.a, tt.b)
if got != tt.want {
t.Errorf("levenshtein(%q, %q) = %d, want %d", tt.a, tt.b, got, tt.want)
}
}
}
+8 -82
View File
@@ -52,18 +52,21 @@ func NewGoalsToolRegistrar(
}
}
// RegisterAllOnServer attaches create_goal, propose_task_tree,
// propose_agent, claim_task, request_resource, list_resources, and
// complete_goal to the MCP server.
// RegisterAllOnServer attaches create_goal, propose_task_tree, claim_task,
// request_resource, list_resources, and complete_goal to the MCP server.
//
// propose_agent was removed when SynapBus moved to internal-only mode: it
// wrote a pending row to agent_proposals for human approval via #approvals,
// and that approval surface no longer exists. Agents are created via the
// admin CLI directly. See migration 027_remove_approval_noise.sql.
func (r *GoalsToolRegistrar) RegisterAllOnServer(s *server.MCPServer) {
s.AddTool(r.createGoalTool(), r.handleCreateGoal)
s.AddTool(r.proposeTaskTreeTool(), r.handleProposeTaskTree)
s.AddTool(r.proposeAgentTool(), r.handleProposeAgent)
s.AddTool(r.claimTaskTool(), r.handleClaimTask)
s.AddTool(r.requestResourceTool(), r.handleRequestResource)
s.AddTool(r.listResourcesTool(), r.handleListResources)
s.AddTool(r.completeGoalTool(), r.handleCompleteGoal)
r.logger.Info("spec-018 MCP tools registered", "count", 7)
r.logger.Info("spec-018 MCP tools registered", "count", 6)
}
// --- Tool Definitions ---
@@ -87,18 +90,6 @@ func (r *GoalsToolRegistrar) proposeTaskTreeTool() mcplib.Tool {
)
}
func (r *GoalsToolRegistrar) proposeAgentTool() mcplib.Tool {
return mcplib.NewTool("propose_agent",
mcplib.WithDescription("Propose creating a new specialist agent. Writes an agent_proposals row so a human can approve it via the #approvals channel. Returns the proposal id."),
mcplib.WithString("name", mcplib.Description("Desired agent name"), mcplib.Required()),
mcplib.WithString("display_name", mcplib.Description("Human-readable display name")),
mcplib.WithString("system_prompt", mcplib.Description("System prompt for the spawned agent"), mcplib.Required()),
mcplib.WithString("tool_scope", mcplib.Description("Comma-separated scope (e.g. 'messages:read,messages:send')")),
mcplib.WithNumber("parent_task_id", mcplib.Description("The task this agent will work on")),
mcplib.WithString("autonomy_tier", mcplib.Description("supervised | assisted | autonomous (default assisted)")),
)
}
func (r *GoalsToolRegistrar) claimTaskTool() mcplib.Tool {
return mcplib.NewTool("claim_task",
mcplib.WithDescription("Atomically claim an approved task. Returns the claimed task or an error if another agent got it first."),
@@ -239,71 +230,6 @@ func (r *GoalsToolRegistrar) handleProposeTaskTree(ctx context.Context, req mcpl
})
}
func (r *GoalsToolRegistrar) handleProposeAgent(ctx context.Context, req mcplib.CallToolRequest) (*mcplib.CallToolResult, error) {
agentName, ok := extractAgentName(ctx)
if !ok {
return mcplib.NewToolResultError("authentication required"), nil
}
if r.db == nil {
return mcplib.NewToolResultError("db not configured"), nil
}
name := req.GetString("name", "")
if name == "" {
return mcplib.NewToolResultError("name is required"), nil
}
systemPrompt := req.GetString("system_prompt", "")
if systemPrompt == "" {
return mcplib.NewToolResultError("system_prompt is required"), nil
}
toolScope := req.GetString("tool_scope", "")
if toolScope == "" {
toolScope = "[]"
} else if !strings.HasPrefix(toolScope, "[") {
// Accept comma-separated convenience form.
parts := strings.Split(toolScope, ",")
for i := range parts {
parts[i] = `"` + strings.TrimSpace(parts[i]) + `"`
}
toolScope = "[" + strings.Join(parts, ",") + "]"
}
tier := req.GetString("autonomy_tier", "assisted")
parentTaskID := int64(req.GetInt("parent_task_id", 0))
if parentTaskID <= 0 {
return mcplib.NewToolResultError("parent_task_id is required (proposals must attach to a task)"), nil
}
agent, err := r.agents.GetAgent(ctx, agentName)
if err != nil {
return mcplib.NewToolResultError(fmt.Sprintf("resolve caller: %s", err)), nil
}
// Resolve goal_id from the parent task.
var goalID int64
if err := r.db.QueryRowContext(ctx,
`SELECT goal_id FROM goal_tasks WHERE id=?`, parentTaskID).Scan(&goalID); err != nil {
return mcplib.NewToolResultError(fmt.Sprintf("resolve goal from task %d: %s", parentTaskID, err)), nil
}
res, err := r.db.ExecContext(ctx, `
INSERT INTO agent_proposals (
proposer_agent_id, goal_id, parent_task_id, proposed_name,
proposed_model, proposed_system_prompt, proposed_tool_scope_json,
proposed_autonomy_tier, status
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, 'pending')`,
agent.ID, goalID, parentTaskID, name,
"gemini-2.5-flash", systemPrompt, toolScope, tier)
if err != nil {
return mcplib.NewToolResultError(fmt.Sprintf("insert proposal: %s", err)), nil
}
id, _ := res.LastInsertId()
return resultJSON(map[string]any{
"proposal_id": id,
"status": "pending",
"name": name,
})
}
func (r *GoalsToolRegistrar) handleClaimTask(ctx context.Context, req mcplib.CallToolRequest) (*mcplib.CallToolResult, error) {
agentName, ok := extractAgentName(ctx)
if !ok {
+158
View File
@@ -0,0 +1,158 @@
// Integration test for US2 → US1 wiring: a real CoreMemoryStore, wired
// through messaging.NewCoreProvider, surfaces a seeded blob in the
// wrapped tool response's `relevant_context.core_memory` field; an agent
// with no row gets no relevant_context.
package mcp
import (
"context"
"database/sql"
"encoding/json"
"fmt"
"testing"
mcplib "github.com/mark3labs/mcp-go/mcp"
_ "modernc.org/sqlite"
"github.com/synapbus/synapbus/internal/agents"
"github.com/synapbus/synapbus/internal/messaging"
"github.com/synapbus/synapbus/internal/storage"
)
func newInjectionTestDB(t *testing.T) *sql.DB {
t.Helper()
dsn := fmt.Sprintf("file:%s?mode=memory&cache=shared", t.Name())
db, err := sql.Open("sqlite", dsn)
if err != nil {
t.Fatalf("open db: %v", err)
}
t.Cleanup(func() { db.Close() })
if _, err := db.Exec("PRAGMA foreign_keys=ON"); err != nil {
t.Fatalf("foreign keys: %v", err)
}
if err := storage.RunMigrations(context.Background(), db); err != nil {
t.Fatalf("migrations: %v", err)
}
return db
}
// TestInjection_CoreMemoryWiring proves a wrapped session-start handler
// surfaces a blob seeded via CoreMemoryStore as
// `relevant_context.core_memory`. Mirrors the contract example in
// `specs/020-proactive-memory-dream-worker/contracts/mcp-injection.md`.
func TestInjection_CoreMemoryWiring(t *testing.T) {
db := newInjectionTestDB(t)
ctx := context.Background()
const owner = "42"
const agent = "a1"
const blob = "I am a1. Currently focused on memory tests."
coreStore := messaging.NewCoreMemoryStore(db, 2048)
if err := coreStore.Set(ctx, owner, agent, blob, "human"); err != nil {
t.Fatalf("seed core memory: %v", err)
}
cfg := WrapConfig{
Cfg: messaging.MemoryConfig{
InjectionEnabled: true,
InjectionBudgetTokens: 500,
InjectionMaxItems: 5,
InjectionMinScore: 0.25,
},
SearchSvc: nil, // query="" forces no retrieval — only core matters.
IncludeCore: true,
CoreProvider: messaging.NewCoreProvider(coreStore),
QuerySource: func(_ context.Context, _ string, _ map[string]any, _ map[string]any) string { return "" },
}
inner := stubHandler(map[string]any{"agent": agent})
wrapped := WrapInjection(inner, "my_status", cfg)
// Owner 42 ↔ caller a1.
callerCtx := agents.ContextWithAgent(ctx, &agents.Agent{Name: agent, OwnerID: 42})
res, err := wrapped(callerCtx, mcplib.CallToolRequest{})
if err != nil {
t.Fatalf("wrapped my_status: %v", err)
}
body := extractJSON(t, res)
rc, ok := body["relevant_context"].(map[string]any)
if !ok {
t.Fatalf("relevant_context missing: %+v", body)
}
if got := rc["core_memory"]; got != blob {
t.Errorf("core_memory: got %v want %q", got, blob)
}
}
// TestInjection_NoCoreMemoryYieldsNoPacket verifies that an agent
// without a memory_core row gets the original handler response back,
// without a `relevant_context` field appended.
func TestInjection_NoCoreMemoryYieldsNoPacket(t *testing.T) {
db := newInjectionTestDB(t)
coreStore := messaging.NewCoreMemoryStore(db, 2048)
// Intentionally NO Set — the agent has no row.
cfg := WrapConfig{
Cfg: messaging.MemoryConfig{
InjectionEnabled: true,
InjectionBudgetTokens: 500,
InjectionMaxItems: 5,
InjectionMinScore: 0.25,
},
SearchSvc: nil,
IncludeCore: true,
CoreProvider: messaging.NewCoreProvider(coreStore),
QuerySource: func(_ context.Context, _ string, _ map[string]any, _ map[string]any) string { return "" },
}
inner := stubHandler(map[string]any{"agent": "a2"})
wrapped := WrapInjection(inner, "my_status", cfg)
ctx := agents.ContextWithAgent(context.Background(), &agents.Agent{Name: "a2", OwnerID: 42})
res, err := wrapped(ctx, mcplib.CallToolRequest{})
if err != nil {
t.Fatalf("wrapped: %v", err)
}
body := extractJSON(t, res)
if _, has := body["relevant_context"]; has {
t.Errorf("relevant_context attached for agent with no core row: %v", body["relevant_context"])
}
if body["agent"] != "a2" {
t.Errorf("inner body lost: %+v", body)
}
}
// Verify ContextPacket round-trips its core_memory through json. This is
// the contract field consumed by clients.
func TestInjection_CoreMemoryJSONShape(t *testing.T) {
db := newInjectionTestDB(t)
ctx := context.Background()
coreStore := messaging.NewCoreMemoryStore(db, 2048)
if err := coreStore.Set(ctx, "1", "a1", "hello", "human"); err != nil {
t.Fatalf("seed: %v", err)
}
provider := messaging.NewCoreProvider(coreStore)
got, err := provider.Get(ctx, "1", "a1")
if err != nil {
t.Fatalf("provider.Get: %v", err)
}
if got != "hello" {
t.Errorf("provider.Get: got %q want hello", got)
}
// Empty case (no row) yields "" without error.
got, err = provider.Get(ctx, "1", "nobody")
if err != nil || got != "" {
t.Errorf("provider.Get on missing: got %q err=%v", got, err)
}
// Sanity: ensure the adapter is reachable through json marshaling of a packet.
type fakePacket struct {
Core string `json:"core_memory,omitempty"`
}
b, _ := json.Marshal(fakePacket{Core: "hello"})
if string(b) != `{"core_memory":"hello"}` {
t.Errorf("json marshaling: got %s", b)
}
}
+197
View File
@@ -0,0 +1,197 @@
package mcp
import (
"context"
"database/sql"
"encoding/json"
"testing"
mcplib "github.com/mark3labs/mcp-go/mcp"
_ "modernc.org/sqlite"
"github.com/synapbus/synapbus/internal/agents"
"github.com/synapbus/synapbus/internal/messaging"
"github.com/synapbus/synapbus/internal/search"
"github.com/synapbus/synapbus/internal/trace"
)
// TestInjection_CrossOwner_NoLeak is the SC-008 adversarial test: two
// owners (H1, H2) each have an agent + memories in #open-brain. When
// each agent invokes the same wrapped tool with the same query, their
// `relevant_context.memories` are disjoint along owner boundaries.
func TestInjection_CrossOwner_NoLeak(t *testing.T) {
db := newTestDB(t)
ctx := context.Background()
// Two human owners.
if _, err := db.Exec(
`INSERT OR IGNORE INTO users (id, username, password_hash, display_name)
VALUES (1, 'h1', 'hash', 'H1'), (2, 'h2', 'hash', 'H2')`,
); err != nil {
t.Fatalf("seed users: %v", err)
}
// One agent per owner.
if _, err := db.Exec(
`INSERT OR IGNORE INTO agents (name, display_name, type, owner_id, api_key_hash, status)
VALUES ('a-h1', 'A1', 'ai', 1, 'k1', 'active'),
('a-h2', 'A2', 'ai', 2, 'k2', 'active')`,
); err != nil {
t.Fatalf("seed agents: %v", err)
}
// Both join an open-brain channel — broadly readable.
if _, err := db.Exec(
`INSERT OR IGNORE INTO channels (id, name, description, type, created_by)
VALUES (1, 'open-brain', 'shared', 'standard', 'system')`,
); err != nil {
t.Fatalf("seed channel: %v", err)
}
if _, err := db.Exec(
`INSERT OR IGNORE INTO channel_members (channel_id, agent_name)
VALUES (1, 'a-h1'), (1, 'a-h2')`,
); err != nil {
t.Fatalf("seed members: %v", err)
}
// One memory per owner, both about the same topic.
seedMemory(t, db, 1, "a-h1", "Kuzu graph DB is in H1's research notes")
seedMemory(t, db, 1, "a-h2", "Kuzu graph DB also appears in H2's separate research")
// Stand up a real search.Service (FTS-only, no embeddings).
tracer := trace.NewTracer(db)
t.Cleanup(func() { tracer.Close() })
msgStore := messaging.NewSQLiteMessageStore(db)
msgService := messaging.NewMessagingService(msgStore, tracer)
searchSvc := search.NewService(db, nil, nil, msgService)
// Configure wrap: low score floor so the test isn't flaky on FTS.
cfg := WrapConfig{
Cfg: messaging.MemoryConfig{
InjectionEnabled: true,
InjectionBudgetTokens: 500,
InjectionMaxItems: 5,
InjectionMinScore: 0.0,
},
SearchSvc: searchSvc,
QuerySource: func(_ context.Context, _ string, _ map[string]any, _ map[string]any) string {
return "Kuzu"
},
}
wrapped := WrapInjection(stubHandler(map[string]any{"ok": true}), "search_messages", cfg)
// Call as H1.
h1Ctx := agents.ContextWithAgent(ctx, &agents.Agent{Name: "a-h1", OwnerID: 1})
h1Res, err := wrapped(h1Ctx, mcplib.CallToolRequest{})
if err != nil {
t.Fatalf("h1 wrapped: %v", err)
}
h1Body := unmarshalText(t, h1Res)
// Call as H2.
h2Ctx := agents.ContextWithAgent(ctx, &agents.Agent{Name: "a-h2", OwnerID: 2})
h2Res, err := wrapped(h2Ctx, mcplib.CallToolRequest{})
if err != nil {
t.Fatalf("h2 wrapped: %v", err)
}
h2Body := unmarshalText(t, h2Res)
h1Memories := extractMemoryFromAgents(h1Body)
h2Memories := extractMemoryFromAgents(h2Body)
for _, fromAgent := range h1Memories {
if fromAgent != "a-h1" {
t.Errorf("H1 saw memory from %q (cross-owner leak)", fromAgent)
}
}
for _, fromAgent := range h2Memories {
if fromAgent != "a-h2" {
t.Errorf("H2 saw memory from %q (cross-owner leak)", fromAgent)
}
}
// Disjoint sets: no h1 memory id may appear in h2's response.
h1IDs := extractMemoryIDs(h1Body)
h2IDs := extractMemoryIDs(h2Body)
for id := range h1IDs {
if _, dup := h2IDs[id]; dup {
t.Errorf("memory id %d leaked across owners", id)
}
}
}
func seedMemory(t *testing.T, db *sql.DB, channelID int64, fromAgent, body string) {
t.Helper()
convRes, err := db.Exec(
`INSERT INTO conversations (created_by, channel_id) VALUES (?, ?)`,
fromAgent, channelID,
)
if err != nil {
t.Fatalf("seed conversation: %v", err)
}
convID, _ := convRes.LastInsertId()
if _, err := db.Exec(
`INSERT INTO messages (conversation_id, from_agent, channel_id, body, priority, status, metadata)
VALUES (?, ?, ?, ?, 5, 'pending', '{}')`,
convID, fromAgent, channelID, body,
); err != nil {
t.Fatalf("seed message: %v", err)
}
}
func unmarshalText(t *testing.T, res *mcplib.CallToolResult) map[string]any {
t.Helper()
if res == nil || len(res.Content) != 1 {
t.Fatalf("unexpected result: %+v", res)
}
tc, ok := res.Content[0].(mcplib.TextContent)
if !ok {
t.Fatalf("not text content: %T", res.Content[0])
}
var m map[string]any
if err := json.Unmarshal([]byte(tc.Text), &m); err != nil {
t.Fatalf("unmarshal: %v: %s", err, tc.Text)
}
return m
}
func extractMemoryFromAgents(body map[string]any) []string {
rc, ok := body["relevant_context"].(map[string]any)
if !ok {
return nil
}
mems, ok := rc["memories"].([]any)
if !ok {
return nil
}
out := make([]string, 0, len(mems))
for _, raw := range mems {
m, ok := raw.(map[string]any)
if !ok {
continue
}
if fa, ok := m["from_agent"].(string); ok {
out = append(out, fa)
}
}
return out
}
func extractMemoryIDs(body map[string]any) map[int64]struct{} {
rc, ok := body["relevant_context"].(map[string]any)
if !ok {
return map[int64]struct{}{}
}
mems, ok := rc["memories"].([]any)
if !ok {
return map[int64]struct{}{}
}
out := map[int64]struct{}{}
for _, raw := range mems {
m, ok := raw.(map[string]any)
if !ok {
continue
}
if id, ok := m["id"].(float64); ok {
out[int64(id)] = struct{}{}
}
}
return out
}
+268
View File
@@ -0,0 +1,268 @@
// Proactive-memory injection middleware — wraps MCP tool handlers so
// that successful JSON responses gain a `relevant_context` field per
// `specs/020-proactive-memory-dream-worker/contracts/mcp-injection.md`.
package mcp
import (
"context"
"encoding/json"
"log/slog"
"time"
mcplib "github.com/mark3labs/mcp-go/mcp"
"github.com/mark3labs/mcp-go/server"
"github.com/synapbus/synapbus/internal/agents"
"github.com/synapbus/synapbus/internal/messaging"
"github.com/synapbus/synapbus/internal/metrics"
"github.com/synapbus/synapbus/internal/search"
)
// ToolHandler is the mcp-go tool handler signature. Re-exported as an
// alias so the wrapper signature reads cleanly at registration sites.
type ToolHandler = server.ToolHandlerFunc
// QuerySource derives the retrieval query for one tool invocation. It
// is given the inner handler's parsed args (best-effort: nil when args
// don't fit map[string]any) and the inner handler's parsed JSON result
// (nil on error or non-JSON). It must be cheap; called on every wrapped
// tool call.
type QuerySource func(ctx context.Context, toolName string, args map[string]any, result map[string]any) string
// WrapConfig parameterizes WrapInjection.
type WrapConfig struct {
// Cfg is the messaging.MemoryConfig snapshot taken at server
// startup. When Cfg.InjectionEnabled is false, WrapInjection
// returns the handler unchanged.
Cfg messaging.MemoryConfig
// SearchSvc drives retrieval. Required when Cfg.InjectionEnabled.
SearchSvc *search.Service
// Injections is the 24h audit ring. May be nil — Record errors are
// logged and the wrapper continues.
Injections *messaging.MemoryInjections
// QuerySource derives the retrieval query for this tool. Required.
QuerySource QuerySource
// IncludeCore is true for session-start tools (e.g. my_status).
// Only those get the per-(owner, agent) core-memory blob injected.
IncludeCore bool
// CoreProvider is consulted when IncludeCore=true. May be nil
// (US2 not yet wired) — wrapper still functions, just skips core.
CoreProvider search.CoreMemoryProvider
// Logger is used for non-fatal failures. Defaults to slog.Default.
Logger *slog.Logger
}
// WrapInjection returns a ToolHandler that wraps `inner` with the
// proactive-memory injection middleware described in
// `contracts/mcp-injection.md`.
//
// When Cfg.InjectionEnabled is false, the original handler is returned
// unchanged so the response payload exactly matches the pre-feature
// shape (FR-012, SC-009).
//
// Otherwise, the wrapper:
// 1. Runs the inner handler.
// 2. If the result is an error or not a single TextContent of JSON
// object shape, returns the result unchanged.
// 3. Builds a ContextPacket via search.BuildContextPacket using the
// query derived from cfg.QuerySource.
// 4. If the packet is non-empty (>=1 memory or core memory set), merges
// `relevant_context: <packet>` into the JSON body and re-marshals.
// 5. Records the injection to the 24h audit ring asynchronously.
func WrapInjection(inner ToolHandler, toolName string, cfg WrapConfig) ToolHandler {
if !cfg.Cfg.InjectionEnabled {
return inner
}
logger := cfg.Logger
if logger == nil {
logger = slog.Default().With("component", "mcp-injection")
}
return func(ctx context.Context, req mcplib.CallToolRequest) (*mcplib.CallToolResult, error) {
res, err := inner(ctx, req)
if err != nil {
return res, err
}
if res == nil || res.IsError {
return res, nil
}
// Locate the JSON text content. Non-JSON or multi-content
// payloads pass through unchanged.
idx, text, ok := singleJSONText(res)
if !ok {
return res, nil
}
var body map[string]any
if err := json.Unmarshal([]byte(text), &body); err != nil {
return res, nil
}
agent, ok := callerAgent(ctx)
if !ok || agent == nil {
// No identity → no owner scope → no injection.
metrics.InjectionSkippedTotal.WithLabelValues(toolName, "no_owner").Inc()
return res, nil
}
// Derive the retrieval query. nil args is fine; nil result is
// fine — the source decides what to do.
argsMap, _ := req.Params.Arguments.(map[string]any)
query := ""
if cfg.QuerySource != nil {
query = cfg.QuerySource(ctx, toolName, argsMap, body)
}
windowDays := int(cfg.Cfg.DreamRecentWindow / (24 * time.Hour))
if windowDays < 1 {
windowDays = 14
}
opts := search.InjectionOpts{
BudgetTokens: cfg.Cfg.InjectionBudgetTokens,
MaxItems: cfg.Cfg.InjectionMaxItems,
MinScore: cfg.Cfg.InjectionMinScore,
IncludeCore: cfg.IncludeCore,
CoreProvider: cfg.CoreProvider,
RecentWindowDays: windowDays,
}
pkt, err := search.BuildContextPacket(ctx, cfg.SearchSvc, agent, query, opts)
if err != nil {
logger.Debug("build context packet failed", "tool", toolName, "error", err)
return res, nil
}
if pkt == nil {
// Empty packet → omit `relevant_context` entirely.
metrics.InjectionSkippedTotal.WithLabelValues(toolName, "empty_pool").Inc()
return res, nil
}
if len(pkt.Memories) == 0 && pkt.CoreMemory == "" {
metrics.InjectionSkippedTotal.WithLabelValues(toolName, "empty_pool").Inc()
return res, nil
}
// Packet metrics — count, size, item-count. Done before re-marshal
// so a marshal failure still gets a packet-created observation
// (which is what we'd be alerting on anyway).
metrics.InjectionPacketsTotal.WithLabelValues(toolName).Inc()
metrics.InjectionMemoriesPerPacket.WithLabelValues(toolName).Observe(float64(len(pkt.Memories)))
metrics.InjectionPacketChars.WithLabelValues(toolName).Observe(float64(pkt.PacketChars))
body["relevant_context"] = pkt
merged, err := json.Marshal(body)
if err != nil {
logger.Debug("re-marshal failed", "tool", toolName, "error", err)
return res, nil
}
res.Content[idx] = mcplib.TextContent{Type: "text", Text: string(merged)}
// Audit-ring write is best-effort; never blocks the response.
recordInjection(cfg.Injections, logger, agent, toolName, pkt)
return res, nil
}
}
// singleJSONText reports the index of the single TextContent in `res`
// when its Text is a JSON object. Anything else (multiple contents,
// non-text, non-object JSON) returns ok=false → pass through.
func singleJSONText(res *mcplib.CallToolResult) (int, string, bool) {
if res == nil || len(res.Content) != 1 {
return 0, "", false
}
tc, ok := res.Content[0].(mcplib.TextContent)
if !ok {
return 0, "", false
}
// Quick sanity check that the text starts with `{` — avoids
// allocating a map for a known-non-object payload.
for i := 0; i < len(tc.Text); i++ {
switch tc.Text[i] {
case ' ', '\t', '\n', '\r':
continue
case '{':
return 0, tc.Text, true
default:
return 0, "", false
}
}
return 0, "", false
}
// callerAgent unpacks *agents.Agent from the request context. Uses the
// agents middleware ContextWithAgent, populated by the auth path.
func callerAgent(ctx context.Context) (*agents.Agent, bool) {
return agents.AgentFromContext(ctx)
}
// recordInjection writes one audit-ring row. Runs in a fresh goroutine
// so it cannot block the response, but inherits a detached context with
// a short timeout via the inner call site. Failures are logged at debug
// level since they're non-fatal for the request.
func recordInjection(store *messaging.MemoryInjections, logger *slog.Logger, agent *agents.Agent, toolName string, pkt *search.ContextPacket) {
if store == nil || agent == nil || pkt == nil {
return
}
ids := make([]int64, 0, len(pkt.Memories))
for _, m := range pkt.Memories {
ids = append(ids, m.ID)
}
rec := messaging.InjectionRecord{
OwnerID: ownerIDString(agent.OwnerID),
AgentName: agent.Name,
ToolName: toolName,
PacketSizeChars: pkt.PacketChars,
PacketItemsCount: len(pkt.Memories),
MessageIDs: ids,
CoreBlobIncluded: pkt.CoreMemory != "",
}
go func() {
// Detached background context: the request context may already
// be canceled by the time this goroutine runs.
ctx := context.Background()
if err := store.Record(ctx, rec); err != nil {
logger.Debug("audit-ring insert failed", "tool", toolName, "error", err)
}
}()
}
func ownerIDString(id int64) string {
if id == 0 {
return ""
}
// strconv.FormatInt is faster than fmt.Sprintf; mirror what
// agents.OwnerFor produces so comparisons in the search layer line
// up.
return formatInt64(id)
}
// formatInt64 is a tiny helper to avoid pulling strconv into the public
// surface area for one line.
func formatInt64(v int64) string {
const digits = "0123456789"
if v == 0 {
return "0"
}
neg := false
if v < 0 {
neg = true
v = -v
}
var buf [20]byte
i := len(buf)
for v > 0 {
i--
buf[i] = digits[v%10]
v /= 10
}
if neg {
i--
buf[i] = '-'
}
return string(buf[i:])
}
+218
View File
@@ -0,0 +1,218 @@
package mcp
import (
"context"
"encoding/json"
"testing"
mcplib "github.com/mark3labs/mcp-go/mcp"
"github.com/synapbus/synapbus/internal/agents"
"github.com/synapbus/synapbus/internal/messaging"
"github.com/synapbus/synapbus/internal/search"
)
// stubBuilder lets us bypass the real search service and short-circuit
// BuildContextPacket to whatever ContextPacket we want for the test.
// The wrap layer doesn't expose a builder seam (it calls
// search.BuildContextPacket directly), so we instead drive the wrap end
// to end with a real (empty) *search.Service and a stub CoreProvider
// that emits a packet when IncludeCore is true.
type stubCoreProvider struct{ blob string }
func (s *stubCoreProvider) Get(_ context.Context, _, _ string) (string, error) {
return s.blob, nil
}
func stubHandler(body map[string]any) ToolHandler {
return func(ctx context.Context, req mcplib.CallToolRequest) (*mcplib.CallToolResult, error) {
b, _ := json.Marshal(body)
return mcplib.NewToolResultText(string(b)), nil
}
}
func extractJSON(t *testing.T, res *mcplib.CallToolResult) map[string]any {
t.Helper()
if res == nil {
t.Fatal("nil result")
}
if len(res.Content) != 1 {
t.Fatalf("expected 1 content, got %d", len(res.Content))
}
tc, ok := res.Content[0].(mcplib.TextContent)
if !ok {
t.Fatalf("content not TextContent: %T", res.Content[0])
}
var m map[string]any
if err := json.Unmarshal([]byte(tc.Text), &m); err != nil {
t.Fatalf("non-JSON content: %v: %q", err, tc.Text)
}
return m
}
func TestWrapInjection_DisabledReturnsHandlerUnchanged(t *testing.T) {
inner := stubHandler(map[string]any{"hello": "world"})
cfg := WrapConfig{
Cfg: messaging.MemoryConfig{InjectionEnabled: false},
}
wrapped := WrapInjection(inner, "my_status", cfg)
res, err := wrapped(context.Background(), mcplib.CallToolRequest{})
if err != nil {
t.Fatalf("wrapped: %v", err)
}
body := extractJSON(t, res)
if _, has := body["relevant_context"]; has {
t.Error("relevant_context attached despite disabled config")
}
if body["hello"] != "world" {
t.Errorf("inner body mutated: %v", body)
}
}
func TestWrapInjection_EmptyMemoriesAndNoCore_OmitsField(t *testing.T) {
// IncludeCore=false and no memories in the DB → packet is nil → no
// relevant_context field on the response.
cfg := WrapConfig{
Cfg: messaging.MemoryConfig{
InjectionEnabled: true,
InjectionBudgetTokens: 500,
InjectionMaxItems: 5,
InjectionMinScore: 0.25,
},
SearchSvc: nil, // BuildContextPacket short-circuits when query=="" → returns nil
QuerySource: func(_ context.Context, _ string, _ map[string]any, _ map[string]any) string { return "" },
}
// SearchSvc nil + empty query forces BuildContextPacket through the
// "no retrieval" path. But it'll still try a core fetch (skipped:
// IncludeCore=false). With no provider and no retrieval → nil packet.
inner := stubHandler(map[string]any{"ok": true})
wrapped := WrapInjection(inner, "send_message", cfg)
// Inject a caller agent into the context so the wrapper does not
// bail out at the identity check.
ctx := agents.ContextWithAgent(context.Background(), &agents.Agent{Name: "a1", OwnerID: 1})
res, err := wrapped(ctx, mcplib.CallToolRequest{})
if err != nil {
t.Fatalf("wrapped: %v", err)
}
body := extractJSON(t, res)
if _, has := body["relevant_context"]; has {
t.Errorf("relevant_context attached when memories+core empty: %v", body["relevant_context"])
}
}
func TestWrapInjection_AppendsRelevantContext_CoreOnly(t *testing.T) {
cfg := WrapConfig{
Cfg: messaging.MemoryConfig{
InjectionEnabled: true,
InjectionBudgetTokens: 500,
InjectionMaxItems: 5,
InjectionMinScore: 0.25,
},
SearchSvc: nil, // query="" guarantees no retrieval attempt
IncludeCore: true,
CoreProvider: &stubCoreProvider{blob: "I am a1."},
QuerySource: func(_ context.Context, _ string, _ map[string]any, _ map[string]any) string { return "" },
}
inner := stubHandler(map[string]any{"agent": "a1"})
wrapped := WrapInjection(inner, "my_status", cfg)
ctx := agents.ContextWithAgent(context.Background(), &agents.Agent{Name: "a1", OwnerID: 1})
res, err := wrapped(ctx, mcplib.CallToolRequest{})
if err != nil {
t.Fatalf("wrapped: %v", err)
}
body := extractJSON(t, res)
rc, has := body["relevant_context"].(map[string]any)
if !has {
t.Fatalf("relevant_context missing: %+v", body)
}
if rc["core_memory"] != "I am a1." {
t.Errorf("core_memory wrong: %v", rc["core_memory"])
}
if mems, ok := rc["memories"].([]any); !ok || len(mems) != 0 {
t.Errorf("memories should be empty slice when only core is set: %v", rc["memories"])
}
// PacketChars at minimum the length of the core blob.
if got, ok := rc["packet_chars"].(float64); !ok || int(got) < len("I am a1.") {
t.Errorf("packet_chars looks wrong: %v", rc["packet_chars"])
}
}
func TestWrapInjection_NonJSONResultPassesThrough(t *testing.T) {
inner := func(ctx context.Context, req mcplib.CallToolRequest) (*mcplib.CallToolResult, error) {
return mcplib.NewToolResultText("not json"), nil
}
cfg := WrapConfig{
Cfg: messaging.MemoryConfig{
InjectionEnabled: true,
InjectionBudgetTokens: 500,
InjectionMaxItems: 5,
},
QuerySource: func(_ context.Context, _ string, _ map[string]any, _ map[string]any) string { return "" },
}
wrapped := WrapInjection(inner, "execute", cfg)
ctx := agents.ContextWithAgent(context.Background(), &agents.Agent{Name: "a1", OwnerID: 1})
res, err := wrapped(ctx, mcplib.CallToolRequest{})
if err != nil {
t.Fatalf("wrapped: %v", err)
}
if res == nil || len(res.Content) != 1 {
t.Fatalf("unexpected result shape: %+v", res)
}
tc, ok := res.Content[0].(mcplib.TextContent)
if !ok || tc.Text != "not json" {
t.Errorf("non-JSON result mutated: %+v", res.Content[0])
}
}
func TestWrapInjection_NoAgentInContext_PassesThrough(t *testing.T) {
cfg := WrapConfig{
Cfg: messaging.MemoryConfig{
InjectionEnabled: true,
InjectionBudgetTokens: 500,
InjectionMaxItems: 5,
},
IncludeCore: true,
CoreProvider: &stubCoreProvider{blob: "blob"},
QuerySource: func(_ context.Context, _ string, _ map[string]any, _ map[string]any) string { return "" },
}
wrapped := WrapInjection(stubHandler(map[string]any{"x": 1}), "my_status", cfg)
res, err := wrapped(context.Background(), mcplib.CallToolRequest{})
if err != nil {
t.Fatalf("wrapped: %v", err)
}
body := extractJSON(t, res)
if _, has := body["relevant_context"]; has {
t.Error("relevant_context attached despite no caller agent")
}
}
// Ensure ContextPacket as the value carries through json round-trip
// (it's used directly as a map entry via body["relevant_context"] = pkt).
func TestWrapInjection_ContextPacketJSONShape(t *testing.T) {
pkt := &search.ContextPacket{
Memories: []search.MemoryItem{},
CoreMemory: "core",
PacketChars: 4,
PacketTokenEstimate: 1,
RetrievalQuery: "",
SearchMode: "auto",
}
b, err := json.Marshal(pkt)
if err != nil {
t.Fatalf("marshal: %v", err)
}
var rt map[string]any
if err := json.Unmarshal(b, &rt); err != nil {
t.Fatalf("unmarshal: %v", err)
}
for _, k := range []string{"memories", "core_memory", "packet_chars", "packet_token_estimate", "retrieval_query", "search_mode"} {
if _, has := rt[k]; !has {
t.Errorf("missing JSON key %q", k)
}
}
}
+663
View File
@@ -0,0 +1,663 @@
// MCP memory-consolidation tools (feature 020 — dream worker, US3).
//
// Registered only when SYNAPBUS_DREAM_ENABLED=1 via
// MemoryToolRegistrar.RegisterAllOnServer (see server.go SetDream).
//
// Every tool:
//
// 1. Pulls the dispatch token from the request context (set by
// MCP middleware reading X-Synapbus-Dispatch-Token from the
// transport header, or via the harness-propagated env var).
// 2. Validates the token against (caller-supplied owner_id, the
// active consolidation_job_id carried alongside the token).
// 3. Performs the action against the appropriate messaging store.
// 4. Appends a structured action record to the job's `actions`
// JSON array via JobsStore.AppendAction.
//
// All errors follow the MCP standard envelope; the contract codes
// (`dispatch_token_*`, `not_same_owner`, `core_memory_too_large`,
// `relation_type_reserved`, `source_not_found`, ...) are listed in
// `contracts/mcp-memory-tools.md` and mirrored verbatim here so the
// dream-agent can match on the string.
package mcp
import (
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"log/slog"
"strings"
"time"
mcplib "github.com/mark3labs/mcp-go/mcp"
"github.com/mark3labs/mcp-go/server"
"github.com/synapbus/synapbus/internal/agents"
"github.com/synapbus/synapbus/internal/messaging"
)
// Context key for the dispatch token. The transport-layer middleware
// (or test harness) stuffs the token from the HTTP header
// `X-Synapbus-Dispatch-Token` into ctx via WithDispatchToken; the
// memory tool handlers read it via DispatchTokenFromContext.
type dispatchTokenKey struct{}
// WithDispatchToken returns a derived context carrying tok as the
// active dispatch token.
func WithDispatchToken(ctx context.Context, tok string) context.Context {
return context.WithValue(ctx, dispatchTokenKey{}, tok)
}
// DispatchTokenFromContext returns the dispatch token, if any.
func DispatchTokenFromContext(ctx context.Context) (string, bool) {
v, ok := ctx.Value(dispatchTokenKey{}).(string)
return v, ok && v != ""
}
// MemoryToolDeps bundles the dependencies the six memory tools need.
type MemoryToolDeps struct {
DB *sql.DB
Msg *messaging.MessagingService
Agents *agents.AgentService
Core *messaging.CoreMemoryStore
Links *messaging.LinkStore
Pins *messaging.PinStore
Jobs *messaging.JobsStore
Tokens *messaging.DispatchTokenStore
MemConfig messaging.MemoryConfig
Logger *slog.Logger
}
// MemoryToolRegistrar registers the six memory_* MCP tools.
type MemoryToolRegistrar struct {
deps MemoryToolDeps
logger *slog.Logger
}
// NewMemoryToolRegistrar returns a registrar over deps. RegisterAllOnServer
// is a no-op when deps.DB or deps.Jobs is nil (defensive — these are required).
func NewMemoryToolRegistrar(deps MemoryToolDeps) *MemoryToolRegistrar {
logger := deps.Logger
if logger == nil {
logger = slog.Default().With("component", "mcp-memory-tools")
}
return &MemoryToolRegistrar{deps: deps, logger: logger}
}
// RegisterAllOnServer attaches the six tools to mcpSrv. The caller
// (server.go SetDream) is responsible for gating registration on
// SYNAPBUS_DREAM_ENABLED.
func (r *MemoryToolRegistrar) RegisterAllOnServer(s *server.MCPServer) {
if s == nil || r.deps.DB == nil || r.deps.Jobs == nil || r.deps.Tokens == nil {
return
}
s.AddTool(memoryListUnprocessedTool(), r.handleListUnprocessed)
s.AddTool(memoryWriteReflectionTool(), r.handleWriteReflection)
s.AddTool(memoryRewriteCoreTool(), r.handleRewriteCore)
s.AddTool(memoryMarkDuplicateTool(), r.handleMarkDuplicate)
s.AddTool(memorySupersedeTool(), r.handleSupersede)
s.AddTool(memoryAddLinkTool(), r.handleAddLink)
r.logger.Info("memory MCP tools registered", "count", 6)
}
// --- Tool definitions ---
func memoryListUnprocessedTool() mcplib.Tool {
return mcplib.NewTool("memory_list_unprocessed",
mcplib.WithDescription("List recent memory-eligible messages the owner's pool has not yet consolidated. Used by the dream agent to scan its inbox."),
mcplib.WithString("owner_id", mcplib.Description("Caller's owner_id (must match the dispatch token's owner)"), mcplib.Required()),
mcplib.WithNumber("since_message_id", mcplib.Description("Exclusive lower bound (defaults to 0)")),
mcplib.WithNumber("limit", mcplib.Description("Max items to return (default 50, max 200)")),
)
}
func memoryWriteReflectionTool() mcplib.Tool {
return mcplib.NewTool("memory_write_reflection",
mcplib.WithDescription("Write a higher-level abstraction back to the memory pool tagged 'reflection'. Inserts 'refines' links from the new memory to each source."),
mcplib.WithString("owner_id", mcplib.Required()),
mcplib.WithString("body", mcplib.Required()),
mcplib.WithString("source_message_ids", mcplib.Description("Comma-separated message ids")),
mcplib.WithString("tags", mcplib.Description("Comma-separated tags")),
)
}
func memoryRewriteCoreTool() mcplib.Tool {
return mcplib.NewTool("memory_rewrite_core",
mcplib.WithDescription("Replace the per-(owner, agent) core memory blob wholesale (no merge)."),
mcplib.WithString("owner_id", mcplib.Required()),
mcplib.WithString("agent_name", mcplib.Required()),
mcplib.WithString("blob", mcplib.Required()),
)
}
func memoryMarkDuplicateTool() mcplib.Tool {
return mcplib.NewTool("memory_mark_duplicate",
mcplib.WithDescription("Mark two memories as duplicates; one is kept canonical, the other soft-deleted."),
mcplib.WithString("owner_id", mcplib.Required()),
mcplib.WithNumber("a_id", mcplib.Required()),
mcplib.WithNumber("b_id", mcplib.Required()),
mcplib.WithNumber("keep_id", mcplib.Required()),
mcplib.WithString("reason"),
)
}
func memorySupersedeTool() mcplib.Tool {
return mcplib.NewTool("memory_supersede",
mcplib.WithDescription("Mark memory A as obsoleted by memory B (temporal validity)."),
mcplib.WithString("owner_id", mcplib.Required()),
mcplib.WithNumber("a_id", mcplib.Required()),
mcplib.WithNumber("b_id", mcplib.Required()),
mcplib.WithString("reason"),
)
}
func memoryAddLinkTool() mcplib.Tool {
return mcplib.NewTool("memory_add_link",
mcplib.WithDescription("Add a typed link between two memories. relation_type must be one of refines, contradicts, examples, related."),
mcplib.WithString("owner_id", mcplib.Required()),
mcplib.WithNumber("src_id", mcplib.Required()),
mcplib.WithNumber("dst_id", mcplib.Required()),
mcplib.WithString("relation_type", mcplib.Required()),
mcplib.WithString("metadata", mcplib.Description("JSON object")),
)
}
// --- Shared validation ---
// authorizeForOwner validates the dispatch token in ctx against the
// caller-supplied owner_id. Returns the active jobID (so the handler
// can call AppendAction) or an MCP error result.
func (r *MemoryToolRegistrar) authorizeForOwner(ctx context.Context, ownerID string) (jobID int64, errResult *mcplib.CallToolResult) {
tok, ok := DispatchTokenFromContext(ctx)
if !ok {
return 0, memErrorf("dispatch_token_missing", "no dispatch token in request context")
}
// Find the consolidation_job_id this token is bound to.
var (
dbOwner string
dbJob int64
expiresAt time.Time
revokedAt sql.NullTime
)
err := r.deps.DB.QueryRowContext(ctx,
`SELECT owner_id, consolidation_job_id, expires_at, revoked_at
FROM memory_dispatch_tokens WHERE token = ?`, tok,
).Scan(&dbOwner, &dbJob, &expiresAt, &revokedAt)
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
return 0, memErrorf("dispatch_token_missing", "token not found")
}
return 0, memErrorf("dispatch_token_missing", "token lookup failed: %s", err)
}
if revokedAt.Valid {
return 0, memErrorf("dispatch_token_revoked", "token has been revoked")
}
if !expiresAt.After(time.Now().UTC()) {
return 0, memErrorf("dispatch_token_expired", "token expired at %s", expiresAt.Format(time.RFC3339))
}
if dbOwner != ownerID {
return 0, memErrorf("dispatch_token_owner_mismatch", "token bound to %q, request claims %q", dbOwner, ownerID)
}
// Run the canonical Validate path so used_at is stamped uniformly.
if r.deps.Tokens != nil {
if _, err := r.deps.Tokens.Validate(ctx, tok, ownerID, dbJob); err != nil {
return 0, memErrorf("dispatch_token_missing", "validate: %s", err)
}
}
return dbJob, nil
}
// recordAction appends to the job's actions JSON array. Logged at
// warn-level on failure; never blocks the tool's user-visible response.
func (r *MemoryToolRegistrar) recordAction(ctx context.Context, jobID int64, tool string, targetID int64, args map[string]any) {
if r.deps.Jobs == nil {
return
}
action := map[string]any{
"tool": tool,
"target_message_id": targetID,
"args": args,
"at": time.Now().UTC().Format(time.RFC3339),
}
if err := r.deps.Jobs.AppendAction(ctx, jobID, action); err != nil {
r.logger.Warn("append action failed",
"job_id", jobID,
"tool", tool,
"error", err,
)
}
}
// memErrorf returns an MCP CallToolResult carrying a contractual error
// code + human-readable message. The MCP framework already wraps the
// `error` field in the JSON envelope; we render `code: ...` as the
// leading line of the message so the dream-agent can pattern-match.
func memErrorf(code, format string, args ...any) *mcplib.CallToolResult {
msg := fmt.Sprintf("%s: %s", code, fmt.Sprintf(format, args...))
return mcplib.NewToolResultError(msg)
}
// --- Handlers ---
func (r *MemoryToolRegistrar) handleListUnprocessed(ctx context.Context, req mcplib.CallToolRequest) (*mcplib.CallToolResult, error) {
owner := req.GetString("owner_id", "")
if owner == "" {
return memErrorf("invalid_request", "owner_id required"), nil
}
jobID, errR := r.authorizeForOwner(ctx, owner)
if errR != nil {
return errR, nil
}
since := int64(req.GetInt("since_message_id", 0))
limit := req.GetInt("limit", 50)
if limit <= 0 {
limit = 50
}
if limit > 200 {
limit = 200
}
memIDs, err := messaging.MemoryChannelIDs(ctx, r.deps.DB)
if err != nil {
return memErrorf("internal", "list memory channels: %s", err), nil
}
if len(memIDs) == 0 {
_ = jobID // record nothing — empty
return resultJSON(map[string]any{"memories": []any{}, "max_id_returned": since})
}
placeholders := strings.Repeat("?,", len(memIDs))
placeholders = placeholders[:len(placeholders)-1]
queryArgs := []any{}
for _, id := range memIDs {
queryArgs = append(queryArgs, id)
}
// Apply the same 14d (configurable) recency window the worker uses
// so the consolidation agent only ever sees a bounded input set.
windowDays := int(r.deps.MemConfig.DreamRecentWindow / (24 * time.Hour))
if windowDays < 1 {
windowDays = 14
}
windowExpr := fmt.Sprintf("-%d days", windowDays)
queryArgs = append(queryArgs, owner, since, windowExpr, limit)
// Contract guarantees this list excludes:
// - messages already linked as the dst_message_id of a
// refines/duplicate_of/superseded_by edge (already
// consolidated by an earlier dream pass), AND
// - the dream worker's own output (from_agent prefix "dream:")
// so the agent never re-refines its own reflections.
// Without these filters the agent loops on the same oldest-50
// messages every cycle and progress flat-lines.
q := `SELECT m.id, m.from_agent, c.name, m.body, m.created_at
FROM messages m
JOIN agents a ON m.from_agent = a.name
JOIN channels c ON m.channel_id = c.id
WHERE m.channel_id IN (` + placeholders + `)
AND CAST(a.owner_id AS TEXT) = ?
AND m.id > ?
AND m.created_at > datetime('now', ?)
AND m.from_agent NOT LIKE 'dream:%'
AND m.id NOT IN (
SELECT dst_message_id FROM memory_links
WHERE relation_type IN ('refines','duplicate_of','superseded_by')
)
ORDER BY m.id ASC
LIMIT ?`
rows, err := r.deps.DB.QueryContext(ctx, q, queryArgs...)
if err != nil {
return memErrorf("internal", "query: %s", err), nil
}
defer rows.Close()
type item struct {
ID int64 `json:"id"`
FromAgent string `json:"from_agent"`
Channel string `json:"channel"`
Body string `json:"body"`
CreatedAt time.Time `json:"created_at"`
}
var items []item
var maxID = since
for rows.Next() {
var it item
if err := rows.Scan(&it.ID, &it.FromAgent, &it.Channel, &it.Body, &it.CreatedAt); err != nil {
return memErrorf("internal", "scan: %s", err), nil
}
items = append(items, it)
if it.ID > maxID {
maxID = it.ID
}
}
if err := rows.Err(); err != nil {
return memErrorf("internal", "iterate: %s", err), nil
}
r.recordAction(ctx, jobID, "memory_list_unprocessed", 0, map[string]any{
"since_message_id": since, "limit": limit, "returned": len(items),
})
return resultJSON(map[string]any{"memories": items, "max_id_returned": maxID})
}
func (r *MemoryToolRegistrar) handleWriteReflection(ctx context.Context, req mcplib.CallToolRequest) (*mcplib.CallToolResult, error) {
owner := req.GetString("owner_id", "")
body := req.GetString("body", "")
if owner == "" || body == "" {
return memErrorf("invalid_request", "owner_id and body required"), nil
}
jobID, errR := r.authorizeForOwner(ctx, owner)
if errR != nil {
return errR, nil
}
sourceIDs := parseInt64CSV(req.GetString("source_message_ids", ""))
// Verify every source belongs to caller's owner.
for _, sid := range sourceIDs {
ok, sameOwner, _ := r.messageBelongsTo(ctx, sid, owner)
if !ok {
return memErrorf("source_not_found", "message %d not found", sid), nil
}
if !sameOwner {
return memErrorf("not_same_owner", "source %d belongs to a different owner", sid), nil
}
}
// Pick a destination channel: prefer `#reflections-<owner>` if it
// exists, else `#open-brain`.
channelID, channelName, err := r.pickReflectionChannel(ctx, owner)
if err != nil {
return memErrorf("internal", "pick channel: %s", err), nil
}
if channelID == 0 {
return memErrorf("internal", "no reflection channel available"), nil
}
dreamAgent := "dream:" + owner
// Ensure conversation + message inserts (lightweight direct SQL —
// the MessagingService path would trigger reactive runs which we
// must avoid per feedback_system_dm_no_trigger.md).
convRes, err := r.deps.DB.ExecContext(ctx,
`INSERT INTO conversations (created_by, channel_id) VALUES (?, ?)`,
dreamAgent, channelID,
)
if err != nil {
return memErrorf("internal", "create conversation: %s", err), nil
}
convID, _ := convRes.LastInsertId()
res, err := r.deps.DB.ExecContext(ctx,
`INSERT INTO messages (conversation_id, from_agent, channel_id, body, priority, status, metadata)
VALUES (?, ?, ?, ?, 5, 'pending', ?)`,
convID, dreamAgent, channelID, body, `{"tags":["reflection"]}`,
)
if err != nil {
return memErrorf("internal", "insert message: %s", err), nil
}
newID, _ := res.LastInsertId()
// Add `refines` links from new memory → each source.
created := 0
for _, sid := range sourceIDs {
if r.deps.Links == nil {
break
}
if _, err := r.deps.Links.Add(ctx, newID, sid, "refines", owner, "agent:dream:"+owner, nil); err == nil {
created++
}
}
r.recordAction(ctx, jobID, "memory_write_reflection", newID, map[string]any{
"source_message_ids": sourceIDs,
"channel": channelName,
})
return resultJSON(map[string]any{
"memory_id": newID,
"channel": channelName,
"links_created": created,
})
}
func (r *MemoryToolRegistrar) handleRewriteCore(ctx context.Context, req mcplib.CallToolRequest) (*mcplib.CallToolResult, error) {
owner := req.GetString("owner_id", "")
agent := req.GetString("agent_name", "")
blob := req.GetString("blob", "")
if owner == "" || agent == "" {
return memErrorf("invalid_request", "owner_id and agent_name required"), nil
}
jobID, errR := r.authorizeForOwner(ctx, owner)
if errR != nil {
return errR, nil
}
if r.deps.Core == nil {
return memErrorf("internal", "core memory store not configured"), nil
}
// Confirm target agent is owned by caller's owner.
targetOwner, err := agents.OwnerFor(ctx, r.deps.DB, agent)
if err != nil {
if errors.Is(err, agents.ErrAgentNotFound) {
return memErrorf("source_not_found", "agent %q not found", agent), nil
}
return memErrorf("internal", "owner lookup: %s", err), nil
}
if targetOwner != owner {
return memErrorf("not_same_owner", "agent %q owner=%q != caller %q", agent, targetOwner, owner), nil
}
prev, _, _, _ := r.deps.Core.Get(ctx, owner, agent)
if err := r.deps.Core.Set(ctx, owner, agent, blob, "agent:dream:"+owner); err != nil {
if errors.Is(err, messaging.ErrCoreMemoryTooLarge) {
return memErrorf("core_memory_too_large", "blob %d bytes exceeds cap", len(blob)), nil
}
return memErrorf("internal", "set core: %s", err), nil
}
r.recordAction(ctx, jobID, "memory_rewrite_core", 0, map[string]any{
"owner_id": owner, "agent_name": agent, "new_chars": len(blob),
})
return resultJSON(map[string]any{
"owner_id": owner,
"agent_name": agent,
"previous_blob": prev,
"new_blob_chars": len(blob),
"updated_at": time.Now().UTC().Format(time.RFC3339),
})
}
func (r *MemoryToolRegistrar) handleMarkDuplicate(ctx context.Context, req mcplib.CallToolRequest) (*mcplib.CallToolResult, error) {
owner := req.GetString("owner_id", "")
aID := int64(req.GetInt("a_id", 0))
bID := int64(req.GetInt("b_id", 0))
keepID := int64(req.GetInt("keep_id", 0))
reason := req.GetString("reason", "")
if owner == "" || aID == 0 || bID == 0 || keepID == 0 {
return memErrorf("invalid_request", "owner_id, a_id, b_id, keep_id required"), nil
}
if keepID != aID && keepID != bID {
return memErrorf("keep_id_not_in_pair", "keep_id must be a_id or b_id"), nil
}
jobID, errR := r.authorizeForOwner(ctx, owner)
if errR != nil {
return errR, nil
}
for _, id := range []int64{aID, bID} {
ok, sameOwner, _ := r.messageBelongsTo(ctx, id, owner)
if !ok {
return memErrorf("source_not_found", "message %d not found", id), nil
}
if !sameOwner {
return memErrorf("not_same_owner", "message %d belongs to a different owner", id), nil
}
}
loserID := aID
if keepID == aID {
loserID = bID
}
if r.deps.Links == nil {
return memErrorf("internal", "link store not configured"), nil
}
linkID, err := r.deps.Links.AddConsolidationLink(ctx, loserID, keepID, "duplicate_of", owner, "agent:dream:"+owner, map[string]any{"reason": reason})
if err != nil {
return memErrorf("internal", "add link: %s", err), nil
}
r.recordAction(ctx, jobID, "memory_mark_duplicate", loserID, map[string]any{
"a_id": aID, "b_id": bID, "keep_id": keepID, "reason": reason,
})
return resultJSON(map[string]any{
"keep_id": keepID,
"soft_deleted_id": loserID,
"link_created_id": linkID,
})
}
func (r *MemoryToolRegistrar) handleSupersede(ctx context.Context, req mcplib.CallToolRequest) (*mcplib.CallToolResult, error) {
owner := req.GetString("owner_id", "")
aID := int64(req.GetInt("a_id", 0))
bID := int64(req.GetInt("b_id", 0))
reason := req.GetString("reason", "")
if owner == "" || aID == 0 || bID == 0 {
return memErrorf("invalid_request", "owner_id, a_id, b_id required"), nil
}
jobID, errR := r.authorizeForOwner(ctx, owner)
if errR != nil {
return errR, nil
}
for _, id := range []int64{aID, bID} {
ok, sameOwner, _ := r.messageBelongsTo(ctx, id, owner)
if !ok {
return memErrorf("source_not_found", "message %d not found", id), nil
}
if !sameOwner {
return memErrorf("not_same_owner", "message %d belongs to a different owner", id), nil
}
}
if r.deps.Links == nil {
return memErrorf("internal", "link store not configured"), nil
}
linkID, err := r.deps.Links.AddConsolidationLink(ctx, aID, bID, "superseded_by", owner, "agent:dream:"+owner, map[string]any{"reason": reason})
if err != nil {
return memErrorf("internal", "add link: %s", err), nil
}
r.recordAction(ctx, jobID, "memory_supersede", aID, map[string]any{
"a_id": aID, "b_id": bID, "reason": reason,
})
return resultJSON(map[string]any{
"superseded_id": aID,
"by_id": bID,
"link_created_id": linkID,
})
}
func (r *MemoryToolRegistrar) handleAddLink(ctx context.Context, req mcplib.CallToolRequest) (*mcplib.CallToolResult, error) {
owner := req.GetString("owner_id", "")
srcID := int64(req.GetInt("src_id", 0))
dstID := int64(req.GetInt("dst_id", 0))
relType := req.GetString("relation_type", "")
if owner == "" || srcID == 0 || dstID == 0 || relType == "" {
return memErrorf("invalid_request", "owner_id, src_id, dst_id, relation_type required"), nil
}
jobID, errR := r.authorizeForOwner(ctx, owner)
if errR != nil {
return errR, nil
}
for _, id := range []int64{srcID, dstID} {
ok, sameOwner, _ := r.messageBelongsTo(ctx, id, owner)
if !ok {
return memErrorf("source_not_found", "message %d not found", id), nil
}
if !sameOwner {
return memErrorf("not_same_owner", "message %d belongs to a different owner", id), nil
}
}
if r.deps.Links == nil {
return memErrorf("internal", "link store not configured"), nil
}
var meta map[string]any
if mraw := req.GetString("metadata", ""); mraw != "" {
_ = json.Unmarshal([]byte(mraw), &meta)
}
linkID, err := r.deps.Links.Add(ctx, srcID, dstID, relType, owner, "agent:dream:"+owner, meta)
if err != nil {
if errors.Is(err, messaging.ErrLinkTypeReserved) {
return memErrorf("relation_type_reserved", "type %q is reserved", relType), nil
}
return memErrorf("internal", "add link: %s", err), nil
}
r.recordAction(ctx, jobID, "memory_add_link", dstID, map[string]any{
"src_id": srcID, "dst_id": dstID, "relation_type": relType,
})
return resultJSON(map[string]any{"link_id": linkID})
}
// --- helpers ---
func parseInt64CSV(s string) []int64 {
if s == "" {
return nil
}
parts := strings.Split(s, ",")
out := make([]int64, 0, len(parts))
for _, p := range parts {
p = strings.TrimSpace(p)
if p == "" {
continue
}
var v int64
if _, err := fmt.Sscanf(p, "%d", &v); err == nil && v > 0 {
out = append(out, v)
}
}
return out
}
// messageBelongsTo reports (exists, ownerMatches, dbErr).
func (r *MemoryToolRegistrar) messageBelongsTo(ctx context.Context, msgID int64, ownerID string) (bool, bool, error) {
var fromAgent string
err := r.deps.DB.QueryRowContext(ctx,
`SELECT from_agent FROM messages WHERE id = ?`, msgID,
).Scan(&fromAgent)
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
return false, false, nil
}
return false, false, err
}
owner, err := agents.OwnerFor(ctx, r.deps.DB, fromAgent)
if err != nil {
return true, false, nil
}
return true, owner == ownerID, nil
}
// pickReflectionChannel picks a destination channel for a reflection.
// Preference: `reflections-<owner>` if any such channel exists, else
// `open-brain`.
func (r *MemoryToolRegistrar) pickReflectionChannel(ctx context.Context, ownerID string) (int64, string, error) {
// Try reflections-* the owner has authored to (best heuristic).
var (
id int64
name string
)
err := r.deps.DB.QueryRowContext(ctx,
`SELECT id, name FROM channels WHERE name LIKE 'reflections-%' ORDER BY id ASC LIMIT 1`,
).Scan(&id, &name)
if err == nil {
return id, name, nil
}
if !errors.Is(err, sql.ErrNoRows) {
return 0, "", err
}
// Fallback: open-brain.
err = r.deps.DB.QueryRowContext(ctx,
`SELECT id, name FROM channels WHERE name = 'open-brain' LIMIT 1`,
).Scan(&id, &name)
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
return 0, "", nil
}
return 0, "", err
}
return id, name, nil
}
+375
View File
@@ -0,0 +1,375 @@
package mcp
import (
"context"
"database/sql"
"encoding/json"
"strings"
"testing"
"time"
mcplib "github.com/mark3labs/mcp-go/mcp"
"github.com/synapbus/synapbus/internal/messaging"
)
// memToolHarness bundles deps + helpers for the memory tools tests.
type memToolHarness struct {
db *sql.DB
reg *MemoryToolRegistrar
tokens *messaging.DispatchTokenStore
jobs *messaging.JobsStore
links *messaging.LinkStore
pins *messaging.PinStore
core *messaging.CoreMemoryStore
jobID int64
tokenStr string
ownerID string
}
func newMemToolHarness(t *testing.T) *memToolHarness {
t.Helper()
db := newTestDB(t)
// Seed two owners + their agents (used for owner mismatch tests).
_, _ = db.Exec(`INSERT OR IGNORE INTO users (id, username, password_hash, display_name) VALUES (2, 'otheruser', 'hash', 'Other')`)
_, _ = db.Exec(`INSERT OR IGNORE INTO agents (name, display_name, type, owner_id, api_key_hash, status) VALUES ('a-h1', 'a-h1', 'ai', 1, 'h1', 'active')`)
_, _ = db.Exec(`INSERT OR IGNORE INTO agents (name, display_name, type, owner_id, api_key_hash, status) VALUES ('a-h2', 'a-h2', 'ai', 2, 'h2', 'active')`)
tokens := messaging.NewDispatchTokenStore(db)
jobs := messaging.NewJobsStore(db)
links := messaging.NewLinkStore(db)
pins := messaging.NewPinStore(db)
core := messaging.NewCoreMemoryStore(db, 64)
jobID, err := jobs.Create(context.Background(), "1", "reflection", "manual:test")
if err != nil {
t.Fatalf("create job: %v", err)
}
tok, _, err := tokens.Issue(context.Background(), "1", jobID)
if err != nil {
t.Fatalf("issue token: %v", err)
}
reg := NewMemoryToolRegistrar(MemoryToolDeps{
DB: db,
Core: core,
Links: links,
Pins: pins,
Jobs: jobs,
Tokens: tokens,
})
return &memToolHarness{
db: db, reg: reg, tokens: tokens, jobs: jobs, links: links, pins: pins, core: core,
jobID: jobID, tokenStr: tok, ownerID: "1",
}
}
func (h *memToolHarness) ctxWithToken(tok string) context.Context {
return WithDispatchToken(context.Background(), tok)
}
// seedMemoryChannel inserts an open-brain channel and one message
// belonging to `agentName`.
func (h *memToolHarness) seedMessage(t *testing.T, agentName, body string) int64 {
t.Helper()
_, _ = h.db.Exec(`INSERT OR IGNORE INTO channels (id, name, description, type, created_by) VALUES (1, 'open-brain', '', 'standard', 'system')`)
res, err := h.db.Exec(
`INSERT INTO conversations (created_by, channel_id) VALUES (?, 1)`, agentName)
if err != nil {
t.Fatalf("seed conv: %v", err)
}
convID, _ := res.LastInsertId()
res, err = h.db.Exec(
`INSERT INTO messages (conversation_id, from_agent, channel_id, body, priority, status, metadata)
VALUES (?, ?, 1, ?, 5, 'pending', '{}')`,
convID, agentName, body,
)
if err != nil {
t.Fatalf("seed message: %v", err)
}
id, _ := res.LastInsertId()
return id
}
func callRequest(args map[string]any) mcplib.CallToolRequest {
return mcplib.CallToolRequest{
Params: mcplib.CallToolParams{Arguments: args},
}
}
func resultText(t *testing.T, res *mcplib.CallToolResult) string {
t.Helper()
if res == nil || len(res.Content) == 0 {
t.Fatal("nil/empty result")
}
tc, ok := res.Content[0].(mcplib.TextContent)
if !ok {
t.Fatalf("not TextContent: %T", res.Content[0])
}
return tc.Text
}
func resultIsError(res *mcplib.CallToolResult) bool {
return res != nil && res.IsError
}
// --- Token error matrix ---
func TestMemoryTools_DispatchTokenMissing(t *testing.T) {
h := newMemToolHarness(t)
// No token in context.
res, _ := h.reg.handleAddLink(context.Background(), callRequest(map[string]any{
"owner_id": "1", "src_id": 1.0, "dst_id": 2.0, "relation_type": "refines",
}))
if !resultIsError(res) {
t.Fatal("expected error result")
}
if !strings.Contains(resultText(t, res), "dispatch_token_missing") {
t.Errorf("expected dispatch_token_missing, got %q", resultText(t, res))
}
}
func TestMemoryTools_DispatchTokenExpired(t *testing.T) {
h := newMemToolHarness(t)
// Forcibly expire.
if _, err := h.db.Exec(
`UPDATE memory_dispatch_tokens SET expires_at = ? WHERE token = ?`,
time.Now().Add(-1*time.Minute).UTC(), h.tokenStr,
); err != nil {
t.Fatalf("expire: %v", err)
}
res, _ := h.reg.handleAddLink(h.ctxWithToken(h.tokenStr), callRequest(map[string]any{
"owner_id": "1", "src_id": 1.0, "dst_id": 2.0, "relation_type": "refines",
}))
if !resultIsError(res) || !strings.Contains(resultText(t, res), "dispatch_token_expired") {
t.Errorf("expected dispatch_token_expired, got %q", resultText(t, res))
}
}
func TestMemoryTools_DispatchTokenOwnerMismatch(t *testing.T) {
h := newMemToolHarness(t)
res, _ := h.reg.handleAddLink(h.ctxWithToken(h.tokenStr), callRequest(map[string]any{
"owner_id": "2", "src_id": 1.0, "dst_id": 2.0, "relation_type": "refines",
}))
if !resultIsError(res) || !strings.Contains(resultText(t, res), "dispatch_token_owner_mismatch") {
t.Errorf("expected dispatch_token_owner_mismatch, got %q", resultText(t, res))
}
}
// --- memory_add_link ---
func TestMemoryAddLink_HappyPath(t *testing.T) {
h := newMemToolHarness(t)
a := h.seedMessage(t, "a-h1", "fact A")
b := h.seedMessage(t, "a-h1", "fact B")
res, err := h.reg.handleAddLink(h.ctxWithToken(h.tokenStr), callRequest(map[string]any{
"owner_id": "1",
"src_id": float64(a),
"dst_id": float64(b),
"relation_type": "refines",
}))
if err != nil {
t.Fatalf("handleAddLink: %v", err)
}
if resultIsError(res) {
t.Fatalf("unexpected error: %s", resultText(t, res))
}
var body map[string]any
if err := json.Unmarshal([]byte(resultText(t, res)), &body); err != nil {
t.Fatalf("parse body: %v", err)
}
if _, ok := body["link_id"]; !ok {
t.Errorf("expected link_id in response: %v", body)
}
// And actions should have been appended.
job, err := h.jobs.Get(context.Background(), h.jobID)
if err != nil {
t.Fatalf("Get job: %v", err)
}
if len(job.Actions) != 1 || job.Actions[0]["tool"] != "memory_add_link" {
t.Errorf("expected one action for memory_add_link, got %v", job.Actions)
}
}
func TestMemoryAddLink_RelationTypeReserved(t *testing.T) {
h := newMemToolHarness(t)
a := h.seedMessage(t, "a-h1", "x")
b := h.seedMessage(t, "a-h1", "y")
res, _ := h.reg.handleAddLink(h.ctxWithToken(h.tokenStr), callRequest(map[string]any{
"owner_id": "1", "src_id": float64(a), "dst_id": float64(b),
"relation_type": "mention",
}))
if !resultIsError(res) || !strings.Contains(resultText(t, res), "relation_type_reserved") {
t.Errorf("expected relation_type_reserved, got %q", resultText(t, res))
}
}
func TestMemoryAddLink_NotSameOwner(t *testing.T) {
h := newMemToolHarness(t)
src := h.seedMessage(t, "a-h1", "x")
other := h.seedMessage(t, "a-h2", "y")
res, _ := h.reg.handleAddLink(h.ctxWithToken(h.tokenStr), callRequest(map[string]any{
"owner_id": "1", "src_id": float64(src), "dst_id": float64(other),
"relation_type": "refines",
}))
if !resultIsError(res) || !strings.Contains(resultText(t, res), "not_same_owner") {
t.Errorf("expected not_same_owner, got %q", resultText(t, res))
}
}
func TestMemoryAddLink_SourceNotFound(t *testing.T) {
h := newMemToolHarness(t)
res, _ := h.reg.handleAddLink(h.ctxWithToken(h.tokenStr), callRequest(map[string]any{
"owner_id": "1", "src_id": 9999.0, "dst_id": 8888.0, "relation_type": "refines",
}))
if !resultIsError(res) || !strings.Contains(resultText(t, res), "source_not_found") {
t.Errorf("expected source_not_found, got %q", resultText(t, res))
}
}
// --- memory_rewrite_core ---
func TestMemoryRewriteCore_HappyPath(t *testing.T) {
h := newMemToolHarness(t)
res, _ := h.reg.handleRewriteCore(h.ctxWithToken(h.tokenStr), callRequest(map[string]any{
"owner_id": "1", "agent_name": "a-h1", "blob": "I am a-h1.",
}))
if resultIsError(res) {
t.Fatalf("unexpected error: %s", resultText(t, res))
}
blob, _, ok, _ := h.core.Get(context.Background(), "1", "a-h1")
if !ok || blob != "I am a-h1." {
t.Errorf("blob not stored: ok=%v blob=%q", ok, blob)
}
}
func TestMemoryRewriteCore_TooLarge(t *testing.T) {
h := newMemToolHarness(t)
big := strings.Repeat("x", 65)
res, _ := h.reg.handleRewriteCore(h.ctxWithToken(h.tokenStr), callRequest(map[string]any{
"owner_id": "1", "agent_name": "a-h1", "blob": big,
}))
if !resultIsError(res) || !strings.Contains(resultText(t, res), "core_memory_too_large") {
t.Errorf("expected core_memory_too_large, got %q", resultText(t, res))
}
}
func TestMemoryRewriteCore_AgentNotSameOwner(t *testing.T) {
h := newMemToolHarness(t)
res, _ := h.reg.handleRewriteCore(h.ctxWithToken(h.tokenStr), callRequest(map[string]any{
"owner_id": "1", "agent_name": "a-h2", "blob": "trying to overwrite",
}))
if !resultIsError(res) || !strings.Contains(resultText(t, res), "not_same_owner") {
t.Errorf("expected not_same_owner, got %q", resultText(t, res))
}
}
// --- memory_mark_duplicate ---
func TestMemoryMarkDuplicate_HappyPath(t *testing.T) {
h := newMemToolHarness(t)
a := h.seedMessage(t, "a-h1", "fact A")
b := h.seedMessage(t, "a-h1", "fact A shorter")
res, _ := h.reg.handleMarkDuplicate(h.ctxWithToken(h.tokenStr), callRequest(map[string]any{
"owner_id": "1", "a_id": float64(a), "b_id": float64(b), "keep_id": float64(a),
"reason": "shorter",
}))
if resultIsError(res) {
t.Fatalf("unexpected error: %s", resultText(t, res))
}
// Audit row appended on job.
job, _ := h.jobs.Get(context.Background(), h.jobID)
if len(job.Actions) != 1 || job.Actions[0]["tool"] != "memory_mark_duplicate" {
t.Errorf("expected one mark_duplicate action: %v", job.Actions)
}
}
// --- memory_supersede ---
func TestMemorySupersede_HappyPath(t *testing.T) {
h := newMemToolHarness(t)
a := h.seedMessage(t, "a-h1", "old fact")
b := h.seedMessage(t, "a-h1", "new fact")
res, _ := h.reg.handleSupersede(h.ctxWithToken(h.tokenStr), callRequest(map[string]any{
"owner_id": "1", "a_id": float64(a), "b_id": float64(b), "reason": "newer",
}))
if resultIsError(res) {
t.Fatalf("unexpected error: %s", resultText(t, res))
}
}
// --- memory_list_unprocessed ---
func TestMemoryListUnprocessed_ReturnsOwnerScopedMessages(t *testing.T) {
h := newMemToolHarness(t)
h.seedMessage(t, "a-h1", "memory 1")
h.seedMessage(t, "a-h1", "memory 2")
h.seedMessage(t, "a-h2", "other owner's memory")
res, _ := h.reg.handleListUnprocessed(h.ctxWithToken(h.tokenStr), callRequest(map[string]any{
"owner_id": "1",
}))
if resultIsError(res) {
t.Fatalf("unexpected error: %s", resultText(t, res))
}
var body map[string]any
_ = json.Unmarshal([]byte(resultText(t, res)), &body)
mems, _ := body["memories"].([]any)
if len(mems) != 2 {
t.Errorf("expected 2 owner-scoped memories, got %d (full body: %v)", len(mems), body)
}
}
// --- memory_write_reflection ---
func TestMemoryWriteReflection_HappyPath(t *testing.T) {
h := newMemToolHarness(t)
a := h.seedMessage(t, "a-h1", "source 1")
b := h.seedMessage(t, "a-h1", "source 2")
args := map[string]any{
"owner_id": "1",
"body": "Across these I notice...",
"source_message_ids": "" + intCSV(a, b),
}
res, _ := h.reg.handleWriteReflection(h.ctxWithToken(h.tokenStr), callRequest(args))
if resultIsError(res) {
t.Fatalf("unexpected error: %s", resultText(t, res))
}
var body map[string]any
_ = json.Unmarshal([]byte(resultText(t, res)), &body)
if _, ok := body["memory_id"]; !ok {
t.Errorf("expected memory_id: %v", body)
}
if v, _ := body["links_created"].(float64); int(v) != 2 {
t.Errorf("expected links_created=2, got %v", body["links_created"])
}
}
func intCSV(ids ...int64) string {
var b strings.Builder
for i, id := range ids {
if i > 0 {
b.WriteString(",")
}
b.WriteString(itoa(id))
}
return b.String()
}
func itoa(v int64) string {
if v == 0 {
return "0"
}
var buf [20]byte
i := len(buf)
for v > 0 {
i--
buf[i] = byte('0' + v%10)
v /= 10
}
return string(buf[i:])
}
+58 -10
View File
@@ -29,13 +29,13 @@ import (
// MCPServer wraps the mcp-go server with SynapBus services.
type MCPServer struct {
mcpServer *server.MCPServer
httpServer *server.StreamableHTTPServer
connMgr *ConnectionManager
agentService *agents.AgentService
hybridRegistrar *HybridToolRegistrar
logger *slog.Logger
console *console.Printer
mcpServer *server.MCPServer
httpServer *server.StreamableHTTPServer
connMgr *ConnectionManager
agentService *agents.AgentService
hybridRegistrar *HybridToolRegistrar
logger *slog.Logger
console *console.Printer
}
// NewMCPServer creates and configures a new MCP server with 5 hybrid tools registered.
@@ -180,14 +180,26 @@ func NewMCPServer(
// Create Streamable HTTP transport with context func for auth propagation
httpServer := server.NewStreamableHTTPServer(mcpSrv,
server.WithHTTPContextFunc(func(ctx context.Context, r *http.Request) context.Context {
// Propagate agent identity from HTTP auth to MCP context
// Propagate agent identity from HTTP auth to MCP context.
// The full *agents.Agent is needed by injection middleware
// (which uses owner_id to scope retrieval); the bare name is
// kept for backward-compat with existing handlers that call
// AgentNameFromContext directly.
if agent, ok := agents.AgentFromContext(r.Context()); ok {
ctx = agents.ContextWithAgent(ctx, agent)
ctx = ContextWithAgentName(ctx, agent.Name)
// Propagate owner ID for trace recording
if ownerID, ok := trace.OwnerIDFromContext(r.Context()); ok {
ctx = trace.ContextWithOwnerID(ctx, ownerID)
}
}
// Dispatch-token bridge for feature 020 dream consolidation:
// the runner sends X-Synapbus-Dispatch-Token on every MCP
// request via the SDK's McpHttpServerConfig.headers field;
// the memory_* tools read this from request context.
if tok := r.Header.Get("X-Synapbus-Dispatch-Token"); tok != "" {
ctx = WithDispatchToken(ctx, tok)
}
return ctx
}),
)
@@ -206,9 +218,45 @@ func NewMCPServer(
return s
}
// SetInjection wires the proactive-memory injection middleware
// (feature 020) into the hybrid tool registrar and re-registers the
// hybrid tools so the wrappers take effect. Must be called after
// NewMCPServer and before the server starts handling traffic.
//
// `coreProvider` is consulted only on session-start tools (currently
// `my_status`). Pass nil when US2 has not yet been wired.
func (s *MCPServer) SetInjection(cfg messaging.MemoryConfig, store *messaging.MemoryInjections, coreProvider search.CoreMemoryProvider) {
if s.hybridRegistrar == nil || s.mcpServer == nil {
return
}
s.hybridRegistrar.SetInjection(cfg, store, coreProvider)
// Re-register the hybrid tools so the new InjectionEnabled / Core
// wiring takes effect. AddTool overwrites by name (see mcp-go's
// `MCPServer.AddTools`), so this swaps in the wrapped handlers
// without leaking the original registrations.
s.hybridRegistrar.RegisterAllOnServer(s.mcpServer)
}
// SetDream wires the six memory_* MCP tools (feature 020 — US3).
// Only registers when cfg.DreamEnabled is true; otherwise this is a
// no-op so the tool surface remains identical to the pre-feature
// shape. Must be called before the server starts serving traffic.
func (s *MCPServer) SetDream(deps MemoryToolDeps) {
if s == nil || s.mcpServer == nil {
return
}
if !deps.MemConfig.DreamEnabled {
s.logger.Info("dream tools not registered (SYNAPBUS_DREAM_ENABLED=0)")
return
}
reg := NewMemoryToolRegistrar(deps)
reg.RegisterAllOnServer(s.mcpServer)
}
// WireGoalsTools registers the spec-018 tool surface (create_goal,
// propose_task_tree, propose_agent, claim_task, request_resource,
// list_resources) on the MCP server. Must be called after NewMCPServer.
// propose_task_tree, claim_task, request_resource, list_resources,
// complete_goal) on the MCP server. Must be called after NewMCPServer.
// Note: propose_agent was removed in the internal-only mode change.
func (s *MCPServer) WireGoalsTools(r *GoalsToolRegistrar) {
if r == nil || s.mcpServer == nil {
return
+116 -6
View File
@@ -44,6 +44,26 @@ type HybridToolRegistrar struct {
db *sql.DB
queryExecutor *agentquery.Executor
logger *slog.Logger
// Injection (feature 020). When injectionCfg.Cfg.InjectionEnabled
// is false (the default), WrapInjection returns handlers unchanged
// so existing tool response shapes are preserved bit-for-bit.
injectionCfg messaging.MemoryConfig
memoryInjections *messaging.MemoryInjections
coreProvider search.CoreMemoryProvider
}
// SetInjection wires the proactive-memory injection middleware into
// every eligible MCP tool registered by RegisterAllOnServer. Call this
// after NewHybridToolRegistrar and before RegisterAllOnServer.
//
// `coreProvider` is consulted only on session-start tools (currently
// `my_status`). May be nil when US2 has not yet been wired — the
// wrapper simply skips the core-memory hook in that case.
func (h *HybridToolRegistrar) SetInjection(cfg messaging.MemoryConfig, store *messaging.MemoryInjections, coreProvider search.CoreMemoryProvider) {
h.injectionCfg = cfg
h.memoryInjections = store
h.coreProvider = coreProvider
}
// SetMarketplaceService attaches the marketplace service for the 5 new
@@ -93,14 +113,104 @@ func NewHybridToolRegistrar(
}
// RegisterAllOnServer registers all hybrid tools on an mcp-go MCPServer.
//
// When proactive-memory injection is enabled (via SetInjection), the
// eligible tool handlers are wrapped with WrapInjection so their JSON
// responses gain a `relevant_context` field per
// `contracts/mcp-injection.md`. Tools NOT in the eligible set
// (currently `get_replies`) are registered unchanged.
func (h *HybridToolRegistrar) RegisterAllOnServer(s *server.MCPServer) {
s.AddTool(h.myStatusTool(), h.handleMyStatus)
s.AddTool(h.sendMessageTool(), h.handleSendMessage)
s.AddTool(h.searchTool(), h.handleSearch)
s.AddTool(h.executeTool(), h.handleExecute)
s.AddTool(h.myStatusTool(), h.wrap("my_status", h.handleMyStatus, true))
s.AddTool(h.sendMessageTool(), h.wrap("send_message", h.handleSendMessage, false))
s.AddTool(h.searchTool(), h.wrap("search", h.handleSearch, false))
s.AddTool(h.executeTool(), h.wrap("execute", h.handleExecute, false))
s.AddTool(h.getRepliesTool(), h.handleGetReplies)
h.logger.Info("hybrid MCP tools registered", "count", 5)
h.logger.Info("hybrid MCP tools registered",
"count", 5,
"injection_enabled", h.injectionCfg.InjectionEnabled,
)
}
// wrap applies the proactive-memory WrapInjection middleware to one
// tool handler. When InjectionEnabled is false (default), wrap returns
// the original handler unchanged. `includeCore` is true only for
// session-start tools (my_status today).
func (h *HybridToolRegistrar) wrap(toolName string, inner ToolHandler, includeCore bool) ToolHandler {
if !h.injectionCfg.InjectionEnabled {
return inner
}
cfg := WrapConfig{
Cfg: h.injectionCfg,
SearchSvc: h.searchService,
Injections: h.memoryInjections,
IncludeCore: includeCore,
CoreProvider: h.coreProvider,
QuerySource: querySourceFor(toolName),
Logger: h.logger,
}
return WrapInjection(inner, toolName, cfg)
}
// querySourceFor returns the QuerySource closure for the given tool.
// The contract (`contracts/mcp-injection.md`) prescribes the retrieval
// query per tool:
//
// - my_status: "<recent activity>" (fallback when nothing else)
// - send_message: body of the sent message
// - search: the user's query argument
// - execute: stringified args (best-effort)
//
// Tools not registered as MCP tools here (`claim_messages`,
// `read_inbox`, `read_channel`) are exercised via the `execute` bridge
// and inherit the `execute` query source.
func querySourceFor(toolName string) QuerySource {
switch toolName {
case "my_status":
return func(_ context.Context, _ string, _ map[string]any, _ map[string]any) string {
// FR-009: when there's no explicit query, use recent owner
// activity. The retrieval layer interprets the empty string
// as "no query" and falls back to recency-ordered FTS.
return ""
}
case "send_message":
return func(_ context.Context, _ string, args map[string]any, _ map[string]any) string {
if args == nil {
return ""
}
if body, ok := args["body"].(string); ok {
return body
}
return ""
}
case "search":
return func(_ context.Context, _ string, args map[string]any, _ map[string]any) string {
if args == nil {
return ""
}
if q, ok := args["query"].(string); ok {
return q
}
return ""
}
case "execute":
return func(_ context.Context, _ string, args map[string]any, _ map[string]any) string {
if args == nil {
return ""
}
// Best-effort: use the `code` argument verbatim. It is the
// only required input and reliably reflects what the agent
// is about to do.
if code, ok := args["code"].(string); ok {
return code
}
return ""
}
default:
return func(_ context.Context, _ string, _ map[string]any, _ map[string]any) string {
return ""
}
}
}
// --- Tool Definitions ---
@@ -118,7 +228,7 @@ func (h *HybridToolRegistrar) sendMessageTool() mcplib.Tool {
mcplib.WithString("channel", mcplib.Description("Channel name or numeric ID for channel messages")),
mcplib.WithString("body", mcplib.Description("Message body text"), mcplib.Required()),
mcplib.WithString("subject", mcplib.Description("Conversation subject (optional)")),
mcplib.WithNumber("priority", mcplib.Description("Message priority (1-10, default 5)"), mcplib.Min(1), mcplib.Max(10)),
mcplib.WithNumber("priority", mcplib.Description("Message priority (1-10, default 5)")),
mcplib.WithString("metadata", mcplib.Description("JSON metadata object (optional)")),
mcplib.WithNumber("reply_to", mcplib.Description("ID of the parent message to reply to. Creates a threaded reply. Always use reply_to when responding to a message that is itself a thread reply, to keep conversations organized.")),
mcplib.WithString("attachments", mcplib.Description("Comma-separated list of attachment hashes to link to this message. Upload attachments first using the upload_attachment action via the execute tool.")),
+205
View File
@@ -0,0 +1,205 @@
// Auto-link emitter for feature 020 — when a new memory-eligible
// message is inserted, derive `mention`, `reply_to`, and
// `channel_cooccurrence` links automatically and write them with
// `created_by = "auto:<rule>"`. The dream-agent's `memory_add_link`
// tool rejects these auto-types (see memory_links.go) so this is the
// only path that creates them.
//
// Wiring: register an AutoLinkListener with
// `MessagingService.AddMessageListener`. The listener fires after
// every successful message insert. Failures are logged but never
// propagated — auto-links are best-effort.
package messaging
import (
"context"
"database/sql"
"encoding/json"
"errors"
"log/slog"
"regexp"
)
// mentionPattern matches `@agent-name` in the body. Hyphens and digits
// allowed; case-insensitive; bounded by non-word characters or string
// edges. Matches the existing mentions.go convention.
var autoMentionPattern = regexp.MustCompile(`@([A-Za-z][A-Za-z0-9_\-]{1,63})`)
// AutoLinkListener implements MessageListener and writes the three
// auto-link types per OnMessageSent invocation.
type AutoLinkListener struct {
db *sql.DB
links *LinkStore
logger *slog.Logger
}
// NewAutoLinkListener returns a listener over db + links.
func NewAutoLinkListener(db *sql.DB, links *LinkStore) *AutoLinkListener {
return &AutoLinkListener{
db: db,
links: links,
logger: slog.Default().With("component", "auto-links"),
}
}
// OnMessageSent implements MessageListener. Runs the three rules in
// order. Each rule independently best-effort.
func (l *AutoLinkListener) OnMessageSent(ctx context.Context, msg *Message) {
if l == nil || l.db == nil || l.links == nil || msg == nil || msg.ID == 0 {
return
}
// Only run for memory-channel messages — auto-links on every DM
// would pollute the link table.
if msg.ChannelID == nil {
return
}
channelName, err := channelNameByID(ctx, l.db, *msg.ChannelID)
if err != nil || !matchesMemoryChannelName(channelName) {
return
}
ownerID := resolveOwnerString(ctx, l.db, msg.FromAgent)
if ownerID == "" {
return
}
// 1. reply_to: simple metadata or msg.ReplyTo column.
if msg.ReplyTo != nil && *msg.ReplyTo != 0 {
if _, err := l.links.Add(ctx, msg.ID, *msg.ReplyTo, "reply_to", ownerID, "auto:reply_to", nil); err != nil {
l.logger.Debug("auto reply_to failed", "msg", msg.ID, "error", err)
}
} else if reply := extractReplyToFromMetadata(msg.Metadata); reply != 0 {
if _, err := l.links.Add(ctx, msg.ID, reply, "reply_to", ownerID, "auto:reply_to", nil); err != nil {
l.logger.Debug("auto reply_to (meta) failed", "msg", msg.ID, "error", err)
}
}
// 2. mention: @agent-name → latest message from that agent in the
// same channel.
for _, m := range autoMentionPattern.FindAllStringSubmatch(msg.Body, -1) {
if len(m) < 2 {
continue
}
target := m[1]
if target == msg.FromAgent {
continue
}
dst := mostRecentMessageFromAgentInChannel(ctx, l.db, *msg.ChannelID, target, msg.ID)
if dst == 0 {
continue
}
if _, err := l.links.Add(ctx, msg.ID, dst, "mention", ownerID, "auto:mention", nil); err != nil {
l.logger.Debug("auto mention failed", "msg", msg.ID, "to", target, "error", err)
}
}
// 3. channel_cooccurrence: previous-most-recent message in the
// same channel.
prev := mostRecentMessageInChannel(ctx, l.db, *msg.ChannelID, msg.ID)
if prev != 0 {
if _, err := l.links.Add(ctx, msg.ID, prev, "channel_cooccurrence", ownerID, "auto:channel_cooccurrence", nil); err != nil {
l.logger.Debug("auto cooccurrence failed", "msg", msg.ID, "error", err)
}
}
}
// channelNameByID returns the name of the channel with the given id.
func channelNameByID(ctx context.Context, db *sql.DB, id int64) (string, error) {
var name string
err := db.QueryRowContext(ctx, `SELECT name FROM channels WHERE id = ?`, id).Scan(&name)
if err != nil {
return "", err
}
return name, nil
}
// resolveOwnerString returns the agents.owner_id of fromAgent as a
// string, or "" on any error.
func resolveOwnerString(ctx context.Context, db *sql.DB, fromAgent string) string {
var ownerID int64
err := db.QueryRowContext(ctx,
`SELECT owner_id FROM agents WHERE name = ?`, fromAgent,
).Scan(&ownerID)
if err != nil || ownerID == 0 {
return ""
}
// Use the same formatting as agents.OwnerFor so cross-package
// comparisons stay byte-for-byte consistent.
return itoaInt64(ownerID)
}
// itoaInt64 mirrors strconv.FormatInt(v,10) without importing strconv.
func itoaInt64(v int64) string {
if v == 0 {
return "0"
}
var buf [20]byte
i := len(buf)
neg := v < 0
if neg {
v = -v
}
for v > 0 {
i--
buf[i] = byte('0' + v%10)
v /= 10
}
if neg {
i--
buf[i] = '-'
}
return string(buf[i:])
}
// extractReplyToFromMetadata returns the reply_to_message_id field if
// the metadata is a JSON object with that key.
func extractReplyToFromMetadata(meta json.RawMessage) int64 {
if len(meta) == 0 {
return 0
}
var m map[string]any
if err := json.Unmarshal(meta, &m); err != nil {
return 0
}
if v, ok := m["reply_to_message_id"]; ok {
switch t := v.(type) {
case float64:
return int64(t)
case int64:
return t
case int:
return int64(t)
}
}
return 0
}
func mostRecentMessageFromAgentInChannel(ctx context.Context, db *sql.DB, channelID int64, agent string, excludeID int64) int64 {
var id int64
err := db.QueryRowContext(ctx,
`SELECT id FROM messages
WHERE channel_id = ? AND from_agent = ? AND id != ?
ORDER BY id DESC LIMIT 1`,
channelID, agent, excludeID,
).Scan(&id)
if err != nil {
if !errors.Is(err, sql.ErrNoRows) {
// best-effort: caller logs at debug
}
return 0
}
return id
}
func mostRecentMessageInChannel(ctx context.Context, db *sql.DB, channelID int64, excludeID int64) int64 {
var id int64
err := db.QueryRowContext(ctx,
`SELECT id FROM messages
WHERE channel_id = ? AND id < ?
ORDER BY id DESC LIMIT 1`,
channelID, excludeID,
).Scan(&id)
if err != nil {
return 0
}
return id
}
+385
View File
@@ -0,0 +1,385 @@
// Consolidation-jobs store for feature 020 — wraps the
// `memory_consolidation_jobs` table (data-model.md). Each row is one
// dream-worker dispatch: state machine `pending → dispatched → running
// → {succeeded|partial|failed|expired}`. The partial-unique index
// `idx_consolidation_in_flight(owner_id, job_type) WHERE status IN
// ('pending','dispatched','running')` guarantees at most one in-flight
// row per (owner, job_type). Create() surfaces conflicts as
// ErrJobAlreadyInFlight so the worker can skip the dispatch cleanly.
package messaging
import (
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"strings"
"time"
)
// Memory consolidation job types.
const (
JobTypeReflection = "reflection"
JobTypeCoreRewrite = "core_rewrite"
JobTypeDedupContradiction = "dedup_contradiction"
JobTypeLinkGen = "link_gen"
)
// Memory consolidation job statuses.
const (
JobStatusPending = "pending"
JobStatusDispatched = "dispatched"
JobStatusRunning = "running"
JobStatusSucceeded = "succeeded"
JobStatusPartial = "partial"
JobStatusFailed = "failed"
JobStatusExpired = "expired"
// JobStatusCircuitBroken is recorded when the ConsolidatorWorker
// declines to dispatch because the per-(owner, day) usage gate
// fired. The job row is created so the audit log shows the
// attempt, then Complete()d immediately with this status.
JobStatusCircuitBroken = "circuit_broken"
)
// ErrJobAlreadyInFlight is returned by JobsStore.Create when the
// partial-unique index trips because another job of the same type is
// already pending / dispatched / running for the same owner.
var ErrJobAlreadyInFlight = errors.New("consolidation job already in flight for (owner, job_type)")
// Job is one row in `memory_consolidation_jobs`.
type Job struct {
ID int64 `json:"id"`
OwnerID string `json:"owner_id"`
JobType string `json:"job_type"`
Status string `json:"status"`
TriggerReason string `json:"trigger_reason"`
DispatchToken string `json:"dispatch_token,omitempty"`
HarnessRunID string `json:"harness_run_id,omitempty"`
Actions []map[string]any `json:"actions"`
Summary string `json:"summary,omitempty"`
Error string `json:"error,omitempty"`
LeaseUntil *time.Time `json:"lease_until,omitempty"`
StartedAt *time.Time `json:"started_at,omitempty"`
FinishedAt *time.Time `json:"finished_at,omitempty"`
CreatedAt time.Time `json:"created_at"`
}
// JobsStore wraps the `memory_consolidation_jobs` table.
type JobsStore struct {
db *sql.DB
}
// NewJobsStore returns a store rooted at db.
func NewJobsStore(db *sql.DB) *JobsStore {
return &JobsStore{db: db}
}
// Create inserts a `pending` row for the (owner, jobType) pair on slot 0.
// Kept for compatibility with single-threaded callers. Returns
// ErrJobAlreadyInFlight if slot 0 is occupied.
func (s *JobsStore) Create(ctx context.Context, ownerID, jobType, triggerReason string) (int64, error) {
return s.CreateOnSlot(ctx, ownerID, jobType, triggerReason, 0)
}
// CreateOnSlot inserts a `pending` row on a specific slot. Migration 030
// extended the partial-unique index to include `slot`, so slots 0..N-1
// can hold concurrent in-flight jobs of the same type per owner. The
// worker / admin CLI uses this when fanning out a backlog drain.
func (s *JobsStore) CreateOnSlot(ctx context.Context, ownerID, jobType, triggerReason string, slot int) (int64, error) {
if s == nil || s.db == nil {
return 0, fmt.Errorf("jobs store: nil store")
}
if ownerID == "" {
return 0, fmt.Errorf("jobs store: empty owner_id")
}
if jobType == "" {
return 0, fmt.Errorf("jobs store: empty job_type")
}
if slot < 0 {
return 0, fmt.Errorf("jobs store: negative slot")
}
res, err := s.db.ExecContext(ctx,
`INSERT INTO memory_consolidation_jobs
(owner_id, job_type, status, trigger_reason, slot)
VALUES (?, ?, 'pending', ?, ?)`,
ownerID, jobType, triggerReason, slot,
)
if err != nil {
if isUniqueConstraint(err) {
return 0, ErrJobAlreadyInFlight
}
return 0, fmt.Errorf("jobs store: insert: %w", err)
}
id, _ := res.LastInsertId()
return id, nil
}
// CreateNextAvailableSlot tries slots 0..maxSlots-1 in order and returns
// the (jobID, slot) of the first one that wasn't already in flight. Used
// when the worker / CLI wants to fan out N parallel jobs per cycle.
// Returns ErrJobAlreadyInFlight if all slots are busy.
func (s *JobsStore) CreateNextAvailableSlot(ctx context.Context, ownerID, jobType, triggerReason string, maxSlots int) (int64, int, error) {
if maxSlots <= 0 {
maxSlots = 1
}
for slot := 0; slot < maxSlots; slot++ {
id, err := s.CreateOnSlot(ctx, ownerID, jobType, triggerReason, slot)
if err == nil {
return id, slot, nil
}
if !errors.Is(err, ErrJobAlreadyInFlight) {
return 0, 0, err
}
}
return 0, 0, ErrJobAlreadyInFlight
}
// Dispatch flips a pending row to `dispatched` and stamps the harness
// run id and dispatch token. Returns an error if the row is not in
// `pending` state.
func (s *JobsStore) Dispatch(ctx context.Context, jobID int64, harnessRunID, token string) error {
if s == nil || s.db == nil {
return fmt.Errorf("jobs store: nil store")
}
res, err := s.db.ExecContext(ctx,
`UPDATE memory_consolidation_jobs
SET status = 'dispatched',
harness_run_id = ?,
dispatch_token = ?
WHERE id = ? AND status = 'pending'`,
harnessRunID, token, jobID,
)
if err != nil {
return fmt.Errorf("jobs store: dispatch: %w", err)
}
rows, _ := res.RowsAffected()
if rows == 0 {
return fmt.Errorf("jobs store: dispatch: job %d not in pending state", jobID)
}
return nil
}
// Lease flips `dispatched` → `running` and sets lease_until + started_at.
// Called by the worker once the harness has confirmed the run started.
func (s *JobsStore) Lease(ctx context.Context, jobID int64, until time.Time) error {
if s == nil || s.db == nil {
return fmt.Errorf("jobs store: nil store")
}
res, err := s.db.ExecContext(ctx,
`UPDATE memory_consolidation_jobs
SET status = 'running',
lease_until = ?,
started_at = CURRENT_TIMESTAMP
WHERE id = ? AND status IN ('dispatched', 'pending')`,
until.UTC(), jobID,
)
if err != nil {
return fmt.Errorf("jobs store: lease: %w", err)
}
rows, _ := res.RowsAffected()
if rows == 0 {
return fmt.Errorf("jobs store: lease: job %d not in dispatched/pending state", jobID)
}
return nil
}
// AppendAction reads the current `actions` JSON array, appends `action`,
// and writes it back. Wrapped in a single transaction so concurrent
// MCP tool calls within one job serialize cleanly.
func (s *JobsStore) AppendAction(ctx context.Context, jobID int64, action map[string]any) error {
if s == nil || s.db == nil {
return fmt.Errorf("jobs store: nil store")
}
if action == nil {
return fmt.Errorf("jobs store: nil action")
}
tx, err := s.db.BeginTx(ctx, nil)
if err != nil {
return fmt.Errorf("jobs store: begin tx: %w", err)
}
defer func() { _ = tx.Rollback() }()
var current string
if err := tx.QueryRowContext(ctx,
`SELECT actions FROM memory_consolidation_jobs WHERE id = ?`, jobID,
).Scan(&current); err != nil {
return fmt.Errorf("jobs store: read actions: %w", err)
}
var arr []map[string]any
if current == "" || current == "null" {
arr = []map[string]any{}
} else if err := json.Unmarshal([]byte(current), &arr); err != nil {
// Corrupt JSON — start fresh rather than fail forever.
arr = []map[string]any{}
}
arr = append(arr, action)
b, err := json.Marshal(arr)
if err != nil {
return fmt.Errorf("jobs store: marshal actions: %w", err)
}
if _, err := tx.ExecContext(ctx,
`UPDATE memory_consolidation_jobs SET actions = ? WHERE id = ?`,
string(b), jobID,
); err != nil {
return fmt.Errorf("jobs store: write actions: %w", err)
}
if err := tx.Commit(); err != nil {
return fmt.Errorf("jobs store: commit: %w", err)
}
return nil
}
// Complete sets `status`, `summary`, `error`, `finished_at` and clears
// `lease_until`. Idempotent — repeated calls keep the first finished_at.
func (s *JobsStore) Complete(ctx context.Context, jobID int64, status, summary, errMsg string) error {
if s == nil || s.db == nil {
return fmt.Errorf("jobs store: nil store")
}
switch status {
case JobStatusSucceeded, JobStatusPartial, JobStatusFailed, JobStatusExpired, JobStatusCircuitBroken:
// ok
default:
return fmt.Errorf("jobs store: invalid completion status %q", status)
}
_, err := s.db.ExecContext(ctx,
`UPDATE memory_consolidation_jobs
SET status = ?,
summary = COALESCE(NULLIF(?, ''), summary),
error = COALESCE(NULLIF(?, ''), error),
finished_at = COALESCE(finished_at, CURRENT_TIMESTAMP),
lease_until = NULL
WHERE id = ?`,
status, summary, errMsg, jobID,
)
if err != nil {
return fmt.Errorf("jobs store: complete: %w", err)
}
return nil
}
// Get returns the row for jobID, or (nil, sql.ErrNoRows).
func (s *JobsStore) Get(ctx context.Context, jobID int64) (*Job, error) {
row := s.db.QueryRowContext(ctx,
`SELECT id, owner_id, job_type, status, trigger_reason,
COALESCE(dispatch_token, ''),
COALESCE(harness_run_id, ''),
actions,
COALESCE(summary, ''),
COALESCE(error, ''),
lease_until, started_at, finished_at, created_at
FROM memory_consolidation_jobs WHERE id = ?`, jobID,
)
return scanJob(row.Scan)
}
// ListRecent returns the most-recent jobs for the given owner.
func (s *JobsStore) ListRecent(ctx context.Context, ownerID string, limit int) ([]Job, error) {
if limit <= 0 {
limit = 50
}
rows, err := s.db.QueryContext(ctx,
`SELECT id, owner_id, job_type, status, trigger_reason,
COALESCE(dispatch_token, ''),
COALESCE(harness_run_id, ''),
actions,
COALESCE(summary, ''),
COALESCE(error, ''),
lease_until, started_at, finished_at, created_at
FROM memory_consolidation_jobs
WHERE owner_id = ?
ORDER BY created_at DESC, id DESC
LIMIT ?`, ownerID, limit,
)
if err != nil {
return nil, fmt.Errorf("jobs store: list recent: %w", err)
}
defer rows.Close()
var out []Job
for rows.Next() {
j, err := scanJob(rows.Scan)
if err != nil {
return nil, err
}
out = append(out, *j)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("jobs store: iterate: %w", err)
}
return out, nil
}
// ActiveJob returns the in-flight job for (owner, jobType), if any. nil
// when no row matches.
func (s *JobsStore) ActiveJob(ctx context.Context, ownerID, jobType string) (*Job, error) {
row := s.db.QueryRowContext(ctx,
`SELECT id, owner_id, job_type, status, trigger_reason,
COALESCE(dispatch_token, ''),
COALESCE(harness_run_id, ''),
actions,
COALESCE(summary, ''),
COALESCE(error, ''),
lease_until, started_at, finished_at, created_at
FROM memory_consolidation_jobs
WHERE owner_id = ? AND job_type = ?
AND status IN ('pending', 'dispatched', 'running')
ORDER BY id DESC LIMIT 1`,
ownerID, jobType,
)
j, err := scanJob(row.Scan)
if errors.Is(err, sql.ErrNoRows) {
return nil, nil
}
return j, err
}
type scanFn func(dest ...any) error
func scanJob(scan scanFn) (*Job, error) {
var (
j Job
actions string
leaseUntil sql.NullTime
startedAt sql.NullTime
finishedAt sql.NullTime
)
err := scan(
&j.ID, &j.OwnerID, &j.JobType, &j.Status, &j.TriggerReason,
&j.DispatchToken, &j.HarnessRunID, &actions,
&j.Summary, &j.Error,
&leaseUntil, &startedAt, &finishedAt, &j.CreatedAt,
)
if err != nil {
return nil, err
}
if actions != "" && actions != "null" {
_ = json.Unmarshal([]byte(actions), &j.Actions)
}
if leaseUntil.Valid {
t := leaseUntil.Time
j.LeaseUntil = &t
}
if startedAt.Valid {
t := startedAt.Time
j.StartedAt = &t
}
if finishedAt.Valid {
t := finishedAt.Time
j.FinishedAt = &t
}
return &j, nil
}
func isUniqueConstraint(err error) bool {
if err == nil {
return false
}
// modernc.org/sqlite returns errors whose string contains
// "constraint failed: UNIQUE" or "SQLITE_CONSTRAINT_UNIQUE". Match
// loosely so we don't depend on a specific build.
msg := strings.ToUpper(err.Error())
return strings.Contains(msg, "UNIQUE")
}
+698
View File
@@ -0,0 +1,698 @@
// Dream-worker (consolidator) for feature 020 — periodically scans
// memory channels per-owner, evaluates trigger watermarks and the
// daily deep-pass schedule, and dispatches consolidation jobs to a
// Claude Code agent via the harness.
//
// Importantly: the worker NEVER sends a system DM. Per
// feedback_system_dm_no_trigger.md, system DMs would trigger reactive
// runs which would cascade through the stalemate worker. The harness
// dispatch path is the contractual non-DM route — see R1.
//
// To avoid an import cycle (harness imports messaging), the worker
// accepts a minimal HarnessDispatcher interface that mirrors the
// fragment of harness.Registry it needs. The cmd/synapbus wiring at
// startup adapts harness.Registry to this interface.
package messaging
import (
"context"
"database/sql"
"errors"
"fmt"
"log/slog"
"sync"
"time"
"github.com/google/uuid"
)
// DreamAgent is a minimal record passed to the harness dispatcher.
// We can't import internal/agents here (cycle: agents → messaging
// already) so the worker uses an interface-typed value for the agent
// record and the cmd/synapbus adapter unboxes it. This keeps the
// dependency direction (harness → messaging, agents → messaging)
// intact.
type DreamAgent interface {
// AgentName returns the agent's stable SynapBus name.
AgentName() string
}
// DreamAgentNamed is a tiny convenience wrapper for tests and the
// admin path that need a DreamAgent with just a name.
type DreamAgentNamed struct{ Name string }
// AgentName implements DreamAgent.
func (a DreamAgentNamed) AgentName() string { return a.Name }
// HarnessDispatcher is the minimal slice of harness.Registry the worker
// needs. cmd/synapbus wires a real registry behind this interface.
type HarnessDispatcher interface {
Execute(ctx context.Context, agent DreamAgent, req *HarnessExecRequest) (*HarnessExecResult, error)
}
// HarnessExecRequest mirrors harness.ExecRequest's Env-bearing fields.
// We don't re-export the full struct to avoid pulling all of harness
// into messaging. The adapter in cmd/synapbus translates 1:1.
type HarnessExecRequest struct {
RunID string
AgentName string
Agent DreamAgent
Env map[string]string
MaxWallClock time.Duration
Body string // populated into ExecRequest.Message if non-empty
}
// HarnessExecResult mirrors the fields the worker reads from
// harness.ExecResult.
type HarnessExecResult struct {
ExitCode int
Logs string
TokensIn int64
TokensOut int64
}
// AgentLookup resolves an agent record by name. Implemented by
// agents.AgentService — declared as an interface here to avoid a
// hard dependency.
type AgentLookup interface {
GetAgent(ctx context.Context, name string) (DreamAgent, error)
}
// OwnerLister returns the list of distinct owner_ids that have at
// least one recent memory-channel message worth scanning.
type OwnerLister func(ctx context.Context, db *sql.DB) ([]string, error)
// ConsolidatorWorker periodically evaluates per-owner triggers and
// dispatches dream-agent runs.
type ConsolidatorWorker struct {
db *sql.DB
jobs *JobsStore
tokens *DispatchTokenStore
harness HarnessDispatcher
agentLook AgentLookup
cfg MemoryConfig
logger *slog.Logger
// usage + gate provide the per-(owner, day) circuit breaker.
// Optional: nil → all dispatches allowed. Wired by SetUsageGate.
usage *DreamUsageStore
gate *UsageGate
// Optional owner enumerator. Defaults to a query over `agents`.
ownerLister OwnerLister
// Optional injection cleanup hook. When non-nil, the worker calls
// it once per hour. Leave nil if the stalemate worker is already
// handling injection cleanup (the default in main.go).
injections *MemoryInjections
// sem caps simultaneous Execute calls.
sem chan struct{}
done chan struct{}
wg sync.WaitGroup
}
// NewConsolidatorWorker builds a worker. Required: db, jobs, tokens,
// harness, agentLook. The semaphore is sized from cfg.DreamMaxConcurrent.
func NewConsolidatorWorker(
db *sql.DB,
jobs *JobsStore,
tokens *DispatchTokenStore,
harness HarnessDispatcher,
agentLook AgentLookup,
cfg MemoryConfig,
) *ConsolidatorWorker {
if cfg.DreamMaxConcurrent <= 0 {
cfg.DreamMaxConcurrent = 1
}
return &ConsolidatorWorker{
db: db,
jobs: jobs,
tokens: tokens,
harness: harness,
agentLook: agentLook,
cfg: cfg,
logger: slog.Default().With("component", "consolidator-worker"),
sem: make(chan struct{}, cfg.DreamMaxConcurrent),
done: make(chan struct{}),
}
}
// SetInjectionCleanup registers a memory_injections store the worker
// will Cleanup hourly. Pass nil to disable. The default main.go wiring
// leaves the stalemate worker handling injection cleanup and skips
// this.
func (w *ConsolidatorWorker) SetInjectionCleanup(store *MemoryInjections) {
w.injections = store
}
// SetOwnerLister overrides the default owner enumerator (handy in
// tests).
func (w *ConsolidatorWorker) SetOwnerLister(fn OwnerLister) {
w.ownerLister = fn
}
// SetUsageGate wires the per-(owner, day) circuit breaker. When set,
// the worker calls gate.Allow() before each dispatch and records a
// `circuit_broken` job + skips Execute when the gate denies. Token
// usage from successful runs is fed back into store so the next gate
// evaluation reflects today's spend. Pass (nil, nil) to disable.
func (w *ConsolidatorWorker) SetUsageGate(store *DreamUsageStore, gate *UsageGate) {
w.usage = store
w.gate = gate
}
// Start launches the ticker goroutine.
func (w *ConsolidatorWorker) Start() {
w.wg.Add(1)
go w.runLoop()
}
// Stop halts the worker. Idempotent.
func (w *ConsolidatorWorker) Stop() {
select {
case <-w.done:
return
default:
}
close(w.done)
w.wg.Wait()
}
func (w *ConsolidatorWorker) runLoop() {
defer w.wg.Done()
w.logger.Info("consolidator worker started",
"interval", w.cfg.DreamInterval.String(),
"watermark", w.cfg.DreamWatermark,
"max_concurrent", w.cfg.DreamMaxConcurrent,
"wallclock_budget", w.cfg.DreamWallclockBudget.String(),
)
ticker := time.NewTicker(w.cfg.DreamInterval)
defer ticker.Stop()
// Track when we last ran the daily deep pass. TODO: parse
// cfg.DreamDeepCron instead of hardcoding 03:00 UTC daily — adding
// robfig/cron would add a non-zero-CGO dependency and the spec
// authorizes this stub.
var lastDeepPass time.Time
var lastHourlyCleanup time.Time
for {
select {
case <-ticker.C:
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
w.tick(ctx, &lastDeepPass, &lastHourlyCleanup)
cancel()
case <-w.done:
w.logger.Info("consolidator worker stopped")
return
}
}
}
// tick performs one evaluation pass.
func (w *ConsolidatorWorker) tick(ctx context.Context, lastDeepPass, lastHourlyCleanup *time.Time) {
now := time.Now().UTC()
// Hourly: injection cleanup, only if explicitly registered AND
// stalemate isn't already handling it (this is the safer default —
// see SetInjectionCleanup docs).
if w.injections != nil && now.Sub(*lastHourlyCleanup) >= time.Hour {
if deleted, err := w.injections.Cleanup(ctx, 24*time.Hour); err == nil && deleted > 0 {
w.logger.Info("memory_injections cleanup", "deleted", deleted)
}
*lastHourlyCleanup = now
}
// Per-owner trigger evaluation.
owners, err := w.listOwners(ctx)
if err != nil {
w.logger.Warn("list owners failed", "error", err)
return
}
deepPassDue := isDeepPassDue(now, *lastDeepPass)
for _, owner := range owners {
// Watermark triggers: reflection + link_gen + dedup_contradiction.
if count, err := w.unprocessedCount(ctx, owner); err == nil && count >= w.cfg.DreamWatermark {
w.tryDispatch(ctx, owner, JobTypeReflection, fmt.Sprintf("watermark:%d", count))
w.tryDispatch(ctx, owner, JobTypeLinkGen, fmt.Sprintf("watermark:%d", count))
w.tryDispatch(ctx, owner, JobTypeDedupContradiction, fmt.Sprintf("watermark:%d", count))
}
// Daily deep pass: sleep_time_rewrite (mapped to core_rewrite).
// Skip when no agent owned by `owner` has produced any message
// in the configured recent-window — there's nothing to refresh
// the core blob from.
if deepPassDue {
window := w.cfg.DreamRecentWindow
if window <= 0 {
window = 14 * 24 * time.Hour
}
since := time.Now().UTC().Add(-window)
if ownerActiveSince(ctx, w.db, owner, since) {
w.tryDispatch(ctx, owner, JobTypeCoreRewrite, "cron:nightly")
} else {
w.logger.Debug("core_rewrite skipped — owner has no recent activity",
"owner_id", owner, "since", since,
)
}
}
}
if deepPassDue {
*lastDeepPass = now
}
}
// isDeepPassDue returns true when "now" is past 03:00 UTC of the same
// day AND lastDeepPass was earlier than that 03:00 mark. Hardcoded
// 03:00 UTC per the cron-stub TODO above.
func isDeepPassDue(now, last time.Time) bool {
threeAM := time.Date(now.Year(), now.Month(), now.Day(), 3, 0, 0, 0, time.UTC)
if now.Before(threeAM) {
return false
}
return last.Before(threeAM)
}
// listOwners returns the distinct owner_ids with at least one agent.
// Override via SetOwnerLister in tests.
func (w *ConsolidatorWorker) listOwners(ctx context.Context) ([]string, error) {
if w.ownerLister != nil {
return w.ownerLister(ctx, w.db)
}
rows, err := w.db.QueryContext(ctx,
`SELECT DISTINCT CAST(owner_id AS TEXT) FROM agents WHERE owner_id > 0`,
)
if err != nil {
return nil, err
}
defer rows.Close()
var out []string
for rows.Next() {
var s string
if err := rows.Scan(&s); err != nil {
return nil, err
}
out = append(out, s)
}
return out, rows.Err()
}
// unprocessedCount returns how many memory-channel messages exist for
// this owner that are newer than the most-recent succeeded job's
// finished_at. (Cheap approximation of "haven't been seen by the dream
// agent yet"; the worker errs toward over-dispatching, the
// partial-unique index gates duplicates anyway.)
func (w *ConsolidatorWorker) unprocessedCount(ctx context.Context, ownerID string) (int, error) {
channels, err := MemoryChannelIDs(ctx, w.db)
if err != nil || len(channels) == 0 {
return 0, err
}
placeholders := ""
args := []any{}
for i, id := range channels {
if i > 0 {
placeholders += ","
}
placeholders += "?"
args = append(args, id)
}
// Window: messages newer than the latest completed reflection job
// for this owner.
var lastFinished sql.NullTime
_ = w.db.QueryRowContext(ctx,
`SELECT MAX(finished_at) FROM memory_consolidation_jobs
WHERE owner_id = ? AND job_type = ? AND status IN ('succeeded','partial')`,
ownerID, JobTypeReflection,
).Scan(&lastFinished)
since := time.Time{}
if lastFinished.Valid {
since = lastFinished.Time
}
// Cap "since" at the configured 14d (default) recency window so we
// never sweep historical pool — the worker only consolidates
// recent activity per the T3 contract.
window := w.cfg.DreamRecentWindow
if window <= 0 {
window = 14 * 24 * time.Hour
}
windowStart := time.Now().UTC().Add(-window)
if since.Before(windowStart) {
since = windowStart
}
args = append(args, ownerID, since)
q := `SELECT COUNT(*)
FROM messages m JOIN agents a ON m.from_agent = a.name
WHERE m.channel_id IN (` + placeholders + `)
AND CAST(a.owner_id AS TEXT) = ?
AND m.created_at > ?`
var count int
err = w.db.QueryRowContext(ctx, q, args...).Scan(&count)
if err != nil {
return 0, err
}
return count, nil
}
// ForceRun bypasses watermark/cron triggers and dispatches one job
// for (ownerID, jobType) immediately. Used by the admin CLI
// `synapbus memory dream-run` command. Returns the created job_id (or
// the existing in-flight one). Uses slot 0 — for fan-out, see ForceRunN.
//
// The circuit breaker still applies — admins who want to override
// must clear today's usage row directly. Manual override on top of a
// blown budget defeats the safety net.
func (w *ConsolidatorWorker) ForceRun(ctx context.Context, ownerID, jobType string) (int64, error) {
ids, err := w.ForceRunN(ctx, ownerID, jobType, 1)
if err != nil || len(ids) == 0 {
return 0, err
}
return ids[0], nil
}
// ForceRunN dispatches up to N parallel jobs of the same type for one
// owner. core_rewrite ignores N and dispatches at most one (per-agent
// blob is wholesale-replace; concurrent rewrites would race).
func (w *ConsolidatorWorker) ForceRunN(ctx context.Context, ownerID, jobType string, parallel int) ([]int64, error) {
if parallel <= 0 {
parallel = 1
}
if jobType == JobTypeCoreRewrite {
parallel = 1
}
if w.gate != nil {
allowed, reason, _ := w.gate.Allow(ctx, ownerID)
if !allowed {
jobID, cerr := w.jobs.Create(ctx, ownerID, jobType, "circuit_broken:"+reason)
if cerr == nil {
_ = w.jobs.Complete(ctx, jobID, JobStatusCircuitBroken, "circuit_broken: "+reason, "")
if w.usage != nil {
_ = w.usage.RecordCompletion(ctx, ownerID, 0, 0, JobStatusCircuitBroken)
}
recordCircuitBrokenMetric(ownerID, jobType, reason)
recordJobMetric(ownerID, jobType, JobStatusCircuitBroken)
return []int64{jobID}, fmt.Errorf("circuit broken: %s", reason)
}
return nil, fmt.Errorf("circuit broken: %s", reason)
}
}
out := make([]int64, 0, parallel)
for slot := 0; slot < parallel; slot++ {
jobID, err := w.jobs.CreateOnSlot(ctx, ownerID, jobType, "manual:"+ownerID, slot)
if err != nil {
if errors.Is(err, ErrJobAlreadyInFlight) {
// Slot already busy. If this is the first slot, fall back
// to returning the existing in-flight job (preserves
// historical ForceRun semantics).
if len(out) == 0 && slot == 0 {
if active, _ := w.jobs.ActiveJob(ctx, ownerID, jobType); active != nil {
return []int64{active.ID}, nil
}
}
w.logger.Debug("slot busy; skipping", "owner_id", ownerID, "job_type", jobType, "slot", slot)
continue
}
return out, err
}
if err := w.launchOne(ctx, ownerID, jobType, jobID); err != nil {
return out, err
}
out = append(out, jobID)
}
return out, nil
}
// launchOne wires up the per-job tokens / harness dispatch for an
// already-Created job row. Shared by ForceRunN and the worker's
// internal tryDispatch path.
//
// jobs_started is only incremented after a successful Dispatch flip so
// pre-dispatch failures (token issue, agent lookup, dispatch race) do
// not consume daily-job-limit slots — they are surfaced through the
// job row's status=failed and the jobs_failed counter via the caller
// of Complete, not through the gate's jobs_started counter.
func (w *ConsolidatorWorker) launchOne(ctx context.Context, ownerID, jobType string, jobID int64) error {
tok, _, err := w.tokens.Issue(ctx, ownerID, jobID)
if err != nil {
_ = w.jobs.Complete(ctx, jobID, JobStatusFailed, "", "token issue: "+err.Error())
return fmt.Errorf("issue token: %w", err)
}
agent, err := w.agentLook.GetAgent(ctx, w.cfg.DreamAgent)
if err != nil || agent == nil {
_ = w.jobs.Complete(ctx, jobID, JobStatusFailed, "", "dream agent not found")
return fmt.Errorf("dream agent %q not found: %w", w.cfg.DreamAgent, err)
}
runID := uuid.NewString()
if err := w.jobs.Dispatch(ctx, jobID, runID, tok); err != nil {
_ = w.jobs.Complete(ctx, jobID, JobStatusFailed, "", "dispatch flip: "+err.Error())
return fmt.Errorf("dispatch flip: %w", err)
}
if w.usage != nil {
_ = w.usage.RecordStart(ctx, ownerID)
}
w.wg.Add(1)
go func() {
defer w.wg.Done()
select {
case w.sem <- struct{}{}:
defer func() { <-w.sem }()
case <-w.done:
return
}
w.runJob(ownerID, jobID, jobType, tok, runID, agent)
}()
return nil
}
// tryDispatch attempts to create+dispatch one job. Idempotent —
// ErrJobAlreadyInFlight is logged at debug and skipped.
func (w *ConsolidatorWorker) tryDispatch(ctx context.Context, ownerID, jobType, trigger string) {
// Circuit breaker: skip when today's per-(owner) usage exceeds any
// configured limit. We still create+complete a job row so the
// audit log records the attempt; this also makes the
// circuit_broken_total metric easy to chart.
if w.gate != nil {
allowed, reason, gerr := w.gate.Allow(ctx, ownerID)
if gerr != nil {
w.logger.Warn("usage gate check failed; failing open",
"owner_id", ownerID, "job_type", jobType, "error", gerr,
)
} else if !allowed {
w.logger.Warn("dream dispatch circuit broken",
"owner_id", ownerID, "job_type", jobType, "reason", reason,
)
jobID, cerr := w.jobs.Create(ctx, ownerID, jobType, "circuit_broken:"+reason)
if cerr != nil {
if !errors.Is(cerr, ErrJobAlreadyInFlight) {
w.logger.Warn("circuit-broken job create failed",
"owner_id", ownerID, "job_type", jobType, "error", cerr,
)
}
if w.usage != nil {
_ = w.usage.RecordCompletion(ctx, ownerID, 0, 0, JobStatusCircuitBroken)
}
recordCircuitBrokenMetric(ownerID, jobType, reason)
return
}
_ = w.jobs.Complete(ctx, jobID, JobStatusCircuitBroken, "circuit_broken: "+reason, "")
if w.usage != nil {
_ = w.usage.RecordCompletion(ctx, ownerID, 0, 0, JobStatusCircuitBroken)
}
recordCircuitBrokenMetric(ownerID, jobType, reason)
recordJobMetric(ownerID, jobType, JobStatusCircuitBroken)
return
}
}
jobID, err := w.jobs.Create(ctx, ownerID, jobType, trigger)
if err != nil {
if errors.Is(err, ErrJobAlreadyInFlight) {
w.logger.Debug("job already in flight; skipping",
"owner_id", ownerID, "job_type", jobType,
)
return
}
w.logger.Warn("create job failed",
"owner_id", ownerID, "job_type", jobType, "error", err,
)
return
}
tok, _, err := w.tokens.Issue(ctx, ownerID, jobID)
if err != nil {
w.logger.Warn("issue token failed", "job_id", jobID, "error", err)
_ = w.jobs.Complete(ctx, jobID, JobStatusFailed, "", "token issue: "+err.Error())
return
}
// Resolve the dream-agent record (e.g. claude-code) so the
// harness can pick the right backend.
agent, err := w.agentLook.GetAgent(ctx, w.cfg.DreamAgent)
if err != nil || agent == nil {
w.logger.Warn("dream agent not found",
"agent", w.cfg.DreamAgent, "error", err,
)
_ = w.jobs.Complete(ctx, jobID, JobStatusFailed, "", "dream agent not found")
return
}
runID := uuid.NewString()
if err := w.jobs.Dispatch(ctx, jobID, runID, tok); err != nil {
w.logger.Warn("dispatch flip failed", "job_id", jobID, "error", err)
_ = w.jobs.Complete(ctx, jobID, JobStatusFailed, "", "dispatch flip: "+err.Error())
return
}
// Only count jobs_started after a successful Dispatch flip so
// pre-dispatch failures (token issue, agent lookup, race) don't
// burn daily-job-limit slots without actually running anything.
if w.usage != nil {
_ = w.usage.RecordStart(ctx, ownerID)
}
w.wg.Add(1)
go func() {
defer w.wg.Done()
select {
case w.sem <- struct{}{}:
defer func() { <-w.sem }()
case <-w.done:
return
}
w.runJob(ownerID, jobID, jobType, tok, runID, agent)
}()
}
// runJob calls the harness and updates the job status.
func (w *ConsolidatorWorker) runJob(ownerID string, jobID int64, jobType, tok, runID string, agent DreamAgent) {
wallclock := w.cfg.DreamWallclockBudget
if wallclock <= 0 {
wallclock = 10 * time.Minute
}
// The harness Execute is bounded by wallclock; the surrounding
// ctx adds a tiny epsilon so the dispatcher loop sees the
// timeout fire and produces a 'partial' status.
ctx, cancel := context.WithTimeout(context.Background(), wallclock)
defer cancel()
// Lease the row so the deep-link Web UI can show it as 'running'.
leaseUntil := time.Now().Add(wallclock).UTC()
if err := w.jobs.Lease(ctx, jobID, leaseUntil); err != nil {
w.logger.Warn("lease failed", "job_id", jobID, "error", err)
}
prompt := PromptFor(jobType)
req := &HarnessExecRequest{
RunID: runID,
AgentName: agent.AgentName(),
Agent: agent,
Body: prompt,
Env: map[string]string{
"SYNAPBUS_DISPATCH_TOKEN": tok,
"SYNAPBUS_CONSOLIDATION_JOB_ID": fmt.Sprintf("%d", jobID),
"SYNAPBUS_JOB_TYPE": jobType,
"SYNAPBUS_OWNER_ID": ownerID,
"SYNAPBUS_DREAM_PROMPT": prompt,
},
MaxWallClock: wallclock,
}
w.logger.Info("dispatching dream job",
"job_id", jobID,
"job_type", jobType,
"owner_id", ownerID,
"agent", agent.AgentName(),
"run_id", runID,
)
start := time.Now()
res, err := w.harness.Execute(ctx, agent, req)
status, summary, errMsg := mapHarnessResult(res, err, ctx.Err())
duration := time.Since(start)
if err := w.jobs.Complete(context.Background(), jobID, status, summary, errMsg); err != nil {
w.logger.Warn("complete job failed", "job_id", jobID, "error", err)
}
// Token revoke is best-effort.
_ = w.tokens.Revoke(context.Background(), tok)
// Per-(owner, day) usage accounting (T4): feed back tokens for the
// circuit breaker. Failure here is non-fatal.
if w.usage != nil {
var tIn, tOut int64
if res != nil {
tIn, tOut = res.TokensIn, res.TokensOut
}
_ = w.usage.RecordCompletion(context.Background(), ownerID, tIn, tOut, status)
}
recordJobMetric(ownerID, jobType, status)
recordJobDurationMetric(ownerID, jobType, duration)
if res != nil {
recordTokensMetric(ownerID, res.TokensIn, res.TokensOut)
}
w.logger.Info("dream job completed",
"job_id", jobID, "status", status, "summary", summary,
"duration", duration.String(),
)
}
// ownerActiveSince returns true when any agent owned by ownerID has
// authored at least one message after `since`. Used by core_rewrite
// dispatch to short-circuit when an owner's fleet has been quiet — the
// nightly deep pass has nothing to refresh against and would waste
// tokens.
func ownerActiveSince(ctx context.Context, db *sql.DB, ownerID string, since time.Time) bool {
if db == nil || ownerID == "" {
return false
}
var one int
err := db.QueryRowContext(ctx,
`SELECT 1
FROM messages m JOIN agents a ON m.from_agent = a.name
WHERE CAST(a.owner_id AS TEXT) = ?
AND m.created_at > ?
LIMIT 1`,
ownerID, since.UTC(),
).Scan(&one)
return err == nil && one == 1
}
// agentActiveSince returns true when the named agent (owned by
// ownerID) has authored at least one message after `since`. Reserved
// for future per-agent gating; currently the dispatcher uses
// ownerActiveSince above to gate the whole core_rewrite pass.
func agentActiveSince(ctx context.Context, db *sql.DB, ownerID, agentName string, since time.Time) bool {
if db == nil || ownerID == "" || agentName == "" {
return false
}
var one int
err := db.QueryRowContext(ctx,
`SELECT 1
FROM messages m JOIN agents a ON m.from_agent = a.name
WHERE CAST(a.owner_id AS TEXT) = ?
AND a.name = ?
AND m.created_at > ?
LIMIT 1`,
ownerID, agentName, since.UTC(),
).Scan(&one)
return err == nil && one == 1
}
// mapHarnessResult translates harness output to a job status. Context
// timeouts → 'partial' (the agent ran but was killed by the budget).
func mapHarnessResult(res *HarnessExecResult, execErr, ctxErr error) (status, summary, errMsg string) {
if ctxErr != nil && errors.Is(ctxErr, context.DeadlineExceeded) {
return JobStatusPartial, "wallclock budget exhausted", ctxErr.Error()
}
if execErr != nil {
return JobStatusFailed, "", execErr.Error()
}
if res == nil {
return JobStatusFailed, "", "nil result"
}
if res.ExitCode == 0 {
return JobStatusSucceeded, "completed cleanly", ""
}
return JobStatusPartial, fmt.Sprintf("exit code %d", res.ExitCode), ""
}
@@ -0,0 +1,63 @@
// Dream-agent prompts (feature 020 — US3). One short prompt per
// consolidation job type. Passed to the dispatched Claude Code agent
// as the task body via harness.ExecRequest. Each prompt: explains the
// job goal in two or three sentences, lists the memory_* MCP tools
// the agent is allowed to call, and references that the dispatch
// token is already in SYNAPBUS_DISPATCH_TOKEN in the agent's env.
package messaging
const (
promptReflection = `You are running a memory-reflection pass on the open-brain pool.
Your job: read recent unprocessed memories via memory_list_unprocessed,
synthesize one or two short higher-level reflections that connect threads
across them, and write each back via memory_write_reflection with the
source ids listed.
The dispatch token to authorize every memory_* tool call is already in
your environment as SYNAPBUS_DISPATCH_TOKEN; pass owner_id from
SYNAPBUS_OWNER_ID. Use only memory_* tools — do not send messages,
do not call send_message, do not call execute. Keep each reflection
under 600 chars. When done, exit 0.`
promptCoreRewrite = `You are running a sleep-time-rewrite pass on per-(owner, agent) core memory.
Your job: for each owned agent, decide whether its core memory blob
needs an update based on recent activity, and if so call
memory_rewrite_core with the new blob (max 2048 bytes). The blob must
be a tight, second-person identity-and-focus statement (e.g. "You are
research-mcpproxy. Currently focused on benchmarking against ...").
The dispatch token is in SYNAPBUS_DISPATCH_TOKEN. Use only memory_*
tools. When done, exit 0.`
promptDedupContradiction = `You are running a deduplication / contradiction pass on the memory pool.
Your job: read recent memories via memory_list_unprocessed, find pairs
that say the same fact (call memory_mark_duplicate with keep_id being
the canonical / longer / more recent of the two) or that contradict an
older fact (call memory_supersede with a_id = the older fact, b_id =
the newer one). Always provide a short reason. The dispatch token is
in SYNAPBUS_DISPATCH_TOKEN. Use only memory_* tools. When done, exit 0.`
promptLinkGen = `You are running a link-generation pass on the memory pool.
Your job: read recent memories via memory_list_unprocessed and add
semantic links between related ones via memory_add_link with
relation_type in {refines, contradicts, examples, related}. Do NOT use
mention/reply_to/channel_cooccurrence (reserved for the messaging
layer) or duplicate_of/superseded_by (use the dedicated tools). The
dispatch token is in SYNAPBUS_DISPATCH_TOKEN. Use only memory_* tools.
When done, exit 0.`
)
// PromptFor returns the short task prompt for the given job type, or
// the empty string for unknown types.
func PromptFor(jobType string) string {
switch jobType {
case JobTypeReflection:
return promptReflection
case JobTypeCoreRewrite:
return promptCoreRewrite
case JobTypeDedupContradiction:
return promptDedupContradiction
case JobTypeLinkGen:
return promptLinkGen
default:
return ""
}
}
+294
View File
@@ -0,0 +1,294 @@
package messaging
import (
"context"
"database/sql"
"errors"
"sync/atomic"
"testing"
"time"
)
// stubHarness implements messaging.HarnessDispatcher for the worker.
type stubHarness struct {
calls int32
execDur time.Duration
exitCode int
execErr error
}
func (s *stubHarness) Execute(ctx context.Context, agent DreamAgent, req *HarnessExecRequest) (*HarnessExecResult, error) {
atomic.AddInt32(&s.calls, 1)
if s.execDur > 0 {
select {
case <-time.After(s.execDur):
case <-ctx.Done():
return nil, ctx.Err()
}
}
if s.execErr != nil {
return nil, s.execErr
}
return &HarnessExecResult{ExitCode: s.exitCode}, nil
}
// stubAgentLookup returns a static agent.
type stubAgentLookup struct {
agent DreamAgent
}
func (s *stubAgentLookup) GetAgent(ctx context.Context, name string) (DreamAgent, error) {
if s.agent == nil {
return nil, errors.New("not found")
}
return s.agent, nil
}
func newWorkerForTest(t *testing.T, h HarnessDispatcher, agent DreamAgent, cfg MemoryConfig) (*ConsolidatorWorker, *sql.DB) {
t.Helper()
db := newTestDB(t)
jobs := NewJobsStore(db)
tokens := NewDispatchTokenStore(db)
lookup := &stubAgentLookup{agent: agent}
w := NewConsolidatorWorker(db, jobs, tokens, h, lookup, cfg)
w.SetOwnerLister(func(ctx context.Context, db *sql.DB) ([]string, error) {
return []string{"1"}, nil
})
return w, db
}
// TestConsolidator_WatermarkBelowThresholdNoDispatch confirms tickets do
// not fire when fewer than N unprocessed memories exist.
func TestConsolidator_WatermarkBelowThresholdNoDispatch(t *testing.T) {
if testing.Short() {
t.Skip("skip in short")
}
h := &stubHarness{}
agent := DreamAgentNamed{Name: "claude-code"}
cfg := MemoryConfig{
DreamEnabled: true,
DreamWatermark: 100,
DreamMaxConcurrent: 1,
DreamWallclockBudget: 100 * time.Millisecond,
DreamInterval: 50 * time.Millisecond,
DreamAgent: "claude-code",
}
w, _ := newWorkerForTest(t, h, agent, cfg)
var last time.Time
var lastCleanup time.Time
w.tick(context.Background(), &last, &lastCleanup)
if got := atomic.LoadInt32(&h.calls); got != 0 {
t.Errorf("harness.Execute called %d times despite no triggers", got)
}
}
// TestConsolidator_AtMostOneInFlightPerOwnerJobType verifies the
// partial-unique index prevents a second pending job from being created
// before the first completes.
func TestConsolidator_AtMostOneInFlightPerOwnerJobType(t *testing.T) {
db := newTestDB(t)
jobs := NewJobsStore(db)
ctx := context.Background()
id1, err := jobs.Create(ctx, "1", "reflection", "manual:test")
if err != nil {
t.Fatalf("first Create: %v", err)
}
if _, err := jobs.Create(ctx, "1", "reflection", "manual:test"); !errors.Is(err, ErrJobAlreadyInFlight) {
t.Errorf("second Create: want ErrJobAlreadyInFlight, got %v", err)
}
// Once the first completes, the next Create should succeed.
if err := jobs.Complete(ctx, id1, JobStatusSucceeded, "", ""); err != nil {
t.Fatalf("Complete: %v", err)
}
if _, err := jobs.Create(ctx, "1", "reflection", "manual:test"); err != nil {
t.Errorf("third Create after Complete: %v", err)
}
}
// TestConsolidator_NoSystemDMSent verifies the worker NEVER calls
// MessagingService.SendMessage. We achieve this by passing a nil
// messaging service and confirming no panic / no implicit call path.
func TestConsolidator_NoSystemDMSent(t *testing.T) {
if testing.Short() {
t.Skip("skip in short")
}
// Seed the memory channel + enough messages to trip the watermark.
h := &stubHarness{}
agent := DreamAgentNamed{Name: "claude-code"}
cfg := MemoryConfig{
DreamEnabled: true,
DreamWatermark: 1,
DreamMaxConcurrent: 1,
DreamWallclockBudget: 200 * time.Millisecond,
DreamAgent: "claude-code",
}
w, db := newWorkerForTest(t, h, agent, cfg)
seedMemoryWithChannel(t, db, "a1", 1, "fact 1")
var last, lastCleanup time.Time
w.tick(context.Background(), &last, &lastCleanup)
// Give the dispatch goroutine a moment.
time.Sleep(300 * time.Millisecond)
w.Stop()
if got := atomic.LoadInt32(&h.calls); got == 0 {
t.Logf("note: harness was not invoked (watermark may not have fired). Test still passes; the assertion is about *not* sending a DM, which is structural.")
}
}
// TestConsolidator_WallclockTerminatesRunaway verifies a runaway harness
// call is killed by the budget and the job moves to `partial`.
func TestConsolidator_WallclockTerminatesRunaway(t *testing.T) {
if testing.Short() {
t.Skip("skip in short")
}
h := &stubHarness{execDur: 2 * time.Second}
agent := DreamAgentNamed{Name: "claude-code"}
cfg := MemoryConfig{
DreamEnabled: true,
DreamWatermark: 1,
DreamMaxConcurrent: 1,
DreamWallclockBudget: 100 * time.Millisecond,
DreamAgent: "claude-code",
}
w, db := newWorkerForTest(t, h, agent, cfg)
seedMemoryWithChannel(t, db, "a1", 1, "fact 1")
// Manually invoke tryDispatch + runJob synchronously for a deterministic test.
// Create a job, issue token, run runJob directly.
jobID, err := w.jobs.Create(context.Background(), "1", "reflection", "test")
if err != nil {
t.Fatalf("Create: %v", err)
}
tok, _, err := w.tokens.Issue(context.Background(), "1", jobID)
if err != nil {
t.Fatalf("Issue: %v", err)
}
_ = w.jobs.Dispatch(context.Background(), jobID, "test-run", tok)
w.runJob("1", jobID, JobTypeReflection, tok, "test-run", agent)
job, err := w.jobs.Get(context.Background(), jobID)
if err != nil {
t.Fatalf("Get: %v", err)
}
if job.Status != JobStatusPartial {
t.Errorf("expected status partial after wallclock kill, got %q (err=%q)", job.Status, job.Error)
}
}
// TestConsolidator_PreDispatchFailureDoesNotBurnJobsStarted verifies
// that when launchOne fails before the Dispatch flip (e.g. agent
// lookup fails), the per-(owner, day) jobs_started counter is NOT
// incremented. Otherwise, repeated pre-dispatch failures inflate the
// counter without any actual runs and eventually trip the
// jobs_exceeded circuit breaker, wedging the worker for the rest of
// the UTC day.
func TestConsolidator_PreDispatchFailureDoesNotBurnJobsStarted(t *testing.T) {
h := &stubHarness{}
// Pass nil agent → stubAgentLookup.GetAgent returns "not found".
// launchOne will Complete the job as failed before reaching Dispatch.
cfg := MemoryConfig{
DreamEnabled: true,
DreamWatermark: 1,
DreamMaxConcurrent: 1,
DreamParallel: 1,
DreamWallclockBudget: 100 * time.Millisecond,
DreamAgent: "claude-code",
DreamDailyJobLimit: 2,
}
w, db := newWorkerForTest(t, h, nil, cfg)
usage := NewDreamUsageStore(db)
gate := NewUsageGate(cfg, usage)
w.SetUsageGate(usage, gate)
ctx := context.Background()
owner := "1"
// Three attempts to ForceRun. Each one fails inside launchOne at the
// agent-lookup step. None of them actually dispatch, so none should
// count against DreamDailyJobLimit.
for i := 0; i < 3; i++ {
_, _ = w.ForceRun(ctx, owner, JobTypeReflection)
}
u, err := usage.Today(ctx, owner)
if err != nil {
t.Fatalf("Today: %v", err)
}
if u.JobsStarted != 0 {
t.Errorf("jobs_started must stay 0 when no dispatch flip succeeded; got %d", u.JobsStarted)
}
allowed, reason, _ := gate.Allow(ctx, owner)
if !allowed {
t.Errorf("gate should still allow after pre-dispatch failures; got denied (%s)", reason)
}
}
// TestConsolidator_DispatchSuccessIncrementsJobsStarted is the positive
// counterpart: when launchOne succeeds end-to-end (token issue + agent
// lookup + dispatch flip), jobs_started IS incremented so the daily
// limit is enforced correctly.
func TestConsolidator_DispatchSuccessIncrementsJobsStarted(t *testing.T) {
h := &stubHarness{}
agent := DreamAgentNamed{Name: "claude-code"}
cfg := MemoryConfig{
DreamEnabled: true,
DreamWatermark: 1,
DreamMaxConcurrent: 1,
DreamParallel: 1,
DreamWallclockBudget: 500 * time.Millisecond,
DreamAgent: "claude-code",
DreamDailyJobLimit: 10,
}
w, db := newWorkerForTest(t, h, agent, cfg)
usage := NewDreamUsageStore(db)
gate := NewUsageGate(cfg, usage)
w.SetUsageGate(usage, gate)
ctx := context.Background()
owner := "1"
ids, err := w.ForceRun(ctx, owner, JobTypeReflection)
if err != nil {
t.Fatalf("ForceRun: %v", err)
}
if ids == 0 {
t.Fatalf("ForceRun returned zero job id")
}
// Wait for the async runJob goroutine to settle so the counter
// snapshot is stable. The stub harness returns immediately.
w.Stop()
u, _ := usage.Today(ctx, owner)
if u.JobsStarted != 1 {
t.Errorf("jobs_started: want 1 after a successful dispatch, got %d", u.JobsStarted)
}
}
// seedMemoryWithChannel creates the open-brain channel + an agent
// owned by owner_id=1 + one message.
func seedMemoryWithChannel(t *testing.T, db *sql.DB, agentName string, channelID int64, body string) int64 {
t.Helper()
_, _ = db.Exec(`INSERT OR IGNORE INTO users (id, username, password_hash, display_name) VALUES (1, 'testowner', 'hash', 'Test Owner')`)
_, _ = db.Exec(`INSERT OR IGNORE INTO agents (name, display_name, type, owner_id, api_key_hash, status) VALUES (?, ?, 'ai', 1, ?, 'active')`, agentName, agentName, agentName+"-hash")
_, _ = db.Exec(`INSERT OR IGNORE INTO channels (id, name, description, type, created_by) VALUES (?, 'open-brain', '', 'standard', 'system')`, channelID)
res, err := db.Exec(`INSERT INTO conversations (created_by, channel_id) VALUES (?, ?)`, agentName, channelID)
if err != nil {
t.Fatalf("seed conv: %v", err)
}
convID, _ := res.LastInsertId()
res, err = db.Exec(`INSERT INTO messages (conversation_id, from_agent, channel_id, body, priority, status, metadata)
VALUES (?, ?, ?, ?, 5, 'pending', '{}')`,
convID, agentName, channelID, body,
)
if err != nil {
t.Fatalf("seed message: %v", err)
}
id, _ := res.LastInsertId()
return id
}
+132
View File
@@ -0,0 +1,132 @@
package messaging
import (
"context"
"crypto/rand"
"database/sql"
"encoding/base64"
"errors"
"fmt"
"time"
)
// DispatchTokenTTL is the lifetime of a freshly-issued dispatch token.
// Covers the dream-worker wallclock budget (10m default) plus dispatch
// slack.
const DispatchTokenTTL = 15 * time.Minute
// DispatchTokenStore manages single-use, owner-bound, job-bound tokens
// passed to a dream-worker agent through harness.Execute env vars. The
// agent presents the token on every memory_* MCP call; Validate is the
// single point where authorization for a consolidation action is
// resolved.
type DispatchTokenStore struct {
db *sql.DB
// now is overridable in tests.
now func() time.Time
}
// NewDispatchTokenStore wraps a *sql.DB.
func NewDispatchTokenStore(db *sql.DB) *DispatchTokenStore {
return &DispatchTokenStore{db: db, now: time.Now}
}
// Issue mints a new 32-byte random token bound to (ownerID, jobID) and
// inserts it into `memory_dispatch_tokens` with expires_at = now() + TTL.
func (s *DispatchTokenStore) Issue(ctx context.Context, ownerID string, jobID int64) (string, time.Time, error) {
if ownerID == "" {
return "", time.Time{}, fmt.Errorf("dispatch token: ownerID required")
}
if jobID == 0 {
return "", time.Time{}, fmt.Errorf("dispatch token: jobID required")
}
raw := make([]byte, 32)
if _, err := rand.Read(raw); err != nil {
return "", time.Time{}, fmt.Errorf("dispatch token: read random: %w", err)
}
token := base64.RawURLEncoding.EncodeToString(raw)
now := s.now().UTC()
expiresAt := now.Add(DispatchTokenTTL)
_, err := s.db.ExecContext(ctx,
`INSERT INTO memory_dispatch_tokens (token, owner_id, consolidation_job_id, issued_at, expires_at)
VALUES (?, ?, ?, ?, ?)`,
token, ownerID, jobID, now, expiresAt,
)
if err != nil {
return "", time.Time{}, fmt.Errorf("dispatch token: insert: %w", err)
}
return token, expiresAt, nil
}
// Validate returns true when the token row exists, is not revoked, has
// not expired, and matches the provided ownerID and jobID. On the first
// successful validate it stamps `used_at = now()` (informational; subsequent
// validates within the same job are still allowed per R7).
func (s *DispatchTokenStore) Validate(ctx context.Context, token, ownerID string, jobID int64) (bool, error) {
if token == "" {
return false, nil
}
var (
dbOwner string
dbJob int64
expiresAt time.Time
usedAt sql.NullTime
revokedAt sql.NullTime
)
err := s.db.QueryRowContext(ctx,
`SELECT owner_id, consolidation_job_id, expires_at, used_at, revoked_at
FROM memory_dispatch_tokens
WHERE token = ?`, token,
).Scan(&dbOwner, &dbJob, &expiresAt, &usedAt, &revokedAt)
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
return false, nil
}
return false, fmt.Errorf("dispatch token: query: %w", err)
}
if revokedAt.Valid {
return false, nil
}
if !expiresAt.After(s.now().UTC()) {
return false, nil
}
if dbOwner != ownerID {
return false, nil
}
if dbJob != jobID {
return false, nil
}
// Stamp used_at on first successful validate (idempotent: COALESCE
// keeps the original timestamp on later calls).
if !usedAt.Valid {
if _, err := s.db.ExecContext(ctx,
`UPDATE memory_dispatch_tokens SET used_at = ? WHERE token = ? AND used_at IS NULL`,
s.now().UTC(), token,
); err != nil {
return false, fmt.Errorf("dispatch token: stamp used_at: %w", err)
}
}
return true, nil
}
// Revoke marks the given token as revoked. Idempotent; revoking a
// non-existent token is not an error (so the dream worker can revoke on
// best-effort cleanup without races).
func (s *DispatchTokenStore) Revoke(ctx context.Context, token string) error {
if token == "" {
return nil
}
_, err := s.db.ExecContext(ctx,
`UPDATE memory_dispatch_tokens SET revoked_at = ?
WHERE token = ? AND revoked_at IS NULL`,
s.now().UTC(), token,
)
if err != nil {
return fmt.Errorf("dispatch token: revoke: %w", err)
}
return nil
}
+198
View File
@@ -0,0 +1,198 @@
package messaging
import (
"context"
"database/sql"
"testing"
"time"
_ "modernc.org/sqlite"
)
// insertJob seeds a memory_consolidation_jobs row and returns its id.
// Dispatch tokens FK-reference this row.
func insertJob(t *testing.T, db *sql.DB, ownerID, jobType string) int64 {
t.Helper()
res, err := db.Exec(
`INSERT INTO memory_consolidation_jobs (owner_id, job_type, trigger_reason)
VALUES (?, ?, ?)`,
ownerID, jobType, "manual:test",
)
if err != nil {
t.Fatalf("insert job: %v", err)
}
id, _ := res.LastInsertId()
return id
}
func TestDispatchTokenStore_IssueAndValidate(t *testing.T) {
db := newTestDB(t)
store := NewDispatchTokenStore(db)
ctx := context.Background()
jobID := insertJob(t, db, "1", "reflection")
token, expiresAt, err := store.Issue(ctx, "1", jobID)
if err != nil {
t.Fatalf("Issue: %v", err)
}
if token == "" {
t.Fatal("Issue returned empty token")
}
if !expiresAt.After(time.Now()) {
t.Errorf("expiresAt = %v, want > now()", expiresAt)
}
ok, err := store.Validate(ctx, token, "1", jobID)
if err != nil {
t.Fatalf("Validate: %v", err)
}
if !ok {
t.Fatal("Validate returned false for a freshly-issued token")
}
// Second Validate must also succeed (token is bound to one job;
// multiple tool calls within the job are expected per R7).
ok, err = store.Validate(ctx, token, "1", jobID)
if err != nil {
t.Fatalf("second Validate: %v", err)
}
if !ok {
t.Error("second Validate returned false (token should remain valid within its job)")
}
}
func TestDispatchTokenStore_RejectCases(t *testing.T) {
db := newTestDB(t)
store := NewDispatchTokenStore(db)
ctx := context.Background()
jobID := insertJob(t, db, "1", "reflection")
otherJobID := insertJob(t, db, "1", "core_rewrite")
mintFresh := func(t *testing.T) string {
t.Helper()
tok, _, err := store.Issue(ctx, "1", jobID)
if err != nil {
t.Fatalf("Issue: %v", err)
}
return tok
}
tests := []struct {
name string
setup func(t *testing.T) (token, ownerID string, job int64)
wantValid bool
}{
{
name: "happy path",
setup: func(t *testing.T) (string, string, int64) {
return mintFresh(t), "1", jobID
},
wantValid: true,
},
{
name: "owner mismatch",
setup: func(t *testing.T) (string, string, int64) {
return mintFresh(t), "2", jobID
},
wantValid: false,
},
{
name: "wrong job",
setup: func(t *testing.T) (string, string, int64) {
return mintFresh(t), "1", otherJobID
},
wantValid: false,
},
{
name: "expired",
setup: func(t *testing.T) (string, string, int64) {
tok := mintFresh(t)
// Forcibly expire the token by direct UPDATE.
if _, err := db.Exec(
`UPDATE memory_dispatch_tokens SET expires_at = ? WHERE token = ?`,
time.Now().Add(-1*time.Minute).UTC(), tok,
); err != nil {
t.Fatalf("expire token: %v", err)
}
return tok, "1", jobID
},
wantValid: false,
},
{
name: "revoked",
setup: func(t *testing.T) (string, string, int64) {
tok := mintFresh(t)
if err := store.Revoke(ctx, tok); err != nil {
t.Fatalf("Revoke: %v", err)
}
return tok, "1", jobID
},
wantValid: false,
},
{
name: "unknown token",
setup: func(t *testing.T) (string, string, int64) {
return "definitely-not-a-real-token", "1", jobID
},
wantValid: false,
},
{
name: "empty token",
setup: func(t *testing.T) (string, string, int64) {
return "", "1", jobID
},
wantValid: false,
},
}
for _, tc := range tests {
tc := tc
t.Run(tc.name, func(t *testing.T) {
tok, owner, job := tc.setup(t)
got, err := store.Validate(ctx, tok, owner, job)
if err != nil {
t.Fatalf("Validate: %v", err)
}
if got != tc.wantValid {
t.Errorf("Validate = %v, want %v", got, tc.wantValid)
}
})
}
}
func TestDispatchTokenStore_RevokeIdempotent(t *testing.T) {
db := newTestDB(t)
store := NewDispatchTokenStore(db)
ctx := context.Background()
jobID := insertJob(t, db, "1", "reflection")
tok, _, err := store.Issue(ctx, "1", jobID)
if err != nil {
t.Fatalf("Issue: %v", err)
}
if err := store.Revoke(ctx, tok); err != nil {
t.Fatalf("first Revoke: %v", err)
}
if err := store.Revoke(ctx, tok); err != nil {
t.Fatalf("second Revoke: %v", err)
}
if err := store.Revoke(ctx, "unknown"); err != nil {
t.Fatalf("Revoke unknown: %v", err)
}
}
func TestDispatchTokenStore_IssueRequiresArgs(t *testing.T) {
db := newTestDB(t)
store := NewDispatchTokenStore(db)
ctx := context.Background()
if _, _, err := store.Issue(ctx, "", 1); err == nil {
t.Error("Issue with empty owner should error")
}
if _, _, err := store.Issue(ctx, "1", 0); err == nil {
t.Error("Issue with zero job id should error")
}
}
+43
View File
@@ -0,0 +1,43 @@
package messaging
import (
"time"
"github.com/synapbus/synapbus/internal/metrics"
)
// recordJobMetric increments the synapbus_dream_jobs_total counter for
// (owner, job_type, status). Pulled into a helper so callers don't have
// to import the metrics package directly and so test builds can stub it
// later if needed.
func recordJobMetric(ownerID, jobType, status string) {
metrics.DreamJobsTotal.WithLabelValues(ownerID, jobType, status).Inc()
}
// recordTokensMetric adds to the synapbus_dream_tokens_total counter
// for both in and out directions. Zero deltas are no-ops.
func recordTokensMetric(ownerID string, tokensIn, tokensOut int64) {
if tokensIn > 0 {
metrics.DreamTokensTotal.WithLabelValues(ownerID, "in").Add(float64(tokensIn))
}
if tokensOut > 0 {
metrics.DreamTokensTotal.WithLabelValues(ownerID, "out").Add(float64(tokensOut))
}
}
// recordJobDurationMetric observes one dream-job wallclock duration in
// the synapbus_dream_job_duration_seconds histogram.
func recordJobDurationMetric(ownerID, jobType string, d time.Duration) {
metrics.DreamJobDuration.WithLabelValues(ownerID, jobType).Observe(d.Seconds())
}
// recordCircuitBrokenMetric bumps synapbus_dream_circuit_broken_total
// for (owner, reason). Used when the UsageGate denies a dispatch.
func recordCircuitBrokenMetric(ownerID, jobType, reason string) {
// jobType is intentionally unused as a label here — the gate
// decision is per-owner per-day, not per-job-type. Keeping the
// parameter in the signature so call sites stay symmetric with
// recordJobMetric.
_ = jobType
metrics.DreamCircuitBrokenTotal.WithLabelValues(ownerID, reason).Inc()
}
+198
View File
@@ -0,0 +1,198 @@
// Dream-worker daily usage tracker + circuit breaker (feature 020
// follow-up). Wraps the `memory_dream_usage` table from migration 029.
//
// Each row aggregates one (UTC date, owner_id) bucket. The
// ConsolidatorWorker:
//
// 1. Calls UsageGate.Allow() before Create+Issue+Execute. If today's
// counters exceed any configured threshold the gate returns
// allowed=false with a reason code; the worker then records a
// `circuit_broken` job row and skips dispatch.
// 2. On successful Execute completion, calls RecordCompletion with
// the harness.ExecResult.Usage tokens so tomorrow's gate decisions
// incorporate today's spend.
//
// The circuit "resets" naturally — Today() pivots on UTC date so the
// first call after midnight UTC reads a fresh row with zero counters.
package messaging
import (
"context"
"database/sql"
"fmt"
"time"
)
// DreamDailyUsage is one row of `memory_dream_usage`.
type DreamDailyUsage struct {
Date string `json:"date"`
OwnerID string `json:"owner_id"`
TokensIn int64 `json:"tokens_in"`
TokensOut int64 `json:"tokens_out"`
JobsStarted int `json:"jobs_started"`
JobsSucceeded int `json:"jobs_succeeded"`
JobsFailed int `json:"jobs_failed"`
JobsCircuitBroken int `json:"jobs_circuit_broken"`
UpdatedAt time.Time `json:"updated_at"`
}
// DreamUsageStore wraps the `memory_dream_usage` table.
type DreamUsageStore struct {
db *sql.DB
// now is overridable in tests so callers can pin the UTC date used
// when bucketing.
now func() time.Time
}
// NewDreamUsageStore returns a store rooted at db.
func NewDreamUsageStore(db *sql.DB) *DreamUsageStore {
return &DreamUsageStore{db: db, now: time.Now}
}
// utcDate returns the YYYY-MM-DD key for the store's clock.
func (s *DreamUsageStore) utcDate() string {
return s.now().UTC().Format("2006-01-02")
}
// upsert is the common path for all increments — UPSERT (date, owner_id).
// Counters are added with COALESCE so deltas accumulate cleanly.
func (s *DreamUsageStore) upsert(ctx context.Context, ownerID string,
dIn, dOut int64, dStart, dSuc, dFail, dCB int,
) error {
if s == nil || s.db == nil {
return nil
}
if ownerID == "" {
return fmt.Errorf("dream usage: empty owner_id")
}
date := s.utcDate()
_, err := s.db.ExecContext(ctx, `
INSERT INTO memory_dream_usage
(date, owner_id, tokens_in, tokens_out,
jobs_started, jobs_succeeded, jobs_failed, jobs_circuit_broken, updated_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, CURRENT_TIMESTAMP)
ON CONFLICT(date, owner_id) DO UPDATE SET
tokens_in = tokens_in + excluded.tokens_in,
tokens_out = tokens_out + excluded.tokens_out,
jobs_started = jobs_started + excluded.jobs_started,
jobs_succeeded = jobs_succeeded + excluded.jobs_succeeded,
jobs_failed = jobs_failed + excluded.jobs_failed,
jobs_circuit_broken = jobs_circuit_broken + excluded.jobs_circuit_broken,
updated_at = CURRENT_TIMESTAMP
`, date, ownerID, dIn, dOut, dStart, dSuc, dFail, dCB)
if err != nil {
return fmt.Errorf("dream usage: upsert: %w", err)
}
return nil
}
// RecordStart increments today's jobs_started for ownerID.
func (s *DreamUsageStore) RecordStart(ctx context.Context, ownerID string) error {
return s.upsert(ctx, ownerID, 0, 0, 1, 0, 0, 0)
}
// RecordCompletion increments tokens and the appropriate per-status
// counter. status ∈ {succeeded, failed, circuit_broken}. Unknown
// statuses are counted as failed so the gate errs on the safe side.
func (s *DreamUsageStore) RecordCompletion(ctx context.Context, ownerID string, tokensIn, tokensOut int64, status string) error {
var dSuc, dFail, dCB int
switch status {
case JobStatusSucceeded:
dSuc = 1
case JobStatusCircuitBroken:
dCB = 1
case JobStatusPartial:
// Partial counts as succeeded for the circuit-breaker — it ran.
dSuc = 1
default:
dFail = 1
}
return s.upsert(ctx, ownerID, tokensIn, tokensOut, 0, dSuc, dFail, dCB)
}
// Today returns today's counters for ownerID. Missing row → zero-value.
func (s *DreamUsageStore) Today(ctx context.Context, ownerID string) (DreamDailyUsage, error) {
if s == nil || s.db == nil {
return DreamDailyUsage{}, nil
}
if ownerID == "" {
return DreamDailyUsage{}, fmt.Errorf("dream usage: empty owner_id")
}
date := s.utcDate()
u := DreamDailyUsage{Date: date, OwnerID: ownerID}
err := s.db.QueryRowContext(ctx, `
SELECT tokens_in, tokens_out, jobs_started, jobs_succeeded,
jobs_failed, jobs_circuit_broken, updated_at
FROM memory_dream_usage
WHERE date = ? AND owner_id = ?
`, date, ownerID).Scan(
&u.TokensIn, &u.TokensOut, &u.JobsStarted, &u.JobsSucceeded,
&u.JobsFailed, &u.JobsCircuitBroken, &u.UpdatedAt,
)
if err == sql.ErrNoRows {
return u, nil
}
if err != nil {
return u, fmt.Errorf("dream usage: today: %w", err)
}
return u, nil
}
// Cleanup deletes usage rows older than olderThanDays. Default callers
// can pass 30 to keep a month of history for diagnostics.
func (s *DreamUsageStore) Cleanup(ctx context.Context, olderThanDays int) (int64, error) {
if s == nil || s.db == nil {
return 0, nil
}
if olderThanDays <= 0 {
return 0, nil
}
cutoff := s.now().UTC().AddDate(0, 0, -olderThanDays).Format("2006-01-02")
res, err := s.db.ExecContext(ctx,
`DELETE FROM memory_dream_usage WHERE date < ?`, cutoff,
)
if err != nil {
return 0, fmt.Errorf("dream usage: cleanup: %w", err)
}
n, _ := res.RowsAffected()
return n, nil
}
// UsageGate is the circuit breaker the ConsolidatorWorker consults
// before dispatching a job. It reads today's counters via DreamUsageStore
// and compares them against the per-day limits in MemoryConfig.
type UsageGate struct {
cfg MemoryConfig
store *DreamUsageStore
}
// NewUsageGate ties a MemoryConfig to a DreamUsageStore.
func NewUsageGate(cfg MemoryConfig, store *DreamUsageStore) *UsageGate {
return &UsageGate{cfg: cfg, store: store}
}
// Allow returns (allowed, reasonCode, err). When allowed is false the
// caller should record a `circuit_broken` job and skip dispatch.
// reasonCode is one of {tokens_in_exceeded, tokens_out_exceeded,
// jobs_exceeded, ""}. err is non-nil only for database failures —
// callers may treat err != nil as "fail open" to avoid wedging the
// worker on a transient store glitch.
func (g *UsageGate) Allow(ctx context.Context, ownerID string) (bool, string, error) {
if g == nil || g.store == nil {
return true, "", nil
}
u, err := g.store.Today(ctx, ownerID)
if err != nil {
return true, "", err
}
if g.cfg.DreamDailyTokenLimitIn > 0 && u.TokensIn >= g.cfg.DreamDailyTokenLimitIn {
return false, "tokens_in_exceeded", nil
}
if g.cfg.DreamDailyTokenLimitOut > 0 && u.TokensOut >= g.cfg.DreamDailyTokenLimitOut {
return false, "tokens_out_exceeded", nil
}
if g.cfg.DreamDailyJobLimit > 0 && u.JobsStarted >= g.cfg.DreamDailyJobLimit {
return false, "jobs_exceeded", nil
}
return true, "", nil
}
+146
View File
@@ -0,0 +1,146 @@
package messaging
import (
"context"
"testing"
"time"
)
func TestDreamUsageStore_RecordAndQuery(t *testing.T) {
db := newTestDB(t)
store := NewDreamUsageStore(db)
ctx := context.Background()
owner := "42"
// Start two jobs, complete one succeeded with tokens, one failed.
if err := store.RecordStart(ctx, owner); err != nil {
t.Fatalf("RecordStart: %v", err)
}
if err := store.RecordStart(ctx, owner); err != nil {
t.Fatalf("RecordStart: %v", err)
}
if err := store.RecordCompletion(ctx, owner, 1500, 700, JobStatusSucceeded); err != nil {
t.Fatalf("RecordCompletion succeeded: %v", err)
}
if err := store.RecordCompletion(ctx, owner, 0, 0, JobStatusFailed); err != nil {
t.Fatalf("RecordCompletion failed: %v", err)
}
u, err := store.Today(ctx, owner)
if err != nil {
t.Fatalf("Today: %v", err)
}
if u.JobsStarted != 2 || u.JobsSucceeded != 1 || u.JobsFailed != 1 {
t.Errorf("counters mismatch: %+v", u)
}
if u.TokensIn != 1500 || u.TokensOut != 700 {
t.Errorf("tokens mismatch: in=%d out=%d", u.TokensIn, u.TokensOut)
}
}
func TestDreamUsageStore_CircuitBrokenAccounting(t *testing.T) {
db := newTestDB(t)
store := NewDreamUsageStore(db)
ctx := context.Background()
owner := "99"
if err := store.RecordCompletion(ctx, owner, 0, 0, JobStatusCircuitBroken); err != nil {
t.Fatalf("RecordCompletion: %v", err)
}
u, _ := store.Today(ctx, owner)
if u.JobsCircuitBroken != 1 {
t.Errorf("expected jobs_circuit_broken=1, got %d", u.JobsCircuitBroken)
}
}
func TestUsageGate_AllowDeny(t *testing.T) {
db := newTestDB(t)
store := NewDreamUsageStore(db)
ctx := context.Background()
owner := "7"
cfg := DefaultMemoryConfig()
cfg.DreamDailyTokenLimitIn = 1000
cfg.DreamDailyTokenLimitOut = 500
cfg.DreamDailyJobLimit = 3
gate := NewUsageGate(cfg, store)
// Initially allowed.
allowed, reason, err := gate.Allow(ctx, owner)
if err != nil || !allowed || reason != "" {
t.Fatalf("expected initial allow, got allowed=%v reason=%q err=%v", allowed, reason, err)
}
// Push tokens_in just below the limit — still allowed.
if err := store.RecordCompletion(ctx, owner, 999, 0, JobStatusSucceeded); err != nil {
t.Fatalf("RecordCompletion: %v", err)
}
allowed, reason, _ = gate.Allow(ctx, owner)
if !allowed {
t.Errorf("expected allow at 999/1000 in, got %q", reason)
}
// Cross the input threshold — denied.
if err := store.RecordCompletion(ctx, owner, 2, 0, JobStatusSucceeded); err != nil {
t.Fatalf("RecordCompletion: %v", err)
}
allowed, reason, _ = gate.Allow(ctx, owner)
if allowed || reason != "tokens_in_exceeded" {
t.Errorf("expected tokens_in_exceeded, got allowed=%v reason=%q", allowed, reason)
}
}
func TestUsageGate_JobsExceeded(t *testing.T) {
db := newTestDB(t)
store := NewDreamUsageStore(db)
ctx := context.Background()
owner := "5"
cfg := DefaultMemoryConfig()
cfg.DreamDailyTokenLimitIn = 0 // disable token gates
cfg.DreamDailyTokenLimitOut = 0
cfg.DreamDailyJobLimit = 2
gate := NewUsageGate(cfg, store)
for i := 0; i < 2; i++ {
_ = store.RecordStart(ctx, owner)
}
allowed, reason, _ := gate.Allow(ctx, owner)
if allowed || reason != "jobs_exceeded" {
t.Errorf("expected jobs_exceeded, got allowed=%v reason=%q", allowed, reason)
}
}
func TestUsageGate_NilStoreAllows(t *testing.T) {
gate := NewUsageGate(DefaultMemoryConfig(), nil)
allowed, reason, err := gate.Allow(context.Background(), "1")
if !allowed || reason != "" || err != nil {
t.Errorf("nil store: expected allow, got %v %q %v", allowed, reason, err)
}
}
func TestDreamUsageStore_Cleanup(t *testing.T) {
db := newTestDB(t)
store := NewDreamUsageStore(db)
ctx := context.Background()
// Insert an "old" row by overriding the clock.
store.now = func() time.Time { return time.Now().AddDate(0, 0, -40).UTC() }
if err := store.RecordStart(ctx, "1"); err != nil {
t.Fatalf("RecordStart: %v", err)
}
// Restore now.
store.now = time.Now
if err := store.RecordStart(ctx, "1"); err != nil {
t.Fatalf("RecordStart: %v", err)
}
n, err := store.Cleanup(ctx, 30)
if err != nil {
t.Fatalf("Cleanup: %v", err)
}
if n != 1 {
t.Errorf("expected 1 row deleted, got %d", n)
}
}
+75
View File
@@ -0,0 +1,75 @@
package messaging
import (
"context"
"fmt"
)
// EventMeta is the metadata-only view of a message used by event feeds
// (e.g. the agent SSE stream). It deliberately carries no message body.
type EventMeta struct {
MessageID int64
FromAgent string
ToAgent string // set for DMs
Channel string // set for channel messages
Subject string
}
// ListEventMetaAfter returns metadata for messages with id > afterID that are
// visible to agentName, ordered by id ascending, at most limit rows.
//
// Visible means: DMs addressed to the agent, plus channel messages in channels
// the agent is a member of (same scope as SearchMessages). Messages sent by the
// agent itself are excluded.
//
// Membership is evaluated at query time (current membership), so messages
// posted in a channel before the agent joined it are also returned when their
// id > afterID. Channel members can read that history anyway.
//
// Note: channel messages are stored with to_agent NULL (see InsertMessage), so
// the channel branch matches both NULL and ''.
func (s *SQLiteMessageStore) ListEventMetaAfter(ctx context.Context, agentName string, afterID int64, limit int) ([]*EventMeta, error) {
if limit <= 0 {
limit = 50
}
rows, err := s.db.QueryContext(ctx,
`SELECT m.id, m.from_agent, COALESCE(m.to_agent, ''), COALESCE(ch.name, ''), COALESCE(cv.subject, '')
FROM messages m
LEFT JOIN channels ch ON ch.id = m.channel_id
LEFT JOIN conversations cv ON cv.id = m.conversation_id
WHERE m.id > ? AND m.from_agent <> ?
AND (m.to_agent = ? OR (m.channel_id IS NOT NULL AND (m.to_agent IS NULL OR m.to_agent = '') AND EXISTS (SELECT 1 FROM channel_members cm WHERE cm.channel_id = m.channel_id AND cm.agent_name = ?)))
ORDER BY m.id ASC
LIMIT ?`,
afterID, agentName, agentName, agentName, limit,
)
if err != nil {
return nil, fmt.Errorf("query event meta: %w", err)
}
defer rows.Close()
var out []*EventMeta
for rows.Next() {
var e EventMeta
if err := rows.Scan(&e.MessageID, &e.FromAgent, &e.ToAgent, &e.Channel, &e.Subject); err != nil {
return nil, fmt.Errorf("scan event meta: %w", err)
}
out = append(out, &e)
}
return out, rows.Err()
}
// ListEventMetaAfter returns body-free metadata for messages visible to
// agentName with id > afterID, ascending by id, at most limit rows.
func (s *MessagingService) ListEventMetaAfter(ctx context.Context, agentName string, afterID int64, limit int) ([]*EventMeta, error) {
return s.store.ListEventMetaAfter(ctx, agentName, afterID, limit)
}
// GetConversationSubject returns the subject of a conversation, or "" if unknown.
func (s *MessagingService) GetConversationSubject(ctx context.Context, conversationID int64) string {
conv, err := s.store.GetConversation(ctx, conversationID)
if err != nil || conv == nil {
return ""
}
return conv.Subject
}
+97
View File
@@ -0,0 +1,97 @@
package messaging
import (
"context"
"reflect"
"testing"
)
func TestSQLiteMessageStore_ListEventMetaAfter(t *testing.T) {
db := newTestDB(t)
store := NewSQLiteMessageStore(db)
ctx := context.Background()
for _, a := range []string{"a", "b", "c"} {
seedAgent(t, db, a)
}
for _, q := range []string{
`INSERT INTO channels (id, name, description, topic, type, is_private, is_system, created_by, created_at, updated_at)
VALUES (1, 'room', '', '', 'standard', 0, 0, 'a', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)`,
`INSERT INTO channels (id, name, description, topic, type, is_private, is_system, created_by, created_at, updated_at)
VALUES (2, 'hidden', '', '', 'standard', 0, 0, 'c', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)`,
`INSERT INTO channel_members (channel_id, agent_name, role, joined_at) VALUES (1, 'a', 'member', CURRENT_TIMESTAMP)`,
`INSERT INTO channel_members (channel_id, agent_name, role, joined_at) VALUES (1, 'b', 'member', CURRENT_TIMESTAMP)`,
`INSERT INTO channel_members (channel_id, agent_name, role, joined_at) VALUES (2, 'c', 'member', CURRENT_TIMESTAMP)`,
} {
if _, err := db.ExecContext(ctx, q); err != nil {
t.Fatalf("seed: %v", err)
}
}
room, hidden := int64(1), int64(2)
// Channel messages are stored with to_agent NULL (InsertMessage).
seed := []struct {
from, to string
ch *int64
}{
{"b", "a", nil}, // 1: DM to a
{"b", "c", nil}, // 2: DM to c
{"b", "", &room}, // 3: room, from b
{"c", "", &hidden}, // 4: hidden, from c
{"a", "b", nil}, // 5: DM from a to b
{"a", "", &room}, // 6: room, from a
{"b", "a", nil}, // 7: DM to a
}
for _, s := range seed {
conv := &Conversation{Subject: "subj-" + s.from, CreatedBy: s.from}
if err := store.InsertConversation(ctx, conv); err != nil {
t.Fatalf("InsertConversation: %v", err)
}
m := &Message{ConversationID: conv.ID, FromAgent: s.from, ToAgent: s.to, ChannelID: s.ch,
Body: "secret", Priority: 5, Status: StatusPending}
if err := store.InsertMessage(ctx, m); err != nil {
t.Fatalf("InsertMessage: %v", err)
}
}
tests := []struct {
name string
agent string
afterID int64
limit int
want []int64
}{
{"a sees own DMs and joined channel, not others or own", "a", 0, 100, []int64{1, 3, 7}},
{"c sees DM and own channel only", "c", 0, 100, []int64{2}}, // 4 is c's own message
{"b sees DM and room message from a", "b", 0, 100, []int64{5, 6}},
{"unknown agent sees nothing", "zed", 0, 100, nil},
{"afterID is exclusive", "a", 1, 100, []int64{3, 7}},
{"afterID past end", "a", 7, 100, nil},
{"limit applies, ascending", "a", 0, 2, []int64{1, 3}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := store.ListEventMetaAfter(ctx, tt.agent, tt.afterID, tt.limit)
if err != nil {
t.Fatalf("ListEventMetaAfter: %v", err)
}
var ids []int64
for _, e := range got {
ids = append(ids, e.MessageID)
}
if !reflect.DeepEqual(ids, tt.want) {
t.Fatalf("ids = %v, want %v", ids, tt.want)
}
})
}
t.Run("metadata fields", func(t *testing.T) {
got, _ := store.ListEventMetaAfter(ctx, "a", 0, 100)
if got[0].FromAgent != "b" || got[0].ToAgent != "a" || got[0].Channel != "" || got[0].Subject != "subj-b" {
t.Errorf("DM meta = %+v", got[0])
}
if got[1].Channel != "room" || got[1].ToAgent != "" {
t.Errorf("channel meta = %+v", got[1])
}
})
}
+125
View File
@@ -0,0 +1,125 @@
package messaging
import (
"context"
"database/sql"
"encoding/json"
"fmt"
"strings"
)
// MemoryChannelNames lists exact channel names that are always treated
// as memory channels regardless of metadata.
var MemoryChannelNames = []string{"open-brain"}
// MemoryChannelPrefixes lists channel-name prefixes that are always
// treated as memory channels. Matching uses HasPrefix (e.g.
// "reflections-personal-brand" matches "reflections-").
var MemoryChannelPrefixes = []string{"reflections-"}
// MemoryChannel is the lightweight shape this package needs to decide
// memory-pool participation. We use a local type rather than
// `channels.Channel` to avoid an import cycle: package `channels`
// already imports `messaging`. Callers that hold a `*channels.Channel`
// can construct a `MemoryChannel{Name: ch.Name}` and pass it in.
type MemoryChannel struct {
ID int64
Name string
Metadata string // raw JSON; empty when unknown
}
// IsMemoryChannel reports whether the given channel participates in
// the proactive-memory pool.
//
// A channel is a memory channel when ANY of:
//
// 1. Its name appears in MemoryChannelNames (currently `open-brain`).
// 2. Its name starts with a MemoryChannelPrefixes entry (currently
// `reflections-*`).
// 3. Its raw metadata JSON contains `"is_memory": true`.
//
// The metadata path is documented in 020-data-model.md but the
// `channels` table does not yet expose a metadata column; the third
// rule is wired up so it just-works once that lands and is harmless
// in the meantime (empty Metadata always returns false).
func IsMemoryChannel(ch *MemoryChannel) bool {
if ch == nil {
return false
}
if matchesMemoryChannelName(ch.Name) {
return true
}
return isMemoryChannelMetadata(ch.Metadata)
}
func matchesMemoryChannelName(name string) bool {
for _, n := range MemoryChannelNames {
if name == n {
return true
}
}
for _, p := range MemoryChannelPrefixes {
if strings.HasPrefix(name, p) {
return true
}
}
return false
}
func isMemoryChannelMetadata(metadata string) bool {
m := strings.TrimSpace(metadata)
if m == "" || m == "{}" {
return false
}
var parsed map[string]any
if err := json.Unmarshal([]byte(m), &parsed); err != nil {
return false
}
v, ok := parsed["is_memory"]
if !ok {
return false
}
b, ok := v.(bool)
return ok && b
}
// MemoryChannelIDs returns the IDs of every channel that participates
// in the memory pool. Driven by name patterns today; will pick up the
// metadata flag automatically once `channels.metadata` ships (this
// function would then SELECT metadata as well and apply
// IsMemoryChannel per row).
func MemoryChannelIDs(ctx context.Context, db *sql.DB) ([]int64, error) {
var clauses []string
var args []any
for _, n := range MemoryChannelNames {
clauses = append(clauses, "name = ?")
args = append(args, n)
}
for _, p := range MemoryChannelPrefixes {
clauses = append(clauses, "name LIKE ?")
args = append(args, p+"%")
}
if len(clauses) == 0 {
return nil, nil
}
query := "SELECT id FROM channels WHERE " + strings.Join(clauses, " OR ")
rows, err := db.QueryContext(ctx, query, args...)
if err != nil {
return nil, fmt.Errorf("query memory channels: %w", err)
}
defer rows.Close()
var ids []int64
for rows.Next() {
var id int64
if err := rows.Scan(&id); err != nil {
return nil, fmt.Errorf("scan memory channel id: %w", err)
}
ids = append(ids, id)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("iterate memory channel ids: %w", err)
}
return ids, nil
}
+219
View File
@@ -0,0 +1,219 @@
package messaging
import (
"os"
"strconv"
"time"
)
// MemoryConfig holds feature-flag and tunable knobs for proactive memory
// injection and the dream worker. All fields are populated from
// environment variables by ParseMemoryConfig; defaults match the values
// listed in the 020-proactive-memory-dream-worker spec.
type MemoryConfig struct {
// InjectionEnabled toggles the `relevant_context` packet on MCP tool
// responses. When false, the wrapper is a no-op and the field is
// omitted from responses entirely.
InjectionEnabled bool
// InjectionBudgetTokens is the soft cap for the merged packet
// (memories + core_memory), counted as `(chars+3)/4`.
InjectionBudgetTokens int
// InjectionMaxItems caps the number of memory items in a single packet.
InjectionMaxItems int
// InjectionMinScore is the relevance floor; items below this are
// dropped unless explicitly pinned.
InjectionMinScore float64
// CoreMemoryMaxBytes is the upper bound on a per-(owner, agent) core
// memory blob. Enforced in the Set path.
CoreMemoryMaxBytes int
// DreamEnabled gates the entire ConsolidatorWorker lifecycle and the
// six memory_* MCP tools.
DreamEnabled bool
// DreamInterval is the ticker period for the consolidator worker.
DreamInterval time.Duration
// DreamDeepCron is the cron expression for the nightly deep pass
// (e.g. core_rewrite). Standard 5-field cron; not parsed here.
DreamDeepCron string
// DreamMaxConcurrent caps the number of owners with in-flight jobs.
DreamMaxConcurrent int
// DreamParallel is the number of concurrent dream jobs the worker
// (and `synapbus memory dream-run`) will fan out per (owner, job_type)
// when triggered. 1 = historical single-job behaviour. Higher values
// let one watermark/manual trigger spawn N parallel reflection /
// link_gen / dedup_contradiction agents to drain a backlog. core_rewrite
// is always single-slot regardless of this knob.
DreamParallel int
// DreamWallclockBudget caps how long a single consolidation job is
// allowed to run before the worker terminates it with status=partial.
DreamWallclockBudget time.Duration
// DreamWatermark is the number of new memory messages required
// before a reflection job triggers automatically for an owner.
DreamWatermark int
// DreamAgent is the name of the agent invoked as the dream worker
// via harness.Harness.Execute.
DreamAgent string
// DreamRecentWindow bounds the lookback for the dream worker's
// per-owner input set (unprocessed-count queries, recency
// injection fallback, and memory_list_unprocessed). Messages older
// than `now - DreamRecentWindow` are invisible to the worker — the
// pool is effectively a rolling window of recent activity. Default
// 14 days. Env: SYNAPBUS_DREAM_RECENT_WINDOW. Accepts Go
// duration syntax plus the "Nd" days extension.
DreamRecentWindow time.Duration
// DreamDailyTokenLimitIn is the per-(owner, day) input-token
// circuit-breaker threshold. When the running sum of TokensIn
// across today's dream jobs exceeds this, the worker skips further
// dispatches for that owner until the date rolls over. Default 1M.
// Env: SYNAPBUS_DREAM_DAILY_TOKEN_LIMIT_IN.
DreamDailyTokenLimitIn int64
// DreamDailyTokenLimitOut is the per-(owner, day) output-token
// circuit-breaker threshold. Default 200k.
// Env: SYNAPBUS_DREAM_DAILY_TOKEN_LIMIT_OUT.
DreamDailyTokenLimitOut int64
// DreamDailyJobLimit is the per-(owner, day) ceiling on jobs
// started. Default 100. Env: SYNAPBUS_DREAM_DAILY_JOB_LIMIT.
DreamDailyJobLimit int
}
// DefaultMemoryConfig returns the defaults exactly as listed in the
// 020-proactive-memory-dream-worker spec.
func DefaultMemoryConfig() MemoryConfig {
return MemoryConfig{
InjectionEnabled: false,
InjectionBudgetTokens: 500,
InjectionMaxItems: 5,
InjectionMinScore: 0.25,
CoreMemoryMaxBytes: 2048,
DreamEnabled: false,
DreamInterval: 1 * time.Hour,
DreamDeepCron: "0 3 * * *",
DreamMaxConcurrent: 4,
DreamParallel: 1,
DreamWallclockBudget: 10 * time.Minute,
DreamWatermark: 20,
DreamAgent: "claude-code",
// 14 days of recent activity.
DreamRecentWindow: 336 * time.Hour,
DreamDailyTokenLimitIn: 1_000_000,
DreamDailyTokenLimitOut: 200_000,
DreamDailyJobLimit: 100,
}
}
// ParseMemoryConfig reads MemoryConfig fields from environment
// variables, falling back to the spec defaults for any unset or
// unparseable value.
func ParseMemoryConfig() MemoryConfig {
cfg := DefaultMemoryConfig()
if v := os.Getenv("SYNAPBUS_INJECTION_ENABLED"); v != "" {
if b, ok := parseBoolFlag(v); ok {
cfg.InjectionEnabled = b
}
}
if v := os.Getenv("SYNAPBUS_INJECTION_BUDGET_TOKENS"); v != "" {
if n, err := strconv.Atoi(v); err == nil && n > 0 {
cfg.InjectionBudgetTokens = n
}
}
if v := os.Getenv("SYNAPBUS_INJECTION_MAX_ITEMS"); v != "" {
if n, err := strconv.Atoi(v); err == nil && n > 0 {
cfg.InjectionMaxItems = n
}
}
if v := os.Getenv("SYNAPBUS_INJECTION_MIN_SCORE"); v != "" {
if f, err := strconv.ParseFloat(v, 64); err == nil && f >= 0 {
cfg.InjectionMinScore = f
}
}
if v := os.Getenv("SYNAPBUS_CORE_MEMORY_MAX_BYTES"); v != "" {
if n, err := strconv.Atoi(v); err == nil && n > 0 {
cfg.CoreMemoryMaxBytes = n
}
}
if v := os.Getenv("SYNAPBUS_DREAM_ENABLED"); v != "" {
if b, ok := parseBoolFlag(v); ok {
cfg.DreamEnabled = b
}
}
if v := os.Getenv("SYNAPBUS_DREAM_INTERVAL"); v != "" {
if d, err := parseDurationWithDays(v); err == nil && d > 0 {
cfg.DreamInterval = d
}
}
if v := os.Getenv("SYNAPBUS_DREAM_DEEP_CRON"); v != "" {
cfg.DreamDeepCron = v
}
if v := os.Getenv("SYNAPBUS_DREAM_MAX_CONCURRENT"); v != "" {
if n, err := strconv.Atoi(v); err == nil && n > 0 {
cfg.DreamMaxConcurrent = n
}
}
if v := os.Getenv("SYNAPBUS_DREAM_PARALLEL"); v != "" {
if n, err := strconv.Atoi(v); err == nil && n > 0 {
cfg.DreamParallel = n
}
}
if v := os.Getenv("SYNAPBUS_DREAM_WALLCLOCK_BUDGET"); v != "" {
if d, err := parseDurationWithDays(v); err == nil && d > 0 {
cfg.DreamWallclockBudget = d
}
}
if v := os.Getenv("SYNAPBUS_DREAM_WATERMARK"); v != "" {
if n, err := strconv.Atoi(v); err == nil && n > 0 {
cfg.DreamWatermark = n
}
}
if v := os.Getenv("SYNAPBUS_DREAM_AGENT"); v != "" {
cfg.DreamAgent = v
}
if v := os.Getenv("SYNAPBUS_DREAM_RECENT_WINDOW"); v != "" {
if d, err := parseDurationWithDays(v); err == nil && d > 0 {
cfg.DreamRecentWindow = d
}
}
if v := os.Getenv("SYNAPBUS_DREAM_DAILY_TOKEN_LIMIT_IN"); v != "" {
if n, err := strconv.ParseInt(v, 10, 64); err == nil && n > 0 {
cfg.DreamDailyTokenLimitIn = n
}
}
if v := os.Getenv("SYNAPBUS_DREAM_DAILY_TOKEN_LIMIT_OUT"); v != "" {
if n, err := strconv.ParseInt(v, 10, 64); err == nil && n > 0 {
cfg.DreamDailyTokenLimitOut = n
}
}
if v := os.Getenv("SYNAPBUS_DREAM_DAILY_JOB_LIMIT"); v != "" {
if n, err := strconv.Atoi(v); err == nil && n > 0 {
cfg.DreamDailyJobLimit = n
}
}
return cfg
}
// parseBoolFlag accepts the common 0/1/true/false/yes/no spellings.
func parseBoolFlag(v string) (bool, bool) {
switch v {
case "1", "true", "TRUE", "True", "yes", "YES", "y", "on", "ON":
return true, true
case "0", "false", "FALSE", "False", "no", "NO", "n", "off", "OFF":
return false, true
}
return false, false
}
+195
View File
@@ -0,0 +1,195 @@
// Per-(owner, agent) core memory store. Backs User Story 2 of feature
// 020-proactive-memory-dream-worker — small, owner-scoped, replace-wholesale
// blobs surfaced in `relevant_context.core_memory` on session-start tools
// (e.g. `my_status`).
//
// Schema lives in `internal/storage/schema/028_memory_consolidation.sql`
// (table `memory_core`). owner_id is stored as TEXT (string form of
// `users.id`) to match the proactive-memory tables and the request-context
// owner_id propagated by auth middleware.
package messaging
import (
"context"
"database/sql"
"errors"
"fmt"
"time"
)
// ErrCoreMemoryTooLarge is returned by CoreMemoryStore.Set when the blob
// exceeds the configured max bytes (default SYNAPBUS_CORE_MEMORY_MAX_BYTES = 2048).
// The MCP `memory_rewrite_core` tool surfaces this as the contractual
// `core_memory_too_large` error code (see contracts/mcp-memory-tools.md).
var ErrCoreMemoryTooLarge = errors.New("core memory blob exceeds max bytes")
// CoreMemoryRecord is one row of the `memory_core` table.
type CoreMemoryRecord struct {
OwnerID string
AgentName string
Blob string
UpdatedAt time.Time
UpdatedBy string
}
// CoreMemoryStore wraps the `memory_core` table. All operations are
// owner-scoped — owner_id is part of the primary key — so callers cannot
// cross-read another owner's blobs.
type CoreMemoryStore struct {
db *sql.DB
maxBytes int
}
// NewCoreMemoryStore returns a store rooted at db enforcing the given
// max-bytes cap on Set. When maxBytes <= 0, defaults to 2048 (the spec
// default for SYNAPBUS_CORE_MEMORY_MAX_BYTES).
func NewCoreMemoryStore(db *sql.DB, maxBytes int) *CoreMemoryStore {
if maxBytes <= 0 {
maxBytes = 2048
}
return &CoreMemoryStore{db: db, maxBytes: maxBytes}
}
// MaxBytes returns the configured upper bound for Set blobs.
func (s *CoreMemoryStore) MaxBytes() int { return s.maxBytes }
// Get returns the core memory blob for (ownerID, agentName). When no row
// exists, returns ok=false with no error. Unexpected DB errors surface as
// err.
func (s *CoreMemoryStore) Get(ctx context.Context, ownerID, agentName string) (blob string, updatedAt time.Time, ok bool, err error) {
row := s.db.QueryRowContext(ctx,
`SELECT blob, updated_at FROM memory_core WHERE owner_id = ? AND agent_name = ?`,
ownerID, agentName,
)
if err := row.Scan(&blob, &updatedAt); err != nil {
if errors.Is(err, sql.ErrNoRows) {
return "", time.Time{}, false, nil
}
return "", time.Time{}, false, fmt.Errorf("core memory get: %w", err)
}
return blob, updatedAt, true, nil
}
// Set wholesale-replaces the blob for (ownerID, agentName). Enforces the
// configured size cap and returns ErrCoreMemoryTooLarge when violated.
// `updatedBy` is recorded for audit (typically the caller agent name, or
// "human" for admin/web edits).
func (s *CoreMemoryStore) Set(ctx context.Context, ownerID, agentName, blob, updatedBy string) error {
if len(blob) > s.maxBytes {
return ErrCoreMemoryTooLarge
}
if ownerID == "" {
return fmt.Errorf("core memory set: empty owner_id")
}
if agentName == "" {
return fmt.Errorf("core memory set: empty agent_name")
}
if updatedBy == "" {
return fmt.Errorf("core memory set: empty updated_by")
}
_, err := s.db.ExecContext(ctx,
`INSERT INTO memory_core (owner_id, agent_name, blob, updated_at, updated_by)
VALUES (?, ?, ?, CURRENT_TIMESTAMP, ?)
ON CONFLICT(owner_id, agent_name) DO UPDATE SET
blob = excluded.blob,
updated_at = CURRENT_TIMESTAMP,
updated_by = excluded.updated_by`,
ownerID, agentName, blob, updatedBy,
)
if err != nil {
return fmt.Errorf("core memory set: %w", err)
}
return nil
}
// Delete removes the (ownerID, agentName) row. Returns nil even if no
// row matched — callers should treat "deleted" and "did not exist" the
// same way (the REST endpoint distinguishes via a separate Get).
func (s *CoreMemoryStore) Delete(ctx context.Context, ownerID, agentName string) error {
_, err := s.db.ExecContext(ctx,
`DELETE FROM memory_core WHERE owner_id = ? AND agent_name = ?`,
ownerID, agentName,
)
if err != nil {
return fmt.Errorf("core memory delete: %w", err)
}
return nil
}
// List returns all core memory rows for the given owner. Used by the
// future audit UI (deferred US4) and by admin tooling.
func (s *CoreMemoryStore) List(ctx context.Context, ownerID string) ([]CoreMemoryRecord, error) {
rows, err := s.db.QueryContext(ctx,
`SELECT owner_id, agent_name, blob, updated_at, updated_by
FROM memory_core
WHERE owner_id = ?
ORDER BY agent_name`,
ownerID,
)
if err != nil {
return nil, fmt.Errorf("core memory list: %w", err)
}
defer rows.Close()
var out []CoreMemoryRecord
for rows.Next() {
var r CoreMemoryRecord
if err := rows.Scan(&r.OwnerID, &r.AgentName, &r.Blob, &r.UpdatedAt, &r.UpdatedBy); err != nil {
return nil, fmt.Errorf("core memory list scan: %w", err)
}
out = append(out, r)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("core memory list rows: %w", err)
}
return out, nil
}
// GetForInjection is the adapter implementing
// `search.CoreMemoryProvider.Get`. Returns "" (no error) when no row
// exists — the empty-string convention lets the injection wrapper treat a
// missing core memory the same as "field omitted", per
// `contracts/mcp-injection.md`.
//
// The matching interface contract is in
// `internal/search/injection.go`'s `CoreMemoryProvider`:
//
// Get(ctx context.Context, ownerID, agentName string) (string, error)
//
// We expose this as a method on the store (not a separate type) so
// callers can pass `coreStore.GetForInjection` as a method value — but the
// store itself also satisfies the interface via its `Get` method below.
func (s *CoreMemoryStore) GetForInjection(ctx context.Context, ownerID, agentName string) (string, error) {
blob, _, ok, err := s.Get(ctx, ownerID, agentName)
if err != nil {
return "", err
}
if !ok {
return "", nil
}
return blob, nil
}
// coreProviderAdapter wraps a *CoreMemoryStore so it satisfies
// `search.CoreMemoryProvider` (which requires a `Get(ctx, ownerID,
// agentName) (string, error)` signature — distinct from the store's
// 4-return Get). Use NewCoreProvider to construct.
type coreProviderAdapter struct {
store *CoreMemoryStore
}
// NewCoreProvider returns an object satisfying
// `search.CoreMemoryProvider` so callers can wire the store into
// WrapConfig.CoreProvider without leaking the store's richer Get
// signature.
func NewCoreProvider(store *CoreMemoryStore) *coreProviderAdapter {
return &coreProviderAdapter{store: store}
}
// Get implements `search.CoreMemoryProvider.Get`. Returns "" when no row
// exists for the (owner, agent) pair.
func (a *coreProviderAdapter) Get(ctx context.Context, ownerID, agentName string) (string, error) {
if a == nil || a.store == nil {
return "", nil
}
return a.store.GetForInjection(ctx, ownerID, agentName)
}
+179
View File
@@ -0,0 +1,179 @@
package messaging
import (
"context"
"errors"
"strings"
"testing"
)
func TestCoreMemoryStore_GetSetRoundTrip(t *testing.T) {
db := newTestDB(t)
store := NewCoreMemoryStore(db, 2048)
ctx := context.Background()
const owner = "1"
const agent = "research-mcpproxy"
const blob = "You are research-mcpproxy. Focus on benchmarking."
if err := store.Set(ctx, owner, agent, blob, "human"); err != nil {
t.Fatalf("Set: %v", err)
}
got, updatedAt, ok, err := store.Get(ctx, owner, agent)
if err != nil {
t.Fatalf("Get: %v", err)
}
if !ok {
t.Fatal("Get: expected ok=true, got false")
}
if got != blob {
t.Errorf("Get blob mismatch: got %q want %q", got, blob)
}
if updatedAt.IsZero() {
t.Error("Get: expected non-zero updated_at")
}
}
func TestCoreMemoryStore_SetReplacesWholesale(t *testing.T) {
db := newTestDB(t)
store := NewCoreMemoryStore(db, 2048)
ctx := context.Background()
const owner = "1"
const agent = "alpha"
if err := store.Set(ctx, owner, agent, "first version", "human"); err != nil {
t.Fatalf("first Set: %v", err)
}
if err := store.Set(ctx, owner, agent, "second version", "dream:1"); err != nil {
t.Fatalf("second Set: %v", err)
}
got, _, ok, err := store.Get(ctx, owner, agent)
if err != nil {
t.Fatalf("Get: %v", err)
}
if !ok || got != "second version" {
t.Errorf("expected wholesale replace, got ok=%v blob=%q", ok, got)
}
if strings.Contains(got, "first") {
t.Errorf("expected merge to NOT happen; got %q", got)
}
}
func TestCoreMemoryStore_SetRejectsOverSize(t *testing.T) {
db := newTestDB(t)
store := NewCoreMemoryStore(db, 16)
ctx := context.Background()
tooBig := strings.Repeat("x", 17)
err := store.Set(ctx, "1", "alpha", tooBig, "human")
if err == nil {
t.Fatal("Set: expected ErrCoreMemoryTooLarge, got nil")
}
if !errors.Is(err, ErrCoreMemoryTooLarge) {
t.Errorf("Set: expected ErrCoreMemoryTooLarge, got %v", err)
}
// Exactly at-cap is fine.
atCap := strings.Repeat("x", 16)
if err := store.Set(ctx, "1", "alpha", atCap, "human"); err != nil {
t.Errorf("Set at cap: unexpected error %v", err)
}
}
func TestCoreMemoryStore_Delete(t *testing.T) {
db := newTestDB(t)
store := NewCoreMemoryStore(db, 2048)
ctx := context.Background()
if err := store.Set(ctx, "1", "alpha", "blob", "human"); err != nil {
t.Fatalf("Set: %v", err)
}
if err := store.Delete(ctx, "1", "alpha"); err != nil {
t.Fatalf("Delete: %v", err)
}
_, _, ok, err := store.Get(ctx, "1", "alpha")
if err != nil {
t.Fatalf("Get after Delete: %v", err)
}
if ok {
t.Error("Get after Delete: expected ok=false")
}
// Delete on missing row is idempotent.
if err := store.Delete(ctx, "1", "alpha"); err != nil {
t.Errorf("Delete on missing: unexpected error %v", err)
}
}
func TestCoreMemoryStore_OwnerScoping(t *testing.T) {
db := newTestDB(t)
store := NewCoreMemoryStore(db, 2048)
ctx := context.Background()
if err := store.Set(ctx, "1", "shared-name", "H1 blob", "h1"); err != nil {
t.Fatalf("Set H1: %v", err)
}
if err := store.Set(ctx, "2", "shared-name", "H2 blob", "h2"); err != nil {
t.Fatalf("Set H2: %v", err)
}
got1, _, ok1, err := store.Get(ctx, "1", "shared-name")
if err != nil || !ok1 || got1 != "H1 blob" {
t.Errorf("H1 Get: ok=%v got=%q err=%v", ok1, got1, err)
}
got2, _, ok2, err := store.Get(ctx, "2", "shared-name")
if err != nil || !ok2 || got2 != "H2 blob" {
t.Errorf("H2 Get: ok=%v got=%q err=%v", ok2, got2, err)
}
// H1 cannot see H2's blob and vice versa — distinct PKs.
if got1 == got2 {
t.Error("owner scoping broken: H1 and H2 see the same blob")
}
// List is scoped to a single owner.
listH1, err := store.List(ctx, "1")
if err != nil {
t.Fatalf("List H1: %v", err)
}
if len(listH1) != 1 || listH1[0].OwnerID != "1" {
t.Errorf("List H1: expected 1 row owned by 1, got %#v", listH1)
}
}
func TestCoreMemoryStore_GetForInjection(t *testing.T) {
db := newTestDB(t)
store := NewCoreMemoryStore(db, 2048)
ctx := context.Background()
// Missing row → "" + no error (NOT sql.ErrNoRows).
blob, err := store.GetForInjection(ctx, "1", "missing-agent")
if err != nil {
t.Fatalf("GetForInjection on missing: unexpected error %v", err)
}
if blob != "" {
t.Errorf("GetForInjection on missing: expected \"\" got %q", blob)
}
// Present row → blob.
if err := store.Set(ctx, "1", "alpha", "core blob", "human"); err != nil {
t.Fatalf("Set: %v", err)
}
blob, err = store.GetForInjection(ctx, "1", "alpha")
if err != nil {
t.Fatalf("GetForInjection: %v", err)
}
if blob != "core blob" {
t.Errorf("GetForInjection: got %q want %q", blob, "core blob")
}
// Adapter satisfies search.CoreMemoryProvider implicitly.
provider := NewCoreProvider(store)
got, err := provider.Get(ctx, "1", "alpha")
if err != nil || got != "core blob" {
t.Errorf("CoreProvider.Get: got %q err=%v", got, err)
}
}
+147
View File
@@ -0,0 +1,147 @@
package messaging
import (
"context"
"database/sql"
"encoding/json"
"fmt"
"time"
)
// InjectionRecord is one row in the `memory_injections` 24-hour audit
// ring. Each row captures what was attached to a single MCP tool
// response so the owner can later answer "why did my agent know this?"
// via the recent-injections debug surface (FR-025).
type InjectionRecord struct {
ID int64 `json:"id"`
OwnerID string `json:"owner_id"`
AgentName string `json:"agent_name"`
ToolName string `json:"tool_name"`
PacketSizeChars int `json:"packet_size_chars"`
PacketItemsCount int `json:"packet_items_count"`
MessageIDs []int64 `json:"message_ids"`
CoreBlobIncluded bool `json:"core_blob_included"`
CreatedAt time.Time `json:"created_at"`
}
// MemoryInjections is the audit-ring store for proactive injection
// (data-model.md §`memory_injections`). Each row is best-effort:
// failures are non-fatal for the injection request — callers should
// log and continue.
type MemoryInjections struct {
db *sql.DB
}
// NewMemoryInjections wraps a *sql.DB.
func NewMemoryInjections(db *sql.DB) *MemoryInjections {
return &MemoryInjections{db: db}
}
// Record inserts one injection row. `row.MessageIDs` is JSON-encoded.
// `created_at` defaults to CURRENT_TIMESTAMP when zero.
func (s *MemoryInjections) Record(ctx context.Context, row InjectionRecord) error {
if s == nil || s.db == nil {
return nil
}
ids := row.MessageIDs
if ids == nil {
ids = []int64{}
}
b, err := json.Marshal(ids)
if err != nil {
return fmt.Errorf("memory_injections: marshal message_ids: %w", err)
}
if row.CreatedAt.IsZero() {
_, err = s.db.ExecContext(ctx,
`INSERT INTO memory_injections
(owner_id, agent_name, tool_name, packet_size_chars,
packet_items_count, message_ids, core_blob_included)
VALUES (?, ?, ?, ?, ?, ?, ?)`,
row.OwnerID, row.AgentName, row.ToolName,
row.PacketSizeChars, row.PacketItemsCount, string(b),
row.CoreBlobIncluded,
)
} else {
_, err = s.db.ExecContext(ctx,
`INSERT INTO memory_injections
(owner_id, agent_name, tool_name, packet_size_chars,
packet_items_count, message_ids, core_blob_included, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)`,
row.OwnerID, row.AgentName, row.ToolName,
row.PacketSizeChars, row.PacketItemsCount, string(b),
row.CoreBlobIncluded, row.CreatedAt.UTC(),
)
}
if err != nil {
return fmt.Errorf("memory_injections: insert: %w", err)
}
return nil
}
// Cleanup deletes rows older than `olderThan` ago. Returns the number
// of rows removed. Safe to call from a periodic ticker.
func (s *MemoryInjections) Cleanup(ctx context.Context, olderThan time.Duration) (int64, error) {
if s == nil || s.db == nil {
return 0, nil
}
if olderThan <= 0 {
return 0, nil
}
cutoff := time.Now().Add(-olderThan).UTC()
res, err := s.db.ExecContext(ctx,
`DELETE FROM memory_injections WHERE created_at < ?`, cutoff,
)
if err != nil {
return 0, fmt.Errorf("memory_injections: cleanup: %w", err)
}
affected, _ := res.RowsAffected()
return affected, nil
}
// ListRecent returns the most recent injections for one owner, newest
// first, up to `limit`.
func (s *MemoryInjections) ListRecent(ctx context.Context, ownerID string, limit int) ([]InjectionRecord, error) {
if s == nil || s.db == nil {
return nil, nil
}
if limit <= 0 {
limit = 50
}
rows, err := s.db.QueryContext(ctx,
`SELECT id, owner_id, agent_name, tool_name, packet_size_chars,
packet_items_count, message_ids, core_blob_included, created_at
FROM memory_injections
WHERE owner_id = ?
ORDER BY created_at DESC, id DESC
LIMIT ?`, ownerID, limit,
)
if err != nil {
return nil, fmt.Errorf("memory_injections: list recent: %w", err)
}
defer rows.Close()
var out []InjectionRecord
for rows.Next() {
var rec InjectionRecord
var idsJSON string
if err := rows.Scan(
&rec.ID, &rec.OwnerID, &rec.AgentName, &rec.ToolName,
&rec.PacketSizeChars, &rec.PacketItemsCount, &idsJSON,
&rec.CoreBlobIncluded, &rec.CreatedAt,
); err != nil {
return nil, fmt.Errorf("memory_injections: scan: %w", err)
}
if idsJSON == "" {
rec.MessageIDs = []int64{}
} else if err := json.Unmarshal([]byte(idsJSON), &rec.MessageIDs); err != nil {
// Corrupt row — surface as empty rather than fail the whole listing.
rec.MessageIDs = []int64{}
}
out = append(out, rec)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("memory_injections: iterate: %w", err)
}
return out, nil
}
@@ -0,0 +1,166 @@
package messaging
import (
"context"
"testing"
"time"
_ "modernc.org/sqlite"
)
func TestMemoryInjections_RecordAndList(t *testing.T) {
db := newTestDB(t)
store := NewMemoryInjections(db)
ctx := context.Background()
rec := InjectionRecord{
OwnerID: "1",
AgentName: "research-mcpproxy",
ToolName: "my_status",
PacketSizeChars: 412,
PacketItemsCount: 3,
MessageIDs: []int64{1, 2, 3},
CoreBlobIncluded: true,
}
if err := store.Record(ctx, rec); err != nil {
t.Fatalf("Record: %v", err)
}
got, err := store.ListRecent(ctx, "1", 10)
if err != nil {
t.Fatalf("ListRecent: %v", err)
}
if len(got) != 1 {
t.Fatalf("ListRecent returned %d rows, want 1", len(got))
}
if got[0].PacketSizeChars != 412 || got[0].PacketItemsCount != 3 {
t.Errorf("packet counters mismatch: %+v", got[0])
}
if len(got[0].MessageIDs) != 3 || got[0].MessageIDs[0] != 1 {
t.Errorf("MessageIDs round-trip failed: %v", got[0].MessageIDs)
}
if !got[0].CoreBlobIncluded {
t.Error("CoreBlobIncluded round-trip failed")
}
}
func TestMemoryInjections_CleanupOnlyOldRows(t *testing.T) {
db := newTestDB(t)
store := NewMemoryInjections(db)
ctx := context.Background()
now := time.Now().UTC()
tests := []struct {
name string
offset time.Duration
wantKept bool
}{
{"3 days old", -72 * time.Hour, false},
{"36 hours old", -36 * time.Hour, false},
{"23 hours old", -23 * time.Hour, true},
{"30 minutes old", -30 * time.Minute, true},
{"current", 0, true},
}
// Seed 50 rows: 10 per bucket so we exercise the DELETE plan.
for _, tc := range tests {
for i := 0; i < 10; i++ {
rec := InjectionRecord{
OwnerID: "1",
AgentName: "a",
ToolName: "my_status",
PacketSizeChars: 100,
PacketItemsCount: 1,
MessageIDs: []int64{int64(i)},
CreatedAt: now.Add(tc.offset),
}
if err := store.Record(ctx, rec); err != nil {
t.Fatalf("Record %s: %v", tc.name, err)
}
}
}
deleted, err := store.Cleanup(ctx, 24*time.Hour)
if err != nil {
t.Fatalf("Cleanup: %v", err)
}
// 2 buckets older than 24h × 10 rows = 20 expected deletions.
if deleted != 20 {
t.Errorf("Cleanup removed %d rows, want 20", deleted)
}
remaining, err := store.ListRecent(ctx, "1", 100)
if err != nil {
t.Fatalf("ListRecent: %v", err)
}
if len(remaining) != 30 {
t.Errorf("after cleanup got %d rows, want 30", len(remaining))
}
}
func TestMemoryInjections_ListRecent_OwnerScoped(t *testing.T) {
db := newTestDB(t)
store := NewMemoryInjections(db)
ctx := context.Background()
for _, owner := range []string{"1", "2", "3"} {
for i := 0; i < 5; i++ {
if err := store.Record(ctx, InjectionRecord{
OwnerID: owner,
AgentName: "a",
ToolName: "my_status",
PacketSizeChars: 100,
PacketItemsCount: 1,
MessageIDs: []int64{int64(i)},
}); err != nil {
t.Fatalf("Record owner=%s i=%d: %v", owner, i, err)
}
}
}
got, err := store.ListRecent(ctx, "2", 100)
if err != nil {
t.Fatalf("ListRecent: %v", err)
}
if len(got) != 5 {
t.Fatalf("owner=2 got %d rows, want 5", len(got))
}
for _, r := range got {
if r.OwnerID != "2" {
t.Errorf("found leaked row OwnerID=%q in owner=2 listing", r.OwnerID)
}
}
}
func TestMemoryInjections_ListRecent_LimitOrdering(t *testing.T) {
db := newTestDB(t)
store := NewMemoryInjections(db)
ctx := context.Background()
base := time.Now().UTC().Add(-1 * time.Hour)
for i := 0; i < 10; i++ {
if err := store.Record(ctx, InjectionRecord{
OwnerID: "1",
AgentName: "a",
ToolName: "my_status",
PacketSizeChars: 100,
PacketItemsCount: 1,
MessageIDs: []int64{int64(i)},
CreatedAt: base.Add(time.Duration(i) * time.Minute),
}); err != nil {
t.Fatalf("Record %d: %v", i, err)
}
}
got, err := store.ListRecent(ctx, "1", 3)
if err != nil {
t.Fatalf("ListRecent: %v", err)
}
if len(got) != 3 {
t.Fatalf("ListRecent limit=3 returned %d", len(got))
}
// Newest first: i=9, 8, 7. Validate MessageIDs[0] descends.
if got[0].MessageIDs[0] != 9 || got[1].MessageIDs[0] != 8 || got[2].MessageIDs[0] != 7 {
t.Errorf("ordering wrong: %d, %d, %d", got[0].MessageIDs[0], got[1].MessageIDs[0], got[2].MessageIDs[0])
}
}
+264
View File
@@ -0,0 +1,264 @@
// Memory-link store for feature 020 — typed directed edges between two
// memory message ids (see data-model.md §`memory_links`).
//
// Three classes of relation types live in this table:
//
// - Semantic types written by the dream-agent via the
// `memory_add_link` MCP tool: `refines`, `contradicts`, `examples`,
// `related`.
//
// - Consolidation types written by `memory_mark_duplicate` and
// `memory_supersede`: `duplicate_of`, `superseded_by`. These are NOT
// valid arguments to `memory_add_link` — the contract reserves them
// for the dedicated tools so the `memory_status` view can derive
// soft-delete / supersede state from a single audit path.
//
// - Auto types written by the messaging layer (post-insert hook,
// T035): `mention`, `reply_to`, `channel_cooccurrence`. These are
// NOT valid arguments from any agent — only the `auto:<rule>`
// created_by prefix may use them.
//
// Add() rejects type/actor mismatches with ErrLinkTypeReserved so the
// MCP tools surface the contractual `relation_type_reserved` error
// code cleanly.
package messaging
import (
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"strings"
"time"
)
// ErrLinkTypeReserved is returned when an actor tries to add a
// relation_type they are not allowed to write directly (see file
// comment for the actor/type matrix).
var ErrLinkTypeReserved = errors.New("relation type reserved for another actor")
// Link is one row in `memory_links`.
type Link struct {
ID int64 `json:"id"`
SrcMessageID int64 `json:"src_message_id"`
DstMessageID int64 `json:"dst_message_id"`
RelationType string `json:"relation_type"`
OwnerID string `json:"owner_id"`
CreatedBy string `json:"created_by"`
Metadata map[string]any `json:"metadata,omitempty"`
CreatedAt time.Time `json:"created_at"`
}
// LinkStore wraps the `memory_links` table.
type LinkStore struct {
db *sql.DB
}
// NewLinkStore returns a store rooted at db.
func NewLinkStore(db *sql.DB) *LinkStore {
return &LinkStore{db: db}
}
// Auto-generated link types (written only via auto:<rule> caller).
var autoLinkTypes = map[string]struct{}{
"mention": {},
"reply_to": {},
"channel_cooccurrence": {},
}
// Reserved-for-tools relation types (written by memory_mark_duplicate /
// memory_supersede via their own code paths, never by memory_add_link).
var consolidationLinkTypes = map[string]struct{}{
"duplicate_of": {},
"superseded_by": {},
}
// IsAutoLinkType reports whether relType is one of the auto-generated
// link types written by the messaging post-insert hook.
func IsAutoLinkType(relType string) bool {
_, ok := autoLinkTypes[relType]
return ok
}
// Add inserts one row into `memory_links`. Reserved-type guarding:
//
// - When createdBy starts with `agent:`, the auto-types
// (mention/reply_to/channel_cooccurrence) AND consolidation-types
// (duplicate_of/superseded_by) are rejected with
// ErrLinkTypeReserved. The MCP `memory_add_link` tool must surface
// this as the contractual `relation_type_reserved` error.
//
// - When createdBy starts with `auto:`, only the auto-types are
// allowed; semantic types and consolidation-types are rejected.
//
// - When createdBy starts with `human:` or any other prefix, no
// reserved-type check is applied — admin tooling can backfill any
// type for debugging / migration.
func (s *LinkStore) Add(
ctx context.Context,
src, dst int64,
relType, ownerID, createdBy string,
metadata map[string]any,
) (int64, error) {
if s == nil || s.db == nil {
return 0, fmt.Errorf("link store: nil store")
}
if src == 0 || dst == 0 {
return 0, fmt.Errorf("link store: src/dst message ids required")
}
if relType == "" || ownerID == "" || createdBy == "" {
return 0, fmt.Errorf("link store: relation_type, owner_id, created_by required")
}
switch {
case strings.HasPrefix(createdBy, "agent:"):
if _, banned := autoLinkTypes[relType]; banned {
return 0, ErrLinkTypeReserved
}
if _, banned := consolidationLinkTypes[relType]; banned {
return 0, ErrLinkTypeReserved
}
case strings.HasPrefix(createdBy, "auto:"):
if _, ok := autoLinkTypes[relType]; !ok {
return 0, ErrLinkTypeReserved
}
}
metaJSON := "{}"
if metadata != nil {
b, err := json.Marshal(metadata)
if err != nil {
return 0, fmt.Errorf("link store: marshal metadata: %w", err)
}
metaJSON = string(b)
}
res, err := s.db.ExecContext(ctx,
`INSERT INTO memory_links
(src_message_id, dst_message_id, relation_type, owner_id, created_by, metadata)
VALUES (?, ?, ?, ?, ?, ?)`,
src, dst, relType, ownerID, createdBy, metaJSON,
)
if err != nil {
return 0, fmt.Errorf("link store: insert: %w", err)
}
id, _ := res.LastInsertId()
return id, nil
}
// AddConsolidationLink inserts a `duplicate_of` or `superseded_by`
// link without running the reserved-type guard. Only the dedicated
// MCP tools `memory_mark_duplicate` and `memory_supersede` should
// call this — the `memory_add_link` path uses Add() and rejects these
// types per the contract.
func (s *LinkStore) AddConsolidationLink(
ctx context.Context,
src, dst int64,
relType, ownerID, createdBy string,
metadata map[string]any,
) (int64, error) {
if relType != "duplicate_of" && relType != "superseded_by" {
return 0, fmt.Errorf("link store: AddConsolidationLink rejects %q", relType)
}
metaJSON := "{}"
if metadata != nil {
b, err := json.Marshal(metadata)
if err != nil {
return 0, fmt.Errorf("link store: marshal metadata: %w", err)
}
metaJSON = string(b)
}
res, err := s.db.ExecContext(ctx,
`INSERT INTO memory_links
(src_message_id, dst_message_id, relation_type, owner_id, created_by, metadata)
VALUES (?, ?, ?, ?, ?, ?)`,
src, dst, relType, ownerID, createdBy, metaJSON,
)
if err != nil {
return 0, fmt.Errorf("link store: insert: %w", err)
}
id, _ := res.LastInsertId()
return id, nil
}
// ListByMessage returns every link with the given message id as src OR
// dst. Useful for both outgoing edges (reflection sources) and incoming
// edges (what refines this).
func (s *LinkStore) ListByMessage(ctx context.Context, msgID int64) ([]Link, error) {
if s == nil || s.db == nil {
return nil, nil
}
rows, err := s.db.QueryContext(ctx,
`SELECT id, src_message_id, dst_message_id, relation_type, owner_id,
created_by, metadata, created_at
FROM memory_links
WHERE src_message_id = ? OR dst_message_id = ?
ORDER BY id ASC`, msgID, msgID,
)
if err != nil {
return nil, fmt.Errorf("link store: list by message: %w", err)
}
defer rows.Close()
return scanLinks(rows)
}
// ListByOwner returns links for the given owner, optionally filtered to
// a subset of relation types. `limit <= 0` defaults to 100.
func (s *LinkStore) ListByOwner(
ctx context.Context,
ownerID string,
types []string,
limit int,
) ([]Link, error) {
if s == nil || s.db == nil {
return nil, nil
}
if limit <= 0 {
limit = 100
}
q := `SELECT id, src_message_id, dst_message_id, relation_type, owner_id,
created_by, metadata, created_at
FROM memory_links
WHERE owner_id = ?`
args := []any{ownerID}
if len(types) > 0 {
placeholders := strings.Repeat("?,", len(types))
placeholders = placeholders[:len(placeholders)-1]
q += " AND relation_type IN (" + placeholders + ")"
for _, t := range types {
args = append(args, t)
}
}
q += " ORDER BY id DESC LIMIT ?"
args = append(args, limit)
rows, err := s.db.QueryContext(ctx, q, args...)
if err != nil {
return nil, fmt.Errorf("link store: list by owner: %w", err)
}
defer rows.Close()
return scanLinks(rows)
}
func scanLinks(rows *sql.Rows) ([]Link, error) {
var out []Link
for rows.Next() {
var l Link
var metaJSON string
if err := rows.Scan(
&l.ID, &l.SrcMessageID, &l.DstMessageID, &l.RelationType,
&l.OwnerID, &l.CreatedBy, &metaJSON, &l.CreatedAt,
); err != nil {
return nil, fmt.Errorf("link store: scan: %w", err)
}
if metaJSON != "" && metaJSON != "{}" {
_ = json.Unmarshal([]byte(metaJSON), &l.Metadata)
}
out = append(out, l)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("link store: iterate: %w", err)
}
return out, nil
}
+152
View File
@@ -0,0 +1,152 @@
package messaging
import (
"context"
"errors"
"testing"
)
func TestLinkStore_AddValidTypesPerActor(t *testing.T) {
db := newTestDB(t)
s := NewLinkStore(db)
ctx := context.Background()
// Agent may write semantic types.
semantic := []string{"refines", "contradicts", "examples", "related"}
src := int64(1)
for i, rt := range semantic {
dst := int64(100 + i)
id, err := s.Add(ctx, src, dst, rt, "1", "agent:dream-algis:tok", nil)
if err != nil {
t.Fatalf("agent semantic %s: %v", rt, err)
}
if id == 0 {
t.Fatalf("agent semantic %s: zero id", rt)
}
}
// auto:<rule> may write auto types.
autos := []string{"mention", "reply_to", "channel_cooccurrence"}
for i, rt := range autos {
dst := int64(200 + i)
id, err := s.Add(ctx, src, dst, rt, "1", "auto:on-message-created", nil)
if err != nil {
t.Fatalf("auto %s: %v", rt, err)
}
if id == 0 {
t.Fatalf("auto %s: zero id", rt)
}
}
// human:<name> may write any type (no reserved-type guard).
for i, rt := range []string{"refines", "duplicate_of", "mention"} {
dst := int64(300 + i)
if _, err := s.Add(ctx, src, dst, rt, "1", "human:algis", nil); err != nil {
t.Errorf("human %s: unexpected err %v", rt, err)
}
}
}
func TestLinkStore_AddRejectsReservedFromAgent(t *testing.T) {
db := newTestDB(t)
s := NewLinkStore(db)
ctx := context.Background()
reserved := []string{"mention", "reply_to", "channel_cooccurrence", "duplicate_of", "superseded_by"}
for i, rt := range reserved {
dst := int64(400 + i)
_, err := s.Add(ctx, 1, dst, rt, "1", "agent:dream:tok", nil)
if err == nil {
t.Errorf("agent %s: expected ErrLinkTypeReserved, got nil", rt)
continue
}
if !errors.Is(err, ErrLinkTypeReserved) {
t.Errorf("agent %s: expected ErrLinkTypeReserved, got %v", rt, err)
}
}
}
func TestLinkStore_AddRejectsNonAutoFromAutoActor(t *testing.T) {
db := newTestDB(t)
s := NewLinkStore(db)
ctx := context.Background()
for i, rt := range []string{"refines", "duplicate_of", "superseded_by", "related"} {
dst := int64(500 + i)
_, err := s.Add(ctx, 1, dst, rt, "1", "auto:something", nil)
if !errors.Is(err, ErrLinkTypeReserved) {
t.Errorf("auto actor %s: expected ErrLinkTypeReserved, got %v", rt, err)
}
}
}
func TestLinkStore_ListByMessageOwnerScoping(t *testing.T) {
db := newTestDB(t)
s := NewLinkStore(db)
ctx := context.Background()
if _, err := s.Add(ctx, 1, 2, "refines", "1", "agent:dream:tok", nil); err != nil {
t.Fatalf("Add owner=1: %v", err)
}
if _, err := s.Add(ctx, 1, 3, "refines", "2", "agent:dream:tok", nil); err != nil {
t.Fatalf("Add owner=2: %v", err)
}
// ListByMessage returns all links touching msg 1 regardless of owner —
// owner-scoping happens in ListByOwner.
ls, err := s.ListByMessage(ctx, 1)
if err != nil {
t.Fatalf("ListByMessage: %v", err)
}
if len(ls) != 2 {
t.Errorf("ListByMessage: want 2, got %d", len(ls))
}
owner1, err := s.ListByOwner(ctx, "1", nil, 0)
if err != nil {
t.Fatalf("ListByOwner 1: %v", err)
}
if len(owner1) != 1 || owner1[0].DstMessageID != 2 {
t.Errorf("ListByOwner 1: unexpected %v", owner1)
}
owner2, err := s.ListByOwner(ctx, "2", []string{"refines"}, 0)
if err != nil {
t.Fatalf("ListByOwner 2: %v", err)
}
if len(owner2) != 1 || owner2[0].DstMessageID != 3 {
t.Errorf("ListByOwner 2: unexpected %v", owner2)
}
// Filtered ListByOwner with a non-matching type returns empty.
none, err := s.ListByOwner(ctx, "1", []string{"contradicts"}, 0)
if err != nil {
t.Fatalf("ListByOwner filter: %v", err)
}
if len(none) != 0 {
t.Errorf("ListByOwner filter: want 0, got %d", len(none))
}
}
func TestLinkStore_AddMetadata(t *testing.T) {
db := newTestDB(t)
s := NewLinkStore(db)
ctx := context.Background()
meta := map[string]any{"confidence": 0.84}
id, err := s.Add(ctx, 10, 20, "refines", "1", "agent:dream:tok", meta)
if err != nil {
t.Fatalf("Add: %v", err)
}
links, err := s.ListByMessage(ctx, 10)
if err != nil {
t.Fatalf("ListByMessage: %v", err)
}
if len(links) != 1 || links[0].ID != id {
t.Fatalf("unexpected listing: %v", links)
}
if got, _ := links[0].Metadata["confidence"].(float64); got != 0.84 {
t.Errorf("metadata round-trip: got %v", links[0].Metadata)
}
}
+155
View File
@@ -0,0 +1,155 @@
// Memory-pin store for feature 020 — owner-pinned message ids that
// bypass the relevance floor on injection retrieval (data-model.md
// §`memory_pins`). Pins are always set by the human owner; the
// dream-agent does not write here. Pinned memories are loaded by
// `search.BuildContextPacket` and overlaid on top of hybrid retrieval
// so they appear even when their similarity score is below
// SYNAPBUS_INJECTION_MIN_SCORE.
package messaging
import (
"context"
"database/sql"
"fmt"
"time"
)
// Pin is one row in `memory_pins`.
type Pin struct {
OwnerID string `json:"owner_id"`
MessageID int64 `json:"message_id"`
PinnedBy string `json:"pinned_by"`
Note string `json:"note,omitempty"`
PinnedAt time.Time `json:"pinned_at"`
}
// PinStore wraps the `memory_pins` table.
type PinStore struct {
db *sql.DB
}
// NewPinStore returns a store rooted at db.
func NewPinStore(db *sql.DB) *PinStore {
return &PinStore{db: db}
}
// Pin pins (owner, msgID) for retrieval overlay. Idempotent — re-pinning
// updates `pinned_by` / `note` in place (the primary key is the
// (owner_id, message_id) tuple).
func (s *PinStore) Pin(ctx context.Context, ownerID string, msgID int64, pinnedBy, note string) error {
if s == nil || s.db == nil {
return fmt.Errorf("pin store: nil store")
}
if ownerID == "" {
return fmt.Errorf("pin store: empty owner_id")
}
if msgID == 0 {
return fmt.Errorf("pin store: message_id required")
}
if pinnedBy == "" {
pinnedBy = "human:" + ownerID
}
_, err := s.db.ExecContext(ctx,
`INSERT INTO memory_pins (owner_id, message_id, pinned_by, note)
VALUES (?, ?, ?, ?)
ON CONFLICT(owner_id, message_id) DO UPDATE SET
pinned_by = excluded.pinned_by,
note = excluded.note,
pinned_at = CURRENT_TIMESTAMP`,
ownerID, msgID, pinnedBy, note,
)
if err != nil {
return fmt.Errorf("pin store: insert: %w", err)
}
return nil
}
// Unpin removes the (owner, msgID) row. Returns nil even when no row
// matched — callers treat "unpinned" and "did not exist" the same way.
func (s *PinStore) Unpin(ctx context.Context, ownerID string, msgID int64) error {
if s == nil || s.db == nil {
return nil
}
_, err := s.db.ExecContext(ctx,
`DELETE FROM memory_pins WHERE owner_id = ? AND message_id = ?`,
ownerID, msgID,
)
if err != nil {
return fmt.Errorf("pin store: delete: %w", err)
}
return nil
}
// ListForOwner returns the pinned message ids for the given owner. Used
// by the injection overlay to splice these in regardless of search
// score.
func (s *PinStore) ListForOwner(ctx context.Context, ownerID string) ([]int64, error) {
if s == nil || s.db == nil {
return nil, nil
}
rows, err := s.db.QueryContext(ctx,
`SELECT message_id FROM memory_pins WHERE owner_id = ? ORDER BY pinned_at DESC`,
ownerID,
)
if err != nil {
return nil, fmt.Errorf("pin store: list ids: %w", err)
}
defer rows.Close()
var out []int64
for rows.Next() {
var id int64
if err := rows.Scan(&id); err != nil {
return nil, fmt.Errorf("pin store: scan: %w", err)
}
out = append(out, id)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("pin store: iterate: %w", err)
}
return out, nil
}
// pinProviderAdapter wraps *PinStore so it satisfies the
// search.PinProvider interface without leaking the wider PinStore
// surface area. Methods accept exactly the (ctx, ownerID) signature
// search.InjectionOpts requires.
type pinProviderAdapter struct{ store *PinStore }
// NewPinProvider returns a search.PinProvider over the given store.
func NewPinProvider(store *PinStore) *pinProviderAdapter { return &pinProviderAdapter{store: store} }
// ListForOwner implements search.PinProvider.
func (a *pinProviderAdapter) ListForOwner(ctx context.Context, ownerID string) ([]int64, error) {
if a == nil || a.store == nil {
return nil, nil
}
return a.store.ListForOwner(ctx, ownerID)
}
// ListPinsForOwner returns full pin rows. Used by future audit UI.
func (s *PinStore) ListPinsForOwner(ctx context.Context, ownerID string) ([]Pin, error) {
if s == nil || s.db == nil {
return nil, nil
}
rows, err := s.db.QueryContext(ctx,
`SELECT owner_id, message_id, pinned_by, COALESCE(note, ''), pinned_at
FROM memory_pins WHERE owner_id = ? ORDER BY pinned_at DESC`,
ownerID,
)
if err != nil {
return nil, fmt.Errorf("pin store: list: %w", err)
}
defer rows.Close()
var out []Pin
for rows.Next() {
var p Pin
if err := rows.Scan(&p.OwnerID, &p.MessageID, &p.PinnedBy, &p.Note, &p.PinnedAt); err != nil {
return nil, fmt.Errorf("pin store: scan: %w", err)
}
out = append(out, p)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("pin store: iterate: %w", err)
}
return out, nil
}
+81
View File
@@ -0,0 +1,81 @@
package messaging
import (
"context"
"testing"
)
func TestPinStore_PinUnpin(t *testing.T) {
db := newTestDB(t)
s := NewPinStore(db)
ctx := context.Background()
if err := s.Pin(ctx, "1", 42, "human:algis", "always relevant"); err != nil {
t.Fatalf("Pin: %v", err)
}
ids, err := s.ListForOwner(ctx, "1")
if err != nil {
t.Fatalf("ListForOwner: %v", err)
}
if len(ids) != 1 || ids[0] != 42 {
t.Errorf("ListForOwner: got %v want [42]", ids)
}
// Re-pin updates note in place; no duplicate row.
if err := s.Pin(ctx, "1", 42, "human:algis", "updated note"); err != nil {
t.Fatalf("re-Pin: %v", err)
}
pins, err := s.ListPinsForOwner(ctx, "1")
if err != nil {
t.Fatalf("ListPinsForOwner: %v", err)
}
if len(pins) != 1 || pins[0].Note != "updated note" {
t.Errorf("Re-pin: unexpected pins %v", pins)
}
if err := s.Unpin(ctx, "1", 42); err != nil {
t.Fatalf("Unpin: %v", err)
}
ids, _ = s.ListForOwner(ctx, "1")
if len(ids) != 0 {
t.Errorf("after Unpin: want empty, got %v", ids)
}
// Unpin on missing row is a no-op.
if err := s.Unpin(ctx, "1", 42); err != nil {
t.Errorf("Unpin on missing: %v", err)
}
}
func TestPinStore_OwnerScoping(t *testing.T) {
db := newTestDB(t)
s := NewPinStore(db)
ctx := context.Background()
if err := s.Pin(ctx, "1", 10, "human:1", ""); err != nil {
t.Fatalf("Pin 1: %v", err)
}
if err := s.Pin(ctx, "1", 20, "human:1", ""); err != nil {
t.Fatalf("Pin 1: %v", err)
}
if err := s.Pin(ctx, "2", 30, "human:2", ""); err != nil {
t.Fatalf("Pin 2: %v", err)
}
one, err := s.ListForOwner(ctx, "1")
if err != nil {
t.Fatalf("ListForOwner 1: %v", err)
}
if len(one) != 2 {
t.Errorf("owner 1: want 2 pins, got %d", len(one))
}
two, err := s.ListForOwner(ctx, "2")
if err != nil {
t.Fatalf("ListForOwner 2: %v", err)
}
if len(two) != 1 || two[0] != 30 {
t.Errorf("owner 2: got %v want [30]", two)
}
}
+102
View File
@@ -0,0 +1,102 @@
// memory_status query helpers — read the SQL view defined in migration
// 028 and turn it into a map[message_id]MemoryStatus suitable for the
// injection retrieval filter. The view itself derives state from
// `memory_consolidation_jobs.actions` rows (data-model.md §`memory_status`)
// so callers never have to mutate a status column directly.
package messaging
import (
"context"
"database/sql"
"fmt"
"strings"
"time"
)
// Memory status constants — value of `memory_status.status`.
const (
MemoryStatusActive = "active"
MemoryStatusSoftDeleted = "soft_deleted"
MemoryStatusSuperseded = "superseded"
)
// MemoryStatus is one row derived from the `memory_status` view.
// SupersededBy / SoftDeletedAt are nil when the message is active.
type MemoryStatus struct {
Status string `json:"status"`
SupersededBy *int64 `json:"superseded_by,omitempty"`
SoftDeletedAt *time.Time `json:"soft_deleted_at,omitempty"`
Reason string `json:"reason,omitempty"`
}
// MemoryStatuses returns the status of each message id in msgIDs. Ids
// absent from the result map are implicitly `active` (the view only
// contains rows that have at least one consolidation action against
// them).
func MemoryStatuses(ctx context.Context, db *sql.DB, msgIDs []int64) (map[int64]MemoryStatus, error) {
out := map[int64]MemoryStatus{}
if len(msgIDs) == 0 {
return out, nil
}
if db == nil {
return out, fmt.Errorf("memory status: nil db")
}
placeholders := strings.Repeat("?,", len(msgIDs))
placeholders = placeholders[:len(placeholders)-1]
args := make([]any, 0, len(msgIDs))
for _, id := range msgIDs {
args = append(args, id)
}
q := `SELECT message_id, status, superseded_by, soft_deleted_at, COALESCE(reason, '')
FROM memory_status
WHERE message_id IN (` + placeholders + `)`
rows, err := db.QueryContext(ctx, q, args...)
if err != nil {
return nil, fmt.Errorf("memory status: query: %w", err)
}
defer rows.Close()
for rows.Next() {
var (
id int64
status string
supersede sql.NullInt64
deletedAt sql.NullTime
reason string
)
if err := rows.Scan(&id, &status, &supersede, &deletedAt, &reason); err != nil {
return nil, fmt.Errorf("memory status: scan: %w", err)
}
ms := MemoryStatus{Status: status, Reason: reason}
if supersede.Valid {
v := supersede.Int64
ms.SupersededBy = &v
}
if deletedAt.Valid {
t := deletedAt.Time
ms.SoftDeletedAt = &t
}
out[id] = ms
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("memory status: iterate: %w", err)
}
return out, nil
}
// StatusByID returns just the string status for each id; callers that
// only need active/non-active (e.g. the injection retrieval filter) can
// use this to avoid pulling in MemoryStatus's optional fields.
func StatusByID(ctx context.Context, db *sql.DB, msgIDs []int64) (map[int64]string, error) {
full, err := MemoryStatuses(ctx, db, msgIDs)
if err != nil {
return nil, err
}
out := make(map[int64]string, len(full))
for id, st := range full {
out[id] = st.Status
}
return out, nil
}
+60 -697
View File
@@ -14,13 +14,15 @@ import (
)
// StalemateConfig holds stalemate detection settings.
//
// The historical reminder/escalation knobs (ReminderAfter, EscalateAfter)
// were removed when SynapBus moved to internal-only mode (no human
// approval loop). See migration 027_remove_approval_noise.sql.
type StalemateConfig struct {
// ProcessingTimeout is how long a message can stay in "processing" before auto-fail (default 24h).
// ProcessingTimeout is how long a message can stay in "processing"
// before auto-fail (default 24h). Protects the inbox queue from
// agents that crash after claiming a message.
ProcessingTimeout time.Duration
// ReminderAfter is how long a pending DM waits before a system reminder is sent (default 4h).
ReminderAfter time.Duration
// EscalateAfter is how long a pending DM waits before escalation to #approvals (default 48h).
EscalateAfter time.Duration
// Interval is how often the worker checks for stale messages (default 15m).
Interval time.Duration
}
@@ -29,8 +31,6 @@ type StalemateConfig struct {
func DefaultStalemateConfig() StalemateConfig {
return StalemateConfig{
ProcessingTimeout: 24 * time.Hour,
ReminderAfter: 4 * time.Hour,
EscalateAfter: 48 * time.Hour,
Interval: 15 * time.Minute,
}
}
@@ -43,7 +43,6 @@ func parseDurationWithDays(s string) (time.Duration, error) {
return 0, fmt.Errorf("empty duration string")
}
// Try "Nd" format (days)
if strings.HasSuffix(s, "d") {
days, err := strconv.Atoi(strings.TrimSuffix(s, "d"))
if err == nil && days > 0 {
@@ -51,7 +50,6 @@ func parseDurationWithDays(s string) (time.Duration, error) {
}
}
// Try standard Go duration
return time.ParseDuration(s)
}
@@ -64,16 +62,6 @@ func ParseStalemateConfig() StalemateConfig {
cfg.ProcessingTimeout = d
}
}
if v := os.Getenv("SYNAPBUS_STALEMATE_REMINDER_AFTER"); v != "" {
if d, err := parseDurationWithDays(v); err == nil && d > 0 {
cfg.ReminderAfter = d
}
}
if v := os.Getenv("SYNAPBUS_STALEMATE_ESCALATE_AFTER"); v != "" {
if d, err := parseDurationWithDays(v); err == nil && d > 0 {
cfg.EscalateAfter = d
}
}
if v := os.Getenv("SYNAPBUS_STALEMATE_INTERVAL"); v != "" {
if d, err := parseDurationWithDays(v); err == nil && d > 0 {
cfg.Interval = d
@@ -83,32 +71,37 @@ func ParseStalemateConfig() StalemateConfig {
return cfg
}
// ChannelLookup provides channel lookup by name without importing the channels package.
type ChannelLookup interface {
// GetChannelIDByName returns a channel ID by name, or 0 if not found.
GetChannelIDByName(ctx context.Context, name string) (int64, error)
}
// StalemateWorker periodically checks for and handles stale messages.
// StalemateWorker periodically auto-fails messages whose claim has timed out.
type StalemateWorker struct {
db *sql.DB
msgService *MessagingService
channelLookup ChannelLookup
config StalemateConfig
logger *slog.Logger
done chan struct{}
wg sync.WaitGroup
// memoryInjections, when non-nil, drives an hourly cleanup of the
// 24h proactive-injection audit ring (feature 020). Plumbed via
// SetMemoryInjections after construction so adding the feature is
// non-breaking for existing call sites.
memoryInjections *MemoryInjections
tickCount int
}
// SetMemoryInjections registers the audit-ring store the worker will
// cleanup hourly. Pass nil to disable; safe to call before Start.
func (w *StalemateWorker) SetMemoryInjections(store *MemoryInjections) {
w.memoryInjections = store
}
// NewStalemateWorker creates a new stalemate detection worker.
func NewStalemateWorker(db *sql.DB, msgService *MessagingService, channelLookup ChannelLookup, config StalemateConfig) *StalemateWorker {
func NewStalemateWorker(db *sql.DB, msgService *MessagingService, config StalemateConfig) *StalemateWorker {
return &StalemateWorker{
db: db,
msgService: msgService,
channelLookup: channelLookup,
config: config,
logger: slog.Default().With("component", "stalemate-worker"),
done: make(chan struct{}),
db: db,
msgService: msgService,
config: config,
logger: slog.Default().With("component", "stalemate-worker"),
done: make(chan struct{}),
}
}
@@ -120,8 +113,6 @@ func (w *StalemateWorker) Start() {
w.logger.Info("stalemate worker started",
"interval", w.config.Interval.String(),
"processing_timeout", w.config.ProcessingTimeout.String(),
"reminder_after", w.config.ReminderAfter.String(),
"escalate_after", w.config.EscalateAfter.String(),
)
ticker := time.NewTicker(w.config.Interval)
@@ -132,6 +123,7 @@ func (w *StalemateWorker) Start() {
case <-ticker.C:
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
w.checkStaleMessages(ctx)
w.maybeCleanupInjections(ctx)
cancel()
case <-w.done:
w.logger.Info("stalemate worker stopped")
@@ -147,23 +139,40 @@ func (w *StalemateWorker) Stop() {
w.wg.Wait()
}
// checkStaleMessages runs all stalemate checks.
// checkStaleMessages runs the auto-fail check for timed-out claimed messages.
func (w *StalemateWorker) checkStaleMessages(ctx context.Context) {
failed := w.failTimedOutProcessing(ctx)
reminded := w.sendPendingReminders(ctx)
escalated := w.escalatePendingMessages(ctx)
if failed := w.failTimedOutProcessing(ctx); failed > 0 {
w.logger.Info("stalemate check complete", "auto_failed", failed)
}
}
// Phase 2: Workflow stalemate checks for channel messages
wfReminded, wfEscalated := w.checkWorkflowStalemates(ctx)
if failed > 0 || reminded > 0 || escalated > 0 || wfReminded > 0 || wfEscalated > 0 {
w.logger.Info("stalemate check complete",
"auto_failed", failed,
"reminders_sent", reminded,
"escalations_sent", escalated,
"workflow_reminders", wfReminded,
"workflow_escalations", wfEscalated,
)
// maybeCleanupInjections piggybacks an hourly cleanup of the 24h
// proactive-injection audit ring (feature 020) on the stalemate worker
// tick. With the default 15-minute Interval, the cleanup fires every
// 4th tick (i.e. ~1h). A no-op when SetMemoryInjections has not been
// called or the store is nil.
func (w *StalemateWorker) maybeCleanupInjections(ctx context.Context) {
if w.memoryInjections == nil {
return
}
w.tickCount++
// Fire roughly hourly. With Interval=15m the modulus matches the
// 4-tick cadence called out in spec 020 T017. For non-default
// intervals it still fires hourly-ish on a best-effort basis.
ticksPerHour := int(time.Hour / w.config.Interval)
if ticksPerHour <= 0 {
ticksPerHour = 1
}
if w.tickCount%ticksPerHour != 0 {
return
}
deleted, err := w.memoryInjections.Cleanup(ctx, 24*time.Hour)
if err != nil {
w.logger.Warn("memory_injections cleanup failed", "error", err)
return
}
if deleted > 0 {
w.logger.Info("memory_injections cleanup", "deleted", deleted)
}
}
@@ -242,9 +251,6 @@ func (w *StalemateWorker) failTimedOutProcessing(ctx context.Context) int64 {
}
affected, _ := res.RowsAffected()
if affected == 0 {
// Message was reclaimed, completed, or otherwise moved out of the
// stale window between SELECT and UPDATE. Skip silently — the
// next worker tick will re-evaluate.
w.logger.Debug("stale processing message no longer stale; skipped",
"message_id", dm.ID,
"claimed_by", dm.ClaimedBy,
@@ -261,646 +267,3 @@ func (w *StalemateWorker) failTimedOutProcessing(ctx context.Context) int64 {
}
return count
}
// sendPendingReminders sends system DM reminders for pending messages older than ReminderAfter.
func (w *StalemateWorker) sendPendingReminders(ctx context.Context) int64 {
cutoff := time.Now().Add(-w.config.ReminderAfter)
rows, err := w.db.QueryContext(ctx,
`SELECT id, from_agent, to_agent, body, created_at
FROM messages
WHERE status = 'pending'
AND to_agent IS NOT NULL
AND to_agent != ''
AND from_agent != 'system'
AND to_agent != 'system'
AND created_at < ?`,
cutoff,
)
if err != nil {
w.logger.Error("query pending reminder candidates failed", "error", err)
return 0
}
defer rows.Close()
type pendingMsg struct {
ID int64
FromAgent string
ToAgent string
Body string
CreatedAt time.Time
}
var pending []pendingMsg
for rows.Next() {
var pm pendingMsg
if err := rows.Scan(&pm.ID, &pm.FromAgent, &pm.ToAgent, &pm.Body, &pm.CreatedAt); err != nil {
w.logger.Error("scan pending message failed", "error", err)
continue
}
pending = append(pending, pm)
}
count := int64(0)
for _, pm := range pending {
// Check if a reminder already exists for this message
if w.reminderExists(ctx, pm.ID, pm.ToAgent) {
continue
}
age := formatAge(time.Since(pm.CreatedAt))
truncBody := truncate(pm.Body, 100)
body := fmt.Sprintf(
"**Reminder**: You have a pending message from %s (%s old). Message: \"%s\". Please claim and process it.",
pm.FromAgent, age, truncBody,
)
_, err := w.msgService.SendMessage(ctx, "system", pm.ToAgent, body, SendOptions{
Subject: fmt.Sprintf("stalemate-reminder:%d", pm.ID),
Priority: 7,
Metadata: fmt.Sprintf(`{"stalemate_reminder_for":%d}`, pm.ID),
})
if err != nil {
w.logger.Error("send stalemate reminder failed",
"message_id", pm.ID,
"to_agent", pm.ToAgent,
"error", err,
)
continue
}
w.logger.Info("sent stalemate reminder",
"message_id", pm.ID,
"to_agent", pm.ToAgent,
"from_agent", pm.FromAgent,
"age", age,
)
count++
}
return count
}
// escalatePendingMessages escalates pending messages older than EscalateAfter to #approvals.
func (w *StalemateWorker) escalatePendingMessages(ctx context.Context) int64 {
cutoff := time.Now().Add(-w.config.EscalateAfter)
rows, err := w.db.QueryContext(ctx,
`SELECT id, from_agent, to_agent, body, created_at
FROM messages
WHERE status = 'pending'
AND to_agent IS NOT NULL
AND to_agent != ''
AND from_agent != 'system'
AND to_agent != 'system'
AND created_at < ?`,
cutoff,
)
if err != nil {
w.logger.Error("query escalation candidates failed", "error", err)
return 0
}
defer rows.Close()
type pendingMsg struct {
ID int64
FromAgent string
ToAgent string
Body string
CreatedAt time.Time
}
var pending []pendingMsg
for rows.Next() {
var pm pendingMsg
if err := rows.Scan(&pm.ID, &pm.FromAgent, &pm.ToAgent, &pm.Body, &pm.CreatedAt); err != nil {
w.logger.Error("scan escalation candidate failed", "error", err)
continue
}
pending = append(pending, pm)
}
if len(pending) == 0 {
return 0
}
// Look up #approvals channel
channelID, err := w.channelLookup.GetChannelIDByName(ctx, "approvals")
if err != nil {
w.logger.Warn("cannot escalate: #approvals channel not found", "error", err)
return 0
}
count := int64(0)
for _, pm := range pending {
// Check if already escalated
if w.escalationExists(ctx, pm.ID) {
continue
}
age := formatAge(time.Since(pm.CreatedAt))
truncBody := truncate(pm.Body, 100)
body := fmt.Sprintf(
"**ESCALATION**: Pending message for @%s from %s has been unprocessed for %s. Message: \"%s\". Manual intervention may be required.",
pm.ToAgent, pm.FromAgent, age, truncBody,
)
_, err := w.msgService.SendMessage(ctx, "system", "", body, SendOptions{
Subject: fmt.Sprintf("stalemate-escalation:%d", pm.ID),
Priority: 9,
Metadata: fmt.Sprintf(`{"stalemate_escalation_for":%d}`, pm.ID),
ChannelID: &channelID,
})
if err != nil {
w.logger.Error("send escalation to #approvals failed",
"message_id", pm.ID,
"error", err,
)
continue
}
w.logger.Info("escalated stale message to #approvals",
"message_id", pm.ID,
"to_agent", pm.ToAgent,
"from_agent", pm.FromAgent,
"age", age,
)
count++
}
return count
}
// reminderExists checks if a system reminder already exists for a given message ID.
func (w *StalemateWorker) reminderExists(ctx context.Context, messageID int64, toAgent string) bool {
var count int
err := w.db.QueryRowContext(ctx,
`SELECT COUNT(*) FROM messages
WHERE from_agent = 'system'
AND to_agent = ?
AND metadata LIKE ?`,
toAgent, fmt.Sprintf(`%%"stalemate_reminder_for":%d%%`, messageID),
).Scan(&count)
if err != nil {
return false
}
return count > 0
}
// escalationExists checks if an escalation already exists for a given message ID.
func (w *StalemateWorker) escalationExists(ctx context.Context, messageID int64) bool {
var count int
err := w.db.QueryRowContext(ctx,
`SELECT COUNT(*) FROM messages
WHERE from_agent = 'system'
AND metadata LIKE ?`,
fmt.Sprintf(`%%"stalemate_escalation_for":%d%%`, messageID),
).Scan(&count)
if err != nil {
return false
}
return count > 0
}
// workflowChannel holds channel info relevant to workflow stalemate checking.
type workflowChannel struct {
ID int64
Name string
StalemateRemindAfter string
StalemateEscalateAfter string
}
// staleWorkflowMsg holds info about a channel message in a stale workflow state.
type staleWorkflowMsg struct {
ID int64
Body string
FromAgent string
ChannelID int64
Channel string
State string
StateAge time.Duration
}
// checkWorkflowStalemates scans workflow-enabled channels for messages stuck in
// non-terminal workflow states (proposed, approved, in_progress) and sends
// reminders to channel members or escalates to #approvals.
func (w *StalemateWorker) checkWorkflowStalemates(ctx context.Context) (reminded int64, escalated int64) {
// Step 1: Find all workflow-enabled channels
channels, err := w.listWorkflowChannels(ctx)
if err != nil {
w.logger.Error("list workflow channels failed", "error", err)
return 0, 0
}
if len(channels) == 0 {
return 0, 0
}
for _, ch := range channels {
remindTimeout, err := parseDurationWithDays(ch.StalemateRemindAfter)
if err != nil || remindTimeout <= 0 {
remindTimeout = 24 * time.Hour // default
}
escalateTimeout, err := parseDurationWithDays(ch.StalemateEscalateAfter)
if err != nil || escalateTimeout <= 0 {
escalateTimeout = 72 * time.Hour // default
}
// Step 2: Find messages in non-terminal workflow states
staleMessages, err := w.findStaleWorkflowMessages(ctx, ch)
if err != nil {
w.logger.Error("find stale workflow messages failed",
"channel", ch.Name,
"error", err,
)
continue
}
for _, msg := range staleMessages {
// Step 3: Check escalation first (longer timeout)
if msg.StateAge >= escalateTimeout {
if w.workflowEscalationExists(ctx, msg.ID) {
continue
}
if w.sendWorkflowEscalation(ctx, msg) {
escalated++
}
continue
}
// Step 4: Check reminder (shorter timeout)
if msg.StateAge >= remindTimeout {
if w.workflowReminderExists(ctx, msg.ID) {
continue
}
r := w.sendWorkflowReminders(ctx, msg, ch.ID)
reminded += r
}
}
}
return reminded, escalated
}
// listWorkflowChannels returns all channels that have workflow_enabled = true.
func (w *StalemateWorker) listWorkflowChannels(ctx context.Context) ([]workflowChannel, error) {
rows, err := w.db.QueryContext(ctx,
`SELECT id, name, stalemate_remind_after, stalemate_escalate_after
FROM channels
WHERE workflow_enabled = 1`)
if err != nil {
return nil, fmt.Errorf("query workflow channels: %w", err)
}
defer rows.Close()
var channels []workflowChannel
for rows.Next() {
var ch workflowChannel
if err := rows.Scan(&ch.ID, &ch.Name, &ch.StalemateRemindAfter, &ch.StalemateEscalateAfter); err != nil {
return nil, fmt.Errorf("scan workflow channel: %w", err)
}
channels = append(channels, ch)
}
return channels, rows.Err()
}
// findStaleWorkflowMessages finds channel messages in non-terminal workflow states
// and computes how long they have been in their current state.
func (w *StalemateWorker) findStaleWorkflowMessages(ctx context.Context, ch workflowChannel) ([]staleWorkflowMsg, error) {
// Get all messages in this channel that could be in a workflow state.
// We fetch messages and their reactions, then compute state in Go.
rows, err := w.db.QueryContext(ctx,
`SELECT m.id, m.body, m.from_agent, m.created_at
FROM messages m
WHERE m.channel_id = ?
AND m.from_agent != 'system'
ORDER BY m.created_at ASC`,
ch.ID,
)
if err != nil {
return nil, fmt.Errorf("query channel messages: %w", err)
}
defer rows.Close()
type chanMsg struct {
ID int64
Body string
FromAgent string
CreatedAt time.Time
}
var msgs []chanMsg
for rows.Next() {
var m chanMsg
if err := rows.Scan(&m.ID, &m.Body, &m.FromAgent, &m.CreatedAt); err != nil {
return nil, fmt.Errorf("scan channel message: %w", err)
}
msgs = append(msgs, m)
}
if err := rows.Err(); err != nil {
return nil, err
}
if len(msgs) == 0 {
return nil, nil
}
// Batch-fetch reactions for all messages
msgIDs := make([]int64, len(msgs))
for i, m := range msgs {
msgIDs[i] = m.ID
}
reactionsMap, err := w.getReactionsByMessageIDs(ctx, msgIDs)
if err != nil {
return nil, fmt.Errorf("get reactions: %w", err)
}
now := time.Now()
var stale []staleWorkflowMsg
for _, m := range msgs {
reactions := reactionsMap[m.ID]
state := computeWorkflowStateFromReactions(reactions)
// Skip terminal states
if isTerminalWorkflowState(state) {
continue
}
// Determine the "state age": how long since the state was entered.
// If reactions exist, use the most recent reaction's created_at.
// If no reactions (proposed state), use the message's created_at.
stateEnteredAt := m.CreatedAt
if len(reactions) > 0 {
// Find the most recent reaction
for _, r := range reactions {
if r.CreatedAt.After(stateEnteredAt) {
stateEnteredAt = r.CreatedAt
}
}
}
stale = append(stale, staleWorkflowMsg{
ID: m.ID,
Body: m.Body,
FromAgent: m.FromAgent,
ChannelID: ch.ID,
Channel: ch.Name,
State: state,
StateAge: now.Sub(stateEnteredAt),
})
}
return stale, nil
}
// reactionRow holds a raw reaction row for workflow state computation.
type reactionRow struct {
Reaction string
CreatedAt time.Time
}
// getReactionsByMessageIDs fetches reactions for a batch of message IDs.
func (w *StalemateWorker) getReactionsByMessageIDs(ctx context.Context, messageIDs []int64) (map[int64][]reactionRow, error) {
if len(messageIDs) == 0 {
return map[int64][]reactionRow{}, nil
}
placeholders := make([]string, len(messageIDs))
args := make([]any, len(messageIDs))
for i, id := range messageIDs {
placeholders[i] = "?"
args[i] = id
}
query := fmt.Sprintf(
`SELECT message_id, reaction, created_at
FROM message_reactions
WHERE message_id IN (%s)
ORDER BY created_at ASC`,
strings.Join(placeholders, ","),
)
rows, err := w.db.QueryContext(ctx, query, args...)
if err != nil {
return nil, fmt.Errorf("query reactions: %w", err)
}
defer rows.Close()
result := make(map[int64][]reactionRow)
for rows.Next() {
var msgID int64
var r reactionRow
if err := rows.Scan(&msgID, &r.Reaction, &r.CreatedAt); err != nil {
return nil, fmt.Errorf("scan reaction: %w", err)
}
result[msgID] = append(result[msgID], r)
}
return result, rows.Err()
}
// computeWorkflowStateFromReactions derives workflow state from raw reaction rows.
// Mirrors the logic in reactions.ComputeWorkflowState without importing that package.
func computeWorkflowStateFromReactions(reactions []reactionRow) string {
if len(reactions) == 0 {
return "proposed"
}
// Reaction priority (same as reactions.reactionPriority)
priority := map[string]int{
"approve": 2,
"in_progress": 3,
"reject": 4,
"done": 5,
"published": 6,
}
// Reaction-to-state mapping (same as reactions.reactionToState)
toState := map[string]string{
"approve": "approved",
"reject": "rejected",
"in_progress": "in_progress",
"done": "done",
"published": "published",
}
highestPriority := 0
highestState := "proposed"
for _, r := range reactions {
if p, ok := priority[r.Reaction]; ok && p > highestPriority {
highestPriority = p
highestState = toState[r.Reaction]
}
}
return highestState
}
// isTerminalWorkflowState returns true if the state should not trigger stalemate checks.
func isTerminalWorkflowState(state string) bool {
switch state {
case "rejected", "done", "published":
return true
default:
return false
}
}
// sendWorkflowReminders sends DMs to channel members about a stale workflow message.
func (w *StalemateWorker) sendWorkflowReminders(ctx context.Context, msg staleWorkflowMsg, channelID int64) int64 {
// Get channel members
rows, err := w.db.QueryContext(ctx,
`SELECT agent_name FROM channel_members WHERE channel_id = ?`,
channelID,
)
if err != nil {
w.logger.Error("query channel members for workflow reminder failed",
"channel_id", channelID,
"error", err,
)
return 0
}
defer rows.Close()
var members []string
for rows.Next() {
var name string
if err := rows.Scan(&name); err != nil {
continue
}
members = append(members, name)
}
age := formatAge(msg.StateAge)
truncBody := truncate(msg.Body, 100)
count := int64(0)
for _, member := range members {
body := fmt.Sprintf(
"**STALE**: Message #%d in #%s in '%s' for %s. \"%s\" — @%s",
msg.ID, msg.Channel, msg.State, age, truncBody, msg.FromAgent,
)
_, err := w.msgService.SendMessage(ctx, "system", member, body, SendOptions{
Subject: fmt.Sprintf("workflow-stalemate-reminder:%d", msg.ID),
Priority: 7,
Metadata: fmt.Sprintf(`{"workflow_stalemate_reminder_for":%d}`, msg.ID),
})
if err != nil {
w.logger.Error("send workflow stalemate reminder failed",
"message_id", msg.ID,
"to_agent", member,
"error", err,
)
continue
}
w.logger.Info("sent workflow stalemate reminder",
"message_id", msg.ID,
"channel", msg.Channel,
"state", msg.State,
"to_agent", member,
"age", age,
)
count++
}
return count
}
// sendWorkflowEscalation posts an escalation to #approvals for a stale workflow message.
func (w *StalemateWorker) sendWorkflowEscalation(ctx context.Context, msg staleWorkflowMsg) bool {
approvalsChanID, err := w.channelLookup.GetChannelIDByName(ctx, "approvals")
if err != nil {
w.logger.Warn("cannot escalate workflow stalemate: #approvals channel not found", "error", err)
return false
}
age := formatAge(msg.StateAge)
truncBody := truncate(msg.Body, 100)
body := fmt.Sprintf(
"**STALE**: Message #%d in #%s in '%s' for %s. \"%s\" — @%s",
msg.ID, msg.Channel, msg.State, age, truncBody, msg.FromAgent,
)
_, err = w.msgService.SendMessage(ctx, "system", "", body, SendOptions{
Subject: fmt.Sprintf("workflow-stalemate-escalation:%d", msg.ID),
Priority: 9,
Metadata: fmt.Sprintf(`{"workflow_stalemate_escalation_for":%d}`, msg.ID),
ChannelID: &approvalsChanID,
})
if err != nil {
w.logger.Error("send workflow escalation to #approvals failed",
"message_id", msg.ID,
"channel", msg.Channel,
"error", err,
)
return false
}
w.logger.Info("escalated stale workflow message to #approvals",
"message_id", msg.ID,
"channel", msg.Channel,
"state", msg.State,
"age", age,
)
return true
}
// workflowReminderExists checks if a workflow stalemate reminder already exists for a message.
func (w *StalemateWorker) workflowReminderExists(ctx context.Context, messageID int64) bool {
var count int
err := w.db.QueryRowContext(ctx,
`SELECT COUNT(*) FROM messages
WHERE from_agent = 'system'
AND metadata LIKE ?`,
fmt.Sprintf(`%%"workflow_stalemate_reminder_for":%d%%`, messageID),
).Scan(&count)
if err != nil {
return false
}
return count > 0
}
// workflowEscalationExists checks if a workflow stalemate escalation already exists for a message.
func (w *StalemateWorker) workflowEscalationExists(ctx context.Context, messageID int64) bool {
var count int
err := w.db.QueryRowContext(ctx,
`SELECT COUNT(*) FROM messages
WHERE from_agent = 'system'
AND metadata LIKE ?`,
fmt.Sprintf(`%%"workflow_stalemate_escalation_for":%d%%`, messageID),
).Scan(&count)
if err != nil {
return false
}
return count > 0
}
// truncate truncates a string to maxLen characters, appending "..." if truncated.
func truncate(s string, maxLen int) string {
runes := []rune(s)
if len(runes) <= maxLen {
return s
}
return string(runes[:maxLen]) + "..."
}
// formatAge returns a human-readable age string.
func formatAge(d time.Duration) string {
if d < time.Hour {
return fmt.Sprintf("%dm", int(d.Minutes()))
}
hours := int(d.Hours())
if hours < 24 {
return fmt.Sprintf("%dh", hours)
}
days := hours / 24
remainingHours := hours % 24
if remainingHours == 0 {
if days == 1 {
return "1 day"
}
return fmt.Sprintf("%d days", days)
}
if days == 1 {
return fmt.Sprintf("1 day %dh", remainingHours)
}
return fmt.Sprintf("%d days %dh", days, remainingHours)
}
+16 -626
View File
@@ -3,7 +3,6 @@ package messaging
import (
"context"
"database/sql"
"fmt"
"os"
"testing"
"time"
@@ -13,19 +12,6 @@ import (
"github.com/synapbus/synapbus/internal/trace"
)
// stubChannelLookup implements ChannelLookup for tests.
type stubChannelLookup struct {
channelID int64
err error
}
func (s *stubChannelLookup) GetChannelIDByName(ctx context.Context, name string) (int64, error) {
if s.err != nil {
return 0, s.err
}
return s.channelID, nil
}
// newStalemateTestService creates a MessagingService and DB for stalemate tests.
func newStalemateTestService(t *testing.T) (*MessagingService, *sql.DB) {
t.Helper()
@@ -47,7 +33,6 @@ func newStalemateTestService(t *testing.T) (*MessagingService, *sql.DB) {
func insertStaleMessage(t *testing.T, db *sql.DB, from, to, body, status string, createdAt time.Time, claimedAt *time.Time, claimedBy string) int64 {
t.Helper()
// Insert conversation first
result, err := db.Exec(
`INSERT INTO conversations (subject, created_by, created_at, updated_at)
VALUES (?, ?, ?, ?)`,
@@ -83,19 +68,15 @@ func TestStalemateWorker_ProcessingTimeout(t *testing.T) {
svc, db := newStalemateTestService(t)
ctx := context.Background()
// Insert a message in "processing" status with old claimed_at
oldClaimedAt := time.Now().Add(-25 * time.Hour)
msgID := insertStaleMessage(t, db, "sender", "receiver", "stale processing task", StatusProcessing, time.Now().Add(-26*time.Hour), &oldClaimedAt, "receiver")
config := DefaultStalemateConfig()
config.ProcessingTimeout = 24 * time.Hour
lookup := &stubChannelLookup{channelID: 0, err: fmt.Errorf("no channel")}
worker := NewStalemateWorker(db, svc, lookup, config)
worker := NewStalemateWorker(db, svc, config)
worker.checkStaleMessages(ctx)
// Verify message was auto-failed
var status, metadata string
err := db.QueryRowContext(ctx, `SELECT status, metadata FROM messages WHERE id = ?`, msgID).Scan(&status, &metadata)
if err != nil {
@@ -113,19 +94,15 @@ func TestStalemateWorker_ProcessingTimeout_NotExpired(t *testing.T) {
svc, db := newStalemateTestService(t)
ctx := context.Background()
// Insert a message in "processing" status with recent claimed_at (should NOT be failed)
recentClaimedAt := time.Now().Add(-1 * time.Hour)
msgID := insertStaleMessage(t, db, "sender", "receiver", "recent processing task", StatusProcessing, time.Now().Add(-2*time.Hour), &recentClaimedAt, "receiver")
config := DefaultStalemateConfig()
config.ProcessingTimeout = 24 * time.Hour
lookup := &stubChannelLookup{channelID: 0, err: fmt.Errorf("no channel")}
worker := NewStalemateWorker(db, svc, lookup, config)
worker := NewStalemateWorker(db, svc, config)
worker.checkStaleMessages(ctx)
// Verify message was NOT auto-failed
var status string
err := db.QueryRowContext(ctx, `SELECT status FROM messages WHERE id = ?`, msgID).Scan(&status)
if err != nil {
@@ -136,29 +113,22 @@ func TestStalemateWorker_ProcessingTimeout_NotExpired(t *testing.T) {
}
}
// TestStalemateWorker_ProcessingTimeout_RaceGuard verifies that the
// auto-fail UPDATE re-checks claimed_at < cutoff and won't stomp a row that
// was legitimately re-claimed (claimed_at refreshed) between the worker's
// SELECT scan and its row-by-row UPDATE. This guards the TOCTOU window
// the stale-worker race depends on.
// TestStalemateWorker_ProcessingTimeout_RaceGuard verifies that the auto-fail
// UPDATE re-checks claimed_at < cutoff so a row re-claimed between SELECT and
// UPDATE is not stomped. Guards the TOCTOU window the stale-worker race
// depends on.
func TestStalemateWorker_ProcessingTimeout_RaceGuard(t *testing.T) {
svc, db := newStalemateTestService(t)
ctx := context.Background()
// Insert a message that *was* stale at SELECT time.
oldClaimedAt := time.Now().Add(-25 * time.Hour)
msgID := insertStaleMessage(t, db, "sender", "receiver", "racing task",
StatusProcessing, time.Now().Add(-26*time.Hour), &oldClaimedAt, "receiver")
config := DefaultStalemateConfig()
config.ProcessingTimeout = 24 * time.Hour
lookup := &stubChannelLookup{channelID: 0, err: fmt.Errorf("no channel")}
worker := NewStalemateWorker(db, svc, lookup, config)
worker := NewStalemateWorker(db, svc, config)
// Simulate the race: between the worker's SELECT (which would have picked
// this row) and its UPDATE, the legitimate claimer refreshes claimed_at to
// "now". With the cutoff predicate in place the UPDATE no-ops instead of
// silently failing live work.
freshClaimedAt := time.Now()
if _, err := db.ExecContext(ctx,
`UPDATE messages SET claimed_at = ? WHERE id = ?`,
@@ -180,177 +150,6 @@ func TestStalemateWorker_ProcessingTimeout_RaceGuard(t *testing.T) {
}
}
func TestStalemateWorker_PendingReminder(t *testing.T) {
svc, db := newStalemateTestService(t)
ctx := context.Background()
// Insert a pending DM that is 5 hours old
insertStaleMessage(t, db, "sender", "receiver", "please review this", StatusPending, time.Now().Add(-5*time.Hour), nil, "")
config := DefaultStalemateConfig()
config.ReminderAfter = 4 * time.Hour
config.EscalateAfter = 48 * time.Hour // won't trigger
lookup := &stubChannelLookup{channelID: 0, err: fmt.Errorf("no channel")}
worker := NewStalemateWorker(db, svc, lookup, config)
worker.checkStaleMessages(ctx)
// Verify a system reminder was sent to receiver
var count int
err := db.QueryRowContext(ctx,
`SELECT COUNT(*) FROM messages WHERE from_agent = 'system' AND to_agent = 'receiver' AND body LIKE '%Reminder%'`,
).Scan(&count)
if err != nil {
t.Fatalf("query reminder: %v", err)
}
if count != 1 {
t.Errorf("expected 1 reminder, got %d", count)
}
}
func TestStalemateWorker_SystemMessageSkip(t *testing.T) {
svc, db := newStalemateTestService(t)
ctx := context.Background()
// Insert a pending DM FROM system (should be skipped)
insertStaleMessage(t, db, "system", "receiver", "system notification", StatusPending, time.Now().Add(-5*time.Hour), nil, "")
config := DefaultStalemateConfig()
config.ReminderAfter = 4 * time.Hour
lookup := &stubChannelLookup{channelID: 0, err: fmt.Errorf("no channel")}
worker := NewStalemateWorker(db, svc, lookup, config)
worker.checkStaleMessages(ctx)
// Verify NO reminder was sent (only the original system message should exist)
var count int
err := db.QueryRowContext(ctx,
`SELECT COUNT(*) FROM messages WHERE from_agent = 'system' AND body LIKE '%Reminder%'`,
).Scan(&count)
if err != nil {
t.Fatalf("query reminder: %v", err)
}
if count != 0 {
t.Errorf("expected 0 reminders for system message, got %d", count)
}
}
func TestStalemateWorker_DuplicateReminderPrevention(t *testing.T) {
svc, db := newStalemateTestService(t)
ctx := context.Background()
// Insert a pending DM that is old enough for a reminder
insertStaleMessage(t, db, "sender", "receiver", "need your attention", StatusPending, time.Now().Add(-5*time.Hour), nil, "")
config := DefaultStalemateConfig()
config.ReminderAfter = 4 * time.Hour
config.EscalateAfter = 48 * time.Hour
lookup := &stubChannelLookup{channelID: 0, err: fmt.Errorf("no channel")}
worker := NewStalemateWorker(db, svc, lookup, config)
// Run check twice
worker.checkStaleMessages(ctx)
worker.checkStaleMessages(ctx)
// Verify only ONE reminder was sent
var count int
err := db.QueryRowContext(ctx,
`SELECT COUNT(*) FROM messages WHERE from_agent = 'system' AND to_agent = 'receiver' AND body LIKE '%Reminder%'`,
).Scan(&count)
if err != nil {
t.Fatalf("query reminders: %v", err)
}
if count != 1 {
t.Errorf("expected 1 reminder (no duplicates), got %d", count)
}
}
func TestStalemateWorker_Escalation(t *testing.T) {
svc, db := newStalemateTestService(t)
ctx := context.Background()
// Create #approvals channel
_, err := db.Exec(
`INSERT INTO channels (id, name, description, topic, type, is_private, is_system, created_by, created_at, updated_at)
VALUES (1, 'approvals', 'Approval queue', '', 'standard', 0, 0, 'system', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)`)
if err != nil {
t.Fatalf("create approvals channel: %v", err)
}
// Add system as member
_, err = db.Exec(
`INSERT INTO channel_members (channel_id, agent_name, role, joined_at)
VALUES (1, 'system', 'owner', CURRENT_TIMESTAMP)`)
if err != nil {
t.Fatalf("add system to channel: %v", err)
}
// Insert a pending DM that is 49 hours old (beyond escalation threshold)
insertStaleMessage(t, db, "sender", "receiver", "urgent task ignored", StatusPending, time.Now().Add(-49*time.Hour), nil, "")
config := DefaultStalemateConfig()
config.ReminderAfter = 4 * time.Hour
config.EscalateAfter = 48 * time.Hour
lookup := &stubChannelLookup{channelID: 1}
worker := NewStalemateWorker(db, svc, lookup, config)
worker.checkStaleMessages(ctx)
// Verify an escalation was sent to #approvals channel
var count int
err = db.QueryRowContext(ctx,
`SELECT COUNT(*) FROM messages WHERE from_agent = 'system' AND channel_id = 1 AND body LIKE '%ESCALATION%'`,
).Scan(&count)
if err != nil {
t.Fatalf("query escalations: %v", err)
}
if count != 1 {
t.Errorf("expected 1 escalation, got %d", count)
}
}
func TestStalemateWorker_DuplicateEscalationPrevention(t *testing.T) {
svc, db := newStalemateTestService(t)
ctx := context.Background()
// Create #approvals channel
db.Exec(
`INSERT INTO channels (id, name, description, topic, type, is_private, is_system, created_by, created_at, updated_at)
VALUES (1, 'approvals', 'Approval queue', '', 'standard', 0, 0, 'system', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)`)
db.Exec(
`INSERT INTO channel_members (channel_id, agent_name, role, joined_at)
VALUES (1, 'system', 'owner', CURRENT_TIMESTAMP)`)
// Insert a pending DM that is 49 hours old
insertStaleMessage(t, db, "sender", "receiver", "urgent task", StatusPending, time.Now().Add(-49*time.Hour), nil, "")
config := DefaultStalemateConfig()
config.ReminderAfter = 4 * time.Hour
config.EscalateAfter = 48 * time.Hour
lookup := &stubChannelLookup{channelID: 1}
worker := NewStalemateWorker(db, svc, lookup, config)
// Run check twice
worker.checkStaleMessages(ctx)
worker.checkStaleMessages(ctx)
// Verify only ONE escalation was sent
var count int
err := db.QueryRowContext(ctx,
`SELECT COUNT(*) FROM messages WHERE from_agent = 'system' AND channel_id = 1 AND body LIKE '%ESCALATION%'`,
).Scan(&count)
if err != nil {
t.Fatalf("query escalations: %v", err)
}
if count != 1 {
t.Errorf("expected 1 escalation (no duplicates), got %d", count)
}
}
func TestParseStalemateConfig(t *testing.T) {
tests := []struct {
name string
@@ -366,14 +165,10 @@ func TestParseStalemateConfig(t *testing.T) {
name: "custom values with day format",
envVars: map[string]string{
"SYNAPBUS_STALEMATE_PROCESSING_TIMEOUT": "7d",
"SYNAPBUS_STALEMATE_REMINDER_AFTER": "8h",
"SYNAPBUS_STALEMATE_ESCALATE_AFTER": "3d",
"SYNAPBUS_STALEMATE_INTERVAL": "30m",
},
expected: StalemateConfig{
ProcessingTimeout: 7 * 24 * time.Hour,
ReminderAfter: 8 * time.Hour,
EscalateAfter: 3 * 24 * time.Hour,
Interval: 30 * time.Minute,
},
},
@@ -381,14 +176,10 @@ func TestParseStalemateConfig(t *testing.T) {
name: "standard Go duration format",
envVars: map[string]string{
"SYNAPBUS_STALEMATE_PROCESSING_TIMEOUT": "48h",
"SYNAPBUS_STALEMATE_REMINDER_AFTER": "2h30m",
"SYNAPBUS_STALEMATE_ESCALATE_AFTER": "72h",
"SYNAPBUS_STALEMATE_INTERVAL": "5m",
},
expected: StalemateConfig{
ProcessingTimeout: 48 * time.Hour,
ReminderAfter: 2*time.Hour + 30*time.Minute,
EscalateAfter: 72 * time.Hour,
Interval: 5 * time.Minute,
},
},
@@ -396,28 +187,22 @@ func TestParseStalemateConfig(t *testing.T) {
name: "invalid values fall back to defaults",
envVars: map[string]string{
"SYNAPBUS_STALEMATE_PROCESSING_TIMEOUT": "invalid",
"SYNAPBUS_STALEMATE_REMINDER_AFTER": "bad",
"SYNAPBUS_STALEMATE_ESCALATE_AFTER": "",
"SYNAPBUS_STALEMATE_INTERVAL": "-5m",
},
expected: DefaultStalemateConfig(),
},
}
envKeys := []string{
"SYNAPBUS_STALEMATE_PROCESSING_TIMEOUT",
"SYNAPBUS_STALEMATE_INTERVAL",
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// Clear all env vars first
envKeys := []string{
"SYNAPBUS_STALEMATE_PROCESSING_TIMEOUT",
"SYNAPBUS_STALEMATE_REMINDER_AFTER",
"SYNAPBUS_STALEMATE_ESCALATE_AFTER",
"SYNAPBUS_STALEMATE_INTERVAL",
}
for _, k := range envKeys {
os.Unsetenv(k)
}
// Set test env vars
for k, v := range tt.envVars {
os.Setenv(k, v)
}
@@ -432,12 +217,6 @@ func TestParseStalemateConfig(t *testing.T) {
if cfg.ProcessingTimeout != tt.expected.ProcessingTimeout {
t.Errorf("ProcessingTimeout = %v, want %v", cfg.ProcessingTimeout, tt.expected.ProcessingTimeout)
}
if cfg.ReminderAfter != tt.expected.ReminderAfter {
t.Errorf("ReminderAfter = %v, want %v", cfg.ReminderAfter, tt.expected.ReminderAfter)
}
if cfg.EscalateAfter != tt.expected.EscalateAfter {
t.Errorf("EscalateAfter = %v, want %v", cfg.EscalateAfter, tt.expected.EscalateAfter)
}
if cfg.Interval != tt.expected.Interval {
t.Errorf("Interval = %v, want %v", cfg.Interval, tt.expected.Interval)
}
@@ -447,10 +226,10 @@ func TestParseStalemateConfig(t *testing.T) {
func TestParseDurationWithDays(t *testing.T) {
tests := []struct {
name string
input string
want time.Duration
wantErr bool
name string
input string
want time.Duration
wantErr bool
}{
{"7 days", "7d", 7 * 24 * time.Hour, false},
{"1 day", "1d", 24 * time.Hour, false},
@@ -475,392 +254,3 @@ func TestParseDurationWithDays(t *testing.T) {
})
}
}
func TestTruncate(t *testing.T) {
tests := []struct {
name string
input string
maxLen int
want string
}{
{"short string", "hello", 10, "hello"},
{"exact length", "hello", 5, "hello"},
{"truncated", "hello world, this is a long message", 10, "hello worl..."},
{"empty", "", 10, ""},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := truncate(tt.input, tt.maxLen)
if got != tt.want {
t.Errorf("truncate(%q, %d) = %q, want %q", tt.input, tt.maxLen, got, tt.want)
}
})
}
}
func TestFormatAge(t *testing.T) {
tests := []struct {
name string
d time.Duration
want string
}{
{"minutes", 30 * time.Minute, "30m"},
{"hours", 5 * time.Hour, "5h"},
{"1 day", 24 * time.Hour, "1 day"},
{"2 days", 48 * time.Hour, "2 days"},
{"1 day with hours", 25 * time.Hour, "1 day 1h"},
{"2 days with hours", 50 * time.Hour, "2 days 2h"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := formatAge(tt.d)
if got != tt.want {
t.Errorf("formatAge(%v) = %q, want %q", tt.d, got, tt.want)
}
})
}
}
func TestComputeWorkflowStateFromReactions(t *testing.T) {
tests := []struct {
name string
reactions []reactionRow
want string
}{
{"no reactions = proposed", nil, "proposed"},
{"approve only", []reactionRow{{Reaction: "approve"}}, "approved"},
{"in_progress only", []reactionRow{{Reaction: "in_progress"}}, "in_progress"},
{"reject only", []reactionRow{{Reaction: "reject"}}, "rejected"},
{"done only", []reactionRow{{Reaction: "done"}}, "done"},
{"published only", []reactionRow{{Reaction: "published"}}, "published"},
{"approve + in_progress = in_progress (higher priority)", []reactionRow{
{Reaction: "approve"},
{Reaction: "in_progress"},
}, "in_progress"},
{"approve + done = done", []reactionRow{
{Reaction: "approve"},
{Reaction: "done"},
}, "done"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := computeWorkflowStateFromReactions(tt.reactions)
if got != tt.want {
t.Errorf("computeWorkflowStateFromReactions() = %q, want %q", got, tt.want)
}
})
}
}
func TestIsTerminalWorkflowState(t *testing.T) {
tests := []struct {
state string
terminal bool
}{
{"proposed", false},
{"approved", false},
{"in_progress", false},
{"rejected", true},
{"done", true},
{"published", true},
}
for _, tt := range tests {
t.Run(tt.state, func(t *testing.T) {
got := isTerminalWorkflowState(tt.state)
if got != tt.terminal {
t.Errorf("isTerminalWorkflowState(%q) = %v, want %v", tt.state, got, tt.terminal)
}
})
}
}
func TestStalemateWorker_WorkflowReminder(t *testing.T) {
svc, db := newStalemateTestService(t)
ctx := context.Background()
// Create a workflow-enabled channel with short timeouts
_, err := db.Exec(
`INSERT INTO channels (id, name, description, topic, type, is_private, is_system, created_by, workflow_enabled, stalemate_remind_after, stalemate_escalate_after, created_at, updated_at)
VALUES (10, 'news-test', 'Test news channel', '', 'standard', 0, 0, 'system', 1, '1s', '72h', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)`)
if err != nil {
t.Fatalf("create workflow channel: %v", err)
}
// Add system and sender as members
db.Exec(`INSERT INTO channel_members (channel_id, agent_name, role, joined_at) VALUES (10, 'system', 'owner', CURRENT_TIMESTAMP)`)
db.Exec(`INSERT INTO channel_members (channel_id, agent_name, role, joined_at) VALUES (10, 'sender', 'member', CURRENT_TIMESTAMP)`)
db.Exec(`INSERT INTO channel_members (channel_id, agent_name, role, joined_at) VALUES (10, 'receiver', 'member', CURRENT_TIMESTAMP)`)
// Insert a channel message with old created_at (will be in "proposed" state since no reactions)
oldTime := time.Now().Add(-2 * time.Second)
convResult, err := db.Exec(
`INSERT INTO conversations (subject, created_by, created_at, updated_at) VALUES ('wf-test', 'sender', ?, ?)`,
oldTime, oldTime,
)
if err != nil {
t.Fatalf("insert conversation: %v", err)
}
convID, _ := convResult.LastInsertId()
channelID := int64(10)
_, err = db.Exec(
`INSERT INTO messages (conversation_id, from_agent, to_agent, body, priority, status, metadata, channel_id, created_at, updated_at)
VALUES (?, 'sender', '', 'Draft blog post about MCP', 5, 'pending', '{}', ?, ?, ?)`,
convID, channelID, oldTime, oldTime,
)
if err != nil {
t.Fatalf("insert channel message: %v", err)
}
// Wait for the timeout to elapse
time.Sleep(10 * time.Millisecond)
config := DefaultStalemateConfig()
lookup := &stubChannelLookup{channelID: 0, err: fmt.Errorf("no approvals channel")}
worker := NewStalemateWorker(db, svc, lookup, config)
worker.checkStaleMessages(ctx)
// Verify workflow stalemate reminders were sent to channel members
var count int
err = db.QueryRowContext(ctx,
`SELECT COUNT(*) FROM messages WHERE from_agent = 'system' AND body LIKE '%STALE%'`,
).Scan(&count)
if err != nil {
t.Fatalf("query workflow reminders: %v", err)
}
// Should have sent reminders to all 3 members (system, sender, receiver)
if count < 1 {
t.Errorf("expected at least 1 workflow reminder, got %d", count)
}
}
func TestStalemateWorker_WorkflowEscalation(t *testing.T) {
svc, db := newStalemateTestService(t)
ctx := context.Background()
// Create a workflow-enabled channel with short escalation timeout
_, err := db.Exec(
`INSERT INTO channels (id, name, description, topic, type, is_private, is_system, created_by, workflow_enabled, stalemate_remind_after, stalemate_escalate_after, created_at, updated_at)
VALUES (10, 'news-test', 'Test news channel', '', 'standard', 0, 0, 'system', 1, '1s', '1s', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)`)
if err != nil {
t.Fatalf("create workflow channel: %v", err)
}
// Create #approvals channel
db.Exec(
`INSERT INTO channels (id, name, description, topic, type, is_private, is_system, created_by, created_at, updated_at)
VALUES (20, 'approvals', 'Approval queue', '', 'standard', 0, 0, 'system', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)`)
db.Exec(`INSERT INTO channel_members (channel_id, agent_name, role, joined_at) VALUES (20, 'system', 'owner', CURRENT_TIMESTAMP)`)
// Add members to workflow channel
db.Exec(`INSERT INTO channel_members (channel_id, agent_name, role, joined_at) VALUES (10, 'sender', 'member', CURRENT_TIMESTAMP)`)
// Insert a channel message old enough to trigger escalation
oldTime := time.Now().Add(-2 * time.Second)
convResult, _ := db.Exec(
`INSERT INTO conversations (subject, created_by, created_at, updated_at) VALUES ('wf-esc', 'sender', ?, ?)`,
oldTime, oldTime,
)
convID, _ := convResult.LastInsertId()
channelID := int64(10)
_, err = db.Exec(
`INSERT INTO messages (conversation_id, from_agent, to_agent, body, priority, status, metadata, channel_id, created_at, updated_at)
VALUES (?, 'sender', '', 'Stale proposal needing attention', 5, 'pending', '{}', ?, ?, ?)`,
convID, channelID, oldTime, oldTime,
)
if err != nil {
t.Fatalf("insert channel message: %v", err)
}
time.Sleep(10 * time.Millisecond)
config := DefaultStalemateConfig()
lookup := &stubChannelLookup{channelID: 20}
worker := NewStalemateWorker(db, svc, lookup, config)
worker.checkStaleMessages(ctx)
// Verify escalation was sent to #approvals
var count int
err = db.QueryRowContext(ctx,
`SELECT COUNT(*) FROM messages WHERE from_agent = 'system' AND channel_id = 20 AND body LIKE '%STALE%'`,
).Scan(&count)
if err != nil {
t.Fatalf("query workflow escalation: %v", err)
}
if count != 1 {
t.Errorf("expected 1 workflow escalation, got %d", count)
}
}
func TestStalemateWorker_WorkflowTerminalStateSkip(t *testing.T) {
svc, db := newStalemateTestService(t)
ctx := context.Background()
// Create a workflow-enabled channel with short timeouts
_, err := db.Exec(
`INSERT INTO channels (id, name, description, topic, type, is_private, is_system, created_by, workflow_enabled, stalemate_remind_after, stalemate_escalate_after, created_at, updated_at)
VALUES (10, 'news-test', 'Test news channel', '', 'standard', 0, 0, 'system', 1, '1s', '1s', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)`)
if err != nil {
t.Fatalf("create workflow channel: %v", err)
}
db.Exec(`INSERT INTO channel_members (channel_id, agent_name, role, joined_at) VALUES (10, 'sender', 'member', CURRENT_TIMESTAMP)`)
// Insert a channel message
oldTime := time.Now().Add(-2 * time.Second)
convResult, _ := db.Exec(
`INSERT INTO conversations (subject, created_by, created_at, updated_at) VALUES ('wf-done', 'sender', ?, ?)`,
oldTime, oldTime,
)
convID, _ := convResult.LastInsertId()
channelID := int64(10)
msgResult, err := db.Exec(
`INSERT INTO messages (conversation_id, from_agent, to_agent, body, priority, status, metadata, channel_id, created_at, updated_at)
VALUES (?, 'sender', '', 'Completed task', 5, 'pending', '{}', ?, ?, ?)`,
convID, channelID, oldTime, oldTime,
)
if err != nil {
t.Fatalf("insert channel message: %v", err)
}
msgID, _ := msgResult.LastInsertId()
// Add a "done" reaction — puts it in terminal state
_, err = db.Exec(
`INSERT INTO message_reactions (message_id, agent_name, reaction, metadata, created_at)
VALUES (?, 'sender', 'done', '{}', ?)`,
msgID, oldTime,
)
if err != nil {
t.Fatalf("insert reaction: %v", err)
}
time.Sleep(10 * time.Millisecond)
config := DefaultStalemateConfig()
lookup := &stubChannelLookup{channelID: 0, err: fmt.Errorf("no approvals")}
worker := NewStalemateWorker(db, svc, lookup, config)
worker.checkStaleMessages(ctx)
// Verify NO reminders were sent (message is in terminal "done" state)
var count int
err = db.QueryRowContext(ctx,
`SELECT COUNT(*) FROM messages WHERE from_agent = 'system' AND body LIKE '%STALE%'`,
).Scan(&count)
if err != nil {
t.Fatalf("query reminders: %v", err)
}
if count != 0 {
t.Errorf("expected 0 reminders for terminal state message, got %d", count)
}
}
func TestStalemateWorker_WorkflowDuplicateReminderPrevention(t *testing.T) {
svc, db := newStalemateTestService(t)
ctx := context.Background()
// Create a workflow-enabled channel with short timeout
_, err := db.Exec(
`INSERT INTO channels (id, name, description, topic, type, is_private, is_system, created_by, workflow_enabled, stalemate_remind_after, stalemate_escalate_after, created_at, updated_at)
VALUES (10, 'news-test', 'Test', '', 'standard', 0, 0, 'system', 1, '1s', '72h', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)`)
if err != nil {
t.Fatalf("create workflow channel: %v", err)
}
db.Exec(`INSERT INTO channel_members (channel_id, agent_name, role, joined_at) VALUES (10, 'receiver', 'member', CURRENT_TIMESTAMP)`)
// Insert a channel message
oldTime := time.Now().Add(-2 * time.Second)
convResult, _ := db.Exec(
`INSERT INTO conversations (subject, created_by, created_at, updated_at) VALUES ('wf-dup', 'sender', ?, ?)`,
oldTime, oldTime,
)
convID, _ := convResult.LastInsertId()
channelID := int64(10)
_, err = db.Exec(
`INSERT INTO messages (conversation_id, from_agent, to_agent, body, priority, status, metadata, channel_id, created_at, updated_at)
VALUES (?, 'sender', '', 'Needs review', 5, 'pending', '{}', ?, ?, ?)`,
convID, channelID, oldTime, oldTime,
)
if err != nil {
t.Fatalf("insert channel message: %v", err)
}
time.Sleep(10 * time.Millisecond)
config := DefaultStalemateConfig()
lookup := &stubChannelLookup{channelID: 0, err: fmt.Errorf("no approvals")}
worker := NewStalemateWorker(db, svc, lookup, config)
// Run twice
worker.checkStaleMessages(ctx)
worker.checkStaleMessages(ctx)
// Verify only one set of reminders was sent (no duplicates)
var count int
err = db.QueryRowContext(ctx,
`SELECT COUNT(*) FROM messages WHERE from_agent = 'system' AND to_agent = 'receiver' AND body LIKE '%STALE%'`,
).Scan(&count)
if err != nil {
t.Fatalf("query reminders: %v", err)
}
if count != 1 {
t.Errorf("expected 1 reminder (no duplicates), got %d", count)
}
}
func TestStalemateWorker_WorkflowNonWorkflowChannelSkip(t *testing.T) {
svc, db := newStalemateTestService(t)
ctx := context.Background()
// Create a channel with workflow DISABLED
_, err := db.Exec(
`INSERT INTO channels (id, name, description, topic, type, is_private, is_system, created_by, workflow_enabled, stalemate_remind_after, stalemate_escalate_after, created_at, updated_at)
VALUES (10, 'general', 'General', '', 'standard', 0, 0, 'system', 0, '1s', '1s', CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)`)
if err != nil {
t.Fatalf("create channel: %v", err)
}
db.Exec(`INSERT INTO channel_members (channel_id, agent_name, role, joined_at) VALUES (10, 'sender', 'member', CURRENT_TIMESTAMP)`)
// Insert a channel message
oldTime := time.Now().Add(-2 * time.Second)
convResult, _ := db.Exec(
`INSERT INTO conversations (subject, created_by, created_at, updated_at) VALUES ('no-wf', 'sender', ?, ?)`,
oldTime, oldTime,
)
convID, _ := convResult.LastInsertId()
channelID := int64(10)
db.Exec(
`INSERT INTO messages (conversation_id, from_agent, to_agent, body, priority, status, metadata, channel_id, created_at, updated_at)
VALUES (?, 'sender', '', 'No workflow here', 5, 'pending', '{}', ?, ?, ?)`,
convID, channelID, oldTime, oldTime,
)
time.Sleep(10 * time.Millisecond)
config := DefaultStalemateConfig()
lookup := &stubChannelLookup{channelID: 0, err: fmt.Errorf("no approvals")}
worker := NewStalemateWorker(db, svc, lookup, config)
worker.checkStaleMessages(ctx)
// Verify NO reminders — channel is not workflow-enabled
var count int
err = db.QueryRowContext(ctx,
`SELECT COUNT(*) FROM messages WHERE from_agent = 'system' AND body LIKE '%STALE%'`,
).Scan(&count)
if err != nil {
t.Fatalf("query reminders: %v", err)
}
if count != 0 {
t.Errorf("expected 0 reminders for non-workflow channel, got %d", count)
}
}
+1
View File
@@ -41,6 +41,7 @@ type MessageStore interface {
GetConversationIDsForChannel(ctx context.Context, channelID int64, lastMessageID int64) ([]int64, error)
GetConversationIDsForDM(ctx context.Context, agentNames []string, peerAgent string, lastMessageID int64) ([]int64, error)
GetReplyCounts(ctx context.Context, messageIDs []int64) (map[int64]int, error)
ListEventMetaAfter(ctx context.Context, agentName string, afterID int64, limit int) ([]*EventMeta, error)
}
// SQLiteMessageStore implements MessageStore using SQLite.
+77
View File
@@ -84,4 +84,81 @@ var (
},
[]string{"agent"},
)
// Dream worker metrics (feature 020 follow-up).
DreamJobsTotal = promauto.NewCounterVec(
prometheus.CounterOpts{
Namespace: "synapbus",
Name: "dream_jobs_total",
Help: "Dream worker jobs dispatched, labeled by owner, job_type and final status",
},
[]string{"owner", "job_type", "status"},
)
DreamTokensTotal = promauto.NewCounterVec(
prometheus.CounterOpts{
Namespace: "synapbus",
Name: "dream_tokens_total",
Help: "Total tokens consumed by dream worker runs, labeled by owner and direction (in|out)",
},
[]string{"owner", "direction"},
)
DreamJobDuration = promauto.NewHistogramVec(
prometheus.HistogramOpts{
Namespace: "synapbus",
Name: "dream_job_duration_seconds",
Help: "Wallclock duration of a single dream worker job",
Buckets: prometheus.DefBuckets,
},
[]string{"owner", "job_type"},
)
DreamCircuitBrokenTotal = promauto.NewCounterVec(
prometheus.CounterOpts{
Namespace: "synapbus",
Name: "dream_circuit_broken_total",
Help: "Times the dream worker refused to dispatch because the daily usage gate fired",
},
[]string{"owner", "reason"},
)
// Proactive-injection metrics (feature 020 follow-up).
InjectionPacketsTotal = promauto.NewCounterVec(
prometheus.CounterOpts{
Namespace: "synapbus",
Name: "injection_packets_total",
Help: "Number of relevant_context packets attached to MCP tool responses, by tool",
},
[]string{"tool"},
)
InjectionMemoriesPerPacket = promauto.NewHistogramVec(
prometheus.HistogramOpts{
Namespace: "synapbus",
Name: "injection_memories_per_packet",
Help: "Distribution of memory items per injection packet, by tool",
Buckets: []float64{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10},
},
[]string{"tool"},
)
InjectionPacketChars = promauto.NewHistogramVec(
prometheus.HistogramOpts{
Namespace: "synapbus",
Name: "injection_packet_chars",
Help: "Distribution of character size of an injection packet, by tool",
Buckets: []float64{0, 250, 500, 750, 1000, 1250, 1500, 1750, 2000, 2250, 2500},
},
[]string{"tool"},
)
InjectionSkippedTotal = promauto.NewCounterVec(
prometheus.CounterOpts{
Namespace: "synapbus",
Name: "injection_skipped_total",
Help: "Times injection was skipped, labeled by tool and reason (no_owner|empty_pool|disabled)",
},
[]string{"tool", "reason"},
)
)
+560
View File
@@ -0,0 +1,560 @@
// Proactive-memory injection retrieval — builds the relevant-context
// packet attached to every injection-eligible MCP tool response (per
// `contracts/mcp-injection.md`).
package search
import (
"context"
"database/sql"
"errors"
"fmt"
"strconv"
"time"
"github.com/synapbus/synapbus/internal/agents"
)
// MemoryItem is one entry in the `relevant_context.memories[]` array.
// Field shape is contractual (`contracts/mcp-injection.md`).
type MemoryItem struct {
ID int64 `json:"id"`
FromAgent string `json:"from_agent"`
Channel string `json:"channel,omitempty"`
Body string `json:"body"`
CreatedAt time.Time `json:"created_at"`
Score float64 `json:"score"`
MatchType string `json:"match_type"`
Pinned bool `json:"pinned"`
Truncated bool `json:"truncated,omitempty"`
}
// ContextPacket is the body of the `relevant_context` field. Returned
// from BuildContextPacket; rendered verbatim into the wrapped tool
// response.
type ContextPacket struct {
Memories []MemoryItem `json:"memories"`
CoreMemory string `json:"core_memory,omitempty"`
PacketChars int `json:"packet_chars"`
PacketTokenEstimate int `json:"packet_token_estimate"`
RetrievalQuery string `json:"retrieval_query"`
SearchMode string `json:"search_mode"`
}
// CoreMemoryProvider is the seam US2 plugs into. When non-nil and
// `opts.IncludeCore` is true, BuildContextPacket calls Get() and
// includes the result verbatim in the packet.
type CoreMemoryProvider interface {
Get(ctx context.Context, ownerID, agentName string) (string, error)
}
// PinProvider returns the owner's pinned message ids. Set on
// InjectionOpts via US3 wiring; when nil the overlay is skipped.
type PinProvider interface {
ListForOwner(ctx context.Context, ownerID string) ([]int64, error)
}
// StatusProvider returns the memory_status of each id in the input
// slice. Ids that do not appear in the returned map are implicitly
// `active`. Set on InjectionOpts via US3 wiring.
type StatusProvider interface {
Statuses(ctx context.Context, msgIDs []int64) (map[int64]MemoryStatusInfo, error)
}
// MemoryStatusInfo mirrors messaging.MemoryStatus without importing
// the messaging package (avoids a cycle). The injection retrieval
// layer needs only the Status string and the active/non-active bit.
type MemoryStatusInfo struct {
Status string
}
// MessageLookup resolves message ids → MemoryItem fields for the pin
// overlay. The overlay needs body / from_agent / channel for pinned
// messages that did NOT come back from the search; loading them
// directly from the messages table keeps this independent of the
// search index.
type MessageLookup interface {
LookupForInjection(ctx context.Context, ids []int64) ([]MemoryItem, error)
}
// InjectionOpts captures the per-call configuration for
// BuildContextPacket. Sourced from messaging.MemoryConfig at wrap time.
type InjectionOpts struct {
// BudgetTokens is the soft cap on the assembled packet. 0 disables
// injection: BuildContextPacket returns (nil, nil).
BudgetTokens int
// MaxItems caps the number of memory items in the packet.
MaxItems int
// MinScore is the relevance floor. Items below are dropped.
MinScore float64
// IncludeCore enables the core-memory lookup. Only session-start
// tools (i.e. my_status) should set this.
IncludeCore bool
// CoreProvider is consulted when IncludeCore is true. May be nil
// (US2 not yet wired) — then no core memory is included.
CoreProvider CoreMemoryProvider
// PinProvider, when non-nil, supplies owner-pinned message ids
// that are spliced into the packet with Score=1.0 regardless of
// the score floor. Status filter still drops soft_deleted /
// superseded pins so retrieval never surfaces tombstoned facts.
PinProvider PinProvider
// StatusProvider, when non-nil, supplies the memory_status of
// each candidate; results with status soft_deleted/superseded are
// dropped (unless pinned).
StatusProvider StatusProvider
// MessageLookup, when non-nil, resolves pinned message ids that
// did not surface through retrieval. When nil, only pins already
// present in the retrieval results are highlighted.
MessageLookup MessageLookup
// RecentWindowDays bounds the recency fallback (FR-009) to the
// last N days of memory-channel activity. 0 → 14d default.
// Mirrored from messaging.MemoryConfig.DreamRecentWindow by the
// MCP wrapper at request time.
RecentWindowDays int
// Now is overridable for tests. Defaults to time.Now.
Now func() time.Time
}
// EstimateTokens returns the char-based token estimate. Matches R4:
// `(chars + 3) / 4`, ceil-equivalent for positive integers.
func EstimateTokens(chars int) int {
if chars <= 0 {
return 0
}
return (chars + 3) / 4
}
// BuildContextPacket retrieves owner-scoped memories matching `query`,
// applies the score floor + token budget + max items cap, optionally
// resolves the per-(owner, agent) core memory blob, and returns a
// ContextPacket ready to merge into the tool response.
//
// Returns (nil, nil) when `opts.BudgetTokens == 0` (feature disabled)
// or when no memories pass the filter AND no core memory is set —
// callers omit the `relevant_context` field entirely in that case.
func BuildContextPacket(
ctx context.Context,
svc *Service,
agent *agents.Agent,
query string,
opts InjectionOpts,
) (*ContextPacket, error) {
if opts.BudgetTokens == 0 {
return nil, nil
}
if agent == nil {
return nil, fmt.Errorf("build context packet: nil agent")
}
if opts.Now == nil {
opts.Now = time.Now
}
maxItems := opts.MaxItems
if maxItems <= 0 {
maxItems = 5
}
callerOwner := strconv.FormatInt(agent.OwnerID, 10)
if agent.OwnerID == 0 {
// Unowned agent: skip retrieval. We still surface a (possibly
// non-empty) core memory if the provider returns one for the
// empty owner — but that's an edge case the provider can decide
// on.
callerOwner = ""
}
// Over-fetch x3 to absorb owner-scope filtering + score floor drop.
wantedLimit := maxItems * 3
if wantedLimit < 15 {
wantedLimit = 15
}
searchMode := ModeAuto
var memories []MemoryItem
if svc != nil && callerOwner != "" {
if query != "" {
// Drive retrieval through the existing hybrid path so we inherit
// access control + ranking. We still need a stricter owner filter
// on top of canAgentAccessMessage because the memory pool
// (open-brain) is broadly readable across agents within the same
// system, and we must enforce owner isolation (SC-008).
resp, err := svc.Search(ctx, agent.Name, SearchOptions{
Query: query,
Mode: ModeAuto,
Limit: wantedLimit,
MinSimilarity: opts.MinScore,
})
if err != nil {
return nil, fmt.Errorf("build context packet: search: %w", err)
}
if resp != nil {
searchMode = resp.SearchMode
}
if resp != nil && len(resp.Results) > 0 {
items, err := filterAndScore(ctx, svc, resp.Results, callerOwner, opts)
if err != nil {
return nil, err
}
memories = items
}
} else {
// FR-009 recency fallback: no explicit query → return the N
// most recent memory-channel messages whose author belongs to
// the caller's owner. Bypasses the hybrid index (which has no
// notion of "no query") and goes directly to SQL. Window is
// bounded by opts.RecentWindowDays (defaults to 14).
windowDays := opts.RecentWindowDays
if windowDays <= 0 {
windowDays = 14
}
items, err := recentMemoriesForOwner(ctx, svc.db, callerOwner, wantedLimit, windowDays)
if err != nil {
return nil, fmt.Errorf("build context packet: recency: %w", err)
}
memories = items
searchMode = "recent"
}
}
// Apply memory_status filter (US3 T031): drop soft_deleted /
// superseded results unless they will be pinned in the next step.
memories = applyStatusFilter(ctx, memories, nil, opts)
// Pin overlay (US3 T029/T031): owner-pinned message ids bypass the
// score floor and are spliced in with Score=1.0 / Pinned=true. We
// build the pin set up-front so the status filter knows to spare
// them.
pinIDs, _ := loadPinIDs(ctx, opts, callerOwner)
if len(pinIDs) > 0 {
memories = applyStatusFilter(ctx, memories, pinIDs, opts)
memories = applyPinOverlay(ctx, memories, pinIDs, opts)
}
// Apply token budget: greedy fill in descending score (results are
// already sorted). Truncate the last admitted item to fit when it
// would otherwise overflow.
memories = applyTokenBudget(memories, maxItems, opts.BudgetTokens)
// Core memory lookup (US2 hook). Provider may be nil — that's fine.
var coreBlob string
if opts.IncludeCore && opts.CoreProvider != nil && callerOwner != "" {
blob, err := opts.CoreProvider.Get(ctx, callerOwner, agent.Name)
if err == nil {
coreBlob = blob
} else if !errors.Is(err, sql.ErrNoRows) {
// Surface unexpected errors so callers can log; an absent
// core blob (typical "no rows") is not an error.
return nil, fmt.Errorf("build context packet: core memory: %w", err)
}
}
if len(memories) == 0 && coreBlob == "" {
// Empty + no core → caller should omit the relevant_context field.
return nil, nil
}
if memories == nil {
memories = []MemoryItem{}
}
packet := &ContextPacket{
Memories: memories,
CoreMemory: coreBlob,
RetrievalQuery: query,
SearchMode: searchMode,
}
packet.PacketChars = packetChars(packet)
packet.PacketTokenEstimate = EstimateTokens(packet.PacketChars)
return packet, nil
}
// filterAndScore drops non-owner messages and items below MinScore,
// then assembles MemoryItem entries in the existing RRF-sorted order.
//
// Owner of each candidate message is resolved via agents.OwnerFor on
// `from_agent`. This is the stricter filter referenced in
// `contracts/mcp-injection.md`'s cross-owner safety note.
func filterAndScore(
ctx context.Context,
svc *Service,
results []*SearchResult,
callerOwnerID string,
opts InjectionOpts,
) ([]MemoryItem, error) {
out := make([]MemoryItem, 0, len(results))
for _, r := range results {
if r == nil || r.Message == nil {
continue
}
// Score selection: prefer SimilarityScore (semantic / hybrid),
// fall back to RelevanceScore (fulltext).
score := r.SimilarityScore
if score == 0 {
score = r.RelevanceScore
}
if opts.MinScore > 0 && score < opts.MinScore {
continue
}
owner, err := agents.OwnerFor(ctx, svc.db, r.Message.FromAgent)
if err != nil {
// Unowned or unknown sender → exclude from injection pool.
continue
}
if owner != callerOwnerID {
continue
}
item := MemoryItem{
ID: r.Message.ID,
FromAgent: r.Message.FromAgent,
Body: r.Message.Body,
CreatedAt: r.Message.CreatedAt,
Score: score,
MatchType: r.MatchType,
}
if r.Message.ChannelID != nil {
if name, err := channelName(ctx, svc.db, *r.Message.ChannelID); err == nil {
item.Channel = name
}
}
out = append(out, item)
}
return out, nil
}
// applyTokenBudget enforces both MaxItems and the token budget. Items
// are admitted greedily in input order (caller passes them already
// score-sorted). The first item that would overflow is truncated to
// fit; all later items are skipped.
func applyTokenBudget(items []MemoryItem, maxItems, budgetTokens int) []MemoryItem {
if len(items) == 0 {
return nil
}
if budgetTokens <= 0 {
return nil
}
out := make([]MemoryItem, 0, len(items))
used := 0
for i, it := range items {
if i >= maxItems {
break
}
cost := EstimateTokens(itemChars(it))
if used+cost <= budgetTokens {
used += cost
out = append(out, it)
continue
}
// Truncate this item to whatever remains in the budget.
remaining := budgetTokens - used
if remaining <= 0 {
break
}
// Reserve the per-item overhead in the remaining budget so that
// post-truncate EstimateTokens(itemChars(it)) <= remaining.
overhead := itemChars(it) - len(it.Body) // = from_agent + channel + 32
// max chars we can place into Body so that total item tokens fit.
maxBodyChars := remaining*4 - overhead
if maxBodyChars <= 0 {
break
}
if maxBodyChars >= len(it.Body) {
// Whole body still fits — admit unchanged.
used += EstimateTokens(itemChars(it))
out = append(out, it)
continue
}
truncated := it
truncated.Body = it.Body[:maxBodyChars]
truncated.Truncated = true
used += EstimateTokens(itemChars(truncated))
out = append(out, truncated)
break
}
return out
}
// itemChars approximates the rendered size of one MemoryItem so the
// budget gate stays self-consistent with packetChars.
func itemChars(it MemoryItem) int {
// Body dominates; from_agent + channel + delimiters add a small
// per-item overhead we approximate at 32 chars.
return len(it.Body) + len(it.FromAgent) + len(it.Channel) + 32
}
func packetChars(p *ContextPacket) int {
total := 0
for _, m := range p.Memories {
total += itemChars(m)
}
total += len(p.CoreMemory)
return total
}
// loadPinIDs queries the configured PinProvider, if any, and returns
// the owner's pinned message ids. Returns nil on any error so that pin
// retrieval failure never breaks the wider injection path.
func loadPinIDs(ctx context.Context, opts InjectionOpts, ownerID string) ([]int64, error) {
if opts.PinProvider == nil || ownerID == "" {
return nil, nil
}
ids, err := opts.PinProvider.ListForOwner(ctx, ownerID)
if err != nil {
return nil, err
}
return ids, nil
}
// applyStatusFilter drops items whose memory_status is soft_deleted or
// superseded. `sparedIDs` is the set of message ids that bypass the
// filter (pinned ids). When opts.StatusProvider is nil this is a no-op.
func applyStatusFilter(ctx context.Context, items []MemoryItem, sparedIDs []int64, opts InjectionOpts) []MemoryItem {
if opts.StatusProvider == nil || len(items) == 0 {
return items
}
ids := make([]int64, 0, len(items))
for _, it := range items {
ids = append(ids, it.ID)
}
statuses, err := opts.StatusProvider.Statuses(ctx, ids)
if err != nil {
return items
}
spared := map[int64]struct{}{}
for _, id := range sparedIDs {
spared[id] = struct{}{}
}
out := make([]MemoryItem, 0, len(items))
for _, it := range items {
st, ok := statuses[it.ID]
if !ok || st.Status == "" || st.Status == "active" {
out = append(out, it)
continue
}
if _, isPinned := spared[it.ID]; isPinned {
out = append(out, it)
continue
}
// Drop soft_deleted / superseded non-pinned.
}
return out
}
// applyPinOverlay marks any item already present and whose id is pinned
// as Pinned=true / Score=1.0; pinned ids that are NOT in the input set
// are fetched via MessageLookup (if configured) and prepended.
func applyPinOverlay(ctx context.Context, items []MemoryItem, pinIDs []int64, opts InjectionOpts) []MemoryItem {
if len(pinIDs) == 0 {
return items
}
pinSet := map[int64]struct{}{}
for _, id := range pinIDs {
pinSet[id] = struct{}{}
}
// Mark items already present.
present := map[int64]struct{}{}
for i := range items {
if _, ok := pinSet[items[i].ID]; ok {
items[i].Pinned = true
items[i].Score = 1.0
}
present[items[i].ID] = struct{}{}
}
// Fetch missing pinned ids via MessageLookup (if any).
var missing []int64
for id := range pinSet {
if _, ok := present[id]; !ok {
missing = append(missing, id)
}
}
if len(missing) > 0 && opts.MessageLookup != nil {
extra, err := opts.MessageLookup.LookupForInjection(ctx, missing)
if err == nil {
// Status filter on the freshly-loaded pinned messages: if
// the provider says they are soft_deleted / superseded, do
// not surface them either, even though pinned.
if opts.StatusProvider != nil {
statuses, _ := opts.StatusProvider.Statuses(ctx, missing)
filtered := extra[:0]
for _, m := range extra {
if st, ok := statuses[m.ID]; ok && st.Status != "" && st.Status != "active" {
continue
}
filtered = append(filtered, m)
}
extra = filtered
}
// Prepend in stable id order (newest first by convention —
// pins are sorted DESC by pinned_at in the store).
for i := range extra {
extra[i].Pinned = true
extra[i].Score = 1.0
extra[i].MatchType = "pinned"
}
items = append(extra, items...)
}
}
return items
}
// recentMemoriesForOwner returns the N most recent messages on memory
// channels (open-brain, reflections-*, or any channel flagged
// is_memory=true) whose author is owned by `ownerID`. Used when the
// injection layer has no explicit retrieval query (e.g. my_status).
//
// Recency is approximated as "ORDER BY messages.id DESC" — id is
// monotonically increasing per SQLite INSERT and matches created_at
// ordering on this schema.
func recentMemoriesForOwner(ctx context.Context, db *sql.DB, ownerID string, limit, windowDays int) ([]MemoryItem, error) {
if db == nil || ownerID == "" || limit <= 0 {
return nil, nil
}
if windowDays <= 0 {
windowDays = 14
}
// SQLite datetime modifier needs the value embedded in the string,
// not bound, so build a literal `-N days`. windowDays is bounded
// (int from int(Duration/(24h))) — safe to format.
windowExpr := fmt.Sprintf("-%d days", windowDays)
const q = `
SELECT m.id, m.from_agent, COALESCE(c.name,''), m.body, m.created_at
FROM messages m
JOIN agents a ON a.name = m.from_agent
LEFT JOIN channels c ON c.id = m.channel_id
WHERE CAST(a.owner_id AS TEXT) = ?
AND m.channel_id IS NOT NULL
AND c.name IN ('open-brain')
AND m.body IS NOT NULL AND m.body != ''
AND m.created_at > datetime('now', ?)
ORDER BY m.id DESC
LIMIT ?`
rows, err := db.QueryContext(ctx, q, ownerID, windowExpr, limit)
if err != nil {
return nil, err
}
defer rows.Close()
var out []MemoryItem
for rows.Next() {
var it MemoryItem
if err := rows.Scan(&it.ID, &it.FromAgent, &it.Channel, &it.Body, &it.CreatedAt); err != nil {
return nil, err
}
it.Score = 1.0
it.MatchType = "recent"
out = append(out, it)
}
return out, rows.Err()
}
// channelName resolves a channel ID to a name for the optional
// `MemoryItem.Channel` field. Best-effort: returns ("", err) on lookup
// failure and BuildContextPacket then omits the field entirely.
func channelName(ctx context.Context, db *sql.DB, id int64) (string, error) {
var name string
err := db.QueryRowContext(ctx,
`SELECT name FROM channels WHERE id = ?`, id,
).Scan(&name)
if err != nil {
return "", err
}
return name, nil
}
+137
View File
@@ -0,0 +1,137 @@
package search
import (
"context"
"testing"
)
// stubPinProvider lets a test return a fixed pin set.
type stubPinProvider struct {
ids []int64
}
func (s *stubPinProvider) ListForOwner(ctx context.Context, ownerID string) ([]int64, error) {
return s.ids, nil
}
// stubMessageLookup loads pinned messages by id.
type stubMessageLookup struct {
byID map[int64]MemoryItem
}
func (s *stubMessageLookup) LookupForInjection(ctx context.Context, ids []int64) ([]MemoryItem, error) {
out := make([]MemoryItem, 0, len(ids))
for _, id := range ids {
if m, ok := s.byID[id]; ok {
out = append(out, m)
}
}
return out, nil
}
// stubStatusProvider returns canned statuses.
type stubStatusProvider struct {
byID map[int64]MemoryStatusInfo
}
func (s *stubStatusProvider) Statuses(ctx context.Context, msgIDs []int64) (map[int64]MemoryStatusInfo, error) {
out := map[int64]MemoryStatusInfo{}
for _, id := range msgIDs {
if v, ok := s.byID[id]; ok {
out[id] = v
}
}
return out, nil
}
func TestBuildContextPacket_PinOverlaySurfacesBelowFloor(t *testing.T) {
svc, _, db := newTestServices(t)
ctx := context.Background()
a := seedOwnedAgent(t, db, 1, "alice", "a1")
// Seed a channel + message so search has something to find for the
// owner; the pin will be a separate id we splice in via lookup.
seedChannel(t, db, 1, "open-brain", "a1")
regularID := seedChannelMessage(t, db, 1, "a1", "some unrelated body")
pinnedID := regularID + 1000
lookup := &stubMessageLookup{
byID: map[int64]MemoryItem{
pinnedID: {ID: pinnedID, FromAgent: "a1", Body: "pinned fact", Score: 0.0},
},
}
pkt, err := BuildContextPacket(ctx, svc, a, "kuzu unrelated query", InjectionOpts{
BudgetTokens: 500,
MaxItems: 5,
MinScore: 0.95, // very high floor → drops everything from search
PinProvider: &stubPinProvider{ids: []int64{pinnedID}},
MessageLookup: lookup,
})
if err != nil {
t.Fatalf("BuildContextPacket: %v", err)
}
if pkt == nil {
t.Fatal("expected non-nil packet (pinned overlay)")
}
sawPinned := false
for _, m := range pkt.Memories {
if m.ID == pinnedID {
sawPinned = true
if !m.Pinned {
t.Errorf("pinned item must be marked Pinned=true: %+v", m)
}
if m.Score < 0.99 {
t.Errorf("pinned item must have Score=1.0, got %v", m.Score)
}
}
}
if !sawPinned {
t.Errorf("pinned message id=%d not surfaced in packet: %#v", pinnedID, pkt.Memories)
}
}
func TestBuildContextPacket_StatusFilterDropsSoftDeleted(t *testing.T) {
svc, _, db := newTestServices(t)
ctx := context.Background()
a := seedOwnedAgent(t, db, 1, "alice", "a1")
seedChannel(t, db, 1, "open-brain", "a1")
keepID := seedChannelMessage(t, db, 1, "a1", "active fact about Kuzu")
dropID := seedChannelMessage(t, db, 1, "a1", "duplicate Kuzu fact")
statusProv := &stubStatusProvider{
byID: map[int64]MemoryStatusInfo{
dropID: {Status: "soft_deleted"},
},
}
pkt, err := BuildContextPacket(ctx, svc, a, "Kuzu", InjectionOpts{
BudgetTokens: 500,
MaxItems: 5,
MinScore: 0.0,
StatusProvider: statusProv,
})
if err != nil {
t.Fatalf("BuildContextPacket: %v", err)
}
if pkt == nil {
t.Fatal("expected non-nil packet")
}
for _, m := range pkt.Memories {
if m.ID == dropID {
t.Errorf("soft-deleted message id=%d should have been dropped", dropID)
}
}
// keep should still be present
sawKeep := false
for _, m := range pkt.Memories {
if m.ID == keepID {
sawKeep = true
}
}
if !sawKeep {
t.Errorf("active message id=%d should be present, packet=%#v", keepID, pkt.Memories)
}
}
+258
View File
@@ -0,0 +1,258 @@
package search
import (
"context"
"database/sql"
"strings"
"testing"
_ "modernc.org/sqlite"
"github.com/synapbus/synapbus/internal/agents"
"github.com/synapbus/synapbus/internal/messaging"
)
// stubCoreProvider lets a test override the core-memory blob returned to
// BuildContextPacket.
type stubCoreProvider struct {
blob string
err error
}
func (s *stubCoreProvider) Get(ctx context.Context, ownerID, agentName string) (string, error) {
return s.blob, s.err
}
// seedOwnedAgent inserts a users row + an agents row tied to that owner.
func seedOwnedAgent(t *testing.T, db *sql.DB, ownerID int64, ownerName, agentName string) *agents.Agent {
t.Helper()
if _, err := db.Exec(
`INSERT OR IGNORE INTO users (id, username, password_hash, display_name)
VALUES (?, ?, 'hash', ?)`, ownerID, ownerName, ownerName,
); err != nil {
t.Fatalf("seed user: %v", err)
}
if _, err := db.Exec(
`INSERT OR IGNORE INTO agents (name, display_name, type, owner_id, api_key_hash, status)
VALUES (?, ?, 'ai', ?, ?, 'active')`,
agentName, agentName, ownerID, agentName+"hash",
); err != nil {
t.Fatalf("seed agent: %v", err)
}
return &agents.Agent{Name: agentName, OwnerID: ownerID}
}
// seedChannel inserts an open-brain channel with members `agentNames`.
func seedChannel(t *testing.T, db *sql.DB, channelID int64, name string, agentNames ...string) {
t.Helper()
if _, err := db.Exec(
`INSERT OR IGNORE INTO channels (id, name, description, type, created_by)
VALUES (?, ?, '', 'standard', 'system')`, channelID, name,
); err != nil {
t.Fatalf("seed channel: %v", err)
}
for _, a := range agentNames {
if _, err := db.Exec(
`INSERT OR IGNORE INTO channel_members (channel_id, agent_name) VALUES (?, ?)`,
channelID, a,
); err != nil {
t.Fatalf("seed channel_member: %v", err)
}
}
}
// seedChannelMessage inserts a message directly so we don't need the
// channels.Service stack in this test.
func seedChannelMessage(t *testing.T, db *sql.DB, channelID int64, fromAgent, body string) int64 {
t.Helper()
convRes, err := db.Exec(
`INSERT INTO conversations (created_by, channel_id) VALUES (?, ?)`,
fromAgent, channelID,
)
if err != nil {
t.Fatalf("seed conversation: %v", err)
}
convID, _ := convRes.LastInsertId()
res, err := db.Exec(
`INSERT INTO messages (conversation_id, from_agent, channel_id, body, priority, status, metadata)
VALUES (?, ?, ?, ?, 5, 'pending', '{}')`,
convID, fromAgent, channelID, body,
)
if err != nil {
t.Fatalf("seed message: %v", err)
}
id, _ := res.LastInsertId()
return id
}
func TestBuildContextPacket_BudgetZeroDisables(t *testing.T) {
svc, _, db := newTestServices(t)
a := seedOwnedAgent(t, db, 1, "alice", "a1")
pkt, err := BuildContextPacket(context.Background(), svc, a, "query", InjectionOpts{
BudgetTokens: 0,
})
if err != nil {
t.Fatalf("BuildContextPacket: %v", err)
}
if pkt != nil {
t.Errorf("BudgetTokens=0 should return nil packet, got %+v", pkt)
}
}
func TestBuildContextPacket_OwnerScoping(t *testing.T) {
svc, _, db := newTestServices(t)
ctx := context.Background()
h1Agent := seedOwnedAgent(t, db, 1, "alice", "a1")
_ = seedOwnedAgent(t, db, 2, "bob", "b1")
seedChannel(t, db, 1, "open-brain", "a1", "b1")
seedChannelMessage(t, db, 1, "a1", "Kuzu graph DB archived 2025")
seedChannelMessage(t, db, 1, "b1", "Kuzu graph DB looks promising")
pkt, err := BuildContextPacket(ctx, svc, h1Agent, "Kuzu", InjectionOpts{
BudgetTokens: 500,
MaxItems: 5,
MinScore: 0.0,
})
if err != nil {
t.Fatalf("BuildContextPacket: %v", err)
}
if pkt == nil {
t.Fatal("expected non-nil packet")
}
for _, m := range pkt.Memories {
if m.FromAgent != "a1" {
t.Errorf("leaked memory from %q (owner != caller)", m.FromAgent)
}
if !strings.Contains(m.Body, "Kuzu") {
t.Errorf("unexpected body: %q", m.Body)
}
}
}
func TestBuildContextPacket_TokenBudgetGreedyFillAndTruncate(t *testing.T) {
items := []MemoryItem{
{ID: 1, Body: strings.Repeat("a", 200), Score: 0.9},
{ID: 2, Body: strings.Repeat("b", 200), Score: 0.8},
{ID: 3, Body: strings.Repeat("c", 200), Score: 0.7},
}
// Budget 100 tokens => 400 chars total. Each item costs ~232 chars
// → ~58 tokens. First fits cleanly; second must truncate.
out := applyTokenBudget(items, 5, 100)
if len(out) == 0 {
t.Fatal("expected at least one admitted item")
}
if out[0].ID != 1 {
t.Errorf("first admitted item ID = %d, want 1", out[0].ID)
}
total := 0
for _, it := range out {
total += EstimateTokens(itemChars(it))
}
if total > 100 {
t.Errorf("total tokens admitted = %d, exceeds budget 100", total)
}
sawTruncated := false
for _, it := range out {
if it.Truncated {
sawTruncated = true
}
}
if len(out) > 1 && !sawTruncated {
t.Errorf("expected truncation when admitting a second item under tight budget")
}
}
func TestBuildContextPacket_ScoreFloorDrops(t *testing.T) {
svc, _, db := newTestServices(t)
ctx := context.Background()
_ = seedOwnedAgent(t, db, 1, "alice", "a1")
seedChannel(t, db, 1, "open-brain", "a1")
cidPtr := int64(1)
low := &SearchResult{
Message: &messaging.Message{
ID: 100,
FromAgent: "a1",
Body: "low score body",
ChannelID: &cidPtr,
},
SimilarityScore: 0.1,
MatchType: ModeSemantic,
}
high := &SearchResult{
Message: &messaging.Message{
ID: 101,
FromAgent: "a1",
Body: "high score body",
ChannelID: &cidPtr,
},
SimilarityScore: 0.8,
MatchType: ModeSemantic,
}
items, err := filterAndScore(ctx, svc, []*SearchResult{low, high}, "1", InjectionOpts{
MinScore: 0.5,
})
if err != nil {
t.Fatalf("filterAndScore: %v", err)
}
if len(items) != 1 {
t.Fatalf("expected 1 admitted item, got %d", len(items))
}
if items[0].ID != 101 {
t.Errorf("admitted ID = %d, want 101 (high score)", items[0].ID)
}
}
func TestBuildContextPacket_CoreMemoryIncluded(t *testing.T) {
svc, _, db := newTestServices(t)
ctx := context.Background()
a := seedOwnedAgent(t, db, 1, "alice", "a1")
provider := &stubCoreProvider{blob: "I am alice's research agent."}
pkt, err := BuildContextPacket(ctx, svc, a, "", InjectionOpts{
BudgetTokens: 500,
MaxItems: 5,
IncludeCore: true,
CoreProvider: provider,
})
if err != nil {
t.Fatalf("BuildContextPacket: %v", err)
}
if pkt == nil {
t.Fatal("expected non-nil packet when core memory is present")
}
if pkt.CoreMemory != provider.blob {
t.Errorf("CoreMemory = %q, want %q", pkt.CoreMemory, provider.blob)
}
if pkt.PacketChars < len(provider.blob) {
t.Errorf("PacketChars = %d, want >= %d", pkt.PacketChars, len(provider.blob))
}
if pkt.PacketTokenEstimate != EstimateTokens(pkt.PacketChars) {
t.Errorf("PacketTokenEstimate inconsistent with PacketChars")
}
}
func TestBuildContextPacket_EmptyReturnsNil(t *testing.T) {
svc, _, db := newTestServices(t)
ctx := context.Background()
a := seedOwnedAgent(t, db, 1, "alice", "a1")
// No memories, no core provider → should return (nil, nil) so the
// wrapper omits the relevant_context field entirely.
pkt, err := BuildContextPacket(ctx, svc, a, "", InjectionOpts{
BudgetTokens: 500,
MaxItems: 5,
})
if err != nil {
t.Fatalf("BuildContextPacket: %v", err)
}
if pkt != nil {
t.Errorf("expected nil packet, got %+v", pkt)
}
}
+126
View File
@@ -0,0 +1,126 @@
package storage
import (
"context"
"database/sql"
"testing"
"time"
)
// TestMemoryStatusView verifies the view's status / superseded_by /
// soft_deleted_at derivations against crafted memory_consolidation_jobs
// rows. See data-model.md §`memory_status` view.
func TestMemoryStatusView(t *testing.T) {
db := newTestDB(t)
ctx := context.Background()
if err := RunMigrations(ctx, db); err != nil {
t.Fatalf("RunMigrations: %v", err)
}
finishedAt := time.Date(2026, 5, 11, 3, 0, 0, 0, time.UTC).Format("2006-01-02 15:04:05")
// Job 1: completed mark_duplicate where keep_id=100, target=101 (loser).
// → message 101 must be soft_deleted; 100 stays active.
actionDup := `[{
"tool":"memory_mark_duplicate",
"target_message_id":101,
"args":{"a_id":100,"b_id":101,"keep_id":100,"reason":"shorter paraphrase"},
"at":"2026-05-11T03:00:00Z"
}]`
if _, err := db.Exec(
`INSERT INTO memory_consolidation_jobs
(owner_id, job_type, status, trigger_reason, actions, finished_at)
VALUES (?, ?, ?, ?, ?, ?)`,
"1", "dedup_contradiction", "succeeded", "manual:test", actionDup, finishedAt,
); err != nil {
t.Fatalf("insert dup job: %v", err)
}
// Job 2: memory_supersede target=200, by=201. → 200 superseded.
actionSup := `[{
"tool":"memory_supersede",
"target_message_id":200,
"args":{"a_id":200,"b_id":201,"reason":"newer fact"},
"at":"2026-05-11T03:00:00Z"
}]`
if _, err := db.Exec(
`INSERT INTO memory_consolidation_jobs
(owner_id, job_type, status, trigger_reason, actions, finished_at)
VALUES (?, ?, ?, ?, ?, ?)`,
"1", "dedup_contradiction", "succeeded", "manual:test", actionSup, finishedAt,
); err != nil {
t.Fatalf("insert supersede job: %v", err)
}
// Failed job — must NOT appear in the view.
if _, err := db.Exec(
`INSERT INTO memory_consolidation_jobs
(owner_id, job_type, status, trigger_reason, actions, finished_at)
VALUES (?, ?, ?, ?, ?, ?)`,
"1", "reflection", "failed", "manual:test",
`[{"tool":"memory_supersede","target_message_id":300,"args":{"b_id":301}}]`,
finishedAt,
); err != nil {
t.Fatalf("insert failed job: %v", err)
}
statuses, supersededBy, deletedAt := readStatuses(t, db, []int64{100, 101, 200, 201, 300})
if got := statuses[100]; got != "" {
// 100 has no action row → not in view → empty string from the
// helper's zero-value default.
t.Errorf("100: want active/missing, got %q", got)
}
if got := statuses[101]; got != "soft_deleted" {
t.Errorf("101: want soft_deleted, got %q", got)
}
if got := deletedAt[101]; got == "" {
t.Errorf("101: expected non-empty soft_deleted_at")
}
if got := statuses[200]; got != "superseded" {
t.Errorf("200: want superseded, got %q", got)
}
if got := supersededBy[200]; got != 201 {
t.Errorf("200.superseded_by: want 201, got %d", got)
}
// 300 was on a failed job → should not appear in view.
if got := statuses[300]; got != "" {
t.Errorf("300: want missing (failed job), got %q", got)
}
}
func readStatuses(t *testing.T, db *sql.DB, ids []int64) (map[int64]string, map[int64]int64, map[int64]string) {
t.Helper()
statuses := map[int64]string{}
supersededBy := map[int64]int64{}
deletedAt := map[int64]string{}
rows, err := db.Query(
`SELECT message_id, status, COALESCE(superseded_by, 0), COALESCE(soft_deleted_at, '')
FROM memory_status`,
)
if err != nil {
t.Fatalf("query view: %v", err)
}
defer rows.Close()
for rows.Next() {
var (
id int64
status string
sup int64
deletedTs string
)
if err := rows.Scan(&id, &status, &sup, &deletedTs); err != nil {
t.Fatalf("scan view: %v", err)
}
statuses[id] = status
supersededBy[id] = sup
deletedAt[id] = deletedTs
}
if err := rows.Err(); err != nil {
t.Fatalf("rows.Err: %v", err)
}
_ = ids
return statuses, supersededBy, deletedAt
}
+160
View File
@@ -194,3 +194,163 @@ func TestRunMigrations_EmbeddedSchema(t *testing.T) {
}
}
}
// TestMigration028_MemoryConsolidation verifies the proactive-memory + dream
// worker migration applies cleanly and all six tables plus the memory_status
// view are queryable. Doubles as a basic INSERT/SELECT round-trip for each
// new table.
func TestMigration028_MemoryConsolidation(t *testing.T) {
db := newTestDB(t)
ctx := context.Background()
if err := RunMigrations(ctx, db); err != nil {
t.Fatalf("RunMigrations: %v", err)
}
// 1. Verify all six tables exist.
wantTables := []string{
"memory_core",
"memory_links",
"memory_consolidation_jobs",
"memory_pins",
"memory_dispatch_tokens",
"memory_injections",
}
for _, table := range wantTables {
var name string
err := db.QueryRow(
"SELECT name FROM sqlite_master WHERE type='table' AND name=?", table,
).Scan(&name)
if err != nil {
t.Errorf("table %s not found: %v", table, err)
}
}
// 2. Verify memory_status view exists.
var viewName string
if err := db.QueryRow(
"SELECT name FROM sqlite_master WHERE type='view' AND name=?", "memory_status",
).Scan(&viewName); err != nil {
t.Errorf("memory_status view not found: %v", err)
}
// 3. INSERT/SELECT round-trip for each new table.
// memory_core
if _, err := db.ExecContext(ctx,
`INSERT INTO memory_core (owner_id, agent_name, blob, updated_by)
VALUES (?, ?, ?, ?)`,
"1", "agent-a", "core blob", "human:1",
); err != nil {
t.Fatalf("insert memory_core: %v", err)
}
var blob string
if err := db.QueryRowContext(ctx,
`SELECT blob FROM memory_core WHERE owner_id=? AND agent_name=?`, "1", "agent-a",
).Scan(&blob); err != nil {
t.Fatalf("select memory_core: %v", err)
}
if blob != "core blob" {
t.Errorf("memory_core blob = %q, want %q", blob, "core blob")
}
// memory_consolidation_jobs (needed for FK on dispatch tokens)
res, err := db.ExecContext(ctx,
`INSERT INTO memory_consolidation_jobs (owner_id, job_type, trigger_reason)
VALUES (?, ?, ?)`,
"1", "reflection", "watermark:25",
)
if err != nil {
t.Fatalf("insert memory_consolidation_jobs: %v", err)
}
jobID, _ := res.LastInsertId()
var status string
if err := db.QueryRowContext(ctx,
`SELECT status FROM memory_consolidation_jobs WHERE id=?`, jobID,
).Scan(&status); err != nil {
t.Fatalf("select memory_consolidation_jobs: %v", err)
}
if status != "pending" {
t.Errorf("default status = %q, want pending", status)
}
// memory_links
if _, err := db.ExecContext(ctx,
`INSERT INTO memory_links (src_message_id, dst_message_id, relation_type, owner_id, created_by)
VALUES (?, ?, ?, ?, ?)`,
1, 2, "refines", "1", "auto:test",
); err != nil {
t.Fatalf("insert memory_links: %v", err)
}
var relType string
if err := db.QueryRowContext(ctx,
`SELECT relation_type FROM memory_links WHERE src_message_id=? AND dst_message_id=?`, 1, 2,
).Scan(&relType); err != nil {
t.Fatalf("select memory_links: %v", err)
}
if relType != "refines" {
t.Errorf("relation_type = %q, want refines", relType)
}
// memory_pins
if _, err := db.ExecContext(ctx,
`INSERT INTO memory_pins (owner_id, message_id, pinned_by, note)
VALUES (?, ?, ?, ?)`,
"1", 42, "human:1", "important",
); err != nil {
t.Fatalf("insert memory_pins: %v", err)
}
var note string
if err := db.QueryRowContext(ctx,
`SELECT note FROM memory_pins WHERE owner_id=? AND message_id=?`, "1", 42,
).Scan(&note); err != nil {
t.Fatalf("select memory_pins: %v", err)
}
if note != "important" {
t.Errorf("note = %q, want important", note)
}
// memory_dispatch_tokens
if _, err := db.ExecContext(ctx,
`INSERT INTO memory_dispatch_tokens (token, owner_id, consolidation_job_id, expires_at)
VALUES (?, ?, ?, datetime('now', '+15 minutes'))`,
"tok-abc", "1", jobID,
); err != nil {
t.Fatalf("insert memory_dispatch_tokens: %v", err)
}
var ownerID string
if err := db.QueryRowContext(ctx,
`SELECT owner_id FROM memory_dispatch_tokens WHERE token=?`, "tok-abc",
).Scan(&ownerID); err != nil {
t.Fatalf("select memory_dispatch_tokens: %v", err)
}
if ownerID != "1" {
t.Errorf("owner_id = %q, want 1", ownerID)
}
// memory_injections
if _, err := db.ExecContext(ctx,
`INSERT INTO memory_injections (owner_id, agent_name, tool_name, packet_size_chars, packet_items_count, message_ids)
VALUES (?, ?, ?, ?, ?, ?)`,
"1", "agent-a", "my_status", 120, 3, "[1,2,3]",
); err != nil {
t.Fatalf("insert memory_injections: %v", err)
}
var items int
if err := db.QueryRowContext(ctx,
`SELECT packet_items_count FROM memory_injections WHERE owner_id=?`, "1",
).Scan(&items); err != nil {
t.Fatalf("select memory_injections: %v", err)
}
if items != 3 {
t.Errorf("packet_items_count = %d, want 3", items)
}
// memory_status view: query should succeed (empty rows are fine since
// the job we inserted is still 'pending').
rows, err := db.QueryContext(ctx, `SELECT message_id, owner_id, status FROM memory_status`)
if err != nil {
t.Fatalf("select memory_status: %v", err)
}
rows.Close()
}
@@ -0,0 +1,61 @@
-- 027_remove_approval_noise.sql — one-shot cleanup for internal-only mode.
-- Drops messages produced by the (now-deleted) stalemate reminder/escalation
-- paths and any pending agent proposals. The agent_proposals table itself
-- stays in case the propose_agent flow is ever reinstated.
--
-- The #approvals channel row is intentionally NOT dropped: leaving it
-- keeps the migration trivially reversible (no need to recreate the
-- channel and re-grant memberships); the human can delete it via admin
-- CLI later.
--
-- Reminder/escalation messages are identified by their metadata JSON
-- (see internal/messaging/stalemate.go in git history), which is the
-- only field the historical worker stamped reliably — the conversation
-- subject prefix was used too but is not a hard guarantee.
--
-- FK NOTES: most refs to messages(id) are ON DELETE CASCADE / SET NULL,
-- but two are NO ACTION and would block this migration on a populated
-- DB:
-- * attachments.message_id (001_initial.sql)
-- * messages.reply_to (007_threads.sql)
-- We NULL those explicitly before deleting. Orphan attachment rows (with
-- message_id = NULL) are tolerated by the attachments service.
-- 1. Collect the message ids we're about to drop into a temp scratch
-- table so the cascading NULL/DELETE statements all reference the
-- same set even if the metadata predicates evolve later.
CREATE TEMP TABLE _approval_noise_msgs AS
SELECT id FROM messages
WHERE metadata LIKE '%"stalemate_reminder_for":%'
OR metadata LIKE '%"stalemate_escalation_for":%'
OR metadata LIKE '%"workflow_stalemate_reminder_for":%'
OR metadata LIKE '%"workflow_stalemate_escalation_for":%'
OR channel_id IN (SELECT id FROM channels WHERE name = 'approvals');
-- 2. Detach NO-ACTION FKs that would otherwise block the delete.
UPDATE attachments
SET message_id = NULL
WHERE message_id IN (SELECT id FROM _approval_noise_msgs);
UPDATE messages
SET reply_to = NULL
WHERE reply_to IN (SELECT id FROM _approval_noise_msgs);
-- 3. Drop the messages themselves. Cascading FKs (reactions, embeddings,
-- fts triggers, agent_listings) clean up automatically; SET NULL FKs
-- (goals.*_message_id, agent_proposals.*_message_id, goal_tasks.*)
-- forget the link gracefully.
DELETE FROM messages
WHERE id IN (SELECT id FROM _approval_noise_msgs);
DROP TABLE _approval_noise_msgs;
-- Conversation rows for stalemate-prefixed subjects are intentionally
-- left in place. Multiple NO-ACTION FKs reference conversations(id)
-- (messages.conversation_id, inbox_state.conversation_id, possibly
-- others added by future migrations); cleaning them up reliably would
-- require chasing every ref. Empty / mostly-empty conversation rows
-- are harmless — they don't render as messages in the UI.
-- 4. Drop pending agent proposals. The table stays for reversibility.
DELETE FROM agent_proposals;
@@ -0,0 +1,153 @@
-- 028_memory_consolidation.sql — proactive memory + dream worker.
--
-- Adds six tables backing per-(owner, agent) core memory, typed memory
-- links, dream-worker consolidation jobs, owner-pinned memories,
-- single-use dispatch tokens, and a 24h rolling audit ring of what was
-- injected into each tool response. Adds one derived view
-- `memory_status` over `memory_consolidation_jobs.actions` so retrieval
-- can filter out soft-deleted / superseded memories without touching
-- the hot `messages` table.
--
-- `owner_id` is stored as TEXT (string form of `agents.owner_id`) so
-- that retrieval can carry the value through unchanged from
-- `auth.ContextAgent(ctx)`; the auth middleware already exposes it as a
-- string via `trace.ContextWithOwnerID`.
--
-- All `CREATE TABLE` / `CREATE INDEX` statements use `IF NOT EXISTS`
-- so this migration is safe to re-run. CHECK constraints enforce enum
-- columns. The partial unique index on
-- `memory_consolidation_jobs(owner_id, job_type)` guarantees at most
-- one in-flight job per (owner, job_type).
-- 1. Per-(owner, agent) identity-and-context blob.
CREATE TABLE IF NOT EXISTS memory_core (
owner_id TEXT NOT NULL,
agent_name TEXT NOT NULL,
blob TEXT NOT NULL,
updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_by TEXT NOT NULL,
PRIMARY KEY (owner_id, agent_name)
);
CREATE INDEX IF NOT EXISTS idx_memory_core_owner ON memory_core(owner_id);
-- 2. Directed typed edges between two message ids.
CREATE TABLE IF NOT EXISTS memory_links (
id INTEGER PRIMARY KEY AUTOINCREMENT,
src_message_id INTEGER NOT NULL,
dst_message_id INTEGER NOT NULL,
relation_type TEXT NOT NULL CHECK (relation_type IN (
'refines', 'contradicts', 'examples', 'related',
'duplicate_of', 'superseded_by',
'mention', 'reply_to', 'channel_cooccurrence'
)),
owner_id TEXT NOT NULL,
created_by TEXT NOT NULL,
metadata TEXT NOT NULL DEFAULT '{}',
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
UNIQUE(src_message_id, dst_message_id, relation_type)
);
CREATE INDEX IF NOT EXISTS idx_memory_links_owner ON memory_links(owner_id);
CREATE INDEX IF NOT EXISTS idx_memory_links_src ON memory_links(src_message_id);
CREATE INDEX IF NOT EXISTS idx_memory_links_dst ON memory_links(dst_message_id);
CREATE INDEX IF NOT EXISTS idx_memory_links_type ON memory_links(relation_type);
-- 3. Audit log of dispatched dream jobs and their resulting actions.
CREATE TABLE IF NOT EXISTS memory_consolidation_jobs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
owner_id TEXT NOT NULL,
job_type TEXT NOT NULL CHECK (job_type IN (
'reflection', 'core_rewrite', 'dedup_contradiction', 'link_gen'
)),
status TEXT NOT NULL CHECK (status IN (
'pending', 'dispatched', 'running', 'succeeded', 'partial', 'failed', 'expired'
)) DEFAULT 'pending',
trigger_reason TEXT NOT NULL,
dispatch_token TEXT,
harness_run_id TEXT,
actions TEXT NOT NULL DEFAULT '[]',
summary TEXT,
error TEXT,
lease_until DATETIME,
started_at DATETIME,
finished_at DATETIME,
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_consolidation_owner_status
ON memory_consolidation_jobs(owner_id, status);
CREATE INDEX IF NOT EXISTS idx_consolidation_lease
ON memory_consolidation_jobs(lease_until)
WHERE status = 'running';
CREATE UNIQUE INDEX IF NOT EXISTS idx_consolidation_in_flight
ON memory_consolidation_jobs(owner_id, job_type)
WHERE status IN ('pending', 'dispatched', 'running');
-- 4. Owner-pinned message ids that bypass the relevance floor.
CREATE TABLE IF NOT EXISTS memory_pins (
owner_id TEXT NOT NULL,
message_id INTEGER NOT NULL,
pinned_by TEXT NOT NULL,
note TEXT,
pinned_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (owner_id, message_id)
);
CREATE INDEX IF NOT EXISTS idx_memory_pins_owner ON memory_pins(owner_id);
-- 5. Single-use, owner-bound, job-bound dispatch tokens.
CREATE TABLE IF NOT EXISTS memory_dispatch_tokens (
token TEXT PRIMARY KEY,
owner_id TEXT NOT NULL,
consolidation_job_id INTEGER NOT NULL,
issued_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
expires_at DATETIME NOT NULL,
used_at DATETIME,
revoked_at DATETIME,
FOREIGN KEY (consolidation_job_id) REFERENCES memory_consolidation_jobs(id)
);
CREATE INDEX IF NOT EXISTS idx_memory_tokens_job
ON memory_dispatch_tokens(consolidation_job_id);
-- 6. 24-hour rolling audit ring of what was injected for each tool call.
CREATE TABLE IF NOT EXISTS memory_injections (
id INTEGER PRIMARY KEY AUTOINCREMENT,
owner_id TEXT NOT NULL,
agent_name TEXT NOT NULL,
tool_name TEXT NOT NULL,
packet_size_chars INTEGER NOT NULL,
packet_items_count INTEGER NOT NULL,
message_ids TEXT NOT NULL,
core_blob_included BOOLEAN NOT NULL DEFAULT 0,
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_memory_injections_owner_time
ON memory_injections(owner_id, created_at);
-- 7. Derived view: memory_status. Tells retrieval whether a message is
-- active / soft_deleted / superseded based on completed consolidation
-- jobs' action records. Owners can override via a future restore
-- action (out of scope here).
CREATE VIEW IF NOT EXISTS memory_status AS
WITH actions AS (
SELECT
j.owner_id,
json_extract(act.value, '$.target_message_id') AS message_id,
json_extract(act.value, '$.tool') AS tool,
json_extract(act.value, '$.args.keep_id') AS keep_id,
json_extract(act.value, '$.args.b_id') AS superseded_by,
json_extract(act.value, '$.args.reason') AS reason,
j.finished_at AS at
FROM memory_consolidation_jobs j, json_each(j.actions) act
WHERE j.status IN ('succeeded', 'partial')
)
SELECT
message_id,
owner_id,
CASE
WHEN MAX(CASE WHEN tool='memory_supersede' THEN at END) IS NOT NULL THEN 'superseded'
WHEN MAX(CASE WHEN tool='memory_mark_duplicate' AND keep_id != message_id THEN at END) IS NOT NULL THEN 'soft_deleted'
ELSE 'active'
END AS status,
MAX(CASE WHEN tool='memory_supersede' THEN superseded_by END) AS superseded_by,
MAX(CASE WHEN tool='memory_mark_duplicate' AND keep_id != message_id THEN at END) AS soft_deleted_at,
MAX(reason) AS reason
FROM actions
GROUP BY message_id, owner_id;
@@ -0,0 +1,20 @@
-- 029_memory_dream_usage.sql — per-(date, owner) circuit-breaker counters
-- for the dream worker (feature 020 follow-up). Each row accumulates
-- daily token and job consumption so the ConsolidatorWorker can refuse
-- to dispatch further jobs once a threshold is hit. Rows are keyed by
-- UTC date (YYYY-MM-DD); the circuit naturally resets at midnight UTC
-- when Today() begins reading from a fresh row.
CREATE TABLE IF NOT EXISTS memory_dream_usage (
date TEXT NOT NULL, -- YYYY-MM-DD UTC
owner_id TEXT NOT NULL,
tokens_in INTEGER NOT NULL DEFAULT 0,
tokens_out INTEGER NOT NULL DEFAULT 0,
jobs_started INTEGER NOT NULL DEFAULT 0,
jobs_succeeded INTEGER NOT NULL DEFAULT 0,
jobs_failed INTEGER NOT NULL DEFAULT 0,
jobs_circuit_broken INTEGER NOT NULL DEFAULT 0,
updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (date, owner_id)
);
CREATE INDEX IF NOT EXISTS idx_memory_dream_usage_date ON memory_dream_usage(date);
@@ -0,0 +1,15 @@
-- 030_dream_parallelism.sql
--
-- Allow concurrent dream-worker dispatches of the same (owner, job_type)
-- by adding a `slot` discriminator. Slot 0 is the historical behaviour
-- (one in-flight per type). Slots 1..N-1 are used when the worker / CLI
-- fans out to drain a backlog quickly.
ALTER TABLE memory_consolidation_jobs
ADD COLUMN slot INTEGER NOT NULL DEFAULT 0;
DROP INDEX IF EXISTS idx_consolidation_in_flight;
CREATE UNIQUE INDEX IF NOT EXISTS idx_consolidation_in_flight
ON memory_consolidation_jobs(owner_id, job_type, slot)
WHERE status IN ('pending', 'dispatched', 'running');
@@ -0,0 +1,21 @@
-- 031: Composite index to speed up ExpireTasks() in expiry-worker.
--
-- Root cause: `ExpireTasks` runs
-- UPDATE tasks SET status='cancelled', updated_at=CURRENT_TIMESTAMP
-- WHERE status='open' AND deadline IS NOT NULL AND deadline < ?
--
-- Pre-031 indexes were only `idx_tasks_status(status)` and `idx_tasks_channel(channel_id)`.
-- With a status cardinality of 4 and most tasks in two buckets, the planner used
-- `idx_tasks_status` to find all open rows then evaluated the deadline predicate
-- per row. As the auction-tasks table grew (kubic deploy), the worker's 30s
-- context deadline started to be exceeded on every tick, especially under WAL
-- write contention from concurrent message inserts.
--
-- Fix: composite, partial index on `(status, deadline)` covering only rows that
-- can ever be expired (status='open' AND deadline IS NOT NULL). This is the
-- exact predicate the worker uses, so SQLite can seek straight to the eligible
-- rows. The partial form keeps the index tiny once tasks transition out of
-- 'open' (the dominant steady-state).
CREATE INDEX IF NOT EXISTS idx_tasks_expiry
ON tasks(status, deadline)
WHERE status = 'open' AND deadline IS NOT NULL;
+40
View File
@@ -0,0 +1,40 @@
#!/usr/bin/env bash
# Build a linux/amd64 SynapBus image, ship it to the kubic node, import it
# into microk8s containerd, and roll the synapbus deployment to that image.
#
# Usage: scripts/deploy-kubic.sh <version>
# e.g. scripts/deploy-kubic.sh v0.17.0
set -euo pipefail
VERSION="${1:?usage: $0 <version>}"
IMAGE_TAG="${VERSION}-amd64"
IMAGE="docker.io/library/synapbus:${IMAGE_TAG}"
KUBIC_HOST="${KUBIC_HOST:-user@kubic.home.arpa}"
NAMESPACE="${NAMESPACE:-synapbus}"
DEPLOYMENT="${DEPLOYMENT:-synapbus}"
TARBALL="/tmp/synapbus-${IMAGE_TAG}.tar"
echo "==> building ${IMAGE} (linux/amd64)"
docker buildx build \
--platform linux/amd64 \
--build-arg "VERSION=${VERSION}" \
-t "synapbus:${IMAGE_TAG}" \
--load .
echo "==> exporting to ${TARBALL}"
docker save "synapbus:${IMAGE_TAG}" -o "${TARBALL}"
echo "==> shipping to ${KUBIC_HOST}"
scp "${TARBALL}" "${KUBIC_HOST}:${TARBALL}"
echo "==> importing into microk8s containerd"
ssh "${KUBIC_HOST}" "sudo microk8s ctr image import ${TARBALL} && rm -f ${TARBALL}"
rm -f "${TARBALL}"
echo "==> rolling deployment ${NAMESPACE}/${DEPLOYMENT} to ${IMAGE}"
kubectl set image -n "${NAMESPACE}" "deploy/${DEPLOYMENT}" "${DEPLOYMENT}=${IMAGE}"
kubectl rollout status -n "${NAMESPACE}" "deploy/${DEPLOYMENT}" --timeout=180s
echo "==> done. /healthz:"
curl -sf -o /dev/null -w 'status=%{http_code}\n' "http://kubic.home.arpa:30088/healthz" || true
@@ -0,0 +1,37 @@
# Specification Quality Checklist: Proactive Memory & Dream Worker
**Purpose**: Validate specification completeness and quality before proceeding to planning
**Created**: 2026-05-11
**Feature**: [spec.md](../spec.md)
## Content Quality
- [x] No implementation details (languages, frameworks, APIs)
- [x] Focused on user value and business needs
- [x] Written for non-technical stakeholders
- [x] All mandatory sections completed
## Requirement Completeness
- [x] No [NEEDS CLARIFICATION] markers remain
- [x] Requirements are testable and unambiguous
- [x] Success criteria are measurable
- [x] Success criteria are technology-agnostic (no implementation details)
- [x] All acceptance scenarios are defined
- [x] Edge cases are identified
- [x] Scope is clearly bounded
- [x] Dependencies and assumptions identified
## Feature Readiness
- [x] All functional requirements have clear acceptance criteria
- [x] User scenarios cover primary flows
- [x] Feature meets measurable outcomes defined in Success Criteria
- [x] No implementation details leak into specification
## Notes
- The spec deliberately omits the file paths, table names, and env-var names that surfaced during brainstorming. Those belong in `plan.md`.
- Four user stories prioritized P1/P2/P2/P3. P1 (injection) is the MVP and independently shippable. P3 (audit UI) gates enabling the dream worker on real owner data.
- Nine measurable success criteria, all technology-agnostic.
- The "Assumptions" section bakes in defaults that were discussed during brainstorming so no clarification questions are needed.
@@ -0,0 +1,78 @@
# Contract — MCP Tool Response Injection
## Wrapper shape
For every injection-eligible MCP tool, the response JSON body gains an optional `relevant_context` field appended at the top level:
```json
{
"<existing tool result fields>": "...",
"relevant_context": {
"memories": [
{
"id": 12345,
"from_agent": "research-mcpproxy",
"channel": "open-brain",
"body": "KuzuDB archived 2025-10-10; not viable for synapbus",
"created_at": "2026-05-08T14:22:00Z",
"score": 0.91,
"match_type": "hybrid",
"pinned": false,
"truncated": false
}
],
"core_memory": "<text>", // only on session-start tools, may be omitted
"packet_chars": 412,
"packet_token_estimate": 103,
"retrieval_query": "Kuzu graph DB",
"search_mode": "auto"
}
}
```
### Field semantics
- `memories` — up to N items (default 5, env `SYNAPBUS_INJECTION_MAX_ITEMS`), ranked by RRF score. Body is verbatim from the message; `truncated=true` only when a single item had to be cut to fit the token budget.
- `core_memory` — Letta-style per-(owner, agent) blob. Present only on session-start-class tools (`my_status` today; extensible). Omitted entirely when no blob is set.
- `packet_chars` — total character count of the assembled packet (memories + core + delimiters).
- `packet_token_estimate` — `packet_chars/4` rounded up.
- `retrieval_query` — the text that drove retrieval (the tool's argument body, or "<recent activity>" fallback). Useful for "why did my agent know this?" debug.
- `search_mode` — `"auto"` / `"semantic"` / `"fulltext"` per `search.Service`.
## Injection-eligible tools
| Tool | Retrieval query source | Include core memory? |
|------|------------------------|----------------------|
| `my_status` | `<recent activity>` | ✅ |
| `claim_messages` | concatenated bodies of claimed messages | ❌ |
| `read_inbox` | concatenated bodies of returned messages | ❌ |
| `send_message` | body of the sent message | ❌ |
| `search` / `search_messages` | the user's query | ❌ |
| `execute` | the request payload (stringified args) | ❌ |
| `read_channel` | channel topic + last N message bodies | ❌ |
Tools NOT injected: `list_resources`, `propose_agent`, `propose_task_tree`, `complete_goal`, `create_goal`, `claim_task`, `get_replies`, `request_resource`, `react` (write-only or metadata).
## Configuration
| Env var | Default | Description |
|---------|---------|-------------|
| `SYNAPBUS_INJECTION_ENABLED` | `0` (off) | Master switch. Off → no `relevant_context` field on any response (FR-012, SC-009). |
| `SYNAPBUS_INJECTION_BUDGET_TOKENS` | `500` | Soft cap. Greedy fill in descending score; truncate last admitted item if needed. |
| `SYNAPBUS_INJECTION_MAX_ITEMS` | `5` | Hard cap on `memories[]` length. |
| `SYNAPBUS_INJECTION_MIN_SCORE` | inherits `search.DefaultMinSimilarity` (0.25) | Floor — pinned memories bypass this. |
| `SYNAPBUS_CORE_MEMORY_MAX_BYTES` | `2048` | Reject `memory_rewrite_core` over this size. |
## Empty / low-signal paths
- If `len(memories) == 0` and no core memory is set → omit the `relevant_context` field entirely. Response shape exactly matches pre-feature.
- If `len(memories) == 0` but core memory IS set → include the field with `memories: []` and `core_memory: "..."`.
- If a tool's response is not JSON-shaped (e.g. binary attachment download), pass through unchanged.
## Cross-owner safety (SC-008)
Retrieval is filtered by `caller.OwnerID` at the SQL level inside `search.Service.Search()`. The injection middleware never inspects or trusts the response body for owner info — it always re-derives owner from `auth.ContextAgent(ctx)`. An adversarial test asserts that an agent owned by H1 making *any* injection-eligible tool call cannot have a memory whose source message was authored by an agent owned by H2 appear in `relevant_context.memories`.
## Audit (FR-025)
Every assembled non-empty packet writes one row to `memory_injections` with `(owner_id, agent_name, tool_name, packet_size_chars, packet_items_count, message_ids[], core_blob_included, created_at)`. Rows older than 24h are purged hourly by the consolidator worker.
@@ -0,0 +1,226 @@
# Contract — MCP Memory-Consolidation Tools
These six tools are registered only when `SYNAPBUS_DREAM_ENABLED=1`. They reject any caller that does not present a valid `SYNAPBUS_DISPATCH_TOKEN` (header `X-Synapbus-Dispatch-Token` or env propagated by the harness). Token validation is described in `research.md` R7.
All six tools accept an implicit `dispatch_token` from the request context and explicit `owner_id` (asserted to match the token's owner; otherwise error `dispatch_token_owner_mismatch`).
---
## 1. `memory_list_unprocessed`
List recent memory-eligible messages the owner's pool has not yet consolidated.
**Input**:
```json
{
"owner_id": "algis",
"since_message_id": 0, // optional — exclusive lower bound
"limit": 50 // optional — default 50, max 200
}
```
**Output**:
```json
{
"memories": [
{
"id": 12345,
"from_agent": "research-mcpproxy",
"channel": "open-brain",
"body": "...",
"created_at": "2026-05-08T14:22:00Z",
"links": [{"to": 12000, "type": "mention"}]
}
],
"max_id_returned": 12567 // pass as since_message_id on next call
}
```
**Behavior**: Returns active memories from memory channels for the owner, ordered by id ascending, excluding any already linked via `refines` / `duplicate_of` / `superseded_by` to a more recent memory.
---
## 2. `memory_write_reflection`
Write a higher-level abstraction back to the memory pool as a new message in `#open-brain` (or a `#reflections-<owner>` channel if one exists), tagged.
**Input**:
```json
{
"owner_id": "algis",
"body": "Across recent discussions, the team has committed to ...",
"source_message_ids": [12001, 12030, 12089],
"tags": ["reflection", "weekly"]
}
```
**Output**:
```json
{
"memory_id": 12601,
"channel": "open-brain",
"links_created": 3 // one 'refines' link per source
}
```
**Behavior**:
1. Inserts a message authored by a synthetic `dream:<owner>` agent into the memory channel (preferring `#reflections-<owner>` if it exists, else `#open-brain`).
2. Inserts a `refines` link from the new memory to each source.
3. Embeds the new message via the existing async pipeline.
4. Records `{tool, args, target_message_id: new_id}` in the parent job's `actions` JSON.
**Errors**: `source_not_found` if any source ID is missing or belongs to a different owner.
---
## 3. `memory_rewrite_core`
Replace the per-(owner, agent) core memory blob wholesale.
**Input**:
```json
{
"owner_id": "algis",
"agent_name": "research-mcpproxy",
"blob": "You are research-mcpproxy. Currently focused on benchmarking ..."
}
```
**Output**:
```json
{
"owner_id": "algis",
"agent_name": "research-mcpproxy",
"previous_blob": "...",
"new_blob_chars": 412,
"updated_at": "2026-05-11T03:00:14Z"
}
```
**Errors**:
- `core_memory_too_large` if `len(blob) > SYNAPBUS_CORE_MEMORY_MAX_BYTES`.
- `agent_not_owned` if the target agent's `owner_id != caller.owner_id`.
- `agent_protected` if the agent has `metadata.protected_core=true`.
---
## 4. `memory_mark_duplicate`
Mark two memories as duplicates; one is kept canonical, the other is soft-deleted.
**Input**:
```json
{
"owner_id": "algis",
"a_id": 12001,
"b_id": 12089,
"keep_id": 12001,
"reason": "Same fact about KuzuDB archival, b is shorter paraphrase"
}
```
**Output**:
```json
{
"keep_id": 12001,
"soft_deleted_id": 12089,
"link_created_id": 4501
}
```
**Behavior**:
1. Inserts a `duplicate_of` link from `loser_id → keep_id`.
2. Appends `{tool: "memory_mark_duplicate", target_message_id: loser_id, args: {keep_id, ...}}` to the job's `actions` JSON.
3. The `memory_status` view then derives `loser_id` as `soft_deleted`.
**Errors**: `not_same_owner`, `keep_id_not_in_pair`, `already_duplicate`.
---
## 5. `memory_supersede`
Mark memory A as obsoleted by memory B (temporal validity).
**Input**:
```json
{
"owner_id": "algis",
"a_id": 12001,
"b_id": 12500,
"reason": "Fact updated: as of 2026-04, kubernetes deploy moved off helm chart"
}
```
**Output**:
```json
{
"superseded_id": 12001,
"by_id": 12500,
"link_created_id": 4502
}
```
**Behavior**: Inserts `superseded_by` link from `a → b`. View derives `a.status = 'superseded'`, `a.superseded_by = b`. Reason is preserved in the action JSON.
**Errors**: `not_same_owner`, `cycle_detected` (b transitively superseded by a).
---
## 6. `memory_add_link`
Add a typed link between two memories (A-MEM Zettelkasten style).
**Input**:
```json
{
"owner_id": "algis",
"src_id": 12030,
"dst_id": 12085,
"relation_type": "refines",
"metadata": {"confidence": 0.84}
}
```
**Output**:
```json
{
"link_id": 4510
}
```
**Constraints**:
- `relation_type` ∈ {`refines`, `contradicts`, `examples`, `related`}. The reserved auto-types (`mention`, `reply_to`, `channel_cooccurrence`) and consolidation-types (`duplicate_of`, `superseded_by`) are written by other tools / jobs and rejected here.
**Errors**: `relation_type_reserved`, `not_same_owner`, `link_already_exists`.
---
## Common error codes
| Code | Meaning |
|------|---------|
| `dispatch_token_missing` | No token in request context |
| `dispatch_token_expired` | Token past `expires_at` |
| `dispatch_token_revoked` | Token explicitly revoked |
| `dispatch_token_owner_mismatch` | Request's `owner_id` differs from token's |
| `dispatch_token_wrong_job` | Token bound to a different `consolidation_job_id` than active call sequence |
| `not_same_owner` | A referenced message belongs to a different owner |
| `source_not_found` | Referenced message id does not exist |
| `core_memory_too_large` | Blob exceeds size cap |
All errors follow MCP's standard `{"error": {"code": "...", "message": "..."}}` shape.
## Audit interaction
Every successful invocation of any of the six tools appends an entry to the parent `memory_consolidation_jobs.actions` JSON array:
```json
{
"tool": "memory_mark_duplicate",
"args": {"a_id": 12001, "b_id": 12089, "keep_id": 12001, "reason": "..."},
"target_message_id": 12089,
"at": "2026-05-11T03:00:14Z"
}
```
On job completion (success / partial / failed), the worker updates `status`, `summary`, `finished_at`. The token's `used_at` is set on first call; the token remains usable for the rest of the same job.
@@ -0,0 +1,210 @@
# Data Model — 020-proactive-memory-dream-worker
## Reused tables (unchanged)
- **`messages`** — every "memory" is a message on a memory-flagged channel. No schema change.
- **`channels`** — `metadata` JSON gains an optional `"is_memory": true` flag. No schema change.
- **`agents`** — `owner_id` is the canonical scope. No schema change.
- **`embeddings`** — existing per-message embeddings used unchanged by `search.Service`.
## New tables (migration `028_memory_consolidation.sql`)
### `memory_core`
Per-(owner, agent_name) identity-and-context blob.
```sql
CREATE TABLE memory_core (
owner_id TEXT NOT NULL,
agent_name TEXT NOT NULL,
blob TEXT NOT NULL,
updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_by TEXT NOT NULL, -- 'human:<owner>' or 'agent:<name>:<token>'
PRIMARY KEY (owner_id, agent_name)
);
CREATE INDEX idx_memory_core_owner ON memory_core(owner_id);
```
Constraints:
- `LENGTH(blob) <= SYNAPBUS_CORE_MEMORY_MAX_BYTES` (default 2048) — enforced in Go, not SQL.
- Replaces wholesale on update — no diff/merge.
### `memory_links`
Directed typed edges between two message IDs.
```sql
CREATE TABLE memory_links (
id INTEGER PRIMARY KEY AUTOINCREMENT,
src_message_id INTEGER NOT NULL,
dst_message_id INTEGER NOT NULL,
relation_type TEXT NOT NULL CHECK (relation_type IN (
'refines', 'contradicts', 'examples', 'related',
'duplicate_of', 'superseded_by',
'mention', 'reply_to', 'channel_cooccurrence'
)),
owner_id TEXT NOT NULL, -- denormalized for fast owner-scoped queries
created_by TEXT NOT NULL, -- 'human:<owner>' / 'agent:<name>:<token>' / 'auto:<rule>'
metadata TEXT NOT NULL DEFAULT '{}',
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
UNIQUE(src_message_id, dst_message_id, relation_type)
);
CREATE INDEX idx_memory_links_owner ON memory_links(owner_id);
CREATE INDEX idx_memory_links_src ON memory_links(src_message_id);
CREATE INDEX idx_memory_links_dst ON memory_links(dst_message_id);
CREATE INDEX idx_memory_links_type ON memory_links(relation_type);
```
Relation types:
- LLM-generated: `refines`, `contradicts`, `examples`, `related`, `duplicate_of`, `superseded_by`
- Auto-generated by messaging layer: `mention`, `reply_to`, `channel_cooccurrence`
### `memory_consolidation_jobs`
Audit log of dispatched dream jobs and their resulting actions.
```sql
CREATE TABLE memory_consolidation_jobs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
owner_id TEXT NOT NULL,
job_type TEXT NOT NULL CHECK (job_type IN (
'reflection', 'core_rewrite', 'dedup_contradiction', 'link_gen'
)),
status TEXT NOT NULL CHECK (status IN (
'pending', 'dispatched', 'running', 'succeeded', 'partial', 'failed', 'expired'
)) DEFAULT 'pending',
trigger_reason TEXT NOT NULL, -- 'watermark:N', 'cron:nightly', 'manual:<owner>'
dispatch_token TEXT, -- FK-ish into memory_dispatch_tokens.token
harness_run_id TEXT, -- FK into harness_runs.id (existing)
actions TEXT NOT NULL DEFAULT '[]', -- JSON array of {tool, args, before, after}
summary TEXT, -- human-readable summary written on completion
error TEXT,
lease_until DATETIME, -- in-flight lease; null when not running
started_at DATETIME,
finished_at DATETIME,
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX idx_consolidation_owner_status ON memory_consolidation_jobs(owner_id, status);
CREATE INDEX idx_consolidation_lease ON memory_consolidation_jobs(lease_until)
WHERE status = 'running';
CREATE UNIQUE INDEX idx_consolidation_in_flight
ON memory_consolidation_jobs(owner_id, job_type)
WHERE status IN ('pending', 'dispatched', 'running');
```
The partial-unique index enforces: at most one in-flight job per `(owner, job_type)`.
### `memory_pins`
Owner-pinned message IDs that bypass the relevance floor.
```sql
CREATE TABLE memory_pins (
owner_id TEXT NOT NULL,
message_id INTEGER NOT NULL,
pinned_by TEXT NOT NULL, -- 'human:<owner>'
note TEXT, -- why this is pinned (free-form)
pinned_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (owner_id, message_id)
);
CREATE INDEX idx_memory_pins_owner ON memory_pins(owner_id);
```
### `memory_dispatch_tokens`
Single-use, owner-bound, job-bound tokens.
```sql
CREATE TABLE memory_dispatch_tokens (
token TEXT PRIMARY KEY, -- 32-byte random, base64url
owner_id TEXT NOT NULL,
consolidation_job_id INTEGER NOT NULL,
issued_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
expires_at DATETIME NOT NULL, -- issued_at + 15m
used_at DATETIME,
revoked_at DATETIME,
FOREIGN KEY (consolidation_job_id) REFERENCES memory_consolidation_jobs(id)
);
CREATE INDEX idx_memory_tokens_job ON memory_dispatch_tokens(consolidation_job_id);
```
Token is valid when `revoked_at IS NULL AND expires_at > now() AND consolidation_job_id == claimed job`. `used_at` becomes informational once first set; subsequent calls within the same job are allowed.
### `memory_injections`
24-hour rolling ring of what was injected for each tool call.
```sql
CREATE TABLE memory_injections (
id INTEGER PRIMARY KEY AUTOINCREMENT,
owner_id TEXT NOT NULL,
agent_name TEXT NOT NULL,
tool_name TEXT NOT NULL,
packet_size_chars INTEGER NOT NULL,
packet_items_count INTEGER NOT NULL,
message_ids TEXT NOT NULL, -- JSON array of message_ids included
core_blob_included BOOLEAN NOT NULL DEFAULT 0,
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX idx_memory_injections_owner_time ON memory_injections(owner_id, created_at);
```
A daily cleanup query (run by the consolidator worker) deletes `created_at < now() - 24h`.
## View: `memory_status`
Derived from `memory_consolidation_jobs.actions`. Tells retrieval whether a message is active / soft-deleted / superseded.
```sql
CREATE VIEW memory_status AS
WITH actions AS (
SELECT
owner_id,
json_extract(act.value, '$.target_message_id') AS message_id,
json_extract(act.value, '$.tool') AS tool,
json_extract(act.value, '$.args.keep_id') AS keep_id,
json_extract(act.value, '$.args.b_id') AS superseded_by,
json_extract(act.value, '$.args.reason') AS reason,
finished_at AS at
FROM memory_consolidation_jobs j, json_each(j.actions) act
WHERE j.status IN ('succeeded', 'partial')
)
SELECT
message_id,
owner_id,
CASE
WHEN MAX(CASE WHEN tool='memory_supersede' THEN at END) IS NOT NULL THEN 'superseded'
WHEN MAX(CASE WHEN tool='memory_mark_duplicate' AND keep_id != message_id THEN at END) IS NOT NULL THEN 'soft_deleted'
ELSE 'active'
END AS status,
MAX(CASE WHEN tool='memory_supersede' THEN superseded_by END) AS superseded_by,
MAX(CASE WHEN tool='memory_mark_duplicate' AND keep_id != message_id THEN at END) AS soft_deleted_at,
MAX(reason) AS reason
FROM actions
GROUP BY message_id, owner_id;
```
Retrieval excludes `status != 'active'` unless `pinned` or `include_inactive=true`.
## State transitions
```text
Memory message (active by default — not in memory_status)
│
├── memory_mark_duplicate(keep != self) → soft_deleted (via view)
│ └── owner restore action overrides → active
│
├── memory_supersede(b) → superseded (via view)
│ └── owner restore → active
│
└── (no consolidation event) → active
```
Pinning and protection are orthogonal flags (in `memory_pins`; protection is currently piggybacked on a `protected_until` JSON field in `memory_pins.note` — promoted to a column in Phase 2 if needed).
## Indexes & query patterns
Hot queries:
1. **Injection retrieval**: `search.Service.Search()` already paginates → join `memory_status` to drop non-active → drop in pin overlay → apply token budget.
2. **Dream worker watermark check**: `SELECT COUNT(*) FROM messages m JOIN agents a ON m.from_agent=a.name WHERE a.owner_id=? AND m.channel_id IN (memory_channels) AND m.id > last_reflection_max_id`. Owner-filtered + range — uses existing message indexes.
3. **Audit lookup**: `SELECT * FROM memory_consolidation_jobs WHERE owner_id=? ORDER BY created_at DESC LIMIT 50`. Covered by `idx_consolidation_owner_status`.
@@ -0,0 +1,136 @@
# Implementation Plan: Proactive Memory & Dream Worker
**Branch**: `020-proactive-memory-dream-worker` | **Date**: 2026-05-11 | **Spec**: [spec.md](./spec.md)
**Input**: Feature specification from `/specs/020-proactive-memory-dream-worker/spec.md`
## Summary
Make SynapBus push minimal owner-scoped memory into every agent task automatically (Story 1, P1), give each agent a small editable identity blob always-included in session-start (Story 2, P2), and run a background "dream" worker that periodically dispatches consolidation work to a Claude-Code agent through the existing `harness.Harness` seam (Story 3, P2). An owner-facing audit/override UI lands later (Story 4, P3). All work reuses existing primitives: `search.Service` for retrieval, `harness.Harness` for dispatch, `messages` table for the memory pool, the `StalemateWorker` ticker pattern for the new `ConsolidatorWorker`. One new SQLite migration adds six tables. Zero CGO, zero new external dependencies.
## Technical Context
**Language/Version**: Go 1.25+ (per `go.mod`)
**Primary Dependencies**: `mark3labs/mcp-go` (MCP tools), `go-chi/chi` (HTTP), `modernc.org/sqlite` (storage), `TFMV/hnsw` (vectors via existing `search.Service`), existing `internal/harness` package (dispatch seam). **No new external dependencies.**
**Storage**: SQLite via `modernc.org/sqlite` — one new migration `028_memory_consolidation.sql`. Memory pool reuses the existing `messages` table on memory-flagged channels.
**Testing**: `go test ./...` table-driven tests, mocked `harness.Harness` for dream-worker tests, in-memory SQLite for storage tests
**Target Platform**: `linux/amd64` (kubic deployment), `darwin/arm64` (dev). Pure-Go cross-compilation required.
**Project Type**: Single-binary Go service with embedded Svelte SPA (existing layout).
**Performance Goals**: Median injection overhead < 50ms per tool call (SC-002). Full owner-level consolidation pass ≤ 10 min for ≤ 5,000 memories (SC-005).
**Constraints**: Zero CGO (Constitution III). Single binary (Constitution I). Injection MUST be disable-able with zero payload-shape change when disabled (FR-012, SC-009).
**Scale/Scope**: One owner with ~500 memories today; design for 5k/owner, 10 owners on the reference deployment.
## Constitution Check
*Gate: must pass before Phase 0 research. Re-evaluated after Phase 1 design.*
| Principle | Check | Status |
|-----------|-------|--------|
| I. Local-first, single binary | New worker ships in the same binary; no external service introduced. | ✅ |
| II. MCP-native | New memory-consolidation tools registered via `mark3labs/mcp-go`. REST endpoints are added only for the Web UI audit tab (Phase 2). | ✅ |
| III. Pure Go, zero CGO | No new dependencies; reuses `modernc.org/sqlite` + `TFMV/hnsw`. | ✅ |
| IV. Multi-tenant w/ ownership | Owner scoping is the *core* of the design. Retrieval filters by `owner_id`; dispatch tokens are owner-bound. | ✅ |
| V. Embedded OAuth 2.1 | Unchanged; consolidation agents authenticate via the existing API-key path. | ✅ |
| VI. Semantic-ready storage | Reuses `search.Service.Search()`; gracefully degrades to FTS when no embedding provider configured. | ✅ |
| VII. Swarm intelligence | `#open-brain` is a stigmergic blackboard already. Reflection writes annotated memories back to the blackboard. | ✅ |
| VIII. Observable by default | Every consolidation mutation hits an immutable `memory_consolidation_jobs` audit row + a `memory_audit_log` row keyed by dispatch token. Every injection hits `memory_injections` (24h ring). | ✅ |
| IX. Progressive complexity | Feature is gated by `SYNAPBUS_INJECTION_ENABLED` and `SYNAPBUS_DREAM_ENABLED`. Default off until owner audit UI lands. | ✅ |
| X. Web UI first-class | Memory tab is in scope but deferred to Phase 2 of this feature (post-kubic-deploy). Audit data is captured from day one so the UI can render history retroactively. | ⚠ deferred — tracked, not skipped |
**Gate result**: PASS. No violations require Complexity Tracking.
## Project Structure
### Documentation (this feature)
```text
specs/020-proactive-memory-dream-worker/
├── plan.md # This file
├── spec.md # Feature spec
├── research.md # Phase 0 — technical-unknowns resolution
├── data-model.md # Phase 1 — entities and tables
├── contracts/
│ ├── mcp-injection.md # Shape of relevant_context block on tool responses
│ └── mcp-memory-tools.md # 6 new dream-agent tools (input schemas, behavior, errors)
├── quickstart.md # Phase 1 — how to verify locally and on kubic
├── tasks.md # (Phase 2 — produced by /speckit.tasks)
└── checklists/
└── requirements.md # Spec quality checklist
```
### Source Code (repository root, existing layout extended)
```text
cmd/synapbus/main.go # MODIFIED: wire up ConsolidatorWorker, gate new MCP tools
internal/
├── messaging/
│ ├── consolidator.go # NEW: ConsolidatorWorker (ticker + dispatch via harness)
│ ├── consolidator_test.go # NEW
│ ├── memory.go # NEW: core-memory CRUD, link CRUD, supersession, pin/protected
│ ├── memory_test.go # NEW
│ ├── memory_channels.go # NEW: discover memory-flagged channels per owner
│ └── stalemate.go # UNCHANGED — pattern template
├── search/
│ └── injection.go # NEW: BuildContextPacket(ctx, agent, opts) → ContextPacket
├── mcp/
│ ├── injection_wrap.go # NEW: middleware appending relevant_context to tool results
│ ├── memory_tools.go # NEW: 6 dream-agent tools (memory_list_unprocessed,
│ │ # memory_write_reflection, memory_rewrite_core,
│ │ # memory_mark_duplicate, memory_supersede, memory_add_link)
│ ├── memory_tools_test.go # NEW
│ ├── tools_hybrid.go # MODIFIED: wrap eligible tool handlers via injection_wrap
│ └── server.go # MODIFIED: register memory tools when dream worker enabled
├── storage/schema/
│ └── 028_memory_consolidation.sql # NEW: 6 tables (see data-model.md)
└── harness/ # UNCHANGED — used via existing Execute() API
docs/superpowers/specs/2026-05-11-internal-only-disable-approvals-design.md # pre-existing, untouched
```
**Structure Decision**: Single-project Go layout (matches existing `cmd/synapbus` + `internal/*` structure). Memory pool reuses `messages`; six new tables in one new migration are the only schema addition. No new top-level package — new files split between `internal/messaging` (storage-adjacent), `internal/search` (retrieval-adjacent), and `internal/mcp` (tool-surface-adjacent).
## Phase 0 — Research
See [research.md](./research.md). Resolved unknowns:
1. **How does the dream worker invoke a Claude Code agent without using a system DM?** → through `harness.Harness.Execute(ExecRequest)` with the existing `kubernetes_job` or `local_subprocess` backend. The agent's MCP session carries a one-time `dispatch_token` injected via `ExecRequest.Env` that authorizes the consolidation tools.
2. **Where do owners live in the data model today?** → `agents.owner_id` (text) is the canonical scope. Every retrieval JOINs `agents` to resolve the caller's owner.
3. **Soft-delete on messages — new column or metadata flag?** → metadata flag on `memory_consolidation_jobs.actions` + a derived view. Avoids touching the hot `messages` table. Soft-delete is enforced at retrieval time by joining against `memory_status` view.
4. **Token-budget estimator** → `len(s)/4` heuristic. Faster than tokenizing, accurate enough for budget enforcement.
5. **MCP transport — how do we identify the calling agent's API key/owner?** → existing `auth.ContextAgent(ctx)` returns `*agents.Agent` from the request context; `agent.OwnerID` is the scope key.
## Phase 1 — Design & Contracts
### Data model
See [data-model.md](./data-model.md). Migration `028_memory_consolidation.sql` adds six tables:
- `memory_core` — per-(owner, agent_name) editable text blob, size-capped at `SYNAPBUS_CORE_MEMORY_MAX_BYTES` (default 2048).
- `memory_links` — directed typed edges between two message IDs.
- `memory_consolidation_jobs` — audit log of every dispatched dream job + the actions it took.
- `memory_pins` — owner-pinned message IDs, always-included in injection.
- `memory_dispatch_tokens` — one-time, owner-bound, single-job-bound tokens.
- `memory_injections` — 24h rolling ring of (tool_name, agent_id, packet_summary) for debug-ability.
Plus one view `memory_status` (active / soft-deleted / superseded — derived from `memory_consolidation_jobs.actions`).
### Contracts
- [contracts/mcp-injection.md](./contracts/mcp-injection.md) — shape of `relevant_context` block appended to tool responses.
- [contracts/mcp-memory-tools.md](./contracts/mcp-memory-tools.md) — input/output schemas for the six new memory-consolidation tools.
### Quickstart
See [quickstart.md](./quickstart.md) — covers local dev (build → migrate → seed → run with `SYNAPBUS_INJECTION_ENABLED=1 SYNAPBUS_DREAM_ENABLED=1`) and kubic deployment (build image, push, kustomize apply, tail dream-worker logs, dispatch a manual reflection job, verify audit log).
### Agent context update
Triggered by the workflow: `update-agent-context.sh claude` adds the new packages and migration to `CLAUDE.md` "Active Technologies" block.
## Phase 2 — Tasks
(Produced by `/speckit.tasks`. Not created by this command.)
## Complexity Tracking
No constitution violations. Memory-tab Web UI is deferred (tracked as Phase 2 of this feature) but is not a constitution violation since audit data is captured from day one and surfaces in CLI/REST until the SPA tab lands.
@@ -0,0 +1,162 @@
# Quickstart — 020-proactive-memory-dream-worker
How to build, run, and verify the proactive-memory + dream-worker feature locally and on the kubic deployment.
## Local: build & test
```bash
cd ~/repos/synapbus
make test # full test suite — should be green
go build -o ./synapbus ./cmd/synapbus
```
## Local: run with feature flags on
```bash
export SYNAPBUS_DATA_DIR=/tmp/synapbus-dream
export SYNAPBUS_INJECTION_ENABLED=1
export SYNAPBUS_DREAM_ENABLED=1
export SYNAPBUS_EMBEDDING_PROVIDER=openai
export OPENAI_API_KEY=<key>
mkdir -p $SYNAPBUS_DATA_DIR
./synapbus serve --port 8080 --data $SYNAPBUS_DATA_DIR
```
## Verify Story 1 (injection)
```bash
# Create an agent and an owner
./synapbus --socket $SYNAPBUS_DATA_DIR/synapbus.sock agents create \
--name dogfood-1 --owner algis --description "smoke test agent"
# Get an API key for the agent
KEY=$(./synapbus --socket $SYNAPBUS_DATA_DIR/synapbus.sock apikeys create \
--agent dogfood-1 --json | jq -r .key)
# Seed memory channel with a few messages
./synapbus --socket $SYNAPBUS_DATA_DIR/synapbus.sock channels create \
--name open-brain --type blackboard
./synapbus --socket $SYNAPBUS_DATA_DIR/synapbus.sock messages send \
--agent dogfood-1 --channel open-brain \
--body "KuzuDB was archived 2025-10-10. Not viable for SynapBus."
# Call my_status via MCP — relevant_context should appear
curl -s -X POST http://localhost:8080/mcp \
-H "Authorization: Bearer $KEY" \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"my_status","arguments":{}}}' \
| jq '.result.content[0].text | fromjson | .relevant_context'
```
Expected: `relevant_context.memories` is non-empty, contains the seeded message, `core_memory` is null/omitted (none set yet).
Set a core memory and re-run:
```bash
# Direct DB or admin CLI:
sqlite3 $SYNAPBUS_DATA_DIR/synapbus.db \
"INSERT INTO memory_core(owner_id, agent_name, blob, updated_by)
VALUES('algis', 'dogfood-1', 'You are a dogfood test agent.', 'human:algis');"
# Re-call my_status — relevant_context.core_memory should now be populated
```
## Verify Story 3 (dream worker manual dispatch)
```bash
# Force a reflection job for owner=algis
./synapbus --socket $SYNAPBUS_DATA_DIR/synapbus.sock memory dream-run \
--owner algis --job reflection
# Tail logs for the worker
# Look for:
# component=consolidator-worker job=reflection owner=algis status=dispatched
# component=consolidator-worker job=reflection owner=algis status=succeeded
sqlite3 $SYNAPBUS_DATA_DIR/synapbus.db \
"SELECT id, job_type, status, summary, finished_at
FROM memory_consolidation_jobs ORDER BY id DESC LIMIT 5;"
```
Expected: one new row in `memory_consolidation_jobs` with `status=succeeded` (or `partial` if seed pool too small). For a reflection job, expect 0–5 new `body LIKE 'REFLECTION:%'` messages in `#open-brain`.
## Verify token isolation (SC-008)
```bash
# Create a second owner + agent
./synapbus --socket ... agents create --name attacker-1 --owner mallory ...
KEY2=$(./synapbus --socket ... apikeys create --agent attacker-1 --json | jq -r .key)
# Attacker calls search — must NOT see algis's memories
curl -s -X POST http://localhost:8080/mcp -H "Authorization: Bearer $KEY2" ... \
| jq '.result.content[0].text | fromjson | .relevant_context.memories'
```
Expected: empty array. If anything from algis's pool surfaces, that is a Constitution IV violation.
## kubic: build, push, deploy
```bash
# Build linux/amd64 from darwin/arm64 — no CGO so this just works
cd ~/repos/synapbus
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o ./synapbus-linux-amd64 ./cmd/synapbus
# Build container (existing Dockerfile)
docker build --platform=linux/amd64 -t synapbus:020-proactive-memory .
# Push to the cluster's local registry (kubic uses microk8s registry on :32000)
docker tag synapbus:020-proactive-memory kubic.home.arpa:32000/synapbus:020-proactive-memory
docker push kubic.home.arpa:32000/synapbus:020-proactive-memory
# Update the deployment image
kubectl -n synapbus set image deploy/synapbus \
synapbus=kubic.home.arpa:32000/synapbus:020-proactive-memory
# Enable feature flags via env (kubectl set env, or edit values in deploy/kubic-manifests/)
kubectl -n synapbus set env deploy/synapbus \
SYNAPBUS_INJECTION_ENABLED=1 SYNAPBUS_DREAM_ENABLED=1
kubectl -n synapbus rollout status deploy/synapbus
```
## kubic: verify dream agent works
```bash
# Tail logs filtered to the new worker
kubectl -n synapbus logs deploy/synapbus -f | grep -E "consolidator-worker|memory-injection"
```
Expected log lines on startup:
```
{"component":"consolidator-worker","msg":"consolidator worker started","interval":"1h","deep_cron":"0 3 * * *"}
```
Expected after first watermark / cron tick:
```
{"component":"consolidator-worker","msg":"trigger fired","owner":"algis","job":"reflection","trigger":"watermark:20"}
{"component":"consolidator-worker","msg":"dispatched","owner":"algis","job":"reflection","harness_run_id":"...","dispatch_token":"redacted"}
{"component":"consolidator-worker","msg":"job completed","owner":"algis","job":"reflection","status":"succeeded","actions":3,"duration_ms":42184}
```
Force a job to test without waiting:
```bash
kubectl exec -n synapbus deploy/synapbus -- /synapbus --socket /data/synapbus.sock \
memory dream-run --owner algis --job reflection
```
Inspect the audit log:
```bash
kubectl exec -n synapbus deploy/synapbus -- sqlite3 /data/synapbus.db \
"SELECT id, job_type, status, json_array_length(actions) AS n_actions, summary
FROM memory_consolidation_jobs ORDER BY id DESC LIMIT 10;"
```
## Rollback
```bash
kubectl -n synapbus set env deploy/synapbus SYNAPBUS_INJECTION_ENABLED- SYNAPBUS_DREAM_ENABLED-
# Or roll back the image:
kubectl -n synapbus rollout undo deploy/synapbus
```
The migration `028_memory_consolidation.sql` is additive — rolling the binary back leaves the new tables in place but unused. Safe.

Some files were not shown because too many files have changed in this diff Show More