specs(020): proactive memory injection + dream worker
Owner-scoped memory the system pushes onto MCP tool responses, plus a background "dream" worker that dispatches consolidation jobs to a Claude Code agent through the existing harness seam. Memory pool reuses the messages table on memory-flagged channels; six new SQLite tables for core blob, links, audit, pins, dispatch tokens, and a 24h injection ring. Zero CGO, no new external deps. Includes: spec.md (4 user stories), plan.md, research.md (10 decisions), data-model.md, contracts/ (injection shape + 6 memory tools), quickstart.md, tasks.md (47 tasks across foundational + 3 stories + polish, with parallel-subagent cluster plan). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Algis Dumbris
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
a3d8681715
commit
84973ad665
@@ -114,6 +114,8 @@ make lint # Run linters
|
||||
- SQLite via `modernc.org/sqlite` — five new migrations (`021_goals_tasks.sql`, `022_agent_proposals.sql`, `023_agent_trust_model.sql`, `024_secrets.sql`, `025_harness_runs_task_id.sql`); existing content-addressable attachment store reused for encrypted secret blobs (018-dynamic-agent-spawning)
|
||||
- Go 1.25+ (per go.mod) + `mark3labs/mcp-go` (MCP), `go-chi/chi` (HTTP), `spf13/cobra` (CLI), `modernc.org/sqlite` (storage), `jmoiron/sqlx` (query helpers), `cloudflare/tableflip` (graceful restart — NEW), `gopkg.in/yaml.v3` (config), `xeipuuv/gojsonschema` (config-schema validation) (019-plugin-system)
|
||||
- SQLite via `modernc.org/sqlite` (pure Go, zero CGO). New core table `plugin_migrations`. Plugin tables namespaced `plugin_<name>_*`. (019-plugin-system)
|
||||
- Go 1.25+ (per `go.mod`) + `mark3labs/mcp-go` (MCP tools), `go-chi/chi` (HTTP), `modernc.org/sqlite` (storage), `TFMV/hnsw` (vectors via existing `search.Service`), existing `internal/harness` package (dispatch seam). **No new external dependencies.** (020-proactive-memory-dream-worker)
|
||||
- SQLite via `modernc.org/sqlite` — one new migration `028_memory_consolidation.sql`. Memory pool reuses the existing `messages` table on memory-flagged channels. (020-proactive-memory-dream-worker)
|
||||
|
||||
## Recent Changes
|
||||
- 002-mcp-auth-ux-polish: Added Go 1.23+ + ory/fosite (OAuth 2.1), mark3labs/mcp-go (MCP server), go-chi/chi (HTTP), Svelte 5 + Tailwind (Web UI)
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
# Specification Quality Checklist: Proactive Memory & Dream Worker
|
||||
|
||||
**Purpose**: Validate specification completeness and quality before proceeding to planning
|
||||
**Created**: 2026-05-11
|
||||
**Feature**: [spec.md](../spec.md)
|
||||
|
||||
## Content Quality
|
||||
|
||||
- [x] No implementation details (languages, frameworks, APIs)
|
||||
- [x] Focused on user value and business needs
|
||||
- [x] Written for non-technical stakeholders
|
||||
- [x] All mandatory sections completed
|
||||
|
||||
## Requirement Completeness
|
||||
|
||||
- [x] No [NEEDS CLARIFICATION] markers remain
|
||||
- [x] Requirements are testable and unambiguous
|
||||
- [x] Success criteria are measurable
|
||||
- [x] Success criteria are technology-agnostic (no implementation details)
|
||||
- [x] All acceptance scenarios are defined
|
||||
- [x] Edge cases are identified
|
||||
- [x] Scope is clearly bounded
|
||||
- [x] Dependencies and assumptions identified
|
||||
|
||||
## Feature Readiness
|
||||
|
||||
- [x] All functional requirements have clear acceptance criteria
|
||||
- [x] User scenarios cover primary flows
|
||||
- [x] Feature meets measurable outcomes defined in Success Criteria
|
||||
- [x] No implementation details leak into specification
|
||||
|
||||
## Notes
|
||||
|
||||
- The spec deliberately omits the file paths, table names, and env-var names that surfaced during brainstorming. Those belong in `plan.md`.
|
||||
- Four user stories prioritized P1/P2/P2/P3. P1 (injection) is the MVP and independently shippable. P3 (audit UI) gates enabling the dream worker on real owner data.
|
||||
- Nine measurable success criteria, all technology-agnostic.
|
||||
- The "Assumptions" section bakes in defaults that were discussed during brainstorming so no clarification questions are needed.
|
||||
@@ -0,0 +1,78 @@
|
||||
# Contract — MCP Tool Response Injection
|
||||
|
||||
## Wrapper shape
|
||||
|
||||
For every injection-eligible MCP tool, the response JSON body gains an optional `relevant_context` field appended at the top level:
|
||||
|
||||
```json
|
||||
{
|
||||
"<existing tool result fields>": "...",
|
||||
"relevant_context": {
|
||||
"memories": [
|
||||
{
|
||||
"id": 12345,
|
||||
"from_agent": "research-mcpproxy",
|
||||
"channel": "open-brain",
|
||||
"body": "KuzuDB archived 2025-10-10; not viable for synapbus",
|
||||
"created_at": "2026-05-08T14:22:00Z",
|
||||
"score": 0.91,
|
||||
"match_type": "hybrid",
|
||||
"pinned": false,
|
||||
"truncated": false
|
||||
}
|
||||
],
|
||||
"core_memory": "<text>", // only on session-start tools, may be omitted
|
||||
"packet_chars": 412,
|
||||
"packet_token_estimate": 103,
|
||||
"retrieval_query": "Kuzu graph DB",
|
||||
"search_mode": "auto"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Field semantics
|
||||
|
||||
- `memories` — up to N items (default 5, env `SYNAPBUS_INJECTION_MAX_ITEMS`), ranked by RRF score. Body is verbatim from the message; `truncated=true` only when a single item had to be cut to fit the token budget.
|
||||
- `core_memory` — Letta-style per-(owner, agent) blob. Present only on session-start-class tools (`my_status` today; extensible). Omitted entirely when no blob is set.
|
||||
- `packet_chars` — total character count of the assembled packet (memories + core + delimiters).
|
||||
- `packet_token_estimate` — `packet_chars/4` rounded up.
|
||||
- `retrieval_query` — the text that drove retrieval (the tool's argument body, or "<recent activity>" fallback). Useful for "why did my agent know this?" debug.
|
||||
- `search_mode` — `"auto"` / `"semantic"` / `"fulltext"` per `search.Service`.
|
||||
|
||||
## Injection-eligible tools
|
||||
|
||||
| Tool | Retrieval query source | Include core memory? |
|
||||
|------|------------------------|----------------------|
|
||||
| `my_status` | `<recent activity>` | ✅ |
|
||||
| `claim_messages` | concatenated bodies of claimed messages | ❌ |
|
||||
| `read_inbox` | concatenated bodies of returned messages | ❌ |
|
||||
| `send_message` | body of the sent message | ❌ |
|
||||
| `search` / `search_messages` | the user's query | ❌ |
|
||||
| `execute` | the request payload (stringified args) | ❌ |
|
||||
| `read_channel` | channel topic + last N message bodies | ❌ |
|
||||
|
||||
Tools NOT injected: `list_resources`, `propose_agent`, `propose_task_tree`, `complete_goal`, `create_goal`, `claim_task`, `get_replies`, `request_resource`, `react` (write-only or metadata).
|
||||
|
||||
## Configuration
|
||||
|
||||
| Env var | Default | Description |
|
||||
|---------|---------|-------------|
|
||||
| `SYNAPBUS_INJECTION_ENABLED` | `0` (off) | Master switch. Off → no `relevant_context` field on any response (FR-012, SC-009). |
|
||||
| `SYNAPBUS_INJECTION_BUDGET_TOKENS` | `500` | Soft cap. Greedy fill in descending score; truncate last admitted item if needed. |
|
||||
| `SYNAPBUS_INJECTION_MAX_ITEMS` | `5` | Hard cap on `memories[]` length. |
|
||||
| `SYNAPBUS_INJECTION_MIN_SCORE` | inherits `search.DefaultMinSimilarity` (0.25) | Floor — pinned memories bypass this. |
|
||||
| `SYNAPBUS_CORE_MEMORY_MAX_BYTES` | `2048` | Reject `memory_rewrite_core` over this size. |
|
||||
|
||||
## Empty / low-signal paths
|
||||
|
||||
- If `len(memories) == 0` and no core memory is set → omit the `relevant_context` field entirely. Response shape exactly matches pre-feature.
|
||||
- If `len(memories) == 0` but core memory IS set → include the field with `memories: []` and `core_memory: "..."`.
|
||||
- If a tool's response is not JSON-shaped (e.g. binary attachment download), pass through unchanged.
|
||||
|
||||
## Cross-owner safety (SC-008)
|
||||
|
||||
Retrieval is filtered by `caller.OwnerID` at the SQL level inside `search.Service.Search()`. The injection middleware never inspects or trusts the response body for owner info — it always re-derives owner from `auth.ContextAgent(ctx)`. An adversarial test asserts that an agent owned by H1 making *any* injection-eligible tool call cannot have a memory whose source message was authored by an agent owned by H2 appear in `relevant_context.memories`.
|
||||
|
||||
## Audit (FR-025)
|
||||
|
||||
Every assembled non-empty packet writes one row to `memory_injections` with `(owner_id, agent_name, tool_name, packet_size_chars, packet_items_count, message_ids[], core_blob_included, created_at)`. Rows older than 24h are purged hourly by the consolidator worker.
|
||||
@@ -0,0 +1,226 @@
|
||||
# Contract — MCP Memory-Consolidation Tools
|
||||
|
||||
These six tools are registered only when `SYNAPBUS_DREAM_ENABLED=1`. They reject any caller that does not present a valid `SYNAPBUS_DISPATCH_TOKEN` (header `X-Synapbus-Dispatch-Token` or env propagated by the harness). Token validation is described in `research.md` R7.
|
||||
|
||||
All six tools accept an implicit `dispatch_token` from the request context and explicit `owner_id` (asserted to match the token's owner; otherwise error `dispatch_token_owner_mismatch`).
|
||||
|
||||
---
|
||||
|
||||
## 1. `memory_list_unprocessed`
|
||||
|
||||
List recent memory-eligible messages the owner's pool has not yet consolidated.
|
||||
|
||||
**Input**:
|
||||
```json
|
||||
{
|
||||
"owner_id": "algis",
|
||||
"since_message_id": 0, // optional — exclusive lower bound
|
||||
"limit": 50 // optional — default 50, max 200
|
||||
}
|
||||
```
|
||||
|
||||
**Output**:
|
||||
```json
|
||||
{
|
||||
"memories": [
|
||||
{
|
||||
"id": 12345,
|
||||
"from_agent": "research-mcpproxy",
|
||||
"channel": "open-brain",
|
||||
"body": "...",
|
||||
"created_at": "2026-05-08T14:22:00Z",
|
||||
"links": [{"to": 12000, "type": "mention"}]
|
||||
}
|
||||
],
|
||||
"max_id_returned": 12567 // pass as since_message_id on next call
|
||||
}
|
||||
```
|
||||
|
||||
**Behavior**: Returns active memories from memory channels for the owner, ordered by id ascending, excluding any already linked via `refines` / `duplicate_of` / `superseded_by` to a more recent memory.
|
||||
|
||||
---
|
||||
|
||||
## 2. `memory_write_reflection`
|
||||
|
||||
Write a higher-level abstraction back to the memory pool as a new message in `#open-brain` (or a `#reflections-<owner>` channel if one exists), tagged.
|
||||
|
||||
**Input**:
|
||||
```json
|
||||
{
|
||||
"owner_id": "algis",
|
||||
"body": "Across recent discussions, the team has committed to ...",
|
||||
"source_message_ids": [12001, 12030, 12089],
|
||||
"tags": ["reflection", "weekly"]
|
||||
}
|
||||
```
|
||||
|
||||
**Output**:
|
||||
```json
|
||||
{
|
||||
"memory_id": 12601,
|
||||
"channel": "open-brain",
|
||||
"links_created": 3 // one 'refines' link per source
|
||||
}
|
||||
```
|
||||
|
||||
**Behavior**:
|
||||
1. Inserts a message authored by a synthetic `dream:<owner>` agent into the memory channel (preferring `#reflections-<owner>` if it exists, else `#open-brain`).
|
||||
2. Inserts a `refines` link from the new memory to each source.
|
||||
3. Embeds the new message via the existing async pipeline.
|
||||
4. Records `{tool, args, target_message_id: new_id}` in the parent job's `actions` JSON.
|
||||
|
||||
**Errors**: `source_not_found` if any source ID is missing or belongs to a different owner.
|
||||
|
||||
---
|
||||
|
||||
## 3. `memory_rewrite_core`
|
||||
|
||||
Replace the per-(owner, agent) core memory blob wholesale.
|
||||
|
||||
**Input**:
|
||||
```json
|
||||
{
|
||||
"owner_id": "algis",
|
||||
"agent_name": "research-mcpproxy",
|
||||
"blob": "You are research-mcpproxy. Currently focused on benchmarking ..."
|
||||
}
|
||||
```
|
||||
|
||||
**Output**:
|
||||
```json
|
||||
{
|
||||
"owner_id": "algis",
|
||||
"agent_name": "research-mcpproxy",
|
||||
"previous_blob": "...",
|
||||
"new_blob_chars": 412,
|
||||
"updated_at": "2026-05-11T03:00:14Z"
|
||||
}
|
||||
```
|
||||
|
||||
**Errors**:
|
||||
- `core_memory_too_large` if `len(blob) > SYNAPBUS_CORE_MEMORY_MAX_BYTES`.
|
||||
- `agent_not_owned` if the target agent's `owner_id != caller.owner_id`.
|
||||
- `agent_protected` if the agent has `metadata.protected_core=true`.
|
||||
|
||||
---
|
||||
|
||||
## 4. `memory_mark_duplicate`
|
||||
|
||||
Mark two memories as duplicates; one is kept canonical, the other is soft-deleted.
|
||||
|
||||
**Input**:
|
||||
```json
|
||||
{
|
||||
"owner_id": "algis",
|
||||
"a_id": 12001,
|
||||
"b_id": 12089,
|
||||
"keep_id": 12001,
|
||||
"reason": "Same fact about KuzuDB archival, b is shorter paraphrase"
|
||||
}
|
||||
```
|
||||
|
||||
**Output**:
|
||||
```json
|
||||
{
|
||||
"keep_id": 12001,
|
||||
"soft_deleted_id": 12089,
|
||||
"link_created_id": 4501
|
||||
}
|
||||
```
|
||||
|
||||
**Behavior**:
|
||||
1. Inserts a `duplicate_of` link from `loser_id → keep_id`.
|
||||
2. Appends `{tool: "memory_mark_duplicate", target_message_id: loser_id, args: {keep_id, ...}}` to the job's `actions` JSON.
|
||||
3. The `memory_status` view then derives `loser_id` as `soft_deleted`.
|
||||
|
||||
**Errors**: `not_same_owner`, `keep_id_not_in_pair`, `already_duplicate`.
|
||||
|
||||
---
|
||||
|
||||
## 5. `memory_supersede`
|
||||
|
||||
Mark memory A as obsoleted by memory B (temporal validity).
|
||||
|
||||
**Input**:
|
||||
```json
|
||||
{
|
||||
"owner_id": "algis",
|
||||
"a_id": 12001,
|
||||
"b_id": 12500,
|
||||
"reason": "Fact updated: as of 2026-04, kubernetes deploy moved off helm chart"
|
||||
}
|
||||
```
|
||||
|
||||
**Output**:
|
||||
```json
|
||||
{
|
||||
"superseded_id": 12001,
|
||||
"by_id": 12500,
|
||||
"link_created_id": 4502
|
||||
}
|
||||
```
|
||||
|
||||
**Behavior**: Inserts `superseded_by` link from `a → b`. View derives `a.status = 'superseded'`, `a.superseded_by = b`. Reason is preserved in the action JSON.
|
||||
|
||||
**Errors**: `not_same_owner`, `cycle_detected` (b transitively superseded by a).
|
||||
|
||||
---
|
||||
|
||||
## 6. `memory_add_link`
|
||||
|
||||
Add a typed link between two memories (A-MEM Zettelkasten style).
|
||||
|
||||
**Input**:
|
||||
```json
|
||||
{
|
||||
"owner_id": "algis",
|
||||
"src_id": 12030,
|
||||
"dst_id": 12085,
|
||||
"relation_type": "refines",
|
||||
"metadata": {"confidence": 0.84}
|
||||
}
|
||||
```
|
||||
|
||||
**Output**:
|
||||
```json
|
||||
{
|
||||
"link_id": 4510
|
||||
}
|
||||
```
|
||||
|
||||
**Constraints**:
|
||||
- `relation_type` ∈ {`refines`, `contradicts`, `examples`, `related`}. The reserved auto-types (`mention`, `reply_to`, `channel_cooccurrence`) and consolidation-types (`duplicate_of`, `superseded_by`) are written by other tools / jobs and rejected here.
|
||||
|
||||
**Errors**: `relation_type_reserved`, `not_same_owner`, `link_already_exists`.
|
||||
|
||||
---
|
||||
|
||||
## Common error codes
|
||||
|
||||
| Code | Meaning |
|
||||
|------|---------|
|
||||
| `dispatch_token_missing` | No token in request context |
|
||||
| `dispatch_token_expired` | Token past `expires_at` |
|
||||
| `dispatch_token_revoked` | Token explicitly revoked |
|
||||
| `dispatch_token_owner_mismatch` | Request's `owner_id` differs from token's |
|
||||
| `dispatch_token_wrong_job` | Token bound to a different `consolidation_job_id` than active call sequence |
|
||||
| `not_same_owner` | A referenced message belongs to a different owner |
|
||||
| `source_not_found` | Referenced message id does not exist |
|
||||
| `core_memory_too_large` | Blob exceeds size cap |
|
||||
|
||||
All errors follow MCP's standard `{"error": {"code": "...", "message": "..."}}` shape.
|
||||
|
||||
## Audit interaction
|
||||
|
||||
Every successful invocation of any of the six tools appends an entry to the parent `memory_consolidation_jobs.actions` JSON array:
|
||||
|
||||
```json
|
||||
{
|
||||
"tool": "memory_mark_duplicate",
|
||||
"args": {"a_id": 12001, "b_id": 12089, "keep_id": 12001, "reason": "..."},
|
||||
"target_message_id": 12089,
|
||||
"at": "2026-05-11T03:00:14Z"
|
||||
}
|
||||
```
|
||||
|
||||
On job completion (success / partial / failed), the worker updates `status`, `summary`, `finished_at`. The token's `used_at` is set on first call; the token remains usable for the rest of the same job.
|
||||
@@ -0,0 +1,210 @@
|
||||
# Data Model — 020-proactive-memory-dream-worker
|
||||
|
||||
## Reused tables (unchanged)
|
||||
|
||||
- **`messages`** — every "memory" is a message on a memory-flagged channel. No schema change.
|
||||
- **`channels`** — `metadata` JSON gains an optional `"is_memory": true` flag. No schema change.
|
||||
- **`agents`** — `owner_id` is the canonical scope. No schema change.
|
||||
- **`embeddings`** — existing per-message embeddings used unchanged by `search.Service`.
|
||||
|
||||
## New tables (migration `028_memory_consolidation.sql`)
|
||||
|
||||
### `memory_core`
|
||||
|
||||
Per-(owner, agent_name) identity-and-context blob.
|
||||
|
||||
```sql
|
||||
CREATE TABLE memory_core (
|
||||
owner_id TEXT NOT NULL,
|
||||
agent_name TEXT NOT NULL,
|
||||
blob TEXT NOT NULL,
|
||||
updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_by TEXT NOT NULL, -- 'human:<owner>' or 'agent:<name>:<token>'
|
||||
PRIMARY KEY (owner_id, agent_name)
|
||||
);
|
||||
CREATE INDEX idx_memory_core_owner ON memory_core(owner_id);
|
||||
```
|
||||
|
||||
Constraints:
|
||||
- `LENGTH(blob) <= SYNAPBUS_CORE_MEMORY_MAX_BYTES` (default 2048) — enforced in Go, not SQL.
|
||||
- Replaces wholesale on update — no diff/merge.
|
||||
|
||||
### `memory_links`
|
||||
|
||||
Directed typed edges between two message IDs.
|
||||
|
||||
```sql
|
||||
CREATE TABLE memory_links (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
src_message_id INTEGER NOT NULL,
|
||||
dst_message_id INTEGER NOT NULL,
|
||||
relation_type TEXT NOT NULL CHECK (relation_type IN (
|
||||
'refines', 'contradicts', 'examples', 'related',
|
||||
'duplicate_of', 'superseded_by',
|
||||
'mention', 'reply_to', 'channel_cooccurrence'
|
||||
)),
|
||||
owner_id TEXT NOT NULL, -- denormalized for fast owner-scoped queries
|
||||
created_by TEXT NOT NULL, -- 'human:<owner>' / 'agent:<name>:<token>' / 'auto:<rule>'
|
||||
metadata TEXT NOT NULL DEFAULT '{}',
|
||||
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(src_message_id, dst_message_id, relation_type)
|
||||
);
|
||||
CREATE INDEX idx_memory_links_owner ON memory_links(owner_id);
|
||||
CREATE INDEX idx_memory_links_src ON memory_links(src_message_id);
|
||||
CREATE INDEX idx_memory_links_dst ON memory_links(dst_message_id);
|
||||
CREATE INDEX idx_memory_links_type ON memory_links(relation_type);
|
||||
```
|
||||
|
||||
Relation types:
|
||||
- LLM-generated: `refines`, `contradicts`, `examples`, `related`, `duplicate_of`, `superseded_by`
|
||||
- Auto-generated by messaging layer: `mention`, `reply_to`, `channel_cooccurrence`
|
||||
|
||||
### `memory_consolidation_jobs`
|
||||
|
||||
Audit log of dispatched dream jobs and their resulting actions.
|
||||
|
||||
```sql
|
||||
CREATE TABLE memory_consolidation_jobs (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
owner_id TEXT NOT NULL,
|
||||
job_type TEXT NOT NULL CHECK (job_type IN (
|
||||
'reflection', 'core_rewrite', 'dedup_contradiction', 'link_gen'
|
||||
)),
|
||||
status TEXT NOT NULL CHECK (status IN (
|
||||
'pending', 'dispatched', 'running', 'succeeded', 'partial', 'failed', 'expired'
|
||||
)) DEFAULT 'pending',
|
||||
trigger_reason TEXT NOT NULL, -- 'watermark:N', 'cron:nightly', 'manual:<owner>'
|
||||
dispatch_token TEXT, -- FK-ish into memory_dispatch_tokens.token
|
||||
harness_run_id TEXT, -- FK into harness_runs.id (existing)
|
||||
actions TEXT NOT NULL DEFAULT '[]', -- JSON array of {tool, args, before, after}
|
||||
summary TEXT, -- human-readable summary written on completion
|
||||
error TEXT,
|
||||
lease_until DATETIME, -- in-flight lease; null when not running
|
||||
started_at DATETIME,
|
||||
finished_at DATETIME,
|
||||
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
CREATE INDEX idx_consolidation_owner_status ON memory_consolidation_jobs(owner_id, status);
|
||||
CREATE INDEX idx_consolidation_lease ON memory_consolidation_jobs(lease_until)
|
||||
WHERE status = 'running';
|
||||
CREATE UNIQUE INDEX idx_consolidation_in_flight
|
||||
ON memory_consolidation_jobs(owner_id, job_type)
|
||||
WHERE status IN ('pending', 'dispatched', 'running');
|
||||
```
|
||||
|
||||
The partial-unique index enforces: at most one in-flight job per `(owner, job_type)`.
|
||||
|
||||
### `memory_pins`
|
||||
|
||||
Owner-pinned message IDs that bypass the relevance floor.
|
||||
|
||||
```sql
|
||||
CREATE TABLE memory_pins (
|
||||
owner_id TEXT NOT NULL,
|
||||
message_id INTEGER NOT NULL,
|
||||
pinned_by TEXT NOT NULL, -- 'human:<owner>'
|
||||
note TEXT, -- why this is pinned (free-form)
|
||||
pinned_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
PRIMARY KEY (owner_id, message_id)
|
||||
);
|
||||
CREATE INDEX idx_memory_pins_owner ON memory_pins(owner_id);
|
||||
```
|
||||
|
||||
### `memory_dispatch_tokens`
|
||||
|
||||
Single-use, owner-bound, job-bound tokens.
|
||||
|
||||
```sql
|
||||
CREATE TABLE memory_dispatch_tokens (
|
||||
token TEXT PRIMARY KEY, -- 32-byte random, base64url
|
||||
owner_id TEXT NOT NULL,
|
||||
consolidation_job_id INTEGER NOT NULL,
|
||||
issued_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
expires_at DATETIME NOT NULL, -- issued_at + 15m
|
||||
used_at DATETIME,
|
||||
revoked_at DATETIME,
|
||||
FOREIGN KEY (consolidation_job_id) REFERENCES memory_consolidation_jobs(id)
|
||||
);
|
||||
CREATE INDEX idx_memory_tokens_job ON memory_dispatch_tokens(consolidation_job_id);
|
||||
```
|
||||
|
||||
Token is valid when `revoked_at IS NULL AND expires_at > now() AND consolidation_job_id == claimed job`. `used_at` becomes informational once first set; subsequent calls within the same job are allowed.
|
||||
|
||||
### `memory_injections`
|
||||
|
||||
24-hour rolling ring of what was injected for each tool call.
|
||||
|
||||
```sql
|
||||
CREATE TABLE memory_injections (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
owner_id TEXT NOT NULL,
|
||||
agent_name TEXT NOT NULL,
|
||||
tool_name TEXT NOT NULL,
|
||||
packet_size_chars INTEGER NOT NULL,
|
||||
packet_items_count INTEGER NOT NULL,
|
||||
message_ids TEXT NOT NULL, -- JSON array of message_ids included
|
||||
core_blob_included BOOLEAN NOT NULL DEFAULT 0,
|
||||
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
CREATE INDEX idx_memory_injections_owner_time ON memory_injections(owner_id, created_at);
|
||||
```
|
||||
|
||||
A daily cleanup query (run by the consolidator worker) deletes `created_at < now() - 24h`.
|
||||
|
||||
## View: `memory_status`
|
||||
|
||||
Derived from `memory_consolidation_jobs.actions`. Tells retrieval whether a message is active / soft-deleted / superseded.
|
||||
|
||||
```sql
|
||||
CREATE VIEW memory_status AS
|
||||
WITH actions AS (
|
||||
SELECT
|
||||
owner_id,
|
||||
json_extract(act.value, '$.target_message_id') AS message_id,
|
||||
json_extract(act.value, '$.tool') AS tool,
|
||||
json_extract(act.value, '$.args.keep_id') AS keep_id,
|
||||
json_extract(act.value, '$.args.b_id') AS superseded_by,
|
||||
json_extract(act.value, '$.args.reason') AS reason,
|
||||
finished_at AS at
|
||||
FROM memory_consolidation_jobs j, json_each(j.actions) act
|
||||
WHERE j.status IN ('succeeded', 'partial')
|
||||
)
|
||||
SELECT
|
||||
message_id,
|
||||
owner_id,
|
||||
CASE
|
||||
WHEN MAX(CASE WHEN tool='memory_supersede' THEN at END) IS NOT NULL THEN 'superseded'
|
||||
WHEN MAX(CASE WHEN tool='memory_mark_duplicate' AND keep_id != message_id THEN at END) IS NOT NULL THEN 'soft_deleted'
|
||||
ELSE 'active'
|
||||
END AS status,
|
||||
MAX(CASE WHEN tool='memory_supersede' THEN superseded_by END) AS superseded_by,
|
||||
MAX(CASE WHEN tool='memory_mark_duplicate' AND keep_id != message_id THEN at END) AS soft_deleted_at,
|
||||
MAX(reason) AS reason
|
||||
FROM actions
|
||||
GROUP BY message_id, owner_id;
|
||||
```
|
||||
|
||||
Retrieval excludes `status != 'active'` unless `pinned` or `include_inactive=true`.
|
||||
|
||||
## State transitions
|
||||
|
||||
```text
|
||||
Memory message (active by default — not in memory_status)
|
||||
│
|
||||
├── memory_mark_duplicate(keep != self) → soft_deleted (via view)
|
||||
│ └── owner restore action overrides → active
|
||||
│
|
||||
├── memory_supersede(b) → superseded (via view)
|
||||
│ └── owner restore → active
|
||||
│
|
||||
└── (no consolidation event) → active
|
||||
```
|
||||
|
||||
Pinning and protection are orthogonal flags (in `memory_pins`; protection is currently piggybacked on a `protected_until` JSON field in `memory_pins.note` — promoted to a column in Phase 2 if needed).
|
||||
|
||||
## Indexes & query patterns
|
||||
|
||||
Hot queries:
|
||||
1. **Injection retrieval**: `search.Service.Search()` already paginates → join `memory_status` to drop non-active → drop in pin overlay → apply token budget.
|
||||
2. **Dream worker watermark check**: `SELECT COUNT(*) FROM messages m JOIN agents a ON m.from_agent=a.name WHERE a.owner_id=? AND m.channel_id IN (memory_channels) AND m.id > last_reflection_max_id`. Owner-filtered + range — uses existing message indexes.
|
||||
3. **Audit lookup**: `SELECT * FROM memory_consolidation_jobs WHERE owner_id=? ORDER BY created_at DESC LIMIT 50`. Covered by `idx_consolidation_owner_status`.
|
||||
@@ -0,0 +1,136 @@
|
||||
# Implementation Plan: Proactive Memory & Dream Worker
|
||||
|
||||
**Branch**: `020-proactive-memory-dream-worker` | **Date**: 2026-05-11 | **Spec**: [spec.md](./spec.md)
|
||||
**Input**: Feature specification from `/specs/020-proactive-memory-dream-worker/spec.md`
|
||||
|
||||
## Summary
|
||||
|
||||
Make SynapBus push minimal owner-scoped memory into every agent task automatically (Story 1, P1), give each agent a small editable identity blob always-included in session-start (Story 2, P2), and run a background "dream" worker that periodically dispatches consolidation work to a Claude-Code agent through the existing `harness.Harness` seam (Story 3, P2). An owner-facing audit/override UI lands later (Story 4, P3). All work reuses existing primitives: `search.Service` for retrieval, `harness.Harness` for dispatch, `messages` table for the memory pool, the `StalemateWorker` ticker pattern for the new `ConsolidatorWorker`. One new SQLite migration adds six tables. Zero CGO, zero new external dependencies.
|
||||
|
||||
## Technical Context
|
||||
|
||||
**Language/Version**: Go 1.25+ (per `go.mod`)
|
||||
**Primary Dependencies**: `mark3labs/mcp-go` (MCP tools), `go-chi/chi` (HTTP), `modernc.org/sqlite` (storage), `TFMV/hnsw` (vectors via existing `search.Service`), existing `internal/harness` package (dispatch seam). **No new external dependencies.**
|
||||
**Storage**: SQLite via `modernc.org/sqlite` — one new migration `028_memory_consolidation.sql`. Memory pool reuses the existing `messages` table on memory-flagged channels.
|
||||
**Testing**: `go test ./...` table-driven tests, mocked `harness.Harness` for dream-worker tests, in-memory SQLite for storage tests
|
||||
**Target Platform**: `linux/amd64` (kubic deployment), `darwin/arm64` (dev). Pure-Go cross-compilation required.
|
||||
**Project Type**: Single-binary Go service with embedded Svelte SPA (existing layout).
|
||||
**Performance Goals**: Median injection overhead < 50ms per tool call (SC-002). Full owner-level consolidation pass ≤ 10 min for ≤ 5,000 memories (SC-005).
|
||||
**Constraints**: Zero CGO (Constitution III). Single binary (Constitution I). Injection MUST be disable-able with zero payload-shape change when disabled (FR-012, SC-009).
|
||||
**Scale/Scope**: One owner with ~500 memories today; design for 5k/owner, 10 owners on the reference deployment.
|
||||
|
||||
## Constitution Check
|
||||
|
||||
*Gate: must pass before Phase 0 research. Re-evaluated after Phase 1 design.*
|
||||
|
||||
| Principle | Check | Status |
|
||||
|-----------|-------|--------|
|
||||
| I. Local-first, single binary | New worker ships in the same binary; no external service introduced. | ✅ |
|
||||
| II. MCP-native | New memory-consolidation tools registered via `mark3labs/mcp-go`. REST endpoints are added only for the Web UI audit tab (Phase 2). | ✅ |
|
||||
| III. Pure Go, zero CGO | No new dependencies; reuses `modernc.org/sqlite` + `TFMV/hnsw`. | ✅ |
|
||||
| IV. Multi-tenant w/ ownership | Owner scoping is the *core* of the design. Retrieval filters by `owner_id`; dispatch tokens are owner-bound. | ✅ |
|
||||
| V. Embedded OAuth 2.1 | Unchanged; consolidation agents authenticate via the existing API-key path. | ✅ |
|
||||
| VI. Semantic-ready storage | Reuses `search.Service.Search()`; gracefully degrades to FTS when no embedding provider configured. | ✅ |
|
||||
| VII. Swarm intelligence | `#open-brain` is a stigmergic blackboard already. Reflection writes annotated memories back to the blackboard. | ✅ |
|
||||
| VIII. Observable by default | Every consolidation mutation hits an immutable `memory_consolidation_jobs` audit row + a `memory_audit_log` row keyed by dispatch token. Every injection hits `memory_injections` (24h ring). | ✅ |
|
||||
| IX. Progressive complexity | Feature is gated by `SYNAPBUS_INJECTION_ENABLED` and `SYNAPBUS_DREAM_ENABLED`. Default off until owner audit UI lands. | ✅ |
|
||||
| X. Web UI first-class | Memory tab is in scope but deferred to Phase 2 of this feature (post-kubic-deploy). Audit data is captured from day one so the UI can render history retroactively. | ⚠ deferred — tracked, not skipped |
|
||||
|
||||
**Gate result**: PASS. No violations require Complexity Tracking.
|
||||
|
||||
## Project Structure
|
||||
|
||||
### Documentation (this feature)
|
||||
|
||||
```text
|
||||
specs/020-proactive-memory-dream-worker/
|
||||
├── plan.md # This file
|
||||
├── spec.md # Feature spec
|
||||
├── research.md # Phase 0 — technical-unknowns resolution
|
||||
├── data-model.md # Phase 1 — entities and tables
|
||||
├── contracts/
|
||||
│ ├── mcp-injection.md # Shape of relevant_context block on tool responses
|
||||
│ └── mcp-memory-tools.md # 6 new dream-agent tools (input schemas, behavior, errors)
|
||||
├── quickstart.md # Phase 1 — how to verify locally and on kubic
|
||||
├── tasks.md # (Phase 2 — produced by /speckit.tasks)
|
||||
└── checklists/
|
||||
└── requirements.md # Spec quality checklist
|
||||
```
|
||||
|
||||
### Source Code (repository root, existing layout extended)
|
||||
|
||||
```text
|
||||
cmd/synapbus/main.go # MODIFIED: wire up ConsolidatorWorker, gate new MCP tools
|
||||
|
||||
internal/
|
||||
├── messaging/
|
||||
│ ├── consolidator.go # NEW: ConsolidatorWorker (ticker + dispatch via harness)
|
||||
│ ├── consolidator_test.go # NEW
|
||||
│ ├── memory.go # NEW: core-memory CRUD, link CRUD, supersession, pin/protected
|
||||
│ ├── memory_test.go # NEW
|
||||
│ ├── memory_channels.go # NEW: discover memory-flagged channels per owner
|
||||
│ └── stalemate.go # UNCHANGED — pattern template
|
||||
├── search/
|
||||
│ └── injection.go # NEW: BuildContextPacket(ctx, agent, opts) → ContextPacket
|
||||
├── mcp/
|
||||
│ ├── injection_wrap.go # NEW: middleware appending relevant_context to tool results
|
||||
│ ├── memory_tools.go # NEW: 6 dream-agent tools (memory_list_unprocessed,
|
||||
│ │ # memory_write_reflection, memory_rewrite_core,
|
||||
│ │ # memory_mark_duplicate, memory_supersede, memory_add_link)
|
||||
│ ├── memory_tools_test.go # NEW
|
||||
│ ├── tools_hybrid.go # MODIFIED: wrap eligible tool handlers via injection_wrap
|
||||
│ └── server.go # MODIFIED: register memory tools when dream worker enabled
|
||||
├── storage/schema/
|
||||
│ └── 028_memory_consolidation.sql # NEW: 6 tables (see data-model.md)
|
||||
└── harness/ # UNCHANGED — used via existing Execute() API
|
||||
|
||||
docs/superpowers/specs/2026-05-11-internal-only-disable-approvals-design.md # pre-existing, untouched
|
||||
```
|
||||
|
||||
**Structure Decision**: Single-project Go layout (matches existing `cmd/synapbus` + `internal/*` structure). Memory pool reuses `messages`; six new tables in one new migration are the only schema addition. No new top-level package — new files split between `internal/messaging` (storage-adjacent), `internal/search` (retrieval-adjacent), and `internal/mcp` (tool-surface-adjacent).
|
||||
|
||||
## Phase 0 — Research
|
||||
|
||||
See [research.md](./research.md). Resolved unknowns:
|
||||
|
||||
1. **How does the dream worker invoke a Claude Code agent without using a system DM?** → through `harness.Harness.Execute(ExecRequest)` with the existing `kubernetes_job` or `local_subprocess` backend. The agent's MCP session carries a one-time `dispatch_token` injected via `ExecRequest.Env` that authorizes the consolidation tools.
|
||||
2. **Where do owners live in the data model today?** → `agents.owner_id` (text) is the canonical scope. Every retrieval JOINs `agents` to resolve the caller's owner.
|
||||
3. **Soft-delete on messages — new column or metadata flag?** → metadata flag on `memory_consolidation_jobs.actions` + a derived view. Avoids touching the hot `messages` table. Soft-delete is enforced at retrieval time by joining against `memory_status` view.
|
||||
4. **Token-budget estimator** → `len(s)/4` heuristic. Faster than tokenizing, accurate enough for budget enforcement.
|
||||
5. **MCP transport — how do we identify the calling agent's API key/owner?** → existing `auth.ContextAgent(ctx)` returns `*agents.Agent` from the request context; `agent.OwnerID` is the scope key.
|
||||
|
||||
## Phase 1 — Design & Contracts
|
||||
|
||||
### Data model
|
||||
|
||||
See [data-model.md](./data-model.md). Migration `028_memory_consolidation.sql` adds six tables:
|
||||
|
||||
- `memory_core` — per-(owner, agent_name) editable text blob, size-capped at `SYNAPBUS_CORE_MEMORY_MAX_BYTES` (default 2048).
|
||||
- `memory_links` — directed typed edges between two message IDs.
|
||||
- `memory_consolidation_jobs` — audit log of every dispatched dream job + the actions it took.
|
||||
- `memory_pins` — owner-pinned message IDs, always-included in injection.
|
||||
- `memory_dispatch_tokens` — one-time, owner-bound, single-job-bound tokens.
|
||||
- `memory_injections` — 24h rolling ring of (tool_name, agent_id, packet_summary) for debug-ability.
|
||||
|
||||
Plus one view `memory_status` (active / soft-deleted / superseded — derived from `memory_consolidation_jobs.actions`).
|
||||
|
||||
### Contracts
|
||||
|
||||
- [contracts/mcp-injection.md](./contracts/mcp-injection.md) — shape of `relevant_context` block appended to tool responses.
|
||||
- [contracts/mcp-memory-tools.md](./contracts/mcp-memory-tools.md) — input/output schemas for the six new memory-consolidation tools.
|
||||
|
||||
### Quickstart
|
||||
|
||||
See [quickstart.md](./quickstart.md) — covers local dev (build → migrate → seed → run with `SYNAPBUS_INJECTION_ENABLED=1 SYNAPBUS_DREAM_ENABLED=1`) and kubic deployment (build image, push, kustomize apply, tail dream-worker logs, dispatch a manual reflection job, verify audit log).
|
||||
|
||||
### Agent context update
|
||||
|
||||
Triggered by the workflow: `update-agent-context.sh claude` adds the new packages and migration to `CLAUDE.md` "Active Technologies" block.
|
||||
|
||||
## Phase 2 — Tasks
|
||||
|
||||
(Produced by `/speckit.tasks`. Not created by this command.)
|
||||
|
||||
## Complexity Tracking
|
||||
|
||||
No constitution violations. Memory-tab Web UI is deferred (tracked as Phase 2 of this feature) but is not a constitution violation since audit data is captured from day one and surfaces in CLI/REST until the SPA tab lands.
|
||||
@@ -0,0 +1,162 @@
|
||||
# Quickstart — 020-proactive-memory-dream-worker
|
||||
|
||||
How to build, run, and verify the proactive-memory + dream-worker feature locally and on the kubic deployment.
|
||||
|
||||
## Local: build & test
|
||||
|
||||
```bash
|
||||
cd ~/repos/synapbus
|
||||
make test # full test suite — should be green
|
||||
go build -o ./synapbus ./cmd/synapbus
|
||||
```
|
||||
|
||||
## Local: run with feature flags on
|
||||
|
||||
```bash
|
||||
export SYNAPBUS_DATA_DIR=/tmp/synapbus-dream
|
||||
export SYNAPBUS_INJECTION_ENABLED=1
|
||||
export SYNAPBUS_DREAM_ENABLED=1
|
||||
export SYNAPBUS_EMBEDDING_PROVIDER=openai
|
||||
export OPENAI_API_KEY=<key>
|
||||
mkdir -p $SYNAPBUS_DATA_DIR
|
||||
|
||||
./synapbus serve --port 8080 --data $SYNAPBUS_DATA_DIR
|
||||
```
|
||||
|
||||
## Verify Story 1 (injection)
|
||||
|
||||
```bash
|
||||
# Create an agent and an owner
|
||||
./synapbus --socket $SYNAPBUS_DATA_DIR/synapbus.sock agents create \
|
||||
--name dogfood-1 --owner algis --description "smoke test agent"
|
||||
|
||||
# Get an API key for the agent
|
||||
KEY=$(./synapbus --socket $SYNAPBUS_DATA_DIR/synapbus.sock apikeys create \
|
||||
--agent dogfood-1 --json | jq -r .key)
|
||||
|
||||
# Seed memory channel with a few messages
|
||||
./synapbus --socket $SYNAPBUS_DATA_DIR/synapbus.sock channels create \
|
||||
--name open-brain --type blackboard
|
||||
./synapbus --socket $SYNAPBUS_DATA_DIR/synapbus.sock messages send \
|
||||
--agent dogfood-1 --channel open-brain \
|
||||
--body "KuzuDB was archived 2025-10-10. Not viable for SynapBus."
|
||||
|
||||
# Call my_status via MCP — relevant_context should appear
|
||||
curl -s -X POST http://localhost:8080/mcp \
|
||||
-H "Authorization: Bearer $KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"my_status","arguments":{}}}' \
|
||||
| jq '.result.content[0].text | fromjson | .relevant_context'
|
||||
```
|
||||
|
||||
Expected: `relevant_context.memories` is non-empty, contains the seeded message, `core_memory` is null/omitted (none set yet).
|
||||
|
||||
Set a core memory and re-run:
|
||||
|
||||
```bash
|
||||
# Direct DB or admin CLI:
|
||||
sqlite3 $SYNAPBUS_DATA_DIR/synapbus.db \
|
||||
"INSERT INTO memory_core(owner_id, agent_name, blob, updated_by)
|
||||
VALUES('algis', 'dogfood-1', 'You are a dogfood test agent.', 'human:algis');"
|
||||
|
||||
# Re-call my_status — relevant_context.core_memory should now be populated
|
||||
```
|
||||
|
||||
## Verify Story 3 (dream worker manual dispatch)
|
||||
|
||||
```bash
|
||||
# Force a reflection job for owner=algis
|
||||
./synapbus --socket $SYNAPBUS_DATA_DIR/synapbus.sock memory dream-run \
|
||||
--owner algis --job reflection
|
||||
|
||||
# Tail logs for the worker
|
||||
# Look for:
|
||||
# component=consolidator-worker job=reflection owner=algis status=dispatched
|
||||
# component=consolidator-worker job=reflection owner=algis status=succeeded
|
||||
sqlite3 $SYNAPBUS_DATA_DIR/synapbus.db \
|
||||
"SELECT id, job_type, status, summary, finished_at
|
||||
FROM memory_consolidation_jobs ORDER BY id DESC LIMIT 5;"
|
||||
```
|
||||
|
||||
Expected: one new row in `memory_consolidation_jobs` with `status=succeeded` (or `partial` if seed pool too small). For a reflection job, expect 0–5 new `body LIKE 'REFLECTION:%'` messages in `#open-brain`.
|
||||
|
||||
## Verify token isolation (SC-008)
|
||||
|
||||
```bash
|
||||
# Create a second owner + agent
|
||||
./synapbus --socket ... agents create --name attacker-1 --owner mallory ...
|
||||
KEY2=$(./synapbus --socket ... apikeys create --agent attacker-1 --json | jq -r .key)
|
||||
|
||||
# Attacker calls search — must NOT see algis's memories
|
||||
curl -s -X POST http://localhost:8080/mcp -H "Authorization: Bearer $KEY2" ... \
|
||||
| jq '.result.content[0].text | fromjson | .relevant_context.memories'
|
||||
```
|
||||
|
||||
Expected: empty array. If anything from algis's pool surfaces, that is a Constitution IV violation.
|
||||
|
||||
## kubic: build, push, deploy
|
||||
|
||||
```bash
|
||||
# Build linux/amd64 from darwin/arm64 — no CGO so this just works
|
||||
cd ~/repos/synapbus
|
||||
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o ./synapbus-linux-amd64 ./cmd/synapbus
|
||||
|
||||
# Build container (existing Dockerfile)
|
||||
docker build --platform=linux/amd64 -t synapbus:020-proactive-memory .
|
||||
|
||||
# Push to the cluster's local registry (kubic uses microk8s registry on :32000)
|
||||
docker tag synapbus:020-proactive-memory kubic.home.arpa:32000/synapbus:020-proactive-memory
|
||||
docker push kubic.home.arpa:32000/synapbus:020-proactive-memory
|
||||
|
||||
# Update the deployment image
|
||||
kubectl -n synapbus set image deploy/synapbus \
|
||||
synapbus=kubic.home.arpa:32000/synapbus:020-proactive-memory
|
||||
|
||||
# Enable feature flags via env (kubectl set env, or edit values in deploy/kubic-manifests/)
|
||||
kubectl -n synapbus set env deploy/synapbus \
|
||||
SYNAPBUS_INJECTION_ENABLED=1 SYNAPBUS_DREAM_ENABLED=1
|
||||
|
||||
kubectl -n synapbus rollout status deploy/synapbus
|
||||
```
|
||||
|
||||
## kubic: verify dream agent works
|
||||
|
||||
```bash
|
||||
# Tail logs filtered to the new worker
|
||||
kubectl -n synapbus logs deploy/synapbus -f | grep -E "consolidator-worker|memory-injection"
|
||||
```
|
||||
|
||||
Expected log lines on startup:
|
||||
```
|
||||
{"component":"consolidator-worker","msg":"consolidator worker started","interval":"1h","deep_cron":"0 3 * * *"}
|
||||
```
|
||||
|
||||
Expected after first watermark / cron tick:
|
||||
```
|
||||
{"component":"consolidator-worker","msg":"trigger fired","owner":"algis","job":"reflection","trigger":"watermark:20"}
|
||||
{"component":"consolidator-worker","msg":"dispatched","owner":"algis","job":"reflection","harness_run_id":"...","dispatch_token":"redacted"}
|
||||
{"component":"consolidator-worker","msg":"job completed","owner":"algis","job":"reflection","status":"succeeded","actions":3,"duration_ms":42184}
|
||||
```
|
||||
|
||||
Force a job to test without waiting:
|
||||
```bash
|
||||
kubectl exec -n synapbus deploy/synapbus -- /synapbus --socket /data/synapbus.sock \
|
||||
memory dream-run --owner algis --job reflection
|
||||
```
|
||||
|
||||
Inspect the audit log:
|
||||
```bash
|
||||
kubectl exec -n synapbus deploy/synapbus -- sqlite3 /data/synapbus.db \
|
||||
"SELECT id, job_type, status, json_array_length(actions) AS n_actions, summary
|
||||
FROM memory_consolidation_jobs ORDER BY id DESC LIMIT 10;"
|
||||
```
|
||||
|
||||
## Rollback
|
||||
|
||||
```bash
|
||||
kubectl -n synapbus set env deploy/synapbus SYNAPBUS_INJECTION_ENABLED- SYNAPBUS_DREAM_ENABLED-
|
||||
# Or roll back the image:
|
||||
kubectl -n synapbus rollout undo deploy/synapbus
|
||||
```
|
||||
|
||||
The migration `028_memory_consolidation.sql` is additive — rolling the binary back leaves the new tables in place but unused. Safe.
|
||||
@@ -0,0 +1,117 @@
|
||||
# Research — 020-proactive-memory-dream-worker
|
||||
|
||||
Resolves all `NEEDS CLARIFICATION` from `plan.md` Technical Context. Each section: **Decision** / **Rationale** / **Alternatives considered**.
|
||||
|
||||
## R1 — Dream-worker dispatch path
|
||||
|
||||
**Decision**: The `ConsolidatorWorker` invokes a consolidation agent via `harness.Harness.Execute(ExecRequest)`, picking the registered backend (`kubernetes_job` on kubic, `local_subprocess` in dev). The triggering message is `nil` (no inbound DM). A one-time dispatch token is passed via `ExecRequest.Env["SYNAPBUS_DISPATCH_TOKEN"]`, and the consolidation job ID via `ExecRequest.Env["SYNAPBUS_CONSOLIDATION_JOB_ID"]`.
|
||||
|
||||
**Rationale**:
|
||||
- The `harness` package was built specifically as "the single seam between SynapBus's reactor / webhook / MCP entry points and whatever actually runs an agent." Reusing it costs nothing.
|
||||
- Avoids the user-flagged trap: per `feedback_system_dm_no_trigger.md`, system DMs trigger reactive runs which cascade through the stalemate worker. Going through `Harness.Execute` directly bypasses messaging entirely.
|
||||
- Tokens via `Env` are already how `harness` propagates run identity (`SYNAPBUS_RUN_ID`); adding two more env vars is idiomatic.
|
||||
|
||||
**Alternatives rejected**:
|
||||
- *System DM into a "dream" channel* — would re-trigger the very pattern we're avoiding.
|
||||
- *New REST endpoint that the consolidation agent polls* — duplicates what `harness` already does; violates Principle II (MCP-native) by adding a non-MCP agent surface.
|
||||
- *Cron via OS cron + curl* — leaves the binary; violates Principle I (single binary, self-contained).
|
||||
|
||||
## R2 — Owner identification
|
||||
|
||||
**Decision**: Owner is `agents.owner_id` (string, non-null per existing schema). Retrieval uses `auth.ContextAgent(ctx).OwnerID` as the scope key.
|
||||
|
||||
**Rationale**: `agents.owner_id` is the authoritative scope key per Constitution IV. The auth middleware already populates the agent record into the request context for every MCP and REST call. No new lookup path needed.
|
||||
|
||||
**Alternatives rejected**:
|
||||
- *New `memory_owner` column on messages* — denormalizes data already available via `messages.from_agent → agents.owner_id`. Adds a write-path constraint on every message insert.
|
||||
|
||||
## R3 — Soft-delete and supersession storage
|
||||
|
||||
**Decision**: Status of a memory (`active` / `soft_deleted` / `superseded`) is derived from rows in `memory_consolidation_jobs.actions` JSON, exposed via a SQL view `memory_status(message_id, status, superseded_by, soft_deleted_at, reason)`. Retrieval joins against this view to filter.
|
||||
|
||||
**Rationale**:
|
||||
- Keeps the hot `messages` table untouched. Migration is additive only.
|
||||
- The audit log IS the source of truth — every status change is already recorded; deriving the view from it eliminates the dual-write/divergence risk.
|
||||
- View can be materialized later if join cost becomes measurable.
|
||||
|
||||
**Alternatives rejected**:
|
||||
- *Add `status` column to `messages`* — touches the most-written table on every soft-delete; requires backfill migration; harder to roll back.
|
||||
- *Separate `memory_status` table updated by triggers* — triggers in SQLite are awkward to test and reason about.
|
||||
|
||||
## R4 — Token-budget estimator
|
||||
|
||||
**Decision**: Estimate tokens as `(len(s) + 3) / 4`. Enforce budget by greedy fill in descending relevance order; truncate the *last admitted* memory if it overflows.
|
||||
|
||||
**Rationale**: For the budget gate (~500 tokens, hard upper bound 1000), char/4 is within ±15% of GPT-4/Claude tokenization for English prose — well inside the margin we'd need to tighten. A real tokenizer would force CGO (tiktoken) or add a meaningful dependency (`pkoukk/tiktoken-go` is pure Go but adds 5MB BPE tables). Not worth it for budget gating.
|
||||
|
||||
**Alternatives rejected**:
|
||||
- `pkoukk/tiktoken-go` — pure Go but bloats the binary and is overkill for a budget gate.
|
||||
- Exact tokenizer per provider — provider-specific, fragile, and we don't know which provider the consuming agent uses.
|
||||
|
||||
## R5 — MCP request-context → caller identification
|
||||
|
||||
**Decision**: Existing `auth.ContextAgent(ctx) *agents.Agent` returns the authenticated agent for the current MCP call. The new injection middleware unpacks this once and passes the agent down to `search.BuildContextPacket(ctx, agent, opts)`.
|
||||
|
||||
**Rationale**: Every existing MCP tool handler in `tools_hybrid.go` already calls `auth.ContextAgent(ctx)`. The middleware wraps the handler, so it can intercept the result before serialization without changing handler signatures.
|
||||
|
||||
**Alternatives rejected**:
|
||||
- *Pass agent through every handler* — touches every existing handler; high blast radius for a P1 feature.
|
||||
|
||||
## R6 — Memory-channel discovery
|
||||
|
||||
**Decision**: A channel is a "memory channel" if either its name matches a hardcoded list (`open-brain`, `reflections-*`) OR its `channels.metadata` JSON includes `{"is_memory": true}`. Owners set the flag via an admin endpoint (Phase 2 UI; CLI/REST in Phase 1).
|
||||
|
||||
**Rationale**:
|
||||
- Keeps the existing #open-brain working with zero migration.
|
||||
- Per-channel flag is a lightweight JSON update — no schema change.
|
||||
- Owners can opt new channels into memory without code changes.
|
||||
|
||||
**Alternatives rejected**:
|
||||
- *New `is_memory` boolean column on channels* — requires migration; the existing JSON metadata column already exists for this kind of extension.
|
||||
- *Make EVERY channel a memory channel* — defeats the owner's ability to keep ephemera (e.g. #bugs-X) out of the memory pool.
|
||||
|
||||
## R7 — Dispatch-token lifecycle
|
||||
|
||||
**Decision**: Tokens are 32-byte random strings (`crypto/rand`, base64-url encoded). Inserted with `expires_at = now() + 15m` and `used_at = NULL`. The first call to any memory-consolidation tool with a valid unexpired unused token marks it `used_at = now()`. Subsequent calls within the same job session reuse the token (checked by `consolidation_job_id` match). Tokens for a different job, or a soft-expired token, are rejected.
|
||||
|
||||
**Rationale**:
|
||||
- Single-use semantic (in the sense of "bound to one job") prevents stolen-token replay against a fresh job.
|
||||
- 15m TTL covers wallclock budget (10m) + dispatch slack.
|
||||
- Random 32 bytes is the standard secret size.
|
||||
|
||||
**Alternatives rejected**:
|
||||
- *JWT* — pulls in a JWT library and signing key management for a 15m intra-process token. Massive overkill.
|
||||
- *Owner-API-key as token* — would let the agent bypass the audit trail by calling tools as the human. Violates Constitution IV (ownership separation).
|
||||
|
||||
## R8 — Concurrent owners
|
||||
|
||||
**Decision**: A global `sync.Semaphore`-equivalent (`chan struct{}` of size `SYNAPBUS_DREAM_MAX_CONCURRENT`, default 4) gates how many owners' jobs can run concurrently. Within one owner, only one job per `(owner, job_type)` runs at a time (enforced by a row-level lock in `memory_consolidation_jobs.lease_until`).
|
||||
|
||||
**Rationale**: Single-binary, single-instance (Constitution Non-Goals) means we don't need distributed locking. A local semaphore + a leased row is enough.
|
||||
|
||||
**Alternatives rejected**:
|
||||
- *Unlimited concurrency* — could overwhelm the embedding provider's rate limits.
|
||||
- *Per-owner goroutine pool* — wastes goroutines when most owners are idle.
|
||||
|
||||
## R9 — Where injection is wired in
|
||||
|
||||
**Decision**: A thin wrapper `mcp.WrapInjection(handler, opts)` is applied at tool-registration time in `internal/mcp/server.go`. The wrapper runs the inner handler, then — if the tool is on the injection-eligible list AND the response is JSON-shaped — appends a `relevant_context` field to the JSON body before returning. Non-JSON results pass through unchanged.
|
||||
|
||||
**Rationale**: One wrapper, one point of change, easy to disable globally. Doesn't pollute handler code.
|
||||
|
||||
**Alternatives rejected**:
|
||||
- *Per-handler injection calls* — duplicates the same five lines in six places; easy to forget for new tools.
|
||||
- *Server-level response interceptor in `chi`* — wrong layer; the MCP framing happens above HTTP.
|
||||
|
||||
## R10 — How to verify dream agent works on kubic
|
||||
|
||||
**Decision**: Watch `kubectl logs -n synapbus deploy/synapbus -f` for log lines tagged `component=consolidator-worker`. Manually trigger a job via admin CLI: `kubectl exec -n synapbus deploy/synapbus -- /synapbus --socket /data/synapbus.sock memory dream-run --owner algis --job reflection`. The worker logs: dispatched, harness-execute started, agent connected with token, tool calls, completion, audit rows.
|
||||
|
||||
**Rationale**: Reuses the existing admin CLI Unix-socket pattern.
|
||||
|
||||
**Alternatives rejected**:
|
||||
- *Wait for cron trigger* — slow feedback loop for verification.
|
||||
|
||||
## Open questions deferred to `/speckit.tasks` decomposition
|
||||
|
||||
None — all blockers resolved.
|
||||
@@ -0,0 +1,163 @@
|
||||
# Feature Specification: Proactive Memory & Dream Worker
|
||||
|
||||
**Feature Branch**: `020-proactive-memory-dream-worker`
|
||||
**Created**: 2026-05-11
|
||||
**Status**: Draft
|
||||
**Input**: User description: "Proactive owner-scoped agent memory: SynapBus pushes minimal relevant memory to agents on every MCP call, and a background 'dream' worker consolidates memory offline."
|
||||
|
||||
## User Scenarios & Testing *(mandatory)*
|
||||
|
||||
### User Story 1 - Agent gets relevant prior context without asking (Priority: P1)
|
||||
|
||||
An agent owned by a human starts working on a task — opening a new session, claiming inbound messages, replying to a teammate, or running a search. Today the agent has to remember to query a memory store first, so it usually doesn't, and it loses continuity with prior work. With this feature, SynapBus automatically attaches a small "what you should know right now" packet to every meaningful tool response. The agent gets the three to five most relevant prior memories scoped to its human owner — without spending a tool call to fetch them.
|
||||
|
||||
**Why this priority**: This is the headline complaint. Today the memory pool exists (about 500 entries in #open-brain) but is invisible per-task. Fixing this restores continuity across agent sessions immediately, even before any background processing exists.
|
||||
|
||||
**Independent Test**: Deploy injection only (no dream worker). Run a single agent against SynapBus; observe that responses from status-check, inbox-claim, message-send, and search tools include a context packet of relevant prior memories. Measure that the agent makes one fewer search call per task on average.
|
||||
|
||||
**Acceptance Scenarios**:
|
||||
|
||||
1. **Given** an agent owned by human H has 200 prior memories in the pool, **When** the agent calls a status-check tool, **Then** the response includes up to 5 relevant memories scoped to H, ranked by combined recency + similarity, sized within a configurable token budget.
|
||||
2. **Given** an agent sends a message containing the phrase "Kuzu graph DB", **When** the send completes, **Then** the response includes prior memories that mention Kuzu, graph databases, or related decisions made by the owner's agent fleet.
|
||||
3. **Given** agent A is owned by human H1 and agent B is owned by human H2, **When** both call the same tool with identical arguments, **Then** A and B receive disjoint memory packets — each only sees memories from its own owner's pool.
|
||||
4. **Given** the configured token budget is set to zero, **When** any agent calls any tool, **Then** no context packet is attached (feature is fully opt-out by configuration).
|
||||
5. **Given** the underlying retrieval finds no memory above the relevance floor, **When** an agent calls a tool, **Then** the response contains a context packet field that is empty or omitted entirely — no irrelevant filler.
|
||||
|
||||
---
|
||||
|
||||
### User Story 2 - Agent has a stable "who am I, what am I doing" anchor (Priority: P2)
|
||||
|
||||
Each agent in an owner's fleet has a small editable identity-and-context blob (a "core memory") that captures its role, current focus, and protocols it has been told to follow. This blob is always included in the agent's session-start response. When the agent's recent activity changes its focus, the dream worker rewrites this blob so the agent doesn't have to re-derive context from scratch each session.
|
||||
|
||||
**Why this priority**: Per Letta's published numbers, sleep-time rewrites of core memory yield meaningful accuracy gains and lower per-turn cost. Without P1 this is invisible; with P1, the core blob becomes the most-injected piece of memory.
|
||||
|
||||
**Independent Test**: Manually write a core-memory blob for one agent via the admin interface. Confirm that the agent receives this blob on every session-start tool call. Have the dream worker (or a human, simulating it) rewrite the blob. Confirm the next session-start call returns the new content.
|
||||
|
||||
**Acceptance Scenarios**:
|
||||
|
||||
1. **Given** human owner H sets a core memory of 800 characters for agent A, **When** A calls a session-start tool, **Then** the response includes the exact core-memory text alongside any relevance-scored memories.
|
||||
2. **Given** a core memory exceeds the configured size cap, **When** anyone attempts to save it, **Then** the save is rejected with a clear error referencing the cap.
|
||||
3. **Given** no core memory exists for agent A, **When** A calls a session-start tool, **Then** the response omits the core-memory field entirely — no errors, no empty placeholder.
|
||||
|
||||
---
|
||||
|
||||
### User Story 3 - Memory pool stays high-quality without human intervention (Priority: P2)
|
||||
|
||||
Without maintenance, the memory pool accumulates near-duplicates, stale facts, and contradictions, and retrieval quality decays. A background "dream" worker periodically dispatches consolidation work to a designated agent (which uses dedicated memory tools to do the actual reading and writing). The worker handles four kinds of jobs: writing higher-level abstractions when enough new memories have accumulated; rewriting each agent's core memory nightly; deduplicating near-identical memories and resolving contradictions; and linking related memories so retrieval surfaces the cluster, not just one item.
|
||||
|
||||
**Why this priority**: Memory hygiene compounds over weeks. Without it, P1 still works on day one but degrades. With it, retrieval quality holds steady and improves as the pool grows.
|
||||
|
||||
**Independent Test**: Seed the pool with 50 deliberately overlapping memories (5 paraphrases of the same fact, 3 contradictions, 20 unrelated items). Run the dream worker once. Verify: paraphrases are marked as duplicates with one canonical winner; contradictions are flagged with an explicit supersession; unrelated items are linked to relevant neighbors where applicable; an audit log records every action taken.
|
||||
|
||||
**Acceptance Scenarios**:
|
||||
|
||||
1. **Given** N new unprocessed memories have accumulated for owner H since the last reflection, where N exceeds the configured watermark, **When** the dream worker runs its periodic check, **Then** it dispatches a reflection job that produces 3–5 higher-level abstractions and writes them back as new memories tagged as reflections, with provenance links to the source memories.
|
||||
2. **Given** two memories with cosine similarity above 0.95 and the same factual claim, **When** the dedup job runs, **Then** one is kept as canonical and the other is soft-deleted with a duplicate-of link.
|
||||
3. **Given** memory A states "X is true" and memory B (newer) states "X is false", **When** the contradiction job runs, **Then** A is marked as superseded by B with a reason recorded, and retrieval no longer surfaces A by default.
|
||||
4. **Given** the dream worker is configured to dispatch jobs to a specific agent, **When** a trigger fires, **Then** the worker invokes that agent through the existing harness/spawn mechanism (not via a direct message), and the dispatched agent uses only the dedicated memory tools to do the work — it cannot bypass the audit log.
|
||||
5. **Given** an owner has zero new memories since the last consolidation, **When** the dream worker runs, **Then** no consolidation job is dispatched for that owner — no wasted cost.
|
||||
|
||||
---
|
||||
|
||||
### User Story 4 - Human owner can inspect and override memory operations (Priority: P3)
|
||||
|
||||
A human owner can see what's in their memory pool, what was injected on recent tool calls, and what the dream worker has done. They can correct mistakes: undo a bad dedup, restore a soft-deleted memory, mark a memory as protected so it can't be auto-consolidated, or pin/unpin a memory so it always (or never) gets injected.
|
||||
|
||||
**Why this priority**: Trust is a precondition for autonomy. Without visibility, owners won't enable the dream worker on real data. The injection layer (P1) can ship before this exists, but the dream worker shouldn't be enabled on production data until owners can audit it.
|
||||
|
||||
**Independent Test**: Use the Web UI to view all memories for one owner. Soft-delete a memory; verify it disappears from retrieval. Undo the delete; verify it comes back. Pin a memory; verify it shows up in every relevant injection regardless of similarity score.
|
||||
|
||||
**Acceptance Scenarios**:
|
||||
|
||||
1. **Given** an owner navigates to a Memory section in the Web UI, **When** they view it, **Then** they see all their memories with provenance, links, and consolidation history.
|
||||
2. **Given** the dream worker has marked memory M as duplicate, **When** the owner clicks "restore", **Then** M is reactivated and the duplicate-of link is removed; the audit log records the override.
|
||||
3. **Given** an owner pins memory M, **When** any of their agents calls an injection-eligible tool, **Then** M is included in the context packet even if its similarity score is below the relevance floor (subject to the token budget).
|
||||
|
||||
---
|
||||
|
||||
### Edge Cases
|
||||
|
||||
- **No embedding provider configured**: The system gracefully degrades to full-text-only retrieval for injection. The dream worker's link/dedup jobs that depend on vector similarity are skipped, with a warning logged. Manual reflections and supersessions still work.
|
||||
- **A dispatched consolidation job fails mid-flight**: The audit log records the failure with an error and the partial work done. The next periodic check retries the job. Repeated failures on the same job for the same owner pause future dispatches for that job type and surface an alert to the owner.
|
||||
- **A consolidation job runs longer than the wallclock budget**: The worker terminates the job at the budget, records what was completed, and resumes on the next run from where it stopped.
|
||||
- **Two consolidation jobs targeting the same memory arrive concurrently** (e.g. a manual owner edit during a nightly run): The owner edit wins; the consolidation job logs a "stale target" outcome and skips the conflicting action.
|
||||
- **Memory pool grows beyond a reasonable working-set size for one owner**: Retrieval still returns top-K within the relevance floor; consolidation jobs operate on a sliding window of recent + cluster-relevant memories rather than the full set.
|
||||
- **A pinned memory loses relevance to current activity**: The pin is honored regardless. Owners are responsible for unpinning. Document this clearly in the UI.
|
||||
- **An agent without an owner attempts to call an injection-eligible tool**: Injection is skipped (no owner to scope to). The tool itself still works.
|
||||
- **The injection token budget would be exceeded by a single memory**: Truncate the memory to fit; mark the truncation in the packet so the agent knows there's more.
|
||||
- **A consolidation tool is called by an agent outside the dream worker dispatch flow**: Reject — only the worker-dispatched session can call these tools, identified via a one-time dispatch token.
|
||||
|
||||
## Requirements *(mandatory)*
|
||||
|
||||
### Functional Requirements
|
||||
|
||||
#### Memory pool and ownership
|
||||
|
||||
- **FR-001**: The system MUST maintain a memory pool scoped to each human owner. Every memory MUST be attributable to the owner of the agent that produced it.
|
||||
- **FR-002**: Retrieval MUST filter strictly by the requesting agent's owner. An agent MUST NOT see any memory belonging to a different owner under any retrieval path.
|
||||
- **FR-003**: The system MUST recognize messages on designated "memory channels" (including `#open-brain` and channels explicitly flagged as memory channels) as eligible memories. Other channel messages are not eligible by default.
|
||||
- **FR-004**: The system MUST support soft-deletion (status flag) for memories, with the ability to restore a soft-deleted memory by an authorized human owner.
|
||||
- **FR-005**: The system MUST support temporal supersession: marking memory A as obsolete because memory B replaces it, with a recorded reason. Superseded memories are excluded from default retrieval but remain in the audit history.
|
||||
|
||||
#### Proactive context injection
|
||||
|
||||
- **FR-006**: The system MUST attach a relevant-context packet to the response of designated tool calls. The packet MUST include up to a configurable number of memories (default 3–5) and stay within a configurable token budget (default approximately 500 tokens).
|
||||
- **FR-007**: Retrieval for injection MUST use the existing hybrid (semantic + full-text) ranking with the existing relevance floor. Items below the floor MUST be excluded.
|
||||
- **FR-008**: The set of injection-eligible tools MUST include at minimum: status-check, inbox-claim, inbox-read, message-send, search, and the generic execute tool. Pure metadata tools (e.g. listing one's own agents) MAY be excluded.
|
||||
- **FR-009**: If the response context for a tool call contains a query, message body, or current channel topic, that text MUST be used as the retrieval query. Otherwise, recent owner activity (last 24h) is used as the implicit query.
|
||||
- **FR-010**: The system MUST support a per-agent "core memory" blob (size-capped, default ~1KB). When present, it MUST be included in every status-check response in addition to retrieval-based memories.
|
||||
- **FR-011**: Owners MUST be able to pin and unpin individual memories. Pinned memories MUST be included in every injection-eligible tool response for that owner's agents, subject only to the token budget — bypassing the relevance floor.
|
||||
- **FR-012**: The injection feature MUST be disable-able per owner and globally via configuration. With injection disabled, tool responses match their current shape exactly (no extra field).
|
||||
|
||||
#### Dream worker (background consolidation)
|
||||
|
||||
- **FR-013**: A background worker MUST periodically check each owner's memory pool against trigger conditions for four consolidation job types: reflection-on-watermark, sleep-time core-memory rewrite, deduplication-and-contradiction-resolution, and link generation.
|
||||
- **FR-014**: When a trigger fires, the worker MUST dispatch the job to a configured consolidation agent via the existing harness/spawn mechanism. The worker MUST NOT use direct messaging or any path that itself triggers reactive runs.
|
||||
- **FR-015**: The dispatched consolidation agent MUST only use the dedicated memory-consolidation tool surface (list, write-reflection, rewrite-core, mark-duplicate, supersede, add-link). Every call MUST be recorded in an immutable audit log scoped to the owner.
|
||||
- **FR-016**: The worker MUST honor a per-owner wallclock budget per consolidation pass (default 10 minutes). When the budget is exceeded, in-progress work is recorded and the pass terminates.
|
||||
- **FR-017**: The system MUST automatically derive lightweight reference links (mention, reply-to, channel co-occurrence) between memories without invoking the consolidation agent. These links MUST be available to retrieval alongside agent-generated links.
|
||||
- **FR-018**: Reflection memories MUST be tagged so they are distinguishable from raw memories. Source-memory provenance MUST be recorded.
|
||||
- **FR-019**: The dream worker MUST be disable-able per owner and globally via configuration. The injection layer (P1) MUST function correctly when the dream worker is disabled.
|
||||
- **FR-020**: A consolidation agent's dispatch session MUST be identified by a one-time dispatch token. Memory-consolidation tools MUST reject calls from sessions without a valid token.
|
||||
|
||||
#### Human-owner audit and override
|
||||
|
||||
- **FR-021**: Human owners MUST be able to view all memories in their pool, including soft-deleted ones, via the Web UI.
|
||||
- **FR-022**: For each memory, the UI MUST surface: original source, provenance (parent reflections / supersessions / duplicates), links to related memories, current status, and pinning state.
|
||||
- **FR-023**: Owners MUST be able to: pin/unpin a memory, soft-delete or restore a memory, undo a consolidation action recorded in the audit log, and mark a memory as "protected" so the dream worker cannot soft-delete or supersede it.
|
||||
- **FR-024**: The audit log MUST be searchable by owner, agent, action type, and time range.
|
||||
- **FR-025**: Recent injections (what was attached to which tool call) MUST be inspectable for at least the last 24 hours so owners can debug "why did my agent know that?"
|
||||
|
||||
### Key Entities *(include if feature involves data)*
|
||||
|
||||
- **Memory**: A retained piece of context belonging to an owner. Has source (origin message or reflection), status (active/soft-deleted/superseded), embedding (when available), provenance (parents), and a pinned/protected flag set.
|
||||
- **Core memory blob**: A small, owner+agent-scoped editable text. One per (owner, agent) pair. Replaces itself wholesale on rewrite.
|
||||
- **Memory link**: A directed typed edge between two memories. Type ∈ {refines, contradicts, examples, related, duplicate-of, superseded-by, mention, reply-to, channel-cooccurrence}.
|
||||
- **Consolidation job**: A scheduled or watermark-triggered unit of dream work. Has owner, type, trigger reason, dispatch token, status, start/end time, summary of actions taken, and outcome (success/partial/failed).
|
||||
- **Audit entry**: An immutable record of every memory mutation — dedup, supersession, link addition, core rewrite, pin/unpin, restore — including actor (agent or human), dispatch token if any, and before/after diff.
|
||||
- **Dispatch token**: A short-lived single-use credential that authorizes a consolidation agent to call memory-consolidation tools for a specific owner. Bound to one job.
|
||||
|
||||
## Assumptions
|
||||
|
||||
- **Default eligible memory channels**: `#open-brain` plus any channel explicitly flagged via channel metadata as a memory channel. Owners can mark additional channels as memory channels through the admin interface.
|
||||
- **Default trigger watermark for reflection**: 20 new unprocessed memories per owner since the last reflection job.
|
||||
- **Default deep-pass schedule**: 03:00 UTC daily for sleep-time core-memory rewrite. Owners can shift this per-owner.
|
||||
- **Default consolidation agent**: Claude Code, invoked through the existing agent-spawn / harness path. Other agents are configurable per-owner.
|
||||
- **Retention**: Memories are NOT subject to the standard message retention policy (currently 12 months). They are retained until explicitly deleted or superseded, regardless of age.
|
||||
- **Token estimation**: Token counts for the injection budget are computed by a simple character-based heuristic (≈ 4 characters per token) — exact tokenization is not required for budget enforcement.
|
||||
- **Concurrent dispatch**: At most one consolidation job per (owner, job type) runs at a time. Jobs for different owners can run concurrently up to a global worker pool limit (default 4).
|
||||
- **Embedding provider availability**: When no embedding provider is configured, the system degrades gracefully — injection uses full-text-only retrieval; dedup and link-generation jobs that depend on vector similarity are skipped with a logged warning.
|
||||
|
||||
## Success Criteria *(mandatory)*
|
||||
|
||||
### Measurable Outcomes
|
||||
|
||||
- **SC-001**: For every injection-eligible tool call where the owner's memory pool contains at least one item, the response includes a non-empty relevant-context packet ≥ 95% of the time (the 5% accounts for cases where the relevance floor rejects everything).
|
||||
- **SC-002**: Median added latency per tool call from injection stays below 50 milliseconds, measured end-to-end including retrieval and packet assembly.
|
||||
- **SC-003**: When measured on a benchmark dogfood run of the searcher swarm before and after this feature, the average number of explicit memory-search tool calls per agent task drops by at least 40%, without a regression in task completion rate.
|
||||
- **SC-004**: Agents complete a "catch up on prior context" benchmark task using at least 30% fewer total tokens (system + tool inputs) with this feature enabled versus disabled.
|
||||
- **SC-005**: A full owner-level dream-worker pass completes within the configured wallclock budget (default 10 minutes per owner) on a memory pool of up to 5,000 memories, on the reference deployment.
|
||||
- **SC-006**: After one week of dream-worker operation on a seeded test pool (500 memories with deliberate duplicates and contradictions), the duplicate rate (memories with cosine similarity > 0.95 and overlapping content) drops below 5%, down from a seeded baseline of 20%.
|
||||
- **SC-007**: Owners can audit every memory mutation made in the past 30 days through the Web UI with response time under 2 seconds.
|
||||
- **SC-008**: Zero cross-owner memory leakage detected in adversarial testing: agents owned by H1 cannot retrieve, view, or be injected with any memory created by H2's agents.
|
||||
- **SC-009**: When injection is disabled via configuration, response payload sizes for the affected tools are within 1% of pre-feature baseline.
|
||||
@@ -0,0 +1,217 @@
|
||||
---
|
||||
description: "Task list — proactive memory + dream worker"
|
||||
---
|
||||
|
||||
# Tasks: Proactive Memory & Dream Worker
|
||||
|
||||
**Input**: Design documents from `/specs/020-proactive-memory-dream-worker/`
|
||||
**Prerequisites**: plan.md ✓, spec.md ✓, research.md ✓, data-model.md ✓, contracts/ ✓, quickstart.md ✓
|
||||
**Tests**: Tests are REQUESTED (test-before-impl pairs marked explicitly).
|
||||
|
||||
## Format: `[ID] [P?] [Story] Description`
|
||||
|
||||
- **[P]**: parallelizable — different files, no in-flight dependencies
|
||||
- **[Story]**: `[US1]`, `[US2]`, `[US3]` — maps to spec user stories
|
||||
- Setup / Foundational / Polish tasks carry no `[Story]` label
|
||||
|
||||
## Scope for this round
|
||||
|
||||
Foundational → US1 (P1 injection) → US2 (P2 core memory) → US3 (P2 dream worker). **US4 (P3 audit UI) is deferred** — captured in plan.md Constitution Check as Phase 2 of this feature.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Setup
|
||||
|
||||
Branch and spec artifacts already exist. Only verifying scaffolding.
|
||||
|
||||
- [ ] T001 Verify branch `020-proactive-memory-dream-worker` is checked out and all spec artifacts (spec.md, plan.md, research.md, data-model.md, contracts/, quickstart.md) are present at `/Users/user/repos/synapbus/specs/020-proactive-memory-dream-worker/`. No code changes.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Foundational (Blocks all user stories)
|
||||
|
||||
**⚠️ CRITICAL**: No `[US*]` task may start until this phase is complete.
|
||||
|
||||
- [ ] T002 Write SQL migration in `/Users/user/repos/synapbus/internal/storage/schema/028_memory_consolidation.sql` per data-model.md (6 tables: `memory_core`, `memory_links`, `memory_consolidation_jobs`, `memory_pins`, `memory_dispatch_tokens`, `memory_injections` + `memory_status` view). Idempotent `CREATE TABLE IF NOT EXISTS`, all indexes, the partial-unique index on `memory_consolidation_jobs`, and the CHECK constraints on enums.
|
||||
- [ ] T003 [P] Migration smoke test in `/Users/user/repos/synapbus/internal/storage/migration_test.go` — verify migration 028 applies cleanly on a fresh in-memory DB and all 6 tables + 1 view are queryable.
|
||||
- [ ] T004 [P] Add feature-flag env parsing in `/Users/user/repos/synapbus/internal/messaging/options.go` (or appropriate config struct): `SYNAPBUS_INJECTION_ENABLED` (default 0), `SYNAPBUS_INJECTION_BUDGET_TOKENS` (default 500), `SYNAPBUS_INJECTION_MAX_ITEMS` (default 5), `SYNAPBUS_INJECTION_MIN_SCORE` (default 0.25), `SYNAPBUS_CORE_MEMORY_MAX_BYTES` (default 2048), `SYNAPBUS_DREAM_ENABLED` (default 0), `SYNAPBUS_DREAM_INTERVAL` (default 1h), `SYNAPBUS_DREAM_DEEP_CRON` (default `0 3 * * *`), `SYNAPBUS_DREAM_MAX_CONCURRENT` (default 4), `SYNAPBUS_DREAM_WALLCLOCK_BUDGET` (default 10m), `SYNAPBUS_DREAM_WATERMARK` (default 20), `SYNAPBUS_DREAM_AGENT` (default `claude-code`).
|
||||
- [ ] T005 [P] Owner resolver helper in `/Users/user/repos/synapbus/internal/agents/owner.go`: `OwnerFor(ctx, db, agentName) (string, error)` that looks up `agents.owner_id`. Returns sentinel error if agent not found or unowned.
|
||||
- [ ] T006 [P] Dispatch-token store in `/Users/user/repos/synapbus/internal/messaging/dispatch_tokens.go`: `Issue(ctx, ownerID, jobID) (token, expiresAt, err)`, `Validate(ctx, token, ownerID, jobID) (ok, err)`, `Revoke(ctx, token)`. Uses `crypto/rand` for 32-byte tokens; 15m TTL.
|
||||
- [ ] T007 [P] Dispatch-token tests in `/Users/user/repos/synapbus/internal/messaging/dispatch_tokens_test.go`: issue→validate success path; reject expired; reject owner mismatch; reject wrong-job; reject revoked.
|
||||
- [ ] T008 [P] Memory-channel discovery in `/Users/user/repos/synapbus/internal/messaging/memory_channels.go`: `IsMemoryChannel(ch *Channel) bool` (true when name matches `open-brain` or `reflections-*`, OR `metadata.is_memory == true`); `MemoryChannelIDs(ctx) ([]int64, error)`.
|
||||
|
||||
**Checkpoint**: Foundation ready — US1/US2/US3 may proceed in parallel.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: User Story 1 — Proactive Injection (Priority: P1) 🎯 MVP
|
||||
|
||||
**Goal**: Every injection-eligible MCP tool response carries a `relevant_context` packet (owner-scoped, hybrid-ranked, token-budgeted).
|
||||
|
||||
**Independent Test**: With `SYNAPBUS_INJECTION_ENABLED=1` and seeded memories, calling `my_status` returns `relevant_context.memories` populated. Toggling `SYNAPBUS_INJECTION_ENABLED=0` removes the field entirely from the response.
|
||||
|
||||
### Tests for User Story 1 (write first, ensure they FAIL)
|
||||
|
||||
- [ ] T009 [P] [US1] Contract test for relevant_context shape in `/Users/user/repos/synapbus/internal/mcp/injection_wrap_test.go`: build a fake handler that returns canned JSON; assert wrapper appends `relevant_context` with the expected fields per `contracts/mcp-injection.md`; assert wrapper omits the field when memories slice is empty AND no core_memory set.
|
||||
- [ ] T010 [P] [US1] Retrieval test in `/Users/user/repos/synapbus/internal/search/injection_test.go`: assert `BuildContextPacket` enforces owner scoping (memory from another owner is excluded), respects the score floor (drops items below 0.25), respects the token budget (truncates last item that overflows), and marks `truncated=true` on the truncated item.
|
||||
- [ ] T011 [P] [US1] Cross-owner adversarial test in `/Users/user/repos/synapbus/internal/mcp/injection_e2e_test.go`: seed memories for owners H1 and H2; call any wrapped tool as an H1 agent and an H2 agent; assert their `relevant_context.memories` are disjoint (SC-008).
|
||||
- [ ] T012 [P] [US1] Audit-ring test in `/Users/user/repos/synapbus/internal/messaging/memory_injections_test.go`: write 50 injection rows; assert cleanup deletes those older than 24h and keeps younger ones; assert ring is owner-scoped on read.
|
||||
|
||||
### Implementation for User Story 1
|
||||
|
||||
- [ ] T013 [P] [US1] Audit ring writer in `/Users/user/repos/synapbus/internal/messaging/memory_injections.go`: `Record(ctx, row)` writes one row; `Cleanup(ctx, olderThan time.Duration)` deletes old rows; `ListRecent(ctx, ownerID, limit)` reads back for debug.
|
||||
- [ ] T014 [P] [US1] Build-context-packet implementation in `/Users/user/repos/synapbus/internal/search/injection.go`: `BuildContextPacket(ctx, agent *agents.Agent, query string, opts InjectionOpts) (*ContextPacket, error)`. Calls existing `search.Service.Search()` with `owner_id` filter, applies pin overlay (always include pinned), greedy-fills under token budget (char/4), returns `ContextPacket{Memories []MemoryItem, CoreMemory string, PacketChars int, RetrievalQuery string, SearchMode string}`. Accepts a `CoreMemoryProvider` interface so US2 can plug in without modifying this file.
|
||||
- [ ] T015 [P] [US1] MCP injection middleware in `/Users/user/repos/synapbus/internal/mcp/injection_wrap.go`: `WrapInjection(handler ToolHandler, cfg WrapConfig) ToolHandler`. Runs inner handler, on JSON-shaped success result calls `BuildContextPacket`, marshals merged JSON, calls `memory_injections.Record`. Skip on non-JSON result.
|
||||
- [ ] T016 [US1] Register middleware against eligible tools in `/Users/user/repos/synapbus/internal/mcp/tools_hybrid.go`. Wrap `my_status`, `claim_messages`, `read_inbox`, `send_message`, `search` / `search_messages`, `execute`, `read_channel`. For each, supply the right `query_source` (recent activity / claimed bodies / message body / query string / args / channel topic). Depends on T014, T015.
|
||||
- [ ] T017 [US1] Wire injection cleanup into stalemate worker tick (or new cleanup task in `consolidator.go` if it lands first) — call `memory_injections.Cleanup(ctx, 24*time.Hour)` once per hour. File: `/Users/user/repos/synapbus/internal/messaging/stalemate.go` (extend existing tick).
|
||||
|
||||
**Checkpoint**: US1 functional — seed → call `my_status` → see `relevant_context`. MVP ready.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: User Story 2 — Per-Agent Core Memory (Priority: P2)
|
||||
|
||||
**Goal**: Each `(owner, agent)` has a small editable blob always included in session-start responses.
|
||||
|
||||
**Independent Test**: Set a core memory via admin CLI; call `my_status` as that agent; assert `relevant_context.core_memory` matches the set blob. Set a blob over the size cap; assert rejection with `core_memory_too_large`.
|
||||
|
||||
### Tests for User Story 2 (write first)
|
||||
|
||||
- [ ] T018 [P] [US2] Core-memory store tests in `/Users/user/repos/synapbus/internal/messaging/memory_core_test.go`: Get/Set/Delete round-trip; reject over-size on Set; replace-wholesale semantic (no merge); owner scoping (cannot read another owner's blob).
|
||||
|
||||
### Implementation for User Story 2
|
||||
|
||||
- [ ] T019 [P] [US2] Core-memory store in `/Users/user/repos/synapbus/internal/messaging/memory_core.go`: `GetCore(ctx, ownerID, agentName) (string, error)`, `SetCore(ctx, ownerID, agentName, blob, updatedBy) error` (enforces `SYNAPBUS_CORE_MEMORY_MAX_BYTES`), `DeleteCore(ctx, ownerID, agentName)`.
|
||||
- [ ] T020 [US2] Implement `CoreMemoryProvider` adapter that `BuildContextPacket` calls — a one-liner wrapper around `messaging.GetCore` registered in `cmd/synapbus/main.go` wiring. Update injection wrapper config so session-start tools (i.e. `my_status`) get a provider; other tools get nil.
|
||||
- [ ] T021 [P] [US2] Admin CLI command in `/Users/user/repos/synapbus/cmd/synapbus/main.go` (add to existing cobra tree under `memory` subcommand): `synapbus memory core get --owner --agent`, `set --owner --agent --blob-file`, `delete --owner --agent`. Uses the existing socket-RPC pattern.
|
||||
- [ ] T022 [P] [US2] REST endpoint for the Web UI placeholder in `/Users/user/repos/synapbus/internal/api/memory_core.go`: `GET/PUT/DELETE /api/owner/{ownerID}/agents/{agentName}/core-memory` guarded by session auth (owner only).
|
||||
|
||||
**Checkpoint**: US2 functional — admin sets a core, agent receives it on session start.
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: User Story 3 — Dream Worker (Priority: P2)
|
||||
|
||||
**Goal**: A background worker dispatches consolidation jobs to a Claude Code agent through `harness.Harness.Execute`. The agent uses 6 new MCP tools, gated by dispatch token, fully audited.
|
||||
|
||||
**Independent Test**: Seed pool with deliberate duplicates + contradictions, force a manual dream-run via admin CLI, observe `memory_consolidation_jobs.actions` populated and the `memory_status` view reflecting the changes. Confirm logs include `component=consolidator-worker` lines through the full dispatch → succeeded lifecycle.
|
||||
|
||||
### Tests for User Story 3 (write first)
|
||||
|
||||
- [ ] T023 [P] [US3] Memory-link store tests in `/Users/user/repos/synapbus/internal/messaging/memory_links_test.go`: insert each relation type; reject reserved types from `memory_add_link` caller path; assert owner-scoped queries.
|
||||
- [ ] T024 [P] [US3] Memory-pin store tests in `/Users/user/repos/synapbus/internal/messaging/memory_pins_test.go`: pin/unpin; pinned memories surface in `BuildContextPacket` even below the score floor.
|
||||
- [ ] T025 [P] [US3] Consolidator-worker tests in `/Users/user/repos/synapbus/internal/messaging/consolidator_test.go` using a mocked `harness.Harness`: assert watermark trigger fires only when N≥threshold; assert cron-deep-pass fires at the configured wallclock; assert at-most-one-in-flight per (owner, job_type) via the partial-unique-index path; assert wallclock budget terminates a runaway job with `partial` status; assert no system-DM is ever sent.
|
||||
- [ ] T026 [P] [US3] MCP-memory-tools tests in `/Users/user/repos/synapbus/internal/mcp/memory_tools_test.go`: each of the 6 tools — happy path + every documented error code (`dispatch_token_missing`, `dispatch_token_expired`, `dispatch_token_owner_mismatch`, `not_same_owner`, `core_memory_too_large`, `relation_type_reserved`, `source_not_found`).
|
||||
- [ ] T027 [P] [US3] memory_status view test in `/Users/user/repos/synapbus/internal/storage/memory_status_view_test.go`: insert `memory_consolidation_jobs.actions` JSON for dedup + supersede; query the view; assert correct derivations of `status`, `superseded_by`, `soft_deleted_at`.
|
||||
|
||||
### Implementation for User Story 3
|
||||
|
||||
- [ ] T028 [P] [US3] Link store in `/Users/user/repos/synapbus/internal/messaging/memory_links.go`: `AddLink(ctx, src, dst, relType, ownerID, createdBy, metadata)`, `ListLinks(ctx, msgID)`, `LinksForOwner(ctx, ownerID, types[], limit)`. Reject reserved types when `createdBy` starts with `agent:` (auto-types still allowed when `createdBy` starts with `auto:`).
|
||||
- [ ] T029 [P] [US3] Pin store in `/Users/user/repos/synapbus/internal/messaging/memory_pins.go`: `Pin(ctx, ownerID, msgID, pinnedBy, note)`, `Unpin(ctx, ownerID, msgID)`, `ListPins(ctx, ownerID)`. Update `BuildContextPacket` overlay path to include pins.
|
||||
- [ ] T030 [P] [US3] memory_status query helpers in `/Users/user/repos/synapbus/internal/messaging/memory_status.go`: `StatusFor(ctx, msgIDs []int64) (map[int64]MemoryStatus, error)`. Used by retrieval (joins to filter out non-active).
|
||||
- [ ] T031 [US3] Plumb status filter into `BuildContextPacket` (modifies `/Users/user/repos/synapbus/internal/search/injection.go`). Depends on T014 and T030.
|
||||
- [ ] T032 [P] [US3] Six MCP memory tools in `/Users/user/repos/synapbus/internal/mcp/memory_tools.go`: `memory_list_unprocessed`, `memory_write_reflection`, `memory_rewrite_core`, `memory_mark_duplicate`, `memory_supersede`, `memory_add_link`. Each: validates dispatch token via `messaging.Validate`, calls the underlying store, appends to `memory_consolidation_jobs.actions` JSON via `Jobs.AppendAction(ctx, jobID, action)`.
|
||||
- [ ] T033 [US3] Tool registration gate in `/Users/user/repos/synapbus/internal/mcp/server.go`: register memory tools only when `SYNAPBUS_DREAM_ENABLED=1`. Depends on T032.
|
||||
- [ ] T034 [P] [US3] Consolidation-jobs store in `/Users/user/repos/synapbus/internal/messaging/consolidation_jobs.go`: `Create(ctx, ownerID, jobType, trigger) (jobID, error)`, `Dispatch(ctx, jobID, harnessRunID, token)`, `Lease(ctx, jobID, until)`, `AppendAction(ctx, jobID, action)`, `Complete(ctx, jobID, status, summary, error)`, `ListRecent(ctx, ownerID, limit)`.
|
||||
- [ ] T035 [P] [US3] Auto-link emitter in `/Users/user/repos/synapbus/internal/messaging/auto_links.go`: on `OnMessageCreated` (callable from existing `MessagingService`), insert `mention`, `reply_to`, `channel_cooccurrence` links automatically. Configurable, no LLM. Wire from `service.go` post-insert hook (one-line addition).
|
||||
- [ ] T036 [P] [US3] ConsolidatorWorker in `/Users/user/repos/synapbus/internal/messaging/consolidator.go`: ticker pattern modeled on `StalemateWorker`. Each tick: per owner, evaluate watermark for reflection / link-gen / dedup; cron-evaluate sleep-time-rewrite. When trigger fires, `Create` job → `Issue` token → `Execute` via `harness.Harness` with `Env={SYNAPBUS_DISPATCH_TOKEN, SYNAPBUS_CONSOLIDATION_JOB_ID, SYNAPBUS_JOB_TYPE, SYNAPBUS_OWNER_ID}` → on `ExecResult`, call `Complete`. Honors `SYNAPBUS_DREAM_MAX_CONCURRENT` global semaphore. Hourly call to `memory_injections.Cleanup(ctx, 24h)`. Wallclock-budget on `Execute` via `Budget.MaxWallClock`.
|
||||
- [ ] T037 [P] [US3] Dream-agent prompts in `/Users/user/repos/synapbus/internal/messaging/consolidator_prompts.go`: one short prompt string per job type (`reflection`, `core_rewrite`, `dedup_contradiction`, `link_gen`) — passed to `harness.Execute` as the task body. Each prompt: explains the job, instructs to use only `memory_*` tools, gives the dispatch token reference.
|
||||
- [ ] T038 [US3] Wire `ConsolidatorWorker` lifecycle into `/Users/user/repos/synapbus/cmd/synapbus/main.go`: start when `SYNAPBUS_DREAM_ENABLED=1` after DB + messaging service ready; stop on shutdown. Depends on T036.
|
||||
- [ ] T039 [P] [US3] Admin CLI `synapbus memory dream-run --owner --job` in `/Users/user/repos/synapbus/cmd/synapbus/main.go` (extend the `memory` subtree from T021): forces a single job dispatch bypassing the trigger check. Useful for kubic verification.
|
||||
|
||||
**Checkpoint**: All three user stories functional. Memory pool is owner-scoped, injection works, dream worker dispatches and audits consolidation jobs end-to-end.
|
||||
|
||||
---
|
||||
|
||||
## Phase 6: Polish & Cross-Cutting
|
||||
|
||||
- [ ] T040 [P] Run `go vet ./...` and `gofmt -l .` — fix any lint findings.
|
||||
- [ ] T041 Run `make test` — full suite must pass; fix any regressions.
|
||||
- [ ] T042 Cross-compile linux/amd64 binary: `GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o ./synapbus-linux-amd64 ./cmd/synapbus`. Confirms zero-CGO constitution constraint.
|
||||
- [ ] T043 Build kubic container image: `docker build --platform=linux/amd64 -t kubic.home.arpa:32000/synapbus:020-proactive-memory .` then push.
|
||||
- [ ] T044 Deploy to kubic: `kubectl -n synapbus set image deploy/synapbus synapbus=kubic.home.arpa:32000/synapbus:020-proactive-memory` then `kubectl set env deploy/synapbus SYNAPBUS_INJECTION_ENABLED=1 SYNAPBUS_DREAM_ENABLED=1`.
|
||||
- [ ] T045 Run quickstart.md "kubic: verify dream agent" section: tail logs, force a manual reflection job, verify `memory_consolidation_jobs` row reaches `status=succeeded` or `partial`. Capture the log excerpt as evidence.
|
||||
- [ ] T046 Update `/Users/user/repos/synapbus/CLAUDE.md` "Active Technologies" with migration 028 entry (already partially done by speckit script).
|
||||
- [ ] T047 Post completion summary to SynapBus `#my-agents-algis` channel (once auth re-established) — what shipped + the kubic log evidence.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Execution Order
|
||||
|
||||
### Phase order
|
||||
|
||||
1. Setup (T001) — single check, ~no work.
|
||||
2. Foundational (T002–T008) — blocks everything.
|
||||
3. Stories run in priority order but US1/US2/US3 can proceed in parallel after Foundational completes:
|
||||
- US1 (T009–T017) — MVP. Ship-able alone with `SYNAPBUS_INJECTION_ENABLED=1`.
|
||||
- US2 (T018–T022) — depends only on `T014` (BuildContextPacket exists) from US1 for the integration point; the rest is parallel.
|
||||
- US3 (T023–T039) — depends on Foundational. Independent of US1/US2 except `T031` (status filter into injection) which extends `T014`.
|
||||
4. Polish (T040–T047).
|
||||
|
||||
### Within each story
|
||||
|
||||
- Tests first; fail; then impl.
|
||||
- Stores before MCP tool wiring before main-wiring.
|
||||
|
||||
### File-level conflicts (do not parallel)
|
||||
|
||||
| Tasks | Shared file | Order |
|
||||
|-------|-------------|-------|
|
||||
| T014 ↔ T031 | `internal/search/injection.go` | T014 first |
|
||||
| T015 ↔ T016 | `internal/mcp/injection_wrap.go` + `tools_hybrid.go` | T015 first |
|
||||
| T032 ↔ T033 | `internal/mcp/memory_tools.go` + `server.go` | T032 first |
|
||||
| T036 ↔ T038 | `cmd/synapbus/main.go` | T036 first (creates worker), T038 wires it |
|
||||
| T021 ↔ T039 | `cmd/synapbus/main.go` (memory CLI subtree) | T021 first (creates subtree), T039 adds command |
|
||||
| T017 ↔ T036 | `internal/messaging/stalemate.go` OR consolidator | one of them owns the hourly cleanup |
|
||||
|
||||
### Parallel opportunities
|
||||
|
||||
After T002 lands, T003/T004/T005/T006/T008 run in parallel. After Foundational completes, all `[P]` tasks within a story run in parallel — for US1 that's T009/T010/T011/T012/T013/T014/T015 simultaneously; for US3 that's T023/T024/T025/T026/T027 (tests) then T028/T029/T030/T032/T034/T035/T036/T037 (impl) in parallel.
|
||||
|
||||
---
|
||||
|
||||
## Parallel-Subagent Cluster Plan
|
||||
|
||||
The next step is to dispatch subagents in parallel. Suggested clusters (each cluster = one subagent):
|
||||
|
||||
- **Cluster F (Foundational)**: T002, T003, T004, T005, T006, T007, T008.
|
||||
- **Cluster I1 (US1 retrieval + middleware)**: T009, T010, T013, T014, T015, T016 — owns `injection.go`, `injection_wrap.go`, `tools_hybrid.go`, `memory_injections.go`.
|
||||
- **Cluster I2 (US1 cross-owner safety + cleanup)**: T011, T012, T017.
|
||||
- **Cluster C (US2 core memory)**: T018, T019, T020, T021, T022.
|
||||
- **Cluster D1 (US3 stores + view)**: T023, T024, T027, T028, T029, T030.
|
||||
- **Cluster D2 (US3 MCP tools + jobs)**: T026, T032, T033, T034, T037.
|
||||
- **Cluster D3 (US3 worker + wiring + CLI)**: T025, T035, T036, T038, T039.
|
||||
|
||||
Clusters that share files are serialized; clusters in different file sets are parallel.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Strategy
|
||||
|
||||
### MVP increments
|
||||
|
||||
1. Foundational → US1 → STOP. Ship `SYNAPBUS_INJECTION_ENABLED=1` to dev. This alone delivers the headline value ("agents get context without asking").
|
||||
2. Add US2. Per-agent core memory active; session-start tools include the blob.
|
||||
3. Add US3. Dream worker dispatches consolidation jobs; verify on kubic.
|
||||
|
||||
### Stop conditions before kubic deploy
|
||||
|
||||
- All `go test ./...` green.
|
||||
- `go vet ./...` clean.
|
||||
- Linux/amd64 binary cross-compiles with `CGO_ENABLED=0`.
|
||||
- A local smoke test of US1 → US2 → US3 quickstart steps passes.
|
||||
|
||||
### Kubic deploy verification
|
||||
|
||||
- Logs show `component=consolidator-worker` startup line.
|
||||
- Manual `dream-run` produces a `memory_consolidation_jobs` row with `status` ∈ {`succeeded`, `partial`}.
|
||||
- An adversarial cross-owner injection test on the deployed instance returns no leakage.
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
- `[P]` = different files AND no in-flight dependency. Conservative parallel marks where doubt.
|
||||
- Auto-types `mention` / `reply_to` / `channel_cooccurrence` are created automatically by T035, NOT through `memory_add_link` (which rejects them — see contracts/mcp-memory-tools.md). The dream-agent only writes the four semantic types.
|
||||
- The harness/dispatch path is the contractual non-system-DM route. Any deviation (e.g. tempted to "just send a DM to dream-agent") violates `feedback_system_dm_no_trigger.md` and recreates the cascading-stalemate bug.
|
||||
- US4 (audit UI) is deferred. Until it ships, owners audit via `sqlite3` or the future REST endpoint from T022.
|
||||
Reference in New Issue
Block a user