diff --git a/CLAUDE.md b/CLAUDE.md index 9a6463c..338df23 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -112,6 +112,8 @@ make lint # Run linters - SQLite via modernc.org/sqlite — new migration 015_reactive_triggers.sql (014-reactive-agent-triggers) - Go 1.25+ (per `go.mod`), no CGO, cross-compiled for `linux/amd64` + `darwin/arm64` + `mark3labs/mcp-go` (MCP tools), `go-chi/chi` (HTTP), `spf13/cobra` (CLI), `modernc.org/sqlite` (storage), `golang.org/x/crypto/nacl/secretbox` (secret encryption — pure Go, already in ecosystem), existing `SherClockHolmes/webpush-go`, `TFMV/hnsw`, `ory/fosite` (018-dynamic-agent-spawning) - SQLite via `modernc.org/sqlite` — five new migrations (`021_goals_tasks.sql`, `022_agent_proposals.sql`, `023_agent_trust_model.sql`, `024_secrets.sql`, `025_harness_runs_task_id.sql`); existing content-addressable attachment store reused for encrypted secret blobs (018-dynamic-agent-spawning) +- Go 1.25+ (per go.mod) + `mark3labs/mcp-go` (MCP), `go-chi/chi` (HTTP), `spf13/cobra` (CLI), `modernc.org/sqlite` (storage), `jmoiron/sqlx` (query helpers), `cloudflare/tableflip` (graceful restart — NEW), `gopkg.in/yaml.v3` (config), `xeipuuv/gojsonschema` (config-schema validation) (019-plugin-system) +- SQLite via `modernc.org/sqlite` (pure Go, zero CGO). New core table `plugin_migrations`. Plugin tables namespaced `plugin__*`. (019-plugin-system) ## Recent Changes - 002-mcp-auth-ux-polish: Added Go 1.23+ + ory/fosite (OAuth 2.1), mark3labs/mcp-go (MCP server), go-chi/chi (HTTP), Svelte 5 + Tailwind (Web UI) diff --git a/specs/019-plugin-system/contracts/host.md b/specs/019-plugin-system/contracts/host.md new file mode 100644 index 0000000..32fc3bd --- /dev/null +++ b/specs/019-plugin-system/contracts/host.md @@ -0,0 +1,78 @@ +# Contract — `Host` struct + +```go +package plugin + +import ( + "context" + "database/sql" + "encoding/json" + "log/slog" + + "github.com/jmoiron/sqlx" + "github.com/prometheus/client_golang/prometheus" + "go.opentelemetry.io/otel/trace" + + "github.com/synapbus/synapbus/internal/attachments" + "github.com/synapbus/synapbus/internal/channels" + "github.com/synapbus/synapbus/internal/eventbus" + "github.com/synapbus/synapbus/internal/messaging" + "github.com/synapbus/synapbus/internal/search" + "github.com/synapbus/synapbus/internal/secrets" + "github.com/synapbus/synapbus/internal/users" +) + +// Host is the bundle of core services handed to a plugin at Init. +// Plugins MUST treat it as read-only and MUST NOT cache sub-handles +// across process lifetimes. +type Host struct { + Logger *slog.Logger // pre-tagged plugin= + DB *sqlx.DB // shared core handle + Messaging messaging.API + Channels channels.API + Attachments attachments.Store + Search search.Index + Secrets secrets.Scoped // scoped to calling plugin + Events eventbus.Bus + Config json.RawMessage // plugins[] YAML sub-tree + DataDir string // <--data>/plugins// + Tracer trace.Tracer + Metrics prometheus.Registerer + DefaultOwner *users.User // for failure notifications + baseURL string // unexported; plugins read via BaseURL() +} + +// BaseURL returns the public base URL of this instance, for building +// absolute links in panel HTML. +func (h Host) BaseURL() string { return h.baseURL } + +// ExecTx runs a function inside a database transaction against the shared +// handle. Plugins should use this for multi-statement writes. +func (h Host) ExecTx(ctx context.Context, fn func(*sqlx.Tx) error) error { … } +``` + +## Security / scope invariants + +- `Host.Secrets` returns only secrets that were written with the caller's plugin name. Calling `.Get("X")` for a secret owned by another plugin returns `secrets.ErrNotFound`, same as missing. +- `Host.DB` gives full SQL access — but tables read/written are expected to be `plugin__*`. A static linter (part of this feature) flags unqualified reads from core tables. +- `Host.DataDir` is guaranteed to exist, to be a directory, and to be writable only by this plugin (permissions 0700). +- `Host.Metrics` is a sub-registerer; labels automatically carry `plugin=""`. + +## Test constructor (from plugintest) + +```go +package plugintest + +// NopHost returns an in-memory Host backed by an in-memory SQLite database +// and a temporary directory. Suitable for unit tests. +func NopHost(t *testing.T) *plugin.Host { … } + +// Run performs a full lifecycle dry-run of a plugin against a NopHost: +// 1. Migrations are applied. +// 2. Init is called. +// 3. If HasLifecycle, Start is called. +// 4. All declared capabilities are asserted to be registered. +// 5. Shutdown + close. +// t.Fatal on any error. +func Run(t *testing.T, p plugin.Plugin) { … } +``` diff --git a/specs/019-plugin-system/contracts/plugin.md b/specs/019-plugin-system/contracts/plugin.md new file mode 100644 index 0000000..b58107d --- /dev/null +++ b/specs/019-plugin-system/contracts/plugin.md @@ -0,0 +1,150 @@ +# Contract — `internal/plugin` interfaces + +## Base interface + +```go +package plugin + +import ( + "context" +) + +// Plugin is the minimum contract. Every compiled-in plugin implements this. +type Plugin interface { + // Name returns the plugin's globally unique identifier. + // Must match ^[a-z][a-z0-9_]{1,31}$. + Name() string + + // Version returns the plugin's semver version. + Version() string + + // Init is called once during startup after migrations have applied. + // The plugin registers its capabilities here. Must be fast (<100 ms) + // and idempotent. Returning an error marks the plugin failed; the + // core continues to start. + Init(ctx context.Context, host Host) error +} +``` + +## Optional capability interfaces + +Plugins implement any subset. Host uses type assertion at Init time. + +```go +// HasMigrations: plugin owns a numbered chain of SQL migrations. +type HasMigrations interface { + Migrations() []Migration +} + +type Migration struct { + Version int // 1..N, monotonic + Name string // "001_initial" + SQL string // single file, runs in one transaction +} + +// HasMCPTools: plugin adds first-class MCP tools (rare; most plugins use HasActions). +type HasMCPTools interface { + MCPTools() []MCPTool +} + +type MCPTool struct { + Name string // "linkedin_post" + Description string + InputSchema json.RawMessage + Handler func(ctx context.Context, args map[string]any) (any, error) +} + +// HasActions: plugin adds bridged actions callable via the core execute() tool. +type HasActions interface { + Actions() []ActionRegistration +} + +type ActionRegistration struct { + Name string // "create_article" + Description string + InputSchema json.RawMessage + RequiredScope Scope // read | write | admin + Handler func(ctx context.Context, args map[string]any) (any, error) +} + +type Scope string +const ( + ScopeRead Scope = "read" + ScopeWrite Scope = "write" + ScopeAdmin Scope = "admin" +) + +// HasHTTPRoutes: plugin mounts REST routes under /api/plugins//*. +type HasHTTPRoutes interface { + RegisterRoutes(r chi.Router) +} + +// HasWebPanels: plugin contributes one or more UI panels, served under /ui/plugins//*. +type HasWebPanels interface { + WebPanels() []PanelManifest + PanelHandler() http.Handler // serves /ui/plugins//* +} + +type PanelManifest struct { + ID string // "wiki" + Title string // "Wiki" + Icon string // lucide-icon name + Route string // "/ui/plugins/wiki" + Scope string // "owner" | "member" +} + +// HasCLICommands: plugin adds subcommands under `synapbus plugin `. +type HasCLICommands interface { + CLICommands() []*cobra.Command +} + +// HasChannelType: plugin defines a channel behavior (blackboard/auction-like). +type HasChannelType interface { + ChannelTypes() []ChannelTypeDef +} + +type ChannelTypeDef struct { + Name string + OnMessage func(ctx context.Context, channelID int64, msgID int64) error + OnReaction func(ctx context.Context, channelID int64, msgID int64, reaction string) error +} + +// HasEventHook: plugin subscribes to internal events. +type HasEventHook interface { + OnEvent(ctx context.Context, e Event) error +} + +type Event struct { + Topic string // "message.created", "reaction.added", "plugin.status.changed" + Payload any // typed by topic + Meta map[string]string +} + +// HasLifecycle: plugin runs background work and needs explicit start/shutdown. +type HasLifecycle interface { + Start(ctx context.Context) error + Shutdown(ctx context.Context) error +} + +// HasConfigSchema: plugin publishes a JSON Schema for its config so the Web UI can generate a form. +type HasConfigSchema interface { + ConfigSchema() json.RawMessage +} +``` + +## Stability hint (optional) + +```go +// HasStability: plugin declares stability level. Default: "stable". +type HasStability interface { + Stability() string // "stable" | "beta" | "experimental" +} +``` + +## Invariants + +- `Plugin.Name()` MUST be constant over the plugin's lifetime (memoized is fine). +- `Init` MUST NOT block for more than 100 ms; long-running work goes in `HasLifecycle.Start`. +- `Init` MUST be safe to call exactly once; calling twice is a host bug, not a plugin bug. +- Plugins MUST NOT hold a package-level reference to the `Host`; only the `Init` ctx-scoped copy. +- `Shutdown` MUST be idempotent; the host may call it more than once on chained signals. diff --git a/specs/019-plugin-system/contracts/rest.md b/specs/019-plugin-system/contracts/rest.md new file mode 100644 index 0000000..eefca0c --- /dev/null +++ b/specs/019-plugin-system/contracts/rest.md @@ -0,0 +1,55 @@ +# Contract — REST endpoints + +## Core endpoints (added by this feature) + +### `GET /api/plugins/status` + +Returns the registry state for all plugins. + +**Auth**: session cookie or `Authorization: Bearer ` (any authenticated user). + +**Response 200**: +```json +{ + "plugins": [ + { + "name": "wiki", + "version": "0.1.0", + "stability": "stable", + "enabled": true, + "status": "started", + "started_at": "2026-04-19T10:30:12Z", + "error": null, + "capabilities": ["migrations", "actions", "http_routes", "web_panel", "config_schema"], + "tools_registered": ["create_article","get_article","list_articles","update_article","get_backlinks"], + "migration_versions": [1] + } + ] +} +``` + +Error shapes: standard `{"error":"…"}` with appropriate HTTP status. + +### `POST /api/plugins/{name}/enable` (admin only) + +Sets `plugins..enabled: true` in `synapbus.yaml` and sends self-SIGHUP. Returns 202 Accepted with `{"restart": true}`; operator observes the graceful restart. + +### `POST /api/plugins/{name}/disable` (admin only) + +Sets `plugins..enabled: false` in `synapbus.yaml` and sends self-SIGHUP. + +### `GET /api/plugins/{name}/schema` (any auth) + +Returns the plugin's `HasConfigSchema().ConfigSchema()` JSON Schema, or 404 if the plugin does not implement it. + +## Plugin-owned endpoints + +Plugins mount routes via `HasHTTPRoutes.RegisterRoutes(r chi.Router)`. The registry mounts the returned router under `/api/plugins//`. Plugins MUST NOT register routes outside this namespace. + +## UI panel endpoints + +Plugins with `HasWebPanels` MUST serve assets under `/ui/plugins//`. The registry mounts `plugin.PanelHandler()` under this prefix. The plugin's `index.html` is served at `/ui/plugins//`. + +## Core shell changes + +The existing `/ui/*` shell (Svelte) fetches `/api/plugins/status` on load and renders plugin panels in the nav based on the `capabilities` including `"web_panel"` and the `panels` field derived from `HasWebPanels().WebPanels()`. Clicking a panel entry in the nav opens the route `/ui/plugins/` in an iframe in the content pane. diff --git a/specs/019-plugin-system/data-model.md b/specs/019-plugin-system/data-model.md new file mode 100644 index 0000000..4ecb500 --- /dev/null +++ b/specs/019-plugin-system/data-model.md @@ -0,0 +1,181 @@ +# Phase 1 Data Model — Plugin System + +## Entities + +### Plugin (in-memory, code-defined) + +Not persisted directly; it is a Go value constructed by a factory. + +| Field | Type | Notes | +|---|---|---| +| Name | string | Unique, lowercased, snake-case. Panic on duplicate at registry build time. | +| Version | string | Semver ("0.1.0"). Informational only in this feature. | +| Stability | string | "stable" / "beta" / "experimental". Defaults to "stable". Non-stable emits a log warning but is not gated. | + +### Plugin Registry (in-memory, one per process) + +Assembled at boot from `defaultPlugins()` filtered by config. + +| Field | Type | Notes | +|---|---|---| +| plugins | `map[string]*entry` | Name → entry, panic on duplicate. | +| order | `[]string` | Registration order, used for migrate/init/start sequence. | + +Per-entry `struct` fields: + +| Field | Type | Notes | +|---|---|---| +| plugin | `Plugin` | The plugin value. | +| config | `json.RawMessage` | From YAML sub-tree. | +| enabled | `bool` | From YAML `plugins..enabled`. Default false unless the plugin has a default-enabled hint. | +| status | `Status` | `registered` → `migrated` → `initialized` → `started` \| `failed` \| `disabled` | +| error | `error` | Populated when status == `failed`. | +| startedAt | `time.Time` | Populated on transition to `started`. | + +### Plugin Migration Record (SQLite, core table) + +Table `plugin_migrations`, created by core migration `000_initial.sql`. + +| Column | Type | Notes | +|---|---|---| +| plugin | TEXT | Plugin name. PK (plugin, version). | +| version | INTEGER | Plugin-local migration number, monotonically increasing within plugin. | +| name | TEXT | Human-readable slug ("001_initial"). | +| applied_at | DATETIME | Default `CURRENT_TIMESTAMP`. | +| checksum | TEXT | SHA-256 of the migration SQL; compared on re-apply to detect drift. | + +### Plugin-owned tables (example: wiki) + +The wiki plugin's migration `001_initial.sql` (executed against the same SQLite handle as core): + +```sql +CREATE TABLE IF NOT EXISTS plugin_wiki_articles ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + slug TEXT NOT NULL UNIQUE, + title TEXT NOT NULL, + body TEXT NOT NULL, + revision INTEGER NOT NULL DEFAULT 1, + word_count INTEGER NOT NULL DEFAULT 0, + created_by TEXT, + created_at DATETIME DEFAULT CURRENT_TIMESTAMP, + updated_by TEXT, + updated_at DATETIME DEFAULT CURRENT_TIMESTAMP +); + +CREATE TABLE IF NOT EXISTS plugin_wiki_backlinks ( + from_slug TEXT NOT NULL, + to_slug TEXT NOT NULL, + PRIMARY KEY (from_slug, to_slug) +); + +CREATE INDEX IF NOT EXISTS idx_plugin_wiki_articles_slug + ON plugin_wiki_articles(slug); +``` + +Tables are prefixed `plugin_wiki_*` for audit / lint clarity. + +### Host (in-memory, one per plugin per process) + +Constructed by the registry when calling `Plugin.Init(ctx, host)`. Not persisted. + +| Field | Type | Purpose | +|---|---|---| +| Logger | `*slog.Logger` | Pre-tagged `plugin=`. | +| DB | `*sqlx.DB` | Shared core handle. | +| Messaging | `messaging.API` | Send DMs / channel messages. | +| Channels | `channels.API` | CRUD, post, reactions. | +| Attachments | `attachments.Store` | CAS blob API. | +| Search | `search.Index` | Read-only semantic + FTS. | +| Secrets | `secrets.Scoped` | Scoped to this plugin only. | +| Events | `eventbus.Bus` | Subscribe / publish. | +| Config | `json.RawMessage` | This plugin's YAML sub-tree. | +| DataDir | `string` | `<--data>/plugins//`, pre-created. | +| Tracer | `trace.Tracer` | OTel tracer scoped to plugin. | +| Metrics | `prometheus.Registerer` | Sub-registry scoped to plugin. | +| DefaultOwner | `*users.User` | For failure notifications. Read-only snapshot. | + +### Plugin Status (runtime JSON, served by `/api/plugins/status`) + +```json +{ + "plugins": [ + { + "name": "wiki", + "version": "0.1.0", + "stability": "stable", + "enabled": true, + "status": "started", + "started_at": "2026-04-19T10:30:12Z", + "error": null, + "capabilities": ["migrations", "mcp_tools", "actions", "http_routes", "web_panel", "config_schema"], + "tools_registered": ["create_article", "get_article", "list_articles", "update_article", "get_backlinks"], + "migration_versions": [1] + }, + { + "name": "demo_broken", + "version": "0.0.1", + "stability": "experimental", + "enabled": true, + "status": "failed", + "started_at": null, + "error": "Init: missing required config field 'token'", + "capabilities": ["config_schema"], + "tools_registered": [], + "migration_versions": [] + } + ] +} +``` + +### Backup Archive (filesystem artifact) + +Produced by `scripts/backup-kubic.sh`. Structure: + +``` +synapbus-backup-YYYY-MM-DDTHH-MM-SSZ.tar.gz +├── manifest.json # name + SHA-256 + size of each entry +├── synapbus.db # SQLite main file +├── synapbus.db-wal # SQLite WAL (if present) +├── attachments/ # CAS directory +│ └── / # existing structure preserved +├── secrets.key # master key (mode 0600) +└── hnsw.idx # vector index snapshot +``` + +`manifest.json` example: + +```json +{ + "created_at": "2026-04-19T11:00:00Z", + "synapbus_version": "0.12.3+pre-plugin-refactor", + "source_host": "hub.synapbus.dev", + "entries": [ + {"path": "synapbus.db", "sha256": "…", "bytes": 48_218_112}, + {"path": "secrets.key", "sha256": "…", "bytes": 32}, + … + ] +} +``` + +## State transitions + +``` +Plugin lifecycle: + registered → migrated → initialized → started (happy path) + │ │ │ + └── failed ──┴── failed ──┴── failed (on any error) + +Disabled plugins stay in: + registered → disabled +``` + +Transitions fire `EventBus` events on `plugin.status.changed` with `{name, from, to, error?}`. + +## Validation rules (derived from FRs) + +- Plugin name MUST match `^[a-z][a-z0-9_]{1,31}$`. (Prevents path traversal, YAML quirks, table name explosions.) +- Duplicate plugin name → panic at `defaultPlugins()` processing (FR-004). +- Duplicate tool name across plugins → panic during `Init` phase (FR-022). +- Plugin tables not prefixed `plugin__` → migration refused at apply time, plugin marked failed (FR-010). +- `HasConfigSchema` schema validation failure → plugin marked failed with the validator diagnostic (edge case: invalid config references). +- Cross-plugin secret access attempt → `secrets.Scoped` returns `ErrNotFound` deterministically; FR-007 + SC-006. diff --git a/specs/019-plugin-system/plan.md b/specs/019-plugin-system/plan.md new file mode 100644 index 0000000..d90cc87 --- /dev/null +++ b/specs/019-plugin-system/plan.md @@ -0,0 +1,119 @@ +# Implementation Plan: Plugin System for SynapBus Core + +**Branch**: `019-plugin-system` | **Date**: 2026-04-19 | **Spec**: [spec.md](./spec.md) +**Input**: Feature specification from `/specs/019-plugin-system/spec.md` + +## Summary + +Refactor SynapBus into a tiny core (messaging, channels, reactions, auth, storage, MCP transport, search, attachments, Web UI shell) plus a set of compile-in plugins. Deliver in three phases: Phase 0 backs up the live kubic instance and squashes 26 migrations into a single `000_initial.sql`; Phase 1 introduces `internal/plugin/` (interface, host, registry, lifecycle) plus the `plugintest` helper and wires SIGHUP-triggered graceful restart; Phase 2 extracts the wiki feature as the canonical pilot plugin. The remaining nine plugin candidates are mechanical follow-ups not in this spec. + +## Technical Context + +**Language/Version**: Go 1.25+ (per go.mod) +**Primary Dependencies**: `mark3labs/mcp-go` (MCP), `go-chi/chi` (HTTP), `spf13/cobra` (CLI), `modernc.org/sqlite` (storage), `jmoiron/sqlx` (query helpers), `cloudflare/tableflip` (graceful restart — NEW), `gopkg.in/yaml.v3` (config), `xeipuuv/gojsonschema` (config-schema validation) +**Storage**: SQLite via `modernc.org/sqlite` (pure Go, zero CGO). New core table `plugin_migrations`. Plugin tables namespaced `plugin__*`. +**Testing**: `go test`, table-driven; new `internal/plugin/plugintest/` package shipped alongside `internal/plugin/`; integration test boots the binary with a fixture config and exercises wiki via curl + MCP. +**Target Platform**: `linux/amd64`, `darwin/arm64`, `darwin/amd64` (per constitution III) +**Project Type**: Single-binary Go server with embedded Svelte Web UI +**Performance Goals**: Graceful restart < 2 s visible downtime on dev laptop; plugin `Init` < 100 ms each; `plugintest.Run` < 1 s per plugin. +**Constraints**: Zero CGO; plugins may not import from `internal/` outside `internal/plugin/*`; core may not import from `internal/plugins/*`; enforced via a compile-time lint pass. +**Scale/Scope**: ~12 eventual plugins; in this feature 1 (wiki) is extracted. Current codebase is ~44 KLOC Go + 28 KLOC tests across 37 `internal/` packages. + +## Constitution Check + +*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.* + +| Principle | Status | Notes | +|---|---|---| +| I. Local-First, Single Binary | ✅ Pass | Plugins compile-in; no RPC, no .so, no Wasm. `cloudflare/tableflip` is pure-Go, no daemon dependency. | +| II. MCP-Native | ✅ Pass | Plugins contribute MCP tools through `HasMCPTools` / `HasActions`; core MCP server remains the only agent interface. | +| III. Pure Go, Zero CGO | ✅ Pass | All new deps (`tableflip`, `yaml.v3`, `gojsonschema`) are pure Go. | +| IV. Multi-Tenant with Ownership | ✅ Pass | Plugin failure notification targets the default-agent owner. Host exposes owner-scoped messaging. | +| V. Embedded OAuth 2.1 | ✅ Pass | Auth stays in core; plugins receive an action-scope check via `Registration.RequiredScope`. | +| VI. Semantic-Ready Storage | ✅ Pass | Search index remains in core and is exposed via `Host.Search`. Plugins use it read-only. | +| VII. Swarm Intelligence Patterns | ✅ Pass | Channel-type pattern (`HasChannelType`) makes auction/blackboard a plug-point rather than a hardcode. | +| VIII. Observable by Default | ✅ Pass | Per-plugin tracer and metrics registerer on Host; `/api/plugins/status` exposes state. Failure paths log + DM the owner. | +| IX. Progressive Complexity | ✅ Pass | Disabling every plugin leaves a working tier-1 system (messaging + agent registration). | +| X. Web UI as First-Class | ✅ Pass | Plugin panels served from `/ui/plugins/`; shell continues to own navigation and auth. | + +All ten gates pass. No complexity-tracking entries required. + +## Project Structure + +### Documentation (this feature) + +``` +specs/019-plugin-system/ +├── plan.md # this file +├── spec.md # feature spec +├── research.md # Phase 0 research synthesis +├── data-model.md # Phase 1 entity model +├── quickstart.md # "build your first plugin" walkthrough +├── contracts/ +│ ├── plugin.md # Plugin + HasX interface contracts +│ ├── host.md # Host API contract +│ └── rest.md # /api/plugins/* contract +├── checklists/ +│ └── requirements.md # spec quality checklist +└── tasks.md # Phase 2 output (generated by /speckit.tasks) +``` + +### Source Code (repository root) + +``` +internal/ +├── plugin/ # NEW — plugin framework (~300 LOC) +│ ├── plugin.go # Plugin, HasX interfaces +│ ├── host.go # Host struct and scoped accessors +│ ├── registry.go # registry, enable/disable, panic-on-dup +│ ├── lifecycle.go # three-phase boot, shutdown in reverse +│ ├── status.go # Plugin status tracking + /api/plugins/status handler +│ ├── config.go # YAML load, per-plugin config extraction +│ ├── migrator.go # per-plugin migration runner using plugin_migrations table +│ ├── restart.go # SIGHUP → tableflip.Upgrade re-exec +│ └── plugintest/ +│ ├── nop_host.go # NopHost with in-memory SQLite +│ ├── run.go # Run(t, plugin) full-lifecycle smoke +│ └── assertions.go # HasTool/HasRoute/HasMigration helpers +├── plugins/ # NEW — all plugins live here +│ ├── standard/ +│ │ └── import.go # var Standard = []plugin.Plugin{ ... } consumed by main.go +│ └── wiki/ # FIRST EXTRACTED PLUGIN +│ ├── plugin.go # WikiPlugin implementing Plugin + HasActions + HasHTTPRoutes + HasWebPanels + HasMigrations + HasMCPTools + HasConfigSchema +│ ├── actions.go # create_article, get_article, list_articles, update_article, get_backlinks +│ ├── routes.go # /api/plugins/wiki/* (if any needed for the UI panel) +│ ├── panel.go # /ui/plugins/wiki/* serving embedded HTML+JS +│ ├── store.go # SQL access against plugin_wiki_articles (moved from internal/wiki) +│ ├── schema/ +│ │ └── 001_initial.sql # creates plugin_wiki_articles, plugin_wiki_backlinks +│ ├── ui/ # embedded assets (Svelte or plain HTML — choose simplest) +│ │ └── index.html +│ └── plugin_test.go # plugintest.Run + action-level unit tests +│ +cmd/synapbus/ +├── main.go # calls plugin.NewRegistry(defaultPlugins()) and bootstraps +└── plugins.go # NEW — defaultPlugins() returns []plugin.Plugin +│ +schema/ # core migrations +├── 000_initial.sql # NEW — squashed from all 26 prior migrations +└── (legacy files archived into ../legacy-migrations/ at squash time) +│ +scripts/ +├── backup-kubic.sh # NEW — SSH into kubic, dump DB + attachments + secrets key + hnsw index, produce manifest +├── verify-backup.sh # NEW — restore archive into scratch dir, schema diff +└── generate-squash.sh # NEW — sqlite .schema + light edits → 000_initial.sql +│ +docs/plugins/ # NEW +├── authoring.md # step-by-step "write your first plugin" +├── lifecycle.md # migrate → init → start, shutdown semantics +└── capabilities.md # each HasX interface with example snippets +│ +test/integration/ +└── plugin_system_test.go # NEW — boots binary, curls MCP + REST, asserts enable/disable +``` + +**Structure Decision**: Standard Go `internal/` layout. The plugin framework lives at `internal/plugin/`; all plugins live at `internal/plugins//`. A single import barrel `internal/plugins/standard/import.go` is consumed by `cmd/synapbus/plugins.go` → `defaultPlugins()`. No `init()` registration is used. The compile-time boundary (core files must not import `internal/plugins/*`) is enforced by a `go vet` analyzer added in Phase 1. + +## Complexity Tracking + +No constitution violations. No entries required. diff --git a/specs/019-plugin-system/quickstart.md b/specs/019-plugin-system/quickstart.md new file mode 100644 index 0000000..f43d467 --- /dev/null +++ b/specs/019-plugin-system/quickstart.md @@ -0,0 +1,178 @@ +# Quickstart — Write your first SynapBus plugin + +Goal: add a "hello" plugin that exposes an MCP action, a REST route, a Web UI panel, and a tiny migration — in under 20 minutes. + +## 1. Create the package + +``` +internal/plugins/hello/ +├── plugin.go +├── schema/ +│ └── 001_initial.sql +├── ui/ +│ └── index.html +└── plugin_test.go +``` + +## 2. Write the plugin + +`internal/plugins/hello/plugin.go`: + +```go +package hello + +import ( + "context" + _ "embed" + "encoding/json" + "net/http" + + "github.com/go-chi/chi/v5" + "github.com/synapbus/synapbus/internal/plugin" +) + +//go:embed schema/001_initial.sql +var migration001 string + +//go:embed ui/index.html +var panelHTML []byte + +type HelloPlugin struct{ host plugin.Host } + +func New() *HelloPlugin { return &HelloPlugin{} } + +func (p *HelloPlugin) Name() string { return "hello" } +func (p *HelloPlugin) Version() string { return "0.1.0" } + +func (p *HelloPlugin) Init(ctx context.Context, host plugin.Host) error { + p.host = host + return nil +} + +func (p *HelloPlugin) Migrations() []plugin.Migration { + return []plugin.Migration{ + {Version: 1, Name: "001_initial", SQL: migration001}, + } +} + +func (p *HelloPlugin) Actions() []plugin.ActionRegistration { + return []plugin.ActionRegistration{{ + Name: "say_hello", + Description: "Return a greeting for the given name.", + InputSchema: json.RawMessage(`{"type":"object","properties":{"name":{"type":"string"}}}`), + RequiredScope: plugin.ScopeRead, + Handler: p.sayHello, + }} +} + +func (p *HelloPlugin) sayHello(ctx context.Context, args map[string]any) (any, error) { + name, _ := args["name"].(string) + if name == "" { name = "world" } + return map[string]string{"greeting": "hello, " + name}, nil +} + +func (p *HelloPlugin) RegisterRoutes(r chi.Router) { + r.Get("/ping", func(w http.ResponseWriter, _ *http.Request) { + _, _ = w.Write([]byte(`{"ok":true}`)) + w.Header().Set("Content-Type", "application/json") + }) +} + +func (p *HelloPlugin) WebPanels() []plugin.PanelManifest { + return []plugin.PanelManifest{{ID: "hello", Title: "Hello", Icon: "hand", Route: "/ui/plugins/hello", Scope: "member"}} +} + +func (p *HelloPlugin) PanelHandler() http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "text/html") + _, _ = w.Write(panelHTML) + }) +} +``` + +## 3. Write the migration + +`internal/plugins/hello/schema/001_initial.sql`: + +```sql +CREATE TABLE IF NOT EXISTS plugin_hello_greetings ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + name TEXT NOT NULL, + created_at DATETIME DEFAULT CURRENT_TIMESTAMP +); +``` + +## 4. Write the UI panel + +`internal/plugins/hello/ui/index.html`: + +```html + +Hello +

Hello, plugin world

+``` + +## 5. Register it + +`cmd/synapbus/plugins.go`: + +```go +package main + +import ( + "github.com/synapbus/synapbus/internal/plugin" + "github.com/synapbus/synapbus/internal/plugins/hello" + "github.com/synapbus/synapbus/internal/plugins/wiki" +) + +func defaultPlugins() []plugin.Plugin { + return []plugin.Plugin{ + wiki.New(), + hello.New(), // ← one line + } +} +``` + +## 6. Write the smoke test + +`internal/plugins/hello/plugin_test.go`: + +```go +package hello_test + +import ( + "testing" + + "github.com/synapbus/synapbus/internal/plugin/plugintest" + "github.com/synapbus/synapbus/internal/plugins/hello" +) + +func TestHelloPlugin(t *testing.T) { + plugintest.Run(t, hello.New()) +} +``` + +## 7. Enable in config + +`synapbus.yaml`: + +```yaml +plugins: + wiki: { enabled: true } + hello: { enabled: true } +``` + +## 8. Run + +``` +make build +./synapbus serve --data ./data +``` + +- `curl http://localhost:8080/api/plugins/status | jq` — you see `hello` status=started. +- `curl http://localhost:8080/api/plugins/hello/ping` — returns `{"ok":true}`. +- Open `http://localhost:8080/ui/plugins/hello` in a browser — renders the HTML. +- Call `say_hello` via MCP: `mcp execute "call('say_hello', {name: 'you'})"` → `{"greeting":"hello, you"}`. +- Flip to `enabled: false`, send SIGHUP — everything above goes away cleanly. + +Total new code: ~60 lines of Go. No core code modified. diff --git a/specs/019-plugin-system/research.md b/specs/019-plugin-system/research.md new file mode 100644 index 0000000..fbdee1f --- /dev/null +++ b/specs/019-plugin-system/research.md @@ -0,0 +1,185 @@ +# Phase 0 Research — Plugin System for SynapBus + +**Status**: All unknowns resolved. Decisions below derived from prior brainstorming session + three parallel research agents (HashiCorp, Caddy v2, broader Go plugin landscape — OpenTelemetry Collector, Prometheus SD, Coraza, Cosmos-SDK, etc.). + +--- + +## Decision 1 — Plugin interface shape + +**Decision**: Tiny `Plugin` interface (Name, Version, Init) + optional `HasX` capability sub-interfaces detected by type assertion at registration. + +**Rationale**: +- Terraform framework's growth from monolithic provider interface → optional sub-interfaces is a cautionary tale in reverse: the decomposition is inevitable. +- OpenTelemetry Collector `component.Component` is this shape and is the gold standard for compile-in Go plugins. +- Plugins pay only for the capabilities they need; a 15-line plugin is a valid plugin. + +**Alternatives considered**: +- Monolithic `Plugin` with all methods — rejected: Cosmos-SDK's `AppModule` is living proof this ossifies. +- `Host.RegisterXxx()` imperative registration during `Init` — viable but harder to reason about ordering; capability interfaces make the contract declarative. + +--- + +## Decision 2 — Registration mechanism + +**Decision**: Explicit list in `defaultPlugins()` (cmd/synapbus/plugins.go), not `init()` side effects. + +**Rationale**: +- OpenTelemetry Collector moved away from `init()` registration because test builds, OSS vs. enterprise distributions, and CI need to supply alternate plugin sets; `init()` makes this painful. +- One line per plugin is trivial ceremony. +- `go test` order-independence is easier to guarantee. + +**Alternatives considered**: +- `init()` + blank imports (Caddy style) — rejected for the above reasons; the ergonomic win doesn't outweigh the control loss. +- Hybrid (init-registers + config allow-list) — rejected as unnecessarily two-layered. + +--- + +## Decision 3 — Host API shape + +**Decision**: Single `Host` struct passed to `Plugin.Init(ctx, host) error`. Fields are typed accessors for core services. Not globals, not a service locator with string keys. + +**Rationale**: +- Vault's `BackendConfig` pattern + Kong's PDK struct both land here. +- Trivially mockable in tests — one `mockHost{}` covers 100% of plugin-side testing. +- Adding a new service means adding a field; no breaking signature change to `Init`. + +**Alternatives considered**: +- `Host` as an interface — rejected: interfaces force one-at-a-time mocking and obscure which fields exist. A struct with exported typed fields is more Go-idiomatic for a closed set. +- Global singletons — rejected outright; violates DI and hurts testability. + +--- + +## Decision 4 — Dynamic enable/disable approach + +**Decision**: Compile-in all plugins. Enable/disable via YAML config. Runtime toggle triggers SIGHUP-based graceful restart using `cloudflare/tableflip`. + +**Rationale**: +- Go's `plugin` package is effectively broken (Linux-only, no unload, deps must match exactly). +- HashiCorp go-plugin (subprocess + gRPC) would require a wire protocol for every extension point, including Web UI panels, which is impractical. +- Wasm (wazero) would constrain plugin authors significantly and adds a whole toolchain. +- Graceful restart with tableflip = ~200–800 ms visible downtime on dev; MCP clients reconnect automatically. This is 99% indistinguishable from real hot-load for the user. + +**Alternatives considered**: +- Real hot-reload via Wasm — rejected for toolchain burden. +- Full restart (kill -TERM + re-spawn by systemd) — rejected: > 1 s downtime, in-flight HTTP/SSE requests lost. + +--- + +## Decision 5 — SQL migration ownership + +**Decision**: Each plugin owns its own numbered migration chain under `internal/plugins//schema/`. Core records applied migrations in a new `plugin_migrations (plugin, version)` table. Plugin tables namespaced `plugin__*`. + +**Rationale**: +- Isolation: a plugin's schema lives with its code. +- Re-enable-after-disable works: tables and data survive. +- Namespace prefix prevents collisions and makes a renegade `SELECT * FROM plugin_wiki_articles` visible in reviews. + +**Alternatives considered**: +- Single monolithic numbered chain — rejected: couples plugin code to a global numbering, makes alt distributions impossible. +- Per-plugin separate SQLite file — rejected: loses the simplicity advantage of one database handle. + +--- + +## Decision 6 — Web UI panel integration + +**Decision**: Each plugin serves its own HTML/JS from `/ui/plugins//*` backed by its own `embed.FS`. Core shell renders panels as iframes in the existing content pane, driven by the plugin-supplied `PanelManifest`. + +**Rationale**: +- Zero build-time coupling between plugin releases and the core Svelte shell — upgrading the wiki doesn't rebuild the shell. +- Security isolation via iframe origin is a free bonus. +- Simple to implement — `http.FS(embed.FS)` with a Chi subrouter. + +**Alternatives considered**: +- Dynamic ES module imports of Svelte components from the shell — rejected: requires a manifest merge at build time and couples versions. +- Server-rendered panels into the shell via fetch — rejected: still couples layout/style. + +--- + +## Decision 7 — Config format + +**Decision**: YAML (`synapbus.yaml`) at a known path (next to binary or `$SYNAPBUS_DATA_DIR/synapbus.yaml`). Each plugin's sub-tree is passed into its `Host.Config` as `json.RawMessage`; the plugin unmarshals into its typed struct. Optional `HasConfigSchema` lets the Web UI generate a form. + +**Rationale**: +- YAML is the prevailing agent-tooling config format (LangChain, AutoGen, Helm, OTel all use YAML). +- `json.RawMessage` is used internally because that's what `yaml.v3` hands back after first-pass decode and Go's JSON-struct-tags are widely known. +- `HasConfigSchema` being optional means "zero-config" plugins still work. + +**Alternatives considered**: +- HCL — rejected: adds HashiCorp dep, less familiar to Python-first audience. +- TOML — rejected: less familiar than YAML in agent ecosystem. +- Environment variables only — rejected: typed per-plugin blobs don't fit env-var flat keyspace. + +--- + +## Decision 8 — Testing approach + +**Decision**: Ship `internal/plugin/plugintest` package with `NopHost()` (in-memory SQLite + tmpdir) and `Run(t, plugin)` smoke helper. + +**Rationale**: +- Copied from OTel Collector `componenttest.NewNopHost`. +- Every plugin gets a ≤ 20-line smoke test for free. +- Mocking the Host struct is the primary testing surface anyway; this formalizes it. + +**Alternatives considered**: +- No helper (each plugin invents its own scaffolding) — rejected: produces inconsistent test quality and discourages testing. + +--- + +## Decision 9 — Boundary enforcement + +**Decision**: A custom `go/analysis` analyzer runs in CI that rejects imports from `internal/plugins/*` by files outside that subtree (and rejects imports into `internal/plugins/*` from core `internal/*` packages that aren't `internal/plugin*`). + +**Rationale**: +- Compile-time enforcement of the architectural invariant. +- Without it, the `internal/` flat layout makes it trivially easy to accidentally cross the line. + +**Alternatives considered**: +- `go-arch-lint` — rejected: another dep; a ~50-line custom analyzer is simpler. +- Honor system — rejected: precisely what we're escaping. + +--- + +## Decision 10 — Squash of existing 26 migrations + +**Decision**: Generate `000_initial.sql` from the live `.schema` output plus curated seed data for reference rows, after the kubic backup has been verified restorable. Archive legacy migration files under `schema/legacy/` for historical reference; they are not applied. + +**Rationale**: +- No external users exist; backward-compat across the 26-step chain is unneeded. +- A single file is ~50× faster to boot and is the new baseline. +- Legacy chain is preserved for human forensic use. + +**Alternatives considered**: +- Keep all 26 migrations forever — rejected: user explicitly authorized squash. +- Truncate legacy and delete — rejected: losing schema history is a small-but-real audit cost. + +--- + +## Decision 11 — Failure-notification channel + +**Decision**: On plugin `Init` error or `Start` panic: (a) mark the plugin failed in the registry, (b) log the error with `slog`, (c) send a direct message to the owner of the default system agent naming the plugin and the error. + +**Rationale**: +- DM integration uses existing messaging; no new ops channel to manage. +- Surfaces plugin problems in the operator's normal workflow (the same inbox they read every day). +- Owner-of-default-agent is always defined and the simplest routing decision. + +**Alternatives considered**: +- Dedicated `#plugin-alerts` channel — rejected: over-designed for a single-operator deployment. +- Webhook to external destination — rejected: violates local-first principle and adds config burden. + +--- + +## Decision 12 — Integration test strategy + +**Decision**: One `test/integration/plugin_system_test.go` boots the binary with a fixture YAML, waits for readiness, exercises the wiki plugin via `curl` + MCP JSON-RPC, then flips the config, sends SIGHUP, waits for restart, and re-checks. Uses `testcontainers`-free approach — just spawns `./synapbus serve` as a subprocess. + +**Rationale**: +- No need for Docker in unit CI. +- End-to-end guarantees the three-phase boot works for real, not just in mocks. + +**Alternatives considered**: +- Containerized e2e — rejected: slow, adds infra; this binary is single-file. + +--- + +All Phase 0 unknowns resolved. Proceeding to Phase 1.