Harness-Agnostic Wrappers &
OpenTelemetry for SynapBus

TL;DR

1 · What scion actually does

Despite the name collision with the SCION internet-architecture project, GoogleCloudPlatform/scion is a multi-agent orchestration harness for evaluating and running "deep agents" (Claude Code, Gemini CLI, Codex, OpenCode) inside isolated containers. It is explicitly not a planner and not a verifier — it is the control plane and observability spine around arbitrary agent CLIs.

The Harness interface — the centrepiece

pkg/api/harness.go:22–68

type Harness interface {
    Name() string
    AdvancedCapabilities() HarnessAdvancedCapabilities
    GetEnv(agentName, agentHome, unixUsername string) map[string]string
    GetCommand(task string, resume bool, baseArgs []string) []string
    DefaultConfigDir() string
    SkillsDir() string
    HasSystemPrompt(agentHome string) bool
    Provision(ctx context.Context, agentName, agentDir, agentHome, agentWorkspace string) error
    GetEmbedDir() string
    GetInterruptKey() string
    GetHarnessEmbedsFS() (embed.FS, string)
    InjectAgentInstructions(agentHome string, content []byte) error
    InjectSystemPrompt(agentHome string, content []byte) error
    // the key OTel seam — returns env vars that the container runtime
    // will merge into the child process env before exec
    GetTelemetryEnv() map[string]string
    ResolveAuth(auth AuthConfig) (*ResolvedAuth, error)
}

Three things to notice:

• Provision is separate from GetCommand: one-shot setup (write .claude.json, pre-approve tool fingerprints, materialise skill files) versus per-invocation command building.
• GetEnv / GetTelemetryEnv / ResolveAuth all return maps of env vars. The container runtime layer merges them. This means every harness is credential-injection-agnostic and telemetry-injection-agnostic — you can point a whole pod at a different OTel collector by changing one map.
• AdvancedCapabilities() lets a dispatcher ask "does this harness support system prompts?" and degrade gracefully (fall back to InjectAgentInstructions) when it doesn't.

The factory

pkg/harness/harness.go:37–57

func New(name string) Harness {
    switch name {
    case "claude":   return &ClaudeCode{}
    case "gemini":   return &GeminiCLI{}
    case "opencode": return &OpenCode{}
    case "codex":    return &Codex{}
    }
    if h := pluginMgr.Lookup(name); h != nil { return h }
    return &Generic{} // universal fallback
}

OTel injection pattern

pkg/harness/claude_code.go:311–320

func (c *ClaudeCode) GetTelemetryEnv() map[string]string {
    return map[string]string{
        "CLAUDE_CODE_ENABLE_TELEMETRY": "1",
        "OTEL_METRICS_EXPORTER":       "otlp",
        "OTEL_LOGS_EXPORTER":          "otlp",
        "OTEL_EXPORTER_OTLP_PROTOCOL": "grpc",
        "OTEL_EXPORTER_OTLP_ENDPOINT": "http://localhost:4317",
        "OTEL_METRIC_EXPORT_INTERVAL": "30000",
    }
}

Scion's own Go code emits OTel logs via the OTLP log exporter (pkg/util/logging/otel_provider.go:26–61) and bridges slog into it (pkg/util/logging/otel.go:85–119). W3C traceparent headers are extracted at HTTP ingress (pkg/util/logging/trace.go) so trace context can flow across the dispatcher → runtime → container boundary.

Coordination & decomposition

Scion does not decompose tasks. A single task string goes to the agent and the agent's own model decides how to break it up. Coordination between agents happens via a structured StructuredMessage envelope (pkg/messages/types.go:46–61) with fields {sender, recipient, msg, type, urgent, broadcasted, attachments} — an on-disk analogue of a SynapBus channel post.

2 · What paperclip actually does

Paperclip is a Node/Express control plane for running 10–20 agent "companies" with org charts, budgets, and approval gates. Wildly different product — but it has a clean adapter interface worth borrowing.

The ServerAdapterModule interface

packages/adapter-utils/src/types.ts:292–331

export interface ServerAdapterModule {
  type: string;
  execute(ctx: AdapterExecutionContext): Promise<AdapterExecutionResult>;
  testEnvironment(ctx: AdapterEnvironmentTestContext): Promise<AdapterEnvironmentTestResult>;
  listSkills?:  (ctx) => Promise<AdapterSkillSnapshot>;
  syncSkills?:  (ctx, desired: string[]) => Promise<AdapterSkillSnapshot>;
  sessionCodec?:     AdapterSessionCodec;       // resume / serialize sessions
  models?:           AdapterModel[];
  listModels?:      () => Promise<AdapterModel[]>;
  agentConfigurationDoc?: string;
  onHireApproved?:  (payload, cfg) => Promise<HireApprovedHookResult>;
  getQuotaWindows?: () => Promise<ProviderQuotaResult>;
}

AdapterExecutionResult — types.ts:64–95

{ exitCode, signal, timedOut, errorMessage, errorCode,
  usage: { inputTokens, outputTokens, cachedInputTokens },
  resultJson, costUsd,
  question?: { prompt, choices }  // can pause for human approval
}

Ten adapters are registered via a mutable map in server/src/adapters/registry.ts:89–222: claude-local, codex-local, cursor, gemini, opencode, pi, openclaw, hermes, http, process. External adapters are loaded from plugins asynchronously (lines 244–270).

Coordination model — heartbeat + atomic checkout

server/src/services/heartbeat.ts

No DAG, no queue, no planner. Agents wake on a heartbeat (schedule or event), atomically claim assigned issues via a per-agent start lock (withAgentStartLock(), lines 331–346), run once, and go back to sleep. Concurrency is per-agent (default 1, configurable to 10). Task decomposition is entirely delegated to the agent's own model.

Verification

None that's interesting. Exit code 0 = success; timeouts and process-loss retries are tracked; there is no LLM judge, no schema validation, no test runner. Verification is whatever the running agent chooses to self-report in resultJson.

Observability

Pino structured logging (server/src/middleware/logger.ts:29–45) + a custom telemetry client (server/src/telemetry.ts:12–26) that batch-flushes events every 60s. No OpenTelemetry. This is the weakest part relative to scion.

3 · Side-by-side comparison

4 · SynapBus current execution surface

5 · Proposed design for SynapBus

New package: internal/harness/

Interface sketch

package harness

type Capabilities struct {
    SystemPrompt    bool
    SessionResume   bool
    Skills          bool
    OTelNative      bool // child process honours OTEL_* env vars
    MaxConcurrency  int
}

type ExecRequest struct {
    AgentName  string
    Message    *messaging.Message   // triggering message
    Context    []*messaging.Message // optional conversation window
    Budget     Budget               // tokens, cost, wallclock
    Env        map[string]string // caller-provided overrides
    Skills     []string
}

type ExecResult struct {
    ExitCode   int
    Logs       string
    ResultJSON json.RawMessage
    Usage      Usage // { in, out, cached tokens, cost }
    TraceID    string // W3C, for correlation
    Err        error
}

type Harness interface {
    Name() string
    Capabilities() Capabilities
    TestEnvironment(ctx context.Context) error                 // preflight
    Provision(ctx context.Context, agent *agents.Agent) error    // one-shot setup
    Execute(ctx context.Context, req *ExecRequest) (*ExecResult, error)
    Cancel(ctx context.Context, runID string) error
}

type Registry struct { /* map[string]Harness + mutex */ }

func (r *Registry) Register(h Harness)
func (r *Registry) Resolve(agent *agents.Agent) (Harness, error)
func (r *Registry) Execute(ctx context.Context, req *ExecRequest) (*ExecResult, error)

Backend implementations

Resolution policy

Registry.Resolve(agent) picks a backend based on:

1. Explicit agent.HarnessName field (new column, nullable)
2. Else: agent has K8sImage and k8s.JobRunner.IsAvailable() → k8sjob
3. Else: agent has Webhooks registered → webhook
4. Else: agent has LocalCommand configured → subprocess
5. Else: typed error ErrNoBackend

Data model additions

• New migration 016_harness.sql: add agents.harness_name, agents.local_command, agents.harness_config_json
• New table harness_runs: mirror of ReactiveRun but backend-agnostic, with backend, trace_id, span_id, usage_in, usage_out, cost_usd, result_json
• Fold ReactiveRun into harness_runs in a follow-up migration

6 · OTel integration points

Scion's pattern is the template: (a) initialize an OTel tracer provider in the main process, (b) start a span per harness invocation, (c) inject the trace context into the child as env vars, (d) ship spans via OTLP gRPC to whatever collector is configured.

Init

New file internal/observability/otel.go:

func Init(ctx context.Context, cfg Config) (shutdown func(context.Context) error, err error) {
    res, _ := resource.New(ctx,
        resource.WithAttributes(semconv.ServiceName("synapbus")),
    )
    exp, _ := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint(cfg.Endpoint),
        otlptracegrpc.WithInsecure(),
    )
    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exp),
        sdktrace.WithResource(res),
    )
    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(propagation.TraceContext{})
    return tp.Shutdown, nil
}

Span taxonomy

Context propagation into children

For each backend, the current span's traceparent is serialised via propagation.TraceContext{}.Inject into an env-var map and merged with Harness.GetTelemetryEnv():

func injectTraceEnv(ctx context.Context, dst map[string]string) {
    carrier := propagation.MapCarrier{}
    otel.GetTextMapPropagator().Inject(ctx, carrier)
    for k, v := range carrier {
        // OTel convention: TRACEPARENT / TRACESTATE env names
        dst[strings.ToUpper(k)] = v
    }
    dst["OTEL_EXPORTER_OTLP_ENDPOINT"] = cfg.ChildEndpoint // same collector
    dst["OTEL_SERVICE_NAME"] = "synapbus-agent-" + agentName
    dst["OTEL_RESOURCE_ATTRIBUTES"] = "synapbus.run_id=" + runID
}

For K8s: merged into corev1.EnvVar slice at k8s/runner.go:105–119. For subprocess: merged into cmd.Env. For webhook: added as HTTP headers (traceparent, tracestate) alongside the existing X-SynapBus-* headers.

Metrics

Keep the existing Prometheus registry (internal/metrics/metrics.go) — it's already wired — but also emit a minimal set via OTel meter, so a single OTLP collector sees both spans and metrics:

• synapbus.harness.runs (counter, labels: harness, status)
• synapbus.harness.duration_ms (histogram)
• synapbus.harness.tokens_in / tokens_out (counters)
• synapbus.harness.cost_usd (counter)

Config

Three new env vars (matching scion naming, with SYNAPBUS_ prefix for ours):

• SYNAPBUS_OTEL_ENDPOINT — e.g. http://otel-collector:4317
• SYNAPBUS_OTEL_INSECURE — bool, default true for LAN
• SYNAPBUS_OTEL_ENABLED — bool, default false (opt-in)

Until a real collector exists on kubic, a file exporter (stdouttrace) or the existing trace.Tracer (SQLite trace table) can back the same interface via an adapter.

7 · Other reusable nuggets

8 · Next steps & open questions

Staged implementation plan (for discussion)

Open questions for you

1. Collector. Is there an OTel collector on kubic.home.arpa already, or do we deploy one first (Tempo? Jaeger? stdout only for now)?
2. Scope of Phase 1. Do you want the new package to land behind a feature flag, or replace the existing reactor path immediately?
3. Subprocess path on the Mac. SynapBus today only runs agents as K8s Jobs. The subprocess backend lets it also run claude-code / gemini-cli locally on your laptop. Is that in-scope now or defer?
4. Session codec. How much of paperclip's session-resume semantics do you want — just "reuse the Claude Code session id" or full conversation replay?
5. Plugin loader. Do we need to load third-party harnesses at runtime (plugin.Plugin / HashiCorp go-plugin), or is a compile-time registry enough?