Multi-Agent Orchestration

Blocks World with partitioned arms

Setup

6-block tower world. Two agents each control one "arm" and see only half the blocks. Cooperate to reach a goal configuration in ≤ N moves.

Why it bites

Classic AI planning with well-defined optimum. Exposes state-sync bugs, stale-view reads, and whether the framework supports atomic claim/release — a direct analog of SynapBus claim_messages.

Werewolf / Mafia social deduction

Setup

6–8 LLM agents with hidden roles, day/night cycles as channels. Private night DMs, public day broadcast. Success = villagers win ≥ baseline rate; agents stay in character.

Why it bites

Stresses private vs. public channels, role-scoped memory, and whether the orchestrator prevents info leaks. Published baseline: Werewolf Arena (DeepMind, arXiv:2407.13943).

Iterated Prisoner's Dilemma tournament

Setup

N agents, round-robin pairings, 200 rounds each, fixed payoff matrix. Success = stable ranking across re-runs with fixed seeds.

Why it bites

Tiny payloads but huge volume. Exposes per-message overhead, trace scaling, and whether the framework can support deterministic replay for debugging. Axelrod's classic is the reference.

Collaborative story writing with critic veto

Setup

Writer + Editor + Fact-Checker + Critic produce a 2000-word story. Critic has veto; loop until approved or budget exhausted.

Why it bites

Tests unbounded loops and budget enforcement. Failure mode: infinite revision — a real production hazard for any "agent with veto" pattern.

Recursive Fermi estimate ("piano tuners in Chicago")

The classic question from physicist Enrico Fermi. You cannot look it up; you must decompose into estimable sub-quantities, estimate each, and multiply. This is a sharp MAS benchmark because the decomposition itself is the work — and decomposition is exactly what agent orchestration should be good at.

Canonical decomposition

tuners = (pianos in Chicago)
       × (tunings per piano per year)
       ÷ (tunings per tuner per year)

pianos = population × (households per capita)
                   × (piano ownership rate)
       + commercial pianos (venues, schools, churches)

Ground truth ≈ 125–250 professional tuners in the Chicago metro area. Success criterion: final estimate within one order of magnitude.

Why it's a sharp MAS test — five failure axes

Dynamic spawning

The orchestrator doesn't know up front how many sub-researchers it needs. Decomposition dictates fan-out. Tests whether the framework supports runtime subagent creation, not a predefined graph.

Duplicate-work detection

Naive orchestrators spawn two sub-agents that both research "Chicago population" independently. A well-designed system deduplicates via a shared scratchpad (blackboard!) or caches sub-results. Direct test of whether shared-memory patterns actually work.

Cost accounting

Fermi estimates are supposed to be cheap. If each sub-agent burns 50k tokens researching census data, you've failed the spirit of the task. Pairs accuracy with a hard token budget.

Aggregation under uncertainty

Each leaf estimate has an uncertainty range. A mature MAS propagates ranges, not point estimates. Tests whether agents can handle structured sub-results instead of concatenating strings.

Orphaned spawns

If a sub-agent fails or times out, does the orchestrator notice, retry, or silently drop the branch? A 5-branch Fermi estimate with one silent drop produces a confident wrong answer — the worst possible failure mode.

Concrete setup for SynapBus

• One fermi-orchestrator agent with authority to spawn ≤ 5 fermi-researcher subagents via reactive triggers.
• Shared channel #fermi-scratchpad as the blackboard — all sub-results posted here with a fixed schema.
• Each sub-agent has a WebSearch tool with per-call token limit.
• Orchestrator monitors scratchpad via list_by_state, marks the task done when all leaves posted.

Scratchpad schema (enforced)

{
  "quantity":  "chicago_population",
  "value":     2.7e6,
  "low":       2.6e6,
  "high":      2.8e6,
  "confidence": 0.95,
  "source":    "census.gov/quickfacts/chicagocityillinois",
  "parent":    null
}

Success criteria (all must hold)

• Final estimate within 1 OOM of ground truth (≥ 25 and ≤ 2,500 tuners)
• Total tokens ≤ 10k across orchestrator + all sub-agents
• Zero duplicate sub-queries (verified by grepping quantity field)
• All spawned branches reach terminal state (done or failed, never orphan)
• Aggregation uses ranges, not point estimates; final answer includes a low/high bound

Failure modes this catches that nothing else does

• Cost explosion from unbounded recursion (sub-agents spawning sub-sub-agents)
• Silent branch loss — a sub-agent returns nothing and the orchestrator averages 0 into the product
• Over-convergence — all sub-agents copying each other's bad assumption because they read the scratchpad before contributing (a subtle failure mode of premature information sharing)
• Point-estimate collapse — losing uncertainty bands so the final answer looks precise when it isn't

This is basically the Anthropic multi-agent researcher pattern reduced to a 10-minute test you can run deterministically. If your framework can't pass Fermi, it can't do Deep Research.

Grid-world treasure hunt with fog of war

Setup

10×10 grid, 3 agents, each sees 3×3 window around itself. Find and retrieve 5 treasures. Communication only via a shared "map" channel.

Why it bites

Forces publishing structured observations and merging them. Tests channel throughput and message-ordering preservation under concurrent writes.

Parallel bug triage from a synthetic issue tracker

Setup

50 synthetic bug reports (frontend/backend/infra mix). 3 specialist agents pull from a queue, classify, fix or escalate. Success = all bugs terminally resolved, zero double-processing, ≥ 90% correct routing.

Why it bites

Almost a unit test for SynapBus's claim_messages / mark_done / StalemateWorker pattern. Exposes lease-expiry bugs and whether failed messages re-queue cleanly.

Overcooked-style real-time coordination

Setup

2 agents in a simplified kitchen grid must prepare N soups under a time limit. Reference env: Carroll et al., NeurIPS 2019 (arXiv:1910.05789).

Why it bites

The only benchmark here that punishes message latency directly. If the framework adds 500ms per hop, the score shows it. Exposes whether turn-based dialogue assumptions break under real-time load.