synapbus/benchmark/requirements.txt at main - synapbus - ilapage Git

OPC/synapbus

Files

T

Algis DumbrisandClaude Opus 4.6 02b8548eac feat(017): MuSiQue benchmark harness — marketplace stub, mixed-tier agents, Pareto scoring, HTML report

MVP implementation of the MuSiQue multi-agent benchmark (spec 017):

- benchmark/setup.py: downloads musique_v1.0.zip from the canonical Google
  Drive source (mirrors upstream download_data.sh). Idempotent.
- benchmark/curate.py: deterministic selection of 3 4-hop questions from
  the dev set sharing a US pivot entity; writes benchmark/trio.jsonl.
- benchmark/marketplace.py: in-process 016-marketplace stub with
  post_auction / bid / award / mark_done / query_reputation and a
  domain-scoped reputation ledger. Designed for mechanical swap to real
  SynapBus MCP tools.
- benchmark/agents.py: HaikuAgent + SonnetAgent, using the official
  anthropic SDK (no Claude Agent SDK, no subprocesses). Models pinned
  to claude-haiku-4-5-20251001 and claude-sonnet-4-6.
- benchmark/baseline.py: single Sonnet call with all 20 distractors
  plus chain-of-thought.
- benchmark/score.py: SQuAD-style normalized F1 + Pareto verdict
  (strictly northwest = PASS).
- benchmark/run.py: main entry. --mode single-shot, --question, --dry-run.
- benchmark/report.py: self-contained HTML with inline SVG scatter plot.
- benchmark/trio.jsonl: curated reproducible trio (all three converge on
  "Treaty of Paris" US territory cession).

Verified with benchmark/run.py --dry-run end-to-end; all 8 files
py_compile clean. Real-token execution is deferred to the user's main
session.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-11 15:12:19 +03:00

3 lines

35 B

Plaintext

Raw Permalink Blame History

	`anthropic>=0.40.0`
	`requests>=2.31.0`