Root cause
----------
ExpireTasks in internal/channels/task_store.go ran a single unbounded
UPDATE filtered on (status='open' AND deadline IS NOT NULL AND deadline < now).
The only indexes on tasks were idx_tasks_status(status) and
idx_tasks_channel(channel_id). With status cardinality of ~4 and a growing
auction-tasks table on kubic, the planner used idx_tasks_status to enumerate
all open rows then evaluated deadline per row, holding a SQLite write
transaction the whole time. Under WAL contention with concurrent writers
(message inserts, consolidator) the worker's 30s context regularly expired,
producing the recurring expiry-worker log line.
Fix
---
1. New migration 031_tasks_expiry_index.sql: partial composite index
idx_tasks_expiry(status, deadline) WHERE status='open' AND deadline IS NOT NULL.
This is the exact predicate ExpireTasks uses, so the planner now seeks
straight to eligible rows. The partial form keeps the index empty for the
steady-state majority of rows (completed/cancelled), so writes elsewhere
aren't penalized.
2. Batch the UPDATE in chunks of 500 (rowid IN subquery; UPDATE ... LIMIT
isn't compiled into modernc.org/sqlite by default). Bounded write
transactions stop the worker from starving other writers and let it
observe context cancellation between batches.
Perf
----
New test exercises 2400 mixed rows (1200 expirable). With the index +
batching, expiry finishes in ~3ms inside a 5s context; without the index a
regression to full status-scan would be measurably worse and is also
guarded by an EXPLAIN QUERY PLAN test.
Operational notes
-----------------
- Migration is additive and idempotent (CREATE INDEX IF NOT EXISTS). No
backfill needed; it will apply on next pod startup.
- After rollout, expiry-worker error logs should clear within one tick
(default 1m).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
