Stage B steps 3 and 4.
The filter runs after loadDailyList has proved the day's snapshot whole,
never before. Filtering first would let drift among other shops' orders
hide a hole in the ones we do want. OrderCount stays the unfiltered total
alongside AcceptedCount and ShopSkipped, so a shop missing from the
allow-list shows up as a number rather than as absent data.
An empty allow-list is a refusal, not "import everything" — the latter
looks exactly like a filter that works. Mutation-tested, along with the
detail-level shop re-check that catches a list and detail response
disagreeing about which shop an order belongs to.
ShopBreakdown counts orders per shop including skipped ones, under the
stored display name rather than SYB's spelling, so one shop cannot appear
under two spellings. #50 renders it.
Discovery lives in sybimport, not sybshop: it needs the SYB client, and
sybimport already depends on sybshop for the filter, so the reverse would
be an import cycle. It reads the list endpoint only and never adds a
shop — widening what gets imported stays an explicit action.
Read and write routes are registered separately. GoAuto inherits
go-admin's per-path permission model, so "配置仅管理员" is expressible but
not enforced in code; the grant is configured in 系统管理.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stage B step 2 (service layer; HTTP handlers next).
Rename rewrites NormalizedName along with DisplayName. Leaving the key
stale would show the new name while still matching the old one, so the
archive would look correctly configured while importing a different shop.
Mutation-tested: updating only display_name makes the rename test fail.
Duplicates are reported against the name already stored, not the one just
submitted — the two can differ only in case or character width, and
echoing back what was typed reads as the system rejecting a name it does
not have.
Delete is a soft delete carrying the id into DeletedFlag, so the same
name can be added again afterwards while past sync records keep resolving
the old row.
EnabledNames returns an empty map without error. Empty is a legitimate
state that callers must turn into "refuse to sync", never "import
everything".
MarkSeen only updates shops already on the list. A sync must not grow the
allow-list as a side effect; discovery is a separate explicit action.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stage B step 1. The shop list decides which of a SYB account's shops get
imported.
Matching is by name, so the normalized form is the real key: whitespace
trimmed, full-width ASCII folded, letters lowercased. " ABC店 ", "ＡＢＣ店"
and "abc店" are one shop. DisplayName is only ever shown to people, and
the unique index sits on the normalized column so a caller that skips
normalization cannot create rows that look identical on screen.
Interior whitespace is deliberately not collapsed: two shops differing
only there are still two shops, and merging them silently would be worse
than the duplicate this prevents.
Soft delete uses the DeletedFlag sentinel rather than a nullable
deleted_at in the unique index, which is silently inert because unique
indexes treat every NULL as distinct. Mutation-tested: dropping the
sentinel from the index makes the recreate-after-delete case fail.
The version-local migration file is included this time — the model alone
would never reach an existing database.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adding SYBSession to migrations.Migrate was not enough. Schema reaches an
existing database only through a version-local file; with the previous
version already recorded in sys_migration, Migrate never re-ran and the
table simply never appeared. The unit tests build a fresh database every
time, so they stayed green while the real database was missing a table —
which surfaced as "Error 1146: Table 'goauto.syb_session' doesn't exist"
on the first import attempt.
server/.gitignore was hiding these files. go-admin ignores version-local
because it is where generated local migrations land, but every GoAuto
migration belongs in version control; the existing ones had been forced
in with `git add -f`. Un-ignoring *.go there also recovers four migrations
that were never committed at all — 1786700000000 through 1786700300000,
covering the base schema, device registration, heartbeat and collection
execution. A fresh clone could not have built a working database.
Guard the class of mistake rather than just this instance:
- migrations.VerifyTables checks every model's table after migrating and
names what is missing along with the fix.
- The migrate command runs it, so the failure lands at migrate time
instead of at the first request that needs the table.
- initDB no longer discards migrateModel's error. Upstream had
`_ = migrateModel()` followed by an unconditional "初始化成功", so a
failed migration reported success and the launcher believed it.
Also records the two-step rule in Common-Changes: a new model needs both
the model registration and a new version file.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The remaining red block came from go-admin's own "config init" line. That
is the third informational stderr writer to surface this way, and the
first two were only fixed by silencing them one at a time — the actual
defect is in the launcher, not in any of the writers.
`2>&1 |` makes PowerShell wrap every stderr line from a native command in
an ErrorRecord, which renders as a NativeCommandError block and reads as
a failed migration regardless of what the line says. Stringifying in the
pipeline fixes the whole class, including SDK output this repo cannot
change.
Verified in Windows PowerShell from WSL: the bare pipeline produces the
red block for a plain stderr line while the stringified one does not, and
$LASTEXITCODE still reports the child's exit code (3) through the extra
stage, so the migration failure check below is unaffected. Script
parse-checked and run with -ValidateConfigOnly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PowerShell 5.1 reads a .ps1 with no BOM using the ANSI code page. Every
byte of a UTF-8 Chinese character is >= 0x80, so GBK pairs them up two at
a time; a comment line carrying an odd number of those bytes pairs its
last byte with the trailing newline and swallows it, folding the next
line into the comment. That commented out the `try {` I had added and
left `} catch { }` orphaned, which is the parse error reported at
startup — and it shifted the line numbers, which is why the earlier
PowerShell error positions never matched the file.
My previous Chinese comment survived only because its byte count happened
to be even. Both scripts are back to ASCII, matching the English already
used throughout them, with a note saying why it matters.
Also restores CRLF: .gitattributes marks *.ps1 eol=crlf, and rewriting
these files from Python had left them LF in the working tree.
Verified this time rather than handed over untested: Windows PowerShell
is reachable from WSL, so both scripts were parse-checked and run with
-ValidateConfigOnly, and the chcp block was executed on its own
(code page 65001, console encoding utf-8).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The startup output was dominated by two upstream behaviours. Migrate
printed a bare count for every already-applied migration — a lone "1"
with no version, no explanation — through the standard logger, hence
stderr, which the launcher's `2>&1 |` renders as a red PowerShell error
block that looks like a failed migration. And the version-existence check
echoed one SELECT count(*) per version. Both replaced: a Warn-level
session for the check, and one summary line on stdout.
Console encoding needed more than [Console]::OutputEncoding: PowerShell
5.1 decodes child-process output by the console code page, so chcp 65001
goes with it.
The regression test capture was wrong twice and mutation testing caught
both. It captured only stdout while the bug wrote to stderr, and it
compared lines to "1" when the logger prefixes a timestamp, so the
assertion could never fail. It now captures the standard logger too and
matches with a pattern that allows the prefix — verified by reinstating
the bug and watching the test fail.
Not verified: the PowerShell edits, which need a Windows run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every API request logged one bare "info" line with no message: upstream
calls log.WithFields(logData).Info(), and the console formatter does not
render fields, so status, latency, IP, method and URI all went nowhere.
One line of pure noise per request.
Give the line its message and demote it to debug. The payload is worth
having when diagnosing a slow or failing request, so it is recoverable
by setting logger.level back to debug rather than deleted.
settings.yml drops from trace to info accordingly. Nothing else in the
project logs below info: the many .Debug() calls are gorm SQL echo, which
is unaffected by logger level.
This patches upstream go-admin middleware, marked as such in a comment.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The startup log proved config.yaml was not being found during migrate.
Lookup covered the working directory and the executable's directory, but
the server runs from server/ while config.yaml sits at the repository
root — so it was found only when a launcher happened to export
GOAUTO_CONFIG. Anyone running `go run .` by hand got no local config at
all. Search the parent directory too, with the working directory still
winning.
The diagnostics also wrote to stderr, and the launcher pipes the server
through `2>&1 | Tee-Object`, which turns every stderr write into a
PowerShell NativeCommandError. The informational line I added to make
this debuggable was itself rendering as a red error block. They go to
stdout now.
Launchers set the console to UTF-8: Go writes UTF-8 while the console
decodes as the ANSI code page, which turned every Chinese log line into
mojibake.
Verified by running the built binary from server/ with no GOAUTO_CONFIG
set: it loads ../config.yaml and the lines survive 2>/dev/null.
Not verified: the two PowerShell edits, which need a Windows run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Diagnosing "credential not configured" required reading the source: the
local config layer was a silent no-op when it found no file, and the
import error named only environment variables even though config.yaml is
now the normal place to configure this — sending operators to look in a
file that was never the problem.
Startup now logs which config.yaml was loaded, or that none was found and
where it looked, followed by whether SYB credentials resolved and from
which layer. Presence only, never values: server logs get pasted into
tickets.
The import error now names the file that was actually consulted and
distinguishes "no config.yaml found" from "found it, but it has no syb
credentials" — two problems with different fixes that previously produced
identical text.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Running gofmt over the whole router directory added a trailing newline to
six upstream files that this ticket does not touch. Harmless, but the
repo rule is to leave unrelated changes alone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The repo already layered file defaults under environment overrides, on
both the backend (settings.yml < GOAUTO_*) and the frontend
(.env.production < process.env). What was missing was a translator for
production: config.yaml only ever existed for the PowerShell launchers,
so a packaged binary read none of it and had no database credentials
either — SYB was inheriting an existing gap, not creating one.
The server now reads config.yaml itself, between settings.yml and the
environment. Lookup is GOAUTO_CONFIG, then ./config.yaml, then beside the
executable, so a packaged binary works wherever it is started. An absent
file is not an error: containers supply everything through the
environment. Scalars are read by YAML type and coerced, so an unquoted
all-digit password cannot take startup down over a quoting detail.
This removed the need for a Read-SybConfig in PowerShell: the launcher
just hands over the path it already knows, rather than reimplementing a
YAML parser.
The server also serves the built frontend when dist is present, which is
what .env.production's empty VUE_APP_BASE_API already assumes. The
history fallback is restricted to non-API GETs, and is not installed at
all without dist, so development 404s stay 404s.
Precedence is mutation-tested: applying the local file after the
environment instead of before makes the layering test fail.
Not verified: the PowerShell change and any Windows deployment — both
need a run on the Windows side.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Runs the whole path against the live API: login, page a day's shipment
orders, fold every detail into the archive, reuse the cached session, and
re-import to prove idempotency. Tagged syblive and gated on E2E_DB, so it
never runs by accident and never picks its own database.
Verified on real data for 2026-08-19: 986 orders, 1524 detail lines,
1056 shopee archives, parse success 1395 / uncertain 125 / failed 4. The
4 failures are all empty productSpec at the source — marked failed rather
than filled in. Re-import created 0 and overwrote 1524, leaving the row
count unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every other GoAuto model declares TableName(); SYBSession did not, so
gorm silently created syb_sessions while the rest of the schema is
singular. Migration succeeds either way, which is what makes it easy to
miss — it only surfaces when someone queries the conventional name.
Caught by running a real import, before the MySQL migration had been run
anywhere.
Migrate now takes its model list from MigratedModels(), so the new
convention test asserts over exactly the set that Migrate creates instead
of a hand-copied list that would drift. Mutation-checked: dropping the
TableName method again makes the test fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stage 2. Completes the path from the SYB API to the archive.
sybimport.Sync pages a date range day by day and folds each detail line
through the existing idempotent ApplyDetail. The paging loop is driven by
listTotal, never by the list response's own total, which live data
confirmed is the page's row count rather than the filtered total. A day
whose completeness cannot be proven — a short page, a total that drifted
while paging, a detail response missing an order — stops the run instead
of reporting a partial import as a whole one. Rows already written stay:
they are idempotent on (order_code, detail_id), so a re-run overwrites.
sybimport.Connect reuses the cached session and only discards it when SYB
explicitly says it is invalid. Caching the numeric user id alongside the
cookies is required, not incidental: session validation calls
/am/user/get?id=, and a wrong id comes back as a business error rather
than a logout, so cookies alone can never be revalidated.
The import endpoint is single-flight and detached from the request
context, so closing the tab cannot abandon a half-finished range. It
reads only; no SYB write endpoint is reachable from GoAuto.
The page swaps its "no import entry" warning for a date-range dialog that
warns when the span is wide — a single week held over 7000 orders — and
reports partial progress when a run fails midway.
Verified: go build, go vet, go test ./... all pass. The drift check was
mutation-tested. Not verified: the MySQL migration for syb_session, and
any browser walkthrough.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Guarded by the syblive build tag, so `go test ./...` still runs entirely
against httptest fakes. Credentials come from the environment; the file
contains none, calls only read endpoints, and logs shapes and counts
rather than order codes, titles or amounts.
Verified against the live API. Confirmed from real data:
- login via OCR works unattended; session is exactly 24h
- §3.1 auth is cookie-based: the jar is populated after login
- §4.3 listTotal is the filtered total (7404 over 7 days) while
list.total is the current page size (20) — a paging loop that
treats the latter as the total stops after one page
- §5.1 productPrice is in yuan, not cents (max observed 403.00)
- §6.1 one order carries nested items, not repeated rows
- details[].id was unique across the batch, closing a gap #41
recorded as unverified
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Port the SYB ERP HTTP client from the upstream cmautobuy project. The
package is stdlib-only (no DB, no gin), so client.go, columns.go, ocr.go
and their httptest suites carry over almost verbatim; only the package
name and doc references changed.
Add on top of the port:
- models.SYBSession + gorm SessionStore, so a restart does not force a
fresh captcha. Cookies are credential-equivalent and carry json:"-".
Expired sessions read as absent because SYB has no rolling renewal.
- config.Extend.SYB. Non-secret settings live in settings.yml;
username and password come only from GOAUTO_SYB_* environment
variables, so no credential lands in a tracked file.
- docs/12-syb-erp-interface.md, the ported interface contract.
The guard tests were mutation-checked: reverting json:"-" and renaming a
settings.yml key each make their test fail.
No import endpoint yet — that needs real credentials and live network
verification, which is Stage 2 of #48.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The blanket "no OCR/VLM" rule was written for Android Agent collection.
Server-side SYB ERP login is a different domain and needs captcha OCR to
run unattended. Scope the prohibition to the Agent side and record the
SYB carve-out, including that captcha images leave the project.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
QiuSW