diff --git a/docs/00-project-profile.md b/docs/00-project-profile.md index 0ca61e8..36e864e 100644 --- a/docs/00-project-profile.md +++ b/docs/00-project-profile.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件) wiki_page: Project-Profile wiki_url: https://git.ilapage.cn/OPC/goauto/wiki/Project-Profile.- -wiki_revision: f92d8e766d3307ac80abfd85f664fb0ab87ec4c2 -synchronized_at: 2026-08-20T03:52:46Z +wiki_revision: c94df9467c938231fb127647d4956c1913375610 +synchronized_at: 2026-08-20T07:13:07Z # 项目档案 @@ -17,7 +17,7 @@ synchronized_at: 2026-08-20T03:52:46Z | 主要使用者 | 管理员、采购人员、开发维护者 | | Gitea 仓库 | `OPC/goauto` | | 默认分支 | `main` | -| 当前已实施范围 | PDD 商品与采集闭环、Shopee/SYB 商品档案、SYB 接口导入与店铺准入管理 | +| 当前已实施范围 | PDD 商品与采集闭环、Shopee/SYB 商品档案、SYB 接口导入、店铺准入管理与后台同步记录 | | 后续设计范围 | 采购演练、创建待付款订单和物流回填 | | 预计规模 | 20 台 Android；每天约 100 个采集任务、200 个采购任务 | @@ -69,3 +69,5 @@ synchronized_at: 2026-08-20T03:52:46Z #31 PDD 商品档案原型、数据库、API、Admin 页面和采集结果写回已于 2026-08-17 通过用户验收；#45 PDD 商品列表批量采集入口已于 2026-08-18 通过用户验收。#32 采购闭环原型及后续采购工单仍按各自门禁推进。采购永不支付，真实地址修改和创建订单属于必须再次人工确认的高风险范围。 #49 已完成 SYB 店铺表、管理 API、Admin 只读/管理界面、真实列表发现及导入双重过滤，正在等待用户验收。真实 MySQL 迁移和既有错误导入数据清理尚未执行；清理必须先确认精确 SQL 与影响行数。 + +#50 原型已于 2026-08-20 通过用户验收，后台导入任务、持久化进度、同步记录页面、单任务互斥和启动中断恢复已实现，正在等待代码验收；尚未对本地真实 MySQL 执行新增迁移，也未用真实 SYB 长任务做在线验证。 diff --git a/docs/02-architecture-and-code-map.md b/docs/02-architecture-and-code-map.md index 24aaf5e..b4ef1ca 100644 --- a/docs/02-architecture-and-code-map.md +++ b/docs/02-architecture-and-code-map.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件) wiki_page: Architecture-and-Code-Map wiki_url: https://git.ilapage.cn/OPC/goauto/wiki/Architecture-and-Code-Map.- -wiki_revision: f92d8e766d3307ac80abfd85f664fb0ab87ec4c2 -synchronized_at: 2026-08-20T03:52:51Z +wiki_revision: c94df9467c938231fb127647d4956c1913375610 +synchronized_at: 2026-08-20T07:13:07Z # 架构与代码地图 @@ -111,10 +111,12 @@ Android Portal/Agent | 顺云宝（SYB）ERP HTTP 客户端与登录会话 | `server/app/goauto/sybclient/`（登录、OCR 验证码、会话缓存、列表与明细读取；见 [SYB-ERP-Interface-Contract](SYB-ERP-Interface-Contract)） | | SYB 商品明细增量迁移 | `server/cmd/migrate/migration/version-local/1786700700000_syb_product_import.go` | | SYB 店铺准入、发现与过滤 | `server/app/goauto/sybshop/`、`server/app/goauto/sybimport/`；迁移 `server/cmd/migrate/migration/version-local/1786700900000_syb_shop.go` | +| SYB 后台导入任务、进度、单任务互斥与启动恢复 | `server/app/goauto/sybimport/sync_run.go`、`sync_run_handler.go`；表 `syb_sync_run`，迁移 `server/cmd/migrate/migration/version-local/1786701000000_syb_sync_run.go` | | 任务领取、结果、重置与删除 | `server/app/goauto/task/` | | 管理端基线 | `web/`（go-admin-ui v3.0.0） | | 管理端闭环页面 | `web/src/views/goauto/` | | SYB 店铺管理页面与接口封装 | `web/src/views/goauto/syb-shops/`、`web/src/api/goauto/syb-shops.js`；确认原型快照 `prototypes/49/v1/index.html` | +| SYB 异步导入入口与同步记录页面 | `web/src/views/goauto/syb-products/`、`web/src/views/goauto/syb-sync-runs/`、`web/src/api/goauto/syb-products.js`；确认原型快照 `prototypes/50/v2/index.html` | | Android Agent 基线 | `android/app/src/main/java/cn/ilapage/goauto/agent/` | | Android 无障碍规则执行 | `android/app/src/main/java/cn/ilapage/goauto/agent/automation/` | | 三端统一验证 | `scripts/verify.ps1` | diff --git a/docs/03-business-rules-and-glossary.md b/docs/03-business-rules-and-glossary.md index 902acb6..07539a5 100644 --- a/docs/03-business-rules-and-glossary.md +++ b/docs/03-business-rules-and-glossary.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件) wiki_page: Business-Rules-and-Glossary wiki_url: https://git.ilapage.cn/OPC/goauto/wiki/Business-Rules-and-Glossary.- -wiki_revision: f92d8e766d3307ac80abfd85f664fb0ab87ec4c2 -synchronized_at: 2026-08-20T03:52:54Z +wiki_revision: c94df9467c938231fb127647d4956c1913375610 +synchronized_at: 2026-08-20T07:13:07Z # 业务规则与术语 @@ -54,6 +54,13 @@ synchronized_at: 2026-08-20T03:52:54Z - 只有列表与明细响应的店铺名都非空且命中启用快照时才允许写入。明细店铺名为空、变化为未启用店铺或无法匹配时跳过，并计入跳过数量。 - 停用或软删除店铺只影响后续导入，不删除历史 SYB 商品、虾皮商品或任务数据。已有错误导入数据的清理必须先给出精确 SQL 和影响行数，再由用户单独确认。 +## SYB 后台导入记录 + +- 导入由管理员创建，创建成功后立即转入后台执行；页面关闭不取消任务。采购员等其他已登录角色可以查看同步记录，不能开始导入。 +- 系统同一时刻只运行一个 SYB 导入任务。`syb_sync_run` 的唯一执行槽负责跨进程互斥，内存锁减少同一进程内的竞争；服务重启后遗留的执行中记录标记为“已中断”。 +- 导入失败或中断时保留已写入商品，记录明确的失败原因和已处理进度；重新导入相同范围按「货运单号 + 明细 ID」覆盖，不产生重复商品。 +- 每条同步记录保存本次启用店铺集合的 SHA-256 哈希以及各店铺“已导入/已跳过”数量，不保存账号、密码、Cookie、验证码图片或 SYB 原始响应。 + ## 采集规则 - 规则创建后立即可用，不存在草稿、发布、版本或停用流程。 diff --git a/docs/08-agent-api-contract.md b/docs/08-agent-api-contract.md index b495ccd..6a201be 100644 --- a/docs/08-agent-api-contract.md +++ b/docs/08-agent-api-contract.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件) wiki_page: Android-Agent-API-Contract wiki_url: https://git.ilapage.cn/OPC/goauto/wiki/Android-Agent-API-Contract.- -wiki_revision: f92d8e766d3307ac80abfd85f664fb0ab87ec4c2 -synchronized_at: 2026-08-20T03:53:07Z +wiki_revision: c94df9467c938231fb127647d4956c1913375610 +synchronized_at: 2026-08-20T07:13:07Z # MVP 共享 API 契约 @@ -63,12 +63,16 @@ POST /api/admin/v1/shopee-products/batch-delete GET /api/admin/v1/syb-products GET /api/admin/v1/syb-products/{productId} POST /api/admin/v1/syb-products/import +GET /api/admin/v1/syb-products/sync-runs +GET /api/admin/v1/syb-products/sync-runs/{runId} POST /api/admin/v1/syb-products/{productId}/reparse POST /api/admin/v1/syb-products/reparse-batch PATCH /api/admin/v1/syb-products/{productId}/correction ``` -`import` 提交日期范围后同步调用 SYB 客户端完成列表完整性校验、启用店铺过滤、明细读取和幂等落库。没有启用店铺时必须在读取凭据、登录、验证码 OCR 和任意 SYB 网络请求之前返回 `422`；#50 将另行把长耗时导入改为后台任务，本接口当前仍为同步执行。 +`import` 仅允许管理员调用。请求体提交 `dateFrom`、`dateTo` 后创建持久化后台任务并立即以 `202` 返回 `runId` 和 `status=running`；关闭弹窗、刷新或离开页面不影响任务。没有启用店铺时必须在读取凭据、登录、验证码 OCR 和任意 SYB 网络请求之前返回 `422`。任意时刻只能有一条 `running` 记录，内存锁与数据库唯一执行槽共同阻止单进程和跨进程重复导入；冲突时返回正在执行任务的日期范围。 + +`sync-runs` 列表支持 `page`、`pageSize`、`status`、`dateFrom`、`dateTo`，详情返回日期范围、状态（`running` / `succeeded` / `failed` / `interrupted`）、处理天数、货运单/明细/新增/覆盖数量、店铺准入与跳过数量、店铺筛选快照哈希、按店铺的 `accepted` / `skipped` 统计、操作人和起止时间。列表和详情对已登录角色只读开放。服务启动时遗留的 `running` 任务改为 `interrupted`；中途失败或中断已经写入的数据保留，重新导入仍按唯一键覆盖。 列表返回结构化字段（`orderCode`、`shopeeItemId`、`productTitle`、`targetColor`、`targetSize`、`quantity`、`unitPriceCent`、`imageUrl`、`parseStatus`、`parseNote`、`manuallyConfirmed`），不含原始 JSON；`keyword` 匹配货运单号、虾皮商品ID 或商品标题，`parseStatus` 筛选 `success`/`uncertain`/`failed`。详情额外返回 `rawJson`（原始 `details[]` 元素，未做任何改写）。 diff --git a/docs/09-delivery-issues.md b/docs/09-delivery-issues.md index e969948..fa18708 100644 --- a/docs/09-delivery-issues.md +++ b/docs/09-delivery-issues.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件) wiki_page: Delivery-Issues wiki_url: https://git.ilapage.cn/OPC/goauto/wiki/Delivery-Issues.- -wiki_revision: f92d8e766d3307ac80abfd85f664fb0ab87ec4c2 -synchronized_at: 2026-08-20T03:53:09Z +wiki_revision: c94df9467c938231fb127647d4956c1913375610 +synchronized_at: 2026-08-20T07:13:07Z # 当前 MVP 交付工单索引 @@ -66,7 +66,7 @@ synchronized_at: 2026-08-20T03:53:09Z | T43 | [#45](https://git.ilapage.cn/OPC/goauto/issues/45) | Admin PDD 商品列表多选与批量采集任务创建（2026-08-18 已验收） | 已完成 | | T46 | [#48](https://git.ilapage.cn/OPC/goauto/issues/48) | SYB ERP 客户端移植与真实导入 | #41；已实施，等待验收 | | T47 | [#49](https://git.ilapage.cn/OPC/goauto/issues/49) | 店铺管理与 SYB 同步店铺过滤 | #48；代码与原型已完成，等待验收；真实迁移和历史数据清理需另行确认 | -| T48 | [#50](https://git.ilapage.cn/OPC/goauto/issues/50) | SYB 导入后台任务与进度展示 | #48、#49 | +| T48 | [#50](https://git.ilapage.cn/OPC/goauto/issues/50) | SYB 导入后台任务与进度展示 | #48、#49；原型已验收，代码已实现并等待验收 | 推荐依赖顺序：#31、#40、#41 完成商品域 → #33、#34 建立采购契约和服务端状态机 → #42 完成不下单演练 → #35 管理端人工处理 → #36 高风险真实订单动作 → #37、#38 物流闭环 → #39 真机总验收。 diff --git a/server/app/goauto/migrations/migrate.go b/server/app/goauto/migrations/migrate.go index f3f609f..2ee2537 100644 --- a/server/app/goauto/migrations/migrate.go +++ b/server/app/goauto/migrations/migrate.go @@ -19,6 +19,7 @@ func MigratedModels() []any { &models.SYBProduct{}, &models.SYBSession{}, &models.SYBShop{}, + &models.SYBSyncRun{}, &models.CollectionRule{}, &models.CollectionTask{}, &models.CollectionDimension{}, diff --git a/server/app/goauto/models/schema.go b/server/app/goauto/models/schema.go index b75985a..88554ee 100644 --- a/server/app/goauto/models/schema.go +++ b/server/app/goauto/models/schema.go @@ -422,6 +422,36 @@ type SYBShop struct { func (SYBShop) TableName() string { return "syb_shop" } +// SYBSyncRun records one asynchronous SYB import. ActiveSlot is set only while +// a run is active; its unique index is the database-level single-flight guard +// across processes and is cleared when the run reaches a terminal state. +type SYBSyncRun struct { + ID uint64 `json:"id" gorm:"primaryKey;autoIncrement"` + DateFrom string `json:"dateFrom" gorm:"size:10;not null"` + DateTo string `json:"dateTo" gorm:"size:10;not null"` + Status string `json:"status" gorm:"size:16;not null;index"` + ActiveSlot *uint8 `json:"-" gorm:"uniqueIndex:ux_syb_sync_run_active_slot"` + DaysTotal int `json:"daysTotal" gorm:"not null;default:0"` + DaysProcessed int `json:"daysProcessed" gorm:"not null;default:0"` + OrderCount int `json:"orderCount" gorm:"not null;default:0"` + DetailCount int `json:"detailCount" gorm:"not null;default:0"` + AcceptedCount int `json:"acceptedCount" gorm:"not null;default:0"` + ShopSkipped int `json:"shopSkipped" gorm:"not null;default:0"` + Created int `json:"created" gorm:"not null;default:0"` + Updated int `json:"updated" gorm:"not null;default:0"` + ShopFilterHash string `json:"shopFilterHash" gorm:"size:64;not null"` + ShopBreakdownJSON string `json:"-" gorm:"type:text;not null"` + ErrorMessage string `json:"errorMessage" gorm:"size:1000;not null;default:''"` + OperatorID uint64 `json:"operatorId" gorm:"not null;default:0"` + OperatorName string `json:"operatorName" gorm:"size:128;not null;default:''"` + StartedAt time.Time `json:"startedAt" gorm:"not null"` + FinishedAt *time.Time `json:"finishedAt"` + CreatedAt time.Time `json:"createdAt"` + UpdatedAt time.Time `json:"updatedAt"` +} + +func (SYBSyncRun) TableName() string { return "syb_sync_run" } + func (product *SYBProduct) BeforeCreate(_ *gorm.DB) error { if product.RawJSON == "" { product.RawJSON = "{}" diff --git a/server/app/goauto/sybimport/import_handler.go b/server/app/goauto/sybimport/import_handler.go index ad4e91a..a4d0a35 100644 --- a/server/app/goauto/sybimport/import_handler.go +++ b/server/app/goauto/sybimport/import_handler.go @@ -3,6 +3,7 @@ package sybimport import ( "context" "net/http" + "strconv" "sync" "time" @@ -11,6 +12,8 @@ import ( "go-admin/config" "github.com/gin-gonic/gin" + jwt "github.com/go-admin-team/go-admin-core/sdk/pkg/jwtauth" + "gorm.io/gorm" ) // importTimeout bounds one import run. A month-wide range over a busy account @@ -38,6 +41,10 @@ type ImportRequest struct { // `[必须]` This is the only endpoint that reaches out to SYB. It performs reads // only — no SYB write endpoint is called from anywhere in GoAuto. func (handler Handler) Import(c *gin.Context) { + if claimString(jwt.ExtractClaims(c)["rolekey"]) != "admin" { + c.JSON(http.StatusForbidden, gin.H{"code": "FORBIDDEN", "message": "只有管理员可以开始导入"}) + return + } var request ImportRequest if err := c.ShouldBindJSON(&request); err != nil { writeError(c, invalidRequest("请求体必须是合法 JSON，且包含 dateFrom 和 dateTo")) @@ -86,47 +93,66 @@ func (handler Handler) Import(c *gin.Context) { writeError(c, &ServiceError{Code: CodeInvalidRequest, Message: "已有一个导入任务正在执行，请等它结束后再试"}) return } - importGate.running = true - importGate.Unlock() - defer func() { - importGate.Lock() - importGate.running = false - importGate.Unlock() - }() - - // Detached from the request context on purpose: an operator closing the tab - // must not abort a half-finished import, which would leave a partial range - // imported with nobody holding the report that says how far it got. - ctx, cancel := context.WithTimeout(context.Background(), importTimeout) - defer cancel() - - client, err := Connect(ctx, sybclient.NewSessionStore(service.DB), ConnectConfig{ - BaseURL: settings.BaseURL, - Username: settings.Username, - Password: settings.Password, - OcrURL: settings.OcrURL, - OcrMaxAttempts: settings.OcrMaxAttempts, + claims := jwt.ExtractClaims(c) + run, err := NewSyncRunService(service.DB).Create(c.Request.Context(), CreateSyncRunInput{ + DateFrom: request.DateFrom, DateTo: request.DateTo, ShopFilterHash: enabledShopHash(enabled), + OperatorID: claimUint64(claims["identity"]), OperatorName: claimString(claims["nice"]), }) if err != nil { - writeError(c, &ServiceError{Code: CodeInvalidRequest, Message: err.Error()}) + importGate.Unlock() + writeError(c, err) return } + importGate.running = true + importGate.Unlock() - report, err := Sync(ctx, service.DB, client, SyncConfig{ - PageSize: settings.PageSize, MaxMatches: settings.MaxMatches, - }, request.DateFrom, request.DateTo) - if err != nil { - // The partial report goes back with the error: rows already written are - // kept (a re-run overwrites them), so the operator needs to see how far - // it got, not just that it failed. - c.JSON(http.StatusBadGateway, gin.H{ - "code": CodeInvalidRequest, "message": err.Error(), "data": report, - }) - return - } - c.JSON(http.StatusOK, gin.H{"code": 200, "data": report}) + go runImport(service.DB, run.ID, request, settings) + c.JSON(http.StatusAccepted, gin.H{"code": 200, "data": gin.H{"runId": run.ID, "status": run.Status}}) } +func runImport(db *gorm.DB, runID uint64, request ImportRequest, settings config.SYB) { + defer func() { importGate.Lock(); importGate.running = false; importGate.Unlock() }() + ctx, cancel := context.WithTimeout(context.Background(), importTimeout) + defer cancel() + runs := NewSyncRunService(db) + report := SyncReport{From: request.DateFrom, To: request.DateTo, StartedAt: time.Now().UTC(), ShopBreakdown: map[string]ShopBreakdown{}} + client, err := Connect(ctx, sybclient.NewSessionStore(db), ConnectConfig{ + BaseURL: settings.BaseURL, Username: settings.Username, Password: settings.Password, + OcrURL: settings.OcrURL, OcrMaxAttempts: settings.OcrMaxAttempts, + }) + if err == nil { + report, err = SyncWithProgress(ctx, db, client, SyncConfig{PageSize: settings.PageSize, MaxMatches: settings.MaxMatches}, request.DateFrom, request.DateTo, + func(progress SyncProgress) error { return runs.UpdateProgress(ctx, runID, progress) }) + } + status := SyncRunSucceeded + if err != nil { + status = SyncRunFailed + } + finishCtx, finishCancel := context.WithTimeout(context.Background(), 10*time.Second) + defer finishCancel() + _ = runs.Finish(finishCtx, runID, status, report, err) +} + +func claimUint64(value any) uint64 { + switch typed := value.(type) { + case float64: + return uint64(typed) + case int: + return uint64(typed) + case int64: + return uint64(typed) + case uint64: + return typed + case string: + parsed, _ := strconv.ParseUint(typed, 10, 64) + return parsed + default: + return 0 + } +} + +func claimString(value any) string { text, _ := value.(string); return text } + type DiscoverRequest struct { DateFrom string `json:"dateFrom"` DateTo string `json:"dateTo"` diff --git a/server/app/goauto/sybimport/import_handler_test.go b/server/app/goauto/sybimport/import_handler_test.go index 2d1f4be..99c31aa 100644 --- a/server/app/goauto/sybimport/import_handler_test.go +++ b/server/app/goauto/sybimport/import_handler_test.go @@ -11,6 +11,7 @@ import ( "go-admin/app/goauto/models" "github.com/gin-gonic/gin" + jwt "github.com/go-admin-team/go-admin-core/sdk/pkg/jwtauth" "gorm.io/driver/sqlite" "gorm.io/gorm" ) @@ -28,6 +29,7 @@ func TestImportRejectsNoEnabledShopBeforeConnect(t *testing.T) { gin.SetMode(gin.TestMode) engine := gin.New() + engine.Use(func(c *gin.Context) { c.Set(jwt.JwtPayloadKey, jwt.MapClaims{"rolekey": "admin"}); c.Next() }) engine.POST("/api/admin/v1/syb-products/import", Handler{DB: db}.Import) body := bytes.NewBufferString(`{"dateFrom":"2026-08-19","dateTo":"2026-08-19"}`) request := httptest.NewRequest(http.MethodPost, "/api/admin/v1/syb-products/import", body) @@ -48,3 +50,17 @@ func TestImportRejectsNoEnabledShopBeforeConnect(t *testing.T) { t.Fatalf("unexpected response: %s", recorder.Body.String()) } } + +func TestImportRequiresAdminRole(t *testing.T) { + gin.SetMode(gin.TestMode) + engine := gin.New() + engine.Use(func(c *gin.Context) { c.Set(jwt.JwtPayloadKey, jwt.MapClaims{"rolekey": "purchaser"}); c.Next() }) + engine.POST("/api/admin/v1/syb-products/import", Handler{}.Import) + request := httptest.NewRequest(http.MethodPost, "/api/admin/v1/syb-products/import", bytes.NewBufferString(`{"dateFrom":"2026-08-19","dateTo":"2026-08-19"}`)) + request.Header.Set("Content-Type", "application/json") + recorder := httptest.NewRecorder() + engine.ServeHTTP(recorder, request) + if recorder.Code != http.StatusForbidden || !strings.Contains(recorder.Body.String(), "只有管理员") { + t.Fatalf("采购员应被拒绝: status=%d body=%s", recorder.Code, recorder.Body.String()) + } +} diff --git a/server/app/goauto/sybimport/router.go b/server/app/goauto/sybimport/router.go index 393695d..8e0552d 100644 --- a/server/app/goauto/sybimport/router.go +++ b/server/app/goauto/sybimport/router.go @@ -7,9 +7,9 @@ import ( jwt "github.com/go-admin-team/go-admin-core/sdk/pkg/jwtauth" ) -// InitRouter mounts the read/reparse/manual-correct surface plus the import -// trigger (#48). Import is the only route that reaches out to SYB, and it only -// reads: no SYB write endpoint is called from anywhere in GoAuto. +// InitRouter mounts the read/reparse/manual-correct surface plus the async +// import trigger and read-only run history (#50). Import is the only route that +// reaches out to SYB, and it only reads. func InitRouter(engine *gin.Engine, auth *jwt.GinJWTMiddleware) { handler := Handler{} admin := engine.Group("/api/admin/v1/syb-products").Use(auth.MiddlewareFunc()).Use(middleware.AuthCheckRole()) @@ -18,6 +18,11 @@ func InitRouter(engine *gin.Engine, auth *jwt.GinJWTMiddleware) { admin.POST("/:productId/reparse", handler.Reparse) admin.POST("/reparse-batch", handler.ReparseBatch) admin.POST("/import", handler.Import) + // Run history is intentionally authenticated but not Casbin-gated: #50 + // makes it read-only for every role, while Import enforces admin itself. + readonly := engine.Group("/api/admin/v1/syb-products").Use(auth.MiddlewareFunc()) + readonly.GET("/sync-runs", handler.ListSyncRuns) + readonly.GET("/sync-runs/:runId", handler.SyncRunDetail) // 「从 SYB 发现店铺」挂在店铺路径下，但实现放在本包：它需要 SYB 客户端， // 而本包已经依赖 sybshop 做导入过滤，反过来会形成包循环。 diff --git a/server/app/goauto/sybimport/router_test.go b/server/app/goauto/sybimport/router_test.go index 837c4af..8710098 100644 --- a/server/app/goauto/sybimport/router_test.go +++ b/server/app/goauto/sybimport/router_test.go @@ -15,12 +15,16 @@ func TestRoutesRegisterWithoutConflict(t *testing.T) { group := engine.Group("/api/admin/v1/syb-products") group.GET("", handler.List) group.GET("/:productId", handler.Detail) + group.GET("/sync-runs", handler.ListSyncRuns) + group.GET("/sync-runs/:runId", handler.SyncRunDetail) group.POST("/:productId/reparse", handler.Reparse) group.POST("/reparse-batch", handler.ReparseBatch) group.POST("/import", handler.Import) want := map[string]bool{ "GET /api/admin/v1/syb-products": false, + "GET /api/admin/v1/syb-products/sync-runs": false, + "GET /api/admin/v1/syb-products/sync-runs/:runId": false, "GET /api/admin/v1/syb-products/:productId": false, "POST /api/admin/v1/syb-products/:productId/reparse": false, "POST /api/admin/v1/syb-products/reparse-batch": false, diff --git a/server/app/goauto/sybimport/sync.go b/server/app/goauto/sybimport/sync.go index fbd4787..1407529 100644 --- a/server/app/goauto/sybimport/sync.go +++ b/server/app/goauto/sybimport/sync.go @@ -2,9 +2,12 @@ package sybimport import ( "context" + "crypto/sha256" "encoding/json" "errors" "fmt" + "sort" + "strings" "time" "go-admin/app/goauto/sybclient" @@ -51,17 +54,31 @@ type SyncReport struct { // allow-list obvious rather than silent. AcceptedCount int `json:"acceptedCount"` ShopSkipped int `json:"shopSkipped"` - // ShopBreakdown counts orders per shop display name, skipped ones included. - // Without it a newly opened shop is never noticed (#50). - ShopBreakdown map[string]int `json:"shopBreakdown"` - OrderCount int `json:"orderCount"` - DetailCount int `json:"detailCount"` - Created int `json:"created"` - Updated int `json:"updated"` - StartedAt time.Time `json:"startedAt"` - FinishedAt time.Time `json:"finishedAt"` + // ShopBreakdown separates accepted and skipped orders for every shop. Without + // it a renamed or newly opened shop is never noticed (#50). + ShopBreakdown map[string]ShopBreakdown `json:"shopBreakdown"` + ShopFilterHash string `json:"shopFilterHash"` + OrderCount int `json:"orderCount"` + DetailCount int `json:"detailCount"` + Created int `json:"created"` + Updated int `json:"updated"` + StartedAt time.Time `json:"startedAt"` + FinishedAt time.Time `json:"finishedAt"` } +type ShopBreakdown struct { + Accepted int `json:"accepted"` + Skipped int `json:"skipped"` +} + +type SyncProgress struct { + Report SyncReport + DaysTotal int + DaysProcessed int +} + +type ProgressFunc func(SyncProgress) error + // Sync pulls every shipment order in [dateFrom, dateTo] and folds each detail // line into the SYB/Shopee archive through ApplyDetail. // @@ -80,8 +97,12 @@ type SyncReport struct { // list endpoint returns the *page* size in `total` (§4.3, confirmed against // live data), so the paging loop is driven by listTotal, never by list.total. func Sync(ctx context.Context, db *gorm.DB, client *sybclient.Client, cfg SyncConfig, dateFrom, dateTo string) (SyncReport, error) { + return SyncWithProgress(ctx, db, client, cfg, dateFrom, dateTo, nil) +} + +func SyncWithProgress(ctx context.Context, db *gorm.DB, client *sybclient.Client, cfg SyncConfig, dateFrom, dateTo string, progress ProgressFunc) (SyncReport, error) { report := SyncReport{From: dateFrom, To: dateTo, StartedAt: time.Now().UTC(), - ShopBreakdown: map[string]int{}} + ShopBreakdown: map[string]ShopBreakdown{}} dates, err := splitDateRange(dateFrom, dateTo) if err != nil { @@ -96,6 +117,16 @@ func Sync(ctx context.Context, db *gorm.DB, client *sybclient.Client, cfg SyncCo if len(allowed) == 0 { return report, ErrNoEnabledShop } + report.ShopFilterHash = enabledShopHash(allowed) + emit := func(daysProcessed int) error { + if progress == nil { + return nil + } + return progress(SyncProgress{Report: report, DaysTotal: len(dates), DaysProcessed: daysProcessed}) + } + if err := emit(0); err != nil { + return report, err + } pageSize := cfg.PageSize if pageSize <= 0 { pageSize = 20 @@ -131,8 +162,11 @@ func Sync(ctx context.Context, db *gorm.DB, client *sybclient.Client, cfg SyncCo plans = append(plans, dayPlan{date: date, total: total}) } - for _, plan := range plans { + for dayIndex, plan := range plans { if plan.total == 0 { + if err := emit(dayIndex + 1); err != nil { + return report, err + } continue } // `[必须]` Completeness first, filtering second. loadDailyList proves the @@ -157,16 +191,24 @@ func Sync(ctx context.Context, db *gorm.DB, client *sybclient.Client, cfg SyncCo if ok { label = display } - report.ShopBreakdown[label]++ if !ok { + entry := report.ShopBreakdown[label] + entry.Skipped++ + report.ShopBreakdown[label] = entry report.ShopSkipped++ continue } + entry := report.ShopBreakdown[label] + entry.Accepted++ + report.ShopBreakdown[label] = entry report.AcceptedCount++ byID[row.ID] = row ids = append(ids, row.ID) } if len(ids) == 0 { + if err := emit(dayIndex + 1); err != nil { + return report, err + } continue } @@ -191,11 +233,13 @@ func Sync(ctx context.Context, db *gorm.DB, client *sybclient.Client, cfg SyncCo // row for a shop nobody enabled. name := stringField(detail.Raw, "shopName") if sybshop.IsBlank(name) { + moveAcceptedToSkipped(&report, stringField(byID[detail.ID].Raw, "shopName"), "(无店铺名)", allowed) report.AcceptedCount-- report.ShopSkipped++ continue } if _, ok := allowed[sybshop.Normalize(name)]; !ok { + moveAcceptedToSkipped(&report, stringField(byID[detail.ID].Raw, "shopName"), name, allowed) report.AcceptedCount-- report.ShopSkipped++ continue @@ -204,6 +248,12 @@ func Sync(ctx context.Context, db *gorm.DB, client *sybclient.Client, cfg SyncCo return report, err } } + if err := emit(dayIndex); err != nil { + return report, err + } + } + if err := emit(dayIndex + 1); err != nil { + return report, err } } @@ -211,6 +261,35 @@ func Sync(ctx context.Context, db *gorm.DB, client *sybclient.Client, cfg SyncCo return report, nil } +func enabledShopHash(allowed map[string]string) string { + names := make([]string, 0, len(allowed)) + for normalized := range allowed { + names = append(names, normalized) + } + sort.Strings(names) + sum := sha256.Sum256([]byte(strings.Join(names, "\n"))) + return fmt.Sprintf("%x", sum) +} + +func moveAcceptedToSkipped(report *SyncReport, listName, detailName string, allowed map[string]string) { + acceptedLabel := listName + if display, ok := allowed[sybshop.Normalize(listName)]; ok { + acceptedLabel = display + } + accepted := report.ShopBreakdown[acceptedLabel] + if accepted.Accepted > 0 { + accepted.Accepted-- + } + report.ShopBreakdown[acceptedLabel] = accepted + label := detailName + if sybshop.IsBlank(label) { + label = "(无店铺名)" + } + skipped := report.ShopBreakdown[label] + skipped.Skipped++ + report.ShopBreakdown[label] = skipped +} + // loadDailyList pages through one day and refuses to return a list it cannot // prove is complete. // diff --git a/server/app/goauto/sybimport/sync_run.go b/server/app/goauto/sybimport/sync_run.go new file mode 100644 index 0000000..ce6f6a2 --- /dev/null +++ b/server/app/goauto/sybimport/sync_run.go @@ -0,0 +1,222 @@ +package sybimport + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "sort" + "strings" + "time" + + "go-admin/app/goauto/models" + "go-admin/app/goauto/sybshop" + + "gorm.io/gorm" +) + +const ( + SyncRunRunning = "running" + SyncRunSucceeded = "succeeded" + SyncRunFailed = "failed" + SyncRunInterrupted = "interrupted" +) + +type SyncRunService struct{ db *gorm.DB } + +func NewSyncRunService(db *gorm.DB) *SyncRunService { return &SyncRunService{db: db} } + +type CreateSyncRunInput struct { + DateFrom, DateTo, ShopFilterHash string + OperatorID uint64 + OperatorName string +} + +func (s *SyncRunService) Create(ctx context.Context, input CreateSyncRunInput) (*models.SYBSyncRun, error) { + dates, err := splitDateRange(input.DateFrom, input.DateTo) + if err != nil { + return nil, invalidRequest(err.Error()) + } + slot := uint8(1) + run := &models.SYBSyncRun{ + DateFrom: input.DateFrom, DateTo: input.DateTo, Status: SyncRunRunning, + ActiveSlot: &slot, DaysTotal: len(dates), ShopFilterHash: input.ShopFilterHash, + ShopBreakdownJSON: "{}", OperatorID: input.OperatorID, + OperatorName: strings.TrimSpace(input.OperatorName), StartedAt: time.Now().UTC(), + } + if err := s.db.WithContext(ctx).Create(run).Error; err != nil { + var active models.SYBSyncRun + if findErr := s.db.WithContext(ctx).Where("status = ?", SyncRunRunning).Order("id desc").First(&active).Error; findErr == nil { + return nil, invalidRequest(fmt.Sprintf("已有导入任务正在执行（%s 至 %s），请等它结束后再试", active.DateFrom, active.DateTo)) + } + return nil, internalError(err) + } + return run, nil +} + +func (s *SyncRunService) UpdateProgress(ctx context.Context, id uint64, progress SyncProgress) error { + breakdown, err := json.Marshal(progress.Report.ShopBreakdown) + if err != nil { + return err + } + return s.db.WithContext(ctx).Model(&models.SYBSyncRun{}). + Where("id = ? AND status = ?", id, SyncRunRunning).Updates(map[string]any{ + "days_total": progress.DaysTotal, "days_processed": progress.DaysProcessed, + "order_count": progress.Report.OrderCount, "detail_count": progress.Report.DetailCount, + "accepted_count": progress.Report.AcceptedCount, "shop_skipped": progress.Report.ShopSkipped, + "created": progress.Report.Created, "updated": progress.Report.Updated, + "shop_filter_hash": progress.Report.ShopFilterHash, "shop_breakdown_json": string(breakdown), + }).Error +} + +func (s *SyncRunService) Finish(ctx context.Context, id uint64, status string, report SyncReport, runErr error) error { + if status != SyncRunSucceeded && status != SyncRunFailed { + return fmt.Errorf("invalid terminal sync status %q", status) + } + breakdown, err := json.Marshal(report.ShopBreakdown) + if err != nil { + return err + } + now := time.Now().UTC() + message := "" + if runErr != nil { + message = truncateRunError(runErr.Error(), 1000) + } + updates := map[string]any{ + "status": status, "active_slot": nil, "finished_at": now, "error_message": message, + "order_count": report.OrderCount, "detail_count": report.DetailCount, + "accepted_count": report.AcceptedCount, "shop_skipped": report.ShopSkipped, + "created": report.Created, "updated": report.Updated, + "shop_breakdown_json": string(breakdown), + } + if report.ShopFilterHash != "" { + updates["shop_filter_hash"] = report.ShopFilterHash + } + if status == SyncRunSucceeded { + updates["days_processed"] = gorm.Expr("days_total") + } + return s.db.WithContext(ctx).Transaction(func(tx *gorm.DB) error { + result := tx.Model(&models.SYBSyncRun{}).Where("id = ? AND status = ?", id, SyncRunRunning).Updates(updates) + if result.Error != nil { + return result.Error + } + if result.RowsAffected == 0 { + return nil + } + if status != SyncRunSucceeded { + return nil + } + for shopName, counts := range report.ShopBreakdown { + if counts.Accepted == 0 { + continue + } + count := counts.Accepted + if err := tx.Model(&models.SYBShop{}).Where("normalized_name = ?", sybshop.Normalize(shopName)).Updates(map[string]any{ + "last_seen_at": now, "last_seen_order_count": count, + }).Error; err != nil { + return err + } + } + return nil + }) +} + +func RecoverInterruptedRuns(ctx context.Context, db *gorm.DB) error { + now := time.Now().UTC() + return db.WithContext(ctx).Model(&models.SYBSyncRun{}).Where("status = ?", SyncRunRunning).Updates(map[string]any{ + "status": SyncRunInterrupted, "active_slot": nil, "finished_at": now, + "error_message": "服务重启，导入任务已中断；已导入的数据会保留，可重新发起导入", + }).Error +} + +type SyncRunListRequest struct { + Page, PageSize int + Status, DateFrom, DateTo string +} +type SyncRunListResponse struct { + Items []SyncRunDTO `json:"items"` + Total int64 `json:"total"` + Page int `json:"page"` + PageSize int `json:"pageSize"` +} +type ShopBreakdownItem struct { + ShopName string `json:"shopName"` + Accepted int `json:"accepted"` + Skipped int `json:"skipped"` +} +type SyncRunDTO struct { + models.SYBSyncRun + ShopBreakdown []ShopBreakdownItem `json:"shopBreakdown"` + ProgressPercent int `json:"progressPercent"` +} + +func (s *SyncRunService) List(ctx context.Context, request SyncRunListRequest) (SyncRunListResponse, error) { + q := s.db.WithContext(ctx).Model(&models.SYBSyncRun{}) + if request.Status != "" { + q = q.Where("status = ?", request.Status) + } + if request.DateFrom != "" { + q = q.Where("date_to >= ?", request.DateFrom) + } + if request.DateTo != "" { + q = q.Where("date_from <= ?", request.DateTo) + } + var total int64 + if err := q.Count(&total).Error; err != nil { + return SyncRunListResponse{}, internalError(err) + } + var rows []models.SYBSyncRun + if err := q.Order("id desc").Offset((request.Page - 1) * request.PageSize).Limit(request.PageSize).Find(&rows).Error; err != nil { + return SyncRunListResponse{}, internalError(err) + } + items := make([]SyncRunDTO, 0, len(rows)) + for _, row := range rows { + items = append(items, syncRunDTO(row)) + } + return SyncRunListResponse{Items: items, Total: total, Page: request.Page, PageSize: request.PageSize}, nil +} + +func (s *SyncRunService) Detail(ctx context.Context, id uint64) (SyncRunDTO, error) { + var row models.SYBSyncRun + if err := s.db.WithContext(ctx).First(&row, id).Error; err != nil { + if errors.Is(err, gorm.ErrRecordNotFound) { + return SyncRunDTO{}, &ServiceError{Code: CodeNotFound, Message: "同步记录不存在"} + } + return SyncRunDTO{}, internalError(err) + } + return syncRunDTO(row), nil +} + +func syncRunDTO(row models.SYBSyncRun) SyncRunDTO { + var values map[string]ShopBreakdown + _ = json.Unmarshal([]byte(row.ShopBreakdownJSON), &values) + names := make([]string, 0, len(values)) + for name := range values { + names = append(names, name) + } + sort.Strings(names) + items := make([]ShopBreakdownItem, 0, len(names)) + for _, name := range names { + v := values[name] + items = append(items, ShopBreakdownItem{ShopName: name, Accepted: v.Accepted, Skipped: v.Skipped}) + } + percent := 0 + if row.DaysTotal > 0 { + percent = row.DaysProcessed * 100 / row.DaysTotal + if percent > 100 { + percent = 100 + } + } + if row.Status == SyncRunSucceeded { + percent = 100 + } + return SyncRunDTO{SYBSyncRun: row, ShopBreakdown: items, ProgressPercent: percent} +} + +func truncateRunError(value string, limit int) string { + runes := []rune(strings.TrimSpace(value)) + if len(runes) <= limit { + return string(runes) + } + return string(runes[:limit]) +} diff --git a/server/app/goauto/sybimport/sync_run_handler.go b/server/app/goauto/sybimport/sync_run_handler.go new file mode 100644 index 0000000..23c2db5 --- /dev/null +++ b/server/app/goauto/sybimport/sync_run_handler.go @@ -0,0 +1,58 @@ +package sybimport + +import ( + "net/http" + "strconv" + "strings" + + "github.com/gin-gonic/gin" +) + +func (handler Handler) ListSyncRuns(c *gin.Context) { + page, err := queryInt(c.Query("page"), 1) + if err != nil { + writeError(c, invalidRequest("page 必须是正整数")) + return + } + pageSize, err := queryInt(c.Query("pageSize"), 20) + if err != nil || pageSize > 100 { + writeError(c, invalidRequest("pageSize 必须是 1 到 100 的整数")) + return + } + status := strings.TrimSpace(c.Query("status")) + if status != "" && status != SyncRunRunning && status != SyncRunSucceeded && status != SyncRunFailed && status != SyncRunInterrupted { + writeError(c, invalidRequest("status 无效")) + return + } + service, ok := handler.service(c) + if !ok { + return + } + response, err := NewSyncRunService(service.DB).List(c.Request.Context(), SyncRunListRequest{ + Page: page, PageSize: pageSize, Status: status, + DateFrom: strings.TrimSpace(c.Query("dateFrom")), DateTo: strings.TrimSpace(c.Query("dateTo")), + }) + if err != nil { + writeError(c, err) + return + } + c.JSON(http.StatusOK, gin.H{"code": 200, "data": response}) +} + +func (handler Handler) SyncRunDetail(c *gin.Context) { + id, err := strconv.ParseUint(c.Param("runId"), 10, 64) + if err != nil || id == 0 { + writeError(c, invalidRequest("runId 无效")) + return + } + service, ok := handler.service(c) + if !ok { + return + } + item, err := NewSyncRunService(service.DB).Detail(c.Request.Context(), id) + if err != nil { + writeError(c, err) + return + } + c.JSON(http.StatusOK, gin.H{"code": 200, "data": gin.H{"item": item}}) +} diff --git a/server/app/goauto/sybimport/sync_run_handler_test.go b/server/app/goauto/sybimport/sync_run_handler_test.go new file mode 100644 index 0000000..cb296a7 --- /dev/null +++ b/server/app/goauto/sybimport/sync_run_handler_test.go @@ -0,0 +1,39 @@ +package sybimport + +import ( + "context" + "net/http" + "net/http/httptest" + "strconv" + "strings" + "testing" + + "github.com/gin-gonic/gin" +) + +func TestSyncRunListAndDetailHandlers(t *testing.T) { + db := newSyncTestDB(t) + run, err := NewSyncRunService(db).Create(context.Background(), CreateSyncRunInput{ + DateFrom: "2026-08-01", DateTo: "2026-08-02", ShopFilterHash: "hash", OperatorName: "管理员", + }) + if err != nil { + t.Fatal(err) + } + gin.SetMode(gin.TestMode) + engine := gin.New() + handler := Handler{DB: db} + engine.GET("/sync-runs", handler.ListSyncRuns) + engine.GET("/sync-runs/:runId", handler.SyncRunDetail) + + list := httptest.NewRecorder() + engine.ServeHTTP(list, httptest.NewRequest(http.MethodGet, "/sync-runs?page=1&pageSize=20&status=running", nil)) + if list.Code != http.StatusOK || !strings.Contains(list.Body.String(), `"total":1`) { + t.Fatalf("列表响应不正确: %d %s", list.Code, list.Body.String()) + } + + detail := httptest.NewRecorder() + engine.ServeHTTP(detail, httptest.NewRequest(http.MethodGet, "/sync-runs/"+strconv.FormatUint(run.ID, 10), nil)) + if detail.Code != http.StatusOK || !strings.Contains(detail.Body.String(), `"operatorName":"管理员"`) { + t.Fatalf("详情响应不正确: %d %s", detail.Code, detail.Body.String()) + } +} diff --git a/server/app/goauto/sybimport/sync_run_test.go b/server/app/goauto/sybimport/sync_run_test.go new file mode 100644 index 0000000..c53090a --- /dev/null +++ b/server/app/goauto/sybimport/sync_run_test.go @@ -0,0 +1,59 @@ +package sybimport + +import ( + "context" + "strings" + "testing" +) + +func TestSyncRunSingleFlightAndTerminalRelease(t *testing.T) { + db := newSyncTestDB(t) + service := NewSyncRunService(db) + first, err := service.Create(context.Background(), CreateSyncRunInput{DateFrom: "2026-08-01", DateTo: "2026-08-03", ShopFilterHash: "hash", OperatorID: 7, OperatorName: "采购员"}) + if err != nil { + t.Fatalf("创建同步任务失败: %v", err) + } + if _, err := service.Create(context.Background(), CreateSyncRunInput{DateFrom: "2026-08-04", DateTo: "2026-08-04", ShopFilterHash: "hash"}); err == nil || !strings.Contains(err.Error(), "2026-08-01 至 2026-08-03") { + t.Fatalf("并发任务应返回正在执行的日期范围，实际: %v", err) + } + report := SyncReport{From: first.DateFrom, To: first.DateTo, ShopFilterHash: "hash", ShopBreakdown: map[string]ShopBreakdown{"测试店铺": {Accepted: 2}}} + if err := service.Finish(context.Background(), first.ID, SyncRunSucceeded, report, nil); err != nil { + t.Fatalf("结束同步任务失败: %v", err) + } + if _, err := service.Create(context.Background(), CreateSyncRunInput{DateFrom: "2026-08-04", DateTo: "2026-08-04", ShopFilterHash: "hash"}); err != nil { + t.Fatalf("终态应释放执行槽位: %v", err) + } +} + +func TestSyncRunProgressDetailAndRecovery(t *testing.T) { + db := newSyncTestDB(t) + service := NewSyncRunService(db) + run, err := service.Create(context.Background(), CreateSyncRunInput{DateFrom: "2026-08-01", DateTo: "2026-08-02", ShopFilterHash: "initial"}) + if err != nil { + t.Fatal(err) + } + progress := SyncProgress{DaysTotal: 2, DaysProcessed: 1, Report: SyncReport{ + OrderCount: 5, DetailCount: 3, AcceptedCount: 4, ShopSkipped: 1, Created: 3, + ShopFilterHash: "actual", ShopBreakdown: map[string]ShopBreakdown{"测试店铺": {Accepted: 4}, "新店": {Skipped: 1}}, + }} + if err := service.UpdateProgress(context.Background(), run.ID, progress); err != nil { + t.Fatal(err) + } + detail, err := service.Detail(context.Background(), run.ID) + if err != nil { + t.Fatal(err) + } + if detail.ProgressPercent != 50 || detail.OrderCount != 5 || len(detail.ShopBreakdown) != 2 { + t.Fatalf("进度详情不正确: %+v", detail) + } + if err := RecoverInterruptedRuns(context.Background(), db); err != nil { + t.Fatal(err) + } + detail, err = service.Detail(context.Background(), run.ID) + if err != nil { + t.Fatal(err) + } + if detail.Status != SyncRunInterrupted || detail.FinishedAt == nil || !strings.Contains(detail.ErrorMessage, "服务重启") { + t.Fatalf("启动恢复状态不正确: %+v", detail) + } +} diff --git a/server/app/goauto/sybimport/sync_test.go b/server/app/goauto/sybimport/sync_test.go index 0f9632b..45699b1 100644 --- a/server/app/goauto/sybimport/sync_test.go +++ b/server/app/goauto/sybimport/sync_test.go @@ -187,6 +187,21 @@ func TestSyncPagesBeyondTheFirstPage(t *testing.T) { } } +func TestSyncProgressCompletesDayWhenEveryShopIsSkipped(t *testing.T) { + db := newSyncTestDB(t) + f := &fakeSYB{perDay: map[string]int{"2026-08-01": 2}, shopNames: []string{"未启用店铺"}} + last := SyncProgress{} + _, err := SyncWithProgress(context.Background(), db, newSyncClient(t, f), + SyncConfig{PageSize: 10, MaxMatches: 1000}, "2026-08-01", "2026-08-01", + func(progress SyncProgress) error { last = progress; return nil }) + if err != nil { + t.Fatalf("同步失败: %v", err) + } + if last.DaysProcessed != 1 || last.DaysTotal != 1 || last.Report.ShopSkipped != 2 { + t.Fatalf("全部跳过的日期也必须完成进度并保存统计: %+v", last) + } +} + // 重跑同一范围必须幂等：按 (order_code, detail_id) 覆盖，不产生重复行。 func TestSyncIsIdempotentAcrossRuns(t *testing.T) { db := newSyncTestDB(t) @@ -351,10 +366,10 @@ func TestSyncSkipsOrdersFromShopsNotOnTheList(t *testing.T) { t.Fatalf("只应写入启用店铺的明细: %d", report.DetailCount) } // `[必须]` 跳过的店铺也要出现在分店铺统计里，否则新开的店永远没人注意到。 - if report.ShopBreakdown["别人的店"] != 3 { + if report.ShopBreakdown["别人的店"].Skipped != 3 { t.Fatalf("被跳过的店铺应出现在统计里: %v", report.ShopBreakdown) } - if report.ShopBreakdown["测试店铺"] != 3 { + if report.ShopBreakdown["测试店铺"].Accepted != 3 { t.Fatalf("接受的店铺统计不对: %v", report.ShopBreakdown) } } @@ -375,7 +390,7 @@ func TestSyncMatchesShopNamesThroughNormalization(t *testing.T) { t.Fatalf("归一化后应全部命中: accepted=%d skipped=%d", report.AcceptedCount, report.ShopSkipped) } // 统计里用名单中的展示名，而不是 SYB 返回的写法，避免同一家店出现两种拼写。 - if report.ShopBreakdown["ＡＢＣ店"] != 4 { + if report.ShopBreakdown["ＡＢＣ店"].Accepted != 4 { t.Fatalf("统计应使用名单里的展示名: %v", report.ShopBreakdown) } } diff --git a/server/cmd/api/server.go b/server/cmd/api/server.go index eb02e06..cc422ad 100644 --- a/server/cmd/api/server.go +++ b/server/cmd/api/server.go @@ -21,6 +21,7 @@ import ( "go-admin/app/admin/models" "go-admin/app/admin/router" goautodevice "go-admin/app/goauto/device" + goautosybimport "go-admin/app/goauto/sybimport" "go-admin/app/jobs" "go-admin/common/database" "go-admin/common/global" @@ -89,6 +90,11 @@ func run() error { for _, f := range AppRouters { f() } + for _, db := range sdk.Runtime.GetDb() { + if err := goautosybimport.RecoverInterruptedRuns(context.Background(), db); err != nil { + return fmt.Errorf("recover interrupted SYB imports: %w", err) + } + } offlineMonitorContext, stopOfflineMonitors := context.WithCancel(context.Background()) defer stopOfflineMonitors() for _, db := range sdk.Runtime.GetDb() { diff --git a/server/cmd/migrate/migration/version-local/1786701000000_syb_sync_run.go b/server/cmd/migrate/migration/version-local/1786701000000_syb_sync_run.go new file mode 100644 index 0000000..18e90da --- /dev/null +++ b/server/cmd/migrate/migration/version-local/1786701000000_syb_sync_run.go @@ -0,0 +1,27 @@ +package version_local + +import ( + "runtime" + + goautomigrations "go-admin/app/goauto/migrations" + "go-admin/cmd/migrate/migration" + common "go-admin/common/models" + + "gorm.io/gorm" +) + +// This additive migration introduces the persistent asynchronous SYB import +// run and its database-level single-flight guard (#50). +func init() { + _, fileName, _, _ := runtime.Caller(0) + migration.Migrate.SetVersion(migration.GetFilename(fileName), migrateSYBSyncRun) +} + +func migrateSYBSyncRun(db *gorm.DB, version string) error { + return db.Transaction(func(tx *gorm.DB) error { + if err := goautomigrations.Migrate(tx); err != nil { + return err + } + return tx.Create(&common.Migration{Version: version}).Error + }) +} diff --git a/web/src/api/goauto/syb-products.js b/web/src/api/goauto/syb-products.js index 396de69..053f39a 100644 --- a/web/src/api/goauto/syb-products.js +++ b/web/src/api/goauto/syb-products.js @@ -20,8 +20,14 @@ export function correctSybProduct(productId, data) { return request({ url: `/api/admin/v1/syb-products/${productId}/correction`, method: 'patch', data }) } -// 从 SYB 拉取指定日期范围的货运单并入库。耗时可能很长（一周可达数千张）， -// 所以单独放大超时，不用 request 的默认值。 export function importSybProducts(data) { - return request({ url: '/api/admin/v1/syb-products/import', method: 'post', data, timeout: 30 * 60 * 1000 }) + return request({ url: '/api/admin/v1/syb-products/import', method: 'post', data }) +} + +export function listSybSyncRuns(params) { + return request({ url: '/api/admin/v1/syb-products/sync-runs', method: 'get', params }) +} + +export function getSybSyncRun(runId) { + return request({ url: `/api/admin/v1/syb-products/sync-runs/${runId}`, method: 'get' }) } diff --git a/web/src/router/index.js b/web/src/router/index.js index 00127bf..29a236a 100644 --- a/web/src/router/index.js +++ b/web/src/router/index.js @@ -145,6 +145,19 @@ export const constantRoutes = [ } ] }, + { + path: '/syb-sync-runs', + component: Layout, + redirect: '/syb-sync-runs/index', + children: [ + { + path: 'index', + component: () => import('@/views/goauto/syb-sync-runs/index'), + name: 'GoAutoSybSyncRuns', + meta: { title: '同步记录', icon: 'time' } + } + ] + }, { path: '/collection-rules', component: Layout, diff --git a/web/src/views/goauto/syb-products/index.vue b/web/src/views/goauto/syb-products/index.vue index f745cdd..a6c3630 100644 --- a/web/src/views/goauto/syb-products/index.vue +++ b/web/src/views/goauto/syb-products/index.vue @@ -3,12 +3,22 @@