diff --git a/docs/02-architecture-and-code-map.md b/docs/02-architecture-and-code-map.md index 13ac5cb..1da0c41 100644 --- a/docs/02-architecture-and-code-map.md +++ b/docs/02-architecture-and-code-map.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件) wiki_page: Architecture-and-Code-Map wiki_url: https://git.ilapage.cn/OPC/goauto/wiki/Architecture-and-Code-Map.- -wiki_revision: b94fd07dfdebe418257a675a779f83b782896543 -synchronized_at: 2026-09-01T14:15:32Z +wiki_revision: 2864a22f9e0f54a0663a56ab27d4b84052be3aac +synchronized_at: 2026-09-01T15:31:13Z @@ -318,3 +318,11 @@ PddProductDetailCollector - `GET /api/admin/v1/pdd-products/{productId}/related-syb-products` 返回统一扁平分页，参数为 `page`、`pageSize`（默认 20、最大 100）、`scope=actionable|all`（默认 actionable）和可选 `shopeeProductId`。管理员和采购员可读，其他角色由 Casbin 拒绝。 - `purchase.Service.ProcessStages` 是 SYB 页和 PDD 关联订单共用的只读阶段入口；批量加载、不调用 AI。当前采购规则缺失或无效时仍返回基础关联、采集和已有任务事实，依赖规则才能创建的行明确标记不可采购。 - Web 的 PDD 商品详情把“创建备货采购”保留在顶部，把“关联订单继续采购”放在关联商品区块；订单行使用统一分页，并复用既有 `batch-preview` / `batch` 创建路径。 + +## 蝦皮规格自动匹配批处理（#195） + +- 服务端在 `server/app/goauto/shopeeproduct/auto_match_batch.go` 复用单商品“一键匹配颜色和尺码”原子服务，定时任务与 Admin 手动执行共用同一批处理入口。 +- `shopee_spec_auto_match_run` 保存触发来源、稳定 `requestId`、运行摘要和可空唯一 `active_slot`；活动槽与租约保证多实例、定时和手动同时触发时全局最多一个运行批次。 +- `shopee_spec_auto_match_work_item` 按蝦皮商品唯一保存输入指纹、尝试次数、下次尝试时间和逐商品租约。已完成或低置信度/无结果的相同输入不重复调用 AI；规格、关联或 AI 设置更新时间变化后才允许重新处理。 +- 迁移 `1788290000000_shopee_spec_auto_match.go` 幂等创建两张表并写入调用目标 `GoAutoShopeeSpecAutoMatch`。系统任务默认关闭，默认 Cron 为每小时第 15 分钟、每批最多 20 个商品。 +- Admin 蝦皮商品列表通过异步手动接口启动同一批次并轮询最近运行摘要；该入口仅管理员可用，不创建采购任务、订单，不调用 Android Agent，也不执行付款。 diff --git a/docs/03-business-rules-and-glossary.md b/docs/03-business-rules-and-glossary.md index 1f33627..72e971a 100644 --- a/docs/03-business-rules-and-glossary.md +++ b/docs/03-business-rules-and-glossary.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件) wiki_page: Business-Rules-and-Glossary wiki_url: https://git.ilapage.cn/OPC/goauto/wiki/Business-Rules-and-Glossary.- -wiki_revision: b94fd07dfdebe418257a675a779f83b782896543 -synchronized_at: 2026-09-01T14:15:42Z +wiki_revision: 0e1df62c48aeba32a7d5e4956783f0af68e8e658 +synchronized_at: 2026-09-01T15:31:24Z # 业务规则与术语 @@ -393,3 +393,11 @@ synchronized_at: 2026-09-01T14:15:42Z - 规格映射摘要使用 `specMappingStatus=complete|incomplete|not_required`、确认数和总数。`not_required` 只表示虾皮商品没有颜色或尺码维度；商品摘要不替代订单行采购资格。 - “关联订单继续采购”只处理已有 SYB 订单；“创建备货采购”不关联 SYB 订单。两者入口和文案必须明确区分。继续采购复用既有批量预检和批量创建，价格、映射、并发、订单和不可逆门禁不变。 - SYB 订单号、目标颜色/尺码、数量和当前采购任务状态只允许管理员与采购员读取；其他角色由服务端拒绝。查询路径不调用 AI Provider。 + +## 定时与手动蝦皮规格自动匹配（#195） + +- 系统只扫描存活且已关联 `active` PDD 商品、两边至少共享颜色或尺码角色、并且至少存在一个未确认或已失效映射的蝦皮商品。 +- 每个商品复用详情页“一键匹配颜色和尺码”的规则：保留当前仍有效的 `confirmed` 映射；唯一确定匹配和达到服务端阈值、理由非空、候选仍有效的 AI 结果直接保存为 `confirmed`；低置信度、无结果、Provider 异常或上下文漂移不猜测、不写入错误映射。 +- 定时与管理员手动执行共用全局活动槽和逐商品工作状态。单批默认最多 20 个商品；同一规格上下文与 AI 设置更新时间未变化时，已完成、低置信度或无结果商品不重复调用 AI。 +- Provider 临时失败最多尝试 3 次，间隔至少 60 分钟；输入变化后重新计算指纹并允许重新处理。运行记录只保存结构化计数和脱敏限长错误，不保存 API Key、Provider 原始响应、商品原始 JSON 或个人数据。 +- 系统定时任务迁移后默认关闭，须由管理员明确启用。该能力仅维护蝦皮与 PDD 颜色/尺码映射，不创建采购任务、PDD 订单，不触发 Agent，也不执行付款。 diff --git a/docs/08-agent-api-contract.md b/docs/08-agent-api-contract.md index 5137805..d060723 100644 --- a/docs/08-agent-api-contract.md +++ b/docs/08-agent-api-contract.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件) wiki_page: Android-Agent-API-Contract wiki_url: https://git.ilapage.cn/OPC/goauto/wiki/Android-Agent-API-Contract.- -wiki_revision: b94fd07dfdebe418257a675a779f83b782896543 -synchronized_at: 2026-09-01T14:16:34Z +wiki_revision: 87219820eec3964ee5379f1edba6a62e94bddd2e +synchronized_at: 2026-09-01T15:32:52Z # MVP 共享 API 契约 @@ -848,3 +848,17 @@ file= | `GET` | `/api/agent/v1/app/releases/{releaseId}/download` | Device Token；私有 APK 下载 | Agent 只比较整数 `versionCode`。设备有活动任务时禁止检查、下载和安装；下载到应用私有缓存并校验响应大小与 SHA-256，失败立即删除。安装使用 FileProvider 和 Android 系统安装确认页；未知来源权限必须由用户在系统设置授权，不静默安装。 + +## Admin 蝦皮规格自动匹配批次（#195） + +以下接口仅管理员可用，采购员与其他角色必须拒绝；两者不属于 Android Agent 接口，不改变任何 Agent 契约。 + +### `POST /api/admin/v1/shopee-spec-auto-match/runs` + +请求：`{ "requestId": "UUID" }`。服务端以 `requestId` 幂等，异步受理默认最多 20 个商品的手动批次，并返回 HTTP 202、`data.run` 运行摘要。若已有活动批次，不再创建第二个批次，返回当前运行并标记 `alreadyRunning=true`；相同请求重放标记 `replayed=true`。 + +### `GET /api/admin/v1/shopee-spec-auto-match/runs/latest` + +返回 `data.run`；从未执行时为 `null`。运行摘要包含 `id/requestId/trigger/status/batchLimit/scannedCount/eligibleCount/processedCount/confirmedCount/unmatchedCount/failedCount/startedAt/finishedAt` 和可选脱敏 `errorSummary`，不得返回 API Key、Provider 原始响应或商品原始 JSON。 + +`status` 当前为 `running`、`completed`、`completed_partial` 或 `failed`。页面只轮询最近摘要；查询本身不调用 AI。批次只更新蝦皮商品规格映射，不创建采购任务、订单或付款动作。 diff --git a/docs/13-deployment-and-operations.md b/docs/13-deployment-and-operations.md index 41a0efa..6c4e030 100644 --- a/docs/13-deployment-and-operations.md +++ b/docs/13-deployment-and-operations.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件) wiki_page: Deployment-and-Operations wiki_url: https://git.ilapage.cn/OPC/goauto/wiki/Deployment-and-Operations.- -wiki_revision: 80ae5c8cea87c624fff63ce3e302aa7bf204811e -synchronized_at: 2026-08-31T16:07:03Z +wiki_revision: 88422ddd1bd239c687b379217e3709f053b77704 +synchronized_at: 2026-09-01T15:31:48Z # 部署与运维 @@ -42,3 +42,11 @@ GOAUTO_ALLOW_INSECURE_AGENT_HTTP=true ## 回滚 服务异常时把 `/home/goauto/current` 切回上一已验证发布目录并重启 `goauto.service`，随后复核服务状态和 Agent 接口。不要删除当前或历史 APK/发布目录来代替回滚；数据库变化如需回退必须单独评估。 + +## 蝦皮规格自动匹配任务运维（#195） + +- 迁移 `1788290000000_shopee_spec_auto_match.go` 创建运行与工作状态表，并幂等写入 `sys_job.invoke_target=GoAutoShopeeSpecAutoMatch`。首次部署状态为关闭，迁移重跑不会覆盖管理员后来修改的 Cron、参数或启停状态。 +- 默认 Cron 为 `0 15 * * * *`（每小时第 15 分钟），默认 `batchLimit` 为 20，合法范围 1～100。上线后先保持关闭，在 Admin 蝦皮商品列表手动执行小批次并核对运行摘要和映射，再由管理员在定时任务页明确启用。 +- 定时和手动触发共享数据库唯一活动槽。看到“已有自动匹配正在运行”属于正常互斥结果；运行租约超过 30 分钟会在下次触发时安全释放并把旧运行标记失败。处理每个商品前会续租。 +- Provider 临时失败的商品至少 60 分钟后重试，最多 3 次；低置信度或无结果在输入指纹不变时不会重复调用。排错优先查看 `shopee_spec_auto_match_run` 的结构化计数与脱敏错误，以及 `shopee_spec_auto_match_work_item` 的状态、尝试次数、下次尝试时间和租约；不得输出或复制 API Key、Provider 原始响应、商品原始 JSON 或个人数据。 +- 紧急回退只需关闭系统任务并停止使用 Admin 手动入口；保留运行记录和既有已确认映射用于审计，不删除数据，也不回滚 #194。 diff --git a/server/app/goauto/access/modules.go b/server/app/goauto/access/modules.go index 1adf63a..cf7793d 100644 --- a/server/app/goauto/access/modules.go +++ b/server/app/goauto/access/modules.go @@ -136,6 +136,8 @@ func moduleKeyForAPI(path string) string { return ModulePDDProducts case strings.HasPrefix(path, "/api/admin/v1/shopee-products"): return ModuleShopeeProducts + case strings.HasPrefix(path, "/api/admin/v1/shopee-spec-auto-match"): + return ModuleShopeeProducts case strings.HasPrefix(path, "/api/admin/v1/syb-products/sync-runs"): return ModuleSYBSyncRuns case strings.HasPrefix(path, "/api/admin/v1/syb-products"): diff --git a/server/app/goauto/access/purchaser.go b/server/app/goauto/access/purchaser.go index 958e12a..264a291 100644 --- a/server/app/goauto/access/purchaser.go +++ b/server/app/goauto/access/purchaser.go @@ -48,6 +48,8 @@ var AdminAPIs = []APIPermission{ {"AI 建议颜色映射", "/api/admin/v1/shopee-products/:productId/specs/mapping/suggest-colors", "POST", true}, {"AI 建议尺码映射", "/api/admin/v1/shopee-products/:productId/specs/mapping/suggest-sizes", "POST", true}, {"一键匹配并确认颜色尺码", "/api/admin/v1/shopee-products/:productId/specs/mapping/auto-match", "POST", true}, + {"手动执行虾皮规格自动匹配", "/api/admin/v1/shopee-spec-auto-match/runs", "POST", false}, + {"查看最近虾皮规格自动匹配", "/api/admin/v1/shopee-spec-auto-match/runs/latest", "GET", false}, {"查看 SYB 商品", "/api/admin/v1/syb-products", "GET", true}, {"查看 SYB 商品详情", "/api/admin/v1/syb-products/:productId", "GET", true}, diff --git a/server/app/goauto/access/purchaser_test.go b/server/app/goauto/access/purchaser_test.go index 7cd208f..89ce9aa 100644 --- a/server/app/goauto/access/purchaser_test.go +++ b/server/app/goauto/access/purchaser_test.go @@ -15,10 +15,12 @@ func TestPurchaserPermissionMatrixHasNoDuplicates(t *testing.T) { func TestPurchaserExcludesAdministratorOperations(t *testing.T) { denied := map[string]bool{ - "POST /api/admin/v1/devices/:deviceId/disable": true, - "POST /api/admin/v1/syb-products/import": true, - "POST /api/admin/v1/collection-rules": true, - "PUT /api/admin/v1/ai-matching-settings": true, + "POST /api/admin/v1/devices/:deviceId/disable": true, + "POST /api/admin/v1/syb-products/import": true, + "POST /api/admin/v1/collection-rules": true, + "PUT /api/admin/v1/ai-matching-settings": true, + "POST /api/admin/v1/shopee-spec-auto-match/runs": true, + "GET /api/admin/v1/shopee-spec-auto-match/runs/latest": true, } for _, permission := range PurchaserAPIs() { if denied[permission.Method+" "+permission.Path] { diff --git a/server/app/goauto/migrations/migrate.go b/server/app/goauto/migrations/migrate.go index c0c0d67..85eaf54 100644 --- a/server/app/goauto/migrations/migrate.go +++ b/server/app/goauto/migrations/migrate.go @@ -34,6 +34,8 @@ func MigratedModels() []any { &models.PDDProduct{}, &models.AIMatchingSetting{}, &models.ShopeeProduct{}, + &models.ShopeeSpecAutoMatchRun{}, + &models.ShopeeSpecAutoMatchWorkItem{}, &models.SYBProduct{}, &models.SYBSession{}, &models.SYBShop{}, diff --git a/server/app/goauto/models/shopee_spec_auto_match.go b/server/app/goauto/models/shopee_spec_auto_match.go new file mode 100644 index 0000000..215207e --- /dev/null +++ b/server/app/goauto/models/shopee_spec_auto_match.go @@ -0,0 +1,56 @@ +package models + +import "time" + +// ShopeeSpecAutoMatchRun is one scheduled or administrator-triggered batch. +// ActiveSlot is 1 only while running; its nullable unique index is the +// database-level cross-process mutex shared by both trigger paths. +type ShopeeSpecAutoMatchRun struct { + ID uint64 `json:"id" gorm:"primaryKey;autoIncrement"` + RequestID string `json:"requestId" gorm:"size:36;not null;uniqueIndex:ux_shopee_spec_auto_match_run_request"` + Trigger string `json:"trigger" gorm:"size:16;not null;index"` + Status string `json:"status" gorm:"size:24;not null;index"` + ActiveSlot *uint8 `json:"-" gorm:"uniqueIndex:ux_shopee_spec_auto_match_run_active"` + LeaseOwner string `json:"-" gorm:"size:64;not null;default:''"` + LeaseExpiresAt *time.Time `json:"-" gorm:"index"` + RequestedBy *uint64 `json:"requestedBy,omitempty"` + BatchLimit int `json:"batchLimit" gorm:"not null;default:20"` + ScannedCount int `json:"scannedCount" gorm:"not null;default:0"` + EligibleCount int `json:"eligibleCount" gorm:"not null;default:0"` + ProcessedCount int `json:"processedCount" gorm:"not null;default:0"` + ConfirmedCount int `json:"confirmedCount" gorm:"not null;default:0"` + UnmatchedCount int `json:"unmatchedCount" gorm:"not null;default:0"` + FailedCount int `json:"failedCount" gorm:"not null;default:0"` + ErrorSummary string `json:"errorSummary,omitempty" gorm:"size:500;not null;default:''"` + StartedAt time.Time `json:"startedAt" gorm:"not null"` + FinishedAt *time.Time `json:"finishedAt,omitempty"` + CreatedAt time.Time `json:"createdAt"` + UpdatedAt time.Time `json:"updatedAt"` +} + +func (ShopeeSpecAutoMatchRun) TableName() string { return "shopee_spec_auto_match_run" } + +// ShopeeSpecAutoMatchWorkItem remembers the last input fingerprint and retry +// state for each product, preventing unchanged low-confidence inputs from +// repeatedly spending AI calls. +type ShopeeSpecAutoMatchWorkItem struct { + ID uint64 `json:"id" gorm:"primaryKey;autoIncrement"` + ShopeeProductID uint64 `json:"shopeeProductId" gorm:"not null;uniqueIndex:ux_shopee_spec_auto_match_work_product"` + RunID *uint64 `json:"runId,omitempty" gorm:"index"` + InputFingerprint string `json:"inputFingerprint" gorm:"size:128;not null;default:'';index"` + Status string `json:"status" gorm:"size:24;not null;index"` + AttemptCount int `json:"attemptCount" gorm:"not null;default:0"` + NextAttemptAt *time.Time `json:"nextAttemptAt,omitempty" gorm:"index"` + LeaseOwner string `json:"-" gorm:"size:64;not null;default:''"` + LeaseExpiresAt *time.Time `json:"-" gorm:"index"` + ConfirmedCount int `json:"confirmedCount" gorm:"not null;default:0"` + UnmatchedCount int `json:"unmatchedCount" gorm:"not null;default:0"` + LastErrorCode string `json:"lastErrorCode,omitempty" gorm:"size:64;not null;default:''"` + LastError string `json:"lastError,omitempty" gorm:"size:500;not null;default:''"` + CreatedAt time.Time `json:"createdAt"` + UpdatedAt time.Time `json:"updatedAt"` +} + +func (ShopeeSpecAutoMatchWorkItem) TableName() string { + return "shopee_spec_auto_match_work_item" +} diff --git a/server/app/goauto/shopeeproduct/auto_match_batch.go b/server/app/goauto/shopeeproduct/auto_match_batch.go new file mode 100644 index 0000000..aaac6ba --- /dev/null +++ b/server/app/goauto/shopeeproduct/auto_match_batch.go @@ -0,0 +1,360 @@ +package shopeeproduct + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "strings" + "time" + + "go-admin/app/goauto/models" + + "github.com/google/uuid" + "gorm.io/gorm" +) + +const ( + SpecAutoMatchInvokeTarget = "GoAutoShopeeSpecAutoMatch" + defaultAutoMatchBatchLimit = 20 + autoMatchLeaseDuration = 30 * time.Minute + autoMatchRetryDelay = time.Hour + maxAutoMatchAttempts = 3 +) + +type AutoMatchRunView struct { + models.ShopeeSpecAutoMatchRun + AlreadyRunning bool `json:"alreadyRunning,omitempty"` + Replayed bool `json:"replayed,omitempty"` +} + +// StartAutoMatchRun acquires the single database-backed activity slot. A +// repeated requestId is idempotent; a concurrent trigger receives the current +// run instead of starting a second batch. +func (service *Service) StartAutoMatchRun(ctx context.Context, trigger, requestID string, requestedBy *uint64, batchLimit int) (AutoMatchRunView, bool, error) { + if _, err := uuid.Parse(strings.TrimSpace(requestID)); err != nil { + return AutoMatchRunView{}, false, invalidRequest("requestId 必须是 UUID") + } + if trigger != "manual" && trigger != "scheduled" { + return AutoMatchRunView{}, false, invalidRequest("trigger 无效") + } + if batchLimit <= 0 { + batchLimit = defaultAutoMatchBatchLimit + } + if batchLimit > 100 { + return AutoMatchRunView{}, false, invalidRequest("batchLimit 不能超过 100") + } + now := time.Now().UTC() + lease := now.Add(autoMatchLeaseDuration) + owner := uuid.NewString() + one := uint8(1) + var result models.ShopeeSpecAutoMatchRun + created := false + err := service.DB.WithContext(ctx).Transaction(func(tx *gorm.DB) error { + if err := tx.Model(&models.ShopeeSpecAutoMatchRun{}). + Where("status = ? AND active_slot = ? AND lease_expires_at < ?", "running", 1, now). + Updates(map[string]any{"status": "failed", "active_slot": nil, "lease_owner": "", "lease_expires_at": nil, "error_summary": "上次运行租约过期，已安全释放", "finished_at": now}).Error; err != nil { + return err + } + if err := tx.Where("request_id = ?", requestID).First(&result).Error; err == nil { + return nil + } else if !errors.Is(err, gorm.ErrRecordNotFound) { + return err + } + if err := tx.Where("status = ? AND active_slot = ?", "running", 1).First(&result).Error; err == nil { + result.ActiveSlot = &one + return nil + } else if !errors.Is(err, gorm.ErrRecordNotFound) { + return err + } + result = models.ShopeeSpecAutoMatchRun{RequestID: requestID, Trigger: trigger, Status: "running", ActiveSlot: &one, LeaseOwner: owner, LeaseExpiresAt: &lease, RequestedBy: requestedBy, BatchLimit: batchLimit, StartedAt: now} + if err := tx.Create(&result).Error; err != nil { + return err + } + created = true + return nil + }) + if err != nil { + // A unique-slot race means another instance won after our read. Return + // its run as the stable, non-error result. + if findErr := service.DB.WithContext(ctx).Where("status = ? AND active_slot = ?", "running", 1).First(&result).Error; findErr == nil { + return AutoMatchRunView{ShopeeSpecAutoMatchRun: result, AlreadyRunning: true}, false, nil + } + return AutoMatchRunView{}, false, internalError(err) + } + view := AutoMatchRunView{ShopeeSpecAutoMatchRun: result} + if !created { + view.AlreadyRunning = result.RequestID != requestID + view.Replayed = result.RequestID == requestID + } + return view, created, nil +} + +func (service *Service) LatestAutoMatchRun(ctx context.Context) (*AutoMatchRunView, error) { + var run models.ShopeeSpecAutoMatchRun + err := service.DB.WithContext(ctx).Order("id DESC").First(&run).Error + if errors.Is(err, gorm.ErrRecordNotFound) { + return nil, nil + } + if err != nil { + return nil, internalError(err) + } + return &AutoMatchRunView{ShopeeSpecAutoMatchRun: run}, nil +} + +// ProcessAutoMatchRun performs a bounded batch. It is safe to call from an +// HTTP-launched goroutine or the scheduler because only the run owning the +// active slot may update and finish itself. +func (service *Service) ProcessAutoMatchRun(ctx context.Context, runID uint64) error { + var run models.ShopeeSpecAutoMatchRun + if err := service.DB.WithContext(ctx).First(&run, runID).Error; err != nil { + return err + } + if run.Status != "running" || run.ActiveSlot == nil || *run.ActiveSlot != 1 { + return nil + } + limit := run.BatchLimit + if limit <= 0 || limit > 100 { + limit = defaultAutoMatchBatchLimit + } + var candidates []models.ShopeeProduct + queryLimit := limit * 25 + if queryLimit < 100 { + queryLimit = 100 + } + if queryLimit > 1000 { + queryLimit = 1000 + } + if err := service.DB.WithContext(ctx). + Joins("JOIN pdd_product ON pdd_product.id = shopee_product.pdd_product_id AND pdd_product.status = ?", "active"). + Where("shopee_product.pdd_product_id IS NOT NULL"). + Order("shopee_product.updated_at ASC, shopee_product.id ASC").Limit(queryLimit).Find(&candidates).Error; err != nil { + service.finishAutoMatchRun(run, "failed", 0, 0, 0, 0, 0, 1, "扫描符合条件的商品失败") + return err + } + + eligible, processed, confirmed, unmatched, failed := 0, 0, 0, 0, 0 + firstError := "" + for _, product := range candidates { + if processed >= limit { + break + } + fingerprint, ok, err := service.autoMatchEligibility(ctx, product) + if err != nil { + failed++ + if firstError == "" { + firstError = safeBatchError(err) + } + continue + } + if !ok { + continue + } + eligible++ + work, claimed, err := service.claimAutoMatchWork(ctx, run, product.ID, fingerprint) + if err != nil { + failed++ + if firstError == "" { + firstError = safeBatchError(err) + } + continue + } + if !claimed { + continue + } + processed++ + service.renewAutoMatchRun(run) + response, matchErr := service.AutoMatchMappings(ctx, product.ID, AutoMatchRequest{RequestID: uuid.NewString(), SpecContextVersion: fingerprint[:64]}) + // fingerprint begins with the 64-character context version. + postFingerprint := fingerprint + if next, _, nextErr := service.autoMatchEligibility(ctx, product); nextErr == nil && next != "" { + postFingerprint = next + } + if matchErr != nil { + failed++ + if firstError == "" { + firstError = safeBatchError(matchErr) + } + service.completeAutoMatchWork(work, postFingerprint, 0, 0, matchErr) + continue + } + confirmed += response.ConfirmedCount + unmatched += response.UnmatchedCount + service.completeAutoMatchWork(work, postFingerprint, response.ConfirmedCount, response.UnmatchedCount, nil) + } + status := "completed" + if failed > 0 { + status = "completed_partial" + } + return service.finishAutoMatchRun(run, status, len(candidates), eligible, processed, confirmed, unmatched, failed, firstError) +} + +func (service *Service) autoMatchEligibility(ctx context.Context, product models.ShopeeProduct) (string, bool, error) { + if product.PDDProductID == nil { + return "", false, nil + } + var pdd models.PDDProduct + if err := service.DB.WithContext(ctx).First(&pdd, *product.PDDProductID).Error; err != nil { + return "", false, err + } + if pdd.Status != "active" { + return "", false, nil + } + shopeeSpecs, err := Unmarshal(product.SpecsJSON) + if err != nil { + return "", false, err + } + shared, needsMatch := false, false + for _, role := range []string{RoleColor, RoleSize} { + pddValues, err := selectablePDDValues(pdd.SpecsJSON, role) + if err != nil { + return "", false, err + } + if len(pddValues) == 0 { + continue + } + for _, dimension := range shopeeSpecs { + if dimension.Role != role || len(dimension.Values) == 0 { + continue + } + shared = true + for _, value := range dimension.Values { + if value.Mapping == nil || value.Mapping.Status != MappingStatusConfirmed || !pddValues[value.Mapping.PDDValue] { + needsMatch = true + } + } + } + } + if !shared || !needsMatch { + return "", false, nil + } + contextVersion := computeSpecContextVersion(product.PDDProductID, product.SpecsJSON, pdd.SpecsJSON) + var setting struct{ UpdatedAt time.Time } + _ = service.DB.WithContext(ctx).Table((models.AIMatchingSetting{}).TableName()).Select("updated_at").Where("id = ?", 1).Scan(&setting).Error + h := sha256.Sum256([]byte(contextVersion + "\x00" + setting.UpdatedAt.UTC().Format(time.RFC3339Nano))) + // Keeping the context version as a prefix lets ProcessAutoMatchRun pass the + // exact version to #194 without re-reading a potentially drifting input. + return contextVersion + hex.EncodeToString(h[:]), true, nil +} + +func (service *Service) claimAutoMatchWork(ctx context.Context, run models.ShopeeSpecAutoMatchRun, productID uint64, fingerprint string) (models.ShopeeSpecAutoMatchWorkItem, bool, error) { + now := time.Now().UTC() + lease := now.Add(autoMatchLeaseDuration) + var work models.ShopeeSpecAutoMatchWorkItem + err := service.DB.WithContext(ctx).Transaction(func(tx *gorm.DB) error { + err := tx.Where("shopee_product_id = ?", productID).First(&work).Error + if errors.Is(err, gorm.ErrRecordNotFound) { + work = models.ShopeeSpecAutoMatchWorkItem{ShopeeProductID: productID, RunID: &run.ID, InputFingerprint: fingerprint, Status: "running", AttemptCount: 1, LeaseOwner: run.LeaseOwner, LeaseExpiresAt: &lease} + return tx.Create(&work).Error + } + if err != nil { + return err + } + if work.InputFingerprint == fingerprint { + if work.Status == "completed" || work.Status == "unmatched" || work.AttemptCount >= maxAutoMatchAttempts || (work.NextAttemptAt != nil && work.NextAttemptAt.After(now)) || (work.Status == "running" && work.LeaseExpiresAt != nil && work.LeaseExpiresAt.After(now)) { + return errWorkNotClaimed + } + } else { + work.AttemptCount = 0 + } + updates := map[string]any{"run_id": run.ID, "input_fingerprint": fingerprint, "status": "running", "attempt_count": work.AttemptCount + 1, "next_attempt_at": nil, "lease_owner": run.LeaseOwner, "lease_expires_at": lease, "last_error_code": "", "last_error": ""} + if err := tx.Model(&models.ShopeeSpecAutoMatchWorkItem{}).Where("id = ?", work.ID).Updates(updates).Error; err != nil { + return err + } + return tx.First(&work, work.ID).Error + }) + if errors.Is(err, errWorkNotClaimed) { + return work, false, nil + } + return work, err == nil, err +} + +var errWorkNotClaimed = errors.New("auto match work not claimed") + +func (service *Service) completeAutoMatchWork(work models.ShopeeSpecAutoMatchWorkItem, fingerprint string, confirmed, unmatched int, matchErr error) { + now := time.Now().UTC() + updates := map[string]any{"input_fingerprint": fingerprint, "lease_owner": "", "lease_expires_at": nil, "confirmed_count": confirmed, "unmatched_count": unmatched} + if matchErr == nil { + if unmatched > 0 { + updates["status"] = "unmatched" + } else { + updates["status"] = "completed" + } + updates["next_attempt_at"], updates["last_error_code"], updates["last_error"] = nil, "", "" + } else { + code := batchErrorCode(matchErr) + updates["status"], updates["last_error_code"], updates["last_error"] = "failed", code, safeBatchError(matchErr) + if code == CodeAIUnavailable && work.AttemptCount < maxAutoMatchAttempts { + next := now.Add(autoMatchRetryDelay) + updates["next_attempt_at"] = next + } else { + updates["next_attempt_at"] = nil + } + } + _ = service.DB.Model(&models.ShopeeSpecAutoMatchWorkItem{}).Where("id = ?", work.ID).Updates(updates).Error +} + +func (service *Service) renewAutoMatchRun(run models.ShopeeSpecAutoMatchRun) { + lease := time.Now().UTC().Add(autoMatchLeaseDuration) + _ = service.DB.Model(&models.ShopeeSpecAutoMatchRun{}).Where("id = ? AND status = ? AND lease_owner = ?", run.ID, "running", run.LeaseOwner).Update("lease_expires_at", lease).Error +} + +func (service *Service) finishAutoMatchRun(run models.ShopeeSpecAutoMatchRun, status string, scanned, eligible, processed, confirmed, unmatched, failed int, summary string) error { + now := time.Now().UTC() + updates := map[string]any{"status": status, "active_slot": nil, "lease_owner": "", "lease_expires_at": nil, "scanned_count": scanned, "eligible_count": eligible, "processed_count": processed, "confirmed_count": confirmed, "unmatched_count": unmatched, "failed_count": failed, "error_summary": truncateBatchText(summary), "finished_at": now} + return service.DB.Model(&models.ShopeeSpecAutoMatchRun{}).Where("id = ? AND status = ? AND lease_owner = ?", run.ID, "running", run.LeaseOwner).Updates(updates).Error +} + +func batchErrorCode(err error) string { + var serviceErr *ServiceError + if errors.As(err, &serviceErr) { + return serviceErr.Code + } + return CodeInternal +} + +func safeBatchError(err error) string { + var serviceErr *ServiceError + if errors.As(err, &serviceErr) { + return truncateBatchText(serviceErr.Message) + } + return "服务端处理失败" +} + +func truncateBatchText(value string) string { + runes := []rune(strings.TrimSpace(value)) + if len(runes) > 500 { + runes = runes[:500] + } + return string(runes) +} + +type scheduledAutoMatchArgs struct { + BatchLimit int `json:"batchLimit"` +} + +type ScheduledAutoMatchJob struct{} + +func (ScheduledAutoMatchJob) Exec(_ interface{}) error { + return errors.New("规格自动匹配定时任务缺少数据库连接") +} + +func (ScheduledAutoMatchJob) ExecWithDB(db *gorm.DB, arg interface{}) error { + args := scheduledAutoMatchArgs{BatchLimit: defaultAutoMatchBatchLimit} + if raw, ok := arg.(string); ok && strings.TrimSpace(raw) != "" { + if err := json.Unmarshal([]byte(raw), &args); err != nil { + return fmt.Errorf("规格自动匹配参数不是合法 JSON: %w", err) + } + } + if args.BatchLimit < 1 || args.BatchLimit > 100 { + return errors.New("规格自动匹配 batchLimit 必须在 1 到 100 之间") + } + service := NewService(db) + run, created, err := service.StartAutoMatchRun(context.Background(), "scheduled", uuid.NewString(), nil, args.BatchLimit) + if err != nil || !created { + return err + } + return service.ProcessAutoMatchRun(context.Background(), run.ID) +} diff --git a/server/app/goauto/shopeeproduct/auto_match_batch_test.go b/server/app/goauto/shopeeproduct/auto_match_batch_test.go new file mode 100644 index 0000000..cdf9cff --- /dev/null +++ b/server/app/goauto/shopeeproduct/auto_match_batch_test.go @@ -0,0 +1,86 @@ +package shopeeproduct + +import ( + "context" + "testing" + + "go-admin/app/goauto/models" + + "github.com/google/uuid" +) + +func TestAutoMatchRunIsIdempotentAndGloballySerialized(t *testing.T) { + db := openTestDB(t) + service := NewService(db) + requestID := uuid.NewString() + first, created, err := service.StartAutoMatchRun(context.Background(), "manual", requestID, nil, 20) + if err != nil || !created { + t.Fatalf("first=%+v created=%v err=%v", first, created, err) + } + replay, created, err := service.StartAutoMatchRun(context.Background(), "manual", requestID, nil, 20) + if err != nil || created || !replay.Replayed || replay.ID != first.ID { + t.Fatalf("replay=%+v created=%v err=%v", replay, created, err) + } + concurrent, created, err := service.StartAutoMatchRun(context.Background(), "scheduled", uuid.NewString(), nil, 20) + if err != nil || created || !concurrent.AlreadyRunning || concurrent.ID != first.ID { + t.Fatalf("concurrent=%+v created=%v err=%v", concurrent, created, err) + } +} + +func TestProcessAutoMatchRunConfirmsExactSizeAndFinishes(t *testing.T) { + db := openTestDB(t) + pdd := seedPDDProduct(t, db, "active") + service := NewService(db) + createdProduct, err := service.Create(context.Background(), CreateRequest{ + RequestID: uuid.NewString(), ShopeeItemID: "SP-BATCH-EXACT", PDDProductID: &pdd.ID, + Specs: []SpecDimension{{Name: "尺码", Role: RoleSize, Values: []SpecValue{{Name: " xl ", Source: ValueSourceImport}}}}, + }) + if err != nil { + t.Fatal(err) + } + run, started, err := service.StartAutoMatchRun(context.Background(), "manual", uuid.NewString(), nil, 20) + if err != nil || !started { + t.Fatalf("run=%+v started=%v err=%v", run, started, err) + } + if err := service.ProcessAutoMatchRun(context.Background(), run.ID); err != nil { + t.Fatal(err) + } + latest, err := service.LatestAutoMatchRun(context.Background()) + if err != nil { + t.Fatal(err) + } + if latest == nil || latest.Status != "completed" || latest.ProcessedCount != 1 || latest.ConfirmedCount != 1 || latest.ActiveSlot != nil { + t.Fatalf("latest=%+v", latest) + } + detail, err := service.Detail(context.Background(), createdProduct.Product.ID) + if err != nil { + t.Fatal(err) + } + mapping := detail.Product.Specs[0].Values[0].Mapping + if mapping == nil || mapping.Status != MappingStatusConfirmed || mapping.PDDValue != "XL" { + t.Fatalf("mapping=%+v", mapping) + } +} + +func TestUnchangedUnmatchedWorkIsNotClaimedAgain(t *testing.T) { + db := openTestDB(t) + service := NewService(db) + one := uint8(1) + run := models.ShopeeSpecAutoMatchRun{RequestID: uuid.NewString(), Trigger: "manual", Status: "running", ActiveSlot: &one, LeaseOwner: uuid.NewString(), BatchLimit: 20} + if err := db.Create(&run).Error; err != nil { + t.Fatal(err) + } + work, claimed, err := service.claimAutoMatchWork(context.Background(), run, 99, "fingerprint") + if err != nil || !claimed { + t.Fatalf("work=%+v claimed=%v err=%v", work, claimed, err) + } + service.completeAutoMatchWork(work, "fingerprint", 0, 1, nil) + _, claimed, err = service.claimAutoMatchWork(context.Background(), run, 99, "fingerprint") + if err != nil || claimed { + t.Fatalf("unchanged unmatched claimed=%v err=%v", claimed, err) + } + _, claimed, err = service.claimAutoMatchWork(context.Background(), run, 99, "changed") + if err != nil || !claimed { + t.Fatalf("changed input claimed=%v err=%v", claimed, err) + } +} diff --git a/server/app/goauto/shopeeproduct/handler.go b/server/app/goauto/shopeeproduct/handler.go index 82e1e19..5d751a5 100644 --- a/server/app/goauto/shopeeproduct/handler.go +++ b/server/app/goauto/shopeeproduct/handler.go @@ -1,6 +1,7 @@ package shopeeproduct import ( + "context" "encoding/json" "errors" "io" @@ -300,6 +301,45 @@ func (handler Handler) AutoMatchMappings(c *gin.Context) { c.JSON(http.StatusOK, gin.H{"code": 200, "data": response}) } +func (handler Handler) StartAutoMatchRun(c *gin.Context) { + var request struct { + RequestID string `json:"requestId"` + } + if err := decodeJSON(c, &request); err != nil { + writeError(c, invalidRequest("请求 JSON 无效")) + return + } + service, ok := handler.service(c) + if !ok { + return + } + operator := currentUserID(c) + run, created, err := service.StartAutoMatchRun(c.Request.Context(), "manual", request.RequestID, &operator, defaultAutoMatchBatchLimit) + if err != nil { + writeError(c, err) + return + } + if created { + go func(runID uint64, db *gorm.DB) { + _ = NewService(db).ProcessAutoMatchRun(context.Background(), runID) + }(run.ID, service.DB) + } + c.JSON(http.StatusAccepted, gin.H{"code": 200, "data": gin.H{"run": run}}) +} + +func (handler Handler) LatestAutoMatchRun(c *gin.Context) { + service, ok := handler.service(c) + if !ok { + return + } + run, err := service.LatestAutoMatchRun(c.Request.Context()) + if err != nil { + writeError(c, err) + return + } + c.JSON(http.StatusOK, gin.H{"code": 200, "data": gin.H{"run": run}}) +} + func (handler Handler) BatchDelete(c *gin.Context) { var request BatchDeleteRequest if err := decodeJSON(c, &request); err != nil { diff --git a/server/app/goauto/shopeeproduct/router.go b/server/app/goauto/shopeeproduct/router.go index ec5982b..5ec039a 100644 --- a/server/app/goauto/shopeeproduct/router.go +++ b/server/app/goauto/shopeeproduct/router.go @@ -10,6 +10,9 @@ import ( func InitRouter(engine *gin.Engine, auth *jwt.GinJWTMiddleware) { handler := Handler{} admin := engine.Group("/api/admin/v1/shopee-products").Use(auth.MiddlewareFunc()).Use(middleware.AuthCheckRole()) + adminOnlyRuns := engine.Group("/api/admin/v1/shopee-spec-auto-match/runs").Use(auth.MiddlewareFunc()).Use(middleware.AuthCheckRole()).Use(middleware.RequireRoleKey("admin")) + adminOnlyRuns.POST("", handler.StartAutoMatchRun) + adminOnlyRuns.GET("/latest", handler.LatestAutoMatchRun) admin.GET("", handler.List) admin.POST("", handler.Create) admin.POST("/batch-delete", handler.BatchDelete) diff --git a/server/app/jobs/examples.go b/server/app/jobs/examples.go index 5937819..dfd2696 100644 --- a/server/app/jobs/examples.go +++ b/server/app/jobs/examples.go @@ -4,6 +4,7 @@ import ( "fmt" "time" + "go-admin/app/goauto/shopeeproduct" "go-admin/app/goauto/sybimport" ) @@ -12,8 +13,9 @@ import ( // 字典 key 可以配置到 自动任务 调用目标 中； func InitJob() { jobList = map[string]JobExec{ - "ExamplesOne": ExamplesOne{}, - sybimport.HourlySyncInvokeTarget: sybimport.HourlySyncJob{}, + "ExamplesOne": ExamplesOne{}, + sybimport.HourlySyncInvokeTarget: sybimport.HourlySyncJob{}, + shopeeproduct.SpecAutoMatchInvokeTarget: shopeeproduct.ScheduledAutoMatchJob{}, // ... } } diff --git a/server/cmd/migrate/migration/version-local/1788290000000_shopee_spec_auto_match.go b/server/cmd/migrate/migration/version-local/1788290000000_shopee_spec_auto_match.go new file mode 100644 index 0000000..4c50b1a --- /dev/null +++ b/server/cmd/migrate/migration/version-local/1788290000000_shopee_spec_auto_match.go @@ -0,0 +1,47 @@ +package version_local + +import ( + "errors" + "runtime" + + goautomigrations "go-admin/app/goauto/migrations" + "go-admin/app/goauto/shopeeproduct" + jobsmodels "go-admin/app/jobs/models" + "go-admin/cmd/migrate/migration" + common "go-admin/common/models" + + "gorm.io/gorm" +) + +func init() { + _, fileName, _, _ := runtime.Caller(0) + migration.Migrate.SetVersion(migration.GetFilename(fileName), migrateShopeeSpecAutoMatch) +} + +func migrateShopeeSpecAutoMatch(db *gorm.DB, version string) error { + return db.Transaction(func(tx *gorm.DB) error { + if err := goautomigrations.Migrate(tx); err != nil { + return err + } + if err := ensureShopeeSpecAutoMatchJob(tx); err != nil { + return err + } + return tx.Create(&common.Migration{Version: version}).Error + }) +} + +func ensureShopeeSpecAutoMatchJob(db *gorm.DB) error { + var existing jobsmodels.SysJob + err := db.Where("invoke_target = ?", shopeeproduct.SpecAutoMatchInvokeTarget).First(&existing).Error + if err == nil { + return nil + } + if !errors.Is(err, gorm.ErrRecordNotFound) { + return err + } + return db.Create(&jobsmodels.SysJob{ + JobName: "蝦皮规格自动匹配", JobGroup: "GoAuto", JobType: 2, + CronExpression: "0 15 * * * *", InvokeTarget: shopeeproduct.SpecAutoMatchInvokeTarget, + Args: `{"batchLimit":20}`, MisfirePolicy: 1, Concurrent: 1, Status: 1, + }).Error +} diff --git a/server/cmd/migrate/migration/version-local/1788290000000_shopee_spec_auto_match_test.go b/server/cmd/migrate/migration/version-local/1788290000000_shopee_spec_auto_match_test.go new file mode 100644 index 0000000..7a2b28e --- /dev/null +++ b/server/cmd/migrate/migration/version-local/1788290000000_shopee_spec_auto_match_test.go @@ -0,0 +1,50 @@ +package version_local + +import ( + "testing" + + "go-admin/app/goauto/shopeeproduct" + jobsmodels "go-admin/app/jobs/models" + + "gorm.io/driver/sqlite" + "gorm.io/gorm" +) + +func TestEnsureShopeeSpecAutoMatchJobIsDisabledIdempotentAndPreservesChanges(t *testing.T) { + db, err := gorm.Open(sqlite.Open("file:shopee-spec-auto-match-job?mode=memory&cache=shared"), &gorm.Config{}) + if err != nil { + t.Fatal(err) + } + if err := db.AutoMigrate(&jobsmodels.SysJob{}); err != nil { + t.Fatal(err) + } + if err := ensureShopeeSpecAutoMatchJob(db); err != nil { + t.Fatal(err) + } + var job jobsmodels.SysJob + if err := db.Where("invoke_target = ?", shopeeproduct.SpecAutoMatchInvokeTarget).First(&job).Error; err != nil { + t.Fatal(err) + } + if job.Status != 1 || job.CronExpression != "0 15 * * * *" || job.Args != `{"batchLimit":20}` { + t.Fatalf("unexpected seed: %+v", job) + } + if err := db.Model(&job).Updates(map[string]any{"status": 2, "cron_expression": "0 30 * * * *", "args": `{"batchLimit":5}`}).Error; err != nil { + t.Fatal(err) + } + if err := ensureShopeeSpecAutoMatchJob(db); err != nil { + t.Fatal(err) + } + var count int64 + if err := db.Model(&jobsmodels.SysJob{}).Where("invoke_target = ?", shopeeproduct.SpecAutoMatchInvokeTarget).Count(&count).Error; err != nil { + t.Fatal(err) + } + if count != 1 { + t.Fatalf("count=%d", count) + } + if err := db.Where("invoke_target = ?", shopeeproduct.SpecAutoMatchInvokeTarget).First(&job).Error; err != nil { + t.Fatal(err) + } + if job.Status != 2 || job.CronExpression != "0 30 * * * *" || job.Args != `{"batchLimit":5}` { + t.Fatalf("admin changes overwritten: %+v", job) + } +} diff --git a/web/src/api/goauto/shopee-products.js b/web/src/api/goauto/shopee-products.js index 77b0983..c6956e2 100644 --- a/web/src/api/goauto/shopee-products.js +++ b/web/src/api/goauto/shopee-products.js @@ -70,6 +70,14 @@ export function autoMatchShopeeSpecMappings(productId, data) { return request({ url: `/api/admin/v1/shopee-products/${productId}/specs/mapping/auto-match`, method: 'post', data, timeout: aiSuggestTimeoutMs }) } +export function startShopeeSpecAutoMatchRun(data) { + return request({ url: '/api/admin/v1/shopee-spec-auto-match/runs', method: 'post', data }) +} + +export function getLatestShopeeSpecAutoMatchRun() { + return request({ url: '/api/admin/v1/shopee-spec-auto-match/runs/latest', method: 'get' }) +} + export function batchDeleteShopeeProducts(data) { return request({ url: '/api/admin/v1/shopee-products/batch-delete', method: 'post', data }) } diff --git a/web/src/views/goauto/shopee-products/index.vue b/web/src/views/goauto/shopee-products/index.vue index e30bdfc..2b2ac0f 100644 --- a/web/src/views/goauto/shopee-products/index.vue +++ b/web/src/views/goauto/shopee-products/index.vue @@ -1,7 +1,7 @@