Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 ila
|
b18f9cc4a5 | ||
|
ila
|
a55708cd37 |
@@ -2,7 +2,7 @@
|
||||
|
||||
面向自托管场景的阅读式语言学习项目，规划提供内容导入、阅读查词、词汇与短语、复习和实例管理。
|
||||
|
||||
已确认：**DevHarness 轻量模式、MySQL 8、go-admin 管理端**。工程基础 #2 已通过验收：两端用户名登录、学习账号管理、可撤销会话和本人英语空空间。管理端基于指定 go-admin/go-admin-ui 选用模块，学习端为独立 Vue 3 + TypeScript + Vite 工程，共用 Go 后端和 MySQL 8.4.3。#18 登录日志与操作审计已通过用户验收，支持管理员查询和 90 天保留清理。阅读、导入、词典与复习尚未实现。MVP 定位为“支持多账号、数据独立的自托管学习工具”，先邀请少量用户使用；F01～F12 已确认，X 系列后置。
|
||||
已确认：**DevHarness 轻量模式、MySQL 8、go-admin 管理端**。工程基础 #2 已通过验收：两端用户名登录、学习账号管理、可撤销会话和本人英语空空间。管理端基于指定 go-admin/go-admin-ui 选用模块，学习端为独立 Vue 3 + TypeScript + Vite 工程，共用 Go 后端和 MySQL 8.4.3。#18 登录日志与操作审计已通过用户验收，支持管理员查询和 90 天保留清理。#5 粘贴导入与章节原文阅读已实现，待用户验收；点词查词、词典、词语状态与复习仍未实现。MVP 定位为“支持多账号、数据独立的自托管学习工具”，先邀请少量用户使用；F01～F12 已确认，X 系列后置。
|
||||
|
||||
- [文档入口](docs/README.md) · [线上 Wiki](https://git.ilapage.cn/OPC/lexgo/wiki/Home)
|
||||
- [英语分词与离线词典验证小样](spikes/english/README.md)（#3 已验收，独立本机入口）
|
||||
@@ -19,7 +19,7 @@
|
||||
|
||||
本地入口：学习端 http://127.0.0.1:5173，管理端 http://127.0.0.1:5174。完整安装与测试命令见[开发与验证](docs/04-local-development-and-verification.md)。账号使用用户名，无需邮箱；没有随代码交付的默认密码。
|
||||
|
||||
本机三个服务已由 `D:/supervisord/supervisord.conf` 中的 lexgo-learner、lexgo-admin、lexgo-api 托管，使用时不要重复手动启动同端口。当前 schema v2；从 #2 升级时停止 API，执行 build、migrate 后再启动。审计过期清理可执行 `python scripts/server.py audit-cleanup`，只影响超过 90 天的日志。
|
||||
本机三个服务已由 `D:/supervisord/supervisord.conf` 中的 lexgo-learner、lexgo-admin、lexgo-api 托管，使用时不要重复手动启动同端口。当前 schema v3；从 #2/#18 升级时停止 API，执行 build、migrate 后再启动。审计过期清理可执行 `python scripts/server.py audit-cleanup`，只影响超过 90 天的日志。
|
||||
|
||||
## 文档与治理
|
||||
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件)
|
||||
wiki_page: Architecture-and-Code-Map
|
||||
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Architecture-and-Code-Map.-
|
||||
wiki_revision: 5bc7ccf33d1cc1cc28598c90e27a2ab6af4146bd
|
||||
synchronized_at: 2026-09-10T14:33:23Z
|
||||
wiki_revision: f981b86a7c6e5d2e353142d3006dcd6e0636c3ee
|
||||
synchronized_at: 2026-09-11T03:01:48Z
|
||||
<!-- gitea-wiki-mirror:end -->
|
||||
|
||||
# 架构与代码地图
|
||||
@@ -175,3 +175,49 @@ WordNet 使用 ZIP 内原始 index/data/exception 文件，不使用 SysDict 或
|
||||
- [resources/js/components/Review/ReviewHotkeyInformationDialog.vue](https://github.com/simjanos-dev/LinguaCafe/blob/c1ea298ce40c65b9dd33e9b26fd2e52fae66f2c8/resources/js/components/Review/ReviewHotkeyInformationDialog.vue)
|
||||
|
||||
上游 LICENSE 为 GPL v3，本单只核对并描述行为，没有移植源码。#1 评论 7497 的四项对照登记由本节补充；原型 v1 的保存/关闭主要流程保持，不更改其历史验收记录，不声称已经在 Quant-UX 新建修订版。范围调整与原生手柄作为可运行小样验证，若真机结果导致主要流程变化，应先更新关键原型状态并由用户确认。
|
||||
|
||||
## 粘贴导入、章节与阅读（#5，schema v3）
|
||||
|
||||
#5 实现了目标路径一的第一段可运行链路：粘贴英语文本 → 持久导入任务 → 固定分章 → 处理中/就绪 → 本人阅读原文。Go 单进程同时承担 API 与后台任务。本单不接入 Python NLP，token、lemma 与词典索引仍待 #6，路线未决边界见业务规则页。
|
||||
|
||||
| 路径 | 职责 |
|
||||
|---|---|
|
||||
| server/app/lexgo/database.go | schema v3 显式迁移：lexgo_books、lexgo_chapters、lexgo_ingest_jobs；按版本累加语句，版本行只在全部语句成功后推进 |
|
||||
| server/app/lexgo/library.go | 粘贴校验与固定分章、书籍/章节/任务写入、本人归属查询、重试与请求幂等 |
|
||||
| server/app/lexgo/ingest.go | 任务声明 claim、处理完成、固定失败原因、启动恢复 |
|
||||
| server/app/lexgo/router.go | 新增书籍/章节/任务路由；粘贴请求使用独立的 4 MiB 体积上限 |
|
||||
| server/cmd/lexgo/main.go | serve 启动时恢复遗留任务，并按秒轮询处理待处理任务 |
|
||||
| learner/src/stores/library.ts、views/ImportView.vue、BookView.vue、ReaderView.vue | 粘贴导入、书库与章节状态、失败重试、原文阅读 |
|
||||
|
||||
任务状态为 pending → processing → ready/failed，章节与任务共用同一套状态词。声明与完成分属两个事务：声明一经提交，即使进程随即退出，也只会留下可被启动恢复重新入队的 processing 记录。
|
||||
|
||||
### 粘贴导入 API v1（#5）
|
||||
|
||||
| 方法与路径 | 行为和权限 |
|
||||
|---|---|
|
||||
| POST /api/v1/books | {requestId,title,text,language?}；创建书籍＋首个章节＋导入任务；重复 requestId 返回首次结果（HTTP 200，duplicate=true） |
|
||||
| POST /api/v1/books/:id/chapters | 向本人书籍追加一个章节 |
|
||||
| GET /api/v1/books | 本人书库与章节状态计数；拒绝查询参数，避免用参数替换认证身份 |
|
||||
| GET /api/v1/books/:id | 本人书籍与章节列表，含 jobId 与可读失败原因 |
|
||||
| GET /api/v1/chapters/:id | 本人章节详情；仅 ready 时返回 originalText，并附带前后章节编号 |
|
||||
| GET /api/v1/jobs/:id | 本人任务状态、尝试次数与失败原因 |
|
||||
| POST /api/v1/jobs/:id/retry | 仅失败任务可重试；复用同一章节，不新建章节 |
|
||||
|
||||
所有接口按认证身份过滤 owner_id；他人书籍、章节或任务编号统一返回 404，管理员角色也不能解除学习数据的本人归属过滤。后台任务只使用任务行内的 owner_id，不接受客户端用户编号；请求体含未知字段（例如 ownerId）直接返回 400。
|
||||
|
||||
### schema v3
|
||||
|
||||
lexgo_books(owner_id, title, language)、lexgo_chapters(book_id, owner_id, ordinal, title, original_text MEDIUMTEXT, char_count, content_sha256, status, error_reason) 与 lexgo_ingest_jobs(owner_id, book_id, chapter_id, request_key, content_sha256, status, attempts, error_reason, finished_at)。owner_id 在章节与任务上冗余存放，使任何查询都能直接按认证身份过滤而不依赖连接；UNIQUE(book_id, ordinal) 与 UNIQUE(owner_id, request_key) 分别阻止重复章节与重复提交。启动检查要求版本 3，服务不自动迁移。
|
||||
|
||||
并发重复提交：请求命中 request_key 唯一键冲突后，用加锁读读取已提交结果，因为该请求事务的快照早于并发提交；因此两个并发相同提交只会产生一个章节，另一个得到 duplicate=true 的首次结果。
|
||||
|
||||
## #5 审核整改（R1～R4，2026-09-11）
|
||||
|
||||
提交见工单 #5 的整改评论；本条记录实现与验证方式。
|
||||
|
||||
- 追加契约（R1）：学习端把新建与追加拆成两个请求类型，追加不发送 language；后端保持严格解码，并新增回归测试断言“追加带 language 返回 400、不带则 201”，学习端单测断言追加请求体只有 requestId/title/text。
|
||||
- 运行期任务恢复（R2）：`server/app/lexgo/ingest.go` 的恢复逻辑合并为一处——启动恢复使用阈值 0，运行期每轮清扫使用 15 秒阈值并把超过 5 次尝试的任务置为 failed（原因码 attempts_exhausted）；`cmd/lexgo/main.go` 的 worker 每秒先清扫再处理，日志分别说明“已重新入队”与“本批未完成、等待下一次清扫”，不再声称已完成实际跳过的重试。
|
||||
- 离页作废在途请求（R3）：`closeBook`/`closeChapter` 推进请求序号并清理 loading；`ImportView` 记录是否已卸载，卸载后的成功响应不再触发跳转。
|
||||
- 重试自愈（R4）：`retryChapter` 先把重试返回的章节状态应用到列表与阅读器并重新安排轮询，再做静默刷新。
|
||||
|
||||
验证：Go 全量用例 20 项通过（新增运行期恢复与尝试上限两项）；学习端单测 38 项通过，其中 7 项在整改前的代码上复现失败；真实联调确认追加路径可用、被中断的任务在运行中被自动恢复（约 0.5 秒，无需重启）、重试在首次刷新失败后仍自动显示最终结果。
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件)
|
||||
wiki_page: Business-Rules-and-Glossary
|
||||
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Business-Rules-and-Glossary.-
|
||||
wiki_revision: 227c6cb92d30490f0f1f4b1b9f29cdbed8559ee4
|
||||
synchronized_at: 2026-09-10T14:33:25Z
|
||||
wiki_revision: 4bd843d1f0134f5688407b7072d98333b40b6da1
|
||||
synchronized_at: 2026-09-11T03:01:48Z
|
||||
<!-- gitea-wiki-mirror:end -->
|
||||
|
||||
# 业务规则与术语
|
||||
@@ -112,3 +112,28 @@ POST /lookup 接收 {surface,lemma?}。查词键单独 casefold/NFC/弯撇号转
|
||||
桌面鼠标/手机长按使用原生 Selection；正文中键盘 ←/→ 选相邻词，Shift+←/→ 从锚点扩缩连续范围，起点/终点按钮可用 Tab/Enter。Esc 或关闭清除选区及未保存编辑，保留滚动和键盘阅读位置；切换章节清除选区，当前页面内分别记住章节滚动位置。纯标点的新选区清除旧面板，防止操作上一个词。个人释义必须显式保存，状态变化也随保存提交；本小样仅写内存。
|
||||
|
||||
手机方案保留浏览器原生长按、选择手柄和滚动，面板最多占底部 42dvh，正文有底部阅读余量；键盘定位用滚动边距避开面板。未获得真实手机结果，不能判定手柄、系统菜单、虚拟键盘或触摸滚动冲突已经解决。
|
||||
|
||||
## #5 粘贴分章、任务与阅读规则 v1
|
||||
|
||||
用户于 2026-09-10 确认两项边界（工单 #5 评论 7644）：本单按 Go 处理、不接入 Python；一次粘贴等于一个章节。
|
||||
|
||||
- 固定分章规则：一次粘贴产生一个章节，不按空行或长度自动再分。新建书籍时书籍标题与首章标题同为提交的标题；追加时标题即新章节标题。分章规则变化属于需求变化，必须重新确认。
|
||||
- 长度与校验：标题去首尾空白后 1～120 个字符；正文必须含至少一个非空白字符；正文上限 100000 Unicode code point，超出返回 400；语言当前只接受 en。
|
||||
- 原文保真：正文按收到的字符串原样保存与返回，不做 NFC、大小写、换行或空白归一化；页面使用 white-space: pre-wrap 展示，制表符、连续空格与空行保持可见。处理完成前不返回原文。
|
||||
- 归属：书籍、章节与任务都记录认证账号的 owner_id；他人编号返回 404；管理员角色不解除学习数据的本人归属；后台任务只使用任务行的 owner，不信任客户端用户编号。
|
||||
- 任务状态：pending、processing、ready、failed，章节与任务共用同一词表。失败时返回固定原因码加可读中文提示，错误字段不保存正文。
|
||||
- 固定失败原因：unsupported_language、too_long、empty_text、content_changed。前三种只能由其他写入路径产生（例如语言调整或后续编辑功能）；content_changed 表示章节内容在处理前被改动，属于过期任务，必须重新提交，或恢复为提交时的内容后重试。
|
||||
- 幂等：客户端 requestId 与账号构成唯一键。同一 requestId 配同标题同正文的重复提交返回首次结果，不新建章节；同一 requestId 配不同标题或正文返回 409；并发重复提交同样只产生一个章节。重试复用原章节，只增加尝试次数。
|
||||
- 恢复：声明与完成分属两个事务。进程在声明后退出时，重启把 processing 的章节与任务放回 pending 并保留尝试次数，不产生重复章节。
|
||||
- 未决边界：Go＋Python NLP 与全 Go 路线尚未确认。本单只做 Go 校验、分章与发布，不产生 token、lemma 或词典索引；正式接入前必须由用户确认路线，再定义生产 token 契约。
|
||||
- 已知限制：本单未设置每账号书籍数量或总容量配额，只限制单次正文与请求体大小；删除书籍/章节属 #10，导入失败不会自动重试，只在启动时恢复被中断的 processing 任务。
|
||||
|
||||
## #5 审核整改（R1～R4，2026-09-11）
|
||||
|
||||
工单 #5 的整改记录见该工单评论；本节只记录长期有效的契约变化。
|
||||
|
||||
- 追加章节沿用所属书籍的语言：`POST /api/v1/books/:id/chapters` 的请求体只有 requestId、title、text，不接受 language；服务端仍拒绝未知字段，客户端发 language 会得到 400。新建书籍的 `POST /api/v1/books` 才带 language。学习端已按此拆分请求体，避免两个契约共用同一结构。
|
||||
- 导入任务的自动重试有上限：同一任务被 worker 领取的次数达到 5 次后，任务与章节转为 failed，原因码 `attempts_exhausted`，提示“处理多次失败，请重试或重新提交”。人工重试（POST /jobs/:id/retry）会重置尝试次数，因此人工操作不受该上限阻塞。
|
||||
- 运行期恢复不依赖重启：除了启动时的恢复，运行中的服务每次轮询都会把停留在 processing 且超过 15 秒的任务放回 pending，因此“领取已提交、完成事务失败”不会让章节永久卡在处理中。该阈值必须长于正常的领取到完成窗口；重复处理同一任务不会产生第二个章节，因为任务从不创建章节。
|
||||
- 学习端离开页面时作废在途请求：目录页与阅读页在关闭时推进各自的请求序号，晚到的响应不会写回状态或重启轮询；导入页在提交过程中离开后，晚到的成功响应不会把用户导航回书籍页。
|
||||
- 重试结果立即生效：重试被接受后先把返回的 pending 状态写入界面并继续轮询，因此紧随其后的一次刷新失败不会让页面停在处理失败。
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件)
|
||||
wiki_page: Local-Development-and-Verification
|
||||
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Local-Development-and-Verification.-
|
||||
wiki_revision: 694facbb7a861182122c9f6c69d28c58b005cb1a
|
||||
synchronized_at: 2026-09-10T14:52:22Z
|
||||
wiki_revision: 3bce5fae964b1050513c37319262679814d68f14
|
||||
synchronized_at: 2026-09-11T03:01:49Z
|
||||
<!-- gitea-wiki-mirror:end -->
|
||||
|
||||
# 本地开发与验证
|
||||
@@ -246,3 +246,62 @@ node --test spikes/english/view.test.mjs
|
||||
|
||||
|
||||
#4 用户验收记录：2026-09-10T22:51:52+08:00 用户确认“#4通过验收”（评论 7636）。未补充手机型号/浏览器/操作记录，未重跑测试，未自动合并 PR。后续正式移动端集成应补真机回归。
|
||||
|
||||
## #5 粘贴导入与章节阅读（schema v3）
|
||||
|
||||
升级步骤（本机，仓库根执行）：停止 lexgo-api → `python scripts/server.py build` → `python scripts/server.py migrate` → 启动 lexgo-api。lexgo_dev 已从 v2 升到 v3，新增 lexgo_books、lexgo_chapters、lexgo_ingest_jobs；迁移前后 sys_user 4、lexgo_spaces 4、lexgo_sessions 3、lexgo_login_logs 23、lexgo_operation_logs 1 完全一致。托管实例重启后 /healthz 返回 200，两端首页仍为 200，lexgo-admin 与 lexgo-learner 的 PID 未变化。
|
||||
|
||||
回退：停止 API，把 lexgo_schema 中 id=1 的版本从 3 改回 2，并恢复上一二进制；三张新表保留不删除，旧程序不读写它们。重新升级时显式 migrate 重新执行 IF NOT EXISTS 语句即可；集成测试覆盖 v2→v3 的既有数据保留与 v2 标记下的重复迁移。未在开发库演练回退。
|
||||
|
||||
学习端入口：http://127.0.0.1:5173 → 登录 → 我的书库 → 粘贴文本导入 → 章节就绪后进入阅读。
|
||||
|
||||
测试命令与结果（仓库根执行；本单使用专用库 lexgo_test_issue5，不借用其他测试库）：
|
||||
|
||||
| 命令 | 本次结果 |
|
||||
|---|---|
|
||||
| `python scripts/server.py test-integration`（LEXGO_TEST_DB_NAME=lexgo_test_issue5） | 全部通过：18 个顶层用例，其中 #5 新增 8 个（7 个书库/章节/任务/阅读 + 1 个 v2→v3 数据保留），另含 12 个子用例；既有 10 个用例保持通过 |
|
||||
| `npx --yes pnpm@9.15.1 --dir learner test:unit --run` | 3 个文件 31 项通过（session 9、library 16、reading 6） |
|
||||
| `npx --yes pnpm@9.15.1 --dir learner build` | vue-tsc 类型检查与 vite 构建通过，退出码 0 |
|
||||
| `npx --yes pnpm@9.15.1 --dir learner test:e2e` | 3 项通过（既有 auth 2 项 + 新增 reading 1 项，均为虚构 API 响应） |
|
||||
|
||||
### 真实 API + MySQL 实测（2026-09-10，lexgo_dev）
|
||||
|
||||
使用本单新建的虚构账号 issue5_a、issue5_b，口令只保存在忽略的 .local/issue5-accounts.json；未改动 admin、dev、learner_a、learner_b。脚本 .local/verify-issue5-api.ps1 只在本机运行，不输出口令。
|
||||
|
||||
- 粘贴：HTTP 201，章节与任务均为 pending，charCount 99。
|
||||
- 处理：实测状态序列 pending → ready，约 1132 ms（后台任务每秒轮询）；job attempts=1。
|
||||
- 阅读：originalText 与提交正文逐字符相等，CRLF、制表符、弯引号、破折号、省略号、é 加组合重音、emoji、行尾空格与空行全部保留；sha256 前缀 ce7357ea22a3。
|
||||
- 幂等：同一 requestId 重复提交 HTTP 200、duplicate=true、章节与任务编号不变；同一 requestId 换正文 HTTP 409；书库仍为 1 本。
|
||||
- 隔离：issue5_b 读取 issue5_a 的书籍、章节、任务以及追加、重试全部 404；请求体带 ownerId 与查询参数 ownerId 均 400；issue5_b 书库为空。
|
||||
- 追加与阅读导航：新章节 ordinal=2，处理后就绪，前后章节编号互相指向。
|
||||
- 校验：空标题、纯空白正文、非 en 语言、缺少 requestId、超过 100000 code point 分别返回 400 与可读中文提示。
|
||||
- 遗留 fixture：lexgo_dev 中 issue5_a 名下 1 本虚构书、2 个就绪章节（bookId=1，章节 1、2）。
|
||||
|
||||
浏览器实测：真实学习端 + 真实 API + 真实 MySQL 联测（临时 Playwright 用例，运行后删除）：issue5_a 登录 → 书库显示既有虚构书与“导入内容”入口 → 导入页粘贴含空行、制表符、行尾空格与 emoji 的正文 → 书库页由“处理中”变为“已就绪” → 阅读页 article.reader-text 的 textContent 与粘贴正文逐字符相等、computed white-space 为 pre-wrap → “下一章”切换到第二章且正文精确相等 → 390×844 视口下横向溢出 0 px。同一轮还运行了 3 项虚构 API 的既有 e2e，共 4 项通过。
|
||||
|
||||
截图保存在本机 .local/evidence/（library.png、book-after-paste.png、reader-desktop.png、reader-mobile-390.png）并作为工单附件上传，便于人工目视复核；本次会话模型不能读取图片，截图未经 Agent 目视检查，功能断言来自上面的程序化检查。
|
||||
|
||||
未验证：处理失败到重试的用户界面路径只由集成测试覆盖（无法通过 API 主动制造处理失败）；真实手机浏览器长按、手柄、滚动与虚拟键盘仍属 #4 缺口，本次只用桌面浏览器窄屏检查，不能当作真机结果；Python NLP 未接入，token、lemma 与词典仍为 #3 小样范围；生产并发、容量、备份恢复与部署不在本单范围。
|
||||
|
||||
## #5 审核整改验证（R1～R4，2026-09-11）
|
||||
|
||||
整改提交与完整证据见工单 #5 的整改评论。本次复核命令与结果（仓库根执行，专用测试库 lexgo_test_issue5）：
|
||||
|
||||
| 命令 | 本次结果 |
|
||||
|---|---|
|
||||
| `python scripts/server.py test-integration` | 20 个顶层用例全部通过（新增 `TestMySQLIngestRecoveryWithoutRestart`、`TestMySQLIngestAttemptsAreBoundedAndManualRetryRestarts`） |
|
||||
| `npx --yes pnpm@9.15.1 --dir learner test:unit --run` | 38 项通过（library 21、reading 8、session 9） |
|
||||
| `npx --yes pnpm@9.15.1 --dir learner build` | 通过（vue-tsc + vite） |
|
||||
| `npx --yes pnpm@9.15.1 --dir learner test:e2e` | 3 项通过（虚构 API 响应） |
|
||||
|
||||
回归测试的有效性：新增的前端 7 项用例先在整改前的 `library.ts`/`ImportView.vue` 上运行并实际失败（追加发送 language、离页后响应写回、重试后停在失败），改回修复版本后全部通过。
|
||||
|
||||
真实联调（lexgo_dev，虚构账号 issue5_a）：
|
||||
|
||||
- R1 追加：真实学习端从书籍页进入“追加章节”，提交后回到书籍页，新章节就绪后可阅读；抓取到的请求体只有 requestId、title、text，无 language；正文逐字符相等。
|
||||
- R2 恢复：新建章节后用 SQL 把任务与章节置为 processing 且 updated_at 早于阈值（UTC 时间），**不重启服务**，运行期清扫在 544 ms 内把任务重新入队并发布为就绪，章节与任务编号不变，正文逐字符相等。
|
||||
- R4 重试：SQL 制造真实失败任务（content_changed）后，在浏览器点击“重试”并中断其后的第一次刷新请求，页面立即由“处理失败”变为“处理中”，随后自行变为“已就绪”，无需手工刷新。
|
||||
|
||||
注意：MySQL 会话时区为 SYSTEM（本机为 UTC+8），而服务按 UTC 存储 DATETIME；核对任务时间时使用 UTC_TIMESTAMP 而不是 NOW()，否则会出现 8 小时的假偏差。
|
||||
|
||||
未在本轮验证：R3 的真实浏览器时序（离页与响应同时发生）只由单测覆盖；真机手机证据仍属 #4 缺口。
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件)
|
||||
wiki_page: Product-Requirements-Overview
|
||||
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Product-Requirements-Overview.-
|
||||
wiki_revision: db82eeb1d509b8d3c4a2204143cfb0e829b6b083
|
||||
synchronized_at: 2026-09-10T12:43:32Z
|
||||
wiki_revision: 125df697d3d45dee098616d25970bb65a96727cd
|
||||
synchronized_at: 2026-09-10T16:35:19Z
|
||||
<!-- gitea-wiki-mirror:end -->
|
||||
|
||||
# 产品需求总览
|
||||
@@ -223,3 +223,9 @@ Anki（U20）、YouTube/Jellyfin 远程字幕（U07/A08）、全量旧实例迁
|
||||
## 试用前管理能力增补
|
||||
|
||||
用户于 2026-09-10 验收 #2，并批准新增 #18 登录日志与操作审计：两个管理员列表、查询筛选分页、必要字段记录、权限及 90 天保留清理。沿用现有 go-admin 管理布局，无需独立高保真原型。已通过用户验收，纳入 #16“邀请用户试用前完成”；不替代 #3 英语分词和 #4 阅读选择验证，也不引入其他 go-admin 默认模块。
|
||||
|
||||
## 实施进度增补（#5）
|
||||
|
||||
2026-09-10：#5“粘贴英语文本，处理后进入本人章节阅读”已实现并待用户验收，覆盖 F01 基础（书库与章节）、F02（粘贴导入）、F04 基础（处理状态与失败重试）与 F05 原文（可读原文与章节切换）。本节取代此前“阅读、导入尚未实现”的表述：粘贴导入与原文阅读已实现；点词查词、词典、个人词语状态、复习与统计仍未实现（#6～#15）。schema 升级为 v3，新增 lexgo_books、lexgo_chapters、lexgo_ingest_jobs。
|
||||
|
||||
范围边界不变：本单按 Go 处理，不接入 Python NLP，因此不产生 token、lemma 或词典索引；Go＋Python NLP 与全 Go 路线仍未确认，正式接入前必须由用户确认。真机手机详细证据仍缺失（#4 缺口保持）。
|
||||
|
||||
+4
-2
@@ -2,8 +2,8 @@
|
||||
generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件)
|
||||
wiki_page: Home
|
||||
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Home
|
||||
wiki_revision: a645b99632c70734b493ed030ad6d4d2acfd13be
|
||||
synchronized_at: 2026-09-10T14:52:14Z
|
||||
wiki_revision: 5b936b75caeec89cd54f7147c842f0b66f901307
|
||||
synchronized_at: 2026-09-10T16:35:18Z
|
||||
<!-- gitea-wiki-mirror:end -->
|
||||
|
||||
# LexGo 文档入口
|
||||
@@ -63,3 +63,5 @@ Quant-UX 原型 v1 已通过用户验收。[桌面预览](https://qux.ilapage.cn
|
||||
|
||||
|
||||
#4 阅读选择小样已实现桌面鼠标/键盘、范围调整与原文位置验证，入口 http://127.0.0.1:5184/；固定版本 LinguaCafe 四项源码对照已记录。用户已验收并关闭 #4；真实手机详细测试证据仍缺失，详见本地验证页。#21 仍待实施。
|
||||
|
||||
#5 粘贴导入与章节阅读已实现，待用户验收：schema v3 新增 books/chapters/ingest_jobs，学习端具备粘贴导入、书库、处理状态与原文阅读；查词、词典与复习仍未实现。本单按 Go 处理，Go＋Python NLP 与全 Go 路线仍未决，正式接入前须用户确认。
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
import { expect, test } from '@playwright/test'
|
||||
|
||||
test('paste English text, watch a chapter finish processing, then read it verbatim', async ({ page }) => {
|
||||
const user = { id: 42, username: 'fictional-reader', role: 'learner' }
|
||||
const book = { id: 1, title: '虚构样例书', language: 'en' }
|
||||
const chapterTitle = '虚构样例第一章'
|
||||
// Line breaks, a tab and repeated spaces must survive the whole round trip.
|
||||
const pasted = 'First line of the chapter.\n\tIndented line.\nTwo spaces kept.\n\nLast line.\n'
|
||||
const timestamps = { createdAt: '2026-01-01T00:00:00Z', updatedAt: '2026-01-01T00:00:00Z' }
|
||||
// The worker reports the fresh chapter as processing until the worker settles it.
|
||||
let status: 'processing' | 'ready' = 'processing'
|
||||
const chapterPayload = () => ({
|
||||
id: 55,
|
||||
bookId: book.id,
|
||||
ordinal: 1,
|
||||
title: chapterTitle,
|
||||
status,
|
||||
charCount: [...pasted].length,
|
||||
errorReason: '',
|
||||
errorMessage: '',
|
||||
jobId: 7,
|
||||
...timestamps,
|
||||
})
|
||||
|
||||
await page.route('**/api/v1/**', async route => {
|
||||
const path = new URL(route.request().url()).pathname
|
||||
const method = route.request().method()
|
||||
let data: unknown = null
|
||||
let statusCode = 200
|
||||
if (path === '/api/v1/login') {
|
||||
expect(route.request().postDataJSON()).toEqual({ username: user.username, password: 'fictional-password' })
|
||||
data = { token: 'fictional-session', user }
|
||||
} else if (path === '/api/v1/me') data = user
|
||||
else if (path === '/api/v1/space') data = { ownerId: user.id, language: 'en' }
|
||||
else if (path === '/api/v1/books' && method === 'GET') {
|
||||
data = { items: [{ ...book, chapterCount: 0, pendingCount: 0, processingCount: 0, readyCount: 0, failedCount: 0, ...timestamps }] }
|
||||
} else if (path === '/api/v1/books' && method === 'POST') {
|
||||
const body = route.request().postDataJSON() as { requestId: string; title: string; text: string; language: string }
|
||||
expect(body.requestId).toMatch(/^[0-9a-f-]{36}$/)
|
||||
expect(body).toMatchObject({ title: chapterTitle, text: pasted, language: 'en' })
|
||||
statusCode = 201
|
||||
data = {
|
||||
book,
|
||||
chapter: chapterPayload(),
|
||||
job: { id: 7, bookId: book.id, chapterId: 55, status, attempts: 0, errorReason: '', errorMessage: '', ...timestamps },
|
||||
duplicate: false,
|
||||
}
|
||||
} else if (path === '/api/v1/books/1') data = { book, chapters: [chapterPayload()] }
|
||||
else if (path === '/api/v1/chapters/55') {
|
||||
data = {
|
||||
book,
|
||||
chapter: { ...chapterPayload(), contentSha256: 'fictional-sha256', ...(status === 'ready' ? { originalText: pasted } : {}) },
|
||||
navigation: { previousChapterId: null, nextChapterId: null },
|
||||
}
|
||||
}
|
||||
await route.fulfill({ status: statusCode, json: { code: 200, data } })
|
||||
})
|
||||
|
||||
await page.goto('/')
|
||||
await page.getByLabel('账号').fill(user.username)
|
||||
await page.getByLabel('密码', { exact: true }).fill('fictional-password')
|
||||
await page.getByRole('button', { name: '登录', exact: true }).click()
|
||||
|
||||
// The library lists the caller's book.
|
||||
await expect(page.getByRole('heading', { name: '我的书库' })).toBeVisible()
|
||||
await expect(page.getByRole('link', { name: book.title })).toBeVisible()
|
||||
|
||||
// Paste text through the import form.
|
||||
await page.getByRole('button', { name: '导入内容' }).click()
|
||||
await expect(page.getByRole('heading', { name: '导入英文内容' })).toBeVisible()
|
||||
await page.getByLabel('标题').fill(chapterTitle)
|
||||
await page.getByLabel('正文').fill(pasted)
|
||||
await page.getByRole('button', { name: '开始处理' }).click()
|
||||
|
||||
// The new book opens with the chapter still processing…
|
||||
await expect(page).toHaveURL(/\/books\/1$/)
|
||||
await expect(page.getByText('处理中')).toBeVisible()
|
||||
|
||||
// …and the browser poll turns it ready without a page reload.
|
||||
status = 'ready'
|
||||
await expect(page.getByText('已就绪')).toBeVisible({ timeout: 15000 })
|
||||
|
||||
// Open the chapter and check the pasted text survived verbatim.
|
||||
await page.getByRole('link', { name: chapterTitle }).click()
|
||||
await expect(page).toHaveURL(/\/chapters\/55$/)
|
||||
const readerText = page.locator('.reader-text')
|
||||
await expect(readerText).toBeVisible()
|
||||
expect(await readerText.evaluate(element => element.textContent)).toBe(pasted)
|
||||
expect(await readerText.evaluate(element => getComputedStyle(element).whiteSpace)).toBe('pre-wrap')
|
||||
await expect(page.getByRole('button', { name: '上一章' })).toBeDisabled()
|
||||
await expect(page.getByRole('button', { name: '下一章' })).toBeDisabled()
|
||||
})
|
||||
@@ -0,0 +1,445 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import { createPinia, setActivePinia } from 'pinia'
|
||||
import {
|
||||
NOT_FOUND_MESSAGE,
|
||||
POLL_INTERVAL_MS,
|
||||
TEXT_MAX_CODE_POINTS,
|
||||
canRetry,
|
||||
statusSummary,
|
||||
useLibraryStore,
|
||||
type ChapterDetail,
|
||||
} from '../stores/library'
|
||||
import { useSessionStore } from '../stores/session'
|
||||
|
||||
// All accounts, books and texts in these tests are deliberately fictitious.
|
||||
const user = { id: 7, username: 'fictional-reader', role: 'learner' as const }
|
||||
const book = { id: 1, title: '虚构样例书', language: 'en' }
|
||||
const navigation = { previousChapterId: null, nextChapterId: null }
|
||||
|
||||
const ok = (data: unknown) => new Response(JSON.stringify({ code: 200, data }), { status: 200 })
|
||||
const created = (data: unknown) => new Response(JSON.stringify({ code: 200, data }), { status: 201 })
|
||||
const httpError = (status: number, msg: string) => new Response(JSON.stringify({ code: status, msg }), { status })
|
||||
|
||||
const chapter = (overrides: Partial<ChapterDetail> = {}): ChapterDetail => ({
|
||||
id: 55,
|
||||
bookId: 1,
|
||||
ordinal: 1,
|
||||
title: '第一篇',
|
||||
status: 'pending',
|
||||
charCount: 120,
|
||||
errorReason: '',
|
||||
errorMessage: '',
|
||||
contentSha256: 'fictional-sha256',
|
||||
jobId: 7,
|
||||
createdAt: '2026-01-01T00:00:00Z',
|
||||
updatedAt: '2026-01-01T00:00:00Z',
|
||||
...overrides,
|
||||
})
|
||||
|
||||
const summary = (overrides: Partial<Record<string, number | string>> = {}) => ({
|
||||
...book,
|
||||
chapterCount: 0,
|
||||
pendingCount: 0,
|
||||
processingCount: 0,
|
||||
readyCount: 0,
|
||||
failedCount: 0,
|
||||
createdAt: '2026-01-01T00:00:00Z',
|
||||
updatedAt: '2026-01-01T00:00:00Z',
|
||||
...overrides,
|
||||
})
|
||||
|
||||
const job = { id: 7, bookId: 1, chapterId: 55, status: 'pending', attempts: 0, errorReason: '', errorMessage: '', createdAt: '2026-01-01T00:00:00Z', updatedAt: '2026-01-01T00:00:00Z' }
|
||||
|
||||
const fetchMock = () => vi.mocked(globalThis.fetch)
|
||||
const paths = () => fetchMock().mock.calls.map(([input]) => String(input))
|
||||
const bodyOf = (index: number): Record<string, unknown> => JSON.parse(String(fetchMock().mock.calls[index]?.[1]?.body)) as Record<string, unknown>
|
||||
|
||||
async function signIn() {
|
||||
fetchMock().mockResolvedValueOnce(ok({ token: 'fictional-token', expiresAt: '2030-01-01', user }))
|
||||
const session = useSessionStore()
|
||||
await session.login(user.username, 'fictional-password')
|
||||
return session
|
||||
}
|
||||
|
||||
describe('learner library store', () => {
|
||||
beforeEach(() => {
|
||||
sessionStorage.clear()
|
||||
setActivePinia(createPinia())
|
||||
vi.restoreAllMocks()
|
||||
// Any request a test did not expect fails loudly instead of hanging.
|
||||
vi.spyOn(globalThis, 'fetch').mockImplementation(input => {
|
||||
throw new Error(`unexpected request: ${String(input)}`)
|
||||
})
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
// Never let a polling timer outlive its test.
|
||||
useLibraryStore().stopPolling()
|
||||
vi.useRealTimers()
|
||||
})
|
||||
|
||||
it('loads the library and summarises the count fields the API reports', async () => {
|
||||
await signIn()
|
||||
const library = useLibraryStore()
|
||||
fetchMock().mockResolvedValueOnce(ok({ items: [summary({ chapterCount: 5, readyCount: 2, processingCount: 1, pendingCount: 1, failedCount: 1 })] }))
|
||||
|
||||
await library.loadBooks()
|
||||
|
||||
expect(library.books).toHaveLength(1)
|
||||
expect(library.booksLoading).toBe(false)
|
||||
expect(library.booksError).toBe('')
|
||||
expect(statusSummary(library.books[0]!)).toBe('已就绪 2 · 处理中 1 · 待处理 1 · 失败 1')
|
||||
expect(paths()).toEqual(['/api/v1/login', '/api/v1/books'])
|
||||
expect(fetchMock().mock.calls[1]?.[1]?.headers).toMatchObject({ Authorization: 'Bearer fictional-token' })
|
||||
})
|
||||
|
||||
it('shows pending and processing separately instead of deriving one from a total', async () => {
|
||||
// processingCount is strictly "processing" now, so pendingCount must be read as given.
|
||||
const queued = summary({ chapterCount: 3, readyCount: 1, processingCount: 0, pendingCount: 2, failedCount: 0 })
|
||||
expect(statusSummary(queued)).toBe('已就绪 1 · 待处理 2')
|
||||
expect(statusSummary(queued)).not.toContain('处理中')
|
||||
const done = summary({ chapterCount: 1, readyCount: 1 })
|
||||
expect(statusSummary(done)).toBe('已就绪 1')
|
||||
expect(statusSummary(summary({ chapterCount: 0 }))).toBe('')
|
||||
})
|
||||
|
||||
it('loads a book detail whose chapters carry the job id used for retry', async () => {
|
||||
await signIn()
|
||||
const library = useLibraryStore()
|
||||
fetchMock().mockResolvedValueOnce(ok({ book, chapters: [chapter({ status: 'failed', jobId: 7 }), chapter({ id: 56, status: 'failed', jobId: null })] }))
|
||||
|
||||
await library.loadBook(1)
|
||||
|
||||
expect(library.book?.title).toBe('虚构样例书')
|
||||
expect(library.chapters).toHaveLength(2)
|
||||
expect(paths()[1]).toBe('/api/v1/books/1')
|
||||
// The chapter itself carries the job id, even for a freshly loaded book.
|
||||
expect(library.chapters[0]?.jobId).toBe(7)
|
||||
expect(canRetry(library.chapters[0]!)).toBe(true)
|
||||
// A null job id means the chapter has nothing to retry yet.
|
||||
expect(canRetry(library.chapters[1]!)).toBe(false)
|
||||
})
|
||||
|
||||
it('retries a failed chapter loaded fresh from the book detail, without any submit in this session', async () => {
|
||||
vi.useFakeTimers()
|
||||
await signIn()
|
||||
const library = useLibraryStore()
|
||||
// No submit() call: this is a plain reload, the old workaround would hide retry here.
|
||||
fetchMock().mockResolvedValueOnce(ok({ book, chapters: [chapter({ status: 'failed', errorMessage: '无法解析正文。', jobId: 7 })] }))
|
||||
await library.loadBook(1)
|
||||
|
||||
fetchMock()
|
||||
.mockResolvedValueOnce(ok({ job: { ...job, status: 'pending', attempts: 1 }, chapter: { id: 55, bookId: 1, jobId: 7 } }))
|
||||
.mockResolvedValueOnce(ok({ book, chapters: [chapter({ jobId: 7 })] }))
|
||||
await library.retryChapter(55)
|
||||
|
||||
expect(paths()).toContain('/api/v1/jobs/7/retry')
|
||||
expect(library.retryingChapterId).toBeNull()
|
||||
expect(library.chapters[0]?.status).toBe('pending')
|
||||
})
|
||||
|
||||
it('pastes a new book and reads the job id from the created chapter', async () => {
|
||||
await signIn()
|
||||
const library = useLibraryStore()
|
||||
fetchMock().mockResolvedValueOnce(created({ book, chapter: chapter(), job, duplicate: false }))
|
||||
|
||||
const bookId = await library.submit({ title: ' 第一篇 ', text: 'Hello world.\nSecond line.', target: { mode: 'new' } })
|
||||
|
||||
expect(bookId).toBe(1)
|
||||
expect(library.submitting).toBe(false)
|
||||
expect(library.submitError).toBe('')
|
||||
expect(paths()[1]).toBe('/api/v1/books')
|
||||
expect(bodyOf(1)).toEqual({ requestId: expect.any(String), title: '第一篇', text: 'Hello world.\nSecond line.', language: 'en' })
|
||||
})
|
||||
|
||||
it('appends to an existing book through the chapter endpoint', async () => {
|
||||
await signIn()
|
||||
const library = useLibraryStore()
|
||||
fetchMock().mockResolvedValueOnce(created({ chapter: chapter(), job, duplicate: false }))
|
||||
|
||||
const bookId = await library.submit({ title: '第二篇', text: 'Another text.', target: { mode: 'append', bookId: 1 } })
|
||||
|
||||
expect(bookId).toBe(1)
|
||||
expect(paths()[1]).toBe('/api/v1/books/1/chapters')
|
||||
})
|
||||
|
||||
it('reuses one requestId while the same unsent content keeps failing', async () => {
|
||||
await signIn()
|
||||
const library = useLibraryStore()
|
||||
const input = { title: '第一篇', text: 'Hello world.', target: { mode: 'new' as const } }
|
||||
fetchMock()
|
||||
.mockResolvedValueOnce(httpError(500, '服务器开小差了'))
|
||||
.mockResolvedValueOnce(created({ book, chapter: chapter(), job, duplicate: false }))
|
||||
|
||||
await expect(library.submit(input)).rejects.toThrow('服务器开小差了')
|
||||
expect(library.submitError).toBe('服务器开小差了')
|
||||
await library.submit(input)
|
||||
|
||||
// One chapter, not two: the retry of unchanged content reuses the requestId.
|
||||
expect(bodyOf(2).requestId).toBe(bodyOf(1).requestId)
|
||||
})
|
||||
|
||||
it('uses a fresh requestId after a successful submit and after the content changes', async () => {
|
||||
await signIn()
|
||||
const library = useLibraryStore()
|
||||
fetchMock()
|
||||
.mockResolvedValueOnce(created({ book, chapter: chapter(), job, duplicate: false }))
|
||||
.mockResolvedValueOnce(created({ book, chapter: chapter({ id: 56 }), job, duplicate: false }))
|
||||
.mockResolvedValueOnce(created({ book, chapter: chapter({ id: 57 }), job, duplicate: false }))
|
||||
|
||||
await library.submit({ title: '第一篇', text: 'Hello world.', target: { mode: 'new' } })
|
||||
await library.submit({ title: '第一篇', text: 'Hello world.', target: { mode: 'new' } })
|
||||
await library.submit({ title: '第一篇', text: 'Hello world changed.', target: { mode: 'new' } })
|
||||
|
||||
expect(bodyOf(2).requestId).not.toBe(bodyOf(1).requestId)
|
||||
expect(bodyOf(3).requestId).not.toBe(bodyOf(2).requestId)
|
||||
})
|
||||
|
||||
it('rejects invalid input before sending anything', async () => {
|
||||
await signIn()
|
||||
const library = useLibraryStore()
|
||||
const sent = paths().length
|
||||
|
||||
await expect(library.submit({ title: ' ', text: 'Hello.', target: { mode: 'new' } })).rejects.toThrow('请填写标题。')
|
||||
await expect(library.submit({ title: '标题', text: ' \n\t ', target: { mode: 'new' } })).rejects.toThrow('请粘贴要导入的英文正文。')
|
||||
await expect(library.submit({ title: 'x'.repeat(121), text: 'Hello.', target: { mode: 'new' } })).rejects.toThrow('标题不能超过 120 个字符。')
|
||||
await expect(library.submit({ title: '标题', text: 'a'.repeat(TEXT_MAX_CODE_POINTS + 1), target: { mode: 'new' } })).rejects.toThrow(`正文不能超过 ${TEXT_MAX_CODE_POINTS} 个字符。`)
|
||||
|
||||
expect(paths().length).toBe(sent)
|
||||
expect(library.submitError).toBe(`正文不能超过 ${TEXT_MAX_CODE_POINTS} 个字符。`)
|
||||
})
|
||||
|
||||
it('measures the text limit in Unicode code points', async () => {
|
||||
await signIn()
|
||||
const library = useLibraryStore()
|
||||
// 100000 astral characters are 200000 UTF-16 units but still within the limit.
|
||||
fetchMock().mockResolvedValueOnce(created({ book, chapter: chapter(), job, duplicate: false }))
|
||||
|
||||
await expect(library.submit({ title: '标题', text: '😀'.repeat(TEXT_MAX_CODE_POINTS), target: { mode: 'new' } })).resolves.toBe(1)
|
||||
expect([...('😀'.repeat(TEXT_MAX_CODE_POINTS))].length).toBe(TEXT_MAX_CODE_POINTS)
|
||||
})
|
||||
|
||||
it('polls a pending chapter until it is ready and then stops', async () => {
|
||||
vi.useFakeTimers()
|
||||
await signIn()
|
||||
const library = useLibraryStore()
|
||||
fetchMock().mockResolvedValueOnce(ok({ book, chapter: chapter(), navigation }))
|
||||
|
||||
await library.loadChapter(55)
|
||||
expect(library.chapter?.status).toBe('pending')
|
||||
expect(library.readerText).toBe('')
|
||||
|
||||
fetchMock().mockResolvedValueOnce(ok({ book, chapter: chapter({ status: 'ready', originalText: 'Hello\nworld.' }), navigation }))
|
||||
await vi.advanceTimersByTimeAsync(POLL_INTERVAL_MS)
|
||||
|
||||
expect(library.chapter?.status).toBe('ready')
|
||||
expect(library.readerText).toBe('Hello\nworld.')
|
||||
|
||||
const settled = paths().length
|
||||
await vi.advanceTimersByTimeAsync(POLL_INTERVAL_MS * 3)
|
||||
expect(paths().length).toBe(settled)
|
||||
})
|
||||
|
||||
it('retries a failed chapter through the chapter job id and resumes polling', async () => {
|
||||
vi.useFakeTimers()
|
||||
await signIn()
|
||||
const library = useLibraryStore()
|
||||
fetchMock().mockResolvedValueOnce(ok({ book, chapter: chapter({ status: 'failed', errorReason: 'decode_error', errorMessage: '无法解析正文。', jobId: 7 }), navigation }))
|
||||
|
||||
await library.loadChapter(55)
|
||||
expect(library.chapter?.status).toBe('failed')
|
||||
expect(canRetry(library.chapter!)).toBe(true)
|
||||
|
||||
fetchMock()
|
||||
.mockResolvedValueOnce(ok({ job: { ...job, status: 'pending', attempts: 1 }, chapter: { id: 55, bookId: 1, jobId: 7 } }))
|
||||
.mockResolvedValueOnce(ok({ book, chapter: chapter(), navigation }))
|
||||
await library.retryChapter(55)
|
||||
|
||||
expect(paths()).toContain('/api/v1/jobs/7/retry')
|
||||
expect(library.retryingChapterId).toBeNull()
|
||||
expect(library.chapter?.status).toBe('pending')
|
||||
|
||||
// The retry restarts polling for the chapter it re-queued.
|
||||
const before = paths().length
|
||||
fetchMock().mockResolvedValueOnce(ok({ book, chapter: chapter({ status: 'ready', originalText: 'Hello world.' }), navigation }))
|
||||
await vi.advanceTimersByTimeAsync(POLL_INTERVAL_MS)
|
||||
expect(paths().length).toBe(before + 1)
|
||||
expect(library.readerText).toBe('Hello world.')
|
||||
})
|
||||
|
||||
it('refuses to retry a chapter whose job id is null', async () => {
|
||||
await signIn()
|
||||
const library = useLibraryStore()
|
||||
fetchMock().mockResolvedValueOnce(ok({ book, chapter: chapter({ status: 'failed', jobId: null }), navigation }))
|
||||
await library.loadChapter(55)
|
||||
|
||||
expect(canRetry(library.chapter!)).toBe(false)
|
||||
await expect(library.retryChapter(55)).rejects.toThrow('这一章暂时没有可重试的任务编号。')
|
||||
expect(paths()).toEqual(['/api/v1/login', '/api/v1/chapters/55'])
|
||||
})
|
||||
|
||||
it('reports another account id as 内容不存在 and stops polling for it', async () => {
|
||||
vi.useFakeTimers()
|
||||
await signIn()
|
||||
const library = useLibraryStore()
|
||||
fetchMock().mockResolvedValueOnce(httpError(404, 'chapter not found'))
|
||||
|
||||
await library.loadChapter(99)
|
||||
expect(library.chapter).toBeNull()
|
||||
expect(library.chapterError).toBe(NOT_FOUND_MESSAGE)
|
||||
|
||||
fetchMock().mockResolvedValueOnce(httpError(404, 'book not found'))
|
||||
await library.loadBook(99)
|
||||
expect(library.book).toBeNull()
|
||||
expect(library.bookError).toBe(NOT_FOUND_MESSAGE)
|
||||
|
||||
const settled = paths().length
|
||||
await vi.advanceTimersByTimeAsync(POLL_INTERVAL_MS * 4)
|
||||
expect(paths().length).toBe(settled)
|
||||
})
|
||||
|
||||
it('stops polling and ignores a late response once the session is cleared', async () => {
|
||||
vi.useFakeTimers()
|
||||
const session = await signIn()
|
||||
const library = useLibraryStore()
|
||||
fetchMock().mockResolvedValueOnce(ok({ book, chapter: chapter(), navigation }))
|
||||
await library.loadChapter(55)
|
||||
expect(library.chapter).not.toBeNull()
|
||||
|
||||
let finish!: (response: Response) => void
|
||||
fetchMock().mockImplementationOnce(() => new Promise<Response>(resolve => { finish = resolve }))
|
||||
fetchMock().mockResolvedValueOnce(ok(null))
|
||||
const late = library.loadChapter(55, { silent: true })
|
||||
const logout = session.logout()
|
||||
|
||||
finish(ok({ book, chapter: chapter({ status: 'ready', originalText: 'Late text.' }), navigation }))
|
||||
await late
|
||||
await logout
|
||||
|
||||
expect(session.user).toBeNull()
|
||||
expect(library.chapter).toBeNull()
|
||||
expect(library.readerText).toBe('')
|
||||
|
||||
const settled = paths().length
|
||||
await vi.advanceTimersByTimeAsync(POLL_INTERVAL_MS * 4)
|
||||
expect(paths().length).toBe(settled)
|
||||
})
|
||||
|
||||
it('keeps newer state when an older request answers later', async () => {
|
||||
await signIn()
|
||||
const library = useLibraryStore()
|
||||
let finish!: (response: Response) => void
|
||||
fetchMock().mockImplementationOnce(() => new Promise<Response>(resolve => { finish = resolve }))
|
||||
fetchMock().mockResolvedValueOnce(ok({ items: [summary({ title: '较新的标题' })] }))
|
||||
|
||||
const stale = library.loadBooks()
|
||||
await library.loadBooks()
|
||||
finish(ok({ items: [summary({ title: '过期的标题' })] }))
|
||||
await stale
|
||||
|
||||
expect(library.books).toHaveLength(1)
|
||||
expect(library.books[0]?.title).toBe('较新的标题')
|
||||
})
|
||||
|
||||
// Regression R1: the append contract has no language field and the server rejects unknown
|
||||
// fields, so a client that sent one could never append.
|
||||
it('sends the language only when creating a book, never when appending', async () => {
|
||||
await signIn()
|
||||
const library = useLibraryStore()
|
||||
fetchMock().mockResolvedValueOnce(created({ book, chapter: chapter({ id: 54 }) }))
|
||||
await library.submit({ title: ' 新书 ', text: 'New book text.', target: { mode: 'new' } })
|
||||
expect(bodyOf(1)).toEqual({ requestId: expect.any(String), title: '新书', text: 'New book text.', language: 'en' })
|
||||
|
||||
fetchMock().mockResolvedValueOnce(created({ chapter: chapter({ id: 55, ordinal: 2 }) }))
|
||||
await library.submit({ title: '第二篇', text: 'Appended text.', target: { mode: 'append', bookId: 1 } })
|
||||
expect(String(fetchMock().mock.calls[2]?.[0])).toBe('/api/v1/books/1/chapters')
|
||||
expect(bodyOf(2)).toEqual({ requestId: expect.any(String), title: '第二篇', text: 'Appended text.' })
|
||||
expect(bodyOf(2)).not.toHaveProperty('language')
|
||||
})
|
||||
|
||||
// Regression R3: leaving a view must invalidate its in-flight request.
|
||||
it('ignores a book response that arrives after the book view was closed', async () => {
|
||||
vi.useFakeTimers()
|
||||
await signIn()
|
||||
const library = useLibraryStore()
|
||||
let finish!: (response: Response) => void
|
||||
fetchMock().mockImplementationOnce(() => new Promise<Response>(resolve => { finish = resolve }))
|
||||
|
||||
const pending = library.loadBook(1)
|
||||
library.closeBook()
|
||||
finish(ok({ book, chapters: [chapter()] }))
|
||||
await pending
|
||||
|
||||
expect(library.book).toBeNull()
|
||||
expect(library.chapters).toEqual([])
|
||||
expect(library.bookLoading).toBe(false)
|
||||
const settled = paths().length
|
||||
await vi.advanceTimersByTimeAsync(POLL_INTERVAL_MS * 3)
|
||||
expect(paths().length).toBe(settled)
|
||||
})
|
||||
|
||||
it('ignores a chapter response that arrives after the reader was closed', async () => {
|
||||
vi.useFakeTimers()
|
||||
await signIn()
|
||||
const library = useLibraryStore()
|
||||
let finish!: (response: Response) => void
|
||||
fetchMock().mockImplementationOnce(() => new Promise<Response>(resolve => { finish = resolve }))
|
||||
|
||||
const pending = library.loadChapter(55)
|
||||
library.closeChapter()
|
||||
finish(ok({ book, chapter: chapter({ status: 'ready', originalText: 'Late text.' }), navigation }))
|
||||
await pending
|
||||
|
||||
expect(library.chapter).toBeNull()
|
||||
expect(library.readerText).toBe('')
|
||||
expect(library.chapterLoading).toBe(false)
|
||||
const settled = paths().length
|
||||
await vi.advanceTimersByTimeAsync(POLL_INTERVAL_MS * 3)
|
||||
expect(paths().length).toBe(settled)
|
||||
})
|
||||
|
||||
// Regression R4: an accepted retry must be visible and tracked even if the refresh fails.
|
||||
it('keeps tracking a retried chapter when the first refresh fails', async () => {
|
||||
vi.useFakeTimers()
|
||||
await signIn()
|
||||
const library = useLibraryStore()
|
||||
fetchMock().mockResolvedValueOnce(ok({ book, chapter: chapter({ status: 'failed', errorReason: 'content_changed', errorMessage: '内容在处理前发生变化。', jobId: 7 }), navigation }))
|
||||
await library.loadChapter(55)
|
||||
expect(library.chapter?.status).toBe('failed')
|
||||
|
||||
fetchMock()
|
||||
.mockResolvedValueOnce(ok({ job: { ...job, status: 'pending' }, chapter: chapter({ status: 'pending' }) }))
|
||||
.mockRejectedValueOnce(new Error('network down'))
|
||||
await library.retryChapter(55)
|
||||
|
||||
expect(paths()).toContain('/api/v1/jobs/7/retry')
|
||||
expect(library.chapter?.status).toBe('pending')
|
||||
expect(library.readerText).toBe('')
|
||||
|
||||
// The next poll still tracks the queued chapter and shows the final result.
|
||||
fetchMock().mockResolvedValueOnce(ok({ book, chapter: chapter({ status: 'ready', originalText: 'Recovered text.' }), navigation }))
|
||||
await vi.advanceTimersByTimeAsync(POLL_INTERVAL_MS)
|
||||
expect(library.chapter?.status).toBe('ready')
|
||||
expect(library.readerText).toBe('Recovered text.')
|
||||
})
|
||||
|
||||
it('keeps tracking a retried chapter from the book page when the first refresh fails', async () => {
|
||||
vi.useFakeTimers()
|
||||
await signIn()
|
||||
const library = useLibraryStore()
|
||||
fetchMock().mockResolvedValueOnce(ok({ book, chapters: [chapter({ status: 'failed', errorReason: 'content_changed', errorMessage: '内容在处理前发生变化。' })] }))
|
||||
await library.loadBook(1)
|
||||
expect(library.chapters[0]?.status).toBe('failed')
|
||||
|
||||
fetchMock()
|
||||
.mockResolvedValueOnce(ok({ job: { ...job, status: 'pending' }, chapter: chapter({ status: 'pending' }) }))
|
||||
.mockRejectedValueOnce(new Error('network down'))
|
||||
await library.retryChapter(55)
|
||||
|
||||
expect(library.chapters[0]?.status).toBe('pending')
|
||||
|
||||
fetchMock().mockResolvedValueOnce(ok({ book, chapters: [chapter({ status: 'ready' })] }))
|
||||
await vi.advanceTimersByTimeAsync(POLL_INTERVAL_MS)
|
||||
expect(library.chapters[0]?.status).toBe('ready')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,265 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import { flushPromises, mount } from '@vue/test-utils'
|
||||
import { createPinia, setActivePinia } from 'pinia'
|
||||
import { createMemoryHistory, createRouter, type Router } from 'vue-router'
|
||||
import { defineComponent, h } from 'vue'
|
||||
import BookView from '../views/BookView.vue'
|
||||
import ImportView from '../views/ImportView.vue'
|
||||
import ReaderView from '../views/ReaderView.vue'
|
||||
import { useLibraryStore, type ChapterStatus } from '../stores/library'
|
||||
import { useSessionStore } from '../stores/session'
|
||||
|
||||
// All accounts, books and texts in these tests are deliberately fictitious.
|
||||
const user = { id: 42, username: 'fictional-reader', role: 'learner' as const }
|
||||
const book = { id: 1, title: '虚构样例书', language: 'en' }
|
||||
const navigation = { previousChapterId: null, nextChapterId: null }
|
||||
const pasted = 'First line.\n\tIndented line.\nTwo spaces kept.\n\nLast line.\n'
|
||||
|
||||
const ok = (data: unknown) => new Response(JSON.stringify({ code: 200, data }), { status: 200 })
|
||||
const created = (data: unknown) => new Response(JSON.stringify({ code: 200, data }), { status: 201 })
|
||||
|
||||
function chapter(status: ChapterStatus, extra: Record<string, unknown> = {}) {
|
||||
return {
|
||||
id: 55,
|
||||
bookId: 1,
|
||||
ordinal: 1,
|
||||
title: '第一篇',
|
||||
status,
|
||||
charCount: 120,
|
||||
errorReason: '',
|
||||
errorMessage: '',
|
||||
jobId: 7,
|
||||
createdAt: '2026-01-01T00:00:00Z',
|
||||
updatedAt: '2026-01-01T00:00:00Z',
|
||||
...extra,
|
||||
}
|
||||
}
|
||||
|
||||
const stub = (name: string) => defineComponent({ name, render: () => h('div') })
|
||||
|
||||
async function viewAt(path: string): Promise<Router> {
|
||||
const router = createRouter({
|
||||
history: createMemoryHistory(),
|
||||
routes: [
|
||||
{ path: '/', component: stub('LibraryStub') },
|
||||
{ path: '/import', component: stub('ImportStub') },
|
||||
{ path: '/books/:id', component: stub('BookStub') },
|
||||
{ path: '/chapters/:id', component: stub('ChapterStub') },
|
||||
],
|
||||
})
|
||||
await router.push(path)
|
||||
await router.isReady()
|
||||
return router
|
||||
}
|
||||
|
||||
function signIn() {
|
||||
useSessionStore().user = { ...user }
|
||||
}
|
||||
|
||||
describe('learner reading views', () => {
|
||||
beforeEach(() => {
|
||||
sessionStorage.clear()
|
||||
setActivePinia(createPinia())
|
||||
vi.restoreAllMocks()
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
useLibraryStore().stopPolling()
|
||||
})
|
||||
|
||||
it('shows inline validation and sends nothing for an empty import form', async () => {
|
||||
const fetchMock = vi.spyOn(globalThis, 'fetch')
|
||||
signIn()
|
||||
const router = await viewAt('/import')
|
||||
const wrapper = mount(ImportView, { global: { plugins: [router] } })
|
||||
await flushPromises()
|
||||
|
||||
await wrapper.find('form').trigger('submit')
|
||||
expect(wrapper.text()).toContain('请填写标题。')
|
||||
expect(wrapper.text()).toContain('请粘贴要导入的英文正文。')
|
||||
expect(fetchMock).not.toHaveBeenCalled()
|
||||
|
||||
await wrapper.find('input#title').setValue(' ')
|
||||
await wrapper.find('textarea#text').setValue(' \n\t ')
|
||||
await wrapper.find('form').trigger('submit')
|
||||
expect(wrapper.text()).toContain('请填写标题。')
|
||||
expect(wrapper.text()).toContain('请粘贴要导入的英文正文。')
|
||||
|
||||
await wrapper.find('input#title').setValue('虚构样例第一章')
|
||||
await wrapper.find('form').trigger('submit')
|
||||
expect(wrapper.text()).not.toContain('请填写标题。')
|
||||
expect(wrapper.text()).toContain('请粘贴要导入的英文正文。')
|
||||
expect(fetchMock).not.toHaveBeenCalled()
|
||||
|
||||
wrapper.unmount()
|
||||
})
|
||||
|
||||
it('submits valid pasted text and opens the created book', async () => {
|
||||
const fetchMock = vi.spyOn(globalThis, 'fetch')
|
||||
.mockResolvedValueOnce(created({ book, chapter: chapter('pending'), job: { id: 7 }, duplicate: false }))
|
||||
signIn()
|
||||
const router = await viewAt('/import')
|
||||
const wrapper = mount(ImportView, { global: { plugins: [router] } })
|
||||
// Element Plus assigns the input ids on mount, so wait for the first update.
|
||||
await flushPromises()
|
||||
|
||||
await wrapper.find('input#title').setValue('虚构样例第一章')
|
||||
await wrapper.find('textarea#text').setValue(pasted)
|
||||
await wrapper.find('form').trigger('submit')
|
||||
await flushPromises()
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledTimes(1)
|
||||
expect(JSON.parse(String(fetchMock.mock.calls[0]?.[1]?.body))).toMatchObject({ title: '虚构样例第一章', text: pasted, language: 'en' })
|
||||
expect(router.currentRoute.value.path).toBe('/books/1')
|
||||
wrapper.unmount()
|
||||
})
|
||||
|
||||
it('renders a ready chapter verbatim, keeping line breaks, tabs and repeated spaces', async () => {
|
||||
vi.spyOn(globalThis, 'fetch').mockResolvedValue(ok({ book, chapter: chapter('ready', { originalText: pasted, contentSha256: 'fictional' }), navigation }))
|
||||
signIn()
|
||||
const router = await viewAt('/chapters/55')
|
||||
const wrapper = mount(ReaderView, { global: { plugins: [router] } })
|
||||
await flushPromises()
|
||||
|
||||
const article = wrapper.find('.reader-text')
|
||||
expect(article.exists()).toBe(true)
|
||||
expect(article.element.textContent).toBe(pasted)
|
||||
expect(wrapper.text()).toContain('已就绪')
|
||||
wrapper.unmount()
|
||||
})
|
||||
|
||||
it('shows the failure message of a failed chapter and no text', async () => {
|
||||
vi.spyOn(globalThis, 'fetch').mockResolvedValue(ok({ book, chapter: chapter('failed', { errorReason: 'decode_error', errorMessage: '无法解析正文，请检查编码。', jobId: null }), navigation }))
|
||||
signIn()
|
||||
const router = await viewAt('/chapters/55')
|
||||
const wrapper = mount(ReaderView, { global: { plugins: [router] } })
|
||||
await flushPromises()
|
||||
|
||||
expect(wrapper.text()).toContain('处理失败')
|
||||
expect(wrapper.text()).toContain('无法解析正文，请检查编码。')
|
||||
expect(wrapper.find('.reader-text').exists()).toBe(false)
|
||||
// A null job id means there is nothing to retry yet.
|
||||
expect(wrapper.text()).toContain('这一章暂时没有可重试的任务编号。')
|
||||
wrapper.unmount()
|
||||
})
|
||||
|
||||
it('offers retry on a failed chapter through its own job id', async () => {
|
||||
const fetchMock = vi.spyOn(globalThis, 'fetch').mockResolvedValue(ok({ book, chapter: chapter('failed', { errorMessage: '解析失败。', jobId: 7 }), navigation }))
|
||||
signIn()
|
||||
const router = await viewAt('/chapters/55')
|
||||
const wrapper = mount(ReaderView, { global: { plugins: [router] } })
|
||||
await flushPromises()
|
||||
|
||||
expect(wrapper.text()).toContain('解析失败。')
|
||||
const retryButton = wrapper.get('.notice .el-button')
|
||||
expect(retryButton.text()).toContain('重试处理')
|
||||
await retryButton.trigger('click')
|
||||
await flushPromises()
|
||||
|
||||
// No prior submit in this session: the job id comes from the chapter payload.
|
||||
const retryCall = fetchMock.mock.calls.find(([input]) => String(input).endsWith('/jobs/7/retry'))
|
||||
expect(retryCall?.[1]?.method).toBe('POST')
|
||||
wrapper.unmount()
|
||||
})
|
||||
|
||||
it('lists chapters with their status labels and links only ready chapters', async () => {
|
||||
const chapters = [
|
||||
chapter('pending', { id: 55, ordinal: 1, title: '第一篇', jobId: 7 }),
|
||||
chapter('processing', { id: 56, ordinal: 2, title: '第二篇', jobId: 8 }),
|
||||
chapter('ready', { id: 57, ordinal: 3, title: '第三篇', jobId: 9 }),
|
||||
chapter('failed', { id: 58, ordinal: 4, title: '第四篇', errorMessage: '解析失败。', jobId: null }),
|
||||
chapter('failed', { id: 59, ordinal: 5, title: '第五篇', errorMessage: '编码错误。', jobId: 10 }),
|
||||
]
|
||||
const fetchMock = vi.spyOn(globalThis, 'fetch').mockImplementation(input => {
|
||||
if (String(input).endsWith('/jobs/10/retry')) {
|
||||
return Promise.resolve(ok({ job: { id: 10, status: 'pending', attempts: 1 }, chapter: { id: 59, bookId: 1, jobId: 10 } }))
|
||||
}
|
||||
return Promise.resolve(ok({ book, chapters }))
|
||||
})
|
||||
signIn()
|
||||
const router = await viewAt('/books/1')
|
||||
const wrapper = mount(BookView, { global: { plugins: [router] } })
|
||||
await flushPromises()
|
||||
|
||||
const text = wrapper.text()
|
||||
expect(text).toContain('待处理')
|
||||
expect(text).toContain('处理中')
|
||||
expect(text).toContain('已就绪')
|
||||
expect(text).toContain('处理失败')
|
||||
expect(text).toContain('解析失败。')
|
||||
expect(text).toContain('编码错误。')
|
||||
expect(wrapper.find('a[href="/chapters/57"]').exists()).toBe(true)
|
||||
expect(wrapper.find('a[href="/chapters/55"]').exists()).toBe(false)
|
||||
|
||||
// Retry is offered only for the failed chapter that carries a job id.
|
||||
const retryButtons = wrapper.findAll('.chapter-row .el-button').filter(button => button.text().includes('重试'))
|
||||
expect(retryButtons).toHaveLength(1)
|
||||
await retryButtons[0]!.trigger('click')
|
||||
await flushPromises()
|
||||
const retryCall = fetchMock.mock.calls.find(([input]) => String(input).endsWith('/jobs/10/retry'))
|
||||
expect(retryCall?.[1]?.method).toBe('POST')
|
||||
wrapper.unmount()
|
||||
})
|
||||
|
||||
// Regression R1: appending must not send the language field that the append contract rejects.
|
||||
it('appends to an existing book through the chapter endpoint without a language field', async () => {
|
||||
const fetchMock = vi.spyOn(globalThis, 'fetch')
|
||||
.mockResolvedValueOnce(ok({
|
||||
items: [{
|
||||
...book,
|
||||
chapterCount: 1,
|
||||
pendingCount: 0,
|
||||
processingCount: 0,
|
||||
readyCount: 1,
|
||||
failedCount: 0,
|
||||
createdAt: '2026-01-01T00:00:00Z',
|
||||
updatedAt: '2026-01-01T00:00:00Z',
|
||||
}],
|
||||
}))
|
||||
.mockResolvedValueOnce(created({ chapter: chapter('pending', { id: 56, ordinal: 2 }), job: { id: 8 } }))
|
||||
signIn()
|
||||
const router = await viewAt('/import?book=1')
|
||||
const wrapper = mount(ImportView, { global: { plugins: [router] } })
|
||||
await flushPromises()
|
||||
|
||||
await wrapper.find('input#title').setValue('第二篇')
|
||||
await wrapper.find('textarea#text').setValue(pasted)
|
||||
await wrapper.find('form').trigger('submit')
|
||||
await flushPromises()
|
||||
|
||||
const appendCall = fetchMock.mock.calls.find(([input]) => String(input).endsWith('/books/1/chapters'))
|
||||
expect(appendCall?.[1]?.method).toBe('POST')
|
||||
const body = JSON.parse(String(appendCall?.[1]?.body)) as Record<string, unknown>
|
||||
expect(body).toEqual({ requestId: expect.any(String), title: '第二篇', text: pasted })
|
||||
expect(body).not.toHaveProperty('language')
|
||||
expect(router.currentRoute.value.path).toBe('/books/1')
|
||||
wrapper.unmount()
|
||||
})
|
||||
|
||||
// Regression R3: a submit that finishes after the user left the page must not navigate back.
|
||||
it('does not navigate after the user left the import page during a submit', async () => {
|
||||
let finish!: (response: Response) => void
|
||||
const fetchMock = vi.spyOn(globalThis, 'fetch')
|
||||
.mockImplementationOnce(() => new Promise<Response>(resolve => { finish = resolve }))
|
||||
signIn()
|
||||
const router = await viewAt('/import')
|
||||
const wrapper = mount(ImportView, { global: { plugins: [router] } })
|
||||
await flushPromises()
|
||||
|
||||
await wrapper.find('input#title').setValue('虚构样例第一章')
|
||||
await wrapper.find('textarea#text').setValue(pasted)
|
||||
await wrapper.find('form').trigger('submit')
|
||||
await flushPromises()
|
||||
expect(fetchMock).toHaveBeenCalledTimes(1)
|
||||
|
||||
// The user leaves the page while the request is still open.
|
||||
wrapper.unmount()
|
||||
await router.push('/')
|
||||
await flushPromises()
|
||||
|
||||
finish(created({ book, chapter: chapter('pending'), job: { id: 7 } }))
|
||||
await flushPromises()
|
||||
|
||||
expect(router.currentRoute.value.path).toBe('/')
|
||||
})
|
||||
})
|
||||
@@ -4,6 +4,10 @@ import App from './App.vue'
|
||||
import router from './router'
|
||||
import 'element-plus/es/components/button/style/css'
|
||||
import 'element-plus/es/components/input/style/css'
|
||||
import 'element-plus/es/components/radio/style/css'
|
||||
import 'element-plus/es/components/radio-group/style/css'
|
||||
import 'element-plus/es/components/select/style/css'
|
||||
import 'element-plus/es/components/option/style/css'
|
||||
import './style.css'
|
||||
|
||||
createApp(App).use(createPinia()).use(router).mount('#app')
|
||||
|
||||
@@ -6,6 +6,9 @@ const router = createRouter({
|
||||
routes: [
|
||||
{ path: '/login', name: 'login', component: () => import('../views/LoginView.vue') },
|
||||
{ path: '/', name: 'library', meta: { private: true }, component: () => import('../views/LibraryView.vue') },
|
||||
{ path: '/import', name: 'import', meta: { private: true }, component: () => import('../views/ImportView.vue') },
|
||||
{ path: '/books/:id', name: 'book', meta: { private: true }, component: () => import('../views/BookView.vue') },
|
||||
{ path: '/chapters/:id', name: 'chapter', meta: { private: true }, component: () => import('../views/ReaderView.vue') },
|
||||
{ path: '/:pathMatch(.*)*', redirect: '/' },
|
||||
],
|
||||
})
|
||||
|
||||
@@ -0,0 +1,467 @@
|
||||
import { defineStore } from 'pinia'
|
||||
import { computed, ref, watch } from 'vue'
|
||||
import { ApiError, useSessionStore } from './session'
|
||||
|
||||
export type ChapterStatus = 'pending' | 'processing' | 'ready' | 'failed'
|
||||
|
||||
export interface BookRef { id: number; title: string; language: string }
|
||||
|
||||
export interface BookSummary extends BookRef {
|
||||
chapterCount: number
|
||||
pendingCount: number
|
||||
processingCount: number
|
||||
readyCount: number
|
||||
failedCount: number
|
||||
createdAt: string
|
||||
updatedAt: string
|
||||
}
|
||||
|
||||
export interface ChapterSummary {
|
||||
id: number
|
||||
bookId: number
|
||||
ordinal: number
|
||||
title: string
|
||||
status: ChapterStatus
|
||||
charCount: number
|
||||
errorReason: string
|
||||
errorMessage: string
|
||||
// Present wherever a chapter appears; null while the job id is unknown.
|
||||
jobId: number | null
|
||||
createdAt: string
|
||||
updatedAt: string
|
||||
}
|
||||
|
||||
export interface ChapterDetail extends ChapterSummary {
|
||||
contentSha256: string
|
||||
// Present only for ready chapters; never cached or faked for other statuses.
|
||||
originalText?: string
|
||||
}
|
||||
|
||||
export interface Job {
|
||||
id: number
|
||||
bookId: number
|
||||
chapterId: number
|
||||
status: ChapterStatus
|
||||
attempts: number
|
||||
errorReason: string
|
||||
errorMessage: string
|
||||
createdAt: string
|
||||
updatedAt: string
|
||||
}
|
||||
|
||||
export interface ChapterNavigation { previousChapterId: number | null; nextChapterId: number | null }
|
||||
|
||||
export type SubmitTarget = { mode: 'new' } | { mode: 'append'; bookId: number }
|
||||
|
||||
export interface SubmitInput { title: string; text: string; target: SubmitTarget }
|
||||
|
||||
export const POLL_INTERVAL_MS = 1500
|
||||
export const TITLE_MAX_LENGTH = 120
|
||||
export const TEXT_MAX_CODE_POINTS = 100000
|
||||
export const NOT_FOUND_MESSAGE = '内容不存在。'
|
||||
export const LANGUAGE_LABEL = '英语'
|
||||
export const LANGUAGE_CODE = 'en'
|
||||
|
||||
const CHAPTER_STATUS_LABELS: Record<ChapterStatus, string> = {
|
||||
pending: '待处理',
|
||||
processing: '处理中',
|
||||
ready: '已就绪',
|
||||
failed: '处理失败',
|
||||
}
|
||||
|
||||
export function statusLabel(status: ChapterStatus): string {
|
||||
return CHAPTER_STATUS_LABELS[status]
|
||||
}
|
||||
|
||||
function isUnsettled(status: ChapterStatus): boolean {
|
||||
return status === 'pending' || status === 'processing'
|
||||
}
|
||||
|
||||
/** Mirrors the server rule: non-empty after trim and at most 120 characters. */
|
||||
export function titleProblem(title: string): string {
|
||||
const trimmed = title.trim()
|
||||
if (!trimmed) return '请填写标题。'
|
||||
if ([...trimmed].length > TITLE_MAX_LENGTH) return `标题不能超过 ${TITLE_MAX_LENGTH} 个字符。`
|
||||
return ''
|
||||
}
|
||||
|
||||
/** Mirrors the server rule: at least one non-whitespace character, at most 100000 code points. */
|
||||
export function textProblem(text: string): string {
|
||||
if (!text.trim()) return '请粘贴要导入的英文正文。'
|
||||
if ([...text].length > TEXT_MAX_CODE_POINTS) return `正文不能超过 ${TEXT_MAX_CODE_POINTS} 个字符。`
|
||||
return ''
|
||||
}
|
||||
|
||||
/** A failed chapter can be retried as soon as the API told us its job id. */
|
||||
export function canRetry(chapter: Pick<ChapterSummary, 'status' | 'jobId'>): boolean {
|
||||
return chapter.status === 'failed' && chapter.jobId !== null
|
||||
}
|
||||
|
||||
/** Compact one-line status summary for a book card, e.g. `已就绪 2 · 处理中 1 · 待处理 1 · 失败 1`. */
|
||||
export function statusSummary(book: BookSummary): string {
|
||||
const parts: string[] = []
|
||||
if (book.readyCount > 0) parts.push(`已就绪 ${book.readyCount}`)
|
||||
if (book.processingCount > 0) parts.push(`处理中 ${book.processingCount}`)
|
||||
if (book.pendingCount > 0) parts.push(`待处理 ${book.pendingCount}`)
|
||||
if (book.failedCount > 0) parts.push(`失败 ${book.failedCount}`)
|
||||
return parts.join(' · ')
|
||||
}
|
||||
|
||||
interface LoadOptions { silent?: boolean }
|
||||
interface SubmitBookBody { requestId: string; title: string; text: string; language: 'en' }
|
||||
// An appended chapter owns the language of its book, so the append contract has no language
|
||||
// field; the server rejects unknown fields, and a client that sends one gets HTTP 400.
|
||||
interface SubmitChapterBody { requestId: string; title: string; text: string }
|
||||
interface Created { bookId: number; chapter: ChapterSummary }
|
||||
|
||||
function emptyNavigation(): ChapterNavigation {
|
||||
return { previousChapterId: null, nextChapterId: null }
|
||||
}
|
||||
|
||||
export const useLibraryStore = defineStore('library', () => {
|
||||
const session = useSessionStore()
|
||||
|
||||
const books = ref<BookSummary[]>([])
|
||||
const booksLoading = ref(false)
|
||||
const booksError = ref('')
|
||||
|
||||
const book = ref<BookRef | null>(null)
|
||||
const chapters = ref<ChapterSummary[]>([])
|
||||
const bookLoading = ref(false)
|
||||
const bookError = ref('')
|
||||
|
||||
const chapter = ref<ChapterDetail | null>(null)
|
||||
const chapterBook = ref<BookRef | null>(null)
|
||||
const navigation = ref<ChapterNavigation>(emptyNavigation())
|
||||
const chapterLoading = ref(false)
|
||||
const chapterError = ref('')
|
||||
|
||||
const submitting = ref(false)
|
||||
const submitError = ref('')
|
||||
const retryingChapterId = ref<number | null>(null)
|
||||
|
||||
/** Reader text exists only for ready chapters and is never taken from a cache. */
|
||||
const readerText = computed(() => (chapter.value?.status === 'ready' ? chapter.value.originalText ?? '' : ''))
|
||||
|
||||
// Every request is tagged with a generation and the owning account so that a
|
||||
// late response can never repopulate the view after logout or an account switch.
|
||||
let generation = 0
|
||||
let booksSeq = 0
|
||||
let bookSeq = 0
|
||||
let chapterSeq = 0
|
||||
let pollTimer: number | undefined
|
||||
// One requestId per unsent form content: a double click or a repeat submit of
|
||||
// unchanged content must create one chapter, not two.
|
||||
let submissionKey = ''
|
||||
let submissionRequestId = ''
|
||||
|
||||
watch(() => session.user?.id ?? null, (next, previous) => {
|
||||
if (next !== previous) reset()
|
||||
}, { flush: 'sync' })
|
||||
|
||||
function ownerId(): number | null {
|
||||
return session.user?.id ?? null
|
||||
}
|
||||
|
||||
function isStale(version: number, owner: number | null): boolean {
|
||||
return version !== generation || ownerId() !== owner
|
||||
}
|
||||
|
||||
function failureMessage(reason: unknown, fallback: string): string {
|
||||
return reason instanceof Error && reason.message ? reason.message : fallback
|
||||
}
|
||||
|
||||
function isNotFound(reason: unknown): boolean {
|
||||
return reason instanceof ApiError && reason.status === 404
|
||||
}
|
||||
|
||||
function stopPolling(): void {
|
||||
if (pollTimer !== undefined) {
|
||||
window.clearTimeout(pollTimer)
|
||||
pollTimer = undefined
|
||||
}
|
||||
}
|
||||
|
||||
function needsPolling(): boolean {
|
||||
return (book.value !== null && chapters.value.some(item => isUnsettled(item.status)))
|
||||
|| (chapter.value !== null && isUnsettled(chapter.value.status))
|
||||
}
|
||||
|
||||
/** Schedules the next refresh, or stops polling when nothing is pending anymore. */
|
||||
function schedulePolling(): void {
|
||||
if (!needsPolling()) {
|
||||
stopPolling()
|
||||
return
|
||||
}
|
||||
if (pollTimer !== undefined || !session.user) return
|
||||
pollTimer = window.setTimeout(() => {
|
||||
pollTimer = undefined
|
||||
void poll()
|
||||
}, POLL_INTERVAL_MS)
|
||||
}
|
||||
|
||||
async function poll(): Promise<void> {
|
||||
if (!session.user) {
|
||||
stopPolling()
|
||||
return
|
||||
}
|
||||
const version = generation
|
||||
const owner = ownerId()
|
||||
const bookId = book.value !== null && chapters.value.some(item => isUnsettled(item.status)) ? book.value.id : null
|
||||
const chapterId = chapter.value !== null && isUnsettled(chapter.value.status) ? chapter.value.id : null
|
||||
if (bookId !== null) await loadBook(bookId, { silent: true })
|
||||
if (chapterId !== null) await loadChapter(chapterId, { silent: true })
|
||||
if (isStale(version, owner)) {
|
||||
stopPolling()
|
||||
return
|
||||
}
|
||||
schedulePolling()
|
||||
}
|
||||
|
||||
async function loadBooks(): Promise<void> {
|
||||
const version = generation
|
||||
const owner = ownerId()
|
||||
const seq = ++booksSeq
|
||||
booksLoading.value = true
|
||||
booksError.value = ''
|
||||
try {
|
||||
const result = await session.request<{ items?: BookSummary[] } | null>('books')
|
||||
if (seq !== booksSeq || isStale(version, owner)) return
|
||||
const items = result?.items
|
||||
books.value = Array.isArray(items) ? items : []
|
||||
} catch (reason) {
|
||||
if (seq !== booksSeq || isStale(version, owner)) return
|
||||
booksError.value = failureMessage(reason, '书库暂时无法加载，请稍后重试。')
|
||||
} finally {
|
||||
if (seq === booksSeq && !isStale(version, owner)) booksLoading.value = false
|
||||
}
|
||||
}
|
||||
|
||||
async function loadBook(id: number, options: LoadOptions = {}): Promise<void> {
|
||||
const version = generation
|
||||
const owner = ownerId()
|
||||
const seq = ++bookSeq
|
||||
if (!options.silent) {
|
||||
bookLoading.value = true
|
||||
bookError.value = ''
|
||||
}
|
||||
try {
|
||||
const result = await session.request<{ book: BookRef; chapters?: ChapterSummary[] }>(`books/${id}`)
|
||||
if (seq !== bookSeq || isStale(version, owner)) return
|
||||
book.value = result.book
|
||||
chapters.value = Array.isArray(result.chapters) ? result.chapters : []
|
||||
bookError.value = ''
|
||||
schedulePolling()
|
||||
} catch (reason) {
|
||||
if (seq !== bookSeq || isStale(version, owner)) return
|
||||
if (isNotFound(reason)) {
|
||||
// Another account's id never resolves for this caller: report it and stop
|
||||
// instead of polling a resource that will not appear.
|
||||
book.value = null
|
||||
chapters.value = []
|
||||
bookError.value = NOT_FOUND_MESSAGE
|
||||
stopPolling()
|
||||
return
|
||||
}
|
||||
// A failed background refresh keeps the data already on screen; the next
|
||||
// tick tries again and the user still sees the last known state.
|
||||
if (!options.silent) bookError.value = failureMessage(reason, '书籍暂时无法加载，请稍后重试。')
|
||||
} finally {
|
||||
if (seq === bookSeq && !isStale(version, owner)) bookLoading.value = false
|
||||
}
|
||||
}
|
||||
|
||||
async function loadChapter(id: number, options: LoadOptions = {}): Promise<void> {
|
||||
const version = generation
|
||||
const owner = ownerId()
|
||||
const seq = ++chapterSeq
|
||||
if (!options.silent) {
|
||||
chapterLoading.value = true
|
||||
chapterError.value = ''
|
||||
// Never keep the previous chapter's text under a new chapter id.
|
||||
if (chapter.value !== null && chapter.value.id !== id) {
|
||||
chapter.value = null
|
||||
chapterBook.value = null
|
||||
navigation.value = emptyNavigation()
|
||||
}
|
||||
}
|
||||
try {
|
||||
const result = await session.request<{ book: BookRef; chapter: ChapterDetail; navigation?: ChapterNavigation }>(`chapters/${id}`)
|
||||
if (seq !== chapterSeq || isStale(version, owner)) return
|
||||
chapterBook.value = result.book
|
||||
chapter.value = result.chapter
|
||||
navigation.value = result.navigation ?? emptyNavigation()
|
||||
chapterError.value = ''
|
||||
schedulePolling()
|
||||
} catch (reason) {
|
||||
if (seq !== chapterSeq || isStale(version, owner)) return
|
||||
if (isNotFound(reason)) {
|
||||
chapter.value = null
|
||||
chapterBook.value = null
|
||||
navigation.value = emptyNavigation()
|
||||
chapterError.value = NOT_FOUND_MESSAGE
|
||||
stopPolling()
|
||||
return
|
||||
}
|
||||
if (!options.silent) chapterError.value = failureMessage(reason, '章节暂时无法加载，请稍后重试。')
|
||||
} finally {
|
||||
if (seq === chapterSeq && !isStale(version, owner)) chapterLoading.value = false
|
||||
}
|
||||
}
|
||||
|
||||
function submissionKeyOf(target: SubmitTarget, title: string, text: string): string {
|
||||
return target.mode === 'new' ? `new\n${title}\n${text}` : `append:${target.bookId}\n${title}\n${text}`
|
||||
}
|
||||
|
||||
async function createBook(body: SubmitBookBody): Promise<Created> {
|
||||
const result = await session.request<{ book: BookRef; chapter: ChapterSummary }>('books', 'POST', body)
|
||||
return { bookId: result.book.id, chapter: result.chapter }
|
||||
}
|
||||
|
||||
async function appendChapter(bookId: number, body: SubmitChapterBody): Promise<Created> {
|
||||
const result = await session.request<{ chapter: ChapterSummary }>(`books/${bookId}/chapters`, 'POST', body)
|
||||
return { bookId: result.chapter.bookId, chapter: result.chapter }
|
||||
}
|
||||
|
||||
/**
|
||||
* Submits pasted text. Returns the book id to open on success and throws on
|
||||
* failure; `submitError` always carries the message shown to the user.
|
||||
*/
|
||||
async function submit(input: SubmitInput): Promise<number> {
|
||||
const title = input.title.trim()
|
||||
const text = input.text
|
||||
const problem = titleProblem(title) || textProblem(text)
|
||||
if (problem) {
|
||||
submitError.value = problem
|
||||
throw new Error(problem)
|
||||
}
|
||||
|
||||
const key = submissionKeyOf(input.target, title, text)
|
||||
if (key !== submissionKey || submissionRequestId === '') {
|
||||
submissionKey = key
|
||||
submissionRequestId = crypto.randomUUID()
|
||||
}
|
||||
const requestId = submissionRequestId
|
||||
|
||||
const version = generation
|
||||
const owner = ownerId()
|
||||
submitting.value = true
|
||||
submitError.value = ''
|
||||
try {
|
||||
// Only the new-book contract carries a language; appending inherits the book's language.
|
||||
const created = input.target.mode === 'new'
|
||||
? await createBook({ requestId, title, text, language: LANGUAGE_CODE })
|
||||
: await appendChapter(input.target.bookId, { requestId, title, text })
|
||||
if (isStale(version, owner)) throw new Error('登录状态已变化，请重新提交。')
|
||||
// The content was accepted; a later submit must use a fresh requestId.
|
||||
submissionKey = ''
|
||||
submissionRequestId = ''
|
||||
return created.bookId
|
||||
} catch (reason) {
|
||||
if (!isStale(version, owner)) submitError.value = failureMessage(reason, '提交失败，请稍后重试。')
|
||||
throw reason instanceof Error ? reason : new Error('提交失败，请稍后重试。')
|
||||
} finally {
|
||||
if (!isStale(version, owner)) submitting.value = false
|
||||
}
|
||||
}
|
||||
|
||||
/** The job id comes from the chapter itself, wherever that chapter was loaded from. */
|
||||
function jobIdOf(chapterId: number): number | null {
|
||||
const target = chapters.value.find(item => item.id === chapterId)
|
||||
?? (chapter.value?.id === chapterId ? chapter.value : null)
|
||||
return target?.jobId ?? null
|
||||
}
|
||||
|
||||
/**
|
||||
* Applies a chapter summary coming from any response to the chapter list entry and to the
|
||||
* open reader, so a queued chapter is never displayed with the state or text it had before.
|
||||
*/
|
||||
function applyChapterSummary(summary: ChapterSummary): void {
|
||||
const index = chapters.value.findIndex(item => item.id === summary.id)
|
||||
if (index >= 0) chapters.value[index] = { ...chapters.value[index], ...summary }
|
||||
if (chapter.value !== null && chapter.value.id === summary.id) {
|
||||
const originalText = summary.status === 'ready' ? chapter.value.originalText : undefined
|
||||
chapter.value = { ...chapter.value, ...summary, originalText }
|
||||
}
|
||||
}
|
||||
|
||||
async function retryChapter(chapterId: number): Promise<void> {
|
||||
const jobId = jobIdOf(chapterId)
|
||||
if (jobId === null) throw new Error('这一章暂时没有可重试的任务编号。')
|
||||
const version = generation
|
||||
const owner = ownerId()
|
||||
retryingChapterId.value = chapterId
|
||||
try {
|
||||
const result = await session.request<{ job: Job; chapter: ChapterSummary }>(`jobs/${jobId}/retry`, 'POST')
|
||||
if (isStale(version, owner)) return
|
||||
// The retry is accepted, so show the queued chapter and keep tracking it even if the
|
||||
// refresh below fails: a failed silent refresh must not freeze the view on the old error.
|
||||
applyChapterSummary(result.chapter)
|
||||
schedulePolling()
|
||||
// Refresh whatever is on screen to pick up the newest job state.
|
||||
if (book.value !== null && book.value.id === result.chapter.bookId) await loadBook(result.chapter.bookId, { silent: true })
|
||||
if (chapter.value !== null && chapter.value.id === result.chapter.id) await loadChapter(result.chapter.id, { silent: true })
|
||||
} catch (reason) {
|
||||
if (isStale(version, owner)) return
|
||||
throw reason instanceof Error ? reason : new Error('重试失败，请稍后重试。')
|
||||
} finally {
|
||||
if (!isStale(version, owner)) retryingChapterId.value = null
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Releases the book view so polling stops when the page is left. The request sequence is
|
||||
* advanced first, so a response that arrives after this call cannot repopulate the view or
|
||||
* restart polling for a page the user already left.
|
||||
*/
|
||||
function closeBook(): void {
|
||||
bookSeq++
|
||||
book.value = null
|
||||
chapters.value = []
|
||||
bookLoading.value = false
|
||||
bookError.value = ''
|
||||
schedulePolling()
|
||||
}
|
||||
|
||||
/** Releases the reader view, invalidating in-flight loads the same way. */
|
||||
function closeChapter(): void {
|
||||
chapterSeq++
|
||||
chapter.value = null
|
||||
chapterBook.value = null
|
||||
navigation.value = emptyNavigation()
|
||||
chapterLoading.value = false
|
||||
chapterError.value = ''
|
||||
schedulePolling()
|
||||
}
|
||||
|
||||
function reset(): void {
|
||||
generation++
|
||||
stopPolling()
|
||||
books.value = []
|
||||
booksLoading.value = false
|
||||
booksError.value = ''
|
||||
book.value = null
|
||||
chapters.value = []
|
||||
bookLoading.value = false
|
||||
bookError.value = ''
|
||||
chapter.value = null
|
||||
chapterBook.value = null
|
||||
navigation.value = emptyNavigation()
|
||||
chapterLoading.value = false
|
||||
chapterError.value = ''
|
||||
submitting.value = false
|
||||
submitError.value = ''
|
||||
retryingChapterId.value = null
|
||||
submissionKey = ''
|
||||
submissionRequestId = ''
|
||||
}
|
||||
|
||||
return {
|
||||
books, booksLoading, booksError,
|
||||
book, chapters, bookLoading, bookError,
|
||||
chapter, chapterBook, navigation, chapterLoading, chapterError,
|
||||
submitting, submitError, retryingChapterId, readerText,
|
||||
loadBooks, loadBook, loadChapter, submit, retryChapter,
|
||||
stopPolling, closeBook, closeChapter, reset,
|
||||
}
|
||||
})
|
||||
@@ -2,6 +2,17 @@ import { defineStore } from 'pinia'
|
||||
import { ref } from 'vue'
|
||||
|
||||
export const TOKEN_KEY = 'lexgo-learner-token'
|
||||
// Carries the real HTTP status alongside the server message so callers can tell
|
||||
// "this id does not exist for me" (404) from a transient failure without
|
||||
// re-parsing the envelope. The message itself is unchanged.
|
||||
export class ApiError extends Error {
|
||||
readonly status: number
|
||||
constructor(message: string, status: number) {
|
||||
super(message)
|
||||
this.name = 'ApiError'
|
||||
this.status = status
|
||||
}
|
||||
}
|
||||
interface User { id: number; username: string; role: 'admin' | 'learner' }
|
||||
interface Space { ownerId: number; language: 'en' }
|
||||
interface Login { token: string; expiresAt: string; user: User }
|
||||
@@ -36,7 +47,7 @@ export const useSessionStore = defineStore('session', () => {
|
||||
notice.value = '登录已失效，请重新登录。'
|
||||
}
|
||||
const result = await response.json()
|
||||
if (!response.ok || result.code !== 200) throw new Error(result.msg || '请求失败，请稍后重试。')
|
||||
if (!response.ok || result.code !== 200) throw new ApiError(result.msg || '请求失败，请稍后重试。', response.status)
|
||||
return result.data as T
|
||||
}
|
||||
|
||||
@@ -91,5 +102,5 @@ export const useSessionStore = defineStore('session', () => {
|
||||
if (previousToken) await request<null>('logout', 'POST', undefined, previousToken, -1)
|
||||
}
|
||||
|
||||
return { user, space, notice, login, restore, logout, loadSpace }
|
||||
return { user, space, notice, login, restore, logout, loadSpace, request }
|
||||
})
|
||||
|
||||
@@ -51,6 +51,50 @@ h1 { font-size: 30px; font-weight: 600; margin: 14px 0; letter-spacing: 1px; }
|
||||
.empty-library .book-mark { color: #6e8967; width: 56px; height: 56px; }
|
||||
.empty-library h2 { font-weight: 500; font-size: 21px; margin: 26px 0 4px; }
|
||||
.loading { padding: 80px 24px; text-align: center; color: #748073; }
|
||||
/* Import / book / reader pages share one shell. */
|
||||
.page { max-width: 1120px; margin: 60px auto; padding: 0 28px; }
|
||||
.page-title { display: flex; align-items: center; justify-content: space-between; gap: 18px; flex-wrap: wrap; margin-bottom: 28px; }
|
||||
.page-title h1 { margin: 14px 0 6px; overflow-wrap: anywhere; }
|
||||
.page-actions { display: flex; align-items: center; gap: 12px; flex-wrap: wrap; }
|
||||
.library-actions { display: flex; align-items: center; gap: 14px; flex-wrap: wrap; }
|
||||
.link-button { display: inline-flex; align-items: center; min-height: 42px; padding: 0 8px; color: #315c43; }
|
||||
.breadcrumb { display: flex; align-items: center; gap: 8px; flex-wrap: wrap; margin: 0 0 4px; font-size: 14px; color: #748073; }
|
||||
.breadcrumb a { color: #315c43; }
|
||||
/* Library book cards. */
|
||||
.book-grid { list-style: none; margin: 0; padding: 0; display: grid; gap: 18px; grid-template-columns: repeat(auto-fill, minmax(290px, 1fr)); }
|
||||
.book-card { display: flex; flex-direction: column; gap: 10px; background: #fffdf8; border: 1px solid #e0e3d8; border-radius: 12px; padding: 22px; }
|
||||
.book-card .subtle { margin: 0; }
|
||||
.book-title { font-family: Georgia, 'Microsoft YaHei', serif; font-size: 21px; font-weight: 600; color: #233d31; text-decoration: none; overflow-wrap: anywhere; }
|
||||
.book-title:hover { color: #315c43; text-decoration: underline; }
|
||||
.status-summary { align-self: flex-start; margin: 0; padding: 6px 13px; border-radius: 20px; background: #eef2eb; color: #3d5b48; font-size: 13px; }
|
||||
/* Chapter and job status, identical vocabulary for both. */
|
||||
.status-chip { display: inline-flex; align-items: center; white-space: nowrap; padding: 5px 13px; border: 1px solid transparent; border-radius: 20px; font-size: 13px; }
|
||||
.status-pending { background: #f5f2e4; border-color: #e2dcc2; color: #7a6a35; }
|
||||
.status-processing { background: #eaf1f7; border-color: #c9dcea; color: #35566e; }
|
||||
.status-ready { background: #eef2eb; border-color: #cbd9c9; color: #315c43; }
|
||||
.status-failed { background: #fff0e7; border-color: #ebc3a8; color: #8b4324; }
|
||||
/* Import form. */
|
||||
.import-form { max-width: 720px; background: #fffdf8; border: 1px solid #e0e3d8; border-radius: 12px; padding: 28px; }
|
||||
.field-label { display: block; font-size: 14px; margin-bottom: 10px; }
|
||||
.fixed-value { margin: 0; padding: 12px 15px; border: 1px solid #d6dccf; border-radius: 8px; background: #fffefa; font-size: 15px; }
|
||||
.field-error { margin: 8px 0 0; color: #8b4324; font-size: 13px; line-height: 1.6; }
|
||||
.counter { margin: 8px 0 0; color: #748073; font-size: 13px; }
|
||||
.form-actions { display: flex; align-items: center; gap: 16px; flex-wrap: wrap; margin-top: 14px; }
|
||||
.import-form .el-textarea__inner { min-height: 220px; line-height: 1.9; }
|
||||
.book-select { width: 100%; }
|
||||
/* Book chapters. */
|
||||
.chapter-list { list-style: none; margin: 0; padding: 0; display: flex; flex-direction: column; gap: 12px; }
|
||||
.chapter-row { display: grid; grid-template-columns: 40px minmax(0, 1fr) auto auto; align-items: center; gap: 14px; background: #fffdf8; border: 1px solid #e0e3d8; border-radius: 12px; padding: 16px 18px; }
|
||||
.chapter-ordinal { font-family: Georgia, serif; font-size: 17px; color: #8b9a8b; text-align: center; }
|
||||
.chapter-info { min-width: 0; }
|
||||
.chapter-name { display: inline-block; font-size: 16px; font-weight: 500; color: #233d31; text-decoration: none; overflow-wrap: anywhere; }
|
||||
a.chapter-name:hover { color: #315c43; text-decoration: underline; }
|
||||
.chapter-meta { margin: 4px 0 0; color: #748073; font-size: 13px; overflow-wrap: anywhere; }
|
||||
/* Reader keeps the pasted text exactly as it was, including line breaks and tabs. */
|
||||
.reader-page { max-width: 820px; }
|
||||
.reader-text { white-space: pre-wrap; overflow-wrap: break-word; margin: 26px 0 0; font-family: Georgia, 'Times New Roman', 'Microsoft YaHei', serif; font-size: 17px; line-height: 2; }
|
||||
.processing-hint { padding: 36px 0; color: #748073; }
|
||||
.reader-nav { display: flex; align-items: center; justify-content: space-between; gap: 14px; flex-wrap: wrap; margin-top: 36px; padding-top: 22px; border-top: 1px solid #e0e3d8; }
|
||||
@media (max-width: 760px) {
|
||||
.login-page { grid-template-columns: 1fr; }
|
||||
.welcome { padding: 28px; }
|
||||
@@ -62,4 +106,15 @@ h1 { font-size: 30px; font-weight: 600; margin: 14px 0; letter-spacing: 1px; }
|
||||
.site-header nav { order: 3; flex-basis: 100%; padding-top: 8px; }
|
||||
.library { margin-top: 32px; padding: 0 20px; }
|
||||
h1 { font-size: 26px; }
|
||||
/* Single column, tap-friendly controls and no horizontal overflow. */
|
||||
.page { margin-top: 32px; padding: 0 20px; }
|
||||
.page-title { align-items: flex-start; }
|
||||
.library-title { align-items: flex-start; }
|
||||
.library-actions { width: 100%; justify-content: space-between; }
|
||||
.book-grid { grid-template-columns: 1fr; }
|
||||
.import-form { padding: 20px; }
|
||||
.chapter-row { grid-template-columns: 30px minmax(0, 1fr); align-items: start; row-gap: 10px; padding: 15px 16px; }
|
||||
.chapter-row .status-chip, .chapter-row .el-button { grid-column: 2; justify-self: start; }
|
||||
.reader-text { font-size: 16px; line-height: 1.95; }
|
||||
.reader-nav .el-button { flex: 1; }
|
||||
}
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
<script setup lang="ts">
|
||||
import { computed, onMounted, onUnmounted, ref, watch } from 'vue'
|
||||
import { RouterLink, useRoute, useRouter } from 'vue-router'
|
||||
import { ElButton } from 'element-plus'
|
||||
import { canRetry, statusLabel, useLibraryStore } from '../stores/library'
|
||||
import { useSessionStore } from '../stores/session'
|
||||
|
||||
const session = useSessionStore()
|
||||
const library = useLibraryStore()
|
||||
const route = useRoute()
|
||||
const router = useRouter()
|
||||
const retryError = ref('')
|
||||
|
||||
const bookId = computed(() => Number(route.params.id))
|
||||
|
||||
async function load() {
|
||||
retryError.value = ''
|
||||
await library.loadBook(bookId.value)
|
||||
}
|
||||
|
||||
async function retry(chapterId: number) {
|
||||
retryError.value = ''
|
||||
try { await library.retryChapter(chapterId) }
|
||||
catch (reason) { retryError.value = reason instanceof Error ? reason.message : '重试失败，请稍后重试。' }
|
||||
}
|
||||
|
||||
async function logout() {
|
||||
try { await session.logout() }
|
||||
catch { session.notice = '已退出此设备。服务器暂时无法连接，请稍后重试。' }
|
||||
finally { await router.replace('/login') }
|
||||
}
|
||||
|
||||
onMounted(load)
|
||||
watch(bookId, () => { void load() })
|
||||
// Leaving the page releases the book so polling stops.
|
||||
onUnmounted(() => library.closeBook())
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<div v-if="session.user">
|
||||
<header class="site-header">
|
||||
<RouterLink to="/" class="brand">LexGo<span class="brand-dot">.</span></RouterLink>
|
||||
<nav aria-label="学习导航"><RouterLink to="/">我的书库</RouterLink></nav>
|
||||
<div class="account">
|
||||
<span class="account-name">{{ session.user.username }}</span>
|
||||
<ElButton text @click="logout">退出登录</ElButton>
|
||||
</div>
|
||||
</header>
|
||||
<main class="page">
|
||||
<p class="breadcrumb"><RouterLink to="/">我的书库</RouterLink><span aria-hidden="true">/</span><span>{{ library.book?.title ?? '书籍' }}</span></p>
|
||||
<p v-if="library.bookLoading && !library.book" role="status" class="loading">正在加载…</p>
|
||||
<div v-else-if="library.bookError" class="notice">
|
||||
<p role="alert">{{ library.bookError }}</p>
|
||||
<ElButton @click="load">重试</ElButton>
|
||||
</div>
|
||||
<template v-else-if="library.book">
|
||||
<div class="page-title">
|
||||
<div>
|
||||
<h1>{{ library.book.title }}</h1>
|
||||
<p class="subtle">{{ library.chapters.length }} 个章节 · 语言 英语</p>
|
||||
</div>
|
||||
<div class="page-actions">
|
||||
<RouterLink :to="`/import?book=${library.book.id}`" class="link-button">追加章节</RouterLink>
|
||||
</div>
|
||||
</div>
|
||||
<p v-if="retryError" role="alert" class="notice">{{ retryError }}</p>
|
||||
<ul v-if="library.chapters.length" class="chapter-list" aria-label="章节列表">
|
||||
<li v-for="item in library.chapters" :key="item.id" class="chapter-row">
|
||||
<span class="chapter-ordinal">{{ item.ordinal }}</span>
|
||||
<div class="chapter-info">
|
||||
<RouterLink v-if="item.status === 'ready'" :to="`/chapters/${item.id}`" class="chapter-name">{{ item.title }}</RouterLink>
|
||||
<span v-else class="chapter-name">{{ item.title }}</span>
|
||||
<p class="chapter-meta">
|
||||
{{ item.charCount }} 字符
|
||||
<template v-if="item.status === 'failed' && item.errorMessage"> · {{ item.errorMessage }}</template>
|
||||
</p>
|
||||
</div>
|
||||
<span class="status-chip" :class="`status-${item.status}`">{{ statusLabel(item.status) }}</span>
|
||||
<ElButton
|
||||
v-if="canRetry(item)"
|
||||
size="small"
|
||||
:loading="library.retryingChapterId === item.id"
|
||||
@click="retry(item.id)"
|
||||
>重试</ElButton>
|
||||
</li>
|
||||
</ul>
|
||||
<section v-else class="empty-library" aria-label="章节列表">
|
||||
<h2>这一本书还没有章节</h2>
|
||||
<p class="subtle">粘贴一段英文即可生成第一章。</p>
|
||||
</section>
|
||||
</template>
|
||||
</main>
|
||||
</div>
|
||||
</template>
|
||||
@@ -0,0 +1,133 @@
|
||||
<script setup lang="ts">
|
||||
import { computed, onMounted, onUnmounted, ref, watch } from 'vue'
|
||||
import { RouterLink, useRoute, useRouter } from 'vue-router'
|
||||
import { ElButton, ElInput, ElOption, ElRadio, ElRadioGroup, ElSelect } from 'element-plus'
|
||||
import { LANGUAGE_LABEL, TEXT_MAX_CODE_POINTS, textProblem, titleProblem, useLibraryStore, type SubmitTarget } from '../stores/library'
|
||||
import { useSessionStore } from '../stores/session'
|
||||
|
||||
const session = useSessionStore()
|
||||
const library = useLibraryStore()
|
||||
const route = useRoute()
|
||||
const router = useRouter()
|
||||
|
||||
const title = ref('')
|
||||
const text = ref('')
|
||||
const mode = ref<'new' | 'append'>('new')
|
||||
const bookId = ref<number | undefined>(undefined)
|
||||
const titleError = ref('')
|
||||
const textError = ref('')
|
||||
const bookError = ref('')
|
||||
|
||||
const length = computed(() => [...text.value].length)
|
||||
const busy = computed(() => library.submitting)
|
||||
|
||||
function requestedBookId(): number | undefined {
|
||||
const raw = Array.isArray(route.query.book) ? route.query.book[0] : route.query.book
|
||||
if (typeof raw !== 'string' || !/^\d+$/.test(raw)) return undefined
|
||||
const value = Number(raw)
|
||||
return Number.isSafeInteger(value) && value > 0 ? value : undefined
|
||||
}
|
||||
|
||||
onMounted(() => {
|
||||
// `?book=<id>` preselects "append to an existing book".
|
||||
const preselect = requestedBookId()
|
||||
if (preselect === undefined) return
|
||||
mode.value = 'append'
|
||||
bookId.value = preselect
|
||||
})
|
||||
|
||||
watch(mode, value => {
|
||||
// The select can only list the caller's own books.
|
||||
if (value === 'append') void library.loadBooks()
|
||||
})
|
||||
|
||||
watch(title, () => { titleError.value = '' })
|
||||
watch(text, () => { textError.value = '' })
|
||||
|
||||
async function submit() {
|
||||
if (busy.value) return
|
||||
titleError.value = titleProblem(title.value)
|
||||
textError.value = textProblem(text.value)
|
||||
bookError.value = mode.value === 'append' && bookId.value === undefined ? '请选择要追加的书籍。' : ''
|
||||
if (titleError.value || textError.value || bookError.value) return
|
||||
const target: SubmitTarget = mode.value === 'append' && bookId.value !== undefined
|
||||
? { mode: 'append', bookId: bookId.value }
|
||||
: { mode: 'new' }
|
||||
try {
|
||||
const createdBookId = await library.submit({ title: title.value, text: text.value, target })
|
||||
// A response that arrives after the user left this page must not navigate them back.
|
||||
if (disposed) return
|
||||
// The requestId was consumed by this submission, so the form starts clean.
|
||||
title.value = ''
|
||||
text.value = ''
|
||||
await router.replace(`/books/${createdBookId}`)
|
||||
} catch {
|
||||
// library.submitError already carries the server message for the template.
|
||||
}
|
||||
}
|
||||
|
||||
// The form may still be awaiting its submit when the user navigates away.
|
||||
let disposed = false
|
||||
onUnmounted(() => {
|
||||
disposed = true
|
||||
library.submitError = ''
|
||||
})
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<div v-if="session.user">
|
||||
<header class="site-header">
|
||||
<RouterLink to="/" class="brand">LexGo<span class="brand-dot">.</span></RouterLink>
|
||||
<nav aria-label="学习导航"><RouterLink to="/">我的书库</RouterLink></nav>
|
||||
<div class="account">
|
||||
<span class="account-name">{{ session.user.username }}</span>
|
||||
</div>
|
||||
</header>
|
||||
<main class="page">
|
||||
<p class="breadcrumb"><RouterLink to="/">我的书库</RouterLink><span aria-hidden="true">/</span><span>导入内容</span></p>
|
||||
<div class="page-title">
|
||||
<div>
|
||||
<h1>导入英文内容</h1>
|
||||
<p class="subtle">粘贴英文正文，提交后系统会自动切分并处理章节。</p>
|
||||
</div>
|
||||
</div>
|
||||
<form class="import-form" novalidate @submit.prevent="submit">
|
||||
<div class="field">
|
||||
<span class="field-label">语言</span>
|
||||
<p class="fixed-value">{{ LANGUAGE_LABEL }}</p>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label for="title">标题</label>
|
||||
<ElInput id="title" v-model="title" type="text" maxlength="200" placeholder="例如：虚构样例第一章" :disabled="busy" />
|
||||
<p v-if="titleError" role="alert" class="field-error">{{ titleError }}</p>
|
||||
</div>
|
||||
<div class="field">
|
||||
<span class="field-label">导入到</span>
|
||||
<ElRadioGroup v-model="mode" :disabled="busy" aria-label="导入目标">
|
||||
<ElRadio value="new">新建书籍</ElRadio>
|
||||
<ElRadio value="append">追加到已有书籍</ElRadio>
|
||||
</ElRadioGroup>
|
||||
</div>
|
||||
<div v-if="mode === 'append'" class="field">
|
||||
<label for="book">选择书籍</label>
|
||||
<ElSelect id="book" v-model="bookId" placeholder="请选择要追加的书籍" :disabled="busy" class="book-select">
|
||||
<ElOption v-for="item in library.books" :key="item.id" :label="item.title" :value="item.id" />
|
||||
</ElSelect>
|
||||
<p v-if="library.booksError" role="alert" class="field-error">{{ library.booksError }}</p>
|
||||
<p v-if="bookError" role="alert" class="field-error">{{ bookError }}</p>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label for="text">正文</label>
|
||||
<ElInput id="text" v-model="text" type="textarea" :rows="12" placeholder="在此粘贴英文正文…" :disabled="busy" />
|
||||
<p class="counter">{{ length }} / {{ TEXT_MAX_CODE_POINTS }} 字符</p>
|
||||
<p v-if="textError" role="alert" class="field-error">{{ textError }}</p>
|
||||
</div>
|
||||
<p v-if="library.submitError" role="alert" class="notice">{{ library.submitError }}</p>
|
||||
<div class="form-actions">
|
||||
<ElButton type="primary" native-type="submit" :loading="busy" :disabled="busy">开始处理</ElButton>
|
||||
<RouterLink to="/" class="subtle">返回书库</RouterLink>
|
||||
</div>
|
||||
</form>
|
||||
</main>
|
||||
</div>
|
||||
</template>
|
||||
@@ -3,8 +3,10 @@ import { onMounted, ref } from 'vue'
|
||||
import { RouterLink, useRouter } from 'vue-router'
|
||||
import { ElButton } from 'element-plus'
|
||||
import BookMark from '../components/BookMark.vue'
|
||||
import { statusSummary, useLibraryStore } from '../stores/library'
|
||||
import { useSessionStore } from '../stores/session'
|
||||
const session = useSessionStore()
|
||||
const library = useLibraryStore()
|
||||
const router = useRouter()
|
||||
const loading = ref(true)
|
||||
const error = ref('')
|
||||
@@ -13,7 +15,9 @@ async function load() {
|
||||
error.value = ''
|
||||
try { await session.loadSpace() }
|
||||
catch (reason) { error.value = reason instanceof Error ? reason.message : '暂时无法加载，请重试。' }
|
||||
finally { loading.value = false }
|
||||
// The book list keeps its own error so one failing call still shows a retry.
|
||||
if (!error.value) await library.loadBooks()
|
||||
loading.value = false
|
||||
}
|
||||
async function logout() {
|
||||
try { await session.logout() }
|
||||
@@ -34,11 +38,28 @@ onMounted(load)
|
||||
</div>
|
||||
</header>
|
||||
<main class="library">
|
||||
<div class="library-title"><div><h1>我的书库</h1><p class="subtle">你的阅读与学习，从这里开始。</p></div><span class="language">英语</span></div>
|
||||
<div class="library-title">
|
||||
<div><h1>我的书库</h1><p class="subtle">你的阅读与学习，从这里开始。</p></div>
|
||||
<div class="library-actions">
|
||||
<span class="language">英语</span>
|
||||
<ElButton type="primary" @click="router.push('/import')">导入内容</ElButton>
|
||||
</div>
|
||||
</div>
|
||||
<p v-if="loading" role="status" class="loading">正在加载…</p>
|
||||
<div v-else-if="error" class="notice"><p role="alert">{{ error }}</p><ElButton @click="load">重试</ElButton></div>
|
||||
<section v-else-if="session.space" class="empty-library" aria-label="书库内容">
|
||||
<BookMark /><h2>书库还是空的</h2><p class="subtle">这里将收纳你的阅读内容。</p>
|
||||
<div v-else-if="error || library.booksError" class="notice">
|
||||
<p role="alert">{{ error || library.booksError }}</p>
|
||||
<ElButton @click="load">重试</ElButton>
|
||||
</div>
|
||||
<ul v-else-if="library.books.length" class="book-grid" aria-label="书籍列表">
|
||||
<li v-for="item in library.books" :key="item.id" class="book-card">
|
||||
<RouterLink :to="`/books/${item.id}`" class="book-title">{{ item.title }}</RouterLink>
|
||||
<p class="subtle">{{ item.chapterCount }} 个章节</p>
|
||||
<p v-if="statusSummary(item)" class="status-summary">{{ statusSummary(item) }}</p>
|
||||
<p v-else class="subtle">尚未导入章节</p>
|
||||
</li>
|
||||
</ul>
|
||||
<section v-else class="empty-library" aria-label="书库内容">
|
||||
<BookMark /><h2>书库还是空的</h2><p class="subtle">粘贴一段英文，开始你的第一篇阅读。</p>
|
||||
</section>
|
||||
</main>
|
||||
</div>
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
<script setup lang="ts">
|
||||
import { computed, onMounted, onUnmounted, ref, watch } from 'vue'
|
||||
import { RouterLink, useRoute, useRouter } from 'vue-router'
|
||||
import { ElButton } from 'element-plus'
|
||||
import { canRetry, statusLabel, useLibraryStore } from '../stores/library'
|
||||
import { useSessionStore } from '../stores/session'
|
||||
|
||||
const session = useSessionStore()
|
||||
const library = useLibraryStore()
|
||||
const route = useRoute()
|
||||
const router = useRouter()
|
||||
const retryError = ref('')
|
||||
|
||||
const chapterId = computed(() => Number(route.params.id))
|
||||
const chapter = computed(() => library.chapter)
|
||||
// Retry uses the job id the chapter carries, no matter where it was loaded from.
|
||||
const retryable = computed(() => chapter.value !== null && canRetry(chapter.value))
|
||||
|
||||
async function load() {
|
||||
retryError.value = ''
|
||||
await library.loadChapter(chapterId.value)
|
||||
}
|
||||
|
||||
async function retry() {
|
||||
if (chapter.value === null) return
|
||||
retryError.value = ''
|
||||
try { await library.retryChapter(chapter.value.id) }
|
||||
catch (reason) { retryError.value = reason instanceof Error ? reason.message : '重试失败，请稍后重试。' }
|
||||
}
|
||||
|
||||
// Switching chapters reuses this component; only the route param changes.
|
||||
function goTo(id: number | null) {
|
||||
if (id === null) return
|
||||
void router.push(`/chapters/${id}`)
|
||||
}
|
||||
|
||||
async function logout() {
|
||||
try { await session.logout() }
|
||||
catch { session.notice = '已退出此设备。服务器暂时无法连接，请稍后重试。' }
|
||||
finally { await router.replace('/login') }
|
||||
}
|
||||
|
||||
onMounted(load)
|
||||
watch(chapterId, () => { void load() })
|
||||
// Leaving the page releases the chapter so polling stops.
|
||||
onUnmounted(() => library.closeChapter())
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<div v-if="session.user">
|
||||
<header class="site-header">
|
||||
<RouterLink to="/" class="brand">LexGo<span class="brand-dot">.</span></RouterLink>
|
||||
<nav aria-label="学习导航"><RouterLink to="/">我的书库</RouterLink></nav>
|
||||
<div class="account">
|
||||
<span class="account-name">{{ session.user.username }}</span>
|
||||
<ElButton text @click="logout">退出登录</ElButton>
|
||||
</div>
|
||||
</header>
|
||||
<main class="page reader-page">
|
||||
<p v-if="library.chapterLoading && !chapter" role="status" class="loading">正在加载…</p>
|
||||
<div v-else-if="library.chapterError" class="notice">
|
||||
<p role="alert">{{ library.chapterError }}</p>
|
||||
<ElButton @click="load">重试</ElButton>
|
||||
</div>
|
||||
<template v-else-if="chapter">
|
||||
<p class="breadcrumb">
|
||||
<RouterLink to="/">我的书库</RouterLink>
|
||||
<span aria-hidden="true">/</span>
|
||||
<RouterLink v-if="library.chapterBook" :to="`/books/${library.chapterBook.id}`">{{ library.chapterBook.title }}</RouterLink>
|
||||
<span v-else>章节</span>
|
||||
</p>
|
||||
<div class="page-title">
|
||||
<h1>{{ chapter.title }}</h1>
|
||||
<span class="status-chip" :class="`status-${chapter.status}`">{{ statusLabel(chapter.status) }}</span>
|
||||
</div>
|
||||
<div v-if="chapter.status === 'failed'" class="notice">
|
||||
<p role="alert">{{ chapter.errorMessage || '这一章处理失败。' }}</p>
|
||||
<ElButton v-if="retryable" :loading="library.retryingChapterId === chapter.id" @click="retry">重试处理</ElButton>
|
||||
<p v-else class="subtle">这一章暂时没有可重试的任务编号。</p>
|
||||
</div>
|
||||
<p v-else-if="chapter.status !== 'ready'" role="status" class="processing-hint">这一章还在{{ statusLabel(chapter.status) }}，页面会自动刷新。</p>
|
||||
<p v-if="retryError" role="alert" class="notice">{{ retryError }}</p>
|
||||
<article v-if="chapter.status === 'ready'" class="reader-text">{{ library.readerText }}</article>
|
||||
<nav class="reader-nav" aria-label="章节切换">
|
||||
<ElButton :disabled="library.navigation.previousChapterId === null" @click="goTo(library.navigation.previousChapterId)">上一章</ElButton>
|
||||
<ElButton :disabled="library.navigation.nextChapterId === null" @click="goTo(library.navigation.nextChapterId)">下一章</ElButton>
|
||||
</nav>
|
||||
</template>
|
||||
</main>
|
||||
</div>
|
||||
</template>
|
||||
@@ -4,10 +4,11 @@ import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
driver "github.com/go-sql-driver/mysql"
|
||||
"gorm.io/gorm"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
driver "github.com/go-sql-driver/mysql"
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
// Migrate takes a connection-scoped lock. Only an empty or LexGo-owned schema is accepted.
|
||||
@@ -59,44 +60,17 @@ func Migrate(db *gorm.DB) error {
|
||||
if err = conn.QueryRowContext(ctx, "SELECT version,product FROM lexgo_schema WHERE id=1").Scan(¤t, &product); err != nil {
|
||||
return err
|
||||
}
|
||||
if product != "lexgo" || current < 0 || current > 2 {
|
||||
if product != "lexgo" || current < 0 || current > schemaVersion {
|
||||
return errors.New("unknown schema version")
|
||||
}
|
||||
if current == 2 {
|
||||
return nil
|
||||
// Each known version contributes its own statements; the version row advances only
|
||||
// after every statement succeeded, so a partially applied migration can be retried.
|
||||
statements := make([]string, 0, 16)
|
||||
if current < 2 {
|
||||
statements = append(statements, schemaV2Statements...)
|
||||
}
|
||||
statements := []string{
|
||||
`CREATE TABLE IF NOT EXISTS sys_user (
|
||||
user_id BIGINT PRIMARY KEY AUTO_INCREMENT, username VARCHAR(32) CHARACTER SET ascii COLLATE ascii_bin NOT NULL UNIQUE,
|
||||
password VARCHAR(128) NOT NULL, nick_name VARCHAR(128) NOT NULL DEFAULT '', phone VARCHAR(11) NOT NULL DEFAULT '',
|
||||
role_id INT NOT NULL, salt VARCHAR(255) NOT NULL DEFAULT '', avatar VARCHAR(255) NOT NULL DEFAULT '',
|
||||
sex VARCHAR(255) NOT NULL DEFAULT '', email VARCHAR(128) NOT NULL DEFAULT '', dept_id BIGINT NOT NULL DEFAULT 0,
|
||||
post_id BIGINT NOT NULL DEFAULT 0, remark VARCHAR(255) NOT NULL DEFAULT '', status VARCHAR(4) NOT NULL DEFAULT '2',
|
||||
create_by BIGINT NOT NULL DEFAULT 0, update_by BIGINT NOT NULL DEFAULT 0,
|
||||
created_at DATETIME(3) NULL, updated_at DATETIME(3) NULL, deleted_at DATETIME(3) NULL,
|
||||
CHECK (role_id IN (1,2)), CHECK (status IN ('1','2'))
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`,
|
||||
`CREATE TABLE IF NOT EXISTS lexgo_spaces (
|
||||
owner_id BIGINT PRIMARY KEY, language VARCHAR(16) NOT NULL DEFAULT 'en',
|
||||
FOREIGN KEY (owner_id) REFERENCES sys_user(user_id) ON DELETE CASCADE
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`,
|
||||
`CREATE TABLE IF NOT EXISTS lexgo_sessions (
|
||||
token_hash CHAR(64) CHARACTER SET ascii COLLATE ascii_bin PRIMARY KEY,
|
||||
owner_id BIGINT NOT NULL, expires_at DATETIME(3) NOT NULL,
|
||||
INDEX (owner_id), INDEX (expires_at),
|
||||
FOREIGN KEY (owner_id) REFERENCES sys_user(user_id) ON DELETE CASCADE
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`,
|
||||
`CREATE TABLE IF NOT EXISTS lexgo_login_logs (
|
||||
id BIGINT UNSIGNED PRIMARY KEY AUTO_INCREMENT, username VARCHAR(32) NOT NULL DEFAULT '',
|
||||
result VARCHAR(16) NOT NULL, reason VARCHAR(32) NOT NULL, ip VARCHAR(45) NOT NULL DEFAULT '',
|
||||
created_at DATETIME(3) NOT NULL, INDEX(created_at,id), INDEX(username,created_at)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`,
|
||||
`CREATE TABLE IF NOT EXISTS lexgo_operation_logs (
|
||||
id BIGINT UNSIGNED PRIMARY KEY AUTO_INCREMENT, actor_id BIGINT NOT NULL, actor_username VARCHAR(32) NOT NULL,
|
||||
target_id BIGINT NOT NULL DEFAULT 0, target_username VARCHAR(32) NOT NULL DEFAULT '',
|
||||
action VARCHAR(32) NOT NULL, result VARCHAR(16) NOT NULL, reason VARCHAR(32) NOT NULL,
|
||||
created_at DATETIME(3) NOT NULL, INDEX(created_at,id), INDEX(actor_username,created_at), INDEX(target_username,created_at)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`,
|
||||
if current < 3 {
|
||||
statements = append(statements, schemaV3Statements...)
|
||||
}
|
||||
for i, s := range statements {
|
||||
if _, err = conn.ExecContext(ctx, s); err != nil {
|
||||
@@ -107,16 +81,89 @@ func Migrate(db *gorm.DB) error {
|
||||
return fmt.Errorf("migration statement %d failed", i+1)
|
||||
}
|
||||
}
|
||||
_, err = conn.ExecContext(ctx, "UPDATE lexgo_schema SET version=2 WHERE id=1")
|
||||
_, err = conn.ExecContext(ctx, fmt.Sprintf("UPDATE lexgo_schema SET version=%d WHERE id=1", schemaVersion))
|
||||
return err
|
||||
}
|
||||
|
||||
const schemaVersion = 3
|
||||
|
||||
var schemaV2Statements = []string{
|
||||
`CREATE TABLE IF NOT EXISTS sys_user (
|
||||
user_id BIGINT PRIMARY KEY AUTO_INCREMENT, username VARCHAR(32) CHARACTER SET ascii COLLATE ascii_bin NOT NULL UNIQUE,
|
||||
password VARCHAR(128) NOT NULL, nick_name VARCHAR(128) NOT NULL DEFAULT '', phone VARCHAR(11) NOT NULL DEFAULT '',
|
||||
role_id INT NOT NULL, salt VARCHAR(255) NOT NULL DEFAULT '', avatar VARCHAR(255) NOT NULL DEFAULT '',
|
||||
sex VARCHAR(255) NOT NULL DEFAULT '', email VARCHAR(128) NOT NULL DEFAULT '', dept_id BIGINT NOT NULL DEFAULT 0,
|
||||
post_id BIGINT NOT NULL DEFAULT 0, remark VARCHAR(255) NOT NULL DEFAULT '', status VARCHAR(4) NOT NULL DEFAULT '2',
|
||||
create_by BIGINT NOT NULL DEFAULT 0, update_by BIGINT NOT NULL DEFAULT 0,
|
||||
created_at DATETIME(3) NULL, updated_at DATETIME(3) NULL, deleted_at DATETIME(3) NULL,
|
||||
CHECK (role_id IN (1,2)), CHECK (status IN ('1','2'))
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`,
|
||||
`CREATE TABLE IF NOT EXISTS lexgo_spaces (
|
||||
owner_id BIGINT PRIMARY KEY, language VARCHAR(16) NOT NULL DEFAULT 'en',
|
||||
FOREIGN KEY (owner_id) REFERENCES sys_user(user_id) ON DELETE CASCADE
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`,
|
||||
`CREATE TABLE IF NOT EXISTS lexgo_sessions (
|
||||
token_hash CHAR(64) CHARACTER SET ascii COLLATE ascii_bin PRIMARY KEY,
|
||||
owner_id BIGINT NOT NULL, expires_at DATETIME(3) NOT NULL,
|
||||
INDEX (owner_id), INDEX (expires_at),
|
||||
FOREIGN KEY (owner_id) REFERENCES sys_user(user_id) ON DELETE CASCADE
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`,
|
||||
`CREATE TABLE IF NOT EXISTS lexgo_login_logs (
|
||||
id BIGINT UNSIGNED PRIMARY KEY AUTO_INCREMENT, username VARCHAR(32) NOT NULL DEFAULT '',
|
||||
result VARCHAR(16) NOT NULL, reason VARCHAR(32) NOT NULL, ip VARCHAR(45) NOT NULL DEFAULT '',
|
||||
created_at DATETIME(3) NOT NULL, INDEX(created_at,id), INDEX(username,created_at)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`,
|
||||
`CREATE TABLE IF NOT EXISTS lexgo_operation_logs (
|
||||
id BIGINT UNSIGNED PRIMARY KEY AUTO_INCREMENT, actor_id BIGINT NOT NULL, actor_username VARCHAR(32) NOT NULL,
|
||||
target_id BIGINT NOT NULL DEFAULT 0, target_username VARCHAR(32) NOT NULL DEFAULT '',
|
||||
action VARCHAR(32) NOT NULL, result VARCHAR(16) NOT NULL, reason VARCHAR(32) NOT NULL,
|
||||
created_at DATETIME(3) NOT NULL, INDEX(created_at,id), INDEX(actor_username,created_at), INDEX(target_username,created_at)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`,
|
||||
}
|
||||
|
||||
// v3 adds the private library: books, chapters with byte-exact original text, and
|
||||
// persistent ingestion jobs. owner_id is denormalized onto chapters and jobs so every
|
||||
// query can filter by the authenticated identity without joining.
|
||||
var schemaV3Statements = []string{
|
||||
`CREATE TABLE IF NOT EXISTS lexgo_books (
|
||||
id BIGINT UNSIGNED PRIMARY KEY AUTO_INCREMENT, owner_id BIGINT NOT NULL,
|
||||
title VARCHAR(120) NOT NULL, language VARCHAR(16) NOT NULL DEFAULT 'en',
|
||||
created_at DATETIME(3) NOT NULL, updated_at DATETIME(3) NOT NULL,
|
||||
INDEX (owner_id, updated_at, id),
|
||||
FOREIGN KEY (owner_id) REFERENCES sys_user(user_id) ON DELETE CASCADE
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`,
|
||||
`CREATE TABLE IF NOT EXISTS lexgo_chapters (
|
||||
id BIGINT UNSIGNED PRIMARY KEY AUTO_INCREMENT, book_id BIGINT UNSIGNED NOT NULL, owner_id BIGINT NOT NULL,
|
||||
ordinal INT NOT NULL, title VARCHAR(120) NOT NULL,
|
||||
original_text MEDIUMTEXT NOT NULL, char_count INT NOT NULL DEFAULT 0,
|
||||
content_sha256 CHAR(64) CHARACTER SET ascii COLLATE ascii_bin NOT NULL DEFAULT '',
|
||||
status VARCHAR(16) NOT NULL DEFAULT 'pending', error_reason VARCHAR(32) NOT NULL DEFAULT '',
|
||||
created_at DATETIME(3) NOT NULL, updated_at DATETIME(3) NOT NULL,
|
||||
UNIQUE KEY uq_chapter_ordinal (book_id, ordinal), INDEX (owner_id, id),
|
||||
CHECK (status IN ('pending','processing','ready','failed')),
|
||||
FOREIGN KEY (book_id) REFERENCES lexgo_books(id) ON DELETE CASCADE
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`,
|
||||
`CREATE TABLE IF NOT EXISTS lexgo_ingest_jobs (
|
||||
id BIGINT UNSIGNED PRIMARY KEY AUTO_INCREMENT, owner_id BIGINT NOT NULL,
|
||||
book_id BIGINT UNSIGNED NOT NULL, chapter_id BIGINT UNSIGNED NOT NULL,
|
||||
request_key CHAR(64) CHARACTER SET ascii COLLATE ascii_bin NOT NULL,
|
||||
content_sha256 CHAR(64) CHARACTER SET ascii COLLATE ascii_bin NOT NULL,
|
||||
status VARCHAR(16) NOT NULL DEFAULT 'pending', attempts INT NOT NULL DEFAULT 0,
|
||||
error_reason VARCHAR(32) NOT NULL DEFAULT '',
|
||||
created_at DATETIME(3) NOT NULL, updated_at DATETIME(3) NOT NULL, finished_at DATETIME(3) NULL,
|
||||
UNIQUE KEY uq_job_request (owner_id, request_key), INDEX (status, id),
|
||||
CHECK (status IN ('pending','processing','ready','failed')),
|
||||
FOREIGN KEY (book_id) REFERENCES lexgo_books(id) ON DELETE CASCADE,
|
||||
FOREIGN KEY (chapter_id) REFERENCES lexgo_chapters(id) ON DELETE CASCADE
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`,
|
||||
}
|
||||
|
||||
func CheckSchema(db *gorm.DB) error {
|
||||
var r struct {
|
||||
Version int
|
||||
Product string
|
||||
}
|
||||
if err := db.Table("lexgo_schema").Where("id=1").First(&r).Error; err != nil || r.Version != 2 || r.Product != "lexgo" {
|
||||
if err := db.Table("lexgo_schema").Where("id=1").First(&r).Error; err != nil || r.Version != schemaVersion || r.Product != "lexgo" {
|
||||
return errors.New("run the explicit migration before starting")
|
||||
}
|
||||
return nil
|
||||
|
||||
@@ -0,0 +1,217 @@
|
||||
package lexgo
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/clause"
|
||||
)
|
||||
|
||||
// The ingestion worker is deliberately small: the fixed paste rule stores one chapter per
|
||||
// submit, so "processing" only validates the persisted content and publishes the chapter.
|
||||
// Claiming and finishing are separate transactions on purpose. A durable claim means a
|
||||
// chapter observed as processing stays recoverable, whether the process stops or only the
|
||||
// finishing transaction fails.
|
||||
|
||||
// Recovery is one mechanism used from two places: the startup pass treats every processing row
|
||||
// as abandoned, while the running worker sweeps rows that have been processing longer than any
|
||||
// legitimate claim-to-finish window. A job that keeps failing ends in a readable failure
|
||||
// instead of looping forever, and a retried job never creates a second chapter.
|
||||
const (
|
||||
maxIngestAttempts = 5
|
||||
// ingestStaleAfter must stay longer than the longest legitimate claim-to-finish window,
|
||||
// otherwise a healthy job could be processed twice. Reprocessing is harmless for content
|
||||
// because a job never creates a chapter, only publishes the one it was created with.
|
||||
ingestStaleAfter = 15 * time.Second
|
||||
)
|
||||
|
||||
// RecoverIngestJobs requeues jobs and chapters left in processing by an unclean stop. It runs
|
||||
// once before the worker starts, for a single-instance deployment.
|
||||
func RecoverIngestJobs(db *gorm.DB, now time.Time) (int64, error) {
|
||||
return requeueStaleIngestJobs(db, now, 0)
|
||||
}
|
||||
|
||||
// RequeueStaleIngestJobs recovers jobs whose finishing transaction did not complete, so a
|
||||
// running service does not depend on a restart to make progress again.
|
||||
func RequeueStaleIngestJobs(db *gorm.DB, now time.Time) (int64, error) {
|
||||
return requeueStaleIngestJobs(db, now, ingestStaleAfter)
|
||||
}
|
||||
|
||||
func requeueStaleIngestJobs(db *gorm.DB, now time.Time, staleAfter time.Duration) (int64, error) {
|
||||
ts := stamp(now)
|
||||
cutoff := stamp(now.Add(-staleAfter))
|
||||
var requeued int64
|
||||
err := db.Transaction(func(tx *gorm.DB) error {
|
||||
if err := exhaustIngestJobs(tx, ts); err != nil {
|
||||
return err
|
||||
}
|
||||
stale := []int64{}
|
||||
if err := tx.Model(&IngestJob{}).
|
||||
Where("status = ? AND attempts < ? AND updated_at <= ?", statusProcessing, maxIngestAttempts, cutoff).
|
||||
Pluck("id", &stale).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
if len(stale) == 0 {
|
||||
return nil
|
||||
}
|
||||
if err := setIngestChapterStatus(tx, stale, statusPending, "", ts); err != nil {
|
||||
return err
|
||||
}
|
||||
result := tx.Model(&IngestJob{}).Where("id IN ?", stale).
|
||||
Updates(map[string]any{"status": statusPending, "updated_at": ts})
|
||||
if result.Error != nil {
|
||||
return result.Error
|
||||
}
|
||||
requeued = result.RowsAffected
|
||||
return nil
|
||||
})
|
||||
return requeued, err
|
||||
}
|
||||
|
||||
// exhaustIngestJobs fails jobs that used up the attempt budget, so nothing can stay queued or
|
||||
// claimed forever. The recorded reason is readable and a manual retry is still allowed.
|
||||
func exhaustIngestJobs(tx *gorm.DB, ts time.Time) error {
|
||||
var exhausted []int64
|
||||
if err := tx.Model(&IngestJob{}).
|
||||
Where("status IN ? AND attempts >= ?", []string{statusPending, statusProcessing}, maxIngestAttempts).
|
||||
Pluck("id", &exhausted).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
if len(exhausted) == 0 {
|
||||
return nil
|
||||
}
|
||||
if err := setIngestChapterStatus(tx, exhausted, statusFailed, reasonAttemptsExhausted, ts); err != nil {
|
||||
return err
|
||||
}
|
||||
return tx.Model(&IngestJob{}).Where("id IN ?", exhausted).Updates(map[string]any{
|
||||
"status": statusFailed, "error_reason": reasonAttemptsExhausted, "updated_at": ts, "finished_at": ts}).Error
|
||||
}
|
||||
|
||||
// setIngestChapterStatus mirrors a job outcome onto the chapters it owns.
|
||||
func setIngestChapterStatus(tx *gorm.DB, jobIDs []int64, status, reason string, ts time.Time) error {
|
||||
var chapterIDs []int64
|
||||
if err := tx.Model(&IngestJob{}).Where("id IN ?", jobIDs).Pluck("chapter_id", &chapterIDs).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
if len(chapterIDs) == 0 {
|
||||
return nil
|
||||
}
|
||||
return tx.Model(&Chapter{}).Where("id IN ?", chapterIDs).
|
||||
Updates(map[string]any{"status": status, "error_reason": reason, "updated_at": ts}).Error
|
||||
}
|
||||
|
||||
// ClaimNextIngestJob takes the oldest pending job with attempts left and marks it processing in
|
||||
// its own transaction. The guarded update means only one worker can own a job.
|
||||
func ClaimNextIngestJob(db *gorm.DB, now time.Time) (IngestJob, bool, error) {
|
||||
ts := stamp(now)
|
||||
var job IngestJob
|
||||
err := db.Transaction(func(tx *gorm.DB) error {
|
||||
if err := tx.Where("status = ? AND attempts < ?", statusPending, maxIngestAttempts).
|
||||
Order("id ASC").First(&job).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
claim := tx.Model(&IngestJob{}).Where("id = ? AND status = ?", job.ID, statusPending).
|
||||
Updates(map[string]any{"status": statusProcessing, "attempts": gorm.Expr("attempts + 1"), "updated_at": ts})
|
||||
if claim.Error != nil {
|
||||
return claim.Error
|
||||
}
|
||||
if claim.RowsAffected != 1 {
|
||||
return errJobTaken
|
||||
}
|
||||
if err := tx.Model(&Chapter{}).Where("id = ? AND owner_id = ?", job.ChapterID, job.OwnerID).
|
||||
Updates(map[string]any{"status": statusProcessing, "updated_at": ts}).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
job.Status = statusProcessing
|
||||
job.Attempts++
|
||||
job.UpdatedAt = ts
|
||||
return nil
|
||||
})
|
||||
if errors.Is(err, gorm.ErrRecordNotFound) || errors.Is(err, errJobTaken) {
|
||||
return IngestJob{}, false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return IngestJob{}, false, err
|
||||
}
|
||||
return job, true, nil
|
||||
}
|
||||
|
||||
var errJobTaken = errors.New("ingestion job already claimed")
|
||||
|
||||
// FinishIngestJob validates the persisted chapter and publishes it, or records a fixed
|
||||
// failure reason. The check runs again here because a worker must not trust that content
|
||||
// reached the table through the paste API.
|
||||
func FinishIngestJob(ctx context.Context, db *gorm.DB, job IngestJob, now time.Time) error {
|
||||
ts := stamp(now)
|
||||
return db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
|
||||
var chapter Chapter
|
||||
if err := tx.Clauses(clause.Locking{Strength: "UPDATE"}).
|
||||
Where("id = ? AND owner_id = ?", job.ChapterID, job.OwnerID).First(&chapter).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
var book Book
|
||||
if err := tx.Where("id = ? AND owner_id = ?", job.BookID, job.OwnerID).First(&book).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
if reason := unprocessableReason(book, chapter, job); reason != "" {
|
||||
if err := tx.Model(&IngestJob{}).Where("id = ?", job.ID).Updates(map[string]any{
|
||||
"status": statusFailed, "error_reason": reason, "updated_at": ts, "finished_at": ts}).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
return tx.Model(&Chapter{}).Where("id = ?", chapter.ID).Updates(map[string]any{
|
||||
"status": statusFailed, "error_reason": reason, "updated_at": ts}).Error
|
||||
}
|
||||
if err := tx.Model(&Chapter{}).Where("id = ?", chapter.ID).Updates(map[string]any{
|
||||
"status": statusReady, "char_count": utf8.RuneCountInString(chapter.OriginalText), "updated_at": ts}).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
return tx.Model(&IngestJob{}).Where("id = ?", job.ID).Updates(map[string]any{
|
||||
"status": statusReady, "error_reason": "", "updated_at": ts, "finished_at": ts}).Error
|
||||
})
|
||||
}
|
||||
|
||||
func unprocessableReason(book Book, chapter Chapter, job IngestJob) string {
|
||||
if book.Language != "en" {
|
||||
return reasonUnsupportedLanguage
|
||||
}
|
||||
if strings.TrimFunc(chapter.OriginalText, unicode.IsSpace) == "" {
|
||||
return reasonEmptyText
|
||||
}
|
||||
if utf8.RuneCountInString(chapter.OriginalText) > maxChapterRunes {
|
||||
return reasonTooLong
|
||||
}
|
||||
// The job accepted a specific content version; a chapter changed after submission is a
|
||||
// different paste and must be submitted again rather than silently processed.
|
||||
if contentSHA(chapter.OriginalText) != job.ContentSHA256 {
|
||||
return reasonContentChanged
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ProcessIngestJobs drains up to limit pending jobs. Claiming and finishing each use their
|
||||
// own transaction, so an interrupted run simply leaves a job for recovery.
|
||||
func ProcessIngestJobs(ctx context.Context, db *gorm.DB, now func() time.Time, limit int) (int, error) {
|
||||
processed := 0
|
||||
for i := 0; i < limit; i++ {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return processed, err
|
||||
}
|
||||
job, claimed, err := ClaimNextIngestJob(db.WithContext(ctx), now())
|
||||
if err != nil {
|
||||
return processed, err
|
||||
}
|
||||
if !claimed {
|
||||
return processed, nil
|
||||
}
|
||||
if err = FinishIngestJob(ctx, db, job, now()); err != nil {
|
||||
return processed, err
|
||||
}
|
||||
processed++
|
||||
}
|
||||
return processed, nil
|
||||
}
|
||||
@@ -0,0 +1,666 @@
|
||||
package lexgo
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
|
||||
driver "github.com/go-sql-driver/mysql"
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/clause"
|
||||
)
|
||||
|
||||
// Chapter and job share one status vocabulary so the client renders either without mapping.
|
||||
const (
|
||||
statusPending = "pending"
|
||||
statusProcessing = "processing"
|
||||
statusReady = "ready"
|
||||
statusFailed = "failed"
|
||||
)
|
||||
|
||||
// Fixed worker-side failure reasons. Only these codes are stored; the readable text is
|
||||
// produced at the API boundary so no user content can leak into an error field.
|
||||
const (
|
||||
reasonUnsupportedLanguage = "unsupported_language"
|
||||
reasonTooLong = "too_long"
|
||||
reasonEmptyText = "empty_text"
|
||||
reasonContentChanged = "content_changed"
|
||||
reasonAttemptsExhausted = "attempts_exhausted"
|
||||
)
|
||||
|
||||
const (
|
||||
maxChapterRunes = 100000
|
||||
maxTitleRunes = 120
|
||||
maxBooksPerList = 200
|
||||
// A 100000 code point paste stays well inside this even with JSON escaping; the limit
|
||||
// exists only so an oversized body is rejected before it is decoded.
|
||||
maxPasteBodyBytes = 4 << 20
|
||||
maxJSONBodyBytes = 16 * 1024
|
||||
)
|
||||
|
||||
var requestIDPattern = regexp.MustCompile(`^[A-Za-z0-9_-]{8,64}$`)
|
||||
|
||||
func reasonMessage(reason string) string {
|
||||
switch reason {
|
||||
case reasonUnsupportedLanguage:
|
||||
return "当前版本只支持英语内容"
|
||||
case reasonTooLong:
|
||||
return "内容超过单章上限（100000 个字符）"
|
||||
case reasonEmptyText:
|
||||
return "章节内容为空"
|
||||
case reasonContentChanged:
|
||||
return "内容在处理前发生变化，请重新提交"
|
||||
case reasonAttemptsExhausted:
|
||||
return "处理多次失败，请重试或重新提交"
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
|
||||
type Book struct {
|
||||
ID int64 `gorm:"primaryKey"`
|
||||
OwnerID int
|
||||
Title string
|
||||
Language string
|
||||
CreatedAt time.Time
|
||||
UpdatedAt time.Time
|
||||
}
|
||||
|
||||
func (Book) TableName() string { return "lexgo_books" }
|
||||
|
||||
type Chapter struct {
|
||||
ID int64 `gorm:"primaryKey"`
|
||||
BookID int64
|
||||
OwnerID int
|
||||
Ordinal int
|
||||
Title string
|
||||
OriginalText string
|
||||
CharCount int
|
||||
ContentSHA256 string
|
||||
Status string
|
||||
ErrorReason string
|
||||
CreatedAt time.Time
|
||||
UpdatedAt time.Time
|
||||
}
|
||||
|
||||
func (Chapter) TableName() string { return "lexgo_chapters" }
|
||||
|
||||
type IngestJob struct {
|
||||
ID int64 `gorm:"primaryKey"`
|
||||
OwnerID int
|
||||
BookID int64
|
||||
ChapterID int64
|
||||
RequestKey string
|
||||
ContentSHA256 string
|
||||
Status string
|
||||
Attempts int
|
||||
ErrorReason string
|
||||
CreatedAt time.Time
|
||||
UpdatedAt time.Time
|
||||
FinishedAt *time.Time
|
||||
}
|
||||
|
||||
func (IngestJob) TableName() string { return "lexgo_ingest_jobs" }
|
||||
|
||||
type BookSummary struct {
|
||||
ID int64 `json:"id"`
|
||||
Title string `json:"title"`
|
||||
Language string `json:"language"`
|
||||
ChapterCount int `json:"chapterCount"`
|
||||
PendingCount int `json:"pendingCount"`
|
||||
ProcessingCount int `json:"processingCount"`
|
||||
ReadyCount int `json:"readyCount"`
|
||||
FailedCount int `json:"failedCount"`
|
||||
CreatedAt time.Time `json:"createdAt"`
|
||||
UpdatedAt time.Time `json:"updatedAt"`
|
||||
}
|
||||
|
||||
type BookRef struct {
|
||||
ID int64 `json:"id"`
|
||||
Title string `json:"title"`
|
||||
Language string `json:"language"`
|
||||
}
|
||||
|
||||
func bookRef(book Book) BookRef { return BookRef{book.ID, book.Title, book.Language} }
|
||||
|
||||
type ChapterSummary struct {
|
||||
ID int64 `json:"id"`
|
||||
BookID int64 `json:"bookId"`
|
||||
Ordinal int `json:"ordinal"`
|
||||
Title string `json:"title"`
|
||||
Status string `json:"status"`
|
||||
CharCount int `json:"charCount"`
|
||||
// JobID lets a client retry a failed chapter without keeping the submit response.
|
||||
JobID *int64 `json:"jobId"`
|
||||
ErrorReason string `json:"errorReason"`
|
||||
ErrorMessage string `json:"errorMessage"`
|
||||
CreatedAt time.Time `json:"createdAt"`
|
||||
UpdatedAt time.Time `json:"updatedAt"`
|
||||
}
|
||||
|
||||
type ChapterView struct {
|
||||
ChapterSummary
|
||||
ContentSHA256 string `json:"contentSha256"`
|
||||
// OriginalText is returned only for a ready chapter, so unprocessed content cannot be
|
||||
// rendered as readable text by the client.
|
||||
OriginalText string `json:"originalText,omitempty"`
|
||||
}
|
||||
|
||||
func chapterSummary(c Chapter) ChapterSummary { return chapterSummaryWithJob(c, nil) }
|
||||
|
||||
func chapterSummaryWithJob(c Chapter, jobID *int64) ChapterSummary {
|
||||
return ChapterSummary{ID: c.ID, BookID: c.BookID, Ordinal: c.Ordinal, Title: c.Title,
|
||||
Status: c.Status, CharCount: c.CharCount, JobID: jobID, ErrorReason: c.ErrorReason,
|
||||
ErrorMessage: reasonMessage(c.ErrorReason), CreatedAt: c.CreatedAt, UpdatedAt: c.UpdatedAt}
|
||||
}
|
||||
|
||||
func chapterView(c Chapter, jobID *int64) ChapterView {
|
||||
view := ChapterView{ChapterSummary: chapterSummaryWithJob(c, jobID), ContentSHA256: c.ContentSHA256}
|
||||
if c.Status == statusReady {
|
||||
view.OriginalText = c.OriginalText
|
||||
}
|
||||
return view
|
||||
}
|
||||
|
||||
type JobView struct {
|
||||
ID int64 `json:"id"`
|
||||
BookID int64 `json:"bookId"`
|
||||
ChapterID int64 `json:"chapterId"`
|
||||
Status string `json:"status"`
|
||||
Attempts int `json:"attempts"`
|
||||
ErrorReason string `json:"errorReason"`
|
||||
ErrorMessage string `json:"errorMessage"`
|
||||
CreatedAt time.Time `json:"createdAt"`
|
||||
UpdatedAt time.Time `json:"updatedAt"`
|
||||
}
|
||||
|
||||
func jobView(j IngestJob) JobView {
|
||||
return JobView{j.ID, j.BookID, j.ChapterID, j.Status, j.Attempts, j.ErrorReason, reasonMessage(j.ErrorReason), j.CreatedAt, j.UpdatedAt}
|
||||
}
|
||||
|
||||
type Navigation struct {
|
||||
PreviousChapterID *int64 `json:"previousChapterId"`
|
||||
NextChapterID *int64 `json:"nextChapterId"`
|
||||
}
|
||||
|
||||
// ReaderResponse carries everything the reader needs for one chapter of the caller's own book.
|
||||
type ReaderResponse struct {
|
||||
Book BookRef `json:"book"`
|
||||
Chapter ChapterView `json:"chapter"`
|
||||
Navigation Navigation `json:"navigation"`
|
||||
}
|
||||
|
||||
type PasteResult struct {
|
||||
Book *BookRef `json:"book,omitempty"`
|
||||
Chapter ChapterSummary `json:"chapter"`
|
||||
Job JobView `json:"job"`
|
||||
Duplicate bool `json:"duplicate"`
|
||||
}
|
||||
|
||||
type PasteBookInput struct {
|
||||
RequestID string `json:"requestId"`
|
||||
Title string `json:"title"`
|
||||
Text string `json:"text"`
|
||||
Language string `json:"language"`
|
||||
}
|
||||
|
||||
type PasteChapterInput struct {
|
||||
RequestID string `json:"requestId"`
|
||||
Title string `json:"title"`
|
||||
Text string `json:"text"`
|
||||
}
|
||||
|
||||
func stamp(now time.Time) time.Time { return now.UTC().Truncate(time.Millisecond) }
|
||||
|
||||
func contentSHA(text string) string {
|
||||
v := sha256.Sum256([]byte(text))
|
||||
return hex.EncodeToString(v[:])
|
||||
}
|
||||
|
||||
func requestKey(requestID string) (string, error) {
|
||||
if !requestIDPattern.MatchString(requestID) {
|
||||
return "", failure(400, "请求编号须为 8～64 位字母、数字、下划线或连字符")
|
||||
}
|
||||
return contentSHA(requestID), nil
|
||||
}
|
||||
|
||||
// validatePaste applies the fixed paste rules: a title within the cap, at least one
|
||||
// non-space character, and at most maxChapterRunes code points. The text itself is stored
|
||||
// exactly as received, so whitespace, punctuation and line breaks survive unchanged.
|
||||
func validatePaste(title, text string) (string, string, int, error) {
|
||||
name := strings.TrimSpace(title)
|
||||
if name == "" {
|
||||
return "", "", 0, failure(400, "请填写标题")
|
||||
}
|
||||
if utf8.RuneCountInString(name) > maxTitleRunes {
|
||||
return "", "", 0, failure(400, "标题最多 120 个字符")
|
||||
}
|
||||
if strings.TrimFunc(text, unicode.IsSpace) == "" {
|
||||
return "", "", 0, failure(400, "请输入正文内容")
|
||||
}
|
||||
count := utf8.RuneCountInString(text)
|
||||
if count > maxChapterRunes {
|
||||
return "", "", 0, failure(400, "正文超过单章上限（100000 个字符）")
|
||||
}
|
||||
return name, contentSHA(text), count, nil
|
||||
}
|
||||
|
||||
// PasteBook creates one book with its first chapter and the ingestion job. The paste and the
|
||||
// job are written in one transaction, so a rejected submit leaves no book behind.
|
||||
func PasteBook(db *gorm.DB, owner int, now time.Time, input PasteBookInput) (PasteResult, error) {
|
||||
title, sha, count, err := validatePaste(input.Title, input.Text)
|
||||
if err != nil {
|
||||
return PasteResult{}, err
|
||||
}
|
||||
if input.Language != "" && input.Language != "en" {
|
||||
return PasteResult{}, failure(400, "当前版本只支持英语内容")
|
||||
}
|
||||
key, err := requestKey(input.RequestID)
|
||||
if err != nil {
|
||||
return PasteResult{}, err
|
||||
}
|
||||
ts := stamp(now)
|
||||
var result PasteResult
|
||||
err = db.Transaction(func(tx *gorm.DB) error {
|
||||
existing, found, err := jobByRequest(tx, owner, key)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if found {
|
||||
reused, err := reusePaste(tx, existing, title, sha)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
result = reused
|
||||
return nil
|
||||
}
|
||||
book := Book{OwnerID: owner, Title: title, Language: "en", CreatedAt: ts, UpdatedAt: ts}
|
||||
if err = tx.Create(&book).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
chapter := Chapter{BookID: book.ID, OwnerID: owner, Ordinal: 1, Title: title,
|
||||
OriginalText: input.Text, CharCount: count, ContentSHA256: sha,
|
||||
Status: statusPending, CreatedAt: ts, UpdatedAt: ts}
|
||||
if err = tx.Create(&chapter).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
job := IngestJob{OwnerID: owner, BookID: book.ID, ChapterID: chapter.ID, RequestKey: key,
|
||||
ContentSHA256: sha, Status: statusPending, CreatedAt: ts, UpdatedAt: ts}
|
||||
if err = tx.Create(&job).Error; err != nil {
|
||||
return pasteInsertError(err)
|
||||
}
|
||||
ref := bookRef(book)
|
||||
result = PasteResult{Book: &ref, Chapter: chapterSummaryWithJob(chapter, &job.ID), Job: jobView(job)}
|
||||
return nil
|
||||
})
|
||||
if errors.Is(err, errRequestReuse) {
|
||||
return reusePasteByRequest(db, owner, key, title, sha)
|
||||
}
|
||||
if err != nil {
|
||||
return PasteResult{}, err
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// PasteChapter appends one chapter to a book the caller already owns. The book row is locked
|
||||
// so two appends cannot claim the same ordinal.
|
||||
func PasteChapter(db *gorm.DB, owner int, bookID int64, now time.Time, input PasteChapterInput) (PasteResult, error) {
|
||||
title, sha, count, err := validatePaste(input.Title, input.Text)
|
||||
if err != nil {
|
||||
return PasteResult{}, err
|
||||
}
|
||||
key, err := requestKey(input.RequestID)
|
||||
if err != nil {
|
||||
return PasteResult{}, err
|
||||
}
|
||||
ts := stamp(now)
|
||||
var result PasteResult
|
||||
err = db.Transaction(func(tx *gorm.DB) error {
|
||||
var book Book
|
||||
if err = lockOwnedBook(tx, owner, bookID, &book); err != nil {
|
||||
return err
|
||||
}
|
||||
existing, found, err := jobByRequest(tx, owner, key)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if found {
|
||||
if existing.BookID != book.ID {
|
||||
return failure(409, "该请求编号已用于其他内容")
|
||||
}
|
||||
reused, err := reusePaste(tx, existing, title, sha)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
result = reused
|
||||
return nil
|
||||
}
|
||||
var last int
|
||||
row := tx.Model(&Chapter{}).Where("book_id = ?", book.ID).Select("COALESCE(MAX(ordinal),0)").Row()
|
||||
if err = row.Scan(&last); err != nil {
|
||||
return err
|
||||
}
|
||||
chapter := Chapter{BookID: book.ID, OwnerID: owner, Ordinal: last + 1, Title: title,
|
||||
OriginalText: input.Text, CharCount: count, ContentSHA256: sha,
|
||||
Status: statusPending, CreatedAt: ts, UpdatedAt: ts}
|
||||
if err = tx.Create(&chapter).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
job := IngestJob{OwnerID: owner, BookID: book.ID, ChapterID: chapter.ID, RequestKey: key,
|
||||
ContentSHA256: sha, Status: statusPending, CreatedAt: ts, UpdatedAt: ts}
|
||||
if err = tx.Create(&job).Error; err != nil {
|
||||
return pasteInsertError(err)
|
||||
}
|
||||
if err = tx.Model(&Book{}).Where("id = ?", book.ID).Update("updated_at", ts).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
result = PasteResult{Chapter: chapterSummaryWithJob(chapter, &job.ID), Job: jobView(job)}
|
||||
return nil
|
||||
})
|
||||
if errors.Is(err, errRequestReuse) {
|
||||
return reusePasteByRequest(db, owner, key, title, sha)
|
||||
}
|
||||
if err != nil {
|
||||
return PasteResult{}, err
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
var errRequestReuse = errors.New("ingestion request already accepted")
|
||||
|
||||
// pasteInsertError turns a unique-key conflict on the job insert into a request reuse. The
|
||||
// transaction must be abandoned: a concurrent submit that already committed is invisible to
|
||||
// this transaction's snapshot, and its rows are re-read with locking reads below.
|
||||
func pasteInsertError(err error) error {
|
||||
var dup *driver.MySQLError
|
||||
if errors.As(err, &dup) && dup.Number == 1062 {
|
||||
return errRequestReuse
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
func reusePasteByRequest(db *gorm.DB, owner int, key, title, sha string) (PasteResult, error) {
|
||||
var result PasteResult
|
||||
err := db.Transaction(func(tx *gorm.DB) error {
|
||||
job, found, err := jobByRequestLatest(tx, owner, key)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !found {
|
||||
// The conflict came from another unique key, not from a repeated request id.
|
||||
return failure(409, "提交冲突，请重试")
|
||||
}
|
||||
result, err = reusePaste(tx, job, title, sha)
|
||||
return err
|
||||
})
|
||||
if err != nil {
|
||||
return PasteResult{}, err
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func jobByRequest(tx *gorm.DB, owner int, key string) (IngestJob, bool, error) {
|
||||
return jobQuery(tx, owner, key, false)
|
||||
}
|
||||
|
||||
// jobByRequestLatest uses a locking read, which sees the latest committed row instead of this
|
||||
// transaction's older snapshot. It is required after a duplicate-key conflict: only the
|
||||
// competing transaction's commit can cause that conflict, and its rows are newer than the
|
||||
// snapshot this request already took.
|
||||
func jobByRequestLatest(tx *gorm.DB, owner int, key string) (IngestJob, bool, error) {
|
||||
return jobQuery(tx, owner, key, true)
|
||||
}
|
||||
|
||||
func jobQuery(tx *gorm.DB, owner int, key string, latest bool) (IngestJob, bool, error) {
|
||||
var job IngestJob
|
||||
query := tx
|
||||
if latest {
|
||||
query = tx.Clauses(clause.Locking{Strength: "UPDATE"})
|
||||
}
|
||||
err := query.Where("owner_id = ? AND request_key = ?", owner, key).First(&job).Error
|
||||
if errors.Is(err, gorm.ErrRecordNotFound) {
|
||||
return job, false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return job, false, err
|
||||
}
|
||||
return job, true, nil
|
||||
}
|
||||
|
||||
// reusePaste answers a repeated submit with the first result instead of creating a second
|
||||
// chapter. A reused request id with different content or title is a conflict, not a retry.
|
||||
// Its reads are locking reads so the same answer works right after a duplicate-key conflict.
|
||||
func reusePaste(tx *gorm.DB, job IngestJob, title, sha string) (PasteResult, error) {
|
||||
var chapter Chapter
|
||||
if err := tx.Clauses(clause.Locking{Strength: "UPDATE"}).
|
||||
Where("id = ? AND owner_id = ?", job.ChapterID, job.OwnerID).First(&chapter).Error; err != nil {
|
||||
if errors.Is(err, gorm.ErrRecordNotFound) {
|
||||
return PasteResult{}, failure(409, "该请求已提交过，请刷新后查看结果")
|
||||
}
|
||||
return PasteResult{}, err
|
||||
}
|
||||
if chapter.ContentSHA256 != sha || chapter.Title != title {
|
||||
return PasteResult{}, failure(409, "该请求编号已用于其他内容")
|
||||
}
|
||||
var book Book
|
||||
if err := tx.Clauses(clause.Locking{Strength: "UPDATE"}).
|
||||
Where("id = ? AND owner_id = ?", job.BookID, job.OwnerID).First(&book).Error; err != nil {
|
||||
return PasteResult{}, err
|
||||
}
|
||||
ref := bookRef(book)
|
||||
result := PasteResult{Chapter: chapterSummaryWithJob(chapter, &job.ID), Job: jobView(job), Duplicate: true}
|
||||
if chapter.Ordinal == 1 {
|
||||
result.Book = &ref
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func lockOwnedBook(tx *gorm.DB, owner int, bookID int64, book *Book) error {
|
||||
err := tx.Clauses(clause.Locking{Strength: "UPDATE"}).
|
||||
Where("id = ? AND owner_id = ?", bookID, owner).First(book).Error
|
||||
if errors.Is(err, gorm.ErrRecordNotFound) {
|
||||
// Another user's book and a missing book are indistinguishable to the caller.
|
||||
return failure(404, "书籍不存在")
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
func ListBooks(db *gorm.DB, owner int) ([]BookSummary, error) {
|
||||
var books []Book
|
||||
if err := db.Where("owner_id = ?", owner).Order("updated_at DESC, id DESC").Limit(maxBooksPerList).Find(&books).Error; err != nil {
|
||||
return nil, err
|
||||
}
|
||||
items := make([]BookSummary, 0, len(books))
|
||||
ids := make([]int64, 0, len(books))
|
||||
for _, b := range books {
|
||||
items = append(items, BookSummary{ID: b.ID, Title: b.Title, Language: b.Language, CreatedAt: b.CreatedAt, UpdatedAt: b.UpdatedAt})
|
||||
ids = append(ids, b.ID)
|
||||
}
|
||||
if len(ids) == 0 {
|
||||
return items, nil
|
||||
}
|
||||
type row struct {
|
||||
BookID int64
|
||||
Status string
|
||||
Total int
|
||||
}
|
||||
var rows []row
|
||||
if err := db.Model(&Chapter{}).Select("book_id, status, COUNT(*) AS total").
|
||||
Where("owner_id = ? AND book_id IN ?", owner, ids).Group("book_id, status").Scan(&rows).Error; err != nil {
|
||||
return nil, err
|
||||
}
|
||||
index := make(map[int64]int, len(items))
|
||||
for i, item := range items {
|
||||
index[item.ID] = i
|
||||
}
|
||||
for _, r := range rows {
|
||||
i, ok := index[r.BookID]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
items[i].ChapterCount += r.Total
|
||||
switch r.Status {
|
||||
case statusReady:
|
||||
items[i].ReadyCount += r.Total
|
||||
case statusProcessing:
|
||||
items[i].ProcessingCount += r.Total
|
||||
case statusFailed:
|
||||
items[i].FailedCount += r.Total
|
||||
default:
|
||||
items[i].PendingCount += r.Total
|
||||
}
|
||||
}
|
||||
return items, nil
|
||||
}
|
||||
|
||||
func BookDetail(db *gorm.DB, owner int, bookID int64) (BookRef, []ChapterSummary, error) {
|
||||
var book Book
|
||||
if err := db.Where("id = ? AND owner_id = ?", bookID, owner).First(&book).Error; err != nil {
|
||||
if errors.Is(err, gorm.ErrRecordNotFound) {
|
||||
return BookRef{}, nil, failure(404, "书籍不存在")
|
||||
}
|
||||
return BookRef{}, nil, err
|
||||
}
|
||||
var chapters []Chapter
|
||||
if err := db.Where("book_id = ? AND owner_id = ?", book.ID, owner).Order("ordinal ASC").Find(&chapters).Error; err != nil {
|
||||
return BookRef{}, nil, err
|
||||
}
|
||||
items := make([]ChapterSummary, 0, len(chapters))
|
||||
ids := make([]int64, 0, len(chapters))
|
||||
for _, c := range chapters {
|
||||
ids = append(ids, c.ID)
|
||||
}
|
||||
jobs, err := jobIDsByChapter(db, owner, ids)
|
||||
if err != nil {
|
||||
return BookRef{}, nil, err
|
||||
}
|
||||
for _, c := range chapters {
|
||||
var jobID *int64
|
||||
if id, ok := jobs[c.ID]; ok {
|
||||
jobID = &id
|
||||
}
|
||||
items = append(items, chapterSummaryWithJob(c, jobID))
|
||||
}
|
||||
return bookRef(book), items, nil
|
||||
}
|
||||
|
||||
// jobIDsByChapter maps chapters to their ingestion job so a client can retry a failed chapter
|
||||
// without having kept the original submit response.
|
||||
func jobIDsByChapter(db *gorm.DB, owner int, chapterIDs []int64) (map[int64]int64, error) {
|
||||
ids := make(map[int64]int64, len(chapterIDs))
|
||||
if len(chapterIDs) == 0 {
|
||||
return ids, nil
|
||||
}
|
||||
var jobs []IngestJob
|
||||
if err := db.Select("id", "chapter_id").Where("owner_id = ? AND chapter_id IN ?", owner, chapterIDs).
|
||||
Order("id ASC").Find(&jobs).Error; err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// Ascending order keeps the newest id if a chapter somehow has more than one job.
|
||||
for _, j := range jobs {
|
||||
ids[j.ChapterID] = j.ID
|
||||
}
|
||||
return ids, nil
|
||||
}
|
||||
|
||||
// ChapterDetail resolves a chapter strictly inside the caller's own books and returns the
|
||||
// original text only once the chapter is ready.
|
||||
func ChapterDetail(db *gorm.DB, owner int, chapterID int64) (ReaderResponse, error) {
|
||||
var chapter Chapter
|
||||
if err := db.Where("id = ? AND owner_id = ?", chapterID, owner).First(&chapter).Error; err != nil {
|
||||
if errors.Is(err, gorm.ErrRecordNotFound) {
|
||||
return ReaderResponse{}, failure(404, "章节不存在")
|
||||
}
|
||||
return ReaderResponse{}, err
|
||||
}
|
||||
var book Book
|
||||
if err := db.Where("id = ? AND owner_id = ?", chapter.BookID, owner).First(&book).Error; err != nil {
|
||||
return ReaderResponse{}, err
|
||||
}
|
||||
navigation := Navigation{}
|
||||
var previous, next Chapter
|
||||
if err := db.Where("book_id = ? AND owner_id = ? AND ordinal < ?", book.ID, owner, chapter.Ordinal).
|
||||
Order("ordinal DESC").First(&previous).Error; err == nil {
|
||||
navigation.PreviousChapterID = &previous.ID
|
||||
} else if !errors.Is(err, gorm.ErrRecordNotFound) {
|
||||
return ReaderResponse{}, err
|
||||
}
|
||||
if err := db.Where("book_id = ? AND owner_id = ? AND ordinal > ?", book.ID, owner, chapter.Ordinal).
|
||||
Order("ordinal ASC").First(&next).Error; err == nil {
|
||||
navigation.NextChapterID = &next.ID
|
||||
} else if !errors.Is(err, gorm.ErrRecordNotFound) {
|
||||
return ReaderResponse{}, err
|
||||
}
|
||||
jobs, err := jobIDsByChapter(db, owner, []int64{chapter.ID})
|
||||
if err != nil {
|
||||
return ReaderResponse{}, err
|
||||
}
|
||||
var jobID *int64
|
||||
if id, ok := jobs[chapter.ID]; ok {
|
||||
jobID = &id
|
||||
}
|
||||
return ReaderResponse{Book: bookRef(book), Chapter: chapterView(chapter, jobID), Navigation: navigation}, nil
|
||||
}
|
||||
|
||||
func JobDetail(db *gorm.DB, owner int, jobID int64) (JobView, error) {
|
||||
var job IngestJob
|
||||
if err := db.Where("id = ? AND owner_id = ?", jobID, owner).First(&job).Error; err != nil {
|
||||
if errors.Is(err, gorm.ErrRecordNotFound) {
|
||||
return JobView{}, failure(404, "任务不存在")
|
||||
}
|
||||
return JobView{}, err
|
||||
}
|
||||
return jobView(job), nil
|
||||
}
|
||||
|
||||
// RetryIngestJob requeues a failed job on the same chapter, so a retry can never create a
|
||||
// second chapter for one paste. An explicit retry also restarts the attempt budget, because a
|
||||
// person asking again should not be blocked by the bound that stops automatic loops.
|
||||
func RetryIngestJob(db *gorm.DB, owner int, jobID int64, now time.Time) (JobView, ChapterSummary, error) {
|
||||
ts := stamp(now)
|
||||
var job IngestJob
|
||||
var chapter Chapter
|
||||
err := db.Transaction(func(tx *gorm.DB) error {
|
||||
if err := tx.Clauses(clause.Locking{Strength: "UPDATE"}).
|
||||
Where("id = ? AND owner_id = ?", jobID, owner).First(&job).Error; err != nil {
|
||||
if errors.Is(err, gorm.ErrRecordNotFound) {
|
||||
return failure(404, "任务不存在")
|
||||
}
|
||||
return err
|
||||
}
|
||||
if job.Status != statusFailed {
|
||||
return failure(409, "只有失败的任务可以重试")
|
||||
}
|
||||
if err := tx.Clauses(clause.Locking{Strength: "UPDATE"}).
|
||||
Where("id = ? AND owner_id = ?", job.ChapterID, owner).First(&chapter).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
if err := tx.Model(&IngestJob{}).Where("id = ?", job.ID).
|
||||
Updates(map[string]any{"status": statusPending, "error_reason": "", "attempts": 0,
|
||||
"updated_at": ts, "finished_at": nil}).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
if err := tx.Model(&Chapter{}).Where("id = ?", chapter.ID).
|
||||
Updates(map[string]any{"status": statusPending, "error_reason": "", "updated_at": ts}).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
job.Status = statusPending
|
||||
job.ErrorReason = ""
|
||||
job.Attempts = 0
|
||||
job.FinishedAt = nil
|
||||
job.UpdatedAt = ts
|
||||
chapter.Status = statusPending
|
||||
chapter.ErrorReason = ""
|
||||
chapter.UpdatedAt = ts
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return JobView{}, ChapterSummary{}, err
|
||||
}
|
||||
return jobView(job), chapterSummaryWithJob(chapter, &job.ID), nil
|
||||
}
|
||||
@@ -0,0 +1,914 @@
|
||||
package lexgo
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/gin-gonic/gin"
|
||||
admin "go-admin/app/admin/models"
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
// Fictional English fixture with the characters the paste contract must preserve exactly:
|
||||
// CRLF and LF, a tab, curly quotes, an em dash, an ellipsis, an emoji, a combining acute
|
||||
// accent, a trailing space run and an empty final line.
|
||||
const fixturePastedText = "Mira opened the workshop.\r\n\r\n\tThe sign read “A small step…” — café e\u0301 🙂\r\nTrailing spaces here: \n\n"
|
||||
|
||||
type libraryAccount struct {
|
||||
ID int
|
||||
Username string
|
||||
Token string
|
||||
}
|
||||
|
||||
type pasteResponse struct {
|
||||
Book *struct {
|
||||
ID int64
|
||||
Title string
|
||||
Language string
|
||||
}
|
||||
Chapter struct {
|
||||
ID int64
|
||||
BookID int64
|
||||
Ordinal int
|
||||
Title string
|
||||
Status string
|
||||
CharCount int
|
||||
ErrorReason string
|
||||
ErrorMessage string
|
||||
}
|
||||
Job struct {
|
||||
ID int64
|
||||
BookID int64
|
||||
ChapterID int64
|
||||
Status string
|
||||
Attempts int
|
||||
ErrorReason string
|
||||
ErrorMessage string
|
||||
}
|
||||
Duplicate bool
|
||||
}
|
||||
|
||||
type readerResponse struct {
|
||||
Book struct {
|
||||
ID int64
|
||||
Title string
|
||||
Language string
|
||||
}
|
||||
Chapter struct {
|
||||
ID int64
|
||||
BookID int64
|
||||
Ordinal int
|
||||
Title string
|
||||
Status string
|
||||
CharCount int
|
||||
ErrorReason string
|
||||
ErrorMessage string
|
||||
ContentSHA256 string
|
||||
OriginalText string
|
||||
JobID *int64
|
||||
}
|
||||
Navigation struct {
|
||||
PreviousChapterID *int64
|
||||
NextChapterID *int64
|
||||
}
|
||||
}
|
||||
|
||||
type bookDetailResponse struct {
|
||||
Book struct {
|
||||
ID int64
|
||||
Title string
|
||||
Language string
|
||||
}
|
||||
Chapters []struct {
|
||||
ID int64
|
||||
Ordinal int
|
||||
Title string
|
||||
Status string
|
||||
CharCount int
|
||||
JobID *int64
|
||||
ErrorMessage string
|
||||
}
|
||||
}
|
||||
|
||||
type bookListResponse struct {
|
||||
Items []struct {
|
||||
ID int64
|
||||
Title string
|
||||
ChapterCount int
|
||||
PendingCount int
|
||||
ProcessingCount int
|
||||
ReadyCount int
|
||||
FailedCount int
|
||||
}
|
||||
}
|
||||
|
||||
// callRaw keeps the API message, which is how a readable failure reason is asserted.
|
||||
func callRaw(t *testing.T, r *gin.Engine, method, path, token string, body any) (int, string, json.RawMessage) {
|
||||
t.Helper()
|
||||
b, _ := json.Marshal(body)
|
||||
q := httptest.NewRequest(method, path, bytes.NewReader(b))
|
||||
q.Header.Set("Content-Type", "application/json")
|
||||
if token != "" {
|
||||
q.Header.Set("Authorization", "Bearer "+token)
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
r.ServeHTTP(w, q)
|
||||
var e struct {
|
||||
Code int `json:"code"`
|
||||
Msg string `json:"msg"`
|
||||
Data json.RawMessage `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &e); err != nil {
|
||||
t.Fatalf("invalid JSON for %s %s (status %d)", method, path, w.Code)
|
||||
}
|
||||
return w.Code, e.Msg, e.Data
|
||||
}
|
||||
|
||||
func libraryFixture(t *testing.T) (*gorm.DB, *gin.Engine, libraryAccount) {
|
||||
t.Helper()
|
||||
db := testDB(t)
|
||||
owner := admin.SysUser{Username: randomName("admin"), Password: fixturePassword, RoleId: 1, Status: "2"}
|
||||
if err := db.Create(&owner).Error; err != nil {
|
||||
t.Fatal("fixture admin creation failed")
|
||||
}
|
||||
r := Router(db, time.Now)
|
||||
return db, r, libraryAccount{owner.UserId, owner.Username, loginToken(t, r, owner.Username, fixturePassword)}
|
||||
}
|
||||
|
||||
func newLearner(t *testing.T, r *gin.Engine, adminToken string) libraryAccount {
|
||||
t.Helper()
|
||||
name := randomName("lib")
|
||||
code, msg, data := callRaw(t, r, "POST", "/api/v1/accounts", adminToken, map[string]string{"username": name, "password": fixturePassword})
|
||||
if code != 201 {
|
||||
t.Fatalf("create learner status %d (%s)", code, msg)
|
||||
}
|
||||
var created struct {
|
||||
ID int
|
||||
Username string
|
||||
}
|
||||
json.Unmarshal(data, &created)
|
||||
return libraryAccount{created.ID, created.Username, loginToken(t, r, name, fixturePassword)}
|
||||
}
|
||||
|
||||
func pasteBook(t *testing.T, r *gin.Engine, token string, body any) (int, pasteResponse) {
|
||||
t.Helper()
|
||||
code, msg, data := callRaw(t, r, "POST", "/api/v1/books", token, body)
|
||||
var out pasteResponse
|
||||
if len(data) > 0 {
|
||||
if err := json.Unmarshal(data, &out); err != nil {
|
||||
t.Fatalf("paste response: %v", err)
|
||||
}
|
||||
}
|
||||
if code >= 400 && msg == "" {
|
||||
t.Fatalf("paste failed with status %d and no message", code)
|
||||
}
|
||||
return code, out
|
||||
}
|
||||
|
||||
func pasteChapter(t *testing.T, r *gin.Engine, token string, bookID int64, body any) (int, pasteResponse) {
|
||||
t.Helper()
|
||||
code, msg, data := callRaw(t, r, "POST", fmt.Sprintf("/api/v1/books/%d/chapters", bookID), token, body)
|
||||
var out pasteResponse
|
||||
if len(data) > 0 {
|
||||
if err := json.Unmarshal(data, &out); err != nil {
|
||||
t.Fatalf("paste chapter response: %v", err)
|
||||
}
|
||||
}
|
||||
if code >= 400 && msg == "" {
|
||||
t.Fatalf("paste chapter failed with status %d and no message", code)
|
||||
}
|
||||
return code, out
|
||||
}
|
||||
|
||||
func readChapter(t *testing.T, r *gin.Engine, token string, chapterID int64) (int, readerResponse) {
|
||||
t.Helper()
|
||||
code, _, data := callRaw(t, r, "GET", fmt.Sprintf("/api/v1/chapters/%d", chapterID), token, nil)
|
||||
var out readerResponse
|
||||
if len(data) > 0 {
|
||||
if err := json.Unmarshal(data, &out); err != nil {
|
||||
t.Fatalf("reader response: %v", err)
|
||||
}
|
||||
}
|
||||
return code, out
|
||||
}
|
||||
|
||||
func bookDetail(t *testing.T, r *gin.Engine, token string, bookID int64) (int, bookDetailResponse) {
|
||||
t.Helper()
|
||||
code, _, data := callRaw(t, r, "GET", fmt.Sprintf("/api/v1/books/%d", bookID), token, nil)
|
||||
var out bookDetailResponse
|
||||
if len(data) > 0 {
|
||||
if err := json.Unmarshal(data, &out); err != nil {
|
||||
t.Fatalf("book response: %v", err)
|
||||
}
|
||||
}
|
||||
return code, out
|
||||
}
|
||||
|
||||
func bookList(t *testing.T, r *gin.Engine, token string) (int, bookListResponse) {
|
||||
t.Helper()
|
||||
code, _, data := callRaw(t, r, "GET", "/api/v1/books", token, nil)
|
||||
var out bookListResponse
|
||||
if len(data) > 0 {
|
||||
if err := json.Unmarshal(data, &out); err != nil {
|
||||
t.Fatalf("book list response: %v", err)
|
||||
}
|
||||
}
|
||||
return code, out
|
||||
}
|
||||
|
||||
func drainIngest(t *testing.T, db *gorm.DB) {
|
||||
t.Helper()
|
||||
if _, err := ProcessIngestJobs(t.Context(), db, time.Now, 50); err != nil {
|
||||
t.Fatalf("ingestion failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func chapterRow(t *testing.T, db *gorm.DB, id int64) Chapter {
|
||||
t.Helper()
|
||||
var c Chapter
|
||||
if err := db.Where("id = ?", id).First(&c).Error; err != nil {
|
||||
t.Fatalf("chapter row: %v", err)
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
func jobRow(t *testing.T, db *gorm.DB, id int64) IngestJob {
|
||||
t.Helper()
|
||||
var j IngestJob
|
||||
if err := db.Where("id = ?", id).First(&j).Error; err != nil {
|
||||
t.Fatalf("job row: %v", err)
|
||||
}
|
||||
return j
|
||||
}
|
||||
|
||||
func TestMySQLPasteToReaderFullPath(t *testing.T) {
|
||||
db, r, owner := libraryFixture(t)
|
||||
learner := newLearner(t, r, owner.Token)
|
||||
// Empty the queue so the claim below takes this test's own job.
|
||||
drainIngest(t, db)
|
||||
|
||||
code, pasted := pasteBook(t, r, learner.Token, map[string]string{
|
||||
"requestId": "fixture-request-full-path", "title": "The Workshop", "text": fixturePastedText, "language": "en"})
|
||||
if code != 201 {
|
||||
t.Fatalf("paste status %d, want 201", code)
|
||||
}
|
||||
if pasted.Book == nil || pasted.Book.ID == 0 || pasted.Book.Title != "The Workshop" || pasted.Book.Language != "en" {
|
||||
t.Fatalf("unexpected book %+v", pasted.Book)
|
||||
}
|
||||
if pasted.Chapter.Ordinal != 1 || pasted.Chapter.BookID != pasted.Book.ID || pasted.Chapter.Title != "The Workshop" {
|
||||
t.Fatalf("unexpected chapter %+v", pasted.Chapter)
|
||||
}
|
||||
if pasted.Chapter.Status != statusPending || pasted.Job.Status != statusPending || pasted.Duplicate {
|
||||
t.Fatalf("paste must queue a pending job, got chapter %q job %q", pasted.Chapter.Status, pasted.Job.Status)
|
||||
}
|
||||
if want := utf8.RuneCountInString(fixturePastedText); pasted.Chapter.CharCount != want {
|
||||
t.Fatalf("charCount %d, want %d", pasted.Chapter.CharCount, want)
|
||||
}
|
||||
|
||||
// A queued chapter has no readable text yet.
|
||||
code, pending := readChapter(t, r, learner.Token, pasted.Chapter.ID)
|
||||
if code != 200 || pending.Chapter.Status != statusPending || pending.Chapter.OriginalText != "" {
|
||||
t.Fatalf("pending chapter must not expose text: status %d, %+v", code, pending.Chapter)
|
||||
}
|
||||
_, queued := bookList(t, r, learner.Token)
|
||||
if len(queued.Items) != 1 || queued.Items[0].PendingCount != 1 || queued.Items[0].ProcessingCount != 0 || queued.Items[0].ReadyCount != 0 {
|
||||
t.Fatalf("queued book counts %+v", queued.Items)
|
||||
}
|
||||
|
||||
// Processing is a durable state: a claim survives a crash and is observable in between.
|
||||
job, claimed, err := ClaimNextIngestJob(db, time.Now())
|
||||
if err != nil || !claimed {
|
||||
t.Fatalf("claim failed (claimed=%v): %v", claimed, err)
|
||||
}
|
||||
if job.ID != pasted.Job.ID || job.Attempts != 1 {
|
||||
t.Fatalf("claimed job %+v, want id %d with 1 attempt", job, pasted.Job.ID)
|
||||
}
|
||||
code, processing := readChapter(t, r, learner.Token, pasted.Chapter.ID)
|
||||
if code != 200 || processing.Chapter.Status != statusProcessing || processing.Chapter.OriginalText != "" {
|
||||
t.Fatalf("processing chapter must not expose text: status %d, %+v", code, processing.Chapter)
|
||||
}
|
||||
if err = FinishIngestJob(t.Context(), db, job, time.Now()); err != nil {
|
||||
t.Fatalf("finish failed: %v", err)
|
||||
}
|
||||
|
||||
code, ready := readChapter(t, r, learner.Token, pasted.Chapter.ID)
|
||||
if code != 200 || ready.Chapter.Status != statusReady {
|
||||
t.Fatalf("ready chapter status %d %q", code, ready.Chapter.Status)
|
||||
}
|
||||
if ready.Chapter.OriginalText != fixturePastedText {
|
||||
t.Fatalf("original text changed:\n got %q\nwant %q", ready.Chapter.OriginalText, fixturePastedText)
|
||||
}
|
||||
if ready.Chapter.ContentSHA256 != contentSHA(fixturePastedText) {
|
||||
t.Fatal("content hash mismatch")
|
||||
}
|
||||
if ready.Chapter.CharCount != utf8.RuneCountInString(fixturePastedText) {
|
||||
t.Fatalf("charCount %d after processing", ready.Chapter.CharCount)
|
||||
}
|
||||
if ready.Navigation.PreviousChapterID != nil || ready.Navigation.NextChapterID != nil {
|
||||
t.Fatal("single chapter must not navigate")
|
||||
}
|
||||
if ready.Book.ID != pasted.Book.ID || ready.Book.Title != "The Workshop" {
|
||||
t.Fatalf("unexpected reader book %+v", ready.Book)
|
||||
}
|
||||
|
||||
code, _, data := callRaw(t, r, "GET", fmt.Sprintf("/api/v1/jobs/%d", pasted.Job.ID), learner.Token, nil)
|
||||
var jobView struct {
|
||||
Job struct {
|
||||
Status string
|
||||
Attempts int
|
||||
ChapterID int64
|
||||
}
|
||||
}
|
||||
json.Unmarshal(data, &jobView)
|
||||
if code != 200 || jobView.Job.Status != statusReady || jobView.Job.Attempts != 1 || jobView.Job.ChapterID != pasted.Chapter.ID {
|
||||
t.Fatalf("job view %+v (status %d)", jobView.Job, code)
|
||||
}
|
||||
|
||||
// Appending keeps the fixed rule: one paste, one more chapter.
|
||||
code, appended := pasteChapter(t, r, learner.Token, pasted.Book.ID, map[string]string{
|
||||
"requestId": "fixture-request-append", "title": "Second Chapter", "text": "A single plain paragraph.\n"})
|
||||
if code != 201 || appended.Chapter.Ordinal != 2 || appended.Book != nil {
|
||||
t.Fatalf("append status %d chapter %+v book %+v", code, appended.Chapter, appended.Book)
|
||||
}
|
||||
drainIngest(t, db)
|
||||
code, second := readChapter(t, r, learner.Token, appended.Chapter.ID)
|
||||
if code != 200 || second.Chapter.Status != statusReady || second.Chapter.OriginalText != "A single plain paragraph.\n" {
|
||||
t.Fatalf("appended chapter %+v", second.Chapter)
|
||||
}
|
||||
if second.Navigation.PreviousChapterID == nil || *second.Navigation.PreviousChapterID != pasted.Chapter.ID || second.Navigation.NextChapterID != nil {
|
||||
t.Fatalf("appended navigation %+v", second.Navigation)
|
||||
}
|
||||
code, first := readChapter(t, r, learner.Token, pasted.Chapter.ID)
|
||||
if code != 200 || first.Navigation.NextChapterID == nil || *first.Navigation.NextChapterID != appended.Chapter.ID {
|
||||
t.Fatalf("first chapter navigation %+v", first.Navigation)
|
||||
}
|
||||
|
||||
code, detail := bookDetail(t, r, learner.Token, pasted.Book.ID)
|
||||
if code != 200 || len(detail.Chapters) != 2 || detail.Chapters[0].Status != statusReady || detail.Chapters[1].Ordinal != 2 {
|
||||
t.Fatalf("book detail %+v", detail)
|
||||
}
|
||||
// The chapter list carries the job id, which is what a client needs to retry a failure.
|
||||
if detail.Chapters[0].JobID == nil || *detail.Chapters[0].JobID != pasted.Job.ID {
|
||||
t.Fatalf("book detail chapter job id %+v, want %d", detail.Chapters[0].JobID, pasted.Job.ID)
|
||||
}
|
||||
if ready.Chapter.JobID == nil || *ready.Chapter.JobID != pasted.Job.ID {
|
||||
t.Fatalf("reader chapter job id %+v, want %d", ready.Chapter.JobID, pasted.Job.ID)
|
||||
}
|
||||
code, list := bookList(t, r, learner.Token)
|
||||
if code != 200 || len(list.Items) != 1 || list.Items[0].ChapterCount != 2 || list.Items[0].ReadyCount != 2 || list.Items[0].FailedCount != 0 {
|
||||
t.Fatalf("book list %+v", list)
|
||||
}
|
||||
if list.Items[0].PendingCount != 0 || list.Items[0].ProcessingCount != 0 {
|
||||
t.Fatalf("published book must have no queued chapter: %+v", list.Items[0])
|
||||
}
|
||||
if list.Items[0].ID != pasted.Book.ID {
|
||||
t.Fatal("book list must only contain the caller's own book")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMySQLIngestFailureReasonsAndRetry(t *testing.T) {
|
||||
db, r, owner := libraryFixture(t)
|
||||
learner := newLearner(t, r, owner.Token)
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
reason string
|
||||
mutate func(book *Book, chapter *Chapter, job *IngestJob)
|
||||
}{
|
||||
{"valid", "", func(*Book, *Chapter, *IngestJob) {}},
|
||||
{"unsupported_language", reasonUnsupportedLanguage, func(book *Book, _ *Chapter, _ *IngestJob) {
|
||||
book.Language = "de"
|
||||
}},
|
||||
{"empty_text", reasonEmptyText, func(_ *Book, chapter *Chapter, job *IngestJob) {
|
||||
chapter.OriginalText = " \r\n\t "
|
||||
chapter.ContentSHA256 = contentSHA(chapter.OriginalText)
|
||||
job.ContentSHA256 = chapter.ContentSHA256
|
||||
}},
|
||||
{"too_long", reasonTooLong, func(_ *Book, chapter *Chapter, job *IngestJob) {
|
||||
chapter.OriginalText = strings.Repeat("a", maxChapterRunes+1)
|
||||
chapter.ContentSHA256 = contentSHA(chapter.OriginalText)
|
||||
job.ContentSHA256 = chapter.ContentSHA256
|
||||
}},
|
||||
{"content_changed", reasonContentChanged, func(_ *Book, chapter *Chapter, _ *IngestJob) {
|
||||
chapter.OriginalText = "Mira opened"
|
||||
}},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
code, pasted := pasteBook(t, r, learner.Token, map[string]string{
|
||||
"requestId": randomName("case"), "title": "Case " + tc.name, "text": fixturePastedText})
|
||||
if code != 201 {
|
||||
t.Fatalf("paste status %d", code)
|
||||
}
|
||||
// These are the states another writing path could leave behind; the worker must
|
||||
// re-validate persisted content instead of trusting the paste API.
|
||||
book := Book{ID: pasted.Book.ID, OwnerID: learner.ID, Language: "en"}
|
||||
chapter := chapterRow(t, db, pasted.Chapter.ID)
|
||||
job := jobRow(t, db, pasted.Job.ID)
|
||||
tc.mutate(&book, &chapter, &job)
|
||||
if err := db.Model(&Book{}).Where("id = ?", book.ID).Update("language", book.Language).Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := db.Model(&Chapter{}).Where("id = ?", chapter.ID).
|
||||
Updates(map[string]any{"original_text": chapter.OriginalText, "content_sha256": chapter.ContentSHA256}).Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := db.Model(&IngestJob{}).Where("id = ?", job.ID).
|
||||
Update("content_sha256", job.ContentSHA256).Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
drainIngest(t, db)
|
||||
code, read := readChapter(t, r, learner.Token, pasted.Chapter.ID)
|
||||
if code != 200 {
|
||||
t.Fatalf("read status %d", code)
|
||||
}
|
||||
if tc.reason == "" {
|
||||
if read.Chapter.Status != statusReady || read.Chapter.OriginalText != fixturePastedText {
|
||||
t.Fatalf("valid content must reach ready, got %+v", read.Chapter)
|
||||
}
|
||||
return
|
||||
}
|
||||
if read.Chapter.Status != statusFailed || read.Chapter.ErrorReason != tc.reason {
|
||||
t.Fatalf("chapter %q with reason %q, want failed/%s", read.Chapter.Status, read.Chapter.ErrorReason, tc.reason)
|
||||
}
|
||||
// The readable reason is produced by the API and never stores user content.
|
||||
if read.Chapter.ErrorMessage == "" || strings.ContainsAny(read.Chapter.ErrorMessage, "\r\n") {
|
||||
t.Fatalf("unreadable failure message %q", read.Chapter.ErrorMessage)
|
||||
}
|
||||
if read.Chapter.OriginalText != "" {
|
||||
t.Fatal("failed chapter must not expose text")
|
||||
}
|
||||
var detail bookDetailResponse
|
||||
_, detail = bookDetail(t, r, learner.Token, pasted.Book.ID)
|
||||
if detail.Chapters[0].Status != statusFailed || detail.Chapters[0].ErrorMessage == "" {
|
||||
t.Fatalf("book detail must show the failure: %+v", detail.Chapters[0])
|
||||
}
|
||||
// Retry through the job id the chapter list exposes, which is the client's only path.
|
||||
if detail.Chapters[0].JobID == nil || *detail.Chapters[0].JobID != pasted.Job.ID {
|
||||
t.Fatalf("failed chapter must expose its job id: %+v", detail.Chapters[0])
|
||||
}
|
||||
code, _, _ = callRaw(t, r, "POST", fmt.Sprintf("/api/v1/jobs/%d/retry", *detail.Chapters[0].JobID), learner.Token, nil)
|
||||
if code != 200 {
|
||||
t.Fatalf("retry status %d, want 200", code)
|
||||
}
|
||||
// Retrying reuses the same chapter: no second chapter for one paste.
|
||||
var count int64
|
||||
db.Model(&Chapter{}).Where("book_id = ?", pasted.Book.ID).Count(&count)
|
||||
if count != 1 {
|
||||
t.Fatalf("retry created %d chapters, want 1", count)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestMySQLRetryAfterContentRestoredPublishesSameChapter(t *testing.T) {
|
||||
db, r, owner := libraryFixture(t)
|
||||
learner := newLearner(t, r, owner.Token)
|
||||
code, pasted := pasteBook(t, r, learner.Token, map[string]string{
|
||||
"requestId": "fixture-retry", "title": "Retry Book", "text": fixturePastedText})
|
||||
if code != 201 {
|
||||
t.Fatalf("paste status %d", code)
|
||||
}
|
||||
// Simulate a chapter whose stored text was replaced before processing.
|
||||
if err := db.Model(&Chapter{}).Where("id = ?", pasted.Chapter.ID).
|
||||
Update("original_text", "Mira opened").Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
drainIngest(t, db)
|
||||
code, failed := readChapter(t, r, learner.Token, pasted.Chapter.ID)
|
||||
if code != 200 || failed.Chapter.Status != statusFailed || failed.Chapter.ErrorReason != reasonContentChanged {
|
||||
t.Fatalf("expected content_changed failure, got %+v", failed.Chapter)
|
||||
}
|
||||
code, msg, _ := callRaw(t, r, "POST", fmt.Sprintf("/api/v1/jobs/%d/retry", pasted.Job.ID), learner.Token, nil)
|
||||
if code != 200 {
|
||||
t.Fatalf("retry status %d (%s)", code, msg)
|
||||
}
|
||||
// A pending retry exposes no text and does not create a new chapter.
|
||||
code, pending := readChapter(t, r, learner.Token, pasted.Chapter.ID)
|
||||
if code != 200 || pending.Chapter.Status != statusPending || pending.Chapter.OriginalText != "" {
|
||||
t.Fatalf("retry must return the chapter to pending, got %+v", pending.Chapter)
|
||||
}
|
||||
var count int64
|
||||
db.Model(&Chapter{}).Where("book_id = ?", pasted.Book.ID).Count(&count)
|
||||
if count != 1 {
|
||||
t.Fatalf("retry created %d chapters, want 1", count)
|
||||
}
|
||||
// Simulate the content being restored to what was submitted, then retry to completion.
|
||||
if err := db.Model(&Chapter{}).Where("id = ?", pasted.Chapter.ID).
|
||||
Update("original_text", fixturePastedText).Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
drainIngest(t, db)
|
||||
code, ready := readChapter(t, r, learner.Token, pasted.Chapter.ID)
|
||||
if code != 200 || ready.Chapter.Status != statusReady || ready.Chapter.OriginalText != fixturePastedText {
|
||||
t.Fatalf("retry did not publish the same chapter: %+v", ready.Chapter)
|
||||
}
|
||||
job := jobRow(t, db, pasted.Job.ID)
|
||||
// A manual retry restarts the attempt budget, so this processing is attempt 1 again.
|
||||
if job.Attempts != 1 {
|
||||
t.Fatalf("attempts %d after a manual retry, want 1", job.Attempts)
|
||||
}
|
||||
code, msg, _ = callRaw(t, r, "POST", fmt.Sprintf("/api/v1/jobs/%d/retry", pasted.Job.ID), learner.Token, nil)
|
||||
if code != 409 || msg == "" {
|
||||
t.Fatalf("retry of a ready job status %d (%s), want 409 with a message", code, msg)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMySQLRepeatedPasteIsIdempotent(t *testing.T) {
|
||||
db, r, owner := libraryFixture(t)
|
||||
learner := newLearner(t, r, owner.Token)
|
||||
body := map[string]string{"requestId": "fixture-request-repeat", "title": "Repeat", "text": fixturePastedText}
|
||||
|
||||
code, first := pasteBook(t, r, learner.Token, body)
|
||||
if code != 201 || first.Duplicate {
|
||||
t.Fatalf("first paste status %d duplicate %v", code, first.Duplicate)
|
||||
}
|
||||
code, second := pasteBook(t, r, learner.Token, body)
|
||||
if code != 200 || !second.Duplicate {
|
||||
t.Fatalf("repeat paste status %d duplicate %v, want 200 with duplicate", code, second.Duplicate)
|
||||
}
|
||||
if second.Chapter.ID != first.Chapter.ID || second.Job.ID != first.Job.ID || second.Book.ID != first.Book.ID {
|
||||
t.Fatalf("repeat paste returned different rows: %+v vs %+v", second, first)
|
||||
}
|
||||
var books, chapters int64
|
||||
db.Model(&Book{}).Where("owner_id = ?", learner.ID).Count(&books)
|
||||
db.Model(&Chapter{}).Where("owner_id = ?", learner.ID).Count(&chapters)
|
||||
if books != 1 || chapters != 1 {
|
||||
t.Fatalf("repeat paste created books=%d chapters=%d, want 1/1", books, chapters)
|
||||
}
|
||||
|
||||
// The same request id with different content is a conflict, not a silent reuse.
|
||||
for name, changed := range map[string]map[string]string{
|
||||
"different text": {"requestId": "fixture-request-repeat", "title": "Repeat", "text": "Mira opened the workshop."},
|
||||
"different title": {"requestId": "fixture-request-repeat", "title": "Another", "text": fixturePastedText},
|
||||
} {
|
||||
code, msg, _ := callRaw(t, r, "POST", "/api/v1/books", learner.Token, changed)
|
||||
if code != 409 || msg == "" {
|
||||
t.Fatalf("%s: status %d (%s), want 409", name, code, msg)
|
||||
}
|
||||
}
|
||||
|
||||
// Two concurrent submits of one request id create exactly one chapter.
|
||||
name := randomName("race")
|
||||
body = map[string]string{"requestId": name, "title": "Race", "text": "A small step; a small step"}
|
||||
var wg sync.WaitGroup
|
||||
type attempt struct {
|
||||
code int
|
||||
msg string
|
||||
}
|
||||
results := make(chan attempt, 2)
|
||||
for i := 0; i < 2; i++ {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
code, msg, _ := callRaw(t, r, "POST", "/api/v1/books", learner.Token, body)
|
||||
results <- attempt{code, msg}
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
close(results)
|
||||
created, reused := 0, 0
|
||||
for result := range results {
|
||||
switch result.code {
|
||||
case 201:
|
||||
created++
|
||||
case 200:
|
||||
reused++
|
||||
default:
|
||||
t.Fatalf("concurrent paste status %d (%s)", result.code, result.msg)
|
||||
}
|
||||
}
|
||||
if created != 1 || reused != 1 {
|
||||
t.Fatalf("concurrent paste created=%d reused=%d", created, reused)
|
||||
}
|
||||
db.Model(&Chapter{}).Where("owner_id = ?", learner.ID).Count(&chapters)
|
||||
if chapters != 2 {
|
||||
t.Fatalf("concurrent paste left %d chapters, want 2", chapters)
|
||||
}
|
||||
|
||||
// A request id already used by another book cannot be replayed by appending, while a
|
||||
// fresh request id appends normally.
|
||||
var raceBook Book
|
||||
if err := db.Where("owner_id = ? AND title = ?", learner.ID, "Race").First(&raceBook).Error; err != nil {
|
||||
t.Fatalf("race book: %v", err)
|
||||
}
|
||||
code, msg, _ := callRaw(t, r, "POST", fmt.Sprintf("/api/v1/books/%d/chapters", first.Book.ID), learner.Token,
|
||||
map[string]string{"requestId": name, "title": "Append", "text": "A second chapter."})
|
||||
if code != 409 || msg == "" {
|
||||
t.Fatalf("cross-book request id status %d (%s), want 409", code, msg)
|
||||
}
|
||||
code, appended := pasteChapter(t, r, learner.Token, first.Book.ID, map[string]string{
|
||||
"requestId": randomName("append"), "title": "Append", "text": "A second chapter."})
|
||||
if code != 201 || appended.Chapter.Ordinal != 2 {
|
||||
t.Fatalf("append status %d chapter %+v", code, appended.Chapter)
|
||||
}
|
||||
if raceBook.ID == first.Book.ID {
|
||||
t.Fatal("idempotency fixtures must use different books")
|
||||
}
|
||||
// Append owns the language of its book, so the field is not part of that contract and the
|
||||
// strict decoder rejects it. The learner client must therefore not send it (regression R1).
|
||||
code, msg, _ = callRaw(t, r, "POST", fmt.Sprintf("/api/v1/books/%d/chapters", first.Book.ID), learner.Token,
|
||||
map[string]string{"requestId": randomName("append"), "title": "Strict", "text": "Strict contract.", "language": "en"})
|
||||
if code != 400 || msg == "" {
|
||||
t.Fatalf("append with language status %d (%s), want 400", code, msg)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMySQLLibraryIsolationAndOwnership(t *testing.T) {
|
||||
db, r, owner := libraryFixture(t)
|
||||
a := newLearner(t, r, owner.Token)
|
||||
b := newLearner(t, r, owner.Token)
|
||||
|
||||
code, pasted := pasteBook(t, r, a.Token, map[string]string{
|
||||
"requestId": "fixture-request-isolation", "title": "Private Book", "text": "Only A may read this."})
|
||||
if code != 201 {
|
||||
t.Fatalf("paste status %d", code)
|
||||
}
|
||||
// Force a failed job so the retry path is checked for another account too.
|
||||
if err := db.Model(&IngestJob{}).Where("id = ?", pasted.Job.ID).Updates(map[string]any{"status": statusFailed, "error_reason": reasonContentChanged}).Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
var book Book
|
||||
var chapter Chapter
|
||||
var job IngestJob
|
||||
db.Where("id = ?", pasted.Book.ID).First(&book)
|
||||
db.Where("id = ?", pasted.Chapter.ID).First(&chapter)
|
||||
db.Where("id = ?", pasted.Job.ID).First(&job)
|
||||
if book.OwnerID != a.ID || chapter.OwnerID != a.ID || job.OwnerID != a.ID {
|
||||
t.Fatal("stored rows must belong to the authenticated account")
|
||||
}
|
||||
if chapter.BookID != book.ID || job.ChapterID != chapter.ID {
|
||||
t.Fatal("job and chapter must stay linked to the book")
|
||||
}
|
||||
|
||||
for name, token := range map[string]string{"other learner": b.Token, "administrator": owner.Token} {
|
||||
for _, path := range []string{
|
||||
fmt.Sprintf("/api/v1/books/%d", book.ID),
|
||||
fmt.Sprintf("/api/v1/chapters/%d", chapter.ID),
|
||||
fmt.Sprintf("/api/v1/jobs/%d", job.ID),
|
||||
} {
|
||||
code, _, _ := callRaw(t, r, "GET", path, token, nil)
|
||||
if code != 404 {
|
||||
t.Fatalf("%s GET %s status %d, want 404", name, path, code)
|
||||
}
|
||||
}
|
||||
code, _, _ = callRaw(t, r, "POST", fmt.Sprintf("/api/v1/books/%d/chapters", book.ID), token,
|
||||
map[string]string{"requestId": randomName("intruder"), "title": "Intruder", "text": "Intruder text."})
|
||||
if code != 404 {
|
||||
t.Fatalf("%s append status %d, want 404", name, code)
|
||||
}
|
||||
code, _, _ = callRaw(t, r, "POST", fmt.Sprintf("/api/v1/jobs/%d/retry", job.ID), token, nil)
|
||||
if code != 404 {
|
||||
t.Fatalf("%s retry status %d, want 404", name, code)
|
||||
}
|
||||
_, list := bookList(t, r, token)
|
||||
if len(list.Items) != 0 {
|
||||
t.Fatalf("%s sees %d books", name, len(list.Items))
|
||||
}
|
||||
}
|
||||
|
||||
// The caller's identity comes from the session, never from the request body or query.
|
||||
for _, payload := range []map[string]any{
|
||||
{"requestId": randomName("owner"), "title": "Spoof", "text": "Spoofed owner.", "ownerId": b.ID},
|
||||
{"requestId": randomName("owner"), "title": "Spoof", "text": "Spoofed owner.", "userId": b.ID},
|
||||
} {
|
||||
code, msg, _ := callRaw(t, r, "POST", "/api/v1/books", a.Token, payload)
|
||||
if code != 400 || msg == "" {
|
||||
t.Fatalf("client-supplied owner status %d (%s), want 400", code, msg)
|
||||
}
|
||||
}
|
||||
code, msg, _ := callRaw(t, r, "GET", fmt.Sprintf("/api/v1/books?ownerId=%d", b.ID), a.Token, nil)
|
||||
if code != 400 || msg == "" {
|
||||
t.Fatalf("query owner override status %d (%s), want 400", code, msg)
|
||||
}
|
||||
var books int64
|
||||
db.Model(&Book{}).Where("owner_id = ?", a.ID).Count(&books)
|
||||
if books != 1 {
|
||||
t.Fatalf("rejected requests created %d books", books)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMySQLIngestRecoveryWithoutRestart(t *testing.T) {
|
||||
db, r, owner := libraryFixture(t)
|
||||
learner := newLearner(t, r, owner.Token)
|
||||
drainIngest(t, db)
|
||||
code, pasted := pasteBook(t, r, learner.Token, map[string]string{
|
||||
"requestId": "fixture-in-service-recovery", "title": "In-service recovery", "text": fixturePastedText})
|
||||
if code != 201 {
|
||||
t.Fatalf("paste status %d", code)
|
||||
}
|
||||
job, claimed, err := ClaimNextIngestJob(db, time.Now())
|
||||
if err != nil || !claimed || job.ID != pasted.Job.ID {
|
||||
t.Fatalf("claim failed (claimed=%v, job=%d): %v", claimed, job.ID, err)
|
||||
}
|
||||
|
||||
// The finishing transaction fails (context cancellation stands in for a timeout or a
|
||||
// database error). The claim is already committed, so the job stays processing.
|
||||
canceled, cancel := context.WithCancel(t.Context())
|
||||
cancel()
|
||||
if err = FinishIngestJob(canceled, db, job, time.Now()); err == nil {
|
||||
t.Fatal("a canceled finishing transaction must report an error")
|
||||
}
|
||||
var stuck IngestJob
|
||||
if err = db.Where("id = ?", pasted.Job.ID).First(&stuck).Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if stuck.Status != statusProcessing {
|
||||
t.Fatalf("job status %q after a failed finish, want processing", stuck.Status)
|
||||
}
|
||||
// A manual retry cannot rescue it: only failed jobs are accepted.
|
||||
code, msg, _ := callRaw(t, r, "POST", fmt.Sprintf("/api/v1/jobs/%d/retry", pasted.Job.ID), learner.Token, nil)
|
||||
if code != 409 || msg == "" {
|
||||
t.Fatalf("retry of a processing job status %d (%s), want 409", code, msg)
|
||||
}
|
||||
|
||||
// A sweep that is too early must leave a healthy claim alone.
|
||||
claimedAt := stuck.UpdatedAt
|
||||
if _, err = RequeueStaleIngestJobs(db, claimedAt.Add(time.Second)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if jobRow(t, db, pasted.Job.ID).Status != statusProcessing {
|
||||
t.Fatal("a fresh claim must not be requeued")
|
||||
}
|
||||
// Once the claim is older than the stale window, the running service recovers it.
|
||||
if _, err = RequeueStaleIngestJobs(db, claimedAt.Add(ingestStaleAfter+time.Second)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
recovered := jobRow(t, db, pasted.Job.ID)
|
||||
if recovered.Status != statusPending || recovered.Attempts != 1 || recovered.FinishedAt != nil {
|
||||
t.Fatalf("recovered job %+v", recovered)
|
||||
}
|
||||
if chapterRow(t, db, pasted.Chapter.ID).Status != statusPending {
|
||||
t.Fatal("recovered chapter must be pending")
|
||||
}
|
||||
// Recovery reuses the same rows: no second chapter, same task id.
|
||||
var chapters int64
|
||||
db.Model(&Chapter{}).Where("book_id = ?", pasted.Book.ID).Count(&chapters)
|
||||
if chapters != 1 {
|
||||
t.Fatalf("recovery left %d chapters, want 1", chapters)
|
||||
}
|
||||
drainIngest(t, db)
|
||||
code, ready := readChapter(t, r, learner.Token, pasted.Chapter.ID)
|
||||
if code != 200 || ready.Chapter.Status != statusReady || ready.Chapter.OriginalText != fixturePastedText {
|
||||
t.Fatalf("recovered chapter %+v", ready.Chapter)
|
||||
}
|
||||
if ready.Chapter.ID != pasted.Chapter.ID || ready.Chapter.JobID == nil || *ready.Chapter.JobID != pasted.Job.ID {
|
||||
t.Fatal("recovery must keep the original chapter and task ids")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMySQLIngestAttemptsAreBoundedAndManualRetryRestarts(t *testing.T) {
|
||||
db, r, owner := libraryFixture(t)
|
||||
learner := newLearner(t, r, owner.Token)
|
||||
drainIngest(t, db)
|
||||
code, pasted := pasteBook(t, r, learner.Token, map[string]string{
|
||||
"requestId": "fixture-attempt-budget", "title": "Attempt budget", "text": fixturePastedText})
|
||||
if code != 201 {
|
||||
t.Fatalf("paste status %d", code)
|
||||
}
|
||||
// Spend the whole budget without any worker running.
|
||||
if err := db.Model(&IngestJob{}).Where("id = ?", pasted.Job.ID).Update("attempts", maxIngestAttempts).Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, claimed, err := ClaimNextIngestJob(db, time.Now()); err != nil || claimed {
|
||||
t.Fatalf("claim claimed=%v (%v), want no claim once the budget is used", claimed, err)
|
||||
}
|
||||
if _, err := RequeueStaleIngestJobs(db, time.Now()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
exhausted := jobRow(t, db, pasted.Job.ID)
|
||||
if exhausted.Status != statusFailed || exhausted.ErrorReason != reasonAttemptsExhausted {
|
||||
t.Fatalf("exhausted job %+v", exhausted)
|
||||
}
|
||||
code, read := readChapter(t, r, learner.Token, pasted.Chapter.ID)
|
||||
if code != 200 || read.Chapter.Status != statusFailed || read.Chapter.ErrorReason != reasonAttemptsExhausted {
|
||||
t.Fatalf("exhausted chapter %+v", read.Chapter)
|
||||
}
|
||||
if read.Chapter.ErrorMessage == "" || read.Chapter.OriginalText != "" {
|
||||
t.Fatalf("exhausted chapter must fail readably without text: %+v", read.Chapter)
|
||||
}
|
||||
// The manual retry is still available and restarts the attempt budget.
|
||||
code, msg, _ := callRaw(t, r, "POST", fmt.Sprintf("/api/v1/jobs/%d/retry", pasted.Job.ID), learner.Token, nil)
|
||||
if code != 200 {
|
||||
t.Fatalf("manual retry status %d (%s), want 200", code, msg)
|
||||
}
|
||||
retried := jobRow(t, db, pasted.Job.ID)
|
||||
if retried.Status != statusPending || retried.Attempts != 0 || retried.ErrorReason != "" {
|
||||
t.Fatalf("retried job %+v", retried)
|
||||
}
|
||||
drainIngest(t, db)
|
||||
code, ready := readChapter(t, r, learner.Token, pasted.Chapter.ID)
|
||||
if code != 200 || ready.Chapter.Status != statusReady || ready.Chapter.ErrorReason != "" {
|
||||
t.Fatalf("chapter after manual retry %+v", ready.Chapter)
|
||||
}
|
||||
if ready.Chapter.ID != pasted.Chapter.ID {
|
||||
t.Fatal("manual retry must reuse the same chapter")
|
||||
}
|
||||
var chapters int64
|
||||
db.Model(&Chapter{}).Where("book_id = ?", pasted.Book.ID).Count(&chapters)
|
||||
if chapters != 1 {
|
||||
t.Fatalf("attempt recovery left %d chapters, want 1", chapters)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMySQLIngestRecoveryAfterRestart(t *testing.T) {
|
||||
db, r, owner := libraryFixture(t)
|
||||
learner := newLearner(t, r, owner.Token)
|
||||
// The test database is shared with other cases, so empty the queue first: the claim below
|
||||
// must take this test's own job, not a leftover one.
|
||||
drainIngest(t, db)
|
||||
code, pasted := pasteBook(t, r, learner.Token, map[string]string{
|
||||
"requestId": "fixture-recovery", "title": "Recovery", "text": fixturePastedText})
|
||||
if code != 201 {
|
||||
t.Fatalf("paste status %d", code)
|
||||
}
|
||||
// A crash between claim and finish leaves rows in processing.
|
||||
job, claimed, err := ClaimNextIngestJob(db, time.Now())
|
||||
if err != nil || !claimed {
|
||||
t.Fatalf("claim failed (claimed=%v): %v", claimed, err)
|
||||
}
|
||||
if job.ID != pasted.Job.ID {
|
||||
t.Fatalf("claimed job %d, want this test's job %d", job.ID, pasted.Job.ID)
|
||||
}
|
||||
if chapterRow(t, db, pasted.Chapter.ID).Status != statusProcessing || jobRow(t, db, pasted.Job.ID).Status != statusProcessing {
|
||||
t.Fatal("claim must persist the processing state")
|
||||
}
|
||||
requeued, err := RecoverIngestJobs(db, time.Now())
|
||||
if err != nil || requeued < 1 {
|
||||
t.Fatalf("recovery requeued %d (%v), want at least this test's job", requeued, err)
|
||||
}
|
||||
pending := jobRow(t, db, pasted.Job.ID)
|
||||
if pending.Status != statusPending || pending.Attempts != 1 || pending.FinishedAt != nil {
|
||||
t.Fatalf("recovered job %+v", pending)
|
||||
}
|
||||
if chapterRow(t, db, pasted.Chapter.ID).Status != statusPending {
|
||||
t.Fatal("recovered chapter must be pending")
|
||||
}
|
||||
drainIngest(t, db)
|
||||
code, ready := readChapter(t, r, learner.Token, pasted.Chapter.ID)
|
||||
if code != 200 || ready.Chapter.Status != statusReady || ready.Chapter.OriginalText != fixturePastedText {
|
||||
t.Fatalf("recovered chapter %+v", ready.Chapter)
|
||||
}
|
||||
if jobRow(t, db, pasted.Job.ID).Attempts != 2 {
|
||||
t.Fatal("the recovered job must be processed as a second attempt")
|
||||
}
|
||||
var chapters int64
|
||||
db.Model(&Chapter{}).Where("book_id = ?", pasted.Book.ID).Count(&chapters)
|
||||
if chapters != 1 {
|
||||
t.Fatalf("recovery left %d chapters, want 1", chapters)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMySQLPasteRejectsInvalidInputAndLimits(t *testing.T) {
|
||||
db, r, owner := libraryFixture(t)
|
||||
learner := newLearner(t, r, owner.Token)
|
||||
|
||||
longTitle := strings.Repeat("T", maxTitleRunes+1)
|
||||
overLimit := strings.Repeat("a", maxChapterRunes+1)
|
||||
cases := []struct {
|
||||
name string
|
||||
payload map[string]string
|
||||
}{
|
||||
{"missing_request_id", map[string]string{"title": "T", "text": "Text."}},
|
||||
{"short_request_id", map[string]string{"requestId": "short", "title": "T", "text": "Text."}},
|
||||
{"empty_title", map[string]string{"requestId": randomName("invalid"), "title": " ", "text": "Text."}},
|
||||
{"long_title", map[string]string{"requestId": randomName("invalid"), "title": longTitle, "text": "Text."}},
|
||||
{"empty_text", map[string]string{"requestId": randomName("invalid"), "title": "T", "text": " \r\n\t "}},
|
||||
{"unsupported_language", map[string]string{"requestId": randomName("invalid"), "title": "T", "text": "Text.", "language": "zh"}},
|
||||
{"over_limit", map[string]string{"requestId": randomName("invalid"), "title": "T", "text": overLimit}},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
code, msg, _ := callRaw(t, r, "POST", "/api/v1/books", learner.Token, tc.payload)
|
||||
if code != 400 || msg == "" {
|
||||
t.Fatalf("status %d (%s), want 400 with a message", code, msg)
|
||||
}
|
||||
})
|
||||
}
|
||||
// The boundary itself is accepted, and an oversized body is refused before decoding.
|
||||
code, accepted := pasteBook(t, r, learner.Token, map[string]string{
|
||||
"requestId": "fixture-boundary-limit", "title": "At the limit", "text": strings.Repeat("a", maxChapterRunes)})
|
||||
if code != 201 || accepted.Chapter.CharCount != maxChapterRunes {
|
||||
t.Fatalf("boundary paste status %d charCount %d", code, accepted.Chapter.CharCount)
|
||||
}
|
||||
code, msg, _ := callRaw(t, r, "POST", "/api/v1/books", learner.Token, map[string]string{
|
||||
"requestId": "fixture-body-limit", "title": "Too large", "text": strings.Repeat("a", maxPasteBodyBytes)})
|
||||
if code != 400 || !strings.Contains(msg, "过大") {
|
||||
t.Fatalf("oversized body status %d (%s), want 400 with 过大", code, msg)
|
||||
}
|
||||
var books, chapters int64
|
||||
db.Model(&Book{}).Where("owner_id = ?", learner.ID).Count(&books)
|
||||
db.Model(&Chapter{}).Where("owner_id = ?", learner.ID).Count(&chapters)
|
||||
if books != 1 || chapters != 1 {
|
||||
t.Fatalf("rejected input created books=%d chapters=%d", books, chapters)
|
||||
}
|
||||
}
|
||||
@@ -88,11 +88,68 @@ func emptyMigrationDB(t *testing.T) *gorm.DB {
|
||||
return db
|
||||
}
|
||||
|
||||
func TestMigrationFromV2PreservesExistingData(t *testing.T) {
|
||||
db := emptyMigrationDB(t)
|
||||
// Build a v2 database by hand: this is the state a deployed instance is in before #5.
|
||||
if err := db.Exec("CREATE TABLE lexgo_schema (id INT PRIMARY KEY,version INT,product VARCHAR(32))").Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := db.Exec("INSERT INTO lexgo_schema VALUES (1,2,'lexgo')").Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, statement := range schemaV2Statements {
|
||||
if err := db.Exec(statement).Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err := db.Exec("INSERT INTO sys_user (user_id,username,password,role_id) VALUES (7,'fixture_v2','x',2)").Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := db.Exec("INSERT INTO lexgo_spaces (owner_id,language) VALUES (7,'en')").Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := Migrate(db); err != nil {
|
||||
t.Fatalf("v2 to v3 migration failed: %v", err)
|
||||
}
|
||||
if err := CheckSchema(db); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, table := range []string{"lexgo_books", "lexgo_chapters", "lexgo_ingest_jobs"} {
|
||||
var count int64
|
||||
db.Raw("SELECT COUNT(*) FROM information_schema.tables WHERE table_schema=DATABASE() AND table_name=?", table).Scan(&count)
|
||||
if count != 1 {
|
||||
t.Fatalf("migration did not create %s", table)
|
||||
}
|
||||
}
|
||||
var users, spaces int64
|
||||
db.Table("sys_user").Where("user_id = 7").Count(&users)
|
||||
db.Table("lexgo_spaces").Where("owner_id = 7").Count(&spaces)
|
||||
if users != 1 || spaces != 1 {
|
||||
t.Fatalf("migration changed existing rows: users=%d spaces=%d", users, spaces)
|
||||
}
|
||||
// A rollback marker set back to 2 can be upgraded again without touching data.
|
||||
if err := db.Exec("UPDATE lexgo_schema SET version=2 WHERE id=1").Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := Migrate(db); err != nil {
|
||||
t.Fatalf("re-upgrade failed: %v", err)
|
||||
}
|
||||
var version int
|
||||
db.Raw("SELECT version FROM lexgo_schema WHERE id=1").Scan(&version)
|
||||
if version != 3 {
|
||||
t.Fatalf("schema version %d after re-upgrade, want 3", version)
|
||||
}
|
||||
db.Table("sys_user").Where("user_id = 7").Count(&users)
|
||||
if users != 1 {
|
||||
t.Fatal("re-upgrade changed existing rows")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMigrationRefusesUnownedOrUnsupportedSchema(t *testing.T) {
|
||||
for _, tc := range []struct{ name, marker string }{
|
||||
{"empty_marker", ""},
|
||||
{"negative_version", "INSERT INTO lexgo_schema VALUES (1,-1,'lexgo')"},
|
||||
{"future_version", "INSERT INTO lexgo_schema VALUES (1,3,'lexgo')"},
|
||||
{"future_version", "INSERT INTO lexgo_schema VALUES (1,4,'lexgo')"},
|
||||
{"wrong_product", "INSERT INTO lexgo_schema VALUES (1,0,'another-app')"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
|
||||
@@ -129,6 +129,12 @@ func Router(db *gorm.DB, now func() time.Time) *gin.Engine {
|
||||
if c.Request.Method == "POST" && c.FullPath() == "/api/v1/accounts" {
|
||||
status = 201
|
||||
}
|
||||
// A repeated paste is answered from the first result, so it is not a new resource.
|
||||
if c.Request.Method == "POST" && (c.FullPath() == "/api/v1/books" || c.FullPath() == "/api/v1/books/:id/chapters") {
|
||||
if paste, ok := data.(PasteResult); ok && !paste.Duplicate {
|
||||
status = 201
|
||||
}
|
||||
}
|
||||
respond(c, status, data, err)
|
||||
}
|
||||
}
|
||||
@@ -213,18 +219,99 @@ func Router(db *gorm.DB, now func() time.Time) *gin.Engine {
|
||||
}
|
||||
return updateAccount(tx, id, u.UserId, input)
|
||||
}))
|
||||
pathID := func(c *gin.Context, message string) (int64, error) {
|
||||
id, err := strconv.ParseInt(c.Param("id"), 10, 64)
|
||||
if err != nil || id <= 0 {
|
||||
return 0, failure(404, message)
|
||||
}
|
||||
return id, nil
|
||||
}
|
||||
v.POST("/books", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) {
|
||||
var input PasteBookInput
|
||||
if err := decodeLimit(c, &input, maxPasteBodyBytes); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return PasteBook(tx, u.UserId, now(), input)
|
||||
}))
|
||||
v.GET("/books", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) {
|
||||
if c.Request.URL.RawQuery != "" {
|
||||
return nil, failure(400, "书库不接受查询参数")
|
||||
}
|
||||
items, err := ListBooks(tx, u.UserId)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return gin.H{"items": items}, nil
|
||||
}))
|
||||
v.GET("/books/:id", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) {
|
||||
id, err := pathID(c, "书籍不存在")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
book, chapters, err := BookDetail(tx, u.UserId, id)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return gin.H{"book": book, "chapters": chapters}, nil
|
||||
}))
|
||||
v.POST("/books/:id/chapters", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) {
|
||||
id, err := pathID(c, "书籍不存在")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var input PasteChapterInput
|
||||
if err := decodeLimit(c, &input, maxPasteBodyBytes); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return PasteChapter(tx, u.UserId, id, now(), input)
|
||||
}))
|
||||
v.GET("/chapters/:id", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) {
|
||||
id, err := pathID(c, "章节不存在")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return ChapterDetail(tx, u.UserId, id)
|
||||
}))
|
||||
v.GET("/jobs/:id", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) {
|
||||
id, err := pathID(c, "任务不存在")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
job, err := JobDetail(tx, u.UserId, id)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return gin.H{"job": job}, nil
|
||||
}))
|
||||
v.POST("/jobs/:id/retry", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) {
|
||||
id, err := pathID(c, "任务不存在")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
job, chapter, err := RetryIngestJob(tx, u.UserId, id, now())
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return gin.H{"job": job, "chapter": chapter}, nil
|
||||
}))
|
||||
r.NoRoute(func(c *gin.Context) { respond(c, 404, nil, failure(404, "页面或接口不存在")) })
|
||||
return r
|
||||
}
|
||||
|
||||
func decode(c *gin.Context, value any) error {
|
||||
func decode(c *gin.Context, value any) error { return decodeLimit(c, value, maxJSONBodyBytes) }
|
||||
|
||||
func decodeLimit(c *gin.Context, value any, limit int64) error {
|
||||
if !strings.HasPrefix(c.GetHeader("Content-Type"), "application/json") {
|
||||
return failure(400, "请使用 JSON 请求")
|
||||
}
|
||||
c.Request.Body = http.MaxBytesReader(c.Writer, c.Request.Body, 16*1024)
|
||||
c.Request.Body = http.MaxBytesReader(c.Writer, c.Request.Body, limit)
|
||||
d := json.NewDecoder(c.Request.Body)
|
||||
d.DisallowUnknownFields()
|
||||
if d.Decode(value) != nil {
|
||||
if err := d.Decode(value); err != nil {
|
||||
var tooLarge *http.MaxBytesError
|
||||
if errors.As(err, &tooLarge) {
|
||||
return failure(400, "内容过大，请减少后重试")
|
||||
}
|
||||
return failure(400, "请求内容无效")
|
||||
}
|
||||
if d.Decode(new(any)) != io.EOF {
|
||||
|
||||
@@ -82,7 +82,7 @@ func run() error {
|
||||
if err = lexgo.Migrate(db); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Println("LexGo schema version 2 ready")
|
||||
fmt.Println("LexGo schema version 3 ready")
|
||||
return nil
|
||||
}
|
||||
if err = lexgo.CheckSchema(db); err != nil {
|
||||
@@ -112,6 +112,36 @@ func run() error {
|
||||
srv := &http.Server{Addr: addr, Handler: lexgo.Router(db, time.Now), ReadHeaderTimeout: 5 * time.Second, ReadTimeout: 15 * time.Second, WriteTimeout: 15 * time.Second, IdleTimeout: 60 * time.Second, MaxHeaderBytes: 1 << 20}
|
||||
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
|
||||
defer stop()
|
||||
// A single instance owns the worker. Startup recovery returns chapters left in processing
|
||||
// by an unclean stop, and the running loop sweeps jobs whose finishing transaction failed,
|
||||
// so a transient database problem does not need a restart.
|
||||
if _, err = lexgo.RecoverIngestJobs(db, time.Now()); err != nil {
|
||||
return errors.New("ingestion recovery failed")
|
||||
}
|
||||
go func() {
|
||||
ticker := time.NewTicker(time.Second)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
jobCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
requeued, recoverErr := lexgo.RequeueStaleIngestJobs(db, time.Now())
|
||||
if recoverErr != nil {
|
||||
if ctx.Err() == nil {
|
||||
log.Print("ingestion recovery failed; the next sweep retries it")
|
||||
}
|
||||
} else if requeued > 0 && ctx.Err() == nil {
|
||||
log.Printf("requeued %d interrupted ingestion job(s)", requeued)
|
||||
}
|
||||
if _, err := lexgo.ProcessIngestJobs(jobCtx, db, time.Now, 20); err != nil && ctx.Err() == nil {
|
||||
log.Print("ingestion batch stopped before finishing; the claimed job stays processing until the next sweep requeues it")
|
||||
}
|
||||
cancel()
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
}
|
||||
}
|
||||
}()
|
||||
go func() {
|
||||
ticker := time.NewTicker(time.Hour)
|
||||
defer ticker.Stop()
|
||||
|
||||
Reference in New Issue
Block a user