diff --git a/README.md b/README.md index 18dee8c..301066d 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ 面向自托管场景的阅读式语言学习项目,规划提供内容导入、阅读查词、词汇与短语、复习和实例管理。 -已确认:**DevHarness 轻量模式、MySQL 8、go-admin 管理端**。工程基础 #2 已通过验收:两端用户名登录、学习账号管理、可撤销会话和本人英语空空间。管理端基于指定 go-admin/go-admin-ui 选用模块,学习端为独立 Vue 3 + TypeScript + Vite 工程,共用 Go 后端和 MySQL 8.4.3。#18 登录日志与操作审计已通过用户验收,支持管理员查询和 90 天保留清理。阅读、导入、词典与复习尚未实现。MVP 定位为“支持多账号、数据独立的自托管学习工具”,先邀请少量用户使用;F01~F12 已确认,X 系列后置。 +已确认:**DevHarness 轻量模式、MySQL 8、go-admin 管理端**。工程基础 #2 已通过验收:两端用户名登录、学习账号管理、可撤销会话和本人英语空空间。管理端基于指定 go-admin/go-admin-ui 选用模块,学习端为独立 Vue 3 + TypeScript + Vite 工程,共用 Go 后端和 MySQL 8.4.3。#18 登录日志与操作审计已通过用户验收,支持管理员查询和 90 天保留清理。#5 粘贴导入与章节原文阅读已实现,待用户验收;点词查词、词典、词语状态与复习仍未实现。MVP 定位为“支持多账号、数据独立的自托管学习工具”,先邀请少量用户使用;F01~F12 已确认,X 系列后置。 - [文档入口](docs/README.md) · [线上 Wiki](https://git.ilapage.cn/OPC/lexgo/wiki/Home) - [英语分词与离线词典验证小样](spikes/english/README.md)(#3 已验收,独立本机入口) @@ -19,7 +19,7 @@ 本地入口:学习端 http://127.0.0.1:5173,管理端 http://127.0.0.1:5174。完整安装与测试命令见[开发与验证](docs/04-local-development-and-verification.md)。账号使用用户名,无需邮箱;没有随代码交付的默认密码。 -本机三个服务已由 `D:/supervisord/supervisord.conf` 中的 lexgo-learner、lexgo-admin、lexgo-api 托管,使用时不要重复手动启动同端口。当前 schema v2;从 #2 升级时停止 API,执行 build、migrate 后再启动。审计过期清理可执行 `python scripts/server.py audit-cleanup`,只影响超过 90 天的日志。 +本机三个服务已由 `D:/supervisord/supervisord.conf` 中的 lexgo-learner、lexgo-admin、lexgo-api 托管,使用时不要重复手动启动同端口。当前 schema v3;从 #2/#18 升级时停止 API,执行 build、migrate 后再启动。审计过期清理可执行 `python scripts/server.py audit-cleanup`,只影响超过 90 天的日志。 ## 文档与治理 diff --git a/docs/02-architecture-and-code-map.md b/docs/02-architecture-and-code-map.md index 6583309..e1a3d4f 100644 --- a/docs/02-architecture-and-code-map.md +++ b/docs/02-architecture-and-code-map.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki,禁止直接编辑本文件) wiki_page: Architecture-and-Code-Map wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Architecture-and-Code-Map.- -wiki_revision: 5bc7ccf33d1cc1cc28598c90e27a2ab6af4146bd -synchronized_at: 2026-09-10T14:33:23Z +wiki_revision: f981b86a7c6e5d2e353142d3006dcd6e0636c3ee +synchronized_at: 2026-09-11T03:01:48Z # 架构与代码地图 @@ -175,3 +175,49 @@ WordNet 使用 ZIP 内原始 index/data/exception 文件,不使用 SysDict 或 - [resources/js/components/Review/ReviewHotkeyInformationDialog.vue](https://github.com/simjanos-dev/LinguaCafe/blob/c1ea298ce40c65b9dd33e9b26fd2e52fae66f2c8/resources/js/components/Review/ReviewHotkeyInformationDialog.vue) 上游 LICENSE 为 GPL v3,本单只核对并描述行为,没有移植源码。#1 评论 7497 的四项对照登记由本节补充;原型 v1 的保存/关闭主要流程保持,不更改其历史验收记录,不声称已经在 Quant-UX 新建修订版。范围调整与原生手柄作为可运行小样验证,若真机结果导致主要流程变化,应先更新关键原型状态并由用户确认。 + +## 粘贴导入、章节与阅读(#5,schema v3) + +#5 实现了目标路径一的第一段可运行链路:粘贴英语文本 → 持久导入任务 → 固定分章 → 处理中/就绪 → 本人阅读原文。Go 单进程同时承担 API 与后台任务。本单不接入 Python NLP,token、lemma 与词典索引仍待 #6,路线未决边界见业务规则页。 + +| 路径 | 职责 | +|---|---| +| server/app/lexgo/database.go | schema v3 显式迁移:lexgo_books、lexgo_chapters、lexgo_ingest_jobs;按版本累加语句,版本行只在全部语句成功后推进 | +| server/app/lexgo/library.go | 粘贴校验与固定分章、书籍/章节/任务写入、本人归属查询、重试与请求幂等 | +| server/app/lexgo/ingest.go | 任务声明 claim、处理完成、固定失败原因、启动恢复 | +| server/app/lexgo/router.go | 新增书籍/章节/任务路由;粘贴请求使用独立的 4 MiB 体积上限 | +| server/cmd/lexgo/main.go | serve 启动时恢复遗留任务,并按秒轮询处理待处理任务 | +| learner/src/stores/library.ts、views/ImportView.vue、BookView.vue、ReaderView.vue | 粘贴导入、书库与章节状态、失败重试、原文阅读 | + +任务状态为 pending → processing → ready/failed,章节与任务共用同一套状态词。声明与完成分属两个事务:声明一经提交,即使进程随即退出,也只会留下可被启动恢复重新入队的 processing 记录。 + +### 粘贴导入 API v1(#5) + +| 方法与路径 | 行为和权限 | +|---|---| +| POST /api/v1/books | {requestId,title,text,language?};创建书籍+首个章节+导入任务;重复 requestId 返回首次结果(HTTP 200,duplicate=true) | +| POST /api/v1/books/:id/chapters | 向本人书籍追加一个章节 | +| GET /api/v1/books | 本人书库与章节状态计数;拒绝查询参数,避免用参数替换认证身份 | +| GET /api/v1/books/:id | 本人书籍与章节列表,含 jobId 与可读失败原因 | +| GET /api/v1/chapters/:id | 本人章节详情;仅 ready 时返回 originalText,并附带前后章节编号 | +| GET /api/v1/jobs/:id | 本人任务状态、尝试次数与失败原因 | +| POST /api/v1/jobs/:id/retry | 仅失败任务可重试;复用同一章节,不新建章节 | + +所有接口按认证身份过滤 owner_id;他人书籍、章节或任务编号统一返回 404,管理员角色也不能解除学习数据的本人归属过滤。后台任务只使用任务行内的 owner_id,不接受客户端用户编号;请求体含未知字段(例如 ownerId)直接返回 400。 + +### schema v3 + +lexgo_books(owner_id, title, language)、lexgo_chapters(book_id, owner_id, ordinal, title, original_text MEDIUMTEXT, char_count, content_sha256, status, error_reason) 与 lexgo_ingest_jobs(owner_id, book_id, chapter_id, request_key, content_sha256, status, attempts, error_reason, finished_at)。owner_id 在章节与任务上冗余存放,使任何查询都能直接按认证身份过滤而不依赖连接;UNIQUE(book_id, ordinal) 与 UNIQUE(owner_id, request_key) 分别阻止重复章节与重复提交。启动检查要求版本 3,服务不自动迁移。 + +并发重复提交:请求命中 request_key 唯一键冲突后,用加锁读读取已提交结果,因为该请求事务的快照早于并发提交;因此两个并发相同提交只会产生一个章节,另一个得到 duplicate=true 的首次结果。 + +## #5 审核整改(R1~R4,2026-09-11) + +提交见工单 #5 的整改评论;本条记录实现与验证方式。 + +- 追加契约(R1):学习端把新建与追加拆成两个请求类型,追加不发送 language;后端保持严格解码,并新增回归测试断言“追加带 language 返回 400、不带则 201”,学习端单测断言追加请求体只有 requestId/title/text。 +- 运行期任务恢复(R2):`server/app/lexgo/ingest.go` 的恢复逻辑合并为一处——启动恢复使用阈值 0,运行期每轮清扫使用 15 秒阈值并把超过 5 次尝试的任务置为 failed(原因码 attempts_exhausted);`cmd/lexgo/main.go` 的 worker 每秒先清扫再处理,日志分别说明“已重新入队”与“本批未完成、等待下一次清扫”,不再声称已完成实际跳过的重试。 +- 离页作废在途请求(R3):`closeBook`/`closeChapter` 推进请求序号并清理 loading;`ImportView` 记录是否已卸载,卸载后的成功响应不再触发跳转。 +- 重试自愈(R4):`retryChapter` 先把重试返回的章节状态应用到列表与阅读器并重新安排轮询,再做静默刷新。 + +验证:Go 全量用例 20 项通过(新增运行期恢复与尝试上限两项);学习端单测 38 项通过,其中 7 项在整改前的代码上复现失败;真实联调确认追加路径可用、被中断的任务在运行中被自动恢复(约 0.5 秒,无需重启)、重试在首次刷新失败后仍自动显示最终结果。 diff --git a/docs/03-business-rules-and-glossary.md b/docs/03-business-rules-and-glossary.md index 98d8d94..ab588a8 100644 --- a/docs/03-business-rules-and-glossary.md +++ b/docs/03-business-rules-and-glossary.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki,禁止直接编辑本文件) wiki_page: Business-Rules-and-Glossary wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Business-Rules-and-Glossary.- -wiki_revision: 227c6cb92d30490f0f1f4b1b9f29cdbed8559ee4 -synchronized_at: 2026-09-10T14:33:25Z +wiki_revision: 4bd843d1f0134f5688407b7072d98333b40b6da1 +synchronized_at: 2026-09-11T03:01:48Z # 业务规则与术语 @@ -112,3 +112,28 @@ POST /lookup 接收 {surface,lemma?}。查词键单独 casefold/NFC/弯撇号转 桌面鼠标/手机长按使用原生 Selection;正文中键盘 ←/→ 选相邻词,Shift+←/→ 从锚点扩缩连续范围,起点/终点按钮可用 Tab/Enter。Esc 或关闭清除选区及未保存编辑,保留滚动和键盘阅读位置;切换章节清除选区,当前页面内分别记住章节滚动位置。纯标点的新选区清除旧面板,防止操作上一个词。个人释义必须显式保存,状态变化也随保存提交;本小样仅写内存。 手机方案保留浏览器原生长按、选择手柄和滚动,面板最多占底部 42dvh,正文有底部阅读余量;键盘定位用滚动边距避开面板。未获得真实手机结果,不能判定手柄、系统菜单、虚拟键盘或触摸滚动冲突已经解决。 + +## #5 粘贴分章、任务与阅读规则 v1 + +用户于 2026-09-10 确认两项边界(工单 #5 评论 7644):本单按 Go 处理、不接入 Python;一次粘贴等于一个章节。 + +- 固定分章规则:一次粘贴产生一个章节,不按空行或长度自动再分。新建书籍时书籍标题与首章标题同为提交的标题;追加时标题即新章节标题。分章规则变化属于需求变化,必须重新确认。 +- 长度与校验:标题去首尾空白后 1~120 个字符;正文必须含至少一个非空白字符;正文上限 100000 Unicode code point,超出返回 400;语言当前只接受 en。 +- 原文保真:正文按收到的字符串原样保存与返回,不做 NFC、大小写、换行或空白归一化;页面使用 white-space: pre-wrap 展示,制表符、连续空格与空行保持可见。处理完成前不返回原文。 +- 归属:书籍、章节与任务都记录认证账号的 owner_id;他人编号返回 404;管理员角色不解除学习数据的本人归属;后台任务只使用任务行的 owner,不信任客户端用户编号。 +- 任务状态:pending、processing、ready、failed,章节与任务共用同一词表。失败时返回固定原因码加可读中文提示,错误字段不保存正文。 +- 固定失败原因:unsupported_language、too_long、empty_text、content_changed。前三种只能由其他写入路径产生(例如语言调整或后续编辑功能);content_changed 表示章节内容在处理前被改动,属于过期任务,必须重新提交,或恢复为提交时的内容后重试。 +- 幂等:客户端 requestId 与账号构成唯一键。同一 requestId 配同标题同正文的重复提交返回首次结果,不新建章节;同一 requestId 配不同标题或正文返回 409;并发重复提交同样只产生一个章节。重试复用原章节,只增加尝试次数。 +- 恢复:声明与完成分属两个事务。进程在声明后退出时,重启把 processing 的章节与任务放回 pending 并保留尝试次数,不产生重复章节。 +- 未决边界:Go+Python NLP 与全 Go 路线尚未确认。本单只做 Go 校验、分章与发布,不产生 token、lemma 或词典索引;正式接入前必须由用户确认路线,再定义生产 token 契约。 +- 已知限制:本单未设置每账号书籍数量或总容量配额,只限制单次正文与请求体大小;删除书籍/章节属 #10,导入失败不会自动重试,只在启动时恢复被中断的 processing 任务。 + +## #5 审核整改(R1~R4,2026-09-11) + +工单 #5 的整改记录见该工单评论;本节只记录长期有效的契约变化。 + +- 追加章节沿用所属书籍的语言:`POST /api/v1/books/:id/chapters` 的请求体只有 requestId、title、text,不接受 language;服务端仍拒绝未知字段,客户端发 language 会得到 400。新建书籍的 `POST /api/v1/books` 才带 language。学习端已按此拆分请求体,避免两个契约共用同一结构。 +- 导入任务的自动重试有上限:同一任务被 worker 领取的次数达到 5 次后,任务与章节转为 failed,原因码 `attempts_exhausted`,提示“处理多次失败,请重试或重新提交”。人工重试(POST /jobs/:id/retry)会重置尝试次数,因此人工操作不受该上限阻塞。 +- 运行期恢复不依赖重启:除了启动时的恢复,运行中的服务每次轮询都会把停留在 processing 且超过 15 秒的任务放回 pending,因此“领取已提交、完成事务失败”不会让章节永久卡在处理中。该阈值必须长于正常的领取到完成窗口;重复处理同一任务不会产生第二个章节,因为任务从不创建章节。 +- 学习端离开页面时作废在途请求:目录页与阅读页在关闭时推进各自的请求序号,晚到的响应不会写回状态或重启轮询;导入页在提交过程中离开后,晚到的成功响应不会把用户导航回书籍页。 +- 重试结果立即生效:重试被接受后先把返回的 pending 状态写入界面并继续轮询,因此紧随其后的一次刷新失败不会让页面停在处理失败。 diff --git a/docs/04-local-development-and-verification.md b/docs/04-local-development-and-verification.md index 1586682..59ee197 100644 --- a/docs/04-local-development-and-verification.md +++ b/docs/04-local-development-and-verification.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki,禁止直接编辑本文件) wiki_page: Local-Development-and-Verification wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Local-Development-and-Verification.- -wiki_revision: 694facbb7a861182122c9f6c69d28c58b005cb1a -synchronized_at: 2026-09-10T14:52:22Z +wiki_revision: 3bce5fae964b1050513c37319262679814d68f14 +synchronized_at: 2026-09-11T03:01:49Z # 本地开发与验证 @@ -246,3 +246,62 @@ node --test spikes/english/view.test.mjs #4 用户验收记录:2026-09-10T22:51:52+08:00 用户确认“#4通过验收”(评论 7636)。未补充手机型号/浏览器/操作记录,未重跑测试,未自动合并 PR。后续正式移动端集成应补真机回归。 + +## #5 粘贴导入与章节阅读(schema v3) + +升级步骤(本机,仓库根执行):停止 lexgo-api → `python scripts/server.py build` → `python scripts/server.py migrate` → 启动 lexgo-api。lexgo_dev 已从 v2 升到 v3,新增 lexgo_books、lexgo_chapters、lexgo_ingest_jobs;迁移前后 sys_user 4、lexgo_spaces 4、lexgo_sessions 3、lexgo_login_logs 23、lexgo_operation_logs 1 完全一致。托管实例重启后 /healthz 返回 200,两端首页仍为 200,lexgo-admin 与 lexgo-learner 的 PID 未变化。 + +回退:停止 API,把 lexgo_schema 中 id=1 的版本从 3 改回 2,并恢复上一二进制;三张新表保留不删除,旧程序不读写它们。重新升级时显式 migrate 重新执行 IF NOT EXISTS 语句即可;集成测试覆盖 v2→v3 的既有数据保留与 v2 标记下的重复迁移。未在开发库演练回退。 + +学习端入口:http://127.0.0.1:5173 → 登录 → 我的书库 → 粘贴文本导入 → 章节就绪后进入阅读。 + +测试命令与结果(仓库根执行;本单使用专用库 lexgo_test_issue5,不借用其他测试库): + +| 命令 | 本次结果 | +|---|---| +| `python scripts/server.py test-integration`(LEXGO_TEST_DB_NAME=lexgo_test_issue5) | 全部通过:18 个顶层用例,其中 #5 新增 8 个(7 个书库/章节/任务/阅读 + 1 个 v2→v3 数据保留),另含 12 个子用例;既有 10 个用例保持通过 | +| `npx --yes pnpm@9.15.1 --dir learner test:unit --run` | 3 个文件 31 项通过(session 9、library 16、reading 6) | +| `npx --yes pnpm@9.15.1 --dir learner build` | vue-tsc 类型检查与 vite 构建通过,退出码 0 | +| `npx --yes pnpm@9.15.1 --dir learner test:e2e` | 3 项通过(既有 auth 2 项 + 新增 reading 1 项,均为虚构 API 响应) | + +### 真实 API + MySQL 实测(2026-09-10,lexgo_dev) + +使用本单新建的虚构账号 issue5_a、issue5_b,口令只保存在忽略的 .local/issue5-accounts.json;未改动 admin、dev、learner_a、learner_b。脚本 .local/verify-issue5-api.ps1 只在本机运行,不输出口令。 + +- 粘贴:HTTP 201,章节与任务均为 pending,charCount 99。 +- 处理:实测状态序列 pending → ready,约 1132 ms(后台任务每秒轮询);job attempts=1。 +- 阅读:originalText 与提交正文逐字符相等,CRLF、制表符、弯引号、破折号、省略号、é 加组合重音、emoji、行尾空格与空行全部保留;sha256 前缀 ce7357ea22a3。 +- 幂等:同一 requestId 重复提交 HTTP 200、duplicate=true、章节与任务编号不变;同一 requestId 换正文 HTTP 409;书库仍为 1 本。 +- 隔离:issue5_b 读取 issue5_a 的书籍、章节、任务以及追加、重试全部 404;请求体带 ownerId 与查询参数 ownerId 均 400;issue5_b 书库为空。 +- 追加与阅读导航:新章节 ordinal=2,处理后就绪,前后章节编号互相指向。 +- 校验:空标题、纯空白正文、非 en 语言、缺少 requestId、超过 100000 code point 分别返回 400 与可读中文提示。 +- 遗留 fixture:lexgo_dev 中 issue5_a 名下 1 本虚构书、2 个就绪章节(bookId=1,章节 1、2)。 + +浏览器实测:真实学习端 + 真实 API + 真实 MySQL 联测(临时 Playwright 用例,运行后删除):issue5_a 登录 → 书库显示既有虚构书与“导入内容”入口 → 导入页粘贴含空行、制表符、行尾空格与 emoji 的正文 → 书库页由“处理中”变为“已就绪” → 阅读页 article.reader-text 的 textContent 与粘贴正文逐字符相等、computed white-space 为 pre-wrap → “下一章”切换到第二章且正文精确相等 → 390×844 视口下横向溢出 0 px。同一轮还运行了 3 项虚构 API 的既有 e2e,共 4 项通过。 + +截图保存在本机 .local/evidence/(library.png、book-after-paste.png、reader-desktop.png、reader-mobile-390.png)并作为工单附件上传,便于人工目视复核;本次会话模型不能读取图片,截图未经 Agent 目视检查,功能断言来自上面的程序化检查。 + +未验证:处理失败到重试的用户界面路径只由集成测试覆盖(无法通过 API 主动制造处理失败);真实手机浏览器长按、手柄、滚动与虚拟键盘仍属 #4 缺口,本次只用桌面浏览器窄屏检查,不能当作真机结果;Python NLP 未接入,token、lemma 与词典仍为 #3 小样范围;生产并发、容量、备份恢复与部署不在本单范围。 + +## #5 审核整改验证(R1~R4,2026-09-11) + +整改提交与完整证据见工单 #5 的整改评论。本次复核命令与结果(仓库根执行,专用测试库 lexgo_test_issue5): + +| 命令 | 本次结果 | +|---|---| +| `python scripts/server.py test-integration` | 20 个顶层用例全部通过(新增 `TestMySQLIngestRecoveryWithoutRestart`、`TestMySQLIngestAttemptsAreBoundedAndManualRetryRestarts`) | +| `npx --yes pnpm@9.15.1 --dir learner test:unit --run` | 38 项通过(library 21、reading 8、session 9) | +| `npx --yes pnpm@9.15.1 --dir learner build` | 通过(vue-tsc + vite) | +| `npx --yes pnpm@9.15.1 --dir learner test:e2e` | 3 项通过(虚构 API 响应) | + +回归测试的有效性:新增的前端 7 项用例先在整改前的 `library.ts`/`ImportView.vue` 上运行并实际失败(追加发送 language、离页后响应写回、重试后停在失败),改回修复版本后全部通过。 + +真实联调(lexgo_dev,虚构账号 issue5_a): + +- R1 追加:真实学习端从书籍页进入“追加章节”,提交后回到书籍页,新章节就绪后可阅读;抓取到的请求体只有 requestId、title、text,无 language;正文逐字符相等。 +- R2 恢复:新建章节后用 SQL 把任务与章节置为 processing 且 updated_at 早于阈值(UTC 时间),**不重启服务**,运行期清扫在 544 ms 内把任务重新入队并发布为就绪,章节与任务编号不变,正文逐字符相等。 +- R4 重试:SQL 制造真实失败任务(content_changed)后,在浏览器点击“重试”并中断其后的第一次刷新请求,页面立即由“处理失败”变为“处理中”,随后自行变为“已就绪”,无需手工刷新。 + +注意:MySQL 会话时区为 SYSTEM(本机为 UTC+8),而服务按 UTC 存储 DATETIME;核对任务时间时使用 UTC_TIMESTAMP 而不是 NOW(),否则会出现 8 小时的假偏差。 + +未在本轮验证:R3 的真实浏览器时序(离页与响应同时发生)只由单测覆盖;真机手机证据仍属 #4 缺口。 diff --git a/docs/09-product-requirements-overview.md b/docs/09-product-requirements-overview.md index 5abbd45..be25207 100644 --- a/docs/09-product-requirements-overview.md +++ b/docs/09-product-requirements-overview.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki,禁止直接编辑本文件) wiki_page: Product-Requirements-Overview wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Product-Requirements-Overview.- -wiki_revision: db82eeb1d509b8d3c4a2204143cfb0e829b6b083 -synchronized_at: 2026-09-10T12:43:32Z +wiki_revision: 125df697d3d45dee098616d25970bb65a96727cd +synchronized_at: 2026-09-10T16:35:19Z # 产品需求总览 @@ -223,3 +223,9 @@ Anki(U20)、YouTube/Jellyfin 远程字幕(U07/A08)、全量旧实例迁 ## 试用前管理能力增补 用户于 2026-09-10 验收 #2,并批准新增 #18 登录日志与操作审计:两个管理员列表、查询筛选分页、必要字段记录、权限及 90 天保留清理。沿用现有 go-admin 管理布局,无需独立高保真原型。已通过用户验收,纳入 #16“邀请用户试用前完成”;不替代 #3 英语分词和 #4 阅读选择验证,也不引入其他 go-admin 默认模块。 + +## 实施进度增补(#5) + +2026-09-10:#5“粘贴英语文本,处理后进入本人章节阅读”已实现并待用户验收,覆盖 F01 基础(书库与章节)、F02(粘贴导入)、F04 基础(处理状态与失败重试)与 F05 原文(可读原文与章节切换)。本节取代此前“阅读、导入尚未实现”的表述:粘贴导入与原文阅读已实现;点词查词、词典、个人词语状态、复习与统计仍未实现(#6~#15)。schema 升级为 v3,新增 lexgo_books、lexgo_chapters、lexgo_ingest_jobs。 + +范围边界不变:本单按 Go 处理,不接入 Python NLP,因此不产生 token、lemma 或词典索引;Go+Python NLP 与全 Go 路线仍未确认,正式接入前必须由用户确认。真机手机详细证据仍缺失(#4 缺口保持)。 diff --git a/docs/README.md b/docs/README.md index f89ed9f..a938808 100644 --- a/docs/README.md +++ b/docs/README.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki,禁止直接编辑本文件) wiki_page: Home wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Home -wiki_revision: a645b99632c70734b493ed030ad6d4d2acfd13be -synchronized_at: 2026-09-10T14:52:14Z +wiki_revision: 5b936b75caeec89cd54f7147c842f0b66f901307 +synchronized_at: 2026-09-10T16:35:18Z # LexGo 文档入口 @@ -63,3 +63,5 @@ Quant-UX 原型 v1 已通过用户验收。[桌面预览](https://qux.ilapage.cn #4 阅读选择小样已实现桌面鼠标/键盘、范围调整与原文位置验证,入口 http://127.0.0.1:5184/;固定版本 LinguaCafe 四项源码对照已记录。用户已验收并关闭 #4;真实手机详细测试证据仍缺失,详见本地验证页。#21 仍待实施。 + +#5 粘贴导入与章节阅读已实现,待用户验收:schema v3 新增 books/chapters/ingest_jobs,学习端具备粘贴导入、书库、处理状态与原文阅读;查词、词典与复习仍未实现。本单按 Go 处理,Go+Python NLP 与全 Go 路线仍未决,正式接入前须用户确认。 diff --git a/learner/e2e/reading.spec.ts b/learner/e2e/reading.spec.ts new file mode 100644 index 0000000..84e1fdf --- /dev/null +++ b/learner/e2e/reading.spec.ts @@ -0,0 +1,92 @@ +import { expect, test } from '@playwright/test' + +test('paste English text, watch a chapter finish processing, then read it verbatim', async ({ page }) => { + const user = { id: 42, username: 'fictional-reader', role: 'learner' } + const book = { id: 1, title: '虚构样例书', language: 'en' } + const chapterTitle = '虚构样例第一章' + // Line breaks, a tab and repeated spaces must survive the whole round trip. + const pasted = 'First line of the chapter.\n\tIndented line.\nTwo spaces kept.\n\nLast line.\n' + const timestamps = { createdAt: '2026-01-01T00:00:00Z', updatedAt: '2026-01-01T00:00:00Z' } + // The worker reports the fresh chapter as processing until the worker settles it. + let status: 'processing' | 'ready' = 'processing' + const chapterPayload = () => ({ + id: 55, + bookId: book.id, + ordinal: 1, + title: chapterTitle, + status, + charCount: [...pasted].length, + errorReason: '', + errorMessage: '', + jobId: 7, + ...timestamps, + }) + + await page.route('**/api/v1/**', async route => { + const path = new URL(route.request().url()).pathname + const method = route.request().method() + let data: unknown = null + let statusCode = 200 + if (path === '/api/v1/login') { + expect(route.request().postDataJSON()).toEqual({ username: user.username, password: 'fictional-password' }) + data = { token: 'fictional-session', user } + } else if (path === '/api/v1/me') data = user + else if (path === '/api/v1/space') data = { ownerId: user.id, language: 'en' } + else if (path === '/api/v1/books' && method === 'GET') { + data = { items: [{ ...book, chapterCount: 0, pendingCount: 0, processingCount: 0, readyCount: 0, failedCount: 0, ...timestamps }] } + } else if (path === '/api/v1/books' && method === 'POST') { + const body = route.request().postDataJSON() as { requestId: string; title: string; text: string; language: string } + expect(body.requestId).toMatch(/^[0-9a-f-]{36}$/) + expect(body).toMatchObject({ title: chapterTitle, text: pasted, language: 'en' }) + statusCode = 201 + data = { + book, + chapter: chapterPayload(), + job: { id: 7, bookId: book.id, chapterId: 55, status, attempts: 0, errorReason: '', errorMessage: '', ...timestamps }, + duplicate: false, + } + } else if (path === '/api/v1/books/1') data = { book, chapters: [chapterPayload()] } + else if (path === '/api/v1/chapters/55') { + data = { + book, + chapter: { ...chapterPayload(), contentSha256: 'fictional-sha256', ...(status === 'ready' ? { originalText: pasted } : {}) }, + navigation: { previousChapterId: null, nextChapterId: null }, + } + } + await route.fulfill({ status: statusCode, json: { code: 200, data } }) + }) + + await page.goto('/') + await page.getByLabel('账号').fill(user.username) + await page.getByLabel('密码', { exact: true }).fill('fictional-password') + await page.getByRole('button', { name: '登录', exact: true }).click() + + // The library lists the caller's book. + await expect(page.getByRole('heading', { name: '我的书库' })).toBeVisible() + await expect(page.getByRole('link', { name: book.title })).toBeVisible() + + // Paste text through the import form. + await page.getByRole('button', { name: '导入内容' }).click() + await expect(page.getByRole('heading', { name: '导入英文内容' })).toBeVisible() + await page.getByLabel('标题').fill(chapterTitle) + await page.getByLabel('正文').fill(pasted) + await page.getByRole('button', { name: '开始处理' }).click() + + // The new book opens with the chapter still processing… + await expect(page).toHaveURL(/\/books\/1$/) + await expect(page.getByText('处理中')).toBeVisible() + + // …and the browser poll turns it ready without a page reload. + status = 'ready' + await expect(page.getByText('已就绪')).toBeVisible({ timeout: 15000 }) + + // Open the chapter and check the pasted text survived verbatim. + await page.getByRole('link', { name: chapterTitle }).click() + await expect(page).toHaveURL(/\/chapters\/55$/) + const readerText = page.locator('.reader-text') + await expect(readerText).toBeVisible() + expect(await readerText.evaluate(element => element.textContent)).toBe(pasted) + expect(await readerText.evaluate(element => getComputedStyle(element).whiteSpace)).toBe('pre-wrap') + await expect(page.getByRole('button', { name: '上一章' })).toBeDisabled() + await expect(page.getByRole('button', { name: '下一章' })).toBeDisabled() +}) diff --git a/learner/src/__tests__/library.spec.ts b/learner/src/__tests__/library.spec.ts new file mode 100644 index 0000000..8182398 --- /dev/null +++ b/learner/src/__tests__/library.spec.ts @@ -0,0 +1,445 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { createPinia, setActivePinia } from 'pinia' +import { + NOT_FOUND_MESSAGE, + POLL_INTERVAL_MS, + TEXT_MAX_CODE_POINTS, + canRetry, + statusSummary, + useLibraryStore, + type ChapterDetail, +} from '../stores/library' +import { useSessionStore } from '../stores/session' + +// All accounts, books and texts in these tests are deliberately fictitious. +const user = { id: 7, username: 'fictional-reader', role: 'learner' as const } +const book = { id: 1, title: '虚构样例书', language: 'en' } +const navigation = { previousChapterId: null, nextChapterId: null } + +const ok = (data: unknown) => new Response(JSON.stringify({ code: 200, data }), { status: 200 }) +const created = (data: unknown) => new Response(JSON.stringify({ code: 200, data }), { status: 201 }) +const httpError = (status: number, msg: string) => new Response(JSON.stringify({ code: status, msg }), { status }) + +const chapter = (overrides: Partial = {}): ChapterDetail => ({ + id: 55, + bookId: 1, + ordinal: 1, + title: '第一篇', + status: 'pending', + charCount: 120, + errorReason: '', + errorMessage: '', + contentSha256: 'fictional-sha256', + jobId: 7, + createdAt: '2026-01-01T00:00:00Z', + updatedAt: '2026-01-01T00:00:00Z', + ...overrides, +}) + +const summary = (overrides: Partial> = {}) => ({ + ...book, + chapterCount: 0, + pendingCount: 0, + processingCount: 0, + readyCount: 0, + failedCount: 0, + createdAt: '2026-01-01T00:00:00Z', + updatedAt: '2026-01-01T00:00:00Z', + ...overrides, +}) + +const job = { id: 7, bookId: 1, chapterId: 55, status: 'pending', attempts: 0, errorReason: '', errorMessage: '', createdAt: '2026-01-01T00:00:00Z', updatedAt: '2026-01-01T00:00:00Z' } + +const fetchMock = () => vi.mocked(globalThis.fetch) +const paths = () => fetchMock().mock.calls.map(([input]) => String(input)) +const bodyOf = (index: number): Record => JSON.parse(String(fetchMock().mock.calls[index]?.[1]?.body)) as Record + +async function signIn() { + fetchMock().mockResolvedValueOnce(ok({ token: 'fictional-token', expiresAt: '2030-01-01', user })) + const session = useSessionStore() + await session.login(user.username, 'fictional-password') + return session +} + +describe('learner library store', () => { + beforeEach(() => { + sessionStorage.clear() + setActivePinia(createPinia()) + vi.restoreAllMocks() + // Any request a test did not expect fails loudly instead of hanging. + vi.spyOn(globalThis, 'fetch').mockImplementation(input => { + throw new Error(`unexpected request: ${String(input)}`) + }) + }) + + afterEach(() => { + // Never let a polling timer outlive its test. + useLibraryStore().stopPolling() + vi.useRealTimers() + }) + + it('loads the library and summarises the count fields the API reports', async () => { + await signIn() + const library = useLibraryStore() + fetchMock().mockResolvedValueOnce(ok({ items: [summary({ chapterCount: 5, readyCount: 2, processingCount: 1, pendingCount: 1, failedCount: 1 })] })) + + await library.loadBooks() + + expect(library.books).toHaveLength(1) + expect(library.booksLoading).toBe(false) + expect(library.booksError).toBe('') + expect(statusSummary(library.books[0]!)).toBe('已就绪 2 · 处理中 1 · 待处理 1 · 失败 1') + expect(paths()).toEqual(['/api/v1/login', '/api/v1/books']) + expect(fetchMock().mock.calls[1]?.[1]?.headers).toMatchObject({ Authorization: 'Bearer fictional-token' }) + }) + + it('shows pending and processing separately instead of deriving one from a total', async () => { + // processingCount is strictly "processing" now, so pendingCount must be read as given. + const queued = summary({ chapterCount: 3, readyCount: 1, processingCount: 0, pendingCount: 2, failedCount: 0 }) + expect(statusSummary(queued)).toBe('已就绪 1 · 待处理 2') + expect(statusSummary(queued)).not.toContain('处理中') + const done = summary({ chapterCount: 1, readyCount: 1 }) + expect(statusSummary(done)).toBe('已就绪 1') + expect(statusSummary(summary({ chapterCount: 0 }))).toBe('') + }) + + it('loads a book detail whose chapters carry the job id used for retry', async () => { + await signIn() + const library = useLibraryStore() + fetchMock().mockResolvedValueOnce(ok({ book, chapters: [chapter({ status: 'failed', jobId: 7 }), chapter({ id: 56, status: 'failed', jobId: null })] })) + + await library.loadBook(1) + + expect(library.book?.title).toBe('虚构样例书') + expect(library.chapters).toHaveLength(2) + expect(paths()[1]).toBe('/api/v1/books/1') + // The chapter itself carries the job id, even for a freshly loaded book. + expect(library.chapters[0]?.jobId).toBe(7) + expect(canRetry(library.chapters[0]!)).toBe(true) + // A null job id means the chapter has nothing to retry yet. + expect(canRetry(library.chapters[1]!)).toBe(false) + }) + + it('retries a failed chapter loaded fresh from the book detail, without any submit in this session', async () => { + vi.useFakeTimers() + await signIn() + const library = useLibraryStore() + // No submit() call: this is a plain reload, the old workaround would hide retry here. + fetchMock().mockResolvedValueOnce(ok({ book, chapters: [chapter({ status: 'failed', errorMessage: '无法解析正文。', jobId: 7 })] })) + await library.loadBook(1) + + fetchMock() + .mockResolvedValueOnce(ok({ job: { ...job, status: 'pending', attempts: 1 }, chapter: { id: 55, bookId: 1, jobId: 7 } })) + .mockResolvedValueOnce(ok({ book, chapters: [chapter({ jobId: 7 })] })) + await library.retryChapter(55) + + expect(paths()).toContain('/api/v1/jobs/7/retry') + expect(library.retryingChapterId).toBeNull() + expect(library.chapters[0]?.status).toBe('pending') + }) + + it('pastes a new book and reads the job id from the created chapter', async () => { + await signIn() + const library = useLibraryStore() + fetchMock().mockResolvedValueOnce(created({ book, chapter: chapter(), job, duplicate: false })) + + const bookId = await library.submit({ title: ' 第一篇 ', text: 'Hello world.\nSecond line.', target: { mode: 'new' } }) + + expect(bookId).toBe(1) + expect(library.submitting).toBe(false) + expect(library.submitError).toBe('') + expect(paths()[1]).toBe('/api/v1/books') + expect(bodyOf(1)).toEqual({ requestId: expect.any(String), title: '第一篇', text: 'Hello world.\nSecond line.', language: 'en' }) + }) + + it('appends to an existing book through the chapter endpoint', async () => { + await signIn() + const library = useLibraryStore() + fetchMock().mockResolvedValueOnce(created({ chapter: chapter(), job, duplicate: false })) + + const bookId = await library.submit({ title: '第二篇', text: 'Another text.', target: { mode: 'append', bookId: 1 } }) + + expect(bookId).toBe(1) + expect(paths()[1]).toBe('/api/v1/books/1/chapters') + }) + + it('reuses one requestId while the same unsent content keeps failing', async () => { + await signIn() + const library = useLibraryStore() + const input = { title: '第一篇', text: 'Hello world.', target: { mode: 'new' as const } } + fetchMock() + .mockResolvedValueOnce(httpError(500, '服务器开小差了')) + .mockResolvedValueOnce(created({ book, chapter: chapter(), job, duplicate: false })) + + await expect(library.submit(input)).rejects.toThrow('服务器开小差了') + expect(library.submitError).toBe('服务器开小差了') + await library.submit(input) + + // One chapter, not two: the retry of unchanged content reuses the requestId. + expect(bodyOf(2).requestId).toBe(bodyOf(1).requestId) + }) + + it('uses a fresh requestId after a successful submit and after the content changes', async () => { + await signIn() + const library = useLibraryStore() + fetchMock() + .mockResolvedValueOnce(created({ book, chapter: chapter(), job, duplicate: false })) + .mockResolvedValueOnce(created({ book, chapter: chapter({ id: 56 }), job, duplicate: false })) + .mockResolvedValueOnce(created({ book, chapter: chapter({ id: 57 }), job, duplicate: false })) + + await library.submit({ title: '第一篇', text: 'Hello world.', target: { mode: 'new' } }) + await library.submit({ title: '第一篇', text: 'Hello world.', target: { mode: 'new' } }) + await library.submit({ title: '第一篇', text: 'Hello world changed.', target: { mode: 'new' } }) + + expect(bodyOf(2).requestId).not.toBe(bodyOf(1).requestId) + expect(bodyOf(3).requestId).not.toBe(bodyOf(2).requestId) + }) + + it('rejects invalid input before sending anything', async () => { + await signIn() + const library = useLibraryStore() + const sent = paths().length + + await expect(library.submit({ title: ' ', text: 'Hello.', target: { mode: 'new' } })).rejects.toThrow('请填写标题。') + await expect(library.submit({ title: '标题', text: ' \n\t ', target: { mode: 'new' } })).rejects.toThrow('请粘贴要导入的英文正文。') + await expect(library.submit({ title: 'x'.repeat(121), text: 'Hello.', target: { mode: 'new' } })).rejects.toThrow('标题不能超过 120 个字符。') + await expect(library.submit({ title: '标题', text: 'a'.repeat(TEXT_MAX_CODE_POINTS + 1), target: { mode: 'new' } })).rejects.toThrow(`正文不能超过 ${TEXT_MAX_CODE_POINTS} 个字符。`) + + expect(paths().length).toBe(sent) + expect(library.submitError).toBe(`正文不能超过 ${TEXT_MAX_CODE_POINTS} 个字符。`) + }) + + it('measures the text limit in Unicode code points', async () => { + await signIn() + const library = useLibraryStore() + // 100000 astral characters are 200000 UTF-16 units but still within the limit. + fetchMock().mockResolvedValueOnce(created({ book, chapter: chapter(), job, duplicate: false })) + + await expect(library.submit({ title: '标题', text: '😀'.repeat(TEXT_MAX_CODE_POINTS), target: { mode: 'new' } })).resolves.toBe(1) + expect([...('😀'.repeat(TEXT_MAX_CODE_POINTS))].length).toBe(TEXT_MAX_CODE_POINTS) + }) + + it('polls a pending chapter until it is ready and then stops', async () => { + vi.useFakeTimers() + await signIn() + const library = useLibraryStore() + fetchMock().mockResolvedValueOnce(ok({ book, chapter: chapter(), navigation })) + + await library.loadChapter(55) + expect(library.chapter?.status).toBe('pending') + expect(library.readerText).toBe('') + + fetchMock().mockResolvedValueOnce(ok({ book, chapter: chapter({ status: 'ready', originalText: 'Hello\nworld.' }), navigation })) + await vi.advanceTimersByTimeAsync(POLL_INTERVAL_MS) + + expect(library.chapter?.status).toBe('ready') + expect(library.readerText).toBe('Hello\nworld.') + + const settled = paths().length + await vi.advanceTimersByTimeAsync(POLL_INTERVAL_MS * 3) + expect(paths().length).toBe(settled) + }) + + it('retries a failed chapter through the chapter job id and resumes polling', async () => { + vi.useFakeTimers() + await signIn() + const library = useLibraryStore() + fetchMock().mockResolvedValueOnce(ok({ book, chapter: chapter({ status: 'failed', errorReason: 'decode_error', errorMessage: '无法解析正文。', jobId: 7 }), navigation })) + + await library.loadChapter(55) + expect(library.chapter?.status).toBe('failed') + expect(canRetry(library.chapter!)).toBe(true) + + fetchMock() + .mockResolvedValueOnce(ok({ job: { ...job, status: 'pending', attempts: 1 }, chapter: { id: 55, bookId: 1, jobId: 7 } })) + .mockResolvedValueOnce(ok({ book, chapter: chapter(), navigation })) + await library.retryChapter(55) + + expect(paths()).toContain('/api/v1/jobs/7/retry') + expect(library.retryingChapterId).toBeNull() + expect(library.chapter?.status).toBe('pending') + + // The retry restarts polling for the chapter it re-queued. + const before = paths().length + fetchMock().mockResolvedValueOnce(ok({ book, chapter: chapter({ status: 'ready', originalText: 'Hello world.' }), navigation })) + await vi.advanceTimersByTimeAsync(POLL_INTERVAL_MS) + expect(paths().length).toBe(before + 1) + expect(library.readerText).toBe('Hello world.') + }) + + it('refuses to retry a chapter whose job id is null', async () => { + await signIn() + const library = useLibraryStore() + fetchMock().mockResolvedValueOnce(ok({ book, chapter: chapter({ status: 'failed', jobId: null }), navigation })) + await library.loadChapter(55) + + expect(canRetry(library.chapter!)).toBe(false) + await expect(library.retryChapter(55)).rejects.toThrow('这一章暂时没有可重试的任务编号。') + expect(paths()).toEqual(['/api/v1/login', '/api/v1/chapters/55']) + }) + + it('reports another account id as 内容不存在 and stops polling for it', async () => { + vi.useFakeTimers() + await signIn() + const library = useLibraryStore() + fetchMock().mockResolvedValueOnce(httpError(404, 'chapter not found')) + + await library.loadChapter(99) + expect(library.chapter).toBeNull() + expect(library.chapterError).toBe(NOT_FOUND_MESSAGE) + + fetchMock().mockResolvedValueOnce(httpError(404, 'book not found')) + await library.loadBook(99) + expect(library.book).toBeNull() + expect(library.bookError).toBe(NOT_FOUND_MESSAGE) + + const settled = paths().length + await vi.advanceTimersByTimeAsync(POLL_INTERVAL_MS * 4) + expect(paths().length).toBe(settled) + }) + + it('stops polling and ignores a late response once the session is cleared', async () => { + vi.useFakeTimers() + const session = await signIn() + const library = useLibraryStore() + fetchMock().mockResolvedValueOnce(ok({ book, chapter: chapter(), navigation })) + await library.loadChapter(55) + expect(library.chapter).not.toBeNull() + + let finish!: (response: Response) => void + fetchMock().mockImplementationOnce(() => new Promise(resolve => { finish = resolve })) + fetchMock().mockResolvedValueOnce(ok(null)) + const late = library.loadChapter(55, { silent: true }) + const logout = session.logout() + + finish(ok({ book, chapter: chapter({ status: 'ready', originalText: 'Late text.' }), navigation })) + await late + await logout + + expect(session.user).toBeNull() + expect(library.chapter).toBeNull() + expect(library.readerText).toBe('') + + const settled = paths().length + await vi.advanceTimersByTimeAsync(POLL_INTERVAL_MS * 4) + expect(paths().length).toBe(settled) + }) + + it('keeps newer state when an older request answers later', async () => { + await signIn() + const library = useLibraryStore() + let finish!: (response: Response) => void + fetchMock().mockImplementationOnce(() => new Promise(resolve => { finish = resolve })) + fetchMock().mockResolvedValueOnce(ok({ items: [summary({ title: '较新的标题' })] })) + + const stale = library.loadBooks() + await library.loadBooks() + finish(ok({ items: [summary({ title: '过期的标题' })] })) + await stale + + expect(library.books).toHaveLength(1) + expect(library.books[0]?.title).toBe('较新的标题') + }) + + // Regression R1: the append contract has no language field and the server rejects unknown + // fields, so a client that sent one could never append. + it('sends the language only when creating a book, never when appending', async () => { + await signIn() + const library = useLibraryStore() + fetchMock().mockResolvedValueOnce(created({ book, chapter: chapter({ id: 54 }) })) + await library.submit({ title: ' 新书 ', text: 'New book text.', target: { mode: 'new' } }) + expect(bodyOf(1)).toEqual({ requestId: expect.any(String), title: '新书', text: 'New book text.', language: 'en' }) + + fetchMock().mockResolvedValueOnce(created({ chapter: chapter({ id: 55, ordinal: 2 }) })) + await library.submit({ title: '第二篇', text: 'Appended text.', target: { mode: 'append', bookId: 1 } }) + expect(String(fetchMock().mock.calls[2]?.[0])).toBe('/api/v1/books/1/chapters') + expect(bodyOf(2)).toEqual({ requestId: expect.any(String), title: '第二篇', text: 'Appended text.' }) + expect(bodyOf(2)).not.toHaveProperty('language') + }) + + // Regression R3: leaving a view must invalidate its in-flight request. + it('ignores a book response that arrives after the book view was closed', async () => { + vi.useFakeTimers() + await signIn() + const library = useLibraryStore() + let finish!: (response: Response) => void + fetchMock().mockImplementationOnce(() => new Promise(resolve => { finish = resolve })) + + const pending = library.loadBook(1) + library.closeBook() + finish(ok({ book, chapters: [chapter()] })) + await pending + + expect(library.book).toBeNull() + expect(library.chapters).toEqual([]) + expect(library.bookLoading).toBe(false) + const settled = paths().length + await vi.advanceTimersByTimeAsync(POLL_INTERVAL_MS * 3) + expect(paths().length).toBe(settled) + }) + + it('ignores a chapter response that arrives after the reader was closed', async () => { + vi.useFakeTimers() + await signIn() + const library = useLibraryStore() + let finish!: (response: Response) => void + fetchMock().mockImplementationOnce(() => new Promise(resolve => { finish = resolve })) + + const pending = library.loadChapter(55) + library.closeChapter() + finish(ok({ book, chapter: chapter({ status: 'ready', originalText: 'Late text.' }), navigation })) + await pending + + expect(library.chapter).toBeNull() + expect(library.readerText).toBe('') + expect(library.chapterLoading).toBe(false) + const settled = paths().length + await vi.advanceTimersByTimeAsync(POLL_INTERVAL_MS * 3) + expect(paths().length).toBe(settled) + }) + + // Regression R4: an accepted retry must be visible and tracked even if the refresh fails. + it('keeps tracking a retried chapter when the first refresh fails', async () => { + vi.useFakeTimers() + await signIn() + const library = useLibraryStore() + fetchMock().mockResolvedValueOnce(ok({ book, chapter: chapter({ status: 'failed', errorReason: 'content_changed', errorMessage: '内容在处理前发生变化。', jobId: 7 }), navigation })) + await library.loadChapter(55) + expect(library.chapter?.status).toBe('failed') + + fetchMock() + .mockResolvedValueOnce(ok({ job: { ...job, status: 'pending' }, chapter: chapter({ status: 'pending' }) })) + .mockRejectedValueOnce(new Error('network down')) + await library.retryChapter(55) + + expect(paths()).toContain('/api/v1/jobs/7/retry') + expect(library.chapter?.status).toBe('pending') + expect(library.readerText).toBe('') + + // The next poll still tracks the queued chapter and shows the final result. + fetchMock().mockResolvedValueOnce(ok({ book, chapter: chapter({ status: 'ready', originalText: 'Recovered text.' }), navigation })) + await vi.advanceTimersByTimeAsync(POLL_INTERVAL_MS) + expect(library.chapter?.status).toBe('ready') + expect(library.readerText).toBe('Recovered text.') + }) + + it('keeps tracking a retried chapter from the book page when the first refresh fails', async () => { + vi.useFakeTimers() + await signIn() + const library = useLibraryStore() + fetchMock().mockResolvedValueOnce(ok({ book, chapters: [chapter({ status: 'failed', errorReason: 'content_changed', errorMessage: '内容在处理前发生变化。' })] })) + await library.loadBook(1) + expect(library.chapters[0]?.status).toBe('failed') + + fetchMock() + .mockResolvedValueOnce(ok({ job: { ...job, status: 'pending' }, chapter: chapter({ status: 'pending' }) })) + .mockRejectedValueOnce(new Error('network down')) + await library.retryChapter(55) + + expect(library.chapters[0]?.status).toBe('pending') + + fetchMock().mockResolvedValueOnce(ok({ book, chapters: [chapter({ status: 'ready' })] })) + await vi.advanceTimersByTimeAsync(POLL_INTERVAL_MS) + expect(library.chapters[0]?.status).toBe('ready') + }) +}) diff --git a/learner/src/__tests__/reading.spec.ts b/learner/src/__tests__/reading.spec.ts new file mode 100644 index 0000000..0fae0bc --- /dev/null +++ b/learner/src/__tests__/reading.spec.ts @@ -0,0 +1,265 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { flushPromises, mount } from '@vue/test-utils' +import { createPinia, setActivePinia } from 'pinia' +import { createMemoryHistory, createRouter, type Router } from 'vue-router' +import { defineComponent, h } from 'vue' +import BookView from '../views/BookView.vue' +import ImportView from '../views/ImportView.vue' +import ReaderView from '../views/ReaderView.vue' +import { useLibraryStore, type ChapterStatus } from '../stores/library' +import { useSessionStore } from '../stores/session' + +// All accounts, books and texts in these tests are deliberately fictitious. +const user = { id: 42, username: 'fictional-reader', role: 'learner' as const } +const book = { id: 1, title: '虚构样例书', language: 'en' } +const navigation = { previousChapterId: null, nextChapterId: null } +const pasted = 'First line.\n\tIndented line.\nTwo spaces kept.\n\nLast line.\n' + +const ok = (data: unknown) => new Response(JSON.stringify({ code: 200, data }), { status: 200 }) +const created = (data: unknown) => new Response(JSON.stringify({ code: 200, data }), { status: 201 }) + +function chapter(status: ChapterStatus, extra: Record = {}) { + return { + id: 55, + bookId: 1, + ordinal: 1, + title: '第一篇', + status, + charCount: 120, + errorReason: '', + errorMessage: '', + jobId: 7, + createdAt: '2026-01-01T00:00:00Z', + updatedAt: '2026-01-01T00:00:00Z', + ...extra, + } +} + +const stub = (name: string) => defineComponent({ name, render: () => h('div') }) + +async function viewAt(path: string): Promise { + const router = createRouter({ + history: createMemoryHistory(), + routes: [ + { path: '/', component: stub('LibraryStub') }, + { path: '/import', component: stub('ImportStub') }, + { path: '/books/:id', component: stub('BookStub') }, + { path: '/chapters/:id', component: stub('ChapterStub') }, + ], + }) + await router.push(path) + await router.isReady() + return router +} + +function signIn() { + useSessionStore().user = { ...user } +} + +describe('learner reading views', () => { + beforeEach(() => { + sessionStorage.clear() + setActivePinia(createPinia()) + vi.restoreAllMocks() + }) + + afterEach(() => { + useLibraryStore().stopPolling() + }) + + it('shows inline validation and sends nothing for an empty import form', async () => { + const fetchMock = vi.spyOn(globalThis, 'fetch') + signIn() + const router = await viewAt('/import') + const wrapper = mount(ImportView, { global: { plugins: [router] } }) + await flushPromises() + + await wrapper.find('form').trigger('submit') + expect(wrapper.text()).toContain('请填写标题。') + expect(wrapper.text()).toContain('请粘贴要导入的英文正文。') + expect(fetchMock).not.toHaveBeenCalled() + + await wrapper.find('input#title').setValue(' ') + await wrapper.find('textarea#text').setValue(' \n\t ') + await wrapper.find('form').trigger('submit') + expect(wrapper.text()).toContain('请填写标题。') + expect(wrapper.text()).toContain('请粘贴要导入的英文正文。') + + await wrapper.find('input#title').setValue('虚构样例第一章') + await wrapper.find('form').trigger('submit') + expect(wrapper.text()).not.toContain('请填写标题。') + expect(wrapper.text()).toContain('请粘贴要导入的英文正文。') + expect(fetchMock).not.toHaveBeenCalled() + + wrapper.unmount() + }) + + it('submits valid pasted text and opens the created book', async () => { + const fetchMock = vi.spyOn(globalThis, 'fetch') + .mockResolvedValueOnce(created({ book, chapter: chapter('pending'), job: { id: 7 }, duplicate: false })) + signIn() + const router = await viewAt('/import') + const wrapper = mount(ImportView, { global: { plugins: [router] } }) + // Element Plus assigns the input ids on mount, so wait for the first update. + await flushPromises() + + await wrapper.find('input#title').setValue('虚构样例第一章') + await wrapper.find('textarea#text').setValue(pasted) + await wrapper.find('form').trigger('submit') + await flushPromises() + + expect(fetchMock).toHaveBeenCalledTimes(1) + expect(JSON.parse(String(fetchMock.mock.calls[0]?.[1]?.body))).toMatchObject({ title: '虚构样例第一章', text: pasted, language: 'en' }) + expect(router.currentRoute.value.path).toBe('/books/1') + wrapper.unmount() + }) + + it('renders a ready chapter verbatim, keeping line breaks, tabs and repeated spaces', async () => { + vi.spyOn(globalThis, 'fetch').mockResolvedValue(ok({ book, chapter: chapter('ready', { originalText: pasted, contentSha256: 'fictional' }), navigation })) + signIn() + const router = await viewAt('/chapters/55') + const wrapper = mount(ReaderView, { global: { plugins: [router] } }) + await flushPromises() + + const article = wrapper.find('.reader-text') + expect(article.exists()).toBe(true) + expect(article.element.textContent).toBe(pasted) + expect(wrapper.text()).toContain('已就绪') + wrapper.unmount() + }) + + it('shows the failure message of a failed chapter and no text', async () => { + vi.spyOn(globalThis, 'fetch').mockResolvedValue(ok({ book, chapter: chapter('failed', { errorReason: 'decode_error', errorMessage: '无法解析正文,请检查编码。', jobId: null }), navigation })) + signIn() + const router = await viewAt('/chapters/55') + const wrapper = mount(ReaderView, { global: { plugins: [router] } }) + await flushPromises() + + expect(wrapper.text()).toContain('处理失败') + expect(wrapper.text()).toContain('无法解析正文,请检查编码。') + expect(wrapper.find('.reader-text').exists()).toBe(false) + // A null job id means there is nothing to retry yet. + expect(wrapper.text()).toContain('这一章暂时没有可重试的任务编号。') + wrapper.unmount() + }) + + it('offers retry on a failed chapter through its own job id', async () => { + const fetchMock = vi.spyOn(globalThis, 'fetch').mockResolvedValue(ok({ book, chapter: chapter('failed', { errorMessage: '解析失败。', jobId: 7 }), navigation })) + signIn() + const router = await viewAt('/chapters/55') + const wrapper = mount(ReaderView, { global: { plugins: [router] } }) + await flushPromises() + + expect(wrapper.text()).toContain('解析失败。') + const retryButton = wrapper.get('.notice .el-button') + expect(retryButton.text()).toContain('重试处理') + await retryButton.trigger('click') + await flushPromises() + + // No prior submit in this session: the job id comes from the chapter payload. + const retryCall = fetchMock.mock.calls.find(([input]) => String(input).endsWith('/jobs/7/retry')) + expect(retryCall?.[1]?.method).toBe('POST') + wrapper.unmount() + }) + + it('lists chapters with their status labels and links only ready chapters', async () => { + const chapters = [ + chapter('pending', { id: 55, ordinal: 1, title: '第一篇', jobId: 7 }), + chapter('processing', { id: 56, ordinal: 2, title: '第二篇', jobId: 8 }), + chapter('ready', { id: 57, ordinal: 3, title: '第三篇', jobId: 9 }), + chapter('failed', { id: 58, ordinal: 4, title: '第四篇', errorMessage: '解析失败。', jobId: null }), + chapter('failed', { id: 59, ordinal: 5, title: '第五篇', errorMessage: '编码错误。', jobId: 10 }), + ] + const fetchMock = vi.spyOn(globalThis, 'fetch').mockImplementation(input => { + if (String(input).endsWith('/jobs/10/retry')) { + return Promise.resolve(ok({ job: { id: 10, status: 'pending', attempts: 1 }, chapter: { id: 59, bookId: 1, jobId: 10 } })) + } + return Promise.resolve(ok({ book, chapters })) + }) + signIn() + const router = await viewAt('/books/1') + const wrapper = mount(BookView, { global: { plugins: [router] } }) + await flushPromises() + + const text = wrapper.text() + expect(text).toContain('待处理') + expect(text).toContain('处理中') + expect(text).toContain('已就绪') + expect(text).toContain('处理失败') + expect(text).toContain('解析失败。') + expect(text).toContain('编码错误。') + expect(wrapper.find('a[href="/chapters/57"]').exists()).toBe(true) + expect(wrapper.find('a[href="/chapters/55"]').exists()).toBe(false) + + // Retry is offered only for the failed chapter that carries a job id. + const retryButtons = wrapper.findAll('.chapter-row .el-button').filter(button => button.text().includes('重试')) + expect(retryButtons).toHaveLength(1) + await retryButtons[0]!.trigger('click') + await flushPromises() + const retryCall = fetchMock.mock.calls.find(([input]) => String(input).endsWith('/jobs/10/retry')) + expect(retryCall?.[1]?.method).toBe('POST') + wrapper.unmount() + }) + + // Regression R1: appending must not send the language field that the append contract rejects. + it('appends to an existing book through the chapter endpoint without a language field', async () => { + const fetchMock = vi.spyOn(globalThis, 'fetch') + .mockResolvedValueOnce(ok({ + items: [{ + ...book, + chapterCount: 1, + pendingCount: 0, + processingCount: 0, + readyCount: 1, + failedCount: 0, + createdAt: '2026-01-01T00:00:00Z', + updatedAt: '2026-01-01T00:00:00Z', + }], + })) + .mockResolvedValueOnce(created({ chapter: chapter('pending', { id: 56, ordinal: 2 }), job: { id: 8 } })) + signIn() + const router = await viewAt('/import?book=1') + const wrapper = mount(ImportView, { global: { plugins: [router] } }) + await flushPromises() + + await wrapper.find('input#title').setValue('第二篇') + await wrapper.find('textarea#text').setValue(pasted) + await wrapper.find('form').trigger('submit') + await flushPromises() + + const appendCall = fetchMock.mock.calls.find(([input]) => String(input).endsWith('/books/1/chapters')) + expect(appendCall?.[1]?.method).toBe('POST') + const body = JSON.parse(String(appendCall?.[1]?.body)) as Record + expect(body).toEqual({ requestId: expect.any(String), title: '第二篇', text: pasted }) + expect(body).not.toHaveProperty('language') + expect(router.currentRoute.value.path).toBe('/books/1') + wrapper.unmount() + }) + + // Regression R3: a submit that finishes after the user left the page must not navigate back. + it('does not navigate after the user left the import page during a submit', async () => { + let finish!: (response: Response) => void + const fetchMock = vi.spyOn(globalThis, 'fetch') + .mockImplementationOnce(() => new Promise(resolve => { finish = resolve })) + signIn() + const router = await viewAt('/import') + const wrapper = mount(ImportView, { global: { plugins: [router] } }) + await flushPromises() + + await wrapper.find('input#title').setValue('虚构样例第一章') + await wrapper.find('textarea#text').setValue(pasted) + await wrapper.find('form').trigger('submit') + await flushPromises() + expect(fetchMock).toHaveBeenCalledTimes(1) + + // The user leaves the page while the request is still open. + wrapper.unmount() + await router.push('/') + await flushPromises() + + finish(created({ book, chapter: chapter('pending'), job: { id: 7 } })) + await flushPromises() + + expect(router.currentRoute.value.path).toBe('/') + }) +}) diff --git a/learner/src/main.ts b/learner/src/main.ts index 7149fd2..f58ee80 100644 --- a/learner/src/main.ts +++ b/learner/src/main.ts @@ -4,6 +4,10 @@ import App from './App.vue' import router from './router' import 'element-plus/es/components/button/style/css' import 'element-plus/es/components/input/style/css' +import 'element-plus/es/components/radio/style/css' +import 'element-plus/es/components/radio-group/style/css' +import 'element-plus/es/components/select/style/css' +import 'element-plus/es/components/option/style/css' import './style.css' createApp(App).use(createPinia()).use(router).mount('#app') diff --git a/learner/src/router/index.ts b/learner/src/router/index.ts index 9707d56..5ee2cb1 100644 --- a/learner/src/router/index.ts +++ b/learner/src/router/index.ts @@ -6,6 +6,9 @@ const router = createRouter({ routes: [ { path: '/login', name: 'login', component: () => import('../views/LoginView.vue') }, { path: '/', name: 'library', meta: { private: true }, component: () => import('../views/LibraryView.vue') }, + { path: '/import', name: 'import', meta: { private: true }, component: () => import('../views/ImportView.vue') }, + { path: '/books/:id', name: 'book', meta: { private: true }, component: () => import('../views/BookView.vue') }, + { path: '/chapters/:id', name: 'chapter', meta: { private: true }, component: () => import('../views/ReaderView.vue') }, { path: '/:pathMatch(.*)*', redirect: '/' }, ], }) diff --git a/learner/src/stores/library.ts b/learner/src/stores/library.ts new file mode 100644 index 0000000..12e363f --- /dev/null +++ b/learner/src/stores/library.ts @@ -0,0 +1,467 @@ +import { defineStore } from 'pinia' +import { computed, ref, watch } from 'vue' +import { ApiError, useSessionStore } from './session' + +export type ChapterStatus = 'pending' | 'processing' | 'ready' | 'failed' + +export interface BookRef { id: number; title: string; language: string } + +export interface BookSummary extends BookRef { + chapterCount: number + pendingCount: number + processingCount: number + readyCount: number + failedCount: number + createdAt: string + updatedAt: string +} + +export interface ChapterSummary { + id: number + bookId: number + ordinal: number + title: string + status: ChapterStatus + charCount: number + errorReason: string + errorMessage: string + // Present wherever a chapter appears; null while the job id is unknown. + jobId: number | null + createdAt: string + updatedAt: string +} + +export interface ChapterDetail extends ChapterSummary { + contentSha256: string + // Present only for ready chapters; never cached or faked for other statuses. + originalText?: string +} + +export interface Job { + id: number + bookId: number + chapterId: number + status: ChapterStatus + attempts: number + errorReason: string + errorMessage: string + createdAt: string + updatedAt: string +} + +export interface ChapterNavigation { previousChapterId: number | null; nextChapterId: number | null } + +export type SubmitTarget = { mode: 'new' } | { mode: 'append'; bookId: number } + +export interface SubmitInput { title: string; text: string; target: SubmitTarget } + +export const POLL_INTERVAL_MS = 1500 +export const TITLE_MAX_LENGTH = 120 +export const TEXT_MAX_CODE_POINTS = 100000 +export const NOT_FOUND_MESSAGE = '内容不存在。' +export const LANGUAGE_LABEL = '英语' +export const LANGUAGE_CODE = 'en' + +const CHAPTER_STATUS_LABELS: Record = { + pending: '待处理', + processing: '处理中', + ready: '已就绪', + failed: '处理失败', +} + +export function statusLabel(status: ChapterStatus): string { + return CHAPTER_STATUS_LABELS[status] +} + +function isUnsettled(status: ChapterStatus): boolean { + return status === 'pending' || status === 'processing' +} + +/** Mirrors the server rule: non-empty after trim and at most 120 characters. */ +export function titleProblem(title: string): string { + const trimmed = title.trim() + if (!trimmed) return '请填写标题。' + if ([...trimmed].length > TITLE_MAX_LENGTH) return `标题不能超过 ${TITLE_MAX_LENGTH} 个字符。` + return '' +} + +/** Mirrors the server rule: at least one non-whitespace character, at most 100000 code points. */ +export function textProblem(text: string): string { + if (!text.trim()) return '请粘贴要导入的英文正文。' + if ([...text].length > TEXT_MAX_CODE_POINTS) return `正文不能超过 ${TEXT_MAX_CODE_POINTS} 个字符。` + return '' +} + +/** A failed chapter can be retried as soon as the API told us its job id. */ +export function canRetry(chapter: Pick): boolean { + return chapter.status === 'failed' && chapter.jobId !== null +} + +/** Compact one-line status summary for a book card, e.g. `已就绪 2 · 处理中 1 · 待处理 1 · 失败 1`. */ +export function statusSummary(book: BookSummary): string { + const parts: string[] = [] + if (book.readyCount > 0) parts.push(`已就绪 ${book.readyCount}`) + if (book.processingCount > 0) parts.push(`处理中 ${book.processingCount}`) + if (book.pendingCount > 0) parts.push(`待处理 ${book.pendingCount}`) + if (book.failedCount > 0) parts.push(`失败 ${book.failedCount}`) + return parts.join(' · ') +} + +interface LoadOptions { silent?: boolean } +interface SubmitBookBody { requestId: string; title: string; text: string; language: 'en' } +// An appended chapter owns the language of its book, so the append contract has no language +// field; the server rejects unknown fields, and a client that sends one gets HTTP 400. +interface SubmitChapterBody { requestId: string; title: string; text: string } +interface Created { bookId: number; chapter: ChapterSummary } + +function emptyNavigation(): ChapterNavigation { + return { previousChapterId: null, nextChapterId: null } +} + +export const useLibraryStore = defineStore('library', () => { + const session = useSessionStore() + + const books = ref([]) + const booksLoading = ref(false) + const booksError = ref('') + + const book = ref(null) + const chapters = ref([]) + const bookLoading = ref(false) + const bookError = ref('') + + const chapter = ref(null) + const chapterBook = ref(null) + const navigation = ref(emptyNavigation()) + const chapterLoading = ref(false) + const chapterError = ref('') + + const submitting = ref(false) + const submitError = ref('') + const retryingChapterId = ref(null) + + /** Reader text exists only for ready chapters and is never taken from a cache. */ + const readerText = computed(() => (chapter.value?.status === 'ready' ? chapter.value.originalText ?? '' : '')) + + // Every request is tagged with a generation and the owning account so that a + // late response can never repopulate the view after logout or an account switch. + let generation = 0 + let booksSeq = 0 + let bookSeq = 0 + let chapterSeq = 0 + let pollTimer: number | undefined + // One requestId per unsent form content: a double click or a repeat submit of + // unchanged content must create one chapter, not two. + let submissionKey = '' + let submissionRequestId = '' + + watch(() => session.user?.id ?? null, (next, previous) => { + if (next !== previous) reset() + }, { flush: 'sync' }) + + function ownerId(): number | null { + return session.user?.id ?? null + } + + function isStale(version: number, owner: number | null): boolean { + return version !== generation || ownerId() !== owner + } + + function failureMessage(reason: unknown, fallback: string): string { + return reason instanceof Error && reason.message ? reason.message : fallback + } + + function isNotFound(reason: unknown): boolean { + return reason instanceof ApiError && reason.status === 404 + } + + function stopPolling(): void { + if (pollTimer !== undefined) { + window.clearTimeout(pollTimer) + pollTimer = undefined + } + } + + function needsPolling(): boolean { + return (book.value !== null && chapters.value.some(item => isUnsettled(item.status))) + || (chapter.value !== null && isUnsettled(chapter.value.status)) + } + + /** Schedules the next refresh, or stops polling when nothing is pending anymore. */ + function schedulePolling(): void { + if (!needsPolling()) { + stopPolling() + return + } + if (pollTimer !== undefined || !session.user) return + pollTimer = window.setTimeout(() => { + pollTimer = undefined + void poll() + }, POLL_INTERVAL_MS) + } + + async function poll(): Promise { + if (!session.user) { + stopPolling() + return + } + const version = generation + const owner = ownerId() + const bookId = book.value !== null && chapters.value.some(item => isUnsettled(item.status)) ? book.value.id : null + const chapterId = chapter.value !== null && isUnsettled(chapter.value.status) ? chapter.value.id : null + if (bookId !== null) await loadBook(bookId, { silent: true }) + if (chapterId !== null) await loadChapter(chapterId, { silent: true }) + if (isStale(version, owner)) { + stopPolling() + return + } + schedulePolling() + } + + async function loadBooks(): Promise { + const version = generation + const owner = ownerId() + const seq = ++booksSeq + booksLoading.value = true + booksError.value = '' + try { + const result = await session.request<{ items?: BookSummary[] } | null>('books') + if (seq !== booksSeq || isStale(version, owner)) return + const items = result?.items + books.value = Array.isArray(items) ? items : [] + } catch (reason) { + if (seq !== booksSeq || isStale(version, owner)) return + booksError.value = failureMessage(reason, '书库暂时无法加载,请稍后重试。') + } finally { + if (seq === booksSeq && !isStale(version, owner)) booksLoading.value = false + } + } + + async function loadBook(id: number, options: LoadOptions = {}): Promise { + const version = generation + const owner = ownerId() + const seq = ++bookSeq + if (!options.silent) { + bookLoading.value = true + bookError.value = '' + } + try { + const result = await session.request<{ book: BookRef; chapters?: ChapterSummary[] }>(`books/${id}`) + if (seq !== bookSeq || isStale(version, owner)) return + book.value = result.book + chapters.value = Array.isArray(result.chapters) ? result.chapters : [] + bookError.value = '' + schedulePolling() + } catch (reason) { + if (seq !== bookSeq || isStale(version, owner)) return + if (isNotFound(reason)) { + // Another account's id never resolves for this caller: report it and stop + // instead of polling a resource that will not appear. + book.value = null + chapters.value = [] + bookError.value = NOT_FOUND_MESSAGE + stopPolling() + return + } + // A failed background refresh keeps the data already on screen; the next + // tick tries again and the user still sees the last known state. + if (!options.silent) bookError.value = failureMessage(reason, '书籍暂时无法加载,请稍后重试。') + } finally { + if (seq === bookSeq && !isStale(version, owner)) bookLoading.value = false + } + } + + async function loadChapter(id: number, options: LoadOptions = {}): Promise { + const version = generation + const owner = ownerId() + const seq = ++chapterSeq + if (!options.silent) { + chapterLoading.value = true + chapterError.value = '' + // Never keep the previous chapter's text under a new chapter id. + if (chapter.value !== null && chapter.value.id !== id) { + chapter.value = null + chapterBook.value = null + navigation.value = emptyNavigation() + } + } + try { + const result = await session.request<{ book: BookRef; chapter: ChapterDetail; navigation?: ChapterNavigation }>(`chapters/${id}`) + if (seq !== chapterSeq || isStale(version, owner)) return + chapterBook.value = result.book + chapter.value = result.chapter + navigation.value = result.navigation ?? emptyNavigation() + chapterError.value = '' + schedulePolling() + } catch (reason) { + if (seq !== chapterSeq || isStale(version, owner)) return + if (isNotFound(reason)) { + chapter.value = null + chapterBook.value = null + navigation.value = emptyNavigation() + chapterError.value = NOT_FOUND_MESSAGE + stopPolling() + return + } + if (!options.silent) chapterError.value = failureMessage(reason, '章节暂时无法加载,请稍后重试。') + } finally { + if (seq === chapterSeq && !isStale(version, owner)) chapterLoading.value = false + } + } + + function submissionKeyOf(target: SubmitTarget, title: string, text: string): string { + return target.mode === 'new' ? `new\n${title}\n${text}` : `append:${target.bookId}\n${title}\n${text}` + } + + async function createBook(body: SubmitBookBody): Promise { + const result = await session.request<{ book: BookRef; chapter: ChapterSummary }>('books', 'POST', body) + return { bookId: result.book.id, chapter: result.chapter } + } + + async function appendChapter(bookId: number, body: SubmitChapterBody): Promise { + const result = await session.request<{ chapter: ChapterSummary }>(`books/${bookId}/chapters`, 'POST', body) + return { bookId: result.chapter.bookId, chapter: result.chapter } + } + + /** + * Submits pasted text. Returns the book id to open on success and throws on + * failure; `submitError` always carries the message shown to the user. + */ + async function submit(input: SubmitInput): Promise { + const title = input.title.trim() + const text = input.text + const problem = titleProblem(title) || textProblem(text) + if (problem) { + submitError.value = problem + throw new Error(problem) + } + + const key = submissionKeyOf(input.target, title, text) + if (key !== submissionKey || submissionRequestId === '') { + submissionKey = key + submissionRequestId = crypto.randomUUID() + } + const requestId = submissionRequestId + + const version = generation + const owner = ownerId() + submitting.value = true + submitError.value = '' + try { + // Only the new-book contract carries a language; appending inherits the book's language. + const created = input.target.mode === 'new' + ? await createBook({ requestId, title, text, language: LANGUAGE_CODE }) + : await appendChapter(input.target.bookId, { requestId, title, text }) + if (isStale(version, owner)) throw new Error('登录状态已变化,请重新提交。') + // The content was accepted; a later submit must use a fresh requestId. + submissionKey = '' + submissionRequestId = '' + return created.bookId + } catch (reason) { + if (!isStale(version, owner)) submitError.value = failureMessage(reason, '提交失败,请稍后重试。') + throw reason instanceof Error ? reason : new Error('提交失败,请稍后重试。') + } finally { + if (!isStale(version, owner)) submitting.value = false + } + } + + /** The job id comes from the chapter itself, wherever that chapter was loaded from. */ + function jobIdOf(chapterId: number): number | null { + const target = chapters.value.find(item => item.id === chapterId) + ?? (chapter.value?.id === chapterId ? chapter.value : null) + return target?.jobId ?? null + } + + /** + * Applies a chapter summary coming from any response to the chapter list entry and to the + * open reader, so a queued chapter is never displayed with the state or text it had before. + */ + function applyChapterSummary(summary: ChapterSummary): void { + const index = chapters.value.findIndex(item => item.id === summary.id) + if (index >= 0) chapters.value[index] = { ...chapters.value[index], ...summary } + if (chapter.value !== null && chapter.value.id === summary.id) { + const originalText = summary.status === 'ready' ? chapter.value.originalText : undefined + chapter.value = { ...chapter.value, ...summary, originalText } + } + } + + async function retryChapter(chapterId: number): Promise { + const jobId = jobIdOf(chapterId) + if (jobId === null) throw new Error('这一章暂时没有可重试的任务编号。') + const version = generation + const owner = ownerId() + retryingChapterId.value = chapterId + try { + const result = await session.request<{ job: Job; chapter: ChapterSummary }>(`jobs/${jobId}/retry`, 'POST') + if (isStale(version, owner)) return + // The retry is accepted, so show the queued chapter and keep tracking it even if the + // refresh below fails: a failed silent refresh must not freeze the view on the old error. + applyChapterSummary(result.chapter) + schedulePolling() + // Refresh whatever is on screen to pick up the newest job state. + if (book.value !== null && book.value.id === result.chapter.bookId) await loadBook(result.chapter.bookId, { silent: true }) + if (chapter.value !== null && chapter.value.id === result.chapter.id) await loadChapter(result.chapter.id, { silent: true }) + } catch (reason) { + if (isStale(version, owner)) return + throw reason instanceof Error ? reason : new Error('重试失败,请稍后重试。') + } finally { + if (!isStale(version, owner)) retryingChapterId.value = null + } + } + + /** + * Releases the book view so polling stops when the page is left. The request sequence is + * advanced first, so a response that arrives after this call cannot repopulate the view or + * restart polling for a page the user already left. + */ + function closeBook(): void { + bookSeq++ + book.value = null + chapters.value = [] + bookLoading.value = false + bookError.value = '' + schedulePolling() + } + + /** Releases the reader view, invalidating in-flight loads the same way. */ + function closeChapter(): void { + chapterSeq++ + chapter.value = null + chapterBook.value = null + navigation.value = emptyNavigation() + chapterLoading.value = false + chapterError.value = '' + schedulePolling() + } + + function reset(): void { + generation++ + stopPolling() + books.value = [] + booksLoading.value = false + booksError.value = '' + book.value = null + chapters.value = [] + bookLoading.value = false + bookError.value = '' + chapter.value = null + chapterBook.value = null + navigation.value = emptyNavigation() + chapterLoading.value = false + chapterError.value = '' + submitting.value = false + submitError.value = '' + retryingChapterId.value = null + submissionKey = '' + submissionRequestId = '' + } + + return { + books, booksLoading, booksError, + book, chapters, bookLoading, bookError, + chapter, chapterBook, navigation, chapterLoading, chapterError, + submitting, submitError, retryingChapterId, readerText, + loadBooks, loadBook, loadChapter, submit, retryChapter, + stopPolling, closeBook, closeChapter, reset, + } +}) diff --git a/learner/src/stores/session.ts b/learner/src/stores/session.ts index 4634fec..1b6e0de 100644 --- a/learner/src/stores/session.ts +++ b/learner/src/stores/session.ts @@ -2,6 +2,17 @@ import { defineStore } from 'pinia' import { ref } from 'vue' export const TOKEN_KEY = 'lexgo-learner-token' +// Carries the real HTTP status alongside the server message so callers can tell +// "this id does not exist for me" (404) from a transient failure without +// re-parsing the envelope. The message itself is unchanged. +export class ApiError extends Error { + readonly status: number + constructor(message: string, status: number) { + super(message) + this.name = 'ApiError' + this.status = status + } +} interface User { id: number; username: string; role: 'admin' | 'learner' } interface Space { ownerId: number; language: 'en' } interface Login { token: string; expiresAt: string; user: User } @@ -36,7 +47,7 @@ export const useSessionStore = defineStore('session', () => { notice.value = '登录已失效,请重新登录。' } const result = await response.json() - if (!response.ok || result.code !== 200) throw new Error(result.msg || '请求失败,请稍后重试。') + if (!response.ok || result.code !== 200) throw new ApiError(result.msg || '请求失败,请稍后重试。', response.status) return result.data as T } @@ -91,5 +102,5 @@ export const useSessionStore = defineStore('session', () => { if (previousToken) await request('logout', 'POST', undefined, previousToken, -1) } - return { user, space, notice, login, restore, logout, loadSpace } + return { user, space, notice, login, restore, logout, loadSpace, request } }) diff --git a/learner/src/style.css b/learner/src/style.css index 49bf1f0..3cc929c 100644 --- a/learner/src/style.css +++ b/learner/src/style.css @@ -51,6 +51,50 @@ h1 { font-size: 30px; font-weight: 600; margin: 14px 0; letter-spacing: 1px; } .empty-library .book-mark { color: #6e8967; width: 56px; height: 56px; } .empty-library h2 { font-weight: 500; font-size: 21px; margin: 26px 0 4px; } .loading { padding: 80px 24px; text-align: center; color: #748073; } +/* Import / book / reader pages share one shell. */ +.page { max-width: 1120px; margin: 60px auto; padding: 0 28px; } +.page-title { display: flex; align-items: center; justify-content: space-between; gap: 18px; flex-wrap: wrap; margin-bottom: 28px; } +.page-title h1 { margin: 14px 0 6px; overflow-wrap: anywhere; } +.page-actions { display: flex; align-items: center; gap: 12px; flex-wrap: wrap; } +.library-actions { display: flex; align-items: center; gap: 14px; flex-wrap: wrap; } +.link-button { display: inline-flex; align-items: center; min-height: 42px; padding: 0 8px; color: #315c43; } +.breadcrumb { display: flex; align-items: center; gap: 8px; flex-wrap: wrap; margin: 0 0 4px; font-size: 14px; color: #748073; } +.breadcrumb a { color: #315c43; } +/* Library book cards. */ +.book-grid { list-style: none; margin: 0; padding: 0; display: grid; gap: 18px; grid-template-columns: repeat(auto-fill, minmax(290px, 1fr)); } +.book-card { display: flex; flex-direction: column; gap: 10px; background: #fffdf8; border: 1px solid #e0e3d8; border-radius: 12px; padding: 22px; } +.book-card .subtle { margin: 0; } +.book-title { font-family: Georgia, 'Microsoft YaHei', serif; font-size: 21px; font-weight: 600; color: #233d31; text-decoration: none; overflow-wrap: anywhere; } +.book-title:hover { color: #315c43; text-decoration: underline; } +.status-summary { align-self: flex-start; margin: 0; padding: 6px 13px; border-radius: 20px; background: #eef2eb; color: #3d5b48; font-size: 13px; } +/* Chapter and job status, identical vocabulary for both. */ +.status-chip { display: inline-flex; align-items: center; white-space: nowrap; padding: 5px 13px; border: 1px solid transparent; border-radius: 20px; font-size: 13px; } +.status-pending { background: #f5f2e4; border-color: #e2dcc2; color: #7a6a35; } +.status-processing { background: #eaf1f7; border-color: #c9dcea; color: #35566e; } +.status-ready { background: #eef2eb; border-color: #cbd9c9; color: #315c43; } +.status-failed { background: #fff0e7; border-color: #ebc3a8; color: #8b4324; } +/* Import form. */ +.import-form { max-width: 720px; background: #fffdf8; border: 1px solid #e0e3d8; border-radius: 12px; padding: 28px; } +.field-label { display: block; font-size: 14px; margin-bottom: 10px; } +.fixed-value { margin: 0; padding: 12px 15px; border: 1px solid #d6dccf; border-radius: 8px; background: #fffefa; font-size: 15px; } +.field-error { margin: 8px 0 0; color: #8b4324; font-size: 13px; line-height: 1.6; } +.counter { margin: 8px 0 0; color: #748073; font-size: 13px; } +.form-actions { display: flex; align-items: center; gap: 16px; flex-wrap: wrap; margin-top: 14px; } +.import-form .el-textarea__inner { min-height: 220px; line-height: 1.9; } +.book-select { width: 100%; } +/* Book chapters. */ +.chapter-list { list-style: none; margin: 0; padding: 0; display: flex; flex-direction: column; gap: 12px; } +.chapter-row { display: grid; grid-template-columns: 40px minmax(0, 1fr) auto auto; align-items: center; gap: 14px; background: #fffdf8; border: 1px solid #e0e3d8; border-radius: 12px; padding: 16px 18px; } +.chapter-ordinal { font-family: Georgia, serif; font-size: 17px; color: #8b9a8b; text-align: center; } +.chapter-info { min-width: 0; } +.chapter-name { display: inline-block; font-size: 16px; font-weight: 500; color: #233d31; text-decoration: none; overflow-wrap: anywhere; } +a.chapter-name:hover { color: #315c43; text-decoration: underline; } +.chapter-meta { margin: 4px 0 0; color: #748073; font-size: 13px; overflow-wrap: anywhere; } +/* Reader keeps the pasted text exactly as it was, including line breaks and tabs. */ +.reader-page { max-width: 820px; } +.reader-text { white-space: pre-wrap; overflow-wrap: break-word; margin: 26px 0 0; font-family: Georgia, 'Times New Roman', 'Microsoft YaHei', serif; font-size: 17px; line-height: 2; } +.processing-hint { padding: 36px 0; color: #748073; } +.reader-nav { display: flex; align-items: center; justify-content: space-between; gap: 14px; flex-wrap: wrap; margin-top: 36px; padding-top: 22px; border-top: 1px solid #e0e3d8; } @media (max-width: 760px) { .login-page { grid-template-columns: 1fr; } .welcome { padding: 28px; } @@ -62,4 +106,15 @@ h1 { font-size: 30px; font-weight: 600; margin: 14px 0; letter-spacing: 1px; } .site-header nav { order: 3; flex-basis: 100%; padding-top: 8px; } .library { margin-top: 32px; padding: 0 20px; } h1 { font-size: 26px; } + /* Single column, tap-friendly controls and no horizontal overflow. */ + .page { margin-top: 32px; padding: 0 20px; } + .page-title { align-items: flex-start; } + .library-title { align-items: flex-start; } + .library-actions { width: 100%; justify-content: space-between; } + .book-grid { grid-template-columns: 1fr; } + .import-form { padding: 20px; } + .chapter-row { grid-template-columns: 30px minmax(0, 1fr); align-items: start; row-gap: 10px; padding: 15px 16px; } + .chapter-row .status-chip, .chapter-row .el-button { grid-column: 2; justify-self: start; } + .reader-text { font-size: 16px; line-height: 1.95; } + .reader-nav .el-button { flex: 1; } } diff --git a/learner/src/views/BookView.vue b/learner/src/views/BookView.vue new file mode 100644 index 0000000..6a7902f --- /dev/null +++ b/learner/src/views/BookView.vue @@ -0,0 +1,94 @@ + + + diff --git a/learner/src/views/ImportView.vue b/learner/src/views/ImportView.vue new file mode 100644 index 0000000..cc34c2d --- /dev/null +++ b/learner/src/views/ImportView.vue @@ -0,0 +1,133 @@ + + + diff --git a/learner/src/views/LibraryView.vue b/learner/src/views/LibraryView.vue index f78b5a3..cdc6cbe 100644 --- a/learner/src/views/LibraryView.vue +++ b/learner/src/views/LibraryView.vue @@ -3,8 +3,10 @@ import { onMounted, ref } from 'vue' import { RouterLink, useRouter } from 'vue-router' import { ElButton } from 'element-plus' import BookMark from '../components/BookMark.vue' +import { statusSummary, useLibraryStore } from '../stores/library' import { useSessionStore } from '../stores/session' const session = useSessionStore() +const library = useLibraryStore() const router = useRouter() const loading = ref(true) const error = ref('') @@ -13,7 +15,9 @@ async function load() { error.value = '' try { await session.loadSpace() } catch (reason) { error.value = reason instanceof Error ? reason.message : '暂时无法加载,请重试。' } - finally { loading.value = false } + // The book list keeps its own error so one failing call still shows a retry. + if (!error.value) await library.loadBooks() + loading.value = false } async function logout() { try { await session.logout() } @@ -34,11 +38,28 @@ onMounted(load)
-

我的书库

你的阅读与学习,从这里开始。

英语
+
+

我的书库

你的阅读与学习,从这里开始。

+
+ 英语 + 导入内容 +
+

正在加载…

-

{{ error }}

重试
-
-

书库还是空的

这里将收纳你的阅读内容。

+
+

{{ error || library.booksError }}

+ 重试 +
+
    +
  • + {{ item.title }} +

    {{ item.chapterCount }} 个章节

    +

    {{ statusSummary(item) }}

    +

    尚未导入章节

    +
  • +
+
+

书库还是空的

粘贴一段英文,开始你的第一篇阅读。

diff --git a/learner/src/views/ReaderView.vue b/learner/src/views/ReaderView.vue new file mode 100644 index 0000000..f871bb7 --- /dev/null +++ b/learner/src/views/ReaderView.vue @@ -0,0 +1,91 @@ + + + diff --git a/server/app/lexgo/database.go b/server/app/lexgo/database.go index 38f0244..0fb89dc 100644 --- a/server/app/lexgo/database.go +++ b/server/app/lexgo/database.go @@ -4,10 +4,11 @@ import ( "context" "errors" "fmt" - driver "github.com/go-sql-driver/mysql" - "gorm.io/gorm" "strings" "time" + + driver "github.com/go-sql-driver/mysql" + "gorm.io/gorm" ) // Migrate takes a connection-scoped lock. Only an empty or LexGo-owned schema is accepted. @@ -59,44 +60,17 @@ func Migrate(db *gorm.DB) error { if err = conn.QueryRowContext(ctx, "SELECT version,product FROM lexgo_schema WHERE id=1").Scan(¤t, &product); err != nil { return err } - if product != "lexgo" || current < 0 || current > 2 { + if product != "lexgo" || current < 0 || current > schemaVersion { return errors.New("unknown schema version") } - if current == 2 { - return nil + // Each known version contributes its own statements; the version row advances only + // after every statement succeeded, so a partially applied migration can be retried. + statements := make([]string, 0, 16) + if current < 2 { + statements = append(statements, schemaV2Statements...) } - statements := []string{ - `CREATE TABLE IF NOT EXISTS sys_user ( - user_id BIGINT PRIMARY KEY AUTO_INCREMENT, username VARCHAR(32) CHARACTER SET ascii COLLATE ascii_bin NOT NULL UNIQUE, - password VARCHAR(128) NOT NULL, nick_name VARCHAR(128) NOT NULL DEFAULT '', phone VARCHAR(11) NOT NULL DEFAULT '', - role_id INT NOT NULL, salt VARCHAR(255) NOT NULL DEFAULT '', avatar VARCHAR(255) NOT NULL DEFAULT '', - sex VARCHAR(255) NOT NULL DEFAULT '', email VARCHAR(128) NOT NULL DEFAULT '', dept_id BIGINT NOT NULL DEFAULT 0, - post_id BIGINT NOT NULL DEFAULT 0, remark VARCHAR(255) NOT NULL DEFAULT '', status VARCHAR(4) NOT NULL DEFAULT '2', - create_by BIGINT NOT NULL DEFAULT 0, update_by BIGINT NOT NULL DEFAULT 0, - created_at DATETIME(3) NULL, updated_at DATETIME(3) NULL, deleted_at DATETIME(3) NULL, - CHECK (role_id IN (1,2)), CHECK (status IN ('1','2')) - ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`, - `CREATE TABLE IF NOT EXISTS lexgo_spaces ( - owner_id BIGINT PRIMARY KEY, language VARCHAR(16) NOT NULL DEFAULT 'en', - FOREIGN KEY (owner_id) REFERENCES sys_user(user_id) ON DELETE CASCADE - ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`, - `CREATE TABLE IF NOT EXISTS lexgo_sessions ( - token_hash CHAR(64) CHARACTER SET ascii COLLATE ascii_bin PRIMARY KEY, - owner_id BIGINT NOT NULL, expires_at DATETIME(3) NOT NULL, - INDEX (owner_id), INDEX (expires_at), - FOREIGN KEY (owner_id) REFERENCES sys_user(user_id) ON DELETE CASCADE - ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`, - `CREATE TABLE IF NOT EXISTS lexgo_login_logs ( - id BIGINT UNSIGNED PRIMARY KEY AUTO_INCREMENT, username VARCHAR(32) NOT NULL DEFAULT '', - result VARCHAR(16) NOT NULL, reason VARCHAR(32) NOT NULL, ip VARCHAR(45) NOT NULL DEFAULT '', - created_at DATETIME(3) NOT NULL, INDEX(created_at,id), INDEX(username,created_at) - ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`, - `CREATE TABLE IF NOT EXISTS lexgo_operation_logs ( - id BIGINT UNSIGNED PRIMARY KEY AUTO_INCREMENT, actor_id BIGINT NOT NULL, actor_username VARCHAR(32) NOT NULL, - target_id BIGINT NOT NULL DEFAULT 0, target_username VARCHAR(32) NOT NULL DEFAULT '', - action VARCHAR(32) NOT NULL, result VARCHAR(16) NOT NULL, reason VARCHAR(32) NOT NULL, - created_at DATETIME(3) NOT NULL, INDEX(created_at,id), INDEX(actor_username,created_at), INDEX(target_username,created_at) - ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`, + if current < 3 { + statements = append(statements, schemaV3Statements...) } for i, s := range statements { if _, err = conn.ExecContext(ctx, s); err != nil { @@ -107,16 +81,89 @@ func Migrate(db *gorm.DB) error { return fmt.Errorf("migration statement %d failed", i+1) } } - _, err = conn.ExecContext(ctx, "UPDATE lexgo_schema SET version=2 WHERE id=1") + _, err = conn.ExecContext(ctx, fmt.Sprintf("UPDATE lexgo_schema SET version=%d WHERE id=1", schemaVersion)) return err } +const schemaVersion = 3 + +var schemaV2Statements = []string{ + `CREATE TABLE IF NOT EXISTS sys_user ( + user_id BIGINT PRIMARY KEY AUTO_INCREMENT, username VARCHAR(32) CHARACTER SET ascii COLLATE ascii_bin NOT NULL UNIQUE, + password VARCHAR(128) NOT NULL, nick_name VARCHAR(128) NOT NULL DEFAULT '', phone VARCHAR(11) NOT NULL DEFAULT '', + role_id INT NOT NULL, salt VARCHAR(255) NOT NULL DEFAULT '', avatar VARCHAR(255) NOT NULL DEFAULT '', + sex VARCHAR(255) NOT NULL DEFAULT '', email VARCHAR(128) NOT NULL DEFAULT '', dept_id BIGINT NOT NULL DEFAULT 0, + post_id BIGINT NOT NULL DEFAULT 0, remark VARCHAR(255) NOT NULL DEFAULT '', status VARCHAR(4) NOT NULL DEFAULT '2', + create_by BIGINT NOT NULL DEFAULT 0, update_by BIGINT NOT NULL DEFAULT 0, + created_at DATETIME(3) NULL, updated_at DATETIME(3) NULL, deleted_at DATETIME(3) NULL, + CHECK (role_id IN (1,2)), CHECK (status IN ('1','2')) + ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`, + `CREATE TABLE IF NOT EXISTS lexgo_spaces ( + owner_id BIGINT PRIMARY KEY, language VARCHAR(16) NOT NULL DEFAULT 'en', + FOREIGN KEY (owner_id) REFERENCES sys_user(user_id) ON DELETE CASCADE + ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`, + `CREATE TABLE IF NOT EXISTS lexgo_sessions ( + token_hash CHAR(64) CHARACTER SET ascii COLLATE ascii_bin PRIMARY KEY, + owner_id BIGINT NOT NULL, expires_at DATETIME(3) NOT NULL, + INDEX (owner_id), INDEX (expires_at), + FOREIGN KEY (owner_id) REFERENCES sys_user(user_id) ON DELETE CASCADE + ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`, + `CREATE TABLE IF NOT EXISTS lexgo_login_logs ( + id BIGINT UNSIGNED PRIMARY KEY AUTO_INCREMENT, username VARCHAR(32) NOT NULL DEFAULT '', + result VARCHAR(16) NOT NULL, reason VARCHAR(32) NOT NULL, ip VARCHAR(45) NOT NULL DEFAULT '', + created_at DATETIME(3) NOT NULL, INDEX(created_at,id), INDEX(username,created_at) + ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`, + `CREATE TABLE IF NOT EXISTS lexgo_operation_logs ( + id BIGINT UNSIGNED PRIMARY KEY AUTO_INCREMENT, actor_id BIGINT NOT NULL, actor_username VARCHAR(32) NOT NULL, + target_id BIGINT NOT NULL DEFAULT 0, target_username VARCHAR(32) NOT NULL DEFAULT '', + action VARCHAR(32) NOT NULL, result VARCHAR(16) NOT NULL, reason VARCHAR(32) NOT NULL, + created_at DATETIME(3) NOT NULL, INDEX(created_at,id), INDEX(actor_username,created_at), INDEX(target_username,created_at) + ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`, +} + +// v3 adds the private library: books, chapters with byte-exact original text, and +// persistent ingestion jobs. owner_id is denormalized onto chapters and jobs so every +// query can filter by the authenticated identity without joining. +var schemaV3Statements = []string{ + `CREATE TABLE IF NOT EXISTS lexgo_books ( + id BIGINT UNSIGNED PRIMARY KEY AUTO_INCREMENT, owner_id BIGINT NOT NULL, + title VARCHAR(120) NOT NULL, language VARCHAR(16) NOT NULL DEFAULT 'en', + created_at DATETIME(3) NOT NULL, updated_at DATETIME(3) NOT NULL, + INDEX (owner_id, updated_at, id), + FOREIGN KEY (owner_id) REFERENCES sys_user(user_id) ON DELETE CASCADE + ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`, + `CREATE TABLE IF NOT EXISTS lexgo_chapters ( + id BIGINT UNSIGNED PRIMARY KEY AUTO_INCREMENT, book_id BIGINT UNSIGNED NOT NULL, owner_id BIGINT NOT NULL, + ordinal INT NOT NULL, title VARCHAR(120) NOT NULL, + original_text MEDIUMTEXT NOT NULL, char_count INT NOT NULL DEFAULT 0, + content_sha256 CHAR(64) CHARACTER SET ascii COLLATE ascii_bin NOT NULL DEFAULT '', + status VARCHAR(16) NOT NULL DEFAULT 'pending', error_reason VARCHAR(32) NOT NULL DEFAULT '', + created_at DATETIME(3) NOT NULL, updated_at DATETIME(3) NOT NULL, + UNIQUE KEY uq_chapter_ordinal (book_id, ordinal), INDEX (owner_id, id), + CHECK (status IN ('pending','processing','ready','failed')), + FOREIGN KEY (book_id) REFERENCES lexgo_books(id) ON DELETE CASCADE + ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`, + `CREATE TABLE IF NOT EXISTS lexgo_ingest_jobs ( + id BIGINT UNSIGNED PRIMARY KEY AUTO_INCREMENT, owner_id BIGINT NOT NULL, + book_id BIGINT UNSIGNED NOT NULL, chapter_id BIGINT UNSIGNED NOT NULL, + request_key CHAR(64) CHARACTER SET ascii COLLATE ascii_bin NOT NULL, + content_sha256 CHAR(64) CHARACTER SET ascii COLLATE ascii_bin NOT NULL, + status VARCHAR(16) NOT NULL DEFAULT 'pending', attempts INT NOT NULL DEFAULT 0, + error_reason VARCHAR(32) NOT NULL DEFAULT '', + created_at DATETIME(3) NOT NULL, updated_at DATETIME(3) NOT NULL, finished_at DATETIME(3) NULL, + UNIQUE KEY uq_job_request (owner_id, request_key), INDEX (status, id), + CHECK (status IN ('pending','processing','ready','failed')), + FOREIGN KEY (book_id) REFERENCES lexgo_books(id) ON DELETE CASCADE, + FOREIGN KEY (chapter_id) REFERENCES lexgo_chapters(id) ON DELETE CASCADE + ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`, +} + func CheckSchema(db *gorm.DB) error { var r struct { Version int Product string } - if err := db.Table("lexgo_schema").Where("id=1").First(&r).Error; err != nil || r.Version != 2 || r.Product != "lexgo" { + if err := db.Table("lexgo_schema").Where("id=1").First(&r).Error; err != nil || r.Version != schemaVersion || r.Product != "lexgo" { return errors.New("run the explicit migration before starting") } return nil diff --git a/server/app/lexgo/ingest.go b/server/app/lexgo/ingest.go new file mode 100644 index 0000000..dd485a3 --- /dev/null +++ b/server/app/lexgo/ingest.go @@ -0,0 +1,217 @@ +package lexgo + +import ( + "context" + "errors" + "strings" + "time" + "unicode" + "unicode/utf8" + + "gorm.io/gorm" + "gorm.io/gorm/clause" +) + +// The ingestion worker is deliberately small: the fixed paste rule stores one chapter per +// submit, so "processing" only validates the persisted content and publishes the chapter. +// Claiming and finishing are separate transactions on purpose. A durable claim means a +// chapter observed as processing stays recoverable, whether the process stops or only the +// finishing transaction fails. + +// Recovery is one mechanism used from two places: the startup pass treats every processing row +// as abandoned, while the running worker sweeps rows that have been processing longer than any +// legitimate claim-to-finish window. A job that keeps failing ends in a readable failure +// instead of looping forever, and a retried job never creates a second chapter. +const ( + maxIngestAttempts = 5 + // ingestStaleAfter must stay longer than the longest legitimate claim-to-finish window, + // otherwise a healthy job could be processed twice. Reprocessing is harmless for content + // because a job never creates a chapter, only publishes the one it was created with. + ingestStaleAfter = 15 * time.Second +) + +// RecoverIngestJobs requeues jobs and chapters left in processing by an unclean stop. It runs +// once before the worker starts, for a single-instance deployment. +func RecoverIngestJobs(db *gorm.DB, now time.Time) (int64, error) { + return requeueStaleIngestJobs(db, now, 0) +} + +// RequeueStaleIngestJobs recovers jobs whose finishing transaction did not complete, so a +// running service does not depend on a restart to make progress again. +func RequeueStaleIngestJobs(db *gorm.DB, now time.Time) (int64, error) { + return requeueStaleIngestJobs(db, now, ingestStaleAfter) +} + +func requeueStaleIngestJobs(db *gorm.DB, now time.Time, staleAfter time.Duration) (int64, error) { + ts := stamp(now) + cutoff := stamp(now.Add(-staleAfter)) + var requeued int64 + err := db.Transaction(func(tx *gorm.DB) error { + if err := exhaustIngestJobs(tx, ts); err != nil { + return err + } + stale := []int64{} + if err := tx.Model(&IngestJob{}). + Where("status = ? AND attempts < ? AND updated_at <= ?", statusProcessing, maxIngestAttempts, cutoff). + Pluck("id", &stale).Error; err != nil { + return err + } + if len(stale) == 0 { + return nil + } + if err := setIngestChapterStatus(tx, stale, statusPending, "", ts); err != nil { + return err + } + result := tx.Model(&IngestJob{}).Where("id IN ?", stale). + Updates(map[string]any{"status": statusPending, "updated_at": ts}) + if result.Error != nil { + return result.Error + } + requeued = result.RowsAffected + return nil + }) + return requeued, err +} + +// exhaustIngestJobs fails jobs that used up the attempt budget, so nothing can stay queued or +// claimed forever. The recorded reason is readable and a manual retry is still allowed. +func exhaustIngestJobs(tx *gorm.DB, ts time.Time) error { + var exhausted []int64 + if err := tx.Model(&IngestJob{}). + Where("status IN ? AND attempts >= ?", []string{statusPending, statusProcessing}, maxIngestAttempts). + Pluck("id", &exhausted).Error; err != nil { + return err + } + if len(exhausted) == 0 { + return nil + } + if err := setIngestChapterStatus(tx, exhausted, statusFailed, reasonAttemptsExhausted, ts); err != nil { + return err + } + return tx.Model(&IngestJob{}).Where("id IN ?", exhausted).Updates(map[string]any{ + "status": statusFailed, "error_reason": reasonAttemptsExhausted, "updated_at": ts, "finished_at": ts}).Error +} + +// setIngestChapterStatus mirrors a job outcome onto the chapters it owns. +func setIngestChapterStatus(tx *gorm.DB, jobIDs []int64, status, reason string, ts time.Time) error { + var chapterIDs []int64 + if err := tx.Model(&IngestJob{}).Where("id IN ?", jobIDs).Pluck("chapter_id", &chapterIDs).Error; err != nil { + return err + } + if len(chapterIDs) == 0 { + return nil + } + return tx.Model(&Chapter{}).Where("id IN ?", chapterIDs). + Updates(map[string]any{"status": status, "error_reason": reason, "updated_at": ts}).Error +} + +// ClaimNextIngestJob takes the oldest pending job with attempts left and marks it processing in +// its own transaction. The guarded update means only one worker can own a job. +func ClaimNextIngestJob(db *gorm.DB, now time.Time) (IngestJob, bool, error) { + ts := stamp(now) + var job IngestJob + err := db.Transaction(func(tx *gorm.DB) error { + if err := tx.Where("status = ? AND attempts < ?", statusPending, maxIngestAttempts). + Order("id ASC").First(&job).Error; err != nil { + return err + } + claim := tx.Model(&IngestJob{}).Where("id = ? AND status = ?", job.ID, statusPending). + Updates(map[string]any{"status": statusProcessing, "attempts": gorm.Expr("attempts + 1"), "updated_at": ts}) + if claim.Error != nil { + return claim.Error + } + if claim.RowsAffected != 1 { + return errJobTaken + } + if err := tx.Model(&Chapter{}).Where("id = ? AND owner_id = ?", job.ChapterID, job.OwnerID). + Updates(map[string]any{"status": statusProcessing, "updated_at": ts}).Error; err != nil { + return err + } + job.Status = statusProcessing + job.Attempts++ + job.UpdatedAt = ts + return nil + }) + if errors.Is(err, gorm.ErrRecordNotFound) || errors.Is(err, errJobTaken) { + return IngestJob{}, false, nil + } + if err != nil { + return IngestJob{}, false, err + } + return job, true, nil +} + +var errJobTaken = errors.New("ingestion job already claimed") + +// FinishIngestJob validates the persisted chapter and publishes it, or records a fixed +// failure reason. The check runs again here because a worker must not trust that content +// reached the table through the paste API. +func FinishIngestJob(ctx context.Context, db *gorm.DB, job IngestJob, now time.Time) error { + ts := stamp(now) + return db.WithContext(ctx).Transaction(func(tx *gorm.DB) error { + var chapter Chapter + if err := tx.Clauses(clause.Locking{Strength: "UPDATE"}). + Where("id = ? AND owner_id = ?", job.ChapterID, job.OwnerID).First(&chapter).Error; err != nil { + return err + } + var book Book + if err := tx.Where("id = ? AND owner_id = ?", job.BookID, job.OwnerID).First(&book).Error; err != nil { + return err + } + if reason := unprocessableReason(book, chapter, job); reason != "" { + if err := tx.Model(&IngestJob{}).Where("id = ?", job.ID).Updates(map[string]any{ + "status": statusFailed, "error_reason": reason, "updated_at": ts, "finished_at": ts}).Error; err != nil { + return err + } + return tx.Model(&Chapter{}).Where("id = ?", chapter.ID).Updates(map[string]any{ + "status": statusFailed, "error_reason": reason, "updated_at": ts}).Error + } + if err := tx.Model(&Chapter{}).Where("id = ?", chapter.ID).Updates(map[string]any{ + "status": statusReady, "char_count": utf8.RuneCountInString(chapter.OriginalText), "updated_at": ts}).Error; err != nil { + return err + } + return tx.Model(&IngestJob{}).Where("id = ?", job.ID).Updates(map[string]any{ + "status": statusReady, "error_reason": "", "updated_at": ts, "finished_at": ts}).Error + }) +} + +func unprocessableReason(book Book, chapter Chapter, job IngestJob) string { + if book.Language != "en" { + return reasonUnsupportedLanguage + } + if strings.TrimFunc(chapter.OriginalText, unicode.IsSpace) == "" { + return reasonEmptyText + } + if utf8.RuneCountInString(chapter.OriginalText) > maxChapterRunes { + return reasonTooLong + } + // The job accepted a specific content version; a chapter changed after submission is a + // different paste and must be submitted again rather than silently processed. + if contentSHA(chapter.OriginalText) != job.ContentSHA256 { + return reasonContentChanged + } + return "" +} + +// ProcessIngestJobs drains up to limit pending jobs. Claiming and finishing each use their +// own transaction, so an interrupted run simply leaves a job for recovery. +func ProcessIngestJobs(ctx context.Context, db *gorm.DB, now func() time.Time, limit int) (int, error) { + processed := 0 + for i := 0; i < limit; i++ { + if err := ctx.Err(); err != nil { + return processed, err + } + job, claimed, err := ClaimNextIngestJob(db.WithContext(ctx), now()) + if err != nil { + return processed, err + } + if !claimed { + return processed, nil + } + if err = FinishIngestJob(ctx, db, job, now()); err != nil { + return processed, err + } + processed++ + } + return processed, nil +} diff --git a/server/app/lexgo/library.go b/server/app/lexgo/library.go new file mode 100644 index 0000000..956f75d --- /dev/null +++ b/server/app/lexgo/library.go @@ -0,0 +1,666 @@ +package lexgo + +import ( + "crypto/sha256" + "encoding/hex" + "errors" + "regexp" + "strings" + "time" + "unicode" + "unicode/utf8" + + driver "github.com/go-sql-driver/mysql" + "gorm.io/gorm" + "gorm.io/gorm/clause" +) + +// Chapter and job share one status vocabulary so the client renders either without mapping. +const ( + statusPending = "pending" + statusProcessing = "processing" + statusReady = "ready" + statusFailed = "failed" +) + +// Fixed worker-side failure reasons. Only these codes are stored; the readable text is +// produced at the API boundary so no user content can leak into an error field. +const ( + reasonUnsupportedLanguage = "unsupported_language" + reasonTooLong = "too_long" + reasonEmptyText = "empty_text" + reasonContentChanged = "content_changed" + reasonAttemptsExhausted = "attempts_exhausted" +) + +const ( + maxChapterRunes = 100000 + maxTitleRunes = 120 + maxBooksPerList = 200 + // A 100000 code point paste stays well inside this even with JSON escaping; the limit + // exists only so an oversized body is rejected before it is decoded. + maxPasteBodyBytes = 4 << 20 + maxJSONBodyBytes = 16 * 1024 +) + +var requestIDPattern = regexp.MustCompile(`^[A-Za-z0-9_-]{8,64}$`) + +func reasonMessage(reason string) string { + switch reason { + case reasonUnsupportedLanguage: + return "当前版本只支持英语内容" + case reasonTooLong: + return "内容超过单章上限(100000 个字符)" + case reasonEmptyText: + return "章节内容为空" + case reasonContentChanged: + return "内容在处理前发生变化,请重新提交" + case reasonAttemptsExhausted: + return "处理多次失败,请重试或重新提交" + default: + return "" + } +} + +type Book struct { + ID int64 `gorm:"primaryKey"` + OwnerID int + Title string + Language string + CreatedAt time.Time + UpdatedAt time.Time +} + +func (Book) TableName() string { return "lexgo_books" } + +type Chapter struct { + ID int64 `gorm:"primaryKey"` + BookID int64 + OwnerID int + Ordinal int + Title string + OriginalText string + CharCount int + ContentSHA256 string + Status string + ErrorReason string + CreatedAt time.Time + UpdatedAt time.Time +} + +func (Chapter) TableName() string { return "lexgo_chapters" } + +type IngestJob struct { + ID int64 `gorm:"primaryKey"` + OwnerID int + BookID int64 + ChapterID int64 + RequestKey string + ContentSHA256 string + Status string + Attempts int + ErrorReason string + CreatedAt time.Time + UpdatedAt time.Time + FinishedAt *time.Time +} + +func (IngestJob) TableName() string { return "lexgo_ingest_jobs" } + +type BookSummary struct { + ID int64 `json:"id"` + Title string `json:"title"` + Language string `json:"language"` + ChapterCount int `json:"chapterCount"` + PendingCount int `json:"pendingCount"` + ProcessingCount int `json:"processingCount"` + ReadyCount int `json:"readyCount"` + FailedCount int `json:"failedCount"` + CreatedAt time.Time `json:"createdAt"` + UpdatedAt time.Time `json:"updatedAt"` +} + +type BookRef struct { + ID int64 `json:"id"` + Title string `json:"title"` + Language string `json:"language"` +} + +func bookRef(book Book) BookRef { return BookRef{book.ID, book.Title, book.Language} } + +type ChapterSummary struct { + ID int64 `json:"id"` + BookID int64 `json:"bookId"` + Ordinal int `json:"ordinal"` + Title string `json:"title"` + Status string `json:"status"` + CharCount int `json:"charCount"` + // JobID lets a client retry a failed chapter without keeping the submit response. + JobID *int64 `json:"jobId"` + ErrorReason string `json:"errorReason"` + ErrorMessage string `json:"errorMessage"` + CreatedAt time.Time `json:"createdAt"` + UpdatedAt time.Time `json:"updatedAt"` +} + +type ChapterView struct { + ChapterSummary + ContentSHA256 string `json:"contentSha256"` + // OriginalText is returned only for a ready chapter, so unprocessed content cannot be + // rendered as readable text by the client. + OriginalText string `json:"originalText,omitempty"` +} + +func chapterSummary(c Chapter) ChapterSummary { return chapterSummaryWithJob(c, nil) } + +func chapterSummaryWithJob(c Chapter, jobID *int64) ChapterSummary { + return ChapterSummary{ID: c.ID, BookID: c.BookID, Ordinal: c.Ordinal, Title: c.Title, + Status: c.Status, CharCount: c.CharCount, JobID: jobID, ErrorReason: c.ErrorReason, + ErrorMessage: reasonMessage(c.ErrorReason), CreatedAt: c.CreatedAt, UpdatedAt: c.UpdatedAt} +} + +func chapterView(c Chapter, jobID *int64) ChapterView { + view := ChapterView{ChapterSummary: chapterSummaryWithJob(c, jobID), ContentSHA256: c.ContentSHA256} + if c.Status == statusReady { + view.OriginalText = c.OriginalText + } + return view +} + +type JobView struct { + ID int64 `json:"id"` + BookID int64 `json:"bookId"` + ChapterID int64 `json:"chapterId"` + Status string `json:"status"` + Attempts int `json:"attempts"` + ErrorReason string `json:"errorReason"` + ErrorMessage string `json:"errorMessage"` + CreatedAt time.Time `json:"createdAt"` + UpdatedAt time.Time `json:"updatedAt"` +} + +func jobView(j IngestJob) JobView { + return JobView{j.ID, j.BookID, j.ChapterID, j.Status, j.Attempts, j.ErrorReason, reasonMessage(j.ErrorReason), j.CreatedAt, j.UpdatedAt} +} + +type Navigation struct { + PreviousChapterID *int64 `json:"previousChapterId"` + NextChapterID *int64 `json:"nextChapterId"` +} + +// ReaderResponse carries everything the reader needs for one chapter of the caller's own book. +type ReaderResponse struct { + Book BookRef `json:"book"` + Chapter ChapterView `json:"chapter"` + Navigation Navigation `json:"navigation"` +} + +type PasteResult struct { + Book *BookRef `json:"book,omitempty"` + Chapter ChapterSummary `json:"chapter"` + Job JobView `json:"job"` + Duplicate bool `json:"duplicate"` +} + +type PasteBookInput struct { + RequestID string `json:"requestId"` + Title string `json:"title"` + Text string `json:"text"` + Language string `json:"language"` +} + +type PasteChapterInput struct { + RequestID string `json:"requestId"` + Title string `json:"title"` + Text string `json:"text"` +} + +func stamp(now time.Time) time.Time { return now.UTC().Truncate(time.Millisecond) } + +func contentSHA(text string) string { + v := sha256.Sum256([]byte(text)) + return hex.EncodeToString(v[:]) +} + +func requestKey(requestID string) (string, error) { + if !requestIDPattern.MatchString(requestID) { + return "", failure(400, "请求编号须为 8~64 位字母、数字、下划线或连字符") + } + return contentSHA(requestID), nil +} + +// validatePaste applies the fixed paste rules: a title within the cap, at least one +// non-space character, and at most maxChapterRunes code points. The text itself is stored +// exactly as received, so whitespace, punctuation and line breaks survive unchanged. +func validatePaste(title, text string) (string, string, int, error) { + name := strings.TrimSpace(title) + if name == "" { + return "", "", 0, failure(400, "请填写标题") + } + if utf8.RuneCountInString(name) > maxTitleRunes { + return "", "", 0, failure(400, "标题最多 120 个字符") + } + if strings.TrimFunc(text, unicode.IsSpace) == "" { + return "", "", 0, failure(400, "请输入正文内容") + } + count := utf8.RuneCountInString(text) + if count > maxChapterRunes { + return "", "", 0, failure(400, "正文超过单章上限(100000 个字符)") + } + return name, contentSHA(text), count, nil +} + +// PasteBook creates one book with its first chapter and the ingestion job. The paste and the +// job are written in one transaction, so a rejected submit leaves no book behind. +func PasteBook(db *gorm.DB, owner int, now time.Time, input PasteBookInput) (PasteResult, error) { + title, sha, count, err := validatePaste(input.Title, input.Text) + if err != nil { + return PasteResult{}, err + } + if input.Language != "" && input.Language != "en" { + return PasteResult{}, failure(400, "当前版本只支持英语内容") + } + key, err := requestKey(input.RequestID) + if err != nil { + return PasteResult{}, err + } + ts := stamp(now) + var result PasteResult + err = db.Transaction(func(tx *gorm.DB) error { + existing, found, err := jobByRequest(tx, owner, key) + if err != nil { + return err + } + if found { + reused, err := reusePaste(tx, existing, title, sha) + if err != nil { + return err + } + result = reused + return nil + } + book := Book{OwnerID: owner, Title: title, Language: "en", CreatedAt: ts, UpdatedAt: ts} + if err = tx.Create(&book).Error; err != nil { + return err + } + chapter := Chapter{BookID: book.ID, OwnerID: owner, Ordinal: 1, Title: title, + OriginalText: input.Text, CharCount: count, ContentSHA256: sha, + Status: statusPending, CreatedAt: ts, UpdatedAt: ts} + if err = tx.Create(&chapter).Error; err != nil { + return err + } + job := IngestJob{OwnerID: owner, BookID: book.ID, ChapterID: chapter.ID, RequestKey: key, + ContentSHA256: sha, Status: statusPending, CreatedAt: ts, UpdatedAt: ts} + if err = tx.Create(&job).Error; err != nil { + return pasteInsertError(err) + } + ref := bookRef(book) + result = PasteResult{Book: &ref, Chapter: chapterSummaryWithJob(chapter, &job.ID), Job: jobView(job)} + return nil + }) + if errors.Is(err, errRequestReuse) { + return reusePasteByRequest(db, owner, key, title, sha) + } + if err != nil { + return PasteResult{}, err + } + return result, nil +} + +// PasteChapter appends one chapter to a book the caller already owns. The book row is locked +// so two appends cannot claim the same ordinal. +func PasteChapter(db *gorm.DB, owner int, bookID int64, now time.Time, input PasteChapterInput) (PasteResult, error) { + title, sha, count, err := validatePaste(input.Title, input.Text) + if err != nil { + return PasteResult{}, err + } + key, err := requestKey(input.RequestID) + if err != nil { + return PasteResult{}, err + } + ts := stamp(now) + var result PasteResult + err = db.Transaction(func(tx *gorm.DB) error { + var book Book + if err = lockOwnedBook(tx, owner, bookID, &book); err != nil { + return err + } + existing, found, err := jobByRequest(tx, owner, key) + if err != nil { + return err + } + if found { + if existing.BookID != book.ID { + return failure(409, "该请求编号已用于其他内容") + } + reused, err := reusePaste(tx, existing, title, sha) + if err != nil { + return err + } + result = reused + return nil + } + var last int + row := tx.Model(&Chapter{}).Where("book_id = ?", book.ID).Select("COALESCE(MAX(ordinal),0)").Row() + if err = row.Scan(&last); err != nil { + return err + } + chapter := Chapter{BookID: book.ID, OwnerID: owner, Ordinal: last + 1, Title: title, + OriginalText: input.Text, CharCount: count, ContentSHA256: sha, + Status: statusPending, CreatedAt: ts, UpdatedAt: ts} + if err = tx.Create(&chapter).Error; err != nil { + return err + } + job := IngestJob{OwnerID: owner, BookID: book.ID, ChapterID: chapter.ID, RequestKey: key, + ContentSHA256: sha, Status: statusPending, CreatedAt: ts, UpdatedAt: ts} + if err = tx.Create(&job).Error; err != nil { + return pasteInsertError(err) + } + if err = tx.Model(&Book{}).Where("id = ?", book.ID).Update("updated_at", ts).Error; err != nil { + return err + } + result = PasteResult{Chapter: chapterSummaryWithJob(chapter, &job.ID), Job: jobView(job)} + return nil + }) + if errors.Is(err, errRequestReuse) { + return reusePasteByRequest(db, owner, key, title, sha) + } + if err != nil { + return PasteResult{}, err + } + return result, nil +} + +var errRequestReuse = errors.New("ingestion request already accepted") + +// pasteInsertError turns a unique-key conflict on the job insert into a request reuse. The +// transaction must be abandoned: a concurrent submit that already committed is invisible to +// this transaction's snapshot, and its rows are re-read with locking reads below. +func pasteInsertError(err error) error { + var dup *driver.MySQLError + if errors.As(err, &dup) && dup.Number == 1062 { + return errRequestReuse + } + return err +} + +func reusePasteByRequest(db *gorm.DB, owner int, key, title, sha string) (PasteResult, error) { + var result PasteResult + err := db.Transaction(func(tx *gorm.DB) error { + job, found, err := jobByRequestLatest(tx, owner, key) + if err != nil { + return err + } + if !found { + // The conflict came from another unique key, not from a repeated request id. + return failure(409, "提交冲突,请重试") + } + result, err = reusePaste(tx, job, title, sha) + return err + }) + if err != nil { + return PasteResult{}, err + } + return result, nil +} + +func jobByRequest(tx *gorm.DB, owner int, key string) (IngestJob, bool, error) { + return jobQuery(tx, owner, key, false) +} + +// jobByRequestLatest uses a locking read, which sees the latest committed row instead of this +// transaction's older snapshot. It is required after a duplicate-key conflict: only the +// competing transaction's commit can cause that conflict, and its rows are newer than the +// snapshot this request already took. +func jobByRequestLatest(tx *gorm.DB, owner int, key string) (IngestJob, bool, error) { + return jobQuery(tx, owner, key, true) +} + +func jobQuery(tx *gorm.DB, owner int, key string, latest bool) (IngestJob, bool, error) { + var job IngestJob + query := tx + if latest { + query = tx.Clauses(clause.Locking{Strength: "UPDATE"}) + } + err := query.Where("owner_id = ? AND request_key = ?", owner, key).First(&job).Error + if errors.Is(err, gorm.ErrRecordNotFound) { + return job, false, nil + } + if err != nil { + return job, false, err + } + return job, true, nil +} + +// reusePaste answers a repeated submit with the first result instead of creating a second +// chapter. A reused request id with different content or title is a conflict, not a retry. +// Its reads are locking reads so the same answer works right after a duplicate-key conflict. +func reusePaste(tx *gorm.DB, job IngestJob, title, sha string) (PasteResult, error) { + var chapter Chapter + if err := tx.Clauses(clause.Locking{Strength: "UPDATE"}). + Where("id = ? AND owner_id = ?", job.ChapterID, job.OwnerID).First(&chapter).Error; err != nil { + if errors.Is(err, gorm.ErrRecordNotFound) { + return PasteResult{}, failure(409, "该请求已提交过,请刷新后查看结果") + } + return PasteResult{}, err + } + if chapter.ContentSHA256 != sha || chapter.Title != title { + return PasteResult{}, failure(409, "该请求编号已用于其他内容") + } + var book Book + if err := tx.Clauses(clause.Locking{Strength: "UPDATE"}). + Where("id = ? AND owner_id = ?", job.BookID, job.OwnerID).First(&book).Error; err != nil { + return PasteResult{}, err + } + ref := bookRef(book) + result := PasteResult{Chapter: chapterSummaryWithJob(chapter, &job.ID), Job: jobView(job), Duplicate: true} + if chapter.Ordinal == 1 { + result.Book = &ref + } + return result, nil +} + +func lockOwnedBook(tx *gorm.DB, owner int, bookID int64, book *Book) error { + err := tx.Clauses(clause.Locking{Strength: "UPDATE"}). + Where("id = ? AND owner_id = ?", bookID, owner).First(book).Error + if errors.Is(err, gorm.ErrRecordNotFound) { + // Another user's book and a missing book are indistinguishable to the caller. + return failure(404, "书籍不存在") + } + return err +} + +func ListBooks(db *gorm.DB, owner int) ([]BookSummary, error) { + var books []Book + if err := db.Where("owner_id = ?", owner).Order("updated_at DESC, id DESC").Limit(maxBooksPerList).Find(&books).Error; err != nil { + return nil, err + } + items := make([]BookSummary, 0, len(books)) + ids := make([]int64, 0, len(books)) + for _, b := range books { + items = append(items, BookSummary{ID: b.ID, Title: b.Title, Language: b.Language, CreatedAt: b.CreatedAt, UpdatedAt: b.UpdatedAt}) + ids = append(ids, b.ID) + } + if len(ids) == 0 { + return items, nil + } + type row struct { + BookID int64 + Status string + Total int + } + var rows []row + if err := db.Model(&Chapter{}).Select("book_id, status, COUNT(*) AS total"). + Where("owner_id = ? AND book_id IN ?", owner, ids).Group("book_id, status").Scan(&rows).Error; err != nil { + return nil, err + } + index := make(map[int64]int, len(items)) + for i, item := range items { + index[item.ID] = i + } + for _, r := range rows { + i, ok := index[r.BookID] + if !ok { + continue + } + items[i].ChapterCount += r.Total + switch r.Status { + case statusReady: + items[i].ReadyCount += r.Total + case statusProcessing: + items[i].ProcessingCount += r.Total + case statusFailed: + items[i].FailedCount += r.Total + default: + items[i].PendingCount += r.Total + } + } + return items, nil +} + +func BookDetail(db *gorm.DB, owner int, bookID int64) (BookRef, []ChapterSummary, error) { + var book Book + if err := db.Where("id = ? AND owner_id = ?", bookID, owner).First(&book).Error; err != nil { + if errors.Is(err, gorm.ErrRecordNotFound) { + return BookRef{}, nil, failure(404, "书籍不存在") + } + return BookRef{}, nil, err + } + var chapters []Chapter + if err := db.Where("book_id = ? AND owner_id = ?", book.ID, owner).Order("ordinal ASC").Find(&chapters).Error; err != nil { + return BookRef{}, nil, err + } + items := make([]ChapterSummary, 0, len(chapters)) + ids := make([]int64, 0, len(chapters)) + for _, c := range chapters { + ids = append(ids, c.ID) + } + jobs, err := jobIDsByChapter(db, owner, ids) + if err != nil { + return BookRef{}, nil, err + } + for _, c := range chapters { + var jobID *int64 + if id, ok := jobs[c.ID]; ok { + jobID = &id + } + items = append(items, chapterSummaryWithJob(c, jobID)) + } + return bookRef(book), items, nil +} + +// jobIDsByChapter maps chapters to their ingestion job so a client can retry a failed chapter +// without having kept the original submit response. +func jobIDsByChapter(db *gorm.DB, owner int, chapterIDs []int64) (map[int64]int64, error) { + ids := make(map[int64]int64, len(chapterIDs)) + if len(chapterIDs) == 0 { + return ids, nil + } + var jobs []IngestJob + if err := db.Select("id", "chapter_id").Where("owner_id = ? AND chapter_id IN ?", owner, chapterIDs). + Order("id ASC").Find(&jobs).Error; err != nil { + return nil, err + } + // Ascending order keeps the newest id if a chapter somehow has more than one job. + for _, j := range jobs { + ids[j.ChapterID] = j.ID + } + return ids, nil +} + +// ChapterDetail resolves a chapter strictly inside the caller's own books and returns the +// original text only once the chapter is ready. +func ChapterDetail(db *gorm.DB, owner int, chapterID int64) (ReaderResponse, error) { + var chapter Chapter + if err := db.Where("id = ? AND owner_id = ?", chapterID, owner).First(&chapter).Error; err != nil { + if errors.Is(err, gorm.ErrRecordNotFound) { + return ReaderResponse{}, failure(404, "章节不存在") + } + return ReaderResponse{}, err + } + var book Book + if err := db.Where("id = ? AND owner_id = ?", chapter.BookID, owner).First(&book).Error; err != nil { + return ReaderResponse{}, err + } + navigation := Navigation{} + var previous, next Chapter + if err := db.Where("book_id = ? AND owner_id = ? AND ordinal < ?", book.ID, owner, chapter.Ordinal). + Order("ordinal DESC").First(&previous).Error; err == nil { + navigation.PreviousChapterID = &previous.ID + } else if !errors.Is(err, gorm.ErrRecordNotFound) { + return ReaderResponse{}, err + } + if err := db.Where("book_id = ? AND owner_id = ? AND ordinal > ?", book.ID, owner, chapter.Ordinal). + Order("ordinal ASC").First(&next).Error; err == nil { + navigation.NextChapterID = &next.ID + } else if !errors.Is(err, gorm.ErrRecordNotFound) { + return ReaderResponse{}, err + } + jobs, err := jobIDsByChapter(db, owner, []int64{chapter.ID}) + if err != nil { + return ReaderResponse{}, err + } + var jobID *int64 + if id, ok := jobs[chapter.ID]; ok { + jobID = &id + } + return ReaderResponse{Book: bookRef(book), Chapter: chapterView(chapter, jobID), Navigation: navigation}, nil +} + +func JobDetail(db *gorm.DB, owner int, jobID int64) (JobView, error) { + var job IngestJob + if err := db.Where("id = ? AND owner_id = ?", jobID, owner).First(&job).Error; err != nil { + if errors.Is(err, gorm.ErrRecordNotFound) { + return JobView{}, failure(404, "任务不存在") + } + return JobView{}, err + } + return jobView(job), nil +} + +// RetryIngestJob requeues a failed job on the same chapter, so a retry can never create a +// second chapter for one paste. An explicit retry also restarts the attempt budget, because a +// person asking again should not be blocked by the bound that stops automatic loops. +func RetryIngestJob(db *gorm.DB, owner int, jobID int64, now time.Time) (JobView, ChapterSummary, error) { + ts := stamp(now) + var job IngestJob + var chapter Chapter + err := db.Transaction(func(tx *gorm.DB) error { + if err := tx.Clauses(clause.Locking{Strength: "UPDATE"}). + Where("id = ? AND owner_id = ?", jobID, owner).First(&job).Error; err != nil { + if errors.Is(err, gorm.ErrRecordNotFound) { + return failure(404, "任务不存在") + } + return err + } + if job.Status != statusFailed { + return failure(409, "只有失败的任务可以重试") + } + if err := tx.Clauses(clause.Locking{Strength: "UPDATE"}). + Where("id = ? AND owner_id = ?", job.ChapterID, owner).First(&chapter).Error; err != nil { + return err + } + if err := tx.Model(&IngestJob{}).Where("id = ?", job.ID). + Updates(map[string]any{"status": statusPending, "error_reason": "", "attempts": 0, + "updated_at": ts, "finished_at": nil}).Error; err != nil { + return err + } + if err := tx.Model(&Chapter{}).Where("id = ?", chapter.ID). + Updates(map[string]any{"status": statusPending, "error_reason": "", "updated_at": ts}).Error; err != nil { + return err + } + job.Status = statusPending + job.ErrorReason = "" + job.Attempts = 0 + job.FinishedAt = nil + job.UpdatedAt = ts + chapter.Status = statusPending + chapter.ErrorReason = "" + chapter.UpdatedAt = ts + return nil + }) + if err != nil { + return JobView{}, ChapterSummary{}, err + } + return jobView(job), chapterSummaryWithJob(chapter, &job.ID), nil +} diff --git a/server/app/lexgo/library_test.go b/server/app/lexgo/library_test.go new file mode 100644 index 0000000..9d0441a --- /dev/null +++ b/server/app/lexgo/library_test.go @@ -0,0 +1,914 @@ +package lexgo + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "net/http/httptest" + "strings" + "sync" + "testing" + "time" + "unicode/utf8" + + "github.com/gin-gonic/gin" + admin "go-admin/app/admin/models" + "gorm.io/gorm" +) + +// Fictional English fixture with the characters the paste contract must preserve exactly: +// CRLF and LF, a tab, curly quotes, an em dash, an ellipsis, an emoji, a combining acute +// accent, a trailing space run and an empty final line. +const fixturePastedText = "Mira opened the workshop.\r\n\r\n\tThe sign read “A small step…” — café e\u0301 🙂\r\nTrailing spaces here: \n\n" + +type libraryAccount struct { + ID int + Username string + Token string +} + +type pasteResponse struct { + Book *struct { + ID int64 + Title string + Language string + } + Chapter struct { + ID int64 + BookID int64 + Ordinal int + Title string + Status string + CharCount int + ErrorReason string + ErrorMessage string + } + Job struct { + ID int64 + BookID int64 + ChapterID int64 + Status string + Attempts int + ErrorReason string + ErrorMessage string + } + Duplicate bool +} + +type readerResponse struct { + Book struct { + ID int64 + Title string + Language string + } + Chapter struct { + ID int64 + BookID int64 + Ordinal int + Title string + Status string + CharCount int + ErrorReason string + ErrorMessage string + ContentSHA256 string + OriginalText string + JobID *int64 + } + Navigation struct { + PreviousChapterID *int64 + NextChapterID *int64 + } +} + +type bookDetailResponse struct { + Book struct { + ID int64 + Title string + Language string + } + Chapters []struct { + ID int64 + Ordinal int + Title string + Status string + CharCount int + JobID *int64 + ErrorMessage string + } +} + +type bookListResponse struct { + Items []struct { + ID int64 + Title string + ChapterCount int + PendingCount int + ProcessingCount int + ReadyCount int + FailedCount int + } +} + +// callRaw keeps the API message, which is how a readable failure reason is asserted. +func callRaw(t *testing.T, r *gin.Engine, method, path, token string, body any) (int, string, json.RawMessage) { + t.Helper() + b, _ := json.Marshal(body) + q := httptest.NewRequest(method, path, bytes.NewReader(b)) + q.Header.Set("Content-Type", "application/json") + if token != "" { + q.Header.Set("Authorization", "Bearer "+token) + } + w := httptest.NewRecorder() + r.ServeHTTP(w, q) + var e struct { + Code int `json:"code"` + Msg string `json:"msg"` + Data json.RawMessage `json:"data"` + } + if err := json.Unmarshal(w.Body.Bytes(), &e); err != nil { + t.Fatalf("invalid JSON for %s %s (status %d)", method, path, w.Code) + } + return w.Code, e.Msg, e.Data +} + +func libraryFixture(t *testing.T) (*gorm.DB, *gin.Engine, libraryAccount) { + t.Helper() + db := testDB(t) + owner := admin.SysUser{Username: randomName("admin"), Password: fixturePassword, RoleId: 1, Status: "2"} + if err := db.Create(&owner).Error; err != nil { + t.Fatal("fixture admin creation failed") + } + r := Router(db, time.Now) + return db, r, libraryAccount{owner.UserId, owner.Username, loginToken(t, r, owner.Username, fixturePassword)} +} + +func newLearner(t *testing.T, r *gin.Engine, adminToken string) libraryAccount { + t.Helper() + name := randomName("lib") + code, msg, data := callRaw(t, r, "POST", "/api/v1/accounts", adminToken, map[string]string{"username": name, "password": fixturePassword}) + if code != 201 { + t.Fatalf("create learner status %d (%s)", code, msg) + } + var created struct { + ID int + Username string + } + json.Unmarshal(data, &created) + return libraryAccount{created.ID, created.Username, loginToken(t, r, name, fixturePassword)} +} + +func pasteBook(t *testing.T, r *gin.Engine, token string, body any) (int, pasteResponse) { + t.Helper() + code, msg, data := callRaw(t, r, "POST", "/api/v1/books", token, body) + var out pasteResponse + if len(data) > 0 { + if err := json.Unmarshal(data, &out); err != nil { + t.Fatalf("paste response: %v", err) + } + } + if code >= 400 && msg == "" { + t.Fatalf("paste failed with status %d and no message", code) + } + return code, out +} + +func pasteChapter(t *testing.T, r *gin.Engine, token string, bookID int64, body any) (int, pasteResponse) { + t.Helper() + code, msg, data := callRaw(t, r, "POST", fmt.Sprintf("/api/v1/books/%d/chapters", bookID), token, body) + var out pasteResponse + if len(data) > 0 { + if err := json.Unmarshal(data, &out); err != nil { + t.Fatalf("paste chapter response: %v", err) + } + } + if code >= 400 && msg == "" { + t.Fatalf("paste chapter failed with status %d and no message", code) + } + return code, out +} + +func readChapter(t *testing.T, r *gin.Engine, token string, chapterID int64) (int, readerResponse) { + t.Helper() + code, _, data := callRaw(t, r, "GET", fmt.Sprintf("/api/v1/chapters/%d", chapterID), token, nil) + var out readerResponse + if len(data) > 0 { + if err := json.Unmarshal(data, &out); err != nil { + t.Fatalf("reader response: %v", err) + } + } + return code, out +} + +func bookDetail(t *testing.T, r *gin.Engine, token string, bookID int64) (int, bookDetailResponse) { + t.Helper() + code, _, data := callRaw(t, r, "GET", fmt.Sprintf("/api/v1/books/%d", bookID), token, nil) + var out bookDetailResponse + if len(data) > 0 { + if err := json.Unmarshal(data, &out); err != nil { + t.Fatalf("book response: %v", err) + } + } + return code, out +} + +func bookList(t *testing.T, r *gin.Engine, token string) (int, bookListResponse) { + t.Helper() + code, _, data := callRaw(t, r, "GET", "/api/v1/books", token, nil) + var out bookListResponse + if len(data) > 0 { + if err := json.Unmarshal(data, &out); err != nil { + t.Fatalf("book list response: %v", err) + } + } + return code, out +} + +func drainIngest(t *testing.T, db *gorm.DB) { + t.Helper() + if _, err := ProcessIngestJobs(t.Context(), db, time.Now, 50); err != nil { + t.Fatalf("ingestion failed: %v", err) + } +} + +func chapterRow(t *testing.T, db *gorm.DB, id int64) Chapter { + t.Helper() + var c Chapter + if err := db.Where("id = ?", id).First(&c).Error; err != nil { + t.Fatalf("chapter row: %v", err) + } + return c +} + +func jobRow(t *testing.T, db *gorm.DB, id int64) IngestJob { + t.Helper() + var j IngestJob + if err := db.Where("id = ?", id).First(&j).Error; err != nil { + t.Fatalf("job row: %v", err) + } + return j +} + +func TestMySQLPasteToReaderFullPath(t *testing.T) { + db, r, owner := libraryFixture(t) + learner := newLearner(t, r, owner.Token) + // Empty the queue so the claim below takes this test's own job. + drainIngest(t, db) + + code, pasted := pasteBook(t, r, learner.Token, map[string]string{ + "requestId": "fixture-request-full-path", "title": "The Workshop", "text": fixturePastedText, "language": "en"}) + if code != 201 { + t.Fatalf("paste status %d, want 201", code) + } + if pasted.Book == nil || pasted.Book.ID == 0 || pasted.Book.Title != "The Workshop" || pasted.Book.Language != "en" { + t.Fatalf("unexpected book %+v", pasted.Book) + } + if pasted.Chapter.Ordinal != 1 || pasted.Chapter.BookID != pasted.Book.ID || pasted.Chapter.Title != "The Workshop" { + t.Fatalf("unexpected chapter %+v", pasted.Chapter) + } + if pasted.Chapter.Status != statusPending || pasted.Job.Status != statusPending || pasted.Duplicate { + t.Fatalf("paste must queue a pending job, got chapter %q job %q", pasted.Chapter.Status, pasted.Job.Status) + } + if want := utf8.RuneCountInString(fixturePastedText); pasted.Chapter.CharCount != want { + t.Fatalf("charCount %d, want %d", pasted.Chapter.CharCount, want) + } + + // A queued chapter has no readable text yet. + code, pending := readChapter(t, r, learner.Token, pasted.Chapter.ID) + if code != 200 || pending.Chapter.Status != statusPending || pending.Chapter.OriginalText != "" { + t.Fatalf("pending chapter must not expose text: status %d, %+v", code, pending.Chapter) + } + _, queued := bookList(t, r, learner.Token) + if len(queued.Items) != 1 || queued.Items[0].PendingCount != 1 || queued.Items[0].ProcessingCount != 0 || queued.Items[0].ReadyCount != 0 { + t.Fatalf("queued book counts %+v", queued.Items) + } + + // Processing is a durable state: a claim survives a crash and is observable in between. + job, claimed, err := ClaimNextIngestJob(db, time.Now()) + if err != nil || !claimed { + t.Fatalf("claim failed (claimed=%v): %v", claimed, err) + } + if job.ID != pasted.Job.ID || job.Attempts != 1 { + t.Fatalf("claimed job %+v, want id %d with 1 attempt", job, pasted.Job.ID) + } + code, processing := readChapter(t, r, learner.Token, pasted.Chapter.ID) + if code != 200 || processing.Chapter.Status != statusProcessing || processing.Chapter.OriginalText != "" { + t.Fatalf("processing chapter must not expose text: status %d, %+v", code, processing.Chapter) + } + if err = FinishIngestJob(t.Context(), db, job, time.Now()); err != nil { + t.Fatalf("finish failed: %v", err) + } + + code, ready := readChapter(t, r, learner.Token, pasted.Chapter.ID) + if code != 200 || ready.Chapter.Status != statusReady { + t.Fatalf("ready chapter status %d %q", code, ready.Chapter.Status) + } + if ready.Chapter.OriginalText != fixturePastedText { + t.Fatalf("original text changed:\n got %q\nwant %q", ready.Chapter.OriginalText, fixturePastedText) + } + if ready.Chapter.ContentSHA256 != contentSHA(fixturePastedText) { + t.Fatal("content hash mismatch") + } + if ready.Chapter.CharCount != utf8.RuneCountInString(fixturePastedText) { + t.Fatalf("charCount %d after processing", ready.Chapter.CharCount) + } + if ready.Navigation.PreviousChapterID != nil || ready.Navigation.NextChapterID != nil { + t.Fatal("single chapter must not navigate") + } + if ready.Book.ID != pasted.Book.ID || ready.Book.Title != "The Workshop" { + t.Fatalf("unexpected reader book %+v", ready.Book) + } + + code, _, data := callRaw(t, r, "GET", fmt.Sprintf("/api/v1/jobs/%d", pasted.Job.ID), learner.Token, nil) + var jobView struct { + Job struct { + Status string + Attempts int + ChapterID int64 + } + } + json.Unmarshal(data, &jobView) + if code != 200 || jobView.Job.Status != statusReady || jobView.Job.Attempts != 1 || jobView.Job.ChapterID != pasted.Chapter.ID { + t.Fatalf("job view %+v (status %d)", jobView.Job, code) + } + + // Appending keeps the fixed rule: one paste, one more chapter. + code, appended := pasteChapter(t, r, learner.Token, pasted.Book.ID, map[string]string{ + "requestId": "fixture-request-append", "title": "Second Chapter", "text": "A single plain paragraph.\n"}) + if code != 201 || appended.Chapter.Ordinal != 2 || appended.Book != nil { + t.Fatalf("append status %d chapter %+v book %+v", code, appended.Chapter, appended.Book) + } + drainIngest(t, db) + code, second := readChapter(t, r, learner.Token, appended.Chapter.ID) + if code != 200 || second.Chapter.Status != statusReady || second.Chapter.OriginalText != "A single plain paragraph.\n" { + t.Fatalf("appended chapter %+v", second.Chapter) + } + if second.Navigation.PreviousChapterID == nil || *second.Navigation.PreviousChapterID != pasted.Chapter.ID || second.Navigation.NextChapterID != nil { + t.Fatalf("appended navigation %+v", second.Navigation) + } + code, first := readChapter(t, r, learner.Token, pasted.Chapter.ID) + if code != 200 || first.Navigation.NextChapterID == nil || *first.Navigation.NextChapterID != appended.Chapter.ID { + t.Fatalf("first chapter navigation %+v", first.Navigation) + } + + code, detail := bookDetail(t, r, learner.Token, pasted.Book.ID) + if code != 200 || len(detail.Chapters) != 2 || detail.Chapters[0].Status != statusReady || detail.Chapters[1].Ordinal != 2 { + t.Fatalf("book detail %+v", detail) + } + // The chapter list carries the job id, which is what a client needs to retry a failure. + if detail.Chapters[0].JobID == nil || *detail.Chapters[0].JobID != pasted.Job.ID { + t.Fatalf("book detail chapter job id %+v, want %d", detail.Chapters[0].JobID, pasted.Job.ID) + } + if ready.Chapter.JobID == nil || *ready.Chapter.JobID != pasted.Job.ID { + t.Fatalf("reader chapter job id %+v, want %d", ready.Chapter.JobID, pasted.Job.ID) + } + code, list := bookList(t, r, learner.Token) + if code != 200 || len(list.Items) != 1 || list.Items[0].ChapterCount != 2 || list.Items[0].ReadyCount != 2 || list.Items[0].FailedCount != 0 { + t.Fatalf("book list %+v", list) + } + if list.Items[0].PendingCount != 0 || list.Items[0].ProcessingCount != 0 { + t.Fatalf("published book must have no queued chapter: %+v", list.Items[0]) + } + if list.Items[0].ID != pasted.Book.ID { + t.Fatal("book list must only contain the caller's own book") + } +} + +func TestMySQLIngestFailureReasonsAndRetry(t *testing.T) { + db, r, owner := libraryFixture(t) + learner := newLearner(t, r, owner.Token) + + cases := []struct { + name string + reason string + mutate func(book *Book, chapter *Chapter, job *IngestJob) + }{ + {"valid", "", func(*Book, *Chapter, *IngestJob) {}}, + {"unsupported_language", reasonUnsupportedLanguage, func(book *Book, _ *Chapter, _ *IngestJob) { + book.Language = "de" + }}, + {"empty_text", reasonEmptyText, func(_ *Book, chapter *Chapter, job *IngestJob) { + chapter.OriginalText = " \r\n\t " + chapter.ContentSHA256 = contentSHA(chapter.OriginalText) + job.ContentSHA256 = chapter.ContentSHA256 + }}, + {"too_long", reasonTooLong, func(_ *Book, chapter *Chapter, job *IngestJob) { + chapter.OriginalText = strings.Repeat("a", maxChapterRunes+1) + chapter.ContentSHA256 = contentSHA(chapter.OriginalText) + job.ContentSHA256 = chapter.ContentSHA256 + }}, + {"content_changed", reasonContentChanged, func(_ *Book, chapter *Chapter, _ *IngestJob) { + chapter.OriginalText = "Mira opened" + }}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + code, pasted := pasteBook(t, r, learner.Token, map[string]string{ + "requestId": randomName("case"), "title": "Case " + tc.name, "text": fixturePastedText}) + if code != 201 { + t.Fatalf("paste status %d", code) + } + // These are the states another writing path could leave behind; the worker must + // re-validate persisted content instead of trusting the paste API. + book := Book{ID: pasted.Book.ID, OwnerID: learner.ID, Language: "en"} + chapter := chapterRow(t, db, pasted.Chapter.ID) + job := jobRow(t, db, pasted.Job.ID) + tc.mutate(&book, &chapter, &job) + if err := db.Model(&Book{}).Where("id = ?", book.ID).Update("language", book.Language).Error; err != nil { + t.Fatal(err) + } + if err := db.Model(&Chapter{}).Where("id = ?", chapter.ID). + Updates(map[string]any{"original_text": chapter.OriginalText, "content_sha256": chapter.ContentSHA256}).Error; err != nil { + t.Fatal(err) + } + if err := db.Model(&IngestJob{}).Where("id = ?", job.ID). + Update("content_sha256", job.ContentSHA256).Error; err != nil { + t.Fatal(err) + } + drainIngest(t, db) + code, read := readChapter(t, r, learner.Token, pasted.Chapter.ID) + if code != 200 { + t.Fatalf("read status %d", code) + } + if tc.reason == "" { + if read.Chapter.Status != statusReady || read.Chapter.OriginalText != fixturePastedText { + t.Fatalf("valid content must reach ready, got %+v", read.Chapter) + } + return + } + if read.Chapter.Status != statusFailed || read.Chapter.ErrorReason != tc.reason { + t.Fatalf("chapter %q with reason %q, want failed/%s", read.Chapter.Status, read.Chapter.ErrorReason, tc.reason) + } + // The readable reason is produced by the API and never stores user content. + if read.Chapter.ErrorMessage == "" || strings.ContainsAny(read.Chapter.ErrorMessage, "\r\n") { + t.Fatalf("unreadable failure message %q", read.Chapter.ErrorMessage) + } + if read.Chapter.OriginalText != "" { + t.Fatal("failed chapter must not expose text") + } + var detail bookDetailResponse + _, detail = bookDetail(t, r, learner.Token, pasted.Book.ID) + if detail.Chapters[0].Status != statusFailed || detail.Chapters[0].ErrorMessage == "" { + t.Fatalf("book detail must show the failure: %+v", detail.Chapters[0]) + } + // Retry through the job id the chapter list exposes, which is the client's only path. + if detail.Chapters[0].JobID == nil || *detail.Chapters[0].JobID != pasted.Job.ID { + t.Fatalf("failed chapter must expose its job id: %+v", detail.Chapters[0]) + } + code, _, _ = callRaw(t, r, "POST", fmt.Sprintf("/api/v1/jobs/%d/retry", *detail.Chapters[0].JobID), learner.Token, nil) + if code != 200 { + t.Fatalf("retry status %d, want 200", code) + } + // Retrying reuses the same chapter: no second chapter for one paste. + var count int64 + db.Model(&Chapter{}).Where("book_id = ?", pasted.Book.ID).Count(&count) + if count != 1 { + t.Fatalf("retry created %d chapters, want 1", count) + } + }) + } +} + +func TestMySQLRetryAfterContentRestoredPublishesSameChapter(t *testing.T) { + db, r, owner := libraryFixture(t) + learner := newLearner(t, r, owner.Token) + code, pasted := pasteBook(t, r, learner.Token, map[string]string{ + "requestId": "fixture-retry", "title": "Retry Book", "text": fixturePastedText}) + if code != 201 { + t.Fatalf("paste status %d", code) + } + // Simulate a chapter whose stored text was replaced before processing. + if err := db.Model(&Chapter{}).Where("id = ?", pasted.Chapter.ID). + Update("original_text", "Mira opened").Error; err != nil { + t.Fatal(err) + } + drainIngest(t, db) + code, failed := readChapter(t, r, learner.Token, pasted.Chapter.ID) + if code != 200 || failed.Chapter.Status != statusFailed || failed.Chapter.ErrorReason != reasonContentChanged { + t.Fatalf("expected content_changed failure, got %+v", failed.Chapter) + } + code, msg, _ := callRaw(t, r, "POST", fmt.Sprintf("/api/v1/jobs/%d/retry", pasted.Job.ID), learner.Token, nil) + if code != 200 { + t.Fatalf("retry status %d (%s)", code, msg) + } + // A pending retry exposes no text and does not create a new chapter. + code, pending := readChapter(t, r, learner.Token, pasted.Chapter.ID) + if code != 200 || pending.Chapter.Status != statusPending || pending.Chapter.OriginalText != "" { + t.Fatalf("retry must return the chapter to pending, got %+v", pending.Chapter) + } + var count int64 + db.Model(&Chapter{}).Where("book_id = ?", pasted.Book.ID).Count(&count) + if count != 1 { + t.Fatalf("retry created %d chapters, want 1", count) + } + // Simulate the content being restored to what was submitted, then retry to completion. + if err := db.Model(&Chapter{}).Where("id = ?", pasted.Chapter.ID). + Update("original_text", fixturePastedText).Error; err != nil { + t.Fatal(err) + } + drainIngest(t, db) + code, ready := readChapter(t, r, learner.Token, pasted.Chapter.ID) + if code != 200 || ready.Chapter.Status != statusReady || ready.Chapter.OriginalText != fixturePastedText { + t.Fatalf("retry did not publish the same chapter: %+v", ready.Chapter) + } + job := jobRow(t, db, pasted.Job.ID) + // A manual retry restarts the attempt budget, so this processing is attempt 1 again. + if job.Attempts != 1 { + t.Fatalf("attempts %d after a manual retry, want 1", job.Attempts) + } + code, msg, _ = callRaw(t, r, "POST", fmt.Sprintf("/api/v1/jobs/%d/retry", pasted.Job.ID), learner.Token, nil) + if code != 409 || msg == "" { + t.Fatalf("retry of a ready job status %d (%s), want 409 with a message", code, msg) + } +} + +func TestMySQLRepeatedPasteIsIdempotent(t *testing.T) { + db, r, owner := libraryFixture(t) + learner := newLearner(t, r, owner.Token) + body := map[string]string{"requestId": "fixture-request-repeat", "title": "Repeat", "text": fixturePastedText} + + code, first := pasteBook(t, r, learner.Token, body) + if code != 201 || first.Duplicate { + t.Fatalf("first paste status %d duplicate %v", code, first.Duplicate) + } + code, second := pasteBook(t, r, learner.Token, body) + if code != 200 || !second.Duplicate { + t.Fatalf("repeat paste status %d duplicate %v, want 200 with duplicate", code, second.Duplicate) + } + if second.Chapter.ID != first.Chapter.ID || second.Job.ID != first.Job.ID || second.Book.ID != first.Book.ID { + t.Fatalf("repeat paste returned different rows: %+v vs %+v", second, first) + } + var books, chapters int64 + db.Model(&Book{}).Where("owner_id = ?", learner.ID).Count(&books) + db.Model(&Chapter{}).Where("owner_id = ?", learner.ID).Count(&chapters) + if books != 1 || chapters != 1 { + t.Fatalf("repeat paste created books=%d chapters=%d, want 1/1", books, chapters) + } + + // The same request id with different content is a conflict, not a silent reuse. + for name, changed := range map[string]map[string]string{ + "different text": {"requestId": "fixture-request-repeat", "title": "Repeat", "text": "Mira opened the workshop."}, + "different title": {"requestId": "fixture-request-repeat", "title": "Another", "text": fixturePastedText}, + } { + code, msg, _ := callRaw(t, r, "POST", "/api/v1/books", learner.Token, changed) + if code != 409 || msg == "" { + t.Fatalf("%s: status %d (%s), want 409", name, code, msg) + } + } + + // Two concurrent submits of one request id create exactly one chapter. + name := randomName("race") + body = map[string]string{"requestId": name, "title": "Race", "text": "A small step; a small step"} + var wg sync.WaitGroup + type attempt struct { + code int + msg string + } + results := make(chan attempt, 2) + for i := 0; i < 2; i++ { + wg.Add(1) + go func() { + defer wg.Done() + code, msg, _ := callRaw(t, r, "POST", "/api/v1/books", learner.Token, body) + results <- attempt{code, msg} + }() + } + wg.Wait() + close(results) + created, reused := 0, 0 + for result := range results { + switch result.code { + case 201: + created++ + case 200: + reused++ + default: + t.Fatalf("concurrent paste status %d (%s)", result.code, result.msg) + } + } + if created != 1 || reused != 1 { + t.Fatalf("concurrent paste created=%d reused=%d", created, reused) + } + db.Model(&Chapter{}).Where("owner_id = ?", learner.ID).Count(&chapters) + if chapters != 2 { + t.Fatalf("concurrent paste left %d chapters, want 2", chapters) + } + + // A request id already used by another book cannot be replayed by appending, while a + // fresh request id appends normally. + var raceBook Book + if err := db.Where("owner_id = ? AND title = ?", learner.ID, "Race").First(&raceBook).Error; err != nil { + t.Fatalf("race book: %v", err) + } + code, msg, _ := callRaw(t, r, "POST", fmt.Sprintf("/api/v1/books/%d/chapters", first.Book.ID), learner.Token, + map[string]string{"requestId": name, "title": "Append", "text": "A second chapter."}) + if code != 409 || msg == "" { + t.Fatalf("cross-book request id status %d (%s), want 409", code, msg) + } + code, appended := pasteChapter(t, r, learner.Token, first.Book.ID, map[string]string{ + "requestId": randomName("append"), "title": "Append", "text": "A second chapter."}) + if code != 201 || appended.Chapter.Ordinal != 2 { + t.Fatalf("append status %d chapter %+v", code, appended.Chapter) + } + if raceBook.ID == first.Book.ID { + t.Fatal("idempotency fixtures must use different books") + } + // Append owns the language of its book, so the field is not part of that contract and the + // strict decoder rejects it. The learner client must therefore not send it (regression R1). + code, msg, _ = callRaw(t, r, "POST", fmt.Sprintf("/api/v1/books/%d/chapters", first.Book.ID), learner.Token, + map[string]string{"requestId": randomName("append"), "title": "Strict", "text": "Strict contract.", "language": "en"}) + if code != 400 || msg == "" { + t.Fatalf("append with language status %d (%s), want 400", code, msg) + } +} + +func TestMySQLLibraryIsolationAndOwnership(t *testing.T) { + db, r, owner := libraryFixture(t) + a := newLearner(t, r, owner.Token) + b := newLearner(t, r, owner.Token) + + code, pasted := pasteBook(t, r, a.Token, map[string]string{ + "requestId": "fixture-request-isolation", "title": "Private Book", "text": "Only A may read this."}) + if code != 201 { + t.Fatalf("paste status %d", code) + } + // Force a failed job so the retry path is checked for another account too. + if err := db.Model(&IngestJob{}).Where("id = ?", pasted.Job.ID).Updates(map[string]any{"status": statusFailed, "error_reason": reasonContentChanged}).Error; err != nil { + t.Fatal(err) + } + + var book Book + var chapter Chapter + var job IngestJob + db.Where("id = ?", pasted.Book.ID).First(&book) + db.Where("id = ?", pasted.Chapter.ID).First(&chapter) + db.Where("id = ?", pasted.Job.ID).First(&job) + if book.OwnerID != a.ID || chapter.OwnerID != a.ID || job.OwnerID != a.ID { + t.Fatal("stored rows must belong to the authenticated account") + } + if chapter.BookID != book.ID || job.ChapterID != chapter.ID { + t.Fatal("job and chapter must stay linked to the book") + } + + for name, token := range map[string]string{"other learner": b.Token, "administrator": owner.Token} { + for _, path := range []string{ + fmt.Sprintf("/api/v1/books/%d", book.ID), + fmt.Sprintf("/api/v1/chapters/%d", chapter.ID), + fmt.Sprintf("/api/v1/jobs/%d", job.ID), + } { + code, _, _ := callRaw(t, r, "GET", path, token, nil) + if code != 404 { + t.Fatalf("%s GET %s status %d, want 404", name, path, code) + } + } + code, _, _ = callRaw(t, r, "POST", fmt.Sprintf("/api/v1/books/%d/chapters", book.ID), token, + map[string]string{"requestId": randomName("intruder"), "title": "Intruder", "text": "Intruder text."}) + if code != 404 { + t.Fatalf("%s append status %d, want 404", name, code) + } + code, _, _ = callRaw(t, r, "POST", fmt.Sprintf("/api/v1/jobs/%d/retry", job.ID), token, nil) + if code != 404 { + t.Fatalf("%s retry status %d, want 404", name, code) + } + _, list := bookList(t, r, token) + if len(list.Items) != 0 { + t.Fatalf("%s sees %d books", name, len(list.Items)) + } + } + + // The caller's identity comes from the session, never from the request body or query. + for _, payload := range []map[string]any{ + {"requestId": randomName("owner"), "title": "Spoof", "text": "Spoofed owner.", "ownerId": b.ID}, + {"requestId": randomName("owner"), "title": "Spoof", "text": "Spoofed owner.", "userId": b.ID}, + } { + code, msg, _ := callRaw(t, r, "POST", "/api/v1/books", a.Token, payload) + if code != 400 || msg == "" { + t.Fatalf("client-supplied owner status %d (%s), want 400", code, msg) + } + } + code, msg, _ := callRaw(t, r, "GET", fmt.Sprintf("/api/v1/books?ownerId=%d", b.ID), a.Token, nil) + if code != 400 || msg == "" { + t.Fatalf("query owner override status %d (%s), want 400", code, msg) + } + var books int64 + db.Model(&Book{}).Where("owner_id = ?", a.ID).Count(&books) + if books != 1 { + t.Fatalf("rejected requests created %d books", books) + } +} + +func TestMySQLIngestRecoveryWithoutRestart(t *testing.T) { + db, r, owner := libraryFixture(t) + learner := newLearner(t, r, owner.Token) + drainIngest(t, db) + code, pasted := pasteBook(t, r, learner.Token, map[string]string{ + "requestId": "fixture-in-service-recovery", "title": "In-service recovery", "text": fixturePastedText}) + if code != 201 { + t.Fatalf("paste status %d", code) + } + job, claimed, err := ClaimNextIngestJob(db, time.Now()) + if err != nil || !claimed || job.ID != pasted.Job.ID { + t.Fatalf("claim failed (claimed=%v, job=%d): %v", claimed, job.ID, err) + } + + // The finishing transaction fails (context cancellation stands in for a timeout or a + // database error). The claim is already committed, so the job stays processing. + canceled, cancel := context.WithCancel(t.Context()) + cancel() + if err = FinishIngestJob(canceled, db, job, time.Now()); err == nil { + t.Fatal("a canceled finishing transaction must report an error") + } + var stuck IngestJob + if err = db.Where("id = ?", pasted.Job.ID).First(&stuck).Error; err != nil { + t.Fatal(err) + } + if stuck.Status != statusProcessing { + t.Fatalf("job status %q after a failed finish, want processing", stuck.Status) + } + // A manual retry cannot rescue it: only failed jobs are accepted. + code, msg, _ := callRaw(t, r, "POST", fmt.Sprintf("/api/v1/jobs/%d/retry", pasted.Job.ID), learner.Token, nil) + if code != 409 || msg == "" { + t.Fatalf("retry of a processing job status %d (%s), want 409", code, msg) + } + + // A sweep that is too early must leave a healthy claim alone. + claimedAt := stuck.UpdatedAt + if _, err = RequeueStaleIngestJobs(db, claimedAt.Add(time.Second)); err != nil { + t.Fatal(err) + } + if jobRow(t, db, pasted.Job.ID).Status != statusProcessing { + t.Fatal("a fresh claim must not be requeued") + } + // Once the claim is older than the stale window, the running service recovers it. + if _, err = RequeueStaleIngestJobs(db, claimedAt.Add(ingestStaleAfter+time.Second)); err != nil { + t.Fatal(err) + } + recovered := jobRow(t, db, pasted.Job.ID) + if recovered.Status != statusPending || recovered.Attempts != 1 || recovered.FinishedAt != nil { + t.Fatalf("recovered job %+v", recovered) + } + if chapterRow(t, db, pasted.Chapter.ID).Status != statusPending { + t.Fatal("recovered chapter must be pending") + } + // Recovery reuses the same rows: no second chapter, same task id. + var chapters int64 + db.Model(&Chapter{}).Where("book_id = ?", pasted.Book.ID).Count(&chapters) + if chapters != 1 { + t.Fatalf("recovery left %d chapters, want 1", chapters) + } + drainIngest(t, db) + code, ready := readChapter(t, r, learner.Token, pasted.Chapter.ID) + if code != 200 || ready.Chapter.Status != statusReady || ready.Chapter.OriginalText != fixturePastedText { + t.Fatalf("recovered chapter %+v", ready.Chapter) + } + if ready.Chapter.ID != pasted.Chapter.ID || ready.Chapter.JobID == nil || *ready.Chapter.JobID != pasted.Job.ID { + t.Fatal("recovery must keep the original chapter and task ids") + } +} + +func TestMySQLIngestAttemptsAreBoundedAndManualRetryRestarts(t *testing.T) { + db, r, owner := libraryFixture(t) + learner := newLearner(t, r, owner.Token) + drainIngest(t, db) + code, pasted := pasteBook(t, r, learner.Token, map[string]string{ + "requestId": "fixture-attempt-budget", "title": "Attempt budget", "text": fixturePastedText}) + if code != 201 { + t.Fatalf("paste status %d", code) + } + // Spend the whole budget without any worker running. + if err := db.Model(&IngestJob{}).Where("id = ?", pasted.Job.ID).Update("attempts", maxIngestAttempts).Error; err != nil { + t.Fatal(err) + } + if _, claimed, err := ClaimNextIngestJob(db, time.Now()); err != nil || claimed { + t.Fatalf("claim claimed=%v (%v), want no claim once the budget is used", claimed, err) + } + if _, err := RequeueStaleIngestJobs(db, time.Now()); err != nil { + t.Fatal(err) + } + exhausted := jobRow(t, db, pasted.Job.ID) + if exhausted.Status != statusFailed || exhausted.ErrorReason != reasonAttemptsExhausted { + t.Fatalf("exhausted job %+v", exhausted) + } + code, read := readChapter(t, r, learner.Token, pasted.Chapter.ID) + if code != 200 || read.Chapter.Status != statusFailed || read.Chapter.ErrorReason != reasonAttemptsExhausted { + t.Fatalf("exhausted chapter %+v", read.Chapter) + } + if read.Chapter.ErrorMessage == "" || read.Chapter.OriginalText != "" { + t.Fatalf("exhausted chapter must fail readably without text: %+v", read.Chapter) + } + // The manual retry is still available and restarts the attempt budget. + code, msg, _ := callRaw(t, r, "POST", fmt.Sprintf("/api/v1/jobs/%d/retry", pasted.Job.ID), learner.Token, nil) + if code != 200 { + t.Fatalf("manual retry status %d (%s), want 200", code, msg) + } + retried := jobRow(t, db, pasted.Job.ID) + if retried.Status != statusPending || retried.Attempts != 0 || retried.ErrorReason != "" { + t.Fatalf("retried job %+v", retried) + } + drainIngest(t, db) + code, ready := readChapter(t, r, learner.Token, pasted.Chapter.ID) + if code != 200 || ready.Chapter.Status != statusReady || ready.Chapter.ErrorReason != "" { + t.Fatalf("chapter after manual retry %+v", ready.Chapter) + } + if ready.Chapter.ID != pasted.Chapter.ID { + t.Fatal("manual retry must reuse the same chapter") + } + var chapters int64 + db.Model(&Chapter{}).Where("book_id = ?", pasted.Book.ID).Count(&chapters) + if chapters != 1 { + t.Fatalf("attempt recovery left %d chapters, want 1", chapters) + } +} + +func TestMySQLIngestRecoveryAfterRestart(t *testing.T) { + db, r, owner := libraryFixture(t) + learner := newLearner(t, r, owner.Token) + // The test database is shared with other cases, so empty the queue first: the claim below + // must take this test's own job, not a leftover one. + drainIngest(t, db) + code, pasted := pasteBook(t, r, learner.Token, map[string]string{ + "requestId": "fixture-recovery", "title": "Recovery", "text": fixturePastedText}) + if code != 201 { + t.Fatalf("paste status %d", code) + } + // A crash between claim and finish leaves rows in processing. + job, claimed, err := ClaimNextIngestJob(db, time.Now()) + if err != nil || !claimed { + t.Fatalf("claim failed (claimed=%v): %v", claimed, err) + } + if job.ID != pasted.Job.ID { + t.Fatalf("claimed job %d, want this test's job %d", job.ID, pasted.Job.ID) + } + if chapterRow(t, db, pasted.Chapter.ID).Status != statusProcessing || jobRow(t, db, pasted.Job.ID).Status != statusProcessing { + t.Fatal("claim must persist the processing state") + } + requeued, err := RecoverIngestJobs(db, time.Now()) + if err != nil || requeued < 1 { + t.Fatalf("recovery requeued %d (%v), want at least this test's job", requeued, err) + } + pending := jobRow(t, db, pasted.Job.ID) + if pending.Status != statusPending || pending.Attempts != 1 || pending.FinishedAt != nil { + t.Fatalf("recovered job %+v", pending) + } + if chapterRow(t, db, pasted.Chapter.ID).Status != statusPending { + t.Fatal("recovered chapter must be pending") + } + drainIngest(t, db) + code, ready := readChapter(t, r, learner.Token, pasted.Chapter.ID) + if code != 200 || ready.Chapter.Status != statusReady || ready.Chapter.OriginalText != fixturePastedText { + t.Fatalf("recovered chapter %+v", ready.Chapter) + } + if jobRow(t, db, pasted.Job.ID).Attempts != 2 { + t.Fatal("the recovered job must be processed as a second attempt") + } + var chapters int64 + db.Model(&Chapter{}).Where("book_id = ?", pasted.Book.ID).Count(&chapters) + if chapters != 1 { + t.Fatalf("recovery left %d chapters, want 1", chapters) + } +} + +func TestMySQLPasteRejectsInvalidInputAndLimits(t *testing.T) { + db, r, owner := libraryFixture(t) + learner := newLearner(t, r, owner.Token) + + longTitle := strings.Repeat("T", maxTitleRunes+1) + overLimit := strings.Repeat("a", maxChapterRunes+1) + cases := []struct { + name string + payload map[string]string + }{ + {"missing_request_id", map[string]string{"title": "T", "text": "Text."}}, + {"short_request_id", map[string]string{"requestId": "short", "title": "T", "text": "Text."}}, + {"empty_title", map[string]string{"requestId": randomName("invalid"), "title": " ", "text": "Text."}}, + {"long_title", map[string]string{"requestId": randomName("invalid"), "title": longTitle, "text": "Text."}}, + {"empty_text", map[string]string{"requestId": randomName("invalid"), "title": "T", "text": " \r\n\t "}}, + {"unsupported_language", map[string]string{"requestId": randomName("invalid"), "title": "T", "text": "Text.", "language": "zh"}}, + {"over_limit", map[string]string{"requestId": randomName("invalid"), "title": "T", "text": overLimit}}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + code, msg, _ := callRaw(t, r, "POST", "/api/v1/books", learner.Token, tc.payload) + if code != 400 || msg == "" { + t.Fatalf("status %d (%s), want 400 with a message", code, msg) + } + }) + } + // The boundary itself is accepted, and an oversized body is refused before decoding. + code, accepted := pasteBook(t, r, learner.Token, map[string]string{ + "requestId": "fixture-boundary-limit", "title": "At the limit", "text": strings.Repeat("a", maxChapterRunes)}) + if code != 201 || accepted.Chapter.CharCount != maxChapterRunes { + t.Fatalf("boundary paste status %d charCount %d", code, accepted.Chapter.CharCount) + } + code, msg, _ := callRaw(t, r, "POST", "/api/v1/books", learner.Token, map[string]string{ + "requestId": "fixture-body-limit", "title": "Too large", "text": strings.Repeat("a", maxPasteBodyBytes)}) + if code != 400 || !strings.Contains(msg, "过大") { + t.Fatalf("oversized body status %d (%s), want 400 with 过大", code, msg) + } + var books, chapters int64 + db.Model(&Book{}).Where("owner_id = ?", learner.ID).Count(&books) + db.Model(&Chapter{}).Where("owner_id = ?", learner.ID).Count(&chapters) + if books != 1 || chapters != 1 { + t.Fatalf("rejected input created books=%d chapters=%d", books, chapters) + } +} diff --git a/server/app/lexgo/migration_test.go b/server/app/lexgo/migration_test.go index b474b44..16c9652 100644 --- a/server/app/lexgo/migration_test.go +++ b/server/app/lexgo/migration_test.go @@ -88,11 +88,68 @@ func emptyMigrationDB(t *testing.T) *gorm.DB { return db } +func TestMigrationFromV2PreservesExistingData(t *testing.T) { + db := emptyMigrationDB(t) + // Build a v2 database by hand: this is the state a deployed instance is in before #5. + if err := db.Exec("CREATE TABLE lexgo_schema (id INT PRIMARY KEY,version INT,product VARCHAR(32))").Error; err != nil { + t.Fatal(err) + } + if err := db.Exec("INSERT INTO lexgo_schema VALUES (1,2,'lexgo')").Error; err != nil { + t.Fatal(err) + } + for _, statement := range schemaV2Statements { + if err := db.Exec(statement).Error; err != nil { + t.Fatal(err) + } + } + if err := db.Exec("INSERT INTO sys_user (user_id,username,password,role_id) VALUES (7,'fixture_v2','x',2)").Error; err != nil { + t.Fatal(err) + } + if err := db.Exec("INSERT INTO lexgo_spaces (owner_id,language) VALUES (7,'en')").Error; err != nil { + t.Fatal(err) + } + if err := Migrate(db); err != nil { + t.Fatalf("v2 to v3 migration failed: %v", err) + } + if err := CheckSchema(db); err != nil { + t.Fatal(err) + } + for _, table := range []string{"lexgo_books", "lexgo_chapters", "lexgo_ingest_jobs"} { + var count int64 + db.Raw("SELECT COUNT(*) FROM information_schema.tables WHERE table_schema=DATABASE() AND table_name=?", table).Scan(&count) + if count != 1 { + t.Fatalf("migration did not create %s", table) + } + } + var users, spaces int64 + db.Table("sys_user").Where("user_id = 7").Count(&users) + db.Table("lexgo_spaces").Where("owner_id = 7").Count(&spaces) + if users != 1 || spaces != 1 { + t.Fatalf("migration changed existing rows: users=%d spaces=%d", users, spaces) + } + // A rollback marker set back to 2 can be upgraded again without touching data. + if err := db.Exec("UPDATE lexgo_schema SET version=2 WHERE id=1").Error; err != nil { + t.Fatal(err) + } + if err := Migrate(db); err != nil { + t.Fatalf("re-upgrade failed: %v", err) + } + var version int + db.Raw("SELECT version FROM lexgo_schema WHERE id=1").Scan(&version) + if version != 3 { + t.Fatalf("schema version %d after re-upgrade, want 3", version) + } + db.Table("sys_user").Where("user_id = 7").Count(&users) + if users != 1 { + t.Fatal("re-upgrade changed existing rows") + } +} + func TestMigrationRefusesUnownedOrUnsupportedSchema(t *testing.T) { for _, tc := range []struct{ name, marker string }{ {"empty_marker", ""}, {"negative_version", "INSERT INTO lexgo_schema VALUES (1,-1,'lexgo')"}, - {"future_version", "INSERT INTO lexgo_schema VALUES (1,3,'lexgo')"}, + {"future_version", "INSERT INTO lexgo_schema VALUES (1,4,'lexgo')"}, {"wrong_product", "INSERT INTO lexgo_schema VALUES (1,0,'another-app')"}, } { t.Run(tc.name, func(t *testing.T) { diff --git a/server/app/lexgo/router.go b/server/app/lexgo/router.go index 8103bd8..0e22757 100644 --- a/server/app/lexgo/router.go +++ b/server/app/lexgo/router.go @@ -129,6 +129,12 @@ func Router(db *gorm.DB, now func() time.Time) *gin.Engine { if c.Request.Method == "POST" && c.FullPath() == "/api/v1/accounts" { status = 201 } + // A repeated paste is answered from the first result, so it is not a new resource. + if c.Request.Method == "POST" && (c.FullPath() == "/api/v1/books" || c.FullPath() == "/api/v1/books/:id/chapters") { + if paste, ok := data.(PasteResult); ok && !paste.Duplicate { + status = 201 + } + } respond(c, status, data, err) } } @@ -213,18 +219,99 @@ func Router(db *gorm.DB, now func() time.Time) *gin.Engine { } return updateAccount(tx, id, u.UserId, input) })) + pathID := func(c *gin.Context, message string) (int64, error) { + id, err := strconv.ParseInt(c.Param("id"), 10, 64) + if err != nil || id <= 0 { + return 0, failure(404, message) + } + return id, nil + } + v.POST("/books", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) { + var input PasteBookInput + if err := decodeLimit(c, &input, maxPasteBodyBytes); err != nil { + return nil, err + } + return PasteBook(tx, u.UserId, now(), input) + })) + v.GET("/books", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) { + if c.Request.URL.RawQuery != "" { + return nil, failure(400, "书库不接受查询参数") + } + items, err := ListBooks(tx, u.UserId) + if err != nil { + return nil, err + } + return gin.H{"items": items}, nil + })) + v.GET("/books/:id", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) { + id, err := pathID(c, "书籍不存在") + if err != nil { + return nil, err + } + book, chapters, err := BookDetail(tx, u.UserId, id) + if err != nil { + return nil, err + } + return gin.H{"book": book, "chapters": chapters}, nil + })) + v.POST("/books/:id/chapters", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) { + id, err := pathID(c, "书籍不存在") + if err != nil { + return nil, err + } + var input PasteChapterInput + if err := decodeLimit(c, &input, maxPasteBodyBytes); err != nil { + return nil, err + } + return PasteChapter(tx, u.UserId, id, now(), input) + })) + v.GET("/chapters/:id", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) { + id, err := pathID(c, "章节不存在") + if err != nil { + return nil, err + } + return ChapterDetail(tx, u.UserId, id) + })) + v.GET("/jobs/:id", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) { + id, err := pathID(c, "任务不存在") + if err != nil { + return nil, err + } + job, err := JobDetail(tx, u.UserId, id) + if err != nil { + return nil, err + } + return gin.H{"job": job}, nil + })) + v.POST("/jobs/:id/retry", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) { + id, err := pathID(c, "任务不存在") + if err != nil { + return nil, err + } + job, chapter, err := RetryIngestJob(tx, u.UserId, id, now()) + if err != nil { + return nil, err + } + return gin.H{"job": job, "chapter": chapter}, nil + })) r.NoRoute(func(c *gin.Context) { respond(c, 404, nil, failure(404, "页面或接口不存在")) }) return r } -func decode(c *gin.Context, value any) error { +func decode(c *gin.Context, value any) error { return decodeLimit(c, value, maxJSONBodyBytes) } + +func decodeLimit(c *gin.Context, value any, limit int64) error { if !strings.HasPrefix(c.GetHeader("Content-Type"), "application/json") { return failure(400, "请使用 JSON 请求") } - c.Request.Body = http.MaxBytesReader(c.Writer, c.Request.Body, 16*1024) + c.Request.Body = http.MaxBytesReader(c.Writer, c.Request.Body, limit) d := json.NewDecoder(c.Request.Body) d.DisallowUnknownFields() - if d.Decode(value) != nil { + if err := d.Decode(value); err != nil { + var tooLarge *http.MaxBytesError + if errors.As(err, &tooLarge) { + return failure(400, "内容过大,请减少后重试") + } return failure(400, "请求内容无效") } if d.Decode(new(any)) != io.EOF { diff --git a/server/cmd/lexgo/main.go b/server/cmd/lexgo/main.go index a267461..38a5cd7 100644 --- a/server/cmd/lexgo/main.go +++ b/server/cmd/lexgo/main.go @@ -82,7 +82,7 @@ func run() error { if err = lexgo.Migrate(db); err != nil { return err } - fmt.Println("LexGo schema version 2 ready") + fmt.Println("LexGo schema version 3 ready") return nil } if err = lexgo.CheckSchema(db); err != nil { @@ -112,6 +112,36 @@ func run() error { srv := &http.Server{Addr: addr, Handler: lexgo.Router(db, time.Now), ReadHeaderTimeout: 5 * time.Second, ReadTimeout: 15 * time.Second, WriteTimeout: 15 * time.Second, IdleTimeout: 60 * time.Second, MaxHeaderBytes: 1 << 20} ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) defer stop() + // A single instance owns the worker. Startup recovery returns chapters left in processing + // by an unclean stop, and the running loop sweeps jobs whose finishing transaction failed, + // so a transient database problem does not need a restart. + if _, err = lexgo.RecoverIngestJobs(db, time.Now()); err != nil { + return errors.New("ingestion recovery failed") + } + go func() { + ticker := time.NewTicker(time.Second) + defer ticker.Stop() + for { + jobCtx, cancel := context.WithTimeout(ctx, 30*time.Second) + requeued, recoverErr := lexgo.RequeueStaleIngestJobs(db, time.Now()) + if recoverErr != nil { + if ctx.Err() == nil { + log.Print("ingestion recovery failed; the next sweep retries it") + } + } else if requeued > 0 && ctx.Err() == nil { + log.Printf("requeued %d interrupted ingestion job(s)", requeued) + } + if _, err := lexgo.ProcessIngestJobs(jobCtx, db, time.Now, 20); err != nil && ctx.Err() == nil { + log.Print("ingestion batch stopped before finishing; the claimed job stays processing until the next sweep requeues it") + } + cancel() + select { + case <-ctx.Done(): + return + case <-ticker.C: + } + } + }() go func() { ticker := time.NewTicker(time.Hour) defer ticker.Stop()