From 61d3f63c69334b5793ca21c0678339de3a5dec1d Mon Sep 17 00:00:00 2001 From: QiuSW Date: Fri, 11 Sep 2026 23:37:13 +0800 Subject: [PATCH 1/2] =?UTF-8?q?feat:=20=E4=B8=8A=E4=BC=A0=20TXT=20?= =?UTF-8?q?=E6=96=87=E4=BB=B6=EF=BC=8C=E6=A0=A1=E9=AA=8C=E7=BC=96=E7=A0=81?= =?UTF-8?q?=E5=90=8E=E5=AF=BC=E5=85=A5=E6=9C=AC=E4=BA=BA=E4=B9=A6=E5=BA=93?= =?UTF-8?q?=20(#9)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - POST /api/v1/books/upload 与 /api/v1/books/:id/chapters/upload：multipart 上传， 字段白名单、未知或重复字段拒绝、非 multipart 拒绝、单槽并发门忙时 429 - 只接受 UTF-8（可选 BOM 剥离且不进入原文），UTF-16 按 BOM 识别并给出针对性提示， 非法字节整体拒绝、不使用替换字符；2 MiB 字节上限之后仍套用单章 100000 码点上限 - 文件只在内存中解码，不创建临时文件；客户端文件名不参与任何路径也不入库 - 解码后交给现有 PasteBook/PasteChapter，分章、任务幂等与崩溃恢复与粘贴完全一致 - 学习端导入页新增「粘贴文本 / TXT 文件」来源切换与客户端预检，session.request 支持 FormData - gofmt 整理 #8 引入的 import 顺序与空行 - 同步 Architecture-and-Code-Map、Business-Rules-and-Glossary、 Local-Development-and-Verification、Product-Requirements-Overview 与 Home --- docs/02-architecture-and-code-map.md | 19 +- docs/03-business-rules-and-glossary.md | 18 +- docs/04-local-development-and-verification.md | 22 +- docs/09-product-requirements-overview.md | 10 +- docs/README.md | 6 +- learner/e2e/upload.spec.ts | 76 ++++ learner/src/__tests__/upload.spec.ts | 223 +++++++++++ learner/src/stores/library.ts | 66 ++- learner/src/stores/session.ts | 6 +- learner/src/views/ImportView.vue | 75 +++- server/app/lexgo/review.go | 3 +- server/app/lexgo/review_test.go | 2 +- server/app/lexgo/router.go | 22 +- server/app/lexgo/upload.go | 183 +++++++++ server/app/lexgo/upload_test.go | 376 ++++++++++++++++++ 15 files changed, 1078 insertions(+), 29 deletions(-) create mode 100644 learner/e2e/upload.spec.ts create mode 100644 learner/src/__tests__/upload.spec.ts create mode 100644 server/app/lexgo/upload.go create mode 100644 server/app/lexgo/upload_test.go diff --git a/docs/02-architecture-and-code-map.md b/docs/02-architecture-and-code-map.md index 858d8da..8e401d4 100644 --- a/docs/02-architecture-and-code-map.md +++ b/docs/02-architecture-and-code-map.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件) wiki_page: Architecture-and-Code-Map wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Architecture-and-Code-Map.- -wiki_revision: f7b41a473c138bedf0baff1544e6480872b92d4d -synchronized_at: 2026-09-11T14:55:11Z +wiki_revision: ea8661cfbc172ca67148b6a14d46204ab7033e35 +synchronized_at: 2026-09-11T15:36:44Z # 架构与代码地图 @@ -272,3 +272,18 @@ schema v6 新增 `lexgo_term_reviews`（每个个人词条一行排期：`due_at 学习端新增 `/review` 路由与书库、阅读器顶栏的「到期复习」入口；`stores/review.ts` 维护队列、本轮计数、评分与重学，`ReviewCard.vue`/`ReviewView.vue` 呈现正面（词＋挖空例句）、答案面（个人释义＋例句＋三个评分按钮）、完成页与空队列页。一次评分对应一个 `answerId`，失败重试复用同一个；换词后重新生成。切换账号或退出登录会清空队列、计数与当前卡片。 参考：[LinguaCafe Review.vue](https://github.com/simjanos-dev/LinguaCafe/blob/c1ea298ce40c65b9dd33e9b26fd2e52fae66f2c8/resources/js/components/Review/Review.vue)。上游的随机抽卡、阶段降级与快捷键不属于本单；#8 只实现本项目的固定间隔、固定顺序与幂等作答。 + +## #9 TXT 上传导入（2026-09-11） + +`server/app/lexgo/upload.go` 负责把上传的 TXT 解码后交给与粘贴相同的核心：解码、multipart 解析与两条路由，schema 无变化（沿用 #5 的 `lexgo_books`/`lexgo_chapters`/`lexgo_ingest_jobs`）。文件只在内存中存在，不写临时文件，客户端文件名不参与任何路径也不入库。 + +| 接口 | 权限与输入/输出 | +|---|---| +| POST /api/v1/books/upload | 本人；multipart：`requestId`、`title`、`language`（可省略，省略即英语）、`file`；新建书籍与首章 | +| POST /api/v1/books/:id/chapters/upload | 本人且本人书籍；multipart：`requestId`、`title`、`file`；追加一章；不接受 `language` | + +字段白名单之外的字段、重复字段、缺失 `file`、非 multipart 请求都返回 400；201 新建、200 重复、409 同编号换内容、404 他人书籍、401 未登录、429 已有文件正在上传（单槽并发门）。响应体与粘贴路径同为 `PasteResult`，所以学习端复用同一套跳转与轮询逻辑。 + +解码规则见业务规则页；实现上 `decodeTextUpload` 先按 UTF-16 BOM 识别并给出针对性提示，再剥离可选 UTF-8 BOM，然后用 `utf8.Valid` 整体校验，最后交给 `validatePaste`（非空、≤100000 码点）。因此上传与粘贴共享同一分章与任务规则：一次提交一章，`requestId` + 内容 SHA 幂等，worker 只发布已落库的原文。 + +学习端 `ImportView.vue` 增加「粘贴文本 / TXT 文件」来源切换（沿用已验收 v1 的切换与状态行），`stores/library.ts` 增加 `upload()` 与 `fileProblem`/`fileSizeLabel`，`session.request` 支持 `FormData`（multipart 请求不再被 JSON 化，边界由浏览器提供）。客户端预检只提前反馈，服务端结论为最终结论。 diff --git a/docs/03-business-rules-and-glossary.md b/docs/03-business-rules-and-glossary.md index 43c3e02..50e085e 100644 --- a/docs/03-business-rules-and-glossary.md +++ b/docs/03-business-rules-and-glossary.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件) wiki_page: Business-Rules-and-Glossary wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Business-Rules-and-Glossary.- -wiki_revision: 1d90013d883de9339865694e0bb0026f9edd3acc -synchronized_at: 2026-09-11T14:44:50Z +wiki_revision: 76289713c11902383031764c90ae9da90bd0ce07 +synchronized_at: 2026-09-11T15:36:44Z # 业务规则与术语 @@ -192,3 +192,17 @@ exact优先；未命中再按WordNet异常表/词尾规则查候选，词性顺 **归属与错误**：词条归属由服务端按会话裁决，`owner` 不接受客户端输入；他人词条与不存在的词条统一 404，未登录 401，未知评分/缺少 `expectedDueAt`/未知字段 400，词条已变成 `已知`/`忽略` 409。复习接口不写审计日志：答题属于私人学习内容。 **范围边界**：本单只做单词复习。短语复习归 #11，到期范围筛选与策略配置界面（X11）不做，练习模式（X08）不做，进度与「已知词/待复习」计数归 #13。#8 不改变 #7 定下的身份口径：复习状态按规范化词形归属，仍不按 WordNet lemma 候选合并。 + +## #9 TXT 上传规则（2026-09-11） + +**支持的编码**：只接受 UTF-8，允许带可选 UTF-8 BOM。BOM 在解码时剥离，不进入原文；其余字节必须整体合法，任何非法序列**直接拒绝**，绝不使用替换字符，因此章节里不会出现学习者没有写过的乱码。UTF-16（含记事本「另存为 Unicode」产生的大小端 BOM）单独识别并提示「请另存为 UTF-8 后重试」；GB18030、Latin-1 等其他编码按非法 UTF-8 拒绝。UTF-16 支持不在本单范围。 + +**换行与空白**：不做任何归一化，CRLF、LF、制表符、行尾空格与空行按原字节保存，阅读器以 `pre-wrap` 原样呈现——与粘贴路径一致。 + +**大小上限**：文件字节上限 2 MiB；解码后再套用单章上限（非空、≤100000 码点）。超限返回 400 并明确提示，不截断、不部分导入。2 MiB 对 100000 码点的 UTF-8 文本有足够余量。 + +**文件生命周期与路径**：上传内容只存在于内存中，解码后直接进入导入事务；服务端**不创建临时文件**，所以没有需要清理或可能泄漏的文件；**客户端文件名不参与任何文件系统路径、也不写入数据库**，它只在选择文件时用于显示（并可预填标题）。因此文件名即使写成 `..\..\windows\system32\evil.txt` 也不会影响任何存储位置。上传并发按单槽限制，忙时返回 429。 + +**导入与幂等**：上传与粘贴共用同一套规则——一次提交一章，`requestId` + 内容 SHA 保证重复上传同一文件只产生一章（返回第一次的章节并标记 `duplicate`），同一 `requestId` 换成其他内容返回 409。任务状态、失败重试与崩溃恢复沿用 #5 的任务机制，不新增状态。标题规则与粘贴完全相同（去空白后非空、≤120 字符）；省略 `language` 时默认英语，与粘贴一致；追加章节不接受 `language`。 + +**范围边界**：不包含 EPUB、PDF、字幕与其他文件格式；不做按空行自动分章；不做 UTF-16/GB18030 转码；不做断点续传；不把来源文件名持久化（若将来需要「导入来源」溯源，另立范围）。 diff --git a/docs/04-local-development-and-verification.md b/docs/04-local-development-and-verification.md index dacbc51..e586692 100644 --- a/docs/04-local-development-and-verification.md +++ b/docs/04-local-development-and-verification.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件) wiki_page: Local-Development-and-Verification wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Local-Development-and-Verification.- -wiki_revision: 1192d23a8198e981961adfc6066f0b3bc3b84058 -synchronized_at: 2026-09-11T14:44:50Z +wiki_revision: 8c0886a74112ea7bff734c04775b56c571797c3d +synchronized_at: 2026-09-11T15:36:44Z # 本地开发与验证 @@ -380,3 +380,21 @@ node --test spikes/english/view.test.mjs **流程记录（R6）**：评论 7769 的方案写的是在 `lexgo_terms` 上增加列，实际实现改为独立表 `lexgo_term_reviews`（加法迁移可重试、不对既有表做 ALTER）。该变更在实施评论 7776 与 Wiki 中说明了原因，但没有按「数据结构变化先更新工单」的要求在实施前追加变更评论；本页与上文契约按实际实现记录，方案评论中的「新增列均有默认值」以独立表为准。 整改后重跑：Go 单元与集成测试（专用库 `lexgo_test_issue8`）44 个顶层用例全部通过、0 跳过；学习端 73 项单测、类型检查、构建与 5 项 E2E 通过；管理端 31 项与 lint 通过；治理 56 项与严格检查通过；真实 API＋MySQL 42 项检查通过（新增 4 项针对 R3 与重放契约）；真实浏览器复核面板保存与复习闭环通过。截图 `.local/evidence/issue8-fixed-summary.png`。 + +## #9 验证与迁移（2026-09-11） + +仓库根执行；Go 工具链由 `python scripts/server.py` 固定 go1.26.5。本单使用专用测试库 lexgo_test_issue9，不借用其他测试库。本单**不改动数据库结构**，所以没有迁移步骤，回退只需换回旧二进制。 + +| 命令 | 结果 | +|---|---| +| `go vet ./...` | 通过 | +| `LEXGO_TEST_DB_NAME=lexgo_test_issue9 python scripts/server.py test-integration` | 50 个顶层用例全部通过、0 跳过；含 #9 新增 6 个上传用例 | +| `cd learner`：`npx vitest --run` / `npx vue-tsc --build` / `npx pnpm run build` / `npx playwright test` | 80 项单测、类型检查、构建、6 项 E2E 全部通过 | +| `cd admin`：`npx pnpm test` / `npx pnpm lint` | 31 项与 lint 通过；管理端本单无代码改动 | +| `python -m unittest discover -s tests` / `python dev_scripts/harness.py check --strict` | 56 项与严格检查通过 | + +覆盖内容：有效 UTF-8（含 CRLF、制表符、弯引号、em dash、省略号、emoji、组合字符）字节级往返、UTF-8 BOM 剥离且不进原文、只有 BOM、非法 UTF-8、Latin-1、UTF-16 大小端、NUL 字节、空文件、只有空白、超限与恰好边界（2 MiB、100000 码点）、缺 `file`、缺标题、缺或错误 `language`、缺或过短 `requestId`、未知字段、追加路径携带 `language`、非 multipart 请求、未登录、恶意文件名不影响存储、重复上传只产生一章、同编号换内容 409、追加他人书籍 404、两账号隔离、上传与粘贴共用同一任务管线。 + +真实链路验证：真实 Go API＋真实 MySQL 共 26 项检查通过（凭据只从本机安全配置读入进程），覆盖有效文件与阅读器原文逐字节一致、BOM 不进入原文、五类无效文件、越权与未登录拒绝、两账号隔离、恶意文件名不泄漏；随后用临时 Playwright 用例在真实学习端完成「登录→切换 TXT→选择真实 UTF-8 文件→上传→处理中就绪→阅读器原文逐字符一致」的闭环，并复核 UTF-16 文件在浏览器预检阶段被拒。截图保存在本机 `.local/evidence/`（issue9-invalid-encoding.png、issue9-upload-processing.png、issue9-reader.png），临时用例运行后删除。 + +未验证：真实手机触屏详细证据与完整备份恢复演练仍属既有缺口（#14/#15）；本单只用桌面浏览器检查。大文件并发上传只按单槽并发门设计，没有做多用户压力测试。UTF-16/GB18030 转码与按空行自动分章不在本单。 diff --git a/docs/09-product-requirements-overview.md b/docs/09-product-requirements-overview.md index a3fb5d3..86f76fd 100644 --- a/docs/09-product-requirements-overview.md +++ b/docs/09-product-requirements-overview.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件) wiki_page: Product-Requirements-Overview wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Product-Requirements-Overview.- -wiki_revision: e6a1f2037516203e9144159af77b25e87638b6fd -synchronized_at: 2026-09-11T14:55:11Z +wiki_revision: f5b38f2d527cbc21eac4512f7c439c013bb221d9 +synchronized_at: 2026-09-11T15:36:45Z # 产品需求总览 @@ -251,3 +251,9 @@ F07 的个人词语记录已于 2026-09-11 通过用户验收：阅读器可以 F10 的单词到期复习已于 2026-09-11 通过用户验收（包含独立审核整改 R1～R3）：按固定间隔表取本人当前语言的到期词条，正面显示词与挖空例句，显示答案后按「认识/答对」「不认识/答错」「再学一次」评分；答对升级并排下次复习，答错降级并立即回到本轮，再学一次不改等级并回到本轮。重复提交、网络重发与双标签页都不会重复更新次数和间隔；已知与忽略的词条不入队。管理端无改动。 仍未实现并留给后续工单：短语复习（#11）、词汇库搜索与编辑（#12）、阅读完成与进度统计（#13）、TXT 导入（#9）、书籍章节编辑删除（#10）。复习范围筛选与策略配置（X11）、练习模式（X08）、FSRS 仍在范围外。 + +## #9 交付范围更新（2026-09-11） + +F03 的 TXT 文件导入已实现，待用户验收：学习端导入页新增「粘贴文本 / TXT 文件」来源切换，选择 UTF-8 的 .txt 文件后经大小、空文件与编码校验进入与粘贴相同的处理与阅读流程，失败可重试。只支持 UTF-8（允许可选 BOM）且不替换损坏字符；UTF-16 与其他编码会被明确拒绝；文件只在内存中解码、不写临时文件，客户端文件名不参与任何路径也不入库；重复上传同一文件只产生一章。schema 无变化。 + +仍未实现并留给后续工单：书籍与章节的编辑删除（#10）、短语选择与保存（#11）、词汇库搜索与编辑（#12）、阅读完成与进度（#13）、桌面与手机体验补齐（#14）、自托管试用交付与完整恢复（#15）。EPUB/PDF/字幕、UTF-16 转码、按空行自动分章与断点续传不在本单范围。 diff --git a/docs/README.md b/docs/README.md index 3698db3..39c6983 100644 --- a/docs/README.md +++ b/docs/README.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件) wiki_page: Home wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Home -wiki_revision: 85c66bb518fcbb166b916549aa6e655215e0ed85 -synchronized_at: 2026-09-11T14:55:10Z +wiki_revision: 75de70aad90f02deeadbed301c9fe85ded1fec6e +synchronized_at: 2026-09-11T15:36:44Z # LexGo 文档入口 @@ -74,3 +74,5 @@ Quant-UX 原型 v1 已通过用户验收。[桌面预览](https://qux.ilapage.cn #7 个人词条已实现并于 2026-09-11 通过用户验收（schema v5 新增 lexgo_terms）：阅读器可保存释义、例句与状态，同一词形在本人其他章节显示一致高亮，两个账号数据独立；保存幂等，跨账号与篡改身份均被拒绝；PR #26 已 fast-forward-only 合入 main。 #8 单词到期复习已于 2026-09-11 通过用户验收（schema v6 新增 lexgo_term_reviews 与 lexgo_review_answers）：固定间隔表（1/2/4/7/15/30/60 天）、答对升级、答错或再学立即回队、已知与忽略不入队；重复提交、网络重发与双标签页都只记账一次，到期判定用 UTC 绝对时刻而不引入本地日边界。学习端新增「到期复习」入口，卡片正面显示词与挖空例句、答案面显示个人释义，并有完成页与空队列页。独立审核指出的并发同键 500、编辑文本重排复习与面板保存重置等级三项已整改并复测；PR #27 已 fast-forward-only 合入 main。第 3 阶段「首条学习闭环」#5～#8 全部验收。 + +#9 TXT 文件导入已实现，待用户验收：导入页新增「粘贴文本 / TXT 文件」来源切换，只接受 UTF-8（允许可选 BOM）且不替换损坏字符，UTF-16 与其他编码会被明确拒绝；文件只在内存中解码、不写临时文件，客户端文件名不参与任何路径也不入库；上传与粘贴共用同一分章、任务与幂等规则，重复上传同一文件只产生一章。本次没有数据库结构变化。 diff --git a/learner/e2e/upload.spec.ts b/learner/e2e/upload.spec.ts new file mode 100644 index 0000000..586edb3 --- /dev/null +++ b/learner/e2e/upload.spec.ts @@ -0,0 +1,76 @@ +import { expect, test } from '@playwright/test' + +// The TXT upload path against a mocked API: pre-checks, multipart body and the hand-off to +// the same processing screen the paste path uses. +test('upload a UTF-8 TXT file and open the created book', async ({ page }) => { + const user = { id: 42, username: 'fictional-uploader', role: 'learner' } + const book = { id: 1, title: 'Studio Notes', language: 'en' } + const timestamps = { createdAt: '2026-01-01T00:00:00Z', updatedAt: '2026-01-01T00:00:00Z' } + const pasted = 'Mira opened the workshop.\r\n\r\n\tThe sign read “A small step…”\n' + let uploaded = false + + await page.route('**/api/v1/**', async route => { + const path = new URL(route.request().url()).pathname + const method = route.request().method() + let data: unknown = null + let status = 200 + if (path === '/api/v1/login') data = { token: 'fictional-session', user } + else if (path === '/api/v1/me') data = user + else if (path === '/api/v1/space') data = { ownerId: user.id, language: 'en' } + else if (path === '/api/v1/books' && method === 'GET') data = { items: uploaded ? [{ ...book, chapterCount: 1, pendingCount: 0, processingCount: 0, readyCount: 1, failedCount: 0, ...timestamps }] : [] } + else if (path === '/api/v1/books/upload') { + // The upload must arrive as multipart: the fields and the file are inspected directly. + expect(route.request().headers()['content-type']).toContain('multipart/form-data') + const raw = route.request().postData() ?? '' + expect(raw).toContain('name="requestId"') + expect(raw).toContain('name="language"') + expect(raw).toContain('name="title"') + expect(raw).toContain('Studio Notes') + expect(raw).toContain('filename="notes.txt"') + expect(raw).toContain('Mira opened the workshop.') + uploaded = true + status = 201 + data = { + book, + chapter: { id: 9, bookId: 1, ordinal: 1, title: 'Studio Notes', status: 'pending', charCount: [...pasted].length, errorReason: '', errorMessage: '', jobId: 5, ...timestamps }, + job: { id: 5, bookId: 1, chapterId: 9, status: 'pending', attempts: 0, errorReason: '', errorMessage: '', ...timestamps }, + duplicate: false, + } + } else if (path === '/api/v1/books/1') data = { book, chapters: [{ id: 9, bookId: 1, ordinal: 1, title: 'Studio Notes', status: 'ready', charCount: [...pasted].length, errorReason: '', errorMessage: '', jobId: 5, ...timestamps }] } + else if (path === '/api/v1/chapters/9') data = { book, chapter: { id: 9, bookId: 1, ordinal: 1, title: 'Studio Notes', status: 'ready', charCount: [...pasted].length, errorReason: '', errorMessage: '', jobId: 5, contentSha256: 'fictional-sha', originalText: pasted, ...timestamps }, navigation: { previousChapterId: null, nextChapterId: null } } + await route.fulfill({ status, json: { code: 200, data } }) + }) + + await page.goto('/') + await page.getByLabel('账号').fill(user.username) + await page.getByLabel('密码', { exact: true }).fill('fictional-password') + await page.getByRole('button', { name: '登录', exact: true }).click() + await expect(page.getByRole('heading', { name: '我的书库' })).toBeVisible() + + await page.getByRole('button', { name: '导入内容' }).click() + // Element Plus hides the native radio behind a styled span, so the label is what a person + // clicks; the input still carries the checked state. + await page.locator('label.el-radio', { hasText: 'TXT 文件' }).click() + await expect(page.getByRole('radio', { name: 'TXT 文件' })).toBeChecked() + await expect(page.locator('textarea#text')).toHaveCount(0) + await expect(page.getByText('仅支持 UTF-8')).toBeVisible() + + // An unusable file is refused in the browser, before any request is made. + await page.setInputFiles('[data-testid="file-input"]', { name: 'notes.md', mimeType: 'text/markdown', buffer: Buffer.from('# heading\n') }) + await expect(page.getByText('请选择 .txt 文件。')).toBeVisible() + + // A UTF-8 file is accepted and its metadata is shown; the title comes from the file name. + await page.setInputFiles('[data-testid="file-input"]', { name: 'notes.txt', mimeType: 'text/plain', buffer: Buffer.from(pasted, 'utf8') }) + await expect(page.getByTestId('file-info')).toContainText('notes.txt · UTF-8') + await expect(page.getByLabel('标题')).toHaveValue('notes') + await page.getByLabel('标题').fill('Studio Notes') + + await page.getByRole('button', { name: '上传并处理' }).click() + await expect(page).toHaveURL(/\/books\/1$/) + await expect(page.getByText('已就绪')).toBeVisible() + await page.getByRole('link', { name: 'Studio Notes' }).click() + const readerText = page.locator('.reader-text') + await expect(readerText).toBeVisible() + // The uploaded bytes reached the reader unchanged. + expect(await readerText.evaluate(element => element.textContent)).toBe(pasted) +}) diff --git a/learner/src/__tests__/upload.spec.ts b/learner/src/__tests__/upload.spec.ts new file mode 100644 index 0000000..3e3cd42 --- /dev/null +++ b/learner/src/__tests__/upload.spec.ts @@ -0,0 +1,223 @@ +import { afterEach, beforeEach, describe, expect, it, vi, type MockInstance } from 'vitest' +import { flushPromises, mount, type VueWrapper } from '@vue/test-utils' +import { createPinia, setActivePinia } from 'pinia' +import { createRouter, createMemoryHistory, type Router } from 'vue-router' +import ImportView from '../views/ImportView.vue' +import { fileProblem, fileSizeLabel, TXT_MAX_BYTES, useLibraryStore } from '../stores/library' +import { useSessionStore } from '../stores/session' + +const user = { id: 42, username: 'fictional-uploader', role: 'learner' as const } +const book = { id: 1, title: 'Uploaded Book', language: 'en' } +const chapter = { id: 9, bookId: 1, ordinal: 1, title: 'Uploaded Book', status: 'pending', charCount: 12, errorReason: '', errorMessage: '', jobId: 5, createdAt: '', updatedAt: '' } +const timestamps = { createdAt: '2026-01-01T00:00:00Z', updatedAt: '2026-01-01T00:00:00Z' } +const ok = (data: unknown) => new Response(JSON.stringify({ code: 200, data })) +let wrapper: VueWrapper | undefined + +function stub(name: string) { + return { template: `
${name}
` } +} + +async function viewAt(path: string): Promise { + const router = createRouter({ + history: createMemoryHistory(), + routes: [ + { path: '/', component: stub('LibraryStub') }, + { path: '/import', component: stub('ImportStub') }, + { path: '/books/:id', component: stub('BookStub') }, + ], + }) + await router.push(path) + await router.isReady() + return router +} + +/** jsdom has no file picker, so the input's files are set directly before dispatching change. */ +async function chooseFile(view: VueWrapper, file: File | null): Promise { + const input = view.get('[data-testid="file-input"]') + Object.defineProperty(input.element, 'files', { value: file ? [file] : [], configurable: true }) + await input.trigger('change') + await flushPromises() +} + +/** The TXT toggle is an Element Plus radio group; its hidden input carries the value. */ +async function useTxt(view: VueWrapper): Promise { + for (const input of view.findAll('input[type="radio"]')) { + if ((input.element as HTMLInputElement).value === 'txt') { + await input.setValue() + await flushPromises() + return + } + } + throw new Error('TXT 文件 toggle not found') +} + +function uploadResponse() { + return ok({ book, chapter, job: { id: 5, bookId: 1, chapterId: 9, status: 'pending', attempts: 0, errorReason: '', errorMessage: '', ...timestamps }, duplicate: false }) +} + +describe('txt upload pre-checks', () => { + it('mirrors the server limits for the file that was picked', () => { + expect(fileProblem({ name: 'reading.txt', size: 2048 })).toBe('') + expect(fileProblem({ name: 'READING.TXT', size: 1 })).toBe('') + expect(fileProblem({ name: 'reading.md', size: 2048 })).toContain('.txt') + expect(fileProblem({ name: 'empty.txt', size: 0 })).toContain('空的') + expect(fileProblem({ name: 'big.txt', size: TXT_MAX_BYTES + 1 })).toContain('2 MiB') + expect(fileProblem({ name: 'limit.txt', size: TXT_MAX_BYTES })).toBe('') + expect(fileSizeLabel(512)).toBe('512 B') + expect(fileSizeLabel(2048)).toBe('2 KB') + expect(fileSizeLabel(1.5 * 1024 * 1024)).toBe('1.5 MB') + }) +}) + +describe('txt upload view', () => { + beforeEach(() => { + setActivePinia(createPinia()) + sessionStorage.clear() + vi.restoreAllMocks() + useSessionStore().user = { ...user } + }) + afterEach(() => { + wrapper?.unmount() + wrapper = undefined + useLibraryStore().stopPolling() + }) + + it('shows the file picker instead of the textarea and reports the picked file', async () => { + const router = await viewAt('/import') + wrapper = mount(ImportView, { global: { plugins: [router] } }) + await flushPromises() + // Paste mode keeps the textarea; TXT mode replaces it with the picker. + expect(wrapper.find('textarea#text').exists()).toBe(true) + expect(wrapper.find('[data-testid="file-input"]').exists()).toBe(false) + await useTxt(wrapper) + expect(wrapper.find('textarea#text').exists()).toBe(false) + expect(wrapper.find('[data-testid="file-input"]').exists()).toBe(true) + expect(wrapper.text()).toContain('仅支持 UTF-8') + + await chooseFile(wrapper, new File(['Mira opened the workshop.\n'], 'reading.txt', { type: 'text/plain' })) + expect(wrapper.get('[data-testid="file-info"]').text()).toBe('reading.txt · UTF-8 · 26 B') + // The title is prefilled from the file name and stays editable. + expect((wrapper.get('input#title').element as HTMLInputElement).value).toBe('reading') + }) + + it('refuses a file the server would refuse, before anything is sent', async () => { + const fetchMock = vi.spyOn(globalThis, 'fetch') + const router = await viewAt('/import') + wrapper = mount(ImportView, { global: { plugins: [router] } }) + await flushPromises() + await useTxt(wrapper) + + await chooseFile(wrapper, new File(['# not a txt file\n'], 'notes.md')) + expect(wrapper.text()).toContain('请选择 .txt 文件。') + // A file that is not valid UTF-8 is rejected by the preview decode. + await chooseFile(wrapper, new File([new Uint8Array([0x63, 0x61, 0x66, 0xe9, 0x0a])], 'latin1.txt')) + expect(wrapper.text()).toContain('文件不是 UTF-8 编码') + await wrapper.find('input#title').setValue('Latin One') + await wrapper.find('form').trigger('submit') + await flushPromises() + expect(fetchMock).not.toHaveBeenCalled() + expect(wrapper.text()).toContain('文件不是 UTF-8 编码') + + // No file at all is also refused locally. + await chooseFile(wrapper, null) + await wrapper.find('form').trigger('submit') + await flushPromises() + expect(fetchMock).not.toHaveBeenCalled() + expect(wrapper.text()).toContain('请选择要导入的 TXT 文件。') + }) + + it('uploads the file as multipart and opens the created book', async () => { + const fetchMock = vi.spyOn(globalThis, 'fetch').mockResolvedValueOnce(uploadResponse()) + const router = await viewAt('/import') + wrapper = mount(ImportView, { global: { plugins: [router] } }) + await flushPromises() + await useTxt(wrapper) + const file = new File(['Mira opened the workshop.\n'], 'reading.txt', { type: 'text/plain' }) + await chooseFile(wrapper, file) + await wrapper.find('input#title').setValue('上传的虚构章节') + await wrapper.find('form').trigger('submit') + await flushPromises() + + expect(fetchMock).toHaveBeenCalledTimes(1) + const [url, init] = fetchMock.mock.calls[0]! + expect(String(url)).toBe('/api/v1/books/upload') + expect(init?.method).toBe('POST') + // A multipart body must not be replaced by JSON and must keep the browser's boundary. + expect(init?.body).toBeInstanceOf(FormData) + expect((init?.headers as Record)['Content-Type']).toBeUndefined() + const body = init?.body as FormData + expect(body.get('requestId')).toMatch(/^[0-9a-f-]{36}$/) + expect(body.get('title')).toBe('上传的虚构章节') + expect(body.get('language')).toBe('en') + expect((body.get('file') as File).name).toBe('reading.txt') + expect(await (body.get('file') as File).text()).toBe('Mira opened the workshop.\n') + expect(router.currentRoute.value.path).toBe('/books/1') + // The form starts clean, so the same file is not submitted twice by accident. + expect((wrapper.get('[data-testid="file-input"]').element as HTMLInputElement).value).toBe('') + }) + + it('reuses one request id when the same upload is retried after a failure', async () => { + let attempt = 0 + const fetchMock = vi.spyOn(globalThis, 'fetch').mockImplementation(async () => { + attempt += 1 + if (attempt === 1) throw new Error('上传中断') + return uploadResponse() + }) + const router = await viewAt('/import') + wrapper = mount(ImportView, { global: { plugins: [router] } }) + await flushPromises() + await useTxt(wrapper) + await chooseFile(wrapper, new File(['Body.\n'], 'retry.txt')) + await wrapper.find('input#title').setValue('Retry Upload') + await wrapper.find('form').trigger('submit') + await flushPromises() + expect(wrapper.text()).toContain('上传中断') + // The picked file stays selected so the learner can retry without choosing it again. + expect(wrapper.get('[data-testid="file-info"]').text()).toContain('retry.txt') + + await wrapper.find('form').trigger('submit') + await flushPromises() + const ids = fetchMock.mock.calls.map(call => (call[1]?.body as FormData).get('requestId')) + expect(ids).toHaveLength(2) + expect(ids[0]).toBe(ids[1]) + expect(router.currentRoute.value.path).toBe('/books/1') + }) + + it('appends an uploaded chapter to a chosen book without a language field', async () => { + const fetchMock = vi.spyOn(globalThis, 'fetch').mockImplementation(async (input, init) => { + const url = String(input) + if (url.endsWith('/books')) return ok({ items: [{ ...book, chapterCount: 1, pendingCount: 0, processingCount: 0, readyCount: 1, failedCount: 0, ...timestamps }] }) + expect(url).toBe('/api/v1/books/1/chapters/upload') + const body = init?.body as FormData + expect(body.get('language')).toBeNull() + return ok({ chapter: { ...chapter, id: 10, ordinal: 2 }, job: { id: 6, bookId: 1, chapterId: 10, status: 'pending', attempts: 0, errorReason: '', errorMessage: '', ...timestamps }, duplicate: false }) + }) + const router = await viewAt('/import?book=1') + wrapper = mount(ImportView, { global: { plugins: [router] } }) + await flushPromises() + await useTxt(wrapper) + await chooseFile(wrapper, new File(['Second chapter.\n'], 'second.txt')) + await wrapper.find('input#title').setValue('Appended Chapter') + await wrapper.find('form').trigger('submit') + await flushPromises() + + const posts = fetchMock.mock.calls.filter(call => String(call[0]).includes('/upload')) + expect(posts).toHaveLength(1) + expect(router.currentRoute.value.path).toBe('/books/1') + }) + + it('keeps the server message and the form when the upload is rejected', async () => { + vi.spyOn(globalThis, 'fetch').mockResolvedValue(new Response(JSON.stringify({ code: 400, msg: '文件不是 UTF-8 编码，请另存为 UTF-8 后重试' }), { status: 400 })) + const router = await viewAt('/import') + wrapper = mount(ImportView, { global: { plugins: [router] } }) + await flushPromises() + await useTxt(wrapper) + await chooseFile(wrapper, new File(['Body.\n'], 'server-rejects.txt')) + await wrapper.find('input#title').setValue('Server Rejects') + await wrapper.find('form').trigger('submit') + await flushPromises() + expect(wrapper.text()).toContain('文件不是 UTF-8 编码，请另存为 UTF-8 后重试') + expect(wrapper.get('[data-testid="file-info"]').text()).toContain('server-rejects.txt') + expect(router.currentRoute.value.path).toBe('/import') + }) +}) diff --git a/learner/src/stores/library.ts b/learner/src/stores/library.ts index 12e363f..ac27d76 100644 --- a/learner/src/stores/library.ts +++ b/learner/src/stores/library.ts @@ -58,6 +58,8 @@ export interface SubmitInput { title: string; text: string; target: SubmitTarget export const POLL_INTERVAL_MS = 1500 export const TITLE_MAX_LENGTH = 120 export const TEXT_MAX_CODE_POINTS = 100000 +// Mirrors the server limit for one TXT upload. +export const TXT_MAX_BYTES = 2 * 1024 * 1024 export const NOT_FOUND_MESSAGE = '内容不存在。' export const LANGUAGE_LABEL = '英语' export const LANGUAGE_CODE = 'en' @@ -92,6 +94,24 @@ export function textProblem(text: string): string { return '' } +/** + * Client-side pre-check for a TXT upload. The server validates the file again and stays the + * only authority; this only tells the learner about an obviously unusable choice earlier. + */ +export function fileProblem(file: { name: string; size: number }): string { + if (!/\.txt$/i.test(file.name)) return '请选择 .txt 文件。' + if (file.size === 0) return '文件是空的，请选择包含英文正文的 UTF-8 TXT。' + if (file.size > TXT_MAX_BYTES) return 'TXT 文件不能超过 2 MiB。' + return '' +} + +/** A readable size for the selected file, e.g. `2 KB` or `1.5 MB`. */ +export function fileSizeLabel(bytes: number): string { + if (bytes < 1024) return `${bytes} B` + if (bytes < 1024 * 1024) return `${Math.round(bytes / 1024)} KB` + return `${(bytes / (1024 * 1024)).toFixed(1)} MB` +} + /** A failed chapter can be retried as soon as the API told us its job id. */ export function canRetry(chapter: Pick): boolean { return chapter.status === 'failed' && chapter.jobId !== null @@ -112,6 +132,7 @@ interface SubmitBookBody { requestId: string; title: string; text: string; langu // An appended chapter owns the language of its book, so the append contract has no language // field; the server rejects unknown fields, and a client that sends one gets HTTP 400. interface SubmitChapterBody { requestId: string; title: string; text: string } +export interface UploadInput { title: string; target: SubmitTarget; file: File } interface Created { bookId: number; chapter: ChapterSummary } function emptyNavigation(): ChapterNavigation { @@ -365,6 +386,49 @@ export const useLibraryStore = defineStore('library', () => { } } + /** + * Uploads one TXT file. The file and the title share the request id discipline of a paste, + * so a repeated upload of the same file answers with the chapter it already created. + */ + async function upload(input: UploadInput): Promise { + const title = input.title.trim() + const problem = titleProblem(title) || fileProblem(input.file) + if (problem) { + submitError.value = problem + throw new Error(problem) + } + const key = `upload\n${title}\n${input.file.name}\n${input.file.size}\n${input.file.lastModified}` + if (key !== submissionKey || submissionRequestId === '') { + submissionKey = key + submissionRequestId = crypto.randomUUID() + } + const requestId = submissionRequestId + const form = new FormData() + form.append('requestId', requestId) + form.append('title', title) + // Only the new-book contract carries a language; appending inherits the book's language. + const path = input.target.mode === 'new' ? 'books/upload' : `books/${input.target.bookId}/chapters/upload` + if (input.target.mode === 'new') form.append('language', LANGUAGE_CODE) + form.append('file', input.file, input.file.name) + + const version = generation + const owner = ownerId() + submitting.value = true + submitError.value = '' + try { + const result = await session.request<{ book?: BookRef; chapter: ChapterSummary }>(path, 'POST', form) + if (isStale(version, owner)) throw new Error('登录状态已变化，请重新提交。') + submissionKey = '' + submissionRequestId = '' + return result.book?.id ?? result.chapter.bookId + } catch (reason) { + if (!isStale(version, owner)) submitError.value = failureMessage(reason, '上传失败，请稍后重试。') + throw reason instanceof Error ? reason : new Error('上传失败，请稍后重试。') + } finally { + if (!isStale(version, owner)) submitting.value = false + } + } + /** The job id comes from the chapter itself, wherever that chapter was loaded from. */ function jobIdOf(chapterId: number): number | null { const target = chapters.value.find(item => item.id === chapterId) @@ -461,7 +525,7 @@ export const useLibraryStore = defineStore('library', () => { book, chapters, bookLoading, bookError, chapter, chapterBook, navigation, chapterLoading, chapterError, submitting, submitError, retryingChapterId, readerText, - loadBooks, loadBook, loadChapter, submit, retryChapter, + loadBooks, loadBook, loadChapter, submit, upload, retryChapter, stopPolling, closeBook, closeChapter, reset, } }) diff --git a/learner/src/stores/session.ts b/learner/src/stores/session.ts index 1b6e0de..9ff472b 100644 --- a/learner/src/stores/session.ts +++ b/learner/src/stores/session.ts @@ -36,10 +36,12 @@ export const useSessionStore = defineStore('session', () => { } async function request(path: string, method = 'GET', body?: unknown, auth = token, version = generation): Promise { + // A multipart body carries its own content type with the boundary, so it is sent as is. + const multipart = body instanceof FormData const response = await fetch(`/api/v1/${path}`, { method, - headers: { ...(auth ? { Authorization: `Bearer ${auth}` } : {}), ...(body ? { 'Content-Type': 'application/json' } : {}) }, - body: body ? JSON.stringify(body) : undefined, + headers: { ...(auth ? { Authorization: `Bearer ${auth}` } : {}), ...(body && !multipart ? { 'Content-Type': 'application/json' } : {}) }, + body: body ? (multipart ? body : JSON.stringify(body)) : undefined, cache: 'no-store', }) if (response.status === 401 && version === generation) { diff --git a/learner/src/views/ImportView.vue b/learner/src/views/ImportView.vue index cc34c2d..a362c6f 100644 --- a/learner/src/views/ImportView.vue +++ b/learner/src/views/ImportView.vue @@ -2,7 +2,7 @@ import { computed, onMounted, onUnmounted, ref, watch } from 'vue' import { RouterLink, useRoute, useRouter } from 'vue-router' import { ElButton, ElInput, ElOption, ElRadio, ElRadioGroup, ElSelect } from 'element-plus' -import { LANGUAGE_LABEL, TEXT_MAX_CODE_POINTS, textProblem, titleProblem, useLibraryStore, type SubmitTarget } from '../stores/library' +import { LANGUAGE_LABEL, TEXT_MAX_CODE_POINTS, fileProblem, fileSizeLabel, textProblem, titleProblem, useLibraryStore, type SubmitTarget } from '../stores/library' import { useSessionStore } from '../stores/session' const session = useSessionStore() @@ -14,13 +14,50 @@ const title = ref('') const text = ref('') const mode = ref<'new' | 'append'>('new') const bookId = ref(undefined) +// The accepted prototype offers both sources on one screen; the server treats them the same. +const source = ref<'paste' | 'txt'>('paste') +const file = ref(null) +const fileInfo = ref<{ name: string; encoding: string; size: string } | null>(null) const titleError = ref('') const textError = ref('') +const fileError = ref('') const bookError = ref('') const length = computed(() => [...text.value].length) const busy = computed(() => library.submitting) +/** The browser pre-check only replaces the server, it never replaces its verdict. */ +async function readFile(selected: File): Promise { + fileError.value = '' + fileInfo.value = null + const problem = fileProblem(selected) + if (problem) { + file.value = null + fileError.value = problem + return + } + try { + const decoder = new TextDecoder('utf-8', { fatal: true }) + decoder.decode(await selected.arrayBuffer()) + } catch { + file.value = null + fileError.value = '文件不是 UTF-8 编码，请另存为 UTF-8 后重试。' + return + } + file.value = selected + fileInfo.value = { name: selected.name, encoding: 'UTF-8', size: fileSizeLabel(selected.size) } + // A file usually defines the title; the learner can still change it before submitting. + if (!title.value.trim()) title.value = selected.name.replace(/\.txt$/i, '').slice(0, 120) +} + +function onFile(event: Event): void { + const selected = (event.target as HTMLInputElement).files?.[0] + file.value = null + fileInfo.value = null + fileError.value = '' + if (selected) void readFile(selected) +} + function requestedBookId(): number | undefined { const raw = Array.isArray(route.query.book) ? route.query.book[0] : route.query.book if (typeof raw !== 'string' || !/^\d+$/.test(raw)) return undefined @@ -43,23 +80,35 @@ watch(mode, value => { watch(title, () => { titleError.value = '' }) watch(text, () => { textError.value = '' }) +// Switching source clears the other source's complaints and its result. +watch(source, () => { + titleError.value = '' + textError.value = '' + fileError.value = '' + library.submitError = '' +}) async function submit() { if (busy.value) return titleError.value = titleProblem(title.value) - textError.value = textProblem(text.value) + textError.value = source.value === 'paste' ? textProblem(text.value) : '' + fileError.value = source.value === 'txt' && !file.value ? (fileError.value || '请选择要导入的 TXT 文件。') : fileError.value bookError.value = mode.value === 'append' && bookId.value === undefined ? '请选择要追加的书籍。' : '' - if (titleError.value || textError.value || bookError.value) return + if (titleError.value || textError.value || fileError.value || bookError.value) return const target: SubmitTarget = mode.value === 'append' && bookId.value !== undefined ? { mode: 'append', bookId: bookId.value } : { mode: 'new' } try { - const createdBookId = await library.submit({ title: title.value, text: text.value, target }) + const createdBookId = source.value === 'txt' && file.value + ? await library.upload({ title: title.value, target, file: file.value }) + : await library.submit({ title: title.value, text: text.value, target }) // A response that arrives after the user left this page must not navigate them back. if (disposed) return // The requestId was consumed by this submission, so the form starts clean. title.value = '' text.value = '' + file.value = null + fileInfo.value = null await router.replace(`/books/${createdBookId}`) } catch { // library.submitError already carries the server message for the template. @@ -92,6 +141,13 @@ onUnmounted(() => {
+
+ 导入方式 + + 粘贴文本 + TXT 文件 + +
语言

{{ LANGUAGE_LABEL }}

@@ -116,15 +172,22 @@ onUnmounted(() => {
-
+

{{ length }} / {{ TEXT_MAX_CODE_POINTS }} 字符

+
+ + +

{{ fileInfo.name }} · {{ fileInfo.encoding }} · {{ fileInfo.size }}

+

仅支持 UTF-8 的 .txt 文件，最多 2 MiB；文件不会保存在服务器上。

+ +
- 开始处理 + {{ source === 'txt' ? '上传并处理' : '开始处理' }} 返回书库
diff --git a/server/app/lexgo/review.go b/server/app/lexgo/review.go index 3253749..cb8faf6 100644 --- a/server/app/lexgo/review.go +++ b/server/app/lexgo/review.go @@ -5,8 +5,8 @@ import ( "strconv" "time" - driver "github.com/go-sql-driver/mysql" "github.com/gin-gonic/gin" + driver "github.com/go-sql-driver/mysql" admin "go-admin/app/admin/models" "gorm.io/gorm" "gorm.io/gorm/clause" @@ -337,6 +337,7 @@ func AnswerReview(tx *gorm.DB, owner int, termID int64, input ReviewAnswerInput, DueAtBefore: before.DueAt, DueAtAfter: next.DueAt, Item: reviewItem(term, review), }, nil } + // staleAnswer records that this attempt changed nothing because the term had already // moved on. It is a normal outcome of two open tabs, not an error. func staleAnswer(tx *gorm.DB, owner int, term Term, review TermReview, input ReviewAnswerInput, now time.Time) (ReviewAnswerResult, error) { diff --git a/server/app/lexgo/review_test.go b/server/app/lexgo/review_test.go index 2bf1511..13d2b05 100644 --- a/server/app/lexgo/review_test.go +++ b/server/app/lexgo/review_test.go @@ -1,9 +1,9 @@ package lexgo import ( - "sync" "encoding/json" "fmt" + "sync" "testing" "time" diff --git a/server/app/lexgo/router.go b/server/app/lexgo/router.go index 636fc64..a568399 100644 --- a/server/app/lexgo/router.go +++ b/server/app/lexgo/router.go @@ -130,7 +130,9 @@ func Router(db *gorm.DB, now func() time.Time) *gin.Engine { status = 201 } // A repeated paste is answered from the first result, so it is not a new resource. - if c.Request.Method == "POST" && (c.FullPath() == "/api/v1/books" || c.FullPath() == "/api/v1/books/:id/chapters") { + // A pasted or uploaded chapter is one resource; a repeated submit is not. + if c.Request.Method == "POST" && (c.FullPath() == "/api/v1/books" || c.FullPath() == "/api/v1/books/:id/chapters" || + c.FullPath() == "/api/v1/books/upload" || c.FullPath() == "/api/v1/books/:id/chapters/upload") { if paste, ok := data.(PasteResult); ok && !paste.Duplicate { status = 201 } @@ -231,13 +233,6 @@ func Router(db *gorm.DB, now func() time.Time) *gin.Engine { } return updateAccount(tx, id, u.UserId, input) })) - pathID := func(c *gin.Context, message string) (int64, error) { - id, err := strconv.ParseInt(c.Param("id"), 10, 64) - if err != nil || id <= 0 { - return 0, failure(404, message) - } - return id, nil - } v.POST("/books", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) { var input PasteBookInput if err := decodeLimit(c, &input, maxPasteBodyBytes); err != nil { @@ -309,10 +304,21 @@ func Router(db *gorm.DB, now func() time.Time) *gin.Engine { registerDictionaryRoutes(v, protect, now) registerTermRoutes(v, protect, now) registerReviewRoutes(v, protect, now) + registerUploadRoutes(v, protect, now) r.NoRoute(func(c *gin.Context) { respond(c, 404, nil, failure(404, "页面或接口不存在")) }) return r } +// pathID reads a positive numeric path parameter; a malformed id is reported like a +// missing resource so it cannot be used to probe for other accounts' rows. +func pathID(c *gin.Context, message string) (int64, error) { + id, err := strconv.ParseInt(c.Param("id"), 10, 64) + if err != nil || id <= 0 { + return 0, failure(404, message) + } + return id, nil +} + func decode(c *gin.Context, value any) error { return decodeLimit(c, value, maxJSONBodyBytes) } func decodeLimit(c *gin.Context, value any, limit int64) error { diff --git a/server/app/lexgo/upload.go b/server/app/lexgo/upload.go new file mode 100644 index 0000000..1d16cf9 --- /dev/null +++ b/server/app/lexgo/upload.go @@ -0,0 +1,183 @@ +package lexgo + +import ( + "bytes" + "errors" + "io" + "net/http" + "time" + "unicode/utf8" + + "github.com/gin-gonic/gin" + admin "go-admin/app/admin/models" + "gorm.io/gorm" +) + +// A TXT upload is decoded in memory and handed to the same paste pipeline. Nothing is +// written to disk: there is no temporary file to leak, and the client file name never +// becomes a path, so it cannot reach outside the request. +const maxTextUploadBytes = 2 << 20 + +var ( + utf8BOM = []byte{0xEF, 0xBB, 0xBF} + utf16BOMBig = []byte{0xFE, 0xFF} + utf16BOMSmall = []byte{0xFF, 0xFE} +) + +// decodeTextUpload turns an uploaded TXT into the exact text the reader will show. Only +// UTF-8 is accepted: an optional BOM is removed before the text is validated, and any byte +// that is not valid UTF-8 rejects the file instead of being replaced, so a chapter never +// contains a substitute character the learner did not write. +func decodeTextUpload(raw []byte) (string, error) { + if len(raw) > maxTextUploadBytes { + return "", failure(400, "TXT 文件不能超过 2 MiB") + } + if bytes.HasPrefix(raw, utf16BOMBig) || bytes.HasPrefix(raw, utf16BOMSmall) { + return "", failure(400, "文件是 UTF-16 编码，请另存为 UTF-8 后重试") + } + raw = bytes.TrimPrefix(raw, utf8BOM) + if !utf8.Valid(raw) { + return "", failure(400, "文件不是 UTF-8 编码，请另存为 UTF-8 后重试") + } + if bytes.IndexByte(raw, 0) >= 0 { + return "", failure(400, "文件包含无法处理的字符，请另存为纯文本后重试") + } + return string(raw), nil +} + +type textUpload struct { + RequestID string + Title string + Language string + Text string +} + +// readTextUpload parses the multipart submission: one file part plus a whitelist of text +// fields. The uploaded name is never read, not even for validation, because it only serves +// display in the browser. +func readTextUpload(c *gin.Context, allowLanguage bool) (textUpload, error) { + bad := failure(400, "TXT 上传无效，请选择 UTF-8 的 .txt 文件并填写标题") + c.Request.Body = http.MaxBytesReader(c.Writer, c.Request.Body, maxTextUploadBytes+(64<<10)) + reader, err := c.Request.MultipartReader() + if err != nil { + return textUpload{}, bad + } + fields := map[string]string{} + var content []byte + for { + part, err := reader.NextPart() + if errors.Is(err, io.EOF) { + break + } + if err != nil { + return textUpload{}, uploadBodyError(err, bad) + } + name := part.FormName() + if name == "file" { + if content != nil { + part.Close() + return textUpload{}, bad + } + content, err = io.ReadAll(io.LimitReader(part, maxTextUploadBytes+1)) + part.Close() + if err != nil { + return textUpload{}, uploadBodyError(err, bad) + } + if len(content) == 0 || len(content) > maxTextUploadBytes { + return textUpload{}, failure(400, "TXT 文件不能为空且不能超过 2 MiB") + } + continue + } + if !textUploadField(name, allowLanguage) { + part.Close() + return textUpload{}, bad + } + if _, exists := fields[name]; exists { + part.Close() + return textUpload{}, bad + } + value, readErr := io.ReadAll(io.LimitReader(part, 1025)) + part.Close() + if readErr != nil { + return textUpload{}, uploadBodyError(readErr, bad) + } + if len(value) > 1024 || !utf8.Valid(value) { + return textUpload{}, bad + } + fields[name] = string(value) + } + if content == nil { + return textUpload{}, bad + } + text, err := decodeTextUpload(content) + if err != nil { + return textUpload{}, err + } + return textUpload{RequestID: fields["requestId"], Title: fields["title"], Language: fields["language"], Text: text}, nil +} + +func textUploadField(name string, allowLanguage bool) bool { + switch name { + case "requestId", "title": + return true + case "language": + return allowLanguage + default: + return false + } +} + +// An oversized body is reported as a size problem, not as an invalid upload. +func uploadBodyError(err error, bad error) error { + var tooLarge *http.MaxBytesError + if errors.As(err, &tooLarge) { + return failure(400, "TXT 文件不能超过 2 MiB") + } + return bad +} + +// registerUploadRoutes reuses the paste pipeline: the decoded file becomes the same +// PasteBook/PasteChapter input, so idempotency, ownership and the ingest job behave exactly +// as they do for pasted text. One upload at a time keeps concurrent large decodes bounded. +func registerUploadRoutes(v *gin.RouterGroup, protect func(bool, func(*gin.Context, *gorm.DB, admin.SysUser) (any, error)) gin.HandlerFunc, now func() time.Time) { + gate := make(chan struct{}, 1) + enter := func() error { + select { + case gate <- struct{}{}: + return nil + default: + return failure(429, "已有文件正在上传，请稍后重试") + } + } + leave := func() { <-gate } + v.POST("/books/upload", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) { + if err := enter(); err != nil { + return nil, err + } + defer leave() + upload, err := readTextUpload(c, true) + if err != nil { + return nil, err + } + return PasteBook(tx, u.UserId, now(), PasteBookInput{ + RequestID: upload.RequestID, Title: upload.Title, Text: upload.Text, Language: upload.Language, + }) + })) + v.POST("/books/:id/chapters/upload", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) { + bookID, err := pathID(c, "书籍不存在") + if err != nil { + return nil, err + } + if err := enter(); err != nil { + return nil, err + } + defer leave() + upload, err := readTextUpload(c, false) + if err != nil { + return nil, err + } + return PasteChapter(tx, u.UserId, bookID, now(), PasteChapterInput{ + RequestID: upload.RequestID, Title: upload.Title, Text: upload.Text, + }) + })) +} diff --git a/server/app/lexgo/upload_test.go b/server/app/lexgo/upload_test.go new file mode 100644 index 0000000..429d123 --- /dev/null +++ b/server/app/lexgo/upload_test.go @@ -0,0 +1,376 @@ +package lexgo + +import ( + "bytes" + "encoding/json" + "fmt" + "mime/multipart" + "net/http/httptest" + "strings" + "testing" + + "github.com/gin-gonic/gin" +) + +// uploadFixture is the same character set the paste contract preserves, written as the bytes +// a file would hold: CRLF and LF, a tab, curly quotes, an em dash, an ellipsis, an emoji, a +// combining acute accent, a trailing space run and an empty final line. +const uploadFixture = "Mira opened the workshop.\r\n\r\n\tThe sign read “A small step…” — café e\u0301 🙂\r\nTrailing spaces here: \n\n" + +// uploadFile posts one multipart TXT submission. A nil file means "no file part", and an +// empty name means "no file name", so the rejection paths stay testable. +func uploadFile(t *testing.T, r *gin.Engine, token, path, fileName string, content []byte, fields map[string]string) (int, string, pasteResponse) { + t.Helper() + var body bytes.Buffer + writer := multipart.NewWriter(&body) + names := make([]string, 0, len(fields)) + for name := range fields { + names = append(names, name) + } + // Field order must be stable so a failure message is reproducible. + for _, name := range []string{"requestId", "title", "language"} { + if value, ok := fields[name]; ok { + if err := writer.WriteField(name, value); err != nil { + t.Fatal(err) + } + names = removeString(names, name) + } + } + for _, name := range names { + if err := writer.WriteField(name, fields[name]); err != nil { + t.Fatal(err) + } + } + if content != nil { + part, err := writer.CreateFormFile("file", fileName) + if err != nil { + t.Fatal(err) + } + if _, err = part.Write(content); err != nil { + t.Fatal(err) + } + } + writer.Close() + request := httptest.NewRequest("POST", path, &body) + request.Header.Set("Content-Type", writer.FormDataContentType()) + if token != "" { + request.Header.Set("Authorization", "Bearer "+token) + } + response := httptest.NewRecorder() + r.ServeHTTP(response, request) + var envelope struct { + Msg string `json:"msg"` + Data json.RawMessage `json:"data"` + } + if err := json.Unmarshal(response.Body.Bytes(), &envelope); err != nil { + t.Fatalf("upload response for %s: %v", path, err) + } + var out pasteResponse + if len(envelope.Data) > 0 { + json.Unmarshal(envelope.Data, &out) + } + return response.Code, envelope.Msg, out +} + +func removeString(values []string, target string) []string { + result := values[:0] + for _, value := range values { + if value != target { + result = append(result, value) + } + } + return result +} + +func TestDecodeTextUploadRules(t *testing.T) { + // Valid UTF-8 is returned exactly as received, including unusual but legal characters. + if text, err := decodeTextUpload([]byte(uploadFixture)); err != nil || text != uploadFixture { + t.Fatalf("valid file: %q %v", text, err) + } + // A UTF-8 BOM is removed and never becomes part of the original text. + withBOM := append(append([]byte{}, utf8BOM...), []byte("BOM before text\n")...) + if text, err := decodeTextUpload(withBOM); err != nil || text != "BOM before text\n" { + t.Fatalf("BOM file: %q %v", text, err) + } + // Only a BOM leaves an empty text, which the paste rules then reject. + if text, err := decodeTextUpload(append([]byte{}, utf8BOM...)); err != nil || text != "" { + t.Fatalf("BOM only: %q %v", text, err) + } + rejected := []struct { + name string + content []byte + message string + }{ + {"invalid UTF-8", []byte{0x41, 0x80, 0x42}, "UTF-8"}, + {"latin-1 text", []byte("caf\xe9 plain\n"), "UTF-8"}, + {"UTF-16 little endian", []byte{0xFF, 0xFE, 0x41, 0x00}, "UTF-16"}, + {"UTF-16 big endian", []byte{0xFE, 0xFF, 0x00, 0x41}, "UTF-16"}, + {"NUL byte", []byte("text\x00more"), "无法处理"}, + } + for _, tc := range rejected { + text, err := decodeTextUpload(tc.content) + if err == nil || text != "" { + t.Fatalf("%s was accepted as %q", tc.name, text) + } + api, ok := err.(*apiError) + if !ok || api.status != 400 || !strings.Contains(api.message, tc.message) { + t.Fatalf("%s message: %v", tc.name, err) + } + } + // The size boundary is exact on both sides, and the byte limit is checked before decoding. + if _, err := decodeTextUpload(bytes.Repeat([]byte("a"), maxTextUploadBytes)); err != nil { + t.Fatalf("a file at the size limit must be accepted: %v", err) + } + if _, err := decodeTextUpload(bytes.Repeat([]byte("a"), maxTextUploadBytes+1)); err == nil { + t.Fatal("a file over the size limit must be rejected") + } + // The paste rules still bound one chapter, so the byte limit cannot smuggle in more text. + if _, _, _, err := validatePaste("title", strings.Repeat("a", maxChapterRunes)); err != nil { + t.Fatalf("the exact chapter limit must be accepted: %v", err) + } + if _, _, _, err := validatePaste("title", strings.Repeat("a", maxChapterRunes+1)); err == nil { + t.Fatal("the chapter code point limit must still apply to uploaded text") + } +} + +// TestMySQLTextUploadImportPath covers the accepted file: it creates the same book, chapter +// and job as a paste, and the reader text equals the file byte for byte. +func TestMySQLTextUploadImportPath(t *testing.T) { + db, r, owner := libraryFixture(t) + learner := newLearner(t, r, owner.Token) + drainIngest(t, db) + + fields := map[string]string{"requestId": "upload-fixture-0001", "title": "The Workshop Upload", "language": "en"} + code, msg, uploaded := uploadFile(t, r, learner.Token, "/api/v1/books/upload", "workshop.txt", []byte(uploadFixture), fields) + if code != 201 { + t.Fatalf("upload status %d (%s)", code, msg) + } + if uploaded.Book == nil || uploaded.Book.Title != "The Workshop Upload" || uploaded.Book.Language != "en" { + t.Fatalf("unexpected book %+v", uploaded.Book) + } + if uploaded.Chapter.Ordinal != 1 || uploaded.Chapter.Status != statusPending || uploaded.Duplicate { + t.Fatalf("unexpected chapter %+v", uploaded.Chapter) + } + + // The worker publishes the chapter, and the reader shows exactly the file content. + drainIngest(t, db) + code, ready := readChapter(t, r, learner.Token, uploaded.Chapter.ID) + if code != 200 || ready.Chapter.Status != statusReady || ready.Chapter.OriginalText != uploadFixture { + t.Fatalf("reader text: status %d, %+v", code, ready.Chapter) + } + if want := len([]rune(uploadFixture)); ready.Chapter.CharCount != want { + t.Fatalf("charCount %d, want %d", ready.Chapter.CharCount, want) + } + + // A UTF-8 BOM is stripped, so the reader never shows it. + bomFields := map[string]string{"requestId": "upload-bom-000002", "title": "BOM Upload", "language": "en"} + withBOM := append(append([]byte{}, utf8BOM...), []byte("Plain text with BOM.\n")...) + code, msg, bom := uploadFile(t, r, learner.Token, "/api/v1/books/upload", "bom.txt", withBOM, bomFields) + if code != 201 { + t.Fatalf("BOM upload status %d (%s)", code, msg) + } + drainIngest(t, db) + if code, read := readChapter(t, r, learner.Token, bom.Chapter.ID); code != 200 || read.Chapter.OriginalText != "Plain text with BOM.\n" { + t.Fatalf("BOM text: status %d, %q", code, read.Chapter.OriginalText) + } +} + +// TestMySQLTextUploadIdempotencyAndAppend reuses the paste job rules: one file yields one +// chapter, a repeated upload answers with that chapter, and the same request id with other +// content is a conflict. +func TestMySQLTextUploadIdempotencyAndAppend(t *testing.T) { + db, r, owner := libraryFixture(t) + learner := newLearner(t, r, owner.Token) + other := newLearner(t, r, owner.Token) + drainIngest(t, db) + + fields := map[string]string{"requestId": "upload-repeat-0001", "title": "Repeat Upload", "language": "en"} + code, msg, first := uploadFile(t, r, learner.Token, "/api/v1/books/upload", "repeat.txt", []byte("First upload body.\n"), fields) + if code != 201 { + t.Fatalf("first upload %d (%s)", code, msg) + } + code, msg, again := uploadFile(t, r, learner.Token, "/api/v1/books/upload", "repeat.txt", []byte("First upload body.\n"), fields) + if code != 200 || !again.Duplicate || again.Chapter.ID != first.Chapter.ID { + t.Fatalf("repeat upload %d (%s): %+v", code, msg, again) + } + var chapters int64 + db.Model(&Chapter{}).Where("book_id = ?", first.Book.ID).Count(&chapters) + if chapters != 1 { + t.Fatalf("a repeated upload created %d chapters", chapters) + } + // The same request id with other content is a conflict, not a second chapter. + code, msg, _ = uploadFile(t, r, learner.Token, "/api/v1/books/upload", "repeat.txt", []byte("Different body.\n"), fields) + if code != 409 { + t.Fatalf("changed content status %d (%s)", code, msg) + } + + // Appending uses the same rules and the same job pipeline. + appendFields := map[string]string{"requestId": "upload-append-0002", "title": "Second Chapter"} + path := fmt.Sprintf("/api/v1/books/%d/chapters/upload", first.Book.ID) + code, msg, appended := uploadFile(t, r, learner.Token, path, "second.txt", []byte("A single plain paragraph.\n"), appendFields) + if code != 201 { + t.Fatalf("append upload %d (%s)", code, msg) + } + if appended.Chapter.Ordinal != 2 || appended.Chapter.BookID != first.Book.ID { + t.Fatalf("unexpected appended chapter %+v", appended.Chapter) + } + code, msg, againAppend := uploadFile(t, r, learner.Token, path, "second.txt", []byte("A single plain paragraph.\n"), appendFields) + if code != 200 || !againAppend.Duplicate || againAppend.Chapter.ID != appended.Chapter.ID { + t.Fatalf("repeated append %d (%s): %+v", code, msg, againAppend) + } + // The append endpoint does not accept a language field: the book owns the language. + code, msg, _ = uploadFile(t, r, learner.Token, path, "second.txt", []byte("Another body.\n"), + map[string]string{"requestId": "upload-append-lang-0004", "title": "Second Chapter", "language": "en"}) + if code != 400 { + t.Fatalf("append with a language field: status %d (%s)", code, msg) + } + + // Another account cannot append into this book, and never learns whether it exists. + code, foreignMsg, _ := uploadFile(t, r, other.Token, path, "second.txt", []byte("Foreign body.\n"), map[string]string{"requestId": "upload-foreign-0003", "title": "Foreign"}) + if code != 404 { + t.Fatalf("foreign append status %d (%s)", code, foreignMsg) + } + // The other account's own library stays empty. + _, list := bookList(t, r, other.Token) + if len(list.Items) != 0 { + t.Fatalf("the other account must not see this book: %+v", list.Items) + } +} + +// TestMySQLTextUploadRejectsInvalidSubmissions covers the validation surface, including the +// client file name, which is never used as a path. +func TestMySQLTextUploadRejectsInvalidSubmissions(t *testing.T) { + db, r, owner := libraryFixture(t) + learner := newLearner(t, r, owner.Token) + drainIngest(t, db) + + base := map[string]string{"requestId": "upload-invalid-0001", "title": "Invalid Upload", "language": "en"} + cases := []struct { + name string + fileName string + content []byte + fields map[string]string + status int + }{ + {"no file part", "", nil, base, 400}, + {"empty file", "empty.txt", []byte{}, base, 400}, + {"empty file part", "empty.txt", []byte{}, base, 400}, + {"whitespace only", "blank.txt", []byte(" \n\t\n"), base, 400}, + {"BOM only", "bom.txt", append([]byte{}, utf8BOM...), base, 400}, + {"invalid UTF-8", "latin1.txt", []byte("caf\xe9\n"), base, 400}, + {"UTF-16 file", "unicode.txt", []byte{0xFF, 0xFE, 0x41, 0x00}, base, 400}, + {"oversized file", "big.txt", bytes.Repeat([]byte("a"), maxTextUploadBytes+1), base, 400}, + {"missing request id", "text.txt", []byte("Body.\n"), map[string]string{"title": "Invalid Upload", "language": "en"}, 400}, + {"missing title", "text.txt", []byte("Body.\n"), map[string]string{"requestId": "upload-invalid-0002", "language": "en"}, 400}, + {"unsupported language", "text.txt", []byte("Body.\n"), map[string]string{"requestId": "upload-invalid-0004", "title": "Invalid Upload", "language": "fr"}, 400}, + {"short request id", "text.txt", []byte("Body.\n"), map[string]string{"requestId": "short", "title": "Invalid Upload", "language": "en"}, 400}, + {"unknown field", "text.txt", []byte("Body.\n"), map[string]string{"requestId": "upload-invalid-0005", "title": "Invalid Upload", "language": "en", "ownerId": "9"}, 400}, + } + for _, tc := range cases { + code, msg, _ := uploadFile(t, r, learner.Token, "/api/v1/books/upload", tc.fileName, tc.content, tc.fields) + if code != tc.status { + t.Fatalf("%s: status %d (%s), want %d", tc.name, code, msg, tc.status) + } + if msg == "" { + t.Fatalf("%s: rejection without a readable message", tc.name) + } + } + // No rejected submission left a book behind. + _, list := bookList(t, r, learner.Token) + if len(list.Items) != 0 { + t.Fatalf("rejected uploads created books: %+v", list.Items) + } + + // A missing language field follows the paste rule and defaults to English. + code, msg, defaulted := uploadFile(t, r, learner.Token, "/api/v1/books/upload", "default.txt", []byte("Body without language.\n"), + map[string]string{"requestId": "upload-default-lang-0007", "title": "Default Language"}) + if code != 201 || defaulted.Book == nil || defaulted.Book.Language != "en" { + t.Fatalf("missing language must default to English: status %d (%s) book %+v", code, msg, defaulted.Book) + } + + // The uploaded name is only a display string: a traversal-shaped name changes nothing. + hostile := map[string]string{"requestId": "upload-hostile-0006", "title": "Hostile Name", "language": "en"} + code, msg, uploaded := uploadFile(t, r, learner.Token, "/api/v1/books/upload", `..\..\windows\system32\evil.txt`, []byte("Hostile but harmless.\n"), hostile) + if code != 201 { + t.Fatalf("hostile name status %d (%s)", code, msg) + } + var chapter Chapter + if err := db.First(&chapter, uploaded.Chapter.ID).Error; err != nil { + t.Fatal(err) + } + var book Book + if err := db.First(&book, uploaded.Book.ID).Error; err != nil { + t.Fatal(err) + } + for _, field := range []string{chapter.Title, chapter.OriginalText, book.Title} { + if strings.Contains(field, "evil") || strings.Contains(field, "system32") || strings.Contains(field, `..`) { + t.Fatalf("the uploaded name leaked into stored data: %q", field) + } + } + + // An unrelated content type is not a multipart upload. + request := httptest.NewRequest("POST", "/api/v1/books/upload", strings.NewReader(`{"requestId":"json-upload-0001"}`)) + request.Header.Set("Content-Type", "application/json") + request.Header.Set("Authorization", "Bearer "+learner.Token) + response := httptest.NewRecorder() + r.ServeHTTP(response, request) + if response.Code != 400 { + t.Fatalf("JSON body accepted as an upload: %d", response.Code) + } + // Uploading without a session is rejected before any parsing. + code, _, _ = uploadFile(t, r, "", "/api/v1/books/upload", "text.txt", []byte("Body.\n"), base) + if code != 401 { + t.Fatalf("anonymous upload status %d", code) + } +} + +// TestMySQLTextUploadKeepsChapterLimit proves the byte cap cannot bypass the one-chapter +// code point rule, and that a large but legal file is stored completely. +func TestMySQLTextUploadKeepsChapterLimit(t *testing.T) { + db, r, owner := libraryFixture(t) + learner := newLearner(t, r, owner.Token) + drainIngest(t, db) + + // Exactly at the chapter limit: accepted, and the stored text keeps its full length. + atLimit := strings.Repeat("a", maxChapterRunes-1) + "\n" + fields := map[string]string{"requestId": "upload-limit-0001", "title": "At The Limit", "language": "en"} + code, msg, uploaded := uploadFile(t, r, learner.Token, "/api/v1/books/upload", "limit.txt", []byte(atLimit), fields) + if code != 201 { + t.Fatalf("upload at the chapter limit %d (%s)", code, msg) + } + drainIngest(t, db) + if code, read := readChapter(t, r, learner.Token, uploaded.Chapter.ID); code != 200 || read.Chapter.OriginalText != atLimit { + t.Fatalf("chapter at the limit: status %d, length %d", code, len([]rune(read.Chapter.OriginalText))) + } + // One code point more is rejected by the same rule that already applies to a paste. + overLimit := strings.Repeat("a", maxChapterRunes+1) + code, msg, _ = uploadFile(t, r, learner.Token, "/api/v1/books/upload", "over.txt", []byte(overLimit), map[string]string{"requestId": "upload-limit-0002", "title": "Over The Limit", "language": "en"}) + if code != 400 || !strings.Contains(msg, "100000") { + t.Fatalf("upload over the chapter limit: status %d (%s)", code, msg) + } +} + +// TestMySQLTextUploadAndPasteShareOnePipeline checks the two entry points cannot produce a +// second chapter for the same submitted content when the client stays on one request id. +func TestMySQLTextUploadAndPasteShareOnePipeline(t *testing.T) { + db, r, owner := libraryFixture(t) + learner := newLearner(t, r, owner.Token) + drainIngest(t, db) + + fields := map[string]string{"requestId": "upload-shared-0001", "title": "Shared Pipeline", "language": "en"} + code, msg, uploaded := uploadFile(t, r, learner.Token, "/api/v1/books/upload", "shared.txt", []byte("Shared body.\n"), fields) + if code != 201 { + t.Fatalf("upload %d (%s)", code, msg) + } + // The same request id through the paste endpoint answers with the uploaded chapter. + code, pasted := pasteBook(t, r, learner.Token, map[string]string{ + "requestId": "upload-shared-0001", "title": "Shared Pipeline", "text": "Shared body.\n", "language": "en"}) + if code != 200 || !pasted.Duplicate || pasted.Chapter.ID != uploaded.Chapter.ID { + t.Fatalf("paste after upload %d: %+v", code, pasted) + } + var chapters int64 + db.Model(&Chapter{}).Where("book_id = ?", uploaded.Book.ID).Count(&chapters) + if chapters != 1 { + t.Fatalf("the two entry points created %d chapters", chapters) + } +} -- 2.34.1 From fb256d49168c13e0b1cecccbf1dad139a331e8e3 Mon Sep 17 00:00:00 2001 From: QiuSW Date: Fri, 11 Sep 2026 23:38:38 +0800 Subject: [PATCH 2/2] =?UTF-8?q?docs:=20=E8=AE=B0=E5=BD=95=20#9=20=E5=B7=B2?= =?UTF-8?q?=E7=A1=AE=E8=AE=A4=E7=9A=84=20TXT=20=E5=AF=BC=E5=85=A5=E5=8F=A3?= =?UTF-8?q?=E5=BE=84=20(#9)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- AGENTS.md | 1 + 1 file changed, 1 insertion(+) diff --git a/AGENTS.md b/AGENTS.md index 157528b..07ea087 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -282,4 +282,5 @@ MVP 内所有单元任务通过后才能做 MVP 集成验收；MVP 通过后才 - #4 独立阅读选择小样位于 `spikes/selection/`，`python spikes/selection/serve.py` 默认仅本机 5184。桌面鼠标/键盘与 11 项测试已验证，#4 已获用户验收并关闭；真实手机长按/手柄/滚动详细证据仍未提供；禁止把窄屏桌面当作真机验收。Intl.Segmenter 只用于 UI 范围验证，不替代 #3 NLP；释义保存只在内存。固定 LinguaCafe 源码对照和与 v1 的差异记录见架构 Wiki。 - 2026-09-11 用户确认正式 NLP/词典采用全 Go。#5 已验收并合入 main；#6 使用 Go WordNet 解析和词形候选、Go Unicode 原文分片、schema v4 共享词典资源表，不调用 Python NLP。WordNet 3.0 ZIP 来源与摘要见 `server/wordnet-resource.json`，许可保留在 `server/WORDNET-LICENSE.txt`。词形候选不等于上下文消歧，不自动合并个人学习状态；#3 Python 小样只保留历史验证。当前词典仅英语释义，个人释义输入为临时草稿，持久化归 #7。 - 2026-09-11 用户确认 #7 个人词条口径（三项由 Agent 定案）：身份为「学习者＋语言＋规范化词形」，大小写合并但**不按 lemma/候选合并**（`dog` 与 `dogs` 是两条记录）；首次保存默认「新词」；状态为 新词/学习中/已知/忽略，只有「学习中」带 1～7 级，对应原版 stage 2/1/0/-1～-7；例句只保存手输内容，不自动关联原文句子。schema v5 新增 `lexgo_terms`（唯一键加状态/等级检查约束），个人释义与共享词典分离且不进入审计日志；等级编辑 UI 归 #8/#12。 +- 2026-09-11 用户确认 #9 TXT 导入口径：只接受 UTF-8（允许可选 BOM，解码时剥离且不进入原文），非法字节整体拒绝、不使用替换字符；UTF-16 按 BOM 识别后明确拒绝，GB18030 等按非法 UTF-8 拒绝。文件字节上限 2 MiB，之后仍套用单章 100000 码点上限；换行与空白不归一化。文件只在内存中解码、不创建临时文件，客户端文件名不参与任何路径也不入库。解码后交给既有 `PasteBook`/`PasteChapter`，分章（一次提交一章）、`requestId` 幂等与任务恢复与粘贴一致；不改 schema。EPUB/PDF/字幕、UTF-16 转码、按空行自动分章与断点续传不在范围内。 - 2026-09-11 用户确认 #8 到期单词复习决策表：固定间隔表 1/2/4/7/15/30/60 天，答对升级封顶 7、答错降级最低 1、再学一次不改等级，答错与再学立即回队；已知/忽略不入队，新保存的词立即到期，显式「学习中 level N」排 now+间隔[N]；只有新建或状态/等级实际变化才移动复习时间，编辑释义或例句保留原排期，保存未提及等级时保留已获得等级。到期判定用 UTC 绝对时刻（`due_at ≤ now`），不引入本地日边界。作答按 `answerId` 去重并以 `expectedDueAt` 判定过期标签页，重复提交、网络重发与双标签页都不得重复更新次数与间隔（作答响应 `result` 只取 applied/stale，重放另用 `duplicate` 标记并返回首次结果）；`correct_count` 只计答对，`wrong_count` 计答错与再学。短语复习归 #11，进度统计归 #13，不做策略配置 UI（X11）、练习模式（X08）与 FSRS。 -- 2.34.1