feat: 上传 TXT 文件，校验编码后导入本人书库 (#9)
- POST /api/v1/books/upload 与 /api/v1/books/:id/chapters/upload：multipart 上传， 字段白名单、未知或重复字段拒绝、非 multipart 拒绝、单槽并发门忙时 429 - 只接受 UTF-8（可选 BOM 剥离且不进入原文），UTF-16 按 BOM 识别并给出针对性提示， 非法字节整体拒绝、不使用替换字符；2 MiB 字节上限之后仍套用单章 100000 码点上限 - 文件只在内存中解码，不创建临时文件；客户端文件名不参与任何路径也不入库 - 解码后交给现有 PasteBook/PasteChapter，分章、任务幂等与崩溃恢复与粘贴完全一致 - 学习端导入页新增「粘贴文本 / TXT 文件」来源切换与客户端预检，session.request 支持 FormData - gofmt 整理 #8 引入的 import 顺序与空行 - 同步 Architecture-and-Code-Map、Business-Rules-and-Glossary、 Local-Development-and-Verification、Product-Requirements-Overview 与 Home
This commit is contained in:
@@ -2,8 +2,8 @@
|
||||
generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件)
|
||||
wiki_page: Architecture-and-Code-Map
|
||||
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Architecture-and-Code-Map.-
|
||||
wiki_revision: f7b41a473c138bedf0baff1544e6480872b92d4d
|
||||
synchronized_at: 2026-09-11T14:55:11Z
|
||||
wiki_revision: ea8661cfbc172ca67148b6a14d46204ab7033e35
|
||||
synchronized_at: 2026-09-11T15:36:44Z
|
||||
<!-- gitea-wiki-mirror:end -->
|
||||
|
||||
# 架构与代码地图
|
||||
@@ -272,3 +272,18 @@ schema v6 新增 `lexgo_term_reviews`（每个个人词条一行排期：`due_at
|
||||
学习端新增 `/review` 路由与书库、阅读器顶栏的「到期复习」入口；`stores/review.ts` 维护队列、本轮计数、评分与重学，`ReviewCard.vue`/`ReviewView.vue` 呈现正面（词＋挖空例句）、答案面（个人释义＋例句＋三个评分按钮）、完成页与空队列页。一次评分对应一个 `answerId`，失败重试复用同一个；换词后重新生成。切换账号或退出登录会清空队列、计数与当前卡片。
|
||||
|
||||
参考：[LinguaCafe Review.vue](https://github.com/simjanos-dev/LinguaCafe/blob/c1ea298ce40c65b9dd33e9b26fd2e52fae66f2c8/resources/js/components/Review/Review.vue)。上游的随机抽卡、阶段降级与快捷键不属于本单；#8 只实现本项目的固定间隔、固定顺序与幂等作答。
|
||||
|
||||
## #9 TXT 上传导入（2026-09-11）
|
||||
|
||||
`server/app/lexgo/upload.go` 负责把上传的 TXT 解码后交给与粘贴相同的核心：解码、multipart 解析与两条路由，schema 无变化（沿用 #5 的 `lexgo_books`/`lexgo_chapters`/`lexgo_ingest_jobs`）。文件只在内存中存在，不写临时文件，客户端文件名不参与任何路径也不入库。
|
||||
|
||||
| 接口 | 权限与输入/输出 |
|
||||
|---|---|
|
||||
| POST /api/v1/books/upload | 本人；multipart：`requestId`、`title`、`language`（可省略，省略即英语）、`file`；新建书籍与首章 |
|
||||
| POST /api/v1/books/:id/chapters/upload | 本人且本人书籍；multipart：`requestId`、`title`、`file`；追加一章；不接受 `language` |
|
||||
|
||||
字段白名单之外的字段、重复字段、缺失 `file`、非 multipart 请求都返回 400；201 新建、200 重复、409 同编号换内容、404 他人书籍、401 未登录、429 已有文件正在上传（单槽并发门）。响应体与粘贴路径同为 `PasteResult`，所以学习端复用同一套跳转与轮询逻辑。
|
||||
|
||||
解码规则见业务规则页；实现上 `decodeTextUpload` 先按 UTF-16 BOM 识别并给出针对性提示，再剥离可选 UTF-8 BOM，然后用 `utf8.Valid` 整体校验，最后交给 `validatePaste`（非空、≤100000 码点）。因此上传与粘贴共享同一分章与任务规则：一次提交一章，`requestId` + 内容 SHA 幂等，worker 只发布已落库的原文。
|
||||
|
||||
学习端 `ImportView.vue` 增加「粘贴文本 / TXT 文件」来源切换（沿用已验收 v1 的切换与状态行），`stores/library.ts` 增加 `upload()` 与 `fileProblem`/`fileSizeLabel`，`session.request` 支持 `FormData`（multipart 请求不再被 JSON 化，边界由浏览器提供）。客户端预检只提前反馈，服务端结论为最终结论。
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件)
|
||||
wiki_page: Business-Rules-and-Glossary
|
||||
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Business-Rules-and-Glossary.-
|
||||
wiki_revision: 1d90013d883de9339865694e0bb0026f9edd3acc
|
||||
synchronized_at: 2026-09-11T14:44:50Z
|
||||
wiki_revision: 76289713c11902383031764c90ae9da90bd0ce07
|
||||
synchronized_at: 2026-09-11T15:36:44Z
|
||||
<!-- gitea-wiki-mirror:end -->
|
||||
|
||||
# 业务规则与术语
|
||||
@@ -192,3 +192,17 @@ exact优先；未命中再按WordNet异常表/词尾规则查候选，词性顺
|
||||
**归属与错误**：词条归属由服务端按会话裁决，`owner` 不接受客户端输入；他人词条与不存在的词条统一 404，未登录 401，未知评分/缺少 `expectedDueAt`/未知字段 400，词条已变成 `已知`/`忽略` 409。复习接口不写审计日志：答题属于私人学习内容。
|
||||
|
||||
**范围边界**：本单只做单词复习。短语复习归 #11，到期范围筛选与策略配置界面（X11）不做，练习模式（X08）不做，进度与「已知词/待复习」计数归 #13。#8 不改变 #7 定下的身份口径：复习状态按规范化词形归属，仍不按 WordNet lemma 候选合并。
|
||||
|
||||
## #9 TXT 上传规则（2026-09-11）
|
||||
|
||||
**支持的编码**：只接受 UTF-8，允许带可选 UTF-8 BOM。BOM 在解码时剥离，不进入原文；其余字节必须整体合法，任何非法序列**直接拒绝**，绝不使用替换字符，因此章节里不会出现学习者没有写过的乱码。UTF-16（含记事本「另存为 Unicode」产生的大小端 BOM）单独识别并提示「请另存为 UTF-8 后重试」；GB18030、Latin-1 等其他编码按非法 UTF-8 拒绝。UTF-16 支持不在本单范围。
|
||||
|
||||
**换行与空白**：不做任何归一化，CRLF、LF、制表符、行尾空格与空行按原字节保存，阅读器以 `pre-wrap` 原样呈现——与粘贴路径一致。
|
||||
|
||||
**大小上限**：文件字节上限 2 MiB；解码后再套用单章上限（非空、≤100000 码点）。超限返回 400 并明确提示，不截断、不部分导入。2 MiB 对 100000 码点的 UTF-8 文本有足够余量。
|
||||
|
||||
**文件生命周期与路径**：上传内容只存在于内存中，解码后直接进入导入事务；服务端**不创建临时文件**，所以没有需要清理或可能泄漏的文件；**客户端文件名不参与任何文件系统路径、也不写入数据库**，它只在选择文件时用于显示（并可预填标题）。因此文件名即使写成 `..\..\windows\system32\evil.txt` 也不会影响任何存储位置。上传并发按单槽限制，忙时返回 429。
|
||||
|
||||
**导入与幂等**：上传与粘贴共用同一套规则——一次提交一章，`requestId` + 内容 SHA 保证重复上传同一文件只产生一章（返回第一次的章节并标记 `duplicate`），同一 `requestId` 换成其他内容返回 409。任务状态、失败重试与崩溃恢复沿用 #5 的任务机制，不新增状态。标题规则与粘贴完全相同（去空白后非空、≤120 字符）；省略 `language` 时默认英语，与粘贴一致；追加章节不接受 `language`。
|
||||
|
||||
**范围边界**：不包含 EPUB、PDF、字幕与其他文件格式；不做按空行自动分章；不做 UTF-16/GB18030 转码；不做断点续传；不把来源文件名持久化（若将来需要「导入来源」溯源，另立范围）。
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件)
|
||||
wiki_page: Local-Development-and-Verification
|
||||
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Local-Development-and-Verification.-
|
||||
wiki_revision: 1192d23a8198e981961adfc6066f0b3bc3b84058
|
||||
synchronized_at: 2026-09-11T14:44:50Z
|
||||
wiki_revision: 8c0886a74112ea7bff734c04775b56c571797c3d
|
||||
synchronized_at: 2026-09-11T15:36:44Z
|
||||
<!-- gitea-wiki-mirror:end -->
|
||||
|
||||
# 本地开发与验证
|
||||
@@ -380,3 +380,21 @@ node --test spikes/english/view.test.mjs
|
||||
**流程记录（R6）**：评论 7769 的方案写的是在 `lexgo_terms` 上增加列，实际实现改为独立表 `lexgo_term_reviews`（加法迁移可重试、不对既有表做 ALTER）。该变更在实施评论 7776 与 Wiki 中说明了原因，但没有按「数据结构变化先更新工单」的要求在实施前追加变更评论；本页与上文契约按实际实现记录，方案评论中的「新增列均有默认值」以独立表为准。
|
||||
|
||||
整改后重跑：Go 单元与集成测试（专用库 `lexgo_test_issue8`）44 个顶层用例全部通过、0 跳过；学习端 73 项单测、类型检查、构建与 5 项 E2E 通过；管理端 31 项与 lint 通过；治理 56 项与严格检查通过；真实 API＋MySQL 42 项检查通过（新增 4 项针对 R3 与重放契约）；真实浏览器复核面板保存与复习闭环通过。截图 `.local/evidence/issue8-fixed-summary.png`。
|
||||
|
||||
## #9 验证与迁移（2026-09-11）
|
||||
|
||||
仓库根执行；Go 工具链由 `python scripts/server.py` 固定 go1.26.5。本单使用专用测试库 lexgo_test_issue9，不借用其他测试库。本单**不改动数据库结构**，所以没有迁移步骤，回退只需换回旧二进制。
|
||||
|
||||
| 命令 | 结果 |
|
||||
|---|---|
|
||||
| `go vet ./...` | 通过 |
|
||||
| `LEXGO_TEST_DB_NAME=lexgo_test_issue9 python scripts/server.py test-integration` | 50 个顶层用例全部通过、0 跳过；含 #9 新增 6 个上传用例 |
|
||||
| `cd learner`：`npx vitest --run` / `npx vue-tsc --build` / `npx pnpm run build` / `npx playwright test` | 80 项单测、类型检查、构建、6 项 E2E 全部通过 |
|
||||
| `cd admin`：`npx pnpm test` / `npx pnpm lint` | 31 项与 lint 通过；管理端本单无代码改动 |
|
||||
| `python -m unittest discover -s tests` / `python dev_scripts/harness.py check --strict` | 56 项与严格检查通过 |
|
||||
|
||||
覆盖内容：有效 UTF-8（含 CRLF、制表符、弯引号、em dash、省略号、emoji、组合字符）字节级往返、UTF-8 BOM 剥离且不进原文、只有 BOM、非法 UTF-8、Latin-1、UTF-16 大小端、NUL 字节、空文件、只有空白、超限与恰好边界（2 MiB、100000 码点）、缺 `file`、缺标题、缺或错误 `language`、缺或过短 `requestId`、未知字段、追加路径携带 `language`、非 multipart 请求、未登录、恶意文件名不影响存储、重复上传只产生一章、同编号换内容 409、追加他人书籍 404、两账号隔离、上传与粘贴共用同一任务管线。
|
||||
|
||||
真实链路验证：真实 Go API＋真实 MySQL 共 26 项检查通过（凭据只从本机安全配置读入进程），覆盖有效文件与阅读器原文逐字节一致、BOM 不进入原文、五类无效文件、越权与未登录拒绝、两账号隔离、恶意文件名不泄漏；随后用临时 Playwright 用例在真实学习端完成「登录→切换 TXT→选择真实 UTF-8 文件→上传→处理中就绪→阅读器原文逐字符一致」的闭环，并复核 UTF-16 文件在浏览器预检阶段被拒。截图保存在本机 `.local/evidence/`（issue9-invalid-encoding.png、issue9-upload-processing.png、issue9-reader.png），临时用例运行后删除。
|
||||
|
||||
未验证：真实手机触屏详细证据与完整备份恢复演练仍属既有缺口（#14/#15）；本单只用桌面浏览器检查。大文件并发上传只按单槽并发门设计，没有做多用户压力测试。UTF-16/GB18030 转码与按空行自动分章不在本单。
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件)
|
||||
wiki_page: Product-Requirements-Overview
|
||||
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Product-Requirements-Overview.-
|
||||
wiki_revision: e6a1f2037516203e9144159af77b25e87638b6fd
|
||||
synchronized_at: 2026-09-11T14:55:11Z
|
||||
wiki_revision: f5b38f2d527cbc21eac4512f7c439c013bb221d9
|
||||
synchronized_at: 2026-09-11T15:36:45Z
|
||||
<!-- gitea-wiki-mirror:end -->
|
||||
|
||||
# 产品需求总览
|
||||
@@ -251,3 +251,9 @@ F07 的个人词语记录已于 2026-09-11 通过用户验收：阅读器可以
|
||||
F10 的单词到期复习已于 2026-09-11 通过用户验收（包含独立审核整改 R1～R3）：按固定间隔表取本人当前语言的到期词条，正面显示词与挖空例句，显示答案后按「认识/答对」「不认识/答错」「再学一次」评分；答对升级并排下次复习，答错降级并立即回到本轮，再学一次不改等级并回到本轮。重复提交、网络重发与双标签页都不会重复更新次数和间隔；已知与忽略的词条不入队。管理端无改动。
|
||||
|
||||
仍未实现并留给后续工单：短语复习（#11）、词汇库搜索与编辑（#12）、阅读完成与进度统计（#13）、TXT 导入（#9）、书籍章节编辑删除（#10）。复习范围筛选与策略配置（X11）、练习模式（X08）、FSRS 仍在范围外。
|
||||
|
||||
## #9 交付范围更新（2026-09-11）
|
||||
|
||||
F03 的 TXT 文件导入已实现，待用户验收：学习端导入页新增「粘贴文本 / TXT 文件」来源切换，选择 UTF-8 的 .txt 文件后经大小、空文件与编码校验进入与粘贴相同的处理与阅读流程，失败可重试。只支持 UTF-8（允许可选 BOM）且不替换损坏字符；UTF-16 与其他编码会被明确拒绝；文件只在内存中解码、不写临时文件，客户端文件名不参与任何路径也不入库；重复上传同一文件只产生一章。schema 无变化。
|
||||
|
||||
仍未实现并留给后续工单：书籍与章节的编辑删除（#10）、短语选择与保存（#11）、词汇库搜索与编辑（#12）、阅读完成与进度（#13）、桌面与手机体验补齐（#14）、自托管试用交付与完整恢复（#15）。EPUB/PDF/字幕、UTF-16 转码、按空行自动分章与断点续传不在本单范围。
|
||||
|
||||
+4
-2
@@ -2,8 +2,8 @@
|
||||
generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件)
|
||||
wiki_page: Home
|
||||
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Home
|
||||
wiki_revision: 85c66bb518fcbb166b916549aa6e655215e0ed85
|
||||
synchronized_at: 2026-09-11T14:55:10Z
|
||||
wiki_revision: 75de70aad90f02deeadbed301c9fe85ded1fec6e
|
||||
synchronized_at: 2026-09-11T15:36:44Z
|
||||
<!-- gitea-wiki-mirror:end -->
|
||||
|
||||
# LexGo 文档入口
|
||||
@@ -74,3 +74,5 @@ Quant-UX 原型 v1 已通过用户验收。[桌面预览](https://qux.ilapage.cn
|
||||
#7 个人词条已实现并于 2026-09-11 通过用户验收（schema v5 新增 lexgo_terms）：阅读器可保存释义、例句与状态，同一词形在本人其他章节显示一致高亮，两个账号数据独立；保存幂等，跨账号与篡改身份均被拒绝；PR #26 已 fast-forward-only 合入 main。
|
||||
|
||||
#8 单词到期复习已于 2026-09-11 通过用户验收（schema v6 新增 lexgo_term_reviews 与 lexgo_review_answers）：固定间隔表（1/2/4/7/15/30/60 天）、答对升级、答错或再学立即回队、已知与忽略不入队；重复提交、网络重发与双标签页都只记账一次，到期判定用 UTC 绝对时刻而不引入本地日边界。学习端新增「到期复习」入口，卡片正面显示词与挖空例句、答案面显示个人释义，并有完成页与空队列页。独立审核指出的并发同键 500、编辑文本重排复习与面板保存重置等级三项已整改并复测；PR #27 已 fast-forward-only 合入 main。第 3 阶段「首条学习闭环」#5～#8 全部验收。
|
||||
|
||||
#9 TXT 文件导入已实现，待用户验收：导入页新增「粘贴文本 / TXT 文件」来源切换，只接受 UTF-8（允许可选 BOM）且不替换损坏字符，UTF-16 与其他编码会被明确拒绝；文件只在内存中解码、不写临时文件，客户端文件名不参与任何路径也不入库；上传与粘贴共用同一分章、任务与幂等规则，重复上传同一文件只产生一章。本次没有数据库结构变化。
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
import { expect, test } from '@playwright/test'
|
||||
|
||||
// The TXT upload path against a mocked API: pre-checks, multipart body and the hand-off to
|
||||
// the same processing screen the paste path uses.
|
||||
test('upload a UTF-8 TXT file and open the created book', async ({ page }) => {
|
||||
const user = { id: 42, username: 'fictional-uploader', role: 'learner' }
|
||||
const book = { id: 1, title: 'Studio Notes', language: 'en' }
|
||||
const timestamps = { createdAt: '2026-01-01T00:00:00Z', updatedAt: '2026-01-01T00:00:00Z' }
|
||||
const pasted = 'Mira opened the workshop.\r\n\r\n\tThe sign read “A small step…”\n'
|
||||
let uploaded = false
|
||||
|
||||
await page.route('**/api/v1/**', async route => {
|
||||
const path = new URL(route.request().url()).pathname
|
||||
const method = route.request().method()
|
||||
let data: unknown = null
|
||||
let status = 200
|
||||
if (path === '/api/v1/login') data = { token: 'fictional-session', user }
|
||||
else if (path === '/api/v1/me') data = user
|
||||
else if (path === '/api/v1/space') data = { ownerId: user.id, language: 'en' }
|
||||
else if (path === '/api/v1/books' && method === 'GET') data = { items: uploaded ? [{ ...book, chapterCount: 1, pendingCount: 0, processingCount: 0, readyCount: 1, failedCount: 0, ...timestamps }] : [] }
|
||||
else if (path === '/api/v1/books/upload') {
|
||||
// The upload must arrive as multipart: the fields and the file are inspected directly.
|
||||
expect(route.request().headers()['content-type']).toContain('multipart/form-data')
|
||||
const raw = route.request().postData() ?? ''
|
||||
expect(raw).toContain('name="requestId"')
|
||||
expect(raw).toContain('name="language"')
|
||||
expect(raw).toContain('name="title"')
|
||||
expect(raw).toContain('Studio Notes')
|
||||
expect(raw).toContain('filename="notes.txt"')
|
||||
expect(raw).toContain('Mira opened the workshop.')
|
||||
uploaded = true
|
||||
status = 201
|
||||
data = {
|
||||
book,
|
||||
chapter: { id: 9, bookId: 1, ordinal: 1, title: 'Studio Notes', status: 'pending', charCount: [...pasted].length, errorReason: '', errorMessage: '', jobId: 5, ...timestamps },
|
||||
job: { id: 5, bookId: 1, chapterId: 9, status: 'pending', attempts: 0, errorReason: '', errorMessage: '', ...timestamps },
|
||||
duplicate: false,
|
||||
}
|
||||
} else if (path === '/api/v1/books/1') data = { book, chapters: [{ id: 9, bookId: 1, ordinal: 1, title: 'Studio Notes', status: 'ready', charCount: [...pasted].length, errorReason: '', errorMessage: '', jobId: 5, ...timestamps }] }
|
||||
else if (path === '/api/v1/chapters/9') data = { book, chapter: { id: 9, bookId: 1, ordinal: 1, title: 'Studio Notes', status: 'ready', charCount: [...pasted].length, errorReason: '', errorMessage: '', jobId: 5, contentSha256: 'fictional-sha', originalText: pasted, ...timestamps }, navigation: { previousChapterId: null, nextChapterId: null } }
|
||||
await route.fulfill({ status, json: { code: 200, data } })
|
||||
})
|
||||
|
||||
await page.goto('/')
|
||||
await page.getByLabel('账号').fill(user.username)
|
||||
await page.getByLabel('密码', { exact: true }).fill('fictional-password')
|
||||
await page.getByRole('button', { name: '登录', exact: true }).click()
|
||||
await expect(page.getByRole('heading', { name: '我的书库' })).toBeVisible()
|
||||
|
||||
await page.getByRole('button', { name: '导入内容' }).click()
|
||||
// Element Plus hides the native radio behind a styled span, so the label is what a person
|
||||
// clicks; the input still carries the checked state.
|
||||
await page.locator('label.el-radio', { hasText: 'TXT 文件' }).click()
|
||||
await expect(page.getByRole('radio', { name: 'TXT 文件' })).toBeChecked()
|
||||
await expect(page.locator('textarea#text')).toHaveCount(0)
|
||||
await expect(page.getByText('仅支持 UTF-8')).toBeVisible()
|
||||
|
||||
// An unusable file is refused in the browser, before any request is made.
|
||||
await page.setInputFiles('[data-testid="file-input"]', { name: 'notes.md', mimeType: 'text/markdown', buffer: Buffer.from('# heading\n') })
|
||||
await expect(page.getByText('请选择 .txt 文件。')).toBeVisible()
|
||||
|
||||
// A UTF-8 file is accepted and its metadata is shown; the title comes from the file name.
|
||||
await page.setInputFiles('[data-testid="file-input"]', { name: 'notes.txt', mimeType: 'text/plain', buffer: Buffer.from(pasted, 'utf8') })
|
||||
await expect(page.getByTestId('file-info')).toContainText('notes.txt · UTF-8')
|
||||
await expect(page.getByLabel('标题')).toHaveValue('notes')
|
||||
await page.getByLabel('标题').fill('Studio Notes')
|
||||
|
||||
await page.getByRole('button', { name: '上传并处理' }).click()
|
||||
await expect(page).toHaveURL(/\/books\/1$/)
|
||||
await expect(page.getByText('已就绪')).toBeVisible()
|
||||
await page.getByRole('link', { name: 'Studio Notes' }).click()
|
||||
const readerText = page.locator('.reader-text')
|
||||
await expect(readerText).toBeVisible()
|
||||
// The uploaded bytes reached the reader unchanged.
|
||||
expect(await readerText.evaluate(element => element.textContent)).toBe(pasted)
|
||||
})
|
||||
@@ -0,0 +1,223 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi, type MockInstance } from 'vitest'
|
||||
import { flushPromises, mount, type VueWrapper } from '@vue/test-utils'
|
||||
import { createPinia, setActivePinia } from 'pinia'
|
||||
import { createRouter, createMemoryHistory, type Router } from 'vue-router'
|
||||
import ImportView from '../views/ImportView.vue'
|
||||
import { fileProblem, fileSizeLabel, TXT_MAX_BYTES, useLibraryStore } from '../stores/library'
|
||||
import { useSessionStore } from '../stores/session'
|
||||
|
||||
const user = { id: 42, username: 'fictional-uploader', role: 'learner' as const }
|
||||
const book = { id: 1, title: 'Uploaded Book', language: 'en' }
|
||||
const chapter = { id: 9, bookId: 1, ordinal: 1, title: 'Uploaded Book', status: 'pending', charCount: 12, errorReason: '', errorMessage: '', jobId: 5, createdAt: '', updatedAt: '' }
|
||||
const timestamps = { createdAt: '2026-01-01T00:00:00Z', updatedAt: '2026-01-01T00:00:00Z' }
|
||||
const ok = (data: unknown) => new Response(JSON.stringify({ code: 200, data }))
|
||||
let wrapper: VueWrapper | undefined
|
||||
|
||||
function stub(name: string) {
|
||||
return { template: `<div>${name}</div>` }
|
||||
}
|
||||
|
||||
async function viewAt(path: string): Promise<Router> {
|
||||
const router = createRouter({
|
||||
history: createMemoryHistory(),
|
||||
routes: [
|
||||
{ path: '/', component: stub('LibraryStub') },
|
||||
{ path: '/import', component: stub('ImportStub') },
|
||||
{ path: '/books/:id', component: stub('BookStub') },
|
||||
],
|
||||
})
|
||||
await router.push(path)
|
||||
await router.isReady()
|
||||
return router
|
||||
}
|
||||
|
||||
/** jsdom has no file picker, so the input's files are set directly before dispatching change. */
|
||||
async function chooseFile(view: VueWrapper, file: File | null): Promise<void> {
|
||||
const input = view.get('[data-testid="file-input"]')
|
||||
Object.defineProperty(input.element, 'files', { value: file ? [file] : [], configurable: true })
|
||||
await input.trigger('change')
|
||||
await flushPromises()
|
||||
}
|
||||
|
||||
/** The TXT toggle is an Element Plus radio group; its hidden input carries the value. */
|
||||
async function useTxt(view: VueWrapper): Promise<void> {
|
||||
for (const input of view.findAll('input[type="radio"]')) {
|
||||
if ((input.element as HTMLInputElement).value === 'txt') {
|
||||
await input.setValue()
|
||||
await flushPromises()
|
||||
return
|
||||
}
|
||||
}
|
||||
throw new Error('TXT 文件 toggle not found')
|
||||
}
|
||||
|
||||
function uploadResponse() {
|
||||
return ok({ book, chapter, job: { id: 5, bookId: 1, chapterId: 9, status: 'pending', attempts: 0, errorReason: '', errorMessage: '', ...timestamps }, duplicate: false })
|
||||
}
|
||||
|
||||
describe('txt upload pre-checks', () => {
|
||||
it('mirrors the server limits for the file that was picked', () => {
|
||||
expect(fileProblem({ name: 'reading.txt', size: 2048 })).toBe('')
|
||||
expect(fileProblem({ name: 'READING.TXT', size: 1 })).toBe('')
|
||||
expect(fileProblem({ name: 'reading.md', size: 2048 })).toContain('.txt')
|
||||
expect(fileProblem({ name: 'empty.txt', size: 0 })).toContain('空的')
|
||||
expect(fileProblem({ name: 'big.txt', size: TXT_MAX_BYTES + 1 })).toContain('2 MiB')
|
||||
expect(fileProblem({ name: 'limit.txt', size: TXT_MAX_BYTES })).toBe('')
|
||||
expect(fileSizeLabel(512)).toBe('512 B')
|
||||
expect(fileSizeLabel(2048)).toBe('2 KB')
|
||||
expect(fileSizeLabel(1.5 * 1024 * 1024)).toBe('1.5 MB')
|
||||
})
|
||||
})
|
||||
|
||||
describe('txt upload view', () => {
|
||||
beforeEach(() => {
|
||||
setActivePinia(createPinia())
|
||||
sessionStorage.clear()
|
||||
vi.restoreAllMocks()
|
||||
useSessionStore().user = { ...user }
|
||||
})
|
||||
afterEach(() => {
|
||||
wrapper?.unmount()
|
||||
wrapper = undefined
|
||||
useLibraryStore().stopPolling()
|
||||
})
|
||||
|
||||
it('shows the file picker instead of the textarea and reports the picked file', async () => {
|
||||
const router = await viewAt('/import')
|
||||
wrapper = mount(ImportView, { global: { plugins: [router] } })
|
||||
await flushPromises()
|
||||
// Paste mode keeps the textarea; TXT mode replaces it with the picker.
|
||||
expect(wrapper.find('textarea#text').exists()).toBe(true)
|
||||
expect(wrapper.find('[data-testid="file-input"]').exists()).toBe(false)
|
||||
await useTxt(wrapper)
|
||||
expect(wrapper.find('textarea#text').exists()).toBe(false)
|
||||
expect(wrapper.find('[data-testid="file-input"]').exists()).toBe(true)
|
||||
expect(wrapper.text()).toContain('仅支持 UTF-8')
|
||||
|
||||
await chooseFile(wrapper, new File(['Mira opened the workshop.\n'], 'reading.txt', { type: 'text/plain' }))
|
||||
expect(wrapper.get('[data-testid="file-info"]').text()).toBe('reading.txt · UTF-8 · 26 B')
|
||||
// The title is prefilled from the file name and stays editable.
|
||||
expect((wrapper.get('input#title').element as HTMLInputElement).value).toBe('reading')
|
||||
})
|
||||
|
||||
it('refuses a file the server would refuse, before anything is sent', async () => {
|
||||
const fetchMock = vi.spyOn(globalThis, 'fetch')
|
||||
const router = await viewAt('/import')
|
||||
wrapper = mount(ImportView, { global: { plugins: [router] } })
|
||||
await flushPromises()
|
||||
await useTxt(wrapper)
|
||||
|
||||
await chooseFile(wrapper, new File(['# not a txt file\n'], 'notes.md'))
|
||||
expect(wrapper.text()).toContain('请选择 .txt 文件。')
|
||||
// A file that is not valid UTF-8 is rejected by the preview decode.
|
||||
await chooseFile(wrapper, new File([new Uint8Array([0x63, 0x61, 0x66, 0xe9, 0x0a])], 'latin1.txt'))
|
||||
expect(wrapper.text()).toContain('文件不是 UTF-8 编码')
|
||||
await wrapper.find('input#title').setValue('Latin One')
|
||||
await wrapper.find('form').trigger('submit')
|
||||
await flushPromises()
|
||||
expect(fetchMock).not.toHaveBeenCalled()
|
||||
expect(wrapper.text()).toContain('文件不是 UTF-8 编码')
|
||||
|
||||
// No file at all is also refused locally.
|
||||
await chooseFile(wrapper, null)
|
||||
await wrapper.find('form').trigger('submit')
|
||||
await flushPromises()
|
||||
expect(fetchMock).not.toHaveBeenCalled()
|
||||
expect(wrapper.text()).toContain('请选择要导入的 TXT 文件。')
|
||||
})
|
||||
|
||||
it('uploads the file as multipart and opens the created book', async () => {
|
||||
const fetchMock = vi.spyOn(globalThis, 'fetch').mockResolvedValueOnce(uploadResponse())
|
||||
const router = await viewAt('/import')
|
||||
wrapper = mount(ImportView, { global: { plugins: [router] } })
|
||||
await flushPromises()
|
||||
await useTxt(wrapper)
|
||||
const file = new File(['Mira opened the workshop.\n'], 'reading.txt', { type: 'text/plain' })
|
||||
await chooseFile(wrapper, file)
|
||||
await wrapper.find('input#title').setValue('上传的虚构章节')
|
||||
await wrapper.find('form').trigger('submit')
|
||||
await flushPromises()
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledTimes(1)
|
||||
const [url, init] = fetchMock.mock.calls[0]!
|
||||
expect(String(url)).toBe('/api/v1/books/upload')
|
||||
expect(init?.method).toBe('POST')
|
||||
// A multipart body must not be replaced by JSON and must keep the browser's boundary.
|
||||
expect(init?.body).toBeInstanceOf(FormData)
|
||||
expect((init?.headers as Record<string, string>)['Content-Type']).toBeUndefined()
|
||||
const body = init?.body as FormData
|
||||
expect(body.get('requestId')).toMatch(/^[0-9a-f-]{36}$/)
|
||||
expect(body.get('title')).toBe('上传的虚构章节')
|
||||
expect(body.get('language')).toBe('en')
|
||||
expect((body.get('file') as File).name).toBe('reading.txt')
|
||||
expect(await (body.get('file') as File).text()).toBe('Mira opened the workshop.\n')
|
||||
expect(router.currentRoute.value.path).toBe('/books/1')
|
||||
// The form starts clean, so the same file is not submitted twice by accident.
|
||||
expect((wrapper.get('[data-testid="file-input"]').element as HTMLInputElement).value).toBe('')
|
||||
})
|
||||
|
||||
it('reuses one request id when the same upload is retried after a failure', async () => {
|
||||
let attempt = 0
|
||||
const fetchMock = vi.spyOn(globalThis, 'fetch').mockImplementation(async () => {
|
||||
attempt += 1
|
||||
if (attempt === 1) throw new Error('上传中断')
|
||||
return uploadResponse()
|
||||
})
|
||||
const router = await viewAt('/import')
|
||||
wrapper = mount(ImportView, { global: { plugins: [router] } })
|
||||
await flushPromises()
|
||||
await useTxt(wrapper)
|
||||
await chooseFile(wrapper, new File(['Body.\n'], 'retry.txt'))
|
||||
await wrapper.find('input#title').setValue('Retry Upload')
|
||||
await wrapper.find('form').trigger('submit')
|
||||
await flushPromises()
|
||||
expect(wrapper.text()).toContain('上传中断')
|
||||
// The picked file stays selected so the learner can retry without choosing it again.
|
||||
expect(wrapper.get('[data-testid="file-info"]').text()).toContain('retry.txt')
|
||||
|
||||
await wrapper.find('form').trigger('submit')
|
||||
await flushPromises()
|
||||
const ids = fetchMock.mock.calls.map(call => (call[1]?.body as FormData).get('requestId'))
|
||||
expect(ids).toHaveLength(2)
|
||||
expect(ids[0]).toBe(ids[1])
|
||||
expect(router.currentRoute.value.path).toBe('/books/1')
|
||||
})
|
||||
|
||||
it('appends an uploaded chapter to a chosen book without a language field', async () => {
|
||||
const fetchMock = vi.spyOn(globalThis, 'fetch').mockImplementation(async (input, init) => {
|
||||
const url = String(input)
|
||||
if (url.endsWith('/books')) return ok({ items: [{ ...book, chapterCount: 1, pendingCount: 0, processingCount: 0, readyCount: 1, failedCount: 0, ...timestamps }] })
|
||||
expect(url).toBe('/api/v1/books/1/chapters/upload')
|
||||
const body = init?.body as FormData
|
||||
expect(body.get('language')).toBeNull()
|
||||
return ok({ chapter: { ...chapter, id: 10, ordinal: 2 }, job: { id: 6, bookId: 1, chapterId: 10, status: 'pending', attempts: 0, errorReason: '', errorMessage: '', ...timestamps }, duplicate: false })
|
||||
})
|
||||
const router = await viewAt('/import?book=1')
|
||||
wrapper = mount(ImportView, { global: { plugins: [router] } })
|
||||
await flushPromises()
|
||||
await useTxt(wrapper)
|
||||
await chooseFile(wrapper, new File(['Second chapter.\n'], 'second.txt'))
|
||||
await wrapper.find('input#title').setValue('Appended Chapter')
|
||||
await wrapper.find('form').trigger('submit')
|
||||
await flushPromises()
|
||||
|
||||
const posts = fetchMock.mock.calls.filter(call => String(call[0]).includes('/upload'))
|
||||
expect(posts).toHaveLength(1)
|
||||
expect(router.currentRoute.value.path).toBe('/books/1')
|
||||
})
|
||||
|
||||
it('keeps the server message and the form when the upload is rejected', async () => {
|
||||
vi.spyOn(globalThis, 'fetch').mockResolvedValue(new Response(JSON.stringify({ code: 400, msg: '文件不是 UTF-8 编码，请另存为 UTF-8 后重试' }), { status: 400 }))
|
||||
const router = await viewAt('/import')
|
||||
wrapper = mount(ImportView, { global: { plugins: [router] } })
|
||||
await flushPromises()
|
||||
await useTxt(wrapper)
|
||||
await chooseFile(wrapper, new File(['Body.\n'], 'server-rejects.txt'))
|
||||
await wrapper.find('input#title').setValue('Server Rejects')
|
||||
await wrapper.find('form').trigger('submit')
|
||||
await flushPromises()
|
||||
expect(wrapper.text()).toContain('文件不是 UTF-8 编码，请另存为 UTF-8 后重试')
|
||||
expect(wrapper.get('[data-testid="file-info"]').text()).toContain('server-rejects.txt')
|
||||
expect(router.currentRoute.value.path).toBe('/import')
|
||||
})
|
||||
})
|
||||
@@ -58,6 +58,8 @@ export interface SubmitInput { title: string; text: string; target: SubmitTarget
|
||||
export const POLL_INTERVAL_MS = 1500
|
||||
export const TITLE_MAX_LENGTH = 120
|
||||
export const TEXT_MAX_CODE_POINTS = 100000
|
||||
// Mirrors the server limit for one TXT upload.
|
||||
export const TXT_MAX_BYTES = 2 * 1024 * 1024
|
||||
export const NOT_FOUND_MESSAGE = '内容不存在。'
|
||||
export const LANGUAGE_LABEL = '英语'
|
||||
export const LANGUAGE_CODE = 'en'
|
||||
@@ -92,6 +94,24 @@ export function textProblem(text: string): string {
|
||||
return ''
|
||||
}
|
||||
|
||||
/**
|
||||
* Client-side pre-check for a TXT upload. The server validates the file again and stays the
|
||||
* only authority; this only tells the learner about an obviously unusable choice earlier.
|
||||
*/
|
||||
export function fileProblem(file: { name: string; size: number }): string {
|
||||
if (!/\.txt$/i.test(file.name)) return '请选择 .txt 文件。'
|
||||
if (file.size === 0) return '文件是空的，请选择包含英文正文的 UTF-8 TXT。'
|
||||
if (file.size > TXT_MAX_BYTES) return 'TXT 文件不能超过 2 MiB。'
|
||||
return ''
|
||||
}
|
||||
|
||||
/** A readable size for the selected file, e.g. `2 KB` or `1.5 MB`. */
|
||||
export function fileSizeLabel(bytes: number): string {
|
||||
if (bytes < 1024) return `${bytes} B`
|
||||
if (bytes < 1024 * 1024) return `${Math.round(bytes / 1024)} KB`
|
||||
return `${(bytes / (1024 * 1024)).toFixed(1)} MB`
|
||||
}
|
||||
|
||||
/** A failed chapter can be retried as soon as the API told us its job id. */
|
||||
export function canRetry(chapter: Pick<ChapterSummary, 'status' | 'jobId'>): boolean {
|
||||
return chapter.status === 'failed' && chapter.jobId !== null
|
||||
@@ -112,6 +132,7 @@ interface SubmitBookBody { requestId: string; title: string; text: string; langu
|
||||
// An appended chapter owns the language of its book, so the append contract has no language
|
||||
// field; the server rejects unknown fields, and a client that sends one gets HTTP 400.
|
||||
interface SubmitChapterBody { requestId: string; title: string; text: string }
|
||||
export interface UploadInput { title: string; target: SubmitTarget; file: File }
|
||||
interface Created { bookId: number; chapter: ChapterSummary }
|
||||
|
||||
function emptyNavigation(): ChapterNavigation {
|
||||
@@ -365,6 +386,49 @@ export const useLibraryStore = defineStore('library', () => {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Uploads one TXT file. The file and the title share the request id discipline of a paste,
|
||||
* so a repeated upload of the same file answers with the chapter it already created.
|
||||
*/
|
||||
async function upload(input: UploadInput): Promise<number> {
|
||||
const title = input.title.trim()
|
||||
const problem = titleProblem(title) || fileProblem(input.file)
|
||||
if (problem) {
|
||||
submitError.value = problem
|
||||
throw new Error(problem)
|
||||
}
|
||||
const key = `upload\n${title}\n${input.file.name}\n${input.file.size}\n${input.file.lastModified}`
|
||||
if (key !== submissionKey || submissionRequestId === '') {
|
||||
submissionKey = key
|
||||
submissionRequestId = crypto.randomUUID()
|
||||
}
|
||||
const requestId = submissionRequestId
|
||||
const form = new FormData()
|
||||
form.append('requestId', requestId)
|
||||
form.append('title', title)
|
||||
// Only the new-book contract carries a language; appending inherits the book's language.
|
||||
const path = input.target.mode === 'new' ? 'books/upload' : `books/${input.target.bookId}/chapters/upload`
|
||||
if (input.target.mode === 'new') form.append('language', LANGUAGE_CODE)
|
||||
form.append('file', input.file, input.file.name)
|
||||
|
||||
const version = generation
|
||||
const owner = ownerId()
|
||||
submitting.value = true
|
||||
submitError.value = ''
|
||||
try {
|
||||
const result = await session.request<{ book?: BookRef; chapter: ChapterSummary }>(path, 'POST', form)
|
||||
if (isStale(version, owner)) throw new Error('登录状态已变化，请重新提交。')
|
||||
submissionKey = ''
|
||||
submissionRequestId = ''
|
||||
return result.book?.id ?? result.chapter.bookId
|
||||
} catch (reason) {
|
||||
if (!isStale(version, owner)) submitError.value = failureMessage(reason, '上传失败，请稍后重试。')
|
||||
throw reason instanceof Error ? reason : new Error('上传失败，请稍后重试。')
|
||||
} finally {
|
||||
if (!isStale(version, owner)) submitting.value = false
|
||||
}
|
||||
}
|
||||
|
||||
/** The job id comes from the chapter itself, wherever that chapter was loaded from. */
|
||||
function jobIdOf(chapterId: number): number | null {
|
||||
const target = chapters.value.find(item => item.id === chapterId)
|
||||
@@ -461,7 +525,7 @@ export const useLibraryStore = defineStore('library', () => {
|
||||
book, chapters, bookLoading, bookError,
|
||||
chapter, chapterBook, navigation, chapterLoading, chapterError,
|
||||
submitting, submitError, retryingChapterId, readerText,
|
||||
loadBooks, loadBook, loadChapter, submit, retryChapter,
|
||||
loadBooks, loadBook, loadChapter, submit, upload, retryChapter,
|
||||
stopPolling, closeBook, closeChapter, reset,
|
||||
}
|
||||
})
|
||||
|
||||
@@ -36,10 +36,12 @@ export const useSessionStore = defineStore('session', () => {
|
||||
}
|
||||
|
||||
async function request<T>(path: string, method = 'GET', body?: unknown, auth = token, version = generation): Promise<T> {
|
||||
// A multipart body carries its own content type with the boundary, so it is sent as is.
|
||||
const multipart = body instanceof FormData
|
||||
const response = await fetch(`/api/v1/${path}`, {
|
||||
method,
|
||||
headers: { ...(auth ? { Authorization: `Bearer ${auth}` } : {}), ...(body ? { 'Content-Type': 'application/json' } : {}) },
|
||||
body: body ? JSON.stringify(body) : undefined,
|
||||
headers: { ...(auth ? { Authorization: `Bearer ${auth}` } : {}), ...(body && !multipart ? { 'Content-Type': 'application/json' } : {}) },
|
||||
body: body ? (multipart ? body : JSON.stringify(body)) : undefined,
|
||||
cache: 'no-store',
|
||||
})
|
||||
if (response.status === 401 && version === generation) {
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
import { computed, onMounted, onUnmounted, ref, watch } from 'vue'
|
||||
import { RouterLink, useRoute, useRouter } from 'vue-router'
|
||||
import { ElButton, ElInput, ElOption, ElRadio, ElRadioGroup, ElSelect } from 'element-plus'
|
||||
import { LANGUAGE_LABEL, TEXT_MAX_CODE_POINTS, textProblem, titleProblem, useLibraryStore, type SubmitTarget } from '../stores/library'
|
||||
import { LANGUAGE_LABEL, TEXT_MAX_CODE_POINTS, fileProblem, fileSizeLabel, textProblem, titleProblem, useLibraryStore, type SubmitTarget } from '../stores/library'
|
||||
import { useSessionStore } from '../stores/session'
|
||||
|
||||
const session = useSessionStore()
|
||||
@@ -14,13 +14,50 @@ const title = ref('')
|
||||
const text = ref('')
|
||||
const mode = ref<'new' | 'append'>('new')
|
||||
const bookId = ref<number | undefined>(undefined)
|
||||
// The accepted prototype offers both sources on one screen; the server treats them the same.
|
||||
const source = ref<'paste' | 'txt'>('paste')
|
||||
const file = ref<File | null>(null)
|
||||
const fileInfo = ref<{ name: string; encoding: string; size: string } | null>(null)
|
||||
const titleError = ref('')
|
||||
const textError = ref('')
|
||||
const fileError = ref('')
|
||||
const bookError = ref('')
|
||||
|
||||
const length = computed(() => [...text.value].length)
|
||||
const busy = computed(() => library.submitting)
|
||||
|
||||
/** The browser pre-check only replaces the server, it never replaces its verdict. */
|
||||
async function readFile(selected: File): Promise<void> {
|
||||
fileError.value = ''
|
||||
fileInfo.value = null
|
||||
const problem = fileProblem(selected)
|
||||
if (problem) {
|
||||
file.value = null
|
||||
fileError.value = problem
|
||||
return
|
||||
}
|
||||
try {
|
||||
const decoder = new TextDecoder('utf-8', { fatal: true })
|
||||
decoder.decode(await selected.arrayBuffer())
|
||||
} catch {
|
||||
file.value = null
|
||||
fileError.value = '文件不是 UTF-8 编码，请另存为 UTF-8 后重试。'
|
||||
return
|
||||
}
|
||||
file.value = selected
|
||||
fileInfo.value = { name: selected.name, encoding: 'UTF-8', size: fileSizeLabel(selected.size) }
|
||||
// A file usually defines the title; the learner can still change it before submitting.
|
||||
if (!title.value.trim()) title.value = selected.name.replace(/\.txt$/i, '').slice(0, 120)
|
||||
}
|
||||
|
||||
function onFile(event: Event): void {
|
||||
const selected = (event.target as HTMLInputElement).files?.[0]
|
||||
file.value = null
|
||||
fileInfo.value = null
|
||||
fileError.value = ''
|
||||
if (selected) void readFile(selected)
|
||||
}
|
||||
|
||||
function requestedBookId(): number | undefined {
|
||||
const raw = Array.isArray(route.query.book) ? route.query.book[0] : route.query.book
|
||||
if (typeof raw !== 'string' || !/^\d+$/.test(raw)) return undefined
|
||||
@@ -43,23 +80,35 @@ watch(mode, value => {
|
||||
|
||||
watch(title, () => { titleError.value = '' })
|
||||
watch(text, () => { textError.value = '' })
|
||||
// Switching source clears the other source's complaints and its result.
|
||||
watch(source, () => {
|
||||
titleError.value = ''
|
||||
textError.value = ''
|
||||
fileError.value = ''
|
||||
library.submitError = ''
|
||||
})
|
||||
|
||||
async function submit() {
|
||||
if (busy.value) return
|
||||
titleError.value = titleProblem(title.value)
|
||||
textError.value = textProblem(text.value)
|
||||
textError.value = source.value === 'paste' ? textProblem(text.value) : ''
|
||||
fileError.value = source.value === 'txt' && !file.value ? (fileError.value || '请选择要导入的 TXT 文件。') : fileError.value
|
||||
bookError.value = mode.value === 'append' && bookId.value === undefined ? '请选择要追加的书籍。' : ''
|
||||
if (titleError.value || textError.value || bookError.value) return
|
||||
if (titleError.value || textError.value || fileError.value || bookError.value) return
|
||||
const target: SubmitTarget = mode.value === 'append' && bookId.value !== undefined
|
||||
? { mode: 'append', bookId: bookId.value }
|
||||
: { mode: 'new' }
|
||||
try {
|
||||
const createdBookId = await library.submit({ title: title.value, text: text.value, target })
|
||||
const createdBookId = source.value === 'txt' && file.value
|
||||
? await library.upload({ title: title.value, target, file: file.value })
|
||||
: await library.submit({ title: title.value, text: text.value, target })
|
||||
// A response that arrives after the user left this page must not navigate them back.
|
||||
if (disposed) return
|
||||
// The requestId was consumed by this submission, so the form starts clean.
|
||||
title.value = ''
|
||||
text.value = ''
|
||||
file.value = null
|
||||
fileInfo.value = null
|
||||
await router.replace(`/books/${createdBookId}`)
|
||||
} catch {
|
||||
// library.submitError already carries the server message for the template.
|
||||
@@ -92,6 +141,13 @@ onUnmounted(() => {
|
||||
</div>
|
||||
</div>
|
||||
<form class="import-form" novalidate @submit.prevent="submit">
|
||||
<div class="field">
|
||||
<span class="field-label">导入方式</span>
|
||||
<ElRadioGroup v-model="source" :disabled="busy" aria-label="导入方式">
|
||||
<ElRadio value="paste">粘贴文本</ElRadio>
|
||||
<ElRadio value="txt">TXT 文件</ElRadio>
|
||||
</ElRadioGroup>
|
||||
</div>
|
||||
<div class="field">
|
||||
<span class="field-label">语言</span>
|
||||
<p class="fixed-value">{{ LANGUAGE_LABEL }}</p>
|
||||
@@ -116,15 +172,22 @@ onUnmounted(() => {
|
||||
<p v-if="library.booksError" role="alert" class="field-error">{{ library.booksError }}</p>
|
||||
<p v-if="bookError" role="alert" class="field-error">{{ bookError }}</p>
|
||||
</div>
|
||||
<div class="field">
|
||||
<div v-if="source === 'paste'" class="field">
|
||||
<label for="text">正文</label>
|
||||
<ElInput id="text" v-model="text" type="textarea" :rows="12" placeholder="在此粘贴英文正文…" :disabled="busy" />
|
||||
<p class="counter">{{ length }} / {{ TEXT_MAX_CODE_POINTS }} 字符</p>
|
||||
<p v-if="textError" role="alert" class="field-error">{{ textError }}</p>
|
||||
</div>
|
||||
<div v-else class="field">
|
||||
<label for="file">TXT 文件</label>
|
||||
<input id="file" data-testid="file-input" type="file" accept=".txt,text/plain" :disabled="busy" @change="onFile" />
|
||||
<p v-if="fileInfo" class="counter" data-testid="file-info">{{ fileInfo.name }} · {{ fileInfo.encoding }} · {{ fileInfo.size }}</p>
|
||||
<p v-else class="subtle">仅支持 UTF-8 的 .txt 文件，最多 2 MiB；文件不会保存在服务器上。</p>
|
||||
<p v-if="fileError" role="alert" class="field-error">{{ fileError }}</p>
|
||||
</div>
|
||||
<p v-if="library.submitError" role="alert" class="notice">{{ library.submitError }}</p>
|
||||
<div class="form-actions">
|
||||
<ElButton type="primary" native-type="submit" :loading="busy" :disabled="busy">开始处理</ElButton>
|
||||
<ElButton type="primary" native-type="submit" :loading="busy" :disabled="busy">{{ source === 'txt' ? '上传并处理' : '开始处理' }}</ElButton>
|
||||
<RouterLink to="/" class="subtle">返回书库</RouterLink>
|
||||
</div>
|
||||
</form>
|
||||
|
||||
@@ -5,8 +5,8 @@ import (
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
driver "github.com/go-sql-driver/mysql"
|
||||
"github.com/gin-gonic/gin"
|
||||
driver "github.com/go-sql-driver/mysql"
|
||||
admin "go-admin/app/admin/models"
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/clause"
|
||||
@@ -337,6 +337,7 @@ func AnswerReview(tx *gorm.DB, owner int, termID int64, input ReviewAnswerInput,
|
||||
DueAtBefore: before.DueAt, DueAtAfter: next.DueAt, Item: reviewItem(term, review),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// staleAnswer records that this attempt changed nothing because the term had already
|
||||
// moved on. It is a normal outcome of two open tabs, not an error.
|
||||
func staleAnswer(tx *gorm.DB, owner int, term Term, review TermReview, input ReviewAnswerInput, now time.Time) (ReviewAnswerResult, error) {
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
package lexgo
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
|
||||
@@ -130,7 +130,9 @@ func Router(db *gorm.DB, now func() time.Time) *gin.Engine {
|
||||
status = 201
|
||||
}
|
||||
// A repeated paste is answered from the first result, so it is not a new resource.
|
||||
if c.Request.Method == "POST" && (c.FullPath() == "/api/v1/books" || c.FullPath() == "/api/v1/books/:id/chapters") {
|
||||
// A pasted or uploaded chapter is one resource; a repeated submit is not.
|
||||
if c.Request.Method == "POST" && (c.FullPath() == "/api/v1/books" || c.FullPath() == "/api/v1/books/:id/chapters" ||
|
||||
c.FullPath() == "/api/v1/books/upload" || c.FullPath() == "/api/v1/books/:id/chapters/upload") {
|
||||
if paste, ok := data.(PasteResult); ok && !paste.Duplicate {
|
||||
status = 201
|
||||
}
|
||||
@@ -231,13 +233,6 @@ func Router(db *gorm.DB, now func() time.Time) *gin.Engine {
|
||||
}
|
||||
return updateAccount(tx, id, u.UserId, input)
|
||||
}))
|
||||
pathID := func(c *gin.Context, message string) (int64, error) {
|
||||
id, err := strconv.ParseInt(c.Param("id"), 10, 64)
|
||||
if err != nil || id <= 0 {
|
||||
return 0, failure(404, message)
|
||||
}
|
||||
return id, nil
|
||||
}
|
||||
v.POST("/books", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) {
|
||||
var input PasteBookInput
|
||||
if err := decodeLimit(c, &input, maxPasteBodyBytes); err != nil {
|
||||
@@ -309,10 +304,21 @@ func Router(db *gorm.DB, now func() time.Time) *gin.Engine {
|
||||
registerDictionaryRoutes(v, protect, now)
|
||||
registerTermRoutes(v, protect, now)
|
||||
registerReviewRoutes(v, protect, now)
|
||||
registerUploadRoutes(v, protect, now)
|
||||
r.NoRoute(func(c *gin.Context) { respond(c, 404, nil, failure(404, "页面或接口不存在")) })
|
||||
return r
|
||||
}
|
||||
|
||||
// pathID reads a positive numeric path parameter; a malformed id is reported like a
|
||||
// missing resource so it cannot be used to probe for other accounts' rows.
|
||||
func pathID(c *gin.Context, message string) (int64, error) {
|
||||
id, err := strconv.ParseInt(c.Param("id"), 10, 64)
|
||||
if err != nil || id <= 0 {
|
||||
return 0, failure(404, message)
|
||||
}
|
||||
return id, nil
|
||||
}
|
||||
|
||||
func decode(c *gin.Context, value any) error { return decodeLimit(c, value, maxJSONBodyBytes) }
|
||||
|
||||
func decodeLimit(c *gin.Context, value any, limit int64) error {
|
||||
|
||||
@@ -0,0 +1,183 @@
|
||||
package lexgo
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/gin-gonic/gin"
|
||||
admin "go-admin/app/admin/models"
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
// A TXT upload is decoded in memory and handed to the same paste pipeline. Nothing is
|
||||
// written to disk: there is no temporary file to leak, and the client file name never
|
||||
// becomes a path, so it cannot reach outside the request.
|
||||
const maxTextUploadBytes = 2 << 20
|
||||
|
||||
var (
|
||||
utf8BOM = []byte{0xEF, 0xBB, 0xBF}
|
||||
utf16BOMBig = []byte{0xFE, 0xFF}
|
||||
utf16BOMSmall = []byte{0xFF, 0xFE}
|
||||
)
|
||||
|
||||
// decodeTextUpload turns an uploaded TXT into the exact text the reader will show. Only
|
||||
// UTF-8 is accepted: an optional BOM is removed before the text is validated, and any byte
|
||||
// that is not valid UTF-8 rejects the file instead of being replaced, so a chapter never
|
||||
// contains a substitute character the learner did not write.
|
||||
func decodeTextUpload(raw []byte) (string, error) {
|
||||
if len(raw) > maxTextUploadBytes {
|
||||
return "", failure(400, "TXT 文件不能超过 2 MiB")
|
||||
}
|
||||
if bytes.HasPrefix(raw, utf16BOMBig) || bytes.HasPrefix(raw, utf16BOMSmall) {
|
||||
return "", failure(400, "文件是 UTF-16 编码，请另存为 UTF-8 后重试")
|
||||
}
|
||||
raw = bytes.TrimPrefix(raw, utf8BOM)
|
||||
if !utf8.Valid(raw) {
|
||||
return "", failure(400, "文件不是 UTF-8 编码，请另存为 UTF-8 后重试")
|
||||
}
|
||||
if bytes.IndexByte(raw, 0) >= 0 {
|
||||
return "", failure(400, "文件包含无法处理的字符，请另存为纯文本后重试")
|
||||
}
|
||||
return string(raw), nil
|
||||
}
|
||||
|
||||
type textUpload struct {
|
||||
RequestID string
|
||||
Title string
|
||||
Language string
|
||||
Text string
|
||||
}
|
||||
|
||||
// readTextUpload parses the multipart submission: one file part plus a whitelist of text
|
||||
// fields. The uploaded name is never read, not even for validation, because it only serves
|
||||
// display in the browser.
|
||||
func readTextUpload(c *gin.Context, allowLanguage bool) (textUpload, error) {
|
||||
bad := failure(400, "TXT 上传无效，请选择 UTF-8 的 .txt 文件并填写标题")
|
||||
c.Request.Body = http.MaxBytesReader(c.Writer, c.Request.Body, maxTextUploadBytes+(64<<10))
|
||||
reader, err := c.Request.MultipartReader()
|
||||
if err != nil {
|
||||
return textUpload{}, bad
|
||||
}
|
||||
fields := map[string]string{}
|
||||
var content []byte
|
||||
for {
|
||||
part, err := reader.NextPart()
|
||||
if errors.Is(err, io.EOF) {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return textUpload{}, uploadBodyError(err, bad)
|
||||
}
|
||||
name := part.FormName()
|
||||
if name == "file" {
|
||||
if content != nil {
|
||||
part.Close()
|
||||
return textUpload{}, bad
|
||||
}
|
||||
content, err = io.ReadAll(io.LimitReader(part, maxTextUploadBytes+1))
|
||||
part.Close()
|
||||
if err != nil {
|
||||
return textUpload{}, uploadBodyError(err, bad)
|
||||
}
|
||||
if len(content) == 0 || len(content) > maxTextUploadBytes {
|
||||
return textUpload{}, failure(400, "TXT 文件不能为空且不能超过 2 MiB")
|
||||
}
|
||||
continue
|
||||
}
|
||||
if !textUploadField(name, allowLanguage) {
|
||||
part.Close()
|
||||
return textUpload{}, bad
|
||||
}
|
||||
if _, exists := fields[name]; exists {
|
||||
part.Close()
|
||||
return textUpload{}, bad
|
||||
}
|
||||
value, readErr := io.ReadAll(io.LimitReader(part, 1025))
|
||||
part.Close()
|
||||
if readErr != nil {
|
||||
return textUpload{}, uploadBodyError(readErr, bad)
|
||||
}
|
||||
if len(value) > 1024 || !utf8.Valid(value) {
|
||||
return textUpload{}, bad
|
||||
}
|
||||
fields[name] = string(value)
|
||||
}
|
||||
if content == nil {
|
||||
return textUpload{}, bad
|
||||
}
|
||||
text, err := decodeTextUpload(content)
|
||||
if err != nil {
|
||||
return textUpload{}, err
|
||||
}
|
||||
return textUpload{RequestID: fields["requestId"], Title: fields["title"], Language: fields["language"], Text: text}, nil
|
||||
}
|
||||
|
||||
func textUploadField(name string, allowLanguage bool) bool {
|
||||
switch name {
|
||||
case "requestId", "title":
|
||||
return true
|
||||
case "language":
|
||||
return allowLanguage
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
// An oversized body is reported as a size problem, not as an invalid upload.
|
||||
func uploadBodyError(err error, bad error) error {
|
||||
var tooLarge *http.MaxBytesError
|
||||
if errors.As(err, &tooLarge) {
|
||||
return failure(400, "TXT 文件不能超过 2 MiB")
|
||||
}
|
||||
return bad
|
||||
}
|
||||
|
||||
// registerUploadRoutes reuses the paste pipeline: the decoded file becomes the same
|
||||
// PasteBook/PasteChapter input, so idempotency, ownership and the ingest job behave exactly
|
||||
// as they do for pasted text. One upload at a time keeps concurrent large decodes bounded.
|
||||
func registerUploadRoutes(v *gin.RouterGroup, protect func(bool, func(*gin.Context, *gorm.DB, admin.SysUser) (any, error)) gin.HandlerFunc, now func() time.Time) {
|
||||
gate := make(chan struct{}, 1)
|
||||
enter := func() error {
|
||||
select {
|
||||
case gate <- struct{}{}:
|
||||
return nil
|
||||
default:
|
||||
return failure(429, "已有文件正在上传，请稍后重试")
|
||||
}
|
||||
}
|
||||
leave := func() { <-gate }
|
||||
v.POST("/books/upload", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) {
|
||||
if err := enter(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer leave()
|
||||
upload, err := readTextUpload(c, true)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return PasteBook(tx, u.UserId, now(), PasteBookInput{
|
||||
RequestID: upload.RequestID, Title: upload.Title, Text: upload.Text, Language: upload.Language,
|
||||
})
|
||||
}))
|
||||
v.POST("/books/:id/chapters/upload", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) {
|
||||
bookID, err := pathID(c, "书籍不存在")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := enter(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer leave()
|
||||
upload, err := readTextUpload(c, false)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return PasteChapter(tx, u.UserId, bookID, now(), PasteChapterInput{
|
||||
RequestID: upload.RequestID, Title: upload.Title, Text: upload.Text,
|
||||
})
|
||||
}))
|
||||
}
|
||||
@@ -0,0 +1,376 @@
|
||||
package lexgo
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"mime/multipart"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/gin-gonic/gin"
|
||||
)
|
||||
|
||||
// uploadFixture is the same character set the paste contract preserves, written as the bytes
|
||||
// a file would hold: CRLF and LF, a tab, curly quotes, an em dash, an ellipsis, an emoji, a
|
||||
// combining acute accent, a trailing space run and an empty final line.
|
||||
const uploadFixture = "Mira opened the workshop.\r\n\r\n\tThe sign read “A small step…” — café e\u0301 🙂\r\nTrailing spaces here: \n\n"
|
||||
|
||||
// uploadFile posts one multipart TXT submission. A nil file means "no file part", and an
|
||||
// empty name means "no file name", so the rejection paths stay testable.
|
||||
func uploadFile(t *testing.T, r *gin.Engine, token, path, fileName string, content []byte, fields map[string]string) (int, string, pasteResponse) {
|
||||
t.Helper()
|
||||
var body bytes.Buffer
|
||||
writer := multipart.NewWriter(&body)
|
||||
names := make([]string, 0, len(fields))
|
||||
for name := range fields {
|
||||
names = append(names, name)
|
||||
}
|
||||
// Field order must be stable so a failure message is reproducible.
|
||||
for _, name := range []string{"requestId", "title", "language"} {
|
||||
if value, ok := fields[name]; ok {
|
||||
if err := writer.WriteField(name, value); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
names = removeString(names, name)
|
||||
}
|
||||
}
|
||||
for _, name := range names {
|
||||
if err := writer.WriteField(name, fields[name]); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if content != nil {
|
||||
part, err := writer.CreateFormFile("file", fileName)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = part.Write(content); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
writer.Close()
|
||||
request := httptest.NewRequest("POST", path, &body)
|
||||
request.Header.Set("Content-Type", writer.FormDataContentType())
|
||||
if token != "" {
|
||||
request.Header.Set("Authorization", "Bearer "+token)
|
||||
}
|
||||
response := httptest.NewRecorder()
|
||||
r.ServeHTTP(response, request)
|
||||
var envelope struct {
|
||||
Msg string `json:"msg"`
|
||||
Data json.RawMessage `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(response.Body.Bytes(), &envelope); err != nil {
|
||||
t.Fatalf("upload response for %s: %v", path, err)
|
||||
}
|
||||
var out pasteResponse
|
||||
if len(envelope.Data) > 0 {
|
||||
json.Unmarshal(envelope.Data, &out)
|
||||
}
|
||||
return response.Code, envelope.Msg, out
|
||||
}
|
||||
|
||||
func removeString(values []string, target string) []string {
|
||||
result := values[:0]
|
||||
for _, value := range values {
|
||||
if value != target {
|
||||
result = append(result, value)
|
||||
}
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
func TestDecodeTextUploadRules(t *testing.T) {
|
||||
// Valid UTF-8 is returned exactly as received, including unusual but legal characters.
|
||||
if text, err := decodeTextUpload([]byte(uploadFixture)); err != nil || text != uploadFixture {
|
||||
t.Fatalf("valid file: %q %v", text, err)
|
||||
}
|
||||
// A UTF-8 BOM is removed and never becomes part of the original text.
|
||||
withBOM := append(append([]byte{}, utf8BOM...), []byte("BOM before text\n")...)
|
||||
if text, err := decodeTextUpload(withBOM); err != nil || text != "BOM before text\n" {
|
||||
t.Fatalf("BOM file: %q %v", text, err)
|
||||
}
|
||||
// Only a BOM leaves an empty text, which the paste rules then reject.
|
||||
if text, err := decodeTextUpload(append([]byte{}, utf8BOM...)); err != nil || text != "" {
|
||||
t.Fatalf("BOM only: %q %v", text, err)
|
||||
}
|
||||
rejected := []struct {
|
||||
name string
|
||||
content []byte
|
||||
message string
|
||||
}{
|
||||
{"invalid UTF-8", []byte{0x41, 0x80, 0x42}, "UTF-8"},
|
||||
{"latin-1 text", []byte("caf\xe9 plain\n"), "UTF-8"},
|
||||
{"UTF-16 little endian", []byte{0xFF, 0xFE, 0x41, 0x00}, "UTF-16"},
|
||||
{"UTF-16 big endian", []byte{0xFE, 0xFF, 0x00, 0x41}, "UTF-16"},
|
||||
{"NUL byte", []byte("text\x00more"), "无法处理"},
|
||||
}
|
||||
for _, tc := range rejected {
|
||||
text, err := decodeTextUpload(tc.content)
|
||||
if err == nil || text != "" {
|
||||
t.Fatalf("%s was accepted as %q", tc.name, text)
|
||||
}
|
||||
api, ok := err.(*apiError)
|
||||
if !ok || api.status != 400 || !strings.Contains(api.message, tc.message) {
|
||||
t.Fatalf("%s message: %v", tc.name, err)
|
||||
}
|
||||
}
|
||||
// The size boundary is exact on both sides, and the byte limit is checked before decoding.
|
||||
if _, err := decodeTextUpload(bytes.Repeat([]byte("a"), maxTextUploadBytes)); err != nil {
|
||||
t.Fatalf("a file at the size limit must be accepted: %v", err)
|
||||
}
|
||||
if _, err := decodeTextUpload(bytes.Repeat([]byte("a"), maxTextUploadBytes+1)); err == nil {
|
||||
t.Fatal("a file over the size limit must be rejected")
|
||||
}
|
||||
// The paste rules still bound one chapter, so the byte limit cannot smuggle in more text.
|
||||
if _, _, _, err := validatePaste("title", strings.Repeat("a", maxChapterRunes)); err != nil {
|
||||
t.Fatalf("the exact chapter limit must be accepted: %v", err)
|
||||
}
|
||||
if _, _, _, err := validatePaste("title", strings.Repeat("a", maxChapterRunes+1)); err == nil {
|
||||
t.Fatal("the chapter code point limit must still apply to uploaded text")
|
||||
}
|
||||
}
|
||||
|
||||
// TestMySQLTextUploadImportPath covers the accepted file: it creates the same book, chapter
|
||||
// and job as a paste, and the reader text equals the file byte for byte.
|
||||
func TestMySQLTextUploadImportPath(t *testing.T) {
|
||||
db, r, owner := libraryFixture(t)
|
||||
learner := newLearner(t, r, owner.Token)
|
||||
drainIngest(t, db)
|
||||
|
||||
fields := map[string]string{"requestId": "upload-fixture-0001", "title": "The Workshop Upload", "language": "en"}
|
||||
code, msg, uploaded := uploadFile(t, r, learner.Token, "/api/v1/books/upload", "workshop.txt", []byte(uploadFixture), fields)
|
||||
if code != 201 {
|
||||
t.Fatalf("upload status %d (%s)", code, msg)
|
||||
}
|
||||
if uploaded.Book == nil || uploaded.Book.Title != "The Workshop Upload" || uploaded.Book.Language != "en" {
|
||||
t.Fatalf("unexpected book %+v", uploaded.Book)
|
||||
}
|
||||
if uploaded.Chapter.Ordinal != 1 || uploaded.Chapter.Status != statusPending || uploaded.Duplicate {
|
||||
t.Fatalf("unexpected chapter %+v", uploaded.Chapter)
|
||||
}
|
||||
|
||||
// The worker publishes the chapter, and the reader shows exactly the file content.
|
||||
drainIngest(t, db)
|
||||
code, ready := readChapter(t, r, learner.Token, uploaded.Chapter.ID)
|
||||
if code != 200 || ready.Chapter.Status != statusReady || ready.Chapter.OriginalText != uploadFixture {
|
||||
t.Fatalf("reader text: status %d, %+v", code, ready.Chapter)
|
||||
}
|
||||
if want := len([]rune(uploadFixture)); ready.Chapter.CharCount != want {
|
||||
t.Fatalf("charCount %d, want %d", ready.Chapter.CharCount, want)
|
||||
}
|
||||
|
||||
// A UTF-8 BOM is stripped, so the reader never shows it.
|
||||
bomFields := map[string]string{"requestId": "upload-bom-000002", "title": "BOM Upload", "language": "en"}
|
||||
withBOM := append(append([]byte{}, utf8BOM...), []byte("Plain text with BOM.\n")...)
|
||||
code, msg, bom := uploadFile(t, r, learner.Token, "/api/v1/books/upload", "bom.txt", withBOM, bomFields)
|
||||
if code != 201 {
|
||||
t.Fatalf("BOM upload status %d (%s)", code, msg)
|
||||
}
|
||||
drainIngest(t, db)
|
||||
if code, read := readChapter(t, r, learner.Token, bom.Chapter.ID); code != 200 || read.Chapter.OriginalText != "Plain text with BOM.\n" {
|
||||
t.Fatalf("BOM text: status %d, %q", code, read.Chapter.OriginalText)
|
||||
}
|
||||
}
|
||||
|
||||
// TestMySQLTextUploadIdempotencyAndAppend reuses the paste job rules: one file yields one
|
||||
// chapter, a repeated upload answers with that chapter, and the same request id with other
|
||||
// content is a conflict.
|
||||
func TestMySQLTextUploadIdempotencyAndAppend(t *testing.T) {
|
||||
db, r, owner := libraryFixture(t)
|
||||
learner := newLearner(t, r, owner.Token)
|
||||
other := newLearner(t, r, owner.Token)
|
||||
drainIngest(t, db)
|
||||
|
||||
fields := map[string]string{"requestId": "upload-repeat-0001", "title": "Repeat Upload", "language": "en"}
|
||||
code, msg, first := uploadFile(t, r, learner.Token, "/api/v1/books/upload", "repeat.txt", []byte("First upload body.\n"), fields)
|
||||
if code != 201 {
|
||||
t.Fatalf("first upload %d (%s)", code, msg)
|
||||
}
|
||||
code, msg, again := uploadFile(t, r, learner.Token, "/api/v1/books/upload", "repeat.txt", []byte("First upload body.\n"), fields)
|
||||
if code != 200 || !again.Duplicate || again.Chapter.ID != first.Chapter.ID {
|
||||
t.Fatalf("repeat upload %d (%s): %+v", code, msg, again)
|
||||
}
|
||||
var chapters int64
|
||||
db.Model(&Chapter{}).Where("book_id = ?", first.Book.ID).Count(&chapters)
|
||||
if chapters != 1 {
|
||||
t.Fatalf("a repeated upload created %d chapters", chapters)
|
||||
}
|
||||
// The same request id with other content is a conflict, not a second chapter.
|
||||
code, msg, _ = uploadFile(t, r, learner.Token, "/api/v1/books/upload", "repeat.txt", []byte("Different body.\n"), fields)
|
||||
if code != 409 {
|
||||
t.Fatalf("changed content status %d (%s)", code, msg)
|
||||
}
|
||||
|
||||
// Appending uses the same rules and the same job pipeline.
|
||||
appendFields := map[string]string{"requestId": "upload-append-0002", "title": "Second Chapter"}
|
||||
path := fmt.Sprintf("/api/v1/books/%d/chapters/upload", first.Book.ID)
|
||||
code, msg, appended := uploadFile(t, r, learner.Token, path, "second.txt", []byte("A single plain paragraph.\n"), appendFields)
|
||||
if code != 201 {
|
||||
t.Fatalf("append upload %d (%s)", code, msg)
|
||||
}
|
||||
if appended.Chapter.Ordinal != 2 || appended.Chapter.BookID != first.Book.ID {
|
||||
t.Fatalf("unexpected appended chapter %+v", appended.Chapter)
|
||||
}
|
||||
code, msg, againAppend := uploadFile(t, r, learner.Token, path, "second.txt", []byte("A single plain paragraph.\n"), appendFields)
|
||||
if code != 200 || !againAppend.Duplicate || againAppend.Chapter.ID != appended.Chapter.ID {
|
||||
t.Fatalf("repeated append %d (%s): %+v", code, msg, againAppend)
|
||||
}
|
||||
// The append endpoint does not accept a language field: the book owns the language.
|
||||
code, msg, _ = uploadFile(t, r, learner.Token, path, "second.txt", []byte("Another body.\n"),
|
||||
map[string]string{"requestId": "upload-append-lang-0004", "title": "Second Chapter", "language": "en"})
|
||||
if code != 400 {
|
||||
t.Fatalf("append with a language field: status %d (%s)", code, msg)
|
||||
}
|
||||
|
||||
// Another account cannot append into this book, and never learns whether it exists.
|
||||
code, foreignMsg, _ := uploadFile(t, r, other.Token, path, "second.txt", []byte("Foreign body.\n"), map[string]string{"requestId": "upload-foreign-0003", "title": "Foreign"})
|
||||
if code != 404 {
|
||||
t.Fatalf("foreign append status %d (%s)", code, foreignMsg)
|
||||
}
|
||||
// The other account's own library stays empty.
|
||||
_, list := bookList(t, r, other.Token)
|
||||
if len(list.Items) != 0 {
|
||||
t.Fatalf("the other account must not see this book: %+v", list.Items)
|
||||
}
|
||||
}
|
||||
|
||||
// TestMySQLTextUploadRejectsInvalidSubmissions covers the validation surface, including the
|
||||
// client file name, which is never used as a path.
|
||||
func TestMySQLTextUploadRejectsInvalidSubmissions(t *testing.T) {
|
||||
db, r, owner := libraryFixture(t)
|
||||
learner := newLearner(t, r, owner.Token)
|
||||
drainIngest(t, db)
|
||||
|
||||
base := map[string]string{"requestId": "upload-invalid-0001", "title": "Invalid Upload", "language": "en"}
|
||||
cases := []struct {
|
||||
name string
|
||||
fileName string
|
||||
content []byte
|
||||
fields map[string]string
|
||||
status int
|
||||
}{
|
||||
{"no file part", "", nil, base, 400},
|
||||
{"empty file", "empty.txt", []byte{}, base, 400},
|
||||
{"empty file part", "empty.txt", []byte{}, base, 400},
|
||||
{"whitespace only", "blank.txt", []byte(" \n\t\n"), base, 400},
|
||||
{"BOM only", "bom.txt", append([]byte{}, utf8BOM...), base, 400},
|
||||
{"invalid UTF-8", "latin1.txt", []byte("caf\xe9\n"), base, 400},
|
||||
{"UTF-16 file", "unicode.txt", []byte{0xFF, 0xFE, 0x41, 0x00}, base, 400},
|
||||
{"oversized file", "big.txt", bytes.Repeat([]byte("a"), maxTextUploadBytes+1), base, 400},
|
||||
{"missing request id", "text.txt", []byte("Body.\n"), map[string]string{"title": "Invalid Upload", "language": "en"}, 400},
|
||||
{"missing title", "text.txt", []byte("Body.\n"), map[string]string{"requestId": "upload-invalid-0002", "language": "en"}, 400},
|
||||
{"unsupported language", "text.txt", []byte("Body.\n"), map[string]string{"requestId": "upload-invalid-0004", "title": "Invalid Upload", "language": "fr"}, 400},
|
||||
{"short request id", "text.txt", []byte("Body.\n"), map[string]string{"requestId": "short", "title": "Invalid Upload", "language": "en"}, 400},
|
||||
{"unknown field", "text.txt", []byte("Body.\n"), map[string]string{"requestId": "upload-invalid-0005", "title": "Invalid Upload", "language": "en", "ownerId": "9"}, 400},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
code, msg, _ := uploadFile(t, r, learner.Token, "/api/v1/books/upload", tc.fileName, tc.content, tc.fields)
|
||||
if code != tc.status {
|
||||
t.Fatalf("%s: status %d (%s), want %d", tc.name, code, msg, tc.status)
|
||||
}
|
||||
if msg == "" {
|
||||
t.Fatalf("%s: rejection without a readable message", tc.name)
|
||||
}
|
||||
}
|
||||
// No rejected submission left a book behind.
|
||||
_, list := bookList(t, r, learner.Token)
|
||||
if len(list.Items) != 0 {
|
||||
t.Fatalf("rejected uploads created books: %+v", list.Items)
|
||||
}
|
||||
|
||||
// A missing language field follows the paste rule and defaults to English.
|
||||
code, msg, defaulted := uploadFile(t, r, learner.Token, "/api/v1/books/upload", "default.txt", []byte("Body without language.\n"),
|
||||
map[string]string{"requestId": "upload-default-lang-0007", "title": "Default Language"})
|
||||
if code != 201 || defaulted.Book == nil || defaulted.Book.Language != "en" {
|
||||
t.Fatalf("missing language must default to English: status %d (%s) book %+v", code, msg, defaulted.Book)
|
||||
}
|
||||
|
||||
// The uploaded name is only a display string: a traversal-shaped name changes nothing.
|
||||
hostile := map[string]string{"requestId": "upload-hostile-0006", "title": "Hostile Name", "language": "en"}
|
||||
code, msg, uploaded := uploadFile(t, r, learner.Token, "/api/v1/books/upload", `..\..\windows\system32\evil.txt`, []byte("Hostile but harmless.\n"), hostile)
|
||||
if code != 201 {
|
||||
t.Fatalf("hostile name status %d (%s)", code, msg)
|
||||
}
|
||||
var chapter Chapter
|
||||
if err := db.First(&chapter, uploaded.Chapter.ID).Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var book Book
|
||||
if err := db.First(&book, uploaded.Book.ID).Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, field := range []string{chapter.Title, chapter.OriginalText, book.Title} {
|
||||
if strings.Contains(field, "evil") || strings.Contains(field, "system32") || strings.Contains(field, `..`) {
|
||||
t.Fatalf("the uploaded name leaked into stored data: %q", field)
|
||||
}
|
||||
}
|
||||
|
||||
// An unrelated content type is not a multipart upload.
|
||||
request := httptest.NewRequest("POST", "/api/v1/books/upload", strings.NewReader(`{"requestId":"json-upload-0001"}`))
|
||||
request.Header.Set("Content-Type", "application/json")
|
||||
request.Header.Set("Authorization", "Bearer "+learner.Token)
|
||||
response := httptest.NewRecorder()
|
||||
r.ServeHTTP(response, request)
|
||||
if response.Code != 400 {
|
||||
t.Fatalf("JSON body accepted as an upload: %d", response.Code)
|
||||
}
|
||||
// Uploading without a session is rejected before any parsing.
|
||||
code, _, _ = uploadFile(t, r, "", "/api/v1/books/upload", "text.txt", []byte("Body.\n"), base)
|
||||
if code != 401 {
|
||||
t.Fatalf("anonymous upload status %d", code)
|
||||
}
|
||||
}
|
||||
|
||||
// TestMySQLTextUploadKeepsChapterLimit proves the byte cap cannot bypass the one-chapter
|
||||
// code point rule, and that a large but legal file is stored completely.
|
||||
func TestMySQLTextUploadKeepsChapterLimit(t *testing.T) {
|
||||
db, r, owner := libraryFixture(t)
|
||||
learner := newLearner(t, r, owner.Token)
|
||||
drainIngest(t, db)
|
||||
|
||||
// Exactly at the chapter limit: accepted, and the stored text keeps its full length.
|
||||
atLimit := strings.Repeat("a", maxChapterRunes-1) + "\n"
|
||||
fields := map[string]string{"requestId": "upload-limit-0001", "title": "At The Limit", "language": "en"}
|
||||
code, msg, uploaded := uploadFile(t, r, learner.Token, "/api/v1/books/upload", "limit.txt", []byte(atLimit), fields)
|
||||
if code != 201 {
|
||||
t.Fatalf("upload at the chapter limit %d (%s)", code, msg)
|
||||
}
|
||||
drainIngest(t, db)
|
||||
if code, read := readChapter(t, r, learner.Token, uploaded.Chapter.ID); code != 200 || read.Chapter.OriginalText != atLimit {
|
||||
t.Fatalf("chapter at the limit: status %d, length %d", code, len([]rune(read.Chapter.OriginalText)))
|
||||
}
|
||||
// One code point more is rejected by the same rule that already applies to a paste.
|
||||
overLimit := strings.Repeat("a", maxChapterRunes+1)
|
||||
code, msg, _ = uploadFile(t, r, learner.Token, "/api/v1/books/upload", "over.txt", []byte(overLimit), map[string]string{"requestId": "upload-limit-0002", "title": "Over The Limit", "language": "en"})
|
||||
if code != 400 || !strings.Contains(msg, "100000") {
|
||||
t.Fatalf("upload over the chapter limit: status %d (%s)", code, msg)
|
||||
}
|
||||
}
|
||||
|
||||
// TestMySQLTextUploadAndPasteShareOnePipeline checks the two entry points cannot produce a
|
||||
// second chapter for the same submitted content when the client stays on one request id.
|
||||
func TestMySQLTextUploadAndPasteShareOnePipeline(t *testing.T) {
|
||||
db, r, owner := libraryFixture(t)
|
||||
learner := newLearner(t, r, owner.Token)
|
||||
drainIngest(t, db)
|
||||
|
||||
fields := map[string]string{"requestId": "upload-shared-0001", "title": "Shared Pipeline", "language": "en"}
|
||||
code, msg, uploaded := uploadFile(t, r, learner.Token, "/api/v1/books/upload", "shared.txt", []byte("Shared body.\n"), fields)
|
||||
if code != 201 {
|
||||
t.Fatalf("upload %d (%s)", code, msg)
|
||||
}
|
||||
// The same request id through the paste endpoint answers with the uploaded chapter.
|
||||
code, pasted := pasteBook(t, r, learner.Token, map[string]string{
|
||||
"requestId": "upload-shared-0001", "title": "Shared Pipeline", "text": "Shared body.\n", "language": "en"})
|
||||
if code != 200 || !pasted.Duplicate || pasted.Chapter.ID != uploaded.Chapter.ID {
|
||||
t.Fatalf("paste after upload %d: %+v", code, pasted)
|
||||
}
|
||||
var chapters int64
|
||||
db.Model(&Chapter{}).Where("book_id = ?", uploaded.Book.ID).Count(&chapters)
|
||||
if chapters != 1 {
|
||||
t.Fatalf("the two entry points created %d chapters", chapters)
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user