fix: 整改 #5 审核问题 R1~R4 (#5)

R1 追加契约:学习端把新建与追加拆成两个请求体,追加不再发送 language(后端严格
解码会拒绝未知字段,此前真实追加返回 400);补充断言真实请求格式的回归测试。
R2 运行期恢复:启动恢复与运行期清扫合并为一处,worker 每秒把停留超过 15 秒的
processing 任务重新入队,超过 5 次尝试的任务置为 failed(attempts_exhausted);
人工重试重置尝试次数;日志如实区分“已入队”与“等待下一次清扫”。
R3 离页作废在途请求:closeBook/closeChapter 推进请求序号并清理 loading,导入页
在卸载后的成功响应不再触发跳转。
R4 重试自愈:重试被接受后先应用返回的 pending 状态并继续轮询,静默刷新失败不再
让页面停在处理失败。

测试:Go 20 个顶层用例通过(新增 2 项);学习端 38 项通过,其中 7 项在整改前代码
上实际失败;真实联调验证追加可用、被中断任务约 0.5 秒内在运行中自动恢复、重试在
首次刷新失败后自动显示最终结果。

文档:Wiki 先写后回读(架构、业务规则、本地验证),导出核心镜像。
This commit was merged in pull request #23.
This commit is contained in:
ila
2026-09-11 11:02:32 +08:00
parent a55708cd37
commit b18f9cc4a5
11 changed files with 497 additions and 39 deletions
+13 -2
View File
@@ -2,8 +2,8 @@
generated: true (请先修改 Gitea Wiki,禁止直接编辑本文件)
wiki_page: Architecture-and-Code-Map
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Architecture-and-Code-Map.-
wiki_revision: fa7d3d66e6e68164ee97a23dc20a737fcaba7a11
synchronized_at: 2026-09-10T16:35:18Z
wiki_revision: f981b86a7c6e5d2e353142d3006dcd6e0636c3ee
synchronized_at: 2026-09-11T03:01:48Z
<!-- gitea-wiki-mirror:end -->
# 架构与代码地图
@@ -210,3 +210,14 @@ WordNet 使用 ZIP 内原始 index/data/exception 文件,不使用 SysDict 或
lexgo_books(owner_id, title, language)、lexgo_chapters(book_id, owner_id, ordinal, title, original_text MEDIUMTEXT, char_count, content_sha256, status, error_reason) 与 lexgo_ingest_jobs(owner_id, book_id, chapter_id, request_key, content_sha256, status, attempts, error_reason, finished_at)。owner_id 在章节与任务上冗余存放,使任何查询都能直接按认证身份过滤而不依赖连接;UNIQUE(book_id, ordinal) 与 UNIQUE(owner_id, request_key) 分别阻止重复章节与重复提交。启动检查要求版本 3,服务不自动迁移。
并发重复提交:请求命中 request_key 唯一键冲突后,用加锁读读取已提交结果,因为该请求事务的快照早于并发提交;因此两个并发相同提交只会产生一个章节,另一个得到 duplicate=true 的首次结果。
## #5 审核整改(R1~R4,2026-09-11)
提交见工单 #5 的整改评论;本条记录实现与验证方式。
- 追加契约(R1):学习端把新建与追加拆成两个请求类型,追加不发送 language;后端保持严格解码,并新增回归测试断言“追加带 language 返回 400、不带则 201”,学习端单测断言追加请求体只有 requestId/title/text。
- 运行期任务恢复(R2):`server/app/lexgo/ingest.go` 的恢复逻辑合并为一处——启动恢复使用阈值 0,运行期每轮清扫使用 15 秒阈值并把超过 5 次尝试的任务置为 failed(原因码 attempts_exhausted);`cmd/lexgo/main.go` 的 worker 每秒先清扫再处理,日志分别说明“已重新入队”与“本批未完成、等待下一次清扫”,不再声称已完成实际跳过的重试。
- 离页作废在途请求(R3):`closeBook`/`closeChapter` 推进请求序号并清理 loading;`ImportView` 记录是否已卸载,卸载后的成功响应不再触发跳转。
- 重试自愈(R4):`retryChapter` 先把重试返回的章节状态应用到列表与阅读器并重新安排轮询,再做静默刷新。
验证:Go 全量用例 20 项通过(新增运行期恢复与尝试上限两项);学习端单测 38 项通过,其中 7 项在整改前的代码上复现失败;真实联调确认追加路径可用、被中断的任务在运行中被自动恢复(约 0.5 秒,无需重启)、重试在首次刷新失败后仍自动显示最终结果。
+12 -2
View File
@@ -2,8 +2,8 @@
generated: true (请先修改 Gitea Wiki,禁止直接编辑本文件)
wiki_page: Business-Rules-and-Glossary
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Business-Rules-and-Glossary.-
wiki_revision: ff2abb673fe97e7eaa6934f76180ffdde701da8d
synchronized_at: 2026-09-10T16:35:18Z
wiki_revision: 4bd843d1f0134f5688407b7072d98333b40b6da1
synchronized_at: 2026-09-11T03:01:48Z
<!-- gitea-wiki-mirror:end -->
# 业务规则与术语
@@ -127,3 +127,13 @@ POST /lookup 接收 {surface,lemma?}。查词键单独 casefold/NFC/弯撇号转
- 恢复:声明与完成分属两个事务。进程在声明后退出时,重启把 processing 的章节与任务放回 pending 并保留尝试次数,不产生重复章节。
- 未决边界:Go+Python NLP 与全 Go 路线尚未确认。本单只做 Go 校验、分章与发布,不产生 token、lemma 或词典索引;正式接入前必须由用户确认路线,再定义生产 token 契约。
- 已知限制:本单未设置每账号书籍数量或总容量配额,只限制单次正文与请求体大小;删除书籍/章节属 #10,导入失败不会自动重试,只在启动时恢复被中断的 processing 任务。
## #5 审核整改(R1~R4,2026-09-11)
工单 #5 的整改记录见该工单评论;本节只记录长期有效的契约变化。
- 追加章节沿用所属书籍的语言:`POST /api/v1/books/:id/chapters` 的请求体只有 requestId、title、text,不接受 language;服务端仍拒绝未知字段,客户端发 language 会得到 400。新建书籍的 `POST /api/v1/books` 才带 language。学习端已按此拆分请求体,避免两个契约共用同一结构。
- 导入任务的自动重试有上限:同一任务被 worker 领取的次数达到 5 次后,任务与章节转为 failed,原因码 `attempts_exhausted`,提示“处理多次失败,请重试或重新提交”。人工重试(POST /jobs/:id/retry)会重置尝试次数,因此人工操作不受该上限阻塞。
- 运行期恢复不依赖重启:除了启动时的恢复,运行中的服务每次轮询都会把停留在 processing 且超过 15 秒的任务放回 pending,因此“领取已提交、完成事务失败”不会让章节永久卡在处理中。该阈值必须长于正常的领取到完成窗口;重复处理同一任务不会产生第二个章节,因为任务从不创建章节。
- 学习端离开页面时作废在途请求:目录页与阅读页在关闭时推进各自的请求序号,晚到的响应不会写回状态或重启轮询;导入页在提交过程中离开后,晚到的成功响应不会把用户导航回书籍页。
- 重试结果立即生效:重试被接受后先把返回的 pending 状态写入界面并继续轮询,因此紧随其后的一次刷新失败不会让页面停在处理失败。
+25 -2
View File
@@ -2,8 +2,8 @@
generated: true (请先修改 Gitea Wiki,禁止直接编辑本文件)
wiki_page: Local-Development-and-Verification
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Local-Development-and-Verification.-
wiki_revision: b8741ac70a524bb79741183f592f19c1206cd71b
synchronized_at: 2026-09-10T16:35:18Z
wiki_revision: 3bce5fae964b1050513c37319262679814d68f14
synchronized_at: 2026-09-11T03:01:49Z
<!-- gitea-wiki-mirror:end -->
# 本地开发与验证
@@ -282,3 +282,26 @@ node --test spikes/english/view.test.mjs
截图保存在本机 .local/evidence/(library.png、book-after-paste.png、reader-desktop.png、reader-mobile-390.png)并作为工单附件上传,便于人工目视复核;本次会话模型不能读取图片,截图未经 Agent 目视检查,功能断言来自上面的程序化检查。
未验证:处理失败到重试的用户界面路径只由集成测试覆盖(无法通过 API 主动制造处理失败);真实手机浏览器长按、手柄、滚动与虚拟键盘仍属 #4 缺口,本次只用桌面浏览器窄屏检查,不能当作真机结果;Python NLP 未接入,token、lemma 与词典仍为 #3 小样范围;生产并发、容量、备份恢复与部署不在本单范围。
## #5 审核整改验证(R1~R4,2026-09-11)
整改提交与完整证据见工单 #5 的整改评论。本次复核命令与结果(仓库根执行,专用测试库 lexgo_test_issue5):
| 命令 | 本次结果 |
|---|---|
| `python scripts/server.py test-integration` | 20 个顶层用例全部通过(新增 `TestMySQLIngestRecoveryWithoutRestart`、`TestMySQLIngestAttemptsAreBoundedAndManualRetryRestarts`) |
| `npx --yes pnpm@9.15.1 --dir learner test:unit --run` | 38 项通过(library 21、reading 8、session 9) |
| `npx --yes pnpm@9.15.1 --dir learner build` | 通过(vue-tsc + vite) |
| `npx --yes pnpm@9.15.1 --dir learner test:e2e` | 3 项通过(虚构 API 响应) |
回归测试的有效性:新增的前端 7 项用例先在整改前的 `library.ts`/`ImportView.vue` 上运行并实际失败(追加发送 language、离页后响应写回、重试后停在失败),改回修复版本后全部通过。
真实联调(lexgo_dev,虚构账号 issue5_a):
- R1 追加:真实学习端从书籍页进入“追加章节”,提交后回到书籍页,新章节就绪后可阅读;抓取到的请求体只有 requestId、title、text,无 language;正文逐字符相等。
- R2 恢复:新建章节后用 SQL 把任务与章节置为 processing 且 updated_at 早于阈值(UTC 时间),**不重启服务**,运行期清扫在 544 ms 内把任务重新入队并发布为就绪,章节与任务编号不变,正文逐字符相等。
- R4 重试:SQL 制造真实失败任务(content_changed)后,在浏览器点击“重试”并中断其后的第一次刷新请求,页面立即由“处理失败”变为“处理中”,随后自行变为“已就绪”,无需手工刷新。
注意:MySQL 会话时区为 SYSTEM(本机为 UTC+8),而服务按 UTC 存储 DATETIME;核对任务时间时使用 UTC_TIMESTAMP 而不是 NOW(),否则会出现 8 小时的假偏差。
未在本轮验证:R3 的真实浏览器时序(离页与响应同时发生)只由单测覆盖;真机手机证据仍属 #4 缺口。
+102
View File
@@ -340,4 +340,106 @@ describe('learner library store', () => {
expect(library.books).toHaveLength(1)
expect(library.books[0]?.title).toBe('较新的标题')
})
// Regression R1: the append contract has no language field and the server rejects unknown
// fields, so a client that sent one could never append.
it('sends the language only when creating a book, never when appending', async () => {
await signIn()
const library = useLibraryStore()
fetchMock().mockResolvedValueOnce(created({ book, chapter: chapter({ id: 54 }) }))
await library.submit({ title: ' 新书 ', text: 'New book text.', target: { mode: 'new' } })
expect(bodyOf(1)).toEqual({ requestId: expect.any(String), title: '新书', text: 'New book text.', language: 'en' })
fetchMock().mockResolvedValueOnce(created({ chapter: chapter({ id: 55, ordinal: 2 }) }))
await library.submit({ title: '第二篇', text: 'Appended text.', target: { mode: 'append', bookId: 1 } })
expect(String(fetchMock().mock.calls[2]?.[0])).toBe('/api/v1/books/1/chapters')
expect(bodyOf(2)).toEqual({ requestId: expect.any(String), title: '第二篇', text: 'Appended text.' })
expect(bodyOf(2)).not.toHaveProperty('language')
})
// Regression R3: leaving a view must invalidate its in-flight request.
it('ignores a book response that arrives after the book view was closed', async () => {
vi.useFakeTimers()
await signIn()
const library = useLibraryStore()
let finish!: (response: Response) => void
fetchMock().mockImplementationOnce(() => new Promise<Response>(resolve => { finish = resolve }))
const pending = library.loadBook(1)
library.closeBook()
finish(ok({ book, chapters: [chapter()] }))
await pending
expect(library.book).toBeNull()
expect(library.chapters).toEqual([])
expect(library.bookLoading).toBe(false)
const settled = paths().length
await vi.advanceTimersByTimeAsync(POLL_INTERVAL_MS * 3)
expect(paths().length).toBe(settled)
})
it('ignores a chapter response that arrives after the reader was closed', async () => {
vi.useFakeTimers()
await signIn()
const library = useLibraryStore()
let finish!: (response: Response) => void
fetchMock().mockImplementationOnce(() => new Promise<Response>(resolve => { finish = resolve }))
const pending = library.loadChapter(55)
library.closeChapter()
finish(ok({ book, chapter: chapter({ status: 'ready', originalText: 'Late text.' }), navigation }))
await pending
expect(library.chapter).toBeNull()
expect(library.readerText).toBe('')
expect(library.chapterLoading).toBe(false)
const settled = paths().length
await vi.advanceTimersByTimeAsync(POLL_INTERVAL_MS * 3)
expect(paths().length).toBe(settled)
})
// Regression R4: an accepted retry must be visible and tracked even if the refresh fails.
it('keeps tracking a retried chapter when the first refresh fails', async () => {
vi.useFakeTimers()
await signIn()
const library = useLibraryStore()
fetchMock().mockResolvedValueOnce(ok({ book, chapter: chapter({ status: 'failed', errorReason: 'content_changed', errorMessage: '内容在处理前发生变化。', jobId: 7 }), navigation }))
await library.loadChapter(55)
expect(library.chapter?.status).toBe('failed')
fetchMock()
.mockResolvedValueOnce(ok({ job: { ...job, status: 'pending' }, chapter: chapter({ status: 'pending' }) }))
.mockRejectedValueOnce(new Error('network down'))
await library.retryChapter(55)
expect(paths()).toContain('/api/v1/jobs/7/retry')
expect(library.chapter?.status).toBe('pending')
expect(library.readerText).toBe('')
// The next poll still tracks the queued chapter and shows the final result.
fetchMock().mockResolvedValueOnce(ok({ book, chapter: chapter({ status: 'ready', originalText: 'Recovered text.' }), navigation }))
await vi.advanceTimersByTimeAsync(POLL_INTERVAL_MS)
expect(library.chapter?.status).toBe('ready')
expect(library.readerText).toBe('Recovered text.')
})
it('keeps tracking a retried chapter from the book page when the first refresh fails', async () => {
vi.useFakeTimers()
await signIn()
const library = useLibraryStore()
fetchMock().mockResolvedValueOnce(ok({ book, chapters: [chapter({ status: 'failed', errorReason: 'content_changed', errorMessage: '内容在处理前发生变化。' })] }))
await library.loadBook(1)
expect(library.chapters[0]?.status).toBe('failed')
fetchMock()
.mockResolvedValueOnce(ok({ job: { ...job, status: 'pending' }, chapter: chapter({ status: 'pending' }) }))
.mockRejectedValueOnce(new Error('network down'))
await library.retryChapter(55)
expect(library.chapters[0]?.status).toBe('pending')
fetchMock().mockResolvedValueOnce(ok({ book, chapters: [chapter({ status: 'ready' })] }))
await vi.advanceTimersByTimeAsync(POLL_INTERVAL_MS)
expect(library.chapters[0]?.status).toBe('ready')
})
})
+62
View File
@@ -200,4 +200,66 @@ describe('learner reading views', () => {
expect(retryCall?.[1]?.method).toBe('POST')
wrapper.unmount()
})
// Regression R1: appending must not send the language field that the append contract rejects.
it('appends to an existing book through the chapter endpoint without a language field', async () => {
const fetchMock = vi.spyOn(globalThis, 'fetch')
.mockResolvedValueOnce(ok({
items: [{
...book,
chapterCount: 1,
pendingCount: 0,
processingCount: 0,
readyCount: 1,
failedCount: 0,
createdAt: '2026-01-01T00:00:00Z',
updatedAt: '2026-01-01T00:00:00Z',
}],
}))
.mockResolvedValueOnce(created({ chapter: chapter('pending', { id: 56, ordinal: 2 }), job: { id: 8 } }))
signIn()
const router = await viewAt('/import?book=1')
const wrapper = mount(ImportView, { global: { plugins: [router] } })
await flushPromises()
await wrapper.find('input#title').setValue('第二篇')
await wrapper.find('textarea#text').setValue(pasted)
await wrapper.find('form').trigger('submit')
await flushPromises()
const appendCall = fetchMock.mock.calls.find(([input]) => String(input).endsWith('/books/1/chapters'))
expect(appendCall?.[1]?.method).toBe('POST')
const body = JSON.parse(String(appendCall?.[1]?.body)) as Record<string, unknown>
expect(body).toEqual({ requestId: expect.any(String), title: '第二篇', text: pasted })
expect(body).not.toHaveProperty('language')
expect(router.currentRoute.value.path).toBe('/books/1')
wrapper.unmount()
})
// Regression R3: a submit that finishes after the user left the page must not navigate back.
it('does not navigate after the user left the import page during a submit', async () => {
let finish!: (response: Response) => void
const fetchMock = vi.spyOn(globalThis, 'fetch')
.mockImplementationOnce(() => new Promise<Response>(resolve => { finish = resolve }))
signIn()
const router = await viewAt('/import')
const wrapper = mount(ImportView, { global: { plugins: [router] } })
await flushPromises()
await wrapper.find('input#title').setValue('虚构样例第一章')
await wrapper.find('textarea#text').setValue(pasted)
await wrapper.find('form').trigger('submit')
await flushPromises()
expect(fetchMock).toHaveBeenCalledTimes(1)
// The user leaves the page while the request is still open.
wrapper.unmount()
await router.push('/')
await flushPromises()
finish(created({ book, chapter: chapter('pending'), job: { id: 7 } }))
await flushPromises()
expect(router.currentRoute.value.path).toBe('/')
})
})
+39 -10
View File
@@ -108,7 +108,10 @@ export function statusSummary(book: BookSummary): string {
}
interface LoadOptions { silent?: boolean }
interface SubmitBody { requestId: string; title: string; text: string; language: 'en' }
interface SubmitBookBody { requestId: string; title: string; text: string; language: 'en' }
// An appended chapter owns the language of its book, so the append contract has no language
// field; the server rejects unknown fields, and a client that sends one gets HTTP 400.
interface SubmitChapterBody { requestId: string; title: string; text: string }
interface Created { bookId: number; chapter: ChapterSummary }
function emptyNavigation(): ChapterNavigation {
@@ -310,12 +313,12 @@ export const useLibraryStore = defineStore('library', () => {
return target.mode === 'new' ? `new\n${title}\n${text}` : `append:${target.bookId}\n${title}\n${text}`
}
async function createBook(body: SubmitBody): Promise<Created> {
async function createBook(body: SubmitBookBody): Promise<Created> {
const result = await session.request<{ book: BookRef; chapter: ChapterSummary }>('books', 'POST', body)
return { bookId: result.book.id, chapter: result.chapter }
}
async function appendChapter(bookId: number, body: SubmitBody): Promise<Created> {
async function appendChapter(bookId: number, body: SubmitChapterBody): Promise<Created> {
const result = await session.request<{ chapter: ChapterSummary }>(`books/${bookId}/chapters`, 'POST', body)
return { bookId: result.chapter.bookId, chapter: result.chapter }
}
@@ -338,16 +341,17 @@ export const useLibraryStore = defineStore('library', () => {
submissionKey = key
submissionRequestId = crypto.randomUUID()
}
const body: SubmitBody = { requestId: submissionRequestId, title, text, language: LANGUAGE_CODE }
const requestId = submissionRequestId
const version = generation
const owner = ownerId()
submitting.value = true
submitError.value = ''
try {
// Only the new-book contract carries a language; appending inherits the book's language.
const created = input.target.mode === 'new'
? await createBook(body)
: await appendChapter(input.target.bookId, body)
? await createBook({ requestId, title, text, language: LANGUAGE_CODE })
: await appendChapter(input.target.bookId, { requestId, title, text })
if (isStale(version, owner)) throw new Error('登录状态已变化,请重新提交。')
// The content was accepted; a later submit must use a fresh requestId.
submissionKey = ''
@@ -368,6 +372,19 @@ export const useLibraryStore = defineStore('library', () => {
return target?.jobId ?? null
}
/**
* Applies a chapter summary coming from any response to the chapter list entry and to the
* open reader, so a queued chapter is never displayed with the state or text it had before.
*/
function applyChapterSummary(summary: ChapterSummary): void {
const index = chapters.value.findIndex(item => item.id === summary.id)
if (index >= 0) chapters.value[index] = { ...chapters.value[index], ...summary }
if (chapter.value !== null && chapter.value.id === summary.id) {
const originalText = summary.status === 'ready' ? chapter.value.originalText : undefined
chapter.value = { ...chapter.value, ...summary, originalText }
}
}
async function retryChapter(chapterId: number): Promise<void> {
const jobId = jobIdOf(chapterId)
if (jobId === null) throw new Error('这一章暂时没有可重试的任务编号。')
@@ -375,9 +392,13 @@ export const useLibraryStore = defineStore('library', () => {
const owner = ownerId()
retryingChapterId.value = chapterId
try {
const result = await session.request<{ job: Job; chapter: Pick<ChapterSummary, 'id' | 'bookId'> }>(`jobs/${jobId}/retry`, 'POST')
const result = await session.request<{ job: Job; chapter: ChapterSummary }>(`jobs/${jobId}/retry`, 'POST')
if (isStale(version, owner)) return
// The retry answer only carries a summary, so refresh whatever is on screen.
// The retry is accepted, so show the queued chapter and keep tracking it even if the
// refresh below fails: a failed silent refresh must not freeze the view on the old error.
applyChapterSummary(result.chapter)
schedulePolling()
// Refresh whatever is on screen to pick up the newest job state.
if (book.value !== null && book.value.id === result.chapter.bookId) await loadBook(result.chapter.bookId, { silent: true })
if (chapter.value !== null && chapter.value.id === result.chapter.id) await loadChapter(result.chapter.id, { silent: true })
} catch (reason) {
@@ -388,19 +409,27 @@ export const useLibraryStore = defineStore('library', () => {
}
}
/** Releases the book view so polling stops when the page is left. */
/**
* Releases the book view so polling stops when the page is left. The request sequence is
* advanced first, so a response that arrives after this call cannot repopulate the view or
* restart polling for a page the user already left.
*/
function closeBook(): void {
bookSeq++
book.value = null
chapters.value = []
bookLoading.value = false
bookError.value = ''
schedulePolling()
}
/** Releases the reader view so polling stops when the page is left. */
/** Releases the reader view, invalidating in-flight loads the same way. */
function closeChapter(): void {
chapterSeq++
chapter.value = null
chapterBook.value = null
navigation.value = emptyNavigation()
chapterLoading.value = false
chapterError.value = ''
schedulePolling()
}
+8 -1
View File
@@ -55,6 +55,8 @@ async function submit() {
: { mode: 'new' }
try {
const createdBookId = await library.submit({ title: title.value, text: text.value, target })
// A response that arrives after the user left this page must not navigate them back.
if (disposed) return
// The requestId was consumed by this submission, so the form starts clean.
title.value = ''
text.value = ''
@@ -64,7 +66,12 @@ async function submit() {
}
}
onUnmounted(() => { library.submitError = '' })
// The form may still be awaiting its submit when the user navigates away.
let disposed = false
onUnmounted(() => {
disposed = true
library.submitError = ''
})
</script>
<template>
+81 -15
View File
@@ -15,38 +15,104 @@ import (
// The ingestion worker is deliberately small: the fixed paste rule stores one chapter per
// submit, so "processing" only validates the persisted content and publishes the chapter.
// Claiming and finishing are separate transactions on purpose. A durable claim means a
// chapter observed as processing stays recoverable, and a crash between the two steps can
// only leave rows that RecoverIngestJobs returns to the queue at the next startup.
// chapter observed as processing stays recoverable, whether the process stops or only the
// finishing transaction fails.
// RecoverIngestJobs requeues jobs and chapters left in processing by an unclean stop. It is
// meant for a single-instance deployment and runs once before the worker starts.
// Recovery is one mechanism used from two places: the startup pass treats every processing row
// as abandoned, while the running worker sweeps rows that have been processing longer than any
// legitimate claim-to-finish window. A job that keeps failing ends in a readable failure
// instead of looping forever, and a retried job never creates a second chapter.
const (
maxIngestAttempts = 5
// ingestStaleAfter must stay longer than the longest legitimate claim-to-finish window,
// otherwise a healthy job could be processed twice. Reprocessing is harmless for content
// because a job never creates a chapter, only publishes the one it was created with.
ingestStaleAfter = 15 * time.Second
)
// RecoverIngestJobs requeues jobs and chapters left in processing by an unclean stop. It runs
// once before the worker starts, for a single-instance deployment.
func RecoverIngestJobs(db *gorm.DB, now time.Time) (int64, error) {
return requeueStaleIngestJobs(db, now, 0)
}
// RequeueStaleIngestJobs recovers jobs whose finishing transaction did not complete, so a
// running service does not depend on a restart to make progress again.
func RequeueStaleIngestJobs(db *gorm.DB, now time.Time) (int64, error) {
return requeueStaleIngestJobs(db, now, ingestStaleAfter)
}
func requeueStaleIngestJobs(db *gorm.DB, now time.Time, staleAfter time.Duration) (int64, error) {
ts := stamp(now)
var jobs int64
cutoff := stamp(now.Add(-staleAfter))
var requeued int64
err := db.Transaction(func(tx *gorm.DB) error {
chapters := tx.Exec(`UPDATE lexgo_chapters c JOIN lexgo_ingest_jobs j ON j.chapter_id = c.id
SET c.status = ?, c.updated_at = ? WHERE j.status = ?`, statusPending, ts, statusProcessing)
if chapters.Error != nil {
return chapters.Error
if err := exhaustIngestJobs(tx, ts); err != nil {
return err
}
result := tx.Model(&IngestJob{}).Where("status = ?", statusProcessing).
stale := []int64{}
if err := tx.Model(&IngestJob{}).
Where("status = ? AND attempts < ? AND updated_at <= ?", statusProcessing, maxIngestAttempts, cutoff).
Pluck("id", &stale).Error; err != nil {
return err
}
if len(stale) == 0 {
return nil
}
if err := setIngestChapterStatus(tx, stale, statusPending, "", ts); err != nil {
return err
}
result := tx.Model(&IngestJob{}).Where("id IN ?", stale).
Updates(map[string]any{"status": statusPending, "updated_at": ts})
if result.Error != nil {
return result.Error
}
jobs = result.RowsAffected
requeued = result.RowsAffected
return nil
})
return jobs, err
return requeued, err
}
// ClaimNextIngestJob takes the oldest pending job and marks it processing in its own
// transaction. The guarded update means only one worker can own a job.
// exhaustIngestJobs fails jobs that used up the attempt budget, so nothing can stay queued or
// claimed forever. The recorded reason is readable and a manual retry is still allowed.
func exhaustIngestJobs(tx *gorm.DB, ts time.Time) error {
var exhausted []int64
if err := tx.Model(&IngestJob{}).
Where("status IN ? AND attempts >= ?", []string{statusPending, statusProcessing}, maxIngestAttempts).
Pluck("id", &exhausted).Error; err != nil {
return err
}
if len(exhausted) == 0 {
return nil
}
if err := setIngestChapterStatus(tx, exhausted, statusFailed, reasonAttemptsExhausted, ts); err != nil {
return err
}
return tx.Model(&IngestJob{}).Where("id IN ?", exhausted).Updates(map[string]any{
"status": statusFailed, "error_reason": reasonAttemptsExhausted, "updated_at": ts, "finished_at": ts}).Error
}
// setIngestChapterStatus mirrors a job outcome onto the chapters it owns.
func setIngestChapterStatus(tx *gorm.DB, jobIDs []int64, status, reason string, ts time.Time) error {
var chapterIDs []int64
if err := tx.Model(&IngestJob{}).Where("id IN ?", jobIDs).Pluck("chapter_id", &chapterIDs).Error; err != nil {
return err
}
if len(chapterIDs) == 0 {
return nil
}
return tx.Model(&Chapter{}).Where("id IN ?", chapterIDs).
Updates(map[string]any{"status": status, "error_reason": reason, "updated_at": ts}).Error
}
// ClaimNextIngestJob takes the oldest pending job with attempts left and marks it processing in
// its own transaction. The guarded update means only one worker can own a job.
func ClaimNextIngestJob(db *gorm.DB, now time.Time) (IngestJob, bool, error) {
ts := stamp(now)
var job IngestJob
err := db.Transaction(func(tx *gorm.DB) error {
if err := tx.Where("status = ?", statusPending).Order("id ASC").First(&job).Error; err != nil {
if err := tx.Where("status = ? AND attempts < ?", statusPending, maxIngestAttempts).
Order("id ASC").First(&job).Error; err != nil {
return err
}
claim := tx.Model(&IngestJob{}).Where("id = ? AND status = ?", job.ID, statusPending).
+9 -2
View File
@@ -30,6 +30,7 @@ const (
reasonTooLong = "too_long"
reasonEmptyText = "empty_text"
reasonContentChanged = "content_changed"
reasonAttemptsExhausted = "attempts_exhausted"
)
const (
@@ -54,6 +55,8 @@ func reasonMessage(reason string) string {
return "章节内容为空"
case reasonContentChanged:
return "内容在处理前发生变化,请重新提交"
case reasonAttemptsExhausted:
return "处理多次失败,请重试或重新提交"
default:
return ""
}
@@ -616,7 +619,8 @@ func JobDetail(db *gorm.DB, owner int, jobID int64) (JobView, error) {
}
// RetryIngestJob requeues a failed job on the same chapter, so a retry can never create a
// second chapter for one paste.
// second chapter for one paste. An explicit retry also restarts the attempt budget, because a
// person asking again should not be blocked by the bound that stops automatic loops.
func RetryIngestJob(db *gorm.DB, owner int, jobID int64, now time.Time) (JobView, ChapterSummary, error) {
ts := stamp(now)
var job IngestJob
@@ -637,7 +641,8 @@ func RetryIngestJob(db *gorm.DB, owner int, jobID int64, now time.Time) (JobView
return err
}
if err := tx.Model(&IngestJob{}).Where("id = ?", job.ID).
Updates(map[string]any{"status": statusPending, "error_reason": "", "updated_at": ts}).Error; err != nil {
Updates(map[string]any{"status": statusPending, "error_reason": "", "attempts": 0,
"updated_at": ts, "finished_at": nil}).Error; err != nil {
return err
}
if err := tx.Model(&Chapter{}).Where("id = ?", chapter.ID).
@@ -646,6 +651,8 @@ func RetryIngestJob(db *gorm.DB, owner int, jobID int64, now time.Time) (JobView
}
job.Status = statusPending
job.ErrorReason = ""
job.Attempts = 0
job.FinishedAt = nil
job.UpdatedAt = ts
chapter.Status = statusPending
chapter.ErrorReason = ""
+134 -2
View File
@@ -2,6 +2,7 @@ package lexgo
import (
"bytes"
"context"
"encoding/json"
"fmt"
"net/http/httptest"
@@ -511,8 +512,9 @@ func TestMySQLRetryAfterContentRestoredPublishesSameChapter(t *testing.T) {
t.Fatalf("retry did not publish the same chapter: %+v", ready.Chapter)
}
job := jobRow(t, db, pasted.Job.ID)
if job.Attempts != 2 {
t.Fatalf("attempts %d, want 2", job.Attempts)
// A manual retry restarts the attempt budget, so this processing is attempt 1 again.
if job.Attempts != 1 {
t.Fatalf("attempts %d after a manual retry, want 1", job.Attempts)
}
code, msg, _ = callRaw(t, r, "POST", fmt.Sprintf("/api/v1/jobs/%d/retry", pasted.Job.ID), learner.Token, nil)
if code != 409 || msg == "" {
@@ -611,6 +613,13 @@ func TestMySQLRepeatedPasteIsIdempotent(t *testing.T) {
if raceBook.ID == first.Book.ID {
t.Fatal("idempotency fixtures must use different books")
}
// Append owns the language of its book, so the field is not part of that contract and the
// strict decoder rejects it. The learner client must therefore not send it (regression R1).
code, msg, _ = callRaw(t, r, "POST", fmt.Sprintf("/api/v1/books/%d/chapters", first.Book.ID), learner.Token,
map[string]string{"requestId": randomName("append"), "title": "Strict", "text": "Strict contract.", "language": "en"})
if code != 400 || msg == "" {
t.Fatalf("append with language status %d (%s), want 400", code, msg)
}
}
func TestMySQLLibraryIsolationAndOwnership(t *testing.T) {
@@ -688,6 +697,129 @@ func TestMySQLLibraryIsolationAndOwnership(t *testing.T) {
}
}
func TestMySQLIngestRecoveryWithoutRestart(t *testing.T) {
db, r, owner := libraryFixture(t)
learner := newLearner(t, r, owner.Token)
drainIngest(t, db)
code, pasted := pasteBook(t, r, learner.Token, map[string]string{
"requestId": "fixture-in-service-recovery", "title": "In-service recovery", "text": fixturePastedText})
if code != 201 {
t.Fatalf("paste status %d", code)
}
job, claimed, err := ClaimNextIngestJob(db, time.Now())
if err != nil || !claimed || job.ID != pasted.Job.ID {
t.Fatalf("claim failed (claimed=%v, job=%d): %v", claimed, job.ID, err)
}
// The finishing transaction fails (context cancellation stands in for a timeout or a
// database error). The claim is already committed, so the job stays processing.
canceled, cancel := context.WithCancel(t.Context())
cancel()
if err = FinishIngestJob(canceled, db, job, time.Now()); err == nil {
t.Fatal("a canceled finishing transaction must report an error")
}
var stuck IngestJob
if err = db.Where("id = ?", pasted.Job.ID).First(&stuck).Error; err != nil {
t.Fatal(err)
}
if stuck.Status != statusProcessing {
t.Fatalf("job status %q after a failed finish, want processing", stuck.Status)
}
// A manual retry cannot rescue it: only failed jobs are accepted.
code, msg, _ := callRaw(t, r, "POST", fmt.Sprintf("/api/v1/jobs/%d/retry", pasted.Job.ID), learner.Token, nil)
if code != 409 || msg == "" {
t.Fatalf("retry of a processing job status %d (%s), want 409", code, msg)
}
// A sweep that is too early must leave a healthy claim alone.
claimedAt := stuck.UpdatedAt
if _, err = RequeueStaleIngestJobs(db, claimedAt.Add(time.Second)); err != nil {
t.Fatal(err)
}
if jobRow(t, db, pasted.Job.ID).Status != statusProcessing {
t.Fatal("a fresh claim must not be requeued")
}
// Once the claim is older than the stale window, the running service recovers it.
if _, err = RequeueStaleIngestJobs(db, claimedAt.Add(ingestStaleAfter+time.Second)); err != nil {
t.Fatal(err)
}
recovered := jobRow(t, db, pasted.Job.ID)
if recovered.Status != statusPending || recovered.Attempts != 1 || recovered.FinishedAt != nil {
t.Fatalf("recovered job %+v", recovered)
}
if chapterRow(t, db, pasted.Chapter.ID).Status != statusPending {
t.Fatal("recovered chapter must be pending")
}
// Recovery reuses the same rows: no second chapter, same task id.
var chapters int64
db.Model(&Chapter{}).Where("book_id = ?", pasted.Book.ID).Count(&chapters)
if chapters != 1 {
t.Fatalf("recovery left %d chapters, want 1", chapters)
}
drainIngest(t, db)
code, ready := readChapter(t, r, learner.Token, pasted.Chapter.ID)
if code != 200 || ready.Chapter.Status != statusReady || ready.Chapter.OriginalText != fixturePastedText {
t.Fatalf("recovered chapter %+v", ready.Chapter)
}
if ready.Chapter.ID != pasted.Chapter.ID || ready.Chapter.JobID == nil || *ready.Chapter.JobID != pasted.Job.ID {
t.Fatal("recovery must keep the original chapter and task ids")
}
}
func TestMySQLIngestAttemptsAreBoundedAndManualRetryRestarts(t *testing.T) {
db, r, owner := libraryFixture(t)
learner := newLearner(t, r, owner.Token)
drainIngest(t, db)
code, pasted := pasteBook(t, r, learner.Token, map[string]string{
"requestId": "fixture-attempt-budget", "title": "Attempt budget", "text": fixturePastedText})
if code != 201 {
t.Fatalf("paste status %d", code)
}
// Spend the whole budget without any worker running.
if err := db.Model(&IngestJob{}).Where("id = ?", pasted.Job.ID).Update("attempts", maxIngestAttempts).Error; err != nil {
t.Fatal(err)
}
if _, claimed, err := ClaimNextIngestJob(db, time.Now()); err != nil || claimed {
t.Fatalf("claim claimed=%v (%v), want no claim once the budget is used", claimed, err)
}
if _, err := RequeueStaleIngestJobs(db, time.Now()); err != nil {
t.Fatal(err)
}
exhausted := jobRow(t, db, pasted.Job.ID)
if exhausted.Status != statusFailed || exhausted.ErrorReason != reasonAttemptsExhausted {
t.Fatalf("exhausted job %+v", exhausted)
}
code, read := readChapter(t, r, learner.Token, pasted.Chapter.ID)
if code != 200 || read.Chapter.Status != statusFailed || read.Chapter.ErrorReason != reasonAttemptsExhausted {
t.Fatalf("exhausted chapter %+v", read.Chapter)
}
if read.Chapter.ErrorMessage == "" || read.Chapter.OriginalText != "" {
t.Fatalf("exhausted chapter must fail readably without text: %+v", read.Chapter)
}
// The manual retry is still available and restarts the attempt budget.
code, msg, _ := callRaw(t, r, "POST", fmt.Sprintf("/api/v1/jobs/%d/retry", pasted.Job.ID), learner.Token, nil)
if code != 200 {
t.Fatalf("manual retry status %d (%s), want 200", code, msg)
}
retried := jobRow(t, db, pasted.Job.ID)
if retried.Status != statusPending || retried.Attempts != 0 || retried.ErrorReason != "" {
t.Fatalf("retried job %+v", retried)
}
drainIngest(t, db)
code, ready := readChapter(t, r, learner.Token, pasted.Chapter.ID)
if code != 200 || ready.Chapter.Status != statusReady || ready.Chapter.ErrorReason != "" {
t.Fatalf("chapter after manual retry %+v", ready.Chapter)
}
if ready.Chapter.ID != pasted.Chapter.ID {
t.Fatal("manual retry must reuse the same chapter")
}
var chapters int64
db.Model(&Chapter{}).Where("book_id = ?", pasted.Book.ID).Count(&chapters)
if chapters != 1 {
t.Fatalf("attempt recovery left %d chapters, want 1", chapters)
}
}
func TestMySQLIngestRecoveryAfterRestart(t *testing.T) {
db, r, owner := libraryFixture(t)
learner := newLearner(t, r, owner.Token)
+12 -3
View File
@@ -112,8 +112,9 @@ func run() error {
srv := &http.Server{Addr: addr, Handler: lexgo.Router(db, time.Now), ReadHeaderTimeout: 5 * time.Second, ReadTimeout: 15 * time.Second, WriteTimeout: 15 * time.Second, IdleTimeout: 60 * time.Second, MaxHeaderBytes: 1 << 20}
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
// A single instance owns the worker; a restart returns chapters left in processing to
// the queue instead of losing them.
// A single instance owns the worker. Startup recovery returns chapters left in processing
// by an unclean stop, and the running loop sweeps jobs whose finishing transaction failed,
// so a transient database problem does not need a restart.
if _, err = lexgo.RecoverIngestJobs(db, time.Now()); err != nil {
return errors.New("ingestion recovery failed")
}
@@ -122,8 +123,16 @@ func run() error {
defer ticker.Stop()
for {
jobCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
requeued, recoverErr := lexgo.RequeueStaleIngestJobs(db, time.Now())
if recoverErr != nil {
if ctx.Err() == nil {
log.Print("ingestion recovery failed; the next sweep retries it")
}
} else if requeued > 0 && ctx.Err() == nil {
log.Printf("requeued %d interrupted ingestion job(s)", requeued)
}
if _, err := lexgo.ProcessIngestJobs(jobCtx, db, time.Now, 20); err != nil && ctx.Err() == nil {
log.Print("ingestion batch failed; retrying next second")
log.Print("ingestion batch stopped before finishing; the claimed job stays processing until the next sweep requeues it")
}
cancel()
select {