Compare commits

...
Author SHA1 Message Date
ila a5a8c3c72d docs: 记录 #7 已确认的个人词条决策 (#7) 2026-09-11 16:34:54 +08:00
ila 8c0946af9f feat: 保存个人词义与状态,并在其他章节同步显示 (#7)
- schema v5 新增 lexgo_terms:身份为学习者+语言+规范化词形,唯一键保证
  重复保存只更新同一条记录,不产生冲突副本
- POST /api/v1/terms 幂等保存并返回 created,GET /api/v1/terms/:id 仅本人可读,
  章节 tokens 为 word 片段附带 term:{id,status,level}
- 状态与等级边界:新词/学习中/已知/忽略,只有学习中带 1~7 级,其余必须为 0,
  并由数据库检查约束守住
- 学习端面板可编辑释义、例句与学习状态,正文按状态高亮;打开已保存词先读取原内容,
  读取失败时禁用保存,切换账号或退出后清理表单、状态与高亮
- 同步 Architecture-and-Code-Map、Business-Rules-and-Glossary、
  Local-Development-and-Verification、Product-Requirements-Overview 与 Home
2026-09-11 16:32:04 +08:00
ila 33182e584b docs: record issue 6 acceptance and merge (#6) 2026-09-11 14:47:22 +08:00
ila ad31ea4225 feat: Go-only WordNet resources and reader lookup (#6) 2026-09-11 11:59:51 +08:00
ila 8310438bee docs: confirm Go-only NLP implementation scope (#6) 2026-09-11 11:51:18 +08:00
34 changed files with 3041 additions and 28 deletions
+2
View File
@@ -280,3 +280,5 @@ MVP 内所有单元任务通过后才能做 MVP 集成验收;MVP 通过后才
- #18 登录日志与操作审计已通过用户验收:schema v2 显式迁移;日志只保存白名单字段,禁止保存凭据、请求/响应正文及私人学习内容。仅管理员查询,默认保留 90 天;启动/每小时及 `python scripts/server.py audit-cleanup` 仅清理两张审计表的过期记录。
- #3 独立小样位于 `spikes/english/`,使用 `.local/nlp-venv/Scripts/python.exe`(3.12.12)运行;固定 spaCy 3.8.7、英语模型 3.8.0、NLTK 3.9.2、WordNet 3.0。资源仅显式准备时下载,摘要见 resources.json。不得把本机无账号的实验接口用于正式学习端;后续集成仍需 Go 授权、数据归属和任务设计。原文不归一化,位置区分 cp/UTF-8/UTF-16,lemma 不自动合并学习状态。
- #4 独立阅读选择小样位于 `spikes/selection/`,`python spikes/selection/serve.py` 默认仅本机 5184。桌面鼠标/键盘与 11 项测试已验证,#4 已获用户验收并关闭;真实手机长按/手柄/滚动详细证据仍未提供;禁止把窄屏桌面当作真机验收。Intl.Segmenter 只用于 UI 范围验证,不替代 #3 NLP;释义保存只在内存。固定 LinguaCafe 源码对照和与 v1 的差异记录见架构 Wiki。
- 2026-09-11 用户确认正式 NLP/词典采用全 Go。#5 已验收并合入 main;#6 使用 Go WordNet 解析和词形候选、Go Unicode 原文分片、schema v4 共享词典资源表,不调用 Python NLP。WordNet 3.0 ZIP 来源与摘要见 `server/wordnet-resource.json`,许可保留在 `server/WORDNET-LICENSE.txt`。词形候选不等于上下文消歧,不自动合并个人学习状态;#3 Python 小样只保留历史验证。当前词典仅英语释义,个人释义输入为临时草稿,持久化归 #7。
- 2026-09-11 用户确认 #7 个人词条口径(三项由 Agent 定案):身份为「学习者+语言+规范化词形」,大小写合并但**不按 lemma/候选合并**(`dog` 与 `dogs` 是两条记录);首次保存默认「新词」;状态为 新词/学习中/已知/忽略,只有「学习中」带 1~7 级,对应原版 stage 2/1/0/-1~-7;例句只保存手输内容,不自动关联原文句子。schema v5 新增 `lexgo_terms`(唯一键加状态/等级检查约束),个人释义与共享词典分离且不进入审计日志;等级编辑 UI 归 #8/#12。
+11 -3
View File
@@ -2,13 +2,13 @@
面向自托管场景的阅读式语言学习项目,规划提供内容导入、阅读查词、词汇与短语、复习和实例管理。
已确认:**DevHarness 轻量模式、MySQL 8、go-admin 管理端**。工程基础 #2 已通过验收:两端用户名登录、学习账号管理、可撤销会话和本人英语空空间。管理端基于指定 go-admin/go-admin-ui 选用模块,学习端为独立 Vue 3 + TypeScript + Vite 工程,共用 Go 后端和 MySQL 8.4.3。#18 登录日志与操作审计已通过用户验收,支持管理员查询和 90 天保留清理。#5 粘贴导入与章节原文阅读已实现,待用户验收;点词查词、词典、词语状态与复习仍未实现。MVP 定位为“支持多账号、数据独立的自托管学习工具”,先邀请少量用户使用;F01~F12 已确认,X 系列后置。
已确认:**DevHarness 轻量模式、MySQL 8、go-admin 管理端**。工程基础 #2 已通过验收:两端用户名登录、学习账号管理、可撤销会话和本人英语空空间。管理端基于指定 go-admin/go-admin-ui 选用模块,学习端为独立 Vue 3 + TypeScript + Vite 工程,共用 Go 后端和 MySQL 8.4.3。#18 登录日志与操作审计已通过用户验收,支持管理员查询和 90 天保留清理。#5 粘贴导入与章节原文阅读已验收并合入 main;#6 全 Go 英语词典与点词查义已通过用户验收,PR #25 已合入 main。词语状态与复习仍未实现。MVP 定位为“支持多账号、数据独立的自托管学习工具”,先邀请少量用户使用;F01~F12 已确认,X 系列后置。
- [文档入口](docs/README.md) · [线上 Wiki](https://git.ilapage.cn/OPC/lexgo/wiki/Home)
- [英语分词与离线词典验证小样](spikes/english/README.md)(#3 已验收,独立本机入口)
- [阅读选择验证小样](spikes/selection/README.md)(#4 已验收,真机详细测试证据缺口保留)
- [项目档案](docs/00-project-profile.md) · [需求总览](docs/09-product-requirements-overview.md)
- [工作量估算](docs/10-workload-estimate.md):#2、#3、#4、#18 已验收,剩余 #5~#15 与新增 #21 规划参考 44~71 人日;后续结合集成结果重估,旧全量研究仅供参考。
- [工作量估算](docs/10-workload-estimate.md):#2、#3、#4、#5、#6、#18 已验收,原规划中的 #5 已完成;剩余 #7~#15 与新增 #21、#24 按工单复核;后续结合集成结果重估,旧全量研究仅供参考。
- [四阶段实施总览 #16](https://git.ilapage.cn/OPC/lexgo/issues/16):14 张单元工单,工程基础 → 技术验证 → 首条学习闭环 → 补齐 MVP;原型 v1 已获用户验收。两端使用账号(用户名)+密码登录,不要求邮箱。
- [原型工单 #1](https://git.ilapage.cn/OPC/lexgo/issues/1):Quant-UX 桌面/手机原型 v1,预览入口与审核记录见工单及需求总览。
- [工作流](docs/01-workflow.md) · [开发与验证](docs/04-local-development-and-verification.md)
@@ -19,7 +19,15 @@
本地入口:学习端 http://127.0.0.1:5173,管理端 http://127.0.0.1:5174。完整安装与测试命令见[开发与验证](docs/04-local-development-and-verification.md)。账号使用用户名,无需邮箱;没有随代码交付的默认密码。
本机三个服务已由 `D:/supervisord/supervisord.conf` 中的 lexgo-learner、lexgo-admin、lexgo-api 托管,使用时不要重复手动启动同端口。当前 schema v3;从 #2/#18 升级时停止 API,执行 build、migrate 后再启动。审计过期清理可执行 `python scripts/server.py audit-cleanup`,只影响超过 90 天的日志。
本机三个服务已由 `D:/supervisord/supervisord.conf` 中的 lexgo-learner、lexgo-admin、lexgo-api 托管,使用时不要重复手动启动同端口。当前 schema v4;从 #2/#18 升级时停止 API,执行 build、migrate 后再启动。审计过期清理可执行 `python scripts/server.py audit-cleanup`,只影响超过 90 天的日志。
## 英语词典与点词查义(#6)
正式后端为纯 Go,`lexgo.exe serve` 不启动 Python NLP。开发辅助脚本与历史小样保留。
管理员在“英语词典”页下载并导入指定 WordNet 3.0 ZIP;来源、固定摘要及许可见 [资源清单](server/wordnet-resource.json)。词典提供英语释义,保存在 MySQL 中;导入失败保留当前资源,重复导入不新增资源并重新启用。也可在管理页停用。
学习者打开本人就绪章节,点击词语或聚焦后按 Enter/空格查询;Escape/关闭返回阅读。精确词形优先,未命中再查规则候选,例如 `went → go`。个人释义当前是未保存的临时草稿,保存功能在 #7 实现。真机测试证据尚未补齐。
## 文档与治理
+47
View File
@@ -0,0 +1,47 @@
export const MAX_DICTIONARY_BYTES = 32 * 1024 * 1024
export function createDictionaryLoader(session, state) {
let revision = 0
function invalidate() {
revision++
Object.assign(state, { items: [], supported: null, loading: false, saving: false, error: '', notice: '' })
}
async function perform(operation, saving, apply) {
const current = ++revision
const generation = session.state.generation
const stale = () => current !== revision || generation !== session.state.generation
Object.assign(state, { loading: !saving, saving, error: '', notice: '' })
try {
const result = await operation()
if (stale()) return false
apply(result)
return true
} catch (error) {
if (!stale()) state.error = error instanceof Error ? error.message : '词典操作失败,请重试'
return false
} finally {
if (!stale()) { state.loading = false; state.saving = false }
}
}
return {
invalidate,
load() {
return perform(() => session.listDictionaries(), false, result => {
state.items = result.items
state.supported = result.supported
})
},
import(form) {
return perform(() => session.importDictionary(form), true, result => {
state.items = [result.resource]
state.notice = result.duplicate ? '该词典已存在,已启用,未重复导入。' : '词典导入成功。'
})
},
toggle(id, enabled) {
return perform(() => session.setDictionaryEnabled(id, enabled), true, result => {
state.items = state.items.map(item => item.id === result.resource.id ? result.resource : item)
state.notice = result.resource.enabled ? '词典已启用。' : '词典已停用。'
})
}
}
}
+1
View File
@@ -5,6 +5,7 @@
<div class="brand">{{ sidebar.opened ? 'LexGo 管理' : 'LG' }}</div>
<el-menu :default-active="$route.path" :collapse="!sidebar.opened" router>
<el-menu-item index="/accounts"><span>账号管理</span></el-menu-item>
<el-menu-item index="/dictionaries"><span>英语词典</span></el-menu-item>
<el-menu-item index="/login-logs"><span>登录日志</span></el-menu-item>
<el-menu-item index="/operation-logs"><span>操作日志</span></el-menu-item>
</el-menu>
+2
View File
@@ -4,11 +4,13 @@ import Layout from '../layout/index.vue'
import Login from '../views/Login.vue'
import Accounts from '../views/Accounts.vue'
import AuditLogs from '../views/AuditLogs.vue'
import Dictionaries from '../views/Dictionaries.vue'
const router = createRouter({ history: createWebHashHistory(), routes: [
{ path: '/login', component: Login },
{ path: '/', component: Layout, children: [
{ path: '', redirect: '/accounts' },
{ path: 'accounts', component: Accounts, meta: { title: '账号管理' } },
{ path: 'dictionaries', component: Dictionaries, meta: { title: '英语词典' } },
{ path: 'login-logs', component: AuditLogs, props: { kind: 'login' }, meta: { title: '登录日志' } },
{ path: 'operation-logs', component: AuditLogs, props: { kind: 'operation' }, meta: { title: '操作日志' } }
] },
+17 -2
View File
@@ -20,10 +20,11 @@ export function createSession({ fetch, storage, changed = () => {} }) {
if (generation !== state.generation) throw new Error('会话已变化,请重新操作')
}
async function request(path, method = 'GET', body, token = state.token, generation = state.generation) {
const multipart = typeof FormData !== 'undefined' && body instanceof FormData
const result = await fetch('/api/v1' + path, {
method,
headers: { 'Content-Type': 'application/json', ...(token ? { Authorization: 'Bearer ' + token } : {}) },
...(body === undefined ? {} : { body: JSON.stringify(body) }),
headers: { ...(!multipart ? { 'Content-Type': 'application/json' } : {}), ...(token ? { Authorization: 'Bearer ' + token } : {}) },
...(body === undefined ? {} : { body: multipart ? body : JSON.stringify(body) }),
cache: 'no-store'
})
const payload = await result.json()
@@ -84,6 +85,20 @@ export function createSession({ fetch, storage, changed = () => {} }) {
} catch (error) { if (generation === state.generation) clear(); throw error }
},
async logout() { const token = state.token; clear(); await revoke(token) },
async listDictionaries() {
authorized()
return request('/dictionaries')
},
async importDictionary(form) {
authorized()
if (!(form instanceof FormData)) throw new Error('请选择词典文件')
return request('/dictionaries/import', 'POST', form)
},
async setDictionaryEnabled(id, enabled) {
authorized()
if (!Number.isSafeInteger(id) || id <= 0 || typeof enabled !== 'boolean') throw new Error('词典参数无效')
return request('/dictionaries/' + id, 'PATCH', { enabled })
},
async queryAuditLogs(kind, filters) {
authorized()
const generation = state.generation
+101
View File
@@ -0,0 +1,101 @@
<template>
<basic-layout><template #wrapper>
<el-card>
<div class="toolbar"><div><h1>英语词典</h1><p class="subtle">供所有学习账号查询</p></div><el-button :disabled="busy" @click="reload">刷新</el-button></div>
<el-alert v-if="resources.error" :title="resources.error" type="error" :closable="false" show-icon />
<el-alert v-if="resources.notice" :title="resources.notice" type="success" :closable="false" show-icon />
<el-table v-loading="resources.loading" :data="resources.items" border :empty-text="resources.error ? '加载失败,请重试' : '尚未导入词典'">
<el-table-column label="名称" prop="name" min-width="175" />
<el-table-column label="语言" width="80"><template #default>英语</template></el-table-column>
<el-table-column label="版本" prop="version" width="80" />
<el-table-column label="来源" prop="source" min-width="160" show-overflow-tooltip />
<el-table-column label="格式" prop="format" min-width="160" />
<el-table-column label="状态" width="95"><template #default="scope"><el-tag :type="scope.row.status === 'ready' ? 'success' : 'info'">{{ statusLabel(scope.row.status) }}</el-tag></template></el-table-column>
<el-table-column label="词条数" prop="entryCount" min-width="95" />
<el-table-column label="操作" width="100"><template #default="scope"><el-button :disabled="busy" link type="primary" @click="toggle(scope.row)">{{ scope.row.enabled ? '停用' : '启用' }}</el-button></template></el-table-column>
</el-table>
</el-card>
<el-card class="import-card">
<h2>导入词典</h2>
<el-form class="dictionary-form" label-position="top" @submit.prevent="upload">
<el-form-item label="名称"><el-input v-model="form.name" maxlength="120" :disabled="busy" /></el-form-item>
<div class="form-row">
<el-form-item label="语言"><el-input model-value="英语" disabled /></el-form-item>
<el-form-item label="版本"><el-input v-model="form.version" readonly /></el-form-item>
</div>
<el-form-item label="来源"><el-input v-model="form.source" readonly /></el-form-item>
<el-form-item label="格式"><el-input v-model="form.format" readonly /></el-form-item>
<el-form-item label="词典文件">
<input id="dictionary-file" ref="fileInput" type="file" accept=".zip,application/zip" aria-label="词典文件" :disabled="busy" @change="selectFile">
</el-form-item>
<p class="subtle">WordNet 3.0 ZIP · 上限 32 MiB · 英语释义</p>
<p v-if="resources.supported?.source"><a :href="downloadURL" target="_blank" rel="noopener noreferrer">下载支持的词典文件</a></p>
<p class="subtle">导入失败时保留当前词典。</p>
<p v-if="fileError" role="alert" class="file-error">{{ fileError }}</p>
<el-button type="primary" native-type="submit" :loading="resources.saving" :disabled="resources.loading || !file">导入并启用</el-button>
</el-form>
</el-card>
</template></basic-layout>
</template>
<script>
import BasicLayout from '../layout/BasicLayout.vue'
import { session } from '../store'
import { createDictionaryLoader, MAX_DICTIONARY_BYTES } from '../dictionaries.mjs'
const downloadURL = 'https://raw.githubusercontent.com/nltk/nltk_data/96f9b3252457a2b97e52aec64c3dfceeb5c312d5/packages/corpora/wordnet.zip'
export default {
name: 'DictionariesView',
components: { BasicLayout },
data: () => ({
resources: { items: [], supported: null, loading: false, saving: false, error: '', notice: '' },
form: { name: 'Princeton WordNet', language: 'en', version: '3.0', source: downloadURL, format: 'wordnet-3.0-zip' },
file: null, fileError: '', downloadURL
}),
computed: { busy() { return this.resources.loading || this.resources.saving } },
watch: { '$store.state.generation': { flush: 'sync', handler() { this.loader.invalidate(); this.clearFile() } } },
created() { this.loader = createDictionaryLoader(session, this.resources) },
mounted() { this.reload() },
beforeUnmount() { this.loader.invalidate(); this.file = null },
methods: {
reload() { return this.loader.load() },
statusLabel(status) { return { ready: '可用', disabled: '已停用', unavailable: '不可用' }[status] || '不可用' },
clearFile() { this.file = null; this.fileError = ''; if (this.$refs.fileInput) this.$refs.fileInput.value = '' },
selectFile(event) {
this.fileError = ''
const selected = event.target.files?.[0]
this.file = null
if (!selected) return
if (!selected.name.toLowerCase().endsWith('.zip') || selected.size === 0 || selected.size > MAX_DICTIONARY_BYTES) {
this.fileError = '请选择不超过 32 MiB 的 ZIP 文件。'
event.target.value = ''
return
}
this.file = selected
},
async upload() {
if (this.busy || !this.file) return
this.fileError = ''
if (!this.form.name.trim() || !this.form.source.trim()) { this.fileError = '请填写名称和来源。'; return }
const data = new FormData()
for (const [key, value] of Object.entries(this.form)) data.append(key, value.trim())
data.append('file', this.file)
if (await this.loader.import(data)) this.clearFile()
},
toggle(resource) { return this.loader.toggle(resource.id, !resource.enabled) }
}
}
</script>
<style scoped>
.toolbar { display:flex; align-items:center; justify-content:space-between; gap:16px; margin-bottom:20px; }
h1 { font-size:20px; margin:0 0 8px; }
h2 { font-size:18px; margin:0 0 24px; }
.subtle { color:#606266; font-size:13px; line-height:1.6; }
.toolbar p { margin:0; }
.import-card { margin-top:20px; }
.dictionary-form { max-width:600px; }
.form-row { display:flex; gap:16px; }
.form-row .el-form-item { flex:1; min-width:0; }
.file-error { color:#b42318; }
.el-alert { margin-bottom:16px; }
input[type=file] { max-width:100%; }
a { color:#176b63; }
</style>
+55
View File
@@ -0,0 +1,55 @@
import test from 'node:test'
import assert from 'node:assert/strict'
import { createSession } from '../src/session.mjs'
const response = (data, status = 200) => ({ ok: status < 400, status, json: async () => ({ data, msg: '导入失败' }) })
function setup() {
const pending = []
const session = createSession({ fetch: (url, options) => new Promise(resolve => pending.push({ url, options, resolve })), storage: { getItem() {}, setItem() {}, removeItem() {} } })
session.state.user = { id: 1, username: 'fixture.admin', role: 'admin' }
session.state.token = 'fictional-test-token'
return { session, pending }
}
test('dictionary upload sends multipart with authorization and lets browser set boundary', async () => {
const { session, pending } = setup()
const form = new FormData()
form.append('name', 'Princeton WordNet')
form.append('file', new Blob(['fictional-archive']), 'wordnet.zip')
assert.equal(typeof session.importDictionary, 'function')
const request = session.importDictionary(form)
assert.equal(pending[0].url, '/api/v1/dictionaries/import')
assert.equal(pending[0].options.method, 'POST')
assert.equal(pending[0].options.body, form)
assert.equal(pending[0].options.headers['Content-Type'], undefined)
assert.equal(pending[0].options.headers.Authorization, 'Bearer fictional-test-token')
pending[0].resolve(response({ resource: { id: 1 }, duplicate: false }))
assert.deepEqual(await request, { resource: { id: 1 }, duplicate: false })
})
test('dictionary read and toggle follow API contract, ordinary accounts cannot mutate', async () => {
const { session, pending } = setup()
assert.equal(typeof session.listDictionaries, 'function')
const read = session.listDictionaries()
assert.equal(pending[0].url, '/api/v1/dictionaries')
pending[0].resolve(response({ items: [] }))
await read
const toggle = session.setDictionaryEnabled(1, false)
assert.equal(pending[1].url, '/api/v1/dictionaries/1')
assert.deepEqual(JSON.parse(pending[1].options.body), { enabled: false })
pending[1].resolve(response({ resource: { id: 1, enabled: false } }))
await toggle
session.state.user.role = 'learner'
await assert.rejects(session.importDictionary(new FormData()), /管理员/)
await assert.rejects(session.setDictionaryEnabled(1, true), /管理员/)
assert.equal(pending.length, 2)
})
test('dictionary response cannot cross a logout or account switch', async () => {
const { session, pending } = setup()
assert.equal(typeof session.importDictionary, 'function')
const request = session.importDictionary(new FormData())
session.clear()
pending[0].resolve(response({ resource: { id: 1 }, duplicate: false }))
await assert.rejects(request, /会话已变化/)
})
+57
View File
@@ -0,0 +1,57 @@
import test from 'node:test'
import assert from 'node:assert/strict'
async function fixture() {
const { createDictionaryLoader } = await import('../src/dictionaries.mjs')
const pending = []
const later = () => new Promise((resolve, reject) => pending.push({ resolve, reject }))
const session = { state: { generation: 1 }, listDictionaries: later, importDictionary: later, setDictionaryEnabled: later }
const state = { items: [{ id: 1, name: 'Existing', enabled: true }], supported: null, loading: false, saving: false, error: '', notice: '' }
return { loader: createDictionaryLoader(session, state), state, session, pending }
}
test('failed upload preserves the existing resource and exposes the error', async () => {
const { loader, state, pending } = await fixture()
const action = loader.import(new FormData())
assert.equal(state.saving, true)
pending[0].reject(new Error('文件摘要不匹配'))
assert.equal(await action, false)
assert.equal(state.items[0].name, 'Existing')
assert.equal(state.error, '文件摘要不匹配')
assert.equal(state.saving, false)
})
test('accepted mutation replaces old state directly, without a second refresh dependency', async () => {
const { loader, state, pending } = await fixture()
const action = loader.toggle(1, false)
pending[0].resolve({ resource: { id: 1, name: 'Existing', enabled: false, status: 'disabled' } })
assert.equal(await action, true)
assert.equal(state.items[0].enabled, false)
assert.equal(pending.length, 1)
})
test('an old list cannot overwrite a newer import and duplicate import is explicit', async () => {
const { loader, state, pending } = await fixture()
const read = loader.load()
const action = loader.import(new FormData())
pending[1].resolve({ resource: { id: 1, name: 'Imported', enabled: true }, duplicate: true })
await action
pending[0].resolve({ items: [{ id: 1, name: 'Old' }] })
await read
assert.equal(state.items[0].name, 'Imported')
assert.match(state.notice, /已存在/)
})
test('page exit or session change discards late responses', async () => {
const { loader, state, pending, session } = await fixture()
const action = loader.import(new FormData())
loader.invalidate()
pending[0].resolve({ resource: { id: 1, name: 'Late' } })
assert.equal(await action, false)
assert.deepEqual(state.items, [])
const read = loader.load()
session.state.generation++
pending[1].resolve({ items: [{ id: 1, name: 'Other session' }] })
await read
assert.deepEqual(state.items, [])
})
+13 -3
View File
@@ -2,8 +2,8 @@
generated: true (请先修改 Gitea Wiki,禁止直接编辑本文件)
wiki_page: Project-Profile
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Project-Profile.-
wiki_revision: 40ae5e3a0521195da4f1a9cd9f42beec15da20d4
synchronized_at: 2026-09-10T12:43:19Z
wiki_revision: 1855b5e99c288de90d3b804f7dfd60c5f2c35b45
synchronized_at: 2026-09-11T06:46:46Z
<!-- gitea-wiki-mirror:end -->
# LexGo 项目档案
@@ -37,7 +37,7 @@ LinguaCafe 参考基线由现有调研记录为 `c1ea298ce40c65b9dd33e9b26fd2e52
|---|---|---|---|
| D01 | 管理端底座 | **已确认使用 `D:/github_project/goadmin` 的 go-admin + go-admin-ui**,依据用户指令“管理端用D:\github_project\goadmin”;替代 gin-vue-admin 建议 | M0 验证指定本地版本配套、MySQL 8 及数据隔离,不再比较管理底座 |
| D02 | 数据库 | **已确认 MySQL 8**;依据 2026-09-10 用户原话“使用mysql8”。覆盖两份分析的数据库分歧,具体小版本待基线验证锁定 | 按单一 MySQL 8 数据层估算,不做 PostgreSQL 双库兼容 |
| D03 | NLP | 保留 Python 为建议,纯 Go 是否硬约束未确认 | 首发语言质量与开发量 |
| D03 | NLP | **用户于2026-09-11确认正式产品全 Go**;Python小样仅历史验证 | 使用词典词形候选,不承诺上下文消歧 |
| D04 | 默认学习语言 | 用户已确认英语;中文/日语后续需独立范围与验收 | 分词、读音和 UI |
| D05 | 用户范围 | **已确认:支持多账号、数据独立的自托管学习工具,先邀请少量用户使用** | 首版按多用户归属和隔离设计;不自动加入公开注册、邀请链接或组织租户 |
| D06 | 旧数据 | 全量迁移是否需要未确认,CSV 与完整迁移不同 | 迁移另估 |
@@ -136,3 +136,13 @@ server 是 go-admin 的选用模块接入:原样保留 SysUser、SysDept、必
## 日志审计基线(#18)
管理端新增两个审计列表,后端为 LexGo 自有日志模型与接口,参照 go-admin 模块布局但不复制其原始参数/响应持久化逻辑。schema v2 新增 lexgo_login_logs、lexgo_operation_logs;保留 90 天,启动和每小时分批清理,也可显式 audit-cleanup。新建/重置密码均为 6~72 UTF-8 字节;初始 bootstrap 保留 10 字节下限。#2 已验收,#18 已验收;默认模块中的其他候选未纳入。
## 全 Go 正式架构决定(2026-09-11,#6)
用户已明确选择全 Go:正式英语分词、原文位置映射、本地词典解析和词形候选查询由 Go 后端完成,不运行 Python NLP 服务。此前“Python 建议/全 Go 未决”仅为历史决策记录,由本决定覆盖;spikes/english 保留历史验证,不接入产品。#6 按该方向实施,当前方案见工单最新启动评论;WordNet 3.0 仍为首个资源(英语释义),词形规则候选不等同于 spaCy 上下文消歧,原文及个人学习状态不按候选合并。
## #6 当前工程状态(2026-09-11)
#5已验收且相关前置PR均已合入main。#6正式词典/点词查义已按全Go实现,已于2026-09-11通过用户验收,PR #25已合入main;server schema v4,管理端新增英语词典页,学习端加入Go分片及查词面板。正式Go进程不依赖Python NLP,WordNet包随数据库持久化。#3小样仍是历史验证;保存个人释义和状态归#7,音频封面#21、列表优化#24尚未实施。
+38 -2
View File
@@ -2,8 +2,8 @@
generated: true (请先修改 Gitea Wiki,禁止直接编辑本文件)
wiki_page: Architecture-and-Code-Map
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Architecture-and-Code-Map.-
wiki_revision: f981b86a7c6e5d2e353142d3006dcd6e0636c3ee
synchronized_at: 2026-09-11T03:01:48Z
wiki_revision: cb6d420632527a436f8b9a7a6a18413a0773f93d
synchronized_at: 2026-09-11T08:31:30Z
<!-- gitea-wiki-mirror:end -->
# 架构与代码地图
@@ -221,3 +221,39 @@ lexgo_books(owner_id, title, language)、lexgo_chapters(book_id, owner_id, ordin
- 重试自愈(R4):`retryChapter` 先把重试返回的章节状态应用到列表与阅读器并重新安排轮询,再做静默刷新。
验证:Go 全量用例 20 项通过(新增运行期恢复与尝试上限两项);学习端单测 38 项通过,其中 7 项在整改前的代码上复现失败;真实联调确认追加路径可用、被中断的任务在运行中被自动恢复(约 0.5 秒,无需重启)、重试在首次刷新失败后仍自动显示最终结果。
## 全 Go 正式架构决定(2026-09-11,#6)
用户已明确选择全 Go:正式英语分词、原文位置映射、本地词典解析和词形候选查询由 Go 后端完成,不运行 Python NLP 服务。此前“Python 建议/全 Go 未决”仅为历史决策记录,由本决定覆盖;spikes/english 保留历史验证,不接入产品。#6 按该方向实施,当前方案见工单最新启动评论;WordNet 3.0 仍为首个资源(英语释义),词形规则候选不等同于 spaCy 上下文消歧,原文及个人学习状态不按候选合并。
## #6 全 Go 词典与阅读器(2026-09-11,已验收并合入 main)
`server/app/lexgo/wordnet.go` 负责固定 WordNet ZIP 校验/内存解析、Unicode 分词及词形候选;`dictionary.go` 负责资源与章节查词 API;`database.go` schema v4 新增单槽共享资源表 lexgo_dictionaries(元数据、enabled、SHA、ZIP LONGBLOB),已有学习数据不改写。每个 Router 按 SHA 缓存一个不可变词典,查询先读资源元数据,缓存未命中才读取 ZIP;进程重启从数据库恢复,不需要 Python NLP 或额外资源目录。
| 接口 | 权限与输入/输出 |
|---|---|
| GET /api/v1/dictionaries | 管理员;items + supported,状态 ready/disabled/unavailable,不返回 archive 或本机路径 |
| POST /api/v1/dictionaries/import | 管理员;multipart name/language/version/source/format/file;返回 resource + duplicate;格式/来源/版本固定 |
| PATCH /api/v1/dictionaries/:id | 管理员;{enabled:boolean};返回 resource |
| GET /api/v1/chapters/:id/tokens | 本人 ready 章节;{textSha256,tokens:[{text,start,end,startUtf16,endUtf16,kind}]} |
| POST /api/v1/lookup | {chapterId,start,end},cp半开范围;本人ready完整单词;返回 status/query/matchedForm/candidates/entries/resource |
管理端 `Dictionaries.vue` + `dictionaries.mjs` + session multipart 方法,复用 go-admin 导航/表单与身份失效保护。学习端 `useReaderLookup.ts` 校验原文片段/SHA/所有位置,`ReaderTokens.vue` 渲染可聚焦单词,`LookupPanel.vue` 展示释义及临时个人草稿。桌面侧栏,手机固定底部45dvh面板;关闭恢复焦点,仅无后续手动滚动时恢复自动调整前位置。旧响应在换词/换章/退出/离页后失效。
参考:[WordNet 数据格式](https://wordnet.princeton.edu/documentation/wndb5wn)、[词形规则](https://wordnet.princeton.edu/documentation/morphy7wn)。#3 仅历史实验,#6 不调用其实验服务。
## #7 个人词条与阅读器状态(2026-09-11)
schema v5 新增 lexgo_terms:一个学习者对一个词形一条记录。身份键为 `(owner_id, language, term)`,`term` 是 Go 侧 `normalizeWord` 的结果(NFC、小写、弯撇号转直撇号),列使用 `utf8mb4_bin`,避免折叠 `resume`/`résumé`;`original_form` 保存最近一次保存的原词形供显示。`definition`/`examples` 是学习者自己的文本,例句按行存储;共享词典仍只在 `lexgo_dictionaries`,两者不混存。`status` 与 `level` 由数据库检查约束守住:只有 `learning` 允许 1~7,其他状态必须为 0。
| 接口 | 权限与输入/输出 |
|---|---|
| POST /api/v1/terms | 本人;`{chapterId,start,end,definition,examples[],status,level?}`;服务端按 #6 同一套 token 范围反推词形,`owner`/`language`/`term` 一律不接受客户端输入;唯一键 upsert,重复保存更新同一行;首次 201、更新 200,返回 `{term,created}` |
| GET /api/v1/terms/:id | 本人;他人编号与不存在编号统一 404,不泄露存在性 |
| GET /api/v1/chapters/:id/tokens | 在原响应上为 word 片段增加可选 `term:{id,status,level}`;服务端按本章词形分批(每批 500)查本人词条,其他章节保存的同形词同样命中 |
`server/app/lexgo/terms.go` 负责身份、状态/等级边界、文本上限、upsert 与章节点词状态;`database.go` 提供 v5;`dictionary.go` 的 tokens 与 lookup 共用 `wordAtRange`,保存与查询必须落在同一个完整单词范围上,所以客户端无法命名自己没有读到的词。个人词条不写审计日志。
学习端 `useReaderLookup.ts` 在原有查询状态上增加个人释义、例句、状态、已保存编号、预填与保存;打开已保存词先读 `GET /terms/:id`,读取失败时禁用保存,避免用空表单覆盖原内容。`LookupPanel.vue` 提供状态单选、释义与例句输入、保存与简短反馈;`ReaderTokens.vue` 按状态高亮 `is-new`/`is-learning`/`is-known`/`is-ignored`。换词、换章、离页、退出或切换账号都会清空表单、状态与高亮。管理端无改动。
+37 -2
View File
@@ -2,8 +2,8 @@
generated: true (请先修改 Gitea Wiki,禁止直接编辑本文件)
wiki_page: Business-Rules-and-Glossary
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Business-Rules-and-Glossary.-
wiki_revision: 4bd843d1f0134f5688407b7072d98333b40b6da1
synchronized_at: 2026-09-11T03:01:48Z
wiki_revision: 71389f97df86e17699f4af930022251fa354631b
synchronized_at: 2026-09-11T08:31:32Z
<!-- gitea-wiki-mirror:end -->
# 业务规则与术语
@@ -137,3 +137,38 @@ POST /lookup 接收 {surface,lemma?}。查词键单独 casefold/NFC/弯撇号转
- 运行期恢复不依赖重启:除了启动时的恢复,运行中的服务每次轮询都会把停留在 processing 且超过 15 秒的任务放回 pending,因此“领取已提交、完成事务失败”不会让章节永久卡在处理中。该阈值必须长于正常的领取到完成窗口;重复处理同一任务不会产生第二个章节,因为任务从不创建章节。
- 学习端离开页面时作废在途请求:目录页与阅读页在关闭时推进各自的请求序号,晚到的响应不会写回状态或重启轮询;导入页在提交过程中离开后,晚到的成功响应不会把用户导航回书籍页。
- 重试结果立即生效:重试被接受后先把返回的 pending 状态写入界面并继续轮询,因此紧随其后的一次刷新失败不会让页面停在处理失败。
## 全 Go 正式架构决定(2026-09-11,#6)
用户已明确选择全 Go:正式英语分词、原文位置映射、本地词典解析和词形候选查询由 Go 后端完成,不运行 Python NLP 服务。此前“Python 建议/全 Go 未决”仅为历史决策记录,由本决定覆盖;spikes/english 保留历史验证,不接入产品。#6 按该方向实施,当前方案见工单最新启动评论;WordNet 3.0 仍为首个资源(英语释义),词形规则候选不等同于 spaCy 上下文消歧,原文及个人学习状态不按候选合并。
## #6 正式词典规则(2026-09-11)
首个资源固定 Princeton WordNet 3.0,英语释义;ZIP 来源及 SHA 以 server/wordnet-resource.json 为准,LICENSE 原样保留于 server/WORDNET-LICENSE.txt。管理员上传指定包并配置显示名称,语言/来源/版本/格式固定;32MiB压缩、128MiB解压上限、成员与结构校验。资源为所有账号共享、仅管理员写;失败保持旧资源,重复上传复用id并启用,停用后查询返回 resource_missing。ZIP 随数据库备份,无运行时网络下载。
正式分词由 Go 完成:字母开始词,组合标记延续词,内部直/弯撇号连接字母;空白逐字保留,数字和符号为不可查询的 punctuation。start/end 是 Unicode code point 半开范围,另有UTF-16偏移;原文、CRLF、组合字符与emoji不归一化。只对查询键小写/NFC/撇号归一化。每次查询必须是本人ready章节内的完整单词,最大128码点,跨账号404,非法范围400。
exact优先;未命中再按WordNet异常表/词尾规则查候选,词性顺序n/v/a/r、最多12条释义,保留原数据s词性。返回lemma表示规则候选,不承诺上下文消歧;歧义不会合并个人学习状态。not_found与resource_missing区分,网络错误可重试,不阻断阅读。#3使用spaCy上下文lemma的实验路径由全Go规则候选替代。
个人释义目前仅当前选择的临时草稿,界面标记未保存;换词、关闭、换章、离页和身份变化清理。持久化与个人词汇状态由#7实现,不把临时输入宣传为保存成功。
## #7 个人词条规则(2026-09-11)
个人词条身份是「学习者+语言+规范化词形」:`normalizeWord` 做 NFC、小写与弯撇号转换,而原文、章节文本与显示用的原词形都不归一化。同一词形不同大小写是同一条记录;不同词形(`dog` 与 `dogs`)是不同记录,不按 WordNet 候选或 lemma 合并,与 #6「不按候选合并个人学习状态」一致。语言取自已登录学习者的英语空间,不从请求读取。
| 业务状态 | level | 含义 | 进入到期复习 | 原版 stage |
|---|---|---|---|---|
| new 新词 | 0 | 已保存、尚未开始学习;首次保存默认 | 由 #8 决定 | 2 |
| learning 学习中 | 1~7 | 正在复习,越接近 7 越熟 | 是 | -1~-7 |
| known 已知 | 0 | 已掌握,不再进入到期复习 | 否 | 0 |
| ignored 忽略 | 0 | 明确忽略,不计入已知 | 否 | 1 |
上表是原版合并编码(状态与等级在同一字段)的显式替代,供 CSV 导出与旧数据迁移映射;#7 只保存与返回等级,#8 负责复习推进与到期时间。非 `learning` 状态携带非 0 等级、`learning` 等级超出 1~7、以及未知状态一律拒绝,`learning` 缺省等级为 1。
个人释义可为空(允许只记录状态),最长 2000 字符,可含换行与制表符;例句最多 5 条、每条最长 500 字符,不能为空行或含换行。保存失败保留学习者已输入的内容,成功后显示「已保存 · 状态」并立即更新正文高亮。
保存幂等由唯一键承担:重复提交同一词形只更新同一行,不产生第二条冲突记录;同一账号多端同时编辑为最后写入生效,本版不引入版本冲突拒绝。所有读写都属于会话本人:章节必须本人且已就绪,篡改 id、owner、language、term 或携带未知字段返回 400,他人编号与本人不可见编号统一 404。跨账号不共享任何数据与前端缓存,退出或切换账号后不保留前一账号的词条与高亮。
例句只保存学习者手输内容,不自动关联原文句子:当前分词只有词/空白/标点边界,没有句子切分规则。原型 v1 中「原文例句已关联到词条」是演示文案,不作为契约。等级选择器属于 #8/#12 的编辑界面,阅读器面板只提供四个状态。
+37 -2
View File
@@ -2,8 +2,8 @@
generated: true (请先修改 Gitea Wiki,禁止直接编辑本文件)
wiki_page: Local-Development-and-Verification
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Local-Development-and-Verification.-
wiki_revision: 3bce5fae964b1050513c37319262679814d68f14
synchronized_at: 2026-09-11T03:01:49Z
wiki_revision: 919545b0d716180a49ac7dd2fe96bbeccdba0880
synchronized_at: 2026-09-11T08:31:32Z
<!-- gitea-wiki-mirror:end -->
# 本地开发与验证
@@ -305,3 +305,38 @@ node --test spikes/english/view.test.mjs
注意:MySQL 会话时区为 SYSTEM(本机为 UTC+8),而服务按 UTC 存储 DATETIME;核对任务时间时使用 UTC_TIMESTAMP 而不是 NOW(),否则会出现 8 小时的假偏差。
未在本轮验证:R3 的真实浏览器时序(离页与响应同时发生)只由单测覆盖;真机手机证据仍属 #4 缺口。
## #6 部署与验证(2026-09-11)
正式后端 schema v4,只增加 lexgo_dictionaries。显式 migrate 后启动纯Go lexgo.exe;Python scripts/server.py 仍是开发命令封装,产品运行不依赖Python NLP。
本机已从v3升级v4并重启lexgo-api,升级前后sys_user/lexgo_spaces/lexgo_sessions/lexgo_books/lexgo_chapters/lexgo_ingest_jobs计数一致。旧二进制保存在忽略的 .local/lexgo-pre-issue6.exe。回退:停止API,恢复旧二进制,将已确认v4的schema标记恢复3,保留新增资源表及全部学习数据,再启动旧API;不要删除数据或重新bootstrap。
管理端 http://127.0.0.1:5174 的“英语词典”页可下载指定包并导入/启停;本机已导入固定WordNet3.0,155287个词形/词性索引项。学习端 http://127.0.0.1:5173 打开本人ready章节,点词或Enter/空格查询,Escape关闭。went/mice应出现go/mouse候选。源包、本机凭据与测试证据仅存在忽略的.local,不进入Git。
验证:Go全包MySQL集成(专用lexgo_test_issue5)与go vet通过;学习端53单测、类型检查/构建、默认5173 Playwright3项通过;管理端31单测与lint通过,构建含已有Sass弃用与bundle体积提示;治理56测试与strict通过。实际ZIP解析、schema3→4原文/任务保留、权限、失败保留、重复启用、冷缓存路由重建都有覆盖;冷缓存测试不是完整备份恢复演练。
主审真实API联调使用真实管理端session模块经5174代理上传;经5173代理两测试账号分别创建虚构章节并验证精确/不规则词形、Unicode原文片段、越权404/普通用户管理403、停用/重复导入启用、错误ZIP保留资源。新增测试书籍id3/4、章节id8/9归issue5_a/issue5_b,没有修改其他账号的书籍。
常驻5173一度返回空白页:Vue模块转换500、代理缺失;同代码隔离服务正常,只重启lexgo-learner加载配置后恢复,默认E2E通过,未改启动配置。桌面交互浏览器工具因旧会话失效未完成手工真实UI联调;已有项目Playwright使用模拟API,真实API验证另列。手机仅窄屏自动测试,真机证据仍未补齐。
## #7 验证与迁移(2026-09-11)
仓库根执行;Go 工具链由 `python scripts/server.py` 固定 go1.26.5。本单使用专用测试库 lexgo_test_issue7,不借用其他测试库。
| 命令 | 结果 |
|---|---|
| `go vet ./...` | 通过 |
| `LEXGO_TEST_DB_NAME=lexgo_test_issue7 python scripts/server.py test-integration` | 34 个顶层用例全部通过、0 跳过;含 #7 新增 3 个 MySQL 用例、4 个单元用例和 1 个 v4→v5 迁移用例 |
| `cd learner`:`npx vitest --run` / `npx vue-tsc --build` / `npx pnpm run build` / `npx playwright test` | 58 项单测、类型检查、构建、3 项默认 E2E 全部通过 |
| `cd admin`:`npx pnpm test` / `npx pnpm lint` | 31 项与 lint 通过;管理端本单无代码改动 |
| `python -m unittest discover -s tests` / `python dev_scripts/harness.py check --strict` | 56 项与严格检查通过 |
覆盖内容:非法状态与等级边界、文本上限、词形身份与大小写合并、词形不按 lemma 合并、重复与并发保存只留一条记录、跨章节同形词状态一致、两账号互不影响、篡改 owner/language/term 被拒绝、非本人章节 404、未登录 401、未就绪章节 409、v4→v5 迁移保留既有数据与检查约束。
本机开发库 lexgo_dev 已显式从 v4 升级到 v5:升级前后 sys_user 6、lexgo_spaces 6、lexgo_sessions 8、lexgo_books 4、lexgo_chapters 9、lexgo_ingest_jobs 9、lexgo_dictionaries 1 全部不变,新增空的 lexgo_terms。旧二进制备份在忽略的 `.local/lexgo-pre-issue7.exe`。回退:停止 lexgo-api,恢复旧二进制,把 `lexgo_schema` 标记改回 4 后启动;保留 lexgo_terms 与全部既有数据,不删除数据、不重新 bootstrap。
真实链路验证:真实 Go API+真实 MySQL 共 30 项检查通过(凭据只从本机安全配置读入进程),覆盖两个虚构测试账号 issue5_a/issue5_b 的登录、保存、幂等、状态边界、跨章节一致、跨账号隔离与越权拒绝;随后用临时 Playwright 用例在真实学习端+真实 API 上以两个账号复核保存、重新加载后的高亮与预填,以及 390×844 窄屏底部面板。截图保存在本机 `.local/evidence/`(issue7-reader-desktop.png、issue7-reader-mobile-390.png、issue7-account-b.png),临时用例运行后删除。
未验证:真实手机触屏详细证据与完整备份恢复演练仍属既有缺口(#14/#15);本单只用桌面浏览器窄屏检查,不当作真机结果。
+18 -2
View File
@@ -2,8 +2,8 @@
generated: true (请先修改 Gitea Wiki,禁止直接编辑本文件)
wiki_page: Product-Requirements-Overview
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Product-Requirements-Overview.-
wiki_revision: 125df697d3d45dee098616d25970bb65a96727cd
synchronized_at: 2026-09-10T16:35:19Z
wiki_revision: b5eaaf1d0664e1e3c564079c857ccb5042540022
synchronized_at: 2026-09-11T08:31:32Z
<!-- gitea-wiki-mirror:end -->
# 产品需求总览
@@ -229,3 +229,19 @@ Anki(U20)、YouTube/Jellyfin 远程字幕(U07/A08)、全量旧实例迁
2026-09-10:#5“粘贴英语文本,处理后进入本人章节阅读”已实现并待用户验收,覆盖 F01 基础(书库与章节)、F02(粘贴导入)、F04 基础(处理状态与失败重试)与 F05 原文(可读原文与章节切换)。本节取代此前“阅读、导入尚未实现”的表述:粘贴导入与原文阅读已实现;点词查词、词典、个人词语状态、复习与统计仍未实现(#6~#15)。schema 升级为 v3,新增 lexgo_books、lexgo_chapters、lexgo_ingest_jobs。
范围边界不变:本单按 Go 处理,不接入 Python NLP,因此不产生 token、lemma 或词典索引;Go+Python NLP 与全 Go 路线仍未确认,正式接入前必须由用户确认。真机手机详细证据仍缺失(#4 缺口保持)。
## 全 Go 正式架构决定(2026-09-11,#6)
用户已明确选择全 Go:正式英语分词、原文位置映射、本地词典解析和词形候选查询由 Go 后端完成,不运行 Python NLP 服务。此前“Python 建议/全 Go 未决”仅为历史决策记录,由本决定覆盖;spikes/english 保留历史验证,不接入产品。#6 按该方向实施,当前方案见工单最新启动评论;WordNet 3.0 仍为首个资源(英语释义),词形规则候选不等同于 spaCy 上下文消歧,原文及个人学习状态不按候选合并。
## #6 交付范围更新(2026-09-11)
用户确认全Go后,英语词典配置与阅读器点词查义已实现待验收:共享WordNet3.0英语释义、管理员导入/启停、本人章节点击/键盘查词、加载/无结果/资源不可用/网络失败/关闭状态。词形结果是规则候选,不提供上下文词性消歧。个人释义输入是未保存临时草稿,#7才持久化。沿用已验收v1;手机底部45dvh面板自动测试通过,真机缺口保留。#5已关闭并合入main,#6尚不关闭。
## #7 交付范围更新(2026-09-11)
F07 的个人词语记录已实现,待用户验收:阅读器可以保存与修改个人释义、例句和状态(新词/学习中/已知/忽略),同一词形在本人其他章节显示相同状态与高亮,两个账号的数据互不影响。个人释义与共享 WordNet 词典分开存储,个人释义不进入审计日志。schema 升级为 v5,新增 lexgo_terms。
仍未实现并留给后续工单:词汇库搜索与编辑(#12)、短语(#11)、到期复习与等级推进(#8)、阅读完成与进度(#13)、TXT 导入(#9)、书籍章节编辑删除(#10)。词形候选仍不提供上下文消歧,个人学习状态不按候选合并;例句不自动关联原文句子。
+9 -2
View File
@@ -2,8 +2,8 @@
generated: true (请先修改 Gitea Wiki,禁止直接编辑本文件)
wiki_page: Home
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Home
wiki_revision: 5b936b75caeec89cd54f7147c842f0b66f901307
synchronized_at: 2026-09-10T16:35:18Z
wiki_revision: 6fd4c754e3bb6ab215edb6077974fbd401f24103
synchronized_at: 2026-09-11T08:31:29Z
<!-- gitea-wiki-mirror:end -->
# LexGo 文档入口
@@ -65,3 +65,10 @@ Quant-UX 原型 v1 已通过用户验收。[桌面预览](https://qux.ilapage.cn
#4 阅读选择小样已实现桌面鼠标/键盘、范围调整与原文位置验证,入口 http://127.0.0.1:5184/;固定版本 LinguaCafe 四项源码对照已记录。用户已验收并关闭 #4;真实手机详细测试证据仍缺失,详见本地验证页。#21 仍待实施。
#5 粘贴导入与章节阅读已实现,待用户验收:schema v3 新增 books/chapters/ingest_jobs,学习端具备粘贴导入、书库、处理状态与原文阅读;查词、词典与复习仍未实现。本单按 Go 处理,Go+Python NLP 与全 Go 路线仍未决,正式接入前须用户确认。
## 当前进度(2026-09-11)
#5已验收,#17/#19/#20/#22/#23按依赖顺序合入main。#6已按用户确认的全Go方向实现英语词典与阅读点词,2026-09-11通过用户验收,PR #25已合入main;原文/账户隔离保留。正式NLP不使用Python服务。管理端“英语词典”导入指定WordNet3.0,学习端打开本人章节即可查词。个人释义与学习状态已实现持久化(#7,待用户验收)。详见#6工单、架构和本地开发页面。
#7 已实现个人词条(schema v5 新增 lexgo_terms):阅读器可保存释义、例句与状态,同一词形在本人其他章节显示一致高亮,两个账号数据独立;保存幂等,跨账号与篡改身份均被拒绝。本单待用户验收。
+65
View File
@@ -9,6 +9,8 @@ test('paste English text, watch a chapter finish processing, then read it verbat
const timestamps = { createdAt: '2026-01-01T00:00:00Z', updatedAt: '2026-01-01T00:00:00Z' }
// The worker reports the fresh chapter as processing until the worker settles it.
let status: 'processing' | 'ready' = 'processing'
// The personal record the learner saves during this run, served back on reload.
let savedTerm: { id: number; term: string; originalForm: string; definition: string; examples: string[]; status: string; level: number } | null = null
const chapterPayload = () => ({
id: 55,
bookId: book.id,
@@ -52,6 +54,29 @@ test('paste English text, watch a chapter finish processing, then read it verbat
chapter: { ...chapterPayload(), contentSha256: 'fictional-sha256', ...(status === 'ready' ? { originalText: pasted } : {}) },
navigation: { previousChapterId: null, nextChapterId: null },
}
} else if (path === '/api/v1/chapters/55/tokens') {
let offset = 0
const tokens = (pasted.match(/[A-Za-z]+|\s+|[^A-Za-z\s]+/g) ?? []).map(text => {
const start = offset
offset += text.length
return { text, start, end: offset, startUtf16: start, endUtf16: offset, kind: /^[A-Za-z]+$/.test(text) ? 'word' : /^\s+$/.test(text) ? 'space' : 'punctuation' }
})
data = {
textSha256: 'fictional-sha256',
tokens: tokens.map(token => savedTerm && token.text === 'First'
? { ...token, term: { id: savedTerm.id, status: savedTerm.status, level: savedTerm.level } }
: token),
}
} else if (path === '/api/v1/terms' && method === 'POST') {
const body = route.request().postDataJSON() as { chapterId: number; start: number; end: number; definition: string; examples: string[]; status: string }
expect(body).toEqual({ chapterId: 55, start: 0, end: 5, definition: '虚构的个人释义', examples: ['A fictional example.'], status: 'new' })
savedTerm = { id: 9, term: 'first', originalForm: 'First', definition: body.definition, examples: body.examples, status: body.status, level: 0 }
statusCode = 201
data = { term: savedTerm, created: true }
} else if (path === '/api/v1/terms/9') data = { term: savedTerm }
else if (path === '/api/v1/lookup') {
expect(route.request().postDataJSON()).toEqual({ chapterId: 55, start: 0, end: 5 })
data = { status: 'exact', query: 'first', matchedForm: 'first', candidates: [], entries: [{ lemma: 'first', pos: 'adjective', definition: 'Coming before all others.', examples: ['The first fictional chapter.'] }] }
}
await route.fulfill({ status: statusCode, json: { code: 200, data } })
})
@@ -89,4 +114,44 @@ test('paste English text, watch a chapter finish processing, then read it verbat
expect(await readerText.evaluate(element => getComputedStyle(element).whiteSpace)).toBe('pre-wrap')
await expect(page.getByRole('button', { name: '上一章' })).toBeDisabled()
await expect(page.getByRole('button', { name: '下一章' })).toBeDisabled()
const word = page.locator('.reader-word').first()
await expect(word).toHaveAttribute('aria-label', '查询 First')
await word.focus()
await word.press('Enter')
await expect(page.getByText('Coming before all others.')).toBeVisible()
expect(await readerText.evaluate(element => element.textContent)).toBe(pasted)
// Browser narrow viewport check only; this is not real-device acceptance.
await page.setViewportSize({ width: 390, height: 844 })
await word.click()
await expect(page.getByText('Coming before all others.')).toBeVisible()
const panelBounds = await page.locator('.lookup-panel').boundingBox()
expect(panelBounds!.y + panelBounds!.height).toBeLessThanOrEqual(845)
expect(panelBounds!.height).toBeLessThanOrEqual(844 * 0.46)
expect(await page.locator('.lookup-panel').evaluate(element => getComputedStyle(element).position)).toBe('fixed')
const wordBounds = await word.boundingBox()
expect(wordBounds!.y + wordBounds!.height).toBeLessThanOrEqual(panelBounds!.y)
expect(await page.getByLabel('我的释义 新词条').inputValue()).toBe('')
// Save a personal record: definition, example and the default status.
await page.setViewportSize({ width: 1280, height: 900 })
await page.getByLabel('我的释义 新词条').fill('虚构的个人释义')
await page.getByLabel('例句 每行一条,最多 5 条').fill('A fictional example.')
await page.getByRole('button', { name: '保存到生词本' }).click()
await expect(page.getByText('已保存 · 新词')).toBeVisible()
await expect(word).toHaveClass(/is-new/)
expect(await readerText.evaluate(element => element.textContent)).toBe(pasted)
await page.getByRole('button', { name: '关闭释义' }).press('Escape')
await expect(page.locator('.lookup-panel')).toHaveCount(0)
await expect(word).toBeFocused()
// Returning to the chapter shows the same state and the stored text.
await page.reload()
const reloadedWord = page.locator('.reader-word').first()
await expect(reloadedWord).toHaveAttribute('aria-label', '查询 First,已保存')
await expect(reloadedWord).toHaveClass(/is-new/)
await reloadedWord.click()
await expect(page.getByLabel('我的释义 已保存')).toHaveValue('虚构的个人释义')
await expect(page.getByLabel('例句 每行一条,最多 5 条')).toHaveValue('A fictional example.')
await expect(page.getByRole('radio', { name: '新词' })).toBeChecked()
})
+252
View File
@@ -0,0 +1,252 @@
import { afterEach, beforeEach, describe, expect, it, vi, type MockInstance } from 'vitest'
import { flushPromises, mount, type VueWrapper } from '@vue/test-utils'
import { createPinia, setActivePinia } from 'pinia'
import { createMemoryHistory, createRouter } from 'vue-router'
import ReaderView from '../views/ReaderView.vue'
import { useSessionStore } from '../stores/session'
import { useReaderLookup, type SavedTerm } from '../composables/useReaderLookup'
import { effectScope, ref } from 'vue'
import type { ChapterDetail } from '../stores/library'
// Fictitious text includes astral, combining, CRLF and repeated whitespace.
const original = '😀 Cats\r\n café!'
const fragments = [['😀', 'punctuation'], [' ', 'space'], ['Cats', 'word'], ['\r\n ', 'space'], ['café', 'word'], ['!', 'punctuation']]
let cp = 0, utf16 = 0
const tokens = fragments.map(([text = '', kind]) => {
const token = { text, kind, start: cp, end: cp + [...text].length, startUtf16: utf16, endUtf16: utf16 + text.length }
cp = token.end; utf16 = token.endUtf16
return token
})
const chapter = { id: 55, bookId: 1, title: '虚构章节', status: 'ready', originalText: original, contentSha256: 'same-sha' }
const ok = (data: unknown) => new Response(JSON.stringify({ code: 200, data }))
const result = (query: string, status = 'exact') => ({ status, query, matchedForm: query, candidates: [], entries: status === 'exact' ? [{ lemma: query, pos: 'noun', definition: `Definition of ${query}`, examples: ['A fictional example.'] }] : [] })
const savedTerm = (overrides: Partial<SavedTerm> = {}): SavedTerm => ({
id: 7, term: 'cats', originalForm: 'Cats', definition: '猫', examples: ['A fictional example.'], status: 'learning', level: 2, ...overrides,
})
let wrapper: VueWrapper | undefined
interface OpenOptions {
lookup?: (body: Record<string, number>) => Promise<Response>
tokenData?: unknown
termRead?: () => Promise<Response>
termWrite?: (body: Record<string, unknown>) => Promise<Response>
}
async function open(options: OpenOptions = {}) {
const { lookup = async () => ok(result('cat')), tokenData = { textSha256: 'same-sha', tokens } } = options
const fetchMock = vi.spyOn(globalThis, 'fetch').mockImplementation(async (input, init) => {
const url = String(input)
if (url.endsWith('/tokens')) return ok(tokenData)
if (url.endsWith('/lookup')) return lookup(JSON.parse(String(init?.body)))
if (url.includes('/terms/')) return options.termRead ? options.termRead() : ok({ term: savedTerm() })
if (url.endsWith('/terms')) return options.termWrite ? options.termWrite(JSON.parse(String(init?.body))) : ok({ term: savedTerm(), created: true })
return ok({ chapter, navigation: { previousChapterId: null, nextChapterId: 56 } })
})
useSessionStore().user = { id: 42, username: 'fictional', role: 'learner' }
const router = createRouter({ history: createMemoryHistory(), routes: [{ path: '/chapters/:id', component: ReaderView }, { path: '/', component: { template: '<div />' } }] })
await router.push('/chapters/55')
wrapper = mount(ReaderView, { attachTo: document.body, global: { plugins: [router] } })
await flushPromises()
return { fetchMock, router, view: wrapper }
}
const callsTo = (fetchMock: MockInstance, suffix: string) => fetchMock.mock.calls.filter(call => String(call[0]).endsWith(suffix))
// Personal state is attached to the word itself, never to a space or punctuation token.
const withTerm = (text: string, term: { id: number; status: string; level: number }) => ({
textSha256: 'same-sha',
tokens: tokens.map(token => token.text === text ? { ...token, term } : token),
})
describe('reader word lookup', () => {
beforeEach(() => { setActivePinia(createPinia()); sessionStorage.clear() })
afterEach(() => { wrapper?.unmount(); wrapper = undefined; vi.restoreAllMocks() })
it('keeps Unicode original text exact and sends only chapter and code-point offsets', async () => {
const { view, fetchMock } = await open()
expect(view.get('.reader-text').element.textContent).toBe(original)
const words = view.findAll('.reader-word')
expect(words).toHaveLength(2)
await words[0]!.trigger('click'); await flushPromises()
const call = callsTo(fetchMock, '/lookup')[0]!
expect(call[1]?.method).toBe('POST')
expect(JSON.parse(String(call[1]?.body))).toEqual({ chapterId: 55, start: 2, end: 6 })
expect(view.text()).toContain('Definition of cat')
expect(view.text()).toContain('A fictional example.')
})
it('discards older word responses and clears the selection form', async () => {
let finish!: (response: Response) => void
const { view } = await open({ lookup: body => body.start === 2 ? new Promise(resolve => { finish = resolve }) : Promise.resolve(ok(result('cafe', 'not_found'))) })
await view.findAll('.reader-word')[0]!.trigger('click')
expect(view.text()).toContain('正在查询')
await view.findAll('.reader-word')[1]!.trigger('click'); await flushPromises()
await view.get('#term-definition').setValue('虚构私人草稿')
finish(ok(result('OLD'))); await flushPromises()
expect(view.text()).not.toContain('Definition of OLD')
expect(view.text()).toContain('未找到释义')
expect(view.text()).toContain('新词条')
await view.findAll('.reader-word')[0]!.trigger('click')
expect((view.get('#term-definition').element as HTMLTextAreaElement).value).toBe('')
})
it.each(['resource_missing', 'not_found', 'error'])('keeps reading and permits a saved word and retry for %s', async status => {
const { view, fetchMock } = await open({ lookup: async () => { if (status === 'error') throw new Error('暂时无法查询'); return ok(result('cat', status)) } })
await view.get('.reader-word').trigger('click'); await flushPromises()
await view.get('#term-definition').setValue('仅当前词的虚构释义')
expect(view.get('.reader-text').element.textContent).toBe(original)
expect(view.get('[data-testid="term-save"]').attributes('disabled')).toBeUndefined()
expect(callsTo(fetchMock, '/lookup')).toHaveLength(1)
await view.get('[data-testid="lookup-retry"]').trigger('click'); await flushPromises()
expect(callsTo(fetchMock, '/lookup')).toHaveLength(2)
})
it.each([{ textSha256: 'wrong', tokens }, { textSha256: 'same-sha', tokens: tokens.slice(1) }])('falls back to original when tokens do not match', async data => {
const { view } = await open({ tokenData: data })
expect(view.get('.reader-text').element.textContent).toBe(original)
expect(view.find('.reader-word').exists()).toBe(false)
expect(view.find('[data-testid="tokens-retry"]').exists()).toBe(true)
})
it('closes with Escape and restores the word focus without scrolling', async () => {
const { view } = await open()
const word = view.get('.reader-word').element as HTMLElement
word.focus()
await view.get('.reader-word').trigger('keydown', { key: 'Enter' }); await flushPromises()
const focus = vi.spyOn(word, 'focus')
await view.get('.lookup-panel').trigger('keydown', { key: 'Escape' })
await flushPromises()
expect(view.find('.lookup-panel').exists()).toBe(false)
expect(document.activeElement).toBe(word)
expect(focus).toHaveBeenCalledWith({ preventScroll: true })
})
it('retries a failed token fetch without losing original text', async () => {
const { view, fetchMock } = await open()
fetchMock.mockRejectedValueOnce(new Error('网络暂不可用'))
// A new chapter load invalidates the old token rendering.
const { useLibraryStore } = await import('../stores/library')
useLibraryStore().chapter = { ...useLibraryStore().chapter!, contentSha256: 'retry-sha' }
await flushPromises()
expect(view.get('.reader-text').element.textContent).toBe(original)
expect(view.find('.reader-word').exists()).toBe(false)
fetchMock.mockResolvedValueOnce(ok({ textSha256: 'retry-sha', tokens }))
await view.get('[data-testid="tokens-retry"]').trigger('click'); await flushPromises()
expect(view.findAll('.reader-word')).toHaveLength(2)
})
it('does not accept a late token response after identity changes', async () => {
const { view, fetchMock } = await open({ tokenData: { textSha256: 'wrong', tokens } })
let finish!: (response: Response) => void
fetchMock.mockImplementationOnce(() => new Promise(resolve => { finish = resolve }))
await view.get('[data-testid="tokens-retry"]').trigger('click')
await useSessionStore().logout()
useSessionStore().user = { id: 42, username: 'fictional', role: 'learner' }
finish(ok({ textSha256: 'same-sha', tokens })); await flushPromises()
expect(view.find('.reader-word').exists()).toBe(false)
})
it('moves a covered selected word above the sheet and restores the prior scroll on close', async () => {
useSessionStore().user = { id: 42, username: 'fictional', role: 'learner' }
vi.spyOn(globalThis, 'fetch').mockResolvedValue(ok(result('cat')))
const scrollBy = vi.spyOn(window, 'scrollBy').mockImplementation(() => {})
const scrollTo = vi.spyOn(window, 'scrollTo').mockImplementation(() => {})
const scope = effectScope()
const lookup = scope.run(() => useReaderLookup(ref(chapter as ChapterDetail)))!
const element = document.createElement('span')
vi.spyOn(element, 'getBoundingClientRect').mockReturnValue({ bottom: 700 } as DOMRect)
lookup.select(tokens[2] as never, element)
lookup.keepSelectionVisible(500)
expect(scrollBy).toHaveBeenCalledWith({ top: 216, behavior: 'instant' })
lookup.close()
expect(scrollTo).toHaveBeenCalledWith({ top: 0, left: 0, behavior: 'instant' })
scope.stop()
})
it('preserves a later manual reading scroll when closing the sheet', async () => {
useSessionStore().user = { id: 42, username: 'fictional', role: 'learner' }
vi.spyOn(globalThis, 'fetch').mockResolvedValue(ok(result('cat')))
let scrollY = 100
vi.spyOn(window, 'scrollY', 'get').mockImplementation(() => scrollY)
vi.spyOn(window, 'scrollBy').mockImplementation((...args: unknown[]) => {
const options = args[0] as ScrollToOptions
scrollY += options.top ?? 0
})
const scrollTo = vi.spyOn(window, 'scrollTo').mockImplementation(() => {})
const scope = effectScope()
const lookup = scope.run(() => useReaderLookup(ref(chapter as ChapterDetail)))!
const element = document.createElement('span')
vi.spyOn(element, 'getBoundingClientRect').mockReturnValue({ bottom: 700 } as DOMRect)
lookup.select(tokens[2] as never, element)
lookup.keepSelectionVisible(500)
scrollY += 300 // The reader continues down the page while the sheet is open.
lookup.close()
expect(scrollTo).not.toHaveBeenCalled()
expect(scrollY).toBe(616)
scope.stop()
})
it.each(['logout', 'chapter', 'unmount'])('invalidates pending lookup on %s', async action => {
let finish!: (response: Response) => void
const { view, router } = await open({ lookup: () => new Promise(resolve => { finish = resolve }) })
await view.get('.reader-word').trigger('click')
if (action === 'logout') await useSessionStore().logout()
else if (action === 'chapter') await router.push('/chapters/56')
else { view.unmount(); wrapper = undefined }
finish(ok(result('OLD'))); await flushPromises()
if (action === 'unmount') expect(document.querySelector('.lookup-panel')).toBeNull()
else expect(view.find('.lookup-panel').exists()).toBe(false)
expect(view.text()).not.toContain('Definition of OLD')
})
})
describe('personal word records', () => {
beforeEach(() => { setActivePinia(createPinia()); sessionStorage.clear() })
afterEach(() => { wrapper?.unmount(); wrapper = undefined; vi.restoreAllMocks() })
it('saves the definition, examples and status, then marks the word in the text', async () => {
let body: Record<string, unknown> | undefined
const { view, fetchMock } = await open({ termWrite: async value => { body = value; return ok({ term: savedTerm({ status: 'learning', level: 2 }), created: true }) } })
await view.findAll('.reader-word')[0]!.trigger('click'); await flushPromises()
await view.get('#term-definition').setValue(' 猫 ')
await view.get('#term-examples').setValue('A fictional example.\n\n Second line ')
await view.findAll('input[type="radio"]')[1]!.setValue()
await view.get('[data-testid="term-save"]').trigger('click'); await flushPromises()
expect(body).toEqual({ chapterId: 55, start: 2, end: 6, definition: ' 猫 ', examples: ['A fictional example.', 'Second line'], status: 'learning' })
expect(view.text()).toContain('已保存 · 学习中')
expect(view.findAll('.reader-word')[0]!.classes()).toContain('is-learning')
expect(callsTo(fetchMock, '/terms')).toHaveLength(1)
})
it('keeps the typed input and reports the failure when saving fails', async () => {
const { view } = await open({ termWrite: async () => { throw new Error('保存暂时不可用') } })
await view.findAll('.reader-word')[0]!.trigger('click'); await flushPromises()
await view.get('#term-definition').setValue('虚构释义')
await view.get('[data-testid="term-save"]').trigger('click'); await flushPromises()
expect(view.text()).toContain('保存暂时不可用')
expect((view.get('#term-definition').element as HTMLTextAreaElement).value).toBe('虚构释义')
expect(view.find('.lookup-panel').exists()).toBe(true)
expect(view.findAll('.reader-word')[0]!.classes()).not.toContain('is-new')
})
it('loads the stored record when a saved word is opened and marks it in the text', async () => {
const tokenData = withTerm('Cats', { id: 7, status: 'known', level: 0 })
const { view, fetchMock } = await open({ tokenData, termRead: async () => ok({ term: savedTerm({ status: 'known', level: 0 }) }) })
expect(view.findAll('.reader-word')[0]!.classes()).toContain('is-known')
expect(view.findAll('.reader-word')[0]!.attributes('aria-label')).toContain('已保存')
await view.findAll('.reader-word')[0]!.trigger('click'); await flushPromises()
expect(callsTo(fetchMock, '/terms/7')).toHaveLength(1)
expect((view.get('#term-definition').element as HTMLTextAreaElement).value).toBe('猫')
expect((view.get('#term-examples').element as HTMLTextAreaElement).value).toBe('A fictional example.')
expect((view.findAll('input[type="radio"]')[2]!.element as HTMLInputElement).checked).toBe(true)
expect(view.text()).toContain('已保存')
})
it('blocks saving until a failed read of the stored record is retried', async () => {
const tokenData = withTerm('Cats', { id: 7, status: 'learning', level: 1 })
let attempts = 0
const { view, fetchMock } = await open({ tokenData, termRead: async () => { attempts++; if (attempts === 1) throw new Error('已保存的内容暂时无法读取'); return ok({ term: savedTerm() }) } })
await view.findAll('.reader-word')[0]!.trigger('click'); await flushPromises()
expect(view.text()).toContain('已保存的内容暂时无法读取')
expect(view.get('[data-testid="term-save"]').attributes('disabled')).toBeDefined()
await view.get('.reader-word').trigger('click'); await flushPromises()
expect((view.get('#term-definition').element as HTMLTextAreaElement).value).toBe('猫')
expect(view.get('[data-testid="term-save"]').attributes('disabled')).toBeUndefined()
expect(callsTo(fetchMock, '/terms')).toHaveLength(0)
})
it('drops the previous account words and highlights after an identity change', async () => {
const { view, fetchMock } = await open({ termWrite: async () => ok({ term: savedTerm({ status: 'new', level: 0 }), created: true }) })
await view.findAll('.reader-word')[0]!.trigger('click'); await flushPromises()
await view.get('[data-testid="term-save"]').trigger('click'); await flushPromises()
expect(view.findAll('.reader-word')[0]!.classes()).toContain('is-new')
fetchMock.mockResolvedValueOnce(ok({ textSha256: 'same-sha', tokens }))
useSessionStore().user = { id: 43, username: 'other-fictional', role: 'learner' }
await flushPromises()
expect(view.find('.lookup-panel').exists()).toBe(false)
// The previous account's words and chapter are gone: the reader falls back to
// nothing until the new identity loads its own chapter.
expect(view.find('.reader-word').exists()).toBe(false)
expect(view.find('.reader-text').exists()).toBe(false)
expect(view.text()).not.toContain('已保存')
})
})
+85
View File
@@ -0,0 +1,85 @@
<script setup lang="ts">
import { onMounted, onUnmounted, ref } from 'vue'
import { ElButton, ElInput, ElOption, ElRadio, ElRadioGroup, ElSelect } from 'element-plus'
import { TERM_STATUSES, type LookupResult, type TermStatus } from '../composables/useReaderLookup'
defineProps<{
word: string
result: LookupResult | null
loading: boolean
error: string
saving: boolean
saveError: string
saved: string
savedTermId: number | null
prefilling: boolean
prefillError: string
canSave: boolean
}>()
const definition = defineModel<string>('definition', { required: true })
const examples = defineModel<string>('examples', { required: true })
const status = defineModel<TermStatus>('status', { required: true })
const emit = defineEmits<{ close: []; retry: []; save: []; resize: [top: number] }>()
const heading = ref<HTMLElement | null>(null)
const panel = ref<HTMLElement | null>(null)
let observer: ResizeObserver | undefined
function resized() {
if (window.innerWidth <= 760 && panel.value) emit('resize', panel.value.getBoundingClientRect().top)
}
onMounted(() => {
heading.value?.focus({ preventScroll: true })
resized()
if (typeof ResizeObserver !== 'undefined' && panel.value) {
observer = new ResizeObserver(resized)
observer.observe(panel.value)
}
window.addEventListener('resize', resized)
})
onUnmounted(() => { observer?.disconnect(); window.removeEventListener('resize', resized) })
</script>
<template>
<aside ref="panel" class="lookup-panel" aria-labelledby="lookup-heading" @keydown.esc.stop.prevent="$emit('close')">
<header class="lookup-heading">
<h2 id="lookup-heading" ref="heading" tabindex="-1">{{ word }}</h2>
<ElButton text aria-label="关闭释义" @click="$emit('close')">关闭</ElButton>
</header>
<div class="lookup-content" aria-live="polite" :aria-busy="loading">
<p v-if="loading" role="status" class="subtle">正在查询…</p>
<p v-else-if="error" role="alert" class="lookup-message">{{ error }}</p>
<template v-else-if="result">
<p v-if="result.status === 'resource_missing'" class="lookup-message">词典资源暂不可用。</p>
<p v-else-if="result.status === 'not_found'" class="lookup-message">未找到释义。</p>
<p v-if="result.status === 'lemma'" class="subtle">词形匹配:{{ result.matchedForm }}(规则候选)</p>
<ol v-if="result.entries.length" class="lookup-senses">
<li v-for="(entry, index) in result.entries" :key="index">
<p class="sense-heading"><strong>{{ entry.lemma }}</strong> <span>{{ entry.pos }}</span></p>
<p lang="en">{{ entry.definition }}</p>
<blockquote v-for="(example, exampleIndex) in entry.examples" :key="exampleIndex" lang="en">{{ example }}</blockquote>
</li>
</ol>
<p v-if="result.resource" class="subtle">{{ result.resource.name }} · {{ result.resource.version }}</p>
</template>
<ElButton v-if="!loading && (error || result?.status === 'resource_missing' || result?.status === 'not_found')" data-testid="lookup-retry" @click="$emit('retry')">重试查询</ElButton>
</div>
<div class="lookup-term">
<p class="lookup-term-title">学习状态</p>
<ElRadioGroup v-model="status" :disabled="saving || prefilling || !!prefillError" aria-label="学习状态">
<ElRadio v-for="item in TERM_STATUSES" :key="item.value" :value="item.value">{{ item.label }}</ElRadio>
</ElRadioGroup>
<p v-if="status === 'learning'" class="subtle">学习中按 1~7 级记录,等级由复习推进。</p>
<label for="term-definition">我的释义 <span>{{ savedTermId ? '已保存' : '新词条' }}</span></label>
<ElInput id="term-definition" v-model="definition" type="textarea" :rows="3" :disabled="saving || prefilling || !!prefillError" placeholder="记下当前词的释义" />
<label for="term-examples">例句 <span>每行一条,最多 5 条</span></label>
<ElInput id="term-examples" v-model="examples" type="textarea" :rows="2" :disabled="saving || prefilling || !!prefillError" placeholder="可选,每行一条" />
<p v-if="prefilling" role="status" class="subtle">正在读取已保存的内容…</p>
<p v-else-if="prefillError" role="alert" class="lookup-message">{{ prefillError }}</p>
<p v-if="saved" role="status" class="lookup-saved">✓ {{ saved }}</p>
<p v-if="saveError" role="alert" class="lookup-message">{{ saveError }}</p>
<div class="lookup-actions">
<ElButton type="primary" data-testid="term-save" :loading="saving" :disabled="!canSave" @click="$emit('save')">保存到生词本</ElButton>
<ElButton text @click="$emit('close')">关闭,继续阅读</ElButton>
</div>
</div>
</aside>
</template>
+10
View File
@@ -0,0 +1,10 @@
<script setup lang="ts">
import { termStatusOf, type ReaderToken } from '../composables/useReaderLookup'
defineProps<{ tokens: ReaderToken[]; original: string; selectedStart?: number }>()
const emit = defineEmits<{ select: [token: ReaderToken, element: HTMLElement] }>()
function select(token: ReaderToken, event: Event) { emit('select', token, event.currentTarget as HTMLElement) }
</script>
<template>
<article class="reader-text"><template v-if="tokens.length"><template v-for="token in tokens" :key="token.start"><span v-if="token.kind === 'word'" role="button" tabindex="0" class="reader-word" :class="[termStatusOf(token) ? `is-${termStatusOf(token)}` : '', { 'is-selected': selectedStart === token.start }]" :aria-label="termStatusOf(token) ? `查询 ${token.text},已保存` : `查询 ${token.text}`" :aria-pressed="selectedStart === token.start" @click="select(token, $event)" @keydown.enter.prevent="select(token, $event)" @keydown.space.prevent="select(token, $event)">{{ token.text }}</span><template v-else>{{ token.text }}</template></template></template><template v-else>{{ original }}</template></article>
</template>
+257
View File
@@ -0,0 +1,257 @@
import { computed, onScopeDispose, ref, watch, type Ref } from 'vue'
import type { ChapterDetail } from '../stores/library'
import { useSessionStore } from '../stores/session'
export type TermStatus = 'new' | 'learning' | 'known' | 'ignored'
export interface TokenTerm { id: number; status: TermStatus; level: number }
export interface ReaderToken {
text: string
start: number
end: number
startUtf16: number
endUtf16: number
kind: 'word' | 'space' | 'punctuation'
term?: TokenTerm | null
}
export interface LookupResult {
status: 'exact' | 'lemma' | 'not_found' | 'resource_missing'
query: string
matchedForm: string | null
candidates: string[]
entries: { lemma: string; pos: string; definition: string; examples: string[] }[]
resource?: { name: string; version: string }
}
export interface SavedTerm {
id: number
term: string
originalForm: string
definition: string
examples: string[]
status: TermStatus
level: number
}
interface TermResponse { term: SavedTerm }
interface TokenResponse { textSha256: string; tokens: ReaderToken[] }
// The four learner-visible states and the level rule behind them. Only a learning
// entry carries a level, so every other status reports level 0.
export const TERM_STATUSES: { value: TermStatus; label: string }[] = [
{ value: 'new', label: '新词' },
{ value: 'learning', label: '学习中' },
{ value: 'known', label: '已知' },
{ value: 'ignored', label: '忽略' },
]
export function termStatusLabel(status: TermStatus): string {
return TERM_STATUSES.find(item => item.value === status)?.label ?? status
}
/** Unknown or malformed personal state never becomes a highlight class. */
export function termStatusOf(token: ReaderToken): TermStatus | null {
const term = token.term
if (!term || typeof term.id !== 'number' || term.id <= 0) return null
return TERM_STATUSES.some(item => item.value === term.status) ? term.status : null
}
// Validate every coordinate before enabling selection; original text is always the fallback.
function matchesChapter(data: TokenResponse, chapter: ChapterDetail): boolean {
if (!data || data.textSha256 !== chapter.contentSha256 || !Array.isArray(data.tokens)) return false
const text = chapter.originalText ?? ''
let cp = 0, utf16 = 0
for (const token of data.tokens) {
if (typeof token.text !== 'string' || !token.text || !['word', 'space', 'punctuation'].includes(token.kind)) return false
if (token.start !== cp || token.startUtf16 !== utf16) return false
cp += [...token.text].length
utf16 += token.text.length
if (token.end !== cp || token.endUtf16 !== utf16 || text.slice(token.startUtf16, utf16) !== token.text) return false
}
return utf16 === text.length && data.tokens.map(token => token.text).join('') === text
}
export function useReaderLookup(chapter: Ref<ChapterDetail | null>) {
const session = useSessionStore()
const tokens = ref<ReaderToken[]>([])
const tokensError = ref('')
const tokensLoading = ref(false)
const selected = ref<ReaderToken | null>(null)
const result = ref<LookupResult | null>(null)
const loading = ref(false)
const error = ref('')
// The learner's own record for the selected word.
const definition = ref('')
const examples = ref('')
const status = ref<TermStatus>('new')
const savedTermId = ref<number | null>(null)
const prefilling = ref(false)
const prefillError = ref('')
const saving = ref(false)
const saveError = ref('')
const saved = ref('')
let tokenSequence = 0, lookupSequence = 0, termSequence = 0, saveSequence = 0
let origin: HTMLElement | null = null
let scrollBeforeAdjustment: { top: number; left: number } | null = null
let lastAdjustment: { top: number; left: number } | null = null
// Sending is blocked until the stored text is on screen, so a slow or failed read
// can never let an empty form overwrite what the learner already wrote.
const canSave = computed(() => selected.value !== null && !saving.value && !prefilling.value && !prefillError.value)
function exampleLines(): string[] {
return examples.value.split('\n').map(line => line.trim()).filter(Boolean)
}
function applyTerm(next: TokenTerm) {
savedTermId.value = next.id
const index = tokens.value.findIndex(token => token.start === selected.value?.start)
const target = index >= 0 ? tokens.value[index] : undefined
if (target) tokens.value[index] = { ...target, term: next }
}
function stillAtAdjustment() {
return lastAdjustment !== null && Math.abs(window.scrollY - lastAdjustment.top) <= 2
&& Math.abs(window.scrollX - lastAdjustment.left) <= 2
}
function keepSelectionVisible(panelTop: number) {
if (!origin) return
const coveredBy = origin.getBoundingClientRect().bottom - panelTop + 16
if (coveredBy <= 0) return
// A later manual scroll becomes the new reading position, including when
// another resize subsequently needs to reveal the selected word again.
if (!scrollBeforeAdjustment || !stillAtAdjustment()) scrollBeforeAdjustment = { top: window.scrollY, left: window.scrollX }
window.scrollBy({ top: coveredBy, behavior: 'instant' })
lastAdjustment = { top: window.scrollY, left: window.scrollX }
}
function close(restoreFocus = true) {
lookupSequence++
termSequence++
saveSequence++
selected.value = null
result.value = null
loading.value = false
error.value = ''
definition.value = ''
examples.value = ''
status.value = 'new'
savedTermId.value = null
prefilling.value = false
prefillError.value = ''
saving.value = false
saveError.value = ''
saved.value = ''
if (restoreFocus && origin?.isConnected) origin.focus({ preventScroll: true })
if (restoreFocus && scrollBeforeAdjustment && stillAtAdjustment()) window.scrollTo({ ...scrollBeforeAdjustment, behavior: 'instant' })
scrollBeforeAdjustment = null
lastAdjustment = null
origin = null
}
function reset() {
tokenSequence++
tokens.value = []
tokensError.value = ''
tokensLoading.value = false
close(false)
}
async function loadTokens() {
const current = chapter.value
if (!session.user || current?.status !== 'ready') return
const seq = ++tokenSequence
tokensLoading.value = true
tokensError.value = ''
try {
const data = await session.request<TokenResponse>(`chapters/${current.id}/tokens`)
if (seq !== tokenSequence) return
if (!matchesChapter(data, current)) throw new Error('分词与正文不一致,请重试。')
tokens.value = data.tokens
} catch (reason) {
if (seq !== tokenSequence) return
tokens.value = []
tokensError.value = reason instanceof Error ? reason.message : '单词暂时无法加载。'
} finally { if (seq === tokenSequence) tokensLoading.value = false }
}
async function lookup() {
const current = chapter.value
const token = selected.value
if (!session.user || !current || !token) return
const seq = ++lookupSequence
result.value = null
error.value = ''
loading.value = true
try {
const data = await session.request<LookupResult>('lookup', 'POST', { chapterId: current.id, start: token.start, end: token.end })
if (seq === lookupSequence) result.value = data
} catch (reason) {
if (seq === lookupSequence) error.value = reason instanceof Error ? reason.message : '暂时无法查询,请重试。'
} finally { if (seq === lookupSequence) loading.value = false }
}
// Opening a word the learner already saved loads that record, so the panel shows
// the stored text and the same id other chapters show.
async function loadTerm(id: number) {
const seq = ++termSequence
prefilling.value = true
prefillError.value = ''
try {
const data = await session.request<TermResponse>(`terms/${id}`)
if (seq !== termSequence) return
definition.value = data.term.definition
examples.value = data.term.examples.join('\n')
status.value = data.term.status
applyTerm({ id: data.term.id, status: data.term.status, level: data.term.level })
} catch (reason) {
if (seq !== termSequence) return
prefillError.value = reason instanceof Error ? reason.message : '已保存的内容暂时无法读取。'
} finally { if (seq === termSequence) prefilling.value = false }
}
async function save() {
const current = chapter.value
const token = selected.value
if (!session.user || !current || !token || !canSave.value) return
const seq = ++saveSequence
saving.value = true
saveError.value = ''
saved.value = ''
try {
const data = await session.request<TermResponse & { created: boolean }>('terms', 'POST', {
chapterId: current.id, start: token.start, end: token.end,
definition: definition.value, examples: exampleLines(), status: status.value,
})
if (seq !== saveSequence) return
definition.value = data.term.definition
examples.value = data.term.examples.join('\n')
status.value = data.term.status
applyTerm({ id: data.term.id, status: data.term.status, level: data.term.level })
saved.value = `已保存 · ${termStatusLabel(data.term.status)}`
} catch (reason) {
if (seq !== saveSequence) return
saveError.value = reason instanceof Error ? reason.message : '保存失败,请稍后重试。'
} finally { if (seq === saveSequence) saving.value = false }
}
function select(token: ReaderToken, element: HTMLElement) {
close(false)
origin = element
selected.value = token
const existing = termStatusOf(token)
if (existing !== null && token.term) {
status.value = existing
void loadTerm(token.term.id)
}
void lookup()
}
watch(() => [chapter.value?.id, chapter.value?.status, chapter.value?.contentSha256, chapter.value?.originalText], () => {
reset()
void loadTokens()
}, { immediate: true, flush: 'sync' })
// Watch the identity object, including clear → login for the same account: leaving
// or switching an account drops every word saved by the previous one.
watch(() => session.user, reset, { flush: 'sync' })
onScopeDispose(reset)
return {
tokens, tokensError, tokensLoading, selected, result, loading, error,
definition, examples, status, savedTermId, prefilling, prefillError, saving, saveError, saved, canSave,
loadTokens, lookup, loadTerm, save, select, close, reset, keepSelectionVisible,
}
}
+36
View File
@@ -95,6 +95,38 @@ a.chapter-name:hover { color: #315c43; text-decoration: underline; }
.reader-text { white-space: pre-wrap; overflow-wrap: break-word; margin: 26px 0 0; font-family: Georgia, 'Times New Roman', 'Microsoft YaHei', serif; font-size: 17px; line-height: 2; }
.processing-hint { padding: 36px 0; color: #748073; }
.reader-nav { display: flex; align-items: center; justify-content: space-between; gap: 14px; flex-wrap: wrap; margin-top: 36px; padding-top: 22px; border-top: 1px solid #e0e3d8; }
.reader-page.has-lookup { max-width: 1120px; }
.reader-workspace { display: grid; grid-template-columns: minmax(0, 1fr); gap: 32px; align-items: start; }
.has-lookup .reader-workspace { grid-template-columns: minmax(0, 1fr) 320px; }
.reader-body { min-width: 0; }
.reader-word { cursor: pointer; border-radius: 3px; }
/* Saved words keep one style per status everywhere they appear. */
.reader-word.is-new { border-bottom: 2px dotted #bc803d; }
.reader-word.is-learning { background: #f6e7c9; }
.reader-word.is-known { background: #dee9d6; }
.reader-word.is-ignored { color: #949c94; }
.reader-word:hover, .reader-word.is-selected { background: #e3e9d9; color: #264a35; }
.reader-word:focus-visible { outline: 2px solid #bc803d; outline-offset: 2px; background: #eef2eb; }
.tokens-notice { color: #748073; font-size: 14px; margin-top: 24px; }
.lookup-panel { position: sticky; top: 24px; margin-top: 26px; padding: 22px; border: 1px solid #d9decf; border-radius: 12px; background: #fffdf8; max-height: calc(100dvh - 48px); overflow-y: auto; overflow-wrap: anywhere; }
.lookup-heading { display: flex; align-items: start; justify-content: space-between; gap: 10px; border-bottom: 1px solid #e0e3d8; padding-bottom: 12px; }
.lookup-heading h2 { margin: 6px 0; font-family: Georgia, serif; font-size: 25px; }
.lookup-content { font-size: 15px; line-height: 1.7; }
.lookup-message { color: #8b4324; }
.lookup-senses { padding-left: 22px; }
.lookup-senses li { padding-left: 3px; margin-bottom: 20px; }
.lookup-senses p { margin: 8px 0; }
.sense-heading span { color: #748073; font-size: 13px; }
.lookup-senses blockquote { border-left: 2px solid #cbd9c9; margin: 10px 0; padding-left: 12px; color: #687568; font-style: italic; }
.lookup-term { border-top: 1px solid #e0e3d8; padding-top: 18px; margin-top: 20px; }
.lookup-term label { display: flex; justify-content: space-between; gap: 10px; font-size: 14px; margin-top: 14px; }
.lookup-term-title { display: flex; font-size: 14px; margin: 0; }
.lookup-term label span { color: #8b794e; font-size: 12px; }
.lookup-term .el-radio-group { margin-top: 8px; flex-wrap: wrap; gap: 4px 12px; }
.lookup-term .el-textarea { margin-top: 8px; }
.lookup-term .el-textarea textarea { font: inherit; line-height: 1.6; }
.lookup-saved { color: #2f6b45; font-size: 14px; margin: 12px 0 0; }
.lookup-actions { display: flex; gap: 8px; margin-top: 14px; flex-wrap: wrap; }
@media (max-width: 760px) {
.login-page { grid-template-columns: 1fr; }
.welcome { padding: 28px; }
@@ -116,5 +148,9 @@ a.chapter-name:hover { color: #315c43; text-decoration: underline; }
.chapter-row { grid-template-columns: 30px minmax(0, 1fr); align-items: start; row-gap: 10px; padding: 15px 16px; }
.chapter-row .status-chip, .chapter-row .el-button { grid-column: 2; justify-self: start; }
.reader-text { font-size: 16px; line-height: 1.95; }
.has-lookup .reader-workspace { grid-template-columns: minmax(0, 1fr); gap: 20px; }
.reader-page.has-lookup { padding-bottom: calc(45dvh + 28px); }
.lookup-panel { position: fixed; inset: auto 0 0; z-index: 20; max-height: 45dvh; margin-top: 0; padding: 16px 20px max(20px, env(safe-area-inset-bottom)); border-radius: 16px 16px 0 0; box-shadow: 0 -5px 24px #233d3114; }
.lookup-heading { position: sticky; top: -16px; z-index: 1; background: #fffdf8; }
.reader-nav .el-button { flex: 1; }
}
+16 -2
View File
@@ -4,6 +4,9 @@ import { RouterLink, useRoute, useRouter } from 'vue-router'
import { ElButton } from 'element-plus'
import { canRetry, statusLabel, useLibraryStore } from '../stores/library'
import { useSessionStore } from '../stores/session'
import { useReaderLookup } from '../composables/useReaderLookup'
import ReaderTokens from '../components/ReaderTokens.vue'
import LookupPanel from '../components/LookupPanel.vue'
const session = useSessionStore()
const library = useLibraryStore()
@@ -13,10 +16,12 @@ const retryError = ref('')
const chapterId = computed(() => Number(route.params.id))
const chapter = computed(() => library.chapter)
const lookup = useReaderLookup(chapter)
// Retry uses the job id the chapter carries, no matter where it was loaded from.
const retryable = computed(() => chapter.value !== null && canRetry(chapter.value))
async function load() {
lookup.reset()
retryError.value = ''
await library.loadChapter(chapterId.value)
}
@@ -56,7 +61,7 @@ onUnmounted(() => library.closeChapter())
<ElButton text @click="logout">退出登录</ElButton>
</div>
</header>
<main class="page reader-page">
<main class="page reader-page" :class="{ 'has-lookup': lookup.selected.value }" @keydown.esc="lookup.close()">
<p v-if="library.chapterLoading && !chapter" role="status" class="loading">正在加载…</p>
<div v-else-if="library.chapterError" class="notice">
<p role="alert">{{ library.chapterError }}</p>
@@ -80,7 +85,16 @@ onUnmounted(() => library.closeChapter())
</div>
<p v-else-if="chapter.status !== 'ready'" role="status" class="processing-hint">这一章还在{{ statusLabel(chapter.status) }},页面会自动刷新。</p>
<p v-if="retryError" role="alert" class="notice">{{ retryError }}</p>
<article v-if="chapter.status === 'ready'" class="reader-text">{{ library.readerText }}</article>
<div v-if="chapter.status === 'ready'" class="reader-workspace">
<div class="reader-body">
<ReaderTokens :tokens="lookup.tokens.value" :original="library.readerText" :selected-start="lookup.selected.value?.start" @select="lookup.select" />
<div v-if="lookup.tokensError.value" class="tokens-notice">
<p role="status">{{ lookup.tokensError.value }}</p>
<ElButton data-testid="tokens-retry" :loading="lookup.tokensLoading.value" @click="lookup.loadTokens">重试加载单词</ElButton>
</div>
</div>
<LookupPanel v-if="lookup.selected.value" :word="lookup.selected.value.text" :result="lookup.result.value" :loading="lookup.loading.value" :error="lookup.error.value" :saving="lookup.saving.value" :save-error="lookup.saveError.value" :saved="lookup.saved.value" :saved-term-id="lookup.savedTermId.value" :prefilling="lookup.prefilling.value" :prefill-error="lookup.prefillError.value" :can-save="lookup.canSave.value" v-model:definition="lookup.definition.value" v-model:examples="lookup.examples.value" v-model:status="lookup.status.value" @close="lookup.close()" @retry="lookup.lookup" @save="lookup.save" @resize="lookup.keepSelectionVisible" />
</div>
<nav class="reader-nav" aria-label="章节切换">
<ElButton :disabled="library.navigation.previousChapterId === null" @click="goTo(library.navigation.previousChapterId)">上一章</ElButton>
<ElButton :disabled="library.navigation.nextChapterId === null" @click="goTo(library.navigation.nextChapterId)">下一章</ElButton>
+31
View File
@@ -0,0 +1,31 @@
WordNet Release 3.0
This software and database is being provided to you, the LICENSEE, by
Princeton University under the following license. By obtaining, using
and/or copying this software and database, you agree that you have
read, understood, and will comply with these terms and conditions.:
Permission to use, copy, modify and distribute this software and
database and its documentation for any purpose and without fee or
royalty is hereby granted, provided that you agree to comply with
the following copyright notice and statements, including the disclaimer,
and that the same appear on ALL copies of the software, database and
documentation, including modifications that you make for internal
use or for distribution.
WordNet 3.0 Copyright 2006 by Princeton University. All rights reserved.
THIS SOFTWARE AND DATABASE IS PROVIDED "AS IS" AND PRINCETON
UNIVERSITY MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PRINCETON
UNIVERSITY MAKES NO REPRESENTATIONS OR WARRANTIES OF MERCHANT-
ABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE OR THAT THE USE
OF THE LICENSED SOFTWARE, DATABASE OR DOCUMENTATION WILL NOT
INFRINGE ANY THIRD PARTY PATENTS, COPYRIGHTS, TRADEMARKS OR
OTHER RIGHTS.
The name of Princeton University or Princeton may not be used in
advertising or publicity pertaining to distribution of the software
and/or database. Title to copyright in this software, database and
any associated documentation shall at all times remain with
Princeton University and LICENSEE agrees to preserve same.
+44 -4
View File
@@ -60,7 +60,7 @@ func Migrate(db *gorm.DB) error {
if err = conn.QueryRowContext(ctx, "SELECT version,product FROM lexgo_schema WHERE id=1").Scan(&current, &product); err != nil {
return err
}
if product != "lexgo" || current < 0 || current > schemaVersion {
if product != "lexgo" || current < 0 || current > SchemaVersion {
return errors.New("unknown schema version")
}
// Each known version contributes its own statements; the version row advances only
@@ -72,6 +72,12 @@ func Migrate(db *gorm.DB) error {
if current < 3 {
statements = append(statements, schemaV3Statements...)
}
if current < 4 {
statements = append(statements, schemaV4Statements...)
}
if current < 5 {
statements = append(statements, schemaV5Statements...)
}
for i, s := range statements {
if _, err = conn.ExecContext(ctx, s); err != nil {
var sqlErr *driver.MySQLError
@@ -81,11 +87,45 @@ func Migrate(db *gorm.DB) error {
return fmt.Errorf("migration statement %d failed", i+1)
}
}
_, err = conn.ExecContext(ctx, fmt.Sprintf("UPDATE lexgo_schema SET version=%d WHERE id=1", schemaVersion))
_, err = conn.ExecContext(ctx, fmt.Sprintf("UPDATE lexgo_schema SET version=%d WHERE id=1", SchemaVersion))
return err
}
const schemaVersion = 3
// SchemaVersion is the version an explicit migration leaves behind, and the
// version the server requires before it starts.
const SchemaVersion = 5
// v5 adds one learner's own records for a word form. Identity is the normalized
// form under the owner's language, so re-saving the same word updates one row
// instead of creating a second, conflicting record. utf8mb4_bin keeps the key
// byte-exact: the Go side normalizes, and an accent-insensitive collation must
// not fold "resume" and "résumé" into the same entry.
var schemaV5Statements = []string{
`CREATE TABLE IF NOT EXISTS lexgo_terms (
id BIGINT UNSIGNED PRIMARY KEY AUTO_INCREMENT, owner_id BIGINT NOT NULL,
language VARCHAR(16) NOT NULL DEFAULT 'en',
term VARCHAR(128) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
original_form VARCHAR(191) NOT NULL,
definition TEXT NOT NULL, examples TEXT NOT NULL,
status VARCHAR(16) NOT NULL, level TINYINT NOT NULL DEFAULT 0,
created_at DATETIME(3) NOT NULL, updated_at DATETIME(3) NOT NULL,
UNIQUE KEY uq_term_identity (owner_id, language, term),
CONSTRAINT ck_term_status CHECK (status IN ('new','learning','known','ignored')),
CONSTRAINT ck_term_level CHECK ((status = 'learning' AND level BETWEEN 1 AND 7) OR (status <> 'learning' AND level = 0)),
FOREIGN KEY (owner_id) REFERENCES sys_user(user_id) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`,
}
var schemaV4Statements = []string{
`CREATE TABLE IF NOT EXISTS lexgo_dictionaries (
id BIGINT PRIMARY KEY, name VARCHAR(120) NOT NULL, language VARCHAR(16) NOT NULL,
version VARCHAR(32) NOT NULL, source VARCHAR(512) NOT NULL, format VARCHAR(32) NOT NULL,
sha256 CHAR(64) CHARACTER SET ascii COLLATE ascii_bin NOT NULL,
entry_count INT NOT NULL, enabled BOOLEAN NOT NULL DEFAULT TRUE,
archive LONGBLOB NOT NULL, updated_at DATETIME(3) NOT NULL,
CHECK (id = 1), CHECK (language = 'en')
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`,
}
var schemaV2Statements = []string{
`CREATE TABLE IF NOT EXISTS sys_user (
@@ -163,7 +203,7 @@ func CheckSchema(db *gorm.DB) error {
Version int
Product string
}
if err := db.Table("lexgo_schema").Where("id=1").First(&r).Error; err != nil || r.Version != schemaVersion || r.Product != "lexgo" {
if err := db.Table("lexgo_schema").Where("id=1").First(&r).Error; err != nil || r.Version != SchemaVersion || r.Product != "lexgo" {
return errors.New("run the explicit migration before starting")
}
return nil
+324
View File
@@ -0,0 +1,324 @@
package lexgo
import (
"errors"
"io"
"net/http"
"strconv"
"strings"
"sync"
"time"
"unicode"
"unicode/utf8"
"github.com/gin-gonic/gin"
admin "go-admin/app/admin/models"
"gorm.io/gorm"
"gorm.io/gorm/clause"
)
// ID 1 is the sole English corpus slot. Keeping the validated ZIP in MySQL
// makes ordinary database backups include the resource needed after a restart.
type DictionaryResource struct {
ID int64 `gorm:"primaryKey;autoIncrement:false"`
Name string
Language string
Version string
Source string
Format string
SHA256 string `gorm:"column:sha256"`
EntryCount int
Enabled bool
Archive []byte
UpdatedAt time.Time
}
func (DictionaryResource) TableName() string { return "lexgo_dictionaries" }
type DictionaryView struct {
ID int64 `json:"id"`
Name string `json:"name"`
Language string `json:"language"`
Version string `json:"version"`
Source string `json:"source"`
Format string `json:"format"`
Status string `json:"status"`
Enabled bool `json:"enabled"`
SHA256 string `json:"sha256"`
EntryCount int `json:"entryCount"`
UpdatedAt time.Time `json:"updatedAt"`
}
func dictionaryView(r DictionaryResource, status string) DictionaryView {
return DictionaryView{r.ID, r.Name, r.Language, r.Version, r.Source, r.Format, status, r.Enabled, r.SHA256, r.EntryCount, r.UpdatedAt}
}
type DictionaryImportResult struct {
Resource DictionaryView `json:"resource"`
Duplicate bool `json:"duplicate"`
}
type ChapterTokens struct {
TextSHA256 string `json:"textSha256"`
Tokens []TextToken `json:"tokens"`
}
// Each router keeps at most one immutable parsed corpus; no private chapter or
// lookup data enters the cache. The mutex also coalesces simultaneous cold loads.
type dictionaryCache struct {
mu sync.Mutex
sha string
engine *WordNet
}
func (cache *dictionaryCache) load(tx *gorm.DB, r DictionaryResource) (*WordNet, error) {
cache.mu.Lock()
defer cache.mu.Unlock()
if r.SHA256 != WordNetSHA {
return nil, errors.New("unsupported resource checksum")
}
if cache.sha == r.SHA256 && cache.engine != nil {
return cache.engine, nil
}
var stored DictionaryResource
if err := tx.Select("id", "archive").Where("id = ? AND sha256 = ?", r.ID, r.SHA256).First(&stored).Error; err != nil {
return nil, err
}
engine, err := ParseWordNet(stored.Archive)
if err != nil {
return nil, err
}
cache.sha = r.SHA256
cache.engine = engine
return engine, nil
}
func readyOwnedChapter(tx *gorm.DB, owner int, id int64) (Chapter, error) {
var chapter Chapter
err := tx.Where("id = ? AND owner_id = ?", id, owner).First(&chapter).Error
if errors.Is(err, gorm.ErrRecordNotFound) {
return chapter, failure(404, "章节不存在")
}
if err != nil {
return chapter, err
}
if chapter.Status != statusReady {
return chapter, failure(409, "章节尚未就绪,请稍后重试")
}
return chapter, nil
}
func readDictionaryUpload(c *gin.Context) (DictionaryResource, error) {
bad := failure(400, "词典上传无效,请使用指定的 WordNet 3.0 ZIP 和完整资源信息")
c.Request.Body = http.MaxBytesReader(c.Writer, c.Request.Body, maxDictionaryZip+(64<<10))
reader, err := c.Request.MultipartReader()
if err != nil {
return DictionaryResource{}, bad
}
fields := map[string]string{}
var archive []byte
for {
part, err := reader.NextPart()
if err == io.EOF {
break
}
if err != nil {
return DictionaryResource{}, bad
}
name := part.FormName()
if name == "file" {
if archive != nil || part.FileName() == "" {
part.Close()
return DictionaryResource{}, bad
}
archive, err = io.ReadAll(io.LimitReader(part, maxDictionaryZip+1))
if err != nil || len(archive) == 0 || len(archive) > maxDictionaryZip {
part.Close()
return DictionaryResource{}, bad
}
} else {
if part.FileName() != "" || (name != "name" && name != "language" && name != "version" && name != "source" && name != "format") {
part.Close()
return DictionaryResource{}, bad
}
if _, exists := fields[name]; exists {
part.Close()
return DictionaryResource{}, bad
}
value, e := io.ReadAll(io.LimitReader(part, 1025))
if e != nil || len(value) > 1024 || !utf8.Valid(value) {
part.Close()
return DictionaryResource{}, bad
}
fields[name] = string(value)
}
part.Close()
}
name := strings.TrimSpace(fields["name"])
if name == "" || utf8.RuneCountInString(name) > 120 || strings.IndexFunc(name, unicode.IsControl) >= 0 || fields["language"] != "en" || fields["version"] != "3.0" || fields["format"] != "wordnet-3.0-zip" || fields["source"] != WordNetSource || len(archive) == 0 {
return DictionaryResource{}, bad
}
return DictionaryResource{ID: 1, Name: name, Language: "en", Version: "3.0", Source: WordNetSource, Format: "wordnet-3.0-zip", SHA256: WordNetSHA, Enabled: true, Archive: archive}, nil
}
func registerDictionaryRoutes(v *gin.RouterGroup, protect func(bool, func(*gin.Context, *gorm.DB, admin.SysUser) (any, error)) gin.HandlerFunc, now func() time.Time) {
cache := &dictionaryCache{}
uploadGate := make(chan struct{}, 1)
v.GET("/dictionaries", protect(true, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) {
if c.Request.URL.RawQuery != "" {
return nil, failure(400, "词典列表不接受查询参数")
}
var rows []DictionaryResource
if err := tx.Omit("archive").Order("id").Find(&rows).Error; err != nil {
return nil, err
}
items := make([]DictionaryView, 0, len(rows))
for _, row := range rows {
status := "disabled"
if row.Enabled {
status = "ready"
if _, err := cache.load(tx, row); err != nil {
status = "unavailable"
}
}
items = append(items, dictionaryView(row, status))
}
return gin.H{"items": items, "supported": gin.H{"name": "Princeton WordNet", "language": "en", "version": "3.0", "format": "wordnet-3.0-zip", "source": WordNetSource, "sha256": WordNetSHA}}, nil
}))
v.POST("/dictionaries/import", protect(true, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) {
select {
case uploadGate <- struct{}{}:
defer func() { <-uploadGate }()
default:
return nil, failure(429, "已有词典正在导入,请稍后重试")
}
resource, err := readDictionaryUpload(c)
if err != nil {
return nil, err
}
engine, err := ParseWordNet(resource.Archive)
if err != nil {
return nil, failure(400, "词典文件校验失败,请选择指定的 WordNet 3.0 ZIP")
}
resource.EntryCount = engine.EntryCount
resource.UpdatedAt = stamp(now())
// INSERT ... ON CONFLICT followed by a locking read serializes even the first
// concurrent import. A validated replacement and its metadata commit together.
created := tx.Clauses(clause.OnConflict{DoNothing: true}).Create(&resource)
if created.Error != nil {
return nil, created.Error
}
var existing DictionaryResource
if err = tx.Omit("archive").Clauses(clause.Locking{Strength: "UPDATE"}).Where("id = 1").First(&existing).Error; err != nil {
return nil, err
}
duplicate := created.RowsAffected == 0 && existing.SHA256 == resource.SHA256
if created.RowsAffected == 0 {
if err = tx.Model(&DictionaryResource{}).Where("id = 1").Select("name", "language", "version", "source", "format", "sha256", "entry_count", "enabled", "archive", "updated_at").Updates(&resource).Error; err != nil {
return nil, err
}
existing = resource
}
status := "ready"
if !existing.Enabled {
status = "disabled"
}
return DictionaryImportResult{dictionaryView(existing, status), duplicate}, nil
}))
v.PATCH("/dictionaries/:id", protect(true, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) {
id, err := strconv.ParseInt(c.Param("id"), 10, 64)
if err != nil || id != 1 {
return nil, failure(404, "词典不存在")
}
var input struct {
Enabled *bool `json:"enabled"`
}
if err := decode(c, &input); err != nil {
return nil, err
}
if input.Enabled == nil {
return nil, failure(400, "请指定词典启用状态")
}
var resource DictionaryResource
if err = tx.Omit("archive").Clauses(clause.Locking{Strength: "UPDATE"}).Where("id = ?", id).First(&resource).Error; errors.Is(err, gorm.ErrRecordNotFound) {
return nil, failure(404, "词典不存在")
} else if err != nil {
return nil, err
}
if *input.Enabled {
if _, err = cache.load(tx, resource); err != nil {
return nil, failure(409, "词典资源不可用,请重新导入")
}
}
resource.Enabled = *input.Enabled
resource.UpdatedAt = stamp(now())
if err = tx.Model(&resource).Updates(map[string]any{"enabled": resource.Enabled, "updated_at": resource.UpdatedAt}).Error; err != nil {
return nil, err
}
status := "disabled"
if resource.Enabled {
status = "ready"
}
return gin.H{"resource": dictionaryView(resource, status)}, nil
}))
v.GET("/chapters/:id/tokens", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) {
if c.Request.URL.RawQuery != "" {
return nil, failure(400, "分词接口不接受查询参数")
}
id, err := strconv.ParseInt(c.Param("id"), 10, 64)
if err != nil || id <= 0 {
return nil, failure(404, "章节不存在")
}
chapter, err := readyOwnedChapter(tx, u.UserId, id)
if err != nil {
return nil, err
}
tokens := Tokenize(chapter.OriginalText)
language, err := languageOf(tx, u.UserId)
if err != nil {
return nil, err
}
// The learner's own saved words are part of the chapter view, so the reader
// shows the same status here as everywhere else.
if err = attachTerms(tx, u.UserId, language, tokens); err != nil {
return nil, err
}
return ChapterTokens{chapter.ContentSHA256, tokens}, nil
}))
v.POST("/lookup", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) {
var input struct {
ChapterID int64 `json:"chapterId"`
Start *int `json:"start"`
End *int `json:"end"`
}
if err := decode(c, &input); err != nil {
return nil, err
}
chapter, err := readyOwnedChapter(tx, u.UserId, input.ChapterID)
if err != nil {
return nil, err
}
if input.Start == nil || input.End == nil {
return nil, failure(400, "请选择完整单词")
}
query, err := wordAtRange(chapter, *input.Start, *input.End)
if err != nil {
return nil, err
}
missing := LookupResult{Status: "resource_missing", Query: query, Candidates: []string{}, Entries: []DictionaryEntry{}}
var resource DictionaryResource
err = tx.Omit("archive").Where("id = 1 AND enabled = ?", true).First(&resource).Error
if errors.Is(err, gorm.ErrRecordNotFound) {
return missing, nil
}
if err != nil {
return nil, err
}
engine, err := cache.load(tx, resource)
if err != nil {
return missing, nil
}
result := engine.Lookup(query)
result.Resource = &LookupResource{Name: resource.Name, Version: resource.Version}
return result, nil
}))
}
+182
View File
@@ -0,0 +1,182 @@
package lexgo
import (
"bytes"
"encoding/json"
"fmt"
"mime/multipart"
"net/http/httptest"
"os"
"testing"
"time"
"github.com/gin-gonic/gin"
admin "go-admin/app/admin/models"
)
func importDictionaryAPI(t *testing.T, r *gin.Engine, token string, raw []byte) (int, DictionaryImportResult) {
t.Helper()
var body bytes.Buffer
w := multipart.NewWriter(&body)
for k, v := range map[string]string{"name": "Princeton WordNet", "language": "en", "version": "3.0", "source": WordNetSource, "format": "wordnet-3.0-zip"} {
if err := w.WriteField(k, v); err != nil {
t.Fatal(err)
}
}
p, err := w.CreateFormFile("file", "wordnet.zip")
if err != nil {
t.Fatal(err)
}
p.Write(raw)
w.Close()
req := httptest.NewRequest("POST", "/api/v1/dictionaries/import", &body)
req.Header.Set("Content-Type", w.FormDataContentType())
req.Header.Set("Authorization", "Bearer "+token)
response := httptest.NewRecorder()
r.ServeHTTP(response, req)
var e envelope
json.Unmarshal(response.Body.Bytes(), &e)
var result DictionaryImportResult
json.Unmarshal(e.Data, &result)
return response.Code, result
}
func TestDictionaryAPIResourcesAndOwnership(t *testing.T) {
db := testDB(t)
raw, err := os.ReadFile("../../../.local/nlp-resources/wordnet.zip")
if os.IsNotExist(err) {
t.Skip("prepare WordNet fixture")
}
if err != nil {
t.Fatal(err)
}
// Test-only dedicated database slot; no development/production resources touched.
if err := db.Exec("DELETE FROM lexgo_dictionaries").Error; err != nil {
t.Fatal(err)
}
t.Cleanup(func() { db.Exec("DELETE FROM lexgo_dictionaries") })
r := Router(db, time.Now)
users := []admin.SysUser{}
tokens := []string{}
for i, role := range []int{1, 2, 2} {
u := admin.SysUser{Username: randomName(fmt.Sprintf("dict%d", i)), Password: fixturePassword, RoleId: role, Status: "2"}
if err := db.Create(&u).Error; err != nil {
t.Fatal(err)
}
users = append(users, u)
tokens = append(tokens, loginToken(t, r, u.Username, fixturePassword))
}
book := Book{OwnerID: users[1].UserId, Title: "Fictional", Language: "en", CreatedAt: stamp(time.Now()), UpdatedAt: stamp(time.Now())}
if err := db.Create(&book).Error; err != nil {
t.Fatal(err)
}
original := "😀 Dogs went. Cafe\u0301 123"
chapter := Chapter{BookID: book.ID, OwnerID: users[1].UserId, Ordinal: 1, Title: "Fictional", OriginalText: original, ContentSHA256: contentSHA(original), CharCount: len([]rune(original)), Status: statusReady, CreatedAt: stamp(time.Now()), UpdatedAt: stamp(time.Now())}
if err := db.Create(&chapter).Error; err != nil {
t.Fatal(err)
}
lookup := func(token string, start, end int) (int, LookupResult) {
code, data := callAPI(t, r, "POST", "/api/v1/lookup", token, map[string]any{"chapterId": chapter.ID, "start": start, "end": end})
var got LookupResult
json.Unmarshal(data, &got)
return code, got
}
if code, _ := callAPI(t, r, "GET", "/api/v1/dictionaries", tokens[1], nil); code != 403 {
t.Fatal("learner resource list", code)
}
if code, _ := importDictionaryAPI(t, r, tokens[1], []byte("bad")); code != 403 {
t.Fatal("learner import", code)
}
if code, _ := lookup(tokens[2], 2, 6); code != 404 {
t.Fatal("foreign lookup", code)
}
if code, _ := callAPI(t, r, "GET", fmt.Sprintf("/api/v1/chapters/%d/tokens", chapter.ID), tokens[2], nil); code != 404 {
t.Fatal("foreign tokens", code)
}
if code, got := lookup(tokens[1], 2, 6); code != 200 || got.Status != "resource_missing" {
t.Fatal("missing", code, got)
}
code, data := callAPI(t, r, "GET", fmt.Sprintf("/api/v1/chapters/%d/tokens", chapter.ID), tokens[1], nil)
var analyzed ChapterTokens
json.Unmarshal(data, &analyzed)
if code != 200 || analyzed.TextSHA256 != contentSHA(original) || analyzed.Tokens[2].StartUtf16 != 3 {
t.Fatal("tokens", code, string(data))
}
for _, span := range [][2]int{{-1, 3}, {2, 5}, {0, 1}, {2, 11}, {18, 21}, {1, 2}, {6, 2}} {
if code, _ := lookup(tokens[1], span[0], span[1]); code != 400 {
t.Fatal("invalid interval", span, code)
}
}
code, imported := importDictionaryAPI(t, r, tokens[0], raw)
if code != 200 || imported.Duplicate || imported.Resource.Status != "ready" {
t.Fatal("import", code, imported)
}
if code, got := lookup(tokens[1], 7, 11); code != 200 || got.Status != "lemma" || *got.MatchedForm != "go" {
t.Fatal("went", code, got)
}
code, duplicate := importDictionaryAPI(t, r, tokens[0], raw)
if code != 200 || !duplicate.Duplicate || duplicate.Resource.ID != imported.Resource.ID {
t.Fatal("duplicate", code, duplicate)
}
if code, _ := importDictionaryAPI(t, r, tokens[0], []byte("broken")); code != 400 {
t.Fatal("bad import", code)
}
if code, got := lookup(tokens[1], 2, 6); code != 200 || got.Status != "lemma" {
t.Fatal("failed import lost old resource", code, got)
}
endpoint := fmt.Sprintf("/api/v1/dictionaries/%d", imported.Resource.ID)
if code, _ := callAPI(t, r, "PATCH", endpoint, tokens[1], map[string]bool{"enabled": false}); code != 403 {
t.Fatal("learner toggle", code)
}
for _, enabled := range []bool{false, true} {
if code, _ := callAPI(t, r, "PATCH", endpoint, tokens[0], map[string]bool{"enabled": enabled}); code != 200 {
t.Fatal("toggle", code)
}
code, got := lookup(tokens[1], 2, 6)
if code != 200 || (enabled && got.Status != "lemma") || (!enabled && got.Status != "resource_missing") {
t.Fatal("enabled state", enabled, code, got)
}
}
r = Router(db, time.Now) // A new router has an empty cache and reloads the persisted ZIP.
if code, got := lookup(tokens[1], 2, 6); code != 200 || got.Status != "lemma" {
t.Fatal("cold restart", code, got)
}
if code, _ := callAPI(t, r, "PATCH", endpoint, tokens[0], map[string]bool{"enabled": false}); code != 200 {
t.Fatal("disable before duplicate", code)
}
if code, result := importDictionaryAPI(t, r, tokens[0], raw); code != 200 || !result.Duplicate || !result.Resource.Enabled {
t.Fatal("duplicate must re-enable", code, result)
}
var count int64
if err := db.Model(&DictionaryResource{}).Count(&count).Error; err != nil || count != 1 {
t.Fatal("duplicate created extra resource", count, err)
}
if code, _ := callAPI(t, r, "PATCH", endpoint, tokens[0], map[string]any{"enabled": nil}); code != 400 {
t.Fatal("null toggle", code)
}
if code, _ := callAPI(t, r, "POST", "/api/v1/lookup", tokens[1], map[string]any{"chapterId": chapter.ID, "start": 2, "end": 6, "ownerId": users[2].UserId}); code != 400 {
t.Fatal("unknown lookup input", code)
}
if err := db.Model(&DictionaryResource{}).Where("id=1").Update("archive", []byte("corrupt fixture")).Error; err != nil {
t.Fatal(err)
}
r = Router(db, time.Now)
if code, got := lookup(tokens[1], 2, 6); code != 200 || got.Status != "resource_missing" {
t.Fatal("corrupt cold resource", code, got)
}
if code, got := callAPI(t, r, "GET", "/api/v1/dictionaries", tokens[0], nil); code != 200 || !bytes.Contains(got, []byte(`"status":"unavailable"`)) {
t.Fatal("corrupt resource state", code, string(got))
}
if code, result := importDictionaryAPI(t, r, tokens[0], raw); code != 200 || !result.Duplicate {
t.Fatal("repair reimport", code, result)
}
if code, got := lookup(tokens[1], 2, 6); code != 200 || got.Status != "lemma" {
t.Fatal("repair lookup", code, got)
}
if err := db.Model(&chapter).Update("status", statusPending).Error; err != nil {
t.Fatal(err)
}
if code, _ := lookup(tokens[1], 2, 6); code != 409 {
t.Fatal("pending lookup", code)
}
}
+125 -3
View File
@@ -136,8 +136,8 @@ func TestMigrationFromV2PreservesExistingData(t *testing.T) {
}
var version int
db.Raw("SELECT version FROM lexgo_schema WHERE id=1").Scan(&version)
if version != 3 {
t.Fatalf("schema version %d after re-upgrade, want 3", version)
if version != SchemaVersion {
t.Fatalf("schema version %d after re-upgrade, want %d", version, SchemaVersion)
}
db.Table("sys_user").Where("user_id = 7").Count(&users)
if users != 1 {
@@ -145,11 +145,83 @@ func TestMigrationFromV2PreservesExistingData(t *testing.T) {
}
}
func TestMigrationFromV4KeepsLibraryAndAddsPersonalTerms(t *testing.T) {
db := emptyMigrationDB(t)
statements := []string{"CREATE TABLE lexgo_schema (id INT PRIMARY KEY,version INT,product VARCHAR(32))", "INSERT INTO lexgo_schema VALUES (1,4,'lexgo')"}
statements = append(statements, schemaV2Statements...)
statements = append(statements, schemaV3Statements...)
statements = append(statements, schemaV4Statements...)
for _, statement := range statements {
if err := db.Exec(statement).Error; err != nil {
t.Fatal(err)
}
}
if err := db.Exec("INSERT INTO sys_user (user_id,username,password,role_id) VALUES (9,'fixture_v4','fictional-not-a-real-hash',2)").Error; err != nil {
t.Fatal(err)
}
now := stamp(time.Now())
book := Book{OwnerID: 9, Title: "Fictional migration", Language: "en", CreatedAt: now, UpdatedAt: now}
if err := db.Create(&book).Error; err != nil {
t.Fatal(err)
}
original := "😀 Original e\u0301\r\n"
chapter := Chapter{BookID: book.ID, OwnerID: 9, Ordinal: 1, Title: "Fixture", OriginalText: original, ContentSHA256: contentSHA(original), Status: statusReady, CharCount: len([]rune(original)), CreatedAt: now, UpdatedAt: now}
if err := db.Create(&chapter).Error; err != nil {
t.Fatal(err)
}
resource := DictionaryResource{ID: 1, Name: "Fixture WordNet", Language: "en", Version: "3.0", Source: WordNetSource, Format: "wordnet-3.0-zip", SHA256: strings.Repeat("a", 64), EntryCount: 5, Enabled: true, Archive: []byte("fixture archive"), UpdatedAt: now}
if err := db.Create(&resource).Error; err != nil {
t.Fatal(err)
}
if err := CheckSchema(db); err == nil {
t.Fatal("old schema accepted before explicit migration")
}
if err := Migrate(db); err != nil {
t.Fatal(err)
}
if err := CheckSchema(db); err != nil {
t.Fatal(err)
}
var restored Chapter
if err := db.First(&restored, chapter.ID).Error; err != nil || restored.OriginalText != original || restored.ContentSHA256 != chapter.ContentSHA256 {
t.Fatal("migration changed original chapter", err)
}
var restoredResource DictionaryResource
if err := db.Omit("archive").First(&restoredResource, 1).Error; err != nil || restoredResource.SHA256 != resource.SHA256 || restoredResource.EntryCount != 5 {
t.Fatal("migration changed the shared dictionary resource", err)
}
var terms int64
if err := db.Model(&Term{}).Count(&terms).Error; err != nil || terms != 0 {
t.Fatal("migration must create an empty personal term table", terms, err)
}
// The new table is usable and enforces the documented status boundary.
valid := Term{OwnerID: 9, Language: "en", Term: "curiosity", OriginalForm: "Curiosity", Status: termStatusNew, CreatedAt: now, UpdatedAt: now}
if err := db.Create(&valid).Error; err != nil {
t.Fatal(err)
}
if err := db.Create(&Term{OwnerID: 9, Language: "en", Term: "forged", OriginalForm: "forged", Status: "deleted", CreatedAt: now, UpdatedAt: now}).Error; err == nil {
t.Fatal("an unknown status must be rejected by the schema check")
}
if err := db.Create(&Term{OwnerID: 9, Language: "en", Term: "leveled", OriginalForm: "leveled", Status: termStatusKnown, Level: 5, CreatedAt: now, UpdatedAt: now}).Error; err == nil {
t.Fatal("a level outside 0~7 must be rejected by the schema check")
}
// Rolling the marker back for a binary rollback and upgrading again keeps rows.
if err := db.Exec("UPDATE lexgo_schema SET version=4 WHERE id=1").Error; err != nil {
t.Fatal(err)
}
if err := Migrate(db); err != nil {
t.Fatal(err)
}
if err := db.Model(&Term{}).Count(&terms).Error; err != nil || terms != 1 {
t.Fatal("re-upgrade changed existing personal terms", terms, err)
}
}
func TestMigrationRefusesUnownedOrUnsupportedSchema(t *testing.T) {
for _, tc := range []struct{ name, marker string }{
{"empty_marker", ""},
{"negative_version", "INSERT INTO lexgo_schema VALUES (1,-1,'lexgo')"},
{"future_version", "INSERT INTO lexgo_schema VALUES (1,4,'lexgo')"},
{"future_version", "INSERT INTO lexgo_schema VALUES (1,99,'lexgo')"},
{"wrong_product", "INSERT INTO lexgo_schema VALUES (1,0,'another-app')"},
} {
t.Run(tc.name, func(t *testing.T) {
@@ -173,3 +245,53 @@ func TestMigrationRefusesUnownedOrUnsupportedSchema(t *testing.T) {
})
}
}
func TestMigrationFromV3PreservesLibraryAndJobs(t *testing.T) {
db := emptyMigrationDB(t)
statements := []string{"CREATE TABLE lexgo_schema (id INT PRIMARY KEY,version INT,product VARCHAR(32))", "INSERT INTO lexgo_schema VALUES (1,3,'lexgo')"}
statements = append(statements, schemaV2Statements...)
statements = append(statements, schemaV3Statements...)
for _, statement := range statements {
if err := db.Exec(statement).Error; err != nil {
t.Fatal(err)
}
}
if err := db.Exec("INSERT INTO sys_user (user_id,username,password,role_id) VALUES (8,'fixture_v3','fictional-not-a-real-hash',2)").Error; err != nil {
t.Fatal(err)
}
now := stamp(time.Now())
book := Book{OwnerID: 8, Title: "Fictional migration", Language: "en", CreatedAt: now, UpdatedAt: now}
if err := db.Create(&book).Error; err != nil {
t.Fatal(err)
}
original := "😀 Original e\u0301\r\n"
chapter := Chapter{BookID: book.ID, OwnerID: 8, Ordinal: 1, Title: "Fixture", OriginalText: original, ContentSHA256: contentSHA(original), Status: statusReady, CharCount: len([]rune(original)), CreatedAt: now, UpdatedAt: now}
if err := db.Create(&chapter).Error; err != nil {
t.Fatal(err)
}
job := IngestJob{OwnerID: 8, BookID: book.ID, ChapterID: chapter.ID, RequestKey: contentSHA("fixture-v3"), ContentSHA256: chapter.ContentSHA256, Status: statusReady, CreatedAt: now, UpdatedAt: now}
if err := db.Create(&job).Error; err != nil {
t.Fatal(err)
}
if err := CheckSchema(db); err == nil {
t.Fatal("old schema accepted before explicit migration")
}
if err := Migrate(db); err != nil {
t.Fatal(err)
}
if err := CheckSchema(db); err != nil {
t.Fatal(err)
}
var restored Chapter
if err := db.First(&restored, chapter.ID).Error; err != nil || restored.OriginalText != original || restored.ContentSHA256 != chapter.ContentSHA256 || restored.Status != statusReady {
t.Fatal("migration changed original chapter", err)
}
var restoredJob IngestJob
if err := db.First(&restoredJob, job.ID).Error; err != nil || restoredJob.RequestKey != job.RequestKey || restoredJob.Status != statusReady {
t.Fatal("migration changed job", err)
}
var resources int64
if err := db.Model(&DictionaryResource{}).Count(&resources).Error; err != nil || resources != 0 {
t.Fatal("migration must create empty resource table", resources, err)
}
}
+8
View File
@@ -135,6 +135,12 @@ func Router(db *gorm.DB, now func() time.Time) *gin.Engine {
status = 201
}
}
// Saving the same word again is an update of one record, not a new resource.
if c.Request.Method == "POST" && c.FullPath() == "/api/v1/terms" {
if saved, ok := data.(TermSave); ok && saved.Created {
status = 201
}
}
respond(c, status, data, err)
}
}
@@ -294,6 +300,8 @@ func Router(db *gorm.DB, now func() time.Time) *gin.Engine {
}
return gin.H{"job": job, "chapter": chapter}, nil
}))
registerDictionaryRoutes(v, protect, now)
registerTermRoutes(v, protect, now)
r.NoRoute(func(c *gin.Context) { respond(c, 404, nil, failure(404, "页面或接口不存在")) })
return r
}
+330
View File
@@ -0,0 +1,330 @@
package lexgo
import (
"errors"
"strconv"
"strings"
"time"
"unicode"
"unicode/utf8"
"github.com/gin-gonic/gin"
admin "go-admin/app/admin/models"
"gorm.io/gorm"
"gorm.io/gorm/clause"
)
// The four learner-visible statuses are stored explicitly instead of the upstream
// merged status/level code. The mapping stays available for CSV export and for
// migrating an existing LinguaCafe instance: new=2, ignored=1, known=0, and
// learning level N=-N. A level is only meaningful while learning, so every other
// status keeps level 0 and the database check constraints repeat that rule.
const (
termStatusNew = "new"
termStatusLearning = "learning"
termStatusKnown = "known"
termStatusIgnored = "ignored"
)
const (
termFormLimit = 128
termDefinitionLimit = 2000
termExampleLimit = 500
termExamplesLimit = 5
termLevelMax = 7
// One chapter may contain many distinct words, so identity lookups are batched
// instead of sending an unbounded IN list.
termLookupBatch = 500
)
var termStatuses = map[string]bool{
termStatusNew: true,
termStatusLearning: true,
termStatusKnown: true,
termStatusIgnored: true,
}
// Term is one learner's own record for a word form. Identity is the normalized
// form: the original spelling is kept for display only.
type Term struct {
ID int64 `gorm:"primaryKey;autoIncrement"`
OwnerID int
Language string
Term string
OriginalForm string `gorm:"column:original_form"`
Definition string
Examples string
Status string
Level int
CreatedAt time.Time
UpdatedAt time.Time
}
func (Term) TableName() string { return "lexgo_terms" }
type TermView struct {
ID int64 `json:"id"`
Language string `json:"language"`
Term string `json:"term"`
OriginalForm string `json:"originalForm"`
Definition string `json:"definition"`
Examples []string `json:"examples"`
Status string `json:"status"`
Level int `json:"level"`
CreatedAt time.Time `json:"createdAt"`
UpdatedAt time.Time `json:"updatedAt"`
}
// TokenTerm is what a reader token needs to render its own highlight: the entry
// id for the follow-up read, and enough state to style the word.
type TokenTerm struct {
ID int64 `json:"id"`
Status string `json:"status"`
Level int `json:"level"`
}
type TermSave struct {
Term TermView `json:"term"`
Created bool `json:"created"`
}
func termView(t Term) TermView {
examples := []string{}
if t.Examples != "" {
examples = strings.Split(t.Examples, "\n")
}
return TermView{t.ID, t.Language, t.Term, t.OriginalForm, t.Definition, examples, t.Status, t.Level, t.CreatedAt, t.UpdatedAt}
}
// termLevel enforces the documented status/level boundary: only a learning entry
// carries a level, entering learning defaults to 1, and every other status must
// leave the level at 0.
func termLevel(status string, level *int) (int, error) {
if !termStatuses[status] {
return 0, failure(400, "词语状态无效")
}
if status == termStatusLearning {
if level == nil || *level == 0 {
return 1, nil
}
if *level < 1 || *level > termLevelMax {
return 0, failure(400, "学习等级须为 1~7")
}
return *level, nil
}
if level != nil && *level != 0 {
return 0, failure(400, "只有学习中的词语可以设置等级")
}
return 0, nil
}
func hasControlRune(text string, allowNewline bool) bool {
for _, r := range text {
if allowNewline && (r == '\n' || r == '\t') {
continue
}
if unicode.IsControl(r) {
return true
}
}
return false
}
// termContent validates the learner's own text and returns it in storage form:
// the definition as typed (trimmed) and examples joined by newline.
func termContent(definition string, examples []string) (string, string, error) {
definition = strings.TrimSpace(definition)
if utf8.RuneCountInString(definition) > termDefinitionLimit {
return "", "", failure(400, "个人释义不能超过 2000 个字符")
}
if hasControlRune(definition, true) {
return "", "", failure(400, "个人释义包含不支持的字符")
}
if len(examples) > termExamplesLimit {
return "", "", failure(400, "例句不能超过 5 条")
}
cleaned := make([]string, 0, len(examples))
for _, example := range examples {
example = strings.TrimSpace(example)
if example == "" {
return "", "", failure(400, "例句不能为空行")
}
if utf8.RuneCountInString(example) > termExampleLimit {
return "", "", failure(400, "每条例句不能超过 500 个字符")
}
if hasControlRune(example, false) {
return "", "", failure(400, "例句包含不支持的字符")
}
cleaned = append(cleaned, example)
}
return definition, strings.Join(cleaned, "\n"), nil
}
// wordAtRange returns the word the learner actually selected. Identity is always
// derived from the server's own tokens, so a client cannot name a word, user or
// language that it did not read from this chapter.
func wordAtRange(chapter Chapter, start, end int) (string, error) {
for _, token := range Tokenize(chapter.OriginalText) {
if token.Kind == "word" && token.Start == start && token.End == end {
if utf8.RuneCountInString(token.Text) <= termFormLimit {
return token.Text, nil
}
break
}
}
return "", failure(400, "请选择不超过 128 个字符的完整单词")
}
// languageOf reads the owner's study language. A missing space row falls back to
// the English default that account creation and login already establish.
func languageOf(tx *gorm.DB, owner int) (string, error) {
var space Space
err := tx.Where("owner_id = ?", owner).First(&space).Error
if errors.Is(err, gorm.ErrRecordNotFound) {
return "en", nil
}
if err != nil {
return "", err
}
return space.Language, nil
}
// saveTerm writes one identity with INSERT ... ON DUPLICATE KEY UPDATE: a repeated
// save updates the same row instead of adding a second, conflicting record, and
// two concurrent saves of the same word still leave exactly one.
func saveTerm(tx *gorm.DB, owner int, language, word string, fields TermFields, now time.Time) (TermSave, error) {
when := stamp(now)
row := Term{
OwnerID: owner, Language: language, Term: normalizeWord(word), OriginalForm: word,
Definition: fields.Definition, Examples: fields.Examples, Status: fields.Status, Level: fields.Level,
CreatedAt: when, UpdatedAt: when,
}
// MySQL reports affected rows 1 for an insert and 0 or 2 for an update, so the
// counter distinguishes "created" from "saved again" without a second read.
insert := tx.Clauses(clause.OnConflict{
Columns: []clause.Column{{Name: "owner_id"}, {Name: "language"}, {Name: "term"}},
DoUpdates: clause.Assignments(map[string]any{
"original_form": row.OriginalForm, "definition": row.Definition, "examples": row.Examples,
"status": row.Status, "level": row.Level, "updated_at": row.UpdatedAt,
}),
}).Create(&row)
if insert.Error != nil {
return TermSave{}, insert.Error
}
var stored Term
if err := tx.Where("owner_id = ? AND language = ? AND term = ?", owner, language, row.Term).First(&stored).Error; err != nil {
return TermSave{}, err
}
return TermSave{termView(stored), insert.RowsAffected == 1}, nil
}
// attachTerms marks the tokens this learner already saved. Matching is by
// normalized form across the whole vocabulary, so a word saved in another chapter
// is highlighted here with the same status.
func attachTerms(tx *gorm.DB, owner int, language string, tokens []TextToken) error {
keys := make([]string, 0, 16)
seen := map[string]bool{}
for _, token := range tokens {
if token.Kind != "word" || utf8.RuneCountInString(token.Text) > termFormLimit {
continue
}
key := normalizeWord(token.Text)
if !seen[key] {
seen[key] = true
keys = append(keys, key)
}
}
byKey := map[string]TokenTerm{}
for start := 0; start < len(keys); start += termLookupBatch {
end := min(start+termLookupBatch, len(keys))
var rows []Term
if err := tx.Select("id", "term", "status", "level").
Where("owner_id = ? AND language = ? AND term IN ?", owner, language, keys[start:end]).
Find(&rows).Error; err != nil {
return err
}
for _, row := range rows {
byKey[row.Term] = TokenTerm{ID: row.ID, Status: row.Status, Level: row.Level}
}
}
for i := range tokens {
if tokens[i].Kind != "word" {
continue
}
if term, ok := byKey[normalizeWord(tokens[i].Text)]; ok {
value := term
tokens[i].Term = &value
}
}
return nil
}
// TermFields carries the validated learner text and state into storage.
type TermFields struct {
Definition string
Examples string
Status string
Level int
}
type TermInput struct {
ChapterID int64 `json:"chapterId"`
Start *int `json:"start"`
End *int `json:"end"`
Definition string `json:"definition"`
Examples []string `json:"examples"`
Status string `json:"status"`
Level *int `json:"level"`
}
// registerTermRoutes exposes the learner's own word records. Nothing here is
// written to the audit log: personal learning content stays out of it, and the
// account-level audit already covers administrative changes.
func registerTermRoutes(v *gin.RouterGroup, protect func(bool, func(*gin.Context, *gorm.DB, admin.SysUser) (any, error)) gin.HandlerFunc, now func() time.Time) {
v.POST("/terms", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) {
var input TermInput
if err := decode(c, &input); err != nil {
return nil, err
}
if input.Start == nil || input.End == nil {
return nil, failure(400, "请选择完整单词")
}
chapter, err := readyOwnedChapter(tx, u.UserId, input.ChapterID)
if err != nil {
return nil, err
}
word, err := wordAtRange(chapter, *input.Start, *input.End)
if err != nil {
return nil, err
}
level, err := termLevel(input.Status, input.Level)
if err != nil {
return nil, err
}
definition, examples, err := termContent(input.Definition, input.Examples)
if err != nil {
return nil, err
}
language, err := languageOf(tx, u.UserId)
if err != nil {
return nil, err
}
fields := TermFields{Definition: definition, Examples: examples, Status: input.Status, Level: level}
return saveTerm(tx, u.UserId, language, word, fields, now())
}))
v.GET("/terms/:id", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) {
id, err := strconv.ParseInt(c.Param("id"), 10, 64)
if err != nil || id <= 0 {
return nil, failure(404, "词条不存在")
}
// Another account's id and a missing id answer identically, so the response
// never confirms that someone else's entry exists.
var term Term
if err := tx.Where("id = ? AND owner_id = ?", id, u.UserId).First(&term).Error; errors.Is(err, gorm.ErrRecordNotFound) {
return nil, failure(404, "词条不存在")
} else if err != nil {
return nil, err
}
return gin.H{"term": termView(term)}, nil
}))
}
+365
View File
@@ -0,0 +1,365 @@
package lexgo
import (
"encoding/json"
"fmt"
"strings"
"testing"
"time"
"github.com/gin-gonic/gin"
admin "go-admin/app/admin/models"
"gorm.io/gorm"
)
func TestTermLevelBoundary(t *testing.T) {
level := func(v int) *int { return &v }
cases := []struct {
status string
level *int
want int
ok bool
}{
{termStatusNew, nil, 0, true},
{termStatusNew, level(0), 0, true},
{termStatusNew, level(1), 0, false},
{termStatusKnown, level(0), 0, true},
{termStatusKnown, level(3), 0, false},
{termStatusIgnored, level(0), 0, true},
{termStatusIgnored, level(-1), 0, false},
{termStatusLearning, nil, 1, true},
{termStatusLearning, level(0), 1, true},
{termStatusLearning, level(1), 1, true},
{termStatusLearning, level(7), 7, true},
{termStatusLearning, level(8), 0, false},
{termStatusLearning, level(-1), 0, false},
{"", nil, 0, false},
{"Learning", nil, 0, false},
{"deleted", nil, 0, false},
}
for _, tc := range cases {
got, err := termLevel(tc.status, tc.level)
if tc.ok && (err != nil || got != tc.want) {
t.Fatalf("%s/%v: got %d, %v; want %d", tc.status, tc.level, got, err, tc.want)
}
if !tc.ok && err == nil {
t.Fatalf("%s/%v: invalid state accepted as %d", tc.status, tc.level, got)
}
}
}
func TestTermContentRules(t *testing.T) {
definition, examples, err := termContent(" 好奇\t心\n求知 ", []string{" A fictional line. ", "Second line"})
if err != nil || definition != "好奇\t心\n求知" || examples != "A fictional line.\nSecond line" {
t.Fatalf("trim and join: %q %q %v", definition, examples, err)
}
if definition, examples, err = termContent(" ", nil); err != nil || definition != "" || examples != "" {
t.Fatalf("an empty definition is allowed: %q %q %v", definition, examples, err)
}
rejected := []struct {
name string
def string
examples []string
}{
{"definition too long", strings.Repeat("字", termDefinitionLimit+1), nil},
{"definition control character", "a\u0007b", nil},
{"empty example", "ok", []string{"fine", " "}},
{"example with newline", "ok", []string{"two\nlines"}},
{"example too long", "ok", []string{strings.Repeat("a", termExampleLimit+1)}},
{"too many examples", "ok", []string{"1", "2", "3", "4", "5", "6"}},
}
for _, tc := range rejected {
if _, _, err := termContent(tc.def, tc.examples); err == nil {
t.Fatalf("%s was accepted", tc.name)
}
}
if _, _, err := termContent(strings.Repeat("字", termDefinitionLimit), []string{strings.Repeat("a", termExampleLimit)}); err != nil {
t.Fatalf("boundary values must be accepted: %v", err)
}
}
func TestWordAtRangeUsesServerTokens(t *testing.T) {
chapter := Chapter{OriginalText: "Cats went. Dogs"}
for _, tc := range []struct {
start, end int
want string
}{
{0, 4, "Cats"},
{5, 9, "went"},
{11, 15, "Dogs"},
} {
got, err := wordAtRange(chapter, tc.start, tc.end)
if err != nil || got != tc.want {
t.Fatalf("%d-%d: got %q %v, want %q", tc.start, tc.end, got, err, tc.want)
}
}
for _, span := range [][2]int{{0, 3}, {0, 5}, {4, 5}, {9, 10}, {10, 11}, {0, 15}, {-1, 3}, {12, 15}, {0, 0}} {
if _, err := wordAtRange(chapter, span[0], span[1]); err == nil {
t.Fatalf("range %v accepted as a word", span)
}
}
long := "a" + strings.Repeat("b", termFormLimit)
if _, err := wordAtRange(Chapter{OriginalText: long}, 0, len([]rune(long))); err == nil {
t.Fatal("over-long word accepted")
}
}
func TestTermViewSplitsExamples(t *testing.T) {
base := Term{ID: 3, Language: "en", Term: "curiosity", OriginalForm: "Curiosity", Status: termStatusNew}
if got := termView(base); len(got.Examples) != 0 || got.Examples == nil {
t.Fatalf("empty examples must serialize as []: %#v", got.Examples)
}
base.Examples = "One.\nTwo."
if got := termView(base); len(got.Examples) != 2 || got.Examples[1] != "Two." {
t.Fatalf("examples: %#v", got.Examples)
}
}
// termFixture creates one learner with a private book holding one ready chapter per
// text. Fictional content only; nothing here touches another account's data.
func termFixture(t *testing.T, db *gorm.DB, r *gin.Engine, texts ...string) (admin.SysUser, string, []Chapter) {
t.Helper()
u := admin.SysUser{Username: randomName("term"), Password: fixturePassword, RoleId: 2, Status: "2"}
if err := db.Create(&u).Error; err != nil {
t.Fatal("fixture learner creation failed")
}
now := stamp(time.Now())
book := Book{OwnerID: u.UserId, Title: "Fictional terms", Language: "en", CreatedAt: now, UpdatedAt: now}
if err := db.Create(&book).Error; err != nil {
t.Fatal(err)
}
var chapters []Chapter
for index, text := range texts {
chapter := Chapter{BookID: book.ID, OwnerID: u.UserId, Ordinal: index + 1, Title: fmt.Sprintf("Fictional %d", index+1),
OriginalText: text, ContentSHA256: contentSHA(text), CharCount: len([]rune(text)), Status: statusReady, CreatedAt: now, UpdatedAt: now}
if err := db.Create(&chapter).Error; err != nil {
t.Fatal(err)
}
chapters = append(chapters, chapter)
}
return u, loginToken(t, r, u.Username, fixturePassword), chapters
}
func saveTermAPI(t *testing.T, r *gin.Engine, token string, body map[string]any) (int, TermSave) {
t.Helper()
code, data := callAPI(t, r, "POST", "/api/v1/terms", token, body)
var saved TermSave
if data != nil {
json.Unmarshal(data, &saved)
}
return code, saved
}
func chapterTokens(t *testing.T, r *gin.Engine, token string, chapterID int64) (int, ChapterTokens) {
t.Helper()
code, data := callAPI(t, r, "GET", fmt.Sprintf("/api/v1/chapters/%d/tokens", chapterID), token, nil)
var analyzed ChapterTokens
if data != nil {
json.Unmarshal(data, &analyzed)
}
return code, analyzed
}
// wordTokens returns only the selectable words, so a test can talk about the third
// word instead of the sixth token.
func wordTokens(tokens []TextToken) []TextToken {
words := make([]TextToken, 0, len(tokens))
for _, token := range tokens {
if token.Kind == "word" {
words = append(words, token)
}
}
return words
}
// TestMySQLTermIdentityAndCrossChapterConsistency covers the acceptance items that
// one saved word stays one record, that case is not a second word, that another
// inflection is its own entry, and that another chapter shows the same state.
func TestMySQLTermIdentityAndCrossChapterConsistency(t *testing.T) {
db := testDB(t)
r := Router(db, time.Now)
learner, token, chapters := termFixture(t, db, r, "Dogs dogs dog.", "Dogs elsewhere")
chapter, elsewhere := chapters[0], chapters[1]
code, saved := saveTermAPI(t, r, token, map[string]any{
"chapterId": chapter.ID, "start": 0, "end": 4, "definition": "狗", "examples": []string{"Fictional dogs."}, "status": termStatusNew,
})
if code != 201 || !saved.Created || saved.Term.ID == 0 {
t.Fatalf("first save: %d %#v", code, saved)
}
if saved.Term.Term != "dogs" || saved.Term.OriginalForm != "Dogs" || saved.Term.Language != "en" || saved.Term.Level != 0 {
t.Fatalf("stored identity: %#v", saved.Term)
}
first := saved.Term.ID
// The exact same save is idempotent: same record, same status code, no duplicate.
code, repeated := saveTermAPI(t, r, token, map[string]any{
"chapterId": chapter.ID, "start": 0, "end": 4, "definition": "狗", "examples": []string{"Fictional dogs."}, "status": termStatusNew,
})
if code != 200 || repeated.Created || repeated.Term.ID != first {
t.Fatalf("identical repeat must update one record: %d %#v", code, repeated)
}
// Repeating the save updates the same record instead of adding a second one.
code, again := saveTermAPI(t, r, token, map[string]any{
"chapterId": chapter.ID, "start": 0, "end": 4, "definition": "狗;犬", "examples": []string{}, "status": termStatusLearning, "level": 3,
})
if code != 200 || again.Created || again.Term.ID != first || again.Term.Definition != "狗;犬" || again.Term.Status != termStatusLearning || again.Term.Level != 3 {
t.Fatalf("repeated save: %d %#v", code, again)
}
// The same word in another case is the same identity.
code, lower := saveTermAPI(t, r, token, map[string]any{
"chapterId": chapter.ID, "start": 5, "end": 9, "definition": "狗", "status": termStatusKnown,
})
if code != 200 || lower.Created || lower.Term.ID != first || lower.Term.OriginalForm != "dogs" || lower.Term.Level != 0 {
t.Fatalf("case folded identity: %d %#v", code, lower)
}
// A different inflection stays its own entry instead of merging by lemma.
code, inflection := saveTermAPI(t, r, token, map[string]any{
"chapterId": chapter.ID, "start": 10, "end": 13, "definition": "狗(单数)", "status": termStatusNew,
})
if code != 201 || inflection.Term.ID == first {
t.Fatalf("inflection must be a separate entry: %d %#v", code, inflection)
}
var owned int64
if err := db.Model(&Term{}).Where("owner_id = ?", learner.UserId).Count(&owned).Error; err != nil || owned != 2 {
t.Fatalf("one word must stay one record: count=%d err=%v", owned, err)
}
// The status saved here is the status the reader shows for this chapter.
code, tokens := chapterTokens(t, r, token, chapter.ID)
words := wordTokens(tokens.Tokens)
if code != 200 || len(words) != 3 {
t.Fatalf("tokens: %d %#v", code, tokens.Tokens)
}
if words[0].Term == nil || words[0].Term.ID != first || words[0].Term.Status != termStatusKnown {
t.Fatalf("first token term: %#v", words[0])
}
if words[2].Term == nil || words[2].Term.ID != inflection.Term.ID {
t.Fatalf("second entry token term: %#v", words[2])
}
for _, token := range tokens.Tokens {
if token.Kind != "word" && token.Term != nil {
t.Fatalf("spaces and punctuation must not carry a term: %#v", token)
}
}
// Another chapter of the same learner shows the same entry without saving again.
code, crossTokens := chapterTokens(t, r, token, elsewhere.ID)
crossWords := wordTokens(crossTokens.Tokens)
if code != 200 || len(crossWords) != 2 || crossWords[0].Term == nil || crossWords[0].Term.ID != first || crossWords[0].Term.Status != termStatusKnown {
t.Fatalf("cross-chapter highlight: %d %#v", code, crossTokens.Tokens)
}
// Another account sees the same text with no personal state at all.
_, otherToken, otherChapters := termFixture(t, db, r, "Dogs elsewhere")
code, otherTokens := chapterTokens(t, r, otherToken, otherChapters[0].ID)
otherWords := wordTokens(otherTokens.Tokens)
if code != 200 || len(otherWords) != 2 || otherWords[0].Term != nil {
t.Fatalf("another account must not see a term: %d %#v", code, otherTokens.Tokens)
}
if code, _ := chapterTokens(t, r, otherToken, chapter.ID); code != 404 {
t.Fatalf("foreign chapter must be 404: %d", code)
}
}
// TestMySQLTermIsolationAndInputRules covers ownership, tampered input and the
// status/level boundary through the real HTTP surface.
func TestMySQLTermIsolationAndInputRules(t *testing.T) {
db := testDB(t)
r := Router(db, time.Now)
_, ownerToken, chapters := termFixture(t, db, r, "Cats went.")
chapter := chapters[0]
otherUser, otherToken, _ := termFixture(t, db, r, "Cats went.")
code, saved := saveTermAPI(t, r, ownerToken, map[string]any{"chapterId": chapter.ID, "start": 0, "end": 4, "definition": "猫", "status": termStatusNew})
if code != 201 {
t.Fatalf("owner save: %d", code)
}
if code, _ := callAPI(t, r, "GET", fmt.Sprintf("/api/v1/terms/%d", saved.Term.ID), ownerToken, nil); code != 200 {
t.Fatalf("owner read: %d", code)
}
if code, _ := callAPI(t, r, "GET", fmt.Sprintf("/api/v1/terms/%d", saved.Term.ID), otherToken, nil); code != 404 {
t.Fatalf("foreign read must be 404: %d", code)
}
for _, id := range []string{"0", "-1", "abc", "99999999999999999999"} {
if code, _ := callAPI(t, r, "GET", "/api/v1/terms/"+id, ownerToken, nil); code != 404 {
t.Fatalf("invalid id %s: %d", id, code)
}
}
if code, _ := callAPI(t, r, "POST", "/api/v1/terms", "", map[string]any{"chapterId": chapter.ID, "start": 0, "end": 4, "status": termStatusNew}); code != 401 {
t.Fatal("anonymous save must be 401")
}
if code, _ := saveTermAPI(t, r, otherToken, map[string]any{"chapterId": chapter.ID, "start": 0, "end": 4, "status": termStatusNew}); code != 404 {
t.Fatal("saving into a foreign chapter must be 404")
}
// A client cannot name the owner, the language, the word or the record.
for _, body := range []map[string]any{
{"chapterId": chapter.ID, "start": 0, "end": 4, "status": termStatusNew, "ownerId": otherUser.UserId},
{"chapterId": chapter.ID, "start": 0, "end": 4, "status": termStatusNew, "language": "fr"},
{"chapterId": chapter.ID, "start": 0, "end": 4, "status": termStatusNew, "term": "forged"},
{"chapterId": chapter.ID, "start": 0, "end": 4, "status": termStatusNew, "id": 1},
} {
if code, _ := saveTermAPI(t, r, ownerToken, body); code != 400 {
t.Fatalf("tampered input %v must be 400: %d", body, code)
}
}
// Invalid or partial ranges, an unknown status and a forbidden level.
for _, body := range []map[string]any{
{"chapterId": chapter.ID, "start": 0, "status": termStatusNew},
{"chapterId": chapter.ID, "start": 0, "end": 3, "status": termStatusNew},
{"chapterId": chapter.ID, "start": 1, "end": 4, "status": termStatusNew},
{"chapterId": chapter.ID, "start": 4, "end": 5, "status": termStatusNew},
{"chapterId": chapter.ID, "start": 0, "end": 4, "status": "deleted"},
{"chapterId": chapter.ID, "start": 0, "end": 4},
{"chapterId": chapter.ID, "start": 0, "end": 4, "status": termStatusKnown, "level": 4},
{"chapterId": chapter.ID, "start": 0, "end": 4, "status": termStatusLearning, "level": 8},
{"chapterId": chapter.ID, "start": 0, "end": 4, "status": termStatusNew, "definition": strings.Repeat("a", termDefinitionLimit+1)},
{"chapterId": chapter.ID, "start": 0, "end": 4, "status": termStatusNew, "examples": []string{"1", "2", "3", "4", "5", "6"}},
} {
if code, _ := saveTermAPI(t, r, ownerToken, body); code != 400 {
t.Fatalf("invalid input %v must be 400: %d", body, code)
}
}
// Entering learning without a level starts at 1 and still updates the same entry.
code, learning := saveTermAPI(t, r, ownerToken, map[string]any{"chapterId": chapter.ID, "start": 0, "end": 4, "status": termStatusLearning})
if code != 200 || learning.Term.ID != saved.Term.ID || learning.Term.Level != 1 {
t.Fatalf("learning default level: %d %#v", code, learning)
}
// A chapter that is not ready refuses the save instead of storing a word.
if err := db.Model(&Chapter{}).Where("id = ?", chapter.ID).Update("status", statusPending).Error; err != nil {
t.Fatal(err)
}
if code, _ := saveTermAPI(t, r, ownerToken, map[string]any{"chapterId": chapter.ID, "start": 0, "end": 4, "status": termStatusNew}); code != 409 {
t.Fatal("pending chapter must be 409")
}
// An unknown chapter id never confirms whether it belongs to someone else.
if code, _ := saveTermAPI(t, r, ownerToken, map[string]any{"chapterId": 999999, "start": 0, "end": 4, "status": termStatusNew}); code != 404 {
t.Fatal("unknown chapter must be 404")
}
}
// TestMySQLTermTextRoundTrip checks the learner's own text, including characters
// that are never normalized away.
func TestMySQLTermTextRoundTrip(t *testing.T) {
db := testDB(t)
r := Router(db, time.Now)
_, token, chapters := termFixture(t, db, r, "Café Dogs")
code, saved := saveTermAPI(t, r, token, map[string]any{
"chapterId": chapters[0].ID, "start": 0, "end": 4,
"definition": "咖啡\n附带说明", "examples": []string{"A fictional example.", "😀 second"},
"status": termStatusLearning, "level": 7,
})
if code != 201 {
t.Fatalf("save: %d", code)
}
code, data := callAPI(t, r, "GET", fmt.Sprintf("/api/v1/terms/%d", saved.Term.ID), token, nil)
if code != 200 {
t.Fatalf("read: %d", code)
}
var response struct {
Term TermView `json:"term"`
}
json.Unmarshal(data, &response)
if response.Term.Definition != "咖啡\n附带说明" || len(response.Term.Examples) != 2 || response.Term.Examples[1] != "😀 second" || response.Term.Level != 7 {
t.Fatalf("round trip: %#v", response.Term)
}
}
+338
View File
@@ -0,0 +1,338 @@
package lexgo
import (
"archive/zip"
"bufio"
"bytes"
"errors"
"io"
"path"
"strconv"
"strings"
"unicode"
"golang.org/x/text/unicode/norm"
)
const WordNetSHA = "cbda5ea6eef7f36a97a43d4a75f85e07fccbb4f23657d27b4ccbc93e2646ab59"
const WordNetSource = "https://raw.githubusercontent.com/nltk/nltk_data/96f9b3252457a2b97e52aec64c3dfceeb5c312d5/packages/corpora/wordnet.zip"
const maxDictionaryZip = 32 << 20
const maxDictionaryInflated = 128 << 20
type TextToken struct {
Text string `json:"text"`
Start int `json:"start"`
End int `json:"end"`
StartUtf16 int `json:"startUtf16"`
EndUtf16 int `json:"endUtf16"`
Kind string `json:"kind"`
Term *TokenTerm `json:"term,omitempty"`
}
// Tokenize never normalizes text. Letters begin words; marks continue them and
// apostrophes join letters. Numbers and symbols are non-selectable punctuation.
// All offsets are half-open, in code points and UTF-16 code units respectively.
func Tokenize(text string) []TextToken {
runes := []rune(text)
result := make([]TextToken, 0)
utf16Offset := 0
for i := 0; i < len(runes); {
start, start16 := i, utf16Offset
kind := "punctuation"
if unicode.IsLetter(runes[i]) {
kind = "word"
} else if unicode.IsSpace(runes[i]) {
kind = "space"
}
i++
for i < len(runes) {
if kind == "word" && (unicode.IsLetter(runes[i]) || unicode.IsMark(runes[i]) || ((runes[i] == '\'' || runes[i] == '’') && i+1 < len(runes) && unicode.IsLetter(runes[i+1]))) {
i++
continue
}
if kind == "space" && unicode.IsSpace(runes[i]) {
i++
continue
}
break
}
for _, r := range runes[start:i] {
utf16Offset++
if r > 0xffff {
utf16Offset++
}
}
result = append(result, TextToken{Text: string(runes[start:i]), Start: start, End: i, StartUtf16: start16, EndUtf16: utf16Offset, Kind: kind})
}
return result
}
func normalizeWord(word string) string {
return norm.NFC.String(strings.ToLower(strings.ReplaceAll(word, "’", "'")))
}
type DictionaryEntry struct {
Lemma string `json:"lemma"`
POS string `json:"pos"`
Definition string `json:"definition"`
Examples []string `json:"examples"`
}
type LookupResult struct {
Status string `json:"status"`
Query string `json:"query"`
MatchedForm *string `json:"matchedForm"`
Candidates []string `json:"candidates"`
Entries []DictionaryEntry `json:"entries"`
Resource *LookupResource `json:"resource,omitempty"`
}
type LookupResource struct {
Name string `json:"name"`
Version string `json:"version"`
}
type wordNetPOS struct {
pos string
index map[string][]int
data map[int]DictionaryEntry
exceptions map[string][]string
}
type WordNet struct {
parts []wordNetPOS
EntryCount int
}
// ParseWordNet accepts only the pinned official corpus. No archive member is ever
// extracted to disk; both declared and actual inflated sizes are bounded.
func ParseWordNet(raw []byte) (*WordNet, error) {
if len(raw) > maxDictionaryZip || contentSHA(string(raw)) != WordNetSHA {
return nil, errors.New("WordNet ZIP checksum mismatch")
}
return parseWordNetArchive(raw)
}
func readWordNetArchive(raw []byte) (map[string][]byte, error) {
if len(raw) > maxDictionaryZip {
return nil, errors.New("archive too large")
}
reader, err := zip.NewReader(bytes.NewReader(raw), int64(len(raw)))
if err != nil {
return nil, err
}
if len(reader.File) > 64 {
return nil, errors.New("too many archive members")
}
files := map[string][]byte{}
seen := map[string]bool{}
total := int64(0)
for _, f := range reader.File {
name := f.Name
if strings.Contains(name, "\\") || path.IsAbs(name) || strings.Contains(name, ":") || path.Clean(name) != strings.TrimSuffix(name, "/") || !strings.HasPrefix(name, "wordnet/") || seen[name] {
return nil, errors.New("unsafe archive member")
}
seen[name] = true
if !f.FileInfo().IsDir() && !f.Mode().IsRegular() {
return nil, errors.New("unsupported archive member")
}
if f.FileInfo().IsDir() {
continue
}
if f.UncompressedSize64 > maxDictionaryInflated || total+int64(f.UncompressedSize64) > maxDictionaryInflated {
return nil, errors.New("inflated archive too large")
}
rc, err := f.Open()
if err != nil {
return nil, err
}
data, err := io.ReadAll(io.LimitReader(rc, maxDictionaryInflated-total+1))
rc.Close()
if err != nil {
return nil, err
}
total += int64(len(data))
if total > maxDictionaryInflated {
return nil, errors.New("inflated archive too large")
}
files[strings.TrimPrefix(name, "wordnet/")] = data
}
for _, name := range []string{"LICENSE", "README", "lexnames", "index.noun", "index.verb", "index.adj", "index.adv", "data.noun", "data.verb", "data.adj", "data.adv", "noun.exc", "verb.exc", "adj.exc", "adv.exc"} {
if len(files[name]) == 0 {
return nil, errors.New("missing required WordNet member")
}
}
return files, nil
}
func parseWordNetArchive(raw []byte) (*WordNet, error) {
files, err := readWordNetArchive(raw)
if err != nil {
return nil, err
}
result := &WordNet{}
for i, suffix := range []string{"noun", "verb", "adj", "adv"} {
part := wordNetPOS{pos: []string{"n", "v", "a", "r"}[i], index: map[string][]int{}, data: map[int]DictionaryEntry{}, exceptions: map[string][]string{}}
scanner := bufio.NewScanner(bytes.NewReader(files["data."+suffix]))
scanner.Buffer(make([]byte, 4096), 1<<20)
for scanner.Scan() {
line := scanner.Text()
if line == "" || line[0] == ' ' {
continue
}
header, gloss, ok := strings.Cut(line, "|")
fields := strings.Fields(header)
if !ok || len(fields) < 6 {
return nil, errors.New("invalid WordNet data record")
}
offset, e := strconv.Atoi(fields[0])
if e != nil {
return nil, e
}
if _, exists := part.data[offset]; exists {
return nil, errors.New("duplicate synset offset")
}
definition, examples := wordNetGloss(gloss)
part.data[offset] = DictionaryEntry{POS: fields[2], Definition: definition, Examples: examples}
}
if err := scanner.Err(); err != nil {
return nil, err
}
scanner = bufio.NewScanner(bytes.NewReader(files["index."+suffix]))
scanner.Buffer(make([]byte, 4096), 1<<20)
for scanner.Scan() {
line := scanner.Text()
if line == "" || line[0] == ' ' {
continue
}
fields := strings.Fields(line)
if len(fields) < 7 {
return nil, errors.New("invalid WordNet index record")
}
count, e := strconv.Atoi(fields[2])
if e != nil || count < 1 || count > 1000 {
return nil, errors.New("invalid sense count")
}
pointers, e := strconv.Atoi(fields[3])
if e != nil || pointers < 0 || len(fields) != 6+pointers+count {
return nil, errors.New("invalid index offsets")
}
offsets := make([]int, 0, count)
for _, rawOffset := range fields[6+pointers:] {
offset, e := strconv.Atoi(rawOffset)
if e != nil {
return nil, e
}
if _, ok := part.data[offset]; !ok {
return nil, errors.New("missing synset")
}
offsets = append(offsets, offset)
}
if _, exists := part.index[fields[0]]; exists {
return nil, errors.New("duplicate index key")
}
part.index[fields[0]] = offsets
}
if err := scanner.Err(); err != nil {
return nil, err
}
scanner = bufio.NewScanner(bytes.NewReader(files[suffix+".exc"]))
for scanner.Scan() {
fields := strings.Fields(scanner.Text())
if len(fields) < 2 {
return nil, errors.New("invalid exception record")
}
part.exceptions[fields[0]] = fields[1:]
}
if err := scanner.Err(); err != nil {
return nil, err
}
result.EntryCount += len(part.index)
result.parts = append(result.parts, part)
}
return result, nil
}
func wordNetGloss(gloss string) (string, []string) {
examples := []string{}
definition := strings.Builder{}
for {
before, after, ok := strings.Cut(gloss, "\"")
definition.WriteString(before)
if !ok {
break
}
example, rest, closed := strings.Cut(after, "\"")
if !closed {
definition.WriteString(after)
break
}
examples = append(examples, example)
gloss = rest
}
return strings.Trim(strings.TrimSpace(definition.String()), "; "), examples
}
var morphyRules = map[string][][2]string{
"n": {{"s", ""}, {"ses", "s"}, {"ves", "f"}, {"xes", "x"}, {"zes", "z"}, {"ches", "ch"}, {"shes", "sh"}, {"men", "man"}, {"ies", "y"}},
"v": {{"s", ""}, {"ies", "y"}, {"es", "e"}, {"es", ""}, {"ed", "e"}, {"ed", ""}, {"ing", "e"}, {"ing", ""}},
"a": {{"er", ""}, {"est", ""}, {"er", "e"}, {"est", "e"}},
}
// Morphy yields possible dictionary forms, never contextual POS or a merged
// learning identity. Exact forms win before exceptions and suffix detachment.
func (wn *WordNet) Lookup(query string) LookupResult {
result := LookupResult{Status: "not_found", Query: query, Candidates: []string{}, Entries: []DictionaryEntry{}}
key := normalizeWord(query)
add := func(form string, part wordNetPOS) {
if len(part.index[form]) == 0 {
return
}
found := false
for _, candidate := range result.Candidates {
if candidate == form {
found = true
break
}
}
if !found {
result.Candidates = append(result.Candidates, form)
}
for _, offset := range part.index[form] {
if len(result.Entries) >= 12 {
break
}
entry := part.data[offset]
entry.Lemma = strings.ReplaceAll(form, "_", " ")
result.Entries = append(result.Entries, entry)
}
}
for _, part := range wn.parts {
add(key, part)
}
if len(result.Entries) > 0 {
result.Status = "exact"
} else {
for _, part := range wn.parts {
forms := part.exceptions[key]
if len(forms) == 0 {
for _, rule := range morphyRules[part.pos] {
if strings.HasSuffix(key, rule[0]) && len(key) > len(rule[0]) {
forms = append(forms, strings.TrimSuffix(key, rule[0])+rule[1])
}
}
}
seen := map[string]bool{}
for _, form := range forms {
if !seen[form] {
add(form, part)
seen[form] = true
}
}
}
if len(result.Entries) > 0 {
result.Status = "lemma"
}
}
if len(result.Candidates) > 0 {
form := result.Candidates[0]
result.MatchedForm = &form
}
return result
}
+112
View File
@@ -0,0 +1,112 @@
package lexgo
import (
"archive/zip"
"bytes"
"os"
"strings"
"testing"
"unicode/utf16"
)
func TestArchiveRejectsUnsafeMembersAndMissingLicense(t *testing.T) {
for _, name := range []string{"../data.noun", "wordnet/../data.noun", "/wordnet/data.noun", "wordnet\\data.noun", "wordnet/C:data.noun", "wordnet/LICENSE"} {
var raw bytes.Buffer
w := zip.NewWriter(&raw)
entry, err := w.Create(name)
if err != nil {
t.Fatal(err)
}
entry.Write([]byte("fictional fixture"))
w.Close()
if _, err := readWordNetArchive(raw.Bytes()); err == nil {
t.Fatalf("accepted unsafe/incomplete archive %s", name)
}
}
var raw bytes.Buffer
w := zip.NewWriter(&raw)
for i := 0; i < 2; i++ {
entry, _ := w.Create("wordnet/LICENSE")
entry.Write([]byte("fixture"))
}
w.Close()
if _, err := readWordNetArchive(raw.Bytes()); err == nil {
t.Fatal("accepted duplicate member")
}
if _, err := ParseWordNet(make([]byte, maxDictionaryZip+1)); err == nil {
t.Fatal("accepted oversized archive")
}
}
func TestMorphyCandidatesAndExactPrecedence(t *testing.T) {
wn := &WordNet{parts: []wordNetPOS{{pos: "v", index: map[string][]int{"axes": {1}, "ax": {1}, "axe": {2}}, data: map[int]DictionaryEntry{1: {POS: "v", Definition: "fixture one", Examples: []string{}}, 2: {POS: "v", Definition: "fixture two", Examples: []string{}}}, exceptions: map[string][]string{}}}}
if got := wn.Lookup("AXES"); got.Status != "exact" || len(got.Candidates) != 1 || *got.MatchedForm != "axes" {
t.Fatal("exact must precede lemma", got)
}
delete(wn.parts[0].index, "axes")
got := wn.Lookup("axes")
if got.Status != "lemma" || strings.Join(got.Candidates, ",") != "axe,ax" {
t.Fatal("deterministic candidate forms", got)
}
definition, examples := wordNetGloss(`a fictional gloss; "one example"; "second example"`)
if definition != "a fictional gloss" || len(examples) != 2 {
t.Fatal("gloss", definition, examples)
}
}
func TestTokensPreserveUnicodeOriginal(t *testing.T) {
text := "😀 Cafe\u0301 can’t\r\nDogs 123 中文!"
tokens := Tokenize(text)
runes := []rune(text)
cursor, u16 := 0, 0
words := []string{}
for _, token := range tokens {
if token.Start != cursor || token.StartUtf16 != u16 || string(runes[token.Start:token.End]) != token.Text {
t.Fatalf("incorrect token %#v", token)
}
cursor = token.End
u16 += len(utf16.Encode([]rune(token.Text)))
if token.EndUtf16 != u16 {
t.Fatal("incorrect UTF-16 end")
}
if token.Kind == "word" {
words = append(words, token.Text)
}
}
if cursor != len(runes) || strings.Join(words, "|") != "Cafe\u0301|can’t|Dogs|中文" {
t.Fatalf("coverage/words %v", words)
}
if normalizeWord("CAFE\u0301") != "café" || normalizeWord("CAN’T") != "can't" {
t.Fatal("normalization")
}
}
func TestOfficialWordNet(t *testing.T) {
raw, err := os.ReadFile("../../../.local/nlp-resources/wordnet.zip")
if os.IsNotExist(err) {
t.Skip("explicitly prepared official WordNet fixture unavailable")
}
if err != nil {
t.Fatal(err)
}
engine, err := ParseWordNet(raw)
if err != nil {
t.Fatal(err)
}
if engine.EntryCount < 150000 {
t.Fatalf("entry count %d", engine.EntryCount)
}
for _, tc := range []struct{ query, status, lemma string }{{"dog", "exact", "dog"}, {"went", "lemma", "go"}, {"mice", "lemma", "mouse"}, {"zzzznonword", "not_found", ""}} {
got := engine.Lookup(tc.query)
if got.Status != tc.status || (tc.lemma != "" && (got.MatchedForm == nil || *got.MatchedForm != tc.lemma)) {
t.Fatalf("%s: %#v", tc.query, got)
}
if tc.lemma != "" && (len(got.Entries) == 0 || got.Entries[0].Definition == "") {
t.Fatal("missing definition")
}
}
raw[100] ^= 1
if _, err := ParseWordNet(raw); err == nil {
t.Fatal("accepted corrupt checksum")
}
}
+1 -1
View File
@@ -82,7 +82,7 @@ func run() error {
if err = lexgo.Migrate(db); err != nil {
return err
}
fmt.Println("LexGo schema version 3 ready")
fmt.Println("LexGo schema version", lexgo.SchemaVersion, "ready")
return nil
}
if err = lexgo.CheckSchema(db); err != nil {
+15
View File
@@ -0,0 +1,15 @@
{
"name": "Princeton WordNet",
"language": "en",
"version": "3.0",
"format": "wordnet-3.0-zip",
"source": "https://raw.githubusercontent.com/nltk/nltk_data/96f9b3252457a2b97e52aec64c3dfceeb5c312d5/packages/corpora/wordnet.zip",
"sha256": "cbda5ea6eef7f36a97a43d4a75f85e07fccbb4f23657d27b4ccbc93e2646ab59",
"license": "WORDNET-LICENSE.txt",
"licenseUrl": "https://wordnet.princeton.edu/license-and-commercial-use",
"documentation": [
"https://wordnet.princeton.edu/documentation/wndb5wn",
"https://wordnet.princeton.edu/documentation/morphy7wn"
],
"runtime": "Go only; explicitly imported ZIP persisted in MySQL; no runtime download or Python process"
}