feat: 全 Go 英语词典管理与阅读点词查义 (#6) #25

Merged
ila merged 2 commits from feat/6-go-dictionary-lookup into main 2026-09-11 14:45:55 +08:00
32 changed files with 1885 additions and 25 deletions
+1
View File
@@ -280,3 +280,4 @@ MVP 内所有单元任务通过后才能做 MVP 集成验收;MVP 通过后才
- #18 登录日志与操作审计已通过用户验收:schema v2 显式迁移;日志只保存白名单字段,禁止保存凭据、请求/响应正文及私人学习内容。仅管理员查询,默认保留 90 天;启动/每小时及 `python scripts/server.py audit-cleanup` 仅清理两张审计表的过期记录。
- #3 独立小样位于 `spikes/english/`,使用 `.local/nlp-venv/Scripts/python.exe`(3.12.12)运行;固定 spaCy 3.8.7、英语模型 3.8.0、NLTK 3.9.2、WordNet 3.0。资源仅显式准备时下载,摘要见 resources.json。不得把本机无账号的实验接口用于正式学习端;后续集成仍需 Go 授权、数据归属和任务设计。原文不归一化,位置区分 cp/UTF-8/UTF-16,lemma 不自动合并学习状态。
- #4 独立阅读选择小样位于 `spikes/selection/`,`python spikes/selection/serve.py` 默认仅本机 5184。桌面鼠标/键盘与 11 项测试已验证,#4 已获用户验收并关闭;真实手机长按/手柄/滚动详细证据仍未提供;禁止把窄屏桌面当作真机验收。Intl.Segmenter 只用于 UI 范围验证,不替代 #3 NLP;释义保存只在内存。固定 LinguaCafe 源码对照和与 v1 的差异记录见架构 Wiki。
- 2026-09-11 用户确认正式 NLP/词典采用全 Go。#5 已验收并合入 main;#6 使用 Go WordNet 解析和词形候选、Go Unicode 原文分片、schema v4 共享词典资源表,不调用 Python NLP。WordNet 3.0 ZIP 来源与摘要见 `server/wordnet-resource.json`,许可保留在 `server/WORDNET-LICENSE.txt`。词形候选不等于上下文消歧,不自动合并个人学习状态;#3 Python 小样只保留历史验证。当前词典仅英语释义,个人释义输入为临时草稿,持久化归 #7。
+11 -3
View File
@@ -2,13 +2,13 @@
面向自托管场景的阅读式语言学习项目,规划提供内容导入、阅读查词、词汇与短语、复习和实例管理。
已确认:**DevHarness 轻量模式、MySQL 8、go-admin 管理端**。工程基础 #2 已通过验收:两端用户名登录、学习账号管理、可撤销会话和本人英语空空间。管理端基于指定 go-admin/go-admin-ui 选用模块,学习端为独立 Vue 3 + TypeScript + Vite 工程,共用 Go 后端和 MySQL 8.4.3。#18 登录日志与操作审计已通过用户验收,支持管理员查询和 90 天保留清理。#5 粘贴导入与章节原文阅读已实现,待用户验收;点词查词、词典、词语状态与复习仍未实现。MVP 定位为“支持多账号、数据独立的自托管学习工具”,先邀请少量用户使用;F01~F12 已确认,X 系列后置。
已确认:**DevHarness 轻量模式、MySQL 8、go-admin 管理端**。工程基础 #2 已通过验收:两端用户名登录、学习账号管理、可撤销会话和本人英语空空间。管理端基于指定 go-admin/go-admin-ui 选用模块,学习端为独立 Vue 3 + TypeScript + Vite 工程,共用 Go 后端和 MySQL 8.4.3。#18 登录日志与操作审计已通过用户验收,支持管理员查询和 90 天保留清理。#5 粘贴导入与章节原文阅读已验收并合入 main;#6 全 Go 英语词典与点词查义已实现,待用户验收。词语状态与复习仍未实现。MVP 定位为“支持多账号、数据独立的自托管学习工具”,先邀请少量用户使用;F01~F12 已确认,X 系列后置。
- [文档入口](docs/README.md) · [线上 Wiki](https://git.ilapage.cn/OPC/lexgo/wiki/Home)
- [英语分词与离线词典验证小样](spikes/english/README.md)(#3 已验收,独立本机入口)
- [阅读选择验证小样](spikes/selection/README.md)(#4 已验收,真机详细测试证据缺口保留)
- [项目档案](docs/00-project-profile.md) · [需求总览](docs/09-product-requirements-overview.md)
- [工作量估算](docs/10-workload-estimate.md):#2、#3、#4、#18 已验收,剩余 #5~#15 与新增 #21 规划参考 44~71 人日;后续结合集成结果重估,旧全量研究仅供参考。
- [工作量估算](docs/10-workload-estimate.md):#2、#3、#4、#5、#18 已验收,原规划中的 #5 已完成;剩余 #6~#15 与新增 #21、#24 按工单复核;后续结合集成结果重估,旧全量研究仅供参考。
- [四阶段实施总览 #16](https://git.ilapage.cn/OPC/lexgo/issues/16):14 张单元工单,工程基础 → 技术验证 → 首条学习闭环 → 补齐 MVP;原型 v1 已获用户验收。两端使用账号(用户名)+密码登录,不要求邮箱。
- [原型工单 #1](https://git.ilapage.cn/OPC/lexgo/issues/1):Quant-UX 桌面/手机原型 v1,预览入口与审核记录见工单及需求总览。
- [工作流](docs/01-workflow.md) · [开发与验证](docs/04-local-development-and-verification.md)
@@ -19,7 +19,15 @@
本地入口:学习端 http://127.0.0.1:5173,管理端 http://127.0.0.1:5174。完整安装与测试命令见[开发与验证](docs/04-local-development-and-verification.md)。账号使用用户名,无需邮箱;没有随代码交付的默认密码。
本机三个服务已由 `D:/supervisord/supervisord.conf` 中的 lexgo-learner、lexgo-admin、lexgo-api 托管,使用时不要重复手动启动同端口。当前 schema v3;从 #2/#18 升级时停止 API,执行 build、migrate 后再启动。审计过期清理可执行 `python scripts/server.py audit-cleanup`,只影响超过 90 天的日志。
本机三个服务已由 `D:/supervisord/supervisord.conf` 中的 lexgo-learner、lexgo-admin、lexgo-api 托管,使用时不要重复手动启动同端口。当前 schema v4;从 #2/#18 升级时停止 API,执行 build、migrate 后再启动。审计过期清理可执行 `python scripts/server.py audit-cleanup`,只影响超过 90 天的日志。
## 英语词典与点词查义(#6)
正式后端为纯 Go,`lexgo.exe serve` 不启动 Python NLP。开发辅助脚本与历史小样保留。
管理员在“英语词典”页下载并导入指定 WordNet 3.0 ZIP;来源、固定摘要及许可见 [资源清单](server/wordnet-resource.json)。词典提供英语释义,保存在 MySQL 中;导入失败保留当前资源,重复导入不新增资源并重新启用。也可在管理页停用。
学习者打开本人就绪章节,点击词语或聚焦后按 Enter/空格查询;Escape/关闭返回阅读。精确词形优先,未命中再查规则候选,例如 `went → go`。个人释义当前是未保存的临时草稿,保存功能在 #7 实现。真机测试证据尚未补齐。
## 文档与治理
+47
View File
@@ -0,0 +1,47 @@
export const MAX_DICTIONARY_BYTES = 32 * 1024 * 1024
export function createDictionaryLoader(session, state) {
let revision = 0
function invalidate() {
revision++
Object.assign(state, { items: [], supported: null, loading: false, saving: false, error: '', notice: '' })
}
async function perform(operation, saving, apply) {
const current = ++revision
const generation = session.state.generation
const stale = () => current !== revision || generation !== session.state.generation
Object.assign(state, { loading: !saving, saving, error: '', notice: '' })
try {
const result = await operation()
if (stale()) return false
apply(result)
return true
} catch (error) {
if (!stale()) state.error = error instanceof Error ? error.message : '词典操作失败,请重试'
return false
} finally {
if (!stale()) { state.loading = false; state.saving = false }
}
}
return {
invalidate,
load() {
return perform(() => session.listDictionaries(), false, result => {
state.items = result.items
state.supported = result.supported
})
},
import(form) {
return perform(() => session.importDictionary(form), true, result => {
state.items = [result.resource]
state.notice = result.duplicate ? '该词典已存在,已启用,未重复导入。' : '词典导入成功。'
})
},
toggle(id, enabled) {
return perform(() => session.setDictionaryEnabled(id, enabled), true, result => {
state.items = state.items.map(item => item.id === result.resource.id ? result.resource : item)
state.notice = result.resource.enabled ? '词典已启用。' : '词典已停用。'
})
}
}
}
+1
View File
@@ -5,6 +5,7 @@
<div class="brand">{{ sidebar.opened ? 'LexGo 管理' : 'LG' }}</div>
<el-menu :default-active="$route.path" :collapse="!sidebar.opened" router>
<el-menu-item index="/accounts"><span>账号管理</span></el-menu-item>
<el-menu-item index="/dictionaries"><span>英语词典</span></el-menu-item>
<el-menu-item index="/login-logs"><span>登录日志</span></el-menu-item>
<el-menu-item index="/operation-logs"><span>操作日志</span></el-menu-item>
</el-menu>
+2
View File
@@ -4,11 +4,13 @@ import Layout from '../layout/index.vue'
import Login from '../views/Login.vue'
import Accounts from '../views/Accounts.vue'
import AuditLogs from '../views/AuditLogs.vue'
import Dictionaries from '../views/Dictionaries.vue'
const router = createRouter({ history: createWebHashHistory(), routes: [
{ path: '/login', component: Login },
{ path: '/', component: Layout, children: [
{ path: '', redirect: '/accounts' },
{ path: 'accounts', component: Accounts, meta: { title: '账号管理' } },
{ path: 'dictionaries', component: Dictionaries, meta: { title: '英语词典' } },
{ path: 'login-logs', component: AuditLogs, props: { kind: 'login' }, meta: { title: '登录日志' } },
{ path: 'operation-logs', component: AuditLogs, props: { kind: 'operation' }, meta: { title: '操作日志' } }
] },
+17 -2
View File
@@ -20,10 +20,11 @@ export function createSession({ fetch, storage, changed = () => {} }) {
if (generation !== state.generation) throw new Error('会话已变化,请重新操作')
}
async function request(path, method = 'GET', body, token = state.token, generation = state.generation) {
const multipart = typeof FormData !== 'undefined' && body instanceof FormData
const result = await fetch('/api/v1' + path, {
method,
headers: { 'Content-Type': 'application/json', ...(token ? { Authorization: 'Bearer ' + token } : {}) },
...(body === undefined ? {} : { body: JSON.stringify(body) }),
headers: { ...(!multipart ? { 'Content-Type': 'application/json' } : {}), ...(token ? { Authorization: 'Bearer ' + token } : {}) },
...(body === undefined ? {} : { body: multipart ? body : JSON.stringify(body) }),
cache: 'no-store'
})
const payload = await result.json()
@@ -84,6 +85,20 @@ export function createSession({ fetch, storage, changed = () => {} }) {
} catch (error) { if (generation === state.generation) clear(); throw error }
},
async logout() { const token = state.token; clear(); await revoke(token) },
async listDictionaries() {
authorized()
return request('/dictionaries')
},
async importDictionary(form) {
authorized()
if (!(form instanceof FormData)) throw new Error('请选择词典文件')
return request('/dictionaries/import', 'POST', form)
},
async setDictionaryEnabled(id, enabled) {
authorized()
if (!Number.isSafeInteger(id) || id <= 0 || typeof enabled !== 'boolean') throw new Error('词典参数无效')
return request('/dictionaries/' + id, 'PATCH', { enabled })
},
async queryAuditLogs(kind, filters) {
authorized()
const generation = state.generation
+101
View File
@@ -0,0 +1,101 @@
<template>
<basic-layout><template #wrapper>
<el-card>
<div class="toolbar"><div><h1>英语词典</h1><p class="subtle">供所有学习账号查询</p></div><el-button :disabled="busy" @click="reload">刷新</el-button></div>
<el-alert v-if="resources.error" :title="resources.error" type="error" :closable="false" show-icon />
<el-alert v-if="resources.notice" :title="resources.notice" type="success" :closable="false" show-icon />
<el-table v-loading="resources.loading" :data="resources.items" border :empty-text="resources.error ? '加载失败,请重试' : '尚未导入词典'">
<el-table-column label="名称" prop="name" min-width="175" />
<el-table-column label="语言" width="80"><template #default>英语</template></el-table-column>
<el-table-column label="版本" prop="version" width="80" />
<el-table-column label="来源" prop="source" min-width="160" show-overflow-tooltip />
<el-table-column label="格式" prop="format" min-width="160" />
<el-table-column label="状态" width="95"><template #default="scope"><el-tag :type="scope.row.status === 'ready' ? 'success' : 'info'">{{ statusLabel(scope.row.status) }}</el-tag></template></el-table-column>
<el-table-column label="词条数" prop="entryCount" min-width="95" />
<el-table-column label="操作" width="100"><template #default="scope"><el-button :disabled="busy" link type="primary" @click="toggle(scope.row)">{{ scope.row.enabled ? '停用' : '启用' }}</el-button></template></el-table-column>
</el-table>
</el-card>
<el-card class="import-card">
<h2>导入词典</h2>
<el-form class="dictionary-form" label-position="top" @submit.prevent="upload">
<el-form-item label="名称"><el-input v-model="form.name" maxlength="120" :disabled="busy" /></el-form-item>
<div class="form-row">
<el-form-item label="语言"><el-input model-value="英语" disabled /></el-form-item>
<el-form-item label="版本"><el-input v-model="form.version" readonly /></el-form-item>
</div>
<el-form-item label="来源"><el-input v-model="form.source" readonly /></el-form-item>
<el-form-item label="格式"><el-input v-model="form.format" readonly /></el-form-item>
<el-form-item label="词典文件">
<input id="dictionary-file" ref="fileInput" type="file" accept=".zip,application/zip" aria-label="词典文件" :disabled="busy" @change="selectFile">
</el-form-item>
<p class="subtle">WordNet 3.0 ZIP · 上限 32 MiB · 英语释义</p>
<p v-if="resources.supported?.source"><a :href="downloadURL" target="_blank" rel="noopener noreferrer">下载支持的词典文件</a></p>
<p class="subtle">导入失败时保留当前词典。</p>
<p v-if="fileError" role="alert" class="file-error">{{ fileError }}</p>
<el-button type="primary" native-type="submit" :loading="resources.saving" :disabled="resources.loading || !file">导入并启用</el-button>
</el-form>
</el-card>
</template></basic-layout>
</template>
<script>
import BasicLayout from '../layout/BasicLayout.vue'
import { session } from '../store'
import { createDictionaryLoader, MAX_DICTIONARY_BYTES } from '../dictionaries.mjs'
const downloadURL = 'https://raw.githubusercontent.com/nltk/nltk_data/96f9b3252457a2b97e52aec64c3dfceeb5c312d5/packages/corpora/wordnet.zip'
export default {
name: 'DictionariesView',
components: { BasicLayout },
data: () => ({
resources: { items: [], supported: null, loading: false, saving: false, error: '', notice: '' },
form: { name: 'Princeton WordNet', language: 'en', version: '3.0', source: downloadURL, format: 'wordnet-3.0-zip' },
file: null, fileError: '', downloadURL
}),
computed: { busy() { return this.resources.loading || this.resources.saving } },
watch: { '$store.state.generation': { flush: 'sync', handler() { this.loader.invalidate(); this.clearFile() } } },
created() { this.loader = createDictionaryLoader(session, this.resources) },
mounted() { this.reload() },
beforeUnmount() { this.loader.invalidate(); this.file = null },
methods: {
reload() { return this.loader.load() },
statusLabel(status) { return { ready: '可用', disabled: '已停用', unavailable: '不可用' }[status] || '不可用' },
clearFile() { this.file = null; this.fileError = ''; if (this.$refs.fileInput) this.$refs.fileInput.value = '' },
selectFile(event) {
this.fileError = ''
const selected = event.target.files?.[0]
this.file = null
if (!selected) return
if (!selected.name.toLowerCase().endsWith('.zip') || selected.size === 0 || selected.size > MAX_DICTIONARY_BYTES) {
this.fileError = '请选择不超过 32 MiB 的 ZIP 文件。'
event.target.value = ''
return
}
this.file = selected
},
async upload() {
if (this.busy || !this.file) return
this.fileError = ''
if (!this.form.name.trim() || !this.form.source.trim()) { this.fileError = '请填写名称和来源。'; return }
const data = new FormData()
for (const [key, value] of Object.entries(this.form)) data.append(key, value.trim())
data.append('file', this.file)
if (await this.loader.import(data)) this.clearFile()
},
toggle(resource) { return this.loader.toggle(resource.id, !resource.enabled) }
}
}
</script>
<style scoped>
.toolbar { display:flex; align-items:center; justify-content:space-between; gap:16px; margin-bottom:20px; }
h1 { font-size:20px; margin:0 0 8px; }
h2 { font-size:18px; margin:0 0 24px; }
.subtle { color:#606266; font-size:13px; line-height:1.6; }
.toolbar p { margin:0; }
.import-card { margin-top:20px; }
.dictionary-form { max-width:600px; }
.form-row { display:flex; gap:16px; }
.form-row .el-form-item { flex:1; min-width:0; }
.file-error { color:#b42318; }
.el-alert { margin-bottom:16px; }
input[type=file] { max-width:100%; }
a { color:#176b63; }
</style>
+55
View File
@@ -0,0 +1,55 @@
import test from 'node:test'
import assert from 'node:assert/strict'
import { createSession } from '../src/session.mjs'
const response = (data, status = 200) => ({ ok: status < 400, status, json: async () => ({ data, msg: '导入失败' }) })
function setup() {
const pending = []
const session = createSession({ fetch: (url, options) => new Promise(resolve => pending.push({ url, options, resolve })), storage: { getItem() {}, setItem() {}, removeItem() {} } })
session.state.user = { id: 1, username: 'fixture.admin', role: 'admin' }
session.state.token = 'fictional-test-token'
return { session, pending }
}
test('dictionary upload sends multipart with authorization and lets browser set boundary', async () => {
const { session, pending } = setup()
const form = new FormData()
form.append('name', 'Princeton WordNet')
form.append('file', new Blob(['fictional-archive']), 'wordnet.zip')
assert.equal(typeof session.importDictionary, 'function')
const request = session.importDictionary(form)
assert.equal(pending[0].url, '/api/v1/dictionaries/import')
assert.equal(pending[0].options.method, 'POST')
assert.equal(pending[0].options.body, form)
assert.equal(pending[0].options.headers['Content-Type'], undefined)
assert.equal(pending[0].options.headers.Authorization, 'Bearer fictional-test-token')
pending[0].resolve(response({ resource: { id: 1 }, duplicate: false }))
assert.deepEqual(await request, { resource: { id: 1 }, duplicate: false })
})
test('dictionary read and toggle follow API contract, ordinary accounts cannot mutate', async () => {
const { session, pending } = setup()
assert.equal(typeof session.listDictionaries, 'function')
const read = session.listDictionaries()
assert.equal(pending[0].url, '/api/v1/dictionaries')
pending[0].resolve(response({ items: [] }))
await read
const toggle = session.setDictionaryEnabled(1, false)
assert.equal(pending[1].url, '/api/v1/dictionaries/1')
assert.deepEqual(JSON.parse(pending[1].options.body), { enabled: false })
pending[1].resolve(response({ resource: { id: 1, enabled: false } }))
await toggle
session.state.user.role = 'learner'
await assert.rejects(session.importDictionary(new FormData()), /管理员/)
await assert.rejects(session.setDictionaryEnabled(1, true), /管理员/)
assert.equal(pending.length, 2)
})
test('dictionary response cannot cross a logout or account switch', async () => {
const { session, pending } = setup()
assert.equal(typeof session.importDictionary, 'function')
const request = session.importDictionary(new FormData())
session.clear()
pending[0].resolve(response({ resource: { id: 1 }, duplicate: false }))
await assert.rejects(request, /会话已变化/)
})
+57
View File
@@ -0,0 +1,57 @@
import test from 'node:test'
import assert from 'node:assert/strict'
async function fixture() {
const { createDictionaryLoader } = await import('../src/dictionaries.mjs')
const pending = []
const later = () => new Promise((resolve, reject) => pending.push({ resolve, reject }))
const session = { state: { generation: 1 }, listDictionaries: later, importDictionary: later, setDictionaryEnabled: later }
const state = { items: [{ id: 1, name: 'Existing', enabled: true }], supported: null, loading: false, saving: false, error: '', notice: '' }
return { loader: createDictionaryLoader(session, state), state, session, pending }
}
test('failed upload preserves the existing resource and exposes the error', async () => {
const { loader, state, pending } = await fixture()
const action = loader.import(new FormData())
assert.equal(state.saving, true)
pending[0].reject(new Error('文件摘要不匹配'))
assert.equal(await action, false)
assert.equal(state.items[0].name, 'Existing')
assert.equal(state.error, '文件摘要不匹配')
assert.equal(state.saving, false)
})
test('accepted mutation replaces old state directly, without a second refresh dependency', async () => {
const { loader, state, pending } = await fixture()
const action = loader.toggle(1, false)
pending[0].resolve({ resource: { id: 1, name: 'Existing', enabled: false, status: 'disabled' } })
assert.equal(await action, true)
assert.equal(state.items[0].enabled, false)
assert.equal(pending.length, 1)
})
test('an old list cannot overwrite a newer import and duplicate import is explicit', async () => {
const { loader, state, pending } = await fixture()
const read = loader.load()
const action = loader.import(new FormData())
pending[1].resolve({ resource: { id: 1, name: 'Imported', enabled: true }, duplicate: true })
await action
pending[0].resolve({ items: [{ id: 1, name: 'Old' }] })
await read
assert.equal(state.items[0].name, 'Imported')
assert.match(state.notice, /已存在/)
})
test('page exit or session change discards late responses', async () => {
const { loader, state, pending, session } = await fixture()
const action = loader.import(new FormData())
loader.invalidate()
pending[0].resolve({ resource: { id: 1, name: 'Late' } })
assert.equal(await action, false)
assert.deepEqual(state.items, [])
const read = loader.load()
session.state.generation++
pending[1].resolve({ items: [{ id: 1, name: 'Other session' }] })
await read
assert.deepEqual(state.items, [])
})
+13 -3
View File
@@ -2,8 +2,8 @@
generated: true (请先修改 Gitea Wiki,禁止直接编辑本文件)
wiki_page: Project-Profile
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Project-Profile.-
wiki_revision: 40ae5e3a0521195da4f1a9cd9f42beec15da20d4
synchronized_at: 2026-09-10T12:43:19Z
wiki_revision: 445504a7f95f9c018216f74e27d433edde521f73
synchronized_at: 2026-09-11T03:58:56Z
<!-- gitea-wiki-mirror:end -->
# LexGo 项目档案
@@ -37,7 +37,7 @@ LinguaCafe 参考基线由现有调研记录为 `c1ea298ce40c65b9dd33e9b26fd2e52
|---|---|---|---|
| D01 | 管理端底座 | **已确认使用 `D:/github_project/goadmin` 的 go-admin + go-admin-ui**,依据用户指令“管理端用D:\github_project\goadmin”;替代 gin-vue-admin 建议 | M0 验证指定本地版本配套、MySQL 8 及数据隔离,不再比较管理底座 |
| D02 | 数据库 | **已确认 MySQL 8**;依据 2026-09-10 用户原话“使用mysql8”。覆盖两份分析的数据库分歧,具体小版本待基线验证锁定 | 按单一 MySQL 8 数据层估算,不做 PostgreSQL 双库兼容 |
| D03 | NLP | 保留 Python 为建议,纯 Go 是否硬约束未确认 | 首发语言质量与开发量 |
| D03 | NLP | **用户于2026-09-11确认正式产品全 Go**;Python小样仅历史验证 | 使用词典词形候选,不承诺上下文消歧 |
| D04 | 默认学习语言 | 用户已确认英语;中文/日语后续需独立范围与验收 | 分词、读音和 UI |
| D05 | 用户范围 | **已确认:支持多账号、数据独立的自托管学习工具,先邀请少量用户使用** | 首版按多用户归属和隔离设计;不自动加入公开注册、邀请链接或组织租户 |
| D06 | 旧数据 | 全量迁移是否需要未确认,CSV 与完整迁移不同 | 迁移另估 |
@@ -136,3 +136,13 @@ server 是 go-admin 的选用模块接入:原样保留 SysUser、SysDept、必
## 日志审计基线(#18)
管理端新增两个审计列表,后端为 LexGo 自有日志模型与接口,参照 go-admin 模块布局但不复制其原始参数/响应持久化逻辑。schema v2 新增 lexgo_login_logs、lexgo_operation_logs;保留 90 天,启动和每小时分批清理,也可显式 audit-cleanup。新建/重置密码均为 6~72 UTF-8 字节;初始 bootstrap 保留 10 字节下限。#2 已验收,#18 已验收;默认模块中的其他候选未纳入。
## 全 Go 正式架构决定(2026-09-11,#6)
用户已明确选择全 Go:正式英语分词、原文位置映射、本地词典解析和词形候选查询由 Go 后端完成,不运行 Python NLP 服务。此前“Python 建议/全 Go 未决”仅为历史决策记录,由本决定覆盖;spikes/english 保留历史验证,不接入产品。#6 按该方向实施,当前方案见工单最新启动评论;WordNet 3.0 仍为首个资源(英语释义),词形规则候选不等同于 spaCy 上下文消歧,原文及个人学习状态不按候选合并。
## #6 当前工程状态(2026-09-11)
#5已验收且相关前置PR均已合入main。#6正式词典/点词查义已按全Go实现,位于feat/6-go-dictionary-lookup,等待用户验收;server schema v4,管理端新增英语词典页,学习端加入Go分片及查词面板。正式Go进程不依赖Python NLP,WordNet包随数据库持久化。#3小样仍是历史验证;保存个人释义和状态归#7,音频封面#21、列表优化#24尚未实施。
+24 -2
View File
@@ -2,8 +2,8 @@
generated: true (请先修改 Gitea Wiki,禁止直接编辑本文件)
wiki_page: Architecture-and-Code-Map
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Architecture-and-Code-Map.-
wiki_revision: f981b86a7c6e5d2e353142d3006dcd6e0636c3ee
synchronized_at: 2026-09-11T03:01:48Z
wiki_revision: f317e21d651f2b5cb66810e9a4abf02831908559
synchronized_at: 2026-09-11T03:58:59Z
<!-- gitea-wiki-mirror:end -->
# 架构与代码地图
@@ -221,3 +221,25 @@ lexgo_books(owner_id, title, language)、lexgo_chapters(book_id, owner_id, ordin
- 重试自愈(R4):`retryChapter` 先把重试返回的章节状态应用到列表与阅读器并重新安排轮询,再做静默刷新。
验证:Go 全量用例 20 项通过(新增运行期恢复与尝试上限两项);学习端单测 38 项通过,其中 7 项在整改前的代码上复现失败;真实联调确认追加路径可用、被中断的任务在运行中被自动恢复(约 0.5 秒,无需重启)、重试在首次刷新失败后仍自动显示最终结果。
## 全 Go 正式架构决定(2026-09-11,#6)
用户已明确选择全 Go:正式英语分词、原文位置映射、本地词典解析和词形候选查询由 Go 后端完成,不运行 Python NLP 服务。此前“Python 建议/全 Go 未决”仅为历史决策记录,由本决定覆盖;spikes/english 保留历史验证,不接入产品。#6 按该方向实施,当前方案见工单最新启动评论;WordNet 3.0 仍为首个资源(英语释义),词形规则候选不等同于 spaCy 上下文消歧,原文及个人学习状态不按候选合并。
## #6 全 Go 词典与阅读器(2026-09-11,已实现待验收)
`server/app/lexgo/wordnet.go` 负责固定 WordNet ZIP 校验/内存解析、Unicode 分词及词形候选;`dictionary.go` 负责资源与章节查词 API;`database.go` schema v4 新增单槽共享资源表 lexgo_dictionaries(元数据、enabled、SHA、ZIP LONGBLOB),已有学习数据不改写。每个 Router 按 SHA 缓存一个不可变词典,查询先读资源元数据,缓存未命中才读取 ZIP;进程重启从数据库恢复,不需要 Python NLP 或额外资源目录。
| 接口 | 权限与输入/输出 |
|---|---|
| GET /api/v1/dictionaries | 管理员;items + supported,状态 ready/disabled/unavailable,不返回 archive 或本机路径 |
| POST /api/v1/dictionaries/import | 管理员;multipart name/language/version/source/format/file;返回 resource + duplicate;格式/来源/版本固定 |
| PATCH /api/v1/dictionaries/:id | 管理员;{enabled:boolean};返回 resource |
| GET /api/v1/chapters/:id/tokens | 本人 ready 章节;{textSha256,tokens:[{text,start,end,startUtf16,endUtf16,kind}]} |
| POST /api/v1/lookup | {chapterId,start,end},cp半开范围;本人ready完整单词;返回 status/query/matchedForm/candidates/entries/resource |
管理端 `Dictionaries.vue` + `dictionaries.mjs` + session multipart 方法,复用 go-admin 导航/表单与身份失效保护。学习端 `useReaderLookup.ts` 校验原文片段/SHA/所有位置,`ReaderTokens.vue` 渲染可聚焦单词,`LookupPanel.vue` 展示释义及临时个人草稿。桌面侧栏,手机固定底部45dvh面板;关闭恢复焦点,仅无后续手动滚动时恢复自动调整前位置。旧响应在换词/换章/退出/离页后失效。
参考:[WordNet 数据格式](https://wordnet.princeton.edu/documentation/wndb5wn)、[词形规则](https://wordnet.princeton.edu/documentation/morphy7wn)。#3 仅历史实验,#6 不调用其实验服务。
+18 -2
View File
@@ -2,8 +2,8 @@
generated: true (请先修改 Gitea Wiki,禁止直接编辑本文件)
wiki_page: Business-Rules-and-Glossary
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Business-Rules-and-Glossary.-
wiki_revision: 4bd843d1f0134f5688407b7072d98333b40b6da1
synchronized_at: 2026-09-11T03:01:48Z
wiki_revision: 935630005c9b1d7103550cf09eb10edfdbf8f3af
synchronized_at: 2026-09-11T03:59:01Z
<!-- gitea-wiki-mirror:end -->
# 业务规则与术语
@@ -137,3 +137,19 @@ POST /lookup 接收 {surface,lemma?}。查词键单独 casefold/NFC/弯撇号转
- 运行期恢复不依赖重启:除了启动时的恢复,运行中的服务每次轮询都会把停留在 processing 且超过 15 秒的任务放回 pending,因此“领取已提交、完成事务失败”不会让章节永久卡在处理中。该阈值必须长于正常的领取到完成窗口;重复处理同一任务不会产生第二个章节,因为任务从不创建章节。
- 学习端离开页面时作废在途请求:目录页与阅读页在关闭时推进各自的请求序号,晚到的响应不会写回状态或重启轮询;导入页在提交过程中离开后,晚到的成功响应不会把用户导航回书籍页。
- 重试结果立即生效:重试被接受后先把返回的 pending 状态写入界面并继续轮询,因此紧随其后的一次刷新失败不会让页面停在处理失败。
## 全 Go 正式架构决定(2026-09-11,#6)
用户已明确选择全 Go:正式英语分词、原文位置映射、本地词典解析和词形候选查询由 Go 后端完成,不运行 Python NLP 服务。此前“Python 建议/全 Go 未决”仅为历史决策记录,由本决定覆盖;spikes/english 保留历史验证,不接入产品。#6 按该方向实施,当前方案见工单最新启动评论;WordNet 3.0 仍为首个资源(英语释义),词形规则候选不等同于 spaCy 上下文消歧,原文及个人学习状态不按候选合并。
## #6 正式词典规则(2026-09-11)
首个资源固定 Princeton WordNet 3.0,英语释义;ZIP 来源及 SHA 以 server/wordnet-resource.json 为准,LICENSE 原样保留于 server/WORDNET-LICENSE.txt。管理员上传指定包并配置显示名称,语言/来源/版本/格式固定;32MiB压缩、128MiB解压上限、成员与结构校验。资源为所有账号共享、仅管理员写;失败保持旧资源,重复上传复用id并启用,停用后查询返回 resource_missing。ZIP 随数据库备份,无运行时网络下载。
正式分词由 Go 完成:字母开始词,组合标记延续词,内部直/弯撇号连接字母;空白逐字保留,数字和符号为不可查询的 punctuation。start/end 是 Unicode code point 半开范围,另有UTF-16偏移;原文、CRLF、组合字符与emoji不归一化。只对查询键小写/NFC/撇号归一化。每次查询必须是本人ready章节内的完整单词,最大128码点,跨账号404,非法范围400。
exact优先;未命中再按WordNet异常表/词尾规则查候选,词性顺序n/v/a/r、最多12条释义,保留原数据s词性。返回lemma表示规则候选,不承诺上下文消歧;歧义不会合并个人学习状态。not_found与resource_missing区分,网络错误可重试,不阻断阅读。#3使用spaCy上下文lemma的实验路径由全Go规则候选替代。
个人释义目前仅当前选择的临时草稿,界面标记未保存;换词、关闭、换章、离页和身份变化清理。持久化与个人词汇状态由#7实现,不把临时输入宣传为保存成功。
+17 -2
View File
@@ -2,8 +2,8 @@
generated: true (请先修改 Gitea Wiki,禁止直接编辑本文件)
wiki_page: Local-Development-and-Verification
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Local-Development-and-Verification.-
wiki_revision: 3bce5fae964b1050513c37319262679814d68f14
synchronized_at: 2026-09-11T03:01:49Z
wiki_revision: bbd6e618f4339aa45d81bba0492045b60581a735
synchronized_at: 2026-09-11T03:59:02Z
<!-- gitea-wiki-mirror:end -->
# 本地开发与验证
@@ -305,3 +305,18 @@ node --test spikes/english/view.test.mjs
注意:MySQL 会话时区为 SYSTEM(本机为 UTC+8),而服务按 UTC 存储 DATETIME;核对任务时间时使用 UTC_TIMESTAMP 而不是 NOW(),否则会出现 8 小时的假偏差。
未在本轮验证:R3 的真实浏览器时序(离页与响应同时发生)只由单测覆盖;真机手机证据仍属 #4 缺口。
## #6 部署与验证(2026-09-11)
正式后端 schema v4,只增加 lexgo_dictionaries。显式 migrate 后启动纯Go lexgo.exe;Python scripts/server.py 仍是开发命令封装,产品运行不依赖Python NLP。
本机已从v3升级v4并重启lexgo-api,升级前后sys_user/lexgo_spaces/lexgo_sessions/lexgo_books/lexgo_chapters/lexgo_ingest_jobs计数一致。旧二进制保存在忽略的 .local/lexgo-pre-issue6.exe。回退:停止API,恢复旧二进制,将已确认v4的schema标记恢复3,保留新增资源表及全部学习数据,再启动旧API;不要删除数据或重新bootstrap。
管理端 http://127.0.0.1:5174 的“英语词典”页可下载指定包并导入/启停;本机已导入固定WordNet3.0,155287个词形/词性索引项。学习端 http://127.0.0.1:5173 打开本人ready章节,点词或Enter/空格查询,Escape关闭。went/mice应出现go/mouse候选。源包、本机凭据与测试证据仅存在忽略的.local,不进入Git。
验证:Go全包MySQL集成(专用lexgo_test_issue5)与go vet通过;学习端53单测、类型检查/构建、默认5173 Playwright3项通过;管理端31单测与lint通过,构建含已有Sass弃用与bundle体积提示;治理56测试与strict通过。实际ZIP解析、schema3→4原文/任务保留、权限、失败保留、重复启用、冷缓存路由重建都有覆盖;冷缓存测试不是完整备份恢复演练。
主审真实API联调使用真实管理端session模块经5174代理上传;经5173代理两测试账号分别创建虚构章节并验证精确/不规则词形、Unicode原文片段、越权404/普通用户管理403、停用/重复导入启用、错误ZIP保留资源。新增测试书籍id3/4、章节id8/9归issue5_a/issue5_b,没有修改其他账号的书籍。
常驻5173一度返回空白页:Vue模块转换500、代理缺失;同代码隔离服务正常,只重启lexgo-learner加载配置后恢复,默认E2E通过,未改启动配置。桌面交互浏览器工具因旧会话失效未完成手工真实UI联调;已有项目Playwright使用模拟API,真实API验证另列。手机仅窄屏自动测试,真机证据仍未补齐。
+12 -2
View File
@@ -2,8 +2,8 @@
generated: true (请先修改 Gitea Wiki,禁止直接编辑本文件)
wiki_page: Product-Requirements-Overview
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Product-Requirements-Overview.-
wiki_revision: 125df697d3d45dee098616d25970bb65a96727cd
synchronized_at: 2026-09-10T16:35:19Z
wiki_revision: 099f1c803ffb69edc3fe07582d22dcf9a57647a1
synchronized_at: 2026-09-11T03:59:10Z
<!-- gitea-wiki-mirror:end -->
# 产品需求总览
@@ -229,3 +229,13 @@ Anki(U20)、YouTube/Jellyfin 远程字幕(U07/A08)、全量旧实例迁
2026-09-10:#5“粘贴英语文本,处理后进入本人章节阅读”已实现并待用户验收,覆盖 F01 基础(书库与章节)、F02(粘贴导入)、F04 基础(处理状态与失败重试)与 F05 原文(可读原文与章节切换)。本节取代此前“阅读、导入尚未实现”的表述:粘贴导入与原文阅读已实现;点词查词、词典、个人词语状态、复习与统计仍未实现(#6~#15)。schema 升级为 v3,新增 lexgo_books、lexgo_chapters、lexgo_ingest_jobs。
范围边界不变:本单按 Go 处理,不接入 Python NLP,因此不产生 token、lemma 或词典索引;Go+Python NLP 与全 Go 路线仍未确认,正式接入前必须由用户确认。真机手机详细证据仍缺失(#4 缺口保持)。
## 全 Go 正式架构决定(2026-09-11,#6)
用户已明确选择全 Go:正式英语分词、原文位置映射、本地词典解析和词形候选查询由 Go 后端完成,不运行 Python NLP 服务。此前“Python 建议/全 Go 未决”仅为历史决策记录,由本决定覆盖;spikes/english 保留历史验证,不接入产品。#6 按该方向实施,当前方案见工单最新启动评论;WordNet 3.0 仍为首个资源(英语释义),词形规则候选不等同于 spaCy 上下文消歧,原文及个人学习状态不按候选合并。
## #6 交付范围更新(2026-09-11)
用户确认全Go后,英语词典配置与阅读器点词查义已实现待验收:共享WordNet3.0英语释义、管理员导入/启停、本人章节点击/键盘查词、加载/无结果/资源不可用/网络失败/关闭状态。词形结果是规则候选,不提供上下文词性消歧。个人释义输入是未保存临时草稿,#7才持久化。沿用已验收v1;手机底部45dvh面板自动测试通过,真机缺口保留。#5已关闭并合入main,#6尚不关闭。
+7 -2
View File
@@ -2,8 +2,8 @@
generated: true (请先修改 Gitea Wiki,禁止直接编辑本文件)
wiki_page: Home
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Home
wiki_revision: 5b936b75caeec89cd54f7147c842f0b66f901307
synchronized_at: 2026-09-10T16:35:18Z
wiki_revision: 241fa63d8141ed9f1992421346d42f9b87453b0a
synchronized_at: 2026-09-11T03:58:55Z
<!-- gitea-wiki-mirror:end -->
# LexGo 文档入口
@@ -65,3 +65,8 @@ Quant-UX 原型 v1 已通过用户验收。[桌面预览](https://qux.ilapage.cn
#4 阅读选择小样已实现桌面鼠标/键盘、范围调整与原文位置验证,入口 http://127.0.0.1:5184/;固定版本 LinguaCafe 四项源码对照已记录。用户已验收并关闭 #4;真实手机详细测试证据仍缺失,详见本地验证页。#21 仍待实施。
#5 粘贴导入与章节阅读已实现,待用户验收:schema v3 新增 books/chapters/ingest_jobs,学习端具备粘贴导入、书库、处理状态与原文阅读;查词、词典与复习仍未实现。本单按 Go 处理,Go+Python NLP 与全 Go 路线仍未决,正式接入前须用户确认。
## 当前进度(2026-09-11)
#5已验收,#17/#19/#20/#22/#23按依赖顺序合入main。#6已按用户确认的全Go方向实现英语词典与阅读点词,待验收;原文/账户隔离保留。正式NLP不使用Python服务。管理端“英语词典”导入指定WordNet3.0,学习端打开本人章节即可查词。个人释义当前不保存,后续#7实现。详见#6工单、架构和本地开发页面。
+32
View File
@@ -52,6 +52,17 @@ test('paste English text, watch a chapter finish processing, then read it verbat
chapter: { ...chapterPayload(), contentSha256: 'fictional-sha256', ...(status === 'ready' ? { originalText: pasted } : {}) },
navigation: { previousChapterId: null, nextChapterId: null },
}
} else if (path === '/api/v1/chapters/55/tokens') {
let offset = 0
const tokens = (pasted.match(/[A-Za-z]+|\s+|[^A-Za-z\s]+/g) ?? []).map(text => {
const start = offset
offset += text.length
return { text, start, end: offset, startUtf16: start, endUtf16: offset, kind: /^[A-Za-z]+$/.test(text) ? 'word' : /^\s+$/.test(text) ? 'space' : 'punctuation' }
})
data = { textSha256: 'fictional-sha256', tokens }
} else if (path === '/api/v1/lookup') {
expect(route.request().postDataJSON()).toEqual({ chapterId: 55, start: 0, end: 5 })
data = { status: 'exact', query: 'first', matchedForm: 'first', candidates: [], entries: [{ lemma: 'first', pos: 'adjective', definition: 'Coming before all others.', examples: ['The first fictional chapter.'] }] }
}
await route.fulfill({ status: statusCode, json: { code: 200, data } })
})
@@ -89,4 +100,25 @@ test('paste English text, watch a chapter finish processing, then read it verbat
expect(await readerText.evaluate(element => getComputedStyle(element).whiteSpace)).toBe('pre-wrap')
await expect(page.getByRole('button', { name: '上一章' })).toBeDisabled()
await expect(page.getByRole('button', { name: '下一章' })).toBeDisabled()
const word = page.getByRole('button', { name: '查询 First', exact: true })
await word.focus()
await word.press('Enter')
await expect(page.getByText('Coming before all others.')).toBeVisible()
expect(await readerText.evaluate(element => element.textContent)).toBe(pasted)
await page.getByLabel('个人释义 未保存').fill('虚构的本地草稿')
await page.getByRole('button', { name: '关闭释义' }).press('Escape')
await expect(page.locator('.lookup-panel')).toHaveCount(0)
await expect(word).toBeFocused()
// Browser narrow viewport check only; this is not real-device acceptance.
await page.setViewportSize({ width: 390, height: 844 })
await word.click()
await expect(page.getByText('Coming before all others.')).toBeVisible()
const panelBounds = await page.locator('.lookup-panel').boundingBox()
expect(panelBounds!.y + panelBounds!.height).toBeLessThanOrEqual(845)
expect(panelBounds!.height).toBeLessThanOrEqual(844 * 0.46)
expect(await page.locator('.lookup-panel').evaluate(element => getComputedStyle(element).position)).toBe('fixed')
const wordBounds = await word.boundingBox()
expect(wordBounds!.y + wordBounds!.height).toBeLessThanOrEqual(panelBounds!.y)
expect(await page.getByLabel('个人释义 未保存').inputValue()).toBe('')
})
+167
View File
@@ -0,0 +1,167 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { flushPromises, mount, type VueWrapper } from '@vue/test-utils'
import { createPinia, setActivePinia } from 'pinia'
import { createMemoryHistory, createRouter } from 'vue-router'
import ReaderView from '../views/ReaderView.vue'
import { useSessionStore } from '../stores/session'
import { useReaderLookup } from '../composables/useReaderLookup'
import { effectScope, ref } from 'vue'
import type { ChapterDetail } from '../stores/library'
// Fictitious text includes astral, combining, CRLF and repeated whitespace.
const original = '😀 Cats\r\n café!'
const fragments = [['😀', 'punctuation'], [' ', 'space'], ['Cats', 'word'], ['\r\n ', 'space'], ['café', 'word'], ['!', 'punctuation']]
let cp = 0, utf16 = 0
const tokens = fragments.map(([text = '', kind]) => {
const token = { text, kind, start: cp, end: cp + [...text].length, startUtf16: utf16, endUtf16: utf16 + text.length }
cp = token.end; utf16 = token.endUtf16
return token
})
const chapter = { id: 55, bookId: 1, title: '虚构章节', status: 'ready', originalText: original, contentSha256: 'same-sha' }
const ok = (data: unknown) => new Response(JSON.stringify({ code: 200, data }))
const result = (query: string, status = 'exact') => ({ status, query, matchedForm: query, candidates: [], entries: status === 'exact' ? [{ lemma: query, pos: 'noun', definition: `Definition of ${query}`, examples: ['A fictional example.'] }] : [] })
let wrapper: VueWrapper | undefined
async function open(mockLookup: (body: Record<string, number>) => Promise<Response> = async () => ok(result('cat')), tokenData: unknown = { textSha256: 'same-sha', tokens }) {
const fetchMock = vi.spyOn(globalThis, 'fetch').mockImplementation(async (input, options) => {
if (String(input).endsWith('/tokens')) return ok(tokenData)
if (String(input).endsWith('/lookup')) return mockLookup(JSON.parse(String(options?.body)))
return ok({ chapter, navigation: { previousChapterId: null, nextChapterId: 56 } })
})
useSessionStore().user = { id: 42, username: 'fictional', role: 'learner' }
const router = createRouter({ history: createMemoryHistory(), routes: [{ path: '/chapters/:id', component: ReaderView }, { path: '/', component: { template: '<div />' } }] })
await router.push('/chapters/55')
wrapper = mount(ReaderView, { attachTo: document.body, global: { plugins: [router] } })
await flushPromises()
return { fetchMock, router, view: wrapper }
}
describe('reader word lookup', () => {
beforeEach(() => { setActivePinia(createPinia()); sessionStorage.clear() })
afterEach(() => { wrapper?.unmount(); wrapper = undefined; vi.restoreAllMocks() })
it('keeps Unicode original text exact and sends only chapter and code-point offsets', async () => {
const { view, fetchMock } = await open()
expect(view.get('.reader-text').element.textContent).toBe(original)
const words = view.findAll('.reader-word')
expect(words).toHaveLength(2)
await words[0]!.trigger('click'); await flushPromises()
const call = fetchMock.mock.calls.find(([url]) => String(url).endsWith('/lookup'))!
expect(call[1]?.method).toBe('POST')
expect(JSON.parse(String(call[1]?.body))).toEqual({ chapterId: 55, start: 2, end: 6 })
expect(view.text()).toContain('Definition of cat')
expect(view.text()).toContain('A fictional example.')
})
it('discards older word responses and clears the selection draft', async () => {
let finish!: (response: Response) => void
const { view } = await open(body => body.start === 2 ? new Promise(resolve => { finish = resolve }) : Promise.resolve(ok(result('cafe', 'not_found'))))
await view.findAll('.reader-word')[0]!.trigger('click')
expect(view.text()).toContain('正在查询')
await view.findAll('.reader-word')[1]!.trigger('click'); await flushPromises()
await view.get('textarea').setValue('虚构私人草稿')
finish(ok(result('OLD'))); await flushPromises()
expect(view.text()).not.toContain('Definition of OLD')
expect(view.text()).toContain('未找到释义')
expect(view.text()).toContain('未保存')
await view.findAll('.reader-word')[0]!.trigger('click')
expect((view.get('textarea').element as HTMLTextAreaElement).value).toBe('')
})
it.each(['resource_missing', 'not_found', 'error'])('keeps reading and permits a local draft and retry for %s', async status => {
const { view, fetchMock } = await open(async () => { if (status === 'error') throw new Error('暂时无法查询'); return ok(result('cat', status)) })
await view.get('.reader-word').trigger('click'); await flushPromises()
await view.get('textarea').setValue('仅当前词的虚构草稿')
expect(view.get('.reader-text').element.textContent).toBe(original)
expect(view.text()).toContain('未保存')
expect(fetchMock.mock.calls.filter(([url]) => String(url).endsWith('/lookup'))).toHaveLength(1)
await view.get('[data-testid="lookup-retry"]').trigger('click'); await flushPromises()
expect(fetchMock.mock.calls.filter(([url]) => String(url).endsWith('/lookup'))).toHaveLength(2)
})
it.each([{ textSha256: 'wrong', tokens }, { textSha256: 'same-sha', tokens: tokens.slice(1) }])('falls back to original when tokens do not match', async data => {
const { view } = await open(undefined, data)
expect(view.get('.reader-text').element.textContent).toBe(original)
expect(view.find('.reader-word').exists()).toBe(false)
expect(view.find('[data-testid="tokens-retry"]').exists()).toBe(true)
})
it('closes with Escape and restores the word focus without scrolling', async () => {
const { view } = await open()
const word = view.get('.reader-word').element as HTMLElement
word.focus()
await view.get('.reader-word').trigger('keydown', { key: 'Enter' }); await flushPromises()
const focus = vi.spyOn(word, 'focus')
await view.get('.lookup-panel').trigger('keydown', { key: 'Escape' })
await flushPromises()
expect(view.find('.lookup-panel').exists()).toBe(false)
expect(document.activeElement).toBe(word)
expect(focus).toHaveBeenCalledWith({ preventScroll: true })
})
it('retries a failed token fetch without losing original text', async () => {
const { view, fetchMock } = await open()
fetchMock.mockRejectedValueOnce(new Error('网络暂不可用'))
// A new chapter load invalidates the old token rendering.
const { useLibraryStore } = await import('../stores/library')
useLibraryStore().chapter = { ...useLibraryStore().chapter!, contentSha256: 'retry-sha' }
await flushPromises()
expect(view.get('.reader-text').element.textContent).toBe(original)
expect(view.find('.reader-word').exists()).toBe(false)
fetchMock.mockResolvedValueOnce(ok({ textSha256: 'retry-sha', tokens }))
await view.get('[data-testid="tokens-retry"]').trigger('click'); await flushPromises()
expect(view.findAll('.reader-word')).toHaveLength(2)
})
it('does not accept a late token response after identity changes', async () => {
const { view, fetchMock } = await open(undefined, { textSha256: 'wrong', tokens })
let finish!: (response: Response) => void
fetchMock.mockImplementationOnce(() => new Promise(resolve => { finish = resolve }))
await view.get('[data-testid="tokens-retry"]').trigger('click')
await useSessionStore().logout()
useSessionStore().user = { id: 42, username: 'fictional', role: 'learner' }
finish(ok({ textSha256: 'same-sha', tokens })); await flushPromises()
expect(view.find('.reader-word').exists()).toBe(false)
})
it('moves a covered selected word above the sheet and restores the prior scroll on close', async () => {
useSessionStore().user = { id: 42, username: 'fictional', role: 'learner' }
vi.spyOn(globalThis, 'fetch').mockResolvedValue(ok(result('cat')))
const scrollBy = vi.spyOn(window, 'scrollBy').mockImplementation(() => {})
const scrollTo = vi.spyOn(window, 'scrollTo').mockImplementation(() => {})
const scope = effectScope()
const lookup = scope.run(() => useReaderLookup(ref(chapter as ChapterDetail)))!
const element = document.createElement('span')
vi.spyOn(element, 'getBoundingClientRect').mockReturnValue({ bottom: 700 } as DOMRect)
lookup.select(tokens[2] as never, element)
lookup.keepSelectionVisible(500)
expect(scrollBy).toHaveBeenCalledWith({ top: 216, behavior: 'instant' })
lookup.close()
expect(scrollTo).toHaveBeenCalledWith({ top: 0, left: 0, behavior: 'instant' })
scope.stop()
})
it('preserves a later manual reading scroll when closing the sheet', () => {
useSessionStore().user = { id: 42, username: 'fictional', role: 'learner' }
vi.spyOn(globalThis, 'fetch').mockResolvedValue(ok(result('cat')))
let scrollY = 100
vi.spyOn(window, 'scrollY', 'get').mockImplementation(() => scrollY)
vi.spyOn(window, 'scrollBy').mockImplementation((...args: unknown[]) => {
const options = args[0] as ScrollToOptions
scrollY += options.top ?? 0
})
const scrollTo = vi.spyOn(window, 'scrollTo').mockImplementation(() => {})
const scope = effectScope()
const lookup = scope.run(() => useReaderLookup(ref(chapter as ChapterDetail)))!
const element = document.createElement('span')
vi.spyOn(element, 'getBoundingClientRect').mockReturnValue({ bottom: 700 } as DOMRect)
lookup.select(tokens[2] as never, element)
lookup.keepSelectionVisible(500)
scrollY += 300 // The reader continues down the page while the sheet is open.
lookup.close()
expect(scrollTo).not.toHaveBeenCalled()
expect(scrollY).toBe(616)
scope.stop()
})
it.each(['logout', 'chapter', 'unmount'])('invalidates pending lookup on %s', async action => {
let finish!: (response: Response) => void
const { view, router } = await open(() => new Promise(resolve => { finish = resolve }))
await view.get('.reader-word').trigger('click')
if (action === 'logout') await useSessionStore().logout()
else if (action === 'chapter') await router.push('/chapters/56')
else { view.unmount(); wrapper = undefined }
finish(ok(result('OLD'))); await flushPromises()
if (action === 'unmount') expect(document.querySelector('.lookup-panel')).toBeNull()
else expect(view.find('.lookup-panel').exists()).toBe(false)
expect(view.text()).not.toContain('Definition of OLD')
})
})
+55
View File
@@ -0,0 +1,55 @@
<script setup lang="ts">
import { onMounted, onUnmounted, ref } from 'vue'
import { ElButton } from 'element-plus'
import type { LookupResult } from '../composables/useReaderLookup'
defineProps<{ word: string; result: LookupResult | null; loading: boolean; error: string }>()
const draft = defineModel<string>('draft', { required: true })
const emit = defineEmits<{ close: []; retry: []; resize: [top: number] }>()
const heading = ref<HTMLElement | null>(null)
const panel = ref<HTMLElement | null>(null)
let observer: ResizeObserver | undefined
function resized() {
if (window.innerWidth <= 760 && panel.value) emit('resize', panel.value.getBoundingClientRect().top)
}
onMounted(() => {
heading.value?.focus({ preventScroll: true })
resized()
if (typeof ResizeObserver !== 'undefined' && panel.value) {
observer = new ResizeObserver(resized)
observer.observe(panel.value)
}
window.addEventListener('resize', resized)
})
onUnmounted(() => { observer?.disconnect(); window.removeEventListener('resize', resized) })
</script>
<template>
<aside ref="panel" class="lookup-panel" aria-labelledby="lookup-heading" @keydown.esc.stop.prevent="$emit('close')">
<header class="lookup-heading">
<h2 id="lookup-heading" ref="heading" tabindex="-1">{{ word }}</h2>
<ElButton text aria-label="关闭释义" @click="$emit('close')">关闭</ElButton>
</header>
<div class="lookup-content" aria-live="polite" :aria-busy="loading">
<p v-if="loading" role="status" class="subtle">正在查询…</p>
<p v-else-if="error" role="alert" class="lookup-message">{{ error }}</p>
<template v-else-if="result">
<p v-if="result.status === 'resource_missing'" class="lookup-message">词典资源暂不可用。</p>
<p v-else-if="result.status === 'not_found'" class="lookup-message">未找到释义。</p>
<p v-if="result.status === 'lemma'" class="subtle">词形匹配:{{ result.matchedForm }}(规则候选)</p>
<ol v-if="result.entries.length" class="lookup-senses">
<li v-for="(entry, index) in result.entries" :key="index">
<p class="sense-heading"><strong>{{ entry.lemma }}</strong> <span>{{ entry.pos }}</span></p>
<p lang="en">{{ entry.definition }}</p>
<blockquote v-for="(example, exampleIndex) in entry.examples" :key="exampleIndex" lang="en">{{ example }}</blockquote>
</li>
</ol>
<p v-if="result.resource" class="subtle">{{ result.resource.name }} · {{ result.resource.version }}</p>
</template>
<ElButton v-if="!loading && (error || result?.status === 'resource_missing' || result?.status === 'not_found')" data-testid="lookup-retry" @click="$emit('retry')">重试查询</ElButton>
</div>
<div class="lookup-draft">
<label for="definition-draft">个人释义 <span>未保存</span></label>
<textarea id="definition-draft" v-model="draft" rows="3" placeholder="记下当前词的释义" />
</div>
</aside>
</template>
+10
View File
@@ -0,0 +1,10 @@
<script setup lang="ts">
import type { ReaderToken } from '../composables/useReaderLookup'
defineProps<{ tokens: ReaderToken[]; original: string; selectedStart?: number }>()
const emit = defineEmits<{ select: [token: ReaderToken, element: HTMLElement] }>()
function select(token: ReaderToken, event: Event) { emit('select', token, event.currentTarget as HTMLElement) }
</script>
<template>
<article class="reader-text"><template v-if="tokens.length"><template v-for="token in tokens" :key="token.start"><span v-if="token.kind === 'word'" role="button" tabindex="0" class="reader-word" :class="{ 'is-selected': selectedStart === token.start }" :aria-label="`查询 ${token.text}`" :aria-pressed="selectedStart === token.start" @click="select(token, $event)" @keydown.enter.prevent="select(token, $event)" @keydown.space.prevent="select(token, $event)">{{ token.text }}</span><template v-else>{{ token.text }}</template></template></template><template v-else>{{ original }}</template></article>
</template>
+128
View File
@@ -0,0 +1,128 @@
import { onScopeDispose, ref, watch, type Ref } from 'vue'
import type { ChapterDetail } from '../stores/library'
import { useSessionStore } from '../stores/session'
export interface ReaderToken { text: string; start: number; end: number; startUtf16: number; endUtf16: number; kind: 'word' | 'space' | 'punctuation' }
export interface LookupResult {
status: 'exact' | 'lemma' | 'not_found' | 'resource_missing'
query: string
matchedForm: string | null
candidates: string[]
entries: { lemma: string; pos: string; definition: string; examples: string[] }[]
resource?: { name: string; version: string }
}
interface TokenResponse { textSha256: string; tokens: ReaderToken[] }
// Validate every coordinate before enabling selection; original text is always the fallback.
function matchesChapter(data: TokenResponse, chapter: ChapterDetail): boolean {
if (!data || data.textSha256 !== chapter.contentSha256 || !Array.isArray(data.tokens)) return false
const text = chapter.originalText ?? ''
let cp = 0, utf16 = 0
for (const token of data.tokens) {
if (typeof token.text !== 'string' || !token.text || !['word', 'space', 'punctuation'].includes(token.kind)) return false
if (token.start !== cp || token.startUtf16 !== utf16) return false
cp += [...token.text].length
utf16 += token.text.length
if (token.end !== cp || token.endUtf16 !== utf16 || text.slice(token.startUtf16, utf16) !== token.text) return false
}
return utf16 === text.length && data.tokens.map(token => token.text).join('') === text
}
export function useReaderLookup(chapter: Ref<ChapterDetail | null>) {
const session = useSessionStore()
const tokens = ref<ReaderToken[]>([])
const tokensError = ref('')
const tokensLoading = ref(false)
const selected = ref<ReaderToken | null>(null)
const result = ref<LookupResult | null>(null)
const loading = ref(false)
const error = ref('')
const draft = ref('')
let tokenSequence = 0, lookupSequence = 0
let origin: HTMLElement | null = null
let scrollBeforeAdjustment: { top: number; left: number } | null = null
let lastAdjustment: { top: number; left: number } | null = null
function stillAtAdjustment() {
return lastAdjustment !== null && Math.abs(window.scrollY - lastAdjustment.top) <= 2
&& Math.abs(window.scrollX - lastAdjustment.left) <= 2
}
function keepSelectionVisible(panelTop: number) {
if (!origin) return
const coveredBy = origin.getBoundingClientRect().bottom - panelTop + 16
if (coveredBy <= 0) return
// A later manual scroll becomes the new reading position, including when
// another resize subsequently needs to reveal the selected word again.
if (!scrollBeforeAdjustment || !stillAtAdjustment()) scrollBeforeAdjustment = { top: window.scrollY, left: window.scrollX }
window.scrollBy({ top: coveredBy, behavior: 'instant' })
lastAdjustment = { top: window.scrollY, left: window.scrollX }
}
function close(restoreFocus = true) {
lookupSequence++
selected.value = null
result.value = null
loading.value = false
error.value = ''
draft.value = ''
if (restoreFocus && origin?.isConnected) origin.focus({ preventScroll: true })
if (restoreFocus && scrollBeforeAdjustment && stillAtAdjustment()) window.scrollTo({ ...scrollBeforeAdjustment, behavior: 'instant' })
scrollBeforeAdjustment = null
lastAdjustment = null
origin = null
}
function reset() {
tokenSequence++
tokens.value = []
tokensError.value = ''
tokensLoading.value = false
close(false)
}
async function loadTokens() {
const current = chapter.value
if (!session.user || current?.status !== 'ready') return
const seq = ++tokenSequence
tokensLoading.value = true
tokensError.value = ''
try {
const data = await session.request<TokenResponse>(`chapters/${current.id}/tokens`)
if (seq !== tokenSequence) return
if (!matchesChapter(data, current)) throw new Error('分词与正文不一致,请重试。')
tokens.value = data.tokens
} catch (reason) {
if (seq !== tokenSequence) return
tokens.value = []
tokensError.value = reason instanceof Error ? reason.message : '单词暂时无法加载。'
} finally { if (seq === tokenSequence) tokensLoading.value = false }
}
async function lookup() {
const current = chapter.value
const token = selected.value
if (!session.user || !current || !token) return
const seq = ++lookupSequence
result.value = null
error.value = ''
loading.value = true
try {
const data = await session.request<LookupResult>('lookup', 'POST', { chapterId: current.id, start: token.start, end: token.end })
if (seq === lookupSequence) result.value = data
} catch (reason) {
if (seq === lookupSequence) error.value = reason instanceof Error ? reason.message : '暂时无法查询,请重试。'
} finally { if (seq === lookupSequence) loading.value = false }
}
function select(token: ReaderToken, element: HTMLElement) {
close(false)
origin = element
selected.value = token
void lookup()
}
watch(() => [chapter.value?.id, chapter.value?.status, chapter.value?.contentSha256, chapter.value?.originalText], () => {
reset()
void loadTokens()
}, { immediate: true, flush: 'sync' })
// Watch the identity object, including clear → login for the same account.
watch(() => session.user, reset, { flush: 'sync' })
onScopeDispose(reset)
return { tokens, tokensError, tokensLoading, selected, result, loading, error, draft, loadTokens, lookup, select, close, reset, keepSelectionVisible }
}
+27
View File
@@ -95,6 +95,29 @@ a.chapter-name:hover { color: #315c43; text-decoration: underline; }
.reader-text { white-space: pre-wrap; overflow-wrap: break-word; margin: 26px 0 0; font-family: Georgia, 'Times New Roman', 'Microsoft YaHei', serif; font-size: 17px; line-height: 2; }
.processing-hint { padding: 36px 0; color: #748073; }
.reader-nav { display: flex; align-items: center; justify-content: space-between; gap: 14px; flex-wrap: wrap; margin-top: 36px; padding-top: 22px; border-top: 1px solid #e0e3d8; }
.reader-page.has-lookup { max-width: 1120px; }
.reader-workspace { display: grid; grid-template-columns: minmax(0, 1fr); gap: 32px; align-items: start; }
.has-lookup .reader-workspace { grid-template-columns: minmax(0, 1fr) 320px; }
.reader-body { min-width: 0; }
.reader-word { cursor: pointer; border-radius: 3px; }
.reader-word:hover, .reader-word.is-selected { background: #e3e9d9; color: #264a35; }
.reader-word:focus-visible { outline: 2px solid #bc803d; outline-offset: 2px; background: #eef2eb; }
.tokens-notice { color: #748073; font-size: 14px; margin-top: 24px; }
.lookup-panel { position: sticky; top: 24px; margin-top: 26px; padding: 22px; border: 1px solid #d9decf; border-radius: 12px; background: #fffdf8; max-height: calc(100dvh - 48px); overflow-y: auto; overflow-wrap: anywhere; }
.lookup-heading { display: flex; align-items: start; justify-content: space-between; gap: 10px; border-bottom: 1px solid #e0e3d8; padding-bottom: 12px; }
.lookup-heading h2 { margin: 6px 0; font-family: Georgia, serif; font-size: 25px; }
.lookup-content { font-size: 15px; line-height: 1.7; }
.lookup-message { color: #8b4324; }
.lookup-senses { padding-left: 22px; }
.lookup-senses li { padding-left: 3px; margin-bottom: 20px; }
.lookup-senses p { margin: 8px 0; }
.sense-heading span { color: #748073; font-size: 13px; }
.lookup-senses blockquote { border-left: 2px solid #cbd9c9; margin: 10px 0; padding-left: 12px; color: #687568; font-style: italic; }
.lookup-draft { border-top: 1px solid #e0e3d8; padding-top: 18px; margin-top: 20px; }
.lookup-draft label { display: flex; justify-content: space-between; gap: 10px; font-size: 14px; }
.lookup-draft label span { color: #8b794e; font-size: 12px; }
.lookup-draft textarea { width: 100%; resize: vertical; padding: 12px; margin-top: 10px; font: inherit; line-height: 1.6; color: #233d31; border: 1px solid #d6dccf; border-radius: 8px; background: #fffefa; }
.lookup-draft textarea:focus-visible { outline: 2px solid #bc803d; outline-offset: 2px; }
@media (max-width: 760px) {
.login-page { grid-template-columns: 1fr; }
.welcome { padding: 28px; }
@@ -116,5 +139,9 @@ a.chapter-name:hover { color: #315c43; text-decoration: underline; }
.chapter-row { grid-template-columns: 30px minmax(0, 1fr); align-items: start; row-gap: 10px; padding: 15px 16px; }
.chapter-row .status-chip, .chapter-row .el-button { grid-column: 2; justify-self: start; }
.reader-text { font-size: 16px; line-height: 1.95; }
.has-lookup .reader-workspace { grid-template-columns: minmax(0, 1fr); gap: 20px; }
.reader-page.has-lookup { padding-bottom: calc(45dvh + 28px); }
.lookup-panel { position: fixed; inset: auto 0 0; z-index: 20; max-height: 45dvh; margin-top: 0; padding: 16px 20px max(20px, env(safe-area-inset-bottom)); border-radius: 16px 16px 0 0; box-shadow: 0 -5px 24px #233d3114; }
.lookup-heading { position: sticky; top: -16px; z-index: 1; background: #fffdf8; }
.reader-nav .el-button { flex: 1; }
}
+16 -2
View File
@@ -4,6 +4,9 @@ import { RouterLink, useRoute, useRouter } from 'vue-router'
import { ElButton } from 'element-plus'
import { canRetry, statusLabel, useLibraryStore } from '../stores/library'
import { useSessionStore } from '../stores/session'
import { useReaderLookup } from '../composables/useReaderLookup'
import ReaderTokens from '../components/ReaderTokens.vue'
import LookupPanel from '../components/LookupPanel.vue'
const session = useSessionStore()
const library = useLibraryStore()
@@ -13,10 +16,12 @@ const retryError = ref('')
const chapterId = computed(() => Number(route.params.id))
const chapter = computed(() => library.chapter)
const lookup = useReaderLookup(chapter)
// Retry uses the job id the chapter carries, no matter where it was loaded from.
const retryable = computed(() => chapter.value !== null && canRetry(chapter.value))
async function load() {
lookup.reset()
retryError.value = ''
await library.loadChapter(chapterId.value)
}
@@ -56,7 +61,7 @@ onUnmounted(() => library.closeChapter())
<ElButton text @click="logout">退出登录</ElButton>
</div>
</header>
<main class="page reader-page">
<main class="page reader-page" :class="{ 'has-lookup': lookup.selected.value }" @keydown.esc="lookup.close()">
<p v-if="library.chapterLoading && !chapter" role="status" class="loading">正在加载…</p>
<div v-else-if="library.chapterError" class="notice">
<p role="alert">{{ library.chapterError }}</p>
@@ -80,7 +85,16 @@ onUnmounted(() => library.closeChapter())
</div>
<p v-else-if="chapter.status !== 'ready'" role="status" class="processing-hint">这一章还在{{ statusLabel(chapter.status) }},页面会自动刷新。</p>
<p v-if="retryError" role="alert" class="notice">{{ retryError }}</p>
<article v-if="chapter.status === 'ready'" class="reader-text">{{ library.readerText }}</article>
<div v-if="chapter.status === 'ready'" class="reader-workspace">
<div class="reader-body">
<ReaderTokens :tokens="lookup.tokens.value" :original="library.readerText" :selected-start="lookup.selected.value?.start" @select="lookup.select" />
<div v-if="lookup.tokensError.value" class="tokens-notice">
<p role="status">{{ lookup.tokensError.value }}</p>
<ElButton data-testid="tokens-retry" :loading="lookup.tokensLoading.value" @click="lookup.loadTokens">重试加载单词</ElButton>
</div>
</div>
<LookupPanel v-if="lookup.selected.value" :word="lookup.selected.value.text" :result="lookup.result.value" :loading="lookup.loading.value" :error="lookup.error.value" v-model:draft="lookup.draft.value" @close="lookup.close()" @retry="lookup.lookup" @resize="lookup.keepSelectionVisible" />
</div>
<nav class="reader-nav" aria-label="章节切换">
<ElButton :disabled="library.navigation.previousChapterId === null" @click="goTo(library.navigation.previousChapterId)">上一章</ElButton>
<ElButton :disabled="library.navigation.nextChapterId === null" @click="goTo(library.navigation.nextChapterId)">下一章</ElButton>
+31
View File
@@ -0,0 +1,31 @@
WordNet Release 3.0
This software and database is being provided to you, the LICENSEE, by
Princeton University under the following license. By obtaining, using
and/or copying this software and database, you agree that you have
read, understood, and will comply with these terms and conditions.:
Permission to use, copy, modify and distribute this software and
database and its documentation for any purpose and without fee or
royalty is hereby granted, provided that you agree to comply with
the following copyright notice and statements, including the disclaimer,
and that the same appear on ALL copies of the software, database and
documentation, including modifications that you make for internal
use or for distribution.
WordNet 3.0 Copyright 2006 by Princeton University. All rights reserved.
THIS SOFTWARE AND DATABASE IS PROVIDED "AS IS" AND PRINCETON
UNIVERSITY MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PRINCETON
UNIVERSITY MAKES NO REPRESENTATIONS OR WARRANTIES OF MERCHANT-
ABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE OR THAT THE USE
OF THE LICENSED SOFTWARE, DATABASE OR DOCUMENTATION WILL NOT
INFRINGE ANY THIRD PARTY PATENTS, COPYRIGHTS, TRADEMARKS OR
OTHER RIGHTS.
The name of Princeton University or Princeton may not be used in
advertising or publicity pertaining to distribution of the software
and/or database. Title to copyright in this software, database and
any associated documentation shall at all times remain with
Princeton University and LICENSEE agrees to preserve same.
+15 -1
View File
@@ -72,6 +72,9 @@ func Migrate(db *gorm.DB) error {
if current < 3 {
statements = append(statements, schemaV3Statements...)
}
if current < 4 {
statements = append(statements, schemaV4Statements...)
}
for i, s := range statements {
if _, err = conn.ExecContext(ctx, s); err != nil {
var sqlErr *driver.MySQLError
@@ -85,7 +88,18 @@ func Migrate(db *gorm.DB) error {
return err
}
const schemaVersion = 3
const schemaVersion = 4
var schemaV4Statements = []string{
`CREATE TABLE IF NOT EXISTS lexgo_dictionaries (
id BIGINT PRIMARY KEY, name VARCHAR(120) NOT NULL, language VARCHAR(16) NOT NULL,
version VARCHAR(32) NOT NULL, source VARCHAR(512) NOT NULL, format VARCHAR(32) NOT NULL,
sha256 CHAR(64) CHARACTER SET ascii COLLATE ascii_bin NOT NULL,
entry_count INT NOT NULL, enabled BOOLEAN NOT NULL DEFAULT TRUE,
archive LONGBLOB NOT NULL, updated_at DATETIME(3) NOT NULL,
CHECK (id = 1), CHECK (language = 'en')
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`,
}
var schemaV2Statements = []string{
`CREATE TABLE IF NOT EXISTS sys_user (
+320
View File
@@ -0,0 +1,320 @@
package lexgo
import (
"errors"
"io"
"net/http"
"strconv"
"strings"
"sync"
"time"
"unicode"
"unicode/utf8"
"github.com/gin-gonic/gin"
admin "go-admin/app/admin/models"
"gorm.io/gorm"
"gorm.io/gorm/clause"
)
// ID 1 is the sole English corpus slot. Keeping the validated ZIP in MySQL
// makes ordinary database backups include the resource needed after a restart.
type DictionaryResource struct {
ID int64 `gorm:"primaryKey;autoIncrement:false"`
Name string
Language string
Version string
Source string
Format string
SHA256 string `gorm:"column:sha256"`
EntryCount int
Enabled bool
Archive []byte
UpdatedAt time.Time
}
func (DictionaryResource) TableName() string { return "lexgo_dictionaries" }
type DictionaryView struct {
ID int64 `json:"id"`
Name string `json:"name"`
Language string `json:"language"`
Version string `json:"version"`
Source string `json:"source"`
Format string `json:"format"`
Status string `json:"status"`
Enabled bool `json:"enabled"`
SHA256 string `json:"sha256"`
EntryCount int `json:"entryCount"`
UpdatedAt time.Time `json:"updatedAt"`
}
func dictionaryView(r DictionaryResource, status string) DictionaryView {
return DictionaryView{r.ID, r.Name, r.Language, r.Version, r.Source, r.Format, status, r.Enabled, r.SHA256, r.EntryCount, r.UpdatedAt}
}
type DictionaryImportResult struct {
Resource DictionaryView `json:"resource"`
Duplicate bool `json:"duplicate"`
}
type ChapterTokens struct {
TextSHA256 string `json:"textSha256"`
Tokens []TextToken `json:"tokens"`
}
// Each router keeps at most one immutable parsed corpus; no private chapter or
// lookup data enters the cache. The mutex also coalesces simultaneous cold loads.
type dictionaryCache struct {
mu sync.Mutex
sha string
engine *WordNet
}
func (cache *dictionaryCache) load(tx *gorm.DB, r DictionaryResource) (*WordNet, error) {
cache.mu.Lock()
defer cache.mu.Unlock()
if r.SHA256 != WordNetSHA {
return nil, errors.New("unsupported resource checksum")
}
if cache.sha == r.SHA256 && cache.engine != nil {
return cache.engine, nil
}
var stored DictionaryResource
if err := tx.Select("id", "archive").Where("id = ? AND sha256 = ?", r.ID, r.SHA256).First(&stored).Error; err != nil {
return nil, err
}
engine, err := ParseWordNet(stored.Archive)
if err != nil {
return nil, err
}
cache.sha = r.SHA256
cache.engine = engine
return engine, nil
}
func readyOwnedChapter(tx *gorm.DB, owner int, id int64) (Chapter, error) {
var chapter Chapter
err := tx.Where("id = ? AND owner_id = ?", id, owner).First(&chapter).Error
if errors.Is(err, gorm.ErrRecordNotFound) {
return chapter, failure(404, "章节不存在")
}
if err != nil {
return chapter, err
}
if chapter.Status != statusReady {
return chapter, failure(409, "章节尚未就绪,请稍后重试")
}
return chapter, nil
}
func readDictionaryUpload(c *gin.Context) (DictionaryResource, error) {
bad := failure(400, "词典上传无效,请使用指定的 WordNet 3.0 ZIP 和完整资源信息")
c.Request.Body = http.MaxBytesReader(c.Writer, c.Request.Body, maxDictionaryZip+(64<<10))
reader, err := c.Request.MultipartReader()
if err != nil {
return DictionaryResource{}, bad
}
fields := map[string]string{}
var archive []byte
for {
part, err := reader.NextPart()
if err == io.EOF {
break
}
if err != nil {
return DictionaryResource{}, bad
}
name := part.FormName()
if name == "file" {
if archive != nil || part.FileName() == "" {
part.Close()
return DictionaryResource{}, bad
}
archive, err = io.ReadAll(io.LimitReader(part, maxDictionaryZip+1))
if err != nil || len(archive) == 0 || len(archive) > maxDictionaryZip {
part.Close()
return DictionaryResource{}, bad
}
} else {
if part.FileName() != "" || (name != "name" && name != "language" && name != "version" && name != "source" && name != "format") {
part.Close()
return DictionaryResource{}, bad
}
if _, exists := fields[name]; exists {
part.Close()
return DictionaryResource{}, bad
}
value, e := io.ReadAll(io.LimitReader(part, 1025))
if e != nil || len(value) > 1024 || !utf8.Valid(value) {
part.Close()
return DictionaryResource{}, bad
}
fields[name] = string(value)
}
part.Close()
}
name := strings.TrimSpace(fields["name"])
if name == "" || utf8.RuneCountInString(name) > 120 || strings.IndexFunc(name, unicode.IsControl) >= 0 || fields["language"] != "en" || fields["version"] != "3.0" || fields["format"] != "wordnet-3.0-zip" || fields["source"] != WordNetSource || len(archive) == 0 {
return DictionaryResource{}, bad
}
return DictionaryResource{ID: 1, Name: name, Language: "en", Version: "3.0", Source: WordNetSource, Format: "wordnet-3.0-zip", SHA256: WordNetSHA, Enabled: true, Archive: archive}, nil
}
func registerDictionaryRoutes(v *gin.RouterGroup, protect func(bool, func(*gin.Context, *gorm.DB, admin.SysUser) (any, error)) gin.HandlerFunc, now func() time.Time) {
cache := &dictionaryCache{}
uploadGate := make(chan struct{}, 1)
v.GET("/dictionaries", protect(true, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) {
if c.Request.URL.RawQuery != "" {
return nil, failure(400, "词典列表不接受查询参数")
}
var rows []DictionaryResource
if err := tx.Omit("archive").Order("id").Find(&rows).Error; err != nil {
return nil, err
}
items := make([]DictionaryView, 0, len(rows))
for _, row := range rows {
status := "disabled"
if row.Enabled {
status = "ready"
if _, err := cache.load(tx, row); err != nil {
status = "unavailable"
}
}
items = append(items, dictionaryView(row, status))
}
return gin.H{"items": items, "supported": gin.H{"name": "Princeton WordNet", "language": "en", "version": "3.0", "format": "wordnet-3.0-zip", "source": WordNetSource, "sha256": WordNetSHA}}, nil
}))
v.POST("/dictionaries/import", protect(true, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) {
select {
case uploadGate <- struct{}{}:
defer func() { <-uploadGate }()
default:
return nil, failure(429, "已有词典正在导入,请稍后重试")
}
resource, err := readDictionaryUpload(c)
if err != nil {
return nil, err
}
engine, err := ParseWordNet(resource.Archive)
if err != nil {
return nil, failure(400, "词典文件校验失败,请选择指定的 WordNet 3.0 ZIP")
}
resource.EntryCount = engine.EntryCount
resource.UpdatedAt = stamp(now())
// INSERT ... ON CONFLICT followed by a locking read serializes even the first
// concurrent import. A validated replacement and its metadata commit together.
created := tx.Clauses(clause.OnConflict{DoNothing: true}).Create(&resource)
if created.Error != nil {
return nil, created.Error
}
var existing DictionaryResource
if err = tx.Omit("archive").Clauses(clause.Locking{Strength: "UPDATE"}).Where("id = 1").First(&existing).Error; err != nil {
return nil, err
}
duplicate := created.RowsAffected == 0 && existing.SHA256 == resource.SHA256
if created.RowsAffected == 0 {
if err = tx.Model(&DictionaryResource{}).Where("id = 1").Select("name", "language", "version", "source", "format", "sha256", "entry_count", "enabled", "archive", "updated_at").Updates(&resource).Error; err != nil {
return nil, err
}
existing = resource
}
status := "ready"
if !existing.Enabled {
status = "disabled"
}
return DictionaryImportResult{dictionaryView(existing, status), duplicate}, nil
}))
v.PATCH("/dictionaries/:id", protect(true, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) {
id, err := strconv.ParseInt(c.Param("id"), 10, 64)
if err != nil || id != 1 {
return nil, failure(404, "词典不存在")
}
var input struct {
Enabled *bool `json:"enabled"`
}
if err := decode(c, &input); err != nil {
return nil, err
}
if input.Enabled == nil {
return nil, failure(400, "请指定词典启用状态")
}
var resource DictionaryResource
if err = tx.Omit("archive").Clauses(clause.Locking{Strength: "UPDATE"}).Where("id = ?", id).First(&resource).Error; errors.Is(err, gorm.ErrRecordNotFound) {
return nil, failure(404, "词典不存在")
} else if err != nil {
return nil, err
}
if *input.Enabled {
if _, err = cache.load(tx, resource); err != nil {
return nil, failure(409, "词典资源不可用,请重新导入")
}
}
resource.Enabled = *input.Enabled
resource.UpdatedAt = stamp(now())
if err = tx.Model(&resource).Updates(map[string]any{"enabled": resource.Enabled, "updated_at": resource.UpdatedAt}).Error; err != nil {
return nil, err
}
status := "disabled"
if resource.Enabled {
status = "ready"
}
return gin.H{"resource": dictionaryView(resource, status)}, nil
}))
v.GET("/chapters/:id/tokens", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) {
if c.Request.URL.RawQuery != "" {
return nil, failure(400, "分词接口不接受查询参数")
}
id, err := strconv.ParseInt(c.Param("id"), 10, 64)
if err != nil || id <= 0 {
return nil, failure(404, "章节不存在")
}
chapter, err := readyOwnedChapter(tx, u.UserId, id)
if err != nil {
return nil, err
}
return ChapterTokens{chapter.ContentSHA256, Tokenize(chapter.OriginalText)}, nil
}))
v.POST("/lookup", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) {
var input struct {
ChapterID int64 `json:"chapterId"`
Start *int `json:"start"`
End *int `json:"end"`
}
if err := decode(c, &input); err != nil {
return nil, err
}
chapter, err := readyOwnedChapter(tx, u.UserId, input.ChapterID)
if err != nil {
return nil, err
}
if input.Start == nil || input.End == nil {
return nil, failure(400, "请选择完整单词")
}
query := ""
for _, token := range Tokenize(chapter.OriginalText) {
if token.Kind == "word" && token.Start == *input.Start && token.End == *input.End {
query = token.Text
break
}
}
if query == "" || utf8.RuneCountInString(query) > 128 {
return nil, failure(400, "请选择不超过 128 个字符的完整单词")
}
missing := LookupResult{Status: "resource_missing", Query: query, Candidates: []string{}, Entries: []DictionaryEntry{}}
var resource DictionaryResource
err = tx.Omit("archive").Where("id = 1 AND enabled = ?", true).First(&resource).Error
if errors.Is(err, gorm.ErrRecordNotFound) {
return missing, nil
}
if err != nil {
return nil, err
}
engine, err := cache.load(tx, resource)
if err != nil {
return missing, nil
}
result := engine.Lookup(query)
result.Resource = &LookupResource{Name: resource.Name, Version: resource.Version}
return result, nil
}))
}
+182
View File
@@ -0,0 +1,182 @@
package lexgo
import (
"bytes"
"encoding/json"
"fmt"
"mime/multipart"
"net/http/httptest"
"os"
"testing"
"time"
"github.com/gin-gonic/gin"
admin "go-admin/app/admin/models"
)
func importDictionaryAPI(t *testing.T, r *gin.Engine, token string, raw []byte) (int, DictionaryImportResult) {
t.Helper()
var body bytes.Buffer
w := multipart.NewWriter(&body)
for k, v := range map[string]string{"name": "Princeton WordNet", "language": "en", "version": "3.0", "source": WordNetSource, "format": "wordnet-3.0-zip"} {
if err := w.WriteField(k, v); err != nil {
t.Fatal(err)
}
}
p, err := w.CreateFormFile("file", "wordnet.zip")
if err != nil {
t.Fatal(err)
}
p.Write(raw)
w.Close()
req := httptest.NewRequest("POST", "/api/v1/dictionaries/import", &body)
req.Header.Set("Content-Type", w.FormDataContentType())
req.Header.Set("Authorization", "Bearer "+token)
response := httptest.NewRecorder()
r.ServeHTTP(response, req)
var e envelope
json.Unmarshal(response.Body.Bytes(), &e)
var result DictionaryImportResult
json.Unmarshal(e.Data, &result)
return response.Code, result
}
func TestDictionaryAPIResourcesAndOwnership(t *testing.T) {
db := testDB(t)
raw, err := os.ReadFile("../../../.local/nlp-resources/wordnet.zip")
if os.IsNotExist(err) {
t.Skip("prepare WordNet fixture")
}
if err != nil {
t.Fatal(err)
}
// Test-only dedicated database slot; no development/production resources touched.
if err := db.Exec("DELETE FROM lexgo_dictionaries").Error; err != nil {
t.Fatal(err)
}
t.Cleanup(func() { db.Exec("DELETE FROM lexgo_dictionaries") })
r := Router(db, time.Now)
users := []admin.SysUser{}
tokens := []string{}
for i, role := range []int{1, 2, 2} {
u := admin.SysUser{Username: randomName(fmt.Sprintf("dict%d", i)), Password: fixturePassword, RoleId: role, Status: "2"}
if err := db.Create(&u).Error; err != nil {
t.Fatal(err)
}
users = append(users, u)
tokens = append(tokens, loginToken(t, r, u.Username, fixturePassword))
}
book := Book{OwnerID: users[1].UserId, Title: "Fictional", Language: "en", CreatedAt: stamp(time.Now()), UpdatedAt: stamp(time.Now())}
if err := db.Create(&book).Error; err != nil {
t.Fatal(err)
}
original := "😀 Dogs went. Cafe\u0301 123"
chapter := Chapter{BookID: book.ID, OwnerID: users[1].UserId, Ordinal: 1, Title: "Fictional", OriginalText: original, ContentSHA256: contentSHA(original), CharCount: len([]rune(original)), Status: statusReady, CreatedAt: stamp(time.Now()), UpdatedAt: stamp(time.Now())}
if err := db.Create(&chapter).Error; err != nil {
t.Fatal(err)
}
lookup := func(token string, start, end int) (int, LookupResult) {
code, data := callAPI(t, r, "POST", "/api/v1/lookup", token, map[string]any{"chapterId": chapter.ID, "start": start, "end": end})
var got LookupResult
json.Unmarshal(data, &got)
return code, got
}
if code, _ := callAPI(t, r, "GET", "/api/v1/dictionaries", tokens[1], nil); code != 403 {
t.Fatal("learner resource list", code)
}
if code, _ := importDictionaryAPI(t, r, tokens[1], []byte("bad")); code != 403 {
t.Fatal("learner import", code)
}
if code, _ := lookup(tokens[2], 2, 6); code != 404 {
t.Fatal("foreign lookup", code)
}
if code, _ := callAPI(t, r, "GET", fmt.Sprintf("/api/v1/chapters/%d/tokens", chapter.ID), tokens[2], nil); code != 404 {
t.Fatal("foreign tokens", code)
}
if code, got := lookup(tokens[1], 2, 6); code != 200 || got.Status != "resource_missing" {
t.Fatal("missing", code, got)
}
code, data := callAPI(t, r, "GET", fmt.Sprintf("/api/v1/chapters/%d/tokens", chapter.ID), tokens[1], nil)
var analyzed ChapterTokens
json.Unmarshal(data, &analyzed)
if code != 200 || analyzed.TextSHA256 != contentSHA(original) || analyzed.Tokens[2].StartUtf16 != 3 {
t.Fatal("tokens", code, string(data))
}
for _, span := range [][2]int{{-1, 3}, {2, 5}, {0, 1}, {2, 11}, {18, 21}, {1, 2}, {6, 2}} {
if code, _ := lookup(tokens[1], span[0], span[1]); code != 400 {
t.Fatal("invalid interval", span, code)
}
}
code, imported := importDictionaryAPI(t, r, tokens[0], raw)
if code != 200 || imported.Duplicate || imported.Resource.Status != "ready" {
t.Fatal("import", code, imported)
}
if code, got := lookup(tokens[1], 7, 11); code != 200 || got.Status != "lemma" || *got.MatchedForm != "go" {
t.Fatal("went", code, got)
}
code, duplicate := importDictionaryAPI(t, r, tokens[0], raw)
if code != 200 || !duplicate.Duplicate || duplicate.Resource.ID != imported.Resource.ID {
t.Fatal("duplicate", code, duplicate)
}
if code, _ := importDictionaryAPI(t, r, tokens[0], []byte("broken")); code != 400 {
t.Fatal("bad import", code)
}
if code, got := lookup(tokens[1], 2, 6); code != 200 || got.Status != "lemma" {
t.Fatal("failed import lost old resource", code, got)
}
endpoint := fmt.Sprintf("/api/v1/dictionaries/%d", imported.Resource.ID)
if code, _ := callAPI(t, r, "PATCH", endpoint, tokens[1], map[string]bool{"enabled": false}); code != 403 {
t.Fatal("learner toggle", code)
}
for _, enabled := range []bool{false, true} {
if code, _ := callAPI(t, r, "PATCH", endpoint, tokens[0], map[string]bool{"enabled": enabled}); code != 200 {
t.Fatal("toggle", code)
}
code, got := lookup(tokens[1], 2, 6)
if code != 200 || (enabled && got.Status != "lemma") || (!enabled && got.Status != "resource_missing") {
t.Fatal("enabled state", enabled, code, got)
}
}
r = Router(db, time.Now) // A new router has an empty cache and reloads the persisted ZIP.
if code, got := lookup(tokens[1], 2, 6); code != 200 || got.Status != "lemma" {
t.Fatal("cold restart", code, got)
}
if code, _ := callAPI(t, r, "PATCH", endpoint, tokens[0], map[string]bool{"enabled": false}); code != 200 {
t.Fatal("disable before duplicate", code)
}
if code, result := importDictionaryAPI(t, r, tokens[0], raw); code != 200 || !result.Duplicate || !result.Resource.Enabled {
t.Fatal("duplicate must re-enable", code, result)
}
var count int64
if err := db.Model(&DictionaryResource{}).Count(&count).Error; err != nil || count != 1 {
t.Fatal("duplicate created extra resource", count, err)
}
if code, _ := callAPI(t, r, "PATCH", endpoint, tokens[0], map[string]any{"enabled": nil}); code != 400 {
t.Fatal("null toggle", code)
}
if code, _ := callAPI(t, r, "POST", "/api/v1/lookup", tokens[1], map[string]any{"chapterId": chapter.ID, "start": 2, "end": 6, "ownerId": users[2].UserId}); code != 400 {
t.Fatal("unknown lookup input", code)
}
if err := db.Model(&DictionaryResource{}).Where("id=1").Update("archive", []byte("corrupt fixture")).Error; err != nil {
t.Fatal(err)
}
r = Router(db, time.Now)
if code, got := lookup(tokens[1], 2, 6); code != 200 || got.Status != "resource_missing" {
t.Fatal("corrupt cold resource", code, got)
}
if code, got := callAPI(t, r, "GET", "/api/v1/dictionaries", tokens[0], nil); code != 200 || !bytes.Contains(got, []byte(`"status":"unavailable"`)) {
t.Fatal("corrupt resource state", code, string(got))
}
if code, result := importDictionaryAPI(t, r, tokens[0], raw); code != 200 || !result.Duplicate {
t.Fatal("repair reimport", code, result)
}
if code, got := lookup(tokens[1], 2, 6); code != 200 || got.Status != "lemma" {
t.Fatal("repair lookup", code, got)
}
if err := db.Model(&chapter).Update("status", statusPending).Error; err != nil {
t.Fatal(err)
}
if code, _ := lookup(tokens[1], 2, 6); code != 409 {
t.Fatal("pending lookup", code)
}
}
+53 -3
View File
@@ -136,8 +136,8 @@ func TestMigrationFromV2PreservesExistingData(t *testing.T) {
}
var version int
db.Raw("SELECT version FROM lexgo_schema WHERE id=1").Scan(&version)
if version != 3 {
t.Fatalf("schema version %d after re-upgrade, want 3", version)
if version != schemaVersion {
t.Fatalf("schema version %d after re-upgrade, want %d", version, schemaVersion)
}
db.Table("sys_user").Where("user_id = 7").Count(&users)
if users != 1 {
@@ -149,7 +149,7 @@ func TestMigrationRefusesUnownedOrUnsupportedSchema(t *testing.T) {
for _, tc := range []struct{ name, marker string }{
{"empty_marker", ""},
{"negative_version", "INSERT INTO lexgo_schema VALUES (1,-1,'lexgo')"},
{"future_version", "INSERT INTO lexgo_schema VALUES (1,4,'lexgo')"},
{"future_version", "INSERT INTO lexgo_schema VALUES (1,99,'lexgo')"},
{"wrong_product", "INSERT INTO lexgo_schema VALUES (1,0,'another-app')"},
} {
t.Run(tc.name, func(t *testing.T) {
@@ -173,3 +173,53 @@ func TestMigrationRefusesUnownedOrUnsupportedSchema(t *testing.T) {
})
}
}
func TestMigrationFromV3PreservesLibraryAndJobs(t *testing.T) {
db := emptyMigrationDB(t)
statements := []string{"CREATE TABLE lexgo_schema (id INT PRIMARY KEY,version INT,product VARCHAR(32))", "INSERT INTO lexgo_schema VALUES (1,3,'lexgo')"}
statements = append(statements, schemaV2Statements...)
statements = append(statements, schemaV3Statements...)
for _, statement := range statements {
if err := db.Exec(statement).Error; err != nil {
t.Fatal(err)
}
}
if err := db.Exec("INSERT INTO sys_user (user_id,username,password,role_id) VALUES (8,'fixture_v3','fictional-not-a-real-hash',2)").Error; err != nil {
t.Fatal(err)
}
now := stamp(time.Now())
book := Book{OwnerID: 8, Title: "Fictional migration", Language: "en", CreatedAt: now, UpdatedAt: now}
if err := db.Create(&book).Error; err != nil {
t.Fatal(err)
}
original := "😀 Original e\u0301\r\n"
chapter := Chapter{BookID: book.ID, OwnerID: 8, Ordinal: 1, Title: "Fixture", OriginalText: original, ContentSHA256: contentSHA(original), Status: statusReady, CharCount: len([]rune(original)), CreatedAt: now, UpdatedAt: now}
if err := db.Create(&chapter).Error; err != nil {
t.Fatal(err)
}
job := IngestJob{OwnerID: 8, BookID: book.ID, ChapterID: chapter.ID, RequestKey: contentSHA("fixture-v3"), ContentSHA256: chapter.ContentSHA256, Status: statusReady, CreatedAt: now, UpdatedAt: now}
if err := db.Create(&job).Error; err != nil {
t.Fatal(err)
}
if err := CheckSchema(db); err == nil {
t.Fatal("old schema accepted before explicit migration")
}
if err := Migrate(db); err != nil {
t.Fatal(err)
}
if err := CheckSchema(db); err != nil {
t.Fatal(err)
}
var restored Chapter
if err := db.First(&restored, chapter.ID).Error; err != nil || restored.OriginalText != original || restored.ContentSHA256 != chapter.ContentSHA256 || restored.Status != statusReady {
t.Fatal("migration changed original chapter", err)
}
var restoredJob IngestJob
if err := db.First(&restoredJob, job.ID).Error; err != nil || restoredJob.RequestKey != job.RequestKey || restoredJob.Status != statusReady {
t.Fatal("migration changed job", err)
}
var resources int64
if err := db.Model(&DictionaryResource{}).Count(&resources).Error; err != nil || resources != 0 {
t.Fatal("migration must create empty resource table", resources, err)
}
}
+1
View File
@@ -294,6 +294,7 @@ func Router(db *gorm.DB, now func() time.Time) *gin.Engine {
}
return gin.H{"job": job, "chapter": chapter}, nil
}))
registerDictionaryRoutes(v, protect, now)
r.NoRoute(func(c *gin.Context) { respond(c, 404, nil, failure(404, "页面或接口不存在")) })
return r
}
+337
View File
@@ -0,0 +1,337 @@
package lexgo
import (
"archive/zip"
"bufio"
"bytes"
"errors"
"io"
"path"
"strconv"
"strings"
"unicode"
"golang.org/x/text/unicode/norm"
)
const WordNetSHA = "cbda5ea6eef7f36a97a43d4a75f85e07fccbb4f23657d27b4ccbc93e2646ab59"
const WordNetSource = "https://raw.githubusercontent.com/nltk/nltk_data/96f9b3252457a2b97e52aec64c3dfceeb5c312d5/packages/corpora/wordnet.zip"
const maxDictionaryZip = 32 << 20
const maxDictionaryInflated = 128 << 20
type TextToken struct {
Text string `json:"text"`
Start int `json:"start"`
End int `json:"end"`
StartUtf16 int `json:"startUtf16"`
EndUtf16 int `json:"endUtf16"`
Kind string `json:"kind"`
}
// Tokenize never normalizes text. Letters begin words; marks continue them and
// apostrophes join letters. Numbers and symbols are non-selectable punctuation.
// All offsets are half-open, in code points and UTF-16 code units respectively.
func Tokenize(text string) []TextToken {
runes := []rune(text)
result := make([]TextToken, 0)
utf16Offset := 0
for i := 0; i < len(runes); {
start, start16 := i, utf16Offset
kind := "punctuation"
if unicode.IsLetter(runes[i]) {
kind = "word"
} else if unicode.IsSpace(runes[i]) {
kind = "space"
}
i++
for i < len(runes) {
if kind == "word" && (unicode.IsLetter(runes[i]) || unicode.IsMark(runes[i]) || ((runes[i] == '\'' || runes[i] == '’') && i+1 < len(runes) && unicode.IsLetter(runes[i+1]))) {
i++
continue
}
if kind == "space" && unicode.IsSpace(runes[i]) {
i++
continue
}
break
}
for _, r := range runes[start:i] {
utf16Offset++
if r > 0xffff {
utf16Offset++
}
}
result = append(result, TextToken{string(runes[start:i]), start, i, start16, utf16Offset, kind})
}
return result
}
func normalizeWord(word string) string {
return norm.NFC.String(strings.ToLower(strings.ReplaceAll(word, "’", "'")))
}
type DictionaryEntry struct {
Lemma string `json:"lemma"`
POS string `json:"pos"`
Definition string `json:"definition"`
Examples []string `json:"examples"`
}
type LookupResult struct {
Status string `json:"status"`
Query string `json:"query"`
MatchedForm *string `json:"matchedForm"`
Candidates []string `json:"candidates"`
Entries []DictionaryEntry `json:"entries"`
Resource *LookupResource `json:"resource,omitempty"`
}
type LookupResource struct {
Name string `json:"name"`
Version string `json:"version"`
}
type wordNetPOS struct {
pos string
index map[string][]int
data map[int]DictionaryEntry
exceptions map[string][]string
}
type WordNet struct {
parts []wordNetPOS
EntryCount int
}
// ParseWordNet accepts only the pinned official corpus. No archive member is ever
// extracted to disk; both declared and actual inflated sizes are bounded.
func ParseWordNet(raw []byte) (*WordNet, error) {
if len(raw) > maxDictionaryZip || contentSHA(string(raw)) != WordNetSHA {
return nil, errors.New("WordNet ZIP checksum mismatch")
}
return parseWordNetArchive(raw)
}
func readWordNetArchive(raw []byte) (map[string][]byte, error) {
if len(raw) > maxDictionaryZip {
return nil, errors.New("archive too large")
}
reader, err := zip.NewReader(bytes.NewReader(raw), int64(len(raw)))
if err != nil {
return nil, err
}
if len(reader.File) > 64 {
return nil, errors.New("too many archive members")
}
files := map[string][]byte{}
seen := map[string]bool{}
total := int64(0)
for _, f := range reader.File {
name := f.Name
if strings.Contains(name, "\\") || path.IsAbs(name) || strings.Contains(name, ":") || path.Clean(name) != strings.TrimSuffix(name, "/") || !strings.HasPrefix(name, "wordnet/") || seen[name] {
return nil, errors.New("unsafe archive member")
}
seen[name] = true
if !f.FileInfo().IsDir() && !f.Mode().IsRegular() {
return nil, errors.New("unsupported archive member")
}
if f.FileInfo().IsDir() {
continue
}
if f.UncompressedSize64 > maxDictionaryInflated || total+int64(f.UncompressedSize64) > maxDictionaryInflated {
return nil, errors.New("inflated archive too large")
}
rc, err := f.Open()
if err != nil {
return nil, err
}
data, err := io.ReadAll(io.LimitReader(rc, maxDictionaryInflated-total+1))
rc.Close()
if err != nil {
return nil, err
}
total += int64(len(data))
if total > maxDictionaryInflated {
return nil, errors.New("inflated archive too large")
}
files[strings.TrimPrefix(name, "wordnet/")] = data
}
for _, name := range []string{"LICENSE", "README", "lexnames", "index.noun", "index.verb", "index.adj", "index.adv", "data.noun", "data.verb", "data.adj", "data.adv", "noun.exc", "verb.exc", "adj.exc", "adv.exc"} {
if len(files[name]) == 0 {
return nil, errors.New("missing required WordNet member")
}
}
return files, nil
}
func parseWordNetArchive(raw []byte) (*WordNet, error) {
files, err := readWordNetArchive(raw)
if err != nil {
return nil, err
}
result := &WordNet{}
for i, suffix := range []string{"noun", "verb", "adj", "adv"} {
part := wordNetPOS{pos: []string{"n", "v", "a", "r"}[i], index: map[string][]int{}, data: map[int]DictionaryEntry{}, exceptions: map[string][]string{}}
scanner := bufio.NewScanner(bytes.NewReader(files["data."+suffix]))
scanner.Buffer(make([]byte, 4096), 1<<20)
for scanner.Scan() {
line := scanner.Text()
if line == "" || line[0] == ' ' {
continue
}
header, gloss, ok := strings.Cut(line, "|")
fields := strings.Fields(header)
if !ok || len(fields) < 6 {
return nil, errors.New("invalid WordNet data record")
}
offset, e := strconv.Atoi(fields[0])
if e != nil {
return nil, e
}
if _, exists := part.data[offset]; exists {
return nil, errors.New("duplicate synset offset")
}
definition, examples := wordNetGloss(gloss)
part.data[offset] = DictionaryEntry{POS: fields[2], Definition: definition, Examples: examples}
}
if err := scanner.Err(); err != nil {
return nil, err
}
scanner = bufio.NewScanner(bytes.NewReader(files["index."+suffix]))
scanner.Buffer(make([]byte, 4096), 1<<20)
for scanner.Scan() {
line := scanner.Text()
if line == "" || line[0] == ' ' {
continue
}
fields := strings.Fields(line)
if len(fields) < 7 {
return nil, errors.New("invalid WordNet index record")
}
count, e := strconv.Atoi(fields[2])
if e != nil || count < 1 || count > 1000 {
return nil, errors.New("invalid sense count")
}
pointers, e := strconv.Atoi(fields[3])
if e != nil || pointers < 0 || len(fields) != 6+pointers+count {
return nil, errors.New("invalid index offsets")
}
offsets := make([]int, 0, count)
for _, rawOffset := range fields[6+pointers:] {
offset, e := strconv.Atoi(rawOffset)
if e != nil {
return nil, e
}
if _, ok := part.data[offset]; !ok {
return nil, errors.New("missing synset")
}
offsets = append(offsets, offset)
}
if _, exists := part.index[fields[0]]; exists {
return nil, errors.New("duplicate index key")
}
part.index[fields[0]] = offsets
}
if err := scanner.Err(); err != nil {
return nil, err
}
scanner = bufio.NewScanner(bytes.NewReader(files[suffix+".exc"]))
for scanner.Scan() {
fields := strings.Fields(scanner.Text())
if len(fields) < 2 {
return nil, errors.New("invalid exception record")
}
part.exceptions[fields[0]] = fields[1:]
}
if err := scanner.Err(); err != nil {
return nil, err
}
result.EntryCount += len(part.index)
result.parts = append(result.parts, part)
}
return result, nil
}
func wordNetGloss(gloss string) (string, []string) {
examples := []string{}
definition := strings.Builder{}
for {
before, after, ok := strings.Cut(gloss, "\"")
definition.WriteString(before)
if !ok {
break
}
example, rest, closed := strings.Cut(after, "\"")
if !closed {
definition.WriteString(after)
break
}
examples = append(examples, example)
gloss = rest
}
return strings.Trim(strings.TrimSpace(definition.String()), "; "), examples
}
var morphyRules = map[string][][2]string{
"n": {{"s", ""}, {"ses", "s"}, {"ves", "f"}, {"xes", "x"}, {"zes", "z"}, {"ches", "ch"}, {"shes", "sh"}, {"men", "man"}, {"ies", "y"}},
"v": {{"s", ""}, {"ies", "y"}, {"es", "e"}, {"es", ""}, {"ed", "e"}, {"ed", ""}, {"ing", "e"}, {"ing", ""}},
"a": {{"er", ""}, {"est", ""}, {"er", "e"}, {"est", "e"}},
}
// Morphy yields possible dictionary forms, never contextual POS or a merged
// learning identity. Exact forms win before exceptions and suffix detachment.
func (wn *WordNet) Lookup(query string) LookupResult {
result := LookupResult{Status: "not_found", Query: query, Candidates: []string{}, Entries: []DictionaryEntry{}}
key := normalizeWord(query)
add := func(form string, part wordNetPOS) {
if len(part.index[form]) == 0 {
return
}
found := false
for _, candidate := range result.Candidates {
if candidate == form {
found = true
break
}
}
if !found {
result.Candidates = append(result.Candidates, form)
}
for _, offset := range part.index[form] {
if len(result.Entries) >= 12 {
break
}
entry := part.data[offset]
entry.Lemma = strings.ReplaceAll(form, "_", " ")
result.Entries = append(result.Entries, entry)
}
}
for _, part := range wn.parts {
add(key, part)
}
if len(result.Entries) > 0 {
result.Status = "exact"
} else {
for _, part := range wn.parts {
forms := part.exceptions[key]
if len(forms) == 0 {
for _, rule := range morphyRules[part.pos] {
if strings.HasSuffix(key, rule[0]) && len(key) > len(rule[0]) {
forms = append(forms, strings.TrimSuffix(key, rule[0])+rule[1])
}
}
}
seen := map[string]bool{}
for _, form := range forms {
if !seen[form] {
add(form, part)
seen[form] = true
}
}
}
if len(result.Entries) > 0 {
result.Status = "lemma"
}
}
if len(result.Candidates) > 0 {
form := result.Candidates[0]
result.MatchedForm = &form
}
return result
}
+112
View File
@@ -0,0 +1,112 @@
package lexgo
import (
"archive/zip"
"bytes"
"os"
"strings"
"testing"
"unicode/utf16"
)
func TestArchiveRejectsUnsafeMembersAndMissingLicense(t *testing.T) {
for _, name := range []string{"../data.noun", "wordnet/../data.noun", "/wordnet/data.noun", "wordnet\\data.noun", "wordnet/C:data.noun", "wordnet/LICENSE"} {
var raw bytes.Buffer
w := zip.NewWriter(&raw)
entry, err := w.Create(name)
if err != nil {
t.Fatal(err)
}
entry.Write([]byte("fictional fixture"))
w.Close()
if _, err := readWordNetArchive(raw.Bytes()); err == nil {
t.Fatalf("accepted unsafe/incomplete archive %s", name)
}
}
var raw bytes.Buffer
w := zip.NewWriter(&raw)
for i := 0; i < 2; i++ {
entry, _ := w.Create("wordnet/LICENSE")
entry.Write([]byte("fixture"))
}
w.Close()
if _, err := readWordNetArchive(raw.Bytes()); err == nil {
t.Fatal("accepted duplicate member")
}
if _, err := ParseWordNet(make([]byte, maxDictionaryZip+1)); err == nil {
t.Fatal("accepted oversized archive")
}
}
func TestMorphyCandidatesAndExactPrecedence(t *testing.T) {
wn := &WordNet{parts: []wordNetPOS{{pos: "v", index: map[string][]int{"axes": {1}, "ax": {1}, "axe": {2}}, data: map[int]DictionaryEntry{1: {POS: "v", Definition: "fixture one", Examples: []string{}}, 2: {POS: "v", Definition: "fixture two", Examples: []string{}}}, exceptions: map[string][]string{}}}}
if got := wn.Lookup("AXES"); got.Status != "exact" || len(got.Candidates) != 1 || *got.MatchedForm != "axes" {
t.Fatal("exact must precede lemma", got)
}
delete(wn.parts[0].index, "axes")
got := wn.Lookup("axes")
if got.Status != "lemma" || strings.Join(got.Candidates, ",") != "axe,ax" {
t.Fatal("deterministic candidate forms", got)
}
definition, examples := wordNetGloss(`a fictional gloss; "one example"; "second example"`)
if definition != "a fictional gloss" || len(examples) != 2 {
t.Fatal("gloss", definition, examples)
}
}
func TestTokensPreserveUnicodeOriginal(t *testing.T) {
text := "😀 Cafe\u0301 can’t\r\nDogs 123 中文!"
tokens := Tokenize(text)
runes := []rune(text)
cursor, u16 := 0, 0
words := []string{}
for _, token := range tokens {
if token.Start != cursor || token.StartUtf16 != u16 || string(runes[token.Start:token.End]) != token.Text {
t.Fatalf("incorrect token %#v", token)
}
cursor = token.End
u16 += len(utf16.Encode([]rune(token.Text)))
if token.EndUtf16 != u16 {
t.Fatal("incorrect UTF-16 end")
}
if token.Kind == "word" {
words = append(words, token.Text)
}
}
if cursor != len(runes) || strings.Join(words, "|") != "Cafe\u0301|can’t|Dogs|中文" {
t.Fatalf("coverage/words %v", words)
}
if normalizeWord("CAFE\u0301") != "café" || normalizeWord("CAN’T") != "can't" {
t.Fatal("normalization")
}
}
func TestOfficialWordNet(t *testing.T) {
raw, err := os.ReadFile("../../../.local/nlp-resources/wordnet.zip")
if os.IsNotExist(err) {
t.Skip("explicitly prepared official WordNet fixture unavailable")
}
if err != nil {
t.Fatal(err)
}
engine, err := ParseWordNet(raw)
if err != nil {
t.Fatal(err)
}
if engine.EntryCount < 150000 {
t.Fatalf("entry count %d", engine.EntryCount)
}
for _, tc := range []struct{ query, status, lemma string }{{"dog", "exact", "dog"}, {"went", "lemma", "go"}, {"mice", "lemma", "mouse"}, {"zzzznonword", "not_found", ""}} {
got := engine.Lookup(tc.query)
if got.Status != tc.status || (tc.lemma != "" && (got.MatchedForm == nil || *got.MatchedForm != tc.lemma)) {
t.Fatalf("%s: %#v", tc.query, got)
}
if tc.lemma != "" && (len(got.Entries) == 0 || got.Entries[0].Definition == "") {
t.Fatal("missing definition")
}
}
raw[100] ^= 1
if _, err := ParseWordNet(raw); err == nil {
t.Fatal("accepted corrupt checksum")
}
}
+1 -1
View File
@@ -82,7 +82,7 @@ func run() error {
if err = lexgo.Migrate(db); err != nil {
return err
}
fmt.Println("LexGo schema version 3 ready")
fmt.Println("LexGo schema version 4 ready")
return nil
}
if err = lexgo.CheckSchema(db); err != nil {
+15
View File
@@ -0,0 +1,15 @@
{
"name": "Princeton WordNet",
"language": "en",
"version": "3.0",
"format": "wordnet-3.0-zip",
"source": "https://raw.githubusercontent.com/nltk/nltk_data/96f9b3252457a2b97e52aec64c3dfceeb5c312d5/packages/corpora/wordnet.zip",
"sha256": "cbda5ea6eef7f36a97a43d4a75f85e07fccbb4f23657d27b4ccbc93e2646ab59",
"license": "WORDNET-LICENSE.txt",
"licenseUrl": "https://wordnet.princeton.edu/license-and-commercial-use",
"documentation": [
"https://wordnet.princeton.edu/documentation/wndb5wn",
"https://wordnet.princeton.edu/documentation/morphy7wn"
],
"runtime": "Go only; explicitly imported ZIP persisted in MySQL; no runtime download or Python process"
}