Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ad31ea4225 | ||
|
|
8310438bee |
@@ -280,3 +280,4 @@ MVP 内所有单元任务通过后才能做 MVP 集成验收;MVP 通过后才
|
||||
- #18 登录日志与操作审计已通过用户验收:schema v2 显式迁移;日志只保存白名单字段,禁止保存凭据、请求/响应正文及私人学习内容。仅管理员查询,默认保留 90 天;启动/每小时及 `python scripts/server.py audit-cleanup` 仅清理两张审计表的过期记录。
|
||||
- #3 独立小样位于 `spikes/english/`,使用 `.local/nlp-venv/Scripts/python.exe`(3.12.12)运行;固定 spaCy 3.8.7、英语模型 3.8.0、NLTK 3.9.2、WordNet 3.0。资源仅显式准备时下载,摘要见 resources.json。不得把本机无账号的实验接口用于正式学习端;后续集成仍需 Go 授权、数据归属和任务设计。原文不归一化,位置区分 cp/UTF-8/UTF-16,lemma 不自动合并学习状态。
|
||||
- #4 独立阅读选择小样位于 `spikes/selection/`,`python spikes/selection/serve.py` 默认仅本机 5184。桌面鼠标/键盘与 11 项测试已验证,#4 已获用户验收并关闭;真实手机长按/手柄/滚动详细证据仍未提供;禁止把窄屏桌面当作真机验收。Intl.Segmenter 只用于 UI 范围验证,不替代 #3 NLP;释义保存只在内存。固定 LinguaCafe 源码对照和与 v1 的差异记录见架构 Wiki。
|
||||
- 2026-09-11 用户确认正式 NLP/词典采用全 Go。#5 已验收并合入 main;#6 使用 Go WordNet 解析和词形候选、Go Unicode 原文分片、schema v4 共享词典资源表,不调用 Python NLP。WordNet 3.0 ZIP 来源与摘要见 `server/wordnet-resource.json`,许可保留在 `server/WORDNET-LICENSE.txt`。词形候选不等于上下文消歧,不自动合并个人学习状态;#3 Python 小样只保留历史验证。当前词典仅英语释义,个人释义输入为临时草稿,持久化归 #7。
|
||||
|
||||
@@ -2,13 +2,13 @@
|
||||
|
||||
面向自托管场景的阅读式语言学习项目,规划提供内容导入、阅读查词、词汇与短语、复习和实例管理。
|
||||
|
||||
已确认:**DevHarness 轻量模式、MySQL 8、go-admin 管理端**。工程基础 #2 已通过验收:两端用户名登录、学习账号管理、可撤销会话和本人英语空空间。管理端基于指定 go-admin/go-admin-ui 选用模块,学习端为独立 Vue 3 + TypeScript + Vite 工程,共用 Go 后端和 MySQL 8.4.3。#18 登录日志与操作审计已通过用户验收,支持管理员查询和 90 天保留清理。#5 粘贴导入与章节原文阅读已实现,待用户验收;点词查词、词典、词语状态与复习仍未实现。MVP 定位为“支持多账号、数据独立的自托管学习工具”,先邀请少量用户使用;F01~F12 已确认,X 系列后置。
|
||||
已确认:**DevHarness 轻量模式、MySQL 8、go-admin 管理端**。工程基础 #2 已通过验收:两端用户名登录、学习账号管理、可撤销会话和本人英语空空间。管理端基于指定 go-admin/go-admin-ui 选用模块,学习端为独立 Vue 3 + TypeScript + Vite 工程,共用 Go 后端和 MySQL 8.4.3。#18 登录日志与操作审计已通过用户验收,支持管理员查询和 90 天保留清理。#5 粘贴导入与章节原文阅读已验收并合入 main;#6 全 Go 英语词典与点词查义已实现,待用户验收。词语状态与复习仍未实现。MVP 定位为“支持多账号、数据独立的自托管学习工具”,先邀请少量用户使用;F01~F12 已确认,X 系列后置。
|
||||
|
||||
- [文档入口](docs/README.md) · [线上 Wiki](https://git.ilapage.cn/OPC/lexgo/wiki/Home)
|
||||
- [英语分词与离线词典验证小样](spikes/english/README.md)(#3 已验收,独立本机入口)
|
||||
- [阅读选择验证小样](spikes/selection/README.md)(#4 已验收,真机详细测试证据缺口保留)
|
||||
- [项目档案](docs/00-project-profile.md) · [需求总览](docs/09-product-requirements-overview.md)
|
||||
- [工作量估算](docs/10-workload-estimate.md):#2、#3、#4、#18 已验收,剩余 #5~#15 与新增 #21 规划参考 44~71 人日;后续结合集成结果重估,旧全量研究仅供参考。
|
||||
- [工作量估算](docs/10-workload-estimate.md):#2、#3、#4、#5、#18 已验收,原规划中的 #5 已完成;剩余 #6~#15 与新增 #21、#24 按工单复核;后续结合集成结果重估,旧全量研究仅供参考。
|
||||
- [四阶段实施总览 #16](https://git.ilapage.cn/OPC/lexgo/issues/16):14 张单元工单,工程基础 → 技术验证 → 首条学习闭环 → 补齐 MVP;原型 v1 已获用户验收。两端使用账号(用户名)+密码登录,不要求邮箱。
|
||||
- [原型工单 #1](https://git.ilapage.cn/OPC/lexgo/issues/1):Quant-UX 桌面/手机原型 v1,预览入口与审核记录见工单及需求总览。
|
||||
- [工作流](docs/01-workflow.md) · [开发与验证](docs/04-local-development-and-verification.md)
|
||||
@@ -19,7 +19,15 @@
|
||||
|
||||
本地入口:学习端 http://127.0.0.1:5173,管理端 http://127.0.0.1:5174。完整安装与测试命令见[开发与验证](docs/04-local-development-and-verification.md)。账号使用用户名,无需邮箱;没有随代码交付的默认密码。
|
||||
|
||||
本机三个服务已由 `D:/supervisord/supervisord.conf` 中的 lexgo-learner、lexgo-admin、lexgo-api 托管,使用时不要重复手动启动同端口。当前 schema v3;从 #2/#18 升级时停止 API,执行 build、migrate 后再启动。审计过期清理可执行 `python scripts/server.py audit-cleanup`,只影响超过 90 天的日志。
|
||||
本机三个服务已由 `D:/supervisord/supervisord.conf` 中的 lexgo-learner、lexgo-admin、lexgo-api 托管,使用时不要重复手动启动同端口。当前 schema v4;从 #2/#18 升级时停止 API,执行 build、migrate 后再启动。审计过期清理可执行 `python scripts/server.py audit-cleanup`,只影响超过 90 天的日志。
|
||||
|
||||
## 英语词典与点词查义(#6)
|
||||
|
||||
正式后端为纯 Go,`lexgo.exe serve` 不启动 Python NLP。开发辅助脚本与历史小样保留。
|
||||
|
||||
管理员在“英语词典”页下载并导入指定 WordNet 3.0 ZIP;来源、固定摘要及许可见 [资源清单](server/wordnet-resource.json)。词典提供英语释义,保存在 MySQL 中;导入失败保留当前资源,重复导入不新增资源并重新启用。也可在管理页停用。
|
||||
|
||||
学习者打开本人就绪章节,点击词语或聚焦后按 Enter/空格查询;Escape/关闭返回阅读。精确词形优先,未命中再查规则候选,例如 `went → go`。个人释义当前是未保存的临时草稿,保存功能在 #7 实现。真机测试证据尚未补齐。
|
||||
|
||||
## 文档与治理
|
||||
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
export const MAX_DICTIONARY_BYTES = 32 * 1024 * 1024
|
||||
|
||||
export function createDictionaryLoader(session, state) {
|
||||
let revision = 0
|
||||
function invalidate() {
|
||||
revision++
|
||||
Object.assign(state, { items: [], supported: null, loading: false, saving: false, error: '', notice: '' })
|
||||
}
|
||||
async function perform(operation, saving, apply) {
|
||||
const current = ++revision
|
||||
const generation = session.state.generation
|
||||
const stale = () => current !== revision || generation !== session.state.generation
|
||||
Object.assign(state, { loading: !saving, saving, error: '', notice: '' })
|
||||
try {
|
||||
const result = await operation()
|
||||
if (stale()) return false
|
||||
apply(result)
|
||||
return true
|
||||
} catch (error) {
|
||||
if (!stale()) state.error = error instanceof Error ? error.message : '词典操作失败,请重试'
|
||||
return false
|
||||
} finally {
|
||||
if (!stale()) { state.loading = false; state.saving = false }
|
||||
}
|
||||
}
|
||||
return {
|
||||
invalidate,
|
||||
load() {
|
||||
return perform(() => session.listDictionaries(), false, result => {
|
||||
state.items = result.items
|
||||
state.supported = result.supported
|
||||
})
|
||||
},
|
||||
import(form) {
|
||||
return perform(() => session.importDictionary(form), true, result => {
|
||||
state.items = [result.resource]
|
||||
state.notice = result.duplicate ? '该词典已存在,已启用,未重复导入。' : '词典导入成功。'
|
||||
})
|
||||
},
|
||||
toggle(id, enabled) {
|
||||
return perform(() => session.setDictionaryEnabled(id, enabled), true, result => {
|
||||
state.items = state.items.map(item => item.id === result.resource.id ? result.resource : item)
|
||||
state.notice = result.resource.enabled ? '词典已启用。' : '词典已停用。'
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -5,6 +5,7 @@
|
||||
<div class="brand">{{ sidebar.opened ? 'LexGo 管理' : 'LG' }}</div>
|
||||
<el-menu :default-active="$route.path" :collapse="!sidebar.opened" router>
|
||||
<el-menu-item index="/accounts"><span>账号管理</span></el-menu-item>
|
||||
<el-menu-item index="/dictionaries"><span>英语词典</span></el-menu-item>
|
||||
<el-menu-item index="/login-logs"><span>登录日志</span></el-menu-item>
|
||||
<el-menu-item index="/operation-logs"><span>操作日志</span></el-menu-item>
|
||||
</el-menu>
|
||||
|
||||
@@ -4,11 +4,13 @@ import Layout from '../layout/index.vue'
|
||||
import Login from '../views/Login.vue'
|
||||
import Accounts from '../views/Accounts.vue'
|
||||
import AuditLogs from '../views/AuditLogs.vue'
|
||||
import Dictionaries from '../views/Dictionaries.vue'
|
||||
const router = createRouter({ history: createWebHashHistory(), routes: [
|
||||
{ path: '/login', component: Login },
|
||||
{ path: '/', component: Layout, children: [
|
||||
{ path: '', redirect: '/accounts' },
|
||||
{ path: 'accounts', component: Accounts, meta: { title: '账号管理' } },
|
||||
{ path: 'dictionaries', component: Dictionaries, meta: { title: '英语词典' } },
|
||||
{ path: 'login-logs', component: AuditLogs, props: { kind: 'login' }, meta: { title: '登录日志' } },
|
||||
{ path: 'operation-logs', component: AuditLogs, props: { kind: 'operation' }, meta: { title: '操作日志' } }
|
||||
] },
|
||||
|
||||
+17
-2
@@ -20,10 +20,11 @@ export function createSession({ fetch, storage, changed = () => {} }) {
|
||||
if (generation !== state.generation) throw new Error('会话已变化,请重新操作')
|
||||
}
|
||||
async function request(path, method = 'GET', body, token = state.token, generation = state.generation) {
|
||||
const multipart = typeof FormData !== 'undefined' && body instanceof FormData
|
||||
const result = await fetch('/api/v1' + path, {
|
||||
method,
|
||||
headers: { 'Content-Type': 'application/json', ...(token ? { Authorization: 'Bearer ' + token } : {}) },
|
||||
...(body === undefined ? {} : { body: JSON.stringify(body) }),
|
||||
headers: { ...(!multipart ? { 'Content-Type': 'application/json' } : {}), ...(token ? { Authorization: 'Bearer ' + token } : {}) },
|
||||
...(body === undefined ? {} : { body: multipart ? body : JSON.stringify(body) }),
|
||||
cache: 'no-store'
|
||||
})
|
||||
const payload = await result.json()
|
||||
@@ -84,6 +85,20 @@ export function createSession({ fetch, storage, changed = () => {} }) {
|
||||
} catch (error) { if (generation === state.generation) clear(); throw error }
|
||||
},
|
||||
async logout() { const token = state.token; clear(); await revoke(token) },
|
||||
async listDictionaries() {
|
||||
authorized()
|
||||
return request('/dictionaries')
|
||||
},
|
||||
async importDictionary(form) {
|
||||
authorized()
|
||||
if (!(form instanceof FormData)) throw new Error('请选择词典文件')
|
||||
return request('/dictionaries/import', 'POST', form)
|
||||
},
|
||||
async setDictionaryEnabled(id, enabled) {
|
||||
authorized()
|
||||
if (!Number.isSafeInteger(id) || id <= 0 || typeof enabled !== 'boolean') throw new Error('词典参数无效')
|
||||
return request('/dictionaries/' + id, 'PATCH', { enabled })
|
||||
},
|
||||
async queryAuditLogs(kind, filters) {
|
||||
authorized()
|
||||
const generation = state.generation
|
||||
|
||||
@@ -0,0 +1,101 @@
|
||||
<template>
|
||||
<basic-layout><template #wrapper>
|
||||
<el-card>
|
||||
<div class="toolbar"><div><h1>英语词典</h1><p class="subtle">供所有学习账号查询</p></div><el-button :disabled="busy" @click="reload">刷新</el-button></div>
|
||||
<el-alert v-if="resources.error" :title="resources.error" type="error" :closable="false" show-icon />
|
||||
<el-alert v-if="resources.notice" :title="resources.notice" type="success" :closable="false" show-icon />
|
||||
<el-table v-loading="resources.loading" :data="resources.items" border :empty-text="resources.error ? '加载失败,请重试' : '尚未导入词典'">
|
||||
<el-table-column label="名称" prop="name" min-width="175" />
|
||||
<el-table-column label="语言" width="80"><template #default>英语</template></el-table-column>
|
||||
<el-table-column label="版本" prop="version" width="80" />
|
||||
<el-table-column label="来源" prop="source" min-width="160" show-overflow-tooltip />
|
||||
<el-table-column label="格式" prop="format" min-width="160" />
|
||||
<el-table-column label="状态" width="95"><template #default="scope"><el-tag :type="scope.row.status === 'ready' ? 'success' : 'info'">{{ statusLabel(scope.row.status) }}</el-tag></template></el-table-column>
|
||||
<el-table-column label="词条数" prop="entryCount" min-width="95" />
|
||||
<el-table-column label="操作" width="100"><template #default="scope"><el-button :disabled="busy" link type="primary" @click="toggle(scope.row)">{{ scope.row.enabled ? '停用' : '启用' }}</el-button></template></el-table-column>
|
||||
</el-table>
|
||||
</el-card>
|
||||
<el-card class="import-card">
|
||||
<h2>导入词典</h2>
|
||||
<el-form class="dictionary-form" label-position="top" @submit.prevent="upload">
|
||||
<el-form-item label="名称"><el-input v-model="form.name" maxlength="120" :disabled="busy" /></el-form-item>
|
||||
<div class="form-row">
|
||||
<el-form-item label="语言"><el-input model-value="英语" disabled /></el-form-item>
|
||||
<el-form-item label="版本"><el-input v-model="form.version" readonly /></el-form-item>
|
||||
</div>
|
||||
<el-form-item label="来源"><el-input v-model="form.source" readonly /></el-form-item>
|
||||
<el-form-item label="格式"><el-input v-model="form.format" readonly /></el-form-item>
|
||||
<el-form-item label="词典文件">
|
||||
<input id="dictionary-file" ref="fileInput" type="file" accept=".zip,application/zip" aria-label="词典文件" :disabled="busy" @change="selectFile">
|
||||
</el-form-item>
|
||||
<p class="subtle">WordNet 3.0 ZIP · 上限 32 MiB · 英语释义</p>
|
||||
<p v-if="resources.supported?.source"><a :href="downloadURL" target="_blank" rel="noopener noreferrer">下载支持的词典文件</a></p>
|
||||
<p class="subtle">导入失败时保留当前词典。</p>
|
||||
<p v-if="fileError" role="alert" class="file-error">{{ fileError }}</p>
|
||||
<el-button type="primary" native-type="submit" :loading="resources.saving" :disabled="resources.loading || !file">导入并启用</el-button>
|
||||
</el-form>
|
||||
</el-card>
|
||||
</template></basic-layout>
|
||||
</template>
|
||||
<script>
|
||||
import BasicLayout from '../layout/BasicLayout.vue'
|
||||
import { session } from '../store'
|
||||
import { createDictionaryLoader, MAX_DICTIONARY_BYTES } from '../dictionaries.mjs'
|
||||
const downloadURL = 'https://raw.githubusercontent.com/nltk/nltk_data/96f9b3252457a2b97e52aec64c3dfceeb5c312d5/packages/corpora/wordnet.zip'
|
||||
export default {
|
||||
name: 'DictionariesView',
|
||||
components: { BasicLayout },
|
||||
data: () => ({
|
||||
resources: { items: [], supported: null, loading: false, saving: false, error: '', notice: '' },
|
||||
form: { name: 'Princeton WordNet', language: 'en', version: '3.0', source: downloadURL, format: 'wordnet-3.0-zip' },
|
||||
file: null, fileError: '', downloadURL
|
||||
}),
|
||||
computed: { busy() { return this.resources.loading || this.resources.saving } },
|
||||
watch: { '$store.state.generation': { flush: 'sync', handler() { this.loader.invalidate(); this.clearFile() } } },
|
||||
created() { this.loader = createDictionaryLoader(session, this.resources) },
|
||||
mounted() { this.reload() },
|
||||
beforeUnmount() { this.loader.invalidate(); this.file = null },
|
||||
methods: {
|
||||
reload() { return this.loader.load() },
|
||||
statusLabel(status) { return { ready: '可用', disabled: '已停用', unavailable: '不可用' }[status] || '不可用' },
|
||||
clearFile() { this.file = null; this.fileError = ''; if (this.$refs.fileInput) this.$refs.fileInput.value = '' },
|
||||
selectFile(event) {
|
||||
this.fileError = ''
|
||||
const selected = event.target.files?.[0]
|
||||
this.file = null
|
||||
if (!selected) return
|
||||
if (!selected.name.toLowerCase().endsWith('.zip') || selected.size === 0 || selected.size > MAX_DICTIONARY_BYTES) {
|
||||
this.fileError = '请选择不超过 32 MiB 的 ZIP 文件。'
|
||||
event.target.value = ''
|
||||
return
|
||||
}
|
||||
this.file = selected
|
||||
},
|
||||
async upload() {
|
||||
if (this.busy || !this.file) return
|
||||
this.fileError = ''
|
||||
if (!this.form.name.trim() || !this.form.source.trim()) { this.fileError = '请填写名称和来源。'; return }
|
||||
const data = new FormData()
|
||||
for (const [key, value] of Object.entries(this.form)) data.append(key, value.trim())
|
||||
data.append('file', this.file)
|
||||
if (await this.loader.import(data)) this.clearFile()
|
||||
},
|
||||
toggle(resource) { return this.loader.toggle(resource.id, !resource.enabled) }
|
||||
}
|
||||
}
|
||||
</script>
|
||||
<style scoped>
|
||||
.toolbar { display:flex; align-items:center; justify-content:space-between; gap:16px; margin-bottom:20px; }
|
||||
h1 { font-size:20px; margin:0 0 8px; }
|
||||
h2 { font-size:18px; margin:0 0 24px; }
|
||||
.subtle { color:#606266; font-size:13px; line-height:1.6; }
|
||||
.toolbar p { margin:0; }
|
||||
.import-card { margin-top:20px; }
|
||||
.dictionary-form { max-width:600px; }
|
||||
.form-row { display:flex; gap:16px; }
|
||||
.form-row .el-form-item { flex:1; min-width:0; }
|
||||
.file-error { color:#b42318; }
|
||||
.el-alert { margin-bottom:16px; }
|
||||
input[type=file] { max-width:100%; }
|
||||
a { color:#176b63; }
|
||||
</style>
|
||||
@@ -0,0 +1,55 @@
|
||||
import test from 'node:test'
|
||||
import assert from 'node:assert/strict'
|
||||
import { createSession } from '../src/session.mjs'
|
||||
|
||||
const response = (data, status = 200) => ({ ok: status < 400, status, json: async () => ({ data, msg: '导入失败' }) })
|
||||
function setup() {
|
||||
const pending = []
|
||||
const session = createSession({ fetch: (url, options) => new Promise(resolve => pending.push({ url, options, resolve })), storage: { getItem() {}, setItem() {}, removeItem() {} } })
|
||||
session.state.user = { id: 1, username: 'fixture.admin', role: 'admin' }
|
||||
session.state.token = 'fictional-test-token'
|
||||
return { session, pending }
|
||||
}
|
||||
|
||||
test('dictionary upload sends multipart with authorization and lets browser set boundary', async () => {
|
||||
const { session, pending } = setup()
|
||||
const form = new FormData()
|
||||
form.append('name', 'Princeton WordNet')
|
||||
form.append('file', new Blob(['fictional-archive']), 'wordnet.zip')
|
||||
assert.equal(typeof session.importDictionary, 'function')
|
||||
const request = session.importDictionary(form)
|
||||
assert.equal(pending[0].url, '/api/v1/dictionaries/import')
|
||||
assert.equal(pending[0].options.method, 'POST')
|
||||
assert.equal(pending[0].options.body, form)
|
||||
assert.equal(pending[0].options.headers['Content-Type'], undefined)
|
||||
assert.equal(pending[0].options.headers.Authorization, 'Bearer fictional-test-token')
|
||||
pending[0].resolve(response({ resource: { id: 1 }, duplicate: false }))
|
||||
assert.deepEqual(await request, { resource: { id: 1 }, duplicate: false })
|
||||
})
|
||||
|
||||
test('dictionary read and toggle follow API contract, ordinary accounts cannot mutate', async () => {
|
||||
const { session, pending } = setup()
|
||||
assert.equal(typeof session.listDictionaries, 'function')
|
||||
const read = session.listDictionaries()
|
||||
assert.equal(pending[0].url, '/api/v1/dictionaries')
|
||||
pending[0].resolve(response({ items: [] }))
|
||||
await read
|
||||
const toggle = session.setDictionaryEnabled(1, false)
|
||||
assert.equal(pending[1].url, '/api/v1/dictionaries/1')
|
||||
assert.deepEqual(JSON.parse(pending[1].options.body), { enabled: false })
|
||||
pending[1].resolve(response({ resource: { id: 1, enabled: false } }))
|
||||
await toggle
|
||||
session.state.user.role = 'learner'
|
||||
await assert.rejects(session.importDictionary(new FormData()), /管理员/)
|
||||
await assert.rejects(session.setDictionaryEnabled(1, true), /管理员/)
|
||||
assert.equal(pending.length, 2)
|
||||
})
|
||||
|
||||
test('dictionary response cannot cross a logout or account switch', async () => {
|
||||
const { session, pending } = setup()
|
||||
assert.equal(typeof session.importDictionary, 'function')
|
||||
const request = session.importDictionary(new FormData())
|
||||
session.clear()
|
||||
pending[0].resolve(response({ resource: { id: 1 }, duplicate: false }))
|
||||
await assert.rejects(request, /会话已变化/)
|
||||
})
|
||||
@@ -0,0 +1,57 @@
|
||||
import test from 'node:test'
|
||||
import assert from 'node:assert/strict'
|
||||
|
||||
async function fixture() {
|
||||
const { createDictionaryLoader } = await import('../src/dictionaries.mjs')
|
||||
const pending = []
|
||||
const later = () => new Promise((resolve, reject) => pending.push({ resolve, reject }))
|
||||
const session = { state: { generation: 1 }, listDictionaries: later, importDictionary: later, setDictionaryEnabled: later }
|
||||
const state = { items: [{ id: 1, name: 'Existing', enabled: true }], supported: null, loading: false, saving: false, error: '', notice: '' }
|
||||
return { loader: createDictionaryLoader(session, state), state, session, pending }
|
||||
}
|
||||
|
||||
test('failed upload preserves the existing resource and exposes the error', async () => {
|
||||
const { loader, state, pending } = await fixture()
|
||||
const action = loader.import(new FormData())
|
||||
assert.equal(state.saving, true)
|
||||
pending[0].reject(new Error('文件摘要不匹配'))
|
||||
assert.equal(await action, false)
|
||||
assert.equal(state.items[0].name, 'Existing')
|
||||
assert.equal(state.error, '文件摘要不匹配')
|
||||
assert.equal(state.saving, false)
|
||||
})
|
||||
|
||||
test('accepted mutation replaces old state directly, without a second refresh dependency', async () => {
|
||||
const { loader, state, pending } = await fixture()
|
||||
const action = loader.toggle(1, false)
|
||||
pending[0].resolve({ resource: { id: 1, name: 'Existing', enabled: false, status: 'disabled' } })
|
||||
assert.equal(await action, true)
|
||||
assert.equal(state.items[0].enabled, false)
|
||||
assert.equal(pending.length, 1)
|
||||
})
|
||||
|
||||
test('an old list cannot overwrite a newer import and duplicate import is explicit', async () => {
|
||||
const { loader, state, pending } = await fixture()
|
||||
const read = loader.load()
|
||||
const action = loader.import(new FormData())
|
||||
pending[1].resolve({ resource: { id: 1, name: 'Imported', enabled: true }, duplicate: true })
|
||||
await action
|
||||
pending[0].resolve({ items: [{ id: 1, name: 'Old' }] })
|
||||
await read
|
||||
assert.equal(state.items[0].name, 'Imported')
|
||||
assert.match(state.notice, /已存在/)
|
||||
})
|
||||
|
||||
test('page exit or session change discards late responses', async () => {
|
||||
const { loader, state, pending, session } = await fixture()
|
||||
const action = loader.import(new FormData())
|
||||
loader.invalidate()
|
||||
pending[0].resolve({ resource: { id: 1, name: 'Late' } })
|
||||
assert.equal(await action, false)
|
||||
assert.deepEqual(state.items, [])
|
||||
const read = loader.load()
|
||||
session.state.generation++
|
||||
pending[1].resolve({ items: [{ id: 1, name: 'Other session' }] })
|
||||
await read
|
||||
assert.deepEqual(state.items, [])
|
||||
})
|
||||
@@ -2,8 +2,8 @@
|
||||
generated: true (请先修改 Gitea Wiki,禁止直接编辑本文件)
|
||||
wiki_page: Project-Profile
|
||||
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Project-Profile.-
|
||||
wiki_revision: 40ae5e3a0521195da4f1a9cd9f42beec15da20d4
|
||||
synchronized_at: 2026-09-10T12:43:19Z
|
||||
wiki_revision: 445504a7f95f9c018216f74e27d433edde521f73
|
||||
synchronized_at: 2026-09-11T03:58:56Z
|
||||
<!-- gitea-wiki-mirror:end -->
|
||||
|
||||
# LexGo 项目档案
|
||||
@@ -37,7 +37,7 @@ LinguaCafe 参考基线由现有调研记录为 `c1ea298ce40c65b9dd33e9b26fd2e52
|
||||
|---|---|---|---|
|
||||
| D01 | 管理端底座 | **已确认使用 `D:/github_project/goadmin` 的 go-admin + go-admin-ui**,依据用户指令“管理端用D:\github_project\goadmin”;替代 gin-vue-admin 建议 | M0 验证指定本地版本配套、MySQL 8 及数据隔离,不再比较管理底座 |
|
||||
| D02 | 数据库 | **已确认 MySQL 8**;依据 2026-09-10 用户原话“使用mysql8”。覆盖两份分析的数据库分歧,具体小版本待基线验证锁定 | 按单一 MySQL 8 数据层估算,不做 PostgreSQL 双库兼容 |
|
||||
| D03 | NLP | 保留 Python 为建议,纯 Go 是否硬约束未确认 | 首发语言质量与开发量 |
|
||||
| D03 | NLP | **用户于2026-09-11确认正式产品全 Go**;Python小样仅历史验证 | 使用词典词形候选,不承诺上下文消歧 |
|
||||
| D04 | 默认学习语言 | 用户已确认英语;中文/日语后续需独立范围与验收 | 分词、读音和 UI |
|
||||
| D05 | 用户范围 | **已确认:支持多账号、数据独立的自托管学习工具,先邀请少量用户使用** | 首版按多用户归属和隔离设计;不自动加入公开注册、邀请链接或组织租户 |
|
||||
| D06 | 旧数据 | 全量迁移是否需要未确认,CSV 与完整迁移不同 | 迁移另估 |
|
||||
@@ -136,3 +136,13 @@ server 是 go-admin 的选用模块接入:原样保留 SysUser、SysDept、必
|
||||
## 日志审计基线(#18)
|
||||
|
||||
管理端新增两个审计列表,后端为 LexGo 自有日志模型与接口,参照 go-admin 模块布局但不复制其原始参数/响应持久化逻辑。schema v2 新增 lexgo_login_logs、lexgo_operation_logs;保留 90 天,启动和每小时分批清理,也可显式 audit-cleanup。新建/重置密码均为 6~72 UTF-8 字节;初始 bootstrap 保留 10 字节下限。#2 已验收,#18 已验收;默认模块中的其他候选未纳入。
|
||||
|
||||
|
||||
## 全 Go 正式架构决定(2026-09-11,#6)
|
||||
|
||||
用户已明确选择全 Go:正式英语分词、原文位置映射、本地词典解析和词形候选查询由 Go 后端完成,不运行 Python NLP 服务。此前“Python 建议/全 Go 未决”仅为历史决策记录,由本决定覆盖;spikes/english 保留历史验证,不接入产品。#6 按该方向实施,当前方案见工单最新启动评论;WordNet 3.0 仍为首个资源(英语释义),词形规则候选不等同于 spaCy 上下文消歧,原文及个人学习状态不按候选合并。
|
||||
|
||||
|
||||
## #6 当前工程状态(2026-09-11)
|
||||
|
||||
#5已验收且相关前置PR均已合入main。#6正式词典/点词查义已按全Go实现,位于feat/6-go-dictionary-lookup,等待用户验收;server schema v4,管理端新增英语词典页,学习端加入Go分片及查词面板。正式Go进程不依赖Python NLP,WordNet包随数据库持久化。#3小样仍是历史验证;保存个人释义和状态归#7,音频封面#21、列表优化#24尚未实施。
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
generated: true (请先修改 Gitea Wiki,禁止直接编辑本文件)
|
||||
wiki_page: Architecture-and-Code-Map
|
||||
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Architecture-and-Code-Map.-
|
||||
wiki_revision: f981b86a7c6e5d2e353142d3006dcd6e0636c3ee
|
||||
synchronized_at: 2026-09-11T03:01:48Z
|
||||
wiki_revision: f317e21d651f2b5cb66810e9a4abf02831908559
|
||||
synchronized_at: 2026-09-11T03:58:59Z
|
||||
<!-- gitea-wiki-mirror:end -->
|
||||
|
||||
# 架构与代码地图
|
||||
@@ -221,3 +221,25 @@ lexgo_books(owner_id, title, language)、lexgo_chapters(book_id, owner_id, ordin
|
||||
- 重试自愈(R4):`retryChapter` 先把重试返回的章节状态应用到列表与阅读器并重新安排轮询,再做静默刷新。
|
||||
|
||||
验证:Go 全量用例 20 项通过(新增运行期恢复与尝试上限两项);学习端单测 38 项通过,其中 7 项在整改前的代码上复现失败;真实联调确认追加路径可用、被中断的任务在运行中被自动恢复(约 0.5 秒,无需重启)、重试在首次刷新失败后仍自动显示最终结果。
|
||||
|
||||
|
||||
## 全 Go 正式架构决定(2026-09-11,#6)
|
||||
|
||||
用户已明确选择全 Go:正式英语分词、原文位置映射、本地词典解析和词形候选查询由 Go 后端完成,不运行 Python NLP 服务。此前“Python 建议/全 Go 未决”仅为历史决策记录,由本决定覆盖;spikes/english 保留历史验证,不接入产品。#6 按该方向实施,当前方案见工单最新启动评论;WordNet 3.0 仍为首个资源(英语释义),词形规则候选不等同于 spaCy 上下文消歧,原文及个人学习状态不按候选合并。
|
||||
|
||||
|
||||
## #6 全 Go 词典与阅读器(2026-09-11,已实现待验收)
|
||||
|
||||
`server/app/lexgo/wordnet.go` 负责固定 WordNet ZIP 校验/内存解析、Unicode 分词及词形候选;`dictionary.go` 负责资源与章节查词 API;`database.go` schema v4 新增单槽共享资源表 lexgo_dictionaries(元数据、enabled、SHA、ZIP LONGBLOB),已有学习数据不改写。每个 Router 按 SHA 缓存一个不可变词典,查询先读资源元数据,缓存未命中才读取 ZIP;进程重启从数据库恢复,不需要 Python NLP 或额外资源目录。
|
||||
|
||||
| 接口 | 权限与输入/输出 |
|
||||
|---|---|
|
||||
| GET /api/v1/dictionaries | 管理员;items + supported,状态 ready/disabled/unavailable,不返回 archive 或本机路径 |
|
||||
| POST /api/v1/dictionaries/import | 管理员;multipart name/language/version/source/format/file;返回 resource + duplicate;格式/来源/版本固定 |
|
||||
| PATCH /api/v1/dictionaries/:id | 管理员;{enabled:boolean};返回 resource |
|
||||
| GET /api/v1/chapters/:id/tokens | 本人 ready 章节;{textSha256,tokens:[{text,start,end,startUtf16,endUtf16,kind}]} |
|
||||
| POST /api/v1/lookup | {chapterId,start,end},cp半开范围;本人ready完整单词;返回 status/query/matchedForm/candidates/entries/resource |
|
||||
|
||||
管理端 `Dictionaries.vue` + `dictionaries.mjs` + session multipart 方法,复用 go-admin 导航/表单与身份失效保护。学习端 `useReaderLookup.ts` 校验原文片段/SHA/所有位置,`ReaderTokens.vue` 渲染可聚焦单词,`LookupPanel.vue` 展示释义及临时个人草稿。桌面侧栏,手机固定底部45dvh面板;关闭恢复焦点,仅无后续手动滚动时恢复自动调整前位置。旧响应在换词/换章/退出/离页后失效。
|
||||
|
||||
参考:[WordNet 数据格式](https://wordnet.princeton.edu/documentation/wndb5wn)、[词形规则](https://wordnet.princeton.edu/documentation/morphy7wn)。#3 仅历史实验,#6 不调用其实验服务。
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
generated: true (请先修改 Gitea Wiki,禁止直接编辑本文件)
|
||||
wiki_page: Business-Rules-and-Glossary
|
||||
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Business-Rules-and-Glossary.-
|
||||
wiki_revision: 4bd843d1f0134f5688407b7072d98333b40b6da1
|
||||
synchronized_at: 2026-09-11T03:01:48Z
|
||||
wiki_revision: 935630005c9b1d7103550cf09eb10edfdbf8f3af
|
||||
synchronized_at: 2026-09-11T03:59:01Z
|
||||
<!-- gitea-wiki-mirror:end -->
|
||||
|
||||
# 业务规则与术语
|
||||
@@ -137,3 +137,19 @@ POST /lookup 接收 {surface,lemma?}。查词键单独 casefold/NFC/弯撇号转
|
||||
- 运行期恢复不依赖重启:除了启动时的恢复,运行中的服务每次轮询都会把停留在 processing 且超过 15 秒的任务放回 pending,因此“领取已提交、完成事务失败”不会让章节永久卡在处理中。该阈值必须长于正常的领取到完成窗口;重复处理同一任务不会产生第二个章节,因为任务从不创建章节。
|
||||
- 学习端离开页面时作废在途请求:目录页与阅读页在关闭时推进各自的请求序号,晚到的响应不会写回状态或重启轮询;导入页在提交过程中离开后,晚到的成功响应不会把用户导航回书籍页。
|
||||
- 重试结果立即生效:重试被接受后先把返回的 pending 状态写入界面并继续轮询,因此紧随其后的一次刷新失败不会让页面停在处理失败。
|
||||
|
||||
|
||||
## 全 Go 正式架构决定(2026-09-11,#6)
|
||||
|
||||
用户已明确选择全 Go:正式英语分词、原文位置映射、本地词典解析和词形候选查询由 Go 后端完成,不运行 Python NLP 服务。此前“Python 建议/全 Go 未决”仅为历史决策记录,由本决定覆盖;spikes/english 保留历史验证,不接入产品。#6 按该方向实施,当前方案见工单最新启动评论;WordNet 3.0 仍为首个资源(英语释义),词形规则候选不等同于 spaCy 上下文消歧,原文及个人学习状态不按候选合并。
|
||||
|
||||
|
||||
## #6 正式词典规则(2026-09-11)
|
||||
|
||||
首个资源固定 Princeton WordNet 3.0,英语释义;ZIP 来源及 SHA 以 server/wordnet-resource.json 为准,LICENSE 原样保留于 server/WORDNET-LICENSE.txt。管理员上传指定包并配置显示名称,语言/来源/版本/格式固定;32MiB压缩、128MiB解压上限、成员与结构校验。资源为所有账号共享、仅管理员写;失败保持旧资源,重复上传复用id并启用,停用后查询返回 resource_missing。ZIP 随数据库备份,无运行时网络下载。
|
||||
|
||||
正式分词由 Go 完成:字母开始词,组合标记延续词,内部直/弯撇号连接字母;空白逐字保留,数字和符号为不可查询的 punctuation。start/end 是 Unicode code point 半开范围,另有UTF-16偏移;原文、CRLF、组合字符与emoji不归一化。只对查询键小写/NFC/撇号归一化。每次查询必须是本人ready章节内的完整单词,最大128码点,跨账号404,非法范围400。
|
||||
|
||||
exact优先;未命中再按WordNet异常表/词尾规则查候选,词性顺序n/v/a/r、最多12条释义,保留原数据s词性。返回lemma表示规则候选,不承诺上下文消歧;歧义不会合并个人学习状态。not_found与resource_missing区分,网络错误可重试,不阻断阅读。#3使用spaCy上下文lemma的实验路径由全Go规则候选替代。
|
||||
|
||||
个人释义目前仅当前选择的临时草稿,界面标记未保存;换词、关闭、换章、离页和身份变化清理。持久化与个人词汇状态由#7实现,不把临时输入宣传为保存成功。
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
generated: true (请先修改 Gitea Wiki,禁止直接编辑本文件)
|
||||
wiki_page: Local-Development-and-Verification
|
||||
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Local-Development-and-Verification.-
|
||||
wiki_revision: 3bce5fae964b1050513c37319262679814d68f14
|
||||
synchronized_at: 2026-09-11T03:01:49Z
|
||||
wiki_revision: bbd6e618f4339aa45d81bba0492045b60581a735
|
||||
synchronized_at: 2026-09-11T03:59:02Z
|
||||
<!-- gitea-wiki-mirror:end -->
|
||||
|
||||
# 本地开发与验证
|
||||
@@ -305,3 +305,18 @@ node --test spikes/english/view.test.mjs
|
||||
注意:MySQL 会话时区为 SYSTEM(本机为 UTC+8),而服务按 UTC 存储 DATETIME;核对任务时间时使用 UTC_TIMESTAMP 而不是 NOW(),否则会出现 8 小时的假偏差。
|
||||
|
||||
未在本轮验证:R3 的真实浏览器时序(离页与响应同时发生)只由单测覆盖;真机手机证据仍属 #4 缺口。
|
||||
|
||||
|
||||
## #6 部署与验证(2026-09-11)
|
||||
|
||||
正式后端 schema v4,只增加 lexgo_dictionaries。显式 migrate 后启动纯Go lexgo.exe;Python scripts/server.py 仍是开发命令封装,产品运行不依赖Python NLP。
|
||||
|
||||
本机已从v3升级v4并重启lexgo-api,升级前后sys_user/lexgo_spaces/lexgo_sessions/lexgo_books/lexgo_chapters/lexgo_ingest_jobs计数一致。旧二进制保存在忽略的 .local/lexgo-pre-issue6.exe。回退:停止API,恢复旧二进制,将已确认v4的schema标记恢复3,保留新增资源表及全部学习数据,再启动旧API;不要删除数据或重新bootstrap。
|
||||
|
||||
管理端 http://127.0.0.1:5174 的“英语词典”页可下载指定包并导入/启停;本机已导入固定WordNet3.0,155287个词形/词性索引项。学习端 http://127.0.0.1:5173 打开本人ready章节,点词或Enter/空格查询,Escape关闭。went/mice应出现go/mouse候选。源包、本机凭据与测试证据仅存在忽略的.local,不进入Git。
|
||||
|
||||
验证:Go全包MySQL集成(专用lexgo_test_issue5)与go vet通过;学习端53单测、类型检查/构建、默认5173 Playwright3项通过;管理端31单测与lint通过,构建含已有Sass弃用与bundle体积提示;治理56测试与strict通过。实际ZIP解析、schema3→4原文/任务保留、权限、失败保留、重复启用、冷缓存路由重建都有覆盖;冷缓存测试不是完整备份恢复演练。
|
||||
|
||||
主审真实API联调使用真实管理端session模块经5174代理上传;经5173代理两测试账号分别创建虚构章节并验证精确/不规则词形、Unicode原文片段、越权404/普通用户管理403、停用/重复导入启用、错误ZIP保留资源。新增测试书籍id3/4、章节id8/9归issue5_a/issue5_b,没有修改其他账号的书籍。
|
||||
|
||||
常驻5173一度返回空白页:Vue模块转换500、代理缺失;同代码隔离服务正常,只重启lexgo-learner加载配置后恢复,默认E2E通过,未改启动配置。桌面交互浏览器工具因旧会话失效未完成手工真实UI联调;已有项目Playwright使用模拟API,真实API验证另列。手机仅窄屏自动测试,真机证据仍未补齐。
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
generated: true (请先修改 Gitea Wiki,禁止直接编辑本文件)
|
||||
wiki_page: Product-Requirements-Overview
|
||||
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Product-Requirements-Overview.-
|
||||
wiki_revision: 125df697d3d45dee098616d25970bb65a96727cd
|
||||
synchronized_at: 2026-09-10T16:35:19Z
|
||||
wiki_revision: 099f1c803ffb69edc3fe07582d22dcf9a57647a1
|
||||
synchronized_at: 2026-09-11T03:59:10Z
|
||||
<!-- gitea-wiki-mirror:end -->
|
||||
|
||||
# 产品需求总览
|
||||
@@ -229,3 +229,13 @@ Anki(U20)、YouTube/Jellyfin 远程字幕(U07/A08)、全量旧实例迁
|
||||
2026-09-10:#5“粘贴英语文本,处理后进入本人章节阅读”已实现并待用户验收,覆盖 F01 基础(书库与章节)、F02(粘贴导入)、F04 基础(处理状态与失败重试)与 F05 原文(可读原文与章节切换)。本节取代此前“阅读、导入尚未实现”的表述:粘贴导入与原文阅读已实现;点词查词、词典、个人词语状态、复习与统计仍未实现(#6~#15)。schema 升级为 v3,新增 lexgo_books、lexgo_chapters、lexgo_ingest_jobs。
|
||||
|
||||
范围边界不变:本单按 Go 处理,不接入 Python NLP,因此不产生 token、lemma 或词典索引;Go+Python NLP 与全 Go 路线仍未确认,正式接入前必须由用户确认。真机手机详细证据仍缺失(#4 缺口保持)。
|
||||
|
||||
|
||||
## 全 Go 正式架构决定(2026-09-11,#6)
|
||||
|
||||
用户已明确选择全 Go:正式英语分词、原文位置映射、本地词典解析和词形候选查询由 Go 后端完成,不运行 Python NLP 服务。此前“Python 建议/全 Go 未决”仅为历史决策记录,由本决定覆盖;spikes/english 保留历史验证,不接入产品。#6 按该方向实施,当前方案见工单最新启动评论;WordNet 3.0 仍为首个资源(英语释义),词形规则候选不等同于 spaCy 上下文消歧,原文及个人学习状态不按候选合并。
|
||||
|
||||
|
||||
## #6 交付范围更新(2026-09-11)
|
||||
|
||||
用户确认全Go后,英语词典配置与阅读器点词查义已实现待验收:共享WordNet3.0英语释义、管理员导入/启停、本人章节点击/键盘查词、加载/无结果/资源不可用/网络失败/关闭状态。词形结果是规则候选,不提供上下文词性消歧。个人释义输入是未保存临时草稿,#7才持久化。沿用已验收v1;手机底部45dvh面板自动测试通过,真机缺口保留。#5已关闭并合入main,#6尚不关闭。
|
||||
|
||||
+7
-2
@@ -2,8 +2,8 @@
|
||||
generated: true (请先修改 Gitea Wiki,禁止直接编辑本文件)
|
||||
wiki_page: Home
|
||||
wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Home
|
||||
wiki_revision: 5b936b75caeec89cd54f7147c842f0b66f901307
|
||||
synchronized_at: 2026-09-10T16:35:18Z
|
||||
wiki_revision: 241fa63d8141ed9f1992421346d42f9b87453b0a
|
||||
synchronized_at: 2026-09-11T03:58:55Z
|
||||
<!-- gitea-wiki-mirror:end -->
|
||||
|
||||
# LexGo 文档入口
|
||||
@@ -65,3 +65,8 @@ Quant-UX 原型 v1 已通过用户验收。[桌面预览](https://qux.ilapage.cn
|
||||
#4 阅读选择小样已实现桌面鼠标/键盘、范围调整与原文位置验证,入口 http://127.0.0.1:5184/;固定版本 LinguaCafe 四项源码对照已记录。用户已验收并关闭 #4;真实手机详细测试证据仍缺失,详见本地验证页。#21 仍待实施。
|
||||
|
||||
#5 粘贴导入与章节阅读已实现,待用户验收:schema v3 新增 books/chapters/ingest_jobs,学习端具备粘贴导入、书库、处理状态与原文阅读;查词、词典与复习仍未实现。本单按 Go 处理,Go+Python NLP 与全 Go 路线仍未决,正式接入前须用户确认。
|
||||
|
||||
|
||||
## 当前进度(2026-09-11)
|
||||
|
||||
#5已验收,#17/#19/#20/#22/#23按依赖顺序合入main。#6已按用户确认的全Go方向实现英语词典与阅读点词,待验收;原文/账户隔离保留。正式NLP不使用Python服务。管理端“英语词典”导入指定WordNet3.0,学习端打开本人章节即可查词。个人释义当前不保存,后续#7实现。详见#6工单、架构和本地开发页面。
|
||||
|
||||
@@ -52,6 +52,17 @@ test('paste English text, watch a chapter finish processing, then read it verbat
|
||||
chapter: { ...chapterPayload(), contentSha256: 'fictional-sha256', ...(status === 'ready' ? { originalText: pasted } : {}) },
|
||||
navigation: { previousChapterId: null, nextChapterId: null },
|
||||
}
|
||||
} else if (path === '/api/v1/chapters/55/tokens') {
|
||||
let offset = 0
|
||||
const tokens = (pasted.match(/[A-Za-z]+|\s+|[^A-Za-z\s]+/g) ?? []).map(text => {
|
||||
const start = offset
|
||||
offset += text.length
|
||||
return { text, start, end: offset, startUtf16: start, endUtf16: offset, kind: /^[A-Za-z]+$/.test(text) ? 'word' : /^\s+$/.test(text) ? 'space' : 'punctuation' }
|
||||
})
|
||||
data = { textSha256: 'fictional-sha256', tokens }
|
||||
} else if (path === '/api/v1/lookup') {
|
||||
expect(route.request().postDataJSON()).toEqual({ chapterId: 55, start: 0, end: 5 })
|
||||
data = { status: 'exact', query: 'first', matchedForm: 'first', candidates: [], entries: [{ lemma: 'first', pos: 'adjective', definition: 'Coming before all others.', examples: ['The first fictional chapter.'] }] }
|
||||
}
|
||||
await route.fulfill({ status: statusCode, json: { code: 200, data } })
|
||||
})
|
||||
@@ -89,4 +100,25 @@ test('paste English text, watch a chapter finish processing, then read it verbat
|
||||
expect(await readerText.evaluate(element => getComputedStyle(element).whiteSpace)).toBe('pre-wrap')
|
||||
await expect(page.getByRole('button', { name: '上一章' })).toBeDisabled()
|
||||
await expect(page.getByRole('button', { name: '下一章' })).toBeDisabled()
|
||||
const word = page.getByRole('button', { name: '查询 First', exact: true })
|
||||
await word.focus()
|
||||
await word.press('Enter')
|
||||
await expect(page.getByText('Coming before all others.')).toBeVisible()
|
||||
expect(await readerText.evaluate(element => element.textContent)).toBe(pasted)
|
||||
await page.getByLabel('个人释义 未保存').fill('虚构的本地草稿')
|
||||
await page.getByRole('button', { name: '关闭释义' }).press('Escape')
|
||||
await expect(page.locator('.lookup-panel')).toHaveCount(0)
|
||||
await expect(word).toBeFocused()
|
||||
|
||||
// Browser narrow viewport check only; this is not real-device acceptance.
|
||||
await page.setViewportSize({ width: 390, height: 844 })
|
||||
await word.click()
|
||||
await expect(page.getByText('Coming before all others.')).toBeVisible()
|
||||
const panelBounds = await page.locator('.lookup-panel').boundingBox()
|
||||
expect(panelBounds!.y + panelBounds!.height).toBeLessThanOrEqual(845)
|
||||
expect(panelBounds!.height).toBeLessThanOrEqual(844 * 0.46)
|
||||
expect(await page.locator('.lookup-panel').evaluate(element => getComputedStyle(element).position)).toBe('fixed')
|
||||
const wordBounds = await word.boundingBox()
|
||||
expect(wordBounds!.y + wordBounds!.height).toBeLessThanOrEqual(panelBounds!.y)
|
||||
expect(await page.getByLabel('个人释义 未保存').inputValue()).toBe('')
|
||||
})
|
||||
|
||||
@@ -0,0 +1,167 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import { flushPromises, mount, type VueWrapper } from '@vue/test-utils'
|
||||
import { createPinia, setActivePinia } from 'pinia'
|
||||
import { createMemoryHistory, createRouter } from 'vue-router'
|
||||
import ReaderView from '../views/ReaderView.vue'
|
||||
import { useSessionStore } from '../stores/session'
|
||||
import { useReaderLookup } from '../composables/useReaderLookup'
|
||||
import { effectScope, ref } from 'vue'
|
||||
import type { ChapterDetail } from '../stores/library'
|
||||
|
||||
// Fictitious text includes astral, combining, CRLF and repeated whitespace.
|
||||
const original = '😀 Cats\r\n café!'
|
||||
const fragments = [['😀', 'punctuation'], [' ', 'space'], ['Cats', 'word'], ['\r\n ', 'space'], ['café', 'word'], ['!', 'punctuation']]
|
||||
let cp = 0, utf16 = 0
|
||||
const tokens = fragments.map(([text = '', kind]) => {
|
||||
const token = { text, kind, start: cp, end: cp + [...text].length, startUtf16: utf16, endUtf16: utf16 + text.length }
|
||||
cp = token.end; utf16 = token.endUtf16
|
||||
return token
|
||||
})
|
||||
const chapter = { id: 55, bookId: 1, title: '虚构章节', status: 'ready', originalText: original, contentSha256: 'same-sha' }
|
||||
const ok = (data: unknown) => new Response(JSON.stringify({ code: 200, data }))
|
||||
const result = (query: string, status = 'exact') => ({ status, query, matchedForm: query, candidates: [], entries: status === 'exact' ? [{ lemma: query, pos: 'noun', definition: `Definition of ${query}`, examples: ['A fictional example.'] }] : [] })
|
||||
let wrapper: VueWrapper | undefined
|
||||
async function open(mockLookup: (body: Record<string, number>) => Promise<Response> = async () => ok(result('cat')), tokenData: unknown = { textSha256: 'same-sha', tokens }) {
|
||||
const fetchMock = vi.spyOn(globalThis, 'fetch').mockImplementation(async (input, options) => {
|
||||
if (String(input).endsWith('/tokens')) return ok(tokenData)
|
||||
if (String(input).endsWith('/lookup')) return mockLookup(JSON.parse(String(options?.body)))
|
||||
return ok({ chapter, navigation: { previousChapterId: null, nextChapterId: 56 } })
|
||||
})
|
||||
useSessionStore().user = { id: 42, username: 'fictional', role: 'learner' }
|
||||
const router = createRouter({ history: createMemoryHistory(), routes: [{ path: '/chapters/:id', component: ReaderView }, { path: '/', component: { template: '<div />' } }] })
|
||||
await router.push('/chapters/55')
|
||||
wrapper = mount(ReaderView, { attachTo: document.body, global: { plugins: [router] } })
|
||||
await flushPromises()
|
||||
return { fetchMock, router, view: wrapper }
|
||||
}
|
||||
describe('reader word lookup', () => {
|
||||
beforeEach(() => { setActivePinia(createPinia()); sessionStorage.clear() })
|
||||
afterEach(() => { wrapper?.unmount(); wrapper = undefined; vi.restoreAllMocks() })
|
||||
it('keeps Unicode original text exact and sends only chapter and code-point offsets', async () => {
|
||||
const { view, fetchMock } = await open()
|
||||
expect(view.get('.reader-text').element.textContent).toBe(original)
|
||||
const words = view.findAll('.reader-word')
|
||||
expect(words).toHaveLength(2)
|
||||
await words[0]!.trigger('click'); await flushPromises()
|
||||
const call = fetchMock.mock.calls.find(([url]) => String(url).endsWith('/lookup'))!
|
||||
expect(call[1]?.method).toBe('POST')
|
||||
expect(JSON.parse(String(call[1]?.body))).toEqual({ chapterId: 55, start: 2, end: 6 })
|
||||
expect(view.text()).toContain('Definition of cat')
|
||||
expect(view.text()).toContain('A fictional example.')
|
||||
})
|
||||
it('discards older word responses and clears the selection draft', async () => {
|
||||
let finish!: (response: Response) => void
|
||||
const { view } = await open(body => body.start === 2 ? new Promise(resolve => { finish = resolve }) : Promise.resolve(ok(result('cafe', 'not_found'))))
|
||||
await view.findAll('.reader-word')[0]!.trigger('click')
|
||||
expect(view.text()).toContain('正在查询')
|
||||
await view.findAll('.reader-word')[1]!.trigger('click'); await flushPromises()
|
||||
await view.get('textarea').setValue('虚构私人草稿')
|
||||
finish(ok(result('OLD'))); await flushPromises()
|
||||
expect(view.text()).not.toContain('Definition of OLD')
|
||||
expect(view.text()).toContain('未找到释义')
|
||||
expect(view.text()).toContain('未保存')
|
||||
await view.findAll('.reader-word')[0]!.trigger('click')
|
||||
expect((view.get('textarea').element as HTMLTextAreaElement).value).toBe('')
|
||||
})
|
||||
it.each(['resource_missing', 'not_found', 'error'])('keeps reading and permits a local draft and retry for %s', async status => {
|
||||
const { view, fetchMock } = await open(async () => { if (status === 'error') throw new Error('暂时无法查询'); return ok(result('cat', status)) })
|
||||
await view.get('.reader-word').trigger('click'); await flushPromises()
|
||||
await view.get('textarea').setValue('仅当前词的虚构草稿')
|
||||
expect(view.get('.reader-text').element.textContent).toBe(original)
|
||||
expect(view.text()).toContain('未保存')
|
||||
expect(fetchMock.mock.calls.filter(([url]) => String(url).endsWith('/lookup'))).toHaveLength(1)
|
||||
await view.get('[data-testid="lookup-retry"]').trigger('click'); await flushPromises()
|
||||
expect(fetchMock.mock.calls.filter(([url]) => String(url).endsWith('/lookup'))).toHaveLength(2)
|
||||
})
|
||||
it.each([{ textSha256: 'wrong', tokens }, { textSha256: 'same-sha', tokens: tokens.slice(1) }])('falls back to original when tokens do not match', async data => {
|
||||
const { view } = await open(undefined, data)
|
||||
expect(view.get('.reader-text').element.textContent).toBe(original)
|
||||
expect(view.find('.reader-word').exists()).toBe(false)
|
||||
expect(view.find('[data-testid="tokens-retry"]').exists()).toBe(true)
|
||||
})
|
||||
it('closes with Escape and restores the word focus without scrolling', async () => {
|
||||
const { view } = await open()
|
||||
const word = view.get('.reader-word').element as HTMLElement
|
||||
word.focus()
|
||||
await view.get('.reader-word').trigger('keydown', { key: 'Enter' }); await flushPromises()
|
||||
const focus = vi.spyOn(word, 'focus')
|
||||
await view.get('.lookup-panel').trigger('keydown', { key: 'Escape' })
|
||||
await flushPromises()
|
||||
expect(view.find('.lookup-panel').exists()).toBe(false)
|
||||
expect(document.activeElement).toBe(word)
|
||||
expect(focus).toHaveBeenCalledWith({ preventScroll: true })
|
||||
})
|
||||
it('retries a failed token fetch without losing original text', async () => {
|
||||
const { view, fetchMock } = await open()
|
||||
fetchMock.mockRejectedValueOnce(new Error('网络暂不可用'))
|
||||
// A new chapter load invalidates the old token rendering.
|
||||
const { useLibraryStore } = await import('../stores/library')
|
||||
useLibraryStore().chapter = { ...useLibraryStore().chapter!, contentSha256: 'retry-sha' }
|
||||
await flushPromises()
|
||||
expect(view.get('.reader-text').element.textContent).toBe(original)
|
||||
expect(view.find('.reader-word').exists()).toBe(false)
|
||||
fetchMock.mockResolvedValueOnce(ok({ textSha256: 'retry-sha', tokens }))
|
||||
await view.get('[data-testid="tokens-retry"]').trigger('click'); await flushPromises()
|
||||
expect(view.findAll('.reader-word')).toHaveLength(2)
|
||||
})
|
||||
it('does not accept a late token response after identity changes', async () => {
|
||||
const { view, fetchMock } = await open(undefined, { textSha256: 'wrong', tokens })
|
||||
let finish!: (response: Response) => void
|
||||
fetchMock.mockImplementationOnce(() => new Promise(resolve => { finish = resolve }))
|
||||
await view.get('[data-testid="tokens-retry"]').trigger('click')
|
||||
await useSessionStore().logout()
|
||||
useSessionStore().user = { id: 42, username: 'fictional', role: 'learner' }
|
||||
finish(ok({ textSha256: 'same-sha', tokens })); await flushPromises()
|
||||
expect(view.find('.reader-word').exists()).toBe(false)
|
||||
})
|
||||
it('moves a covered selected word above the sheet and restores the prior scroll on close', async () => {
|
||||
useSessionStore().user = { id: 42, username: 'fictional', role: 'learner' }
|
||||
vi.spyOn(globalThis, 'fetch').mockResolvedValue(ok(result('cat')))
|
||||
const scrollBy = vi.spyOn(window, 'scrollBy').mockImplementation(() => {})
|
||||
const scrollTo = vi.spyOn(window, 'scrollTo').mockImplementation(() => {})
|
||||
const scope = effectScope()
|
||||
const lookup = scope.run(() => useReaderLookup(ref(chapter as ChapterDetail)))!
|
||||
const element = document.createElement('span')
|
||||
vi.spyOn(element, 'getBoundingClientRect').mockReturnValue({ bottom: 700 } as DOMRect)
|
||||
lookup.select(tokens[2] as never, element)
|
||||
lookup.keepSelectionVisible(500)
|
||||
expect(scrollBy).toHaveBeenCalledWith({ top: 216, behavior: 'instant' })
|
||||
lookup.close()
|
||||
expect(scrollTo).toHaveBeenCalledWith({ top: 0, left: 0, behavior: 'instant' })
|
||||
scope.stop()
|
||||
})
|
||||
it('preserves a later manual reading scroll when closing the sheet', () => {
|
||||
useSessionStore().user = { id: 42, username: 'fictional', role: 'learner' }
|
||||
vi.spyOn(globalThis, 'fetch').mockResolvedValue(ok(result('cat')))
|
||||
let scrollY = 100
|
||||
vi.spyOn(window, 'scrollY', 'get').mockImplementation(() => scrollY)
|
||||
vi.spyOn(window, 'scrollBy').mockImplementation((...args: unknown[]) => {
|
||||
const options = args[0] as ScrollToOptions
|
||||
scrollY += options.top ?? 0
|
||||
})
|
||||
const scrollTo = vi.spyOn(window, 'scrollTo').mockImplementation(() => {})
|
||||
const scope = effectScope()
|
||||
const lookup = scope.run(() => useReaderLookup(ref(chapter as ChapterDetail)))!
|
||||
const element = document.createElement('span')
|
||||
vi.spyOn(element, 'getBoundingClientRect').mockReturnValue({ bottom: 700 } as DOMRect)
|
||||
lookup.select(tokens[2] as never, element)
|
||||
lookup.keepSelectionVisible(500)
|
||||
scrollY += 300 // The reader continues down the page while the sheet is open.
|
||||
lookup.close()
|
||||
expect(scrollTo).not.toHaveBeenCalled()
|
||||
expect(scrollY).toBe(616)
|
||||
scope.stop()
|
||||
})
|
||||
it.each(['logout', 'chapter', 'unmount'])('invalidates pending lookup on %s', async action => {
|
||||
let finish!: (response: Response) => void
|
||||
const { view, router } = await open(() => new Promise(resolve => { finish = resolve }))
|
||||
await view.get('.reader-word').trigger('click')
|
||||
if (action === 'logout') await useSessionStore().logout()
|
||||
else if (action === 'chapter') await router.push('/chapters/56')
|
||||
else { view.unmount(); wrapper = undefined }
|
||||
finish(ok(result('OLD'))); await flushPromises()
|
||||
if (action === 'unmount') expect(document.querySelector('.lookup-panel')).toBeNull()
|
||||
else expect(view.find('.lookup-panel').exists()).toBe(false)
|
||||
expect(view.text()).not.toContain('Definition of OLD')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,55 @@
|
||||
<script setup lang="ts">
|
||||
import { onMounted, onUnmounted, ref } from 'vue'
|
||||
import { ElButton } from 'element-plus'
|
||||
import type { LookupResult } from '../composables/useReaderLookup'
|
||||
defineProps<{ word: string; result: LookupResult | null; loading: boolean; error: string }>()
|
||||
const draft = defineModel<string>('draft', { required: true })
|
||||
const emit = defineEmits<{ close: []; retry: []; resize: [top: number] }>()
|
||||
const heading = ref<HTMLElement | null>(null)
|
||||
const panel = ref<HTMLElement | null>(null)
|
||||
let observer: ResizeObserver | undefined
|
||||
function resized() {
|
||||
if (window.innerWidth <= 760 && panel.value) emit('resize', panel.value.getBoundingClientRect().top)
|
||||
}
|
||||
onMounted(() => {
|
||||
heading.value?.focus({ preventScroll: true })
|
||||
resized()
|
||||
if (typeof ResizeObserver !== 'undefined' && panel.value) {
|
||||
observer = new ResizeObserver(resized)
|
||||
observer.observe(panel.value)
|
||||
}
|
||||
window.addEventListener('resize', resized)
|
||||
})
|
||||
onUnmounted(() => { observer?.disconnect(); window.removeEventListener('resize', resized) })
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<aside ref="panel" class="lookup-panel" aria-labelledby="lookup-heading" @keydown.esc.stop.prevent="$emit('close')">
|
||||
<header class="lookup-heading">
|
||||
<h2 id="lookup-heading" ref="heading" tabindex="-1">{{ word }}</h2>
|
||||
<ElButton text aria-label="关闭释义" @click="$emit('close')">关闭</ElButton>
|
||||
</header>
|
||||
<div class="lookup-content" aria-live="polite" :aria-busy="loading">
|
||||
<p v-if="loading" role="status" class="subtle">正在查询…</p>
|
||||
<p v-else-if="error" role="alert" class="lookup-message">{{ error }}</p>
|
||||
<template v-else-if="result">
|
||||
<p v-if="result.status === 'resource_missing'" class="lookup-message">词典资源暂不可用。</p>
|
||||
<p v-else-if="result.status === 'not_found'" class="lookup-message">未找到释义。</p>
|
||||
<p v-if="result.status === 'lemma'" class="subtle">词形匹配:{{ result.matchedForm }}(规则候选)</p>
|
||||
<ol v-if="result.entries.length" class="lookup-senses">
|
||||
<li v-for="(entry, index) in result.entries" :key="index">
|
||||
<p class="sense-heading"><strong>{{ entry.lemma }}</strong> <span>{{ entry.pos }}</span></p>
|
||||
<p lang="en">{{ entry.definition }}</p>
|
||||
<blockquote v-for="(example, exampleIndex) in entry.examples" :key="exampleIndex" lang="en">{{ example }}</blockquote>
|
||||
</li>
|
||||
</ol>
|
||||
<p v-if="result.resource" class="subtle">{{ result.resource.name }} · {{ result.resource.version }}</p>
|
||||
</template>
|
||||
<ElButton v-if="!loading && (error || result?.status === 'resource_missing' || result?.status === 'not_found')" data-testid="lookup-retry" @click="$emit('retry')">重试查询</ElButton>
|
||||
</div>
|
||||
<div class="lookup-draft">
|
||||
<label for="definition-draft">个人释义 <span>未保存</span></label>
|
||||
<textarea id="definition-draft" v-model="draft" rows="3" placeholder="记下当前词的释义" />
|
||||
</div>
|
||||
</aside>
|
||||
</template>
|
||||
@@ -0,0 +1,10 @@
|
||||
<script setup lang="ts">
|
||||
import type { ReaderToken } from '../composables/useReaderLookup'
|
||||
defineProps<{ tokens: ReaderToken[]; original: string; selectedStart?: number }>()
|
||||
const emit = defineEmits<{ select: [token: ReaderToken, element: HTMLElement] }>()
|
||||
function select(token: ReaderToken, event: Event) { emit('select', token, event.currentTarget as HTMLElement) }
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<article class="reader-text"><template v-if="tokens.length"><template v-for="token in tokens" :key="token.start"><span v-if="token.kind === 'word'" role="button" tabindex="0" class="reader-word" :class="{ 'is-selected': selectedStart === token.start }" :aria-label="`查询 ${token.text}`" :aria-pressed="selectedStart === token.start" @click="select(token, $event)" @keydown.enter.prevent="select(token, $event)" @keydown.space.prevent="select(token, $event)">{{ token.text }}</span><template v-else>{{ token.text }}</template></template></template><template v-else>{{ original }}</template></article>
|
||||
</template>
|
||||
@@ -0,0 +1,128 @@
|
||||
import { onScopeDispose, ref, watch, type Ref } from 'vue'
|
||||
import type { ChapterDetail } from '../stores/library'
|
||||
import { useSessionStore } from '../stores/session'
|
||||
|
||||
export interface ReaderToken { text: string; start: number; end: number; startUtf16: number; endUtf16: number; kind: 'word' | 'space' | 'punctuation' }
|
||||
export interface LookupResult {
|
||||
status: 'exact' | 'lemma' | 'not_found' | 'resource_missing'
|
||||
query: string
|
||||
matchedForm: string | null
|
||||
candidates: string[]
|
||||
entries: { lemma: string; pos: string; definition: string; examples: string[] }[]
|
||||
resource?: { name: string; version: string }
|
||||
}
|
||||
interface TokenResponse { textSha256: string; tokens: ReaderToken[] }
|
||||
|
||||
// Validate every coordinate before enabling selection; original text is always the fallback.
|
||||
function matchesChapter(data: TokenResponse, chapter: ChapterDetail): boolean {
|
||||
if (!data || data.textSha256 !== chapter.contentSha256 || !Array.isArray(data.tokens)) return false
|
||||
const text = chapter.originalText ?? ''
|
||||
let cp = 0, utf16 = 0
|
||||
for (const token of data.tokens) {
|
||||
if (typeof token.text !== 'string' || !token.text || !['word', 'space', 'punctuation'].includes(token.kind)) return false
|
||||
if (token.start !== cp || token.startUtf16 !== utf16) return false
|
||||
cp += [...token.text].length
|
||||
utf16 += token.text.length
|
||||
if (token.end !== cp || token.endUtf16 !== utf16 || text.slice(token.startUtf16, utf16) !== token.text) return false
|
||||
}
|
||||
return utf16 === text.length && data.tokens.map(token => token.text).join('') === text
|
||||
}
|
||||
|
||||
export function useReaderLookup(chapter: Ref<ChapterDetail | null>) {
|
||||
const session = useSessionStore()
|
||||
const tokens = ref<ReaderToken[]>([])
|
||||
const tokensError = ref('')
|
||||
const tokensLoading = ref(false)
|
||||
const selected = ref<ReaderToken | null>(null)
|
||||
const result = ref<LookupResult | null>(null)
|
||||
const loading = ref(false)
|
||||
const error = ref('')
|
||||
const draft = ref('')
|
||||
let tokenSequence = 0, lookupSequence = 0
|
||||
let origin: HTMLElement | null = null
|
||||
let scrollBeforeAdjustment: { top: number; left: number } | null = null
|
||||
let lastAdjustment: { top: number; left: number } | null = null
|
||||
|
||||
function stillAtAdjustment() {
|
||||
return lastAdjustment !== null && Math.abs(window.scrollY - lastAdjustment.top) <= 2
|
||||
&& Math.abs(window.scrollX - lastAdjustment.left) <= 2
|
||||
}
|
||||
|
||||
function keepSelectionVisible(panelTop: number) {
|
||||
if (!origin) return
|
||||
const coveredBy = origin.getBoundingClientRect().bottom - panelTop + 16
|
||||
if (coveredBy <= 0) return
|
||||
// A later manual scroll becomes the new reading position, including when
|
||||
// another resize subsequently needs to reveal the selected word again.
|
||||
if (!scrollBeforeAdjustment || !stillAtAdjustment()) scrollBeforeAdjustment = { top: window.scrollY, left: window.scrollX }
|
||||
window.scrollBy({ top: coveredBy, behavior: 'instant' })
|
||||
lastAdjustment = { top: window.scrollY, left: window.scrollX }
|
||||
}
|
||||
|
||||
function close(restoreFocus = true) {
|
||||
lookupSequence++
|
||||
selected.value = null
|
||||
result.value = null
|
||||
loading.value = false
|
||||
error.value = ''
|
||||
draft.value = ''
|
||||
if (restoreFocus && origin?.isConnected) origin.focus({ preventScroll: true })
|
||||
if (restoreFocus && scrollBeforeAdjustment && stillAtAdjustment()) window.scrollTo({ ...scrollBeforeAdjustment, behavior: 'instant' })
|
||||
scrollBeforeAdjustment = null
|
||||
lastAdjustment = null
|
||||
origin = null
|
||||
}
|
||||
function reset() {
|
||||
tokenSequence++
|
||||
tokens.value = []
|
||||
tokensError.value = ''
|
||||
tokensLoading.value = false
|
||||
close(false)
|
||||
}
|
||||
async function loadTokens() {
|
||||
const current = chapter.value
|
||||
if (!session.user || current?.status !== 'ready') return
|
||||
const seq = ++tokenSequence
|
||||
tokensLoading.value = true
|
||||
tokensError.value = ''
|
||||
try {
|
||||
const data = await session.request<TokenResponse>(`chapters/${current.id}/tokens`)
|
||||
if (seq !== tokenSequence) return
|
||||
if (!matchesChapter(data, current)) throw new Error('分词与正文不一致,请重试。')
|
||||
tokens.value = data.tokens
|
||||
} catch (reason) {
|
||||
if (seq !== tokenSequence) return
|
||||
tokens.value = []
|
||||
tokensError.value = reason instanceof Error ? reason.message : '单词暂时无法加载。'
|
||||
} finally { if (seq === tokenSequence) tokensLoading.value = false }
|
||||
}
|
||||
async function lookup() {
|
||||
const current = chapter.value
|
||||
const token = selected.value
|
||||
if (!session.user || !current || !token) return
|
||||
const seq = ++lookupSequence
|
||||
result.value = null
|
||||
error.value = ''
|
||||
loading.value = true
|
||||
try {
|
||||
const data = await session.request<LookupResult>('lookup', 'POST', { chapterId: current.id, start: token.start, end: token.end })
|
||||
if (seq === lookupSequence) result.value = data
|
||||
} catch (reason) {
|
||||
if (seq === lookupSequence) error.value = reason instanceof Error ? reason.message : '暂时无法查询,请重试。'
|
||||
} finally { if (seq === lookupSequence) loading.value = false }
|
||||
}
|
||||
function select(token: ReaderToken, element: HTMLElement) {
|
||||
close(false)
|
||||
origin = element
|
||||
selected.value = token
|
||||
void lookup()
|
||||
}
|
||||
watch(() => [chapter.value?.id, chapter.value?.status, chapter.value?.contentSha256, chapter.value?.originalText], () => {
|
||||
reset()
|
||||
void loadTokens()
|
||||
}, { immediate: true, flush: 'sync' })
|
||||
// Watch the identity object, including clear → login for the same account.
|
||||
watch(() => session.user, reset, { flush: 'sync' })
|
||||
onScopeDispose(reset)
|
||||
return { tokens, tokensError, tokensLoading, selected, result, loading, error, draft, loadTokens, lookup, select, close, reset, keepSelectionVisible }
|
||||
}
|
||||
@@ -95,6 +95,29 @@ a.chapter-name:hover { color: #315c43; text-decoration: underline; }
|
||||
.reader-text { white-space: pre-wrap; overflow-wrap: break-word; margin: 26px 0 0; font-family: Georgia, 'Times New Roman', 'Microsoft YaHei', serif; font-size: 17px; line-height: 2; }
|
||||
.processing-hint { padding: 36px 0; color: #748073; }
|
||||
.reader-nav { display: flex; align-items: center; justify-content: space-between; gap: 14px; flex-wrap: wrap; margin-top: 36px; padding-top: 22px; border-top: 1px solid #e0e3d8; }
|
||||
.reader-page.has-lookup { max-width: 1120px; }
|
||||
.reader-workspace { display: grid; grid-template-columns: minmax(0, 1fr); gap: 32px; align-items: start; }
|
||||
.has-lookup .reader-workspace { grid-template-columns: minmax(0, 1fr) 320px; }
|
||||
.reader-body { min-width: 0; }
|
||||
.reader-word { cursor: pointer; border-radius: 3px; }
|
||||
.reader-word:hover, .reader-word.is-selected { background: #e3e9d9; color: #264a35; }
|
||||
.reader-word:focus-visible { outline: 2px solid #bc803d; outline-offset: 2px; background: #eef2eb; }
|
||||
.tokens-notice { color: #748073; font-size: 14px; margin-top: 24px; }
|
||||
.lookup-panel { position: sticky; top: 24px; margin-top: 26px; padding: 22px; border: 1px solid #d9decf; border-radius: 12px; background: #fffdf8; max-height: calc(100dvh - 48px); overflow-y: auto; overflow-wrap: anywhere; }
|
||||
.lookup-heading { display: flex; align-items: start; justify-content: space-between; gap: 10px; border-bottom: 1px solid #e0e3d8; padding-bottom: 12px; }
|
||||
.lookup-heading h2 { margin: 6px 0; font-family: Georgia, serif; font-size: 25px; }
|
||||
.lookup-content { font-size: 15px; line-height: 1.7; }
|
||||
.lookup-message { color: #8b4324; }
|
||||
.lookup-senses { padding-left: 22px; }
|
||||
.lookup-senses li { padding-left: 3px; margin-bottom: 20px; }
|
||||
.lookup-senses p { margin: 8px 0; }
|
||||
.sense-heading span { color: #748073; font-size: 13px; }
|
||||
.lookup-senses blockquote { border-left: 2px solid #cbd9c9; margin: 10px 0; padding-left: 12px; color: #687568; font-style: italic; }
|
||||
.lookup-draft { border-top: 1px solid #e0e3d8; padding-top: 18px; margin-top: 20px; }
|
||||
.lookup-draft label { display: flex; justify-content: space-between; gap: 10px; font-size: 14px; }
|
||||
.lookup-draft label span { color: #8b794e; font-size: 12px; }
|
||||
.lookup-draft textarea { width: 100%; resize: vertical; padding: 12px; margin-top: 10px; font: inherit; line-height: 1.6; color: #233d31; border: 1px solid #d6dccf; border-radius: 8px; background: #fffefa; }
|
||||
.lookup-draft textarea:focus-visible { outline: 2px solid #bc803d; outline-offset: 2px; }
|
||||
@media (max-width: 760px) {
|
||||
.login-page { grid-template-columns: 1fr; }
|
||||
.welcome { padding: 28px; }
|
||||
@@ -116,5 +139,9 @@ a.chapter-name:hover { color: #315c43; text-decoration: underline; }
|
||||
.chapter-row { grid-template-columns: 30px minmax(0, 1fr); align-items: start; row-gap: 10px; padding: 15px 16px; }
|
||||
.chapter-row .status-chip, .chapter-row .el-button { grid-column: 2; justify-self: start; }
|
||||
.reader-text { font-size: 16px; line-height: 1.95; }
|
||||
.has-lookup .reader-workspace { grid-template-columns: minmax(0, 1fr); gap: 20px; }
|
||||
.reader-page.has-lookup { padding-bottom: calc(45dvh + 28px); }
|
||||
.lookup-panel { position: fixed; inset: auto 0 0; z-index: 20; max-height: 45dvh; margin-top: 0; padding: 16px 20px max(20px, env(safe-area-inset-bottom)); border-radius: 16px 16px 0 0; box-shadow: 0 -5px 24px #233d3114; }
|
||||
.lookup-heading { position: sticky; top: -16px; z-index: 1; background: #fffdf8; }
|
||||
.reader-nav .el-button { flex: 1; }
|
||||
}
|
||||
|
||||
@@ -4,6 +4,9 @@ import { RouterLink, useRoute, useRouter } from 'vue-router'
|
||||
import { ElButton } from 'element-plus'
|
||||
import { canRetry, statusLabel, useLibraryStore } from '../stores/library'
|
||||
import { useSessionStore } from '../stores/session'
|
||||
import { useReaderLookup } from '../composables/useReaderLookup'
|
||||
import ReaderTokens from '../components/ReaderTokens.vue'
|
||||
import LookupPanel from '../components/LookupPanel.vue'
|
||||
|
||||
const session = useSessionStore()
|
||||
const library = useLibraryStore()
|
||||
@@ -13,10 +16,12 @@ const retryError = ref('')
|
||||
|
||||
const chapterId = computed(() => Number(route.params.id))
|
||||
const chapter = computed(() => library.chapter)
|
||||
const lookup = useReaderLookup(chapter)
|
||||
// Retry uses the job id the chapter carries, no matter where it was loaded from.
|
||||
const retryable = computed(() => chapter.value !== null && canRetry(chapter.value))
|
||||
|
||||
async function load() {
|
||||
lookup.reset()
|
||||
retryError.value = ''
|
||||
await library.loadChapter(chapterId.value)
|
||||
}
|
||||
@@ -56,7 +61,7 @@ onUnmounted(() => library.closeChapter())
|
||||
<ElButton text @click="logout">退出登录</ElButton>
|
||||
</div>
|
||||
</header>
|
||||
<main class="page reader-page">
|
||||
<main class="page reader-page" :class="{ 'has-lookup': lookup.selected.value }" @keydown.esc="lookup.close()">
|
||||
<p v-if="library.chapterLoading && !chapter" role="status" class="loading">正在加载…</p>
|
||||
<div v-else-if="library.chapterError" class="notice">
|
||||
<p role="alert">{{ library.chapterError }}</p>
|
||||
@@ -80,7 +85,16 @@ onUnmounted(() => library.closeChapter())
|
||||
</div>
|
||||
<p v-else-if="chapter.status !== 'ready'" role="status" class="processing-hint">这一章还在{{ statusLabel(chapter.status) }},页面会自动刷新。</p>
|
||||
<p v-if="retryError" role="alert" class="notice">{{ retryError }}</p>
|
||||
<article v-if="chapter.status === 'ready'" class="reader-text">{{ library.readerText }}</article>
|
||||
<div v-if="chapter.status === 'ready'" class="reader-workspace">
|
||||
<div class="reader-body">
|
||||
<ReaderTokens :tokens="lookup.tokens.value" :original="library.readerText" :selected-start="lookup.selected.value?.start" @select="lookup.select" />
|
||||
<div v-if="lookup.tokensError.value" class="tokens-notice">
|
||||
<p role="status">{{ lookup.tokensError.value }}</p>
|
||||
<ElButton data-testid="tokens-retry" :loading="lookup.tokensLoading.value" @click="lookup.loadTokens">重试加载单词</ElButton>
|
||||
</div>
|
||||
</div>
|
||||
<LookupPanel v-if="lookup.selected.value" :word="lookup.selected.value.text" :result="lookup.result.value" :loading="lookup.loading.value" :error="lookup.error.value" v-model:draft="lookup.draft.value" @close="lookup.close()" @retry="lookup.lookup" @resize="lookup.keepSelectionVisible" />
|
||||
</div>
|
||||
<nav class="reader-nav" aria-label="章节切换">
|
||||
<ElButton :disabled="library.navigation.previousChapterId === null" @click="goTo(library.navigation.previousChapterId)">上一章</ElButton>
|
||||
<ElButton :disabled="library.navigation.nextChapterId === null" @click="goTo(library.navigation.nextChapterId)">下一章</ElButton>
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
WordNet Release 3.0
|
||||
|
||||
This software and database is being provided to you, the LICENSEE, by
|
||||
Princeton University under the following license. By obtaining, using
|
||||
and/or copying this software and database, you agree that you have
|
||||
read, understood, and will comply with these terms and conditions.:
|
||||
|
||||
Permission to use, copy, modify and distribute this software and
|
||||
database and its documentation for any purpose and without fee or
|
||||
royalty is hereby granted, provided that you agree to comply with
|
||||
the following copyright notice and statements, including the disclaimer,
|
||||
and that the same appear on ALL copies of the software, database and
|
||||
documentation, including modifications that you make for internal
|
||||
use or for distribution.
|
||||
|
||||
WordNet 3.0 Copyright 2006 by Princeton University. All rights reserved.
|
||||
|
||||
THIS SOFTWARE AND DATABASE IS PROVIDED "AS IS" AND PRINCETON
|
||||
UNIVERSITY MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
|
||||
IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PRINCETON
|
||||
UNIVERSITY MAKES NO REPRESENTATIONS OR WARRANTIES OF MERCHANT-
|
||||
ABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE OR THAT THE USE
|
||||
OF THE LICENSED SOFTWARE, DATABASE OR DOCUMENTATION WILL NOT
|
||||
INFRINGE ANY THIRD PARTY PATENTS, COPYRIGHTS, TRADEMARKS OR
|
||||
OTHER RIGHTS.
|
||||
|
||||
The name of Princeton University or Princeton may not be used in
|
||||
advertising or publicity pertaining to distribution of the software
|
||||
and/or database. Title to copyright in this software, database and
|
||||
any associated documentation shall at all times remain with
|
||||
Princeton University and LICENSEE agrees to preserve same.
|
||||
@@ -72,6 +72,9 @@ func Migrate(db *gorm.DB) error {
|
||||
if current < 3 {
|
||||
statements = append(statements, schemaV3Statements...)
|
||||
}
|
||||
if current < 4 {
|
||||
statements = append(statements, schemaV4Statements...)
|
||||
}
|
||||
for i, s := range statements {
|
||||
if _, err = conn.ExecContext(ctx, s); err != nil {
|
||||
var sqlErr *driver.MySQLError
|
||||
@@ -85,7 +88,18 @@ func Migrate(db *gorm.DB) error {
|
||||
return err
|
||||
}
|
||||
|
||||
const schemaVersion = 3
|
||||
const schemaVersion = 4
|
||||
|
||||
var schemaV4Statements = []string{
|
||||
`CREATE TABLE IF NOT EXISTS lexgo_dictionaries (
|
||||
id BIGINT PRIMARY KEY, name VARCHAR(120) NOT NULL, language VARCHAR(16) NOT NULL,
|
||||
version VARCHAR(32) NOT NULL, source VARCHAR(512) NOT NULL, format VARCHAR(32) NOT NULL,
|
||||
sha256 CHAR(64) CHARACTER SET ascii COLLATE ascii_bin NOT NULL,
|
||||
entry_count INT NOT NULL, enabled BOOLEAN NOT NULL DEFAULT TRUE,
|
||||
archive LONGBLOB NOT NULL, updated_at DATETIME(3) NOT NULL,
|
||||
CHECK (id = 1), CHECK (language = 'en')
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4`,
|
||||
}
|
||||
|
||||
var schemaV2Statements = []string{
|
||||
`CREATE TABLE IF NOT EXISTS sys_user (
|
||||
|
||||
@@ -0,0 +1,320 @@
|
||||
package lexgo
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/gin-gonic/gin"
|
||||
admin "go-admin/app/admin/models"
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/clause"
|
||||
)
|
||||
|
||||
// ID 1 is the sole English corpus slot. Keeping the validated ZIP in MySQL
|
||||
// makes ordinary database backups include the resource needed after a restart.
|
||||
type DictionaryResource struct {
|
||||
ID int64 `gorm:"primaryKey;autoIncrement:false"`
|
||||
Name string
|
||||
Language string
|
||||
Version string
|
||||
Source string
|
||||
Format string
|
||||
SHA256 string `gorm:"column:sha256"`
|
||||
EntryCount int
|
||||
Enabled bool
|
||||
Archive []byte
|
||||
UpdatedAt time.Time
|
||||
}
|
||||
|
||||
func (DictionaryResource) TableName() string { return "lexgo_dictionaries" }
|
||||
|
||||
type DictionaryView struct {
|
||||
ID int64 `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Language string `json:"language"`
|
||||
Version string `json:"version"`
|
||||
Source string `json:"source"`
|
||||
Format string `json:"format"`
|
||||
Status string `json:"status"`
|
||||
Enabled bool `json:"enabled"`
|
||||
SHA256 string `json:"sha256"`
|
||||
EntryCount int `json:"entryCount"`
|
||||
UpdatedAt time.Time `json:"updatedAt"`
|
||||
}
|
||||
|
||||
func dictionaryView(r DictionaryResource, status string) DictionaryView {
|
||||
return DictionaryView{r.ID, r.Name, r.Language, r.Version, r.Source, r.Format, status, r.Enabled, r.SHA256, r.EntryCount, r.UpdatedAt}
|
||||
}
|
||||
|
||||
type DictionaryImportResult struct {
|
||||
Resource DictionaryView `json:"resource"`
|
||||
Duplicate bool `json:"duplicate"`
|
||||
}
|
||||
type ChapterTokens struct {
|
||||
TextSHA256 string `json:"textSha256"`
|
||||
Tokens []TextToken `json:"tokens"`
|
||||
}
|
||||
|
||||
// Each router keeps at most one immutable parsed corpus; no private chapter or
|
||||
// lookup data enters the cache. The mutex also coalesces simultaneous cold loads.
|
||||
type dictionaryCache struct {
|
||||
mu sync.Mutex
|
||||
sha string
|
||||
engine *WordNet
|
||||
}
|
||||
|
||||
func (cache *dictionaryCache) load(tx *gorm.DB, r DictionaryResource) (*WordNet, error) {
|
||||
cache.mu.Lock()
|
||||
defer cache.mu.Unlock()
|
||||
if r.SHA256 != WordNetSHA {
|
||||
return nil, errors.New("unsupported resource checksum")
|
||||
}
|
||||
if cache.sha == r.SHA256 && cache.engine != nil {
|
||||
return cache.engine, nil
|
||||
}
|
||||
var stored DictionaryResource
|
||||
if err := tx.Select("id", "archive").Where("id = ? AND sha256 = ?", r.ID, r.SHA256).First(&stored).Error; err != nil {
|
||||
return nil, err
|
||||
}
|
||||
engine, err := ParseWordNet(stored.Archive)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
cache.sha = r.SHA256
|
||||
cache.engine = engine
|
||||
return engine, nil
|
||||
}
|
||||
|
||||
func readyOwnedChapter(tx *gorm.DB, owner int, id int64) (Chapter, error) {
|
||||
var chapter Chapter
|
||||
err := tx.Where("id = ? AND owner_id = ?", id, owner).First(&chapter).Error
|
||||
if errors.Is(err, gorm.ErrRecordNotFound) {
|
||||
return chapter, failure(404, "章节不存在")
|
||||
}
|
||||
if err != nil {
|
||||
return chapter, err
|
||||
}
|
||||
if chapter.Status != statusReady {
|
||||
return chapter, failure(409, "章节尚未就绪,请稍后重试")
|
||||
}
|
||||
return chapter, nil
|
||||
}
|
||||
|
||||
func readDictionaryUpload(c *gin.Context) (DictionaryResource, error) {
|
||||
bad := failure(400, "词典上传无效,请使用指定的 WordNet 3.0 ZIP 和完整资源信息")
|
||||
c.Request.Body = http.MaxBytesReader(c.Writer, c.Request.Body, maxDictionaryZip+(64<<10))
|
||||
reader, err := c.Request.MultipartReader()
|
||||
if err != nil {
|
||||
return DictionaryResource{}, bad
|
||||
}
|
||||
fields := map[string]string{}
|
||||
var archive []byte
|
||||
for {
|
||||
part, err := reader.NextPart()
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return DictionaryResource{}, bad
|
||||
}
|
||||
name := part.FormName()
|
||||
if name == "file" {
|
||||
if archive != nil || part.FileName() == "" {
|
||||
part.Close()
|
||||
return DictionaryResource{}, bad
|
||||
}
|
||||
archive, err = io.ReadAll(io.LimitReader(part, maxDictionaryZip+1))
|
||||
if err != nil || len(archive) == 0 || len(archive) > maxDictionaryZip {
|
||||
part.Close()
|
||||
return DictionaryResource{}, bad
|
||||
}
|
||||
} else {
|
||||
if part.FileName() != "" || (name != "name" && name != "language" && name != "version" && name != "source" && name != "format") {
|
||||
part.Close()
|
||||
return DictionaryResource{}, bad
|
||||
}
|
||||
if _, exists := fields[name]; exists {
|
||||
part.Close()
|
||||
return DictionaryResource{}, bad
|
||||
}
|
||||
value, e := io.ReadAll(io.LimitReader(part, 1025))
|
||||
if e != nil || len(value) > 1024 || !utf8.Valid(value) {
|
||||
part.Close()
|
||||
return DictionaryResource{}, bad
|
||||
}
|
||||
fields[name] = string(value)
|
||||
}
|
||||
part.Close()
|
||||
}
|
||||
name := strings.TrimSpace(fields["name"])
|
||||
if name == "" || utf8.RuneCountInString(name) > 120 || strings.IndexFunc(name, unicode.IsControl) >= 0 || fields["language"] != "en" || fields["version"] != "3.0" || fields["format"] != "wordnet-3.0-zip" || fields["source"] != WordNetSource || len(archive) == 0 {
|
||||
return DictionaryResource{}, bad
|
||||
}
|
||||
return DictionaryResource{ID: 1, Name: name, Language: "en", Version: "3.0", Source: WordNetSource, Format: "wordnet-3.0-zip", SHA256: WordNetSHA, Enabled: true, Archive: archive}, nil
|
||||
}
|
||||
|
||||
func registerDictionaryRoutes(v *gin.RouterGroup, protect func(bool, func(*gin.Context, *gorm.DB, admin.SysUser) (any, error)) gin.HandlerFunc, now func() time.Time) {
|
||||
cache := &dictionaryCache{}
|
||||
uploadGate := make(chan struct{}, 1)
|
||||
v.GET("/dictionaries", protect(true, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) {
|
||||
if c.Request.URL.RawQuery != "" {
|
||||
return nil, failure(400, "词典列表不接受查询参数")
|
||||
}
|
||||
var rows []DictionaryResource
|
||||
if err := tx.Omit("archive").Order("id").Find(&rows).Error; err != nil {
|
||||
return nil, err
|
||||
}
|
||||
items := make([]DictionaryView, 0, len(rows))
|
||||
for _, row := range rows {
|
||||
status := "disabled"
|
||||
if row.Enabled {
|
||||
status = "ready"
|
||||
if _, err := cache.load(tx, row); err != nil {
|
||||
status = "unavailable"
|
||||
}
|
||||
}
|
||||
items = append(items, dictionaryView(row, status))
|
||||
}
|
||||
return gin.H{"items": items, "supported": gin.H{"name": "Princeton WordNet", "language": "en", "version": "3.0", "format": "wordnet-3.0-zip", "source": WordNetSource, "sha256": WordNetSHA}}, nil
|
||||
}))
|
||||
v.POST("/dictionaries/import", protect(true, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) {
|
||||
select {
|
||||
case uploadGate <- struct{}{}:
|
||||
defer func() { <-uploadGate }()
|
||||
default:
|
||||
return nil, failure(429, "已有词典正在导入,请稍后重试")
|
||||
}
|
||||
resource, err := readDictionaryUpload(c)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
engine, err := ParseWordNet(resource.Archive)
|
||||
if err != nil {
|
||||
return nil, failure(400, "词典文件校验失败,请选择指定的 WordNet 3.0 ZIP")
|
||||
}
|
||||
resource.EntryCount = engine.EntryCount
|
||||
resource.UpdatedAt = stamp(now())
|
||||
// INSERT ... ON CONFLICT followed by a locking read serializes even the first
|
||||
// concurrent import. A validated replacement and its metadata commit together.
|
||||
created := tx.Clauses(clause.OnConflict{DoNothing: true}).Create(&resource)
|
||||
if created.Error != nil {
|
||||
return nil, created.Error
|
||||
}
|
||||
var existing DictionaryResource
|
||||
if err = tx.Omit("archive").Clauses(clause.Locking{Strength: "UPDATE"}).Where("id = 1").First(&existing).Error; err != nil {
|
||||
return nil, err
|
||||
}
|
||||
duplicate := created.RowsAffected == 0 && existing.SHA256 == resource.SHA256
|
||||
if created.RowsAffected == 0 {
|
||||
if err = tx.Model(&DictionaryResource{}).Where("id = 1").Select("name", "language", "version", "source", "format", "sha256", "entry_count", "enabled", "archive", "updated_at").Updates(&resource).Error; err != nil {
|
||||
return nil, err
|
||||
}
|
||||
existing = resource
|
||||
}
|
||||
status := "ready"
|
||||
if !existing.Enabled {
|
||||
status = "disabled"
|
||||
}
|
||||
return DictionaryImportResult{dictionaryView(existing, status), duplicate}, nil
|
||||
}))
|
||||
v.PATCH("/dictionaries/:id", protect(true, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) {
|
||||
id, err := strconv.ParseInt(c.Param("id"), 10, 64)
|
||||
if err != nil || id != 1 {
|
||||
return nil, failure(404, "词典不存在")
|
||||
}
|
||||
var input struct {
|
||||
Enabled *bool `json:"enabled"`
|
||||
}
|
||||
if err := decode(c, &input); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if input.Enabled == nil {
|
||||
return nil, failure(400, "请指定词典启用状态")
|
||||
}
|
||||
var resource DictionaryResource
|
||||
if err = tx.Omit("archive").Clauses(clause.Locking{Strength: "UPDATE"}).Where("id = ?", id).First(&resource).Error; errors.Is(err, gorm.ErrRecordNotFound) {
|
||||
return nil, failure(404, "词典不存在")
|
||||
} else if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if *input.Enabled {
|
||||
if _, err = cache.load(tx, resource); err != nil {
|
||||
return nil, failure(409, "词典资源不可用,请重新导入")
|
||||
}
|
||||
}
|
||||
resource.Enabled = *input.Enabled
|
||||
resource.UpdatedAt = stamp(now())
|
||||
if err = tx.Model(&resource).Updates(map[string]any{"enabled": resource.Enabled, "updated_at": resource.UpdatedAt}).Error; err != nil {
|
||||
return nil, err
|
||||
}
|
||||
status := "disabled"
|
||||
if resource.Enabled {
|
||||
status = "ready"
|
||||
}
|
||||
return gin.H{"resource": dictionaryView(resource, status)}, nil
|
||||
}))
|
||||
v.GET("/chapters/:id/tokens", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) {
|
||||
if c.Request.URL.RawQuery != "" {
|
||||
return nil, failure(400, "分词接口不接受查询参数")
|
||||
}
|
||||
id, err := strconv.ParseInt(c.Param("id"), 10, 64)
|
||||
if err != nil || id <= 0 {
|
||||
return nil, failure(404, "章节不存在")
|
||||
}
|
||||
chapter, err := readyOwnedChapter(tx, u.UserId, id)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return ChapterTokens{chapter.ContentSHA256, Tokenize(chapter.OriginalText)}, nil
|
||||
}))
|
||||
v.POST("/lookup", protect(false, func(c *gin.Context, tx *gorm.DB, u admin.SysUser) (any, error) {
|
||||
var input struct {
|
||||
ChapterID int64 `json:"chapterId"`
|
||||
Start *int `json:"start"`
|
||||
End *int `json:"end"`
|
||||
}
|
||||
if err := decode(c, &input); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
chapter, err := readyOwnedChapter(tx, u.UserId, input.ChapterID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if input.Start == nil || input.End == nil {
|
||||
return nil, failure(400, "请选择完整单词")
|
||||
}
|
||||
query := ""
|
||||
for _, token := range Tokenize(chapter.OriginalText) {
|
||||
if token.Kind == "word" && token.Start == *input.Start && token.End == *input.End {
|
||||
query = token.Text
|
||||
break
|
||||
}
|
||||
}
|
||||
if query == "" || utf8.RuneCountInString(query) > 128 {
|
||||
return nil, failure(400, "请选择不超过 128 个字符的完整单词")
|
||||
}
|
||||
missing := LookupResult{Status: "resource_missing", Query: query, Candidates: []string{}, Entries: []DictionaryEntry{}}
|
||||
var resource DictionaryResource
|
||||
err = tx.Omit("archive").Where("id = 1 AND enabled = ?", true).First(&resource).Error
|
||||
if errors.Is(err, gorm.ErrRecordNotFound) {
|
||||
return missing, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
engine, err := cache.load(tx, resource)
|
||||
if err != nil {
|
||||
return missing, nil
|
||||
}
|
||||
result := engine.Lookup(query)
|
||||
result.Resource = &LookupResource{Name: resource.Name, Version: resource.Version}
|
||||
return result, nil
|
||||
}))
|
||||
}
|
||||
@@ -0,0 +1,182 @@
|
||||
package lexgo
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"mime/multipart"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/gin-gonic/gin"
|
||||
admin "go-admin/app/admin/models"
|
||||
)
|
||||
|
||||
func importDictionaryAPI(t *testing.T, r *gin.Engine, token string, raw []byte) (int, DictionaryImportResult) {
|
||||
t.Helper()
|
||||
var body bytes.Buffer
|
||||
w := multipart.NewWriter(&body)
|
||||
for k, v := range map[string]string{"name": "Princeton WordNet", "language": "en", "version": "3.0", "source": WordNetSource, "format": "wordnet-3.0-zip"} {
|
||||
if err := w.WriteField(k, v); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
p, err := w.CreateFormFile("file", "wordnet.zip")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
p.Write(raw)
|
||||
w.Close()
|
||||
req := httptest.NewRequest("POST", "/api/v1/dictionaries/import", &body)
|
||||
req.Header.Set("Content-Type", w.FormDataContentType())
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
response := httptest.NewRecorder()
|
||||
r.ServeHTTP(response, req)
|
||||
var e envelope
|
||||
json.Unmarshal(response.Body.Bytes(), &e)
|
||||
var result DictionaryImportResult
|
||||
json.Unmarshal(e.Data, &result)
|
||||
return response.Code, result
|
||||
}
|
||||
|
||||
func TestDictionaryAPIResourcesAndOwnership(t *testing.T) {
|
||||
db := testDB(t)
|
||||
raw, err := os.ReadFile("../../../.local/nlp-resources/wordnet.zip")
|
||||
if os.IsNotExist(err) {
|
||||
t.Skip("prepare WordNet fixture")
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Test-only dedicated database slot; no development/production resources touched.
|
||||
if err := db.Exec("DELETE FROM lexgo_dictionaries").Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { db.Exec("DELETE FROM lexgo_dictionaries") })
|
||||
r := Router(db, time.Now)
|
||||
users := []admin.SysUser{}
|
||||
tokens := []string{}
|
||||
for i, role := range []int{1, 2, 2} {
|
||||
u := admin.SysUser{Username: randomName(fmt.Sprintf("dict%d", i)), Password: fixturePassword, RoleId: role, Status: "2"}
|
||||
if err := db.Create(&u).Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
users = append(users, u)
|
||||
tokens = append(tokens, loginToken(t, r, u.Username, fixturePassword))
|
||||
}
|
||||
book := Book{OwnerID: users[1].UserId, Title: "Fictional", Language: "en", CreatedAt: stamp(time.Now()), UpdatedAt: stamp(time.Now())}
|
||||
if err := db.Create(&book).Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
original := "😀 Dogs went. Cafe\u0301 123"
|
||||
chapter := Chapter{BookID: book.ID, OwnerID: users[1].UserId, Ordinal: 1, Title: "Fictional", OriginalText: original, ContentSHA256: contentSHA(original), CharCount: len([]rune(original)), Status: statusReady, CreatedAt: stamp(time.Now()), UpdatedAt: stamp(time.Now())}
|
||||
if err := db.Create(&chapter).Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
lookup := func(token string, start, end int) (int, LookupResult) {
|
||||
code, data := callAPI(t, r, "POST", "/api/v1/lookup", token, map[string]any{"chapterId": chapter.ID, "start": start, "end": end})
|
||||
var got LookupResult
|
||||
json.Unmarshal(data, &got)
|
||||
return code, got
|
||||
}
|
||||
if code, _ := callAPI(t, r, "GET", "/api/v1/dictionaries", tokens[1], nil); code != 403 {
|
||||
t.Fatal("learner resource list", code)
|
||||
}
|
||||
if code, _ := importDictionaryAPI(t, r, tokens[1], []byte("bad")); code != 403 {
|
||||
t.Fatal("learner import", code)
|
||||
}
|
||||
if code, _ := lookup(tokens[2], 2, 6); code != 404 {
|
||||
t.Fatal("foreign lookup", code)
|
||||
}
|
||||
if code, _ := callAPI(t, r, "GET", fmt.Sprintf("/api/v1/chapters/%d/tokens", chapter.ID), tokens[2], nil); code != 404 {
|
||||
t.Fatal("foreign tokens", code)
|
||||
}
|
||||
if code, got := lookup(tokens[1], 2, 6); code != 200 || got.Status != "resource_missing" {
|
||||
t.Fatal("missing", code, got)
|
||||
}
|
||||
code, data := callAPI(t, r, "GET", fmt.Sprintf("/api/v1/chapters/%d/tokens", chapter.ID), tokens[1], nil)
|
||||
var analyzed ChapterTokens
|
||||
json.Unmarshal(data, &analyzed)
|
||||
if code != 200 || analyzed.TextSHA256 != contentSHA(original) || analyzed.Tokens[2].StartUtf16 != 3 {
|
||||
t.Fatal("tokens", code, string(data))
|
||||
}
|
||||
for _, span := range [][2]int{{-1, 3}, {2, 5}, {0, 1}, {2, 11}, {18, 21}, {1, 2}, {6, 2}} {
|
||||
if code, _ := lookup(tokens[1], span[0], span[1]); code != 400 {
|
||||
t.Fatal("invalid interval", span, code)
|
||||
}
|
||||
}
|
||||
code, imported := importDictionaryAPI(t, r, tokens[0], raw)
|
||||
if code != 200 || imported.Duplicate || imported.Resource.Status != "ready" {
|
||||
t.Fatal("import", code, imported)
|
||||
}
|
||||
if code, got := lookup(tokens[1], 7, 11); code != 200 || got.Status != "lemma" || *got.MatchedForm != "go" {
|
||||
t.Fatal("went", code, got)
|
||||
}
|
||||
code, duplicate := importDictionaryAPI(t, r, tokens[0], raw)
|
||||
if code != 200 || !duplicate.Duplicate || duplicate.Resource.ID != imported.Resource.ID {
|
||||
t.Fatal("duplicate", code, duplicate)
|
||||
}
|
||||
if code, _ := importDictionaryAPI(t, r, tokens[0], []byte("broken")); code != 400 {
|
||||
t.Fatal("bad import", code)
|
||||
}
|
||||
if code, got := lookup(tokens[1], 2, 6); code != 200 || got.Status != "lemma" {
|
||||
t.Fatal("failed import lost old resource", code, got)
|
||||
}
|
||||
endpoint := fmt.Sprintf("/api/v1/dictionaries/%d", imported.Resource.ID)
|
||||
if code, _ := callAPI(t, r, "PATCH", endpoint, tokens[1], map[string]bool{"enabled": false}); code != 403 {
|
||||
t.Fatal("learner toggle", code)
|
||||
}
|
||||
for _, enabled := range []bool{false, true} {
|
||||
if code, _ := callAPI(t, r, "PATCH", endpoint, tokens[0], map[string]bool{"enabled": enabled}); code != 200 {
|
||||
t.Fatal("toggle", code)
|
||||
}
|
||||
code, got := lookup(tokens[1], 2, 6)
|
||||
if code != 200 || (enabled && got.Status != "lemma") || (!enabled && got.Status != "resource_missing") {
|
||||
t.Fatal("enabled state", enabled, code, got)
|
||||
}
|
||||
}
|
||||
r = Router(db, time.Now) // A new router has an empty cache and reloads the persisted ZIP.
|
||||
if code, got := lookup(tokens[1], 2, 6); code != 200 || got.Status != "lemma" {
|
||||
t.Fatal("cold restart", code, got)
|
||||
}
|
||||
if code, _ := callAPI(t, r, "PATCH", endpoint, tokens[0], map[string]bool{"enabled": false}); code != 200 {
|
||||
t.Fatal("disable before duplicate", code)
|
||||
}
|
||||
if code, result := importDictionaryAPI(t, r, tokens[0], raw); code != 200 || !result.Duplicate || !result.Resource.Enabled {
|
||||
t.Fatal("duplicate must re-enable", code, result)
|
||||
}
|
||||
var count int64
|
||||
if err := db.Model(&DictionaryResource{}).Count(&count).Error; err != nil || count != 1 {
|
||||
t.Fatal("duplicate created extra resource", count, err)
|
||||
}
|
||||
if code, _ := callAPI(t, r, "PATCH", endpoint, tokens[0], map[string]any{"enabled": nil}); code != 400 {
|
||||
t.Fatal("null toggle", code)
|
||||
}
|
||||
if code, _ := callAPI(t, r, "POST", "/api/v1/lookup", tokens[1], map[string]any{"chapterId": chapter.ID, "start": 2, "end": 6, "ownerId": users[2].UserId}); code != 400 {
|
||||
t.Fatal("unknown lookup input", code)
|
||||
}
|
||||
if err := db.Model(&DictionaryResource{}).Where("id=1").Update("archive", []byte("corrupt fixture")).Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r = Router(db, time.Now)
|
||||
if code, got := lookup(tokens[1], 2, 6); code != 200 || got.Status != "resource_missing" {
|
||||
t.Fatal("corrupt cold resource", code, got)
|
||||
}
|
||||
if code, got := callAPI(t, r, "GET", "/api/v1/dictionaries", tokens[0], nil); code != 200 || !bytes.Contains(got, []byte(`"status":"unavailable"`)) {
|
||||
t.Fatal("corrupt resource state", code, string(got))
|
||||
}
|
||||
if code, result := importDictionaryAPI(t, r, tokens[0], raw); code != 200 || !result.Duplicate {
|
||||
t.Fatal("repair reimport", code, result)
|
||||
}
|
||||
if code, got := lookup(tokens[1], 2, 6); code != 200 || got.Status != "lemma" {
|
||||
t.Fatal("repair lookup", code, got)
|
||||
}
|
||||
if err := db.Model(&chapter).Update("status", statusPending).Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if code, _ := lookup(tokens[1], 2, 6); code != 409 {
|
||||
t.Fatal("pending lookup", code)
|
||||
}
|
||||
}
|
||||
@@ -136,8 +136,8 @@ func TestMigrationFromV2PreservesExistingData(t *testing.T) {
|
||||
}
|
||||
var version int
|
||||
db.Raw("SELECT version FROM lexgo_schema WHERE id=1").Scan(&version)
|
||||
if version != 3 {
|
||||
t.Fatalf("schema version %d after re-upgrade, want 3", version)
|
||||
if version != schemaVersion {
|
||||
t.Fatalf("schema version %d after re-upgrade, want %d", version, schemaVersion)
|
||||
}
|
||||
db.Table("sys_user").Where("user_id = 7").Count(&users)
|
||||
if users != 1 {
|
||||
@@ -149,7 +149,7 @@ func TestMigrationRefusesUnownedOrUnsupportedSchema(t *testing.T) {
|
||||
for _, tc := range []struct{ name, marker string }{
|
||||
{"empty_marker", ""},
|
||||
{"negative_version", "INSERT INTO lexgo_schema VALUES (1,-1,'lexgo')"},
|
||||
{"future_version", "INSERT INTO lexgo_schema VALUES (1,4,'lexgo')"},
|
||||
{"future_version", "INSERT INTO lexgo_schema VALUES (1,99,'lexgo')"},
|
||||
{"wrong_product", "INSERT INTO lexgo_schema VALUES (1,0,'another-app')"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
@@ -173,3 +173,53 @@ func TestMigrationRefusesUnownedOrUnsupportedSchema(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestMigrationFromV3PreservesLibraryAndJobs(t *testing.T) {
|
||||
db := emptyMigrationDB(t)
|
||||
statements := []string{"CREATE TABLE lexgo_schema (id INT PRIMARY KEY,version INT,product VARCHAR(32))", "INSERT INTO lexgo_schema VALUES (1,3,'lexgo')"}
|
||||
statements = append(statements, schemaV2Statements...)
|
||||
statements = append(statements, schemaV3Statements...)
|
||||
for _, statement := range statements {
|
||||
if err := db.Exec(statement).Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err := db.Exec("INSERT INTO sys_user (user_id,username,password,role_id) VALUES (8,'fixture_v3','fictional-not-a-real-hash',2)").Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
now := stamp(time.Now())
|
||||
book := Book{OwnerID: 8, Title: "Fictional migration", Language: "en", CreatedAt: now, UpdatedAt: now}
|
||||
if err := db.Create(&book).Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
original := "😀 Original e\u0301\r\n"
|
||||
chapter := Chapter{BookID: book.ID, OwnerID: 8, Ordinal: 1, Title: "Fixture", OriginalText: original, ContentSHA256: contentSHA(original), Status: statusReady, CharCount: len([]rune(original)), CreatedAt: now, UpdatedAt: now}
|
||||
if err := db.Create(&chapter).Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
job := IngestJob{OwnerID: 8, BookID: book.ID, ChapterID: chapter.ID, RequestKey: contentSHA("fixture-v3"), ContentSHA256: chapter.ContentSHA256, Status: statusReady, CreatedAt: now, UpdatedAt: now}
|
||||
if err := db.Create(&job).Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := CheckSchema(db); err == nil {
|
||||
t.Fatal("old schema accepted before explicit migration")
|
||||
}
|
||||
if err := Migrate(db); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := CheckSchema(db); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var restored Chapter
|
||||
if err := db.First(&restored, chapter.ID).Error; err != nil || restored.OriginalText != original || restored.ContentSHA256 != chapter.ContentSHA256 || restored.Status != statusReady {
|
||||
t.Fatal("migration changed original chapter", err)
|
||||
}
|
||||
var restoredJob IngestJob
|
||||
if err := db.First(&restoredJob, job.ID).Error; err != nil || restoredJob.RequestKey != job.RequestKey || restoredJob.Status != statusReady {
|
||||
t.Fatal("migration changed job", err)
|
||||
}
|
||||
var resources int64
|
||||
if err := db.Model(&DictionaryResource{}).Count(&resources).Error; err != nil || resources != 0 {
|
||||
t.Fatal("migration must create empty resource table", resources, err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -294,6 +294,7 @@ func Router(db *gorm.DB, now func() time.Time) *gin.Engine {
|
||||
}
|
||||
return gin.H{"job": job, "chapter": chapter}, nil
|
||||
}))
|
||||
registerDictionaryRoutes(v, protect, now)
|
||||
r.NoRoute(func(c *gin.Context) { respond(c, 404, nil, failure(404, "页面或接口不存在")) })
|
||||
return r
|
||||
}
|
||||
|
||||
@@ -0,0 +1,337 @@
|
||||
package lexgo
|
||||
|
||||
import (
|
||||
"archive/zip"
|
||||
"bufio"
|
||||
"bytes"
|
||||
"errors"
|
||||
"io"
|
||||
"path"
|
||||
"strconv"
|
||||
"strings"
|
||||
"unicode"
|
||||
|
||||
"golang.org/x/text/unicode/norm"
|
||||
)
|
||||
|
||||
const WordNetSHA = "cbda5ea6eef7f36a97a43d4a75f85e07fccbb4f23657d27b4ccbc93e2646ab59"
|
||||
const WordNetSource = "https://raw.githubusercontent.com/nltk/nltk_data/96f9b3252457a2b97e52aec64c3dfceeb5c312d5/packages/corpora/wordnet.zip"
|
||||
const maxDictionaryZip = 32 << 20
|
||||
const maxDictionaryInflated = 128 << 20
|
||||
|
||||
type TextToken struct {
|
||||
Text string `json:"text"`
|
||||
Start int `json:"start"`
|
||||
End int `json:"end"`
|
||||
StartUtf16 int `json:"startUtf16"`
|
||||
EndUtf16 int `json:"endUtf16"`
|
||||
Kind string `json:"kind"`
|
||||
}
|
||||
|
||||
// Tokenize never normalizes text. Letters begin words; marks continue them and
|
||||
// apostrophes join letters. Numbers and symbols are non-selectable punctuation.
|
||||
// All offsets are half-open, in code points and UTF-16 code units respectively.
|
||||
func Tokenize(text string) []TextToken {
|
||||
runes := []rune(text)
|
||||
result := make([]TextToken, 0)
|
||||
utf16Offset := 0
|
||||
for i := 0; i < len(runes); {
|
||||
start, start16 := i, utf16Offset
|
||||
kind := "punctuation"
|
||||
if unicode.IsLetter(runes[i]) {
|
||||
kind = "word"
|
||||
} else if unicode.IsSpace(runes[i]) {
|
||||
kind = "space"
|
||||
}
|
||||
i++
|
||||
for i < len(runes) {
|
||||
if kind == "word" && (unicode.IsLetter(runes[i]) || unicode.IsMark(runes[i]) || ((runes[i] == '\'' || runes[i] == '’') && i+1 < len(runes) && unicode.IsLetter(runes[i+1]))) {
|
||||
i++
|
||||
continue
|
||||
}
|
||||
if kind == "space" && unicode.IsSpace(runes[i]) {
|
||||
i++
|
||||
continue
|
||||
}
|
||||
break
|
||||
}
|
||||
for _, r := range runes[start:i] {
|
||||
utf16Offset++
|
||||
if r > 0xffff {
|
||||
utf16Offset++
|
||||
}
|
||||
}
|
||||
result = append(result, TextToken{string(runes[start:i]), start, i, start16, utf16Offset, kind})
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
func normalizeWord(word string) string {
|
||||
return norm.NFC.String(strings.ToLower(strings.ReplaceAll(word, "’", "'")))
|
||||
}
|
||||
|
||||
type DictionaryEntry struct {
|
||||
Lemma string `json:"lemma"`
|
||||
POS string `json:"pos"`
|
||||
Definition string `json:"definition"`
|
||||
Examples []string `json:"examples"`
|
||||
}
|
||||
type LookupResult struct {
|
||||
Status string `json:"status"`
|
||||
Query string `json:"query"`
|
||||
MatchedForm *string `json:"matchedForm"`
|
||||
Candidates []string `json:"candidates"`
|
||||
Entries []DictionaryEntry `json:"entries"`
|
||||
Resource *LookupResource `json:"resource,omitempty"`
|
||||
}
|
||||
type LookupResource struct {
|
||||
Name string `json:"name"`
|
||||
Version string `json:"version"`
|
||||
}
|
||||
type wordNetPOS struct {
|
||||
pos string
|
||||
index map[string][]int
|
||||
data map[int]DictionaryEntry
|
||||
exceptions map[string][]string
|
||||
}
|
||||
type WordNet struct {
|
||||
parts []wordNetPOS
|
||||
EntryCount int
|
||||
}
|
||||
|
||||
// ParseWordNet accepts only the pinned official corpus. No archive member is ever
|
||||
// extracted to disk; both declared and actual inflated sizes are bounded.
|
||||
func ParseWordNet(raw []byte) (*WordNet, error) {
|
||||
if len(raw) > maxDictionaryZip || contentSHA(string(raw)) != WordNetSHA {
|
||||
return nil, errors.New("WordNet ZIP checksum mismatch")
|
||||
}
|
||||
return parseWordNetArchive(raw)
|
||||
}
|
||||
|
||||
func readWordNetArchive(raw []byte) (map[string][]byte, error) {
|
||||
if len(raw) > maxDictionaryZip {
|
||||
return nil, errors.New("archive too large")
|
||||
}
|
||||
reader, err := zip.NewReader(bytes.NewReader(raw), int64(len(raw)))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(reader.File) > 64 {
|
||||
return nil, errors.New("too many archive members")
|
||||
}
|
||||
files := map[string][]byte{}
|
||||
seen := map[string]bool{}
|
||||
total := int64(0)
|
||||
for _, f := range reader.File {
|
||||
name := f.Name
|
||||
if strings.Contains(name, "\\") || path.IsAbs(name) || strings.Contains(name, ":") || path.Clean(name) != strings.TrimSuffix(name, "/") || !strings.HasPrefix(name, "wordnet/") || seen[name] {
|
||||
return nil, errors.New("unsafe archive member")
|
||||
}
|
||||
seen[name] = true
|
||||
if !f.FileInfo().IsDir() && !f.Mode().IsRegular() {
|
||||
return nil, errors.New("unsupported archive member")
|
||||
}
|
||||
if f.FileInfo().IsDir() {
|
||||
continue
|
||||
}
|
||||
if f.UncompressedSize64 > maxDictionaryInflated || total+int64(f.UncompressedSize64) > maxDictionaryInflated {
|
||||
return nil, errors.New("inflated archive too large")
|
||||
}
|
||||
rc, err := f.Open()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
data, err := io.ReadAll(io.LimitReader(rc, maxDictionaryInflated-total+1))
|
||||
rc.Close()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
total += int64(len(data))
|
||||
if total > maxDictionaryInflated {
|
||||
return nil, errors.New("inflated archive too large")
|
||||
}
|
||||
files[strings.TrimPrefix(name, "wordnet/")] = data
|
||||
}
|
||||
for _, name := range []string{"LICENSE", "README", "lexnames", "index.noun", "index.verb", "index.adj", "index.adv", "data.noun", "data.verb", "data.adj", "data.adv", "noun.exc", "verb.exc", "adj.exc", "adv.exc"} {
|
||||
if len(files[name]) == 0 {
|
||||
return nil, errors.New("missing required WordNet member")
|
||||
}
|
||||
}
|
||||
return files, nil
|
||||
}
|
||||
|
||||
func parseWordNetArchive(raw []byte) (*WordNet, error) {
|
||||
files, err := readWordNetArchive(raw)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
result := &WordNet{}
|
||||
for i, suffix := range []string{"noun", "verb", "adj", "adv"} {
|
||||
part := wordNetPOS{pos: []string{"n", "v", "a", "r"}[i], index: map[string][]int{}, data: map[int]DictionaryEntry{}, exceptions: map[string][]string{}}
|
||||
scanner := bufio.NewScanner(bytes.NewReader(files["data."+suffix]))
|
||||
scanner.Buffer(make([]byte, 4096), 1<<20)
|
||||
for scanner.Scan() {
|
||||
line := scanner.Text()
|
||||
if line == "" || line[0] == ' ' {
|
||||
continue
|
||||
}
|
||||
header, gloss, ok := strings.Cut(line, "|")
|
||||
fields := strings.Fields(header)
|
||||
if !ok || len(fields) < 6 {
|
||||
return nil, errors.New("invalid WordNet data record")
|
||||
}
|
||||
offset, e := strconv.Atoi(fields[0])
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
if _, exists := part.data[offset]; exists {
|
||||
return nil, errors.New("duplicate synset offset")
|
||||
}
|
||||
definition, examples := wordNetGloss(gloss)
|
||||
part.data[offset] = DictionaryEntry{POS: fields[2], Definition: definition, Examples: examples}
|
||||
}
|
||||
if err := scanner.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
scanner = bufio.NewScanner(bytes.NewReader(files["index."+suffix]))
|
||||
scanner.Buffer(make([]byte, 4096), 1<<20)
|
||||
for scanner.Scan() {
|
||||
line := scanner.Text()
|
||||
if line == "" || line[0] == ' ' {
|
||||
continue
|
||||
}
|
||||
fields := strings.Fields(line)
|
||||
if len(fields) < 7 {
|
||||
return nil, errors.New("invalid WordNet index record")
|
||||
}
|
||||
count, e := strconv.Atoi(fields[2])
|
||||
if e != nil || count < 1 || count > 1000 {
|
||||
return nil, errors.New("invalid sense count")
|
||||
}
|
||||
pointers, e := strconv.Atoi(fields[3])
|
||||
if e != nil || pointers < 0 || len(fields) != 6+pointers+count {
|
||||
return nil, errors.New("invalid index offsets")
|
||||
}
|
||||
offsets := make([]int, 0, count)
|
||||
for _, rawOffset := range fields[6+pointers:] {
|
||||
offset, e := strconv.Atoi(rawOffset)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
if _, ok := part.data[offset]; !ok {
|
||||
return nil, errors.New("missing synset")
|
||||
}
|
||||
offsets = append(offsets, offset)
|
||||
}
|
||||
if _, exists := part.index[fields[0]]; exists {
|
||||
return nil, errors.New("duplicate index key")
|
||||
}
|
||||
part.index[fields[0]] = offsets
|
||||
}
|
||||
if err := scanner.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
scanner = bufio.NewScanner(bytes.NewReader(files[suffix+".exc"]))
|
||||
for scanner.Scan() {
|
||||
fields := strings.Fields(scanner.Text())
|
||||
if len(fields) < 2 {
|
||||
return nil, errors.New("invalid exception record")
|
||||
}
|
||||
part.exceptions[fields[0]] = fields[1:]
|
||||
}
|
||||
if err := scanner.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
result.EntryCount += len(part.index)
|
||||
result.parts = append(result.parts, part)
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func wordNetGloss(gloss string) (string, []string) {
|
||||
examples := []string{}
|
||||
definition := strings.Builder{}
|
||||
for {
|
||||
before, after, ok := strings.Cut(gloss, "\"")
|
||||
definition.WriteString(before)
|
||||
if !ok {
|
||||
break
|
||||
}
|
||||
example, rest, closed := strings.Cut(after, "\"")
|
||||
if !closed {
|
||||
definition.WriteString(after)
|
||||
break
|
||||
}
|
||||
examples = append(examples, example)
|
||||
gloss = rest
|
||||
}
|
||||
return strings.Trim(strings.TrimSpace(definition.String()), "; "), examples
|
||||
}
|
||||
|
||||
var morphyRules = map[string][][2]string{
|
||||
"n": {{"s", ""}, {"ses", "s"}, {"ves", "f"}, {"xes", "x"}, {"zes", "z"}, {"ches", "ch"}, {"shes", "sh"}, {"men", "man"}, {"ies", "y"}},
|
||||
"v": {{"s", ""}, {"ies", "y"}, {"es", "e"}, {"es", ""}, {"ed", "e"}, {"ed", ""}, {"ing", "e"}, {"ing", ""}},
|
||||
"a": {{"er", ""}, {"est", ""}, {"er", "e"}, {"est", "e"}},
|
||||
}
|
||||
|
||||
// Morphy yields possible dictionary forms, never contextual POS or a merged
|
||||
// learning identity. Exact forms win before exceptions and suffix detachment.
|
||||
func (wn *WordNet) Lookup(query string) LookupResult {
|
||||
result := LookupResult{Status: "not_found", Query: query, Candidates: []string{}, Entries: []DictionaryEntry{}}
|
||||
key := normalizeWord(query)
|
||||
add := func(form string, part wordNetPOS) {
|
||||
if len(part.index[form]) == 0 {
|
||||
return
|
||||
}
|
||||
found := false
|
||||
for _, candidate := range result.Candidates {
|
||||
if candidate == form {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
result.Candidates = append(result.Candidates, form)
|
||||
}
|
||||
for _, offset := range part.index[form] {
|
||||
if len(result.Entries) >= 12 {
|
||||
break
|
||||
}
|
||||
entry := part.data[offset]
|
||||
entry.Lemma = strings.ReplaceAll(form, "_", " ")
|
||||
result.Entries = append(result.Entries, entry)
|
||||
}
|
||||
}
|
||||
for _, part := range wn.parts {
|
||||
add(key, part)
|
||||
}
|
||||
if len(result.Entries) > 0 {
|
||||
result.Status = "exact"
|
||||
} else {
|
||||
for _, part := range wn.parts {
|
||||
forms := part.exceptions[key]
|
||||
if len(forms) == 0 {
|
||||
for _, rule := range morphyRules[part.pos] {
|
||||
if strings.HasSuffix(key, rule[0]) && len(key) > len(rule[0]) {
|
||||
forms = append(forms, strings.TrimSuffix(key, rule[0])+rule[1])
|
||||
}
|
||||
}
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for _, form := range forms {
|
||||
if !seen[form] {
|
||||
add(form, part)
|
||||
seen[form] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(result.Entries) > 0 {
|
||||
result.Status = "lemma"
|
||||
}
|
||||
}
|
||||
if len(result.Candidates) > 0 {
|
||||
form := result.Candidates[0]
|
||||
result.MatchedForm = &form
|
||||
}
|
||||
return result
|
||||
}
|
||||
@@ -0,0 +1,112 @@
|
||||
package lexgo
|
||||
|
||||
import (
|
||||
"archive/zip"
|
||||
"bytes"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"unicode/utf16"
|
||||
)
|
||||
|
||||
func TestArchiveRejectsUnsafeMembersAndMissingLicense(t *testing.T) {
|
||||
for _, name := range []string{"../data.noun", "wordnet/../data.noun", "/wordnet/data.noun", "wordnet\\data.noun", "wordnet/C:data.noun", "wordnet/LICENSE"} {
|
||||
var raw bytes.Buffer
|
||||
w := zip.NewWriter(&raw)
|
||||
entry, err := w.Create(name)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
entry.Write([]byte("fictional fixture"))
|
||||
w.Close()
|
||||
if _, err := readWordNetArchive(raw.Bytes()); err == nil {
|
||||
t.Fatalf("accepted unsafe/incomplete archive %s", name)
|
||||
}
|
||||
}
|
||||
var raw bytes.Buffer
|
||||
w := zip.NewWriter(&raw)
|
||||
for i := 0; i < 2; i++ {
|
||||
entry, _ := w.Create("wordnet/LICENSE")
|
||||
entry.Write([]byte("fixture"))
|
||||
}
|
||||
w.Close()
|
||||
if _, err := readWordNetArchive(raw.Bytes()); err == nil {
|
||||
t.Fatal("accepted duplicate member")
|
||||
}
|
||||
if _, err := ParseWordNet(make([]byte, maxDictionaryZip+1)); err == nil {
|
||||
t.Fatal("accepted oversized archive")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMorphyCandidatesAndExactPrecedence(t *testing.T) {
|
||||
wn := &WordNet{parts: []wordNetPOS{{pos: "v", index: map[string][]int{"axes": {1}, "ax": {1}, "axe": {2}}, data: map[int]DictionaryEntry{1: {POS: "v", Definition: "fixture one", Examples: []string{}}, 2: {POS: "v", Definition: "fixture two", Examples: []string{}}}, exceptions: map[string][]string{}}}}
|
||||
if got := wn.Lookup("AXES"); got.Status != "exact" || len(got.Candidates) != 1 || *got.MatchedForm != "axes" {
|
||||
t.Fatal("exact must precede lemma", got)
|
||||
}
|
||||
delete(wn.parts[0].index, "axes")
|
||||
got := wn.Lookup("axes")
|
||||
if got.Status != "lemma" || strings.Join(got.Candidates, ",") != "axe,ax" {
|
||||
t.Fatal("deterministic candidate forms", got)
|
||||
}
|
||||
definition, examples := wordNetGloss(`a fictional gloss; "one example"; "second example"`)
|
||||
if definition != "a fictional gloss" || len(examples) != 2 {
|
||||
t.Fatal("gloss", definition, examples)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTokensPreserveUnicodeOriginal(t *testing.T) {
|
||||
text := "😀 Cafe\u0301 can’t\r\nDogs 123 中文!"
|
||||
tokens := Tokenize(text)
|
||||
runes := []rune(text)
|
||||
cursor, u16 := 0, 0
|
||||
words := []string{}
|
||||
for _, token := range tokens {
|
||||
if token.Start != cursor || token.StartUtf16 != u16 || string(runes[token.Start:token.End]) != token.Text {
|
||||
t.Fatalf("incorrect token %#v", token)
|
||||
}
|
||||
cursor = token.End
|
||||
u16 += len(utf16.Encode([]rune(token.Text)))
|
||||
if token.EndUtf16 != u16 {
|
||||
t.Fatal("incorrect UTF-16 end")
|
||||
}
|
||||
if token.Kind == "word" {
|
||||
words = append(words, token.Text)
|
||||
}
|
||||
}
|
||||
if cursor != len(runes) || strings.Join(words, "|") != "Cafe\u0301|can’t|Dogs|中文" {
|
||||
t.Fatalf("coverage/words %v", words)
|
||||
}
|
||||
if normalizeWord("CAFE\u0301") != "café" || normalizeWord("CAN’T") != "can't" {
|
||||
t.Fatal("normalization")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOfficialWordNet(t *testing.T) {
|
||||
raw, err := os.ReadFile("../../../.local/nlp-resources/wordnet.zip")
|
||||
if os.IsNotExist(err) {
|
||||
t.Skip("explicitly prepared official WordNet fixture unavailable")
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
engine, err := ParseWordNet(raw)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if engine.EntryCount < 150000 {
|
||||
t.Fatalf("entry count %d", engine.EntryCount)
|
||||
}
|
||||
for _, tc := range []struct{ query, status, lemma string }{{"dog", "exact", "dog"}, {"went", "lemma", "go"}, {"mice", "lemma", "mouse"}, {"zzzznonword", "not_found", ""}} {
|
||||
got := engine.Lookup(tc.query)
|
||||
if got.Status != tc.status || (tc.lemma != "" && (got.MatchedForm == nil || *got.MatchedForm != tc.lemma)) {
|
||||
t.Fatalf("%s: %#v", tc.query, got)
|
||||
}
|
||||
if tc.lemma != "" && (len(got.Entries) == 0 || got.Entries[0].Definition == "") {
|
||||
t.Fatal("missing definition")
|
||||
}
|
||||
}
|
||||
raw[100] ^= 1
|
||||
if _, err := ParseWordNet(raw); err == nil {
|
||||
t.Fatal("accepted corrupt checksum")
|
||||
}
|
||||
}
|
||||
@@ -82,7 +82,7 @@ func run() error {
|
||||
if err = lexgo.Migrate(db); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Println("LexGo schema version 3 ready")
|
||||
fmt.Println("LexGo schema version 4 ready")
|
||||
return nil
|
||||
}
|
||||
if err = lexgo.CheckSchema(db); err != nil {
|
||||
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"name": "Princeton WordNet",
|
||||
"language": "en",
|
||||
"version": "3.0",
|
||||
"format": "wordnet-3.0-zip",
|
||||
"source": "https://raw.githubusercontent.com/nltk/nltk_data/96f9b3252457a2b97e52aec64c3dfceeb5c312d5/packages/corpora/wordnet.zip",
|
||||
"sha256": "cbda5ea6eef7f36a97a43d4a75f85e07fccbb4f23657d27b4ccbc93e2646ab59",
|
||||
"license": "WORDNET-LICENSE.txt",
|
||||
"licenseUrl": "https://wordnet.princeton.edu/license-and-commercial-use",
|
||||
"documentation": [
|
||||
"https://wordnet.princeton.edu/documentation/wndb5wn",
|
||||
"https://wordnet.princeton.edu/documentation/morphy7wn"
|
||||
],
|
||||
"runtime": "Go only; explicitly imported ZIP persisted in MySQL; no runtime download or Python process"
|
||||
}
|
||||
Reference in New Issue
Block a user