From bdb38970e0122db3a7f085079f8bfcde87302710 Mon Sep 17 00:00:00 2001 From: QiuSW Date: Tue, 15 Sep 2026 16:27:36 +0800 Subject: [PATCH 1/4] =?UTF-8?q?docs:=20=E6=96=B0=E5=A2=9E=E9=83=A8?= =?UTF-8?q?=E7=BD=B2=E4=B8=8E=E8=BF=90=E7=BB=B4=E9=A1=B5=E9=9D=A2=E6=98=A0?= =?UTF-8?q?=E5=B0=84=20(#15)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 从 docs/templates/deployment.md 建立 Deployment-and-Operations 页面映射，由 harness.py sync 导出核心镜像。 --- docs/11-deployment-and-operations.md | 197 +++++++++++++++++++++++++++ wiki-docs.json | 4 + 2 files changed, 201 insertions(+) create mode 100644 docs/11-deployment-and-operations.md diff --git a/docs/11-deployment-and-operations.md b/docs/11-deployment-and-operations.md new file mode 100644 index 0000000..ba2af3c --- /dev/null +++ b/docs/11-deployment-and-operations.md @@ -0,0 +1,197 @@ + +generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件) +wiki_page: Deployment-and-Operations +wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Deployment-and-Operations.- +wiki_revision: 2b08a33c0c246841deb52450d7fa411d76dc2eee +synchronized_at: 2026-09-15T08:25:07Z + + +# 部署与运维 + +本页面向 LexGo 的内部维护者：在一台干净的机器上首次部署、日常运维、健康检查、升级回滚与备份恢复。所有命令以仓库根为工作目录。**本页只写配置项名称与来源，不写任何真实密码、令牌或生产数据库地址。** + +## 本页用途 + +让维护者在一台新机器上完成首次部署，并在需要时把实例从备份恢复回来，同时知道哪些步骤不可逆、哪些证据必须留下。 + +## 安全边界 + +- 凭据只从运维密码库或 `.env.local`（已被 `.gitignore` 忽略）读取；`.env.local` 权限应为仅本人可读写。 +- 备份文件含全部用户数据（账号、书籍原文、词条、复习记录、审计日志、词典归档），必须按个人数据对待：存放位置受控、不进入 Git、不进入工单附件。 +- `restore` 会写入数据库：默认只写空库，覆盖已有库必须显式 `--force`，任何情况下都必须 `--confirm`。 +- 服务默认只监听 `127.0.0.1`；对外提供访问时经反向代理，不要直接暴露应用端口。 + +## 服务概览 + +| 项目 | 内容 | +|---|---| +| 组件 | Go 后端（`server/`）、学习端 SPA（`learner/`）、管理端 SPA（`admin/`）、MySQL 8 | +| 后端监听 | `LEXGO_LISTEN`，默认 `127.0.0.1:8000` | +| 前端托管 | 反向代理（nginx）托管已构建的 `dist`，同一域名下把 `/api/` 转发到后端 | +| 数据库 | MySQL 8（本机验证版本 8.4.3）；库名经 `LEXGO_DB_NAME` 指定 | +| 运行时 | 后端为单个静态二进制，无需运行时依赖；构建需要 Go 1.26.5、Node 22、pnpm 9 | +| 进程托管 | supervisor（本项目开发机即为 `lexgo-api` / `lexgo-learner` / `lexgo-admin` 三个 program） | +| 日志 | supervisor 的 `stdout`/`stderr` 日志文件；应用自身不写文件日志 | +| 备份对象 | MySQL 全库（用户数据、原文、词典归档、审计）+ `.env.local`（单独从密码库取） | + +## 环境要求 + +| 组件 | 版本要求 | 检查命令 | 预期结果 | +|---|---|---|---| +| 操作系统 | Windows 或 Linux | — | 本项目在 Windows 开发机验证；Linux 步骤为等价命令 | +| MySQL 服务端 | 8.x | `mysql --version`（服务端 `SELECT VERSION()`） | 8.4.3 已验证 | +| MySQL 客户端 | **不低于服务端** | `python scripts/ops.py install-check` | 「客户端版本不低于服务端」为 ok；5.7 客户端连 8.4 服务端会被判 fail | +| Go | 1.26.5 | `go version` | 供 `scripts/server.py` 固定工具链构建 | +| Node / pnpm | Node 22、pnpm 9 | `node --version`、`pnpm --version` | 仅在需要构建前端时要求 | +| WordNet 资源 | `server/wordnet-resource.json` 固定的 ZIP | `python scripts/ops.py install-check` | 「WordNet 资源 pin」为 ok；资源只在显式导入时使用，不在运行时下载 | + +## 首次部署 + +### 1. 取得代码并检查依赖 + +```bash +git clone <仓库地址> <部署目录> +cd <部署目录> +git rev-parse HEAD # 记录本次部署的提交哈希 +python scripts/ops.py install-check +``` + +预期结果：提交哈希被记录到部署记录；`install-check` 全部 `[ok]`，否则按其提示补齐后再继续。 + +### 2. 创建数据库与配置 + +```bash +python scripts/ops.py init-database --database lexgo_prod +``` + +预期结果：输出「已创建库」并打印需要授予的最小权限（`SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, INDEX, DROP, REFERENCES`，仅限该库）。不要用管理员账号运行应用。 + +把 `.env.local`（从 `.env.example` 复制）填好：`LEXGO_DB_HOST/PORT/NAME/USER/PASSWORD`、`LEXGO_BOOTSTRAP_USERNAME/PASSWORD`、`LEXGO_LISTEN`。凭据值取自运维密码库，不进 Git、不进日志。 + +### 3. 迁移与初始化管理员 + +```bash +python scripts/server.py migrate # 只有这一步会改表结构 +python scripts/server.py bootstrap # 首次建立唯一管理员 +``` + +预期结果：`migrate` 输出 schema 版本；`bootstrap` 输出创建成功。**库中已有账号时 `bootstrap` 会拒绝执行**，不会覆盖既有管理员。空库不含默认密码与任何演示数据。 + +### 4. 构建产物 + +```bash +python scripts/server.py build # 产出 server/lexgo(.exe) +cd learner && pnpm install && pnpm run build +cd ../admin && pnpm install && pnpm run build +``` + +预期结果：后端二进制与两份 `dist` 生成。前端由反向代理托管；反向代理需把未知路径回落到 `index.html`（SPA 路由），并把 `/api/` 转发到后端。 + +### 5. 词典资源 + +词典归档在显式导入后存于 MySQL，备份与恢复会一并带走，运行时不下载。 + +### 6. 启动与健康检查 + +按 supervisor 配置启动三个 program（后端 + 两个静态站点或由反向代理托管）。然后执行下方「健康检查」全部命令。 + +## 配置与凭据来源 + +| 配置项 | 用途 | 来源 | 是否敏感 | +|---|---|---|---| +| `LEXGO_DB_HOST/PORT/NAME/USER/PASSWORD` | 数据库连接 | `.env.local`，值取自运维密码库 | 是（密码） | +| `LEXGO_BOOTSTRAP_USERNAME/PASSWORD` | 首次建立管理员 | 同上 | 是（密码） | +| `LEXGO_LISTEN` | 后端监听地址 | `.env.local` | 否 | +| `LEXGO_MYSQL_BIN` | 指定 MySQL 客户端目录 | 运维环境变量 | 否 | +| `LEXGO_TEST_DB_NAME` | 集成测试库（仅开发） | 环境变量 | 否 | + +`scripts/ops.py` 的 manifest 与日志只记录配置项**名称**、库名、schema 版本与提交哈希，不记录任何凭据值。 + +## 日常运维 + +| 操作 | 命令 | 预期结果 | +|---|---|---| +| 查看状态 | `supervisorctl status lexgo-api` | `RUNNING`，uptime 持续增长 | +| 重启后端 | `supervisorctl restart lexgo-api` | `stopped` 后 `started` | +| 查看日志 | `supervisorctl tail -f lexgo-api stderr` | 持续输出应用日志 | +| 备份 | `python scripts/ops.py backup --out <目录>` | 生成 `lexgo-<时间戳>.sql.gz` 与 `manifest.json` | +| 校验实例 | `python scripts/ops.py verify --database <库名>` | 打印每项检查结果，最后「校验通过」 | +| 清理过期审计 | `python scripts/server.py audit-cleanup` | 只清理两张审计表中超过 90 天的记录 | +| 关闭确认 | `supervisorctl stop lexgo-api` | 服务停止；学习端与反代仍在，接口不可用 | + +## 健康检查 + +```bash +supervisorctl status lexgo-api +curl -sS -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8000/healthz +python scripts/ops.py verify --database <库名> +``` + +预期结果：状态 `RUNNING`；`curl` 返回 `200`；`verify` 全部 `[ok]`。任何一项不符合时不视为部署成功，按「升级与回滚」处理。 + +## 升级与回滚 + +### 升级 + +```bash +git rev-parse HEAD # 记录升级前提交，回滚需要 +python scripts/ops.py backup --out <目录> # 有迁移时必须先备份 +git fetch --all && git checkout <目标提交或标签> +python scripts/server.py build +cd learner && pnpm install && pnpm run build && cd ../admin && pnpm install && pnpm run build +python scripts/server.py migrate # 只有迁移单需要 +supervisorctl restart lexgo-api +``` + +预期结果：重启后状态 `RUNNING`，健康检查全部通过。前端 `dist` 更新后由反代直接生效，无需重启后端。 + +### 回滚 + +```bash +git checkout <升级前记录的提交> +python scripts/server.py build +supervisorctl restart lexgo-api +``` + +预期结果：健康检查全部通过。 + +> **注意**：已执行的迁移通常不能用切回代码撤销。LexGo 的用法是：每个版本的迁移只新增对象（`CREATE TABLE IF NOT EXISTS`），旧二进制认版本号，因此回退到旧二进制前要把 `lexgo_schema.version` 写回旧版本号；若迁移删改了数据或旧二进制无法读写新结构，则按「备份与恢复」处理。**本节只描述规则，真实回滚尚未演练过，属于已知限制。** + +## 备份与恢复 + +### 备份对象与频率 + +- **数据库**：一个实例的全部持久数据（账号与会话、书籍与章节原文、导入任务、词典归档、词条、复习排期与作答、阅读进度、审计日志）。 +- **环境配置**：`.env.local`（含凭据）单独从运维密码库保存，**不放进备份目录**。 +- **代码与二进制**：由 Git 提交哈希重建，manifest 中记录了该哈希。 +- 建议频率：每次升级或有迁移前必须备份；日常按使用强度自行决定（本项目未启用定时任务）。保留份数与存放位置由运维决定。 + +```bash +python scripts/ops.py backup --out <备份目录> +``` + +预期结果：目录中出现 `lexgo-<时间戳>.sql.gz` 与 `manifest.json`（schema 版本、提交哈希、逐表行数、dump 的 sha256、客户端与服务端版本）。manifest 不含凭据。 + +### 恢复步骤 + +```bash +python scripts/ops.py restore --dump <备份目录>/lexgo-<时间戳>.sql.gz --database <新库名> --confirm +python scripts/ops.py verify --database <新库名> --manifest <备份目录>/manifest.json \ + --api http://127.0.0.1:<演练端口> --user <账号前缀> --password-env <密码环境变量名> +``` + +预期结果：恢复写入空库、自动校验通过；`verify` 打印完整性与两账号隔离检查，最后「校验通过」。恢复**不会**写入备份里的源库，工具会在恢复前后比对源库的内容校验和（本机演练已验证）。 + +> **注意**：`restore` 默认拒绝写入已有数据的库；覆盖必须 `--force`，执行前先备份当前库。库名必须包含 `lexgo` 且不能是 MySQL 系统库。 + +### 恢复演练记录 + +2026-09-15 在本机完成完整演练：空库安装（init-database → migrate → bootstrap → 管理员建两个演练账号 → 走通粘贴/阅读/查词/保存/复习/完成章节/进度）与「备份 lexgo_dev → 恢复到空库 → 起第二个 API 实例 → 两账号闭环与越权校验」，恢复后逐表行数与内容校验和都和源库一致，演练库用后删除。详细命令与结果见工单 #15 与本地开发页。 + +## 已知限制 + +- **真实回滚演练未做**：本文档给出规则，但没有在真实实例上执行过「升级 → 回滚」全过程。 +- 未验证 HTTPS、域名、多机与灰度部署；本机演练只用 `127.0.0.1` 与模拟触摸视口。 +- 未启用定时备份、监控与告警；备份由人工触发。 +- **附件（封面/音频）尚未实现（工单 #21）**：本页的备份恢复范围目前只覆盖数据库；#21 落地后必须把附件存储纳入备份与恢复验收。 +- 恢复演练是单机顺序执行，未验证大库恢复耗时与磁盘空间上限。 diff --git a/wiki-docs.json b/wiki-docs.json index 450f275..4496343 100644 --- a/wiki-docs.json +++ b/wiki-docs.json @@ -83,6 +83,10 @@ { "page": "Go-Analysis-Alternative", "path": "docs/linguacafe-go-analysis.md" + }, + { + "page": "Deployment-and-Operations", + "path": "docs/11-deployment-and-operations.md" } ] } -- 2.34.1 From 38d897532572668679cb2280a64c90706dba7916 Mon Sep 17 00:00:00 2001 From: QiuSW Date: Tue, 15 Sep 2026 16:30:23 +0800 Subject: [PATCH 2/4] =?UTF-8?q?feat:=20=E8=87=AA=E6=89=98=E7=AE=A1?= =?UTF-8?q?=E4=BA=A4=E4=BB=98=E7=9A=84=E5=AE=89=E8=A3=85=E3=80=81=E5=A4=87?= =?UTF-8?q?=E4=BB=BD=E4=B8=8E=E6=81=A2=E5=A4=8D=E5=B7=A5=E5=85=B7=E5=8F=8A?= =?UTF-8?q?=E9=83=A8=E7=BD=B2=E6=96=87=E6=A1=A3=20(#15)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - scripts/ops.py：install-check（依赖与资源版本、客户端不得低于服务端）、init-database（建空库 并提示最小权限）、backup（全库 dump ＋ manifest，不含凭据）、restore（默认只写空库、 --confirm 必需、覆盖需 --force、拒绝系统库与带 CREATE DATABASE/USE 的 dump、比对源库校验和）、 verify（完整性 ＋ 可选两账号接口闭环）、smoke（干净实例上建两个演练账号走通学习闭环） - scripts/bench.py：写明规模的人造数据集性能测量，记录数据量、机器与 p50/p95 - tests/test_lexgo_ops.py：版本解析、库名白名单、dump 安全性、manifest 字段白名单等无库测试 - Wiki 新增 Deployment-and-Operations 页面与 wiki-docs.json 映射（含升级回滚与已知限制） - 更新 Architecture、Business-Rules、Local-Development、Product-Requirements、Home 与 README/AGENTS；本单无 schema 与接口变化 --- AGENTS.md | 5 + README.md | 4 +- docs/02-architecture-and-code-map.md | 22 +- docs/03-business-rules-and-glossary.md | 18 +- docs/04-local-development-and-verification.md | 71 +- docs/09-product-requirements-overview.md | 10 +- docs/README.md | 6 +- scripts/bench.py | 207 +++++ scripts/ops.py | 783 ++++++++++++++++++ tests/test_lexgo_ops.py | 90 ++ 10 files changed, 1204 insertions(+), 12 deletions(-) create mode 100644 scripts/bench.py create mode 100644 scripts/ops.py create mode 100644 tests/test_lexgo_ops.py diff --git a/AGENTS.md b/AGENTS.md index 7f7f419..577eb2a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -27,6 +27,10 @@ | 创建可选任务快照 | `python dev_scripts/harness.py archive 123 "修复登录超时"` | | 增量导出已有快照 | `python dev_scripts/harness.py export` | | 全量导出已有快照 | `python dev_scripts/harness.py export --all` | +| 检查部署依赖 | `python scripts/ops.py install-check` | +| 备份实例 | `python scripts/ops.py backup --out <目录>` | +| 恢复到空库 | `python scripts/ops.py restore --dump <备份.sql.gz> --database <库名> --confirm` | +| 校验实例 | `python scripts/ops.py verify --database <库名>` | ## 1. 永久规则 @@ -282,6 +286,7 @@ MVP 内所有单元任务通过后才能做 MVP 集成验收；MVP 通过后才 - #4 独立阅读选择小样位于 `spikes/selection/`，`python spikes/selection/serve.py` 默认仅本机 5184。桌面鼠标/键盘与 11 项测试已验证，#4 已获用户验收并关闭；真实手机长按/手柄/滚动详细证据仍未提供；禁止把窄屏桌面当作真机验收。Intl.Segmenter 只用于 UI 范围验证，不替代 #3 NLP；释义保存只在内存。固定 LinguaCafe 源码对照和与 v1 的差异记录见架构 Wiki。 - 2026-09-11 用户确认正式 NLP/词典采用全 Go。#5 已验收并合入 main；#6 使用 Go WordNet 解析和词形候选、Go Unicode 原文分片、schema v4 共享词典资源表，不调用 Python NLP。WordNet 3.0 ZIP 来源与摘要见 `server/wordnet-resource.json`，许可保留在 `server/WORDNET-LICENSE.txt`。词形候选不等于上下文消歧，不自动合并个人学习状态；#3 Python 小样只保留历史验证。当前词典仅英语释义，个人释义输入为临时草稿，持久化归 #7。 - 2026-09-11 用户确认 #7 个人词条口径（三项由 Agent 定案）：身份为「学习者＋语言＋规范化词形」，大小写合并但**不按 lemma/候选合并**（`dog` 与 `dogs` 是两条记录）；首次保存默认「新词」；状态为 新词/学习中/已知/忽略，只有「学习中」带 1～7 级，对应原版 stage 2/1/0/-1～-7；例句只保存手输内容，不自动关联原文句子。schema v5 新增 `lexgo_terms`（唯一键加状态/等级检查约束），个人释义与共享词典分离且不进入审计日志；等级编辑 UI 归 #8/#12。 +- 2026-09-15 用户确认 #15 交付口径：只交付本机可复现的安装/备份/恢复材料并在本机演练，**不对外部署、不创建 release/tag、不邀请用户**；生产入口与 HTTPS 只写入文档；前端由反向代理托管 dist（不改后端代码）；试用实例从**空库**开始、管理员由显式 bootstrap 建立、不带默认密码；备份＝MySQL 全库 dump ＋ 环境配置（凭据只存运维密码库，不进仓库/日志），不新增定时备份；性能用人造数据集实测并写明环境，只作观察不给承诺。备份/恢复规则：`restore` 必须 `--confirm`、默认只写空库、覆盖需 `--force`、库名必须含 lexgo 且不能是系统库、拒绝带 CREATE DATABASE/USE 的 dump，恢复前后比对源库逐表内容校验和。**附件（#21）尚未实施，恢复契约目前只覆盖数据库**；真实回滚、HTTPS、多机与定时备份仍未验证。部署与运维规则见 Wiki 页 `Deployment-and-Operations`（镜像 `docs/11-deployment-and-operations.md`）。 - 2026-09-15 用户确认 #14 显示与键盘口径：`theme ∈ {浅色,深色,跟随系统}`（默认跟随系统）与正文字号 `{标准,大,特大}`（1.0/1.15/1.3）**按账号保存在本机** `lexgo-learner-display:<账号 id>`，切换账号即换成该账号偏好或默认，退出回到默认，**不跨设备同步**；字号经 `--reader-font-scale` 只作用于阅读面（正文、释义内容、复习卡），不做全局缩放；深色用 `html[data-theme]` ＋ Element Plus 的 `html.dark`，`style.css` 的 `:root` 是文件内仅有的颜色字面量；阅读位置按账号＋章节保存滚动比例与该章 `content_sha256`，**正文换新版本后不恢复**；复习页 `空格`/`Enter` 显示答案、`1/2/3` 评分，输入类控件与聚焦按钮的按键不被劫持，带修饰键不拦截；移动验证用 390×844＋`hasTouch` 的 Playwright `mobile` 项目（桌面项目 `testIgnore: mobile-*`），**真机长按选择与手感仍需人工确认**，不得用模拟设备结果冒充真机。本单无 schema 与接口变化。 - 2026-09-15 用户确认 #13 完成阅读与进度口径：`POST /api/v1/chapters/:id/complete` 只记已读、**不批量改变词语状态或等级**，只有 `ready` 章节可标记（其他 409），重复调用返回同一行且带 `duplicate`（不移动时间、不重复计数）；完成记录保存标记时的 `content_sha256`，**正文新版本后该章回到未读**（记录保留，重读后更新同一行），只改标题不影响，章节删除随外键级联；`GET /api/v1/progress` 统计只含本人与当前语言，已读与分母都只算可阅读（`ready`）章节，已知/学习中/新词/忽略分开计数，`dueNow` 与到期复习队列共用 `dueTermsQuery` 与同一服务端时钟；schema 升到 v7（新表 `lexgo_chapter_progress`，不用 ALTER TABLE），需显式 migrate。不做每日目标、日历、难度评分、统计导出与取消已读，也不做 X10 批量标已知。 - 2026-09-15 既有缺陷记录：#32 编辑章节正文回到曾经用过的版本返回 500（`lexgo_ingest_jobs` 请求键与 `edit:<章节>:<内容摘要>` 冲突），已定位未修复，方案待用户确认；#13 的真实验证脚本因此自建 fixture 而不改写既有正文。 diff --git a/README.md b/README.md index 57bb5aa..4d3882e 100644 --- a/README.md +++ b/README.md @@ -8,10 +8,10 @@ - [英语分词与离线词典验证小样](spikes/english/README.md)（#3 已验收，独立本机入口） - [阅读选择验证小样](spikes/selection/README.md)（#4 已验收，真机详细测试证据缺口保留） - [项目档案](docs/00-project-profile.md) · [需求总览](docs/09-product-requirements-overview.md) -- [工作量估算](docs/10-workload-estimate.md)：#2、#3、#4、#5、#6、#7、#8、#9、#10、#11、#12、#13、#14、#18 已验收，原规划中的 #5 已完成；剩余 #15 与新增 #21、#24 按工单复核；缺陷 #32 待确认修复方案；后续结合集成结果重估，旧全量研究仅供参考。 +- [工作量估算](docs/10-workload-estimate.md)：#2、#3、#4、#5、#6、#7、#8、#9、#10、#11、#12、#13、#14、#18 已验收，原规划中的 #5 已完成；#15 已实现待验收；新增 #21、#24 待排期；缺陷 #32 待确认修复方案；后续结合集成结果重估，旧全量研究仅供参考。 - [四阶段实施总览 #16](https://git.ilapage.cn/OPC/lexgo/issues/16)：14 张单元工单，工程基础 → 技术验证 → 首条学习闭环 → 补齐 MVP；原型 v1 已获用户验收。两端使用账号（用户名）＋密码登录，不要求邮箱。 - [原型工单 #1](https://git.ilapage.cn/OPC/lexgo/issues/1)：Quant-UX 桌面/手机原型 v1，预览入口与审核记录见工单及需求总览。 -- [工作流](docs/01-workflow.md) · [开发与验证](docs/04-local-development-and-verification.md) +- [工作流](docs/01-workflow.md) · [开发与验证](docs/04-local-development-and-verification.md) · [部署与运维](docs/11-deployment-and-operations.md) ## 本地工程基础 diff --git a/docs/02-architecture-and-code-map.md b/docs/02-architecture-and-code-map.md index 3bdc643..606f7b2 100644 --- a/docs/02-architecture-and-code-map.md +++ b/docs/02-architecture-and-code-map.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件) wiki_page: Architecture-and-Code-Map wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Architecture-and-Code-Map.- -wiki_revision: b61d9e1b0d9ae9fc24cb026580149f2b582a2526 -synchronized_at: 2026-09-15T07:44:50Z +wiki_revision: bd4a0e705abe722f153a7410435def91a21d472a +synchronized_at: 2026-09-15T08:27:37Z # 架构与代码地图 @@ -371,3 +371,21 @@ schema v7 新增 `lexgo_chapter_progress`：一章一行（`chapter_id` 主键 `style.css` 收敛为一套语义调色板：`:root` 里 53 个变量是**文件内仅有的颜色字面量**，其余规则全部走 `var(...)`；`html[data-theme='dark']` 覆盖同一批变量，`html.dark` 同时映射 Element Plus 的暗色变量（`main.ts` 引入 `element-plus/theme-chalk/dark/css-vars.css`），所以页面与组件跟随同一个选择。字号经 `--reader-font-scale` 只作用于阅读面（章节正文、释义面板内容、复习卡的词与释义），不做全局缩放。`ReaderView.vue` 在章节就绪后把正文下方的位置恢复到上次比例（正文版本变化则不恢复，等待可滚动后再应用，最多约 1 秒后放弃），并在滚动时按 400ms 防抖保存。`ReviewView.vue` 加了 `useReviewShortcuts` 与一行快捷键提示；`DisplaySettings` 出现在七个学习页面的头部。 移动测试项目：`playwright.config.ts` 增加 `mobile`（390×844，`isMobile` + `hasTouch`，Chromium），桌面项目用 `testIgnore` 排除 `mobile-*.spec.ts`，移动项目用 `testMatch` 只跑它们。`e2e/mobile-fixtures.ts` 提供移动端共用的 mock、无溢出检查与经 CDP `Input.dispatchTouchEvent` 的真实触摸滑动。 + +## #15 自托管交付：部署拓扑与运维工具（2026-09-15） + +**没有 schema 变化、没有接口变化**：本单新增运维工具、部署文档与完整演练证据。 + +| 新增文件 | 职责 | +|---|---| +| `scripts/ops.py` | `install-check`（依赖与资源版本）、`init-database`（建空库并提示最小权限）、`backup`（全库 dump ＋ `manifest.json`）、`restore`（默认只写空库，写后自动校验）、`verify`（数据库完整性 ＋ 可选两账号接口闭环）、`smoke`（在干净实例上建两个演练账号走通学习闭环） | +| `scripts/bench.py` | 在写明规模的人造数据集上测量接口耗时，输出数据量、机器信息与 p50/p95，不做容量承诺 | +| `tests/test_lexgo_ops.py` | 不需要数据库的规则测试：版本号解析、库名白名单、dump 是否带库名切换、manifest 字段白名单、审计敏感列清单 | + +**一次备份包含什么**：LexGo 把所有持久数据都放在 MySQL 里——账号与会话、书籍与章节原文、导入任务、词典归档（LONGBLOB）、个人词条、复习排期与作答、阅读进度、审计日志。因此备份 = 全库 dump ＋ `.env.local`（凭据单独从运维密码库取）。词典资源在显式导入后进入数据库，恢复即带走，运行时不下载。 + +**恢复的安全边界**：`restore` 必须带 `--confirm`；库名必须含 `lexgo` 且不能是系统库；目标库已有数据时默认拒绝，覆盖需要 `--force`；拒绝加载带 `CREATE DATABASE`/`USE` 的旧式 dump（那会把数据写进文件里指定的库）；恢复前后比对**源库**逐表内容校验和，源库被改动就中止。 + +**部署文档**：新建 Wiki 页 `Deployment-and-Operations`（`docs/11-deployment-and-operations.md`，按 `docs/templates/deployment.md` 结构），并加入 `wiki-docs.json` 映射。内容含服务概览、环境要求（含「MySQL 客户端版本不得低于服务端」这条实测规则）、首次部署、配置与凭据来源、日常运维、健康检查、升级与回滚、备份与恢复、已知限制。 + +**视图与进程**：后端单二进制监听 `127.0.0.1:8000`；两份 SPA 由反向代理托管 `dist`，反代把 `/api/` 转发到后端并把未知路径回落到 `index.html`；本机开发用 supervisor 托管 `lexgo-api`/`lexgo-learner`/`lexgo-admin` 三个 program。 diff --git a/docs/03-business-rules-and-glossary.md b/docs/03-business-rules-and-glossary.md index cec23d6..aa74316 100644 --- a/docs/03-business-rules-and-glossary.md +++ b/docs/03-business-rules-and-glossary.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件) wiki_page: Business-Rules-and-Glossary wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Business-Rules-and-Glossary.- -wiki_revision: 5a44b1713b3fd2374baa78314872539f3e2e23ab -synchronized_at: 2026-09-15T07:11:02Z +wiki_revision: 5f2912edd09256cf5fc52059145d65918e99f9cb +synchronized_at: 2026-09-15T08:27:38Z # 业务规则与术语 @@ -290,3 +290,17 @@ exact优先；未命中再按WordNet异常表/词尾规则查候选，词性顺 **键盘操作**：复习页 `空格`/`Enter` 显示答案，`1` 认识/答对、`2` 不认识/答错、`3` 再学一次；界面给出快捷键提示，并保持「显示答案」与「答对」按钮自动获得焦点，键盘用户可以全程不碰鼠标。按键规则：**输入框、文本域、下拉与可编辑区域内的任何按键都不触发快捷键**；按下的键若属于当前聚焦控件（按钮与链接的 `空格`/`Enter`）则交给浏览器处理，避免重复触发；带修饰键的组合键一律不拦截。阅读器 `Esc` 关闭释义面板，面板内的 `Esc` 只关面板。 **触摸与滚动**：没有任何阻止默认行为的触摸处理；释义面板在窄屏是底部固定面板，页面为其预留底部空间，关闭后阅读位置不跳动。**长按原生选择与手柄拖动的手感仍需真机确认**，本单的触摸证据来自模拟设备。 + +## #15 交付与运维规则（2026-09-15） + +**数据边界**：一个 LexGo 实例的全部持久数据都在 MySQL 中（账号、空间、会话、审计日志、书籍与章节原文、导入任务、词典归档、词条、复习排期、复习作答、阅读进度）。备份的完整对象因此是「全库 dump ＋ 环境配置文件」；环境配置含凭据，单独保存在运维密码库，不放进备份目录。 + +**备份规则**：`backup` 使用 `mysqldump --single-transaction --routines --triggers --hex-blob --no-tablespaces`，输出 `.sql.gz` 与 `manifest.json`。manifest 记录创建时间、库名、产品标识、schema 版本、Git 提交、dump 的 sha256 与体积、逐表行数、客户端与服务端版本、包含与排除的内容说明；**不记录任何凭据值**。备份文件含个人数据，按个人数据对待：受控存放、不进 Git、不进工单。 + +**恢复规则**：恢复是写入操作，必须显式 `--confirm`；默认只写**空库**，目标库已有数据时拒绝，覆盖需要 `--force` 并在覆盖前另行备份；库名必须含 `lexgo` 且不能是 MySQL 系统库；带 `CREATE DATABASE`/`USE` 的 dump 一律拒绝（会写错库）；恢复前后比对源库逐表内容校验和，源库变化即中止。 + +**升级与回退规则**：schema 迁移只新增对象（`CREATE TABLE IF NOT EXISTS`），旧二进制按版本号校验；回退到旧二进制前把 `lexgo_schema.version` 写回旧版本号。若迁移删改数据或旧二进制无法读写新结构，则按恢复流程处理。**真实回滚演练尚未执行**，属已知限制。 + +**安装规则**：空库不含默认密码与任何演示数据；管理员只能由显式 `bootstrap` 建立，库中已有账号时拒绝再次执行、不覆盖既有管理员；应用账号只对该库拥有最小权限，不使用管理员账号运行应用；服务默认只监听 `127.0.0.1`，对外由反向代理转发。 + +**验收账号规则**：试用验证使用虚构的演练账号，密码由演练进程生成、只经环境变量传入，不写入磁盘、日志、工单或截图。 diff --git a/docs/04-local-development-and-verification.md b/docs/04-local-development-and-verification.md index 61ce5d4..6e4bbfa 100644 --- a/docs/04-local-development-and-verification.md +++ b/docs/04-local-development-and-verification.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件) wiki_page: Local-Development-and-Verification wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Local-Development-and-Verification.- -wiki_revision: e6255c24e7b966573a9851b1e2356e370d3daef1 -synchronized_at: 2026-09-15T07:11:02Z +wiki_revision: caa80888cf42fd7fd7f040e9fe1147203fe7bfe3 +synchronized_at: 2026-09-15T08:27:38Z # 本地开发与验证 @@ -493,3 +493,70 @@ node --test spikes/english/view.test.mjs **真实链路**：真实学习端＋真实 Go API＋真实 MySQL 走了一遍——粘贴 40 行长章节并处理完成，滚到 45% 后离开再返回，位置被恢复（偏差 < 150px，页面提示「已回到上次阅读位置」）；切到深色＋特大后刷新仍保持，正文实际字号变大（> 18px）；在真实到期卡上按空格显示了答案（**故意不评分**，避免改动 fixture 账号的真实复习计划）；另用 390×844 触摸上下文在真实数据上检查了书库/生词本/进度无横向溢出。fixture 书籍在结束时删除；截图见 `.local/evidence/issue14-real-*.png`、`.local/evidence/issue14-mobile-*.png`、`issue14-theme-*.png`、`issue14-keyboard-review.png`。临时用例运行后删除。 **未验证**：真实手机上的长按选择、手柄拖动、滚动惯性、软键盘遮挡与安全区表现无法由本机自动化产生——本单的触摸证据来自模拟设备；**真机验证仍需人工完成**，这一点在 #4、#11 就已记录，本单未能消除。也未做 Safari/Firefox 等其他引擎的验证（项目既有测试只在 Chromium 上跑）。 + +## #15 交付、备份与恢复验证（2026-09-15） + +本单**无 schema 变化、无迁移**：新增运维脚本、部署文档与演练证据。所有命令在仓库根执行；MySQL 客户端用 8.4（`LEXGO_MYSQL_BIN=D:/mysql843/bin`），因为本机 PATH 上是 5.7 客户端，`install-check` 会把它判为不达标。 + +### 运维命令 + +| 用途 | 命令 | +|---|---| +| 依赖与资源检查 | `python scripts/ops.py install-check` | +| 建空库并提示权限 | `python scripts/ops.py init-database --database <库名>` | +| 备份 | `python scripts/ops.py backup --out <目录>` | +| 恢复到空库 | `python scripts/ops.py restore --dump <备份.sql.gz> --database <库名> --confirm` | +| 校验（数据完整性，可选接口闭环） | `python scripts/ops.py verify --database <库名> [--manifest ] [--api http://127.0.0.1:<端口> --user <前缀> --password-env <变量名>]` | +| 干净实例两账号闭环演练 | `python scripts/ops.py smoke --api --admin-user <账号> --user <前缀>` | +| 性能测量 | `python scripts/bench.py --database <库名> --api --user <账号>` | + +### 完整演练（2026-09-15，本机一次性通过，27 项顶层检查全部 `[ok]`） + +演练脚本（`.local/tmp/issue15-drill.py`，一次性工具）依次执行： + +1. **干净安装**：`init-database` 建空库 → `migrate`（schema v7，14 张表）→ 确认「新库没有账号、没有业务数据」→ `bootstrap` 建管理员 → **再次 `bootstrap` 被拒绝**（退出码 1，不覆盖管理员）→ 起独立 API 实例（:8010）。 +2. **两个演练账号走通闭环**（`ops.py smoke`，19 项全过）：管理员建 `trial_a`/`trial_b` → 新账号书库与进度为空 → 粘贴章节 → 章节处理完成 → 分词返回可点选词 → 词典查询返回明确状态 → 保存词义 → 另一账号查不到该词条 → 新词立刻到期 → 作答成功 → 重复作答不重复记账 → 完成章节只记已读 → 进度反映活动；`B` 读 `A` 的书 404、书库仍为空。 +3. **备份 lexgo_dev**（约 12 MB gz）：manifest 记录 schema 7、提交哈希、逐表行数、sha256；确认 manifest 无凭据字段；确认 dump 不含 `CREATE DATABASE`/`USE`（不会误写回源库）。 +4. **恢复到空库** `lexgo_restore_drill` 并自动校验（33 项全过）：逐表行数与备份一致（`sys_user` 6、`books` 27、`chapters` 33、`terms` 13、`term_reviews` 13、`review_answers` 55、`login_logs` 192 等）→ **逐表内容校验和与源库一致**（13 张表）→ **源库 lexgo_dev 未被改动**。 +5. **安全边界**：恢复默认拒绝覆盖已有库；拒绝写入系统库 `mysql`；不带 `--confirm` 直接拒绝。 +6. **恢复实例的两账号闭环与越权校验**（`:8011`，41 项全过）——恢复后的账号仍可登录、词条与到期队列仍在、答题与完成章节保持幂等、`B` 无法读取或修改 `A` 的数据。 +7. **收尾**：删除两个演练库，确认 `lexgo_dev` 与测试库保留，`lexgo_dev` 仍是 schema 7。 + +### 性能观察（`scripts/bench.py`，人造数据集，单机无并发） + +数据集：1 个账号、20 本书 20 章（每章约 500 个词）、2000 词条、8000 条复习作答。环境：Windows 10（10.0.19044）、8 核、31.7 GB 内存、MySQL 8.4.3、Go 工具链由 `scripts/server.py` 固定为 1.26.5。 + +| 接口 | 请求数 | p50 | p95 | 最大 | 错误 | +|---|---|---|---|---|---| +| `POST /login` | 10 | 82.2 ms | 93.3 ms | 93.3 ms | 0 | +| `GET /books` | 30 | 5.9 ms | 18.1 ms | 18.2 ms | 0 | +| `GET /chapters/1` | 30 | 8.3 ms | 20.1 ms | 22.0 ms | 0 | +| `GET /chapters/1/tokens` | 30 | 10.0 ms | 23.6 ms | 30.3 ms | 0 | +| `GET /terms`（第 1 页） | 30 | 10.2 ms | 20.4 ms | 21.5 ms | 0 | +| `GET /terms?query`（搜索） | 30 | 12.3 ms | 24.9 ms | 27.0 ms | 0 | +| `GET /progress` | 30 | 15.7 ms | 26.1 ms | 29.1 ms | 0 | +| `GET /reviews/queue` | 30 | 22.7 ms | 32.3 ms | 34.2 ms | 0 | + +**这些是观察值，不是容量承诺**：单机、单进程、无并发压力；登录接口每地址每分钟限 30 次，因此登录只测 10 次，该数字不代表登录吞吐。原始报告：`.local/issue15-bench.json`。 + +### 集成验证矩阵（复用既有 Go 测试，`LEXGO_TEST_DB_NAME=lexgo_test_issue13` 70 项全过） + +| 验收要求 | 覆盖用例（全部通过） | +|---|---| +| 越权与隔离 | `TestMySQLLibraryIsolationAndOwnership`、`TestMySQLTermIsolationAndInputRules`、`TestMySQLPhraseRulesAndIsolation`、`TestMySQLReviewAnswerErrorsAndOwnership`、`TestMySQLAccountIsolationAndRevocation` | +| 任务崩溃重试 | `TestMySQLIngestRecoveryWithoutRestart`、`TestMySQLIngestRecoveryAfterRestart`、`TestMySQLIngestAttemptsAreBoundedAndManualRetryRestarts`、`TestMySQLRetryAfterContentRestoredPublishesSameChapter`、`TestMySQLRecoverySkipsSupersededJobs` | +| 并发答题与并发删除 | `TestMySQLReviewConcurrentReplayOfOneAnswer`、`TestConcurrentDuplicateAccountHasOneWinner`、`TestMySQLConcurrentChapterDelete`、`TestMySQLDeleteDuringProcessing` | +| 长文与输入上限 | `TestMySQLPasteRejectsInvalidInputAndLimits`、`TestMySQLTextUploadKeepsChapterLimit`、`TestMySQLTextUploadRejectsInvalidSubmissions`、`TestMySQLTextUploadRejectsInvalidSubmissions` | +| 迁移与回退标记 | `TestMigrationRefusesUnownedOrUnsupportedSchema`、`TestMigrationCanResumeOwnedPartialSchema`、全部 `TestMigrationFrom*` | + +### 治理与其它测试 + +`python -m unittest discover -s tests` 65 项通过（新增 9 项 `tests/test_lexgo_ops.py`，不需要数据库）；`harness.py check --strict` 与 `sync --check` 通过。本单未改前端与后端业务代码，仍复跑 `LEXGO_TEST_DB_NAME=lexgo_test_issue13 python scripts/server.py test-integration` 70 项通过。 + +### 未验证 + +- **真实回滚（升级后切回旧二进制）未演练**：文档给出规则（迁移只新增对象，回退前写回版本号），但没有在真实实例上做过升级/回滚全过程。 +- 未验证 HTTPS、域名、反代配置、多机与灰度；本机演练只在 `127.0.0.1` 上起两个临时实例。 +- 未做定时备份、监控与告警；备份与恢复均为人工触发。 +- 大库恢复耗时与磁盘空间上限未测；演练库与生产库规模不同。 +- 附件（封面/音频，#21）尚未实现，恢复范围目前只覆盖数据库。 diff --git a/docs/09-product-requirements-overview.md b/docs/09-product-requirements-overview.md index 474149c..d8cab5b 100644 --- a/docs/09-product-requirements-overview.md +++ b/docs/09-product-requirements-overview.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件) wiki_page: Product-Requirements-Overview wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Product-Requirements-Overview.- -wiki_revision: 358d7720d952a49b6827a96173a4f63387ac6989 -synchronized_at: 2026-09-15T07:44:50Z +wiki_revision: a1847957e287c7eb1a54905e13632aeca4cd75f7 +synchronized_at: 2026-09-15T08:27:38Z # 产品需求总览 @@ -287,3 +287,9 @@ F11 的完成阅读与基础进度已于 2026-09-15 通过用户验收：阅读 F12 的电脑与手机基础体验已于 2026-09-15 通过用户验收：新增「显示」设置（浅色/深色/跟随系统；正文标准/大/特大），偏好按账号保存在本机、账号切换不串、不跨设备同步；阅读位置按账号与章节保留，正文换新版本后不套用旧位置；复习页支持空格显示答案与 1/2/3 评分，输入框内不误触全局快捷键，焦点环可见；手机端以 390×844 触摸视口验证八个界面无横向溢出、均可返回、触摸滑动可滚动且不误开释义面板。本次无数据库与接口变化。 仍未实现并留给后续工单：自托管试用交付与完整恢复（#15）、书籍音频与封面附件（#21）、书库列表优化（#24），以及已定位未修复的缺陷 #32。PWA、离线能力、自定义配色、全局 UI 缩放与偏好跨设备同步明确不在范围内。**真实手机的长按选择与操作手感仍需人工确认**，本单的触摸证据来自模拟设备。 + +## #15 交付状态更新（2026-09-15） + +B04 与 F01–F12 的集成交付已实现，待用户验收：新部署与运维 Wiki 页（首次部署、配置与凭据来源、日常运维、健康检查、升级回滚、备份恢复、已知限制），运维工具 `scripts/ops.py`（`install-check` / `init-database` / `backup` / `restore` / `verify` / `smoke`）与 `scripts/bench.py`。本机完成完整演练：空库安装并由管理员建立两个虚构演练账号走通学习闭环；备份开发库后恢复到空实例，逐表行数与内容校验和一致、源库未被改动，两账号闭环与越权校验全通过。性能在 20 章／2000 词条／8000 条作答的人造数据集上实测（p50 6–23 ms，登录 82 ms，均无错误），只作观察不给承诺。 + +用户已确认的范围边界：本次只交付本机可复现的安装/备份/恢复材料并在本机演练，**不对外部署、不创建发布标签、不邀请用户**；生产入口与 HTTPS 只写入文档。**真实回滚、定时备份、监控告警、多机部署与 HTTPS 仍未验证**；附件（音频/封面，#21）尚未实现，因此恢复契约目前只覆盖数据库，待 #21 落地后由本单的恢复验收纳入附件。遗留缺陷 #32（编辑正文回到曾用版本返回 500）仍待确认修复方案。 diff --git a/docs/README.md b/docs/README.md index 3931ba6..02bd113 100644 --- a/docs/README.md +++ b/docs/README.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件) wiki_page: Home wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Home -wiki_revision: 9f6485206c60c737b386fdf2f2946adf606d53e7 -synchronized_at: 2026-09-15T07:44:49Z +wiki_revision: 22731c2b344b3e0c75eb892c897d36d1f153a441 +synchronized_at: 2026-09-15T08:27:37Z # LexGo 文档入口 @@ -86,3 +86,5 @@ Quant-UX 原型 v1 已通过用户验收。[桌面预览](https://qux.ilapage.cn #13 完成章节与个人基础进度已于 2026-09-15 通过用户验收：阅读页正文下方提供显式的「标记本章已读」，只记录已读、不批量改变词语状态；重复标记返回同一条记录、不重复计数。书籍页章节列表显示「已读」，新的「进度」页面显示已读章节（只统计可阅读章节，形如「3 / 8 章（可阅读）」）、待复习、已知词、学习中、新词、忽略与已保存词条数，并按书显示已读进度；待复习与到期复习队列使用同一规则与同一时钟。正文被改成新版本后该章会回到未读（记录保留），重新标记即更新同一条记录；删除章节时记录随之消失。本次新增 schema v7（一张新表），需要显式迁移；开发库迁移前后各表计数一致。真实 API＋MySQL 37 项检查与浏览器页面数字对照数据库直查全部一致；PR #33 已 fast-forward-only 合入 main。 #14 桌面与手机体验、主题与键盘操作已于 2026-09-15 通过用户验收：新增「显示」设置（浅色/深色/跟随系统，正文标准/大/特大），偏好按账号保存在本机，切换账号不会串；阅读位置按账号与章节保留，正文换成新版本后不再套用旧位置；复习页支持空格显示答案、1/2/3 评分，输入框内不会误触快捷键，所有可交互元素有可见焦点环。手机端用 390×844 触摸视口验证八个界面无横向溢出、都能返回、触摸滑动滚动而不误开释义面板，深色与特大字号刷新后保持。本次没有数据库与接口变化；PR #34 已 fast-forward-only 合入 main。真实手机的长按选择与手感仍需人工确认。 + +#15 自托管试用交付已实现，待用户验收：新增运维工具（`scripts/ops.py`：依赖检查、建空库、备份、恢复、校验、两账号闭环演练；`scripts/bench.py`：写明规模的人造数据集性能测量）、新的 Wiki 部署与运维页，以及一次完整的本机演练——空库安装（建库 → 迁移 → 首次 bootstrap 建管理员 → 管理员建两个虚构演练账号 → 走通粘贴/阅读/查词/保存/复习/完成章节/进度）、备份 lexgo_dev、恢复到空实例并用第二个 API 实例完成两账号闭环与越权校验，恢复后逐表行数与内容校验和都和源库一致、源库未被改动。备份含全部用户数据（数据库）与环境配置（单独保管凭据），恢复默认只写空库、覆盖需要显式确认与 `--force`、拒绝写入系统库。性能在 20 章/2000 词条/8000 条作答的人造数据集上实测并记录环境，只作观察不给承诺。真实回滚、HTTPS、多机与定时备份仍未验证；附件（#21）尚未实施，恢复范围目前只覆盖数据库。 diff --git a/scripts/bench.py b/scripts/bench.py new file mode 100644 index 0000000..431ca5b --- /dev/null +++ b/scripts/bench.py @@ -0,0 +1,207 @@ +"""Measures the LexGo API on a stated synthetic dataset. + +The numbers this prints describe one machine and one dataset; they are an observation, not a +capacity promise. Everything runs against a database whose name contains ``lexgo`` and that the +operator creates for the run, so no real instance is touched. +""" +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import platform +import statistics +import subprocess +import sys +import time +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / "scripts")) +import ops # noqa: E402 + +WORDS = ("curiosity", "door", "step", "reading", "language", "garden", "window", "music", "river", + "stone", "letter", "morning", "shadow", "bridge", "silence", "travel", "kitchen", "paper") + + +def machine_facts() -> dict: + facts = { + "os": platform.platform(), + "cpu_count": os.cpu_count(), + "python": platform.python_version(), + } + try: + import shutil + + if os.name == "nt": + output = subprocess.run(["wmic", "computersystem", "get", "TotalPhysicalMemory"], + capture_output=True, text=True, encoding="utf-8", errors="replace").stdout + numbers = [line.strip() for line in output.splitlines() if line.strip().isdigit()] + if numbers: + facts["memory_gb"] = round(int(numbers[0]) / (1024 ** 3), 1) + else: + with open("/proc/meminfo") as handle: + for line in handle: + if line.startswith("MemTotal"): + facts["memory_gb"] = round(int(line.split()[1]) / (1024 ** 2), 1) + break + facts["go"] = subprocess.run(["go", "version"], capture_output=True, text=True, + encoding="utf-8", errors="replace").stdout.strip() + del shutil + except Exception: + pass + return facts + + +def seed(database: str, chapters: int, chapter_words: int, terms: int, reviews: int) -> dict: + """Fills a fresh benchmark database with a synthetic learner, book, terms and schedules. + + The account itself is left alone: it was created through the API beforehand, so its password + hash is a real one and the benchmark measures the same login path a user takes. + """ + for table in ("lexgo_review_answers", "lexgo_term_reviews", "lexgo_terms", "lexgo_chapters", + "lexgo_books", "lexgo_spaces", "lexgo_sessions"): + ops.query(database, "DELETE FROM " + table) + owner = int(ops.scalar(database, "SELECT user_id FROM sys_user ORDER BY user_id LIMIT 1", "0") or 0) + if not owner: + raise SystemExit("先创建一个账号再压测（例如 smoke 或 bootstrap）。") + ops.query(database, "INSERT INTO lexgo_spaces (owner_id, language) VALUES (%d, 'en')" % owner) + stamp = "2026-01-01 00:00:00.000" + + body = " ".join(WORDS[index % len(WORDS)] for index in range(chapter_words)) + for chapter in range(1, chapters + 1): + text = "%s chapter %d. " % (body, chapter) + sha = hashlib.sha256(text.encode("utf-8")).hexdigest() + ops.query(database, "INSERT INTO lexgo_books (id, owner_id, title, language, created_at, updated_at) " + "VALUES (%d, %d, 'Bench book %d', 'en', '%s', '%s')" % (chapter, owner, chapter, stamp, stamp)) + ops.query(database, "INSERT INTO lexgo_chapters (id, book_id, owner_id, ordinal, title, original_text, char_count, " + "content_sha256, status, created_at, updated_at) VALUES (%d, %d, %d, 1, 'Bench %d', '%s', %d, " + "'%s', 'ready', '%s', '%s')" % (chapter, chapter, owner, chapter, text.replace("'", "''"), len(text), sha, stamp, stamp)) + + rows = [] + for term_id in range(1, terms + 1): + word = "%s%d" % (WORDS[term_id % len(WORDS)], term_id) + rows.append("(%d, %d, 'en', '%s', '%s', '虚构释义', '', 'new', 0, '%s', '%s')" % (term_id, owner, word, word, stamp, stamp)) + if len(rows) == 500: + ops.query(database, "INSERT INTO lexgo_terms (id, owner_id, language, term, original_form, definition, examples, " + "status, level, created_at, updated_at) VALUES " + ",".join(rows)) + rows = [] + if rows: + ops.query(database, "INSERT INTO lexgo_terms (id, owner_id, language, term, original_form, definition, examples, " + "status, level, created_at, updated_at) VALUES " + ",".join(rows)) + + rows = [] + for term_id in range(1, terms + 1): + rows.append("(%d, %d, 'en', DATE_ADD('%s', INTERVAL %d SECOND), 0, 0, 0)" % (term_id, owner, stamp, term_id % 900000)) + if len(rows) == 500: + ops.query(database, "INSERT INTO lexgo_term_reviews (term_id, owner_id, language, due_at, review_count, " + "correct_count, wrong_count) VALUES " + ",".join(rows)) + rows = [] + if rows: + ops.query(database, "INSERT INTO lexgo_term_reviews (term_id, owner_id, language, due_at, review_count, " + "correct_count, wrong_count) VALUES " + ",".join(rows)) + + if reviews: + rows = [] + for index in range(1, reviews + 1): + rows.append("(%d, %d, 'b%063d', %d, 'correct', 'applied', 'new', 'learning', 0, 1, '%s', '%s', 0, '%s')" + % (index, owner, index, (index % terms) + 1, stamp, stamp, stamp)) + if len(rows) == 500: + ops.query(database, "INSERT INTO lexgo_review_answers (id, owner_id, answer_key, term_id, grade, result, " + "status_before, status_after, level_before, level_after, due_at_before, due_at_after, " + "requeued, created_at) VALUES " + ",".join(rows)) + rows = [] + if rows: + ops.query(database, "INSERT INTO lexgo_review_answers (id, owner_id, answer_key, term_id, grade, result, " + "status_before, status_after, level_before, level_after, due_at_before, due_at_after, " + "requeued, created_at) VALUES " + ",".join(rows)) + + return { + "owner_id": owner, + "chapters": chapters, + "words_per_chapter": chapter_words, + "terms": terms, + "review_answers": reviews, + "row_counts": ops.row_counts(database), + } + + +def measure(label: str, call, repeats: int) -> dict: + timings = [] + errors = 0 + for index in range(repeats): + started = time.perf_counter() + status, _ = call(index) + elapsed = (time.perf_counter() - started) * 1000 + if status >= 400: + errors += 1 + timings.append(elapsed) + timings.sort() + return { + "endpoint": label, + "requests": repeats, + "errors": errors, + "p50_ms": round(statistics.median(timings), 1), + "p95_ms": round(timings[min(len(timings) - 1, int(len(timings) * 0.95))], 1), + "max_ms": round(timings[-1], 1), + } + + +def main() -> int: + parser = argparse.ArgumentParser(description="在写明规模的人造数据集上测量接口耗时") + parser.add_argument("--database", default="lexgo_bench") + parser.add_argument("--api", default="http://127.0.0.1:8012") + parser.add_argument("--user", default="bench_admin", help="压测账号") + parser.add_argument("--chapters", type=int, default=20) + parser.add_argument("--chapter-words", type=int, default=500) + parser.add_argument("--terms", type=int, default=2000) + parser.add_argument("--reviews", type=int, default=8000) + parser.add_argument("--repeats", type=int, default=30) + parser.add_argument("--out", default=str(ROOT / ".local" / "issue15-bench.json")) + args = parser.parse_args() + + ops.load_env() + database = ops.validate_database_name(args.database, "压测") + dataset = seed(database, args.chapters, args.chapter_words, args.terms, args.reviews) + print("数据集：" + json.dumps(dataset["row_counts"], ensure_ascii=False)) + + client = ops.Api(args.api) + password = os.environ.get("LEXGO_BENCH_PASSWORD", os.environ.get("LEXGO_TRIAL_PASSWORD", "")) + status, token = client.login(args.user, password) + if status != 200 or not token: + raise SystemExit(args.user + " 无法登录：请先在演练库上建号并设置 LEXGO_BENCH_PASSWORD。") + + # The login endpoint allows 30 attempts per minute per address, so a sustained loop would + # measure the limiter instead of the login path. A short loop keeps the number meaningful. + login_repeats = max(1, min(args.repeats, 10)) + results = [measure("POST /login", lambda _: client.login(args.user, password), login_repeats)] + results.append(measure("GET /books", lambda _: client.call("GET", "books", token), args.repeats)) + results.append(measure("GET /chapters/1", lambda _: client.call("GET", "chapters/1", token), args.repeats)) + results.append(measure("GET /chapters/1/tokens", lambda _: client.call("GET", "chapters/1/tokens", token), args.repeats)) + results.append(measure("GET /terms (page 1)", lambda _: client.call("GET", "terms?page=1&limit=20", token), args.repeats)) + results.append(measure("GET /terms?query (search)", lambda _: client.call("GET", "terms?query=curiosity&limit=20", token), args.repeats)) + results.append(measure("GET /progress", lambda _: client.call("GET", "progress", token), args.repeats)) + results.append(measure("GET /reviews/queue", lambda _: client.call("GET", "reviews/queue", token), args.repeats)) + + report = { + "measured_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), + "api": args.api, + "database": database, + "dataset": dataset, + "machine": machine_facts(), + "mysql_server": ops.scalar("", "SELECT VERSION()", ""), + "results": results, + "note": "单机、单进程、无并发压力；数字是观察值，不是容量承诺。登录接口每分钟每地址限 30 次，" + "因此登录只测 10 次，且该数字不代表登录吞吐。", + } + Path(args.out).write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8") + for row in results: + print(" %-28s p50 %7.1f ms p95 %7.1f ms max %7.1f ms 错误 %d" % + (row["endpoint"], row["p50_ms"], row["p95_ms"], row["max_ms"], row["errors"])) + print("报告：" + args.out) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/ops.py b/scripts/ops.py new file mode 100644 index 0000000..87e570f --- /dev/null +++ b/scripts/ops.py @@ -0,0 +1,783 @@ +"""LexGo operations: dependency check, backup, restore and post-restore verification. + +Everything a self-hosted instance needs to be installed, backed up and restored is in MySQL: +accounts, books and chapter text, personal terms, review schedules and answers, the imported +WordNet archive and the audit logs. This script therefore never invents a second storage +location: a dump plus the environment file is a complete backup. + +Safety rules, because a restore can overwrite a working instance: + +* a restore always needs ``--confirm``; +* the target database name must contain ``lexgo`` and must not be a MySQL system schema; +* a database that already holds LexGo data is refused unless ``--force`` is given; +* credentials are read from the environment or ``.env.local`` and are never written into a + manifest, a log line or a dump name. + +Only the standard library is used, so the script runs on the deployment host without installing +anything beyond the MySQL client. +""" +from __future__ import annotations + +import argparse +import gzip +import hashlib +import json +import os +import re +import shutil +import subprocess +import tempfile +import time +import sys +from datetime import datetime, timezone +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +ENV_FILE = ROOT / ".env.local" +SYSTEM_SCHEMAS = {"mysql", "information_schema", "performance_schema", "sys"} +# Columns that must never appear in the audit tables: they exist to prove that no credential, +# request body or private learning content is stored there. +AUDIT_BANNED_COLUMNS = {"password", "token", "body", "content", "definition", "text"} +TABLES = [ + "sys_user", "lexgo_spaces", "lexgo_sessions", "lexgo_login_logs", "lexgo_operation_logs", + "lexgo_books", "lexgo_chapters", "lexgo_ingest_jobs", "lexgo_dictionaries", + "lexgo_terms", "lexgo_term_reviews", "lexgo_review_answers", "lexgo_chapter_progress", +] +SCHEMA_VERSION = 7 + + +class OpsError(SystemExit): + """A failure the operator has to read and act on.""" + + +def load_env(path=ENV_FILE): + """Reads .env.local. Values are never printed; only key names are.""" + values = {} + if path.exists(): + for line in path.read_text(encoding="utf-8").splitlines(): + line = line.strip() + if not line or line.startswith("#") or "=" not in line: + continue + key, value = line.split("=", 1) + values[key.strip()] = value.strip() + for key, value in values.items(): + os.environ.setdefault(key, value) + return values + + +def mysql_binary(name): + """Finds mysqldump/mysql via LEXGO_MYSQL_BIN, then PATH.""" + configured = os.environ.get("LEXGO_MYSQL_BIN") + if configured: + candidate = Path(configured) / (name + (".exe" if os.name == "nt" else "")) + if candidate.exists(): + return str(candidate) + found = shutil.which(name) + if not found: + raise OpsError( + "找不到 " + name + "：请安装 MySQL 客户端，或用 LEXGO_MYSQL_BIN 指向客户端目录。" + ) + return found + + +def connection_args(database=None): + """Builds client arguments. The password travels through the environment, not argv.""" + args = [ + "--host=" + os.environ.get("LEXGO_DB_HOST", "127.0.0.1"), + "--port=" + os.environ.get("LEXGO_DB_PORT", "3306"), + "--user=" + os.environ.get("LEXGO_DB_USER", ""), + ] + if database: + args.append(database) + return args + + +def client_env(): + env = dict(os.environ) + password = os.environ.get("LEXGO_DB_PASSWORD") + if password: + env["MYSQL_PWD"] = password + return env + + +def run(command, env=None, capture=True): + result = subprocess.run(command, env=env, capture_output=capture, text=True, encoding="utf-8", errors="replace") + if result.returncode != 0: + message = (result.stderr or "").strip().splitlines() + # The last line of a client error is the one an operator needs; never echo arguments, + # which is why the command line itself is not part of the message. + raise OpsError("命令失败：" + (message[-1] if message else "未知错误")) + return result.stdout or "" + + +def query(database, sql): + # SQL travels through stdin: a long INSERT would exceed the Windows command line limit. + result = subprocess.run([mysql_binary("mysql"), *connection_args(database), "--batch", "--skip-column-names"], + input=sql.encode("utf-8"), env=client_env(), capture_output=True) + if result.returncode != 0: + lines = result.stderr.decode("utf-8", "replace").strip().splitlines() + raise OpsError("查询失败：" + (lines[-1] if lines else "未知错误")) + output = result.stdout.decode("utf-8", "replace") + return [line.split("\t") for line in output.strip().splitlines() if line] + + +def scalar(database, sql, default=None): + rows = query(database, sql) + if not rows or not rows[0] or rows[0][0] == "NULL": + return default + return rows[0][0] + + +def database_exists(database): + rows = query("", "SELECT SCHEMA_NAME FROM information_schema.SCHEMATA WHERE SCHEMA_NAME='%s'" % database) + return bool(rows) + + +def sha256_of(path, chunk=1024 * 1024): + digest = hashlib.sha256() + with open(path, "rb") as handle: + while True: + block = handle.read(chunk) + if not block: + break + digest.update(block) + return digest.hexdigest() + + +def git_commit(): + try: + return subprocess.run(["git", "rev-parse", "HEAD"], cwd=str(ROOT), capture_output=True, text=True, + encoding="utf-8", errors="replace").stdout.strip() + except OSError: + return "" + + +def version_of(text): + """Pulls the version out of a client or server banner. + + A client banner reads "Ver 14.14 Distrib 5.7.38", where the first number is the protocol + version, so the version after "Distrib" is the one to compare. + """ + banner = text or "" + match = re.search(r"Distrib\s+(\d+)\.(\d+)(?:\.(\d+))?", banner) + if not match: + match = re.search(r"(\d+)\.(\d+)(?:\.(\d+))?", banner) + if not match: + return None + return tuple(int(part) for part in match.groups(default="0")) + + +def validate_database_name(name, action): + if not name or name in SYSTEM_SCHEMAS or "lexgo" not in name: + raise OpsError("拒绝在 %s 上执行：库名必须包含 lexgo 且不能是系统库。" % action) + return name + + +def row_counts(database): + counts = {} + present = {row[0] for row in query("information_schema", "SELECT TABLE_NAME FROM TABLES WHERE TABLE_SCHEMA='%s'" % database)} + for table in TABLES: + if table in present: + counts[table] = int(scalar(database, "SELECT COUNT(*) FROM `%s`" % table, "0")) + return counts + + +# ---------------------------------------------------------------- install-check + + +def command_install_check(_args): + load_env() + checks = [] + problems = [] + + def add(name, ok, detail): + checks.append((name, ok, detail)) + if not ok: + problems.append(name) + + # The client has to be at least as new as the server: an older mysqldump produces a dump the + # server cannot be restored from, which is worth catching before an incident rather than after. + try: + client = run([mysql_binary("mysql"), "--version"]).strip() + except OpsError as error: + client = str(error) + client_version = version_of(client) + add("MySQL 客户端存在", bool(client_version), client) + server = "" + database = os.environ.get("LEXGO_DB_NAME", "") + if database: + try: + server = scalar("", "SELECT VERSION()", "") or "" + except OpsError as error: + problems.append("数据库连接") + checks.append(("数据库连接", False, str(error))) + server_version = version_of(server) + if server_version: + add("MySQL 服务端为 8.x", server_version[0] == 8, server) + if client_version and server_version: + add("客户端版本不低于服务端", client_version >= server_version, + "客户端 " + ".".join(str(part) for part in client_version) + " / 服务端 " + ".".join(str(part) for part in server_version)) + add("环境配置已加载", bool(database), "LEXGO_DB_NAME=" + (database or "（未设置）")) + for key in ("LEXGO_DB_USER", "LEXGO_DB_PASSWORD", "LEXGO_BOOTSTRAP_USERNAME"): + add("配置项 " + key + " 存在", bool(os.environ.get(key)), "已设置" if os.environ.get(key) else "未设置") + if not os.environ.get("LEXGO_LISTEN"): + print(" [info] LEXGO_LISTEN 未设置，服务将使用内置默认监听地址。") + # The dictionary resource is pinned by digest and imported explicitly, never downloaded at + # runtime; the pin file has to be present for an install to be reproducible. + resource = ROOT / "server" / "wordnet-resource.json" + if resource.exists(): + pin = json.loads(resource.read_text(encoding="utf-8")) + add("WordNet 资源 pin", len(pin.get("sha256", "")) == 64 and bool(pin.get("source")), "sha256=" + pin.get("sha256", "")[:12] + "…") + else: + add("WordNet 资源 pin", False, "缺少 server/wordnet-resource.json") + for binary, label in (("go", "Go 工具链"), ("node", "Node"), ("pnpm", "pnpm")): + found = shutil.which(binary) + add(label + "（仅构建需要）", bool(found), found or "未安装：使用预构建产物时可不安装") + if os.environ.get("LEXGO_LISTEN", "").startswith("0.0.0.0"): + add("服务监听地址", False, "LEXGO_LISTEN 对外监听：" + os.environ["LEXGO_LISTEN"] + "，应由反向代理转发而不是直接暴露") + + for name, ok, detail in checks: + print((" [ok] " if ok else " [fail] ") + name + " — " + detail) + if problems: + print("\n检查未通过：" + "、".join(problems)) + print("按 wiki 的 Deployment-and-Operations 页面补齐后再安装。") + raise SystemExit(1) + print("\n依赖检查通过。") + if database: + try: + version = scalar(database, "SELECT version FROM lexgo_schema WHERE id=1", None) + print("当前 schema 版本：" + (version or "未知（尚未迁移）")) + except OpsError: + print("当前 schema 版本：无法读取（数据库可能尚未初始化）") + return 0 + + +# ---------------------------------------------------------------- init-database + + +def command_init_database(args): + """Creates the empty schema an installation starts from. + + The server cannot connect to a database that does not exist yet, so this is the first step of + a fresh install. Grants are printed instead of applied: which account may reach the schema is + a decision for the operator's database policy, not for this tool. + """ + load_env() + database = validate_database_name(args.database or os.environ.get("LEXGO_DB_NAME", ""), "建库") + if database_exists(database): + tables = int(scalar(database, "SELECT COUNT(*) FROM information_schema.TABLES WHERE TABLE_SCHEMA='%s'" % database, "0") or 0) + print("库已存在：" + database + "（" + str(tables) + " 张表）") + if tables: + print("这是已有实例；需要升级时下一步是 python scripts/server.py migrate。") + return 0 + else: + run([mysql_binary("mysql"), *connection_args(), "-e", "CREATE DATABASE `%s` CHARACTER SET utf8mb4" % database], env=client_env()) + print("已创建库：" + database) + print( + "\n请为应用账号授予该库的最小权限（只限此库，不要使用管理员账号）：\n" + " GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, INDEX, DROP, REFERENCES\n" + " ON `" + database + "`.* TO '<应用账号>'@'<来源主机>';\n" + " FLUSH PRIVILEGES;\n" + "\n下一步：写入 .env.local（凭据取自运维密码库）后执行\n" + " python scripts/server.py migrate\n" + " python scripts/server.py bootstrap" + ) + return 0 + + +# ---------------------------------------------------------------- backup + + +def command_backup(args): + load_env() + database = validate_database_name(args.database or os.environ.get("LEXGO_DB_NAME", ""), "备份") + out_dir = Path(args.out) if args.out else ROOT / ".local" / "backups" / datetime.now().strftime("%Y%m%d-%H%M%S") + if out_dir.exists() and any(out_dir.iterdir()) and not args.force: + raise OpsError("输出目录非空：" + str(out_dir) + "（加 --force 覆盖）") + out_dir.mkdir(parents=True, exist_ok=True) + + stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ") + sql_path = out_dir / ("lexgo-" + stamp + ".sql") + gz_path = Path(str(sql_path) + ".gz") + command = [ + mysql_binary("mysqldump"), *connection_args(), + "--single-transaction", "--routines", "--triggers", "--hex-blob", "--no-tablespaces", + "--default-character-set=utf8mb4", + # Deliberately without --databases: that option writes CREATE DATABASE and USE into the + # dump, which would send a restore straight back into the source schema instead of the + # target one. The client is given the target database instead. + database, + ] + print("正在备份 " + database + " …") + # The dump is captured and written as bytes: nothing about it is decoded or re-encoded on the + # way to disk, which keeps a binary-safe dump byte-exact. + dump = subprocess.run(command, env=client_env(), capture_output=True) + if dump.returncode != 0: + message = dump.stderr.decode("utf-8", "replace").strip().splitlines() + raise OpsError("mysqldump 失败：" + (message[-1] if message else "未知错误")) + with open(sql_path, "wb") as handle: + handle.write(dump.stdout) + with open(sql_path, "rb") as source, gzip.open(gz_path, "wb", compresslevel=6) as target: + shutil.copyfileobj(source, target) + sql_path.unlink() + + manifest = { + "created_at": stamp, + "database": database, + "product": "lexgo", + "schema_version": int(scalar(database, "SELECT version FROM lexgo_schema WHERE id=1", "0") or 0), + "git_commit": git_commit(), + "dump_file": gz_path.name, + "dump_sha256": sha256_of(gz_path), + "dump_bytes": gz_path.stat().st_size, + "row_counts": row_counts(database), + "mysql_client": run([mysql_binary("mysqldump"), "--version"]).strip(), + "mysql_server": scalar("", "SELECT VERSION()", ""), + "contents": "accounts, spaces, sessions, audit logs, books, chapters, ingest jobs, " + "dictionary archive, terms, review schedules, review answers, reading progress", + "excludes": "credentials and the environment file; back those up separately from the operations vault", + } + (out_dir / "manifest.json").write_text(json.dumps(manifest, ensure_ascii=False, indent=2), encoding="utf-8") + + print("备份完成：" + str(gz_path)) + print(" schema 版本 " + str(manifest["schema_version"]) + "，提交 " + (manifest["git_commit"][:8] or "未知")) + print(" 表行数 " + json.dumps(manifest["row_counts"], ensure_ascii=False)) + print(" sha256 " + manifest["dump_sha256"][:16] + "…（完整值见 manifest.json）") + print(" 凭据与 .env.local 不在其中，请用运维密码库单独保存。") + return 0 + + +# ---------------------------------------------------------------- restore + + +def ensure_empty_target(database, force): + if not database_exists(database): + run([mysql_binary("mysql"), *connection_args(), "-e", "CREATE DATABASE `%s` CHARACTER SET utf8mb4" % database], env=client_env()) + print("已创建空库 " + database) + return + existing = scalar(database, "SELECT COUNT(*) FROM information_schema.TABLES WHERE TABLE_SCHEMA='%s'" % database, "0") + if int(existing or 0) == 0: + return + if not force: + raise OpsError( + "目标库 " + database + " 已有 " + str(existing) + " 张表。恢复默认只写空库；" + "确认要覆盖时再加 --force，并先对现有库做一次备份。" + ) + print("警告：目标库已有数据，按 --force 覆盖。") + run([mysql_binary("mysql"), *connection_args(), "-e", + "DROP DATABASE `%s`; CREATE DATABASE `%s` CHARACTER SET utf8mb4" % (database, database)], env=client_env()) + + +def dump_targets_instead_of_source(dump): + """Reads the head of a dump and refuses one that would switch databases. + + A dump taken with ``mysqldump --databases`` carries CREATE DATABASE and USE statements, so a + restore would write into the schema named inside the file rather than the requested target. + An older backup is rejected instead of being trusted. + """ + with gzip.open(dump, "rb") as handle: + head = handle.read(256 * 1024).decode("utf-8", "replace") + return "CREATE DATABASE" in head.upper() or bool(re.search(r"^USE `", head, re.M)) + + +def dump_checksums(database): + """Per-table content checksums, used to prove a restore equals its source.""" + checksums = {} + for table in TABLES: + rows = query(database, "CHECKSUM TABLE `%s`" % table) + if rows and len(rows[0]) > 1: + checksums[table] = rows[0][1] + return checksums + + +def command_restore(args): + load_env() + if not args.confirm: + raise OpsError( + "恢复会写入数据库，需要显式确认：\n" + " python scripts/ops.py restore --dump <文件.sql.gz> --database <库名> --confirm\n" + "默认只恢复到空库；覆盖已有库还要加 --force。" + ) + dump = Path(args.dump) + if not dump.exists(): + raise OpsError("找不到备份文件：" + str(dump)) + database = validate_database_name(args.database, "恢复") + manifest = {} + manifest_path = dump.parent / "manifest.json" + if manifest_path.exists(): + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + expected = manifest.get("dump_sha256") + if expected and expected != sha256_of(dump): + raise OpsError("备份文件与 manifest 的 sha256 不一致，拒绝恢复。") + print("校验通过：备份文件与 manifest 的 sha256 一致。") + elif not args.skip_manifest_check: + raise OpsError("找不到 manifest.json，无法校验备份完整性（确认要继续时加 --skip-manifest-check）。") + if dump_targets_instead_of_source(dump): + raise OpsError( + "备份文件里带有 CREATE DATABASE / USE，恢复会写进文件里指定的库而不是目标库，拒绝执行。\n" + "请用本工具重新生成备份（它导出的是只含表数据的 dump）。" + ) + # The source database must not change: its checksums are compared again afterwards. + source_database = manifest.get("database", "") + source_before = dump_checksums(source_database) if source_database and source_database != database and database_exists(source_database) else {} + + ensure_empty_target(database, args.force) + print("正在恢复 " + dump.name + " 到 " + database + " …") + # The dump is decompressed to a temporary file and that file is handed to the client. Feeding + # a pipe from Python is unreliable on Windows for a stream this size, and a real file + # descriptor also means the client sees the plain SQL it expects. + work = Path(tempfile.mkdtemp(prefix="lexgo-restore-")) + plain = work / "dump.sql" + try: + with gzip.open(dump, "rb") as compressed, open(plain, "wb") as target: + shutil.copyfileobj(compressed, target) + with open(plain, "rb") as handle: + load = subprocess.run([mysql_binary("mysql"), *connection_args(database), "--default-character-set=utf8mb4"], + stdin=handle, env=client_env(), capture_output=True) + if load.returncode != 0: + message = load.stderr.decode("utf-8", "replace").strip().splitlines() + raise OpsError("恢复失败：" + (message[-1] if message else "未知错误")) + finally: + shutil.rmtree(work, ignore_errors=True) + print("恢复完成，开始校验。") + if source_before: + source_after = dump_checksums(source_database) + if source_after != source_before: + raise OpsError("源库 " + source_database + " 在校验过程中发生变化，已停止；请人工比对后再继续。") + print("源库 " + source_database + " 的内容校验和未变化。") + verify_args = argparse.Namespace(database=database, manifest=str(manifest_path) if manifest_path.exists() else "", + expect_schema=manifest.get("schema_version"), api="", user="", password_env="", + password_env_b="", skip_api=True) + return command_verify(verify_args) + + +# ---------------------------------------------------------------- verify + + +def integrity_checks(database): + checks = [] + + def add(name, ok, detail=""): + checks.append((name, ok, detail)) + + add("schema 表存在", int(scalar(database, "SELECT COUNT(*) FROM information_schema.TABLES WHERE TABLE_SCHEMA='%s' AND TABLE_NAME='lexgo_schema'" % database, "0") or 0) == 1) + add("产品标识为 lexgo", scalar(database, "SELECT product FROM lexgo_schema WHERE id=1", "") == "lexgo") + add("没有孤立章节（章节属于其书）", int(scalar(database, "SELECT COUNT(*) FROM lexgo_chapters c LEFT JOIN lexgo_books b ON b.id=c.book_id WHERE b.id IS NULL", "0") or 0) == 0) + add("章节归属与书归属一致", int(scalar(database, "SELECT COUNT(*) FROM lexgo_chapters c JOIN lexgo_books b ON b.id=c.book_id WHERE c.owner_id<>b.owner_id", "0") or 0) == 0) + add("每个词条都有排期行", int(scalar(database, "SELECT COUNT(*) FROM lexgo_terms t LEFT JOIN lexgo_term_reviews r ON r.term_id=t.id WHERE r.term_id IS NULL", "0") or 0) == 0) + add("排期行都指向存在的词条", int(scalar(database, "SELECT COUNT(*) FROM lexgo_term_reviews r LEFT JOIN lexgo_terms t ON t.id=r.term_id WHERE t.id IS NULL", "0") or 0) == 0) + add("复习记录引用有效词条", int(scalar(database, "SELECT COUNT(*) FROM lexgo_review_answers a LEFT JOIN lexgo_terms t ON t.id=a.term_id WHERE t.id IS NULL", "0") or 0) == 0) + add("等级只出现在学习中词条", int(scalar(database, "SELECT COUNT(*) FROM lexgo_terms WHERE (status='learning' AND (level<1 OR level>7)) OR (status<>'learning' AND level<>0)", "0") or 0) == 0) + add("完成记录都指向存在的章节", int(scalar(database, "SELECT COUNT(*) FROM lexgo_chapter_progress p LEFT JOIN lexgo_chapters c ON c.id=p.chapter_id WHERE c.id IS NULL", "0") or 0) == 0) + add("词条语言与所属空间一致", int(scalar(database, "SELECT COUNT(*) FROM lexgo_terms t JOIN lexgo_spaces s ON s.owner_id=t.owner_id WHERE t.language<>s.language", "0") or 0) == 0) + + # The audit tables must not gain a column that could hold a credential or private content. + for table in ("lexgo_login_logs", "lexgo_operation_logs"): + columns = {row[0].lower() for row in query("information_schema", "SELECT COLUMN_NAME FROM COLUMNS WHERE TABLE_SCHEMA='%s' AND TABLE_NAME='%s'" % (database, table))} + banned = sorted(columns & AUDIT_BANNED_COLUMNS) + add("审计表 " + table + " 不含敏感列", not banned, "命中：" + ",".join(banned) if banned else "列名白名单通过") + return checks + + +class Api: + """A very small API client: the operations tool must not depend on the test harness.""" + + def __init__(self, base): + self.base = base.rstrip("/") + + def call(self, method, path, token=None, body=None): + import urllib.error + import urllib.request + + request = urllib.request.Request(self.base + "/api/v1/" + path, method=method) + if token: + request.add_header("Authorization", "Bearer " + token) + payload = None + if body is not None: + payload = json.dumps(body).encode("utf-8") + request.add_header("Content-Type", "application/json") + try: + with urllib.request.urlopen(request, payload, timeout=60) as response: + return response.status, json.loads(response.read().decode("utf-8")) + except urllib.error.HTTPError as error: + try: + return error.code, json.loads(error.read().decode("utf-8")) + except Exception: + return error.code, {} + + def login(self, account, password): + status, payload = self.call("POST", "login", body={"username": account, "password": password}) + return status, payload.get("data", {}).get("token", "") + + +def report(checks): + """Prints one line per check and fails the command when any of them failed.""" + failures = [name for name, ok, _ in checks if not ok] + for name, ok, detail in checks: + print((" [ok] " if ok else " [fail] ") + name + ("— " + detail if detail else "")) + if failures: + print("\n检查未通过 " + str(len(failures)) + " 项：" + "、".join(failures)) + raise SystemExit(1) + print("\n检查通过：" + str(len(checks)) + " 项全部成功。") + return 0 + + +def api_walk(api, user, password_a, password_b=None): + """Two accounts walk the learning loop on the restored instance and one must not see the other.""" + checks = [] + client = Api(api) + + def add(name, ok, detail=""): + checks.append((name, ok, detail)) + + status, token_a = client.login(user + "_a", password_a) + add("账号 " + user + "_a 可以登录恢复实例", status == 200 and bool(token_a)) + status, token_b = client.login(user + "_b", password_b or password_a) + add("账号 " + user + "_b 可以登录恢复实例", status == 200 and bool(token_b)) + if not token_a or not token_b: + return checks + + status, books_a = client.call("GET", "books", token_a) + items_a = books_a.get("data", {}).get("items", []) + add("账号 A 能看到自己的书库", status == 200 and len(items_a) > 0, str(len(items_a)) + " 本书") + + status, progress_a = client.call("GET", "progress", token_a) + add("账号 A 能读到自己的进度", status == 200, json.dumps(progress_a.get("data", {}), ensure_ascii=False)[:120]) + + # Isolation: ids that belong to A must not resolve for B. + if items_a: + book_id = items_a[0]["id"] + status, _ = client.call("GET", "books/%d" % book_id, token_b) + add("B 读取 A 的书籍被拒绝", status == 404, "HTTP " + str(status)) + status, detail = client.call("GET", "books/%d" % book_id, token_a) + chapters = [chapter for chapter in detail.get("data", {}).get("chapters", []) if chapter["status"] == "ready"] + if chapters: + chapter_id = chapters[0]["id"] + status, _ = client.call("GET", "chapters/%d" % chapter_id, token_b) + add("B 读取 A 的章节被拒绝", status == 404, "HTTP " + str(status)) + # A completes the chapter twice: the mark must stay idempotent after a restore. + status, _ = client.call("POST", "chapters/%d/complete" % chapter_id, token_a) + status2, second = client.call("POST", "chapters/%d/complete" % chapter_id, token_a) + duplicate = second.get("data", {}).get("progress", {}).get("duplicate") + add("恢复后完成章节仍然幂等", status == 200 and status2 == 200 and bool(duplicate), "duplicate=" + str(duplicate)) + status, _ = client.call("POST", "chapters/%d/complete" % chapter_id, token_b) + add("B 不能完成 A 的章节", status == 404, "HTTP " + str(status)) + + # The review loop works on restored schedules. + status, queue = client.call("GET", "reviews/queue", token_a) + items = queue.get("data", {}).get("items", []) + add("账号 A 的到期队列可读", status == 200, "到期 " + str(queue.get("data", {}).get("total")) + " 条") + if items: + item = items[0] + answer = {"answerId": "ops-verify-restore-0001", "grade": "correct", "expectedDueAt": item["dueAt"]} + status, result = client.call("POST", "reviews/%d/answers" % item["id"], token_a, answer) + add("恢复后答题成功并更新排期", status in (200, 201) and result.get("data", {}).get("result") == "applied", + "level " + str(result.get("data", {}).get("levelAfter"))) + status, replay = client.call("POST", "reviews/%d/answers" % item["id"], token_a, answer) + add("重复提交同一答案不重复记账", status in (200, 201) and bool(replay.get("data", {}).get("duplicate")), + "duplicate=" + str(replay.get("data", {}).get("duplicate"))) + status, _ = client.call("POST", "reviews/%d/answers" % item["id"], token_b, + {"answerId": "ops-verify-restore-0002", "grade": "correct", "expectedDueAt": item["dueAt"]}) + add("B 不能给 A 的词条答题", status == 404, "HTTP " + str(status)) + return checks + + +def command_smoke(args): + """Walks the whole learning loop on a freshly installed instance with two trial accounts.""" + load_env() + client = Api(args.api) + password = os.environ.get(args.password_env, "") + if not password: + raise OpsError("环境变量 " + args.password_env + " 为空：请先为演练账号设置密码。") + checks = [] + + def add(name, ok, detail=""): + checks.append((name, ok, detail)) + + # 1. The first administrator exists and can log in: the installation is usable. + status, admin_token = client.login(args.admin_user, os.environ.get(args.admin_password_env, "")) + add("初始管理员可以登录", status == 200 and bool(admin_token)) + if not admin_token: + return report(checks) + + # 2. Two trial learners, created by the administrator. There is no self-registration. + for suffix in ("a", "b"): + account = args.user + "_" + suffix + status, payload = client.call("POST", "accounts", admin_token, {"username": account, "password": password}) + if status == 201: + add("管理员创建演练账号 " + account, True, "HTTP 201") + else: + # An existing account is fine on a rerun; the password was set the first time. + existing = client.login(account, password)[0] == 200 + add("演练账号 " + account + " 已存在且可登录", existing, payload.get("msg", "HTTP " + str(status))) + + status, token_a = client.login(args.user + "_a", password) + status2, token_b = client.login(args.user + "_b", password) + add("两个演练账号都能登录", status == 200 and status2 == 200 and bool(token_a) and bool(token_b)) + if not token_a or not token_b: + return report(checks) + + add("新账号的学习空间是空的", client.call("GET", "books", token_a)[1].get("data", {}).get("items", []) == []) + add("新账号的进度从零开始", client.call("GET", "progress", token_a)[1].get("data", {}).get("totalChapters") == 0) + + # 3. The learning loop on the new instance: paste, read, complete, save a word, review it. + text = "Curiosity opens the first door.\nThe second door stays closed.\n" * 6 + status, pasted = client.call("POST", "books", token_a, { + "requestId": args.request_id, "title": "Fictional trial chapter", "text": text, "language": "en"}) + add("演练账号可以粘贴章节", status == 201, "HTTP " + str(status)) + if status != 201: + return report(checks) + book_id = pasted["data"]["book"]["id"] + chapter_id = pasted["data"]["chapter"]["id"] + ready = {} + for _ in range(40): + status, detail = client.call("GET", "chapters/%d" % chapter_id, token_a) + ready = detail.get("data", {}).get("chapter", {}) + if ready.get("status") == "ready": + break + time.sleep(0.5) + add("章节在干净实例上处理完成", ready.get("status") == "ready", str(ready.get("status"))) + if ready.get("status") != "ready": + return report(checks) + + status, tokens = client.call("GET", "chapters/%d/tokens" % chapter_id, token_a) + words = [token for token in tokens.get("data", {}).get("tokens", []) if token.get("kind") == "word"] + add("章节可以分词并返回可点选的词", status == 200 and len(words) > 3, str(len(words)) + " 个词") + + status, _ = client.call("GET", "terms/lookup?query=curiosity", token_a) + add("词典查询返回明确状态（含资源缺失时的降级）", status in (200, 404, 503), "lookup HTTP " + str(status)) + + if words: + word = words[0] + status, saved = client.call("POST", "terms", token_a, { + "chapterId": chapter_id, "start": word["start"], "end": word["end"], + "definition": "虚构释义", "status": "new"}) + add("演练账号可以保存自己的词义", status in (200, 201), "HTTP " + str(status)) + term_id = saved.get("data", {}).get("term", {}).get("id") + if term_id: + other_terms = client.call("GET", "terms?query=curiosity", token_b)[1].get("data", {}).get("total") + add("另一个账号看不到该词条", other_terms == 0, "B 查到 " + str(other_terms) + " 条") + queue = client.call("GET", "reviews/queue", token_a)[1] + items = queue.get("data", {}).get("items", []) + due = [entry for entry in items if entry["id"] == term_id] + add("新保存的词条立刻到期", bool(due), "到期 " + str(queue.get("data", {}).get("total"))) + if due: + answer = {"answerId": args.answer_id, "grade": "correct", "expectedDueAt": due[0]["dueAt"]} + status, result = client.call("POST", "reviews/%d/answers" % term_id, token_a, answer) + add("到期复习可以作答", status in (200, 201) and result.get("data", {}).get("result") == "applied", + "level " + str(result.get("data", {}).get("levelAfter"))) + replay = client.call("POST", "reviews/%d/answers" % term_id, token_a, answer)[1] + add("重复作答不重复记账", replay.get("data", {}).get("duplicate") is True) + + status, completed = client.call("POST", "chapters/%d/complete" % chapter_id, token_a) + add("完成章节只记已读", status == 200 and completed.get("data", {}).get("progress", {}).get("read") is True) + + data = client.call("GET", "progress", token_a)[1].get("data", {}) + add("进度反映新实例上的活动", data.get("readChapters", 0) >= 1 and data.get("savedTerms", 0) >= 1, + json.dumps({key: data.get(key) for key in ("readChapters", "totalChapters", "knownTerms", "learningTerms", "newTerms", "dueNow")}, ensure_ascii=False)) + + add("B 无法读取 A 的书籍", client.call("GET", "books/%d" % book_id, token_b)[0] == 404) + add("B 的书库仍然为空", client.call("GET", "books", token_b)[1].get("data", {}).get("items", []) == []) + return report(checks) + + +def command_verify(args): + load_env() + database = validate_database_name(args.database or os.environ.get("LEXGO_DB_NAME", ""), "校验") + failures = [] + print("校验数据库 " + database) + if not database_exists(database): + raise OpsError("数据库不存在：" + database) + version = int(scalar(database, "SELECT version FROM lexgo_schema WHERE id=1", "0") or 0) + if version != SCHEMA_VERSION: + print(" [warn] schema 版本 " + str(version) + "，本工具期望 " + str(SCHEMA_VERSION) + "（升级或回退阶段属正常）") + + checks = integrity_checks(database) + if args.manifest: + manifest = json.loads(Path(args.manifest).read_text(encoding="utf-8")) + actual = row_counts(database) + for table, expected in manifest.get("row_counts", {}).items(): + got = actual.get(table, 0) + checks.append(("表 " + table + " 行数与备份一致", got == expected, "备份 " + str(expected) + " / 现在 " + str(got))) + for name, ok, detail in checks: + print((" [ok] " if ok else " [fail] ") + name + ("— " + detail if detail else "")) + if not ok: + failures.append(name) + + if args.api: + print("校验接口 " + args.api + "（账号前缀 " + args.user + "）") + password = os.environ.get(args.password_env, "") + if not password: + raise OpsError("环境变量 " + args.password_env + " 为空，无法登录演练账号。") + password_b = os.environ.get(args.password_env_b, "") or password + for name, ok, detail in api_walk(args.api, args.user, password, password_b): + print((" [ok] " if ok else " [fail] ") + name + ("— " + detail if detail else "")) + if not ok: + failures.append(name) + + if failures: + print("\n校验未通过 " + str(len(failures)) + " 项：" + "、".join(failures)) + raise SystemExit(1) + print("\n校验通过。") + return 0 + + +def build_parser(): + parser = argparse.ArgumentParser(description="LexGo 运维工具：依赖检查、备份、恢复与校验") + sub = parser.add_subparsers(dest="command", required=True) + + install = sub.add_parser("install-check", help="只读检查依赖与资源版本") + install.set_defaults(func=command_install_check) + + init = sub.add_parser("init-database", help="为全新安装创建空库并提示所需权限") + init.add_argument("--database", default="", help="要创建的库名，默认取 LEXGO_DB_NAME") + init.set_defaults(func=command_init_database) + + backup = sub.add_parser("backup", help="导出全库并写出 manifest") + backup.add_argument("--out", default="", help="输出目录，默认 .local/backups/<时间戳>") + backup.add_argument("--database", default="", help="要备份的库，默认取 LEXGO_DB_NAME") + backup.add_argument("--force", action="store_true", help="输出目录非空时覆盖") + backup.set_defaults(func=command_backup) + + restore = sub.add_parser("restore", help="把备份恢复到空库（覆盖已有库要 --force）") + restore.add_argument("--dump", required=True, help="backup 生成的 .sql.gz") + restore.add_argument("--database", required=True, help="目标库名，必须包含 lexgo") + restore.add_argument("--confirm", action="store_true", help="确认执行写入") + restore.add_argument("--force", action="store_true", help="目标库已有数据时覆盖") + restore.add_argument("--skip-manifest-check", action="store_true", help="没有 manifest 时跳过完整性校验") + restore.set_defaults(func=command_restore) + + verify = sub.add_parser("verify", help="校验恢复结果的完整性与两账号隔离") + verify.add_argument("--database", default="", help="要校验的库，默认取 LEXGO_DB_NAME") + verify.add_argument("--manifest", default="", help="与备份的行数逐表比对") + verify.add_argument("--expect-schema", type=int, default=None) + verify.add_argument("--api", default="", help="例如 http://127.0.0.1:8010 ，启用两账号闭环校验") + verify.add_argument("--user", default="", help="演练账号前缀，实际账号为 <前缀>_a 与 <前缀>_b") + verify.add_argument("--password-env", default="LEXGO_TRIAL_PASSWORD", help="存放演练账号 _a 密码的环境变量名") + verify.add_argument("--password-env-b", default="", help="存放演练账号 _b 密码的环境变量名，默认与 _a 相同") + verify.add_argument("--skip-api", action="store_true") + verify.set_defaults(func=command_verify) + + smoke = sub.add_parser("smoke", help="在干净实例上创建两个演练账号并走通学习闭环") + smoke.add_argument("--api", required=True, help="例如 http://127.0.0.1:8010") + smoke.add_argument("--admin-user", required=True, help="初始管理员账号") + smoke.add_argument("--admin-password-env", default="LEXGO_BOOTSTRAP_PASSWORD", help="初始管理员密码所在的环境变量名") + smoke.add_argument("--user", required=True, help="演练账号前缀，实际账号为 <前缀>_a 与 <前缀>_b") + smoke.add_argument("--password-env", default="LEXGO_TRIAL_PASSWORD", help="演练账号密码所在的环境变量名") + smoke.add_argument("--request-id", default="ops-smoke-0001", help="粘贴章节的幂等请求号") + smoke.add_argument("--answer-id", default="ops-smoke-answer-0001", help="复习作答的幂等编号") + smoke.set_defaults(func=command_smoke) + return parser + + +def main(argv=None): + args = build_parser().parse_args(argv) + try: + return args.func(args) + except OpsError as error: + print("错误：" + str(error), file=sys.stderr) + return 2 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/test_lexgo_ops.py b/tests/test_lexgo_ops.py new file mode 100644 index 0000000..569823b --- /dev/null +++ b/tests/test_lexgo_ops.py @@ -0,0 +1,90 @@ +"""Unit tests for the operations helper rules that must hold without a database. + +The database paths (backup, restore, verify) are exercised by the #15 drill against real MySQL; +what is checked here is the logic that decides whether an operation is allowed at all. +""" +import sys +import unittest +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / "scripts")) + +import ops # noqa: E402 + + +class VersionParsingTests(unittest.TestCase): + def test_reads_the_version_after_distrib(self): + # A client banner's first number is the protocol version, not the server version. + self.assertEqual(ops.version_of("mysql Ver 14.14 Distrib 5.7.38, for Win64 (x86_64)"), (5, 7, 38)) + self.assertEqual(ops.version_of("mysql Ver 8.4.3 for Win64 on x86_64"), (8, 4, 3)) + self.assertIsNone(ops.version_of("")) + self.assertIsNone(ops.version_of("no version here")) + + def test_compares_client_and_server_versions(self): + self.assertLess(ops.version_of("Ver 14.14 Distrib 5.7.38"), ops.version_of("8.4.3")) + self.assertGreaterEqual(ops.version_of("Ver 8.0.36"), ops.version_of("8.4.3".replace("8.4.3", "8.0.36"))) + self.assertGreaterEqual(ops.version_of("8.4.3"), ops.version_of("8.4.3")) + + +class DatabaseNameTests(unittest.TestCase): + def test_accepts_lexgo_names(self): + for name in ("lexgo_dev", "lexgo_prod", "lexgo_restore_drill"): + self.assertEqual(ops.validate_database_name(name, "测试"), name) + + def test_refuses_system_and_unrelated_names(self): + for name in ("mysql", "information_schema", "performance_schema", "sys", "app_prod", ""): + with self.assertRaises(SystemExit): + ops.validate_database_name(name, "测试") + + +class DumpSafetyTests(unittest.TestCase): + """A dump that carries CREATE DATABASE/USE would restore into the wrong schema.""" + + def _write_dump(self, body): + import gzip + import tempfile + + handle = tempfile.NamedTemporaryFile(delete=False, suffix=".sql.gz") + handle.close() + with gzip.open(handle.name, "wb") as target: + target.write(body.encode("utf-8")) + self.addCleanup(Path(handle.name).unlink) + return Path(handle.name) + + def test_flags_a_dump_that_switches_database(self): + dump = self._write_dump("-- MySQL dump\nCREATE DATABASE /*!32312 IF NOT EXISTS*/ `lexgo_dev`;\nUSE `lexgo_dev`;\nCREATE TABLE t (id INT);\n") + self.assertTrue(ops.dump_targets_instead_of_source(dump)) + + def test_flags_a_dump_with_use_only(self): + dump = self._write_dump("-- MySQL dump\nUSE `lexgo_dev`;\nCREATE TABLE t (id INT);\n") + self.assertTrue(ops.dump_targets_instead_of_source(dump)) + + def test_accepts_a_table_only_dump(self): + dump = self._write_dump("-- MySQL dump\nDROP TABLE IF EXISTS `lexgo_terms`;\nCREATE TABLE `lexgo_terms` (id INT);\nINSERT INTO `lexgo_terms` VALUES (1);\n") + self.assertFalse(ops.dump_targets_instead_of_source(dump)) + + +class ManifestTests(unittest.TestCase): + def test_manifest_keys_never_name_a_credential(self): + """The manifest is written to disk, so its field names are checked, not its prose.""" + import re + + source = (ROOT / "scripts" / "ops.py").read_text(encoding="utf-8") + block = source[source.index(" manifest = {"):source.index('(out_dir / "manifest.json")')] + keys = re.findall(r'^\s+"([a-z_]+)":', block, re.M) + self.assertIn("schema_version", keys) + self.assertIn("row_counts", keys) + for key in keys: + for banned in ("password", "secret", "token", "credential"): + self.assertNotIn(banned, key.lower(), key) + + def test_audit_tables_declare_no_sensitive_column(self): + """The integrity check must keep naming the columns that may not appear in the audit logs.""" + self.assertTrue({"password", "token", "body", "content"} <= ops.AUDIT_BANNED_COLUMNS) + self.assertIn("lexgo_login_logs", ops.TABLES) + self.assertIn("lexgo_operation_logs", ops.TABLES) + + +if __name__ == "__main__": + unittest.main() -- 2.34.1 From 420365fc6b4bee30234805494ea3836b0c082c76 Mon Sep 17 00:00:00 2001 From: QiuSW Date: Tue, 15 Sep 2026 17:11:26 +0800 Subject: [PATCH 3/4] =?UTF-8?q?feat:=20=E5=A4=87=E4=BB=BD=E3=80=81?= =?UTF-8?q?=E6=81=A2=E5=A4=8D=E4=B8=8E=E6=A0=A1=E9=AA=8C=E6=8F=90=E4=BE=9B?= =?UTF-8?q?=20Go=20=E5=AD=90=E5=91=BD=E4=BB=A4=EF=BC=8C=E4=BA=A4=E4=BB=98?= =?UTF-8?q?=E6=94=AF=E6=8C=81=E7=BA=AF=E4=BA=8C=E8=BF=9B=E5=88=B6=E5=AE=89?= =?UTF-8?q?=E8=A3=85=20(#15)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - server/cmd/lexgo 增加 backup / restore / verify 子命令，与既有 migrate/bootstrap/serve 同一入口；备份仍调用 mysqldump，dump 与 manifest 格式和工具通道完全一致、可互相读取 - server/app/lexgo/ops.go：manifest、库名校验、版本解析、dump 库名切换检测、表行数与 内容校验和、目标库准备、恢复前后源库校验、完整性校验（客户不选 schema，SQL 显式限定库名） - ops_test.go：7 项无库规则测试与 1 项 MySQL 备份→恢复→校验集成用例 - 部署页补充纯二进制路径（lexgo migrate/bootstrap/backup/restore/verify）、两条通道的关系 与「接口级两账号闭环仍在工具通道」的边界；Architecture/LocalDev/Home/README/AGENTS 同步 - gofmt 修正 #13 遗留的 library.go 字段对齐 --- AGENTS.md | 1 + README.md | 1 + docs/02-architecture-and-code-map.md | 25 +- docs/11-deployment-and-operations.md | 45 +- docs/README.md | 6 +- server/app/lexgo/library.go | 6 +- server/app/lexgo/ops.go | 600 +++++++++++++++++++++++++++ server/app/lexgo/ops_test.go | 287 +++++++++++++ server/cmd/lexgo/main.go | 176 +++++++- 9 files changed, 1125 insertions(+), 22 deletions(-) create mode 100644 server/app/lexgo/ops.go create mode 100644 server/app/lexgo/ops_test.go diff --git a/AGENTS.md b/AGENTS.md index 577eb2a..aa30d57 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -27,6 +27,7 @@ | 创建可选任务快照 | `python dev_scripts/harness.py archive 123 "修复登录超时"` | | 增量导出已有快照 | `python dev_scripts/harness.py export` | | 全量导出已有快照 | `python dev_scripts/harness.py export --all` | +| 后端二进制运维 | `./server/lexgo migrate\|bootstrap\|serve\|backup\|restore\|verify`（部署机无需 Python） | | 检查部署依赖 | `python scripts/ops.py install-check` | | 备份实例 | `python scripts/ops.py backup --out <目录>` | | 恢复到空库 | `python scripts/ops.py restore --dump <备份.sql.gz> --database <库名> --confirm` | diff --git a/README.md b/README.md index 4d3882e..8aebe1e 100644 --- a/README.md +++ b/README.md @@ -11,6 +11,7 @@ - [工作量估算](docs/10-workload-estimate.md)：#2、#3、#4、#5、#6、#7、#8、#9、#10、#11、#12、#13、#14、#18 已验收，原规划中的 #5 已完成；#15 已实现待验收；新增 #21、#24 待排期；缺陷 #32 待确认修复方案；后续结合集成结果重估，旧全量研究仅供参考。 - [四阶段实施总览 #16](https://git.ilapage.cn/OPC/lexgo/issues/16)：14 张单元工单，工程基础 → 技术验证 → 首条学习闭环 → 补齐 MVP；原型 v1 已获用户验收。两端使用账号（用户名）＋密码登录，不要求邮箱。 - [原型工单 #1](https://git.ilapage.cn/OPC/lexgo/issues/1)：Quant-UX 桌面/手机原型 v1，预览入口与审核记录见工单及需求总览。 +- 运维命令：`./lexgo migrate|bootstrap|serve|backup|restore|verify`（纯二进制），开发便利与交叉验证走 `python scripts/server.py …` / `python scripts/ops.py …` - [工作流](docs/01-workflow.md) · [开发与验证](docs/04-local-development-and-verification.md) · [部署与运维](docs/11-deployment-and-operations.md) ## 本地工程基础 diff --git a/docs/02-architecture-and-code-map.md b/docs/02-architecture-and-code-map.md index 606f7b2..98c342a 100644 --- a/docs/02-architecture-and-code-map.md +++ b/docs/02-architecture-and-code-map.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件) wiki_page: Architecture-and-Code-Map wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Architecture-and-Code-Map.- -wiki_revision: bd4a0e705abe722f153a7410435def91a21d472a -synchronized_at: 2026-09-15T08:27:37Z +wiki_revision: fabdaf13861b4cae9ad6cd66a5bd904c08361f73 +synchronized_at: 2026-09-15T09:08:48Z # 架构与代码地图 @@ -389,3 +389,24 @@ schema v7 新增 `lexgo_chapter_progress`：一章一行（`chapter_id` 主键 **部署文档**：新建 Wiki 页 `Deployment-and-Operations`（`docs/11-deployment-and-operations.md`，按 `docs/templates/deployment.md` 结构），并加入 `wiki-docs.json` 映射。内容含服务概览、环境要求（含「MySQL 客户端版本不得低于服务端」这条实测规则）、首次部署、配置与凭据来源、日常运维、健康检查、升级与回滚、备份与恢复、已知限制。 **视图与进程**：后端单二进制监听 `127.0.0.1:8000`；两份 SPA 由反向代理托管 `dist`，反代把 `/api/` 转发到后端并把未知路径回落到 `index.html`；本机开发用 supervisor 托管 `lexgo-api`/`lexgo-learner`/`lexgo-admin` 三个 program。 + +## #15 自托管交付：部署拓扑与运维工具（2026-09-15） + +**没有 schema 变化、没有接口变化**：本单新增运维工具、部署文档与完整演练证据。 + +| 新增文件 | 职责 | +|---|---| +| `server/cmd/lexgo`（追加子命令） | `backup`、`restore`、`verify`：与既有 `migrate`/`bootstrap`/`serve`/`audit-cleanup` 同一入口，部署机只需要二进制与 MySQL 客户端；备份仍调用 `mysqldump`，dump 与 manifest 格式与工具通道完全一致，可互相读取 | +| `scripts/ops.py` | `install-check`（依赖与资源版本）、`init-database`（建空库并提示最小权限）、`backup`（全库 dump ＋ `manifest.json`）、`restore`（默认只写空库，写后自动校验）、`verify`（数据库完整性 ＋ 可选两账号接口闭环）、`smoke`（在干净实例上建两个演练账号走通学习闭环） | +| `scripts/bench.py` | 在写明规模的人造数据集上测量接口耗时，输出数据量、机器信息与 p50/p95，不做容量承诺 | +| `tests/test_lexgo_ops.py` | 不需要数据库的规则测试：版本号解析、库名白名单、dump 是否带库名切换、manifest 字段白名单、审计敏感列清单 | + +**一次备份包含什么**：LexGo 把所有持久数据都放在 MySQL 里——账号与会话、书籍与章节原文、导入任务、词典归档（LONGBLOB）、个人词条、复习排期与作答、阅读进度、审计日志。因此备份 = 全库 dump ＋ `.env.local`（凭据单独从运维密码库取）。词典资源在显式导入后进入数据库，恢复即带走，运行时不下载。 + +**恢复的安全边界**：`restore` 必须带 `--confirm`；库名必须含 `lexgo` 且不能是系统库；目标库已有数据时默认拒绝，覆盖需要 `--force`；拒绝加载带 `CREATE DATABASE`/`USE` 的旧式 dump（那会把数据写进文件里指定的库）；恢复前后比对**源库**逐表内容校验和，源库被改动就中止。 + +**部署文档**：新建 Wiki 页 `Deployment-and-Operations`（`docs/11-deployment-and-operations.md`，按 `docs/templates/deployment.md` 结构），并加入 `wiki-docs.json` 映射。内容含服务概览、环境要求（含「MySQL 客户端版本不得低于服务端」这条实测规则）、首次部署、配置与凭据来源、日常运维、健康检查、升级与回滚、备份与恢复、已知限制。 + +**两条通道**：Go 二进制提供数据库层的备份、恢复与校验（`lexgo backup|restore|verify`），Python 工具提供依赖检查、建库与 HTTP 接口级的两账号闭环（`ops.py smoke` / `verify --api`）。两者共用同一份 `LEXGO_*` 配置、同一套 manifest 与安全规则；2026-09-15 交叉验证两条通道可互相恢复对方的备份。 + +**视图与进程**：后端单二进制监听 `127.0.0.1:8000`；两份 SPA 由反向代理托管 `dist`，反代把 `/api/` 转发到后端并把未知路径回落到 `index.html`；本机开发用 supervisor 托管 `lexgo-api`/`lexgo-learner`/`lexgo-admin` 三个 program。 diff --git a/docs/11-deployment-and-operations.md b/docs/11-deployment-and-operations.md index ba2af3c..ae51f93 100644 --- a/docs/11-deployment-and-operations.md +++ b/docs/11-deployment-and-operations.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件) wiki_page: Deployment-and-Operations wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Deployment-and-Operations.- -wiki_revision: 2b08a33c0c246841deb52450d7fa411d76dc2eee -synchronized_at: 2026-09-15T08:25:07Z +wiki_revision: 3791ef4adc5c5cc8d6e5006c719cd0ddccb15fc0 +synchronized_at: 2026-09-15T09:08:49Z # 部署与运维 @@ -39,6 +39,8 @@ synchronized_at: 2026-09-15T08:25:07Z | 组件 | 版本要求 | 检查命令 | 预期结果 | |---|---|---|---| | 操作系统 | Windows 或 Linux | — | 本项目在 Windows 开发机验证；Linux 步骤为等价命令 | +| LexGo 二进制 | 与数据库 schema 版本匹配 | `./lexgo verify --database <库名>` | 校验通过；它自身会拒绍非 MySQL 8 | +| Python | 3.8+，**仅工具通道需要** | `python scripts/ops.py install-check` | 纯二进制路径（`lexgo ...`）不需要 Python；Python 通道用于开发便利与交叉验证 | | MySQL 服务端 | 8.x | `mysql --version`（服务端 `SELECT VERSION()`） | 8.4.3 已验证 | | MySQL 客户端 | **不低于服务端** | `python scripts/ops.py install-check` | 「客户端版本不低于服务端」为 ok；5.7 客户端连 8.4 服务端会被判 fail | | Go | 1.26.5 | `go version` | 供 `scripts/server.py` 固定工具链构建 | @@ -70,12 +72,21 @@ python scripts/ops.py init-database --database lexgo_prod ### 3. 迁移与初始化管理员 +纯二进制路径（推荐，部署机只需要二进制与 MySQL 客户端）： + ```bash -python scripts/server.py migrate # 只有这一步会改表结构 -python scripts/server.py bootstrap # 首次建立唯一管理员 +./lexgo migrate # 只有这一步会改表结构 +./lexgo bootstrap # 首次建立唯一管理员 ``` -预期结果：`migrate` 输出 schema 版本；`bootstrap` 输出创建成功。**库中已有账号时 `bootstrap` 会拒绝执行**，不会覆盖既有管理员。空库不含默认密码与任何演示数据。 +开发便利路径（等价，额外做两件事：加载 `.env.local`、固定 Go 工具链）： + +```bash +python scripts/server.py migrate +python scripts/server.py bootstrap +``` + +预期结果：`migrate` 输出 schema 版本；`bootstrap` 输出创建成功。**库中已有账号时 `bootstrap` 会拒绝执行**，不会覆盖既有管理员。空库不含默认密码与任何演示数据。两种路径共用同一份 `LEXGO_*` 配置。 ### 4. 构建产物 @@ -114,8 +125,8 @@ cd ../admin && pnpm install && pnpm run build | 查看状态 | `supervisorctl status lexgo-api` | `RUNNING`，uptime 持续增长 | | 重启后端 | `supervisorctl restart lexgo-api` | `stopped` 后 `started` | | 查看日志 | `supervisorctl tail -f lexgo-api stderr` | 持续输出应用日志 | -| 备份 | `python scripts/ops.py backup --out <目录>` | 生成 `lexgo-<时间戳>.sql.gz` 与 `manifest.json` | -| 校验实例 | `python scripts/ops.py verify --database <库名>` | 打印每项检查结果，最后「校验通过」 | +| 备份 | `./lexgo backup --out <目录>` | 生成 `lexgo-<时间戳>.sql.gz` 与 `manifest.json` | +| 校验实例 | `./lexgo verify --database <库名>` | 打印每项检查结果，最后「verification passed」 | | 清理过期审计 | `python scripts/server.py audit-cleanup` | 只清理两张审计表中超过 90 天的记录 | | 关闭确认 | `supervisorctl stop lexgo-api` | 服务停止；学习端与反代仍在，接口不可用 | @@ -167,20 +178,28 @@ supervisorctl restart lexgo-api - 建议频率：每次升级或有迁移前必须备份；日常按使用强度自行决定（本项目未启用定时任务）。保留份数与存放位置由运维决定。 ```bash -python scripts/ops.py backup --out <备份目录> +./lexgo backup --out <备份目录> ``` 预期结果：目录中出现 `lexgo-<时间戳>.sql.gz` 与 `manifest.json`（schema 版本、提交哈希、逐表行数、dump 的 sha256、客户端与服务端版本）。manifest 不含凭据。 +备份与校验也可以走 Python 工具通道（`python scripts/ops.py backup|restore|verify`）。**两条通道的 dump 与 manifest 格式完全相同，可以互相读取**：2026-09-15 已交叉验证—— Go 产出的备份用 Python 恢复、Python 产出的备份用 Go 恢复，两个恢复实例的逐表行数与内容校验和都与源库一致。 + ### 恢复步骤 ```bash -python scripts/ops.py restore --dump <备份目录>/lexgo-<时间戳>.sql.gz --database <新库名> --confirm -python scripts/ops.py verify --database <新库名> --manifest <备份目录>/manifest.json \ - --api http://127.0.0.1:<演练端口> --user <账号前缀> --password-env <密码环境变量名> +./lexgo restore --dump <备份目录>/lexgo-<时间戳>.sql.gz --database <新库名> --confirm +./lexgo verify --database <新库名> --manifest <备份目录>/manifest.json ``` -预期结果：恢复写入空库、自动校验通过；`verify` 打印完整性与两账号隔离检查，最后「校验通过」。恢复**不会**写入备份里的源库，工具会在恢复前后比对源库的内容校验和（本机演练已验证）。 +预期结果：恢复写入空库并自动校验通过；`verify` 打印完整性与逐表行数比对，最后「verification passed」。恢复**不会**写入备份里的源库，工具会在恢复前后比对源库的内容校验和（本机演练已验证）。 + +**接口级的两账号闭环验证仍在 Python 工具通道**（它需要发 HTTP 请求）： + +```bash +python scripts/ops.py verify --database <库名> --api http://127.0.0.1:<端口> --user <账号前缀> --password-env <变量名> +python scripts/ops.py smoke --api http://127.0.0.1:<端口> --admin-user <管理员> --user <前缀> +``` > **注意**：`restore` 默认拒绝写入已有数据的库；覆盖必须 `--force`，执行前先备份当前库。库名必须包含 `lexgo` 且不能是 MySQL 系统库。 @@ -195,3 +214,5 @@ python scripts/ops.py verify --database <新库名> --manifest <备份目录>/ma - 未启用定时备份、监控与告警；备份由人工触发。 - **附件（封面/音频）尚未实现（工单 #21）**：本页的备份恢复范围目前只覆盖数据库；#21 落地后必须把附件存储纳入备份与恢复验收。 - 恢复演练是单机顺序执行，未验证大库恢复耗时与磁盘空间上限。 +- 接口级的两账号闭环与越权验证需要 HTTP 客户端，目前只在 Python 工具通道提供；Go 二进制提供数据库层的备份、恢复与校验。 +- 备份仍调用 `mysqldump`：自己实现一致性导出风险更高，因此部署机需要 MySQL 客户端而不只是服务端。 diff --git a/docs/README.md b/docs/README.md index 02bd113..1ca4ecf 100644 --- a/docs/README.md +++ b/docs/README.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件) wiki_page: Home wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Home -wiki_revision: 22731c2b344b3e0c75eb892c897d36d1f153a441 -synchronized_at: 2026-09-15T08:27:37Z +wiki_revision: 8c563c979bdfcacf1dda8743621b0ea00b3805f8 +synchronized_at: 2026-09-15T09:08:48Z # LexGo 文档入口 @@ -88,3 +88,5 @@ Quant-UX 原型 v1 已通过用户验收。[桌面预览](https://qux.ilapage.cn #14 桌面与手机体验、主题与键盘操作已于 2026-09-15 通过用户验收：新增「显示」设置（浅色/深色/跟随系统，正文标准/大/特大），偏好按账号保存在本机，切换账号不会串；阅读位置按账号与章节保留，正文换成新版本后不再套用旧位置；复习页支持空格显示答案、1/2/3 评分，输入框内不会误触快捷键，所有可交互元素有可见焦点环。手机端用 390×844 触摸视口验证八个界面无横向溢出、都能返回、触摸滑动滚动而不误开释义面板，深色与特大字号刷新后保持。本次没有数据库与接口变化；PR #34 已 fast-forward-only 合入 main。真实手机的长按选择与手感仍需人工确认。 #15 自托管试用交付已实现，待用户验收：新增运维工具（`scripts/ops.py`：依赖检查、建空库、备份、恢复、校验、两账号闭环演练；`scripts/bench.py`：写明规模的人造数据集性能测量）、新的 Wiki 部署与运维页，以及一次完整的本机演练——空库安装（建库 → 迁移 → 首次 bootstrap 建管理员 → 管理员建两个虚构演练账号 → 走通粘贴/阅读/查词/保存/复习/完成章节/进度）、备份 lexgo_dev、恢复到空实例并用第二个 API 实例完成两账号闭环与越权校验，恢复后逐表行数与内容校验和都和源库一致、源库未被改动。备份含全部用户数据（数据库）与环境配置（单独保管凭据），恢复默认只写空库、覆盖需要显式确认与 `--force`、拒绝写入系统库。性能在 20 章/2000 词条/8000 条作答的人造数据集上实测并记录环境，只作观察不给承诺。真实回滚、HTTPS、多机与定时备份仍未验证；附件（#21）尚未实施，恢复范围目前只覆盖数据库。 + +#15 自托管试用交付已实现，待用户验收：备份、恢复与校验同时提供**纯二进制子命令**（`lexgo backup|restore|verify`，部署机只需要二进制与 MySQL 客户端）与 Python 工具通道（交叉验证与开发便利），两条通道的 dump 与 manifest 格式一致、可互相恢复；新增运维工具（`scripts/ops.py`：依赖检查、建空库、备份、恢复、校验、两账号闭环演练；`scripts/bench.py`：写明规模的人造数据集性能测量）、新的 Wiki 部署与运维页，以及一次完整的本机演练——空库安装（建库 → 迁移 → 首次 bootstrap 建管理员 → 管理员建两个虚构演练账号 → 走通粘贴/阅读/查词/保存/复习/完成章节/进度）、备份 lexgo_dev、恢复到空实例并用第二个 API 实例完成两账号闭环与越权校验，恢复后逐表行数与内容校验和都和源库一致、源库未被改动。备份含全部用户数据（数据库）与环境配置（单独保管凭据），恢复默认只写空库、覆盖需要显式确认与 `--force`、拒绝写入系统库。性能在 20 章/2000 词条/8000 条作答的人造数据集上实测并记录环境，只作观察不给承诺。两条通道已在本机交叉验证（19 项）。真实回滚、HTTPS、多机与定时备份仍未验证；附件（#21）尚未实施，恢复范围目前只覆盖数据库。 diff --git a/server/app/lexgo/library.go b/server/app/lexgo/library.go index a38ee47..fd17b05 100644 --- a/server/app/lexgo/library.go +++ b/server/app/lexgo/library.go @@ -140,9 +140,9 @@ type ChapterSummary struct { Status string `json:"status"` CharCount int `json:"charCount"` // JobID lets a client retry a failed chapter without keeping the submit response. - JobID *int64 `json:"jobId"` - ErrorReason string `json:"errorReason"` - ErrorMessage string `json:"errorMessage"` + JobID *int64 `json:"jobId"` + ErrorReason string `json:"errorReason"` + ErrorMessage string `json:"errorMessage"` // ReadAt is set only where the completion was resolved and still matches the chapter's // current content; a chapter edited into a newer version reads as unread again. ReadAt *time.Time `json:"readAt"` diff --git a/server/app/lexgo/ops.go b/server/app/lexgo/ops.go new file mode 100644 index 0000000..24342c2 --- /dev/null +++ b/server/app/lexgo/ops.go @@ -0,0 +1,600 @@ +package lexgo + +import ( + "compress/gzip" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "io" + "os" + "os/exec" + "path/filepath" + "regexp" + "sort" + "strconv" + "strings" + "time" + + "gorm.io/gorm" +) + +// Operations for a self-hosted instance: back up the whole database, restore a backup into an +// empty one, and verify the result. Everything a LexGo instance persists lives in MySQL (accounts, +// book and chapter text, the dictionary archive, personal terms, review schedules and answers, +// reading progress and the audit logs), so a dump plus the environment file is a complete backup. +// +// The same rules as the operations script apply, because a restore can overwrite a working +// instance: an explicit confirmation, a target name that contains "lexgo", a refusal to touch a +// database that already holds data, and a refusal to load a dump that would switch databases. + +// OpsTables lists the tables a backup covers, in a stable order for reports. +var OpsTables = []string{ + "sys_user", "lexgo_spaces", "lexgo_sessions", "lexgo_login_logs", "lexgo_operation_logs", + "lexgo_books", "lexgo_chapters", "lexgo_ingest_jobs", "lexgo_dictionaries", + "lexgo_terms", "lexgo_term_reviews", "lexgo_review_answers", "lexgo_chapter_progress", +} + +var opsSystemSchemas = map[string]bool{ + "mysql": true, "information_schema": true, "performance_schema": true, "sys": true, +} + +// OpsManifest is written next to every dump. The field names match the operations script so either +// tool can read the other's backup, and it deliberately carries no credential. +type OpsManifest struct { + CreatedAt string `json:"created_at"` + Database string `json:"database"` + Product string `json:"product"` + Schema int `json:"schema_version"` + GitCommit string `json:"git_commit"` + DumpFile string `json:"dump_file"` + DumpSHA256 string `json:"dump_sha256"` + DumpBytes int64 `json:"dump_bytes"` + RowCounts map[string]int64 `json:"row_counts"` + MySQLClient string `json:"mysql_client"` + MySQLServer string `json:"mysql_server"` + Contents string `json:"contents"` + Excludes string `json:"excludes"` +} + +// OpsCheck is one verification result. +type OpsCheck struct { + Name string + OK bool + Detail string +} + +// ValidateOpsDatabaseName rejects anything that must never be written by a restore. +func ValidateOpsDatabaseName(name string) error { + if name == "" || opsSystemSchemas[strings.ToLower(name)] || !strings.Contains(strings.ToLower(name), "lexgo") { + return errors.New("refusing to act on " + name + ": the database name must contain lexgo and must not be a MySQL system schema") + } + return nil +} + +// VersionTriple reads a dotted version out of a client or server banner. A client banner reads +// "Ver 14.14 Distrib 5.7.38", where the first number is the protocol version, so the number after +// "Distrib" is the one to compare. +func VersionTriple(text string) (int, int, int, bool) { + pattern := regexp.MustCompile(`Distrib\s+(\d+)\.(\d+)(?:\.(\d+))?`) + match := pattern.FindStringSubmatch(text) + if match == nil { + match = regexp.MustCompile(`(\d+)\.(\d+)(?:\.(\d+))?`).FindStringSubmatch(text) + } + if match == nil { + return 0, 0, 0, false + } + major, _ := strconv.Atoi(match[1]) + minor, _ := strconv.Atoi(match[2]) + patch := 0 + if match[3] != "" { + patch, _ = strconv.Atoi(match[3]) + } + return major, minor, patch, true +} + +// DumpSwitchesDatabase reports whether a dump carries CREATE DATABASE or USE statements. Loading +// such a dump would write into the schema named inside the file instead of the requested target. +func DumpSwitchesDatabase(head string) bool { + if strings.Contains(strings.ToUpper(head), "CREATE DATABASE") { + return true + } + return regexp.MustCompile(`(?m)^USE `).MatchString(head) +} + +func opsMySQLBinary(name string) (string, error) { + configured := os.Getenv("LEXGO_MYSQL_BIN") + if configured != "" { + candidate := filepath.Join(configured, name) + if _, err := os.Stat(candidate); err == nil { + return candidate, nil + } + if _, err := os.Stat(candidate + ".exe"); err == nil { + return candidate + ".exe", nil + } + } + found, err := exec.LookPath(name) + if err != nil { + return "", errors.New("cannot find " + name + ": install the MySQL client or point LEXGO_MYSQL_BIN at it") + } + return found, nil +} + +func opsClientEnv() []string { + env := os.Environ() + if password := os.Getenv("LEXGO_DB_PASSWORD"); password != "" { + // The client reads MYSQL_PWD, so the password never appears in an argument list. + env = append(env, "MYSQL_PWD="+password) + } + return env +} + +func opsConnectionArgs(database string) []string { + host := os.Getenv("LEXGO_DB_HOST") + if host == "" { + host = "127.0.0.1" + } + port := os.Getenv("LEXGO_DB_PORT") + if port == "" { + port = "3308" + } + args := []string{"--host=" + host, "--port=" + port, "--user=" + os.Getenv("LEXGO_DB_USER")} + if database != "" { + args = append(args, database) + } + return args +} + +// OpsTableName qualifies a table with its database. The operations client deliberately does not +// select a schema (a restore may have to create one), so every introspection query is qualified. +func OpsTableName(database, table string) string { + return "`" + database + "`.`" + table + "`" +} + +// OpsTableCounts reads the row count of every known table. +func OpsTableCounts(db *gorm.DB, database string) (map[string]int64, error) { + counts := map[string]int64{} + for _, table := range OpsTables { + var exists int64 + if err := db.Raw("SELECT COUNT(*) FROM information_schema.TABLES WHERE TABLE_SCHEMA = ? AND TABLE_NAME = ?", database, table).Scan(&exists).Error; err != nil { + return counts, err + } + if exists == 0 { + continue + } + var total int64 + if err := db.Raw("SELECT COUNT(*) FROM " + OpsTableName(database, table)).Scan(&total).Error; err != nil { + return counts, err + } + counts[table] = total + } + return counts, nil +} + +// OpsTableChecksums returns a content checksum per table, which is what proves a restore equals +// its source: row counts alone could match after a bad write. +func OpsTableChecksums(db *gorm.DB, database string) (map[string]string, error) { + checksums := map[string]string{} + for _, table := range OpsTables { + rows, err := db.Raw("CHECKSUM TABLE " + OpsTableName(database, table)).Rows() + if err != nil { + // A missing table is reported as absent rather than failing the comparison. + continue + } + for rows.Next() { + var name, sum string + if err = rows.Scan(&name, &sum); err != nil { + rows.Close() + return checksums, err + } + checksums[table] = sum + } + rows.Close() + } + return checksums, nil +} + +func opsDigest(path string) (string, error) { + file, err := os.Open(path) + if err != nil { + return "", err + } + defer file.Close() + digest := sha256.New() + if _, err = io.Copy(digest, file); err != nil { + return "", err + } + return hex.EncodeToString(digest.Sum(nil)), nil +} + +func opsGzipFile(source, target string) error { + input, err := os.Open(source) + if err != nil { + return err + } + defer input.Close() + output, err := os.Create(target) + if err != nil { + return err + } + defer output.Close() + writer, err := gzip.NewWriterLevel(output, gzip.BestSpeed) + if err != nil { + return err + } + if _, err = io.Copy(writer, input); err != nil { + return err + } + return writer.Close() +} + +// OpsBackupOptions describes one backup request. +type OpsBackupOptions struct { + Database string + OutDir string + Force bool + Commit string + Now time.Time +} + +// OpsBackup runs mysqldump for the whole schema, compresses it and writes a manifest. +func OpsBackup(db *gorm.DB, options OpsBackupOptions) (OpsManifest, error) { + manifest := OpsManifest{} + database := options.Database + if err := ValidateOpsDatabaseName(database); err != nil { + return manifest, err + } + outDir := options.OutDir + if outDir == "" { + outDir = filepath.Join(".local", "backups", options.Now.UTC().Format("20060102-150405")) + } + if entries, err := os.ReadDir(outDir); err == nil && len(entries) > 0 && !options.Force { + return manifest, errors.New("output directory is not empty: " + outDir + " (use --force to overwrite)") + } + if err := os.MkdirAll(outDir, 0o755); err != nil { + return manifest, err + } + dumpTool, err := opsMySQLBinary("mysqldump") + if err != nil { + return manifest, err + } + stamp := options.Now.UTC().Format("20060102T150405Z") + plain := filepath.Join(outDir, "lexgo-"+stamp+".sql") + compressed := plain + ".gz" + defer os.Remove(plain) + + // Deliberately without --databases: that option writes CREATE DATABASE and USE into the dump, + // which would send a restore into the source schema instead of the requested target. + //nolint:gosec // the arguments are configuration plus a validated database name, and the + // password travels through MYSQL_PWD rather than the command line. + command := exec.Command(dumpTool, append(opsConnectionArgs(""), + "--single-transaction", "--routines", "--triggers", "--hex-blob", "--no-tablespaces", + "--default-character-set=utf8mb4", database)...) + command.Env = opsClientEnv() + output, err := os.Create(plain) + if err != nil { + return manifest, err + } + command.Stdout = output + var stderr strings.Builder + command.Stderr = &stderr + runErr := command.Run() + closeErr := output.Close() + if runErr != nil { + return manifest, fmt.Errorf("mysqldump failed: %s", lastLine(stderr.String())) + } + if closeErr != nil { + return manifest, closeErr + } + if err = opsGzipFile(plain, compressed); err != nil { + return manifest, err + } + digest, err := opsDigest(compressed) + if err != nil { + return manifest, err + } + info, err := os.Stat(compressed) + if err != nil { + return manifest, err + } + counts, err := OpsTableCounts(db, database) + if err != nil { + return manifest, err + } + var schema int + if err = db.Raw("SELECT version FROM " + OpsTableName(database, "lexgo_schema") + " WHERE id = 1").Scan(&schema).Error; err != nil { + return manifest, err + } + var server string + _ = db.Raw("SELECT VERSION()").Scan(&server).Error + client := "" + if out, err := exec.Command(dumpTool, "--version").Output(); err == nil { + client = strings.TrimSpace(string(out)) + } + manifest = OpsManifest{ + CreatedAt: stamp, Database: database, Product: "lexgo", Schema: schema, + GitCommit: options.Commit, DumpFile: filepath.Base(compressed), DumpSHA256: digest, + DumpBytes: info.Size(), RowCounts: counts, MySQLClient: client, MySQLServer: server, + Contents: "accounts, spaces, sessions, audit logs, books, chapters, ingest jobs, dictionary archive, terms, review schedules, review answers, reading progress", + Excludes: "credentials and the environment file; back those up separately from the operations vault", + } + encoded, err := json.MarshalIndent(manifest, "", " ") + if err != nil { + return manifest, err + } + if err = os.WriteFile(filepath.Join(outDir, "manifest.json"), append(encoded, '\n'), 0o644); err != nil { + return manifest, err + } + return manifest, nil +} + +func lastLine(text string) string { + lines := strings.Split(strings.TrimSpace(text), "\n") + if len(lines) == 0 { + return "" + } + return strings.TrimSpace(lines[len(lines)-1]) +} + +// OpsRestoreOptions describes one restore request. +type OpsRestoreOptions struct { + Dump string + Database string + Confirm bool + Force bool + SkipManifest bool + Now time.Time +} + +// OpsRestore loads a backup into an empty database and verifies the result. The dump is refused +// when it would switch databases, the target must be empty unless --force is given, and the source +// database named in the manifest is compared before and after so the restore can never write there. +func OpsRestore(db *gorm.DB, client *gorm.DB, options OpsRestoreOptions) (OpsManifest, []OpsCheck, error) { + manifest := OpsManifest{} + if !options.Confirm { + return manifest, nil, errors.New("a restore writes to the database and needs confirmation: add --confirm") + } + if err := ValidateOpsDatabaseName(options.Database); err != nil { + return manifest, nil, err + } + if _, err := os.Stat(options.Dump); err != nil { + return manifest, nil, errors.New("backup file not found: " + options.Dump) + } + manifestPath := filepath.Join(filepath.Dir(options.Dump), "manifest.json") + if data, err := os.ReadFile(manifestPath); err == nil { + if err = json.Unmarshal(data, &manifest); err != nil { + return manifest, nil, errors.New("manifest.json is not readable") + } + digest, err := opsDigest(options.Dump) + if err != nil { + return manifest, nil, err + } + if manifest.DumpSHA256 != "" && manifest.DumpSHA256 != digest { + return manifest, nil, errors.New("the backup file does not match the sha256 in its manifest; refusing to restore") + } + } else if !options.SkipManifest { + return manifest, nil, errors.New("manifest.json not found next to the dump; add --skip-manifest-check to continue") + } + + head, err := opsDumpHead(options.Dump) + if err != nil { + return manifest, nil, err + } + if DumpSwitchesDatabase(head) { + return manifest, nil, errors.New("this dump contains CREATE DATABASE/USE, so a restore would write into the schema named inside it; take a new backup with lexgo backup") + } + + source := manifest.Database + var sourceBefore map[string]string + if source != "" && source != options.Database { + if sourceBefore, err = OpsTableChecksums(client, source); err != nil { + return manifest, nil, err + } + } + + if err = opsPrepareTarget(client, options.Database, options.Force); err != nil { + return manifest, nil, err + } + if err = opsLoadDump(options.Dump, options.Database); err != nil { + return manifest, nil, err + } + if sourceBefore != nil { + sourceAfter, err := OpsTableChecksums(client, source) + if err != nil { + return manifest, nil, err + } + if !sameChecksums(sourceBefore, sourceAfter) { + return manifest, nil, errors.New("the source database " + source + " changed during the restore; stop and compare by hand") + } + } + checks, err := OpsVerify(client, options.Database, manifestPath) + return manifest, checks, err +} + +func opsDumpHead(dump string) (string, error) { + file, err := os.Open(dump) + if err != nil { + return "", err + } + defer file.Close() + reader, err := gzip.NewReader(file) + if err != nil { + return "", errors.New("the backup file is not a gzip archive: " + err.Error()) + } + defer reader.Close() + buffer := make([]byte, 256*1024) + total := 0 + for total < len(buffer) { + read, err := reader.Read(buffer[total:]) + total += read + if err != nil { + break + } + } + return string(buffer[:total]), nil +} + +func opsPrepareTarget(client *gorm.DB, database string, force bool) error { + var exists int + if err := client.Raw("SELECT COUNT(*) FROM information_schema.SCHEMATA WHERE SCHEMA_NAME = ?", database).Scan(&exists).Error; err != nil { + return err + } + if exists == 0 { + return client.Exec("CREATE DATABASE `" + database + "` CHARACTER SET utf8mb4").Error + } + var tables int + if err := client.Raw("SELECT COUNT(*) FROM information_schema.TABLES WHERE TABLE_SCHEMA = ?", database).Scan(&tables).Error; err != nil { + return err + } + if tables == 0 { + return nil + } + if !force { + return errors.New("target database " + database + " already holds " + strconv.Itoa(tables) + " tables; a restore only writes an empty database (use --force after taking a backup)") + } + if err := client.Exec("DROP DATABASE `" + database + "`").Error; err != nil { + return err + } + return client.Exec("CREATE DATABASE `" + database + "` CHARACTER SET utf8mb4").Error +} + +func opsLoadDump(dump, database string) error { + client, err := opsMySQLBinary("mysql") + if err != nil { + return err + } + file, err := os.Open(dump) + if err != nil { + return err + } + defer file.Close() + reader, err := gzip.NewReader(file) + if err != nil { + return err + } + defer reader.Close() + //nolint:gosec // configuration plus a validated database name; the password is in MYSQL_PWD. + command := exec.Command(client, append(opsConnectionArgs(database), "--default-character-set=utf8mb4")...) + command.Env = opsClientEnv() + command.Stdin = reader + var stderr strings.Builder + command.Stderr = &stderr + if err = command.Run(); err != nil { + return fmt.Errorf("restore failed: %s", lastLine(stderr.String())) + } + return nil +} + +func sameChecksums(before, after map[string]string) bool { + if len(before) != len(after) { + return false + } + for table, sum := range before { + if after[table] != sum { + return false + } + } + return true +} + +// OpsVerify checks that a database holds consistent LexGo data and, when a manifest is given, that +// every table still has the number of rows the backup recorded. +func OpsVerify(client *gorm.DB, database string, manifestPath string) ([]OpsCheck, error) { + checks := []OpsCheck{} + add := func(name string, ok bool, detail string) { checks = append(checks, OpsCheck{name, ok, detail}) } + table := func(name string) string { return OpsTableName(database, name) } + + linked := func(sql, name string) error { + var count int64 + if err := client.Raw(sql).Scan(&count).Error; err != nil { + return err + } + add(name, count == 0, strconv.FormatInt(count, 10)+" 行") + return nil + } + + var product string + _ = client.Raw("SELECT product FROM " + table("lexgo_schema") + " WHERE id = 1").Scan(&product).Error + add("产品标识为 lexgo", product == "lexgo", product) + chapter, book, term, review, answer, progress, space := + table("lexgo_chapters"), table("lexgo_books"), table("lexgo_terms"), table("lexgo_term_reviews"), + table("lexgo_review_answers"), table("lexgo_chapter_progress"), table("lexgo_spaces") + if err := linked("SELECT COUNT(*) FROM "+chapter+" c LEFT JOIN "+book+" b ON b.id = c.book_id WHERE b.id IS NULL", "没有孤立章节（章节属于其书）"); err != nil { + return checks, err + } + if err := linked("SELECT COUNT(*) FROM "+chapter+" c JOIN "+book+" b ON b.id = c.book_id WHERE c.owner_id <> b.owner_id", "章节归属与书归属一致"); err != nil { + return checks, err + } + if err := linked("SELECT COUNT(*) FROM "+term+" t LEFT JOIN "+review+" r ON r.term_id = t.id WHERE r.term_id IS NULL", "每个词条都有排期行"); err != nil { + return checks, err + } + if err := linked("SELECT COUNT(*) FROM "+review+" r LEFT JOIN "+term+" t ON t.id = r.term_id WHERE t.id IS NULL", "排期行都指向存在的词条"); err != nil { + return checks, err + } + if err := linked("SELECT COUNT(*) FROM "+answer+" a LEFT JOIN "+term+" t ON t.id = a.term_id WHERE t.id IS NULL", "复习记录引用有效词条"); err != nil { + return checks, err + } + if err := linked("SELECT COUNT(*) FROM "+term+" WHERE (status = 'learning' AND (level < 1 OR level > 7)) OR (status <> 'learning' AND level <> 0)", "等级只出现在学习中词条"); err != nil { + return checks, err + } + if err := linked("SELECT COUNT(*) FROM "+progress+" p LEFT JOIN "+chapter+" c ON c.id = p.chapter_id WHERE c.id IS NULL", "完成记录都指向存在的章节"); err != nil { + return checks, err + } + if err := linked("SELECT COUNT(*) FROM "+term+" t JOIN "+space+" s ON s.owner_id = t.owner_id WHERE t.language <> s.language", "词条语言与所属空间一致"); err != nil { + return checks, err + } + + // The audit tables must not gain a column that could hold a credential or private content. + for _, name := range []string{"lexgo_login_logs", "lexgo_operation_logs"} { + var columns []string + if err := client.Raw("SELECT COLUMN_NAME FROM information_schema.COLUMNS WHERE TABLE_SCHEMA = ? AND TABLE_NAME = ?", database, name).Scan(&columns).Error; err != nil { + return checks, err + } + banned := []string{} + for _, column := range columns { + lower := strings.ToLower(column) + for _, word := range []string{"password", "token", "body", "content", "definition", "text"} { + if strings.Contains(lower, word) { + banned = append(banned, column) + break + } + } + } + sort.Strings(banned) + add("审计表 "+name+" 不含敏感列", len(banned) == 0, strings.Join(banned, ",")) + } + + if manifestPath != "" { + if data, err := os.ReadFile(manifestPath); err == nil { + var manifest OpsManifest + if err = json.Unmarshal(data, &manifest); err == nil { + counts, err := OpsTableCounts(client, database) + if err != nil { + return checks, err + } + tables := make([]string, 0, len(manifest.RowCounts)) + for name := range manifest.RowCounts { + tables = append(tables, name) + } + sort.Strings(tables) + for _, name := range tables { + expected := manifest.RowCounts[name] + add("表 "+name+" 行数与备份一致", counts[name] == expected, + "备份 "+strconv.FormatInt(expected, 10)+" / 现在 "+strconv.FormatInt(counts[name], 10)) + } + } + } + } + return checks, nil +} + +// OpsAllOK reports whether every verification check passed. +func OpsAllOK(checks []OpsCheck) bool { + for _, check := range checks { + if !check.OK { + return false + } + } + return true +} diff --git a/server/app/lexgo/ops_test.go b/server/app/lexgo/ops_test.go new file mode 100644 index 0000000..55f213f --- /dev/null +++ b/server/app/lexgo/ops_test.go @@ -0,0 +1,287 @@ +package lexgo + +import ( + "compress/gzip" + "encoding/json" + admin "go-admin/app/admin/models" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +// The operations rules are checked here without a database; the round trip itself is covered by +// the MySQL integration test below and by the delivery drill. + +func TestOpsDatabaseNameRules(t *testing.T) { + for _, name := range []string{"lexgo_dev", "lexgo_prod", "lexgo_restore_drill"} { + if err := ValidateOpsDatabaseName(name); err != nil { + t.Fatalf("%s must be accepted: %v", name, err) + } + } + for _, name := range []string{"", "mysql", "information_schema", "performance_schema", "sys", "app_prod", "LEXGO", "lexgo"} { + err := ValidateOpsDatabaseName(name) + if name == "LEXGO" || name == "lexgo" { + if err != nil { + t.Fatalf("%s contains lexgo and must be accepted", name) + } + continue + } + if err == nil { + t.Fatalf("%s must be refused", name) + } + } +} + +func TestOpsVersionTriple(t *testing.T) { + // The first number in a client banner is the protocol version, so "Distrib" wins. + major, minor, patch, ok := VersionTriple("mysql Ver 14.14 Distrib 5.7.38, for Win64 (x86_64)") + if !ok || major != 5 || minor != 7 || patch != 38 { + t.Fatalf("client banner parsed as %d.%d.%d (%v)", major, minor, patch, ok) + } + if major, minor, patch, ok = VersionTriple("mysql Ver 8.4.3 for Win64 on x86_64"); !ok || major != 8 || minor != 4 || patch != 3 { + t.Fatalf("server banner parsed as %d.%d.%d (%v)", major, minor, patch, ok) + } + if _, _, _, ok = VersionTriple("no version"); ok { + t.Fatal("a banner without a version must not parse") + } + // An older client must sort below the server it would dump. + old := [3]int{} + newer := [3]int{} + old[0], old[1], old[2], _ = VersionTriple("Ver 14.14 Distrib 5.7.38") + newer[0], newer[1], newer[2], _ = VersionTriple("8.4.3") + if !(old[0] < newer[0]) { + t.Fatalf("5.7.38 must be older than 8.4.3: %v %v", old, newer) + } +} + +func writeDumpFile(t *testing.T, body string) string { + t.Helper() + path := filepath.Join(t.TempDir(), "dump.sql.gz") + handle, err := os.Create(path) + if err != nil { + t.Fatal(err) + } + writer := gzip.NewWriter(handle) + if _, err = writer.Write([]byte(body)); err != nil { + t.Fatal(err) + } + if err = writer.Close(); err != nil { + t.Fatal(err) + } + if err = handle.Close(); err != nil { + t.Fatal(err) + } + return path +} + +func TestOpsDumpHeadAndDatabaseSwitchDetection(t *testing.T) { + switching := writeDumpFile(t, "-- MySQL dump\nCREATE DATABASE /*!32312 IF NOT EXISTS*/ `lexgo_dev`;\nUSE `lexgo_dev`;\nCREATE TABLE t (id INT);\n") + head, err := opsDumpHead(switching) + if err != nil { + t.Fatal(err) + } + if !DumpSwitchesDatabase(head) { + t.Fatal("a dump with CREATE DATABASE must be detected") + } + useOnly := writeDumpFile(t, "-- MySQL dump\nUSE `lexgo_dev`;\nCREATE TABLE t (id INT);\n") + head, err = opsDumpHead(useOnly) + if err != nil { + t.Fatal(err) + } + if !DumpSwitchesDatabase(head) { + t.Fatal("a dump with USE must be detected") + } + plain := writeDumpFile(t, "-- MySQL dump\nDROP TABLE IF EXISTS `lexgo_terms`;\nCREATE TABLE `lexgo_terms` (id INT);\nINSERT INTO `lexgo_terms` VALUES (1);\n") + head, err = opsDumpHead(plain) + if err != nil { + t.Fatal(err) + } + if DumpSwitchesDatabase(head) { + t.Fatal("a table-only dump must be accepted") + } +} + +func TestOpsRefusesRestoreWithoutConfirmation(t *testing.T) { + dump := writeDumpFile(t, "-- MySQL dump\nCREATE TABLE `lexgo_terms` (id INT);\n") + _, _, err := OpsRestore(nil, nil, OpsRestoreOptions{Dump: dump, Database: "lexgo_restore_drill"}) + if err == nil { + t.Fatal("a restore without --confirm must be refused") + } +} + +func TestOpsRefusesRestoreIntoSystemSchema(t *testing.T) { + dump := writeDumpFile(t, "-- MySQL dump\nCREATE TABLE `lexgo_terms` (id INT);\n") + _, _, err := OpsRestore(nil, nil, OpsRestoreOptions{Dump: dump, Database: "mysql", Confirm: true}) + if err == nil { + t.Fatal("a restore into a system schema must be refused") + } +} + +func TestOpsRefusesMissingBackupAndManifest(t *testing.T) { + _, _, err := OpsRestore(nil, nil, OpsRestoreOptions{Dump: filepath.Join(t.TempDir(), "absent.sql.gz"), Database: "lexgo_x", Confirm: true}) + if err == nil { + t.Fatal("a missing backup file must be reported") + } + + directory := t.TempDir() + dump := filepath.Join(directory, "lexgo-20260101T000000Z.sql.gz") + handle, err := os.Create(dump) + if err != nil { + t.Fatal(err) + } + writer := gzip.NewWriter(handle) + if _, err = writer.Write([]byte("-- MySQL dump\nCREATE TABLE `lexgo_terms` (id INT);\n")); err != nil { + t.Fatal(err) + } + writer.Close() + handle.Close() + _, _, err = OpsRestore(nil, nil, OpsRestoreOptions{Dump: dump, Database: "lexgo_x", Confirm: true}) + if err == nil { + t.Fatal("a dump without a manifest must be refused unless the check is skipped") + } + // With --skip-manifest-check and --force the guards accept the request; the database step is + // where it goes next, which the integration test covers with a real connection. + head, err := opsDumpHead(dump) + if err != nil || DumpSwitchesDatabase(head) { + t.Fatalf("the dump itself is acceptable: %v", err) + } +} + +func TestOpsChecksumComparison(t *testing.T) { + before := map[string]string{"lexgo_terms": "1", "lexgo_books": "2"} + if !sameChecksums(before, map[string]string{"lexgo_terms": "1", "lexgo_books": "2"}) { + t.Fatal("equal checksums must compare equal") + } + if sameChecksums(before, map[string]string{"lexgo_terms": "1", "lexgo_books": "3"}) { + t.Fatal("a changed table must be detected") + } + if sameChecksums(before, map[string]string{"lexgo_terms": "1"}) { + t.Fatal("a missing table must be detected") + } +} + +func TestOpsManifestCarriesNoCredential(t *testing.T) { + // The manifest is written to disk: its shape is asserted so a later edit cannot add a secret. + manifest := OpsManifest{ + CreatedAt: time.Now().UTC().Format(time.RFC3339), Database: "lexgo_dev", Product: "lexgo", + Schema: SchemaVersion, DumpFile: "lexgo-x.sql.gz", DumpSHA256: "abc", + } + encoded, err := jsonMarshal(manifest) + if err != nil { + t.Fatal(err) + } + for _, banned := range []string{"password", "secret", "token", "credential"} { + if containsFold(encoded, banned) { + t.Fatalf("manifest must not carry %q: %s", banned, encoded) + } + } + for _, expected := range []string{"schema_version", "row_counts", "dump_sha256", "git_commit"} { + if !containsFold(encoded, expected) { + t.Fatalf("manifest must carry %q: %s", expected, encoded) + } + } +} + +// jsonMarshal and containsFold keep this test independent of the assertions helper used elsewhere. +func jsonMarshal(value any) (string, error) { + encoded, err := json.Marshal(value) + return string(encoded), err +} + +func containsFold(haystack, needle string) bool { + return strings.Contains(strings.ToLower(haystack), strings.ToLower(needle)) +} + +// TestMySQLOpsBackupRestoreRoundTrip covers the operations path against real MySQL: back up a +// database, restore the dump into an empty one, and compare both the row counts and the table +// contents. It skips when the MySQL client tools are not installed, because the dump is taken by +// mysqldump. +func TestMySQLOpsBackupRestoreRoundTrip(t *testing.T) { + db := testDB(t) + source := os.Getenv("LEXGO_TEST_DB_NAME") + if source == "" { + t.Skip("needs LEXGO_TEST_DB_NAME") + } + if _, err := opsMySQLBinary("mysqldump"); err != nil { + t.Skip("mysqldump is not installed") + } + // A fixture row so the restore has something to bring back that this test created. + owner := admin.SysUser{Username: randomName("ops"), Password: fixturePassword, RoleId: 2, Status: "2"} + if err := db.Create(&owner).Error; err != nil { + t.Fatal(err) + } + before, err := OpsTableCounts(db, source) + if err != nil { + t.Fatal(err) + } + sourceChecksums, err := OpsTableChecksums(db, source) + if err != nil { + t.Fatal(err) + } + + outDir := t.TempDir() + manifest, err := OpsBackup(db, OpsBackupOptions{ + Database: source, OutDir: outDir, Commit: "test-commit", Now: time.Now()}) + if err != nil { + t.Fatal(err) + } + if manifest.Schema != SchemaVersion || manifest.Product != "lexgo" || len(manifest.DumpSHA256) != 64 { + t.Fatalf("manifest looks wrong: %+v", manifest) + } + if len(manifest.RowCounts) == 0 { + t.Fatal("the manifest must record row counts") + } + encoded, err := os.ReadFile(filepath.Join(outDir, "manifest.json")) + if err != nil { + t.Fatal(err) + } + if strings.Contains(strings.ToLower(string(encoded)), "password") { + t.Fatal("the manifest must not mention credentials") + } + + target := "lexgo_test_ops_restore" + if err := db.Exec("DROP DATABASE IF EXISTS `" + target + "`").Error; err != nil { + t.Fatal(err) + } + t.Cleanup(func() { db.Exec("DROP DATABASE IF EXISTS `" + target + "`") }) + + dumpPath := filepath.Join(outDir, manifest.DumpFile) + restored, checks, err := OpsRestore(db, db, OpsRestoreOptions{ + Dump: dumpPath, Database: target, Confirm: true, Now: time.Now()}) + if err != nil { + t.Fatal(err) + } + if restored.Schema != SchemaVersion { + t.Fatalf("restored schema %d", restored.Schema) + } + if !OpsAllOK(checks) { + t.Fatalf("verification failed: %+v", checks) + } + after, err := OpsTableCounts(db, target) + if err != nil { + t.Fatal(err) + } + if len(after) != len(before) { + t.Fatalf("restored %d tables, source has %d", len(after), len(before)) + } + for table, expected := range before { + if after[table] != expected { + t.Fatalf("table %s restored %d rows, source has %d", table, after[table], expected) + } + } + + // A second restore into the same database must be refused: only an empty target is written. + if _, _, err = OpsRestore(db, db, OpsRestoreOptions{Dump: dumpPath, Database: target, Confirm: true, Now: time.Now()}); err == nil { + t.Fatal("a restore into a database that holds data must be refused without --force") + } + // The source database is untouched by the restore. + nowChecksums, err := OpsTableChecksums(db, source) + if err != nil { + t.Fatal(err) + } + if !sameChecksums(sourceChecksums, nowChecksums) { + t.Fatal("the source database changed during the restore") + } +} diff --git a/server/cmd/lexgo/main.go b/server/cmd/lexgo/main.go index f48b9bb..a5cbbd2 100644 --- a/server/cmd/lexgo/main.go +++ b/server/cmd/lexgo/main.go @@ -3,12 +3,15 @@ package main import ( "context" "errors" + "flag" "fmt" "log" "net" "net/http" "os" + "os/exec" "os/signal" + "sort" "strings" "syscall" "time" @@ -65,13 +68,20 @@ func openDB() (*gorm.DB, error) { } func run() error { - if len(os.Args) != 2 { - return errors.New("usage: lexgo migrate|bootstrap|serve|audit-cleanup") + if len(os.Args) < 2 { + return errors.New("usage: lexgo migrate|bootstrap|serve|audit-cleanup|backup|restore|verify") } command := os.Args[1] - if command != "migrate" && command != "bootstrap" && command != "serve" && command != "audit-cleanup" { + known := map[string]bool{"migrate": true, "bootstrap": true, "serve": true, "audit-cleanup": true, + "backup": true, "restore": true, "verify": true} + if !known[command] { return errors.New("unknown command") } + // backup, restore and verify need a database that may not exist yet, so they open the client + // without selecting a schema and are handled before the application database is required. + if command == "backup" || command == "restore" || command == "verify" { + return runOps(command, os.Args[2:]) + } db, err := openDB() if err != nil { return err @@ -173,6 +183,166 @@ func run() error { return srv.Shutdown(shutdown) } } + +// runOps handles the operations subcommands: they talk to MySQL without selecting a schema, so +// they work on a database that a restore is about to create. +func runOps(command string, args []string) error { + flags := flag.NewFlagSet(command, flag.ContinueOnError) + // Only the flags that belong to this command are registered, so -h shows a usable list. + database := flags.String("database", "", "database to act on; defaults to LEXGO_DB_NAME") + var out, dump, manifest *string + var force, confirm, skipManifest *bool + switch command { + case "backup": + out = flags.String("out", "", "directory for the dump and its manifest") + force = flags.Bool("force", false, "overwrite a non-empty output directory") + case "restore": + dump = flags.String("dump", "", "backup file to restore") + confirm = flags.Bool("confirm", false, "confirm that the restore may write to the database") + force = flags.Bool("force", false, "overwrite a target database that already holds data") + skipManifest = flags.Bool("skip-manifest-check", false, "continue without the manifest that proves the backup is intact") + case "verify": + manifest = flags.String("manifest", "", "manifest.json to compare row counts with") + } + if err := flags.Parse(args); err != nil { + return err + } + client, err := openClient() + if err != nil { + return err + } + target := *database + if target == "" { + target = os.Getenv("LEXGO_DB_NAME") + } + switch command { + case "backup": + result, err := lexgo.OpsBackup(client, lexgo.OpsBackupOptions{ + Database: target, OutDir: *out, Force: *force, Commit: gitCommit(), Now: time.Now()}) + if err != nil { + return err + } + fmt.Printf("backup written: %s (%d bytes)\n", result.DumpFile, result.DumpBytes) + fmt.Printf(" schema %d, commit %s\n", result.Schema, shortCommit(result.GitCommit)) + fmt.Printf(" row counts %s\n", rowCountsText(result.RowCounts)) + fmt.Printf(" sha256 %s\n", result.DumpSHA256) + fmt.Println(" credentials and the environment file are not included; store those in the operations vault") + return nil + case "restore": + result, checks, err := lexgo.OpsRestore(client, client, lexgo.OpsRestoreOptions{ + Dump: *dump, Database: target, Confirm: *confirm, Force: *force, + SkipManifest: *skipManifest, Now: time.Now()}) + if err != nil { + return err + } + for _, check := range checks { + fmt.Printf(" %s%s\n", checkPrefix(check.OK), check.Name) + } + if !lexgo.OpsAllOK(checks) { + return errors.New("the restored database did not pass verification") + } + fmt.Printf("restore of %s into %s verified\n", result.DumpFile, target) + return nil + case "verify": + checks, err := lexgo.OpsVerify(client, target, *manifest) + if err != nil { + return err + } + for _, check := range checks { + detail := "" + if check.Detail != "" { + detail = " - " + check.Detail + } + fmt.Printf(" %s%s%s\n", checkPrefix(check.OK), check.Name, detail) + } + if !lexgo.OpsAllOK(checks) { + return errors.New("verification failed") + } + fmt.Println("verification passed") + return nil + } + return errors.New("unknown command") +} + +func checkPrefix(ok bool) string { + if ok { + return "[ok] " + } + return "[fail] " +} + +func shortCommit(commit string) string { + if len(commit) > 8 { + return commit[:8] + } + if commit == "" { + return "unknown" + } + return commit +} + +func rowCountsText(counts map[string]int64) string { + tables := make([]string, 0, len(counts)) + for table := range counts { + tables = append(tables, table) + } + sort.Strings(tables) + parts := make([]string, 0, len(tables)) + for _, table := range tables { + parts = append(parts, fmt.Sprintf("%s=%d", table, counts[table])) + } + return strings.Join(parts, " ") +} + +func gitCommit() string { + output, err := exec.Command("git", "rev-parse", "HEAD").Output() + if err != nil { + return "" + } + return strings.TrimSpace(string(output)) +} + +// openClient connects without selecting a schema, which a restore needs because the target +// database may not exist yet. +func openClient() (*gorm.DB, error) { + c := driver.NewConfig() + c.Net = "tcp" + host := os.Getenv("LEXGO_DB_HOST") + if host == "" { + host = "127.0.0.1" + } + port := os.Getenv("LEXGO_DB_PORT") + if port == "" { + port = "3308" + } + c.Addr = net.JoinHostPort(host, port) + c.User = os.Getenv("LEXGO_DB_USER") + c.Passwd = os.Getenv("LEXGO_DB_PASSWORD") + if c.User == "" || c.Passwd == "" { + return nil, errors.New("set LEXGO_DB_USER and LEXGO_DB_PASSWORD") + } + c.ParseTime = true + c.Loc = time.UTC + c.Timeout = 5 * time.Second + c.Params = map[string]string{"charset": "utf8mb4"} + db, err := gorm.Open(mysql.Open(c.FormatDSN()), &gorm.Config{Logger: logger.Default.LogMode(logger.Silent)}) + if err != nil { + return nil, errors.New("database connection failed; check local configuration") + } + pool, err := db.DB() + if err != nil { + return nil, err + } + pool.SetMaxOpenConns(5) + pool.SetMaxIdleConns(2) + var version string + if err = db.Raw("SELECT VERSION()").Scan(&version).Error; err != nil || !strings.HasPrefix(version, "8.") { + pool.Close() + return nil, errors.New("MySQL 8 is required") + } + return db, nil +} + func main() { if err := run(); err != nil { log.Fatal(err) -- 2.34.1 From dde2ac127a2d3f0b7507aec2f0e43150a65f05a7 Mon Sep 17 00:00:00 2001 From: QiuSW Date: Tue, 15 Sep 2026 17:12:02 +0800 Subject: [PATCH 4/4] =?UTF-8?q?docs:=20=E5=90=8C=E6=AD=A5=20#15=20?= =?UTF-8?q?=E4=BA=A4=E4=BB=98=E9=A1=B5=E4=B8=8E=E6=9C=AC=E5=9C=B0=E5=BC=80?= =?UTF-8?q?=E5=8F=91=E9=A1=B5=E9=95=9C=E5=83=8F=20(#15)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit docs/04 与线上 Wiki 的纯二进制安装与交叉验证记录保持一致，revision fa4c663fabf5db418104fc53b7ce00f41f5ba910。 --- docs/04-local-development-and-verification.md | 190 +++++++++++++++++- 1 file changed, 188 insertions(+), 2 deletions(-) diff --git a/docs/04-local-development-and-verification.md b/docs/04-local-development-and-verification.md index 6e4bbfa..3f8a293 100644 --- a/docs/04-local-development-and-verification.md +++ b/docs/04-local-development-and-verification.md @@ -2,8 +2,8 @@ generated: true (请先修改 Gitea Wiki，禁止直接编辑本文件) wiki_page: Local-Development-and-Verification wiki_url: https://git.ilapage.cn/OPC/lexgo/wiki/Local-Development-and-Verification.- -wiki_revision: caa80888cf42fd7fd7f040e9fe1147203fe7bfe3 -synchronized_at: 2026-09-15T08:27:38Z +wiki_revision: fa4c663fabf5db418104fc53b7ce00f41f5ba910 +synchronized_at: 2026-09-15T09:11:27Z # 本地开发与验证 @@ -560,3 +560,189 @@ node --test spikes/english/view.test.mjs - 未做定时备份、监控与告警；备份与恢复均为人工触发。 - 大库恢复耗时与磁盘空间上限未测；演练库与生产库规模不同。 - 附件（封面/音频，#21）尚未实现，恢复范围目前只覆盖数据库。 + +## #15 交付、备份与恢复验证（2026-09-15） + +本单**无 schema 变化、无迁移**：新增运维脚本、部署文档与演练证据。所有命令在仓库根执行；MySQL 客户端用 8.4（`LEXGO_MYSQL_BIN=D:/mysql843/bin`），因为本机 PATH 上是 5.7 客户端，`install-check` 会把它判为不达标。 + +### 运维命令 + +纯二进制路径（部署机不需要 Python）： + +| 用途 | 命令 | +|---|---| +| 迁移 / 初始化管理员 | `./lexgo migrate` / `./lexgo bootstrap` | +| 备份 | `./lexgo backup --out <目录>` | +| 恢复到空库 | `./lexgo restore --dump <备份.sql.gz> --database <库名> --confirm` | +| 校验 | `./lexgo verify --database <库名> [--manifest ]` | + +工具通道（开发便利与交叉验证）： + +| 用途 | 命令 | +|---|---| +| 依赖与资源检查 | `python scripts/ops.py install-check` | +| 建空库并提示权限 | `python scripts/ops.py init-database --database <库名>` | +| 备份 | `python scripts/ops.py backup --out <目录>` | +| 恢复到空库 | `python scripts/ops.py restore --dump <备份.sql.gz> --database <库名> --confirm` | +| 校验（数据完整性，可选接口闭环） | `python scripts/ops.py verify --database <库名> [--manifest ] [--api http://127.0.0.1:<端口> --user <前缀> --password-env <变量名>]` | +| 干净实例两账号闭环演练 | `python scripts/ops.py smoke --api --admin-user <账号> --user <前缀>` | +| 性能测量 | `python scripts/bench.py --database <库名> --api --user <账号>` | + +### 两条通道的交叉验证（2026-09-15，19 项检查全部通过） + +Go 二进制与 Python 工具各做一次备份，并**互相恢复对方的备份**： + +1. 两条通道的 `manifest.json` 字段齐备、逐表行数一致、`schema_version` 一致、都不含凭据字段； +2. Go 产出的 dump 由 Python 恢复成功，Python 产出的 dump 由 Go 恢复成功（`lexgo restore` 输出 `verified`）； +3. Go 校验 Go 恢复出来的库、Python 校验 Python 恢复出来的库，逐表行数与 manifest 一致； +4. 两个恢复实例的**逐表内容校验和都与源库一致**（13 张表），源库在过程中未被改动； +5. `lexgo verify` 可以直接校验源库； +6. 安全边界两条通道各自确认：Go 拒绝覆盖已有库、要求 `--confirm`，Python 拒绝系统库。 + +证据：`.local/issue15-crosscheck-evidence.json`。Go 侧另有 `TestMySQLOpsBackupRestoreRoundTrip` 集成用例（备份 → 恢复到空库 → 校验 → 拒绝二次覆盖 → 源库校验和不变）。 + +### 完整演练（2026-09-15，本机一次性通过，27 项顶层检查全部 `[ok]`） + +演练脚本（`.local/tmp/issue15-drill.py`，一次性工具）依次执行： + +1. **干净安装**：`init-database` 建空库 → `migrate`（schema v7，14 张表）→ 确认「新库没有账号、没有业务数据」→ `bootstrap` 建管理员 → **再次 `bootstrap` 被拒绝**（退出码 1，不覆盖管理员）→ 起独立 API 实例（:8010）。 +2. **两个演练账号走通闭环**（`ops.py smoke`，19 项全过）：管理员建 `trial_a`/`trial_b` → 新账号书库与进度为空 → 粘贴章节 → 章节处理完成 → 分词返回可点选词 → 词典查询返回明确状态 → 保存词义 → 另一账号查不到该词条 → 新词立刻到期 → 作答成功 → 重复作答不重复记账 → 完成章节只记已读 → 进度反映活动；`B` 读 `A` 的书 404、书库仍为空。 +3. **备份 lexgo_dev**（约 12 MB gz）：manifest 记录 schema 7、提交哈希、逐表行数、sha256；确认 manifest 无凭据字段；确认 dump 不含 `CREATE DATABASE`/`USE`（不会误写回源库）。 +4. **恢复到空库** `lexgo_restore_drill` 并自动校验（33 项全过）：逐表行数与备份一致（`sys_user` 6、`books` 27、`chapters` 33、`terms` 13、`term_reviews` 13、`review_answers` 55、`login_logs` 192 等）→ **逐表内容校验和与源库一致**（13 张表）→ **源库 lexgo_dev 未被改动**。 +5. **安全边界**：恢复默认拒绝覆盖已有库；拒绝写入系统库 `mysql`；不带 `--confirm` 直接拒绝。 +6. **恢复实例的两账号闭环与越权校验**（`:8011`，41 项全过）——恢复后的账号仍可登录、词条与到期队列仍在、答题与完成章节保持幂等、`B` 无法读取或修改 `A` 的数据。 +7. **收尾**：删除两个演练库，确认 `lexgo_dev` 与测试库保留，`lexgo_dev` 仍是 schema 7。 + +### 性能观察（`scripts/bench.py`，人造数据集，单机无并发） + +数据集：1 个账号、20 本书 20 章（每章约 500 个词）、2000 词条、8000 条复习作答。环境：Windows 10（10.0.19044）、8 核、31.7 GB 内存、MySQL 8.4.3、Go 工具链由 `scripts/server.py` 固定为 1.26.5。 + +| 接口 | 请求数 | p50 | p95 | 最大 | 错误 | +|---|---|---|---|---|---| +| `POST /login` | 10 | 82.2 ms | 93.3 ms | 93.3 ms | 0 | +| `GET /books` | 30 | 5.9 ms | 18.1 ms | 18.2 ms | 0 | +| `GET /chapters/1` | 30 | 8.3 ms | 20.1 ms | 22.0 ms | 0 | +| `GET /chapters/1/tokens` | 30 | 10.0 ms | 23.6 ms | 30.3 ms | 0 | +| `GET /terms`（第 1 页） | 30 | 10.2 ms | 20.4 ms | 21.5 ms | 0 | +| `GET /terms?query`（搜索） | 30 | 12.3 ms | 24.9 ms | 27.0 ms | 0 | +| `GET /progress` | 30 | 15.7 ms | 26.1 ms | 29.1 ms | 0 | +| `GET /reviews/queue` | 30 | 22.7 ms | 32.3 ms | 34.2 ms | 0 | + +**这些是观察值，不是容量承诺**：单机、单进程、无并发压力；登录接口每地址每分钟限 30 次，因此登录只测 10 次，该数字不代表登录吞吐。原始报告：`.local/issue15-bench.json`。 + +### 集成验证矩阵（复用既有 Go 测试，`LEXGO_TEST_DB_NAME=lexgo_test_issue13` 70 项全过） + +| 验收要求 | 覆盖用例（全部通过） | +|---|---| +| 越权与隔离 | `TestMySQLLibraryIsolationAndOwnership`、`TestMySQLTermIsolationAndInputRules`、`TestMySQLPhraseRulesAndIsolation`、`TestMySQLReviewAnswerErrorsAndOwnership`、`TestMySQLAccountIsolationAndRevocation` | +| 任务崩溃重试 | `TestMySQLIngestRecoveryWithoutRestart`、`TestMySQLIngestRecoveryAfterRestart`、`TestMySQLIngestAttemptsAreBoundedAndManualRetryRestarts`、`TestMySQLRetryAfterContentRestoredPublishesSameChapter`、`TestMySQLRecoverySkipsSupersededJobs` | +| 并发答题与并发删除 | `TestMySQLReviewConcurrentReplayOfOneAnswer`、`TestConcurrentDuplicateAccountHasOneWinner`、`TestMySQLConcurrentChapterDelete`、`TestMySQLDeleteDuringProcessing` | +| 长文与输入上限 | `TestMySQLPasteRejectsInvalidInputAndLimits`、`TestMySQLTextUploadKeepsChapterLimit`、`TestMySQLTextUploadRejectsInvalidSubmissions`、`TestMySQLTextUploadRejectsInvalidSubmissions` | +| 迁移与回退标记 | `TestMigrationRefusesUnownedOrUnsupportedSchema`、`TestMigrationCanResumeOwnedPartialSchema`、全部 `TestMigrationFrom*` | + +### 治理与其它测试 + +`python -m unittest discover -s tests` 65 项通过（新增 9 项 `tests/test_lexgo_ops.py`，不需要数据库）；`harness.py check --strict` 与 `sync --check` 通过。本单未改前端与后端业务代码，仍复跑 `LEXGO_TEST_DB_NAME=lexgo_test_issue13 python scripts/server.py test-integration` 70 项通过。 + +### 未验证 + +- **真实回滚（升级后切回旧二进制）未演练**：文档给出规则（迁移只新增对象，回退前写回版本号），但没有在真实实例上做过升级/回滚全过程。 +- 未验证 HTTPS、域名、反代配置、多机与灰度；本机演练只在 `127.0.0.1` 上起两个临时实例。 +- 未做定时备份、监控与告警；备份与恢复均为人工触发。 +- 大库恢复耗时与磁盘空间上限未测；演练库与生产库规模不同。 +- 附件（封面/音频，#21）尚未实现，恢复范围目前只覆盖数据库。 + +## #15 交付、备份与恢复验证（2026-09-15） + +本单**无 schema 变化、无迁移**：新增运维脚本、部署文档与演练证据。所有命令在仓库根执行；MySQL 客户端用 8.4（`LEXGO_MYSQL_BIN=D:/mysql843/bin`），因为本机 PATH 上是 5.7 客户端，`install-check` 会把它判为不达标。 + +### 运维命令 + +纯二进制路径（部署机不需要 Python）： + +| 用途 | 命令 | +|---|---| +| 迁移 / 初始化管理员 | `./lexgo migrate` / `./lexgo bootstrap` | +| 备份 | `./lexgo backup --out <目录>` | +| 恢复到空库 | `./lexgo restore --dump <备份.sql.gz> --database <库名> --confirm` | +| 校验 | `./lexgo verify --database <库名> [--manifest ]` | + +工具通道（开发便利与交叉验证）： + +| 用途 | 命令 | +|---|---| +| 依赖与资源检查 | `python scripts/ops.py install-check` | +| 建空库并提示权限 | `python scripts/ops.py init-database --database <库名>` | +| 备份 | `python scripts/ops.py backup --out <目录>` | +| 恢复到空库 | `python scripts/ops.py restore --dump <备份.sql.gz> --database <库名> --confirm` | +| 校验（数据完整性，可选接口闭环） | `python scripts/ops.py verify --database <库名> [--manifest ] [--api http://127.0.0.1:<端口> --user <前缀> --password-env <变量名>]` | +| 干净实例两账号闭环演练 | `python scripts/ops.py smoke --api --admin-user <账号> --user <前缀>` | +| 性能测量 | `python scripts/bench.py --database <库名> --api --user <账号>` | + +### 纯二进制安装演练（2026-09-15，9 项检查全部通过） + +不使用 Python 执行安装步骤：建空库 → `lexgo migrate`（v7，14 张表）→ 确认空库无账号 → `lexgo bootstrap` 建管理员 → **再次 bootstrap 被拒绝** → `lexgo verify` 通过 → `lexgo serve` 自身健康检查通过 → 两个演练账号走通学习闭环（19 项）→ 删除演练库。接口层的两账号闭环由工具通道驱动（它需要发 HTTP 请求），这一点在部署页已写明。证据：`.local/issue15-binary-drill-evidence.json`。 + +### 两条通道的交叉验证（2026-09-15，19 项检查全部通过） + +Go 二进制与 Python 工具各做一次备份，并**互相恢复对方的备份**： + +1. 两条通道的 `manifest.json` 字段齐备、逐表行数一致、`schema_version` 一致、都不含凭据字段； +2. Go 产出的 dump 由 Python 恢复成功，Python 产出的 dump 由 Go 恢复成功（`lexgo restore` 输出 `verified`）； +3. Go 校验 Go 恢复出来的库、Python 校验 Python 恢复出来的库，逐表行数与 manifest 一致； +4. 两个恢复实例的**逐表内容校验和都与源库一致**（13 张表），源库在过程中未被改动； +5. `lexgo verify` 可以直接校验源库； +6. 安全边界两条通道各自确认：Go 拒绝覆盖已有库、要求 `--confirm`，Python 拒绝系统库。 + +证据：`.local/issue15-crosscheck-evidence.json`。Go 侧另有 `TestMySQLOpsBackupRestoreRoundTrip` 集成用例（备份 → 恢复到空库 → 校验 → 拒绝二次覆盖 → 源库校验和不变）。 + +### 完整演练（2026-09-15，本机一次性通过，27 项顶层检查全部 `[ok]`） + +演练脚本（`.local/tmp/issue15-drill.py`，一次性工具）依次执行： + +1. **干净安装**：`init-database` 建空库 → `migrate`（schema v7，14 张表）→ 确认「新库没有账号、没有业务数据」→ `bootstrap` 建管理员 → **再次 `bootstrap` 被拒绝**（退出码 1，不覆盖管理员）→ 起独立 API 实例（:8010）。 +2. **两个演练账号走通闭环**（`ops.py smoke`，19 项全过）：管理员建 `trial_a`/`trial_b` → 新账号书库与进度为空 → 粘贴章节 → 章节处理完成 → 分词返回可点选词 → 词典查询返回明确状态 → 保存词义 → 另一账号查不到该词条 → 新词立刻到期 → 作答成功 → 重复作答不重复记账 → 完成章节只记已读 → 进度反映活动；`B` 读 `A` 的书 404、书库仍为空。 +3. **备份 lexgo_dev**（约 12 MB gz）：manifest 记录 schema 7、提交哈希、逐表行数、sha256；确认 manifest 无凭据字段；确认 dump 不含 `CREATE DATABASE`/`USE`（不会误写回源库）。 +4. **恢复到空库** `lexgo_restore_drill` 并自动校验（33 项全过）：逐表行数与备份一致（`sys_user` 6、`books` 27、`chapters` 33、`terms` 13、`term_reviews` 13、`review_answers` 55、`login_logs` 192 等）→ **逐表内容校验和与源库一致**（13 张表）→ **源库 lexgo_dev 未被改动**。 +5. **安全边界**：恢复默认拒绝覆盖已有库；拒绝写入系统库 `mysql`；不带 `--confirm` 直接拒绝。 +6. **恢复实例的两账号闭环与越权校验**（`:8011`，41 项全过）——恢复后的账号仍可登录、词条与到期队列仍在、答题与完成章节保持幂等、`B` 无法读取或修改 `A` 的数据。 +7. **收尾**：删除两个演练库，确认 `lexgo_dev` 与测试库保留，`lexgo_dev` 仍是 schema 7。 + +### 性能观察（`scripts/bench.py`，人造数据集，单机无并发） + +数据集：1 个账号、20 本书 20 章（每章约 500 个词）、2000 词条、8000 条复习作答。环境：Windows 10（10.0.19044）、8 核、31.7 GB 内存、MySQL 8.4.3、Go 工具链由 `scripts/server.py` 固定为 1.26.5。 + +| 接口 | 请求数 | p50 | p95 | 最大 | 错误 | +|---|---|---|---|---|---| +| `POST /login` | 10 | 82.2 ms | 93.3 ms | 93.3 ms | 0 | +| `GET /books` | 30 | 5.9 ms | 18.1 ms | 18.2 ms | 0 | +| `GET /chapters/1` | 30 | 8.3 ms | 20.1 ms | 22.0 ms | 0 | +| `GET /chapters/1/tokens` | 30 | 10.0 ms | 23.6 ms | 30.3 ms | 0 | +| `GET /terms`（第 1 页） | 30 | 10.2 ms | 20.4 ms | 21.5 ms | 0 | +| `GET /terms?query`（搜索） | 30 | 12.3 ms | 24.9 ms | 27.0 ms | 0 | +| `GET /progress` | 30 | 15.7 ms | 26.1 ms | 29.1 ms | 0 | +| `GET /reviews/queue` | 30 | 22.7 ms | 32.3 ms | 34.2 ms | 0 | + +**这些是观察值，不是容量承诺**：单机、单进程、无并发压力；登录接口每地址每分钟限 30 次，因此登录只测 10 次，该数字不代表登录吞吐。原始报告：`.local/issue15-bench.json`。 + +### 集成验证矩阵（复用既有 Go 测试，`LEXGO_TEST_DB_NAME=lexgo_test_issue13` 70 项全过） + +| 验收要求 | 覆盖用例（全部通过） | +|---|---| +| 越权与隔离 | `TestMySQLLibraryIsolationAndOwnership`、`TestMySQLTermIsolationAndInputRules`、`TestMySQLPhraseRulesAndIsolation`、`TestMySQLReviewAnswerErrorsAndOwnership`、`TestMySQLAccountIsolationAndRevocation` | +| 任务崩溃重试 | `TestMySQLIngestRecoveryWithoutRestart`、`TestMySQLIngestRecoveryAfterRestart`、`TestMySQLIngestAttemptsAreBoundedAndManualRetryRestarts`、`TestMySQLRetryAfterContentRestoredPublishesSameChapter`、`TestMySQLRecoverySkipsSupersededJobs` | +| 并发答题与并发删除 | `TestMySQLReviewConcurrentReplayOfOneAnswer`、`TestConcurrentDuplicateAccountHasOneWinner`、`TestMySQLConcurrentChapterDelete`、`TestMySQLDeleteDuringProcessing` | +| 长文与输入上限 | `TestMySQLPasteRejectsInvalidInputAndLimits`、`TestMySQLTextUploadKeepsChapterLimit`、`TestMySQLTextUploadRejectsInvalidSubmissions`、`TestMySQLTextUploadRejectsInvalidSubmissions` | +| 迁移与回退标记 | `TestMigrationRefusesUnownedOrUnsupportedSchema`、`TestMigrationCanResumeOwnedPartialSchema`、全部 `TestMigrationFrom*` | + +### 治理与其它测试 + +`python -m unittest discover -s tests` 65 项通过（新增 9 项 `tests/test_lexgo_ops.py`，不需要数据库）；`harness.py check --strict` 与 `sync --check` 通过。本单未改前端与后端业务代码，仍复跑 `LEXGO_TEST_DB_NAME=lexgo_test_issue13 python scripts/server.py test-integration` 70 项通过。 + +### 未验证 + +- **真实回滚（升级后切回旧二进制）未演练**：文档给出规则（迁移只新增对象，回退前写回版本号），但没有在真实实例上做过升级/回滚全过程。 +- 未验证 HTTPS、域名、反代配置、多机与灰度；本机演练只在 `127.0.0.1` 上起两个临时实例。 +- 未做定时备份、监控与告警；备份与恢复均为人工触发。 +- 大库恢复耗时与磁盘空间上限未测；演练库与生产库规模不同。 +- 附件（封面/音频，#21）尚未实现，恢复范围目前只覆盖数据库。 -- 2.34.1