Files
sub2api-cn-relay-manager/docs/plans/2026-05-22-batch-auto-import-v2-implementation-plan.md

785 lines
21 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Batch Auto-Import V2 Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
**Goal:** 实现 V2 的 URL + key 批量导入能力,覆盖模型发现、同模型别名归并、重复导入复用、异步确认、最终 gateway 验证、结果 API 与结果页所需状态投影。
**Architecture:** 采用 `BatchImportService + ConfirmationWorker + ValidationService + RunStateStore + ResultProjection` 分层架构。V2 只以 `import_runs / import_run_items / import_run_item_events` 作为运行态真相,旧 `import_batches/*` 仅保留 legacy linkage。重复导入决策基于 `provider_id + api_key_fingerprint + canonical_model_family`,最终可用性只认宿主真实 `/v1/chat/completions`
**Tech Stack:** Go 1.22.2、`database/sql` + SQLite、Chi、OpenAPI 3.1、Go `testing``httptest`、现有 `internal/host/sub2api` 适配层与 `tests/integration` 集成测试套件。
---
## 0. 实施约束
- 只通过宿主 HTTP API 工作,不直写宿主数据库。
- 所有状态枚举、字段名、API 路由必须遵循当前 canonical contract。
- 每个任务都先写失败测试,再做最小实现,再跑验证。
- 每个任务独立提交,避免大而混杂的 commit。
- 任何 UI/API 展示都只能读 V2 canonical state不得回退到日志拼接。
## 1. 任务总览
```text
T1 Canonical types and enums
T2 Probe models + alias normalization + canonical family
T3 Capability profile + smoke completion routing
T4 Provider ID + reuse policy
T5 Run/item/event state store repositories
T6 BatchImportService: Stage 0~2
T7 ConfirmationWorker + retry + lease
T8 ValidationService + access status
T9 ResultProjection
T10 HTTP API: runs/items
T11 CLI: batch-import
T12 Integration + contract verification
T13 Design restoration audit
```
## 2. 设计还原验证矩阵
### 2.1 目标覆盖矩阵
| 设计目标 | 对应任务 | 验证方式 |
|---|---|---|
| URL + key 自动发现模型 | T2, T6, T12 | `/v1/models` 拉取、集成测试 |
| 模型纠错与别名归一化 | T2, T4, T9, T12 | unit + item detail projection |
| 同模型跨中转快速识别 | T2, T4, T12 | `canonical_model_family` 测试 |
| 重复导入自动复用 | T4, T6, T9, T12 | reuse decision + projection |
| 已启用重复账号直接复用 | T4, T6, T9, T12 | `matched_account_state=active` |
| 已停用/已弃用账号快速启用 | T4, T6, T7, T9, T12 | `account_resolution=reactivated` |
| transport + model capability profile | T3, T9, T10, T12 | profile persistence + API schema |
| channel/account 演化 | T6, T12 | patch contract + host stub |
| 异步确认与重试 | T7, T12 | lease/retry/event trail |
| gateway completion 最终判定 | T8, T12 | `access_status` 唯一写入 |
| 结果 API 与结果页数据源 | T5, T9, T10, T12 | run/item/event projection |
| 单一状态源 | T5, T7, T8, T9 | 只读 `import_runs/*` |
### 2.2 契约覆盖矩阵
| 契约 | 对应任务 |
|---|---|
| `run_id / item_id / provider_id` | T1, T4, T5 |
| `run.state` | T1, T5, T9 |
| `current_stage / confirmation_status / access_status` | T1, T5, T7, T8 |
| `matched_account_state / account_resolution` | T4, T5, T6, T9, T10 |
| `api_key_fingerprint` | T4, T5, T6 |
| `canonical_model_families` | T2, T4, T5, T9, T10 |
| `provision_reused / reused_from_*` | T4, T5, T6, T9, T10 |
| `/api/batch-import/runs*` | T10, T12 |
如果 T1~T12 全部完成并通过验证T13 必须能证明上述矩阵全部为“已覆盖”,否则不得宣称 V2 可按设计实现。
## 3. 实施任务
### Task 1: Canonical Types And Enums
**Files:**
- Create: `internal/batch/types.go`
- Test: `internal/batch/types_test.go`
- Reference: `docs/2026-05-21-BATCH_AUTO_IMPORT_SPEC.md`
**Step 1: Write the failing test**
为以下枚举写失败测试:
- `RunState`
- `ItemStage`
- `ConfirmationStatus`
- `AccessStatus`
- `MatchedAccountState`
- `AccountResolution`
至少覆盖:
- 常量值是否与文档一致
- 非法字符串是否会在后续解析层被拒绝
**Step 2: Run test to verify it fails**
Run:
```bash
go test ./internal/batch -run 'TestRunStateConstants|TestItemStateConstants' -count=1
```
Expected: FAIL提示类型或常量不存在。
**Step 3: Write minimal implementation**
`internal/batch/types.go` 中定义上述类型与常量,不提前引入不需要的 helper。
**Step 4: Run test to verify it passes**
Run:
```bash
go test ./internal/batch -run 'TestRunStateConstants|TestItemStateConstants' -count=1
```
Expected: PASS
**Step 5: Commit**
```bash
git add internal/batch/types.go internal/batch/types_test.go
git commit -m "feat(batch): add canonical v2 state enums"
```
### Task 2: Probe Models, Alias Normalization, Canonical Family
**Files:**
- Create: `internal/probe/models.go`
- Create: `internal/probe/aliases.go`
- Test: `internal/probe/models_test.go`
- Test: `internal/probe/aliases_test.go`
- Reference: `docs/2026-05-21-BATCH_AUTO_IMPORT_TDD_PLAN.md`
**Step 1: Write the failing test**
覆盖:
- `/v1/models` OpenAI 格式解析
- 空模型列表
- 鉴权失败
- `kimi 2.6 / kimi-2.6 / kimi-k2.6` 归并到同一 `canonical_model_family`
- `deepseek-ai/DeepSeek-V4-Pro` vendor 前缀归一化
**Step 2: Run test to verify it fails**
Run:
```bash
go test ./internal/probe -run 'TestProviderModels|TestCanonicalModelFamily' -count=1
```
Expected: FAIL
**Step 3: Write minimal implementation**
实现:
- `ProviderModels`
- `NormalizeModelID`
- `CanonicalModelID`
- `CanonicalModelFamily`
- `BuildAliasTable`
- `ResolveRequestedModel`
- `RecommendModels`
**Step 4: Run test to verify it passes**
Run:
```bash
go test ./internal/probe -run 'TestProviderModels|TestCanonicalModelFamily' -count=1
```
Expected: PASS
**Step 5: Commit**
```bash
git add internal/probe/models.go internal/probe/aliases.go internal/probe/models_test.go internal/probe/aliases_test.go
git commit -m "feat(probe): add model discovery and canonical family normalization"
```
### Task 3: Capability Profile And Smoke Completion Routing
**Files:**
- Create: `internal/probe/capability.go`
- Create: `internal/probe/completion.go`
- Test: `internal/probe/capability_test.go`
- Test: `internal/probe/completion_test.go`
- Reference: `docs/2026-05-22-BATCH_AUTO_IMPORT_V2_ARCHITECTURE.md`
**Step 1: Write the failing test**
覆盖:
- `responses` 不支持但 `chat/completions` 可用
- transport profile 的 advisory 记录
- per-model profile 记录
- `ResolveSmokeModel` 基于别名与能力选择 smoke model
**Step 2: Run test to verify it fails**
Run:
```bash
go test ./internal/probe -run 'TestProbeCapabilities|TestResolveSmokeModel|TestSmokeCompletion' -count=1
```
Expected: FAIL
**Step 3: Write minimal implementation**
实现:
- `TransportProfile`
- `ModelCapabilityProfile`
- `CapabilityProfile`
- `ProbeCapabilities`
- `CompletionResult`
- `ResolveSmokeModel`
- `SmokeCompletion`
**Step 4: Run test to verify it passes**
Run:
```bash
go test ./internal/probe -run 'TestProbeCapabilities|TestResolveSmokeModel|TestSmokeCompletion' -count=1
```
Expected: PASS
**Step 5: Commit**
```bash
git add internal/probe/capability.go internal/probe/completion.go internal/probe/capability_test.go internal/probe/completion_test.go
git commit -m "feat(probe): add capability profile and smoke completion routing"
```
### Task 4: Provider ID And Reuse Policy
**Files:**
- Create: `internal/batch/provider_id.go`
- Create: `internal/batch/reuse_policy.go`
- Test: `internal/batch/provider_id_test.go`
- Test: `internal/batch/reuse_policy_test.go`
- Reference: `docs/2026-05-21-BATCH_AUTO_IMPORT_SPEC.md:336`
**Step 1: Write the failing test**
覆盖:
- 同 host 不同 path 生成不同 `provider_id`
- 已存在 active provider 且 family 已覆盖 -> `reused`
- 已存在 active account -> `matched_account_state=active`, `account_resolution=reused`
- `disabled/deprecated` 账号 -> `reactivated`
- `broken` provider/account -> `replace`
- 同 family 不同 alias -> 视为已覆盖
**Step 2: Run test to verify it fails**
Run:
```bash
go test ./internal/batch -run 'TestNormalizeProviderID|TestDecideReuse' -count=1
```
Expected: FAIL
**Step 3: Write minimal implementation**
实现:
- `NormalizeProviderID`
- `ReuseDecision`
- `DecideReuse`
不要在这一步直接改 service。
**Step 4: Run test to verify it passes**
Run:
```bash
go test ./internal/batch -run 'TestNormalizeProviderID|TestDecideReuse' -count=1
```
Expected: PASS
**Step 5: Commit**
```bash
git add internal/batch/provider_id.go internal/batch/reuse_policy.go internal/batch/provider_id_test.go internal/batch/reuse_policy_test.go
git commit -m "feat(batch): add provider id and reuse policy"
```
### Task 5: Run/Item/Event State Store Repositories
**Files:**
- Modify: `internal/store/migrations/0007_batch_import_runs.sql`
- Modify: `internal/store/migrations/0008_batch_import_run_events.sql`
- Modify: `internal/store/sqlite/import_runs_repo.go`
- Create: `internal/store/sqlite/import_run_items_repo.go`
- Create: `internal/store/sqlite/import_run_item_events_repo.go`
- Modify: `internal/store/sqlite/db.go`
- Test: `internal/store/sqlite/import_runs_repo_test.go`
- Test: `tests/integration/store_init_test.go`
**Step 1: Write the failing test**
覆盖:
- run 创建/更新
- item upsert 持久化 `api_key_fingerprint / canonical_model_families / matched_account_state / account_resolution / provision_reused`
- event append/list
- lease 字段持久化
**Step 2: Run test to verify it fails**
Run:
```bash
go test ./internal/store/sqlite/... ./tests/integration/... -run 'TestRunStateStore|TestStoreAppliesLatestMigration' -count=1
```
Expected: FAIL
**Step 3: Write minimal implementation**
补足 repo 与 migration确保 schema 与文档完全一致。
**Step 4: Run test to verify it passes**
Run:
```bash
go test ./internal/store/sqlite/... ./tests/integration/... -run 'TestRunStateStore|TestStoreAppliesLatestMigration' -count=1
```
Expected: PASS
**Step 5: Commit**
```bash
git add internal/store/migrations/0007_batch_import_runs.sql internal/store/migrations/0008_batch_import_run_events.sql internal/store/sqlite/import_runs_repo.go internal/store/sqlite/import_run_items_repo.go internal/store/sqlite/import_run_item_events_repo.go internal/store/sqlite/db.go internal/store/sqlite/import_runs_repo_test.go tests/integration/store_init_test.go
git commit -m "feat(store): complete v2 runtime state repositories"
```
### Task 6: BatchImportService Stage 0~2
**Files:**
- Create: `internal/batch/service.go`
- Create: `internal/batch/capability_profile.go`
- Create: `internal/batch/channel_evolution.go`
- Test: `internal/batch/service_test.go`
- Test: `internal/batch/channel_evolution_test.go`
- Reference: `internal/provision/import_service.go`
**Step 1: Write the failing test**
覆盖:
- 创建 run + items
- reuse preflight 跳过重复 provision
- active 账号重复导入 -> reused
- deprecated 账号重复导入 -> reactivated
- patch-only 新 alias
- legacy batch/provider link 回写
**Step 2: Run test to verify it fails**
Run:
```bash
go test ./internal/batch -run 'TestBatchImport_StartRun|TestModelMappingDelta' -count=1
```
Expected: FAIL
**Step 3: Write minimal implementation**
实现:
- `BatchImportService.StartRun`
- `ImportRoutingStrategy`
- `BuildImportRoutingStrategy`
- `ChannelPatchContract`
- `ModelMappingDelta`
先接现有 `provision.ImportService`,不要提前扩展 UI/API。
**Step 4: Run test to verify it passes**
Run:
```bash
go test ./internal/batch -run 'TestBatchImport_StartRun|TestModelMappingDelta' -count=1
```
Expected: PASS
**Step 5: Commit**
```bash
git add internal/batch/service.go internal/batch/capability_profile.go internal/batch/channel_evolution.go internal/batch/service_test.go internal/batch/channel_evolution_test.go
git commit -m "feat(batch): implement v2 run setup and provision stages"
```
### Task 7: ConfirmationWorker, Lease And Retry
**Files:**
- Create: `internal/batch/confirmation.go`
- Test: `internal/batch/confirmation_test.go`
- Reference: `docs/2026-05-22-BATCH_AUTO_IMPORT_V2_ARCHITECTURE.md:398`
**Step 1: Write the failing test**
覆盖:
- 只捞 `confirm + pending + retry_due + lease_expired`
- `403` probe race -> advisory
- 初次 `503 no available accounts` -> retry -> success
- 多 worker lease 互斥
- `disabled/deprecated` 命中后 reactivated 投影正确
**Step 2: Run test to verify it fails**
Run:
```bash
go test ./internal/batch -run 'TestConfirmationWorker' -count=1
```
Expected: FAIL
**Step 3: Write minimal implementation**
实现:
- `ConfirmationWorker.Tick`
- `ConfirmationWorker.ConfirmItem`
- retry 计划
- lease 生命周期
- advisory event 写入
**Step 4: Run test to verify it passes**
Run:
```bash
go test ./internal/batch -run 'TestConfirmationWorker' -count=1
```
Expected: PASS
**Step 5: Commit**
```bash
git add internal/batch/confirmation.go internal/batch/confirmation_test.go
git commit -m "feat(batch): add confirmation worker and retry handling"
```
### Task 8: ValidationService And Final Access Status
**Files:**
- Create: `internal/batch/validation.go`
- Test: `internal/batch/validation_test.go`
- Reference: `internal/access/closure.go`
**Step 1: Write the failing test**
覆盖:
- `confirmed/advisory + chat 200 -> active`
- exhausted transient -> `degraded`
- definitive invalid path -> `broken`
- 只有 ValidationService 可以写 `access_status`
**Step 2: Run test to verify it fails**
Run:
```bash
go test ./internal/batch -run 'TestValidationService' -count=1
```
Expected: FAIL
**Step 3: Write minimal implementation**
实现:
- `ValidationService.ValidateItem`
- `access_status` 映射
- 对 run summary 的最小更新
**Step 4: Run test to verify it passes**
Run:
```bash
go test ./internal/batch -run 'TestValidationService' -count=1
```
Expected: PASS
**Step 5: Commit**
```bash
git add internal/batch/validation.go internal/batch/validation_test.go
git commit -m "feat(batch): add validation service for final access status"
```
### Task 9: ResultProjection
**Files:**
- Create: `internal/batch/status_projection.go`
- Test: `internal/batch/status_projection_test.go`
- Reference: `docs/2026-05-22-BATCH_AUTO_IMPORT_V2_API_SCHEMAS.md`
**Step 1: Write the failing test**
覆盖:
- run summary 聚合
- item summary/detail projection
- warning 文案模板
- `provision_reused` badge
- `matched_account_state / account_resolution` 文案与 badge
**Step 2: Run test to verify it fails**
Run:
```bash
go test ./internal/batch -run 'TestStatusProjection' -count=1
```
Expected: FAIL
**Step 3: Write minimal implementation**
实现:
- run list projection
- item list projection
- item detail projection
- warning/badge mapping
**Step 4: Run test to verify it passes**
Run:
```bash
go test ./internal/batch -run 'TestStatusProjection' -count=1
```
Expected: PASS
**Step 5: Commit**
```bash
git add internal/batch/status_projection.go internal/batch/status_projection_test.go
git commit -m "feat(batch): add result projection for v2 runs and items"
```
### Task 10: HTTP API For Runs And Items
**Files:**
- Create: `internal/app/http_batch_import.go`
- Create: `internal/app/http_batch_runs.go`
- Modify: `internal/app/http_api.go`
- Test: `internal/app/http_batch_import_test.go`
- Test: `internal/app/http_batch_runs_test.go`
- Reference: `docs/openapi.yaml`
**Step 1: Write the failing test**
覆盖:
- `POST /api/batch-import/runs`
- `GET /api/batch-import/runs`
- `GET /api/batch-import/runs/{run_id}`
- `GET /api/batch-import/runs/{run_id}/items`
- `GET /api/batch-import/runs/{run_id}/items/{item_id}`
- `subscription/self_service` 条件必填
- 列表过滤 `matched_account_state / account_resolution`
**Step 2: Run test to verify it fails**
Run:
```bash
go test ./internal/app -run 'TestBatchImportHTTP|TestBatchRunsHTTP' -count=1
```
Expected: FAIL
**Step 3: Write minimal implementation**
按 OpenAPI 只输出 projection不泄漏 legacy 表结构。
**Step 4: Run test to verify it passes**
Run:
```bash
go test ./internal/app -run 'TestBatchImportHTTP|TestBatchRunsHTTP' -count=1
```
Expected: PASS
**Step 5: Commit**
```bash
git add internal/app/http_batch_import.go internal/app/http_batch_runs.go internal/app/http_api.go internal/app/http_batch_import_test.go internal/app/http_batch_runs_test.go
git commit -m "feat(api): add batch import v2 endpoints"
```
### Task 11: CLI Entry For Batch Import
**Files:**
- Modify: `cmd/cli/main.go`
- Create: `cmd/cli/batch_import.go`
- Test: `cmd/cli/batch_import_test.go`
**Step 1: Write the failing test**
覆盖:
- 参数解析
- `subscription` 必填订阅参数
- `self_service` 必填 `probe_api_key`
- `--confirm-timeout`
- 结果输出 `run_id/result_page`
**Step 2: Run test to verify it fails**
Run:
```bash
go test ./cmd/cli -run 'TestBatchImportCLI' -count=1
```
Expected: FAIL
**Step 3: Write minimal implementation**
实现 CLI 到 V2 API/service 的入口,不在 CLI 层重复业务逻辑。
**Step 4: Run test to verify it passes**
Run:
```bash
go test ./cmd/cli -run 'TestBatchImportCLI' -count=1
```
Expected: PASS
**Step 5: Commit**
```bash
git add cmd/cli/main.go cmd/cli/batch_import.go cmd/cli/batch_import_test.go
git commit -m "feat(cli): add v2 batch import command"
```
### Task 12: Integration And End-To-End Verification
**Files:**
- Create: `tests/integration/batch_import_v2_test.go`
- Modify: `tests/integration/host_stub_test.go`(如需 stub 扩展)
**Step 1: Write the failing test**
至少覆盖 6 条真实业务链:
- 发现模型并归一化
- 重复导入 active 账号 -> reused
- deprecated 账号 -> reactivated
- 同 family 不同 alias -> patch_only
- probe race + warmup retry -> advisory + active
- run/item/event 详情可从 V2 新表完全读出
**Step 2: Run test to verify it fails**
Run:
```bash
go test ./tests/integration/... -run 'TestBatchImportV2' -count=1
```
Expected: FAIL
**Step 3: Write minimal implementation**
补齐 host stub、fake adapter、seed data确保每条链路都可复现。
**Step 4: Run test to verify it passes**
Run:
```bash
go test ./tests/integration/... -run 'TestBatchImportV2' -count=1
```
Expected: PASS
**Step 5: Commit**
```bash
git add tests/integration/batch_import_v2_test.go tests/integration/host_stub_test.go
git commit -m "test(integration): cover batch import v2 flows"
```
### Task 13: Design Restoration Audit
**Files:**
- Create: `docs/2026-05-22-BATCH_AUTO_IMPORT_V2_RESTORATION_CHECKLIST.md`
- Modify: `docs/EXECUTION_BOARD.md`
**Step 1: Write the failing audit checklist**
列出必须逐项勾选的设计恢复项:
- 8 项 Objective
- canonical contract
- 结果 API
- migration
- worker/retry/lease
- reuse/reactivation
**Step 2: Run verification to identify gaps**
Run:
```bash
go test ./... -count=1
go test ./tests/integration/... -count=1
go test -cover ./internal/... -count=1
go vet ./...
gofmt -l .
```
Expected: 在实现完成前,这一步用来发现剩余设计缺口;在最终完成时必须全绿。
**Step 3: Write the audit artifact**
将每一项设计要求映射到:
- 代码文件
- 测试文件
- API 路由
- 状态字段
**Step 4: Update board with true gate**
在执行板中明确:
- 哪些任务完成
- 哪些设计要求已还原
- 是否可宣称“V2 设计已被完整实现”
**Step 5: Commit**
```bash
git add docs/2026-05-22-BATCH_AUTO_IMPORT_V2_RESTORATION_CHECKLIST.md docs/EXECUTION_BOARD.md
git commit -m "docs(v2): add restoration checklist and completion gate"
```
## 4. 全局验证门禁
完成 T1~T13 后,必须一次性通过:
```bash
gofmt -l .
go vet ./...
go test ./... -count=1
go test ./tests/integration/... -count=1
go test -cover ./internal/... -count=1
```
额外检查:
- `docs/openapi.yaml` 与 handler 响应字段一致
- `import_runs/*` 足以支撑结果页,不依赖 legacy 表拼接
- `matched_account_state / account_resolution / provision_reused` 能在 item detail 里直接读到
- `canonical_model_family` 能把同模型别名判定为同一族
## 5. 计划完整性结论
这份计划只有在满足以下条件时,才算“任务可以完全还原规划设计”:
1. T1~T12 实现完成并全部通过验证
2. T13 的还原清单中不存在未映射设计项
3. 任一 Objective 都能指向至少一条:
- 实现任务
- 自动化测试
- API 或状态字段证据
4. 结果页/API 不需要额外新增未规划字段才能解释最终状态
如果 T13 审核时发现任何一项设计要求无法映射到任务或测试,这份计划必须回退修改,不能直接进入实现。
## 6. 推荐提交顺序
建议按以下小步提交:
1. `feat(batch): add canonical v2 state enums`
2. `feat(probe): add model discovery and canonical family normalization`
3. `feat(probe): add capability profile and smoke completion routing`
4. `feat(batch): add provider id and reuse policy`
5. `feat(store): complete v2 runtime state repositories`
6. `feat(batch): implement v2 run setup and provision stages`
7. `feat(batch): add confirmation worker and retry handling`
8. `feat(batch): add validation service for final access status`
9. `feat(batch): add result projection for v2 runs and items`
10. `feat(api): add batch import v2 endpoints`
11. `feat(cli): add v2 batch import command`
12. `test(integration): cover batch import v2 flows`
13. `docs(v2): add restoration checklist and completion gate`
Plan complete and saved to `docs/plans/2026-05-22-batch-auto-import-v2-implementation-plan.md`. Two execution options:
**1. Subagent-Driven (this session)** - I dispatch fresh subagent per task, review between tasks, fast iteration
**2. Parallel Session (separate)** - Open new session with executing-plans, batch execution with checkpoints
Which approach?