ai-daily-report/docs/pipeline-optimization-plan.md

# AI Daily Report Pipeline Optimization Plan

## Objective

This project should become a stable, long-running AI daily report system for Hermes, OpenClaw, and similar agents. The goal is not only to keep the current script runnable, but to make the whole pipeline observable, replayable, maintainable, and safe to run on a daily schedule.

The recommended direction is:

```text
stable core library + CLI + skill wrapper
```

Core business logic should live in deterministic code. The skill should describe how agents run, diagnose, replay, publish, and extend the pipeline.

## Stage Model

Use this stage model going forward:

```text
Stage 0: Collect Sources
Stage 1: Normalize Items
Stage 2: Hard Dedup
Stage 3: Semantic Dedup
Stage 4: Rewrite Titles and Summaries
Stage 5: Classify and Order
Stage 6: Guide and Daily Threads
Stage 7: Assemble and Validate Markdown
Stage 8: Publish and Deliver
```

The current script names script-level deduplication as Stage 0. That should be treated as old terminology. In the long-term pipeline, the first stage is source collection.

## Architecture

Recommended structure:

```text
ai-daily-report/
├── ai_daily_report/
│   ├── models.py
│   ├── sources/
│   │   ├── aihot.py
│   │   ├── rss.py
│   │   ├── juya.py
│   │   └── registry.py
│   ├── collect.py
│   ├── normalize.py
│   ├── dedupe.py
│   ├── llm.py
│   ├── rewrite.py
│   ├── classify.py
│   ├── assemble.py
│   ├── validate.py
│   ├── publish.py
│   └── cli.py
├── config/
│   ├── sources.json
│   └── pipeline.json
├── docs/
├── skill/
│   ├── SKILL.md
│   ├── scripts/
│   └── references/
├── tests/
│   └── fixtures/
└── script/
    └── ai_daily_blog_pipeline.py
```

Keep `script/ai_daily_blog_pipeline.py` as a compatibility entrypoint during migration, but move implementation into importable modules.

## Data Model

### SourceResult

Every data source should return a structured result:

```json
{
  "source": "AI HOT",
  "role": "primary",
  "ok": true,
  "status": "ok",
  "items": [],
  "error": null,
  "elapsed_ms": 820,
  "retry_count": 0,
  "fetched_at": "2026-06-04T10:00:00+08:00"
}
```

Supported statuses:

```text
ok
empty
not_ready
timeout
http_error
parse_error
disabled
```

### NewsItem

All raw source items should be normalized into one structure:

```json
{
  "id": "item_...",
  "source_group": "AI HOT",
  "source_label": "OpenAI: Blog",
  "source_role": "primary",
  "source_priority": 10,
  "title_raw": "...",
  "title_norm": "...",
  "summary_raw": "...",
  "title": null,
  "summary": null,
  "url": "...",
  "canonical_url": "...",
  "published_at": "...",
  "collected_at": "...",
  "origin_type": "aihot_json",
  "section_hint": "...",
  "section": null,
  "language_hint": "zh",
  "quality_flags": [],
  "duplicate_sources": []
}
```

Do not overwrite raw fields with LLM output. Keep display fields separate.

## Stage 0: Collect Sources

### Goal

Collect candidate news from all configured sources in a stable, observable, and recoverable way.

### Design

Use a primary-plus-supplement model at the quality layer, and parallel execution at the scheduling layer.

```text
Quality layer:
AI HOT = primary source
RSS / Juya / InfoQ / QbitAI / MIT = supplement sources

Execution layer:
start all sources concurrently with per-source timeout, retry, and reporting
```

### Source Config

Example:

```json
{
  "name": "AI HOT",
  "type": "aihot",
  "role": "primary",
  "required": true,
  "priority": 10,
  "timeout_seconds": 20,
  "retries": 2,
  "min_items": 10,
  "enabled": true
}
```

Supplement source example:

```json
{
  "name": "Juya AI Daily",
  "type": "juya_rss",
  "url": "https://imjuya.github.io/juya-ai-daily/rss.xml",
  "role": "supplement",
  "required": false,
  "priority": 20,
  "timeout_seconds": 45,
  "retries": 2,
  "enabled": true
}
```

### Optimizations

- Run supplement sources concurrently.
- Do not let one slow source block the whole pipeline.
- Replace the fixed Juya `sleep(120)` with bounded short retries and a clear `not_ready` or `timeout` status.
- Treat AI HOT 404 as "not ready" rather than a generic failure.
- Allow degraded generation if the primary source has a temporary network failure and supplement sources are usable.
- Persist raw source results for replay.

### Artifacts

```text
source_results.json
raw_items.json
stage0_collect_report.json
```

## Stage 1: Normalize Items

### Goal

Convert heterogeneous source output into clean, comparable, traceable `NewsItem` objects.

### Optimizations

- Normalize text with HTML stripping, entity decoding, whitespace cleanup, and RSS boilerplate removal.
- Generate stable `id` values from canonical URL when possible, otherwise from source, normalized title, and date.
- Canonicalize URLs:
  - Lowercase scheme and host.
  - Remove `utm_*`, `fbclid`, `gclid`, `spm`, `from`, and fragments.
  - Normalize trailing slashes.
  - Normalize `twitter.com` and `x.com` URLs.
- Generate `title_norm`:
  - Unicode NFKC normalization.
  - Lowercase English text.
  - Normalize whitespace and weak punctuation.
  - Preserve numbers, versions, model names, and product names.
- Standardize source labels:
  - X links as `X:@username`.
  - Official blogs as `OpenAI: Blog`, `Google Research: Blog`, etc.
  - Avoid generic labels such as "technology media" when a domain label is available.
- Add `quality_flags` instead of silently dropping items:
  - `missing_url`
  - `missing_summary`
  - `short_title`
  - `bad_url`
  - `old_item`
  - `parse_suspect`

### Non-goals

- Do not dedupe.
- Do not rewrite content.
- Do not call the LLM.
- Do not remove items based on importance.

### Artifacts

```text
normalized_items.json
stage1_normalize_report.json
```

## Stage 2: Hard Dedup

### Goal

Remove only high-confidence duplicates with deterministic rules. Mark uncertain similarities for Stage 3.

### Rules

High-confidence removal:

- Same canonical URL.
- Same normalized title.
- Same platform entity, such as the same X status ID.
- Same source and same exact normalized title.

Uncertain cases:

- Similar title but different URL.
- Same company or model, but unclear whether the event is identical.
- Same topic across multiple sources with different factual details.

Uncertain cases should go to `possible_duplicates`, not be removed.

### Replacement for Current Logic

The current `SequenceMatcher > 0.7` direct deletion is too risky. Replace it with:

- Exact deterministic deletion.
- Similarity-based candidate marking only.

### Keep Item Selection

When merging a duplicate group, choose the item with a local score:

```text
official source bonus
+ primary source bonus
+ source priority
+ has URL
+ has summary
+ has section hint
+ newer published_at
- quality flag penalty
```

Attach removed items to `duplicate_sources` on the kept item.

### Artifacts

```text
deduped_items.json
stage2_dedupe_report.json
```

## Stage 3: Semantic Dedup

### Goal

Use the LLM to identify semantic duplicates that deterministic rules cannot safely remove.

### Principles

- The LLM judges duplicate candidates; local code enforces safety.
- The LLM must not select, curate, or remove items by importance.
- Only remove `confidence = high` duplicate groups.
- Treat medium or uncertain results as non-removal.

### Input

Prefer candidate groups from Stage 2. Avoid sending all items at once unless the item count is small.

Example item payload:

```json
{
  "id": "item_123",
  "title": "...",
  "summary": "...",
  "source": "QbitAI",
  "url_host": "qbitai.com",
  "published_at": "...",
  "section_hint": "Company and Capital"
}
```

### Output Schema

```json
{
  "duplicate_groups": [
    {
      "keep_id": "item_123",
      "remove_ids": ["item_456"],
      "confidence": "high",
      "reason": "Both items report the same concrete event."
    }
  ],
  "not_duplicates": [],
  "uncertain": []
}
```

### Safety Checks

- Validate all IDs exist.
- Validate confidence values.
- Apply local keep-item scoring instead of blindly trusting `keep_id`.
- Skip deletion if the deletion ratio exceeds a configured threshold.
- Skip deletion when versions, product names, or dates conflict.

### Failure Behavior

If timeout, JSON parse failure, or schema validation failure occurs, keep Stage 2 output and continue.

### Artifacts

```text
semantic_dedup_input.json
semantic_dedup_output.json
stage3_semantic_dedup_report.json
```

## Stage 4: Rewrite Titles and Summaries

### Goal

Produce concise, accurate Chinese display titles and summaries.

### Rules

- Keep `title_raw` and `summary_raw` unchanged.
- Write display fields to `title` and `summary`.
- Preserve brand names, model names, API names, and common technical acronyms in English.
- Translate the rest into natural Chinese.
- Avoid marketing words such as "heavyweight", "explosive", or "just now" unless they are factual and necessary.
- Summaries should be factual, concise, and usually 80-140 Chinese characters.
- Do not add facts not present in the raw title or summary.
- Do not write advice or commentary.

### Batch Strategy

- Process 8-12 items per batch.
- Allow limited parallel batches.
- Retry a failed batch once.
- Fall back per item or per batch if needed.

### Validation

Check:

- Non-empty title and summary.
- No markdown links in title.
- No URL in summary.
- No `[N]` or reference markers.
- No emoji.
- Summary length under limit.
- Key numbers, versions, and model names are preserved when present in raw input.

### Artifacts

```text
rewritten_items.json
rewrite_llm_outputs.json
stage4_rewrite_report.json
```

## Stage 5: Classify and Order

### Goal

Place each item into a stable section and order items for readable scanning.

### Recommended Sections

Use a fixed section whitelist:

```text
模型与能力
产品与应用
开发与基础设施
公司与资本
政策与安全
论文与研究
观点与教程
人物与动态
```

Hide empty sections. Do not create dynamic section names.

### Classification Strategy

Use a three-layer approach:

1. Source hint mapping.
2. Local rule fallback.
3. LLM classification for ambiguous items only.

Example alias mapping:

```text
模型发布/更新 -> 模型与能力
产品发布/更新 -> 产品与应用
产品与工具 -> 产品与应用
开发与工程 -> 开发与基础设施
行业动态 -> 公司与资本
行业与公司 -> 公司与资本
论文研究 -> 论文与研究
技巧与观点 -> 观点与教程
人物与花絮 -> 人物与动态
```

### Ordering Strategy

Do not let the LLM freely order all items. Use local scoring:

```text
rank_score =
  source priority
  + official source bonus
  + primary source bonus
  + recency score
  + key metric bonus
  + duplicate source bonus
  - quality flag penalty
```

Ordering is for readability only. It must not remove items.

### Artifacts

```text
classified_items.json
stage5_classify_order_report.json
```

## Stage 6: Guide and Daily Threads

### Goal

Generate a concise top guide and a bottom "daily threads" section that helps readers understand the day's shape without turning the report into an investment memo.

### Replace Current Summary Style

Do not use:

```text
强信号 / 中信号 / 待验证
```

This style feels too much like an industry rating or investment brief.

Use:

```text
导览
今日脉络
仍待确认, when needed
```

### Output Schema

The LLM should output structured JSON, not Markdown:

```json
{
  "theme": "One concise daily theme.",
  "threads": [
    {
      "title": "模型能力继续向长上下文、实时语音、多模态生成推进",
      "text": "MiniMax M3、Miso One、Ideogram v4.0 分别从长上下文解码、语音克隆和图像生成质量上更新能力边界。",
      "item_ids": ["item_1", "item_2", "item_3"],
      "kind": "thread"
    },
    {
      "title": "仍待确认",
      "text": "融资传闻、排行榜和单源爆料类消息需要等待官方或更多来源确认。",
      "item_ids": ["item_8"],
      "kind": "uncertain"
    }
  ]
}
```

### Rules

- Theme should be one paragraph under 120 Chinese characters.
- Threads should be 2-4 items.
- Each thread must bind to existing `item_ids`.
- Do not add facts absent from the item list.
- Do not write advice.
- Do not include reference numbers.
- Do not include Markdown blockquote syntax. Stage 7 will render Markdown.

### Failure Behavior

- If theme generation fails, omit the guide or use a conservative fallback.
- If threads fail, omit `今日脉络`.
- Invalid thread IDs should drop that thread.

### Artifacts

```text
guide_input.json
guide_output.json
stage6_guide_report.json
```

## Stage 7: Assemble and Validate Markdown

### Goal

Render final Markdown deterministically and validate it before publishing.

### Recommended Structure

```markdown
## 导览

> 一句话主线。

## 模型与能力

**1. 新闻标题**

> 新闻摘要。[来源 ↗](https://example.com)

## 今日脉络

- **主题**
  说明...
```

### Rendering Rules

- Render Markdown in code only.
- Use global continuous numbering.
- Hide empty sections.
- Add blockquote syntax for the guide in code.
- Strip any leading `>` from LLM-provided theme text before rendering.
- Use source links consistently:

```markdown
[OpenAI: Blog ↗](https://example.com)
```

If URL is unavailable, render the source label without a link.

### Auto-fixes

- Remove `> >`.
- Remove `[N]` and numeric reference markers.
- Remove code fences from guide/thread text.
- Normalize extra blank lines.
- Add missing Chinese punctuation to summaries.
- Remove `主线判断:` prefixes if present.

### Blocking Checks

Block publish or downgrade to draft when:

- Item count is zero.
- No sections are rendered.
- Markdown is abnormally short.
- Section name is outside the whitelist.
- JSON fragments remain in Markdown.
- Link formatting is broadly broken.
- Forbidden advisory language appears in guide/thread text.

### Artifacts

```text
blog_markdown.md
stage7_markdown_report.json
```

## Stage 8: Publish and Deliver

### Goal

Publish only validated Markdown, verify the public page, and make the operation idempotent and recoverable.

### Modes

```text
dry-run
draft
publish
```

### Requirements

- Do not publish when Stage 7 has blocking errors.
- Use a deterministic slug such as `ai-YYYY-MM-DD`.
- Check whether the slug already exists before creating a new post.
- Support existence strategies:
  - `skip`
  - `update-draft`
  - `replace`
  - `republish`
- Verify the public URL with retries.
- Preserve Markdown and reports when publishing fails.
- Support publishing from an existing run directory.

### Artifacts

```text
stage8_publish_report.json
run_report.json
```

## Run Directory

Every run should write to an isolated directory:

```text
runs/2026-06-04/
  source_results.json
  raw_items.json
  stage0_collect_report.json
  normalized_items.json
  stage1_normalize_report.json
  deduped_items.json
  stage2_dedupe_report.json
  semantic_dedup_output.json
  stage3_semantic_dedup_report.json
  rewritten_items.json
  stage4_rewrite_report.json
  classified_items.json
  stage5_classify_order_report.json
  guide_output.json
  stage6_guide_report.json
  blog_markdown.md
  stage7_markdown_report.json
  stage8_publish_report.json
  run_report.json
```

This makes the pipeline replayable and debuggable.

## CLI

Provide agent-friendly commands:

```bash
ai-daily-report run --date today --mode publish
ai-daily-report run --date today --mode dry-run
ai-daily-report run --date 2026-06-04 --mode draft
ai-daily-report replay --run-id 2026-06-04 --from-stage 4
ai-daily-report publish --from-run 2026-06-04
ai-daily-report status --date 2026-06-04
```

The current cron can keep invoking the compatibility script, which should delegate to the CLI.

## Skill Strategy

Create or update an `ai-daily-report` skill for Hermes/OpenClaw. The skill should not contain business logic. It should provide:

- How to run daily generation.
- How to dry-run.
- How to replay from an existing run.
- How to publish already generated Markdown.
- How to diagnose source, LLM, Markdown, or publish failures.
- How to add a new RSS source.
- How to adjust output style without breaking the pipeline.

Suggested skill references:

```text
skill/references/sources.md
skill/references/output-style.md
skill/references/troubleshooting.md
skill/references/llm-config.md
```

## Testing

Add fixtures and tests for:

- AI HOT sample parsing.
- RSS parsing.
- Juya `content:encoded` parsing.
- URL canonicalization.
- Title normalization.
- Deterministic deduplication.
- LLM JSON schema validation.
- Rewrite output validation.
- Section alias mapping.
- Markdown rendering.
- Markdown validation.
- Publish dry-run behavior.

Start with local fixture tests. They will give most of the stability benefit without needing live network calls.

## Migration Plan

### Phase 1: Stabilize Current Script

- Add run directories.
- Add SourceResult and stage reports.
- Add URL canonicalization.
- Replace risky Stage 0 dedupe with hard dedup.
- Add Markdown validation and auto-fixes.

### Phase 2: Improve Quality

- Add semantic dedup schema and safety checks.
- Batch rewrite title and summary.
- Add section alias mapping and rule-first classification.
- Replace the current summary with `今日脉络`.

### Phase 3: Modularize

- Extract modules under `ai_daily_report/`.
- Add CLI.
- Keep old script as compatibility entrypoint.
- Add fixture tests.

### Phase 4: Skill Integration

- Update `skill/SKILL.md`.
- Add references for sources, style, troubleshooting, and LLM config.
- Make Hermes/OpenClaw call the CLI.

## Success Criteria

The optimized pipeline should satisfy:

- A usable Markdown report is generated whenever enough source data exists.
- Optional source failures degrade the run but do not stop it.
- LLM failures degrade individual stages but do not destroy the whole report.
- No non-duplicate item is removed by importance or editorial selection.
- Every removed duplicate has a reason.
- Every stage writes inspectable artifacts.
- A failed publish can be retried from an existing run.
- Agents can run, diagnose, replay, and publish via stable commands.