first commit
This commit is contained in:
Hermes Agent
163
research/llm-model-comparison/SKILL.md
Normal file
163
research/llm-model-comparison/SKILL.md
Normal file
@@ -0,0 +1,163 @@
|
||||
---
|
||||
name: llm-model-comparison
|
||||
description: Compare LLM models across benchmarks, pricing, and capabilities. For evaluating new models, recommending providers, and maintaining benchmark knowledge.
|
||||
version: 1.0.0
|
||||
author: Hermes Agent
|
||||
license: MIT
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [llm, benchmark, model-comparison, evaluation, provider-selection]
|
||||
triggers:
|
||||
- user asks "which model is better" or "compare X vs Y"
|
||||
- user asks about a new model they saw in news/早报
|
||||
- user wants to know if they should switch models
|
||||
- user asks "what level is this model" or "is X any good"
|
||||
- selecting a model provider for a new project
|
||||
---
|
||||
|
||||
# LLM Model Comparison Skill
|
||||
|
||||
## When to Use
|
||||
- User asks about a model they saw in news, 早报, or social media
|
||||
- User wants to compare two or more models for a specific use case
|
||||
- User asks "should I switch to X" or "is Y worth it"
|
||||
- Selecting models for deployment, API integration, or fine-tuning
|
||||
- **User asks to elaborate on a model or product mentioned in 橘鸦AI早报 or other news digests**
|
||||
|
||||
## Comparison Framework
|
||||
|
||||
### Step 1: Identify the Question
|
||||
- Is this a "what is it?" question → give overview + positioning
|
||||
- Is this a "should I use it?" question → compare against user's current stack
|
||||
- Is this a "which is better?" question → structured comparison table
|
||||
|
||||
### Step 2: Gather Data
|
||||
Use `mmx search` to find:
|
||||
1. Official announcements and benchmark numbers
|
||||
2. Third-party evaluations (non-linear benchmark, LMSYS, Artificial Analysis)
|
||||
3. Community feedback and real-world usage reports
|
||||
|
||||
Search patterns:
|
||||
```
|
||||
mmx search query "<model name> benchmark MMLU 评测 2026"
|
||||
mmx search query "<model name> vs <model name> comparison"
|
||||
mmx search query "<model name> API pricing performance"
|
||||
mmx search query "<模型中文名> 评测 benchmark"
|
||||
```
|
||||
|
||||
For Chinese platform-specific models (SenseNova, Volcengine, Qwen, etc.), search in Chinese:
|
||||
```
|
||||
mmx search query "商汤 sensenova 模型 评测"
|
||||
mmx search query "火山引擎 doubao 模型列表"
|
||||
```
|
||||
|
||||
See `references/chinese-model-platforms.md` for known provider APIs and model catalogs.
|
||||
|
||||
### Step 3: Structure the Comparison
|
||||
|
||||
Use this table format for multi-model comparison:
|
||||
|
||||
| 维度 | Model A | Model B | Model C |
|
||||
|------|---------|---------|---------|
|
||||
| **开发者** | Company | Company | Company |
|
||||
| **参数规模** | XxB | XxB | XxB |
|
||||
| **架构** | Dense/MoE | Dense/MoE | Dense/MoE |
|
||||
| **开源** | ✅/❌ | ✅/❌ | ✅/❌ |
|
||||
| **中文能力** | ⭐⭐⭐ | ⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
|
||||
| **编程能力** | ⭐⭐⭐ | ⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
|
||||
| **Agent能力** | ⭐⭐⭐ | ⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
|
||||
| **性价比** | 描述 | 描述 | 描述 |
|
||||
|
||||
### Step 4: Scenario-Based Recommendation
|
||||
|
||||
Always end with a scenario table:
|
||||
|
||||
| 场景 | 推荐模型 | 理由 |
|
||||
|------|----------|------|
|
||||
| 中文日常对话 | X | 理由 |
|
||||
| 编程任务 | Y | 理由 |
|
||||
| Agent 开发 | Z | 理由 |
|
||||
| 开源自部署 | W | 理由 |
|
||||
| 成本敏感 | V | 理由 |
|
||||
|
||||
### Step 5: Actionable Next Steps
|
||||
- If user already uses a model, compare against their current stack
|
||||
- Offer to configure the new model in their environment
|
||||
- Note any migration costs or compatibility issues
|
||||
|
||||
## Key Benchmark Sources
|
||||
|
||||
| Source | URL | What it measures |
|
||||
|--------|-----|------------------|
|
||||
| Artificial Analysis | artificialanalysis.ai | Speed, quality, price |
|
||||
| LMSYS Chatbot Arena | lmarena.ai | Human preference (Elo) |
|
||||
| non-linear ReLE | github.com/jeinlee1991/chinese-llm-benchmark | Chinese LLM comprehensive |
|
||||
| SWE-bench Pro | swebench.com | Coding agent capability |
|
||||
| BFCL-V3 | gorilla.cs.berkeley.edu | Function calling |
|
||||
| MMLU | Various | General knowledge |
|
||||
|
||||
## Elaborating on 橘鸦AI早报 Items
|
||||
|
||||
When user says "细说X" or "elaborate on item X" from the daily news digest:
|
||||
|
||||
### Step 1: Find the source
|
||||
```bash
|
||||
# Search session history for the cron output
|
||||
ls ~/.hermes/cron/output/9733a9cabb44/ | sort | tail -5
|
||||
# Read the relevant file
|
||||
cat ~/.hermes/cron/output/9733a9cabb44/<date>.md
|
||||
```
|
||||
|
||||
### Step 2: Extract the specific item
|
||||
Parse the numbered list and identify the item by number.
|
||||
|
||||
### Step 3: Deep research
|
||||
Use `mmx search` to find:
|
||||
1. Official announcements and product pages
|
||||
2. Technical documentation or blog posts
|
||||
3. Community reactions and early adopter feedback
|
||||
4. Benchmark data if applicable
|
||||
|
||||
### Step 4: Structure the response
|
||||
- One-line summary of what it is
|
||||
- Detailed breakdown (features, specs, implications)
|
||||
- Comparison with alternatives if relevant
|
||||
- Actionable recommendation (try it? wait? skip?)
|
||||
|
||||
## Pitfalls
|
||||
|
||||
### Don't compare apples to oranges
|
||||
- MoE models (e.g., 400B total, 13B active) ≠ Dense models of same total params
|
||||
- Always note activated parameters for MoE models
|
||||
- Pricing varies wildly: per-token vs per-request vs subscription
|
||||
|
||||
### Benchmark ≠ real-world performance
|
||||
- Benchmark scores don't capture latency, rate limits, or availability
|
||||
- Chinese benchmark scores may not reflect English performance and vice versa
|
||||
- Agent benchmarks (SWE-bench, τ³-Bench) are more relevant for agentic use cases than MMLU
|
||||
|
||||
### Free tier traps
|
||||
- "Free" models on platforms may have rate limits, latency, or availability issues
|
||||
- Check if the free offer is temporary (e.g., "一周免费") before recommending
|
||||
- Self-hosted "free" models still have compute costs
|
||||
|
||||
### Don't over-hype new releases
|
||||
- New model announcements often cherry-pick favorable benchmarks
|
||||
- Wait for third-party evaluations before making strong claims
|
||||
- If user saw it in 早报/news, note it's worth watching but not necessarily switching
|
||||
|
||||
### ALWAYS use mmx search, NOT curl/browser
|
||||
- **Never** fall back to curl-based scraping (Google, Baidu, DuckDuckGo) for model research — they all block or return empty
|
||||
- **Never** try browser navigation for model research — sandbox issues are common and pages are SPAs
|
||||
- `mmx search` is the only reliable research tool. If it fails, say so and give your best assessment from training data
|
||||
- Do NOT attempt 10+ curl variations hoping one works — one `mmx search` call is worth 20 failed curl attempts
|
||||
|
||||
## Current User Stack (Reference)
|
||||
- Primary model: MiMo 2.5 Pro (via Xiaomi API)
|
||||
- Also available: MiniMax M2.7
|
||||
- Hermes Agent: v0.12.0
|
||||
- Use case: Agent tasks, coding, Chinese content
|
||||
|
||||
## References
|
||||
- See `references/model-benchmarks-2026-05.md` for curated benchmark data
|
||||
- See `references/chinese-model-platforms.md` for Chinese AI provider APIs, model naming conventions, and research heuristics
|
||||
Reference in New Issue
Block a user