first commit

research/arxiv/references/skillrouter-methodology.md

@@ -0,0 +1,62 @@
+# SkillRouter: Key Takeaways for LLM Agent Skill Routing
+
+Paper: https://arxiv.org/abs/2603.22455 (Apr 2026, Alibaba)
+Code: https://github.com/zhengyanzhao1997/SkillRouter
+Models: https://huggingface.co/pipizhao/SkillRouter-Embedding-0.6B, SkillRouter-Reranker-0.6B
+
+## Core Finding
+
+At ~80K skill scale with heavy overlap, exposing only name+description causes 31-44pp Hit@1 drop vs full skill text. Full body is THE critical routing signal, not metadata.
+
+## Architecture (1.2B total)
+
+```
+query → SR-Emb-0.6B (bi-encoder) → top-20 from 80K → SR-Rank-0.6B (cross-encoder) → final rank
+```
+
+## Training Recipe
+
+### Data: 37,979 synthetic (query, skill) pairs
+- Skills sampled with category stratification from ~80K pool
+- Queries generated by GPT-4o-mini; prompt forbids revealing skill name
+- Benchmark skills excluded from training
+
+### Hard Negative Mining (10 per query)
+- 4 semantic neighbors (embedding NN)
+- 3 BM25 lexical matches
+- 2 same-category distractors
+- 1 random cross-category
+
+### False Negative Filtering (critical — +4.0pp)
+Three-layer filter removes ~10% of mined negatives:
+1. Name dedup (24,879 pairs)
+2. Body trigram Jaccard > 0.6 (13,860 pairs)
+3. Embedding cosine > 0.92 (326 pairs)
+
+### Loss: Listwise CE >> Pointwise BCE
+- Pointwise: 43.3% Hit@1 (fails because homogeneous candidates get similar scores)
+- Listwise: 74.0% Hit@1 (compares candidates against each other)
+- This is THE key training choice for reranker
+
+### Hyperparams
+- Encoder: InfoNCE τ=0.05, LR 2e-5, batch 8, GA 4, 1 epoch, max 2048 tokens
+- Reranker: Listwise CE τ=1.0, LR 1e-5, 1 epoch, max 4096 tokens
+- Both: single GPU, Qwen3-Emb/Rank-0.6B base
+
+### Input Templates
+- Encoder query: `Instruct: ...\nQuery: <text>` (1500 char cap)
+- Encoder skill: `<name> | <desc:300> | <body:2500>` (no instruction prefix)
+- Reranker: `<Instruct>: ...\n<Query>: ...\n<Document>: <name> | <desc:500> | <body:2000>`
+
+## Results
+| System | Params | Avg Hit@1 | Speed |
+|--------|--------|-----------|-------|
+| Qwen3-Emb-8B + Qwen3-Rank-8B | 16B | 68.0% | 0.32 QPS |
+| SR-Emb-0.6B + SR-Rank-0.6B | 1.2B | 74.0% | 1.83 QPS |
+| SR-Emb-8B + SR-Rank-8B | 16B | 76.0% | - |
+
+## Relevance to Hermes
+- Hermes currently exposes ~100 skills via name+desc in system prompt, full SKILL.md on demand
+- At current scale this works; at 1000+ skills, a routing layer becomes necessary
+- False-negative filtering concept applies to Hermes skill deduplication
+- Listwise reranking matters when many skills look similar (e.g., multiple research skills)