agent-skills/dogfood/SKILL.md

name, description, version, metadata

name	description	version	metadata
dogfood	Exploratory QA of web apps: find bugs, evidence, reports.	1.0.0	hermes tags related_skills qa testing browser web dogfood

Dogfood: Systematic Web Application QA Testing

Overview

This skill guides you through systematic exploratory QA testing of web applications. It supports two execution modes depending on context:

1. Browser-first (default): Use browser toolset to navigate, interact, and capture evidence live.
2. Source-code-first (fallback): When browser automation is unavailable, slow, or times out, analyze the source code to enumerate all routes/endpoints/pages, then build a comprehensive test plan document. Execute browser tests selectively afterward.

For multi-service sites (e.g., separate auth/blog/canvas services), prefer the source-code-first approach — it produces more complete coverage faster than crawling.

Prerequisites

• Browser toolset: either the built-in tools (browser_navigate, browser_snapshot, etc.) or Playwright Python (see references/playwright-qa.md) — optional if using source-code-first mode
• A target URL and testing scope from the user
• Source code access (repo clone or codebase) — strongly recommended for multi-service sites

Inputs

The user provides:

1. Target URL — the entry point for testing
2. Scope — what areas/features to focus on (or "full site" for comprehensive testing)
3. Output directory (optional) — where to save screenshots and the report (default: ./dogfood-output)

Workflow

Follow this 5-phase systematic workflow:

Phase 1: Plan

1. Create the output directory structure:

   {output_dir}/
   ├── screenshots/       # Evidence screenshots
   └── report.md          # Final report (generated in Phase 5)
   

2. Identify the testing scope based on user input.
3. Build a rough sitemap by planning which pages and features to test:
   • Landing/home page
   • Navigation links (header, footer, sidebar)
   • Key user flows (sign up, login, search, checkout, etc.)
   • Forms and interactive elements
   • Edge cases (empty states, error pages, 404s)

Phase 2: Explore

For each page or feature in your plan:

1. Navigate to the page:

   browser_navigate(url="https://example.com/page")
   

2. Take a snapshot to understand the DOM structure:

   browser_snapshot()
   

3. Check the console for JavaScript errors:

   browser_console(clear=true)
   

   Do this after every navigation and after every significant interaction. Silent JS errors are high-value findings.

4. Take an annotated screenshot to visually assess the page and identify interactive elements:

   browser_vision(question="Describe the page layout, identify any visual issues, broken elements, or accessibility concerns", annotate=true)
   

   The annotate=true flag overlays numbered [N] labels on interactive elements. Each [N] maps to ref @eN for subsequent browser commands.

5. Test interactive elements systematically:

   • Click buttons and links: browser_click(ref="@eN")
   • Fill forms: browser_type(ref="@eN", text="test input")
   • Test keyboard navigation: browser_press(key="Tab"), browser_press(key="Enter")
   • Scroll through content: browser_scroll(direction="down")
   • Test form validation with invalid inputs
   • Test empty submissions

6. After each interaction, check for:

   • Console errors: browser_console()
   • Visual changes: browser_vision(question="What changed after the interaction?")
   • Expected vs actual behavior

Phase 3: Collect Evidence

For every issue found:

1. Take a screenshot showing the issue:

   browser_vision(question="Capture and describe the issue visible on this page", annotate=false)
   

   Save the screenshot_path from the response — you will reference it in the report.

2. Record the details:

   • URL where the issue occurs
   • Steps to reproduce
   • Expected behavior
   • Actual behavior
   • Console errors (if any)
   • Screenshot path

3. Classify the issue using the issue taxonomy (see references/issue-taxonomy.md):

   • Severity: Critical / High / Medium / Low
   • Category: Functional / Visual / Accessibility / Console / UX / Content

Phase 4: Categorize

1. Review all collected issues.
2. De-duplicate — merge issues that are the same bug manifesting in different places.
3. Assign final severity and category to each issue.
4. Sort by severity (Critical first, then High, Medium, Low).
5. Count issues by severity and category for the executive summary.

Phase 5: Report

Generate the final report using the template at templates/dogfood-report-template.md.

The report must include:

1. Executive summary with total issue count, breakdown by severity, and testing scope
2. Per-issue sections with:
   • Issue number and title
   • Severity and category badges
   • URL where observed
   • Description of the issue
   • Steps to reproduce
   • Expected vs actual behavior
   • Screenshot references (use MEDIA:<screenshot_path> for inline images)
   • Console errors if relevant
3. Summary table of all issues
4. Testing notes — what was tested, what was not, any blockers

Save the report to {output_dir}/report.md.

Alternative Workflow: Source-Code-First (for multi-service / slow-browser sites)

When the target site has source code available, or browser automation is too slow/times out:

Step 1: Clone and Map the Codebase

git clone <repo_url> /tmp/qa-target
cd /tmp/qa-target
find . -name "routes*.py" -o -name "main.py" -o -name "pages.py" -o -name "admin.py" | sort

Step 2: Enumerate All Routes

Read each route file and extract:

• HTTP method + path (e.g., GET /posts/{slug})
• Required auth/permissions
• Rate limits
• Form fields and validation rules

Build a complete URL inventory — this is your test matrix.

Step 3: Analyze Static Assets and Templates

Check template files for:

• CSS variable definitions (look for :root blocks)
• JS includes (what scripts are loaded vs missing)
• Encoding issues (BOM markers, leading newlines before DOCTYPE)
• Accessibility: alt attributes, user-scalable, skip links

Step 4: HTTP-Level Testing (no browser needed)

Use curl to test:

• Page loads (HTTP status codes)
• Static asset availability
• Response headers (security headers, CSP)
• Redirect chains (login flows)
• API endpoints (with/without auth cookies)

Step 5: Generate Structured Test Plan

Output a markdown document with:

• Service architecture table
• Test accounts and auth mechanism
• Per-module test case tables: 编号 | 测试内容 | 测试步骤 | 预期结果 | 优先级
• Known issues found during source analysis
• Cross-cutting concerns (consistency, accessibility, security)

Step 6: Selective Browser Execution

Only use browser automation for:

• Login/register interactive flows
• Visual verification of known issues
• Console error capture
• Screenshot evidence

Alternative Workflow: Test Plan Gap Analysis (improving existing plans)

When the user already has a test plan and wants to improve/complete it against source code:

Step 1: Load Both Inputs in Parallel

1. Read the existing test-plan.md
2. Clone or pull the target repo
3. Use delegate_task to analyze ALL route files + security mechanisms in one pass
   - Extract: endpoints, form fields, rate limits, cookie params, CSRF mechanism,
     validation rules, ownership models, public APIs
   - Focus on FEATURES not COVERED by the existing plan

Step 2: Systematic Comparison

For each service, compare source code findings against existing test cases:

• Endpoints: Are all HTTP methods + paths covered?
• Validation rules: Password complexity, username blacklists, email uniqueness, slug format
• Rate limits: Are all limiters documented with correct values?
• Security mechanisms: CSRF token format/expiry, cookie attributes, redirect validation
• APIs: Public JSON APIs, service-to-service APIs, ownership isolation
• Edge cases: CRLF normalization, content size limits, cascading deletes

Step 3: Patch, Don't Rewrite

Use targeted patch edits to add missing test cases within existing sections:

• Insert after the related existing case (e.g., A-015a after A-015)
• Use sub-numbering convention: X-NNNa for insertions between X-0NN and X-0NN+1
• Preserve existing case numbers — never renumber
• Add new subsections only when the entire category is missing (e.g., Service API)

Step 4: Update Statistics

After all patches:

# Count per-module
grep -c '^| H-[0-9]' test-plan.md  # Home
grep -c '^| A-[0-9]' test-plan.md  # Auth
# ... etc for each prefix

# Count total (catches sub-numbered too)
grep -c '^| [A-Z]*-[0-9]' test-plan.md

Update both the header stats table and the footer summary table. Bump the version number.

Step 5: Commit

git add test-plan.md && git commit -m "vN.M: 完善测试计划，新增 X 个测试用例 (old→new)" && git push

Pitfalls for Gap Analysis

• Don't renumber existing cases: Use X-NNNa sub-numbering to insert between existing cases. Renumbering breaks any existing references (issue tracker, test automation).
• Count carefully: grep -c '^| X-[0-9]' misses sub-numbered entries like A-015a. Use '^| [A-Z]*-[0-9]' for total count, but per-module counts with the prefix filter are usually accurate enough.
• Don't duplicate: Check if a concept is already covered under a different name before adding. "草稿可见" and "草稿预览" might be the same test.
• delegate_task for source analysis: Don't read 40+ route files manually. A single delegate_task with a well-structured prompt produces a complete analysis in one pass.

Alternative Workflow: Module-by-Module Testing with Incremental Commits

When the user has an existing test plan (e.g., test-plan.md in a repo) and wants to execute it module by module, committing results after each:

Step 1: Initialize Results Document

Create test-results.md with a summary table and placeholder sections for every module. Include: module name, status (⏳), execution time, and empty test result tables.

Step 2: Test Module → Update → Commit Loop

For each module:

1. Execute tests (curl for HTTP-level, Playwright for browser-level)
2. Update the module's section in test-results.md with results
3. Update the summary table (pass/fail/blocked counts)
4. Add a "模块 N 小结" section with key findings
5. Add a "💡 模块 N 优化建议" section with prioritized recommendations (user explicitly wants these persisted in the document, not just in chat)
6. git add test-results.md && git commit -m "模块N: 通过X/失败Y" && git push
7. Report progress to user before starting next module
8. ⚠️ If any test modified content, restore it BEFORE committing

Step 3: Parallel Delegation

Use delegate_task with 3 parallel tasks for curl-based modules. Each task tests a group of modules and returns JSON results. Browser-based modules must run sequentially.

Pitfalls for Module-by-Module

• Don't wait until the end to commit: Session may break, losing all work
• Restore content after destructive tests: Save state before, verify after
• Rate limiting blocks repeated tests: Test rate-limited endpoints last
• CSRF token sync: Use same cookie jar for GET+POST (see references/multi-service-qa.md)
• Optimization suggestions go in the document, not just the chat: User wants them persisted

Deliverables: Split Test Plan + Issue List

Always split QA output into two separate documents:

1. test-plan.md — Structured test cases with execution steps and expected results
2. issue-list.md — Known issues found during analysis, with severity and fix suggestions

Do NOT merge them into one report. Users need the test plan for execution assignment and the issue list for bug tracking. Each document should be self-contained.

Test Plan Structure

• Service architecture table (services, URLs, ports)
• Test accounts and auth mechanism explanation
• Per-module test case tables: 编号 | 测试内容 | 测试步骤 | 预期结果 | 账号 | 优先级
• Cover both public pages AND admin/management pages
• Include a cross-cutting section for security, consistency, accessibility

Issue List Structure

• Summary table (count by severity level)
• Per-issue entries: module, page, phenomenon, impact, root cause, source code location, fix suggestion, priority
• Consistency matrix (which pages have which features/assets)
• Fix priority recommendation (immediate / soon / backlog)

Comprehensive QA Dimensions Checklist

When the user asks for "full" or "complete" testing, cover ALL of these dimensions. If you only covered page navigation and login flows, the plan is incomplete.

Core (always cover)

1. Functional — Page loads, navigation, CRUD operations, form submissions
2. Auth & Permissions — Login/logout, RBAC, cross-service cookie propagation, admin vs user access
3. Input Validation — Form validation, empty submissions, boundary values, special characters

Security (cover for any site with user input)

4. Cookie Security — HttpOnly, Secure, SameSite attributes; Max-Age; domain scope
5. CSRF Protection — Token presence, double-submit pattern, token expiry, replay resistance
6. Redirect Safety — Open redirect via redirect parameter; validate against allowed domains
7. Rate Limiting — Per-endpoint limits; account lockout; IP-based limits
8. File Upload Safety — Allowed extensions, size limits, filename sanitization, path traversal prevention
9. Input Injection — XSS in user-generated content, SQL injection attempts, path traversal in slugs

Session & State

10. Token Lifecycle — Expiration behavior, role changes mid-session (DB role vs token role), token format validation
11. Concurrent Access — Race conditions on shared resources, optimistic locking

Content & Rendering

12. Edge Case Content — Empty states, very long text, special characters (CJK, emoji), Markdown/LaTeX rendering
13. Encoding — BOM markers, UTF-8 consistency, DOCTYPE prefix cleanliness

SEO & Metadata

14. Meta Tags — <title>, <meta description>, canonical URLs
15. Open Graph — og:title, og:description, og:image, og:url per page
16. Structured Data — robots.txt, sitemap.xml, RSS feed validity

Accessibility

17. WCAG Basics — alt attributes, user-scalable, color contrast, skip-to-content links
18. Keyboard Navigation — Tab order, focus management, ARIA labels

Performance & Compatibility

19. Page Load — Static asset availability, CDN reliability (especially in China), resource count
20. Responsive Design — Breakpoints, mobile layout, touch targets
21. Cross-Browser — Chrome/Firefox/Safari/Edge rendering differences

Operations

22. Health Checks — /health endpoint availability per service
23. Error Handling — 404 pages, 500 error responses, graceful degradation
24. Logging & Audit — Audit trail for admin actions, login attempts

Consistency (cross-service)

25. Asset Inclusion — Which pages include mobile.css, loader.js, etc.
26. Navigation — Which pages have site-wide nav bar
27. Security Headers — X-Content-Type-Options, X-Frame-Options, Referrer-Policy, CSP

Pitfalls

• 🔴 CRITICAL: Always backup content before write operations: When testing CRUD endpoints (save, publish, create, update), the test payload (including XSS test strings, dummy data, empty fields) CAN overwrite real production content. Before any write test:
  1. curl -s -b cookies SITE/admin → extract current content_json / initialContent → save to /tmp/backup_<service>.json
  2. Perform test
  3. Restore original content via Playwright (set form fields + collectFormData() + submit)
  • This is not optional. A session that deletes user content without restoring it is a failed session.
• 🔴 CRITICAL: Restore content IMMEDIATELY after destructive tests: Don't wait until end of session. If a test modifies content, restore it in the same turn. Session interruptions, timeouts, or context limits can prevent later restoration.
• 🔴 CRITICAL: XSS payloads in form fields persist: When you fill a form field with <script>alert(1)</script> for XSS testing, that value gets saved to the database if the form is submitted. Always use Playwright's page.evaluate() to set values directly on form elements, NOT page.fill() which triggers input events that may activate auto-save.
• ⚠️ Do NOT parallelize browser delegate_tasks for QA: Each browser interaction is slow (navigate + snapshot + screenshot = 10-30s). 3 parallel browser tasks will all timeout at 600s. Run browser tests sequentially or use source-code-first mode.
• ⚠️ Curl-only delegate tasks also timeout with large batches: A delegate_task with 30+ curl test cases can hit the 600s limit (each curl call = 1-3s + overhead). Split large test batches into smaller tasks (~15-20 cases each) or use execute_code with from hermes_tools import terminal for direct in-process execution (faster, no delegation overhead).
• ⚠️ Client-side-only validation is a security finding: When CSP blocks inline JS (see script-src-elem pitfall), any validation that only exists in client JS (password strength, field format, confirmation matching) becomes bypassable. Always test registration/submission with curl to verify server-side validation exists independently.
• ⚠️ API authentication order matters: Some endpoints validate request body BEFORE checking authentication, returning 422 (validation error) instead of 401 (unauthenticated). Test: curl -X POST /api/endpoint -d 'invalid' without auth — should get 401, not 422. This is a security issue (leaks endpoint existence and field requirements).
• ⚠️ Fulltext search can silently fail: Search endpoints with mode=fulltext may return 0 results while mode=simple works fine. Always test both modes with the same query. Common causes: search index not built, tokenizer (jieba) not installed, BM25 ranking misconfigured.
• ⚠️ Rate limiting blocks subsequent tests: Registration endpoints with strict limits (e.g., 6/hour) will block all remaining registration-related tests with 429. Strategy: test non-registration endpoints first, registration tests last, and note which tests were blocked.
• ⚠️ Present the test plan BEFORE executing: Show the user the complete test plan first. If they say "is this really all of it?", the plan is missing dimensions. Refer to the Comprehensive QA Dimensions Checklist above.
• ⚠️ "全部加上" means ALL dimensions: When the user says to add everything, do not skip any dimension. Write all 25+ categories into the test plan even if some have only 1-2 test cases.
• Multi-service auth: Sites with shared cookies (e.g., .ephron.ren domain) need login on ONE service first, then verify cookie propagation to others. Don't try to login on each service independently.
• Encoding bugs: Always hex-dump HTML source to check for BOM markers (ef bb bf) or leading newlines before DOCTYPE. Use: xxd file.html | head -5. For Python source files, also check: xxd file.py | head -1.
• CSRF tokens: Many form submissions require CSRF tokens. Extract from the page first, then include in POST requests. Don't forget the CSRF cookie (ephron_csrf). Note: CSRF cookies are HttpOnly=false (by design, so JS can read them).
• Rate limits: Note rate limit values from source code (e.g., @limiter.limit("5/minute")). When testing auth failures, stay under the limit or you'll get 429s that mask the real bug.
• Template vs runtime issues: Some issues (empty content, missing sections) may be data issues, not code bugs. Verify by checking if the data source (database/content files) actually has content.
• File delivery fallback: When sending files via QQ/WeChat fails, push to a Gitea repo as a fallback delivery mechanism.
• Source code security analysis: Always check these files when available: cookie_utils.py (cookie params), csrf.py (CSRF mechanism), redirect.py (open redirect validation), security_headers.py (CSP/headers), auth.py (token format, lockout), validators.py (slug/path validation), limiter.py (rate limit config).
• ⚠️ CSP script-src-elem silently kills inline JS: When a page has inline <script> but buttons call functions defined there (e.g., onclick="saveDraft()"), always verify the CSP header. The script-src-elem directive overrides script-src for script elements — so script-src 'unsafe-inline' combined with script-src-elem 'self' https://cdn.example.com blocks ALL inline scripts. Symptoms: functions report "not defined", buttons do nothing, no network requests on click. Detection: check typeof fnName in browser console, or look for CSP error in console: Executing inline script violates the following Content Security Policy directive 'script-src-elem'. Fix: add 'unsafe-inline' to script-src-elem, use nonce/hash, or extract inline scripts to external .js files.
• ⚠️ CSP form-action 'self' blocks cross-origin redirects after form submission: When a form POSTs to a same-origin endpoint (allowed by form-action 'self'), but the server responds with 303 redirect to a different origin (e.g., auth.example.com → www.example.com), the browser blocks the redirect. CSP form-action applies to the entire redirect chain resulting from form submission, not just the form's action URL. Symptoms: form appears to submit (POST in network tab), cookie gets set server-side, but page stays on the form URL — no navigation. Console error: Sending form data to '...' violates the following Content Security Policy directive: "form-action 'self'". Detection: (1) test same-origin redirect (should work) vs cross-origin redirect (should fail); (2) curl -sI the 303 response — if it carries CSP with form-action 'self', that's the blocker. Fix options: (a) skip CSP header on 303 redirect responses (empty body, CSP adds no protection); (b) use JS-based redirect instead of server-side 303; (c) add allowed origins to form-action. Key insight: this breaks any auth flow where login service is on a different subdomain than target pages. See references/session-learnings-ephron-qa.md for full reproduction steps.

Scope Ambiguity Pitfall

When the user asks to "inspect a server" or "巡检服务器" without providing a URL:

• Clarify whether they mean the local machine Hermes runs on (system resources, running processes, disk/memory) or a remote web service (HTTP endpoints, app health).
• Default assumption: If the user mentions a domain name (e.g., "巡检 ephron.ren" or "check blog.ephron.ren"), they mean the remote web service. If they say "your server" or "the machine you're on", they mean the local machine.
• When in doubt, ask: "是巡检本机还是远程服务？"

Tools Reference

Tool	Purpose
browser_navigate	Go to a URL
browser_snapshot	Get DOM text snapshot (accessibility tree)
browser_click	Click an element by ref (@eN) or text
browser_type	Type into an input field
browser_scroll	Scroll up/down on the page
browser_back	Go back in browser history
browser_press	Press a keyboard key
browser_vision	Screenshot + AI analysis; use annotate=true for element labels
browser_console	Get JS console output and errors
Playwright Python	Full browser automation via script — use when built-in tools unavailable or need programmatic control (see references/playwright-qa.md)

Related References

• references/issue-taxonomy.md — severity and category classification for issues
• references/server-inspection.md — local server inspection checklist: system resources, listening ports, processes, Docker, security services; also covers scope ambiguity (local vs. remote), route file reading strategy, cross-service cookie auth testing, static analysis checks
• references/qa-dimensions-checklist.md — comprehensive 25-dimension QA checklist for "full site" testing requests
• references/playwright-qa.md — Playwright Python setup, patterns, event monitoring, CSP bug detection
• references/session-learnings-ephron-qa.md — concrete findings from ephron.ren QA: CSP override, password validation gaps, fulltext search failure, delegate sizing

Templates

• templates/dogfood-report-template.md — issue list template (the output with bugs found)
• templates/test-plan-template.md — test plan template (structured test cases with steps)

Tips

• Always check browser_console() after navigating and after significant interactions. Silent JS errors are among the most valuable findings.
• Use annotate=true with browser_vision when you need to reason about interactive element positions or when the snapshot refs are unclear.
• Test with both valid and invalid inputs — form validation bugs are common.
• Scroll through long pages — content below the fold may have rendering issues.
• Test navigation flows — click through multi-step processes end-to-end.
• Check responsive behavior by noting any layout issues visible in screenshots.
• Don't forget edge cases: empty states, very long text, special characters, rapid clicking.
• When reporting screenshots to the user, include MEDIA:<screenshot_path> so they can see the evidence inline.