first commit
This commit is contained in:
109
dogfood/references/issue-taxonomy.md
Normal file
109
dogfood/references/issue-taxonomy.md
Normal file
@@ -0,0 +1,109 @@
|
||||
# Issue Taxonomy
|
||||
|
||||
Use this taxonomy to classify issues found during dogfood QA testing.
|
||||
|
||||
## Severity Levels
|
||||
|
||||
### Critical
|
||||
The issue makes a core feature completely unusable or causes data loss.
|
||||
|
||||
**Examples:**
|
||||
- Application crashes or shows a blank white page
|
||||
- Form submission silently loses user data
|
||||
- Authentication is completely broken (can't log in at all)
|
||||
- Payment flow fails and charges the user without completing the order
|
||||
- Security vulnerability (e.g., XSS, exposed credentials in console)
|
||||
|
||||
### High
|
||||
The issue significantly impairs functionality but a workaround may exist.
|
||||
|
||||
**Examples:**
|
||||
- A key button does nothing when clicked (but refreshing fixes it)
|
||||
- Search returns no results for valid queries
|
||||
- Form validation rejects valid input
|
||||
- Page loads but critical content is missing or garbled
|
||||
- Navigation link leads to a 404 or wrong page
|
||||
- Uncaught JavaScript exceptions in the console on core pages
|
||||
|
||||
### Medium
|
||||
The issue is noticeable and affects user experience but doesn't block core functionality.
|
||||
|
||||
**Examples:**
|
||||
- Layout is misaligned or overlapping on certain screen sections
|
||||
- Images fail to load (broken image icons)
|
||||
- Slow performance (visible loading delays > 3 seconds)
|
||||
- Form field lacks proper validation feedback (no error message on bad input)
|
||||
- Console warnings that suggest deprecated or misconfigured features
|
||||
- Inconsistent styling between similar pages
|
||||
|
||||
### Low
|
||||
Minor polish issues that don't affect functionality.
|
||||
|
||||
**Examples:**
|
||||
- Typos or grammatical errors in text content
|
||||
- Minor spacing or alignment inconsistencies
|
||||
- Placeholder text left in production ("Lorem ipsum")
|
||||
- Favicon missing
|
||||
- Console info/debug messages that shouldn't be in production
|
||||
- Subtle color contrast issues that don't fail WCAG requirements
|
||||
|
||||
## Categories
|
||||
|
||||
### Functional
|
||||
Issues where features don't work as expected.
|
||||
|
||||
- Buttons/links that don't respond
|
||||
- Forms that don't submit or submit incorrectly
|
||||
- Broken user flows (can't complete a multi-step process)
|
||||
- Incorrect data displayed
|
||||
- Features that work partially
|
||||
|
||||
### Visual
|
||||
Issues with the visual presentation of the page.
|
||||
|
||||
- Layout problems (overlapping elements, broken grids)
|
||||
- Broken images or missing media
|
||||
- Styling inconsistencies
|
||||
- Responsive design failures
|
||||
- Z-index issues (elements hidden behind others)
|
||||
- Text overflow or truncation
|
||||
|
||||
### Accessibility
|
||||
Issues that prevent or hinder access for users with disabilities.
|
||||
|
||||
- Missing alt text on meaningful images
|
||||
- Poor color contrast (fails WCAG AA)
|
||||
- Elements not reachable via keyboard navigation
|
||||
- Missing form labels or ARIA attributes
|
||||
- Focus indicators missing or unclear
|
||||
- Screen reader incompatible content
|
||||
|
||||
### Console
|
||||
Issues detected through JavaScript console output.
|
||||
|
||||
- Uncaught exceptions and unhandled promise rejections
|
||||
- Failed network requests (4xx, 5xx errors in console)
|
||||
- Deprecation warnings
|
||||
- CORS errors
|
||||
- Mixed content warnings (HTTP resources on HTTPS page)
|
||||
- Excessive console.log output left from development
|
||||
|
||||
### UX (User Experience)
|
||||
Issues where functionality works but the experience is poor.
|
||||
|
||||
- Confusing navigation or information architecture
|
||||
- Missing loading indicators (user doesn't know something is happening)
|
||||
- No feedback after user actions (e.g., button click with no visible result)
|
||||
- Inconsistent interaction patterns
|
||||
- Missing confirmation dialogs for destructive actions
|
||||
- Poor error messages that don't help the user recover
|
||||
|
||||
### Content
|
||||
Issues with the text, media, or information on the page.
|
||||
|
||||
- Typos and grammatical errors
|
||||
- Placeholder/dummy content in production
|
||||
- Outdated information
|
||||
- Missing content (empty sections)
|
||||
- Broken or dead links to external resources
|
||||
- Incorrect or misleading labels
|
||||
356
dogfood/references/multi-service-qa.md
Normal file
356
dogfood/references/multi-service-qa.md
Normal file
@@ -0,0 +1,356 @@
|
||||
# Multi-Service Site QA Patterns
|
||||
|
||||
## Architecture Recognition
|
||||
|
||||
When a site has multiple subdomains or services, first map the architecture:
|
||||
|
||||
| Indicator | What it means |
|
||||
|-----------|--------------|
|
||||
| Multiple `main.py` files in subdirectories | Separate service entry points |
|
||||
| `shared/` directory with auth/cookie modules | Shared authentication across services |
|
||||
| Different port numbers in config | Local dev runs separate processes |
|
||||
| Subdomain routing (auth.ephron.ren, blog.ephron.ren) | Production reverse proxy setup |
|
||||
|
||||
## Common Multi-Service Patterns (FastAPI)
|
||||
|
||||
```
|
||||
project/
|
||||
├── auth/src/main.py # Auth service (login, register, RBAC)
|
||||
├── blog/src/main.py # Blog service (posts, comments, likes)
|
||||
├── canvas/src/main.py # Canvas service (AI-generated pages)
|
||||
├── prompt/src/main.py # Prompt service (prompt CRUD)
|
||||
├── home/src/main.py # Homepage service
|
||||
├── shared/ # Shared modules (auth, CSRF, audit, templating)
|
||||
│ ├── auth_users.py
|
||||
│ ├── cookie_utils.py
|
||||
│ ├── csrf.py
|
||||
│ ├── templating.py
|
||||
│ └── ports.py # Service URL configuration
|
||||
└── main.py # Unified launcher (starts all services)
|
||||
```
|
||||
|
||||
## Cross-Service Cookie Auth Testing
|
||||
|
||||
1. Login on auth service → get `ephron_auth` cookie
|
||||
2. Verify cookie domain is `.example.com` (not service-specific)
|
||||
3. Test cookie propagation: visit each service, check logged-in state
|
||||
4. Test logout: logout on one service, verify all services see logged-out state
|
||||
|
||||
## Route File Reading Strategy
|
||||
|
||||
For each service, read these files in order:
|
||||
1. `src/routes/pages.py` — public page routes
|
||||
2. `src/routes/admin.py` — admin/management routes
|
||||
3. `src/routes/api.py` — API endpoints
|
||||
4. `src/routes/service_api.py` — inter-service APIs
|
||||
5. `src/services/auth.py` — auth helpers (what permissions are checked)
|
||||
|
||||
Extract from each route:
|
||||
- `@router.get("/path")` or `@router.post("/path")` → HTTP method + path
|
||||
- `_require_auth(ephron_auth, request, permission="X.Y.Z")` → required permission
|
||||
- `@limiter.limit("N/minute")` → rate limit
|
||||
- `Form(...)` parameters → required form fields
|
||||
- `Cookie(default=None)` → cookie dependencies
|
||||
|
||||
## Test Matrix Generation
|
||||
|
||||
For each discovered route, create test cases:
|
||||
- **Happy path**: valid inputs, correct auth → expected success
|
||||
- **Auth failure**: no cookie / wrong role → expected redirect or 403
|
||||
- **Validation failure**: missing fields, invalid data → expected error
|
||||
- **Rate limit**: exceed the limit → expected 429
|
||||
- **CSRF**: missing/invalid CSRF token → expected rejection
|
||||
|
||||
## Consistency Checks Across Services
|
||||
|
||||
Build a comparison table:
|
||||
| Feature | Service A | Service B | Service C |
|
||||
|---------|-----------|-----------|-----------|
|
||||
| mobile.css loaded? | ✅ | ❌ | ❌ |
|
||||
| loader.js loaded? | ❌ | ✅ | ✅ |
|
||||
| Site navigation? | ✅ | ✅ | ❌ |
|
||||
| user-scalable? | yes | no | no |
|
||||
|
||||
Inconsistencies are bugs — all services sharing a design system should be consistent.
|
||||
|
||||
## Curl-Based QA Techniques (Session-Proven)
|
||||
|
||||
When browser automation is unavailable, these curl patterns reliably test multi-service sites:
|
||||
|
||||
### Cookie Management
|
||||
```bash
|
||||
# Each curl -c (save) / -b (read) needs a SEPARATE cookie file per request chain
|
||||
curl -s -c /tmp/c1.txt https://auth.example.com/login > /tmp/login.html
|
||||
curl -s -b /tmp/c1.txt -c /tmp/c2.txt -X POST https://auth.example.com/api/login \
|
||||
-d "username=user&password=pass&csrf_token=$CSRF" > /dev/null
|
||||
# Verify: grep ephron /tmp/c2.txt
|
||||
```
|
||||
|
||||
### CSRF Token Extraction (FastAPI/Tortoise patterns)
|
||||
```bash
|
||||
# Most reliable — matches name= then grabs value:
|
||||
grep -oP 'name="csrf_token"[^>]*value="\K[^"]+' /tmp/page.html | head -1
|
||||
|
||||
# Fallback variants:
|
||||
grep -oP 'csrf_token.*?value="\K[^"]+' /tmp/page.html | head -1
|
||||
grep -i 'csrf' /tmp/page.html | grep -oP 'value="\K[^"]+' | head -1
|
||||
```
|
||||
|
||||
### API Login: JSON vs Form-Encoded
|
||||
```bash
|
||||
# Modern FastAPI services use /api/login with JSON:
|
||||
curl -s -b /tmp/c.txt -c /tmp/c.txt -X POST https://auth.example.com/api/login \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"username":"user","password":"pass","csrf_token":"TOKEN"}'
|
||||
|
||||
# Legacy form-encoded (action="/login"):
|
||||
curl -s -b /tmp/c.txt -c /tmp/c.txt -X POST https://auth.example.com/login \
|
||||
-H "Content-Type: application/x-www-form-urlencoded" \
|
||||
-d "username=user&password=pass&csrf_token=$CSRF"
|
||||
```
|
||||
|
||||
### Post-Login Redirect Chain
|
||||
```bash
|
||||
# Follow 303 redirect chain automatically:
|
||||
curl -sL -b /tmp/c.txt -c /tmp/c.txt -X POST https://auth.example.com/api/login \
|
||||
-d "username=u&password=p&csrf_token=$CSRF" -w "\nHTTP:%{http_code}"
|
||||
# Get final status: curl -sL ... -o /dev/null -w "%{http_code}"
|
||||
```
|
||||
|
||||
### Health Checks (All Services at Once)
|
||||
```bash
|
||||
for svc in www auth blog canvas prompt; do
|
||||
result=$(curl -s "https://$svc.example.com/health")
|
||||
echo "$svc: $result"
|
||||
done
|
||||
```
|
||||
|
||||
### Security Headers (All Services)
|
||||
```bash
|
||||
for svc in www auth blog canvas prompt; do
|
||||
echo "=== $svc ==="
|
||||
curl -sI "https://$svc.example.com/" | grep -iE \
|
||||
'x-content-type|x-frame|referrer-policy|content-security|set-cookie'
|
||||
done
|
||||
```
|
||||
|
||||
### CSP Deep Analysis — script-src-elem Override Trap
|
||||
```bash
|
||||
# Extract full CSP header
|
||||
curl -sI https://www.example.com/admin | grep -i content-security-policy
|
||||
|
||||
# Look for script-src-elem which OVERRIDES script-src for <script> elements:
|
||||
# BAD: script-src 'self' 'unsafe-inline'; script-src-elem 'self' https://cdn.example.com;
|
||||
# GOOD: script-src 'self' 'unsafe-inline'; script-src-elem 'self' 'unsafe-inline' https://cdn.example.com;
|
||||
#
|
||||
# If script-src-elem exists without 'unsafe-inline', ALL inline <script> tags are blocked.
|
||||
# Symptoms: onclick handlers call undefined functions, buttons do nothing, no JS errors in console
|
||||
# (CSP violations appear as pageerror events, not console.error)
|
||||
```
|
||||
|
||||
### Cookie Security Verification
|
||||
```bash
|
||||
# Capture Set-Cookie on login response:
|
||||
curl -sI -c /tmp/c.txt -X POST https://auth.example.com/api/login \
|
||||
-d "username=u&password=p&csrf_token=t" 2>/dev/null | grep -i set-cookie
|
||||
# Expected: HttpOnly; Secure; SameSite=lax; Max-Age=604800; Domain=.example.com
|
||||
```
|
||||
|
||||
### Session Fixation Check
|
||||
```bash
|
||||
# Before login: record cookie
|
||||
curl -sI -c /tmp/before.txt https://auth.example.com/login | grep -i set-cookie
|
||||
# (GET requests rarely set auth cookies)
|
||||
|
||||
# After login: cookie must change
|
||||
curl -s -b /tmp/before.txt -c /tmp/after.txt -X POST .../api/login ...
|
||||
grep ephron_auth /tmp/after.txt
|
||||
# Session ID must be different from before
|
||||
```
|
||||
|
||||
### Known Rate Limits (ephron.ren observed)
|
||||
```bash
|
||||
# Auth login failures: 5/min → 429
|
||||
# Auth registration: 6/hour → 429 (use existing test accounts)
|
||||
# Blog comments: 6/min
|
||||
# Blog likes toggle: 11/min
|
||||
# Save/publish ops: 21/min
|
||||
```
|
||||
|
||||
### Delegate Task Sizing for Large Test Suites
|
||||
|
||||
When testing 100+ cases across multiple modules, delegate_task has a 600s timeout. Size tasks carefully:
|
||||
|
||||
| Task Type | Max Cases per Delegate | Reason |
|
||||
|-----------|----------------------|--------|
|
||||
| Curl-only HTTP tests | 15-20 | Each curl = 1-3s + overhead |
|
||||
| Browser interactions | 5-8 | Each interaction = 10-30s |
|
||||
| Mixed curl + Playwright | 8-12 | Browser calls dominate time |
|
||||
|
||||
**Faster alternative**: Use `execute_code` with `from hermes_tools import terminal` for in-process execution. No delegation overhead, same capabilities.
|
||||
|
||||
```python
|
||||
from hermes_tools import terminal
|
||||
results = {}
|
||||
r = terminal("curl -s -o /dev/null -w '%{http_code}' https://example.com/")
|
||||
results["T-001"] = {"status": "PASS" if "200" in r["output"] else "FAIL", "detail": f"HTTP {r['output']}"}
|
||||
```
|
||||
|
||||
### CSRF Token Synchronization Pitfall (curl)
|
||||
|
||||
When testing forms that require CSRF tokens, the token in the cookie changes on every GET request. If you GET a page, extract the CSRF token, then POST with a **different** cookie jar, the tokens won't match and you'll get "CSRF token 验证失败".
|
||||
|
||||
```bash
|
||||
# WRONG: separate cookie jars for GET and POST
|
||||
curl -s -b /tmp/jar1.txt https://example.com/admin > /tmp/page.html # sets new CSRF cookie
|
||||
curl -s -b /tmp/jar2.txt -X POST ... -d "csrf_token=$CSRF" # different jar = mismatch!
|
||||
|
||||
# RIGHT: same cookie jar for GET and POST in sequence
|
||||
curl -s -b /tmp/jar.txt -c /tmp/jar.txt https://example.com/admin > /tmp/page.html
|
||||
CSRF=$(grep -oP 'name="csrf_token"[^>]*value="\K[^"]+' /tmp/page.html | head -1)
|
||||
curl -s -b /tmp/jar.txt -c /tmp/jar.txt -X POST ... -d "csrf_token=$CSRF"
|
||||
```
|
||||
|
||||
**Why this happens**: FastAPI/Starlette CSRF middleware generates a new token on each GET and stores it in the `ephron_csrf` cookie. The POST handler compares the form token against the cookie token — they must come from the same request chain.
|
||||
|
||||
**Multiple forms on one page**: If a page has N forms, there will be N CSRF tokens in the HTML but only ONE in the cookie. Each form's token is unique. Extract the token from the specific form you need (use context-aware parsing, not just `head -1`).
|
||||
|
||||
### Owner vs Admin Permission Testing Pattern
|
||||
|
||||
When a site has RBAC (user < admin < owner), test with all roles:
|
||||
|
||||
```bash
|
||||
# Login as each role
|
||||
for role in owner admin user; do
|
||||
curl -s -c /tmp/$role.txt -X POST https://auth.example.com/api/login \
|
||||
-d "username=Elaina_$role&password=Pass123!" -o /dev/null
|
||||
done
|
||||
|
||||
# Test each protected endpoint with each role
|
||||
for role in owner admin user; do
|
||||
status=$(curl -s -b /tmp/$role.txt -o /dev/null -w '%{http_code}' https://example.com/admin/roles)
|
||||
echo "$role -> /admin/roles: $status"
|
||||
done
|
||||
```
|
||||
|
||||
**Key insight**: If admin role can't access a page but the nav bar shows the link, it's a UX bug (hidden nav items for unauthorized roles) or a permission misconfiguration.
|
||||
|
||||
### Content Restoration for Destructive Tests
|
||||
|
||||
When tests modify content (create invite codes, publish posts, change settings):
|
||||
|
||||
1. **Before testing**: Save current state
|
||||
```bash
|
||||
# Save homepage content
|
||||
curl -s -b /tmp/admin.txt https://www.example.com/admin | grep -oP 'initialContent = JSON\.parse\("\K[^"]*' > /tmp/homepage_backup.json
|
||||
|
||||
# Save blog post slugs
|
||||
curl -s https://blog.example.com/ | grep -oP '/posts/[a-z0-9-]+' | sort -u > /tmp/blog_slugs.txt
|
||||
```
|
||||
|
||||
2. **During testing**: Create test data with identifiable markers (e.g., `QA_TEST_TEMP` in notes/titles)
|
||||
|
||||
3. **After testing**: Clean up test data
|
||||
```bash
|
||||
# Delete test invite codes
|
||||
curl -s -b /tmp/owner.txt -X POST https://auth.example.com/admin/invites/delete \
|
||||
-d "csrf_token=$CSRF&code=$TEST_CODE"
|
||||
```
|
||||
|
||||
4. **Verify restoration**: Check that original content is unchanged
|
||||
```bash
|
||||
for slug in $(cat /tmp/blog_slugs.txt); do
|
||||
status=$(curl -s -o /dev/null -w '%{http_code}' "https://blog.example.com/posts/$slug")
|
||||
echo "$slug: $status"
|
||||
done
|
||||
```
|
||||
|
||||
### Module-by-Module Testing with Incremental Commits
|
||||
|
||||
For large QA tasks (100+ test cases across many modules), the user may want results committed after each module:
|
||||
|
||||
1. Create `test-results.md` with placeholder sections for all modules
|
||||
2. Test module N → update the module section in test-results.md
|
||||
3. `git add test-results.md && git commit -m "模块N完成: 通过X/失败Y" && git push`
|
||||
4. Report progress to user
|
||||
5. Repeat for next module
|
||||
|
||||
**Document structure per module**:
|
||||
```markdown
|
||||
## 模块 N:名称
|
||||
|
||||
**状态**: ✅ 已完成
|
||||
**执行时间**: YYYY-MM-DD HH:MM - HH:MM
|
||||
**测试结果**: 通过 X / 失败 Y / 阻塞 Z(共 N 项)
|
||||
|
||||
| 编号 | 结果 | 备注 |
|
||||
|------|------|------|
|
||||
| X-001 | ✅ 通过 | detail |
|
||||
| X-002 | ❌ 失败 | 🔴 description |
|
||||
|
||||
### 模块 N 小结
|
||||
- Summary bullets
|
||||
|
||||
### 💡 模块 N 优化建议
|
||||
1. **🔴 [Critical]**: description
|
||||
2. **🟡 [High]**: description
|
||||
```
|
||||
|
||||
**Why per-module commits**: Gives the user incremental visibility, prevents data loss if the session breaks, and creates a clean git history.
|
||||
|
||||
### Registration Rate Limiting Pitfall
|
||||
|
||||
Registration endpoints typically have strict rate limits (e.g., 6/hour). When testing multiple registration scenarios (password validation, username checks, invite codes), the rate limit kicks in and blocks subsequent tests with 429, masking the real behavior.
|
||||
|
||||
**Workaround**:
|
||||
- Test rate-limited endpoints LAST in each module
|
||||
- Use existing test accounts for non-registration tests
|
||||
- Note which tests were blocked by rate limiting in results
|
||||
- Space out registration tests or use different IPs if possible
|
||||
|
||||
### Common API Field Names (FastAPI/Pydantic patterns)
|
||||
```bash
|
||||
# Blog likes toggle: field is `post_slug` (NOT `slug`)
|
||||
curl -X POST https://blog.example.com/api/likes/toggle \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"post_slug":"article-slug"}'
|
||||
|
||||
# Blog comments: post_slug + content + parent_id (nullable)
|
||||
curl -X POST https://blog.example.com/api/comments/ \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"post_slug":"article-slug","content":"text","parent_id":null}'
|
||||
```
|
||||
|
||||
### Template Encoding Checks (BOM / Leading Whitespace)
|
||||
```bash
|
||||
# BOM marker: UTF-8 EF BB BF appears before DOCTYPE
|
||||
xxd /tmp/page.html | head -3
|
||||
|
||||
# Leading newline before DOCTYPE: 0a 3c 21 44 4f ...
|
||||
head -c 20 /tmp/page.html | xxd
|
||||
|
||||
# Python source BOM check:
|
||||
xxd app.py | head -1
|
||||
```
|
||||
|
||||
## Static Analysis Checks (no browser needed)
|
||||
|
||||
```bash
|
||||
# Check for BOM markers
|
||||
xxd file.html | head -3
|
||||
# Look for: ef bb bf (UTF-8 BOM)
|
||||
|
||||
# Check for leading whitespace before DOCTYPE
|
||||
head -c 20 file.html | xxd
|
||||
|
||||
# Check CSS variable definitions
|
||||
grep -n "\-\-warning-bg|--error-bg|--success-bg" file.html
|
||||
|
||||
# Check for accessibility issues
|
||||
grep -n 'user-scalable=no' *.html
|
||||
grep -n 'alt=""' *.html
|
||||
grep -n 'aria-hidden' *.html
|
||||
|
||||
# Check security headers
|
||||
curl -sI https://example.com | grep -i "x-content-type|x-frame|referrer-policy|content-security"
|
||||
```
|
||||
156
dogfood/references/playwright-qa.md
Normal file
156
dogfood/references/playwright-qa.md
Normal file
@@ -0,0 +1,156 @@
|
||||
# Playwright Python for QA Testing
|
||||
|
||||
## Environment Setup
|
||||
|
||||
Playwright Python is available on this system:
|
||||
- **Package**: `/home/ubuntu/.hermes/hermes-agent/venv/lib/python3.11/site-packages/playwright/`
|
||||
- **Chromium**: `~/.cache/ms-playwright/chromium-1217/`
|
||||
- **Import**: `from playwright.sync_api import sync_playwright`
|
||||
- **Run**: `python3 script.py` (not `node` — Playwright Node module may not be installed)
|
||||
|
||||
## Basic Pattern
|
||||
|
||||
```python
|
||||
from playwright.sync_api import sync_playwright
|
||||
|
||||
with sync_playwright() as p:
|
||||
browser = p.chromium.launch(headless=True)
|
||||
context = browser.new_context()
|
||||
page = context.new_page()
|
||||
|
||||
# Login
|
||||
page.goto('https://auth.example.com/login')
|
||||
page.wait_for_load_state('networkidle')
|
||||
page.fill('#username', 'admin_user')
|
||||
page.fill('#password', 'password')
|
||||
page.click('button[type="submit"]')
|
||||
page.wait_for_url('**/login-success**', timeout=10000)
|
||||
|
||||
# Navigate to target
|
||||
page.goto('https://www.example.com/admin')
|
||||
page.wait_for_load_state('networkidle')
|
||||
|
||||
# Interact and test...
|
||||
|
||||
browser.close()
|
||||
```
|
||||
|
||||
## Event Monitoring (Critical for QA)
|
||||
|
||||
### Console Messages and JS Errors
|
||||
```python
|
||||
console_msgs = []
|
||||
page_errors = []
|
||||
page.on("console", lambda m: console_msgs.append(f"[{m.type}] {m.text}"))
|
||||
page.on("pageerror", lambda e: page_errors.append(str(e)))
|
||||
|
||||
# After interactions:
|
||||
for m in console_msgs:
|
||||
print(f" {m}")
|
||||
for e in page_errors:
|
||||
print(f" ERROR: {e}")
|
||||
```
|
||||
|
||||
### Network Requests and Responses
|
||||
```python
|
||||
api_responses = []
|
||||
def on_response(r):
|
||||
if '/api/' in r.url or '/admin/' in r.url:
|
||||
api_responses.append({"url": r.url, "status": r.status})
|
||||
page.on("response", on_response)
|
||||
|
||||
# After interactions:
|
||||
for r in api_responses:
|
||||
print(f" {r['url']} -> {r['status']}")
|
||||
```
|
||||
|
||||
### Failed Resource Loads
|
||||
```python
|
||||
failed_resources = []
|
||||
page.on("requestfailed", lambda r: failed_resources.append({"url": r.url, "error": r.failure}))
|
||||
```
|
||||
|
||||
## Element Query Patterns
|
||||
|
||||
```python
|
||||
# By text content
|
||||
btn = page.query_selector('button:has-text("Save")')
|
||||
links = page.query_selector_all('a:has-text("Login")')
|
||||
|
||||
# By CSS selector with attribute
|
||||
input_el = page.query_selector('input[name="csrf_token"]')
|
||||
form = page.query_selector('#contentForm')
|
||||
|
||||
# By role
|
||||
submit = page.query_selector('button[type="submit"]')
|
||||
|
||||
# Get all buttons (for debugging)
|
||||
all_btns = page.query_selector_all('button')
|
||||
btn_texts = [b.inner_text().strip() for b in all_btns]
|
||||
```
|
||||
|
||||
## JavaScript Evaluation
|
||||
|
||||
```python
|
||||
# Check if function is defined
|
||||
is_defined = page.evaluate("typeof saveDraft === 'function'")
|
||||
|
||||
# Get element properties
|
||||
val = page.evaluate("document.getElementById('contentJson').value")
|
||||
|
||||
# Get page content
|
||||
page_html = page.content()
|
||||
body_text = page.inner_text('body')
|
||||
|
||||
# Execute arbitrary JS
|
||||
result = page.evaluate("() => { return document.title; }")
|
||||
```
|
||||
|
||||
## Screenshots
|
||||
|
||||
```python
|
||||
# Full page
|
||||
page.screenshot(path='/tmp/screenshot.png', full_page=True)
|
||||
|
||||
# Then analyze with vision tool
|
||||
```
|
||||
|
||||
## Cookie Inspection
|
||||
|
||||
```python
|
||||
cookies = context.cookies()
|
||||
for c in cookies:
|
||||
print(f"{c['name']}: domain={c['domain']}, httpOnly={c['httpOnly']}, "
|
||||
f"secure={c['secure']}, sameSite={c.get('sameSite','N/A')}")
|
||||
```
|
||||
|
||||
## Testing with Custom Headers (e.g., Bearer Token)
|
||||
|
||||
```python
|
||||
# Create separate context with extra headers
|
||||
context2 = browser.new_context(extra_http_headers={"Authorization": "Bearer fake_token"})
|
||||
page2 = context2.new_page()
|
||||
page2.goto('https://www.example.com/admin')
|
||||
# Check if redirected to login
|
||||
print(f"URL: {page2.url}")
|
||||
page2.close()
|
||||
context2.close()
|
||||
```
|
||||
|
||||
## CSP Bug Detection Pattern
|
||||
|
||||
When buttons with `onclick="fnName()"` do nothing:
|
||||
|
||||
1. Check console for CSP violation: `"Executing inline script violates the following Content Security Policy directive 'script-src-elem'"`
|
||||
2. Verify function availability: `page.evaluate("typeof fnName")` returns `"undefined"`
|
||||
3. Confirm script tag exists but is blocked by CSP
|
||||
4. Check CSP header: `curl -sI URL | grep content-security-policy`
|
||||
5. Look for `script-src-elem` directive that overrides `script-src`
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Use Python, not Node.js**: The `playwright` npm module may not be installed. Python works.
|
||||
- **`expect_response` timeout**: Don't use broad URL patterns. Use specific path matches or handle timeout gracefully.
|
||||
- **`expect_navigation` for SPA**: Single-page apps may not trigger navigation events. Use `wait_for_timeout` or check state changes instead.
|
||||
- **Rate limit testing**: Don't try to trigger rate limits via Playwright — too slow. Use curl for rate limit tests.
|
||||
- **`page.on("console")` misses CSP errors**: CSP violations appear as `pageerror` events, not console messages. Listen to both.
|
||||
152
dogfood/references/qa-dimensions-checklist.md
Normal file
152
dogfood/references/qa-dimensions-checklist.md
Normal file
@@ -0,0 +1,152 @@
|
||||
# Comprehensive QA Dimensions Checklist
|
||||
|
||||
Use this checklist when the user asks for "full", "complete", or "comprehensive" QA testing.
|
||||
Each dimension should appear as a section in the test plan with at least 1 test case.
|
||||
|
||||
## Core Functional (always cover)
|
||||
- [ ] Page loads (HTTP 200) for all public pages
|
||||
- [ ] Navigation links work (header, footer, sidebar)
|
||||
- [ ] CRUD operations (create, read, update, delete)
|
||||
- [ ] Form submissions (valid data, empty data, invalid data)
|
||||
- [ ] Search/filter functionality
|
||||
- [ ] Pagination
|
||||
- [ ] Error pages (404, 500)
|
||||
|
||||
## Auth & Permissions
|
||||
- [ ] Login page loads and form works
|
||||
- [ ] Valid credentials → success + cookie set
|
||||
- [ ] Invalid credentials → error message
|
||||
- [ ] Logout clears cookie
|
||||
- [ ] Cross-service cookie propagation (shared domain cookies)
|
||||
- [ ] Admin pages: admin user can access
|
||||
- [ ] Admin pages: regular user gets denied
|
||||
- [ ] Admin pages: unauthenticated user redirects to login
|
||||
- [ ] RBAC: different roles see different features
|
||||
- [ ] Permission checks on API endpoints
|
||||
|
||||
## Input Validation
|
||||
- [ ] Empty form submissions (browser validation or server error)
|
||||
- [ ] Boundary values (min/max length, special chars)
|
||||
- [ ] Password strength requirements
|
||||
- [ ] Username format validation
|
||||
- [ ] Email format validation (if applicable)
|
||||
- [ ] Invite code validation (if invite-based registration)
|
||||
|
||||
## Security — Cookie
|
||||
- [ ] Auth cookie: HttpOnly=true
|
||||
- [ ] Auth cookie: Secure=true (production)
|
||||
- [ ] Auth cookie: SameSite=Lax or Strict
|
||||
- [ ] Auth cookie: Max-Age is reasonable (not infinite)
|
||||
- [ ] Auth cookie: Domain scope correct (e.g., `.example.com` for subdomains)
|
||||
- [ ] CSRF cookie: HttpOnly=false (by design, JS needs to read it)
|
||||
|
||||
## Security — CSRF
|
||||
- [ ] All state-changing POST endpoints require CSRF token
|
||||
- [ ] CSRF token matches between form field and cookie
|
||||
- [ ] CSRF token expires (check timestamp-based expiry)
|
||||
- [ ] Missing/invalid CSRF token returns 403 or error
|
||||
|
||||
## Security — Redirect
|
||||
- [ ] `redirect` parameter accepts valid same-domain URLs
|
||||
- [ ] `redirect` parameter rejects external domains (open redirect prevention)
|
||||
- [ ] `redirect` parameter rejects protocol-relative URLs (`//evil.com`)
|
||||
- [ ] Default redirect when parameter is empty/invalid
|
||||
|
||||
## Security — Rate Limiting
|
||||
- [ ] Login rate limit (e.g., 5/minute per IP)
|
||||
- [ ] Registration rate limit (e.g., 5/hour per IP)
|
||||
- [ ] API rate limits (comments, likes, uploads)
|
||||
- [ ] Account lockout after N failed attempts
|
||||
- [ ] IP-based lockout after N failed attempts
|
||||
- [ ] Rate limit returns 429 status
|
||||
|
||||
## Security — File Upload
|
||||
- [ ] Allowed file types enforced (extension check)
|
||||
- [ ] File size limit enforced
|
||||
- [ ] Filename sanitized (no path traversal)
|
||||
- [ ] Uploaded files stored safely (UUID names, outside web root or in controlled dir)
|
||||
- [ ] Image processing (resize, format conversion) doesn't crash on malformed files
|
||||
|
||||
## Security — Input Injection
|
||||
- [ ] XSS: user input rendered as text, not HTML (test `<script>alert(1)</script>`)
|
||||
- [ ] Path traversal: slug validation prevents `../` sequences
|
||||
- [ ] SQL injection: parameterized queries (verify from source code)
|
||||
|
||||
## Session & Token
|
||||
- [ ] Token expiration: expired token redirects to login
|
||||
- [ ] Token format validation (reject malformed tokens)
|
||||
- [ ] Role changes: DB role takes precedence over token role
|
||||
- [ ] Token max-age from configuration
|
||||
|
||||
## Content & Rendering
|
||||
- [ ] Empty state (no content) shows appropriate message
|
||||
- [ ] Long content doesn't break layout
|
||||
- [ ] Special characters (CJK, emoji, HTML entities) render correctly
|
||||
- [ ] Markdown rendering (code blocks, tables, lists)
|
||||
- [ ] LaTeX/MathJax rendering (if applicable)
|
||||
- [ ] Code syntax highlighting (if applicable)
|
||||
|
||||
## Encoding
|
||||
- [ ] No BOM markers in HTML templates (`ef bb bf`)
|
||||
- [ ] No leading whitespace before `<!DOCTYPE>`
|
||||
- [ ] UTF-8 charset declared in meta tag
|
||||
- [ ] Python source files: no BOM
|
||||
|
||||
## SEO & Metadata
|
||||
- [ ] `<title>` tag present and descriptive on each page
|
||||
- [ ] `<meta name="description">` present
|
||||
- [ ] Open Graph tags (`og:title`, `og:description`, `og:url`, `og:image`)
|
||||
- [ ] Twitter Card tags
|
||||
- [ ] Canonical URL (`<link rel="canonical">`)
|
||||
- [ ] `robots.txt` exists
|
||||
- [ ] `sitemap.xml` exists and is valid
|
||||
- [ ] RSS feed (if blog) exists and is valid XML
|
||||
|
||||
## Accessibility
|
||||
- [ ] All `<img>` have `alt` text (or `aria-hidden` for decorative)
|
||||
- [ ] No `user-scalable=no` in viewport meta
|
||||
- [ ] Sufficient color contrast (text vs background)
|
||||
- [ ] Skip-to-content link (visually hidden)
|
||||
- [ ] Keyboard navigation: Tab order logical
|
||||
- [ ] ARIA labels on interactive elements without visible text
|
||||
- [ ] Form labels associated with inputs
|
||||
|
||||
## Performance
|
||||
- [ ] All static assets return 200 (CSS, JS, images)
|
||||
- [ ] No broken links (404s in static resources)
|
||||
- [ ] CDN reliability (especially for users in China — jsDelivr may timeout)
|
||||
- [ ] Page load doesn't hang on slow external resources
|
||||
- [ ] Resource count reasonable (no excessive requests)
|
||||
|
||||
## Responsive Design
|
||||
- [ ] Layout at 375px (mobile) — no horizontal overflow
|
||||
- [ ] Layout at 768px (tablet) — breakpoint works
|
||||
- [ ] Layout at 1440px (desktop) — content centered
|
||||
- [ ] Touch targets large enough (44x44px minimum)
|
||||
|
||||
## Cross-Browser
|
||||
- [ ] Chrome/Chromium rendering
|
||||
- [ ] Firefox rendering
|
||||
- [ ] Safari rendering (WebKit differences)
|
||||
- [ ] Edge rendering
|
||||
|
||||
## Operations
|
||||
- [ ] `/health` endpoint returns `{"status":"ok"}` per service
|
||||
- [ ] 404 page is custom (not default framework error)
|
||||
- [ ] 500 errors don't leak stack traces to users
|
||||
- [ ] Audit log captures admin actions (verify from source)
|
||||
- [ ] Audit log captures login attempts (success/failure)
|
||||
|
||||
## Consistency (cross-service)
|
||||
- [ ] All pages include same CSS files (mobile.css, etc.)
|
||||
- [ ] All pages include same JS files (loader.js, etc.)
|
||||
- [ ] All pages have site-wide navigation bar
|
||||
- [ ] All pages have same security headers
|
||||
- [ ] All pages have same viewport meta
|
||||
|
||||
## Security Headers
|
||||
- [ ] `X-Content-Type-Options: nosniff`
|
||||
- [ ] `X-Frame-Options: DENY`
|
||||
- [ ] `Referrer-Policy: strict-origin-when-cross-origin`
|
||||
- [ ] `Content-Security-Policy` present and reasonable
|
||||
- [ ] No `unsafe-eval` in CSP (check for `'unsafe-eval'`)
|
||||
69
dogfood/references/server-inspection.md
Normal file
69
dogfood/references/server-inspection.md
Normal file
@@ -0,0 +1,69 @@
|
||||
# Server Inspection Reference
|
||||
|
||||
When asked to inspect a server without a URL, assume the **local machine Hermes runs on**.
|
||||
|
||||
## Quick Checklist
|
||||
|
||||
### System Resources
|
||||
```bash
|
||||
# CPU, load, uptime
|
||||
uptime && top -bn1 | head -3 && nproc
|
||||
|
||||
# Memory
|
||||
free -h
|
||||
|
||||
# Disk
|
||||
df -h
|
||||
```
|
||||
|
||||
### Running Services & Processes
|
||||
```bash
|
||||
# All listening ports
|
||||
ss -tlnp | grep LISTEN
|
||||
|
||||
# Top processes by CPU
|
||||
ps aux --sort=-%cpu | head -10
|
||||
|
||||
# Docker containers
|
||||
docker ps -a
|
||||
```
|
||||
|
||||
### Service Manager
|
||||
```bash
|
||||
systemctl list-units --type=service | grep running
|
||||
# or
|
||||
service --status-all
|
||||
```
|
||||
|
||||
### Network
|
||||
```bash
|
||||
# All LISTEN ports (not just common ones)
|
||||
ss -tlnp
|
||||
|
||||
# DNS resolution test
|
||||
nslookup example.com
|
||||
```
|
||||
|
||||
### Security
|
||||
```bash
|
||||
# fail2ban status
|
||||
fail2ban-client status
|
||||
|
||||
# UFW firewall (if enabled)
|
||||
ufw status
|
||||
```
|
||||
|
||||
## Scope Signals
|
||||
|
||||
| User says | Means |
|
||||
|-----------|-------|
|
||||
| "服务器巡检" / "server inspection" | Local machine (no URL given) |
|
||||
| "巡检 ephron.ren" | Remote web service at that domain |
|
||||
| "check the service on port 8000" | Likely remote host:port |
|
||||
| "你的服务器" / "this machine" | Local machine explicitly |
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
- **Don't** default to checking remote web services when no URL is provided
|
||||
- **Don't** assume the remote service is on the same machine as Hermes
|
||||
- **Do** ask for clarification if "server" could mean local or remote
|
||||
97
dogfood/references/session-learnings-ephron-qa.md
Normal file
97
dogfood/references/session-learnings-ephron-qa.md
Normal file
@@ -0,0 +1,97 @@
|
||||
# Session Learnings: ephron.ren QA (2026-05-03)
|
||||
|
||||
## Environment Facts
|
||||
- 5 services: Home(8000), Auth(8001), Blog(8002), Canvas(8003), Prompt(8004)
|
||||
- Auth: FastAPI + Tortoise ORM, `.ephron.ren` domain cookie
|
||||
- RBAC: user(10) < admin(20) < owner(30)
|
||||
- CSRF: `{unix_timestamp}:{sha256_hex}` format, 75 chars, per-GET refresh
|
||||
- Rate limits: login 5/min, register 6/hour, comments 6/min, likes 11/min, save 21/min
|
||||
|
||||
## High-Value Findings (Reproducible Patterns)
|
||||
|
||||
### CSP script-src-elem Override (Critical)
|
||||
- **Symptom**: Buttons with `onclick="fnName()"` do nothing, `typeof fnName` returns `undefined`
|
||||
- **Root cause**: `script-src-elem 'self' https://cdn.example.com` overrides `script-src 'unsafe-inline'`
|
||||
- **Detection**: `curl -sI URL | grep content-security-policy`, look for `script-src-elem` without `'unsafe-inline'`
|
||||
- **Impact**: All inline JS blocked → save/publish/discard buttons broken, client-only validation bypassed
|
||||
|
||||
### CSP form-action Blocks Cross-Origin Redirects (Critical)
|
||||
- **Date**: 2026-05-05
|
||||
- **Symptom**: Login form submits (POST appears in network tab), server sets cookie, but browser stays on login page — no redirect
|
||||
- **Root cause**: CSP `form-action 'self'` on the 303 redirect response blocks navigation to cross-origin targets
|
||||
- **Reproduction**:
|
||||
1. Visit `https://auth.ephron.ren/login?redirect=aHR0cHM6Ly93d3cuZXBocm9uLnJlbi8=` (redirect=base64 of `https://www.ephron.ren/`)
|
||||
2. Fill username/password, click submit
|
||||
3. Browser sends POST to `/api/login` (same origin ✅ allowed)
|
||||
4. Server returns 303 to `https://www.ephron.ren/` with CSP header containing `form-action 'self'`
|
||||
5. Browser blocks redirect: `https://www.ephron.ren/` ≠ `self` (`https://auth.ephron.ren`)
|
||||
- **Controlled test**: Same-origin redirect (`auth.ephron.ren/admin`) works fine; cross-origin fails
|
||||
- **Console error**: `Sending form data to 'https://auth.ephron.ren/api/login' violates the following Content Security Policy directive: "form-action 'self'"`
|
||||
- **Fix**: Skip CSP header on 303 responses (empty body, no protection value), or use JS redirect
|
||||
- **Affected pages**: ALL pages that redirect to login with a cross-origin redirect target (www/blog/canvas/prompt subdomains)
|
||||
- **Key source files**: `shared/security_headers.py` (CSP middleware), `auth/src/routes/api.py` (login endpoint), `auth/src/utils/redirect.py` (redirect validation)
|
||||
|
||||
### Server-Side Password Validation Missing
|
||||
- **Test**: `curl -X POST /api/register -d 'username=test&password=123&password_confirm=456&invite_code=CODE'`
|
||||
- **Expected**: 400/422 with validation error
|
||||
- **Actual**: 303 redirect (registration succeeds with weak/mismatched passwords)
|
||||
- **Root cause**: Validation only in client JS (blocked by CSP)
|
||||
- **Lesson**: Always test form validation with curl, not just browser
|
||||
|
||||
### Fulltext Search Silent Failure
|
||||
- **Test**: `GET /posts?q=openclaw&mode=fulltext` returns 0 results, `mode=simple` returns 6
|
||||
- **Root cause**: BM25 index not built or jieba tokenizer not installed
|
||||
- **Detection**: Compare simple vs fulltext results for same query
|
||||
|
||||
### API Auth Order Bug
|
||||
- **Test**: `POST /api/service/posts` without token, with invalid body
|
||||
- **Expected**: 401 (unauthenticated)
|
||||
- **Actual**: 422 (body validation error — leaks endpoint info)
|
||||
- **Root cause**: Pydantic validation middleware runs before auth middleware
|
||||
|
||||
## Delegate Task Sizing
|
||||
- Curl-only tasks: max ~15-20 test cases per delegate (30+ cases timeout at 600s)
|
||||
- Browser tasks: max ~5-8 interactions per delegate (each = 10-30s)
|
||||
- Use `execute_code` with `from hermes_tools import terminal` for fastest execution
|
||||
- Parallel delegates: 3 max, but each should be independently scoped
|
||||
|
||||
## Cookie Jar Synchronization
|
||||
- CSRF token changes on every GET request
|
||||
- Must use SAME cookie jar for GET (extract token) and POST (submit form)
|
||||
- Multiple CSRF tokens on one page (one per form) — extract from specific form context
|
||||
- Cross-service cookies: Domain=.ephron.ren should work for all subdomains
|
||||
- If cross-service test fails, check cookie jar file, not the cookie itself
|
||||
|
||||
## Content Restoration Pattern (Playwright)
|
||||
When homepage/admin content is accidentally overwritten, restore via Playwright:
|
||||
1. Prepare JSON with original content (experience/projects/skills/contact/footer)
|
||||
2. Login → navigate to admin page
|
||||
3. Use `page.evaluate()` to set form fields by `id=` (NOT `name=` — admin forms use id):
|
||||
```js
|
||||
document.getElementById('contact_email').value = '...';
|
||||
document.getElementById('footer_copyright').value = '...';
|
||||
```
|
||||
4. Set structured data: `initialContent.experience = [...]; renderExperience();`
|
||||
5. Set `is_draft: false` for items that should be published
|
||||
6. Collect and publish:
|
||||
```js
|
||||
const content = collectFormData();
|
||||
document.querySelector('input[name="content_json"]').value = JSON.stringify(content);
|
||||
// Find form with content_json input, set action=/admin/publish, submit
|
||||
```
|
||||
7. Verify with `curl -s https://site/` checking for restored content strings
|
||||
|
||||
## Form Field Discovery
|
||||
- Admin page fields may use `id=` instead of `name=` — check both:
|
||||
```bash
|
||||
curl -s -b cookies /admin | grep -oP 'id="[^"]*"' | sort -u
|
||||
curl -s -b cookies /admin | grep -oP 'name="[^"]*"' | sort -u
|
||||
```
|
||||
- `collectFormData()` reads from visible form elements, not hidden `content_json`
|
||||
- Setting `content_json` directly is overwritten by `collectFormData()` on submit
|
||||
|
||||
## Playwright vs curl for Form Submission
|
||||
- **curl**: CSRF token sync is fragile (token changes per-GET, cookie jar must match)
|
||||
- **Playwright**: Handles cookies/CSRF automatically, but CSP may block inline JS
|
||||
- **Best approach**: Use Playwright + `page.evaluate()` to bypass CSP-blocked functions
|
||||
- **Pattern**: Set form fields via JS → call `collectFormData()` → set `content_json` → submit form directly
|
||||
Reference in New Issue
Block a user