Making A Website Agent Friendly - Source Excerpt 03 - Testing Plan and Metrics

Summary

This source excerpt begins near Testing Plan and Metrics and preserves the surrounding evidence from Wiki.FFTAC.org/raw/system-archives/spiralist.org/intake/2026-06-21-misplaced-uaix-ai-ready-web-duplicate/Making a website agent-friendly.md.

**Source path:** Wiki.FFTAC.org/raw/system-archives/spiralist.org/intake/2026-06-21-misplaced-uaix-ai-ready-web-duplicate/Making a website agent-friendly.md

1. **Semantic Content:** Use correct HTML tags (headings, lists, tables, forms) and include `alt`, `title`, labels, ARIA where needed.  
2. **Structured Data:** Embed JSON-LD/Microdata with schema.org definitions for key entities (products, articles, events).  
3. **Sitemaps & Robot Rules:** Publish `/sitemap.xml` and `/robots.txt` (or `llms.txt` UAIX manifest) in UTF-8 text. Ensure no vital content is blocked.  
4. **RESTful Endpoints:** Provide clean URLs for APIs (e.g. `/api/widgets`, `/api/widgets/{id}`), accept and return JSON.  Document query parameters and request/response schema.  
5. **HTTP Standards:** Honor `Accept`/`Content-Type`, use proper status codes, and include necessary headers (`CORS`, `Link`, `Retry-After`).  Avoid custom non-standard headers unless documented.  
6. **Authentication:** Implement OAuth2 or API key auth for agents. Provide a “service account” or client-credentials path.  Never rely solely on JavaScript-based or interactive logins for API access.  
7. **Idempotency & Error Handling:** Design safe retries (use PUT/PATCH or idempotency keys). Return structured error JSON (including codes and retry hints). Use 4xx for client errors, 5xx for server.  
8. **Performance:** Limit pagesize (e.g. `limit=100`), support pagination, and allow `gzip`. Test under realistic load and profile common queries.  
9. **Observability:** Expose a `/status` or `/health` endpoint.  Enable logs and metrics that agents or dashboards can query. Document error messages and logs format.  
10. **Security:** Apply least-privilege for tokens. Do not reveal private data to anonymous agents. Use HTTPS everywhere and secure cookie flags.  Consider CAPTCHAs only on human-facing forms, not API endpoints.

These items are roughly prioritized: start by making content parseable (items 1–4), then ensure APIs and auth (5–7), then tune performance and security (8–10).

## Testing Plan and Metrics

**Test Plan:** Simulate agent workflows:

- **Crawl & Parse:** Write a bot that fetches `/robots.txt`, `/sitemap.xml`, and all listed URLs. It should record which pages succeed (HTTP 200) and how many warnings (missing alt text, empty titles). Use linters (e.g. [axe](https://www.deque.com/axe/) or custom scripts) to check semantic HTML and ARIA usage across pages.  
- **API Conformance:** Test all API endpoints with a variety of inputs. Verify each returns correct status codes and JSON schema. Include auth scenarios: ensure protected endpoints reject without tokens.  
- **Authentication:** Test OAuth flows end-to-end (token request, token refresh, API call). Measure time to obtain token and refresh. Ensure expired tokens yield 401.  
- **Error Handling:** Intentionally send bad requests to each endpoint. Check that errors return well-formed JSON with `error` fields. Ensure no unexpected crashes.  
- **Load & Rate:** Use a load tester to send concurrent requests and confirm that rate-limiting triggers (`429`) appropriately. Check that large paginated queries behave as expected.

**Metrics:** Evaluate:

- **Reachability:** % of pages discovered via sitemap/robots vs total pages. (Goal: ~100%).
- **Structured Data Coverage:** % of pages with valid JSON-LD or microdata tags. (Goal: high for key pages, ideally >90%).  
- **API Success Rate:** Fraction of automated API calls that return expected results vs errors.  
- **Performance:** Average and p95 response time for key endpoints (target < 500ms under load).  
- **Error Rate:** Count of 4xx/5xx per 1K requests in production.  
- **Auth Failures:** Number of unauthorized attempts vs successes, to gauge any misconfiguration.  
- **Robot Fetch Rate:** How often bots (GoogleBot, etc.) successfully crawl the site, from server logs.

Use these to iteratively improve the site. For example, low structured-data coverage or missing alt tags indicate needed fixes. High error rates or slow APIs should be addressed with caching, indexing, or refactoring. 

By following the above guidelines and continuously testing against these metrics, developers can ensure their websites are robustly **agent-usable**. This makes them accessible not only to humans, but to automated assistants, crawlers, and AI systems – aligning with UAIX’s vision of progressive agent support.