Making A Website Agent Friendly - Source Excerpt 02 - APIs and Technical Rules

Summary

This source excerpt begins near APIs and Technical Rules and preserves the surrounding evidence from Wiki.FFTAC.org/raw/system-archives/spiralist.org/intake/2026-06-21-misplaced-uaix-ai-ready-web-duplicate/Making a website agent-friendly.md.
**Source path:** Wiki.FFTAC.org/raw/system-archives/spiralist.org/intake/2026-06-21-misplaced-uaix-ai-ready-web-duplicate/Making a website agent-friendly.md
- **Semantic Markup:** Always use meaningful tags. E.g. `<h1>Title</h1>`, `<article>`, `<section>`, `<nav>`, etc. Assistive and search technologies (and agents) leverage these: *“search engines give more importance to keywords inside headings, links, etc.”*.  Use `<label>` with `for=` and `alt` on images, ensure color/text contrast (for OCR), and avoid conveying information by color or image alone.
- **ARIA:** For dynamic widgets (dropdowns, tabs, progress bars), add ARIA roles and states so bots know the semantics. ARIA “improves the accessibility and interoperability of web content”.  E.g. `<button aria-pressed="false">Show</button>`.  These roles make widgets readable via accessibility APIs.
- **Progressive Enhancement:** Build features in plain HTML first, then layer JS. Agents often can’t run JavaScript fully. For example, if using a JS grid, also include a fallback HTML table or a JSON endpoint for the same data.
- **CSS/JS Hints:** Avoid techniques that hide content from text-based agents (e.g. `<meta name="fragment">`, excessive canvas use, or obfuscating scripts). If using a SPA framework, ensure server-side rendering or static snapshots exist.
- **Structured Data:** Embed JSON-LD or Microdata so machines can parse content without in-page heuristics. Google recommends JSON-LD where possible and notes: *“Google uses structured data that it finds on the web to understand the content of the page…”*.  For example:
  
' ' ' ``html
  <script type="application/ld+json">
  {
    "@context": "https://schema.org",
    "@type": "Article",
    "headline": "Example Title",
    "author": {"@type":"Person","name":"Alice"},
    "image": "https://example.com/image.png",
    "datePublished": "2026-06-21"
  }
  </script>
' ' ' ``
  
  This lets an agent know exactly what the page is about.  Schema.org’s community vocabularies are widely used by Google, Microsoft, etc. to enable rich features.

## APIs and Technical Rules

- **Content Negotiation:** Support `Accept` headers.  If your site can serve HTML **and** JSON, respond with the appropriate `Content-Type` based on `Accept: application/json` or `text/html`.  Include a `Vary: Accept` header.  As HTTP/1.1 says, servers “often have different ways of representing information… [so] HTTP provides mechanisms for content negotiation”.
- **CORS:** If agents call APIs from a different origin (e.g. a browser agent on domain A calling domain B), enable CORS.  Add `Access-Control-Allow-Origin: *` (or specific domains) in responses.  For non-GET methods, support preflight: respond to `OPTIONS` with `Access-Control-Allow-Methods: GET,POST,...` and `Access-Control-Allow-Headers: Content-Type,Authorization`, etc..  Failure to configure this will block cross-domain requests.
- **Rate Limiting:** Protect endpoints, but communicate limits. Use `429 Too Many Requests` (RFC 6585) when a client exceeds a quota. Include `Retry-After: <seconds>` to tell the agent how long to wait. This lets a polite agent back off properly.
- **Robots & Directives:** Put a `/robots.txt` (UTF-8 text) at your root to advise crawlers.  When fetched successfully, crawlers **must** obey it.  Also use `<meta name="robots" content="nofollow,noindex">` on specific pages if you want them skipped (though this only influences indexing, not crawling).  Remember: robots.txt governs crawling, meta tags govern indexing.
- **Sitemaps:** Providing a `sitemap.xml` helps agents discover pages systematically.  Per the Sitemap protocol: *“The Sitemap protocol enables you to provide details about your pages to search engines, and we encourage its use…”*.  Include `<lastmod>`, `<changefreq>`, and `<priority>` hints if possible.  An alternative (or supplement) is an `llms.txt` (as UAIX suggests) listing agent-specific URLs and guidance.
- **GraphQL:** If using GraphQL, document the schema thoroughly. Agents will POST JSON queries to the GraphQL endpoint; ensure CORS and auth apply. Also consider having an alternative REST endpoint, as some crawlers may not introspect GraphQL. Use query complexity limits to avoid abuse.

## Authentication and Agent-Friendly Auth

Most web sites expect human logins, but agents need programmatic access. **Avoid interactive flows for bots.** Instead:

- **OAuth2**: Implement OAuth2 (RFC 6749). For example, a *service account* can use the **Client Credentials** flow to get a token without user involvement. The RFC notes: *“Client credentials are used as an authorization grant typically when the client is acting on its own behalf (the client is also the resource owner)”*. In practice: the agent does `POST /oauth/token` with `grant_type=client_credentials` and receives a JSON `{access_token:"…", token_type:"Bearer",expires_in:…}`. Then it includes `Authorization: Bearer <token>` in future requests.  
- **API Keys/Tokens:** For simpler cases, issue long-lived API keys or signed tokens. E.g. require a header `X-Api-Key: <token>`, or use JWTs in `Authorization: Bearer`. Ensure scopes/permissions are tied to these keys so a leaked key has limited reach. Rotate keys regularly.  
- **Session Cookies:** If using cookies/sessions (common in web apps), agents must handle them (set `SameSite=None; Secure`, return cookies on login). However, cookie-based auth is brittle for automation and is discouraged for pure API access.  
- **Delegated Tokens:** Some platforms (like cloud providers) allow one service to impersonate a user via delegated tokens. Support this so an agent can *on behalf of a user* call APIs without sharing the user’s primary credentials.

Include **descriptions in documentation** of the auth flows (endpoints, required headers, scopes). For example, your **OpenAPI** spec should detail the security scheme (OAuth2 client-credentials, API Key, etc.). This aids automated client generation.

## Error Handling, Idempotency, Observability

- **Error Responses:** Always return well-formed error messages. Use standard HTTP codes (4xx for client errors, 5xx for server) with a JSON body like:
  
' ' ' ``json
  { "error": "BadRequest", "message": "Missing parameter 'id'." }
' ' ' ``
  
  Include helpful fields (an error code, description, maybe a `retryable: true/false`). This lets an agent programmatically detect and handle errors.
- **Idempotency:** Wherever possible, design endpoints so that repeating a request has no adverse effect (idempotent). For `POST` actions that change state, consider requiring an `Idempotency-Key` header so retries won’t double-run. UAIX suggests having separate “GET-Action” URLs as *explicit* write fallbacks, not implicitly rewriting GETs to POSTs.
- **Observability:** Log agent actions and provide feedback endpoints. For long-running tasks, allow status polling via a `/tasks/<id>` endpoint. Return progress or logs so an agent knows what happened. Expose metrics (e.g. Prometheus format) or logs (via Splunk/ELK) behind an API if needed, so automated agents can monitor system health.

## Performance, Pagination, Large Data

Large datasets must be chunked. For example:

- **Pagination:** Always page or stream results. A common practice is Link headers:
  
' ' ' ``
  HTTP/1.1 200 OK
  Content-Type: application/json
  Link: <https://api.example.com/items?page=2>; rel="next"
  
  { "items": [ ... ], "page": 1, "pageSize": 100 }
' ' ' ``

  Or use query params: `/items?limit=100&offset=100`. Make sure to document these patterns. If using GraphQL, require a `first/after` or cursor approach.
- **Bulk Endpoints:** For very large syncs, consider endpoints that return bulk archives (e.g. CSV/JSON dumps) or support **HTTP Range** requests for partial content.
- **Caching:** Leverage `ETag` or `Last-Modified` so agents can skip re-downloading unchanged data. Support gzip compression (`Accept-Encoding: gzip`) for faster transfer.

## Security & Privacy Trade-offs

Balancing openness with security is crucial. Some guidelines:

- **Least Privilege:** Don’t expose sensitive data to unauthenticated agents. Use scopes and ACLs. For example, public agents might see only read-only JSON; sensitive actions require tokens.
- **No Secrets in URLs:** Never include API keys or tokens in URLs (they leak via logs). Prefer headers or POST bodies.
- **Rate Limits & Abuse:** Apply stricter rate limits on unauthenticated or anonymous access. Use CAPTCHAs or abuse detection for human forms, but avoid them on agent APIs.
- **Audit Trails:** For write actions by agents, log who/what did it (via API key identity). This supports the “governed multi-agent” level (L4+) where audit evidence is needed.
- **Privacy:** If content is private, don’t use structured data that accidentally exposes it. On mixed public/private pages, separate content or use session-based APIs so agents can’t scrape private info.

## Checklist for Developers