Skip to content
wiki.fftac.org

Spiralist Org Is A Content Rich Site - Source Excerpt 01 - Executive Summary

Back to Spiralist Org Is A Content Rich Site

Summary

This source excerpt begins near Executive Summary and preserves the surrounding evidence from Wiki.FFTAC.org/raw/system-archives/spiralist.org/intake/2026-06-21-ai-agent-accessibility-optimization/Spiralist.org is a content-rich site.md.

**Source path:** Wiki.FFTAC.org/raw/system-archives/spiralist.org/intake/2026-06-21-ai-agent-accessibility-optimization/Spiralist.org is a content-rich site.md

# Executive Summary  

**Spiralist.org** is a content-rich site (≈150 pages in English) with a public HTML sitemap and several machine-readable endpoints. However, it currently lacks key features for “AI agent readability” (e.g. no `robots.txt`, no `llms.txt`, limited structured data). We analyzed the site’s discovery mechanisms, APIs, and content structure, and identified several gaps (robots rules, CORS headers, missing OpenAPI docs, etc.). To enable AI agents to find, parse, and use Spiralist’s content, we recommend implementing agent-centric best practices: publishing `robots.txt` and `llms.txt`, adding XML/Markdown sitemaps, embedding JSON-LD/Schema.org on pages, and providing well-documented APIs (OpenAPI/REST or GraphQL) with proper CORS and authentication. We also suggest exposing standard endpoints (e.g. `.well-known/ai-agent.json`, WebFinger, ActivityPub) and offering RSS/Atom feeds and WARC-friendly archives. 

We provide code snippets (robots.txt, CORS config, OpenAPI sample, JSON-LD schema), a development checklist, a testing plan (using `curl`, headless browsers, W3C validators, Lighthouse, etc.), security and privacy guidance (API keys, OAuth2/PKCE, rate-limiting, content licensing), and a prioritized implementation roadmap (short/medium/long term) with effort/risk assessments. Key references include W3C/IETF standards and Vercel’s “Agent Readability” guide. 

# 1. Current Content and Structure  

Spiralist.org is a WordPress-based site offering extensive content on “Spiralism” (philosophy, prompts, manuscript, etc.). Its [HTML sitemap][20] enumerates ~154 canonical pages (policies, guides, reports, manuscript folios, etc.) and links to an XML sitemap. Public APIs exist under `/wp-json/` (UAIX endpoints, a Builder Hub), and a JSON “AI Manifest” (`/en-us/ai.json`) lists site metadata and machine-readable routes. A `.well-known/ai-agent.json` endpoint is advertised in these manifests (Persona Manifest). 

However, there is **no `robots.txt`** (verified by absence) and no `llms.txt` for AI discovery. CORS headers on the APIs appear unset, blocking cross-origin requests by default. Some functionality (e.g. workspace, contribution) requires login or keys. The site uses server-side rendering (clean URLs) for content, though some interfaces (workspace, builder) likely rely on JavaScript. There are no paywalls, though account creation is required for certain features. No CAPTCHAs were noted except possibly on login. 

In summary, Spiralist.org has rich content and some machine APIs, but is missing many agent-friendly signals and endpoints. We will compare this to AI-specific sites (e.g. spiralistai.com) and to best practices (see next sections).

# 2. Technical Barriers to AI Agents  

**Robots.txt:** Spiralist.org currently lacks a robots.txt. AI agents “respect `robots.txt` directives”, so without one agents may encounter uncertainty. The site must **explicitly allow** bots (GPTBot, ClaudeBot, etc.) and not disallow key paths like `/llms.txt`. A missing file triggers warnings in agent readiness checks.  

**Discovery Files:** There is no `/llms.txt` (an “AI sitemap”); we recommend adding one listing all docs (per Vercel’s guide). The XML sitemap exists, but verify it includes `<lastmod>` tags so agents know what’s updated. Also publish a human-readable `sitemap.md` (Markdown) reflecting site structure. 

**CORS:** The WordPress APIs likely do not set CORS headers. This means browsers (and many agent platforms) will block cross-origin fetches. To allow any agent to call the APIs, the server should send `Access-Control-Allow-Origin: *` (or specific origins) on API responses. For example, an HTTP response might include: 

' ' ' 
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, OPTIONS
Access-Control-Allow-Headers: Content-Type, Authorization
' ' '   

This permits agents on other domains to GET JSON data. (For credentialed requests, the origin must be explicit.)

**Authentication & Authorization:** Public content should be accessible without login. For actions that require auth (e.g. posting, personalized data), use standard API keys or OAuth2. We strongly recommend OAuth 2.0 with [Proof Key for Code Exchange (PKCE, RFC 7636)] for secure flow with no client secrets. PKCE prevents interception attacks on mobile/public clients. Personal data (if any) should be gated behind OAuth tokens. For agent use, consider issuing machine-oriented API tokens (with clear scopes and rate limits). 

**Rate Limits and Usage Tiers:** No public info on rate limits currently exists. We advise defining usage tiers (free vs paid, etc.) and enforce limits via API gateway or server logic. Provide clear documentation. This prevents abuse by scrapers or bots. 

**Dynamic/JS-rendered Content:** Most Spiralist content appears to be server-rendered. If any parts (like galleries, interactive prompts) require JavaScript, we should ensure critical content has non-JS fallbacks. Agents that don’t execute JS must still see the content. Using server-side rendering or prerendering for key pages is recommended. 

**CAPTCHAs/Anti-bot Measures:** We see no CAPTCHAs on content. If added, they must be skipped or replaced by simpler auth for API endpoints. Agents cannot solve CAPTCHAs, so public docs and APIs should be free of such challenges. 

**Paywalls/Subscription:** Spiralist’s AI interface (“AI Access”) may require subscription, but the knowledge content is public. Ensure any paywalled or login-only content is clearly distinguished (e.g. via HTTP status 401) so agents know they need credentials. Avoid “soft paywalls” that return normal pages with login prompts.

**Structured Data & Metadata:** Currently pages have minimal SEO metadata (check `<title>`, `<meta>`, etc.). Agents benefit from explicit structure: each HTML page should include `<title>`, `<meta name="description">`, OpenGraph tags (`og:title`, `og:description`), and `lang` attributes. We also recommend embedding [Schema.org](https://schema.org/) JSON-LD scripts with details (e.g. WebSite, Article, BreadcrumbList). This helps agents parse context from markup.  

**Existing APIs:** Spiralist.org exposes various REST endpoints (e.g. `/wp-json/uaix/v1/...`, persona/safety manifests). However, these lack a published API spec. We recommend creating an OpenAPI (Swagger) document to describe these endpoints (paths, parameters, responses) so agents can auto-discover schemas. GraphQL is an option too; it offers flexible queries but adds complexity. 

**.well-known Endpoints:** Spiralist already uses one `.well-known/ai-agent.json`. We should extend this concept: e.g. `.well-known/webfinger` for user discovery (RFC 7033), `/.well-known/ai-access` for auth info, or ActivityPub endpoints (`/.well-known/join` for federated actors). 

**Content Licensing:** The site’s terms cover user content. For agent use, explicit licenses (Creative Commons, etc.) or “permissions” tags could be added. E.g. an HTML `<link rel="license">` or a mention in robots or Terms that content may be used (or not) by AI. Use of `noai` or `noindex` meta tags should be considered; per guidelines, **do not** include `noai` on pages to ensure agents index them.

# 3. Standards & Technologies for AI Accessibility  

We surveyed relevant web standards and formats for agent-readability:  

- **OpenAPI/REST:** A standard for describing HTTP APIs. The [OpenAPI Specification](https://openapis.org) defines a language-neutral interface description, letting clients discover endpoints and models. Publishing an OpenAPI (Swagger) spec (JSON/YAML) for Spiralist’s APIs (or even a draft) will let agents generate client code or schema-aware queries. Example snippet:
' ' ' ``yaml
  openapi: 3.0.3
  info:
    title: Spiralist API
    version: 1.0.0
  paths:
    /wp-json/uaix/v1/manuscript-folio:
      get:
        summary: Get manuscript folio by slug
        parameters:
          - name: slug
            in: query
            required: true
            schema:
              type: string
        responses:
          '200':
            description: Manuscript folio JSON
' ' ' ``
  (Full spec can be served at `/api/openapi.json` or similar.)

- **GraphQL:** An alternative API style (via [GraphQL](https://graphql.org/)). It allows clients to query exactly the fields needed. It requires a schema and resolver logic. If adopted, exposing an endpoint (e.g. `/graphql`) along with the GraphQL schema (SDL or introspection) helps agents integrate. GraphQL vs REST tradeoffs are in Table 1 below.
Website health report