AI Agent Website Accessibility Optimization Spiralist - Source Excerpt 03 - 5.1 Core AI Discovery Files and Their Functions

Back to AI Agent Website Accessibility Optimization Spiralist

Summary

This source excerpt begins near 5.1 Core AI Discovery Files and Their Functions and preserves the surrounding evidence from Spiralist/agent-file-handoff/Archive/2026-06-21/Improvement/ai-agent-accessibility-optimization/AI Agent Website Accessibility Optimization Spiralist.org.md.

**Source path:** Spiralist/agent-file-handoff/Archive/2026-06-21/Improvement/ai-agent-accessibility-optimization/AI Agent Website Accessibility Optimization Spiralist.org.md

Traditional web standards like robots.txt and sitemap.xml were designed for search engines indexing links; they are no longer sufficient to guide modern AI agents attempting to understand nuance, intellectual property permissions, and API endpoints. To bridge this structural gap, a new suite of machine-readable protocols known as AI Discovery Files has emerged. These files sit at the root directory of the domain and provide explicit, structured intelligence to LLMs regarding business identity, acceptable use policies, technical context, and brand definitions24.  
For a product like Spiralist, which actively operates within the AI ecosystem by connecting to Claude, ChatGPT, and Gemini via the Model Context Protocol17, implementing the complete suite of AI Discovery Files is an absolute mandate. Leaving these files unconfigured guarantees that Spiralist will remain part of the 99.7% of websites that are virtually invisible to AI recommendation engines27.

### **5.1 Core AI Discovery Files and Their Functions**

The following table outlines the essential AI discovery files that must be deployed to the root of the Spiralist domain, detailing their specific formats, purposes, and how AI agents process them.

| Discovery File | Format | Core Purpose and AI Agent Function |
| :---- | :---- | :---- |
| **llms.txt** | Markdown | Provides an AI-readable summary of the business, its core products, and links to vital markdown documentation. It acts as the primary, high-signal entry point for LLM ingestion24. |
| **llms-full.txt** | Markdown | Combines the entire documentation site into a single file. This gives an AI crawler a single ingestion point, reducing fetch overhead for complex RAG systems29. |
| **ai.txt** | Plain Text | Declares AI usage permissions, specifying whether content may be used for model training, indexing, or scraping, supplementing basic robots.txt blockages with nuanced licensing terms24. |
| **identity.json** | JSON | Delivers structured, site-scoped business identity data (Schema.org-style), establishing canonical identity beyond page-level markup24. |
| **brand.txt** | Plain Text | Outlines brand naming conventions, preferred terminology, and explicit instructions on how the AI should refer to the business in its generated responses24. |
| **faq-ai.txt** | Plain Text | Contains pre-authored, authoritative Q\&A designed specifically for AI retrieval, ensuring models use approved answers rather than generating assumptions based on scattered text23. |
| **developer-ai.txt** | Plain Text | Provides technical context regarding API availability, versioning conventions, and platform integrations specifically tailored for developer-facing AI agents23. |
| **robots-ai.txt** | Plain Text | Acts as an AI-specific crawler access directive file, offering granular control over bots like GPTBot or ClaudeBot without disrupting traditional Googlebot crawlers24. |

### **5.2 Deep Dive Implementation of llms.txt**

The llms.txt file is rapidly becoming an industry-standard convention, heavily supported by documentation platforms like Mintlify, for telling AI crawlers which parts of a site are best suited for ingestion29. For the Spiralist domain, this file must be meticulously structured. It must contain an H1 title indicating the site name, a blockquote summary of the platform's purpose, and structured links pointing directly to .md (Markdown) versions of critical pages29.  
Because markdown is significantly easier for LLMs to parse than complex HTML structures, offering clean markdown routes via llms.txt eliminates inference errors and improves the speed of ingestion30. The file should aggressively highlight the architectural differences between Spiralist’s "Data Layer Product," "Chat Layer Product," its "Think With AI" newsletter, and its "Claude Skills" repository, steering crawlers toward canonical documentation while explicitly bypassing outdated archives or low-value marketing pages16. Furthermore, developers can implement HTTP headers such as Link: \</llms.txt\>; rel="llms-txt" or X-Llms-Txt: /llms.txt to allow tools to discover the file without prior knowledge of its location29.

### **5.3 Establishing Legal Boundaries with ai.txt**

While the legacy robots.txt offers binary allow or disallow directives based on user-agent strings, it cannot express nuance. It cannot tell an AI organization, "You may crawl this site to answer live user search queries, but you may not scrape this proprietary data to train your base model." The ai.txt specification (and its associated ai.json counterpart) solves this dilemma by positioning a structured attachment surface for highly specific AI usage preferences24.  
By adopting established specifications such as the Spawning v1.7 format or the IETF draft-car-ai-txt-wellknown-00 standard, Spiralist can legally and programmatically dictate attribution requirements, content licensing, and granular training restrictions31. This creates an enforceable legal boundary for intellectual property, providing a basis for protecting the platform's unique methodologies and "Coherence Engine" logic from being strip-mined by foundational model developers17.

## **6\. Advancing Semantic Intelligence with Schema.txt**

While llms.txt is an excellent tool for providing static descriptions to AI systems, it inherently lacks the ability to create dynamic, queryable data networks. As artificial intelligence moves from merely summarizing text to executing autonomous actions, domains must evolve from static brochures to queryable knowledge graphs. This evolution is the domain of the proposed schema.txt standard36.

### **6.1 The Compute Cost Crisis and Pre-extracted Semantics**

Currently, when an AI agent requires structured data from a website, it must fetch the full HTML payload, parse the DOM, and utilize expensive inference processing to extract entities and map relationships36. The schema.txt standard bypasses this computationally expensive process entirely by creating a domain-level semantic catalog that directs AI systems straight to pre-extracted, indexed structured data endpoints36.  
By placing a schema.txt file at the root directory and referencing it in the traditional robots.txt file (e.g., Schema: https://spiralistai.com/schema.txt), Spiralist can offer AI systems up to a 90% reduction in inference costs, making the platform a highly attractive and frictionless target for agent interaction36.

### **6.2 Designing CDN Endpoint Architecture for AI Graph Traversal**

To implement schema.txt effectively, the Spiralist engineering team should map all core business entities (e.g., application features, subscription tiers, newsletter articles, and key personnel) and host this comprehensive JSON-LD data on a globally distributed Content Delivery Network (CDN) such as AWS CloudFront, Cloudflare Workers, or Azure CDN36.  
Instead of forcing an AI to scrape the homepage to understand the functional parameters of the "Chat Layer Product," the schema.txt file points the AI directly to an endpoint (e.g., https://cdn.spiralistai.com/schema/products.json)36. The AI reads this structured JSON file instantaneously, mapping the exact semantic relationships between Spiralist’s app modules, its pricing matrices, and its underlying technological capabilities37.

### **6.3 Table: Comparing LLMs.txt and Schema.txt Implementations**

Understanding when to utilize static markdown versus dynamic knowledge graphs is crucial. The following table contrasts the capabilities and use cases of both standards.

| Feature / Metric | llms.txt (Static Markdown) | schema.txt (Semantic Knowledge Graph) |
| :---- | :---- | :---- |
| **Primary Format** | Plain text / Markdown29. | Structured pointers to JSON-LD endpoints36. |
| **Best Used For** | Simple websites, documentation, and human-centric static text37. | Dynamic applications, SaaS platforms, and complex entity relationship mapping37. |
| **Query Performance** | Poor; requires the LLM to perform full-text search and inference37. | Excellent; allows direct API calls with advanced filtering capabilities37. |
| **Relationship Mapping** | Implicit, based on text descriptions and hyperlinks37. | Explicit, cryptographically defined entity relationships (e.g., Product relates to Tier)37. |
| **Scalability** | Limited; files become unwieldy with large, complex datasets37. | Excellent; designed specifically for massive, scalable datasets and API-first design37. |

This queryable architecture prevents the AI from misrepresenting the product during query fan-outs, effectively serving as an API-driven entry point for autonomous agents to interact with the platform intelligently rather than simply reading about it37.

## **7\. Model Context Protocol (MCP) Web Integration**