Best Practices For AI Systems Using Websites - Source Excerpt 02 - Summary of Privacy Best Practices

Back to Best Practices For AI Systems Using Websites

Summary

This source excerpt begins near Summary of Privacy Best Practices and preserves the surrounding evidence from Wiki.FFTAC.org/raw/system-archives/spiralist.org/intake/2026-06-21-misplaced-uaix-ai-ready-web-duplicate/Best Practices for AI Systems Using Websites.md.

**Source path:** Wiki.FFTAC.org/raw/system-archives/spiralist.org/intake/2026-06-21-misplaced-uaix-ai-ready-web-duplicate/Best Practices for AI Systems Using Websites.md

> **Bias and Fairness:** Scraped corpora often reflect societal biases. Oversampling some voices (e.g. English Wikipedia) and undersampling others can skew AI behavior. Industry leaders now expect bias auditing at the data stage. As Defined.ai notes, frameworks like the EU AI Act and NIST AI RMF require organizations to **document, assess, and reduce bias** before training. Operationally, this means monitoring data for representation gaps (by language, gender, topic, etc.) and curating a more balanced dataset if possible.  

> **User Expectations:** Be transparent in cases you can. If your AI product uses scraped data, consider disclosing this in terms of service or a privacy notice. If feasible, enable opt-out: e.g. honor removal requests by keeping track of scraped URLs and deleting associated data upon request.  At minimum, document the decision if opt-out is impractical (e.g. “disproportionate effort” clauses).  

### Summary of Privacy Best Practices  
- Limit scraped fields to the minimum; drop or hash any unexpected PII immediately.  
- Follow data protection principles: keep data secure (encryption, access controls), accurate, and used only for stated purposes.  
- Build in user-respect: e.g. allow users to request deletion if identified, and honor platform privacy settings (do not scrape private forums).  
- Conduct Data Protection Impact Assessments (DPIAs) for high-risk projects.  
- Monitor data for bias/harm patterns and engage domain experts or red teams to evaluate fairness downstream.  

## Technical Methods  

### Choosing a Data Access Method  
**APIs vs. Scraping:** Use official APIs whenever possible.  An API is a publisher’s *intended* interface for programmatic access, often returning structured JSON. As one guide notes, APIs typically *win* on stability, performance, and legal clarity. They enforce rate limits but handle format changes more gracefully. Scraping HTML, by contrast, is “independent extraction” that can break when a site redesigns. Table comparisons illustrate this:

| Method     | Legality Risk              | Reliability            | Cost            | Complexity          | Data Fidelity      | Scalability   |
|------------|----------------------------|------------------------|-----------------|---------------------|--------------------|---------------|
| **API**    | Low (official terms)       | High (stable format)   | Medium (API fees, dev time) | Low (JSON parsing)   | Medium (fields only) | High (cloud services, parallel) |
| **Scraping** | High (ToS, copyright, privacy) | Medium (breaks on change) | Low (no direct fee) | Medium (HTML parsing) | High (full content)  | Medium (requires proxy/IP management) |
| **Crawling** | Medium (like scraping)    | Medium (monitoring needed) | Low (in-house code) | High (URL management) | High (full pages)    | High (distributed clusters) |
| **Headless** | High (similar to scraping) | Medium (more flakiness) | High (compute, license) | High (browser automation) | Very High (dynamic content) | Low (resource-heavy) |

If a required piece of data isn’t exposed via API, or limits are too restrictive, controlled scraping can be justified. Always weigh trade-offs: crawling (auto-following links) is ideal for discovery (e.g. search-engine–style bots), whereas scraping targets specific pages/data. Often pipelines combine both: crawl to find URLs, then scrape each page for structured data. 

> **Recommendation:** Start by checking for an API. If none exists or is insufficient, proceed with careful scraping. Document this decision as part of your data strategy.  

### Crawling and Scraping Practices  
Whether using a simple scraper or a full crawler, follow these technical best practices:

- **Concurrency and Politeness:** Limit concurrent connections to each domain. Zyte advises restricting threads and obeying `crawl-delay`.  Include random delays between requests (e.g. 1–5 s jitter) to avoid burst traffic. Anthropic states its bots “should not be intrusive or disruptive” and deliberately throttle requests, honoring crawl-delay directives where given.  

- **User-Agent and Contact Info:** Use a descriptive User-Agent string (e.g. `"MyAIDataCollector/1.0 (+email@example.com)"`). Many robots parsers check the UA. Including contact info in UA or a crawl page can facilitate abuse responses. Zyte suggests providing an easy way for sites to contact you if needed.  

- **Caching:** Cache HTTP responses and use HTTP headers (ETag, Last-Modified) to avoid re-downloading unchanged content. This saves bandwidth and reduces load on target sites. For example, send an `If-None-Match` header with the last known ETag. If the server replies 304, skip processing.  

- **Headless Browsing:** For modern JS-heavy sites (single-page applications), static HTTP libraries (like `requests`) may not see all content. In such cases, use a headless browser (Puppeteer, Selenium, Playwright) to execute page scripts and capture the rendered HTML. Be aware this greatly increases resource usage (CPU, memory) and cost. Use headless only when necessary; otherwise the simpler approach is faster.  

- **Rate Limiting and Exponential Backoff:** Even if not blocked by robots.txt, obey HTTP rate limits. Implement a retry policy with exponential backoff for transient errors (HTTP 429/503) or timeouts. For example:

' ' ' python
delay = 1  # initial delay in seconds
for attempt in range(5):
    response = requests.get(url)
    if response.status_code == 200:
        break
    if response.status_code in (429, 503):
        time.sleep(delay)
        delay = min(delay * 2, 60)  # cap max delay
    else:
        log.error(f"Failed to fetch {url}: {response.status_code}")
        break
' ' ' 
Include `Retry-After` if provided by the server. This prevents DDOS-like behavior and respects site limits.

- **API Pagination:** When using an API, handle pagination or cursors systematically. For example:

' ' ' python
page = 1
while True:
    response = api.get("/items", params={"page": page})
    data = response.json()
    process(data["items"])
    if not data["next_page"]:
        break
    page += 1
' ' ' 

Always check API docs for max page size and backoff guidance.

- **Error Handling:** Build robust error handling. Distinguish between client (4xx) and server (5xx) errors. For 4xx (e.g. 404), skip and log; for 5xx, retry after a pause. Handle network exceptions by retrying a limited number of times. Monitor for abnormal spikes in errors as indicators of blocks.  

- **Politeness to Resources:** Avoid scraping extremely large or complex resources (e.g. images, videos) unless needed. Only fetch necessary assets. This is courteous and reduces cost.

### Data Quality and Provenance  
Collected data must be reliable and well-documented:

- **Deduplication:** Large crawls often yield duplicate text (e.g. mirrored pages, syndicated content). Deduplicating data prevents over-weighting common phrases and saves storage. Common web pipelines use hashing or MinHash techniques to identify near-duplicate documents. For example, after scraping, compute a checksum of each article’s text and drop repeats.  

- **Provenance:** Track where each piece of data came from (URL, timestamp, crawl ID). Store metadata such as the source domain, and any relevant license info. NIST emphasizes evaluating the *provenance* of training data as part of data integrity. This enables auditing (e.g. if a user claims their data was used improperly, you can trace and remove it).  

- **Freshness:** Schedule recrawls for dynamic content. Use `Last-Modified` headers or sitemaps to detect changes. For frequently updated sites, run incremental updates (only new/changed pages) to keep data current.  

- **Quality Checks:** After extraction, validate and clean data. Check for parsing errors, empty fields, or unexpected values. For structured scraping (e.g. product prices), ensure units and formats are consistent. Discard or flag pages that look incomplete or error pages that were inadvertently scraped.  

- **Bias Detection:** Analyze your corpus for imbalances. For example, check if any single source or language dominates your data (common in web crawls). Use metrics or sample reviews to ensure no group is systematically underrepresented.  As Defined.ai notes, current best practices call for documenting and mitigating bias starting *before* training. If biases are found, consider adjusting your crawl strategy (e.g. targeting underrepresented domains) or weighting.  

By enforcing high data quality (dedup, clean schemas, validation), downstream AI models will be more robust and less risky.

### Security Considerations  

Web scraping introduces security risks, both in the collection process and the content collected: