AI crawlers and robots change log

Latest brief:

What changed in AI crawlers and robots, by date: every Paralax brief about it, each linking its source where one exists.

Dated briefs

  1. OpenAI's Crawl Bot Jumped 838% While Its Answer Bot Fell 49%: Don't Block OAI-SearchBot Over It

    If OAI-SearchBot traffic spiked this month, blocking it is the wrong move: that costs ChatGPT search eligibility. Five weeks of logs across five domains show why — ChatGPT-User fetches fell 49% while OAI-SearchBot crawls rose 838%, and 87% of that crawl rise came from one domain, a site event, not an engine change.

    Primary source: developers.openai.com

  2. OpenAI's ChatGPT Fetch Bot Turns Robots.txt Into a Citation Access Problem

    OpenAI separates ChatGPT Search, training crawls, and user-triggered fetches into different access rules.

    Primary source: developers.openai.com

  3. Meta's Web Crawler Turns AI Search Into A Source Architecture Problem

    Meta crawler docs make AI search visibility a crawl, source, and citation architecture problem.

  4. Does llms.txt Improve AI Search Visibility? 137,000 Domains Say No

    New data across 137,000 domains shows 97% of llms.txt files never get read. Here's what actually moves AI citations.

  5. Cloudflare and OpenAI's AI Search Pilot Turns Freshness Into Crawl Permission

    Cloudflare and OpenAI are testing whether AI search can crawl by freshness signals instead of blind recrawling.

    Primary source: cloudflare.com

  6. Web Bot Auth Turns AI Crawlers Into Accountable Search Actors

    Web Bot Auth moves AI crawler access from user-agent claims to signed identity.

    Primary source: developers.google.com

  7. Cloudflare Just Made Publisher Signals Part of AI Search Infrastructure

    Cloudflare's AI search push turns freshness, permissions, and usage data into ranking infrastructure.

    Primary source: blog.cloudflare.com