AI crawlers and robots change log
What changed in AI crawlers and robots, by date: every Paralax brief about it, each linking its source where one exists.
Dated briefs
Google, Microsoft and OpenAI Will Honor One AI-Training Opt-Out. Microsoft Also Splits Training From Answers
A shared robots.txt token for AI training was announced October 30. Bing adds a separate page-level grounding setting; Google's post does not say what opting out does to answers.
Primary source: blogs.bing.com
OpenAI's Crawl Bot Jumped 838% While Its Answer Bot Fell 49%: Don't Block OAI-SearchBot Over It
If OAI-SearchBot traffic spiked this month, blocking it is the wrong move: that costs ChatGPT search eligibility. Five weeks of logs across five domains show why — ChatGPT-User fetches fell 49% while OAI-SearchBot crawls rose 838%, and 87% of that crawl rise came from one domain, a site event, not an engine change.
Primary source: developers.openai.com
OpenAI's ChatGPT Fetch Bot Turns Robots.txt Into a Citation Access Problem
OpenAI separates ChatGPT Search, training crawls, and user-triggered fetches into different access rules.
Primary source: developers.openai.com
Meta's Web Crawler Turns AI Search Into A Source Architecture Problem
Meta crawler docs make AI search visibility a crawl, source, and citation architecture problem.
Does llms.txt Improve AI Search Visibility? 137,000 Domains Say No
New data across 137,000 domains shows 97% of llms.txt files never get read. Here's what actually moves AI citations.
Cloudflare and OpenAI Discovery Pilot Moves the Contest From Being Found to Being Chosen
Cloudflare says its OpenAI pilot added hundreds of millions of new pages to ChatGPT search, and our crawl logs show discovery was already fast while selection was not.
Primary source: blog.cloudflare.com
Web Bot Auth Turns AI Crawlers Into Accountable Search Actors
Web Bot Auth moves AI crawler access from user-agent claims to signed identity.
Primary source: developers.google.com
Cloudflare Just Made Publisher Signals Part of AI Search Infrastructure
Cloudflare's AI search push turns freshness, permissions, and usage data into ranking infrastructure.
Primary source: blog.cloudflare.com