# AI crawlers and robots change log

> What changed in AI crawlers and robots, by date: every Paralax brief about it, each linking its source where one exists.

- Canonical: https://paralax.ai/engines/crawlers
- Latest brief: 2026-09-29
- Change log index: https://paralax.ai/engines

## Dated briefs

### 2026-09-29 — [OpenAI's Crawl Bot Jumped 838% While Its Answer Bot Fell 49%: Don't Block OAI-SearchBot Over It](https://paralax.ai/blog/openai-answer-time-fetch-index-crawl-divergence-2026)

If OAI-SearchBot traffic spiked this month, blocking it is the wrong move: that costs ChatGPT search eligibility. Five weeks of logs across five domains show why — ChatGPT-User fetches fell 49% while OAI-SearchBot crawls rose 838%, and 87% of that crawl rise came from one domain, a site event, not an engine change.

- Primary source: [developers.openai.com](https://developers.openai.com/api/docs/bots)
- Markdown: https://paralax.ai/blog/openai-answer-time-fetch-index-crawl-divergence-2026.md

### 2026-08-16 — [OpenAI's ChatGPT Fetch Bot Turns Robots.txt Into a Citation Access Problem](https://paralax.ai/blog/openai-chatgpt-fetch-bot-robots-txt-citation-access)

OpenAI separates ChatGPT Search, training crawls, and user-triggered fetches into different access rules.

- Primary source: [developers.openai.com](https://developers.openai.com/api/docs/bots)
- Markdown: https://paralax.ai/blog/openai-chatgpt-fetch-bot-robots-txt-citation-access.md

### 2026-08-12 — [Meta's Web Crawler Turns AI Search Into A Source Architecture Problem](https://paralax.ai/blog/meta-web-crawler-ai-search-source-architecture)

Meta crawler docs make AI search visibility a crawl, source, and citation architecture problem.

- Markdown: https://paralax.ai/blog/meta-web-crawler-ai-search-source-architecture.md

### 2026-07-20 — [Does llms.txt Improve AI Search Visibility? 137,000 Domains Say No](https://paralax.ai/blog/llms-txt-137000-domains-zero-ai-requests-2026)

New data across 137,000 domains shows 97% of llms.txt files never get read. Here's what actually moves AI citations.

- Markdown: https://paralax.ai/blog/llms-txt-137000-domains-zero-ai-requests-2026.md

### 2026-07-08 — [Cloudflare and OpenAI's AI Search Pilot Turns Freshness Into Crawl Permission](https://paralax.ai/blog/cloudflare-openai-ai-search-indexing-pilot-2026)

Cloudflare and OpenAI are testing whether AI search can crawl by freshness signals instead of blind recrawling.

- Primary source: [cloudflare.com](https://www.cloudflare.com/press/press-releases/2026/cloudflare-announces-research-pilot-with-openai/)
- Markdown: https://paralax.ai/blog/cloudflare-openai-ai-search-indexing-pilot-2026.md

### 2026-07-03 — [Web Bot Auth Turns AI Crawlers Into Accountable Search Actors](https://paralax.ai/blog/web-bot-auth-ai-crawler-authentication-search-2026)

Web Bot Auth moves AI crawler access from user-agent claims to signed identity.

- Primary source: [developers.google.com](https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth)
- Markdown: https://paralax.ai/blog/web-bot-auth-ai-crawler-authentication-search-2026.md

### 2026-07-02 — [Cloudflare Just Made Publisher Signals Part of AI Search Infrastructure](https://paralax.ai/blog/cloudflare-ai-search-smarter-publisher-signals-2026)

Cloudflare's AI search push turns freshness, permissions, and usage data into ranking infrastructure.

- Primary source: [blog.cloudflare.com](https://blog.cloudflare.com/making-ai-search-smarter/)
- Markdown: https://paralax.ai/blog/cloudflare-ai-search-smarter-publisher-signals-2026.md