OpenAI's Crawl Bot Jumped 838% While Its Answer Bot Fell 49%: Don't Block OAI-SearchBot Over It

If OAI-SearchBot's requests to your site jumped sharply this month, the move is not to block it: OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, so blocking the crawl bot trades away eligibility, not a cost. The metric that actually tells you whether AI visibility is slipping is ChatGPT-User, the separate bot that fetches a page live when a real question needs it. Between the week of August 24 and the week of September 21, 2026, the two moved in opposite directions on the same corpus: ChatGPT-User fell from 997 requests a day to 507, a 49% decline across four consecutive weekly drops, while OAI-SearchBot rose from 201 a day to 1,886, up 838%.

Both numbers are true and they mean different things. Most AI-visibility dashboards add them together, which is how a crawl spike gets mistaken for good news while the answer-time decline — the one that tracks with citation — goes unnoticed.

There is a second finding underneath the first, and it changes what the crawl number is worth: 86.9% of that OAI-SearchBot rise landed on a single domain, alongside simultaneous surges from Applebot and PerplexityBot on the same domain in the same week. Three unrelated vendors accelerating on one property and not on the other four is the signature of a site-level recrawl, not a change in how OpenAI crawls the web. The answer-time decline is the part that generalised across properties. The crawl spike is the part that did not.

The two OpenAI bots have different jobs

OpenAI documents four relevant user agents and says plainly that they are not interchangeable. OAI-SearchBot crawls for search; sites that opt out of it "will not be shown in ChatGPT search answers." GPTBot crawls content that may be used to train foundation models. ChatGPT-User fires on a user-triggered fetch — a person asked something and the model went to get the page (OpenAI developer documentation).

That split is why a single "AI bot traffic" line is not a metric. OAI-SearchBot volume tells you about eligibility: whether your pages are being catalogued for retrieval later. ChatGPT-User volume tells you about use: whether, at the moment a real question was asked, your page was worth going to get. They can move together. Over the last five weeks, on our logs, they did not.

The five-week record

Requests per collected day across five content domains under one Cloudflare account, weeks beginning Monday. The week of August 31 returned five of seven days of logs, which is why every figure is a rate per collected day and not a weekly total.

Week beginning	ChatGPT-User (answer-time)	OAI-SearchBot (index)	PerplexityBot (index)	Applebot
2026-08-24	997	201	171	388
2026-08-31	863	200	181	343
2026-09-07	788	215	244	359
2026-09-14	568	306	463	398
2026-09-21	507	1,886	613	1,512

Read the answer-time column first. It falls in every one of the four steps: 997, 863, 788, 568, 507. There is no week in the window where it recovered. Four consecutive declines on a weekly rate is a trend, not noise.

Read the index columns second. OAI-SearchBot sits between 200 and 215 for three weeks, rises 42% in the fourth, then jumps more than sixfold in the fifth. A single-week step is a crawl wave. It has not yet earned the word trend.

The answer-time decline held across properties, the crawl surge did not

Broken out per domain, first week against last, requests per day:

Property	ChatGPT-User	OAI-SearchBot
Practitioner site, 1,361 URLs	739 → 284 (−62%)	114 → 158 (+39%)
machinerelations.ai, 296 URLs	197 → 179 (−9%)	57 → 1,639 (+2,775%)
Founder blog, 51 URLs	32 → 12 (−63%)	11 → 15 (+36%)
Marketing blog, 110 URLs	23 → 23 (0%)	15 → 50 (+233%)
Product-research blog, 112 URLs	6 → 9	4 → 25

The answer-time fall is concentrated but real in more than one place: −62% on the largest property and −63% on the smallest text property, −9% and flat on two others. The crawl rise is not distributed at all. Of 1,886 OAI-SearchBot requests a day in the final week, 1,639 hit one domain. On the largest property in the network, the one with by far the most pages and the most search demand, OAI-SearchBot rose 39% — real, and more than twenty times smaller than the headline figure.

Anyone reporting "OpenAI is crawling us 838% harder" from the aggregate would be describing something that happened to one site.

Three vendors moved together on that one domain

The strongest evidence that the surge is a site event rather than an engine event is that it was not an OpenAI surge. On the same domain, in the same week, Applebot went from 39 requests a day to 1,101 and PerplexityBot from 39 to 388. Apple, OpenAI and Perplexity do not coordinate crawl schedules. What they share is a set of signals a site emits — sitemap changes, lastmod dates, new URLs, an internal link graph that shifted — and that domain published two dated index releases inside the window.

The general rule this supports: when several unrelated crawlers accelerate on one property at once, look at the property. When one vendor's crawler moves across many properties at once, look at the vendor. Neither pattern is visible in a network-wide total, which is the only view most dashboards offer.

Perplexity and Apple split their bots the same way

This is not an OpenAI quirk to be handled once and forgotten. Perplexity documents PerplexityBot as the crawler that indexes content for search results and Perplexity-User as the agent that visits a page because a user's request required it, and states that the user-triggered fetch is not used for training (Perplexity documentation). Apple documents Applebot for its search and Siri surfaces and Applebot-Extended as a separate control over training use (Apple).

The industry has converged on separating index-building from answer-time fetching and from training, and on exposing each as its own user agent with its own robots.txt control. A measurement practice that collapses them back into one row is throwing away the distinction the vendors went to the trouble of making.

What the aggregate hides

Take the network total for the first and last week of the window. Summed across all four user agents, activity went from 1,757 requests a day to 4,518 — up 157%. Inside that total, answer-time fetching fell by half. A single "AI bot traffic is up" line would have been literally accurate and would have pointed a search team in exactly the wrong direction — toward a celebration, when the signal that correlates with being used in an answer had halved.

The inverse error is equally available. On the domain that absorbed the crawl wave, a team reading only ChatGPT-User would have seen a 9% dip and concluded nothing was happening, in the week its pages were being recatalogued by three engines.

Tooling does not rescue you here. Cloudflare's AI Crawl Control exposes per-operator, per-user-agent request data and per-agent allow and block rules (Cloudflare), which is enough to build the split. The default view in most reporting is still one number.

What to change in the report

Four changes, in order of how much they fix:

1. Split every AI bot line by role, not by vendor. Three buckets: answer-time fetch (ChatGPT-User, Perplexity-User, Claude-User), index crawl (OAI-SearchBot, PerplexityBot, Applebot), training crawl (GPTBot, Applebot-Extended, ClaudeBot). A vendor-only split still mixes eligibility with use.
2. Report per collected day, not per week or per month. Log collection has gaps. A week with five days of data compared against a week with seven manufactures a 29% decline that never happened.
3. Never quote a network total for the crawl bucket without the per-domain split. Crawl volume concentrates. One recrawled property will carry an aggregate on its own, and it did here.
4. Treat a single-week step as a wave until a second week confirms it. The answer-time series earned its reading with four consecutive moves. The crawl series has one.

None of this tells you whether you were cited. Fetches are retrieval; citations are a separate measurement on the answer itself, which is what the Machine Relations Index observes against a fixed question set across six engines. Crawl logs tell you whether the machine came to the door. They never tell you what it said afterwards.

What would falsify this read

Stated plainly, because a five-week window on one network is a small instrument:

• If the answer-time decline is spoof attrition. Requests carrying a bot's name without a verifiable identity are common; Cloudflare maintains a verified-bot list and flags requests whose claimed identity it can confirm (Cloudflare). In the final week, where that flag was collected for all seven days, verified ChatGPT-User requests ran at 481 a day against 507 total — 94.9%. That leaves little room for the decline to be fake traffic leaving, but the earlier weeks in this window were counted by user agent alone and cannot rule it out directly.
• If ChatGPT simply answered from its index instead of fetching. This is the reading the two series most naturally suggest, and it is a hypothesis, not a finding. It would be supported if answer-time fetches stay low while citation rates hold, and refuted if citations fall with the fetches. That test runs on citation data, not logs.
• If the corpus changed. A network that publishes daily changes its own crawl surface. Publication rate held roughly steady across the window on the four properties that showed the answer-time decline, but a larger effect on one of them would confound it.

The finding we will stand behind today is narrow and survives all three: the two OpenAI user agents are separate signals, they diverged, and a network-wide crawl total attributed the divergence to the wrong cause.

FAQ

Is a drop in ChatGPT-User requests bad? It is a drop in answer-time use of your pages, which is closer to being cited than any crawl metric is. Whether it is bad depends on whether citations moved with it. Measure both.

Should I block OAI-SearchBot to save crawl budget? No. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Blocking the index crawler removes your eligibility to appear.

Why normalise per collected day rather than per week? Because log collection is not guaranteed. One of the five weeks here returned five days. Comparing raw weekly totals across it would have produced a 29% swing that is an artefact of collection, not behaviour.

Can I reproduce this on my own site? Yes, with any log source that records user agent and verified bot identity. Split by role, normalise per collected day, and run at least five weeks before reading a direction.

Method

Source: Cloudflare request logs for five content domains operated by the same team, 2026-08-24 to 2026-09-27, 33 of 35 days collected. Counts are HTTP 200 responses attributed by user agent; from 2026-09-19 they also carry Cloudflare's verified-bot identity flag. Rates are requests per collected day. Weeks begin Monday. Properties are described by role rather than named, except machinerelations.ai, whose publishing schedule is material to the crawl-wave reading; the network's ownership is described on machinerelations.ai/about.