Cloudflare and OpenAI Discovery Pilot Moves the Contest From Being Found to Being Chosen
Cloudflare says its OpenAI pilot added hundreds of millions of new pages to ChatGPT search, and our crawl logs show discovery was already fast while selection was not.
On October 15, 2026, Cloudflare published the first results of the research pilot it began with OpenAI in July (Cloudflare blog). Signals from a participating cohort of about 1% of Cloudflare's network helped OpenAI discover and index hundreds of millions of web pages it did not previously know about. OpenAI still evaluates and ranks those pages on its own, and Cloudflare says the work uses metadata about the web for search and discovery, not page content for model training.
The implication for a brand is not the one the headline suggests. For pages on established sites, being found by ChatGPT search was already fast. In our own crawl logs, OpenAI's verified search crawler fetched every one of 85 new pages, most within a day. ChatGPT then read only about half of them while answering a user. The pilot pours more candidates into the pool ChatGPT chooses from. It does not change the step where most brands lose, which is being chosen.
What changed
| Date | Release | What it says |
|---|---|---|
| 2026-07-08 | Research pilot announced (Cloudflare press release) | Cloudflare network signals will be tested as a way to help OpenAI's search discover relevant content |
| 2026-10-15 | Early results (Cloudflare blog) | A cohort of about 1% of Cloudflare's network brought hundreds of millions of previously unknown pages into OpenAI's search index |
| Next phase | Stated plans in the same post | Better page selection, faster and more reliable change signals, and measuring what happens after a page is discovered |
Three details in the results post matter more than the headline number.
- Discovery is filtered before the fetch. Cloudflare says the signals keep crawlers away from addresses that do not exist, that redirect almost everything to a homepage, or that duplicate other pages. A site whose broken and duplicate URLs look like content is now easier to discount before a crawler arrives.
- Change signals decide revisits. The post says change signals tell a crawler when a page is worth another visit and when it is not. A date that moves without a real change is the kind of signal this work is built to see through.
- Single pages can be flagged. Cloudflare is testing signals that flag a potentially harmful page, such as a planted fake login form, without penalizing the whole domain.
Neither announcement describes a way for a site to join the cohort, and neither says whether ranking or citation in ChatGPT changes for pages that were already known. The full record of OpenAI search changes is on the ChatGPT Change Log, and crawler policy is tracked on the crawler log.
What our crawl logs show about finding and choosing
We measured how OpenAI's agents treated new pages on five sites run by the team behind Paralax and the Machine Relations Index. We do not know whether any of these sites is in the pilot cohort; Cloudflare does not publish its membership.
| Measure, 85 new pages published 2026-09-19 to 2026-10-12 | Result |
|---|---|
| Fetched by OAI-SearchBot, OpenAI's search crawler | 85 of 85 |
| Fetched by OAI-SearchBot within one day of publication | 63 of 85 |
| Fetched by ChatGPT-User, the agent ChatGPT sends when it reads a page for a user | 45 of 85 |
| Read by ChatGPT-User within seven days | 43 of 85 |
| Fetched by Bingbot over the same period | 19 of 85, median 17 days after publication |
The gap sits between crawling and reading, and it tracks the site rather than the page. On machinerelations.ai, the site in the set with the longest AI retrieval history, ChatGPT read all 15 new pages, most within a day. On the next largest site it read 26 of 41. Across the three sites with the least retrieval history it read 4 of 29, although OpenAI's crawler had fetched all 29.
That is the pattern the pilot amplifies. A crawler that already reaches a new page within a day cannot make that page more likely to be read. Adding hundreds of millions of long-tail pages to the index gives ChatGPT more candidates for the same answers. When finding gets cheaper, the scarce thing is a reason to be chosen: corroboration from sources the engines already trust, a clear entity, and a page that answers the question better than the alternatives. The Machine Relations framework calls that earned authority, and it is where this change moves the work.
Most failed OpenAI requests are not from OpenAI
Cloudflare's post notes that bots repeatedly ask for files that are not there. Our logs show who is asking. From 2026-10-02 to 2026-10-15, our sites returned 3,722 error responses to requests carrying OpenAI agent names. Of those, 3,581, or 96%, did not come from OpenAI's published IP ranges; most were scanners probing for credential files under a borrowed user agent. Among successful requests carrying OpenAI names, 10,654 of 71,831, about 15%, also failed that check.
A brand that reads raw user-agent strings will overstate both OpenAI's activity on its site and the errors OpenAI hits. Count only requests whose identity is verified before deciding anything about how ChatGPT treats a site.
What to do now
- Do not chase enrollment. No public way to join the pilot has been described, and nothing in either announcement suggests that sites outside the cohort are penalized.
- Check what crawlers can reach. Confirm in Cloudflare's AI Crawl Control or your own logs that OAI-SearchBot and ChatGPT-User receive 200 responses, and remove catch-all redirects and duplicate URL patterns that the pilot's filtering is designed to discount. Blocking OAI-SearchBot costs ChatGPT search eligibility, as our OpenAI crawler brief explains.
- Keep change dates honest. Move a modified date when the substance changes. Change signals are how revisits get earned.
- Measure reads, not crawls. A crawl means the page is a candidate. A ChatGPT-User fetch means ChatGPT went to the page while answering someone. Track the second, by identity-verified agent.
- Spend on being chosen. If discovery is already fast on your site, the constraint is selection, and selection follows the authority of the sources that describe you.
Method note. Cloudflare AI crawl logs for each site's zone, read 2026-10-16 with logs through 2026-10-15. A request counts as verified when Cloudflare matched it to the operator's published IP ranges. A page counts as fetched on its first 200 response from that agent, measured in UTC days from its publication date. The 85 pages are every new page published on the five sites in the window and still live on 2026-10-16. paralax.ai is not included: its traffic passes through Cloudflare only from 2026-09-29, and it published no new page between then and 2026-10-12. One team, five sites and 85 pages describe how OpenAI treated these sites; they are not an estimate for the web.
FAQ
What did the Cloudflare and OpenAI discovery pilot find?
Cloudflare reported on October 15, 2026 that signals from a participating cohort of about 1% of its network helped OpenAI discover and index hundreds of millions of previously unknown web pages for ChatGPT search. OpenAI evaluates and ranks those pages independently.
Can a website join the Cloudflare and OpenAI pilot?
Neither the July announcement nor the October results describe a way for a site to join. Cloudflare's AI Crawl Control lets a site owner see and manage how AI crawlers access the site, which is separate from the pilot.
Does being discovered mean ChatGPT will cite a page?
No. OpenAI decides independently how discovered pages appear. In our logs OpenAI's crawler fetched all 85 new pages, while ChatGPT read 45 of them on behalf of users.
Is Cloudflare sharing website content with OpenAI for training?
Cloudflare says the research uses metadata about the web for search and discovery rather than sharing page content for model training, and that a signal about a page does not grant permission to access it.
How can a site tell whether OpenAI is really crawling it?
Check whether requests named OAI-SearchBot, ChatGPT-User or GPTBot come from OpenAI's published IP ranges. On our sites, 96% of failed requests carrying those names did not.
More on ChatGPT
- ChatGPT Referral Tags Reportedly Went Missing for About a Week After Its October Model Rollout
- OpenAI's Crawl Bot Jumped 838% While Its Answer Bot Fell 49%: Don't Block OAI-SearchBot Over It
- ChatGPT Citation Support Rate Holds at 5x the Sample
- ChatGPT Citation Change Log: It Skips 144 of the 500 Most-Cited Domains
- OpenAI's ChatGPT Fetch Bot Turns Robots.txt Into a Citation Access Problem
More on AI crawlers and robots
- Google, Microsoft and OpenAI Will Honor One AI-Training Opt-Out. Microsoft Also Splits Training From Answers
- OpenAI's Crawl Bot Jumped 838% While Its Answer Bot Fell 49%: Don't Block OAI-SearchBot Over It
- OpenAI's ChatGPT Fetch Bot Turns Robots.txt Into a Citation Access Problem
- Meta's Web Crawler Turns AI Search Into A Source Architecture Problem
- Does llms.txt Improve AI Search Visibility? 137,000 Domains Say No