Is Your Site Blocking ChatGPT Search? How to Check and Fix It

Cloudflare and OpenAI say a pilot added hundreds of millions of pages to ChatGPT search. Here is how to check your robots.txt and CDN so yours can be found.

On October 15, Cloudflare and OpenAI shared early results from a research pilot that helped OpenAI discover and index "hundreds of millions" of new web pages for ChatGPT search. The pilot covered a cohort of about 1% of Cloudflare's network, and it was announced back on July 8.

That is a lot of new pages. It also raises a simple question for any business that wants to be found in ChatGPT: can OpenAI's search crawler actually reach your site? Plenty of sites block it by accident, through a copied robots.txt file, a security plugin or a CDN setting nobody remembers changing. This guide walks through how to check, and how to fix it if needed.

What the pilot actually showed

According to Search Engine Journal's report, OpenAI uses signals from Cloudflare's network to find pages worth indexing. The July announcement named three: how recently a page was updated, the quality of traffic to a website, and what changed on a page.

The examples in the Cloudflare post are deliberately ordinary: a washing machine's user guide, current passport requirements and fees, a restaurant's menu and hours. The point is that useful, specific pages can be valuable even when few other sites link to them. If you run a local service business, a shop or a B2B company with detailed product and pricing pages, that describes you.

Two limits matter. First, the post states that a signal about a page does not grant permission to access it, so your own access rules still decide what gets crawled. Second, OpenAI ranks the pages it indexes on its own, so being discovered does not guarantee a citation. The posts also did not report how often the new pages were cited or how many visits they sent.

Step 1: Know the three OpenAI bots

OpenAI's crawler documentation lists separate bots for separate jobs. Each setting is independent.

• OAI-SearchBot surfaces sites in ChatGPT's search features. If you block it, your site won't appear in ChatGPT search answers, though it can still show up as a navigational link.
• GPTBot crawls content that may be used to train OpenAI's models. Blocking it signals that your content should not be used for training.
• ChatGPT-User handles actions a person triggers, such as visiting a page when they ask a question. It doesn't crawl automatically, and because users trigger these visits, robots.txt rules may not apply.

OpenAI says that if you allow both OAI-SearchBot and GPTBot, it may use a single crawl for both purposes. If you want search visibility without training use, the documentation gives allowing OAI-SearchBot while blocking GPTBot as an example setup.

Step 2: Read your robots.txt

1. Open yourdomain.com/robots.txt in a browser.
2. Search the file for OAI-SearchBot and GPTBot. If neither appears, check the rules under `User-agent: *`, because a blanket `Disallow: /` blocks everything.
3. Look for a Disallow rule on folders that hold your best pages, such as /blog/, /services/ or /pricing/.
4. If you want ChatGPT search visibility, make sure OAI-SearchBot is allowed. Ask your developer to add an explicit rule if it is unclear.
5. Note the date you changed it. OpenAI says search changes can take about 24 hours to take effect after a robots.txt update.

This is a decision about your business, not only a technical tweak. Some companies block training and still want to be cited in answers. Others block both. Decide first, then write the rules to match.

Step 3: Check your CDN and security layer

A clean robots.txt is not enough if a firewall or bot-protection rule turns the crawler away before it reads the file. OpenAI recommends allowing requests from its published IP ranges, and it lists them for each bot (for example openai.com/searchbot.json for OAI-SearchBot).

If your site sits behind Cloudflare, AI Crawl Control is available on every plan. It shows which AI services access your site and lets you allow or block each crawler. Open it and confirm the settings match the decision you made in step 2.

Step 4: Make your best pages worth finding

Access is the entry ticket. Content is what earns the citation. The pilot's examples point to pages that answer a specific, current question, so audit yours with that in mind.

• Keep prices, opening hours, service areas and policies current, and update the page when they change.
• Give every important service its own page rather than burying it in a long catch-all page.
• Remove or redirect dead URLs. The Cloudflare post says its signals aim to cut wasted requests such as repeated attempts at missing files and near-any-address redirects to the homepage.
• Avoid duplicate pages that say the same thing at different URLs.

This is the same groundwork as good technical SEO, and it feeds GEO and AI search optimisation as well.

Step 5: Set up tracking before you expect results

Search Engine Journal notes that tracking ChatGPT visits relies on a UTM tag or a referrer, and that some ChatGPT links lacked both earlier in October. One analyst said a fix began rolling out on October 15. So some past visits may be sitting in your direct traffic.

Set up a GA4 report or segment for referrals from chatgpt.com and any utm_source=chatgpt.com traffic, and check it weekly. Our web analytics setup team can help if your reporting needs tidying first.

Common mistakes that keep sites invisible

Most blocking is accidental. These are the patterns worth ruling out before you spend time on anything else.

• A staging-site robots.txt with a blanket Disallow that was copied to the live site at launch.
• A security or caching plugin that blocks unfamiliar user agents by default.
• Blocking GPTBot and assuming that also covers search. It doesn't, because OAI-SearchBot is a separate rule.
• Blocking the bot in robots.txt but forgetting a firewall rule that still returns errors to it, which makes logs hard to read.
• Changing the file and checking results the same day. Allow about 24 hours before you judge the change.

If you are not sure who controls each layer (your developer, your host or your CDN), write it down. Crawler problems usually survive because everyone assumes someone else owns the setting.

What to do next

1. Decide whether you want to be in ChatGPT search, and whether you allow training use.
2. Check robots.txt for OAI-SearchBot and GPTBot rules.
3. Check your firewall, CDN or AI Crawl Control settings match.
4. Update and consolidate the pages you most want cited.
5. Add a ChatGPT referral report in GA4 and review it weekly.

Expect crawling to change: the posts say that finding new pages can initially increase crawling, with the aim of reducing wasted crawling over time. Keep an eye on your server load after you open things up.

If you would like a second pair of eyes on your crawler access and AI search visibility, you can get a free quote and we will take a look.

Sources

• Cloudflare: Cloudflare and OpenAI discovery pilot
• Search Engine Journal: ChatGPT Search Can Draw On More Pages After A Cloudflare Pilot
• OpenAI: Crawler documentation
• Cloudflare: AI Crawl Control

About the author

David Banjo

Founder & CEO, Binatec Agency

David Banjo is an entrepreneur and web developer, and the Founder & CEO of Binatec Agency, where he helps businesses scale through custom software, digital products, and e-commerce growth strategies. He is also the creator of ProposelyAI, an AI-powered platform designed to streamline and automate proposal generation for freelancers and agencies. Recognized for his work in web development, AI workflow automation, and growth marketing, David equips brands with the modern tools needed to optimize performance and expand their digital reach.

Follow