Does Blocking AI Bots in robots.txt Protect Your Content?

Google says robots.txt controls crawling, not what happens to your content afterwards. Here's what that means for AI bots, Meta's Oct 30 terms and your site.

Did you add a few lines to your robots.txt file to keep AI companies away from your content, then assume the job was done?

A lot of site owners did. It felt like the sensible, grown-up move: block the bots, protect the work.

This week, Google's own search team explained what that file really does. According to Search Engine Journal's report on a new Google Search Central video, robots.txt is about access. It says nothing binding about what happens to content after it has been crawled.

Meanwhile, Meta's new commercial terms start on Oct 30, and ChatGPT's search crawler is finding more pages than before. So here's what changed, who it affects and what to do about it.

Key Takeaways

Google's Gary Illyes said robots.txt is an access preference protocol and doesn't dictate what a system does with data after it has crawled it.
To block a specific AI crawler, you have to name its user agent token explicitly in your robots.txt file.
Google-Extended controls whether crawled content is used for Gemini training and grounding, and Google says it doesn't affect Search inclusion or ranking.
OpenAI uses separate bots for search and training, so blocking one doesn't block the other.
A proposed Content-Usage rule would let sites state how content may be used, but it is still an unfinished draft, not a standard.

What Google Actually Said About robots.txt

The comments came from the second episode of Google's video series The Non-Canonical Cabinet, hosted by Gary Illyes and Cherry Prommawin of Google Search Relations. You can watch it on Google's YouTube channel.

Illyes said that, as defined by RFC 9309, robots.txt is an "access preference protocol." It doesn't dictate what a system does with the data after it has already crawled it.

That's the line that matters. A robots.txt file asks a crawler whether it may fetch a page. It doesn't create a rule about training, summarising or reusing what the crawler already holds.

The episode also showed a made-up directive on screen, "No-AI-Chicken-Training", to make a point: invented lines like that get ignored. Per the SEJ report, Google's parser only supports User-Agent, Allow, Disallow and Sitemap.

Marketing manager checking website files on a laptop in an office — Photo by Rodeo Project Management Software on Unsplash

How to Block an AI Crawler the Right Way

If you want to stop a specific AI crawler, the instruction from the video is simple: target its user agent token explicitly. A general "disallow everything AI" line doesn't exist.

The three you're most likely to meet are worth knowing by name.

Google-Extended

Per Google's crawler documentation, Google-Extended is a standalone product token that controls whether crawled content can be used for AI training and grounding in future Gemini models, Gemini Apps and Vertex AI.

Google also says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." So blocking it won't cost you rankings.

OpenAI's Bots

OpenAI's documentation splits its bots by job. OAI-SearchBot surfaces sites in ChatGPT's search features. GPTBot crawls content that may be used to train its foundation models.

Block OAI-SearchBot and you won't appear in ChatGPT search answers, although you can still show up as a navigational link. Block GPTBot and you signal that your content shouldn't be used for training. They are two separate choices.

There's a third, ChatGPT-User, which visits pages when a person asks ChatGPT a question. OpenAI notes robots.txt rules may not apply to those user-initiated visits.

Small business owner reading about website settings on a laptop — Photo by Ninthgrid on Unsplash

Why Meta's Oct 30 Terms Make This Urgent

Meta's new Commercial Terms take effect on Oct 30. They apply to any business use of Meta's products, which means they also cover companies that manage a Facebook Page without buying ads, according to SEJ.

The first draft said Meta may use any content it collected to "provide, develop, improve, research, train, and maintain the integrity of" its products. After criticism, Meta removed the word "train". A spokesperson said the change "does not change how Meta uses web crawlers."

Legal experts quoted by SEJ aren't convinced. Matt Topic of Loevy & Loevy said, "I think 'develop' is broad enough to include 'train.'" The new language also doesn't mention robots.txt.

Meta's crawler documentation lists separate crawlers for AI training and for Meta AI search, so you may be able to block one and allow the other. Whether robots.txt limits what you grant by accepting the terms is, per SEJ, unclear.

ChatGPT Search Is Finding More of Your Pages

There's a flip side. Blocking everything also cuts you out of AI search, and that channel is widening.

In an Oct 15 post, Cloudflare and OpenAI reported that a pilot, covering about 1% of Cloudflare's network, helped OpenAI discover and index hundreds of millions of new web pages, as SEJ reported.

The post says a signal about a page "doesn't grant permission to access it." It also doesn't promise citations: OpenAI ranks indexed pages independently, and no referral figures were published.

A Quick Example: Two Different Goals

Take a local service business with a blog full of how-to guides. It has two different goals, and they need two different settings.

Goal: be found in ChatGPT answers. Allow OAI-SearchBot, and keep pages fast, clear and easy to quote.
Goal: keep content out of model training. Disallow GPTBot and Google-Extended by name, and check Meta's crawler list.
Goal: stay visible in Google Search. Leave Googlebot alone. Google-Extended doesn't touch your rankings.

One file, three goals, three separate lines. That's the point of the video: robots.txt works per bot, not per intention.

Where This Breaks Down

Not every AI product behaves like a polite search engine. Some tools fetch pages on demand when a user asks a question, and OpenAI says robots.txt may not apply to those visits. If a bot ignores your file, the file can't stop it.

Robots.txt only works on crawlers that choose to follow it. As Illyes put it, it records a preference. Following it is voluntary.

There's also no standard yet for saying "crawl me, but don't train on me." A draft co-authored by Illyes and Martin Thomson of Mozilla proposes a Content-Usage rule, for example "train-ai=n", on paths you allow crawling. It's a working group document, and the training label comes from a separate vocabulary draft that states it reflects no working group consensus.

You can read the IETF draft, but don't build a policy on it today. If your content is truly sensitive, put it behind a login rather than relying on a preference file.

What to Do This Week

Open yourdomain.com/robots.txt and read what's actually in it.
Delete invented lines that no crawler recognises.
Decide separately on search visibility and training, then add explicit user agent tokens for each bot.
Before Oct 30, ask whoever manages your Facebook Page to read Meta's updated Commercial Terms.
Re-check in a few weeks. OpenAI says search changes can take about 24 hours to apply.

If you'd like help deciding which bots to allow, our GEO optimisation and technical SEO work covers exactly this kind of setup.

FAQs

Does robots.txt stop AI companies using my content?

Not by itself. Google says robots.txt is an access preference protocol: it governs whether a crawler fetches your pages, not what happens to data afterwards. Naming specific AI user agents can stop those bots from crawling, but compliance is voluntary.

Will blocking Google-Extended hurt my Google rankings?

No. Google's documentation says Google-Extended doesn't affect inclusion in Google Search and isn't a ranking signal. It only controls whether crawled content is used for Gemini training and grounding.

Should I block OAI-SearchBot?

Only if you don't want to appear in ChatGPT search results. OpenAI says blocked sites won't show in ChatGPT search answers, though they may appear as navigational links. Training is controlled separately through GPTBot.

What is the Content-Usage rule?

It's a draft IETF proposal that would let sites state how crawled content may be used, such as "train-ai=n". It is not an adopted standard yet, so crawlers aren't required to honour it.

Conclusion

Remember one idea: robots.txt controls who may fetch your pages, not what happens next. Treat it as a per-bot request, not a shield.

Your first move this week is to read your own file and make separate choices for search and training. If you want a second pair of eyes, get a free quote and we'll take a look.

Sources

Add Binatec as a preferred source on Google

About the author

David Banjo

Founder & CEO, Binatec Agency

David Banjo is an entrepreneur and web developer, and the Founder & CEO of Binatec Agency, where he helps businesses scale through custom software, digital products, and e-commerce growth strategies. He is also the creator of ProposelyAI, an AI-powered platform designed to streamline and automate proposal generation for freelancers and agencies. Recognized for his work in web development, AI workflow automation, and growth marketing, David equips brands with the modern tools needed to optimize performance and expand their digital reach.

Does Blocking AI Bots in robots.txt Protect Your Content?

Key Takeaways

What Google Actually Said About robots.txt