Technical GEO

AI Crawlers and robots.txt: Search Visibility Is Not Model Training

By GEO Specialist Perth. Updated . 6 minute read.

Allowing an AI search crawler is not the same as allowing content for model training. Understand the practical difference between OAI-SearchBot, GPTBot and ChatGPT-User before changing robots.txt or security rules.

Take Free AI Readiness Quiz Book My Free Strategy Session

AI crawler access is not one decision. A service business can permit a crawler used to find pages for ChatGPT search while separately declining a crawler whose content may be used to train foundation models. It can also receive a request triggered by a person using ChatGPT, which is a different situation again.

That distinction matters before changing robots.txt, a CDN rule or a web application firewall (WAF). A broad block can reduce discoverability in an AI search experience, while an overly permissive rule can expose content or consume server resources without giving you the control you intended.

Three kinds of AI access

Search crawling

OpenAI identifies OAI-SearchBot as the crawler used to surface websites in ChatGPT search features. OpenAI says that sites opting out of OAI-SearchBot will not be shown in ChatGPT search answers, although they can still appear as navigational links. Its documentation recommends allowing the bot in robots.txt if you want to help your site appear in search results. [1]

This is closest to a search visibility choice. It does not promise that a page will be selected, cited or shown. As with other search systems, crawling, eligibility and inclusion are separate outcomes.

Training access

GPTBot is described by OpenAI as being used to crawl content that may be used to train its generative AI foundation models. Disallowing GPTBot indicates that the site’s content should not be used for that training purpose. [1]

Allowing OAI-SearchBot does not require allowing GPTBot. The settings are independent. That is the central point: search visibility and model training are not automatically bundled into one permission.

A request initiated by a user

ChatGPT-User is used for certain actions initiated by a person in ChatGPT or a Custom GPT. OpenAI says it is not an automatic web crawler, does not determine whether content appears in Search, and that robots.txt rules may not apply to these user-initiated actions. [1]

A rule intended to manage automatic search crawling therefore should not be treated as a complete access-control system for private or sensitive information. Keep private material behind authentication and server-side authorisation. Do not publish confidential details and rely on robots.txt to hide them.

What robots.txt can and cannot do

A robots.txt file tells crawlers which URLs they may access. Google describes its main purpose as managing crawler traffic, not keeping a web page securely out of Google. A disallowed URL can still be discovered through links and appear without a normal page description. For private content, use authentication and server-side access control. For public content that should not appear in Google, use an appropriate indexing control such as noindex, applied where the crawler can see it. A noindex directive does not restrict public access. [2]

The practical lesson is to match the control to the objective:

  1. Manage automatic crawler access: use user-agent-specific robots.txt rules.
  2. Reduce or prevent search indexing: use noindex or authentication as appropriate, remembering that a blocked crawler may not see a noindex directive.
  3. Protect confidential or restricted content: use authentication, authorisation and server controls, not a public text file.
  4. Manage request volume: use sensible rate limits, caching and infrastructure controls, while checking that legitimate search crawlers are not accidentally challenged.

Do not copy a permissive snippet from a forum and assume it is safe. In particular, User-agent: * with Allow: / is not a special AI setting. It simply permits all crawlers that choose to obey the file. Review each directive, the URL scope and the effect on staging, account, search and internal paths.

CDN and WAF checks are part of GEO

A correct robots.txt file can be undermined by infrastructure. Google’s AI features guidance specifically includes allowing crawling through robots.txt and any CDN or hosting infrastructure among sound SEO practice. [3] A Perth business should therefore check both the published file and the request path that reaches the origin.

Ask your developer or hosting provider to check:

  • whether the CDN serves the current /robots.txt rather than a stale cached copy
  • whether a WAF blocks, challenges or rate-limits recognised crawler requests
  • whether bot protection depends on JavaScript or a browser challenge that an automated crawler cannot complete
  • whether geo-blocking, IP allowlists or an origin firewall conflict with the crawler’s published IP ranges
  • whether logs record user agent, response status, challenge events and request volume for investigation

Do not allow a request merely because its user-agent string claims to be a known bot. User-agent strings can be impersonated. Where a provider publishes IP ranges, your technical team can use those details as one verification input, alongside normal security monitoring. Keep rules narrow and test them on a non-sensitive URL before applying them site-wide.

A practical decision for a service business

Hypothetical example: a Perth accounting practice wants its service pages to remain discoverable when someone asks ChatGPT for local accounting help. It could choose to allow OAI-SearchBot while disallowing GPTBot, subject to reviewing OpenAI’s current documentation and its own commercial and legal preferences. It should still keep customer portals, quote records and staff areas behind authentication, and ensure the WAF does not accidentally block legitimate search access.

That decision does not guarantee a mention, a citation, a lead or a ranking. It simply separates one access preference from another. The business should also maintain clear service information, location coverage and contact pathways using ordinary SEO foundations, rather than treating crawler permissions as a substitute for useful content. For a broader technical review, see what generative engine optimisation means and consider GEO web design if the site makes important information difficult to access.

A safer review process

Start by writing down the purpose for each area of the site: search discovery, training preference, user access, indexing control or security. Then inspect the current robots.txt, CDN configuration, WAF events and server logs together. Record changes, test expected responses and schedule a review when provider documentation changes. OpenAI notes that a robots.txt update for its search systems can take about 24 hours to adjust, so do not judge a change immediately. [1]

For Google AI features, there is no special AI file or markup required. Google says its generative search features rely on foundational SEO, and specifically says llms.txt and other special machine-readable files are not used by Google Search. [3] An optional file may be relevant to another service, but it is not a Google visibility lever.

If you need help connecting access rules with a sound site structure, explore our schema and entity mapping service. For a tailored starting point, take the Free AI Readiness Quiz or Book My Free Strategy Session.

Frequently asked questions

No. OpenAI documents OAI-SearchBot and GPTBot as independent choices. OAI-SearchBot is the relevant setting for automatic ChatGPT search crawling; GPTBot relates to content that may be used for training. [1]

Not reliably. Google says a blocked URL may still be indexed if discovered through links. Use authentication or suitable indexing controls when exclusion is important. [2]

Does ChatGPT-User obey my robots.txt file?

OpenAI says ChatGPT-User requests are user initiated and robots.txt rules may not apply. Treat access control and sensitive data protection as separate security responsibilities. [1]

Do I need llms.txt for Google AI visibility?

No. Google says there is no need to create special AI text files or markup for its generative search features, and that Google Search ignores llms.txt. [3]

Sources

  1. Overview of OpenAI Crawlers
  2. Introduction to robots.txt
  3. Optimizing your website for generative AI features on Google Search

Related GEO and SEO guides