GPTBot:
How to Allow or Block It in robots.txt

GPTBot is OpenAI’s crawler for content that may be used to train its generative AI models. Allow it by leaving it out of robots.txt or giving it Allow: /, block it with Disallow: /. That decision covers training only. ChatGPT search relies on a second crawler, OAI-SearchBot, and requests a person triggers inside ChatGPT come from a third, ChatGPT-User, so each one needs its own decision.

Our own robots.txt has 22 User-agent lines and gives GPTBot, OAI-SearchBot and ChatGPT-User an explicit Allow: / (checked 15.10.2026). We open everything because we sell AI visibility tracking. A publisher with paid content can reasonably choose differently. If you want to see where you stand, AskAiRank’s free website scanner checks nine AI crawler tokens, GPTBot among them, against your robots.txt.

Why it matters for AEO. GPTBot is the crawler OpenAI uses to collect content that may be used to train its generative AI foundation models. It is one of three OpenAI user agents, and robots.txt controls each one on its own: GPTBot for training, OAI-SearchBot for ChatGPT's search results, ChatGPT-User for actions a person starts inside ChatGPT. Treat them as one switch and you either leave training open when you meant to opt out, or drop out of ChatGPT search without noticing.

1. Open yourdomain.com/robots.txt and note which of GPTBot, OAI-SearchBot and ChatGPT-User have their own group. A bot with no group of its own follows the `User-agent: *` rules.
2. Decide per bot. Allow OAI-SearchBot if you want to be shown in ChatGPT search. Allow or disallow GPTBot depending on whether you accept your pages being used for model training. OpenAI says each setting is independent of the others.
3. Write one group per user agent with `Allow: /` or `Disallow: /`. To limit a bot to part of the site, add path rules such as `Disallow: /drafts/`. When rules overlap, the longest matching rule wins (RFC 9309).
4. Check your CDN, firewall and security plugins. robots.txt is a request, not access control, so a firewall rule can still answer 403 to a crawler you allowed.
5. Give it a day. OpenAI says it can take about 24 hours after a robots.txt update for its search systems to adjust.

How to verify. Search your access logs for the user-agent strings GPTBot and OAI-SearchBot. Allowed paths should return 200, and disallowed paths should stop being requested after the change.

# Be shown in ChatGPT search
User-agent: OAI-SearchBot
Allow: /

# Opt out of model training only
User-agent: GPTBot
Disallow: /