Most AI answer engines run a real crawler with its own user agent string, separate from Googlebot. robots.txt is a plain text file at your domain root that tells each of those crawlers whether they may fetch your pages - block it, even by accident, and that engine has nothing of yours to cite. This is the first item to check because every other guide in this library assumes AI crawlers can already reach your site.
Why it matters for AEO. ChatGPT (GPTBot), Claude (ClaudeBot), Perplexity (PerplexityBot), and Google's AI Overviews (Google-Extended) each send a named crawler. If robots.txt blocks or omits one, your content is invisible to that engine no matter how good it is.
How to do it
- Open yourdomain.com/robots.txt in a browser.
- Confirm there is no "Disallow: /" rule under User-agent: * or under GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, or Applebot-Extended specifically.
- If your CDN or CMS (Cloudflare, Vercel, WordPress security plugins) has an "AI scrapers" or bot-blocking toggle, explicitly allow these crawlers even if you block generic scrapers.
- Re-check after any CDN or WAF change - security tools often add new bot rules silently during upgrades.
How to verify. Run curl -A "GPTBot" yourdomain.com/robots.txt and confirm no Disallow rule matches your page paths.
Example
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /