llms.txt: what it is and how to write one
llms.txt is a short Markdown file that gives AI assistants a map of your most important content. Here is the format, an example and how to keep it useful.
GEO & AI search
BoostSEO Team
2 min read
robots.txt for AI crawlers: which bots to allow or block
GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended… what each one does and ready-to-use robots.txt examples.
BoostSEO Team
2 min read
On this page
robots.txt is a simple text file at the root of your domain that tells crawlers which paths they may fetch. With AI crawlers multiplying, it is now also where you decide whether AI products can learn from or quote your content.
Know the two families of AI bots
| Company | Training / data collection | Search & user-triggered fetching |
|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot, ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot, Claude-User |
| Perplexity | — | PerplexityBot, Perplexity-User |
| Google-Extended (control token) | Googlebot (Search & AI Overviews) | |
| Microsoft | — | Bingbot (Bing & Copilot) |
| Apple | Applebot-Extended (control token) | Applebot |
| Common Crawl | CCBot (dataset used by many models) | — |
Vendors update their crawler lists from time to time, so check each company's documentation before relying on a name.
Option A — allow everything (maximum visibility)
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlOption B — allow answers, block training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlThis keeps you eligible for citations in ChatGPT search, Claude, Perplexity, Copilot and Google while opting out of model training by those vendors.
Common mistakes
- Leftover staging rules.
Disallow: /underUser-agent: *blocks everything. - Blocking CSS and JavaScript. Search engines need them to render and evaluate your pages.
- Using robots.txt to hide pages from search. A blocked URL can still be indexed from links. Use
noindexfor that, and do not block the page so the tag can be seen. - Relying on robots.txt for security. It is public and voluntary. Protect private areas with authentication.
- Firewall rules that contradict robots.txt. Some CDNs block AI bots by default; check your WAF settings too.
Check yours in one minute
The BoostSEO audit reads your robots.txt, lists which AI crawlers are blocked and generates suggested rules in the GEO tab.
SEO & GEO guides from the team behind BoostSEO and ToolCMB.