robots.txt for AI crawlers: which bots to allow or block
GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended… what each one does and ready-to-use robots.txt examples.
llms.txt is a short Markdown file that gives AI assistants a map of your most important content. Here is the format, an example and how to keep it useful.
llms.txt is a community proposal for a simple file, placed at https://yourdomain.com/llms.txt, that tells AI tools what your site is about and which pages matter most. Think of it as a curated table of contents written for machines in plain Markdown.
Support varies between AI products and the standard is still evolving, so treat llms.txt as a low-cost, low-risk addition rather than a ranking factor. It takes minutes to create, forces you to define your key pages, and is useful for developer tools and agents that look for it.
- [Title](URL): short description.# Acme Analytics
> Acme Analytics is privacy-friendly web analytics for small businesses in the US and EU. No cookies, GDPR-compliant, from $9/month.
## Product
- [Features](https://acme.example/features): what Acme measures and how
- [Pricing](https://acme.example/pricing): plans, limits and free trial
- [Security & GDPR](https://acme.example/security): data storage and compliance
## Guides
- [Getting started](https://acme.example/docs/start): install the script in 5 minutes
- [Goals & funnels](https://acme.example/docs/goals): track conversions
## Optional
- [Blog](https://acme.example/blog)
text/plain or text/markdown with UTF-8.BoostSEO checks whether your llms.txt exists and follows the format, and Pro members can generate a draft from their audit in AI Studio.
GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended… what each one does and ready-to-use robots.txt examples.
A step-by-step playbook to increase the chance that AI assistants mention your brand and link your pages.
AI assistants now answer many questions before anyone clicks a link. GEO is how you make sure your content is the one they read, trust and cite.