llms.txt and robots.txt for AI Crawlers: What to Allow, What to Block

Which AI crawlers should you allow in robots.txt? Allow the ones that fetch pages so an AI search product can cite you (OAI-SearchBot, Claude-SearchBot, PerplexityBot), decide separately about the ones that collect training data (GPTBot, ClaudeBot, Google-Extended), and accept that the ones that fetch on a user's behalf (ChatGPT-User, Perplexity-User) may not obey robots.txt at all. llms.txt is an optional proposal that no vendor I checked says it reads for answers.

I read each vendor's own page on 2026-10-08. Where a page was silent, this post says so rather than filling in from memory. Vendors rename and add bots often, so re-check the linked pages before you ship a file.

What does each vendor say about its crawlers?

Sources: OpenAI's bots page (developers.openai.com/api/docs/bots, no date shown), Anthropic's Privacy Center article on its crawlers (dated 7 April 2026), Google's common crawlers page (no date seen) and its AI features page (updated 10 December 2025), and Perplexity's bots guide (no date shown). OpenAI says robots.txt changes can take about 24 hours to show up in search; Perplexity says the same window can apply.

Notes worth knowing:

• Block training without blocking search. Both OpenAI and Perplexity document separate agents for training versus search, so the split is real on their side. Anthropic documents three bots with different purposes, and says blocking Claude-SearchBot may reduce visibility and accuracy in search results.
• Blocking is a request, not a lock. Anthropic states that IP-based blocking may not persistently guarantee an opt-out, and publishes its IP ranges at claude.com/crawling/bots.json. OpenAI and Perplexity publish IP lists too. robots.txt only works on bots that choose to obey it.
• Anthropic's article only shows robots.txt examples for ClaudeBot. It names Claude-User and Claude-SearchBot as separate bots, and says disabling them reduces visibility, but it gives no sample lines for them. The sample file below uses their names as listed; confirm the tokens against the article.
• I did not verify other crawlers. Common Crawl, Apple, Meta, Amazon, ByteDance and others also fetch pages. They are not covered here, and I cannot tell you their rules from this research.

Which bots should I allow? A decision table

Decide by what you want, not by fear of "AI".

Three more rules of thumb:

1. Do not block Googlebot to avoid AI features. Google says the pages in AI Overviews and AI Mode are ones that are indexed and eligible for a Search snippet, with no extra requirements. Blocking Googlebot removes you from Search itself. To limit what Search shows, Google documents nosnippet, data-nosnippet, max-snippet and noindex.
2. Google-Extended is not a switch for AI Overviews. Google's page on it says it does not affect Search inclusion or ranking, and its AI features page does not list it as an AI Overviews control. If you only wanted to stop AI Overviews, this token is the wrong tool based on what Google has written.
3. Block by purpose, not by vendor. A blanket ban on every AI user agent also blocks the bots that could cite you.

A sample robots.txt I syntax-checked

This file allows the three search bots (while keeping private paths blocked), blocks the three training controls, and leaves everyone else on the default rules.

# Search and citation bots: allowed, but repeat the private paths
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /admin/
Disallow: /api/

# Training bots: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

# Everyone else, including Googlebot and the user-triggered fetchers
User-agent: *
Disallow: /admin/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

Why the private paths are repeated: a crawler follows the most specific group that names it and ignores the * group. My first draft used Allow: / for the search bots, and the check below showed they could then fetch /admin/x, because the * rules no longer applied to them. That is the common mistake with this layout.

I tested it with Python's standard urllib.robotparser (Python 3, run on 2026-10-08):

OAI-SearchBot      /blog/post: True   /admin/x: False
Claude-SearchBot   /blog/post: True   /admin/x: False
PerplexityBot      /blog/post: True   /admin/x: False
GPTBot             /blog/post: False  /admin/x: False
ClaudeBot          /blog/post: False  /admin/x: False
Google-Extended    /blog/post: False  /admin/x: False
Googlebot          /blog/post: True   /admin/x: False
ChatGPT-User       /blog/post: True   /admin/x: False
Claude-User        /blog/post: True   /admin/x: False
Perplexity-User    /blog/post: True   /admin/x: False

That is the intent. Caveats: Python's parser is not Google's or any vendor's parser, so edge cases can differ, and it cannot tell you how a given bot will behave. It only confirms the file parses the way the grouping rules say. Also, Googlebot is only blocked from /admin/ and /api/ here because of the * group. Replace example.com and the paths with your own.

Two things this file does not do:

• It does not stop ChatGPT-User or Perplexity-User if they choose to ignore it (see the table above).
• It does not block OAI-AdsBot. OpenAI's page does not say whether robots.txt applies to it, and it only visits pages submitted as ads.

Is my site actually reachable by these bots?

robots.txt is only half the story. A CDN or firewall rule can return 403 to a bot the file allows. Perplexity's guide explicitly recommends allow rules in Cloudflare or AWS WAF that combine the user agent with its published IP ranges.

Checks, in order:

1. Fetch https://yourdomain/robots.txt in a browser. Confirm it returns 200 and the text you expect.
2. Test with a spoofed user agent. This is an untested template; the output depends on your host:

curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot" https://yourdomain/

A 403 or a challenge page suggests a firewall rule. A spoofed user agent comes from your IP, so a bot-management rule that also checks the source IP can still behave differently for the real bot. 3. Check your access logs for real visits. Match the user agent and then confirm the IP against the vendor's published list (OpenAI: openai.com/searchbot.json; Perplexity: perplexity.com/perplexitybot.json; Anthropic: claude.com/crawling/bots.json). 4. Re-check after 24 hours, since the vendors say changes need time. 5. Then run the AI citation test protocol to see whether any of this changes what people see.

For the content edits that make a fetched page worth citing, see the GEO guide.

What is llms.txt, and should I publish one?

llms.txt is a proposal from Jeremy Howard, published at llmstxt.org on 3 September 2024. The page I read on 2026-10-08 is marked as v2, modified 10 August 2026. The idea: put a Markdown file at /llms.txt that gives AI agents a curated map of your site, so they do not have to wade through navigation and scripts.

The format, per the proposal:

• An H1 with the site or project name. This is the only required part.
• A blockquote with a short summary.
• Optional free-text sections.
• H2 sections containing lists of links in the form [name](url), optionally followed by a colon and notes.
• An "Optional" H2 by convention for links an agent can skip.
• Optionally, clean Markdown copies of pages at the same URL with .md added.

A minimal valid example, checked by a short script I wrote against those rules (H1 present, blockquote present, two H2 sections, three well-formed link items):

# Example Co

> Example Co sells invoicing software for freelancers. This file lists the pages that answer most support and pricing questions.

Pricing is per seat and billed monthly. Free plan limits are on the pricing page.

## Docs

- [Getting started](https://example.com/docs/start.md): create an account and send a first invoice
- [Pricing](https://example.com/pricing.md): plans, limits and refund terms

## Optional

- [Changelog](https://example.com/changelog.md): release notes, newest first

That check only confirms the structure matches the written spec. It says nothing about whether any engine uses the file.

What evidence is there that engines use it?

Honestly, thin. What I could confirm:

• llmstxt.org says thousands of sites publish the file, that documentation platforms generate it, that Lighthouse audits for it, and that OpenAI, Anthropic and Google's Gemini publish llms.txt files for their developer docs. It does not claim any AI product reads the file when answering.
• Google's page on AI features states there are no additional requirements or special optimizations for AI Overviews and AI Mode, and that you do not need new machine-readable files or AI text files.
• I found no OpenAI, Anthropic or Perplexity page saying their search or answer products read llms.txt. Absence on pages I read is not proof of absence in the product.

What I could not verify: whether any retrieval system uses llms.txt behind the scenes, and whether having one changes citation rates. If you read a claim that it does, ask for the test. You can run your own with the citation protocol linked above, but a single site's result is weak evidence.

My recommendation

• Developer docs or API references: worth an hour. Agents and coding tools that fetch docs may benefit from a clean index, and it costs little. That is a plausible use, not a proven one.
• A marketing or blog site chasing citations: skip it until you have fixed access, content quality and sources. Robots rules and good pages have documented effects; llms.txt does not.
• If you publish one: keep it short, keep links current, and make sure every URL in it returns 200. A stale index is worse than none.

What to do now

1. Look at your logs for the bots in the table. Know who visits before you block anyone.
2. Decide: cited-but-not-trained, or fully open, or fully closed. Use the decision table.
3. Copy the sample file, edit the paths, run the parser check against your real file.
4. Confirm your CDN or firewall is not blocking the bots you allowed.
5. Re-check the vendor pages every few months. The bot lists change.

Access is the first gate. After it, the usual SEO work still applies, and the AI SEO content strategy prompts post covers the briefs and drafting side.

Related reading

• nosnippet, max-snippet and data-nosnippet explained
• IndexNow setup with a Python script
• AEO checklist: what Google and OpenAI actually say