They sound similar and do completely different jobs. robots.txt is permission; llms.txt is comprehension. You need both — here's the difference in one table.
robots.txt answers "may I?" — it grants or denies crawlers access. llms.txt answers "what is this?" — it hands a model a curated map of your site so it doesn't have to guess. One gates the pipeline; the other improves what flows through it.
| robots.txt | llms.txt | |
|---|---|---|
| Purpose | Crawl permission per bot | Content overview for LLMs |
| Format | Plain-text directives (User-agent / Allow / Disallow) | Markdown: H1, summary, sections of links |
| Location | https://site.com/robots.txt | https://site.com/llms.txt |
| Who obeys it | All reputable crawlers (enforced by convention) | AI assistants, optionally, when reading your site |
| Without it | Bots crawl by default rules | Models guess your structure from crawling alone |
| Age | 1994, universal | 2024 proposal, early adoption |
Exact allow-blocks: how to allow AI crawlers.
Sequence matters: robots.txt decides whether an AI crawler can fetch anything at all; the fetched pages' schema and structure decide what it understands; llms.txt gives the model your own curated summary and priority map. A site with perfect llms.txt and a blocked PerplexityBot is invisible to Perplexity; a site with open crawling and no structure forces engines to reverse-engineer it. Do all three, in that order — the free checker scores each layer so you know which one is costing you points. Related: build the file in the llms.txt generator, see a real one in the annotated example.