How to Write robots.txt: Syntax, Examples and Common Mistakes

robots.txtis a plain-text file that tells search-engine crawlers which paths they may not fetch. The syntax fits on a napkin, yet in practice three things go wrong over and over: the file sits in the wrong place, the longest-match and wildcard rules get misread so the wrong URLs end up blocked, and — worst of all — people use it to “remove pages from Google”, which it fundamentally cannot do.

This guide covers the exact grammar (User-agent groups, Disallow/Allow, the * and $ wildcards, the longest-match rule), gives copy-paste examples for common site types, and then walks a real five-line rule set against four URLs line by line so you can see the matching logic in action. If you would rather not hand-write the file, build it with the robots.txt generator, and after deploying, confirm individual URLs are allowed or blocked with the robots.txt checker.

Location: one file per host, always at the root

There is exactly one robots.txt per host (scheme + domain + port), and it must live at the root. Crawlers only ever request https://example.com/robots.txt — a file tucked away at /seo/robots.txt simply does not exist as far as they are concerned.

• https://example.com/robots.txt covers example.com only.
• blog.example.com needs its own file; subdomains never inherit the parent domain’s rules.
• http:// and https:// are technically separate hosts too. If everything redirects to https, maintaining the https file is enough.
• The file must be UTF-8 plain text; Google reads at most 500 KiB and ignores the rest.

The HTTP status matters as well. A 404 means “no restrictions” and everything gets crawled, which is fine. Persistent 5xx errors are not: Google may throttle crawling of the whole site while the file is unreachable. If you do not need the file, serve a clean 404 or an empty 200 — never a server error.

Syntax: User-agent groups, Disallow/Allow, wildcards

Crawlers read robots.txt in groups: one or more User-agent lines open a group, and the Disallow/Allow lines beneath it are that group’s rules. Each crawler obeys only the single most specific group that names it — if a User-agent: Googlebot group exists, Googlebot ignores the User-agent: * group entirely.

User-agent: *
Disallow: /admin/

User-agent: Googlebot
Disallow: /beta/

Given this file, the only rule that applies to Googlebot is Disallow: /beta/. It is free to crawl /admin/ — the groups do not merge. If you want Googlebot kept out of /admin/ too, repeat that line inside the Googlebot group.

• Disallow: blocks any path that starts with the value. An empty value (Disallow:) blocks nothing.
• Allow: punches an exception through a Disallow. Supported by Google and Bing.
• * matches zero or more of any character: Disallow: /*.pdf matches any URL containing .pdf anywhere in the path.
• $ anchors the end of the path: Disallow: /*.pdf$ only matches URLs that end in .pdf (note that /file.pdf?v=2 escapes it, since matching includes the query string).
• Longest match wins: when both an Allow and a Disallow match a URL, the rule with the longer matching path string applies. On a tie, Google picks the less restrictive rule (Allow). The order of lines in the file is irrelevant.

Longest match is the rule people trip on. With Disallow: /admin/ (7 characters) and Allow: /admin/help/ (12 characters) in the same group, /admin/help/faq is crawlable because the longer Allow wins. When the mental arithmetic gets fiddly, paste the rules and a test URL into the robots.txt checker and read the verdict instead of guessing.

Ready-made examples, plus the Sitemap directive

The Sitemap: directive is group-independent — put it anywhere in the file, and declare several if you have several sitemaps. The value must be an absolute URL (a relative Sitemap: /sitemap.xml is invalid). Since robots.txt is the first file every crawler fetches, it is the most reliable place to announce your sitemap. Before pointing crawlers at it, run the file through the sitemap validator to make sure the XML parses and the listed URLs actually resolve.

Worked example: five lines of rules vs four URLs

Suppose this file is live.

User-agent: *
Disallow: /admin/
Allow: /admin/help/
Disallow: /*.pdf$
Disallow: /search

The last row is the classic accident. If you only meant to block the /search page itself, write Disallow: /search$ (exactly that path) or Disallow: /search/ (only the directory beneath it).

The critical misconception: robots.txt does not deindex

robots.txt controls crawling, not indexing.If other sites link to a blocked URL, Google can still index it without ever fetching it — the result is a bare, title-less “No information is available for this page” entry in search results. Blocking a page in robots.txt is therefore the wrong tool for getting it out of Google.

1. To remove a page from search results: add <meta name="robots" content="noindex"> (or an X-Robots-Tag response header) and make sure the page is not blocked in robots.txt — a crawler that cannot fetch the page can never see the noindex.
2. For urgent removals: use the URL removal tool in Search Console alongside noindex.
3. What robots.txt is actually for: saving crawl budget (infinite filter/sort parameters, internal search results) and shielding expensive endpoints. It is not a secrecy mechanism — the file is public, so listing sensitive paths in it hands attackers a map.

Two final pitfalls: paths are case-sensitive (Disallow: /Admin/ does not block /admin/), and as the worked example showed, a trailing slash changes the meaning (/admin is a prefix, /admin/ is a directory). After every edit, test a handful of representative URLs with the robots.txt checker, or start from a safe template in the robots.txt generator and let it handle the syntax for you.