robots.txt Generator

robots.txt is a plain-text file placed at your site root (/robots.txt) that tells search-engine crawlers which paths they may crawl. This generator turns a User-agent, Allow/Disallow rules, an optional Crawl-delay and one or more Sitemap URLs into a standard, ready-to-paste robots.txt.

Start from a preset — allow all, block all, or the WordPress default — then add or remove paths to fine-tune it. Everything is processed in your browser and nothing is sent to a server.

robots.txt controls crawling, not indexing

The most common misconception is that Disallow removes a page from search results. It does not.robots.txt only stops a crawler from fetching a URL; it does not prevent indexing. If many external links point to that URL, Google can still index the URL without reading its content and show it in results as "No information is available for this page."

To reliably keep a page out of search results, use noindex on the page itself — not robots.txt — via the meta tag <meta name="robots" content="noindex"> or the HTTP header X-Robots-Tag: noindex. Because a crawler must be able to fetch the page to see thenoindex, blocking that same URL in robots.txt while also setting noindexis contradictory.

Directives at a glance

• User-agent: the crawler the rules apply to. * means all bots, Googlebot only Google.
• Disallow: a path to block from crawling. It matches by prefix (e.g. /admin/). An empty value allows everything.
• Allow: an exception inside a disallowed area. The more specific rule wins.
• Crawl-delay: seconds between requests. Google ignores it; only some crawlers (Bing/Yandex) honor it.
• Sitemap: the absolute URL of a sitemap. You can list several on separate lines.

Common mistakes

1. Accidentally blocking the whole site with Disallow: /, dropping it from the index — always verify after deploy.
2. Blocking CSS/JS so Google can't render the page properly. Don't disallow your static assets.
3. Missing leading slashes or wrong casing. Paths are case-sensitive and must always start with /.

After deploying, use the robots.txt checker to confirm the live file behaves as intended, and the sitemap validator to verify the sitemaps you declared.

Wildcard * and $ pattern table

Googlebot and Bingbot support two patterns that aren't in the original standard: * matches any run of characters (zero or more) and $ anchors to the end of the URL. Combine them to target file extensions or query strings precisely. Remember every match is anchored to the start of the path.

How Allow vs Disallow conflicts resolve (walkthrough)

When both an Allow and a Disallow match the same URL, Google obeys the rule with the longer (more specific) path pattern. If the lengths tie, the least restrictive rule (Allow) wins. Consider this block:

• User-agent: *
• Disallow: /folder/ (8 chars)
• Allow: /folder/public.html (19 chars)

For /folder/public.html, both rules match but the Allow pattern is longer, so the URL is crawlable. Meanwhile /folder/secret.html matches only Disallow and is blocked. The key takeaway: it is not "the first line wins" but pattern length that decides — reordering the lines changes nothing.

Common pitfall

Treating Disallow: /folder (no trailing slash) as equivalent to Disallow: /folder/. The former is a prefix match, so it also blocks /folder-archive and /folderX.html, quietly dropping pages you never meant to hide. To block only what's inside a specific folder, always add the trailing /.