Build a complete robots.txt with guided choices: set a default allow-all or block-all policy, list paths to Disallow, add a Crawl-delay and Sitemap, then allow or block specific crawlers from a built-in catalog — Google, Bing and other search engines, AI crawlers (GPTBot, Google-Extended, ClaudeBot, PerplexityBot, CCBot), SEO bots (Ahrefs, Semrush, Majestic) and social preview bots. One-click presets block all AI or SEO crawlers, and you can import a live site's existing robots.txt to edit it.
Pro tip: Only bots set differently from the default policy add rules, so leave most on Default to keep the file clean.
A robots.txt sits at the root of a domain and tells crawlers which paths they may or may not request. You set a default policy for all crawlers, optionally list paths to keep them out of, add a Crawl-delay and a Sitemap URL, and then allow or block individual crawlers from a built-in catalog of search engines, AI bots, SEO crawlers, and social preview bots. The output is plain text in the exact one-directive-per-line format crawlers expect, ready to paste into a file named robots.txt.
The policy switch is the main decision. "Allow all" emits Allow: / and lets you add specific Disallow paths beneath it — the normal setup for a public site that wants most pages crawled but a few folders left alone. "Block all crawlers" emits a single Disallow: / and nothing else, which keeps every well-behaved crawler off the entire site. That second mode is for staging or private builds, not for a site you want to appear in search.
The file always opens with a User-agent: * group carrying your default policy — Allow: / (with any Disallow paths beneath it) or a single Disallow: / to block every crawler. Each crawler you switch to Allow or Block in the catalog adds its own group below, for example User-agent: GPTBot followed by Disallow: /. The builder only writes a group when a crawler differs from your default, so the file stays clean. Disallow paths get a leading slash added automatically (typing admin and /admin both produce Disallow: /admin), and a Crawl-delay plus the Sitemap line are appended at the end.
The robots.txt text is assembled in your browser as you type, and nothing you build is uploaded. If you paste a URL or domain into the optional import box, the tool fetches that site's live /robots.txt once — server-side, read-only — and pre-fills the policy, User-agent, Disallow paths, Crawl-delay, and Sitemap so you can edit an existing file instead of starting from scratch. It never writes anything back to your server. After you copy the output, you still have to save it as robots.txt and place it at your domain root so it resolves at /robots.txt.
The importer models a single user-agent group — the wildcard * by default — because that is what this builder produces. If the fetched file has several user-agent groups or Allow: exceptions, only the * group is imported and a note tells you what was left out; the tool also shows the full fetched file beneath the output so you can copy any extra rules by hand.
Keep the file short and deliberate. Every Disallow you add removes pages from crawling, so list only paths you are sure should never be fetched — a stray /blog or / can quietly hide your whole site. Use an absolute URL for the Sitemap line, including the https:// and the full host, because relative sitemap paths are not valid here. If you maintain several subdomains, remember each one needs its own robots.txt; a file on the apex domain does not cover blog.example.com.
When you only need to keep one specific page out of search, robots.txt is usually the wrong tool — reach for a noindex tag instead (see the limitations below). Save robots.txt for crawl management at the folder level, where it does its real job.
robots.txt controls crawling, not indexing. A page you Disallow can still appear in search results if other sites link to it, because the crawler simply skips the body rather than forgetting the URL exists. To keep a page out of the index, allow it to be crawled and add a noindex meta tag, or protect it behind authentication. Treat robots.txt as a politeness request: reputable crawlers obey it, but it is not a security boundary and malicious bots can ignore it entirely.
This generator writes a default group plus a separate group for any crawler you allow or block from the catalog, which covers the vast majority of real robots.txt files. For very granular setups — per-crawler Allow exceptions to a Disallow, or wildcard patterns within a specific bot's group — you may still hand-edit afterward. And note that Google ignores Crawl-delay; it is emitted for the engines that read it.
Input: Policy: Allow all / User-agent: * / Disallow paths: admin, /cart / Sitemap: https://example.com/sitemap.xml
Output: User-agent: *\nAllow: /\nDisallow: /admin\nDisallow: /cart\n\nSitemap: https://example.com/sitemap.xml
The bare "admin" entry is normalized to /admin automatically.
Input: Policy: Block all crawlers / User-agent: *
Output: User-agent: *\nDisallow: /
Use this only where you do not want the site in search at all.
At the root of the host, so it loads at https://yourdomain.com/robots.txt. A file in a subfolder is ignored, and each subdomain needs its own copy.
No. It stops the crawler from fetching the page, but the URL can still be indexed from external links. Use a noindex tag on a crawlable page to drop it from results.
This builder targets the common case: allow the site, then carve out folders with Disallow. Per-path Allow exceptions that override a Disallow are an advanced pattern you can add by hand after copying.
Not reliably. Google ignores Crawl-delay and uses its own crawl-rate logic; some other engines honor it. It is included for the crawlers that read it, but do not depend on it for rate control.
The tool adds a single Sitemap line. If you need several, add the extra Sitemap lines manually after copying — each goes on its own line and applies to the whole site.
Building is local: as you type, the file is assembled in your browser and nothing you build is uploaded. The optional "Import from a URL" box is the one exception — it fetches that site's live robots.txt once, server-side and read-only, to pre-fill the form. It never writes anything back to your server.
Our editorial and data-source notes explain how each check works and where estimates come from.