Developer

Robots.txt Generator

Generate a valid robots.txt with user-agent, allow/disallow, sitemap, and host directives.

Loading interactive tool…

Embed this tool on your site

About this tool

A robots.txt file is the first thing search engine crawlers check before visiting any page on your site. Get it wrong and you may accidentally block Google from your whole site, or fail to keep well-behaved bots out of admin and API paths. The rules follow a simple but finicky syntax: User-agent lines select which bots to address, Disallow paths block crawling, Allow paths create exceptions, and Sitemap lines help crawlers discover your content. This tool writes one User-agent block from a short form: type the user-agent, list Allow and Disallow paths one per line, add Sitemap URLs and an optional Host line. Paths without a leading slash get one added, and the output updates as you type with line and byte counts. Note that an empty Allow box becomes Allow: / and an empty Disallow box becomes Disallow: /, so if you list only Allow paths, everything else is blocked.

Why use it

  • Form Instead of Raw Syntax: Fill in Allow and Disallow paths one per line and the tool writes each as a correctly spelled directive under a single User-agent line, so you do not have to remember the exact keywords.

  • Leading Slash Added for You: Paths typed without a leading slash, such as admin/, are written as /admin/, which avoids one of the most common robots.txt mistakes.

  • Sitemap Inclusion: Add one or more sitemap URLs and each becomes its own Sitemap: line, which helps crawlers like Googlebot and Bingbot discover your content.

  • Wildcard Paths Passed Through: Patterns like /*?sort= are kept exactly as typed, so you can block parameter-based URL variants without blocking the base URL.

  • Plain-Text Output: The output is plain-text robots.txt content with a live line and byte count, ready to save as robots.txt and upload — no build step or server restart required.

How to use

  1. Type the User-agent — keep * for all bots or enter a specific crawler name like Googlebot
  2. List Disallow paths, one per line, for directories and pages you want to block from crawling
  3. List Allow paths, one per line, to create explicit exceptions — or leave the box empty to write Allow: /
  4. Add one or more Sitemap URLs and, if you want, a Host value
  5. Check the live robots.txt output and the Lines and Bytes counts
  6. Copy the output and upload it to your site's domain root as /robots.txt

When it helps

  • You are launching a new site and need to create a robots.txt that allows indexing of public pages and blocks admin and API paths
  • You are migrating a site and need to ensure robots.txt does not accidentally block your new URL structure
  • You want to block specific scrapers or SEO tools from accessing your site without blocking legitimate search engines
  • You need to add your sitemap URL to robots.txt for better crawl discovery by search engines
  • You want to block URL parameter variants from crawling to save crawl budget for canonical pages
  • You are setting up a staging environment and need a robots.txt that blocks all crawlers from indexing it

Examples

Block admin, allow rest

Input
User-agent: * | Disallow: /admin/ | Sitemap: https://example.com/sitemap.xml
Expected result
User-agent: *
Allow: /
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml

Single named bot with Host

Input
User-agent: Googlebot | Disallow: /staging/ | Host: example.com
Expected result
User-agent: Googlebot
Allow: /
Disallow: /staging/
Host: example.com

Block parameter URL variants

Input
User-agent: * | Disallow: /*?sort= and /*?filter=
Expected result
User-agent: *
Allow: /
Disallow: /*?sort=
Disallow: /*?filter=

Tips

  • Disallow blocks crawling, not indexing — to prevent a page from appearing in search results, add a noindex meta robots tag in addition to the disallow rule.
  • Always include a Sitemap: directive at the bottom of your robots.txt — it is one of the most reliable ways to ensure new content is discovered by crawlers promptly.
  • Test your robots.txt with Google Search Console's robots.txt Tester before deploying to verify the rules have the intended effect on specific URLs.
  • Do not block CSS and JavaScript files — Google needs them to render pages correctly for ranking. Blocking them can reduce your pages' ranking.
  • Add specific User-agent rules above the wildcard * block — when both match a bot, the more specific named rule takes precedence.

Frequently Asked Questions

Where does robots.txt go on my site?
robots.txt must be placed at the root of your domain at exactly https://yourdomain.com/robots.txt. It cannot be in a subdirectory. A robots.txt at /blog/robots.txt controls nothing — crawlers only check the root location.
Does Disallow guarantee a page will not appear in search results?
No. Disallow blocks crawling but not indexing. A page can still appear in search results if it has inbound links, even if it is disallowed. To prevent indexing, use the noindex meta robots tag or X-Robots-Tag HTTP header.
What is the difference between robots.txt and the meta robots tag?
robots.txt controls crawling at the URL level — blocked URLs are not fetched. The meta robots tag controls what happens after the page is crawled: noindex prevents indexing, nofollow prevents link equity passing. Use robots.txt to save crawl budget; use noindex to prevent indexing.
Can I block specific bots like AhrefsBot or SemrushBot?
Yes. Enter AhrefsBot as the User-agent and / as the Disallow path. The tool writes one User-agent block at a time, so to combine several bots generate each block and paste them into one file separated by a blank line. Well-behaved bots respect robots.txt, but malicious scrapers typically do not — robots.txt is not a security measure.
What does User-agent: * mean?
The wildcard * matches all robots and crawlers that have not been given a specific named user-agent block. Rules in the * block apply to any bot not addressed by a more specific block in the file.
What happens if I leave the Allow or Disallow box empty?
An empty Allow box is written as Allow: / and an empty Disallow box as Disallow: /. With only Disallow paths filled in, the result allows everything except those paths. With only Allow paths filled in, the result blocks everything except those paths, so add a Disallow path if that is not what you want.
Should I add my sitemap URL to robots.txt?
Yes. The Sitemap: directive is one of the most reliable ways to ensure all major crawlers discover your sitemap. Google, Bing, and other compliant crawlers read this directive when processing robots.txt.
Does case matter in robots.txt paths?
Yes. Path matching in robots.txt is case-sensitive. Disallow: /Admin does not block /admin. Always match the exact case of your actual URL paths.
Why is Google still indexing my disallowed page?
Google may have indexed the page before you added the disallow rule, or it may have discovered the URL from an external link and indexed it from that signal. Remove indexed pages using Google Search Console's URL removal tool, and add a noindex meta tag for belt-and-suspenders protection.

Explore the category

Robots.txt Tester Fetch a site's robots.txt through the UtilityKit server and test whether a path is allowed for a user-agent (fetching currently returns no file content). XML Sitemap Generator Generate XML sitemap files from URL lists with optional metadata defaults. User Agent Parser Parse user-agent strings into browser, engine, OS, and device details. Hash Generator Generate SHA-256, SHA-1, MD5 hashes UUID Generator Generate random UUIDs v4 HTML Minifier Minify HTML by removing comments and unnecessary whitespace while preserving valid markup.

Glossary

robots.txt
A plain-text file placed at the root of a domain that instructs web crawlers which pages and directories they may or may not access, following the Robots Exclusion Protocol.
User-agent
A robots.txt directive that specifies which crawler the following rules apply to. Use * for all crawlers, or a specific name like Googlebot, Bingbot, or AhrefsBot for targeted rules.
Disallow
A robots.txt directive that instructs matching crawlers not to fetch URLs matching the specified path. An empty Disallow: value means the crawler is allowed everywhere on the site.
Allow
A robots.txt directive that explicitly permits crawling of a path, used to create exceptions within a broader Disallow rule. Allow takes precedence over Disallow when both match the same URL.
Sitemap
A robots.txt directive providing the absolute URL of an XML sitemap file, helping crawlers discover all indexable pages on a site. Multiple Sitemap: lines are allowed in a single robots.txt.
Crawl-delay
An optional robots.txt directive requesting a minimum wait time in seconds between requests from a crawler. Supported by some bots like Bingbot but not by Googlebot, which manages crawl rate separately via Search Console.
A LITTLE HELP. A LOT DONE.
Good tools.
Great flow.

Your shortcut to the everyday. Create, convert, calculate,
and get back to what you love.

✓ Free to use✓ No sign-up✓ 500+ possibilities