Robots.txt Tester

About this tool

Robots.txt Tester is meant to load a site's robots.txt and check whether a URL path is allowed for a crawler. You enter a site URL; the tool takes its origin, asks the UtilityKit server to fetch /robots.txt, and would then show the raw file, the parsed User-agent groups with their Allow and Disallow lines, and a Test path form with a user-agent field (Googlebot by default). Today the server request it uses returns only the HTTP status and headers, never the file body, so the tool always reports “robots.txt is empty or the backend did not return a body” and the test form never appears. There is no field to paste robots.txt text. Until this is fixed, use Google Search Console's robots.txt report or open https://<site>/robots.txt and check the rules by hand.

Why use it

• Builds the robots.txt address from any page URL on the site.

• Designed to show parsed User-agent groups with their Allow and Disallow lines next to the raw file.

• The test form reports the matched rule as well as the Allowed or Disallowed verdict.

How to use

1. Enter the site URL (https:// is added if you leave it out); only the origin is used.
2. Click Fetch robots.txt to ask the UtilityKit server for <origin>/robots.txt.
3. If the file loads, enter a Test path and a User-agent, then click Test path to see Allowed or Disallowed and the matched rule.
4. If you see “robots.txt is empty or the backend did not return a body”, the fetch failed; open /robots.txt in a new tab and read the rules instead.

When it helps

• Verifying important pages (blog posts, product pages) are not blocked from Googlebot.
• Diagnosing why a page is not being indexed despite being published.
• Auditing a live robots.txt for accidental blocks on crawl-critical paths.
• Confirming AI crawler directives (GPTBot, ClaudeBot, CCBot) match your stated policy.

Examples

https://example.com

robots.txt is empty or the backend did not return a body. Paste text manually or check the raw URL.

https://example.com/blog/post-1

The tool requests https://example.com/robots.txt; the result is the same message.

Tips

• Always declare your sitemap with 'Sitemap: https://example.com/sitemap.xml' at the top or bottom — search engines pick it up regardless of user-agent group.
• Use 'Disallow: /' under a specific user-agent like GPTBot to block AI training while allowing search engines.
• Wildcards (*) match any character sequence; $ anchors to URL end. 'Disallow: /*.pdf$' blocks all PDFs but not PDF query parameters.
• Test each major bot — Googlebot, Bingbot, AppleBot, GPTBot — separately. Different bots may match different groups.
• Avoid blocking critical CSS/JS via robots.txt — Googlebot needs them to render pages and judge mobile-friendliness.
• Robots.txt only suggests crawler behavior — it does not enforce it. Use authentication or noindex meta tags for true privacy.
• Keep robots.txt under 500KB; Google ignores anything past that size limit per the official spec.

Frequently Asked Questions

Glossary

robots.txt directives

The set of instructions in a robots.txt file: User-agent, Allow, Disallow, Crawl-delay, and Sitemap. Each directive has a name and value separated by a colon.

User-agent

An identifier string declared in robots.txt that specifies which crawler the following rules apply to. '*' matches all crawlers; specific names (Googlebot, Bingbot) override the wildcard.

Allow

A directive that explicitly permits a URL pattern to be crawled, often used to override a broader Disallow rule. Google honors Allow; some older crawlers do not.

Disallow

A directive that tells the matching user-agent not to fetch URLs matching the pattern. Empty Disallow value (Disallow:) is a no-op.

Sitemap declaration

A line of the form 'Sitemap: https://example.com/sitemap.xml' that tells crawlers where to find the XML sitemap. Independent of User-agent groups.

Wildcard (*)

A pattern character that matches any sequence of characters in a URL path. Google supports * in both Allow and Disallow values.

End-of-URL ($)

A pattern character that anchors the match to the end of the URL path. 'Disallow: /*.pdf$' blocks .pdf URLs but allows /file.pdf?query=1.

Crawl-delay

A directive specifying minimum seconds between successive crawler requests. Honored by Bing and Yandex but ignored by Google (which throttles automatically).

RFC 9309

The IETF standard published in 2022 that formalized the robots.txt protocol after decades of de-facto implementation. Defines parsing, precedence, and limits.