Robots.txt Tester

Free robots.txt tester: fetch any site’s robots.txt, test URLs against Googlebot, Bingbot or AI crawlers, see the matching rule and find syntax errors.

Loading tool…

Load the robots.txt of any website (or paste your own), then test which URLs are allowed or blocked for Googlebot, Bingbot, AI crawlers or any user agent. The tester follows Google’s rules for wildcards, longest match and Allow vs Disallow, shows the exact rule that decided each URL and checks the file for mistakes.

How to use the Robots.txt Tester

  1. Enter a website and click Fetch robots.txt, or paste robots.txt rules into the box.
  2. Add the URLs or paths you want to test, one per line, and choose a crawler.
  3. Click Test URLs to see whether each one is allowed or blocked and which rule matched.

Frequently asked questions

How does Google decide if a URL is blocked?

Google uses the group for the most specific matching user agent, then picks the rule with the longest matching path. If an Allow and a Disallow rule are the same length, Allow wins. URLs with no matching rule are allowed.

Which wildcards does robots.txt support?

Google and Bing support * (any sequence of characters) and $ (end of the URL). For example Disallow: /*.pdf$ blocks every URL that ends in .pdf.

Can I edit the rules and test again?

Yes. After loading a robots.txt you can change any line in the box and test again. This is a safe way to try new rules before you upload them to your site.

What happens if robots.txt does not exist?

If robots.txt returns a 404 or another 4xx error, Google treats the whole site as allowed. If it returns a 5xx server error, Google may stop crawling the site until the file is reachable again.

Does Disallow remove a page from Google?

No. Disallow stops crawling, but the URL can still be indexed if other pages link to it. To remove a page, allow crawling and add a noindex meta tag, or protect it with a password.

Is Crawl-delay supported by Google?

No. Google ignores Crawl-delay. Bing and some other crawlers respect it.

How many URLs can I check?

You can check up to 20 URLs at a time. To keep the service fast for everyone there is a fair-use limit of about 60 checks every 10 minutes per visitor.

Can I check pages on localhost or a private network?

No. For security our server only connects to public websites. Local, private and reserved IP addresses are blocked.

Is the Robots.txt Tester free to use?

Yes. The Robots.txt Tester on BookmarkBuild is completely free, with no hidden charges, no trial period and no watermark.

Do I need to create an account?

No. You can use the tool straight away without signing up or logging in.

Is my data stored or sent anywhere?

Only the URL you enter is sent to our server so it can be requested from there. We do not store the URL or the result, and the report is built in your browser.

Does the Robots.txt Tester work on mobile phones?

Yes. It works in any modern browser on Android phones, iPhones, tablets, laptops and desktop computers.