See what crawlers are blocked
Instantly see which paths are disallowed for Googlebot, Bingbot, or all crawlers — before an accidental rule hides important pages from search.
Robots.txt Tester
Fetch and analyze any site's robots.txt. See crawl rules, blocked paths, declared sitemaps, and raw content in one view.
Why test robots.txt?
Find declared sitemaps
robots.txt is the standard place to declare sitemap URLs. Check that your sitemap is listed so crawlers can discover it automatically.
Catch misconfigurations
A single Disallow: / blocks every crawler from your entire site. Spot critical mistakes before they tank your rankings.
What is robots.txt and why does it matter?
robots.txt is a plain-text file at the root of your domain that tells crawlers which URLs they may access. It is one of the first things Googlebot fetches when it visits a site — and a misconfigured file can silently block your entire site from being indexed.
How does robots.txt work?
The file uses a simple format: User-agent lines specify which crawler the rules apply to (* means all), followed by Disallow and Allow directives for path patterns. Crawlers are expected to respect these rules, though they are not technically enforced.
robots.txt vs. noindex
Disallowing a URL in robots.txt prevents crawling but not necessarily indexing — a page can still appear in search results if other sites link to it. To prevent indexing, use a noindex meta tag on the page itself. Use both tools deliberately.
Common robots.txt mistakes
Disallow: / blocks everything. Blocking /wp-admin/ is fine; blocking /wp-content/ can hide images and assets. Forgetting to add a Sitemap: directive means crawlers must guess where your sitemap lives. Always test after changes.
How does this tester work?
Enter any URL and we fetch the robots.txt from the site's root domain. We parse every User-agent block, list Allow and Disallow rules, highlight Sitemap declarations, and show the raw file so you can verify the exact content crawlers see.
Also check your sitemap with our Sitemap Check.
Robots.txt FAQ
Enter a URL above to inspect a live robots.txt, or read the sections below for syntax and best-practice tips.
Always at the root of your domain: https://example.com/robots.txt. Subdomains need their own file. A robots.txt at a subdirectory path is ignored by crawlers.