Free Robots.txt Checker

A wrong Disallow rarely shows up as a red banner in your CMS. You notice it when organic traffic dips, AI answers stop citing you, or Search Console says “blocked by robots.txt” on URLs you thought were open.

You do not need Screaming Frog running to answer the first question: “Is GPTBot blocked on / right now?” Enter a domain, set a test path if you need one, and read allow/block per bot. You get the matched User-agent group from your file, HTTP status, sitemap lines, Host directive, and parse notes when relevant.

Re-run after you deploy a fix. That is the loop.

What is robots.txt?

Robots.txt is a plain-text file at the root of your site (https://example.com/robots.txt) that tells crawlers which paths they may or may not request. It uses User-agent groups and Allow / Disallow rules (plus optional Sitemap and Host lines).

Example:

User-agent: GPTBot
Disallow: /

User-agent: *
Allow: /
Disallow: /admin/

Important nuances:

• Robots.txt is a crawl directive, not a guarantee of indexing. Pages can still be indexed without being crawled in some cases, but blocking crawl is the first gate.
• Most specific User-agent group wins for a given bot token (we match the way major engines describe substring match on the crawler product name).
• Longest matching path rule wins. When Allow and Disallow tie on the same path length, Allow takes precedence (Google’s documented behavior).

Google’s robots.txt documentation remains the reference for how Googlebot interprets the file. This tool is a practical checker, not an official Google tester.

Search bots vs. AI training vs. AI retrieval

Not every bot on the internet appears in your marketing slide deck. Sprout groups 35 named tokens into four buckets:

Training and retrieval are different decisions. Many sites block training (GPTBot, Google-Extended) but allow retrieval (OAI-SearchBot, PerplexityBot) so they stay out of model datasets but remain eligible for citations. This tool shows both side by side so you can spot lopsided policies.

Complete list of crawlers we check (35)

For each bot we evaluate Allow/Disallow for your test path and show which User-agent group in your file matched. The robots.txt token is the string to use in User-agent: lines (must match how vendors document it).

Search crawlers (12)

AI training crawlers (10)

AI retrieval crawlers (7)

SEO and audit crawlers (6)

Note: The catalog is updated when vendors publish new tokens. The live tool shows the current count in the header. Bots not listed here are not in the report unless your file’s User-agent: * group applies to them.

Why robots.txt mistakes hurt SEO and AEO

Robots.txt sits at the intersection of crawl budget, index coverage, and AI visibility.

• Accidental full-site block: Disallow: / on User-agent: * (or on the wrong specialized agent) can starve crawl across the site.
• Split Google policies: Googlebot allowed but Googlebot-Image or Google-Extended blocked affects different surfaces (images, AI training) independently.
• False confidence: a 200 response that returns an HTML login or CDN page is not a robots file. Parsers and humans both get misled.
• AI citation drift: blocking retrieval bots does not always remove you from every AI product, but it is the lever you control in robots.txt for many providers’ documented tokens.
• SEO tool blind spots: blocking AhrefsBot or SemrushBot does not hurt Google. It hurts your visibility in those tools.

International and large sites often maintain one robots file for dozens of hostnames. One stale group after a migration can block an entire path prefix for every bot using *.

What this tool checks

After you submit a domain:

1. Fetches https://your-domain/robots.txt (HTTP or HTTPS normalized from your input), fetching the page directly and safely.
2. Reports HTTP status: including 404 (missing), 403/401 (inaccessible), empty 200, and non-text/HTML responses flagged as unreliable.
3. Parses User-agent groups, Allow/Disallow rules, Sitemap lines, and Host directive; surfaces parse warnings for unrecognized lines.
4. Tests a path: default /, or a path you enter, or path+query extracted from a pasted full URL.
5. Evaluates each catalog bot: allow or block for that path, with the matched User-agent group from your file.
6. Summarizes AI bots: how many of the 17 AI training + retrieval bots we track are blocked for the test path.
7. Shows an AI crawler access score: weighted toward retrieval bots, then training bots, then whether a usable robots.txt was found (see below).
8. Displays the response body when the server returns one (including some error pages for manual inspection) with copy to clipboard.

What we do not do: live HTTP requests as each bot (we fetch the file once as Sprout’s checker), sitewide crawl simulation, or validation of rules on hostnames other than the origin you entered. Custom or unpublished user-agent strings not in our catalog are not listed unless they share a group with *.

How to check robots.txt

1. Enter your domain (example.com or https://example.com).
2. Optionally set a test path (/, /pricing, or a full URL like https://example.com/blog/post?q=1).
3. Click Check robots.txt and read the summary, score hint, HTTP pill, and bot list.
4. Filter by AI bots, Search, SEO tools, Blocked, or All.
5. Open the robots.txt panel, copy the file, and edit on your server or CDN.
6. Re-run the checker on the same path after deploy (allow time for CDN cache purge).

For launches, test / plus one representative blocked section (e.g. /admin/, /api/) and one AI-sensitive path (e.g. /blog/).

Sprout SEO Extension

See crawler access, right from your browser

Checks robots.txt, meta robots, and your page's access for 14 named AI crawlers, instantly, on every page you visit.

Add to Chrome Also available for Firefox and Edge.

Learn more →

Common robots.txt issues (and what to do)

Missing robots.txt (404)

Symptom: File health missing; all bots shown as allowed (default when no rules exist).
Fix: Decide intentionally. If you want restrictions, publish a robots.txt. If you rely on “allow all,” document that choice for compliance and AI policy.

Empty file or HTML at /robots.txt

Symptom: empty or invalid file health; notice that the response is not usable rules.
Fix: Serve plain-text UTF-8 rules. Fix CDN/origin routing so /robots.txt is not the SPA shell or login page.

GPTBot blocked but Google-Extended allowed (or vice versa)

Symptom: Mixed allow/block across AI training bots.
Fix: Align with your legal/product policy. Block or allow each token you care about; do not assume one line covers all OpenAI or Google AI crawlers.

Specialized Googlebot groups

Symptom: Googlebot allowed, Googlebot-Image blocked on /.
Fix: Expected if intentional (images vs web). If unintended, add explicit groups per Google’s list of user agents.

Allow/Disallow overlap on the same path length

Symptom: Surprising allow on /public/ when both Allow and Disallow exist.
Fix: Remember allow wins on ties at the same specificity; use longer, more specific paths for the exception you mean.

Wildcard and end-anchored rules

Symptom: PDFs or query strings behave unexpectedly.
Fix: Use * and $ carefully; test the exact path in the tool (including query string if your rules match it).

Sitemap not declared

Symptom: No Sitemap lines in parse output.
Fix: Add Sitemap: https://example.com/sitemap.xml. Robots.txt is a standard discovery path (we surface lines we parse; we do not validate that the sitemap URL works).

Blocking SEO tool bots

Symptom: AhrefsBot or SemrushBot shows Block.
Fix: Harmless for Google rankings; only affects those vendors’ ability to crawl you for reports and backlinks.

Robots.txt vs. meta robots vs. noindex

Rule of thumb: if you need a URL out of the index, rely on noindex (and allow crawl so the tag can be seen), not Disallow alone. Use robots.txt for crawl budget, AI bot policy, and large path prefixes.

Pair with Sprout’s Free Canonical Tag Checker when coverage issues mix crawl blocks with duplicate URL problems.

AI crawler access score (what the number means)

The ring score is an AI crawler access score, not a Lighthouse grade and not Google’s official robots tester.

• ~60%: share of AI retrieval bots in our catalog allowed on the test path.
• ~25%: share of AI training bots allowed.
• ~15%: file health (usable robots.txt found vs missing, empty, invalid, or inaccessible).

A high score with missing robots.txt means “everyone is allowed by default,” not “you have a great file.” Read the summary and file-health pill together. The UI states that this is based on our catalog, not a guarantee any vendor will crawl you.

How this compares to other robots.txt tools

Check technical SEO as you browse (Chrome, Firefox, Edge)

Robots.txt review is a deliberate, enter-a-domain task. Day-to-day SEO is browsing: templates change, staging leaks, headers drift.

Install the Sprout SEO extension once. On any page, you get fast on-page and technical checks without opening a separate audit, the same workflow you use on canonical, hreflang, and broken link checks.

Use this page when you need crawler allow/block for AI and search bots. Use the extension when you need eyes on the live page while you move through a site.

Add Sprout SEO to Chrome (also available for Firefox and Edge). Free, no account required.

In short

1. Robots.txt errors block crawl and shape AI training/retrieval access; the file is small, but the blast radius is not.
2. Sprout fetches live /robots.txt, parses groups, tests a path with longest-match rules, and maps 35 catalog bots (including 17 AI bots) to allow/block with matched User-agent groups.
3. Fix file availability and HTML-at-robots-url problems first, then align AI tokens with policy. Re-run after every CDN or edge deploy.

Check your robots.txt, then stop guessing on every crawler after this one

A wrong Disallow doesn’t throw an error. It just quietly shapes which bots can see you, and which ones give up and leave.

Add Sprout SEO to Chrome to see technical SEO, including robots.txt, on every page you browse, free, with no crawl and no account required. It also works in Firefox and Edge.