robots.txt Tester
Check whether a crawler is allowed to fetch a URL, and why.
Find Out What Your robots.txt Actually Blocks
robots.txt looks simple and behaves subtly. The rule that decides an outcome is not the first one that matches or the last — it is the longest pattern, with Allow winning a tie against Disallow. That is how a single Allow: /blog/ can override a broad Disallow: /, and how a rule you meant as a catch-all quietly does nothing.
This applies that rule to whatever paths you give it and tells you which line decided each answer.
Key features
- Correct precedence — longest match wins, Allow beats Disallow on a tie, exactly as Google and Bing implement it.
- Wildcards and anchors —
*and a trailing$are both supported. - Per-crawler groups — Googlebot, Bingbot, GPTBot and others, falling back to
*when no group names them. - The deciding line named — so you can see which rule produced the verdict.
- File problems flagged — rules before any user-agent line, a sitemap that is not a full URL, lines missing a colon.
This tool runs entirely inside your browser using native Web APIs. Your files and text are never uploaded to a server, never logged and never shared with third parties.
How to use: robots.txt Tester
- Paste your robots.txt
The whole file. Parse problems appear immediately. - List the paths to test
Full URLs or paths — the host is stripped either way. - Choose a crawler
Rules are often different for Googlebot than for the wildcard group. - Read the verdicts
Crawl or blocked, with the deciding rule and its line number.
Technical specifications
| Processing location | Entirely in your browser — no server round trip |
|---|---|
| Data uploaded | None. Files and text never leave your device |
| Price | Free — no account, no trial, no usage cap |
| Category | Developer |
| Works offline | Yes, once the page has loaded |
| Browser support | Chrome 90+, Edge 90+, Firefox 90+, Safari 15+ |
| Interface languages | English, 中文, हिन्दी, Español, العربية |
Frequently asked questions
Which rule wins when two match?
The one with the longest pattern. If two matching rules are the same length, Allow wins. That is why <code>Disallow: /</code> plus <code>Allow: /public/</code> lets <code>/public/</code> through — the Allow pattern is longer for those URLs.
Does blocking a page in robots.txt keep it out of search results?
No, and this is the most common misunderstanding. robots.txt controls <em>crawling</em>, not indexing. A blocked URL can still appear in results if other pages link to it — search engines simply cannot see what is on it. To keep a page out of the index, let it be crawled and serve a <code>noindex</code> meta tag.
Why is my rule being ignored?
Most often it appears before any <code>User-agent</code> line, which makes it belong to no group at all. The tester flags that specifically. The other common cause is a more specific user-agent group elsewhere in the file taking precedence for that crawler.
Should I block AI crawlers?
That is a judgement about your content, not a technical question. Adding a <code>User-agent: GPTBot</code> group with <code>Disallow: /</code> is how it is done, and this tester will show you the effect. Be aware it is a request that well-behaved crawlers honour, not an enforcement mechanism.