Which crawlers can actually reach this page
This fetches the real robots.txt and evaluates it against twenty seven named agents using the actual precedence rules: most specific group wins, longest matching pattern wins, and a named group inherits nothing from the wildcard. That last rule is where most robots.txt files quietly break.
Crawler Access Matrix
This site's robots.txt, fetched live
loading...
Path to evaluate
Or paste a different robots.txt
Matching follows the specification rather than intuition. Group selection picks the single most specific matching user agent token, with no inheritance from the wildcard group once a named group exists. Rule selection within that group picks the longest matching path pattern, and an Allow wins over a Disallow of identical length. Wildcards and end anchors are supported. Those two precedence rules are where almost every real robots.txt error lives.
The file everyone writes and nobody tests
Robots.txt looks like a list of instructions read top to bottom. It is not. It is a specificity contest, twice over, and the intuitive reading gives the wrong answer surprisingly often.
Named groups do not inherit
The moment you write a group for a specific agent, that agent stops reading the wildcard group entirely. Rules you assumed were global silently stop applying to it.
Longest match wins, not first
Order in the file is irrelevant. A three character disallow later in the file loses to a longer allow earlier in it, which is the opposite of how people read the thing.
Blocking is not the same as removing
A disallowed URL can still be indexed from external links, just without its content. If you want it gone you need a noindex directive on a page that is crawlable enough to be read.