Robots.txt Audit Claude Skill
This skill audits a website's robots.txt and produces an actionable report. It fetches and parses the file to find a redirected or HTML-served file, a 5xx that Google reads as block-everything, a sitewide Disallow: / that blocks the whole site, syntax errors, CSS and JS blocked from rendering, missing or unreachable Sitemap directives, a missing default User-agent group, unsupported directives that Google ignores, files over the 500 KiB limit, and BOM or encoding issues.
Quick Take
Point the skill at a site or robots.txt URL. It fetches the file without following redirects, parses every user-agent group and directive, runs fourteen severity-ranked checks, and returns a prioritized report with fixes.
What The Skill Checks
- Delivery problems that change how the file is read: a redirected or HTML-served file, a 5xx that Google interprets as block-everything, files over the 500 KiB limit, and BOM or encoding issues.
- Blocking mistakes: a sitewide
Disallow: /that blocks the whole site, CSS and JS resources blocked from rendering, a missing defaultUser-agent: *group, and syntax errors or directives that appear before any User-agent line. - Directive hygiene: missing or unreachable Sitemap directives, unsupported directives like noindex, nofollow, and crawl-delay that Google ignores in robots.txt, and duplicate or empty rules.
How The Skill Is Packaged
The skill follows the standard Claude Agent Skill structure: a SKILL.md file with YAML frontmatter and workflow instructions, a references/ folder with the full audit check definitions and report template, and a scripts/ folder with Python scripts that fetch and parse robots.txt and audit it. Copy the skill folder into your Claude skills directory and Claude invokes it automatically when a request matches its description.
Skill Files
Every file in the skill is embedded below directly from the Claude-SEO-Skills repository, so you can review exactly what the skill instructs Claude to do before installing it.
SKILL.md
The skill definition: frontmatter, inputs to collect, the four-step workflow, and the resources the skill relies on.
Audit Checks Reference (references/audit-checks.md)
Full definitions, severities, and rationale for all fourteen checks, spanning HTTP errors, file format problems, blocking rules, and missing directives.
Report Template (references/report-template.md)
The output structure the skill follows, with prioritized fixes for each failure.
Robots Collector Script (scripts/collect_robots.py)
Fetches robots.txt without following redirects, recording HTTP status, content type, file size, and gzip flags, and parses user-agent groups, sitemap declarations, and syntax issues.
Robots Auditor Script (scripts/audit_robots.py)
Runs the audit checks against the parsed robots.txt data and outputs the structured report data grouped by severity.