Crawlability & Indexation

CMS-specific implementation guides

Operational runbooks translating this playbook onto each major CMS, including hosting edges, authoring workflows, and integration seams that typically move rankings and AI retrieval outcomes.

Prefer a CMS-wide lens before tackling another topic? Review every SEO & GEO playbook surfaced for WordPress, Shopify, Webflow, Drupal, HubSpot CMS, Contentful, or Adobe Experience Manager.

What Is Crawlability & Indexation?

Crawlability refers to how easily search engine bots and AI crawlers can access your site's pages. Indexation refers to whether those pages are actually added to the search engine's index. A page that can't be crawled won't be indexed — and a page that isn't indexed can't rank.

Why It Matters More Than Ever

AI search systems rely on fresh, comprehensive crawl data. If your best content is blocked, buried, or crawl-budget-wasted on low-value pages, AI systems simply won't know it exists. Crawlability and indexation are the foundation everything else is built on — no amount of content quality helps if the crawler can't reach it.

The Crawl Priority Hierarchy

• Robots.txt — The first gate; controls which paths crawlers can access at all
• Noindex meta tags — Page-level control; prevents indexation even if crawled
• Canonical tags — Signals the preferred URL when duplicate or similar content exists
• XML sitemaps — Proactively tells crawlers what pages exist and when they were updated
• Internal links — How crawlers navigate your site; pages with no internal links are often missed

Large sites have a finite crawl budget — the number of pages Googlebot will crawl in a given window. Wasting budget on thin pages, faceted navigation, or URL parameters means important pages get crawled less frequently. For AI-era SEO, prioritizing crawl budget toward your highest-value content is critical.

• Audit robots.txt — Check that no important sections are accidentally blocked; verify at search.google.com/search-console/robots-testing-tool
• Run a crawl audit — Use Screaming Frog or Sitebulb to crawl your site and identify noindex tags, canonical issues, and orphan pages
• Check Index Coverage in GSC — Review all "Excluded" URLs; fix "Crawled - currently not indexed" and "Discovered - currently not indexed" pages
• Generate and submit an XML sitemap — Include only canonical, indexable URLs; resubmit after major content updates
• Fix orphan pages — Any important page with zero internal links needs to be added to your internal linking structure
• Consolidate thin and duplicate content — Use canonical tags or noindex to prevent crawl budget waste on low-value URLs
• Block parameter URLs — Use robots.txt or URL parameter handling in GSC to prevent crawling of filtered/sorted/paginated duplicates
• Monitor crawl stats in GSC — Check Crawl Stats report monthly; drops in crawl rate signal problems
• Accidental robots.txt blocks — A single wildcard rule can accidentally disallow entire directories; always test after changes
• Noindex on important pages — Often left from dev/staging environments; audit before and after every migration
• Missing or outdated sitemaps — A sitemap with 404s or excluded URLs signals poor site hygiene to crawlers
• Faceted navigation crawl traps — E-commerce filter combinations can generate millions of low-value URLs that consume crawl budget
• Ignoring "Discovered - currently not indexed" — These pages are known to Google but not crawled; usually a crawl budget or quality signal issue
• Google Search Console — Index Coverage, Crawl Stats, and URL Inspection tools
• Screaming Frog SEO Spider — Comprehensive site crawl and technical audit tool
• Sitebulb — Visual crawl auditing with prioritized issue reporting
• XML Sitemaps Generator — Simple sitemap generation for smaller sites

How long does it take Google to index a new page?

Anywhere from hours to weeks depending on your site's crawl frequency, the page's internal link depth, and whether you submit it via Google Search Console's URL Inspection tool. Submitting the URL directly and including it in your sitemap can accelerate this to 24-72 hours for most sites.

Can a page be crawled but not indexed?

Yes — this is one of the most common and confusing crawl states. "Crawled - currently not indexed" means Google reached the page but chose not to index it, often due to thin content, low E-E-A-T signals, or duplicate content. Improving content quality is the fix, not a technical one.

Does blocking Googlebot hurt SEO?

Blocking Googlebot from specific sections (like admin pages or staging environments) is appropriate and necessary. The risk is accidental blocks to content you want indexed. Always use robots.txt testing tools after any changes.

How a Retailer Recovered 40% More Indexed Pages by Fixing Crawl Budget

A mid-size e-commerce retailer with 50,000 product pages noticed that only about 30,000 were indexed in GSC, despite all having sitemaps and no noindex tags. Log file analysis revealed that faceted navigation (color, size, and price filter combinations) was generating over 200,000 unique URL variants — consuming crawl budget and leaving thousands of product pages crawled infrequently or not at all. After blocking faceted navigation URLs via robots.txt and canonicalizing remaining parameter variants, Google's crawl budget refocused on actual product pages. Indexed page count climbed from 30,000 to 42,000 over three months — a 40% increase with no new content published.