Updated Googlebot Docs Clarify File Size Limits

SEO
1

image
image760×400 46 KB

Google Updates Googlebot Documentation on File Size Limits

Google has updated its documentation to clarify file size limits for Googlebot and other crawlers. The update moves default limits from the Googlebot page to Google’s broader crawling infrastructure documentation, while also specifying Googlebot’s own limits more precisely.

What’s New

Google describes the update as a two-part clarification:

  1. Default limits moved to crawler documentation

    • Previously listed on the Googlebot page, the default file size limits now appear under Google’s crawling infrastructure docs.

    • Reason: These limits apply to all Google crawlers and fetchers, not just Googlebot.

  2. Googlebot-specific limits clarified

    • Googlebot page now lists limits for Google Search crawling:

      • 2 MB for HTML and other supported text-based files

      • 64 MB for PDFs

    • In contrast, the crawler overview shows a 15 MB default across all Google crawlers and fetchers.

    • Each referenced resource (CSS, JavaScript, images) is fetched separately.

Why This Matters

This change continues Google’s documentation reorganization that began in late 2025. Key points:

  • November 2025: Core crawling docs migrated to a standalone site separate from Search Central, reflecting that Google’s crawlers serve multiple products (Search, Shopping, News, Gemini, AdSense).

  • December 2025: Additional updates covered faceted navigation and crawl budget optimization.

  • File size context: The 15 MB limit was first documented in 2022. It was already in effect but hadn’t been clearly recorded.

For SEOs managing crawl budgets or content-heavy pages, it’s important to note that limits now differ depending on the resource:

  • Crawling infrastructure overview: 15 MB default for all crawlers

  • Googlebot page: 2 MB for HTML/text files, 64 MB for PDFs

Google’s changelog doesn’t explicitly explain how the two limits relate, but separating crawler-wide defaults from Google Search–specific limits provides more clarity for developers.

Looking Ahead

Google’s reorganization suggests the crawling infrastructure documentation will continue evolving. By separating crawler-wide defaults from product-specific limits, Google can more easily introduce new crawlers and fetchers, and document their behavior consistently.