In short. RFC 9309 requires parsing capacity of at least 500 KiB, while Google documents a 500 KiB limit and ignores content beyond it.
Definition: A robots.txt file contains rules that request how crawlers access a site's URLs. RFC 9309 requires a parsing limit of at least 500 KiB; as of 17 October 2026, Google documents a 500 KiB limit and ignores content beyond it.
Key facts
- The IETF's RFC 9309, section 2.5 requires crawlers to support parsing at least 500 kibibytes and recommends a limit to protect their systems.
- Google's robots.txt specification says content after its 500 KiB maximum is ignored.
- Google's file creation guide identifies its open source robots.txt library as a way to test files locally with the parser used in Google Search.
What does the standard require?
A parser is software that reads the file and interprets its rules. The limit in RFC 9309 sets a minimum capacity: an implementation can support a larger file. Section 3 describes this requirement as the lower limit of bytes that must be processed. File size is therefore measured in bytes, rather than a count of rules.
Section 2.3.1.5 also requires crawlers to try to parse each line and use parseable rules. The standard gives implementers room to cap processing while requiring capacity for at least 500 KiB.
What happens to rules beyond Google's limit?
According to Google's file format documentation, content beyond 500 KiB is ignored. A rule placed entirely beyond that boundary therefore has no effect on Google's parsing of the file.
Keeping the complete file within the limit avoids putting intended instructions in the ignored portion.
How can a file stay within the limit?
Google's specification recommends consolidating rules and placing excluded material in a separate directory. A directory rule can cover its contents without listing individual URLs. The following example uses the specification's documented /fish/ path pattern:
User-agent: *
Disallow: /fish/
The pattern matches /fish/ and /fish/salmon.htm, while /fish.html remains outside that match. These path differences matter when replacing individual rules.
How can the uploaded file be checked?
Google's creation guide recommends opening the robots.txt URL in a private browsing window to check public access. It also identifies the robots.txt report in Search Console for files already accessible on the site, and its open source parser for local testing.
Related terms
FAQ
Must every crawler stop at 500 KiB?
RFC 9309 requires a parsing limit of at least 500 KiB. A crawler can support more; Google's stated limit applies to its implementation.
Can I shorten a file by grouping excluded URLs?
Yes. Google's specification recommends a separate directory for excluded material. Check the resulting path matches before replacing your existing rules.
Sources
- IETF, "RFC 9309: Robots Exclusion Protocol", September 2022
- Google, "How Google interprets the robots.txt specification", updated 31 August 2026
- Google, "How to write and submit a robots.txt file", updated 21 November 2025
Check how a page reads to answer engines. Open the GEO score analyzer