In short. Google can index a robots.txt-blocked URL through links from other pages; excluding a page with noindex requires crawler access to the instruction.
Definition: A
Disallowrule inrobots.txt, a site's crawler instructions file, tells crawlers to avoid fetching matching URLs. As of October 2026, Google's robots.txt introduction explains that Google can still index a blocked web page's URL through links from other pages.
Key facts
- Google's robots.txt introduction states that a blocked web page's URL can appear in search results without a description.
- Google's noindex documentation requires crawler access for Google to read a
noindexexclusion rule in page markup or a response header. - Google's URL Inspection documentation distinguishes the fields “Crawl allowed?” and “Indexing allowed?” when reporting a page's status.
Why can a blocked URL still appear?
Crawling means fetching a page. Google's robots.txt introduction explains that links from other sites can supply enough information for Google to index a disallowed URL without visiting it. Publicly available information, including anchor text, the clickable text of those links, can appear alongside the URL.
This is URL-only indexing: the address can appear in search results while robots.txt blocks access to the page's content. Google's documentation describes the resulting search listing as having no description. A crawl restriction therefore leaves a route for a linked URL to appear in Search.
What does a Disallow rule look like?
Google's robots.txt creation guide includes this rule group:
User-agent: Googlebot
Disallow: /nogooglebot/
User-agent identifies the crawler addressed by the group. Here, Googlebot is instructed to avoid URLs whose paths start with /nogooglebot/. The rule controls fetching those URLs.
The same guide places the file at the site's root, for example https://www.example.com/robots.txt. Files are allowed for crawling by default unless a rule restricts them. Selecting a path for Disallow is therefore a decision about crawler access.
How can a page be excluded from Google's index?
noindex is an instruction to exclude content from search indexing. Google's implementation guide provides this tag for an HTML page's <head> section:
<meta name="robots" content="noindex">
An alternative is an HTTP response header, which also works for non-HTML resources such as PDFs:
X-Robots-Tag: noindex
Google must be able to fetch the resource and read the instruction. For a blocked page, this requires removing the robots.txt restriction while keeping the noindex instruction accessible. The same documentation says Google removes the page from Search after crawling and reading the rule, even when other sites link to it. Google does not support placing noindex in robots.txt.
How can the result be checked?
Google's URL Inspection tool reports information about the indexed version of a URL and offers a live test for its current version. Within the page indexing details, “Crawl allowed?” reports robots.txt access, while “Page fetch” reports whether Google actually obtained the page.
The same help page explains that “Indexing allowed?” always shows “Yes” for a robots.txt-blocked page because Google cannot read its noindex instructions. These fields need to be read together. The indexed report reflects the last crawled version; after a configuration change, the live test checks current access.
Related terms
FAQ
Does Disallow protect private information?
Google's robots.txt introduction recommends password protection for private files because crawlers choose whether to obey robots.txt. Use access controls when you need to restrict who can read the content.
Why does my page still appear after I add noindex?
Google's noindex troubleshooting guidance says Google must crawl the page after the rule is added. If your robots.txt blocks the URL, remove that restriction so Google can read the instruction.
Sources
- Google Search Central, "Introduction to robots.txt", 10 December 2025
- Google Crawling Infrastructure, "How to write and submit a robots.txt file", 21 November 2025
- Google Search Central, "Block Search indexing with noindex", 10 December 2025
- Google Search Console Help, "URL Inspection tool", undated
Check how a page reads to answer engines. Open the GEO score analyzer