In short. Googlebot cannot crawl content hidden behind login, while subscription or registration content can be eligible when crawler access and paywall guidance are satisfied.
Definition: As of 17 October 2026, Google's Search technical requirements state that Googlebot, its search crawler, does not crawl pages made private by a login requirement. Subscription or registration content can be eligible for indexing when Googlebot has access and the publisher follows Google's paywalled-content guidance.
Key facts
- Google's Search technical requirements require access for Googlebot, an HTTP
200success response and indexable content. - Google's paywalled-content documentation covers content requiring a subscription or registration and explicitly requires crawler access.
- Google's robots.txt guide says a blocked URL can appear in results without Google visiting the page.
- Google's URL Inspection help requires internet access without login for a live test.
What does public access require?
Google's technical requirements say that pages must be accessible to the public and allow Googlebot to crawl them. A private page requiring login fails that access requirement. The same documentation requires an HTTP 200 response: client and server error pages are excluded from indexing.
Indexable content means text in a supported file type that complies with Google's spam policies. Access and a successful response establish technical eligibility; Google still decides whether to index the page. For a website, the relevant content is therefore the version accessible to Google's crawler. A successful response alone is insufficient when the intended content remains behind authentication.
What is the exception for subscription or registration content?
A paywall restricts content to subscribers or registered readers. Google's subscription and paywalled-content guide expressly covers both arrangements. It instructs publishers seeking indexing of restricted sections to make those sections accessible to Googlebot, and Googlebot-News where applicable.
Structured data, machine-readable page annotations, identifies the access restriction. Google's article example sets isAccessibleForFree to false. The following fragment matches the properties used in that example and belongs within the article's existing structured data:
{
"isAccessibleForFree": false,
"hasPart": {
"@type": "WebPageElement",
"isAccessibleForFree": false,
"cssSelector": ".paywall"
}
}
The .paywall selector references the HTML class surrounding the restricted section. The guide accepts class selectors for this property and asks publishers to validate their markup with the Rich Results Test. Crawler access remains a separate requirement alongside these annotations.
Why can a blocked URL still appear in results?
A robots.txt file controls crawler access to URLs. Google's robots.txt guide explains that links from other pages can supply enough information to index a blocked URL. The result can show the address and publicly available link text, while lacking a description of the page.
For private files, the same guide recommends password protection. For an accessible page that should be excluded from search, Google's noindex documentation supplies this instruction for the HTML head:
<meta name="robots" content="noindex">
Googlebot needs access to read that instruction. Placing it only in content hidden behind login prevents the crawler from seeing it.
How can access be checked?
In Google's URL Inspection tool, the inspected URL must belong to the currently opened Search Console property. Test live URL checks the current page; View tested page provides the returned HTML, HTTP headers and a rendered screenshot when response data is available.
These views help check whether the returned page contains the intended content or a login screen. The indexed result and live test are separate views. The live test checks current access.
Related terms
FAQ
Can I add paywall markup while keeping Googlebot blocked?
Google's paywall guide requires Googlebot access to the content intended for indexing, including restricted sections. Markup describes the restriction; the server must also provide access.
Can I use robots.txt to protect private customer pages?
Google's robots.txt guide recommends password protection for private files. Robots.txt instructions rely on crawlers following them and leave the possibility of a URL appearing in results through external links.
Can I run a live inspection using my account login?
Google's URL Inspection instructions require the tested page to be accessible from the internet without login. Your own authenticated browser session does not satisfy that requirement.
Sources
- Google Search Central, "Google Search technical requirements", 18 December 2025
- Google Search Central, "Structured data for subscription and paywalled content (CreativeWork)", 8 September 2026
- Google Search Central, "Introduction to robots.txt", 10 December 2025
- Google Search Central, "Block Search indexing with noindex", 10 December 2025
- Google Search Console Help, "URL Inspection tool", accessed 17 October 2026
Check how a page reads to answer engines. Open the GEO score analyzer