In short. Perplexity documents that Perplexity-User generally ignores robots.txt rules because a user requested the fetch.
Definition: As of 17 October 2026, Perplexity documents that Perplexity-User generally ignores robots.txt rules because its page fetches respond to user requests. A robots.txt file publishes instructions for automated crawlers; it provides no access authorization.
Key facts
- Perplexity's Crawlers documentation attributes this policy to the user requesting the fetch.
- The IETF's Robots Exclusion Protocol requires crawlers following the protocol to obey their matching rule groups.
- Google's robots.txt introduction explains that robots.txt instructions cannot enforce crawler behaviour and recommends password protection for private files.
Why does Perplexity distinguish user requests?
Perplexity's Crawlers documentation describes Perplexity-User as supporting actions within Perplexity: a question can prompt a page visit to inform an answer and link to that page. The operator excludes web crawling and foundation-model training collection from this agent's stated purpose.
The same documentation assigns PerplexityBot a separate role: surfacing and linking websites in search results. It recommends allowing that agent through robots.txt and its published IP ranges. A site's policy for PerplexityBot therefore addresses a different request category from Perplexity-User.
What does a robots.txt restriction control?
The IETF's RFC 9309, section 2.2.1, defines how a crawler selects the rules for its product token, the name used to identify it. If no specific group matches, the crawler follows the wildcard group, when present. Section 2.2.2 defines how Allow and Disallow rules match URL paths.
These rules depend on the client applying the protocol. Perplexity's stated user-request policy therefore makes a robots.txt Disallow rule unsuitable as the sole means of preventing Perplexity-User access.
For content that requires restricted access, section 3 of the RFC points to application-level security and gives HTTP Authentication as an example. It also warns that listing restricted paths in robots.txt exposes those paths publicly. Publishing a restriction can reveal a URL even when a crawler respects it.
How can a request be identified?
A user-agent string is an identifier sent with a request. Perplexity's Crawlers documentation publishes this string:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
Its web application firewall guidance combines user-agent matching with IP address verification, using the current published IP lists. It also recommends monitoring logs to check that firewall rules work as expected. These checks help distinguish the agent concerned when reviewing access rules.
Related terms
FAQ
Does Perplexity say the agent always ignores robots.txt?
Perplexity's policy uses the qualifier “generally.” It states the reason as the user having requested the fetch.
How should I protect private pages?
You can require authentication. The IETF's security considerations identify HTTP Authentication as an access-control measure for protected paths.
Sources
- Perplexity, "Perplexity Crawlers", undated
- IETF, "RFC 9309: Robots Exclusion Protocol", September 2022
- Google Search Central, "Introduction to robots.txt", 10 December 2025
Check how a page reads to answer engines. Open the GEO score analyzer