Does a Baiduspider robots rule apply to both desktop and mobile crawling?

Definition: Yes. Baidu's documentation says that a robots rule addressed to Baiduspider applies to both its PC and mobile crawling, because the two use different user-agent strings but the same Baiduspider agent name.

Key facts

• Baidu's website construction guidance says a Baiduspider robots block takes effect for PC and mobile at the same time.
• The same guidance says the PC and mobile versions of Baiduspider differ in their UA strings, while both are Baiduspider.
• Baidu's Baiduspider help page defines a User-agent value as the robot name used by a search engine and says * applies to any robot.
• RFC 9309 says a crawler selects the group matching its product token, and uses the * group when no specific group matches.
• Baidu places robots.txt at the site's root directory, according to its help page.

Why does one Baiduspider rule cover both variants?

Baidu's guidance, dated 18 September 2019, lists a mobile UA and a PC UA under its instructions for identifying Baiduspider. It then states that a robots block naming Baiduspider applies to PC and mobile simultaneously. The distinction between the two request forms is therefore in the UA string, not in the robots product name.

The practical consequence is narrow: this rule selects both desktop and mobile Baiduspider requests for the same path scope. A site cannot use User-agent: Baiduspider to block only the PC crawler while leaving the mobile crawler open. Baidu specifically describes that attempted arrangement as unsuitable for code-adaptive sites where the same URL returns a PC page or a mobile page according to the request.

How does the robots group get selected?

RFC 9309 defines a robots group as one or more user-agent lines followed by rules. It says crawlers use a product token to find relevant groups, match that token case-insensitively, and obey the matching group's rules. A product token should occur in the crawler's identification string, such as the HTTP User-Agent header.

For a shared Baiduspider exclusion, the relevant artifact is:

User-agent: Baiduspider
Disallow: /private/

The path rule applies to the matching group. If a request has no matching specific group, RFC 9309 says the crawler uses the User-agent: * group when one is present. Baidu's help page also says that * takes effect for any robot. A wildcard group therefore has a broader scope than the Baiduspider-specific group.

What does the rule block?

Baidu describes Disallow as specifying URLs that the robot should not visit. It treats a value as a full path or path prefix, so a rule such as Disallow: /private/ covers URLs beginning with that path under the selected group. Baidu's help page also says that the file belongs at the site's root, for example /robots.txt.

The robots protocol controls requested access to matching URLs. Baidu's help page notes that a blocked page can still appear in search results when other sites link to it, while Baidu does not crawl, index, or show the blocked page's content. A robots block is therefore a crawl instruction with search-result consequences described separately by Baidu.

How can you check the scope?

Read the User-agent group in the root robots.txt, then compare it with the crawler identity in server logs. A group named Baiduspider is shared by Baidu's PC and mobile variants under the cited guidance. Use separate log checks for the PC and mobile UA strings when verifying which requests reached a path.

RFC 9309 adds that the most specific matching path wins when Allow and Disallow rules overlap. Baidu's help page describes rule order as significant and says the robot acts on the first match, so overlapping rules should be reviewed with the crawler implementation being tested in mind.

Related terms

• Baiduspider
• robots.txt User-agent directive
• robots.txt group precedence
• robots.txt

FAQ

Can I block Baiduspider on desktop but allow it on mobile?

Not with a single distinction based only on User-agent: Baiduspider. Baidu's guidance assigns the same Baiduspider agent name to PC and mobile, so that group covers both.

Does a wildcard group have the same scope?

User-agent: * is broader. Baidu's Baiduspider help page says it applies to any robot, while a specific Baiduspider group targets the Baiduspider identity.

Where should I place the file?

Place robots.txt at the site's root. Baidu's help page gives the root URL as the location a spider checks before accessing the site.

Sources

1. 百度搜索资源平台, "网站建设", 18 September 2019
2. Baidu, "Baiduspider Help Center: How to block the crawling", undated
3. IETF, "RFC 9309: Robots Exclusion Protocol", September 2022

---

Check how a page reads to answer engines. Open the GEO score analyzer