In short. Baiduspider is Baidu’s search crawler, with documented robots.txt controls and reverse DNS identification.

Definition: Baiduspider is Baidu's automated search crawler. It visits web pages to build the index database used by Baidu Search.

Key facts

As of 17 October 2026, Baidu's crawler introduction lists Baiduspider for web search and Baiduspider-image for image search.
Baidu's search workflow documentation describes following hyperlinks from seed URLs and revisiting previously crawled pages.
The platform's robots manual places robots.txt, a plain text file defining crawler access rules, in the website's root directory.

How does crawling feed Baidu Search?

Baidu's search workflow documentation separates crawling, indexing and result presentation. Crawling collects and updates web information. The indexing system then analyzes collected pages, extracts information and builds the database from which search results are selected.

The same documentation says Baiduspider adjusts crawl frequency according to factors including content update frequency and server pressure. It directs websites experiencing excessive crawl pressure to the platform's crawl frequency tool. It also describes a resource submission tool for sending URLs to Baidu and an index volume tool for checking how many pages can serve as search result candidates.

These tools measure different stages of the process: a crawl visit records collection, while the index volume concerns pages available for result selection.

How does robots.txt control Baiduspider?

Baidu's crawler introduction gives this configuration for blocking Baiduspider:

User-agent: Baiduspider
Disallow: /

The same introduction allows product-specific rules, including a separate Baiduspider-image group. It identifies exceptions: Baiduspider-cpro and Baiduspider-ads carry out customer-agreed operations, do not add fetched pages to the index and do not follow robots rules. It directs complaints about those crawlers to the advertising service contacts.

Baidu's robots manual defines Disallow as a path or path prefix to exclude and Allow as a path or prefix to permit. Its example Disallow: /help covers /help.html and /help/index.html; adding a trailing slash, as in /help/, restricts the match to that directory prefix.

The manual also documents * for matching zero or more characters and $ for an ending match. Path case matters: the manual requires an exact match between the case used in the rules and the target directories. Its Robots tool can create, validate and update the file and show its effective status in Baidu.

How can a website check crawler identity?

Baidu's crawler introduction recommends reverse DNS, which resolves an IP address to a hostname. It identifies *.baidu.com and *.baidu.jp as hostname patterns and gives host and nslookup examples. It also recommends checking access logs for impersonation.

FAQ

Does adding a robots block immediately remove existing results?

Baidu's crawler introduction says existing index records can take months to clear after crawling stops. You can also check the robots configuration for errors.

How can I check whether Baidu has indexed pages?

Baidu's search workflow documentation points to its index volume tool. You can use it to view the number of pages eligible as search result candidates.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer