In short. Diffbot combines link crawling with structured extraction and documents crawler identities, robots.txt defaults and customer overrides.

Definition: Diffbot crawler refers to Diffbot software that follows website links and supplies pages to APIs that extract structured data. Its Crawl API combines link traversal with extraction, producing a collection of page records that can be downloaded or queried.

Key facts

As of October 2026, Diffbot's Crawl API documentation describes a service that passes website links to automatic or custom extraction APIs and compiles the results into a collection.
The Extract API renders and classifies pages, then returns structured objects in JSON, a text format for named fields and values.
Diffbot's crawler FAQ distinguishes proactive crawling under Diffbot from requests made for a human user under Diffbot-User.

How does structured extraction work?

A seed URL is a starting address for following links. Diffbot's Create a Crawl documentation requires a job name, seed URLs and an apiUrl specifying the extraction API. Starting a job immediately begins link traversal and processing. Separate crawling and processing patterns can limit which links are followed and which pages undergo extraction.

Diffbot's Extract documentation describes Analyze as the API that classifies a page and routes it to the appropriate extractor. For a known page type, a client can call an extractor directly. The Article API returns fields such as author, date and body text; the Product API returns fields such as price and availability. These named fields are the concrete output of structured web extraction.

The result retrieval documentation specifies JSON as the default download format. A CSV export contains only top-level fields, so the choice of format affects how much of the extracted structure is retained.

Which requests use the Diffbot name?

A User-Agent is a string that an HTTP client sends to identify itself. Diffbot's robots.txt FAQ describes Diffbot as proactive crawling for a general search engine and associates it with the Diffbot Knowledge Graph and web search services. The same page describes Diffbot-User as requests made in response to a human entering a URL in Diffbot software.

For customers using Extract or Crawlbot, the FAQ recommends setting a User-Agent that represents their own organization. A customer-run extraction job can therefore identify itself differently from Diffbot's proactive crawler. The FAQ also states that Diffbot's proactive crawler is not used for AI training.

What site controls apply?

A robots.txt file expresses crawling rules. The Robots Exclusion Protocol places it at the service's top-level /robots.txt path and defines Disallow rules for matching URL paths. Combining that syntax with Diffbot's documented token gives this example requesting exclusion of all paths for Diffbot:

User-agent: Diffbot
Disallow: /

Diffbot's Crawl API overview states that Crawl follows disallow and crawl-delay directives by default. It explicitly says Crawl ignores Allow, so an Allow exception beneath a broader exclusion is unsuitable for expressing Crawl access.

The same overview describes robots.txt overrides for specific cases, typically a partnership or agreement with the website. The creation reference exposes this through obeyRobots, whose default is 1; setting it to 0 ignores the site's instructions. Site owners should account for that documented override when assessing this control's scope. The protocol itself states that robots.txt rules are not access authorization.

FAQ

How can you inspect what a crawl encountered?

If you operate the crawl, you can request type=urls from the crawl data endpoint. It returns a URL Report in CSV with diagnostics for every crawled URL, separate from the extraction output.

Can you provide a complete URL list instead of following links?

Diffbot's Crawl API overview directs clients with a complete list of URLs to the Bulk API. Crawl serves jobs that need to follow links from seed URLs.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer