How do I include crawler requests in a Matomo log import?

Definition: A Matomo log import reads web server request records into analytics reports. As of October 2026, the --enable-bots option includes requests identified as bots and adds a visit custom variable containing their user-agent string.

Key facts

• Matomo's bot import FAQ states that the default importer excludes bot traffic and detects bots through the User-Agent field.
• The importer source assigns the visit custom variable Bot to matched bots and Not-Bot to other imported traffic when the option is enabled.
• Matomo's default-options guide warns that importing the same data again duplicates it at the destination.

Which command enables crawler requests?

Matomo's default-options guide locates the On-Premise importer at misc/log-analytics/import_logs.py. The following example adapts its command to enable bot traffic while leaving the other optional request categories at their defaults:

python3 /path/to/matomo/misc/log-analytics/import_logs.py \
  --url=https://analytics.example.com \
  --idsite=1234 \
  --enable-bots \
  /path/to/your/web/access.log

The URL, site identifier and file path are example values to replace with the destination Matomo installation, the intended site and the input log. The same guide documents --idsite as the destination selector and recommends importing into a test site first because the importer has no duplicate-data check.

Matomo's bot import FAQ states that enabling bots increases processing time because bot page views are sent to Matomo.

How does the option affect imported visits?

A user-agent string is the client identifier recorded in the log's User-Agent field. The importer's detection code converts this string to lowercase and searches it for entries in its bot-matching list. That list includes googlebot, crawl and spider. A matching request passes the user-agent check when --enable-bots is active.

The importer then attaches a visit custom variable, a named value associated with the visit. Its name is Bot for a match, and its value is the request's user-agent string. Other imported requests receive Not-Bot with their user-agent string as the value. These labels reflect the importer's string matching; an AI crawler's classification depends on the user-agent text recorded in the input log.

Other request filters still apply. The same code provides separate switches for static resources (--enable-static), HTTP errors (--enable-http-errors) and redirects (--enable-http-redirects). HTTP errors include status codes in the 4xx and 5xx ranges. Including failed crawler requests therefore requires the error switch alongside the bot switch.

How can an import be checked?

The importer source defines --dry-run as a trial run that inserts no tracking data into Matomo. Adding it to the command allows a trial before an actual import.

The script's Logs import summary reports successful requests and ignored requests, with separate counts for bots, HTTP errors, redirects and static resources. These are request counts. The Bot custom variable is attached at visit scope.

Related terms

• How do I import website access logs into Matomo?
• How do I include failed crawler requests in Matomo log analytics?
• Does Matomo's JavaScript tracker record search engine bots?

FAQ

Can I use the bot switch alone to import failed crawler requests?

The importer source gives HTTP errors their own filter. You need --enable-http-errors as well as --enable-bots to include bot requests returning error responses.

Can I rerun an existing import with bots enabled?

Matomo's default-options guide warns that repeated imports duplicate destination data. You can select a test destination with --idsite before importing the same log with different options.

Sources

1. Matomo, "Import additional data including bots, static files, and HTTP errors tracking", undated
2. Matomo, "How do I run the Log File importer script with default options?", undated
3. Matomo, "import_logs.py", 5.x-dev branch, undated

---

Check how a page reads to answer engines. Open the GEO score analyzer