In short. AI2Bot collects web content for open language model training, and Ai2 publishes its User-Agent string so website operators can filter or reject its traffic.

Definition: AI2Bot is Ai2's web crawler, an automated program that explores selected domains to collect web content for training open language models. As of October 2026, Ai2's crawling notice identifies its requests through a published User-Agent HTTP header.

Key facts

Ai2's crawling notice states that AI2Bot collects web content used to train open language models.
Ai2's Olmo 3 announcement describes publicly releasing training datasets and tools for processing them.
The IETF's Robots Exclusion Protocol defines robots.txt rules for crawlers and specifies HTTP authentication as an example of access protection.

What does collection for open models mean?

Ai2's Olmo 3 announcement, published on 20 November 2025, gives context for its open-model work. It describes making collections of training material, called datasets, available for download and releasing open-source tools to process them.

The announcement describes training-data processing that includes removing duplicate material and filtering for quality. Collection is therefore one step in a broader preparation process. These details describe Ai2's model-development work; the AI2Bot notice itself specifies collection for open language model training.

How can a website recognize AI2Bot requests?

A User-Agent header is a request field containing the client's identification string. Ai2's technical details state that the crawler always uses this header when looking for documents:

User-Agent: Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler)

Ai2 explicitly permits using this string to filter or reject crawler traffic. For a website operator, this is the concrete identifier to look for in recorded request headers and the documented basis for a traffic-filtering rule.

How do collection policy and access controls fit together?

In its Olmo 3 announcement, Ai2 states that it follows established web standards when collecting training data and excludes sites that explicitly disallow collection, including paywalled content. The AI2Bot notice does not specify how this particular crawler processes robots.txt.

Under the IETF's protocol specification, robots.txt sits at the service's top-level path, /robots.txt, and uses the text/plain media type. Website operators who need to protect access can use HTTP authentication, the security mechanism named in the specification. Ai2's published User-Agent string separately supports filtering requests that identify themselves as AI2Bot.

FAQ

Can I reject AI2Bot traffic?

Yes. Ai2's crawling notice says you can use its published User-Agent string to filter or reject crawler traffic.

Does robots.txt protect private pages?

The IETF's security guidance specifies that robots.txt rules are publicly accessible and recommends application-level security measures such as HTTP authentication to control access.

Where can I raise a collection concern?

Ai2's contact page lists support@allenai.org for questions about models, datasets or research, and info@allenai.org for general inquiries.

Sources

Check how a page reads to answer engines. Open the GEO score analyzer