Does KagiBot use rules written for Googlebot?

Definition: As of 17 October 2026, Kagi documents that KagiBot, its search engine's web crawler, follows Googlebot directives in robots.txt when no rule targets KagiBot and a rule targets Googlebot.

A robots.txt file contains instructions about which website paths crawlers may access. A group associates a crawler name in a User-agent line with the rules that follow it.

Key facts

• Kagi's About KagiBot page documents both respect for KagiBot-specific directives and the fallback to Googlebot directives when KagiBot rules are absent.
• The IETF's Robots Exclusion Protocol specifies the wildcard group, User-agent: *, as the fallback when no matching group exists.
• As of 17 October 2026, Google's robots.txt documentation states that its crawlers select the most specific matching user-agent group and ignore other groups.

What does Kagi's fallback example show?

Kagi's published example contains a wildcard block and a Googlebot group:

User-Agent: *
Disallow: /
User-Agent: Googlebot
Allow: /
Disallow: /private/

Kagi states that KagiBot fetches all pages except those under /private/ with this configuration.

How can a website set an explicit KagiBot policy?

Kagi's crawler instructions provide this KagiBot-specific example:

User-Agent: Kagibot
Allow: /
Disallow: /private/

The example permits crawling outside /private/. An explicit KagiBot group provides a place to maintain its crawling policy separately from Googlebot rules.

Under RFC 9309, section 2.2.2, Allow and Disallow indicate whether access to a matching path is permitted. The most specific matching path determines the applicable rule. Here, /private/ is more specific than /, so the exclusion applies within that directory.

What should a robots.txt review check?

RFC 9309, section 2.3 places the file at the service's top-level /robots.txt path. Section 2.2.1 requires case-insensitive matching of crawler product tokens, the names used to select groups, and combines rules from multiple matching groups. A review therefore needs to inspect every group naming the relevant crawler.

For Googlebot, Google's interpretation of robots.txt states that group order is irrelevant and crawler-specific groups are separate from wildcard groups. It also limits a file's scope to its host, protocol and port. These details matter when locating the Googlebot rules involved in Kagi's fallback: a file on one host does not set the policy for another host.

Related terms

• KagiBot
• Googlebot
• robots.txt
• robots.txt group precedence

FAQ

Is a wildcard block enough to exclude KagiBot?

Kagi's fallback example shows KagiBot following the Googlebot group despite a wildcard Disallow: /. If your file contains Googlebot rules and no KagiBot rules, review that Googlebot group too.

Where does the ordinary wildcard fallback come from?

RFC 9309, section 2.2.1 specifies the * group when no matching group exists. Kagi documents its additional Googlebot fallback on its crawler page.

Sources

1. Kagi, "About KagiBot", undated, accessed 17 October 2026
2. IETF, "RFC 9309: Robots Exclusion Protocol", September 2022
3. Google, "How Google interprets the robots.txt specification", updated 31 August 2026

---

Check how a page reads to answer engines. Open the GEO score analyzer