Why can Apple's iTMS fetcher ignore robots.txt restrictions?

Definition: Apple’s iTMS user agent is a special fetcher for Apple Podcasts content. Apple documents that it does not follow robots.txt because it is not a general search crawler, and that it only crawls URLs associated with registered content on Apple Podcasts.

Key facts

• Apple identifies the fetcher with the request header User-Agent: iTMS in its About Applebot documentation.
• Apple says iTMS traffic may come from applebot.apple.com hosts.
• The documented scope is URLs associated with registered content on Apple Podcasts.
• Apple separately says that Applebot follows standard robots.txt directives in general search crawls targeted at Applebot.
• Apple’s podcast submission workflow accepts an RSS feed URL, and a validated feed makes its associated episodes available on Apple Podcasts, according to Submit a new show.

Why is iTMS outside the usual robots.txt rule?

robots.txt is the Robots Exclusion Protocol. The IETF specification describes it as a set of rules that crawlers are requested to honor when accessing URLs. It also states that these rules are not access authorization. A crawler normally uses its product token to select a matching group in the file, then applies that group’s rules.

Apple defines iTMS as a special Apple online properties user agent rather than as its general search crawler. Its documented behavior is therefore an Apple policy for this fetcher: iTMS does not consult the site’s robots.txt rules, while Applebot’s general search crawl does. The reason Apple gives is functional scope. iTMS retrieves URLs connected to content that has been registered on Apple Podcasts.

Apple’s identifier is:

User-Agent: iTMS

The header is useful when classifying requests in access logs. Apple says a user agent helps website operators identify crawler traffic and control access through robots.txt; for iTMS, the same documentation gives the specific exception above.

What makes a podcast URL registered?

Apple’s submission guidance says a show can be submitted through an RSS feed hosted by a third-party provider or through Apple Podcasts Connect. The workflow includes entering the RSS feed URL. Apple requires all shows to pass technical validations and a review process before they become available on Apple Podcasts. When the RSS feed validates, Apple says all episodes associated with that feed become available on Apple Podcasts.

The associated URLs are therefore part of a podcast publishing relationship, rather than URLs selected by an open web search crawl. Apple’s RSS feed requirements say that feed URLs must use RSS 2.0, be publicly addressable, and not be password-protected. The same page requires HTTP HEAD and byte-range requests for streaming playback. Each episode must have a unique <enclosure> with a URL, length and type that specify the episode location and host. Apple Podcasts ignores duplicate <enclosure> URLs.

What does a site owner control?

A robots.txt group aimed at Applebot controls Apple’s documented general search crawler. It does not create an opt-out for iTMS. For example, Apple’s general search example is:

User-agent: Applebot
Allow: /
Disallow: /private/

The Robots Exclusion Protocol says that robots.txt is not a substitute for valid content security measures. A server can therefore enforce access separately from crawler preferences, while robots.txt remains the documented control for crawlers that honor it.

Related terms

• Applebot
• robots.txt
• robots.txt User-agent directive
• Crawler IP verification

FAQ

Does User-agent: iTMS with Disallow: / stop the fetcher?

Apple documents that the iTMS user agent ignores robots.txt, so a matching group in that file has no effect on the fetcher.

Is iTMS the same as Applebot’s general search crawler?

Apple lists Search and Podcasts as separate user agents powered by Applebot. It describes iTMS separately and gives it a different robots.txt behavior.

How can a site identify an iTMS request?

Check the User-Agent value in the access log for iTMS. Apple also says that iTMS traffic may come from applebot.apple.com hosts.

Sources

1. Apple, "About Applebot", 4 September 2026
2. Apple Podcasts for Creators, "Submit a new show", undated
3. Apple Podcasts for Creators, "Podcast RSS feed requirements", undated
4. IETF, "Robots Exclusion Protocol", September 2022

---

Check how a page reads to answer engines. Open the GEO score analyzer