How do I edit robots.txt for a HubSpot-hosted domain?

Definition: A robots.txt file gives crawlers instructions about which website paths they may access. As of July 2026, HubSpot provides domain-specific editing in Settings > Content > Pages, on the SEO & Crawlers tab.

Key facts

• HubSpot's search exclusion documentation provides a domain selector and an Override default settings control for domain-specific rules.
• Google's November 2025 file creation guide states that rules apply within the protocol, host and port where the file is posted.
• Google's December 2025 robots.txt introduction explains that a blocked page's URL can still appear in search results when other pages link to it.

Where are the domain settings?

HubSpot's editing instructions specify this sequence:

1. The settings icon opens account settings; the sidebar path is Content > Pages.
2. Choose a domain to edit its settings selects the target domain. Override default settings, where necessary, replaces that domain's default robots.txt settings.
3. The SEO & Crawlers tab contains the Robots.txt editor.
4. Save, at the bottom left, saves the changes.

The selector also offers Default settings for all domains for shared settings across connected domains. HubSpot limits this control to content hosted on domains connected to HubSpot.

What goes in the editor?

A User-agent line identifies the crawler addressed by a rule group. A Disallow line identifies a path excluded from crawling. HubSpot's documented example uses /welcome:

User-agent: *
Disallow: /welcome

This is an example rule to adapt to the intended path. Google's syntax guidance requires paths relative to the root, beginning with /, and treats rules as case-sensitive. For example, /file.asp and /FILE.asp are different paths. Each directive occupies its own line, and each group starts with a User-agent line.

How is the saved file checked?

Google's testing instructions recommend opening the live file in a private browser window. For a placeholder host, the address is:

https://www.example.com/robots.txt

The contents should display at that host's root. Google's robots.txt report in Search Console checks files already accessible on the site. The same guide says Google's crawlers automatically find and use accessible robots.txt files.

For search removal, noindex is a directive instructing a search engine to exclude a page from its index. Google's robots.txt introduction identifies noindex as a method for keeping pages out of Google Search. A robots.txt block can leave the URL visible through links from elsewhere.

Related terms

• robots.txt
• noindex
• How do I prevent indexing of a HubSpot page?
• Can Google read noindex on a page blocked by robots.txt?

FAQ

Can I keep HubSpot site search working?

HubSpot's documentation requires a separate HubSpotContentSearchBot user-agent when you use its site search module, allowing the search feature to crawl pages.

Can I combine a crawl block with noindex?

HubSpot warns in the same documentation that a robots.txt block prevents search engines from seeing the noindex tag. If you use noindex, the page must remain accessible to the crawler so it can read the instruction.

Sources

1. HubSpot, "Prevent content from appearing in search results", 28 July 2026
2. Google, "How to write and submit a robots.txt file", 21 November 2025
3. Google, "Introduction to robots.txt", 10 December 2025

---

Check how a page reads to answer engines. Open the GEO score analyzer