In short. HubSpot documents robots rules, access responses, URL and DNS problems, and timeouts as causes of SEO crawl failures.
Definition: A HubSpot SEO crawl failure means that an SEO crawler fails to index a page. HubSpot attributes these failures to HTTP responses, robots.txt rules or retrieval problems, page access restrictions, URL or DNS problems, and crawler allow-listing.
Key facts
- HubSpot reports status 301 and 302 when a redirect prevents access, status 403 when the server is reachable but denies access, and status 404 when the crawler fails to find a live page in its SEO crawling errors documentation.
- For an external page, HubSpot labels a robots rule as “Scan blocked by robots.txt file” and says to add the user agent “HubSpot Crawler” as an exemption in the same documentation.
- HubSpot says that a robots.txt file must be accessible in the site's top-level directory for its crawlers to retrieve it, while the Robots Exclusion Protocol defines the file at
/robots.txtin the service's top-level path. - HubSpot's troubleshooting steps include checking the URL, confirming that the page is live, verifying DNS resolution, and asking the site administrator to allow “HubSpot Crawler” in the crawler error guide.
Which responses and restrictions stop a scan?
HubSpot's SEO crawling errors documentation says its SEO tools display a crawling error when an SEO crawler fails to index a page. For HubSpot-hosted content, the documented messages identify 301 Moved Permanently and 302 Object moved as redirects that prevent access. A 403 Forbidden response means the server can be reached but access to the content is denied. A 404 Not Found response means the crawler fails to find a live version because the content was deleted or moved.
For an external page, HubSpot also names a timeout when an audit targets the root domain instead of the subdomain connected to HubSpot. Its documentation lists a noindex meta tag in the page head as another reason an external SEO tool may fail to crawl or index a page. Private or password-protected pages are not crawled, so the noindex tag remains unreadable.
How can robots.txt block HubSpot's crawler?
HubSpot's search exclusion documentation describes two robots-related failures. “Scan blocked by robots.txt file” means the external page is excluded from indexing by its robots.txt rules. “Robots.txt file couldn't be retrieved” means HubSpot's crawler fails to access the file. HubSpot says the file should be accessible in the top-level directory and recommends asking the site administrator to add “HubSpot Crawler” to the allow list as an exemption.
HubSpot describes User-agent as the field that identifies the search engine or web bot covered by a rule, and Disallow as the field that prevents crawling of a URL slug. Its example has this form:
User-agent: *
Disallow: /url-slug
The Robots Exclusion Protocol defines the matching model behind these fields. A crawler looks for a product-token group, falls back to the * group when no specific group matches, and uses the most specific matching path. The standard requires the file at /robots.txt, says that a successful download makes the crawler follow parseable rules, and recommends following at least five consecutive redirects when fetching the file.
What should be checked when the error mentions access or connection?
Start with the exact URL reported by HubSpot. The product documentation says to verify that the URL was entered correctly, that the page is currently live, and that DNS can resolve the URL. It also tells site administrators to add “HubSpot Crawler” to the allow list when the crawler fails to scan a specific URL.
The robots file itself can create a separate failure from the page response. RFC 9309 classifies a robots.txt response in the 400 to 499 range as unavailable to the crawler and permits access to resources in that case. A server or network error in the 500 to 599 range makes the file unreachable; the standard requires the crawler to assume complete disallow in that situation. The standard also permits caching the file and recommends using a cached version for no more than 24 hours unless the file is unreachable.
Related terms
- robots.txt
- robots.txt caching and error handling
- robots.txt User-agent directive
- robots.txt Disallow directive
FAQ
Can a noindex tag cause a HubSpot scan problem?
Yes. HubSpot's SEO crawling errors documentation lists a noindex meta tag in the head HTML as a cause that can prevent an external SEO tool from crawling or indexing a page. You also need to keep the page accessible so the crawler can read that tag.
Does allowing HubSpot Crawler resolve every failure?
No. Allow-listing addresses the robots exclusion case HubSpot documents, while a bad URL, missing live page, DNS failure, redirect, HTTP access response, or timeout has a different cause.
Where must robots.txt be available?
The file belongs at /robots.txt in the service's top-level path under RFC 9309. HubSpot likewise tells site owners to make the file accessible in the top-level directory.
Sources
- HubSpot, "Understand SEO crawling errors", January 11, 2026
- HubSpot, "Prevent content from appearing in search results", July 28, 2026
- IETF, "Robots Exclusion Protocol", September 2022
Check how a page reads to answer engines. Open the GEO score analyzer