Which AI Crawlers Should a B2B Site Allow or Block?

B2B teams do not have to make one all-or-nothing choice about “AI bots.” Search discovery, user-requested page access, and model-development collection can involve different crawlers and controls. Decide separately for each purpose and content class. What should be discoverable? What may be fetched at a user’s request? Which uses are acceptable under your rights and policies? Then verify the current provider rules and the requests your site actually receives. A robots.txt directive expresses a crawler preference; it does not secure private material.

Which AI crawlers should a B2B site allow or block?

Allow or block crawlers according to their documented purpose, your content rights, and the discovery outcome you want—not simply because a user agent (the identifier a client sends with a request) is labeled an AI bot. A team may want search discovery while declining a model-training crawler, or may treat user-triggered fetches differently. These are separate policy decisions, and providers do not necessarily offer identical controls.

Start by separating public marketing pages and documentation from restricted customer areas, licensed material, and sensitive information. Robots directives can communicate preferences to crawlers that honor them, but they are not authentication. A page that must remain private needs a real access-control boundary, such as authenticated access, with appropriate server-side enforcement.

What do the documented crawler agents do?

Official documentation distinguishes agents used for search from agents associated with user-directed access or model development. The names and controls can change, so treat the table as a decision aid, not a permanent allowlist. Check each provider’s current documentation before changing production rules.

Provider	Documented agent	Stated purpose in provider documentation	Policy decision to verify
OpenAI	OAI-SearchBot	Surfaces websites in ChatGPT search	Whether search visibility is desired and whether the agent can reach the relevant public pages
OpenAI	GPTBot	Crawls content that may be used in training OpenAI foundation models	Whether this collection is acceptable for the content and rights involved
OpenAI	ChatGPT-User	May fetch pages in response to certain user actions; not automatic web crawling	Whether user-triggered access is acceptable; its robots handling differs from automatic crawling
Anthropic	ClaudeBot	Collects web content that could potentially contribute to model training	Whether potential model-development collection is acceptable
Anthropic	Claude-SearchBot	Improves search-result quality by navigating the web	Whether search crawling and its intended visibility are acceptable
Anthropic	Claude-User	Retrieves websites in response to user-directed requests	Whether that user-directed access is acceptable
Perplexity	PerplexityBot	Surfaces and links websites in Perplexity search; documentation says it is not used to crawl content for AI foundation models	Whether search surfacing is desired and the site permits the agent to reach public pages
Perplexity	Perplexity-User	May fetch pages to answer a user’s request; documentation says it is not used for web crawling or training collection	Whether user-requested access is acceptable; provider documentation says this fetch generally ignores `robots.txt`

These descriptions reflect the providers’ published crawler documentation reviewed for this draft. They describe stated agent purposes, not every product feature, contractual term, or downstream use. Review the linked documentation and any applicable agreements before turning a general policy into a rights decision.

How do OpenAI’s OAI-SearchBot, GPTBot, and ChatGPT-User differ?

OpenAI’s crawler documentation describes OAI-SearchBot as supporting discovery in ChatGPT search, GPTBot as crawling content that may be used to train its foundation models, and ChatGPT-User as fetching pages for certain user actions. OpenAI says its OAI-SearchBot and GPTBot robots settings are independent. It also notes that robots rules may not apply to ChatGPT-User because requests are initiated by users. Do not treat a setting for one agent as a switch for all OpenAI access.

That distinction matters when a company wants search visibility but has a separate policy for training-related collection. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they may still appear as navigational links. Confirm the current wording and eligibility guidance before making a visibility decision.

How do ClaudeBot and Claude-User differ?

Anthropic’s crawler documentation describes ClaudeBot as collecting web content that could potentially contribute to model training, while Claude-User retrieves pages for user-directed requests. Its current documentation also lists Claude-SearchBot, which navigates the web to improve search-result quality. These are distinct purposes, not one general “Claude crawler” setting.

A team evaluating access should consider each agent separately and review Anthropic’s latest instructions. User-directed retrieval is not the same as permission for model-development collection, and allowing one agent should not be interpreted as permission for every purpose.

What is PerplexityBot used for?

Perplexity’s crawler documentation says PerplexityBot is designed to surface and link websites in its search results and is not used to crawl content for AI foundation models. It describes Perplexity-User as a fetcher for user actions and says that fetch generally ignores robots.txt. This is a provider-specific distinction; it should not be generalized to other products or services.

If your policy depends on controlling user-triggered access, review the provider’s current explanation and consider how your own authentication and server controls behave. A robots rule alone may not produce the result a team expects for a user-requested fetch.

How should a SaaS team choose a crawler policy by content class?

Map each content class to its discovery goal, rights, and acceptable access method before choosing a crawler rule. The table below poses decisions; it is not a legal recommendation or a default allowance. Have the relevant rights owner review licensed or sensitive material, and use technical access controls wherever access must be restricted.

Content class	Search discovery desired?	User fetch acceptable?	Training/model-development collection acceptable?	Protection or next step
Public marketing pages	Decide whether each provider’s search discovery supports the site’s goals	Decide whether answering user requests with the public page is acceptable	Make a separate rights and policy decision	Apply provider-specific crawler preferences where appropriate; monitor actual access
Public product documentation	Decide which search agents should be able to discover it	Decide whether user-directed retrieval is useful and acceptable	Assess documentation rights and internal policy separately	Keep public pages distinct from authenticated support or customer content
Customer-only or account-specific material	Usually not a public discovery target; confirm the intended audience	Do not rely on a crawler preference to prevent access	Do not assume a robots directive makes it private	Require authentication and enforce authorization server-side; review caching and access paths
Licensed, partner, or otherwise restricted material	Check the license and the intended audience before enabling discovery	Check whether user-triggered retrieval is covered	Obtain the appropriate rights review before permitting collection	Use access controls where required and document the rights decision
Sensitive operational pages	Exclude from public discovery where appropriate	Do not permit unauthenticated access	Do not rely on crawler compliance	Protect with authentication, authorization, and suitable technical controls

There is a practical tension to resolve: blocking a provider’s search agent while expecting that provider’s search product to cite the site may be contradictory. Check the provider’s own eligibility guidance rather than assuming another agent or a user-triggered fetch will substitute.

How do you implement and verify the policy?

Implement purpose-specific preferences only after checking the provider’s current documentation, and protect restricted content with access controls rather than robots.txt. A rule in robots.txt is a crawler directive, not a reliable security boundary. Also, a permitted directive does not prove that a request reached the site successfully or that the expected page content was served.

Classify content and rights. Identify public, customer-only, licensed, and sensitive paths. Record the discovery goals and rights decisions for each class.
Check agent names and purposes. Review the current OpenAI, Anthropic, and Perplexity documentation. Confirm exact user-agent names, stated purposes, robots behavior, and any provider-specific search eligibility conditions.
Configure rules for intended paths. Have a technical owner review the robots.txt syntax, user-agent matching, rule precedence, and path coverage. Also review controls on the content delivery network (CDN) and web application firewall. Do not copy a generic rule without testing how it applies to the site.
Protect restricted content with access controls. Require authentication and authorization for nonpublic pages. Do not expose private information on the assumption that a crawler will honor a disallow rule.
Inspect verified requests and responses. Review logs for the claimed user agent and, where necessary, validate the request source using provider-published information. Check requested paths, status codes, redirects, and the content actually served. A user-agent string alone should not be treated as proof of identity.
Recheck documentation and outcomes. Revisit provider rules after material changes and periodically review access logs and visibility measures. Keep an audit trail of policy changes and their intended effect.

Google’s guidance for its own Search AI features is a separate case: it ties eligibility to Google Search requirements and Googlebot controls. That does not establish access by OpenAI, Anthropic, or Perplexity agents. The operational question is provider-specific: what did each system’s documentation say, what did your site serve to its requests, and what did you observe afterward? The related guide on Google indexing vs AI crawlers covers that distinction in more detail.

Keep crawler access controls separate from content-guidance files. The related explainer, “does llms.txt matter?”, addresses what such a file can communicate; it is not an access-control mechanism. If a site is changing infrastructure, the AI crawler site migration guide is a useful companion for checking policy during releases.

Will allowing a bot guarantee citations or leads?

No. Provider documentation describes intended crawler functions and controls; it does not establish that allowing an agent will generate incremental citations, referrals, qualified leads, or sales. Treat these as outcomes to measure, not results to promise.

For a defined test period, track verified crawler fetches alongside repeated citation observations, referral traffic, and qualified outcomes. Record the pages, queries, and measurement window. Sparse observations and other changes—such as content updates or shifts in demand—can confound comparisons. A site-level test may inform a decision, but it does not automatically prove causation.

What is the practical policy takeaway?

Choose crawler access by provider, purpose, content class, and rights. Use robots directives to express preferences to crawlers, and real access controls to protect restricted material. Then verify requests and served content, and measure visibility and business outcomes without assuming that permission leads to citations.

Frequently Asked Questions

Should I block AI crawlers?

Not as a group by default. Decide separately whether each provider’s search, user-requested access, or model-development collection is acceptable for each content class, and use authentication and authorization—not robots.txt—to protect private material.

How Nortav Can Help

For B2B teams setting crawler rules, the decision turns on provider purpose, content rights, and the difference between crawler preferences and real access controls. Any production policy still needs current provider documentation, technical review, and rights review.

Our approach starts with understanding your current search visibility, identifying gaps against competitors, and developing a strategy focused on driving qualified traffic, generating leads, and delivering measurable business value.

Book a Discovery Call →