AI Crawlers: Whom to Let In, Whom to Block, and Why It Matters

Every site owner now makes a choice nobody asked for: block the AI crawlers and protect the content, or open the door and exist in the answers.

Streams of green code on a dark screen

Somewhere in your server logs, machines with names like GPTBot, ClaudeBot and PerplexityBot are requesting your pages. Each visit poses a question you never asked to answer: is being read by an AI a theft of your content, or the only distribution channel that is still growing? The answer depends on what your site sells - and most owners decide by accident, through a robots.txt they have not read in years.

Training and retrieval are different doors

The single most useful distinction in this debate is the one vendors blur. Training crawlers - GPTBot, Google-Extended, ClaudeBot in its training role - collect text to teach future models; blocking them keeps your words out of the next model's weights, slowly and invisibly. Retrieval crawlers - OAI-SearchBot for ChatGPT Search, PerplexityBot, Bingbot feeding Copilot - fetch pages to answer a live question, today, with a citation and often a link. Blocking training is a philosophical stance with delayed effects. Blocking retrieval is deletion from the answers your buyers read this afternoon.

The decision, by business model

  1. You sell the content itself - journalism, research, data. Blocking training bots and negotiating licences is coherent; your words are the product, and models ingesting them is unpaid supply.
  2. You sell services or products the content promotes. Open everything. A citation in an answer is the new referral, and the content was always bait, not merchandise. Blocking here is guarding the flyer while closing the shop.
  3. You are mixed - a product with a substantial blog. Allow retrieval everywhere; decide training case by case. The blog exists to be found, whatever your stance on model weights.
  4. Whatever you choose, choose it. The worst position is the accidental one: a JavaScript-only site that blocks every AI reader unintentionally while its robots.txt says welcome.

Being worth reading once they are in

  • Serve real HTML. Most AI crawlers execute no JavaScript; a client-rendered site is a closed door regardless of policy.
  • Publish an llms.txt - what the company is, what it sells, where the substantive pages live. One fetch, machine-shaped, at the root.
  • Announce changes through IndexNow so Bing's index, which feeds ChatGPT Search and Copilot, hears about a new page in hours rather than weeks.
  • Keep the entity consistent everywhere, and date what you update - the groundwork covered in generative engine optimization.

The costs, stated honestly

Open doors have a price. Crawler traffic is real load - mostly trivial for a static site, worth rate-limiting for a heavy application. Your words will be paraphrased without a click more often than cited with one. And no policy is permanent: the bots, their names and their behaviour change quarterly, which is why the robots file deserves a yearly review with an owner, like any other contract. This is part of what a technical SEO and GEO engagement audits.

  • 2 kinds of crawler - training and retrieval - with opposite stakes
  • 1 accidental policy: a JS-only site blocks every AI reader silently
  • 1x/year how often the robots.txt deserves a deliberate review

Blocking a training bot is an opinion. Blocking a retrieval bot is an absence.

Frequently asked questions

If I block GPTBot, do I disappear from ChatGPT?

Not immediately, and not entirely. GPTBot feeds training; ChatGPT's live search uses OAI-SearchBot and Bing's index, so blocking GPTBot alone leaves you citable in search-mode answers. Block all three - GPTBot, OAI-SearchBot and Bingbot - and you disappear from both the model's memory and its lookups. Know which door each directive closes before closing it.

What exactly is llms.txt?

A plain-text file at your site's root, written for machines that read neither JavaScript nor sitemaps for meaning: what the organisation is, what it offers, and a curated list of the pages that matter, with one-line descriptions. It is an emerging convention rather than a standard, it costs an hour to produce, and it is the cheapest clarity you can hand an AI reader.

Can AI crawlers overload a site?

They can be heavy-handed - Perplexity and the training bots have both drawn complaints for aggressive fetching. A prerendered or static site barely notices; an application that renders each page on demand should rate-limit by user agent rather than block. If a crawler ignores robots.txt, that is a firewall conversation, not a policy one.

More on this topic: SEO & Visibility.

Keep reading

Want this built for your business? See what we do.