Which AI Bots Should Crawl Your Store, and How to Block the Rest

November 17, 2026 · Clever Bean

DRAFTED OVERNIGHT. APPROVED AND PUBLISHED BY DARIA BYKOVA. November 17, 2026

Most store owners treat AI bots as one group and block them all or allow them all. Both choices cost money, because the bots do three different jobs.

Split bots by job. Allow search bots and user-triggered agents, because they send buyers to your store. Decide separately on training crawlers. Put the rule in robots.txt for bots that obey it, then enforce at the CDN with verified identity for the rest, since robots.txt asks politely and enforces nothing.

Which AI crawlers and agents should I allow on my online store?

Allow any bot whose job is to find your products for a buyer. That means AI search crawlers and the agents that fetch a page because a shopper asked about it. These are the bots that decide whether your store can appear in an answer.

Be selective with training crawlers. They collect pages to improve a model and send no visit back. Whether that matters depends on what you sell. A brand with original product photography, written guides and a distinct voice has more to protect than a store of standard catalogue items.

Keep every bot away from cart, checkout and account pages. No bot has a reason to be there.

What is the difference between AI training bots, search bots and user-triggered shopping agents?

The difference is who asked for the visit and what happens to the page afterwards.

Three kinds of AI bot visiting your store

Why it visits
Indexes pages so an assistant can cite them
Example user agents
OAI-SearchBot, Claude-SearchBot, PerplexityBot
Sends buyers to you
Yes, through citations and links
Usual stance for a store
Allow
Obeys robots.txt
Major operators say yes

AI bots fall into three groups. Training crawlers collect pages for models and send no visit back. Search crawlers index pages so assistants can cite them, and a store usually wants them. User-triggered agents fetch a page because one shopper asked, and they are the closest thing to a customer walking in. Each operator publishes its own user agent names, so check the current list before you write rules.

How do I block GPTBot, ClaudeBot or other AI crawlers in robots.txt?

Add one group per bot to the robots.txt file at the root of your domain. Match the user agent name exactly as the operator publishes it. This example blocks training crawlers, keeps search and shopper bots allowed and protects the pages that hold customer data:

# Block training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

# Keep search and shopper-triggered bots welcome
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

# Keep customer data out of reach for everyone
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account

Google publishes a separate control token for its AI model training, so you can keep Google Search crawling while opting out of that use. Check that token in Google's own documentation before you add it.

Test the file after every change by opening yourdomain.com/robots.txt in a browser. A typo in a wildcard group can block Google by accident.

Does blocking AI bots in robots.txt stop AI shopping agents from browsing my store?

Only the ones that choose to obey it. Robots.txt is a published request. Reputable operators follow it for their crawlers. An agent that drives a real browser on a shopper's behalf can look like an ordinary visitor, and a scraper can ignore the file entirely.

So treat robots.txt as your statement of policy and the firewall as your enforcement. You need both.

How do I block AI bots at the firewall or CDN level without hurting Google indexing?

Write rules that name the bot you want to stop and leave search engines alone by default. A safe order:

  1. Allow verified Google and Bing crawlers first, using the verification methods those companies publish.
  2. Block the user agents of training crawlers you chose to refuse.
  3. Rate limit unknown automated traffic on product, search and cart pages.
  4. Challenge traffic that claims a known bot name but fails verification.

Never block by guesswork on IP ranges you copied from a forum. Ranges change, and a stale list can lock out a search engine for weeks before anyone notices. After each rule change, inspect a few product URLs in Google Search Console and read your server logs for a day.

How do I allow trusted AI shopping agents while still blocking scrapers?

Allow by verified identity, not by name. A user agent string is text anyone can type. Three signals are stronger:

  • The visiting IP address belongs to a range the operator publishes.
  • A reverse DNS lookup of the IP resolves to the operator's domain and a forward lookup returns the same IP.
  • The request carries a cryptographic signature the operator documents, where the operator supports it.

Many CDNs now maintain verified bot lists and apply these checks for you. Use them where available, then add your own limits: read-only access, no cart or checkout paths and a rate cap per IP.

Which stance fits your store?

Question 1 of 3

How much of your catalogue is original (your photos, your descriptions, your guides)?

The right stance depends on three things: how much of your content is original, whether assistants already mention you and whether you read your own logs. Original content and an informed eye on logs favour selective blocking with CDN enforcement. Stores with mostly supplier copy have less to protect and more to gain from staying visible to search and shopper bots.

Will blocking AI crawlers reduce how often ChatGPT or other assistants recommend my products?

Blocking a search crawler can remove you from the pool of pages that assistant cites. Blocking a training crawler works differently: it keeps your pages out of future training data, and no one outside the operators can say how that changes recommendations. Nobody can read live ChatGPT queries or measure the effect of a single rule, so any promise on this point is a guess.

A sensible test is to change one thing at a time, then ask the same product questions in the assistants each month and keep notes. That gives you your own evidence.

What did the news of 15 November 2026 change, and what do we think?

A news brief published on 15 November 2026 put AI agents and site access back in front of store owners. Our view, as of 17 November 2026, is plain. Headlines will keep changing which bots exist and what they are called. The decision you make does not change: sort bots by job, allow the ones that bring buyers, and confirm identity before you trust a name. Rules written around categories and verification survive new bots. Rules written around a list of names go stale within months.

How do I verify that a bot claiming to be an AI agent is genuine?

Compare what the request says with what you can check. Start with the logs, then run the steps below on any visitor that matters.

Verify a bot in five steps

0 of 5 done

  • Details

    Note the IP address, the user agent string, the time and the pages requested.

  • Details

    Use the list on the operator's own documentation page. Ignore third-party copies.

  • Details

    The reverse lookup should name a domain owned by the operator. A forward lookup of that name should return the same IP.

  • Details

    Signed requests are the hardest to fake. Your CDN may check them for you.

  • Details

    A genuine shopper agent reads a few pages. A fake one requests thousands of product URLs in minutes or visits cart and checkout paths.

To verify a bot, find it in your logs, match its IP to the operator's published ranges, confirm it with reverse and forward DNS and check a signature if the operator offers one. Then look at behaviour: real shopper agents read a handful of pages, and fakes hammer product URLs or poke at checkout.

Should I block all AI crawlers to protect my content?

A full block also removes you from AI search results. Block training crawlers if you want to protect original content, and keep search and shopper bots allowed.

Can I block AI bots without touching Google Search?

Yes. Write rules for named AI user agents and leave Googlebot alone. Test a few product URLs in Google Search Console afterwards.

Where does the robots.txt file go?

At the root of the domain, for example yourdomain.com/robots.txt. Subdomains need their own file.

How often should I review bot rules?

Quarterly is a reasonable rhythm, plus whenever your logs show a new automated visitor you cannot explain.

  1. AI Brief - November 15, 2026 - Sam on AI 2026-11
  2. List of Top AI Search Crawlers + User Agents (October 2026) 2026-10

Your decision comes down to three questions: does this bot send me buyers, does it take content I would rather keep, and can I prove who it is. Allow the first group, decide on the second with your catalogue in mind and enforce the third at the CDN. Results take time and no setting guarantees a recommendation, so review your logs each quarter.

To see how your store appears in Google and in five AI assistants today, and the three things to fix first, run the free check.