Is Your Firewall Blocking AI Shopping Agents? How to Check
November 6, 2026 · Clever Bean
DRAFTED OVERNIGHT. APPROVED AND PUBLISHED BY DARIA BYKOVA. November 6, 2026
You pay for bot protection to keep scrapers and fraud out. The same tool may also be turning away the AI agents that shoppers now use to compare products, and it will not tell you.
Check your firewall's blocked-request log and your server logs for requests from AI assistants, then look at the status codes they received. A 403, 429 or challenge page means your bot protection is turning them away. Allow them by verifying each request against the vendor's published details, and keep admin, cart and checkout closed.
What did the 5 November 2026 report find about bot protection and AI shopping agents?
On 5 November 2026 PPC Land reported on an audit by Blend Commerce. Security software at 29 of 200 UK and US retailers refused its shopping agent on two separate attempts, which is 14.5% or one in seven. The article says the robots.txt file could be read at 24 of those 29 stores, and every readable file allowed the search bots of ChatGPT, Perplexity and Claude.
Our opinion, in plain sentences. The audit measures one agent on two attempts, so read it as a signal and not as a census. The part worth acting on is the mismatch. A robots.txt file is a polite request that someone in marketing can edit. A firewall rule is enforcement that someone in security or a hosting provider set up, often years ago or by default. When the two disagree, the firewall wins and nobody is told. If you sell through a store platform with bot protection switched on, assume you have the same gap until your own logs say otherwise.
Share of retailers whose bot protection refused the test shopping agent
Source: Bot protection blocks Blend's AI shopping agent at 29 of 200 retailers, 2026-11
In the audit, 14.5% of 200 UK and US retailers refused the test agent. Large retailers refused it at a 20% rate and emerging ones at 2.5%. The more security tooling a store runs, the more likely it is to turn agents away, so brands with a managed firewall should check first.
How do I tell if Cloudflare, AWS WAF or another bot protection tool is blocking AI shopping agents?
Open the security event or firewall log in your protection tool and filter for actions such as block, challenge or rate limit. Then check whether any of those events carry an AI assistant's user agent. If they do, your tool is turning those agents away, and the log entry names the rule that fired.
Three places give you evidence, and they catch different problems:
- The firewall or bot dashboard. Most tools list blocked and challenged requests with the rule name, the country, the user agent and the path. Sort by rule to see which one fires most.
- Your origin or CDN access logs. Look at the status code. A 403 or 429 is a refusal. A 200 with a short page that says "verify you are human" is also a refusal, and it hides in the logs because the code looks healthy.
- A direct test. Fetch one product URL the way an agent would and read what comes back. The last section of this guide walks through it.
The tool name matters less than the rule. Cloudflare, AWS WAF and others all have a mode that treats automated traffic as suspect by default. Find that setting and read who owns it.
How can I see AI agent requests in my server or CDN logs?
Search the user agent field of your access logs for the names of the assistants you care about, then count the status codes they received. If your server writes the common combined log format, one command does it:
grep -iE "chatgpt|claude|perplexity" access.log | awk '{print $9}' | sort | uniq -c | sort -rn
The output is a list of status codes with a count beside each. Your format may put the status code in a different field, and a hosted store may only give you logs through a CDN or platform export. The idea stays the same: filter by user agent, group by status.
Two limits apply. First, you will see that a fetch arrived and what it asked for, but nobody can read the live questions shoppers type into ChatGPT, so you cannot see what the shopper wanted. Second, a request that your firewall dropped before it reached your server will only appear in the firewall log, so check both. Put your own counts into the calculator below.
Your AI agent block rate
- Share refused
- 15%
- Requests that got through
- 170
Use one period, for example the last 30 days, for both numbers. Include firewall log entries as well as server logs.
Divide the AI assistant requests that were refused by the total you found in your logs. Any share above zero on public product pages is worth a look at the rule that fired. If you find no AI requests at all, either they are blocked before your server sees them or no assistant has fetched your pages yet.
Which user agents do ChatGPT, Claude and Perplexity shopping agents send?
Each of these vendors documents its own bot names and the IP addresses it uses, and those details change. Copy them from the vendor's current documentation and not from a blog post, including this one. A list pasted into a firewall in 2026 can be wrong by 2027.
When you read the documentation, sort each bot by purpose and decide per bot:
- A bot that indexes pages so the assistant can cite you in answers.
- A bot that fetches a page because a person asked the assistant about it.
- A bot that collects pages to train models.
The first two put your products in front of buyers. The third is a separate choice about your content, and you can allow the first two while disallowing the third. The audit from 5 November 2026 matters here too: the refusing stores allowed the search bots in robots.txt and still got blocked by firewall rules, so the name on the bot is only half of the story.
Should I allowlist AI shopping agent IP ranges or user agents in my firewall?
Allow on both together. A user agent is a line of text that anyone can type into a script. An IP range proves where the request came from but goes stale when the vendor changes it. A rule that needs both to match is stricter than either one and still lets the real agents in.
Three ways to write the allow rule
- What it proves
- The request claims a name and comes from the vendor's address
- Can a scraper fake it
- Only by using the vendor's network and the name
- Upkeep
- Same refresh as the IP list
- Risk if it goes stale
- Real agents get blocked again, and the log shows why
Allowing by user agent alone lets any scraper pass by copying a name. Allowing by IP range alone works but breaks when the list changes. Requiring both matches is the safest of the three, at the cost of refreshing the vendor's published list on a schedule you set.
How do I verify that a request really comes from an AI agent and not a scraper?
Take the source IP of the request from your log and check it against the list the vendor publishes for that bot. If the address is on the list and the user agent matches, treat it as the agent. If the user agent says ChatGPT and the address is somewhere else, it is a scraper using a borrowed name.
Build the check into a routine:
- Export the AI-labelled requests from your logs once a month.
- Compare each source IP with the vendor's current list.
- Count the mismatches. Mismatches are impostors, and they should stay under your normal bot rules.
- Refresh the allowlist from the vendor's page at the same time.
Some vendors are also moving toward cryptographically signed requests, so check each vendor's documentation for what they offer today. Where signing exists, it beats an IP list, because an address can be shared and a signature cannot be copied.
Will allowing AI shopping agents hurt my site's security or expose checkout data?
Letting an agent read your public product pages exposes nothing a visitor with a browser could not already see. The risk sits in how broad the rule is. A rule that says "skip all protection for anything labelled AI" is dangerous. A rule that says "skip the bot challenge on product and collection URLs for verified agents" is narrow and safe.
Keep these guardrails in place:
- Scope by path. The allow rule applies to product, collection and content pages only. Admin, login, account, cart and checkout keep full protection.
- Keep attack rules on. Skipping a bot challenge does not mean skipping the rules that catch injection attempts and other attacks.
- Keep rate limits. A real agent should not need thousands of requests a minute. Set a ceiling and watch for agents that hit it.
- Treat buying as a separate decision. Reading a product page is one thing and completing a purchase for a shopper is another. Decide that one on purpose, later, with your payment provider. Until you have a reason, leave checkout closed.
Checkout data belongs to a session and a customer. An agent that never reaches those paths never touches that data.
How do I set robots.txt and bot rules so AI agents can browse product pages but not admin or checkout?
Use both layers and make them say the same thing. In robots.txt, allow the public site and disallow the private paths. In the firewall, enforce the same split, because robots.txt is a request that well-behaved bots follow and a scraper ignores.
A starting robots.txt looks like this, with your own paths in place of the examples:
User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /checkout
Disallow: /account
Allow: /
Path names differ between store platforms, so check yours. If you want different rules for one vendor's bots, add a separate block for that bot's name, copied from its documentation.
Then mirror it in the firewall: one rule that skips the bot challenge for verified agents on public paths, and one rule that keeps full protection on the private paths whoever asks. Write down who owns each file. The audit from 5 November 2026 found stores whose robots.txt was open and whose firewall was shut, and a short note saying who owns which file prevents that.
How do I test that AI shopping agents can reach my product pages after changing firewall rules?
Fetch a product URL with a command-line client or a testing tool, read the status code and the body, and then read your logs for the same request. A real fetch returns 200 with your product HTML. A 403, a 429 or a page asking to verify you are human means the rule still blocks.
One trap: a test from your own laptop that copies an agent's user agent will be refused by a rule that also checks the IP address. That refusal is the rule working. To test the verified path, use the vendor's own tools where they exist, or ask an assistant to open one of your product URLs and then look for the request in your logs.
Allow AI agents safely and retest
0 of 6 done
Details
Filter your firewall log for block and challenge actions on AI user agents and note the rule name and owner.
Details
Take user agent names and IP lists from each vendor's current documentation and note the date you copied them.
Details
Match user agent and IP range together, limited to product, collection and content paths.
Details
Admin, login, account, cart and checkout stay protected and disallowed in robots.txt.
Details
Fetch a product URL, check the status code and body, then find the request in your logs.
Details
Compare AI-labelled source IPs with the vendor list, count mismatches and refresh the allowlist.
The sequence is: find the rule, copy vendor details from the source, allow narrowly, keep private paths closed, retest from outside and review monthly. Each step is a log search or one rule change. Skipping the retest is how a fix looks done while agents still get refused.
If my robots.txt allows AI bots, am I safe from being blocked?
No. Robots.txt is a request to bots, while your firewall enforces. The 5 November 2026 audit found stores whose readable robots.txt allowed the search bots of ChatGPT, Perplexity and Claude and whose bot protection still refused the test agent.
Can I see what shoppers ask ChatGPT about my products?
No. Nobody outside the vendor can read live ChatGPT queries. Your logs show that a fetch arrived and which page it asked for, and that is the extent of it.
Does allowing AI shopping agents guarantee more sales?
No. It removes one reason an assistant cannot read your pages. Whether an assistant recommends you depends on many other things, and results take time.
Should I allow AI training bots too?
That is a separate choice about your content. You can allow bots that fetch pages for shoppers and answers while disallowing bots that collect pages for training, using each vendor's documented bot names.
Your firewall and your robots.txt are usually owned by different people, and AI agents get stopped in the gap between them. Read your logs once, fix the rule that fires, verify who you let in, keep checkout closed and retest. Then look at the other half: whether the assistants describe your business correctly. The free check shows how your business appears in Google and in five AI assistants, the same day, with three things to fix first: https://cleverbean.ai/#check