How to See Which AI Bots Crawl Your Site

November 13, 2026 · Clever Bean

DRAFTED OVERNIGHT. APPROVED AND PUBLISHED BY DARIA BYKOVA. November 13, 2026

Blocking every AI bot is the common reflex and usually the wrong first move. The useful move is to find out which bots visit, then decide for each one whether its job helps you.

Your server access logs show it. Filter requests by user agent for GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, then check the IP addresses against each company's published list. You control access in robots.txt with one block per bot. It is a request that honest crawlers follow and others ignore.

How do I find out which AI bots are crawling my website?

Start with the one record every web server keeps: the access log. Each request lists a time, a URL, an IP address and a user agent string. AI crawlers announce themselves in that last field, so a search for their names tells you who came and how often.

Three places can hold that record:

  • Your hosting panel, usually under logs, raw access or statistics.
  • A content delivery network or firewall dashboard, if your site sits behind one. Many of these list bot traffic by name.
  • Your web developer or host, who can export the last 30 days on request.

A word on analytics: tools that run a script in the visitor's browser mostly miss crawlers, because a crawler that fetches the page text never runs the script. Logs are the more complete source.

How do I check AI crawler activity in my server logs or analytics?

If you can open the log file, one command lists the AI bots by request count. This example works on a standard Apache or Nginx log:

grep -iE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot" access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn

The output is a ranked list of user agent strings with a count beside each. Add the date range you care about and keep the first export as a baseline, so next month's numbers mean something.

On a managed platform such as Shopify or Squarespace you may have no raw log. In that case ask the platform what bot reporting it offers, or put a CDN in front that does.

Read your logs for AI bots in six steps

0 of 6 done

  • Details

    Ask your host or developer for the full file, not a summary. Summaries often group crawlers under one label.

  • Details

    Look for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User and PerplexityBot.

  • Details

    A bot that fetches your product pages is different from one that only reads your robots.txt file.

  • Details

    A run of 403 or 404 responses means the bot is knocking and being turned away or finding nothing.

  • Details

    Compare them with the lists the company publishes. Section below explains how.

  • Details

    Repeat monthly. The change from one month to the next tells you more than any single count.

Export 30 days of raw logs, search them for the AI bot names, count requests per bot and per page, read the status codes and check the IP addresses of the bots you care about. Save the first result as a baseline and repeat it every month.

What share of your requests come from AI bots?

AI bot share of all requests
2.5%
All other requests
97,500

The default numbers are an illustration, not a benchmark. Replace them with the counts from your own log.

Divide the requests from AI bot names by all requests in the same period to get your own share. The default figures are a model. Only the number from your log counts, and the trend across months matters more than the first reading.

Which user agents do GPTBot, ClaudeBot, PerplexityBot and Google-Extended use?

One company can run several bots with different jobs, and each has its own robots.txt token. OpenAI's crawler page (2026-10) lists OAI-SearchBot, which surfaces sites in ChatGPT search, and ChatGPT-User, which acts on a person's request. It also publishes IP addresses for its bots. GPTBot is the bot that guides on OpenAI crawlers (2026-08) tie to training opt-outs.

Anthropic runs three. A 2026-11 explainer describes ClaudeBot as the collector of pages that may train Claude models, with Claude-SearchBot for search indexing and Claude-User for fetches a person requests. Each can be allowed or blocked on its own. Older tokens such as Claude-Web and anthropic-ai are flagged as deprecated (2026-03), so a rule written only for them does not block the current ClaudeBot.

Google-Extended behaves differently. A bot reference (2026-08) describes it as a robots.txt control token and no fetching crawler, so it never shows up as a visitor in your log. Treat that point as secondary reporting and confirm it on Google's own crawler documentation before you rely on it.

PerplexityBot is the token that crawler guides (2026-06) list for Perplexity. Check Perplexity's own documentation for its current bots and IP list.

Four bot names, four different jobs

Operator
OpenAI
Job
Surfaces sites in ChatGPT search
Appears as a visitor in your log
Yes
robots.txt token
OAI-SearchBot

GPTBot and ClaudeBot relate to training. OAI-SearchBot relates to search results in ChatGPT. Google-Extended is a control token that, per secondary reporting, never visits your site itself. Each has its own robots.txt token, so you can treat them separately.

How do I block AI training bots using robots.txt?

Robots.txt is a plain text file at the root of your domain, at yourdomain.com/robots.txt. Add one block per bot token and a Disallow: / line to refuse the whole site:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

To exclude only part of the site, replace / with a path such as /journal/. Save the file, open the URL in a browser to confirm it loads, and check your log a week later to see whether the bot still fetches pages. Changes take effect when the bot next reads the file, so expect a delay.

Can I allow AI search crawlers while blocking AI training crawlers?

Yes, because the companies separate the two jobs into separate tokens. This file refuses training collection and keeps the search bots in:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

The trade is concrete. A search bot that is allowed can put your page in an answer with a link. A search bot that is blocked cannot. Owners who sell products usually want the first outcome, because a citation is how a buyer finds the brand. Whether to refuse training is a separate call about how you feel about your photos and copy being used to build models.

User-initiated bots need a decision too. OpenAI's page says that because a person starts ChatGPT-User actions, robots.txt rules may not apply (2026-10). Writing a rule for it may change nothing.

Does robots.txt actually stop AI companies from using my content?

It stops the companies and bots that choose to read it. The file is a request. Nothing in it enforces anything, and a crawler that ignores it can still fetch every public page.

Three limits are worth stating plainly:

  • Robots.txt controls future crawling. Content collected before you added the rule is outside its reach.
  • It governs the bot that fetches your page. Your text can also reach a model through a third party that copied it, such as a forum post quoting your product description.
  • Compliance is each company's own claim. Anthropic, for example, documents robots.txt as the way to opt out of its bots, according to the 2026-11 explainer. Your log is the only way to see whether a bot behaves as documented on your site.

If a page must stay private, put it behind a login. A public URL with a robots.txt rule is still a public URL.

What did the VerticalScope news of 2026-11-11 change?

On 2026-11-11 PYMNTS reported that VerticalScope signed data agreements with two frontier AI labs for content from an archive of more than 2 billion user-generated posts, with revenue expected to begin in the fourth quarter. According to the same report, one of the two arrangements pays VerticalScope when its content contributes to an AI-generated answer. The report also says AI shopping recommendations helped lift affiliate commerce revenue 29% across more than 1,200 forums.

Our opinion, in plain sentences. Content that AI answers draw on now has a buyer, and the owner sets the terms. A publisher with 2 billion posts negotiates a contract. An owner of a product brand has a robots.txt file and a log. The principle is the same: decide which bots may use your pages and on what terms, and do it before a bot decides for you.

For a product brand the practical reading is narrower than a licensing deal. Your product pages are what you want AI answers to cite. Your photographs and long guides are what you may want to keep out of training. Those two goals map onto the separate search and training tokens above.

How do I set up llms.txt or meta tags to control AI use of my content?

Llms.txt is a proposed plain text file at the root of a site that points AI systems to your most useful pages. It works as a reading list. We found no company page that names it as a permission control, and the OpenAI crawler page we cite lists robots.txt tags as the way to manage access. Add one if you like, to make your key pages easier to find, and keep your access rules in robots.txt.

Meta tags split into two kinds. The standard robots meta tag, such as noindex, tells search engines how to treat a page, and it works on the search side. Tags aimed at AI training, often written as noai, are a convention we could not confirm on any primary company page, so we do not rely on them. If a page needs a firm barrier, use a login or a server rule that returns an error for the bot's verified address range.

How do I verify that a bot claiming to be an AI crawler is genuine?

A user agent string is text that any program can write. Anyone can send a request that says GPTBot. The IP address is much harder to fake, so verification goes through it:

  1. Pick the log lines that carry the bot name and note their IP addresses.
  2. Open the company's published list. OpenAI publishes IP addresses for OAI-SearchBot at openai.com/searchbot.json, as its crawler page (2026-10) states.
  3. Check whether each address sits inside a listed range.
  4. For bots without a published list, run a reverse DNS lookup on the address, then a forward lookup on the name that returns. A genuine crawler resolves back to the same address under the company's domain.
  5. Block or rate-limit the addresses that fail, in your firewall or CDN, since robots.txt will not stop an impostor.

An impostor tends to show up as a burst of requests under a famous name from an address outside every published range. That pattern is worth a firewall rule.

Will blocking AI training bots hurt my Google rankings?

Blocking Google-Extended does not change how Google's regular search crawling treats your site, according to secondary reporting we checked. Confirm this on Google's own documentation before changing the file, and watch your search traffic for a month afterwards.

Can I see what people ask ChatGPT about my business?

No. Nobody outside the AI companies can read live ChatGPT queries. Your log shows which bots fetch your pages, and you can test what assistants say about your business by asking them yourself.

How often should I check my logs for AI bots?

Monthly is enough for most sites. Compare each month with the baseline you saved and look for new bot names, new IP ranges and large changes in volume.

  1. VerticalScope Turns 2 Billion Forum Posts Into AI Shopping Revenue (PYMNTS) 2026-11
  2. Overview of OpenAI Crawlers (OpenAI) 2026-10
  3. What Is ClaudeBot? Anthropic's 3 Crawlers (SEOmator) 2026-11
  4. GPTBot & OAI-SearchBot: robots.txt Guide (icoda) 2026-08
  5. How to Block AI Bots in robots.txt (Mersel AI) 2026-03
  6. Google-Extended: User-Agent, Robots.txt and Verification (Trakkr) 2026-08
  7. AI Crawlers Explained: GPTBot, ClaudeBot, PerplexityBot (Anagram) 2026-06

Take away three things. The log tells you who visits. Robots.txt states your rules for the bots that listen. The business decision is which bots earn access because they send buyers your way. If you would rather see how your business appears in Google and in five AI assistants before touching any file, run the free check. It is free and arrives the same day.