Free AI Bot Traffic Checker
Find out which AI crawlers can read your site — for training, for AI search citations, and for live user requests in ChatGPT, Claude and Perplexity. No signup.
Check if GPTBot, ClaudeBot and PerplexityBot can crawl your site
We read your robots.txt and request your homepage once as each AI crawler that has a published user agent.
Why “allowed in robots.txt” isn’t the same as “reachable”. robots.txt is a request, not a lock. Many sites allow GPTBot in robots.txt while a Cloudflare rule, bot-fight mode or hosting firewall returns a 403 to it. The result is the same as blocking: the crawler can’t read the page. That’s why this checker tests both.
Can AI crawlers reach you? Good. Now find out if AI actually recommends you. We ask ChatGPT, Claude and Perplexity the questions your customers ask, and report whether your brand is mentioned, cited or ignored, and who gets recommended instead. Plans: $4.99 / $7.49 / $9.99.
GPTBot user agent and official IP ranges
Every major AI company publishes the user-agent token its crawlers send and, for most of them, a JSON list of official IP ranges. Match on the token (for example GPTBot), not the full string. Version numbers like GPTBot/1.4 change over time, so use a wildcard or substring match, never an exact string. Because anyone can copy a user agent, the IP list is how you confirm a request really came from that vendor. The table below lists each bot’s token, its purpose, and links to the vendor’s own documentation (checked 18 October 2026).
| Crawler | Company | Purpose | Official user agent | Official IP list |
|---|---|---|---|---|
GPTBot | OpenAI | Training crawlers | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbotVendor docs | openai.com/gptbot.json |
ClaudeBot | Anthropic | Training crawlers | No user agent of its own / see vendor docs Anthropic links one IP list for all of its bots. Vendor docs | claude.com/crawling/bots.json |
Google-Extended | Training crawlers | No user agent of its own / see vendor docs robots.txt token only. Google says it has no separate HTTP user agent; fetching is done by existing Google crawlers. Vendor docs | Not published | |
CCBot | Common Crawl | Training crawlers | CCBot/2.0 (https://commoncrawl.org/faq/)Vendor docs | index.commoncrawl.org/ccbot.json |
Applebot-Extended | Apple | Training crawlers | No user agent of its own / see vendor docs robots.txt token only; Apple says it does not crawl. It controls whether Applebot-crawled content may train Apple models. Vendor docs | Not published |
Meta-ExternalAgent | Meta | Training crawlers | meta-externalagent/1.1Product token shown as documented by Meta; no official IP list. Vendor docs | Not published |
MistralAI-Training | Mistral AI | Training crawlers | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Training/1.0; +https://docs.mistral.ai/robots)Vendor docs | Not published |
Amazonbot | Amazon | Training crawlers | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36Chrome/W.X.Y.Z is a placeholder in Amazon’s docs. Vendor docs | developer.amazon.com/amazonbot/ip-addresses/ |
OAI-SearchBot | OpenAI | Search & citation crawlers | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbotVendor docs | openai.com/searchbot.json |
Claude-SearchBot | Anthropic | Search & citation crawlers | No user agent of its own / see vendor docs Vendor docs | claude.com/crawling/bots.json |
PerplexityBot | Perplexity | Search & citation crawlers | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)Vendor docs | www.perplexity.com/perplexitybot.json |
Applebot | Apple | Search & citation crawlers | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)Vendor docs | search.developer.apple.com/applebot.json |
Meta-WebIndexer | Meta | Search & citation crawlers | meta-webindexer/1.1No official IP list. Vendor docs | Not published |
Amzn-SearchBot | Amazon | Search & citation crawlers | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36Vendor docs | developer.amazon.com/amazonbot/searchbot-ip-addresses/ |
DuckAssistBot | DuckDuckGo | Search & citation crawlers | DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)Vendor docs | duckduckgo.com/duckassistbot.json |
MistralAI-Index | Mistral AI | Search & citation crawlers | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots)Vendor docs | mistral.ai/mistralai-index-ips.json |
ChatGPT-User | OpenAI | User-triggered fetchers | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/botUser-initiated; OpenAI says robots.txt rules may not apply. Vendor docs | openai.com/chatgpt-user.json |
Claude-User | Anthropic | User-triggered fetchers | No user agent of its own / see vendor docs Vendor docs | claude.com/crawling/bots.json |
Perplexity-User | Perplexity | User-triggered fetchers | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)User-initiated; Perplexity says it generally ignores robots.txt. Vendor docs | www.perplexity.com/perplexity-user.json |
Meta-ExternalFetcher | Meta | User-triggered fetchers | meta-externalfetcher/1.1User-initiated; Meta says it may bypass robots.txt. No official IP list. Vendor docs | Not published |
Amzn-User | Amazon | User-triggered fetchers | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-User/0.1) Chrome/W.X.Y.Z Safari/537.36Vendor docs | developer.amazon.com/amazonbot/live-ip-addresses/ |
MistralAI-User | Mistral AI | User-triggered fetchers | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)Vendor docs | mistral.ai/mistralai-user-ips.json |
AI bot traffic FAQ
What is the GPTBot user agent?
GPTBot is OpenAI’s crawler for gathering content that may be used to train its AI models. Its user agent contains the token GPTBot (currently GPTBot/1.4; +https://openai.com/gptbot). OpenAI publishes its IP ranges at https://openai.com/gptbot.json. To opt out of training, add User-agent: GPTBot and Disallow: / to robots.txt.
How do I check if AI bots are crawling my site?
There are two ways. To see whether bots can crawl, run the checker above: it tests robots.txt and a live request for each bot. To see whether they actually visit, search your server access logs for tokens like GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, then confirm the source IPs against each vendor’s published JSON IP ranges.
Does blocking GPTBot affect ChatGPT search visibility?
No. OpenAI says GPTBot and OAI-SearchBot are separate and controlled independently. GPTBot covers training. OAI-SearchBot decides whether your site can appear in ChatGPT search answers. You can block GPTBot and still allow OAI-SearchBot. OpenAI says robots.txt changes take about 24 hours to apply to search.
Can user agents be faked?
Yes. Any script can send a GPTBot or ClaudeBot user agent. To confirm a request is real, check that its IP falls inside the vendor’s published range (for example https://openai.com/gptbot.json, https://openai.com/searchbot.json or https://www.perplexity.com/perplexitybot.json). If the user agent says it’s a known bot but the IP isn’t in that range, treat the request as unverified.
How is this different from Cloudflare AI Crawl Control?
Cloudflare AI Crawl Control monitors and blocks AI crawlers for sites that run on Cloudflare. On the free plan, it identifies crawlers by user agent only, and the Metrics tab shows just the past 24 hours. This checker is a free one-off test that works on any site, on Cloudflare or not.
Is there an AI crawler log analyzer for nginx?
You don’t need a dedicated tool to start. With the default “combined” log format, nginx already records user agents, so grep -E "GPTBot|OAI-SearchBot|ClaudeBot|PerplexityBot" /var/log/nginx/access.log shows AI crawler hits. Count them per bot with awk, then check the IPs against each vendor’s JSON range list.
Operator: RETRETO NEXT LIMITED.