Published November 11, 2026
How can I tell if AI crawlers are visiting my website?
Check your server's access log. Every request to your site records a user-agent name, and the major AI companies publish the names their bots use: OAI-SearchBot, GPTBot, and ChatGPT-User for OpenAI, PerplexityBot and Perplexity-User for Perplexity, and Googlebot for Google. Find those names, then confirm the requests came from the IP addresses each company publishes, because anyone can fake a user-agent name.
What this means
Most business owners assume that if their site is live, AI systems can read it. Often that's true. But a firewall rule, a "block bots" setting at the host, or an old robots.txt line can quietly turn AI crawlers away, and nothing on the site itself will tell you. The log will.
It also helps to know that these bots don't all do the same job. OpenAI lists separate agents for separate purposes: OAI-SearchBot finds pages to show in ChatGPT's search features, GPTBot collects content that may be used to train its models, and ChatGPT-User visits a page when a person in ChatGPT asks something that calls for it (OpenAI, n.d.). Perplexity draws the same line between PerplexityBot, which indexes pages for its search results, and Perplexity-User, which fetches a page in response to a user's question (Perplexity AI, n.d.). So seeing GPTBot in your logs tells you something different from seeing OAI-SearchBot.
Who this applies to
Any business that wants to be named when customers ask ChatGPT, Perplexity, or Google's AI features for a recommendation. It matters most if you've recently changed hosts, added a security service or CDN, or had a developer edit your robots.txt, since those are the moments a block tends to slip in unnoticed.
How we'd diagnose it
We'd work through four checks, in this order:
- Pull 30 days of raw access logs from the host or CDN and search them for each AI user-agent name.
- Look at the response codes, not just whether the bot showed up. A 200 means the page was served. A 403 or a challenge page means something turned the bot away.
- Verify the IP addresses. OpenAI and Perplexity each publish JSON files of the IP ranges their bots use (OpenAI, n.d.; Perplexity AI, n.d.). For Google, the documented method is a reverse DNS lookup on the requesting IP, a check that the hostname ends in googlebot.com, google.com, or googleusercontent.com, then a forward lookup to confirm it points back to the same IP (Google, 2026a). A request that claims to be GPTBot but comes from somewhere else isn't GPTBot.
- Compare against robots.txt, so you can tell a deliberate rule apart from an accidental one.
Available options
- Read the raw logs yourself. Free, if your host provides them. A text search for each bot name gets you most of the way.
- Use your CDN or host's bot reports. Some security and CDN services label known AI crawlers for you, which saves the manual work.
- Have someone run the full check. Worth it if the logs are messy, you run several sites, or you need the IP verification done properly.
Benefits, limitations, and tradeoffs
The log is the most direct evidence you'll get that an AI system can reach your pages. That's also its limit. A visit from OAI-SearchBot tells you OpenAI can read the page. It doesn't tell you whether ChatGPT will ever cite it, and no crawl report can tell you that.
There are other gaps. Some user-requested fetchers don't follow robots.txt the way indexing crawlers do. OpenAI says robots.txt rules "may not apply" to ChatGPT-User (OpenAI, n.d.), and Perplexity says Perplexity-User generally ignores them (Perplexity AI, n.d.). And Google-Extended won't show up in your logs at all. Google says it has no separate user agent string. It's a robots.txt token that controls whether content Google has crawled can be used to train Gemini models, and it doesn't affect inclusion or ranking in Google Search (Google, 2026b).
What we know
The companies are fairly explicit about which bot controls what. OpenAI says sites that opt out of OAI-SearchBot won't be shown in ChatGPT search answers, though they can still appear as navigational links, and that it can take about 24 hours for its systems to pick up a robots.txt change (OpenAI, n.d.). Perplexity gives a similar window of up to 24 hours (Perplexity AI, n.d.). Perplexity says neither of its agents collects content to train AI foundation models (Perplexity AI, n.d.). OpenAI handles the training question separately: disallowing GPTBot is how a site signals that its content shouldn't be used for training, and that setting is independent of the search one (OpenAI, n.d.).
Volume is another matter. Cloudflare reported that over the 12 months to mid-2025, about 80% of the AI crawling it saw was for model training, 18% was for search, and 2% was for user actions (Tomé, 2025). The same analysis put OpenAI at roughly 1,091 crawls for every visitor it referred back to a site in July 2025 (Tomé, 2025). Those are Cloudflare's own network measurements, not an audit of any single site. What they mean in practice is that most AI-bot traffic in a log is likely to be training crawls rather than the search crawlers that decide whether you can be cited.
Next steps
Open your robots.txt (it's at your-domain.com/robots.txt) and note which AI agents it mentions. Then ask your host or web developer for 30 days of raw access logs and search them for OAI-SearchBot, PerplexityBot, and Googlebot. If the search crawlers are getting through with 200 responses, access isn't your problem and the work is elsewhere. If they're getting 403s, or never show up at all, start there.
Orlando considerations
A lot of Central Florida service businesses run on hosted website builders or a site a contractor set up years ago, and some of those platforms don't expose raw server logs at all. If that's your situation, you can still confirm the robots.txt side yourself, ask your platform whether it offers a bot or crawler report, and test the outcome directly by asking ChatGPT and Perplexity about your business by name and by category.
Frequently asked questions
If AI crawlers visit my site, does that mean I'll show up in AI answers?
No. A crawl means the system can read your pages. Whether it then names or cites your business depends on many other signals, and no one controls that outcome except the company running the AI system.
Why don't I see Google-Extended in my logs?
Google says Google-Extended has no separate user agent string. Google crawls with its existing user agents, and Google-Extended is only a robots.txt token that controls whether crawled content can be used for Gemini model training and grounding.
Can I see AI crawler visits in Google Analytics?
Generally not. Crawlers fetch pages without running the analytics script the way a browser does, so they rarely appear in tools like Google Analytics. Server logs or your CDN's bot reports are the reliable place to look.
My website builder doesn't give me server logs. What can I do?
Check whether your host or CDN offers a bot or crawler report, since some do. If neither is available, you can still confirm the robots.txt side yourself and test the outcome directly by asking AI engines about your business.
References
Google. (2026a, March 20). Verify requests from Google crawlers and fetchers. Google for Developers. https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests
Google. (2026b, October 30). Google's common crawlers. Google for Developers. https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
OpenAI. (n.d.). Overview of OpenAI crawlers. OpenAI Developers. Retrieved November 11, 2026, from https://developers.openai.com/api/docs/bots
Perplexity AI. (n.d.). Perplexity crawlers. Perplexity Docs. Retrieved November 11, 2026, from https://docs.perplexity.ai/docs/resources/perplexity-crawlers
Tomé, J. (2025, August 29). The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals. Cloudflare Blog. https://blog.cloudflare.com/crawlers-click-ai-bots-training/
Last updated: November 11, 2026
If the logs show the crawlers are getting through and you're still not being named, access isn't the bottleneck. That's the gap an AI visibility audit is built to map, with a prioritized list of what to fix first.