Search discovery
Crawlers that help providers find pages for search experiences.
OAI-SearchBotPerplexityBotClaude-SearchBotSee which AI crawlers
reach your website.
See which AI systems request your site, which pages they reach, and where access breaks down. Turn crawler activity into a clearer picture of your discoverability.
NikeAI crawler visibility
24-hour activityObserved requests1,280
Successful responses1,240
Access denied40
Requests by hour
00:00 — 23:0000:0006:0012:0018:0023:00
| Page | User agent | Purpose | HTTP status |
|---|
Connect your server logs.
Connect your CDN, web server or edge logs. Browser analytics alone can miss requests from bots that never execute your tracking script.
Vasa groups observed requests by provider, crawler purpose and URL, then highlights which important pages are being reached successfully.
01
Collect
Timestamp, path, status, user agent and source IP from your connected logs.
02
Verify
Match a recognised agent with provider-published network ranges where available. Keep unverified matches separate.
03
Interpret
Group requests by page and purpose. Compare successful fetches, blocked requests, redirects and server errors.
Identify why each crawler visits.
A visit from an AI company can serve search, model development or a specific user request. Classify the purpose before deciding what to allow.
Model development
Crawlers collecting content that may contribute to training.
GPTBotClaudeBotUser-directed visits
A fetch initiated by someone asking an assistant to retrieve content.
ChatGPT-UserPerplexity-UserClaude-UserProvider definitions: OpenAIPerplexityAnthropic
Find and resolve access problems.
Follow a Nike running-shoe page through three checks. See how an access problem becomes a specific action.
The page was requested.
GET /running/shoes
OAI-SearchBotAgent matched · Network verified
OAI-SearchBotAgent matched · Network verified
An identifiable crawler reached the edge. Record the request before drawing conclusions about what the provider read.
The response was blocked.
403Access denied
Edge rule matched
Edge rule matched
Inspect the firewall event and the page’s crawl policy. A user-agent string alone is not enough evidence to trust a request.
Review the rule.
Watch the next visit.
403200
If the restriction was unintended, propose a narrowly scoped correction. Confirm access from subsequent logs, then measure answer visibility separately.
Access is the first signal.
A successful request shows that your server returned a response. It does not establish that a page was indexed, used for training, cited or recommended.
Connect crawler activity with Prompt Analysis to compare what providers can reach with what they actually say. Missing log coverage stays visible as missing data.
Make your best pages
easier to discover.
Find the access gaps behind your AI visibility.
Request your free reportExplore Agent Analysis