Crawler Catalogue
44 configured crawlers. Definitions, verification configuration and associated crawler IP assessment evidence.
AbbySEOBot
| Category | MONITORED | Confidence | 50 |
|---|---|---|---|
| Source | auto | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| — | |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
AffsignalCrawler
| Category | MONITORED | Confidence | 50 |
|---|---|---|---|
| Source | auto | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| — | |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
AhrefsBot
| Category | MONITORED | Confidence | 50 |
|---|---|---|---|
| Source | auto | Verification | |
| Source URL | — | ||
| IP List URL | https://api.ahrefs.com/v3/public/crawler-ip-ranges | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| — | |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
AionBot
| Category | MONITORED | Confidence | 50 |
|---|---|---|---|
| Source | auto | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| — | |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
Amazonbot
| Category | SEARCH | Confidence | 50 |
|---|---|---|---|
| Source | observed | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| Amazonbot/ | contains |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
Amzn-SearchBot
| Category | SEARCH | Confidence | 50 |
|---|---|---|---|
| Source | observed | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| Amzn-SearchBot | contains |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
Applebot
| Category | SEARCH | Confidence | 50 |
|---|---|---|---|
| Source | observed | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| Applebot | contains |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
Baiduspider
| Category | SEARCH | Confidence | 50 |
|---|---|---|---|
| Source | observed | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| Baiduspider | contains |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
Bingbot
| Category | SEARCH | Confidence | 50 |
|---|---|---|---|
| Source | observed | Verification | |
| Source URL | — | ||
| IP List URL | https://www.bing.com/toolbox/bingbot.json | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| bingbot | contains |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
Bytespider
| Category | SEARCH | Confidence | 50 |
|---|---|---|---|
| Source | observed | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| Bytespider | contains |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
CCBot
| Category | SEARCH | Confidence | 50 |
|---|---|---|---|
| Source | observed | Verification | |
| Source URL | — | ||
| IP List URL | https://index.commoncrawl.org/ccbot.json | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| CCBot | contains |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
ChatGPT-User
| Category | AI | Confidence | 50 |
|---|---|---|---|
| Source | observed | Verification | |
| Source URL | — | ||
| IP List URL | https://openai.com/chatgpt-user.json | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| ChatGPT-User | contains |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
ClaudeBot
| Category | AI | Confidence | 50 |
|---|---|---|---|
| Source | observed | Verification | |
| Source URL | — | ||
| IP List URL | https://claude.com/crawling/bots.json | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| ClaudeBot | contains |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
Claude-SearchBot
| Category | MONITORED | Confidence | 50 |
|---|---|---|---|
| Source | auto | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| — | |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
Claude-Web
| Category | AI | Confidence | 50 |
|---|---|---|---|
| Source | observed | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| Claude-Web | contains |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
Cohere
| Category | AI | Confidence | 50 |
|---|---|---|---|
| Source | observed | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| cohere-ai | contains |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
DataForSeoBot
| Category | MONITORED | Confidence | 50 |
|---|---|---|---|
| Source | auto | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| — | |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
DeepSeekBot
| Category | AI | Confidence | 50 |
|---|---|---|---|
| Source | observed | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| DeepSeekBot | contains |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
DotBot
| Category | SEARCH | Confidence | 50 |
|---|---|---|---|
| Source | observed | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| DotBot | contains |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
DuckAssistBot
| Category | AI | Confidence | 50 |
|---|---|---|---|
| Source | auto | Verification | |
| Source URL | — | ||
| IP List URL | https://duckduckgo.com/duckassistbot.json | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| — | |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
DuckDuckBot
| Category | SEARCH | Confidence | 50 |
|---|---|---|---|
| Source | auto | Verification | |
| Source URL | — | ||
| IP List URL | https://duckduckgo.com/duckduckbot.json | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| — | |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
FossickBot
| Category | MONITORED | Confidence | 50 |
|---|---|---|---|
| Source | auto | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| — | |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
Googlebot
| Category | SEARCH | Confidence | 50 |
|---|---|---|---|
| Source | observed | Verification | |
| Source URL | — | ||
| IP List URL | https://developers.google.com/search/apis/ipranges/googlebot.json | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| Googlebot | contains |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
Google-Extended
| Category | AI | Confidence | 50 |
|---|---|---|---|
| Source | observed | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| Google-Extended | contains |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
GPTBot
| Category | AI | Confidence | 50 |
|---|---|---|---|
| Source | observed | Verification | |
| Source URL | — | ||
| IP List URL | https://openai.com/gptbot.json | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| GPTBot | contains |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
l9scan
| Category | MONITORED | Confidence | 50 |
|---|---|---|---|
| Source | auto | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| — | |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
LinkupBot
| Category | MONITORED | Confidence | 50 |
|---|---|---|---|
| Source | auto | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| — | |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
LohiSoftBot
| Category | MONITORED | Confidence | 50 |
|---|---|---|---|
| Source | auto | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| — | |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
meta-externalagent
| Category | SOCIAL | Confidence | 50 |
|---|---|---|---|
| Source | observed | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| meta-externalagent | contains |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
MJ12bot
| Category | MONITORED | Confidence | 50 |
|---|---|---|---|
| Source | auto | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| — | |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
OAI-SearchBot
| Category | AI | Confidence | 50 |
|---|---|---|---|
| Source | observed | Verification | |
| Source URL | — | ||
| IP List URL | https://openai.com/searchbot.json | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| OAI-SearchBot | contains |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
PerplexityBot
| Category | AI | Confidence | 50 |
|---|---|---|---|
| Source | observed | Verification | |
| Source URL | — | ||
| IP List URL | https://www.perplexity.ai/perplexitybot.json | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| PerplexityBot | contains |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
Perplexity-User
| Category | AI | Confidence | 50 |
|---|---|---|---|
| Source | auto | Verification | |
| Source URL | — | ||
| IP List URL | https://www.perplexity.ai/perplexity-user.json | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| — | |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
PetalBot
| Category | SEARCH | Confidence | 50 |
|---|---|---|---|
| Source | observed | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| PetalBot | contains |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
PipericBot
| Category | MONITORED | Confidence | 50 |
|---|---|---|---|
| Source | auto | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| — | |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
Reflectionbot
| Category | MONITORED | Confidence | 50 |
|---|---|---|---|
| Source | auto | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| — | |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
SectorBot
| Category | MONITORED | Confidence | 50 |
|---|---|---|---|
| Source | auto | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| — | |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
SemrushBot
| Category | MONITORED | Confidence | 50 |
|---|---|---|---|
| Source | auto | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| — | |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
SERankingBacklinksBot
| Category | MONITORED | Confidence | 50 |
|---|---|---|---|
| Source | auto | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| — | |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
SeznamBot
| Category | MONITORED | Confidence | 50 |
|---|---|---|---|
| Source | auto | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| — | |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
SleepBot
| Category | MONITORED | Confidence | 50 |
|---|---|---|---|
| Source | auto | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| — | |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
xAI-SearchBot
| Category | AI | Confidence | 50 |
|---|---|---|---|
| Source | observed | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| xAI-SearchBot | contains |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
YaK
| Category | MONITORED | Confidence | 50 |
|---|---|---|---|
| Source | auto | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| — | |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.
YandexBot
| Category | SEARCH | Confidence | 50 |
|---|---|---|---|
| Source | observed | Verification | |
| Source URL | — | ||
| IP List URL | — | ||
| Allowed DNS domains | — | ||
| Confirmation | |||
| Registered networks | 0 | Assessed IPs | 0 |
| First seen | Last seen | ||
User-Agent Patterns
| Pattern | Match type |
|---|---|
| YandexBot | contains |
Assessment Evidence
No crawler IP assessments currently associated.
No associated IP assessments.