Free AI crawler checker
See how your robots.txt rules apply to ChatGPT, Perplexity, Claude and Google for one page. Each result shows the crawler’s purpose, selected group and matching rule, with its source line.
Test the rules for a specific page
Copy the plain text from your site’s /robots.txt, then enter the page URL or path to check. The test runs in your browser. It does not fetch your site, save the pasted text or require email.
Search, training and user requests are different
OAI-SearchBot, PerplexityBot and Claude-SearchBot serve search discovery. GPTBot and ClaudeBot serve model training. Blocking a training crawler is a separate choice from allowing its search crawler.
Google-Extended controls certain AI uses of content; it does not control inclusion or ranking in Google Search. ChatGPT-User and Perplexity-User perform user-requested fetches, where robots rules may not apply. Their results here are rule simulations, not access predictions.
Why a specific path matters
A file can allow the homepage and disallow /private/. A more specific Allow rule can reopen a path within that directory. This checker combines matching groups, prefers an agent-specific group over the wildcard group, and selects the longest matching path rule. Allow wins when matching rules have equal specificity. Paths are case-sensitive.
What to check after robots.txt
Confirm that the deployed file contains the intended rules. Then inspect HTTP responses, firewall logs, page-level noindex directives and whether useful content is available in the returned HTML. Robots permission alone does not establish search eligibility or citations.
Run an AI readiness audit for broader page diagnostics, or read the measured noindex example to see why a good total score can hide a search exclusion.
Crawler documentation
Purpose descriptions checked on 13 October 2026. Consult the provider documentation when setting policy.