Skip to tools

AI crawler access test

Check robots.txt access for GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, and other major AI crawlers.

Rate limit
100 URLs / 24 h

Other available SEO & GEO tools

01

Compare two lists

Compare two lists online and separate common values, items found only in list A, and items found only in list B.
Local
02

Remove duplicates from a list

Remove duplicate lines from a list online while keeping the first occurrence and the original order. Optional case-sensitive matching.
Local
03

List intersection

Find the intersection of two lists and copy only the values present in both data sets, with optional case-sensitive matching.
Local
04

List difference

Compare two lists and return their symmetric difference, containing every value found in only one of the two lists.
Local
05

Online word counter

Count words, characters, sentences, and paragraphs in pasted text or a web page, with an estimated reading time.
Local + URL
06

Online character counter

Count characters with and without spaces in pasted text or a web page, plus words, sentences, paragraphs, and reading time.
Local + URL
07

Keyword research tool

Find keywords for free and compare search volume, SEO difficulty, and traffic potential in the Cadrank France or US database.
Cadrank data
08

Keyword search volume

Enter a keyword and check its monthly search volume for free in the Cadrank France or United States database.
Cadrank data
09

Extract links from a page

Extract every link from a web page or HTML source, including its anchor text, rel attribute, and internal or external type.
Local + URL
10

Google SERP preview

Preview a Google result with its title, URL, and meta description, including estimated character and pixel widths.
Local + URL
11

SEO title length checker

Check an SEO title tag’s length in characters and pixels against a 580-pixel desktop display guide.
Local + URL
12

Meta description length checker

Check meta description length in characters and pixels against a 920-pixel desktop display guide.
Local + URL
13

HTML meta tag generator

Generate essential HTML head tags: title, meta description, canonical URL, robots directive, and viewport.
Local + URL
14

Open Graph and Twitter Card preview

Test a page’s Open Graph preview and generate og:title, og:description, og:image, og:url, and Twitter Card tags.
Local + URL
15

robots.txt generator

Create a robots.txt file with User-agent, Allow, Disallow, Sitemap, and Crawl-delay directives, then copy the result.
Local
16

robots.txt validator

Validate robots.txt syntax and find unknown directives, incomplete groups, and malformed lines before publishing.
Local + URL
17

Test a URL against robots.txt

Test whether robots.txt allows or blocks a URL for Googlebot or another User-agent, and identify the deciding rule.
Local + URL
19

URL indexability test

Check a URL’s indexability using HTTP status, robots.txt, robots meta, X-Robots-Tag, canonical URL, and redirects.
URL
20

Hreflang tag tester

Test a page’s hreflang tags, list each language code and URL, and identify invalid international annotations.
Local + URL
21

Hreflang reciprocity checker

Check hreflang return links between pages in a multilingual cluster and identify non-reciprocal annotations.
URL
22

Canonical and hreflang conflicts

Find conflicts between canonical URLs and hreflang tags that can invalidate an international SEO cluster.
Local + URL
23

LocalBusiness Schema generator

Generate Schema.org LocalBusiness JSON-LD with a business name, URL, telephone, address, and price range.
Local
24

FAQPage Schema generator

Turn questions and answers into valid FAQPage JSON-LD structured data, ready to add to a web page.
Local
25

Content similarity checker

Compare two texts or pages with cosine and 3-gram Jaccard similarity scores to identify content that may be too close.
Local + URL
26

Frequency and n-gram analyzer

Analyze the frequency of words and groups of 2 to 4 words in pasted text or a web page, with counts and percentages.
Local + URL
27

HTTP status checker

Check a URL’s HTTP status code, final destination, and observed response time after redirects.
URL
28

Redirect chain checker

Check a URL redirect chain and display every HTTP code, Location destination, loop, and final URL.
URL
29

HTTP header viewer

View a URL’s HTTP response headers, including Content-Type, Cache-Control, Location, CSP, HSTS, and X-Robots-Tag.
URL
30

HTTP cache and security analysis

Test HTTP security and cache headers, including CSP, HSTS, Cache-Control, X-Content-Type-Options, and Referrer-Policy.
URL
31

Image dimensions and metadata reader

View EXIF metadata, MIME type, file size, dimensions, and aspect ratio for a local image or a public image URL.
Local + URL

This tool checks how robots.txt treats the leading AI crawlers. For each recognized User-agent, it displays the crawler’s purpose, whether access is allowed or blocked, and the rule behind that decision.

How to use this tool

Paste robots.txt for a local analysis, or enter the website URL to retrieve its public file. Compare the results for GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, and the other listed crawlers.

Why check AI crawler access?

AI crawlers do not all serve the same purpose. Some collect training data, while others retrieve pages to answer a request or support search. A general User-agent: * rule may also apply when no specific group exists.

The test makes your configuration visible but does not choose a policy for you. Before blocking a crawler, consider content control, visibility in generated answers, and the operator’s published terms. Review server logs as well, because robots.txt relies on voluntary compliance.

How do I block GPTBot in robots.txt?

Add a User-agent: GPTBot group with Disallow: /, then validate the file. This applies to that specific User-agent and does not automatically block other crawlers from the same company.

Does this test measure ChatGPT visibility?

No. It only checks crawling permission declared in robots.txt. Brand presence in generated answers depends on many other signals and requires separate measurement.

Data processing

Pasted data is processed in your browser. If you provide a URL, only that URL is sent to the public Cadrank proxy.

Browse all 31 free SEO & GEO tools in the Pied Web toolbox.