Other available SEO & GEO tools
Compare two lists
Compare two lists online and separate common values, items found only in list A, and items found only in list B. Local 02Remove duplicates from a list
Remove duplicate lines from a list online while keeping the first occurrence and the original order. Optional case-sensitive matching. Local 03List intersection
Find the intersection of two lists and copy only the values present in both data sets, with optional case-sensitive matching. Local 04List difference
Compare two lists and return their symmetric difference, containing every value found in only one of the two lists. Local 05Online word counter
Count words, characters, sentences, and paragraphs in pasted text or a web page, with an estimated reading time. Local + URL 06Online character counter
Count characters with and without spaces in pasted text or a web page, plus words, sentences, paragraphs, and reading time. Local + URL 07Keyword research tool
Find keywords for free and compare search volume, SEO difficulty, and traffic potential in the Cadrank France or US database. Cadrank data 08Keyword search volume
Enter a keyword and check its monthly search volume for free in the Cadrank France or United States database. Cadrank data 09Extract links from a page
Extract every link from a web page or HTML source, including its anchor text, rel attribute, and internal or external type. Local + URL 10Google SERP preview
Preview a Google result with its title, URL, and meta description, including estimated character and pixel widths. Local + URL 11SEO title length checker
Check an SEO title tag’s length in characters and pixels against a 580-pixel desktop display guide. Local + URL 12Meta description length checker
Check meta description length in characters and pixels against a 920-pixel desktop display guide. Local + URL 13HTML meta tag generator
Generate essential HTML head tags: title, meta description, canonical URL, robots directive, and viewport. Local + URL 14Open Graph and Twitter Card preview
Test a page’s Open Graph preview and generate og:title, og:description, og:image, og:url, and Twitter Card tags. Local + URL 16robots.txt validator
Validate robots.txt syntax and find unknown directives, incomplete groups, and malformed lines before publishing. Local + URL 17Test a URL against robots.txt
Test whether robots.txt allows or blocks a URL for Googlebot or another User-agent, and identify the deciding rule. Local + URL 18AI crawler access test
Check robots.txt access for GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, and other major AI crawlers. Local + URL 19URL indexability test
Check a URL’s indexability using HTTP status, robots.txt, robots meta, X-Robots-Tag, canonical URL, and redirects. URL 20Hreflang tag tester
Test a page’s hreflang tags, list each language code and URL, and identify invalid international annotations. Local + URL 21Hreflang reciprocity checker
Check hreflang return links between pages in a multilingual cluster and identify non-reciprocal annotations. URL 22Canonical and hreflang conflicts
Find conflicts between canonical URLs and hreflang tags that can invalidate an international SEO cluster. Local + URL 23LocalBusiness Schema generator
Generate Schema.org LocalBusiness JSON-LD with a business name, URL, telephone, address, and price range. Local 24FAQPage Schema generator
Turn questions and answers into valid FAQPage JSON-LD structured data, ready to add to a web page. Local 25Content similarity checker
Compare two texts or pages with cosine and 3-gram Jaccard similarity scores to identify content that may be too close. Local + URL 26Frequency and n-gram analyzer
Analyze the frequency of words and groups of 2 to 4 words in pasted text or a web page, with counts and percentages. Local + URL 27HTTP status checker
Check a URL’s HTTP status code, final destination, and observed response time after redirects. URL 28Redirect chain checker
Check a URL redirect chain and display every HTTP code, Location destination, loop, and final URL. URL 29HTTP header viewer
View a URL’s HTTP response headers, including Content-Type, Cache-Control, Location, CSP, HSTS, and X-Robots-Tag. URL 30HTTP cache and security analysis
Test HTTP security and cache headers, including CSP, HSTS, Cache-Control, X-Content-Type-Options, and Referrer-Policy. URL 31Image dimensions and metadata reader
View EXIF metadata, MIME type, file size, dimensions, and aspect ratio for a local image or a public image URL. Local + URLThis robots.txt generator creates a clean file from your crawling rules. Define the crawler, allowed and disallowed paths, and sitemap address, then copy the output to publish at the root of your website.
How to use this tool
- Enter the
User-agent, such as*for all crawlers. - Add one path per line under Allow and Disallow.
- Enter the absolute sitemap URL when available.
- Optionally set a
Crawl-delay, then generate the file.
robots.txt example
User-agent: *
Allow: /
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml
Publish the file at https://example.com/robots.txt. A robots.txt rule controls crawling but does not guarantee removal of an already known URL from the index. To remove a page from search results, use an accessible noindex directive.
Should robots.txt block CSS and JavaScript?
Usually not. Search engines need these assets to understand page rendering. Block only areas that provide no crawling value, then check the file with the robots.txt validator.
What does the Sitemap directive do?
It tells crawlers where to find the XML sitemap. This aids discovery but does not replace submitting the sitemap through webmaster tools.
Data processing
Processing runs entirely in your browser. Entered data is sent neither to Pied Web nor to Cadrank.
Browse all 31 free SEO & GEO tools in the Pied Web toolbox.