Free · No signup · Unlimited
Free Robots.txt Tester
Paste your robots.txt and test specific URLs against it. See which rule allows or blocks each URL, per user-agent.
What this tool checks
Test specific URLs against your rules (Googlebot, Bingbot, GPTBot, others)
Detects accidental site-wide Disallow: / (the #1 SEO disaster)
Validates Sitemap: directives are reachable
Warns about wildcard and crawl-delay misconfigurations
Want this monitored weekly?
Get alerts the moment your meta tags, schema, OG tags or robots.txt regress.
Sign up freeRelated SEO issues we'll catch in a full audit
Missing Meta Description
What a missing meta description means for SEO, why it lowers click-through rate, and how to fix it on any platform in under 2 minutes.
Meta Description Too Long
Meta descriptions over 160 characters get truncated in search results. Here's how to shorten them without losing the message.
Meta Description Too Short
Descriptions under 50 characters tell Google you didn't write one on purpose. Here's how to expand without padding.
Duplicate Meta Descriptions
Multiple pages sharing the same meta description signal thin or templated content. Here's how to detect and fix duplicates.
Missing <title> Tag
Pages without a title tag fall back to URL or domain in search results. Here's how to add one in any framework.
Title Tag Too Long
Titles over 60 characters get truncated. Here's how to shorten without losing the keyword.
Duplicate Title Tags
Multiple pages with identical titles compete with each other in search. Here's how to deduplicate.
Missing H1 Tag
Pages without an H1 lose a key topical signal. Here's why one H1 per page matters and how to add it correctly.
Multiple H1 Tags on One Page
Multiple H1s dilute the topical signal. Here's how to consolidate to one.
FAQ
What's the worst robots.txt mistake?
User-agent: * / Disallow: /, blocks the entire site from every crawler. We flag this in red.
Does robots.txt block crawlers?
Only if you add user-agent blocks for GPTBot, Google-Extended, CCBot, etc. Standard 'User-agent: *' applies, but most respect dedicated rules.