High impactIndexing & crawl
robots.txt Is Blocking Googlebot
Disallow rules in robots.txt can block Google from your entire site. Here's how to audit and fix safely.
What it means
Your robots.txt contains a Disallow rule that blocks Googlebot from important pages — sometimes the entire site.
Why it matters
Blocked pages can't be crawled, can't be indexed, can't rank. The classic launch-day disaster: forgetting to remove `Disallow: /` from the staging environment.
How to fix it
- Visit yoursite.com/robots.txt and read it line-by-line.
- Confirm there's no `Disallow: /` under `User-agent: *` or `User-agent: Googlebot`.
- Allow CSS, JS, and image folders — Google needs them to render pages.
- Use Search Console's robots.txt Tester to verify.
Example
Before
User-agent: *
Disallow: /After
User-agent: *
Allow: /
Sitemap: https://yoursite.com/sitemap.xmlFind this issue on your site automatically
FreeSEO scans for robots.txt is blocking googlebot and 140+ other issues, free, no signup.
Frequently asked questions
How fast can I recover after fixing this?
Submit your sitemap in Search Console and request indexing. Most sites recover within 1–2 weeks.