robots.txt & sitemap.xml: 2026 Guide

These two files control how Google crawls your site. Get them wrong and your best pages disappear from search. Get them right and indexing takes care of itself. Here's the no-fluff guide.

robots.txt — what it does

robots.txt sits at yoursite.com/robots.txt and tells crawlers which paths they can fetch. It does NOT prevent indexing — only crawling. To prevent indexing, use a noindex meta tag on the page itself.

Minimum viable robots.txt

User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

Common mistakes

• Disallow: / blocks the entire site from Google (we see this on launches at least monthly)
• Sitemap directive points at the wrong domain (staging.yoursite.com instead of yoursite.com)
• Disallowing /wp-admin but accidentally also blocking /wp-admin/admin-ajax.php breaks Google's rendering
• Blocking CSS or JS folders — Google needs them to render the page

sitemap.xml — what it does

Your sitemap is a list of every URL on your site that you want indexed. It's a hint, not a command — Google still decides what to actually index.

What to include

• Every public, indexable URL (one entry per page, including dynamic ones)
• Use absolute URLs with the same domain Google sees (no http vs https mismatch)
• Include a <lastmod> date when content changes
• Skip pages with noindex, login pages, search results, and admin URLs

What to exclude

• Tag/category pages that just paginate (unless they're real landing pages)
• URL parameters that produce duplicate content (?utm_source=...)
• Anything you've blocked in robots.txt

How to verify both

1. Visit yoursite.com/robots.txt directly in your browser
2. Visit yoursite.com/sitemap.xml — should be valid XML, not HTML
3. In Search Console, submit the sitemap URL and check the "Discovered" vs "Indexed" counts
4. Run FreeSEO — it cross-checks routes against sitemap entries automatically