Technical SEO & Website Performance
XML Sitemaps and robots.txt: A Practical Guide for Beginners
What XML sitemaps and robots.txt files do, what they do not do, how to check them on a WordPress site, and the common mistakes that stop important pages from being found or indexed.
Last reviewed: 14 October 2026
An XML sitemap tells search engines which URLs you consider important. A robots.txt file tells crawlers which parts of your site they may or may not crawl. They are small files, but a mistake in either can hide important pages or waste crawling on unimportant ones. This guide explains both in practical terms.
What an XML sitemap does
A sitemap is a list of the URLs you want search engines to know about. It helps discovery, especially for new sites, large sites and pages with few internal links. It does not force a search engine to crawl or index a URL: Google describes submitting a sitemap as a hint.
Key facts from Google’s documentation (checked October 2026):
- A single sitemap can contain up to 50,000 URLs or 50 MB uncompressed. Larger sites split their URLs across several sitemaps listed in a sitemap index file.
- Google ignores the
priorityandchangefreqvalues in a sitemap. - Google uses
lastmod(last modified date) only when it is consistently accurate, meaning it reflects a significant change to the page, not a small edit such as updating a copyright year. - XML is the most flexible format, but RSS/Atom feeds and simple text files are also accepted.
What to include in your sitemap
- Pages you want to appear in search: services, important blog posts, guides, category pages that are useful to visitors.
- Only the canonical version of each page (the main URL, not duplicates with parameters).
What to leave out
- Pages with
noindex, redirects, error pages and duplicate URLs. - Thank-you pages, internal search results and login pages.
Sitemaps on WordPress
Most SEO plugins, including Rank Math and Yoast, create a sitemap index automatically, often at /sitemap_index.xml. WordPress itself also offers a basic sitemap at /wp-sitemap.xml when no plugin replaces it. Open your sitemap in a browser, check that it lists the pages you expect, then submit it in the Sitemaps report in Google Search Console. You can also add a line such as Sitemap: https://www.example.com/sitemap_index.xml to robots.txt.
What robots.txt does
robots.txt is a plain text file at the root of your domain, for example example.com/robots.txt. It gives crawlers rules about which URLs they may request. It is mainly used to manage crawling, for example to keep crawlers out of endless filter combinations or admin areas.
A simple WordPress-style robots.txt might look like this:
User-agent: *(the rules below apply to all crawlers)Disallow: /wp-admin/(do not crawl the admin area)Allow: /wp-admin/admin-ajax.php(but this file may be requested)Sitemap: https://www.example.com/sitemap_index.xml
The most important rule: robots.txt does not remove pages from Google
Google states that robots.txt is not a mechanism for keeping a page out of Google. If a blocked URL is linked from elsewhere, it can still be indexed without its content. To keep a page out of search results, allow it to be crawled and use a noindex robots meta tag, or protect it with a password.
| Goal | Use | Do not use |
|---|---|---|
| Keep a page out of search results | noindex on a crawlable page, or password protection | robots.txt Disallow |
| Reduce crawling of low-value URLs | robots.txt Disallow | noindex alone (crawlers still visit) |
| Help discovery of important pages | XML sitemap and internal links | Relying on robots.txt |
| Choose the main version of duplicate pages | Canonical tags and redirects | robots.txt Disallow |
Common mistakes and how to check for them
- Blocking the whole site. A line
Disallow: /underUser-agent: *blocks all crawling. This sometimes happens when a development site is copied to live. Check your live robots.txt after any site launch or migration. - Blocking CSS or JavaScript. If crawlers cannot load the files that build your layout, they may not render pages correctly.
- Using robots.txt to hide private pages. Anyone can read robots.txt, so listing secret folders there can draw attention to them. Use proper access control instead.
- Sitemaps full of redirects or noindex pages. These send mixed signals. Keep the sitemap clean.
- Forgetting to resubmit after major changes. After a redesign or URL change, check that the sitemap reflects the new URLs.
Controlling AI crawlers
Some site owners now ask how robots.txt relates to AI. Google explains that its AI features in Search (AI Overviews and AI Mode) are part of Google Search, so they follow the same Googlebot rules. Google also provides a separate product token, Google-Extended, that lets you limit the use of your content in some of Google’s other AI systems without affecting how your site appears in Search. Other companies publish their own crawler names and rules; check each provider’s documentation before blocking anything, and remember that blocking a crawler can also remove a source of visitors. For more context, read Google AI Overviews and AI Mode: what website owners should know.
A five-minute check
- Open /robots.txt: is anything important disallowed?
- Open your sitemap: does it list your key pages and only canonical URLs?
- In Search Console, open Sitemaps: was the sitemap read successfully?
- In Search Console, open Page indexing: are important pages excluded for an unexpected reason?
- Inspect one key URL with URL Inspection: is crawling allowed and is the page indexed?
Common questions
Does a small website need a sitemap?
It is not strictly required for a small, well-linked site, but it is easy to create with a WordPress SEO plugin and can help search engines find new pages faster. There is little downside as long as the sitemap is accurate.
Should I block AI crawlers in robots.txt?
It depends on your goals. Blocking Googlebot would remove your pages from Google Search, including AI features in Search. The Google-Extended token only affects some of Google’s other AI systems, not Search. Other AI companies document their own crawlers; read each provider’s documentation and consider whether those tools also send you visitors before blocking them.
How do I check whether Google can crawl a page?
Use the URL Inspection tool in Google Search Console. It shows whether crawling is allowed by robots.txt, whether the page is indexed and which canonical URL Google selected.
Next step
Sitemaps and robots.txt help search engines find your pages; canonical tags help them choose the right version. Continue with canonical tags and duplicate URLs and how search engines crawl, index and rank pages. This guide is part of Module 4 of our free SEO course (Technical SEO). If you run a WordPress site, creating an XML sitemap and connecting Google Search Console are practised in Module 5 of our WordPress Course in Pune.
Sources (checked 14 October 2026): Google Search Central, Build and submit a sitemap, Introduction to robots.txt and AI features and your website.