XML Sitemaps: The Complete Guide to Getting Every Page Indexed

An XML sitemap is a file that lists all the pages on your website that you want search engines to index. It does not directly help you rank — ranking is determined by content quality, links, and hundreds of other factors. But it helps you get indexed, which is the prerequisite for ranking. Pages that are not indexed cannot rank for anything. This guide covers how to build, submit, and maintain an XML sitemap that ensures every valuable page on your site is crawled and indexed.

What an XML sitemap actually does

A sitemap serves three functions:

1. Discovery. It tells search engines about pages they might not discover through normal crawling — new pages, isolated pages, pages with no internal links.
2. Priority hint. The "priority" field (0.0 to 1.0) suggests which pages are most important. Google has stated that priority is mostly ignored, but it does not hurt to include.
3. Refresh hint. The "lastmod" (last modified) and "changefreq" (change frequency) fields suggest when to re-crawl. Google uses lastmod but largely ignores changefreq.

The most important function is discovery. For most sites, search engines discover pages through crawling — they follow links from known pages to find new ones. But for new sites, sites with weak internal linking, or sites with isolated pages (pages with few or no internal links), the sitemap is the primary discovery mechanism.

The sitemap structure

A minimal XML sitemap looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2025-06-30</lastmod>
    <changefreq>weekly</changefreq>
    <priority>1.0</priority>
  </url>
  <url>
    <loc>https://example.com/about</loc>
    <lastmod>2025-06-15</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.6</priority>
  </url>
</urlset>

The required elements are the XML declaration, the urlset root element with the sitemaps.org namespace, and at least one url element containing a loc (location) element. The lastmod, changefreq, and priority elements are optional but recommended.

What to include in your sitemap

Include every page that you want search engines to index and rank. This typically means:

• The homepage
• All published blog posts and articles
• All category and tag pages (if you want them indexed — see below)
• All product or service pages (for e-commerce or business sites)
• All location or landing pages (for multi-location businesses)
• The About, Contact, and other key static pages

What to exclude:

• Pages blocked by robots.txt (search engines cannot crawl them anyway)
• Pages with "noindex" meta tags (you are telling search engines not to index them, so including them in the sitemap sends a contradictory signal)
• Redirect URLs (the destination should be in the sitemap, not the source)
• Pages that return 404 or 410 status codes
• Pagination pages (page 2, page 3 of archives) — generally exclude these to avoid duplicate-content issues
• Faceted navigation pages (filter combinations) — these create thousands of low-value URLs that dilute crawl budget
• Logged-in-only pages (user dashboard, account settings) — these are not accessible to crawlers

The category and tag page question

Whether to include category and tag pages in your sitemap depends on your site architecture and content strategy:

Include if: your category and tag pages have unique, valuable content (custom intros, curated lists) and you want them to rank for category-level queries. WordPress category archives, when properly curated, can be valuable landing pages.

Exclude if: your category and tag pages are just lists of post excerpts with no unique content. These are thin pages that can dilute crawl budget and compete with your individual posts for rankings.

For most content sites, include category pages and exclude tag pages. Tag pages tend to multiply (each post can have 5–10 tags) and produce thin, overlapping content.

Sitemap size limits

A single sitemap file is limited to 50,000 URLs and 50MB uncompressed. For most sites, this is more than enough. For sites that exceed these limits (large e-commerce sites, news sites with thousands of articles), use a sitemap index file that references multiple individual sitemaps:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemap-posts.xml</loc>
    <lastmod>2025-06-30</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-products.xml</loc>
    <lastmod>2025-06-30</lastmod>
  </sitemap>
</sitemapindex>

Submit the sitemap index file to Google Search Console; Google will discover and process the individual sitemaps.

Submitting your sitemap to Google

Once your sitemap is published (typically at example.com/sitemap.xml), submit it through Google Search Console:

1. Navigate to Search Console → Sitemaps
2. Enter the sitemap URL (just "sitemap.xml" — the domain is pre-filled)
3. Click Submit
4. Wait for Google to process (typically 1–7 days for the first submission)

After processing, Search Console shows the number of discovered URLs, the number of indexed URLs, and any errors. The "discovered but not indexed" status is common for new sites and means Google knows about the pages but has not yet decided to index them. This typically resolves over weeks to months as Google builds trust in the site.

The robots.txt connection

You can reference your sitemap in your robots.txt file, which helps search engines discover it:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

This is good practice even if you have also submitted the sitemap through Search Console — it ensures that search engines other than Google (Bing, DuckDuckGo, etc.) can also discover it.

Dynamic sitemaps vs static sitemaps

For sites that update infrequently (a few times per month), a static sitemap.xml file that you regenerate when content changes is sufficient. Use the XML Sitemap Generator to create the file from your current list of URLs.

For sites that update frequently (daily or more), a dynamic sitemap generated by your content management system is preferable. WordPress, Shopify, and most modern CMS platforms generate sitemaps automatically. If your CMS does not, a simple PHP or Python script can generate the sitemap on demand from your database.

For very large sites (10,000+ URLs), a dynamic sitemap with proper caching is essential. Generating a 50,000-URL sitemap on every request would be prohibitively expensive.

Common sitemap mistakes

1. Including non-canonical URLs

If your site has URL parameters (utm_source, session IDs, etc.), your sitemap should include only the canonical URLs. Including parameterised URLs creates duplicate-content issues and wastes crawl budget.

2. Including redirected URLs

If a URL has been redirected (301 or 302), remove it from the sitemap and add the destination URL. Including redirected URLs creates redirect chains that waste crawl budget.

3. Forgetting to update lastmod

When you update a page, update the lastmod date in the sitemap. Google uses lastmod to prioritise crawling — pages with recent lastmod dates are crawled sooner. A sitemap with stale lastmod dates for all pages signals that the site is not being maintained.

4. Including 404 pages

If a page has been deleted, remove it from the sitemap immediately. Leaving 404 URLs in the sitemap tells Google to crawl pages that do not exist, which wastes crawl budget and signals poor site maintenance.

5. Not submitting the sitemap at all

A surprising number of sites publish a sitemap.xml but never submit it through Search Console. While Google can discover sitemaps through robots.txt, explicit submission accelerates the process and provides diagnostic information that is not available otherwise.

Sitemap monitoring

Check your sitemap status in Google Search Console monthly. Look for:

• Errors (URLs that could not be processed)
• Warnings (URLs with issues that may affect indexing)
• The ratio of discovered to indexed URLs (a low ratio suggests quality issues)
• The total URL count (should match your expectation; large discrepancies suggest a sitemap generation problem)

Investigate any errors or significant changes. Sitemap health is a leading indicator of broader site health — if your sitemap has issues, your site probably has other issues that need attention.

The bottom line

An XML sitemap is foundational SEO infrastructure. It does not directly improve rankings, but it ensures that search engines discover and index all your valuable pages — the prerequisite for any ranking. Build a clean sitemap that includes all indexable pages and excludes non-canonical, redirected, or deleted URLs. Submit it through Google Search Console and reference it in robots.txt. Monitor its status monthly and fix issues promptly. Use the XML Sitemap Generator to create a valid sitemap from your URL list, or configure your CMS to generate one dynamically if your site updates frequently.