Microsys
  

Exclude URLs When Creating XML Sitemaps

You can exclude URLs by various means, e.g. response code. This works with all kinds of sitemaps including HTML and XML sitemaps.

General About Filtering URLs before Building Sitemaps

Normally, filtering of URLs are done by the site crawler during website scan, e.g. through output filters, filtering session IDs in URLs and robots.txt file, nofollow and noindex.

Depending on the program configuration, not all URLs shown in the website tree view will be included in generated sitemaps.

You can control if exclusion related filters are after a website scan has finished or when building sitemaps:
  • Older versions:
    • Scan website | Crawler options | Apply "webmaster" and "output" filters after website scan stops
  • Newer versions:
    • Scan website | Output filters | After website scan stops: Remove URLs excluded
    • Scan website | Webmaster filters | After website scan stops: Remove URLs with noindex/disallow
  • And then:
    • Check Create sitemap | Document options | Remove URLs excluded by "webmaster" and "output" filters

Note: You can also edit state flags of URLs such as do not output after a website crawl has finished:

mass edit state flags


Exclude URLs by Response Code in Sitemap Builder

In addition to general filtering, you can also exclude URLs when building sitemap files (including HTML sitemaps and XML sitemaps) based on HTTP response codes.

Generally speaking, when using the default configuration, only URLs with a valid response code are included when building sitemaps. There are a few specific exceptions when creating HTML sitemaps, but otherwise all unwanted URLs are left out.

Example: URLs that redirect with e.g. response 301 : Moved Permanently are not included when building XML sitemaps.

Which response codes the sitemap builder will accept can be set in option Create sitemap | Document options.

xml sitemap response code
TechSEO360
TechSEO360 | help | previous | next
SEO website crawler tool that can find broken links, analyze internal link juice flow, show duplicate titles, perform custom code/text search and much more.
This help page is maintained by
As one of the lead developers, his hands have touched most of the code in the software from Microsys. If you email any questions, chances are that he will be the one answering.
Share this page with friends   LinkedIn   Twitter   Facebook   Pinterest   YouTube  
 © Copyright 1997-2024 Microsys

 Usage of this website constitutes an accept of our legal, privacy policy and cookies information.