Sitemap Builder Can Exclude URLs When Building XML Sitemaps
Microsys
  

Sitemap Builder Can Exclude URLs When Building XML Sitemaps

The sitemap builder can exclude URLs by various means, e.g. response code. This works with all kinds of sitemaps including HTML and XML sitemaps.
Help: overview | previous | next

General About Filtering URLs before Building Sitemaps

Normally, filtering of URLs are done by the site crawler during website scan, e.g. through output filters, filtering session IDs in URLs and robots.txt file, nofollow and noindex.

Depending on the program configuration, not all URLs shown in the website tree view will be included in generated sitemaps.

You can control if and at which point filters are applied during website scan or sitemap building:
  • Disable Scan website | Crawler options | Apply "webmaster" and "output filters" after website scan stops.
  • Enable Create sitemap | Document options | Remove URLs excluded by "webmaster" and "output" filters.


Exclude URLs by Response Code in Sitemap Builder

In addition to general filtering, you can also exclude URLs when building sitemap files (including HTML sitemaps and XML sitemaps) based on HTTP response codes.

Generally speaking, when using the default configuration, only URLs with a valid response code are included when building sitemaps. There are a few specific exceptions when creating HTML sitemaps, but otherwise all unwanted URLs are left out.

Example: URLs that redirect with e.g. response 301 : Moved Permanently are not included when building XML sitemaps.

Which response codes the sitemap builder will accept can be set in option Create sitemap | Documment options.

xml sitemap response code
This help page is maintained by

As one of the lead developers, his hands have touched most of the code in the software from Microsys.

If you email any questions, chances are that he will be the one answering them.
TechSEO360About TechSEO360

SEO website crawler tool that can find broken links, analyze internal link juice flow, show duplicate titles, perform custom code/text search and much more.
Share this page with friends   LinkedIn   Twitter   Facebook   Pinterest   Google+   YouTube  
 © Copyright 1997-2018 Microsys

 Usage of this website constitutes an accept of our legal, privacy policy and cookies information.