Sitemap Builder can Exclude URLs when Building XML Sitemaps

Normally, filtering of URLs are done by crawler during website scan, e.g. through list filters, filtering session IDs in URLs and robots.txt file, nofollow and noindex.

Note: Depending on sitemap generator configuration, not all URLs shown in the website tree view will necessarily get included in generated sitemaps. You can control this behavior through both website scan and sitemap generation options:

• Disable Scan website | Crawler options | Apply "webmaster" and "output filters" after website scan stops.
• Enable Create sitemap | Document options | Remove URLs excluded by "webmaster" and "list" filters.

In addition to general filtering, you can also exclude URLs when building sitemap files (including HTML sitemaps and XML sitemaps) based on HTTP response code. In default configuration, only URLs with valid response codes are included by the sitemap builder tool. There are a few specific exceptions depending on the sitemap file type built, but generally A1 Sitemap Generator takes great care only to include valid URLs.

For instance, URLs that redirect with e.g. response 301 : Moved Permanently are not included when building XML sitemaps.

Which response codes the sitemap builder will accept can be set in option Create sitemap | Documment options.