Microsys
  

Crawl Error Response Code Page URLs with Sitemap Generator Tool

A1 Sitemap Generator has en option to crawl error pages for links. The sitemap generator software has builtin protection against crawling endless error pages.
Help: overview | previous | next

 To see all the options available, you will have to switch off easy mode 

 With options that use a dropdown list, any [+] or [-] button next to adds or removes items in the list itself 

Some websites include important links in pages returned for e.g. 404 - not found errors. You can have A1 Sitemap Generator scan error pages for links by checking option: scan website | crawler options | crawl error pages.

Please note that the program will ignore links relative to current path when analyzing error pages. It does so to avoid getting caught in an endless crawling loop. To understand the reason, take a look at following example of the process in a naive website crawler:
    • Crawler detects url http://www.example.com/directory/ gives 404 - not found.
    • Crawler finds http://www.example.com/directory/ links to directory/something.
    • Crawler concatenates http://www.example.com/directory/ and directory/something into http://www.example.com/directory/directory/something.
    • Crawler detects url http://www.example.com/directory/directory/ gives 404 - not found.
    • Crawler finds http://www.example.com/directory/directory/ links to directory/something.
    • Crawler concatenates http://www.example.com/directory/directory/ and directory/something into http://www.example.com/directory/directory/directory/something.
    • Classic spider trap that continues forever.

To have error page URLs scanned for links, use one of the following kinds instead:
  • /directory/something
  • http://www.example.com/directory/something
This help page is maintained by

As one of the lead developers, his hands have touched most of the code in the software from Microsys.

If you email any questions, chances are that he will be the one answering them.
A1 Sitemap GeneratorAbout A1 Sitemap Generator

Build all kinds of sitemaps including text, visual HTML / CSS, RSS, XML, image, video, news and mobile for all your websites no matter the platform they use.
     
share   LinkedIn   Twitter   Facebook   Pinterest   Google+   YouTube  
 © Copyright 1997-2016 Microsys
 Usage of this website constitutes an accept of our legal, privacy and cookies information.