Robots File for Disallow URLs and XML Sitemaps Autodiscovery

If you intend to manually create and/or edit robots.txt files, you can see a complete example here with disallow of URLs and XML Sitemaps Autodiscovery.

User-agent: * Disallow: /home/feedback.php Disallow: /home/link-us.php? Disallow: /home/social-bookmark.php? Sitemap: http://www.example.com/sitemap.xml Sitemap: http://www.example.com/sitemap-blog.xml

See below for more information.

First read out help about configuring output and analysis filters in website scan.
It is only standard path filters that are added to robots.txt file, i.e. filters starting with a single : colon.



From sitemap generator tool tip:

Text string matches: "mypics". Path relative to root: ":mypics/", subpaths only: ":mypics/*", regex search: "::mypics[0-9]*/"

In 2007 it became possible to indirectly submit XML sitemaps to search engines by listing listing them in robots.txt file. This concept is called XML sitemaps autodiscovery, and it is part of the XML sitemaps protocol. To add XML sitemaps autodiscovery to a robots.txt file, add the fully qualified XML sitemap file path like this: Sitemap: http://www.example.com/sitemap.xml. Some complete examples using XML sitemaps autodiscovery in robots.txt file:

If you created multiple XML sitemap files covering different parts of your website:

User-agent: * Sitemap: http://www.example.com/sitemap.xml Sitemap: http://www.example.com/sitemap-1.xml

Or refer to the XML sitemap index file instead that links all XML sitemap files:

User-agent: * Sitemap: http://www.example.com/sitemap-index.xml

Some URLs for more information about robots text file:

• Official sitemaps documentation about robots file and autodiscovery.

Sitemap generator can create robots.txt files.



Generated robots.txt files are ready to be uploaded and used by search engines.