Robots File for Disallow URLs and XML Sitemaps Autodiscovery

Sitemap generator tool can create robots.txt files that contain disallow instructions taken from sitemap generator project exclude path filters. It can also add reference to XML sitemap for autodiscovery.
Help: overview | previous | next

 To see all the options available, you will have to switch off easy mode 

 With options that use a dropdown list, any [+] or [-] button next to adds or removes items in the list itself 

Robots.txt File Example

If you intend to manually create and/or edit robots.txt files, you can see a complete example here with disallow of URLs and XML Sitemaps Autodiscovery.

User-agent: *
Disallow: /home/feedback.php
Disallow: /home/link-us.php?
Disallow: /home/social-bookmark.php?

Sitemap: http://www.example.com/sitemap.xml
Sitemap: http://www.example.com/sitemap-blog.xml

See below for more information.

Robots Disallow URLs with Sitemapper "List" and "Analysis" Filters

If you want our sitemap builder to create your robots.txt file, you will need to read the help about configuring output and analysis filters.

Note: It is only standard path filters that are added to robots.txt file, i.e. filters starting with a single : colon.

exclude path filters

From sitemap generator tool tip:
Text string matches: "mypics". Path relative to root: ":mypics/", subpaths only: ":mypics/*", regex search: "::mypics[0-9]*/"

Robots Text File and XML Sitemaps Autodiscovery

In 2007 it became possible to indirectly submit XML sitemaps to search engines by listing listing them in robots.txt file. This concept is called XML sitemaps autodiscovery, and it is part of the XML sitemaps protocol. To add XML sitemaps autodiscovery to a robots.txt file, add the fully qualified XML sitemap file path like this: Sitemap: http://www.example.com/sitemap.xml. Some complete examples using XML sitemaps autodiscovery in robots.txt file:

If you created multiple XML sitemap files covering different parts of your website:

User-agent: *

Sitemap: http://www.example.com/sitemap.xml
Sitemap: http://www.example.com/sitemap-1.xml

Or refer to the XML sitemap index file instead that links all XML sitemap files:

User-agent: *

Sitemap: http://www.example.com/sitemap-index.xml

Some URLs for more information about robots text file:

Create Robots Text File with Sitemap Generator

Sitemap generator can create robots.txt files.

robots text file disallow urls option

Generated robots.txt files are ready to be uploaded and used by search engines.

sitemap generator robots file
This help page is maintained by

As one of the lead developers, his hands have touched most of the code in the software from Microsys.

If you email any questions, chances are that he will be the one answering them.
A1 Sitemap GeneratorAbout A1 Sitemap Generator

Build all kinds of sitemaps including text, visual HTML / CSS, RSS, XML, image, video, news and mobile for all your websites no matter the platform they use.
Share this page with friends   LinkedIn   Twitter   Facebook   Pinterest   Google+   YouTube  
 © Copyright 1997-2017 Microsys
 Usage of this website constitutes an accept of our legal, privacy and cookies information.