A1 Sitemap Generator - Website crawler webmaster filters
 |
 |
 |
|
|
|
|
|
 |
|
 |
Webmaster filters overview
A1 Sitemap Generator supports:
- The robots.txt file.
- Meta tag nofollow.
- Meta tag noindex.
- Anchor tag nofollow.
Webmaster filters with robots.txt
- If checked the robots.txt option, A1 Sitemap Generator will respond to:
- Exact user agent id match as defined in website crawler identification.
- If the user agent id contains A1 Sitemap Generator that is matched as well
- * (all crawlers)
Robots.txt file filters and wildcards
The match behavior in the website crawling engine used by A1 Sitemap Generator is similar to most search engines.
This includes support for wildcard symbols in robots.txt file.
-
Standard: Match from beginning to length of filter.
gre will match: greyfox, greenfox and green/fox. -
Wildcard *: Match any character until another match becomes possible.
gr*fox will match: greyfox, grayfox, growl-fox and green/fox.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|