Abstract: If your website is powered by NetSuite, A1 Keyword Research has to configured quite precisely to successfully crawl and create XML sitemaps. Navigate:Keyword Research | Buy | Download | Help Index
NetSuite Websites and Sitemap Generators
From feedback received by customers of A1 Keyword Research and NetSuite we
know one has to configure our sitemapper program quite carefully. The reason appears to be that NetSuite powered websites reserves
bandwidth and server usage for real visitors and search engines, i.e. unknown crawlers are bandwidth throttled.
While our sitemap generator program features lots of
intelligent behavior to crawl such websites, NetSuite websites require
you configure our sitemapper crawler quite precisely.
Sitemap Generator Settings for NetSuite
First read about
sitemap crawler.
Then remove all
analysis
and
output
file extension filters. Use the [-] button until all extensions in both have been deleted.
This will make our sitemap generator only use MIME filters.
This is necessary since NetSuite uses various uncommon file extensions for redirects etc.
In Scan website | Crawler engine set max simultaneous connections/threads to one.
In Crawler engine | Advanced engine settings set/enable GET to be default for page requests.
At this point one can do one of two things. Either attempt to mask the sitemap crawler as a search engine crawler or as a user surfing the website.
Settings to mimic "user surfing website":
In General options | Internet crawler set user agent to Mozilla/4.0 (compatible; MSIE 7.0; Win32).
In Scan website | Webmaster filters disable/uncheck Download "robots.txt" and Obey "robots.txt" file if found.
You can download demo project for creating sitemaps of NetSuite websites.
In our sitemap generator tool, make sure to use File | Load Project
and select the downloaded file netsuite-sitemaps-demo.ini.
After that, just enter which website to scan.
You should be able to scan any NetSuite based website
including generating HTML sitemaps and XML sitemaps.
Online Store
