|
|
Website Analyzer and NetSuite CRM/ERP Websites
If your website is powered by NetSuite, A1 Website Analyzer has to be configured quite precisely to crawl it successfully.
NetSuite Sites and Website Analyzer Program
From feedback received by customers of A1 Website Analyzer and NetSuite we
know one has to configure our software quite carefully. The reason appears to be that NetSuite powered websites reserves
bandwidth and server usage for real visitors and search engines, i.e. unknown crawlers are bandwidth throttled.
While our website analyzer program features lots of intelligent behavior to crawl such websites, NetSuite websites require you configure the crawler engine quite precisely.
While our website analyzer program features lots of intelligent behavior to crawl such websites, NetSuite websites require you configure the crawler engine quite precisely.
Website Analyzer Tool Settings for NetSuite
-
Remove all
analysis
and
output
file extension filters. Use the [-] button until all extensions in both have been deleted.
This will make our website analyzer only use MIME filters.
This is necessary since NetSuite uses various uncommon file extensions for redirects etc.
-
In Scan website | Crawler engine set max simultaneous connections/threads to one.
-
In Crawler engine | Advanced engine settings set/enable GET to be default for page requests.
-
At this point one can do one of two things. Either attempt to mask the website analyzer as a search engine crawler or as a user surfing the website:
- Settings to mimic "user surfing website":
- In General options and tools | Internet crawler set user agent to Mozilla/4.0 (compatible; MSIE 7.0; Win32).
- In Scan website | Webmaster filters disable/uncheck Download "robots.txt" and Obey "robots.txt" file if found.
- Settings to mimic "user surfing website":
