


	A1 Website Search Engine

Website Search Engine and NetSuite CRM/ERP Websites

If your website is powered by NetSuite, A1 Website Search Engine has to configured quite precisely to crawl it successfully.

Help: overview | previous | next

To see all the options available, you will have to switch off easy mode

With options that use a dropdown list, any [+] or [-] button next to adds or removes items in the list itself

NetSuite Sites and Website Search Engine Program

From feedback received by customers of A1 Website Search Engine and NetSuite we know one has to configure our software quite carefully. The reason appears to be that NetSuite powered websites reserves bandwidth and server usage for real visitors and search engines, i.e. unknown crawlers are bandwidth throttled.

While our website search engine program features lots of intelligent behavior to crawl such websites, NetSuite websites require you configure the crawler engine quite precisely.

Website Search Engine Tool Settings for NetSuite

Remove all analysis and output file extension filters. Use the [-] button until all extensions in both have been deleted. This will make our website search engine only use MIME filters. This is necessary since NetSuite uses various uncommon file extensions for redirects etc.
In Scan website | Crawler engine set max simultaneous connections/threads to one.
In Crawler engine | Advanced engine settings set/enable GET to be default for page requests.
At this point one can do one of two things. Either attempt to mask the website search engine as a search engine crawler or as a user surfing the website:
- Settings to mimic "user surfing website":
  - In General options | Internet crawler set user agent to Mozilla/4.0 (compatible; MSIE 7.0; Win32).
  - In Scan website | Webmaster filters disable/uncheck Download "robots.txt" and Obey "robots.txt" file if found.

This help page is maintained by Thomas Schulz

As one of the lead developers, his hands have touched most of the code in the software from Microsys.

If you email any questions, chances are that he will be the one answering them.

About A1 Website Search Engine

By giving your offline or online website a capabale search engine, you can ensure more of your visitors stay on your site. Having a search box helps visitors find what they are searching for.