Firewall Software Causes Problems for Automation Batch Tools

Most firewall software programs default to silently block all internet enabled applications unless explicitly specified otherwise in configuration. This includes most Windows programs such as our automation batch tools program.

• If you get no URLs found in website crawl and related tools, firewalls are often the reason.
• If you get flaky results, odd errors etc. net traffic filtering can be the reason.

NOD32 client version 3:

• View advanced mode
• Select Setup
• Click - Antivirus and antispyware
• In Web access protection click Configure
• Expand HTTP and click Web browsers
• NOD32 will automatically consider Automation Batch Tools as web browser (checked) - you must uncheck it for Automation Batch Tools to work.

ESET Smart Security:

• Set it to learning mode

Kaspersky anti-virus:
Symptoms:

• URL timeouts with Indy HTTP engine.
• URL 404 response codes wiith WinInet HTTP engine

Solution:

• Pause it

Hardware firewall:
Symptoms:

• Various errors and/or no crawling

Solution: Mimic user browser behavior (like some other programs also do):

• In Scan website | Crawler engine to HTTP using WinInet engine and settings (Internet Explorer)
• In General Options | Internet Crawler to Mozilla/4.0 (compatible; MSIE 8.0; Win32)
• In Scan website | Crawler engine lower amount of simultaneous conections, possibly all down to one.