Crawl HTTPS Websites with A1 Website Download
Scan and crawl websites that use HTTPS or mix HTTP and HTTPS with our website download software.
A1 Website Download allows the user to select a
HTTP solution in
Scan website | Crawler engine.
The default setting is
Auto detect which translates to:
- Windows: The setting HTTP using Windows API
- Mac: The setting HTTP using Mac API
It is also possible to select
HTTP using Indy library which is an alternative solution.
Tip:
If you have problems getting crawling working,
be sure to check if
A1 Website Download is getting blocked by
firewalls or similar solutions.
Note:
This section is
only necessary if:
- Your website uses HTTPS.
- You use Indy in Scan website | Crawler engine.
Configuring
OpenSSL / Configuring
LibreSSL for use with A1 Website Download will help for
all
HTTPS /
SSL based websites.
To add support for this, see
General options and tools | Tool paths.
Clicking the button at the right will show a menu with information and links.
In newer versions of A1 Website Download the menu will also show which version you should download for your computer system.
While this will usually work
out of the box, you may sometimes need to do some configuration, especially on older systems
This will mainly be in
Tools | Internet Options | Advanced | Security:
Windows 11:
- Download and apply all Windows updates, e.g. by using Windows Update.
- Enable TLS 1.1, TLS 1.2, TLS 1.3 and newer if available in Windows internet settings at Tools | Internet Options | Advanced | Security.
Windows 10:
- Download and apply all Windows updates, e.g. by using Windows Update.
- Enable TLS 1.1, TLS 1.2, TLS 1.3 and newer if available in Windows internet settings at Tools | Internet Options | Advanced | Security.
- If crawling using the embedded system browser option, download the Edge / Chromium a.k.a. WebView2 runtime from Microsoft:
Windows 8.1 without Internet Explorer 11:
- Download and apply all Windows and IE updates, e.g. by using Windows Update.
- Enable TLS 1.1, TLS 1.2 and newer if available in Windows / IE internet settings at Tools | Internet Options | Advanced | Security.
- If crawling using the embedded system browser option, download the Edge / Chromium a.k.a. WebView2 runtime from Microsoft:
Windows 8 without Internet Explorer 11:
- Download and apply all Windows and IE updates, e.g. by using Windows Update.
- Enable TLS 1.1, TLS 1.2 and newer if available in Windows / IE internet settings at Tools | Internet Options | Advanced | Security.
Windows 7:
- Download and apply all Windows and IE updates, e.g. by using Windows Update. You at minimum need to use SP1 / service pack one.
- Enable TLS 1.1, TLS 1.2 and newer if available in Windows / IE internet settings at Tools | Internet Options | Advanced | Security.
- If crawling using the embedded system browser option, download the Edge / Chromium a.k.a. WebView2 runtime from Microsoft:
Windows Vista:
- Download and apply all Windows and IE updates, e.g. by using Windows Update.
- Enable TLS 1.1, TLS 1.2 and newer if available in Windows / IE internet settings at Tools | Internet Options | Advanced | Security.
- Experts: If you do not see these options then check these guides:
Windows XP and derivatives:
- Download and apply all Windows and IE updates, e.g. by using Windows Update.
- Experts: Investigate if Windows Embedded POSReady 2009 is applicable to you and that you are eligible.
- Experts: A registry setting essentially turns your system into a different edition for which updates were released until 2019 April.
- Experts: You can get started learning more here:
- Download and apply all Windows and IE updates again, e.g. by using Windows Update.
- Enable TLS 1.1, TLS 1.2 and newer if available in Windows / IE internet settings at Tools | Internet Options | Advanced | Security.
- Experts: If you do not see these options then check this guide:
Note: Windows systems with IE 8 or less will still not work with websites using
elliptic curve cryptography.
Alternative solution: Use
Indy as the chosen crawler engine and configure as explained further above.