Microsys
  

Schedule and Automate Website Scraper Tool with Command Line

Automate website scrape in A1 Website Scraper. Download websites using automation, e.g. during the night.
Help: overview | previous | next

 To see all the options available, you will have to switch off easy mode 

 With options that use a dropdown list, any [+] or [-] button next to adds or removes items in the list itself 

Command Line Support in A1 Website Scraper

You can use a command line interface to automate all the major website scraper tools in the program.

This means that you can also use external applications and bat / batch / script files.

This enables you to run our website scraper software at regular intervals using e.g. Windows Task Scheduler.
  • Parameters:
    • ":%project-path%" : Where %project-path% should contain the path to the project. (Remember the colon before project path.)
    • "@override_rootpath=http://example.com@" : Overwrite website rootpath.
    • "-exit" : Exits when done.
    • "-hide" : Always invisible and exits when done.
    • "-scan" : Runs website scanner.
    • "-stop0000" : Stops scan after a number of seconds, e.g. -stop600 stops scan after 10 minutes.
    • "-save" : Saves project.
    • "@override_exportpathdir=c:\example\exports\@" : Overwrite general directory path used for e.g. CSV export data files.
    • "-exportsitemapcsv" : Exports all URLs data listed in "sitemap" tree view into a file called "sitemap.csv" located in project directory.
    • "-exportexternalcsv" : Exports all URLs data listed in "external" tree view into a file called "external.csv" located in project directory.

  • Examples for usage on Windows:
    • [ "c:\microsys\website\scraper.exe" -exit -scan -build -save ":c:\microsys\website\scraper\my-project.ini" ].
    • [ "Scraper.exe" -exit -scan -build -save ":my-project.ini" ] (Good for testing when placing all files in the same directory.)
    • [ "Scraper.exe" -scan -build @override_rootpath=http://example.com@ ]

  • Examples for usage on Mac OS:
    • [ open A1WebsiteScraper.app --args -scan -build @override_rootpath=http://example.com@ ].

  • Tips:
    • To prevent a parameter value that contains spaces (e.g. if you are passing a directory path) from being broken up, enclose it inside a couple of "".


Automate Website Scraper with Windows Command Line and Batch Files

  • Create a batch file, e.g. "batch-file.bat", using any standard text editor
  • Example of what to write underlined: [ "c:\microsys\website\scraper.exe" -exit -scan -build -save ":c:\microsys\website\scraper\my-project.ini" ].
  • Save your batch file. You can now call it yourself or from other programs and scripts.


Schedule and automate Website Scraper with Windows Task Scheduler


schedule automate website scraper

  • Open Control Panel | Scheduled Tasks | Add Scheduled Task. Follow the guide.
  • Open the generated website scraper time scheduled item to edit details, e.g. command line parameters.
This help page is maintained by

As one of the lead developers, his hands have touched most of the code in the software from Microsys.

If you email any questions, chances are that he will be the one answering them.
A1 Website ScraperAbout A1 Website Scraper

Extract data from sites into CSV files. By scraping websites, you can grab data on websites and transform it into CSV files ready to be imported anywhere, e.g. SQL databases
     
Share this page with friends   LinkedIn   Twitter   Facebook   Pinterest   Google+   YouTube  
 © Copyright 1997-2016 Microsys
 Usage of this website constitutes an accept of our legal, privacy and cookies information.