Microsys
  

Import and Crawl List of Pages in Website With Keyword Research

Explains the easiest way to setup the keyword Research program to crawl and analyze a list of specific pages from a website.
Help: overview | previous | next

 To see all the options available, you will have to switch off easy mode 

 With options that use a dropdown list, any [+] or [-] button next to adds or removes items in the list itself 

Import The Page URLs You Want Keyword Research to Handle

Before doing anything else, you will first have to import the list of pages you want. You can do so from the File menu.

import list of pages

Select a file containing the list of URLs you wish to import. It can be in a variety of formats including .CSV, .SQL and .TXT.

The software will automatically (try to) determine which URLs go into the internal and external tabs.

It will do so by recognizing if the majority of the imported URLs are:
  • From the same domain and place those in the internal category tab. (The rest will be ignored.)
  • From multiple domains and place those in the exernal category tab. (The rest will be ignored.)


Simple Way to Crawl Imported URLs

Tick either Scan website | Recrawl (full) or Scan website | Recrawl (listed only). The latter of the two options will avoid including any new URLs for analysis or scan results.

You can now Start scan

Note: If you want to have the URLs in the external tab recrawled, see further below.


Further Limit The Crawl of Internal URLs

Note: You can skip this step if you are only interested in external URLs.

Use the button shown in the picture below to quickly add all selected website URLs to analysis filters and output filters.

This button sets limit include to in both of the above filter types.

Note:
When selecting and filtering items, it is often easier to switch the left view to list mode and sort after the response code or similar. In tree mode, the filter button will try optimize the "limit" filter, e.g. by only filtering the "root" directories selected.

Note:
If you want to have URLs checked that are not in the imported list, you will need to ensure the crawler is allowed to analyze and include them in results. An example could be to add image directories if you want to have linked/used images included.

import list of pages

Note: Remember to keep option Scan website | Crawler options | Apply "webmaster" and "ouput" filters after website scan stops checked if you use output filters. That way, only the URLs you are interested in will be shown after the site crawl has finished.

Note: If you forgot to use one of the recrawl and you use limit crawl to filters, the scan may even be unable to start its scan since all the "start crawl from" URLs are excluded.


More Options to Fine Tune Crawl

  • In case you want to have external URLs checked:
    • Untick the Scan website | Crawler Engine | Default to GET for page requests option.
    • Tick the Scan website | Data collection | Store found external URLs option.
    • Tick the Scan website | Data collection | Verify external URLs (and analyze if applicable) option.

  • In case you want to have external URLs analyzed:
    • Tick the Scan website | Crawler Engine | Default to GET for page requests option.
    • Tick the Scan website | Data collection | Store found external URLs option.
    • Tick the Scan website | Data collection | Verify external URLs (and analyze if applicable) option.
    • Note: Only external URLs found in the list imported will be analyzed.

  • Depending on the server(s) and need(s) for which you are checking URLs, you may want to switch off Default to GET for page requests in Scan website | Crawler engine. If you just want to check URLs, using HEAD instead of GET for HTTP requests may be sufficient.


Start The Crawl and View The Results

  • Hit the start scan button.

    import list of pages

  • Wait for the scan to finish.

  • View results.

    Note: It is usually easier viewing the results when switching the left view to list mode.

  • If you want to export the results, see the help page about exporting data to CSV files.
This help page is maintained by

As one of the lead developers, his hands have touched most of the code in the software from Microsys.

If you email any questions, chances are that he will be the one answering them.
A1 Keyword ResearchAbout A1 Keyword Research

Complete PPC and SEO keywords toolset. Position check including history, analyze competition, analyze keyword density and content, organize and combine keyword lists and much more.
     
Share this page with friends   LinkedIn   Twitter   Facebook   Pinterest   Google+   YouTube  
 © Copyright 1997-2016 Microsys
 Usage of this website constitutes an accept of our legal, privacy and cookies information.