Microsys
  

Forums and Website Crawling with Website Search Engine

Crawling website for blogs and forums such as SMF, VBulletin etc. can sometimes take a long time. However, proper configuration of website search engine can speedup website forum scan.
Help: overview | previous | next

 To see all the options available, you will have to switch off easy mode 

 With options that use a dropdown list, any [+] or [-] button next to adds or removes items in the list itself 

General Website Search Engine Tips for Crawling Forums and Blogs

Forums and blogs are no different from other websites. Rarely will you ever need to configure website search engine in a special way. However, here is a list of common topics for large and/or database websites:

  • How some website platforms cause crawling problems.
  • Use resume scan support in our website search engine tool.
    • Notice that you can improve resume by disabling option Scan website | Crawler options | Apply "webmaster" and "output filters" after website scan stops.
  • About crawling and finding links in websites.
  • Adjusting server load and website crawl speed.
  • Including content otherwise only available for subscribers using password protected pages.
  • Use output filters to exclude certain URLs from being included website scan output.
  • Use analysis filters to prevent certain URLs in being crawled / analyzed.


Website Search Engine Example Settings for Popular Forums and Blogs

The following settings are for demonstration purposes. Most likely you will never need to configure these options. Should you need to configure settings, take time to investigate above links and what you need. Then possibly look at underneath for inspiration. Remember, few blogs and forums are exactly the same.

Note: There may already be Quick presets... available in Scan website that match your website platform and crawl needs.

Note: If in doubt what Login path and Post form data corresponds to see the help page about password protected pages and login.

List of examples:
  • phpbb
    • Configure login
      • Login path : http://forum.example.com/login.php
      • Post form data : username=yourusername&password=yourpassword&redirect=index.php?&login=Log in
    • Configure crawler/analysis and output/list exclude filters
      • Necessary
        • :login.php?logout
      • Recommended
        • :profile.php
        • :login.php
        • :newreply.php
        • :printthread.php
        • :sendmessage.php
        • :search.php
        • :threadrate.php


  • vBulletin
    • Configure login
      • Login path : http://forum.example.com/login.php?do=login
      • Post form data : vb_login_username=yourusername&vb_login_password=yourpassword&cookieuser=1&s=&do=login&vb_login_md5password=&vb_login_md5password_utf=
    • Configure crawler/analysis and output/list exclude filters
      • Necessary
        • :login.php?logout
      • Recommended
        • :profile.php
        • :login.php


  • WordPress
    • Configure login
      • Login path : http://blog.example.com/wp-login.php
      • Post form data : log=yourusername&pwd=yourpassword&rememberme=forever&wp-submit=Log+ind&redirect_to=wp-admin%2F&testcookie=1
    • Configure crawler/analysis and output/list exclude filters
      • Necessary
        • :wp-admin/
        • :wp-login.php?action=logout
      • Recommended
        • :wp-login.php
      • Note
        • If you do not exclude "admin" section using filters, try avoid edit, post, delete, trash, logout and related link types.
This help page is maintained by

As one of the lead developers, his hands have touched most of the code in the software from Microsys.

If you email any questions, chances are that he will be the one answering them.
A1 Website Search EngineAbout A1 Website Search Engine

By giving your offline or online website a capabale search engine, you can ensure more of your visitors stay on your site. Having a search box helps visitors find what they are searching for.
     
Share this page with friends   LinkedIn   Twitter   Facebook   Pinterest   Google+   YouTube  
 © Copyright 1997-2016 Microsys
 Usage of this website constitutes an accept of our legal, privacy and cookies information.