MiggiBot™ website crawler engine
Abstract: About MiggiBot website crawler technology
Created: 2006-01-01 (yyyy/mm/dd)
Updated: 2006-04-24
Author: Thomas Schulz
|
Current feautures - MiggiBot capabilities today
- Can extract vast amounts of meta data about all pages in a website. File sizes, download times, server response codes etc.
- Has various tools for analyzing keyword density. Can use stop words and similar filters.
- Can generate sitemaps of websites with the crawler supporting advanced filters.
- Can check, validate and store links, redirects and even source from data (e.g. images and frames).
- Can calculate the internal importance of all pages within a site.
- Can generate CSV and XML files for all crawled and derived data.
- Has website download and links conversion functionality.
Future development - MiggiBot adventures tomorrow
- Continue to develop and polish the engine.
- Some very exotic and secret ideas.
Demonstrations - MiggiBot technology in use
Licensing - For developers by developers
- Generated output XML data
- Having easy-to-parse data can be useful for companies building custom research tools.
- See A1 Website Analyzer for more information.
- Generated output XML data
- We are currently considering if and how to expose our crawler engine for developers.
- Feel free to contact us if you are interested.
About Miggi / MiggiBot - fun facts
The name is a contraction of following words:
Micro-Sys,
Microsys,
microbot, microbotic, robot, bot, big etc.
Language
