Abstract: Crawling website for blogs and forums such as SMF, VBulletin etc. can sometimes take a long time. However, proper configuration of keyword research can speedup website forum scan. Navigate:Keyword Research | Buy | Download | Help Index
General Tips Creating Sitemaps for Forum Websites
Forums and blogs are no different from other websites. Rarely will you ever need to configure keyword research in a special way.
However, here is a list of common topics for large and/or database websites:
Use
resume scan support in sitemap generator tool.
Notice that you can improve resume by disabling option
Scan website | Crawler options | Apply "webmaster" and "list" filters after website scan.
Pages marked noindex can still be filtered off when creating the forum XML sitemap.
Including content otherwise only available for subscribers using password protected pages.
Use output filters to exclude certain URLs from being included in generated forum sitemaps.
Use analysis filters to prevent certain URLs in being crawled / analyzed.
Keyword Research Example Settings for Popular Forums and Blogs
The following settings are for demonstration purposes.
Most likely you will never need to configure these options.
Should you need to configure settings, take time to investigate above
links and what you need. Then possibly look at underneath for inspiration. Remember, few blogs and
forums are exactly the same.
phpbb
Configure login
Login path : http://forum.example.com/login.php
Post form data : username=yourusername&password=yourpassword&redirect=index.php?&login=Log in
Configure crawler/analysis and output/list exclude filters
Post form data : vb_login_username=yourusername&vb_login_password=yourpassword&cookieuser=1&s=&do=login&vb_login_md5password=&vb_login_md5password_utf=
Configure crawler/analysis and output/list exclude filters
Necessary
:login.php?logout
Recommended
:profile.php
:login.php
WordPress
Configure login
Login path : http://blog.example.com/wp-login.php
Post form data : log=yourusername&pwd=yourpassword&rememberme=forever&submit=Login+%C2%BB&redirect_to=wp-admin%2F
Configure crawler/analysis and output/list exclude filters
Necessary
:wp-login.php?action=logout
Recommended
:wp-login.php
Note
Avoid edit, delete, logout and related link types.
