Herself’s Webtools

There are good bots and bad bots. Some bots crawl your site and stick you in their search engines. The Google bot is your friend. Some bots scrape your site for email addresses, or just to copy your site. Bots are small programs that traverse the web, usually traveling from one link to another and downloading part or all of what they find.

You can tell by looking at your log files when you’ve been botted. Several pages will have been loaded in a very short time by one ip address. Often the pages will be loaded in alphabetical order, or by the link list you provide to various pages.

So if you see a bot has been viewing your website how do you know who it is?
BotSpot: The List of all bots
Kloth.net Bad Bots List
Robotstxt.org, Database of Web Robots
IP Addresses of Search Engine Spiders
Search Engine Robots
List of User-Agents ( Spiders, Robots, Browsers )

What can you do about bad bots? Probably not much. Some hosting services let you ban specific ip numbers from getting to your site. However, bots don’t always come from the same ip number twice.

There is Bot Trap ( I haven’t tried it but is sounds promising)

Fleiner has some tips on how to ban bad bots using your .htaccess file. There are also some bot traps available for download on that site.

** update: I wrote a WP plugin to block most bots WP Security Plugin if you are having trouble with bot registrations try WP plugin bot blocker