Herself's Webtools

A friend runs a local forum and she mentioned several GoogleBots were hanging around all day and night on her forum. I thought this a bit odd so I did some digging. Several forums about running forums turned up lots of GoogleBots hanging out at forums day and night. Many of these forums were still new and low traffic.

Further investigation turned up Instructions on how to surf as the GoogleBot to gain access to restricted areas of forums. Not so nice. This will often let you into restricted areas and avoid detection while on a forum. Forum webmasters need to make sure that the GoogleBot is not given extra privileges to their sites with out first verifying it is in fact Google.

A user need only install User Agent Switcher to Firefox to hide behind Google so this means anyone who can install a plugin can pull this off. Slightly more sophisticated users can do this with IE Become a GoogleBot get a free pass. Further instructions are at GoogleBot Extension.

All these howtos I’ve posted are not to tell your forum members how to hide behind GoogleBot, they already know. These are to show you, the forum master what those pretend GoogleBots are able to do so that you can properly lock down your forum. So fire up your browser, set it up to hide as Google and see what you can get into on your forum. Make sure it is only areas the general public should be able to access.

** update: I wrote a WP plugin to block most bots WP Security Plugin if you are having trouble with bot registrations try WP plugin bot blocker

I’ve one site that seems to attracts bots. It was attracting bots when it was just a directory on TimesToCome eight years ago. Who’d've thought house plants would be such an attraction for evil bots? There are several sites with directions for building bot traps. The main idea is the same for all.

First you create a bot_trap directory. Second you add a line to your robots.txt file telling legitimate bots not to enter that directory.

User-agent: *
Disallow: /bot_trap/

Then you put a hidden link in one of your busy pages, perhaps the main page or archives page that humans can’t see but that bots can see. <a href=”/bot_trap/got_you.php”><img src=”/images/1_pixel_clear.gif” border=”0″></a>. Since most evil bots are aware of the clear image you should give your 1×1 pixel image a different name.

Lastly you create a got_you.php file and place it in your bot_trap directory. What to put in your got_you.php file is a matter of taste. You can have the got_you.php file, email you, you can ban the ip address by adding it to your .htaccess file, or you can do evil deeds to the bots ip address.

PHP email and ban with .htaccess examples
Kloth: Bot trap emails you and bans ip number
735: Ban bad bots and email yourself

Or you can just write the bans into your .htaccess file yourself.
A close to perfect .htaccess ban list
Block known evil bots by manually re-writing .htaccess

An alternative for those of you not comfortable messing with .htaccess is to add a page ( The ultimate spam bot bait and trap page ) to your site with a hidden, banned link.

You do need to be careful. The smallest mistake in .htaccess will take down your site. Also you don’t want .htaccess to get so large it slows down your website. Another concern is that you don’t want to trap and ban good bots. Banishing Googlebots, sending Yahoo bots to the depths of the internet will not do your search engine rankings any favors.

Since my bot attacks are not coming from known bots I’ll be using one of the php trap and ban versions on my sites.

More information:
Everything you wanted to know about bots