Posted by ljmacphee on October 31, 2007 under security, things you should know | 
A friend runs a local forum and she mentioned several GoogleBots were hanging around all day and night on her forum. I thought this a bit odd so I did some digging. Several forums about running forums turned up lots of GoogleBots hanging out at forums day and night. Many of these forums were still new and low traffic.
Further investigation turned up Instructions on how to surf as the GoogleBot to gain access to restricted areas of forums. Not so nice. This will often let you into restricted areas and avoid detection while on a forum. Forum webmasters need to make sure that the GoogleBot is not given extra privileges to their sites with out first verifying it is in fact Google.
A user need only install User Agent Switcher to Firefox to hide behind Google so this means anyone who can install a plugin can pull this off. Slightly more sophisticated users can do this with IE Become a GoogleBot get a free pass. Further instructions are at GoogleBot Extension.
All these howtos I’ve posted are not to tell your forum members how to hide behind GoogleBot, they already know. These are to show you, the forum master what those pretend GoogleBots are able to do so that you can properly lock down your forum. So fire up your browser, set it up to hide as Google and see what you can get into on your forum. Make sure it is only areas the general public should be able to access.
Posted by ljmacphee on October 29, 2007 under security, tools |
** update: I wrote a WP plugin to block most bots WP Security Plugin if you are having trouble with bot registrations try WP plugin bot blocker
I’ve one site that seems to attracts bots. It was attracting bots when it was just a directory on TimesToCome eight years ago. Who’d've thought house plants would be such an attraction for evil bots? There are several sites with directions for building bot traps. The main idea is the same for all.
First you create a bot_trap directory. Second you add a line to your robots.txt file telling legitimate bots not to enter that directory.
User-agent: *
Disallow: /bot_trap/
Then you put a hidden link in one of your busy pages, perhaps the main page or archives page that humans can’t see but that bots can see. <a href=”/bot_trap/got_you.php”><img src=”/images/1_pixel_clear.gif” border=”0″></a>. Since most evil bots are aware of the clear image you should give your 1×1 pixel image a different name.
Lastly you create a got_you.php file and place it in your bot_trap directory. What to put in your got_you.php file is a matter of taste. You can have the got_you.php file, email you, you can ban the ip address by adding it to your .htaccess file, or you can do evil deeds to the bots ip address.
PHP email and ban with .htaccess examples
Kloth: Bot trap emails you and bans ip number
735: Ban bad bots and email yourself
Or you can just write the bans into your .htaccess file yourself.
A close to perfect .htaccess ban list
Block known evil bots by manually re-writing .htaccess
An alternative for those of you not comfortable messing with .htaccess is to add a page ( The ultimate spam bot bait and trap page ) to your site with a hidden, banned link.
You do need to be careful. The smallest mistake in .htaccess will take down your site. Also you don’t want .htaccess to get so large it slows down your website. Another concern is that you don’t want to trap and ban good bots. Banishing Googlebots, sending Yahoo bots to the depths of the internet will not do your search engine rankings any favors.
Since my bot attacks are not coming from known bots I’ll be using one of the php trap and ban versions on my sites.
More information:
Everything you wanted to know about bots
Posted by ljmacphee on October 26, 2007 under how to, osx, wordpress |
Gad, it is just as miserable and painful as it was under Panther.
1) Install MySQL 4.1 or 5.x
2) Turn on PHP in Apache
3) Fix broken socket
4) Download and install WordPress
Step 1 MySQL:
- Download MySQL 4.1
Unpack it and install and tell it to start on startup.
- Turn on SU account. Open up a Terminal Window and type:
sudo passwd root
Give it a good password. You’ll need su to edit files in a bit
- Configure MySQL
cd /usr/local/mysql/bin
./mysqladmin -u root password
[Give it a good password, if it refuses to connect reboot your computer. Also check in your System Preferences MySQL that MySQL is running and told to turn on at boot. ]
- Once you have the root user set up on MySQL set up an account for yourself
./mysql -u root -pYOURPASSWORD
- You should now see a sql> prompt
GRANT ALL PRIVILEGES ON *.* TO username@localhost IDENTIFIED BY ‘userPassword’ WITH GRANT OPTION;
exit;
- Make sure it worked
mysql -u username -puserPassword
Did you get a sql> prompt? If so you’re cool, if not reboot, I’m finding that helps and if not try again. You need to get MySQL running to use Wordpress.
- Life is much easier if you set up a link in your path to mysql
as su
cd /usr/bin
ln -s /usr/local/mysql/bin mysql
Now you should be able to access mysql from any where on the computer.
Step 2: Turn on PHP in Apache
- Turn on WebSharing in System Preferences Sharing
cd /etc/apache2
- Edit httpd.conf , you need to uncomment the line
LoadModule php5_module libexec/apache2/libphp5.so
Thanks and more info here if you are stuck
Step 3 Fix broken socket
- cd/etc/
- copy php.ini.default to php.ini
- edit php.ini
Change the line
mysql.default.socket =
to
mysql.default.socket = /tmp/mysql.sock
Thanks and more help here if you need it for this step
Step 4 Wordpress:
- Download and unpack your favorite version to /Library/WebServer/Documents
- Edit config.php to use the password, and user name and database you created in step 1. If you didn’t create a database do so now.
- type 127.0.0.1/wordpress/ in your browser window and you should see the familiar WP set up screen.
If you do not see the set up screen try this: create this test.php file and it should tell you what is your problem:
if ( $cn = mysql_connect ( “localhost”, “username”, “password” )){
echo “connected”;
}else{
echo “not connected - ” .mysql_error();
}
Use the username and password you set up in mysql.
NOTES:
If you decide to wipe and re-install everything on your computer, you must get PHP/Apache/MySQL up and running BEFORE you do the 10.5.1 update.
If after installing MySQL you are unable to create a database ( errno 13 ) then log on to a terminal as root and change the owner / group of everything in the mysql directories to mysql/mysql.
> chgrp -R mysql *
> chown -R mysql *
You must also create a var and run directory in your /usr/local/mysql directory and make sure they are owned and in the group mysql.
Posted by ljmacphee on under security, tools |
There are good bots and bad bots. Some bots crawl your site and stick you in their search engines. The Google bot is your friend. Some bots scrape your site for email addresses, or just to copy your site. Bots are small programs that traverse the web, usually traveling from one link to another and downloading part or all of what they find.
You can tell by looking at your log files when you’ve been botted. Several pages will have been loaded in a very short time by one ip address. Often the pages will be loaded in alphabetical order, or by the link list you provide to various pages.
So if you see a bot has been viewing your website how do you know who it is?
BotSpot: The List of all bots
Kloth.net Bad Bots List
Robotstxt.org, Database of Web Robots
IP Addresses of Search Engine Spiders
Search Engine Robots
List of User-Agents ( Spiders, Robots, Browsers )
What can you do about bad bots? Probably not much. Some hosting services let you ban specific ip numbers from getting to your site. However, bots don’t always come from the same ip number twice.
There is Bot Trap ( I haven’t tried it but is sounds promising)
Fleiner has some tips on how to ban bad bots using your .htaccess file. There are also some bot traps available for download on that site.
** update: I wrote a WP plugin to block most bots WP Security Plugin if you are having trouble with bot registrations try WP plugin bot blocker