blogZero - FreeBSD

Authenticating a Googlebot in PHP and Perl

dwclifton@gmail.com (Douglas Clifton) — Sun, 24 Sep 2006 18:08:26 -0400

Following a tip from Russ I was pleased to find an interesting post on the Official Google Webmaster Central Blog titled How to verify Googlebot. In a nutshell, it explains how to use the Unix shell program host to authenticate that an IP address copied from your Web server log file really is a Googlebot and not some email harvester (or whatever).

I decided to take this a step further and demonstrate how you can automate this procedure using a scripting language. For these examples I chose PHP and Perl, although you could certainly use Python or Ruby or whatever your preferred language is, as long as it has an interface to the gethostbyname and gethostbyaddr system calls.

Using these calls under PHP is the simpler of the two approaches, as the interface to these routines are written at a more abstract level than using the Perl Socket module. Below is an example googlebot() function in PHP that returns true if the IP address parameter authenticates, although there is no 100% guarantee of preventing a spoof getting through (but it will catch the vast majority of them). A bit of test code is included.

<?php

function googlebot($ip)  {

    // check to see if this IP really is a Googlebot

    $bot = 'googlebot.com';
    $name = gethostbyaddr($ip);
    if ($name == $ip) return false;

    return (strpos($name, $bot) !== false and gethostbyname($name) == $ip) ? true : false;
}

// test it

$ip = '66.249.66.1';

echo $ip . ' is ';
if (!googlebot($ip)) echo 'not ';
echo 'a Google bot' . "\n";
?>

The Perl version is at a much lower level, very similar to the corresponding C system calls. In fact, the module is derived directly from the sys/sockets.h header file and the functions are just wrappers around these Standard C library calls. See Berkeley Sockets for more information. If you have a copy of Programming Perl, the chapter 16 Interprocess Communications section on socket programming will help, and if you are lucky enough to have a copy of the Perl Cookbook, chapter 18 Internet Services has some great recipes for DNS lookups. For really gory details, refer to chapter 14 DNS: The Domain Name System of TCP/IP Illustrated, Volume I—The Protocols.

#!/usr/bin/perl

use Socket;

sub googlebot($)  {

    # check to see if this IP really is a Googlebot

    my $ip = shift;
    my $bot = 'googlebot\.com';
    my $name = gethostbyaddr(inet_aton($ip), AF_INET) or return 0;
    my @addr = gethostbyname($name);
    my $addr = inet_ntoa($addr[4]);

    return ($name =~ m/$bot/ and $ip eq $addr) ? 1 : 0;
}

# test it

$ip = '66.249.66.1';

print $ip . ' is ';
unless (googlebot($ip)) { print 'not '; }
print 'a Google bot' . "\n";

Finally, in case anyone is interested why it's been so long since I posted anything, much of the summer I was sick as a dog and since recovering, busy as a bee. It's nice to be feeling better and back to work!

Advanced crontab Tutorial

dwclifton@gmail.com (Douglas Clifton) — Fri, 28 Apr 2006 00:00:00 -0400

Most crontab entries are very simple. For instance, "run this script once every day exactly at midnight." To do this you would add the following line to your crontab file:

0 0 * * * /path/to/script

Depending on your version of cron, you could also use the shorthand string @midnight or @daily. But what if you need to run a script between the hours of 8am and 6pm every two hours, but only on Mondays through Saturdays? That's exactly the problem I had to solve earlier today. Although I use cron quite a bit, I needed a quick syntax refresher course covering some of the more advanced features.

So naturally (like we all do) I reached for my browser and fired up Google. After 20 wasted minutes trying various keyword search combinations, wading several result pages deep and following links, it hit me. Do'h! All the information I needed was right on my FreeBSD server. Man pages contain a wealth of information about your system and have been around for many years. Decades actually. Many people forget that Unix was originally designed as a text processing and technical manual printing system. The entire system is documented using these manual pages and they are designed for viewing, searching and printing. I once kept large chunks of the standard C library in a 3-ring binder.

Since crontab is a file format, I went back to my shell and entered:

$ man 5 crontab

And within a matter of minutes I had the solution. The point of all of this is sometimes we are over reliant on the Web and search. Often the information we are seeking has been there all along. Call me old school, I still like books too.

By the way, the crontab entry looks like this:

0 8-18/2 * * 1-6 /path/to/script

If you really have to use a browser, try this "advanced crontab tutorial."

How ironic, I posted this exactly at midnight.

Long Live the King

dwclifton@gmail.com (Douglas Clifton) — Thu, 18 Aug 2005 00:15:33 -0400

When Bell Labs finally closed its doors on Department 1127 this month, it didn't signal the end of Unix. Perhaps the end of an operating system research environment at Lucent, as most of the original team are long gone to other ventures (Google, Pixar, NASA/JPL, Princeton, Dartmouth...) or retired anyway.

Hardly. Unix was first developed around 1970 (aka the "epoch"), and the original concepts and many of the technologies are stronger than ever. The vast majority of the servers that drive the Internet and the Web are some form of Unix operating system. Hell, we wouldn't have an Internet if wasn't for the role that Unix played in it.

I use Linux, FreeBSD and OS X every day. Both as servers and as desktop environments. There are countless other Unix variants, both open-source (OpenBSD, NetBSD, DragonflyBSD, Darwin...) as well as commercial ones (Sun, HP...).

And all of this, thanks to a couple of guys who were allowed to tinker with an old PDP-11 in exchange for developing an electronic typesetting system so AT&T could publish their own technical manuals. Oh, and while they were at it they threw in a little thing called the C programming language. But that's another story.

The King is dead, long live the King!