blogZero - Perl

W3C LogValidator

dwclifton@gmail.com (Douglas Clifton) — Tue, 04 Nov 2008 02:33:00 -0500

This article documents my experience installing, configuring, and using the W₃C LogValidator. Hopefully it will be useful to anyone new to this, and in particular those who are not comfortable installing Perl CPAN modules.

Goals

When I set out on this project I wanted to satisfy the following criteria. First, determine the N most popular documents where N is some configurable number. And second, amongst those pages determine:

Is the markup, be it HTML or XHTML, valid?
Are the CSS stylesheets valid?
Do any of them have broken links?

As it turns out, all three of these criteria can be met with LogValidator and they correspond with the following modules:

Both the HTML and CSS modules use the W3C validation services that most developers are familiar with, so there is no need to install HTML Tidy or any complicated Java packages. For the LinkChecker module to function, you also need to install checklink (which is a useful little utility on its own).

All this is fine and dandy, but how do you view the results? I'll get to this in more detail below, but the two most popular methods are to have LogValidator generate an HTML document, or send you an email—which is handy if you want to set-up the system to run automatically using cron.

Installation

This is command-line land folks, so I'm hoping you are comfortable using a shell. If you are, and you do have any experience installing Perl CPAN modules, you should be all set. Otherwise you may consult a complete session of my own installation experience. Good luck with all that, it's kind of gory.

First, you need the W3C::LogValidator module, which may or may not require some other prerequisite modules. Thankfully, the CPAN module itself is smart enough to determine if anything is missing and automatically install them as well. I'm not going to get into installing this software in non-standard locations, so you are going to have to be root (aka super-user) to do most of what follows. Note that the `#' character prompt indicates the command is run as root. To get started, install LogValidator like so:

# perl -MCPAN -e 'install W3C::LogValidator'

I found that installing LinkChecker did not work using CPAN, so I used the old-school method of downloading the package (aka "tarball"), unpacking it in some temporary directory, and building it and installing it from there. Again, only the actual installation step requires you to be root, all others you are better off as an ordinary user (as in you). Note the change in shell prompt to the `>' character.

/tmp/CPAN> tar xzvf W3C-LinkChecker-4.3.tar.gz
W3C-LinkChecker-4.3/
W3C-LinkChecker-4.3/...
/tmp/CPAN> cd W3C-LinkChecker-4.3
/tmp/CPAN/W3C-LinkChecker-4.3> perl Makefile.PL
/tmp/CPAN/W3C-LinkChecker-4.3> make
/tmp/CPAN/W3C-LinkChecker-4.3> make test

I'm not showing you the output here because it gets a little messy. Again, consult my session if you are new to this. The final step is to install LinkChecker as root:

/tmp/CPAN/W3C-LinkChecker-4.3> su
Password:
# make install
...
# ^d

Now you should have all the software installed and can proceed to the configuration step. Note that the ^d above means Ctrl+D (EOF), which exits you from the root login (typing "exit" also works).

Configuration

LogValidator comes with a sample configuration file that you need to copy to some directory of your choice. The sample will be in root's hidden .cron directory, but it's readable so exit from your root login and create this directory.

For example:

/tmp/CPAN/W3C-LinkChecker-4.3> cd /var/www/mysite
/var/www/mysite> mkdir config
/var/www/mysite> cd config
/var/www/mysite/config> cp ~root/.cpan/build/W3C-LogValidator-1.3.1/samples/logprocess.conf .

Pay attention to the LogValidator version number because this article can (and will) become obsolete at some point.

Now you're ready to edit the configuration file and run the script. It is very similar to the Apache httpd.conf file (and some of the directives are straight out of it), so if you're familiar with editing that file you're ahead of the game.

There are several options that you will want to alter, and the file is fully documented. Well almost fully documented. I'll get to that in a minute.

##  [apache] ServerAdmin : e-mail address to send the reports

ServerAdmin doug@example.com

##  MailFrom : From: address for e-mail output
##
## Unless the relevant option is specified when running the LogValidator,
## the mail output will use ServerAdmin (see above) as From: and To:
## This option allows you to override the From: parameter
## DEFAULT  = ServerAdmin

MailFrom logvalidator@example.com

## Title : a more useful Subject: for the Mail Output
##           and <title> for HTML Output
##
## Tell the Mail/HTML Output what this config is all about
## and make them use a better subject than the vanilla default
## DEFAULT = Logvalidator results

Title LogValidator - Results for example.com

##  [apache] DocumentRoot : where the files are located
##
## For some log formats, it is necessary to know where the actual files
## reside on the server

DocumentRoot /var/www/mysite/docroot/

##  [apache] ServerName : full address for the web server
##
## should be of the form host.domain
## NOTE: no need to prepend http://

ServerName example.com

##  [apache] CustomLog : log file and format
##
## Add as many entries as you like.
## The Log Validator will process all log files listed below
## formats: see http://httpd.apache.org/docs/mod/mod_log_config.html
#  NOTE: only the following formats are currently supported:
#                    common, combined, w3, full, plain (list of addresses)
# CustomLog /var/log/apache/access.log.1 combined
# CustomLog /home/me/path/to/list plain

CustomLog /var/www/mysite/logs/access_log combined

## [apache] DirectoryIndex : document equivalent to "/"
##
## See http://httpd.apache.org/docs/mod/mod_dir.html#directoryindex
## Used by the validator to compute the "canonical" URLs for Documents
## DEFAULT = index.html index.htm index

DirectoryIndex index.php index.html

There are a large number of other options to play with, but many of the defaults are sufficient until you get everything running and you want to tweak things. The options that don't seem to be documented very well, and are omitted from the sample configuration file, relate to output. So I took the time to do so.

# UseOutputModule : method and location to send output

# --email from your shell, or
# UseOutputModule W3C::LogValidator::Output::Mail
# -s|--sendto <address> from your shell, or
# SendTo doug@example.com

# --HTML from your shell, or

UseOutputModule W3C::LogValidator::Output::HTML

# -o|--output <path> from your shell, or

OutputTo /var/www/mysite/www/admin/logvalidator.html

# output will go to console if not specified

Notice I have both options set, but the email method is commented out (a `#' character precedes comments). You can override either option by using command-line switches as described above. Also note that I am placing the HTML output in an admin directory, which I happen to have password protected because it contains other stuff that I don't want just anyone to have access to.

Once you have everything configured correctly it's time to run the script and view the output. Back at your command prompt, run the script and tell it where to find the configuration file:

~config> logprocess.pl -f logprocess.conf

This should work if you are in the same directory as the configuration file, otherwise you may have to specify the full (or relative) path to its location. Also, the CPAN installation typically puts the Perl script in /usr/local/bin/, so if that directory isn't in your PATH or depending on which shell you're using, it may not find the script at first. Either try using the full path to the script or run the rehash shell built-in command to update its database.

Also, depending on how much traffic you get, the server log file may be quite large and running all four modules on the results will take some time, so be patient. When the script exits and depending on the output option you selected, you will either get an email with the results or you can visit the generated HTML report with your browser.

Reports

To generate the report the script took about 20 minutes running in the background. This was with a full day's worth of requests stored in my Apache access log from yesterday. I quick check with wc -l on the log returned around 50,000 total requests. Keep in mind that many of these are for images, CSS, JavaScript, and other resources—not complete pages. Below is a thumbnail with my results, minus the Basic module, which was configured to list the top 100 most popular pages (the default).

I kept an eye on the process while it ran and by far the longest time was spent during the LinkChecker phase, which is not surprising given all the external HTTP requests necessary to check each link. And I have a lot of outbound links. I ran checklink manually on my home page, and that in itself took a minute and change.

Conclusion

LogValidator is a great tool, although it takes some work to get it running. The main idea here is to target those pages on your Web site that get the most traffic and fix problems there first, before moving on. If you have thousands of pages like I do, it can be a pretty daunting task to find and fix every error on your site.

One final note. I use newsyslog(8) to automatically rotate and gzip compress my Apache access logs. I was hoping that LogValidator would have the ability to deal with gzipped log files, but as far as I could determine it does not. Although it really wouldn't be that difficult to modify the code to do so, I opted for using a simple wrapper shell script that uncompresses yesterday's access log, runs the logprocess script, and then recompresses the log file. Below is some sample code if you want to take this route.

# newsyslog.conf -- rotate apache logs
/var/www/mysite/logs/error_log root:staff 640 6 * @T00 BZN
/var/www/mysite/logs/access_log root:staff 640 6 * @T00 BZ /var/run/httpd.pid 30

#!/bin/sh

# logprocess -- W3C::LogValidator wrapper script

config=/var/www/mysite/config/logprocess.conf
access=/var/www/mysite/logs/access_log.0

gunzip ${access} 2>&1 /dev/null
logprocess.pl --quiet --config ${config} 2>&1 /dev/null
gzip ${access} 2>&1 /dev/null

In case you're wondering, the 2>&1 /dev/null at the end of each line above means "really quiet" as in all output, both stdout (console) and stderror (also console, but another file handle), is sent to the "bit bucket" (/dev/null). And the reason I'm doing this is because I can now execute this script via cron and not have a bunch of useless messages in my inbox.

That all he wrote folks! And you thought I was a softy after my last post.

Authenticating a Googlebot in PHP and Perl

dwclifton@gmail.com (Douglas Clifton) — Sun, 24 Sep 2006 18:08:26 -0400

Following a tip from Russ I was pleased to find an interesting post on the Official Google Webmaster Central Blog titled How to verify Googlebot. In a nutshell, it explains how to use the Unix shell program host to authenticate that an IP address copied from your Web server log file really is a Googlebot and not some email harvester (or whatever).

I decided to take this a step further and demonstrate how you can automate this procedure using a scripting language. For these examples I chose PHP and Perl, although you could certainly use Python or Ruby or whatever your preferred language is, as long as it has an interface to the gethostbyname and gethostbyaddr system calls.

Using these calls under PHP is the simpler of the two approaches, as the interface to these routines are written at a more abstract level than using the Perl Socket module. Below is an example googlebot() function in PHP that returns true if the IP address parameter authenticates, although there is no 100% guarantee of preventing a spoof getting through (but it will catch the vast majority of them). A bit of test code is included.

<?php

function googlebot($ip)  {

    // check to see if this IP really is a Googlebot

    $bot = 'googlebot.com';
    $name = gethostbyaddr($ip);
    if ($name == $ip) return false;

    return (strpos($name, $bot) !== false and gethostbyname($name) == $ip) ? true : false;
}

// test it

$ip = '66.249.66.1';

echo $ip . ' is ';
if (!googlebot($ip)) echo 'not ';
echo 'a Google bot' . "\n";
?>

The Perl version is at a much lower level, very similar to the corresponding C system calls. In fact, the module is derived directly from the sys/sockets.h header file and the functions are just wrappers around these Standard C library calls. See Berkeley Sockets for more information. If you have a copy of Programming Perl, the chapter 16 Interprocess Communications section on socket programming will help, and if you are lucky enough to have a copy of the Perl Cookbook, chapter 18 Internet Services has some great recipes for DNS lookups. For really gory details, refer to chapter 14 DNS: The Domain Name System of TCP/IP Illustrated, Volume I—The Protocols.

#!/usr/bin/perl

use Socket;

sub googlebot($)  {

    # check to see if this IP really is a Googlebot

    my $ip = shift;
    my $bot = 'googlebot\.com';
    my $name = gethostbyaddr(inet_aton($ip), AF_INET) or return 0;
    my @addr = gethostbyname($name);
    my $addr = inet_ntoa($addr[4]);

    return ($name =~ m/$bot/ and $ip eq $addr) ? 1 : 0;
}

# test it

$ip = '66.249.66.1';

print $ip . ' is ';
unless (googlebot($ip)) { print 'not '; }
print 'a Google bot' . "\n";

Finally, in case anyone is interested why it's been so long since I posted anything, much of the summer I was sick as a dog and since recovering, busy as a bee. It's nice to be feeling better and back to work!

Sendmail in Perl

dwclifton@gmail.com (Douglas Clifton) — Wed, 03 May 2006 14:44:21 -0400

People love to argue. A recent thread on the DC Perl Mongers mailing list opened yet another discussion of which CPAN module is the best for sending email.

Most Unix systems, at least the open-source flavors I use most frequently (FreeBSD and Linux), have sendmail installed. So when I need to send an email from a Perl script I just roll my own code to do it. You could argue that it's too simplistic because it's procedural or it isn't portable, but it works, it's fast and I have control over the code.

The problem with using someone else's module is you're tied into the way they think, and it can be difficult or time consuming to modify the code to add a feature or fix a bug.

Who made this rule that you have to use a module for everything or that object-oriented methods are somehow intrinsically superior to procedural programming? Don't get the wrong idea, I have nothing against modules or OOP. It would have taken me a hell of a lot longer to write many of the Perl scripts I've written without wonderful tools like CGI.pm and DBI. Not to mention about a 100 others.

#!/usr/bin/perl

$to = 'you@somehost.com';
$from = 'me@somehost.com';
$subject = 'test sendmail';
$body = <<EOB;
Hello from Perl/sendmail
EOB

if ($error = sendmail($to, $from, $subject, $body))  {
    die "Can't sendmail: $error\n";
}
print 'mail sent';

sub sendmail($$$$)  {

   my @args = (
       'to',
       'from',
       'subject',
       'body'
   );

   my $arg;

   foreach $arg (@args)  {
       unless ($$arg = shift)  {
           return (caller(0))[3] . ': missing $' . $arg . ' parameter';
       }
   }

   my $sendmail = '/usr/sbin/sendmail';
   my $switch = '-t';

   open MAIL, "|$sendmail $switch" or return $!;

   print MAIL <<EOM;
To: $to
From: $from
Subject: $subject

$body
EOM
   close MAIL;
   return;

} # sendmail()

Love Perl

dwclifton@gmail.com (Douglas Clifton) — Thu, 09 Feb 2006 20:46:42 -0500

My first true love as a language, and although I don't use it much for Web stuff, I do still use it quite a bit. In fact I have a whole bin/cron directory full of Perl scripts. I love Perl!

From the creator of the Perl Advent Calendar, launching Valentine's Day, comes LovePerl.

Via O'Reilly Radar.

Bricolage Adds PHP Support

dwclifton@gmail.com (Douglas Clifton) — Thu, 25 Aug 2005 10:24:00 -0400

Version 1.9.0 of David Wheeler's excellent Bricolage Perl CMS adds the ability to call PHP from inside the system. It also sports a new UI, built with standards compliant XHMTL 1.0 Strict and CSS.

But the really exciting news is the new PHP 5 Templating Architecture, on top of the existing Mason, Template Toolkit, and HTML::Template systems. To pull it off, the Bricolage team brought in George Schlossnagle of Omni TI to write a full-blown PHP::Interpreter. The new Perl module is not restricted to deployment inside of Bricolage, you can install and use it from any Perl application. Not only does it allow you to call PHP from Perl, with you can reach back and access Perl objects and methods from inside PHP.

You heard me, you can access any CPAN module from inside PHP. Wow. I have goosebumps on my arms.

With the addition of the leaner UI and new support for mod_gzip, the Bricolage system gets a nice performance boost as well.

Via O'Reilly Radar.

Blue Moon

dwclifton@gmail.com (Douglas Clifton) — Sat, 20 Aug 2005 02:08:26 -0400

Dammit! I complete forgot tonight is the Blue Moon.

Check out: dcal: Calendar for August, 2005.

And some details on how it is implemented: dbrowse: Moon Phase Data and the Blue Moon.