W3Perl 3.09

Copyright W3Perl is a logfile analyser based on logfiles parsing and distributed under the GPL licence. It require a configuration file which can be build from a web interface. A complete list of features is available in this section, with a short description of all scripts. The glossary is here to help you understanding your report. W3Perl is a free software distributed under the GNU General Public License. It means you are free to use the package as long as you want, you can modify the code and distribute the new code to others. This section describes the W3Perl package. Each script compute a specific stats and some are optional. The job of the master script is to run all stats in the right order. Included in the package are some useful tools and the list of resources used by the package. Selecting a script will display its command lines parameters. This glossary will help you to understand W3Perl output. The main features of W3Perl is shown, with the kind of logfile name the package can read and the W3Perl output structure. This is the place to download the package. Make sure you have the latest version and read about the last news. Scripts contents Glossary Main features Downloading

What you need This section will teach you how to use W3Perl. First, check that your system has everything to run this package. An installation manual is available for each platform. You can install the package without telnet access (FTP) nor CGI available. A Web adminstration interface allows you to manage your stats but all scripts can be run using command lines. For easy installation, RPM is available for Linux and there is a .exe file for Windows. Informations are available on the configuration files to customize your output, choose the right logfile format. You can run the package from the web administration interface or from a crontab. Don't mix both as web user has very limited privileges and won't be able to read/write file owned by a user. W3Perl provides the usual stats as Pages, Hosts, Countries, Directories, Download, Hourly, Daily, Weekly, Monthly, Referer, Browsers, Traffic, Search engine, Error, Scripts ... but many more as Real-time, Session, Virus, URL mapping ... If you want to upgrade, see first the latest news about the package and read how to make the upgrade. At last, some informations about the next release and the package limitations. To use W3Perl , you don't need to install a lots of package or external Perl modules. It can be run as a standalone package. As W3Perl is written in Perl, you should have a copy of Perl on your system. This is not a problem for unix user as Perl is always installed. For windows user, you'll have to install ActivePerl, the Windows port of Perl. W3Perl use the fly package to build graphics. This is a small tool which allow the use of the gd library in Perl. W3Perl is based on the 1.6.5 Fly version as Gif support have been removed from the latest version. You can install optional Perl modules which allow you to connect to a database (DBI) if you're using a CMS as Spip or a module which allow you to locate IP when the reverse DNS fail. If your logfiles are compressed, W3Perl need to uncompress them on the fly so you need at least to have the tool to do the job. Installing W3Perl is quite easy. You can choose between the RPM package or the tarball. RPM is available for Mandrake users. If you want to build your own RPM, the SRPM is there. If you want to share, send me your RPM so it will be available to others. To install W3Perl from the tarball, extract the package inside your server tree (could be a user alias). Edit the install.pl script to alter the perl path, your cgi path (if available) and the w3perl path. Run this script. You can then use the web interface to manage your stats output or build your configuration files. Some predefined configuration are available, you can use them as a template. Once everything is fine, you could either run the stats from the web interface or from the command lines with cron-w3perl.pl script. To automatically update your stats, use a crontab to launch this master script with the incremental mode. More details are provided in the documentation. The windows installer have been greatly improved recently. Thanks to the NSIS tool, installing W3Perl is just a matter of clicking on the .exe file. First, the installer will check Perl is available and then will look for the IIS server. If found, W3Perl install its scripts, create a virtual directory inside your IIS server and allow this directory to have executable scripts and finally add the perl mapping extension. Then the install script is running. When everything is fine, your favourite browser is call with the W3Perl web administration page. A default configuration file is provided for IIS, so you can straight away to launch the stats. Many things need to be improved as the ability to detect several IIS Webserver or the Apache server, and then the ability to select on which servers you want to install W3Perl. Also building configuration files according to your server by reading the IIS Metadata or Apache configuration files. I haven't install W3Perl on a Mac for a while. Some people have done the job so a little help is available. If you want to add some comments or update this documentation, you are welcome. W3Perl can use different plug-in to build extra stats or add new features. The first one is the Geo::IP perl module which allow to map IP to country code, allowing to get country stats with IP logfiles. It can be used as an alternative to reverse dns as the perl module use a local database to map IP to country code. (update are freely available on the maxmind website). Making DNS queries can be very long, but this is the only way to resolve IP to hostname, GeoIP doesn't do the same job, IP are not translated to hostname. To get cities stats, you can use the GeoLiteCity module from maxmind which require the GeoIP library. Stats reports can be exported in PDF thanks to the htmldoc program. In order to send stats report via email, you'll need to install the perl MIME::Lite module. At last, if you want W3Perl to extract data from your SPIP database, the perl DBI module should be installed. If you are running SELinux (Security-Enhanced Linux), additionnal customisations are required in order to run W3Perl. Basically, few chcon commands and some others security changes. Configuration files are the way to customize your stats. It includes how the stats will be build, which stats you want to produce, the display and many filtering options. A Web Administration interface is available to help you managing those files. You can create one from scratch, from a previous template or modify an existing one. Sadly, people without cgi access won't be able to get this facility, then use your favourite editor and select an existing one to build your own. Some predefined configuration are availble for newbies. There are ready to use with default values. Feel free to change them with the administration interface. To create a configuration file, you should first define some basic setup in order to W3Perl to run : where your logfiles are, which kind of logfile you are using, if your logfiles are split/compressed, your logfile filename ... Then the filtering option to select on which directories you want to get stats, which hosts you want to exclude, the threshold you want on display ... And finally, how the stats will be displayed using a background, in which language ... You can select how much stats will be output. If you just need to have a glimpse, use the lower level and only one page figuring the main basic stats will be produced. This is the quick way to know the number of access, hits, hosts... If you want to see stats versus time, level 2 is the minimum. More informations will be available but of course it will take longer to process. Default configuration is level 3 which give a very good amount of stats. At last, if you are not in a hurry and want to have the maximum stats, select level 4 but you'll need some hard disk space left as the stats become very precise. As the logfiles are the input files for W3Perl, you should carefully select the ones you are using or the scripts will fail. Sadly, there are a lot of logfile format to choose. Configuration files provided with the package use the default predefined. With IIS, they use the W3C format (which is really a minimal format), with Apache, they use the CLF (Common Log File) which have more informations but still lack the referer, browser and OS informations. My recommandation would be to change your logfile format to a more precise one as the ECLF (Extended Common Log File). If you use your own non-standard logfile format, W3Perl is able to read it because you can build your own string format using a list of keywords so there is no limit on logfile format the scripts can read. Ftp and Squid logfile format have been added recently. You can have only one logfile but most servers are splitting the logfiles to avoid the file becoming too large. W3Perl is able to cope with daily and monthly split logfiles (they can be compressed also to save disk space). Log filename should follow some basic rules. They should have a core filename (i.e access,ex,in ...) and informations about the date. If you choose to have daily logfiles, the daily date should be in the log filename, but it can be anything from number to string date. For the W3C format, it is really necessary as the date is not store in the log. News about updated package is available here. If you want to be informed about the latest release by email, subscribe to the list in the w3perl homepage. Want to upgrade to the latest W3Perl version ? Please read carefully those instructions. There are three steps in upgrading. First is to backup your configuration files just in case ... and any files you have make some modifications by yourself. Then run the upgrade script either from command line or from the web interface. It will update your configuration files if needed. At last, run the fixperpath script to change the perl path in the new installed scripts. Any problem using W3Perl ? Please read first the FAQ. Many questions have been solved here, including customisation, installation, logfile, setup and windows issues. If you don't find your answer there, feel free to ask in the forum, maybe someone have the answers. You can also send me an email. Various hints are available here. If you want to add yours, tell me. Best advice is to test your configuration on a small scale (with short logfile or using a starting/ending date for the scripts or with the lowest precision level). Once everything is fine, go for the complete run. The major criticism about W3Perl is the speed. The package is written in Perl so it can't be as fast as C package of course. You could disable some options to get faster (reverse dns is very slow as it queries DNS server for each new IP found). Some tests have been done with a 'slow' computer (right now in 2006) and it show that W3Perl can process some quite large logfile in a few hours. This is the place to ask questions, to submit features and requests help. If you want new forum to be added, send me an email. I'll try to answer everyone as fast as possible. If you want to be keep inform about the latest release, subscribe to this notification mailing list. Don't be afraid, you will receive very few emails a year. This is a easy way for me to announce a new version available to all W3Perl users. A form have been installed to keep track of bugs. Please use it to send problems you have found. Try to be as precise as possible as it will help me a lot to reproduce the problem. The list of information I would like to receive is the following : W3Perl version Name of the script which fail How do you install W3Perl (FTP or telnet) Your OS The step where the problem occur (installation, configuration, running, web interface...) Which kind of logfile are you using (split, compressed, filename) The logfile format you are using (CLF,IIS-W3C,IIS-Microsoft...) If you have used the web interface to build the configuration file The URL where I can view the problem (if available). Your email of course (or I won't be able to contact you back) Any comments which could be useful to debug Your configuration file and scripts output. Watch the kind of features I would like to add in the next release. Of course, this is a wish list. If you have others idea, let me know, I'm always open to suggestions. About the display, I would like to add some CSV, PDF or XML output. About installation, providing more predefined configuration files would be helpful. Improving the windows installer with the ability to choose on which server you want to install the package or to disable the webserver detection to allow installation without IIS or Apache. About features, I've planned to add more support for FTP logfiles, to improve speed and memory usage and to increase the whole package stability. W3Perl have some limitations as a minimum of one request by week for the logfile activities or the maximum number of logfiles to scan. You'll find the last bug reports and how they have been fixed. Installation - Unix - Windows - Macintosh - Plug-in - SELinux Configuration - Configuration file - Level of accuracy - Log format - Log filename What's new Upgrading Help - FAQ - Hints - Speed - Forum - Mailing list - Bugs report Future developements Serious bugs

Demo If you want to check other web stats tools, I've listed some of the best available. They have all their pro/cons, it's up to you to make the final choice. Included in this section is a list of useful tools, especially to convert log files. Finally, I've included this program history to see what's have been done during the last 10 years ... and the greetings to thanks many people who have greatly help. If you want to look at the W3Perl output, you could navigate through my website stats. The stats are updated once a day via a crontab. You can see which parts of my website attract people and how many people are coming each day. There are many stats package based on the logfile scanning. Here is a list of my favourite. Each have its pro and cons. Feel free to look at them and choose the one which fit best your need. Many tools exists to make life easier especially with log files. Some can converts from one format to another, others can resolve IP to hostname allowing then W3Perl to compute faster. If you know others tools which can be added in the list, let me know. Two small reviews about W3Perl in french. If you aware of others reviews, let me know. Just for fun, some stats about the growth of W3Perl code. Not really uptodate. Need some kind of cleaning ! W3Perl have begun ten years ago in 1995 with my first job as a webmaster. I was looking for a stats analyser to know how many people were visiting the website. Started as a shell script, I move quickly the code to Perl. Adding more features with time, W3Perl grow quite fast in the first years. Now the package is working fine but there are always some new idea to implemente. W3Perl exist because of many people around the world who help me to develop new features, test the package and bring new idea. I would like to thanks everyone who help here, if your name is not in the list, I apologie because I can't remenber everyone. Others stats packages Logfile tools Reviews about W3Perl W3Perl own stats Program history Greetings