Splitting Apache log files

[Gerald Combs]

On Wed, 24 Sep 2003, Bradley Miller wrote:

> I've run into a situation, I've got all my log files in one "BIG" 
> logfile.  I want to run Webalizer and get stats info for clients, I managed 
> to get stats to run for one client today (yea), but they had a separate log 
> file.  I know the Apache Split-Logfile script is there, but how does one go 
> about using it?   It theoretically should make all my little log files by 
> domain so I can give my hosting customers stats . . .

The utility that comes with Apache is "rotatelogs"; you use it in a
pipeline like so:

  CustomLog "|/path/to/rotatelogs /var/log/access_log 86400" common

Under Apache 1.3, this will create /var/log/access_log.nnnn, where "nnnn"  
is a series of timestamps one day apart.  It's a bit ham-fisted in that
the timestamp starts whenever you happen to fire up Apache.  The logrotate
that comes with 2.0 has a few other features.

I personally use cronolog (http://www.cronolog.org/).  Using

  CustomLog "| /path/to/cronolog /web/site/path/logs/%Y/%m/access.%Y%m%d" combined

Gives me a directory tree with logs arranged by year, month and date.
E.g., yesterday's logs are in /web/site/path/logs/2003/09/access.20030924.
This really appeals to my sense of aesthetics.  I need to get out more.

>From there it's a simple matter to feed all of the past year's logs (or
a particular month's logs, or any combination thereof) to Webalizer.
It also makes it easy to feed the day's logs to adnslogres (which is
another utility you should check out).

I've been using this scheme since the 2.2 days and filehandles have
been a non-issue, even when the server in question has more than 500
concurrent httpd processes.

> Also, I just want to note that I kept everything in one big log file 
> because of conflicting information here and there.  Some say you'll run out 
> of file handles (with 2.2 kernel ? ? ) and some say you are ok with newer 
> kernels.  Nobody ever says "here's how you do it and why" . . . it's all 
> sorta wishy washy and never a definitive source of info.   That's one minor 
> frustration with Linux in general.   At least Microsoft has one central 
> "this is why our software is crap" page that is the gospel.
> 
> 
> 
> 
>