-----Original Message----- From: kclug-bounces@kclug.org [mailto:kclug-bounces@kclug.org] On Behalf Of Dave Hull Sent: Thursday, October 21, 2004 11:21 AM To: kclug@kclug.org Subject: RE: It was bound to happen - suspected hack
[snip]
What kind of storage space is required for this? How long to do you keep all that traffic? That sounds impressive.
Well, it depends on a number of factors - primarily the speed of the link being monitored, and the utilization of the link as well. Monitoring 10/100 is pretty well sustainable without packet loss. Moving into gigabit is another world entirely.
I have been known to use the client components from the SHADOW IDS project to log traffic into binary tcpdump format, on one hour boundaries. Thus, I wind up with files like:
-rw-r--r-- 1 ids ids 3043 Oct 21 02:00 tcp.2004102101.gz -rw-r--r-- 1 ids ids 7076 Oct 21 03:00 tcp.2004102102.gz -rw-r--r-- 1 ids ids 1456 Oct 21 04:00 tcp.2004102103.gz -rw-r--r-- 1 ids ids 160916 Oct 21 05:00 tcp.2004102104.gz -rw-r--r-- 1 ids ids 9574 Oct 21 06:00 tcp.2004102105.gz -rw-r--r-- 1 ids ids 194180 Oct 21 07:00 tcp.2004102106.gz -rw-r--r-- 1 ids ids 161428 Oct 21 08:00 tcp.2004102107.gz -rw-r--r-- 1 ids ids 5623292 Oct 21 09:00 tcp.2004102108.gz -rw-r--r-- 1 ids ids 638976 Oct 21 09:36 tcp.2004102109.gz
I've a handful of methods to pull the files from remote sensors which I monitor, etc. and clean the sensor hard drive to ensure I don't run out of disk space. I have as of late been moving toward the use of IDABench, which is based on the SHADOW IDS, which functions nearly identical but is a bit more current vis-à-vis features.
You can see the file sizes on the snippet of directory listing above. The particular sensor this is from is on a DSL link which can sustain 5.99mpbs inbound, and 384kbps outbound, so storage isn't a huge issue. What's important is to pull the files from the sensor frequently enough to ensure your logging partition isn't getting full. (<= 24 hours for me, depends on client. I have one for which I'm running an hourly pass.)
I can generally store about a month of compressed files on a CD, which is a good immutable storage mechanism if I need to present them in the future.
Folks who are really interested in going down this road will find Snort 2.1 Intrusion Detection 2nd Edition as useful as the Stevens TCP/IP bible. It's from Syngress press, and a host of contributing authors. 'Tis around $42.05 @ MicroCenter.
There are several vendors out there that like to throw around the buzz-phrase "real time"... don't believe the hype. In my experience, anything less than $50K will only get you to within the past half an hour at best... and you still have false positives (or worse, negatives) to deal with.
What did Bruce Schneier say? Oh yeah, "Security is a _process_, not a _product_." And much like Neil Stephenson pointed out on /., the hardware is there, but software is still pretty much shit.
Dustin