high softirq

[brad at bradandkim.net]

I have 6 identical IBM blades and all are running RHEL 3, kernel
2.4.21-40.ELsmp.  They are dual 2.8 Xeons with 4 GB RAM.  Most of the
blades that are not in production have a resting load average of 0.00 to
0.08.  One particular blade has a resting load average of 0.2 to 1.3. 
Even when I shut down just about every service it hovers around 0.3.  The
load looks to be coming from softirq.  How can I track it further?

irq looks fine, as does iowait and context switching, at least as far as I
can tell.

Here is a snip from top:

 16:30:49  up 1 day, 9 min,  2 users,  load average: 0.51, 0.65, 0.83
46 processes: 45 sleeping, 1 running, 0 zombie, 0 stopped
CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
           total    0.0%    0.0%    0.0%   0.0%     3.2%    0.3%   96.3%
           cpu00    0.2%    0.0%    0.0%   0.0%     0.0%    0.0%   99.7%
           cpu01    0.0%    0.0%    0.0%   0.0%     0.0%    0.0%  100.0%
           cpu02    0.0%    0.0%    0.0%   0.0%     0.0%    0.6%   99.4%
           cpu03    0.0%    0.0%    0.0%   0.0%    13.0%    0.8%   86.2%
Mem:  4093432k av,  763352k used, 3330080k free,       0k shrd,  213936k buff
       278660k active,             220840k inactive
Swap: 2048276k av,       0k used, 2048276k free                  347588k
cached



Thanks,

Brad Crotchett
brad at bradandkim.net