Follow-up on OSSEC alerts for packet loss

This is a follow-up to my recent post "How do I receive an email when my sensor stops receiving traffic?".  That post explains the core idea which I have since refined.


Refinement #1: Tell me which interface stopped receiving traffic
The first area of refinement is making the output a little more verbose so that, if we have multiple interfaces, we know exactly which interface stopped receiving traffic.  We do that by modifying the "bandwidth" command in /var/ossec/etc/ossec.conf as follows:

  <localfile>
    <log_format>command</log_format>
    <command>grep -v "^#" /etc/nsm/sensortab |awk '{print $1}' |while read SENSOR; do INTERFACE=`echo $SENSOR|cut -d\- -f3`; echo -n "$INTERFACE: "; tail -1 /nsm/sensor_data/$SENSOR/snort.st
ats |cut -d\, -f3; done</command>
    <alias>bandwidth</alias>
  </localfile>

Refinement #2: Give me more flexibility in the OSSEC rule structure
The second area of refinement is implementing a tiered OSSEC rule structure.  This gives us more flexibility and troubleshooting capability.  We do this by editing /var/ossec/rules/local_rules.xml and replacing our previous single rule with these two rules:

 <rule id="100001" level="1">
    <if_sid>530</if_sid>
    <match>ossec: output: 'bandwidth':</match>
    <description>Bandwidth statistics from snort.stats</description>
  </rule>

  <rule id="100002" level="7">
    <if_sid>100001</if_sid>
    <regex>0.000</regex>
    <description>Bandwidth down to 0.000.  Please check interface, cabling, and tap/span!</description>
  </rule>

The first rule just identifies "bandwidth" output and only logs it to disk (level 1 alerts do not generate email by default).  The second rule is a child rule of the first and alerts/emails (level 7) when bandwidth is down to 0.000. 

Since we're now logging all "bandwidth" output, we can search for it in the OSSEC logs:

grep "bandwidth" /var/ossec/logs/alerts/alerts.log
2011 Nov 17 14:28:50 so->bandwidth
ossec: output: 'bandwidth': eth4: 8.940
2011 Nov 17 14:28:50 so->bandwidth
ossec: output: 'bandwidth': eth5: 7.189
2011 Nov 17 14:38:54 so->bandwidth
ossec: output: 'bandwidth': eth4: 8.920
2011 Nov 17 14:38:54 so->bandwidth
ossec: output: 'bandwidth': eth5: 7.223

Refinement #3: Use Linux kernel's built-in packet counters instead of relying on snort.stats
The third area of refinement is not relying on snort.stats but instead using the Linux kernel's built-in packet counters.  (I hinted at this in the previous post.)  This could be used to replace the entire "bandwidth" configuration above, or to complement it for a belt-and-suspenders approach.  We start off by adding the following to /var/ossec/etc/ossec.conf:

  <localfile>
    <log_format>command</log_format>
    <command>grep -v "^#" /etc/nsm/sensortab |awk '{print $4}' |while read SENSOR; do echo -n "$SENSOR: "; RX1=`ifconfig $SENSOR |awk '/RX packets/ {print $2}' |cut -d\: -f2`; sleep 300; RX2
=`ifconfig $SENSOR |awk '/RX packets/ {print $2}' |cut -d\: -f2`; expr $RX2 - $RX1; done</command>
    <alias>packets_received</alias>
  </localfile>

This follows the same format as the "bandwidth" command, but pulls the count of received packets from ifconfig, waits 5 minutes, pulls the RX count from ifconfig a second time, and subtracts the first from the second to get the total number of packets received in the 5-minute interval.

Next, we add these two rules to /var/ossec/rules/local_rules.xml:

  <rule id="100003" level="1">
    <if_sid>530</if_sid>
    <match>ossec: output: 'packets_received':</match>
    <description>Number of packets received in 5-minute interval</description>
  </rule>

<rule id="100004" level="7">
    <if_sid>100003</if_sid>
    <regex> 0</regex>
    <description>Received 0 packets in a 5-minute interval.  Please check interface, cabling, and tap/span!</description>
  </rule>

Since we're now logging all "packets_received" output, we can search for it in the OSSEC logs:

grep "packets_received" /var/ossec/logs/alerts/alerts.log

2011 Nov 17 14:33:50 so->packets_received
ossec: output: 'packets_received': eth4: 70969
2011 Nov 17 14:38:50 so->packets_received
ossec: output: 'packets_received': eth5: 63059
2011 Nov 17 14:43:54 so->packets_received
ossec: output: 'packets_received': eth4: 71030
2011 Nov 17 14:48:54 so->packets_received
ossec: output: 'packets_received': eth5: 67475

When the number of received packets drops to 0, rule 100004 triggers a level 7 alert, generating an email if configured to do so.