Thursday, November 17, 2011

Follow-up on OSSEC alerts for packet loss


This is a follow-up to my recent post "How do I receive an email when my sensor stops receiving traffic?".  That post explains the core idea which I have since refined.


Refinement #1: Tell me which interface stopped receiving traffic
The first area of refinement is making the output a little more verbose so that, if we have multiple interfaces, we know exactly which interface stopped receiving traffic.  We do that by modifying the "bandwidth" command in /var/ossec/etc/ossec.conf as follows:
  <localfile>
    <log_format>command</log_format>
    <command>grep -v "^#" /etc/nsm/sensortab |awk '{print $1}' |while read SENSOR; do INTERFACE=`echo $SENSOR|cut -d\- -f3`; echo -n "$INTERFACE: "; tail -1 /nsm/sensor_data/$SENSOR/snort.st
ats |cut -d\, -f3; done</command>
    <alias>bandwidth</alias>
  </localfile>
Refinement #2: Give me more flexibility in the OSSEC rule structure
The second area of refinement is implementing a tiered OSSEC rule structure.  This gives us more flexibility and troubleshooting capability.  We do this by editing /var/ossec/rules/local_rules.xml and replacing our previous single rule with these two rules:

 <rule id="100001" level="1">
    <if_sid>530</if_sid>
    <match>ossec: output: 'bandwidth':</match>
    <description>Bandwidth statistics from snort.stats</description>
  </rule>
  <rule id="100002" level="7">
    <if_sid>100001</if_sid>
    <regex>0.000</regex>
    <description>Bandwidth down to 0.000.  Please check interface, cabling, and tap/span!</description>
  </rule>
The first rule just identifies "bandwidth" output and only logs it to disk (level 1 alerts do not generate email by default).  The second rule is a child rule of the first and alerts/emails (level 7) when bandwidth is down to 0.000. 

Since we're now logging all "bandwidth" output, we can search for it in the OSSEC logs:
grep "bandwidth" /var/ossec/logs/alerts/alerts.log
2011 Nov 17 14:28:50 so->bandwidth
ossec: output: 'bandwidth': eth4: 8.940
2011 Nov 17 14:28:50 so->bandwidth
ossec: output: 'bandwidth': eth5: 7.189
2011 Nov 17 14:38:54 so->bandwidth
ossec: output: 'bandwidth': eth4: 8.920
2011 Nov 17 14:38:54 so->bandwidth
ossec: output: 'bandwidth': eth5: 7.223
Refinement #3: Use Linux kernel's built-in packet counters instead of relying on snort.stats
The third area of refinement is not relying on snort.stats but instead using the Linux kernel's built-in packet counters.  (I hinted at this in the previous post.)  This could be used to replace the entire "bandwidth" configuration above, or to complement it for a belt-and-suspenders approach.  We start off by adding the following to /var/ossec/etc/ossec.conf:

  <localfile>
    <log_format>command</log_format>
    <command>grep -v "^#" /etc/nsm/sensortab |awk '{print $4}' |while read SENSOR; do echo -n "$SENSOR: "; RX1=`ifconfig $SENSOR |awk '/RX packets/ {print $2}' |cut -d\: -f2`; sleep 300; RX2
=`ifconfig $SENSOR |awk '/RX packets/ {print $2}' |cut -d\: -f2`; expr $RX2 - $RX1; done</command>
    <alias>packets_received</alias>
  </localfile>
This follows the same format as the "bandwidth" command, but pulls the count of received packets from ifconfig, waits 5 minutes, pulls the RX count from ifconfig a second time, and subtracts the first from the second to get the total number of packets received in the 5-minute interval.

Next, we add these two rules to /var/ossec/rules/local_rules.xml:
  <rule id="100003" level="1">
    <if_sid>530</if_sid>
    <match>ossec: output: 'packets_received':</match>
    <description>Number of packets received in 5-minute interval</description>
  </rule>
<rule id="100004" level="7">
    <if_sid>100003</if_sid>
    <regex> 0</regex>
    <description>Received 0 packets in a 5-minute interval.  Please check interface, cabling, and tap/span!</description>
  </rule>
Since we're now logging all "packets_received" output, we can search for it in the OSSEC logs:
grep "packets_received" /var/ossec/logs/alerts/alerts.log
2011 Nov 17 14:33:50 so->packets_received
ossec: output: 'packets_received': eth4: 70969
2011 Nov 17 14:38:50 so->packets_received
ossec: output: 'packets_received': eth5: 63059
2011 Nov 17 14:43:54 so->packets_received
ossec: output: 'packets_received': eth4: 71030
2011 Nov 17 14:48:54 so->packets_received
ossec: output: 'packets_received': eth5: 67475
When the number of received packets drops to 0, rule 100004 triggers a level 7 alert, generating an email if configured to do so.

No comments:

Search This Blog

Featured Post

State of the Onion 2024

We usually have our State of the Onion at the annual Security Onion Conference, but we had to cancel the conference due to Hurricane Helene ...

Popular Posts

Blog Archive