securityonion-setup - 20120912-0ubuntu0securityonion296 now available for Security Onion!

securityonion-setup - 20120912-0ubuntu0securityonion296 is now available and should resolve the following issues:

so-allow: add OSSEC/Wazuh registration service option #1506
https://github.com/Security-Onion-Solutions/security-onion/issues/1506

Setup: /etc/network/interfaces ethtool rx setting should be commented out by default #1508
https://github.com/Security-Onion-Solutions/security-onion/issues/1508

Discussion
Richard Bejtlich recently blogged about an issue with Virtualbox and /etc/network/interfaces:
https://taosecurity.blogspot.com/2019/04/troubleshooting-nsm-virtualization.html

We were able to duplicate the issue and determine that it had to do with the ethtool -G rx setting.  Traditionally, our Setup script has used ethtool -g to determine the maximum rx setting and then ethtool -G to enforce that maximum rx setting.  It seems as if VirtualBox 6.0.4 may have an issue whereby its virtual network interfaces report a maximum rx setting of 4096 but are unable to reliably be set to that value.  Therefore, the safest option for widest compatibility is to keep the rx setting at its default value.  Additionally, some folks are recommending lower rx values for better performance:
https://github.com/pevma/SEPTun/blob/master/SEPTun.rst

Our new Setup script continues to write the ethtool -G rx setting into /etc/network/interfaces but it is now commented out by default.  If you need to modify this, you can certainly do so.

For more information, please see the Network Configuration page on our Documentation site:
https://securityonion.readthedocs.io/en/latest/network-configuration.html

Thanks
Thanks to Richard Bejtlich for reporting the /etc/network/interfaces issue!
Thanks to Dustin Lee for duplicating the /etc/network/interfaces issue!
Thanks to Wes Lambert for testing!

Updating
Please see the following page for full update instructions:
https://securityonion.net/docs/Upgrade

Conference
Please mark your calendar! Security Onion Conference 2019 will be on Friday, October 4, 2019 and registration will open July 18! CFP is open now and we want to hear from you!
https://blog.securityonion.net/2019/04/security-onion-conference-2019-cfp.html

Training
We have a 4-day Security Onion Basic Training class coming up in Costa Mesa CA!  If you can't make it to an onsite class, we have a new online training platform.  For more information and other training options, please see:
https://securityonionsolutions.com

Appliances
We now offer hardware appliances!  For more information, please see:
https://blog.securityonion.net/2018/10/introducing-security-onion-solutions.html

Documentation
We've got a brand new documentation site!  Please let us know if anything needs to be updated:
https://securityonion.net/docs

Support
Need support?  Please see:
https://securityonion.net/docs/Support

Thanks!

When is full packet capture NOT full packet capture?

I was looking at some packets recently and noticed the Wireshark message "Packet size limited during capture".  This was strange since the packets came from a Sguil sensor performing full packet capture using Snort's default snaplen on a standard Ethernet connection (no Jumbo frames and no VLAN tags).  Drilling down into the packet capture, some of the packets were 2900 bytes and Snort was only capturing the first 1500 bytes.  The full packet capture was not "full" packet capture after all.

So where did the 2900-byte packets come from?

The OS had enabled by default the following NIC offload features:
tcp-segmentation-offload (tso)
generic-segmentation-offload (gso)
generic-receive-offload (gro)

For more information about these features and their side effects, please see:
http://www.unleashnetworks.com/blog/?p=307
http://wiki.wireshark.org/CaptureSetup/Offloading
http://manual.snort.org/node7.html
http://www.inliniac.net/blog/2007/04/20/snort_inline-and-tcp-segmentation-offloading.html

I won't repeat all the information in those links, but I'll summarize by saying that the NIC was reassembling packets before being passed up the stack to Snort.  I disabled the offload features and then verified that this resulted in no more packets larger than 1500 bytes.  The packet capture truly was "full" packet capture.

[ UPDATE: A reader asked why we couldn't simply change Snort's default snaplen to a larger value to capture the 2900-byte packets.  While it's true that would solve the "full" packet capture problem, another problem would remain.  Since the packets are being reassembled on the NIC, Snort is not able to properly perform target-based reassembly (see the Snort manual link above).  This opens the door for potential IDS evasion/insertion attacks.  NIC offload settings need to be disabled so that Snort sees the same packets the destination host does. ]

Some NIC/driver combinations may disable these offload settings by default, while others enable it by default.  You should check your sensors now before you get into a situation where you really need that full packet capture and find out that you don't actually have it.  To check, run ethtool with the "-k" (lower-case k) option on the interface you'd like to check.  For example, to check eth0:

ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off

You should repeat this for every interface in your system, as you may have NICs from different manufacturers with different defaults.

You can set these options using the "-K" (upper-case K) option to ethtool and specify which option you'd like to set.  For example, to disable tcp-segmentation-offload for eth0:

ethtool -K eth0 tso off

You can set multiple options in one "ethtool" command, but this can be problematic if your card doesn't support all of the settings.  To avoid this, you could invoke ethtool for each option like this:

ethtool -K eth0 rx off
ethtool -K eth0 tx off
ethtool -K eth0 sg off
ethtool -K eth0 tso off
ethtool -K eth0 ufo off
ethtool -K eth0 gso off
ethtool -K eth0 gro off
ethtool -K eth0 lro off 

Or we could simply wrap the ethtool command in a for-loop like this:

for i in rx tx sg tso ufo gso gro lro; do ethtool -K eth0 $i off; done

These settings will remain in effect only while the OS is booted, so this needs to be applied at every boot.  This can be done by adding the above for-loop as a "post-up" command for each of the interfaces in /etc/network/interfaces.  If you're still using the graphical Network Manager to configure your interfaces, I've put together some documentation on disabling it and configuring interfaces and their offloading features via /etc/network/interfaces:

https://securityonion.net/docs/network-configuration.html

I'd really like some feedback on this:

• What were your default settings? (ethtool -k eth0)
• Did you have any problems disabling the offload features?
• Did you notice any difference in performance after disabling the offload features?
• Is there a better way of disabling offload features globally?  I tried putting the commands in /etc/rc.local and /etc/init/securityonion.conf, but the only way I could get it to work consistently was via /etc/network/interfaces as documented above.
• I'm considering disabling offload features by default in the Security Onion Setup script.  Can anyone think of any reason why this might be a bad idea?