When is full packet capture NOT full packet capture?

I was looking at some packets recently and noticed the Wireshark message "Packet size limited during capture".  This was strange since the packets came from a Sguil sensor performing full packet capture using Snort's default snaplen on a standard Ethernet connection (no Jumbo frames and no VLAN tags).  Drilling down into the packet capture, some of the packets were 2900 bytes and Snort was only capturing the first 1500 bytes.  The full packet capture was not "full" packet capture after all.

So where did the 2900-byte packets come from?

The OS had enabled by default the following NIC offload features:
tcp-segmentation-offload (tso)
generic-segmentation-offload (gso)
generic-receive-offload (gro)

For more information about these features and their side effects, please see:
http://www.unleashnetworks.com/blog/?p=307
http://wiki.wireshark.org/CaptureSetup/Offloading
http://manual.snort.org/node7.html
http://www.inliniac.net/blog/2007/04/20/snort_inline-and-tcp-segmentation-offloading.html

I won't repeat all the information in those links, but I'll summarize by saying that the NIC was reassembling packets before being passed up the stack to Snort.  I disabled the offload features and then verified that this resulted in no more packets larger than 1500 bytes.  The packet capture truly was "full" packet capture.

[ UPDATE: A reader asked why we couldn't simply change Snort's default snaplen to a larger value to capture the 2900-byte packets.  While it's true that would solve the "full" packet capture problem, another problem would remain.  Since the packets are being reassembled on the NIC, Snort is not able to properly perform target-based reassembly (see the Snort manual link above).  This opens the door for potential IDS evasion/insertion attacks.  NIC offload settings need to be disabled so that Snort sees the same packets the destination host does. ]

Some NIC/driver combinations may disable these offload settings by default, while others enable it by default.  You should check your sensors now before you get into a situation where you really need that full packet capture and find out that you don't actually have it.  To check, run ethtool with the "-k" (lower-case k) option on the interface you'd like to check.  For example, to check eth0:

ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off

You should repeat this for every interface in your system, as you may have NICs from different manufacturers with different defaults.

You can set these options using the "-K" (upper-case K) option to ethtool and specify which option you'd like to set.  For example, to disable tcp-segmentation-offload for eth0:

ethtool -K eth0 tso off

You can set multiple options in one "ethtool" command, but this can be problematic if your card doesn't support all of the settings.  To avoid this, you could invoke ethtool for each option like this:

ethtool -K eth0 rx off
ethtool -K eth0 tx off
ethtool -K eth0 sg off
ethtool -K eth0 tso off
ethtool -K eth0 ufo off
ethtool -K eth0 gso off
ethtool -K eth0 gro off
ethtool -K eth0 lro off 

Or we could simply wrap the ethtool command in a for-loop like this:

for i in rx tx sg tso ufo gso gro lro; do ethtool -K eth0 $i off; done

These settings will remain in effect only while the OS is booted, so this needs to be applied at every boot.  This can be done by adding the above for-loop as a "post-up" command for each of the interfaces in /etc/network/interfaces.  If you're still using the graphical Network Manager to configure your interfaces, I've put together some documentation on disabling it and configuring interfaces and their offloading features via /etc/network/interfaces:

https://securityonion.net/docs/network-configuration.html

I'd really like some feedback on this:

• What were your default settings? (ethtool -k eth0)
• Did you have any problems disabling the offload features?
• Did you notice any difference in performance after disabling the offload features?
• Is there a better way of disabling offload features globally?  I tried putting the commands in /etc/rc.local and /etc/init/securityonion.conf, but the only way I could get it to work consistently was via /etc/network/interfaces as documented above.
• I'm considering disabling offload features by default in the Security Onion Setup script.  Can anyone think of any reason why this might be a bad idea?