Wednesday, October 19, 2011

When is full packet capture NOT full packet capture?

I was looking at some packets recently and noticed the Wireshark message "Packet size limited during capture".  This was strange since the packets came from a Sguil sensor performing full packet capture using Snort's default snaplen on a standard Ethernet connection (no Jumbo frames and no VLAN tags).  Drilling down into the packet capture, some of the packets were 2900 bytes and Snort was only capturing the first 1500 bytes.  The full packet capture was not "full" packet capture after all.

So where did the 2900-byte packets come from?

The OS had enabled by default the following NIC offload features:
tcp-segmentation-offload (tso)
generic-segmentation-offload (gso)
generic-receive-offload (gro)

For more information about these features and their side effects, please see:
http://www.unleashnetworks.com/blog/?p=307
http://wiki.wireshark.org/CaptureSetup/Offloading
http://manual.snort.org/node7.html
http://www.inliniac.net/blog/2007/04/20/snort_inline-and-tcp-segmentation-offloading.html

I won't repeat all the information in those links, but I'll summarize by saying that the NIC was reassembling packets before being passed up the stack to Snort.  I disabled the offload features and then verified that this resulted in no more packets larger than 1500 bytes.  The packet capture truly was "full" packet capture.

[ UPDATE: A reader asked why we couldn't simply change Snort's default snaplen to a larger value to capture the 2900-byte packets.  While it's true that would solve the "full" packet capture problem, another problem would remain.  Since the packets are being reassembled on the NIC, Snort is not able to properly perform target-based reassembly (see the Snort manual link above).  This opens the door for potential IDS evasion/insertion attacks.  NIC offload settings need to be disabled so that Snort sees the same packets the destination host does. ]

Some NIC/driver combinations may disable these offload settings by default, while others enable it by default.  You should check your sensors now before you get into a situation where you really need that full packet capture and find out that you don't actually have it.  To check, run ethtool with the "-k" (lower-case k) option on the interface you'd like to check.  For example, to check eth0:
ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
You should repeat this for every interface in your system, as you may have NICs from different manufacturers with different defaults.

You can set these options using the "-K" (upper-case K) option to ethtool and specify which option you'd like to set.  For example, to disable tcp-segmentation-offload for eth0:
ethtool -K eth0 tso off
You can set multiple options in one "ethtool" command, but this can be problematic if your card doesn't support all of the settings.  To avoid this, you could invoke ethtool for each option like this:
ethtool -K eth0 rx off
ethtool -K eth0 tx off
ethtool -K eth0 sg off
ethtool -K eth0 tso off
ethtool -K eth0 ufo off
ethtool -K eth0 gso off
ethtool -K eth0 gro off
ethtool -K eth0 lro off 
Or we could simply wrap the ethtool command in a for-loop like this:
for i in rx tx sg tso ufo gso gro lro; do ethtool -K eth0 $i off; done
These settings will remain in effect only while the OS is booted, so this needs to be applied at every boot.  This can be done by adding the above for-loop as a "post-up" command for each of the interfaces in /etc/network/interfaces.  If you're still using the graphical Network Manager to configure your interfaces, I've put together some documentation on disabling it and configuring interfaces and their offloading features via /etc/network/interfaces:


I'd really like some feedback on this:
  • What were your default settings? (ethtool -k eth0)
  • Did you have any problems disabling the offload features?
  • Did you notice any difference in performance after disabling the offload features?
  • Is there a better way of disabling offload features globally?  I tried putting the commands in /etc/rc.local and /etc/init/securityonion.conf, but the only way I could get it to work consistently was via /etc/network/interfaces as documented above.
  • I'm considering disabling offload features by default in the Security Onion Setup script.  Can anyone think of any reason why this might be a bad idea?

4 comments:

Anonymous said...

Thanks for this excellent post. I was troubleshooting "large ip packet" capture issues on Ubuntu 12.04 vs Wireshark, and this post explains everything nicely.

Apparently these nic features are now enabled by default in modern linux distros, so these issues are beginning to surface.

My Xubuntu 12.04 desktop with Marvel Yukon nic (sky2 kernel driver) has:

tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off

/Sune

Anonymous said...

My Snort machine that I'm going to install SO on, has the following defaults, Intel gigabit card:

Features for em1:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: on
loopback: off [fixed]
rx-fcs: off
rx-all: off

-sjm

Anonymous said...

The former output was from Ubuntu 12.10 - sorry.

This is from Ubuntu 12.04.1 LTS on the same hardware.

Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: off

-sjm

Anonymous said...

This is good stuff. On CentOS/RHEL you can create a /sbin/ifup-local file, then add those commands to it. The /etc/sysconfig/network-scripts/ifup-post script looks for a /sbin/ifup-local and if there runs the commands on boot within that file. I have other offloading showing (rxhash, rxvlan, and txvlan) but I am not sure if others do as well. This was very helpful. thanks.
Here is what I added to my /sbin/ifup-local

/sbin/ethtool -K em2 rx off
/sbin/ethtool -K em2 tx off
/sbin/ethtool -K em2 sg off
/sbin/ethtool -K em2 tso off
/sbin/ethtool -K em2 ufo off
/sbin/ethtool -K em2 gso off
/sbin/ethtool -K em2 gro off
/sbin/ethtool -K em2 lro off