Linux Networking: How to disable/enable offload features, RX/TX checksum, scatter, gather and beyond

root's picture

TCP offload engine or TOE is a technology used in network interface cards (NIC) to offload processing of the entire TCP/IP stack to the network controller. It is primarily used with high-speed network interfaces, such as gigabit Ethernet and 10 Gigabit Ethernet, where processing overhead of the network stack becomes significant.
The term, TOE, is often used to refer to the NIC itself, although circuit board engineers may use it to refer only to the integrated circuit included on the card which processes the TCP headers. TOEs are often suggested as a way to reduce the overhead associated with IP storage protocols such as iSCSI and NFS. Read more about this on wikipedia.

Linux doesn't support TOE by default because of many reasons mentioned here.
But even if TOE is not suitable, at least for now, there are other settings that could improve your network (especially if you use it for iSCSI). So read on and remember: in most of the cases, these settings must be ON but you have to test and see what is suitable for your environment. Enjoy!

Offload features


Most of these offload features are configured using the ethtool tool. In order to check which features are set, you can use the -k parameter. Also, in order to disable/enable, use -K or --offload parameter along with the desired features.
Display: ethtool -k ethX
Enable (recommended): ethtool -K ethX rx on tx on
Disable: ethtool -K ethX rx off tx off

Example:

root@pluto:~# ethtool -k eth2
Features for eth2:
rx-checksumming: off
tx-checksumming: on
        tx-checksum-ipv4: off [fixed]
        tx-checksum-unneeded: off [fixed]
        tx-checksum-ip-generic: on
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp6-segmentation: off [fixed]
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: on
loopback: off [fixed]

root@pluto:~# ethtool --offload eth2 rx off tx off
Actual changes:
tx-checksumming: off
        tx-checksum-ip-generic: off
scatter-gather: off
        tx-scatter-gather: off [requested on]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off [requested on]
generic-segmentation-offload: off [requested on]

root@pluto:~# ethtool --offload eth2 rx on tx on
Actual changes:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ip-generic: on
scatter-gather: on
        tx-scatter-gather: on
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
generic-segmentation-offload: on

TCP Segmentation Offload (TSO)


This is useful to reduce CPU overhead and it is also called Large Segment Offload (LSO).
In order to enable/disable TCP segmentation offload, you must use ethtool command with tso option:
Display: ethtool -k ethX | grep tcp-segmentation
Enable (recommended): ethtool -K ethX tso on
Disable: ethtool -K ethX tso off

Example:

root@pluto:~# ethtool -K eth2 tso off
root@pluto:~# ethtool -k eth2 | grep segmentation
tcp-segmentation-offload: off
        tx-tcp-segmentation: off
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp6-segmentation: off [fixed]
generic-segmentation-offload: on
tx-fcoe-segmentation: off [fixed]

root@pluto:~# ethtool -K eth2 tso on
root@pluto:~# ethtool -k eth2 | grep segmentation
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp6-segmentation: off [fixed]
generic-segmentation-offload: on
tx-fcoe-segmentation: off [fixed]

Scatter and Gather


Scatter and Gather (Vectored I/O) is a concept that was primarily used in hard disks and it enhances large I/O request performance, if supported by the hardware.
To manipulate this feature, you must use the same ethtool command with the following parameter:

Display: ethtool -k ethX | grep scatter-gather
Enable (recommended): ethtool -K ethX sg on
Disable: ethtool -K ethX sg off

Example:

root@pluto:~# ethtool -k eth2 | grep scatter-gather
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]

root@pluto:~# ethtool -K eth2 sg off
Actual changes:
scatter-gather: off
        tx-scatter-gather: off
tcp-segmentation-offload: off
        tx-tcp-segmentation: off [requested on]
generic-segmentation-offload: off [requested on]

root@pluto:~# ethtool -K eth2 sg on
Actual changes:
scatter-gather: on
        tx-scatter-gather: on
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
generic-segmentation-offload: on

Generic Segmentation Offload


This feature is related to TSO so it is advisable to enable them together. It can be done with ethtool command also:
Display: ethtool -k ethX | grep generic-segmentation-offload
Enable (recommended): ethtool -K ethX gso on
Disable: ethtool -K ethX gso off

Example:

root@pluto:~# ethtool -K eth2 gso off
root@pluto:~# ethtool -k eth2 | grep generic-segmentation-offload
generic-segmentation-offload: off

root@pluto:~# ethtool -K eth2 gso on
root@pluto:~# ethtool -k eth2 | grep generic-segmentation-offload
generic-segmentation-offload: on

UDP / TCP Checksum errors in tcpdump output


To avoid any extra search from your part, if you have offload features enabled and you see cksum incorrect in tcpdump output, without any packet errors and your network is working properly: it is nothing to worry about because the checksum is actually calculated on the network adapter and the tcpdump is showing the checksum calculated on kernel level. Read more here if you are curious.

Thou shalt not steal!

If you want to use this information on your own website, please remember: by doing copy/paste entirely it is always stealing and you should be ashamed of yourself!
Have at least the decency to create your own text and comments and run the commands on your own servers and provide your output, not what I did!

Or at least link back to this website.