TCP offload engine or TOE is a technology used in network interface cards (NIC) to offload processing of the entire TCP/IP stack to the network controller. It is primarily used with high-speed network interfaces, such as gigabit Ethernet and 10 Gigabit Ethernet, where processing overhead of the network stack becomes significant.
The term, TOE, is often used to refer to the NIC itself, although circuit board engineers may use it to refer only to the integrated circuit included on the card which processes the TCP headers. TOEs are often suggested as a way to reduce the overhead associated with IP storage protocols such as iSCSI and NFS. Read more about this on wikipedia.
Linux doesn't support TOE by default because of many reasons mentioned here.
But even if TOE is not suitable, at least for now, there are other settings that could improve your network (especially if you use it for iSCSI). So read on and remember: in most of the cases, these settings must be ON but you have to test and see what is suitable for your environment. Enjoy!
Offload features
Most of these offload features are configured using the ethtool tool. In order to check which features are set, you can use the -k parameter. Also, in order to disable/enable, use -K or --offload parameter along with the desired features.
Display: ethtool -k ethX
Enable (recommended): ethtool -K ethX rx on tx on
Disable: ethtool -K ethX rx off tx off
Example:
root@pluto:~# ethtool -k eth2
Features for eth2:
rx-checksumming: off
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-unneeded: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp6-segmentation: off [fixed]
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: on
loopback: off [fixed]root@pluto:~# ethtool --offload eth2 rx off tx off
Actual changes:
tx-checksumming: off
tx-checksum-ip-generic: off
scatter-gather: off
tx-scatter-gather: off [requested on]
tcp-segmentation-offload: off
tx-tcp-segmentation: off [requested on]
generic-segmentation-offload: off [requested on]root@pluto:~# ethtool --offload eth2 rx on tx on
Actual changes:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ip-generic: on
scatter-gather: on
tx-scatter-gather: on
tcp-segmentation-offload: on
tx-tcp-segmentation: on
generic-segmentation-offload: onTCP Segmentation Offload (TSO)
This is useful to reduce CPU overhead and it is also called Large Segment Offload (LSO).
In order to enable/disable TCP segmentation offload, you must use ethtool command with tso option:
Display: ethtool -k ethX | grep tcp-segmentation
Enable (recommended): ethtool -K ethX tso on
Disable: ethtool -K ethX tso off
Example:
root@pluto:~# ethtool -K eth2 tso off
root@pluto:~# ethtool -k eth2 | grep segmentation
tcp-segmentation-offload: off
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp6-segmentation: off [fixed]
generic-segmentation-offload: on
tx-fcoe-segmentation: off [fixed]root@pluto:~# ethtool -K eth2 tso on
root@pluto:~# ethtool -k eth2 | grep segmentation
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp6-segmentation: off [fixed]
generic-segmentation-offload: on
tx-fcoe-segmentation: off [fixed]Scatter and Gather
Scatter and Gather (Vectored I/O) is a concept that was primarily used in hard disks and it enhances large I/O request performance, if supported by the hardware.
To manipulate this feature, you must use the same ethtool command with the following parameter:
Display: ethtool -k ethX | grep scatter-gather
Enable (recommended): ethtool -K ethX sg on
Disable: ethtool -K ethX sg off
Example:
root@pluto:~# ethtool -k eth2 | grep scatter-gather
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]root@pluto:~# ethtool -K eth2 sg off
Actual changes:
scatter-gather: off
tx-scatter-gather: off
tcp-segmentation-offload: off
tx-tcp-segmentation: off [requested on]
generic-segmentation-offload: off [requested on]root@pluto:~# ethtool -K eth2 sg on
Actual changes:
scatter-gather: on
tx-scatter-gather: on
tcp-segmentation-offload: on
tx-tcp-segmentation: on
generic-segmentation-offload: onGeneric Segmentation Offload
This feature is related to TSO so it is advisable to enable them together. It can be done with ethtool command also:
Display: ethtool -k ethX | grep generic-segmentation-offload
Enable (recommended): ethtool -K ethX gso on
Disable: ethtool -K ethX gso off
Example:
root@pluto:~# ethtool -K eth2 gso off root@pluto:~# ethtool -k eth2 | grep generic-segmentation-offload generic-segmentation-offload: off
root@pluto:~# ethtool -K eth2 gso on root@pluto:~# ethtool -k eth2 | grep generic-segmentation-offload generic-segmentation-offload: on
UDP / TCP Checksum errors in tcpdump output
To avoid any extra search from your part, if you have offload features enabled and you see cksum incorrect in tcpdump output, without any packet errors and your network is working properly: it is nothing to worry about because the checksum is actually calculated on the network adapter and the tcpdump is showing the checksum calculated on kernel level. Read more here if you are curious.
