Issue 11
© Solarflare Communications 2014
85
Solarflare Server Adapter
User Guide
TCP Protocol Tuning
TCP Performance can also be improved by tuning kernel TCP settings. Settings include adjusting send
and receive buffer sizes, connection backlog, congestion control, etc.
For Linux kernel versions, including 2.6.16 and later, initial buffering settings should provide good
performance. However for earlier kernel versions, and for certain applications even on later kernels,
tuning buffer settings can significantly benefit throughput. To change buffer settings, adjust the
tcp_rmem
and
tcp_wmem
using the
sysctl
command:
Receive buffering:
sysctl net.ipv4.tcp_rmem="<min> <default> <max>"
Transmit buffering:
sysctl net.ipv4.tcp_wmem="<min> <default> <max>"
(
tcp_rmem
and
tcp_wmem
can also be adjusted for IPV6 and globally with the net.ipv6 and net.core
variable prefixes respectively).
Typically it is sufficient to tune just the max buffer value. It defines the largest size the buffer can
grow to. Suggested alternate values are max=500000 (1/2 Mbyte). Factors such as link latency,
packet loss and CPU cache size all influence the affect of the max buffer size values. The minimum
and default values can be left at their defaults minimum=4096 and default=87380.
Receive Side Scaling (RSS)
Solarflare adapters support Receive Side Scaling (RSS). RSS enables packet receive-processing to
scale with the number of available CPU cores. RSS requires a platform that supports MSI-X
interrupts. RSS is enabled by default.
When RSS is enabled the controller uses multiple receive queues into which to deliver incoming
packets. The receive queue selected for an incoming packet is chosen in such a way as to ensure that
packets within a TCP stream are all sent to the same receive queue – this ensures that packet-
ordering within each stream is maintained. Each receive queue has its own dedicated MSI-X
interrupt which ideally should be tied to a dedicated CPU core. This allows the receive side TCP
processing to be distributed amongst the available CPU cores, providing a considerable performance
advantage over a conventional adapter architecture in which all received packets for a given
interface are processed by just one CPU core. RSS can be restricted to only process receive queues
on the NUMA node local to the Solarflare adapter. To configure this the driver module option
rss_numa_local
should be set to 1.
By default the drivers enables RSS and configures one RSS Receive queue per CPU core. The number
of RSS Receive queues can be controlled via the driver module parameter
rss_cpus
. The following
options are supported: