Issue 11
© Solarflare Communications 2014
261
Solarflare Server Adapter
User Guide
TCP Protocol Tuning
TCP Performance can also be improved by tuning kernel TCP settings. Settings include adjusting send
and receive buffer sizes, connection backlog, congestion control, etc.
Typically it is sufficient to tune just the max buffer value. It defines the largest size the buffer can
grow to. Suggested alternate values are max=500000 (1/2 Mbyte). Factors such as link latency,
packet loss and CPU cache size all influence the affect of the max buffer size values. The minimum
and default values can be left at their defaults minimum=4096 and default=87380.
For advice on tuning the guest TCP stack consult the documentation for the guest operating system.
Receive Side Scaling (RSS)
Solarflare adapters support Receive Side Scaling (RSS). RSS enables packet receive-processing to
scale with the number of available CPU cores. RSS requires a platform that supports MSI-X
interrupts. RSS is enabled by default.
When RSS is enabled the controller uses multiple receive queues into which to deliver incoming
packets. The receive queue selected for an incoming packet is chosen in such a way as to ensure that
packets within a TCP stream are all sent to the same receive queue – this ensures that packet-
ordering within each stream is maintained. Each receive queue has its own dedicated MSI-X
interrupt which ideally should be tied to a dedicated CPU core. This allows the receive side TCP
processing to be distributed amongst the available CPU cores, providing a considerable performance
advantage over a conventional adapter architecture in which all received packets for a given
interface are processed by just one CPU core.
RSS will be enabled whenever NetQueue is not and Solarflare recommend using NetQueue on
VMware ESX hosts.
Other Considerations
PCI Express Lane Configurations
The PCI Express (PCIe) interface used to connect the adapter to the server can function at different
widths. This is independent of the physical slot size used to connect the adapter. The possible widths
are multiples x1, x2, x4, x8 and x16 lanes of (2.5Gbps for PCIe Gen. 1, 5.0 Gbps for PCIe Gen. 2) in
each direction. Solarflare Adapters are designed for x8 lane operation.
On some server motherboards, choice of PCIe slot is important. This is because some slots (including
ones that are physically x8 or x16 lanes) may only electrically support x4 lanes. In x4 lane slots,
Solarflare PCIe adapters will continue to operate, but not at full speed. The Solarflare driver will warn
you if it detects the adapter is plugged into a PCIe slot which electrically has fewer than x8 lanes.
For SFN5xxxx adapters, which require a PCIe Gen. 2 slot for optimal operation, a warning will be
given if they are installed in a PCIe Gen. 1 slot. Warning messages can be viewed in
dmesg
from
/
var/log/messages
.
Memory bandwidth
Many chipsets/CPUs use multiple channels to access main system memory. Maximum memory
performance is only achieved when the server can make use of all channels simultaneously. This