manualshive.com logo in svg

Solarflare Solarflare XtremeScale series, User Manual, Page: 88 / 400

Share
Download
Solarflare Solarflare XtremeScale series User Manual Download Page 88

Solarflare

 

Server

 

Adapter

 

User

 

Guide

Solarflare

 

Adapters

 

on

 

Linux

Issue

 

20

©

 

Solarflare

 

Communications

 

2017

81

LACP

 

Bonding

LACP

 

Bonding

 

is

 

not

 

currently

 

supported

 

using

 

the

 

NIC

 

Partitioning

 

configuration

 

mode

 

as

 

the

 

LACP

 

partner

 

i.e.

 

the

 

switch

 

will

 

be

 

unaware

 

of

 

the

 

configured

 

partitions.

Users

 

are

 

advised

 

to

 

refer

 

to

 

the

 

sfc

 

driver

 

release

 

notes

 

for

 

current

 

limitations

 

when

 

using

 

the

 

NIC

 

partitioning

 

features.

3.16

 

Receive

 

Side

 

Scaling

 

(RSS)

Solarflare

 

adapters

 

support

 

Receive

 

Side

 

Scaling

 

(RSS).

 

RSS

 

enables

 

packet

 

receive

‐

processing

 

to

 

scale

 

with

 

the

 

number

 

of

 

available

 

CPU

 

cores.

 

RSS

 

requires

 

a

 

platform

 

that

 

supports

 

MSI

‐

X

 

interrupts.

 

RSS

 

is

 

enabled

 

by

 

default.

When

 

RSS

 

is

 

enabled

 

the

 

controller

 

uses

 

multiple

 

receive

 

queues

 

to

 

deliver

 

incoming

 

packets.

 

The

 

receive

 

queue

 

selected

 

for

 

an

 

incoming

 

packet

 

is

 

chosen

 

to

 

ensure

 

that

 

packets

 

within

 

a

 

TCP

 

stream

 

are

 

all

 

sent

 

to

 

the

 

same

 

receive

 

queue

 

–

 

this

 

ensures

 

that

 

packet

‐

ordering

 

within

 

each

 

stream

 

is

 

maintained.

 

Each

 

receive

 

queue

 

has

 

its

 

own

 

dedicated

 

MSI

‐

X

 

interrupt

 

which

 

ideally

 

should

 

be

 

tied

 

to

 

a

 

dedicated

 

CPU

 

core.

 

This

 

allows

 

the

 

receive

 

side

 

TCP

 

processing

 

to

 

be

 

distributed

 

amongst

 

the

 

available

 

CPU

 

cores,

 

providing

 

a

 

considerable

 

performance

 

advantage

 

over

 

a

 

conventional

 

adapter

 

architecture

 

in

 

which

 

all

 

received

 

packets

 

for

 

a

 

given

 

interface

 

are

 

processed

 

by

 

just

 

one

 

CPU

 

core.

 

RSS

 

can

 

be

 

restricted

 

to

 

only

 

process

 

receive

 

queues

 

on

 

the

 

NUMA

 

node

 

local

 

to

 

the

 

Solarflare

 

adapter.

 

To

 

configure

 

this

 

the

 

driver

 

module

 

option

 

rss_numa_local

 

should

 

be

 

set

 

to

 

1.

By

 

default

 

the

 

driver

 

enables

 

RSS

 

and

 

configures

 

one

 

RSS

 

Receive

 

queue

 

per

 

CPU

 

core.

 

The

 

number

 

of

 

RSS

 

Receive

 

queues

 

can

 

be

 

controlled

 

via

 

the

 

driver

 

module

 

parameter

 

rss_cpus

.

 

The

 

following

 

table

 

identifies

 

rss_cpus

 

options.

Table

 

17:

 

rss_cpus

 

Options

Option

Description

Interrupt

 

Affinity

 

(MSI

‐

X)

<num_cpus>

Indicates

 

the

 

number

 

of

 

RSS

 

queues

 

to

 

create.

A

 

separate

 

MSI

‐

X

 

interrupt

 

for

 

a

 

receive

 

queue

 

is

 

affinitized

 

to

 

each

 

CPU.

packages

An

 

RSS

 

queue

 

will

 

be

 

created

 

for

 

each

 

multi

‐

core

 

CPU

 

package.

 

The

 

first

 

CPU

 

in

 

the

 

package

 

will

 

be

 

chosen.

A

 

separate

 

MSI

‐

X

 

interrupt

 

for

 

a

 

receive

 

queue,

 

is

 

affinitized

 

to

 

each

 

of

 

the

 

designated

 

package

 

CPUs.

Summary of Contents for Solarflare XtremeScale series

Page 1: ...e part of SOLARFLARE Communications Inc or its affiliates The only warranties granted by SOLARFLARE Communications Inc or its affiliates in connection with the Product described in this document are t...

Page 2: ...pter User Guide Issue 20 Solarflare Communications 2017 ii Trademarks OpenOnload EnterpriseOnload XtremeScale and Flareon are registered trademarks of Solarflare Communications Inc in the United State...

Page 3: ...ttaching a Cable RJ 45 35 2 5 Attaching a Cable SFP 36 2 6 Supported SFP Cables 38 2 7 Supported SFP 10G SR Optical Transceivers 40 2 8 Supported SFP 10G LR Optical Transceivers 41 2 9 QSFP Transceive...

Page 4: ...SE Linux Enterprise Server 69 3 11 Configuring the Solarflare Adapter 70 3 12 Setting Up VLANs 73 3 13 Setting Up Teams 73 3 14 NIC Partitioning 74 3 15 NIC Partitioning with SR IOV 78 3 16 Receive Si...

Page 5: ...ostics 177 4 16 Using SAM for Boot ROM Configuration 181 4 17 Managing Firmware with SAM 183 4 18 Configuring Network Adapter Properties in Windows 184 4 19 Windows Command Line Tools 189 4 20 Sfboot...

Page 6: ...Kernel and Network Adapter Statistics 317 7 SR IOV Virtualization Using KVM 326 7 1 Introduction 326 7 2 SR IOV 331 7 3 KVM Network Architectures 333 7 4 PF IOV 346 7 5 General Configuration 348 7 6...

Page 7: ...2017 vii 9 Solarflare Boot Manager 374 9 1 Introduction 374 9 2 Solarflare Boot Manager 375 9 3 iPXE Support 376 9 4 sfupdate Options for PXE upgrade downgrade 376 9 5 Starting PXE Boot 378 9 6 iPXE...

Page 8: ...Onload to achieve minimum latency and maximum throughput 1 1 Virtual NIC Interface Solarflare s VNIC architecture provides the key to efficient server I O and is flexible enough to be applied to mult...

Page 9: ...s can be instantiated for each running guest operating system or application giving it a direct pipeline to the Ethernet network This architecture provides the most efficient way to maximize network a...

Page 10: ...ployment of 1024 channels between Virtual and Physical Functions Support Alternate Routing ID ARI SR IOV is not supported for Solarflare adapters on IBM System p servers 10 Gigabit Ethernet Supports t...

Page 11: ...ardware to steer packets based on the MAC address to a VNIC Hardware timestamps The Solarflare Flareon SFN7000 and XtremeScale SFN8000 series adapters can support hardware timestamping for all packets...

Page 12: ...bled for the Plus version PTP and hardware timestamps Yes factory enabled for the Plus version 1PPS Optional bracket and cable assembly not factory installed SR IOV Yes Network ports 2 x QSFP 40G 10G...

Page 13: ...240 VF 2048 VI 2048 MSI X Interrupts Supports OpenOnload Yes factory enabled for the Onload and Plus versions PTP and hardware timestamps Yes factory enabled for the Plus version 1PPS Optional bracke...

Page 14: ...16 PF 240 VF 2048 VI 2048 MSI X Interrupts Supports OpenOnload Yes factory enabled PTP and hardware timestamps Yes factory enabled 1PPS Optional bracket and cable assembly not factory installed SR IO...

Page 15: ...2048 MSI X Interrupts Supports OpenOnload Yes factory enabled PTP and hardware timestamps Enabled by installing AppFlex license 1PPS Optional bracket and cable assembly not factory installed SR IOV Y...

Page 16: ...VI 2048 MSI X Interrupts Supports OpenOnload Enabled by installing AppFlex license PTP and hardware timestamps Enabled by installing AppFlex license 1PPS Optional bracket and cable assembly not factor...

Page 17: ...VI 2048 MSI X Interrupts Supports OpenOnload Enabled by installing AppFlex license PTP and hardware timestamps Enabled by installing AppFlex license 1PPS Optional bracket and cable assembly not factor...

Page 18: ...slots PCIe features support Per adapter 16 PF 240 VF 2048 VI 2048 MSI X Interrupts Supports OpenOnload Enabled by installing AppFlex license PTP and hardware timestamps Enabled by installing AppFlex l...

Page 19: ...SR IOV virtual functions per port Virtual NIC support 1024 vNIC interfaces per port Supports OpenOnload Yes SR IOV Yes Network ports 2 x SFP 10G 1G Part number SFN6122F Controller silicon SFC9020 Pow...

Page 20: ...edge connector usable in x8 and x16 slots 127 SR IOV virtual functions per port Virtual NIC support 1024 vNIC interfaces per port Supports OpenOnload Yes SR IOV Yes Network ports 2 x SFP 10G 1G Part...

Page 21: ...lanes Gen2 5 0GT s x8 edge connector usable in x8 and x16 slots Virtual NIC support 1024 vNIC interfaces per port Supports OpenOnload No SR IOV Yes1 1 SR IOV is not supported for Solarflare adapters o...

Page 22: ...con SFC9020 Power 5 9W typical PCI Express 8 lanes Gen2 5 0GT s x8 edge connector usable in x8 and x16 slots 127 SR IOV virtual functions per port Virtual NIC support 1024 vNIC interfaces per port Sup...

Page 23: ...edge connector usable in x8 and x16 slots 127 SR IOV virtual functions per port Virtual NIC support 1024 vNIC interfaces per port Supports OpenOnload Yes SR IOV Yes Ports 4 x 10GBASE KX4 backplane tra...

Page 24: ...are ESX 5 0 ESXi 5 1 5 5 and 6 0 Linux KVM Support includes all minor updates releases service packs of the above major releases for which the distributor has not yet declared end of life support Sola...

Page 25: ...page at https support solarflare com myappflex The sfkey utility application is used to install the generated license key file on selected adapters For detailed instructions for sfkey and license ins...

Page 26: ...butors may be used to endorse or promote products derived from this software without specific prior written permission THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS AS IS AND ANY...

Page 27: ...ision reception which can be determined by turning the equipment off and on the user is encouraged to try to correct the interference by one or more of the following measures Reorient or relocate the...

Page 28: ...r example equipment may use one of a multiple of different LDMs depending on path length of the laser communication signal This equipment is not basic consumer ITE The equipment is installed and maint...

Page 29: ...tory Approval Additional Regulatory Information for SFN8722 SFN8542 SFN8522M and SFN8042 adapters VCCI A A A Pb Hg Cd Cr VI PB B PBDE RC0402FR series resistor x o o o o o ERJ 2GE series resistor x o o...

Page 30: ...an CNS 13438 2006 Class A Japan VCCI Regulations V 3 2014 04 Class A South Korea KCC KN 32 KN 35 Australia AS NZS CISPR 22 2009 A1 2010 Safety1 Europe BS EN 60950 1 2006 A11 2009 A1 2010 A12 2011 A2 2...

Page 31: ...ations 2017 24 Additional Regulatory Information for SFN8522 adapter B VCCI B B B B Pb Hg Cd Cr VI PB B PBDE RC0402FR series resistor x o o o o o ERJ 2GE series resistor x o o o o o CRCW0402 series re...

Page 32: ...an CNS 13438 2006 Class B Japan VCCI Regulations V 3 2014 04 Class B South Korea KCC KN 32 KN 35 Australia AS NZS CISPR 22 2009 A1 2010 Safety1 Europe BS EN 60950 1 2006 A11 2009 A1 2010 A12 2011 A2 2...

Page 33: ...1998 A1 2001 A2 2003 US FCC Part 15 Class B Canada ICES 003 NMB 003 Class B Taiwan CNS 13438 2006 Class B Japan VCCI Regulations V 3 2010 Class B South Korea KCC KN 22 KN 24 Australia AS NZS CISPR 22...

Page 34: ...ass B Canada ICES 003 NMB 003 Class B Taiwan CNS 13438 2006 Class B Japan VCCI Regulations V 3 2014 04 Class B South Korea KCC KN 32 KN 24 Australia AS NZS CISPR 22 2009 A1 2010 Safety1 1 The safety a...

Page 35: ...ss B Canada ICES 003 NMB 003 Class B Taiwan CNS 13438 2006 Class B Japan VCCI Regulations V 3 2010 Class B South Korea KCC KN 22 KN 24 Australia AS NZS CISPR 22 2009 Safety1 1 The safety assessment ha...

Page 36: ...55024 1998 A1 2001 A2 2003 US FCC Part 15 Class B Canada ICES 003 NMB 003 Class B Taiwan CNS 13438 2006 Class A Japan VCCI Regulations V 3 2010 Class A Australia AS NZS CISPR 22 2009 Safety1 1 The sa...

Page 37: ...Details EMC Europe BS EN 55022 2006 BS EN 55024 1998 A1 2001 A2 2003 US FCC Part 15 Class B Canada ICES 003 NMB 003 Class B Safety1 1 The safety assessment has been concluded on this product as a com...

Page 38: ...d Mode on page 46 LED States on page 48 Configure QSFP Adapter on page 49 Single Optical Fiber RX Configuration on page 51 Installing the OCP Mezzanine AdapterSolarflare Mezzanine Adapters SFN5814H an...

Page 39: ...ort 40GbE PCIe 3 0 QSFP Server Adapter Solarflare Flareon Ultra SFN7124F Quad Port 10GbE PCIe 3 0 SFP Server Adapter Solarflare Flareon Ultra SFN7122F Dual Port 10GbE PCIe 3 0 Server I O Adapter Solar...

Page 40: ...rEdge C6200 series servers Solarflare SFN6822F Dual Port 10GbE SFP FlexibleLOM Onload Server Adapter Solarflare SFN5814H Quad Port 10G Ethernet Mezzanine Adapter for IBM BladeCenter Solarflare SFN5812...

Page 41: ...dapter remove the screws securing the bracket 2 Slide the bracket away from the adapter 3 Taking care not the overtighten the screws attach the full height bracket to the adapter 2 3 Inserting the Ada...

Page 42: ...to install drivers for the new hardware Click Cancel or abort the installation and refer to the relevant chapter in this manual for how to install the Solarflare adapter drivers for your operating sys...

Page 43: ...re SFP Server Adapters can be connected to the network using either an SFP Direct Attach cable or a fiber optic cable Attaching an SFP Direct Attach Cable To attach an SFP Direct Attach cable 1 Turn t...

Page 44: ...e from the adapter socket Attaching a fiber optic cable WARNING Do not look directly into the fiber transceiver or cables as the laser beams can damage your eyesight To attach a fiber optic cable 1 Re...

Page 45: ...ody of the adapter bracket and remove it from the adapter 2 6 Supported SFP Cables Table 2 is a list of supported SFP cables that have been tested by Solarflare Solarflare is not aware of any issues p...

Page 46: ...SFP 10GE DAC 3m 3m Molex 74752 1101 1m Molex 74752 2301 3m Molex 74752 3501 5m Molex 74752 9093 1m 37 0960 01 0K585N Molex 74752 9094 3m 37 0961 01 0J564N Molex 74752 9096 5m 37 0962 01 0H603N Panduit...

Page 47: ...e supported Table 3 Supported SFP 10G Optical SR Transceivers Manufacturer Product Code Notes SFN5xxx SFN6xxx SFN7xxx SFN8xxx Arista SFP 10G SR 10G Arista XVR 00002 02 10G Avago AFBR 703SDZ 10G Avago...

Page 48: ...rflare Solarflare is not aware of any issues preventing the use of other brands of 10G LR transceivers with Solarflare network adapters However only transceivers in the table below have been fully ver...

Page 49: ...adapters However only products listed in the tables below have been fully verified and are therefore supported Supported QSFP 40GBASE SR4 Transceivers Solarflare Flareon Ultra SFN7000 series and Xtrem...

Page 50: ...ing QSFP Direct Attach Cables DAC QSFP cables may not work with all switches Table 6 Supported QSFP Active Optical Cables Manufacturer Product Code Notes SFN5xxx SFN6xxx SFN7xxx SFN8x42 Avago AFBR 7QE...

Page 51: ...R QSFP2SFP 1M 3M copper DAC cables are fully tested and compatible with the Solarflare SFN7142Q and SFN7042Q server I O adapters These cables are compliant with the SFF 8431 SFF 8432 SFF 8436 SFF 8472...

Page 52: ...sted by Solarflare Solarflare is not aware of any issues preventing the use of other brands of 1000BASE T transceivers with the Solarflare network adapters However only transceivers in the table below...

Page 53: ...tiation to automatically select the highest speed supported in common with the link partner On SFP adapters the SFP module transceiver determines the supported speeds typically SFP modules only suppor...

Page 54: ...SFP optical module 10G No 10G SFPopticalmodule 1G No 1G SFP optical module 10G 1G No 10G or 1G Dual speed modules run at the maximum speed 10G unless explicitly configured to the lower speed 1G SFP 10...

Page 55: ...LED states are as follows Table 13 LED States Adapter Type LED Description State QSFP SFP SFP Speed Green solid at all speeds Activity Flashing green when network traffic is present LEDs are OFF when...

Page 56: ...re 40G breakout cables have only 2 physical cables see Supported QSFP to SFP Breakout Cables on page 44 Breakout cables from other suppliers may have 4 physical cables When connecting a third party br...

Page 57: ...tion for SFN8x42 The sfboot utility from the Solarflare Linux Utilities package SF 107601 LS is used to configure a Solarflare QSFP adapter for 10G or 40G operation For example sfboot port mode 2x40G...

Page 58: ...e light signal to achieve a ratio that will deliver sufficient signal strength to all endpoints Solarflare adapters do not support a receive only configuration on 1G links 2 16 Solarflare Mezzanine Ad...

Page 59: ...chassis HS22 HS22V and HX5 servers The IBM BladeCenter blade supports a single Solarflare mezzanine adapter 1 The blade should be extracted from the BladeCenter in order to install the mezzanine adap...

Page 60: ...adapter See Figure 5 Figure 5 In position mezzanine adapter 5 Press the port connector gently into the connector block ensuring that the adapter is firmly and correctly seated in the connector block...

Page 61: ...Dual Port SFP are 10GbE Mezzanine Adapters for the DELL PowerEdge C6100 series rack server Each DELL PowerEdge node supports a single Solarflare mezzanine adapter 1 The node should be extracted from...

Page 62: ...PCB card into the slot as shown in Figure 5 top diagram Note that the riser card only fits one way 4 Offer the adapter to the node and ensure it lies underneath the chassis cover 5 Lower the adapter...

Page 63: ...s rack server Each DELL PowerEdge node supports a single Solarflare mezzanine adapter 1 The node should be extracted from the rack server in order to install the mezzanine adapter Refer to the PowerEd...

Page 64: ...er to the Solarflare Enhanced PTP User Guide SF 109110 CD 2 21 Solarflare ApplicationOnload Engine The ApplicationOnload Engine AOE SFA7942Q is a half length full height PCIe form factor adapter combi...

Page 65: ...ed Installation Red Hat Enterprise Linux on page 68 Unattended Installation SUSE Linux Enterprise Server on page 69 Configuring the Solarflare Adapter on page 70 Setting Up VLANs on page 73 Setting Up...

Page 66: ...Solarflare adapters on Red Hat and SUSE Linux distributions Table 14 Linux Feature Set Fault diagnostics Support for comprehensive adapter and cable fault diagnostics and system reports See Linux Util...

Page 67: ...ed RFS See Solarflare Accelerated RFS SARFS on page 84 Transmit Packet Steering XPS Supported on Linux 2 6 38 and later kernels Selects the transmit queue when transmitting on multi queue devices See...

Page 68: ...DKMS is installed To install the Solarflare driver DKMS package execute the following command rpm i sfc dkms version noarch rpm Load the driver modprobe sfc Teaming Improve server reliability and ban...

Page 69: ...e full path to the source RPM 3 To build for a different kernel to the running system enter the following command rpmbuild define kernel kernel version rebuild package_name 4 Install the resulting RPM...

Page 70: ...Enterprise Linux Distributions These instructions cover installation and configuration of the Solarflare network adapter drivers on Red Hat Enterprise Linux Server Refer to Software Driver Support on...

Page 71: ...to yes For SLES 11 edit the last line in etc modprobe d unsupported modules to allow_unsupported_modules 1 2 Install the RPMs rpm ivh kernel module sfc 2 6 5 7 244 smp 2 1 0111 0 sf 1 SLES9 i586 rpm...

Page 72: ...driver sfc version 4 0 this might also be the 4 1 driver 2 To remove the in tree driver and rebuild the initramfs so that the in tree driver does not automatically reload following reboot find lib mo...

Page 73: ...nux unattended installation requires building two drivers A minimal installation Solarflare driver that only provides networking support This driver is used for network access during the installation...

Page 74: ...ended Installations Table 15 below identifies the various stages of an unattended installation process Table 15 Installation Stages In Control Stages of Boot Setup needed BIOS PXE code on the adapter...

Page 75: ...t contains the kickstart file or a network location where the kickstart file can be accessed A Solarflare driver disk Unattended Red Hat Enterprise Linux installations are configured with Kickstart Th...

Page 76: ...RPM mnt bin rpm Uvh mnt filename of Solarflare RPM bin umount mnt 3 10 Unattended Installation SUSE Linux Enterprise Server Unattended SUSE Linux Enterprise Server installations are configured with Au...

Page 77: ...boot_ pxe 3 11 Configuring the Solarflare Adapter Ethtool is a standard Linux tool that you can use to query and change Ethernet adapter settings Ethtool can be downloaded from http sourceforge net pr...

Page 78: ...onfigured to operate at a lower speed 1G The following commands demonstrate ethtool to configure the network adapter Ethernet settings Identify interface configuration settings ethtool ethX Set link s...

Page 79: ...e ethtool g ethX To set the new transmit or receive ring size to value N ethtool G ethX rx N tx N The ring buffer size must be a value between 128 and 4096 On the SFN7000 and SFN8000 series adapters t...

Page 80: ...d VLAN traffic only on Red Hat Enterprise Linux see http kbase redhat com faq docs DOC 8062 To configure mixed VLAN tagged and untagged traffic on Red Hat Enterprise Linux see http kbase redhat com fa...

Page 81: ...twork port can be exposed to the host as multiple PCIe Physical Functions PF with each having a unique interface name and unique MAC address When the Solarflare NET driver sfc ko is loaded in the host...

Page 82: ...re delivered to each subscriber Received unicast packets are delivered to the PF with a matching MAC address Because the TCP IP stack has multiple network interfaces on the same broadcast domain there...

Page 83: ...nt to the OS and driver VLAN tags can be assigned when PFs are enabled using the sfboot command A single PF can be assigned VLAN tag 0 allowing it to receive untagged traffic sfboot switch mode partit...

Page 84: ...24 07 00 0 Ethernet controller Solarflare Communications Device 0903 rev 01 07 00 1 Ethernet controller Solarflare Communications Device 0903 rev 01 07 00 2 Ethernet controller Solarflare Communicatio...

Page 85: ...nd confirmed with rx0 tx0 appearing after the version number in the output from ethtool as shown above The firmware update utility sfupdate and boot ROM configuration tool sfboot are available in the...

Page 86: ...ilable via eth4 00 0F 53 21 00 60 eth13 Interface specific boot options are not available Adapter wide options are available via eth4 00 0F 53 21 00 60 eth14 Interface specific boot options are not av...

Page 87: ...k encap Ethernet HWaddr 00 0F 53 21 00 63 eth14 Link encap Ethernet HWaddr 00 0F 53 21 00 62 eth15 Link encap Ethernet HWaddr 00 0F 53 21 00 67 3 The output from steps 1 and 2 above identifies a serve...

Page 88: ...queue has its own dedicated MSI X interrupt which ideally should be tied to a dedicated CPU core This allows the receive side TCP processing to be distributed amongst the available CPU cores providing...

Page 89: ...of RSS receive queues to a CPU is governed by the receive queue s MSI X interrupt affinity See Interrupt Affinity on page 113 for more details rmmod sfc modprobe sfc NOTE RSS also works for UDP packe...

Page 90: ...ature supported since Linux 2 6 35 that attempts to schedule protocol processing of incoming packets on the same processor as the user thread that will consume the packets Accelerated RFS requires Lin...

Page 91: ...le rps_sock_flow_entries 32768 and there are 16 queues then rps_flow_cnt for each queue will be configured as 2048 echo 2048 sys class net eth3 queues rx 0 rps_flow_cnt echo 2048 sys class net eth3 qu...

Page 92: ...Linux 2 6 38 and later XPS is a mechanism for selecting which transmit queue to use when transmitting a packet on a multi queue device XPS is configured on a per transmit queue basis where a bitmap of...

Page 93: ...to CPU2 and so on Map TX queue to CPU Hyperthreaded cores are included with the associated physical core echo 110011 sys class net eth3 queues tx 0 xps_cpus echo 11001100 sys class net eth3 queues tx...

Page 94: ...M package is supplied as 64bit and 32bit binaries compiled to be compatible with GLIBC versions for all supported distributions The Solarflare utilities RPM file can be downloaded from the following l...

Page 95: ...agent during server startup See Solarflare Boot Manager on page 374 for more information on the Boot Rom agent PXE and UEFI network boot is not supported for Solarflare adapters on IBM System p serve...

Page 96: ...mpletion code to determine the outcome of commands when operating silently l list Lists all available Solarflare adapters This option shows the ifname and MAC address Note this option may not be used...

Page 97: ...adapter dependent SFN8722 2x10G SFN8x42 4x10G 2x40G default SFN8522 M 2x10G default SFN7xx4F 2x10G 4x10G default SFN7xx2Q 2x10G 4x10G 2x40G default SFN7xx2F 1x10G 2x10G default Changes to this settin...

Page 98: ...rts on the NIC rx dc size 8 16 32 64 Specifies the size of the descriptor cache for each receive queue This is for SFN7000 and SFN8000 series adapters only The default is 16 if the port mode supports...

Page 99: ...capabilities of the link partner which may result in an inability to establish the link auto Auto negotiate link speed default 10G 10G bit sec 1G 1G bit sec 100M 100M bit sec linkup delay delay time...

Page 100: ...eries adapters vf count vf count The number of virtual functions VF advertised to the operating system for each Physical Function on this physical network port SFN5000 6000 series adapters support 102...

Page 101: ...Setting pf vlans none will clear all VLAN tags on the port pf vlans should be included after the pf count option on the sfboot command line If the number of PFs is changed then the VLAN tags will be...

Page 102: ...s 2002 2005 eth4 Boot image Option ROM only Link speed Negotiated automatically Link up delay time 7 seconds Banner delay time 3 seconds Boot skip delay time 6 seconds Boot type PXE MSI X interrupt li...

Page 103: ...he general usage for sfupdate is as follows as root sfupdate adapter eth N options where ethN is the interface name ifname of the Solarflare adapter to be upgraded option is one of the command options...

Page 104: ...unbind done 2 Identify the bus device function for all Solarflare interfaces Using ifconfig a will not discover any Solarflare interfaces Use lspci lspci D d 1924 Output similar to the following will...

Page 105: ...ivers sfc bind done Alternatively reload the sfc driver onload_tool reload or modprobe r sfc modprobe sfc 8 Run ifconfig a again to find that all the interfaces are reported and all have been firmware...

Page 106: ...n is the same as or more recent then the firmware embedded in sfupdate backup Backup existing firmware image before updating This option may be used with write and force image filename Update the firm...

Page 107: ...ed to install Solarflare AppFlex licenses and enable selected on board services for Solarflare adapters For more information about license requirements see Solarflare AppFlex Technology Licensing on p...

Page 108: ...arSystem License Inventory Use the combined inventory and keys options to identify the licenses installed on an adapter sfkey adapter eth2 inventory keys eth2 714100101282140148200014 Flareon ONL PTP...

Page 109: ...the keys installed sfc driver reload is required after sfkey installs certain types of license e g a PTP license To reload the sfc driver modprobe r sfc modprode sfc or when Onload is installed onloa...

Page 110: ...tilization Different applications may be more or less affected by improvements in these three metrics For example transactional request response network applications can be very sensitive to latency w...

Page 111: ...igured potentially improving performance Adapter MTU Maximum Transmission Unit The default MTU of 1500 bytes ensures that the adapter is compatible with legacy 10 100Mbps Ethernet endpoints However if...

Page 112: ...mmediately A new moderation interval then starts during which no interrupt is raised Solarflare adapters by default use an adaptive algorithm where the interrupt moderation delay is automatically adju...

Page 113: ...the transmit interrupt moderation interval unless the driver module parameter separate_tx_channels is enabled Normally packet RX and TX completions will share interrupts so RX and TX interrupt modera...

Page 114: ...ecommend that you do not disable this setting TCP Large Receive Offload LRO TCP Large Receive Offload LRO is a feature whereby the adapter coalesces multiple packets received on a TCP connection into...

Page 115: ...only the Solarflare interfaces and re enable LRO sysctl w net ipv4 conf ethX forwarding 0 ethtool K ethX lro on where X is the id of the Solarflare interface Disabling the libvirtd service is a perma...

Page 116: ...t throughput To change buffer settings adjust the tcp_rmem and tcp_wmem using the sysctl command Receive buffering sysctl net ipv4 tcp_rmem min default max Transmit buffering sysctl net ipv4 tcp_wmem...

Page 117: ...le option piobuf_size 3 25 Web Server Driver Optimization Introduction The Solarflare net driver from version 4 4 1 1017 on Solarflare SFN7000 and SFN8000 series adapters includes optimizations aimed...

Page 118: ...MSI X interrupt is created for every core rx_recycle_ring_size The default value for the maximum number of receive buffers to recycle pages for has been changed to 512 and in newer drivers will be fur...

Page 119: ...d to have all interrupts on the NUMA node local to the Solarflare adapter rss_numa local 1 and pin nginx threads to the free CPUs even when these are on the non local node When this is not possible CP...

Page 120: ...isable irqbalance permanently run sbin chkconfig level 12345 irqbalance off To see whether irqbalance is currently running run sbin service irqbalance status To disable irqbalance temporarily run sbin...

Page 121: ...Set all network interface interrupts to a single CPU in this case CPU 0 echo 1 proc irq 123 smp_affinity echo 1 proc irq 131 smp_affinity echo 1 proc irq 139 smp_affinity echo 1 proc irq 147 smp_affi...

Page 122: ...0 Other Considerations PCI Express Lane Configurations The PCI Express PCIe interface used to connect the adapter to the server can function at different speeds and widths This is independent of the p...

Page 123: ...ation CPU Speed Service Most Linux distributions will have the cpuspeed service running by default This service controls the CPU clock speed dynamically according to current processing demand For late...

Page 124: ...the number of memory modules DIMMs to populate in the server For optimal memory bandwidth in the system it is likely that all DIMM slots should be populated all NUMA nodes should have memory installed...

Page 125: ...m Offload Leave at default Enabled TCP Segmentation Offload Leave at default Enabled TCP Large Receive Offload Leave at default Enabled TCP Protocol Tuning Leave at default for 2 6 16 and later kernel...

Page 126: ...probe ioatdma Table 25 Latency Tuning Settings Tuning Parameter How MTU Size Configure to maximum supported by network sbin ifconfig ethX mtu size Interrupt moderation Disable with ethtool C ethX rx u...

Page 127: ...ported speed reads back as x8 and 5GT s or x8 and 8GT s or x8 and Unknown CPU Speed Service cpuspeed Disable with sbin service cpuspeed stop CPU Power Service cpupower Disable with systemctl stop cpup...

Page 128: ...s class net ethX device lro TCP Protocol Tuning Leave at default for 2 6 16 and later kernels For earlier kernels sysctl net core tcp_rmem 4096 87380 524288 sysctl net core tcp_wmem 4096 87380 524288...

Page 129: ...option see Driver Tuning on page 111 tx_push_max_fill 0 Direct copying Disable copying directly from the network stack for transmits by setting the following driver option see Driver Tuning on page 11...

Page 130: ...rsordecreasedtoincrease responsiveness uint 10ms sarfs_sample_rate The frequency at which TCP packets are inspected by the SARFS feature This can be increased on heavily loaded servers to reduce the C...

Page 131: ...e separate channels for TX and RX uint 0 rss_cpus Number of CPUs to use for Receive Side Scaling or packages cores or hyperthreads uint or string empty irq_adapt_enable Enable adaptive interrupt moder...

Page 132: ...ailable to reset stats is to cold reboot power OFF ON the server or reload the firmware image Per port statistics port_ are from the physical adapter port Other statistics are from the specified PCIe...

Page 133: ...than 64 bytes port_tx_64 Number of frames transmitted where the length is exactly 64 bytes port_tx_65_to_127 Number of frames transmitted where the length is between 65 and 127 bytes port_tx_128_to_25...

Page 134: ...eived port_rx_broadcast Number of broadcasted packets received port_rx_lt64 Number of packets received where the length is less than 64 bytes port_rx_64 Number of packets received where the length is...

Page 135: ...hen there are insufficient RX descriptors in the RX queue to allocate to the packet This problem occurs if the receive rate is very high and the network adapter receive cycle process has insufficient...

Page 136: ...or performance for the adapter port_rx_pm_discard_vfifo_full Count of the number of packets dropped because of a lack of main packet memory on the adapter to receive the packet into port_rx_pm_trunc_q...

Page 137: ...bytes Number of unicast bytes received rx_multicast Number of multicast packets received rx_multicast_bytes Number of multicast bytes received rx_broadcast Number of broadcast packets received rx_broa...

Page 138: ...rs tx_tso_packets Number of physical packets produced by TSO tx_pushes Number of times a packet descriptor is pushed to the adapter from the network adapter driver tx_pio_packets Number of packets sen...

Page 139: ...rames truncated because an internal FIFO is full As a packet is received it is fed by the MAC into a 128K FIFO If for any reason the PCI interface cannot keep pace and is unable to empty the FIFO at a...

Page 140: ...ounter is incremented otherwise the bad_syncs counter is incremented If sfptpd is unable to synchronize the clocks at this event the sync_timeout counter is incremented sfptpd will synchronize clocks...

Page 141: ...tion of the values in proc sys kernel printk The netif message level provides additional logging control for a specified interface These message levels are documented in Documentation networking netif...

Page 142: ...e 29 Adapter Diagnostic Tests Diagnostic Test Purpose core nvram Verifies the flash memory board configuration area by parsing and examining checksums core registers Verifies the adapter registers by...

Page 143: ...gth 9 phy cable pairA status 1 phy cable pairB status 1 phy cable pairC status 1 phy cable pairD status 1 Cable length is the estimated length in metres A length value of 65535 indicates length not es...

Page 144: ...mmand Prompt on page 146 Unattended Installation on page 150 Managing Adapters with SAM on page 154 Managing Adapters Remotely with SAM on page 156 Using SAM on page 156 Configuring Network Adapter Pr...

Page 145: ...loads Solarflare adapters support Large Segmentation Offload LSO Receive Segment Coalescing RSC and TCP UDP IP checksum offload for improved adapter performance and reduced CPU processing requirements...

Page 146: ...to Configure Teams and VLANs on page 168 See Sfteam Adapter Teaming and VLAN Tool on page 200 See Teaming and VLANs on page 216 PXE and UEFI booting Solarflare adapters support PXE and UEFI booting en...

Page 147: ...follow the instructions below Solarflare does not recommend installing drivers via Remote Desktop Protocol RDP For example via Terminal Services The drivers install package is named after the Solarfl...

Page 148: ...indows To install the drivers from the command line see Install Drivers and Options From a Windows Command Prompt on page 146 1 Double click the supplied Setup exe to start the Solarflare Driver Packa...

Page 149: ...143 Solarflare Package Installation Procedure on page 143 Repair Remove and Change Drivers and Utilities on page 145 Prerequisites The Solarflare Adapter Manager Utility SAM requires Microsoft NET Fra...

Page 150: ...ge To install this from the command line see Install Drivers and Options From a Windows Command Prompt on page 146 1 Double click the supplied Setup exe The Solarflare Driver Package Setup wizard star...

Page 151: ...Installs Solarflare drivers for Windows The Solarflare drivers are installed by default Install Solarflare command line tools Installs the following Solarflare Windows command line tools sfboot exe B...

Page 152: ...anel Programs Programs and Features select the Solarflare Driver Package then select Uninstall Change or Repair from the menu bar above the program list Install Solarflare Adapter Manager Installs Sol...

Page 153: ...on of the core Solarflare drivers and package utilities The drivers install package is named after the Solarflare document part number e g SF 107785 LS 2_Solarflare_Windows_x64_64 bit_Driver_Package e...

Page 154: ...drivers and utilities rebooting the host to complete the installation as required setup exe Install Quiet Performs a silent installation of the drivers and utilities rebooting the host without prompt...

Page 155: ...installed if either NetworkAdapterManager or CommandLineTools are not installed or are still being installed ADDLOCAL examples Install the package interactively with the default installation options...

Page 156: ...ADDLOCAL NetworkAdapterManager REBOOT Suppress Extract Solarflare Drivers If it is necessary to extract the Solarflare Windows drivers e g before WDS installs this can be done from the Windows command...

Page 157: ...using WDS Windows Deployment Services WDS enables the deployment of Windows over a network from a WDS server avoiding the need to install each operating system directly from a CD or DVD This guide ass...

Page 158: ...lare drivers into the Boot Image Modifying the Boot Image You next need to modify the boot image to include the Solarflare Drivers extracted from the setup package Table 33 identifies drivers required...

Page 159: ...nto the Custom Install Image These are the same Solarflare drivers added to the boot image Preparing the Custom Install Image 1 From WDS locate the install image from the Install Images folder on your...

Page 160: ...ger Part of the Windows Automated Installation Kit to create the WDSClientUnattend xml file To associate your WDSClientUnattend xml file with your modified boot image 1 Copy the WDSClientUnattend xml...

Page 161: ...microsoft com en us library cc749082 28WS 10 29 aspx Performing Unattended Installations http technet microsoft com en us library cc771830 28WS 10 29 aspx How to add network driver to WDS boot image h...

Page 162: ...Ethernet and task offloading settings set up teams and VLANs configure the Boot ROM for PXE or UEFI booting and upgrade the adapter firmware Figure 17 SAM Main Screen Windows Server 2012 SAM s diagno...

Page 163: ...Using SAM Starting SAM There are various ways of starting SAM To manage a local computer do one of the following If the Solarflare notification area icon is installed right click the icon and choose...

Page 164: ...te computer On Windows Server 2008 R2 choose Start All Programs Solarflare Network Adapters Manage network adapters on a remote computer On Windows Server 2012 or later click the Start button followed...

Page 165: ...pter and select Show graphs from the menu By default SAM shows the load transmitted packets and received packets graphs only To view other available graphs Select Graphs from the right click menu or f...

Page 166: ...r the value 2 300 000 Enabled by default This can be useful when dealing with the large Tx Rx numbers that can accumulate with 10Gb networking Note The Transmit and Receive bytes columns ignore this s...

Page 167: ...tures on Solarflare adapters Accessing Adapter Feature Settings on page 160 Checksum Offload on page 163 RSS and Interrupts on page 163 Segmentation Offload Ethernet Link Speed on page 166 Ethernet fl...

Page 168: ...Solarflare Adapters on Windows Issue 20 Solarflare Communications 2017 161 From SAM select an adapter and from the Action menu select Configure Offload tasks Ethernet and other features The Adapter F...

Page 169: ...y or OK when changes to Adapter Features are modified Note that the Receive legend in the Segmentation Offload field differs depending on the version of Windows that is installed for Windows Server 20...

Page 170: ...SS Receive Side Scaling and interrupt moderation Both are enabled by default and can significantly improve the performance of the host CPU when handling large amounts of network data RSS attempts to d...

Page 171: ...A node but there is no dynamic load balancing NUMA Scaling CPUs are assigned on a round robin basis across every NUMA node NUMA Scaling Static As for NUMA Scaling but without dynamic load balancing Co...

Page 172: ...received packets from one interface to another then Windows will automatically disable RSC RSC is enabled by default Large Receive Offload LRO is a Solarflare proprietary mechanism similar to RSC It...

Page 173: ...ach other when they are being overloaded by received data This prevents one device from overwhelming the other device with network packets For instance when a switch is unable to keep up with forwardi...

Page 174: ...spread across the specified cores however the static configuration does not provide best performance when the network load varies over time Dynamic VMQ supported in Windows Server 2012 and later will...

Page 175: ...lerant teams Teaming allows the user to configure teams consisting of all Solarflare adapter ports on all installed Solarflare adapters or might consist of only selected adapter ports e g from a dual...

Page 176: ...am The Solarflare Create a team Wizard starts 4 Team Create Wizard 5 The wizard will guide you through the process of creating a team and optionally adding VLANs to your team see Table 40 on page 175...

Page 177: ...n the server Solarflare recommend only changing network settings when disruption to the services can be tolerated 1 Start SAM and from the Network Adapter list select the team you want to reconfigure...

Page 178: ...the key button Note Before physically removing an adapter from a server first check it is not the key adapter You must reassign the key adapter if you want to remove it from the team to avoid duplica...

Page 179: ...tions list click Add one or more adapters or choose Actions Add one or more adapters The Available Network Adapters dialog box is displayed To specify a new active adapter For Fault Tolerant Teams onl...

Page 180: ...onfiguration settings and become available on the server once again Any VLANs set up for the team will be deleted when the team is deleted To delete a team NOTE Changing team settings can disrupt curr...

Page 181: ...anager through which the operating system is able to receive data tagged with the correct VLAN ID VID You may assign one VLAN to accept VLAN 0 or untagged traffic which allows the interface to communi...

Page 182: ...when remotely administering the server Supports the handling of priority traffic Enables the handling of traffic that is tagged as priority Supports untagged and VLAN 0 traffic Restricts the VLAN to h...

Page 183: ...2 In the Network adapter list select the VLAN to delete If necessary expand the team if the VLAN is attached to a team then select the VLAN 3 From the Actions list click Delete this VLAN or choose Ac...

Page 184: ...nstalling of the adapter drivers If necessary you can reset the adapter statistics to restart the accumulated data values at any time 1 Start SAM and select a Solarflare adapter from the Network Adapt...

Page 185: ...e selected by default See Table 41 for a description of the tests that are available 4 To stop as soon as a failure is detected select Stop on first test failure 5 To run all the tests more than once...

Page 186: ...its in a selection of registers Interrupts Examines the available hardware interrupts by requesting the controller to generate an interrupt and verifying that the interrupt has been processed by the n...

Page 187: ...ce more detailed results but at the expense of disrupting the connection while tests are running CAUTION The offline tests will cause the network link to momentarily drop and disrupt data flow Solarfl...

Page 188: ...ager To configure PXE or UEFI booting on the Solarflare Boot ROM 1 Start SAM and select a Solarflare adapter from the Network Adapter list From the Action menu select the Configure Boot ROM option The...

Page 189: ...m the Link tab change the Link Speed option depending on your link requirement Note that Auto negotiated is correct for most links and should not be changed unless advised The Link Speed options will...

Page 190: ...K or Apply to save your settings to the Boot ROM 4 17 Managing Firmware with SAM SAM allows you to monitor the firmware PHY Boot ROM and Adapter for your Solarflare adapters Either select Manage firmw...

Page 191: ...not be reflected in SAM until you close the Advanced Properties page To configure network adapter properties 1 From the Control Panel select System 2 Select Device Manager from the left hand menu 3 Ex...

Page 192: ...hat it is overloaded such as when a device is receiving data faster than it can process it The adapter does this by generating a pause frame to request the sending device to temporarily stop transmitt...

Page 193: ...end Offload Version 2 IPv4 and IPv6 Enabled Disabled Large Send Offload LSO is an offload technology for reducing the load on a CPU by processing TCP segmentation for transmitted packets in the adapte...

Page 194: ...ter driver Default is 8 Preferred Numa Node All 0 to 15 The adapter attempts to use only the CPUs from the specified NUMA node for RSS If this is set to All or is greater than or equal to the number o...

Page 195: ...abled Rx Tx Enabled Rx Enabled Tx Enabled UDP checksum offload is a hardware offload technology for reducing the load on a CPU by processing UDP checksums in the adapter hardware Default setting Rx Tx...

Page 196: ...ws console to run Solarflare command line tools Table 44 List Available Command Line Utilities Utility Description sfboot exe A tool for configuring Solarflare adapter Boot ROM options including PXE a...

Page 197: ...o using Ctrl B to access the Boot Rom agent during server startup See Solarflare Boot Manager on page 374 for more information on the Boot Rom agent Sfboot Command Usage 1 Login with an administrator...

Page 198: ...ename Logs output to the specified file in the current folder or an existing folder Specify Silent to suppress simultaneous output to screen if required Computer ComputerName Performs the operation on...

Page 199: ...tors available after using any splitter cables The usable values are adapter dependent SFN8722 2x10G SFN8x42 4x10G 2x40G default SFN8522 M 2x10G default SFN7xx4F 2x10G 4x10G default SFN7xx2Q 2x10G 4x1...

Page 200: ...cache for each receive queue This is for SFN7000 and SFN8000 series adapters only The default is 16 if the port mode supports the maximum number of connectors for the adapter 32 if the port mode suppo...

Page 201: ...e capabilities of the link partner which may result in an inability to establish the link auto Auto negotiate link speed default 10G 10G bit sec 1G 1G bit sec 100M 100M bit sec linkup delay delay time...

Page 202: ...00 series adapters vf count vf count The number of virtual functions VF advertised to the operating system for each Physical Function on this physical network port SFN7000 and SFN8000 series adapters...

Page 203: ...h PF in the range 0 4094 see sfboot help for details Setting pf vlans none will clear all VLAN tags on the port pf vlans should be included after the pf count option on the sfboot command line If the...

Page 204: ...Solarflare adapters installed on the localhost sfboot List Sample console output Solarflare boot ROM configuration utility v4 1 4 Copyright Solarflare Communications 2006 2014 Level 5 Networks 2002 2...

Page 205: ...owing syntax sfupdate Adapter Identifier options where Identifier is the name or ID of the adapter that you want to manage Specifying the adapter is optional if it is not included the command is appli...

Page 206: ...required Adapter Identifier Performs the action on the identified Solarflare network adapter The identifier can be the adapter ID number name or MAC address Force Forces a firmware update Can be used...

Page 207: ...le output Solarflare firmware update utility v4 1 4 Copyright Solarflare Communications 2006 2014 Level 5 Networks 2002 2005 1 Solarflare SFN7122F SFP Server Adapter MAC address 00 0F 53 21 9B B1 Firm...

Page 208: ...variable SFTOOLS to set the path to the Solarflare tools For example SET PATH PATH SFTOOLS or refer to sfteam as SFTOOLS sfteam Sfteam Command Line Options Table 49 lists the command line options sfte...

Page 209: ...created sfteam does not allow you to change its adapters VLANs or team name Either delete the team and set it up again or use SAM instead to configure the team Configure TeamIdentifier Configures the...

Page 210: ...g priority name DHCP to be explicit Formats tag e g 2 assumes no priority tag priority e g 2 p tag priority name e g 2 p my name tag priority name DHCP e g 2 p my name DHCP tag priority name addr mask...

Page 211: ...s either dynamic or static The options are auto default faulttolerant bandwidth key adapter See Teaming and VLANs on page 216 for an explanation of the different teaming modes Distribution Distributio...

Page 212: ...dapter 2 with VLAN tag 4 and priority traffic handling enabled sfteam Create Adapter 2 Vlan 4 P Sample output Solarflare teaming configuration utility v4 1 4 Copyright Solarflare Communications 2006 2...

Page 213: ...ne options are case insensitive and may be abbreviated NOTE Abbreviations in scripts should be avoided since future updates to the application may render your abbreviated scripts invalid Table 50 Sfca...

Page 214: ...9m 2 Solarflare SFN5121T 10GBASE T Server Adapter 2 MAC address 00 0F 53 01 40 8D Link state Up Link speed 10 Gbps Pair 1 OK length 9m Pair 2 OK length 9m Pair 3 OK length 9m Pair 4 OK length 9m Adapt...

Page 215: ...Command Prompt window enter the following command sfkey Adapter Identifier options where Identifier is the name or ID of the adapter that you want to manage Specifying the adapter is optional if it i...

Page 216: ...number name and MAC address Backup Filename Output a report of the installed keys in all adapters The report can be saved to file and later used with the Install option or can be output to stdout by...

Page 217: ...fkey Inventory Keys Solarflare Key Management Application v4 8 2 Copyright Solarflare Communications 2006 2016 Level 5 Networks 2002 2005 42 43 000F53430560 8xxx Onload Precision Time TCP Direct Low L...

Page 218: ...e tools instead 3 In the Command Prompt window enter your command using the following syntax sfnet Adapter Identifier options where Identifier is the name or ID of the adapter that you want to manage...

Page 219: ...he operation on the identified remote host Administrator rights on the remote host computer is required Adapter Identifier Perform the action on the identified Solarflare physical or virtual network a...

Page 220: ...ue is given it must formated as group number where group is in the range 0 9 and number in the range 0 to 63 rssmaxprocessor group number The maximum number of processors available for RSS If a value...

Page 221: ...bled disabled Allows the adapter to vary interrupt moderation automatically if interruptmoderation is enabled wake enabled disabled Specify whether Wake on LAN is enabled sleep enabled disabled Specif...

Page 222: ...apters were not able to be added 32 Application failed initialization 33 Access denied Either the remote host refused a connection on the basis of account privileges or a file could not be opened 34 C...

Page 223: ...the same team This section is only relevant to teams of Solarflare adapters Solarflare adapters can be used in multi vendor teams when teamed using another vendor s teaming driver NOTE Windows Server...

Page 224: ...nected to the same switch In case of link failover traffic on the failed link will be re distributed to the remaining links Link aggregation offers the following functionality Teams can be built from...

Page 225: ...of a link aggregation until a link within the aggregation fails VLANs are supported within 802 1AX teams In the event of failover the load on the failed link is redistributed over the remaining links...

Page 226: ...mode that requires manual configuring of the ports at both ends of the link Static link aggregation is protocol independent and typically inter operates with common link aggregation schemes such as In...

Page 227: ...ch It can also be used with each network link connected to separate switches A fault tolerant team is a set of one or more network adapters bound together by the adapter driver A fault tolerant team i...

Page 228: ...n Figure 39 Fault Tolerant Team Failover The teaming driver monitors the state of the active adapter and in the event that its physical link is lost down or that it fails in service swaps to one of th...

Page 229: ...7 222 Note that in this example Adapter 3 is now active The order in which the adapters are used is determined by a number of factors including user definable rank VLANs VLANs offer a method of dividi...

Page 230: ...on Windows Issue 20 Solarflare Communications 2017 223 VLANs and Teaming VLANs are supported on all Solarflare adapter teaming configurations VLANs with Fault Tolerant Teams Figure 42 shows a fault to...

Page 231: ...gardless of the number of VLANs as show in Figure 43 Figure 43 Failover in Fault Tolerant Team VLAN VLANs with Dynamic or Static Link Aggregation Teams VLANs work in the same way with either Dynamic o...

Page 232: ...of link failure all traffic is distributed over the remaining links as in Figure 45 Figure 45 VLAN with Failed Dynamic or Static Team Link Key Adapter Every team must have a key adapter Figure 46 show...

Page 233: ...e Receive Side Scaling RSS on page 233 and provides the MAC Address that will be used for all traffic sent and received by the team When a link failure occurs in the active adapter for example the phy...

Page 234: ...tions 2017 227 Dynamic and Static Link Aggregation Teams The assignment of key adapters is supported in both dynamic and static link aggregated teams and works in the same way for both Any link failur...

Page 235: ...apter driver settings that affect the performance metrics described This section covers the tuning of all Solarflare adapters Latency will be affected by the type of physical medium used 10GBase T twi...

Page 236: ...rver 2008 R2 Supported on Windows Server 2012 Windows Server 2012 R2 Adaptive Interrupt Moderation Yes Yes Interrupt Moderation Yes Yes Interrupt Moderation Time Yes Yes Large Receive Offload IPv4 Yes...

Page 237: ...es 802 1Q tag control information and 2 bytes EtherType The default maximum frame size is therefore 1518 bytes Since the maximum frame size should ideally be matched across all endpoints in the same L...

Page 238: ...duce peak throughput Turning off interrupt moderation will generate the most interrupts give the highest CPU utilization give the lowest latency give the biggest reduction in peak throughput For many...

Page 239: ...ling LSO will reduce CPU utilization on the transmit side of a TCP connection and improve peak throughput if the CPU is fully utilized Since LSO has no effect on latency it can be enabled at all times...

Page 240: ...use LRO instead The Solarflare network adapter driver disables LRO by default NOTE LRO should NOT be enabled when using the host to forward packets from one interface to another For example if the ho...

Page 241: ...increased to spread the load over more cores This will also increase the amount of receive buffering due to a larger number of RX queues NOTE If hyper threading is enabled RSS will only select one thr...

Page 242: ...e AutoSize Id Name PreferredNumaNode On Windows Server 2012 and later you can instead run the PowerShell Get NetAdapterRss cmdlet and look at lines starting RssProcessorArray Group Number NUMA Distanc...

Page 243: ...rRss Other Considerations PCI Express Lane Configurations The PCI Express PCIe interface used to connect the adapter to the server can function at different speeds and widths This is independent of th...

Page 244: ...when selecting the number of memory modules DIMMs to populate in the server For optimal memory bandwidth in the system it is likely that all DIMM slots should be populated all NUMA nodes should have m...

Page 245: ...t hyper threading is generally beneficial on systems fitted with Core i5 Core i7 and Xeon Nehalem or later CPUs TCP IP Options TCP timestamps window scaling and selective acknowledgments are enabled b...

Page 246: ...ce idle policy to favor performance over power savings Disables PCIe ASPM Active State Power Management The installer does not enable this by default as on certain systems it might significantly incre...

Page 247: ...or any third party Firewall should be disabled with caution The network administrator should be consulted before making any changes Disable the Windows Firewall a Set the Startup Type of the Windows F...

Page 248: ...nterface 2 On client run ntttcps test ntttcps exe rb 500000 a 24 n 100000 l 524288 m 1 1 server_adapter_IP_interface C ntttcps exe rb 500000 a 24 n 100000 l 524288 m 1 1 server adapter IP interface Co...

Page 249: ...Receive Offloads Enable in Network Adapter Advanced Properties Large Send Offloads Leave at default Enabled Max Frame Size Configure to maximum supported by network in Network Adapter s Advanced Prope...

Page 250: ...Max Frame Size Configure to maximum supported by network in Network Adapter s Advanced Properties Memory bandwidth Ensure memory utilizes all memory channels on system motherboard Offload Checksums L...

Page 251: ...al 0xA0000004L The driver can t add a device due to the system being started in safe mode SAFEMODE_MINIMAL Warning 0xA0000005L The driver could not allocate memory on a specific NUMA node Warning For...

Page 252: ...g The adapter or server maybe faulty 0xA0010017L Hardware MCDI communication suffered an error Warning None required 0xA0010019L Hardware MCDI boot from non primary flash Possible flash corruption War...

Page 253: ...e has reported an error condition Error 0xE0010018L Hardware MCDI communication timed out Error None required Table 60 NDIS Driver Status Codes Value Description Severity Notes 0x60030001L The driver...

Page 254: ...pecified in the registry is out of range Using default value Warning 0xA0010007L The parameter IfType is missing from the registry Using default value Warning 0xA0010008L The parameter IfType specifie...

Page 255: ...missing from the registry Using default value Warning 0xA0010010L The parameter LROIPv4 specified in the registry is out of range Using default value Warning 0xA0010011L Theparameter LSOv2IPv4 is mis...

Page 256: ...ecified in the registry is invalid Using default value Warning 0xA0010019L The parameter PhysicalMediaType is missing from the registry Using default value Warning 0xA001001AL The parameter PhysicalMe...

Page 257: ...L The parameter SpeedDuplex is missing from the registry Using default value Warning 0xA0010022L The parameter SpeedDuplex specified in the registry is out of range Using default value Warning 0xA0010...

Page 258: ...er VlanId specified in the registry is out of range Using default value Warning 0xA001002BL The parameter WarnLevel is missing from the registry Using default value Warning 0xA001002CL The parameter W...

Page 259: ...fault value Warning 0xA0010033L The parameter UDPChecksum OffloadIPv4ismissingfrom the registry Using default value Warning 0xA0010034L The parameter UDPChecksum OffloadIPv4 specified in the registry...

Page 260: ...t value Warning 0xA001003BL The parameter TCPChecksum OffloadIPv6ismissingfrom the registry Using default value Warning 0xA001003CL The parameter TCPChecksumOffload IPv6 specified in the registry is o...

Page 261: ...leepOn Disconnect is missing from the registry Using default value Warning 0xA0010044L The parameter DeviceSleepOn Disconnect specified in the registry is out of range Using default value Warning 0xA0...

Page 262: ...f range Using default value Warning 0xA001004DL The parameter FlowControlAuto Negotiation is missing from the registry Using default value Warning 0xA001004EL The parameter FlowControlAuto Negotiation...

Page 263: ...issing from the registry Using default value Warning 0xA0010056L The parameter VMQLookaheadSplit specified in the registry is out of range Using default value Warning 0xA0010057L The parameter VMQVlan...

Page 264: ...specified in the registry is out of range Using default value Warning 0xA001005FL The parameter RssProfile is missing from the registry Using default value Warning 0xA0010060L The parameter RssProfile...

Page 265: ...parameter RssMaxProcNumber is missing from the registry Using default value Warning 0xA0010068L The parameter RssMaxProcNumber specified in the registry is out of range Using default value Warning 0xA...

Page 266: ...he LSOv2IPv4 and LSOv2IPv6 registry parameters Warning 0xA0020003L The driver has disabled Large Receive Offload overriding the LROIPv4 registry parameter as it is incompatible with IEEE 802 1p 802 1Q...

Page 267: ...support TCP Connection Offload and therefore has overridden the TCPConnection OffloadIPv4 registry parameter Warning 0xA0020009L ThedriverhasdisabledTCP Connection Offload overriding the TCPConnectio...

Page 268: ...t support IPv6 checksum offloads and therefore has overridden the TCPChksumOffv6 and UDPChksumOffv6 registry parameters Warning 0xA002000DL The driver has determined that the network adapter does not...

Page 269: ...upport will be unavailable whilst IPv6 TCP transmit checksum offload is disabled Warning 0xA0020012L The driver has determined that the network adapter does not support Virtual Machine Queues and ther...

Page 270: ...cessary for operation Error 0xE0000002L The driver has determined thatthenetworkadapteris not functioning properly Error 0xE0000003L The driver has encountered an internal error and has failed Error 0...

Page 271: ...em Requirements on page 264 VMware Feature Set on page 265 Installing Solarflare Drivers and Utilities on VMware on page 266 Configuring Teams on page 267 Configuring VLANs on page 268 Running Adapter...

Page 272: ...ter performance and reduced CPU processing requirements See TCP IP Checksum Offload on page 285 See TCP Segmentation Offload TSO on page 285 NetQueue Support for NetQueue a performance technology that...

Page 273: ...their configurations Installing on VMware ESX 5 0 ESXi 5 1 5 5 and 6 0 To install or update the VIB through the CLI esxcli software vib install v absolute PATH to the vib To install or update the off...

Page 274: ...on type and click Next 7 Choose Create a Virtual Switch or Use vSwitchX as desired 8 Follow the remaining on screen instructions 5 4 Configuring Teams A team allows two or more network adapters to be...

Page 275: ...tch must have an appropriate VLAN ID specified NOTE VMware recommend that you create or amend VLAN details from the physical console of the server not via the Infrastructure Client to prevent potentia...

Page 276: ...nvram Verifies the flash memory board configuration area by parsing and examining checksums core registers Verifies the adapter registers by attempting to modify the writable bits in a selection of r...

Page 277: ...Wrote line at the end of the build process identifies the location of the binary RPM Wrote root rpmbuild RPMS x86_64 sfutils version x86_64 rpm 3 Identify where the binary tools would be installed bu...

Page 278: ...ng the Boot ROM with Sfboot Sfboot Command Usage on page 271 Sfboot Command Line Options on page 272 Sfboot Examples on page 278 Sfboot is a command line utility for configuring Solarflare adapter Boo...

Page 279: ...iases silent Suppresses all output including warnings and errors no user interaction You should query the completion code to determine the outcome of commands when operating silently see Performance T...

Page 280: ...s Table 63 Sfboot Options Option Description Table 64 Sfboot Global Parameters Parameter Description boot image all optionrom uefi disabled Specifies which boot firmware images are served up to the BI...

Page 281: ...security on non privileged functions This is for SFN7000 and SFN8000 series adapters only This reduces security in virtualized environments The default is disabled When enabled a function PF or VF ca...

Page 282: ...rts a reduced number of connectors change mac default enabled disabled This is for SFN7000 and SFN8000 series adapters only Change the unicast MAC address for non privileged functions on this port Thi...

Page 283: ...apabilities of the link partner which may result in an inability to establish the link auto Auto negotiate link speed default 10G 10G bit sec 1G 1G bit sec 100M 100M bit sec linkup delay delay time in...

Page 284: ...0 series adapters vf count vf count The number of virtual functions VF advertised to the operating system for each Physical Function on this physical network port SFN7000 and SFN8000 series adapters h...

Page 285: ...pf vlans none will clear all VLAN tags on the port pf vlans should be included after the pf count option on the sfboot command line If the number of PFs is changed then the VLAN tags will be cleared...

Page 286: ...ice Console as root and enter the following command sfupdate adapter vmnicX options where vmnicX is the interface name of the Solarflare adapter you want to upgrade Specifying the adapter is optional...

Page 287: ...pdate To use an external image specify image filename in the command write fails if the embedded image is the same or a previous version to that in the adapter To force a write in this case specify fo...

Page 288: ...The PHY firmware is up to date 5 10 Performance Tuning on VMware Introduction on page 281 Tuning Settings on page 282 Other Considerations on page 286 Introduction The Solarflare family of network ad...

Page 289: ...virtual adapter of type vmxnet To check that VMware Tools are installed 1 From the VMware Infrastructure Client power on the virtual machine and click the Summary tab 2 In the General panel check the...

Page 290: ...pt vector in hex and CPU is the CPU number in decimal To determine the value for IRQVEC enter cat proc vmware interrupts Locate the interrupts associated with the Solarflare adapter e g vmnic2 Interru...

Page 291: ...ver software adapts the interrupt moderation setting according to the traffic and workloads it sees Alternatively you can set the moderation interval manually You would normally only do this if you ar...

Page 292: ...olarflare recommend you do not disable checksum offload For advice on configuring checksum offload in the guest consult the relevant Solarflare section for that guest or the documentation for the gues...

Page 293: ...with the number of available CPU cores RSS requires a platform that supports MSI X interrupts RSS is enabled by default When RSS is enabled the controller uses multiple receive queues into which to de...

Page 294: ...ould be taken into account when selecting the number of DIMMs to populate in the server Consult your motherboard documentation for details Intel QuickData Intel QuickData Technology allows VMware ESX...

Page 295: ...7 288 Server Motherboard Server BIOS Chipset Drivers Tuning or enabling other system capabilities may further enhance adapter performance Readers should consult their server user guide Possible opport...

Page 296: ...s on page 295 FreeBSD Utilities Package on page 296 Configuring the Boot ROM with sfboot on page 297 Upgrading Adapter Firmware with sfupdate on page 303 Performance Tuning on FreeBSD on page 305 Modu...

Page 297: ...eBSD Feature Set Jumbo frames Support for MTUs Maximum Transmission Units to 9000 bytes See Configuring Jumbo Frames on page 295 Task offloads Support for TCP Segmentation Offload TSO Large Receive Of...

Page 298: ...add or pkg_add The following instructions assume that the source package has been downloaded to the tmp directory 1 Ensure you are the root user If not su 2 To avoid using the previous driver that is...

Page 299: ...Ensure that DHCP is available with PXE boot options Ensure that a TFTP server is available Ensure that a FreeBSD server is available This is required only to generate the FreeBSD PXE boot image Downlo...

Page 300: ...e for an unattended installation Table 68 Installation Stages In Control Stages of Boot Setup needed BIOS PXE code on the adapter runs Adapter must be in PXE boot mode See Solarflare Boot Manager on p...

Page 301: ...tings permanent create entries in the configuration file as described below Using IPv6 To configure using IPv6 create an IPv6 interface sfxge n interface with a link local IPv6 address by entering ifc...

Page 302: ...fconfig sfxge n sfxge0 flags 8843 UP BROADCAST RUNNING SIMPLEX MULTICAST metric 0 mtu 1500 If you want to have an MTU configured when the interface is brought up add an mtu parameter to the single lin...

Page 303: ...cd source package dir 3 Build and install a binary package from the source make install Alternatively to build and install in separate steps a Build a binary package from the source make package b In...

Page 304: ...e target host The format for the parameters are parameter value Sfboot Command Line Options Table 70 lists the options for sfboot Table 71 lists the available global parameters and Table 72 lists the...

Page 305: ...t values If an adapter is specified options for the given adapter are reset but global options shown in Table 71 are not reset Note that clear can also be used with parameters allowing you to reset to...

Page 306: ...ilter security on non privileged functions This is for SFN7000 and SFN8000 series adapters only This reduces security in virtualized environments The default is disabled When enabled a function PF or...

Page 307: ...meout timeout in nanoseconds Specifies the timeout in nanoseconds for RX event merging A timeout of 0 means that event merging is disabled Table 71 Sfboot Global Parameters Parameter Description Table...

Page 308: ...apter booting delay time in seconds 0 256 boot type pxe uefi disabled Sets the adapter boot type effective on next boot pxe PXE Preboot eXecution Environment booting disabled Disable adapter booting p...

Page 309: ...greater than zero Changes to this setting with sfboot require a cold reboot to become effective vf msix limit 1 2 4 8 16 32 64 128 256 The maximum number of interrupts a virtual function may use pf vl...

Page 310: ...rading Adapter Firmware with sfupdate To Update Adapter Firmware Reinstall the sfutils package as described in FreeBSD Utilities Package on page 296 Sfupdate Command Usage The general usage for sfupda...

Page 311: ...or scripts V version Display version information and exit i adapter sfxge n Specifies the target adapter when more than one adapter is installed in the machine sfxge n Adapter ifname or MAC address as...

Page 312: ...page 306 Other Considerations on page 311 Introduction The Solarflare family of network adapters are designed for high performance network applications The adapter driver is pre configured with defau...

Page 313: ...pter MTU Maximum Transmission Unit The default MTU of 1500 bytes ensures that the adapter is compatible with legacy 10 100Mbps Ethernet endpoints However if a larger MTU is used adapter throughput and...

Page 314: ...val then starts during which no interrupt is raised Interrupt moderation settings are critical for tuning adapter latency Increasing the interrupt moderation interval will generate less interrupts red...

Page 315: ...Checksum offload is changed dynamically using ifconfig with the following parameters rxcsum txcsum rxcsum6 txcsum6 Enable Rx and Tx checksum offload for IPv4 and IPv6 rxcsum txcsum rxcsum6 txcsum6 Dis...

Page 316: ...config_sfxge n inet IPv4 address netmask netmask tso TCP and IP checksum offloads must be enabled for TSO to work NOTE Solarflare recommend that you do not disable this setting TCP Large Receive Offlo...

Page 317: ...e size all influence the affect of the max buffer size values The minimum and default values can be left at their defaults minimum 4096 and default 87380 See https wiki freebsd org NetworkPerformanceT...

Page 318: ...widths This is independent of the physical slot size used to connect the adapter The possible widths are multiples x1 x2 x4 x8 and x16 lanes of 2 5Gbps for PCIe Gen 1 5 0 Gbps for PCIe Gen 2 and 8 0G...

Page 319: ...e channels to access main system memory Maximum memory performance is only achieved when the chipset can make use of all channels simultaneously This should be taken into account when selecting the nu...

Page 320: ...t default Enabled TCP Protocol Tuning Leave at default Receive Side Scaling RSS Application dependent Buffer Allocation Method Leave at default Some applications may benefit from specific setting PCI...

Page 321: ...nt speed not the supported speed reads back as x8 and 5GT s or x8 and 8GT s or x8 and Unknown CPU Power Management Disable with etc rc d powerd stop Memory bandwidth Ensure memory utilizes all memory...

Page 322: ...bandwidth Ensure memory utilizes all memory channels on system motherboard Table 76 Forwarding Tuning Settings Tuning Parameter How Table 77 Driver Module Parameters Parameter Description Possible Va...

Page 323: ...w counter grows uint 65536 tx_dpl_get_non_tcp_max Maximum number of non TCP packets queued in the software get list for a transmit queue This parameter can restrict utilizing the queue for non TCP e g...

Page 324: ...cified PCIe function NOTE sysctl dev sfxge n stats output depends on the features supported by the adapter type tso_fw_assisted Whether to assist TSO using the firmware Applicable to SFN7000 and SFN80...

Page 325: ...following mismatched unicast address and unicast promiscuous mode is not enabled packet is a pause frame packet has length discrepancy internal FIFO overflow condition length 60 bytes ev_rx_pause_frm...

Page 326: ...ber of transmitted packets dropped because of FIFO overrun ev_tx_pkt_err Number of transmitted packets dropped because of driver error ev_tx_pkt_too_big Number of transmitted packets dropped because o...

Page 327: ...many streams lro_new_stream Number of distinct streams the driver has tracked lro_drop_idle Number of streams discarded because they went idle lro_drop_closed Number of streams that have seen a FIN o...

Page 328: ...max in Table 77 on page 315 tx_netdown_drops Number of packets early dropped by the driver because of link is down tso_pdrop_too_many Number of TSO packets partially dropped by the driver because TSO...

Page 329: ...s 1522 with VLAN tag rx_ge_15xx_pkts Number of packets received where the length is between 1518 bytes 1522 with VLAN tag and 9000 bytes rx_errors Number of packets received with errors rx_fcs_errors...

Page 330: ...o refill the queue with new descriptors A number of different steps can be tried to resolve this issue Disable the irqbalance daemon in the OS Distribute the traffic load across the available CPU core...

Page 331: ...ngth is between 512 and 1023 bytes tx_1024_to_15xx_pkts Number of frames transmitted where the length is between 1024 and 1518 bytes 1522 with VLAN tag tx_ge_15xx_pkts Number of frames transmitted whe...

Page 332: ...hey are periodically updated from the port and driver statistics See Table 81 Table 81 Netstat statistics Field Value Ipkts rx_pkts Ierrs rx_errors Idrop 0 Ibytes rx_octets Opkts tx_pkts Oerrs tx_erro...

Page 333: ...number of features to support SR IOV These features can be summarized as follows PCIe Virtual Functions VF A PCIe physical function PF can support a configurable number of PCIe virtual functions In t...

Page 334: ...ethtool as shown above The firmware update utility sfupdate and boot ROM configuration tool sfboot are available in the Solarflare Linux Utilities package SF 107601 LS issue 28 or later Platform suppo...

Page 335: ...unsafe_assigned_interrupts 1 RHEL 7 options vfio_iommu_type1 allow_unsafe_assigned_interrupts 1 Alternative Routing ID Interpretation ARI The ARI extension to the PCI Express Base Specification extend...

Page 336: ...p delay time 5 seconds Boot type Disabled Physical Functions per port 1 MSI X interrupt limit 32 Number of Virtual Functions 2 VF MSI X interrupt limit 8 Firmware variant full feature virtualization I...

Page 337: ...for a PF must be a power of 2 switch mode mode default Specifies the mode of operation that the port will be used in default single PF created zero VFs created sriov SR IOV enabled single PF created...

Page 338: ...e the virtual adapters and virtual ports as required Adapter firmware will also configure the firmware switching functions allowing packets to pass between PF and VFs or from VF to VF Figure 49 SR IOV...

Page 339: ...ver kernel command line Refer to Platform support SR IOV on page 327 1 The example configures 1 PF per port default 2 VFs per PF sfboot switch mode sriov pf count 1 vf count 2 Solarflare boot configur...

Page 340: ...ware addresses MAC addresses assigned to VFs in the above example output have been randomly generated by the host MAC addresses visible to the host will be replaced by libvirt generated MAC addresses...

Page 341: ...count 1 The sfboot settings shown above are the default shipping state settings for the SFN7000 series adapter A cold reboot of the server is only required when changes are made using sfboot 3 Create...

Page 342: ...6 errors 0 dropped 0 overruns 0 carrier 0 collisions 0 txqueuelen 0 RX bytes 55760 54 4 KiB TX bytes 468 468 0 b 8 Define a network in an XML file i e host network xml network name host network name...

Page 343: ...address 52 54 00 96 0a 8a 3 source bridge br1 4 model type virtio 5 address type pci domain 0x0000 bus 0x00 slot 0x07 function 0x0 6 interface 1 Interface type must be specified by the user as bridge...

Page 344: ...ion and increasing performance When using macvtap there is no link state propagation to the guest which is unable to identify if a physical link is up or down Macvtap does not currently forward multic...

Page 345: ...start the GUI interface A VM can also be created from an existing VM XML file The following procedure assumes the VM is created The example procedure will create an interface configuration file and c...

Page 346: ...n 0x0 6 interface 1 Interface type must be specified by the user as direct 2 The MAC address If not specified by the user this will be automatically assigned a random MAC address by libvirt 3 The sour...

Page 347: ...F network interface and is not visible to the host OS Each macvtap instance builds over a different network interface so there is no implicit macvtap bridge Macvtap does not currently forward multicas...

Page 348: ...flare Communications Device 1903 0000 03 00 4 Ethernet controller Solarflare Communications Device 1903 0000 03 00 5 Ethernet controller Solarflare Communications Device 1903 VFs will also be listed u...

Page 349: ...type component is required for each VF 9 Restart the VM after editing the XML file virsh start vmname The passed through VF interface is visible when viewed from the GUI Virtual Machine Manager Figur...

Page 350: ...automatically assigned by libvirt when the guest is started 5 The VF PCIe address as known by the guest will be added automatically by libvirt For further information about the direct passthrough conf...

Page 351: ...the Ethernet interfaces and PCIe addresses VFs shown below in bold text lspci D d1924 0000 03 00 0 Ethernet controller Solarflare Communications SFC9120 rev 01 0000 03 00 1 Ethernet controller Solarfl...

Page 352: ...utomatically reattached to the host after the guest exits If managed no the user must call virNodeDeviceDetach or use the command virsh nodedev detach before starting the guest or hot plugging the dev...

Page 353: ...o a different Virtual Machine VM Each VM supports a TCP IP stack and Solarflare adapter driver sfc ko This mode allows switching between PFs via the Layer 2 switch function configured in firmware Figu...

Page 354: ...sical Functions per port 4 MSI X interrupt limit 32 Number of Virtual Functions 0 VF MSI X interrupt limit 8 Firmware variant full feature virtualization Insecure filters Disabled VLAN tags None Switc...

Page 355: ...l via sysfs the Solarflare net driver module option max_vfs can be used to enable VFs The max_vfs value applies to all adapters and can be set to a single integer i e all adapter physical functions wi...

Page 356: ...e driver after setting changing this value onload_tool reload 7 6 Feature Summary Table 83 Feature Summary Default SRIOV Partitioning Partitioning SRIOV PFIOV Number of PFs per adapter num ports num p...

Page 357: ...ort have the same expansion ROM configuration where PXE UEFI settings are stored This means that all PFs will PXE boot or none will attempt to PXE boot Users should ensure that a DHCP server responds...

Page 358: ...ers provides accelerated cut through performance and is compatible with hypervisor based services and management tools The advanced design of Solarflare SFN7000 and SFN8000 series adapters incorporate...

Page 359: ...hp Ensure the BIOS has all SR IOV Virtualization options enabled On a server with SR IOV correctly configured identify if Virtual Functions VF can be exposed to the host OS Refer to sfboot options bel...

Page 360: ...rt SR IOV Features described in this chapter are not supported by Solarflare SFN5000 or SFN6000 series adapters Solarflare Driver Firmware Features described in the chapter require the following minim...

Page 361: ...tilities on the ESXi host on page 355 Install Solarflare Drivers in the Guest on page 354 Partition the Adapter on page 370 Virtual Machine on page 358 Make PF Passthrough Devices available to the Gue...

Page 362: ...mber VFs exposed per port to host and or Virtual Machine VM The number of MSI X interrupts assigned to each PF or VF Firmware Variant and switch mode To check the current adapter configuration run the...

Page 363: ...or each physical port sfboot switch mode sriov pf count 1 vf count 4 When used without the adapter option the command applies to all adapters DirectPath I O PF Passthrough sfboot Settings The followin...

Page 364: ...Following restart list VFs exposed in the host lspci grep Solarflare 0000 04 00 0 Network controller Solarflare SFC9120 vmnic6 0000 04 00 1 Network controller Solarflare SFC9120 vmnic7 0000 04 00 2 N...

Page 365: ...later VM Compatibility The VM must be compatible with ESXi 5 5 or later When the VM is not compatible the following procedure via the vSphere Web Client will upgrade compatibility Locate the VM from t...

Page 366: ...on Using ESXi Issue 20 Solarflare Communications 2017 359 8 9 List Adapters Web Client To list available adapters Navigate to the Host Networking Physical Adapters Selecting an adapter and clicking th...

Page 367: ...Port Group Configuration Using the vSphere Web Client navigate to the host Right click and select All vCenter Actions Add Networking to display the Add Networking wizard Select Virtual Machine Port G...

Page 368: ...V Virtualization Using ESXi Issue 20 Solarflare Communications 2017 361 When creating a new standard switch click the icon under Assigned adapters Select the required physical adapter s which as uplin...

Page 369: ...e 20 Solarflare Communications 2017 362 Label the portgroup and assign a network label and VLAN ID if required VFs will later be assigned to the same portgroup and will be able to send receive traffic...

Page 370: ...er User Guide SR IOV Virtualization Using ESXi Issue 20 Solarflare Communications 2017 363 The vSwitch and associated uplink adapter s topology can be viewed as follows Select the host Manage tab Netw...

Page 371: ...OS installed A Solarflare SFN7000 or SFN8000 series adapter is physically installed on the host The Solarflare VMware net driver package is installed on the host A Solarflare net driver is installed i...

Page 372: ...Settings dialog enable SRIOV and specify the number of VFs which can be used by the VM NOTE The number of virtual functions should not exceed the value set by the max vfs Solarflare driver option 5 T...

Page 373: ...the Manage tab Settings VM Hardware then click the Edit button to display the VM Edit Settings window 4 To add a VF select Network from the New device drop down list and click the Add button 5 Expand...

Page 374: ...Issue 20 Solarflare Communications 2017 367 7 Select Adapter Type as SR IOV passthrough 8 Select the physical adapter associated with the port group and which will back the passthrough virtual machin...

Page 375: ...indow expand the Memory section and check the Reserve all guest memory All locked check box Reserving all memory is required for both SR IOV and DirectPath configurations because the adapter PF VF mus...

Page 376: ...selected click the Manage tab and the VM Hardware option to view hardware configuration The Solarflare adapter PF and VF are listed in this window Listing Passthrough Devices in a Linux Guest When th...

Page 377: ...partitioning The server must be cold power cycled When the server restarts PFs will be visible in the host lspci vvv grep Solarflare 0000 07 00 0 Ethernet controller Network controller Solarflare SFC9...

Page 378: ...t and follows the procedure from the vmware documentation for DirectPath I O VMware Networking User Guide for ESXi 5 5 1 Navigate to the host Select the Manage tab Settings option 2 Under the Hardware...

Page 379: ...as Available pending The host must be restarted to effect the changes Assign PF Passthrough Devices to the VM 1 Locate and select the VM in the vSphere Web Client 2 Power OFF the VM 3 Select the Manag...

Page 380: ...dd for each PF to be passed through 9 Click the OK button when done 10 Power ON the VM Listing Devices in a Windows Guest Start the Solarflare Adapter Manager GUI by selecting Start select arrow icon...

Page 381: ...so be invoked using the Solarflare supplied sfboot utility For instructions on the sfboot method refer to the sfboot commands in the relevant OS specific sections of this user guide Using the sfboot u...

Page 382: ...upport on page 376 sfupdate Options for PXE upgrade downgrade on page 376 Starting PXE Boot on page 378 iPXE Image Create on page 382 Multiple PF PXE Boot on page 384 9 2 Solarflare Boot Manager The s...

Page 383: ...evelopment mailing lists and participate to ensure the iPXE driver for Solarflare adapters operates correctly NOTE It is recommended that customers having support questions on the iPXE feature set wor...

Page 384: ...ite img image dat Use the force option when downgrading Use the backup option to create a backup image dat file of the current firmware and Solarflare Boot Manager image sfupdate iPXE image To install...

Page 385: ...ation Tool on page 190 From a Linux environment you can use the sfboot utility See Configuring the Boot Manager with sfboot on page 88 From a VMware environment you can use the sfboot utility See Sola...

Page 386: ...enable ANSI terminal emulation on both the BIOS and your terminal Some BIOSs are known to not render the Solarflare Boot Manager properly when using vt100 terminal emulation 1 On starting or re start...

Page 387: ...The Global Adapter Options menu is displayed 4 Select the required boot image a Use the arrow keys to highlight the Boot Image b Use the space bar to choose the required image c Press the Esc key to e...

Page 388: ...the arrow keys to highlight the Boot Type b Use the space bar to select PXE Solarflare recommend leaving other settings at their default values For details on the default values for the various adapt...

Page 389: ...Communications 2017 382 9 6 iPXE Image Create Solarflare do not provide pre built iPXE images The Solarflare iPXE boot ROM image can be generated from the rom o matic iPXE web builder wizard available...

Page 390: ...vendor identifier 1924 b Enter the SFN8000 PCI Device Code 0a03 c Select the GIT version master is recommended 4 Generate the image file Click the Proceed button to start image generation then downloa...

Page 391: ...ical interface sfboot Solarflare boot configuration utility v4 5 0 eth2 Boot image Option ROM only Link speed Negotiated automatically Link up delay time 5 seconds Banner delay time 2 seconds Boot ski...

Page 392: ...s are not available Adapter wide options are available via eth2 00 0F 53 25 39 90 eth5 Interface specific boot options are not available Adapter wide options are available via eth2 00 0F 53 25 39 90 U...

Page 393: ...e a server is moved to a different part of the network PXE booting will then fail To correct these settings you must use the Solarflare Boot Manger GUI utility You cannot use the sfboot command line u...

Page 394: ...Settings Resetting an adapter does not change the boot ROM image To reset an adapter to its default settings 1 Start or re start the server and when prompted press Ctrl B The Solarflare Boot Manager...

Page 395: ...s Enter The settings are reset to the defaults NOTE Under Windows you can also reset the settings by clicking Default from SAM Table 9 8 lists the various adapter settings and their default values Tab...

Page 396: ...Solarflare Server Adapter User Guide Solarflare Boot Manager Issue 20 Solarflare Communications 2017 389 DHCP Vendor MPIO attempts MSIX Limit 32 Table 85 Default Adapter Settings Setting Default Value...

Page 397: ...ith SAM 166 Ethtool Configure Interrupt moderation on Linux 105 Configure Interrupt moderation on VMware 284 Configure segmentation offload 107 Configuring adapter 70 Running adapter diagnostics on Li...

Page 398: ...ies Configuration 184 NIC Partitioning 74 O OpenOnload 2 P PCI Express Lane Configuration On FreeBSD 311 On Linux 115 On VMware 286 On Windows 236 PF IOV 346 Powerd Service Tuning on FreeBSD 312 PXE C...

Page 399: ...T TCP Protocol Tuning On Linux 109 On VMware 286 On Windows 238 Teaming see also sfteam on Windows 200 Adding adapters with SAM 172 Configure on VMware 267 Deleting from SAM 173 Key adapter 225 Reconf...

Page 400: ...Solarflare Server Adapter User Guide Index Issue 20 Solarflare Communications 2017 393...

Reviews:

No comments