manualshive.com logo in svg

Cavium FastLinQ 45000 Series, User Manual

The Cavium FastLinQ 45000 Series brings high-performance networking solutions for seamless connectivity. Unlock the full potential of your device by downloading the comprehensive and free User Manual from manualshive.com. Get ready to maximize your networking capabilities with this powerful product.

Share
Download
background image

7–RoCE Configuration
Configuring RoCE on the Adapter for Linux

143

BC0154501-00  P

Configuring RoCE on the Adapter for Linux

This section describes the RoCE configuration procedure for RHEL and SLES. It 
also describes how to verify the RoCE configuration and provides some guidance 
about using group IDs (GIDs) with vLAN interfaces.

RoCE Configuration for RHEL

RoCE Configuration for SLES

RoCE Configuration for Ubuntu

Verifying the RoCE Configuration on Linux

vLAN Interfaces and GID Index Values

RoCE v2 Configuration for Linux

Responder 
Local QP 
Operation 
error

An internal QP consistency error 
was detected while processing this 
work request.

Yes

Yes

Indicates a software 
issue.

Responder 
remote 
invalid 
request

The responder detected an invalid 
inbound message on the channel.

Yes

Yes

Indicates possible mis-
behavior by a remote 
peer. Possible causes 
include: the operation is 
not supported by this 
receive queue, insuffi-
cient buffering to 
receive a new RDMA 
request, or the length 
specified in an RDMA 
request is greater than 
2

31

 bytes.

Table 7-3. Cavium FastLinQ RDMA Error Counters (Continued)

RDMA Error 

Counter

Description

Applies 

to 

RoCE?

Applies 

to 

iWARP?

Troubleshooting

Summary of Contents for FastLinQ 45000 Series

Page 1: ...BC0154501 00 P User s Guide Converged Network Adapters and Intelligent Ethernet Adapters FastLinQ 45000 Series ...

Page 2: ...back page Removed references to Windows Nano Server System Requirements on page 7 Table 2 2 Chapter 3 Driver Installation Chapter 4 Firmware Upgrade Utility Chapter 14 Windows Server 2016 Before Table 3 4 added Cavium has certified qedrntv for ESXi 6 5 and ESXi 6 7 which are bun dled along with qedentv as one package in vCG listings VMware Drivers and Driver Packages on page 24 Updated the Linux f...

Page 3: ...e ESXi OS Supported Operating Systems for iSCSI Boot on page 60 Following Step 1 of the To install SLES 12 SP3 and later procedure added this note Although they are not necessarily required for iSCSI boot from SAN for SUSE Cavium recommends that you also complete Step 2 through Step 15 of the Configuring iSCSI Boot from SAN for RHEL 7 5 and Later procedure Configuring iSCSI Boot from SAN for SLES ...

Page 4: ...ew Step 1 that describes how to create an nfs client directory and grant permission In Step 3 updated the command used for NFS Version 4 Configuring NFS RDMA on page 175 Following Step 12 added a fourth bullet to the note Switch dependent teaming IEEE 802 3ad LACP and Generic Static Link Aggregation Trunk ing cannot use a switch independent partitioned virtual adapter IEEE standards require Switch...

Page 5: ...tep 1 On all physical interfaces set the value of the NetworkDirect Functional ity parameter to Enabled Creating a Hyper V Virtual Switch with an RDMA NIC on page 242 In the To add host virtual NICs procedure In Step 2 corrected the last digit of the com mand to S was 5 Changed the note s first bullet second sen tence to The same VLAN ID must be assigned to ports on the switch was The same VLAN ID...

Page 6: ...EMI and EMC Requirements xxv KCC Class A xxvi VCCI Class A xxvi Product Safety Compliance xxvi 1 Product Overview Functional Description 1 Features 1 Adapter Management 3 QLogic Control Suite CLI 4 QLogic QConvergeConsole GUI 4 QLogic QConvergeConsole vCenter Plug in 4 QConvergeConsole PowerKit 5 FastLinQ ESXCLI VMware Plug in 5 Adapter Specifications 5 Physical Characteristics 5 Standards Specifi...

Page 7: ...lic Key for Secure Boot 19 Installing Windows Driver Software 20 Installing the Windows Drivers 21 Removing the Windows Drivers 21 Managing Adapter Properties 22 Setting Power Management Options 23 Installing VMware Driver Software 23 VMware Drivers and Driver Packages 24 Installing VMware Drivers 25 VMware NIC Driver Optional Parameters 26 VMware Driver Parameter Defaults 28 Removing the VMware D...

Page 8: ...SCSI Boot Parameters 65 Configuring Adapter UEFI Boot Mode 65 Configuring iSCSI Boot Options 67 Configuring the DHCP Server to Support iSCSI Boot 77 Configuring iSCSI Boot from SAN on Windows 81 Before You Begin 81 Selecting the Preferred iSCSI Boot Mode 82 Configuring iSCSI General Parameters 82 Configuring the iSCSI Initiator 82 Configuring the iSCSI Targets 83 Detecting the iSCSI LUN and Inject...

Page 9: ...apter 129 Preparing the Ethernet Switch 129 Configuring the Cisco Nexus 6000 Ethernet Switch 129 Configuring the Dell Z9100 Ethernet Switch 131 Configuring the Arista 7060X Ethernet Switch 133 Configuring RoCE on the Adapter for Windows Server 135 Viewing RDMA Counters 138 Configuring RoCE on the Adapter for Linux 143 RoCE Configuration for RHEL 144 RoCE Configuration for SLES 144 RoCE Configurati...

Page 10: ...reparing the Adapter for iWARP 168 Configuring iWARP on Windows 169 Configuring iWARP on Linux 172 Installing the Driver 172 Configuring iWARP and RoCE 172 Detecting the Device 173 Supported iWARP Applications 174 Running Perftest for iWARP 174 Configuring NFS RDMA 175 iWARP RDMA Core Support on SLES 12 SP3 and RHEL 7 4 177 9 iSER Configuration Before You Begin 179 Configuring iSER for RHEL 180 Co...

Page 11: ... 205 iSCSI Offload in Linux Environments 205 Differences from bnx2i 206 Configuring qedi ko 206 Verifying iSCSI Interfaces in Linux 206 11 FCoE Configuration Configuring Linux FCoE Offload 209 Differences Between qedf and bnx2fc 210 Configuring qedf ko 210 Verifying FCoE Devices in Linux 211 12 SR IOV Configuration Configuring SR IOV on Windows 213 Configuring SR IOV on Linux 220 Configuring SR IO...

Page 12: ... DCBX on the Adapter 249 Configuring QoS by Enabling DCBX on the Adapter 253 Configuring VMMQ 257 Enabling VMMQ on the Adapter 258 Creating a Virtual Machine Switch with or Without SR IOV 258 Enabling VMMQ on the Virtual Machine Switch 259 Getting the Virtual Machine Switch Capability 260 Creating a VM and Enabling VMMQ on VMNetworkadapters in the VM 260 Default and Maximum VMMQ Virtual NIC 262 En...

Page 13: ...hat Current Drivers Are Loaded 275 Verifying Drivers in Windows 275 Verifying Drivers in Linux 275 Verifying Drivers in VMware 276 Testing Network Connectivity 276 Testing Network Connectivity for Windows 276 Testing Network Connectivity for Linux 277 Microsoft Virtualization with Hyper V 277 Linux specific Issues 277 Miscellaneous Issues 277 Troubleshooting Windows FCoE and iSCSI Boot from SAN 27...

Page 14: ...ith FCoE Offload and iSCSI Offload 48 5 16 Global Bandwidth Allocation Page 49 5 17 Partition 1 Configuration 50 5 18 Partition 2 Configuration FCoE Offload 51 5 19 Partition 3 Configuration iSCSI Offload 52 5 20 Partition 4 Configuration Ethernet 52 5 21 Changing the Port Mode to 4x25 54 5 22 Changing the Port Mode to 1x100 55 5 23 Changing the Port Mode to 4x10 56 5 24 Changing the Port Mode to ...

Page 15: ...Box Installation 121 6 37 Red Hat Enterprise Linux 7 4 Configuration 122 6 38 ESXi Customizer Dialog Box 124 6 39 Select a VMware Disk to Install 125 6 40 VMware Generic USB Boot Options 126 7 1 Configuring RoCE Properties 136 7 2 Add Counters Dialog Box 138 7 3 Performance Monitor Cavium FastLinQ Counters 140 7 4 Switch Settings Server 154 7 5 Switch Settings Client 155 7 6 Configuring RDMA_CM Ap...

Page 16: ...Host Edit Settings 227 13 1 NVMe oF Network 230 13 2 Subsystem NQN 234 13 3 Confirm NVMe oF Connection 235 13 4 FIO Utility Installation 236 14 1 Enabling RDMA in Host Virtual NIC 242 14 2 Hyper V Virtual Ethernet Adapter Properties 243 14 3 Windows PowerShell Command Get VMNetworkAdapter 244 14 4 Windows PowerShell Command Get NetAdapterRdma 244 14 5 Add Counters Dialog Box 246 14 6 Performance M...

Page 17: ...faults 28 5 1 Adapter Properties 37 6 1 iSCSI Out of Box and Inbox Boot from SAN Support 59 6 2 iSCSI General Configuration Options 70 6 3 DHCP Option 17 Parameter Definitions 77 6 4 DHCP Option 43 Sub option Definitions 78 6 5 DHCP Option 17 Sub option Definitions 80 6 6 FCoE Out of Box and Inbox Boot from SAN Support 107 7 1 OS Support for RoCE v1 RoCE v2 iWARP iSER and OFED 127 7 2 Advanced Pro...

Page 18: ...BK SP CK QL45631HOCU BK SP CK NOTE QConvergeConsole QCC GUI is the only GUI management tool across all Cavium QLogic adapters QLogic Control Suite QCS GUI is no longer supported for the FastLinQ 45000 Series Adapters and adapters based on 57xx 57xxx controllers and has been replaced by the QCC GUI management tool The QCC GUI provides single pane of glass GUI management for all Cavium adapters In W...

Page 19: ... the use of the utility to upgrade adapter firmware and boot code Chapter 5 Adapter Preboot Configuration describes the preboot adapter configuration tasks using the Human Infrastructure Interface HII application Chapter 6 Boot from SAN Configuration covers boot from SAN configuration for both iSCSI and FCoE Chapter 7 RoCE Configuration describes how to configure the adapter the Ethernet switch an...

Page 20: ...uments that are available on the Marvell Web Site Installation Guide QConvergeConsole GUI part number SN0051105 00 contains detailed information on how to install and use the QConvergeConsole GUI management tool User s Guide QLogic Control Suite CLI part number BC0054511 00 contains detailed information on how to install start and use QLogic Control Suite CLI User s Guide PowerShell part number BC...

Page 21: ...d links to Web sites are shown in underlined blue For example Table 9 2 lists problems related to the user interface and remote agent See Installation Checklist on page 6 For more information visit www marvell com Text in bold font indicates user interface elements such as a menu items buttons check boxes or column headings For example Click the Start button point to Programs point to Accessories ...

Page 22: ... type as shown For example qaucli pr nic ei angle brackets indicate a variable whose value you must specify For example serial_number square brackets indicate an optional parameter For example file_name means specify a file name or omit it to select the default file name vertical bar indicates mutually exclusive options select one option only For example on off 1 2 3 4 ellipsis indicates that the ...

Page 23: ...ot have an account click Sign Up You are directed to a page to create a support account If you have an account enter your User Name and Password and then click Sign In 5 From the links at the top of the page click Submit a Case Downloading Updates and Documentation The Marvell Web site provides periodic updates to product firmware software and documentation To download QLogic firmware software and...

Page 24: ...s Legal notices covered in this section include laser safety FDA notice agency certification and product safety compliance Laser Safety FDA Notice This product complies with DHHS Rules 21CFR Chapter I Subchapter J This product has been designed and manufactured according to IEC60825 1 on the safety label of laser product Class 1 Laser Product Caution Class 1 laser radiation when open Do not view d...

Page 25: ...sired operation ICES 003 Compliance Class A This Class A digital apparatus complies with Canadian ICES 003 Cet appareil numériqué de la classe A est conformé à la norme NMB 003 du Canada CE Mark 2014 30 EU 2014 35 EU EMC Directive Compliance EN55032 2012 CISPR 32 2015 Class A EN55024 2010 EN61000 3 2 Harmonic Current Emission EN61000 3 3 Voltage Fluctuation and Flicker Immunity Standards EN61000 4...

Page 26: ...No 60950 1 07 2nd Edition A1 A2 2014 10 Use only with listed ITE or equivalent Complies with 21 CFR 1040 10 and 1040 11 2014 30 EU 2014 35 EU Product Name Model Converged Network Adapters and Intelligent Ethernet Adapters Certification holder QLogic Corporation Manufactured date Refer to date code listed on product Manufacturer Country of origin QLogic Corporation USA A class equipment Business pu...

Page 27: ...e Legal Notices xxvii BC0154501 00 P 2006 95 EC low voltage directive TUV EN60950 1 2006 A11 A1 A12 A2 2nd Edition TUV IEC 60950 1 2005 2nd Edition Am1 2009 Am2 2013 CB CB Certified to IEC 60950 1 2nd Edition ...

Page 28: ...ity Features The 45000 Series Adapters provide the following features Some features may not be available on all adapters NIC partitioning NPAR Universal RDMA Remote Direct Memory Access over converged Ethernet versions 1 and 2 RoCE v1 v2 and Internet wide area Protocol iWARP iSCSI Extensions for RDMA iSER NVMe over Fabrics NVMe oF For details download the Deployment Guide Non volatile Memory Expre...

Page 29: ...nt coalescing RSC Microsoft dynamic virtual machine queue VMQ and Linux Multiqueue Adaptive interrupts Transmit receive side scaling TSS RSS Accelerated Receive Flow Steering ARFS Stateless offloads for Network Virtualization using Generic Routing Encapsulation NVGRE Generic Network Virtualization Encapsulation GENEVE and virtual LAN VXLAN L2 L3 GRE tunneled traffic1 Manageability System managemen...

Page 30: ...emory not applicable to all models 1 024 classification filters not applicable to all models Support for multicast addresses through 128 bit hashing hardware function Serial flash NVRAM memory PCI Power Management Interface v1 1 64 bit base address register BAR support EM64T processor support iSCSI and FCoE boot support3 Adapter Management The following applications are available to manage 45000 S...

Page 31: ...ic adapters on both local and remote computer systems QCC GUI is dependent upon additional software a management agent for the adapter The management agent is installed separately from QCC GUI QCC GUI cannot communicate with the hardware until the management agent has been installed For information about installing QCC GUI see the Installation Guide QConvergeConsole GUI For information about using...

Page 32: ... single function to monitor and manage your Cavium QLogic FastLinQ Adapters For information about installing and using QConvergeConsole PowerKit refer to the User s Guide PowerShell FastLinQ ESXCLI VMware Plug in FastLinQ ESXCLI VMware Plug in extends the capabilities of the ESX CLI to manage Cavium QLogic FastLinQ 3400 8400 41000 45000 Series Adapters installed in VMware ESX ESXi hosts For inform...

Page 33: ...apter Specifications 6 BC0154501 00 P 802 3 2015 IEEE Standard for Ethernet flow control 802 3 2015 Clause 78 Energy Efficient Ethernet EEE 1588 2002 PTPv1 Precision Time Protocol 1588 2008 PTPv2 IPv4 RFQ 791 IPv6 RFC 2460 ...

Page 34: ...ems visit the Marvell Web site Table 2 1 Host Hardware Requirements Hardware Requirement Architecture IA 32 or EMT64 that meets operating system requirements PCIe PCIe Gen 2 x8 2x25G NIC x16 2x40G 1x100G NIC PCIe Gen 3 x8 2x25G NIC x16 2x40G 1x100G NIC Full dual port 40Gb and single port 100Gb bandwidth is sup ported on PCIe Gen3 x16 or faster slots Full dual port 25Gb bandwidth is supported on PC...

Page 35: ...a complete list of supported operating systems visit the Marvell Web site WARNING The adapter is being installed in a system that operates with voltages that can be lethal Before you open the case of your system observe the following precautions to protect yourself and to prevent damage to the system components Remove any metallic objects or jewelry from your hands and wrists Make sure to use only...

Page 36: ...ut performing these tasks refer to the manuals that were supplied with the system To install the adapter 1 Review Safety Precautions on page 8 and Preinstallation Checklist on page 9 Before you install the adapter ensure that the system power is OFF the power cord is unplugged from the power outlet and that you are following proper electrical grounding procedures 2 Open the system case and select ...

Page 37: ...lush against the system chassis 6 Secure the adapter with the adapter clip or screw as recommended by the system vendor 7 Close the system case and disconnect any personal anti static devices 8 Connect to the Ethernet network using your preferred and supported cabling option see Tested Cables and Optical Modules on page 282 CAUTION Do not use excessive force when seating the card because this may ...

Page 38: ...Linux Driver Operation Defaults Linux Driver Messages Statistics Importing a Public Key for Secure Boot NOTE Cavium now supports QConvergeConsole QCC GUI as the only GUI management tool across all Cavium QLogic adapters QLogic Control Suite QCS GUI is no longer supported for the 45000 Series Adapters and adapters based on 57xx 57xxx controllers and has been replaced by the QCC GUI management tool ...

Page 39: ...qed interfaces with the qede qedr qedi and qedf drivers The Linux core module manages all PCI device resources registers host interface queues and so on The qed core module requires Linux kernel version 2 6 32 or later Testing was concentrated on the x86_64 architecture qede Linux Ethernet driver for the 45000 Series Adapter This driver directly controls the hard ware and is responsible for sendin...

Page 40: ...rvell Web site 2 Remove the existing Linux drivers as described in Removing the Linux Drivers on page 14 3 Install the new Linux drivers using one of the following methods Installing Linux Drivers Using the src RPM Package Installing Linux Drivers Using the kmp kmod RPM Package Installing Linux Drivers Using the TAR File NOTE For network installations through NFS FTP or HTTP using a network boot d...

Page 41: ...inux drivers were installed using an RPM package issue the following commands rmmod qede rmmod qed depmod a rpm e qlgc fastlinq kmp default version arch If the Linux drivers were installed using a TAR file issue the following commands rmmod qede rmmod qed depmod a For RHEL and CentOS cd lib modules version extra qlgc fastlinq rm rf qed ko qede ko qedr ko For SLES cd lib modules version updates qlg...

Page 42: ...dules version updates qlgc fastlinq rm rf qed ko qede ko qedr ko Installing Linux Drivers Using the src RPM Package To install Linux drivers using the src RPM package 1 Issue the following at a command prompt rpm ivh qlgc fastlinq version src rpm 2 Change the directory to the RPM path and build the binary RPM for the kernel For RHEL and CentOS cd root rpmbuild rpmbuild bb SPECS fastlinq version sp...

Page 43: ...stall kmod RPM package 1 Issue the following command at a command prompt rpm ivh qlgc fastlinq version arch rpm 2 Reload the driver modprobe r qede modprobe qede Installing Ubuntu Linux Drivers To install Ubuntu Linux drivers issue the following command dpkg i fastlinq dkms_ version _all deb Installing Linux Drivers Using the TAR File To install Linux drivers using the TAR file 1 Create a director...

Page 44: ...s described in Removing the Linux Drivers on page 14 4 Install the new Linux drivers using one of the following methods Installing Linux Drivers Using the kmp kmod RPM Package Installing Linux Drivers Using the TAR File 5 Install libqedr libraries to work with RDMA user space applications The libqedr RPM is available only for inbox OFED Issue the following command rpm ivh qlgc libqedr version arch...

Page 45: ...le Enables or disables the hardware generic receive offload GRO feature This feature is similar to the kernel s software GRO but is only performed by the device hardware err_flags_override A bitmap for disabling or forcing the actions taken in case of a hardware error bit 31 An enable bit for this bitmask bit 0 Prevent hardware attentions from being reasserted bit 1 Collect debug data bit 2 Trigge...

Page 46: ... the QLogic public key to load the drivers in a Secure Boot environment Before you begin ensure that your server supports Secure Boot This section provides two methods for importing and enrolling the public key To import and enroll the QLogic public key 1 Download the public key from the following Web page http ldriver qlogic com Module public key 2 To install the public key issue the following co...

Page 47: ...the C key 3 Issue the following commands chainloader efibootdir MokManager efi boot 4 Select Enroll key from disk 5 Navigate to the cert der file and then press ENTER 6 Follow the instructions to enroll the key Generally this includes pressing the 0 zero key and then pressing the Y key to confirm For additional information about Secure Boot refer to the following Web page https www suse com docume...

Page 48: ...agreement c Click Install to start the installation d When the installation is complete click Finish 4 Verify that the Windows drivers have been installed a Click Start and then click Control Panel b In the Control Panel click Programs and then click Programs and Features c In the installed programs list locate QLogic FastLinQ Driver Installer Removing the Windows Drivers To remove the Windows dri...

Page 49: ... change the 45000 Series Adapter properties 1 In the Control Panel click Device Manager 2 On the properties of the selected adapter click the Advanced tab 3 On the Advanced page Figure 3 1 select an item under Property and then change the Value for that item as needed Figure 3 1 Setting Advanced Adapter Properties ...

Page 50: ...mputer attempts to go into hibernation To have the controller remain on at all times do not select the Allow the computer to turn off the device to save power check box Figure 3 2 Figure 3 2 Power Management Options Installing VMware Driver Software This section describes the qedentv VMware ESXi driver for the 45000 Series Adapters VMware Drivers and Driver Packages Installing VMware Drivers VMwar...

Page 51: ...ns and applicable driver versions Table 3 4 VMware Drivers VMware Driver Description qedentv Native networking driver qedrntv Native RDMA Offload RoCE and RoCEv2 drivera a The certified RoCE driver is not included in this release The uncertified driver may be available as an early preview qedf Native FCoE Offload driver qedil Legacy iSCSI Offload driver Table 3 5 ESXi Driver Packages by Release ES...

Page 52: ...rivers from different ZIP files will cause problems To install the VMware driver 1 Download the VMware driver for the 45000 Series Adapter from the VMware support page www vmware com support html 2 Power up the ESX host and then log into an account with administrator authority 3 Use the Linux scp utility to copy the driver bundle from a local system into the tmp directory on an ESX server with IP ...

Page 53: ...e the knowledge base article here Updating an ESXi ESX host using VMware vCenter Update Manager 4 x and 5 x 1019545 To upgrade the existing driver bundle Issue the following command esxcli software vib update d tmp qedentv bundle 2 0 3 zip To upgrade an individual driver Follow the steps for a new installation see To install the VMware driver except replace the command in Option 1 with the followi...

Page 54: ...source allocation constraint RSS Specifies the number of receive side scaling queues used by the host or vir tual extensible LAN VXLAN tunneled traffic for a PF RSS can be 2 3 4 or one of the following values 1 uses the default number of queues 0 or 1 disables RSS queues You can specify multiple values delimited by commas for multiport or multi function configurations debug Specifies the level of ...

Page 55: ...ity The default is enable_vxlan_offld 1 You can specify multiple values delimited by commas for multiport or multifunction configurations Table 3 7 VMware Driver Parameter Defaults Parameter Default Speed Autonegotiation with all speeds advertised The speed parameter must be the same on all ports If auto negotiation is enabled on the device all of the device ports will use autonegotiation Flow Con...

Page 56: ...d network interface controllers C NICs The driver is a kernel mode driver that provides a translation layer between the VMware SCSI stack and the QLogic FCoE firmware and hardware The FCoE and DCB feature set is supported on VMware ESXi 5 0 and later iSCSI Support The QLogic VMware iSCSI qedil Host Bus Adapter HBA driver similar to qedf is a kernel mode driver that provides a translation layer bet...

Page 57: ...nfwnx2 for Windows Both utilities are console applications that can be called from other processes or used as a command line tool with parameters For more information about these utilities their command syntax and parameters see their respective ReadMe files This chapter includes the following firmware upgrade information Image Verification on page 31 Upgrading Adapter Firmware on Linux on page 31...

Page 58: ...running 2 Go to the Marvell Web site www marvell com 3 Point to Support and then under Driver Downloads click Marvell QLogic FastLinQ Drivers 4 On the Cavium Downloads and Documentation page select your adapter model to locate and download the Firmware Upgrade Utility for Linux 5 Unzip the Firmware Upgrade Utility on the system where the adapter is installed All files should reside in the same fol...

Page 59: ... Firmware Upgrade Utility on the system where the adapter is installed All files should reside in the same folder 6 Run the Windows upgrade utility script by issuing the following command C Windows_FWupg AMD64 WinQlgcUpg bat Converting a 100G Adapter to Four port 25G Adapter To convert a FastLinQ QL45611HLCU 100G adapter to a four port 25G adapter use the Firmware Upgrade Utility v2 10 47 or later...

Page 60: ...following Getting Started on page 34 Displaying Firmware Information on page 38 Configuring Device level Parameters on page 38 Configuring Port level Parameters on page 41 Configuring FCoE Boot on page 43 Configuring iSCSI Boot on page 45 Configuring Partitions on page 47 Changing the Adapter Port Mode on page 53 NOTE The HII screen shots in this chapter are representative and may not match the sc...

Page 61: ...or information about launching the System Setup consult the user guide for your system 2 In the System Setup window Figure 5 1 select Device Settings and then press ENTER Figure 5 1 System Setup 3 In the Device Settings window Figure 5 2 select the 45000 Series Adapter port that you want to configure and then press ENTER Figure 5 2 System Setup Device Settings ...

Page 62: ... P The Main Configuration Page presents the adapter management options where you can set the partitioning mode If you are not using NPAR set the Partitioning Mode to Default as shown in Figure 5 3 Figure 5 3 Main Configuration Page Setting Default Partitioning Mode ...

Page 63: ...onfiguration see Configuring Device level Parameters on page 38 Port Level Configuration see Configuring Port level Parameters on page 41 iSCSI Configuration if iSCSI remote boot is available see Configuring iSCSI Boot on page 45 FCoE Configuration if FCoE boot from SAN is available see Configuring FCoE Boot on page 43 Partitions Configuration if NPAR is selected as the Partitioning Mode see Confi...

Page 64: ...anufacturer assigned permanent device MAC address Virtual MAC Address User defined device MAC address iSCSI MAC Address Manufacturer assigned permanent device iSCSI Offload MAC address iSCSI Virtual MAC Address User defined device iSCSI Offload MAC address FCoE MAC Address Manufacturer assigned permanent device FCoE Offload MAC address FCoE Virtual MAC Address User defined device FCoE Offload MAC ...

Page 65: ...e firmware interface UEFI driver version L2B Firmware Version is the Ethernet component version not shown in Figure 5 5 Figure 5 5 Firmware Information Window Configuring Device level Parameters Device level configuration includes the following parameters Link Speed FEC Mode forward error correction SR IOV MFW Crash Dump Feature BAR 2 Size UEFI Driver Debug Level Port Mode NOTE The iSCSI physical ...

Page 66: ...y set up a link between the adapter and the switch over DAC or optical media without user intervention or manual setup The IEEE standards do not provide a standards based method to auto negotiate between a 10G switch and a 25G adapter Cavium s SmartAN provides an automatic and convenient method to detect the switch and to determine and set the link speed FEC types media type and length and so on A...

Page 67: ... is set explicitly to 10 Gbps or 25 Gbps on 25Gb adapters 10 Gbps or 40 Gbps on 40Gb adapters or 100 Gbps on 100Gb adapters None disables FEC Auto sets the FEC mode to automatic This setting allows the adapter to try the different applicable for that speed FEC modes while in a fixed speed mode that is not SmartAN or Auto Negotiated modes until a link is established Fire Code enables FEC BASE R FEC...

Page 68: ...arameters 1 On the Main Configuration Page select Port Level Configuration and then press ENTER 2 For Boot Mode select values for the port level parameters as shown in Figure 5 7 The available parameters will vary depending on whether you are using a 25Gb 40Gb or a 100Gb adapter and what boot modes are available PXE enables PXE boot FCoE enables FCoE boot from SAN over the hardware offload pathway...

Page 69: ...y value of the attached DCB switch Typically 0 is used for the default common lossy traffic class 3 is used for the lossless FCoE traffic class and 4 is used for lossless iSCSI TLV over DCB traffic class 5 For iSCSI Offload select either Enabled or Disabled This setting is only visible on 45000 Series Adapters that are enabled with iSCSI Offload 6 For FCoE Offload select either Enabled or Disabled...

Page 70: ...f the FIP vLAN ID is known beforehand you can set the value here If a non zero value is used FIP vLAN discovery is not performed Fabric Login Retry Count Target Login Retry Count options include Connect 1 8 is Enabled or Disabled WWPN 1 8 shows the WWPN in use all zeros if the target is offline A indicates that the specified target is online while a indicates that the specified target is offline B...

Page 71: ...figuration Configuring FCoE Boot 44 BC0154501 00 P Figure 5 8 FCoE General Parameters Figure 5 9 FCoE Target Configuration 4 Click Back 5 When prompted click Yes to save the changes Changes take effect after a system reset ...

Page 72: ... 3 Choose values for the appropriate iSCSI configuration parameters iSCSI General Configuration Figure 5 10 on page 46 TCP IP Parameters Via DHCP iSCSI Parameters Via DHCP CHAP Authentication CHAP Mutual Authentication IP Version ARP Redirect DHCP Request Timeout Target Login Timeout DHCP Vendor ID iSCSI Initiator Configuration Figure 5 11 on page 46 IPv4 Address IPv4 Subnet Mask IPv4 Default Gate...

Page 73: ...et Configuration Figure 5 13 on page 47 Connect IPv4 Address TCP Port Boot LUN iSCSI Name CHAP ID CHAP Secret 4 Click Back 5 When prompted click Yes to save the changes Changes take effect after a system reset Figure 5 10 iSCSI General Configuration Figure 5 11 iSCSI Initiator Configuration ...

Page 74: ...Second Target Configuration Configuring Partitions You can configure bandwidth ranges for each partition on the adapter To configure the maximum and minimum bandwidth allocations 1 On the Main Configuration Page select Partitions Configuration and then press ENTER 2 On the Partitions Configuration page select Global Bandwidth Allocation ...

Page 75: ...ns Configuration Page No FCoE Offload or iSCSI Offload Figure 5 15 shows the page when NPAR mode is enabled with FCoE Offload and iSCSI Offload enabled Figure 5 15 Partitions Configuration Page with FCoE Offload and iSCSI Offload 3 On the Global Bandwidth Allocation page Figure 5 16 click each partition minimum and maximum TX bandwidth field for which you want to allocate bandwidth There are eight...

Page 76: ...h to all zeros is similar to equally dividing the available bandwidth over every active partition however the bandwidth is dynamically allocated over all actively sending partitions A zero value when one or more of the other values are set to a non zero value allocates a minimum of one percent to that partition when congestion from all of the partitions is restricting TX bandwidth Partition n Maxi...

Page 77: ... Number of VFs per PF if SR IOV was enabled on the Device Level Configuration page PCI Device ID PCI bus Address Permanent MAC Address Virtual MAC Address Partition 1 is always present and cannot be disabled Figure 5 17 Partition 1 Configuration 3 To configure the second partition select Partition 2 Configuration to open the Partition 2 Configuration page If FCoE Offload is present the Partition 2...

Page 78: ... 3 Configuration to open the Partition 3 Configuration page Figure 5 19 If iSCSI Offload is present the Partition 3 Configuration shows the Personality as iSCSI Figure 5 19 and the following additional parameters NIC Mode Disabled iSCSI Offload Mode Enabled iSCSI Offload MAC Address2 Virtual iSCSI Offload MAC Address1 PCI Device ID PCI Address 2 This parameter is only present on the third partitio...

Page 79: ...a partition 2 or greater Ethernet partition The Personality is specified as Ethernet Figure 5 20 and includes the following additional parameters NIC Mode Enabled or Disabled When disabled the partition is hidden such that it does not appear to the OS if fewer than the maximum quantity of partitions or PCI PFs are detected PCI Device ID PCI Address Permanent MAC Address Virtual MAC Address Figure ...

Page 80: ...verting a 100G Adapter to Four port 25G Adapter Follow the steps in this procedure to convert a single QSFP28 ported QLogic FastLinQ QL45611HLCU or QL45631HOCU from single port 100G mode to four port 25G mode To convert a single port 100G adapter to a four port 25G adapter 1 Boot to the UEFI or HII setup utility by pressing the appropriate function key F1 through F12 depending on the platform 2 Se...

Page 81: ...ported QLogic FastLinQ QL45611HLCU or QL45631HOCU from four port 25G mode to single port 100G mode To convert a four port 25G adapter to a single port 100G adapter 1 Boot to the UEFI or HII setup utility by pressing the appropriate function key F1 through F12 depending on the platform 2 Select a QLogic FastLinQ QL45611HxCU or QL45631HOCU Adapter 3 On the Main Configuration Page select Device Level...

Page 82: ...dure to convert a QLogic FastLinQ QL454xxHLCU or QL45412HLCU from 40G mode to four port 10G mode To convert a 40G adapter to a four port 10G adapter 1 Boot to the UEFI or HII setup utility by pressing the appropriate function key F1 through F12 depending on the platform 2 Select a QL454xxHLCU or QL45412HLCU Adapter 3 On the Main Configuration Page select Device Level Configuration 4 On the Device ...

Page 83: ...mode to 40G mode To convert a four port 10G adapter to a 40G adapter 1 Boot to the UEFI or HII setup utility by pressing the appropriate function key F1 through F12 depending on the platform 2 Select a QLogic FastLinQ QL454xx Adapter 3 On the Main Configuration Page select Device Level Configuration CAUTION Adapter port mode changes alter the base PCIe configuration Cavium recommends that you perf...

Page 84: ... power cycle the system The adapter port mode change takes effect after the cold reboot CAUTION Adapter port mode changes alter the base PCIe configuration Cavium recommends that you perform a cold boot power cycle to prevent undefined results NOTE On the dual QSFP port 40G QL454x2HxCU adapter the one QSFP port that was previously disabled is now enabled and both QSFP ports are placed into 40G mod...

Page 85: ...from SAN FCoE Boot from SAN on page 107 iSCSI Boot from SAN Cavium 45000 Series gigabit Ethernet GbE adapters support iSCSI boot to enable network boot of operating systems to diskless systems iSCSI boot allows a Windows Linux or VMware operating system to boot from an iSCSI target machine located remotely over a standard IP network This section provides the following configuration information abo...

Page 86: ...are offload Yes Yes No No Windows 2012 R2 a Yes Yes No No Windows 2016b b Windows Server 2016 does not support the inbox iSCSI driver for hardware offload Yes Yes Yes No Windows 2019 Yes Yes Yes Yes RHEL 6 10 Yes Yes Yes No RHEL 7 4c c RHEL 7 4 supports the inbox qedi driver but does not support hardware offload iSCSI boot from SAN Yes Yes Yes No RHEL 7 5 Yes Yes Yes Yes SLES 12 SP3 Yes Yes Yes Ye...

Page 87: ...tocol Configuring the iSCSI Target Configuring iSCSI Boot Parameters Configuring Adapter UEFI Boot Mode Configuring iSCSI Boot Options Configuring the DHCP Server to Support iSCSI Boot Supported Operating Systems for iSCSI Boot The Cavium FastLinQ 45000 Series Adapters support iSCSI boot on the following operating systems Windows Server 2012 and later 64 bit supports offload and non offload paths ...

Page 88: ...s the OEM hotkey to enter the System setup or configuration menu This is also known as UEFI HII For example the HPE Gen 9 systems use F9 as a hotkey to access the System Utilities menu at boot time Figure 6 1 Figure 6 1 Systems Utilities at Boot Time NOTE SAN boot is supported in UEFI environment only Make sure the system boot option is UEFI and not legacy ...

Page 89: ... guide on accessing the PCI device configuration menu For example on a HPE Gen 9 server the System Utilities for QLogic devices are listed on the System Configuration menu Figure 6 2 Configuration Utility 4 On the Main Configuration Page select Port Level Configuration Figure 6 3 and then press ENTER Figure 6 3 Selecting Port Level Configuration ...

Page 90: ... of the following configuration options Static iSCSI Boot Configuration on page 68 Dynamic iSCSI Boot Configuration on page 74 NOTE The iSCSI HW option is not listed if the iSCSI Offload feature is disabled at the port level If the preferred boot mode is iSCSI HW make sure the iSCSI offload feature is enabled Not all adapter versions support iSCSI offload and iSCSI HW offloaded boot Additionally n...

Page 91: ...nterface in UEFI mode Figure 6 5 Port Level Configuration Configuring the iSCSI Target Configuring the iSCSI target varies by target vendors For information on configuring the iSCSI target refer to the documentation provided by the vendor To configure the iSCSI target 1 Select the appropriate procedure based on your iSCSI target either Create an iSCSI target using software such as SANBlaze or Linu...

Page 92: ...ther static or dynamic configuration For configuration options available from the General Parameters window see Table 6 2 on page 70 which lists parameters for both IPv4 and IPv6 Parameters that are specific to either IPv4 or IPv6 are noted Configuring Adapter UEFI Boot Mode To configure the boot mode 1 Restart the system 2 Press the OEM hotkey to enter the System setup or configuration menu This ...

Page 93: ...ot Time 3 On the Main Configuration Page select Port Level Configuration Figure 6 7 and then press ENTER Figure 6 7 Selecting Port Level Configuration 4 On the Port Level Configuration page Figure 6 8 select Boot Mode and then press ENTER to select one of the following iSCSI boot modes iSCSI SW iSCSI HW ...

Page 94: ...iSCSI boot configuration options include Static iSCSI Boot Configuration Dynamic iSCSI Boot Configuration Enabling CHAP Authentication NOTE The iSCSI HW option is not listed if the iSCSI Offload feature is disabled at port level If the preferred boot mode is iSCSI HW make sure the iSCSI offload feature is enabled Not all adapter versions support iSCSI offload and iSCSI HW offloaded boot Additional...

Page 95: ... address Initiator IQN Target parameters obtained in Configuring the iSCSI Target on page 64 For information on configuration options see Table 6 2 on page 70 To configure the iSCSI boot parameters using static configuration 1 In the Device UEFI HII Main Configuration Page select iSCSI Configuration Figure 6 9 and then press ENTER Figure 6 9 Selecting iSCSI Boot Configuration ...

Page 96: ...e DOWN ARROW key to select a parameter and then press the ENTER key to input the following values Table 6 2 on page 70 provides descriptions of these parameters TCP IP Parameters via DHCP Disabled iSCSI Parameters via DHCP Disabled CHAP Authentication As required IP Version As required IPv4 or IPv6 ARP Redirect Not applicable for boot DHCP Request Timeout Default value or as required Target Login ...

Page 97: ...formation is entered on the iSCSI Initiator Parameters Configuration page CHAP Authentication Controls whether the iSCSI boot host software uses CHAP authentication when connecting to the iSCSI target If CHAP Authentication is enabled configure the CHAP ID and CHAP Secret on the iSCSI Initiator Parameters Configuration page IP Version This option is specific to IPv6 Toggles between IPv4 and IPv6 A...

Page 98: ...eneral traffic If this is the case make the iS2CSI interface on the adapter a member of that VLAN by setting this value iSCSI Name Corresponds to the iSCSI initiator name to be used by the client system CHAP ID CHAP Secret NOTE For the preceding items with asterisks note the following The label will change to IPv6 or IPv4 default based on the IP version set on the iSCSI General Configuration page ...

Page 99: ...ess ESC 8 Select iSCSI First Target Parameters Figure 6 14 and then press ENTER Figure 6 14 iSCSI First Target Parameters 9 On the iSCSI First Target Parameters page set the Connect option to Enabled for the iSCSI target 10 Type values for the following parameters for the iSCSI target and then press ENTER IPv4 Address TCP Port Boot LUN iSCSI Name CHAP ID CHAP Secret ...

Page 100: ...d target is used if the first target cannot be connected to Otherwise proceed to Step 13 Figure 6 16 iSCSI Second Target Configuration 13 Press ESC once and a second time to exit 14 Press the Y key to save changes or follow the OEM guidelines to save the device level configuration For example in an HPE Gen 9 system press Y to confirm the setting change Figure 6 17 NOTE For the preceding parameters...

Page 101: ...ystem s IP address and target or initiator information are provided by a DHCP server see IPv4 and IPv6 configurations in Configuring the DHCP Server to Support iSCSI Boot on page 77 Any settings for the following parameters are ignored and do not need to be cleared with the exception of the initiator iSCSI name for IPv4 CHAP ID and CHAP secret for IPv6 Initiator Parameters First Target Parameters ...

Page 102: ...a DHCP Enabled iSCSI Parameters via DHCP Enabled CHAP Authentication As required IP Version As required IPv4 or IPv6 ARP Redirect Not applicable for boot DHCP Request Timeout Default value or as required Target Login Timeout Default value or as required DHCP Vendor ID As required NOTE When using a DHCP server the DNS server entries are overwritten by the values provided by the DHCP server This ove...

Page 103: ...ng CHAP ID up to 255 characters CHAP Secret if authentication is required must be 12 to 16 characters in length 4 Press ESC to return to the iSCSI Boot Configuration page 5 On the iSCSI Boot Configuration Menu select iSCSI First Target Parameters 6 In the iSCSI First Target Parameters window type values used when configuring the iSCSI target CHAP ID optional if two way CHAP CHAP Secret optional if...

Page 104: ...ormation to the DHCP client For iSCSI boot Cavium QLogic adapters support the following DHCP configurations DHCP Option 17 Root Path DHCP Option 43 Vendor specific Information DHCP Option 17 Root Path Option 17 is used to pass the iSCSI target information to the iSCSI client The format of the root path as defined in IETC RFC 4173 is iscsi servername protocol port LUN targetname Table 6 3 lists the...

Page 105: ...ration options You should use DHCP Option 43 when performing dynamic iSCSI boot configuration LUN Logical unit number to use on the iSCSI target The value of the LUN must be represented in hexadecimal format A LUN with an ID of 64 must be configured as 40 within the Option 17 parameter on the DHCP server targetname Target name in either IQN or EUI format For details on both IQN and EUI formats ref...

Page 106: ...QLGC ISAN as shown in the General Parameters of the iSCSI Boot Configuration menu The content of Option 16 should be 2 byte length DHCP Vendor ID DHCPv6 Option 17 Vendor Specific Information DHCPv6 Option 17 vendor specific information provides more configuration options to the iSCSI client In this configuration three additional sub options are provided that assign the initiator IQN to the iSCSI b...

Page 107: ...ember of that vLAN To configure vLAN for iSCSI boot 1 Go to the iSCSI Boot Configuration Menu for the port 2 Select iSCSI Initiator Configuration 3 Select VLAN ID to enter and set the vLAN value as shown in Figure 6 19 Table 6 5 DHCP Option 17 Sub option Definitions Sub option Definition 201 First iSCSI target information in the standard root path format iscsi servername protocol port LUN targetna...

Page 108: ... Protocol to UEFI iSCSI iSCSI boot from SAN for Windows information includes the following Before You Begin Selecting the Preferred iSCSI Boot Mode Configuring iSCSI General Parameters Configuring the iSCSI Initiator Configuring the iSCSI Targets Detecting the iSCSI LUN and Injecting the QLogic Drivers Before You Begin Before you begin configuring iSCSI boot from SAN on a Windows machine note the ...

Page 109: ...n the iSCSI General Configuration page see Figure 6 11 on page 69 press the DOWN ARROW key to select a parameter and then press the ENTER key to input the following values see Table 6 2 on page 70 provides descriptions of these parameters TCP IP Parameters via DHCP Disabled for static iSCSI boot or Enabled for dynamic iSCSI boot iSCSI Parameters via DHCP Disabled CHAP Authentication As required IP...

Page 110: ...the iSCSI first target second target or both at once To set the iSCSI target parameters on Windows 1 From the Main Configuration page select iSCSI Configuration and then select iSCSI First Target Configuration 2 On the iSCSI First Target Parameters page set the Connect option to Enabled for the iSCSI target 3 Type values for the following parameters for the iSCSI target and then press ENTER IPv4 A...

Page 111: ...s 1 Reboot the system access the HII and determine if the iSCSI LUN is detected Issue the following UEFI Shell version 2 script command map r b The output from the preceding command shown in Figure 6 20 indicates that the iSCSI LUN was detected successfully at the preboot level Figure 6 20 Detecting the iSCSI LUN Using UEFI Shell Version 2 2 On the newly detected iSCSI LUN select an installation s...

Page 112: ...s in the virtual media a Click Load driver and then click Browse see Figure 6 22 Figure 6 22 Windows Setup Selecting Driver to Install b Navigate to the driver location and choose the qevbd driver c Choose the adapter on which to install the driver and then click Next to continue 5 Repeat Step 4 to load the qeios driver QLogic L4 iSCSI driver 6 After injecting the qevbd and qeios drivers click Nex...

Page 113: ...N for RHEL 7 5 and Later Configuring iSCSI Boot from SAN for SLES 12 SP3 and Later Configuring iSCSI Boot from SAN for Other Linux Distributions Configuring iSCSI Boot from SAN for RHEL 7 4 To install RHEL 7 4 1 Boot from the RHEL 7 x installation media with the iSCSI target already connected in UEFI Install Red Hat Enterprise Linux 7 x Test this media install Red Hat Enterprise 7 x Troubleshootin...

Page 114: ...igure 6 23 example Figure 6 23 Prompt for Out of Box Installation 5 If required for your setup load the FastLinQ driver update disk when prompted for additional driver disks Otherwise if you have no other driver update disks to install press the C key 6 Continue with the installation You can skip the media test Click Next to continue with the installation ...

Page 115: ...section select the iSCSI LUN 10 Type the root user s password and then click Next to complete the installation 11 During the first boot add the following kernel command line to fall into shell rd iscsi firmware rd break pre pivot rd driver pre qed qede qedr qedf qedi selinux 0 12 Issue the following commands umount sysroot boot efi umount sysroot boot umount sysroot home umount sysroot mount dev m...

Page 116: ...ontain other parameters that can remain Change only the iscsi_firmware string as follows GRUB_CMDLINE_LINUX rd iscsi firmware selinux 0 18 Create a new grub cfg file by issuing the following command grub2 mkconfig o boot efi EFI redhat grub cfg 19 Rebuild the ramdisk by issuing the dracut f command and then reboot NOTE If the iSCSI target is on a different subnet than the initiator update the open...

Page 117: ...ver update disk when prompted for additional driver disks Otherwise if you have no other driver update disks to install press the C key 6 Continue with the installation You can skip the media test Click Next to continue with the installation 7 In the Configuration window select the language to use during the installation process and then click Continue 8 In the Installation Summary window click In...

Page 118: ...l SLES 12 SP3 and later 1 Boot from the SLES 12 SP3 installation media with the iSCSI target pre configured and connected in UEFI 2 Update the latest driver package by adding the dud 1 parameter in the installer command parameter The driver update disk is required because the necessary iSCSI drivers are not inbox 3 Complete the installation steps specified by the SLES 12 SP3 OS NOTE Although they ...

Page 119: ...a driver update disk DUD However no interface or process exists to update userspace inbox utilities which causes the iSCSI target login and boot from SAN installation to fail To overcome this limitation perform the initial boot from SAN with the pure L2 interface do not use hardware offloaded iSCSI using the following procedure during the boot from SAN iSCSI offload for other distributions of Linu...

Page 120: ... the kernel boot parameters f Issue the following command BOARDTYPE pc KEYTABLE us rd_LVM_LV vg_prebooteit lv_root rhgb quiet selinux 0 For RHEL 7 2 and 7 3 At the beginning of the installation pass the following boot parameter with the DUD option rd iscsi ibft inst dd For SLES 11 SP4 and SLES 12 SP1 SP2 At the beginning of the installation pass the following boot parameter with the DUD option ip ...

Page 121: ...obe q iscsi_tcp modprobe q ib_iser modprobe q cxgb3i modprobe q cxgb4i modprobe q bnx2i modprobe q be2iscsi modprobe q qedi daemon iscsiuio 3 Open the boot efi EFI redhat grub conf file make the following changes and save the file Remove ifname eth5 14 02 ec ce dc 6d Remove ip ibft Add selinux 0 For example kernel vmlinuz 2 6 32 696 el6 x86_64 ro root dev mapper vg_prebooteit lv_root rd_NO_LUKS is...

Page 122: ...conf file as follows a Locate string node session timeo replacement_timeout 120 b Change value from 120 to 5 seconds 6 Reboot the server and then open the UEFI HII 7 In the HII open System Configuration select the adapter port and then select Port Level Configuration 8 On the Port Level Configuration page set the Boot Mode to iSCSI HW and set iSCSI Offload to Enabled 9 Save the configuration and r...

Page 123: ...o systemctl restart iscsid 4 Edit the usr libexec iscsi mark root nodes file as follows a Locate the following statement if transport bnx2i then b Change the statement to if transport bnx2i transport qedi then 5 Edit the etc default grub file as follows a Locate the following statement GRUB_CMDLINE_LINUX iscsi_firmware ip ibft b Change this statement to GRUB_CMDLINE_LINUX rd iscsi firmware 6 Creat...

Page 124: ...ns by issuing the following commands rpm ivh qlgc open iscsi 2 0_873 111 sles11sp4 3 x86_64 rpm force rpm ivh iscsiuio 2 11 5 3 2 sles11sp4 x86_64 rpm force 2 Edit the etc elilo conf file make the following changes and then save the file Remove the ip ibft parameter if present Add iscsi_firmware 3 Edit the etc sysconfig kernel file as follows a Locate the line that begins with INITRD_MODULES This ...

Page 125: ...t Level Configuration 10 On the Port Level Configuration page set the Boot Mode to iSCSI HW and set iSCSI Offload to Enabled 11 Save the configuration and reboot the server The OS can now boot through the offload interface Migrating to Offload iSCSI for SLES 12 SP1 SP2 To migrate from a software iSCSI installation to an offload iSCSI for SLES 12 SP1 SP2 1 Boot into the iSCSI non offload L2 boot fr...

Page 126: ... skip to Step 6 c 6 Edit the etc default grub file and modify the GRUB_CMDLINE_LINUX value a Remove rd iscsi ibft if present b Remove any ip value boot options if present c Add rd iscsi firmware For older distros add iscsi_firmware 7 Create a backup of the original grub cfg file The file is in the following locations Legacy boot boot grub2 grub cfg UEFI boot boot efi EFI sles grub cfg for SLES 8 C...

Page 127: ...he installation in a single path using either an L2 or L4 path After the server boots into the installed operating system perform the required configurations for enabling multipath I O MPIO See the appropriate procedure in this section to migrate from L2 to L4 and configure MPIO for your OS Migrating and Configuring MPIO to Offloaded Interface for RHEL 6 9 6 10 Migrating and Configuring MPIO to Of...

Page 128: ... L2 5 Update Open iSCSI tools and iscsiuio by issuing the following commands rpm ivh qlgc open iscsi 2 0_873 111 rhel6u9 3 x86_64 rpm force rpm ivh iscsiuio 2 11 5 5 6 rhel6u9 x86_64 rpm force 6 Edit the boot efi EFI redhat grub conf file make the following changes and then save the file a Remove ifname eth5 14 02 ec ce dc 6d b Remove ip ibft c Add selinux 0 7 Build the initramfs file by issuing t...

Page 129: ...t to the OS with L2 5 Update Open iSCSI tools and iscsiuio by issuing the following commands rpm ivh qlgc open iscsi 2 0_873 111 rhel7u3 3 x86_64 rpm force rpm ivh iscsiuio 2 11 5 5 6 rhel7u3 x86_64 rpm force 6 Reload all the daemon services by issuing the following command systemctl daemon reload 7 Restart iscsid and iscsiuio services by issuing the following commands systemctl restart iscsiuio s...

Page 130: ...mfs with multipath support dracut force add multipath include etc multipath 16 Reboot the server and boot into the OS with multipath Migrating and Configuring MPIO to Offloaded Interface for SLES 11 SP4 To migrate from L2 to L4 and configure MPIO to boot the OS over an offloaded interface for SLES 11 SP4 1 Follow all the steps necessary to migrate a non offload L2 interface to an offload L4 interf...

Page 131: ...s etc init d boot multipath start etc init d multipathd start 5 Run multipath v2 d to display multipath configuration with a dry run 6 Locate the multipath conf file under etc multipath conf 7 Edit the multipath conf to enable the default section 8 Rebuild initrd image to include MPIO support mkinitrd f multipath 9 Reboot the server and boot the OS with multipath support NOTE If the file is not pr...

Page 132: ... 0_873 111 sles12sp1 3 x86_64 rpm force rpm ivh iscsiuio 2 11 5 5 6 sles12sp1 x86_64 rpm force 6 Edit the etc default grub file by changing the rd iscsi ibft parameter to rd iscsi firmware and then save the file 7 Issue the following command grub2 mkconfig o boot efi EFI suse grub cfg 8 To load the multipath module issue the following command modprobe dm_multipath 9 To enable the multipath daemon ...

Page 133: ...t Level Configuration b On the Port Level Configuration page set the Boot Mode to iSCSI HW and set iSCSI Offload to Enabled c Save the settings and then exit the System Configuration Menu 14 Reboot the system The OS should now boot through the offload interface NOTE For any additional changes in the etc multipath conf file to take effect you must rebuild the initrd image and reboot the server ...

Page 134: ...t machine located remotely over an FCoE supporting network You can set the FCoE option offload path with QLogic offload FCoE driver by opening the NIC Configuration menu and setting the Boot Protocol option to FCoE This section provides the following configuration information about FCoE boot from SAN FCoE Out of Box and Inbox Support FCoE Preboot Configuration Configuring FCoE Boot from SAN on Win...

Page 135: ...FI mode only Set the platform in boot mode protocol using the system BIOS configuration to UEFI RHEL 6 10 Yes No RHEL 7 4 Yes No RHEL 7 5 Yes Yes SLES 12 SP4 Yes Yes SLES 15 Yes Yes ESXi 6 0 U3 Yes No ESXi 6 5 U1 Yes No ESXi 6 5 U2 Yes No ESXi 6 7 Yes No NOTE FCoE boot from SAN is supported on ESXi 5 5 and later Not all adapter versions support FCoE and FCoE boot from SAN NOTE FCoE BFS is not supp...

Page 136: ...m 2 Press the OEM hot key to enter System Setup or the configuration menu Figure 6 25 This is also known as UEFI HII For example the HPE Gen 9 system uses F9 as a hotkey to access the System Utilities menu at boot time Figure 6 25 System Utilities NOTE SAN boot is supported in the UEFI environment only Make sure the system boot option is UEFI and not legacy ...

Page 137: ...n accessing PCI device configuration menu For example on an HPE Gen 9 server the System Utilities for Cavium QLogic devices are listed under the System Configuration menu Figure 6 26 System Configuration Port Selection 4 On the Main Configuration Page select Port Level Configuration Figure 6 27 and then press ENTER Figure 6 27 Port Level Configuration ...

Page 138: ...elect FCoE as a preferred boot mode Figure 6 28 Boot Mode in Port Level Configuration Figure 6 29 FCoE Offload Enabled NOTE FCoE is not listed as a boot option if the FCoE Offload feature is disabled at the port level If the Boot Mode preferred is FCoE make sure the FCoE Offload feature is enabled as shown in Figure 6 29 Not all adapter versions support FCoE ...

Page 139: ...ameters 1 On the Device HII Main Configuration Page select FCoE Configuration Figure 6 30 and then press ENTER Figure 6 30 Selecting FCoE Boot Configuration 2 In the FCoE Boot Configuration Menu select FCoE General Parameters Figure 6 31 and then press ENTER Figure 6 31 Selecting General Parameters ...

Page 140: ...c Login Retry Count Default value or as required Target Login Retry Count Default value or as required Figure 6 32 FCoE General Parameters 4 Return to the FCoE Boot Configuration page 5 Press ESC and then select FCoE Target Parameters 6 Press ENTER 7 In the FCoE Target Configuration enable Connect to the preferred FCoE target 8 Type values for the following parameters Figure 6 33 for the FCoE targ...

Page 141: ...E Boot Installation Configuring FCoE on Windows FCoE Crash Dump on Windows Injecting Slipstreaming Adapter Drivers into Windows Image Files Windows Server 2012 R2 and 2016 FCoE Boot Installation For Windows Server 2012R2 2016 boot from SAN installation Cavium requires the use of a slipstream DVD or ISO image with the latest Cavium QLogic drivers injected See Injecting Slipstreaming Adapter Drivers...

Page 142: ...ce 7 Boot from DVD and begin installation 8 Follow the on screen instructions On the window that shows the list of disks available for the installation the FCoE target disk should be visible This target is a disk connected through the FCoE boot protocol located in the remote FCoE target 9 To proceed with Windows Server 2012R2 2016 installation select Next and then follow the on screen instructions...

Page 143: ... driver package For example type c temp d Follow the driver installer instructions to install the drivers in the specified folder In this example the Cavium QLogic driver files are installed here c temp Program File 64 QLogic Corporation QDrivers 3 Download the Windows Assessment and Deployment Kit ADK version 10 from Microsoft https developer microsoft com en us windows hardware windows assessmen...

Page 144: ... interface and are automatically activated by the qedf driver SLES 11 The driver update disk is required because the necessary FCoE drivers are not inbox The installer parameter dud 1 is required to ensure that the installer will ask for the driver update disk Do not use the installer parameter withfcoe 1 because the software FCoE will conflict with the hardware offload if network interfaces from ...

Page 145: ... the inbox drivers to ensure that the out of box drivers from the driver update disk will load correctly For details see Configuring FCoE Boot from SAN for RHEL 7 4 and Later on page 120 Configuring Linux FCoE Boot from SAN This section provides FCoE boot from SAN procedures for the following Linux distributions Configuring FCoE Boot from SAN for RHEL 6 10 Configuring FCoE Boot from SAN for RHEL 7...

Page 146: ...the B key to continue with the installation 8 At the Select the file which is your driver disk image prompt select the out of box driver FastLinQ driver update disk and then click OK Figure 6 34 shows an example Figure 6 34 Selecting the Driver Disk Image 9 Load the FastLinQ driver update disk and then click Next to continue with the installation You can skip the media test 10 At the What type of ...

Page 147: ...ce lldpad stop chkconfig fcoe off chkconfig lldpad off Configuring FCoE Boot from SAN for RHEL 7 4 and Later To install RHEL 7 4 and later 1 Boot from the RHEL 7 x installation media with the FCoE target already connected in UEFI Install Red Hat Enterprise Linux 7 x Test this media install Red Hat Enterprise 7 x Troubleshooting Use the UP and DOWN keys to change the selection Press e to edit the s...

Page 148: ...Figure 6 23 example Figure 6 36 Prompt for Out of Box Installation 6 If required for your setup load the FastLinQ driver update disk when prompted for additional driver disks Otherwise press the C key if you have no other driver update disks to install 7 Continue with the installation You can skip the media test Click Next to continue with the installation ...

Page 149: ...ck Next to complete the installation 12 During the first boot add the following kernel command line to fall into shell rd driver pre qed qede qedr qedf qedi 13 After a successful system boot edit the etc modprobe d anaconda blacklist conf file to remove the blacklist entry for the selected driver 14 Rebuild the ramdisk by issuing the dracut f command and then reboot 15 Turn off the lldpad and fcoe...

Page 150: ...umentation Configuring FCoE Boot from SAN on VMware For VMware ESXi 6 5 6 7 boot from SAN installation Cavium requires that you use a customized ESXi ISO image that is built with the latest Cavium QLogic Converged Network Adapter bundle injected This section covers the following VMware FCoE boot from SAN procedures Injecting Slipstreaming ESXi Adapter Drivers into Image Files Installing the Custom...

Page 151: ...example Figure 6 38 ESXi Customizer Dialog Box 5 Burn a DVD that contains the customized ISO build located in the working directory specified in Step 4c 6 Use the new DVD to install the ESXi OS Installing the Customized ESXi ISO 1 Load the latest Cavium QLogic FCOE boot images into the adapter NVRAM 2 Configure the FCOE target to allow a valid connection with the remote machine Ensure that the tar...

Page 152: ...e visible because the injected Converged Network Adapter bundle is inside the customized ESXi ISO Figure 6 39 shows an example Figure 6 39 Select a VMware Disk to Install 7 Select the LUN on which ESXi can install and then press ENTER 8 On the next window click Next and then follow the on screen instructions 9 When installation completes reboot the server and eject the DVD 10 During the server boo...

Page 153: ...6 Boot from SAN Configuration FCoE Boot from SAN 126 BC0154501 00 P In the example shown in Figure 6 40 the first two ports indicate Cavium QLogic adapters Figure 6 40 VMware Generic USB Boot Options ...

Page 154: ...or Linux on page 143 Configuring RoCE on the Adapter for VMware ESX on page 156 Configuring DCQCN on page 162 Supported Operating Systems and OFED Table 7 1 shows the Linux operating system support for RoCE v1 RoCE v2 iWARP iSER and OpenFabrics Enterprise Distribution OFED OFED is not supported on Windows or VMware ESXi NOTE Some RoCE features may not be fully enabled in the current release Table ...

Page 155: ...ce applications in inbox OFED requires installing the InfiniBand Support group by yum groupinstall InfiniBand Support that contains libibcm libibverbs and more OFED and RDMA applications that depend on libibverbs also require the QLogic RDMA user space library libqedr Install libqedr using the libqedr RPM or source packages If the OS or OFED contains the RDMA Core package installation of libqedr m...

Page 156: ...c class and 4 is used for lossless iSCSI TLV over DCB traffic class 4 On the Main Configuration Page for the selected adapter click Port Level Configuration 5 On the Port Level Configuration page set RDMA Protocol Support to RoCE or iWARP RoCE 6 Click Back 7 When prompted click Yes to save the changes Changes will take effect after a system reset For Windows you can configure DCBX using the HII or...

Page 157: ...ollows switch config policy map type qos roce switch config class type qos class roce switch config set qos group 3 6 Configure queuing policy maps to assign network bandwidth In this example use a value of 50 percent switch config policy map type queuing roce switch config class type queuing class roce switch config bandwidth percent 50 7 Configure network QoS policy maps to set priority flow con...

Page 158: ...nd ETS on the port specifying the DCB protocol and assigning a VLAN ID to the switch port To configure the Dell switch 1 Create a DCB map Dell configure Dell conf dcb map roce Dell conf dcbmap roce 2 Configure two ETS traffic classes in the DCB map with 50 percent bandwidth assigned for RoCE group 1 Dell conf dcbmap roce priority group 0 bandwidth 50 pfc off Dell conf dcbmap roce priority group 1 ...

Page 159: ...C is 4 Number of Traffic Classes is 8 Admin mode is on Admin Parameters Admin is enabled PG grp Priority BW BW COMMITTED BW PEAK TSA Rate Mbps Burst KB Rate Mbps Burst KB 0 0 1 2 3 4 6 7 40 ETS 1 5 60 ETS 2 3 Dell Conf do show interfaces twentyFiveGigE 1 8 1 pfc detail Interface twentyFiveGigE 1 8 1 Admin mode is on Admin is enabled Priority list is 5 Remote is enabled Priority list is 5 Remote Wi...

Page 160: ...Switch The following examples show Arista 7060X Switch configuration Configuring RoCE APP TLVs for both RoCE and RoCE v2 The following is an example of configuring RoCE APP TLV for both RoCE and RoCE v2 Arista 7060X EIT config dcbx application ether 0x8915 priority 5 Arista 7060X EIT config dcbx application udp 4791 priority 5 Mapping Priority to Traffic Class In the following example Priority cos...

Page 161: ... EIT config qos map traffic class 1 to tx queue 1 Assign tx queue bandwidth to all switch ports connected to adapters Arista 7060X EIT config interface ethernet 26 1 Arista 7060X EIT config if Et26 1 tx queue 0 Arista 7060X EIT config if Et26 1 txq 0 bandwidth percent 5 Arista 7060X EIT config if Et26 1 txq 0 exit Arista 7060X EIT config if Et26 1 tx queue 1 Arista 7060X EIT config if Et26 1 txq 1...

Page 162: ...rkDirect Functionality Enabled Quality of Service For RoCE v1 v2 always select Enabled to allow Win dows DCB QoS service to control and monitor DCB For more information see Configuring QoS by Disabling DCBX on the Adapter on page 249 and Configuring QoS by Enabling DCBX on the Adapter on page 253 NetworkDirect Technology RoCE or RoCE v2 The iWARP value applies only when configuring ports for iWARP...

Page 163: ...PowerShell verify that RDMA is enabled on the adapter The Get NetAdapterRdma command lists the adapters that support RDMA both ports are enabled PS C Users Administrator Get NetAdapterRdma Name InterfaceDescription Enabled SLOT 4 3 Port 1 QLogic FastLinQ QL45212 True SLOT 4 3 Port 2 QLogic FastLinQ QL45212 True NOTE If you are configuring RoCE over Hyper V do not assign a vLAN ID to the physical i...

Page 164: ...affic and verify the results To set up and connect to an SMB drive view the information available online from Microsoft https technet microsoft com en us library hh831795 v ws 11 aspx 5 By default Microsoft s SMB Direct establishes two RDMA connections per port which provides good performance including line rate at a higher block size for example 64KB To optimize performance you can change the qua...

Page 165: ...Windows Server 138 BC0154501 00 P Viewing RDMA Counters The following procedure also applies to iWARP To view RDMA counters for RoCE 1 Launch Performance Monitor 2 Open the Add Counters dialog box Figure 7 2 shows an example Figure 7 2 Add Counters Dialog Box ...

Page 166: ... there is congestion in the network Pause counters increment when flow control or global pause is configured and there is a congestion in the network PFC counters increment when priority flow control is configured and there is a congestion in the network Cavium FastLinQ RDMA Error Counters Increment if any error occurs in transport operations For details see Table 7 3 4 Under Instances of selected...

Page 167: ...nters RDMAError Counter Description Applies to RoCE Applies to iWARP Troubleshooting CQ overflow A completion queue on which an RDMA work request is posted This counter specifies the quantity of instances where there was a com pletion for a work request on the send or receive queue but no space on the associated completion queue Yes Yes Indicates a software design issue causing an insufficient com...

Page 168: ... not reference a memory region that is valid for the requested operation Yes Yes Usually indicates an issue with the host soft ware components Requestor local QP operation An internal QP consistency error was detected while processing this work request Yes Yes Requestor Remote access A protection error occurred on a remote data buffer to be read by an RDMA Read written by an RDMA Write or accessed...

Page 169: ...rk issue is preventing messages acknowledgment Responder CQE flushed Posted work requests receive buf fers on RQ may be flushed by send ing completions with a flush status to the CQ if the QP moves to an error state for any reason and pending receive buffers exist on the RQ If a work request completed with an error status all other pending work requests for that QP are flushed Yes Yes Responder lo...

Page 170: ...dex Values RoCE v2 Configuration for Linux Responder Local QP Operation error An internal QP consistency error was detected while processing this work request Yes Yes Indicates a software issue Responder remote invalid request The responder detected an invalid inbound message on the channel Yes Yes Indicates possible mis behavior by a remote peer Possible causes include the operation is not suppor...

Page 171: ...m groupinstall Infiniband Support yum install perftest yum install tcl tcl devel tk zlib devel libibverbs libibverbs devel 3 Install the new Linux drivers as described in Installing the Linux Drivers with RDMA on page 17 RoCE Configuration for SLES To configure RoCE on the adapter for a SLES host OFED must be installed and configured on the SLES host To install inbox OFED for SLES 1 While installi...

Page 172: ...lan automake autoconf dkms git b Install the following RDMA packages required for Ubuntu apt get install f libibverbs librdma libibcm libibmad libibumad c Install RDMA user space tools and libraries required for Ubuntu apt get install f libtool ibutils ibverbs utils rdmacm utils infiniband diags perftest librdmacm dev libibverbs dev numactl libnuma dev libnl 3 200 libnl route 3 200 libnl route 3 d...

Page 173: ...put as unlimited 5 Reboot the system 6 To allow the device to be recognized as an InfiniBand device that can be used by OFED install the FastLinQ package by issuing the following commands cd fastlinq X X X X make clean make install 7 Install the libqedr libraries to work with RDMA user space applications using one of the following command options Option 1 cd fastlinq X X X X make libqedr_install O...

Page 174: ...gateway 192 168 10 254 To assign the DHCP IP address edit the etc network interfaces file with the following entries auto eth0 iface eth0 inet dhcp Verifying the RoCE Configuration on Linux After installing OFED installing the Linux driver and loading the RoCE drivers verify that the RoCE devices were detected on all Linux operating systems To verify RoCE configuration on Linux 1 Stop firewall tab...

Page 175: ...of the modules have been loaded For example lsmod grep qedr qedr 89871 0 qede 96670 1 qedr qed 2075255 2 qede qedr ib_core 88311 16 qedr rdma_cm ib_cm ib_sa iw_cm xprtrdma ib_mad ib_srp ib_ucm ib_iser ib_srpt ib_umad ib_uverbs rdma_ucm ib_ipoib ib_isert 5 Configure the IP address and enable the port using a configuration method such as ifconfig For example ifconfig ethX 192 168 10 10 24 up 6 Issue...

Page 176: ...bv_rc_pingpong d ib dev g 0 server L2 IP address The following are examples of successful ping pong tests on the server and the client Server Ping root captain ibv_rc_pingpong d qedr0 g 0 local address LID 0x0000 QPN 0xff0000 PSN 0xb3e07e GID fe80 20e 1eff fe50 c7c0 remote address LID 0x0000 QPN 0xff0000 PSN 0x934d28 GID fe80 20e 1eff fe50 c570 8192000 bytes in 0 05 seconds 1436 97 Mbit sec 1000 i...

Page 177: ...fe50 c5b0 GID 1 0000 0000 0000 0000 0000 ffff c0a8 0103 GID 2 2001 0db1 0000 0000 020e 1eff fe50 c5b0 GID 3 2001 0db2 0000 0000 020e 1eff fe50 c5b0 GID 4 0000 0000 0000 0000 0000 ffff c0a8 0b03 IP address for vLAN interface GID 5 fe80 0000 0000 0000 020e 1e00 0350 c5b0 vLAN ID 3 RoCE v2 Configuration for Linux To verify RoCE v2 functionality you must use RoCE v2 supported kernels To configure RoCE...

Page 178: ...0000 0000 0000 0000 0000 ffff c0a8 6403 GID 7 0000 0000 0000 0000 0000 ffff c0a8 6403 Verifying the RoCE v1 or RoCE v2 GID Index and Address from sys and class Parameters Use one of the following options to verify the RoCE v1 or RoCE v2 GID Index and address from the sys and class parameters Option 1 cat sys class infiniband qedr0 ports 1 gid_attrs types 0 IB RoCE v1 cat sys class infiniband qedr0...

Page 179: ...c0a8 6403 192 168 100 3 v2 p4p1 100 qedr1 1 0 fe80 0000 0000 0000 020e 1eff fec4 1b21 v1 p4p2 qedr1 1 1 fe80 0000 0000 0000 020e 1eff fec4 1b21 v2 p4p2 Verifying the RoCE v1 or RoCE v2 Function Through perftest Applications This section shows how to verify the RoCE v1 or RoCE v2 function through perftest applications In this example the following server IP and client IP are used Server IP 192 168 ...

Page 180: ...ip route add 192 168 101 0 24 via 192 168 100 1 dev p4p1 100 Client Configuration sbin ip link add link p4p1 name p4p1 101 type vlan id 101 ifconfig p4p1 101 192 168 101 3 24 up ip route add 192 168 100 0 24 via 192 168 101 1 dev p4p1 101 2 Set the switch settings using the following procedure Use any flow control method Pause DCBX CEE or DCBX IEEE and enable IP routing for RoCE v2 See Preparing t...

Page 181: ... P If you are using PFC configuration and L3 routing run RoCE v2 traffic over the vLAN using a different subnet and use the RoCE v2 vLAN GID index Server ib_send_bw d qedr0 F x 5 Client ib_send_bw d qedr0 F x 5 192 168 100 3 Server Switch Settings Figure 7 4 Switch Settings Server ...

Page 182: ...ings for RDMA_CM Applications To configure RoCE use the following scripts from the FastLinQ source package show_rdma_cm_roce_ver sh qedr0 is configured to IB RoCE v1 qedr1 is configured to IB RoCE v1 config_rdma_cm_roce_ver sh v2 configured rdma_cm for qedr0 to RoCE v2 configured rdma_cm for qedr1 to RoCE v2 Server Settings Figure 7 6 Configuring RDMA_CM Applications Server ...

Page 183: ...ll both QLogic NIC and RoCE drivers 2 Using the module parameter enable the RoCE function from the NIC driver by issuing the following command esxcfg module s enable_roce 1 qedentv To apply the change reload the NIC driver or reboot the system 3 To view a list of the NIC interfaces issue the esxcfg nics l command For example esxcfg nics l Name PCI Driver Link Speed Duplex MAC Address MTU Descripti...

Page 184: ...ciate the QLogic NIC port to the vSwitch issue the following command esxcli network vswitch standard uplink add u uplink device v roce vswitch For example esxcli network vswitch standard uplink add u vmnic0 v roce_vs 7 To create a new port group on this vSwitch issue the following command esxcli network vswitch standard portgroup add p roce_pg v roce_vs For example esxcli network vswitch standard ...

Page 185: ... Interface RoCE Mode and Statistics For the RoCE mode ESXi requires concurrent support of both RoCE v1 and v2 The decision regarding which RoCE mode to use is made during queue pair creation The ESXi driver advertises both modes during registration and initialization To view RoCE statistics issue the following command esxcli rdma device stats get d vmrdma0 Packets received 0 Packets sent 0 Bytes r...

Page 186: ...onfiguring a Paravirtual RDMA Device PVRDMA To configure PVRDMA using a vCenter interface 1 Create and configure a new distributed virtual switch as follows a In the VMware vSphere Web Client right click the RoCE node in the left pane of the Navigator window b On the Actions menu point to Distributed Switch and then click New Distributed Switch c Select version 6 5 0 d Under New Distributed Switch...

Page 187: ...ign uplinks Select from the list of available uplinks Manage VMkernel network adapters Accept the default and then click Next Migrate VM networking Assign the port group created in Step 1 3 Assign a vmknic for PVRDMA to use on ESX hosts a Right click a host and then click Settings b On the Settings page expand the System node and then click Advanced System Settings c The Advanced System Settings p...

Page 188: ...for the PVRDMA a Right click a host and then click Settings b On the Settings page expand the System node and then click Security Profile c On the Firewall Summary page click Edit d In the Edit Security Profile dialog box under Name scroll down select the pvrdma check box and then select the Set Firewall check box Figure 7 10 shows an example Figure 7 10 Setting the Firewall Rule ...

Page 189: ...pe e After the VM is booted ensure that the PVRDMA guest driver is loaded Configuring DCQCN Data Center Quantized Congestion Notification DCQCN is a feature that determines how an RoCE receiver notifies a transmitter that a switch between them has provided an explicit congestion notification notification point and how a transmitter reacts to such notification reaction point This section provides t...

Page 190: ...of an incoming packet for PFC purposes rather than according to the vLAN priority or the DSCP field in the IPv4 header You may use an indirection table to indicate a specified DSCP value to a vLAN priority value DSCP PFC can work across L2 networks because it is an L3 IPv4 feature Traffic classes also known as priority groups are groups of vLAN priorities or DSCP values if DSCP PFC is used that ca...

Page 191: ...ission rate and use bandwidth effectively after congestion ceases DCQCN is described in the 2015 SIGCOMM paper Congestion Control for Large Scale RDMA Deployments located here http conferences sigcomm org sigcomm 2015 pdf papers p523 pdf DCB related Parameters Use DCB to map priorities to traffic classes priority groups DCB also controls which priority groups are subject to PFC lossless traffic an...

Page 192: ... enabled Configuring DSCP PFC Use dscp_pfc nodes to configure the dscp priority association for PFC You must enable the feature before you can add entries to the map For example to map DSCP value 6 to priority 5 issue the following commands debugfs sh n eth0 t dscp_pfc_enable 1 debugfs sh n eth0 t dscp_pfc_set 6 5 Enabling DCQCN To enable DCQCN for RoCE traffic probe the qed driver with the dcqcn_...

Page 193: ...to be used on CNPs Values range between 0 63 dcqcn_cnp_vlan_priority vLAN priority to be used on CNPs Values range between 0 7 FCoE Offload uses 3 and iSCSI Offload TLV generally uses 4 Cavium rec ommends that you specify a number from 1 2 or 5 7 Use this same value throughout the entire net work dcqcn_notification_point 0 Disable DCQCN notification point 1 Enable DCQCN notification point dcqcn_re...

Page 194: ...priorities 5 and 6 as subject to pfc debugfs sh n ens6f0 t dcbx_set_mode 0x10 debugfs sh n ens6f0 t dcbx_set_pfc 5 1 debugfs sh n ens6f0 t dcbx_set_pfc 6 1 set roce global overrides for qp params enable exn and open QPs with dscp 20 debugfs sh n ens6f0 t rdma_glob_ecn 1 debugfs sh n ens6f0 t rdma_glob_dscp 20 open some QPs DSCP 20 ib_write_bw d qedr0 q 16 F x 1 run_infinitely change global dscp qp...

Page 195: ...nformation about preboot adapter configuration see Chapter 5 Adapter Preboot Configuration To configure iWARP through HII 1 Access the server BIOS System Setup and then click Device Settings 2 On the Device Settings page select a port for the 25G or 40G 45000 Series Adapter 3 On the Main Configuration Page for the selected adapter click Port Level Configuration 4 On the Port Level Configuration pa...

Page 196: ...affic on Windows For a list of OSs that support iWARP see Table 7 1 on page 127 To enable iWARP on the Windows host and verify RDMA 1 Enable iWARP on the Windows host a Open the Windows Device Manager and then open the 45000 Series Adapter NDIS Miniport Properties b On the FastLinQ Adapter properties click the Advanced tab c On the Advanced page under Property do the following Select Network Direc...

Page 197: ...mand output Figure 8 2 shows NetworkDirect as Enabled Figure 8 2 Windows PowerShell Command Get NetOffloadGlobalSetting To verify iWARP traffic 1 Map SMB drives and run iWARP traffic 2 Launch Performance Monitor Perfmon 3 In the Add Counters dialog box click RDMA Activity and then select the adapter instances Figure 8 3 shows an example Figure 8 3 Perfmon Add Counters ...

Page 198: ... 10 Share1 Microsoft Windows Network The command completed successfully b Issue the netstat xan command as follows where Share1 is mapped as an SMB share C Users Administrator netstat xan Active NetworkDirect Connections Listeners ShareEndpoints Mode IfIndex Type Local Address Foreign Address PID Kernel 56 Connection 192 168 11 20 16159 192 168 11 10 445 0 Kernel 56 Connection 192 168 11 20 15903 ...

Page 199: ...stem includes the following Installing the Driver Configuring iWARP and RoCE Detecting the Device Supported iWARP Applications Running Perftest for iWARP Configuring NFS RDMA iWARP RDMA Core Support on SLES 12 SP3 and RHEL 7 4 Installing the Driver Install the RDMA drivers as shown in Chapter 3 Driver Installation Configuring iWARP and RoCE To enable iWARP and RoCE 1 Unload all FastLinQ drivers as...

Page 200: ...rep iWARP transport iWARP 1 transport iWARP 1 transport iWARP 1 transport iWARP 1 transport iWARP 1 transport iWARP 1 transport iWARP 1 transport iWARP 1 Detecting the Device To detect the device 1 To verify whether RDMA devices are detected view the dmesg logs dmesg grep qedr 10500 191047 qedr 0000 04 00 0 registered qedr0 10500 221726 qedr 0000 04 00 1 registered qedr1 2 Issue the ibv_devinfo co...

Page 201: ... communication manager rdma_cm using the R option rdma_server rdma_client rdma_xserver rdma_xclient rping NFS over RDMA NFSoRDMA iSER for details see Chapter 9 iSER Configuration NVMe oF for details see Chapter 13 NVMe oF Configuration with RDMA Running Perftest for iWARP All perftest tools are supported over the iWARP transport type You must run the tools using the RDMA connection manager with th...

Page 202: ... 62 00 00 00 00 00 00 00 00 00 00 bytes iterations BW peak MB sec BW average MB sec MsgRate Mpps 65536 1000 2250 38 2250 36 0 036006 Configuring NFS RDMA NFS RDMA for iWARP includes both server and client configuration steps To configure the NFS server 1 Create an nfs server directory and grant permission by issuing the following commands mkdir tmp nfs server chmod 777 tmp nfs server 2 In the etc ...

Page 203: ...start status nfs 5 Include the default RDMA port 20049 into this file as follows echo rdma 20049 proc fs nfsd portlist 6 To make local directories available for NFS clients to mount issue the exportfs command as follows exportfs v To configure the NFS client 1 Create an nfs client directory and grant permission by issuing the following commands mkdir tmp nfs client chmod 777 tmp nfs client 2 Load ...

Page 204: ...e source go to the following link and choose RDMA Core 15 or later https github com linux rdma rdma core releases Or to use the latest RDMA Core source issue the following command git clone https github com linux rdma rdma core git 2 Install all OS dependent packages libraries as described in the RDMA Core README For RHEL and CentOS issue the following command yum install cmake gcc libnl3 devel li...

Page 205: ...FGHIJKLMNOPQRSTUVWXYZ _ abcdefghijklmnopqrs ping data rdma ping 2 CDEFGHIJKLMNOPQRSTUVWXYZ _ abcdefghijklmnopqrst ping data rdma ping 3 DEFGHIJKLMNOPQRSTUVWXYZ _ abcdefghijklmnopqrstu ping data rdma ping 4 EFGHIJKLMNOPQRSTUVWXYZ _ abcdefghijklmnopqrstuv client DISCONNECT EVENT 5 To run inbox OFED applications such as perftest and other InfiniBand applications issue the following command to set the...

Page 206: ...on ESXi 6 7 on page 192 Before You Begin As you prepare to configure iSER consider the following iSER is supported only in inbox OFED for the following operating systems RHEL 7 5 and 7 6 SLES12 SP2 SP3 SLES 15 CentOS 7 5 and 7 6 Ubuntu 16 04 LTS and 18 04 LTS VMware ESXi 6 7 After logging into the targets or while running I O traffic unloading the Linux RoCE qedr driver may crash the system While ...

Page 207: ..._isert 5 Verify that all RDMA and iSER modules are loaded on the initiator and target devices by issuing the lsmod grep qed and lsmod grep iser commands 6 Verify that there are separate hca_id instances by issuing the ibv_devinfo command as shown in Step 6 on page 148 7 Check the RDMA connection on the initiator device and the target device a On the initiator device issue the following command rpi...

Page 208: ...the same for any iSCSI target except that you issue the command enable_iser Boolean true on the applicable portals The portal instances are identified as iser in Figure 9 2 Figure 9 2 iSER Portal Instances 9 Install Linux iSCSI Initiator Utilities using the yum install iscsi initiator utils commands a To discover the iSER target issue the iscsiadm command For example iscsiadm m discovery t st p 19...

Page 209: ...er c To connect to or log in to the iSER target issue the iscsiadm command For example iscsiadm m node l p 192 168 100 99 3260 T iqn 2015 06 test target1 d Confirm that the Iface Transport is iser in the target connection as shown in Figure 9 3 Issue the iscsiadm command for example iscsiadm m session P2 Figure 9 3 Iface Transport Confirmed e To check for a new iSCSI device as shown in Figure 9 4 ...

Page 210: ...python configobj 4 7 2 18 10 noarch rpm python PrettyTable 0 7 2 8 5 noarch rpm python configshell 1 5 1 44 noarch rpm python pyparsing 2 0 1 4 10 noarch rpm python netifaces 0 8 6 55 x86_64 rpm python rtslib 2 2 6 6 noarch rpm python urwid 1 1 1 6 144 x86_64 rpm targetcli 2 1 3 8 x86_64 rpm 2 Before starting the targetcli load all RoCE device drivers and iSER modules as follows modprobe qed modpr...

Page 211: ...target 1 Install targetcli by issuing the following command sudo apt get install targetcli 2 To enter the LIO CLI console issue the targetcli command root captain targetcli targetcli GIT_VERSION rtslib GIT_VERSION Copyright c 2011 2013 by Datera Inc All rights reserved ls o o backstores o fileio 0 Storage Object o iblock 0 Storage Object o pscsi 0 Storage Object o rd_dr 0 Storage Object o rd_mcp 0...

Page 212: ...0 Storage Object o rd_mcp 1 Storage Object o iSERPort1 1 ramdisk deactivated o ib_srpt 0 Targets o iscsi 0 Targets o loopback 0 Targets o qla2xxx 0 Targets o tcm_fc 0 Targets 4 Create an iSCSI target by issuing the following command iscsi create wwn iqn 2004 01 com qlogic iSERPort1 Target1 Created target iqn 2004 01 com qlogic iSERPort1 Target1 Selected TPG Tag 1 Successfully created TPG 1 ls o o ...

Page 213: ...fully created LUN 0 ls o o backstores o fileio 0 Storage Object o iblock 0 Storage Object o pscsi 0 Storage Object o rd_dr 0 Storage Object o rd_mcp 1 Storage Object o iSERPort1 1 ramdisk activated o ib_srpt 0 Targets o iscsi 1 Target o iqn 2004 01 com qlogic iSERPort1 Target1 1 TPG o tpgt1 enabled o acls 0 ACLs o luns 1 LUN o lun0 rd_mcp iSERPort1 1 ramdisk o portals 0 Portals o loopback 0 Target...

Page 214: ...d o acls 0 ACLs o luns 1 LUN o lun0 rd_mcp iSERPort1 1 ramdisk o portals 1 Portal o 192 168 10 5 3260 OK iser disabled o loopback 0 Targets o qla2xxx 0 Targets o tcm_fc 0 Targets 7 Enable iSER on the portal by issuing the following command iscsi iqn 2004 01 com qlogic iSERPort1 Target1 tpgt1 port als 192 168 10 103 3260 iser_enable iser operation has been enabled ls o o backstores o fileio 0 Stora...

Page 215: ...ite_protect 0 generate_node_acls 1 cache_dynamic_acls 1 Parameter demo_mode_write_protect is now 0 Parameter authentication is now 0 Parameter generate_node_acls is now 1 Parameter cache_dynamic_acls is now 1 ls o o backstores o fileio 0 Storage Object o iblock 0 Storage Object o pscsi 0 Storage Object o rd_dr 0 Storage Object o rd_mcp 1 Storage Object o iSERPort1 1 ramdisk activated o ib_srpt 0 T...

Page 216: ...ted default config etc target loopback_start sh Making backup of srpt ConfigFS with timestamp 2015 06 09_19 07 37 855693 Successfully updated default config etc target srpt_start sh Making backup of qla2xxx ConfigFS with timestamp 2015 06 09_19 07 37 855693 Successfully updated default config etc target qla2xxx_start sh Making backup of LIO Target ConfigFS with timestamp 2015 06 09_19 07 37 855693...

Page 217: ...ansport_iscsi 100628 4 iscsi_tcp ib_iser libiscsi 2 Issue the following iscsiadm commands to discover the target and change the transport mode to iSER iscsiadm m discovery t st p 192 168 10 5 3260 I iser 192 168 10 5 3260 1 iqn 2004 01 com qlogic iSERPort1 Target1 iscsiadm m node T iqn 2004 01 com qlogic iSERPort1 Target1 o update n iface transport_name v iser 3 Log into the target device by issui...

Page 218: ... scaling_governor do f CPUFREQ continue echo n performance CPUFREQ done Verify that all CPU cores are set to maximum performance mode by issuing the following command cat sys devices system cpu cpu cpufreq scaling_governor Configuring Kernel sysctl Settings Set the kernel sysctl settings as follows sysctl w net ipv4 tcp_mem 4194304 4194304 4194304 sysctl w net ipv4 tcp_wmem 4096 65536 4194304 sysc...

Page 219: ...cho 0 sys block sdd queue add_random echo 1 sys block sdd queue rq_affinity Configuring iSER on ESXi 6 7 This section provides information for configuring iSER for VMware ESXi 6 7 Before You Begin Before you configure iSER for ESXi 6 7 ensure that the following is complete The CNA package with NIC and RoCE drivers is installed on the ESXi 6 7 system and the devices are listed To view RDMA devices ...

Page 220: ...ame PCI Driver Link Speed Duplex MAC Address MTU Description vmnic0 0000 01 00 0 ntg3 Up 1000Mbps Full e0 db 55 0c 5f 94 1500 Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet vmnic1 0000 01 00 1 ntg3 Down 0Mbps Half e0 db 55 0c 5f 95 1500 Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet vmnic2 0000 02 00 0 ntg3 Down 0Mbps Half e0 db 55 0c 5f 96 1500 Broadcom Corporation NetXtreme BCM...

Page 221: ...rsion TCP Protocol Supported false Bidirectional Transfers Supported false Maximum Cdb Length 64 Can Be NIC true Is NIC true Is Initiator true Is Target false Using TCP Offload Engine true Using ISCSI Offload Engine true 4 Add the target to the iSER initiator as follows esxcli iscsi adapter target list esxcli iscsi adapter discovery sendtarget add A vmhba65 a 192 168 10 11 esxcli iscsi adapter tar...

Page 222: ...supported naa 6001405e81ae36b771c418b89c85dae0 Device Type Direct Access Size 512 MB Display Name LIO ORG iSCSI Disk naa 6001405e81ae36b771c418b89c85dae0 Multipath Plugin NMP Console Device vmfs devices disks naa 6001405e81ae36b771c418b89c85dae0 Devfs Path vmfs devices disks naa 6001405e81ae36b771c418b89c85dae0 Vendor LIO ORG Model ram1 Revis 4 0 SCSI Level 5 Is Pseudo false Status degraded Is RDM...

Page 223: ...I offload is a technology that offloads iSCSI protocol processing overhead from host processors to the iSCSI HBA iSCSI offload increases network performance and throughput while helping to optimize server processor use This section covers how to configure the Windows iSCSI offload feature for the Cavium FastLinQ 45000 Series Adapters With the proper iSCSI offload licensing you can configure your i...

Page 224: ...r an automatic service start Confirm the selection for the applet to launch Configuring Microsoft Initiator to Use Cavium s iSCSI Offload After the IP address is configured for the iSCSI adapter you must use Microsoft Initiator to configure and add a connection to the iSCSI target using the Cavium QLogic iSCSI adapter For more details on Microsoft Initiator see the Microsoft user guide To configur...

Page 225: ...00 P Figure 10 1 iSCSI Initiator Properties Configuration Page c In the iSCSI Initiator Name dialog box type the new initiator IQN name and then click OK Figure 10 2 Figure 10 2 iSCSI Initiator Node Name Change 3 On the iSCSI Initiator Properties click the Discovery tab ...

Page 226: ...P 4 On the Discovery page Figure 10 3 under Target portals click Discover Portal Figure 10 3 iSCSI Initiator Discover Target Portal 5 In the Discover Target Portal dialog box Figure 10 4 a In the IP address or DNS name box type the IP address of the target b Click Advanced ...

Page 227: ...0154501 00 P Figure 10 4 Target Portal IP Address 6 In the Advanced Settings dialog box Figure 10 5 complete the following under Connect using a For Local adapter select the QLogic name or model Adapter b For Initiator IP select the adapter IP address c Click OK ...

Page 228: ...10 iSCSI Configuration iSCSI Offload in Windows Server 201 BC0154501 00 P Figure 10 5 Selecting the Initiator IP Address 7 On the iSCSI Initiator Properties Discovery page click OK ...

Page 229: ...10 iSCSI Configuration iSCSI Offload in Windows Server 202 BC0154501 00 P 8 Click the Targets tab and then on the Targets page Figure 10 6 click Connect Figure 10 6 Connecting to the iSCSI Target ...

Page 230: ...sole GUI Question What tools should I use to create the connection to the target Answer Use Microsoft iSCSI Software Initiator version 2 08 or later NOTE Some limitations of the teaming functionality include Teaming does not support iSCSI adapters Teaming does not support NDIS adapters that are in the boot path Teaming supports NDIS adapters that are not in the iSCSI boot path but only for the SLB...

Page 231: ... booting in either the offload or non offload path To set up Windows Server 2012 R2 2016 2019 iSCSI boot 1 Remove any local hard drives on the system to be booted remote system 2 Prepare the Windows OS installation media by following the slipstreaming steps in Injecting Slipstreaming Adapter Drivers into Windows Image Files on page 116 3 Load the latest Cavium QLogic iSCSI boot images into the NVR...

Page 232: ...nal parts of the Linux kernel for specific functionality qed ko is the Linux eCore kernel module used for common Cavium FastLinQ 45000 hardware initialization routines scsi_transport_iscsi ko is the Linux iSCSI transport library used for upcall and downcall for session management libiscsi ko is the Linux iSCSI library function needed for protocol data unit PDU and task processing as well as sessio...

Page 233: ...le for exporting boot information Configuring qedi ko The qedi driver automatically binds to the exposed iSCSI functions of the CNA and the target discovery and binding is done through the Open iSCSI tools This functionality and operation is similar to that of the bnx2i driver To load the qedi ko kernel module issue the following commands modprobe qed modprobe libiscsi modprobe uio modprobe iscsi_...

Page 234: ... 6c 192 168 25 91 empty iqn 1994 05 com redhat 534ca9b6adf 4 To ensure that the iscsiuio service is running issue the following command systemctl status iscsiuio service iscsiuio service iSCSI UserSpace I O driver Loaded loaded usr lib systemd system iscsiuio service disabled vendor preset disabled Active active running since Fri 2017 01 27 16 33 58 IST 6 days ago Docs man iscsiuio 8 Process 3745 ...

Page 235: ... 1e c4 e1 6c target iqn 2003 04 com sanblaze virtualun virtualun target 05000007 portal 192 168 25 100 3260 successful 7 To verify that the iSCSI session was created issue the following command iscsiadm m session qedi 297 192 168 25 100 3260 1 iqn 2003 04 com sanblaze virtualun virtualun target 05000007 non flash 8 To check for iSCSI devices issue the iscsiadm command iscsiadm m session P3 Attache...

Page 236: ...e ko is the Linux FCoE kernel library needed to conduct FCoE forwarder FCF solicitation and FCoE initialization protocol FIP fabric login FLOGI libfc ko is the Linux FC kernel library needed for several functions including Name server login and registration rport session management scsi_transport_fc ko is the Linux FC SCSI transport library used for remote port and SCSI target management These mod...

Page 237: ...it on top of the net_device qedf is not dependent on network drivers such as bnx2x and cnic qedf will automatically initiate FCoE discovery on link up because it is not dependent on fipvlan or fcoemon for FCoE interface creation Configuring qedf ko No explicit configuration is required for qedf ko The driver automatically binds to the exposed FCoE functions of the CNA and begins discovery This fun...

Page 238: ...d 806912 1 qedf scsi_mod 262144 14 sg hpsa qedf scsi_dh_alua scsi_dh_rdac dm_multipath scsi_transport_fc scsi_transport_sas libfc scsi_transport_iscsi scsi_dh_emc libata sd_mod sr_mod 2 Check dmesg to verify that the FCoE devices were detected properly In this example the two detected FCoE CNA devices are SCSI host numbers 4 and 5 dmesg grep qedf 235 321185 0000 00 00 0 qedf_init 3728 QLogic FCoE ...

Page 239: ...L TYPE VENDOR MODEL REV TRAN sdb 5 0 0 0 disk SANBlaze VLUN P2T1L0 V7 3 fc sdc 5 0 0 1 disk SANBlaze VLUN P2T1L1 V7 3 fc sdd 5 0 0 2 disk SANBlaze VLUN P2T1L2 V7 3 fc sde 5 0 0 3 disk SANBlaze VLUN P2T1L3 V7 3 fc sdf 5 0 0 4 disk SANBlaze VLUN P2T1L4 V7 3 fc sdg 5 0 0 5 disk SANBlaze VLUN P2T1L5 V7 3 fc sdh 5 0 0 6 disk SANBlaze VLUN P2T1L6 V7 3 fc sdi 5 0 0 7 disk SANBlaze VLUN P2T1L7 V7 3 fc sdj...

Page 240: ...iguring SR IOV on Windows Configuring SR IOV on Linux on page 220 Configuring SR IOV on VMware on page 224 Configuring SR IOV on Windows To configure SR IOV on Windows 1 Access the server BIOS System Setup and then click System BIOS Settings 2 On the System BIOS Settings page click Integrated Devices 3 On the Integrated Devices page a Set the SR IOV Global Enable option to Enabled b Click Back 4 O...

Page 241: ... 6 On the Main Configuration Page click Finish 7 In the Warning Saving Changes message box click Yes to save the configuration 8 In the Success Saving Changes message box click OK 9 To enable SR IOV on the miniport adapter a Access Device Manager b Open the miniport adapter properties and then click the Advanced tab c On the Advanced properties page Figure 12 2 under Property select SR IOV and the...

Page 242: ...Figure 12 3 on page 216 a Launch the Hyper V Manager b Select Virtual Switch Manager c In the Name box type a name for the virtual switch d Under Connection type select External network e Select the Enable single root I O virtualization SR IOV check box and then click Apply NOTE Be sure to enable SR IOV when you create the vSwitch This option is unavailable after the vSwitch is created ...

Page 243: ...g SR IOV f The Apply Networking Changes message box advises you that Pending changes may disrupt network connectivity To save your changes and continue click Yes 11 To get the virtual machine switch capability issue the following Windows PowerShell command PS C Users Administrator Get VMSwitch Name SR IOV_vSwitch fl ...

Page 244: ... function VF in the VM a Create a virtual machine b Add the VMNetworkadapter to the virtual machine c Assign a virtual switch to the VMNetworkadapter d In the Settings for VM VM_Name dialog box Figure 12 4 Hardware Acceleration page under Single root I O virtualization select the Enable SR IOV check box and then click OK NOTE After the virtual adapter connection is created the SR IOV setting can b...

Page 245: ...Cavium QLogic drivers for the adapters detected in the VM Use the latest drivers available from your vendor for your host OS do not use inbox drivers NOTE Be sure to use the same driver package on both the VM and the host system For example use the same qeVBD and qeND driver version on the Windows VM and in the Windows Hyper V host ...

Page 246: ...pter is listed in the VM Figure 12 5 shows an example Figure 12 5 Device Manager VM with QLogic Adapter 14 To view the SR IOV VF details issue the following Windows PowerShell command PS C Users Administrator Get NetadapterSriovVf Figure 12 6 shows example output Figure 12 6 Windows PowerShell Command Get NetadapterSriovVf ...

Page 247: ...or Settings page a Set the Virtualization Technology option to Enabled b Click Back 6 On the System Setup page select Device Settings 7 On the Device Settings page select Port 1 for the Cavium QLogic adapter 8 On the Device Level Configuration page Figure 12 7 a Set the Virtualization Mode to SR IOV b Click Back Figure 12 7 Integrated Devices 9 On the Main Configuration Page click Finish save your...

Page 248: ...I iommu A successful input output memory management unit IOMMU command output should show for example Intel IOMMU enabled d To view VF details number of VFs and total VFs issue the following command find sys grep I sriov 11 For a specific port enable a quantity of VFs a Issue the following command to enable for example 8 VFs on PCI instance 04 00 0 bus 4 device 0 function 0 root ah rh68 echo 8 sys...

Page 249: ... different on the PFs 8070 in this example versus the VFs 8090 in this example Figure 12 9 Command Output for sriov_numvfs 12 To view a list of all PF and VF interfaces issue the following command ip link show grep i vf b2 Figure 12 10 shows example output Figure 12 10 Command Output for ip link show Command 13 Assign and verify MAC addresses a To assign a MAC address to the VF issue the following...

Page 250: ...ress 14 Power off the VM and attach the VF Some OSs support hot plugging of VFs to the VM a In the Virtual Machine dialog box Figure 12 11 click Add Hardware Figure 12 11 RHEL68 Virtual Machine b In the left pane of the Add New Virtual Hardware dialog box Figure 12 12 click PCI Host Device c In the right pane select a host device d Click Finish ...

Page 251: ...he host and the VM 17 As needed add more VFs in the VM Configuring SR IOV on VMware To configure SR IOV on VMware 1 Access the server BIOS System Setup and then click System BIOS Settings 2 On the System BIOS Settings page click Integrated Devices 3 On the Integrated Devices page a Set the SR IOV Global Enable option to Enabled b Click Back 4 In the System Setup window click Device Settings 5 On t...

Page 252: ... Network controller QLogic Corp QLogic FastLinQ QL45xxx 10 25 GbE Ethernet Adapter vmnic7 0000 05 02 0 Network controller QLogic Corp QLogic FastLinQ QL45xxx Series 10 25 GbE Controller SR IOV VF PF_0 5 0_VF_0 0000 05 02 1 Network controller QLogic Corp QLogic FastLinQ QL45xxx Series 10 25 GbE Controller SR IOV VF PF_0 5 0_VF_1 0000 05 02 2 Network controller QLogic Corp QLogic FastLinQ QL45xxx Se...

Page 253: ..._14 0000 05 0f 7 Network controller QLogic Corp QLogic FastLinQ QL45xxx Series 10 25 GbE Controller SR IOV VF PF_0 5 1_VF_15 13 Attach VFs to the VM as follows a Power off the VM and attach the VF Some OSs support hot plugging of VFs to the VM b Add a host to a VMware vCenter Server Virtual Appliance vCSA c Click Edit Settings of the VM 14 Complete the Edit Settings dialog box Figure 12 13 as foll...

Page 254: ...C0154501 00 P Figure 12 13 VMware Host Edit Settings 15 To validate the VFs per port issue the esxcli command as follows root localhost esxcli network sriovnic vf list n vmnic6 VF ID Active PCI Address Owner World ID 0 true 005 02 0 60591 1 true 005 02 1 60591 ...

Page 255: ...2 false 005 03 4 13 false 005 03 5 14 false 005 03 6 15 false 005 03 7 16 Install the Cavium QLogic drivers for the adapters detected in the VM Use the latest drivers available from your vendor for your host OS do not use inbox drivers The same driver version must be installed on the host and the VM 17 Power on the VM and then issue the ifconfig a command to verify that the added network interface...

Page 256: ...ptions described in this chapter apply to Ethernet based RDMA protocols including RoCE and iWARP The development of NVMe oF with RDMA is defined by a technical sub group of the NVMe organization This chapter demonstrates how to configure NVMe oF for a simple network The example network comprises the following Two servers an initiator and a target The target server is equipped with a PCIe SSD drive...

Page 257: ...VMe oF Network The NVMe oF configuration process covers the following procedures Installing Device Drivers on Both Servers Configuring the Target Server Configuring the Initiator Server Preconditioning the Target Server Testing the NVMe oF Devices Optimizing Performance 45000 Series Adapter 45000 Series Adapter ...

Page 258: ...d libraries by issuing the following commands yum groupinstall Infiniband Support yum install tcl devel libibverbs devel libnl devel glib2 devel libudev devel lsscsi perftest yum install gcc make git ctags ncurses ncurses devel openssl openssl devel elfutils libelf devel c To ensure that NVMe OFED support is in the selected OS kernel issue the following command make menuconfig d Under Device Drive...

Page 259: ... command finishes before executing the next command To configure the target service 1 Load target modules Issue the following commands after each server reboot modprobe qedr modprobe nvmet modprobe nvmet rdma lsmod grep nvme confirm that the modules are loaded 2 Create the target subsystem NVMe Qualified Name NQN with the name indicated by nvme subsystem name Use the NVMe oF specifications for exa...

Page 260: ...vice path using the lsblk command This system has two NVMe devices nvme0n1 and nvme1n1 echo 1 namespaces 1 enable Enables the namespace mkdir sys kernel config nvmet ports 1 cd sys kernel config nvmet ports 1 Creates NVMe port 1 echo 1 1 1 1 addr_traddr Sets the same IP address For example 1 1 1 1 is the IP address for the target port of the 45000 Series Adapter echo rdma addr_trtype Sets the tran...

Page 261: ...ecuting the next command To configure the initiator server 1 Load the NVMe modules Issue these commands after each server reboot modprobe qedr modprobe nvme rdma 2 Download compile and install the nvme cli initiator utility Issue these commands at the first configuration you do not need to issue these commands after each reboot git clone https github com linux nvme nvme cli git cd nvme cli make ma...

Page 262: ...r NVMe target servers that are tested out of the box show a higher than expected performance Before running a benchmark the target server needs to be prefilled or preconditioned To precondition the target server 1 Secure erase the target server with vendor specific tools similar to formatting This test example uses an Intel NVMe SSD device which requires the Intel Data Center Tool that is availabl...

Page 263: ... oF device Issue the following command fio filename dev nvme0n1 direct 1 time_based rw randread refill_buffers norandommap randrepeat 0 ioengine libaio bs 4k iodepth 1 numjobs 1 runtime 60 group_reporting name temp out FIO reports two latency types submission and completion Submission latency slat measures application to kernel latency Completion latency clat measures end to end kernel latency The...

Page 264: ...d for reads only randwrite for writes only or randrw for reads and writes Optimizing Performance To optimize performance on both initiator and target servers 1 Configure the following system BIOS settings Power Profiles Max Performance or equivalent ALL C States Disabled Hyperthreading Disabled 2 Configure the Linux kernel parameters by editing the grub file etc default grub a Add parameters to en...

Page 265: ...fin sh The following script sets the IRQ affinity bin bash RSS affinity setup script input the device name ethX OFFSET 0 0 1 0 1 2 0 1 2 3 FACTOR 1 2 3 4 OFFSET 0 FACTOR 1 LASTCPU cat proc cpuinfo grep processor tail n1 cut d f2 MAXCPUID echo 2 LASTCPU p dc OFFSET echo 2 OFFSET p dc FACTOR echo 2 FACTOR p dc CPUID 1 for eth in do NUM grep eth proc interrupts wc l NUM_FP NUM INT grep m 1 eth proc i...

Page 266: ...r for CPUFREQ in sys devices system cpu cpu cpufreq scaling_governor do f CPUFREQ continue echo n performance CPUFREQ done cat sys devices system cpu cpu0 cpufreq scaling_governor To configure the network or memory settings sysctl w net ipv4 tcp_mem 16777216 16777216 16777216 sysctl w net ipv4 tcp_wmem 4096 65536 16777216 sysctl w net ipv4 tcp_rmem 4096 87380 16777216 sysctl w net core wmem_max 16...

Page 267: ... Configuration with RDMA Optimizing Performance 240 BC0154501 00 P echo 0 sys block nvme0n1 queue add_random echo 2 sys block nvme0n1 queue nomerges NOTE The following commands apply only to the initiator server ...

Page 268: ... Kernel Provider Interface NDKPI Mode 2 host virtual network adapters host virtual NICs support RDMA RoCE configuration procedures in this section include Creating a Hyper V Virtual Switch with an RDMA NIC Adding a vLAN ID to Host Virtual NIC Verifying If RoCE is Enabled Adding Host Virtual NICs Virtual Ports Mapping the SMB Drive and Running RoCE Traffic NOTE For NDKPI Mode 2 you must enable Netw...

Page 269: ...nterfaces set the value of the NetworkDirect Functionality parameter to Enabled 2 Launch Hyper V Manager 3 Click Virtual Switch Manager see Figure 14 1 Figure 14 1 Enabling RDMA in Host Virtual NIC 4 Create a virtual switch 5 Select the Allow management operating system to share this network adapter check box In Windows Server 2016 a new parameter Network Direct RDMA is added in the Host virtual N...

Page 270: ... V Virtual Ethernet Adapter Properties 4 To enable RDMA issue the following Windows PowerShell command PS C Users Administrator Enable NetAdapterRdma vEthernet New Virtual Switch PS C Users Administrator Adding a vLAN ID to Host Virtual NIC To add a vLAN ID to a host virtual NIC 1 To find the host virtual NIC name issue the following Windows PowerShell command PS C Users Administrator Get VMNetwor...

Page 271: ... To verify if the RoCE is enabled Issue the following Windows PowerShell command Get NetAdapterRdma Command output lists the RDMA supported adapters as shown in Figure 14 4 Figure 14 4 Windows PowerShell Command Get NetAdapterRdma NOTE Note the following about adding a vLAN ID to a host virtual NIC A vLAN ID must be assigned to a host virtual NIC The same vLAN ID must be assigned to ports on the s...

Page 272: ...hown in To enable RDMA in a host virtual NIC on page 242 3 To assign a vLAN ID to the virtual port issue the following command Set VMNetworkAdapterVlan VMNetworkAdapterName SMB VlanId 5 Access ManagementOS Mapping the SMB Drive and Running RoCE Traffic To map the SMB drive and run the RoCE traffic 1 Launch the Performance Monitor Perfmon 2 Complete the Add Counters dialog box Figure 14 5 as follow...

Page 273: ...016 Configuring RoCE Interfaces with Hyper V 246 BC0154501 00 P Figure 14 5 Add Counters Dialog Box If the RoCE traffic is running counters appear as shown in Figure 14 6 Figure 14 6 Performance Monitor Shows RoCE Traffic ...

Page 274: ...d on a team SET member network adapters must all be installed in the same physical Hyper V host RoCE over SET procedures included in this section Creating a Hyper V Virtual Switch with SET and RDMA Virtual NICs Enabling RDMA on SET Assigning a vLAN ID on SET Running RDMA Traffic on SET Creating a Hyper V Virtual Switch with SET and RDMA Virtual NICs To create a Hyper V virtual switch with SET and ...

Page 275: ...n VMNetworkAdapterName SET VlanId 5 Access ManagementOS Running RDMA Traffic on SET For information about running RDMA traffic on SET go to https technet microsoft com en us library mt403349 aspx Configuring QoS for RoCE The two methods of configuring quality of service QoS include Configuring QoS by Disabling DCBX on the Adapter Configuring QoS by Enabling DCBX on the Adapter NOTE Note the follow...

Page 276: ...CBX 1 Disable DCBX on the adapter 2 Using HII set the RoCE Priority to 0 3 To install the DCB role in the host issue the following Windows PowerShell command PS C Users Administrators Install WindowsFeature Data Center Bridging 4 To set the DCBX Willing mode to False issue the following Windows PowerShell command PS C Users Administrators set NetQosDcbxSetting Willing 0 5 Enable QoS in the minipor...

Page 277: ...ble QoS 6 Assign the vLAN ID to the interface as follows a Open the miniport properties and then click the Advanced tab b On the adapter properties Advanced page Figure 14 10 under Property select VLAN ID and then set the value c Click OK NOTE The preceding step is required for priority flow control PFC ...

Page 278: ...tors Enable NetQoSFlowControl Priority 5 8 To disable priority flow control on any other priority issue the following commands PS C Users Administrator Disable NetQosFlowControl 0 1 2 3 4 6 7 PS C Users Administrator Get NetQosFlowControl Priority Enabled PolicySet IfIndex IfAlias 0 False Global 1 False Global 2 False Global NOTE If configuring RoCE over Hyper V do not assign a vLAN ID to the phys...

Page 279: ...on TCP PriorityValue8021Action 0 Policystore ActiveStore PS C Users Administrator Get NetQosPolicy PolicyStore activestore Name tcp Owner PowerShell WMI NetworkProfile All Precedence 127 JobObject IPProtocol TCP PriorityValue 0 Name smb Owner PowerShell WMI NetworkProfile All Precedence 127 JobObject NetDirectPort 445 PriorityValue 5 10 To configure ETS for all traffic classes defined in the previ...

Page 280: ...S PFC 4 4 4 4 4 4 OperationalTrafficClasses TC TSA Bandwidth Priorities 0 ETS 20 1 4 6 7 1 ETS 50 5 2 ETS 30 0 OperationalFlowControl Priority 5 Enabled OperationalClassifications Protocol Port Type Priority Default 0 NetDirect 445 5 12 Create a startup script to make the settings persistent across the system reboots 13 Run RDMA traffic and verify as described in RoCE Configuration on page 127 Con...

Page 281: ...ers Administrators set NetQosDcbxSetting Willing 1 5 Enable QoS in the miniport properties as follows a On the adapter properties Advanced page Figure 14 11 under Property select Quality of Service and then set the value to Enabled b Click OK NOTE If the switch does not have a way of designating the RoCE traffic you may need to set the RoCE Priority to the number used by the switch Arista switches...

Page 282: ... Advanced Properties Enabling QoS 6 Assign the vLAN ID to the interface required for PFC as follows a Open the miniport properties and then click the Advanced tab b On the adapter properties Advanced page Figure 14 12 under Property select VLAN ID and then set the value c Click OK ...

Page 283: ...ors Get NetAdapterQoS Name Ethernet 5 Enabled True Capabilities Hardware Current MacSecBypass NotSupported NotSupported DcbxSupport CEE CEE NumTCs Max ETS PFC 4 4 4 4 4 4 OperationalTrafficClasses TC TSA Bandwidth Priorities 0 ETS 5 0 4 6 7 1 ETS 95 5 OperationalFlowControl Priority 5 Enabled OperationalClassifications Protocol Port Type Priority NetDirect 445 5 RemoteTrafficClasses TC TSA Bandwid...

Page 284: ...adapters in the VM Default and Maximum VMMQ Virtual NIC Enabling and Disabling VMMQ on a Management NIC Monitoring Traffic Statistics NOTE The preceding example is taken when the adapter port is connected to an Arista 7060X switch In this example the switch PFC is enabled on Priority 5 RoCE App TLVs are defined The two traffic classes are defined as TC0 and TC1 where TC1 is defined for RoCE DCBX P...

Page 285: ...then set the value to Enabled 3 Click OK Figure 14 13 Advanced Properties Enabling Virtual Switch RSS Creating a Virtual Machine Switch with or Without SR IOV To create a virtual machine switch with or without SR IOV 1 Launch the Hyper V Manager 2 Select Virtual Switch Manager see Figure 14 14 3 In the Name box type a name for the virtual switch 4 Under Connection type a Click External network b S...

Page 286: ...Virtual Switch Manager 5 Click OK Enabling VMMQ on the Virtual Machine Switch To enable VMMQ on the virtual machine switch Issue the following Windows PowerShell command PS C Users Administrators Set VMSwitch name q1 defaultqueuevmmqenabled true defaultqueuevmmqqueuepairs 4 ...

Page 287: ...l command PS C Users Administrator Get VMSwitch Name ql fl Figure 14 15 shows example output Figure 14 15 Windows PowerShell Command Get VMSwitch Creating a VM and Enabling VMMQ on VMNetworkadapters in the VM To create a virtual machine VM and enable VMMQ on VMNetworksadapters in the VM 1 Create a VM 2 Add the VMNetworkadapter to the VM 3 Assign a virtual switch to the VMNetworkadapter ...

Page 288: ...et 3 64 0 0 62 Activated Unknown 1 Ethernet 3 65 0 0 63 Activated Unknown 1 Ethernet 3 66 00 15 5D 36 0A 04 0 16 PF Activated Adaptive 4 Ethernet 3 67 00 15 5D 36 0A 05 1 0 PF Activated Adaptive 4 Ethernet 3 68 00 15 5D 36 0A 06 0 0 PF Activated Adaptive 4 Name ID MacAddress VID ProcMask FID State ITR QPairs Ethernet 3 69 00 15 5D 36 0A 07 0 8 PF Activated Adaptive 4 Ethernet 3 70 00 15 5D 36 0A 0...

Page 289: ...fy the maximum default queue use the VMswitch capability Enabling and Disabling VMMQ on a Management NIC To enable or disable VMMQ on a management NIC To enable VMMQ on a management NIC issue the following command PS C Users Administrator Set VMNetworkAdapter ManagementOS vmmqEnabled true The Management OS VNIC has four VMMQs To disable VMMQ on a management NIC issue the following command PS C Use...

Page 290: ... Offload Figure 14 16 Advanced Properties Enabling VXLAN 3 Set the Value to Enabled 4 Click OK Deploying a Software Defined Network To take advantage of VXLAN encapsulation task offload on virtual machines you must deploy a Software Defined Networking SDN stack that utilizes a Microsoft Network Controller For more details refer to the following Microsoft TechNet link on Software Defined Networking...

Page 291: ... storage systems with local storage For more information refer to the following Microsoft TechNet link https technet microsoft com en us windows server docs storage storage spaces storage spaces direct windows server 2016 Configuring the Hardware Figure 14 17 shows an example of hardware configuration Figure 14 17 Example Hardware Configuration NOTE The disks used in this example are 4 400G NVMe a...

Page 292: ...Windows Server roles Hyper V 3 Install the following features Failover Cluster Data center bridging DCB 4 Connect the nodes to a domain and add domain accounts Configuring the Network To deploy Storage Spaces Direct the Hyper V switch must be deployed with RDMA enabled host virtual NICs To configure the network on each server 1 Configure the physical network switch as follows a Connect all adapter...

Page 293: ...s and to then enable SET issue the following command New VMSwitch Name SETswitch NetAdapterName port1 port2 port3 port4 EnableEmbeddedTeaming true c To add host virtual NICs to the virtual switch issue the following commands Add VMNetworkAdapter SwitchName SETswitch Name SMB_1 managementOS Add VMNetworkAdapter SwitchName SETswitch Name SMB_2 managementOS NOTE Network Quality of Service is used to ...

Page 294: ...e NetAdapter vEthernet SMB_1 Disable NetAdapter vEthernet SMB_2 Enable NetAdapter vEthernet SMB_2 g To enable RDMA on the host virtual NIC adapters issue the following command Enable NetAdapterRdma SMB1 SMB2 h To verify RDMA capabilities issue the following command Get SmbClientNetworkInterface where RdmaCapable EQ true Configuring Storage Spaces Direct Configuring Storage Spaces Direct in Windows...

Page 295: ...hem before enabling Storage Spaces Direct Otherwise they will not be included in the Storage Spaces Direct storage pool Step 3 Configuring a Cluster Witness You should configure a witness for the cluster so that this four node system can withstand two nodes failing or being offline With these systems you can configure file share witness or cloud witness For more information go to https blogs msdn ...

Page 296: ...eOEM Confirm false _ Set Disk isreadonly true _ Set Disk isoffline true Get Disk Number ne null IsBoot ne true IsSystem ne true PartitionStyle eq RAW Group NoElement Property FriendlyName Sort Property PsComputerName Count Step 5 Enabling Storage Spaces Direct After creating the cluster issue the Enable ClusterStorageSpacesDirect Windows PowerShell cmdlet The cmdlet places the storage system into ...

Page 297: ... the storage pool New Volume StoragePoolFriendlyName S2D FriendlyName VirtualDiskName FileSystem CSVFS_ReFS StorageTierfriendlyNames Capacity Performance StorageTierSizes Size of capacity tier in size units example 800GB Size of Performance tier in size units example 80GB CimSession ClusterName Step 7 Creating or Deploying Virtual Machines You can provision the virtual machines onto the nodes of t...

Page 298: ...n page 273 TC MQPRIO Offload on page 273 Requirements Traffic control offload requires An upstream kernel with the CONFIG_NET_CLS_FLOWER m kernel set The latest OSs including SLES 12 SP3 and Ubuntu Optional The latest TC or IP utility recommended if you encounter problems with inbox utilities Current Support The traffic control offload feature currently supports the following Ingress packet redire...

Page 299: ... l2 fwd offload on ethtool K p5p1 hw tc offload on To create and delete offloaded MAC vLAN devices use the IP utility as follows ip link add link p5p1 name mvlan_1 type macvlan ifconfig mvlan_1 up ip link delete mvlan_1 To add and delete filters for MAC vLAN devices issue commands in the tc utility as follows tc qdisc add dev p5p1 ingress A one time job to add ingress Qdisc until you delete it Whe...

Page 300: ... tc filter add dev p5p1 protocol ip parent ffff pref 0x2 flower skip_sw dst_ip 192 168 50 100 action drop TC MQPRIO Offload You can provide a priority to a traffic control map to enable packets with corresponding priorities to use those traffic classes on which to transmit packets Cavium supports a maximum of four different traffic classes For example ethtool K ethx hw tc offload on tc qdisc delet...

Page 301: ...eries Adapter or running it in your system Inspect all cables and connections Verify that the cable connections at the network adapter and the switch are attached properly Verify the adapter installation by reviewing Installing the Adapter on page 9 Ensure that the adapter is properly seated in the slot Check for specific hardware problems such as obvious damage to board components or the PCI edge...

Page 302: ... drivers are loaded for your Windows Linux or VMware system Verifying Drivers in Windows See the Device Manager to view vital information about the adapter link status and network connectivity Verifying Drivers in Linux To verify that the qed ko driver is loaded properly issue the following command lsmod grep i module name If the driver is loaded the output of this command shows the size of the dr...

Page 303: ... Driver qede x x x x Verifying Drivers in VMware To verify that the VMware ESXi drivers are loaded issue the following command esxcli software vib list Testing Network Connectivity This section provides procedures for testing network connectivity in Windows and Linux environments Testing Network Connectivity for Windows Test network connectivity using the ping command To determine if the network c...

Page 304: ...SSS Set both ports on the 45000 Series Adapter to the same speed both 10Gbps or both 25Gbps Microsoft Virtualization with Hyper V Microsoft Virtualization is a hypervisor virtualization system for Windows Server 2012 R2 For more information on Hyper V go to https technet microsoft com en us library Dn282278 aspx Linux specific Issues Problem Errors appear when compiling driver source code Solution...

Page 305: ...t Windows OS installer reports is shown at the bottom of the Windows Setup dialog box as shown in Figure 16 1 Figure 16 1 Windows Setup Error Message In other cases the error message may indicate a need to ensure that the disk s controller is enabled in the computer s BIOS menu To avoid any of the depicted error messages ensure that you do not attach a USB flash drive until after the setup asks fo...

Page 306: ...01 00 P Collecting Debug Data Use the commands in Table 16 1 to collect debug data Table 16 1 Collecting Debug Data Commands Debug Data Description demesg T Kernel logs ethtool d Register dump sys_info sh System information available in the driver bundle ...

Page 307: ...ors for the state of the adapter port link and activity Table A 1 Adapter Port Link and Activity LEDs Port LED LED Appearance Network State Link LED Off No link cable disconnected Continuously illuminated Link Activity LED Off No port activity Blinking Port activity ...

Page 308: ...8024 Specific form factor compliance is as follows SFPs SFF8472 for memory map SFF8419 or SFF8431 low speed signals and power Quad small form factor pluggable QSFP SFF8636 for memory map SFF8679 or SFF8436 low speed signals and power Optical modules electrical input output active copper cables ACC and active optical cables AOC 10G SFF8431 limiting interface 25G IEEE 802 3by Annex 109B 25GAUI does ...

Page 309: ...45000 Series Adapters Cavium has tested the components listed in Table B 1 and presents this list for your convenience This list is based on cable and optics components that are available at the time of product release and is subject to change over time as new components enter the market or are discontinued To view the most current list of supported cables and optical modules view the Cavium FastL...

Page 310: ...win axial 10G 1 3 5 HP 2074260 2 AP784A AP820A SFP Twin axial 10G SFP Twin axial 10G SFP Twin axial 10G 1 3 5 25G DAC Amphenol NDCCGF0001 NDCCGF0003 NDCCGJ0003 SFP28 to SFP28 SFP28 to SFP28 SFP28 to SFP28 1 3 3 HP 844471 B21 844474 B21 844477 B21 SFP28 to SFP28 SFP28 to SFP28 SFP28 to SFP28 0 5 1 3 40G DAC HPE FCI 845408 B21 10137498 4050LF Dual mode 40 100G QSFP40GB to QSFP40GB 5 LuXshare ICT LP9...

Page 311: ...FP100GB to QSFP100GB QSFP100GB to QSFP100GB QSFP100GB to QSFP100GB 1 1 3 3 5 5 Arista CAB Q 4S 100G 3M Rev 1 QSFP100GB to QSFP100GB 3 Dell 03CC35 Rev A00 QSFP100GB to QSFP100GB 3 HPE 845402 B21 845404 B21 845406 B21 845408 B21 QSFP100GB to QSFP100GB QSFP100GB to QSFP100GB QSFP100GB to QSFP100GB QSFP100GB to QSFP100GB 0 5 1 3 5 Table B 1 Tested Cables and Optical Modules Continued Speed Form Factor...

Page 312: ...4050LF QSFP100GB to 4XS FP28GB 5 Optical Solutions 10G Optical Transceiver Avago AFBR 703SMZ AFBR 701SDZ SFP 10G Optical Transceiver SR SFP 10G Optical Transceiver LR N A N A Finisar FTLX8571D3BCL QL FTLX1471D3BCL QL SFP 10G Optical Transceiver SR SFP 10G Optical Transceiver LR N A N A 25G Optical Transceiver Finisar FTLF8536P4BCL FTLF8538P4BCL SFP28 Optical Transceiver SR SFP28 Optical Trans ceiv...

Page 313: ...M FTLC1151SDPL 100G SR 4 Optical Transceiver 100GBASE LR4 WDMc N A 10Km Sumitomo SQF1002L4LNGG01P 100GBASE LR4 WDM QSFP28 10Km 10G AOCd Dell 470 ABLV 470 ABLZ 470 ABLT 470 ABML 470 ABLU 470 ABMD 470 ABMJ SFP 10G AOC SFP 10G AOC SFP 10G AOC SFP 10G AOC SFP 10G AOC SFP 10G AOC SFP 10G AOC 2 3 5 7 10 15 15 25G AOC InnoLighte TF PY003 N00 TF PY020 N00 SFP28 AOC SFP28 AOC 3 20 a Cable length is indicat...

Page 314: ...nge over time as new switches enter the market or are discontinued To view the most current list of supported switches view the Cavium FastLinQ 45000 Series Interoperability Matrix located here LineCards Cavium_FastLinQ_45000_Series_Interoperability_Matrix pdf Table B 2 Switches Tested for Interoperability Manufacturer Ethernet Switch Model Arista 7060X 7160 Cisco Nexus 3132 Nexus 3232C Nexus 5548...

Page 315: ...g either HII or Cavium QLogic management tools configuration of the storage protocol on another PF is disallowed by those management tools Because storage personality is disabled by default only the personality that has been configured using HII or Cavium QLogic management tools is written in NVRAM configuration When this limitation is removed users can configure additional PFs on the same port fo...

Page 316: ...nd disabled However RDMANICModeOnPartition is currently set to disabled and cannot be enabled NIC and SAN Boot to Base Is Supported Only on Select PFs Ethernet and PXE boot are currently supported only on PF0 and PF1 In NPAR configuration other PFs do not support Ethernet and PXE boot When the Virtualization Mode is set to NPAR non offloaded FCoE boot is supported on Partition 2 PF2 and PF3 and iS...

Page 317: ...apter board Advanced Configuration and Power Interface See ACPI bandwidth A measure of the volume of data that can be transmitted at a specific transmission rate A 1Gbps or 2Gbps Fibre Channel port can transmit or receive at nominal rates of 1 or 2Gbps depending on the device to which it is connected This corre sponds to actual bandwidth values of 106MB and 212MB respectively BAR Base address regi...

Page 318: ...management and LAN connec tivity DCB enables 802 1 bridges to be used for the deployment of a converged network where all applications can be run over a single physical infrastructure DCBX Data center bridging exchange A protocol used by DCB devices to exchange config uration information with directly connected peers The protocol may also be used for misconfiguration detection and for configu rati...

Page 319: ...smits information between computers typically at speeds of 10 and 100 million bits per second Mbps ETS Enhanced transmission selection A standard that specifies the enhancement of transmission selection to support the allocation of bandwidth among traffic classes When the offered load in a traffic class does not use its allocated bandwidth enhanced transmission selection allows other traffic class...

Page 320: ...ancement of technology related to electricity Internet Protocol See IP Internet small computer system interface See iSCSI Internet wide area RDMA protocol See iWARP IP Internet protocol A method by which data is sent from one computer to another over the Internet IP specifies the format of packets also called datagrams and the addressing scheme IQN iSCSI qualified name iSCSI node name based on the...

Page 321: ...nit See MTU message signaled interrupts See MSI MSI X MSI MSI X Message signaled interrupts One of two PCI defined extensions to support message signaled interrupts MSIs in PCI 2 2 and later and PCI Express MSIs are an alternative way of generating an interrupt through special messages that allow emulation of a pin assertion or deassertion MSI X defined in PCI 3 0 allows a device to allocate any n...

Page 322: ... setting priorities and allocating bandwidth quality of service See QoS PF Physical function RDMA Remote direct memory access The ability for one node to write directly to the memory of another with address and size semantics over a network This capability is an important feature of VI networks reduced instruction set computer See RISC remote direct memory access See RDMA RISC Reduced instruction ...

Page 323: ...target A target can contain many LUNs A target is a device that responds to a requested by an initiator the host system Peripherals are targets but for some commands for example a SCSI COPY command the peripheral may act as an initiator TCP Transmission control protocol A set of rules to send data in packets over the Internet protocol TCP IP Transmission control protocol Internet protocol Basic co...

Page 324: ...ical area network See vLAN virtual machine See VM virtual port See vPort vLAN Virtual logical area network LAN A group of hosts with a common set of require ments that communicate as if they were attached to the same wire regardless of their physical location Although a vLAN has the same attributes as a physical LAN it allows for end stations to be grouped together even if they are not located on ...

Page 325: ... configuring device settings 34 FCoE boot type on 115 link and activity LEDs 280 link status confirming 31 server VLAN ID 134 Add Counters dialog box 138 245 adding host VNIC 245 VLAN ID to host VNIC 243 address MAC permanent and virtual 37 PCI 37 ADK downloading Windows 116 Advanced Configuration and Power Interface See ACPI 290 affinity settings IRQ 192 agency certifications xxv algorithm parame...

Page 326: ...282 case submitting for Tech Support xxiii Cavium FastLinQ 41000 Series Interoperability Matrix accessing 287 Cavium FastLinQ 45000 Series Interoperability Matrix accessing 282 Cavium FastLinQ error counters 140 Cavium Technical Support xxiii CEE DCBX protocol 129 DCBX protocol configuring 133 protocol supported 129 protocol DCBX 133 CentOS host requirement 8 Linux drivers installing 17 OS support...

Page 327: ...R IOV on Linux 220 SR IOV on VMware 224 SR IOV on Windows 213 Storage Spaces Direct 264 target server for NVMe oF 232 VXLAN 262 Z9100 switch 131 connections DAC SerDes interface 2 inspecting 274 L2 verifying 149 network verifying 277 RDMA SMB Direct 137 RoCE verifying 149 constraints on features in this release 288 controller power management options 23 conventions documentation xxi converged enha...

Page 328: ...definition of 291 drivers installing on servers for NVMe oF 231 FCoE verifying in Linux 211 ID 37 device continued name 37 NVMe oF testing 236 Device Manage verifying Windows driver 275 DHCP 75 definition of 291 dynamic iSCSI boot 74 enabling on interface for SLES 16 IP address assigning 147 iSCSI configuration 45 70 iSCSI general parameters 69 75 82 known issue with configuration 92 Option 17 75 ...

Page 329: ...thout RoCE 13 RoCE inbox OFED 17 source RPM 15 TAR file Linux 16 VMware 25 Windows 21 driver removal Linux non RoCE 14 Linux RoCE 14 VMware 29 Windows 21 drop action support 273 DSCP PFC configuring 165 dynamic host configuration protocol See DHCP E eCore definition of 291 qed ko Linux kernel module 205 209 EFI definition of 292 driver version 38 EMC requirements xxv EMI requirements xxv enable_vx...

Page 330: ...ding xxiii upgrading on Linux 31 upgrading on Windows 31 version for L2B 38 flow control default Linux 18 default VMware 28 troubleshooting Linux issue 277 FTP definition of 293 network installation 13 functional description of adapter 1 G GID index values VLAN 150 VLAN interfaces configuring RoCE 143 global bandwidth allocation 47 global ID See GID glossary terms and definitions 290 guide documen...

Page 331: ...nux 16 drivers VMware 25 drivers Windows 21 hardware 7 Institute of Electrical and Electronics Engineers See IEEE intended audience of guide xix interfaces configuring RoCE 241 Internet Protocol definition of 293 Internet small computer system interface See iSCSI Internet wide area RDMA protocol See iWARP interoperability switches 287 IP definition of 293 checksum offloads support 2 IPv4 standards...

Page 332: ... Boot 19 kmod RPM package 16 knowledgebase information xxiv L L2B firmware version 38 large send offload See LRO large send offload See LSO laser safety xxiv Layer 2 definition of 294 VLAN segregation 71 80 82 LED port state indicators 280 legal notices xxiv Lenovo servers NPAR mode issue with 288 libfc ko in Linux FCoE offload 209 libfcoe ko in Linux FCoE offload 209 libiscsi ko in Linux iSCSI of...

Page 333: ...ts See MSI MSI X messages Linux driver 19 MFW version 38 Microsoft Virtualization with Hyper V 277 migrating from software iSCSI installation to offload iSCSI 93 minimum bandwidth allocating 47 models supported adapters xviii modules tested optical 282 MokManager importing public key with 20 monitoring traffic statistics 262 MPIO Linux considerations for iSCSI boot 100 MQPRIO traffic control offlo...

Page 334: ...erver configuring 232 target server preconditioning 235 NVRAM boot images loading FCoE 115 boot images loading iSCSI 204 definition of 295 feature support 3 O OFED limitations 128 OS support 127 OFED continued preparing for RHEL 144 preparing for SLES 144 preparing for Ubuntu 145 working with qedr driver 12 offload checksum support 2 iSCSI FAQs 203 iSCSI migrating to 93 Linux FCoE configuring 209 ...

Page 335: ...23 PowerShell Windows verifying RDMA 136 preboot configuration 33 precautions safety 8 preconditioning target server 235 preface guide introduction xviii preinstallation checklist 9 priority PFC 131 RoCE 129 priority based flow control See PFC product functional description 1 overview of 1 product safety compliance xxvi public key importing for Secure Boot 19 PVRDMA configuring 159 Q QCC GUI adapt...

Page 336: ...load native 24 on Ubuntu host 145 packages for Ubuntu 145 running traffic on SET 248 services Linux 147 starting 148 user space applications 17 RDMA continued verifying 169 virtual switch creating Hyper V 242 RDMA over Converged Ethernet See RoCE RDMA Core support on Linux 177 redirection ingress packet 272 reduced instruction set computer See RISC registering for support account xxiii related doc...

Page 337: ... for DAC transceiver 2 serializer deserializer See SerDes server initiator configuring for NVMe oF 234 performance optimizing 237 target configuring for NVMe oF 232 server message block See SMB SET creating virtual switch with 247 defined 247 procedures 247 RDMA traffic running 248 VLAN ID assigning 248 SFF8024 compliant specifications supported 281 SFP Flow Control troubleshooting Linux issue 277...

Page 338: ...raffic control offload TCP checksum offloads support 2 definition of 296 TCP segmentation offload See TSO TCs assigning to tx queues 134 technical support contacting xxiii downloading updates and documentation xxiii knowledgebase xxiv terms defined 290 tested cables and optical modules 282 tested switches 287 testing network connectivity Linux 277 Windows 276 testing NVMe of devices 236 TLV defini...

Page 339: ...ware xxiii upgrading adapter firmware on Linux 31 adapter firmware on Windows 31 uplinks assign 160 user datagram protocol See UDP V VCCI Class A certification xxvi vCenter Plug In for QCC GUI 4 verbose level 27 verifying FCoE devices in Linux 211 iSCSI interfaces in Linux 206 network connection 277 Network Direct MTU size for RoCE 135 RDMA 169 RoCE configuration on Linux 147 VMware driver 276 VF ...

Page 340: ... qedf 29 driver installing 25 driver iSCSI qedil 29 driver removing 29 drivers and packages 24 drivers downloading 25 drivers verifying current 276 ESX configuring RoCE on adapter 156 ESXi driver packages 24 image files injecting 123 VMware continued minimum host OS requirements 8 NIC driver parameters optional 26 SR IOV configuring 224 VMware Update Manager installing driver with 26 VNIC adding V...

Page 341: ...boot installation 204 iSCSI offload configuring 196 minimum host OS requirements 8 RoCE configuring 135 Windows Server R2 Microsoft Virtualization with Hyper V 277 Windows Setup dialog box installation error message 278 wnfwnx2 utility firmware upgrade utility for Windows 30 WoL definition of 297 VMware driver parameter 28 Z Z9100 Ethernet switch configuring 131 ...

Page 342: ...tent s Pending Products identified in this document may be covered by one or more Marvell patents and or patent applications This document is provided for informational purposes only and may contain errors Marvell reserves the right without notice to make changes to this document or in product design or specifications Marvell disclaims any warranty of any kind expressed or implied and does not gua...

Reviews:

No comments

Related manuals for FastLinQ 45000 Series

X-Micro XBT-PACX Quick Start Manual preview
XBT-PACX
Brand: X-Micro Pages: 6
D-Link 32-bit PCI Fast Ethernet Network Adapter... Quick Installation Manual preview
32-bit PCI Fast Ethernet Network Adapter...
Brand: D-Link Pages: 16
Radiant Solutions Therma-Cord TC-2000 Quick Start Manual preview
Therma-Cord TC-2000
Brand: Radiant Solutions Pages: 2
Oakton 91100-85 User Manual preview
91100-85
Brand: Oakton Pages: 9
Future Domain TMC-8S0M User Manual preview
TMC-8S0M
Brand: Future Domain Pages: 48
AMEC NK-80 Quick Start Manual preview
NK-80
Brand: AMEC Pages: 4
Planet WNL-U553 Quick Installation Manual preview
WNL-U553
Brand: Planet Pages: 2
Hama 121976 User Manual preview
121976
Brand: Hama Pages: 4
LSI PCI to USB Host Adapter 60800 User Manual preview
PCI to USB Host Adapter 60800
Brand: LSI Pages: 10
Nvidia MCX4121A-ACAT User Manual preview
MCX4121A-ACAT
Brand: Nvidia Pages: 62
Daikin BRP069C81 Installation Manual preview
BRP069C81
Brand: Daikin Pages: 8
Enercell Enercell 273-331 User Manual preview
Enercell 273-331
Brand: Enercell Pages: 1
Devolo dLAN duo Quick Start Manual preview
dLAN duo
Brand: Devolo Pages: 2
HQ P.SUP.SMP36-HQ Manual preview
P.SUP.SMP36-HQ
Brand: HQ Pages: 30
Bernafon Phone Adapter Installation Manual preview
Phone Adapter
Brand: Bernafon Pages: 20
Axial DSU-09M Manual preview
DSU-09M
Brand: Axial Pages: 6
3idee hp-z Assembly Instructions Manual preview
hp-z
Brand: 3idee Pages: 7
Accell UltraAV DisplayPort Setup Instructions preview
UltraAV DisplayPort
Brand: Accell Pages: 2

Brands by name

Popular brands

Load more brands