Qlogic QL45212 User Manual Download Page 66

Page: 66 / 112

BC0154503-00 C

Configuring RoCE

This chapter describes RoCE (v1 and v2) configuration on the QL45212 Adapter,
the Ethernet switch, and the Windows or Linux host. This chapter describes the
following topics:



Supported Operating Systems and OFED



Planning



Preparing the Adapter



Preparing the Ethernet Switch



Configuring RoCE on the Adapter for Windows Server



Configuring RoCE on the Adapter for Linux

Summary of Contents for QL45212

Page 1: ...BC0154503 00 C Third party information brought to you courtesy of Dell EMC User s Guide 25Gb Intelligent Ethernet Adapter QL45212...

Page 2: ...page xiii Added RoCE v1 and v2 Under Performance Features TCP segmentation offload added virtual machine multiqueues VMMQ Features on page 2 Added Applicable IEEE802 3 Ethernet Standards Specificatio...

Page 3: ...RoCE environ ment on page 12 Added new Step 1 Type the following command to get the path to the currently installed drivers mod info drivername Deleted reference to modprobe r qede modprobe r qed and...

Page 4: ...n Table 3 2 on page 17 Added See Step 7 on Verifying the RoCE Configu ration on Linux on page 59 for more information Statistics on page 18 Updated reference from click View Log to click View Installa...

Page 5: ...dated RoCE priority value from 4 to 5 on Step 2 Preparing the Ethernet Switch on page 51 Updated adapter priority from 4 to 5 on Step 3 Also updated code reference from 00001000 to 00000100 on Step 3...

Page 6: ...n figuring for Linux RoCE v2 Configuring for Linux on page 59 Added reference to CentOS Added reference to SLES 12 iSCSI Extensions for RDMA on page 69 Deleted reference to modprobe r qed and modprobe...

Page 7: ...EMI and EMC Requirements xvii KCC Class A xviii Product Safety Compliance xviii 1 Product Overview Functional Description 2 Features 2 Adaptive Interrupt Frequency 3 ASIC with Embedded RISC Processor...

Page 8: ...date Package in the GUI 19 Installation Options 24 Examples 25 Removing the Windows Drivers 25 Managing Adapter Properties 26 Setting Power Management Options 26 VMware Driver Software 27 Installing V...

Page 9: ...figuration for RHEL 64 RoCE Configuration for SLES 65 Verifying the RoCE Configuration on Linux 65 VLAN Interfaces and GID Index Values 68 7 iSCSI Extensions for RDMA Configuring iSER for RHEL 70 Conf...

Page 10: ...User s Guide 25Gb Intelligent Ethernet Adapter QL45212 x BC0154503 00 C A Adapter LEDS B Dell Z9100 Switch Configuration Glossary Index...

Page 11: ...3 6 InstallShield Wizard Ready to Install the Program Window 23 3 7 InstallShield Wizard Completed Window 23 3 8 Dell Update Package Window 24 3 9 Power Management 26 4 1 Dell Update Package Splash Sc...

Page 12: ...c QL45212 Linux Drivers 10 3 2 qede Driver Optional Parameters 17 3 3 Linux Driver Parameter Defaults 17 3 4 VMware Driver Optional Parameters 28 3 5 VMware Driver Parameter Defaults 30 5 1 Adapter Pr...

Page 13: ...roduct Overview provides a product functional description a list of features a list of supported operating systems and the adapter specifications Chapter 2 Installing the Hardware describes how to ins...

Page 14: ...death Text in blue font indicates a hyperlink jump to a figure table or section in this guide and links to websites are shown in underlined blue For example Table 9 2 lists problems related to the us...

Page 15: ...ement What are shortcut keys To enter the date type mm dd yyyy where mm is the month dd is the day and yyyy is the year Topic titles between quotation marks identify related topics either within this...

Page 16: ...el of laser product Class 1 Laser Product Appareil laser de classe 1 Produkt der Laser Klasse 1 Luokan 1 Laserlaite Caution Class 1 laser radiation when open Do not view directly with optical instrume...

Page 17: ...e received including interference that may cause undesired operation ICES 003 Compliance Class A This Class A digital apparatus complies with Canadian ICES 003 Cet appareil num riqu de la classe A est...

Page 18: ...UL60950 1 2nd Edition 2007 03 3 27 UL CSA C22 2 60950 1 07 2nd Edition Use only with listed ITE or equivalent Complies with 21 CFR 1040 10 and 1040 11 2006 95 EC low voltage directive TUV EN60950 1 20...

Page 19: ...1 BC0154503 00 C 1 Product Overview This chapter provides information about the following topics Functional Description Features Adapter Specifications...

Page 20: ...lution 25Gb MAC SerDes interface for direct attach copper DAC transceiver connection PCI Express Gen3 x8 Zero copy capable hardware Performance features TCP IP UDP checksum offloads TCP segmentation o...

Page 21: ...t Virtualization Adaptive Interrupt Frequency The adapter driver intelligently adjusts host interrupt frequency based on traffic conditions to increase overall application throughput When traffic is l...

Page 22: ...a full height or a low profile bracket for use in a standard PCIe slot Standards Specifications PCI Express Base Specification rev 3 0 PCI Express Card Electromechanical Specification rev 3 0 PCI Bus...

Page 23: ...5 BC0154503 00 C 2 Installing the Hardware This chapter provides information about the following topics System Requirements Safety Precautions Preinstallation Checklist Installing the Adapter...

Page 24: ...Architecture IA 32 or EMT64 that meets operating system require ments Memory 8GB RAM minimum Direct Attach Cables DAC Amphenol NDDCCGF 0001 Amphenol NDAQGF 0003 Amphenol NDCCGF 0005 Leoni ParaLink LA0...

Page 25: ...a damaged adapter WARNING The adapter is being installed in a system that operates with voltages that can be lethal Before you open the case of your system observe the following precautions to protect...

Page 26: ...ater width slot x8 in a x16 but a greater width adapter cannot be seated into a lesser width slot x8 in a x4 If you do not know how to identify a PCIe slot refer to your system documentation 3 Remove...

Page 27: ...9 BC0154503 00 C 3 Installing Drivers This chapter provides information about the following topics Linux Driver Software Windows Driver Software VMware Driver Software...

Page 28: ...Linux Driver Description qed The Linux core module manages all PCI device resources registers host interface queues and so on The qed core module directly controls the firmware handles interrupts and...

Page 29: ...ers without RoCE To install the Linux drivers without RoCE do the following 1 Download the QL45212 Adapter Linux drivers from dell support com 2 Remove the existing Linux drivers as described in Remov...

Page 30: ...ing command to get the path to the currently installed drivers modinfo driver name 2 Unload and remove the Linux drivers If the Linux drivers were installed using an RPM package type the following com...

Page 31: ...move the Linux drivers modprobe r qedr 3 Remove driver module files If the drivers were installed using an RPM package type the following command rpm e qlgc fastlinq kmp default version arch If the dr...

Page 32: ...For SLES cd usr src packages rpmbuild bb SPECS fastlinq version spec 3 Install the newly compiled RPM rpm ivh RPMS arch qlgc fastlinq version arch rpm The drivers will be installed in the following p...

Page 33: ...using the TAR file 1 Create a directory and extract the TAR files to the directory tar xjvf fastlinq minor version tar bz2 2 Change to the recently created directory and then install the drivers cd fa...

Page 34: ...he Linux Drivers on page 12 4 Install the new Linux drivers using one of the following methods Installing Linux Drivers Using the src RPM Package Installing Linux Drivers Using the kmp kmod RPM Packag...

Page 35: ...ce HW err_flags_override A bitmap for disabling or forcing the actions taken in case of a HW error bit 31 an enable bit for this bitmask bit 0 prevent HW attentions from being reasserted bit 1 collect...

Page 36: ...istics and configuration information can be viewed using the ethtool utility See the ethtool man page for more information Also see Step 7 on Verifying the RoCE Configuration on Linux on page 65 for m...

Page 37: ...ick the icon representing the Dell update package file 2 In the Dell Update Package window Figure 3 1 click Install Figure 3 1 Dell Update Package Window 3 In the QLogic Super Installer InstallShield...

Page 38: ...mplete the following in the wizard s License Agreement window Step 3 3 a Read the QLogic End User Software License Agreement b To continue select I accept the terms in the license agreement c Click Ne...

Page 39: ...ow Figure 3 4 as follows a Select one of the following setup types Click Complete to install all program features Click Custom to manually select the features to be installed b To continue click Next...

Page 40: ...select one of the following options This feature will be installed on the local hard drive This setting marks the feature for installation without affecting any of its subfeatures This feature and all...

Page 41: ...he InstallShield Wizard installs the QLogic Adapter drivers and Management Software Installer Figure 3 6 InstallShield Wizard Ready to Install the Program Window 8 When the installation is complete th...

Page 42: ...installed versions any error messages and other information about the installation To close the Update Package window click CLOSE Figure 3 8 Dell Update Package Window Installation Options To customi...

Page 43: ...xe s To extract the update contents to the C mydir directory DUP_file_name exe s e C mydir To extract the driver components to the C mydir directory DUP_file_name exe s drivers C mydir To install only...

Page 44: ...computer If the device is busy doing something servicing a call for example the operating system will not shut down the device The operating system attempts to shut down every possible device only whe...

Page 45: ...and then extract the vib file 4 Copy the vib file to the ESX server You can place the file anywhere that is accessible to the ESX console shell Type the following command to use the Linux scp utility...

Page 46: ...or receive fully formed packets hw_vlan 1 is the default num_queues The num_queues parameter specifies the number of Tx Rx queue pairs num_queues can be 1 11 or one of the follow ing 1 Allow the drive...

Page 47: ...The auto_fw_reset parameter enables 1 or disables 0 the driver automatic firmware recovery capability When this parameter is enabled the driver attempts to recover from events such as transmit timeou...

Page 48: ...the following command vmkload_mod u qedentv Table 3 5 VMware Driver Parameter Defaults Parameter Default Speed Auto negotiation with speed advertised Flow Control Auto negotiation with Rx and Tx adve...

Page 49: ...ne Updating Dell DUP Using BIN File The firmware Dell update package is a Flash update utility only it is not used for adapter configuration You run the firmware Dell update package by double clicking...

Page 50: ...re Dell update package by double clicking the executable file follow these instructions 1 Double click the icon representing the firmware Dell update package file The Dell update package splash screen...

Page 51: ...ading Firmware Run by Double Clicking 33 BC0154503 00 C 2 Follow the on screen instructions Click Yes to continue the installation as shown in Figure 4 2 Figure 4 2 Continue Dell Update Package Instal...

Page 52: ...4 Upgrading Firmware Run by Double Clicking 34 BC0154503 00 C 3 The installer indicates that it is loading the new firmware as shown in Figure 4 3 Figure 4 3 Loading New Firmware...

Page 53: ...03 00 C 4 When complete the installer indicates the result of the installation as shown in Figure 4 4 Click Yes to reboot Figure 4 4 Result of Installation 5 Click Finish to complete the installation...

Page 54: ...line with no options specified results in the same behavior as double clicking the Dell update package icon Note that the actual file name of the Dell update package will vary C Network_Firmware_2T12N...

Page 55: ...ently installed version Software application name BCM57810 10 Gigabit Ethernet rev 10 p2p1 Package version 08 07 26 Installed version 08 07 26 BCM57810 10 Gigabit Ethernet rev 10 p2p2 The version of t...

Page 56: ...onfiguration This chapter provides information about the following topics Getting Started Displaying Firmware Image Properties Device Level Configuration NIC Configuration Data Center Bridging Configu...

Page 57: ...rmware Image Properties Device Level Configuration NIC Configuration Data Center Bridging Configuration NIC Partitioning Configuration To start the HII application 1 Open the System Setup page for you...

Page 58: ...tarted 40 BC0154503 00 C 3 In the Device Settings page select the QL45212 adapter port that you want to configure 4 The Main Configuration Page presents the adapter management options as shown in Figu...

Page 59: ...image version which comprises several firmware images Controller BIOS Version management firmware version EFI Version Unified extensible firmware interface UEFI extensible firmware interface EFI driv...

Page 60: ...Device Level Configuration Page To enable NIC partitioning NPAR single root input output virtualization SR IOV or both 1 Select Device Level Configuration in the Main Configuration Page Figure 5 1 an...

Page 61: ...hanges will take effect after a system reset NOTE The QL45212 Adapter supports 8 virtual functions per port with a total of 16 virtual functions per adapter If the NPAReP is enabled the QL45212 Adapte...

Page 62: ...is not set to Auto Negotiated and the link speed is set to 25G Table 5 3 shows the FEC Mode options 3 Click Back 4 When prompted click Yes to save the changes Changes will take effect after a system r...

Page 63: ...he Main Configuration Page Figure 5 1 and then click Finish 2 In the Data Center Bridging DCB Settings page click the DCBX Protocol drop down and choose from the following options Disabled Disables DC...

Page 64: ...t to allocate bandwidth There are four partitions per port if NPAReP mode is disabled eight partitions per port if NPAReP mode is enabled NPAReP mode requires that ARI be enabled in the BIOS for adapt...

Page 65: ...can be 1 100 The per partition maximum TX bandwidth value applies regardless of the DCBX ETS mode setting If the maximum TX bandwidth for a partition is set to 0 the effective bandwidth is one percen...

Page 66: ...45212 Adapter the Ethernet switch and the Windows or Linux host This chapter describes the following topics Supported Operating Systems and OFED Planning Preparing the Adapter Preparing the Ethernet S...

Page 67: ...stem and the perftest application have different OFED versions To confirm the perftest OFED version type the following command ib_send_bw version qedr and libqedr RPM packages are required for in box...

Page 68: ...idging DCB Settings and then click Finish 2 In the Data Center Bridging DCB Settings window click the DCBX Protocol drop down The QL45212 Adapter supports both CEE and IEEE protocols This value should...

Page 69: ...2 Configure quality of service QoS class maps setting the RoCE priority to match the adapter 5 switch config class map type qos class roce switch config match cos 5 3 Configure queuing class maps swi...

Page 70: ...net x x switch config switchport mode trunk switch config switchport trunk allowed vlan 1 5 Dell Z9100 Ethernet Switch Configuring the Dell Z9100 Ethernet Switch for RoCE comprises configuring a DCB m...

Page 71: ...BW BW COMMITTED BW PEAK TSA Rate Mbps Burst KB Rate Mbps Burst KB 0 0 1 2 3 4 6 7 40 ETS 1 5 60 ETS 2 3 Dell Conf do show interfaces twentyFiveGigE 1 5 1 pfc detail Interface twentyFiveGigE 1 5 1 Admi...

Page 72: ...dapter 5 Dell conf interface vlan 5 Dell conf if vl 5 tagged twentyFiveGigE 1 8 1 Configuring RoCE on the Adapter for Windows Server Configuring RoCE on the adapter for Windows Server comprises enabli...

Page 73: ...3 00 C 2 Select Network Direct Mtu Size from the Property menu The network direct MTU size must be less than the Ethernet jumbo packet size In this example select 1024 Click OK 3 Select RoCE Mode from...

Page 74: ...Port 2 QLogic FastLinQ QL45212 True 5 Verify that RoCE is enabled on the host operating system using PowerShell The Get NetOffloadGlobalSetting command shows NetworkDirect is enabled PS C Users Admini...

Page 75: ...ter for Windows Server 57 BC0154503 00 C c Select Local computer under Select counters from computer Select Add RDMA Activity and then select All instances under Instances of selected object The follo...

Page 76: ...192 168 10 20 16159 192 168 10 10 445 0 Kernel 60 Connection 192 168 10 20 15903 192 168 10 10 445 0 Kernel 56 Listener fe80 e11d 9ab5 a47d 4f0a 56 445 NA 0 Kernel 56 Listener 192 168 11 20 445 NA 0 K...

Page 77: ...s To find RoCE v1 and v2 specific GIDs use either sys or class parameters or run RoCE scripts from the 45000 Series FastLinQ source package To check the default RoCE GID Index and address run the ibv_...

Page 78: ...0000 020e 1eff fec4 1b20 v1 p4p1 qedr0 1 1 fe80 0000 0000 0000 020e 1eff fec4 1b20 v2 p4p1 qedr0 1 2 0000 0000 0000 0000 0000 ffff 1e01 010a 30 1 1 10 v1 p4p1 qedr0 1 3 0000 0000 0000 0000 0000 ffff 1...

Page 79: ...end_bw d qedr0 F x 1 192 168 100 3 Verifying RoCE v2 through Different Subnets Use the following procedure to verify RoCE v2 through different subnets 1 Set the route configuration for the Server and...

Page 80: ...switch settings using the following procedure Use any flow control method Pause DCBX CEE DCBX IEEE and enable IP routing for RoCE v2 See Preparing the Ethernet Switch on page 51 for RoCE v2 configura...

Page 81: ...v2 Settings for RDMA_CM Applications Use the following scripts from fastlinq minor source package show_rdma_cm_roce_ver sh qedr0 is configured to IB RoCE v1 qedr1 is configured to IB RoCE v1 config_r...

Page 82: ...EL ISO image libibverbs devel x x x x86_64 rpm required for libqedr library perftest x x x x86_64 rpm required for IB bandwidth and latency applications or using Yum install the in box OFED yum groupi...

Page 83: ...oCE Configuration on Linux After installing OFED installing the Linux driver and loading the RoCE drivers verify that the RoCE devices were detected on all Linux operating systems 1 Stop firewall tabl...

Page 84: ...niBand 0 fw_ver 8 3 9 0 node_guid 020e 1eff fe50 c7c0 sys_image_guid 020e 1eff fe50 c7c0 vendor_id 0x1077 vendor_part_id 5684 hw_ver 0x0 phys_port_cnt 1 port 1 state PORT_ACTIVE 1 max_mtu 4096 5 activ...

Page 85: ...570 8192000 bytes in 0 05 seconds 1436 97 Mbit sec 1000 iters in 0 05 seconds 45 61 usec iter Client root lambodar ibv_rc_pingpong d qedr0 g 0 192 168 10 165 local address LID 0x0000 QPN 0xff0000 PSN...

Page 86: ...be used for any perftest applications ibv_devinfo d qedr0 v grep GID GID 0 fe80 0000 0000 0000 020e 1eff fe50 c5b0 GID 1 0000 0000 0000 0000 0000 ffff c0a8 0103 GID 2 2001 0db1 0000 0000 020e 1eff fe5...

Page 87: ...INBOX OFED for following operating systems RHEL 7 1 7 2 SLES 12 and 12 SP1 CentOS 7 2 Out of box OFED does not support iSER target configuration and no ib_isert module is available for latest out of...

Page 88: ...ing Linux Drivers in a RoCE Environment on page 13 3 Install the latest FastLinQ driver and libqedr packages as described in Installing the Linux Drivers with RoCE on page 16 4 Load the RDMA services...

Page 89: ...ean true on the applicable portals The portal instances are identified as iser as shown in the following figure 9 Install Linux iSCSI Initiator Utilities using the yum install iscsi initiator utils co...

Page 90: ...following iscsiadm m node l p 192 168 100 99 3260 T iqn 2015 06 test target1 d Confirm that the Iface Transport is iser in the target connection as shown in the following figure Type a command such a...

Page 91: ...ch rpm python netifaces 0 8 6 55 x86_64 rpm python rtslib 2 2 6 6 noarch rpm python urwid 1 1 1 6 144 x86_64 rpm targetcli 2 1 3 8 x86_64 rpm 2 Before starting the TargetCLI load all RoCE device drive...

Page 92: ...ormance mode by typing the following command cat sys devices system cpu cpu cpufreq scaling_governor Set Kernel sysctl Settings Set the kernel sysctl settings as follows sysctl w net ipv4 tcp_mem 4194...

Page 93: ...ance systemctl stop irqbalance cat proc interrupts grep qedr Shows IRQ assigned to each port queue echo 1 proc irq XX smp_affinity_list echo 2 proc irq YY smp_affinity_list echo 4 proc irq ZZ smp_affi...

Page 94: ...his chapter provides information about the following topics Troubleshooting Checklist Verifying that Current Drivers are Loaded Testing Network Connectivity Microsoft Virtualization with Hyper V Linux...

Page 95: ...settings and change them if they are in conflict with another device Verify that your server is using the latest BIOS Try inserting the adapter in another slot If the new position works the original s...

Page 96: ...currently loaded driver is the correct version modinfo qede Or you can use the following command root test1 ethtool i eth2 driver qede version 8 4 7 0 firmware version mfw 8 4 7 0 storm 8 4 7 0 bus in...

Page 97: ...e displayed indicate whether the network connection is working or not The adapter link speed can be forced to 10Gbps or 25Gbps using the following methods QConvergeConsole GUI Network Adapter Advanced...

Page 98: ...hypervisor virtualization system for Windows Server 2012 R2 For more information on Hyper V see https technet microsoft com en us library Dn282278 aspx Linux Problem QL45212 devices with SFP Flow Cont...

Page 99: ...Collecting Debug Data Use the information in table x x to collect debug data Table 8 1 Collecting Debug Data Commands Debug Data Description demesg T Kernel logs ethtool d Register dump sys_info sh S...

Page 100: ...nk and Activity LEDs located adjacent to the port connector as described in Table A 1 Table A 1 Port LED Indications Port LED LED Appearance Network State Link LED Off No link cable discon nected Cont...

Page 101: ...t be explicitly configured to connect to the adapter at 25Gbps To configure a Dell Z9100 switch port to connect to the QL45212 Adapter at 25Gbps 1 Establish a serial port connection between your manag...

Page 102: ...10G speed 25G Quad port mode with 25G speed Dell conf stack unit 1 port 5 portmode quad speed 25G For information about changing the adapter link speed see Testing Network Connectivity on page 79 5 V...

Page 103: ...onf if tf 1 5 1 do show run interface tw 1 5 1 interface twentyFiveGigE 1 5 1 no ip address mtu 9416 switchport flowcontrol rx on tx on no shutdown no intf type cr4 autoneg For more information about...

Page 104: ...ns to satisfy the require ments of protocols and applications in the data center Because existing high perfor mance data centers typically comprise multiple application specific networks that run on d...

Page 105: ...ads human interface infrastructure HII A specification part of UEFI 2 1 for managing user input localized strings fonts and forms that allows OEMs to develop graphical interfaces for preboot configura...

Page 106: ...the memory of another with address and size semantics over a network This capability is an important feature of VI networks serializer deserializer SerDes A pair of functional blocks commonly used in...

Page 107: ...e Tx Microseconds 17 connection manager 49 converged enhanced Ethernet See CEE D data center bridging exchange See DCBX DCB map 52 54 DCBX 45 enable 50 protocol 54 debug parameter 17 29 Dell update pa...

Page 108: ...rupt 3 G GID 68 global ID See GID gro_disable parameter 17 H hardware installation 6 HII 39 50 human interface infrastructure See HII hw_vlan parameter 28 30 Hyper V 80 I IEEE protocol 45 in box OFED...

Page 109: ...58 PF 28 PFC 52 physical characteristics 4 physical function See PF policy map 51 port speed 44 power management 26 PowerShell Windows 56 priority PFC 52 RoCE 50 priority based flow control See PFC pr...

Page 110: ...m requirements 6 T TAR file 15 TCP segmentation offload See TSO TPA 28 transmit send offload See TSO troubleshooting 76 TSO 17 29 30 Tx Ring Size 17 30 U UEFI 41 Unified Extensible Firmware Interface...

Page 111: ......

Page 112: ...Shell are registered trademarks of Microsoft Corporation Linux is a registered trademark of Linus Torvalds VMware is a registered trademark of VMware Inc RHEL and CentOS are registered trademarks of R...

Search results

Summary of Contents for QL45212

Reviews:

Related manuals for QL45212

Brands by name

Popular brands

Qlogic QL45212, User Manual

Search results

Summary of Contents for QL45212

Reviews:

Related manuals for QL45212

Brands by name

Popular brands