manualshive.com logo in svg

Mellanox Technologies 0NHYP5, User Manual

Share
Download
Mellanox Technologies 0NHYP5 User Manual Download Page 3

ConnectX-4 100Gb/s Ethernet Dual Port Adapter Card User Manual for Dell PowerEdge

Rev 1.2

Mellanox Technologies

3

Table of Contents

Table of Contents  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
List of Tables   . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
About this Manual  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Intended Audience  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   9
Related Documentations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   9
Document Conventions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   10
Technical Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   10
Firmware Updates. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   10

Chapter 1  Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

1.1  Product Overview  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.2  Features and Benefits  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.3  Operating Systems/Distributions   . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.4  Connectivity  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Chapter 2  Interfaces  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.1  Ethernet QSFP28 Interface  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2  PCI Express Interface  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.3  LED Interface  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Chapter 3  Hardware Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.1  System Requirements  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.1.1  Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   16

3.2  Operating Systems/Distributions   . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.2.1  Software Stacks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   16

3.3  Safety Precautions   . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.4  Pre-installation Checklist  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.5  Bracket Installation Instructions  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.5.1  Removing the Existing Bracket  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   17
3.5.2  Installing the New Bracket . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   17

3.6  Card Installation Instructions  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.7  Cables and Modules  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.7.1  Cable Installation  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   18

3.8  Adapter Card Un-installation Instructions  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.8.1  Safety Precautions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   19
3.8.2  Card Un-installation  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   20

3.9  Identify the Card in Your System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3.9.1  On Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   20
3.9.2  On Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   21

Chapter 4  Driver Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

4.1  Linux Driver  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

4.1.1  Hardware and Software Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   23

Summary of Contents for 0NHYP5

Page 1: ...www mellanox com ConnectX 4 100Gb s Ethernet Dual QSFP28 Adapter Card User Manual for Dell PowerEdge P N 0NHYP5 0XR0K2 Rev 1 2...

Page 2: ...OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLI...

Page 3: ...I Express Interface 15 2 3 LED Interface 15 Chapter 3 Hardware Installation 16 3 1 System Requirements 16 3 1 1 Hardware 16 3 2 Operating Systems Distributions 16 3 2 1 Software Stacks 16 3 3 Safety P...

Page 4: ...ng the Device Online 33 4 1 7 2 Updating the Device Manually 33 4 1 7 3 Updating the Device Firmware Automatically upon System Boot 33 4 1 8 UEFI Secure Boot 34 4 1 8 1 Enrolling Mellanox s x 509 Publ...

Page 5: ...wing and Dimensions 55 7 5 Bracket Mechanical Drawings 56 Appendix A Interface Connectors Pinout 57 A 1 QSFP28 Connector Pinout 57 A 2 PCI Express x16 Edge Connector Pinout 58 Appendix B Finding the M...

Page 6: ...ion History Table 8 Table 2 Documents List 9 Table 3 Dual port Ethernet Adapter Cards 12 Table 4 Features 13 Table 5 Hardware and Software Requirements 23 Table 6 Hardware and Software Requirements 36...

Page 7: ...Installation Results Example 43 Figure 3 Mechanical Drawing of the Single port x16 Adapter Card 55 Figure 4 Mechanical Drawing of the Dual port x16 Adapter Card 55 Figure 5 Dual port Low Profile Brack...

Page 8: ...pdated Linux Driver on page 23 Updated Windows Driver on page 36 Updated VMware Driver on page 47 Updated 0NHYP5 0XR0K2 Specifications on page 53 Updated Airflow Specifications on page 54 Updated Adap...

Page 9: ...mlxup Firmware Utility Mellanox OFED for Linux User Manual Document no 2877 User Manual describing OFED features performance InfiniBand diagnostic tools content and configuration See http www mella no...

Page 10: ...who purchased Mellanox products through a Mellanox approved reseller should first seek assistance through their reseller Firmware Updates The Mellanox support downloader contains software firmware and...

Page 11: ...cuit device These adapters connectivity provide the highest performing and most flexible interconnect solution for PCI Express Gen3 servers used in Enterprise Data Centers High Performance Computing a...

Page 12: ...ed Each adapter comes with two bracket heights short and tall Table 3 Dual port Ethernet Adapter Cards Ordering Part Number OPN 0XR0K2 short bracket 0NHYP5 tall bracket Data Transmission Rate Ethernet...

Page 13: ...verlay networks that carry traffic from individual virtual machines over logical tunnels in encapsulated formats such as NVGRE and VXLAN While this solves network scalability issues it hides the TCP p...

Page 14: ...r latency and SLA Hardware based I O Virtualization ConnectX 4 provides dedicated adapter resources and guaranteed isolation and protection for virtual machines within the server Storage Acceleration...

Page 15: ...eatures on page 13 Ethernet traffic is transmitted through the cards QSFP28 connectors 2 2 PCI Express Interface The ConnectX 4 adapter card supports PCI Express 3 0 1 1 and 2 0 compatible through an...

Page 16: ...stated above 2 Shut down your system if active 3 After shutting down the system turn off power and unplug the cord 4 Remove the card from its package Please note that the card must be placed on an ant...

Page 17: ...lbf in torque on the screws 3 6 Card Installation Instructions 1 Before installing the card make sure that the system is off and the power cord is not con nected to the server Please follow proper el...

Page 18: ...ligned with the slot opening and the adapter faceplate is visible against the system chassis 6 Secure the adapter with the adapter clips or screws 7 Close the system case 3 7 Cables and Modules To obt...

Page 19: ...ons on page 54 4 After plugging in a cable lock the connector using the latching mechanism particular to the cable vendor When data is being transferred the Green LED will blink See Section 7 3 Adapte...

Page 20: ...nisms on the bracket clips or screws 4 Holding the adapter card from its center gently pull the adapter card from the PCI Express slot 5 When the port connectors reach the top of the chassis window ge...

Page 21: ...Value display box check the fields VEN and DEV fields are separated by In the display example above notice the sub string PCI VEN_15B3 DEV_1003 VEN is equal to 0x15B3 this is the Vendor ID of Mellanox...

Page 22: ...onnectX 4 100Gb s Ethernet Dual Port Adapter Card User Manual for Dell PowerEdge Rev 1 2 Mellanox Technologies 22 lspci grep i Mellanox Network controller Mellanox Technologies MT4115 Family ConnectX...

Page 23: ...arch iso You can download it from http www mellanox com Products Soft ware Ethernet Drivers Table 5 Hardware and Software Requirements Requirements Description Platforms A server platform with an ada...

Page 24: ...nodes and then copy it to a shared file system such as NFS To install on all the cluster nodes use cluster aware tools such as pdsh If your kernel version does not match with any of the offered pre b...

Page 25: ...machine as root Step 2 Mount the ISO image on your machine Step 3 Run the installation script MLNX_OFED_LINUX x x x rhel6 3 x86_64 mlnx_add_kernel_support sh m tmp MLNX_OFED_ LINUX x x x rhel6 3 x86_6...

Page 26: ...installation such as prefix kernel version and installation parameters can be retrieved by running the command etc infiniband info Most of the Mellanox OFED components can be configured or reconfigure...

Page 27: ...the opt directory openshmem bupc fca and ibutils The kernel modules are installed under lib modules uname r updates on SLES and Fedora Distributions lib modules uname r extra mlnx ofa_kernel on RHEL a...

Page 28: ...allation machine as root Step 2 Mount the ISO image on your machine and copy its content to a shared location in your net work You can download it from http www mellanox com Products Software Ether ne...

Page 29: ...nox 100 1 354 K s in 0s 2014 04 20 13 52 30 247 MB s RPM GPG KEY Mellanox saved 1354 1354 sudo rpm import RPM GPG KEY Mellanox warning rpmts_HdrFromFdno Header V3 DSA SHA1 Signature key ID 6224c050 NO...

Page 30: ...a vpi installer package with KMP support mlnx ofed all Installs all available packages in MLNX_OFED mlnx ofed basic Installs basic packages required for running Mellanox cards mlnx ofed guest Installs...

Page 31: ...e of installation is applicable to Debian and Ubuntu Operating Systems 4 1 6 1 Setting up MLNX_OFED apt get Repository Step 1 Log into the installation machine as root Step 2 Extract the MLNX_OFED pac...

Page 32: ...4g 09FCC269 2013 08 11 sudo apt get update apt cache search mlnx ofed mlnx ofed vma eth MLNX_OFED vma eth installer package with DKMS support mlnx ofed hpc MLNX_OFED hpc installer package with DKMS su...

Page 33: ...the firmware Step 4 Reboot your machine after the firmware burning is completed 4 1 7 3 Updating the Device Firmware Automatically upon System Boot As of MLNX_OFED v3 1 x x x firmware can be automati...

Page 34: ...onto the system key ring by the kernel Follow these steps below to add the Mellanox s x 509 public key to your system Step 1 Download the x 509 public key Step 2 Add the public key to the MOK list us...

Page 35: ...each boot for the first module that either has no signature or whose key is not in the kernel key ring So it s much easier to miss this mes sage You won t see it on repeated tests where you unload an...

Page 36: ...n 2 Open a CMD console Click Task Manager File Run new task and enter CMD 3 Enter the following command On an x64 64 bit machine the output will be AMD64 Step 2 Go to the Mellanox WinOF 2 web page at...

Page 37: ...equires no user intervention 4 2 3 1 Attended Installation The following is an example of an installation session Step 1 Double click the exe and follow the GUI instructions to install MLNX_WinOF2 Ste...

Page 38: ...Gb s Ethernet Dual Port Adapter Card User Manual for Dell PowerEdge Rev 1 2 Mellanox Technologies 38 Step 5 Read then accept the license agreement and click Next Step 6 Select the target folder for th...

Page 39: ...following cases If the user has an OEM card In this case the firmware will not be displayed If the user has a standard Mellanox card with an older firmware version the firmware will be updated accord...

Page 40: ...ver and installed automatically Performances tools install the performance tools that are used to measure performance in user envi ronment Analyze tools install the tools that can be used either to di...

Page 41: ...t Adapter Card User Manual for Dell PowerEdge Rev 1 2 Mellanox Technologies 41 b Click Install to start the installation Step 9 In case firmware upgrade option was checked in Step 7 you will be notifi...

Page 42: ...s option Step 4 Optional if you want to control whether to install ND provider or not1 Step 5 Optional If you do not wish to upgrade your firmware version2 If no reboot options are specified the insta...

Page 43: ...F2 Drivers OS To see the Mellanox network adapters display the Device Manager and pull down the Net work adapters menu Figure 2 Installation Results Example 4 2 5 Extracting Files Without Running Inst...

Page 44: ...ual for Dell PowerEdge Rev 1 2 Mellanox Technologies 44 To extract only the driver files Step 3 Click Next to create a server image Step 4 Click Change and specify the location in which the files are...

Page 45: ...Click Install to extract this folder or click Change to install to a different folder Step 6 To complete the extraction click Finish 4 2 6 Uninstalling Mellanox WinOF 2 Driver 4 2 6 1 Attended Uninst...

Page 46: ...ill be automatically updated as part of the WinOF 2 package installation For information on how to upgrade firmware manually please refer to MFT User Manual www mellanox com Products Ethernet Drivers...

Page 47: ...r was installed successfully Table 7 Hardware and Software Requirements Requirements Description Platforms A server platform with an adapter card based on one of the following Mellanox Technologies HC...

Page 48: ...0 from the Mellanox Firmware Tools MFT site ESX 5 5 File mft 4 3 0 25 10EM 550 0 0 1391871 x86_64 vib MD5SUM 62f953b77f4ea49c420ff4e1ce8b925b ESX 6 0 File mft 4 3 0 25 10EM 600 0 0 2768847 x86_64 vib...

Page 49: ...ard User Manual for Dell PowerEdge Rev 1 2 Mellanox Technologies 49 2 Install the image according to the steps described in the MFT User Manual The following procedure requires custom boot image downl...

Page 50: ...n are available from http www mellanox com Products Software Firmware Tools 5 1 Firmware Update Example server1 mlxup Querying Mellanox devices firmware Device Type ConnectX4 Part Number 0XR0K2 Descri...

Page 51: ...adapter in its slot or a different slot if necessary Try using another cable Reinstall the drivers for the network driver files may be damaged or deleted Reboot the server Adapters stopped working af...

Page 52: ...ibdev2netdev Card Detection lspci grep i Mellanox Mellanox Firmware Tool MFT Download and install MFT http www mellanox com content pages php pg management_tools menu_section 34 Refer to the User Manu...

Page 53: ...G R EC 10GBASE SR 10GBASE LR 10GBASE ER 10GBASE CX4 SGMII 1000BASE X Data Rate 10 25 40 50 100 Gb s Ethernet PCI Express Gen3 SERDES 8 0GT s 16 lanes 2 0 and 1 1 compatible Power and Environmental Vol...

Page 54: ...vide additional cooling For information on how to adjust the fan speed on Dell PowerEdge servers please refer to the Dell iDRAC User s Guide for that server Please contact your Dell account team for g...

Page 55: ...llanox Technologies 55 7 4 Board Mechanical Drawing and Dimensions Figure 3 Mechanical Drawing of the Single port x16 Adapter Card Figure 4 Mechanical Drawing of the Dual port x16 Adapter Card All dim...

Page 56: ...2 Mellanox Technologies 56 7 5 Bracket Mechanical Drawings Figure 5 Dual port Low Profile Bracket Figure 6 Dual port Full Height Bracket Port 1 Link and Activity Bi color LED Port 2 Link and Activity...

Page 57: ...Connector Pinout Figure 7 Connector and Cage Views Table 10 Connector Pin Number and Name to Signal Name Map Connector Pin Number Connector Pin Name Port A Signal Name 1 GND GND 2 TXN_2 Tx2n 3 TXP_2 T...

Page 58: ...rding to the PCI Express 3 0 specification 16 GND GND 17 RXP_1 Rx1p 18 RXN_1 Rx1n 19 GND GND 20 GND GND 21 RXN_2 Rx2n 22 RXP_2 Rx2p 23 GND GND 24 RXN_4 Rx4n 25 RXP_4 Rx4p 26 GND GND 27 ModPrsl_Port0 M...

Page 59: ...ConnectX 4 100Gb s Ethernet Dual Port Adapter Card User Manual for Dell PowerEdge Rev 1 2 Mellanox Technologies 59 Figure 8 PCIe x16 Connector Pinout...

Page 60: ...C and Serial Number on the Adapter Card Each Mellanox adapter card has a different identifier printed on the label serial number and the card MAC for the Ethernet protocol Figure 9 0NHYP5 0XR0K2 Board...

Page 61: ...ConnectX 4 100Gb s Ethernet Dual Port Adapter Card User Manual for Dell PowerEdge Rev 1 2 Mellanox Technologies 61...

Page 62: ...proper air flow allow at least 8cm 3 inches of clearance around the ven tilation openings During periods of lightning activity do not work on the equipment or connect or dis connect cables Some coppe...

Page 63: ...Dual Port Adapter Card User Manual for Dell PowerEdge Rev 1 2 Mellanox Technologies 63 CLASS 1 LASER PRODUCT and reference to the most recent laser standards IEC 60 825 1 1993 A1 1997 A2 2001 and EN...

Page 64: ...air de 200LFM cette temp rature ambiante maximale est n cessaire En outre pour garantir un bon coulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des ouver tures de ventilation P...

Page 65: ...65 8 Exposition au rayonnement grave Mise en garde l utilisation de commandes ou de r glages ou l ex cution de proc dures autres que ce qui est sp cifi dans les pr sentes peut engendrer une expositio...

Page 66: ...peratur erforderlich Au erdem sollten mindestens 8 cm 3 in Freiraum um die Bel ftungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten Arbeiten Sie w hrend eines Gewitters und Blitzschl...

Page 67: ...2 Mellanox Technologies 67 8 Strahlenkontak Achtung Nutzung von Steuerungen oder Einstellungen oder Ausf hrung von Prozeduren die hier nicht spezifiziert sind kann zu gef hrlichem Strahlenkon takt f h...

Page 68: ...ire adecuada se debe dejar como m nimo un espacio de 8 cm 3 pulgadas alrededor de las aberturas de ventilaci n No utilizar el equipo ni conectar o desconectar cables durante per odos de actividad de r...

Page 69: ...onnectX 4 100Gb s Ethernet Dual Port Adapter Card User Manual for Dell PowerEdge Rev 1 2 Mellanox Technologies 69 PRODUCTO L SER DE CLASE 1 y referencia a las normas de l ser m s recientes IEC 60825 1...

Reviews:

No comments

Brands by name

Popular brands

Load more brands