Nvidia Mellanox User Manual Download

Page: 1 / 64

Exported on Oct/23/2020 12:05 PM

https://docs.mellanox.com/x/TwvTAQ

NVIDIA Mellanox ConnectX-6 InfiniBand/VPI

Adapter Cards for OCP Spec 3.0 User Manual

Summary of Contents for Mellanox

Page 1: ...Exported on Oct 23 2020 12 05 PM https docs mellanox com x TwvTAQ NVIDIA Mellanox ConnectX 6 InfiniBand VPI Adapter Cards for OCP Spec 3 0 User Manual...

Page 2: ...allation 15 Safety Warnings 15 Installation Procedure Overview 15 System Requirements 15 Hardware Requirements 16 Airflow Requirements 16 Software Requirements 16 Safety Precautions 17 Pre Installatio...

Page 3: ...ation Results 31 Extracting Files Without Running Installation 31 Uninstalling Mellanox WinOF 2 Driver 33 Attended Uninstallation 33 Unattended Uninstallation 33 Firmware Upgrade 33 Deploying the Driv...

Page 4: ...m kernel Modules 46 Performance Tuning 47 VMware Driver Installation 47 Hardware and Software Requirements 47 Installing Mellanox NATIVE ESXi Driver for VMware vSphere 48 Removing Earlier Mellanox Dri...

Page 5: ...CX653435A HDAI Board Labels Example TBD 61 MCX653436A HDAI Board Labels Example TBD 61 MCX653435A EDAI Board Labels Example TBD 61 MCX653435A HDAE Board Labels Example TBD 61 Document Revision History...

Page 6: ...his manual is intended for the installer and user of these cards The manual assumes basic familiarity with InfiniBand VPI network and architecture specifications Technical Support Customers who purcha...

Page 7: ...ables and transceivers are designed to maximize the performance of High Performance Computing networks requiring high bandwidth low latency connections between compute nodes and switch nodes Mellanox...

Page 8: ...dst of retesting these cards to comply with OCP spec 3 0 Products Overview Model ConnectX 6 VPI Cards for OCP Spec 3 0 Part Number MCX653435A HDAI MCX653436A HDAI MCX653435A HDAE a MCX653435A EDAI Hos...

Page 9: ...ncoding resulting in an effective bandwidth of 100Gb s Up to 200 Gigabit Ethernet Mellanox adapters comply with the following IEEE 802 3 standards 200GbE 100GbE 50GbE 40GbE 25GbE 10GbE 1GbE IEEE 802 3...

Page 10: ...nds of nodes CPU Offload Adapter functionality enabling reduced CPU overhead allowing more available CPU for computation tasks Open vSwitch OVS offload using ASAP2 TM Flexible match action flow tables...

Page 11: ...VMware OpenFabrics Enterprise Distribution OFED OpenFabrics Windows Distribution WinOF 2 Connectivity Interoperable with 1 10 25 40 50 100 200 Gb s Ethernet switches Passive copper cable with ESD prot...

Page 12: ...ce The table below describes the supported PCIe interface in ConnectX 6 adapter cards Supported PCIe Interface Features PCIe Gen 3 0 4 0 1 1 and 3 0 compatible through x16 edge connectors Link Rates 2...

Page 13: ..._ID0 and SLOT_ID1 and its capacity is 4Kb Heat Sink Interface A heatsink is attached to the ConnectX 6 IC in order to dissipate the heat from the ConnectX 6 IC It is attached either by using four spri...

Page 14: ...14 CPLD Interface The adapter card incorporates a CPLD device that controls the networking port LEDs and the scan chain It draws its power supply from 3 3V_EDGE...

Page 15: ...Pre Installation Checklist 5 Optional Replace the assembled OCP 3 0 bracket with the desired form factor bracket Refer to OCP 3 0 Bracket Replacement Instructions 6 Install ConnectX 6 adapter card for...

Page 16: ...rection Cold Aisle Cooling Port to heatsink Airflow Direction Software Requirements See Operating Systems Distributions section under the Introduction section Software Stacks Mellanox OpenFabric softw...

Page 17: ...ation Instructions This section provides detailed instructions on how to install your adapter card in a system The below table lists the different ConnectX 6 OCP 3 0 retention mechanisms and provides...

Page 18: ...re installing the card make sure that the system is off and the power cord is not connected to the server Please follow proper electrical grounding procedures Open the system case Align the card with...

Page 19: ...er Please follow proper electrical grounding procedures Open the system case Align the card with the system rails while making sure the ejector latch is open Push the card until connectors are in full...

Page 20: ...p Cable Installation All cables can be inserted or removed with the unit powered on To insert a cable press the connector into the port receptacle until the connector is firmly seated Support the weig...

Page 21: ...he rack To remove a cable disengage the locks and slowly pull the connector away from the port receptacle LED indicator will turn off when the cable is unseated Identifying the Card in Your System On...

Page 22: ...nstalled in a system that operates with voltages that can be lethal Before uninstalling the adapter card please observe the following precautions to avoid injury and prevent damage to system component...

Page 23: ...t the adapter card Cards with Internal Lock Bracket Extraction Instructions Verify that the system is powered off and unplugged Wait 30 seconds Gently extract the adapter card from the server Cards wi...

Page 24: ...24 1 2 3 4 Verify that the system is powered off and unplugged Wait 30 seconds Open the ejector latch Gently extract the adapter card from the server...

Page 25: ...d above must run with administrator privileges Downloading Mellanox WinOF 2 Driver To download the executable file according to your operating system please follow the steps below Obtain the machine a...

Page 26: ...you do not want to upgrade your firmware version Note MT_SKIPFWUPGRD default value is False MLNX_WinOF2 1_10_ revision_version _All_x64 exe v MT_SKIPFWUPGRD 1 Installing the incorrect exe file is pro...

Page 27: ...the installation The firmware upgrade screen will be displayed in the following cases If the user has an OEM card In this case the firmware will not be displayed If the user has a standard Mellanox ca...

Page 28: ...ion follow Step a and on Select the desired feature to install Performance tools install the performance tools that are used to measure performance in user environment Documentation contains the User...

Page 29: ...ools used for diagnostics such as mlx5cmd Click Next to install the desired tools Click Install to start the installation In case that firmware upgrade option was checked in Step 7 you will be notifie...

Page 30: ...ant to control whether to install ND provider or not Note MT_NDPROPERTY default value is True MLNX_WinOF2 1_10_All_x64 exe vMT_NDPROPERTY 1 Optional If you do not wish to upgrade your firmware version...

Page 31: ...ers OS To see the Mellanox network adapters display the Device Manager and pull down the Network adapters menu Extracting Files Without Running Installation To extract the files without running instal...

Page 32: ...3 4 5 Click Next to create a server image Click Change and specify the location in which the files are extracted to Click Install to extract this folder or click Change to install to a different fold...

Page 33: ...CMD console Click Task Manager File Run new task and enter CMD To uninstall the driver run MLNX_WinOF2 1_10_All_x64 exe S x v qn Firmware Upgrade If the machine has a standard Mellanox card with an ol...

Page 34: ...tPath InjectingDrivers vhdx DriversPath C WS2016TP5_Drivers Create the Nano server image Follow the instructions in the TechNet article for Getting Started with Nano Server Online Update Go to the Mel...

Page 35: ...The following example shows a system with an installed Mellanox adapter card lspci v grep Mellanox 86 00 0 Network controller 0207 Mellanox Technologies TBD Family Subsystem Mellanox Technologies Devi...

Page 36: ...e the deletion of the old packages If you need to install Mellanox OFED on an entire homogeneous cluster a common strategy is to mount the ISO image on one of the cluster nodes and then copy it to a s...

Page 37: ...stalled the OpenSM Subnet Manager on your machine For the list of installation options run mlnxofedinstall h Installation Procedure Login to the installation machine as root Mount the ISO image on you...

Page 38: ...installation a message similar to the following Device 1 Device Type ConnectX 6 Part Number MCX653436A HDAI Description ConnectX 6 VPI adapter card 200Gb s HDR IB and 200GbE for OCP 3 0 with host mana...

Page 39: ...tallation script is run in default mode that is without the option without fw update The firmware version of the adapter device is older than the firmware version included with the Mellanox OFED ISO i...

Page 40: ...ocess of updating the initramfs will add the blacklists from step 1 and will prevent the kernel from loading the modules automatically mlnxofedinstall Return Codes The table below lists the mlnxofedin...

Page 41: ...cted HTTP request sent awaiting response 200 OK Length 1354 1 3K text plain Saving to RPM GPG KEY Mellanox 100 1 354 K s in 0s 2014 04 20 13 52 30 247 MB s RPM GPG KEY Mellanox saved 1354 1354 Install...

Page 42: ...port for RHEL and SLES For other operating systems kernel module RPM packages are provided only for the operating systems default kernel In this case the group RPM packages have the supported kernel v...

Page 43: ...rating systems Setting up MLNX_OFED apt get Repository Log into the installation machine as root Extract the MLNX_OFED pacakge on a shared location in your network You can download it from http www me...

Page 44: ...pi installer package Install the desired group apt get install group name Example apt get install mlnx ofed all Uninstalling MLNX_OFED Using the apt get Tool Use the script usr sbin ofed_uninstall sh...

Page 45: ...Firmware Download Get the device s PSID mlxfwmanager_pci grep PSID PSID MT_1210110019 Download the firmware BIN file from the Mellanox website or the OEM website Burn the firmware mlxfwmanager_pci i...

Page 46: ...ill launch MokManager efi to allow you to complete the enrollment from the UEFI console You will need to enter the password you previously associated with this request and confirm the enrollment Once...

Page 47: ...ssage will appear Note that this message cannot be eliminated Update the initramfs on RHEL systems with the stripped modules mkinitrd boot initramfs uname r img uname r force Performance Tuning Depend...

Page 48: ...7 01 31 Removing Earlier Mellanox Drivers To remove all the drivers Log into the ESXi server with root permissions List all the existing NATIVE ESXi driver modules See Step 4 in Installing Mellanox NA...

Page 49: ...Firmware Tools MFT site ESXi 6 5 File mft 4 6 0 48 10EM 650 0 0 4598673 x86_64 vib MD5SUM 0804cffe30913a7b4017445a0f0adbe1 Install the image according to the steps described in the MFT User Manual Th...

Page 50: ...server1 mlxup Querying Mellanox devices firmware Device Type ConnectX 6 Part Number MCX653436A HDAI Description ConnectX 6 VPI adapter card 200Gb s HDR IB and 200GbE for OCP 3 0 with host management...

Page 51: ...ng another adapter Try removing and re installing all adapters Check that cables are connected properly Make sure your motherboard has the latest BIOS Link indicator light is off Try another port on t...

Page 52: ...ows desktop choose the Start menu and run msinfo32 To export system information to a text file choose the Export option from the File menu Assign a file name and save Mellanox Firmware Tool MFT Downlo...

Page 53: ...0BASE CX 1000BASE KX 10GBASE SR InfiniBand IBTA v1 3 and 1 4 Auto Negotiation a 1X 2X 4X SDR 2 5Gb s per lane DDR 5Gb s per lane QDR 10Gb s per lane FDR 14 0625Gb s per lane EDR 25Gb s per lane port H...

Page 54: ...DAI Specifications Physical Size 2 99 in x 4 52 in 76 00mm x 115 00mm Connector Dual QSFP56 InfiniBand and Ethernet copper and optical Retention Mechanism Internal Lock Protocol Support Ethernet 200GB...

Page 55: ...de Passive Cable 700LFM 100LFM Not Supported Not Supported Active 4 55W Cable 700LFM 45C 500LFM Not Supported Not Supported Regulator y Safety CB cTUVus CE EMC CE FCC VCCI ICES RCM RoHS RoHS compliant...

Page 56: ...lane port HDR100 2 lane x 50Gb s per lane HDR 50Gb s per lane port Data Rate Ethernet 1 10 25 40 100 Gb s InfiniBand SDR DDR QDR FDR EDR HDR100 PCI Express Gen 3 0 4 0 SERDES 8 0GT s 16 0GT s 16 lanes...

Page 57: ...BASE R4 40GBASE CR4 40GBASE KR4 40GBASE SR4 40GBASE LR4 40GBASE ER4 40GBASE R2 25GBASE R 20GBASE KR2 10GBASE LR 10GBASE ER 10GBASE CX4 10GBASE CR 10GBASE KR SGMII 1000BASE CX 1000BASE KX 10GBASE SR In...

Page 58: ...ConnectX 6 adapters supplement the IBTA auto negotiation specification to get better bit error rates and longer cable reaches This supplemental feature only initiates when connected to another Mellano...

Page 59: ...59 Brackets Mechanical Drawings and Dimensions Cards with Ejector Latch Bracket All dimensions are in millimeters The brackets mechanical tolerance is 0 25mm...

Page 60: ...60 Cards with Internal Lock Bracket...

Page 61: ...UID and a MAC derived from the GUID MCX653435A HDAI Board Labels Example TBD MCX653436A HDAI Board Labels Example TBD MCX653435A EDAI Board Labels Example TBD MCX653435A HDAE Board Labels Example TBD...

Page 62: ...fications Updated networking ports LEDs indications in Interfaces Sep 2020 1 3 Added OCP 3 0 bracket replacement instructions Nov 2020 1 2 Updated Specifications with airflow numbers Oct 2019 1 1 Adde...

Page 63: ...arily performed by NVIDIA It is customer s sole responsibility to evaluate and determine the applicability of any information contained in this document ensure the product is suitable and fit for the...

Page 64: ...A the NVIDIA logo and Mellanox are trademarks and or registered trademarks of Mellanox Technologies Ltd and or NVIDIA Corporation in the U S and in other countries Other company and product names may...

Search results

Summary of Contents for Mellanox

Reviews:

Related manuals for Mellanox

Brands by name

Popular brands

Nvidia Mellanox, User Manual

Search results

Summary of Contents for Mellanox

Reviews:

Related manuals for Mellanox

Brands by name

Popular brands