Nvidia ConnectX-6 Dx MCX623430MS-CDAB User Manual Download Page 3

Page: 3 / 75

Cards with Internal Lock Bracket............................................................ 26

Cards with Ejector-Latch Bracket ........................................................... 26

Cables and Modules................................................................................ 27

Identifying the Card in Your System............................................................. 28

Card Extraction Instructions ..................................................................... 29

Safety Precautions ............................................................................. 24

Cards Extraction Instructions ................................................................ 30

Driver Installation.............................................................................32

Windows Driver Installation ...................................................................... 32

Hardware and Software Requirements ..................................................... 32

Downloading NVIDIA WinOF-2 Driver ........................................................ 32

Installing NVIDIA WinOF-2 Driver ............................................................ 33

Installation Results ............................................................................ 38

Extracting Files Without Running Installation ............................................. 38

Uninstalling NVIDIA WinOF-2 Driver ......................................................... 40

Firmware Upgrade ............................................................................. 40

Deploying the Driver on a Nano Server ..................................................... 41

Linux Driver Installation .......................................................................... 42

Prerequisites.................................................................................... 42

Downloading NVIDIA OFED .................................................................... 42

Installing NVIDIA OFED ........................................................................ 43

Installing MLNX_OFED Using YUM ............................................................ 48

Installing MLNX_OFED Using apt-get Tool .................................................. 51

Updating Firmware After Installation ....................................................... 52

UEFI Secure Boot............................................................................... 53

Performance Tuning ........................................................................... 55

VMware Driver Installation ....................................................................... 55

Hardware and Software Requirements ..................................................... 55

Installing NVIDIA NATIVE ESXi Driver for VMware vSphere ............................... 55

Removing Earlier NVIDIA Drivers ............................................................. 56

Firmware Programming ....................................................................... 56

Updating Adapter Firmware .................................................................57

Troubleshooting ...............................................................................58

General Troubleshooting .......................................................................... 58

Summary of Contents for ConnectX-6 Dx MCX623430MS-CDAB

Page 1: ...Exported on Sep 19 2022 12 36 PM NVIDIA ConnectX 6 Dx Ethernet Adapter Cards for OCP Spec 3 0 User Manual...

Page 2: ...t 16 Ethernet Networking Interfaces 18 PCI Express Interface 18 Networking Ports LEDs Interface 18 SMBus Interface 19 Voltage Regulators 19 CPLD Interface 19 Heatsink 20 Thermal Sensors 20 Hardware In...

Page 3: ...ning Installation 38 Uninstalling NVIDIA WinOF 2 Driver 40 Firmware Upgrade 40 Deploying the Driver on a Nano Server 41 Linux Driver Installation 42 Prerequisites 42 Downloading NVIDIA OFED 42 Install...

Page 4: ...DAB MCX623436MC CDAB MCX623436MN CDAB MCX623436MS CDAB MCX623436AS CDAI Specifications 64 MCX623430MS CDAB Specifications 65 MCX623435AC CDAB MCX623435AE CDAB MCX623435AN CDAB MCX623435AC VDAB MCX6234...

Page 5: ...t management Single port QSFP56 PCIe 4 0 x16 No Crypto Thumbscrew Pull Tab Bracket MCX623435AC CDAB ConnectX 6 Dx EN adapter card 100GbE OCP3 0 With Host Management Single port QSFP56 PCIe 4 0 x16 Cry...

Page 6: ...nt Single port QSFP56 Multi Host or Socket Direct PCIe 4 0 x16 Crypto and Secure Boot Thumbscrew Pull Tab Bracket MCX623432A C ADAB ConnectX 6 Dx EN adapter card 25GbE OCP3 0 With Host Management Dual...

Page 7: ...e http www nvidia com Products Software Firmware Tools mlxup Firmware Utility NVIDIA Firmware Tools MFT User Manual User Manual describing the set of MFT firmware management tools for a single node Se...

Page 8: ...en discussing memory sizes MB and MBytes are used in this document to mean size in MegaBytes The use of Mb or Mbits small b indicates the size in Megabits In this document PCIe is used to mean PCI Exp...

Page 9: ...er port speed number of ports and PCI Express speed Products Overview ConnectX 6 Dx 25GbE Adapter Cards for OCP 3 0 OPN MCX623432AN ADAB MCX623432AC ADAB MCX623432AE ADAB MCX623432AS ADAB Network Conn...

Page 10: ...Device ID 4125 for Physical Function PF 4126 for Virtual Function VF ConnectX 6 Dx 100GbE Adapter Cards for OCP 3 0 OPN MCX 6234 36AN CDAB MCX 6234 36AC CDAB MCX 6234 35M N CDAB MCX 6234 36M N CDA B...

Page 11: ...ant RoHS Compl iant RoHS Compl iant RoHS Compl iant RoHS Compl iant RoHS Compl iant RoHS Compl iant RoHS Compli ant RoHS Compl iant IC Part Numb er MT289 28A0 XCCF CE MT289 28A0 XCCF CE MT289 28A0 XCC...

Page 12: ...3bm 100 Gigabit Ethernet IEEE 802 3by Ethernet Consortium25 50 Gigabit Ethernet supporting all FEC modes IEEE 802 3ba 40 Gigabit Ethernet IEEE 802 3by 25 Gigabit Ethernet IEEE 802 3ae 10 Gigabit Ether...

Page 13: ...y of Service QoS Support for port based Quality of Service enabling various application requirements for latency and SLA Hardware based I O Virtualization ConnectX 6 Dx provides dedicated adapter reso...

Page 14: ...ompute and storage racks with direct connectivity between multiple hosts and the centralized network controller This enables direct data access with the lowest latency to significantly improve densiti...

Page 15: ...0 through an x16 edge connector 3 Ethernet Networking Interface Ethernet traffic is transmitted through the adapter s SFP28 SFP56 QSFP56 connectors The networking connectors allow for the use of modul...

Page 16: ...AES data at rest cryptographic operations Stateful firewall solution acceleration powered by Open vSwitch connection tracking and NVIDIA s ASAP2 technology Embedded hardware root of trust and support...

Page 17: ...on through the BIF 2 0 pins which it drives to the adapter card The NVIDIA OCP3 0 card has an internal logic that uses the BIF 2 0 data and determines the correct operating mode to boot at The combina...

Page 18: ...atures PCIe Gen 3 0 4 0 1 1 and 2 0 compatible through x16 edge connectors Link Rates 2 5 5 0 8 0 or 16GT s Auto Negotiation to x16 x8 x4 x2 or x1 Support for MSI MSI X mechanisms Networking Ports LED...

Page 19: ...ferent types of Mezzanine cards FRU EEPROM is accessible through SMCLK and SMDATA FRU EEPROM address is defined according to SLOT_ID0 and SLOT_ID1 and its capacity is 4Kb SMBus Interface ConnectX 6 Dx...

Page 20: ...the device crosses the 100 C threshold a Warning Threshold message will be issued by the management SW indicating to system administration that the card has crossed the Warning threshold Note that thi...

Page 21: ...w of 200LFM at this maximum ambient temperature is required for HCA cards and NICs To guarantee proper airflow allow at least 8cm 3 inches of clearance around the ventilation openings During Lightning...

Page 22: ...uirements 3 Follow the safety precautions Refer to Safety Precautions 4 Follow the pre installation checklist Refer to Pre Installation Checklist 5 Optional Replace the assembled OCP 3 0 bracket with...

Page 23: ...ow Direction Cold Aisle Cooling Port to heatsink Airflow Direction Software Requirements See Operating Systems Distributions section under the Introduction section Software Stacks NVIDIA OpenFabric so...

Page 24: ...Replacement Instructions Unable to render include or excerpt include Could not retrieve page OCP 3 0 Adapter Card Installation Instructions This section provides detailed instructions on installing y...

Page 25: ...ds with Thumbscrew Pull tab Bracket Before installing the card ensure the system is off and the power cord is not connected to the server Please follow proper electrical grounding procedures Open the...

Page 26: ...ard with the system rails Push the card until connectors are in full mate and a clicking sound is heard Cards with Ejector Latch Bracket Before installing the card ensure the system is off and the pow...

Page 27: ...cable to the rack Determine the correct orientation of the connector to the card before inserting the connector Do not try and insert the connector upside down This may damage the adapter card Insert...

Page 28: ...To remove a cable disengage the locks and slowly pull the connector away from the port receptacle The LED indicator will turn off when the cable is unseated Identifying the Card in Your System On Linu...

Page 29: ...a system that operates with voltages that can be lethal Before uninstalling the adapter card please observe the following precautions to avoid injury and prevent damage to system components Remove any...

Page 30: ...ut the adapter card Cards with Internal Lock Bracket Extraction Verify that the system is powered off and unplugged Wait 30 seconds Gently extract the adapter card from the server Cards with Ejector L...

Page 31: ...31 1 2 3 4 Verify that the system is powered off and unplugged Wait 30 seconds Open the ejector latch Gently extract the adapter card from the server...

Page 32: ...Windows Server 2012 Windows Server 2016 Windows 8 1 Client 64 bit only Windows 10 Client 64 bit only Note The Operating System listed above must run with administrator privileges Downloading NVIDIA Wi...

Page 33: ...n the logs option replace LogFile with the relevant directory MLNX_WinOF2 1_10_ revision_version _All_x64 exe v l vx LogFile Optional If you do not want to upgrade your firmware version Note MT_SKIPFW...

Page 34: ...Read then accept the license agreement and click Next Select the target folder for the installation The firmware upgrade screen will be displayed in the following cases If the user has an OEM card In...

Page 35: ...DIA card only the NVIDIA card will be updated Select a Complete or Custom installation follow Step a and on Select the desired feature to install Performance tools install the performance tools that a...

Page 36: ...ools used for diagnostics such as mlx5cmd Click Next to install the desired tools Click Install to start the installation In case that firmware upgrade option was checked in Step 7 you will be notifie...

Page 37: ...ant to control whether to install ND provider or not Note MT_NDPROPERTY default value is True MLNX_WinOF2 1_10_All_x64 exe vMT_NDPROPERTY 1 Optional If you do not wish to upgrade your firmware version...

Page 38: ...vers OS To see the NVIDIA network adapters display the Device Manager and pull down the Network adapters menu Extracting Files Without Running Installation To extract the files without running install...

Page 39: ...3 4 5 Click Next to create a server image Click Change and specify the location in which the files are extracted to Click Install to extract this folder or click Change to install to a different fold...

Page 40: ...o uninstall MLNX_WinOF2 in unattended mode Open a CMD console Click Task Manager File Run new task and enter CMD To uninstall the driver run MLNX_WinOF2 1_10_All_x64 exe S x v qn Firmware Upgrade If t...

Page 41: ...lows This location should be specified for DriversPath property when injecting driver into the Nano server image New NanoServerImage MediaPath Path To Media en_us BasePath Base TargetPath InjectingDri...

Page 42: ...OFED Verify that the system has a NVIDIA network adapter installed The following example shows a system with an installed NVIDIA adapter card lspci v grep Mellanox 86 00 0 Network controller 0207 Mell...

Page 43: ...install script with the without fw update option Note If you wish to perform a firmware upgrade using customized FW binaries you can provide a path to the folder that contains the FW binary files by...

Page 44: ...he Ethernet fabric a Subnet Manager must be running on one of the fabric nodes At this point NVIDIA OFED for Linux has already installed the OpenSM Subnet Manager on your machine For the list of insta...

Page 45: ...edinstall force MLNX_OFED for Ubuntu should be installed with the following flags in the root environment mlnxofedinstall without dkms add kernel support kernel kernel version in chroot without fw upd...

Page 46: ...directory except for the following packages which are installed under the opt directory fca and ibutils The kernel modules are installed under lib modules uname r updates on SLES and Fedora Distributi...

Page 47: ...ontact your HW vendor Installation Logs While installing MLNX_OFED the install log for each selected package will be saved in a separate log file The path to the directory containing the log files wil...

Page 48: ...anings Return Code Meaning 0 The installation ended successfully 1 The installation failed 2 No firmware was found for the adapter device 22 Invalid parameter 28 Not enough free space 171 Not applicab...

Page 49: ...pg Mellanox Technologies support mellanox com Create a yum repository configuration file called etc yum repos d mlnx_ofed repo with the following content mlnx_ofed name MLNX_OFED Repository baseurl fi...

Page 50: ...noarch MLNX_OFED hypervisor installer package for kernel 3 17 4 301 fc21 x86_64 without KMP support mlnx ofed vma 3 17 4 301 fc21 x86_64 noarch MLNX_OFED vma installer package for kernel 3 17 4 301 f...

Page 51: ...2013 08 11 Update the apt get cache sudo apt get update Installing MLNX_OFED Using the apt get Tool After setting up the apt get repository for MLNX_OFED package perform the following View the availa...

Page 52: ...0 x16 Crypto and Secure Boot Thumbscrew Pull Tab Bracket PSID MT_1020120019 PCI Device Name 0000 09 00 0 Port1 GUID 0002c9000100d051 Port2 MAC 0002c9000002 Versions Current Available FW 2 32 5000 2 3...

Page 53: ...n boot If the firmware is updated the following message is printed to the system s standard logging file fw_updater Firmware was updated Please reboot your system for the changes to take effect Otherw...

Page 54: ...nt Once a key is in the MOK list it will be automatically propagated to the system key ring and subsequent will be booted when the UEFI Secure Boot is enabled Removing Signature from kernel Modules Th...

Page 55: ...lication of the user s system it may be necessary to modify the default configuration of network adapters based on the ConnectX adapters In case that tuning is required please refer to the Performance...

Page 56: ...er with root permissions List all the existing NATIVE ESXi driver modules See Step 4 in Installing NVIDIA NATIVE ESXi Driver for VMware vSphere Remove each module esxcli software vib remove n nmlx5 rd...

Page 57: ...Querying Mellanox devices firmware Device Type ConnectX 6 Dx Part Number MCX623436AC CDAB Description ConnectX 6 Dx EN adapter card 100GbE OCP3 0 With Host Management Dual port QSFP56 PCIe 4 0 x16 Cr...

Page 58: ...ng another adapter Try removing and re installing all adapters Check that cables are connected properly Make sure your motherboard has the latest BIOS Link indicator light is off Try another port on t...

Page 59: ...ows desktop choose the Start menu and run msinfo32 To export system information to a text file choose the Export option from the File menu Assign a file name and save NVIDIA Firmware Tool NFT Download...

Page 60: ...6 0GT s 16 lanes 2 0 and 1 1 compatible Power and Environmen tal Voltage 3 3VAUX 12V Power Cable Type Active Mode Standby Mode Typical Power b Passive Cables 15 7W 3 8W Maximum Power Passive Cables 19...

Page 61: ...SFP56 Ethernet copper and optical Retention Mechanism MCX623432A N C GDAB Thumbscrew Pull Tab Bracket MCX623432AS GDAI Internal Lock Protocol Support Ethernet 50GBASE R2 50GBASE R4 40GBASE CR4 40GBAS...

Page 62: ...4 52 in x 2 99 in 115 00mm x 76 00mm Connector Single QSFP56 Ethernet copper and optical Retention Mechanism Thumbscrew Pull Tab Bracket Protoco l Support Ethernet 100GBASE CR4 100GBASE KR4 100GBASE...

Page 63: ...er for ATIS traffic load c For both operational and non operational states MCX623439MC CDAB MCX623439MN CDAB Specifications Physi cal Form Factor Small Form Factor SFF OCP 3 0 Size 4 52 in x 2 99 in 1...

Page 64: ...y CB cTUVus CE EMC CE FCC VCCI ICES RCM RoHS RoHS compliant MCX623436AC CDAB MCX623436AN CDAB MCX623436AS CDAB MCX623436MC CDAB MCX623436MN CDAB MCX623436MS CDAB MCX623436AS CDAI Specifications Physi...

Page 65: ...CE FCC VCCI ICES RCM RoHS RoHS compliant a ConnectX 6 Dx adapters supplement the auto negotiation specification to get better bit error rates and longer cable reaches This supplemental feature only in...

Page 66: ...ve Cable 600LFM 100LFM 300LFM 100LFM Regula tory Safety CB cTUVus CE EMC CE FCC VCCI ICES RCM RoHS RoHS compliant a ConnectX 6 Dx adapters supplement the auto negotiation specification to get better b...

Page 67: ...9W 5 6W Maximum Power Passive Cables 24W 11 4W Maximum power available through QSFP56 port 5W Temperature Operation al 0 C to 55 C Non operation al 40 C to 70 C Humidity 90 relative humidity c Altitud...

Page 68: ...56 OCP 3 0 Mechanical Drawing and Dimensions ConnectX 6 Dx Dual Port QSFP56 OCP 3 0 Mechanical Drawing and Dimensions ConnectX 6 Dx Single Port QSFP56 OCP 3 0 Mechanical Drawing and Dimensions All dim...

Page 69: ...9 Bracket Mechanical Drawings and Dimensions Dual port SFP28 SFP56 Thumbscrew Bracket Dual port QSFP56 Thumbscrew Bracket All dimensions are in millimeters The bracket s mechanical tolerance is 0 25mm...

Page 70: ...70 Single port QSFP56 Thumbscrew Bracket...

Page 71: ...al number and the card MAC for the Ethernet protocol MCX623432AN ADAB Board Label Example MCX623432AC ADAB Board Label Example MCX623432AN GDAB Board Label Example MCX623432AC GDAB Board Label Example...

Page 72: ...72 MCX623436AN CDAB Board Label Example MCX623436AC CDAB Board Label Example MCX623435AN VDAB Board Label Example MCX623435AC VDAB Board Label Example...

Page 73: ...36MN CDAB and MCX623436MS CDAB Mar 2021 1 9 Added OPN MCX623432AS ADAB Feb 2021 1 8 Added OPN MCX623439MC N CDAB Jan 2021 1 7 Added OPNs MCX623435AE CDAB and MCX623435AN CDAB Oct 2020 1 6 Updated airf...

Page 74: ...by customer and perform the necessary testing for the application in order to avoid a default of the application or the product Weaknesses in customer s product designs may affect the quality and rel...

Search results

Summary of Contents for ConnectX-6 Dx MCX623430MS-CDAB

Reviews:

Related manuals for ConnectX-6 Dx MCX623430MS-CDAB

Brands by name

Popular brands

Nvidia ConnectX-6 Dx MCX623430MS-CDAB, User Manual

Search results

Summary of Contents for ConnectX-6 Dx MCX623430MS-CDAB

Reviews:

Related manuals for ConnectX-6 Dx MCX623430MS-CDAB

Brands by name

Popular brands