Nvidia Mellanox ConnectX-5 User Manual Download Page 1

Page: 1 / 71

Exported on Sep/19/2022 09:36 AM

NVIDIA ConnectX-5 Ethernet Adapter Cards

for OCP Spec 3.0 User Manual

Summary of Contents for Mellanox ConnectX-5

Page 1: ...Exported on Sep 19 2022 09 36 AM NVIDIA ConnectX 5 Ethernet Adapter Cards for OCP Spec 3 0 User Manual...

Page 2: ...14 Hardware Installation 15 Safety Warnings 15 Installation Procedure Overview 16 System Requirements 17 Hardware Requirements 17 Airflow Requirements 17 Software Requirements 17 Safety Precautions 18...

Page 3: ...tware Requirements 49 Installing NVIDIA NATIVE ESXi Driver for VMware vSphere 50 Removing Earlier NVIDIA Drivers 50 Firmware Programming 51 Updating Adapter Firmware 52 Troubleshooting 53 General Trou...

Page 4: ...Lock bracket MCX565A CCAB ConnectX 5 EN network interface card for OCP 3 0 with host management 100GbE Single port QSFP28 PCIe3 0 x16 Thumbscrew Pull Tab bracket ConnectX 5 Ex MCX566M GDAI ConnectX 5...

Page 5: ...Firmware Utility mlxup User Manual and Release Notes NVIDIA firmware update and query utility used to update the firmware See NVIDIA Firmware Utility mlxup Documentation NVIDIA Firmware Tools MFT Use...

Page 6: ...ventions When discussing memory sizes MB and MBytes are used in this document to mean size in mega Bytes The use of Mb or Mbits small b indicates size in mega bits In this document PCIe is used to mea...

Page 7: ...ar PCI connector may exceed the maximum insertion force that is allowed by the connector s spec NVIDIA is updating the OCP 3 0 cards to ensure the proper insertion process The following tables provide...

Page 8: ...ock Bracket Pull tab Bracket Internal Lock Bracket Pull tab Bracket Host Management IC Part Number MT27808A0 FCCF AE MT27808A0 FCCF AE MT27808A0 FCCF CE MT27808A0 FCCF CE Device ID 4119 for Physical F...

Page 9: ...from individual virtual machines over logical tunnels in encapsulated formats such as NVGRE and VXLAN While this solves network scalability issues it hides the TCP packet from the hardware offloading...

Page 10: ...s optimizes board management and power performance and firmware update management via NC SI MCTP over SMBus and MCTP over PCIe as well as PLDM for Monitor and Control DSP0248 and PLDM for Firmware Upd...

Page 11: ...EL CentOS Windows FreeBSD VMware OpenFabrics Enterprise Distribution OFED OpenFabrics Windows Distribution WinOF 2 Connectivity Interoperable with 1 10 25 40 50 100 Gb s Ethernet switches Passive copp...

Page 12: ...s a bicolor Yellow and Green LED and LED1 is a single color Green LED Link Indications LED and State Description 1Hz blinking Yellow Beacon command for locating the adapter card 4Hz blinking Yellow In...

Page 13: ...or x1 Support for MSI MSI X mechanisms ConnectX 5 Ex PCIe Gen 3 0 4 0 1 1 and 2 0 compatible through x16 edge connectors Link Rates 2 5 5 0 8 0 or 16 0GT s Auto Negotiation to x16 x8 x4 x2 or x1 Supp...

Page 14: ...the server please contact NVIDIA Support Voltage Regulators The voltage regulator power is derived from the OCP 3 0 edge connector 12V and 3 3V supply pins These voltage supply pins feed onboard regul...

Page 15: ...this maximum ambient temperature is required for HCA cards and NICs To guarantee proper airflow allow at least 8cm 3 inches of clearance around the ventilation openings During Lightning Electrical Haz...

Page 16: ...software requirements Refer to System Requirements 2 Pay attention to the airflow consideration within the host system Refer to Airflow Requirements 3 Follow the safety precautions Refer to Safety Pre...

Page 17: ...g Systems Distributions section under the Introduction section Unless otherwise specified NVIDIA products are designed to work in an environmentally controlled data center with low levels of gaseous a...

Page 18: ...ssembled either with a thumbscrew pull tab bracket an internal lock bracket or an ejector latch bracket If this form factor is suitable for your requirements you can skip the remainder of this section...

Page 19: ...ng the torx tool type listed in the below table remove the screws according to the instructions per OCP 3 0 bracket type Internal Lock Bracket Pull tab Thumbscrew Bracket Ejector Latch Bracket Gently...

Page 20: ...d screws that came with the new bracket kit Use the specified torx tool type and apply the specified torque on the screws per bracket form factor Internal Lock Bracket Pull tab Bracket Ejector Latch B...

Page 21: ...cord is not connected to the server Please follow proper electrical grounding procedures Open the system case Align the card with the system rails Internal Lock Bracket Pull tab Thumbscrew Bracket Eje...

Page 22: ...tor will light when the physical connection is established that is when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a functioning port...

Page 23: ...s tab and select Hardware Ids Windows 2012 R2 2016 from the Property pull down menu PCI Device Example In the Value display box check the fields VEN and DEV fields are separated by In the display exam...

Page 24: ...24 Cards Extraction Instructions Unable to render include or excerpt include Could not retrieve page...

Page 25: ...012 Windows Server 2016 Windows Server 2019 Windows 8 1 Client 64 bit only Windows 10 Client 64 bit only Note The Operating System listed above must run with administrator privileges Downloading NVIDI...

Page 26: ...Manually configure your setup to contain the logs option replace LogFile with the relevant directory MLNX_WinOF2 Driver Version _ revision_version _All_Arch exe v l vx LogFile Optional If you do not w...

Page 27: ...27 4 5 6 Click Next in the Welcome screen Read and accept the license agreement and click Next Select the target folder for the installation...

Page 28: ...he user has an OEM card In this case the firmware will not be displayed If the user has a standard NVIDIA card with an older firmware version the firmware will be updated accordingly However if the us...

Page 29: ...Performances tools install the performance tools that are used to measure performance in user environment Documentation contains the User Manual and Release Notes Management tools installation tools u...

Page 30: ...30 b Click Next to install the desired tools 9 Click Install to start the installation...

Page 31: ...31 10 In case firmware upgrade option was checked in Step 7 you will be notified if a firmware upgrade is required see 11 Click Finish to complete the installation 1...

Page 32: ...l_Arch exe vMT_NDPROPERTY 1 Optional If you do not wish to upgrade your firmware version i e MT_SKIPFWUPGRD default value is False MLNX_WinOF2 Driver Version _ revision_version _All_Arch exe vMT_SKIPF...

Page 33: ...rt Control Panel Programs and Features MLNX_WinOF2 Uninstall NOTE This requires elevated administrator privileges Unattended Uninstallation To uninstall MLNX_WinOF2 in unattended mode Open a CMD conso...

Page 34: ...onsole Click Start Task Manager File Run new task and enter CMD Extract the driver and the tools MLNX_WinOF2 2_0_ revision_version _All_x64 a To extract only the driver file MLNX_WinOF2 2_0_ revision_...

Page 35: ...35 5 Click Install to extract this folder or click Change to install to a different folder...

Page 36: ...e refer to the MFT User Manual at https docs nvidia com category mft Linux Driver Installation This section describes how to install and test the NVIDIA OFED for Linux package on a single server with...

Page 37: ...MLNX_OFED_LINUX ver OS label CPU arch iso You can download and install the latest OpenFabrics Enterprise Distribution OFED software package available via https docs nvidia com category mlnxofedib Scr...

Page 38: ...upport sh script can be executed directly from the mlnxofedinstall script For further information please see add kernel support option below Example The following command will create a MLNX_OFED_LINUX...

Page 39: ...l other Mellanox OEM OFED RDMA or Distribution IB packages will be removed Those packages are removed due to conflicts with MLNX_OFED_LINUX do not reinstall them Starting MLNX_OFED_LINUX x x x install...

Page 40: ...ll occur and the installation script will print at the end of installation a message similar to the following Device 1 Device Type ConnectX 5 Ex Part Number MCX546A CDAN Description ConnectX 5 Ex netw...

Page 41: ...originally programmed with an Expansion ROM image the automatic firmware update will also burn an Expansion ROM image In case that your machine has an unsupported network adapter device no firmware u...

Page 42: ...the blacklists from step 1 and will prevent the kernel from loading the modules automatically mlnxofedinstall Return Codes The table below lists the mlnxofedinstall script return codes and their mean...

Page 43: ...TP request sent awaiting response 200 OK Length 1354 1 3K text plain Saving to RPM GPG KEY Mellanox 100 1 354 K s in 0s 2014 04 20 13 52 30 247 MB s RPM GPG KEY Mellanox saved 1354 1354 Install the ke...

Page 44: ...with KMP support for RHEL and SLES For other operating systems kernel module RPM packages are provided only for the operating systems default kernel In this case the group RPM packages have the suppor...

Page 45: ...installation is applicable to Debian and Ubuntu operating systems Setting up MLNX_OFED apt get Repository Log into the installation machine as root Extract the MLNX_OFED pacakge on a shared location i...

Page 46: ...NX_OFED vma vpi installer package Install the desired group apt get install group name Example apt get install mlnx ofed all Uninstalling MLNX_OFED Using the apt get Tool Use the script usr sbin ofed_...

Page 47: ...aded from the NVIDIA web site http www nvidia com Support Firmware Download Get the device s PSID mlxfwmanager_pci grep PSID PSID MT_1210110019 Download the firmware BIN file from the NVIDIA website o...

Page 48: ...er Add the public key to the MOK list using the mokutil utility mokutil import mlnx_signing_key_pub der Reboot the system The pending MOK key enrollment request will be noticed by shim efi and it will...

Page 49: ...ose key is not in the kernel key ring Therefore this message may go unnoticed Once the system is rebooted after unloading and reloading a kernel module the message will appear Note that this message c...

Page 50: ...ore 4 16 8 8 1OEM 650 0 0 4240417 MEL PartnerSupported 2017 01 31 nmlx5 rdma 4 16 8 8 1OEM 650 0 0 4240417 MEL PartnerSupported 2017 01 31 Removing Earlier NVIDIA Drivers To remove all the drivers Log...

Page 51: ...mft 4 6 0 48 10EM 650 0 0 4598673 x86_64 vib MD5SUM 0804cffe30913a7b4017445a0f0adbe1 Install the image according to the steps described in the MFT User Manual To remove the modules you must run the co...

Page 52: ...pdate Example server1 mlxup Querying Mellanox devices firmware Device Type ConnectX 5 Part Number MCX562A ACAI Description ConnectX 5 EN network interface card for OCP 3 0 with host management 25GbE D...

Page 53: ...ng another adapter Try removing and re installing all adapters Check that cables are connected properly Make sure your motherboard has the latest BIOS Link indicator light is off Try another port on t...

Page 54: ...ows desktop choose the Start menu and run msinfo32 To export system information to a text file choose the Export option from the File menu Assign a file name and save NVIDIA Firmware Tool MFT Download...

Page 55: ...E 3 3V_E DGE Typical Power a Passive Cables 12 9W 0 33W TBD 1 5W Active Cables 16 35W 0 33W TBD Maximum Power Passive Cables 15 3W 0 33W TBD 0 75W Active Cables 18 82W 0 33W TBD Maximum power availabl...

Page 56: ...ble 400LFM 100LFM TBD TBD Temperature Operational 0 C to 55 C Non operational 40 C to 70 C Humidity 90 relative humidity b Altitude Operational 3050m Regulato ry Safety CB cTUVus CE EMC CE FCC VCCI IC...

Page 57: ...Cable Type Active Mode Standby Mode 12V_EDGE 3 3V_EDGE Typical Power a Passive Cables 16 5W 0 33W TBD 2 2W Active Cables 21 7W 0 33W TBD Maximum Power Passive Cables 19 3W 0 33W TBD 2 2W Active Cables...

Page 58: ...58 Altitude Operational 3050m Regulatory Safety CB cTUVus CE EMC CE FCC VCCI ICES RCM RoHS RoHS compliant a Typical power for ATIS traffic load b For both operational and non operational states...

Page 59: ...16 lanes 2 0 and 1 1 compatible Power Voltage 3 3V_EDGE 12V_EDGE Power Cable Type Active Mode Standby Mode 12V_EDGE 3 3V_E DGE Typical Power a Passive Cables 14 5W 0 33W TBD 2 2W Active Cables 17W 0 3...

Page 60: ...60 Regulatory Safety CB cTUVus CE EMC CE FCC VCCI ICES RCM RoHS RoHS compliant a Typical power for ATIS traffic load b For both operational and non operational states...

Page 61: ...6 0GT s x16 lanes 2 0 and 1 1 compatible Power Voltage 3 3V_EDGE 12V_EDGE Power Cable Type Active Mode Standby Mode 12V_EDGE 3 3V_EDGE Typical Power a Passive Cables 19 65W 0 33W TBD 2 2W Active Cable...

Page 62: ...70 C Humidity 90 relative humidity b Altitude Operational 3050m Regulatory Safety CB cTUVus CE EMC CE FCC VCCI ICES RCM RoHS RoHS compliant a Typical power for ATIS traffic load b For both operational...

Page 63: ...0 and 1 1 compatible Power Voltage 3 3V_EDGE 12V_EDGE Power Cable Type Active Mode Standby Mode 12V_EDGE 3 3V_EDG E Typical Power a Passive Cables 19W 0 33W TBD 2 2W Active Cables 21 5W 0 33W TBD Max...

Page 64: ...64 EMC CE FCC VCCI ICES RCM RoHS RoHS compliant a Typical power for ATIS traffic load b For both operational and non operational states...

Page 65: ...Dimensions MCX562A ACAI Drawings MCX562A ACAB Drawings MCX566A CCAI Drawings MCX566A CDAB Drawings MCX566A CDAI Drawings MCX565M CDAI MCX565M CDAB Drawings All dimensions are in millimeters All the m...

Page 66: ...ons MCX562A ACAI Bracket MCX562A ACAB Bracket MCX566A CCAI MCX566M GDAI MCX566A CDAI Bracket MCX565A CCAB MCX565M CDAB Bracket MCX565M CDAI Bracket MCX566A CDAB Bracket All dimensions are in millimete...

Page 67: ...entifier printed on the label serial number and the card MAC for the Ethernet protocol MCX562A ACAB Board Label MCX566A CCAI Board Label MCX565M CDAB Board Label The product revisions indicated on the...

Page 68: ...68 MCX566A CDAB Board Label MCX566A CDAI Board Label MCX565M CDAI Board Label MCX562A ACAI Board Label...

Page 69: ...airflow numbers May 2020 2 0 Updated board labels Apr 2020 1 9 Updated Specifications Feb 2020 1 8 Added altitude criteria to Specifications Nov 2019 1 7 Added a note to Introduction Oct 2019 1 6 Add...

Page 70: ...by customer and perform the necessary testing for the application in order to avoid a default of the application or the product Weaknesses in customer s product designs may affect the quality and rel...

Search results

Summary of Contents for Mellanox ConnectX-5

Reviews:

Related manuals for Mellanox ConnectX-5

Brands by name

Popular brands

Nvidia Mellanox ConnectX-5, User Manual

Search results

Summary of Contents for Mellanox ConnectX-5

Reviews:

Related manuals for Mellanox ConnectX-5

Brands by name

Popular brands