background image

12

Feature

Description

Up to 200 Gigabit 
Ethernet

Mellanox adapters comply with the following IEEE 802.3 standards:

• 200GbE / 100GbE / 50GbE / 40GbE / 25GbE / 10GbE / 1GbE
• IEEE 802.3bj, 802.3bm 100 Gigabit Ethernet
• IEEE 802.3by, Ethernet Consortium25, 50 Gigabit Ethernet, supporting all FEC modes
• IEEE 802.3ba 40 Gigabit Ethernet
• IEEE 802.3by 25 Gigabit Ethernet
• IEEE 802.3ae 10 Gigabit Ethernet
• IEEE 802.3ap based auto-negotiation and KR startup
• Proprietary Ethernet protocols (20/40GBASE-R2, 50GBASE-R4)
• IEEE 802.3ad, 802.1AX Link Aggregation
• IEEE 802.1Q, 802.1P VLAN tags and priority
• IEEE 802.1Qau (QCN)
• Congestion Notification
• IEEE 802.1Qaz (ETS)
• IEEE 802.1Qbb (PFC)
• IEEE 802.1Qbg
• IEEE 1588v2
• Jumbo frame support (9.6KB)

Memory Components

EEPROM - The EEPROM capacity is 32Kbit

SPI Quad - includes 256Mbit SPI Quad Flash device (MX25L25645GXDI-08G 
device by Macronix)

Overlay Networks

In order to better scale their networks, data center operators often create overlay 
networks that carry traffic from individual virtual machines over logical tunnels in 
encapsulated formats such as NVGRE and VXLAN. While this solves network scalability 
issues, it hides the TCP packet from the hardware offloading engines, placing higher 
loads on the host CPU. ConnectX®-6 Dx effectively addresses this by providing 
advanced NVGRE and VXLAN hardware offloading engines that encapsulate and de-
capsulate the overlay protocol.

RDMA and RDMA 

over Converged 

Ethernet (RoCE)

ConnectX®-6 Dx, utilizing RDMA (Remote Data Memory Access) and RoCE (RDMA over 
Converged Ethernet) technology, delivers low-latency and high-performance over Band 
and Ethernet networks. Leveraging data center bridging (DCB) capabilities as well as 
ConnectX®-6 Dx advanced congestion control hardware mechanisms, RoCE provides 
efficient low-latency RDMA services over Layer 2 and Layer 3 networks.

Mellanox 
PeerDirect®

PeerDirect

®

 communication provides high-efficiency RDMA access by eliminating 

unnecessary internal data copies between components on the PCIe bus (for example, 
from GPU to CPU), and therefore significantly reduces application run time. 
ConnectX®-6 Dx advanced acceleration technology enables higher cluster efficiency 
and scalability to tens of thousands of nodes.

CPU Offload

Adapter functionality enabling reduced CPU overhead allowing more available CPU for 
computation tasks.
Open vSwitch (OVS) offload using ASAP

2

 - Accelerated Switch and Packet Processing®

• Flexible match-action flow tables
• Tunneling encapsulation/decapsulation

Quality of Service 
(QoS)

Support for port-based Quality of Service enabling various application requirements for 
latency and SLA.

Hardware-based I/

O Virtualization

ConnectX®-6 Dx provides dedicated adapter resources and guaranteed isolation and 
protection for virtual machines within the server.

Summary of Contents for C-ADAB

Page 1: ...Exported on Oct 22 2020 06 48 PM https docs mellanox com x 2wPjAQ NVIDIA Mellanox ConnectX 6 Dx Ethernet Adapter Cards for OCP Spec 3 0 User Manual...

Page 2: ...8 SFP56 QSFP56 Interfaces 16 PCI Express Interface 16 Networking Ports LEDs Interface 17 SMBus Interface 18 Voltage Regulators 18 CPLD Interface 18 Heatsink 18 Hardware Installation 19 Safety Warnings...

Page 3: ...xtracting Files Without Running Installation 33 Uninstalling Mellanox WinOF 2 Driver 35 Attended Uninstallation 35 Unattended Uninstallation 35 Firmware Upgrade 35 Deploying the Driver on a Nano Serve...

Page 4: ...emoving Signature from kernel Modules 48 Performance Tuning 49 VMware Driver Installation 49 Hardware and Software Requirements 49 Installing Mellanox NATIVE ESXi Driver for VMware vSphere 50 Removing...

Page 5: ...5 Single port QSFP56 Thumbscrew Bracket 63 Finding the MAC and Serial Number on the Adapter Card 64 Document Revision History 66...

Page 6: ...pter card 50GbE OCP3 0 With Host Management Dual port SFP56 PCIe 4 0 x16 No Crypto Thumbscrew Pull Tab Bracket MCX623432A C GDAB ConnectX 6 Dx EN adapter card 50GbE OCP3 0 With Host Management Dual po...

Page 7: ...tion WinOF 2 for Windows User Manual and Release Notes User Manual describing WinOF 2 features performance Ethernet diagnostic tools content and configuration See WinOF 2 for Windows Documentation Mel...

Page 8: ...irect Attach Copper cables DACs copper splitter cables Active Optical Cables AOCs and transceivers in a wide range of lengths from 0 5m to 10km Mellanox tests every product in an end to end environmen...

Page 9: ...provides the ordering part number port speed number of ports and PCI Express speed Products Overview ConnectX 6 Dx 25GbE Adapter Cards for OCP 3 0 OPN MCX623432AN ADAB MCX623432AC ADAB MCX623432AE ADA...

Page 10: ...sical Function PF 4126 for Virtual Function VF ConnectX 6 Dx 100GbE Adapter Cards for OCP 3 0 OPN MCX623436AN CDAB MCX623436AC CDAB MCX623435MN CDAB MCX623436AS CDAI Network Connector Type Dual port Q...

Page 11: ...52 in x 2 99 in 115 00mm x 76 00mm Retention Mechanism Thumbscrew Pull Tab Bracket Thumbscrew Pull Tab Bracket Internal Lock Bracket RoHS RoHS Compliant RoHS Compliant RoHS Compliant IC Part Number MT...

Page 12: ...dvanced NVGRE and VXLAN hardware offloading engines that encapsulate and de capsulate the overlay protocol RDMA and RDMA over Converged Ethernet RoCE ConnectX 6 Dx utilizing RDMA Remote Data Memory Ac...

Page 13: ...Management Controller BMC interface supporting both NC SI and PLDM management protocols using these interfaces Mellanox OCP 3 0 adapters support these protocols to offer numerous Host Management feat...

Page 14: ...14...

Page 15: ...16 edge connector 3 Ethernet SFP28 SFP56 QSFP56 Interface Ethernet traffic is transmitted through the adapter s SFP28 SFP56 QSFP56 connectors The networking connectors allow for the use of modules opt...

Page 16: ...ta in motion and XTS AES data at rest cryptographic operations Stateful firewall solution acceleration powered by Open vSwitch connection tracking and Mellanox s ASAP2 technology Embedded hardware roo...

Page 17: ...the following Error Type Description LED Behavior I2C I2C access to the networking ports fails Blinks until the error is fixed Over current Over current condition of the networking ports Blinks until...

Page 18: ...3V supply pins These voltage supply pins feed onboard regulators that provide the necessary power to the various components on the card CPLD Interface The adapter card incorporates a CPLD device that...

Page 19: ...o Airflow Requirements 3 Follow the safety precautions Refer to Safety Precautions 4 Follow the pre installation checklist Refer to Pre Installation Checklist 5 Optional Replace the assembled OCP 3 0...

Page 20: ...ements See Operating Systems Distributions section under the Introduction section Unless otherwise specified Mellanox products are designed to work in an environmentally controlled data center with lo...

Page 21: ...g shipping Please note that the cards must be placed on an antistatic surface Shut down your system if active Turn off the power to the system and disconnect the power cord Refer to the system documen...

Page 22: ...in a system Cards with Thumbscrew Pull tab Bracket Before installing the card make sure that the system is off and the power cord is not connected to the server Please follow proper electrical ground...

Page 23: ...make sure that the system is off and the power cord is not connected to the server Please follow proper electrical grounding procedures Open the system case Align the card with the system rails Push t...

Page 24: ...is established that is when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a functioning port See Adapter Card LED Operations After pluggi...

Page 25: ...xample above notice the sub string PCI VEN_15B3 DEV_1003 VEN is equal to 0x15B3 this is the Vendor ID of Mellanox Technologies and DEV is equal to 1018 for ConnectX 6 Dx this is a valid Mellanox Techn...

Page 26: ...devices Turn off the system and disconnect the power cord from the server Extraction Instructions Verify that the system is powered off and unplugged Wait 30 seconds Rotate the captive screw countercl...

Page 27: ...ted above must run with administrator privileges Downloading Mellanox WinOF 2 Driver To download the executable file according to your operating system please follow the steps below Obtain the machine...

Page 28: ...you do not want to upgrade your firmware version Note MT_SKIPFWUPGRD default value is False MLNX_WinOF2 1_10_ revision_version _All_x64 exe v MT_SKIPFWUPGRD 1 Installing the incorrect exe file is pro...

Page 29: ...the installation The firmware upgrade screen will be displayed in the following cases If the user has an OEM card In this case the firmware will not be displayed If the user has a standard Mellanox ca...

Page 30: ...ion follow Step a and on Select the desired feature to install Performance tools install the performance tools that are used to measure performance in user environment Documentation contains the User...

Page 31: ...ools used for diagnostics such as mlx5cmd Click Next to install the desired tools Click Install to start the installation In case that firmware upgrade option was checked in Step 7 you will be notifie...

Page 32: ...ant to control whether to install ND provider or not Note MT_NDPROPERTY default value is True MLNX_WinOF2 1_10_All_x64 exe vMT_NDPROPERTY 1 Optional If you do not wish to upgrade your firmware version...

Page 33: ...ers OS To see the Mellanox network adapters display the Device Manager and pull down the Network adapters menu Extracting Files Without Running Installation To extract the files without running instal...

Page 34: ...3 4 5 Click Next to create a server image Click Change and specify the location in which the files are extracted to Click Install to extract this folder or click Change to install to a different fold...

Page 35: ...CMD console Click Task Manager File Run new task and enter CMD To uninstall the driver run MLNX_WinOF2 1_10_All_x64 exe S x v qn Firmware Upgrade If the machine has a standard Mellanox card with an ol...

Page 36: ...tPath InjectingDrivers vhdx DriversPath C WS2016TP5_Drivers Create the Nano server image Follow the instructions in the TechNet article for Getting Started with Nano Server Online Update Go to the Mel...

Page 37: ...mple shows a system with an installed Mellanox adapter card lspci v grep Mellanox 86 00 0 Network controller 0207 Mellanox Technologies Family Subsystem Mellanox Technologies Device 0014 86 00 1 Netwo...

Page 38: ...he deletion of the old packages If you need to install Mellanox OFED on an entire homogeneous cluster a common strategy is to mount the ISO image on one of the cluster nodes and then copy it to a shar...

Page 39: ...led the OpenSM Subnet Manager on your machine For the list of installation options run mlnxofedinstall h Installation Procedure Login to the installation machine as root Mount the ISO image on your ma...

Page 40: ...ectX 6 Dx Part Number MCX623436AC CDAB Description ConnectX 6 Dx EN adapter card 100GbE OCP3 0 With Host Management Dual port QSFP56 PCIe 4 0 x16 Crypto and Secure Boot Thumbscrew Pull Tab Bracket PSI...

Page 41: ...updated if the following two conditions are fulfilled The installation script is run in default mode that is without the option without fw update The firmware version of the adapter device is older t...

Page 42: ...process of updating the initramfs will add the blacklists from step 1 and will prevent the kernel from loading the modules automatically mlnxofedinstall Return Codes The table below lists the mlnxofed...

Page 43: ...cted HTTP request sent awaiting response 200 OK Length 1354 1 3K text plain Saving to RPM GPG KEY Mellanox 100 1 354 K s in 0s 2014 04 20 13 52 30 247 MB s RPM GPG KEY Mellanox saved 1354 1354 Install...

Page 44: ...port for RHEL and SLES For other operating systems kernel module RPM packages are provided only for the operating systems default kernel In this case the group RPM packages have the supported kernel v...

Page 45: ...operating systems Setting up MLNX_OFED apt get Repository Log into the installation machine as root Extract the MLNX_OFED pacakge on a shared location in your network You can download it from http www...

Page 46: ...pi installer package Install the desired group apt get install group name Example apt get install mlnx ofed all Uninstalling MLNX_OFED Using the apt get Tool Use the script usr sbin ofed_uninstall sh...

Page 47: ...om Support Firmware Download Get the device s PSID mlxfwmanager_pci grep PSID PSID MT_1210110019 Download the firmware BIN file from the Mellanox website or the OEM website Burn the firmware mlxfwmana...

Page 48: ...ill launch MokManager efi to allow you to complete the enrollment from the UEFI console You will need to enter the password you previously associated with this request and confirm the enrollment Once...

Page 49: ...ssage will appear Note that this message cannot be eliminated Update the initramfs on RHEL systems with the stripped modules mkinitrd boot initramfs uname r img uname r force Performance Tuning Depend...

Page 50: ...7 01 31 Removing Earlier Mellanox Drivers To remove all the drivers Log into the ESXi server with root permissions List all the existing NATIVE ESXi driver modules See Step 4 in Installing Mellanox NA...

Page 51: ...are bootable binary images from the Mellanox Firmware Tools MFT site Install the image according to the steps described in the MFT User Manual The following procedure requires custom boot image downlo...

Page 52: ...xup Querying Mellanox devices firmware Device Type ConnectX 6 Dx Part Number MCX623436AC CDAB Description ConnectX 6 Dx EN adapter card 100GbE OCP3 0 With Host Management Dual port QSFP56 PCIe 4 0 x16...

Page 53: ...ng another adapter Try removing and re installing all adapters Check that cables are connected properly Make sure your motherboard has the latest BIOS Link indicator light is off Try another port on t...

Page 54: ...ows desktop choose the Start menu and run msinfo32 To export system information to a text file choose the Export option from the File menu Assign a file name and save Mellanox Firmware Tool MFT Downlo...

Page 55: ...16 lanes 2 0 and 1 1 compatible Power and Environmental Voltage 3 3VAUX 12V Power Cable Type Active Mode Standby Mode Typical Power b Passive Cables 15 7W 3 8W Maximum Power Passive Cables 19 74W 7 95...

Page 56: ...raffic load c For both operational and non operational states MCX623432A N C GDAB MCX623432AS GDAI Specifications Physical Form Factor Small Form Factor SFF OCP 3 0 Size 4 52 in x 2 99 in 115 00mm x 7...

Page 57: ...to HSK Active Mode 55C Standby Mode 45C Active Mode Standby Mode Passive Cable 400LFM 100LFM TBD TBD Active 2W Cable 800LFM 200LFM TBD TBD Regulatory Safety CB cTUVus CE EMC CE FCC VCCI ICES RCM RoHS...

Page 58: ...e Cables 27 4W 14W Maximum power available through QSFP56 port 3 5W Temperature Operational 0 C to 55 C Non operational 40 C to 70 C Humidity 90 relative humidity c Altitude Operational 3050m Environ...

Page 59: ...ernet 1 10 25 40 50 100 Gb s PCI Express Gen 4 0 SERDES 16 0GT s 16 lanes 2 0 and 1 1 compatible Powe r and Envir onme ntal Voltage 3 3VAUX 12V Power Cable Type Active Mode Standby Mode Typical Power...

Page 60: ...bscrew Pull Tab Bracket MCX623435AS VDAI Internal Lock Protocol Support Ethernet 200GBASE CR4 200GBASE KR4 200GBASE SR4 100GBASE CR4 100GBASE KR4 100GBASE SR4 50GBASE R2 50GBASE R4 40GBASE CR4 40GBASE...

Page 61: ...x adapters supplement the auto negotiation specification to get better bit error rates and longer cable reaches This supplemental feature only initiates when connected to another Mellanox Ethernet pro...

Page 62: ...62 ConnectX 6 Dx Dual Port QSFP56 OCP 3 0 Mechanical Drawing and Dimensions ConnectX 6 Dx Single Port QSFP56 OCP 3 0 Mechanical Drawing and Dimensions...

Page 63: ...l Drawings and Dimensions Dual port SFP28 SFP56 Thumbscrew Bracket Dual port QSFP56 Thumbscrew Bracket Single port QSFP56 Thumbscrew Bracket All dimensions are in millimeters The bracket s mechanical...

Page 64: ...ial number and the card MAC for the Ethernet protocol MCX623432AN ADAB Board Label Example MCX623432AC ADAB Board Label Example MCX623432AN GDAB Board Label Example MCX623432AC GDAB Board Label Exampl...

Page 65: ...65 MCX623436AN CDAB Board Label Example MCX623436AC CDAB Board Label Example MCX623435AN VDAB Board Label Example MCX623435AC VDAB Board Label Example...

Page 66: ...g ports LEDs indications Sep 2020 1 5 Added OCP 3 0 bracket replacement instructions Jul 2020 1 4 Updated airflow numbers in Specifications Jul 2020 1 3 Updated all power and airflow numbers in Specif...

Page 67: ...arily performed by NVIDIA It is customer s sole responsibility to evaluate and determine the applicability of any information contained in this document ensure the product is suitable and fit for the...

Page 68: ...A the NVIDIA logo and Mellanox are trademarks and or registered trademarks of Mellanox Technologies Ltd and or NVIDIA Corporation in the U S and in other countries Other company and product names may...

Reviews: