background image

8

IEEE Std 802.3 

Specification

IEEE Ethernet specification at 

http://standards.ieee.org/

PCI Express 

Specifications

Industry Standard PCI Express Base and Card Electromechanical Specifications at 

https://pcisig.com/specifications

Mellanox LinkX 

Interconnect 

Solutions

Mellanox LinkX Ethernet cables and transceivers are designed to maximize the 
performance of High-Performance Computing networks, requiring high-
bandwidth, low-latency connections between compute nodes and switch nodes. 
Mellanox offers one of industry’s broadest portfolio 40Gb/s, 56Gb/s, 100Gb/s and 
200Gb/s cables, including Direct Attach Copper cables (DACs), copper splitter 
cables, Active Optical Cables (AOCs) and transceivers in a wide range of lengths 
from 0.5m to 10km.Mellanox tests every product in an end-to-end environment 
ensuring a Bit Error Rate of less than 1E-15. Read more at 

https://

www.mellanox.com/products/interconnect/infiniband-overview.php

Document Conventions

When discussing memory sizes, MB and MBytes are used in this document to mean size in mega 
Bytes. The use of Mb or Mbits (small b) indicates size in mega bits. In this document PCIe is used to 
mean PCI Express.

Revision History

A list of the changes made to this document are provided in 

Document Revision History

.

Содержание Mellanox MCX621102ACADAT

Страница 1: ...Exported on Oct 22 2020 06 45 PM https docs mellanox com x esV AQ NVIDIA Mellanox ConnectX 6 Dx PCIe HHHL Ethernet Adapter Cards User Manual...

Страница 2: ...cheme 3 One Bi Color LED and one Single Color LED 17 Voltage Regulators 18 Heatsink 18 Hardware Installation 19 Safety Warnings 19 Installation Procedure Overview 19 System Requirements 20 Hardware Re...

Страница 3: ...Using the YUM Tool 36 Installing MLNX_OFED Using apt get Tool 36 Setting up MLNX_OFED apt get Repository 37 Installing MLNX_OFED Using the apt get Tool 37 Uninstalling MLNX_OFED Using the apt get Too...

Страница 4: ...55 Windows Troubleshooting 56 Specifications 57 MCX621102A C E N ADAT Specifications 57 MCX623102A C N ADAT Specifications 58 MCX623102A C N S E GDAT Specifications 59 MCX623106A C N S E CDAT Specific...

Страница 5: ...5 Short Bracket 66 Tall Bracket 66 Dual Port QSFP56 Adapter Card 67 Short Bracket 67 Tall Bracket 67 Finding the MAC on the Adapter Card 68 Document Revision History 71...

Страница 6: ...CIe 4 0 x16 No Crypto Tall Bracket MCX623102AS GDAT ConnectX 6 Dx EN adapter card 50GbE Dual port SFP56 PCIe 4 0 x16 Secure Boot No Crypto Tall Bracket MCX623102AE GDAT ConnectX 6 Dx EN adapter card 5...

Страница 7: ...please see your contract for details regarding Technical Support Customers who purchased Mellanox products through a Mellanox approved reseller should first seek assistance through their reseller Rela...

Страница 8: ...portfolio 40Gb s 56Gb s 100Gb s and 200Gb s cables including Direct Attach Copper cables DACs copper splitter cables Active Optical Cables AOCs and transceivers in a wide range of lengths from 0 5m t...

Страница 9: ...ease refer to Fe ature and Benefits for more details Mellanox ConnectX 6 Dx 25GbE Adapter Cards Part Number MCX621102AC ADAT MCX621102AE ADAT MCX621102AN ADAT MCX623102AC ADAT MCX623102AN ADAT Form Fa...

Страница 10: ...al port QSFP56 PCIe Interface PCIe Gen 4 0 SERDES 16 0GT s x16 Crypto Secure Boot Adapter IC Part Number MT28928A0 XCCF CE MT28928A0 XCCF CE MT28928A0 CCCF CE MT28928A0 CCCF CE MT28928A0 NCCF CE MT289...

Страница 11: ...face PCIe Gen 4 0 SERDES 16 0GT s x16 Crypto Secure Boot Adapter IC Part Number MT28928A0 CCCF CE MT28928A0 CCCF CE MT28928A0 NCCF CE MT28928A0 NCCF CE RoHS RoHS Compliant For more detailed informatio...

Страница 12: ...on the host CPU Mellanox ConnectX 6 Dx effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that encapsulate and de capsulate the overlay protocol RDMA over Con...

Страница 13: ...that allows ConnectX 6 Dx to achieve sub 20u Sec accuracy and also offers many timing related functions such as time triggered scheduling or time based SND accelerations time based ASAP Furthermore th...

Страница 14: ...MC Mellanox ConnectX 6 Dx PCIe stand up adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as if it is a standard Mellanox PCIe stand up adapter For configuring the ad...

Страница 15: ...et SFP28 SFP56 QSFP56 Interface Ethernet traffic is transmitted through the adapter s SFP28 SFP56 QSFP56 connectors The networking connectors allow for the use of modules optical and passive cable int...

Страница 16: ...The network ports of the ConnectX 6 Dx adapter card are compliant with the IEEE 802 3 Ethernet standards listed in Features and Benefits Ethernet traffic is transmitted through the SFP28 SFP56 QSFP56...

Страница 17: ...Physical Activity A constant Green indicates a physical link with the maximum networking speed A constant Yellow indicates a physical link with less than the maximum networking speed Link Up A consta...

Страница 18: ...er SMBus or MCTP over PCIe protocols as if it is a standard Mellanox PCIe stand up adapter For configuring the adapter for the specific manageability solution in use by the server please contact Mella...

Страница 19: ...ents 2 Pay attention to the airflow consideration within the host system Refer to Airflow Requirements 3 Follow the safety precautions Refer to Safety Precautions 4 Unpack the package Refer to Unpack...

Страница 20: ...em observe the following precautions to avoid injury and prevent damage to system components Remove any metallic objects from your hands and wrists Make sure to use only insulated tools Verify that th...

Страница 21: ...e Shut down your system if active Turn off the power to the system and disconnect the power cord Refer to the system documentation for instructions Before you install the ConnectX 6 Dx card make sure...

Страница 22: ...the bracket from the ConnectX 6 Dx card Save the two screws Installing the New Bracket Place the bracket onto the card until the screw holes line up Screw on the bracket using the screws saved from t...

Страница 23: ...card insert the adapter card in a PCI Express slot until firmly seated Secure the adapter card to the chassis Step 1 Secure the bracket to the chassis with the bracket screw Do not use excessive forc...

Страница 24: ...off the system and disconnect the power cord from the server Card Removal Verify that the system is powered off and unplugged Wait 30 seconds To remove the card disengage the retention mechanisms on t...

Страница 25: ...ished that is when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a functioning port See LED Interface under the Interfaces section After...

Страница 26: ...example above notice the sub string PCI VEN_15B3 DEV_1003 VEN is equal to 0x15B3 this is the Vendor ID of Mellanox Technologies and DEV is equal to 1018 for ConnectX 6 Dx this is a valid Mellanox Tech...

Страница 27: ...r cord from the server Card Removal Verify that the system is powered off and unplugged Wait 30 seconds To remove the card disengage the retention mechanisms on the bracket clips or screws Holding the...

Страница 28: ...vileges on the target machine Downloading Mellanox OFED Verify that the system has a Mellanox network adapter installed The following example shows a system with an installed Mellanox adapter card lsp...

Страница 29: ...ing customized FW binaries you can provide a path to the folder that contains the FW binary files by running fw image dir Using this option the FW version embedded in the MLNX_OFED package will be ign...

Страница 30: ...RAM For your machine to be part of the fabric a Subnet Manager must be running on one of the fabric nodes At this point Mellanox OFED for Linux has already installed the OpenSM Subnet Manager on your...

Страница 31: ...s in the root environment mlnxofedinstall without dkms add kernel support kernel kernel version in chroot without fw update force For example mlnxofedinstall without dkms add kernel support kernel 3 1...

Страница 32: ...the usr directory except for the following packages which are installed under the opt directory fca and ibutils The kernel modules are installed under lib modules uname r updates on SLES and Fedora Di...

Страница 33: ...FED the install log for each selected package will be saved in a separate log file The path to the directory containing the log files will be displayed after running the installation script in the fol...

Страница 34: ...e space 171 Not applicable to this system configuration This can occur when the required hardware is not present on the system 172 Prerequisites are not met For example missing the required software i...

Страница 35: ...content mlnx_ofed name MLNX_OFED Repository baseurl file path to extracted MLNX_OFED package RPMS enabled 1 gpgkey file path to the downloaded key RPM GPG KEY Mellanox gpgcheck 1 Check that the reposi...

Страница 36: ...mlnx ofed vma 3 17 4 301 fc21 x86_64 noarch MLNX_OFED vma installer package for kernel 3 17 4 301 fc21 x8 6_64 without KMP support mlnx ofed vma eth 3 17 4 301 fc21 x86_64 noarch MLNX_OFED vma eth ins...

Страница 37: ...D Using the apt get Tool After setting up the apt get repository for MLNX_OFED package perform the following View the available package groups by invoking pre apt cache search mlnx ofed mlnx ofed vma...

Страница 38: ...vailable FW 2 32 5000 2 33 5000 Status Update required Found 1 device s requiring firmware update Please use u flag to perform the update Updating the Device Manually To update the device manually ple...

Страница 39: ...ameter to yes RUN_FW_UPDATER_ONBOOT yes in the openibd service configuration file etc infiniband openib conf You can opt to exclude a list of devices from the automatic firmware update procedure To do...

Страница 40: ...a copy prior to removing the signature To remove the signature from the MLNX_OFED kernel modules Remove the signature rpm qa grep E kernel ib mlnx ofa_kernel iser srp knem mlnx rds mlnx nfsrdma mlnx...

Страница 41: ...Description Package Windows Server 2012 R2 MLNX_WinOF2 2_10_All_x64 exe Windows Server 2012 Windows Server 2016 Windows Server 2019 Windows 8 1 Client 64 bit only Windows 10 Client 64 bit only Note T...

Страница 42: ...tructions to install MLNX_WinOF2 Optional Manually configure your setup to contain the logs option replace LogFile with the relevant directory MLNX_WinOF2 Driver Version _ revision_version _All_Arch e...

Страница 43: ...43 4 5 6 Click Next in the Welcome screen Read and accept the license agreement and click Next Select the target folder for the installation...

Страница 44: ...user has an OEM card In this case the firmware will not be displayed If the user has a standard Mellanox card with an older firmware version the firmware will be updated accordingly However if the use...

Страница 45: ...install the performance tools that are used to measure performance in user environment Documentation contains the User Manual and Release Notes Management tools installation tools used for management...

Страница 46: ...9 Click Install to start the installation 10 In case firmware upgrade option was checked in Step 7 you will be notified if a firmware upgrade is required see 11 Click Finish to complete the installati...

Страница 47: ...ptional if you wish to control whether to install ND provider or not i e MT_NDPROPERTY default value is True MLNX_WinOF2 Driver Version _ revision_version _All_Arch exe vMT_NDPROPERTY 1 Optional If yo...

Страница 48: ...Drivers To see the Mellanox network adapters display the Device Manager and pull down the Network adapters menu Uninstalling Mellanox WinOF 2 Driver Attended Uninstallation To uninstall MLNX_WinOF2 o...

Страница 49: ...without running installation perform the following steps Open a CMD console Click Start Task Manager File Run new task and enter CMD Extract the driver and the tools MLNX_WinOF2 2_0_ revision_version...

Страница 50: ...50 5 Click Install to extract this folder or click Change to install to a different folder...

Страница 51: ...er to the MFT User Manual at www m ellanox com Products Ethernet Drivers Firmware Tools VMware Driver Installation This section describes VMware Driver Installation Hardware and Software Requirements...

Страница 52: ...0 4240417 zip Reboot the machine Verify the driver was installed successfully esxcli software vib list grep nmlx nmlx5 core 4 16 8 8 1OEM 650 0 0 4240417 MEL PartnerSupported 2017 01 31 nmlx5 rdma 4 1...

Страница 53: ...the server Firmware Programming Download the VMware bootable binary images v4 6 0 from the Mellanox Firmware Tools MFT site ESXi 6 5 File mft 4 6 0 48 10EM 650 0 0 4598673 x86_64 vib MD5SUM 0804cffe30...

Страница 54: ...ware Tools Firmware Update Example server1 mlxup Querying Mellanox devices firmware Device Type ConnectX 6 Dx Part Number MCX623105AN VDAT Description ConnectX 6 Dx EN adapter card 200GbE Single port...

Страница 55: ...ng another adapter Try removing and re installing all adapters Check that cables are connected properly Make sure your motherboard has the latest BIOS Link indicator light is off Try another port on t...

Страница 56: ...ows desktop choose the Start menu and run msinfo32 To export system information to a text file choose the Export option from the File menu Assign a file name and save Mellanox Firmware Tool MFT Downlo...

Страница 57: ...mum current 100mA Environmen tal Temperature Operational 0 C to 55 C Non operational 40 C to 70 C Humidity 90 relative humidity c Altitude Operational 3050m Airflow Requirements 55C d e Cable Type Hot...

Страница 58: ...current 100mA Environmenta l Temperature Operational 0 C to 55 C Non operational 40 C to 70 C Humidity 90 relative humidity c Altitude Operational 3050m Airflow Requirements 55C d e Cable Type Hot Ais...

Страница 59: ...through SFP56 port 2 5W each port Voltage 3 3Aux Maximum current 100mA Environmental Temperature Operational 0 C to 55 C Non operational 40 C to 70 C Humidity 90 relative humidity c Altitude Operation...

Страница 60: ...m power available through QSFP56 port 5W each port Voltage 3 3Aux Maximum current 100mA Environme ntal Temperature Operational 0 C to 55 C Non operational 40 C to 70 C Humidity 90 relative humidity c...

Страница 61: ...available through QSFP56 port 5W each port Voltage 3 3Aux Maximum current 100mA Environme ntal Temperature Operational 0 C to 55 C Non operational 40 C to 70 C Humidity 90 relative humidity c Altitude...

Страница 62: ...ower available through QSFP56 port 5W each port Voltage 3 3Aux Maximum current 100mA Environm ental Temperature Operational 0 C to 55 C Non operational 40 C to 70 C Humidity 90 relative humidity c Alt...

Страница 63: ...D 24W Maximum power available through QSFP56 port 5W each port Voltage 3 3Aux Maximum current 100mA Environmental Temperature Operational 0 C to 55 C Non operational 40 C to 70 C Humidity 90 relative...

Страница 64: ...chanical Drawing Dual Port SFP28 SFP56 x16 Adapter Cards Mechanical Drawing For the 3D Model of the card please refer to 3D Models at http www mellanox com page 3d_models All dimensions are in millime...

Страница 65: ...le Port QSFP56 x16 Adapter Cards Mechanical Drawing Dual Port QSFP56 x16 Adapter Cards Mechanical Drawing Bracket Mechanical Drawing All dimensions are in millimeters All the mechanical tolerances are...

Страница 66: ...66 Dual Port SFP28 SFP56 Adapter Card Short Bracket Tall Bracket Single Port QSFP56 Adapter Card Short Bracket Tall Bracket...

Страница 67: ...67 Dual Port QSFP56 Adapter Card Short Bracket Tall Bracket...

Страница 68: ...rotocol MCX621102AC ADAT Board Label Example MCX621102AN ADAT Board Label Example MCX623102AC ADAT Board Label Example MCX623102AN ADAT Board Label Example MCX623102AC GDAT Board Label Example MCX6231...

Страница 69: ...AS GDAT Board Label Example MCX623106AS CDAT Board Label Example MCX623106AC CDAT Board Label Example MCX623106AN CDAT Board Label Example MCX623105AC VDAT Board Label Example MCX623105AN VDAT Board L...

Страница 70: ...70 MCX623105AS VDAT Board Label Example...

Страница 71: ...pecifications Jul 2020 1 4 Updated power numbers in Specifications Jun 2020 1 3 Updated airflow numbers Added the following OPNs to all relevant sections MCX621102AE ADAT MCX623102AS GDAT MCX623102AC...

Страница 72: ...arily performed by NVIDIA It is customer s sole responsibility to evaluate and determine the applicability of any information contained in this document ensure the product is suitable and fit for the...

Страница 73: ...A the NVIDIA logo and Mellanox are trademarks and or registered trademarks of Mellanox Technologies Ltd and or NVIDIA Corporation in the U S and in other countries Other company and product names may...

Отзывы: