Nvidia Mellanox MCX621102ACADAT User Manual Download Page 1

Page: 1 / 73

Exported on Oct/22/2020 06:45 PM

https://docs.mellanox.com/x/esV-AQ

NVIDIA Mellanox ConnectX-6 Dx PCIe HHHL

Ethernet Adapter Cards User Manual

Summary of Contents for Mellanox MCX621102ACADAT

Page 1: ...Exported on Oct 22 2020 06 45 PM https docs mellanox com x esV AQ NVIDIA Mellanox ConnectX 6 Dx PCIe HHHL Ethernet Adapter Cards User Manual...

Page 2: ...cheme 3 One Bi Color LED and one Single Color LED 17 Voltage Regulators 18 Heatsink 18 Hardware Installation 19 Safety Warnings 19 Installation Procedure Overview 19 System Requirements 20 Hardware Re...

Page 3: ...Using the YUM Tool 36 Installing MLNX_OFED Using apt get Tool 36 Setting up MLNX_OFED apt get Repository 37 Installing MLNX_OFED Using the apt get Tool 37 Uninstalling MLNX_OFED Using the apt get Too...

Page 4: ...55 Windows Troubleshooting 56 Specifications 57 MCX621102A C E N ADAT Specifications 57 MCX623102A C N ADAT Specifications 58 MCX623102A C N S E GDAT Specifications 59 MCX623106A C N S E CDAT Specific...

Page 5: ...5 Short Bracket 66 Tall Bracket 66 Dual Port QSFP56 Adapter Card 67 Short Bracket 67 Tall Bracket 67 Finding the MAC on the Adapter Card 68 Document Revision History 71...

Page 6: ...CIe 4 0 x16 No Crypto Tall Bracket MCX623102AS GDAT ConnectX 6 Dx EN adapter card 50GbE Dual port SFP56 PCIe 4 0 x16 Secure Boot No Crypto Tall Bracket MCX623102AE GDAT ConnectX 6 Dx EN adapter card 5...

Page 7: ...please see your contract for details regarding Technical Support Customers who purchased Mellanox products through a Mellanox approved reseller should first seek assistance through their reseller Rela...

Page 8: ...portfolio 40Gb s 56Gb s 100Gb s and 200Gb s cables including Direct Attach Copper cables DACs copper splitter cables Active Optical Cables AOCs and transceivers in a wide range of lengths from 0 5m t...

Page 9: ...ease refer to Fe ature and Benefits for more details Mellanox ConnectX 6 Dx 25GbE Adapter Cards Part Number MCX621102AC ADAT MCX621102AE ADAT MCX621102AN ADAT MCX623102AC ADAT MCX623102AN ADAT Form Fa...

Page 10: ...al port QSFP56 PCIe Interface PCIe Gen 4 0 SERDES 16 0GT s x16 Crypto Secure Boot Adapter IC Part Number MT28928A0 XCCF CE MT28928A0 XCCF CE MT28928A0 CCCF CE MT28928A0 CCCF CE MT28928A0 NCCF CE MT289...

Page 11: ...face PCIe Gen 4 0 SERDES 16 0GT s x16 Crypto Secure Boot Adapter IC Part Number MT28928A0 CCCF CE MT28928A0 CCCF CE MT28928A0 NCCF CE MT28928A0 NCCF CE RoHS RoHS Compliant For more detailed informatio...

Page 12: ...on the host CPU Mellanox ConnectX 6 Dx effectively addresses this by providing advanced NVGRE and VXLAN hardware offloading engines that encapsulate and de capsulate the overlay protocol RDMA over Con...

Page 13: ...that allows ConnectX 6 Dx to achieve sub 20u Sec accuracy and also offers many timing related functions such as time triggered scheduling or time based SND accelerations time based ASAP Furthermore th...

Page 14: ...MC Mellanox ConnectX 6 Dx PCIe stand up adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as if it is a standard Mellanox PCIe stand up adapter For configuring the ad...

Page 15: ...et SFP28 SFP56 QSFP56 Interface Ethernet traffic is transmitted through the adapter s SFP28 SFP56 QSFP56 connectors The networking connectors allow for the use of modules optical and passive cable int...

Page 16: ...The network ports of the ConnectX 6 Dx adapter card are compliant with the IEEE 802 3 Ethernet standards listed in Features and Benefits Ethernet traffic is transmitted through the SFP28 SFP56 QSFP56...

Page 17: ...Physical Activity A constant Green indicates a physical link with the maximum networking speed A constant Yellow indicates a physical link with less than the maximum networking speed Link Up A consta...

Page 18: ...er SMBus or MCTP over PCIe protocols as if it is a standard Mellanox PCIe stand up adapter For configuring the adapter for the specific manageability solution in use by the server please contact Mella...

Page 19: ...ents 2 Pay attention to the airflow consideration within the host system Refer to Airflow Requirements 3 Follow the safety precautions Refer to Safety Precautions 4 Unpack the package Refer to Unpack...

Page 20: ...em observe the following precautions to avoid injury and prevent damage to system components Remove any metallic objects from your hands and wrists Make sure to use only insulated tools Verify that th...

Page 21: ...e Shut down your system if active Turn off the power to the system and disconnect the power cord Refer to the system documentation for instructions Before you install the ConnectX 6 Dx card make sure...

Page 22: ...the bracket from the ConnectX 6 Dx card Save the two screws Installing the New Bracket Place the bracket onto the card until the screw holes line up Screw on the bracket using the screws saved from t...

Page 23: ...card insert the adapter card in a PCI Express slot until firmly seated Secure the adapter card to the chassis Step 1 Secure the bracket to the chassis with the bracket screw Do not use excessive forc...

Page 24: ...off the system and disconnect the power cord from the server Card Removal Verify that the system is powered off and unplugged Wait 30 seconds To remove the card disengage the retention mechanisms on t...

Page 25: ...ished that is when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a functioning port See LED Interface under the Interfaces section After...

Page 26: ...example above notice the sub string PCI VEN_15B3 DEV_1003 VEN is equal to 0x15B3 this is the Vendor ID of Mellanox Technologies and DEV is equal to 1018 for ConnectX 6 Dx this is a valid Mellanox Tech...

Page 27: ...r cord from the server Card Removal Verify that the system is powered off and unplugged Wait 30 seconds To remove the card disengage the retention mechanisms on the bracket clips or screws Holding the...

Page 28: ...vileges on the target machine Downloading Mellanox OFED Verify that the system has a Mellanox network adapter installed The following example shows a system with an installed Mellanox adapter card lsp...

Page 29: ...ing customized FW binaries you can provide a path to the folder that contains the FW binary files by running fw image dir Using this option the FW version embedded in the MLNX_OFED package will be ign...

Page 30: ...RAM For your machine to be part of the fabric a Subnet Manager must be running on one of the fabric nodes At this point Mellanox OFED for Linux has already installed the OpenSM Subnet Manager on your...

Page 31: ...s in the root environment mlnxofedinstall without dkms add kernel support kernel kernel version in chroot without fw update force For example mlnxofedinstall without dkms add kernel support kernel 3 1...

Page 32: ...the usr directory except for the following packages which are installed under the opt directory fca and ibutils The kernel modules are installed under lib modules uname r updates on SLES and Fedora Di...

Page 33: ...FED the install log for each selected package will be saved in a separate log file The path to the directory containing the log files will be displayed after running the installation script in the fol...

Page 34: ...e space 171 Not applicable to this system configuration This can occur when the required hardware is not present on the system 172 Prerequisites are not met For example missing the required software i...

Page 35: ...content mlnx_ofed name MLNX_OFED Repository baseurl file path to extracted MLNX_OFED package RPMS enabled 1 gpgkey file path to the downloaded key RPM GPG KEY Mellanox gpgcheck 1 Check that the reposi...

Page 36: ...mlnx ofed vma 3 17 4 301 fc21 x86_64 noarch MLNX_OFED vma installer package for kernel 3 17 4 301 fc21 x8 6_64 without KMP support mlnx ofed vma eth 3 17 4 301 fc21 x86_64 noarch MLNX_OFED vma eth ins...

Page 37: ...D Using the apt get Tool After setting up the apt get repository for MLNX_OFED package perform the following View the available package groups by invoking pre apt cache search mlnx ofed mlnx ofed vma...

Page 38: ...vailable FW 2 32 5000 2 33 5000 Status Update required Found 1 device s requiring firmware update Please use u flag to perform the update Updating the Device Manually To update the device manually ple...

Page 39: ...ameter to yes RUN_FW_UPDATER_ONBOOT yes in the openibd service configuration file etc infiniband openib conf You can opt to exclude a list of devices from the automatic firmware update procedure To do...

Page 40: ...a copy prior to removing the signature To remove the signature from the MLNX_OFED kernel modules Remove the signature rpm qa grep E kernel ib mlnx ofa_kernel iser srp knem mlnx rds mlnx nfsrdma mlnx...

Page 41: ...Description Package Windows Server 2012 R2 MLNX_WinOF2 2_10_All_x64 exe Windows Server 2012 Windows Server 2016 Windows Server 2019 Windows 8 1 Client 64 bit only Windows 10 Client 64 bit only Note T...

Page 42: ...tructions to install MLNX_WinOF2 Optional Manually configure your setup to contain the logs option replace LogFile with the relevant directory MLNX_WinOF2 Driver Version _ revision_version _All_Arch e...

Page 43: ...43 4 5 6 Click Next in the Welcome screen Read and accept the license agreement and click Next Select the target folder for the installation...

Page 44: ...user has an OEM card In this case the firmware will not be displayed If the user has a standard Mellanox card with an older firmware version the firmware will be updated accordingly However if the use...

Page 45: ...install the performance tools that are used to measure performance in user environment Documentation contains the User Manual and Release Notes Management tools installation tools used for management...

Page 46: ...9 Click Install to start the installation 10 In case firmware upgrade option was checked in Step 7 you will be notified if a firmware upgrade is required see 11 Click Finish to complete the installati...

Page 47: ...ptional if you wish to control whether to install ND provider or not i e MT_NDPROPERTY default value is True MLNX_WinOF2 Driver Version _ revision_version _All_Arch exe vMT_NDPROPERTY 1 Optional If yo...

Page 48: ...Drivers To see the Mellanox network adapters display the Device Manager and pull down the Network adapters menu Uninstalling Mellanox WinOF 2 Driver Attended Uninstallation To uninstall MLNX_WinOF2 o...

Page 49: ...without running installation perform the following steps Open a CMD console Click Start Task Manager File Run new task and enter CMD Extract the driver and the tools MLNX_WinOF2 2_0_ revision_version...

Page 50: ...50 5 Click Install to extract this folder or click Change to install to a different folder...

Page 51: ...er to the MFT User Manual at www m ellanox com Products Ethernet Drivers Firmware Tools VMware Driver Installation This section describes VMware Driver Installation Hardware and Software Requirements...

Page 52: ...0 4240417 zip Reboot the machine Verify the driver was installed successfully esxcli software vib list grep nmlx nmlx5 core 4 16 8 8 1OEM 650 0 0 4240417 MEL PartnerSupported 2017 01 31 nmlx5 rdma 4 1...

Page 53: ...the server Firmware Programming Download the VMware bootable binary images v4 6 0 from the Mellanox Firmware Tools MFT site ESXi 6 5 File mft 4 6 0 48 10EM 650 0 0 4598673 x86_64 vib MD5SUM 0804cffe30...

Page 54: ...ware Tools Firmware Update Example server1 mlxup Querying Mellanox devices firmware Device Type ConnectX 6 Dx Part Number MCX623105AN VDAT Description ConnectX 6 Dx EN adapter card 200GbE Single port...

Page 55: ...ng another adapter Try removing and re installing all adapters Check that cables are connected properly Make sure your motherboard has the latest BIOS Link indicator light is off Try another port on t...

Page 56: ...ows desktop choose the Start menu and run msinfo32 To export system information to a text file choose the Export option from the File menu Assign a file name and save Mellanox Firmware Tool MFT Downlo...

Page 57: ...mum current 100mA Environmen tal Temperature Operational 0 C to 55 C Non operational 40 C to 70 C Humidity 90 relative humidity c Altitude Operational 3050m Airflow Requirements 55C d e Cable Type Hot...

Page 58: ...current 100mA Environmenta l Temperature Operational 0 C to 55 C Non operational 40 C to 70 C Humidity 90 relative humidity c Altitude Operational 3050m Airflow Requirements 55C d e Cable Type Hot Ais...

Page 59: ...through SFP56 port 2 5W each port Voltage 3 3Aux Maximum current 100mA Environmental Temperature Operational 0 C to 55 C Non operational 40 C to 70 C Humidity 90 relative humidity c Altitude Operation...

Page 60: ...m power available through QSFP56 port 5W each port Voltage 3 3Aux Maximum current 100mA Environme ntal Temperature Operational 0 C to 55 C Non operational 40 C to 70 C Humidity 90 relative humidity c...

Page 61: ...available through QSFP56 port 5W each port Voltage 3 3Aux Maximum current 100mA Environme ntal Temperature Operational 0 C to 55 C Non operational 40 C to 70 C Humidity 90 relative humidity c Altitude...

Page 62: ...ower available through QSFP56 port 5W each port Voltage 3 3Aux Maximum current 100mA Environm ental Temperature Operational 0 C to 55 C Non operational 40 C to 70 C Humidity 90 relative humidity c Alt...

Page 63: ...D 24W Maximum power available through QSFP56 port 5W each port Voltage 3 3Aux Maximum current 100mA Environmental Temperature Operational 0 C to 55 C Non operational 40 C to 70 C Humidity 90 relative...

Page 64: ...chanical Drawing Dual Port SFP28 SFP56 x16 Adapter Cards Mechanical Drawing For the 3D Model of the card please refer to 3D Models at http www mellanox com page 3d_models All dimensions are in millime...

Page 65: ...le Port QSFP56 x16 Adapter Cards Mechanical Drawing Dual Port QSFP56 x16 Adapter Cards Mechanical Drawing Bracket Mechanical Drawing All dimensions are in millimeters All the mechanical tolerances are...

Page 66: ...66 Dual Port SFP28 SFP56 Adapter Card Short Bracket Tall Bracket Single Port QSFP56 Adapter Card Short Bracket Tall Bracket...

Page 67: ...67 Dual Port QSFP56 Adapter Card Short Bracket Tall Bracket...

Page 68: ...rotocol MCX621102AC ADAT Board Label Example MCX621102AN ADAT Board Label Example MCX623102AC ADAT Board Label Example MCX623102AN ADAT Board Label Example MCX623102AC GDAT Board Label Example MCX6231...

Page 69: ...AS GDAT Board Label Example MCX623106AS CDAT Board Label Example MCX623106AC CDAT Board Label Example MCX623106AN CDAT Board Label Example MCX623105AC VDAT Board Label Example MCX623105AN VDAT Board L...

Page 70: ...70 MCX623105AS VDAT Board Label Example...

Page 71: ...pecifications Jul 2020 1 4 Updated power numbers in Specifications Jun 2020 1 3 Updated airflow numbers Added the following OPNs to all relevant sections MCX621102AE ADAT MCX623102AS GDAT MCX623102AC...

Page 72: ...arily performed by NVIDIA It is customer s sole responsibility to evaluate and determine the applicability of any information contained in this document ensure the product is suitable and fit for the...

Page 73: ...A the NVIDIA logo and Mellanox are trademarks and or registered trademarks of Mellanox Technologies Ltd and or NVIDIA Corporation in the U S and in other countries Other company and product names may...

Search results

Summary of Contents for Mellanox MCX621102ACADAT

Reviews:

Related manuals for Mellanox MCX621102ACADAT

Brands by name

Popular brands

Nvidia Mellanox MCX621102ACADAT, User Manual

Search results

Summary of Contents for Mellanox MCX621102ACADAT

Reviews:

Related manuals for Mellanox MCX621102ACADAT

Brands by name

Popular brands