Nvidia MCX621202AC-ADAT User Manual Download Page 77

Page: 77 / 79

•
•
•
•
•
•
•
•

•
•
•

Date

Revision

Description of Changes

Jun. 2020

1.3

Updated airflow numbers.

Added the following OPNs to all relevant sections:

MCX621102AE-ADAT

MCX623102AS-GDAT
MCX623102AC-GDAT

MCX623106AE-CDAT

MCX623106PC-CDAT
MCX623106PN-CDAT
MCX623106PE-CDAT
MCX623105AE-VDAT

May. 2020

1.2

Updated power numbers.

Feb. 2020

1.1

Added the following OPNs to all relevant sections:

MCX623106AS-CDAT
MCX623105AS-VDAT
MCX623102AS-GDAT

Nov. 2019

1.0

First release

Summary of Contents for MCX621202AC-ADAT

Page 1: ...Exported on Sep 16 2022 01 57 PM NVIDIA ConnectX 6 Dx Ethernet Adapter Cards User Manual...

Page 2: ...e Color LED 20 Voltage Regulators 20 Heatsink 21 Thermal Sensors 21 Hardware Installation 22 Safety Warnings 22 Installation Procedure Overview 22 System Requirements 23 Safety Precautions 23 Unpackin...

Page 3: ...hooting 59 General Troubleshooting 59 Linux Troubleshooting 60 Windows Troubleshooting 60 Specifications 62 MCX621102AC ADAT MCX621102AN ADAT MCX621102AN ADAT Specifications 62 MCX623102AC ADAT MCX623...

Page 4: ...4 Board Mechanical Drawing and Dimensions 72 Bracket Mechanical Drawing 73 Finding the MAC on the Adapter Card 75 Document Revision History 76...

Page 5: ...port SFP28 PCIe 4 0 x8 Secure Boot No Crypto Tall Bracket MCX621202AC ADAT ConnectX 6 Dx EN adapter card 25GbE with active cooling Dual port SFP28 PCIe 4 0 x8 Crypto and Secure Boot Tall Bracket 50GbE...

Page 6: ...6 Dx EN adapter card 25GbE Dual port SFP28 PCIe 4 0 x16 No Crypto Tall Bracket MCX621102AE ADAT ConnectX 6 Dx EN adapter card 25GbE Dual port SFP28 PCIe 4 0 x8 Crypto No Secure Boot Tall Bracket MCX62...

Page 7: ...installer and user of these cards The manual assumes basic familiarity with Ethernet network and architecture specifications Technical Support Customers who purchased NVIDIA products directly from NV...

Page 8: ...designed to maximize the performance of High Performance Computing networks requiring high bandwidth low latency connections between compute nodes and switch nodes NVIDIA offers one of the industry s...

Page 9: ...tX 6 Dx 25GbE Adapter Cards OPN Form Factor Dimensions Data Transmiss ion Rate No of Ports and Type PCIe Interface Secure Boot Cry pto Ro HS Bracket Type MCX62110 2AC ADAT 4 89in x 2 71in 124 22mm x 6...

Page 10: ...ry pto Ro HS Bracket Type MCX62310 2AC GDAT 5 59in x 2 71in 142 00mm x 68 90mm 50 25 10 1 GbE Dual port SFP56 PCIe Gen 4 0 SERDES 16 0GT s x16 Tall Bracket MCX62310 2AE GDAT 5 59in x 2 71in 142 00mm x...

Page 11: ...s x16 Tall Bracket MCX623105 AE CDAT 5 59in x 2 71in 142 00mm x 68 90mm 100 50 25 10 1 GbE Single port QSFP56 PCIe Gen 4 0 SERDES 16 0GT s x16 Tall Bracket MCX623106 AE CDAT 5 59in x 2 71in 142 00mm x...

Page 12: ...5 59in x 2 71in 142 00m m x 68 90mm 100 50 25 10 1 GbE Dual port QSFP56 PCIe Gen 4 0 SERDES 16 0GT s x16 Tall Bracke t MCX623 106GC CDAT 5 59in x 2 71in 142 00m m x 68 90mm 100 50 25 10 1 GbE Dual por...

Page 13: ...3ba 40 Gigabit Ethernet IEEE 802 3by 25 Gigabit Ethernet IEEE 802 3ae 10 Gigabit Ethernet IEEE 802 3ap based auto negotiation and KR startup IEEE 802 3ad 802 1AX Link Aggregation IEEE 802 1Q 802 1P VL...

Page 14: ...rom GPU to CPU and therefore significantly reduces application run time ConnectX 6 Dx advanced acceleration technology enables higher cluster efficiency and scalability to tens of thousands of nodes C...

Page 15: ...bled in MCX623106G N C CDAT PPS In Out SMAs NVIDIA offers a full IEEE 1588v2 PTP software solution as well as time sensitive related features called 5T NVIDIA PTP and 5T software solutions are designe...

Page 16: ...16 Connectivity Interoperable with 1 10 25 40 50 100 200 Gb s Ethernet switches Passive copper cable with ESD protection Powered connectors for optical and active cable support...

Page 17: ...s SFP28 SFP56 QSFP56 connectors The networking connectors allow for the use of modules optical and passive cable interconnect solutions 3 PCI Express Interface PCIe Gen 3 0 4 0 through an x8 x16 edge...

Page 18: ...egrity of the network adapter Ethernet SFP28 SFP56 QSFP56 Interfaces The network ports of the ConnectX 6 Dx adapter card are compliant with the IEEE 802 3 Ethernet standards listed in Features and Ben...

Page 19: ...LED Scheme 1 One Bi Color LED There is one bicolor Yellow and Green I O LED per port to indicate speed and link status Link Indications State Bi Color LED Yellow Green Physical link speed Beacon comma...

Page 20: ...ent condition of the networking ports Blinks until error is fixed ON Physical Activity The Green LED will blink Blinking Link Up The Green LED will be solid ON SMBus Interface ConnectX 6 Dx technology...

Page 21: ...ded push pins that insert into four mounting holes or by screws ConnectX 6 Dx IC has a thermal shutdown safety mechanism that automatically shuts down the ConnectX 6 Dx card in cases of high temperatu...

Page 22: ...quirements 2 Pay attention to the airflow consideration within the host system Refer to Airflow Requirements 3 Follow the safety precautions Refer to Safety Precautions 4 Unpack the package Refer to U...

Page 23: ...an be lethal Before opening the case of the system observe the following precautions to avoid injury and prevent damage to system components Remove any metallic objects from your hands and wrists Make...

Page 24: ...ctX 6 Dx adapter card Accessories 1 Adapter card short bracket 1 Adapter card tall bracket shipped assembled on the card Pre Installation Checklist Verify that your system meets the hardware and softw...

Page 25: ...ht The 2 screws saved from the removal of the bracket Removing the Existing Bracket Using a torque driver remove the two screws holding the bracket in place Separate the bracket from the ConnectX 6 Dx...

Page 26: ...pplying even pressure at both corners of the card insert the adapter card in a PCI Express slot until firmly seated Secure the adapter card to the chassis Step 1 Secure the bracket to the chassis with...

Page 27: ...sert the connector upside down This may damage the adapter card Insert the connector into the adapter card Be careful to insert the connector straight into the cage Do not apply any torque up or down...

Page 28: ...y from the port receptacle The LED indicator will turn off when the cable is unseated Identifying the Card in Your System On Linux Get the device location on the PCI bus by running lspci and locating...

Page 29: ...l Before uninstalling the adapter card please observe the following precautions to avoid injury and prevent damage to system components Remove any metallic objects from your hands and wrists It is str...

Page 30: ...ff and unplugged Wait 30 seconds To remove the card disengage the retention mechanisms on the bracket clips or screws Holding the adapter card from its center gently pull the ConnectX 6 and Auxiliary...

Page 31: ...at the system has a network adapter installed by running lspci command The below table provides output examples per ConnectX 6 Dx card configuration lspci v grep Mellanox 86 00 0 Network controller 02...

Page 32: ...re upgrade using customized FW binaries you can provide a path to the folder that contains the FW binary files by running fw image dir Using this option the FW version embedded in the MLNX_OFED packag...

Page 33: ...InfiniBand VPI fabric a Subnet Manager must be running on one of the fabric nodes At this point MLNX_OFED for Linux has already installed the OpenSM Subnet Manager on your machine For the list of inst...

Page 34: ...D installation script mnt mlnxofedinstall force MLNX_OFED for Ubuntu should be installed with the following flags in chroot environment mlnxofedinstall without dkms add kernel support kernel kernel ve...

Page 35: ...are installed under the usr directory except for the following packages which are installed under the opt directory fca and ibutils The kernel modules are installed under lib modules uname r updates...

Page 36: ...tact your HW vendor Installation Logs While installing MLNX_OFED the install log for each selected package will be saved in a separate log file The path to the directory containing the log files will...

Page 37: ...nings Return Code Meaning 0 The installation ended successfully 1 The installation failed 2 No firmware was found for the adapter device 22 Invalid parameter 28 Not enough free space 171 Not applicabl...

Page 38: ...ellanox Technologies support mellanox com Create a yum repository configuration file called etc yum repos d mlnx_ofed repo with the following content mlnx_ofed name MLNX_OFED Repository baseurl file p...

Page 39: ..._64 noarch MLNX_OFED hypervisor installer package for kernel 3 17 4 01 fc21 x86_64 without KMP support mlnx ofed vma 3 17 4 301 fc21 x86_64 noarch MLNX_OFED vma installer package for kernel 3 17 4 301...

Page 40: ...lanox com sub 1024g 09FCC269 2013 08 11 Update the apt get cache sudo apt get update Installing MLNX_OFED Using the apt get Tool After setting up the apt get repository for MLNX_OFED package perform t...

Page 41: ...ectX 6 Dx EN adapter card 100GbE Dual port QSFP56 PCIe 4 0 x16 Crypto and Secure Boot Tall Bracket PCI Device Name 0b 00 0 Base MAC 0000e41d2d5cf810 Versions Current Available FW 28 33 0800 28 33 1000...

Page 42: ...boot If the firmware is updated the following message is printed to the system s standard logging file fw_updater Firmware was updated Please reboot your system for the changes to take effect Otherwi...

Page 43: ...stent Once a key is in the MOK list it will be automatically propagated to the system key ring and subsequent will be booted when the UEFI Secure Boot is enabled Removing Signature from kernel Modules...

Page 44: ...it may be necessary to modify the default configuration of network adapters based on the ConnectX adapters In case that tuning is required please refer to the Performance Tuning Guide for NVIDIA Netw...

Page 45: ...iver This section provides instructions for two types of installation procedures and both require administrator privileges Attended Installation An installation procedure that requires frequent user i...

Page 46: ...replace LogFile with the relevant directory MLNX_WinOF2_ revision_version _All_Arch exe v l vx LogFile Optional If you do not want to upgrade your firmware version Note MT_SKIPFWUPGRD default value is...

Page 47: ...ollowing cases If the user has an OEM card In this case the firmware will not be displayed If the user has a standard adapter card with an older firmware version the firmware will be updated according...

Page 48: ...lect the desired feature to install Performances tools install the performance tools that are used to measure performance in user environment Documentation contains the User Manual and Release Notes M...

Page 49: ...49 b Diagnostic Tools installation tools used for diagnostics such as mlx5cmd Click Next to install the desired tools 9 Click Install to start the installation...

Page 50: ...50 10 In case firmware upgrade option was checked in Step 7 you will be notified if a firmware upgrade is required see 11 Click Finish to complete the installation...

Page 51: ...ontrol whether to install ND provider or not i e MT_NDPROPERTY default value is True MLNX_WinOF2 Driver Version _ revision_version _All_Arch exe vMT_NDPROPERTY 1 Optional If you do not wish to upgrade...

Page 52: ...can be located at ProgramFiles Mellanox MLNX_WinOF2 Drivers To see the network adapters display the Device Manager and pull down the Network adapters menu Uninstalling WinOF 2 Driver Attended Uninsta...

Page 53: ...e files without running installation perform the following steps Open a CMD console Click Start Task Manager File Run new task and enter CMD Extract the driver and the tools MLNX_WinOF2 revision_versi...

Page 54: ...54 5 Click Install to extract this folder or click Change to install to a different folder...

Page 55: ...n how to upgrade firmware manually please refer to the MFT User Manual VMware Driver Installation This section describes VMware Driver Installation Hardware and Software Requirements Requirement Descr...

Page 56: ...650 0 0 4240417 MEL PartnerSupported 2017 01 31 Removing Earlier NVIDIA Drivers To remove all the drivers Log into the ESXi server with root permissions List all the existing NATIVE ESXi driver module...

Page 57: ...mware Tools MFT site ESXi 6 5 File mft 4 6 0 48 10EM 650 0 0 4598673 x86_64 vib MD5SUM 0804cffe30913a7b4017445a0f0adbe1 Install the image according to the steps described in the MFT User Manual The fo...

Page 58: ...e Update Example server1 mlxup Querying Mellanox devices firmware Device Type ConnectX 6 Dx Part Number MCX623105AN VDAT Description ConnectX 6 Dx EN adapter card 200GbE Single port QSFP56 PCIe 4 0 x1...

Page 59: ...s securely attached Check you are using the proper cables that do not exceed the recommended lengths Verify that your switch and adapter port are compatible Link light is on but with no communication...

Page 60: ...supportdownloader Collect Log File cat var log messages dmesg system log journalctl Applicable on new operating systems cat var log syslog Windows Troubleshooting Environment Information From the Wind...

Page 61: ...61 Collect Log File Event log viewer MST device logs mst start mst status flint d mst_device dc dump_configuration log mstdump mst_device dc mstdump log...

Page 62: ...ompatible Power Specification s a Voltage 12V Power Cable PCIe Gen 3 0 PCIe Gen 4 0 Typical Power b Passive Cables 10 88W 11 29W Maximum Power Passive Cables 15 55W 15 96W Maximum power available thro...

Page 63: ...t Data Rate Ethernet 1 10 25 Gb s Ethernet 25GBASE R 20GBASE KR2 10GBASE LR 10GBASE ER 10GBASE CX4 10GBASE CR 10GBASE KR SGMII 1000BASE CX 1000BASE KX 10GBASE SR PCI Express Gen 3 0 4 0 SERDES 16 0GT...

Page 64: ...Gb s Ethernet 25GBASE R 20GBASE KR2 10GBASE LR 10GBASE ER 10GBASE CX4 10GBASE CR 10GBASE KR SGMII 1000BASE CX 1000BASE KX 10GBASE SR PCI Express Gen 3 0 4 0 SERDES 16 0GT s 8 lanes 2 0 and 1 1 compati...

Page 65: ...of_Modules x 1 1 efficiency factor b Typical power for ATIS traffic load c For both operational and non operational states MCX623102AC GDAT MCX623102AE GDAT MCX623102AN GDAT MCX623102AS GDAT Specifica...

Page 66: ...1 1 efficiency factor b Typical power for ATIS traffic load c For both operational and non operational states d Airflow is measured in wind tunnel e Contact NVIDIA for airflow numbers with other acti...

Page 67: ...0LFM Regulato ry Safety CB cTUVus CE EMC CE FCC VCCI ICES RCM RoHS RoHS compliant a Power numbers are provided for passive cables only For board power numbers while using active cables please add the...

Page 68: ...port Voltage 3 3Aux Maximum current 100mA Environ mental Temperature Operational 0 C to 55 C Non operational 40 C to 70 C Humidity 90 relative humidity c Altitude Operational 3050m Airflow Requirement...

Page 69: ...er b Passive Cables TBD 18 96W Maximum Power Passive Cables TBD 26 64W Maximum power available through QSFP56 port 5W each port Voltage 3 3Aux Maximum current 100mA Environmen tal Temperature Operatio...

Page 70: ...PCI Express Gen 3 0 4 0 SERDES 16 0GT s 16 lanes 2 0 and 1 1 compatible Power Specificatio ns a Voltage 12V Power Cable Type PCIe Gen 3 0 PCIe Gen 4 0 Typical Power b Passive Cables TBD 18 96W Maximum...

Page 71: ...SE CR4 40GBASE KR4 40GBASE SR4 40GBASE LR4 40GBASE ER4 40GBASE R2 25GBASE R 20GBASE KR2 10GBASE LR 10GBASE ER 10GBASE CX4 10GBASE CR 10GBASE KR SGMII 1000BASE CX 1000BASE KX 10GBASE SR 100GBASE CR2 10...

Page 72: ...nal states d Airflow is measured in wind tunnel e Contact NVIDIA for airflow numbers with other active modules power levels Board Mechanical Drawing and Dimensions Dual Port SFP28 SFP56 x8 Adapter Car...

Page 73: ...Mechanical Drawing Mechanical Tolerance Width 0 13mm Height 0 0 13mm Dual Port QSFP56 x16 Adapter Cards Mechanical Drawing Mechanical Tolerance Width 0 13mm Height 0 0 13mm Bracket Mechanical Drawing...

Page 74: ...74 Card Configuration Short Bracket Tall Bracket Single Port QSFP56 Cards Dual Port QSFP56 Cards...

Page 75: ...different identifier printed on the label serial number and the card MAC for the Ethernet protocol MCX623105AS VDAT Board Label Example The product revisions indicated on the labels in the following f...

Page 76: ...wing OPNs MCX623106TN CDAT MCX623106TC CDAT MCX623106GN CDAT MCX623106GC CDAT MCX621202AS ADAT MCX621202AC ADAT Jun 2021 2 4 Updated Interfaces Mar 2021 2 3 Updated Troubleshooting Mar 2021 2 2 Update...

Page 77: ...ections MCX621102AE ADAT MCX623102AS GDAT MCX623102AC GDAT MCX623106AE CDAT MCX623106PC CDAT MCX623106PN CDAT MCX623106PE CDAT MCX623105AE VDAT May 2020 1 2 Updated power numbers Feb 2020 1 1 Added th...

Page 78: ...by customer and perform the necessary testing for the application in order to avoid a default of the application or the product Weaknesses in customer s product designs may affect the quality and rel...

Search results

Summary of Contents for MCX621202AC-ADAT

Reviews:

Related manuals for MCX621202AC-ADAT

Brands by name

Popular brands

Nvidia MCX621202AC-ADAT, User Manual

Search results

Summary of Contents for MCX621202AC-ADAT

Reviews:

Related manuals for MCX621202AC-ADAT

Brands by name

Popular brands