Nvidia C-ADAB User Manual Download

Page: 1 / 68

Exported on Oct/22/2020 06:48 PM

https://docs.mellanox.com/x/2wPjAQ

NVIDIA Mellanox ConnectX-6 Dx Ethernet

Adapter Cards for OCP Spec 3.0 User Manual

Summary of Contents for C-ADAB

Page 1: ...Exported on Oct 22 2020 06 48 PM https docs mellanox com x 2wPjAQ NVIDIA Mellanox ConnectX 6 Dx Ethernet Adapter Cards for OCP Spec 3 0 User Manual...

Page 2: ...8 SFP56 QSFP56 Interfaces 16 PCI Express Interface 16 Networking Ports LEDs Interface 17 SMBus Interface 18 Voltage Regulators 18 CPLD Interface 18 Heatsink 18 Hardware Installation 19 Safety Warnings...

Page 3: ...xtracting Files Without Running Installation 33 Uninstalling Mellanox WinOF 2 Driver 35 Attended Uninstallation 35 Unattended Uninstallation 35 Firmware Upgrade 35 Deploying the Driver on a Nano Serve...

Page 4: ...emoving Signature from kernel Modules 48 Performance Tuning 49 VMware Driver Installation 49 Hardware and Software Requirements 49 Installing Mellanox NATIVE ESXi Driver for VMware vSphere 50 Removing...

Page 5: ...5 Single port QSFP56 Thumbscrew Bracket 63 Finding the MAC and Serial Number on the Adapter Card 64 Document Revision History 66...

Page 6: ...pter card 50GbE OCP3 0 With Host Management Dual port SFP56 PCIe 4 0 x16 No Crypto Thumbscrew Pull Tab Bracket MCX623432A C GDAB ConnectX 6 Dx EN adapter card 50GbE OCP3 0 With Host Management Dual po...

Page 7: ...tion WinOF 2 for Windows User Manual and Release Notes User Manual describing WinOF 2 features performance Ethernet diagnostic tools content and configuration See WinOF 2 for Windows Documentation Mel...

Page 8: ...irect Attach Copper cables DACs copper splitter cables Active Optical Cables AOCs and transceivers in a wide range of lengths from 0 5m to 10km Mellanox tests every product in an end to end environmen...

Page 9: ...provides the ordering part number port speed number of ports and PCI Express speed Products Overview ConnectX 6 Dx 25GbE Adapter Cards for OCP 3 0 OPN MCX623432AN ADAB MCX623432AC ADAB MCX623432AE ADA...

Page 10: ...sical Function PF 4126 for Virtual Function VF ConnectX 6 Dx 100GbE Adapter Cards for OCP 3 0 OPN MCX623436AN CDAB MCX623436AC CDAB MCX623435MN CDAB MCX623436AS CDAI Network Connector Type Dual port Q...

Page 11: ...52 in x 2 99 in 115 00mm x 76 00mm Retention Mechanism Thumbscrew Pull Tab Bracket Thumbscrew Pull Tab Bracket Internal Lock Bracket RoHS RoHS Compliant RoHS Compliant RoHS Compliant IC Part Number MT...

Page 12: ...dvanced NVGRE and VXLAN hardware offloading engines that encapsulate and de capsulate the overlay protocol RDMA and RDMA over Converged Ethernet RoCE ConnectX 6 Dx utilizing RDMA Remote Data Memory Ac...

Page 13: ...Management Controller BMC interface supporting both NC SI and PLDM management protocols using these interfaces Mellanox OCP 3 0 adapters support these protocols to offer numerous Host Management feat...

Page 14: ...14...

Page 15: ...16 edge connector 3 Ethernet SFP28 SFP56 QSFP56 Interface Ethernet traffic is transmitted through the adapter s SFP28 SFP56 QSFP56 connectors The networking connectors allow for the use of modules opt...

Page 16: ...ta in motion and XTS AES data at rest cryptographic operations Stateful firewall solution acceleration powered by Open vSwitch connection tracking and Mellanox s ASAP2 technology Embedded hardware roo...

Page 17: ...the following Error Type Description LED Behavior I2C I2C access to the networking ports fails Blinks until the error is fixed Over current Over current condition of the networking ports Blinks until...

Page 18: ...3V supply pins These voltage supply pins feed onboard regulators that provide the necessary power to the various components on the card CPLD Interface The adapter card incorporates a CPLD device that...

Page 19: ...o Airflow Requirements 3 Follow the safety precautions Refer to Safety Precautions 4 Follow the pre installation checklist Refer to Pre Installation Checklist 5 Optional Replace the assembled OCP 3 0...

Page 20: ...ements See Operating Systems Distributions section under the Introduction section Unless otherwise specified Mellanox products are designed to work in an environmentally controlled data center with lo...

Page 21: ...g shipping Please note that the cards must be placed on an antistatic surface Shut down your system if active Turn off the power to the system and disconnect the power cord Refer to the system documen...

Page 22: ...in a system Cards with Thumbscrew Pull tab Bracket Before installing the card make sure that the system is off and the power cord is not connected to the server Please follow proper electrical ground...

Page 23: ...make sure that the system is off and the power cord is not connected to the server Please follow proper electrical grounding procedures Open the system case Align the card with the system rails Push t...

Page 24: ...is established that is when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a functioning port See Adapter Card LED Operations After pluggi...

Page 25: ...xample above notice the sub string PCI VEN_15B3 DEV_1003 VEN is equal to 0x15B3 this is the Vendor ID of Mellanox Technologies and DEV is equal to 1018 for ConnectX 6 Dx this is a valid Mellanox Techn...

Page 26: ...devices Turn off the system and disconnect the power cord from the server Extraction Instructions Verify that the system is powered off and unplugged Wait 30 seconds Rotate the captive screw countercl...

Page 27: ...ted above must run with administrator privileges Downloading Mellanox WinOF 2 Driver To download the executable file according to your operating system please follow the steps below Obtain the machine...

Page 28: ...you do not want to upgrade your firmware version Note MT_SKIPFWUPGRD default value is False MLNX_WinOF2 1_10_ revision_version _All_x64 exe v MT_SKIPFWUPGRD 1 Installing the incorrect exe file is pro...

Page 29: ...the installation The firmware upgrade screen will be displayed in the following cases If the user has an OEM card In this case the firmware will not be displayed If the user has a standard Mellanox ca...

Page 30: ...ion follow Step a and on Select the desired feature to install Performance tools install the performance tools that are used to measure performance in user environment Documentation contains the User...

Page 31: ...ools used for diagnostics such as mlx5cmd Click Next to install the desired tools Click Install to start the installation In case that firmware upgrade option was checked in Step 7 you will be notifie...

Page 32: ...ant to control whether to install ND provider or not Note MT_NDPROPERTY default value is True MLNX_WinOF2 1_10_All_x64 exe vMT_NDPROPERTY 1 Optional If you do not wish to upgrade your firmware version...

Page 33: ...ers OS To see the Mellanox network adapters display the Device Manager and pull down the Network adapters menu Extracting Files Without Running Installation To extract the files without running instal...

Page 34: ...3 4 5 Click Next to create a server image Click Change and specify the location in which the files are extracted to Click Install to extract this folder or click Change to install to a different fold...

Page 35: ...CMD console Click Task Manager File Run new task and enter CMD To uninstall the driver run MLNX_WinOF2 1_10_All_x64 exe S x v qn Firmware Upgrade If the machine has a standard Mellanox card with an ol...

Page 36: ...tPath InjectingDrivers vhdx DriversPath C WS2016TP5_Drivers Create the Nano server image Follow the instructions in the TechNet article for Getting Started with Nano Server Online Update Go to the Mel...

Page 37: ...mple shows a system with an installed Mellanox adapter card lspci v grep Mellanox 86 00 0 Network controller 0207 Mellanox Technologies Family Subsystem Mellanox Technologies Device 0014 86 00 1 Netwo...

Page 38: ...he deletion of the old packages If you need to install Mellanox OFED on an entire homogeneous cluster a common strategy is to mount the ISO image on one of the cluster nodes and then copy it to a shar...

Page 39: ...led the OpenSM Subnet Manager on your machine For the list of installation options run mlnxofedinstall h Installation Procedure Login to the installation machine as root Mount the ISO image on your ma...

Page 40: ...ectX 6 Dx Part Number MCX623436AC CDAB Description ConnectX 6 Dx EN adapter card 100GbE OCP3 0 With Host Management Dual port QSFP56 PCIe 4 0 x16 Crypto and Secure Boot Thumbscrew Pull Tab Bracket PSI...

Page 41: ...updated if the following two conditions are fulfilled The installation script is run in default mode that is without the option without fw update The firmware version of the adapter device is older t...

Page 42: ...process of updating the initramfs will add the blacklists from step 1 and will prevent the kernel from loading the modules automatically mlnxofedinstall Return Codes The table below lists the mlnxofed...

Page 43: ...cted HTTP request sent awaiting response 200 OK Length 1354 1 3K text plain Saving to RPM GPG KEY Mellanox 100 1 354 K s in 0s 2014 04 20 13 52 30 247 MB s RPM GPG KEY Mellanox saved 1354 1354 Install...

Page 44: ...port for RHEL and SLES For other operating systems kernel module RPM packages are provided only for the operating systems default kernel In this case the group RPM packages have the supported kernel v...

Page 45: ...operating systems Setting up MLNX_OFED apt get Repository Log into the installation machine as root Extract the MLNX_OFED pacakge on a shared location in your network You can download it from http www...

Page 46: ...pi installer package Install the desired group apt get install group name Example apt get install mlnx ofed all Uninstalling MLNX_OFED Using the apt get Tool Use the script usr sbin ofed_uninstall sh...

Page 47: ...om Support Firmware Download Get the device s PSID mlxfwmanager_pci grep PSID PSID MT_1210110019 Download the firmware BIN file from the Mellanox website or the OEM website Burn the firmware mlxfwmana...

Page 48: ...ill launch MokManager efi to allow you to complete the enrollment from the UEFI console You will need to enter the password you previously associated with this request and confirm the enrollment Once...

Page 49: ...ssage will appear Note that this message cannot be eliminated Update the initramfs on RHEL systems with the stripped modules mkinitrd boot initramfs uname r img uname r force Performance Tuning Depend...

Page 50: ...7 01 31 Removing Earlier Mellanox Drivers To remove all the drivers Log into the ESXi server with root permissions List all the existing NATIVE ESXi driver modules See Step 4 in Installing Mellanox NA...

Page 51: ...are bootable binary images from the Mellanox Firmware Tools MFT site Install the image according to the steps described in the MFT User Manual The following procedure requires custom boot image downlo...

Page 52: ...xup Querying Mellanox devices firmware Device Type ConnectX 6 Dx Part Number MCX623436AC CDAB Description ConnectX 6 Dx EN adapter card 100GbE OCP3 0 With Host Management Dual port QSFP56 PCIe 4 0 x16...

Page 53: ...ng another adapter Try removing and re installing all adapters Check that cables are connected properly Make sure your motherboard has the latest BIOS Link indicator light is off Try another port on t...

Page 54: ...ows desktop choose the Start menu and run msinfo32 To export system information to a text file choose the Export option from the File menu Assign a file name and save Mellanox Firmware Tool MFT Downlo...

Page 55: ...16 lanes 2 0 and 1 1 compatible Power and Environmental Voltage 3 3VAUX 12V Power Cable Type Active Mode Standby Mode Typical Power b Passive Cables 15 7W 3 8W Maximum Power Passive Cables 19 74W 7 95...

Page 56: ...raffic load c For both operational and non operational states MCX623432A N C GDAB MCX623432AS GDAI Specifications Physical Form Factor Small Form Factor SFF OCP 3 0 Size 4 52 in x 2 99 in 115 00mm x 7...

Page 57: ...to HSK Active Mode 55C Standby Mode 45C Active Mode Standby Mode Passive Cable 400LFM 100LFM TBD TBD Active 2W Cable 800LFM 200LFM TBD TBD Regulatory Safety CB cTUVus CE EMC CE FCC VCCI ICES RCM RoHS...

Page 58: ...e Cables 27 4W 14W Maximum power available through QSFP56 port 3 5W Temperature Operational 0 C to 55 C Non operational 40 C to 70 C Humidity 90 relative humidity c Altitude Operational 3050m Environ...

Page 59: ...ernet 1 10 25 40 50 100 Gb s PCI Express Gen 4 0 SERDES 16 0GT s 16 lanes 2 0 and 1 1 compatible Powe r and Envir onme ntal Voltage 3 3VAUX 12V Power Cable Type Active Mode Standby Mode Typical Power...

Page 60: ...bscrew Pull Tab Bracket MCX623435AS VDAI Internal Lock Protocol Support Ethernet 200GBASE CR4 200GBASE KR4 200GBASE SR4 100GBASE CR4 100GBASE KR4 100GBASE SR4 50GBASE R2 50GBASE R4 40GBASE CR4 40GBASE...

Page 61: ...x adapters supplement the auto negotiation specification to get better bit error rates and longer cable reaches This supplemental feature only initiates when connected to another Mellanox Ethernet pro...

Page 62: ...62 ConnectX 6 Dx Dual Port QSFP56 OCP 3 0 Mechanical Drawing and Dimensions ConnectX 6 Dx Single Port QSFP56 OCP 3 0 Mechanical Drawing and Dimensions...

Page 63: ...l Drawings and Dimensions Dual port SFP28 SFP56 Thumbscrew Bracket Dual port QSFP56 Thumbscrew Bracket Single port QSFP56 Thumbscrew Bracket All dimensions are in millimeters The bracket s mechanical...

Page 64: ...ial number and the card MAC for the Ethernet protocol MCX623432AN ADAB Board Label Example MCX623432AC ADAB Board Label Example MCX623432AN GDAB Board Label Example MCX623432AC GDAB Board Label Exampl...

Page 65: ...65 MCX623436AN CDAB Board Label Example MCX623436AC CDAB Board Label Example MCX623435AN VDAB Board Label Example MCX623435AC VDAB Board Label Example...

Page 66: ...g ports LEDs indications Sep 2020 1 5 Added OCP 3 0 bracket replacement instructions Jul 2020 1 4 Updated airflow numbers in Specifications Jul 2020 1 3 Updated all power and airflow numbers in Specif...

Page 67: ...arily performed by NVIDIA It is customer s sole responsibility to evaluate and determine the applicability of any information contained in this document ensure the product is suitable and fit for the...

Page 68: ...A the NVIDIA logo and Mellanox are trademarks and or registered trademarks of Mellanox Technologies Ltd and or NVIDIA Corporation in the U S and in other countries Other company and product names may...

Search results

Summary of Contents for C-ADAB

Reviews:

Related manuals for C-ADAB

Brands by name

Popular brands

Nvidia C-ADAB, User Manual

Search results

Summary of Contents for C-ADAB

Reviews:

Related manuals for C-ADAB

Brands by name

Popular brands