Nvidia MCX4121A-ACAT User Manual Download

Page: 1 / 62

NVIDIA Mellanox ConnectX-4 Lx Ethernet
Adapter Cards User Manual

Exported on Dec/24/2020 12:18 PM

https://docs.mellanox.com/x/mYaO

Summary of Contents for MCX4121A-ACAT

Page 1: ...NVIDIA Mellanox ConnectX 4 Lx Ethernet Adapter Cards User Manual Exported on Dec 24 2020 12 18 PM https docs mellanox com x mYaO ...

Page 2: ...rs 12 Hardware Installation 13 Safety Warnings 13 Installation Procedure Overview 13 System Requirements 14 Hardware Requirements 14 Airflow Requirements 14 Software Requirements 14 Unpacking the Product 14 Bracket Replacement Instructions 15 Installation Instructions 16 Cables and Modules 17 Identifying the Card inYour System 18 Uninstalling the Card 19 Driver Installation 21 Linux Driver Install...

Page 3: ...d Software Requirements 44 Installing Mellanox NATIVE ESXi Driver for VMware vSphere 45 Removing Earlier Mellanox Drivers 45 Firmware Programming 46 Updating Adapter Firmware 47 Troubleshooting 48 General Troubleshooting 48 Linux Troubleshooting 48 Windows Troubleshooting 49 Specifications 50 MCX4121A XCAT Specifications 50 MCX4121A ACAT Specifications 51 Airflow Specifications 53 Board Mechanical...

Page 4: ...0GbE dual port SFP28 PCIe3 0 x8 tall bracket MCX4121A ACAT ConnectX 4 Lx EN network interface card 25GbE dual port SFP28 PCIe3 0 x8 tall bracket Intended Audience This manual is intended for the installer and user of these cards The manual assumes basic familiarity with Ethernet network and architecture specifications Technical Support Customers who purchased Mellanox products directly from Mellan...

Page 5: ...ware management tools for a single node See MFT User Manual IEEE Std 802 3 Specification IEEE Ethernet specification at http standards ieee org PCI Express Specifications Industry Standard PCI Express Base and Card Electromechanical Specifications at https pcisig com specifications Mellanox LinkX Interconnect Solutions Mellanox LinkX InfiniBand cables and transceivers are designed to maximize the ...

Page 6: ...Lx 10Gb s Ethernet Adapter Cards Model ConnectX 4 Lx 10Gb s Ethernet Adapter Cards Part Number MCX4121A XCAT Ethernet Data Rate 1 10 Gb s Network Connector Type Dual port SFP28 PCI Express Connectors PCIe Gen 3 0 SERDES 8 0GT s Dimensions 2 71 in x 5 6 in 68 90mm x 142 24 mm low profile UEFI Host Management RoHS RoHS Compliant IC Part Number MT27712A0 FDCF XE Device ID 4117 for Physical Function P...

Page 7: ... 25Gb s Ethernet Adapter Cards Secure Firmware Host Management RoHS RoHS Compliant IC Part Number MT27712A0 FDCF AE Device ID 4117 for Physical Function PF 4118 for Virtual Function VF For more detailed information see Specifications Features and Benefits Feature Description PCI Express PCIe Uses PCIe Gen 3 0 8GT s through an x8 edge connector Gen 1 1 and 2 0 compatible ...

Page 8: ...g advanced NVGRE and VXLAN hardware offloading engines that encapsulate and de capsulate the overlay protocol RDMAandRDMAoverConverged Ethernet RoCE ConnectX 4 Lx utilizing IBTA RDMA Remote Data Memory Access and RoCE RDMA over Converged Ethernet technology delivers low latency and high performance over Band and Ethernet networks Leveraging data center bridging DCB capabilities as well as ConnectX...

Page 9: ...icated adapter resources and guaranteed isolation and protection for virtual machines VM within theserver NC SI The adapter supports a Network Controller Sideband Interface NC SI MCTP over SMBus and MCTP over PCIe Baseboard Management Controller interface High Performance Accelerations TagMatching andRendezvousOffloads Adaptive Routing on ReliableTransport Burst Buffer Offloads for Background Chec...

Page 10: ...10 Connectivity Interoperable with 1 10 25 Gb s Ethernet switches Passive copper cable with ESDprotection Powered connectors for optical and active cablesupport ...

Page 11: ...5 5 0 or 8 0GT s Auto Negotiation to x8 x4 x2 or x1 Support for MSI MSI X mechanisms LED Interface There is one bi color I O LED per port to indicate link status located on the adapter card Link Indications Ethernet Mode LED Color and State Description Off A link has not been established Blinking Amber a 4 Hz blinking amber indicates a problem with the link Solid Green Indicates a valid link with ...

Page 12: ...CIe stand up adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as if it is a standard Mellanox PCIe stand up adapter For configuring the adapter for the specific manageability solution in use by the server please contact Mellanox Support Voltage Regulators The voltage regulator power is derived from the PCI Express edge connector 12V supply pins These voltage supp...

Page 13: ...lude Could not retrieve page Installation Procedure Overview The installation procedure of ConnectX 4 Lx adapter cards involves the following steps Step Procedure Direct Link 1 Check the system s hardware and software requirements Refer to System Requirements 2 Pay attention to the airflow consideration within the host system Refer to Airflow Requirements 3 Unpack the product Refer to Unpacking th...

Page 14: ...c objects from your hands and wrists Make sure to use only insulatedtools Verify that the system is powered off and is unplugged It is strongly recommended to use an ESD strap or other antistatic devices System Requirements Hardware Requirements A system with a PCI Express x8 edge connector is required for installing the card Airflow Requirements ConnectX 4 Lx adapter cards are offered with two ai...

Page 15: ...acement Instructions The card is usually shipped with an assembled high profile bracket If this form factor is suitable for your requirements you can skip the remainder of this section and move to Installation Instructions If you need to replace the high profile bracket with the short bracket that is included in the shipping box please follow the instructions in this section To replace the bracket...

Page 16: ...Installation Instructions This section provides detailed instructions on how to install your adapter card in a system 1 Before installing the card make sure that the system is off and the power cord is notconnected to the server Please follow proper electrical groundingprocedures 2 Open the system case 3 Place the adapter in an available PCI Express slot Use a torque driver to apply up to 2 lbs in...

Page 17: ...pter faceplate is visible against the system chassis 6 Secure the adapter with the adapter clip orscrew Cables and Modules To obtain the list of supported Mellanox cables for your adapter please refer to the Cables Reference Table at http www mellanox com products interconnect cables configurator php Cable Installation 1 All cables can be inserted or removed with the unit powered on 2 To insert a ...

Page 18: ...m particular to the cable vendor When data is being transferred the Green LED will blink See LED Operations under the Interfaces section 5 Care should be taken as not to impede the air exhaust flow through the ventilation holes Use cable lengths which allow for routing horizontally around to the side of the chassis before bending upward or downward in therack 6 To remove a cable disengage the lock...

Page 19: ...echnologies and DEV is equal to 1018 for ConnectX 4 Lx this is a valid Mellanox Technologies PCI Device ID Uninstalling the Card Safety Precautions The adapter is installed in a system that operates with voltages that can be lethal Before uninstalling the adapter card please observe the following precautions to avoid injury and prevent damage to system components 1 Remove any metallic objects from...

Page 20: ...ms on the bracket clips or screws 4 Holding the adapter card from its center gently pull the adapter card out of the PCI Express slot 5 When the port connectors reach the top of the chassis window gently pull the adapter card in parallel to the motherboard Please note that the following images are for illustration purposes only ...

Page 21: ...s please visit the Mellanox website at http www mellanox com page firmware_HCA_FW_identification Operating System Linux operating system For the list of supported operating system distributions and kernels please refer to the MellanoxOFED Release Notes file Installer Privileges The installation requires administrator root privileges on the target machine Downloading Mellanox OFED 1 Verify that the...

Page 22: ... firmware will not be updated if you run the install script with the without fw update option Note If you wish to perform a firmware upgrade using customized FW binaries you can provide a path to the folder that contains the FW binary files by running fw image dir Using this option the FW version embedded in the MLNX_OFED package will be ignored Example Usage The installation script removes all pr...

Page 23: ...t o ro loop MLNX_OFED_LINUX ver OS label CPU arch iso mnt Example The following command will create a MLNX_OFED_LINUX ISO image for RedHat 6 3 under the tmp directory The script adds the following lines to etc security limits conf for the userspace components such as MPI soft memlock unlimited hard memlock unlimited These settings set the amount of memory that can be pinned by a userspace applicat...

Page 24: ...e following flags in the root environment mlnxofedinstall without dkms add kernel support kernel kernel version in chroot without fw update force For example mlnxofedinstall without dkms add kernel support kernel 3 13 0 85 generic without fw update force Note that the path to kernel sources kernel sources should be added if the sources are not in their defaultlocation In case your machine has the ...

Page 25: ...nstalledunderthe usr directory except for the following packages which are installed under the opt directory fca and ibutils The kernel modules are installedunder lib modules uname r updates onSLES and Fedora Distributions lib modules uname r extra mlnx ofa_kernel on RHELand other Red Hat likeDistributions Software In case your machine has an unsupported network adapter device no firmware update w...

Page 26: ... modprobe d mlnx conf file OPENIBD_POST_START sbin openibd_post_start sh An example of OPENIBD_POST_START script for activating all interfaces is provided in the MLNX_OFED package under the docs scripts openibd post start configure interfaces folder The firmware of existing network adapter devices will be updated if the following two conditions are fulfilled The installation script is run in defau...

Page 27: ...pplicable to this system configuration This can occur when the required hardware is not present on the system 172 Prerequisites are not met For example missing the required software installed or the hardware is not configured correctly 173 Failed to start the mst driver Uninstalling MLNX_OFED Use the script usr sbin ofed_uninstall sh to uninstall the Mellanox OFED package The script is part of the...

Page 28: ...package with KMP support mlnx ofed vma noarch MLNX_OFED vma installer package with KMP support mlnx ofed vma eth noarch MLNX_OFED vma eth installer package with KMP support mlnx ofed vma vpi noarch MLNX_OFED vma vpi installer package with KMP support 4 Install the key 5 Check that the key was successfullyimported 6 Create a yum repository configuration file called etc yum repos d mlnx_ofed repo wi...

Page 29: ...sdpnetstat x86_64 0 1 60 26 srptools x86_64 0 1 0 2 12 Complete Note MLNX_OFED provides kernel module RPM packages with KMP support for RHEL and SLES For other operating systems kernel module RPM packages are provided only for the operating systems default kernel In this case the group RPM packages have the supported kernel version in their package s name Example If you have an operating system di...

Page 30: ...network You can download it from http www mellanox com Products Software Ethernet Drivers 3 Create an apt get repository configuration file called etc apt sources list d mlnx_ofed list with the following content 4 Download and install Mellanox TechnologiesGPG KEY 5 Check that the key was successfullyimported 6 Update the apt get cache Installing MLNX_OFED Using the apt getTool After setting up the...

Page 31: ...ox site use the following command line Example Port2 MAC Versions 0002c9000002 Current Available FW 2 32 5000 2 33 5000 Status Update required Updating the Device Manually To update the device manually please refer to the OEM Firmware Download page at http www mellanox com page firmware_table_dell mtag oem_firmware_download In case that you ran the mlnxofedinstall script with the without fw update...

Page 32: ...s disabled by default To enable the automatic firmware update upon system boot set the following parameter to yes RUN_FW_UPDATER_ONBOOT yes in the openibd service configuration file etc infiniband openib conf You can opt to exclude a list of devices from the automatic firmware update procedure To do so edit theconfigurationsfile opt mellanox mlnx fw updater mlnx fw updater conf and provide a comma...

Page 33: ...kernel Modules The signature can be removed from a signed kernel module using the strip utility which is provided by the binutils package The strip utility will change the given file without saving a backup The operation can be undo only by resigning the kernel module Hence we recommend backing up a copy prior to removing the signature To remove the signature from the MLNX_OFED kernel modules 1 Re...

Page 34: ...ents Description Package Windows Server 2012 R2 MLNX_WinOF2 2_10_All_x64 exe Windows Server 2012 Windows Server 2016 Windows Server 2019 Windows 8 1 Client 64 bit only Windows 10 Client 64 bit only Note The Operating System listed above must run with administrator privileges Downloading Mellanox WinOF 2 Driver To download the executable file according to your operating system please follow the ste...

Page 35: ...rvention Unattended Installation An automated installation procedure that requires no user intervention Attended Installation The following is an example of an installation session 1 Double click the exe and follow the GUI instructions to install MLNX_WinOF2 2 Optional Manually configure your setup to contain the logs option replace LogFile with the relevant directory 3 Optional If you do not want...

Page 36: ...36 4 Click Next in the Welcome screen 5 Read and accept the license agreement and click Next 6 Select the target folder for the installation ...

Page 37: ... user has an OEM card In this case the firmware will not be displayed If the user has a standard Mellanox card with an older firmware version the firmware will be updated accordingly However if the user has both an OEM card and a Mellanox card only the Mellanox card will be updated ...

Page 38: ...ols install the performance tools that are used to measure performance in user environment Documentation contains the User Manual and Release Notes Managementtools installationtools usedfor management such asmlxstat DiagnosticTools installationtoolsusedfor diagnostics suchasmlx5cmd b Click Next to install the desiredtools ...

Page 39: ...9 9 Click Install to start theinstallation 10 In case firmware upgrade option was checked in Step 7 you will be notified if a firmware upgrade is required see 11 Click Finish to complete the installation ...

Page 40: ... Version _ revision_version _All_Arch exe vMT_SKIPFWUPGRD 1 1 Unattended Installation The following is an example of an unattended installation session 1 Open a CMD console Click Start Task Manager File Run new task and enter CMD 2 Install the driver Run 3 Optional Manually configure your setup to contain the logsoption 4 Optional if you wish to control whether to install ND provider or not i e MT...

Page 41: ...ated at To see the Mellanox network adapters display the Device Manager and pull down the Network adapters menu Uninstalling Mellanox WinOF 2 Driver Attended Uninstallation To uninstall MLNX_WinOF2 on a single node 1 Click Start Control Panel Programs and Features MLNX_WinOF2 Uninstall NOTE This requires elevated administrator privileges Unattended Uninstallation To uninstall MLNX_WinOF2 in unatte...

Page 42: ...r CMD 2 To uninstall the driver run Extracting Files Without Running Installation To extract the files without running installation perform the following steps 1 Open a CMD console Click Start Task Manager File Run new task and enter CMD 2 Extract the driver and the tools To extract only the driver file 3 Click Next to create a serverimage 4 Click Change and specify the location in which the files...

Page 43: ...43 5 Click Install to extract this folder or click Change to install to a different folder ...

Page 44: ...er Manual at www m ellanox com Products Ethernet Drivers Firmware Tools VMware Driver Installation Hardware and Software Requirements Requirement Description Platforms A server platform with an adapter card based on one of the following Mellanox Technologies devices ConnectX 4 Lx EN firmware fw ConnectX4Lx Device ID For the latest list of device IDs please visit Mellanox s website Operating System...

Page 45: ...ATIVE ESXi Driver for VMware vSphere To install the driver 1 Log into the ESXi server with rootpermissions 2 Install the driver Example 3 Reboot the machine 4 Verify the driver was installed successfully esxcli software vib list grep nmlx nmlx5 core 4 16 8 8 1OEM 650 0 0 4240417 MEL PartnerSupported 2017 01 31 nmlx5 rdma 4 16 8 8 1OEM 650 0 0 4240417 MEL PartnerSupported 2017 01 31 Removing Earlie...

Page 46: ...ware Tools MFT site a ESXi 6 5 File mft 4 6 0 48 10EM 650 0 0 4598673 x86_64 vib b MD5SUM 0804cffe30913a7b4017445a0f0adbe1 2 Install the image according to the steps described in the MFT User Manual The following procedure requires custom boot image downloading mounting and booting from a USB device ...

Page 47: ...p to date Perform FW update y N y Device 1 Up to date Device 2 Updating FW Done Restart needed for updates to take effect Log File var log mlxup mlxup yyyymmdd log Updating Adapter Firmware Each adapter card is shipped with the latest version of qualified firmware at the time of manufacturing However Mellanox issues firmware updates occasionally that provide new features and bug fixes To check tha...

Page 48: ...therboard has the latest BIOS Link indicator light is off Try another port on theswitch Make sure the cable is securelyattached Check you are using the proper cables that do not exceed the recommended lengths Verify that your switch and adapter port arecompatible Link light is on but with no communication established Check that the latest driver is loaded Check that both the adapter and its link a...

Page 49: ...from the Filemenu Assign a file name and save Mellanox Firmware Tool MFT Download and install MFT http www mellanox com content pages php pg management_tools menu_section 34 Refer to the User Manual for installation instructions Once installed open a CMD window and run WinMFT mst start mst status flint d mst_device q Ports Information vstat Firmware Version Upgrade Download the latest firmware ver...

Page 50: ...pecifications Physical Size 2 71 in x 5 6 in 68 90mm x 142 24 mm low profile Connector Single SFP28 copper and optical Protocol Support Ethernet 10GBASE SR 10GBASE LR 10GBASE ER 10GBASE CX4 SGMII 1000BASE X Data Rate 1 10 Gb s Ethernet ...

Page 51: ... b For both operational and non operational states MCX4121A ACAT Specifications Physical Size 2 71 in x 5 6 in 68 90mm x 142 24 mm low profile Connector Dual SFP28 copper and optical Protocol Support Ethernet 25GBASE R 20GBASE KR2 1000BASE CX 1000BASE KX 10GBASE SR 10GBASE LR 10GBASE ER 10GBASE CX4 10GBASE CR 10GBASE KR SGMII Data Rate 1 10 25 Gb s Ethernet PCI Express Gen3 SERDES 8 0GT s 8 lanes ...

Page 52: ...Airflow See Airflow Specifications Regulatory Safety CB cTUVus CE EMC CE FCC VCCI ICES RCM RoHS RoHS compliant Notes a Typical power for ATIS traffic load b For both operational and non operational states ...

Page 53: ... Port Port to Heatsink OPN Passive Active 1W Passive Active 1W MCX4121A XCAT 300 300 300 300 25Gb s Adapter Cards Airflow Airflow LFM Airflow Direction Heatsink to Port Port to Heatsink OPN Passive Active 1 5W Passive Active 1 5W MCX4121A ACAT 300 400 300 300 a Cells marked with are not applicable ...

Page 54: ...54 All PSB dimensions are in millimeters All the mechanical tolerances are 0 13mm Board Mechanical Drawing andDimensions Mechanical Drawing of Dual port 10 25 Gb s Adapter Card ...

Page 55: ...del of the card please refer to 3D Models at http www mellanox com page 3d_models All bracket dimensions are in millimeters All the mechanical tolerances are 0 25mm Dimensions in millimeters Bracket Mechanical Drawing ...

Page 56: ...56 Dual port Tall Bracket ...

Page 57: ... the following figures do not necessarily represent the latest revisions of thecards Finding the MAC on the Adapter Card Each Mellanox adapter card has a different identifier printed on the label serial number and the card MAC for the Ethernet protocol ...

Page 58: ...58 MCX4121A ACAT Board Label Example ...

Page 59: ...2016 1 9 Updated Adapter Card Installation Instructions on page 19 Added Card Un installation on page25 Updated the following tables MCX4121A XCAT Specifications Table on page 58 MCX4111A ACAT Specifications Table on page 62 MCX4121A ACAT Specifications on page63 Jun 2016 1 7 Added a note to SystemRequirements Apr 2016 1 6 Updated Physical and Logical LinkIndications Mar 2016 1 5 Removed I2C compa...

Page 60: ...nts List on page 9 and Updating Adapter Card Firmware Updated Device ID in ProductOverview Added IEEE 802 3by support See Features and Benefits Updated PCI Device Added VMware support See VMwareDriver Updated Specifications Added Single port Bracket Nov 2015 1 0 First release ...

Page 61: ... product is not necessarily performed by NVIDIA It is customer s sole responsibility to evaluate and determine the applicability of any information contained in this document ensure the product is suitable and fit for the application planned by customer and perform the necessary testing for the application in order to avoid a default of the application or the product Weaknesses in customer s produ...

Page 62: ... NVIDIA the NVIDIA logo and Mellanox are trademarks and or registered trademarks of Mellanox Technologies Ltd and or NVIDIA Corporation in the U S and in other countries Other company and product names may be trademarks of the respective companies with which they are associated Forthecomplete andmostupdatedlistofMellanoxtrademarks visithttp www mellanox com page trademarks Copyright 2020 Mellanox ...

Search results

Summary of Contents for MCX4121A-ACAT

Reviews:

Related manuals for MCX4121A-ACAT

Brands by name

Popular brands

Nvidia MCX4121A-ACAT, User Manual

Search results

Summary of Contents for MCX4121A-ACAT

Reviews:

Related manuals for MCX4121A-ACAT

Brands by name

Popular brands