Mellanox Technologies ConnectX-5 Ex User Manual Download Page 1

Page: 1 / 75

Mellanox Technologies

www.mellanox.com

Mellanox ConnectX®-5 Ex 100Gb/s VPI Single and Dual
Adapter Cards User Manual

P/N:

MCX555A-ECAT, MCX556A-ECAT, MCX556A-EDAT

Rev 1.5

Summary of Contents for ConnectX-5 Ex

Page 1: ...Mellanox Technologies www mellanox com Mellanox ConnectX 5 Ex 100Gb s VPI Single and Dual Adapter Cards User Manual P N MCX555A ECAT MCX556A ECAT MCX556A EDAT Rev 1 5...

Page 2: ...t updated list of Mellanox trademarks visit http www mellanox com page trademarks NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TEC...

Page 3: ...ace 13 2 4 LED Interface 13 Chapter 3 Hardware Installation 14 3 1 System Requirements 14 3 1 1 Hardware 14 3 2 Operating Systems Distributions 14 3 2 1 Software Stacks 14 3 3 Safety Precautions 14 3...

Page 4: ...apt get Repository 32 4 1 5 6 Installing MLNX_OFED Using the apt get Tool 33 4 1 5 7 Uninstalling Mellanox OFED Using the apt get Tool 34 4 1 6 Updating Firmware After Installation 34 4 1 6 1 Updatin...

Page 5: ...56A EDAT Specifications 57 7 4 MCX555A ECAT Specifications 58 7 5 MCX556A ECAT Specifications 59 7 6 MCX556A EDAT Specifications 60 7 7 Airflow Specifications 61 7 8 Adapter Card LED Operations 61 7 9...

Page 6: ...able 9 General Troubleshooting 52 Table 10 Linux Troubleshooting 53 Table 11 Windows Troubleshooting 54 Table 12 MCX555A ECAT Specification Table 55 Table 13 MCX556A ECAT Specification Table 56 Table...

Page 7: ...ple 20 Figure 2 Installation Results 46 Figure 3 Mechanical Drawing of the Dual port Adapter Cards 62 Figure 4 Dual port Tall Bracket 63 Figure 5 MCX555A ECAT Board Label Example 64 Figure 6 MCX556A E...

Page 8: ...page 61 Added a note to Hardware on page 14 November 2017 1 3 Updated Adapter Card LED Operations on page 61 October 2017 1 2 Updated the following specification tables MCX555A ECAT Specification Tab...

Page 9: ...Firmware Utility Mellanox OFED for Linux User Manual Document no 2877 User Manual describing OFED features performance Band diagnostic tools content and configuration See http www mellanox com Produc...

Page 10: ...e see your contract for details regarding Technical Support Customers who purchased Mellanox products through a Mellanox approved reseller should first seek assistance through their reseller Firmware...

Page 11: ...ECAT Dual port cards MCX556A ECAT MCX556A EDAT Data Transmission Rate InfiniBand SDR DDR QDR FDR EDR Ethernet 1 10 25 40 50 100 Gb s Network Connector Types Single and dual port QSFP28 PCI Express PCI...

Page 12: ...thernet IEEE 802 3ap based auto negotiation and KR startup Proprietary Ethernet protocols 20 40GBASE R2 50GBASE R4 IEEE 802 3ad 802 1AX Link Aggregation IEEE 802 1Q 802 1P VLAN tags and priority IEEE...

Page 13: ...llowing more available CPU for computation tasks Open VSwitch OVS offload using ASAP2 TM Flexible match action flow tables Tunneling encapsulation decapsulation Quality of Service QoS Support for port...

Page 14: ...on Rev 1 5 12 Mellanox Technologies 1 4 Connectivity Interoperable with 1 10 25 40 50 100 Gb s Ethernet switches Passive copper cable with ESD protection Powered connectors for optical and active cabl...

Page 15: ...d are compliant with the IEEE 802 3 Ethernet standards listed in Table 4 Features on page 10 Ethernet traffic is transmitted through the cards QSFP28 connectors 2 3 PCI Express Interface The ConnectX...

Page 16: ...list 1 Verify that your system meets the hardware and software requirements stated above 2 Shut down your system if active 3 After shutting down the system turn off power and unplug the cord Remove th...

Page 17: ...ews 3 5 2 Installing the New Bracket 1 Place the bracket onto the card until the screw holes line up 2 Screw on the bracket using the screws saved from the bracket removal procedure above 3 Make sure...

Page 18: ...y seated the port connectors are aligned with the slot opening and the adapter faceplate is visible against the system chassis 4 Secure the adapter with the adapter clip or screw 5 Close the system ca...

Page 19: ...mber LED indicator will light when the physical con nection is established that is when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a f...

Page 20: ...on 1 Verify that the system is powered off and unplugged 2 Wait 30 seconds 3 To remove the card disengage the retention mechanisms on the bracket clips or screws 4 Holding the adapter card from its ce...

Page 21: ...n Your System 3 9 1 On Windows 1 Open Device Manager on the server Click Start Run and then enter devmgmt msc 2 Expand System Devices and locate your Mellanox ConnectX 5 adapter card 3 Right click the...

Page 22: ...Mellanox Technologies and DEV is equal to 1018 for ConnectX 5 this is a valid Mellanox Technologies PCI Device ID 3 9 2 On Linux Get the device location on the PCI bus by running lspci and locating l...

Page 23: ...Hardware Installation Rev 1 5 21 Mellanox Technologies Network controller Mellanox Technologies MT28800 Family ConnectX 5...

Page 24: ...MLNX_OFED Step a Scroll down to the Download wizard and click the Download tab Step b Choose your relevant package depending on your host operating system Step c Click the desired ISO tgz package Tab...

Page 25: ...f you wish to perform a firmware upgrade using customized FW binaries you can provide a path to the folder that contains the FW binary files by running fw image dir Using this option the FW version em...

Page 26: ...odes At this point Mellanox OFED for Linux has already installed the OpenSM Subnet Manager on your machine For the list of installation options run mlnxofedinstall h On Redhat and SLES distributions w...

Page 27: ...s dir tmp MLNX_OFED_LINUX x x x logs This program will install the MLNX_OFED_LINUX package on your machine Note that all other Mellanox OEM OFED RDMA or Distribution IB packages will be removed Those...

Page 28: ...100GbE dual port QSFP28 PCIe4 0 x16 tall bracket ROHS R6 PSID MT_2190110032 PCI Device Name 0b 00 0 Base MAC 0000e41d2d5cf810 Versions Current Available FW 16 22 0228 16 22 0228 Status Up to date In c...

Page 29: ...nstall log for each selected package will be saved in a sepa rate log file Table 6 Installation Results Software Most of MLNX_OFED packages are installed under the usr directory except for the follow...

Page 30: ...es to the etc modprobe d mlnx conf file Step 2 Set ONBOOT no in the etc infiniband openib conf file Step 3 If the modules exist in the initramfs file they can automatically be loaded by the kernel To...

Page 31: ...unt the ISO image on your machine and copy its content to a shared location in your net work Step 3 Download and install Mellanox Technologies GPG KEY The key can be downloaded via the following link...

Page 32: ...0 13 52 30 247 MB s RPM GPG KEY Mellanox saved 1354 1354 sudo rpm import RPM GPG KEY Mellanox warning rpmts_HdrFromFdno Header V3 DSA SHA1 Signature key ID 6224c050 NOKEY Retrieving key from file repo...

Page 33: ...support mlnx ofed all Installs all available packages in MLNX_OFED mlnx ofed basic Installs basic packages required for running Mellanox cards mlnx ofed guest Installs packages required by guest OS m...

Page 34: ...Products Software Ethernet Drivers Step 3 Create an apt get repository configuration file called etc apt sources list d mlnx_ofed list with the following content yum install mlnx ofed all Loaded plugi...

Page 35: ...MLNX_OFED hpc installer package with DKMS support mlnx ofed vma vpi MLNX_OFED vma vpi installer package with DKMS support mlnx ofed basic MLNX_OFED basic installer package with DKMS support mlnx ofed...

Page 36: ...to manually update firmware on your adapter card s you need to perform the steps below The following steps are also appropriate in case that you wish to burn newer firmware that you have downloaded fr...

Page 37: ...r conf and provide a comma separated list of PCI devices to exclude from the firmware update Example 4 1 7 UEFI Secure Boot All kernel modules included in MLNX_OFED for RHEL7 and SLES12 are signed wit...

Page 38: ...nature from the MLNX_OFED kernel modules Step 1 Remove the signature After the signature has been removed a massage as the below will no longer be presented upon module loading However please note tha...

Page 39: ...Tuning Depending on the application of the user s system it may be necessary to modify the default con figuration of network adapters based on the ConnectX adapters In case that tuning is required ple...

Page 40: ...k Task Manager File Run new task and enter CMD 3 Enter the following command On an x64 64 bit machine the output will be AMD64 Step 2 Go to the Mellanox WinOF 2 web page at http www mellanox com Produ...

Page 41: ...nOF2 Step 2 Optional Manually configure your setup to contain the logs option replace LogFile with the relevant directory Step 3 Optional If you do not want to upgrade your firmware version1 WinOF 2 s...

Page 42: ...Driver Installation Rev 1 5 40 Mellanox Technologies Step 4 Click Next in the Welcome screen Step 5 Read then accept the license agreement and click Next...

Page 43: ...Driver Installation Rev 1 5 41 Mellanox Technologies Step 6 Select the target folder for the installation...

Page 44: ...firmware will be updated accordingly However if the user has both an OEM card and a Mellanox card only the Mellanox card will be updated Step 8 Select a Complete or Custom installation follow Step a a...

Page 45: ...Installation Rev 1 5 43 Mellanox Technologies Diagnostic Tools installation tools used for diagnostics such as mlx5cmd b Click Next to install the desired tools Step 9 Click Install to start the insta...

Page 46: ...tion Rev 1 5 44 Mellanox Technologies Step 10 In case that firmware upgrade option was checked in Step 7 you will be notified if a firm ware upgrade is required See Step 11 Click Finish to complete th...

Page 47: ...Upon installation completion the inf files can be located at ProgramFiles Mellanox MLNX_WinOF2 Drivers OS To see the Mellanox network adapters display the Device Manager and pull down the Network adap...

Page 48: ...Installation To extract the files without running installation perform the following steps Step 1 Open a CMD console Click Start Task Manager File Run new task and enter CMD Step 2 Extract the driver...

Page 49: ...Driver Installation Rev 1 5 47 Mellanox Technologies Step 3 Click Next to create a server image Step 4 Click Change and specify the location in which the files are extracted to...

Page 50: ...Driver Installation Rev 1 5 48 Mellanox Technologies Step 5 Click Install to extract this folder or click Change to install to a different folder Step 6 To complete the extraction click Finish...

Page 51: ...de firmware manually please refer to MFT User Manual www mellanox com Products InfiniBand VPIEthernet Drivers Firmware Tools 4 2 8 Deploying the Driver on a Nano Server 4 2 8 1 Offline Installation To...

Page 52: ...load the driver package Step 3 Extract the Mellanox drivers to a local directory see Section 4 2 5 Extracting Files With out Running Installation on page 46 Step 4 Open a remote connection to the Nano...

Page 53: ...oducts Software Firmware Tools 5 1 Firmware Update Example server1 mlxup Querying Mellanox devices firmware Device Type ConnectX 5 Part Number MCX556A EDAT Description ConnectX 5 Ex VPI adapter card E...

Page 54: ...Try using another cable Reinstall the drivers for the network driver files may be damaged or deleted Reboot the server Adapters stopped working after installing another adapter Try removing and re in...

Page 55: ...x Mellanox Firmware Tool MFT Download and install MFT http www mellanox com content pages php pg management_tools menu_section 34 Refer to the User Manual for installation instructions Once installed...

Page 56: ...ww mellanox com content pages php pg management_tools menu_section 34 Refer to the User Manual for installation instructions Once installed open a CMD window and run WinMFT mst start mst status flint...

Page 57: ...BASE CR4 100GBASE KR4 100GBASE SR4 50GBASE R2 50GBASE R4 40GBASE CR4 40GBASE KR4 40GBASE SR4 40GBASE LR4 40GBASE ER4 40GBASE R2 25GBASE R 20GBASE KR2 10GBASE LR 10GBASE ER 10GBASE CX4 10GBASE KX4 10GB...

Page 58: ...40GBASE ER4 40GBASE R2 25GBASE R 20GBASE KR2 10GBASE LR 10GBASE ER 10GBASE CX4 10GBASE KX4 10GBASE CR 10GBASE KR SGMII 1000BASE CX 1000BASE KX 10GBASE SR Data Rate InfiniBand SDR DDR QDR FDR EDR Ethe...

Page 59: ...40GBASE ER4 40GBASE R2 25GBASE R 20GBASE KR2 10GBASE LR 10GBASE ER 10GBASE CX4 10GBASE KX4 10GBASE CR 10GBASE KR SGMII 1000BASE CX 1000BASE KX 10GBASE SR Data Rate InfiniBand SDR DDR QDR FDR EDR Ethe...

Page 60: ...100GBASE KR4 100GBASE SR4 50GBASE R2 50GBASE R4 40GBASE CR4 40GBASE KR4 40GBASE SR4 40GBASE LR4 40GBASE ER4 40GBASE R2 25GBASE R 20GBASE KR2 10GBASE LR 10GBASE ER 10GBASE CX4 10GBASE KX4 10GBASE CR 1...

Page 61: ...40GBASE ER4 40GBASE R2 25GBASE R 20GBASE KR2 10GBASE LR 10GBASE ER 10GBASE CX4 10GBASE KX4 10GBASE CR 10GBASE KR SGMII 1000BASE CX 1000BASE KX 10GBASE SR Data Rate InfiniBand SDR DDR QDR FDR EDR Ethe...

Page 62: ...40GBASE ER4 40GBASE R2 25GBASE R 20GBASE KR2 10GBASE LR 10GBASE ER 10GBASE CX4 10GBASE KX4 10GBASE CR 10GBASE KR SGMII 1000BASE CX 1000BASE KX 10GBASE SR Data Rate InfiniBand SDR DDR QDR FDR EDR Ethe...

Page 63: ...400 LED Color and State Description Off A link has not been established Blinking Ambera a 1 Hz Blinking Amber occurs due to running a beacon command for locating the adapter card 4 Hz blinking Amber i...

Page 64: ...s 7 9 Board Mechanical Drawing and Dimensions Figure 3 Mechanical Drawing of the Dual port Adapter Cards All dimensions are in millimeters All the mechanical tolerances are 0 1mm Fore the 3D Model of...

Page 65: ...Specifications Rev 1 5 63 Mellanox Technologies 7 10 Bracket Mechanical Drawing Figure 4 Dual port Tall Bracket 21 59 121 05...

Page 66: ...abel serial number and the card MAC for the Ethernet protocol and the card GUID for the InfiniBand protocol VPI cards have both a GUID and a MAC derived from the GUID Figure 5 MCX555A ECAT Board Label...

Page 67: ...Rev 1 5 65 Mellanox Technologies Figure 7 MCX556A EDAT Board Label Example...

Page 68: ...1 F To guarantee proper air flow allow at least 8cm 3 inches of clearance around the ven tilation openings During periods of lightning activity do not work on the equipment or connect or dis connect c...

Page 69: ...se of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure CLASS 1 LASER PRODUCT and reference to the most recent laser stan...

Page 70: ...maximale est n cessaire En outre pour garantir un bon coulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des ouver tures de ventilation Pendant un orage il ne faut pas utiliser l...

Page 71: ...e en garde l utilisation de commandes ou de r glages ou l ex cution de proc dures autres que ce qui est sp cifi dans les pr sentes peut engendrer une exposition au rayonnement grave PRODUIT LASER DE C...

Page 72: ...gstemperatur erforderlich Au erdem sollten mindestens 8 cm 3 in Freiraum um die Bel ftungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten Arbeiten Sie w hrend eines Gewitters und Blit...

Page 73: ...ak Achtung Nutzung von Steuerungen oder Einstellungen oder Ausf hrung von Prozeduren die hier nicht spezifiziert sind kann zu gef hrlichem Strahlenkon takt f hren Klasse 1 Laserprodukt und Referenzen...

Page 74: ...ar una circulaci n de aire adecuada se debe dejar como m nimo un espacio de 8 cm 3 pulgadas alrededor de las aberturas de ventilaci n No utilizar el equipo ni conectar o desconectar cables durante per...

Page 75: ...ligrosos Precauci n el uso de controles o ajustes o la realizaci n de procedimientos distintos de los que aqu se especifican podr an causar exposici n a niveles de radiaci n peligrosos PRODUCTO L SER...

Search results

Summary of Contents for ConnectX-5 Ex

Reviews:

Related manuals for ConnectX-5 Ex

Brands by name

Popular brands

Mellanox Technologies ConnectX-5 Ex, User Manual

Search results

Summary of Contents for ConnectX-5 Ex

Reviews:

Related manuals for ConnectX-5 Ex

Brands by name

Popular brands