Mellanox Technologies MCX456M-ECAT User Manual Download Page 6

Page: 6 / 80

Rev 1.0

Mellanox Technologies

List of Tables

Revision History Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Table 2:

Documents List. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Table 3:

Dual Port EDR Adapter Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Table 4:

Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Table 5:

Hardware and Software Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

Table 6:

Hardware and Software Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

Table 7:

MCX456M-ECAT Specification Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

Table 8:

Physical and Logical Link Indications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

Table 9:

Slim-Line SAS Pinout Connector on the Auxiliary PCIe Connection Card . . . . . . . 68

Table 10: Slim-Line SAS Pinout Connector on the Adapter Card . . . . . . . . . . . . . . . . . . . . . . 70

Summary of Contents for MCX456M-ECAT

Page 1: ...Mellanox Technologies www mellanox com ConnectX 4 VPI Dual Port QSFP28 Socket Direct Adapter Card User Manual for Dual Socket Servers P N MCX456M ECAT Rev 1 0...

Page 2: ...tp www mellanox com page trademarks NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AN...

Page 3: ...eability 15 Chapter 2 Interfaces 16 2 1 InfiniBand Interface 16 2 2 EthernetQSFP28 Interface 16 2 3 PCI Express Interface 16 2 4 LED Interface 16 Chapter 3 Hardware Installation 17 3 1 System Requirem...

Page 4: ...37 4 1 6 Installing MLNX_OFED using apt get 37 4 1 6 1 Setting up MLNX_OFED apt get Repository 37 4 1 6 2 Installing MLNX_OFED using the apt get Tool 38 4 1 6 3 Uninstalling Mellanox OFED using the a...

Page 5: ...MCX456M ECAT Specifications 61 7 2 Airflow Specifications 62 7 3 Adapter Card LED Operations 62 7 4 Board Mechanical Drawing and Dimensions 63 7 5 Bracket Mechanical Drawings 64 Appendix A Interface...

Page 6: ...ble 4 Features 13 Table 5 Hardware and Software Requirements 28 Table 6 Hardware and Software Requirements 42 Table 7 MCX456M ECAT Specification Table 61 Table 8 Physical and Logical Link Indications...

Page 7: ...apter Card 63 Figure 4 Mechanical Drawing of the Auxiliary PCIe Connection Card 63 Figure 5 Mechanical Drawing of the Slim Line SAS Cable 63 Figure 6 Adapter Card Tall Bracket 64 Figure 7 Auxiliary PC...

Page 8: ...able Date Rev Comments Changes December 2016 1 1 Added Section 1 5 Manageability on page 15 Updated Section 3 1 1 Hardware on page 17 Updated Section 4 1 3 1 Installation Procedure on page 31 Added Se...

Page 9: ...nd query utility used to update the firmware See http www mellanox com Products Software Firmware Tools mlxup Firmware Utility Mellanox OFED for Linux User Manual Document no 2877 User Manual describi...

Page 10: ...port Services please see your contract for details regarding Technical Support Customers who purchased Mellanox products through a Mellanox approved reseller should first seek assistance through their...

Page 11: ...le through a PCIe x8 edge connector and the other bus through an x8 parallel connector to an Auxiliary PCIe Connection Card The two cards should be installed into two adjacent PCIe x8 slots and connec...

Page 12: ...r OPN MCX456M ECAT includes the following components Low Profile Adapter Card Dual port Auxiliary PCIe Connection Card Harness Slim Line SAS Cable Data Transmission Rate InfiniBand SDR DDR QDR EDR 100...

Page 13: ...it Ethernet IEEE 802 3by 25 Gigabit Ethernet IEEE 802 3ae 10 Gigabit Ethernet IEEE 802 3az Energy Efficient Ethernet IEEE 802 3ap based auto negotiation and KRstartup IEEE 802 3ad 802 1AX Link Aggrega...

Page 14: ...GPU to CPU and therefore significantly reduces application run time ConnectX 4 advanced acceleration technology enables higher cluster effi ciency and scalability to tens of thousands of nodes CPU off...

Page 15: ...geability through a BMC The Socket Direct PCIe stand up adapter can be connected to a BMC using MCTP over SMBus or MCTP over PCIe protocols as if it is a standard Mellanox PCIe stand up adapter For co...

Page 16: ...Table 4 Features on page 13 Ethernet traffic is transmitted through the cards QSFP28 connectors 2 3 PCI Express Interface The ConnectX 4 adapter card supports PCI Express 3 0 1 1 and 2 0 compatible th...

Page 17: ...4 3 2 1 Software Stacks Mellanox OpenFabric software package MLNX_OFED for Linux and WinOF 2 for Windows See Chapter 4 Driver Installation on page 28 3 3 Safety Precautions 1 Remove any metallic objec...

Page 18: ...this section To replace the brackets you will need the following parts The new bracket of the proper height The 2 screws saved from the removal of the bracket 3 5 1 Removing the Existing Bracket 1 Re...

Page 19: ...3 Connect the slim line SAS connector male on the slim line SAS cable to the female connec tor on the auxiliary PCIe connection card 4 Locate two available PCI Express slots on the server one for the...

Page 20: ...insert the auxiliary PCIe connection card into the PCI Express slot until firmly seated 6 Secure the auxiliary PCIe connection card using server s retention mechanisms Do not use excessive force when...

Page 21: ...ard into the PCI Express slot adjacent to the auxiliary PCIe connection card until firmly seated 8 Secure the adapter card using the server s retention mechanisms 9 Connect the slim line SAS cable to...

Page 22: ...termine the correct orientation of the connector to the adapter card before inserting the connector Do not try and insert the connector upside down This may damage the adapter card c Insert the connec...

Page 23: ...a cable disengage the locks and slowly pull the connector away from the port receptacle LED indicator on the adapter card will turn off when the cable is unseated 3 8 Cards Un Installation Instruction...

Page 24: ...connector on the adapter card 4 Disengage the retention mechanisms clips or screws on adapter card 5 Holding the card from its center gently pull the adapter card from the PCI Express slot 6 Disengage...

Page 25: ...0 25 Mellanox Technologies 7 Holding the card from its center gently pull the auxiliary PCIe connection card from the PCI Express slot 8 Disconnect the slim line SAS cable from the connector on the au...

Page 26: ...d properties window 4 Click the Details tab and select Hardware Ids Windows 2012 R2 from the Properties pull down menu Figure 1 PCI Device 5 In the Value display box check the fields VEN and DEV field...

Page 27: ...he PCI bus by running lspci and locating lines with the string Mella nox Technologies The list of Mellanox Technologies PCI Device IDs can be found in the PCI ID reposi tory at http pci ids ucw cz rea...

Page 28: ...llanox com Products Soft ware InfiniBand VPI Drivers Table 5 Hardware and Software Requirements Requirements Description Platforms A server platform with an adapter card based on one of the following...

Page 29: ...prompted to acknowledge the deletion of the old packages If you need to install Mellanox OFED on an entire homogeneous cluster a common strategy is to mount the ISO image on one of the cluster nodes a...

Page 30: ...information please see add kernel support option below On Ubuntu and Debian distributions drivers installation use Dynamic Kernel Module Support DKMS framework Thus the drivers compilation will take...

Page 31: ...e ISO image on your machine Step 3 Run the installation script MLNX_OFED_LINUX x x x rhel6 3 x86_64 mlnx_add_kernel_support sh m tmp MLNX_OFED_ LINUX x x x rhel6 3 x86_64 make tgz Note This program wi...

Page 32: ...Mellanox OFED components can be configured or reconfigured after the installation by modifying the relevant configuration files See the relevant chapters in this manual for details The list of the mod...

Page 33: ...es are installed under lib modules uname r updates on SLES and Fedora Distributions lib modules uname r extra mlnx ofa_kernel on RHEL and other RedHat like Distributions lib modules uname r updates dk...

Page 34: ...achine as root Step 2 Mount the ISO image on your machine and copy its content to a shared location in your net work You can download it from http www mellanox com Products Software Infini Band VPI Dr...

Page 35: ...20 13 52 30 247 MB s RPM GPG KEY Mellanox saved 1354 1354 sudo rpm import RPM GPG KEY Mellanox warning rpmts_HdrFromFdno Header V3 DSA SHA1 Signature key ID 6224c050 NOKEY Retrieving key from file re...

Page 36: ...d basic Installs basic packages required for running Mellanox cards mlnx ofed guest Installs packages required by guest OS mlnx ofed hpc Installs packages required for HPC mlnx ofed hypervisor Install...

Page 37: ...ng the YUM Tool Use the script usr sbin ofed_uninstall sh to uninstall the Mellanox OFED package The script is part of the ofed scripts RPM 4 1 6 Installing MLNX_OFED using apt get This type of instal...

Page 38: ...apt get update apt cache search mlnx ofed mlnx ofed vma eth MLNX_OFED vma eth installer package with DKMS support mlnx ofed hpc MLNX_OFED hpc installer package with DKMS support mlnx ofed vma vpi MLN...

Page 39: ...following steps are also appropriate in case you wish to burn newer firmware that you have downloaded from Mellanox Technologies Web site http www mellanox com Support Firmware Download Step 1 Get the...

Page 40: ...provide a comma separated list of PCI devices to exclude from the firmware update Example 4 1 8 UEFI Secure Boot All kernel modules included in MLNX_OFED for RHEL7 and SLES12 are signed with x 509 ke...

Page 41: ...ature To remove the signature from the MLNX_OFED kernel modules Step 1 Remove the signature After the signature has been removed a message as the below will no longer be presented upon module loading...

Page 42: ...e according to your Operating System please follow the steps below Step 1 Obtain the machine architecture For Windows Server 2012 2012 R2 8 1 Client 1 To go to the Start menu position your mouse in th...

Page 43: ...ed Installation The following is an example of an installation session Step 1 Double click the exe and follow the GUI instructions to install MLNX_WinOF2 Step 2 Optional Manually configure your setup...

Page 44: ...Rev 1 0 44 Mellanox Technologies Step 4 Click Next in the Welcome screen Step 5 Read then accept the license agreement and click Next...

Page 45: ...Driver Installation Rev 1 0 45 Mellanox Technologies Step 6 Select the target folder for the installation...

Page 46: ...ing cases If the user has an OEM card In this case the firmware will not be displayed If the user has a standard Mellanox card with an older firmware version the firmware will be updated accordingly H...

Page 47: ...net from a host OpenSM is part of the driver and installed automatically Performances tools install the performance tools that are used to measure performance in user environment Analyze tools install...

Page 48: ...Rev 1 0 48 Mellanox Technologies Management tools installation tools used for management such as mlxstat b Click Install to start the installation...

Page 49: ...stallation Rev 1 0 49 Mellanox Technologies Step 9 In case firmware upgrade option was checked in Step 7 you will be notified if a firmware upgrade is required Step 10 Click Finish to complete the ins...

Page 50: ...pon installation completion the inf files can be located at ProgramFiles Mellanox MLNX_WinOF2 Drivers OS To see the Mellanox network adapters display the Device Manager and pull down the Net work adap...

Page 51: ...ning Installation To extract the files without running installation perform the following steps Step 1 Open a CMD console Click Start Task Manager File Run new task and enter CMD Step 2 Extract the dr...

Page 52: ...Rev 1 0 52 Mellanox Technologies Step 3 Click Next to create a server image Step 4 Click Change and specify the location in which the files are extracted to...

Page 53: ...folder or click Change to install to a different folder Step 6 To complete the extraction click Finish 4 2 6 Uninstalling Mellanox WinOF 2 Driver 4 2 6 1 Attended Uninstallation To uninstall MLNX_WinO...

Page 54: ...n a Nano Server 4 2 8 1 Offline Installation To deploy the Driver in Nano Server Step 1 Go to the Mellanox WinOF web page at http www mellanox com Products InfiniBand VPI Drivers Windows SW Drivers St...

Page 55: ...l directory see Section 4 2 5 Extracting Files With out Running Installation on page 51 Step 4 Open a remote connection to the Nano server Step 5 Copy all the driver files to the Nano server Step 6 In...

Page 56: ...bug fixes To check that your card is programmed with the latest available firmware version down load the mlxup firmware update and query utility The utility can query for available Mellanox adapters...

Page 57: ...Status Up to date Device Type ConnectX4 Part Number MCX456M ECAT Description ConnectX 4 VPI adapter card with Multi Host supporting dual socket server EDR IB 100Gb s and 100GbE dual port QSFP28 dual P...

Page 58: ...ecessary Try using another cable Reinstall the drivers for the network driver files may be damaged or deleted Reboot the server Adapters stopped working after installing another adapter Try removing a...

Page 59: ...grep i Mellanox Mellanox Firmware Tool MFT Download and install MFT http www mellanox com con tent pages php pg management_tools menu_section 34 Refer to the User Manual for installation instructions...

Page 60: ...ction 34 Refer to the User Manual for installation instructions Once installed open a CMD window and run cd C Program Files Mellanox WinMFT mst start mst status flint d mst_device q Ports Information...

Page 61: ...E R4 50GBASE R2 50GBASE R4 40GBASE CR4 40GBASE KR4 40GBASESR4 40GBASE LR4 40GBASE ER4 40GBASE R2 25GBASE R 20GBASEKR2 1000BASE CX 1000BASE KX 10GBASE SR 10GBASE LR 10GBASE ER 10GBASE CX4 10GBASE KX4 1...

Page 62: ...rked with are not applicable Air Flow Direction Heat Sink to Port Port to Heat Sink Cable Type Passive Active 3 5W Passive Active 3 5W MCX456M ECAT 400 700 300 300 LED Color and State Description Off...

Page 63: ...Figure 3 Mechanical Drawing of the Dual port Adapter Card Figure 4 Mechanical Drawing of the Auxiliary PCIe Connection Card Figure 5 Mechanical Drawing of the Slim Line SAS Cable All dimensions are i...

Page 64: ...Rev 1 0 64 Mellanox Technologies 7 5 Bracket Mechanical Drawings Figure 6 Adapter Card Tall Bracket Figure 7 Auxiliary PCIe Connection Card Tall Bracket Port 1 Bi Color LED Bi Color LED Port 2...

Page 65: ...r and Cage Views Table 9 Connector Pin Number and Name to Signal Name Map Connector Pin Number Connector Pin Name Port A Signal Name 1 GND GND 2 TXN_2 Tx2n 3 TXP_2 Tx2p 4 GND GND 5 TXN_4 Tx4n 6 TXP_4...

Page 66: ...tan dards 15 RXN_3 Rx3n 16 GND GND 17 RXP_1 Rx1p 18 RXN_1 Rx1n 19 GND GND 20 GND GND 21 RXN_2 Rx2n 22 RXP_2 Rx2p 23 GND GND 24 RXN_4 Rx4n 25 RXP_4 Rx4p 26 GND GND 27 ModPrsl_Port0 Mod PrsL 28 IntL Int...

Page 67: ...Rev 1 0 67 Mellanox Technologies Figure 9 PCIe x8 Edge Connector Pinout Applies to both to the adapter card and the auxiliary PCIe connection card...

Page 68: ...7 GND A8 NC A9 S_WAKE1_CONN_L A10 GND A11 PCIE_REFCLK1_P A12 PCEI_REFCLK1_N A13 GND A14 PCIE_CX_CPU_1CP A15 PCIE_CX_CPU_1CN A16 GND A17 PCIE_CX_CPU_1DP A18 PCIE_CX_CPU_1DN A19 GND A20 PCIE_CX_CPU_1EP...

Page 69: ...CX_1CP B15 PCIE_CPU_CX_1CN B16 GND B17 PCIE_CPU_CX_1D B18 PCIE_CPU_CX_1DN B19 GND B20 PCIE_CPU_CX_1EP B21 PCIE_CPU_CX_1EN B22 GND B23 PCIE_CPU_CX_1FP B24 PCIE_CPU_CX_1FN B25 GND B26 NC B27 NC B28 GND...

Page 70: ...I2C_SMDAT_H1 A10 GND A11 S_PERST1_CONN_L A12 S_PRSNT1_L A13 GND A14 PCIE_CPU_CX_1CP A15 PCIE_CPU_CX_1CN A16 GND A17 PCIE_CPU_CX_1D A18 PCIE_CPU_CX_1DN A19 GND A20 PCIE_CPU_CX_1EP A21 PCIE_CPU_CX_1EN A...

Page 71: ...B15 PCIE_CX_CPU_1CN B16 GND B17 PCIE_CX_CPU_1DP B18 PCIE_CX_CPU_1DN B19 GND B20 PCIE_CX_CPU_1EP B21 PCIE_CX_CPU_1EN B22 GND B23 PCIE_CX_CPU_1FP B24 PCIE_CX_CPU_1FN B25 GND B26 NC B27 NC B28 GND B29 NC...

Page 72: ...er printed on the label serial number and the card MAC for the Ethernet protocol and the card GUID for the InfiniBand protocol VPI cards have both a GUID and a MAC derived from the GUID Figure 10 MCX4...

Page 73: ...1 F To guarantee proper air flow allow at least 8cm 3 inches of clearance around the ven tilation openings During periods of lightning activity do not work on the equipment or connect or dis connect c...

Page 74: ...se of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure CLASS 1 LASER PRODUCT and reference to the most recent laser stan...

Page 75: ...maximale est n cessaire En outre pour garantir un bon coulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des ouver tures de ventilation Pendant un orage il ne faut pas utiliser l...

Page 76: ...e en garde l utilisation de commandes ou de r glages ou l ex cution de proc dures autres que ce qui est sp cifi dans les pr sentes peut engendrer une exposition au rayonnement grave PRODUIT LASER DE C...

Page 77: ...gstemperatur erforderlich Au erdem sollten mindestens 8 cm 3 in Freiraum um die Bel ftungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten Arbeiten Sie w hrend eines Gewitters und Blit...

Page 78: ...ak Achtung Nutzung von Steuerungen oder Einstellungen oder Ausf hrung von Prozeduren die hier nicht spezifiziert sind kann zu gef hrlichem Strahlenkon takt f hren Klasse 1 Laserprodukt und Referenzen...

Page 79: ...ar una circulaci n de aire adecuada se debe dejar como m nimo un espacio de 8 cm 3 pulgadas alrededor de las aberturas de ventilaci n No utilizar el equipo ni conectar o desconectar cables durante per...

Page 80: ...ligrosos Precauci n el uso de controles o ajustes o la realizaci n de procedimientos distintos de los que aqu se especifican podr an causar exposici n a niveles de radiaci n peligrosos PRODUCTO L SER...

Search results

Summary of Contents for MCX456M-ECAT

Reviews:

Related manuals for MCX456M-ECAT

Brands by name

Popular brands

Mellanox Technologies MCX456M-ECAT, User Manual

Search results

Summary of Contents for MCX456M-ECAT

Reviews:

Related manuals for MCX456M-ECAT

Brands by name

Popular brands