Mellanox Technologies Mellanox ConnectX MCX556M-ECAT-S25 User Manual Download Page 1

Page: 1 / 85

Mellanox Technologies

www.mellanox.com

Mellanox ConnectX®-5 VPI Dual Port Socket Direct
Adapter Card User Manual for Dual-Socket Servers

P/N:
MCX556M-ECAT-S25

Rev 1.6

Summary of Contents for Mellanox ConnectX MCX556M-ECAT-S25

Page 1: ...Mellanox Technologies www mellanox com Mellanox ConnectX 5 VPI Dual Port Socket Direct Adapter Card User Manual for Dual Socket Servers P N MCX556M ECAT S25 Rev 1 6...

Page 2: ...pdated list of Mellanox trademarks visit http www mellanox com page trademarks NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNO...

Page 3: ...2 3 PCI Express Interface 15 2 4 LED Interface 15 Chapter 3 Hardware Installation 16 3 1 System Requirements 16 3 1 1 Hardware 16 3 1 2 Operating Systems Distributions 16 3 1 3 Software Stacks 16 3 2...

Page 4: ...1 5 4 Installing MLNX_OFED Using apt get Tool 40 4 1 5 5 Setting up MLNX_OFED apt get Repository 40 4 1 5 6 Installing MLNX_OFED Using the apt get Tool 40 4 1 5 7 Uninstalling Mellanox OFED Using the...

Page 5: ...61 7 2 Airflow Specifications 62 7 3 Adapter Card LED Operations 62 7 4 Board Mechanical Drawing and Dimensions 63 7 5 Bracket Mechanical Drawing 65 Appendix A Interface Connectors Pinout 66 A 1 QSFP...

Page 6: ...nd Software Requirements 44 Table 10 General Troubleshooting 58 Table 11 Linux Troubleshooting 59 Table 12 Windows Troubleshooting 60 Table 13 MCX556M ECAT S25 Specification Table 61 Table 14 Airflow...

Page 7: ...Drawing of the Auxiliary PCIe Connection Card 63 Figure 5 Mechanical Drawing of the Slim Line SAS Cable 64 Figure 6 Adapter Card Tall Bracket 65 Figure 7 Auxiliary PCIe Connection Card Tall Bracket 65...

Page 8: ...n page 44 January 2018 1 5 Added a note to System Requirements on page 16 Updated MCX556M ECAT S25 Specifications on page 61 Added a note to Board Mechanical Drawing and Dimensions on page 63 Updated...

Page 9: ...irmware Utility mlxup User Manual and Release Notes Mellanox firmware update and query utility used to update the firmware See http www mellanox com Products Software Firmware Tools mlxup Firmware Uti...

Page 10: ...purchased Mellanox products through a Mellanox approved reseller should first seek assistance through their reseller Firmware Updates The Mellanox support downloader contains software firmware and kn...

Page 11: ...talled in two PCIe slots each card gets a unique PCI Bus and Device number Each of the PCIe x8 busses sees two network ports in effect the two 100Gb s physical ports of the ConnectX 5 Socket Direct Ad...

Page 12: ...R QDR FDR EDR Ethernet 1 10 25 40 50 100 Gb s Network Connector Types Dual port QSFP28 located on the low profile adapter card PCI Express PCIe SerDes Speed Dual PCIe 3 0 x8 8GT s RoHS R6 Adapter IC P...

Page 13: ...Socket Direct supporting dual socket server EDR IB 100Gb s and 100GbE dual port QSFP28 2x PCIe3 0 x8 tall bracket ROHS R6 1 SA002142 Auxiliary PCIe Connection Card 1 HAR000629 25cm Slim Line SAS Harn...

Page 14: ...IEEE 802 3ae 10 Gigabit Ethernet IEEE 802 3ap based auto negotiation and KR startup Proprietary Ethernet protocols 20 40GBASE R2 50GBASE R4 IEEE 802 3ad 802 1AX Link Aggregation IEEE 802 1Q 802 1P VL...

Page 15: ...ad using ASAP2 TM Flexible match action flow tables Tunneling encapsulation decapsulation Quality of Service QoS Support for port based Quality of Service enabling various application requirements for...

Page 16: ...sive copper cable with ESD protection Powered connectors for optical and active cable support 1 6 Manageability Socket Direct technology maintains support for manageability through a BMC The Socket Di...

Page 17: ...t standards listed in Table 5 Features on page 12 Ethernet traffic is transmitted through the cards SFP28 connectors 2 3 PCI Express Interface The ConnectX 5 adapter card supports PCI Express Gen 3 0...

Page 18: ...inOF 2 for Windows See Chapter 4 Driver Installation 3 2 Safety Precautions 1 Remove any metallic objects from your hands and wrists 2 Make sure to use only insulated tools 3 Verify that the system is...

Page 19: ...is section To replace the brackets you will need the following parts The new brackets of the proper height The 2 screws saved from the removal of the brackets 3 4 1 Removing the Existing Bracket 1 Rem...

Page 20: ...Express slots on the server one for the adapter card and one for the auxiliary PCIe connection card Please note that the following figures are for illustration purposes only Use the Slim Line SAS harn...

Page 21: ...the card insert the auxiliary PCIe connection card into the PCI Express slot until firmly seated 6 Secure the auxiliary PCIe connection card using the server s retention mechanisms Do not use excessi...

Page 22: ...ard into the PCI Express slot adjacent to the auxiliary PCIe connection card until firmly seated 8 Secure the adapter card using the server s retention mechanisms 9 Connect the slim line SAS cable to...

Page 23: ...cable to the adapter card Do this by using a cable holder or tying the cable to the rack b Determine the correct orientation of the connector to the card before inserting the connector Do not try and...

Page 24: ...which allow for routing horizontally around to the side of the chassis before bending upward or downward in the rack 6 To remove a cable disengage the locks and slowly pull the connector away from the...

Page 25: ...e from the connector on the adapter card 4 Disengage the retention mechanisms clips or screws on adapter card 5 Holding the card from its center gently pull the adapter card from the PCI Express slot...

Page 26: ...6 24 Mellanox Technologies 7 Holding the card from its center gently pull the auxiliary PCIe connection card from the PCI Express slot 8 Disconnect the slim line SAS cable from the connector on the au...

Page 27: ...fields VEN and DEV fields are separated by In the display example above notice the sub string PCI VEN_15B3 DEV_1003 VEN is equal to 0x15B3 this is the Vendor ID of Mellanox Technologies and DEV is eq...

Page 28: ...nique PCI Bus and Device number Each of the PCIe x8 busses sees two network ports in effect the two 100Gb s physical ports of the ConnectX 5 Socket Direct Adapter are viewed as four net devices by the...

Page 29: ...software stacks that are part of the standard operating system distribution or another vendor s commercial stack Installs the MLNX_OFED_LINUX binary RPMs if they are available for the current kernel I...

Page 30: ...pecific amount of RAM For your machine to be part of the InfiniBand VPI fabric a Subnet Manager must be running on one of the fabric nodes At this point Mellanox OFED for Linux has already installed t...

Page 31: ...s dir tmp MLNX_OFED_LINUX x x x logs This program will install the MLNX_OFED_LINUX package on your machine Note that all other Mellanox OEM OFED RDMA or Distribution IB packages will be removed Those...

Page 32: ...cket ROHS R6 PSID MT_2190110032 PCI Device Name 0b 00 0 Base MAC 0000e41d2d5cf810 Versions Current Available FW 16 22 0228 16 22 0228 Status Up to date In case that your machine has an unsupported net...

Page 33: ...est ofed see the file docs readme_and_user_manual hca_self_test readme After installation completion information about the Mellanox OFED installation such as prefix kernel version and installation par...

Page 34: ...tes on SLES and Fedora Distributions lib modules uname r extra mlnx ofa_kernel on RHEL and other RedHat like Distributions Firmware The firmware of existing network adapter devices will be updated if...

Page 35: ...ules exist in the initramfs file they can automatically be loaded by the kernel To prevent this behavior update the initramfs using the operating systems standard tools Note The process of updating th...

Page 36: ...sical port of PCI function 0000 84 00 0 MT4119 is the PCI Device ID of the Mellanox ConnectX 5 adapters family Step 2 Check to which NUMA each netdevice is connected to Table 8 mlnxofedinstall Return...

Page 37: ...ped 0 overruns 0 carrier 0 collisions 0 p2p1 flags 4163 UP BROADCAST RUNNING MULTICAST mtu 1500 inet 2 1 203 1 netmask 255 0 0 0 broadcast 2 255 255 255 inet6 fe80 268a 7ff fe9d 4624 prefixlen 64 scop...

Page 38: ...CA mlx5_10 CA type MT4119 Number of ports 1 Firmware version 16 22 0228 Hardware version 0 Node GUID 0x248a0703009d4624 System image GUID 0x248a0703009d4624 Port 1 State Active Physical state LinkUp...

Page 39: ...194 0 Connecting to www mellanox com 72 3 194 0 80 connected HTTP request sent awaiting response 200 OK Length 1354 1 3K text plain Saving to RPM GPG KEY Mellanox 100 1 354 K s in 0s 2014 04 20 13 52...

Page 40: ...pc installer package with KMP support mlnx ofed hypervisor noarch MLNX_OFED hypervisor installer package with KMP support mlnx ofed vma noarch MLNX_OFED vma installer package with KMP support mlnx ofe...

Page 41: ...4 301 fc21 x86_64 noarch MLNX_OFED vma installer package for kernel 3 17 4 301 fc21 x86_64 without KMP support mlnx ofed vma eth 3 17 4 301 fc21 x86_64 noarch MLNX_OFED vma eth installer package for k...

Page 42: ...by invoking Where Step 2 Install the desired group deb file path to extracted MLNX_OFED package DEBS wget qO http www mellanox com downloads ofed RPM GPG KEY Mellanox sudo apt key add apt key list pu...

Page 43: ...nloaded from Mellanox Technologies Web site http www mellanox com Support Firmware Download Step 1 Get the device s PSID Step 2 Download the firmware BIN file from the Mellanox website or the OEM webs...

Page 44: ...enabled the Mellanox x 509 public key should be added to the UEFI Secure Boot key database and loaded onto the system key ring by the kernel Follow these steps below to add the Mellanox s x 509 publi...

Page 45: ...whose key is not in the kernel key ring So it s much easier to miss this mes sage You won t see it on repeated tests where you unload and reload a kernel module until you reboot There is no way to eli...

Page 46: ...CMD 3 Enter the following command On an x64 64 bit machine the output will be AMD64 Step 2 Go to the Mellanox WinOF 2 web page at http www mellanox com Products InfiniBand VPI Drivers Windows SW Driv...

Page 47: ...to upgrade your firmware version1 Installing the incorrect exe file is prohibited If you do so an error message will be displayed For example if you try to install a 64 bit exe on a 32 bit machine the...

Page 48: ...Driver Installation Rev 1 6 46 Mellanox Technologies Step 4 Click Next in the Welcome screen Step 5 Read then accept the license agreement and click Next...

Page 49: ...Driver Installation Rev 1 6 47 Mellanox Technologies Step 6 Select the target folder for the installation...

Page 50: ...firmware will be updated accordingly However if the user has both an OEM card and a Mellanox card only the Mellanox card will be updated Step 8 Select a Complete or Custom installation follow Step a a...

Page 51: ...Installation Rev 1 6 49 Mellanox Technologies Diagnostic Tools installation tools used for diagnostics such as mlx5cmd b Click Next to install the desired tools Step 9 Click Install to start the insta...

Page 52: ...tion Rev 1 6 50 Mellanox Technologies Step 10 In case that firmware upgrade option was checked in Step 7 you will be notified if a firm ware upgrade is required See Step 11 Click Finish to complete th...

Page 53: ...Upon installation completion the inf files can be located at ProgramFiles Mellanox MLNX_WinOF2 Drivers OS To see the Mellanox network adapters display the Device Manager and pull down the Network adap...

Page 54: ...e two 100Gb s physical ports of the ConnectX 5 Socket Direct Adapter are viewed as four netdevices by the system 4 2 5 Extracting Files Without Running Installation To extract the files without runnin...

Page 55: ...Driver Installation Rev 1 6 53 Mellanox Technologies Step 3 Click Next to create a server image Step 4 Click Change and specify the location in which the files are extracted to...

Page 56: ...Driver Installation Rev 1 6 54 Mellanox Technologies Step 5 Click Install to extract this folder or click Change to install to a different folder Step 6 To complete the extraction click Finish...

Page 57: ...o upgrade firmware manually please refer to MFT User Manual www mellanox com Products Ethernet Drivers Firmware Tools 4 2 8 Deploying the Driver on a Nano Server 4 2 8 1 Offline Installation To deploy...

Page 58: ...load the driver package Step 3 Extract the Mellanox drivers to a local directory see Section 4 2 5 Extracting Files With out Running Installation on page 52 Step 4 Open a remote connection to the Nano...

Page 59: ...rying Mellanox devices firmware Device Type ConnectX 5 Part Number MCX556M ECAT S25 Description ConnectX 5 VPI adapter card with Multi Host Socket Direct supporting dual socket server EDR IB 100Gb s a...

Page 60: ...Try using another cable Reinstall the drivers for the network driver files may be damaged or deleted Reboot the server Adapters stopped working after installing another adapter Try removing and re in...

Page 61: ...x Mellanox Firmware Tool MFT Download and install MFT http www mellanox com content pages php pg management_tools menu_section 34 Refer to the User Manual for installation instructions Once installed...

Page 62: ...ww mellanox com content pages php pg management_tools menu_section 34 Refer to the User Manual for installation instructions Once installed open a CMD window and run WinMFT mst start mst status flint...

Page 63: ...s when connected to another Mellanox Infini Band product Ethernet 100GBASE CR4 100GBASE KR4 100GBASE SR4 50GBASE R2 50GBASE R4 40GBASE CR4 40GBASE KR4 40GBASE SR4 40GBASE LR4 40GBASE ER4 40GBASE R2 25...

Page 64: ...t been established Blinking Ambera a 1 Hz Blinking Amber occurs due to running a beacon command for locating the adapter card 6 Hz blinking Amber indicates a problem with the link Solid Green Indicate...

Page 65: ...nsions Figure 3 Mechanical Drawing of the Dual port Adapter Card Figure 4 Mechanical Drawing of the Auxiliary PCIe Connection Card All dimensions are in millimeters All the mechanical tolerances are 0...

Page 66: ...Specifications Rev 1 6 64 Mellanox Technologies Figure 5 Mechanical Drawing of the Slim Line SAS Cable...

Page 67: ...Specifications Rev 1 6 65 Mellanox Technologies 7 5 Bracket Mechanical Drawing Figure 6 Adapter Card Tall Bracket Figure 7 Auxiliary PCIe Connection Card Tall Bracket...

Page 68: ...2n Transmitter Inverted Data Input 3 Tx2p Transmitter Non Inverted Data Input 4 GND Ground 5 Tx4n Transmitter Inverted Data Input 6 Tx4p Transmitter Non Inverted Data Input 7 GND Ground 8 ModSelL Modu...

Page 69: ...Ground 24 Rx2n Receiver Inverted Data Output 25 Rx2p Receiver Non Inverted Data Output 26 GND Ground 27 Mod PrsL Module Present 28 IntL Interrupt 29 VccTx 3 3 V Power supply transmitter 30 Vcc1 3 3 V...

Page 70: ...port PCI Express 3 0 two PCIe x8 connectors a standard PCI Express x8 edge connector and Slim Line SAS connector pinout according to SFF 9401 standards PCIe x16 Connector Pinout Figure 9 PCIe x8 Edge...

Page 71: ...CX_1BN A7 GND A8 I2C_SMCLK_H1 A9 I2C_SMDAT_H1 A10 GND A11 S_PERST1_CONN_L A12 S_PRSNT1_L A13 GND A14 PCIE_CPU_CX_1CP A15 PCIE_CPU_CX_1CN A16 GND A17 PCIE_CPU_CX_1D A18 PCIE_CPU_CX_1DN A19 GND A20 PCIE...

Page 72: ..._P B12 PCEI_REFCLK1_N B13 GND B14 PCIE_CX_CPU_1CP B15 PCIE_CX_CPU_1CN B16 GND B17 PCIE_CX_CPU_1DP B18 PCIE_CX_CPU_1DN B19 GND B20 PCIE_CX_CPU_1EP B21 PCIE_CX_CPU_1EN B22 GND B23 PCIE_CX_CPU_1FP B24 PC...

Page 73: ..._1AP A3 PCIE_CX_CPU_1AN A4 GND A5 PCIE_CX_CPU_1BP A6 PCIE_CX_CPU_1BN A7 GND A8 NC A9 S_WAKE1_CONN_L A10 GND A11 PCIE_REFCLK1_P A12 PCEI_REFCLK1_N A13 GND A14 PCIE_CX_CPU_1CP A15 PCIE_CX_CPU_1CN A16 GN...

Page 74: ...E_CPU_CX_1BP B6 PCIE_CPU_CX_1BN B7 GND B8 I2C_SMCLK_H1 B9 I2C_SMDAT_H1 B10 GND B11 S_PERST1_CONN_L B12 S_PRSNT1_L B13 GND B14 PCIE_CPU_CX_1CP B15 PCIE_CPU_CX_1CN B16 GND B17 PCIE_CPU_CX_1D B18 PCIE_CP...

Page 75: ...30 S_PRSNT1_R_L B31 GND B32 PCIE_CPU_CX_1GP B33 PCIE_CPU_CX_1GN B34 GND B35 PCIE_CPU_CX_1HP B36 PCIE_CPU_CX_1HN B37 GND SH1 GND SH2 GND SH3 GND SH4 GND Table 19 Slim Line SAS Pinout Connector on the A...

Page 76: ...ial number and the card MAC for the Ethernet protocol and the card GUID for the InfiniBand protocol VPI cards have both a GUID and a MAC derived from the GUID Figure 10 MCX556M ECAT S25 Board Label Ex...

Page 77: ...Rev 1 6 75 Mellanox Technologies Figure 12 Slim Line SAS Harness Board Label...

Page 78: ...1 F To guarantee proper air flow allow at least 8cm 3 inches of clearance around the ven tilation openings During periods of lightning activity do not work on the equipment or connect or dis connect c...

Page 79: ...se of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure CLASS 1 LASER PRODUCT and reference to the most recent laser stan...

Page 80: ...maximale est n cessaire En outre pour garantir un bon coulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des ouver tures de ventilation Pendant un orage il ne faut pas utiliser l...

Page 81: ...e en garde l utilisation de commandes ou de r glages ou l ex cution de proc dures autres que ce qui est sp cifi dans les pr sentes peut engendrer une exposition au rayonnement grave PRODUIT LASER DE C...

Page 82: ...gstemperatur erforderlich Au erdem sollten mindestens 8 cm 3 in Freiraum um die Bel ftungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten Arbeiten Sie w hrend eines Gewitters und Blit...

Page 83: ...ak Achtung Nutzung von Steuerungen oder Einstellungen oder Ausf hrung von Prozeduren die hier nicht spezifiziert sind kann zu gef hrlichem Strahlenkon takt f hren Klasse 1 Laserprodukt und Referenzen...

Page 84: ...ar una circulaci n de aire adecuada se debe dejar como m nimo un espacio de 8 cm 3 pulgadas alrededor de las aberturas de ventilaci n No utilizar el equipo ni conectar o desconectar cables durante per...

Page 85: ...ligrosos Precauci n el uso de controles o ajustes o la realizaci n de procedimientos distintos de los que aqu se especifican podr an causar exposici n a niveles de radiaci n peligrosos PRODUCTO L SER...

Search results

Summary of Contents for Mellanox ConnectX MCX556M-ECAT-S25

Reviews:

Related manuals for Mellanox ConnectX MCX556M-ECAT-S25

Brands by name

Popular brands

Mellanox Technologies Mellanox ConnectX MCX556M-ECAT-S25, User Manual

Search results

Summary of Contents for Mellanox ConnectX MCX556M-ECAT-S25

Reviews:

Related manuals for Mellanox ConnectX MCX556M-ECAT-S25

Brands by name

Popular brands