background image

 ConnectX-2 Ethernet Card User Manual

Rev 1.6

Mellanox Technologies

3

Table of Contents

Table of Contents

 3

List of Figures

 5

List of Tables

 6

Revision History

 7

About this Manual

 8

Intended Audience

 8

Related Documentation

 8

Online Resources

 8

Document Conventions

 8

Technical Support

 8

Firmware and Software Updates

 9

Chapter 1    Overview

 10

1.1    Adapter Cards Covered in this Manual 

11

1.2    Mellanox Part Numbering Legend 

13

1.3    Finding the MAC and Serial Number on the Adapter Cards 

14

1.4    Safety Warnings 

14

Chapter 2    Adapter Card Interfaces

 16

2.1    I/O Interfaces 

16

2.2    Power 

18

2.3    Memory 

18

2.4    VPD Layout 

19

Chapter 3    Driver Software and Firmware

 22

3.1    Driver Software 

22

3.2    RDMA over Converged Ethernet (RoCE) 

22

3.3    Updating Card Firmware 

23

3.4    ConnectX EN PXE 

24

Chapter 4    Ethernet Adapter  Card Installation

 25

4.1    Hardware and Software Requirements 

25

4.2    Installation Instructions 

25

4.3    Set Up 

26

4.4    Cables and Modules 

31

Appendix A  Specifications

 34

 A.1 MNPH29D-X[ST]R Specifications

 34

 A.2 MNPH29C-X[ST]R Specifications

 35

 A.3 Board Mechanical Drawing and Dimensions

 35

 A.4 EMC Certification Statements

 37

Appendix B  Interface Connectors Pinout

 40

 B.1 I2C-Compatible Connector Pinout

 40

 B.2 PCI Express x8Connector Pinout

 40

 B.3 PCI Express Connector Pinout

 41

 B.4 SFP+ Connector Pinout

 41

Appendix C  Replacing a Tall Bracket With a Short Bracket

 43

 C.1 Remove the Existing Bracket from the Adapter Card

 43

 C.2 Installing the New Bracket

 43

Appendix D  Avertissements de sécurité d’installation (Warnings in French)

 45

Summary of Contents for MNPH29C-XSR

Page 1: ...www mellanox com ConnectX 2 EN Dual Port SFP Ethernet Adapter Card User Manual P N MNPH29C XSR MNPH29C XTR MNPH29D XSR MNPH29D XTR Rev 1 6...

Page 2: ...FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TE...

Page 3: ...Software 22 3 2 RDMA over Converged Ethernet RoCE 22 3 3 Updating Card Firmware 23 3 4 ConnectX EN PXE 24 Chapter 4 Ethernet Adapter Card Installation 25 4 1 Hardware and Software Requirements 25 4 2...

Page 4: ...Rev 1 6 Mellanox Technologies 4 Appendix E Sicherheitshinweise Warnings in German 46 Appendix F Advertencias de seguridad para la instalaci n Warnings in Spanish 47...

Page 5: ...h Jumper 19 Figure 8 Support Download Assistant 24 Figure 9 Hardware Devices 27 Figure 10 PCI Device 28 Figure 11 Module With Locking Mechanism Closed 32 Figure 12 Module With Locking Mechanism Open 3...

Page 6: ...le 5 LEDs 17 Table 6 Jumper Configuration 19 Table 7 VPD Layout for MNPH29D X ST R 19 Table 8 VPD Layout for MNPH29C X ST R 20 Table 9 Hardware and Software Requirements 25 Table 10 Specifications for...

Page 7: ...ni Form Factor Oct 28th 2010 1 3 Added Technical Support and Firmware and Software Updates sections to the Pref ace section Added RoCE Added ConnectX EN PXE Added Set up section This is a sanity check...

Page 8: ...ellanox com Document Conventions When discussing memory sizes MB and MBytes are used in this document to mean size in mega bytes The use of Mb or Mbits small b indicates size in mega bits Technical Su...

Page 9: ...ts through a Mellanox approved reseller should first seek assistance through their reseller Firmware and Software Updates The Mellanox support downloader contains software firmware and knowledge datab...

Page 10: ...circuit device The cards described in this manual have the following main features IEEE Std 802 3 compliant 10 Gb s Ethernet Dual SFP ports for connection Ethernet traffic PCI Express 2 0 1 1 compatib...

Page 11: ...ess SERDES Speed Data Transmission Rate of ports Short Tall Bracket RoHS Adapter IC Part Number MNPH29D XTR PCIe Gen2 5 0 GT s 10 10 Gb s Ethernet 2 ports Tall R 6 MT25408B0 FCCR TE MNPH29D XSR PCIe G...

Page 12: ...Overview Rev 1 6 Mellanox Technologies 12 Figure 1 SFP NIC Component Side and Bracket Views J1 U1 MNPH29C J1 U1 MNPH29D...

Page 13: ...two ports field I 9 to indicate PCI Express 2 0 x8 running at 5 0GT s field G C to indicate Generation C ConnectX 2 field X X to indicate no on board memory field B T to indicate a tall bracket and fi...

Page 14: ...page 46 For safety warnings in Spanish see Advertencias de seguridad para la instalaci n Warn ings in Spanish on page 47 1 Installation Instructions 2 Over temperature Port 1 uses the GUID or MAC ID d...

Page 15: ...connect or dis connect cables Some copper cables are heavy and not flexible as such they should be carefully attached to or detached from the connectors Refer to the cable manufacturer for spe cial w...

Page 16: ...e IEEE Std 802 3 Specification Figure 3 Port Numbering 2 1 1 Ethernet SFP Interface The ConnectX 2 device is compliant with the IEEE Std 802 3 The SFP port has one Tx Rx pair of SerDes Ethernet NIC ca...

Page 17: ...ion A valid data activity link with data transfer is designated by a blinking yel low LED indication If the LEDs are not active either the physical link or the logical link or both connections have no...

Page 18: ...pter cards support multiple memory devices through the PCI Flash and I2C 2 3 1 System Memory The card utilizes the PCI Express interface to store and access Ethernet fabric connection informa tion and...

Page 19: ...efault Configuration Comments Flash present not present connection open Flash present connection shorted Flash not present Figure 7 Flash Jumper connection open Flash present Header 1x2 Table 7 VPD La...

Page 20: ...STR_ZERO Reserved 0x00 255 Small Resource Type END Tag 0x11 0x78 Table 8 VPD Layout for MNPH29C X ST R Offset Decimal Item Value Format Description 0 Large Resource Type ID String Tag 0x02 0x82 1 Leng...

Page 21: ...ag 0x11 0x91 100 Length 7 0 LSB 0x99 101 Length 15 8 MSB 0x00 102 VPD Keyword V1 STR EFI Driver version 104 Length 0x6 105 Data N A STR_SPC 111 VPD Keyword YA STR Asset Tag 113 Length 0x20 114 Data N...

Page 22: ...n Data Center Bridging DCB to enable efficient and low cost implementations of RDMA over Ethernet supporting the entire breadth of RDMA and low latency features This includes reliable connected servic...

Page 23: ...le card using the flint tool of the Mellanox Firm ware Tools MFT package This package is available for download along with its user manual from the Mellanox Firmware Tools page See http www mellanox c...

Page 24: ...ased LAN environment To enable or disable PXE use the following procedure The user must use the MFT tools To disable one must delete the PXE ROM image from the flash To enable one must place the PXE R...

Page 25: ...ructions on how to install a PCI Express card Table 9 Hardware and Software Requirements Requirement Description Hardware PCI Express x8 or x16 slots Firmware and Software Operating Systems Distributi...

Page 26: ...Identify the Card in Your System For instructions to identify your adapter card using specific tools see http www mellanox com content pages php pg firmware_HCA_FW_identification Otherwise use the ins...

Page 27: ...Hardware Devices 3 Select a PCI Device InfiniBand Controller entry 4 Right click 5 Select Properties to display the PCI Device Properties window 6 Click the Details tab and select Device Instance Id...

Page 28: ...he sub string PCI VEN_15B3 DEV_6368 VEN is equal to 0x15B3 this is the Vendor ID of Mellanox Technologies and DEV is equal to 0x6368 this is a valid Mellanox Technologies PCI Device ID 8 If the PCI de...

Page 29: ...eth1 Supported ports TP Supported link modes 10baseT Half 10baseT Full 100baseT Half 100baseT Full 1000baseT Half 1000baseT Full Supports auto negotiation Yes Advertised link modes 10baseT Half 10base...

Page 30: ...new firmware Unzip the downloaded file and run flint mstflint command flint d device i fw file guid GUID guids 4 GUIDS mac MAC macs 2 MACs burn Check VPI setup Ethernet InfiniBand configuration For Ex...

Page 31: ...mlx4_en To set non default values to module parameters the following line should be added to the file etc modprobe conf options mlx4_en param_name value param_name value Ethernet Network Tuning To imp...

Page 32: ...t the module into the adapter card module cage 4 Close the locking Mechanism To remove the module from the cage 1 Unlock the locking mechanism by opening the handle 2 Pull the module out of the cage 4...

Page 33: ...ward or downward in the rack To remove a cable disengage the locks and slowly pull the connector away from the port recepta cle Both LED indicators will turn off when the cable is unseated Always inst...

Page 34: ...t IEEE Std 802 3ad Link Aggregation and Failover IEEE Std 802 3x Pause IEEE Std 802 1Q VLAN tags 1p priorities IEEE Std 802 1p Priorities IEEE P802 1au D2 0 Congestion Notification IEEE P802 1az D0 2...

Page 35: ...8W 0 C to 55 C Protocol Support Regulatory Ethernet PCI Express IEEE Std 802 3ae 10 Gigabit Ether net IEEE Std 802 3ad Link Aggregation and Failover IEEE Std 802 3x Pause IEEE Std 802 1Q VLAN tags 1p...

Page 36: ...re 13 Schematic of the ConnectX 2 Small Form Factor MNPH29D Adapter Card All dimensions are in millimeters All the mechanical tolerances are 0 1mm J4 J3 U1 J1 U3 Q1 L8 U4 U14 45 01 3 65 1 90 15 00 45...

Page 37: ...world Table 12 Adapter Cards Certification Status Adapter Card P N FCC VCCI EN ICES CE CB cTUVus KCC C TICK MNPH29C XTR YES YES YES YES YES YES YES YES YES MNPH29C XSR YES YES YES YES YES YES YES YES...

Page 38: ...e limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial envi ronment This equipment generates uses and can radiate radio freq...

Page 39: ...in a domestic environment radio interference may occur in which case the user may be required to take corrective actions A 4 5 KCC Certification Korea Device User s information A CLASS A device comme...

Page 40: ...ble Connector Plug and Pinout B 2 PCI Express x8Connector Pinout These cards use a standard PCI Express x8 edge connector and the PCI Express x8 standard pinout according to the PCI Express 2 0 specif...

Page 41: ...in Placement 13 70 8 50 Top VeeT VeeT VeeT TX_Fault TX_Disable SDA SCL MOD_ABS RS0 RX_LOS RS1 VeeR VeeR VeeR VccT VccR TD TD RD RD 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 0 1 1 1 1 1 1 1 1 1 2 10 VeeT Vee...

Page 42: ...th 4 7k 10k on host board to a voltage between 2 0V and 3 6V MOD_ABS pulls line low to indicate module is plugged in 5 SCL 2 wire Serial Interface Clock Line d 6 MOD_ABS Module Absent Grounded within...

Page 43: ...s holding the bracket in place 2 Push the bracket off using equal pressure at the top and bottom of the bracket See Figure 17 3 Save the two screws and the two fiber washers C 2 Installing the New Bra...

Page 44: ...Rev 1 6 Mellanox Technologies 44 Figure 18 Putting on the Bracket 6 Make sure that the LEDs are aligned onto the bracket holes 7 Use a torque driver to apply up to 2 lbs in torque on the screws...

Page 45: ...mp rature ambiante maximale est n cessaire En outre pour garantir un bon coulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des ouver tures de ventilation Pendant un orage il ne f...

Page 46: ...gstemperatur erforderlich Au erdem sollten mindestens 8 cm 3 in Freiraum um die Bel ftungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten Arbeiten Sie w hrend eines Gewitters und Blit...

Page 47: ...mendada 55 C 131 F Adem s para garantizar una circu laci n de aire adecuada se debe dejar como m nimo un espacio de 8 cm 3 pul gadas alrededor de las aberturas de ventilaci n No utilizar el equipo ni...

Page 48: ...Rev 1 6 Mellanox Technologies 48 C digos el ctricos locales y nacionales Este equipo se debe instalar conforme a los c digos el ctricos locales y nacio nales...

Reviews: