manualshive.com logo in svg

Mellanox Technologies MCX4411A-ACAN, User Manual, Page: 69 / 79

Share
Download
Mellanox Technologies MCX4411A-ACAN User Manual Download Page 69

Rev 2.5

69

Mellanox Technologies

A.2 PCI Express x8 Connector Pinout

The adapter cards use a standard PCI Express x8 FCI B2B connector and the PCI Express x8 
standard pinout according to the PCI Express 3.0 specification.

Figure 8: PCIe x8 Connector Pinout

Summary of Contents for MCX4411A-ACAN

Page 1: ...logies www mellanox com ConnectX 4 Lx Single and Dual 10 25 Gb s Ethernet SFP28 Port Adapter Card User Manual for Open Compute Project P N MCX4421A XCQN MCX4411A ACAN MCX4412 ACAN MCX4411A ACQN MCX442...

Page 2: ...http www mellanox com page trademarks NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND...

Page 3: ...2 PCI Express Interface 18 2 3 LEDs Interface 18 Chapter 3 Hardware Installation 19 3 1 System Requirements 19 3 1 1 Hardware 19 3 1 2 Operating Systems Distributions 19 3 1 3 Software Stacks 19 3 2 S...

Page 4: ...e After Installation 37 4 1 7 1 Updating the Device Online 37 4 1 7 2 Updating the Device Manually 37 4 1 7 3 Updating the Device Firmware Automatically upon System Boot 38 4 1 8 UEFI Secure Boot 38 4...

Page 5: ...A ACQN Specifications 62 7 5 MCX4421A ACQN Specifications 63 7 6 Adapter LEDs Operations 64 7 7 Board Mechanical Drawing and Dimensions 65 Appendix A Interface Connectors Pinouts 67 A 1 SFP28 Connecto...

Page 6: ...oftware Requirements 26 Table 7 mlnxofedinstall Return Codes 32 Table 8 Hardware and Software Requirements 41 Table 9 Hardware and Software Requirements 52 Table 10 MCX4421A XCQN Specifications Table...

Page 7: ...xample 48 Figure 5 Mechanical Drawing of Single port Cards 65 Figure 6 Mechanical Drawing of Dual port Cards 66 Figure 7 Rear View of Module With Pin Placement 67 Figure 8 PCIe x8 Connector Pinout 69...

Page 8: ...Specifications on page 61 MCX4411A ACQN Specifications on page 62 MCX4421A ACQN Specifications on page 63 Updated Adapter LEDs Operations on page 64 Added Mechanical Drawing of Single port Cards on p...

Page 9: ...on page 60 Added MCX4421A ACQN Board Label on page 63 and MCX4411A ACQN Board Label on page 63 December 2015 1 2 Added MCX4411A ACQN and updated necessary sections accordingly Updated Related Document...

Page 10: ...ftware Firm ware Tools mlxup Firmware Utility Mellanox OFED for Linux User Manual Document no 2877 User Manual describing OFED features performance tools con tent and configuration See http www mellan...

Page 11: ...ee http www mellanox com Products Software Ether net Drivers VMware Driver User Manual v4 15 2 0 Mellanox VMware for Ethernet Release Notes Release notes for Mellanox ConnectX 4 NATIVE ESXi stack See...

Page 12: ...ributions on page 14 Section 1 3 Connectivity on page 15 Section 1 4 Features and Benefits on page 16 1 1 Product Overview The following section provides the ordering part number port speed number of...

Page 13: ...t card with Host Management support MCX4421A ACQN Dual port card with Host Management support a See OCP 2 0 Spec Stacking Height on page 14 Data Transmission Rate Ethernet 1 10 25 Gb s Network Connect...

Page 14: ...2 0 Type 1 with 8mm stacking height See Figure 1 for the single port front view and Figure 2 for the dual port card front view Figure 1 Type 1 Vertical Stack Front View Single Port Cards Figure 2 Typ...

Page 15: ...duction Rev 2 5 15 Mellanox Technologies 1 3 Connectivity Interoperable with 1 10 25 Gb s Ethernet switches Passive copper cable with ESD protection Powered connectors for optical and active cable sup...

Page 16: ...es that encapsulate and de capsulate the overlay proto col header as well as offloads TCP stateless activities on the encapsulated packet RDMA and RDMA over Converged Ethernet RoCE ConnectX 4 Lx utili...

Page 17: ...Sideband Interface NC SI that can be connected to a BMC Enabled for MCX4421A XCQN MCX4411A ACQN and MCX 4421A ACQN Wake on LAN WoL Supported in MCX4421A XCQN MCX4411A ACQN and MCX4421A ACQN Reset on...

Page 18: ...e Tx Rx pair of SerDes Ethernet traffic is transmitted through the cards SFP28 connectors 2 2 PCI Express Interface The ConnectX 4 Lx adapter cards support PCI Express 3 0 1 1 and 2 0 compatible throu...

Page 19: ...lation Checklist 1 Verify that your system meets the hardware and software requirements stated above 2 Shut down your system if active 3 After shutting down the system turn off power and unplug the co...

Page 20: ...cate an available PCI Express slot for the adapter card Make sure the adapter clips or screws are open 4 Place the adapter card on the clips without applying any pressure Please note that the followin...

Page 21: ...ter please refer to http www mellanox com products interconnect cables configurator php 3 5 1 Cable Installation 1 All cables can be inserted or removed with the unit powered on 2 To insert a cable pr...

Page 22: ...around to the side of the chassis before bending upward or downward in the rack 5 To remove a cable disengage the locks and slowly pull the connector away from the port receptacle LED indicator will t...

Page 23: ...nstallation Rev 2 5 23 Mellanox Technologies 1 To remove the card disengage clip 1 and 2 on connector 2 To disconnect connector A gently pull the adapter card upwards Connector Side Clip 1 Clip 2 Conn...

Page 24: ...dapter card 4 To remove the card gently pull the adapter card upwards 3 7 Identify the Card in Your System 3 7 1 On Windows 1 Open Device Manager on the server Click start Run and then enter devmgmt m...

Page 25: ...b string PCI VEN_15B3 DEV_1003 VEN is equal to 0x15B3 this is the Vendor ID of Mellanox Technologies and DEV is equal to 1003 this is a valid Mellanox Technologies PCI Device ID 3 7 2 On Linux Get the...

Page 26: ...er OS label CPU arch iso You can download it from http www mellanox com Products Soft ware Ethernet Drivers Requirements Description Platforms A server platform with an adapter card based on one of th...

Page 27: ...prompted to acknowledge the deletion of the old packages If you need to install Mellanox OFED on an entire homogeneous cluster a common strategy is to mount the ISO image on one of the cluster nodes a...

Page 28: ...Linux has already installed the OpenSM Subnet Manager on your machine 4 1 3 1 Installation Procedure Step 1 Login to the installation machine as root Step 2 Mount the ISO image on your machine On Ubun...

Page 29: ...re update will occur and the installation script will print at the end of installation a message similar to the follow ing Device 1 Device Type ConnectX4 Lx Part Number MCX4411A ACAN Description Conne...

Page 30: ...ca_self_test ofed see the file hca_self_test readme under docs After the installer completes information about the Mellanox OFED installation such as prefix kernel version and installation parameters...

Page 31: ...n SLES and Fedora Distributions lib modules uname r extra mlnx ofa_kernel on RHEL and other RedHat like Distributions lib modules uname r updates dkms on Ubuntu Firmware The firmware of existing netwo...

Page 32: ...pplicable to RedHat OEL Fedora XenServer Operating Systems 4 1 5 1 Setting up MLNX_OFED YUM Repository Step 1 Log into the installation machine as root Step 2 Mount the ISO image on your machine and c...

Page 33: ...30 247 MB s RPM GPG KEY Mellanox saved 1354 1354 sudo rpm import RPM GPG KEY Mellanox warning rpmts_HdrFromFdno Header V3 DSA SHA1 Signature key ID 6224c050 NOKEY Retrieving key from file repos MLNX_O...

Page 34: ...red for running Mellanox cards mlnx ofed guest Installs packages required by guest OS mlnx ofed hpc Installs packages required for HPC mlnx ofed hypervisor Installs packages required by hypervisor OS...

Page 35: ...lable on Mellanox Technologies Web site as described in Section 4 1 7 Updating Firmware After Installation on page 37 4 1 5 3 Uninstalling Mellanox OFED using the YUM Tool Use the script usr sbin ofed...

Page 36: ...lanox sudo apt key add apt key list pub 1024D A9E4B643 2013 08 11 uid Mellanox Technologies support mellanox com sub 1024g 09FCC269 2013 08 11 sudo apt get update apt cache search mlnx ofed mlnx ofed...

Page 37: ...low The following steps are also appropriate in case you wish to burn newer firmware that you have downloaded from Mellanox Technologies Web site http www mellanox com Support Firmware Download Instal...

Page 38: ...e automatic firmware update procedure To do so edit the configurations file opt mellanox mlnx fw updater mlnx fw updater conf and provide a comma separated list of PCI devices to exclude from the firm...

Page 39: ...emoving the signature To remove the signature from the MLNX_OFED kernel modules Step 1 Remove the signature After the signature has been removed a massage as the below will no longer be presented upon...

Page 40: ...e Tuning Depending on the application of the user s system it may be necessary to modify the default con figuration of network adapters based on the ConnectX adapters In case tuning is required please...

Page 41: ...Task Manager File Run new task and enter CMD 3 Enter the following command On an x64 64 bit machine the output will be AMD64 Step 2 Go to the Mellanox WinOF 2 web page at http www mellanox com Produc...

Page 42: ...2 3 1 Attended Installation The following is an example of an installation session Step 1 Double click the exe and follow the GUI instructions to install MLNX_WinOF2 Step 2 Optional Manually configure...

Page 43: ...Driver Installation Rev 2 5 43 Mellanox Technologies Step 5 Read then accept the license agreement and click Next Step 6 Select the target folder for the installation...

Page 44: ...s an OEM card In this case the firmware will not be displayed If the user has a standard Mellanox card with an older firmware version the firmware will be updated accordingly However if the user has b...

Page 45: ...install Performance tools install the performance tools that are used to measure performance in user environment Documentation contains the User Manual and Release Notes Management tools installation...

Page 46: ...stallation Rev 2 5 46 Mellanox Technologies Step 9 In case firmware upgrade option was checked in Step 7 you will be notified if a firmware upgrade is required Step 10 Click Finish to complete the ins...

Page 47: ...Manager Upon installation completion the inf files can be located at ProgramFiles Mellanox MLNX_WinOF2 Drivers OS To see the Mellanox network adapters display the Device Manager and pull down the Net...

Page 48: ...ning Installation To extract the files without running installation perform the following steps Step 1 Open a CMD console Click Start Task Manager File Run new task and enter CMD Step 2 Extract the dr...

Page 49: ...Driver Installation Rev 2 5 49 Mellanox Technologies Step 3 Click Next to create a server image Step 4 Click Change and specify the location in which the files are extracted to...

Page 50: ...Driver Installation Rev 2 5 50 Mellanox Technologies Step 5 Click Install to extract this folder or click Change to install to a different folder Step 6 To complete the extraction click Finish...

Page 51: ...the driver Run 4 2 7 Firmware Upgrade If the machine has a standard Mellanox card with an older firmware version the firmware will be automatically updated as part of the WinOF 2 package installation...

Page 52: ...ble 9 Hardware and Software Requirements Requirements Description Platforms A server platform with an adapter card based on one of the following Mellanox Technologies HCA devices MT27630 ConnectX 4 LX...

Page 53: ...o unload the driver To load the driver To restart the driver Please unload the driver before removing it esxcli software vib remove n nmlx5 core To remove the modules the command must be run in the sa...

Page 54: ...File mft 4 3 0 25 10EM 550 0 0 1391871 x86_64 vib MD5SUM 62f953b77f4ea49c420ff4e1ce8b925b ESX 6 0 File mft 4 3 0 25 10EM 600 0 0 2768847 x86_64 vib MD5SUM 6a3bd63882f2ca3f696c1a96d62112df 2 Install th...

Page 55: ...Products Software Firmware Tools 5 1 Firmware Update Example server1 mlxup Querying Mellanox devices firmware Device Type ConnectX4 Lx Part Number MCX4411A ACAN Description ConnectX 4 Lx EN network in...

Page 56: ...deleted Reboot the server Adapters stopped working after installing another adapter Try removing and re installing all adapters Check that cables are connected properly Make sure your motherboard has...

Page 57: ...rep i Mellanox Mellanox Firmware Tool MFT Download and install MFT http www mellanox com content pages php pg management_tools menu_section 34 Refer to the User Manual for installation instructions On...

Page 58: ...menu_section 34 Refer to the User Manual for installation instructions Once installed open a CMD window and run cd C Program Files Mellanox WinMFT mst start mst status flint d mst_device q Ports Info...

Page 59: ...Power and Environmental Voltage 5VAUX 3 3VAUX Typical Powera a Typical power for ATIS traffic load Passive Cables 9 00W Maximum Power Passive Cables 10 63W 1 5W Active Cables 13 63W Maximum power ava...

Page 60: ...2 0 and 1 1 compatible Power and Environmental Voltage 5VAUX 3 3VAUX Typical Powera a Typical power for ATIS traffic load Passive Cables 8 41W Maximum Power Passive Cables 10 05W 1 5W Active Cables 11...

Page 61: ...0 and 1 1 compatible Power and Environmental Voltage 5VAUX 3 3VAUX Typical Powera a Typical power for ATIS traffic load Passive Cables 9 01W Maximum Power Passive Cables 10 83W 1 5W Active Cables 13...

Page 62: ...s 8 41W Maximum Power Passive Cables 10 05W 1 5W Active Cables 11 55W Max power available through SFP28 port 1 5W Temperature Operational 0 C to 35 C Non operational 40 C to 70 C Humidity 90 relative...

Page 63: ...bles 9 01W Maximum Power Passive Cables 10 83W 1 5W Active Cables 13 83W Max power available through SFP28 port 1 5W Temperature Operational 0 C to 35 C Non operational 40 C to 70 C Humidity 90 relati...

Page 64: ...nals LED0 Physical link speed A constant Green indicates physical link with the high est rated speed A constant Yellow indicates physical link with degraded speed If LED0 is off then the physical link...

Page 65: ...cations Rev 2 5 65 Mellanox Technologies 7 7 Board Mechanical Drawing and Dimensions Figure 5 Mechanical Drawing of Single port Cards All dimensions are in millimeters All the mechanical tolerances ar...

Page 66: ...Specifications Rev 2 5 66 Mellanox Technologies Figure 6 Mechanical Drawing of Dual port Cards...

Page 67: ...e 7 Rear View of Module With Pin Placement Table 16 SFP28 Connector Pinout Pin Symbol Name Description 1 VeeT Transmitter Ground Common with Receiver Ground a 2 TX_Fault Transmitter Fault b 3 TX_Disab...

Page 68: ...ound Common with Receiver Ground a a Circuit ground is internally isolated from chassis ground b TFAULT is an open collector drain output which should be pulled up with a 4 7k 10k Ohms resis tor on th...

Page 69: ...ogies A 2 PCI Express x8 Connector Pinout The adapter cards use a standard PCI Express x8 FCI B2B connector and the PCI Express x8 standard pinout according to the PCI Express 3 0 specification Figure...

Page 70: ...card has a different identifier printed on the label serial number and the card MAC for the Ethernet protocol Figure 9 MCX4421A XCQN Board Label Example Figure 10 MCX4411A ACAN Board Label Example The...

Page 71: ...Rev 2 5 71 Mellanox Technologies Figure 11 MCX4421A ACAN Board Label Example Figure 12 MCX4411A ACQN Board Label Example Figure 13 MCX4421A ACQN Board Label Example...

Page 72: ...1 F To guarantee proper air flow allow at least 8cm 3 inches of clearance around the ven tilation openings During periods of lightning activity do not work on the equipment or connect or dis connect c...

Page 73: ...se of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure CLASS 1 LASER PRODUCT and reference to the most recent laser stan...

Page 74: ...maximale est n cessaire En outre pour garantir un bon coulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des ouver tures de ventilation Pendant un orage il ne faut pas utiliser l...

Page 75: ...e en garde l utilisation de commandes ou de r glages ou l ex cution de proc dures autres que ce qui est sp cifi dans les pr sentes peut engendrer une exposition au rayonnement grave PRODUIT LASER DE C...

Page 76: ...gstemperatur erforderlich Au erdem sollten mindestens 8 cm 3 in Freiraum um die Bel ftungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten Arbeiten Sie w hrend eines Gewitters und Blit...

Page 77: ...ak Achtung Nutzung von Steuerungen oder Einstellungen oder Ausf hrung von Prozeduren die hier nicht spezifiziert sind kann zu gef hrlichem Strahlenkon takt f hren Klasse 1 Laserprodukt und Referenzen...

Page 78: ...ar una circulaci n de aire adecuada se debe dejar como m nimo un espacio de 8 cm 3 pulgadas alrededor de las aberturas de ventilaci n No utilizar el equipo ni conectar o desconectar cables durante per...

Page 79: ...ligrosos Precauci n el uso de controles o ajustes o la realizaci n de procedimientos distintos de los que aqu se especifican podr an causar exposici n a niveles de radiaci n peligrosos PRODUCTO L SER...

Reviews:

No comments