Nvidia MCX653105A-HDAL Скачать руководство пользователя

Страница: 65 / 107

•

•
•
•

Select a Complete or Custom installation, follow

Step a

onward.

Select the desired feature to install:

Performances tools - install the performance tools that are used to measure

performance in user environment

Documentation - contains the User Manual and Release Notes

Management tools - installation tools used for management, such as mlxstat

Diagnostic Tools - installation tools used for diagnostics, such as mlx5cmd

Содержание MCX653105A-HDAL

Страница 1: ...Exported on Sep 19 2022 12 34 PM NVIDIA ConnectX 6 InfiniBand VPI Adapter Cards User Manual...

Страница 2: ...cooled Intel Server System D50TNP Platforms 15 ConnectX 6 Socket Direct Cards 2x PCIe x16 15 Features and Benefits 16 Operating Systems Distributions 19 Connectivity 19 Manageability 19 Interfaces 20...

Страница 3: ...ing the Card 43 Driver Installation 47 Linux Driver Installation 47 Prerequisites 47 Downloading NVIDIA OFED 47 Installing NVIDIA OFED 49 Installing MLNX_OFED Using YUM 54 Installing MLNX_OFED Using a...

Страница 4: ...ions 85 MCX653105A ECAT Specifications 86 MCX653106A ECAT Specifications 88 MCX654105A HCAT Specifications 89 MCX654106A HCAT Specifications 91 MCX654106A ECAT Specifications 93 MCX653105A EFAT Specif...

Страница 5: ...6A ECAT ConnectX 6 VPI adapter card 100Gb s HDR100 EDR IB and 100GbE dual port QSFP56 PCIe3 0 4 0 x16 tall bracket MCX65310 5A HDAT ConnectX 6 VPI adapter card HDR IB 200Gb s and 200GbE single port Q...

Страница 6: ...lxup Documentation NVIDIA Firmware Tools MFT User Manual User Manual describing the set of MFT firmware management tools for a single node See MFT User Manual InfiniBand Architecture Specification Rel...

Страница 7: ...7 Revision History A list of the changes made to this document are provided in Document Revision History...

Страница 8: ...based on either ConnectX 6 or ConnectX 6 DE ConnectX 6 Dx enhanced for HPC applications Configuration OPN Marketing Description ConnectX 6 PCIe x8 Card MCX651105 A EDAT ConnectX 6 VPI adapter card 100...

Страница 9: ...ConnectX 6 Single slot Socket Direct Cards 2x PCIe x8 in a row MCX653105 A EFAT ConnectX 6 VPI adapter card 100Gb s HDR100 EDR IB and 100GbE single port QSFP56 PCIe3 0 4 0 Socket Direct 2x8 in a row t...

Страница 10: ...System D50TNP Platforms where an Intel liquid cooled cold plate is used for adapter cooling mechanism Part Number MCX653105A ECAT MCX653106A ECAT MCX653105A HDAT MCX653106A HDAT Form Factor Dimension...

Страница 11: ...CIe Gen 4 0 slot Part Number MCX683105AN HDAT Form Factor Dimensions PCIe Half Height Half Length 167 65mm x 68 90mm Data Transmission Rate InfiniBand SDR DDR QDR FDR EDR HDR100 HDR Network Connector...

Страница 12: ...s by enabling direct access from each CPU in a dual socket server to the network through its dedicated PCIe interface Please note that ConnectX 6 Socket Direct cards do not support Multi Host function...

Страница 13: ...Card PCIe Half Height Half Length 167 65mm x 68 90mm Auxiliary PCIe Connection Card 5 09 in x 2 32 in 129 30mm x 59 00mm Two 35cm Cabline CA II Plus harnesses Data Transmission Rate Ethernet 10 25 40...

Страница 14: ...ndles only its own traffic and not traffic from the other CPU A system with a custom PCI Express x16 slot that includes special signals is required for installing the card Please refer to PCI Express...

Страница 15: ...ms Category Qty Item Cards 1 ConnectX 6 adapter card Accessories 1 Adapter card short bracket 1 Adapter card tall bracket shipped assembled on the card 1 Accessory Kit with two 2 TIMs MEB000386 Connec...

Страница 16: ...Ie x8 x16 configurations PCIe Gen 3 0 8GT s and Gen 4 0 16GT s through an x8 x16 edge connector 2x PCIe x16 configurations PCIe Gen 3 0 4 0 SERDES 8 0 16 0 GT s through Edge Connector PCIe Gen 3 0 SER...

Страница 17: ...where each lane of a 2X port runs a bit rate of 53 125Gb s with a 64b 66b encoding resulting in an effective bandwidth of 100Gb s InfiniBa nd HDR A standard InfiniBand data rate where each lane of a 4...

Страница 18: ...PU resources available for computation tasks Open vSwitch OVS offload using ASAP2 TM Flexible match action flow tables Tunneling encapsulation decapsulation Quality of Service QoS Support for port bas...

Страница 19: ...rs for optical and active cable support Manageability ConnectX 6 technology maintains support for manageability through a BMC ConnectX 6 PCIe stand up adapter can be connected to a BMC using MCTP over...

Страница 20: ...8 x16 edge connectors The device can be either a master initiating the PCI Express bus operations or a subordinate responding to PCI bus operations The following lists PCIe interface features PCIe Gen...

Страница 21: ...following Error Type Description LED Behavior I2C I2C access to the networking ports fails Blinks until error is fixed Over current Over current condition of the networking ports Blinks until error i...

Страница 22: ...sm that automatically shuts down the ConnectX 6 card in cases of high temperature events improper thermal coupling or heatsink removal For the required airflow LFM per OPN please refer to Specificatio...

Страница 23: ...23 Thermal Sensors Unable to render include or excerpt include Could not retrieve page...

Страница 24: ...d 55 C 131 F An airflow of 200LFM at this maximum ambient temperature is required for HCA cards and NICs To guarantee proper airflow allow at least 8cm 3 inches of clearance around the ventilation ope...

Страница 25: ...ation Checklist 6 Optional Replace the full height mounting bracket with the supplied short bracket Bracket Replacement Instructions 7 Install the ConnectX 6 PCIe x8 x16 adapter card in the system Con...

Страница 26: ...ots is required for installing the cards Airflow Requirements ConnectX 6 adapter cards are offered with two airflow patterns from the heatsink to the network ports and vice versa as shown below Please...

Страница 27: ...ontents please refer to Package Contents Shut down your system if active Turn off the power to the system and disconnect the power cord Refer to the system documentation for instructions Before you in...

Страница 28: ...card in a system Choose the installation instructions according to the ConnectX 6 configuration you have purchased OPNs Installation Instructions MCX651105A EDAT MCX653105A HDAT MCX653106A HDAT MCX65...

Страница 29: ...ght when the physical connection is established that is when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a functioning port See LED Int...

Страница 30: ...us address 82 hexa decimal PCI Device numb er 00 and PCI Function number 0 and 1 Since the two PCIe cards are installed in two PCIe slots each card gets a unique PCI Bus and Device number Each of the...

Страница 31: ...al to 1018 for ConnectX 6 this is a valid NVIDIA PCI Device ID PCIe x8 16 Cards Installation Instructions Installing the Card If the PCI device does not have a NVIDIA adapter ID return to Step 2 to ch...

Страница 32: ...chassis Step 2 Applying even pressure at both corners of the card insert the adapter card in a PCI Express slot until firmly seated Please make sure to install the ConnectX 6 cards in a PCIe slot that...

Страница 33: ...ng precautions to avoid injury and prevent damage to system components Remove any metallic objects from your hands and wrists It is strongly recommended to use an ESD strap or other antistatic devices...

Страница 34: ...on Instructions The hardware installation section uses the terminology of white and black harnesses to differentiate between the two supplied cables Due to supply chain variations some cards may be su...

Страница 35: ...nesses Step 1 Slide the black and white Cabline CA II Plus harnesses through the retention clip while making sure the clip opening is facing the plugs The harnesses minimal bending radius is 10 mm App...

Страница 36: ...to the color coding As indicated on both sides of the card plug the black harness to the component side and the white harness to the print side Step 2 Verify the plugs are locked Step 3 Slide the ret...

Страница 37: ...37 Step 4 Clamp the retention clip Verify both latches are firmly locked...

Страница 38: ...harnesses on the PCIe Auxiliary Card As indicated on both sides of the Auxiliary connection card plug the black harness to the component side and the white harness to the print side Step 7 Verify the...

Страница 39: ...ConnectX 6 adapter and PCIe Auxiliary Connection cards in available PCI Express x16 slots in the chassis Step 1 Locate two available PCI Express x16 slots Step 2 Applying even pressure at both corner...

Страница 40: ...on card in the PCI Express slots until firmly seated Secure the ConnectX 6 adapter and PCIe Auxiliary Connection Cards to the chassis Step 1 Secure the brackets to the chassis with the bracket screw D...

Страница 41: ...o system components Remove any metallic objects from your hands and wrists It is strongly recommended to use an ESD strap or other antistatic devices Turn off the system and disconnect the power cord...

Страница 42: ...the PCI Express slot Cards for Intel Liquid Cooled Platforms Installation Instructions The below instructions apply to ConnectX 6 cards designed for Intel liquid cooled platforms with ASIC interposer...

Страница 43: ...efully apply the thermal pad on the coldplate while ensuring it thoroughly covers it The below figure indicates the position of the thermal pad Extra care should be taken not to damage the pad The bel...

Страница 44: ...is in place and intact Once the thermal pad is applied to the ASIC interposer the non tacky side should be visible on the card s faceplate Gently peel the liner of the pad s non tacky side visible on...

Страница 45: ...tructions Applying even pressure at both corners of the card insert the adapter card into the adapter riser until firmly seated Care must be taken to not harm the black bumpers located on the print si...

Страница 46: ...46 1 2 Applying even pressure on the riser gently insert the riser into the server Secure the riser with the supplied screws Please refer to the server blade documentation for more information...

Страница 47: ...ce ID For the latest list of device IDs please visit the NVIDIA website at http www nvidia com page firmware_HCA_FW_identification Operating System Linux operating system For the list of supported ope...

Страница 48: ...umber Each of the PCIe x16 busses sees two network ports in effect the two physical ports of the ConnectX 6 Socket Direct adapter are viewed as four net devices by the system Single port PCIe x16 Card...

Страница 49: ...The installation script removes all previously installed NVIDIA OFED packages and re installs from scratch You will be prompted to acknowledge the deletion of the old packages If you need to install N...

Страница 50: ...running on one of the fabric nodes At this point NVIDIA OFED for Linux has already installed the OpenSM Subnet Manager on your machine For the list of installation options run mlnxofedinstall h Instal...

Страница 51: ...sources should be added if the sources are not in their default location In case your machine has the latest firmware no firmware update will occur and the installation script will print at the end o...

Страница 52: ...nel on RHEL and other Red Hat like Distributions Firmware The firmware of existing network adapter devices will be updated if the following two conditions are fulfilled The installation script is run...

Страница 53: ...mlx4_core blacklist mlx4_en blacklist mlx5_core blacklist mlx5_ib Set ONBOOT no in the etc infiniband openib conf file If the modules exist in the initramfs file they can automatically be loaded by t...

Страница 54: ...r network mount o ro loop MLNX_OFED_LINUX ver OS label CPU arch iso mnt Download and install NVIDIA GPG KEY The key can be downloaded via the following link http www nvidia com downloads ofed RPM GPG...

Страница 55: ...ofed hypervisor noarch MLNX_OFED hypervisor installer package with KMP support mlnx ofed vma noarch MLNX_OFED vma installer package with KMP support mlnx ofed vma eth noarch MLNX_OFED vma eth installe...

Страница 56: ...y will contain unsigned RPMs therefore you should set gpgcheck 0 in the repository configuration file Install the desired group yum install mlnx ofed all Loaded plugins langpacks product id subscripti...

Страница 57: ...vma MLNX_OFED vma installer package with DKMS support mlnx ofed all MLNX_OFED all installer package with DKMS support Where mlnx ofed all MLNX_OFED all installer package mlnx ofed basic MLNX_OFED bas...

Страница 58: ...ease use u flag to perform the update Updating the Device Manually To update the device manually please refer to the OEM Firmware Download page at http www nvidia com page firmware_table_dell mtag oem...

Страница 59: ...from the automatic firmware update procedure To do so edit the configurations file opt mellanox mlnx fw updater mlnx fw updater conf and provide a comma separated list of PCI devices to exclude from t...

Страница 60: ...ature To remove the signature from the MLNX_OFED kernel modules Remove the signature rpm qa grep E kernel ib mlnx ofa_kernel iser srp knem mlnx rds mlnx nfsrdma mlnx nvme mlnx rdma rxe xargs rpm ql gr...

Страница 61: ...nload page Software Requirements Description Package Windows Server 2012 R2 MLNX_WinOF2 2_10_All_x64 exe Windows Server 2012 Windows Server 2016 Windows Server 2019 Windows 8 1 Client 64 bit only Wind...

Страница 62: ...an example of an installation session Double click the exe and follow the GUI instructions to install MLNX_WinOF2 Optional Manually configure your setup to contain the logs option replace LogFile with...

Страница 63: ...63 4 5 6 MLNX_WinOF2 2_10_50000_All_x64 exe v l vx MyLog txt 1 Click Next in the Welcome screen Read and accept the license agreement and click Next Select the target folder for the installation...

Страница 64: ...he user has an OEM card In this case the firmware will not be displayed If the user has a standard NVIDIA card with an older firmware version the firmware will be updated accordingly However if the us...

Страница 65: ...Performances tools install the performance tools that are used to measure performance in user environment Documentation contains the User Manual and Release Notes Management tools installation tools u...

Страница 66: ...66 b Click Next to install the desired tools 9 Click Install to start the installation...

Страница 67: ...67 10 In case firmware upgrade option was checked in Step 7 you will be notified if a firmware upgrade is required see 11 Click Finish to complete the installation 1 Unattended Installation...

Страница 68: ...MT_NDPROPERTY 1 Optional If you do not wish to upgrade your firmware version i e MT_SKIPFWUPGRD default value is False MLNX_WinOF2 Driver Version _ revision_version _All_Arch exe vMT_SKIPFWUPGRD 1 Ins...

Страница 69: ...ms and Features MLNX_WinOF2 Uninstall NOTE This requires elevated administrator privileges Unattended Uninstallation To uninstall MLNX_WinOF2 in unattended mode Open a CMD console Click Task Manager F...

Страница 70: ...Manager File Run new task and enter CMD Extract the driver and the tools MLNX_WinOF2 2_0_ revision_version _All_x64 a To extract only the driver file MLNX_WinOF2 2_0_ revision_version _All_x64 a vMT_...

Страница 71: ...71 5 Click Install to extract this folder or click Change to install to a different folder...

Страница 72: ...ease refer to the MFT User Manual at www nvidia com Products Ethernet Drivers Firmware Tools VMware Driver Installation This section describes VMware Driver Installation Hardware and Software Requirem...

Страница 73: ...8 1OEM 650 0 0 4240417 MEL PartnerSupported 2017 01 31 nmlx5 rdma 4 16 8 8 1OEM 650 0 0 4240417 MEL PartnerSupported 2017 01 31 Removing Earlier NVIDIA Drivers To remove all the drivers Log into the...

Страница 74: ...mft 4 6 0 48 10EM 650 0 0 4598673 x86_64 vib MD5SUM 0804cffe30913a7b4017445a0f0adbe1 Install the image according to the steps described in the MFT User Manual To remove the modules you must run the co...

Страница 75: ...Update Example server1 mlxup Querying Mellanox devices firmware Device Type ConnectX 6 Part Number MCX654106A HCAT Description ConnectX 6 VPI adapter card HDR IB 200Gb s and 200GbE dual port QSFP56 So...

Страница 76: ...s securely attached Check you are using the proper cables that do not exceed the recommended lengths Verify that your switch and adapter port are compatible Link light is on but with no communication...

Страница 77: ...supportdownloader Collect Log File cat var log messages dmesg system log journalctl Applicable on new operating systems cat var log syslog Windows Troubleshooting Environment Information From the Wind...

Страница 78: ...78 Collect Log File Event log viewer MST device logs mst start mst status flint d mst_device dc dump_configuration log mstdump mst_device dc mstdump log...

Страница 79: ...GBASE ER4 40GBASE R2 25GBASE R 20GBASE KR2 10GBASE LR 10GBASE ER 10GBASE CX4 10GBASE CR 10GBASE KR SGMII 1000BASE CX 1000BASE KX 10GBASE SR Data Rate InfiniBand SDR DDR QDR FDR EDR HDR100 Ethernet 1 1...

Страница 80: ...Auto Negotiation 1X 2X 4X SDR 2 5Gb s per lane DDR 5Gb s per lane QDR 10Gb s per lane FDR10 10 3125Gb s per lane FDR 14 0625Gb s per lane EDR 25Gb s per lane port HDR100 2 lane x 50Gb s per lane HDR 5...

Страница 81: ...ications requires NVONline login credentials Voltage 3 3Aux Maximum current 100mA Maximum power available through QSFP56 port 5W Envir onm ental Temperature Operational 0 C to 55 C Non operational 40...

Страница 82: ...ane x 50Gb s per lane HDR 50Gb s per lane port Ethernet 200GBASE CR4 200GBASE KR4 200GBASE SR4 100GBASE CR4 100GBASE KR4 100GBASE SR4 50GBASE R2 50GBASE R4 40GBASE CR4 40GBASE KR4 40GBASE SR4 40GBASE...

Страница 83: ...pliant Notes a The ConnectX 6 adapters supplement the IBTA auto negotiation specification to get better bit error rates and longer cable reaches This supplemental feature only initiates when connected...

Страница 84: ...R Data Rate InfiniBand SDR DDR QDR FDR EDR HDR100 HDR Ethernet 1 10 25 40 50 100 200 Gb s PCI Express Gen3 4 SERDES 8 0GT s 16GT s x16 lanes 2 0 and 1 1 compatible Adapt er Card Power Voltage 12V 3 3V...

Страница 85: ...port HDR100 2 lane x 50Gb s per lane HDR 50Gb s per lane port Ethernet 200GBASE CR4 200GBASE KR4 200GBASE SR4 100GBASE CR4 100GBASE KR4 100GBASE SR4 50GBASE R2 50GBASE R4 40GBASE CR4 40GBASE KR4 40GBA...

Страница 86: ...al and non operational states MCX653105A ECAT Specifications Physic al Adapter Card Size 6 6 in x 2 71 in 167 65mm x 68 90mm Connector Single QSFP56 InfiniBand and Ethernet copper and optical Protoc o...

Страница 87: ...s requires NVONline login credentials Voltage 3 3Aux Maximum current 100mA Maximum power available through QSFP56 port 5W Enviro nment al Temperature Operational 0 C to 55 C Non operational 40 C to 70...

Страница 88: ...Gb s per lane port Ethernet 100GBASE CR4 100GBASE KR4 100GBASE SR4 50GBASE R2 50GBASE R4 40GBASE CR4 40GBASE KR4 40GBASE SR4 40GBASE LR4 40GBASE ER4 40GBASE R2 25GBASE R 20GBASE KR2 10GBASE LR 10GBASE...

Страница 89: ...pplement the IBTA auto negotiation specification to get better bit error rates and longer cable reaches This supplemental feature only initiates when connected to another NVIDIA InfiniBand product b T...

Страница 90: ...R 20GBASE KR2 10GBASE LR 10GBASE ER 10GBASE CX4 10GBASE CR 10GBASE KR SGMII 1000BASE CX 1000BASE KX 10GBASE SR Data Rate InfiniBand SDR DDR QDR FDR EDR HDR100 HDR Ethernet 1 10 25 40 50 100 200 Gb s G...

Страница 91: ...duct b Typical power for ATIS traffic load c For both operational and non operational states d For engineering samples add 250LFM MCX654106A HCAT Specifications Physical Low Profile Adapter Card Size...

Страница 92: ...ASE ER 10GBASE CX4 10GBASE CR 10GBASE KR SGMII 1000BASE CX 1000BASE KX 10GBASE SR Data Rate InfiniBand SDR DDR QDR FDR EDR HDR100 HDR Ethernet 1 10 25 40 50 100 200 Gb s Gen3 SERDES 8 0GT s x16 lanes...

Страница 93: ...traffic load c For both operational and non operational states MCX654106A ECAT Specifications Physical Adapter Card Size 6 6 in x 2 71 in 167 65mm x 68 90mm Auxiliary PCIe Connection Card Size 5 09 in...

Страница 94: ...pter Card Power Voltage 12V 3 3VAUX Power Cable Typical Powerb Passive Cables 27 1W Maximum Power Please refer to ConnectX 6 VPI Power Specifications requires NVONline login credentials Voltage 3 3Aux...

Страница 95: ...niBand IBTA v1 4a Auto Negotiation 1X 2X 4X SDR 2 5Gb s per lane DDR 5Gb s per lane QDR 10Gb s per lane FDR10 10 3125Gb s per lane FDR 14 0625Gb s per lane EDR 25Gb s per lane port HDR100 2 lane x 50G...

Страница 96: ...Temperature Cable Type Airflow Direction Heatsink to Port Por t to He atsi nk Passive Cables 300 55 C 200 35 C NVIDIA Active 2 75W Cables 300 55 C 200 35 C Regula tory Safety CB cTUVus CE EMC CE FCC...

Страница 97: ...10GBASE ER 10GBASE CX4 10GBASE CR 10GBASE KR SGMII 1000BASE CX 1000BASE KX 10GBASE SR Data Rate InfiniBand SDR DDR QDR FDR EDR HDR100 Ethernet 1 10 25 40 50 100 Gb s Gen3 4 SERDES 8 0GT s 16GT s x16 l...

Страница 98: ...ectX 6 adapters supplement the IBTA auto negotiation specification to get better bit error rates and longer cable reaches This supplemental feature only initiates when connected to another NVIDIA Infi...

Страница 99: ...VONline login credentials Voltage 3 3Aux Maximum current 100mA Maximum power available through QSFP port 5W Enviro nment al Temperature Operational 0 C to 55 C Non operational 40 C to 70 C Humidity 90...

Страница 100: ...ional and non operational states Adapter Card and Bracket Mechanical Drawings and Dimensions Adapter Cards ConnectX 6 PCIe x16 Adapter Card ConnectX 6 PCIe x8 Adapter Card MCX683105AN HDAT Auxiliary P...

Страница 101: ...101 Brackets Dimensions Tall Bracket Short Bracket...

Страница 102: ...other CPU In order to allow this capability a system with a special PCI Express x16 slot is required Table 31 provides the pin definitions of the required four special PCIe pins P CI e p in Server Co...

Страница 103: ...d MAC for the Ethernet protocol and the card GUID for the InfiniBand protocol VPI cards have both a GUID and a MAC derived from the GUID MCX653105A HDAT Board Label Example MCX683105AN HDAT Board Labe...

Страница 104: ...19 2 4 Updated PCI Express Pinouts Description Aug 2019 2 3 Updated Hardware Installation Jul 2019 2 2 Updated Linux Driver and Identifying the card in the system to include lspci command output examp...

Страница 105: ...105 Date Revi sion Comments Changes Dec 2018 1 2 Updated Hardware Requirements Updated Product Overview Nov 2018 1 1 Updated Hardware Requirements Oct 2018 1 0 First release...

Страница 106: ...by customer and perform the necessary testing for the application in order to avoid a default of the application or the product Weaknesses in customer s product designs may affect the quality and rel...

Результаты поиска

Содержание MCX653105A-HDAL

Отзывы:

Похожие инструкции для MCX653105A-HDAL

Бренды по названию

Популярные бренды

Nvidia MCX653105A-HDAL, Руководство пользователя

Результаты поиска

Содержание MCX653105A-HDAL

Отзывы:

Похожие инструкции для MCX653105A-HDAL

Бренды по названию

Популярные бренды