background image

Notice

This document is provided for information purposes only and shall not be regarded as a warranty of a certain 
functionality, condition, or quality of a product. Neither NVIDIA Corporation nor any of its direct or indirect subsidiaries 

and affiliates (collectively: “NVIDIA”) make any representations or warranties, expressed or implied, as to the accuracy 

or completeness of the information contained in this document and assumes no responsibility for any errors contained 
herein. NVIDIA shall have no liability for the consequences or use of such information or for any infringement of patents 
or other rights of third parties that may result from its use. This document is not a commitment to develop, release, or 
deliver any Material (defined below), code, or functionality.
NVIDIA reserves the right to make corrections, modifications, enhancements, improvements, and any other changes to 
this document, at any time without notice.
Customer should obtain the latest relevant information before placing orders and should verify that such information is 
current and complete.
NVIDIA products are sold subject to the NVIDIA standard terms and conditions of sale supplied at the time of order 
acknowledgement, unless otherwise agreed in an individual sales agreement signed by authorized representatives of 

NVIDIA and customer (“Terms of Sale”). NVIDIA hereby expressly objects to applying any customer general terms and 

conditions with regards to the purchase of the NVIDIA product referenced in this document. No contractual obligations 
are formed either directly or indirectly by this document.
NVIDIA products are not designed, authorized, or warranted to be suitable for use in medical, military, aircraft, space, or 
life support equipment, nor in applications where failure or malfunction of the NVIDIA product can reasonably be 
expected to result in personal injury, death, or property or environmental damage. NVIDIA accepts no liability for 
inclusion and/or use of NVIDIA products in such equipment or applications and therefore such inclusion and/or use is at 

customer’s own risk.

NVIDIA makes no representation or warranty that products based on this document will be suitable for any specified use. 

Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to 

evaluate and determine the applicability of any information contained in this document, ensure the product is suitable 
and fit for the application planned by customer, and perform the necessary testing for the application in order to avoid a 

default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability 

of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in 
this document. NVIDIA accepts no liability related to any default, damage, costs, or problem which may be based on or 
attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this document or (ii) customer product 
designs.
No license, either expressed or implied, is granted under any NVIDIA patent right, copyright, or other NVIDIA intellectual 
property right under this document. Information published by NVIDIA regarding third-party products or services does not 
constitute a license from NVIDIA to use such products or services or a warranty or endorsement thereof. Use of such 
information may require a license from a third party under the patents or other intellectual property rights of the third 
party, or a license from NVIDIA under the patents or other intellectual property rights of NVIDIA.
Reproduction of information in this document is permissible only if approved in advance by NVIDIA in writing, reproduced 
without alteration and in full compliance with all applicable export laws and regulations, and accompanied by all 
associated conditions, limitations, and notices.
THIS DOCUMENT AND ALL NVIDIA DESIGN SPECIFICATIONS, REFERENCE BOARDS, FILES, DRAWINGS, DIAGNOSTICS, LISTS, 

AND OTHER DOCUMENTS (TOGETHER AND SEPARATELY, “MATERIALS”) ARE BEING PROVIDED “AS IS.” NVIDIA MAKES NO 

WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE MATERIALS, AND EXPRESSLY 
DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. 
TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL NVIDIA BE LIABLE FOR ANY DAMAGES, INCLUDING WITHOUT 
LIMITATION ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND 
REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF ANY USE OF THIS DOCUMENT, EVEN IF NVIDIA HAS BEEN 
ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Notwithstanding any damages that customer might incur for any reason 

whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the products described herein shall be 

limited in accordance with the Terms of Sale for the product.

Trademarks
NVIDIA, the NVIDIA logo, and Mellanox are trademarks and/or registered trademarks of NVIDIA Corporation and/

or Mellanox Technologies Ltd. in the U.S. and in other countries. Other company and product names may be trademarks 

Summary of Contents for QM87 Series

Page 1: ...QM87xx 1U HDR 200Gb s InfiniBand Switch Systems User Manual QM8700 QM8790 1U HDR 200Gb s InfiniBand Switch Systems User Manual...

Page 2: ...he Rack 15 Telescopic Rail Kit 15 Removing the System from the Rack 19 Cable Installation 20 Splitter Breakout Cables and Adapters 21 Initial Power On 23 System Bring Up of Managed Systems 24 Configur...

Page 3: ...8 Unit Identification LED 39 Port LEDs 40 Inventory Pull out Tab 41 Troubleshooting 42 Specifications 43 Appendixes 44 Accessory and Replacement Parts 44 Thermal Threshold Definitions 44 Interface Spe...

Page 4: ...itch 40 QSFP56 ports 2 Power Supplies AC unmanaged standard depth P2C airflow Rail Kit 920 9B110 00 RH 0D0 MQM8790 HS2R Mellanox Quantum HDR InfiniBand Switch 40 QSFP56 ports 2 Power Supplies AC unman...

Page 5: ...tocol SHARP technology SHARP architecture enables the usage of all active data center devices to accelerate the communications frameworks resulting in order of magnitude applications performance impro...

Page 6: ...m model System Model HDR 200Gb s QSFP56 Interfaces Max Throughput QM8700 40 16Tb s QM8790 40 16Tb s Management Interfaces PSUs and Fans The table below lists the various management interfaces and avai...

Page 7: ...n the main menu click on Products InfiniBand VPI Switch Systems and select the desired product page Certifications The list of certifications such as EMC Safety and others per system for different reg...

Page 8: ...wer On 6 Perform system bring up System Bring Up of Managed Systems 7 Optional FRU replacements FRU Replacements Safety Warnings Prior to the installation please review the Safety Warnings Note that s...

Page 9: ...o power side outlet Red latches are placed on the power inlet side OPN designation is R Power side inlet to connector side outlet Blue latches are placed on the power inlet side OPN designation is F P...

Page 10: ...ctions see Telescopic Rail Kit Fixed Rail Kit Kit OPN Legacy Kit OPN Rack Size and Rack Depth Range 930 9BRKT 00JF 000 MTEF KIT C 430 800 mm The following parts are included in the fixed rail kit see...

Page 11: ...tallation selection It is important to keep the airflow within the rack in the same direction Note that the part of the system to which you choose to attach the rails the front panel direction as demo...

Page 12: ...s Short Racks 430 580 mm Installation Side View Front side ports Rear side FRUs Standard Racks 580 800 mm Installation Side View In short racks the system s ventilation openings should be framed by th...

Page 13: ...he Rails to the Chassis Attach the left and right rack mount ears C to the switch by gently pushing the switch chassis pins through the slider key holes until locking occurs Secure the system in the b...

Page 14: ...is supporting the system s weight perform the following steps Attach the two rack mount blades B to the back side FRU side of the rack by inserting four M6 screws E in the designated cage nuts Do not...

Page 15: ...stem s weight Loosen the screws attaching the rack mount ears C to the rack Do not remove them yet Loosen the screws attaching the rack mount blades B to the rack and pull the blades towards you while...

Page 16: ...ont rail D 10x M6 Standard cage nuts E 10x M6 Standard pan head Phillips screws F 2x Phillips100 DEG F H TYPE I ST ST 6 32 X 1 4 screw with around patch G 6x Flat head 100 DEG Phillips 4 40X3 16 ST ST...

Page 17: ...thin the rack or in case more space is needed for cable bending radius it is possible to recess the connector side or the FRU side by 3 15 8cm by optional placement of the system s rails The FRU side...

Page 18: ...ng the Inner Rails for Cable Accommodation Route the power cable through either of the inner rails and reassemble the brackets by screwing the 3 screws per rail provided with the rail kit H with a tor...

Page 19: ...ening the 8 screws inserted in Step 2 with a torque of 4 5 0 5 Nm Removing the System from the Rack To remove a unit from the rack Turn off the system and disconnect it from peripherals and from the e...

Page 20: ...en the physical connection is established When a logical connection is made the relevant port LED will turn on To remove a cable disengage the locks and slowly pull the connector away from the port re...

Page 21: ...he port prior to the split and z indicating the number of the resulting single lane port 1 2 Each sub physical port is then handled as an individual port For example splitting port 5 into 2 lanes give...

Page 22: ...o two 2X HDR100 ports The following diagrams attempt to show how the logical ports map onto the physical QSFP ports as viewed by the IB tools e g ibnetdiscover Switch Profile Non Splittable Suitable f...

Page 23: ...r Inlets Electric Caution Notifications MLNX OS will refer to this 4X port as 1 13 The system platform will automatically power on when AC power is applied There is no power system Check all boards po...

Page 24: ...ind the explanation in Disable Dynamic Host Configuration Protocol DHCP sufficient In case manual configuration is required please refer to the instructions in Manual Host Configuration Disable Dynami...

Page 25: ...s described in the table below Once you perform that you should get the CLI prompt of the system Serial Terminal Program Configuration Parameter Setting Baud Rate 115200 Data bits 8 Stop bits 1 Parity...

Page 26: ...ble DHCPv6 on the MGMT0 interface Step 6 Admin password Press Enter to leave unchanged new_password Step 6 Confirm admin password new_password To avoid illegal access to the machine please type a pass...

Page 27: ...Link up yes DHCP running yes IP address 10 209 28 50 Netmask 255 255 255 0 IPv6 enabled yes Autoconf enabled no Autoconf route yes Autoconf privacy no DHCPv6 running no IPv6 addresses 1 IPv6 address...

Page 28: ...t may be extracted without bringing down the system To extract a power supply unit Remove the power cord from the power supply unit Grasping the handle with your hand push the latch release with your...

Page 29: ...ush the latch release with your thumb while pulling the handle outward As the fan unit unseats the fan unit status LEDs will turn off Remove the fan unit Fan Module Latches To remove or replace a fan...

Page 30: ...resistance is felt Continue pressing the fan unit until it seats completely The green Fan Status LED should light If not extract the fan unit and reinsert it After two unsuccessful attempts to install...

Page 31: ...e subnet Each network requires a Subnet Manager to be running in either the system itself system based or on one of the nodes which is connected to the fabric host based The subnet manager OpenSM assi...

Page 32: ...manual are available for download under https network nvidia com products adapter software firmware tools Please select the package that suits your operating system In order to obtain information rega...

Page 33: ...vidia com support firmware firmware downloads select the Quantum System page If the current version is not the latest version follow the directions in the MFT User manual to burn the new firmware inba...

Page 34: ...otocol Mellanox systems support QDR FDR EDR HDR InfiniBand FDR is an InfiniBand data rate where each lane of a 4X port runs a bit rate of 14 0625Gb s with 64b 66b encoding resulting in an effective ba...

Page 35: ...the Console connector and is located on the front side of the system the RJ45 connector It can be used with the I C DB9 to RJ45 splitting harness This interface is not found in externally managed syst...

Page 36: ...e user admin LEDs See LED Notifications LED Notifications The system s LEDs are an important tool for hardware event notification and troubleshooting LEDs Symbols Symbol Name Description Normal Condit...

Page 37: ...Solid Amber Major error has occurred For example corrupted firmware system is overheated etc If the System Status LED shows amber five minutes after starting the system unplug the system and call you...

Page 38: ...uld be replaced Power Supply Status LEDs There are two power supply inlets in the system for redundancy The system can operate with only one power supply connected Each power supply unit has a single...

Page 39: ...AC cord unplugged or AC power lost while the second power supply still has AC input power Plug in the AC cord of the faulty PSU PS failure including voltage out of range and power cord disconnected Ch...

Page 40: ...of a single 4 lane port or of the higher 2 lane split port if a splitter cable is used Splitting Indication LEDs Each time you press on the Lane Select Button the Port LEDs display will switch to a d...

Page 41: ...able is plugged into the port with the other end of the connector plugged into a functioning port When a logical connection is made the LED will change to green When data is being transferred the ligh...

Page 42: ...e power cable Replace the PSU if needed The activity LED does not light up InfiniBand Make sure that there is an SM running in the fabric System boot failure The last software upgrade failed on x86 ba...

Page 43: ...RoHS compliant Power Input Voltage 100 127VAC 50 60Hz 4 5A 200 240 50 60Hz 4 4A Global Power Consumption Full power specifications are provided in NVIDIA QM87XX Full Technical Specifications document...

Page 44: ...MTEF PSF AC C 200G 1U systems 1100W AC Power Supply w P2C airflow 930 9BPSU 00JG 0 00 MTEF PSR AC C 200G 1U systems 1100W AC Power Supply w C2P airflow HAR000631 Harness RS232 2M cable DB9 to RJ 45 fo...

Page 45: ...reshold the device will auto shutdown upon crossing the Emergency 130 C threshold Interface Specifications QSFP Pin Description QSFP Pin Description Connector Pin Number Pin Name Signal Description 1...

Page 46: ...ND Ground 21 Rx2n Receiver Inverted Data Output 3 22 Rx2p Receiver Non Inverted Data Output 3 23 GND Ground 24 Rx4n Receiver Inverted Data Output 3 25 Rx4p Receiver Non Inverted Data Output 3 26 GND G...

Page 47: ...upplied RJ45 to DB9 Harness Pinout RJ 45 Console and I C interfaces are integrated in the same connector Due to that connecting any cable other than the Mellanox supplied console cable may cause an I...

Page 48: ...uts Disposal According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and not disposed of with regular household waste Dispose of thi...

Page 49: ...I C under Interfaces Added power consumption measuring details in Specifications January 2020 1 6 Added a warning to Interfaces under I2C section Interface Specifications under RJ45 to DB9 Harness Pi...

Page 50: ...by customer and perform the necessary testing for the application in order to avoid a default of the application or the product Weaknesses in customer s product designs may affect the quality and rel...

Page 51: ...of the respective companies with which they are associated Copyright 2022 NVIDIA Corporation affiliates All Rights Reserved...

Reviews: