Mellanox Technologies InfiniBand X User Manual Download

Page: 13 / 96

SwitchX 36-Port QSFP InfiniBand- Ethernet Gateway System Hardware User Manual

Rev 1.0

Mellanox Technologies

Table 4 - Management Software

Feature

Description

Management Software
(MLNX-OS®)

Software Management

•

Dual software image

•

Software and firmware updates

File management

•

FTP

•

TFTP

•

SCP

Logging

•

Event history log

•

SysLog support

Management Interface

•

DHCP/Zeroconf

•

IPv6

Chassis Management

•

Monitoring environmental controls

•

Power management

•

Auto temperature control

•

High availability

Network Management Inter-
faces

•

SNMP v1,v2c,v3

•

REST interfaces (XML Gateway)

Security

•

SSH

•

Telnet

•

RADIUS

•

Date and Time

•

NTP

Cables & Transceivers

•

Transceiver info

Virtual Port Interconnect®
(VPI)

•

Ethernet

•

InfiniBand

Table 5 - Ethernet Features

Feature

Description

General

•

Jumbo Frames (9K)

•

ACL - 24K rules (permit/deny)

•

Breakout cables

Ethernet support

•

48K Unicast MAC addresses

•

VLAN (802.1Q) - 4K

•

LAG/LACP (802.3ad), 16 links per LAG (36 LAGs)

•

Rapid Spanning Tree (802.1w)

•

Flow control (802.3x)

•

IGMP snooping v1,2

•

LLDP

•

ETS (802.1Qaz)

•

PFC (802.1Qbb)

•

Port mirroring

Summary of Contents for InfiniBand X

Page 1: ...www mellanox com SwitchX 36 Port QSFP InfiniBand Ethernet Gateway System Hardware User Manual P N MSX6036G 2SFS MSX6036G 2SRS Rev 1 0...

Page 2: ...IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HA...

Page 3: ...y Platform Hardware Overview 16 2 2 Status LEDs 16 2 2 1 System Status LEDs 17 2 2 2 Gateway Status LED 18 2 2 3 Fan Status LED 19 2 2 4 Power Supply Status LEDs 20 2 2 5 Bad Port LED 22 2 2 6 UID LED...

Page 4: ...56 5 1 Upgrading Software 56 Chapter 6 Troubleshooting 57 Appendix A Specification 59 A 1 Approved Cables 60 A 2 EMC Certifications 60 A 3 China CCC Warning Statement 61 Appendix B Thermal Threshold D...

Page 5: ...14 Port LEDs 23 Figure 15 Component Air Flow 25 Figure 16 Component Airflow Mismatch Wrong 25 Figure 17 Management Interfaces 27 Figure 18 Reset Button 28 Figure 19 Port Numbering 29 Figure 20 Top and...

Page 6: ...v 1 0 Mellanox Technologies 6 Figure 36 Port Splitting Options 52 Figure 37 Port Splitting Options 53 Figure 38 3 Examples of Port Mapping Assignment 55 Figure 39 QSFP Connector Male and Female Views...

Page 7: ...rt LED Assignments 22 Table 11 Port Connector Physical and Logical Link Assignments for Ethernet Mode 23 Table 12 Connector Physical and Logical Link Assignments for IB Mode 24 Table 13 Air Flow Direc...

Page 8: ...Rev 1 0 Mellanox Technologies 8 Revision History Table 1 Revision History Table Date Revision Description August 2013 1 0 Initial Release...

Page 9: ...ateway are used to describe the gate way unless explicitly indicated otherwise The following icons are used throughout this document to indicate information that is important to the user Table 2 Refer...

Page 10: ...icates a situation that can potentially cause personal injury or damage to hardware or software Place Field Decoder M Mellanox Technologies SX System Type SwitchX switch Family P Data Transfer Protoco...

Page 11: ...nd I O infra structure Combining InfiniBand and Ethernet into a single solution provides the ideal rack back bone for next generation data centers Built with Mellanox s SwitchX based switches the Infi...

Page 12: ...tures Network Interfaces 36 QSPF interfaces Interface types InfiniBand and Ethernet InfiniBand speed FDR 56Gb s FDR10 QDR 40Gb s DDR 20Gb s SDR 10Gb s Ethernet speed 10 40 56 GbE FRUs Redundant power...

Page 13: ...l controls Power management Auto temperature control High availability Network Management Inter faces SNMP v1 v2c v3 REST interfaces XML Gateway Security SSH Telnet RADIUS TACACS Date and Time NTP Cab...

Page 14: ...QSFP port is capable of up to 40GbE upgradable to 56 GbE with a license as well as 10GbE with a QSA QSFP to SFP adapter or QSFP to SFP cables 1 4 FDR Transmission Rate Mellanox systems support FDR fou...

Page 15: ...where each lane of a 4X port runs a bit rate of 10 3125 Gb s with a 64b 66b encoding resulting in an effective bandwidth of 40 Gb s FDR10 supports 20 more bandwidth over QDR using the same QSFP cables...

Page 16: ...nfigurations for the systems 2 Ethernet RJ 45 connectors for management 1 RJ 45 connector for connecting to a host PC 1 USB connector All gateways have various status LEDs for on site status informati...

Page 17: ...n Normal Conditions Gateway Status Shows the health of the gateway Solid green after 5 minutes Fan Status LED Shows the health of the fans Green Power supply 1 Shows the health of the right side power...

Page 18: ...r supply the PS2 LED will be red if only one PS unit is installed in the gateway the PS2 LED will be off If the PS LEDs are not green this indicates a problem with the power supplies The gateway is op...

Page 19: ...on managed systems only Solid Yellow To be implemented in a future SW release Solid Red Major Error Possible damage can result to the gateway Turn off immediately e g bad firmware can t boot overheate...

Page 20: ...upply Unit can be added to support hot swap ability and to add redundancy See Accessory and Replacement Parts on page 66 for ordering part numbers Table 8 Fan Status LED Assignments LED Configuration...

Page 21: ...e 11 PS Unit Status LEDs Table 9 shows the power supply unit status LED assignment Table 9 PS Unit Status LED Assignments LED Color Status Solid Green OK The Power supply is delivering the correct vol...

Page 22: ...ol error is received 2 2 6 UID LED Gateway Identifier1 The UID LED is a debug feature that lights a blue LED on the gateway connector side panel for ease in finding a particular gateway within a clust...

Page 23: ...according to the LED indication for Ethernet protocol Table 12 shows the port status according to the LED indication for InfiniBand protocol Figure 14 Port LEDs Table 11 Port Connector Physical and L...

Page 24: ...inlet to power side outlet Power side inlet to connector side outlet The air flow is specified in the product model number See Mellanox Part Numbering Legend on page 10 On the power side panel the PS...

Page 25: ...w Mismatch Wrong Table 13 shows the icons that indicate airflow using arrows and colors All components need the same colored airflow icons OK OK OK OK 12C These air flow labels must be the same Red is...

Page 26: ...owing interfaces connect to the gateway 2 100 M 1 Gb Ethernet connectors labeled MGT 1 USB port that is labeled This interface can be used to update software or firmware 1 connector that is labeled CO...

Page 27: ...e management is available For the Socket pinout see RJ 45 CONSOLE and I2C Inter face on page 65 RJ 45 Ethernet Connector MGT The 2 Ethernet connections labeled MGT provides access for remote managemen...

Page 28: ...utton is pushed A quick push of this button performs this reset When the button is held down for 15 sec onds the gateway is reset and the password is deleted You will then be able to enter without a p...

Page 29: ...anox MLNX OS user manual for instructions on port re config uration A future software release will support each port to be manually configured to be either Infiniband or Ethernet Specific ports can be...

Page 30: ...rack capable of supporting the mechanical and environmental charac teristics of a fully populated platform 3 1 Installation Safety Warnings For Safety Warnings in French see Avertissements de s curit...

Page 31: ...ernal parts Risk of electric shock and energy hazard The PSUs are all independent Disconnect all power supplies to ensure a powered down state inside of the gateway platform During periods of lightnin...

Page 32: ...the Canadian Electrical Code Warning Replace only with UL Recognized battery certified for maximum abnormal charging current not less than 4mA There is a risk of explosion should the battery be replac...

Page 33: ...l to the protective earth wire in the building The equipment shall be installed in area where equipotential bonding exists such as a telecommunication centre or a dedicated computer room Cables for co...

Page 34: ...ts for visible damage that may have occurred during shipping The gateway comes packed with the following items 1 gateway 1 power cable for each PS unit Type C13 C14 According to the WEEE Directive 200...

Page 35: ...strap Make sure the ESD strap is touching your skin and that the other end is connected to a verified ground If anything is damaged or missing contact your customer representative immediately For cus...

Page 36: ...can be as many as36 cables connected to the gateway Do you want the connector side recessed in the rack to allow for a larger cable bending radius It is possible to recess the connector side by 2 5cm...

Page 37: ...which mounting option you want to use 2 Screw the gateway slides onto the gateway Use 5 flat head screws for short gateway and 7 screws for standard depth gateway to connect each gateway slide Figure...

Page 38: ...on the opposite side of the rack Check that both sides of the gateway left and right are the same level in the rack 5 Slide the rail into the rail slide 6 Using two of the bolts for each corner insta...

Page 39: ...onal electrical codes require an external ground to all IT components If so connect a ground wire to one of the casing screws and connect the other end to a valid ground If you choose to not use the g...

Page 40: ...r cable This unit is intended for installation in a Restricted Access Location A restricted access area can be accessed only through the use of a special tool lock and key or other means of security C...

Page 41: ...immediately installing a new PSU you must install a blank so that the air flow for cooling the gateway is not compromised Never run the gateway for more than 5 minutes with an uncovered open PSU or F...

Page 42: ...ely The latch will snap into place confirm ing the proper installation 4 Insert the power cord into the supply connector 5 Insert the other end of the power cord into an outlet of the correct voltage...

Page 43: ...us indicator will turn off 2 Remove the fan unit Figure 33 Fan Module Latch To insert a FAN Unit 1 Make sure the mating connector of the new unit is free of any dirt and or obstacles Operation without...

Page 44: ...nsole RJ 45 port of the gateway system using the supplied cable The Console ports for systems are shown below as examples The port labelled CONSOLE must be connected to a local host PC This must be us...

Page 45: ...guration Example Wizard Session Display Comments Mellanox configuration wizard Do you want to use the wizard for initial configuration yes You must perform this configuration the first time you operat...

Page 46: ...nter Then confirm the password by re entering it Note that password characters are not printed You have entered the following information A summary of the configura tion is now displayed To change an...

Page 47: ...y IP address for example 192 168 10 4 10 10 10 10 Mask length may not be zero if address is not zero interface eth0 Step 5 Netmask 0 0 0 0 255 255 255 0 Step 6 Default gateway for example 192 168 10 1...

Page 48: ...0 interface yes no Step 3 Use zeroconf on mgmt0 interface no yes Step 4 Default gateway For example 192 168 10 1 Step 5 Primary DNS server Step 6 Domain name Step 7 Enable IPv6 yes Step 8 Enable IPv6...

Page 49: ...to the gateway using the command ssh l username gateway _IP_address where the gateway _IP_address is the IP address of the gateway RX bytes 1343502058 TX bytes 313920869 RX packets 17589211 TX packets...

Page 50: ...re ready to use the system Refer to MLNX OS Command Ref erence Guide for additional information on the CLI commands 5 You can enter any supported command now Rerunning the Wizard 6 If you want to reru...

Page 51: ...ector away from the port receptacle The LED indicator for that port will turn off when the cable is unseated Care should be taken not to impede the air flow through the ventilation holes next to the c...

Page 52: ...plitting Table Port Can be split to 4 Turns off port Can be split to 2 1 2 3 4 1 5 6 6 7 8 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 This port...

Page 53: ...Manual for instructions on port configuration When using this feature you MUST go into the MLNX OS chassis management system and reconfigure the individual ports to split 4 or split 2 See the MLNX OS...

Page 54: ...echnologies 54 10 7 28 25 11 29 12 30 13 31 14 13 32 31 15 18 33 36 16 34 17 35 18 36 Table 19 Port Splitting Options Port Can be split to 4 Turnsoffport Can be split to 2 Port Can be split to 4 Turns...

Page 55: ...orts used as 10GbE ports 8 10GbE ports Total 64 x 10GbE ports 12 x 40GbE ports split to 4 10GbE ports 48 10GbE ports 4 x 40GbE ports split to 2 10GbE ports 8 10GbE ports 12 x 40GbE ports blocked 00 10...

Page 56: ...n Mellanox support web 5 1 Upgrading Software Software and firmware updates are available from the Mellanox Support website Check that your current revision is the same one that is on the Mellanox web...

Page 57: ...ck that the power cable is plugged into a work ing outlet 1 Check that the power cable has a voltage within the range of 100 240 volts AC 2 Remove and reinstall the power cable Issue 3 The green LED f...

Page 58: ...he image to upload Only use image 1 or image 2 5 Select the image to boot U Boot 2009 01 mlnx1 4 date CPU AMCC PowerPC 460EX Rev A at 1000 MHz PLB 200 OPB 100 EBC 100 MHz Security Kasumi support Boots...

Page 59: ...SerDes Speeds 10 20 40 or 56 Gb s per port 10 GbE or 40GbE Connector Types QSFP Can use QSFP to SFP adapter modules Air Flow Heat dissipation Air Flow 62 CFM Heat dissipa tion Maximum 798 BTUs hr Pow...

Page 60: ...y Compliance Safety EMC Emissions Safety US Canada cTUVus EU IEC60950 International CB Russia GOST R Argentina S mark DoC EMC Emissions USA FCC Class A Canada ICES Class A EU EN55022 Class A EU EN5502...

Page 61: ...itchX 36 Port QSFP InfiniBand Ethernet Gateway System Hardware User Manual Rev 1 0 Mellanox Technologies 61 EMC statements are also in the Regulatory and Compliance Guide A 3 China CCC Warning Stateme...

Page 62: ...ill be issued by the MLNX OS management SW indicating to system administration that the switch has crossed the Warning threshold Note that this temperature threshold does not require nor lead to any a...

Page 63: ...lL Module Select 9 ResetL Module Reset 10 Vcc Rx 3 3 V Power supply receiver 11 SCL 2 wire serial interface clock 12 SDA 2 wire serial interface data 13 GND Ground 14 Rx3p Receiver Non Inverted Data O...

Page 64: ...8 3 3 3 3 3 3 3 3 RX4p GND GND GND RX4n RX2p RX2n TX1n GND TX1p GND TX3n TX3p GND IntL ModPrsL VccTx Vcc1 LPMode 18 35 8 50 Top ResetL SDA SCL ModSelL RX3p TX2n GND VccRx 1 2 3 4 5 6 7 8 9 1 0 1 2 3...

Page 65: ...in 1 Pin 8 Looking into the Socket CONSOLE TXD TXD RXD GND GND RXD GND GND RJ45_SDA GND GND GND GND RJ45_SCL GND GND J34 Pin 1 Pin 8 Looking into the Socket I2C Table 21 RJ 45 CONSOLE Pinout Connectio...

Page 66: ...232 2M cable DB9 to RJ 45 for managed switches only HAR000028 Fan module with power supply side to connector side air flow MSX60 FF Fan module with connector side to power supply side air flow MSX60 F...

Page 67: ...n the Mellanox Web site at mellanox com for proper instructions to disassemble and dispose of the Switch according to the WEEE directive According to the WEEE Directive 2002 96 EC all waste electrical...

Page 68: ...doit tre d plac avec un syst me de levage m ca nique pour viter les blessures Cette unit doit tre install e dans un emplacement acc s limit Danger d lectrocution Lorsque le module de ventilation est r...

Page 69: ...retir ou mal fix Ce syst me dispose de fusibles phase neutre D branchez tous les cordons d alimenta tion avant d ouvrir le capot ou de toucher tout l ment l int rieur Risque et danger d lectrocution L...

Page 70: ...nal Electrical Code et du Code canadien de l lectricit Avertissement ne remplacer qu avec une batterie UL certifi e pour accepter un cou rant de charge anormal maximal sup rieur ou gal 4 mA Si la batt...

Page 71: ...entre de t l communications ou une salle informatique d di e Cet appareil doit tre install conform ment la version la plus r cente des codes lec trique nationaux En Am rique du Nord l quipement doit t...

Page 72: ...sont destin s une installation permanente l emplacement final pas pour l entretien ni la maintenance Selon la Directive 2002 96 CE DEEE tous les d chets d quipements lectriques et lectroniques EEE doi...

Page 73: ...verschoben werden um Verletzungen zu vermeiden Diese Ger t ist f r die Installation in einem Bereich mit beschr nktem Zugang vorge sehen Stromschlagrisiko Bei abgenommenem Ventilatormodul sind die Str...

Page 74: ...tromschlags und Stomgefahr Alle Stromversorgungseinheiten sind unabh ngig Trennen Sie alle Stomversorgungen um einen abgeschalteten Zustand im Inneren der Switch Plattform sicherzustellen Arbeiten Sie...

Page 75: ...ng Nur durch von UL anerkannten Akkus ersetzen die f r maximalen anor malen Ladestrom von nicht weniger als 4mA zertifiziert sind Es besteht Explosiionsgefahr wenn der Akku durch einen Akku eines fals...

Page 76: ...g Nutzung von Steuerungen oder Einstellungen oder Ausf hrung von Proze duren die hier nicht spezifiziert sind kann zu gef hrlichem Strahlenkontakt f hren Klasse 1 Laserprodukt und Referenzen zu den ak...

Page 77: ...s muy pesado se debe mover nicamente mediante un elevador mec nico para evitar lesiones Esta unidad ha sido ideada para instalar en lugares de acceso restringido Riesgo de descarga el ctrica Con el m...

Page 78: ...ables de corriente antes de abrir la tapa de este producto o tocar cualquier componente interno Riesgo de descarga el ctrica y peligro de corriente Todas las fuentes de alimentaci n son independientes...

Page 79: ...do con las disposiciones vigentes del C digo El ctrico Nacional de los EE UU y del C digo El ctrico de Canad II y a danger d explosion s il y a remplacement incorrect de la batterie Remplacer uniqueme...

Page 80: ...nstalar en un rea donde haya conexi n equipotencial como por ejemplo un centro de telecomunicaciones o una sala de computadoras dedicada Este dispositivo se debe instalar conforme a la versi n m s rec...

Page 81: ...La finalidad de los rieles no es deslizar la unidad hacia afuera del bastidor Sirven solo para la instalaci n permanente en el lugar de destino final no para fines de servicio o mantenimiento Conform...

Page 82: ...Rev 1 0 Mellanox Technologies 82 Appendix J Russian 1 2 3 4 5 6 45 C 113 F 8 3 40 lbs 18 kgs 40 70 lbs 18 32 kgs 70 121 lbs 32 55 kgs 121 lbs 55 kgs...

Page 83: ...SwitchX 36 Port QSFP InfiniBand Ethernet Gateway System Hardware User Manual Rev 1 0 Mellanox Technologies 83 7 8 9 10 11 12 InfiniBand 13 InfiniBand...

Page 84: ...Rev 1 0 Mellanox Technologies 84 14 15 16 17 18 19 UL CSA UL 4 UL CSA 3 16 AWG 1 5 6 4 5 125 13 HAR 3 1 0 300 250 10...

Page 85: ...rt QSFP InfiniBand Ethernet Gateway System Hardware User Manual Rev 1 0 Mellanox Technologies 85 20 21 22 23 24 25 RS232 Ethernet DP 1 DP 2 UL 20 1 IEC 60 825 1 1993 A1 1997 A2 2001 EN 60825 1 1994 A1...

Page 86: ...Rev 1 0 Mellanox Technologies 86 26 27 WEEE 2002 96 EC WEEE...

Page 87: ...e foarte greu i trebuie s fie mutat folosind un dispozitiv meca nic de ridicare pentru a evita producerea de leziuni Aceast unitate este destinat instal rii ntr un loc cu acces limitat Risc de oc elec...

Page 88: ...e prev zut cu siguran fuzibil bipolar neutr ndep rta i toate cor doanele de alimentare nainte de a deschide capacul acestui produs sau nainte de a atinge orice componente interne Risc de oc electric i...

Page 89: ...dul electric na ional al SUA i Codul electric canadian Avertisment nlocui i numai cu o baterie recunoscut UL certificat pentru curent de nc rcare anormal maxim de minimum 4 mA Exist risc de explozie n...

Page 90: ...amer a computerelor dedicat Acest dispozitiv trebuie s fie instalat n conformitate cu ultima versiune a codurilor electrice na ionale ale rii n cauz Pentru America de Nord echipamentul trebuie s fie i...

Page 91: ...ologies 91 27 Directiva DEEE n conformitate cu Directiva DEEE 2002 96 CE toate de eurile de echipamente elec trice i electronice EEE trebuie colectate separat i nu trebuie eliminate mpreun cu de euril...

Page 92: ...Rev 1 0 Mellanox Technologies 92 Appendix L Chinese 1 2 3 4 5 6 45 C 113 F 8 3 40 lbs 18 kgs 40 70 lbs 18 32 kgs 70 121 lbs 32 55 kgs 121 lbs 55 kgs...

Page 93: ...SwitchX 36 Port QSFP InfiniBand Ethernet Gateway System Hardware User Manual Rev 1 0 Mellanox Technologies 93 7 8 9 10 11 12 InfiniBand 13 PSU InfiniBand...

Page 94: ...Rev 1 0 Mellanox Technologies 94 14 15 16 17 18 19 UL CSA 20 UL 4mA UL CSA 16 AWG 125 V 13 A 1 5 4 5 HAR 1 0 300 V PVC 250 V 10 A...

Page 95: ...P InfiniBand Ethernet Gateway System Hardware User Manual Rev 1 0 Mellanox Technologies 95 21 GND 22 23 24 25 26 GND RS232 UL DP 1 DP 2 LPS 20 A CLASS 1 IEC 60 825 1 1993 A1 1997 A2 2001 EN 60825 1 19...

Page 96: ...Rev 1 0 Mellanox Technologies 96 27 WEEE 28 WEEE 2002 96 EC EEE TN IT...

Search results

Summary of Contents for InfiniBand X

Reviews:

Related manuals for InfiniBand X

Brands by name

Popular brands

Mellanox Technologies InfiniBand X, User Manual

Search results

Summary of Contents for InfiniBand X

Reviews:

Related manuals for InfiniBand X

Brands by name

Popular brands