manualshive.com logo in svg

Voltaire ISR 9024, User Manual, Page: 1 / 363

Share
Download
Voltaire ISR 9024 User Manual Download Page 1

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Voltaire Switch User Manual 

ISR 9024, ISR 9096, and ISR 9288 Switches 

Business Headquarters 

 

Voltaire Inc.

 

6 Fortune Drive, Suite 301

 

Billerica, MA USA 01821

 

Tel: 978-439-5400

 

Fax: 978-439-5401 

 

Israel Office 

 

Voltaire Ltd. 

9 Hamenofim St. 

Bldg. A Herzeliya 

46725, Israel 

Tel: +972 (9) 971-7666 

Fax: +972 (9) 971-7660 

Release AAA-BAA 
May 2006 
P/N 399Z00038 

Summary of Contents for ISR 9024

Page 1: ...Business Headquarters Voltaire Inc 6 Fortune Drive Suite 301 Billerica MA USA 01821 Tel 978 439 5400 Fax 978 439 5401 Israel Office Voltaire Ltd 9 Hamenofim St Bldg A Herzeliya 46725 Israel Tel 972 9...

Page 2: ...mation in this manual is furnished for informational use only is subject to change without notice and should not be construed as a commitment by Voltaire Voltaire Ltd assumes no responsibility or liab...

Page 3: ...through our Web Request Form by using the following link http www voltaire com support html Contact Us Please send your documentation related comments and feedback or report mistakes to docs Voltaire...

Page 4: ...formation specific instructions commentary sidelights or interesting points of information IMPORTANT Text set off in this manner indicates important information regarding a specific feature CAUTION Te...

Page 5: ...aintenance functions for the Switch and provides operation instructions Chapter 7 Names Settings This section provides instructions on how to set names to Switches and unmanaged ISR 9024 Chapter 8 Rac...

Page 6: ...details the CLI command menus The CLI provides configuration and management functions for Voltaire switches as well as a wide range of provisioning configuration and maintenance functions for the Infi...

Page 7: ...x Rack Grouping Creating renaming deleting rack groups and viewing rack information via the GUI and the CLI groups csv file The Voltaire switch software supports a new hardware module sLB 8 12 Diagnos...

Page 8: ...2 2 3 CLI Usage Tips 2 3 2 4 Keyboard Usage in CLI Operation 2 3 2 5 CLI Command Structure 2 5 2 6 CLI Case Sensitivity 2 5 2 7 CLI Command Prompts 2 6 Chapter 3 Initial System Setup 3 1 3 1 Prerequi...

Page 9: ...tch Information 5 16 5 6 Viewing Module Information 5 27 5 7 Using the Voltaire Device Manager for Switch Management 5 40 5 8 Reset Operations 5 45 Chapter 6 Voltaire Fabric Manager VFM 6 1 6 1 Fabric...

Page 10: ...nfiguring Routing via the VFM 10 4 10 6 Configuring Routing via the CLI 10 7 Chapter 11 Configuring Partitioning 11 1 11 1 Partitioning Overview 11 2 11 2 Mechanisms Used to Create and Enforce Partiti...

Page 11: ...ntents iv P N 399Z00038 2006 Voltaire Inc Appendix A CLI Commands Reference A 1 Appendix B Port Counters Fields B 1 Appendix C Supported Traps and Events C 1 Appendix D Glossary D 1 Appendix E Index E...

Page 12: ...log Box 5 4 Figure 5 3 Exit Warning Message 5 6 Figure 5 4 ISR 9288 Voltaire Device Manager Main Window 5 7 Figure 5 5 ISR 9096 Voltaire Device Manager Main Window 5 8 Figure 5 6 Device Tree 5 10 Figu...

Page 13: ...Figure 5 29 Router Module Information 5 39 Figure 5 30 IP Settings 5 40 Figure 5 31 Remote Manager Settings SNMP Tab 5 41 Figure 5 32 Remote Manager Settings General tab 5 42 Figure 5 33 Configuring U...

Page 14: ...Figure 6 22 Link Properties 6 31 Figure 6 23 List of Racks 6 31 Figure 6 24 Racks Components 6 32 Figure 6 25 Hosts List 6 33 Figure 6 26 Hosts Ports 6 34 Figure 6 27 List of Switches 6 35 Figure 6 2...

Page 15: ...Group Properties 6 63 Figure 6 52 Creating a New Rack 6 63 Figure 6 53 Current Alarms of a Single Node 6 64 Figure 7 1 Node Properties 7 3 Figure 8 1 Rack Groups VFM 8 3 Figure 8 2 Rack Grouping Topo...

Page 16: ...ia the VDM 12 4 Figure 13 1 Cluster Monitor Dialog Box 13 2 Figure 13 2 Ganglia Cluster Monitor 13 3 Figure 14 1 Diagnostic Block Diagram 14 3 Figure 14 2 PM Settings 14 9 Figure 14 3 Port Counters In...

Page 17: ...ief introduction to InfiniBand technology and describes Voltaire ISR 9024 ISR 9096 and ISR 9288 switches The following information is included in these pages About InfiniBand on page 1 2 ISR 9024 Syst...

Page 18: ...manageability InfiniBand enabled the introduction of data clusters an invaluable next step to data center construction In effect InfiniBand is a fabric based switched network that allows devices such...

Page 19: ...discovery failures and resource allocation 1 2 ISR 9024 System Overview The Voltaire ISR 9024 Grid Switch is a high performance low latency fully non blocking switch for high performance computing HPC...

Page 20: ...96 or 288 4X 12X ports in a single chassis multiple ISR 9288 9096 can also be interconnected to form very large clusters Possible configurations range from dozens to thousands of nodes with many upgr...

Page 21: ...es providing seamless connectivity between server clusters FC SANs NAS appliances IP SANs and TCP IP networks LANs For further flexibility InfiniBand 4X 10 Gbps and 12X 30Gbps ports InfiniBand to TCP...

Page 22: ...h 8 12X IB ports The ISR 9288 can host up to 12 sLB 24 or sLB 8 12 or any combination thereof The ISR 9096 can host up to 4 sLB 24 or sLB 8 12 or any combination thereof Management Board sMB One or tw...

Page 23: ...ementing fat tree topology when the system is fully populated sLBs are defined as Line Boards and sFBs as Fabric Boards Each Line Board chip element is connected to each element of the Fabric Board th...

Page 24: ...e ISR 9288 InfiniBand internal switching matrix Figure 1 4 ISR 9288 InfiniScale Switching Topology Fat Tree Topology The ISR 9288 features six symmetrical Line Boards sLB 24 at both ends and four inte...

Page 25: ...a 12X port Each 4X port has an orange and a green LED When a group of three 4X ports is operating as a 12X port the LEDs of the lowest numbered 4X are orange and green and function in the same manner...

Page 26: ...et SSH session Voltaire Device Manager described in Chapter 5 of this manual Provides management of a switch with an advanced Graphic User Interface GUI The Device Manager can be connected via HTTP se...

Page 27: ...ted directly through a network through the local Ethernet subnet in order to manage the Switch via the sMB 1 4 2 In Band Management Figure 1 7 illustrates in band management which is performed via the...

Page 28: ...IPs so the management station will always use the same IP address for system management Management boards redundancy is based on keep alive signals that are being sent from the active board to the pas...

Page 29: ...remote FTP server for downloading new switch software versions and for uploading backup files FTP configuration settings are made using the CLI and are detailed on as detailed in Appendix A FTP Mode S...

Page 30: ...e on the active sMB the IP address of the IB interface is moved to the backup sMB LOCAL The Local interface is an optional IP address which can be attached to the Ethernet interface of the sMB module...

Page 31: ...nagement functions This chapter contains the following sections Interfaces for CLI Connection on page 2 2 CLI Command Modes on page 2 2 CLI Usage Tips on page 2 3 Keyboard Usage in CLI Operation on pa...

Page 32: ...mode Allows view only access of switch configuration parameters Privileged mode Allows the user to configure basic parameters such as date time and reload reset switch and to perform software and fir...

Page 33: ...all of the options available at that point in the command line along with a brief description of the command If you enter a question mark before a partial command use a space after the mark the syste...

Page 34: ...er Ctrl D Deletes the character at the cursor Ctrl E Moves the cursor to the end of the command line Ctrl F Moves the cursor forward one character Ctrl K Deletes all characters from the cursor to the...

Page 35: ...e only the first word of the command You can also type only a part of the command s first word Example ISR9288 04eb ftp ftp show ftp show To display help for commands starting with the same letters ty...

Page 36: ...e Configuration changes are automatically saved as they are made The or sign respectively show if you are in Exec or Privileged configuration mode as shown in the Table 2 3 Table 2 3 Command Prompts C...

Page 37: ...t Time Configuration on page 3 2 Collecting Required Information on page 3 2 Connecting a Console on page 3 3 Starting a CLI Session on page 3 3 Configuring the Management Interface IP Address on page...

Page 38: ...e switch installation checklist Table 3 1 to record the system and network information required for first time configuration The following information is required Passwords A password for users who wi...

Page 39: ...trol None 3 5 Starting a CLI Session To start a CLI management session via serial connection to the switch Step 1 Connect the PC to the switch via its serial port using the cable supplied by Voltaire...

Page 40: ...m the Privileged mode Type f the password when prompted Step 3 interface fast Enters Fast interface configuration Step 4 ip address fast set 192 168 70 60 255 255 255 0 Note The above IP address and m...

Page 41: ...3 7 Configuring the Management IP Address in ISR 9288 9096 9024 Switches The GUI must first be enabled via the CLI The CLI does not require initial configuration The active management interface of th...

Page 42: ...ove specifies that the Fast interface is the active management interface To set the Fast interface as the active management interface ISR 9288 9096 9024 config if fast management interface set 3 7 2 C...

Page 43: ...ernet Management port to the same physical network when using Ethernet as active management 3 8 1 Management over the FAST ETH Interface When the system is configured to manage a chassis with I O modu...

Page 44: ...ess set command The configured modules can be viewed using CLI io show command Use the table below to enter the appropriate commands To set the I O address Use the following table to enter the appropr...

Page 45: ...e time stamped Command Description Step 1 enable password Enter Privileged mode from Exec mode the default password is voltaire Step 2 clock set 072805202003 Enter time and date in military format For...

Page 46: ...aire OK 3 11 Verifying Configuration From the Exec Privileged mode verify the system parameters using the steps below Command Description Step 1 version show Displays the version of the current switch...

Page 47: ...rs using CLI commands including software upgrades and maintenance and contains information on the following topics Prerequisites on page 4 2 Updating Software on page 4 2 Prerequisites for Updating So...

Page 48: ...operations The switch is designed to run on a continuous basis without significant maintenance However from time to time you may need to install updated software or modify device configuration The swi...

Page 49: ...ion files to the FTP directory II Configure FTP Server Settings Identify the location from which to retrieve the updated software If the default download location is not appropriate you can reset it I...

Page 50: ...r sLB and per sFB 4 4 Backing up the Switch Configuration It is recommended that you back up the switch configuration settings and topology file periodically The switch configuration settings can be r...

Page 51: ...ed to the CPU units on the sMB modules These two sFB 12 modules are at the very left and very right of the chassis slots sFB1 and sFB4 4 5 2 Upgrade Process I Upgrade Procedure with a Single sMB Insta...

Page 52: ...ng steps To upgrade the ISR 9288 9096 with two sMBs installed Step 1 Make sure only one sMB is installed in the chassis If two sMBs are installed remove the second sMB Step 2 Follow the steps describe...

Page 53: ...formation on the following topics Device Manager Description on page 5 2 Getting Started with the Device Manager on page 5 3 The Device Manager Main Window on page 5 6 Voltaire Device Manager Operatin...

Page 54: ...th a web based GUI that provides InfiniBand management functionality for Voltaire ISR 9096 and ISR 9288 and ISR 9024 switches It enables users to view a graphical representation of the modules its LED...

Page 55: ...the application Step 1 Set the Fast Ethernet IP address using the CLI through a serial console connection as described in Chapter 3 Step 2 Open the Internet Browser and connect to the fast Ethernet IP...

Page 56: ...use the Voltaire Vision GUI NOTE When opening the Voltaire Device Manager Initiate window for the first time click to start the Sun Java Web Start installation The computer that is being used must be...

Page 57: ...ll configuration and provisioning operations Default password 123456 Manager user type Enables you to set the parameters Default password voltaire Guest user type This user can only view device config...

Page 58: ...message is displayed Figure 5 3 Exit Warning Message Step 2 Click Yes to confirm 5 3 The Device Manager Main Window This section describes the Main window of the ISR 9096 ISR 9288 and the ISR 9024 Vol...

Page 59: ...038 2006 Voltaire Inc 5 3 1 ISR 9288 Main Window Figure 5 4 shows the ISR 9288 Main window Figure 5 4 ISR 9288 Voltaire Device Manager Main Window Information Area Trap Table ISR 9288 Device Tree Devi...

Page 60: ...e Device Manager Main Window 5 8 P N 399Z00038 2006 Voltaire Inc 5 3 2 ISR 9096 Main Window Figure 5 5 shows the ISR 9096 Main window Figure 5 5 ISR 9096 Voltaire Device Manager Main Window ISR 9096 D...

Page 61: ...Ethernet IP and routing mode settings Remote Management Settings Displays and sets the SNMP Manager s IP addresses and general remote management setting details Users Configures user authorization pas...

Page 62: ...hical format This includes sFB boards line boards sMB sPSU sFU sCTRL boards sRBDs IO drawers and installed router modules Figure 5 6 shows the Device tree The ISR 9288 was used here as an example Figu...

Page 63: ...to the ISR 9024 Device Displays device data in the Information Area CPU Displays CPU data in the Information Area SM Displays local SM data in the Information Area Temp Displays the temperature readin...

Page 64: ...w This does not apply to the ISR 9024 NOTE When two Management boards sMB are installed in failover configuration the active sMB board is displayed with a border line around it as shown on the sMB at...

Page 65: ...399Z00038 2006 Voltaire Inc ISR 9288 Switch Figure 5 9 and Figure 5 10 show the graphical representations of the ISR 9288 switch Figure 5 9 ISR 9288 Switch Graphical Representation Front View Click t...

Page 66: ...r Main Window 5 14 P N 399Z00038 2006 Voltaire Inc Figure 5 10 ISR 9288 Switch Graphical Representation Rear View ISR 9024 Figure 5 11 shows the graphical representation of the ISR 9024 switch Figure...

Page 67: ...nformation Area is a read only area that displays information based on user selection both in the main and advanced windows When the Device Manager is connected to ISR 9288 9096 switches the Informati...

Page 68: ...olumns by dragging the column header to a new location 5 4 3 Instructional Tool Tips You can obtain helpful information Voltaire Device Manager operation by positioning the mouse cursor in various loc...

Page 69: ...mation area Figure 5 12 Device Information Table 5 5 describes the parameters displayed for the device Table 5 5 Device Information Parameters Parameter Description Device Name Displays a text descrip...

Page 70: ...ion number Device ID Displays the identification number of the device Contact Displays the user defined contact person assigned to this device Location Displays the user defined location assigned to t...

Page 71: ...99Z00038 2006 Voltaire Inc 5 5 2 Port Information To view port information Step 1 In the Main Window Switch graphical representation view click on a port The following information is displayed in the...

Page 72: ...is attempting to get a link following a problem such as a short cabled pin Link Width Displays the width data of the link Link Width Supported Displays the range of supported width data of the link L...

Page 73: ...PU information Step 1 On the Device toolbar click CPU icon The following is displayed in the Information Area Figure 5 14 CPU Information For ISR 9288 9096 switches the status bar displays the relevan...

Page 74: ...tion Parameters Parameter Description User Time The percentage of time of the total usage time that the CPU is used System Time The percentage of time of the total usage time the system spends using t...

Page 75: ...embedded temperature sensor On the device toolbar click Temp icon The following dialog box is displayed in the main window Information area Figure 5 15 Temperature Readings All fields in this dialog...

Page 76: ...he sCTRL and the sFU 8 front panels The fan speed also increases when one of the sFUs is not present or when one of the sFU internal fans is faulty On the toolbar the Temp the Device and the Fan icons...

Page 77: ...port where the SM resides Has SM Indicates if the SM is present in the switch SM Mode If an SM is present in the switch displays whether the SM is enabled or disabled 5 5 6 Obtaining Detailed Module I...

Page 78: ...meters displayed in each field Table 5 9 Detailed Module Window Fields Parameter Description Module Name More textual information regarding the module Port Number The part number if applicable Hardwar...

Page 79: ...e view of the Voltaire Device Manager main window The LEDs displayed in the graphical information views reflect the actual state of the LEDs on the module For a description of the ISR 9288 9096 module...

Page 80: ...The sLB Information Window appears Figure 5 18 sLB Information sLB Information Window Description The sLB 24 graphical display of the Line Board at the top of the window shows which ports are active...

Page 81: ...itch is operational The color coding of these fields reflects the status of the port as follows Green indicates that the port is in active state Red indicates that the port is either in Initialized or...

Page 82: ...or optical adapters on InfiniBand ports When installed on sLB ports optional Optical Adapter have a distinctive graphical representation in the VDM to differentiate them from the IB connector as shown...

Page 83: ...9288 and ISR 9096 Fabric Information Windows Figure 5 20 ISR 9288 Fabric Board Information Figure 5 21 ISR 9096 Fabric Board Information sFB Information Window Description Each LED corresponds to a g...

Page 84: ...ard provides two RS 232 ports one I2C port for debug purposes by authorized personnel only the second port CLI can be used to establish a CLI connection locally to the chassis To view the Management B...

Page 85: ...on Window Fields Parameter Description Module Name More textual information regarding the module Part Number The part number if applicable Hardware Version The hardware version of the module if applic...

Page 86: ...information Click sCTRL on the Device Tree The sCTRL Information Window appears shown in Figure 5 24 Figure 5 24 sCTRL Information sCTRL Information Window Description The sCTRL graphical display show...

Page 87: ...appears as shown in Figure 5 25 Figure 5 25 sPSU Information sPSU Information Description The green line around the power connector indicates that the power supply is connected to a source of current...

Page 88: ...layed Figure 5 26 ISR 9288 Fan Information Figure 5 27 ISR 9096 Fan Information sFU Information Description Green indicates normal operation Should a specific fan unit be faulty the fan unit is displa...

Page 89: ...sets the fan speed to turbo After reboot the sMB scans the temperature sensors of the chassis modules including all sFB modules all sLB modules sMB sFU 8 and sFU 4 modules If all the temperature sens...

Page 90: ...nd before the temperature LED turns off Whenever a temperature of a specific module goes beyond the WARNING threshold the system generates a trap reporting the high temperature on the problematic modu...

Page 91: ...Information Window appears as shown in Figure 5 29 Figure 5 29 Router Module Information The board s VPD Vital Product Data is also displayed For instructions on how to manage the router modules via t...

Page 92: ...ettings The IP Settings dialog box enables you to view the routing mode of your device To View Configured IP Settings Step 1 Select Device IP Settings from the Main Menu The IP Settings dialog box ope...

Page 93: ...management system To Configure Remote Management Settings Step 1 Select Device Remote Management Settings from the Main Menu The Remote Management Settings dialog box opens as displayed in Figure 5 3...

Page 94: ...ral tab Step 5 Fill in the following fields Device Name Describes the device name in a read only field SNMP Version Displays the software version of the SNMP in a read only field Contact Details of th...

Page 95: ...t the parameters The default password is voltaire The Guest level enables you to view configurations only The default password is voltaire To Configure Authorization Passwords Step 1 Select Device Use...

Page 96: ...itory file To Configure FTP Settings Step 1 Select Device Config FTP from the Main Menu The Config FTP window opens as displayed in Figure 5 34 Figure 5 34 Configuring FTP Step 2 To configure the FTP...

Page 97: ...esets a Line or Fabric Board in ISR 9288 9096 switches only Reset the CPU software reset available in all switches To reset the entire switch NOTE This option is only relevant for ISR 9288 9096 switch...

Page 98: ...switches Right click on a Fabric Board front panel of device view or a Line Board rear panel of device view and select Reset sFB or Reset sLB CAUTION This is a traffic affecting operation To reset th...

Page 99: ...Fabric Manager VFM application It includes information on the following topics Fabric Manager Description on page 6 2 Fabric Manager Installation on page 6 3 Getting Started with the Fabric Manager on...

Page 100: ...Fabric Manager is available for the following hardware platforms ISR 9096 switch system ISR 9288 switch system ISR 9024 switch system 6 1 1 Voltaire Fabric Manager Components The Voltaire Fabric Manag...

Page 101: ...ng of fabric elements e g Node info Port info Port counters and performance Provides advanced Performance Management PM of ports in the fabric with Quality of Service QoS functionality including SL to...

Page 102: ...the Sun Java Web Start Application on a Linux Workstation Installing the Sun Java Web Start Application on a Linux Workstation Step 1 Execute the file with the command jre 1_5_0_05 linux platform arch...

Page 103: ...t interface using the CLI through a serial console connection Refer to page 3 4 for instructions on these configuration settings Step 2 Open the Internet Browser and connect to the fast Ethernet IP ad...

Page 104: ...When opening the Voltaire Fabric Manager Initiate Window for the first time click to start the Sun Java Web Start installation The computer that is being used must be connected to the Internet for suc...

Page 105: ...s Default password voltaire Guest user type This user can only view device configuration and status information Default password voltaire Step 2 Enter the Password and click OK The Voltaire Fabric Man...

Page 106: ...nctions and provides a view of the topology managed by the system with color coded visual indicators that display alarms and important events in the managed fabric Figure 6 4 shows the main areas in t...

Page 107: ...iled port or board and a Critical event Color coding is defined according to severity For more information refer to The Event Notification Mechanism Section 14 1 1 6 4 2 Finding the Source of Alarms i...

Page 108: ...tion on use of other advanced fabric diagnostic capabilities The Configuration Config Mode Appendix A 4 3 details how to manage Events and Alarms via the CLI 6 4 3 Topology Map The Topology Map provid...

Page 109: ...displayed it indicates that the device is a Voltaire device this can either be a device manufactured by Voltaire such as an ISR 9288 switch for example or a device with the Vendor ID Voltaire set in t...

Page 110: ...ink speed are displayed Moving Icons on the Map The location of the device icons can be rearranged in the Topology Map To move a device icon click and drag the icon to the desired location The icon lo...

Page 111: ...oltaire Fabric Manager In general when a menu heading is not available disabled it appears grayed out Table 6 4 details the Main Menu options Table 6 4 Main Menu Options Menu Submenu Action File Exit...

Page 112: ...information on routing algorithm Current Alarms Opens the Current Alarms window which lists the open alarms in the fabric This window is described in Section 6 7 6 and 0 Config FTP Opens the Config FT...

Page 113: ...d set device parameters See Chapter 2 for instructions on CLI usage Properties Opens the Node Properties window which enables you to change the node name view the node properties GUID Vendor Device ID...

Page 114: ...d in Section 14 3 1 Attached to Displays the nodes which the port is connected to at both ends of the link Help About Displays the Voltaire Fabric Manager software version 6 4 5 Right Click Context Se...

Page 115: ...ic Manager Main Window 6 17 P N 399Z00038 2006 Voltaire Inc Figure 6 9 Right Click a Port Information Area Right Panel Figure 6 10 Right Click a Port Table View Figure 6 11 Right Click an Event Figure...

Page 116: ...nc Figure 6 13 Right Click Racks 6 4 6 Subnet Tree The Subnet Tree displays all the elements monitored by the Fabric Manager in a top down hierarchical format Figure 6 14 Subnet Tree The IP address di...

Page 117: ...st Information window described in Section 6 6 2 Switches Logical object which represents all the switches in the fabric Click on this object to display the Switches Lists described in Section 6 6 3 E...

Page 118: ...mmediately displayed underneath the parent and the icon is displayed to the left of the object icon Click once on the icon to collapse the parent object hide the children 6 4 7 Toolbar The toolbar is...

Page 119: ...on Pan Interactor Allows you to drag and drop the icons in the topology map Select Allows you to select an object in the Topology Map Zoom in area Enlarges the objects when centering selected area Zoo...

Page 120: ...15 Topology Table View Table 6 8 describes the parameters displayed in the Topology table view Table 6 8 Topology Table View Fields Field Description State Indicates the alarm state of the node host...

Page 121: ...ding Use Ctrl click for a secondary sort order Filtering the Table View You can filter the Topology Table to only display nodes with a certain alarm state according to the following filters Normal War...

Page 122: ...event alarm occurrence Note events originating in the fabric from a node or host are time stamped with the date time of the node or host clock which could differ from the PC time Events originating i...

Page 123: ...ote NOTE This feature is only supported for certain events when a specific type of event does not support this feature the View Event option is grayed out in the menu Right Click the event listing in...

Page 124: ...the link or node with which the event is associated Table 6 10 Events Data Fields Parameter Description Node GUID Displays the Global Unique Identification number GUID of the device Platform The swit...

Page 125: ...ring Events Clearing an event means that you have taken the steps to correct the problem and now wish to remove the event from the Trap Table Select Clear Event the event is cleared Select Clear All a...

Page 126: ...ties The following information is displayed in the lower part of the Information panel when selecting a node host or switch in the main window Figure 6 20 Node Properties Table 6 11 describes the para...

Page 127: ...the node type and model Node GUID Displays the Global Unique Identification number GUID of the device Node Name Displays the user assigned name of the node Located Displays the location of the node in...

Page 128: ...he LID Mask Control value of the port Possible values are 0 to 7 GID Prefix The GID is a concatenation of a 64 bit subnet prefix an identifier of the subnet and a manufacturer assigned globally unique...

Page 129: ...the number of connections between devices a link with more than one connection is depicted as a thick line on the topology map Link Width Displays the connection width 6 6 Information Windows The Volt...

Page 130: ...ts nodes contained in the specific Rack Option Description Name Displays the Name allocated to this Rack by the user Location Displays the rack location as defined by the user Number of Nodes Displays...

Page 131: ...ormation parameters Table 6 14 Hosts List Information Parameters Parameter Description GUID Displays the Global Unique Identification number GUID of the device Name Displays the user assigned device n...

Page 132: ...igure 6 26 Hosts Ports Table 6 15 details the Host Ports information parameters Table 6 15 Host Ports Information Parameters Parameter Description Port Displays the port number according to the number...

Page 133: ...escription GUID Displays the Global Unique Identification number GUID of the switch Name Displays the user assigned switch name The switch name is either set by the ATS mechanism or can be set manuall...

Page 134: ...Parameters Option Description Port Displays the port number according to the numbering scheme of the switch or line board State Displays the state of the port which can be one of the following ACT po...

Page 135: ...nformation parameters Table 6 18 Port Information Parameters Parameter Description Port GUID Displays the Global Unique ID of the port The GUID is a unique 128 bit number that identifies the port GID...

Page 136: ...ocal ID of the Subnet Manager Active Link Width Displays the active width of the link 1X 4X or 12X Active Link Speed Displays the active current speed of the port Gbps Port MTU Displays the Maximum Tr...

Page 137: ...d in the main window Figure 6 30 Link Information This window displays information regarding the ports at both end of the link This includes the alarm status color coded according to the standard conv...

Page 138: ...s an example representing a chassis regrouping all its components The grouping information window allows you to view the information of the various modules in the chassis To display grouping informati...

Page 139: ...operties Window This section describes viewing and configuring SM parameters via the Voltaire Fabric Manager To view Subnet Manager general parameters Select Fabric SM Properties from the main menu Th...

Page 140: ...wer priority will take over Possible values are 0 15 SM State Read only field that displays the current SM state Possible values are Standby and Master SM Mode Defines the SM functionality mode Possib...

Page 141: ...Contact The configuration of Subnet Parameters can also be performed via the CLI Refer to the Subnet Manager SM Mode Appendix A 4 13 for more information 6 7 2 Fabric Inspect Features Fabric Inspect...

Page 142: ...ings from the main menu The following dialog box is displayed Figure 6 33 Event Filters Settings By default all events are reported You can Uncheck an event to filter Click Default to the return to th...

Page 143: ...ll devices in the fabric Scope field is set to All To change these settings select one of the following values from the Scope field Hosts Only PM data is only collected for HCAs in the fabric Switches...

Page 144: ...he Thresholds you don t want to have checked Step 4 Click Reset to reset the PM counters in all fabric devices Step 5 Click Apply to save PM configuration Note Data entered appear in red until clickin...

Page 145: ...2006 Voltaire Inc Error Log To show the Error Log Select Fabric Fabric Inspect Error Log from the VFM main menu The following window is displayed Figure 6 36 Error Log Example Refer to Section 14 3 3...

Page 146: ...ow is displayed Figure 6 37 Bad Ports Log Refer to Section 14 4 3 for further information on Identifying a Bad Port Fabric Statistics The VFM Fabric summary tools Fabric Fabric inspect Fabric statisti...

Page 147: ...portcounters csv file 6 7 3 Cluster Config Use the Cluster Config dialog box to set the path of the Ganglia Monitoring tool Step 1 Select Fabric Cluster Config from the main menu The Cluster Monitor d...

Page 148: ...taire Inc Example http 172 25 2 175 ganglia m load_one r hour s descending c Support Cluster h HOST sh 1 hc 4 In the example below the Host is called hydra3 voltaire com Figure 6 40 Ganglia Cluster Mo...

Page 149: ...llowing dialog box is displayed Figure 6 41 QoS SL to VL Mapping Table This dialog box is used to set the SL to VL mapping Step 3 Click Disable and the Subnet Manager will not configure the QoS parame...

Page 150: ...ical for both The following dialog box is displayed Figure 6 42 QoS VL High Arbitration Table Step 3 In the High Priority Limit field enter the number of high priority data that can be transmitted wit...

Page 151: ...ring the Fabric 6 53 P N 399Z00038 2006 Voltaire Inc 6 7 5 Routing Configuration To perform a routing configuration Step 1 Select Fabric Routing Configuration from the main menu The following dialog b...

Page 152: ...the desired routing algorithm from the Algorithm drop down menu Figure 6 44 Routing Algorithm Step 3 If you selected either the Rearrangeable or Up down algorithm then select a value from the Topolog...

Page 153: ...Routing dialog box The following dialog box is displayed Figure 6 45 Add to Spine List c Click OK to add selected spine and close the dialog box Step 5 Click Apply this saves your configuration Note...

Page 154: ...alarms present in the fabric It is used to perform a fabric diagnostic Refer to Chapter 14 Section 14 3 4 for further information on how to manage alarms To display the Current Alarm window Select Fab...

Page 155: ...which can be one of the following Critical Minor or Warning Alarm Displays a textual description of the event alarm type Source Displays the object where the event originated Date Time The date and t...

Page 156: ...umber Description A textual description of the alarm Alarms types are detailed in the Current Alarm Table in Appendix C on page C 2 Clearing Alarms Clearing an alarm means that you have taken the step...

Page 157: ...Server Define enter the IP of the FTP Server User Name Define enter your user name to access the FTP server Password Define enter your password to access the FTP server Remote path Enter the remote FT...

Page 158: ...ocal library This file can then be exported to the FTP using Fabric Config FTP Grouping import checks and updates the current configuration with the rack grouping defined in the group csv file importe...

Page 159: ...ng Nodes 6 8 1 Viewing Nodes From the Main Menu select Node Open The contents of the specified node appears OR From the Graphics display select the desired node right click and select Open OR From the...

Page 160: ...ping Info From the Main Menu select Node Grouping Info The grouping details of the specified switch appears OR From the Graphics display select the desired node ISR 9096 9288 switches only right click...

Page 161: ...s Figure 6 52 Creating a New Rack 6 8 4 Nodes Menu Cluster Monitor The browser window opens with the Ganglia link directed to the specific IP displaying a detailed host report Example http 172 25 2 17...

Page 162: ...en configuring routing through the Fabric Manager Routing Configuration To configure routing algorithm Refer to Chapter 10 for further information on Configuring Routing Algorithms 6 8 6 Nodes Menu Cu...

Page 163: ...Chapter This chapter provides instructions on how to set switch and node name settings via the CLI and the GUI It includes information on the following topics CLI Switch Name Settings on page 7 2 GUI...

Page 164: ...me which is the platform name only Default unique default name set Changes the unique default name option unique default name show Displays the unique default name option Example ISR9288 04c3 config n...

Page 165: ...s option the Node Properties window opens Step 2 Click the Manual radio button Step 3 In the Node Name field rename the Node as desired Step 4 Click OK to apply your changes and close the window Figur...

Page 166: ...N 399Z00038 2006 Voltaire Inc 7 3 Naming an Unmanaged ISR 9024 You can rename an unmanaged ISR 9024 from the GUI the exact same way it is performed for a regular switch See Section 7 2 for further de...

Page 167: ...how to perform rack grouping It includes information on the following topics Rack Grouping Overview on page 8 2 Rack Grouping Settings via the VFM on page 8 3 Elements of the Group csv File on page 8...

Page 168: ...ping can be configured either via the GUI or CLI The user can create groups in two ways by 1 Importing a file stored on an FTP server that describes the names and physical locations of the elements in...

Page 169: ...8 2 1 Creating Rack Groups Step 1 Choose the element s to be assigned to a rack group from one of the following screens topology map view host list switch list table view Several elements can be selec...

Page 170: ...Rack Grouping Rack Grouping Settings via the VFM 8 4 P N 399Z00038 2006 Voltaire Inc Figure 8 2 Rack Grouping Topology Map Figure 8 3 Rack Grouping Table View...

Page 171: ...VFM 8 5 P N 399Z00038 2006 Voltaire Inc Figure 8 4 Rack Grouping Switches Host Table The Node Properties dialog box is displayed Figure 8 5 Node Properties Dialog Box Step 3 In the Properties dialog b...

Page 172: ...ox Step 4 In the Name field set a Name for the Rack Step 5 Optional In the Location field set the Rack position to define the physical position of the rack according the user s needs Step 6 Click OK t...

Page 173: ...e Topology Map right click the desired Rack group that you wish to rename and select properties The Group Properties dialog box opens You can also access this by clicking Racks in the left panel and s...

Page 174: ...elected nodes are deleted Note Individual nodes can also be deleted from a Rack group by assign it to No Rack in the Properties Node Rack dialog box simply by manually deleting the name of the Rack Th...

Page 175: ...Figure 8 10 Rack Groups Table View 8 2 5 Showing Absent Elements Elements unconnected element can be viewed from the table view of the fabric The user can show the element s information or permanently...

Page 176: ...8 1 details the elements of the Group csv file Table 8 1 Group csv file Element Description Row Description Element_type Type of element RACK SYSTEM SWITCH SWITCH HCA ID unique key 0 or none for unkno...

Page 177: ...this field is 0 You may download the group csv and edit view it using any CSV editor e g Excel 8 3 1 Group csv Format The following example shows a single rack called DemoRack that contains 15 HCA ele...

Page 178: ...0 U HCA 8f10403965204 Nestor14 0 1 0 U HCA 8f10403965200 Nestor10 0 1 0 U HCA 8f104039651f4 Nestor8 0 1 0 U HCA 8f104039651e4 Nestor15 0 1 0 U HCA 8f104039651b4 Nestor12 0 1 0 U HCA 8f10403962b24 Nes...

Page 179: ...88 0967 config ftp server 172 25 5 36 ISR9288 0967 config ftp password 123 ISR9288 0967 config ftp username ftp ISR9288 0967 config ftp exportFile group Once the group csv file has been created AND ex...

Page 180: ...in Chapter 8 for the nodes connected in the previous step Repeat Steps 1 and 2 until all racks are connected and all elements are assigned to the appropriate group Step 3 Export the Elements of the g...

Page 181: ...In This Chapter This chapter provides information about how to configure the Quality of Service feature It includes information on the following topics QoS Overview on page 9 2 SL to VL Mapping on pa...

Page 182: ...L is VL15 All other VLs are for normal traffic Service Level SL is an indication of the service class of the packet SLs provide differentiated services improving fabric utilization and avoiding deadlo...

Page 183: ...ic Quality of Service from the main menu The following dialog box is displayed Figure 9 1 QoS SL to VL Mapping Table This dialog box is used to set the SL to VL mapping Step 3 Click Disable and the Su...

Page 184: ...ates the amount of high priority packets that can be transmitted without an opportunity to send a low priority packet The High Priority table consists of VL Weight pairs that deserve high bandwidth re...

Page 185: ...that only a single packet from the high priority table may be sent before an opportunity is given to the low priority table The High Priority list has a minimum length of one and a maximum of length...

Page 186: ...os sl vl set 1 1 Step 3 Set the High Arbitration table using the qos vl high arbitration set row virtual lane priority weight Example Switch config sm qos qos vl high arbitration set 1 1 5 Step 4 Set...

Page 187: ...values Switch Lifetime Limit SLL and Head of Queue Lifetime Limit HLL SLL is defined as 4 096us 2 LV if 0 LV 19 5 55 LV is the LifeTimeValue component of the SwitchInfo attribute If LV 19 then SLL is...

Page 188: ...S Example of SM Parameter Configuration 9 8 P N 399Z00038 2006 Voltaire Inc HCA SW to the default IPoIB multicast group MGID 0xFF12401BFEFF000000000000FFFFFFFF Refer to the CLI Subnet Manager SM Mode...

Page 189: ...to configure routing Algorithms It includes information on the following topics Supported Algorithms on page 10 2 Balanced Routing Scheme on page 10 2 Rearrangeable Routing Multi Path with sFB Consist...

Page 190: ...to better utilize network bandwidth the Subnet Manager balances its selections such that the traffic between all end ports is balanced over all possible paths in the fabric NOTE If the algorithm canno...

Page 191: ...sFB consistency rule 10 4 Up Down Routing Scheme Up Down routing is designed for CLOS networks The algorithm objective is to assure that given an N stage CLOS network each lid route between a couple o...

Page 192: ...via the VFM 10 4 P N 399Z00038 2006 Voltaire Inc 10 5 Configuring Routing via the VFM To configure routing via the Fabric Manager Step 1 Select Fabric Routing Configuration from the main menu The fol...

Page 193: ...Select the desired routing algorithm from the Algorithm drop down menu Figure 10 2 Routing Algorithm Step 3 If you selected either the Rearrangeable or Up down algorithm then select a value from the T...

Page 194: ...ick Add in the Routing dialog box The following dialog box is displayed Figure 10 3 Add to Spine List f Click OK to add selected spine and close the dialog box Step 5 Click Apply this saves your confi...

Page 195: ...m info topology set command as follows Command sm info topology set Description Sets the number of levels in a subnet configured as a Clos fat tree topology This is an odd number from 3 and up or 0 fo...

Page 196: ...tion set Step 5 Press enter the new routing configuration is activated NOTE When configuring spines in CLOS network routing configuration can also be performed via the CLI by adding each GUID of each...

Page 197: ...itioning In This Chapter This chapter provides information about how to perform Partitioning It includes information on the following topics Partitioning Overview on page 11 2 Mechanisms Used to Creat...

Page 198: ...iscarded Switches and routers may optionally be used to enforce partitioning Partitioning can be used to associate hosts or devices with common attributes as shown in the diagram below Figure 11 1 Par...

Page 199: ...B there is a well known MGID Pkey mapping 11 3 Setting Partitioning Using ISR 9288 9096 9024 CLI it is possible to configure fabric partitioning by setting the Partition Keys Pkey to a partition table...

Page 200: ...Step 1 Enable password Enter Privileged mode from Exec mode the default password is voltaire Step 2 Config Enter Config mode from the Exec mode Step 3 sm Enter sm configuration Step 4 pkey Enter pkey...

Page 201: ...p 3 Use the pkey set CLI command to add a partition to the ISR 9288 9096 9024 partitioning table When adding a partition you need to specify the Pkey number 0 15 table number row number and the Pkey v...

Page 202: ...o erase a row from the Pkey Table Step 1 Use the Pkey delete command as shown in the example below ISR9XXX config sm pkey low show entry Port GUID Pkey0 Pkey1 Pkey2 Pkey3 Pkey4 Pkey5 Pkey6 Pkey7 1 000...

Page 203: ...0 0 0 0 ISR9XXX config sm pkey set pkey1 1 0x0 Change will be effective only after pkey update command Step 2 Verify that the pkey 7554 in this example was removed from the row by using the CLI pkey...

Page 204: ...r This chapter provides information about how to update the firmware when physically replacing modules It includes information on the following topics Replacing sFBs Spines on page 12 2 Replacing sLBs...

Page 205: ...e version of the newly inserted sFB is identical to rest of the sFBs as shown in the example below Switch module firmware show Grab I2C lock Anafa No Node GUID System GUID LID FW Version Line 2 1 8f10...

Page 206: ...irmware version included in the software image Syntax update firmware update file dir The update firmware command will automatically burn all sLB and sFB modules in the chassis NOTE It is possible to...

Page 207: ...nd width check on page A 37 Step 5 Perform a routing configuration via the CLI using the sm initiate routing reconfiguration set command see Configuring Routing via the CLI Section 10 6 This is to man...

Page 208: ...g Routing via the VFM Section 10 5 Command spine add Description Adds an sFB to the sFB list Used for Clos topology configuration Syntax spine add row Guid However it is highly recommended to use the...

Page 209: ...0697 5f8 11 0 3 3 Line 6 1 8f104003f0694 5f8 25 0 3 3 2 8f104003f0695 5f8 26 0 3 3 Line 9 1 0 0 0 0 0 0 2 0 0 0 0 0 0 Step 2 If the firmware version is not identical to the other sLBs firmware version...

Page 210: ...the fabric configuration before removing the sMB To back up the rack grouping configuration Step 1 Export the fabric configuration by using the grouping export command as detailed in Chapter 8 Step 2...

Page 211: ...irmware version of the newly inserted ISR 9024 must be identical to the firmware of all the other ISR 9024 boards installed in the system To define the firmware version of an unmanaged ISR9024 via the...

Page 212: ...Inc fw_ini_ver 0 sw_version 00 00 00 ISR9288 0b5b utilities In this example the firmware version is fw_version 00 08 00 NOTE If the firmware version of the ISR 9024 is not identical to the other ISR 9...

Page 213: ...Chapter 13 Configuring Cluster Monitor About this Chapter This chapter provides instructions on how to configure the Cluster Monitor It includes information on the following topics What is Ganglia on...

Page 214: ...refer to http ganglia info page_id 45 13 2 Cluster Configuration To configure a Cluster Step 1 In the VFM use the Cluster Config dialog box to set the path of the Ganglia Monitoring tool by selecting...

Page 215: ...guration 13 3 P N 399Z00038 2006 Voltaire Inc Example http 172 25 2 175 ganglia m load_one r hour s descending c Support Cluster h HOST sh 1 hc 4 In the example below the Host is called hydra3 voltair...

Page 216: ...ities It includes information on the following topics Fabric Diagnostic Overview on page 14 2 Fabric Inspect Settings on page 14 7 Diagnostic Tools on page 14 10 Additional Tools and Examples on page...

Page 217: ...zed by the user It also monitors port counters errors and reports every port that passed its error threshold limit as configured by the user Event Logging Creates an event log file for both IB traps a...

Page 218: ...if a counter exceeds its threshold The SM monitors the fabric detects configuration changes and dynamically configures the new elements and new routes in the fabric The SM can detect fabric errors wa...

Page 219: ...t fails to operate Invalid link Duplicate or conflicting ports or path Yellow Warning Minor Warning minor reflects a problem in the fabric but does not prevent its operation A warning is asserted when...

Page 220: ...ture Additional problems related to fabric stability blocking or other 14 1 3 Potential Fabric Problems It is critical to verify the that fabric is clean to ensure optimum application performance and...

Page 221: ...to help identify where the problem is located 5 Use the Topology Map to identify nodes with a current alarm 6 Proactively look for increasing error counters using the statistics feature and running t...

Page 222: ...tion 6 7 2 14 2 1 Export Log This Export Log gathers a bunch of information and creates a tar file out of it This tar file is a mandatory piece of information for analyzing switch issues Sometimes tha...

Page 223: ...your FTP server and send the tar file to R D 14 2 2 Configuring the PortCounters csv file via the CLI This allows you to collect all the port data and errors within the system Telnet the switch IP and...

Page 224: ...must first be enabled using the CLI as described in Section 14 2 2 This tool is used to configure the current Alarms and Event Logs To define the PM settings using the VFM Step 1 Select Fabric Fabric...

Page 225: ...g if the threshold is passed between samples interval The counter is reset when it reaches its maximum value as defined in the InfiniBand specification Step 3 Disable the Thresholds you don t want to...

Page 226: ...following dialog box is displayed Figure 14 3 Port Counters Information Step 2 In the Interval SEC field enter the time interval in seconds After each interval has elapsed the counters are reset and...

Page 227: ...ounters Graph tab The following dialog box is displayed Figure 14 4 Port Counters Graph Step 2 In the Interval SEC field enter the time interval in seconds After each interval has elapsed the counters...

Page 228: ...information refer to Identifying a Bad Port Section 14 4 3 These features are also available via the CLI See CLI section below Viewing Logs in a Browser You can view error and event logs from a browse...

Page 229: ...Voltaire Subnet Manager VSM internal log vsm log show Show Voltaire Subnet Manager VSM log Logs can be exported to the remote FTP server using the following CLI commands exportFile topology Uploads t...

Page 230: ...a problem in the fabric The following is an example of an error log reported through the CLI menu ISR9XXX utilities event log show Jul 13 09 49 55 0 vsm 414 CREATE MC GROUP MGID ff12401bffff000000000...

Page 231: ...er can view the alarms via CLI GUI The current alarm Database includes Unique objects system subsystem port Alarm description Severities Manually corrective actions such as clear alarms The GUI will g...

Page 232: ...Look for the Alarm Appearance Traps and alarms are issued according to the degree of severity which can be one of the following Normal Warning Minor and Critical In the GUI the following color coding...

Page 233: ...s for all active nodes and their active ports in the fabric This includes information such as Node name InfiniBand IPoIB IP address Chassis component PM data failure reports and a detailed port counte...

Page 234: ...N A 0x005795e8 N A XmitWaits PeerLID PeerIBPort PeerPortGUID PeerPlatformType PeerModuleType PeerModuleIndex 0 16 9 0008f1040040061b ISR 9288 sFB 12 4 PeerPort PeerName Status internal 57 ISR9288 006...

Page 235: ...e system tries to detect the maximum link width between any two end points to optimize performance If a 1X link is detected either one of the ports is faulty or the cable is damaged A trap is then iss...

Page 236: ...rify if the problem remains on the switch port Step 2 If it remains use the second port on the HCA if available or use another HCA Step 3 If it remains replace the ISR 9288 9096 Line Board sLB or the...

Page 237: ...w as shown in Figure 14 7 opens and displays ISR 9288 9096 node information the sLB sFB chip and the port connection matrix shows each ISR 9288 9096 external port internal chip port and its Channel Ad...

Page 238: ...8 0061b ALERT This example shows the status of the sLB 24 module and its sFB 12 peer Module Using the Port LID and GUID of each board you can use the Node Info Window to identify the specific sLB and...

Page 239: ...during the fabric configuration cycle The subnet manager supports the dynamic recovery of a malfunctioning port bad port unknown peer status The user may manually define a bad port User Ignored is the...

Page 240: ...3f0b52 4 0008f104003f0b52 0 0 0 0 Unknown Peer 1280 8f104003f0b52 6 0008f104003f0b52 0 0 0 0 Unknown Peer 1408 8f104003f0b53 4 0008f104003f0b53 0 0 0 0 Unknown Peer 1408 8f104003f0b53 6 0008f104003f0b...

Page 241: ...CLI command under the CLI Config menu 14 4 4 Some CLI Diagnostic Tools port verify Utility This script checks the link width link speed port counters ports connectivity and provides a different kinds...

Page 242: ...ounters and or width and or speed NOTE When diagnosing a specific LID the peer connection is not checked As a result a false alarm might be generated for the width and or speed check That happens when...

Page 243: ...e Check Error counters w Check port s Width 1X 4X s Check port s Speed SDR DDR z Zero port counters u LID Check port s of the specified LID If LID is a switch LID all ports of the switch scanned If L...

Page 244: ...wed by plug in This debugging utility is for advanced users only This utility enables the user to manage ports on the fabric and to change the status of a port enable disable port manage sh is used to...

Page 245: ...error found will be printed lid 1 guid 0008f104004004d7 ports 24 lid 5 guid 0008f104003f0723 ports 24 lid 4 guid 0008f104003f0722 ports 24 lid 3 guid 0008f104003f071f ports 24 lid 2 guid 0008f104003f...

Page 246: ...A A Appendix A CLI Commands Reference This section provides information about the following CLI topics Voltaire CLI Basic Concept on page A 2 Exec Admin Mode on page A 3 Privileged Mode on page A 9 Co...

Page 247: ...mands Reference Voltaire CLI Basic Concept A 2 P N 399Z00038 2006 Voltaire Inc A 1 Voltaire CLI Basic Concept The Voltaire CLI concept implements a hierarchical organization as described in the diagra...

Page 248: ...the interface IP address and configuration ftp show Shows the FTP client configuration ib interface show Shows the InfiniBand interface IPoIB address and configuration io connect Connects to I O modu...

Page 249: ...lnet session via a Telnet connection or closes an SSH session Syntax end Command exit Description Exits to previous menu In this case this command is equal to the end command Syntax exit Command fast...

Page 250: ...p mask is 255 255 0 0 broadcast ip is 172 25 255 0 management interface is eth0 1 Command io connect Description Establishes a CLI session with a router module installed in an ISR 9288 9096 switch Syn...

Page 251: ...O Type IPR IP 172 25 3 95 Mask 255 255 0 0 In this example an FCR router module is installed in the slot located at the right hand side of the sRBD itself installed in slot 2 of the ISR 9288 9096 rea...

Page 252: ...session is connected which can be one of the following enabled or disabled When enabled the standby sMB maintains a mirror image of the active sMB and is ready to replace the active sMB in the event...

Page 253: ...ble grouping mode enable NOTE Refer to Section 6 7 1 for a description of the Subnet Manager parameters Command smb state show Description Shows the sMB redundancy status Syntax smb state show Example...

Page 254: ...SRXXXX A 3 2 Privileged Mode Commands Reference The commands available in Privileged mode and their descriptions are listed below NOTE Note The prompt shows that you are in Privileged Mode ISR9096 Dis...

Page 255: ...utilities Changes to switch debug Utilities mode version show Shows the software version information Syntax usage is as follows in Privileged mode Menu clock set clock set MMDDhhmmYYYY clock show clo...

Page 256: ...16 UTC 2005 Command clock show Description Displays the time and date currently set in the switch Syntax clock show Example Switch clock show Tue Sep 20 18 39 35 UTC 2005 Command config Description Ch...

Page 257: ...ription Displays a list of the modules installed on the front panel of an ISR 9288 9096 switch including the hardware version and serial number of the module Syntax front show Example Device HW Versio...

Page 258: ...itch For more information regarding QoS Arbitration see Section 9 3 Syntax qos info show Example QoS info is qos vl high arbitration limit 0 qos configuration enable enable qos apply configuration no...

Page 259: ...replication status show Description Shows the status of the sMB replication enabled or disabled When enabled the standby sMB maintains a mirror image of the active sMB and is ready to replace the act...

Page 260: ...escription Returns the current parameter settings of the Subnet Manager in the ISRXXXX Syntax sm info show Example subnet manager info is smName LAVISM port guid 0008f1040040061b topology CLOS stage 3...

Page 261: ...ch snmp community show get community public set community private trap community public Command update firmware Description Updates the firmware version included in the software image Syntax update fi...

Page 262: ...mote logger delete Delete the remote logger syslog of this switch remote logger set Set the remote logger syslog of this switch remote logger show Show the remote logger syslog of this switch smAgent...

Page 263: ...16136 IN CLI Run Sep 7 19 10 18 0 cli 16136 GetStr 279 IN Sep 7 19 10 18 0 cli 16136 Get 289 IN Command error log show Description Shows the error log file Syntax error log show Example Switch logs e...

Page 264: ...11 IST 2005 Command log mask set Description Sets the mask for event logging which enables the user to filter traps using a log mask The log mask sets a mask field in the software The following option...

Page 265: ...show Sep 8 08 29 15 0 sudo root TTY unknown PWD usr voltaire USER root COMMAND usr voltaire scripts ftpCli sh get root br6000 172 25 3 94 var www html jarPropertyFile txt tmp jarPropertyFile txt var f...

Page 266: ...tus of Voltaire Subnet Manager VSM internal counters Syntax vsm internal log show Example ISR9096 302f logs vsm internal log show Sun Feb 5 07 00 36 2006 Traps GidInService 64 111 GidOutOfService 65 4...

Page 267: ...Manager VSM internal log listing the traps and events according to the user configuration of verbosity Syntax vsm log show Example Switch logs vsm log show 17 50 36 327 1013 SM BUILD VERSION IBSW_OSM...

Page 268: ...and fabric topology port manage Performs a port manage command port verify Performs a port verify command sminfo Queries information from active Subnet Manager SM smpdump Dumps Subnet Manager SM attri...

Page 269: ...n g et l eave mctrace mctrace d ebug D irect src v erbose s imple t timeout_ms mlid src lid dest lid port manage port manage sh v f d e r LID PORT port verify port verify OPTIONS netdiscover netdiscov...

Page 270: ...H 0008f10403965224 Voltaire HCA400 1 LID 32 S 0008f104003f06c1 22 alert errs sym 73 alert linkrecovers 1 alert linkdowned 8 Human translation All HCA ports are scanned One HCA port displays a problem...

Page 271: ...cked against the supported maximum Usage port verify sh OPTIONS Options t Print output as a plain text without rich text options b Report only faulty ports a Check port s without zeroing counters this...

Page 272: ...h cases a component connected to the sFB is not responding it could either be sLB the IPR or the GER If it is the sLB try to remove and insert the sLB A device is probably set in continuous reset If t...

Page 273: ...tive port Example ISR9288 08cf utilities getpathrecord 0x8f10001020001 0x8f1040396086d Output path 0 src GUID 0x8f10001020001 LID 4608 dest GUID 0x8f1040396086d LID 128 Command lidtrace Description Pr...

Page 274: ...port Get Port settings madstat X lid path port Get Advanced port settings madstat A lid path Get smA port settings madstat C lid path port Get port Counters madstat D lid path port Mtl Get port Count...

Page 275: ...lues ISR9288 08cf utilities madstat C 128 1 errs sym 0 linkrecovers 0 linkdowned 0 errs rcv 0 errs remphysrcv 0 errs rcvswitchrelay 0 xmitdiscards 0 errs xmtconstraint 0 errs rcvconstraint 0 errs loca...

Page 276: ...FFFFFFF if p option is not specified then self port portguid of connected to CPU switch chip is used Command mctrace Description Finds multicast port path between nodes MC FDB searches from port to po...

Page 277: ...0008f104003f074e 16 10 S 0008f10400403002 22 4 S 0008f1040040300e 22 1 S 0008f10400403009 22 switchguids 0x8f104003f0764 Switch 24 S 0008f104003f0764 ISR9288 ISR9096 Voltaire sLB 24 smalid 8 21 S 0008...

Page 278: ...rom the IB fabric using LID and direct routed MADs This information includes subnet manager GUID activity count priority and state Syntax sminfo d ebug s state p prio a activity D irect t timeout_ms s...

Page 279: ...ebug s tring x hex b binary data D irect t timeout_ms dest atrrid mod Subnet Management Attributes AttributeName AttrId AttrModifier Notice 2 0 NodeDescription 16 0 NodeInfo 17 0 SwitchInfo 18 0 GUID...

Page 280: ...that port The direct path 1 7 led us to element 2 The corresponding MAD will retrieve the required data from the element provided this is a switch element Command smpex Description smpex is a powerful...

Page 281: ...turn failed In that case the access session will abort 3 If there is no current session alive and they were granted a lock the utility returns success In that case the access session is holding the lo...

Page 282: ...orts 24 port 22 width 1X port 24 width 1X lid 10 guid 0008f10400403002 ports 24 lid 11 guid 0008f1040040300e ports 24 lid 5 guid 0008f100010c0000 ports 2 lid 4 guid 0008f100010c0000 ports 2 Command ze...

Page 283: ...er Changes to Cluster configuration mode end Ends the CLI exit Exits to previous menu factory default Switches back to factory default and reboots the system ftp Changes to FTP configuration mode grou...

Page 284: ...arm Command cluster Description Changes to Cluster configuration mode that is used to assign the IP of router modules installed in the switch See Appendix A 4 4 for a list of commands available under...

Page 285: ...f commands available under this menu Syntax interface IB Command interface LOCAL Description Type this command to access the Local Interface configuration mode that is used to define the settings of L...

Page 286: ...sm Description Type this command to access the Subnet Manager configuration mode which is used to set SM parameters See Appendix A 4 13 for a list of commands available under this menu Syntax sm Comm...

Page 287: ...d or disabled Syntax watchdog mode show Example Switch config watchdog mode show Watchdog mode is enable Command watchdog timeout show Description Returns the WDT timeout setting in milliseconds which...

Page 288: ...clear all alarm show alarm show The following details the commands available under the Alarms Menu Command alarm clear Description Clears the selected current alarm entry Syntax alarm clear Example Sw...

Page 289: ...ption Establishes a CLI session with a router module installed in an ISR 9288 9096 switch Syntax io connect slot number position number Example Switch config cluster io connect 12 2 This example shows...

Page 290: ...nfiguration Config Mode A 45 P N 399Z00038 2006 Voltaire Inc Example Switch config cluster io show slot 2 position 2 I O Type FCR IP 172 25 3 98 Mask 255 255 0 0 slot 2 position 3 I O Type FCR IP 172...

Page 291: ...ote FTP server using previously defined server name user name and password exportLOGs Uploads Logs to the remote FTP server using previously defined server name user name and password exportSm Uploads...

Page 292: ...fined server name user name and password The rack grouping file is created by using the group export command from the Group Mode Syntax exportFile group remote path Example ISR9288 0967 config ftp ser...

Page 293: ...configuration file from the FTP server Syntax import remote path Example ISR9288 config ftp import pub Command importFile group Description Downloads group csv from the remote FTP server using a prev...

Page 294: ...CLI Commands Reference Configuration Config Mode A 49 P N 399Z00038 2006 Voltaire Inc Description Defines the user name that the switch uses to access the file server Syntax username username...

Page 295: ...s the group info group import Imports the group info group verify Verifies import group info Group syntax usage is as follows group error log show group error log show group export group export group...

Page 296: ...ommand verifies and updates the current group information with the imported group information Syntax group import Example ISR9288 0957 config group group import Summary report Racks 1 Elements 6 Norma...

Page 297: ...management interface set Sets this interface as the Management interface mtu set Sets the interface mtu mtu show Shows the interface mtu configuration ping Sends echo messages IB Interface syntax usag...

Page 298: ...Interface broadcasts Syntax ip address ib show Example Switch config if IB ip address ib show IB interface ip is 172 25 7 11 ip mask is 255 255 0 0 broadcast ip is 172 25 255 255 management interface...

Page 299: ...w Shows the interface IP address and configuration ping Sends echo messages Local Interface syntax usage is as follows ip address local set ip address local ip address netmask ip address local show ip...

Page 300: ...terface as the Management interface ping Sends echo messages Fast Interface syntax usage is as follows broadcast fast set broadcast fast set broadcast ip address ip address fast set ip address fast se...

Page 301: ...ent interface is ipoib0 Command mac address fast show Description Shows the interface MAC address Syntax mac address fast show Example Switch config if fast mac address fast show 00 80 F9 69 03 08 Com...

Page 302: ...ge is as follows manual node names show manual node names show system name set system name set name system name show system name show unique default name set unique default name set disable enable uni...

Page 303: ...For example ISR9096 300b Use the system name set command to name the Switch manually or to get the default name Perform the following steps to name your system 1 Remove the previous name system name s...

Page 304: ...ity errors set Sets the local link integrity errors for the PM pm mode set Sets the PM to enable pm polling interval set Sets the polling interval for the PM pm port rcv constraint errors set Sets the...

Page 305: ...polling interval set 60 65535 pm port rcv constraint errors set pm port rcv constraint errors set 0 256 pm port rcv errors set pm port rcv errors set 0 65535 pm port rcv remote physical errors set pm...

Page 306: ...n is supported the PM monitors the difference per interval for each counter and generates an error when it passes the predefined threshold Syntax pm counter operation set delta avg avg_sum min max Com...

Page 307: ...ines PM polling interval in seconds default 180 Syntax pm polling interval set 60 65535 Example pm polling interval set 60 Command pm port rcv constraint errors set Description Defines the error thres...

Page 308: ...eshold for this counter Setting to 0 disables counter monitoring default 5 Syntax pm port xmit discards set 0 65535 Note In this case it is not recommended to change the default settings Command pm re...

Page 309: ...ort counters Syntax pm sLBPortCounter show slot Example pm sLBPortCounter show 2 Command pm scope set Description Defines the scope for PM monitoring Options include all all nodes in the fabric switch...

Page 310: ...0 Report mode enable alert join enable alert ATS enable ISR9096 300b config pm Command pm symbol error counter set Description Defines the error threshold for this counter Setting to 0 disables counte...

Page 311: ...Ends the CLI exit Exits to previous menu The following details the commands available under the Route menu Command default gw IB set Description Sets the IB address of the switch default gateway Synt...

Page 312: ...ommand usage string badports Changes to Bad Ports mode end Ends the CLI exit Exits to previous menu mc Changes to MC mode pkey Changes to Pkey mode qos Changes to QoS configuration mode sm info algori...

Page 313: ...polling retries set sm info polling retries set int 3 180 sm info port state change trap set sm info port state change trap set enable disable sm info priority set sm info priority set int 0 15 sm in...

Page 314: ...ss the QoS configuration menu which is used to define the Quality of Service QoS settings See page A 80 for a list of commands available under this menu Syntax qos Command sm info algorithm set Descri...

Page 315: ...c set Description Sets the number of LIDs per node in the fabric a power of 2 from 0 to 7 Syntax sm info lmc set int 0 7 Example sm info lmc set 3 Note 8 LIDS set for every node in the fabric Command...

Page 316: ...ription Sets the priority of the Subnet Manager the higher the number the higher the priority When there are two Subnet Managers in the network the one that has the higher priority will be the Master...

Page 317: ...abric sweep Syntax sm info sm initiate fabric configuration set Example sm info sm initiate fabric configuration set Command sm info sm initiate fabric reconfiguration set Description Applies topology...

Page 318: ...ty level to sm log file Syntax sm info verbosity set error info verbose debug function frames Example sm info verbosity set verbose Command sm info verbosity Topology set Description Sets the verbosit...

Page 319: ...4004c7 3 0008f104004004c5 4 0008f104004004ea 5 0008f104004004e9 Command spines show Description Displays the sFBs list defined in the switch including entry numbers and GUIDs Syntax spines show Exampl...

Page 320: ...foEvent OFF volSmaTopologyWarningEvent OFF volSmaTopologyErrorEvent OFF volPmSymbolErrorEvent is ON volPmLinkErrorRecoveryEvent is ON volPmLinkDownedCounterEvent is ON volPmPortRcvErrorsEvent is ON vo...

Page 321: ...rts remove guid port 1 255 bad ports show bad ports show The following details the commands available in the Subnet Manager badports sub menu Command bad ports add Description Adds the bad port to the...

Page 322: ...ion Expected Results Switch config sm badports bad ports mode show Bad Ports mode is enable Command bad ports mode show Description Shows the status of bad port data collection When bad port collectio...

Page 323: ...CA Peer GUID MC Multicast Mode This menu allows you to set the multicast parameters The following are the commands available under the Subnet Manager MC sub menu Switch config sm mc Displays the list...

Page 324: ...rence Configuration Config Mode A 79 P N 399Z00038 2006 Voltaire Inc Command mc show Description Shows the Multicast mode setting enabled to disabled Syntax mc show Example Switch config sm mc mc show...

Page 325: ...s the Partition Key Table show active nodes Shows the active nodes in the fabric pkey syntax usage is as follows partitioning enable set partitioning enable set enable disable partitioning enable show...

Page 326: ...as defined in the range range is the range of rows you want to erase Syntax pkey delete row 1 range 0 Example pkey delete 1 10 Command pkey high show Description Shows the high part of Pkey table Synt...

Page 327: ...f 0 0 0 0 0 0 0 Command pkey port guid set Description Sets the port guid in the Partition Key Table Syntax pkey port guid set row 1 guid Example Switch config sm pkey pkey port guid set 2 8f10001a100...

Page 328: ...active nodes Example ISR9096 300b config sm pkey show active nodes LID GUID NAME DESCRIPTION 0x0002 0002c9000100d150 segin VoltaireHCA400 0x0003 0002c901074364d0 subra VoltaireHCA400 0x000A 0008f1040...

Page 329: ...itration limit set Sets the limit of High Arbitration Table qos vl low arbitration set Sets the Low Arbitration Table qos vl low arbitration show Shows the Low Arbitration Table QoS syntax usage is as...

Page 330: ...m qos qos configuration enable set enable Expected Result Switch config sm qos qos info show QoS info is qos vl high arbitration limit 0 qos configuration enable enable qos apply configuration no Comm...

Page 331: ...0 2 1 1 3 2 2 4 3 3 5 4 4 6 5 5 7 6 6 8 7 7 9 8 8 10 9 9 11 10 10 12 11 11 13 12 12 14 13 13 15 14 14 16 15 7 Command qos vl high arbitration set Description Sets the High Arbitration table as describ...

Page 332: ...ration Table as described on page 9 4 Syntax qos vl high arbitration limit set number Example Switch config sm qos qos vl high arbitration limit set 20 Expected result Switch config sm qos qos info sh...

Page 333: ...nes operational on this port are indicated as follows 1 VL0 2 VL0 VL1 3 VL0 VL3 4 VL0 VL7 5 VL0 VL14 HOQ Life Sets the time a packet can live at the head of a VL queue LifeTimeValue Sets the time a pa...

Page 334: ...llowing commands config sm sm info hoq set 1 31 config sm sm info slv set 1 31 config sm sm info mopvl set vl0 vl0 1 vl0 3 vl0 7 vl0 14 config sm mc config sm mc mc auto join set disable enable SM Aut...

Page 335: ...s snmp description set Sets the SNMP description snmp description show Shows the SNMP agent description snmp location set Sets the SNMP location snmp location show Shows the SNMP agent location SNMP s...

Page 336: ...ster monitor string Syntax snmp cluster monitor show Example ISR9096 3009 config snmp snmp cluster monitor show cluster monitor is http 172 25 2 175 ganglia m load_one r hour s descending c Support Cl...

Page 337: ...snmp community show get community public set community private trap community public Command snmp community trap Description Sets the SNMP trap community The default trap community string is public S...

Page 338: ...iption show Description Displays version information on the SNMP agent in the switch Syntax snmp description show Command snmp location set Description Sets the SNMP location Syntax snmp location set...

Page 339: ...number Port GUID Displays the Global Unique Identification number GUID of the port Node Type Displays the node type and model Width Displays the current link width 1X 4X or 12X State Displays the cur...

Page 340: ...xternal port number or internal for internal ports Name Node name where possible otherwise the nodes GUID NodeIP The IP address associated where this node where possible otherwise 0 0 0 0 DeviceID Vol...

Page 341: ...rt number or internal for internal ports where the remote connection exists PeerName The remote connected Ports node name node name or GUID Status OK or ALERT port counter exceed threshold 1X link etc...

Page 342: ...threshold by entering a different value in the respective field Table B 3 PM Counters Tresholds Monitored Counter Values Default Threshold SymbolErrorCounter 0 65535 5 LinkErrorRecoveryCounter 0 255 1...

Page 343: ...should not be increasing and a constantly increasing number probably indicates a bad link PortRcvRemotePhysicalErrors Total number of packets marked with the EBP delimiter received on the port This in...

Page 344: ...transmitted from the port for the following reasons FilterRawOutbound is true and packet is raw PartitionEnforcementOutbound is true and packet fails partition check IP version check or transport head...

Page 345: ...received on all VLs from the port selected by PortSelect This includes all octets between and not including the start of packet delimiter and VCRC It excludes all link packets PortXmitPackets Total nu...

Page 346: ...er severity according to Voltaire or user defined thresholds C 1 Trap Structure Each trap is characterized by the parameters listed below A Header invisible to the user which includes the following fi...

Page 347: ...Jul 13 09 49 56 0 vsm 440 0008f104003f074e port 22 1X port was found Port GUID 8f104003f074e IB Port 22 Peer at 0008f104003f074e port 24 Port GUID 8f104003f074e IB Port 24 Jul 13 09 49 56 0 vsm 440 00...

Page 348: ...current value last value Y warning enable manual PortXmitDiscards counter threshold was exceeded New value is XXXs previous value was XYZ PM_PortXmitConstraintErrors Object port Parameters current val...

Page 349: ...onflicting ports or path OSM_EVENT_DUP_SWITCH_GUID Object port Parameters node_type1 string node_type2 string Y critical enable manual Found duplicate s port GUID 016Lx port XYZ another s already exis...

Page 350: ..._TIMEOUT_EVENT Object port Parameters method string attr string path N warning enable Link timeout Method s attribute s DR path s SECURITY_ERROR SRV__BAD_P_KEY_TRAP_257 Object node Parameters p_key ui...

Page 351: ...016Lx joined MGID 016Lx 016Lx MLID 0x X SRV__LID_CHANGE_EVENT Object node Parameters min lid max lid old min lid old max lid Y normal enable manual Assigned new LID XYZ XYZ for port GUID 016Lx old LID...

Page 352: ...distance XYZ and port GUID 016Lx port XYZ distance XYZ OSM_EVENT_SWITCH_UNEXPECTED_D ISTANCE Object node Parameters distance expected distance N minor enable manual Unexpected distance XYZ hops from...

Page 353: ...ITCH_OUT_OF_CLOS Object node N warning enable manual Found a Switch GUID 016Lx outside the CLOS OSM_EVENT_HCA_OUT_OF_CLOS Object node N warning enable manual Found an HCA Router GUID 016Lx outside the...

Page 354: ...mand Line Interface CPU Central Processing Unit CQ Completion Queues DMA Direct Memory Access FC Fibre Channel FE Fast Ethernet FTP File Transfer Protocol FRU Field Replaceable Unit GbE Gigabit Ethern...

Page 355: ...on RDMA Remote Direct Memory Access SAN Storage Area Network SA Subnet Administration SDP Socket Direct Protocol SM Subnet Manager SMP Short for Simple Management Protocol another name for SNMP2 SNMP2...

Page 356: ...99Z00038 2006 Voltaire Inc TCP Transmission Control Protocol UDP Unreliable Datagram Protocol VDM Voltaire Device Manager VFM Voltaire Fabric Manager VL Virtual Lane VSM Voltaire Subnet Manager WQE Wo...

Page 357: ...A 52 C Case Sensitivity CLI 2 5 Channel Adapter connection 14 22 chip sLB sFB 14 22 Clearing Events 6 27 CLI case sensitivity 2 3 Command Reference A 1 Exec mode A 3 FTP configuration A 43 A 44 Group...

Page 358: ...6 56 Current Alarms window 6 56 D Data center reconstruction 1 2 Data Clusters 1 2 deadlock avoiding 9 2 Debug mode CLI A 11 Default Gateway A 63 default gw IB set A 63 Default passwords 3 2 default...

Page 359: ...outer 1 5 Filters Event 6 27 find_bad_ports A 26 findpath A 27 Firmware Updates A 9 Firmware Upgrade 4 5 First Time Configuration of switch 3 2 Fit to contents button 6 21 Flow Control 3 3 front show...

Page 360: ...al set A 51 ip address local show A 51 IPR 1 6 ISR 9024 Switch 5 14 ISR 9288 Switch 5 13 5 14 ISR9288 FRUs 1 6 J Java Web Start application Sun 6 4 L LCL SM 5 11 LED 5 12 lidtrace A 28 Limit of High P...

Page 361: ...rt information Infiniband See Infiniband port information Port Information viewing in VFM 6 37 Port LID 6 29 Port MTU 5 20 Port Properties 6 28 Port manage A 25 port verify A 25 Power Supplies 1 6 Pri...

Page 362: ...A 15 SM KEY 6 42 smpdump A 34 smpex A 35 snmp A 41 SNMP 1 10 1 13 5 9 5 41 5 42 A 8 A 16 A 41 A 52 A 85 A 87 A 88 A 89 SNMP commands CLI A 85 snmp community show A 8 A 16 software reboot 5 46 softwar...

Page 363: ...View 6 22 topology log show A 21 U umServer log show A 21 update firmware 12 3 12 6 A 16 User name field 5 44 User settings using the Device Manager 5 43 Users window 5 5 V VDM Main Window 5 7 vendst...

Reviews:

No comments