HP Cluster Platform Express v2010 Скачать руководство пользователя страница 68

Страница: 68 / 140

Содержание Cluster Platform Express v2010

Страница 1: ...HP Cluster Platform InfiniBand Interconnect Installation and User s Guide HP Part Number A CPIBI 1E Published October 2006 ...

Страница 2: ...istered trademark and the Topspin logo TopspinOS Topspin Switched Computing System Grid to Go and VFrame are trademarks of Topspin Communications Inc owned by Cisco Systems Inc Microsoft and Windows are U S registered trademarks of Microsoft Corporation Windows Server 2003 and Compute Cluster Server 2003 are U S trademarks of Microsoft Corporation Voltaire and the V logo V are registered trademark...

Страница 3: ...view of the ISR 9024 Interconnect 29 2 1 1 Internally Managed ISR 9024 Front Panel 30 2 1 2 ISR 9024 Management Card 31 2 1 3 Externally Managed ISR 9024 Front Panel 32 2 1 4 ISR 9024 Rear Panel 32 2 1 5 Cooling and Airflow 33 2 2 Unpacking the ISR 9024 Interconnect 33 2 3 Mounting the ISR 9024 in the Rack 34 2 4 Installing the ISR 9024 Power Supply Unit 36 2 5 Operating the ISR 9024 Interconnect ...

Страница 4: ...h Module 69 5 1 1 Subnet Manager 70 6 Installing Field Replaceable Units in the ISR 9XXX Chassis 71 6 1 Guidelines for Inserting and Extracting Boards 71 6 2 Installing or Replacing the sFB 12 Fabric Board 72 6 3 Installing or Replacing sLB 24 Line Boards 73 6 4 Installing or Replacing the sMB Management Board 75 6 5 Installing and Replacing Power Supply Units 76 6 6 Populating the Router Blade Dr...

Страница 5: ...02 8 8 Mellanox Memory Free PCI Express HCA DDR 102 8 8 1 LEDs 103 8 9 Mellanox PCI Express HCA DDR 103 8 9 1 Mellanox PCI Express DDR HCA LEDs 104 8 9 2 HP HPC 4x DDR IB Mezzanine HCA 105 8 9 3 Operating System and Software Requirements 105 9 Postinstallation Troubleshooting and Diagnostics 107 9 1 Postinstallation Troubleshooting 107 9 2 Startup Checks for the ISR 9XXX 107 9 3 Debugging a Fabric...

Страница 6: ...ellanox Memory Free PCI Express HCA SDR Specifications 129 F 7 Mellanox Memory Free PCI Express HCA DDR Specifications 129 F 8 Mellanox PCI Express DDR HCA Specifications 130 F 9 HPC 4x DDR IB Mezzanine HCA Specifications 130 Index 133 6 Table of Contents ...

Страница 7: ...3 ISR 9288 Crate Door 54 4 4 Top Compartment in the ISR 9096 Box 55 4 5 ISR 9096 Accessories Box 56 4 6 ISR 9288 Front View 57 4 7 ISR 9288 Rear View 58 4 8 ISR 9288 Interconnect Rail Kit 59 4 9 ISR 9XXX Rack Mount Procedure 60 4 10 Telescoping Rail Assembly 60 4 11 Assembling 9XXX Rails 61 4 12 Attaching the Cable Management Hooks 62 4 13 Installing the Cabling Clamps 63 4 14 ISR 9288 Chassis Fro...

Страница 8: ... PCI Express HCA 98 8 5 Mellanox PCI X HCA 99 8 6 Mellanox PCI Express HCA 100 8 7 Mellanox Memory Free PCI Express HCA SDR 101 8 8 Mellanox Memory Free PCI Express HCA DDR 102 8 9 Mellanox PCI Express HCA DDR 104 8 10 4x DDR IB Mezzanine HCA 105 8 List of Figures ...

Страница 9: ...ric Board LED Status 73 6 2 Line Board Status LEDs 74 6 3 sMB Board LEDs 75 6 4 Power Supply Unit LEDs 77 6 5 IPR Blade LEDs 78 6 6 FCR Blade LEDs 79 6 7 sCTRL Board LEDs 81 6 8 sFU 8 Fan Module LEDs 84 7 1 InfiniBand Cable Characteristics 85 8 1 HCA 400 Status LEDs 97 9 ...

Страница 10: ...10 ...

Страница 11: ...escribes how to install maintain and operate the 96 port Voltaire interconnect model ISR 9096 and the 288 port Voltaire interconnect model ISR 9288 Chapter 4 Installing and Maintaining the ISR 9096 and ISR 9288 Interconnects Describes how to install field replaceable units modules in the 288 port Voltaire interconnect model ISR 9288 Chapter 6 Installing Field Replaceable Units in the ISR 9XXX Chas...

Страница 12: ...luster Platform architecture and concepts Related Documentation The following documents may be useful references when you are installing and administering the HP Cluster Platform cluster Voltaire Documentation The Voltaire documentation consists of the following information Voltaire ISR 9024 S D Installation Manual Explains how to install the ISR 9024 Switch Voltaire Switch 9024 9096 and 9288 User...

Страница 13: ... compute or utility node ProCurve network switches such as the ProCurve 2824 Gigabit Ethernet switch HP 10000 series rack including the following rack components Rackmount workstation or KVM keyboard video and mouse Power distribution units PDU Optional storage in the form of disk shelves just a bunch of disks JBOD a storage area network SAN unit or a separate modular storage cluster Depending on ...

Страница 14: ...e This type denotes literal items such as command names file names routines directory names path names signals messages and programming language structures Boldface type In command and interactive examples this type denotes literal items entered by the user typed user input Boldface type in text indicates a word that is defined in the glossary Italic type Italic slanted type indicates variable val...

Страница 15: ...cumentation CD ROM before using your system Moving and Stabilizing Racks To reduce the risk of personal injury or damage to equipment in the cluster racks do not attempt to move the racks without adequate assistance due to their height and weight Also do not attempt to move a rack on an incline that is greater than 10 degrees from the horizontal No rack stabilization kits are provided with the clu...

Страница 16: ...ttery with the same or equivalent type as recommended by the manufacturer The battery in this system is a lithium battery that does not contain any heavy metals However to protect the environment do not dispose of batteries in household waste Return used batteries either to the shop from which you bought them to the dealer from whom you purchased your system or to HP so that they can either be rec...

Страница 17: ...ke back your old system for recycling when it reaches the end of its useful life HP has a product take back program in several countries The collected equipment is sent to HP recycling facilities in Europe or the U S A As many parts as possible are reused The remainder is recycled Special care is taken for batteries and other potentially toxic substances these are reduced into non harmful componen...

Страница 18: ...18 ...

Страница 19: ...ata transfers required by shared I O buses It provides inter processor communication and memory sharing at speeds from 2 5 Gb s to 30 Gb s It has advanced fault isolation controls that provide fault tolerance InfiniBand Network Features The InfiniBand architecture consists of single data rate SDR and double data rate DDR channels created by linking host channel adapters HCAs and target channel ada...

Страница 20: ...r provides a reliable connection between connected applications nodes or an unreliable datagram message queue The communications protocol consists of four operations Transmit a message Using remote data memory access RDMA writes data directly into a device s memory Using remote data memory access RDMA reads data directly from a device s memory Atomic operations updating remote memory words The HCA...

Страница 21: ...ia RDMA 2 Server CPU 3 HCA installed in the server s PCI bus 4 InfiniBand Interconnect 5 Servers as nodes in an InfiniBand network 6 A Storage unit with an integrated HCA 7 A Fibre Channel gateway router with an integrated HCA linked to a Fibre Channel storage array 8 A Fibre Channel storage array accessed through a Fibre Channel gateway router 9 An Ethernet router with an integrated HCA 10 Device...

Страница 22: ...atus LEDs and power supplies Figure 1 2 ISR 9024 Interconnect Front Panel View The ISR 9024 interconnect is a fully non blocking interconnect with a theoretical throughput of 480 Gb s This device has the following physical and operational features A 1U chassis designed for industry standard racks 24 ports of 10 Gb s throughput Fabric scalability from several nodes to hundreds of nodes Provides a m...

Страница 23: ...hundreds of nodes Provides a modular building block for fat tree topologies also known as Clos topologies of hundreds of nodes When combined with the ISR 9288 the ISR 9024 can scale topologies to thousands of nodes 1 5 Identifying the ISR 9096 Interconnect The ISR 9096 InfiniBand interconnect features a fat tree Clos topology that provides full bisectional bandwidth for each port Figure 1 4 shows ...

Страница 24: ... redundant management boards model sMB including a CPU Mezzanine Up to 4 router blade drawers model sRBD that each support up to three router blades which are either of the following models TCP IP internet protocol router blade model IPR that provides four Gigabit Ethernet ports and an RJ 45 management port that includes both 10 100 Base T Ethernet and serial ports Fibre Channel router blade model...

Страница 25: ...to thousands of nodes Provides a modular building block for fat tree Clos topologies providing full bisectional bandwidth for each port Multi protocol connectivity in a single chassis Server clusters Fiber Channel FC Storage Area Networks SANs NAS appliances Internet Protocol IP SANs TCP IP Ethernet networks LANs You can configure other InfiniBand to TCP IP and InfiniBand to Fibre channel router b...

Страница 26: ...r adapter in the case of the ISR 9024 from a dumb terminal or a PC running a terminal or telnet application If you are unable to manage an interconnect by any other method this interface provides a fallback All three InfiniBand interconnects have an embedded CPU that provides the following interconnect management software Command Line Interface CLI You connect to the CLI by using a serial console ...

Страница 27: ...anagement and Diagnostic Guide and the HP Cluster Platform InfiniBand Fabric Management and Diagnostic Guide for information on how to launch and use the management interfaces 1 7 Understanding the Interconnect Management Capabilities 27 ...

Страница 28: ...28 ...

Страница 29: ...u to administer the interconnect by using SNMP or InfiniBand in band management You configure and manage the fabric by using either the integrated graphical user interface GUI or the command line interface CLI You use a console remote telnet or HTTP Java client connection to access the interconnect s management interfaces The management interfaces enable you to monitor upgrade and troubleshoot the...

Страница 30: ... four 4x ports each port with a 10 Gb s throughput Twelve 4X ports and four 12x ports with 10 Gb s and 40 Gb s throughputs respectively The ISR 9024 is internally managed master mode by an add on management card or externally managed slave mode where no management card is installed You can manage any ISR 9024 slave from any ISR 9024 master or from any other ISRxxxx device in the fabric that includ...

Страница 31: ...ED amber for port 13 10 Physical link indicator LED green for port 13 2 1 2 ISR 9024 Management Card The management card provides the ISR 9024 internal management capabilities enabling you not only to manage the interconnect in which the card is installed but also to manage any other manageable device in the fabric The Ethernet and serial ports accessible at the front mounted on the management car...

Страница 32: ... 3 Physical link indicator LED green for port 13 4 Reset button 5 Status indicator LED indicating the power supply status and hot swap status 6 Power supply module 7 I2C communications port 8 InfiniBand port indicator LEDs amber for logical link status green for physical link status 9 Three interconnect status LEDs that show the status of the entire unit the subnet manager active passive or standb...

Страница 33: ... heat sink and to the chassis Heated air flows out by convection through the vent holes along the top of the front and rear of the chassis Cooled air flows into the chassis through the side vent holes The side entry of air is made possible by a chassis width slightly narrower than the standard rack width Internally managed ISR 9024 interconnects are equipped with an additional heat sink on the man...

Страница 34: ...eps in reverse order to repack a chassis for shipping ensuring that the chassis is secure in the shock absorbing materials Do not reuse any damaged packaging contact your HP sales and service representative if you are unsure about return shipping requirements 2 3 Mounting the ISR 9024 in the Rack You mount the ISR 9024 interconnect with its ports facing towards the rear of the rack Figure 2 5 show...

Страница 35: ... is approximately 29 2 inches and within the L bracket s range of adjustment 8 Repeat steps 2 4 for the right rail 9 For each bracket clip two M6 cage nuts into the back of each of the four rack columns at the appropriate U location Ensure that you install a total of eight M6 cage nuts 10 Working at the front of the rack lift and slide the chassis into its location and secure the front L brackets ...

Страница 36: ...plete a The chassis is secured in the rack b Cabling is complete as described in Chapter 7 c If not already connected connect a power supply cable to the IEC inlets for each PSU installed maximum two cables to the adjacent power strip outlets located in the sides of the rack d All other cluster components are installed cabled and powered up 2 Apply power to the interconnect and verify the followin...

Страница 37: ...M LED does not illuminate if the SR 9024 is externally managed there is no management card present If the SM indicator does not illuminate on an internally managed ISR 9024 the management card is not running the subnet management software Call HP service 6 The InfiniBand port links do not illuminate as follows The green physical link indicator does not illuminate The green physical link indicator ...

Страница 38: ...38 ...

Страница 39: ...s The ISR 9024 S D incorporates redundant hot swappable power supplies for high availability as well as a hot swappable fan unit The ISR 9024 S D offers a plug and play environment allowing servers to be added without taking down the fabric The ISR 9024 S D is available with an active CPU board ISR 9024S D M internally managed with the embedded GridVision Device and Fabric Manager or without a CPU...

Страница 40: ...9024 M InfiniBand and system indicator LEDs A reset button for software reset and unit initialization The ISR 9024 S D is available in four configurations ISR 9024S 24 4X ports externally managed ISR 9024S M 24 4X ports internally managed ISR 9024D 24 4X DDR ports externally managed ISR 9024D M 24 4X DDR ports internally managed The ISR 9024 S D is internally managed master mode by an add on manag...

Страница 41: ...anagement interface over a local network front and rear RJ 45 connectors cannot be used simultaneously 8 CLI connector Provides serial CLI console interface 9 I C Connector Provides serial I C interface 10 Power supply indicator 11 Hot swappable power supply 3 1 1 1 ISR 9024S D M Management Card The internally managed ISR 9024S D M have an on board CPU running embedded GridVision device and fabric...

Страница 42: ...al I C interface 8 Power supply indicator 9 Power supply module 3 1 3 ISR 9024 S D Rear Panel The rear panel is identical in all ISR 9024 S D RoHS compliant configurations and is shown in Figure 3 3 Figure 3 3 ISR 9024 S D Rear Panel InfiniBand Ports 2 1 3 4 5 6 The following list corresponds to the callouts shown in Figure 3 3 1 IEC power receptacle 2 24 4x InfiniBand ports 3 Port LEDs include Lo...

Страница 43: ...f the box shows sign of damage or is unsealed contact your HP sales or service representative 3 Use only short bladed safety knife to slit the sealing tape ensuring that you do not damage the packaging or shock absorption material callout 6 in Figure 3 4 4 Remove the top box callout 4 in Figure 3 4 which contains small parts Retain the packing list for kit verification 5 Grasp the sides of the cha...

Страница 44: ...sales and service representative if you are unsure about return shipping requirements 3 3 Mounting the ISR 9024 S D in the Rack You mount the ISR 9024 S D interconnect with its ports facing towards the rear of the rack Figure 3 5 shows the mounting kit in use Figure 3 5 The ISR 9024 S D Rack Kit 1 2 3 4 Use the following procedure to mount the chassis in the rack 1 If you are replacing an intercon...

Страница 45: ... by using four M6 machine screws Torque each screw to 30 in lb 12 Tighten the screws that secure the rear L bracket to the track and verify that all fasteners are tight 13 How you power up your interconnect depends on the current state of the cluster and the requirements of the operating environment If the cluster is not running you are now ready to cable the interconnect following the cabling ins...

Страница 46: ... new fan in the slot 3 Holding the fan level slide it into the slot until it meets resistance at the chassis connector It should slide smoothly and easily 4 Push the module further until it is completely seated 5 Use the captive fasteners at each side of the fan to secure it in place 3 5 Installing the ISR 9024 S D Power Supply Units An ISR 9024 S D contains two identical power supplies in the sid...

Страница 47: ...ront panel that is labelled SM is illuminated as follows Constant for active mode Flashing for standby mode d The green status LED on the front panel that is labelled PS is illuminated 3 For each port that has a connected InfiniBand link cable a The green physical link LED is illuminated b The amber logical link LED is illuminated in one of the following states Constant indicating the presence of ...

Страница 48: ...cator does not illuminate The green physical link indicator is illuminated but the logical link indicator is not These conditions indicate that the InfiniBand cable or its connector is faulty For both cases verify that the cable is connected correctly at both ends Try swapping the cable with a spare or use the cable from an adjacent port temporarily to eliminate the cable as the source of the prob...

Страница 49: ...nce computing HPC clusters The ISR 9XXX enables high performance applications to run on distributed server storage and network resources You can interconnect multiple ISR 9XXXs to create large clusters The ISR 9XXX provides a fat tree Clos topology that provides full non blocking bisectional bandwidth for each port The ISR 9288 supports up to 288 InfiniBand 4X 10 Gb s ports The ISR 9096 supports u...

Страница 50: ...IP router and FCR Fibre Channel router modules Each router module drawer can contain up to three router blades Supports up to four vertical fabric boards sFB Supports one or two redundant hot swappable management boards sMB for fabric management and chassis management when two sMB boards are installed they support failover capabilities in the event of failure to one of the boards Supports multiple...

Страница 51: ...mbination thereof The ISR 9288 can host up to 12 sLB 24 or sLB 8 12 or any combination thereof Management board one or two redundant hot swappable management sMB boards including a CPU Mezzanine Router blade drawer a mechanical and electrical interface with up to three router blades IPR and or FCR each The ISR 9096 enclosure can contain up to 4 hot swappable sRBDs The ISR 9288 enclosure can contai...

Страница 52: ...L board Use the following procedure to reset the ISR 9XXX 1 Using a thick wire or tip of a pen not a pencil press and hold the recessed button for the following time periods a 1 second Press and hold the reset button for one second to reset the sMB board only This does not disrupt data traffic over the interconnect b 6 seconds Press and hold the reset button for six seconds to reset the entire int...

Страница 53: ... the ISR 9288 container 1 Remove the top cover of the crate after releasing the four clamps 2 Verify that the top compartment of the crate Figure 4 1 contains the following components 1 Rail kit 2 Cabling Guide brackets these brackets are not used for HP Cluster platform installations 3 Screw kit 4 Grounding kit 5 Console cable 6 Power cables optional 7 Packing foam 8 Documentation Figure 4 1 Top ...

Страница 54: ...2 Documentation and CD Location 1 2 Description Item Getting Started Short Guide 1 ISR 9288 Product CD and other CDs according to system configuration 2 5 Open up the front door of the wooden crate by releasing the clamps Figure 4 3 Two people are required to complete the next step Figure 4 3 ISR 9288 Crate Door 54 Installing and Maintaining the ISR 9096 and ISR 9288 Interconnects ...

Страница 55: ...mponents 1 Cabling Guide brackets these brackets are not used for HP Cluster platform installations 2 Small cardboard box for accessories 3 Packing foam 4 Packing list 5 Documentation Figure 4 4 Top Compartment in the ISR 9096 Box 3 4 1 3 2 4 Description Item Accessories box 1 Packing list 2 Cabling brackets 3 Getting Started Short Guide 4 3 Remove the accessories box Figure 4 5 from the crate and...

Страница 56: ...n a separate carton A shipment typically contains the following items in addition to the chassis Generic brackets for rack mounting These brackets are not used in an HP Cluster Platform configuration use only the HP 10000 series rack mounting kit that is provided with the HP Cluster Platform Generic cable guide hooks that are not used in an HP cluster platform installation Use only the cable guide...

Страница 57: ... transfer the modules to the new chassis after you install the chassis in the rack Figure 4 6 shows a front view of the chassis and Figure 4 7 shows the rear port view Figure 4 6 ISR 9288 Front View 1 3 2 4 6 5 7 Description Item Master sMB module 1 sFU 8 fan module 2 sFU 4 fan module 3 sFB 12 fabric board 4 Fabric board latch 5 Slave redundant sMB module 6 L bracket for rack mount applications 7 ...

Страница 58: ...oam grips on either side of the ISR 9XXX chassis 2 Place the ISR 9XXX chassis with the two large foam grips in the crate 3 Insert the two small foam grips on either side of the chassis 4 Place the small cardboard box on top of the ISR 9XXX chassis 5 Place the rail kit in the space allocated for it in the foam 6 Close the box using the four clamps verifying that the box is securely closed 4 4 Mount...

Страница 59: ...ndant power distribution units This section provides step by step instructions for installing the ISR 9XXX chassis Two people are required to remove the interconnect from its box and mount it in a rack due to its weight 4 4 1 Resources Required to Perform an Installation The following materials and tools are required to mount the ISR 9XXX chassis into a rack Flat blade screwdrivers of varying leng...

Страница 60: ...1 Starting at the front of the rack assemble and install the rail kit as follows 1 Determine the correct mounting location for the interconnect in your model of cluster platform The position is specified as a U location which relates to the marked locations on the vertical columns of an HP 10000 series rack Each U location has three holes top middle and bottom Mark the location with masking tape o...

Страница 61: ...e the following procedure to mount the chassis on the assembled rail kit 1 If you are replacing an interconnect chassis bring the cluster to an appropriate state as described in the operating environment documentation 2 Working at the front of the switch remove the pull handles from the long L brackets that are attached to the sides of the chassis The handles are not required 3 Working at the fron...

Страница 62: ...handles should be fully engaged b InfiniBand cables which must be connected according to the port address specifications for your cluster as described in Chapter 7 c Communications cables such as the DB9 serial console cable or a CAT V Ethernet cable d The power cord which you must connect to the correct rack PDU as specified for your model of Cluster Platform Otherwise if the cluster is running y...

Страница 63: ...t A newer 800mm wide cabinet is now used in large InfiniBand configurations eliminating the need to use this procedure Use the following procedure to install the four clamps 1 Identify locations U21 and U23 identified by callout 1 in Figure 4 13 on the inner side of both rear rack columns Mark these locations with a pen or masking tape 2 Clip an M6 cage nut in the back of each location indicated b...

Страница 64: ...k gripping bracket kit Cabling guide bracket kit rack gripping sLB 24 Voltaire 24 4X InfiniBand ports modular Line Board Fibre MediaConverter support sLB 8 12 Voltaire 8 12X InfiniBand ports modular Line Board sRBD Voltaire Router Blade Drawer IPR Voltaire IP Router InfiniBand form factor 4 Optical SFPs included FCI Voltaire FC Router InfiniBand form factor 4 FC ports iSER initiator stack In an HP...

Страница 65: ...ure 4 16 An optional router blade drawer sRBD might be installed in any of the chassis rear slots Figure 4 15 shows the chassis configured with the maximum line ports and no sRBD Figure 4 15 Maximum InfiniBand Ports No sRBD Installed 1 3 2 4 The following features are shown in Figure 4 15 1 Up to 12 sLB 24 InfiniBand line boards 2 Location of the locking levers for inserting and removing the line ...

Страница 66: ... Cluster Platform the ISR 9096 front panel is configured as shown in Figure 4 17 Figure 4 17 ISR 9096 Chassis Front Panel Configuration 1 3 2 4 The following features are shown in Figure 4 17 1 Master sMB module 2 Redundant slave sMB module 3 Up to 4 sFB 4 fabric boards 4 sFU 8 eight fan horizontal cooling module In an HP Cluster Platform the ISR 9096 rear panel is configured as shown in Figure 4 ...

Страница 67: ...ng IP addresses and fabric device names as described in the Voltaire InfiniBand Fabric Management and Diagnostic Guide 4 Verifying the cluster configuration as described in Voltaire InfiniBand Fabric Management and Diagnostic Guide Use the following procedure to verify that the ISR 9XXX starts up properly 1 Check the status of the power supply LEDs on the front panel to ensure that the power suppl...

Страница 68: ...68 ...

Страница 69: ... following Aggregated fabric and resource views Access to a suite of fabric and switch diagnostics Fail over management on all levels Provisioning of InfiniBand fabrics and the attached server Networking and storage resources The management capabilities can be accessed through the command line interface CLI graphical user interface GUI or simple network management protocol SNMP managers or in band...

Страница 70: ...anagement Bracket Installation Guide 5 1 1 Subnet Manager A subnet manager is required to establish the InfiniBand fabric and provide InfiniBand fabric services The subnet manager can be running on the rack mount InfiniBand switches connected to the uplinks of the 4x DDR IB switch modules or on the server blades The Voltaire Grid switch product family is supported to provide subnet manager service...

Страница 71: ...RL controller board Section 6 8 Installing the sFU 4 fan unit Section 6 9 Installing the sFU 8 fan unit Section 6 10 6 1 Guidelines for Inserting and Extracting Boards Observe the following precautions when inserting and extracting boards Wear grounding wrist straps to avoid ESD damage To avoid risk of electric shock do not touch the backplane with your hand or any metal tool Line sFB 12 sMB and o...

Страница 72: ...t the board 4 The hot swap LED When hot swapping a board it is safe to lift the ejector latches only while this LED is illuminated 6 2 Installing or Replacing the sFB 12 Fabric Board You can install up to four fabric boards at the front of the ISR 9288 chassis Figure 6 2 shows the front panel of the fabric board Figure 6 2 The sFB 12 Fabric Board 1 3 2 4 The following features are shown in Figure ...

Страница 73: ...s in place gently When sliding in the board do so by pushing at the center of the board front panel and not by pushing on the ejectors Use the following procedure to insert a fabric board to the chassis 1 Verify that the ejectors are unlocked by pressing the red button 2 Carefully seat the board into the side guide rails 3 Slowly slide the board into the chassis until the ejectors begin to engage ...

Страница 74: ...link Illuminated A logical link is present Off No logical link detected Flashing There is a problem with the logical link Link state amber LED logical link Illuminated All board voltage levels are operating normally Off There is a problem in the power supply to the board Power Green This is a general purpose LED for system management use Various diagnostic procedures will instruct you to check its...

Страница 75: ...FB 12 card An sMB cannot function without an adjacent sFB 12 is missing and you must install one sFB 12 for each sMB that you install Figure 6 4 shows the front panel of the sMB board Figure 6 4 The sMB Management Board 1 3 2 4 5 Figure 6 3 shows the following line board features 1 The board s top ejector latch 2 Red latch release buttons 3 The hot swap status LED 4 The sMB status LEDs labelled as...

Страница 76: ...quence a Top right b Bottom left c Top left d Bottom right 6 Verify the status of the following LEDs a The power LED is illuminated b The info LED is off c The hot swap LED is off Use the following procedure to extract an sMB board from the chassis 1 Disconnect and unsecure all cables 2 Release the security screws 3 Press the red buttons to unlock the ejectors 4 Wait for the blue hot swap LED to i...

Страница 77: ...el 2 Slowly slide the PSU into the chassis until the PSU s bezel is flush with the chassis front panel 3 Secure the PSU bezel screws 4 Connect route and secure the 48V power cable in accordance with the power cabling layout for your model of HP cluster platform Use the following procedure to extract a PSU from the chassis 1 Disconnect power cable after taking careful note of the rack power outlet ...

Страница 78: ... port Table 6 5 lists the IPR LEDs and provides a description of the status signalled by these LEDs Table 6 5 IPR Blade LEDs Description Color and State LED Label Port is operational and Ethernet packets are detected Flashing green Link Activity 1 4 Boot sequence is completed and the system is operational Illuminated green System Boot sequence initiated and in process Flashing green Port is operat...

Страница 79: ...uence initiated and in process Flashing green Fibre Channel driver is loaded and no Fibre Channel link is attached to the port Flashing amber FC Fibre is connected to the port and the link is up Illuminated green Port is operational and a physical link exists between the port and the InfiniBand fabric Illuminated amber InfiniBand Port is operational and has a logical connection to the InfiniBand f...

Страница 80: ...install a populated or unpopulated drawer in the chassis Figure 6 9 shows a populated sRBD Figure 6 9 A Populated sRBD 1 3 2 4 Figure 6 9 shows the following sRBD features 1 Ejector latch incorporating a red unlock button 2 Blade insertion and removal levers 3 Blade modules three per sRBD 4 Security screw Use the following procedure to insert an sRBD to the chassis 1 Verify that the ejectors are u...

Страница 81: ...e or tip of a pen not a pencil until the system reboots Press the reset button for one second to perform a software reset that does not disrupt data traffic over the interconnect Press and hold the reset button for six seconds to reset the entire interconnect which will also disrupt any data traffic that is travelling over the high speed network 7 One of the security screws that secure the sCTRL t...

Страница 82: ...Installing the sFU 4 Fan Module The sFU 4 vertical fan unit directs ambient computer room air over the line boards It contains four variable speed fans A second fan unit the sFU 8 provides cooling for the fabric boards Both units must operate together for normal operation of the interconnect but either can provide short term cooling alone for hot swap operations The speed of the fans varies dynami...

Страница 83: ...cond fan module the sFU 4 provides cooling for the line boards Both modules must operate together but can provide short term cooling alone for hot swap operations The speed of the fans varies dynamically as the fabric board s temperature changes When a fabric board heats up and reaches a preset threshold a sensor detects the over temperature condition and the fans speed up to pass more air over th...

Страница 84: ...it from power Warning Never remove both the sFU 4 and sFU 8 modules at the same time One fan module must be installed and running at all times when the interconnect is operating In the unlikely event that the replacement sFU 8 is defective be prepared to shut the interconnect down quickly to prevent an over temperature condition Use the following procedure to replace a defective fan module by hot ...

Страница 85: ...ation ports when the cluster is assembled at the factory You can connect cables to a chassis that is powered up and has completed all its initialization steps This chapter Contains the following information 4X InfiniBand port cabling See Section 7 1 Administrative connections See Section 7 3 7 1 InfiniBand Cabling InfiniBand Cables have the following characteristics Table 7 1 InfiniBand Cable Char...

Страница 86: ...ocs hp com under the heading High Performance Computing 7 1 3 Connecting InfiniBand Cables Use the following procedure to connect an InfiniBand cable 1 Align the InfiniBand cable connector with the port on a line board 2 Depending on the type of cable in use Squeeze the tabs on either side of the head shell see callout 1 in Figure 7 1 and push the connector onto the port releasing the tabs to lock...

Страница 87: ...loop straps to loom the cables into bundles at the sides of the rack ensuring that you maintain the minimum bend radius 5 When cabling federated ISR 9024 interconnects together use the following procedure a Route and bundle the interconnect to node cables together so that the main bundle exits the bottom left of the rack b Cable the federated leaf to spine interconnects together using cables of 0 ...

Страница 88: ... GbE Ports Use the following procedure to connect a cable to the Gigabit Ethernet port 1 Connect the small form factor SFP GBIC connector to the GbE port on the IPR module 2 Connect the other end of the cable to the appropriate device 7 1 6 2 Connecting to FCR FC Ports Use the following procedure to connect a cable to the Fibre Channel port 1 Connect the small form factor FC 1G 2G 850nm LC transce...

Страница 89: ...racket hooks on the right side as shown by callout 3 in Figure 7 2 4 All cables are routed through the comb brackets and down the sides of the cabinet Note In systems with greater than 128 nodes the new 800mm wide cabinet is used and it is not necessary to route the cables up through the cable clamps Section 4 4 3 page 63 All cables are routed through the comb brackets and down the sides of the ca...

Страница 90: ...he appropriate ProCurve switch port 7 3 2 Cabling the ISR 9096 and the ISR 9288 Management Connections The ISR 9096 and the ISR 9288 management ports are located on the front panel of the sCTRL board The following ports are provided A DB9 serial interface for local console access Use this serial port to connect to a PC running terminal emulation software You can use this connection to run the mana...

Страница 91: ...nection must support VT100 terminal emulation The PC must be equipped with a terminal emulation software such as HyperTerminal or minicom Note Recommend serial terminal application for Windows is HyperTerminal Recommended serial terminal application for Linux is minicom Use the following procedure to connect to the management port serial Interface 1 Connect the end of the management cable with the...

Страница 92: ...92 ...

Страница 93: ...latform Topspin Mellanox SDR PCI X See Section 8 3 Topspin Mellanox SDR PCI Express See Section 8 4 Mellanox SDR PCI X See Section 8 5 Mellanox SDR PCI Express See Figure 8 6 Mellanox Memory Free SDR PCI Express See Section 8 7 Mellanox Memory Free DDR PCI Express See Section 8 8 Mellanox DDR PCI Express See Section 8 9 HP HPC 4x DDR IB Mezzanine HCA See Section 8 9 2 Note Specific combinations of...

Страница 94: ...h the etched circuit tracks Caution Do not mix single data rate SDR components with double data rate DDR components as this may damage the components or the cluster or both 8 2 Installing the HCA The installation steps described in this section are generic because the actual procedure is specific to server models that are used as nodes in the cluster Clusters might contain only one server model ho...

Страница 95: ... server before the server is mounted in the rack Figure 8 2 shows a typical installation procedure Figure 8 2 Typical HCA 400 PCI Card Installation 1 3 2 4 3 The following installation features are identified by the numbered callouts in Figure 8 2 1 The PCI X installation slot is defined for each model of server Always used the specified slot to obtain maximum performance Do not add additional car...

Страница 96: ...twork connections 4 Remove any cable management parts that prevent you from extending the server from the rack 5 Depending on the requirements for access to the server s PCI slots either remove the server from the rack or remove the required panels 6 Identify the correct Installation slot Always install the HCA in the PCI slot that is specified for your model of server This slot is usually a PCI X...

Страница 97: ... No physical link detected Flashing There is a problem with the physical link Green The amber LED logical link displays the following status Illuminated A logical link is present Off No logical link detected Flashing There is a problem with the logical link Amber After installing the HCA your first task is usually to determine that the its link is correctly cabled and functioning Consult the inter...

Страница 98: ...ellanox PCI Express HCA The Topspin Mellanox PCI Express HCA supports InfiniBand protocols including IPoIB SDP SRP UDAPL and MPI The Topspin Mellanox PCI Express HCA is a single data rate SDR card with two 4X InfiniBand 10 Gb s ports and 128 MB local memory Figure 8 4 shows a Topspin Mellanox PCI Express HCA Figure 8 4 Topspin Mellanox SDR PCI Express HCA Features of the Topspin Mellanox PCI Expre...

Страница 99: ... up to 133 MHz Supports InfiniBand protocols Installation of the Mellanox PCI X HCA is similar to that of the Voltaire HCAs described previously in this chapter 8 5 1 Mellanox PCI X SDR HCA LEDs The board has four LEDs located on the I O panel 2 LEDs per 4X port The physical link green illuminates once VAPI InfiniBand Verbs API is started and a physical connection is made between two nodes The dat...

Страница 100: ...mory PCI Express x8 edge connector I O Panel LEDs I2C compatible connector for debug Supports InfiniBand protocols Installation of the Mellanox card is similar to that of the Voltaire HCAs described previously in this chapter 8 6 1 Mellanox PCI Express SDR HCA LEDs The board has four LEDs located on the I O panel 2 LEDs per 4X port The physical link green illuminates once VAPI InfiniBand Verbs API...

Страница 101: ...8 7 Mellanox Memory Free PCI Express HCA SDR Features of the Mellanox Mem Free PCI Express HCA include PCI Express x8 version 1 0a compatible card Single 4X InfiniBand port Version 1 2 compatible Host Channel Adapter InfiniBand Compatible Verbs API interface for both Linux and Windows operating systems 4X 10 Gb s InfiniBand port with standard copper connector Hardware support for up to 16 million ...

Страница 102: ...eing passed If the LEDs are not active either the physical or the logical or both connections have not been established LED Name 4X Port Physical Link Green Port 1 Logical Link Yellow Installation of the Mellanox card is similar to that of the Voltaire HCAs described previously in this chapter 8 8 Mellanox Memory Free PCI Express HCA DDR The Mellanox Memory Free PCI Express HCA supports InfiniBand...

Страница 103: ...iBand Verbs API is started and a physical connection is made between two nodes The data activity link yellow illuminates once the InfiniBand network is discovered over the physical link The activity link is a steady yellow when it is discovered but no data is being passed The activity link blinks when data is being passed If the LEDs are not active either the physical or the logical or both connec...

Страница 104: ...four LEDs located on the I O panel 2 LEDs per 4X port The physical link green illuminates once VAPI InfiniBand Verbs API is started and a physical connection is made between two nodes The data activity link yellow illuminates once the InfiniBand network is discovered over the physical link The activity link is a steady yellow when it is discovered but no data is being passed The activity link blin...

Страница 105: ...ons Figure 8 10 shows a 4x DDR IB Mezzanine HCA Figure 8 10 4x DDR IB Mezzanine HCA To obtain the best performance plug the 4x DDR IB Mezzanine HCA into the x8 PCI Express connector on the server blade The 4x DDR IB Mezzanine HCA is connected to the 4x DDR Switch Module see Chapter 5 page 69 in the c Class switch bay through the c Class enclosure midplane Multiple HCAs might be plugged into availa...

Страница 106: ...106 ...

Страница 107: ... are usually isolated to a single component and are more difficult to isolate than a problem with a subsystem When troubleshooting first test each separate subsystem in the ISR 9288 since there are fewer subsystems than components The ISR 9XXX chassis consists of the following subsystems The power supplies operate whenever rack power is connected The chassis fan modules operate when the system pow...

Страница 108: ... creates an event log file for both IB traps and SM internal events Examine the log to find problem ports Refer to the Voltaire InfiniBand Fabric Management and Diagnostic Guide and the HP Cluster Platform InfiniBand Fabric Management and Diagnostic Guide for information on using the PM and interpreting the log files 9 4 Identifying a Leaf or Spine Port Malfunction on an ISR 9XXX You can identify ...

Страница 109: ...t connected In future polls or initialization events the port is not discovered unless the problem is corrected and you clear the entry in the failed port table Use the following procedure to display and clear failed ports 1 To display the bad port table select the following option from the device manager s main menu Device Bad Ports Log A browser window is displayed showing the bad ports 2 To cle...

Страница 110: ...110 ...

Страница 111: ... and ports ndicators unit status subnet management Linear forwarding table 48K entries Switch Specifications Multicast table size 1K entries Data virtual lanes 8 MTU 4096 bytes maximum Power and Cooling One or two factory installed Power Supplies Supply Optional second Power Supply in case single Power Supply is factory installed Supply option with optical adapters 55 Watt maximum Wattage Internal...

Страница 112: ...y managed Environmental Operating 15 to 80 non condensing Humidity 0 to 9843 ft 3000m Altitude Environmental Storage 13 F to 185 F 25 C to 85 C Temperature 5 to 90 non condensing Humidity 0 ft to 15 000 ft 4570 m Altitude 112 ISR 9024 Interconnect Specifications ...

Страница 113: ...ed devices On board processor running GridVision fabric and device management software with GUI CLI and SNMP on ISR 9024S D M only Connectors EIA TIA 232 console and RJ 45 Ethernet on ISR 9024S M and ISR 9024D M only Aggregate Data Throughput 960 Gb s DDR or 480 Gb s SDR Switch Specifications Port to port Latency 140 nanoseconds max Linear Forwarding Table 48K entries Multicast table size 1K entri...

Страница 114: ...30 mm x 20 59 523 mm Dimensions H x W x D 19 482 6 mm with optional front or rear rack mounting Rack column width 17 lb 7 7 kg excluding rack mounting Weight Reliability over 160 000 hours Internally managed over 200 000 hours Externally managed Environmental Operating 32 F to 113 F 0 C to 45 C Temperature 15 to 80 non condensing Humidity 0 to 9843 ft 3000m Altitude Environmental Storage 13 F to 1...

Страница 115: ...3 2 00 EN61000 3 3 95 EMC This device complies with par 15 of FCC rules Operation is subject to the following two conditions 1 This device may not cause harmful interference 2 this device must accept any interference received VCCI A ICES 003 C Tick RoHS5 RoHS 115 ...

Страница 116: ...116 ...

Страница 117: ...power and info LEDs Hot pluggable Media Converter for single optical cable Line Board One to four per system Indicators Physical connectivity and logical connectivity LEDs per Line Board link port Power Info LEDs Hot swap LEDs Fabric Board VoltaireVision including Fabric Management Chassis and Device Management Network Traffic Management Storage Resource Management InfiniBand managers InfiniBand A...

Страница 118: ... activity LEDs system LED and management link LEDs Blade reset button Router Blade Drawer Connectors EIA TIA 232 Console DB 9 and 10 100Ethernet RJ45 Indicators Subnet manager activity 2 LEDs Chassis management activity 2 LEDs Fabric boards 4 LEDs Temperature LED Management device reset button Control rear Consumption is 1000W maximum full configuration Operating voltage 85 265V AC 50 60 Hz auto s...

Страница 119: ...d hot swap LEDs on boards Fabric Board Two hot swappable sMB board including a CPU Mezzanine On board Temperature monitoring reported to optional Management system InfiniBand managers Subnet manager Subnet administrator InfiniBand Agents Subnet manager agent Performance manager agent Baseboard manager agent Supports IETF Standard SNMPv2 Provides the following connectors 10 100 Ethernet RJ45 EIA TI...

Страница 120: ... LEDs Four GbE link activity LED System LED Management link LEDs Blade reset button Router Blades Connectors EIA TIA 232 Console DB 9 and 10 100Ethernet RJ45 Indicators Subnet manager activity 2 LEDs Chassis management activity 2 LEDs Fabric boards 4 LEDs Temperature LED Management device reset button Control rear Consumption is 2 500W maximum full configuration Operating voltage 85 240V AC 50 60 ...

Страница 121: ...Weight Ambient Operating Temperature 32 to 113 F 0 to 45 C Operating Humidity 15 to 80 non condensing Operating Altitude 0 to 9843 ft 3000m Storage Temperature 13 to 158 F 25 to 70 C Storage Humidity 5 to 90 non condensing Storage Altitude 0 to 15 000 ft 4570m Environmental 121 ...

Страница 122: ...122 ...

Страница 123: ... non blocking 4X DDR switching with 16 downlinks and 8 uplinks Data path 15 3 x 10 6 inches Dimensions LxW Power and Environmental Specifications 0 to 400 C Operating Temperature 5 to 85 Operating Humidity non condensing 40 to 700 C Non operating Temperature 5 to 95 Non operating Humidity non condensing 4 0 A at 12V max 48 W Power requirement FCC CFR 47 Part 15 Class A Emissions Classifications CI...

Страница 124: ...124 ...

Страница 125: ...s of the HCA 400 are as follows Specification Feature Dual port 4X 10 Gb s InfiniBand PCI Express low profile host channel adapter PCI Express 1 0 4X or 8X HCA 400Ex format and bus interface Dual port 4X 10 Gb s InfiniBand PCI PCI X low profile host channel adapter PCI X 1 0 3 3V 64 bit 133 MHz PCI 2 2 32 64 bit 33 66 MHz HCA 400 format and bus interface InfiniBand v1 1 compatible design InfiniBan...

Страница 126: ...sor General Specifications 4x 10 Gb s per port 40 Gb s aggregate full duplex Line rate bandwidth half duplex and full duplex 7 Gb s Observed bandwitdh configuration dependant 6 μs Latency configuration dependant 128 MB On board memory 64 bit 133Mhz PCI X Bus Dual InfiniBand 4x copper Ports Two InfiniBand 4x copper Connectors Up to 15 m 50 ft with 4x InfiniBand copper cabling Cabling 6 6 x 2 5 in 1...

Страница 127: ...ion dependant 128 MB On board memory 64 bit 133Mhz PCI X Bus Dual InfiniBand 4x copper Ports Two InfiniBand 4x copper Connectors Up to 15 m 50 ft with 4x InfiniBand copper cabling Cabling 6 6 x 2 5 in 16 77 cm x 6 40 cm without bracket Dimensions LxW 32 to 131 F 0 to 55 C Operating Temperature Power and Environmental 5 to 85 non condensing Operating Humidity 40 to 149 F Non operating Temperature 5...

Страница 128: ... 03 Class B EN 55024 1998 A1 01 A2 03 IEC EN 60950 1 2001 Safety ETSI EN 300 019 2 2 IEC 60068 2 64 29 32 Environmental F 5 Mellanox PCI Express HCA Specifications The specifications listed below cover the Mellanox PCI Express SDR HCA 2 5in x 6 6in Size Physical 200 LFM 55 C Air Flow InfiniBand Copper 4X 20 Gb s Connector 12V 3 3V Voltage Power and Environmental 10W Maximum Power 0 to 55 Celsius T...

Страница 129: ...15 subpart B ICES 003 EMC Regulatory VCCI IEC EN 55022 IEC EN 55024 IEC EN 61000 IEC EN 60950 Safety F 7 Mellanox Memory Free PCI Express HCA DDR Specifications The specifications listed below cover the Mellanox Mem Free PCI Express HCA DDR 54mm x 102mm 2 13 in x 4 in Size Physical 200 LFM 55 C Air Flow Amphenol InfiniBand MicroGigaCN 20 Gb s Connector 3 3V Voltage Power and Environmental 4 2W Max...

Страница 130: ...1 A2 03 IEC EN 60950 1 2001 Safety ETSI EN 300 019 2 2 IEC 60068 2 64 29 32 Environmental F 9 HPC 4x DDR IB Mezzanine HCA Specifications The specifications listed below cover the HPC 4x DDR IB Mezzanine HCA 4x DDR IB Mezzanine HCA Specifications PCI Express revision 1 0a Compliance IBTA version 1 2 compatible ROHS R5 General Specifications MT25204A0 FCC D InfiniHost III Lx Communications Processor...

Страница 131: ...mperature 5 to 95 Non operating Humidity non condensing 1 35 A at 3 3V max 4 5W Power requirement FCC CFR 47 Part 15 Class A Emissions Classifications CISPR 22 Class A ICES 003 Class A VCCI Class A ACA CISPR 22 Class A F 9 HPC 4x DDR IB Mezzanine HCA Specifications 131 ...

Страница 132: ...132 ...

Страница 133: ...ector head shell 85 guidelines 85 cable management hooks ISR 9288 62 cabling 85 administrative cabling 90 Ethernet tables 85 ISR 9024 administrative 90 ISR 9096 administrative 90 ISR 9288 administrative 90 sCTRL 90 sMB 90 tables 85 cabling clamps rack mounting 63 cabling procedure 86 87 88 cage nut usage 35 45 card PCI 93 caution information 14 cellular telephones 16 channel based 19 chassis ISR 9...

Страница 134: ...40 InfiniBand 19 management 26 topology 26 fabric board 24 25 50 see also sFB see also sFB 12 see also sFB 4 fabric failure debugging 108 fabric topology 29 fan unit SFU 2 24 see also sFU 2 fan unit SFU 4 26 see also sFU 4 fan unit SFU 8 24 26 see also sFU 8 fan units ISR 9096 51 ISR 9288 51 fast interface 26 fat tree 22 25 49 fat tree network 19 fault tolerance 19 FCR ISR 9288 78 LEDs 79 FCR blad...

Страница 135: ...erations 15 Installing ISR 9096 49 ISR 9288 49 installing cabling clamps 63 HCA 94 inter integrated circuits 30 see also I C interconnect cabling 85 interconnect administration 26 see also management internally managed 29 39 internet protocol router 24 26 77 see also IPR IPR ISR 9288 77 LEDs 78 IPR blade 24 26 ISR 9024 12x ports 30 administrative cabling connections 90 cabling procedure 86 chassis...

Страница 136: ... management agent 52 unpacking 55 ISR 9096 management 26 see also management ISR 9288 administrative cabling connections 90 architecture 50 backplane 51 building blocks 50 cable management hooks 62 cabling clamps 63 cabling procedure 88 control panels 52 fabric board 50 fan units 51 FCR swapping 78 features 49 firmware 51 front view 57 FRU 71 handling boards 71 identifying 24 Installing and Mainta...

Страница 137: ...er 26 management board 24 25 50 see also sMB management card ISR 9024 31 ISR 9024S D M 41 management information base 26 see also MIB manangement sMB board 24 25 master mode 30 40 media access control 30 see also MAC MediaConverter 50 Mellanox PCI X SDR HCA 99 Mellanox Mem Free PCI Express DDR HCA specifications 129 Mellanox Mem Free PCI Express SDR HCA specifications 129 mellanox memory free PCI ...

Страница 138: ...er supply unit 24 26 see also PSU powered screwdriver 71 ProCurve switch 90 protocol 20 InfiniBand 19 PrPMC 30 see also PCI mezzanine card PS LED 37 47 PSU 24 26 50 Installing in ISR 9024 S D 46 Installing in ISR9024 36 ISR 9024 30 ISR 9096 51 ISR 9288 51 76 LEDs 77 Q queue pairs 20 R rack mounting location 35 44 rack 10000 component safety 15 safety 15 rack kit ISR 9024 34 ISR 9024 S D 44 rack mo...

Страница 139: ... ISR 9288 75 LEDs 75 sMB management board 24 25 SNMP 26 specifications 4x DDR IB mezzanine HCA 130 4x DDR IB switch module 123 HCA 125 ISR 9024 111 ISR 9024 S D 113 ISR 9096 117 ISR 9288 119 Mellanox Mem Free PCI Express DDR HCA 129 Mellanox Mem Free PCI Express SDR HCA 129 Mellanox PCI Express DDR HCA 130 Mellanox PCI Express SDR HCA 128 Mellanox PCI X HCA 128 Topspin Mellanox PCI Express HCA 127...

Страница 140: ...s 108 U U location 35 44 unpacking ISR 9024 33 ISR 9024 S D 43 ISR 9096 55 ISR 9288 53 unreliable datagram 20 V verifying shipment ISR 9288 56 vertical fan 51 vertical fan unit sFU 2 50 sFU 4 50 virtual lanes 19 virtual router 50 Voltaire Web site 12 VoltaireVision device manager 26 VoltaireVision fabric manager 26 W warning hazard 13 warning information 14 Web browser 26 Web site login Voltaire 1...

Результаты поиска

Содержание Cluster Platform Express v2010

Отзывы:

Похожие инструкции для Cluster Platform Express v2010

Бренды по названию

Популярные бренды

HP Cluster Platform Express v2010, Руководство пользователя

Результаты поиска

Содержание Cluster Platform Express v2010

Отзывы:

Похожие инструкции для Cluster Platform Express v2010

Бренды по названию

Популярные бренды