IBM NeXtScale System Page 118 Planning And Implementation Manual Download

Page: 118 / 266

100

IBM NeXtScale System Planning and Implementation Guide

More network adapters are offered as part of the IBM Intelligent Cluster program,
as listed in Table 4-26.

Table 4-26 Network adapters that are offered as part of the Intelligent Cluster program

47C9977

A522

Solarflare SFN7122F 2x10GbE SFP+ Flareon Ultra for IBM System x

Gigabit Ethernet

90Y9352

A2V3

Broadcom NetXtreme I Quad Port GbE Adapter for IBM System x

90Y9370

A2V4

Broadcom NetXtreme I Dual Port GbE Adapter for IBM System x

49Y4230

5767

Intel Ethernet Dual Port Server Adapter I340-T2 for IBM System x

49Y4240

5768

Intel Ethernet Quad Port Server Adapter I340-T4 for IBM System x

InfiniBand

00D9550

A3PN

Mellanox ConnectX-3 FDR VPI IB/E Adapter for IBM System x

Part number

Feature code

Description

Part number

Feature code

Description

40 Gb Ethernet

46W0620

A4H5

Chelsio T580-LP-CR Dual-port (QSFP+) 40GbE PCI-E 3.0 Adapter

95Y3459

A2F8

Mellanox ConnectX-3 EN Dual-port QSFP+ 40GbE Adapter

10 Gb Ethernet

00AE047

A4K1

Mellanox ConnectX-3 EN Single-port SFP+ 10GbE Adapter

00W0053

A2ZQ

Mellanox ConnectX-3 EN Dual-port SFP+ 10GbE Adapter

00Y7006

A3G7

Brocade 1860 Dual-port SFP+ 10GbE Fabric Adapter

00Y7026

A3GE

Brocade 1860 Single-port SFP+ 10GbE Fabric Adapter

46W0609

A4H3

Chelsio T520-LL-CR Dual-port (SFP+) 10GbE PCI-E 3.0 Adapter

46W0615

A4H4

Chelsio T540-CR Quad-port (SFP+) 10GbE PCI-E 3.0 Adapter

InfiniBand

00W0037

A2YE

Mellanox ConnectX-3 VPI Single-port QSFP FDR IB HCA

00W0041

A2YF

Mellanox ConnectX-3 VPI Dual-port QSFP FDR IB HCA

46W0571

A44E

Mellanox Connect-IB Dual-port QSFP FDR IB PCI-E 3.0 x16 HCA

59Y1888

5763

Intel QLE7340 single-port 4X QDR IB x8 PCI-E 2.0 HCA

Summary of Contents for NeXtScale System

Page 1: ...lementation Guide David Watts Jordi Caubet Duncan Furniss David Latino Introduces the new high density x86 solution for scale out environments Covers the n1200 Enclosure and nx360 M4 Compute Node Addr...

Page 2: ......

Page 3: ...IBM NeXtScale System Planning and Implementation Guide July 2014 International Technical Support Organization SG24 8152 01...

Page 4: ...s Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp Second Edition July 2014 This edition applies to IBM NeXtScale n1200 Enclosure machine type 5456 IBM NeXtScale nx3...

Page 5: ...1 2 1 IBM NeXtScale n1200 Enclosure 3 1 2 2 IBM NeXtScale nx360 M4 compute node 4 1 3 Design points of the system 6 1 4 This book 7 Chapter 2 Positioning 9 2 1 Market positioning 10 2 1 1 Three key m...

Page 6: ...ale nx360 M4 compute node 59 4 1 Overview 60 4 1 1 Physical design 61 4 2 System architecture 63 4 3 Specificiations 67 4 4 Standard models 69 4 5 Processor options 70 4 6 Memory options 71 4 6 1 DIMM...

Page 7: ...Fibre Channel switches 145 5 7 Rack level networking Sample configurations 146 5 7 1 Non blocking InfiniBand 147 5 7 2 50 blocking InfiniBand 148 5 7 3 10 Gb Ethernet one port per node 149 5 7 4 10 G...

Page 8: ...20 8 3 1 IBM GPFS FPO 222 8 3 2 IBM System x GPFS Storage Server 224 8 4 IBM Platform LSF family 227 8 5 IBM Platform HPC 229 8 6 IBM Platform Symphony family 232 8 7 IBM Parallel Environment for x86...

Page 9: ...materials for this IBM product and use of those websites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any oblig...

Page 10: ...ountries or both AIX BladeCenter Blue Gene GPFS IBM IBM Flex System iDataPlex Intelligent Cluster LoadLeveler LSF POWER Power Systems PowerPC PureFlex RackSwitch Redbooks Redpaper Redbooks logo Server...

Page 11: ...the innovations in its design outlines its benefits and positions it with other IBM x86 servers The book provides details about NeXtScale System components and the supported options It also provides...

Page 12: ...sight for the implementation of many large scale solutions for HPC distributed databases and rendering of computer generated images He is an IBM Certified IT Specialist and member of the IT Specialist...

Page 13: ...n Paget Scott Tease Swarna Tsai Matt Ziegler From IBM Development David Brenchley Vincent Chao Kelly Chen Jason Cheng Marty Crippen Chris Hsieh Christina Hsu Jim Huang Cathy Lin Bruce Smith Brad Taylo...

Page 14: ...ength and you can participate either in person or as a remote resident working from your home base Find out more about the residency program browse the residency index and apply online at this website...

Page 15: ...ibmredbooks Look for us on LinkedIn http www linkedin com groups home gid 2130806 Explore new Redbooks publications residencies and workshops with the IBM Redbooks weekly newsletter https www redbooks...

Page 16: ...xiv IBM NeXtScale System Planning and Implementation Guide...

Page 17: ...tion Added 1300W power supply efficiency values Table 3 10 on page 40 Added tables showing quantities of compute nodes supported based on processor selection power supply selection and input voltage 3...

Page 18: ...XtScale System Planning and Implementation Guide New RDIMM memory options Support for 2 5 inch SSD options and other drive options Support for ServeRAID M5120 RAID controller for external SAS storage...

Page 19: ...ribe the client requirements that led us to its design the computing environment it is meant to work in and how this architecture was created to meet current and future business and technical challeng...

Page 20: ...r of servers in clusters grows and as data center real estate cost increases the number of servers in a unit of space also known as the compute density becomes an increasingly important consideration...

Page 21: ...or billing Analytics Distributed databases and extensive use of data mining or analytics is another use case that is increasing in prevalence and is being applied to a greater range of business and t...

Page 22: ...wo sources of three phase power Also in the rear of the chassis is the Fan and Power Controller which controls power and cooling aspects of the chassis 1 2 2 IBM NeXtScale nx360 M4 compute node The fi...

Page 23: ...el Xeon E5 2600 v2 series processors eight DDR3 DIMMs and a hard drive carrier Hard disk drive carrier options include one 3 5 inch drive two 2 5 inch drives or four 1 8 inch solid state drives The se...

Page 24: ...natively you can have IBM install them into racks with switches and power distribution units and connect all the cables IBM factory integration Further configuration and testing are done when the syst...

Page 25: ...d power controller FPC Next we take a broader view covering implementations at scale reviewing racks and cooling We then describe IBM s process for assembling and testing complete systems in to Intell...

Page 26: ...8 IBM NeXtScale System Planning and Implementation Guide...

Page 27: ...rketplace compared with other systems that are equipped with Intel processors The information helps you to understand the NeXtScale target audience and the types of workloads for which it is intended...

Page 28: ...including SSCT Blue Horizon and x config IBM NeXtScale System includes the following key features Supports up to seven chassis1 in a 42U rack which means up to a total of 84 systems and 2 016 process...

Page 29: ...lution can increase the speed of outcome prediction engineering analysis and design and modeling 2 1 1 Three key messages with NeXtScale The three key messages about IBM NeXtScale System is that it is...

Page 30: ...e V2 Dynamic Rack because it provides the best cabling features However the chassis can also be installed in many third party four post 19 inch racks This ensures maximum flexibility when it comes to...

Page 31: ...h cart access Also each compute node has a pull out tab at the front for system and customer labeling needs as shown in Figure 2 2 Figure 2 2 Front of the IBM NeXtScale nx360 M4 Because the cables do...

Page 32: ...w enormously NeXtScale is designed to be run and managed at scale as a single solution NeXtScale System is built on what we learned about the financial aspects of scale out Every decision that we made...

Page 33: ...HPC and technical computing have many of the same attributes as cloud a key factor is the need for the top bin 130 W processors NeXtScale System can support top bin 130 W processors which means more c...

Page 34: ...et and up systems that offer unprecedented x86 performance resiliency and security Blades and integrated systems Integrated systems is a fast growing market where IBM adds value by packaging IBM softw...

Page 35: ...1200 Enclosure is designed to fit in the IBM 42U 1100mm Enterprise V2 Dynamic Rack but it also fits in many standard 19 inch racks Although iDataPlex can also be installed in a standard 19 inch rack t...

Page 36: ...servers that can be fitted in a single row of racks while using top of rack switches We use a single iDataPlex rack for iDataPlex servers that is 1200 mm wide and we compare it with two standard racks...

Page 37: ...be the perfect server for clients that require a scale out infrastructure NeXtScale System uses industry standard components including I O cards and top of rack networking switches for flexibility of...

Page 38: ...with shared power and cooling that is designed for multiple generations of technology 6U chassis holding 12 servers with shared power and cooling designed for multiple generations of technology Hetero...

Page 39: ...ating cost through higher density up to 4X compared to 2U servers and higher energy efficiency because of the shared power and cooling infrastructure 2 6 Ordering and fulfillment We put careful attent...

Page 40: ...guration ordering and pricing tools Tool System x iDataPlex NeXtScale Presence in SSCT Blue Horizon x config Yes Yes Yes Special bid in leads Yes Yes Yes Available in IBM Intelligent Cluster Yes Yes Y...

Page 41: ...Adding compute storage or acceleration capability is as simple as adding nodes to the chassis There is no built in networking or switching capabilities which requires no chassis level management beyon...

Page 42: ...space that is compared to traditional 1U rack servers Figure 3 1 IBM NeXtScale n1200 Enclosure with 12 compute nodes The founding principle behind IBM NeXtScale System is to allow clients to adopt th...

Page 43: ...200 Enclosure includes the following components Up to 12 compute nodes Six power supplies each separately powered A total of 10 fan modules in two cooling zones One Fan and Power Controller Shipping b...

Page 44: ...NeXtScale n1200 Enclosure Front View with 12 compute nodes This new enclosure not only supports dense high performance compute nodes but also expanded compute nodes with more I O slots for adapters GP...

Page 45: ...re 3 4 IBM NeXtScale n1200 Enclosure rear view At the rear of the chassis the following types of components are accessible Power supplies The IBM NeXtScale n1200 Enclosure has a six power supply desig...

Page 46: ...ity and keeps the IBM NeXtScale n1200 Enclosure flexible and low cost 3 1 3 Fault tolerance features The chassis implements a fault tolerant design The following components in the chassis enable conti...

Page 47: ...components such as the drive cage which is mounted on the rear of the chassis One AC power cord for each power supply installed 1 5m 10A IEC320 C14 to C13 part number 39Y7937 3 3 Supported compute no...

Page 48: ...235 W GPUs Table 3 5 on page 34 200 240V AC input GPU Trays with 300 W GPUs Table 3 6 on page 35 900 W power supplies 200 240V AC input no GPU Trays Table 3 7 on page 36 100 127V AC input no GPU Trays...

Page 49: ...for more efficient use of the available system power Compute nodes Power policy 6 x 1300W power supplies CPU TDP Number of CPUs Non redundant or N 1 with OVSa N 1 N N N N with OVSa Note Some cells in...

Page 50: ...mber of CPUs Non redundant or N 1 with OVSa a OVS oversubscription of the power system allows for more efficient use of the available system power N 1 N N N N with OVSa High line AC input 200 240 V 50...

Page 51: ...of compute nodes supported each with two 225 W GPUs installed 6 x 1300 W PSUs Compute nodes Power policy 6 x 1300W power supplies CPU TDP Number of CPUs Non redundant or N 1 with OVSa a OVS oversubsc...

Page 52: ...with two 235 W GPUs installed 6 x 1300 W PSUs Compute nodes Power policy 6 x 1300W power supplies CPU TDP Number of CPUs Non redundant or N 1 with OVSa a OVS oversubscription of the power system allo...

Page 53: ...0 W GPUs installed 6 x 1300 W PSUs Compute nodes Power policy 6 x 1300W power supplies CPU TDP Number of CPUs Non redundant or N 1 with OVSa a OVS oversubscription of the power system allows for more...

Page 54: ...with 6 x 900 W PSUs Compute nodes Power policy 6 x 900W power supplies CPU TDP Number of CPUs Non redundant or N 1 with OVSa a OVS oversubscription of the power system allows for more efficient use o...

Page 55: ...Input with 6 x 900 W PSUs Compute nodes Power policy 6 x 900W power supplies CPU TDP Number of CPUs Non redundant or N 1 with OVSa a OVS oversubscription of the power system allows for more efficient...

Page 56: ...supported power supplies Table 3 9 Power supplies The power supply options have the following features Supports N N or N 1 Power Redundancy or Non redundant power configurations to support higher den...

Page 57: ...ering The power supplies that are used in IBM NeXtScale System are hot swap high efficiency 80 PLUS Platinum power supplies that are operating at 94 efficiency The efficiency varies by load as shown i...

Page 58: ...le than is needed to provide full power to all chassis components AC redundancy is achieved by distributing the AC power cord connections between independent AC circuits For more information see 5 1 P...

Page 59: ...D is lit green it indicates that DC power is supplied from the power supply to the chassis midplane Fault LED When this LED is lit yellow it indicates that there is a fault with the power supply Remov...

Page 60: ...modules have a dual rotor design for high efficiency and high reliability Air flow is front to back Ordering information for the fan modules is shown in Table 3 11 Table 3 11 Fan Modules Part number...

Page 61: ...e fan module failed The fan modules are not dedicated to cool a specific node If there is a fan module failure the remaining functional fans speed up if required under the control of the FPC to provid...

Page 62: ...values that are required to cool those nodes The FPC varies the speeds of the fans in zone 1 and zone 2 by at most a 20 difference to avoid unbalanced air flow distribution Fan removal To maintain pr...

Page 63: ...ont and rear view of the n1200 Enclosure Midplane Assembly The midplane is used to provide power to all elements in the chassis It also provides signals to control fan speed power consumption and node...

Page 64: ...e Figure 3 11 shows the connectivity of the chassis components through the midplane Figure 3 11 Midplane connectivity Midplane Fan and Power Controller Servers Power Supplies Fan Modules Power Supplie...

Page 65: ...ower permission to each node and controls the speed of the fans The FPC is installed inside the chassis and is accessible from the rear of the chassis as shown in Figure 3 12 The FPC is a hot swap com...

Page 66: ...n this LED is lit green it indicates that the FPC has power Heartbeat LED When this LED is lit green it indicates that the FPC is actively controlling the chassis Locator LED When this LED is lit blue...

Page 67: ...onnector is connected to the management network through a top of rack switch 3 7 2 Internal USB memory key The FPC also includes a USB key that is housed inside the unit as shown in Figure 3 14 Figure...

Page 68: ...mps up fan speeds if conditions require more cooling or slow down the fans to conserve energy if higher fan speeds are not required Provides the following user interfaces Web interface IPMI command li...

Page 69: ...iption Select N N 1 or N N mode Enable or disable oversubscription mode View and set power capping Node level Set value within a defined range for each node separately or choose between one of the thr...

Page 70: ...power usage that is based on power maximizer test The following values are generated Maximum power usage value under stressed condition Maximum power usage value under stressed condition when P state...

Page 71: ...ode power consumption is capped at an assigned level When it is applied to a chassis the whole chassis power usage is capped When power saving is enabled an individual node or all nodes chassis level...

Page 72: ...This means we can count on the power of six power supplies instead of five for normal operation When oversubscription mode is enabled with redundant power N 1 or N N redundancy the total available po...

Page 73: ...t P state to reduce their power usage back to a supported range By design the compute nodes perform this action quickly enough and operation continues The Table 3 13 shows the consequences of redundan...

Page 74: ...speeds change as required for optimal cooling When the option is set to On the chassis offers the following set points where the fan speeds are capped 1 highest acoustics attenuation lowest cooling 2...

Page 75: ...tts Full configuration 20 470 84 Btu hr 6 000 watts Declared sound power level 7 5 bels Chassis airflow Full chassis configuration with all compute nodes FPC power supplies and fan modules installed M...

Page 76: ...e 40 C to 60 C 40 F 140 F Altitude 10700 m 35 105 ft Relative humidity 5 100 Maximum dew point 29 C 84 2 F 6 3 5 C per hour for data centers that use tape drives and 20 C per hour for data centers tha...

Page 77: ...e computing power per watt and the latest Intel Xeon processors you can reduce costs while maintaining speed and availability A total of 12 nx360 M4 servers can be installed into the 6U NeXtScale n120...

Page 78: ...ntains only the essential components in the base architecture to provide a cost optimized platform The nx360 M4 compute node provides a dense flexible solution with a low total cost of ownership Figur...

Page 79: ...server with up to 32 TB of total disk capacity within 1U of comparable rack density A slot for 10 Gb Ethernet or FDR InfiniBand mezzanine for network connectivity without using a PCIe slot PCI Express...

Page 80: ...y components inside the server Figure 4 3 Inside view of the IBM NeXtScale nx360 M4 CPU 2 and four DIMMs CPU 1 and four DIMMs Mezz card connector PCIe 3 0 riser slot 1 PCIe 3 0 riser slot 2 PCIe Tray...

Page 81: ...E5 2600 v2 series processors formerly known by the Intel code name Ivy Bridge EP are the successors of the first implementation of Intel s micro architecture that is based on tri gate transistors Inte...

Page 82: ...d power management capabilities Turbo Boost automatically turns off unused processor cores and increases the clock speed of the cores in use if thermal requirements are still met Turbo Boost Technolog...

Page 83: ...es TDP values W 130 115 96 80 70 60 50 W Idle power targets W 15 W or higher 12 W for low voltage SKUs 10 5 W or higher 7 5 W for low voltage SKUs Xeon E5 2600 Sandy Bridge EP Xeon E5 2600 v2 Ivy Brid...

Page 84: ...IMMs A PCI Express 3 0 x8 slot for Mezzanine card connection that connects to CPU 1 A PCI Express 3 0 x24 slot for PCIe riser cage that provides a full height half length FHHL slot that connects to CP...

Page 85: ...ng DDR3 DIMMs up to 1866 MHz memory speeds RDIMMs UDIMMs and LRDIMMs supported Four memory channels per processor one DIMM per channel Memory maximums Up to 256 GB with 8x 32 GB LRDIMMs and two proces...

Page 86: ...Power Controller Module for chassis management Gb Ethernet connection RJ45 for browser based remote management mini USB serial port for local management Cooling Supplied by the NeXtScale n1200 enclos...

Page 87: ...dth 447 mm 17 6 in height 263 mm 10 4 in depth 915 mm 36 in Weight NeXtScale nx360 M4 maximum weight 6 05 kg 13 31 lb NeXtScale n1200 enclosure Fully configured stand alone 112 kg 247 lb empty chassis...

Page 88: ...0MHz 80W 00FL131 A55R A55Z Intel Xeon E5 2630L v2 6C 2 4GHz 15MB 1600MHz 60W 00Y8632 A4MD A4MH Intel Xeon E5 2637 v2 4C 3 5GHz 15MB 1866MHz 130W 46W2719 A42B A42M Intel Xeon E5 2640 v2 8C 2 0GHz 20MB...

Page 89: ...s There is one DIMM per memory channel 1 DPC Figure 4 6 shows the memory channel layout Figure 4 6 Memory channel layout of the IBM NeXtScale nx360 M4 b Processor detail Model cores core speed L3 cach...

Page 90: ...ance For optimal performance populate the four memory channels when one processor is installed and the eight DIMMs when two processors are installed Table 4 5 lists the memory options that are availab...

Page 91: ...1 and DIMM 2 DIMM 3 and DIMM 4 to be populated by the same memory regards the size and the organization Lock step mode allows Single Device Data Correction SDDC memory protection for x8 based memory...

Page 92: ...sable memory with two installed microprocessor using 16 GB DIMMs Table 4 6 shows DIMM installation if you have one processor that is installed Table 4 6 Memory population with one processor installed...

Page 93: ...ncy the effective memory capacity of the compute node is half the installed memory capacity The pair of DIMMs that are installed in each channel must be identical in capacity type and rank count Table...

Page 94: ...IMM 6 DIMM 7 DIMM 8 4 x x x x 6 x x x x x x 8 x x x x x x x x Spec RDIMMs Rank Single rank Dual rank Part numbers 00D5024 4GB 00D5036 8GB 46W0735 4 GB 00D5044 8 GB 46W0672 16GB 00D5028 4 GB 00D5040 8...

Page 95: ...HDDs can be intermixed in the system but cannot be intermixed in the same RAID array Mixing HDDs and SSDs Both simple swap SATA HDDs and simple swap SAS HDDs can be intermixed with SSDs in the system...

Page 96: ...4 8 IBM NeXtScale Storage Native Expansion Tray on page 86 Drive cages for the drives internal to the nx360 M4 are as listed in Table 4 12 Drives used in the Storage Native Expansion Tray do not need...

Page 97: ...a ServeRAID M1115 adapter or N2115 SAS HBA and the single drive in the nx360 M4 is connected to the ServeRAID C100 4 7 1 Controllers for internal storage The nx360 M4 server support the following dis...

Page 98: ...ibm com portals systemx Open page pg cat raid ServeRAID H1110 SAS SATA controller The ServeRAID H1110 SAS SATA Controller for IBM System x offers a low cost enterprise grade RAID solution for internal...

Page 99: ...flash modules energy packs and software feature upgrades in an ultra flexible offerings structure M1115 also offers a low cost RAID 0 1 10 The IBM ServeRAID M1115 adapter has the following specificat...

Page 100: ...file half length MD2 form factor PCI Express 3 0 x8 host interface Eight internal 6 Gbps SAS SATA ports support for 6 3 or 1 5 Gbps speeds Up to 6 Gbps throughput per port Two internal x4 Mini SAS con...

Page 101: ...SSD 4 No Yes Yes Yes Yes 2 5 inch SS SATA HDD 2 No Yes Yes Yes Yes 2 5 inch SS SAS HDD 2 No No Yes Yes Yes 3 5 inch SS SATA HDD 1 Yes No No No No With Storage Native Expansion Tray attached adds 7x 3...

Page 102: ...wo at the bottom operate at 3 Gbps An array that features drives at different speeds performs at the lowest speed For example a paired array between top drives operates at 6 Gbps however a paired arra...

Page 103: ...7 2K 6Gbps SATA 2 5 HDD for NeXtScale System 2 00AD035 A48B IBM 500GB 7 2K 6Gbps SATA 2 5 HDD for NeXtScale System 2 00AD040 A48C IBM 1TB 7 2K 6Gbps SATA 2 5 HDD for NeXtScale System 2 2 5 inch simpl...

Page 104: ...allows the configuration of storage rich nx360 M4 compute nodes Using eight 4 TB drives such a configuration offers 32 TB of internal direct attach storage 49Y5834 A3AQ IBM 64GB SATA 1 8 MLC Enterpris...

Page 105: ...ompute node Ordering information is listed in Table 4 16 Table 4 16 IBM NeXtScale System Internal Storage tray Figure 4 11 shows the IBM NeXtScale Storage Native Expansion Tray with the cover removed...

Page 106: ...er is used for RAID with those two drives if two 2 5 inch SAS drives are installed the onboard ServeRAID C100 SATA controller cannot be used the C100 does not support SAS drives and only operating sys...

Page 107: ...acing with a new hard drive Some drive bays can be left empty without a filler Only bays 0 3 require fillers to ensure proper airflow Table 4 18 shows the contents of the slots and drive position Tabl...

Page 108: ...ots The tray is designed to support two GPU adapters or coprocessors Part number Feature code Description Maximum supported 3 5 inch Simple Swap SATA HDDs 00AD025 A4GC IBM 4TB 7 2K 6Gbps SATA 3 5 HDD...

Page 109: ...the GPUs or coprocessors installed in the tray Riser cards are as follows 2 slot PCIe 3 0 x24 riser card installed in the front riser slot riser slot 1 see Figure 3 This riser card replaces the standa...

Page 110: ...n 4 18 Operating systems support on page 112 Configuration rules are as follows The use of GPUs or coprocessors require the use of the IBM NeXtScale PCIe Native Expansion Tray One or two GPUs or copro...

Page 111: ...ons Figure 4 14 GPU accelerated computing GPUs are used for everything from consumer gaming and professional graphics to high performance computing to virtualized and cloud environments Two of these a...

Page 112: ...family Scientific grade computation and analytics NVIDIA Tesla GPU cards are built to achieve extremely accurate results and withstand long periods of intensive use The Tesla drivers are coded specifi...

Page 113: ...Number of CUDA cores 2 880 2 688 2 496 3 072 2 x 1 536 768 4 x 192 Memory size per board 12 GB 6 GB 5 GB 8 GB 2 x 4 GB 16 GB 4 x 4 GB Memory bandwidth per board 288 Gb s 250 Gb s 208 Gb s 320 Gb s 116...

Page 114: ...ASE T 100BASE TX and 10BASE T applications 802 3 802 3u and 802 3ab IPv6 Offloads Checksum and LSO Wake on LAN support Virtualization I OAT VMDq eight queues per port and SR IOV PCI SIG compliant 16 T...

Page 115: ...gure 4 16 Figure 4 16 Mezzanine card slot location in IBM NeXtScale nx360 M4 Table 4 23 lists the mezzanine adapters that are supported in nx360 M4 Table 4 23 Mezzanine adapters Mezzanine card Part nu...

Page 116: ...PCIe raiser cage option InfiniBand Mezzanine Card 00AM476 A4WA IBM Dual Port FDR10 QDR embedded adapter for nx360 M4 00D4143 A36R IBM Dual Port FDR Embedded Adapter FCoE iSCSI upgrades IBM Features o...

Page 117: ...ectX 3 FDR VPI IB E Adapter for IBM System x 10 Gb Ethernet 94Y5180 A4Z6 Broadcom NetXtreme Dual Port 10GbE SFP Adapter for IBM System x 49Y7910 A18Y Broadcom NetXtreme II Dual Port 10GBaseT Adapter f...

Page 118: ...System x Part number Feature code Description Part number Feature code Description 40 Gb Ethernet 46W0620 A4H5 Chelsio T580 LP CR Dual port QSFP 40GbE PCI E 3 0 Adapter 95Y3459 A2F8 Mellanox ConnectX...

Page 119: ...FC Dual port HBA for IBM System x 81Y1655 A2W5 Emulex 16Gb FC Single port HBA for IBM System x 81Y1662 A2W6 Emulex 16Gb FC Dual port HBA for IBM System x 00Y3337 A3KW QLogic 16Gb FC Single port HBA fo...

Page 120: ...VMware vSphere images can be downloaded from this website http ibm com systems x os vmware Figure 4 18 shows the USB port inside the nx360 M4 where the IBM USB Memory for VMWare ESXi is connected Figu...

Page 121: ...Figure 4 19 Console breakout cable One console breakout cable is shipped with the IBM NeXtScale n1200 Enclosure Other cables can be ordered The ordering part number is listed in Table 4 29 Table 4 29...

Page 122: ...off and is ready to be turned on Fading on and off Server is in a reduced power state To wake up the server press the power button or use the web interface of the integrated IMM which is only availab...

Page 123: ...ndition exceeds a threshold or if a system component fails the IMM2 lights LEDs to help you diagnose the problem records the error in the event log and alerts you to the problem The server includes IM...

Page 124: ...ed by using the Feature on Demand software license key that uses part number 90Y3901 adds the following features in addition to those of IMM Standard Remotely viewing video with graphics resolutions u...

Page 125: ...storage Note The IMM2 Advanced upgrade requires the IMM2 Standard upgrade Part number Feature codes Description Maximum supported 90Y3900 A1MK IBM Integrated Management Module Standard Upgrade 1 90Y39...

Page 126: ...ormation see the IBM Redbooks Product Guide ServeRAID M5120 SAS SATA Controller for IBM System x TIPS0858 http www redbooks ibm com abstracts tips0858 html Open The ServeRAID M5120 SAS SATA Controller...

Page 127: ...closure 39R6531 IBM 3 m SAS Cable 1 39R6529 IBM 1 m SAS Cable 1 Part number Description Maximum supported per one enclosure 3 5 NL SAS HS HDDs 49Y1903 1TB 7 200 rpm 6Gb SAS NL 3 5 HDD 12 49Y1902 2TB 7...

Page 128: ...imum altitude 3048 m 10 000 ft Maximum rate of temperature change 5 C per hour 41 F per hour 4 81Y9944 300GB 15 000 rpm 6Gb SAS 2 5 HDD 24 00W1595 600GB 10 000 rpm 6Gb SAS 2 5 HDD 24 46W0970 900GB 10...

Page 129: ...nishings and equipment must be connected to ground via an appropriate static control system The following items are considered the minimum requirements a Conductive materials conductive flooring condu...

Page 130: ...Linux 5 Server x64 Edition U9 Red Hat Enterprise Linux 6 Server x64 Edition U4 SUSE Linux Enterprise Server 11 for AMD64 EM64T SP3 VMware vSphere 5 0 U2 VMware vSphere 5 1 U1 Table 4 36 lists the ope...

Page 131: ...IBM ServerProven website http www ibm com systems info x86servers serverproven compat us nos m atrix shtml VMware vSphere ESXi 5 0 U3 N N N N N N N VMware vSphere ESXi 5 1 U2 N N N Y Y N N VMware vSph...

Page 132: ...114 IBM NeXtScale System Planning and Implementation Guide...

Page 133: ...e racks In this chapter we describe best practices for configuration of the individual racks After the rack level design is established we provide some guidance for designing multiple rack solutions T...

Page 134: ...ting current VAC or 200 240 VAC For NeXtScale System servers in production environments 200 240 VAC is preferred because it reduces the electrical current requirement Another consideration is that the...

Page 135: ...single power feed which can be protected by a facility UPS The power cabling that is shown in Figure 5 1 on page 118 uses three PDUs The PDUs can be supplied by using a 60 A 200 240 V three phase sour...

Page 136: ...hassis 1 41 42 39 40 37 38 35 36 33 34 31 32 29 30 27 28 25 26 23 24 21 22 19 20 17 18 15 16 13 14 11 12 09 10 07 08 05 06 03 04 01 02 41 42 39 40 37 38 35 36 33 34 31 32 29 30 27 28 25 26 23 24 21 22...

Page 137: ...ctions from four 1U PDUs Each PDU has 12 outlets thus 48 outlets are available There are six NeXtScale n1200 Enclosures each with six power supplies so there are 36 power supplies to be connected This...

Page 138: ...he specifications of the equipment Rear of NeXtScale Chassis 1 41 42 39 40 37 38 35 36 33 34 31 32 29 30 27 28 25 26 23 24 21 22 19 20 17 18 15 16 13 14 11 12 09 10 07 08 05 06 03 04 01 02 41 42 39 40...

Page 139: ...ee one of the following guides For North America and Japan http www ibm com support techdocs atsmastr nsf WebIndex WP101526 For the International guide http www ibm com support techdocs atsmastr nsf W...

Page 140: ...It can require careful planning for large scale environments After the power planning is done calculating the amount of heat to be dissipated is relatively straight forward For each W of power that is...

Page 141: ...NeXtScale are recessed 75 mm behind the front rack mounting brackets to provide sufficient room for cables A standard filler panel does not contact a switch that is recessed 75 mm therefore hot air r...

Page 142: ...ear door heat exchangers as described in 5 5 Rear Door Heat Exchanger on page 141 This option is a relatively low cost low complexity space and power efficient solution to the cooling challenge 5 3 De...

Page 143: ...e described Further considerations for installing NeXtScale System servers in other racks are covered and other rack options are also described 5 4 1 The IBM 42U 1100mm Enterprise V2 Dynamic Rack The...

Page 144: ...f open systems rack for the enterprise The base model of the rack comes complete with side panels The expansion model of the rack comes with the hardware that is used to join it to another rack and no...

Page 145: ...Figure 5 6 Figure 5 6 Rack with stabilizer bracket attached being bolted to the floor As shown in Figure 5 7 on page 128 a recirculation plate is used to prevent warm air that is coming from the rear...

Page 146: ...rough the doors of most elevators and doorways Reusable ship loadable packaging For more information about the transportation system see this website http ibm com support entry portal docdisplay lndoc...

Page 147: ...ting brackets in the rear post flanges which can be used for power distribution units switches or other 1U devices as shown in Figure 5 8 Figure 5 8 1U Power distribution unit mounted in 1U pocket on...

Page 148: ...igure 5 9 Openings in the side walls behind the rear posts through which cables can be routed between racks in a row as indicated by arrows in Figure 5 5 on page 126 Also includes are attachment point...

Page 149: ...cks are listed in Table 5 6 Table 5 6 Enterprise V2 Dynamic Rack part numbers For more information about this rack see the Installation Guide which available at this website http www ibm com support e...

Page 150: ...mm including handles and latches the distance is approximately 935 mm D The distance from front EIA flange to the bend radius of the rear cables is 980 mm E The minimum distance from the front EIA fla...

Page 151: ...ack and be approximately 25 x 75 mm in size Some cable configurations might require more cable routing space We suggest installing the chassis 1 or 2 rack units from the bottom to allow space for cabl...

Page 152: ...ack Figure 5 11 shows the cable bracket Figure 5 11 Cable management bracket kit for third party racks part 00Y3040 The part number of the Cable Management Bracket kit is listed in Table 5 7 Table 5 7...

Page 153: ...s must be occupied at the front by a device or a blank filler panel All other openings should be covered including air openings around the EIA flanges rack posts and cable passage ways If multiple rac...

Page 154: ...ole Manager LCM8 Console cables 43V6147 IBM Single Cable USB Conversion Option UCO 39M2895 IBM USB Conversion Option four Pack UCO 39M2897 IBM Long KVM Conversion Option four Pack Long KCO 46M5383 IBM...

Page 155: ...irflow around any cables that are passed through it It can also serve to block air flow around a switch that is recessed in the rack for cable routing reasons that pass around a blank filler panel Fig...

Page 156: ...it part number 00Y3016 that attaches to the front of the rack is shown in Figure 5 14 This kit includes four brackets which are enough for one rack two brackets are installed on each side of the rack...

Page 157: ...number 00Y3001 has the following purposes Provides a means to seal the opening in the bottom front of the IBM 42U 1100 mm Enterprise V2 Dynamic Rack The opening at the bottom front of the rack must b...

Page 158: ...ch Seal Kit contents Figure 5 17 shows where these pieces are used Figure 5 17 Placement of the components of the Rack and Switch Seal Kit Switch seals Seal kit foam blocks Kit contains 12 pieces enou...

Page 159: ...a few drops at most when the doors are connected or disconnected Each door has a capacity of 9 liters 2 4 US gallons and supports flow rates of 22 7 liters 6 US gallons to 56 8 liters 15 US gallons pe...

Page 160: ...ir conditioning requirement typically saves about 1 KW per rack that is used to compress refrigerant and move air The reduction in air conditioner noise coupled with the acoustic dampening effect of t...

Page 161: ...IBM NeXtScale System features forward facing cabling To connect cables from the NeXtScale servers to Ethernet switches it is easiest to mount the switches in the racks with the switch ports facing the...

Page 162: ...available through the Intelligent cluster program Part number Description IBM System Networking 1 Gb top of rack switches 7309CFC IBM System Networking RackSwitch G8000F 730952F IBM System Networking...

Page 163: ...Channel networks which are compatible with 8 Gb and 4 Gb storage devices are popular because of their high I Os per second IOPS capability and high reliability In larger clusters or systems with high...

Page 164: ...e location of the chassis and the switches within the rack are shown in a way that optimizes the cabling of the solution The chassis and switches are color coded to indicate which InfiniBand or Ethern...

Page 165: ...rt 1GBe 36 Down 2 10G Up 36 Port IB 18 Down 18 UP 36 Port IB 18 Down 18 UP IB Switch 1 IB Switch 2 IB Switch 3 IB Switch 4 Cable Channel 1 Cable Channel 2 Cable Channel 3 Cable Channel 4 1 2 3 4 5 6 1...

Page 166: ...el 2 Cable Channel 3 Cable Channel 4 1 2 3 4 5 6 13 14 15 16 17 18 7 8 9 10 11 12 19 20 21 22 23 24 31 32 33 34 35 36 25 26 27 28 29 30 37 38 39 40 41 42 1G Ethernet Switch 1 1G Ethernet Switch 2 1 3...

Page 167: ...Cable Channel 2 Cable Channel 3 Cable Channel 4 1 2 3 4 5 6 13 14 15 16 17 18 7 8 9 10 11 12 19 20 21 22 23 24 31 32 33 34 35 36 25 26 27 28 29 30 37 38 39 40 41 42 1G Ethernet Switch 1 1G Ethernet S...

Page 168: ...8 Port 1GBe 36 Down 2 10G Up 48 Port 1GBe 36 Down 2 10G Up 48 Port 10 Gb Ethernet Switch 48 Port 10 Gb Ethernet Switch 10 Gb Ethernet Switch 1 10 Gb Ethernet Switch 2 10 Gb Ethernet Switch 3 Cable Cha...

Page 169: ...2 10G Up 36 Port IB 18 Down 18 UP 36 Port IB 18 Down 18 UP IB Switch 1 IB Switch 2 IB Switch 3 IB Switch 4 Cable Channel 1 Cable Channel 2 Cable Channel 3 Cable Channel 4 1 2 3 4 5 6 13 14 15 16 17 1...

Page 170: ...anagement port that plugs to the Fan and Power Controller which is at the rear of the chassis Each PDU can also have a management port Note The management cables that connect to devices at the rear of...

Page 171: ...Intelligent Cluster testing process was enhanced and all NeXtScale System configurations that are integrated by IBM benefit This chapter describes what IBM factory integration provides what testing i...

Page 172: ...hipping These standards govern the range of systems that can be factory integrated by IBM and we consider them to follow best practices given our design criteria There are a number of alternative conf...

Page 173: ...x_p df intelligent_cluster_factory_settings_102411 pdf Power redundancy testing If there is a client provided redundant power domain scheme IBM tests with one domain powered down then the opposite dom...

Page 174: ...esting is performed Interconnect networks are exercised HPLinpack is run over the highest bandwidth lowest latency network As an added benefit clients can see a performance baseline measurement if req...

Page 175: ...pe model number and serial number of devices Firmware levels For servers these levels include UEFI version IMM version On system diagnostics version Memory per server CPU type per server Proof that an...

Page 176: ...ll clock Right y axis Efficiency Figure 6 1 Example of HPLinpack output graph Although open systems can be racked cabled configured and tested by users we encourage clients to evaluate the benefits of...

Page 177: ...ibe the management capabilities and interfaces that are integrated in the system and the middleware and software layers that are often used in High Performance Computing to manage clusters This chapte...

Page 178: ...functionality that is available at each level see 4 15 Remote server management on page 105 7 1 1 Integrated Management Module II In 2009 IBM introduced the IPMI compliant service processor that is ca...

Page 179: ...or CLI SSH Web interface if IMM Standard and Advanced FoD is available IPMI 2 0 local or remote Advanced Settings Utility ASU SNMP v1 and v3 Figure 7 1 shows the available IMM2 access methods Figure 7...

Page 180: ...ess to the IMM and provides a separate physical connection However IMM can be configured to be accessed through the first on board 1 Gb Ethernet port which results in less cabling and a shared physica...

Page 181: ...is accessed can be selected manually through F1 UEFI setup menu or by using the ASU tool that allows to modify firmware settings through a command line interface CLI remotely or locally to the node F...

Page 182: ...n operating system and running pre boot applications For more information about UEFI see this website http www uefi org home UEFI provides the following improvements over BIOS ASU now has complete cov...

Page 183: ...anagement Reduces the number of error messages and eliminates outdated errors A complete setup solution is provided by allowing adapter configuration function to be moved into UEFI Complete out of ban...

Page 184: ...e node is designed to provide optimal performance with reasonable power usage which depends on the operating frequency and voltage of the processors and memory subsystem In most operating conditions t...

Page 185: ...as operating modes Access the menu in UEFI by selecting System Settings Operating Modes Choose Operating Mode You see the five operating modes from which to choose as shown in Figure 7 6 When a mode...

Page 186: ...imal Power predetermined values They emphasize power saving server operation by setting the processors QPI link and memory subsystem to a lowest working frequency Minimal Power provides less heat and...

Page 187: ...predetermined values They emphasize power saving server operation by setting the processors QPI link and memory subsystem to a balanced working frequency Efficiency Favor Power provides more performa...

Page 188: ...or Performance Figure 7 9 shows the Efficiency Favor Performance predetermined values They emphasize performance server operation by setting the processors QPI link and memory subsystem to a high work...

Page 189: ...c values that they want as shown in Figure 7 10 The recommended factory default setting values provide optimal performance with reasonable power usage However with this mode users can individually set...

Page 190: ...cessor to provide the maximum performance from the processors and memory subsystem Figure 7 11 UEFI operation mode Maximum Performance Performance related individual system settings The UEFI default s...

Page 191: ...12 UEFI Processor system settings panel The following processor feature options are available Turbo Mode Default Enable This mode enables the processor to increase its clock speed dynamically if the...

Page 192: ...ble Bit Default Enable This option enables the processor to disable the running of certain memory areas which prevents buffer overflow attacks Intel Virtualization Technology Default Enable This optio...

Page 193: ...memory operation options as shown in Figure 7 13 Figure 7 13 UEFI Memory system settings panel The following memory feature options are available Memory Mode Default Independent This option selects m...

Page 194: ...rs whereas Non NUMA memory is interleaved accross processors Memory Data Scrambling Default Enabled This option enables a memory data scrambling feature to further minimize bit data errors Page Policy...

Page 195: ...side the node to modify node settings or to change a remote node through its IMM interface When local to the node ASU configures the in band LAN over the USB interface and performs the wanted action W...

Page 196: ...or use with UpdateXpress system packs Updates are available as Windows exe or Linux bin files and are applied inband or remotely through the IMM UpdateXpress System Packs UpdateXpress System Packs are...

Page 197: ...n the chassis The FPC module has the following features Power usage information at the chassis node power supply units PSU and fan levels PSU and fan status information Can configure wanted redundancy...

Page 198: ...on page 202 Complete the following steps to access the FPC web interface 1 Point your browser to the FPC interface URL that is defined for your FPC module By default the module is configured with the...

Page 199: ...enclosure elements and allows the configuration of power supply redundancy modes power capping or saving policies and power restore policies Cooling Provides information about fan speed and allows the...

Page 200: ...s that can appear in each column at the systems table Table 7 1 Front overview systems table Column Description Node Indicates slot number Width Possible values Half Represents a half wide node Full R...

Page 201: ...system fans and FPC module information It displays characteristics of the available elements and a summary of health conditions with which the system administrator can easily identify the source of a...

Page 202: ...wer Off Warning EPOW and Throttle see 3 8 Power management on page 52 Possible values Assert Power supply is in AC lost condition Normal Power supply is in healthy operating condition Throttlea Possib...

Page 203: ...ng tabs are available Power Overview tab on page 185 PSU Configuration tab on page 186 Power Cap tab on page 187 Voltage Overview tab on page 189 Power Restore Policy tab on page 189 Power Overview ta...

Page 204: ...for the enclosure and enable oversubscription mode if needed The following redundancy modes can be selected No redundancy Compute nodes can be throttled or shutdown if any power supply is in faulty co...

Page 205: ...that is available to grant power permission to systems that are installed in the chassis depends directly on the capacity of the power supply units that are installed the demanding power of the nodes...

Page 206: ...apping at node level is selected via the drop down menu The specific node is selected in the drop down menu that appears inside the table Here the suggested range is based on the minimum and maximum c...

Page 207: ...ting to favor performance over power savings Mode 3 dynamic favor power The system adjusts the throttling levels that are based on workload attempting to favor power savings over performance Voltage O...

Page 208: ...s shown in Figure 7 23 The Status changes from Disable to Enable or vice versa Figure 7 23 Power Restore Policy tab window Cooling The Cooling function provides information about fan speed for system...

Page 209: ...tab displays system fan speeds and their healthy condition Each fan is equipped with dual motor so A displays the primary fan motor speed and B displays the redundant fan motor speed System fan speed...

Page 210: ...n Acoustic Mode tab As shown in Figure 7 26 the Acoustic Mode is set in the Acoustic Mode tab and is intended to reduce the noise of the IBM NeXtScale n1200 Enclosure The following acoustic levels whi...

Page 211: ...em information VPD windows for the chassis and the midplane Figure 7 27 Chassis Vital Product Data window Figure 7 28 Midplane Vital Product Data window Event Log The Event Log function displays the S...

Page 212: ...perations on the internal USB are automatically done by the FPC Any change that is done through the web interface or the IPMI interface that is part of the following settings is saved in the internal...

Page 213: ...backup and recovery tab Configuration The Configuration function displays and configures the FPC module All settings under the Configuration function are non volatile so they are kept between FPC reb...

Page 214: ...ocal firmware file that is uploaded and verified to be valid Second after the firmware is checked a confirmation is requested A table shows the actual firmware version the new firmware version and a p...

Page 215: ...o send the events to the destination email addresses and SMTP server as shown in Figure 7 32 The Global Alerting Enable option at the Platform Event Filters PEF tab must be selected to enable SMTP tra...

Page 216: ...iguration to send as SNMP traps the events that happen as shown in Figure 7 33 The specific event types that are sent are selected at the Platform Event Filter PEF tab The Global Alerting Enable optio...

Page 217: ...obal Alerting Enable option Figure 7 34 Platform Event Filters window Network Configuration tab In the Network Configuration tab users can configure the network setting for the FPC module as shown in...

Page 218: ...and time configuration window User Account tab In the User Account tab users can add or remove users and assign one of the following user roles as shown in Figure 7 37 on page 201 Administrator Full...

Page 219: ...37 User Configuration window Web Service tab User can configure the web interface ports for HTTP and HTTPS access in the Web Service tab as shown in Figure 7 38 Figure 7 38 Configuration of the HTTP...

Page 220: ...l list OEM IPMI command extensions require the use of the raw interface ipmitool provides The syntax for such interface has the following format ipmitool I lanplus U USERID P PASSW0RD H 192 168 0 100...

Page 221: ...IN DC OUT Byte 1 Completion code 0x00 Byte 2 Sum of MIN AC IN DC OUT Least Significant Bit LSB Byte 3 Sum of MIN AC IN DC OUT Most Significant Bit MSB Byte 4 Sum of average AC IN DC OUT LSB Byte 5 Sum...

Page 222: ...Node 1 to 12 Chassis 0x0d Byte 2 Capping value LSB Byte 3 Capping value MSB Response Data Byte 1 completion code 0x00 or out of range 0xC9 or cur not support 0xD5 Set power saving state 0x32 0x9f Requ...

Page 223: ...0x32 0xa3 Request Data Byte 1 PSU Policy 0 No redundancy 1 N 1 2 N N Response Data Byte 1 Completion code 0x00 or out of range 0xC9 or config not allowed 0x01 or bank lack 0x02 Set Over Subscription...

Page 224: ...0x7 0000 0111 Response Data Byte 1 Completion code 0x00 or out of range 0xC9 Set Restore Policy 0x32 0xaa Request Data None Response Data Byte 1 Completion code 0x00 or out of range 0xC9 Byte 2 Node...

Page 225: ...nk SysLocated LED only Response Data Byte 1 Completion code 0x00 Description NetFn CMD Data Get Node Status 0x32 0xa7 Request Data Byte 1 Node number 0x1 to 0x0c for Node 1 to 12 Response Data Byte 1...

Page 226: ...al Width Byte 3 Node Physical Height Byte 4 Add on Valid Byte 5 Add on Width Byte 6 Add on Height Show Node Power Consumptionin watts 0x32 0x98 Request Data Byte 1 options Node number 0x1 to 0x0c for...

Page 227: ...le 7 10 on page 207 Table 7 11 on page 207 Table 7 12 on page 209 Description NetFn CMD Data Set Time 0x32 0xa1 Request Data Byte 1 Year MSB 1970 2037 Byte 2 Year LSB 1970 2037 Byte 3 Month 0x01 0x12...

Page 228: ...0 100 raw 0x32 0x98 0x3 00 93 00 94 00 97 00 The output string has the following meaning Byte 1 Completion code 0x00 Byte 2 and 3 Power minimum 0x0093 147 W Byte 4 and 5 Power average 0x0094 148 W By...

Page 229: ...AID support must be enabled by pressing F1 at the setup menu By using the F1 setup menu the MegaCLI command line utility and the MegaRAID Storage Manager a storage configuration must be created for th...

Page 230: ...USB memory key where to install VMware vSphere Hypervisor ESXi The VMware ESXi embedded hypervisor software is a virtualization platform with which multiple operating systems can be run on a host syst...

Page 231: ...t IBM provides for technical computing high performance computing analytics and cloud environments This chapter includes the following topics 8 1 eXtreme Cloud Administration Toolkit xCAT on page 214...

Page 232: ...tecture includes the following main features Client server architecture Clients can run on any Perl compliant system including Windows All communications are SSL encrypted Role based administration Di...

Page 233: ...for compartmental development Add your own xCAT functionally to do whatever you want New plug ins extend the xCAT vocabulary that is available to xCAT clients Notification infrastructure By using thi...

Page 234: ...raditional local disk SAN disk and stateful diskless which provisions via native deployment methods Also support is provided for stateless diskless nodes including ramfs root compressed ramfs root and...

Page 235: ...more information and the latest list of supported hardware see this website http xcat sourceforge net 8 2 IBM Platform Cluster Manager The IBM Platform Cluster Manager family provides full cluster li...

Page 236: ...es such as provisioning and maintaining a cluster By using a centralized user interface system administrators can manage complex clusters as a single system from anywhere A next generation web portal...

Page 237: ...y which expands the ability to meet and exceed service levels and lower operating costs Workload monitoring resource monitoring alerting and troubleshooting Integration with the xCAT or multiple insta...

Page 238: ...e without disrupting applications thus lowering storage costs and reducing management overhead GPFS ensures that there is no single point of failure Unlike some file systems designs GPFS is not relian...

Page 239: ...you can create associations from a local GPFS cluster to a remote cluster or storage and define the location and flow of file data to automate the management of the data You can implement a single nam...

Page 240: ...ge system For more information about IBM GPFS FPO and IBM GPFS Native RAID see the following sections 8 3 1 IBM GPFS FPO 8 3 2 IBM System x GPFS Storage Server on page 224 8 3 1 IBM GPFS FPO The GPFS...

Page 241: ...ap Reduce Environment that uses GPFS FPO Figure 8 3 Map Reduce Environment that uses GPFS FPO For organizations that want to run more applications on their cluster in a multi tenant environment IBM Pl...

Page 242: ...uses decades of IBM experience to reduce the complexity of deployment with integrated delivered and fully supported solutions that match best in industry components with optimized solution design The...

Page 243: ...s The final storage system benefits from the aggregated performance of its building blocks 2 TB and 3 TB drives are available that provide an enormous storage capacity with the high density enclosures...

Page 244: ...performance and rapid rebuilds Data and parity is distributed between several disks so it benefits performance and reliability In the case of a disk failure all of the disks participate in the recons...

Page 245: ...help ensure optimal application performance IBM Platform LSF includes the following features Manages batch workloads Allows a distributed compute network to function as a large supercomputer by matchi...

Page 246: ...onitoring facility that is designed specifically for Platform LSF environments IBM Platform License Scheduler IBM Platform License Scheduler enables license sharing between global project teams It ens...

Page 247: ...Cluster Platform Dynamic Cluster turns static IBM Platform LSF clusters into dynamic shared cloud infrastructure By automatically changing the composition of clusters to meet ever changing workload de...

Page 248: ...e sized HPC clusters Cluster provisioning This function is provided by the elements of IBM Platform Cluster Manager Standard Edition that is included in the IBM Platform HPC product Physical machines...

Page 249: ...effectively the available capacity is used These monitoring facilities are a simplified subset of those facilities that are provided by the IBM Platform Application Center MPI libraries HPC clusters f...

Page 250: ...p you achieve the following goals Obtain higher quality business results faster Reduce infrastructure and management costs Combine compute intensive and data intensive applications on a single shared...

Page 251: ...signed for developing and running parallel Fortran C or C programs IBM PE Runtime Edition consists of components and command line tools for developing running debugging profiling and tuning parallel p...

Page 252: ...ition is a capability rich development and run environment for parallel applications IBM PE Runtime Edition offers parallel application programming interfaces and run environment for parallel applicat...

Page 253: ...g operation It supports nonblocking and ad hoc geometry group communicator creation and nonblocking collective allreduce reduce broadcast gather v scatter v alltoall v reduce scatter and ex scan opera...

Page 254: ...kbench software is delivered in the following packages An all in one bundle with Eclipse basics and the IBM PE Developer Edition additions An Eclipse update site archive where you can add the client a...

Page 255: ...g and hyperlink navigation Source code refactoring and code generation Visual debugging tools including memory registers and disassembly viewers Photran Photran is an IDE and refactoring tool for Fort...

Page 256: ...forming low level analysis of an application including analyzing cache usage and floating point performance Profile and trace an MPI application for analyzing MPI communication patterns and performanc...

Page 257: ...ommand CMOS complementary metal oxide semiconductor CNA Converged Network Adapter CNFS Clustered Network File System Abbreviations and acronyms COD configure on disk CPU central processing unit CSM Cl...

Page 258: ...l IPMI Intelligent Platform Management Interface ISO International Organization for Standards IT information technology ITSO International Technical Support Organization JBOD just a bunch of disks KB...

Page 259: ...endent disks RAM random access memory RAS remote access services row address strobe RDIMM registered DIMM RHEL Red Hat Enterprise Linux RMC Resource Monitoring and Control ROC RAID on card RSS Receive...

Page 260: ...red DIMM UDP user datagram protocol UEFI Unified Extensible Firmware Interface UPC Unified Parallel C UPS uninterruptible power supply URL Uniform Resource Locator USB universal serial bus UXSP Update...

Page 261: ...NeXtScale nx360 M4 TIPS1051 xREF IBM x86 Server Reference REDP XREF You can search for view download or order these documents and other Redbooks Redpapers Web Docs draft and other materials at this w...

Page 262: ...ocs atsmastr nsf WebIndex PRS5196 ServerProven hardware compatibility page for the IBM NeXtScale nx360 M4 http ibm com systems info x86servers serverproven compat us NeXtSc ale 5455 html ServerProven...

Page 263: ...0 5 spine 0 475 0 875 250 459 pages IBM NeXtScale System Planning and Implementation Guide...

Page 264: ......

Page 265: ......

Page 266: ...erience with IBM iDataPlex and IBM BladeCenter with a tight focus on emerging and future client requirements The IBM NeXtScale n1200 Enclosure and IBM NeXtScale nx360 M4 Compute Node are designed to o...

Search results

Summary of Contents for NeXtScale System

Reviews:

Related manuals for NeXtScale System

Brands by name

Popular brands

IBM NeXtScale System, Planning And Implementation Manual

Search results

Summary of Contents for NeXtScale System

Reviews:

Related manuals for NeXtScale System

Brands by name

Popular brands