IBM 88743BU - System x3950 E User Manual Download Page 398

Page: 398 / 426

380

Planning, Installing, and Managing the IBM System x3950 M2

Summary of Contents for 88743BU - System x3950 E

Page 1: ...he IBM System x3950 M2 David Watts Jens Reizel Paul Tan Kevin Galloway Understand the IBM System x3950 M2 and IBM x3850 M2 Learn the technical details of these high performance servers See how to configure install manage multinode complexes Front cover ...

Page 2: ......

Page 3: ...Planning Installing and Managing the IBM System x3950 M2 November 2008 International Technical Support Organization SG24 7630 00 ...

Page 4: ... or disclosure restricted by GSA ADP Schedule Contract with IBM Corp First Edition November 2008 This edition applies to the following systems IBM System x3950 M2 machine types 7141 and 7233 IBM System x3850 M2 machine types 7141 and 7233 Note Before using this information and the product it supports read the information in Notices on page ix ...

Page 5: ...ndows Datacenter models 15 1 4 1 IBM Datacenter Unlimited Virtualization offering 16 1 4 2 IBM Datacenter Unlimited Virtualization with High Availability 16 1 4 3 Upgrading to Datacenter Edition 17 1 4 4 Datacenter multinode configurations 19 1 4 5 Datacenter cluster configurations 19 1 5 Integrated virtualization VMware ESXi 19 1 5 1 Key features of VMware ESXi 20 1 5 2 VMware ESXi on x3850 M2 21...

Page 6: ...2 IBM BladeCenter and iDataPlex 57 2 3 Comparing x3850 M2 to x3850 59 2 4 Comparing x3950 M2 to x3950 62 2 5 System scalability 64 2 6 Operating system scalability 66 2 6 1 Scaling VMware ESX 68 2 6 2 Scaling Microsoft Windows Server 2003 73 2 6 3 Scaling Microsoft Windows Server 2008 and Hyper V 75 2 6 4 Scaling Linux server operating systems 76 2 7 Application scalability 79 2 7 1 Microsoft SQL ...

Page 7: ...5 Supported PCI Express adapter options 194 Chapter 4 Multinode hardware configurations 195 4 1 Introduction and terminology 196 4 2 Multinode capabilities 198 4 3 Understanding scalability 199 4 3 1 Complex Descriptor 199 4 3 2 Complex Descriptor contents 200 4 4 Prerequisites to create a multinode complex 201 4 5 Upgrading an x3850 M2 to an x3950 M2 204 4 5 1 Installing the ScaleXpander key chip...

Page 8: ...guring VMware ESXi 3 5 embedded 264 5 4 2 Installing VMware ESXi 3 5 Installable 279 5 4 3 Installing VMware ESX 3 5 Update 1 281 5 4 4 Installing Windows Server 2003 285 5 4 5 Installing Windows Server 2008 289 5 4 6 Installing Red Hat Enterprise Linux 5 Update 1 293 5 4 7 Installing SUSE Linux Enterprise Server 10 SP1 295 Chapter 6 Management 299 6 1 BMC configuration options 300 6 1 1 BMC conne...

Page 9: ...nagement with VMware ESXi 3 5 355 6 6 1 Hypervisor systems 355 6 6 2 Implementation of x3850 M2 Hypervisor systems 355 6 7 Power management 356 6 7 1 Processor features 356 6 7 2 Power consumption measurement and capping 357 6 7 3 Virtualization 357 6 8 Power Distribution Units PDU 357 6 8 1 Key features 358 6 8 2 Availability and flexibility of Enterprise PDUs 359 6 8 3 Comparing PDU and intellig...

Page 10: ...viii Planning Installing and Managing the IBM System x3950 M2 ...

Page 11: ...ditions of the publication IBM may make improvements and or changes in the product s and or the program s described in this publication at any time without notice Any references in this information to non IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites The materials at those Web sites are not part of the materials for this IBM pro...

Page 12: ...rrow logo and combinations thereof are trademarks of Advanced Micro Devices Inc Cognos and the Cognos logo are trademarks or registered trademarks of Cognos Incorporated an IBM Company in the United States and or other countries Snapshot and the NetApp logo are trademarks or registered trademarks of NetApp Inc in the U S and other countries Novell SUSE the Novell logo and the N logo are registered...

Page 13: ...rademark of The Open Group in the United States and other countries Linux is a trademark of Linus Torvalds in the United States other countries or both Other company product or service names may be trademarks or service marks of others ...

Page 14: ...xii Planning Installing and Managing the IBM System x3950 M2 ...

Page 15: ...eed memory and support for up to 28 PCI Express adapters The capacity gives you the ultimate in processing power ideally suited for very large relational databases The x3850 M2 is the equivalent of the x3950 M2 however it can only be used as a single four processor node This IBM Redbooks publication describes the technical details of the x3950 M2 scalable server and the x3850 M2 server We explain ...

Page 16: ...r presentations and solution workshops based around key leading IBM technologies with a particular focus on x86 based virtualization products such as VMware He has been working in this role for more than two years and prior to that for five years as an IBM Infrastructure Consultant specializing in Microsoft and Linux systems He holds a Bachelor of Science Computer Science and Bachelor of Engineeri...

Page 17: ...rom IBM Development Paul Anderson Chia Yu Chu Richard French Joe Jakubowski Mark Kapoor Don Keener Dan Kelaher Randy Kolvick Josh Miller Thanh Ngo Chuck Stephan From IBM Service and Support Khalid Ansari Brandon Church Become a published author Join us for a two to six week residency program Help write a book dealing with specific products or solutions while getting hands on experience with leadin...

Page 18: ...ndex and apply online at ibm com redbooks residencies html Comments welcome Your comments are important to us We want our books to be as helpful as possible Send us your comments about this book or other IBM Redbooks in one of the following ways Use the online Contact us review Redbooks form found at ibm com redbooks Send your comments in an e mail to redbooks us ibm com Mail your comments to IBM ...

Page 19: ...hapters discusses the following topics 1 1 IBM eX4 based servers on page 2 1 2 Model numbers and scalable upgrade options on page 9 1 3 Multinode capabilities on page 14 1 4 x3950 M2 Windows Datacenter models on page 15 1 5 Integrated virtualization VMware ESXi on page 19 1 6 IBM fourth generation XA 64e chipset on page 27 1 7 Processors on page 33 1 8 Memory subsystem on page 39 1 9 SAS controlle...

Page 20: ...icant processing and I O performance and greater power efficiency The two servers based on IBM eX4 technology are IBM System x3850 M2 IBM System x3950 M2 Although they have the same technical specifications and features the x3850 M2 cannot be used to form a multinode unless you upgrade it to an IBM System x3950 M2 by adding the ScaleXpander Option Kit as described in section 1 2 Model numbers and ...

Page 21: ... LEDs indicating information such as scalability network activity and system errors and warnings The scalability LED on an x3950 M2 indicates whether the node building block in a scalable system is participating in a multinode x3950 M2 complex After each node has successfully merged with the primary node in a partition the scalability LED is lit on all nodes in a partition of a multinode complex 1...

Page 22: ... servers video connector port 10 100 Mbps RJ45 out of band remote management port there is also a mini USB port and a power adapter socket that is not used for the x3850 M2 x3950 M2 Two hot swap redundant power supplies Hypervisor models of the x3850 M2 Inside the server is an additional USB socket used exclusively for the embedded virtualization feature This device shown in Figure 1 4 on page 5 i...

Page 23: ...g Intel processors Xeon 7200 series Tigerton dual core processors Xeon 7300 series Tigerton quad core processors Xeon 7400 series Dunnington quad core processors Xeon 7400 series Dunnington 6 core processors Two processors standard with support for up to four processors One IBM eX4 Hurricane 4 chipset with four 1066 MHz front side buses Support for Intel Virtualization Technology Intel VT Intel 64...

Page 24: ...ID controller and HDD slots Disk subsystem features include Integrated LSI 1078 SAS controller with support for RAID 0 and RAID 1 External JBOD SAS storage through external SAS x4 port if IBM ServeRAID MR10k SAS SATA Controller is installed The SAS SFF 8088 connector is located above SMP Expansion Port 2 in Figure 1 3 on page 4 Up to four hot swap 2 5 inch SAS hard drives up to a maximum of 584 GB...

Page 25: ...page 5 An ATI Radeon ES1000 SVGA video controller DB 15 video connector on RSA II card as shown in Figure 1 3 on page 4 on the Remote Supervisor Adapter II Optical drive On machine type 7141 One standard 24x 8x IDE CD RW DVD ROM combo drive One machine type 7233 SATA CD RW DVD ROM combo drive USB keyboard and mouse System serial port Three SMP expansion ports for use in scalable multinode complex ...

Page 26: ... of three x3950 M2 servers However to form a four node x3950 M2 complex you must have four x3950 M2 and a Scalability Upgrade Option 2 which contains one 3 08m and one 3 26m Scalability cable see Table 4 1 on page 202 for details of part numbers Refer to Chapter 4 Multinode hardware configurations on page 195 for more details about scaling the x3950 M2 to complexes of two three and four nodes Figu...

Page 27: ...for each server and where to find more information about models available in your country The tables in this section use the following nomenclature n Indicates variations between server models relating to the processor type and the number of memory cards and memory DIMMs installed c Indicates the country in which the model is available U is for countries in North America and South America G is for...

Page 28: ... IBM BladeCenter and System x Reference Sheets xREF http www redbooks ibm com xref 1 2 2 x3850 M2 model information The model numbers of the x3850 M2 are listed in Table 1 1 Table 1 1 Models of x3850 M2 Models Description 7141 nRc Standard models of x3850 M2 with dual core or quad core Xeon 7200 and Xeon 7300 Tigerton processors 7141 3Hc Integrated hypervisor models of x3850 M2 with Xeon E7330 Tig...

Page 29: ... Xeon 7200 or Xeon 7300 Tigerton processors 7233 nSc Standard Models of x3950 M2 with quad core and six core Xeon 7400 Dunnington processorsa a Dunnington quad core and six core processors include L2 and L3 shared cache unlike Tigerton processors with only L2 shared cache See 1 7 Processors on page 33 for more details 7141 nAc Datacenter Unlimited Virtualization with High Availability models certi...

Page 30: ...red to convert the x3850 M2 to an x3950 M2 See Figure 1 5 on page 8 x3950 M2 bezel which replaces the existing bezel and shows the x3850 M2 has the kit installed and is now functionally equal to an x3950 M2 See Figure 1 9 on page 13 Figure 1 7 x3950 M2 enterprise cable management arm ...

Page 31: ...ver rails Figure 1 9 x3950 M2 bezel Scalability cable brackets to guide the scalability cables from the scalability expansion ports on one node to another node Route power Ethernet fibre cables video mouse and keyboard through here Scalability expansion ports 1 2 and 3 from left to right ...

Page 32: ...n 1 2 4 Scalable upgrade option for x3850 M2 on page 11 Multinode configurations The x3950 M2 can form a multinode configuration by adding one or more x3950 M2 servers A number of configurations are possible as shown in Figure 1 10 Figure 1 10 Possible multinode configurations Note When we refer to an x3950 M2 we mean either an x3950 M2 or an x3850 M2 that has the ScaleXpander Option Kit installed...

Page 33: ...ries The interface where you set up and maintain partitions is an extension of the Remote Supervisor Adapter II Web interface It is used to create delete control and view scalable partitions Multinode complexes support partitioning on node boundaries This means for example you can logically partition your 2 node 8 way system as two 4 way systems while still leaving the complex cabled as 2 nodes Th...

Page 34: ...ited Virtualization option the x3950 M2 models come with two processors 8 GB of memory eight 1 GB DIMMs four memory cards and no disks The system is shipped with the Datacenter installation CD OS documentation recovery CD and a 4 socket Certificate of Authenticity COA to license the system Windows Server 2003 R2 Datacenter Edition is not preloaded This offering is available in both English and Jap...

Page 35: ...te of Authenticity COA to license the system This offering is available in both English and Japanese languages 1 4 3 Upgrading to Datacenter Edition If you are using another Windows operating system on your x3950 M2 such as Windows Server 2003 Enterprise Edition and want to upgrade to Datacenter Edition you can order the appropriate upgrade as described in this section IBM Datacenter preload upgra...

Page 36: ...BM Datacenter Unlimited Virtualization with High Availability UVHA offering order one or more of the part numbers listed in Table 1 4 You must have one 4 CPU license for each x3950 M2 in your configuration Licenses are cumulative Table 1 4 Upgrade options for IBM Datacenter UVHA Upgrade kits Order number Windows Server 2003 Datacenter Edition R2 32 bit 1 4 CPUs 4818 NCU Windows Server 2003 Datacen...

Page 37: ...SXi is the next generation hypervisor that is integrated into IBM servers such as the x3850 M2 It provides a cost effective high capacity virtual machine platform with advanced resource management capabilities This innovative architecture operates independently from any general purpose operating system offering improved security increased reliability and simplified management The compact architect...

Page 38: ...ages all hardware devices on the server and manages resources for the applications The main processes that run on top of VMkernel are Direct Console User Interface DCUI which is the low level configuration and management interface accessible through the console of the server and used primarily for initial basic configuration The virtual machine monitor which is the process that provides the execut...

Page 39: ...consideration when selecting a server to run VMware ESX is to ensure you have sufficient headroom in capacity The IBM x3850 M2 is optimized for VMware ESXi because of its vertical scalability in the key areas such as processor memory and I O subsystems VMware discusses the benefits of CPU dense ESX server hosts by saying2 The chance that the scheduler can find room for a particular workload withou...

Page 40: ...I Editions being offered from VMware and provides the same functionality as ESX 3 5 VMware ESXi can be upgraded to VI Foundation Standard and Enterprise Editions to provide additional management features as detailed in Table 1 6 Table 1 6 Feature comparison ESX Host Server Number of Cores Scheduling opportunities VM 2 vCPUs 4 way Dual Core 8 28 8 way Dual Core 16 120 8 way Quad Core 32 496 Feature...

Page 41: ...Mware HA VMware ESX Server VMware ESXi VMware Consolidated Backup VMware Update Manager VMware Infrastructure Foundation Edition Unlike the previous VMware Infrastructure 3 Starter Edition VMware Infrastructure Foundation Edition has no restrictions on shared storage connectivity memory utilization or number of CPUs of the physical server It includes VMware ESX Server VMware ESXi VMware Consolidat...

Page 42: ...on that is supported on the x3850 M2 and x3950 M2 single node Table 1 7 Features of the VMware ESX family Note For all VMware VI3 Infrastructure editions Enterprise Standard and Foundation two Socket licenses must be purchased with a corresponding subscription and support for the VI3 Edition purchased The licenses are also valid for use with ESXi Installable Edition VMware ESXi is now available fr...

Page 43: ...ubscriptions for updates to VMware ESXi V3 5 are recommended but not mandatory to be purchased for each ESXi V3 5 four socket license using product number 5773 VMW Subscription for two processor sockets Feature code 0997 Subscription for four processor sockets Feature code 0998 For more details see the IBM Announcement Letter 208 071 http www ibm com isource cgi bin goto it usa_annred on 208 071 L...

Page 44: ... support License upgrades subscription upgrades and support must be purchased as a complete set to upgrade ESXi Edition to Virtual Infrastructure Foundation Standard and Enterprise Editions 1 5 5 Support for applications running on VMware ESX and ESXi Ensure that the applications you plan to use on the x3850 M2 and x3950 M2 running VMware ESX Server are supported by the application vendor Microsof...

Page 45: ...ion The IBM VMware Support Line is available to provide assistance in working with the customer and the VMware Business Partner to resolve this type of problem Customer implementations that are not covered by an IBM VMware Support Line agreement are required to re create the problem in a native environment without VMware in order to use Software Maintenance support services for the IBM software th...

Page 46: ...and peak write bandwidth is 17 1 GBps The memory controller routes all traffic from the eight memory ports four microprocessor ports and the three PCIe bridge ports The memory controller also has an embedded DRAM that in the x3850 M2 and x3950 M2 holds a snoop filter lookup table This filter ensures that snoop B bytes b bits Seven PCI Express x8 slots slots 6 7 are hot swap IBM X4 Architecture cor...

Page 47: ...rate of 2 GBps The Southbridge supplies all the other onboard PCI devices such as the USB ports onboard Ethernet and the standard RSA II 1 6 1 Hurricane 4 Hurricane 4 is the north bridge component of the IBM eX4 chipset designed for latest Intel Core Architecture based processors which feature a new architecture for the processor front side bus Hurricane 4 supports the processors in the Xeon 7000 ...

Page 48: ... chip Two single chip PCIe host bridges are designed to support PCIe adapters for IBM x3850 M2 and x3950 M2 servers The PCIe bridges each have one HSS IB port that provide connectivity to Hurricane 4 memory controller chip and also another HSS IB link between the PCIe bridges for redundancy in the event one of the links from the Hurricane 4 chipset to the two PCIe bridges are not working The HSS I...

Page 49: ... DRAM is sometimes written on the label of the DIMM such as a DRAM chip on a DIMM DIMMs are typically configured as either single rank or double rank devices but four rank devices are becoming more prevalent The DRAM devices that make up a rank are often but not always mounted on one side of the DIMM so a single rank DIMMs can also be referred to as a single sided DIMM Likewise a double ranked DIM...

Page 50: ...t Snoop filter is for quad FSB coherency tracking compared to X3 with only dual FSB coherency tracking Memory Increased four fold memory capacity 2X from chipset 2X from DRAM technology compared to X3 66 Six PCI X 2 0 slots 64 bit 266 MHz IBM XA 64e core chipset CPU 2 CPU 3 6 5 4 DDR2 SMI2 DDR2 SMI2 DDR2 SMI2 DDR2 SMI2 CPU 1 CPU 4 Memory controller Hurricane 3 2 1 HDD backplane Adaptec SAS Gigabit...

Page 51: ...cated PCIe slot the ServeRAID MR10k has improved throughput and ability to support external SAS devices through the integrated external SAS port Dedicated Southbridge ESI port to support Southbridge devices such as RSA II dual Gigabit Ethernet controllers IDE DVD ROM USB port Serial port and Video interface Scalability Almost twice the increase in scalability port bandwidth for improved scaling Th...

Page 52: ...e from the top of the server after opening the media hood The media hood is hinged at the middle of the system and contains the SAS drives optical media USB ports and light path diagnostic panel Figure 1 14 shows the media hood half way open Figure 1 14 The x3950 M2 with the media hood partially open The processors are each packaged in the 604 pin Flip Chip Micro Pin Grid Array FC mPGA package It ...

Page 53: ... quad core processors Figure 1 15 Comparing the dual core and quad core Tigerton The Xeon E7400 Series Dunnington processors both 4 core and 6 core models have shared L2 cache between each pair of cores but also have a shared L3 cache across all cores of the processor While technically all Dunnington Xeon processors have 16MB of L3 cache 4 core models only have 12MB of L3 cache enabled and availab...

Page 54: ... gain in performance The Tigerton quad core processors add two more cores onto the same die and some Dunnington processors also add two more Hyper Threading Technology is not supported Each core has separate L1 instruction and data caches and separate execution units integer floating point and so on registers issue ports and pipelines for each core A multi core processor achieves more parallelism ...

Page 55: ...Although the names of these extensions suggest that the improvements are simply in memory addressability Intel 64 Technology is in fact a fully functional 64 bit processor The processors in the x3850 M2 and x3950 M2 include the Intel 64 Technology extensions from Intel This technology is compatible with IA 32 software while enabling new software to access a larger memory address space To realize t...

Page 56: ...2 bit process on this system gets its very own 4 GB of physical memory space provided sufficient physical memory is installed This is already a huge improvement compared to IA32 where the operating system kernel and the application had to share 4 GB of physical memory Because the compatibility mode does not support the virtual 8086 mode real mode applications are not supported However sixteen bit ...

Page 57: ...r can hold 256 GB of RAM With four nodes the combined complex can hold up to 1 TB RAM With a multinode configuration the memory in all nodes is combined to form a single coherent physical address space For any given region of physical memory in the resulting system certain processors are closer to the memory than other processors Conversely for any processor some memory is considered local and oth...

Page 58: ...ata automatically without issuing a Predictive Failure Analysis PFA or light path diagnostics alerts to the administrator although an event is recorded to the service processor log After the second DIMM failure PFA and light path diagnostics alerts would occur on that DIMM as normal Memory scrubbing Memory scrubbing is an automatic daily test of all the system memory that detects and reports memor...

Page 59: ...it does not require special Chipkill DIMMs and is transparent to the operating system When combining Chipkill with Memory ProteXion and Active Memory the x3850 M2 and x3950 M2 provides very high reliability in the memory subsystem When a memory chip failure occurs Memory ProteXion transparently handles the rerouting of data around the failed component as previously described However if a further f...

Page 60: ...failure without shutting down 3 Provided that memory mirroring is enabled the third chip failure on that port sends the alert and takes the DIMM offline but keeps the system running out of the redundant memory bank 1 9 SAS controller and ports The x3850 M2 and x3950 M2 have a disk subsystem comprised of an LSI Logic 1078 serial attached SCSI SAS controller and four internal 2 5 inch SAS hot swap d...

Page 61: ...slots and devices PCI Express x8 40 Gbps full duplex assumes PCIe v1 1 x1 capable of maximum 2 5 Gbps unidirectional or half duplex Slots 6 and 7 also support Active PCI hot swap adapters PCI Express x8 x8 lanes slot 1 PCI Express x8 x8 lanes slot 2 PCI Express x8 x8 lanes slot 3 PCI Express x8 x8 lanes slot 4 PCI Express x8 x8 lanes slot 5 PCI Express x8 x8 lanes slot 6 PCI Express x8 x8 lanes sl...

Page 62: ...dual 10 100 1000 Ethernet controller that uses the Broadcom NetXtreme II BCM5709C controller The controller contains two standard IEEE 802 3 Ethernet MACs which can operate in either full duplex or half duplex mode 1 11 1 Main features The Broadcom NetXtreme II dual port Gigabit capable Ethernet ports have the following main features Shared PCIe interface across two internal PCI functions with sep...

Page 63: ...ng until the fan can be hot swapped by the IT administrator In general failed fans should be replaced within 48 hours following failure The two Gigabit Ethernet ports can be configured as a team to form a redundant pair The memory subsystem has a number of redundancy features including memory mirroring and Memory ProteXion as described in 1 8 Memory subsystem on page 39 RAID disk arrays are suppor...

Page 64: ...t of the x3850 M2 and x3950 M2 showing the location of the memory cards power supplies and fans is displayed in Figure 1 20 Figure 1 20 Redundant memory fans and power supplies features Memory redundancy features memory mirroring Memory ProteXion and Chipkill Six hot swap fans one in each pair is redundant Two hot swap redundant power supplies one below the other Two hot swap redundant 1440 W powe...

Page 65: ... monitored server subsystems In the event that maintenance is required the customer can slide the server out of the rack and using the LEDs find the failed or failing component Light path diagnostics can monitor and report on the health of microprocessors main memory hard disk drives PCI adapters fans power supplies VRMs and the internal system temperature See 6 1 BMC configuration options on page...

Page 66: ...o determine if the system configuration is safe for power on BMC event log maintenance The BMC maintains and updates an IPMI specified event log in nonvolatile storage Critical system information is recorded and made available for external viewing System power state tracking The BMC monitors the system power state and logs transitions into the system event log System initialization The BMC has I2C...

Page 67: ...is installed in a dedicated PCI 33 MHz slot and provides functionality similar to the Remote Supervisor Adapter II PCI option available for other System x servers However only the Ethernet and video connectors are used on the x3850 M2 and x3950 M2 The other external ports including remote power and the ASM interconnect are not supported on these servers The video adapter on this RSA II card is an ...

Page 68: ...onitors all important system parameters such as temperature voltage and so on If a fan fails for example the RSA II forces the remaining fans to increase speed to compensate for the failing fan Event log You can access the server event logs and the power on self test POST log and export them while the server is running Operating system failure window capture When the operating system hangs for exa...

Page 69: ...the consumption if required The application software enables you to track actual power consumption trends and corresponding thermal loading of servers running in your environment with your applications Active Energy Manager enables customers to monitor actual power draw and thermal loading information This helps you with More efficient planning of new datacenter construction or modification Proper...

Page 70: ... has support written into its code to use the security features of this module The TPM is TCG V1 2 compliant and is ready for use with software purchased from the third party list of the TPM Ecosystem partners who are also in compliance with the TPM V1 2 specification TPM can be used for the following purposes Disk encryption For example BitLocker Drive Encryption in Windows Server 2008 Digital Ri...

Page 71: ...r additional processing expandability and high availability features over those of their predecessors the IBM System x3850 and x3950 servers They are ideal for handling complex business critical On Demand Business applications that must be supported by space saving rack optimized servers This chapters discusses the following topics 2 1 Focus market segments and target applications on page 54 2 2 P...

Page 72: ...ons use an external storage enclosure or SAN depending on the size of the database which is driven by the number of users The 16 way four node configuration can deliver a highly reliable and capable platform for clients who have to run multiple instances of databases that can scale beyond eight processors Key database software vendors are Microsoft SQL Server 2005 and 2008 IBM DB2 and Oracle Enter...

Page 73: ...iler to consumer SCM involves coordinating and integrating these flows both within and among companies The application also can use a four tier three tier or two tier architecture Key SCM software vendors are I2 SAP SAP Business Suite and SAP Business All in One Oracle JD Edwards and PeopleSoft and International Business System IBS Business Intelligence BI BI is a broad category of applications an...

Page 74: ...anagement features to the rack optimized two way servers with increased memory and I O expansion higher levels of redundancy and increased system management features The IBM eX4 servers are part of the IBM System x scale up server offering which is designed to provide the highest level of processor scalability with support for up to 16 multi core processors up to 1 TB of addressable memory with hi...

Page 75: ...le up portfolio of products Scale up SMP computing Scale out distributed computing BladeCenter E x3550 cale up SMP computing tended Design Architecture tended Design Architecture 3550 x3650 X3400 x3500 0 x3250 X34 X x3850 M2 Clusters and virtualization High density Large symmetrical multiprocessing SMP x3200 M2 x3950 M2 Cluster 1350 x3100 x3455 DS3200 3300 3400 Storage Servers S3200 3300 34 0 00 0...

Page 76: ...ons are stateless and use software for workload allocation across all nodes For more information about iDataPlex refer to the paper Building an Efficient Data Center with IBM iDataPlex REDP 4418 available from http www redbooks ibm com abstracts redp4418 html IBM BladeCenter IBM BladeCenter products are designed for complete infrastructure integration ease of management energy efficient servers ha...

Page 77: ...y and choice in creating customized infrastructures and solutions BladeCenter Open Fabric Manager can virtualize the Ethernet and Fibre Channel I O on BladeCenter BladeCenter has a long life cycle and preserves system investment with compatible proven field tested platforms and chassis For more information about IBM BladeCenter refer to the IBM Redbooks publication IBM BladeCenter Products and Tec...

Page 78: ...0 s Adaptec ServeRAID 8i RAID controller which shared a slower common PCI X 66 MHz bus to the Southbridge with the onboard Broadcom Gigabit Ethernet controllers The Hurricane 4 has a dedicated Enterprise Southbridge Interface ESI for the dual port onboard PCIe x4 Broadcom 5709C controllers RSA II Video USB 2 0 and Serial interfaces The x3850 M2 also has an onboard 4x SAS port which can be used in ...

Page 79: ... 8i Standard RAID 0 and RAID 1 additional RAID features through optional ServeRAID MR10k PCI X slots Two or six depending on model None PCI Express slots Some models have four PCI Express x8 full length slots Seven PCI Express x8 half length slots Active PCI slots hot swap Six Two Video controller ATI Radeon 7000M 16 MB onboard ATI ES1000 16 MB memory on RSA II USB ports Three front one rear two S...

Page 80: ...uffer chips Furthermore the x3950 M2 has support for more I O slots from previously having six PCI X slots to seven PCIe slots The onboard LSI 1078 RAID controller and the optional ServeRAID MR10k installed in a dedicated PCIe x8 slot have significantly improved storage subsystem bandwidth compared to the x3950 s Adaptec ServeRAID 8i RAID controller which shared a slower common PCI X 66 MHz bus to...

Page 81: ...m of 64 cores with 6 core Dunnington processors the maximum core count is 96 Memory 16 DDR2 DIMM sockets per node Maximum of four memory cards each with four DDR2 DIMM slots running at 333 MHz 64 GB maximum per node 512 GB maximum with eight nodes 32 DDR2 DIMM sockets per node Maximum of four memory cards each with eight DDR2 DIMM slots running at 533 MHz 256 GB maximum per node 1 TB maximum with ...

Page 82: ...not guarantee a performance benefit For example not all applications can use the full power of four processors in one server File and print servers often only take advantage of one or two processors and popular mail systems typically only scale well to four processors Table 2 3 on page 65 shows the suitability of multi processor systems to application types Active PCI slots Hot Swap Six Two Ethern...

Page 83: ...ents for example if the operating system does not support more than four processors or your server is not able to hold more than seven PCIe adapters Table 2 4 shows how application types scale and what is required to achieve peak performance This table lists the server configurations used to produce several recent benchmark results As you can see the amount of memory and disks varies widely depend...

Page 84: ...wing operating systems VMware ESX Microsoft Windows Server 2003 Microsoft Windows Server 2008 and Hyper V Linux server operating systems In the single chassis 4 way configuration the IBM eX4 server acts as an industry standard symmetric multiprocessor SMP system Each processor has equal access to all system resources In most industry standard SMP systems scaling beyond 4 way configurations has inh...

Page 85: ...ocessor some memory is considered local and other memory is remote The term NUMA is not completely correct because memory and I O resources can be accessed in a non uniform manner PCIe and USB devices may be associated with nodes The exceptions to this situation are existing I O devices such as DVD ROM drives which are disabled because the classic PC architecture precludes multiple copies of these...

Page 86: ...ing administrators to manual oversee the complex task of balancing VMs across multiple NUMA nodes VMware ESX does provide manual override controls for administrators with advanced skills to optimize their systems to the specific requirements of their environments These optimizations work seamlessly regardless of the types of guest operating systems running VMware ESX provides transparent NUMA supp...

Page 87: ...m examines the loads of the various nodes and determines whether it should rebalance the load by moving a virtual machine from one node to another This calculation takes into account the relative priority of each virtual machine to guarantee that performance is not compromised for the sake of fairness The rebalancer selects an appropriate VM and changes its home node to the least loaded node When ...

Page 88: ...ch cases VMware ESX systems use a proprietary transparent page sharing technique to securely eliminate redundant copies of memory pages With memory sharing a workload running in virtual machines often consumes less memory than it would when running on physical machines As a result higher levels of overcommitment can be supported efficiently Transparent page sharing for VMware ESX systems has also ...

Page 89: ... only the processors on a given node Memory affinity The server should allocate memory only on the specified node If both options are set before a virtual machine starts the virtual machine runs only on the selected node and all of its memory is allocated locally An administrator can also manually move a virtual machine to another node after the virtual machine has started running In this case the...

Page 90: ... experimentally by VMware To enable experimental support for systems with up to 64 logical processors in ESX Server 3 5 run the following commands in the service console and then reboot the system esxcfg advcfg k 64 maxPCPUS esxcfg boot b SATA support ESX Server 3 5 supports selected SATA devices connected to dual SAS SATA controllers For a list of supported dual SAS SATA controllers see the ESX S...

Page 91: ...ultinode complex Windows Server 2003 Enterprise and Datacenter Editions are NUMA aware and are able to assign application threads to use processor and memory resource pools on local NUMA nodes Scheduling application threads to run on local resource pools can improve application performance because it minimizes internode traffic on the x3950 M2 scalability ports and also reduces contention for reso...

Page 92: ...64 release Yes Yes Yes No 64 bit release Itanium Yes Yes Yes No Scalability Maximum Processors Sockets 4 8 32 bit 32 x64 64a a At the time of writing Windows 2003 Datacenter Edition x64 is licensed for up to 64 sockets but can detect a maximum of only 64 cores Windows 2003 Datacenter Edition 32 bit is licensed for up to 32 sockets but can detect a maximum of only 64 cores 2 Number of x3950 M2 node...

Page 93: ...s available from http technet microsoft com en us magazine cc194386 aspx Table 2 6 lists features supported by the various Windows Server 2008 editions Table 2 6 Features of the Windows Server 2008 family Features Standard Edition Enterprise Edition Datacenter Edition Web Edition Edition availability 32 bit releasea Yes Yes Yes Yes x64 release Yes Yes Yes Yes 64 bit release Itanium b No No No No S...

Page 94: ...nterprise and Datacenter Editions are not yet supported by IBM on x3850 M2 x3950 M2 single or multinode Windows 2008 Web Edition 32 bit and 64 bit are not supported on IBM x3850 M2 or x3950 M2 b Microsoft has released a separate Windows Server 2008 Itanium Edition for IA64 to be used solely on Itanium based hardware platforms Previously Windows Server 2003 family had IA64 versions of Windows Serve...

Page 95: ... core s cache data and also for reducing the likelihood of a remote CPU core having to access data in the local node s memory The ratio of local node to remote node memory accesses made by all CPU cores Remote memory accesses should be kept to a minimum because it increases latency and reduces the performance of that task It can also reduce the performance of other tasks because of the contention ...

Page 96: ...d guests instances 4 Unlimited Technology capabilities and limits Maximum Logical Processors x86 a a Red Hat defines a logical CPU as any schedulable entity So every core thread in a multi core thread processor is a logical CPU 8 assuming maximum of quad core processors limited by subscription to 2 sockets 32 Maximum Logical Processors EM64T and AMD64 a 8 limited by subscription to 2 sockets 64 ce...

Page 97: ...nagement CRM Enterprise Resource Planning ERP Supply Chain Management SCM Enterprise applications work with your most critical business data so it is important that these applications are highly available and secure As as shown in Figure 2 5 on page 80 the following three general architectures are used by these applications A three tier architecture often referred to as an Internet architecture in...

Page 98: ...on Figure 2 5 Enterprise solution architectures 2 7 1 Microsoft SQL Server 2005 Microsoft SQL Server 2005 has many features that allow it to scale from a small single user database to a huge enterprise wide mission critical multi user database This section highlights these features and discusses how they are relevant to server consolidation Support for 64 bit computing x64 The combination of Windo...

Page 99: ... new databases created with SQL Server 2005 and demonstrates the ease with which databases can be scaled up on this platform Hot add memory Additional physical memory can be installed in a running server and SQL Server 2005 will recognize and use the additional memory immediately This could prove useful if you must increase available memory to service new business requirements without affecting da...

Page 100: ...roup CPUs you must edit the registry directly using a node configuration affinity mask After the soft NUMA nodes have been created you can assign individual SQL Server instances to one or more soft NUMA nodes You might create soft NUMA nodes if your server hardware does not have hardware NUMA capabilities or to sub divide a NUMA node further Each soft NUMA node gets its own I O thread and lazy wri...

Page 101: ...ropped to record lows organizations can store more data per dollar Users must be able to examine and analyze this data quickly and easily on any device using their regular office productivity programs The management of this information explosion and hike in user expectations creates severe challenges for the enterprise Microsoft has positioned its new database platform as the answer to these chall...

Page 102: ... scaling up is that it is relatively easy because in general it requires only hardware or software that is designed to take advantage of additional memory processor and storage resources With the mass adoption of virtualization as a means for server consolidation and driving up resource utilization on under utilized mulit core processor based systems virtualization infrastructures are increasingly...

Page 103: ...ivers investment protection without the up front costs of expensive switch based alternatives Advantages of scale up include Easier to configure and administer Good when most queries access small blocks of data Best for applications that maintain state OLTP Add CPU and memory as required scheduled downtime especially for CPUs All tools and queries work as expected Can be maintained by lesser skill...

Page 104: ...ieve near linear scalability That is as you add each additional server to the system you effectively increase your system capacity proportionally Thus scaling out provides much better returns in terms of the additional costs associated with adding more servers to the system Another benefit inherent with scaling out is that a cluster of smaller servers generally costs less than a single large syste...

Page 105: ...t that as more servers are added to the system each server manages a clearly defined portion of the database The fact that the database is partitioned should not imply that the system cannot be implemented on shared storage IBM DB2 and Microsoft SQL Server both implement a shared nothing approach Choosing scale up or scale out Microsoft SQL Server 2005 and SQL Server 2008 are well suited for scale...

Page 106: ...88 Planning Installing and Managing the IBM System x3950 M2 ...

Page 107: ...derstand and configure before the hardware configuration at your x3850 M2 or x3950 M2 is completed This chapters discusses the following topics 3 1 Processor subsystem on page 90 3 2 Memory subsystem on page 111 3 3 Internal drive options and RAID controllers on page 124 3 4 Configuring RAID volumes on page 154 3 5 PCI Express options on page 188 3 ...

Page 108: ...ch server can be upgraded to a maximum of four processors One two three or four processors are supported Installed processors must be identical in model speed and cache size Figure 3 1 on page 90 shows the locations of the four processors CPUs locations of the required voltage regulator modules VRMs which you can identify by the blue handles and the memory cards that we describe later in 3 2 Memor...

Page 109: ...will automatically reboot without the memory resources and I O resources that were in that node Therefore from a system availability perspective we recommend you have at least two processors in every node Processor Cores Speed GHz L2 L3 cache TDPa a Thermal Design Power TDP The thermal specification shown is the maximum case temperature at the maximum TDP value for that processor It is measured at...

Page 110: ... MB or 4 MB of shared L2 cache for a total of 4 MB 6 MB or 8 MB of L2 cache The L2 cache implements the Advanced Transfer Cache technology L3 cache Tigerton processors do not have L3 cache Processor Cores Speed L2 L3 cache TDPa a For a description of TDP see footnote a in Table 3 1 Part number Feature codeb b The first feature code is the base configuration Additional processor orders should use t...

Page 111: ...al level of cache saves decoding time on cache hits L2 cache Each pair of cores in the processor has 3 MB of shared L2 cache for a total of 6 MB or 9 MB of L2 cache The L2 cache implements the Advanced Transfer Cache technology L3 cache Dunnington processors have 12 MB 4 core or 16 MB 6 core shared L3 cache 3 1 2 Installation of processor options The processors are accessible from the top of the s...

Page 112: ...and heat sink included in the option packaging If the VRM is missing the following error message is displayed and the system does not power up Processor configuration missmatch error To install the processor options 1 Turn off the server and peripheral devices 2 Disconnect the power cords 3 Wait approximately 20 seconds then make sure the blue LED light is off ...

Page 113: ...stallation and removal of processors or VRM can result in damage if the system is not removed from the AC power source Check that the system is without power after removal of the power cables The blue locator LED is located on the rear side of the server This LED indicates whether AC power is connected to the system but the system is powered off The LED remains lit for up to 20 seconds after you r...

Page 114: ...tion of the socket before you install the processor in either of these sockets Figure 3 5 on page 97 shows the orientation of the sockets 10 Note that the processor air baffle is always located between socket 1 and socket 2 as shown in Figure 3 1 on page 90 11 Lift the processor release lever to the fully opened position which is approximately a 135 angle See in Figure 3 5 on page 97 Fan 4 Fan 1 F...

Page 115: ...essor into the socket Close the processor release lever Figure 3 6 Processor orientation 13 Remove the heat sink from its package and remove the cover from the bottom of it Hint The heat sink in the processor option kit is already covered with the correct amount of thermal grease Lever closed Lever fully opened Processor orientation indicator Processor release lever Processor Processor socket ...

Page 116: ... Put the heat sink on the processor press on the top of the heat sink and rotate the heat sink retention lever up until it is locked Figure 3 7 Heat sink installation 15 Ensure that the air baffle between processors is correctly installed as shown in Figure 3 1 on page 90 16 Install a VRM in the connector next to the processor socket The VRM and handle are shown in Figure 3 8 on page 99 Alignment ...

Page 117: ...essor CPU configuration options The BIOS includes various processor settings which you can adjust to the installed operating system and performance purposes To access the processor settings 1 Power on or reboot the system 2 When prompted during system startup press F1 3 In the Main Menu window select Advanced Setup 4 Select CPU Options The selection menu in Figure 3 9 on page 100 opens Note Make s...

Page 118: ...t this option to enable or disable the Active Energy Manager Power Capping When power capping is enabled the Active Energy Manager application can limit the maximum power that this system consumes Active Energy Manager is part of a larger power management implementation that includes hardware and firmware components Use Active Energy Manager to manage the power and thermal requirements of IBM serv...

Page 119: ...passive cooling mechanism which allows the platform to control and indicate the temperature at which clock throttling for example will be applied to the processor residing in a given thermal zone Unlike other cooling policies during passive cooling of processors operating system power management OSPM may take the initiative to actively monitor the temperature in order to control the platform Proce...

Page 120: ...es to run The Processor Performance States option shown in Figure 3 9 on page 100 is disabled by default you enable it in BIOS The following sections show examples in Linux Windows Server 2003 Windows 2008 and how you can identify the operating system power management Linux Various distributions of Linux operating systems such as SUSE SLES10 SP2 or RHEL5U2 based on kernel 2 6 integrate power manag...

Page 121: ...Change the governor to USERSPACE cpufreq set g USERSPACE Example 3 5 Command to change the frequency cpufreq info f c 5 1862000 cpufreq set c 5 f 2128000 cpufreq info f c 5 2128000 Windows Server 2003 To utilize processor performance states Windows Server 2003 must be using a power policy that enables DBS By default the power scheme Always On is set This has the effect of running the system at ful...

Page 122: ...ed driver can be found in the following folder SYSTEMDRIVE Windows Inf SYSTEMDRIVE Windows Inf For Intel processors the Intelppm sys file is used Because the Intel Xeon is not listed in the support matrix the Microsoft generic driver Processr sys is used instead until the support is available The power policies can be adjusted in the Power Options window Each processor power policy includes an upp...

Page 123: ... power consumption efficiency see the following Microsoft Web page http www microsoft com whdc system pnppwr powermgmt ProcPowerMgmt mspx Clustering Technology For certain operating systems you must configure how the routing of processor interrupts in a multi processor system is handled A low level value sets the multi processor interrupt communication protocol XAPIC The settings are functional on...

Page 124: ...s created temporarily to allow 64 bit Linux operating systems to run on the eX4 technology servers and the older X3 technology servers if they do not support clustering Although flat mode is not really a clustering mode it is a mode by which most non clustering operating systems can work simply because it abstracts them from the real physical clustering of the processors as a result of the archite...

Page 125: ...cessor agents The actual number of processor agents supported by a particular Linux distribution depends on the vendor The assumption is that the newer IA32E 64T distributions will support greater than 32 agents Windows operating systems Windows Server does not require a custom hardware abstraction layer HAL because it has been incorporated into 2003 SP1 in order to support scaled systems The 32 b...

Page 126: ...r Execute Disable Bit Processor Execute Disable Bit EDB or XD is a function of new Intel processors which lets you prevent the execution of data that is in memory as though it was code When this setting is enabled the default viruses or other malicious code are prevented from gaining unauthorized access to applications by exploiting buffer overruns in those applications If this option is enabled a...

Page 127: ...istance or stride between subsequent accesses to the same memory area The IP prefetcher then generates a prefetch request with the predicted address and brings the resulting data to the Level 1 data cache This setting is disabled by IBM the opposite of the Intel default The observations in the IBM performance lab have shown that this prefetcher is negligible in most real applications Although Inte...

Page 128: ...ssor to alter the core frequency after sending an idle command such as HALT or MWAIT The processor core speed slows down and then transitions to the lower voltage Enhanced Halt State is a low power state entered when all processor cores have executed the HALT or MWAIT instructions and Extended HALT state has been enabled When one of the processor cores executes the HALT instruction that processor ...

Page 129: ... FSB frequency is not altered only the internal core frequency is changed When entering the low power state the processor first switches to the lower bus to core frequency ratio and then transitions to the lower voltage While in the Extended HALT state the processor will process bus snoops Important All changes you make within the BIOS affects the particular local node only Any changes you make mu...

Page 130: ...e have better timing parameters than the 533 MHz equivalent The memory throughput is 4 26 GBps or 533 MHz x 8 bytes per memory port for a total of 34 1 GBps with four memory cards The server supports up to four memory cards See Figure 3 12 on page 113 Each memory card holds up to eight DIMMs to allow thirty two DIMMs per chassis 43V7356c 3938 16 GB kit 2x 8 GB DIMM PC2 5300 CL5 ECC DDR2 667 MHz SD...

Page 131: ...ds use the part number in Table 3 4 Table 3 4 Memory card part number Fan 4 Fan 1 Fan 6 Fan 5 Fan 3 Fan 2 Power Backplane Power Backplane VRM 3 VRM 4 VRM 1 VRM 2 Memory Card 1 Memory Card 2 Memory Card 3 Memory Card 4 CPU 1 CPU 2 CPU 3 CPU 4 Fan 4 Fan 1 Fan 6 Fan 5 Fan 3 Fan 2 Power Backplane Power Backplane VRM 3 VRM 4 VRM 1 VRM 2 Memory Card 1 Memory Card 2 Memory Card 3 Memory Card 4 CPU 1 CPU ...

Page 132: ...space If you replace the standard pair of DIMMs and install 32x 8 GB DIMMs and four memory cards both the x3850 M2 and x3950 M2 can be expanded to 256 GB The XceL4v Dynamic Server Cache consumes 256 MB in each chassis of the main memory for use as L4 cache if a multinode system is formed therefore Note When you install additional DIMMs on a memory card or a new memory card make sure they are insta...

Page 133: ... is reflected in the power on self test POST with the addition of a new line of text specifying the amount of available system main memory after the L4 scalability cache for each node has been subtracted Example 3 6 shows what you see in a two node complex Example 3 6 Two node system 64GB Memory Installed 512MB Memory Consumed by Scalability To replace or add any DIMMs remove one or more of the in...

Page 134: ...s have the part number 44E4252 Two or four are standard in the x3850 M2 and x3950 M2 depending on the model A maximum of four memory cards can be installed in a node See the Reference Sheets xREF for specific information about models currently available http www redbooks ibm com Redbooks nsf pages xref Each memory card has eight DIMM sockets The three ways to fill the DIMMs sockets depending on wh...

Page 135: ...uence Reliability increased configuration To improve the reliability of your system fill your memory cards as listed in Table 3 7 Depending on the memory population if a DIMM fails it can be removed and replaced by a new DIMM The use of mirroring halves the memory that is available for the operating system Table 3 7 Memory mirroring configuration DIMM pair Memory card Connector 1 4 1 1 5 2 6 3 7 4...

Page 136: ...ockets and Memory card LEDs are shown in Figure 3 14 on page 116 Mirroring takes place across two memory cards as follows The memory DIMMs in card 1 are mirrored to the memory DIMMs in card 2 The memory DIMMs in card 3 are mirrored to the memory DIMMs in card 4 Therefore with memory mirroring enabled in the BIOS you can hot swap any memory card if the hot swap enabled LED is lit For instructions t...

Page 137: ...ilability Hot swap memory requires that memory mirroring be enabled This setting can be changed as described in section 3 2 6 Memory configuration in BIOS on page 121 To easily identify whether hot swap is enabled and the status of power to the memory card each memory card has a green memory hot swap enabled LED and a green memory card power LED on the top panel of the memory card as shown in Figu...

Page 138: ...y feature enables you to add DIMMs without turning off the server This setting can be changed as described in 3 2 6 Memory configuration in BIOS on page 121 Requirements for enabling the hot add memory feature on the server are The operating system must support the adding of usable system memory to a running operating system This is done with an ACPI sequence Currently the only operating systems t...

Page 139: ...ww ibm com support docview wss uid psg1MIGR 5073029 3 2 6 Memory configuration in BIOS The BIOS includes settings related to processor and operating system configuration and performance purposes You can adjust the settings To access the settings and configure the memory subsystem in the server s BIOS setup 1 Power on or reboot the system 2 Press F1 during system startup when prompted 3 From the Ma...

Page 140: ...nable only one of these features If you plan to enable hot add memory and you have a x3850 M2 or x3950 M2 system that comes standard with two memory cards move memory card 2 to slot 3 to be able to hot add memory cards in slots 2 and 4 After you add a memory card that has two DIMMs you cannot add more memory to that same memory card without powering off the server Enabling hot add reserves a porti...

Page 141: ...orrectable memory error is still detected and the failing memory device or devices are logged in the system event logs Run Time Scrub Rate Use this setting to configure the rate of the run time hardware memory scrub engine The hardware memory scrub engine is responsible for checking system memory looking for correctable or uncorrectable memory errors If you set the scrub rate to default the chipse...

Page 142: ...0 M2 and x3950 M2 SAS storage subsystem The LSI 1078 SAS controller operates in the Integrated RAID IR mode that supports the internal port by communicating through a x4 SAS cable connection to the SAS 4 port hot swap backplane The SAS LSI1078 IR controller has the following features RAID level 0 Integrated Mirroring IM RAID level 1 Integrated Striping IS Two disk IM mirrored volumes Two volumes m...

Page 143: ...ing of up to two hot spare drives which will also be imported to the MR mode Figure 3 17 shows the internal SAS cable with the hot swap backplane The cable is an industry standard MiniSAS 4i cable with SFF 8087 cable and board connectors from the I O board to the hot swap backplane Figure 3 17 x3850 M2 and x3950 M2 showing internal SAS cabling The hot swap backplane is mounted in the media hood as...

Page 144: ...or By upgrading your server with the ServeRAID MR10k adapter you enable further RAID features This card works in the MegaRAID MR mode The adapter is described in 3 3 3 ServeRAID MR10k RAID controller on page 128 Note For significant performance reasons IBM recommends you attach an EXP3000 to the external SAS port only when you have the ServeRAID MR10k installed Connecting an EXP3000 to the externa...

Page 145: ...embly in the middle of the server as shown in Figure 3 20 Figure 3 20 Media hood hard disk drive bays Depending on the level of RAID you want to configure up to 584 GB of disk space can be used internally by using four 146 GB drives Table 3 10 shows the supported disks Table 3 10 Supported internal disk options for x3850 M2 and x3950 M2 Important To prevent damage to the disk we recommend you firm...

Page 146: ...re 3 21 Figure 3 21 Optional installed ServeRAID MR10k The ServeRAID MR10k supports stripe sizes from 8 KB to 1024 KB The default stripe size is 128 KB The SAS drive can be driven with up to 3 GBps for each port in full duplex mode The controller supports up to 64 virtual disks and up to 64 TB logical unit numbers LUNs It supports up to 120 devices on the external x4 port IBM supports the use of u...

Page 147: ... survive the loss of two disks without losing data RAID 10 A combination of RAID 0 and RAID 1 consists of striped data across mirrored spans It provides high data throughput and complete data redundancy but uses a larger number of spans RAID 50 A combination of RAID 0 and RAID 5 uses distributed parity and disk striping and works best with data that requires high reliability high request rates hig...

Page 148: ...he assigned virtual drive only Rebuild and rapid restore features These reliability features help to secure your data Check consistency This feature fixes media errors and inconsistencies Patrol read This background operation also known as data scrubbing checks for media errors in configured drives Selectable boot virtual disk The first eight virtual disks can be chosen as a boot device Intelligen...

Page 149: ...tes data by using the system management bus SMBus 2 wire protocol The data available includes the battery s remaining capacity temperature voltage current and remaining run time predictions The ServeRAID MR10k controller and the iTBBU battery package shown in Figure 3 22 can be ordered with the part number in Table 3 11 Table 3 11 Part number for ServeRAID MR10k Figure 3 22 ServeRAID MR10k and iTB...

Page 150: ...grated RAID controller Important Before the battery in the iTBBU can effectively protect from a failure it must be charged for at least six hours under normal operating conditions To protect your data the firmware changes the Write Policy to write through effectively bypassing the cache and battery until the battery unit is sufficiently charged When the battery unit is charged the RAID controller ...

Page 151: ...r 5 Touch the static protective package that contains the DIMM to any unpainted metal surface on the outside of the server then remove the DIMM from the package 6 Turn the DIMM so that the keys align correctly with the slot 7 Insert the DIMM into the connector by aligning the edges of the DIMM with the slots at the ends of the connector 8 Firmly press the DIMM straight down into the connector by a...

Page 152: ...ServeRAID MR10k 13 Reinstall the server cover and reconnect the power cords 14 Turn on the power first to the enclosure if one is connected then to the system 15 Check that the ServeRAID MR10k controller is initialized correctly during POST The text in Example 3 7 appears Example 3 7 ServeRAID MR10k initialization in POST LSI MegaRAID SAS MFI BIOS Version NT16 Build Nov 20 2007 Copyright c 2007 LS...

Page 153: ...nd is a small form factor MD2 adapter with 2U or 3U bracket capability Use the 3U bracket for the x3950 M2 and x3850 M2 servers Figure 3 24 is a block diagram of the MR10M Figure 3 24 ServeRAID MR10M layout Each channel is attached to an external x4 port SAS SFF 8088 connector You can use these ports to connect more local disks to your x3850 M2 and x3950 M2 using external expansion boxes Read more...

Page 154: ...veRAID MR10M has the same key features as the MR10k as described in Key features on page 130 Figure 3 25 shows the components of the ServeRAID MR10M Figure 3 25 ServeRAID MR10M This controller can be ordered with the part number listed in Table 3 13 Table 3 13 ServeRAID MR10M part number Part number Description 43W4339a a Kit includes SeveRAID MR10M iBBU battery brackets and installation guide IBM...

Page 155: ...M can be used in any slot of a x3850 M2 and x3950 M2 complex The ServeRAID MR10M is not supported with the ServeRAID manager Use the MegaRAID Storage Manager MSM instead One or more arrays can be formed using up to nine cascaded EXP3000 An x4 SAS SFF8088 cable is required to attach the external storage subsystem to the ServeRAID MR10M connectors You may have 32 ServeRAID adapters installed in a 4 ...

Page 156: ...aRAID Storage Manager cannot recognize greater than 16 controllers http www ibm com support docview wss uid psg1MIGR 5076491 RETAIN tip H193591 WebBIOS only recognizes sixteen ServeRAID controllers http www ibm com support docview wss uid psg1MIGR 5076492 Installation instructions To install the ServeRAID MR10M SAS SATA controller and the battery package 1 Mount the iBBU package on the controller ...

Page 157: ...nd x3950 M2 PCIe slot location The PCIe slots are beyond the adapter retention bracket on the right at the rear of the server shown in Figure 3 26 and Figure 3 27 on page 140 PCIe adapter retention brackets blue Rear side of the server Front side of the server ...

Page 158: ...Remove the screw for the non hot plugable slot or push the orange adapter retention latch toward the rear of the server at the hot plug and open the tab That is where you add a PCIe adapter The power LED for slot 6 and slot 7 turn off 6 Insert the controller in a PCIe slot as shown in Figure 3 28 on page 141 ...

Page 159: ...e bracket screw Close the adapter retention bracket and use the retention pin as shown in Figure 3 29 on page 142 Note Secure all adapters with the retention pin Adapters can loosen during shipment or pushing the server out of the racks Edge of Motherboard iBBU Top View Bracket Screw Edge of Motherboard Press Here 85021 06 Press Here ...

Page 160: ...on HA 0 Bus 84 Dev 0 IBM ServeRAID MR10M SAS SATA Controller FW package 8 0 1 0029 10 Power off the system again You can now install the disks in the external enclosures and cable with the adapter 3 3 5 SAS expansion enclosure unit The x3850 M2 and x3950 M2 SAS storage controllers can be extended to expand your disk space by the attachment of the EXP3000 external storage expansion unit This expans...

Page 161: ...27 01X is available and shipped in the following configurations Dual hot swap redundant power supplies and two rack power cables No disk add up to 12 SAS SATA or SATA II 3 5 inch hard disk drives One ESM board 2 SAS ports a second ESM is required for attachment to dual controller models of DS3200 DS3400 Part number Description 39M4558 500 GB 3 5 inch hot swap SATA II 43W7580 750 GB 3 5 inch hot sw...

Page 162: ...6529 Performance Depending on your disk space requirement type of application and required performance the amount of disks in a number of chained EXP3000 can result in higher performance but does not increase linearly up to the maximum of nine EXP3000 enclosures with up to 108 disk drives Part number Description 39R6531 IBM 3m SAS cable 39R6529 IBM 1m SAS cable 39R6515a a Second ESM required for a...

Page 163: ...rives 3 Start with the cabling of your expansion unit by connecting the power cables a Attach one end of your 3 m SAS cable to the host port shown in Figure 3 32 on page 146 b Connect the other end to the closest EXP3000 IN port c Attach one end of the 1 m SAS cable to the EXP3000 OUT port and the other end to the IN port of the next expansion Repeat this step if you want to attach multiple expans...

Page 164: ...e IBM System x3950 M2 Figure 3 32 EXP3000 rear view cabling 4 Turn on the EXP3000 expansion units before or at the same time as you turn on the device that contains the RAID controller Figure 3 33 on page 147 shows the components of the ESM ...

Page 165: ...w prepared for configuration RAID volumes which is described in the next section Note The IBM Support for System x Web page contains useful technical documentation user guides and so on It is located at http www ibm com systems support x Enter the system Type and Model 172701X in the Quick path field to link to the information for this enclosure We recommend you also check regularly for new codes ...

Page 166: ...ard drive update program http www ibm com support docview wss uid psg1MIGR 62832 See also Updating the SAS hard disk drives on page 152 EXP3000 ESM update http www ibm com support docview wss uid psg1MIGR 5073877 See also Updating the EXP3000 on page 153 Updating the onboard SAS LSI1078 The program updates all SAS LSI1078 controllers found in the system This allows you to update all referred contr...

Page 167: ... to flash the onboard LSI 1078 controllers in you system You might see warning messages stating that this flash is not compatible with all controllers as shown in Figure 3 35 These are not error messages Figure 3 35 LSI1078 update process 4 Press Y to start the flash process See Figure 3 35 SAS Firmware BIOS Flash Disk NOTE This utility will scan all LSI 1064 1068 and 1078 based controllers in the...

Page 168: ...tes With these diskettes you can boot the server by an attached USB floppy drive or mount it in the Remote Supervisor Adapter II remote drive in the Web interface as described in 6 2 4 Remote console and media on page 324 3 Start the executable package and create three diskettes the first is bootable by selecting Extract to Floppy in Figure 3 37 on page 151 This update is for the LSI 1078 onboard ...

Page 169: ...ou might see warning messages stating that this flash is not compatible with all controllers as shown in Figure 3 38 These are not error messages Figure 3 38 ServeRAID MR10k update process 5 Press Y to start the flash process The update starts as shown in Figure 3 39 on page 152 SAS Firmware BIOS Flash Disk This program will update the firmware on all IBM ServeRAID MR10k controllers in the system ...

Page 170: ...the Web interface as described in 6 2 4 Remote console and media on page 324 3 Boot your server from this CD image 4 Update the hard drive disks by clicking the Update button The update program window is shown in Figure 3 40 on page 153 This update is for the ServeRAID MR10k controllers Controller 1 is a ServeRAID MR10k controller Attempting to flash controller 1 Updating firmware on controller 1 ...

Page 171: ...t the time of writing the latest update contained a critical fix for the ESM which fixed the problem of an unexpected reboot of the EXP3000 To update the EXP3000 1 Download the bootable CD image from the following Web address http www ibm com support docview wss uid psg1MIGR 5073877 2 Burn a CD or mount the ISO image in the Remote Supervisor Adapter II remote drive of the Web interface as describe...

Page 172: ...s on page 154 3 4 Configuring RAID volumes In this section we describe the various ways to configure your RAID volumes We provide the steps and rules for the controllers and describe the differences 3 4 1 Starting the LSI1078 controller BIOS The onboard LSI controller is initialized in POST if it is enabled in the BIOS of the server Figure 3 42 on page 155 and Figure 3 43 on page 155 shows the BIO...

Page 173: ...ultinode complex you must first break the partition and boot the nodes into standalone mode You can then change the BIOS settings as required Serial Port A Port 3F8 IRQ4 Remote Console Redirection Mouse Installed Planar Ethernet Enabled Planar SAS Enabled High Precision Event Timer HPET Enabled Video IDE Configuration Menu System MAC Addresses Devices and I O Ports Startup Sequence Options Planar ...

Page 174: ...iguration utility has three submenus RAID Properties Use to create and manage your RAID volumes SAS Topology Shows a tree of all available devices Advanced Adapter Properties Use advanced settings for the controller and devices Creating volumes To create a volume select the LSI controller you want to configure by highlighting it as shown in Figure 3 45 and then pressing Enter From the submenu that...

Page 175: ... Mirroring IM or Integrated Striping IS Select the RAID level IM or IS In the RAID creation menu Figure 3 47 select the hard disk drives you want to add for example to a RAID 0 IS You can add up to 10 hard disk drives to one volume Press C to create this array Figure 3 47 LSI1078 array creation Figure 3 47 shows a previously created volume in RAID 1 IM and one assigned hard disk drive as a hot spa...

Page 176: ...e of the created volumes Figure 3 49 shows a missing disk in the mirror which will be synchronized after a new disk is inserted Figure 3 49 LSI1078 Manage Array 3 4 2 Starting the ServeRAID MR10k controller WebBIOS After correct installation of the ServeRAID card the card is initialized in POST as shown in Figure 3 50 on page 159 Slot RAID Hot Drive Pred Size Num Disk Spr Status Fail MB 2 IBM ESXS...

Page 177: ...nstallation of this controller 3 4 3 Working with LSI MegaRAID controller WebBIOS The WebBIOS starts up with the initial window which indicates all available LSI SAS controllers that are installed in either the x3850 M2 x3950 M2 or x3950 M2 multinode complex and initialized in the POST correctly Figure 3 51 shows both installed ServeRAID MR10k controllers within a two node x3950 M2 system Figure 3...

Page 178: ... the adapter and then click Start or press Enter to begin the process The WebBIOS main window opens WebBIOS main window The WebBIOS has been started As shown in Figure 3 52 the window is separated into three areas the Controller Function menu Physical Drives panel and Virtual Drives panel Figure 3 52 WebBIOS main window The window also includes icons as shown in Figure 3 53 on page 161 Note If a m...

Page 179: ...d of onboard controller alarm Click to exit the application Click to move to previous window you were viewing Click to return to the home window Show the information for the selected adapter Rescan your physical devices Check your virtual disks logical drives settings and properties RAID subset and hard drive properties Delete add or create new array configurations according the setup you want Swa...

Page 180: ...al drives Figure 3 55 Adapter properties page 1 physical specifications Click Next to view the second page of the Adapter Properties window shown in Figure 3 56 on page 163 On the second page click Next again to view the third page shown in Figure 3 57 on page 163 You may change any of the default settings of the ServeRAID adapter properties Note If a background initialization is in progress you m...

Page 181: ... properties page 3 settings In the second and third pages change any of the following default settings Battery Backup This indicates whether you installed a battery backup unit BBU on the selected controller If a BBU is present click Present to view details about it See details in Figure 3 58 on page 164 ...

Page 182: ... drive The higher the value is the more system resources are devoted to a rebuild The value can be 1 100 BGI Rate 48 is the default This affects the amount of system resources dedicated to background initialization BGI of virtual disks connected to the selected adapter The value can be 1 100 Tip We do not recommend setting the lowest nor the highest value A very high percentage can effect performa...

Page 183: ... The Coercion Mode can be set to None 128 MB way and 1 GB way This number depends on how much the drives from various vendors vary in their actual size PDF Interval 300 seconds is the default this equals 5 minutes This specifies how frequently the controller polls for physical drives report a Predictive Drive Failure PDF Self Monitoring Analysis and Reporting Technology S M A R T error Alarm Contr...

Page 184: ...ain on the system s power supply that would occur if all disks spin up at the same time StopOnError Enabled is the default When the controller encounters an error during boot up it stops if this setting is enabled Stop CC On Error No is the default An error found in checking the consistency causes further checking to stop if you enable this setting Maintain PD Fail History Enabled is the default I...

Page 185: ... be continuous hourly daily weekly monthly or disable CC Start Time No default This is the time of day the consistency check should start CC Start mm dd yyyy No default This is the date the first consistency check should start Important You should not disable the consistency check to prevent any impact of data lost in case the amount of disk errors may reach a rate of inconsistency Consistency che...

Page 186: ...isks only Virtual disks also known as logical drives are arrays or spanned arrays that are available to the operating system The storage space in a virtual disk is spread across all the physical drives in the array A virtual drive can have the following states Optimal The virtual disk operating condition is good All configured physical drives are online Degraded The virtual disk operating conditio...

Page 187: ...s policies and operations are displayed as shown in Figure 3 61 Figure 3 61 Virtual disk Properties Policies and Operations panels Operations panel Many of the same operations listed in Figure 3 60 on page 168 can be performed from the Operations panel shown in Figure 3 62 Figure 3 62 WebBIOS virtual disk Operations panel Note You can watch the status of any ongoing background process by clicking ...

Page 188: ...round Slow Initialize Slow Init This also initializes the selected virtual disk but it is not complete until the entire virtual disk has been initialized with zeroes Check Consistency CC If the controller finds a difference between the data and the parity value on the redundant array it assumes that the data is accurate and automatically corrects the parity value Important Be aware that any kind o...

Page 189: ...ty which allows the controller to read sequentially ahead of requested data and to store the additional data in cache memory anticipating that the data will be needed soon This speeds up reads for sequential data but there is little improvement when accessing random data Adaptive This enables the controller to begin using read ahead if the two most recent disk accesses occurred in sequential secto...

Page 190: ...t disk cache policy I O Direct is the default The I O policy applies to reads on a specific virtual disk It does not affect the read ahead cache The settings are Direct In direct I O mode reads are not buffered in cache memory Data is transferred to the cache and the host concurrently If the same data block is read again it comes from cache memory This is the default Cached In cached I O mode all ...

Page 191: ...umber of disk drives in your system increases you can use RAID level migration to change a virtual disk from one RAID level to another You do not have to power down or reboot the system When you migrate a virtual disk you can keep the same number of drives or you can add drives You can use the WebBIOS CU to migrate the RAID level of an existing virtual disk Important It is strongly recommended tha...

Page 192: ...ives from the Controller Function menu see Figure 3 54 on page 161 On the panel that is displayed shown in Figure 3 65 on page 175 you can rebuild a drive either by selecting Rebuild or by selecting Properties and then working with the extended properties to do the rebuilding 1 At the time of writing the software User s Guide does not list this migration option However we verified that the option ...

Page 193: ...ct the appropriate physical disk 2 Select Properties 3 Click Go The extended properties about the selected disk are displayed as shown in Figure 3 65 Figure 3 66 WebBIOS physical properties The properties are Revision information about the firmware Enclosure ID slot number device type connected port SAS World Wide Name WWN ...

Page 194: ...7 Physical drives unconfigured good drive state Select one of the following settings Make Global HSP Set the selected disk to a global hotspare which is assigned to all defined virtual disks A reconstruction process starts immediately if any degraded virtual drive has been found Make Dedicated HSP Set the selected disk to a dedicated hotspare for a specific selected virtual disk only A reconstruct...

Page 195: ...sk is assigned as global hotspare or as dedicated hotspare for this virtual disk Locate This flashes the LED on the selected disk Offline A drive you forced from Online to Offline can result in loss of data redundancy or both This depends of the RAID level you have configured Figure 3 69 on page 178 shows the choices Important If you set a drive offline this can result in loss of data or redundanc...

Page 196: ... physical disk that was configured as online or hotspare but failed but the controller firmware detected an unrecoverable error and marked this disk as failed Missing A missing drive is a physical drive that is removed or marked missing from an enclosure and is a member of a configured virtual disk A physical disk previously forced to offline can be marked as missing also See Offline on page 177 T...

Page 197: ...n Wizard advises you to clear create new or add a virtual disk to any type of array that is supported at the type of installed adapter you selected in the Adapter Selection menu in Figure 3 51 on page 159 To start the Configuration Wizard select it from the Function Controller panel shown in Figure 3 54 on page 161 The Configuration Wizard window opens as shown in Figure 3 71 on page 180 Tip A dis...

Page 198: ...Disk definitions Define virtual disks using those arrays Within the virtual disk definition window you create and adapt the attributes of any virtual disk 3 Configuration Preview Preview a configuration before it is saved This shows you your configured virtual disk before you save it The three configuration types shown in the figure are discussed in the following sections Clear Configuration on pa...

Page 199: ...Configuration Wizard starting a new configuration Important If you clear the configuration all data will be lost The following reminder appears in the window This is a Destructive Operation Original configuration and data will be lost Select YES if desired so YES NO Important If you use this operation all existing data will be lost Be sure that you do not require any existing virtual disks and the...

Page 200: ...hen you are prompted to initialize the new virtual disks WebBIOS CU begins a background initialization of the virtual disks Table 3 17 Algorithm to select the RAID level for automatic configuration Custom Configuration When you want to set up different or more virtual disks use the custom setup Note Use the Back and Next buttons to display the previous or next window in the wizard You may also can...

Page 201: ...groups as follow a Select the disk in the panel view b Click AddtoArray to assign to a disk group Repeat this step to add more disks c Confirm this disk group by clicking on the Accept DG button shown in Figure 3 74 on page 184 The WebBIOS increases the disk groups and allows you to assign more disks Repeat this step if you want another disk group d If you want to undo a drive addition click the R...

Page 202: ... disk Figure 3 75 WebBIOS Configuration Wizard custom Span definition Tips All disks that you define within a disk group will be assigned to the virtual drives that you create later At the time of writing the Reclaim button does not have any affect to undo a disk addition Instead go back one window to the configuration selection menu and start the process of creating a disk group again Note The We...

Page 203: ...n Wizard custom with RAID properties You can change the following settings RAID Level Select the RAID level to use with the disks you selected Stripe Size Select a stripe size from 8 KB to 1024 KB blocks The recommended size is 128 KB Access Policy Select a policy for the virtual drive Values are read write read only or not blocked Read Policy Specify the read policy for this virtual drive Normal ...

Page 204: ...ived all the data in a transaction This is the default Bad BBU Select this mode if you want the controller to use Writeback mode but the controller has no BBU or the BBU is bad If you do not choose this option the controller firmware automatically switches to Writethrough mode if it detects a bad or missing BBU IO Policy The IO Policy applies to reads on a specific virtual disk It does not affect ...

Page 205: ...dd Configuration This option retains the old configuration and then adds new drives to the configuration You can add new virtual disks by using Auto Configuration after inserting new disks or by using Custom Configuration if you have unallocated space a hole in the assigned disk groups Events option The WebBIOS Event Information panel keeps any actions and errors that are reported by the selected ...

Page 206: ...troller The onboard SAS controller and the optional ServeRAID MR10k are discussed in 3 3 Internal drive options and RAID controllers on page 124 The Remote Supervisor Adapter II and the new Broadcom NIC interface are linked to the Intel ICH7 Southbridge so that it is no longer shared with the onboard SAS and RAID controller because it is at X3 chipset based systems 3 5 1 PCI and I O devices The fo...

Page 207: ...t you want to boot from as shown in Figure 3 78 The default is Planar SAS Figure 3 78 BIOS PCI Device Boot Priority 3 5 3 PCI adapter installation order We recommend that you install the adapters in a specific add in order to balance bandwidth between the two PCIe controllers The recommended order for PCIe adapter installation is 1 Slot 1 2 Slot 5 3 Slot 2 4 Slot 6 5 Slot 3 6 Slot 7 7 Slot 4 Note ...

Page 208: ...ccur in POST after you power on the server This means that not all PCI Express adapters are initialized during POST because the PCI ROM space limit was reached or any overlapping of PCI resources remained This behavior is seen typically if you add in for example Ethernet adapters in slots with a higher slot scan priority before the Fibre channel HBA or ServeRAID adapters Disabling of specific feat...

Page 209: ...ration PCI Express Preferred Max Payload Menu This sets the Preferred Max Payload Size for each PCI Express slot At POST BIOS sets the working Max Payload Size to the lower of the Preferred Max Payload Size and the Max Payload Size capability reported by the adapter The default Preferred Max Payload Size is 512 bytes For multinode configurations this setting must be changed independently on each n...

Page 210: ...figuration space access through the Memory Mapped PCI Configuration Space MCFG table Hot plug Message Signaled Interrupt MSI and Power Management Event PME IBM provides Active PCI software for Windows Server 2008 from the following Web address http www ibm com support docview wss uid psg1MIGR 5074966 PCI device information You can review detailed information at all onboard devices and optional ins...

Page 211: ...ation that can be checked here is the initial ROM size and whether all resources can be assigned correctly If a system reports an initialization error by showing a message in POST or in the server error logs The initial ROM size of each installed and enabled PCI device must be determined Slot Device Information information ROM usage in POST and Run time Vendor based Scroll to the next previous PCI...

Page 212: ...options that are supported with IBM System x systems in table form on the IBM ServerProven Web page http www ibm com servers eserver serverproven compat us This list is updated regularly and might contain limitations for using a specific PCI Express option Choose your system and scroll to the required subcategory based on the type of adapter you are looking for Select the link at the relevant opti...

Page 213: ...d how to work with multinode systems This chapters discusses the following topics 4 1 Introduction and terminology on page 196 4 3 Understanding scalability on page 199 4 4 Prerequisites to create a multinode complex on page 201 4 5 Upgrading an x3850 M2 to an x3950 M2 on page 204 4 6 Cabling of multinode configurations on page 209 4 7 Configuring partitions on page 220 4 8 Working with partitions...

Page 214: ...de systems A scaled system of two nodes three nodes or four nodes Complex This is at least two systems that have the ScaleXpander chip installed as described in 4 5 Upgrading an x3850 M2 to an x3950 M2 on page 204 and are cabled with the scalability cables as described in 4 6 Cabling of multinode configurations on page 209 Complex Descriptor This is a data record that is generated by a BMC in an x...

Page 215: ...artition is the merging process of one or more nodes within a multinode complex It is seen in any type of a supported operating system as one system Partitioning can be performed at any time to add or remove a node in it to scale resources but does require a reboot to activate This is a more flexible alternative to recabling your systems in a complex ScaleXpander This is the scalability enablement...

Page 216: ...vers with four or eight processors and up to 512 GB RAM installed A three node complex comprised of three x3950 M2 servers with six or 12 processors and up to 768 GB RAM installed A four node complex comprised of four x3950 M2 servers with eight or 16 processors and up to 1 TB RAM installed Note When we refer to an x3950 M2 we mean both an x3950 M2 or an x3850 M2 that has the ScaleXpander Option K...

Page 217: ...system and then later form one partition out of all four nodes A recabling in these circumstances is not required The new partitioning technology is managed by the Baseboard Management Controller BMC firmware in the x3950 M2 systems and a ScaleXpander key is required to enable the scalability features This is different from the previous X3 generation of x3950 systems which manages the partitioning...

Page 218: ...on page 211 shows the scalability ports 4 3 2 Complex Descriptor contents The contents of the Complex Descriptor include A unique Complex Descriptor signature Number of chassis UUID and structure of each chassis Logical component ID to specific which one is the primary Partition ID a unique partition identifier Node power state Number of partitions Note The Complex Descriptor is unique it contains...

Page 219: ...hown in Figure 4 24 on page 224 4 4 Prerequisites to create a multinode complex This section describes what to consider before you can create a multinode complex Before you configure a scalability partition be aware of the following prerequisites A scalable system must have a scalability hardware enabler key also known as the ScaleXpander chip installed into the x3850 M2 and x3950 M2 This provides...

Page 220: ... three node complexes the necessary cables are included with the x3950 M2 or with the ScaleXpander Option Kit For a four node complex additional cables are required they are included in the IBM XpandOnDemand Scalability Kit 2 part number 44E4250 Table 4 1 lists part numbers for the components required when you set up a multinode configuration Table 4 1 Supported and required scalability option par...

Page 221: ...of at least one node to manage the complex This can be important if you require assistance by your local IBM technical support if the system experiences any hardware related error Remember the remote video and remote drive features are available from the RSA II interface of the primary system only after a merge is completed The BIOS date and time settings should be adjusted to your local time sett...

Page 222: ...grade your x3850 M2 system to a x3950 M2 system For each x3850 M2 that you want to upgrade add a scalability partition by ordering the IBM ScaleXpander option kit part number 44E4249 Review section 4 4 Prerequisites to create a multinode complex on page 201 4 5 1 Installing the ScaleXpander key chip Figure 4 5 on page 205 shows the ScaleXpander key chip which must be installed in a vertical positi...

Page 223: ...t bezel of x3850 M2 server See instructions in the System x3850 M2 and x3950 M2 User s Guide 4 Loosen the captive screws and rotate the media hood Figure 3 3 on page 95 to the fully opened position You now an open view to the processor board 5 Locate the ScaleXpander key connector at the front of the processor board See Figure 4 5 and Figure 4 7 on page 206 6 Check that the ScaleXpander key is ori...

Page 224: ...ast one system in a complex must have a valid LAN connection to the Remote Supervisor Adapter RSA II The RSA II has a 10 100 Mbps network interface To configure the LAN interface 1 Plug in the power cables and power on your system 2 When prompted press F1 to enter the System Setup in BIOS 3 Select Advanced Setup RSA II Settings RSA LAN interface Note The new bezel does not have a sticker where you...

Page 225: ...he RSA II The RSA II resets enabling the new settings 6 Use your workstation to ping and access the Remote Supervisor adapter s Web interface as described in 6 2 3 Web interface on page 321 Tip Another way to access the RSA II Web interface is by connecting your workstation directly to the RSA II using a crossover Ethernet cable and then connecting to the RSA II with the following address and subn...

Page 226: ...required build level for a two node system is shown in Table 4 2 Table 4 2 Two node x3950 M2 system code levels The minimum required code levels for three node and four node system are shown in Table 4 3 Table 4 3 Three node and four node x3950 M2 system code level Download the following firmware Remote Supervisor Adapter II firmware http www ibm com support docview wss uid psg1MIGR 5073124 Field ...

Page 227: ...urations This section describes how the x3950 M2 systems are cabled This section assumes you have already installed the systems into your rack using the instructions in Rack Installation Instructions for System x3850 M2 and x3950 M2 available from http www ibm com support docview wss uid psg1MIGR 5073030 Before you begin the multinode cabling install the Enterprise Cable Management Arm CMA that is...

Page 228: ... M2 Enterprise Cable Management Arm Figure 4 10 on page 211 shows the ports of the x3950 M2 where the scalability cables are installed Depending on the number of nodes you plan to connect together you use either two or three of these ports in each node The ports are named port 1 port 2 and port 3 starting from the left Each ports has an indicator LED that shows an active connection on this port ...

Page 229: ...ure 4 11 x3950 M2 Scalability cable connector Insert the cables on one server first follow the installation guidance in the next sections to route the scalability cables through the cable management arm Figure 4 12 on page 212 shows cables inserted Tip After you insert a cable into the scalability port ensure that the cable is fully inserted and securely in place Scalability port 1 2 and 3 Port 1 ...

Page 230: ...re listed in Table 4 4 Table 4 4 Scalability cables for two node configuration To cable a two node configuration 1 Label each end of each ScaleXpander cable according to where it will be connected to each server 2 Connect the ScaleXpander cables to node1 a Connect one end of a ScaleXpander cable to port 1 on node 1 then connect the opposite end of the cable to port 1 of node 2 b Connect one end of...

Page 231: ...ngth of the cable between the rear side of the server and the first hanger bracket and on the front side of the cable management arm a Route the cable connected on port 1 of node 1 through the hanger brackets on the cable management arms b Route the cable connected on port 2 of node 1 through the hanger brackets on the cable management arms Node 1 Node 2 Management Network Scalability cabling One ...

Page 232: ...is included with every x3950 M2 system or with the ScaleXpander Option Kit for upgraded x3850 M2 systems The scalability cables required for a three node configuration are listed in Table 4 5 Table 4 5 Scalability cables for three node configuration Route the cables through hanger brackets balancing the slack in the cables between the CMAs and the nodes and in the side of the rack Scalability cabl...

Page 233: ...site end of the cable to port 1 of node 2 b Connect one end of a ScaleXpander cable to port 2 on node 1 then connect the opposite end of the cable to port 1 of node 3 c Connect one end of a ScaleXpander cable to port 2 on node 2 then connect the opposite end of the cable to port 2 of node 3 3 Route the cables through the enterprise cable management arms as follows refer to Figure 4 14 on page 214 ...

Page 234: ...able connected on port 2 of node 2 through the hanger brackets on the cable management arms 4 Ensure each server can be pulled out fully at the front of the rack and the scalability cables are clearly routed 4 6 3 Four node configuration A four node configuration requires five 3 08 m 9 8 feet and one 3 26m 10 7 feet ScaleXpander cables The cable diagram is shown in Figure 4 16 on page 217 One 3 0 ...

Page 235: ...caleXpander cable port assignment The scalability cables required for a four node configuration are listed in Table 4 6 on page 218 Node 1 Node 2 Node 3 Node 4 Management Network Scalability cabling One RSA LAN connection required for configuration Recommended RSA cabling optional ...

Page 236: ...e server and the first hanger bracket and on the front side of the cable management arm a Route the cable connected on port 1 of node 1 through the hanger brackets on the cable management arms b Route the cable connected on port 2 of node 1 through the hanger brackets on the cable management arms c Route the cable connected on port 2 of node 2 through the hanger brackets on the cable management ar...

Page 237: ...e opposite end of the cable to port 3 of node 4 6 Repeat step 3 on page 218 7 Ensure each server can be pulled out fully at the front of the rack and the scalability cables are clear routed Figure 4 17 shows the completed cabling Figure 4 17 x3950 M2 four node scalability cabling The cabling of the scalability cables is now finished Read the next section to create a partition ...

Page 238: ...scalable partitions by first discussing the options in the scalability management menu 4 7 1 Understanding the Scalable Partitioning menu The Remote Supervisor Adapter II RSA II Web interface includes panels to configure scalable partitions as shown in Figure 4 18 Figure 4 18 RSA II Web interface with enabled scalability features ...

Page 239: ...on of one Multiple systems and partitions can be controlled at the same time If the systems are in a partition the partitions must be shut down first to make sure that these will merge Reset partition The Reset button reboots a system or partition that is currently powered on If the system is off the command is ignored To reset the partition select the SN or a PID check box then click Reset Notes ...

Page 240: ...ck Force Notes The stand alone mode allows the systems to boot separately for debugging or diagnostic purposes If the system is in this mode already the command is ignored Multinode mode A system in a partition that is started in stand alone mode can be reassigned to boot in the multinode mode again Select the SN serial number then click Undo Notes One or more systems that are members in a partiti...

Page 241: ... The Delete button removes systems in the complex from a partition To delete a partition first select the PID of the partition that you want to delete and then click Delete If the systems are powered on they need to be powered off first Reset Defaults Figure 4 22 RSA II Web interface Reset Defaults reset the partition information The Reset button clears all the partition information in the complex...

Page 242: ...overview of the following information as indicated in Figure 4 24 Scalability ports Scalability cable connection Complex Descriptors Firmware problems Complex and Partition overview Figure 4 24 RSA II Web interface complex health The section on the right side in Figure 4 18 on page 220 and also shown in Figure 4 25 on page 225 indicates System power state Partition state Scalability mode Partition...

Page 243: ... Cabling of multinode configurations on page 209 b Confirm the boxes representing each node are colored gray If any box representing a scalability port is red or has a red circle this indicates an error You must resolve the error before you can configure System power mode Started powered on and in POST or OS Stopped powered off An entire red row indicates an error with the server as due to firmwar...

Page 244: ... connectors are fixed in the scalability cable slots If a behavior is unexpected such as a red indicator on the scalability ports an unreadable partition descriptor or other error condition that is unexpected our tests showed that an AC power cycle of all nodes in the complex may help to return the complex to a healthy state Note The state Invalid also indicates a non partitioned system Note An In...

Page 245: ...artition and assign the box that is the primary system Then click Create The default settings are the same as in autopartitioning Partition merge timeout 6 minutes On merge failure attempt partial merge Yes Partial merge allowed Yes To review and change the settings for the partition click the Partition ID n link where n is the number of the partition as shown in Figure 4 26 on page 228 This can o...

Page 246: ... power off and delete partitions You can also swap between Standalone and Multinode mode and clear partition information in a complex Start a partition All nodes in the partition power on immediately after a partition is started To start a partition select the Partition ID PID at the top of the blue symbolic box at the primary node Click Start under Partition Control The following events occur as ...

Page 247: ...for the primary node The primary node searches for the secondary node shown in Example 4 2 Example 4 2 x3950 M2 secondary server merge process IBM BIOS c Copyright IBM Cooperation 2008 Symmetric Multiprocessing System Quad Core Intel Xeon MP 2 93GHz Searching for primary server Press BLUE Remind button to bypass partition merge and boot standalone The merging process can bypassed by pressing the b...

Page 248: ...2GB Partition merge successful 64 GB Memory Installed 512 MB Memory Consumed by Scalability Press ESC to reboot and bypass merge attempt on next boot Press F1 for Setup Press F2 for Preboot Diagnostics DSA Press F12 to select boot devices Secondary server display shown in Example 4 4 Example 4 4 x3950 M2 merging process successful Secondary server Merge complete see primary server display The whit...

Page 249: ...rtition Control The following events occur All nodes that are configured in this partition are powered off The System state changes back to Stopped Swap between stand alone and multinode mode Depending on the partition you can change the scalability modes To switch all nodes in the partition to stand alone mode select the Partition ID PID and then click Force under Partition Control The systems re...

Page 250: ...he typical behaviors you might observe when working with systems in a multinode complex are described in this section First however review the following checklist again if you experience difficulties creating partitions Did you upgrade each x3850 M2 system to a x3950 M2 system to be capable for scalability configuration as described in 4 5 Upgrading an x3850 M2 to an x3950 M2 on page 204 Does each...

Page 251: ...ystem behaves as a single server system as follows When it is not configured as a member in a multinode partition All changes of BIOS settings in BIOS affect this particular node only If it is part of a partition but is bypassed during the merging process when you press the blue REMIND button shown in Figure 4 28 on page 230 If its part of a partition but is forced to boot a stand alone node The s...

Page 252: ...ity partition to get a multinode system you can observe the following behaviors All nodes of a partition in a complex can be managed as one system All nodes of a partition in a complex can be managed as single systems If you work remotely you must be able to access each RSA II Start a multinode system If you start a multinode system after creating the required first partition after all nodes in a ...

Page 253: ...the front and the port link LEDs at the rear of the systems go off again if the scalability mode is changed after a partition is started and rebooted to start in stand alone mode If a merge is unsuccessful The node or nodes that could not be merged boot in stand alone mode All nodes that could be merged boot as multinode system if the partial merge flag is enabled If the scalability LED at the fro...

Page 254: ...ing system RSA II and the Baseboard Management Controller might contain logged events The x3950 M2 embedded error correction features try to recover a hardware error and boot up the partition The operating system attempts to reboot Power off a Multinode system Powering off a started multinode system affects all nodes in a partition To power off a running Multinode system use any of the following m...

Page 255: ... not found During the process of merging a successful merge is not completed The members in this partition do not find each other and cannot find a member to merge The systems cannot be bypassed for merging to boot in stand alone mode This cannot be done remotely if the merge process is not completed If the RSA II scalability management menu does not enable you to manage any partition restart or p...

Page 256: ...f any events are colored with red or yellow at this specific time frame see Figure 4 32 You can filter the events at Severity Source and Date Figure 4 32 RSA II Web interface event log filtering How to resolve the problem To resolve the problem try the following steps 1 Power off both systems and start the multinode again If the scalability management menu is unresponsive perform a power off at al...

Page 257: ... in Example 4 6 Example 4 6 x3950 M2 merging process unsuccessful Primary server Chassis Number Partition Merge Status Installed Memory 1 Primary 32GB 2 Failed CPU Mismatch Partition merge failed No secondary chasiss to merge successful 64 GB Memory Installed 512 MB Memory Consumed by Scalability How to identify the problem To identify the issue Check that all systems in this complex have two or f...

Page 258: ...ing System Dual Core Intel Xeion MP 2 4 GHz Primarynode has less than 4GB memory installed merge not supported 2GB Memory Installed 4 Processor Packages Installed BIOS Version 1 03 How to identify the problem Verify that the primary system in this complex is added with at least one pair of DIMMs according the DIMM installation rules This is important if a system is shipped with 2 GB DIMM How to re...

Page 259: ...th merging if prerequisites were met on page 237 if for example a CPU error occurred that is not linked to having the wrong type of CPU See Example 4 8 Example 4 8 RSA II event log 1 Err SERVPROC 04 16 08 22 02 03 Resetting system due to an unrecoverable error 2 Err SERVPROC 04 16 08 22 02 03 Machine check asserted SPINT North Bridge Solution and workaround This problem is reported to IBM developm...

Page 260: ...u The system power state is incorrectly shown as powered off How to identify the problem In the scalability management menu check the System power state of each system in the partition See Figure 4 34 Figure 4 34 RSA II Web interface two node Partition configuration As Figure 4 34 shows the first system in the list is Stopped however the second system in the list is Started The power state is show...

Page 261: ...he AC power source Wait 30 seconds and then replug the AC power We recommend that you regularly check for newer system firmware code and updated product publications such as the Problem Determination and Support Guide and new tips on the Support for IBM System Support Web pages at http www ibm com systems support x As shown in Figure 4 36 on page 244 select your product family x3850 M2 or x3950 M2...

Page 262: ...244 Planning Installing and Managing the IBM System x3950 M2 Figure 4 36 IBM System x technical support on the Web ...

Page 263: ...mware and BIOS updates BIOS settings and operating system support are also discussed as important preliminary tasks prior to beginning the installation of operating systems on the server This chapters discusses the following topics 5 1 Updating firmware and BIOS on page 246 5 2 Confirming BIOS settings on page 254 5 3 Supported operating systems on page 257 5 4 Installing the operating system on p...

Page 264: ... 1 1 Prerequisite checklist Make sure to perform the following tasks before updating the firmware and installing the operating systems You have installed servers and additional components according to instructions in Chapter 3 Hardware configuration on page 89 and Chapter 4 Multinode hardware configurations on page 195 Check the BIOS and firmware levels on the items listed before you start install...

Page 265: ...we followed and the server components that require firmware updates To prepare the x3950 M2 servers in our labs for operating system installation we used the following firmware update sequence 1 Remote Supervisor Adapter II firmware RSA II requires a firmware update specific to system x3850 M2 or x3950 M2 2 FPGA firmware 3 BMC firmware Note When you further update the systems in a multinode comple...

Page 266: ...ely on each node preferably before forming a multinode partition Figure 5 1 Remote Supervisor Adapter II firmware separately on each x3950 M2 node FPGA firmware This update is in the form of a bootable diskette or ISO image If you have the RSA II configured and connected to your network you can use the Remote Console feature to mount a remote diskette IMG file or ISO file and boot the x3850 M2 or ...

Page 267: ...I ROM Programming is complete Erasing the PCI Card SPI ROM Programming PCI Card SPI ROM with Verification Sector 4 Page 54 PCI Card SPI ROM Programming is complete MNFPGA EXE v2 60 Node 1 0x604E 0x604F Erasing the CPU Card SPI ROM Programming CPU Card SPI ROM with Verification Sector 7 Page 54 CPU Card SPI ROM Programming is complete Erasing the PCI Card SPI ROM Programming PCI Card SPI ROM with V...

Page 268: ...RAMDRIVE D Decompressing BMC firmware reading fullfw cmt wrote 16360 SRECs to fullfw mot Acquiring BMC attributes Updating BMC Firmware Do you want to clear the SEL y or n y SEL Cleared Flash Loader v1 30 0 54 OSA Technologies Inc c 2007 firmware image IPMI Version 2 0 2 0 major Revision 2 2 minor Revision 32 32 manufacturer ID 2 2 product ID 77 77 build Name A3BT31B A3BT31B Firmware and the image...

Page 269: ...tion from the primary node The BIOS update process involves a sequence of erasing the current flash image updating the new flash image requesting confirmation of the serial number SN and model type for each node in consecutive order from first node to last node Preboot Diagnostics DSA Preboot This update is in the form of a bootable diskette If you have the RSA II configured and connected to your ...

Page 270: ... drive cache write through SCSI device sda 1024000 512 byte hdwr sectors 524 MB sda Write Protect is off sda assuming drive cache write through sda sda1 sd 1 0 0 0 Attached scsi removable disk sda sd 1 0 0 0 Attached scsi generic sg1 type 0 Device flashing in progress Flashing node 1 Waiting for device to become available sda READ CAPACITY failed sda status 0 message 00 host 7 driver 00 sda sense ...

Page 271: ...sda 1024000 512 byte hdwr sectors 524 MB sda Write Protect is off sda assuming drive cache write through SCSI device sda 1024000 512 byte hdwr sectors 524 MB sda Write Protect is off sda assuming drive cache write through sda sda1 sd 2 0 0 0 Attached scsi removable disk sda sd 2 0 0 0 Attached scsi generic sg1 type 0 Device flashing in progress DSA key has been flashed successfully usb 10 6 USB di...

Page 272: ...correctly detected in the BIOS and that the settings are correct for those devices To check the settings 1 At System Boot up press F1 to enter the BIOS Configuration Setup Utility window shown in Figure 5 2 Figure 5 2 BIOS Configuration Setup Utility window 2 Select System Summary to open the System Summary window shown in Figure 5 3 Figure 5 3 System Summary window ...

Page 273: ...he System Summary window select Processor Summary CPUIDs Figure 5 4 and check that all processors on stand alone or merged nodes are detected as expected Figure 5 4 CPUIDs on a two node x3950 M2 4 From the System Summary window select Processor Summary Processor Speeds Figure 5 5 to ensure that all processors are matched and check L2 Cache Sizes Figure 5 6 on page 256 Figure 5 5 Processor Speeds o...

Page 274: ...mory Installed detailed in System Summary 6 Check that the memory cards installed in the server are detected as shown in the Memory Settings window Figure 5 8 which also shows memory array scrub on every boot and the scrub rate at run time To open the Memory Settings window select Advanced Setup in the Configuration Setup Utility window shown in Figure 5 2 on page 254 Figure 5 8 Detected memory ca...

Page 275: ...5 9 PCIe slot information and the installed adapters 8 Depending on the numbers and types of PCIe adapters in a multinode configuration you might have to configure the Reserved Memory Mapped I O Size shown in Figure 5 10 For details see 3 5 4 PCI Express device related information in the BIOS on page 190 Figure 5 10 Advanced PCI Settings 5 3 Supported operating systems Before you install your inte...

Page 276: ... In addition we also recommend that you check the particular operating system vendor s hardware compatibility list HCL Operating system vendors typically publish and update their HCL as new models of servers are introduced by the server hardware vendors The commonly referenced Web pages of the operating system vendors are VMware http www vmware com pdf vi35_systems_guide pdf http www vmware com pd...

Page 277: ...www ibm com servers eserver serverproven compat us nos microsoft html At the time of writing this book the Windows operating systems are supported on x3850 M2 and x3950 M2 as indicated in Table 5 2 on page 260 The shaded table cells indicate Yes they are supported Operating System x3850 M2 x3950 M2 1 node x3950 M2 2 node x3950 M2 3 node x3950 M2 4 node VMware ESX 3 5 Yes Yes with patch ESX350 200 ...

Page 278: ...ise x64 Edition with Hyper V Yes Yes Planned No No Microsoft Windows Server 2008 Standard x64 Edition Yes Yes Yes limited to 2 sockets in each node No No Microsoft Windows Server 2008 Standard x64 Edition with Hyper V Yes Yes Planned No No Microsoft Windows Server 2008 Web x64 Edition No No No No No Microsoft Windows Server 2008 Datacenter x86 Edition No No No No No Microsoft Windows Server 2008 E...

Page 279: ...mited Virtualization with High Availability Program Yes Yes with SP2 or later No No No Microsoft Windows Server 2003 2003 R2 Enterprise x64 Edition Yes Yes with SP2 or later Yes with SP2 or later No No Microsoft Windows Server 2003 2003 R2 Standard x64 Edition Yes Yes with SP2 or later Yes with SP2 or later limited to 2 sockets in each node No No Microsoft Windows Server 2003 2003 R2 Web x64 Editi...

Page 280: ...ells indicate Yes they are supported Operating System x3850 M2 x3950 M2 1 node x3950 M2 2 node x3950 M2 3 node x3950 M2 4 node Red Hat Enterprise Linux 5 Red Hat Enterprise Linux 5 Server x64 Edition Yes Yes Yes with Update 1 or later No No Red Hat Enterprise Linux 5 Server with Xen x64 Edition Yes Yes Yes with Update 1 or later No No Red Hat Enterprise Linux 5 Server Edition Yes Yes No No No Red ...

Page 281: ...T Yes with SP1 or later Yes with SP1 or later Yes with SP1 or later No No SUSE Linux Enterprise Server 10 with Xen for AMD64 EM64T Yes with SP1 or later Yes with SP1 or later Yes with SP1 or later No No SUSE Linux Enterprise Server 10 with Xen for x86 Yes with SP1 or later Yes with SP1 or later No No No SUSE Linux Enterprise Server 10 for x86 Yes with SP1 or later Yes with SP1 or later No No No SU...

Page 282: ...ded VMware ESXi 3 5 boots from an IBM customized USB Flash Drive It does not have to be installed onto a local server storage IBM does provide ESXi 3 5 Installable Edition which can be installed onto local storage but cannot be installed onto remote storage such as a SAN LUN or iSCSI target The following topics are described in this section Prerequisites on page 264 Configure server BIOS for Embed...

Page 283: ...s F1 to boot the server in the BIOS 2 Go to Start Options USB Disk and set it to Enabled as shown in Figure 5 11 Figure 5 11 Example of x3850 M2 BIOS Start Options with USB Disk enabled 3 In Start Options Startup Sequence Options set the following startup sequences to IBM Embedded Hypervisor also shown in Figure 5 12 on page 266 Primary Startup Sequence Third Startup Device Wake on LAN Startup Seq...

Page 284: ...artup device after CD ROM and diskette drive 0 in system BIOS 4 Exit the BIOS utility and save you settings when prompted Boot ESXi hypervisor and customize settings After setting the IBM Embedded Hypervisor to boot as a startup device in the system BIOS and after POST completes the server starts to boot VMware ESXi ...

Page 285: ...nce of events that take place and how you interact with the process 1 After the server POST has completed the server boots from the IBM USB Flash Disk and loads ESXi hypervisor shown in Figure 5 13 Figure 5 13 ESXi starting to boot from IBM customized USB Flash Disk with IBM Customized ESXi image ...

Page 286: ...nfiguration of ESXi The default network setting and behavior for ESXi is to request a DHCP IP Address In the event that it cannot find a DHCP server to obtain a DHCP lease it defaults to the 169 254 0 1 255 255 0 0 IP address and Class B netmask Figure 5 14 The x3850 M2 is successfully booting the IBM Customized ESXi on the USB Flash Disk and detecting processor sockets and total memory installed ...

Page 287: ...he Customize System menu when you press F2 at the DCUI ESXi interface 4 Set the host name for example DNS FQDN and DNS servers IP Addresses by selecting Configure Management Network See Figure 5 16 on page 270 You might want to set the ESXi host management IP Address this interface should be separate to the interfaces you would set in VI Client for Virtual Machine traffic to static rather than DHC...

Page 288: ...270 Planning Installing and Managing the IBM System x3950 M2 Figure 5 16 Configuring the management network IP address settings ...

Page 289: ...hen set the IP Address to match your management network policies static IP or DHCP See Figure 5 18 on page 272 Make sure to register the host name in your DNS server or servers and set DNS server IP addresses and DNS suffixes for the ESXi host especially if you plan to use management tools like VMware VirtualCenter which relies heavily on DNS name resolution IP Configuration Figure 5 18 on page 27...

Page 290: ... interface settings are correctly configured to ensure that you can connect to the ESXi host for further configuration using VMware VI Client or VirtualCenter By default this test tries to ping your Default Gateway DNS server IP addresses and performs DNS resolution of your ESXi host name See Figure 5 19 Figure 5 19 Testing of the management network settings ...

Page 291: ...allation 273 8 Most of the other settings typically configured for ESX 3 5 can be also configured similarly by using VI Client as shown in Figure 5 20 Figure 5 20 Summary tab after connecting to ESXi using VI Client ...

Page 292: ...ee Figure 5 21 Figure 5 21 The VI Client Configuration tab for ESXi host displaying Health Status basic alerts for the ESXi s server major subsystems 10 In the Hardware panel on the left of the Configurator tab in VI Client check that ESXi can successfully detect each processor s sockets and cores total memory installed network adapters installed and storage adapters installed Figure 5 22 on page ...

Page 293: ...ves connected through the x3850 M2 s external SAS connector to EXP3000 enclosures To confirm this Check Storage sensors in the Hardware panel of the Configurator tab Hardware pane under Health Status Check the ESXi Web Interface accessible by using the address format https IP Address of ESXi host management interface Click Browse datastores in this host s inventory link as shown in Figure 5 24 on ...

Page 294: ...e of the following actions If you intend to use VirtualCenter to manage all your ESXi hosts under Licensed Features License Source Edit configure the ESXi host s license source to point to the IP address of your designated License Server this is typically configured before or during the installation of VMware VirtualCenter Server If you use host based license files configure the ESXi host s licens...

Page 295: ... hosts and connect to your VirtualCenter server using VI Client by using your VirtualCenter IP address add the appropriate ESXi hosts using DNS registered host names and ESXi host root password to VirtualCenter and make all configuration changes to ESXi hosts HA DRS Clusters through VirtualCenter 14 If you intend to use VMware VirtualCenter 2 5 Update 1 to manage ESXi hosts in a High Availability ...

Page 296: ...hLocation to a valid directory with sufficient space for example 1 GB to hold the userworld swap file The userworld swap can be configured on local storage or shared storage For example the VMFS volume vmfs volumes DatastoreName after reboot ESXi might not show DatastoreName but instead a unique volume ID e Check the ScratchConfig ConfiguredSwapState check box f Click OK g Reboot the ESXi host ...

Page 297: ...o install ESXi Installable onto a hard drive SAS or SATA This procedure presumes you are using a keyboard and monitor attached to the server to perform the installation To install VMware ESXi 1 Configure any RAID arrays using the MegaRAID Manager tool 2 Insert the ESX Server 3i Installable CD into the CD drive and boot the system 3 When prompted during the boot process press F12 to boot from the C...

Page 298: ...visor Boot Menu select the ThinESX Installer and press Enter to boot the installer 5 The ESX Server runs through its boot process until the Welcome window opens as shown in Figure 5 27 Figure 5 27 Welcome window for ESXi Installer 6 Press Enter to continue with the Installation 7 If you accept the End User License Agreement press F11 ...

Page 299: ...r ESX Server host configuration as described in Boot ESXi hypervisor and customize settings on page 266 For detailed information about installing ESX Server 3i see the ESX Server 3i Embedded Setup Guide http www vmware com pdf vi3_35 esx_3i_e r35 vi3_35_25_3i_setup pdf 5 4 3 Installing VMware ESX 3 5 Update 1 VMware ESX 3 5 is can be installed on local server storage or remote storage like a SAN L...

Page 300: ... and configuration guides to establish the appropriate settings for booting from the SAN iSCSI remote storage Check for storage system compatibility in Storage SAN Compatibility Guide For ESX Server 3 5 and ESX Server 3i including support for boot from SAN http www vmware com pdf vi35_san_guide pdf Download the VMware ESX 3 5 installation and configuration guides http www vmware com pdf vi3_35 esx...

Page 301: ...ocal or remote disk bootable SAN LUN on which to install ESX 3 5 Update 1 If multiple disks are available be careful to select the correct disk to install ESX 3 5 Update 1 otherwise the server might not boot correctly into ESX 3 5 Update 1 hypervisor Figure 5 30 ESX 3 5 Update 1 Partitioning Options 5 Click Next to continue with the installation ...

Page 302: ...u selected earlier on which to install VMware ESX 3 5 Update 1 7 Network Configuration enables you to select the available Network Adapters to use for the default vSwitch and vSwif0 interface for shared Virtual Machine and Service Console access You may modify the physical Network Adapter that is configured for the vSwitch and the vSwif interface 8 Select the time zone as appropriate to your locat...

Page 303: ...X 3 5 startup window with the Service Console IP Address as shown in Figure 5 32 Figure 5 32 ESX 3 5 Update 1 Hypervisor startup window 13 Press Alt F1 to get the service console logon prompt Figure 5 33 to return to the startup window press Alt F11 Figure 5 33 Logging on to ESX 3 5 Update 1 service console 14 To logon to the ESX 3 5 Service Console type root and the password you set earlier 5 4 4...

Page 304: ... 5709 drivers from http www ibm com support docview wss uid psg1MIGR 5070012 Download the Intel based Gigabit Ethernet drivers from http www ibm com support docview wss uid psg1MIGR 5070807 In BIOS ensure the following parameters are set In CPU Options ensure that the Clustering Technology parameter is set to Logical Mode as shown in Figure 3 9 on page 100 In Advanced Settings RSA II Settings ensu...

Page 305: ...skette d Click Mount drive Install Windows Server 2003 To install Windows Server 2003 1 Boot the server from Windows Server 2003 Installation CD DVD or ISO file 2 Press F6 to load additional storage device drivers when you see the following text Setup is inspecting your computer s hardware configuration The Windows Setup window opens see Figure 5 34 Figure 5 34 Specify additional storage device dr...

Page 306: ...txtsetup oem Figure 5 35 Windows Setup detects the LSI MegaRAID SAS RAID Controller Driver 5 Proceed with the normal installation steps to complete the installation of Windows Server 2003 6 After he installation finishes and the server boots into the partition on which Windows Server 2003 has successfully installed log on as Administrator Proceed to install additional drivers for any other adapter...

Page 307: ...he last line in Example 5 4 Example 5 4 boot ini for accessing more than 4 GB memory boot loader timeout 3 default multi 0 disk 0 rdisk 1 partition 1 WINDOWS operating systems multi 0 disk 0 rdisk 1 partition 1 WINDOWS Windows Server 2003 Enterprise fastdetect PAE 5 4 5 Installing Windows Server 2008 This section describes the key aspects of installing Windows Server 2008 on the x3950 M2 Prerequis...

Page 308: ... ensure the following parameters are set In CPU Options ensure that the Clustering Technology parameter is set to Logical Mode as shown in Figure 3 9 on page 100 In Advanced Settings RSA II Settings ensure that the OS USB Selection setting is set to Other OS as shown in Figure 6 14 on page 320 A USB diskette drive is not required because Windows Server 2008 already includes boot drives for the IBM...

Page 309: ...Chapter 5 Installation 291 Figure 5 36 Select type of installation Upgrade or Custom clean installation ...

Page 310: ...t the appropriate storage host bus adapter driver from diskette drive A Figure 5 37 Selecting the disk on which to install Windows Server 2008 7 Continue with the installation process until completed 8 After Windows Server 2008 boots from the disk partition you selected during the installation process log on as Administrator and follow the instructions listed in the next section to install additio...

Page 311: ...for accessing more than 4 GB memory boot loader timeout 3 default multi 0 disk 0 rdisk 1 partition 1 WINDOWS operating systems multi 0 disk 0 rdisk 1 partition 1 WINDOWS Windows Server 2008 fastdetect PAE 5 4 6 Installing Red Hat Enterprise Linux 5 Update 1 This section describes the key aspects of installing RHEL 5 Update 1 Prerequisites Before you install and configure Red Hat Enterprise Linux 5...

Page 312: ...agement workstation or a mount point if you are managing the RSA from a linux management workstation or choose Select File and browse to the CD DVD ISO file second and click the button followed by the Mount Drive button Install Red Hat Enterprise Linux 5 Update 1 To install Red Hat Enterprise Linux 5 Update 1 1 Boot the server from the Red Hat Enterprise Linux 5 Update 1 DVD CD or ISO file 2 When ...

Page 313: ... Trusted Computing Group s Software Stack TSS specification You can use TrouSerS to write applications that make use of TPM hardware The tpm tools is a suite of tools used to manage and utilize TPM hardware For more information about TrouSerS refer to http trousers sourceforge net 5 4 7 Installing SUSE Linux Enterprise Server 10 SP1 This section describes the key aspects of installing SLES 0 SP1 P...

Page 314: ...e Pack 1 CD DVD for example D in your windows management workstation or a mount point if you manage the RSA from a linux management workstation or choose Select File and browse to the CD DVD ISO file second and click the button then click Mount Drive button Install SUSE Enterprise Linux 10 SP1 To install SUSE Enterprise Linux 10 SP1 1 Boot the server from the SUSE Enterprise LInux 10 SP1 Installat...

Page 315: ...tion Mode window 4 Continue with the installation process Refer to the installation instructions for additional information Post Installation If you are using a 32 bit version of SUSE Enterprise Linux 10 then install the kernel PAE kernel to access more than 4 GB of memory ...

Page 316: ...298 Planning Installing and Managing the IBM System x3950 M2 ...

Page 317: ...edded features to help you reduce energy costs This chapters discusses the following topics 6 1 BMC configuration options on page 300 6 2 Remote Supervisor Adapter II on page 316 6 3 Use of IBM Director with VMware ESX on page 334 6 4 Active Energy Manager on page 334 6 5 IBM Director Implementation of servers on page 346 6 6 System management with VMware ESXi 3 5 on page 355 6 7 Power management ...

Page 318: ... a whole set of standardized and system specific sensors The components communicate with the BMC through an embedded Inter Integrated Circuit I C bus interface Logged events that are defined as errors are reported as follows Into the RSA II event data store Through the BMC driver interface into the operation system and management applications Through error indicator LEDs on different components in...

Page 319: ...settings for battery and system components System LED Control power HDD activity error and others Fan speed control Power and reset control Interface to subassemblies to provide the vital product data VPD which contains information about components and the system such as Field replacement unit FRU information System firmware code levels like BIOS or BMC Non maskable interrupt NMI system management...

Page 320: ...llows access also to an internal 1 Gbps Broadcom interface 6 1 2 BMC LAN configuration in BIOS By default the BMC is configured with the following defaults IP address 10 1 1 97 Subnet 255 0 0 0 You can change the IP settings in the server BIOS as follows 1 Boot or reset the server and press F1 when prompted to enter Setup 2 Select the Advanced Setup Baseboard Management Controller BMC Settings The...

Page 321: ...k Configuration to open the BMC Network Configuration menu shown in Figure 6 2 Figure 6 2 BIOS BMC Network Configuration settings menu 4 Enter a host name 5 Enter the appropriate IP address subnet mask and gateway address or enable DHCP control 6 Select Save Network Settings in BMC and press Enter ...

Page 322: ...nu shown in Figure 6 1 on page 303 or by using tools such as OSA SMBridge SvcCon Dynamic System Analysis DSA or IBM Director An example event log is shown in Figure 6 3 You can select Get Next Entry and Get Previous Entry to page through the events Figure 6 3 BMC System Event Log entry in BIOS The total number of events recorded is listed in the Entry Number field Tip You might notice that several...

Page 323: ... Account Settings menus You can add change or set user privileges in the BMC User access is through account UserID 2 as shown in Figure 6 4 UserID 1 is NULL and cannot be changed The default credentials for UserID 2 are Username USERID all uppercase Password PASSW0RD all uppercase where 0 is the number zero Figure 6 4 BIOS BMC User Account Settings menu and individual user settings Select the User...

Page 324: ...vior of the server and how to proceed if an error occurs Various system management routines are controlled by the BMC Figure 6 5 BIOS BMC configuration settings The settings as shown in Figure 6 5 are BMC POST Watchdog This is Disabled by default It monitors the system initialization in the POST phase If you enable this watchdog the system remains at reboot or shutdown after its defined time out A...

Page 325: ...that cannot be handled by the chipset Typically an MCK is issued by an essential component of the IBM eXA4 chipset particular components such as a DIMM processor or I O linked devices Ring Dump after SPINT This setting is Disabled by default By enabling the setting the BMC catches Ring Dump information from the chipset after an MCK occurred to log unexpected error conditions IBM Technical Support ...

Page 326: ...er any third party driver must be uninstalled first 2 IBM IPMI ASR service Download the BMC device drivers To download the drivers appropriate for your server 1 Go to the Support for IBM System x Web page http www ibm com systems support x 2 Under Popular links click Software and device drivers 3 Select your product for example System x3850 M2 7141 7144 4 Use the category OSA IPMI Device driver Ad...

Page 327: ...d through the usual initial windows 2 When prompted select Perform Update and click the Next button Figure 6 6 on page 310 Note The IBM System x Web page does not have OSA IPMI driver and layer software available for Linux Novell and Windows operating systems to download For instructions see the following sections Notes On multinode x3950 M2 configurations the Microsoft IPMI device driver is the o...

Page 328: ...n Enable Show hidden devices in the submenu View The ISMP device driver is listed in the section System devices as shown in Figure 6 7 Figure 6 7 Device Manager showing list of system devices OSA IPMI mapping layer library files in Windows Server 2003 To install the IPMI mapping layer library files 1 Ensure that the OSA IPMI device driver is installed before installing the library software 2 Downl...

Page 329: ..._server or IPMI Driver depending on the version you have installed and click the Remove button 3 The Add or Remove Programs utility prompts you to restart the system to complete the driver removal process Note If the Microsoft IPMI driver is installed you do not have to install the OSA IPMI mapping layer Note The OSA IPMI driver should not be removed using Device Manager Doing so only partially re...

Page 330: ...ion if you have applied Release 2 Service Pack 2 as follows 1 Click Start Settings Control Panel Add or Remove Programs Add Remove Windows Components The window shown in Figure 6 9 opens Figure 6 9 Windows Server 2003 adding or removing Windows components 2 Select Management and Monitoring Tools and click Details The window shown in Figure 6 10 on page 313 opens 3 Add a check mark to Hardware Mana...

Page 331: ... the OSA IBM driver is uninstalled as described in OSA IPMI driver uninstall in Windows Server 2003 on page 311 Figure 6 11 Microsoft hardware management notice of third party drivers 5 Click OK 6 Click Next 7 Click Finish When the installation is completed the Microsoft compliant IPMI device is listed as a hidden device under Device Manager System Devices as shown in Figure 6 12 on page 314 ...

Page 332: ...n in the window 4 Reboot the server if the installation procedure prompts you to do so Microsoft IPMI driver in Windows 2008 The Microsoft IPMI driver is installed in Windows 2008 by default Install the device drivers on Linux The support for OSA IPMI driver and layer software for RHEL4 and SLES9 are deprecated In May 2008 IBM transitioned to the Open Source IPMI OpenIPMI software instead See the ...

Page 333: ...9 Service Pack 2 Service Pack 3 SUSE Linux SLES 9 for AMD64 EM64T Service Pack 2 VMware ESX Server 3 0 2 any update VMware ESX 3 5 any update VMware ESX 3i installable Working with Open Source IPMITool Download the latest version of ipmitool from the following location http ipmitool sourceforge net To build ipmitool Note SUSE Linux Enterprise Server 10 and later and Red Hat Enterprise Linux 5 and ...

Page 334: ... communicate properly Table 6 2 TCP IP ports used by the BMC 6 2 Remote Supervisor Adapter II The Remote Supervisor Adapter II shown in Figure 6 13 on page 317 is a system management card that ships with the x3850 M2 and x3950 M2 server systems The adapter includes two components the main board has an embedded video chip ATI RN50 and the networking interface and the daughter card is the RSA II ada...

Page 335: ...ns The most useful functions and features of the RSA II include Web and Telnet interface The RSA II is managed through its built in Web interface From the interface you can manage the local server plus other nodes if the local server is part of a multinode complex Vital Product Data This provides an overview of the most essential system firmware codes for BIOS RSA II BMC without rebooting the serv...

Page 336: ...nstalled and enabled Remote media As a part of the remote control feature the remote media capability lets you use diskette drives diskette images optical drives such as DVD or CD ROM ISO files on your hard disk mounted in a virtual drive or anywhere in your network You only have to map this network drive or file system to the local workstation to which you have established the connection to the s...

Page 337: ...ation through the RSA II driver interface allowing any system management software or utility that you installed in the operation system to communicate with the RSA II Out of band communication through the RSA II network interface allowing various features such as alerting or Web interface access 6 2 2 RSA LAN configuration in BIOS The default network address for the RSA II is IP address 192 168 70...

Page 338: ...r Other OS for Windows or select Linux OS Use the right and left arrow keys to make the selection The purpose of this selection is to prevent a known problem with Linux and its generic human interface device HID driver Linux cannot establish USB communication with the RSA II using the generic HID which Windows uses By selecting Linux OS here it makes the RSA II appear as an OEM HID instead of gene...

Page 339: ...k for a DHCP server to obtain an IP address and if none is available to use the default IP address of 192 168 70 125 You can view the assigned address from the BIOS in the Advanced Settings menu To access the RSA II Web interface 1 Open a supported browser Enable the use of a Java Plugin if you want to use remote console features as described in 6 2 4 Remote console and media on page 324 Use the d...

Page 340: ...ter you log on the welcome window opens 2 Set the session time out values as shown in Figure 6 16 on page 323 Note In the Web interface you will have to change the login credentials Safeguard your credentials Losing them requires rebooting the server BIOS and restoring the RSA settings to the factory defaults ...

Page 341: ...Chapter 6 Management 323 Figure 6 16 RSA II Web interface Welcome window The home window of the Web interface opens See Figure 6 17 on page 324 ...

Page 342: ...Managing the IBM System x3950 M2 Figure 6 17 RSA II Web interface 6 2 4 Remote console and media To manage servers from a remote location you often use more than just keyboard video mouse KVM redirection For example for the remote ...

Page 343: ...I Settings menu shown in Figure 6 14 on page 320 Remote media works with the following operating systems Windows Server 2003 Windows Server 2008 64bit Red Hat Enterprise Linux AS 4 but not for OS installation Red Hat Enterprise Linux AS 5 but not for OS installation SUSE LINUX Enterprise Server 9 but not for OS installation SUSE LINUX Enterprise Server 10 but not for OS installation VMware ESX 3i ...

Page 344: ...troke sequence is sent to the server If you require additional buttons click Preferences and then modify or create new key buttons To detach the button bar click anywhere in the grey background and then dragging and dropping the bar a process that creates a separate window Figure 6 19 RSA II Web interface softkeys and preferences The preferences link allows you to specify up to twelve key stroke s...

Page 345: ... selector You may reduce or even stop video data to allow more bandwidth for Remote Disk Move the slider left or right until you find the bandwidth that achieves the best results The changes do not require a restart of RSA II or the server 6 2 5 Updating firmware Flash the RSA II card if it is not at the latest level of firmware to ensure that all known fixes are implemented As described in the 4 ...

Page 346: ...is method 2 Unpack the ZIP file and you will find the two PKT firmware files as shown in Figure 6 22 Figure 6 22 RSA II firmware files 3 Connect to the RSA II by using a Web browser by simply entering the IP address in the address bar Log in using your credentials the default user and password are USERID and PASSW0RD 4 Select Tasks Firmware Update 5 Click Browse to select the first of two files fo...

Page 347: ...Line adapter Download the RSA II device driver Download one of the following drivers RSA II service for 32 bit Windows Server 2003 and Windows Server 2008 http www ibm com support docview wss uid psg1MIGR 5071025 RSA II service for x64 Windows Server 2003 and Windows Server 2008 http www ibm com support docview wss uid psg1MIGR 5071027 RSA II daemon for Linux RHEL 3 and 4 SLES 9 and 10 ESX 3 http ...

Page 348: ... RSA II service in Windows operating system Linux daemon installation To install the RSA II device driver daemon on Linux ensure that you downloaded the correct RPM for your Linux distribution Available packages at the time of writing this book are Red Hat Enterprise Linux 3 Red Hat Enterprise Linux 4 Red Hat Enterprise Linux 5 SUSE Linux Enterprise Server 9 SUSE Linux Enterprise Server 10 VMware ...

Page 349: ...package ibmusbasm 1 xx 2 src rpm for VMware ESX 3 x and 32 bit Linux except RHEL5 ibmusbasm64 1 xx 2 src rpm for 64 bit Linux except RHEL5 ibmusbasm rhel5 1 xx 2 src rpm for 32 bit RHEL5 ibmusbasm64 rhel5 1 xx 2 src rpm for 64 bit RHEL5 2 Install the daemon for example SUSE where xx is the version by running rpm ivh ibmusbasm 1 xx i386 rpm 3 Check that the daemon is running by using the ps command...

Page 350: ...age ibm_svc_rsa2_hlp2xxa_linux_32 64 tgz 2 Expand the tgz file by using the following command tar xzvf ibm_svc_rsa2_hlp2xxa_linux_32 64 tgz Expanding the file creates an SRPMS directory 3 Change directory to SRPMS cd SRPMS 4 Install the 32 bit source RPM Package by issuing an rpm command changing directories and issuing a second rpm command xx is the version number of the RSA II package rpmbuild r...

Page 351: ...n the NVRAM of the RSA II and kept there until deleted or the battery is exhausted To access the event log data select the task Port name Port number Description http 80 default Web server HTTP connection TCP https 443 default SSL connection TCP telnet 23 default Telnet command line interface connection TCP SSH 22 default Secure Shell SSH command line interface TCP SNMP Agent 161 default SNMP get ...

Page 352: ...nstalled The system management driver is required to manage Level 2 systems by the IBM Director agent For more information about implementing and managing objects of levels 0 1 and 2 see section 6 5 IBM Director Implementation of servers on page 346 The IBM Systems Software Information center has additional support and guidance for using the IBM Director http publib boulder ibm com infocenter eser...

Page 353: ... and10 Windows Server 2003 Enterprise Edition Supported managed hardware Rack mounted server BladeCenter Director managed objects iPDUs 6 4 1 Active Energy Manager terminology The following terms apply to the hardware in an Active Energy Manager environment management server A server on which both IBM Director Server and Active Energy Manager Server are installed management console A system on whi...

Page 354: ...e Energy Manager Database This section also discusses AEM s monitoring capabilities Active Energy Manager Server Active Energy Manager Server maintains the AEM environment and manages all AEM operations AEM Server communicates out of band with each managed object to collect power information In the case of a BladeCenter chassis it communicates with the management module in the chassis In the case ...

Page 355: ...equest that customizations be deleted during the uninstallation of IBM Director Server The size of the AEM Database directly correlates to the short term data collection interval the long term data collection interval and the number of days that short term and long term trend data is kept All of these values are configurable You can control them and the size of the AEM Database by using the Manage...

Page 356: ...a export data and manage energy on certain hardware for example by using capping features A new Active Energy Manager task icon is added in the IBM Director Management console after installation of the AEM extension see Figure 6 25 Figure 6 25 Active Energy Manager task icon You use the Active Energy Manager task on the IBM Director Console menu to launch the Active Energy Manager GUI Use the GUI ...

Page 357: ...s of the server with the server itself preselected If the target is a group of rack mounted servers or BladeCenter chassis or both the display consists of all chassis or rack mounted servers in the group with the first chassis or Start Active Energy Manager for all managed systems If the target is an iPDU the current date for the PDU and the capacity and usage for the load groups is displayed with...

Page 358: ...ged systems that you are interested in The disadvantage is that you can manage only those managed systems that were selected when you started Active Energy Manager If you have to manage other managed systems at a later time you must start another Active Energy Manager task for those additional managed systems 6 4 4 Active Energy Manager 3 1 functions This section provides an overview of the functi...

Page 359: ...f data trends and also export the tabular data as a spreadsheet XML or HTML formats Power trending The Energy Scale architecture provides continuous power usage data collection The power usage data can be displayed from Active Energy Manager Data administrators can use the information to project the power consumption of the data center at various times of the day week or month and use that informa...

Page 360: ...ol The types of events monitored by the Active Energy Manager include power component failures offline and online actions power management actions and power and thermal critical warnings Thermal trending The thermal sensors are primarily based on the digital thermal sensors available on the systems All of the logical sensors are the result of firmware running on the thermal and power management de...

Page 361: ...e current data was collected Name assigned to the iPDU Firmware level of the iPDU Average input and output watts for the iPDU over the last polling interval Minimum and maximum output watts for the iPDU over the last polling interval Current minimum maximum ambient temperature of the iPDU at the last polling interval You can also display detailed information about each load group in the iPDU Name ...

Page 362: ...en system or group of systems over a specified period of time and calculates the corresponding cost of that energy shown in Figure 6 30 on page 345 It gives a visual indication of the number of watt hours consumed by the target object or objects over the specified time period and provides a comparison to what would have been consumed had nameplate power been drawn over that entire period ...

Page 363: ... percentage This percentage is predetermined to be within a safe operating limit and is not user configurable Under current implementation this is a 14 frequency drop In the Active Energy Manager user interface the Power Saver mode is set to enable or disable One possible use for Power Saver would be to enable it when workloads are minimal such as at night and then disable it in the morning When P...

Page 364: ...Generally the amount is what is considered to be a safe value which is often the label power for the system This means that a large amount of reserved extra power is never used This is called the margined power The main purpose of the power cap is not to save power but rather to allow a data center operator the ability to reallocate power from current systems to new systems by reducing the margin ...

Page 365: ...management driver is required for managing Level 2 systems by the IBM Director agent 6 5 1 Integrating x3850 M2 and x3950 M2 into IBM Director Understand the requirements and detection of manageable objects Requirements Integrating your x3850 M2 or x3950 M2 server as a Level 0 managed system does not require that the drivers or the IBM Director agent be installed Level 0 systems are managed by the...

Page 366: ...cessors Understand configuration of LAN interface and implementation of Level 0 LAN interface Systems that have an embedded service processor or installed RSA II can be managed and monitored if the communication to the LAN interface is guaranteed The RSA II adapter in the x3850 M2 and x3950 M2 replaces the embedded BMC so the LAN interface for the RSA II must be configured only to manage it by the...

Page 367: ...ation Model Object Manager CIMOM with SSL support at Linux CIM classification for WMI at Windows and SSH server and platform specific instrumentation Supported tasks Core services allow the following tasks to be performed Inventory information Upgrading to a Level 2 by deployment of the IBM Director agent Management of events by using of event action plans Monitoring hardware status Deployment of ...

Page 368: ...oreservices_windows zip 2 Change to the extraction folder cd install_files_directory FILES 3 Copy the response file directory FILES coresvcs rsp to another location 4 Open the coresvc rsp with an ASCII editor and change the contents as commented in this file Save it with a new filename such as responsefile rsp 5 Select Start Run and then type directory FILES dir5 20 2_coreservices_windows exe s a ...

Page 369: ...re installed and an automatic or manual Discovery is performed the IBM Director detects the systems After IBM Director detects the system that system is listed under Groups as Level1 IBM Director Core Services Systems To unlock the system right click the system name select Request Access and provide your operating system logon credentials See Figure 6 32 The lock disappears and the system can now ...

Page 370: ...e Services must be installed before you can install the LSI MegaRAID Provider Download this extension from http www ibm com systems management director downloads html LSI MegaRAID Provider is not supported with VMware operating systems Operating systems with enabled Xen virtualization layer Linux installation of LSI MegaRAID Provider Install this extension by using the following command rpm ivh ls...

Page 371: ... agent IBM Director agent is on IBM Director installation CD or you can download it as single installation package from http www ibm com systems management director downloads html Linux installation of IBM Director agent To install IBM Director agent 1 Extract the RPM Package by using the following command tar xvf dir5 20 2_agent_linux tar 2 Change to the extraction folder cd install_files_directo...

Page 372: ...older or cd rom path CD coresvc agent windows i386 s is optional to hide file extraction dialog installationtype can be unattended installation progress no user intervention silent hide progress responsefile rsp is path and name of the created response file option can be waitforme ensure complete installation debug logging of the Microsoft Windows installation program messages log logfile creates ...

Page 373: ...ies on CIM and SNMP for remote management 6 6 2 Implementation of x3850 M2 Hypervisor systems IBM Director manages an embedded hypervisor system as a Level 0 system to discover inventory and report color coded hardware failure and RAID events You do not have to possess special operating system knowledge do not have to maintain user accounts and passwords and do not have to install additional secur...

Page 374: ...our IT solution 6 7 1 Processor features The Intel Xeon processors have a number of power saving features The C1E Enhanced Halt State as discussed in C1E on page 110 reduces the internal core frequency and is followed by reducing the internal core voltages if the operating system is in a low state Enhanced Intel SpeedStep Technology EIST developed by Intel as the advancement of the C1E Enhanced Ha...

Page 375: ...Power Distribution Unit iPDU An iPDU has power usage measurement capabilities that report through SNMP to applications such as IBM Active Energy Manager We discuss these in detail in 6 8 Power Distribution Units PDU on page 357 6 7 3 Virtualization Intelligent applications are developed to help a system be more efficient and fully loaded Virtualization technologies in the various NUMA aware operat...

Page 376: ...are to mount in either a standard rack space or side pocket of rack Easily accessible individual breakers per load group for high availability environments Single phase 30A 32A 60A and 63A offerings dependent on the line cord selected Three phase 32A removable line cord and 60A fixed line cord offerings 60A 208V 3 phase model includes fixed IEC60309 3P G 60A line cord The intelligent PDU models ha...

Page 377: ...de Description Outlets Input power C19 C13 71762MX 6090 Ultra Density Enterprise PDU 9 3 Varies by line cord see Table 6 6 71763MU 6091 Ultra Density Enterprise PDU 9 3 60A 208V 3 phased fixed cable 71762NX 6050 Ultra Density Enterprise PDU 9 3 Varies by line cord see Table 6 6 71763NU 6051 Ultra Density Enterprise PDU 9 3 60A 208V 3 phase fixed cable 39M2816 6030 DPI C13 Enterprise PDU 0 12 Varie...

Page 378: ...ovides a description of the Web interface The iPDU collects power and temperature data at the power outlets and can send this information through SNMP This data is collected once per second You can also view this data in table form from the iPDU Web interface or by using tools such as IBM Active Energy Manager They report power and thermal trending also for devices that is plugged into their indiv...

Page 379: ...sage on earlier servers that do not have power metering built in Additionally Active Energy Manager monitors how many amps are being consumed by an iPDU overall and how this compares to the maximum current that the iPDU can support Users are alerted when an iPDU approaches its capacity 6 8 4 Assembling of intelligent PDU The IBM enterprise intelligent PDUs are available in three outlet versions fo...

Page 380: ...PDU part number 39M2818 Figure 6 34 IBM Enterprise DPI C19 PDU UTG connector part number 39M2818 The back of the PDU shows from left to right Left Management interface with RJ45 LAN RJ45 serial console port RS232 serial port DB 9 interface reset button input voltage status LED and operating DIP switch Middle 6x IEC 320 C19 outlets Right UTG0247 inlet connector The front of the PDU has 3x IEC 320 C...

Page 381: ... to nine load groups overall Load group 1 4 7 each are shared with one C13 outlets at the rear side of the PDU Description of the PDU components The following components are on the iPDUs in the previous figures Input power connector Connect a power cord to this connector Some PDU models have an attached power cord Power outlets You can connect a device to each power outlet There are either nine or...

Page 382: ...functions This reset does not affect the loads RJ 45 console connector Connect the DB9 to RJ45 cable that comes with the iPDU to this connector and to the serial COM connector on a workstation or notebook computer and use the workstation or notebook as a configuration console You can also connect an environment monitored probe to this connector The environment monitored probe monitors humidity and...

Page 383: ...Figure 6 36 Home page of the IBM Enterprise DPI C19 PDU Web interface The iPDU stores the measurement of power data and reports it to a graph shown in Figure 6 37 on page 366 and presents a chart of power consumption tendencies ...

Page 384: ...use the previous tool PC Doctor did not meet the requirements for newer systems DSA Preboot will become the standard embedded diagnostic tool for System x and BladeCenter servers DSA Preboot is an NVRAM based version of the of the DSA tool which is used by the Technical Support teams to collect the following information when determining the cause of errors System and component level information Op...

Page 385: ...PCI and USB information Optical devices LSI 1064 1068 1078 SAS RAID controllers Remote Supervisor Adapter BMC Check point panel test CPU and Memory Stress tests IBM light path diagnostics status Service Processor status and configuration Vital product data firmware and BIOS information Drive Health Information ServeRAID configuration LSI RAID Controller configuration Event logs for Service Process...

Page 386: ... DSA The DSA Preboot is then started A list of commands is displayed is shown in Figure 6 39 Figure 6 39 DSA Preboot Commands on the main menu Starting IBM DSA Preboot v1 00 Extracting Commands gui Enter GUI Environment cmd Enter Command Line Environment copy Copy DSA results to removable media exit Quit the program Note This will reboot the system help Display this help message Please enter a com...

Page 387: ...ibed in 6 2 4 Remote console and media on page 324 To update the DSA Preboot 1 Mount the ISO image or created CD ROM in the RSA II Web interface remote control window or insert the CD ROM in the CD ROM drive of your server 2 Boot the server to the CD ROM drive The initial sequence detects the internal 4 GB USB flash device and addresses it by SCSI emulation for USB Mass Storage support shown in Ex...

Page 388: ... Waiting for device to become available usb 5 6 new high speed USB device using ehci_hcd and address 3 usb 5 6 new device found idVendor 0483 idProduct fada usb 5 6 Product ST72682 High Speed Mode usb 5 6 Manufacturer STMicroelectronics scsi1 SCSI emulation for USB Mass Storage devices Vendor ST Model ST72682 Rev 2 10 Type Direct Access ANSI SCSI revision 02 SCSI device sda_ 1024000 512 byte hdwr ...

Page 389: ... directory usr 2 Unpack the package by using the command in Example 6 5 Example 6 5 DSA Preboot extraction in Linux linux x3850m2 usr ibm_fw_dsa_a3yt71a_linux_32_64 sh x usr dsa 3 Change to this directory 4 Perform the update as shown in Example 6 6 Example 6 6 DSA Preboot flash progress of the Update process in Linux srv 253 217 usr dsa lflash64 z Node count 1 Flashing node 0 Waiting for device t...

Page 390: ... DSA Preboot on page 369 the main menu is started as shown in Figure 6 39 on page 368 Select any of the commands listed in Table 6 7 Table 6 7 DSA Preboot commands Command Description gui This starts the graphical user interface cmd The command line environment interface you can use to perform the set of diagnostics collect the system information and show the date in the local text viewer The data...

Page 391: ...ewer To exit viewer type x then press Enter enumtests Lists available tests 1 BMC I2C Bus 2 17 Extended onboard network port tests 18 CPU stress test 19 Memory stress test 20 22 Optical media tests 23 Checkpoint panel test 24 RSA restart test exectest Presents a menu in which you can select a test to execute Use this command to execute the specific test getextendedresults Retrieves and displays ad...

Page 392: ...u and by IBM Service Save the collected information in the operating system The internal 4 GB flash device is hidden in the operating system by default To retrieve the DSA logs that the DSA Preboot collected enable this device temporarily and make it show up as a drive in your operating system Do this by pressing Ctrl End on your keyboard during POST when the IBM logo appears Although no message i...

Page 393: ... so you can then use the following command mount dev sdb1 mnt dsa The results are shown in Example 6 8 Example 6 8 Flash storage device in Linux linux x3850m2 2 fdisk l dev sdb1 Disk dev sdb1 1036 MB 1036353024 bytes linux x3850m2 2 dmesg grep i usb usb 1 6 new high speed USB device using ehci_hcd and address 4 usb 1 6 new device found idVendor 0483 idProduct fada usb 1 6 new device strings Mfr 2 ...

Page 394: ...or power cycle your server this device is hidden again 6 9 3 Working with the graphical user interface GUI The new embedded DSA Preboot in your x3850 M2 and x3950 M2 enables a graphical user interface GUI that you control by keyboard and mouse You start the Dynamic System Analysis GUI in the main menu by using the command gui The GUI window opens shown in Figure 6 44 Figure 6 44 DSA Preboot home w...

Page 395: ...his book are in Use of the interactive menu on page 373 Figure 6 45 DSA Preboot Diagnostics window in the GUI After adding tests you select start the tests by clicking the button Start Tests The Status column shows the result of this test For details about the results click Get Status Details System Information Use the system information window to get an overview about your system or your multinod...

Page 396: ... memory scalability ports PCI Info PCI slot device description with information about the installed PCIe adapters Firmware Details about the installed firmware of your system Environmentals Snapshot of fan speed and environmental monitor LSI SAS subsystem information such as type and firmware of the controller physical and virtual drive information Lightpath Snapshot information about error LED st...

Page 397: ...ions known problems and provides tips for using the DSA Preboot Figure 6 47 DSA Preboot list of help topics in the GUI 6 9 4 Scalability partition management You manage scalable partitions by using the RSA II Web interface This is discussed in Chapter 4 Multinode hardware configurations on page 195 Note If your system indicates an error and you are able to boot to the DSA Preboot diagnostics we re...

Page 398: ...380 Planning Installing and Managing the IBM System x3950 M2 ...

Page 399: ...stency check CD compact disc CD ROM compact disc read only memory CIM Common Information Model Abbreviations and acronyms CIMOM Common Information Model Object Manager CLI command line interface COA Certificate of Authenticity COG configuration and option guide COM Component Object Model CPU central processing unit CRM Customer Relationship Management CRTM Core Root of Trusted Measurements CTO con...

Page 400: ...re Compatibility List HDD hard disk drive HID human interface device HPC high performance computing HPMA High Performance Memory Array HSP hotspare HTML Hypertext Markup Language HW hardware I O input output IBM International Business Machines IBS International Business System ID identifier IDE integrated drive electronics IEEE Institute of Electrical and Electronics Engineers IERR internal error ...

Page 401: ...uter PCI Peripheral Component Interconnect PCIe PCI Express PD problem determination PDF Predictive Drive Failure PDU power distribution unit PEF platform event filtering PFA Predictive Failure Analysis PID partition ID PKT packet PME power management event POST power on self test PPM processor power management PSHED Platform Specific Hardware Error Driver PXE Preboot eXecution Environment RAC Rea...

Page 402: ...l Internet Protocol TOE TCP offload engine TPM Trusted Platform Module TPMD thermal and power management device TSS Trusted Computing Group Software Stack URL Uniform Resource Locator USB universal serial bus UUID Universally Unique Identifier UVHA Unlimited Virtualization with High Availability VCB VMware Consolidated Backup VI VMware Infrastructure VLAN virtual LAN VM virtual machine VMFS virtua...

Page 403: ... 7523 Building an Efficient Data Center with IBM iDataPlex REDP 4418 Enabling Serial Over LAN for a Remote Windows Text Console using OSA SMBridge TIPS0551 Product publications These product publications are also relevant as further information sources System x3850 M2 and x3950 M2 Installation Guide http www ibm com support docview wss uid psg1MIGR 5073028 System x3850 M2 and x3950 M2 User s Guide...

Page 404: ... also relevant as further information sources IBM production information See the following Web addresses for more information IBM Systems home page http www ibm com systems IBM System x3850 M2 product page http www ibm com systems x hardware enterprise x3850m2 IBM System x3950 M2 product page http www ibm com systems x hardware enterprise x3950m2 IBM Announcement letter search http www ibm com com...

Page 405: ...er http www ibm com developerworks linux library l scheduler IBM support documents See the following Web addresses for more information IBM System x support home page http www ibm com systems support x ServerProven compatibility home page http www ibm com servers eserver serverproven compat us ServerProven operating system compatibility home page http www ibm com servers eserver serverproven compa...

Page 406: ...294 Installing SUSE Linux Enterprise Server 10 SP 1 http www ibm com support docview wss uid psg1MIGR 5073088 Flash BIOS update v1 06 DOS package http www ibm com support docview wss uid psg1MIGR 5073120 Remote Supervisor Adapter II update Linux package http www ibm com support docview wss uid psg1MIGR 5073123 Remote Supervisor Adapter II update DOS package http www ibm com support docview wss uid...

Page 407: ...w ibm com support docview wss uid psg1MIGR 5074896 Installing Microsoft Windows Server 2003 x64 Edition http www ibm com support docview wss uid psg1MIGR 5075237 Installing Microsoft Windows Server 2003 Edition 32 bit http www ibm com support docview wss uid psg1MIGR 5075238 IBM Active PCI Software for Windows Server 2008 http www ibm com support docview wss uid psg1MIGR 5074966 IBM Active PCI Sof...

Page 408: ...ndard and Enterprise Editions http www microsoft com sql editions enterprise comparison mspx SQL Server 2005 Features Comparison http www microsoft com sql prodinfo features compare features mspx Application Software Considerations for NUMA Based Systems http www microsoft com whdc archive numa_isv mspx Products Designed for Microsoft Windows Windows Catalog and HCL http www microsoft com whdc hcl...

Page 409: ...5 and ESX Server 3i http www vmware com pdf vi35_systems_guide pdf VMware Infrastructure 3 Configuration Maximums http www vmware com pdf vi3_301_201_config_max pdf http www vmware com pdf vi3_35 esx_3 r35 vi3_35_25_config_max pdf ESX Server 3 Configuration Guide http www vmware com pdf vi3_35 esx_3 r35 vi3_35_25_3_server_config pdf ESX Server 3 Installation Guide http www vmware com pdf vi3_35 es...

Page 410: ...er novell com yessearch Search jsp A NUMA API for LINUX http www novell com collateral 4621437 4621437 pdf SUSE Linux Enterprise Server 10 Tech Specs System Requirements http www novell com products server techspecs html Intel See the following Web addresses for more information Execute Disable Bit and Enterprise Security http www intel com technology xdbit index htm Intelligent Platform Managemen...

Page 411: ...hat com drepper cpumemory pdf Red Hat Enterprise Linux Server Version comparison chart http www redhat com rhel compare Red Hat Hardware Catalog https hardware redhat com Other See the following Web addresses for more information IPMItool http ipmitool sourceforge net OpenIPMI http openipmi sourceforge net TrouSerS http trousers sourceforge net Java Downloads for all operating systems http www jav...

Page 412: ...books You can search for view or download Redbooks Redpapers Technotes draft publications and Additional materials as well as order hardcopy Redbooks at this Web site ibm com redbooks Help from IBM IBM Support and downloads ibm com support IBM Global Services ibm com services ...

Page 413: ...Manager 51 100 334 346 BIOS setting 337 capping 346 components 336 console 337 database 337 functions 340 GUI 337 iPDU support 343 managed systems 337 PDU support 343 power capping 346 Power Saver mode 345 power trending 341 sensors 342 server 336 tasks 338 terminology 335 thermal trending 342 trending 341 watt hour meter 344 Active Memory 40 addressing 37 air baffle 96 applications 54 64 scalabil...

Page 414: ...ating 251 BitLocker 52 BladeCenter 57 58 block diagram 28 Dunnington processor 36 ServeRAID MR10M 135 Tigerton processor 35 x3850 M2 x3950 M2 27 BMC account settings 305 BIOS settings 302 configuration 300 connectivity 302 drivers 308 environmental monitoring 301 event log 304 features 301 firmware update 208 307 overview 48 300 POST watchdog 306 SMBridge 305 updating 250 user account settings 305...

Page 415: ...d 130 ports 125 RAID controller 124 RAID levels 129 redundancy features 45 SAS onboard controller 124 ServeRAID MR10k controller 128 ServeRAID MR10M controller 135 drive encryption 52 DSA Preboot updating 251 369 DSA preboot 366 379 dual core processors 33 Dunnington 5 33 90 DVD drive 7 E eCommerce 55 EDB Execute Disable Bit 108 EIDE interface 44 EM64T 37 embedded virtualization 4 19 264 encryptio...

Page 416: ...Plex 58 IBM Systems Director Active Energy Manager 51 iDataPlex 57 58 initial ROM size 193 Initialization Scrub Control 118 123 installation EXP3000 145 IBM Director 349 IPMI driver 309 processors 93 Red Hat Enterprise Linux 293 295 ServeRAID MR10k 132 ServeRAID MR10M 137 138 SUSE Linux Enterprise Server 295 297 VMware ESX 281 285 VMware ESXi 264 281 Windows Server 2003 285 289 Windows Server 2008...

Page 417: ...ing 40 minimum 115 mirroring 41 118 122 modes 122 multinode configurations 114 NUMA 67 options 111 order of population 116 pairs of DIMMs 114 performance optimized configuration 117 ports 28 replacing a hot swap DIMM 120 replacing or adding 115 request queue 67 Run Time Scrub Rate 123 scalability allocation 115 scrub control 118 123 XceL4v allocation in multinode 114 merge defined 196 failures 239...

Page 418: ... 227 clearing all 232 configuration steps 225 configure 222 creating 226 defined 197 199 delete a partition 232 manual partitioning 227 power off 231 reorder 223 reset 231 standalone boot 222 starting 228 status 224 Partitions Configure buttons 222 patrol read 130 165 PCI Device Boot Priority 155 189 PCI Express subsystem 43 188 bridges 29 30 card installation order 189 hot plug support 192 initia...

Page 419: ...from internal drives 293 features 78 installation 293 295 LSI MegaRAID Provider 352 memory limits 78 NUMA 77 RSA II driver 330 scaling 77 support 261 Trusted Platform Module 295 Redbooks Web site 394 Contact us xvi redundancy 45 redundant power supplies 4 remote console 324 Remote Supervisor Adapter II See RSA II Remote Supervisor Adapter II See RSA II request queue 67 reset a partition 231 ROM si...

Page 420: ...ebBIOS battery backup 137 block diagram 135 cache 135 external x4 port 135 firmware update 150 iBBU 137 IBM Director provider 352 installation 137 138 virtual disks 136 ServerProven 258 SFF 8087 internal SAS cable 125 SFF 8088 external SAS port 126 slots PCI 188 SMART error 165 SMBridge 305 SMP 66 SMP expansion ports 4 snoop filter 30 snoop filter lookup table 28 socket 34 Soft NUMA 82 Solaris sup...

Page 421: ...configuration 284 NUMA controls 70 page migration 69 page sharing 70 partitions 283 prerequisites 281 rebalancer 69 RSA II driver 330 SATA support 72 scalability 68 72 scheduling 68 sharing memory 70 support 259 transparent page 70 VMotion processor compatibility 34 VMware HA 73 VMware ESXi 19 BIOS settings 265 boot order 265 CIM 274 comparison 22 customizing 269 DNS configuration 271 features 20 ...

Page 422: ...T error 165 spinup delay 166 spinup drive count 166 stop on error 166 Stripe Size 185 Unconfigured good 183 virtual disk properties 168 169 virtual drive states 168 wizard 179 WHEA support 7 Windows Datacenter Server 15 19 Windows Server 2003 BIOS settings 286 290 boot from internal drives 286 driver diskette 288 hot add memory 74 120 installation 285 287 289 IPMI drivers 309 LSI MegaRAID Provider...

Page 423: ... 0 5 spine 0 475 0 875 250 459 pages Planning Installing and Managing the IBM System x3950 M2 ...

Page 424: ......

Page 425: ......

Page 426: ...with enhanced reliability and availability features to enable optimal performance for databases enterprise applications and virtualized environments The x3950 M2 features make the server ideal for handling complex business critical On Demand Business applications such as database serving business intelligence transaction processing enterprise resource planning collaboration applications and server...

Search results

Summary of Contents for 88743BU - System x3950 E

Reviews:

Related manuals for 88743BU - System x3950 E

Brands by name

Popular brands

IBM 88743BU - System x3950 E, User Manual

Search results

Summary of Contents for 88743BU - System x3950 E

Reviews:

Related manuals for 88743BU - System x3950 E

Brands by name

Popular brands