manualshive.com logo in svg

Fujitsu PRIMEQUEST 440, System Design Manual, Page: 246 / 280

Share
Download
Fujitsu PRIMEQUEST 440 System Design Manual Download Page 246

10-12

C122-B001-02EN

CHAPTER 10 Backup and Restoration

 

10.3

 Comparison of Backup Methods

Table 10.1

 lists the usage method and characteristics of various backup and 

restoration methods.  The requirements for the backup methods vary with the 
operation.  The methods appropriate to each requirement need to be combined to back 
up data.

Table 10.1  Comparison of backup methods

Method

Main usage

Characteristics

OS-standard

dump/restore 
(Linux)
NTBackup 
(Windows)

Backup of a 
system volume 
(single)

• It is possible to back up and restore a 

system volume.

• These are OS-standard functions.  The 

user is normally accustomed to their 
operation.

Cloning 
software

Systemcast 
Wizard Lite 
(Linux/
Windows)

Backup of a 
system volume 
(batch)

• It is possible to back up and restore a 

system volume.

• Automatic processing is possible in 

cooperation with a MMB via a network.

• It is possible to perform concurrent 

parallel backup of multiple partitions.

Summary of Contents for PRIMEQUEST 440

Page 1: ...C122 B001 02EN Front Cover SYSTEM DESIGN GUIDE ...

Page 2: ......

Page 3: ...as office work personal device household appliance and general industrial applications This product is not intended for use in nuclear reactor control systems aeronautical and space systems air traffic control systems mass transportation control systems medical devices for life support missile launch control systems or other specialized uses in which extremely high levels of reliability are requir...

Page 4: ...re designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment This equipment generates uses and can radiate radio frequency energy and if not installed and used in accordance with the instruction manual may cause harmful interference to radio communications Operation of this equipment in a residential area is likely to cause ha...

Page 5: ...ther countries z Intel Xeon and Itanium2 are trademarks or registered trademarks of Intel Corporation z Ethernet is a registered trademark of Fuji Xerox Co Ltd and Xerox Corporation in the United States and other countries z All other product names mentioned herein are the trademarks or registered trademarks of their respective owners z System and product names in this manual are not always noted ...

Page 6: ......

Page 7: ...tails 01 2005 07 11 02 2005 09 16 All chapters correction Section 9 3 addition Technical brushup Addition of description for cluster of Windows Note In this table the revised section is indicated by its section number in the current edition An asterisk indicates a section in the previous edition ...

Page 8: ......

Page 9: ...Contents of This Manual This manual is organized as described below CHAPTER 1 PRIMEQUEST Overview Describes features of the main unit and software CHAPTER 2 General Hardware Configuration Describes the hardware mounted in the main unit CHAPTER 3 General Software Configuration Describes the OS firmware supplied software and software that can be linked CHAPTER 4 Hardware System Management Describes ...

Page 10: ...ng a cluster system CHAPTER 10 Backup and Restoration Describes system maintenance by explaining the necessity of backups how to back up and restore the PRIMEQUEST system and other such topics CHAPTER 11 Considerations for Maintenance Describes the system maintenance considerations that must be made during system design Appendix A Software Supplied with PRIMEQUEST Hardware Describes the software s...

Page 11: ...E001EN b PDF manuals included on the PRIMEQUEST Manual CD ROM disk C122 E013EN C2 PRIMEQUEST 480 440 Installation Planning Manual C122 H001EN PRIMEQUEST 480 440 Installation Manual C122 E001EN PRIMEQUEST 480 440 Operation Manual C122 E002EN PRIMEQUEST 480 440 Reference Manual GUI Commands C122 E003EN PRIMEQUEST 480 440 Reference Manual Messages Logs C122 E004EN ...

Page 12: ...for a mandatory keyword must be enclosed in and delimited by Environmental Requirements for Using This Product This product is a computer which is intended to be used in a computer room For details on the operational environment see the PRIMEQUEST 480 440 Installation Planning Manual C122 H001EN Fonts symbols Meaning Example Italic Indicates names of manuals See the PRIMEQUEST 480 440 System Desig...

Page 13: ...s in this manual should only be performed by a certified service engineer Users must not perform these tasks Incorrect operation of these tasks may cause electric shock injury or fire z Installation and reinstallation of all components and initial settings z Removal of front rear or side covers z Mounting de mounting of optional internal devices z Plugging or unplugging of external interface cards...

Page 14: ......

Page 15: ...2 B001 02EN vii NOTE ON SAFETY 2 Alert Labels The following labels are attached to this product These labels provide information to the users of this product Main unit Do not peel off the labels Front view ...

Page 16: ...viii C122 B001 02EN NOTE ON SAFETY Rear view Unit rear ...

Page 17: ...C122 B001 02EN ix NOTE ON SAFETY Extended Power Cabinet Extended I O cabinet Front view Rear view Front view Rear view ...

Page 18: ...x C122 B001 02EN NOTE ON SAFETY PCI_Box Left side view Front view without cover ...

Page 19: ...Plugging or unplugging of external interface cards z Maintenance and inspections repairing and regular diagnosis and maintenance The following tasks regarding this product and the optional products provided from Fujitsu should only be performed by a certified service engineer Users must not perform these tasks Incorrect operation of these tasks may cause malfunction z Unpacking optional adapters a...

Page 20: ......

Page 21: ...omponents 1 4 1 1 2 Product lineup 1 5 1 2 PRIMEQUEST Features 1 7 1 2 1 Hardware technologies 1 8 1 2 1 1 PRIMEQUEST dedicated chipset 1 8 1 2 1 2 IPF 1 9 1 2 1 3 Symmetric multi processing SMP technology 1 9 1 2 1 4 System Mirror 1 10 1 2 1 5 Partition function 1 12 1 2 1 6 Flexible I O mechanism 1 15 1 2 1 7 Hardware redundancy and hot maintenance function 1 19 1 2 1 8 Management board MMB 1 21...

Page 22: ...e BP 2 21 2 5 2 XAI 2 22 2 5 3 XDI 2 23 2 6 GSWB Gigabit Switch Board 2 24 2 7 KVM Interface Unit KVM 2 26 2 8 Clock and PCI_Box Control Board CPCB 2 27 2 9 OPL 2 29 2 10 AC Section ACS 2 30 2 11 Power Supply Unit PSU 2 31 2 12 Fan Cooling Unit 2 33 2 13 Dual Power Feed Option 2 34 2 14 PCI_Box 2 34 2 14 1 PCIU 2 37 2 14 2 PCI slot 2 37 2 14 3 PCI card cassette 2 38 2 14 4 Compatible PCI cards 2 3...

Page 23: ...e LAN business LAN 4 3 4 1 1 1 Management LAN 4 4 4 1 1 2 Private LAN 4 8 4 1 1 3 Business LAN 4 9 4 1 2 IP addresses of PRIMEQUEST management LAN 4 10 4 2 MMB Functions 4 12 4 2 1 User privilege management 4 13 4 2 2 Hardware configuration display 4 14 4 2 3 Hardware monitor 4 15 4 2 4 Partition settings and configuration display 4 17 4 2 5 KVM USB connection switching 4 18 4 2 6 Console redirect...

Page 24: ...6 Port ranking function 5 18 5 2 7 IGMP snooping 5 19 5 2 8 Port mirroring 5 20 5 2 9 Statistical information and log information function 5 21 5 2 10 SNMP 5 21 5 2 11 Managing units and others 5 22 5 3 Notes on System Design 5 23 5 3 1 Learning table 5 23 5 3 2 Relay inhibited frame 5 26 5 3 3 Port mirroring 5 27 5 3 4 Notes on the Speed Duplex setting of the interface 5 29 5 3 5 Flow control 5 3...

Page 25: ... and reserved SB 8 5 8 3 3 Notes on System Mirror operation 8 6 8 3 4 Notes on the memory configuration 8 7 8 3 5 Operating environment 8 7 8 4 Function for Changing Partition Configurations 8 8 8 5 Notes to Observe when Configuring Partitions 8 8 CHAPTER 9 Clustering 9 1 9 1 The basics of clustering 9 1 9 2 Cluster Functions Provided in Linux Environments 9 5 9 2 1 Cluster Configurations 9 5 9 2 ...

Page 26: ... Maintenance 11 1 11 1 Estimating the Dump use Area 11 1 Appendix A Software Supplied with PRIMEQUEST Hardware A 1 Appendix B Status Confirmation from LED B 1 B 1 LED Display on the Operator Panel B 2 B 2 LED Display of Each Board or Component B 3 B 2 1 LED display for each board or component B 3 B 2 1 1 LED display of the MMB B 4 B 2 1 2 LED display of a GSWB B 5 B 2 2 Other boards B 5 B 2 3 List...

Page 27: ...in components 1 20 Figure 1 14 MMB 1 21 Figure 1 15 GSWB 1 22 Figure 1 16 Clustering 1 25 Figure 1 17 Fujitsu middleware products 1 26 Figure 2 1 PRIMEQUEST 480 physical configuration full package 2 2 Figure 2 2 PRIMEQUEST 440 physical configuration full package 2 3 Figure 2 3 Main unit 2 5 Figure 2 4 SB 2 6 Figure 2 5 Memory DIMM slot numbers and locations 2 8 Figure 2 6 IO_Unit Full package 2 10...

Page 28: ...on switching setting screen 4 19 Figure 4 14 Flow of redirected information 4 20 Figure 4 15 Console Redirection Switch screen 4 21 Figure 4 16 System account management method 4 22 Figure 4 17 Example of setting the usage evironment 4 23 Figure 4 18 Example of the access control settings screen 4 23 Figure 4 19 Time synchronization image 4 24 Figure 4 20 Example of the scheduled operation display...

Page 29: ...mple of implementing redundancy using the high speed switching method 6 17 Figure 6 13 High speed switching method 6 18 Figure 6 14 Example of implementing redundancy using the NIC switch method 6 18 Figure 6 15 NIC switching method 6 19 Figure 7 1 Layout of included components 7 2 Figure 8 1 Example of partition configuration 8 2 Figure 8 2 Allocation of a PCI_Box to partitions 8 3 Figure 8 3 Par...

Page 30: ...10 5 Figure 10 4 Backup of data on the ETERNUS by using ACM 10 7 Figure 10 5 Data backup using VERITAS NetBackup 10 8 Figure 10 6 Online backup of a DB using VERITAS NetBackup 10 9 Figure 10 7 Backup of a system volume using NetVault 10 10 Figure 10 8 Configuration for data backup with the ETERNUS SP5000 10 11 Figure B 1 LEDs on the operator panel B 2 Figure B 2 LEDs of the MMB B 4 Figure B 3 LED ...

Page 31: ... external interfaces 2 26 Table 2 16 CPCB external interfaces 2 28 Table 2 17 LED display on the OPL 2 29 Table 2 18 ACS specifications 2 30 Table 2 19 PSU specifications 2 31 Table 2 20 Number of mounted SBs and IO_Units and PSU configuration PRIMEQUEST 480 2 32 Table 2 21 Number of mounted SBs and IO_Units and PSU configuration PRIMEQUEST 440 2 32 Table 2 22 PCI_Box configuration for PRIMEQUEST ...

Page 32: ... enabled disabled 5 27 Table 5 17 VLAN Tag of mirroring frame transfer from port A to port B 5 28 Table 5 18 VLAN Tag of mirroring frame transfer from port B to port A 5 29 Table 5 19 Notes on the Speed Duplex setting of the interface 5 29 Table 5 20 Flow control setting 5 30 Table 5 21 Load distribution algorithm 5 31 Table 5 22 Example of load distribution 5 32 Table 6 1 System Mirror Specificat...

Page 33: ...ium Processor Family processors hereafter referred to as IPF processors or simply as CPUs which are 64 bit processors manufactured by Intel Corporation The PRIMEQUEST lineup consists of two models the PRIMEQUEST 480 and the PRIMEQUEST 440 The PRIMEQUEST 480 is a high end server model that houses up to 32 CPUs and the PRIMEQUEST 440 is a middle range server model that houses up to 16 CPUs The PRIME...

Page 34: ...e models support Linux and Windows operating systems hereafter sometimes abbreviated as OSs that are available in 64 bit processor configurations Figure 1 1 shows the appearance of the PRIMEQUEST 440 PRIMEQUEST 480 Figure 1 1 PRIMEQUEST 440 PRIMEQUEST 480 ...

Page 35: ...cation programs as well as hardware resources such as system boards SBs and IO_Units Because partitions are independent of each other they can be used as single servers that are assigned different tasks Each partition can be connected to an external storage device via a PCI card within the cabinet or through a PCI_Box It can also interact with the operating system via a remotely connected or seria...

Page 36: ...ossbar MMB and GSWB Figure 1 3 shows the connection configuration of these main components In Figure 1 3 a rectangle represents one SB IO_Unit MMB or GSWB board System boards SBs are boards on which CPUs memory modules PRIMEQUEST dedicated chipsets and other components are mounted The PRIMEQUEST 480 can contain up to eight SBs and the PRIMEQUEST 440 can contain up to four SBs within the cabinet On...

Page 37: ...nstruct a business LAN One GSWB has eight 1000Base T ports The GSWB also can be duplicated to increase its reliability Figure 1 3 Configuration of all hardware PRIMEQUEST 480 1 1 2 Product lineup The PRIMEQUEST lineup consists of two models a high end server that accommodates up to 32 CPUs and a mid range server that accommodates up to 16 CPUs Table 1 1 Hardware specifications Item Specification P...

Page 38: ...SB Video keyboard USB mouse USB DVD ROM drive 2 1 1 Number of PCI slots Up to 128 slots Built in 32 slots PCI_Box 96 slots 12 8 boxes Up to 64 slots Built in 16 slots PCI_Box 48 slots 12 4 boxes PCI_Box option Up to 8 boxes 12 PCI slots per PCI_Box Up to 4 boxes 12 PCI slots per PCI_Box Server management MMB MMB Redundant configuration SB GSWB MMB PSU fan SB GSWB MMB PSU fan Hot swapping SB IO_Uni...

Page 39: ...l server management operations such as those for partition related settings and status monitoring can be centrally managed from the MMB Web UI Figure 1 4 shows the technologies underlying the three features The following sections describe the overview of these technologies Figure 1 4 PRIMEQUEST technologies Linkage of functions High reliability and high availability High performance High operabili...

Page 40: ...Us memory and IO_Units The PRIMEQUEST series machines contain this new chipset that has been developed by Fujitsu for improved performance and increased reliability The following three achievements are made by the PRIMEQUEST dedicated chipset Large scale SMP The high speed memory access technology in the chipset suppresses deteriorations in performance in a large scale SMP configuration resulting ...

Page 41: ...penness Since the IPF processors are open standard IA processors a wide range of IPF independent software vendor ISV and independent hardware vendor IHV products can be used thereby expanding the selection of products for system construction 1 2 1 3 Symmetric multi processing SMP technology Each processor is the same as any other processor in a multiprocessor system using SMP technology so process...

Page 42: ...r a license key optional product is required For details of System Mirror see Section 6 3 System Mirror Figure 1 5 System Mirror Remarks This function is different from disk mirroring RAID1 which writes data to two hard disk drives at the same time For disk mirroring see Section 6 4 Redundancy of an HDD Use of PRIMECLUSTER GDS Improved business continuity with System Mirror The PRIMEQUEST System M...

Page 43: ...e faulty component can be replaced quickly and operation can return to system mirrored operation quickly Thus System Mirror can improve business continuity in the event of a hardware failure Figure 1 6 Improved business continuity with System Mirror Operating Operating Task stopped Operating Operating Operating Another system continues processing One system stopped Failure occurrence Failure occur...

Page 44: ...40 Figure 1 7 Partitions The partition function provides the following advantages Flexible system construction The partition function can be used to construct multiple business systems in the same cabinet thereby enabling flexible system construction Robustness Hardware protects each partition from being affected by another partition If a failure occurs in a partition the other partitions are not ...

Page 45: ...ities For details on the flexible I O mechanism see Section 1 2 1 6 Flexible I O mechanism Figure 1 8 Independent resource expansion in each partition Partition 0 Partition 1 Partition 2 Partition 0 Partition 1 Partition 2 Crossbar MMB 0 MMB 1 IOU 3 IOU 4 IOU 5 IOU 6 IOU 7 SB 1 SB 0 SB 2 SB 3 SB 6 SB 7 IOU 0 MMB 0 MMB 1 IOU 1 IOU 2 IOU 3 IOU 4 IOU 5 IOU 6 IOU 7 SB 1 SB 0 SB 2 SB 3 SB 4 SB 5 SB 6 S...

Page 46: ...fferent OS version This makes possible flexible development and capability improvements so hardware resources can be used effectively Figure 1 9 Operation on different platforms Partition 0 Partition 1 Partition 2 Crossbar MMB 0 MMB 1 IOU 1 IOU 2 IOU 3 IOU 4 IOU 5 IOU 6 IOU 7 SB 1 SB 0 SB 2 SB 3 SB 4 SB 5 SB 6 IOU 0 SB 7 Windows Linux RHEL4 Linux RHELn Windows Linux RHEL4 Linux RHELn RHEL Red Hat ...

Page 47: ...ase processing and other processing that handles a large volume of data In the above figure Partition 1 operation requires more CPU resources than I O resources and the I O resources are underused This state is typical in processing that requires a high CPU processing capacity such as scientific and technological computational processing As explained above resources such as CPUs and memory cannot ...

Page 48: ...aulty computer does not operate before it is repaired The shorter the MTTR the easier the maintenance and repair Free resource allocation with the flexible I O mechanism The flexible I O mechanism enables flexible use of hardware resources Specifically SBs and IO_Units are freely connected by the flexible I O mechanism in the crossbar and the flexible I O mechanism provides diversity and flexibili...

Page 49: ...umber of IO_Units As shown in this example CPUs memory and I O resources can be allocated independently because of the flexible I O mechanism Therefore system resources can be used more effectively That is resources can be allocated as required according to the operation mode If an SB fails in a partition that has a reserved SB prepared the current task can be continued with the reserved SB which ...

Page 50: ...d SB 4 as is by the flexible I O mechanism This shortens the MTTR to the time necessary for rebooting only resulting in a great improvement in system availability Figure 1 12 Reduced MTTR Crossbar Crossbar SB 0 SB SB SB SB 1 2 3 4 IOU IOU IOU IOU IOU 0 1 2 3 4 SB 0 SB SB SB SB 1 2 3 4 IOU IOU IOU IOU IOU 0 1 2 3 4 Partition 0 Partition 1 Reserved SB Partition 0 Partition 1 Rebooting partition 1 SB...

Page 51: ...edicated internal chipset is duplicated z Board For most board types two or more boards can be mounted for a redundant configuration Therefore if one board fails operation can continue with another board z Power supply unit Redundant power supply units are provided so if one power supply unit fails operation can continue with another power supply unit A dual power feed has been prepared as an opti...

Page 52: ...supply units PSUs and fans are in redundant configurations as standard hot maintenance of PSUs and fans is possible z Since the MMB which provides the server management function is in a redundant configuration hot maintenance of the MMB is possible z Hot maintenance of an SB IO_Unit HDD system interconnect PCI card GSWB and other devices is possible under specific conditions z Hot maintenance of a...

Page 53: ...duplicate MMB can continue processing Hot maintenance of the MMB is possible Figure 1 14 MMB The built in MMB provides the following advantages Reduced management costs Management costs can be greatly reduced because the server management function does not require a dedicated management server or dedicated software Centralized management The MMB can centrally monitor control and set up hardware re...

Page 54: ... connecting an external LAN switch hub wireless the time necessary for cable connection and management can be reduced This also can prevent mistaken cable connections Centralized management Centralized management of GSWB status monitoring for failure occurrences and GSWB control can be implemented from the MMB Web UI Virtual LAN VLAN A single GSWB port can be shared by multiple groups and each gro...

Page 55: ...ws z Clustering software z Fujitsu middleware products 1 2 2 1 Firmware The following hardware management and operation firmware is installed on PRIMEQUEST series machines For details see CHAPTER 3 General Software Configuration z MMB firmware Monitoring of all hardware and centralized operation z PSA system management software Monitoring of hardware in individual partitions z GSWB firmware Centra...

Page 56: ...f any particular enterprise OS vendor Continuous technological improvement and development of new technologies by many developers and researchers High reliability Objective of continuous increases in reliability such as with the following high reliability functions Quick fixes to security holes because engineers around the world tackle defects Logging of detailed hardware error information Enhance...

Page 57: ...es ranging from hardware to middleware and applications implements autonomous control such as failover and thereby enables operation to continue safely Even the work done in periodic maintenance and scheduled stoppages such as a system configuration change can be performed without affecting the resources in use As a result the service availability time is greater Figure 1 16 Clustering PRIMEQUEST ...

Page 58: ...d know how z Interstage Business integration software z Symfoware Database software z Systemwalker Integrated operation management software Figure 1 17 Fujitsu middleware products OS Linux Windows PRIMEQUEST Interstage business integration Symfoware database Systemwalker integrated operation management PRIMECLUSTER clustering Linkage Middleware products using Fujitsu s reliable technologies Middle...

Page 59: ...compose a PRIMEQUEST series machine z Main unit Cabinet System Board SB IO_Unit Management Board MMB Crossbar GSWB Gigabit Switch Board KVM Interface Unit KVM Clock and PCI_Box Control Board CPCB OPL AC Section ACS Power Supply Unit PSU Fan Cooling Unit Dual Power Feed Option PCI_Box z Other Options and Other Devices UPS ...

Page 60: ...ware Configuration The configurations of PRIMEQUEST components are shown below Physical configuration PRIMEQUEST 480 Figure 2 1 PRIMEQUEST 480 physical configuration full package FAN_A 1 FAN_A 0 FAN_A 3 FAN_A 2 FAN_A 5 FAN_A 4 FAN_B 1 FAN_B 0 ...

Page 61: ...2 B001 02EN 2 3 CHAPTER 2 General Hardware Configuration Physical configuration PRIMEQUEST 440 Figure 2 2 PRIMEQUEST 440 physical configuration full package FAN_A 1 FAN_A 0 FAN_A 3 FAN_A 2 FAN_B 1 FAN_B 0 ...

Page 62: ...ne CPU 32 4 8 SB 16 4 4 SB Each SB requires at least one CPU Memory DIMM 256 32 8 SB 128 32 4 SB Each SB requires at least four memory modules IO_Unit 8 4 The minimum is one BMM 8 4 PCI 32 16 HDD 32 16 BP 1 1 XDI 4 4 XAI 2 2 MMB 2 2 KVM 1 1 DVD drive 1 1 CPCB 1 1 GSWB 2 2 Optional product PDB 4 4 The minimum is four ACS 2 1 PSU 12 12 The minimum is six FAN FAN_A 6 4 FAN_B 2 2 OPL 1 1 PCI_Box 8 4 O...

Page 63: ...et The MMB can be operated through a Web interface from a general purpose personal computer The gigabit switch board GSWB which is a switching hub compatible with Gigabit Ethernet is an optional product that can be mounted in the cabinet Figure 2 3 shows the appearance of the cabinet with its front and rear covers removed example of the full package of the PRIMEQUEST 480 Figure 2 3 Main unit Fan F...

Page 64: ...SBs z The SB contains one to four CPUs and one set four modules to eight sets 32 modules of memory z At least one CPU and one set of memory must be mounted on an SB that composes a partition Note For details on the partition function see CHAPTER 8 Partition Figure 2 4 SB Table 2 2 SB specifications Item Specifications Number of CPUs 1 to 4 Number of memory modules DDR2 DIMM 4 to 32 added in sets o...

Page 65: ...anced by the error detection and correction function incorporated for cache memory etc This improves business continuity z The IPF improves performance because its 64 bit processors use an expanded memory address space and have higher computing speeds z One to four CPUs can be mounted on one SB Table 2 3 CPU specifications Core frequency FSB frequency Cache size 1 5 GHz 400 MHz 4 MB 1 6 GHz 400 MH...

Page 66: ...r correction function for high system reliability S4EC D4ED is the DIMM error check function for 16 bytes 128 bits z The memory module is handled in sets of four modules and one four modules to eight set 32 modules can be mounted on one SB Figure 2 5 Memory DIMM slot numbers and locations Table 2 4 Memory DIMM types System clock FSB DIMM operation clock frequency 800 MHz 400 MHz 400 MHz 2A0 2A1 3A...

Page 67: ...m Slot 0 z The DIMMs in a channel mounted only to Slot 1 are isolated by the initial diagnosis at boot z If the initial diagnosis detects a fixed one bit error in a DIMM during normal mode operation isolation of the DIMM is by the unit of addition four modules For this reason it is recommended to mount two sets of DIMMs or more eight modules or more 2 2 2 1 Memory interleave Memory interleave is t...

Page 68: ...CSI USB USB 2 0 and COM ports are provided as the external interfaces z One to four HDDs which are hot pluggable can be mounted z Up to four PCI cards which are hot pluggable can be mounted z Up to two BMMs which would be IO_Unit control boards can be mounted as options z Four PCI Box interface ports of a serial bus can be mounted Figure 2 6 IO_Unit Full package HDD 0 HDD 1 HDD 2 HDD 3 HDD 1 HDD 2...

Page 69: ...s and a COM port are also mounted on the BMM 2 3 2 HDD The HDD is connected to the SCSI BP SCBP Four U320 SCSI 3 5 inch HDDs which are hot pluggable can be mounted HDD mirroring is supported with optional software For details see Section 6 4 Redundancy of an HDD Use of PRIMECLUSTER GDS External interface Count Remarks USB 4 Mounted on optional BMM product RS 232C 2 SCSI 2 Ultra320 SCSI PCI Box int...

Page 70: ...hot swappable Figure 2 7 PCI card cassette Table 2 6 PCI slot specifications Notes 1 The PCI slot is compatible with only the standard type full height PCI card bracket 2 Only short PCI slots are mounted in an IO_Unit If a long PCI slot is required use an optional PCI_Box PCI slot No Bus width Operating clock PCI bus operating voltage Size PCI Slot 0 64 bit 100 MHz 3 3 V Short PCI Slot 1 64 bit 13...

Page 71: ...g PCI cards Table 2 7 Compatible PCI cards Type Function SCSI card Ultra320 SCSI dual channel Fibre Channel card Fibre Channel 2 Gbps 1 port Network related PCI card Gigabit Ethernet 1000Base T dual channel Gigabit Ethernet 1000Base SX single channel Gigabit Ethernet 1000Base SX dual channel ...

Page 72: ...y rank of 1 or 2 see Table 2 9 If two or more partitions are defined and operated these priority ranks apply to each partition 2 An SAN bootable FC card can be mounted in only PCI Slot 1 and PCI Slot 3 of an IO_Unit with a priority rank 1 or 2 see Table 2 9 When two or more partitions are defined for operation these priority ranks apply to each partition same conditions as the SCSI card slots in 1...

Page 73: ...mallest after that of the IO_Unit with a priority rank of 3 except the home IO_Unit 5 The IO_Unit with the IO_Unit number next smallest after that of the IO_Unit with a priority rank of 4 except the home IO_Unit 6 The IO_Unit with the IO_Unit number next smallest after that of the IO_Unit with a priority rank of 5 except the home IO_Unit 7 The IO_Unit with the IO_Unit number next smallest after th...

Page 74: ...to the KVM z Video port on the BMM connected to the KVM video port z COM port on the BMM z 100MbE port on the BMM z Built in HDD z External SCSI port z GbE port Table 2 10 Usable built in IOs within the same partition Y usable built in I O N unusable built in I O Built in IO IO_Unit with a priority rank of 1 Home IO_Unit IO_Unit with a priority rank of 2 IO_Unit with a priority rank of 3 IO_Unit w...

Page 75: ...U 0 IOU 2 and IOU 3 which have priority ranks of 1 2 3 and 4 respectively can be used If a built in HDD is mounted in an IO_Unit in this partition it needs to be mounted in IOU 1 IOU 0 IOU 2 or IOU 3 which have priority ranks of 1 2 3 and 4 respectively Figure 2 8 Mounting conditions of SCSI cards and usable built in I Os within the same partition Home IOU IOU 1 IOU 0 0 1 2 3 0 1 2 3 PCI_Box 0 7 P...

Page 76: ... IO_Unit only PCI Slot 0 and PCI Slot 1 can be used The mapping mode can be defined for each partition Table 2 11 lists the PCI bus mapping modes supported by each operating system Table 2 11 PCI bus mapping modes 1 Fujitsu recommends the PCI segment mode when Red Hat Enterprise Linux is used OS PCI segment mode PCI bus mode Red Hat Enterprise Linux Supported 1 Supported Windows Server 2003 Enterp...

Page 77: ... Since the MMB centrally manages the whole system unit management does not require any specific console or software The MMB has the following management functions Cabinet power control Remote operations including configuring settings displaying the status and various other operations from the MMB Web UI Remote control of unit power and partition resets Monitoring sensors and other devices Partitio...

Page 78: ...e REMCS port and a partition thereby achieving a high security level For details see Section 4 1 1 1 Management LAN Remarks When Speed Duplex for the LAN port on the MMB is set to an item other than AUTO use a cross cable to connect a switching hub to the MMB LAN port External interface Count Remarks RS 232C 1 Used when the Fujitsu certified service engineer performs the setup of the equipment It ...

Page 79: ... of boards Back Plane BP Xbar address interconnect board XAI Xbar data interconnect board XDI Table 2 13 Crossbar specifications 2 5 1 Back Plane BP Connectors for SBs IO_Units and other devices are mounted on the BP Only wiring and connectors are mounted on the BP and no DDC or control chip is mounted Item Specifications Transmission mode Stored and forward Number of ports SB port 8 IO_Unit port ...

Page 80: ...ing the address bus system of the crossbar is mounted on the Xbar address interconnect board XAI The XAI ports connect eight SBs and eight IO_Units and the XAI transmits address information between the ports z Two XAIs are mounted in the main unit Figure 2 10 XAI XAI Power XAI Alarm Location ...

Page 81: ...the data bus system of the crossbar is mounted on the Xbar data interconnect board XDI The XDI ports connect eight SBs and eight IO_Units and the XDI transmits data between the ports z Four XDIs are mounted in the main unit Figure 2 11 XDI XDI Power XDI Alarm Location ...

Page 82: ...al product The main unit accommodates up to two GSWBs The GSWB functions as a Gigabit Ethernet switching hub Figure 2 12 GSWB The following table lists the external interfaces provided by GSWB Table 2 14 GSWB external interfaces Interface Count LAN 1000Base TX 8 GSWB 1000 3 4 7 8 1 2 5 6 ID A0 Location GSWB Power GSWB Alarm LAN 1Gb 8 ...

Page 83: ...Figure 2 13 1 Gb LAN GSWB connections 1000BASE T RJ45 x8 1000BASE T RJ45 x8 Crossbar MMB 0 MMB 1 IOU IOU 0 1 IOU 2 IOU 3 IOU 4 IOU 5 IOU 6 IOU 7 SB 1 SB 0 SB 2 SB 3 SB 4 SB 5 SB 6 SB 7 GSWB 0 GSWB 1 Switching hub function Switching hub function BP BP BMM 0 BMM 1 Private LAN Manage ment LAN GbE PXH 0 SCSI PXH 1 GbE SCSI FLP 0 FLP 1 FLI Private LAN Manage ment LAN ...

Page 84: ...d selects one of them to provide external output The interface that is output from the KVM is specified with the Web UI of the MMB For information on KVM operation with the Web UI of the MMB see Section 4 2 5 KVM USB connection switching Figure 2 14 KVM Table 2 15 KVM external interfaces Interface Count Remarks USB 4 Uses USB 1 1 VGA 1 D sub 15 pin DVD ROM drive 1 For software installation KVM Pow...

Page 85: ...cuits are mounted on the CPCB If one clock generator circuit fails MMB firmware control automatically switches operation to the other one However the whole system must be rebooted for this switching operation z The CPCB contains eight interface connectors D SUB 9 pin small connector for PCI_Box control z The CPCB contains two UPC interface connectors D SUB 9 pin for un interruptable power supply U...

Page 86: ...dware Configuration Table 2 16 CPCB external interfaces Interface Count Remarks UPC interface 2 Connected to an external uninterruptible power supply UPS It can be used for the dual power feed option PCI_Box interface 8 Connected to a PCI_Box ...

Page 87: ...under operation Location LED z Displaying the MMB status for MMB status checks from outside the main unit MMB Ready LED Figure 2 16 OPL Table 2 17 LED display on the OPL LED Color LED status Unit status Power LED Green Off Power to unit is off On Power to unit is on Alarm LED Orange Off Normal status On Unit failure Location LED Blue Off Unit cannot be recognized On Unit is recognized Blinking Uni...

Page 88: ...e power supply units PSUs can be connected to a PDB z Two PSUs can be used with one AC cable z The primary power feed option consists of six PSUs Up to 12 PSUs can be connected to the option z The AC section supports dual power feed as an option Table 2 18 lists ACS specifications Table 2 18 ACS specifications Item Specifications Input Voltage Rated voltage 200 to 240 VAC Number of phases Single p...

Page 89: ...y power feed as standard z The PSU supports dual power feed as an option z For PRIMEQUEST 480 the number of mounted PSUs depends on the number of mounted SBs and IO_Units Figure 2 17 PSU Table 2 19 PSU specifications Item Specifications Output Rated voltage 48 VDC Rated power 1450 W Size 69 5 mm 123 5 mm 344 3 mm Weight 3 3 kg ...

Page 90: ...or fewer Primary power feed 6 5 1 redundancy Five SBs or IO_Units or more 10 9 1 redundancy Four SBs and IO_Units or fewer Dual power feed 12 5 1 5 1 redundancy six per power feed An expansion cabinet is required Five SBs or IO_Units or more 20 9 1 9 1 redundancy 10 per power feed An expansion cabinet is required Power feed Number of mounted PSUs PSU configuration Remarks Primary power feed 6 5 1 ...

Page 91: ...hole This unit has the following two types of fans Fan A front FANA 0 to FANA 5 Fan B rear FANB 0 to FANB 1 Figure 2 18 Fan types z The fans have redundant configuration as standard Periodic replacement is not required z Hot maintenance of fans is possible in units of trays Fan_A front Fan_B rear FAN Alarm ...

Page 92: ...I_Box is an optional unit for PCI slot expansion and it is mounted in an extended I O cabinet z Up to eight PCI_Boxes can be mounted in one PRIMEQUEST 480 main unit and up to four PCI_Boxes can be mounted in one PRIMEQUEST 440 main unit z Four PCI Slot Units PCIUs can be mounted in one PCI_Box Three PCI X cards can be inserted in one PCIU z The PCI_Box is connected to the CPCB by a CPCB control ca...

Page 93: ...C122 B001 02EN 2 35 2 14 PCI_Box Figure 2 19 PCI_Box Location PCI_Box Power PCI_Box Alarm IO PSU 0 Alarm IO PSU 1 Alarm FAN Alarm CTL interface CH interface Rear ...

Page 94: ...slot 1 3 PCIU 3 PCIU 96 48 FA N PCI X 133MHz 64bit PCI X 100MHz 64bit x 2 slots PCI_Box 0 PCI_Box 1 PCI_Box 2 PCI slot Unit PCI slot Unit PCI slot Unit PCI slot Unit PCI slot Unit IOB x1 IOB x1 IOB x1 IOB x1 IOB x1 PCI Express 4Lane PCIU control PCI Express 4Lane PCIU control PCI Box control FAN PCI card Full PCI card Full PCI card Full Crossbar SB 0 SB SB SB SB SB SB SB 1 2 3 4 5 6 7 CPCB IOU IOU...

Page 95: ...igure 2 21 PCIU with PCI card cassettes inserted 2 14 2 PCI slot The following table lists the types of PCI slots in the PRIMEQUEST series machines Only the standard type full height PCI card bracket is supported Table 2 23 PCI slot specifications PCI_Box PCI slot No Operating clock Operating voltage Bus width Size PCI slot 1 100 MHz 3 3 V 64 bit Long length PCI slot 2 100 MHz 3 3 V 64 bit Long le...

Page 96: ...d A PCI card is mounted in a PCI card cassette which is then mounted in a PCI_Box Remarks This PCI cassette is different from the PCI cassette used in the IO_Unit Figure 2 22 PCI card cassette PCI_Box 2 14 4 Compatible PCI cards The PCI cards compatible with the PRIMEQUEST PCI_Box are the same as the PCI cards compatible with IO_Units For details see Section 2 3 4 Compatible PCI cards ...

Page 97: ...mounted corresponding to dual power feed option In primary power feed a UPS is connected to the CPCB UPC0 connector of the main unit In dual power feed one UPS is connected to CPCB UPC0 connector of the main unit and another UPS is connected to CPCB UPC1 connector of the main unit The UPS connected to the UPC0 connector must be connected with an AC cable to the ACS of the PRIMEQUEST main unit The ...

Page 98: ...in unit Expansion power cabinet ACS AC cable UPS interface cable UPS interface cable AC cable ACS MMB UPS UPS CPCB 0 1 UPS Unit Power failure notification UPS hardware failure Notification of end of battery discharge Unit UPS Power on off status UPC0 UPC1 UPS interface card UPS interface card ...

Page 99: ...agement operations including operations related to PSA installed in a partition can be performed from a Web browser running on a general purpose PC through linkage with a Web server using the MMB firmware PRIMEQUEST supports large scale enterprise environments that require high reliability by ensuring close linkage between hardware and software Figure 3 1 PRIMEQUEST software configuration PRIMEQUE...

Page 100: ...ction Layer SAL SAL is firmware that abstracts the platform and provides the OS with the specified interface The SAL functions include diagnosis and initialization of the PRIMEQUEST platform including ASIC and memory configuration control 3 1 3 Extensible Firmware Interface EFI EFI is PRIMEQUEST boot firmware EFI provides a boot environment that is not dependent on any platform for an OS such as L...

Page 101: ...SWB firmware GSWB firmware is incorporated in the GSWB and it controls a LAN switching hub GSWB GSWB firmware is operated via MMB firmware GSWB firmware is used to construct and manage network systems using user LANs business LANs For details on the GSWB see CHAPTER 5 Business LAN Management by GSWB 3 2 OS Linux or Windows PRIMEQUEST supports the following OSs The OSs are installed and used in ind...

Page 102: ...ncreased number of connected devices Large scale storage systems can be managed Support of large capacity storage Systems that handle large volume data can be managed 2 Higher reliability and availability Linux works together with hardware to increase reliability and availability as follows Enhancing drivers Error handling is enhanced thereby increasing reliability Machine Check Abort MCA informat...

Page 103: ...ailure monitoring configuration management and other tasks in a partition For details on the PSA see CHAPTER 4 Hardware System Management SIRMS When REMCS is used for operation SIRMS collects software configuration information on each partition and troubleshooting materials in case of a software failure 3 3 2 Drivers SCSI related drivers LAN related drivers and other types of drivers are supplied ...

Page 104: ...ms as necessary Cluster operations can be performed between cabinets and between partitions in a cabinet in the PRIMEQUEST system For details on cluster operations see CHAPTER 9 Clustering For details on PRIMECLUSTER see the PRIMECLUSTER manuals 3 4 2 PRIMECLUSTER GDS PRIMECLUSTER GDS is volume management software that helps increase the availability of data stored in disk drive units and facilita...

Page 105: ...nt of multiple PRIMEQUEST servers For details on Server System Manager see the Server System Manager manuals 3 4 5 Systemwalker Systemwalker is integrated operation management software that manages systems networks resources etc to support Internet based businesses For details on Systemwalker see the Systemwalker manuals 3 4 6 Backup and restoration software The following backup and restoration so...

Page 106: ......

Page 107: ...ardware system Business LAN PRIMEQUEST Sensors Management LAN Console Business LAN Management LAN KVM KVM PSA MMB FW SB Linux windows HDD Memory Partition 0 Partition n BMC_FW IOU CPU Memory Dedicated CPU Dedicated CPU SB IOU Fan Power supply Voltage Temperature Others Web UI PSA Log message Log message Log message Business application Higher system management software MMB Setting display Various ...

Page 108: ...d in the partition such as the CPUs and memory of the SB the HDDs and PCI slots of the IO_Unit and if the PCI_Box is connected to the IO_Unit the PCI card to which the PCI_Box is connected The PSA of each partition operating independently of the PSAs in the other partitions links with the MMB The PSA is operated from the MMB Web UI The GSWB is a switching hub unit that manages the business LAN The...

Page 109: ...rpose PC connected to the management LAN Accordingly functions used by the system administrator or user during operation through the MMB and MMB Web UI are operated through the Web UI However special functions used by service engineers are operated in command line interface CLI operation using commands The main functions of the MMB are described in Section 4 2 MMB Functions The main functions of t...

Page 110: ...rtition and the MMB to a LAN outside the cabinet The MMB operates and controls each partition through the management LAN Each partition sends its log information and hardware configuration information to the MMB through the management LAN Applications of the management LAN are described below z Communication between the MMB and each partition Sends and receives information about Web linkage SNMP l...

Page 111: ... as connecting an external management server and using the operation console Fujitsu certified services engineer s port For connection to a maintenance terminal REMCS port For connection to the REMCS Center The ports for REMCS connections can be selected based on the customer s requirements as follows P P connection Connect a dial up router directly to the REMCS port Si R30 130 dial up router 10BA...

Page 112: ... between ports of all partitions However communication between the REMCS Fujitsu certified service engineer s port and the user port and communication between the REMCS Fujitsu certified service engineer s port and partition ports are not supported Y in Table 4 1 indicates that communication between the corresponding ports is available Table 4 1 Inter partition communication connection mode No VLA...

Page 113: ...hat communication between the corresponding ports is available Table 4 3 All partition communication disconnection mode Port disable mode VLAN mode User port Fujitsu certified service engineer s port REMCS port MMB Partition ports User port Y N N Y Y Fujitsu certified service engineer s port N Y N Y N REMCS port N N Y Y N MMB Y Y Y Y Y Partition ports Y N N Y N Port disable mode User port Fujitsu ...

Page 114: ...ation among the BMC firmware mounted on an IO_Unit the MMB firmware and the GSWB firmware The private LAN cannot be used from the OS or an applications 100BASE T RJ45 100BASE T RJ45 MMB 0 MMB 1 Hub for private LAN Hub for private LAN Hubfor managementLAN HubformanagementLAN Crossbar IOU IOU 0 1 IOU 2 IOU 3 IOU 4 IOU 5 IOU 6 IOU 7 SB 1 SB 0 SB 2 SB 3 SB 4 SB 5 SB 6 SB 7 BP BP BP BMM 0 BMM 1 GbE PXH...

Page 115: ...ports layer 2 switching hub functions including the tag VLAN function and port VLAN function In Figure 4 4 four 1 Gigabit LAN ports are provided for each IO_Unit and two ports are connected to each GSWB The two GSWBs are connected to all IO_Units in a dual configuration One GSWB has eight 1 Gigabit LAN ports for LAN connection with external devices outside the cabinet The GSWB specifications and g...

Page 116: ... GSWB 0 and GSWB 1 if they are accessed through the management LAN The IP addresses are accessed through the GSWB command line interface CLI operation or by using SNMP from the management server If an IP address is assigned for the management LAN business LAN port access needs to be separated through the VLAN setting of a GSWB BMM 1 BMM 1 IOU 0 BMM 0 BMM 0 IOU 7 MMB 1 REMCS Center Dial up router o...

Page 117: ...set by the OS on a partition Each of IO_Units from IOU 0 to IOU 7 has one IP address The IP addresses are used for the control of Web UI linkage SNMP linkage and REMCS linkage performed between PSA and the MMB and for linking the PRIMECLUSTER and the MMB Two network interfaces are provided on the BMM of each IO_Unit They form a redundant configuration of the interface by using the Linux Bonding dr...

Page 118: ...nctions The MMB functions are indicated by MMB in Figure 4 6 in which all of the PRIMEQUEST functions are shown This section describes the MMB functions The MMB functions include the switching function for using the PSA and GSWB functions Figure 4 6 Main MMB functions ...

Page 119: ... only to reference the system state The user cannot set system configuration information and cannot turn on and off power supplies of partitions CE A user having this privilege is allowed to reference the system state and to perform maintenance The user cannot manage users and change network settings Operator A user having this privilege is allowed to reference and set a system state The user cann...

Page 120: ...iguration information For example whether the configuration elements are in the normal or abnormal state is displayed In some cases where these elements are in the abnormal state the system administrator or person in charge of maintenance needs take corrective action Detailed information can be displayed by tracing the link in the display item field Also the configuration can be displayed in units...

Page 121: ...connection are collected and error messages are issued Failures and errors are displayed by LEDs according to their severity In addition different types of reports are supported as follows reports to applications for operation management following SNMP trap configuration mail reports to the system administrator following alarm e mail configuration and reports to the REMCS Center following REMCS re...

Page 122: ...4 16 C122 B001 02EN CHAPTER 4 Hardware System Management Figure 4 9 Example of the filter condition setting screen ...

Page 123: ...ying a combination of SBs and IO_Units The partition configuration information can be displayed in detail showing incorporated SBs and IO_Units and their configuration elements As with the settings the partition can be changed by changing the combination of the SBs and IO_Units An example of the display of the partition configuration screen is shown in Figure 4 11 PRIMEQUEST A mixture of different...

Page 124: ...for connecting the keyboard mouse display and DVD drive As shown in Figure 4 12 arbitrary partitions comprise the connection destinations The connection destination can be set from the MMB Web UI An example of the switching setting screen of the KVM switch of the MMB Web UI is shown in Figure 4 13 This function enables the keyboard or mouse USB video output and DVD ROM drive USB to be connected to...

Page 125: ...4 13 Example of the KVM USB individual connection switching setting screen MMB MMB KVM KVM Partition 0 Partition n PRIMEQUEST Web UI Partition 1 IOU IOU IOU IOU IOU Management LAN n 0 to 15 A partition to which the KVM is connected is switched Connection switching instruction SB SB SB SB ...

Page 126: ...he IO_Unit incorporated in each partition 2 Using the management LAN through the MMB Figure 4 14 shows the flow of console information sent when the information is redirected to the management LAN Figure 4 14 Flow of redirected information When the information is redirected to the management LAN a terminal connected to the management LAN can be used as a console MMB HUB Partition 0 SB IOU BMM COM ...

Page 127: ...C122 B001 02EN 4 21 4 2 MMB Functions Figure 4 15 Console Redirection Switch screen ...

Page 128: ...s how system accounts are managed Figure 4 16 System account management method z User management For information on the user privilege for accessing the MMB see Section 4 2 1 User privilege management z Making and changing the network environment settings In this kind of operation the MMB usage environments for the HTTP and Telnet is set The network environments of a business LAN is set up and cha...

Page 129: ...ting access to the MMB is set up Usable IP addresses are specified for each protocol Access to the MMB is allowed only from the set IP addresses Figure 4 18 Example of the access control settings screen z Secure Sockets Layer SSL support Web and Telnet access is ciphered in the SSL Secret keys and electronic certificates are generated in this layer ...

Page 130: ... time is possible by accessing the external NTP server Figure 4 19 Time synchronization image NTP client NTP server MMB MMB NTP client External NTP server Partition 0 Partition 1 Partition n NTP client NTP client NTP client Linux Windows Linux Windows Linux Windows SB IOU SB IOU SB IOU SB IOU SB IOU SB IOU PRIMEQUEST Time synchronization is possible by using the NTP GSWB GSWB External server Exter...

Page 131: ...f the display for scheduled operation Figure 4 20 Example of the scheduled operation display screen 4 2 10 Firmware maintenance This function updates various types of firmware The update is performed by Fujitsu certified service engineer The following firmware components are subject to update MMB firmware installed in the MMB GSWB firmware installed in the GSWB BMC firmware installed in each parti...

Page 132: ...f each partition This function enables the following operations 1 After PAL SAL EFI settings are made from the PAL SAL EFI SETUP screen in one partition the PAL SAL EFI setting information for that partition can be saved and applied to another partition from the MMB 2 If a failure occurs in an SB and the SB is swapped the saved PAL SAL EFI information enables restoration of the original state Save...

Page 133: ...U MMB firmware MMB configuration information GSWB firmware GSWBconfiguration information Configuration and setting information Web UI MMB configuration information Configuration and setting information can be saved and restored MMB GSWB PAL SAL EFI setting information Linux Windows PSA SB IOU MMB firmware MMB configuration information GSWB firmware GSWBconfiguration information Configuration and s...

Page 134: ...e performed Partition monitoring by using the watchdog function cannot be performed The following error notification through failure prediction and trap notification to the administrator cannot be performed Correctable error count of a CPU DIMM or chip set that exceeds the threshold Exceeded threshold in the S M A R T Self Monitoring Analysis and Reporting Technology system monitoring of an HDD Th...

Page 135: ... the Common Gateway Interface CGI WebGateCGI which is used for communication with the Web server function The PSA in each partition has the operation management GUI functions comprising the WebGate and HTML template group When receiving a request from a user the WebGateCGI communicates with the WebGate by TCP IP and delivers a corresponding HTML The WebGate acquires information from the data sourc...

Page 136: ...TE operation command used by a Fujitsu SE when hot swapping disks PSA start stop command PSA survey data collection command Filter definition update command Local partition number acquisition command Serial number acquisition command MMB Partition n PSA Linux Windows Webgate CGI WWW Server MMB firmware Management LAN Configuration information log information etc PRIMEQUEST User System administrato...

Page 137: ...atus information PCI configuration PCI card mounting PCI device mounting detailed information such as the PCI device type error status information SCSI FC connection unit configuration HDD tape drive etc Network configuration network interfaces error status information 4 3 3 OS information display This function displays information on the OS installed in a partition This function displays the foll...

Page 138: ...he OS and hardware detect in the CPU memory chip sets and inter chip busses The PSA performs an error analysis to identify a component or unit that had more correctable errors than defined in the threshold for a given period The PSA then records this information as log information and reports the error information to the MMB and upper level system management software The PSA notifies the MMB for t...

Page 139: ...onent need not be disconnected immediately However if a CE occurs frequently the component may be degraded making it likely that a fatal failure will occur in the future PRIMEQUEST Notification Degradation reservation PAL SAL EFI PAL SAL EFI MMB firmware SB IOU MMB Linux Windows Interrupt Notification Recording PSA SYSLOG AgentLOG Monitoring Recording Partition n Statistical information Reference ...

Page 140: ...og file information Log file type Description Agent message log Information related to the events excluding event IDs 00000 to 09999 detected in the PSA for which the PSA took action recording to an OS log SNMP trap etc is stored in the agent log This log can be displayed in a format supported by the GUI and can be downloaded in the form of a CSV format file Error record log Information on firmwar...

Page 141: ...ion and errors of the partition to the REMCS Center remote maintenance center REMCS Agent reports PRIMEQUEST system errors and log information to the REMCS Center via the Internet or a P to P connection PRIMEQUEST REMCS Agent consists of the MMB firmware and the PSA and SIRMS installed in each partition As shown in Figure 4 25 the MMB firmware monitors for errors in the entire system and when it d...

Page 142: ...sable such as because of a fault the operation on the active system can be taken over by the standby system The PRIMECLUSTER linkage function is installed in MMB firmware and PSA to support the following functions System status monitor display Monitors and displays the status of the specified node partition System status notification Notifies the associated nodes which together with the local node...

Page 143: ...PRIMEQUEST For REMCS System administrator Management LAN Console Interconnect Business LAN Active system Standby system Switching Partition i Partition j SB 0 SB 1 SB 2 SB 3 IOU 0 IOU 1 IOU 2 IOU 3 PRIMECLUSTER Failure PRIMECLUSTER Taking over of control information Linux Windows Linux Windows GSWB MMB Partition i Partition j SB 0 SB 1 SB 2 SB 3 IOU 0 IOU 1 IOU 2 IOU 3 PRIMECLUSTER Failure PRIMECL...

Page 144: ...with operation management software such as Systemwalker It uses SNMP Simple Network Management Protocol for the linkage Figure 4 27 Operation management software linkage For the REMCS System administrator Management LAN PRIMEQUEST SNMP Operation management software console Reference PSA Linux Windows MMB FW SNMPD SNMPD Log information Operation management software MIB MIB PSA Linux Windows MMB FW ...

Page 145: ...5 1 List of provided main functions Item Description Remarks Interface Front 1000 BASE T 8 ports Auto negotiation Only 1000 BASE T is supported Full duplex half duplex 10 100 1000 Mbps LAN protocol IPv4 IPv6 is not supported LAN swtiching mode Store and forward Buffer size 1 MB DEV Forwarding rate 10 100 1000 32 Mpps Flow control PAUSE frame Full duplex Back pressure Half duplex Number of address ...

Page 146: ...P snooping IGMPv1 RFC1112 IGMPv2 RFC2236 Supported by software control Jumbo frame Up to 9020 byte 9 K Rate control function Broadcast multicast DLF storm control Network management function SNMP v1 v3 RMON Port mirroring Operation management function Local console IPMI telnet ssh ping traceroute tftp multiple access rights connection timeout Log function Elog llog mlog tlog Private LAN function B...

Page 147: ... Unit MAC address 1 Unit IP address 1 Number of L2 address entries 16 K Flow control Y Auto sense auto negotiation Y Port ranking Y Port mirror Y VLAN function Y IGMP snooping Y CoS function Y Rate control function Y STP Y Jumbo frame Y Connection timeout Y SNMP MIB RMON Y Telnet SSH Y BOOTP DHCP client Y TFTP client Y IP TCP UDP ARP ICMP Y Diagnosis command ping traceroute and others Y IPMI N NTP...

Page 148: ... LAN Management by GSWB 5 2 Main GSWB Functions The main GSWB functions are shown in the GSWB box included in the main PRIMEQUEST functions shown in Figure 5 1 This section describes the GSWB functions Figure 5 1 Main PRIMEQUEST functions ...

Page 149: ...omplies with IEEE802 3ab 1000BASE T Complies with IEEE802 3x 10 100 BASE TX Twisted pair cable category 5 UTP category 5 enhanced 1000BASE T only Connector type RJ 45 Transmission distance Up to 100 m Transmission speed 1 Gbps 1000BASE T 100 Mbps 100BASE TX 10 Mbps 10BASE T Business LAN SerDes BackPlane side Complies with IEEE802 3ab Connector type HS 3 connector Transmission distance Transmission...

Page 150: ...munication mode The communication modes that the communication interfaces of this unit support are listed below Table 5 5 Supported communication modes LAN type Location Interface Jumbo frame setting Maximum frame length 1 Without VLAN Tag With VLAN Tag Business LAN Front side 1000BASE T Invalid 1516 bytes 1520 bytes Valid 9016 bytes 9020 bytes BackPlane side SerDes Invalid 1516 bytes 1520 bytes V...

Page 151: ... other the common mode is determined in accordance with the priority stipulated in IEEE802 3u Interface support The table below lists whether interfaces have the auto negotiation function Table 5 6 Interface support for auto negotiation Y Supported N Unsupported Setting priority The table below lists the setting priority for auto negotiation Table 5 7 Auto negotiation priority LAN type Location In...

Page 152: ...t the occurrence of data overflow This type of traffic control mechanism is referred to as flow control Flow control varies depending on the communication mode of a port If a port uses half duplex mode a receiving unit sends a collision back pressure to cause the sending unit to wait enter standby mode before transmitting again If a port uses the full duplex mode a receiving unit sends a PAUSE fra...

Page 153: ...ified for full duplex communication and supports the following three modes Two way control Symmetric One way control Asymmetric accept One way control Asymmetric transmit Restrictions The restriction list for communication speeds and communication modes is shown below Table 5 9 Flow control and restriction list Y Supported N Unsupported Remark Both functions cannot be executed at the same time Com...

Page 154: ...s transferred Interface support The table below lists whether interfaces have the jumbo frame function Table 5 10 Interface support for jumbo frames Y Supported Supported functions z A jumbo frame expansion frame supports a maximum transmission unit MTU size of up to 9 KB z The allowable range is between 1514 and 9020 bytes z The function can be set to enabled or disabled only for the entire unit ...

Page 155: ... for broadcast multicast and Destination Lookup Failure DLF it is possible to discard frames greater than or equal to a threshold value on a PPS basis for each port Interface support The following table lists whether interfaces have rate control Table 5 11 Interface support for rate control Y Supported N Unsupported LAN type Location Interface Rate control function Business LAN Front 1000 BASE T Y...

Page 156: ... use of the learning table The user can set the aging time Figure 5 2 shows why this function is effective Figure 5 2 Address learning and age out a b c d e a b d e c MAC address Port number Addresslearning Addresslearning Partition n Partition n PRIMEQUEST Adding Self learning with communication on a LAN 1 2 5 If however there is no communication for a certain amount of time information is delete...

Page 157: ...nd switches to a new path automatically whenever one part of a path is blocked due to a failure is also provided Figure 5 3 shows the difference in the system depending on whether this function is enabled or disabled Figure 5 3 Spanning tree function SW SW SW SW Partition n PRIMEQUEST GSWB Partition n GSWB When the spanning tree function is disabled When the spanning tree function is enabled This ...

Page 158: ...dentification methods referred to as the port method and the tag method The port method identifies groups on a per port basis The tag method identifies which VLAN a frame belongs to by adding additional information referred to as a tag to the frames z Port method Figure 5 4 Port method GSWB Partition 0 Partition 1 Partition 2 Partition 3 Web server DB server Server for development Internet Group 1...

Page 159: ...ed depends on the port setting statuses and frame types The table below lists how transmission and reception are handled depending on the port setting statuses and frame types Suitable attention is required when setting up a VLAN Web server DB server Server for development GSWB GSWB Partition 0 Partition 1 Partition 2 Partition 3 Internet SW PRIMEQUEST Firewall To identify packets additional infor...

Page 160: ...VLAN is specified for the interface a frame without a VLAN tag as indicated by PVID can be transmitted Port status Frame type Port VLAN Tag VLAN PVID at tag VLAN UP DOWN UP DOWN UP DOWN With a receive VLAN tag The same VLAN ID as port specification Unavailable discard Available Available VLAN ID different from port specification Unavailable discard Unavailable discard Unavailable discard Without a...

Page 161: ...es with the default priority of a receive port The priority can be changed Figure 5 6 Priority control function Web server DB server Server for development PRIMEQUEST GSWB Partition 0 Partition 1 Partition 2 Partition 3 Application 1 SW Application 2 Application 3 Application 4 B4 B1 B2 B3 B4 B1 B2 B3 Breakdown example of packets that transit through B Priority High Low Priority Output Input Packe...

Page 162: ...ges One advantage is that distributing the load among the ports grouped enables the use of a bandwidth comprising the total of the bandwidth of all individual ports The other is that if one of the ports grouped fails this function detects the failure and uses the remaining ports to continue communication Figure 5 7 Port ranking function Web server Application server DB server Server for developmen...

Page 163: ...required switch port As shown in Figure 5 8 a switch that does not support IGMP snooping broadcasts multicast stream to all ports For a switch that supports IGMP snooping destinations to which specific multicast data is relayed can be restricted to a specific switch port The GSWB can enable or disable this function IGMP snooping prevents multicast data from being broadcast thereby optimizing the n...

Page 164: ...tions z Multiple mirror ports cannot be created at the same time Multiple ports to be monitored can be created at the same time however z A port set as the mirror port does not work as a normal switch port Figure 5 9 Port mirroring Web server Application server DB server Server for development PRIMEQUEST Partition 0 Partition 1 Partition 2 Partition 3 Application A Application B Application C Appl...

Page 165: ... is displayed z mlog Message log Messages that programs output as needed is archived as a log The log is displayed 5 2 10 SNMP Simple Network Management Protocol SNMP supports the agent functions in v1 v2 and v3 SNMP can implement read Get and write Set targeting agent unit information or MIB information in response to an inquiry from an SNMP manager belonging to a community to which access right ...

Page 166: ...n The client function for transmitting files from remote units are supported for maintenance management purposes z Network Time Protocol NTP functions The client functions for setting a standard time are supported z Lightweight Directory Access Protocol LDAP functions The client functions for managing passwords for login telnet ssh and ftp in a centralized manner are provided z Firmware update fun...

Page 167: ...on information when ports are moved such as because of a partition reconfiguration Upon receipt of polling data from the GSWB the function also automatically finds the relevant port to implement communication Age out of learning table Symptom When a bridge relay is used age out of the learning table is performed at a different time from the setting Cause A gap between the MAC address learning timi...

Page 168: ...toring MAC address learning w K t O t t O t Monitoring Deletion Flag assignment The monitoring interval is the specified aging time α occurs if there is even a slight gap between the monitoring timing and learning timing α increases if the monitoring interval specified aging time is longer The maximum age out time is specified aging time x 2 Learning Specified aging time Monitoring ...

Page 169: ...ip Remarks When a mismatch occurs the CPU makes each chip relearn the contents of the relevant MAC address and therefore a frame transfer to a port other than those existing in the learning table is a temporary symptom Item Deletion conditions Deletion of an entry Age out of aging timer When a frame which was sent from the relevant MAC address has been received from a port other than an already le...

Page 170: ...ilter is set to Enable The BPDU frame is processed or abandoned in other cases Table 5 14 Handling of BPDU and PAUSE frames according to the STP setting conditions Addition of a VLAN Tag to BPDU When STP is set to Enable a VLAN Tag is not attached to the BPDU that is output When BPDU is relayed it is relayed in the original form When a VLAN Tag is attached to the original form it is relayed with a...

Page 171: ...VLAN Disable OFF Without a VLAN Tag Port VLAN Without a VLAN Tag With a VLAN Tag Port VLAN With a VLAN Tag The Tag is the same as that of the source BPDU ON Mirror port destination Monitored port source IOU 0 0 3 1 GigabitEthenet0 3 0 4 GigabitEthenet 0 7 0 8 IOU 4 0 7 1 GigabitEthenet 0 1 0 2 GigabitEthenet 0 5 0 6 TenGigabitEth enet 1 1 TenGigabitEth enet 1 2 IOU 0 0 3 1 GigabitEthenet 0 3 0 4 G...

Page 172: ...t A to port B The following explains how the frame tag is handled when a frame is transferred from port A to port B Table 5 17 VLAN Tag of mirroring frame transfer from port A to port B 1 A frame without a tag is sent from the monitored port port B but a frame with a tag is mirrored to the mirror port port C Transfer from port B to port A The following explains how the frame tag is handled when a ...

Page 173: ...when the fixed setting is used Port A Port B Port C Receiving a frame without a tag Sending a frame without a tag Frame without a tag Receiving a frame with a VLAN 10 tag Following the setting of port A Frame with a VLAN 10 tag Receiving a non affiliated VLAN frame No transmission Non affiliated VLAN frame 2 Speed Duplex setting Setting of opposed port auto 1G 100Mbps FULL 100Mbps HALF 10Mbps FULL...

Page 174: ...epending on the result of the auto negotiation Table 5 20 Flow control setting Notes on using flow control Note that enabling the flow control may affect the transfer of other ports because the flow control is also performed between GSWB internal chips In the sample configuration shown below the transfer from IOU 0 0 to GigabitEthernet 0 6 enables flow control and then a PAUSE frame is sent from B...

Page 175: ...e LinkAggregation Load distribution method Source MAC address No Load distribution method Key 1 Source MAC address 3 low order bits of source MAC address 2 Destination MAC address 3 low order bits of destination MAC address 3 Combination of source and destination MAC addresses 3 low order bits of exclusive OR of source and destination MAC addresses 4 Source IP address 3 low order bits of source IP...

Page 176: ...fault and connected to the 1G bit ports the linkup timing of all ports at startup may vary by about five seconds Source MAC address of send frame Key Modulo result Destination port 00 0B 5D 70 80 01 1 1 2 00 0B 5D 70 80 02 2 2 3 00 0B 5D 70 80 03 3 0 1 00 0B 5D 70 80 04 4 1 2 00 0B 5D 70 80 05 5 2 3 00 0B 5D 70 80 06 6 3 1 00 0B 5D 70 80 07 7 1 2 00 0B 5D 70 80 08 0 0 1 00 0B 5D 70 80 09 1 1 2 00 ...

Page 177: ...s divided STP determines a loop and blocks one port Figure 5 13 STP setting for VLAN A loop configuration is avoided by separating the A and B pair of ports from the C and D pair of ports using divided VLANs However STP detects a loop and blocks port C and accordingly the communication between ports C and D is disabled GSWB SW VLAN2 VLAN3 PortA Port B PortC Port D ...

Page 178: ...5 34 C122 B001 02EN CHAPTER 5 Business LAN Management by GSWB ...

Page 179: ...ost elements of the hardware configuration such as memory CPUs IO_Units chip sets transmission lines and power units to achieve high reliability and high availability This chapter describes the following types of redundant configuration adopted by PRIMEQUEST to improve reliability and availability Redundancy of Components Redundancy of Private LAN and Management LAN System Mirror Redundancy of an ...

Page 180: ... can be mounted Redundant configuration can be adopted depending on the operation methods For details see CHAPTER 8 Partition and CHAPTER 9 Clustering An SB can have one to four IPF CPUs and up to 8 set 32 memory boards DIMMs Redundant configuration is possible Redundant configuration is possible depending on operation PRIMEQUEST PCI_Box PCIU CPCB SB XDI XAI GSWB MMB OPL FAN TRAY KVM IOU BP CPU FA...

Page 181: ... Board MMB Two MMBs are mounted in the basic configuration One of them is in the active state while the other is in the standby state IO_Unit One to up to eight IO_Units can be mounted Redundant configuration can be adopted depending on the operation methods For details see 6 4 Redundancy of an HDD Use of PRIMECLUSTER GDS CHAPTER 8 Partition and CHAPTER 9 Clustering An IO_Unit has four HDDs and fo...

Page 182: ...ST 480 cabinet Figure 6 2 Redundant configuration of management LANs and private LANs A CPU in an MMB and a Baseboard Management Controller BMC mounted on a BMM in an IO_Unit manage partitions in a cabinet A BMC performs management in partitions An MMB manages functions common across the cabinet and partitions 100BASE T RJ45 100BASE T RJ45 MMB 0 MMB 1 CPU CPU Hub for a private LAN Hub for a privat...

Page 183: ...tem management tasks such as connection to a console unit such as a general purpose PC and node to node connection in a cluster configuration In the same way as for a private LAN a hub incorporated in an MMB connects the components in a cabinet CPUs of an MMB are also connected to an management LAN An management LAN has three external interface ports Two ports are allocated for maintenance tasks p...

Page 184: ...MB 0 to MMB 1 Figure 6 3 Example of automatic switching between MMB and management LAN at the occurrence of an MMB failure MMB 0 CPU HUB NIC MMB 0 CPU HUB NIC IOU BMM NIC NIC BMC SB CP U SB CP U IOU BMM NIC NIC BMC SB CP U SB CP U te MMB 0 CPU HUB NIC MMB 1 CPU HUB NIC IOU BMM NIC NIC BMC SB CP U SB CP U IOU BMM NIC NIC BMC SB CPU SB CPU IOU BMM NIC NIC BMC SB CP U SB CP U IOU BMM NIC NIC BMC SB C...

Page 185: ...for the amount of installed memory Figure 6 4 outlines the system configuration of System Mirror System Mirror duplicates important hardware in the system i e the Xbar address interconnect board XAI on which an address cross bar is mounted and Xbar data interconnect board XDI on which a data cross bar is mounted memory and the inside of the chip set Data is also simultaneously processed by using b...

Page 186: ...ng of a crossbar SB 0 SB SB MMB 0 1 n MMB 1 IOU 1 IOU n LDX LDX LDX LDX FLN Memory Memory IOU 0 Manage ment LAN Manage ment LAN GbE SCSI BMM 0 BMM 0 BMM 1 FLP 0 FLP 1 FLI PXH 0 PXH 2 PXH 3 PXH 1 HDDx2 HDDx2 PCI card PCI card PCI card PCI card GbE SCSI Private LAN Private LAN Mirroring of memory Mirroring of a chip set ...

Page 187: ...stalled However data read and write operation is duplicated and performed simultaneously Thus even in the case where a part of memory fails the operation continues using the rest of the memory Reliability increases with the mirroring of memory Figure 6 5 shows the logical numbers of memory DIMMs that are mirrored Figure 6 5 Logical numbers of memory DIMM for which a redundant configuration has bee...

Page 188: ...independent from without software Therefore to configure redundancy of the I O function combine multiple I O devices for the intended purpose and install the software required to implement the redundancy function Optional software is required for redundancy Only Linux supports redundancy using optional software Windows supports mirroring using external RAID Figure 6 6 I O Configuration XAI XAI XDI...

Page 189: ...D or an ISV product Minimum configuration for HDD redundancy Figure 6 7 shows the minimum configuration for HDD redundancy In this configuration a redundant configuration of HDDs is implemented by connecting two HDDs to one SCSI bus Redundancy however is not applied to SCSI controllers and other I O devices other than HDDs Figure 6 7 Minimum redundancy configuration XAI XAI XDI XDI XDI XDI Crossba...

Page 190: ... HDD SCSI controller PXHs are mirrored I O paths including FLP and FLI are not mirrored Figure 6 8 Redundancy including SCSI controllers XAI XAI XDI XDI XDI XDI Crossbar SB 0 SB SB MMB 0 1 n MMB 1 LDX LDX LDX LDX FLN Memory Memory IOU 0 Manage ment LAN Manage ment LAN GbE SCSI BMM 0 BMM 0 BMM 1 FLP 0 FLP 1 FLI PXH 0 PXH 2 PXH 3 PXH 1 HDDx2 HDDx2 PCI card PCI card PCI card PCI card GbE SCSI Private...

Page 191: ...ed XAI XAI XDI XDI XDI XDI Crossbar SB 0 SB SB MMB 0 1 n MMB 1 IOU n LDX LDX LDX LDX FLN Memory Memory IOU 0 Manage ment LAN Manage ment LAN GbE SCSI BMM 0 BMM 0 BMM 1 FLP 0 FLP 1 FLI PXH 0 PXH 2 PXH 3 PXH 1 HDDx2 HDDx2 PCI card PCI card PCI card PCI card GbE SCSI Private LAN Private LAN IOU 1 Manage ment LAN Manage ment LAN GbE SCSI BMM 0 BMM 0 BMM 1 FLP 0 FLP 1 FLI PXH 0 PXH 2 PXH 3 PXH 1 HDDx2 ...

Page 192: ...ts of different IO_Units Figure 6 10 Disk device mirroring between cabinets Path redundancy To connect PRIMEQUEST series machines to ETERNUS3000 ETERNUS6000 or GR series machines over multiple paths multipaths use the ETERNUS multipath driver If one of the paths in use becomes unavailable operation is switched to another path working normally to continue operation When this happens an application ...

Page 193: ...2 B001 02EN 6 15 6 4 Redundancy of an HDD Use of PRIMECLUSTER GDS Figure 6 11 ETERNUS multipath connection FC FC FC driver GRMPD APPL Standard Driver Controller Controller ETERNUS PRIMEQUEST 0 Partition 0 ...

Page 194: ...cy for a business LAN as related to the construction of a business system To implement redundancy for of a network and the network interface at least two network interfaces and optional software to switch the interfaces are needed Linux supports PRIMECLUSTER Global Link Services GLS as optional software Windows supports duplication using a driver This section shows an example of network redundancy...

Page 195: ...hing method With this method as shown in Figure 6 13 each of Network Interface Cards NICs is connected to a different network and these NICs are activated and used at the same time Transmission packets are sent to appropriate paths according to the operational status whether operation is normal of transmission paths A virtual interface is generated to logically handle multiple NICs as a single NIC...

Page 196: ...h this method communication destinations are not restricted It is possible to communicate with hosts on the same network and on different networks Since the range of duplication is restricted to the directly connected switches and hubs to duplicate the entire communication path it is necessary to duplicate routers and other network devices located halfway as well as the transmission line of the co...

Page 197: ...ng of NICs by using the IP address set in the physical interface in the up status as the own system IP address As the connection pattern duplicated NICs should be connected on the same network It is possible to connect to a communication destination system either on the same network or on a different network via a router If each of the network devices in a multi vendor environment hub router and o...

Page 198: ...re it cannot be identified Case where the location of a failure can be identified If the location of a failure can be identified degradation is performed as appropriate for the failed location If an SB or an IO_Unit fails and reserved SB set to be switched to in case of an SB failure exists and an IO_Unit to which redundant configuration is applied exists reserve degradation and stop the partition...

Page 199: ...tire server operation The component that had been swapped can be incorporated into the running system again In this chapter hot swapping also refers to deleting or adding a component without stopping the entire server such as deliberately deleting a component from system operation adding a component for expansion purposes as well as replacing a component that was operating but stopped due to a fai...

Page 200: ...re 7 1 shows the layout of components included in PRIMEQUEST series machines Table 7 1 lists whether hot swapping and or redundant configuration is available for components Figure 7 1 Layout of included components FAN_A 1 FAN_A 0 FAN_A 3 FAN_A 2 FAN_A 5 FAN_A 4 FAN_B 1 FAN_B 0 ...

Page 201: ... Y Redundancy is ensured by PRIMECLUSTER GDS software mirror See Section 6 4 Redundancy of an HDD Use of PRIMECLUSTER GDS 7 PCI Card Y 8 BP N N 9 XAI Y Y Redundancy is ensured when System Mirror is set or not set 10 XDI Y Y Redundancy is ensured when System Mirror is set 11 MMB Y Y 12 KVM Y N 13 DVD drive Y N 14 CPCB N N 15 GSWB Y Y optional Redundancy is ensured by PRIMECLUSTER GLS software mirro...

Page 202: ...ition stops isolates the failed part and restarts automatically To maintain the failed part stop the partition and then perform hot swapping over the failed SB To add a new SB to the partition the partition must be restarted 2 CPUs Redundant configuration and hot swapping are not available If a CPU fails the partition stops isolates the failed CPU and restarts automatically Isolating the failed CP...

Page 203: ...ailed DIMM may cause no normal DIMM to be available in the SB In this case the entire SB is subjected to isolation even if a normal CPU exists If no normal SB is available in the partition the partition stops 4 IO_Units A redundant configuration is not available but hot swapping is available The partition stops if an IO_Unit fails After the partition stops the failed component can be hot swapped T...

Page 204: ...n is performed in System Mirror Two XAI systems form a redundant configuration If an XAI fails the failed XAI is automatically isolated from the system operation configuration and operation continues with the other normal XAI in degradation mode without a reboot The failed XAI thus isolated can be hot swapped To incorporate a new XAI into the system all the partitions must be restarted When operat...

Page 205: ...the XDI fails all partitions stop The failed XDI can be maintained swapped while all partitions are stopped A new XDI after swapping is automatically incorporated into the system when any partition is started 11 MMBs The MMB is usually duplicated and hot swapping is enabled If the active MMB fails it automatically switches to the other normal MMB The failed MMB can be hot swapped 12 KVM Redundant ...

Page 206: ...undant configuration is adopted and hot swapping is available However since FANs are contained in a FAN TRAY replacement of only a single FAN is not possible 20 PCI_Boxes Hot swapping is available Hot swapping is available for the entire PCI_Box A redundant configuration is also available by using mirroring software such as PRIMECLUSTER GDS 21 PCIUs Hot swapping is available A redundant configurat...

Page 207: ...C122 B001 02EN 7 9 7 1 Hot Swappable Components 23 IO_PSUs Redundant configuration is adopted and hot swapping is available ...

Page 208: ......

Page 209: ... systems resulting from such division is called a partition An operating system OS can run independently on each partition This enables running multiple servers which operate independently from one another as a single server Each partition is protected by hardware so that it is not affected by invalid accesses from another partition Therefore a failure such as an OS panic occurring in one partitio...

Page 210: ...e effective CPU and four memory modules must be mounted z An LSB having memory mapped in the area from 0 GB to 4 GB is called the home LSB To configure a partition at least one home LSB must be included z On each SB at least one CPU and one set of memory modules four modules must be mounted z CPUs on the same SB mounted on the same partition must be of the same type z CPUs mounted on the same part...

Page 211: ...led the home IO_Unit Remark The home IOU can be set on the Home of Partition screen which is displayed by clicking the Home button on the Partition Configuration screen of the MMB Web UI 8 1 2 PCI_Box allocation Each PCIU of a PCI_Box can be allocated to a partition One PCI_Box can be divided and allocated to up to four partitions Each PCIU is allocated to the partition to which the PCI_Box interf...

Page 212: ...n define the partition by using the MMB Web UI To do this first create a partition having a specific partition ID PartitionName and then register SBs and IO_Units belonging to the partition on the Partition Configuration screen of the MMB Web UI For information on the setting procedure see the PRIMEQUEST 480 440 Installation Manual C122 E001EN Figure 8 3 Partition Configuration screen ...

Page 213: ...in PRIMEQUEST series machines The flexible I O component provides a function which separates SBs and IO_Units and allows the SBs and IO_Units to be combined freely This component can connect SBs and IO_Units freely in comparison with a method that allocates SBs and IO_Units permanently Although a reserved SB does not belong to any partition it can be incorporated in a partition to replace a faulty...

Page 214: ...t becomes faulty job processing need not be stopped and can be continued Operation in System Mirror provides higher reliability For details of System Mirror see Section 6 3 System Mirror Crossbar Crossbar SB 0 SB SB SB SB 1 2 3 4 IOU IOU IOU IOU IOU 0 1 2 3 4 SB 0 SB SB SB SB 1 2 3 4 IOU IOU IOU IOU IOU 0 1 2 3 4 Partition 0 Partition 1 Reserved SB Partition 0 Partition 1 Rebooting partition 1 SB ...

Page 215: ...CPU from being degraded concurrently with the SB degradation two or more sets of memory should be mounted in each SB Remark The term degradation means that if a hardware error is detected by the initial diagnosis hardware executed when the power is turned on or during system operation the system runs after disconnecting the respective hardware elements 8 3 5 Operating environment Two types of OSs ...

Page 216: ...p sets GDX 0 and GDX 1 and the chip sets are connected by four internal ports These internal ports are referred to as Other Ports When a partition configuration contains Other Ports the partition may stop due to a port failure because if an Other Port is shared with another partition a failure in one partition may affect the other partition However the possibility of an Other Port failure is very ...

Page 217: ...Units and Other Ports Receive SB 3 IO_Unit 5 SB 2 IO_Unit 1 SB 1 IO_Unit 4 SB 0 IO_Unit 0 SB 7 IO_Unit 7 SB 6 IO_Unit 3 SB 5 IO_Unit 6 SB 4 IO_Unit 2 Send SB 0 SB 3 IO_Units 0 1 4 5 Other Port 0 Other Port 1 Other Port 2 Other Port 3 SB 4 SB 7 IO_Units 2 3 6 7 Other Port 0 Other Port 1 Other Port 2 Other Port 3 ...

Page 218: ......

Page 219: ...Clustering can increase the availability of the entire system Availability is an index for the reliability of a system A system in which few failures occur which continues operation even if a partial error occurs and which can be restored to its original state quickly following the occurrence of a system failure is called a system having high availability By implementing auto detection of hardware...

Page 220: ... and MMB are duplicated to increase the reliability of the system TCP IP is used as the communication protocol to prevent data delay and loss when the amount of information on the communication path between a active node and a standby node increases temporarily Cluster operation modes are described below 1 1 standby A standby node is prepared for the active node This configuration features a high ...

Page 221: ...de takes over job processing During normal operation all node can be used but when a malfunction occurs the job processing for two nodes needs to be performed on a single node Therefore the node needs to be constructed in such a way that a single node can perform the processing for the jobs on both nodes Figure 9 3 Example for mutual standby Node A Node B Node C When node A is faulty Node A Node B...

Page 222: ...ode When one node stops the remaining nodes provide a redundant configuration thereby maintaining availability during periodical maintenance tasks etc Figure 9 4 Example for a cascade Node A Node B Node C When node A is faulty Standby A Standby B Active Node A Node B Node C Active Standby Error Error ...

Page 223: ...er configuration the cabinet is treated as one node or multiple nodes to construct a cluster system Because the same operation is performed in another cabinet the degree of redundancy is high compared with the intra cluster configuration thereby increasing reliability The following example shows a connection of shared disk units by fibre channel The system disk is connected to two IO_Units and mir...

Page 224: ...Inter cabinet cluster system configuration Cabinet Cabinet MMB 1 MMB 0 GSWB GSWB GSWB GSWB MMB 1 MMB 0 Fabric SW RAID Chipset Memory HDD CPU Management LAN Business LAN FC SB 3 SB 2 SB 0 SB 1 IOU 0 IOU 1 SB 3 SB 2 SB 0 SB 1 IOU 0 IOU 1 Crossbar Crossbar ...

Page 225: ...onnection of shared disk units by fibre channel The system disk is connected to two IO_Units and mirroring is applied by the PRIMECLUSTER Global Disk Services GDS Two separately mounted IO_Units increase the resilience against failures Inter system paths are connected via the GSWB In this configuration Fujitsu recommends disabling the spanning tree protocol STP Figure 9 6 Intra cabinet cluster sys...

Page 226: ... node to another node 1 OS detected events such as OS running and shutdown are reported to the MMB 2 When a change in the partition state is detected the MMB sends a trap to an associated node registered in advance 3 A trap is sent to both of the duplicated management LANs z Event reception from the other system in the pair active server standby server This function receives information about chan...

Page 227: ...ins the inter partition cluster as the basic configuration Remarks 1 The partitions referred to in this section may be contained in a single or multiple PRIMEQUEST cabinets 9 3 1 Basic cluster configuration The following figure shows a typical two node cluster configuration Figure 9 7 Basic cluster configuration Windows z Business application software Business application software must be installe...

Page 228: ...The virtual server can be referenced from clients as a Windows network computer As a rule clients access the application software via a virtual server During switching from a node on which business application software is running the corresponding session is disconnected once However the session can be resumed by reestablishing the connection to the same virtual server after the business applicati...

Page 229: ...s disposition In the PRIMEQUEST cluster configuration all nodes must be connected by a dedicated LAN The network connecting nodes is called interconnect Multiplexing the interconnect increases the availability of the system The PRIMEQUEST interconnect should be connected to either of the following types of NIC card z NIC card added to the PCI slot in an IO_Unit z NIC card added to the PCI slot in ...

Page 230: ...eration in System Mirror provides higher availability Thus the PRIMEQUEST series increases the availability of hardware by operation in System Mirror However System Mirror does not support software redundancy Installation of a cluster system can increase the availability of the entire system also as far as software is concerned By constructing a system using both System Mirror and clustering funct...

Page 231: ...I O s yst em i nterconnect Memo ry I O s yst em i nterconnect SB 0 Cabinet Memo ry I O s yst em i nterconnect Crossbar SB 0 SB 0 SB 1 SB 0 SB 2 SB 0 SB 3 IOU 0 IOU 0 IOU IOU 1 IOU IOU 2 IOU IOU 3 GSWB System Mirror Mode Mirror Mirror Mirror Mirror Mirror Mirror Mirror Mirror Inter cabinet cluster Management LAN Private LAN Business LAN CPU Chipset Memory Memo ry I O s yst em i nterconnect Memo ry ...

Page 232: ...er a restart Figure 9 11 Example of using flexible operation in an intra cabinet cluster configuration Switching to standby cluster Switching to the reserved SB SB 4 The reserved SB is automatically changed to the standby system by reboot The SB 3 is replaced and changed to a reserved SB Crossbar Crossbar Crossbar Crossbar SB 0 SB SB SB SB 1 2 3 4 IOU IOU IOU IOU 0 1 2 3 SB 5 SB 5 SB 0 SB SB SB SB...

Page 233: ... active node collection of active node dump and system switching to the standby node are executed at the same time to shorten the time necessary for taking over job processing z System Mirror If an error occurs in hardware during operation in System Mirror switching time is not required This is because the faulty location is simply disconnected However if an error is detected in software this appr...

Page 234: ......

Page 235: ...previous status As long as data is backed up if data in a hard disk drive is damaged due to a hardware failure an operator error and the like the user can restore the backup data and recover normal system operation If the data is not backed up it will be lost Be sure to back up data periodically to operate a system with confidence 10 2 General Methods of Backup and Restoration The following four m...

Page 236: ...es and the backup and restoration of files by using NTBackup which is a Windows standard utility Configuration example when using an OS standard utility Local backup For Linux Prepare an external unit to which backup data is to be saved for each partition Enter the dump or restore command from the OS console of the partition Partition 0 in Figure 10 1 to be backed up or restored A serial console o...

Page 237: ... the network needs to have been activated For the data transmission line Fujitsu recommends using a LAN exclusively prepared for backup purposes It is also possible to use the management LAN Backup is performed by outputting data using the standard output as the output destination of the dump command and copying dd the result to a tape unit of a remote host after logging in remotely using the ssh ...

Page 238: ...cted to the KVM console by using remote desktop connection Backup and restoration can be performed after assigning the Partition 0 as a network drive However the server to be backed up and the remote host must be placed in the same network Figure 10 2 Backup to the remote host by using a standard utility when using the KVM console Partition n Partition 1 Partition 0 Backup Tape unit Restoration Re...

Page 239: ...etwork and also recover and expand the image back to the target system Configuration example when using SystemcastWizard Lite common to Linux and Windows Figure 10 3 Backup using SystemcastWizard Lite z Terminal for an MMB console A Web browser must be installed It can be used on unit b deployment server console z Deployment server console This is a server or PC with Windows OS installed Systemcas...

Page 240: ...agement LAN Place deployment server console in the same segment as the PRIMEQUEST system a terminal for an MMB console can be used also when placed in another segment SystemcastWizard Lite uses MMB 0 for communication concerning remote startup backup or restoration of a target partition Therefore make sure that MMB 0 is connected to the management LAN However MMB 0 need not be active ...

Page 241: ...CM VERITAS NetBackup NetVault NetWorker Softek TSM and others The characteristics and configuration examples of each software programs are described below Configuration example when using Softek AdvancedCopy Manager Linux Softek AdvancedCopy Manager ACM software for backup purposes is used in combination with the ETERNUS6000 ETERNUS3000 disk array unit Data of a disk array unit can be backed up by...

Page 242: ...om tape media The tape library is connected to a partition in which ACM Tape Server is installed ACM Tape Server and ACM Manager can be installed in the same partition Configuration example when VERITAS NetBackup is used Linux Normal backup of data Use the LAN or SAN back up client data to a tape unit connected to a backup server As a backup destination the user can specify a disk area connected t...

Page 243: ...ux Normal backup of data The configuration is the same as the case where VERITAS NetBackup is used The user can use the GUI on a NetVault server terminal or a NetVault remote management terminal via a LAN to back up or restore data To perform backup of multiple terminals at the same time a NetVault server disk as a backup destination should be a virtual tape library Online backup of a DB Oracle Th...

Page 244: ... a system volume using NetVault Configuration example when using NetWorker Linux z Normal backup of data Similar to the case where VERITAS NetBackup is used z Online backup of a DB Oracle NetWorker for Oracle and Oracle Recovery Manager are used The configuration is similar to the case where VERITAS NetBackup is used z Backup of a system volume Not supported Configuration example when using Softek...

Page 245: ... array data without going through a business server or a LAN Configuration example when ETERNUS SP5000 is used Common to Linux and Windows It is possible to back up the data to the ETERNUS system in Figure 10 8 Figure 10 8 Configuration for data backup with the ETERNUS SP5000 1 Support of EMC and Hitachi products is planned 2 DirectBackup and AdvancedCopy Manager ACM are mounted 3 ACM Agent is mou...

Page 246: ...backup methods Method Main usage Characteristics OS standard dump restore Linux NTBackup Windows Backup of a system volume single It is possible to back up and restore a system volume These are OS standard functions The user is normally accustomed to their operation Cloning software Systemcast Wizard Lite Linux Windows Backup of a system volume batch It is possible to back up and restore a system ...

Page 247: ...nux Backup of a system volume and backup of data areas small to middle scale DB It is possible to perform backup during DB Oracle operation Setup for a small scale system is possible at low cost It is possible to back up and restore a system volume disaster recovery NetWorker Linux Backup of data areas It is possible to perform backup during DB Oracle operation High affinity with PRIMECLUSTER Soft...

Page 248: ......

Page 249: ...e following items in the order given Area used to collect dump data Area used to save dump data Area used to collect dump data When a system becomes faulty a disk partition dump device area exclusively used for dump collection is required A dump device is a device on which the dump function temporarily stores information on problem occurrences Table 11 1 Areas used to collect dump data lists the s...

Page 250: ...ed as a dump device As the capacity of internal disks is limited the user may be unable to construct a system using only internal disks In such a case prepare a dump device on an external disk Note that a dump device prepared on an external disk cannot be shared by multiple clusters Area used to save dump data When the system is restarted the dump information that has been collected by diskdump on...

Page 251: ...Fusion MPT EFI appl Y 6 1 Flash utility F W update tool Y 7 Fusion MPT management tool CIMS Browser Y Y Broadcom LAN Related 8 Driver Driver Y Y Y 9 BCM EFI appl Y 10 BACS Win VLAN aggregation etc Y 11 BASP RHL SUSE VLAN aggregation etc Y Intel LAN Related 12 Driver Driver Y Y Y 13 Intel PROSET Win VLAN AFT ALB etc Y Emulex FC Related 14 Driver Driver Y Y Y 15 lputil Y Y 16 HBAnywhere Y Y Others 1...

Page 252: ...eck Tool Checking the environment variable argument necessary for OS operation that the user is going to set Y 24 System Data Output Tool fjsnap Executing the command for collecting the information required for support under OS operation Y 25 Server System Manager Batch operation of the power of multiple servers of different types Monitoring of abnormal states and alarm notification Y 26 Systemcas...

Page 253: ...ponents are verified as being normal the Alarm LEDs go off Each component is equipped with the following LEDs z Power LED Green Indicates the power state within the component or indicates that hot swapping of the component is being performed Also when this LED is off hot swapping hot removal of the component can be performed z Alarm LED Orange Indicates whether an error has occurred in a component...

Page 254: ... power status of the cabinet and whether an error has occurred z The operator panel has a MMB Ready LED green from which the MMB status can be checked from the outside of the cabinet z The operator panel has a Location LED blue from which the device to be operated can be identified Detailed information that cannot be represented by only the LEDs can be checked through the management LAN by using a...

Page 255: ...component Each board or component has the following display functions z Each of the main boards has a Power LED green which indicates the power status and an Alarm LED orange which indicates failure z When the Power LED is turned off the component can be deleted regardless of the states of the other LEDs z Multiple components in the system that are of the same type have their own Location LEDs blu...

Page 256: ... which indicates that the MMB is the Active one of duplicated MMBs Since the Ready LED is synchronized with MMB Ready on the operator panel the MMB status can also be checked from the operator panel Also the MMB has LEDs for a LAN The Speed LED is placed to the left of the LAN connector and the Link Act LED is placed to the right of it Figure B 2 LEDs of the MMB MMB Power Speed LAN MMB Alarm Locat...

Page 257: ...ght of it Figure B 3 LED display of the GSWB B 2 2 Other boards Each of the boards other than the OPL MMB or GSWB has a Power LED green Alarm LED orange and Location LED blue The following shows two examples in which LEDs are arranged vertically or horizontally Figure B 4 LEDs of other boards 1000 3 4 7 8 1 2 5 6 ID A0 Location GSWB Power GSWB Alarm Speed LAN Link Act LAN LAN IOU Power IOU Alarm L...

Page 258: ...ification SB system board SB Power green 1 On 48V supplied in the SB Blinking Hot swapping being performed SB Alarm orange 1 On An error in the SB Location blue 1 On A component identified Blinking A component identified MMB Management Board MMB Power green 1 On 12V supplied to the MMB MMB Alarm orange 1 On An error in the MMB Location blue 1 On A component identified Blinking A component identifi...

Page 259: ...s green On Operation being performed at 1000 Mbps IO_Unit IOB IOU Power green 1 On 48V supplied into the IO_Unit Blinking Hot swapping being performed IOU Alarm orange 1 On An error in the IO_Unit BMM or PERS Location blue 1 On A component identified Blinking A component identified BMM HOME BMM green 1 BMM On BMM on which the home ICH mounted Location blue 1 On A component identified Blinking A co...

Page 260: ...onent identified Blinking A component identified CPCB Alarm orange 1 On An error in the CPCB FAN TRAY Alarm orange 1 TRAY On An error in the FAN TRAY FAN error PSU Power Good green 1 PSU On Supplied power is sufficient Power Alarm orange 1 PSU On Supplied power is insufficient Predictive Alarm FAN speed Alarm orange 1 PSU On Fan rotation speed has decreased KVM Power green 1 On 48V supplied to the...

Page 261: ...nit On 3 3V supplied to the PCIU Blinking Hot swapping being performed Alarm orange 1 Unit On An error in the PCIU PCI card cassette Power green 1 cassette On Power ON PHP Blinking Hot swapping being performed Alarm orange 1 cassette On An error in the PCI slot Blinking A PCI slot identified FAN TRAY Alarm orange 1 TRAY On An error in the FAN TRAY FAN error FAN individual Alarm orange 1 FAN On FAN...

Page 262: ...tive green All plugs disengaged Off Off Off Off Off Off Off Off Power failure 1 Off Off Off Off Off Off Off Off Power supply started Off Off On On Off On Off Off MMB being initialized Off Blinking Off On Blinking Off Off Off Hangup during MMB initialization 2 Off Undefined Off On Undefined On Off Off Error detected during MMB initialization Off Blinking On On Blinking On Off Off Standby state MMB ...

Page 263: ...nction of HDD LEDs The LED display of an HDD is as follows Table B 3 LED display of an HDD Access green Check orange Power OFF not mounted Off Off Power OFF mounted Off On Power ON no disk access Off Off Power ON disk being accessed On Off Power ON error location identified Off On ...

Page 264: ......

Page 265: ...ss system CLI Command Line Interface This interface with UNIX or DOS allows the user to enter commands and optional arguments to communicate with the OS COM Port Communication Port RS 232C serial port for PC AT compatible machines A COM port is also called an RS 232C port Most PC AT compatible machines each have two COM ports which are often used to connect a modem terminal adapter or scanner Most...

Page 266: ... Data Xbar One of the ASICs developed by Fujitsu for the PRIMEQUEST series machine GLS Abbreviation for PRIMECLUSTER GLS GSWB Gigabit Switch Board Board with a switching hub function and a connector that receives Gigabit Ethernet interface output from an IO_Unit via the BP and outputs it to a destination outside the cabinet HBA Abbreviation for a host bus adapter HDD Hard Disk Drive Device that re...

Page 267: ... light emitting diode MAC address Media Access Control Address Unique address assigned to each network interface device switch or router mounted on a network interface card NIC or motherboard Management LAN This LAN connects the MMB to partitions and to LANs outside the cabinet so that the PRIMEQUEST system can be managed MIB Management Information Base Information released by a network device man...

Page 268: ...WB firmware and BMC firmware installed on I O_Units can use a private LAN for communication with one another OSs and applications cannot use a private LAN PSA PRIMEQUEST Server Agent Software that performs hardware error monitoring and configuration management over PRIMEQUEST partitions PSU Power Supply Unit Component that converts AC voltage to DC voltage as a DC power supply RAID Redundant Array...

Page 269: ...system in which all processors work together through common memory resources SNMP Simple Network Management Protocol TCP IP compliant protocol for managing devices in a network SSL Secure Sockets Layer Protocol under which information is encrypted for transmission SSL was developed by Netscape Communications Corp System Mirror Mechanism for duplicating memory system interconnects and internal hard...

Page 270: ...ormation to users and for user operations XAI Xbar Address Interconnect Board Board that transfers address information and controls the information transfer between SBs and IO_Unit boards XDI Xbar Data Interconnect Board Board that transfers data and controls the data transfer between SBs and IO_Unit boards ...

Page 271: ...essing at failure occurrence 9 15 clustering software 1 25 clustering basic of 9 1 COM port 2 11 communication function support list 5 3 communication mode supported 5 6 compatible PCI card 2 13 conceptual diagram PRIMEQUEST conceptual diagram 1 3 console redirection 4 3 4 20 cooling unit 2 33 CoS Class of Service 5 17 CPCB 2 27 CPCB Clock and PCI_Box Control Board 1 4 CPU 2 7 crossbar 2 21 D DDR2...

Page 272: ...dware error monitoring 4 32 hardware technology 1 8 monitor 4 15 hardware component 1 4 ACS 2 30 BP 2 21 CPCB 2 27 crossbar 2 21 dual power feed option 2 34 fan 2 33 GSWB 2 24 GTHB 2 24 IO_Unit 2 10 KVM 2 26 MMB 2 19 OPL 2 29 PCI_Box 2 34 PSU 2 31 SB 2 6 XAI 2 22 XDI 2 23 HDD 2 11 redundancy of HDD 6 10 high speed switch method duplication of transmission line between servers 6 17 hot maintenance ...

Page 273: ... 8 5 management board 1 4 2 19 management LAN 4 4 redundancy of management LAN 6 4 managing unit and other 5 22 memory 2 8 memory configuration note on 8 7 memory interleave 2 9 MIB 5 21 middleware product 1 26 mirror mode operation note on 8 6 mlog 5 21 MMB 1 4 1 21 2 19 function 4 12 IP address 4 10 MMB external interface 2 20 MMB firmware 3 3 MMB use environment setting 4 22 MMB Ready LED 2 29 ...

Page 274: ...r supply unit 2 31 Power LED 2 29 PRIMECLUSTER 3 6 9 8 linkage 4 36 PRIMECLUSTER GDS 3 6 6 11 redundancy of HDD use of PRIMECLUSTER GDS 6 10 PRIMECLUSTER GLS 3 7 6 16 redundancy of business LAN use of PRIMECLUSTER GLS 6 16 PRIMEQUEST feature 1 7 PRIMEQUEST conceptual diagram 1 3 PRIMEQUEST management LAN IP address of 4 10 PRIMEQUEST dedicated chipset 1 8 priority control function 5 17 private LAN...

Page 275: ... STP 9 12 STP setting for VLAN 5 33 structure and contents of manual i supplied software 3 5 symmetric multi processing technology 1 9 system account 4 22 system board 2 6 system clock 2 27 system event log 4 34 system interconnect 1 5 system management basic mechanism 4 1 System Mirror 1 8 6 7 improved business continuity with 1 10 note on operation 8 6 System Mirror and clustering 9 12 system ne...

Page 276: ......

Page 277: ...od Fair Poor Organization Accuracy Clarity Overall rating of this publication Use of examples Index coverage Cross referencing Legibility Binding Figures and tables General appearance Technical level Too detailed Appropriate Not enough detail For Users in U S A Canada and Mexico Fold and fasten as shown on back No postage necessary if mailed in U S A Fujitsu Computer Systems Attention Engineering ...

Page 278: ...Y MAIL FIRST CLASS MAIL PERMIT NO 741 SUNNYVALE CA NO POSTAGE NECESSARY IF MAILED IN THE UNITED STATES POSTAGE WILL BE PAID BY ADDRESSEE FUJITSU COMPUTER SYSTEMS ATTENTION ENGINEERING OPS M S 249 1250 EAST ARQUES AVENUE SUNNYVALE CA 94085 5401 ...

Page 279: ......

Page 280: ...Back Cover ...

Reviews:

No comments