manualshive.com logo in svg

HP 9000 V2500 SCA, Operator'S Manual, Page: 159 / 206

Share
Download
HP 9000 V2500 SCA Operator'S Manual Download Page 159

Chapter 7

137

7

Recovering from failures

This chapter provides detailed information on recovering from HP-UX 
system interruptions.

Usually, the first indication of a problem is that the system does not 
respond to user input. This lack of response indicates either a 
performance problem or system interruption.

Performance problems are generally characterized by:

• The system responds to one or more programs/users but not all, or 

sluggishly to others.

• The system seems to be very slow

System interruptions usually result in a total loss of CPU resources for 
all users/programs due to a:

• System hang

• System panic

• HPMC

Summary of Contents for 9000 V2500 SCA

Page 1: ...Operator s Guide HP 9000 V2500 V2600 SCA Server First Edition A5845 96001 Customer Order Number A5845 90001 July 1999 Printed in USA ...

Page 2: ...ght laws The information contained in this document is subject to change without notice Hewlett Packard makes no warranty of any kind with regard to this material including but not limited to the implied warranties of merchantability and fitness for a particular purpose Hewlett Packard shall not be liable for errors contained herein or for incidental or consequential damages in connection with the...

Page 3: ...rent xviii Installation conditions U S xix Fuse cautions xix Associated documents xx Technical assistance xxi Reader feedback xxii 1 Overview 1 V Class System Components 2 The Service Support Processor 3 Server Console and Diagnostic Connections 4 V Class Server Architecture 6 V2500 V2600 Crossbar Interconnection 6 V2500 V2600 Cabinet Components 8 Core Utilities Board 9 Processors 9 Memory 9 Input...

Page 4: ...console Node X console 40 Console bar 40 ksh shell windows 40 Using the CDE Common Desktop Environment Workspace menu 41 CDE Workspace menu 41 Using the console 45 Creating new console windows 45 Starting the console 45 Starting the console from the Workspace menu 46 Starting the console using the sppconsole command 46 Starting the console using ts_config 47 Starting the console using the consoleb...

Page 5: ...lities 71 ts_config 72 Starting ts_config 72 ts_config operation 73 Configuration procedures 75 Upgrade JTAG firmware 75 Configure a Node 77 Configure the scub_ip address 81 Reset the Node 82 Deconfigure a Node 84 Add Configure the Terminal Mux 84 Remove terminal mux 85 Console sessions 85 V2500 V2600 SCA multinode configuration 87 V2500 V2600 split SCA configuration 92 ts_config files 95 SSP to s...

Page 6: ...8 Configuring HP UX for V Class Servers 120 HP UX parameter sets 120 Multiple cabinet kernel configurations 121 Process and Thread Gang Scheduling 122 HP UX 11 10 SCA Enhancements 122 HP UX SCA Features 123 Starting HP UX 125 Power On Sequence 126 Boot variables 127 Reviewing the state of the file system 128 Stopping HP UX 130 Shutdown considerations 130 Rebooting the system 132 Shutting down the ...

Page 7: ...eds 154 Defining dump devices 155 Kernel dump device definitions 156 Runtime dump device definitions 158 Dump order 160 What happens when the system crashes 160 Operator override options 161 The dump 161 The reboot 162 What to do after the system has rebooted 162 Using crashutil to complete the saving of a dump 163 Crash dump format conversion 163 Analyzing crash dumps 164 Appendix A LED codes 165...

Page 8: ...viii Table of Contents ...

Page 9: ... LCD 28 Figure 19 SSP user windows for V2500 V2600 servers with one node 38 Figure 20 SSP user windows for V2500 V2600 servers with more than two nodes 39 Figure 21 SSP Workspace submenus for V2500 V2600 42 Figure 22 SSP Workspace submenus for V2500 V2600 42 Figure 23 SSP file system for V2500 V2600 servers 54 Figure 24 Boot process 61 Figure 25 ts_config sample display 73 Figure 26 ts_config show...

Page 10: ...re 49 Configuration started information box 90 Figure 50 ts_config showing newly configured complexes 92 Figure 51 ts_config Split Multinode complex operation 93 Figure 52 ts_config Split Multinode complex panel 93 Figure 53 ts_config Split Multinode complex panel filled in 94 Figure 54 Split Multinode confirmation panel 94 Figure 55 ts_config Split Multinode operation complete 94 Figure 56 SSP to...

Page 11: ...g pathnames 58 Table 10 Boot menu commands 65 Table 11 ts_config status values 74 Table 12 report_cfg options 111 Table 13 Hardware Path Numbering for V2500 V2600 Cabinets 119 Table 14 Boot variables 128 Table 15 CUB detects power on error 166 Table 16 CUB detects memory power fail 171 Table 17 CUB detects processor power fail 172 Table 18 CUB detects I O IOB power fail 173 Table 19 CUB detects fa...

Page 12: ...xii List of Tables ...

Page 13: ... Operator s Guide HP 9000 V2500 V2600 Server documents the information necessary to operate and monitor HP V Class servers This book is intended to be a reference for system administrators system operators and system managers ...

Page 14: ...n paragraph text italic identifies titles of documents In command syntax diagrams italic identifies variables that you must provide The following command example uses brackets to indicate that the variable output_file is optional command input_file output_file Brackets In command examples square brackets designate optional entries Curly brackets Pipe In command syntax diagrams text surrounded by c...

Page 15: ...nd how to avoid them WARNING A warning highlights procedures or information necessary to avoid damage to equipment damage to software loss of data or invalid test results Horizontal ellipses In command examples horizontal ellipses show repetition of the preceding items Vertical ellipses Vertical ellipses show that lines of code have been left out of an example Keycap Keycap indicates the keyboard ...

Page 16: ... the users of this product NOTE This equipment has been tested and found to comply with the limits for a Class A digital device pursuant to Part 15 of the FCC Rules These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment This equipment generates uses and can radiate radio frequency energy and if not installe...

Page 17: ...to radio interference may be caused to radios and TV receivers etc Read the instructions for correct handling EMI statement European Union only This is a Class A product In a domestic environment this product may cause radio interference in which case the user may be required to take adequate measures Digital apparatus statement Canada This Class A digital apparatus meets all requirements of the C...

Page 18: ... LpA 65 3 dB IT power system This product has not been evaluated for connection to an IT power system an AC distribution system having no direct connection to earth according to IEC 950 High leakage current CAUTION High leakage current Ground earth connection essential before connecting the supply Attention Forts courants de peretes Connection a une borne de terre est essentielle avant tout raccor...

Page 19: ...upplied by a separately derived system at the supply transformer or motor generator set The attachment plug receptacles in the vicinity of the unit or system are all to be of an earthing type and the earthing conductors serving these receptacles are to be connected to earth at the service equipment CAUTION For supply connections use wires suitable for at least 60 C Utillser des fils convenant à un...

Page 20: ...are doc for HP UX 11 10 HP UX 11 0 Configurable Kernel Parameters Available online at http docs hp com hpux os HP UX 11 10 Installation and Configuration Notes HP V2500 Servers A5532 90005 HP V Class Server HP UX Configuration Notes for 11 0 A4801 90001 Managing Systems and Workgroups B2355 90157 PA RISC 2 0 Architecture Reference Manual ISBN 0 13 182734 0 V2500 SCA HP UX System Guide A5532 90003 ...

Page 21: ...questions that are not answered in this book contact the Hewlett Packard Response Center at the following locations Within the continental U S call 1 800 633 3600 All others contact your local Hewlett Packard Response Center or sales office for assistance ...

Page 22: ...tion SSL FES If you have editorial suggestions or recommended improvements for this document please write to us Please report any technical inaccuracies immediately You can reach us through email at fes_feedback rsn hp com Please include the following information with your email Title and part number of the document Edition number ...

Page 23: ...ted to form a single HP UX system These SCA features are made available through HP s Coherent Toroidal Interconnect CTI technology A V2500 V2600 server can include from one to four cabinets that contain the server resources with each V2500 V2600 cabinet containing from two to 32 processors from 512 Mbytes to 32 Gbytes of memory and up to 28 PCI I O cards Each V Class system also includes a dedicat...

Page 24: ...lass cabinet The V Class server and the Service Support Processor run separate instances of the HP UX operating system Multiple cabinet servers may contain up to four V2500 V2600 cabinets which are booted as a single HP UX system Each cabinet has its own cabinet ID 0 2 4 or 6 and contains processors memory and I O resources that are available to HP UX and the applications that run on the server Ca...

Page 25: ...ge 15 Connections among the Service Support Processor and V2500 V2600 cabinets are covered in Server Console and Diagnostic Connections on page 4 Figure 4 Four Cabinet V2500 V2600 Server Components The Service Support Processor The Service Support Processor SSP workstation is an HP 712 or B180 workstation connected to the V Class server Key operations supported by the Service Support Processor inc...

Page 26: ...rom the Service Support Processor to a V2500 V2600 server s cabinet or cabinets Both the console port and diagnostic LAN on each cabinet are connected to the Service Support Processor for system monitoring booting and other operations The Service Support Processor connections to a V2500 V2600 server provide only console diagnostics and preliminary booting support For multiple cabinet servers the C...

Page 27: ...port Processor and console ports on cabinet IDs 2 4 and 6 connect to the terminal server port numbers 2 3 and 4 respectively The diagnostic LAN connects between and is terminated at the Service Support Processor and the terminal server Between these two points the diagnostic LAN runs in sequence to cabinet IDs 0 2 4 and 6 0 1 2 Util Util Util 2 6 0 4 diagnostic LAN Term Server console SSP Workstat...

Page 28: ...terconnection The primary interconnecting component of each V2500 V2600 server cabinet is the HyperPlane Crossbar which provides connections from processors and I O to memory The V2500 V2600 crossbar is a non blocking 8x8 crossbar which supports eight send messages and eight receive messages simultaneously This crossbar provides a central connection among the processor agents memory controllers an...

Page 29: ...ssor Agent Memory Controller CPU CPU CPU CPU I O I O I O Memory CTI PCI Controller Processor Agent Memory Controller CPU CPU CPU CPU I O I O I O Memory CTI PCI Controller Processor Agent Memory Controller CPU CPU CPU CPU I O I O I O Memory CTI PCI Controller Processor Agent Memory Controller CPU CPU CPU CPU I O I O I O Memory CTI PCI Controller Processor Agent Memory Controller CPU CPU CPU CPU Hyp...

Page 30: ... simultaneously V2500 V2600 Cabinet Components The key components within a V2500 V2600 server cabinet include Core Utilities Board on page 9 Processors on page 9 Memory on page 9 Input Output on page 12 In Chapter 2 you can find details on the V2500 V2600 cabinet external controls such as the on off key switch panel and cabinet displays including the LCD and attention light ...

Page 31: ...z PA 8600 processor Each processor board contains one or two processors with up to two processor boards connecting to each of the eight processor agents per cabinet The PA 8500 and PA 8600 processors are based on version 2 0 of Hewlett Packard s Reduced Instruction Set Computer RISC processor architecture NOTE The PA RISC architecture is presented in the PA RISC 2 0 Architecture reference manual P...

Page 32: ...multiple cabinet servers Details on multiple cabinet CTI connections are available in Multiple Cabinet Server Connections on page 15 Figure 8 Conceptual Overview of V2500 V2600 Memory Board The V Class cabinet has a Symmetric Multi Processing SMP design which gives all processors equal access to all memory and a uniform latency for memory accesses Multiple cabinet V2500 V2600 servers include memor...

Page 33: ...d to implement the CTI cache memory The CTI cache also called network cache is memory that is used to minimize the latency of remote memory accesses The CTI cache is directly mapped and physically indexed unlike the processor data caches The size of the CTI cache is tunable and may range from 8MB to 16GB depending on the configuration of the memory boards To set the CTI cache size use the xconfig ...

Page 34: ... by HP UX or applications Input Output A multiple cabinet V2500 V2600 server can contain up to 112 PCI I O cards with each cabinet containing up to 28 PCI I O cards Each V2500 V2600 cabinet includes 64 bit PCI chassis eight PCI buses and connections for either three or four PCI cards per PCI bus The following I O cards are supported on HP V2500 V2600 servers Tachyon Fibre Channel HVD FWD SCSI LVD ...

Page 35: ...e I O card cages are accessible from either the top left or the bottom right sides of the V2500 V2600 cabinet Figure 9 Numbering and Locations of Single Cabinet V2500 V2600 PCI I O The PCI busses in a single cabinet server are numbered from 0 to 7 as shown above This numbering also is used for the PCI busses in cabinet 0 of a multiple cabinet server 0 to crossbar 1 2 3 4 5 6 7 6 2 7 3 0 4 1 5 Top ...

Page 36: ...lustrated in Figure 10 on page 14 Figure 10 Numbering and Locations of Multiple Cabinet V2500 V2600 PCI I O 198 194 199 195 192 196 193 197 Top left PCI card cage cabinet ID 6 0 1 2 3 0 1 2 3 3 2 1 0 3 2 1 0 2 1 0 2 1 0 0 1 2 0 1 2 Bottom right PCI card cage cabinet ID 6 70 66 71 67 64 68 65 69 Top left PCI card cage cabinet ID 2 0 1 2 3 0 1 2 3 3 2 1 0 3 2 1 0 2 1 0 2 1 0 0 1 2 0 1 2 Bottom right...

Page 37: ... on page 118 Multiple Cabinet Server Connections All cabinets in a multiple cabinet V2500 V2600 server are tightly connected using HP s Coherent Toroidal Interconnect CTI technology CTI is an extension of the Scalable Coherent Interface standard defined by the IEEE CTI cables connect among the CTI controllers on the various cabinets An overview of the CTI connections for a four cabinet V2500 V2600...

Page 38: ...4 X dimension cables connect cabinets 0 and 4 and cabinets 6 and 2 Send and receive connections are provided in two dimensions on each controller for a total of four connections per controller possible In a two cabinet server cabinets 0 and 2 are connected via Y dimension CTI cables only For a three cabinet server cabinet 0 has Y dimension CTI connections to cabinet 2 and X dimension CTI connectio...

Page 39: ...tions CTI cables connect to the opposite controller on the remote cabinet This means for X dimension CTI connections memory boards connect in the following pairs 0 and 2 1 and 3 4 and 6 and 5 and 7 For details on CTI cable connections refer to qualified HP service personnel ...

Page 40: ...to one half processor capacity and to one half and full memory capacity respectively Each V2500 V2600 cabinet can contain up to 32 processors 32 Gbytes of memory and 28 PCI cards with up to four cabinets up to 128 processors 128 Gbytes of memory and 112 I O cards comprising a V2500 V2600 server Additional server configuration and ordering information is available from the following Web site http e...

Page 41: ...igurations Figure 12 Sample V2500 V2600 Cabinet Configurations A single cabinet V2500 V2600 server with 16 pro cessors and 16 Gbytes memory using 256 MByte A three cabinet V2500 V2600 server with 48 pro cessors and 96 Gbytes memory using 256 MByte ...

Page 42: ...20 Chapter1 Overview V2500 V2600 Cabinet Configurations ...

Page 43: ...Chapter 2 21 2 Indicators switches and displays This section describes indicators switches and displays of the HP 9000 V2500 server ...

Page 44: ... the server and contains the key switch panel DVD ROM drive optional DAT tape drive and the LCD display Figure 13 shows the location of the operator panel and its components Figure 13 Operator panel 3 24 99 V25U102 TOC DC ON DC OFF CON SOL E ENA BLE CON SLO LE SEC URE DVD ROM drive Key switch panel LCD display Optional DAT drive ...

Page 45: ...wn in Figure 14 Figure 14 Key switch panel Key switch The key switch has two positions DC OFF DC power is not applied to the system Placing the key switch in this position is the normal method for turning off power to the system ON DC power is applied to the system POST Power On Self Test begins executing and brings up the system from an indeterminate state and then calls OBP DC ON LED This LED in...

Page 46: ...nel as shown in Figure 13 on page 22 Figure 15 shows the DVD ROM drive front panel in detail Figure 15 DVD ROM drive Disk loading slot Place the disk into the slot with the label side up Gently push the front edge of the disk to load it into the drive When an 8 cm disk is used it must be set into an adapter prior to loading 3 17 99 V25U101 Headphone jack Volume control Busy indicator Disk loading ...

Page 47: ... of data Eject button Push the eject button to eject DVD ROMs from the drive Optional DAT drive The DAT drive is located on the right of the operator panel as shown in Figure 13 on page 22 The DAT drive front panel contains two indicator LEDs and an eject button as shown in Figure 16 Figure 16 DDS 3 DAT drive front panel LEDs The two LEDs provide operating information for normal as well as error c...

Page 48: ...rom the mechanism and ejected WARNING Do not push the eject button while the LED is flashing If you do the operation in progress is aborted and the cartridge is ejected possibly causing a loss of data Tape Activity LED green Clean Attention LED amber Meaning Flashing slowly Off A load or unload of a cartridge is in progress Flashing rapidly Off A cartridge is loaded and a read or write is in progr...

Page 49: ...ays System Displays The V Class servers provide two means of displaying status and error reporting an LCD and an Attention light bar Figure 17 System displays 9 18 97 IOLM010 TOC DC ON DC OFF CON SOL E ENA BLE CON SLO LE SEC URE LCD display Attention light bar ...

Page 50: ...art displaying output to the LCD POST is described in the HP Diagnostics Guide V2500 V2600 Servers The following explains the output shown in Figure 18 Node status line The Node Status Line shows the node ID in both decimal and X Y topology formats Processor status line The processor status line shows the current run state for each processor in the node Table 3 shows the initialization step code d...

Page 51: ... final initialization 7 Processor basic instruction set testing optional 8 Processor basic instruction cache testing optional 9 Processor basic data cache testing optional a Processor basic TLB testing optional b Processor post selftest internal register cleanup optional Status Description R RUN Performing system initialization operations I IDLE Processor is in an idle loop awaiting a command M MO...

Page 52: ...atus Description Message display code Description a Utilities board SCUB hardware initialization b Processor initialization selftest rendezvous c Utilities board SCUB SRAM test optional d Utilities board SCUB SRAM initialization e Reading Node ID and serial number f Verifying non volatile RAM NVRAM data structures g Probing system hardware ASICs h Initializing system hardware ASICs i Probing proce...

Page 53: ...lashing There is an environmental error warning or hard error condition Also indicates scanning during diagnostic execution NOTE The light bar flashing during initial start up does not indicate a fault The types of environmental conditions that are monitored include ASIC installation error sensing ASIC configuration or status 48V failure NOTE 48V failures are cleared only after a power cycle Power...

Page 54: ...nd pce to read the status of the CUB However this feature will only work after database generation is complete not before Using the SSP utility man leds to decode the CUB status nibbles The current environmental temperature set points are Warm 32 degrees Celsius 89 6 degrees Fahrenheit Hot 37 degrees Celsius 98 6 degrees Fahrenheit Displaying the CUB LED values using pce Use the sppdsh command pce...

Page 55: ...ist on the node Step 1 Bring up the sppdsh prompt at a sppuser window by entering sppdsh Step 2 Use the blink command to cause the attention light bar to blink on a specific node by entering the blink command followed by the node number For example sppdsh blink 0 For more information about the blink command see the sppdsh man page Step 3 After you have physically identified the node cause the atte...

Page 56: ...34 Chapter2 Indicators switches and displays System Displays ...

Page 57: ...ration This chapter describes the operation the SSP in conjunction with a V Class server and includes SSP log on Using the CDE Common Desktop Environment Workspace menu Using the console SSP file system System log pathnames ...

Page 58: ... console The SSP is closely interfaced with the Core Utility Board CUB located on the Mid plane Interconnect Board MIB It has the ability to access each section of the CUB allowing for control verification testing and normal management of the V Class server A private ethernet bus called the test bus connects the SSP to the CUB located within the V Class server The SSP has HP UX installed and opera...

Page 59: ... on the SSP Default password serialbus NOTE If the passwords to these accounts are changed by the customer the new passwords must be supplied to the Hewlett Packard Customer Engineer CE upon request SSP sppuser windows When the user is logged on to the SSP on a V2500 V2600 server that consists of less than two nodes the windows appear in the configuration shown in Figure 19 The Workspace however d...

Page 60: ...38 Chapter3 SSP operation SSP log on Figure 19 SSP user windows for V2500 V2600 servers with one node ...

Page 61: ...Chapter 3 39 SSP operation SSP log on Figure 20 SSP user windows for V2500 V2600 servers with more than two nodes ...

Page 62: ...nodes exist Console window sppconsole Node X console sppconsole windows for each node in the complex are spawned using the consolebar which is available from the desktop Workspace menu All POST Power On Self Test status for node X is displayed here The user can boot and configure the server from this window using the boot menu The user can also enter the special forth mode to perform special confi...

Page 63: ...ts create new windows initiate diagnostic tools and perform other tasks CDE Workspace menu The following section describes how to use the CDE Workspace menu on V2500 V2600 servers Step 1 Move the pointer over the CDE workspace backdrop Step 2 Press and hold down any mouse button The Workspace root menu appears Step 3 Drag the mouse pointer to an option Step 4 Release the mouse button to select the...

Page 64: ...42 Chapter3 SSP operation Using the CDE Common Desktop Environment Workspace menu Figure 21 SSP Workspace submenus for V2500 V2600 Figure 22 SSP Workspace submenus for V2500 V2600 ...

Page 65: ... Configures the SSP console system monitoring and diagnostic capabilities Runs as root requires the password allows reconfiguration of nodes multinode complexes and configuration of the terminal mux teststation console Creates a new console window on the screen NOTE Only one SSP console per node can be active at any time If a new SSP console is started any existing SSP console sessions for that pa...

Page 66: ...Chapter3 SSP operation Using the CDE Common Desktop Environment Workspace menu Restart Workspace Manager Stops and restarts the Workspace Manager logout Closes all open windows and stops Workspace Manager ...

Page 67: ...e console The console server program automatically starts the console on the SSP when you log on as sppuser If the console stops running restart it from the SSP using one of the following methods the Workspace menu the sppconsole command ts_config consolebar or logging back on Methods for starting the console V2500 V2600 servers are Workspace menu sppconsole command ts_config consolebar logging ba...

Page 68: ... menu restart the Workspace manager The desktop Workspace menus are updated whenever a node is configured or deconfigured but the new menus are not activated until the Workspace Manager is restarted Step 8 Select Console menu Step 9 Select Node X complex The new console window appears Starting the console using the sppconsole command This method of starting the console works from the SSP or after ...

Page 69: ...nnn where nnnn is the Node ID extended to four digits and zero filled on the left These names can be viewed using jf ccmd info or ts_config For example sppconsole guardian 0000 To start a console on Node 2 of the complex named guardian enter sppconsole guardian 0002 Starting the console using ts_config This method of starting the console works from the SSP or after logging on from another system T...

Page 70: ...ethod of starting the console works from the SSP or after logging on from another system The consolebar utility is an GUI that shows the configured nodes grouped by complex Each node is a push button that when pushed activates a console session for that node If more than two nodes are configured on a V2500 V2600 SSP consolebar automatically starts when the sppuser logs in at the SSP display To sta...

Page 71: ...utton to select the option The SSP closes all open windows and returns a HP UX login prompt Step 5 Log into the SSP as sppuser The new sppconsole window displays Console commands Use the sppconsole commands to control the console Using these commands allows the user to watch or to assume control of the console window Table 7 sppconsole commands NOTE E is the Ctrl and e keys pressed simultaneously ...

Page 72: ...the console on page 51 for more information To monitor the console from a system other than the SSP complete the following steps Step 1 Remotely log in to the SSP as sppuser default password spp user with the following command rlogin hostname login sppuser Password spp user Step 2 Access the system console with the following command sppconsole At this point the console is in spy mode meaning the u...

Page 73: ... as sppuser default password spp user with the following command rlogin hostname login sppuser Password spp user Step 2 Access the system console with the following command sppconsole At this point the console is in spy mode meaning the user can only monitor what is going on at the system console If commands are entered the following message is displayed read only use Ecf to attach Ec for help Ste...

Page 74: ...he set_complex command From an existing shell on the SSP the user can set the default complex for the shell by executing the set_complex command set_complex complex name The set_complex command lists the configured complex names and prompts the user for a selection After a complex has been selected the user can issue diagnostic scan and console commands against a particular node ID e g 0 The SSP s...

Page 75: ... interfaces have been set to on an SPP The command provides a list with the following information Ethernet Address MAC IP Address Complex Serial Number Node Number Environmental LED s Power Status SCUB Status Diagnostic JTAG node names The diagnostic node name or the JTAG IP address is required when using the load_eprom command Since the SSP software allows changing the default JTAG hostnames and ...

Page 76: ...he unique daemons that run on the SSP These daemons manage of the V Class node Two daemons that are always running on the SSP are ccmd A daemon that maintains a database of information about the V Class hardware It also monitors the system and reports any significant changes in system status For more information see the ccmd man page spp etc spp specific daemons bin compiled executables scripts ex...

Page 77: ...t receives messages from diagnostic utilities through rpc calls and writes them to the event log for later review or processing dcm Dump Configuration Map dcm dumps the boot configuration map information for the specified node spp scripts The spp scripts directory contains scripts that perform a variety of functions sppconsole The console utility hard_logger The hard error logger script run automa...

Page 78: ...e useful in troubleshooting intermittent ASIC failures event_log Log of all event information A read only file which captures information generated by the ccmd daemon spp firmware The spp firmware directory is where firmware files are written when SSP software is installed The firmware files are loaded from this directory into flash or SRAM spp est The est directory contains files used during scan...

Page 79: ... Location on workstation Device file Private diagnostic LAN RS 232 dev lan1 LAN AUI dev lan0 Global customer LAN LAN TP dev lan0 Slot 2 dev lan1 Node 0 console port RS 232 dev tty0p0 Serial 1 dev tty1p0 Terminal mux configuration port 9 PIN connector of the Y Cable dev ty1p0 Serial 2 dev tty0p0 Remote modem 9 PIN connector of the Y Cable dev ttyd1p0 or dev cua1p0 or dev cul1p0 Serial 2 dev ttyd0p0...

Page 80: ...old new pathname mappings are shown in Table 9 Table 9 System log pathnames Log name V2500 V2600 pathname CCMD log file spp data ccmd_log CCMD old log file spp data ccmd_log old Z Node 0 consolelog spp data complex consolelog0 Node 2 consolelog spp data complex consolelog2 Node CFG file spp data complex node_0 cfg Node PWR file spp data complex node_0 pwr Event log spp data complex event_log Event...

Page 81: ...Chapter 4 59 4 Firmware OBP and PDC This chapter discusses the boot sequence and the commands available from the boot menu ...

Page 82: ...g process occurs 1 Power On Self Test POST runs POST is described in LCD Liquid Crystal Display on page 28 and in the HP Diagnostics Guide V2500 V2600 Servers 2 OBP probes all the devices 3 OBP loads SPP_PDC in RAM 4 OBP starts the HP UX loader which in turns calls SPP_PDC to set up CPU s memory and I O devices in a way that HP UX understands 5 The next action depends on whether Autoboot is enable...

Page 83: ... Figure 24 Boot process NO NO YES YES Boot menu Prompt displays To discontinue press any key within 10 seconds Continue Automatically Press any key HP UX boots Processor is starting the autoboot process displays Autoboot Enabled to display boot menu ...

Page 84: ...____ ____ ____ ____ ____ PB4L_A MB3R ____ ____ ____ ____ ____ ____ Building main memory map Main memory initialization complete Starting multinode initialization Collecting memory configuration from nodes 0 2 Initializing ERI rings for node 0 2 Synchronizing nodes 0 2 Initializing CTI cache r0 r1 r2 r3 PB4L_A MB0L LLLL ____ CCCC ____ ____ ____ ____ ____ PB4L_A MB1L LLLL ____ CCCC ____ ____ ____ __...

Page 85: ...3 PCIs 0 3 4 7 NODE 2 UART No CORE MAC address 0 a0 d9 0 be eb IP 15 99 111 167 0x0f636fa7 JTAG MAC address 0 a0 d9 0 c3 a3 IP 15 99 111 117 0x0f636f75 MEMORY 2048 MB memory installed 1024 MB CTI cache configured CPUs 15 PACs 0 1 2 3 4 5 6 7 MACs 0 1 2 3 TACs 0 1 2 3 PCIs 2 6 4096 MB memory installed 2048 MB CTI cache configured total all nodes Primary boot path 1 0 0 6 0 Primary boot arguments Al...

Page 86: ...RI ALT CON path Display or modify a path PDT CLEAR DEBUG Display clear Non Volatile PDT state PIM_info cpu HPMC TOC LPMC Display PIM of current or any CPU RemoteCommand node command Execute command on a remote node RESET hard debug Force a reset of the system RESTrict ON OFF Display Select restricted access to Forth SCSI INIT RATE bus slot val List Set SCSI controller parms SEArch path Search for ...

Page 87: ... or sets a delay time for the system to wait for external mass storage devices to come online CLEARPIM Clears zeros Processor Internal Memory PIM storage after a system crash CAUTION this command can delete important troubleshooting information do not enter the CLEARPIM command unless directed to CPUconfig proc ON OFF Displays or sets the configuration of processors DEfault Sets the system environ...

Page 88: ...d command on the remote node identified by node number RESET hard debug Resets the system state RESTrict ON OFF Displays or sets restricted access to Forth mode SCSI INIT RATE bus slot val Displays or sets SCSI controller initiator ID or transfer rate SEArch path Displays pathnames for devices with bootable media in the system SECure ON OFF Displays or sets secure boot mode If secure mode is set t...

Page 89: ...h Autoboot and Autosearch is OFF Syntax AUto BOot SEArch ON OFF Used alone this command displays the current status of the Autoboot and Autosearch flags BOot If ON the OS is automatically loaded from the primary boot path after a power up or reset Otherwise the system displays the boot menu and waits for interactive boot commands During an autoboot the process pauses for 10 seconds to allow the op...

Page 90: ...t Examples au This command displays the status of the Autoboot and Autosearch flags Autoboot ON Autosearch ON au bo This command displays the current setting of the Autoboot flag Autoboot ON au bo on This command sets the Autoboot flag ON Autoboot ON ...

Page 91: ... named command Examples The following example illustrate use of this command help au This command displays information for the auto command AUto BOot SEArch ON OFF Display or set the specified flag AUto boot on Enable auto boot on next boot AUto boot off Disable auto boot on next boot AUto search on Enable auto search on next boot AUto search off Disable auto search on next boot Auto search enable...

Page 92: ...70 Chapter4 Firmware OBP and PDC HElp command ...

Page 93: ...nfiguration information OBP can also be used to modify the configuration The SSP allows the user to configure the node using the ts_config utility This is the preferred method for V2500 V2600 servers ts_config configures the SSP to communicate with the node The SSP daemon ccmd monitors the node and reports back configuration information error information and general status ts_config must be run be...

Page 94: ...g a node scub_ip address and resetting a node Configuration of Multiple node complex Configuring V2500 V2600 nodes into a single complex and splitting V2500 V2600 nodes out of a multiple node complex Operational support Resetting a V2500 V2600 node or multiple node complex and starting console sessions The user must have root privilege to configure a node or the terminal mux because several HP UX ...

Page 95: ...powered up and connected to the SSP diagnostic LAN The operator selects a node and configures the selected node A sample display is shown below Figure 25 ts_config sample display The window has three main parts the drop down menu bar the display panel and the message panel The display panel contains a list of nodes and their status To select a node click with the left mouse button the line contain...

Page 96: ...the nodes Table 11 shows the possible status values Table 11 ts_config status values Configuration Status Description Action Required Upgrade JTAG firmware The version of JTAG firmware running on the SCUB does not support the capabilities required to complete the node configuration process Select the node and follow the instructions given at the bottom of the ts_config window ts_config guides the ...

Page 97: ...SP node configuration file contains information about the specified node but the node is not responding to requests on the Diagnostic LAN This status is also shown if a node was configured and then removed from the SSP LAN without being deconfigured Power up the node and or check for a LAN connection problem If the node information shown is for a node that has been removed select the node then sel...

Page 98: ...ser what action to take next This node s JTAG firmware must be upgraded Select Actions Upgrade JTAG firmware and Yes to upgrade Step 2 Select Actions to drop the pop down menu and then click Upgrade JTAG firmware as shown in Figure 27 Figure 27 ts_config Upgrade JTAG firmware selection Step 3 A message panel appears as the one shown in Figure 28 Read the message If this is the desired action click...

Page 99: ... the new firmware Figure 29 ts_config power cycle panel When the node is powered up the Configuration Status should change to Not Configured Configure a Node Step 1 Select the desired node from the list of available nodes When the node is selected the appropriate line is highlighted as shown in Figure 30 Notice the bottom of the display indicates the Node 0 is not configured and provides the steps...

Page 100: ...fig indicating Node 0 as not configured Step 2 Select Actions and then click Configure Node as shown in Figure 31 Figure 31 ts_config Configure Node selection After invoking ts_config to configure the node a node configuration panel appears as the one in Figure 32 ...

Page 101: ...iguration panel ts_config automatically assigns the first unused serial port If the terminal mux has been configured the terminal mux ports are included in the list of available serial connections The IP address information for the Diagnostic interface is provided The ts_config utility automatically changes the IP address of the diagnostic LAN interface to prevent a duplicate when other nodes are ...

Page 102: ...nges to Active as shown in Figure 34 Figure 34 ts_config indicating Node 0 is configured Step 7 Restart the Workspace Manager Click the right mouse button on the desktop background to activate the root menu Select the Restart or Restart Workspace Manager option then OK to activate the new desktop menu NOTE If adding multiple nodes to the SSP wait until the final node is added before restarting the...

Page 103: ...he scub_ip address stored in NVRAM on the SCUB in the node This would initially be the default address set at the factory If the scub_ip address is correct the panel shown in Figure 36 is displayed and no action is required If the node is not detected and scanned by ccmd ts_config may ask you to try again later The ccmd detection scan process should take less than a minute Figure 36 ts_config SCUB...

Page 104: ... in Figure 38 appears confirming that the scub_ip address is set Click OK Figure 38 ts_config scub_ip address set confirmation panel Initiate a node reset to activate the new scub_ip address Reset the Node Step 1 Select the desired node from the list of available nodes Step 2 Select Actions then Reset Node This is indicated in Figure 39 ...

Page 105: ...ties ts_config Figure 39 ts_config Reset Node selection A panel as the one shown in Figure 40 appears Figure 40 ts_config node reset panel Step 3 In the Node Reset panel select the desired Reset Level and Boot Options then click Reset ...

Page 106: ...available nodes Step 2 Select Actions then Deconfigure Node then click Yes Add Configure the Terminal Mux To add or reconfigure the terminal mux perform the following procedure Step 1 In the ts_config display select Actions then Configure Terminal Mux Select Add Configure Terminal Mux This is indicated in Figure 41 Figure 41 ts_config Add Configure Terminal Mux selection Step 2 Connect a serial ca...

Page 107: ... node consoles are assigned to terminal mux ports Step 1 Select Actions then Configure Terminal Mux Step 2 Select Remove Terminal Mux then click Yes Console sessions ts_config may also start console sessions by selecting the desired node s and then selecting the Start Console Session action as shown in Figure 43 Figure 44 shows the started console sessions ...

Page 108: ...86 Chapter5 Configuration utilities ts_config Figure 43 Start Console Session selection Figure 44 Started console sessions ...

Page 109: ... To configure the two node system in the example start ts_config as described in Starting ts_config on page 72 Once ts_config has started a window like that shown in Figure 45 is displayed Figure 45 SSP supporting two single node complexes The following procedure configures the two node SCA system in the example Step 1 Select the nodes by clicking anywhere in the information display for each node ...

Page 110: ...lities ts_config Figure 46 ts_config Configure Multinode complex selection Step 3 When Configure Multinode complex is selected a configuration dialog appears as shown in Figure 57 Figure 47 Configure Multinode Complex dialog window ...

Page 111: ...the SCA system complex serial numbers are the same no complex key is required Step 5 Select the desired node IDs from the New Node ID drop down lists Step 6 If the console connection must be changed select appropriate connection from the Console Connection drop down list Step 7 In the Hypernode bitmask section select POST will determine bitmask Step 8 If necessary select the desired CTI cache size...

Page 112: ...age box appears indicating that the configuration has started Figure 49 Configuration started information box The following activities occur during the configuration process SSP files are updated based on the new complex and node names Essential console server processes are started and the now obsolete server processes are halted New node information is written to the COP chip in each node ...

Page 113: ...s are written to NVRAM in each node These include Hypernode bitmask X and Y ring information Node count CTI cache size Node local memory size The boot vector of each node is set to OBP and each node is reset When the configuration process is complete ts_config shows the new multinode complex as in Figure 50 The restart process activates the new SSP root menu which includes customized menus for eac...

Page 114: ...e and choosing the Configure Multinode complex action Set the desired options and click Configure During a reconfiguration several of the required fields in the Multinode Configuration dialog are filled in by ts_config V2500 V2600 split SCA configuration ts_config also provides a Split Multinode complex action that takes an SCA complex and logically splits it into single node systems Each node bec...

Page 115: ... Figure 52 ts_config Split Multinode complex panel Step 3 Enter the complex names for each node New complex serial numbers may be assigned Each node becomes node 0 in a new complex Figure 53 shows the Split Multinode panel filled in Click the Split Complex button to initiate the configuration process ...

Page 116: ...dicating the configuration is taking place Figure 54 Split Multinode confirmation panel Figure 55 shows the main ts_config display after the split multinode operation has completed It shows the resulting configuration two single node complexes two node 0s with names assigned in the prior step Figure 55 ts_config Split Multinode operation complete ...

Page 117: ...an also update this file delete_node configure_node and split_multinode spp data conserver cf Connection definitions for the console interface spp data consoles conf Console name to ttylink number resolution This file is maintained by the Configure Mux Action of ts_config spp data complex_name For each newly configured complex there is a complex specific directory that contains complex specific fi...

Page 118: ...me Host name used for OBP communication not normally referenced while administering the complex OBP IP addresses are 15 99 111 166 through 15 99 111 181 SSP hostname Local host name of the private diagnostic LAN The default host name for this interface is tsdart d Console port Name of the physical connection to the node RS 232 console port The port name is linked to a ttylink service entry via the...

Page 119: ...h OBP spp_pdc is platform dependent code and runs on top of OBP providing access to the devices and OBP configuration properties ethernet DUART HPUX ethernet Scan console NFS FWCP modem remote diagnostic RS 232 ccmd event_logger hard_logger sppconsole ttylink LCD OBP POST testcontroller consolemessages LCD Global LAN global ethernet Private LAN privateethernet RS 232 JTAGFW ethernet consolemessage...

Page 120: ...tion database on the SSP The board names and revisions the device names and revisions and the start up information generated by POST are all read and stored in memory for use by other diagnostic tools IMPORTANT Both the B180L and the 712 workstations must have two ethernet connections one for the private LAN and one for the global LAN These ports are different on each model of workstation It is im...

Page 121: ... Test and the Test Controller send console messages The SSP processes these messages using the sppconsole and ttylink utilities and the consolelogx log file POST and OBP also send system status to the LCD connected to the DUART For more information on sppconsole ttylink and consolelogx see the appropriate man pages NOTE The second RS 232 port on the workstations are unused and not enabled at this ...

Page 122: ... a node powers up ccmd reads hardware information from the node and interrogates it through scan to determine the node configuration From this data a complete database is built on the SSP that will be used for all scan based diagnostics Once running ccmd checks for power up power down reset error and environmental conditions on regular intervals If at any time ccmd detects a change in the configur...

Page 123: ... share a common ethernet port and Diagnostic DART bus In general the scan data is sent via UDP The DART bus should be separate from any general purpose ethernet bus If the DART bus is improperly set up ccmd cannot run properly Since ccmd can become corrupted by bad data it may be necessary to kill the ccmd process to refresh the SSPs configuration image Killing the ccmd process is not always enoug...

Page 124: ...mote debug xconfig is started from a shell Information on node 0 is read and interpreted to form the starting X windows display shown in Figure 57 The xconfig window appears on the system indicated by the environmental variable DISPLAY This may be overridden however by using the following command xconfig display system_name 0 0 The xconfig window has two display views one shows each component as a...

Page 125: ...Chapter 5 103 Configuration utilities xconfig Figure 57 xconfig window physical location names ...

Page 126: ...ed the item selected changes state and color There is a legend on the screen to explain the color and status The change is recorded in the SSP s image of the node When the user is satisfied with the new configuration it should be copied back into the node and the node should be reset to enable the changes ...

Page 127: ...ror Enable menu Displays the device menu options for error enabling and configuration Help menu Displays the help and about options The menu bar is shown in Figure 59 Figure 59 xconfig window menu bar The File menu provides the capability to save and restore node images and to exit xconfig The Memory menu provides the capability to enable or disable memory at the memory DIMM level by the total mem...

Page 128: ...Configuration utilities xconfig Node configuration map The node configuration map is a representation of the left and right side views of a node as shown in Figure 60 Figure 60 xconfig window node configuration map ...

Page 129: ...igure this component in order to properly execute Grey button Indicates a hardware component that did not properly initialize The colors are shown in the legend box of the node control panel Components can change from enabled to disabled or disabled to unknown by clicking on the appropriate button with the left mouse button A multinode system requires an additional component on a memory board to e...

Page 130: ...umber is shown in the node box A new number can be selected by clicking on the node box and selecting the node from the pull down menu A new complex can be selected by clicking on the complex box and selecting it from the pull down A node IP address is displayed along with the node number and complex ...

Page 131: ... one or all nodes of a system A multinode system requires a reset all to properly function A Retrieve button is available on the node control panel to get a fresh copy of the parameters settings in the system Clicking this button overwrites the setting local to the SSP and xconfig The Stop on hard button is typically used to assist in fault isolation It stops all system clocks shortly after an err...

Page 132: ...isabled NOTE autoreset determines the behavior of ccmd when it encounters an error condition ccmd makes its decision whether to reset a complex immediately after running hard_logger Enabling autoreset after hard_logger has run does not reset the complex est_config est_config is a utility that builds the node and complex descriptions used by est est_config reads support files at spp data DB_RING_FI...

Page 133: ...ons If the report_cfg tool detects any nodes of complexes that contain SCA DIMMS and some memory boards that are not populated with STACS it generates a report Example configuration report The system inventory has determined that you ll need to order 8 SCA Upgrade Kits in order to connect this cabinet with other SCA cabinets These upgrade kits are available by additionally ordering opt 010 of the ...

Page 134: ...lid size information into the BCM for a DIMM report_cfg reports the physical size reported by POST For example if a node has both 80 and 88 bit DIMMs POST reconfigures the 88 bit DIMMs to behave as 80 bit DIMMs and the system logically behaves as if it has all 80 bit DIMMs report_cfg however distinguishes using the physical attribute in the BCM between the 80 and 88 bit DIMMs in its reports Anothe...

Page 135: ...2 1 2 hw2a 0 3 2 2 1 2 hw2a 0 4 2 2 1 hw2a 0 5 2 2 1 hw2a 0 6 2 2 1 hw2a 0 7 2 2 1 hw2a 2 0 2 2 1 2 hw2a 2 1 2 2 1 2 hw2a 2 2 2 2 1 2 hw2a 2 3 2 2 1 2 hw2a 2 4 2 2 1 hw2a 2 5 2 2 1 hw2a 2 6 2 2 1 hw2a 2 7 2 2 1 report_cfg I O report To obtain an I O report use the i option The following is a sample I O report by report_cfg report_cfg i Complex Node MIB COP SCUB COP hw2a 0 A5074 60002 00 a 3845 A50...

Page 136: ...e Board COP 2 8 6 2 8 6 hw4a 0 MB0L A5078 60003 01 a 00XB 8 hw4a 0 MB1L A5078 60003 01 a 00XB 8 hw4a 0 MB2R A5078 60003 01 a 00X2 8 hw4a 0 MB3R A5078 60003 01 a 00X2 8 hw4a 0 MB4L A5078 60003 01 a 00XA 8 hw4a 0 MB5L A5078 60003 01 a 00X2 8 hw4a 0 MB6R A5078 60003 01 a 00XA 8 hw4a 0 MB7R A5078 60003 01 a 00X2 8 hw4a 2 MB0L A5078 60003 01 a 00XA 8 hw4a 2 MB1L A5078 60003 01 a 3842 8 hw4a 2 MB2R A507...

Page 137: ...60001 00 a 00XA 2 0 hw2a 0 PB4L_B A5492 60001 00 b 00XB 2 0 hw2a 0 PB5L_B A5492 60001 00 a 00XB 2 0 Complex Node Processor COP CPU rev hw2a 2 PB2L_A A5491 60001 00 a 00XA 2 0 hw2a 2 PB2R_A A5492 60001 00 a 00XA 2 0 hw2a 2 PB3R_A A5491 60001 00 a 00XA 2 0 hw2a 2 PB4L_A A5492 60001 00 b 00XC 2 0 hw2a 2 PB5L_A A5491 60001 00 a 00XA 2 0 hw2a 2 PB4L_B A5492 60001 00 b 00XC 2 3 xsecure xsecure is an app...

Page 138: ...s begun The label near the red button will inform the user when the SSP is secure A green indicator and the appropriate label shows that the network is available and the SSP may be accessed through the ethernet port In order for xsecure to work properly the SSP console cables terminal mux and modems must be configured in specific ways The SSP JTAG connections OBP connections and an optional termin...

Page 139: ... its Service Support Processor This section covers issues related to using HP UX V11 0 and HP UX V11 10 on V Class servers Multiple cabinet server configurations and HP UX SCA features require that HP UX V11 10 be installed Topics covered in this section include HP UX on the V2500 V2600 Starting HP UX Stopping HP UX ...

Page 140: ... as shown in the command example below model 9000 800 V2500 For details see the model 1 man pages The top command displays information on the top processes based on CPU use on the system and lists CPU utilization data for the system s processors Because V Class servers can have many processors you may want to issue top h when using this command The h option suppresses printing individual lines of ...

Page 141: ...cation but on cabinet ID 2 has a hardware path of 65 2 0 9 0 The above disk is on cabinet ID 2 and is connected to PCI bus 65 slot 2 and has a target SCSI ID of 9 This corresponds to the top left PCI card cage of cabinet ID 2 as shown in Table 13 The following table lists the hardware path numbering for key V2500 V2600 system components as they are numbered on the various cabinets in a multiple ca...

Page 142: ...tions menu HP UX parameter sets HP UX kernel configurations are provided for the following types of V Class server use Scientific and technical use Servers running applications that have very large data sets and may have long processing times Examples include NASTRAN Abaqus mechanical and electrical design applications and fluid dynamics applications The V Class Technical Server tuned parameter se...

Page 143: ...ernel configurations The following are notable initial kernel parameter settings for multiple cabinet V2500 V2600 servers There settings are provided when installing HP UX 11 10 on multiple cabinet servers maxuprc Maximum number of user processes Initial SCA value 256 maxusers The MAXUSERS value used in various kernel formulae Initial SCA value 256 max_thread_proc Maximum number of threads allows ...

Page 144: ...cessors in the system Gang scheduling also permits low latency interactions among threads in shared memory parallel applications Only applications using the HP UX V11 0 MPI or pthread libraries can be gang scheduled Because HP compiler parallelism is primarily built on the pthread library programs compiled with HP compilers can benefit from gang scheduling For more details refer to the gang_sched ...

Page 145: ...e locality domain processor and memory targeting and binding You can specify the locality domain or processor on which a process runs by using the mpsched command the mpctl system call and the pthread interfaces pthread_processor_bind_np and pthread_ldom_bind_np You also can specify the locality from which a process memory is allocated via the mmap and shmget system calls Launch and Scheduling Pro...

Page 146: ...process in the locality that is least loaded at the time of its creation Gang scheduling of threads and processes is supported through the mpsched g option and the MP_GANG HP UX environment variable Details are provided in the gang_sched 7 man page and in the section Process and Thread Gang Scheduling on page 122 More details about HP UX programming scheduling and launch enhancements are available...

Page 147: ...are Runs HP UX The main firmware interface the OBP boot menu provides a straightforward interface for managing a system before HP UX boots The OBP menu is available through the V Class console interface In multiple cabinet V2500 V2600 SCA systems each V2500 V2600 cabinet runs its own copy of the firmware Normally during booting you only need to interact with the OBP menu on cabinet ID 0 which serv...

Page 148: ...er On Sequence Turn on the SSP power and allow it to boot before powering on the V Class server s cabinets This allows the SSP to be used to monitor and control the V Class server as it boots is used The following is the sequence for powering on an HP V Class server and its SSP Step 1 Power on the SSP and allow it to boot HP UX Step 2 Power on the V Class server cabinets On multiple cabinet V2500 ...

Page 149: ...enu For example you can set the server to automatically proceed to boot HP UX by setting the AUTO BOOT option to ON and setting the PRI boot path variable Boot variable settings are stored in non volatile memory NVRAM residing on the utility board of each V Class server cabinet These variables are stored permanently until they are modified using the HP boot menu interface Refer to Chapter 4 Firmwa...

Page 150: ... When set to OFF the server boots to the OBP menu interface AUto SEArch ON OFF If set to ON the server searches for and lists all bootable I O devices AUto Force ON OFF If set to ON then OBP allows HP UX to boot even if one or more cabinets does not complete power on self test When set to OFF all cabinets must successfully pass power on self test for OBP to permit the server to boot HP UX BootTime...

Page 151: ...nconsistencies in the file systems without your intervention and without removing data The fsck command does one of the following Repairs and reboots the system incorporating the changes Prompts the user to run the fsck command manually If fsck needs to be run manually see the fsck 1m manpage 3 Other errors detected An error message displays for example unable to open a specified device file the s...

Page 152: ...rning off the power Stopping the system improperly can corrupt the file system Use the shutdown or reboot commands Shutdown considerations Only the system administrator or a designated superuser can shut down the system The sbin shutdown command Warns all users to log out of the system within a grace period you specify Halts daemons Kills unauthorized processes Unmounts file systems Puts the syste...

Page 153: ...Chapter 6 131 HP UX Operating System Stopping HP UX See the shutdown man page for a complete description of the shutdown process and available options ...

Page 154: ... to reboot HP UX you may need to reset the server See Resetting the V2500 V2600 server hardware on page 134 Step 3 Change to the root directory Enter cd Step 4 Shut down the system using the shutdown or reboot command Enter shutdown r or reboot Progress messages detailing system shutdown activities print to the terminal Upon reaching run level 0 the system Restarts in single user mode Displays the...

Page 155: ...er shutdown Progress messages detailing system shutdown activities print to the terminal Upon reaching run level 0 the system Restarts in single user mode Displays the root prompt Step 5 Shut down and halt HP UX using the shutdown or reboot command Enter shutdown h or reboot h Progress messages detailing system shutdown activities print to the terminal CAUTION Turn power off to the cabinet only af...

Page 156: ...me to reset The do_reset command s syntax is as follows do_reset node_id all level boot_option If do_reset is specified with no arguments then the default level 1 reset of all cabinets is performed rebooting the server to OBP Either a cabinet ID node_id or the keyword all can be specified to indicate which cabinets are to be reset If a cabinet ID is specified the reset level also must be specified...

Page 157: ...ut down HP UX you can proceed with Step Two and may want to perform a level 4 reset at Step Three Step 2 Access a Service Support Processor login shell You can do this directly at the Service Support Processor workstation or by remotely logging in with a telnet or rlogin session Step 3 Issue the do_reset command from a Service Support Processor login shell By default do_reset performs a level 1 re...

Page 158: ...136 Chapter6 HP UX Operating System Stopping HP UX ...

Page 159: ...pond to user input This lack of response indicates either a performance problem or system interruption Performance problems are generally characterized by The system responds to one or more programs users but not all or sluggishly to others The system seems to be very slow System interruptions usually result in a total loss of CPU resources for all users programs due to a System hang System panic ...

Page 160: ...oubleshooting information Step 1 If an error message is displayed on the system console record it Step 2 Record the information displayed on the system LCD See LCD Liquid Crystal Display on page 28 for more information Step 3 Record any relevant information contained in the log files in the spp data complex directory on the SSP event_log Main log file hard_hist Filtered output from the hard_logger...

Page 161: ...rces ps top See Managing Systems and Workgroups and the ps and top man pages for more information about options and usage Step 2 Enter a Ctrl C from the terminal exhibiting the problem to abort an executing command Step 3 Check another terminal to verify that the problem is not just a console hang Step 4 Contact the Hewlett Packard Customer Response Center HP UX kernel configuration can affect per...

Page 162: ...f the following utilities to communicate with the server ping telnet rlogin See the ping telnet and rlogin man pages for more information about options and usage Step 5 If possible wait about 15 minutes to see if the computer is really hung or if it has a performance problem With some performance problems a computer may not respond to user input for 15 minutes or longer Step 6 If the computer is r...

Page 163: ...ed in the event of a disk head crash or similar situation How frequently the backups are updated depends on how much data one can afford to be lose For information on how to back up data refer to Managing Systems and Workgroups After HP UX experiences a system panic the system May display an HPMC tombstone on the console if panic was caused by an HPMC A tombstone is a list of register values used ...

Page 164: ...wording of the panic message should allow the problem to be classified into one of the following areas Peripheral problem Server or I O card problem File system problem LAN communication problem Logical Volume Manager LVM related problem Other Peripheral problem Use the following procedure to troubleshoot an apparent peripheral hardware failure Step 1 Check to ensure the device is powered on and o...

Page 165: ...the device there may be an interface card or system problem If the problem reappears it might be necessary to have the problem fixed by Hewlett Packard service personnel Interface card and system problem Use the following procedure if a hardware failure appears to be associated with an interface card or with the an internal component of the system Step 1 If an HPMC tombstone is displayed record it...

Page 166: ...one with the directory has problems it is important to use the n option to the reboot command right after fsck completes fsck is normally run automatically at boot time See Rebooting the system on page 146 LAN communication problem Use the following procedure if the panic messages indicate a problem with LAN communication Step 1 Check LAN cable and media access unit MAU connections Step 2 Ensure t...

Page 167: ...t show up immediately It will occur when the truncated part of the file system is overwritten by something else such as a new logical volume or the extension of a logical volume in the same volume group as the truncated file system For more information on LVM see Managing Systems and Workgroups Recovery from other situations When a problem appears with something other than that has previously been...

Page 168: ... fsck will ask the operator to reboot the system when it finishes Use the command reboot n The n option tells reboot not to sync the file system before rebooting Since fsck has made all the corrections on disk this will not undo the changes by writing over them with the still corrupt memory buffers Monitoring the system after a system panic If the system successfully reboots there is a good chance...

Page 169: ... memory or all memory without operator interaction By default fast dump selectively dumps only the parts of memory that are expected to be useful in debugging It improves system availability in terms of both the time and space needed to dump and analyze a large memory system The following commands allow the operator to configure save and manipulate the fast core dump crashconf Configures the desti...

Page 170: ...for later analysis Prior to HP UX 11 0 dump devices had to be defined in the kernel configuration and they still can be using Release 11 0 Beginning with Release 11 0 however a new more flexible method for defining dump devices is available using crashconf Beginning with HP UX Release 11 0 there are three places where dump devices are configured 1 In the kernel same as releases prior to Release 11...

Page 171: ...rash dump format is provided to support dumping selected memory on multiple cabinet V2500 V2600 servers This new SCA extended crash dump format is used only on V2500 V2600 SCA systems and requires the crash dump utilities provided in the HP UX 11 10 release Non SCA systems including single cabinet V2500 V2600 servers and all other HP systems use the non SCA crash dump format Unlike the SCA extende...

Page 172: ... configure system dumps The criteria are System recovery time Get the system back up as soon as possible Crash information integrity Capture the correct information Disk space needs Conserve available disk space System recovery time To get the system back up and running as soon as possible consider the following factors Dump level Compressed save vs noncompressed save Using a device for both pagin...

Page 173: ...rash can be configured by editing the file etc rc config d savecrash to compress or not compress the data as it copies the memory image from the dump devices to the HP UX file system area during the reboot process This effects system recovery time in that data compression takes longer Therefore if there is enough disk space and the fastest system recovery is required configure savecrash to not com...

Page 174: ...l instead of savecrash to complete the copy Crash information integrity This section discusses how to make sure the part of memory that contains the instruction or piece of data that caused the crash is captured The factors that must be considered are Full dump vs selective dump Dump definitions built into the kernel vs defined at runtime Using a device for both paging and as a dump device Full du...

Page 175: ...mount of dump space already configured is available at the time of the crash in this example it is 256 Mbytes of space Define enough dump space in the kernel configuration if it is critical to capture every byte of memory in all instances including the early stages of the boot process NOTE This example is presented for completeness The actual amount of time between the point where kernel dump devi...

Page 176: ...p and or the post reboot save of the memory image The factors to consider are Dump level Compressed save vs noncompressed save Partial save savecrash p Dump level There are three levels of core dumps full dump selective dump and no dump The fewer pages required to dump the less space is required to hold them Therefore a full dump is not recommended If disk space is really at a premium one option i...

Page 177: ...es only those pages on dump devices that are endangered by paging activity i e pages on the devices used for both paging and as dump devices Pages that are on dedicated dump devices remain there To configure this option into the boot process edit the file etc rc config savecrash and comment out the line that sets the environment variable SAVE_PART 1 Defining dump devices When defining dump devices...

Page 178: ...for a margin of safety In the above example the calculation would be 6208 x 4 Kbytes x 1 25 approx 30 Mbytes Kernel dump device definitions Capturing dumps for crashes that occur during early stages of the boot process requires sufficient dump space in the kernel configuration Using SAM to configure dump devices into the kernel The easiest way to configure dump devices is to use SAM A screen for d...

Page 179: ...sing HP UX commands Step 1 Edit the system file the file that config uses to build the new kernel This is usually the file stand system but it can be another file if that is preferred Dump to Hardware Device For each hardware dump device to be configured into the kernel add a dump statement in the area of the file designated Kernel Device info immediately prior to any tunable parameter definitions...

Page 180: ...e primary paging device swap device as the dump device Step 2 Once the system file has been edited build a new kernel file using the config command Step 3 Save the existing kernel file probably stand vmunix to a safe place such as stand vmunix safe in case the new kernel file can not be booted Step 4 Boot the system from the new kernel file to activate the new dump device definitions Runtime dump ...

Page 181: ...volumes from volume groups other than the root volume group can be used The crashconf command Use the sbin crashconf command to add to remove or redefine dump devices The following are two ways to do this Reread the etc fstab file using the crashconf a option Use device arguments with crashconf to configure the devices With either method use the crashconf r option to specify that new definitions r...

Page 182: ...ices was built from a kernel build from the etc fstab file from use of the crashconf command or any combination of the these dump devices are used dumped to in the reverse order from which they were defined The last dump device in the list is the first one used and the first device in the list is the last one used Place devices that are used for both paging and dumping early in the list of dump de...

Page 183: ...ring N for no dump at the system console within the 10 second override period If disk space is limited but the operator feels that a dump is important the operator can enter S for selective dump regardless of the currently defined dump level The dump After the operator overrides the current dump level or the 10 second override period expires HP UX writes the physical memory contents to the dump de...

Page 184: ...es WARNING If using devices for both paging and dumping do not disable savecrash boot processing Loss of the dumped memory image to subsequent system paging activity can occur What to do after the system has rebooted After the system reboots make sure that the physical memory image dumped to the dump devices is copied to the HP UX file system area then either package and send it in for analysis or...

Page 185: ...em area has been configured before doing so If a partial save is being done the only pages copied to the HP UX file system area during the boot process are those that were on paging devices Pages residing on dedicated dump devices are still there A partial save can be selected by leaving the SAVECRASH environment set to 1 and setting the environment variable SAVE_PART 1 in etc rc config d savecras...

Page 186: ...he source will be overwritten See the crashutil 1M manpage for more information Analyzing crash dumps Analyzing crash dumps is not a trivial task It requires intimate knowledge of HP UX internals and the use of debuggers It is beyond the scope of this document to cover the actual analysis process Contact the Hewlett Packard representative for help in analyzing a crash dump ...

Page 187: ...ion LED on the core utilities board CUB turns on and the Attention light bar on the front of the node flashes to indicate the presence of an error code listed Table 15 Additionally only the highest priority error is displayed Once remedied an error that is cleared may expose a lesser priority error ...

Page 188: ... lowest priority NOTE Errors from LED hex code 00 through hex code 67 shut the system down and errors from hex code 68 through 73 leave the system up Table 15 CUB detects power on error LED Fault Symptoms Corrective action 00 3 3V error highest priority 1 5V is up 3 3V is not 2 SSP interface will not function Call the Response Center 01 ASIC Install 0 MIB 1 Incorrect rotation or part in one of the...

Page 189: ...OK error Upper Right 1 Power supply is reporting failure dc OK after keyswitch is turned on but prior to CUB power on sequence 2 This is the first of two or more supplies reporting failure Call the Response Center 06 dc OK error Lower Left 1 Power supply is reporting failure dc OK after keyswitch is turned on but prior to CUB power on sequence 2 This is the first of two or more supplies reporting ...

Page 190: ...overload condition on 48V bus 3 Possible node power supply NPS upper left failure Call the Response Center 12 1B 48V error NPSUR failure PWRUP 0 9 1 Error occurs when 48 volt distribution falls below 42 volts during powerup state displayed Powerup state indicates which loads are being turned on 2 Excessive load on 48 volts due to an inadequate number of functioning 48 volt supplies or overload con...

Page 191: ...overload condition on 48V bus 3 Possible node power supply NPS lower left failure Call the Response Center 26 2F 48V error NPSLR failure PWRUP 0 9 1 Error occurs when 48 volt distribution falls below 42 volts during powerup state displayed Powerup state indicates which loads are being turned on 2 Excessive load on 48 volts due to an inadequate number of functioning 48 volt supplies or overload con...

Page 192: ...s board CUB lost and then regained 48V power without the machine being turned off or ac power failure 2 Core utilities board CUB will display this error and not power on the system Cycle dc power to the node using the keyswitch to attempt to clear the Yo Yo bit Call the Response Center 3B MIB power fail MIBPB 1 VDD 3 3V error on MidPlane power board MIBPB 2 Midplane power fails and entire node wil...

Page 193: ...ble 16 CUB detects memory power fail LED Fault Symptoms Corrective action 40 MB0L Power Fail 1 3 3V dropped below acceptable level 2 Core utilities board CUB detected a power loss on reported memory board MB 3 Core utilities board CUB powers down the system Call the Response Center 41 MB1L Power Fail 42 MB2R Power Fail 43 MB3R Power Fail 44 MB4L Power Fail 45 MB5L Power Fail 46 MB6R Power Fail 47 ...

Page 194: ...B0L Power Fail 1 3 3V dropped below acceptable level 2 Core utilities board CUB detected a power loss on the reported processor board PB 3 Core utilities board CUB powers down the system Call the Response Center 49 PB1R Power Fail 4A PB2R Power Fail 4B PB3R Power Fail 4C PB4L Power Fail 4D PB5R Power Fail 4E PB6L Power Fail 4F PB7R Power Fail 50 PB0R Power Fail 51 PB1L Power Fail 52 PB2R Power Fai...

Page 195: ...etects I O IOB power fail LED Fault Symptoms Corrective action 58 Left Front I O Board failure 1 3 3V or 5V dropped below acceptable level 12V and 12V not monitored 2 Core utilities board CUB detected a power loss on reported I O board IOB 3 Core utilities board CUB powers down the system Call the Response Center 59 Left Rear I O Board failure 5A Right Front I O Board failure 5B Right Rear I O Boa...

Page 196: ...positions are referred to as viewed from the rear of the server Table 19 CUB detects fan power fail LED Fault Symptoms Corrective action 5C Fan failure Upper Right Sensor in the reported fan as viewed from rear of system determines fan failure Call the Response Center 5D Fan failure Upper Middle 5E Fan failure Upper Left 5F Fan failure Lower Right 60 Fan failure Lower Middle 61 Fan failure Lower L...

Page 197: ... sensed overtemp on MidPlane power board MIBPB and powers down the system Check that airflow is not blocked Check fans Call the Response Center 64 QUADRL 0 1 Board overheated in Quadrant 0 2 Core utilities board CUB sensed overtemp in Quadrant 0 and powers down the system Call the Response Center 65 QUADRU 1 1 Board overheated in Quadrant 1 2 Core utilities board CUB sensed overtemp in Quadrant 1 ...

Page 198: ...ring utilities chip MUC on the core utilities board after power on Table 21 Hard error LED Fault Symptoms Corrective action 68 Hard error RAC PAC MAC TAC SAGA 1 Hard error lines to core utilities board CUB reported ASIC problem 2 Bit and hard error bus determine which ASIC to check Read spp data hard_list Call the Response Center ...

Page 199: ...by the monitoring utilities chip MUC on the core utilities board after power on Table 22 Ambient air intake error LED Fault Symptoms Corrective action 69 Ambient air too warm is an environmental warning Intake air through CUB too warm Check site temperature and correct If the fault reoccurs when room temperature is within spec call the Response Center ...

Page 200: ... MUC on the core utilities board after power on Table 23 dc error LED Fault Symptoms Corrective action 70 NPSUL failure warning 1 Node power supply Viewed from Node front failure reported 2 Low priority error for redundant power configurations Call the Response Center 71 NPSUR failure warning 72 NPSLL failure warning 73 NPSLR failure warning ...

Page 201: ...D 2 configurations 18 numbering 2 119 cache 11 CTI 11 12 cards I O supported 12 physical access 14 caution defined xv ccmd 71 98 100 101 ccNUMA 10 cmd 54 command prompt see boot menu commands see also utilities autoboot 67 blink 33 crashconf 147 159 crashutil 147 do_reset 134 ioscan 118 jf ccmd_info 53 model 118 mpsched 123 reboot 132 savecrash 147 set_complex 46 52 53 shutdown 132 sppconsole 46 t...

Page 202: ...n 25 EMI xvii enable Autoboot 67 environmental errors 32 error codes see LED errors error indicator 31 error information 138 est 55 event_log 52 56 event_logger 55 Exemplar Routing Access Controllers ERACs 6 F failure information see also logs and LED errors failures recovery 137 fast dump 147 FCC xvi FDDI 12 file system and bcheckrc script 128 firmware 60 97 Forbin Project The 47 force 128 FORTH ...

Page 203: ...0 core utility 3B 170 core utility 40 171 core utility 41 171 core utility 42 171 core utility 43 171 core utility 44 171 core utility 45 171 core utility 46 171 core utility 47 171 core utility 48 172 core utility 49 172 core utility 4A 172 core utility 4B 172 core utility 4C 172 core utility 4D 172 core utility 4E 172 core utility 4F 172 core utility 50 172 core utility 51 172 core utility 52 17...

Page 204: ...figure 77 80 deconfigure 84 reset 82 83 node routing board MUC detected errors 171 poweron detected errors 166 node status line 28 node see cabinet node_0 cfg 55 notational conventions xiv note defined xv NRB 36 numbering I O devices 14 of hardware components 119 NVRAM 127 O OBP 60 71 127 128 commands auto 65 67 autoboot 128 autoforce 128 autosearch 128 boot 65 boottimer 65 128 clearpim 65 cpuconf...

Page 205: ...rash 147 SCA 1 see also multiple cabinet configuration split 92 94 HP UX support 122 kernel configuration 121 Scalable Coherent Interface 15 Scalable Computing Architecture see SCA SCSI 12 scub_ip address 81 82 configure 81 82 server configurations 18 Service Support Processor 2 3 connections to V2500 V2600 server 4 operations 3 ts_config utility 11 utilities board connection 9 xconfig utility 11 ...

Page 206: ...upgrade JTAG firmware JTAG upgrade firmware 75 77 USA radio frequency notice xvi using the console 45 utilities autoreset 110 ccmd 71 98 100 101 consolelog 99 dfdutil 98 est_config 110 pcirom 98 spp_pdc 97 sppconsole 99 sppdsh 71 ts_config 71 96 98 xconfig 71 102 109 xsecure 115 utilities board 4 9 multiple cabinet connections 5 numbering 119 V V2500 V2600 differences 9 V2500 V2600 server overview...

Reviews:

No comments