Oracle SPARC T5-8 Service Manual Download

Page: 150 / 206

Understanding PCIe Root Complex Connections

150

SPARC T5-8 Server Service Manual • November 2015

■

“Root Complex Connections (Four Processor Modules)” on page 150

■

“Root Complex Connections (Two Processor Modules)” on page 151

Root Complex Connections (Four Processor

Modules)

This diagram illustrates the root complex connections between the four processor modules and

the 16 PCIe I/O slots in a fully-populated server. Each CPU supports two I/O root complexes.

Each root complex connects to two I/O slots through two of five multiplexing switches.

The port ID values shown in the diagram correspond to the

pci@

values reported in the

OpenBoot

show-devs

command output. For example:

CPU

Switch

I/O Slot

Root Complex Path

/pci@300/pci@1/pci@0/pci@6

Summary of Contents for SPARC T5-8

Page 1: ...Part No E35082 14 November 2015 SPARC T5 8 Server Service Manual ...

Page 2: ......

Page 3: ...including applications that may create a risk of personal injury If you use this software or hardware in dangerous applications then you shall be responsible to take all appropriate fail safe backup redundancy and other measures to ensure its safe use Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications Or...

Page 4: ...ions à risque notamment dans des applications pouvant causer des dommages corporels Si vous utilisez ce logiciel ou matériel dans le cadre d applications dangereuses il est de votre responsabilité de prendre toutes les mesures de secours de sauvegarde de redondance et autres mesures nécessaires à son utilisation dans des conditions optimales de sécurité Oracle Corporation et ses affiliés déclinent...

Page 5: ...mponents 22 Component Service Task Reference 23 Detecting and Managing Faults 25 Understanding Diagnostics 25 Diagnostics Process 25 Tool Availability 27 Log In to Oracle ILOM Service 28 Oracle ILOM Service Related Tools 29 Interpreting LEDs 29 Front Panel Controls and LEDs 31 Rear Panel Controls and LEDs 33 Configuring POST 35 POST Overview 35 Oracle ILOM Properties That Affect POST Behavior 35 C...

Page 6: ...ver 53 Removing Power From the Server 53 Prepare to Power Off the Server 54 Power Off the Server SP Command 54 Power Off the Server Power Button Graceful 55 Power Off the Server Emergency Shutdown 55 Disconnect the Power Cords 56 Prevent ESD Damage 57 Servicing Processor Modules 59 Server Upgrade Process 59 Processor Module LEDs 61 Determine Which Processor Module Is Faulty 62 Remove a Processor M...

Page 7: ...odule 95 Verify the Main Module 97 Servicing the Storage Backplanes 99 Remove a Storage Backplane 99 Install a Storage Backplane 103 Verify the Storage Backplane 107 Servicing the Service Processor Card 109 Determine if the Service Processor Card Is Faulty 109 Remove the Service Processor Card 110 Install the Service Processor Card 112 Verify the Service Processor Card 114 Servicing the System Con...

Page 8: ... Fan Module Is Faulty 142 Remove a Fan Module 143 Install a Fan Module 145 Verify the Fan Module 146 Servicing PCIe Cards 149 Understanding PCIe Root Complex Connections 149 Root Complex Connections Four Processor Modules 150 Root Complex Connections Two Processor Modules 151 PCIe Card Installation Guidelines 153 PCIe Carrier LEDs and Controls 154 Determine Which PCIe Card Is Faulty 155 Remove a C...

Page 9: ...y 181 Determine if the Rear Chassis Subassembly Is Faulty 181 Remove the Rear Chassis Subassembly 182 Install the Rear Chassis Subassembly 185 Verify the Rear Chassis Subassembly 188 Returning the Server to Operation 191 Connect the Power Cords 191 Power On the Server Oracle ILOM 192 Power On the Server Power Button 193 Glossary 195 Index 201 ...

Page 10: ...10 SPARC T5 8 Server Service Manual November 2015 ...

Page 11: ...uthorized service providers Required knowledge Advanced experience troubleshooting and replacing hardware Product Documentation Library Late breaking information and known issues for this product are included in the documentation library at http www oracle com goto T5 8 docs Feedback Provide feedback about this documentation at http www oracle com goto docfeedback ...

Page 12: ...12 SPARC T5 8 Server Service Manual November 2015 ...

Page 13: ...14 Rear Panel Components Service on page 15 DIMM Locations on page 16 Main Module Internal Component Locations on page 17 Supported Storage Devices on page 18 Fan Module Locations on page 20 Rear I O Module Port Locations on page 21 Chassis Subassembly Components on page 22 Component Service Task Reference on page 23 Related Information Detecting and Managing Faults Preparing for Service Returning...

Page 14: ...Hard Drives Supported Storage Devices on page 18 2 Processor modules 0 to 3 bottom to top or processor filler modules slot 1 and slot 2 Servicing Processor Modules Server Upgrade Process on page 59 3 Main module Servicing the Main Module 4 Power supplies 0 to 3 left to right Servicing Power Supplies Related Information Rear Panel Components Service on page 15 ...

Page 15: ...he server These components are not part of the rear chassis subassembly You must remove these components to access the rear chassis subassembly No Description Links 1 Fan modules 10 Servicing Fan Modules Fan Module Locations on page 20 2 AC power connectors 0 to 3 right to left Servicing the Rear Chassis Subassembly 3 Rear I O module Servicing the Rear I O Module 4 PCIe carriers 1 to 16 left to ri...

Page 16: ...he rear chassis subassembly is removed and serviced as a single unit No Description Links 1 Server chassis Servicing the Rear Chassis Subassembly 2 Midplane 3 Rear chassis subassembly Related Information Front Panel Components Service on page 14 Chassis Subassembly Components on page 22 DIMM Locations DIMMs are located in each of the processor modules ...

Page 17: ...M slots populated with DIMMs All of the DIMM slots must be populated with the same type of DIMMs For guidelines see DIMM Configuration on page 71 No Description Link 1 DIMMs Servicing DIMMs Main Module Internal Component Locations These components are accessible after you remove the main module from the front of the server ...

Page 18: ... Storage Backplanes 4 Motherboard Servicing the Main Module 5 SP Servicing the Service Processor Card 6 System configuration PROM Servicing the System Configuration PROM 7 Battery Servicing the Battery Supported Storage Devices The server supports these storage devices Fibre channel arrays SATA FC flash and SAS 2 SAS arrays SAS 2 ZFS appliances SAS 2 The server also supports these types of tape ba...

Page 19: ...annel SAS LVD SCSI You can install a mixture of storage devices but the server requires at least one storage device to be installed and operational No Description Link 1 Drive 1 Servicing Hard Drives 2 Drive 0 3 Drive 3 4 Drive 2 5 Drive 5 6 Drive 4 7 Drive 7 8 Drive 6 ...

Page 20: ... Service Manual November 2015 Fan Module Locations No Description Link 1 Fan module 5 Servicing Fan Modules 2 Fan module 0 3 Fan module 6 4 Fan module 1 5 Fan module 7 6 Fan module 2 7 Fan module 8 8 Fan module 3 9 Fan module 9 10 Fan module 4 ...

Page 21: ...et port enables you to connect the server to your local network so that you can manage the server from a remote location 2 Serial management SER MGT This serial port enables you to connect directly to the SP 3 Ethernet network NETx Where x is the number of the port You can use these Ethernet ports to connect the server to local or wide area networks 4 USB These ports provide access for input devic...

Page 22: ...ystem controls and indicators Front Panel Components Service on page 14 5 Processor modules 4 fully populated or processor filler modules slot 1 and slot 2 Servicing Processor Modules 6 Chassis Servicing the Rear Chassis Subassembly 7 Rear chassis subassembly Servicing the Rear Chassis Subassembly 8 Fan modules 4 Servicing Fan Modules 9 PCIe carriers 16 Servicing PCIe Cards 10 Rear I O module Serv...

Page 23: ...em Refer to the SPARC T5 Server Installation Guide DIMMs 128 SYS PMx CMx CMP BOBx CHx D0 System Memory DIMMs DIMM_x 16 or 32 GB Servicing DIMMs Disk drives 8 SYS SASBPx HDDx System Storage Disks Disks_x SAS 300 GB or 600 GB or SSD 100 GB or 300 GB Servicing Hard Drives Fan modules 10 SYS RCSA FBDx FMx System Cooling Fans Fan_x Servicing Fan Modules Front I O assembly 1 SYS FIO These components are...

Page 24: ...odules Server Upgrade Process on page 59 Rear I O module 1 SYS RIO Servicing the Rear I O Module Power supplies 4 SYS PSx System Power Power_Supplies Power_Supply_x Servicing Power Supplies Rear chassis subassembly 1 SYS RSCA Servicing the Rear Chassis Subassembly SCC PROM 1 SYS MB SCC Servicing the System Configuration PROM SP 1 SP SP Servicing the Service Processor Card Storage backplanes 2 SYS ...

Page 25: ...ge 45 Related Information Identifying Components Preparing for Service Component Service Task Reference on page 23 Returning the Server to Operation Oracle ILOM Documentation Library Understanding Diagnostics These topics explain the diagnostic process and tools Diagnostics Process on page 25 Tool Availability on page 27 Log In to Oracle ILOM Service on page 28 Oracle ILOM Service Related Tools on...

Page 26: ... Service Manual November 2015 Note The diagnostic tools you use and the order in which you use them depend on the nature of the problem you are troubleshooting However for descriptive purposes this table follows the steps given in the illustration ...

Page 27: ...rpreting Log Files and System Messages on page 45 4 RunOracle VTS software To run Oracle VTS the server must be running the Oracle Solaris OS If Oracle VTS reports a faulty component replace it If Oracle VTS does not report a faulty component run POST Refer to the Oracle VTS software documentation Configuring POST on page 35 Contact technical support if the problem persists Related Information Too...

Page 28: ...fault password changeme for the default Administrator account root after your initial login to Oracle ILOM If this default Administrator account has since been changed contact your system administrator for an Oracle ILOM user account with Administrator privileges 2 Enable the Oracle ILOM 3 0 legacy name spaces set SP cli legacy_targets enabled Note In Oracle ILOM 3 1 the name spaces for SYS and ST...

Page 29: ... a Stop A depending on the mode in which the Oracle Solaris OS was booted start HOST console Connects to the host show HOST console history Displays the contents of the host s console buffer set HOST bootmode property value Controls the method of booting for the host server s firmware The value of property can be state config or script stop System or stop SYS Powers off the host server start Syste...

Page 30: ... in these links to determine if the component has been diagnosed as being faulty Determine if the Main Module Is Faulty on page 91 Determine Which Processor Module Is Faulty on page 62 Determine Which DIMM Is Faulty LEDs on page 75 Determine Which Hard Drive Is Faulty on page 82 Determine Which Power Supply Is Faulty on page 134 Determine Which Fan Module Is Faulty on page 142 Determine Which PCIe...

Page 31: ...adm faulty command provides details about any faults that cause this indicator to light See Check for Faults on page 41 Under some fault conditions individual component fault LEDs are lit in addition to the Service Required LED 3 Power OK LED green Indicates these conditions Off Server is not running in its normal state Server power might be off The SP might be running Steady on Server is powered ...

Page 32: ...dicates that a temperature failure event has been acknowledged and a service action is required 6 Fan Module Fault LED amber Rear FM Indicates these conditions Off Indicates a steady state no service action is required Steady on Indicates that a fan module failure event has been acknowledged and a service action is required on at least one of the fan modules 7 PCIe Card Fault LED amber Rear PCIe I...

Page 33: ... these conditions Off The link is operating as a 10 Mbps connection On or blinking The link is operating as a 100 Mbps connection 4 Network port link LED Indicates these conditions Off No link is established Blinking A link is established 5 Network port speed LED Indicates these conditions Off The link is operating as a 10 Mbps connection or there is no link Amber on The link is operating as a 100...

Page 34: ...eturned to full function Slow blink A normal but transitory activity is taking place Slow blinking might indicate that system diagnostics are running or that the system is booting 10 SP LED SP Indicates these conditions Off AC power might have been connected to the power supplies Steady on green SP is running in its normal operating state No service actions are required Blink green SP is initializ...

Page 35: ... diagnostic information POST displays These properties are described in Oracle ILOM Properties That Affect POST Behavior on page 35 If POST detects a faulty component the component is disabled automatically If the server is able to run without the disabled component the server boots when POST completes its tests For example if POST detects a faulty processor core the core is disabled POST complete...

Page 36: ... processor and memory tests min If diag mode normal runs minimum set of tests TABLE 3 HOST diag trigger Value Description hw change default Runs POST following a FRU replacement or an AC power cycle all resets Runs POST on all resets error reset Runs POST on all error resets power on reset Runs POST on every power on none Does not run POST on reset TABLE 4 HOST diag hw_change_level Value Descripti...

Page 37: ...luding component names and test results debug Displays extensive debugging information none Disables the output TABLE 8 HOST diag error_reset_level Value Description max default Runs the maximum set of tests min Runs a minimum set of tests TABLE 9 HOST diag error_reset_verbosity Value Description min default Displays the minimum level of output max Displays information for each step normal Display...

Page 38: ...tput max Displays all test informational and some debugging messages in POST output debug Displays extensive debugging information none Does not display POST output This flowchart illustrates the same set of Oracle ILOM set command variables Related Information POST Overview on page 35 Configure POST on page 39 Run POST With Maximum Testing on page 40 ...

Page 39: ...roperties That Affect POST Behavior on page 35 3 If the virtual keyswitch is set to normal and you want to define the mode level verbosity or trigger set the respective parameters Syntax set HOST diag property value See Oracle ILOM Properties That Affect POST Behavior on page 35 for a list of parameters and values Examples set HOST diag mode normal set HOST diag verbosity max 4 View the current va...

Page 40: ...Log in to Oracle ILOM See Log In to Oracle ILOM Service on page 28 2 Set the virtual keyswitch to diag so that POST runs in service mode Alternatively you can use the System target set HOST keyswitch_state diag Set keyswitch_state to Diag 3 Run POST Alternatively you can use the System target start SYS Are you sure you want to start SYS y n y Starting SYS 4 Start the system console to view the out...

Page 41: ...ult notification with a message ID You can use the message ID to get additional information about the problem from the knowledge article database A PSH console message provides this information about each detected fault Type Severity Description Automated response Impact Suggested action for system administrator If PSH detects a faulty component use the fmadm faulty command to display information ...

Page 42: ...xxxxxxxxxx Suspect 1 of 1 Fault class fault io pciex device interr corr Certainty 100 Affects hc chassis 0 motherboard 0 cpuboard 0 chip 0 hostbridge 0 pciexrc 0 Status faulted but still in service FRU Status faulty Location SYS PM0 Manufacturer Oracle Corporation Name TLA PN NRM T5 1 2 Part_Number 7061001 Revision 01 Serial_Number 465769T 12445102WR Chassis Manufacturer Oracle Corporation Name SP...

Page 43: ...o faults were detected return to Broken Link Target ID Z40019D01512366 If a fault is detected proceed to Step 4 4 Use the message ID to obtain more information about this type of fault a Obtain the message ID from console output b Go to https support oracle com and search on the message ID in the Knowledge tab 5 Follow the suggested actions to repair the fault 6 Determine your next step If you fou...

Page 44: ...nt fault records In some cases even though the fault is cleared some persistent fault information remains and results in erroneous fault messages at boot time To ensure that these messages are not displayed type this PSH command faultmgmtsp fmadm acquit UUID 4 If required reset the server In some cases the output of the fmadm faulty command might include this message for the faulty component fault...

Page 45: ...have the full complement of Oracle Solaris OS files and commands available for collecting information and for troubleshooting If PSH does not indicate the source of a fault check the message buffer and log files for notifications for faults Drive faults are usually captured by the Oracle Solaris message files These topics explain how to view the log files and system messages Check the Message Buff...

Page 46: ...s file After a period of time usually every week a new messages file is automatically created The original contents of the messages file are rotated to a file named messages 1 Over a period of time the messages are further rotated to messages 2 and messages 3 and then deleted 1 Log in as superuser 2 Type more var adm messages 3 To view all logged messages type more var adm messages Related Informa...

Page 47: ...View Log Files Oracle ILOM Detecting and Managing Faults 47 View Log Files Oracle Solaris on page 46 ...

Page 48: ...48 SPARC T5 8 Server Service Manual November 2015 ...

Page 49: ... 53 7 Prevent ESD damage before you handle any server component Prevent ESD Damage on page 57 Related Information Identifying Components Detecting and Managing Faults Component Service Task Reference on page 23 Returning the Server to Operation Safety Information For your protection observe the following safety precautions when setting up your equipment Follow all cautions and instructions marked ...

Page 50: ...electronic components that are extremely sensitive to static electricity Ordinary amounts of static electricity from clothing or the work environment can destroy the components located on these boards Do not touch the components along their connector edges Caution You must disconnect all power supplies before servicing any of the components that are inside the chassis Antistatic Wrist Strap Use We...

Page 51: ...ng types of filler panels and modules Hard drive filler panels DIMM filler panels these are used only to ship new processor modules You must remove all of the DIMM filer panels and replace them with DIMMs before installing the new processor modules DIMM filler panels are not supported in running processor modules PCIe card carriers these function like filler panels when a card is not installed Pro...

Page 52: ...s On or off Servicing Power Supplies Processor modules Off Servicing Processor Modules Rear I O module Off Servicing the Rear I O Module Rear chassis subassembly Off X Servicing the Rear Chassis Subassembly SP Off Servicing the Service Processor Card Storage backplanes Off Servicing the Storage Backplanes SCC PROM Off X Servicing the System Configuration PROM Related Information Removing Power Fro...

Page 53: ...cle ILOM prompt type set SYS LOCATE value Fast_Blink Alternatively you can type set System locator_indicator on The white Locator LEDs one on the front panel and one on the rear panel blink 2 After locating the server with the blinking Locator LED turn it off by pressing the Locator button Note Alternatively you can turn off the Locator LED by running the Oracle ILOM set SYS LOCATE value off comma...

Page 54: ...t down the Oracle Solaris OS Refer to the Oracle Solaris administration documentation for details 5 Power off the server Related Information Power Off the Server SP Command on page 54 Power Off the Server Power Button Graceful on page 55 Power Off the Server Emergency Shutdown on page 55 Configuring Boot and Restart Behavior in SPARC and Netra SPARC T5 Series Servers Administration Guide Power Off...

Page 55: ...tration Guide Power Off the Server Power Button Graceful This procedure places the system in the power standby mode To service cold replaceable components you must remove the power 1 Press and release the recessed Power button The Power OK LED blinks rapidly 2 If you are servicing a cold service component or if you are upgrading the server disconnect the power cords See Disconnect the Power Cords ...

Page 56: ...ide Disconnect the Power Cords 1 Ensure that you have shut down the system See Power Off the Server SP Command on page 54 Power Off the Server Power Button Graceful on page 55 2 Disconnect all of the power cords Caution Because 3 3v standby power is always present in the server you must unplug the power cords before accessing any cold serviceable components See Component Service Categories on page...

Page 57: ... as an antistatic mat Antistatic bag used to wrap a replacement part ESD mat A disposable ESD mat shipped with some replacement parts or optional server components 2 Attach an antistatic wrist strap When servicing or removing server components attach an antistatic strap to your wrist and then to a metal area on the chassis Related Information Safety Information on page 49 Servicing Processor Modul...

Page 58: ...58 SPARC T5 8 Server Service Manual November 2015 ...

Page 59: ...omponents that are sensitive to electrostatic discharge This discharge can cause server components to fail These topics describe how to service the processor modules Server Upgrade Process on page 59 Processor Module LEDs on page 61 Determine Which Processor Module Is Faulty on page 62 Remove a Processor Module or Processor Filler Module on page 63 Install a Processor Module or Processor Filler Mo...

Page 60: ...odules The steps to remove DIMM filler panels are the same as the steps for removing DIMMs Remove a DIMM or DIMM Filler Panel on page 76 4 Install the DIMMs All of the DIMMs must be either 16 or 32 GB and they must match the size and capacity of the DIMMs that are already installed in the server Install a DIMM on page 78 5 Replace the covers on the new processor modules Step 1 in Install a Process...

Page 61: ...cessor Filler Module on page 67 Returning the Server to Operation Root Complex Connections Four Processor Modules on page 150 PCIe Card Installation Guidelines on page 153 Processor Module LEDs No LED Icon Description 1 Ready to Remove blue Indicates that a processor module can be removed 2 Service Required amber Indicates that the processor module has experienced a fault condition 3 OK green Indi...

Page 62: ...essor module at PM0 results in server shutdown and failure to reboot If your server experiences a fault at PM0 and you do not have a replacement processor module available you can move one of the other processor modules to PM0 and then boot the server in a degraded state Caution In order to maintain system cooling all four processor module slots must be occupied either with a processor module or a...

Page 63: ... components that can be replaced only after you power off the system For the location of the modules see Front Panel Components Service on page 14 Caution This procedure requires that you handle components that are sensitive to electrostatic discharge This discharge can cause server components to fail 1 Remove all of the power from the system See Removing Power From the Server on page 53 2 Take th...

Page 64: ...ber 2015 4 Squeeze the release latches together on the two extraction levers and pull the extraction levers out to disengage the processor module or processor filler module from the server 5 Pull the processor module or processor filler module halfway out of the server and then close the levers ...

Page 65: ...ng two hands completely remove the processor module or processor filler module and place the module on an antistatic mat 7 Determine your next step a If you are replacing DIMMs in an existing processor module go to Step 8 b If you are upgrading the server to a fully populated configuration repeat Step 4 through Step 6 to remove the second processor filler module in slot 2 and then go to Server Upg...

Page 66: ...9 Determine your next step If you are replacing DIMMs see Servicing DIMMs If you are installing new processor modules to upgrade the server return to Server Upgrade Process on page 59 If you are replacing a faulty processor module follow these steps a Remove all of the DIMMs from the faulty processor module and set them in a safe place See Remove a DIMM or DIMM Filler Panel on page 76 b Install th...

Page 67: ...p If you are replacing the cover as part of the upgrade process or if you are servicing a processor module go to Step 2 If you are installing new processor modules as part of the upgrade process go to Step 4 2 Place the cover back onto the processor module and slide the cover forward until the latch clicks into place 3 Determine your next step If you are installing a new processor module upgrading...

Page 68: ...tches on the processor module or processor filler module and insert the module into the empty processor module slot in the server 5 Push the levers together toward the center of the processor module or processor filler module and press the levers firmly against the module to fully seat the module back into the server ...

Page 69: ...ermine your next step If you replaced a faulty processor module or DIMM see Returning the Server to Operation If you installed new processor modules to upgrade the server return to Server Upgrade Process on page 59 7 Determine your next step If you replaced DIMMs see Verify the DIMM on page 80 If you replaced a processor module see Verify the Processor Module on page 70 ...

Page 70: ... shell Do you want to start the SP faultmgmt shell y n y faultmgmtsp fmadm faulty a If the output shows the replacement PM as enabled go to Step 3 b If the output shows the replacement PM as disabled go to Detecting and Managing Faults to clear the fault from the server 3 Verify that the OK LED is lit on the PM and that the Fault LED is not lit See Processor Module LEDs on page 61 4 Verify that th...

Page 71: ...tep Description Links 1 Understand how to configure the DIMMs DIMM Configuration on page 71 2 Locate a faulty DIMM Determine Which DIMM Is Faulty FMA on page 73 Determine Which DIMM Is Faulty LEDs on page 75 DIMM Configuration Fault Messages on page 76 3 Replace a DIMM Remove a DIMM or DIMM Filler Panel on page 76 Install a DIMM on page 78 Verify the DIMM on page 80 Related Information Identifying...

Page 72: ...me type and size of DIMMs that are already in the existing processor modules If you are reviewing this information because you are upgrading the server return to Server Upgrade Process on page 59 DIMM Locations DIMM addresses and consequently their NAC names are based on their location on the processor module motherboard as well as the slot in which the processor is installed For example the full ...

Page 73: ...rmine Which DIMM Is Faulty LEDs on page 75 Install a DIMM on page 78 Server Upgrade Process on page 59 Determine Which DIMM Is Faulty FMA The FMA fmadm faulty command displays current server faults including DIMM failures Type fmadm faulty at the faultmgmtsp prompt start SP faultmgmt shell Are you sure you want to start SP faultmgmt shell y n y faultmgmtsp fmadm faulty ...

Page 74: ... BOB0 CH0 D0 Manufacturer Samsung Name 8192MB DDR3 SDRAM DIMM Part_Number 07042208 M393B1K70DH0 YK0 Revision 04 Serial_Number 00CE02121585C74755 Chassis Manufacturer Oracle Corporation Name T5 chassis Part_Number abcd Serial_Number dbca Description The number of correctable errors associated with this memory module has exceeded acceptable levels Response An attempt will be made to remove the affec...

Page 75: ...uired LED is lit on one of the processor modules See Processor Module LEDs on page 61 3 Remove the PM with the faulty DIMM See Remove a Processor Module or Processor Filler Module on page 63 4 Locate the DIMM Fault Remind button on the front right corner of the motherboard 5 Verify that the DIMM Fault Remind Power LED next to the button is lit An illuminated DIMM Fault Remind Power LED indicates t...

Page 76: ...re checks the memory configuration against the rules described in DIMM Configuration on page 71 If it discovers any faults one or more rule specific messages will be displayed in the POST output indicating the type of configuration fault that has been discovered Related Information DIMM Configuration on page 71 Determine Which DIMM Is Faulty FMA on page 73 Determine Which DIMM Is Faulty LEDs on pa...

Page 77: ...to be replaced See Determine Which DIMM Is Faulty FMA on page 73 or Determine Which DIMM Is Faulty LEDs on page 75 4 Push down on the ejector tabs on each side of the DIMM until the DIMM is released Caution DIMMs and heat sinks on the motherboard might be hot 5 Grasp the top corners of the faulty DIMM and lift it out of its slot 6 Place the DIMM on an antistatic mat 7 Repeat Step 4 through Step 6 ...

Page 78: ...ocess on page 59 Install a DIMM Before beginning this procedure ensure that you are familiar with the information provided in these topics Safety Information on page 49 DIMM Configuration on page 71 1 Take the necessary ESD precautions See Prevent ESD Damage on page 57 2 Ensure that you have removed the processor module if you are replacing a faulty DIMM See Remove a Processor Module or Processor ...

Page 79: ...he DIMM in place If the DIMM does not easily seat into the connector check the DIMM s orientation 8 Determine your next step If you replaced a faulty DIMM go to Step 9 If you are upgrading the server repeat Step 5 through Step 7 until all of the new DIMMs are installed Then go to Server Upgrade Process on page 59 9 Install the PM See Install a Processor Module or Processor Filler Module on page 67...

Page 80: ...cle ILOM Service on page 28 3 Start the faultmgmt shell start SP faultmgmt shell Are you sure you want to start the faultmgmt shell y n y faultmgmtsp 4 Use the fmadm faulty command to determine if the server is operating normally If a fault was detected the server is not operating normally See Diagnostics Process on page 25 If no fault was detected the DIMM was installed successfully Related Infor...

Page 81: ...system and the operating system is not mirrored on another drive If the drive cannot be logically isolated from the online operations of the server If either of these conditions apply to the drive being serviced you must take the server offline shut down the operating system before you replace the drive For the location of the hard drives see Supported Storage Devices on page 18 These topics descr...

Page 82: ...ates the drive s availability for use On Read or write activity is in progress Off Drive is idle and available for use 3 Okay OK Indicates normal operation Blinking indicates that the drive is in use Related Information Determine Which Hard Drive Is Faulty on page 82 Remove a Hard Drive on page 83 Determine Which Hard Drive Is Faulty The following LEDs are lit when a hard drive fault is detected S...

Page 83: ... unless it has an alternate I O path Caution This procedure requires that you handle components that are sensitive to electrostatic discharge This discharge can cause server components to fail 1 Locate the drive in the server that you want to remove See Front Panel Components Service on page 14 for the locations of the drives in the server See Determine Which Hard Drive Is Faulty on page 82 to loc...

Page 84: ...onnected configured unknown c2 w5000cca00a76d1f5 0 disk path connected configured unknown c3 scsi sas connected configured unknown c3 w5000cca00a772bd1 0 disk path connected configured unknown c4 scsi sas connected configured unknown c4 w5000cca00a59b0a9 0 disk path connected configured unknown You must unconfigure any drive whose status is listed as configured as described in Step 3b b Unconfigur...

Page 85: ...o far to the right Doing so can damage the latch 6 Install the replacement drive or a filler tray See Install a Hard Drive on page 85 Related Information Determine Which Hard Drive Is Faulty on page 82 Install a Hard Drive on page 85 Install a Hard Drive 1 Align the replacement drive to the drive slot and slide the drive in until it is seated ...

Page 86: ...s running if you hot plugged the hard drive then no further action is necessary The Oracle Solaris OS will automatically configure the hard drive If you replaced or installed a hard drive in a powered down server then continue with these steps to configure the hard drive 2 If the OS is shut down and the drive you replaced was not the boot device boot the OS Depending on the nature of the replaced ...

Page 87: ...mand to list all drives in the device tree including any drives that are not configured cfgadm al The replacement drive is now listed as configured Example Ap_id Type Receptacle Occupant Condition c2 scsi sas connected configured unknown c2 w5000cca00a76d1f5 0 disk path connected configured unknown c3 scsi sas connected configured unknown c3 w5000cca00a772bd1 0 disk path connected configured unkno...

Page 88: ...Verify the Hard Drive 88 SPARC T5 8 Server Service Manual November 2015 Related Information Determine Which Hard Drive Is Faulty on page 82 Install a Hard Drive on page 85 ...

Page 89: ...r components to fail These topics explain how to service the main module Steps Description Links 1 Replace the main module Remove the Main Module on page 91 Install the Main Module on page 95 2 Remove the main module as part of another component s service operation Remove the Main Module on page 91 3 Install the main module as part of another component s service operation Install the Main Module o...

Page 90: ...mal state System power might be off The SP might be running Steady on System is powered on and is running in its normal operating state No service actions are required Fast blink System is running in standby mode and can be quickly returned to full function Slow blink A normal but transitory activity is taking place Slow blinking might indicate that system diagnostics are running or that the syste...

Page 91: ...ain Module on page 97 Remove the Main Module The main module is a cold service component that can be replaced only after you have powered off the server Caution This procedure requires that you handle components that are sensitive to electrostatic discharge This discharge can cause server components to fail 1 Optional If you are replacing a faulty main module you must back up ILOM configuration se...

Page 92: ...ee Removing Power From the Server on page 53 3 Locate the main module in the server See Front Panel Components Service on page 14 4 Squeeze the green latches together on the two extraction levers and pull the extraction levers out to disengage the main module from the server ...

Page 93: ...odule This will keep the levers from getting damaged when you pull the main module out 7 Remove the main module completely from the server 8 Press down on the green button at the top of the cover to disengage the cover from the main module and push the cover toward the rear of the module as you lift the cover up and away from the chassis 9 Determine your next step ...

Page 94: ...rage Backplane on page 99 System battery Remove the Battery on page 123 System configuration PROM Remove the System Configuration PROM on page 118 System processor card Remove the Service Processor Card on page 110 b If you are replacing a component inside the main module use one of the following links Servicing the Service Processor Card Servicing the Battery Servicing the System Configuration PR...

Page 95: ...l the Main Module 1 Place the cover back onto the main module and slide the cover forward until the latch clicks into place 2 Open the levers so that they are fully open 3 Insert the main module back into its slot in the server until the levers begin to engage ...

Page 96: ...ick into place when the module is fully seated in the server 5 Determine your next step a If you replaced an internal component return to the procedure for that component Verify the Battery on page 126 Verify the Front I O Assembly on page 131 Verify the Service Processor Card on page 114 Verify the Storage Backplane on page 107 Verify the System Configuration PROM on page 120 b If you replaced th...

Page 97: ...iguration instructions b Download the system firmware Follow the firmware download instructions in the Oracle ILOM documentation Note You can load any supported system firmware version including the firmware revision that had been installed prior to the replacement of the main module However Oracle strongly recommends installing the newest version of the system firmware 8 Power on the server See R...

Page 98: ...n y faultmgmtsp 3 Use the fmadm faulty command to determine if the server is operating normally If a fault was detected see Diagnostics Process on page 25 If no fault was detected the main module was installed successfully Related Information Determine if the Main Module Is Faulty on page 91 Install the Main Module on page 95 ...

Page 99: ...on Links 1 Remove a storage backplane Remove a Storage Backplane on page 99 2 Install a storage backplane Install a Storage Backplane on page 103 3 Verify the installation Verify the Storage Backplane on page 107 Related Information Identifying Components Detecting and Managing Faults Preparing for Service Component Service Task Reference on page 23 Returning the Server to Operation Remove a Stora...

Page 100: ... Note the locations of the drives before removing them so that you can install them in their original slots You have to remove only hard drives 0 3 or drives 4 7 depending on which storage backplane you want to replace See Remove a Hard Drive on page 83 4 Remove the main module from the server See Remove the Main Module on page 91 5 Locate the storage backplane that you want to remove No Descripti...

Page 101: ...emove a Storage Backplane Servicing the Storage Backplanes 101 6 Disconnect the two storage backplane cables from the storage backplane that you want to replace No Description 1 Data cable 2 Power cable ...

Page 102: ...015 7 Lift up on the plastic retaining panel for the storage backplane that you want to remove to disengage the plastic panel from the top of the hard drive assembly 8 Push the plastic panel toward the rear of the main module and remove the plastic panel from the main module ...

Page 103: ...ackplane slightly toward the rear of the main module then lift the storage backplane up and remove it from the main module Related Information Install a Storage Backplane on page 103 Verify the Storage Backplane on page 107 Install a Storage Backplane 1 Position the storage backplane in the main module ...

Page 104: ...Install a Storage Backplane 104 SPARC T5 8 Server Service Manual November 2015 2 Lower the storage backplane into place ...

Page 105: ...he plastic retaining panel into place over the storage backplane so that the two notches in the panel slide underneath the two metal mounting studs on the hard drive assembly 4 Press on the press point on the retaining panel to secure it to the top of the hard drive assembly ...

Page 106: ...the main module back into the server See Install the Main Module on page 95 7 Install the hard drives that you removed back into the main module Refer to the notes that you took when removing the hard drives to install them back into their original slots See Install a Hard Drive on page 85 8 Power on the server See Returning the Server to Operation 9 Verify the installation See Verify the Storage ...

Page 107: ...On the Server Oracle ILOM on page 192 2 Log in to Oracle ILOM See Log In to Oracle ILOM Service on page 28 3 Start the faultmgmt shell start SP faultmgmt shell Are you sure you want to start the faultmgmt shell y n y faultmgmtsp 4 Use the fmadm faulty command to determine if the server is operating normally If a fault was detected see Diagnostics Process on page 25 If no fault was detected the sto...

Page 108: ...108 SPARC T5 8 Server Service Manual November 2015 ...

Page 109: ...aulty Determine if the Service Processor Card Is Faulty on page 109 2 Remove the main module so that you can replace the service processor card Remove the Main Module on page 91 3 Replace the service processor card Remove the Service Processor Card on page 110 Install the Service Processor Card on page 112 4 Install the main module after you have replaced the service processor card Install the Mai...

Page 110: ...ee Remove the Service Processor Card on page 110 Related Information Remove the Service Processor Card on page 110 Install the Service Processor Card on page 112 Verify the Service Processor Card on page 114 Remove the Service Processor Card The service processor card is a cold service component that can be replaced only after you have removed the main module Caution This procedure requires that y...

Page 111: ...me 2 Take the necessary ESD precautions See Prevent ESD Damage on page 57 3 Remove the main module from the server See Remove the Main Module on page 91 4 Locate the service processor card on the main module See Main Module Internal Component Locations on page 17 5 Grasp the service processor card by the two grasp points and lift up to disengage the service processor card from the connectors on th...

Page 112: ...ft the service processor card up and away from the motherboard Related Information Determine if the Service Processor Card Is Faulty on page 109 Install the Service Processor Card on page 112 Install the Service Processor Card 1 Take appropriate ESD measures See Prevent ESD Damage on page 57 ...

Page 113: ...turn the server to operation See Returning the Server to Operation If the replacement service processor card detects that the SP firmware is not compatible with the existing host firmware further action is suspended and the following message is delivered over the SER MGT port Unrecognized Chassis This module is installed in an unknown or unsupported chassis You must upgrade the firmware to a newer...

Page 114: ... you stored the configuration information For example load source tftp 129 99 99 99 pathname 9 Verify the installation of the service processor card See Verify the Service Processor Card on page 114 Related Information Remove the Service Processor Card on page 110 Verify the Service Processor Card on page 114 Verify the Service Processor Card 1 Ensure that you have completed the following Applied ...

Page 115: ...he fmadm faulty command to determine if the server is operating normally If a fault was detected see Diagnostics Process on page 25 If no fault was detected the SP was installed successfully 7 Verify that the SP date is correct show SP clock SP clock Targets Properties datetime Tue Jan 01 23 15 59 1970 timezone GMT GMT uptime 0 days 00 01 18 usentpserver disabled Commands cd set show 8 Set the dat...

Page 116: ... the Service Processor Card 116 SPARC T5 8 Server Service Manual November 2015 Related Information Determine if the Service Processor Card Is Faulty on page 109 Install the Service Processor Card on page 112 ...

Page 117: ...onents to fail These topics describe service procedures for the system configuration PROM in the server Steps Description Links 1 Determine if the system configuration PROM is faulty Interpreting LEDs on page 29 2 Remove the main module so that you can replace the system configuration PROM Remove the Main Module on page 91 3 Replace the system configuration PROM Remove the System Configuration PRO...

Page 118: ...configuration PROM see Main Module Internal Component Locations on page 17 Caution This procedure involves handling circuit boards that are extremely sensitive to static electricity Ensure that you follow ESD preventative practices to avoid damaging the circuit boards 1 Remove the main module from the server See Remove the Main Module on page 91 2 Take the necessary ESD precautions See Prevent ESD...

Page 119: ...instructions described in Safety Information on page 49 Caution This procedure requires that you handle components that are sensitive to electrostatic discharge This discharge can cause server components to fail 1 Take all necessary ESD precautions See Prevent ESD Damage on page 57 2 Orient the system configuration PROM properly onto the main module 3 Press down on the system configuration PROM un...

Page 120: ...r presence in the banner verifies that the service processor and the host can read the System Configuration PROM SPARC T5 8 No Keyboard OpenBoot X XX 16256 MB memory available Serial 87304604 Ethernet address Host ID 3 Log in to Oracle ILOM See Log In to Oracle ILOM Service on page 28 4 Start the faultmgmt shell start SP faultmgmt shell Are you sure you want to start the faultmgmt shell y n y faul...

Page 121: ...ress hostid 8534299c ifconfig a lo0 flags 2001000849 UP LOOPBACK RUNNING MULTICAST IPv4 VIRTUAL mtu 8232 index 1 inet 127 0 0 1 netmask ff000000 igb0 flags 201004843 UP BROADCAST RUNNING MULTICAST IPv4 mtu 1500 index 2 inet 10 6 88 150 netmask fffffe00 broadcast 10 6 89 255 ether Related Information Remove the System Configuration PROM on page 118 Install the System Configuration PROM on page 119 ...

Page 122: ...122 SPARC T5 8 Server Service Manual November 2015 ...

Page 123: ...y Remove the Main Module on page 91 3 Replace the battery Remove the Battery on page 123 Install the Battery on page 125 4 Install the main module after you have replaced the battery Install the Main Module on page 95 5 Verify the replacement Verify the Battery on page 126 Related Information Identifying Components Detecting and Managing Faults Preparing for Service Component Service Task Referenc...

Page 124: ...e 57 2 Remove the main module See Remove the Main Module on page 91 3 Locate the system battery in the main module See Main Module Internal Component Locations on page 17 4 Push the top edge of the battery against the spring and lift it out of the carrier Related Information Install the Battery on page 125 Verify the Battery on page 126 ...

Page 125: ...e server using the Network Time Protocol the Oracle ILOM clock will be reset as soon as the server is powered on and connected to the network Otherwise proceed to the next step 3 If the SP is not configured to use NTP use the Oracle ILOM clock command to set the day and time The following example sets the date to June 17 2013 set SP clock datetime 061716192010 show SP clock SP clock Targets Proper...

Page 126: ...er Cords on page 191 Started the system See Power On the Server Oracle ILOM on page 192 2 Log in to Oracle ILOM See Log In to Oracle ILOM Service on page 28 3 Start the faultmgmt shell start SP faultmgmt shell Are you sure you want to start the faultmgmt shell y n y faultmgmtsp 4 Use the fmadm faulty command to determine if the server is operating normally If a fault was detected see Diagnostics P...

Page 127: ...that you can replace the front I O assembly Remove the Main Module on page 91 2 Replace the I O assembly Remove the Front I O Assembly on page 127 Install the Front I O Assembly on page 130 3 Install the main module after you have replaced the I O assembly Install the Main Module on page 95 4 Verify the installation Verify the Front I O Assembly on page 131 Related Information Identifying Componen...

Page 128: ...can cause server components to fail 1 Take the necessary ESD precautions See Prevent ESD Damage on page 57 2 Remove the main module from the server See Remove the Main Module on page 91 3 Locate the front I O assembly on the main module See Main Module Internal Component Locations on page 17 4 Locate the cable that connects the front I O assembly to the motherboard No Description 1 Front I O assem...

Page 129: ...O assembly to the motherboard 7 Gently pull the front I O assembly toward the rear of the main module until the ports at the front of the assembly clear the front of the main module and then remove the front I O assembly from the main module Related Information Install the Front I O Assembly on page 130 Verify the Front I O Assembly on page 131 ...

Page 130: ...ischarges can cause the components to fail 1 Take the necessary ESD precautions See Prevent ESD Damage on page 57 2 Insert the front I O assembly into position in the main module Gently slide the front I O assembly into position with the ports inserted into the port holes in the front of the main module Lower the rear of the front I O assembly so that the captive screw is aligned with the screw ho...

Page 131: ... Remove the Front I O Assembly on page 127 Verify the Front I O Assembly on page 131 Verify the Front I O Assembly 1 Ensure that you have completed the following Applied power to the server See Connect the Power Cords on page 191 Started the system See Power On the Server Oracle ILOM on page 192 2 Log in to Oracle ILOM See Log In to Oracle ILOM Service on page 28 3 Start the faultmgmt shell start ...

Page 132: ...adm faulty command to determine if the server is operating normally If a fault was detected see Diagnostics Process on page 25 If no fault was detected the front I O assembly was installed successfully Related Information Remove the Front I O Assembly on page 127 Install the Front I O Assembly on page 130 ...

Page 133: ...e power sources wired to two separate circuits using a 2 2 configuration Caution This procedure requires that you handle components that are sensitive to electrostatic discharge This discharge can cause failure of server components These topics describe service procedures for the power supplies in the server Steps Description Links 1 Understand the power supply LEDs Power Supply LEDs on page 134 1...

Page 134: ...card is operating normally On PCIe slot has been identified by a user or the button has been pushed 2 OK green Lights when the power supply DC voltage from the PSU to the server is within tolerance 3 AC Present green Lights when the AC voltage is applied to the power supply Related Information Determine Which Power Supply Is Faulty on page 134 Remove a Power Supply on page 135 Verify the Power Sup...

Page 135: ...pply on page 135 Verify the Power Supply on page 138 Remove a Power Supply The power supplies are hot service components that can be replaced at any time For the location of the power supplies see Front Panel Components Service on page 14 Caution This procedure requires that you handle components that are sensitive to electrostatic discharge This discharge can cause server components to fail 1 Loc...

Page 136: ...r Service Manual November 2015 4 At the front of the server squeeze the release latches together on the power supply to be removed then pull the extraction lever toward you to disengage the power supply 5 Pull the power supply out of the server ...

Page 137: ... Supply on page 137 Install a Power Supply The power supply is a hot service component that can be replaced by a customer Caution This procedure requires that you handle components that are sensitive to electrostatic discharge This discharge can cause failure of server components 1 Open the latch on the replacement power supply and align the power supply with the empty bay 2 Slide the power supply...

Page 138: ... Related Information Remove a Power Supply on page 135 Verify the Power Supply on page 138 Verify the Power Supply 1 Verify that the power supply Power OK and AC Present LEDs are lit and that the Fault LED is not lit See Rear Panel Controls and LEDs on page 33 2 Verify that the front and rear Service Required LEDs are not lit See Interpreting LEDs on page 29 3 Log in to Oracle ILOM See Log In to O...

Page 139: ...ultmgmt shell y n y faultmgmtsp 5 Use the fmadm faulty command to determine if the server is operating normally If a fault was detected see Diagnostics Process on page 25 If no fault was detected the power supply was installed successfully Related Information Determine Which Power Supply Is Faulty on page 134 Install a Power Supply on page 137 ...

Page 140: ...140 SPARC T5 8 Server Service Manual November 2015 ...

Page 141: ...ve to electrostatic discharge This discharge can cause server components to fail These topics describe service procedures for the fan modules in the server Steps Description Links 1 Understand the fan module LEDs Fan Module LED on page 141 2 Determine which fan module is faulty Determine Which Fan Module Is Faulty on page 142 3 Replace the fan module Remove a Fan Module on page 143 Install a Fan M...

Page 142: ...le 1 Determine if the System Service Required LEDs are lit on the front panel See Interpreting LEDs on page 29 2 Determine if the Server Fan Fail LED on the front panel is lit See Front Panel Controls and LEDs on page 31 3 From the rear of the server check the fan module LEDs to identify which fan module needs to be replaced See Fan Module LED on page 141 4 Remove the faulty fan module See Remove ...

Page 143: ...he server See Rear Panel Components Service on page 15 for the locations of the fan modules in the server See Determine Which Fan Module Is Faulty on page 142 to locate a faulty fan module 2 Determine if you can remove the fan module with the server running You can remove a fan module with the server running only if one fan module is faulty and if you have a replacement fan module If more than one...

Page 144: ...Remove a Fan Module 144 SPARC T5 8 Server Service Manual November 2015 3 Press the release button to disengage the fan latch 4 Pull out on the fan module to remove it from the server ...

Page 145: ...ng Fan Modules 145 Related Information Determine Which Fan Module Is Faulty on page 142 Install a Fan Module on page 145 Verify the Fan Module on page 146 Install a Fan Module 1 Insert the fan module into the empty fan module slot ...

Page 146: ...ore removing and installing a new fan module see Returning the Server to Operation to power on the server again 4 Verify the fan module functionality See Verify the Fan Module on page 146 Related Information Determine Which Fan Module Is Faulty on page 142 Remove a Fan Module on page 143 Verify the Fan Module on page 146 Verify the Fan Module 1 Ensure that you have completed the following Applied ...

Page 147: ...ations If these conditions are not met perform the actions described in Diagnostics Process on page 25 3 Log in to Oracle ILOM See Log In to Oracle ILOM Service on page 28 4 Start the faultmgmt shell start SP faultmgmt shell Are you sure you want to start the faultmgmt shell y n y faultmgmtsp 5 Use the fmadm faulty command to determine if the server is operating normally If a fault was detected se...

Page 148: ...148 SPARC T5 8 Server Service Manual November 2015 ...

Page 149: ... PCIe card carriers in the server Understanding PCIe Root Complex Connections on page 149 PCIe Card Installation Guidelines on page 153 PCIe Carrier LEDs and Controls on page 154 Determine Which PCIe Card Is Faulty on page 155 Remove a Card Carrier on page 155 Remove a PCIe Card on page 159 Remove a Carrier Extension on page 162 Install a Carrier Extension on page 164 Install a PCIe Card on page 1...

Page 150: ...iagram illustrates the root complex connections between the four processor modules and the 16 PCIe I O slots in a fully populated server Each CPU supports two I O root complexes Each root complex connects to two I O slots through two of five multiplexing switches The port ID values shown in the diagram correspond to the pci values reported in the OpenBoot show devs command output For example PM CP...

Page 151: ...1 6 pci 600 pci 1 pci 0 pci 8 3 6 2 8 pci 640 pci 1 pci 0 pci 8 3 7 3 14 pci 680 pci 1 pci 0 pci 4 3 7 4 16 pci 6c0 pci 1 pci 0 pci 4 Note If you are upgrading the server return to Server Upgrade Process on page 59 Related Information Root Complex Connections Two Processor Modules on page 151 PCIe Card Installation Guidelines on page 153 Server Upgrade Process on page 59 Root Complex Connections T...

Page 152: ...pci 6 0 0 0 2 pci 300 pci 1 pci 0 pci c 0 0 1 3 pci 340 pci 1 pci 0 pci 6 0 0 1 4 pci 340 pci 1 pci 0 pci c 3 2 1 5 pci 600 pci 1 pci 0 pci e 3 2 1 6 pci 600 pci 1 pci 0 pci 8 3 2 2 7 pci 640 pci 1 pci 0 pci e 3 2 2 8 pci 640 pci 1 pci 0 pci 8 0 1 2 9 pci 380 pci 1 pci 0 pci a 0 1 2 10 pci 380 pci 1 pci 0 pci 4 0 1 3 11 pci 3c0 pci 1 pci 0 pci e 0 1 3 12 pci 3c0 pci 1 pci 0 pci 8 3 3 3 13 pci 680 ...

Page 153: ...ded to the card s lowest 8 lanes Note Some PCIe cards are restricted to specific I O slots to meet system cooling requirements Other I O cards provide better performance when installed in particular slots For more information about PCIe slot restrictions for specific devices see I O Slot and Quantity Restrictions in SPARC T5 8 Server Product Notes To achieve optimal load balancing in a fully popul...

Page 154: ...ng the server you are done with the process Related Information Root Complex Connections Four Processor Modules on page 150 Root Complex Connections Two Processor Modules on page 151 Server Upgrade Process on page 59 PCIe Carrier LEDs and Controls No LED Icon Description 1 CAR PCIe carrier 2 Attention Required button ATTN Indicates an operational fault You can also press this button to alert the s...

Page 155: ...tected System Service Required LEDs on the front panel and rear I O module System Rear PCIe LED on the front panel Service Required LED on the faulty PCIe card 1 Determine if the System Service Required LEDs are lit on the front panel or the rear I O module See Interpreting LEDs on page 29 2 Determine if the rear PCIe LED is lit on the front panel See Front Panel Controls and LEDs on page 31 3 Fro...

Page 156: ...itive to electrostatic discharge This discharge can cause server components to fail 1 Take the necessary ESD precautions See Prevent ESD Damage on page 57 2 Start this procedure with the primary domain at the Oracle Solaris prompt Note Removing a PCIe card carrier while the primary domain is at the Open Boot prompt is not supported 3 Locate the PCIe card carrier at the rear of the server See Rear ...

Page 157: ...ci 1 pci 0 pci c PCIE2 EMPTY pci 440 pci 1 pci 0 pci c PCIE4 EMPTY pci 480 pci 1 pci 0 pci 4 PCIE10 EMPTY pci 4c0 pci 1 pci 0 pci 8 PCIE12 EMPTY pci 500 pci 1 pci 0 pci e PCIE5 EMPTY pci 540 pci 1 pci 0 pci e PCIE7 ENABLED pci 580 pci 1 pci 0 pci a PCIE13 EMPTY pci 5c0 pci 1 pci 0 pci 8 PCIE15 EMPTY pci 600 pci 1 pci 0 pci 8 PCIE6 EMPTY pci 640 pci 1 pci 0 pci 8 PCIE8 EMPTY pci 680 pci 1 pci 0 pci...

Page 158: ...Service Manual November 2015 8 Pull the carrier s handle down to disengage the carrier from the card cage 9 Remove the carrier from the server Related Information Determine Which PCIe Card Is Faulty on page 155 Install a Card Carrier on page 168 ...

Page 159: ... to avoid damaging the circuit boards See Prevent ESD Damage on page 57 1 Ensure that you have already taken antistatic measures See Prevent ESD Damage on page 57 2 Determine your next step a If a carrier extension is installed go to Remove a Carrier Extension on page 162 To provide adequate access to the card you must remove the carrier extension if one is installed b If no carrier extension is i...

Page 160: ...Remove a PCIe Card 160 SPARC T5 8 Server Service Manual November 2015 3 Unlatch and open the top cover of the carrier ...

Page 161: ...rrier slot 5 Place the PCIe card on an antistatic mat or into its antistatic packaging 6 Determine your next step If you are replacing the PCIe card go to Install a PCIe Card on page 165 If you are removing the PCIe card return the card carrier to the server See Install a Card Carrier on page 168 Related Information Remove a Carrier Extension on page 162 Install a PCIe Card on page 165 ...

Page 162: ...e replacing the PCIe card you must remove the carrier extension to provide enough clearance to access the card Also if you are reconfiguring the server to remove a double wide carrier you must remove the carrier extension so that you can re install the main carrier 1 Unlock the carrier extension at the front panel ...

Page 163: ...her card go to Install a PCIe Card on page 165 If you are removing the PCIe card from the server go to Step 4 4 Set the carrier extension aside remove the PCIe card and replace the two main PCIe card carriers that you removed to install the carrier extension See Install a Card Carrier on page 168 Caution You must reinstall both main PCIe card carriers to maintain the proper air flow Related Inform...

Page 164: ...that is required for proper cooling 1 Ensure that you have removed two PCIe card carriers from the server See Remove a Card Carrier on page 155 Retain the extra card carrier in a suitable storage space in case you want to remove the carrier extension 2 Install the PCIe card See Install a PCIe Card on page 165 3 Align the rear flange on the carrier extension with the rear wall of the main carrier a...

Page 165: ...city Ensure that you follow ESD preventative practices to avoid damaging the circuit boards See Prevent ESD Damage on page 57 1 Determine your first step a If you are installing a new card and need an empty carrier see Remove a Card Carrier on page 155 b If you are replacing a faulty card and have already removed its carrier from the server go to Step 3 2 Unlatch and swing open the top of the PCIe...

Page 166: ...he PCIe card into the carrier until the bottom connector is firmly seated in the carrier s connector Caution Do not twist or turn the PCIe card as you insert it into the carrier The card s connector must be fully seated in the carrier s slot before you attempt to close the top cover ...

Page 167: ...Install a PCIe Card Servicing PCIe Cards 167 5 Ensure that the card s bulkhead notch is aligned properly with the carrier s bulkhead pin so that the cover enages the pin ...

Page 168: ... server See Install a Card Carrier on page 168 Related Information Install a Card Carrier on page 168 Verify the PCIe Card on page 171 Install a Card Carrier The SPARC T5 8 server supports single wide and double wide card carriers The installation steps are the same for both carrier widths This topic includes illustrations only for the single wide carrier ...

Page 169: ...ng the circuit boards See Prevent ESD Damage on page 57 1 Ensure that the primary domain is at the Oracle Solaris prompt Note Installing a PCIe card carrier while the primary domain is at the Open Boot prompt is not supported 2 Insert the PCIe card carrier into the card cage until it stops Caution Do not press on the PCIe back panel or force the PCIe card carrier into the card cage 3 Close the lat...

Page 170: ...installed a PCIe card in a powered down server power on the server using the instructions provided in Returning the Server to Operation then go to Step 7 6 Use one of the following methods to bring the PCIe card online a Press the Attention button The card s Power OK LED should illuminate indicating that the card is online b At the Oracle Solaris prompt power on and enable the PCIe card Example ho...

Page 171: ...anel and rear I O module are not lit See Interpreting LEDs on page 29 4 Verify that the System PCIe Fault LED on the front panel is not lit See Front Panel Controls and LEDs on page 31 5 Perform one of the following tasks based on your verification results If any of the LEDs are lit see Diagnostics Process on page 25 If none of the LEDs are lit go to Step 6 6 Verify that the green Power LED is lit...

Page 172: ...pci 0 pci e PCIE11 EMPTY pci 400 pci 1 pci 0 pci c PCIE2 EMPTY pci 440 pci 1 pci 0 pci c PCIE4 EMPTY pci 480 pci 1 pci 0 pci 4 PCIE10 EMPTY pci 4c0 pci 1 pci 0 pci 8 PCIE12 EMPTY pci 500 pci 1 pci 0 pci e PCIE5 EMPTY pci 540 pci 1 pci 0 pci e PCIE7 ENABLED pci 580 pci 1 pci 0 pci a PCIE13 EMPTY pci 5c0 pci 1 pci 0 pci 8 PCIE15 EMPTY pci 600 pci 1 pci 0 pci 8 PCIE6 EMPTY pci 640 pci 1 pci 0 pci 8 P...

Page 173: ... for the rear I O module in the server Steps Description Links 1 Determine if the rear I O module is faulty Determine if the Rear I O Module Is Faulty on page 173 2 Replace the rear I O module Remove the Rear I O Module on page 174 Install the Rear I O Module on page 176 3 Verify the replacement Verify the Rear I O Module on page 178 Related Information Identifying Components Detecting and Managin...

Page 174: ...be replaced only after you have powered off the server For the location of the rear I O module see Rear I O Module Port Locations on page 21 Caution This procedure requires that you handle components that are sensitive to electrostatic discharge This discharge can cause server components to fail 1 Take the necessary ESD precautions See Prevent ESD Damage on page 57 2 Remove all of the power from t...

Page 175: ...Remove the Rear I O Module Servicing the Rear I O Module 175 5 Press the green buttons on the rear I O module ejection levers and spread the levers open to eject the rear I O module ...

Page 176: ...odule away from the server to remove it Related Information Determine if the Rear I O Module Is Faulty on page 173 Install the Rear I O Module on page 176 Verify the Rear I O Module on page 178 Install the Rear I O Module 1 Take the necessary ESD precautions See Prevent ESD Damage on page 57 ...

Page 177: ...Install the Rear I O Module Servicing the Rear I O Module 177 2 With the levers in the extended position insert the rear I O module into the slot at the rear of the server ...

Page 178: ... rear I O module 5 Power on the server See Returning the Server to Operation 6 Verify the installation See Verify the Rear I O Module on page 178 Related Information Determine if the Rear I O Module Is Faulty on page 173 Remove the Rear I O Module on page 174 Verify the Rear I O Module on page 178 Verify the Rear I O Module 1 Ensure that you have completed the following Applied power to the server...

Page 179: ...ervice on page 28 4 Start the faultmgmt shell start SP faultmgmt shell Are you sure you want to start the faultmgmt shell y n y faultmgmtsp 5 Use the fmadm faulty command to determine if the server is operating normally If a fault was detected see Diagnostics Process on page 25 If no fault was detected the rear I O module was installed successfully Related Information Determine if the Rear I O Mod...

Page 180: ...180 SPARC T5 8 Server Service Manual November 2015 ...

Page 181: ...r Steps Description Links 1 Determine if the rear chassis subassembly is faulty using the system software Determine if the Rear Chassis Subassembly Is Faulty on page 181 2 If the rear chassis subassembly is faulty shut the server down Removing Power From the Server on page 53 3 Remove the rear chassis subassembly Remove the Rear Chassis Subassembly on page 182 4 Install the rear chassis subassembl...

Page 182: ...5 Remove the Rear Chassis Subassembly The rear chassis subassembly is a cold service component that can be replaced only by qualified service personnel For its location see Rear Panel Components Service on page 15 Caution This procedure requires that you handle components that are sensitive to electrostatic discharge This discharge can cause server components to fail Caution Do not attempt to remo...

Page 183: ...ower Supply on page 135 5 At the rear of the server remove all of these components Component Link Cables Ensure that you lable the cables Fan modules Remove a Fan Module on page 143 PCIe carriers Remove a Card Carrier on page 155 Rear I O module Remove the Rear I O Module on page 174 Caution Do not attempt to remove the chassis alone without the aide of another person or a mechanical lift 6 Determ...

Page 184: ...bassembly 184 SPARC T5 8 Server Service Manual November 2015 7 Place a mechanical lift under the chassis and remove the screws that fasten it to the rack 8 Remove the chassis from the rack and place it on an appropriate surface ...

Page 185: ...ar chassis subassembly out and away from the chassis 10 Slide the rear chassis subassembly out and away from the chassis Related Information Determine if the Rear Chassis Subassembly Is Faulty on page 181 Install the Rear Chassis Subassembly on page 185 Install the Rear Chassis Subassembly 1 If you have not already done so remove the faulty rear chassis subassembly See Remove the Rear Chassis Suba...

Page 186: ...2 Slide the new rear chassis subassembly into the chassis and then using a Phillips screwdriver tighten the five green screws to secure the rear chassis subassembly in the server 3 Determine your next step If you have another person to assist you go to Step 5 If you are alone go to Step 4 ...

Page 187: ...secure the chassis with the four screws that were removed 6 At the rear of the server install these components Component Link Rear I O module Install the Rear I O Module on page 176 PCIe carriers Install a Card Carrier on page 168 Fan modules Install a Fan Module on page 145 Cables According to their lables 7 At the front of the server install these components ...

Page 188: ...hassis Subassembly on page 182 Verify the Rear Chassis Subassembly on page 188 Verify the Rear Chassis Subassembly 1 Ensure that you have completed the following Applied power to the server See Connect the Power Cords on page 191 Started the system See Power On the Server Oracle ILOM on page 192 2 Log in to Oracle ILOM See Log In to Oracle ILOM Service on page 28 3 Start the faultmgmt shell start ...

Page 189: ...y the Rear Chassis Subassembly Servicing the Rear Chassis Subassembly 189 Related Information Determine if the Rear Chassis Subassembly Is Faulty on page 181 Install the Rear Chassis Subassembly on page 185 ...

Page 190: ...190 SPARC T5 8 Server Service Manual November 2015 ...

Page 191: ... that you were performing if applicable or return to the upgrade process Component Service Categories on page 52 Server Upgrade Process on page 59 Related Information Identifying Components Detecting and Managing Faults Preparing for Service Component Service Task Reference on page 23 Connect the Power Cords 1 Connect a terminal device if necessary 2 Find the AC connectors on the rear of the serve...

Page 192: ...cle ILOM Service on page 28 2 Determine if the system has been started show System power_state If the system is running go to the verification task for the component that you replaced If the system is not running go to Step 3 3 Start the system start System 4 Determine your next step If you are servicing a component return to the procedure for that component If you are upgrading the server return ...

Page 193: ...ols and LEDs on page 31 for the location of the Power button 3 Ensure that the system has been started show System power_state If the system is running go to the verification task for the component If the system is not running go to Step 3 4 Start the system start System 5 Determine your next step If you are servicing a component return to the procedure for that component If you are upgrading the ...

Page 194: ...194 SPARC T5 8 Server Service Manual November 2015 ...

Page 195: ...le BMC Baseboard management controller BOB Memory buffer on board C chassis For servers refers to the server enclosure For server modules refers to the modular system enclosure CMA Cable management assembly CMM Chassis monitoring module server modules only The CMM is the service processor in the modular system that contains server modules Oracle ILOM runs on the CMM providing lights out management...

Page 196: ...n NEMs See NEM FRU Field replaceable unit H HBA Host bus adapter host The part of the server or server module with the CPU and other hardware that runs the Oracle Solaris OS and other applications The term host is used to distinguish the primary computer from the SP See SP hot pluggable Describes a component that can be replaced with power applied but the component must be prepared for removal hot...

Page 197: ...nly The modular system provides Oracle ILOM through its CMM MSGID Message identifier N name space Top level Oracle ILOM target NEBS Network Equipment Building System Netra products only NEM Network express module server modules only NEMs provide Ethernet and SAS connectivity to storage modules NET MGT Network management port An Ethernet port on the server SP the server module SP and the CMM NIC Ne...

Page 198: ... O features such as Gigabit Ethernet and Fibre Channel POST Power on self test PROM Programmable read only memory PSH Predictive self healing R REM RAID expansion module server modules only Sometimes referred to as an HBA See HBA Supports the creation of RAID volumes on drives S SAS Serial attached SCSI SCC System configuration chip SER MGT Serial management port A serial port on the server SP the...

Page 199: ... Telecommunications Industry Association Netra products only Tma Maximum ambient temperature U U S NEC United States National Electrical Code UCP Universal connector port UI User interface UL Underwriters Laboratory Inc UTC Coordinated Universal Time UUID Universal unique identifier W WWN World wide name A unique number that identifies a SAS target ...

Page 200: ...200 SPARC T5 8 Server Service Manual November 2015 ...

Page 201: ... POST runs 39 D diag_level parameter 35 diag_mode parameter 35 diag_trigger parameter 35 diag_verbosity parameter 35 DIMMs capacities 71 configuration 71 failures 73 Fault Remind button 75 faulty 66 general error message 76 installing 78 locating faulty using DIMM Fault Remind button 75 usingfmadm faulty command 73 NAC name 23 orientation 79 removing 76 verifying functionality 80 dmesg command 45 ...

Page 202: ...ront I O assembly 130 hard drives 85 main module 95 PCIe carriers 168 power supplies 137 processor modules 67 rear chassis subassembly 185 rear I O module 176 SP card 112 storage backplanes 103 system configuration PROM 119 K Knowledge Base articles 41 L LEDs front panel 31 NET Link and Activity 33 Net Management Link and Activity 33 Net Management Speed 33 NET Speed 33 processor modules 61 rear I...

Page 203: ...g 15 NAC name 24 numbering 153 removing 155 PCIe slots configuration reference 153 port ID values 150 ports front panel 14 rear I O module 21 POST configuration examples 39 configuring 39 overview 35 running in Diag Mode 40 power cords connecting to server 191 power supplies installing 137 locating 14 locating faulty 134 NAC name 24 removing 135 verifying functionality 138 powering off emergency s...

Page 204: ...name 24 server connecting power cords 191 powering off emergency shutdown 55 gracefully with power button 55 using SP command 54 powering on using power button 193 usingstart SYS command 192 server upgrade overview 59 Service Required LED storage 82 SP installing card 112 LEDs 33 locating 17 locating faulty 109 NAC name 24 removing 110 verifying functionality 114 storage device tree 86 hot service...

Page 205: ... ports 21 using Oracle VTS 25 27 UUID 41 V var adm messages file 46 verifying functionality DIMMs 80 fan modules 146 hard drives 86 PCIe cards 171 power supplies 138 processor modules 70 rear I O module 178 SP 114 video port 21 viewing system message log files 46 ...

Page 206: ...206 SPARC T5 8 Server Service Manual November 2015 ...

Search results

Summary of Contents for SPARC T5-8

Reviews:

Related manuals for SPARC T5-8

Brands by name

Popular brands

Oracle SPARC T5-8, Service Manual

Search results

Summary of Contents for SPARC T5-8

Reviews:

Related manuals for SPARC T5-8

Brands by name

Popular brands