manualshive.com logo in svg

Fujitsu PRIMERGY BX924 S4, Upgrade And Maintenance Manual, Page: 110 / 286

Share
Download
Fujitsu PRIMERGY BX924 S4 Upgrade And Maintenance Manual Download Page 110

110

Upgrade and Maintenance Manual

 BX924 

S4

Hard disk drives / Solid state drives

©

 c

o

gn

ita

s.

 G

e

se

lls

ch

ft fü

r T

ech

n

ik

-D

o

ku

m

en

ta

tio

n

 m

b

H

 2

0

13

 

Pf

a

d: 

C

:\P

ro

g

ra

m

m

e

\F

C

T

\ti

m

_a

pp

\ti

m

_l

oca

l\w

or

k\PI

C

H

O

L\

O

B

J_

D

O

KU

-2

0

56

5-

00

1.

fm

Removing SSD 0 from the mounting frame

Figure 26: Removing the screws from the mounting frame

Ê

Remove the 4 screws from the mounting frame.

Ê

Slide the drive from the mounting frame (1) in the direction of the arrow. 

Summary of Contents for PRIMERGY BX924 S4

Page 1: ...Upgrade and Maintenance Manual English PRIMERGY BX924 S4 Server Blade Upgrade and Maintenance Manual Edition February 2014 ...

Page 2: ...on mbH www cognitas de Copyright 2014 Fujitsu Technology Solutions GmbH All rights reserved Delivery subject to availability right of technical modifications reserved All hardware and software names used are trademarks of their respective manufacturers The contents of this manual may be revised without prior notice Fujitsu assumes no liability for damages to third party copyrights or other rights ...

Page 3: ...of these components beyond their operating life may result in electrolyte leakage or depletion potentially causing emission of foul odor or smoke As a guideline in a normal office environment 25 C operating life is not expected to be reached within the maintenance support period 5 years However operating life may be reached more quickly if for example the product is used in a hot environment The c...

Page 4: ...notice follows the guidelines of Voltage Dip Immunity of Personal Computer issued by JEITA the Japan Electronics and Information Technology Industries Association Technology controlled by the Foreign Exchange and Foreign Trade Control Law of Japan Documents produced by Fujitsu may contain technology controlled by the Foreign Exchange and Foreign Trade Control Law of Japan Documents which contain s...

Page 5: ...4 Upgrade and Maintenance Manual Only for the Japanese market I Although described in this manual some sections do not apply to the Japanese market These options and routines include CSS Customer Self Service ...

Page 6: ...Upgrade and Maintenance Manual BX924 S4 cognitas Gesellschft für Technik Dokumentation mbH 2013 Pfad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20557 001 fm ...

Page 7: ...onformity 36 3 3 FCC Class A Compliance Statement 37 3 4 Environmental protection 38 4 Basic hardware procedures 41 4 1 Using diagnostics information 41 4 1 1 Accessing the management blade web interface 41 4 1 2 Locating the defective server blade 43 4 1 3 Determining the error class 44 4 1 3 1 Global Error indicator 45 4 1 3 2 Customer Self Service CSS indicator 46 4 1 4 Locating the defective c...

Page 8: ...erver blade status 63 5 1 2 1 Checking the server blade status via management blade web interface 63 5 1 2 2 Checking the server blade status via iRMC 64 5 1 3 Saving BIOS settings 65 5 1 4 Saving iRMC settings 65 5 1 5 Connecting virtual media to the managed server blade 65 5 1 6 Disabling BitLocker functionality 66 5 1 7 Disabling boot watchdog functionality 67 5 1 7 1 Viewing boot watchdog sett...

Page 9: ...r functionality 91 5 2 15 Performing a RAID array rebuild 92 5 2 16 Looking up changed MAC WWN addresses 92 5 2 16 1 Looking up MAC addresses 92 5 2 16 2 Looking up WWN addresses 93 5 2 17 Using the Chassis ID Prom Tool 94 5 2 18 Configuring LAN teaming 94 5 2 18 1 After replacing upgrading LAN CNA controllers 94 5 2 18 2 After replacing the server blade 95 5 2 19 Switching off the ID indicator 95...

Page 10: ...ing the SAS PCH backplane 118 6 6 5 Concluding steps 120 7 Mezzanine cards 121 7 1 Basic information 122 7 1 1 Installing riser cards 124 7 1 2 Removing riser cards 126 7 1 3 Population rules for mezzanine cards 128 7 2 Installing mezzanine cards 134 7 2 1 Required tools 134 7 2 2 Preliminary steps 134 7 2 3 Installing a mezzanine card 135 7 2 4 Concluding steps 139 7 3 Removing mezzanine cards 14...

Page 11: ...iminary steps 164 8 3 3 Removing a memory module 165 8 3 4 Concluding steps 166 8 4 Replacing memory modules 167 8 4 1 Required tools 167 8 4 2 Preliminary steps 167 8 4 3 Removing a memory module 168 8 4 4 Installing a memory module 168 8 4 5 Concluding steps 169 8 5 Handling of memory air cowls 170 8 5 1 Installing the air cowls 170 8 5 2 Removing the air cowls 171 9 Processors 173 9 1 Basic inf...

Page 12: ...Installing the heat sink for CPU 1 214 9 5 3 3 Installing the heat sink for CPU 2 217 9 5 4 Removing processor heat sinks 219 9 5 5 Removing the dummy heat sink of CPU 2 220 9 5 6 Replacing processor heat sinks 221 9 5 6 1 Removing the processor heat sink 221 9 5 6 2 Applying thermal paste 221 9 5 6 3 Installing the processor heat sink 221 9 5 7 Concluding steps 221 9 6 Applying thermal paste 222 ...

Page 13: ...ding steps 247 10 3 2 Removing the TPM 249 10 3 2 1 Required tools 249 10 3 2 2 Preliminary steps 251 10 3 2 3 Removing the TPM 252 10 3 2 4 Concluding steps 253 10 3 3 Replacing the TPM 254 10 3 3 1 Required tools 254 10 3 3 2 Preliminary steps 255 10 3 3 3 Removing the TPM 256 10 3 3 4 Re installing the TPM 256 10 3 3 5 Concluding steps 256 10 4 Onboard SAS Enabling Key SCU key 258 10 4 1 Requir...

Page 14: ... Mechanical overview 275 12 1 1 Server blade front 275 12 1 2 Server blade interior 276 12 2 Configuration tables 277 12 2 1 Memory configuration table 277 12 2 2 Mezzanine card configuration table 277 12 3 Connectors and indicators 278 12 3 1 Connectors and indicators on the system board 278 12 3 1 1 Onboard connectors 278 12 3 1 2 Onboard settings 279 12 3 1 3 Onboard indicators and controls 280...

Page 15: ... S4 Upgrade and Maintenance Manual 15 Version history Issue number Reason for update 1 0 October 2013 Initial release 2 0 December 2013 Changing cross references in chapter 5 3 0 February 2014 Chapter 5 2 15 ...

Page 16: ...Upgrade and Maintenance Manual BX924 S4 cognitas Gesellschft für Technik Dokumentation mbH 2013 Pfad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20559 001 fm ...

Page 17: ...ing defective hardware components This manual focuses on on site maintenance tasks It is recommended to prepare each service assignment following remote diagnostics procedures as described in the ServerView Suite Local Service Concept LSC manual see section Documents you need at hand on page 26 V CAUTION The document at hand comprises procedures of a wide range of complexity Check the profile of q...

Page 18: ...the user Quotation marks indicate names of chapters and terms that are being emphasized Ê describes activities that must be performed in the order shown Abc indicates keys on the keyboard V CAUTION Pay particular attention to texts marked with this symbol Failure to observe this warning may endanger your life destroy the system or lead to the loss of data I indicates additional information notes a...

Page 19: ...the Fujitsu ServerView Suite management software to prepare hardware expansions ServerView Suite documentation is available online at http manuals ts fujitsu com http jp fujitsu com platform server primergy manual for the Japanese market Please refer to the following ServerView Suite topics Operation Virtualization Maintenance I For the latest information on hardware options refer to your server s...

Page 20: ... to the defective component The Indicate CSS button enables the indicator next to the defective component even if the server has been switched off and disconnected from the mains For further information please refer to sections Using diagnostics information on page 41 and Connectors and indicators on the front panel on page 283 If the defective component is a customer replaceable unit included in ...

Page 21: ...ries significantly Procedures have been assigned to one of two unit categories indicating the level of difficulty and required qualification At the beginning of each procedure the involved unit type is indicated by one of the symbols introduced in this section I Please ask your local Fujitsu service center for more detailed information ...

Page 22: ...es and diagnostic indicators on the front panel and system board will report defective Upgrade and Repair Units as customer replaceable CSS components Upgrade and repair procedures involve shutting down and opening the server V CAUTION The device may be seriously damaged or cause damage if it is opened without authorization or if repairs are attempted by unauthorized and untrained personnel Compon...

Page 23: ...edures involving Field Replaceable Units must be performed exclusively by Fujitsu service personnel or technicians trained by Fujitsu Please note that unauthorized interference with the system will void the warranty and exempt the manufacturer from all liability Components that are handled as Field Replaceable Units Processors replacements SAS SATA backplanes Trusted Platform Module TPM USB Flash ...

Page 24: ...cantly Software tasks necessary before maintenance are described in section Starting the maintenance task on page 63 System unit removal disassembly yes Making the server blade available removing the server blade from the system unit Transport no Transporting the server blade to the service table where required depends on local customer conditions Maintenance procedures yes Maintenance procedures ...

Page 25: ...kup drives optical disk drives chassis M3 x 4 5 mm silver C26192 Y10 C67 Phillips PH2 No 2 hexagonal cross SW5 PZ2 System board M3 x 6 mm silver C26192 Y10 C68 Phillips PH2 No 2 hexagonal cross SW5 PZ2 Backup drives with UNC thread UNC 6 32 x 4 76 mm black C26192 Y10 C75 Phillips PH0 No 0 2 5 inch HDDs SSDs M3 x 3 5 mm Wafer head screw silver C26192 Y10 C102 TPM bit insert Dedicated TPM screw driv...

Page 26: ...e overview below I Ensure to store all printed manuals enclosed with your server in a save place for future reference Unless stated otherwise all manuals are available online at http manuals ts fujitsu com under Industry standard servers For the Japanese market please use the following address http jp fujitsu com platform server primergy manual Phillips PH1 No 1 UFM nylon screw M3 x 4 5 mm white A...

Page 27: ...basic maintenance tasks Software documentation PRIMERGY BX900 Blade Server Systems ServerView Management Blade S1 user interface description PRIMERGY BX400 Blade Server Systems ServerView Management Blade S1 user interface description ServerView Suite Local Service Concept LSC user guide ServerView Operations Manager Server Management user guide iRMC S4 integrated Remote Management Controller user...

Page 28: ...contact information available online or as a printed copy Service Desk leaflet サポート サービス for the Japanese market Additional documentation iRMC S4 user guide available online RAID documentation available online at http manuals ts fujitsu com under Industry standard servers Expansion Cards Storage Adapters For the Japanese market please use the following address http jp fujitsu com platform server p...

Page 29: ... a person who is trained to install the server including hardware and software Repairs to the device that do not relate to CSS failures shall be performed by service personnel Please note that unauthorized interference with the system will void the warranty and exempt the manufacturer from all liability Any failure to observe the guidelines in this manual and any improper repairs could expose the ...

Page 30: ...he requirements for non industrial power supply networks for type B connectors The unit automatically adjusts itself to a mains voltage in a range of 100 VAC to 240 VAC Ensure that the local mains voltage lies within these limits This device must only be connected to properly grounded power outlets or connected to the grounded rack internal power distribution system with tested and approved power ...

Page 31: ...casing is completely assembled and the rear covers for the installation slots have been fitted electric shock cooling fire protection interference suppression Only install system expansions that satisfy the requirements and rules governing safety and electromagnetic compatibility and those relating to telecommunication terminals If you install other expansions they may damage the system or violate...

Page 32: ...he edges of the circuit boards Install the screw removed during installation detaching internal options in former device position To use a screw of the different kind can cause a breakdown of equipment The installation indicated on this document is sometimes changed to the kind of possible options without notice Batteries Incorrect replacement of batteries may lead to a risk of explosion The batte...

Page 33: ...ve equipment damage and can fly out of the device danger of injury particularly to uncovered body parts such as the face or neck High humidity and airborne dust levels are to be avoided Electric shocks and or server failures may be caused by liquids such as water or metallic items such as paper clips entering a drive Shocks and vibrations are also to be avoided Do not insert any objects other than...

Page 34: ...ipe the CD DVD BD with a soft dry cloth then let it air dry Do not dry the CD DVD BD using devices such as a hair dryer To avoid dust damage and deformation keep the CD DVD BD in its case whenever it is not in use Do not store CDs DVDs BDs at high temperatures Areas exposed to prolonged direct sunlight or near heating appliances are to be avoided I You can prevent damage from the optical disk driv...

Page 35: ...c electricity To ensure reliable protection you must wear an earthing band on your wrist when working with this type of module and connect it to an unpainted non conducting metal part of the system Any devices or tools that are used must be free of electrostatic charge Wear a suitable grounding cable that connects you to the external chassis of the system unit Always hold components with ESDs at t...

Page 36: ...ual for the corresponding rack The technical manual is supplied with the corresponding rack When installing the rack make sure that the anti tilt protection is correctly fitted For safety reasons no more than one unit may be removed from the rack at any one time during installation and maintenance work If several units are simultaneously removed from the rack there is a risk that the rack could ti...

Page 37: ... harmful interference to radio or television reception which can be determined by turning the equipment off and on the user is encouraged to try to correct the interference by one or more of the following measures Reorient or relocate the receiving antenna Increase the separation between equipment and the receiver Connect the equipment into an outlet on a circuit different from that to which the r...

Page 38: ...market Energy saving information Devices that do not need to be constantly switched on should be switched off until they are needed as well as during long breaks and after completion of work Packaging information This packaging information doesn t apply to the Japanese market Do not throw away the packaging You may need it later for transporting the system If possible the equipment should only be ...

Page 39: ...ing used devices manual via your local Fujitsu branch or from our recycling center in Paderborn Fujitsu Technology Solutions Recycling Center D 33106 Paderborn Tel 49 5251 525 1410 Fax 49 5251 525 32 1410 The device must not be disposed of with domestic waste This device is labeled in compliance with European directive 2002 96 EC on waste electrical and electronic equipment WEEE This directive set...

Page 40: ...40 Upgrade and Maintenance Manual BX924 S4 Important information cognitas Gesellschft für Technik Dokumentation mbH 2013 Pfad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20562 001 fm ...

Page 41: ...al Service Concept LSC manual Please contact your local Fujitsu customer service partner for details on the service concept and on how to order expansion kits or spare parts Use the Fujitsu Illustrated Spares Catalog to identify the required spare part and obtain technical data and order information Illustrated Spares catalogs are available online at http manuals ts fujitsu com illustrated_spares ...

Page 42: ...ment Agent Administrative URL to login to the management blade web interface For further information refer to the PRIMERGY BX900 Blade Server Systems ServerView Management Blade S1 User Interface Description or PRIMERGY BX400 Blade Server Systems ServerView Management Blade S1 User Interface Description I If you don t know the Management Agent Administrative URL proceed as follows Ê Connect the FS...

Page 43: ...ement blade web interface or use the ServerView Operation Manager user interface to switch on the ID indicators 2 I For further information refer to the ServerView Suite Local Service Concept LSC manual Ê When using management blade web interface to toggle the ID indicator open the Components System Server Blades Server Blade x menu for the desired server blade and press the Locate button in the s...

Page 44: ...ice Concept LSC allows you to identify defective server blade components Failure events are assigned to one of two error classes Global Error events that need to be resolved by maintenance personnel Customer Self Service CSS error events that may be resolved by operating personnel Global Error and CSS LEDs indicate if the defective component is a customer replaceable unit or if maintenance personn...

Page 45: ...ndicator on the front of the server blade Ê For further diagnostics proceed as follows Indicator Status Description Global error indicator off no critical event non CSS component orange on prefailure detected non CSS component requires precautionary service intervention orange flashing non CSS component failure or software agent related error requires service intervention ...

Page 46: ...g SEL as described in section Viewing the SEL on page 87 Software agent related errors Check the ServerView System Monitor available on Windows or Linux based server blades with ServerView agents installed I For further information please refer to the ServerView System Monitor user guide 4 1 3 2 Customer Self Service CSS indicator Figure 5 CSS error indicator in front panel Ê Check the CSS indicat...

Page 47: ...de and Maintenance Manual 47 Basic hardware procedures Indicator Status Description CSS indicator off no critical event CSS component yellow on prefailure detected CSS component yellow flashing CSS component failure ...

Page 48: ...verView Suite Local Service Concept LSC manual Local diagnostic indicators on the system board Figure 6 Onboard indicators and Indicate CSS button Using the Indicate CSS button Ê Shut down the server blade as described in section Shutting down the server blade on page 52 Ê Remove the server blade from the system unit as described in section Removing the server blade from the system unit on page 54...

Page 49: ...andling of memory air cowls on page 170 Indicator Status Description A Mezzanine card 1 off Mezzanine card 1 operational orange on Mezzanine card 1 failure B Memory off memory module operational orange on memory module failure C Fabric 3 4 off Fabric 3 4 no link green flashing Fabric 3 4 link active green on Fabric 3 4 link established D Fabric 2 off Fabric 2 no link green flashing Fabric 2 link a...

Page 50: ...ce the part see section Determining the error class on page 44 If the system has been powered off to replace a non hot plug unit a system of PRIMERGY diagnostics indicators guides you to the faulty component G Global Error off No critical event orange flashing Error detected requires service intervention orange on Prefailure event detected requires service intervention H Mezzanine card 2 off Mezza...

Page 51: ...ocking To unlock and open the rack proceed as follows Ê Insert and turn the key counter clockwise by 180 degrees If applicable turn the door knob counter clockwise by 180 degrees Ê Open the left hand door first then the right hand door I For further information refer to the PRIMECENTER assembly guide available online at http manuals ts fujitsu com 4 2 2 Opening the rack door of a PRIMECENTER M1 ra...

Page 52: ...l be shut down and put offline Ê Terminate all applications Ê In case of Multipath I O environments please refer to section Note on server maintenance in a Multipath I O environment on page 70 Figure 7 Power button on the front panel Ê Shut down the server blade by pressing the On Off button 1 The power indicator 2 turns off I If the system is running an ACPI compliant operating system pressing th...

Page 53: ...N For safety information please refer to chapter Important information on page 29 4 4 1 Preliminary steps Ê If applicable open the rack door as described in section Opening the rack door on page 51 Ê Shut down and power off the server blade as described in section Shutting down the server blade on page 52 ...

Page 54: ... FCT tim_app tim_local work PICHOL OBJ_DOKU 20563 001 fm 4 4 2 Removing the server blade from the system unit Figure 8 Removing the server blade from the system unit Ê Push the release lever 1 up slightly to unlock the ejection lever 2 Ê Swivel the ejection lever down until it is horizontal Ê Pull the server blade out of the system unit 햲 햳 ...

Page 55: ...er blade V CAUTION For safety information please refer to chapter Important information on page 29 Removing the cover Figure 9 Removing the cover Ê Push the locking lever in the direction of the arrow 1 Ê Slide the housing cover backwards as far as possible 2 Ê Take off the housing cover 0 ...

Page 56: ... order to comply with applicable EMC regulations regulations on electromagnetic compatibility and satisfy cooling requirements the PRIMERGY BX924 S4 server blade must not run while the cover is removed For further safety information please refer to chapter Important information on page 29 Figure 10 Closing the server blade Ê Place the cover on the server blade housing such that it lies flush with ...

Page 57: ...sensitive devices in section Safety instructions on page 29 Note the population rules for power supply units and fan modules to ensure sufficient cooling of the system You will find more detailed information on this in the operating manual for the system unit Installing the Server Blade V CAUTION Follow the safety instructions and information in section Modules with Electrostatic Sensitive Devices...

Page 58: ...procedures cognitas Gesellschft für Technik Dokumentation mbH 2013 Pfad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20563 001 fm Figure 11 Installing the Server Blade Ê Open the release lever Ê Push the server blade as far as possible into the slot ...

Page 59: ...the release lever up until it engages It is not a breakdown though the power supply of the server blade intermittently repeats turning on cutting when Automatic inventory collection Automatic Inventory Retrieval of the management blade is set to Automatic default value after the installation of the server blade ...

Page 60: ...r to comply with applicable EMC regulations regulations on electromagnetic compatibility and satisfy cooling requirements the PRIMERGY BX924 S4 server blade must not run while the cover is removed Follow the safety instructions in chapter Important information on page 29 Figure 13 Power button on the front panel Ê Press the On Off button 1 to start up the server blade Ê Ensure that the power on in...

Page 61: ... into operation Resetting the boot retry counter on page 82 Verifying the system time settings on page 86 Viewing and clearing the System Event Log SEL on page 87 Updating the NIC configuration file in a Linux environment on page 89 Enabling BitLocker functionality on page 91 Performing a RAID array rebuild on page 92 Looking up changed MAC WWN addresses on page 92 ...

Page 62: ...s locking To close and lock the rack proceed as follows Ê Close the right hand door first then the left hand door Ê Insert and turn the key clockwise by 180 degrees If applicable turn the door knob clockwise by 180 degrees I For further information refer to the PRIMECENTER assembly guide available online 4 10 2 Closing the rack door of a PRIMECENTER M1 rack The new PRIMECENTER M1 rack is equipped ...

Page 63: ...the management blade web interface Ê Open the Components System Server Blades Server Blade x menu for the desired server blade Ê Open the Configuration tab Ê Scroll to the iRMC Address Configuration box Ê Make sure that Management LAN is selected in the LAN Port list box Ê Click the Video Redirection button in the status frame of the server blade menu I For detailed information on iRMC Advanced Vi...

Page 64: ...sired server blade Ê Scroll down in the Information tab to get status information about the following components Operating System BIOS Version Processors Memory modules Network components on board CNA controller and mezzanine cards Operating System Ê Open the Event Log tab to read the system event log entries 5 1 2 2 Checking the server blade status via iRMC Ê Enter the management blade web interf...

Page 65: ...supported Ê Open the iRMC S4 Network Settings Ethernet and confirm IP Address Subnet Mask and Gateway Ê Open the iRMC S4 Save Configuration Ê Click the Save All button in section of Save iRMC S4 Firmware settings in ServerView WinSCU XML format Ê Store the backup file in the system of the FST 5 1 5 Connecting virtual media to the managed server blade The management blade web interface uses the iRM...

Page 66: ...ess has been modified from its original state Disabling BitLocker Drive Encryption is a temporary method for removing BitLocker protection without decrypting the drive Windows is installed on Disable BitLocker before modifying the server s hardware configuration or startup files Enable BitLocker again after the maintenance procedure is complete V CAUTION With BitLocker features enabled modifying t...

Page 67: ...osoft Knowledge Base Fujitsu service partners will find additional information also available in Japanese on the Fujitsu Extranet web pages 5 1 7 Disabling boot watchdog functionality The boot watchdog determines whether the server blade boots within a preset time frame If the watchdog timer expires the system will automatically reboot 5 1 7 1 Viewing boot watchdog settings To view boot watchdog s...

Page 68: ...ng boot watchdog settings in the BIOS Ê Open a virtual console for your server blade as described in section Launching a video redirection to a server blade on page 63 Ê Switch on or restart your server blade Ê As soon as the startup screen appears press the F2 function key to enter the BIOS Ê Select the Server Mgmt menu Ê Under Boot Watchdog set the Action setting to Continue Ê Save your changes ...

Page 69: ...up software solution it may be necessary to disable or delete the backup drive from the backup software drive list before starting the maintenance task This is the case for the following backup software solution BackupExec I Procedures may differ depending on the backup software For details refer to the dedicated documentation provided separately Further information on suitable backup software sol...

Page 70: ...istrator to disable the FC iSCSI CNA SAS connections to the server blade via FC LAN SAS connection blade command After executing work do the reconnection Ask the system administrator Continue as follows Ê If performing an offline driver update first of all prepare the ServerView Update DVD Ê Download the latest ServerView Update DVD image from the Fujitsu FTP server at ftp ftp ts fujitsu com image...

Page 71: ...DVD into the DVD drive before proceeding Ê Finish the intended maintenance task For further information refer to the following manuals available online ServerView Update Manager Express Local System Update for PRIMERGY Servers user guide PrimeCollect PrimeCollect user guide Ê After the update or diagnostic procedure has been completed it is necessary to connect FC iSCSI CNA SAS connections on the ...

Page 72: ...h on the ID indicators I When the ID button is pushed for five seconds or more and separated it lights to blue In addition when the ID button is pushed again within one second NMI is issued I For further information refer to section Front panel indicators on page 284 Using management blade web interface Ê Open the Components System Server Blades Server Blade x menu for the desired server blade Ê P...

Page 73: ...rd BIOS TFTP update procedure Ê Ensure that the server blade has been shut down as described in section Shutting down the server blade on page 52 Ê Login to the management blade web interface Ê Open the Information Operation Operation Firmware Update menu Ê Open the Server Blade tab Ê In the Server Blade to be Updated box select the desired server Ê In the Update settings box enter the TFTP IPv4 o...

Page 74: ...er blade on page 52 Ê Login to the management blade web interface Ê Open the Components System Server Blades Server Blade x menu for the desired server blade Ê Select the Configuration tab Ê In the Boot Options box activate the BIOS Recovery Flash Bit Enabled option Ê Connect the monitor keyboard mouse and USB memory stick to the port on the front of the server blade using the Y cable see the Oper...

Page 75: ... blade on page 56 Ê Reinstall and secure the server blade in the system unit as described in section Installing the server blade in the system unit on page 57 Ê Connect the monitor keyboard mouse and USB memory stick to the port on the front of the server blade using the Y cable see the Operating Manual Ê Switch on the server blade as described in section Switching on the server blade on page 60 A...

Page 76: ...r blade on page 60 Ê You can now put the server blade back into operation 5 2 1 2 Updating or recovering the iRMC TFTP update procedure Ê Login to the management blade web interface Ê Open the Information Operation Operation Firmware Update menu Ê Open the Server Blade tab Ê In the Server Blade to be Updated box select the desired server Ê In the Update settings box enter the TFTP IPv4 or IPv6 add...

Page 77: ...sing the Y cable see the Operating Manual Ê Switch on the server blade while as described in section Switching on the server blade on page 60 to boot from the USB memory stick After completion of the boot operation the FlashDisk menu opens Ê Select Recovery _L to carry out the recovery flash for firmware image 1 low firmware image Ê Select Recovery _U to carry out the recovery flash for firmware i...

Page 78: ...ress Subnet Mask Gateway Ê Click the Apply button Ê Open the iRMC web interface FUJITSU ServerView iRMC S4 Web Server directly V CAUTION Restoring functionality of iRMC configuration is only supported with iRMC web interface directly Restoring functionality of iRMC configuration via MMB web interface is not supported Ê Open the iRMC S4 Save Configuration Ê Select the backup file stored in the file...

Page 79: ... are available as ASPs Autonomous Support Packages for Windows or as DOS tools from the Fujitsu support web pages at http ts fujitsu com support Ê Select Drivers Downloads Ê From the Select Product drop down lists choose your PRIMERGY server or enter its serial or ident number into the search field Ê Select your operating system and version Ê Select the desired component type e g SAS RAID Ê Select...

Page 80: ...ion ROM keep in mind that only two Option ROMs can be activated in the system board BIOS at a time Ê Open a virtual console for your server blade as described in section Launching a video redirection to a server blade on page 63 Ê Switch on or restart your server blade Ê As soon as the startup screen appears press the F2 function key to enter the BIOS Ê From the Advanced menu select Option ROM Con...

Page 81: ...M has to remain enabled 5 2 6 Verifying and configuring the backup software solution I This task only applies to the Japanese market Disabling backup drives Depending on the backup software solution it may be necessary to disable or delete the backup drive from the backup software drive list and reconfigure backup jobs after completing the maintenance task This is the case for the following backup...

Page 82: ...ter The boot retry counter is decremented from its preset value every time the POST watchdog initiates a system reboot When the value has reached 0 the system will shut down and power off 5 2 7 1 Viewing the boot retry counter The current boot retry counter status is available in the management blade web interface Ê Enter the management blade web interface Ê Open the Components System Server Blade...

Page 83: ...an only be determined after this period of time If the customer knows about the original boot retry values proceed as follows to reset or configure the boot retry counter Resetting the boot retry counter in the management blade web interface Ê Enter the management blade web interface Ê Open the Components System Server Blades Server Blade x menu for the desired server blade Ê Select the Configurat...

Page 84: ...r blade on page 63 Ê Switch on or restart your server blade Ê As soon as the startup screen appears press the F2 function key to enter the BIOS Ê Select the Server Mgmt menu Ê Under Boot Watchdog set the Action setting to Reset Ê Save your changes and exit the BIOS I For detailed information on how to access the BIOS and modify settings refer to the D3143 BIOS Setup Utility for PRIMERGY BX924 S4 r...

Page 85: ...S Ê Select the Advanced menu Ê Select the status menu of the desired component Processors CPU Status I This option is only available for multi processor systems Memory Memory Status Expansion cards PCI Status Ê Reset replaced components to Enable Ê Save your changes and exit the BIOS I For detailed information on how to access the BIOS and modify settings refer to the D3143 BIOS Setup Utility for ...

Page 86: ...n how to access the BIOS and modify settings refer to the D3143 BIOS Setup Utility for PRIMERGY BX924 S4 reference manual 5 2 11 Verifying the system time settings I This task only applies to Linux environments After the system board has been replaced the system time is set automatically By default the RTC Real Time Clock time standard is set as the local time If a Linux OS is used and the hardwar...

Page 87: ...ence manual 5 2 12 Viewing and clearing the System Event Log SEL 5 2 12 1 Viewing the SEL You can view the System Event Log SEL of the management blades and server blades using the management blade web interface and ServerView Operations Manager frontend Viewing the SEL using the management blade web interface Ê Enter the management blade web interface Ê Select the Components System Server Blades ...

Page 88: ...onfirms the event on the server side You may have to log on to the server beforehand The status of the component will then be reset to ok To see the new status you must refresh the Driver Monitor view with Refresh I For detailed information on how to view and sort the SEL using ServerView Operations Manager refer to the ServerView Operations Manager Server Management user guide 5 2 12 2 Saving the...

Page 89: ...sing the management blade web interface on page 41 Ê Select the Components System Server Blades Server Blade x menu for the desired server blade and open the Event Log tab Ê In the Event Log Filter box click Clear All Entries to clear the SEL 5 2 13 Updating the NIC configuration file in a Linux environment In order to prevent errors caused by changing network device names eth x it is recommended ...

Page 90: ...t depending on the client s environment Ê Select Keep Configuration and Ignore to complete the boot process Ê Use the vi text editor to specify the MAC address in the HWADDR section of the ifcfg eth x file I The MAC address can be found on the type label attached to the system board or network controller Example In order to modify the definition file for network controller 1 enter the following co...

Page 91: ...y in order to boot the operating system Ê Ask the system administrator to enable BitLocker protection on the operating system drive using the BitLocker setup wizard available either from the Control Panel or Windows Explorer Ê Open Bitlocker Drive Encryption by clicking the Start button clicking Control Panel clicking Security and then clicking Bitlocker Drive Encryption I Administrator permission...

Page 92: ...ncy will not be available until the RAID array rebuild is complete Depending on the hard disk drive capacity the overall process can take up to several hours in some cases even days I You may notice a slight performance impact during rebuild 5 2 16 Looking up changed MAC WWN addresses When replacing a network controller or SFP transceiver module the MAC Media Access Control and WWN World Wide Name...

Page 93: ... including its MAC address Ê Note down the new 12 digit MAC address Ê Press Esc to exit the BIOS Ê Inform the customer about the changed MAC address 5 2 16 2 Looking up WWN addresses Emulex FC FCoE adapters Ê Enable the network controller s Option ROM in the system board BIOS as described in section Enabling Option ROM scan on page 80 Ê Restart the server Ê During boot as soon as the Emulex BIOS u...

Page 94: ...tp partners ts fujitsu com com service intelservers tools I For the Japanese market follow the instructions provided separately 5 2 18 Configuring LAN teaming Use ServerView Operations Manager to obtain detailed information on existing LAN teams Ê In ServerView Operations Manager Single System View select System Status from the Information Operation menu Ê Under Network Interfaces select LAN Teami...

Page 95: ...g the CNA driver utility after replacing the server blade I For details refer to the relevant CNA driver manual 5 2 19 Switching off the ID indicator Press the ID button on the front panel or use management blade web interface to switch off the ID indicator after the maintenance task has been concluded successfully I For further information refer to section Locating the defective server blade on p...

Page 96: ...96 Upgrade and Maintenance Manual BX924 S4 Basic software procedures cognitas Gesellschft für Technik Dokumentation mbH 2013 Pfad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20564 001 fm ...

Page 97: ...n disposing of transferring or returning a hard disk or solid state drive wipe out the data on the drive for your own security Rough handling of drives may damage the stored data To cope with any unexpected problems always back up important data When backing up data to another hard disk drive you should make backups on a file or partition basis Do not use the device in extremely hot or cold locati...

Page 98: ...heir respective mounting locations see section Locations of the drives inside the server blade on page 100 There are two kinds of disk mounting kits available An SAS disk mounting kit see pos 1 2 in figure 15 A PCH SATA disk mounting kit see pos 4 5 in figure 15 The following figure shows both kinds of kits together with a HDD SSD module see pos 3 in figure 15 Figure 15 SAS and PCH disk mounting k...

Page 99: ...r as described in section Opening the rack door on page 51 Ê Locate the desired server blade as described in section Locating the defective server blade on page 43 Ê Shut down the server blade as described in section Shutting down the server blade on page 52 Ê Remove the server blade from the system unit as described in section Removing the server blade from the system unit on page 54 Ê Open the s...

Page 100: ...565 001 fm 6 3 3 Locations of the drives inside the server blade The following figure shows the mounting locations of the drives Figure 16 Mounting locations of the drives inside the server blade I The first drive has to be mounted on the location for HDD SSD 1 see pos 2 in figure above Description 1 Mounting location of SSD 0 below mezzanine card 1 2 Mounting location of HDD SSD 1 on top of CPU 2...

Page 101: ... 0 Ê Remove the mezzanine carrier as described in section Removing mezzanine cards on page 140 Ê Choose the desired mounting frame PCH or SAS for SSD 0 Figure 17 Securing the drive Ê Shift the drive into the connector on the mounting frame 1 Ê Secure the drive with 4 screws see arrows Torque 0 6 Nm not applicable for the Japanese market ...

Page 102: ... on the system board see arrow Figure 18 Connector for the SSD 0 module Figure 19 Mounting the SSD 0 module into its location Ê Slide the mounting kit into the cut out 1 of the mounting platform Ê Plug the mounting kit into the connector see arrow in figure 18 on page 102 for SSD 0 on the mounting platform 2 Ê Secure the mounting kit with the knurled screw 3 Ê Install the mezzanine carrier as desc...

Page 103: ...talling HDD SSD 1 Mounting HDD SSD 1 into the corresponding mounting frame Ê Choose the desired mounting frame PCH or SAS for HDD SSD 1 Figure 20 Securing the drive in the mounting frame Ê Shift the drive into the connector on the mounting frame 1 Ê Secure the drive with 4 screws see arrows ...

Page 104: ...Technik Dokumentation mbH 2013 Pfad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20565 001 fm Installing HDD SSD 1 onto the dummy heat sink SSD holder The following figure shows the connector for HDD SSD 1 see arrow on the mounting platform Figure 21 Connector for HDD SSD 1 on the dummy heat sink ...

Page 105: ...igure 22 Mounting the HDD SSD 1 module into its location Ê Slide the mounting kit into the cut out 1 of the mounting platform Ê Plug the mounting kit into the connector see arrow in figure 21 for HDD SSD 1 on the mounting platform 2 Ê Secure the mounting kit with the knurled screw 3 ...

Page 106: ... figure shows the connector for HDD SSD 1 on the mounting frame see arrow Figure 23 Connector for HDD SSD 1 on top of the heat sink for CPU 2 Figure 24 Mounting the HDD SSD 1 into its location Ê Slide the mounting kit into the cut out 1 of the mounting platform Ê Plug the mounting kit into the connector see arrow in figure 23 for HDD SSD 1 on the mounting platform 2 Ê Secure the mounting kit with ...

Page 107: ...described in section Closing the server blade on page 56 Ê Reinstall and secure the server blade in the system unit as described in section Installing the server blade in the system unit on page 57 Ê If applicable close the rack door as described in section Closing the rack door on page 62 Ê Please observe the notes on RAID rebuild in section Performing a RAID array rebuild on page 92 ...

Page 108: ...e removed is not combined into a RAID array If the drive is part of a RAID array you first need to delete the array using ServerView RAID Manager V CAUTION All data on all HDDs SSDs in the array will be lost Be sure to back up your data before deleting a RAID array I For further information please refer to the ServerView Suite RAID Management user guide available online Ê If applicable open the ra...

Page 109: ...s described in section Opening the server blade on page 55 6 4 3 Removing SSD 0 Ê Remove the mezzanine carrier as described in section Removing mezzanine cards on page 140 Figure 25 Removing SSD 0 from the mounting platform Ê Loosen the knurled screw of the mounting kit 1 Ê Unplug the mounting kit from the connector by lifting it off from the mounting platform 2 Ê Slide the mounting kit out of the...

Page 110: ...ik Dokumentation mbH 2013 Pfad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20565 001 fm Removing SSD 0 from the mounting frame Figure 26 Removing the screws from the mounting frame Ê Remove the 4 screws from the mounting frame Ê Slide the drive from the mounting frame 1 in the direction of the arrow ...

Page 111: ...moving HDD SSD 1 from the dummy heat sink Figure 27 Removing HDD SSD 1 from the mounting platform Ê Loosen the knurled screw of the mounting kit 1 Ê Unplug the mounting kit from the connector by lifting it off from the mounting platform 2 Ê Slide the mounting kit out of the cut out 3 of the mounting platform ...

Page 112: ... tim_local work PICHOL OBJ_DOKU 20565 001 fm Removing HDD SSD 1 from top of the heat sink of CPU 2 Figure 28 Removing HDD SSD 1 from the mounting platform Ê Loosen the knurled screw of the mounting kit 1 Ê Unplug the mounting kit from the connector by lifting it off from the mounting platform 2 Ê Slide the mounting kit out of the cut out 3 of the mounting platform ...

Page 113: ... the mounting frame 1 in the direction of the arrow 6 4 5 Concluding steps Perform the following procedures to complete the task Ê Close the server blade as described in section Closing the server blade on page 56 Ê Reinstall and secure the server blade in the system unit as described in section Installing the server blade in the system unit on page 57 Ê If applicable close the rack door as descri...

Page 114: ...acing a HDD SSD perform the following steps Ê Disable BitLocker functionality as described in section Disabling BitLocker functionality on page 66 Ê If applicable open the rack door as described in section Opening the rack door on page 51 Ê Locate the desired server blade as described in section Locating the defective server blade on page 43 Ê Shut down the server blade as described in section Shu...

Page 115: ...mpty drive bay as described in section Installing SSD 0 on page 101 6 5 5 Concluding steps Perform the following procedures to complete the task Ê Close the server blade as described in section Closing the server blade on page 56 Ê Reinstall and secure the server blade in the system unit as described in section Installing the server blade in the system unit on page 57 Ê If applicable close the rac...

Page 116: ...CH backplane perform the following steps Ê Disable BitLocker functionality as described in section Disabling BitLocker functionality on page 66 Ê If applicable open the rack door as described in section Opening the rack door on page 51 Ê Locate the desired server blade as described in section Locating the defective server blade on page 43 Ê Shut down the server blade as described in section Shutti...

Page 117: ...that you can reinsert them into their original locations after replacing the backplane Ê Remove the 4 screws from the mounting frame as described in section Removing SSD 0 on page 109 or Removing HDD SSD 1 on page 111 Ê Remove the HDD SSD from the mounting frame as described in section Removing SSD 0 on page 109 or Removing HDD SSD 1 on page 111 6 6 3 Removing the SAS PCH backplane Figure 30 Remov...

Page 118: ...BJ_DOKU 20565 001 fm 6 6 4 Installing the SAS PCH backplane The type of the backplane and the type of the mounting frame to whom it belongs can be seen on the backplane as shown in the following figure Figure 31 Types of backplanes I The different types of mounting kits are shown in section Overview of the disk mounting kits on page 98 1 Type of backplane PCH or SAS 2 SSD 0 or HDD SSD 1 ...

Page 119: ...aintenance Manual 119 Hard disk drives Solid state drives Figure 32 Installing the SAS PCH backplane Ê Install the new SAS PCH backplane to the mounting frame Ê Tighten the two screws see arrows to the new SAS PCH backplane ...

Page 120: ...SSD 1 on page 103 Ê Reinstall the HDD SSD to their original mounting platforms as described in section Installing HDD SSD on page 99 Ê If applicable install the mezzanine carrier as described in chapter Installing mezzanine cards on page 134 Ê Close the server blade as described in section Closing the server blade on page 56 Ê Reinstall and secure the server blade in the system unit as described i...

Page 121: ...oving internal options Circuit boards and soldered parts of internal options are exposed and can be damaged by static electricity Always discharge static build up e g by touching a grounded object before handling electrostatic sensitive devices ESDs Do not touch the circuitry on boards or soldered parts Hold circuit boards by their metallic areas or edges If devices are installed or disassembled u...

Page 122: ...mation One or two mezzanine cards can be installed in a BX924 S4 server blade Additional Fibre Channel SAS Ethernet and or Infiniband I O channels can be set up using these cards All mezzanine cards have the same form factor Figure 33 Sample of a 8 Gbit s fibre channel card with 2 ports The mezzanine cards are fastened on a special carrier and then connected together with the carrier to the system...

Page 123: ...f unpopulated carriers for mezzanine cards All installation removal procedures are identical for both carriers Figure 34 Carrier for mezzanine cards with two x8 riser cards Figure 35 Carrier for mezzanine cards with one x8 1 and one x16 2 riser card I Note the numbering of the mezzanine card slots 0 ...

Page 124: ...chnik Dokumentation mbH 2013 Pfad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20566 001 fm 7 1 1 Installing riser cards Installing the riser card x8 Figure 36 Installing the riser card x8 Ê Connect the riser card to the carrier Make sure that the green clips click into place 0 ...

Page 125: ... Upgrade and Maintenance Manual 125 Mezzanine cards Installing the riser card x16 Figure 37 Installing the riser card x16 Ê Connect the riser card to the carrier Make sure that the green clips click into place ...

Page 126: ...chft für Technik Dokumentation mbH 2013 Pfad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20566 001 fm 7 1 2 Removing riser cards Removing the riser card x8 Figure 38 Removing the riser card x8 Ê Remove the riser card from the holder at the slot of mezzanine card 2 0 ...

Page 127: ...BX924 S4 Upgrade and Maintenance Manual 127 Mezzanine cards Removing the riser card x16 Figure 39 Removing the riser card x16 Ê Remove the riser card from the holder ...

Page 128: ...ds The slots of the mezzanine cards in the server blade are connected to certain connection blade slots on the back of the system unit You therefore need to observe how the connection blade slots are populated on the back of the system unit when installing mezzanine cards Population rules for mezzanine cards in the BX900 S1 S2 system unit Figure 41 Connection blade slots Mezzanine Card 1 Mezzanine...

Page 129: ...ade slots Fabric 1 CB1 1Gb Ethernet or CB2 1Gb Ethernet or Onboard CNA controller 10 Gb Ethernet 10Gb Ethernet Fabric 2 CB3 1Gb Ethernet or CB4 1Gb Ethernet or Mezzanine card 1 Eth 1Gb 4port or Eth 10Gb 2port FC 8Gb 2port or IB 40 56 Gb 2port 10Gb Ethernet or 10Gb Ethernet or Fibre Channel or Fibre Channel or Infiniband CB3 4 Fabric 3 CB5 1Gb Ethernet or CB6 1Gb Ethernet or Mezzanine card 2 Eth 1G...

Page 130: ... fabric 2 I In this case only one of two channels of the Infiniband mezzanine card is used If a 1 Gb Ethernet mezzanine card is installed in slot 2 of a server blade at least one 1 Gb Ethernet connection blade must be installed in fabric 3 or in fabric 4 of the system unit If a 10 Gb Ethernet mezzanine card or a 10Gb CNA mezzanine card is installed in slot 2 of a server blade at least one 10 Gb Et...

Page 131: ...bx924s4 html for the Japanese market http jp fujitsu com platform server primergy system Mezzanine slot 1 Mezzanine slot 2 1 Gb Ethernet 10 Gb Ethernet 10 Gb CNA 1 Gb Ethernet Fibre Channel 1 Gb Ethernet Infiniband 1 Gb Ethernet SAS SAS RAID 10 Gb Ethernet 10 Gb CNA Fibre Channel 10 Gb Ethernet 10 Gb CNA Infiniband 10 Gb Ethernet 10 Gb CNA SAS SAS RAID Fibre Channel Infiniband Fibre Channel SAS SA...

Page 132: ...he rules for populating the mezzanine card slots are as follows If a 1 Gb Ethernet mezzanine card is installed in slot 1 of a server blade at least one 1 Gb Ethernet connection blade must be installed in CB slot 2 of the system unit If a 10 Gb Ethernet mezzanine card is installed in slot 1 of a server blade at least one 10 Gb Ethernet connection blade must be installed in CB slot 2 of the system u...

Page 133: ...e installed in CB slot 3 or in CB slot 4 of the system unit If an FC mezzanine card is installed in slot 2 at least one FC connection blade must be installed in CB slot 3 or in CB slot 4 of the system unit If an Infiniband mezzanine card is installed in slot 2 an Infiniband connection blade must be installed in CB slots 3 and 4 of the system unit If a SAS SAS RAID mezzanine card is installed in sl...

Page 134: ...ble boot watchdog functionality as described in section Disabling boot watchdog functionality on page 67 Ê If applicable open the rack door as described in section Opening the rack door on page 51 Ê Locate the desired server blade as described in section Locating the defective server blade on page 43 Ê Shut down the server blade as described in section Shutting down the server blade on page 52 Ê R...

Page 135: ...ine card The following section illustrates how to install a mezzanine card in slot 2 Removing the mezzanine card carrier Figure 43 Removing the mezzanine card carrier Ê Remove the mezzanine card carrier from the server blade housing by lifting it up keeping it as horizontal as possible ...

Page 136: ... work PICHOL OBJ_DOKU 20566 001 fm Figure 44 Removing the riser card Ê Remove the riser card from the holder at the slot of mezzanine card 2 Figure 45 Inserting the mezzanine card Ê Place the mezzanine card on the two guide pins 1 at the slot of mezzanine card 2 and press the mezzanine card down so that it clicks into place between the two clips 2 0 0 0 ...

Page 137: ...ecting the riser card Ê Connect the riser card to the mezzanine card Make sure that the green clips click into place I Mezzanine card 1 is fastened to the carrier with the component side facing downward Mezzanine card 1 is otherwise installed in the same way as mezzanine card 2 ...

Page 138: ...re 47 Installing the mezzanine card carrier Ê Install the carrier with the mezzanine cards in the server blade housing As you do this the riser cards are inserted in the corresponding system board slots Make sure that the coding on the carrier matches that on the server blade housing Ê Close the server blade re insert it in the system unit and switch it on as described in section Closing the serve...

Page 139: ...ine card that has been installed or replaced the card s Option ROM has to be enabled in the system board BIOS For SAN iSCSI boot the card s Option ROM has to be enabled permanently If applicable proceed as described in section Enabling Option ROM scan on page 80 Ê Enable boot watchdog functionality as described in section Enabling boot watchdog functionality on page 84 Ê If BitLocker functionality...

Page 140: ... boot watchdog functionality as described in section Disabling boot watchdog functionality on page 67 Ê If applicable open the rack door as described in section Opening the rack door on page 51 Ê Locate the desired server blade as described in section Locating the defective server blade on page 43 Ê Shut down the server blade as described in section Shutting down the server blade on page 52 Ê Remo...

Page 141: ... to remove a mezzanine card in slot 2 Removing the mezzanine card carrier Figure 48 Removing the mezzanine card carrier Ê Remove the carrier from the server blade housing by lifting it up keeping it as horizontal as possible Figure 49 Removing the riser card Ê Remove the riser card from the holder at the slot of mezzanine card 2 ...

Page 142: ...bH 2013 Pfad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20566 001 fm Figure 50 Removing the mezzanine card Ê Press the two clips 1 and replace the mezzanine card 2 Figure 51 Reconnecting the riser card Ê Connect the riser card to the mezzanine card Make sure that the green clips click into place 0 ...

Page 143: ... system unit on page 57 Ê If applicable close the rack door as described in section Closing the rack door on page 62 Ê Enable boot watchdog functionality as described in section Enabling boot watchdog functionality on page 84 Ê If BitLocker functionality is used and has been disabled before starting the maintenance task re enable BitLocker as described in section Enabling BitLocker functionality o...

Page 144: ... applies to all static IP address and LAN teaming configurations Ensure to note down your current network settings before replacing the controller Before replacing an mezzanine card perform the following steps Ê Disable BitLocker functionality as described in section Disabling BitLocker functionality on page 66 Ê Disable boot watchdog functionality as described in section Disabling boot watchdog f...

Page 145: ...ation system according to the original configuration of the replaced controller expansion card or onboard CNA For further information please refer to section Note on network settings recovery on page 144 Perform the following procedures to complete the task Ê Close the server blade as described in section Closing the server blade on page 56 Ê Reinstall and secure the server blade in the system uni...

Page 146: ...rder to configure a mezzanine card that has been installed or replaced the card s Option ROM has to be enabled in the system board BIOS For SAN iSCSI boot the card s Option ROM has to be enabled permanently If applicable proceed as described in section Enabling Option ROM scan on page 80 Ê Enable boot watchdog functionality as described in section Enabling boot watchdog functionality on page 84 Ê ...

Page 147: ...ter shutdown Wait for components to cool down before installing or removing memory modules to prevent burns Do not insert and remove memory modules repeatedly Doing so may cause failures Pressing out the securing clips on the memory module connector will eject the installed memory module To prevent damage and injuries eject memory modules carefully without applying excessive force For further info...

Page 148: ...of RAM 1536 GB Supported memory modules 8 1 1 Memory sequence 8 1 1 1 Population rules Populate memory slot 1 channel A DIMM 1A first In case of dual processor configurations populate memory slot 1 channel E DIMM 1E second Within all channels memory slot 1 must be populated prior to slot 2 Slot 2 must be populated prior to Slot 3 for all CPUs If memory modules with different ranks are used always ...

Page 149: ... and the maximum speed of the selected configuration Mixing RDIMMs or LRDIMMs is not allowed Mixing ECC and non ECC DIMMs is not allowed Mixing of quad rank DIMMs in one channel and three DIMMs in another channel 3DPC on the same CPU is not allowed This is not valid for LRDIMMs Figure 52 Overview of the slot numbering and the memory channels CPU 1 CPU 2 Front Channel B Channel A DIMM 3B DIMM 1B DI...

Page 150: ... U P C l a u D s M M I D 1 1 2 1 2 3 1 3 2 4 1 3 2 4 5 1 5 3 2 4 6 1 5 3 2 6 4 7 1 5 7 3 2 6 4 8 1 5 7 3 2 6 8 4 9 1 9 5 7 3 2 6 8 4 10 1 9 5 7 3 2 10 6 8 4 11 1 9 5 7 3 11 2 10 6 8 4 12 1 9 5 7 3 11 2 10 6 8 4 12 13 1 9 5 13 7 3 11 2 10 6 8 4 12 14 1 9 5 13 7 3 11 2 10 6 14 8 4 12 15 1 9 5 13 7 15 3 11 2 10 6 14 8 4 12 16 1 9 5 13 7 15 3 11 2 10 6 14 8 16 4 12 17 1 9 17 5 13 7 15 3 11 2 10 6 14 8...

Page 151: ...9 5 13 7 3 11 2 10 6 8 4 12 1 9 5 13 7 3 11 2 10 6 14 8 4 12 1 9 5 13 7 15 3 11 2 10 6 14 8 4 12 1 9 5 13 7 15 3 11 2 10 6 14 8 16 4 12 1 9 17 5 13 7 15 3 11 2 10 6 14 8 16 4 12 1 9 17 5 13 7 15 3 11 2 10 18 6 14 8 16 4 12 1 9 17 5 13 7 15 3 11 19 2 10 18 6 14 8 16 4 12 1 9 17 5 13 7 15 3 11 19 2 10 18 6 14 8 16 4 12 20 1 9 17 5 13 7 15 3 11 19 2 10 18 6 14 21 8 16 4 12 20 1 9 17 5 13 7 15 3 11 19...

Page 152: ... G DIMM 1A 2A 3A 1B 2B 3B 1D 2D 3D 1C 2C 3C 1E 2E 3E 1F 2F 3F 1H 2H 3H 1G 2G 3G n o i t a r u g i f n o c U P C o n o M s M M I D 4 1 1 1 1 8 1 2 1 2 1 2 1 2 12 1 2 3 1 2 3 1 2 3 1 2 3 n o i t a r u g i f n o c U P C l a u D s M M I D 8 1 1 1 1 2 2 2 2 12 1 3 1 3 1 3 1 3 2 2 2 2 16 1 3 1 3 1 3 1 3 2 4 2 4 2 4 2 4 20 1 3 5 1 3 5 1 3 5 1 3 5 2 4 2 4 2 4 2 4 24 1 3 5 1 3 5 1 3 5 1 3 5 2 4 6 2 4 6 2 4...

Page 153: ...are of the same capacity 4 o o o o 5 o o o o o 6 o o o o o o 7 o o o o o o o 8 o o o o o o o o 9 o o o o o o o o o 10 o o o o o o o o o o 11 o o o o o o o o o o o 12 o o o o o o o o o o o o 13 o o o o o o o o o o o o o 14 o o o o o o o o o o o o o o 15 o o o o o o o o o o o o o o o 16 o o o o o o o o o o o o o o o o 17 o o o o o o o o o o o o o o o o o 18 o o o o o o o o o o o o o o o o o o 19 o o...

Page 154: ... 1H 2H 3H 1G 2G 3G Dual CPU configuration if modules are of different capacities Dual CPU configuration if modules are of different capacities 2 o o 2 o o 3 o o o 3 o o o 4 o o o o 4 o o o o 5 o o o o o 5 o o o o o 6 o o o o o o 6 o o o o o o 7 o o o o o o o 7 o o o o o o o 8 o o o o o o o o 8 o o o o o o o o 9 o o o o o o o o o 9 o o o o o o o o o 10 o o o o o o o o o o 10 o o o o o o o o o o 11 ...

Page 155: ...1 1 1 1 1 1 2 2 1 1 1 2 2 1 1 1 2 2 2 1 1 1 1 1 2 2 1 1 1 1 1 2 2 2 1 1 1 1 1 1 2 2 2 1 1 1 1 1 2 2 2 2 2 1 1 1 1 1 1 2 2 2 2 2 1 1 1 1 1 1 2 2 2 2 2 2 1 1 1 1 1 1 1 1 2 2 2 2 2 1 1 1 1 1 1 1 1 2 2 2 2 2 2 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 2 2 ...

Page 156: ...o o o o o o o o o DIMMs Dual CPU configuration if all modules are of the same capacity 1 o 2 o o 3 o o o 4 o o o o 5 o o o o o 6 o o o o o o 7 o o o o o o o 8 o o o o o o o o 9 o o o o o o o o o 10 o o o o o o o o o o 11 o o o o o o o o o o o 12 o o o o o o o o o o o o 13 o o o o o o o o o o o o o 14 o o o o o o o o o o o o o o 15 o o o o o o o o o o o o o o o 16 o o o o o o o o o o o o o o o o 17...

Page 157: ... capacities Dual CPU configuration if modules are of different capacities 1 o 1 o 2 o o 2 o o 3 o o o 3 o o o 4 o o o o 4 o o o o 5 o o o o o 5 o o o o o 6 o o o o o o 6 o o o o o o 7 o o o o o o o 7 o o o o o o o 8 o o o o o o o o 8 o o o o o o o o 9 o o o o o o o o o 9 o o o o o o o o o 10 o o o o o o o o o o 10 o o o o o o o o o o 11 o o o o o o o o o o o 11 o o o o o o o o o o o 12 o o o o o o...

Page 158: ...A 2A 3A 1B 2B 3B 1D 2D 3D 1C 2C 3C 1E 2E 3E 1F 2F 3F 1H 2H 3H 1G 2G 3G 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 2 1 1 2 2 1 1 1 2 2 1 1 1 1 2 2 1 1 1 1 2 2 2 1 1 1 1 2 2 2 2 1 1 1 1 1 2 2 2 2 1 1 1 1 1 1 2 2 2 2 1 1 1 1 1 1 2 2 2 2 2 1 1 1 1 1 1 2 2 2 2 2 2 1 1 1 1 1 1 1 2 2 2 2 2 2 1 1 1 1 1 1 1 1 2 2 2 2 2 2 1 1 1 1 1 1...

Page 159: ...d 8 x 4 GB and 4 x 8 GB In this case proceed as follows Ê Use the capacity group with more memory modules for CPU 1 and populate them according to the population sequence for mono CPU configurations Ê Populate the other capacity group on CPU 2 according to the population rules for mono CPU configurations In order to satisfy cooling requirements populate memory channels on alternating sides of the ...

Page 160: ...sabling boot watchdog functionality on page 67 Ê If applicable open the rack door as described in section Opening the rack door on page 51 Ê Locate the desired server blade as described in section Locating the defective server blade on page 43 Ê Shut down the defective server blade as described in section Shutting down the server blade on page 52 Ê Remove the server blade from the system unit as d...

Page 161: ...n section Memory sequence on page 148 Ê If required remove the mezzanine carrier for better handling of the DIMMs of CPU 1 Figure 53 Opening the securing clips Ê Press the securing clips on both sides of the relevant memory slot outward Figure 54 Inserting the memory module Ê Align the notch on the bottom of the module with the crossbar in the connector see close up ...

Page 162: ...ls as described in section Installing the air cowls on page 170 Ê If applicable reinstall the mezzanine carrier as described in section Installing mezzanine cards on page 134 Ê Close the server blade as described in section Closing the server blade on page 56 Ê Reinstall and secure the server blade in the system unit as described in section Installing the server blade in the system unit on page 57...

Page 163: ... boot watchdog functionality on page 84 Ê If applicable configure the memory mode as described in section Verifying the memory mode on page 85 Ê If BitLocker functionality is used and has been disabled before starting the maintenance task re enable BitLocker as described in section Enabling BitLocker functionality on page 91 ...

Page 164: ...ling boot watchdog functionality on page 67 Ê If applicable open the rack door as described in section Opening the rack door on page 51 Ê Locate the desired server blade as described in section Locating the defective server blade on page 43 Ê Shut down the defective server blade as described in section Shutting down the server blade on page 52 Ê Remove the server blade from the system unit as desc...

Page 165: ...ure to maintain an operational configuration when removing memory modules For additional information please refer to section Memory sequence on page 148 Ê If required remove the mezzanine carrier for better handling of the DIMMs of CPU 1 Figure 56 Removing memory modules Ê Eject the desired memory module by pressing out the securing clips at each end of the memory module connector Ê Remove the eje...

Page 166: ...ion Closing the server blade on page 56 Ê Reinstall and secure the server blade in the system unit as described in section Installing the server blade in the system unit on page 57 Ê If applicable close the rack door as described in section Closing the rack door on page 62 Ê If available update the system board BIOS to the latest version as described in section Updating or recovering the system bo...

Page 167: ... the rack door as described in section Opening the rack door on page 51 Ê Locate the defective server blade as described in section Locating the defective server blade on page 43 Ê Shut down the defective server blade as described in section Shutting down the server blade on page 52 Ê Remove the server blade from the system unit as described in section Removing the server blade from the system uni...

Page 168: ...he defective memory module using the onboard Local Diagnostic LEDs as described in section Onboard indicators and controls on page 280 8 4 3 Removing a memory module Ê Remove the defective memory module as described in section Removing a memory module on page 165 8 4 4 Installing a memory module Ê Replace the defective memory module as described in section Installing a memory module on page 161 ...

Page 169: ...cable close the rack door as described in section Closing the rack door on page 62 Ê If available update the system board BIOS to the latest version as described in section Updating or recovering the system board BIOS and iRMC on page 73 Ê Enable boot watchdog functionality as described in section Enabling boot watchdog functionality on page 84 Ê Enable the replaced memory module s as described in...

Page 170: ... work PICHOL OBJ_DOKU 20567 001 fm 8 5 Handling of memory air cowls The memory modules of CPU 1 have to be covered by air cowls for cooling reasons 8 5 1 Installing the air cowls Figure 57 Installing the air cowls Ê Place the air cowls over the DIMMs of CPU 1 as shown in the figure I For cooling reasons the air cowls must be installed when operating ...

Page 171: ...BX924 S4 Upgrade and Maintenance Manual 171 Main memory 8 5 2 Removing the air cowls Figure 58 Lifting up the air cowls Ê Lift up the air cowls from the DIMMs of CPU 1 ...

Page 172: ...172 Upgrade and Maintenance Manual BX924 S4 Main memory cognitas Gesellschft für Technik Dokumentation mbH 2013 Pfad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20567 001 fm ...

Page 173: ... up e g by touching a grounded object before handling electrostatic sensitive devices ESDs Do not touch the circuitry on boards or soldered parts Hold circuit boards by their metallic areas or edges When removing or installing the processor be careful not to touch or bend the spring contacts on the processor socket Never touch the underside of the processor Even minor soiling such as grease from t...

Page 174: ...o 130 W 9 2 Installing processors V CAUTION Processors are extremely sensitive to electrostatic discharge and must be handled with care After a processor has been removed from its protective sleeve or from its socket place it upside down on a nonconducting antistatic surface Never push a processor over a surface 9 2 1 Required tools Preliminary and concluding steps tool less Removing and installin...

Page 175: ...section Opening the rack door on page 51 Ê Locate the desired server blade as described in section Locating the defective server blade on page 43 Ê Shut down the server blade as described in section Shutting down the server blade on page 52 Ê Remove the server blade from the system unit as described in section Removing the server blade from the system unit on page 54 Ê Open the server blade as des...

Page 176: ...iption applies to the following procedures Installing the second CPU in a single processor configuration Transferring a CPU after replacing the system board see section Removing processors on page 186 and Installing processors on page 174 Figure 59 Opening socket release lever 1 Ê Unlatch the socket release lever marked Open 1st by pushing it down and away from the socket 1 Ê The socket release le...

Page 177: ...aintenance Manual 177 Processors Figure 60 Opening socket release lever 2 Ê Unlatch the socket release lever marked Close 1st by pushing it down and away from the socket 1 Ê Fully fold back the second socket release lever 2 ...

Page 178: ...cognitas Gesellschft für Technik Dokumentation mbH 2013 Pfad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20568 001 fm Figure 61 Opening the load plate A Ê Push down on the first socket release lever 1 to lift the load plate away from the socket 2 ...

Page 179: ... Manual 179 Processors Figure 62 Opening the load plate B Ê Fully open the load plate V CAUTION Be careful not to touch or bend the spring contacts on the processor socket Ê Make sure that the load plate is in the fully open position ...

Page 180: ...t the four notches on the processor align with the posts on the socket see close up Ê Lower the processor straight down without tilting or sliding it in the socket V CAUTION Ensure that the processor is level in the socket Be careful not to touch or bend the pins on the processor socket Never touch the underside of the processor Even minor soiling such as grease from the skin can impair the proces...

Page 181: ...BX924 S4 Upgrade and Maintenance Manual 181 Processors Figure 64 Closing the load plate Ê Carefully lower the load plate over the processor ...

Page 182: ...e FCT tim_app tim_local work PICHOL OBJ_DOKU 20568 001 fm Figure 65 Removing the protective socket cover Ê When closing the load plate the protective socket cover will automatically detach Ê Remove the socket cover and save it for future use V CAUTION Always replace the socket cover if you remove the processor from the socket ...

Page 183: ...Manual 183 Processors Figure 66 Closing socket release lever 2 Ê Fully close and hold shut the load plate Ê Close the socket release lever marked Close 1st 1 and latch it under the load plate retention tab to lock down the load plate 2 ...

Page 184: ...kumentation mbH 2013 Pfad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20568 001 fm Figure 67 Closing socket release lever 1 Ê Close the socket release lever marked Open 1st 1 and latch it under the load plate retention tab 2 Ê If applicable install the second processor accordingly ...

Page 185: ...ge 57 Ê If applicable close the rack door as described in section Closing the rack door on page 62 Ê If available update the system board BIOS and the iRMC to the latest version as described in section Updating or recovering the system board BIOS and iRMC on page 73 Ê Enable boot watchdog functionality as described in section Enabling boot watchdog functionality on page 84 Ê If BitLocker functiona...

Page 186: ...ing steps tool less Removing and installing the processor heat sink Phillips PH2 No 2 screw driver Removing the processor tool less 9 3 2 Preliminary steps Before removing a processor perform the following steps Ê Disable BitLocker functionality as described in section Disabling BitLocker functionality on page 66 Ê Disable boot watchdog functionality as described in section Disabling boot watchdog...

Page 187: ...ving a memory module on page 165 Ê Open the server blade as described in section Opening the server blade on page 55 Ê Remove the desired processor heat sink as described in section Removing processor heat sinks on page 219 9 3 3 Removing a processor I This description applies to the following procedures Removing CPU 2 from a dual processor configuration Removing CPUs from a system board see secti...

Page 188: ...ik Dokumentation mbH 2013 Pfad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20568 001 fm Figure 68 Opening socket release lever 1 Ê Unlatch the socket release lever marked Open 1st by pushing it down and away from the socket 1 Ê The socket release lever will slightly lift up 2 ...

Page 189: ...aintenance Manual 189 Processors Figure 69 Opening socket release lever 2 Ê Unlatch the socket release lever marked Close 1st by pushing it down and away from the socket 1 Ê Fully fold back the second socket release lever 2 ...

Page 190: ...tas Gesellschft für Technik Dokumentation mbH 2013 Pfad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20568 001 fm Figure 70 Opening the load plate A Ê Push down on the socket release lever marked Open 1st 1 to lift the load plate away from the socket 2 ...

Page 191: ...BX924 S4 Upgrade and Maintenance Manual 191 Processors Figure 71 Opening the load plate B Ê Fully open the load plate ...

Page 192: ...pp tim_local work PICHOL OBJ_DOKU 20568 001 fm Figure 72 Removing the processor Ê Carefully remove the defective processor from its socket in a vertical motion V CAUTION Be careful not to touch or bend the spring contacts on the processor socket Ê Thoroughly clean residual thermal paste from the processor surface using a lint free cloth ...

Page 193: ...must be handled with care After a processor has been removed from its protective sleeve or from its socket place it upside down on a nonconducting antistatic surface Never push a processor over a surface Figure 73 Closing the load plate Ê Carefully close the load plate over the empty processor socket V CAUTION Be careful not to touch or bend the spring contacts on the processor socket ...

Page 194: ...d C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20568 001 fm Figure 74 Attaching the protective socket cover A Ê Carefully lower the protective socket cover onto the CPU socket in a vertical motion until it snaps in place V CAUTION Always replace the socket cover if you remove the processor from the socket ...

Page 195: ...BX924 S4 Upgrade and Maintenance Manual 195 Processors Figure 75 Attaching the protective socket cover B Ê Ensure that the protective socket cover is properly installed on the socket as shown above ...

Page 196: ...ion mbH 2013 Pfad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20568 001 fm Figure 76 Closing socket release lever 2 Ê Fully close and hold shut the load plate Ê Close the socket release lever marked Close 1st 1 and latch it under the load plate retention tab to lock down the load plate 2 ...

Page 197: ...BX924 S4 Upgrade and Maintenance Manual 197 Processors Figure 77 Closing socket release lever 1 Ê Close the socket release lever marked Open 1st 1 and latch it under the load plate retention tab 2 ...

Page 198: ...e as described in section Closing the server blade on page 56 Ê Reinstall and secure the server blade in the system unit as described in section Installing the server blade in the system unit on page 57 Ê If applicable close the rack door as described in section Closing the rack door on page 62 Ê If available update the system board BIOS and the iRMC to the latest version as described in section U...

Page 199: ...lips PH2 No 2 screw driver Upgrading or replacing the processor tool less 9 4 2 Preliminary steps Before upgrading or replacing the processor perform the following steps Ê Disable BitLocker functionality as described in section Disabling BitLocker functionality on page 66 Ê Disable boot watchdog functionality as described in section Disabling boot watchdog functionality on page 67 Ê If applicable ...

Page 200: ...de from the system unit as described in section Removing the server blade from the system unit on page 54 Ê Open the server blade as described in section Opening the server blade on page 55 Ê If necessary remove the memory as described in section Removing a memory module on page 165 Ê Remove the desired processor heat sink as described in section Removing processor heat sinks on page 219 9 4 3 Upg...

Page 201: ... Open 1st by pushing it down and away from the socket 1 Ê The socket release lever will slightly lift up 2 Figure 79 Opening socket release lever 2 Ê Unlatch the socket release lever marked Close 1st by pushing it down and away from the socket 1 Ê Fully fold back the second socket release lever 2 ...

Page 202: ...tas Gesellschft für Technik Dokumentation mbH 2013 Pfad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20568 001 fm Figure 80 Opening the load plate A Ê Push down on the socket release lever marked Open 1st 1 to lift the load plate away from the socket 2 ...

Page 203: ...BX924 S4 Upgrade and Maintenance Manual 203 Processors Figure 81 Opening the load plate B Ê Fully open the load plate ...

Page 204: ...igure 82 Removing the processor Ê Carefully remove the processor from its socket in a vertical motion V CAUTION Be careful not to touch or bend the spring contacts on the processor socket Ê Thoroughly clean residual thermal paste from the processor surface using a lint free cloth Ê If the processor is being replaced during a CPU upgrade and may be reused later store it in a save place ...

Page 205: ...s been removed from its protective sleeve or from its socket place it upside down on a nonconducting antistatic surface Never push a processor over a surface 9 4 3 2 Installing the processor Figure 83 Installing the processor Ê Hold the processor with your thumb and index finger Ê Make sure that the four notches on the processor align with the posts on the socket see close up ...

Page 206: ...re that the processor is level in the socket Be careful not to touch or bend the pins on the processor socket Never touch the underside of the processor Even minor soiling such as grease from the skin can impair the processor s operation or destroy the processor Ensure not to scrape or dent the processor edges Figure 84 Closing the load plate Ê Carefully lower the load plate over the processor V C...

Page 207: ...Manual 207 Processors Figure 85 Closing socket release lever 2 Ê Fully close and hold shut the load plate Ê Close the socket release lever marked Close 1st 1 and latch it under the load plate retention tab to lock down the load plate 2 ...

Page 208: ...s Gesellschft für Technik Dokumentation mbH 2013 Pfad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20568 001 fm Figure 86 Closing socket release lever 1 Ê Close the socket release lever marked Open 1st 1 and latch it under the load plate retention tab 2 ...

Page 209: ...s described in section Installing the server blade in the system unit on page 57 Ê If applicable close the rack door as described in section Closing the rack door on page 62 Ê If available update the system board BIOS and the iRMC to the latest version as described in section Updating or recovering the system board BIOS and iRMC on page 73 Ê Enable the replaced processor as described in section En...

Page 210: ...eliminary steps Before installing removing or replacing the processor heat sink perform the following steps Ê Shut down the server blade as described in section Shutting down the server blade on page 52 Ê Remove the server blade from the system unit as described in section Removing the server blade from the system unit on page 54 Ê Open the server blade as described in section Opening the server b...

Page 211: ...ce Manual 211 Processors 9 5 3 Installing processor heat sinks The following figures show the heat sink types used in the BX924 S4 server blade Figure 87 Heat sink for CPU 1 Figure 88 Heat sink for CPU 2 with holder for SSD 1 HDD 1 ...

Page 212: ...essors cognitas Gesellschft für Technik Dokumentation mbH 2013 Pfad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20568 001 fm Figure 89 Special heat sink for CPU 1 only for CPUs with a power consumption of Xeon E5 2637v2 or a Xeon E5 2643v2 ...

Page 213: ...move the protective cover from the heat sink see arrow V CAUTION Ensure not to touch the heat conductive paste on the bottom of the heat sink When reusing a heat sink Ê Ensure that all residual thermal paste has been thoroughly cleaned off the copper surface of the heat sink Ê Apply thermal paste to the processor surface as described in section Applying thermal paste on page 222 ...

Page 214: ...ling the heat sink for CPU 1 Ê Align the heat sink with the four threaded holes of the processor socket Figure 91 Installing the heat sink for CPU 1 Ê Carefully seat the heat sink on the four threaded holes as shown see arrows V CAUTION Ensure that the captive screws on the heat sink are properly seated on the threaded holes Ensure that the heat sink cooling fins match the direction of the airflow...

Page 215: ... on the heat sink to prevent it from tilting until two screws 1 and 2 are fixed Figure 92 Fastening the heat sink for CPU 1 Ê Fasten the four captive screws combihexagon on the heat sink in the following pattern 1 2 3 4 Torque 0 6 Nm not applicable for the Japanese market ...

Page 216: ...cognitas Gesellschft für Technik Dokumentation mbH 2013 Pfad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20568 001 fm Figure 93 Special heat sink for CPU 1 I When using the special heat sink two memory slots 3B and 3D are not available see arrows ...

Page 217: ... Ê Align the edge of the heat sink with the memory slot levers see red line Ê Carefully seat the captive screws of the heat sink onto the four threaded holes see arrows V CAUTION Ensure that the captive screws of the heat sink are properly seated on the threaded holes Ensure that the heat sink cooling fins match the direction of the airflow ...

Page 218: ...CT tim_app tim_local work PICHOL OBJ_DOKU 20568 001 fm Figure 95 Fastening the heat sink for CPU 2 Ê Keep pressing on the heatsink to avoid tilting until the screws 1 and 2 are fixed Ê Fasten the four captive screws combihexagon on the heat sink in the following pattern 1 2 3 4 Torque 0 6 Nm not applicable for the Japanese market ...

Page 219: ...lowing pattern 4 3 2 1 Ê Carefully turn the heat sink back and forth to detach it from the processor I This may be necessary due to the adhesive quality of the thermal paste located between the heat sink and processor V CAUTION Pay special attention not to damage any system board components surrounding the processor socket Ê Lift the heat sink out of the chassis Ê Thoroughly clean residual thermal...

Page 220: ...nk of CPU 2 I In case that no CPU 2 is installed a dummy heat sink will be installed for cooling reasons Figure 97 Removing the dummy heat sink of CPU 2 Ê Loosen the four captive screws combihexagon of the dummy heat sink in the following pattern 4 3 2 1 see figure Ê Lift the dummy heat sink out of the chassis V CAUTION Pay special attention not to damage any system board components surrounding th...

Page 221: ... Ê Install the processor heat sink as described in sections Preparing the heat sink and processor on page 213 and Installing the heat sink for CPU 1 on page 214 9 5 7 Concluding steps Perform the following procedures to complete the task Ê If memory modules have been removed install the memory into their original location as described in section Installing a memory module on page 161 Ê Close the s...

Page 222: ...t follow the instruction provided separately If the processor upgrade or replacement kit contains a new CPU heat sink a thin layer of thermal compound has already been pre applied to its lower surface In this case please proceed with section Installing processor heat sinks on page 211 Figure 98 Thermal paste syringe One thermal compound syringe FSP P304000003 contains thermal paste for two process...

Page 223: ...rea of the syringe up into two equal segments I Add graduation marks to the syringe using a permanent marker to help you apply the thermal paste Figure 99 Applying thermal paste Ê Apply a small streak of thermal paste 1 0 gram see description above to the center of the processor surface as shown V CAUTION Do not mix different types of thermal paste ...

Page 224: ...224 Upgrade and Maintenance Manual BX924 S4 Processors cognitas Gesellschft für Technik Dokumentation mbH 2013 Pfad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20568 001 fm ...

Page 225: ...ive devices ESDs Do not touch the circuitry on boards or soldered parts Hold circuit boards by their metallic areas or edges For further information please refer to chapter Important information on page 29 10 1 Replacing the CMOS battery CMOS memory volatile BIOS memory and the real time clock are powered by a lithium coin cell CMOS battery If the CMOS battery is deleted or falls below minimum vol...

Page 226: ...facing up 10 1 1 Required tools Preliminary and concluding steps Phillips PH2 No 2 screw driver Replacing the battery tool less 10 1 2 Preliminary steps Before replacing the CMOS battery perform the following steps Ê If applicable open the rack door as described in section Opening the rack door on page 51 Ê Locate the desired server blade as described in section Locating the defective server blade...

Page 227: ...intenance Manual 227 System board components 10 1 3 Removing the battery The CMOS battery is located on the system board between the CPUs see following figure Figure 100 Location of the CMOS battery on the system board D3143 ...

Page 228: ...L OBJ_DOKU 20569 001 fm Figure 101 Replacing the CMOS battery Ê Press out on the locking spring to eject the deleted CMOS battery see arrow Ê Carefully pry the deleted CMOS battery out of its socket Ê Remove the CMOS battery I Do not throw the CMOS battery into the trash can Lithium batteries must be disposed of in accordance with local regulations concerning special waste ...

Page 229: ...stalling the CMOS battery Ê At a slight angle fit the new CMOS battery into its socket as shown in the figure above V CAUTION Ensure to insert the CMOS battery with the positive pole label side facing up as shown Ê Press on the CMOS battery until it locks in place Ê Ensure that the locking spring is properly engaged ...

Page 230: ...stalled after the exchange of the CMOS battery the operating system starts regardless of the Automatic inventory collection Automatic Inventory Retrieval Therefore after installed the server blade the BIOS setup utility menu is displayed once and the power supply is cut After the above mentioned operation the server blade returns to usual operation It is not a breakdown though the power supply of ...

Page 231: ...ibed in section Disabling boot watchdog functionality on page 67 Ê If applicable open the rack door as described in section Opening the rack door on page 51 Ê Locate the desired server blade as described in section Locating the defective server blade on page 43 Ê Shut down the server blade as described in section Shutting down the server blade on page 52 Ê Remove the server blade from the system u...

Page 232: ...d C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20569 001 fm 10 2 1 3 Installing the UFM Figure 103 UFM kit Pre assembled UFM flash module kit S26361 F3514 V3 1 2 GB UFM SLC A3C40104433 2 UFM spacer A3C40109081 I This black spacer will not be used A white spacer is already mounted instead 3 UFM nylon screw A3C40109082 ...

Page 233: ...BX924 S4 Upgrade and Maintenance Manual 233 System board components Figure 104 UFM mounting location UFM mounting location on the system board 1 UFM connector 2 UFM spacer ...

Page 234: ... see arrow 10 2 1 4 Concluding steps Perform the following procedures to complete the task Ê Close the server blade as described in section Closing the server blade on page 56 Ê Reinstall and secure it in the system unit as described in section Installing the server blade in the system unit on page 57 Ê If applicable close the rack door as described in section Closing the rack door on page 62 Ê En...

Page 235: ... CD to setup the ESXi configuration Proceed as follows Ê Switch on the server blade Ê Right after switching on the server insert the Recovery Tool CD into the DVD drive and close the drive tray Ê The server should now boot from the Recovery Tool CD Ê Follow the on screen instructions 10 2 2 Removing the UFM 10 2 2 1 Required tools Preliminary and concluding steps tool less Removing the UFM Phillip...

Page 236: ...tchdog functionality as described in section Disabling boot watchdog functionality on page 67 Ê If applicable open the rack door as described in section Opening the rack door on page 51 Ê Locate the desired server blade as described in section Locating the defective server blade on page 43 Ê Shut down the server blade as described in section Shutting down the server blade on page 52 Ê Remove the s...

Page 237: ...efully The UFM spacer remains on the system board 10 2 2 4 Concluding steps Perform the following procedures to complete the task Ê Close the server blade as described in section Closing the server blade on page 56 Ê Reinstall and secure the server blade in the system unit as described in section Installing the server blade in the system unit on page 57 Ê If applicable close the rack door as descr...

Page 238: ...illips PH0 No 0 screw driver 10 2 3 2 Preliminary steps Before replacing the UFM perform the following steps Ê Disable BitLocker functionality as described in section Disabling BitLocker functionality on page 66 Ê Disable boot watchdog functionality as described in section Disabling boot watchdog functionality on page 67 Ê If applicable open the rack door as described in section Opening the rack d...

Page 239: ...n section Removing the server blade from the system unit on page 54 Ê Open the server blade as described in section Opening the server blade on page 55 10 2 3 3 Removing the UFM Figure 107 Removing the UFM Ê Grasp the UFM on its corners see arrows then pull it out gradually and carefully The UFM spacer remains on the system board ...

Page 240: ...fad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20569 001 fm 10 2 3 4 Re installing the UFM Figure 108 Preparing the new UFM Ê Remove the nylon screw 1 and the black spacer 2 from the new UFM Ê Fit the new UFM on the UFM connector and the remaining UFM spacer Ê Secure the UFM to the UFM spacer with the nylon screw ...

Page 241: ...ustomer information e g IP address license numbers After replacing the UFM hand the defective UFM over to the customer If the customer requests disposal of the defective UFM proceed as follows Figure 109 Destroying the defective UFM Ê Use a pair of combination pliers 1 and flat nose pliers 2 to break the UFM in half as shown ...

Page 242: ...or on page 62 Ê Enable boot watchdog functionality as described in section Enabling boot watchdog functionality on page 84 Ê If BitLocker functionality is used and has been disabled before starting the maintenance task re enable BitLocker as described in section Enabling BitLocker functionality on page 91 10 2 3 6 Software configuration The UFM delivery set includes the Recovery Tool CD to setup t...

Page 243: ...insert For the Japanese market TPM module fixing tool S26361 F3552 L909 10 3 1 2 Preliminary steps Before installing the TPM perform the following steps Ê Disable BitLocker functionality as described in section Disabling BitLocker functionality on page 66 Ê If applicable open the rack door as described in section Opening the rack door on page 51 Ê Locate the desired server blade as described in se...

Page 244: ... on page 52 Ê Remove the server blade from the rack as described in section Removing the server blade from the system unit on page 54 Ê Open the server blade as described in section Opening the server blade on page 55 10 3 1 3 Installing the TPM Figure 110 TPM kit TPM kit S26361 F3299 E2 1 TPM module S26361 D2727 A10 2 TPM spacers I Use the black TPM spacer The white TPM spacer is not used in this...

Page 245: ...BX924 S4 Upgrade and Maintenance Manual 245 System board components Figure 111 TPM mounting location TPM mounting location on the system board 1 TPM connector 2 Cut out for TPM spacer ...

Page 246: ...tion mbH 2013 Pfad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20569 001 fm Figure 112 Installing the TPM spacer Ê Snap the TPM spacer into the cut out in the system board see circle Figure 113 TPM bit insert Ê Attach the TPM bit insert or TPM module fixing tool Japanese market to a bit screw driver ...

Page 247: ...s Perform the following procedures to complete the task Ê Close the server blade as described in section Closing the server blade on page 56 Ê Reinstall and secure the server blade in the system unit as described in section Installing the server blade in the system unit on page 57 Ê If applicable close the rack door as described in section Closing the rack door on page 62 Ê Enable TPM in the syste...

Page 248: ...Ê Select the Trusted Computing submenu Ê Set the TPM Support and TPM State settings to Enabled Ê Under Pending TPM operation select the desired TPM operation mode Ê Save your changes and exit the BIOS I For detailed information on how to access the BIOS and modify settings refer to the D3143 BIOS Setup Utility for PRIMERGY BX924 S4 reference manual available online Ê If BitLocker functionality is ...

Page 249: ...BIOS V CAUTION Before replacing the server blade ask the customer whether TPM functionality is used If the customer is using TPM functionality remove the TPM from the old system board and install it on the new system board Advise your contact persons that they must provide you with TPM backup copies For security reasons the TPM must be restored re saved by the customer After installing a new serve...

Page 250: ...rd components cognitas Gesellschft für Technik Dokumentation mbH 2013 Pfad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20569 001 fm Removing the TPM thin slotted screw driver 2 x 0 4 mm For the Japanese market Dedicated TPM screw driver CWZ8291A ...

Page 251: ...ion Ê To turn off BitLocker and decrypt the volume click Turn Off BitLocker and then click Decrypt the volume I Decrypting the volume may be time consuming By decrypting the volume all of the information stored on that computer is decrypted For further information on how to disable BitLocker drive encryption please refer to the Microsoft Knowledge Base Fujitsu service partners will find additional...

Page 252: ...age 43 Ê Shut down the server blade as described in section Shutting down the server blade on page 52 Ê Remove the server blade from the system unit as described in section Removing the server blade from the system unit on page 54 Ê Open the server blade as described in section Opening the server blade on page 55 10 3 2 3 Removing the TPM Ê Remove the system board as described in section Removing ...

Page 253: ...ing it impossible to loosen the screw Ê Remove the TPM screw Ê Remove the defective TPM on the upper side of the system board 10 3 2 4 Concluding steps Perform the following procedures to complete the task Ê Close the server blade as described in section Closing the server blade on page 56 Ê Reinstall and secure the server blade in the system unit as described in section Installing the server blad...

Page 254: ...fter installing a new system board the TPM must be enabled You may not clear the TPM data If the contact persons DO NOT have a backup copy available inform them that replacing the TPM will cause to lose all data 10 3 3 1 Required tools Preliminary and concluding steps tool less Removing the system board Phillips PH2 No 2 screw driver Replacing the TPM Bit screw driver TPM bit insert thin slotted s...

Page 255: ... rack door on page 51 Ê Locate the desired server blade as described in section Locating the defective server blade on page 43 Ê Shut down the server blade as described in section Shutting down the server blade on page 52 Ê Remove the server blade from the system unit as described in section Removing the server blade from the system unit on page 54 Ê Open the server blade as described in section O...

Page 256: ... the system board Ê Re install the TPM as described in section Installing the TPM on page 243 10 3 3 5 Concluding steps Perform the following procedures to complete the task Ê Install the system board as described in section Installing the system board on page 268 Ê Close the server blade as described in section Closing the server blade on page 56 Ê Reinstall and secure the server blade in the sys...

Page 257: ...tenance Manual 257 System board components Ê If BitLocker functionality is used and has been disabled before starting the maintenance task re enable BitLocker as described in section Enabling BitLocker functionality on page 91 ...

Page 258: ... enables the SAS functionality of the Onboard controller 10 4 1 Required tools Preliminary and concluding steps tool less Main steps tool less 10 4 2 Preliminary steps Perform the following procedures Ê Opening the rack door on page 51 Ê Locating the defective server blade on page 43 Ê Shutting down the server blade on page 52 Ê Removing a server blade on page 53 Ê Opening the server blade on page...

Page 259: ...ual 259 System board components 10 4 3 Removing the defective Onboard SAS Enabling Key Figure 117 Removing the Onboard SAS Enabling Key Ê Pull the Onboard SAS Enabling Key from the connector on the system board using the green touch point ...

Page 260: ...ft für Technik Dokumentation mbH 2013 Pfad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20569 001 fm 10 4 4 Installing the new Onboard SAS Enabling Key Figure 118 Installing the Onboard SAS Enabling Key Ê Connect the Onboard SAS Enabling Key to the connector on the system board ...

Page 261: ...ystem board components Figure 119 Routing the holding rope Ê Route the holding rope see oval of the onboard SAS enabling key into the gap between its own socket and the connector of the mezzanine bracket on the system board as shown in the figure ...

Page 262: ... mbH 2013 Pfad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20569 001 fm 10 4 5 Concluding steps Perform the following procedures Ê Closing the server blade on page 56 Ê Installing the server blade in the system unit on page 57 Ê Switching on the server blade on page 60 Ê Closing the rack door on page 62 ...

Page 263: ...TPM on page 254 The TPM is activated in the system BIOS V CAUTION Before replacing the system board ask the customer whether TPM functionality is used If the customer is using TPM functionality remove the TPM from the old system board and install it on the new system board Advise your contact persons that they must provide you with TPM backup copies For security reasons the TPM must be restored re...

Page 264: ...contains system information like server name and model housing type serial number and manufacturing data Preliminary and concluding steps tool less Replacing the system board Phillips PH2 No 2 screw driver Replacing the system board Magnifying glass for inspecting CPU socket springs recommended If a TPM module is installed Bit screw driver TPM bit insert thin slotted screw driver 2 x 0 4 mm For th...

Page 265: ...ot watchdog functionality on page 67 Ê If applicable open the rack door as described in section Opening the rack door on page 51 Ê Locate the desired server blade as described in section Locating the defective server blade on page 43 Ê Shut down the server blade as described in section Shutting down the server blade on page 52 Ê Remove the server blade from the rack as described in section Removin...

Page 266: ...ocessor heat sinks on page 219 I Leave the processor on the defective board for now Memory modules refer to section Removing processor heat sinks on page 219 I Ensure to take note of the memory modules mounting positions for reassembly Mezzanine cards refer to the section Removing processor heat sinks on page 219 I Ensure to take note of the controllers mounting positions and cable connections for...

Page 267: ...d A Ê Carefully shift the system board to the rear see arrow in order to detach it from the centering bolts see circles Ê Hold the system board by the memory module ejectors and at a slight angle lift it out of the chassis Ê If applicable remove the TPM as described in section Removing the TPM on page 252 ...

Page 268: ...em board 10 5 4 1 Mounting the system board Ê Hold the new system board by the memory module ejectors Ê At a slight angle lower the system board into the chassis Figure 122 Detaching the system board B Ê Lower the system board onto the centering bolts see circles Ensure that the system board is properly seated on the centering bolts Ê Carefully shift the system board towards the server front as fa...

Page 269: ...e refer to section Onboard settings on page 279 10 5 5 Concluding steps Ê If applicable reinstall the air cowls as described in section Handling of memory air cowls on page 170 Ê Reinstall all remaining system board components as shown in the related sections Mezzanine cards with carrier refer to section Installing mezzanine cards on page 134 I Install all expansion cards into their original slots...

Page 270: ... in section Verifying the system time settings on page 86 Ê Inform the customer about changed WWN and MAC addresses For further information refer to section Looking up changed MAC WWN addresses on page 92 Ê After replacing the system board in a server blade running Linux OS update the MAC address of the onboard network controller in the related NIC definition file as described in section Updating ...

Page 271: ... Saving BIOS settings on page 65 Ê Save the iRMC settings as described in section Saving iRMC settings on page 65 Ê Verify if your backup software solution requires preparative configuration before starting the procedure as described in section Verifying and configuring the backup software solution on page 81 Ê Exit all applications and shut down the server blade correctly If your operating system...

Page 272: ...the new server blade see section Replacing memory modules on page 167 Ê If an UFM is installed remove it from the defective server blade and install it into the new server blade see section USB Flash Module UFM on page 231 Ê If an TPM is installed remove it from the defective server blade and install it into the new server blade see section Trusted Platform Module TPM on page 243 Ê If an Onboard S...

Page 273: ...ent network settings before replacing the controller Ê Insert the server blade in the system unit see section Installing the server blade in the system unit on page 57 Ê Restore the BIOS settings as described in section Restoring BIOS settings on page 78 Ê Restore the iRMC settings as described in section Restoring iRMC settings on page 78 I In order to enable ServerView Operations Manager and Ser...

Page 274: ...274 Upgrade and Maintenance Manual BX924 S4 Server blade cognitas Gesellschft für Technik Dokumentation mbH 2013 Pfad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20570 001 fm ...

Page 275: ...ance Manual 275 12 Appendix 12 1 Mechanical overview 12 1 1 Server blade front Figure 124 PRIMERGY BX924 S4 server blade front Pos Component 1 ID card 2 Y cable connector 3 Front panel buttons and indicators 4 USB connector 0 2 1 ...

Page 276: ...RIMERGY BX924 S4 interior example no CPU 2 installed Pos Component 1 Mounting socket for SSD 0 2 Mezzanine card 1 slot 3 Memory modules for CPU 1 4 CPU 1 processor heat sink 5 CMOS battery 6 CPU 2 processor not installed 7 Dummy modules for CPU 2 8 Mounting socket for SSD 1 when no CPU 2 is installed 9 Dummy modules for CPU 2 10 Memory modules for CPU 1 11 Onboard SAS Enabling Key SCU key 12 Onboa...

Page 277: ...al 277 Appendix 12 2 Configuration tables 12 2 1 Memory configuration table Please refer to chapter Main memory on page 147 12 2 2 Mezzanine card configuration table Please refer to chapter Population rules for mezzanine cards on page 128 ...

Page 278: ...ystem board D3143 No Print Description A SSD 0 Connector for SSD 0 B Mezzanine card 1 Connector for mezzanine card 1 C Memory modules Connectors for memory modules D SSD 1 Connector for SSD 1 F Onboard USB Connector for onboard USB port G Mezzanine card 2 Connector for mezzanine card 2 CPU 1 CPU 2 Batt TPM UFM DIMM 3H DIMM 2H DIMM 1H DIMM 3G DIMM 1G DIMM 2G DIMM 1E DIMM 2E DIMM 3E DIMM 1F DIMM 3F ...

Page 279: ...Description Switch 1 On CMOS clear Switch 2 On Clear Password Switch 3 On System BIOS recovery NVRAM clear Switch 4 On ME_RCVR for service personnel only CPU 1 CPU 2 Batt TPM UFM DIMM 3H DIMM 2H DIMM 1H DIMM 3G DIMM 1G DIMM 2G DIMM 1E DIMM 2E DIMM 3E DIMM 1F DIMM 3F DIMM 2F DIMM 1C DIMM 3D DIMM 2D DIMM 2C DIMM 3C DIMM 1D DIMM 3B DIMM 2B DIMM 1B DIMM 3A DIMM 1A DIMM 2A 1 2 3 4 ON ...

Page 280: ...tem unit on page 54 Ê Open the server blade as described in section Opening the server blade on page 55 Ê Press the Indicate CSS button 1 to highlight defective components Component LEDs I The LEDs C to G are visible from the outside All other LEDs are only visible if the server blade has been opened In order to access memory LEDs B the air cowls need to be removed see section Removing the air cow...

Page 281: ... an active network connection E Fabric 1 off Fabric 1 no network connection green flashing Fabric 1 network connection green on Fabric 1 an active network connection F CSS off System is ok yellow flashing An error was detected that you can fix yourself with the CSS concept yellow on A prefailure event was detected for a CSS component that you can fix yourself for reasons of precaution with the CSS...

Page 282: ...itas Gesellschft für Technik Dokumentation mbH 2013 Pfad C Programme FCT tim_app tim_local work PICHOL OBJ_DOKU 20571 001 fm I CPU 1 off CPU 1 operational green on CPU 1 failure J CPU 2 off CPU 2 operational green on CPU 2 failure Indicator Status Description ...

Page 283: ...4 Upgrade and Maintenance Manual 283 Appendix 12 3 2 Connectors and indicators on the front panel 12 3 2 1 Front panel connectors Figure 129 Front panel connectors Pos Component 1 Y cable port 2 USB connector ...

Page 284: ...01 fm 12 3 2 2 Front panel indicators Figure 130 I O panel indicators Indicator Status Description 1 Fabric 3 4 off Fabric 3 4 no network connection green flashing Fabric 3 4 an active network connection green on Fabric 3 4 network connection 2 Fabric 2 off Fabric 2 no network connection green flashing Fabric 2 an active network connection green on Fabric 2 network connection ...

Page 285: ...l error indicator off No critical event orange on A prefailure event has been detected that requires precautionary service intervention orange flashing An error was detected that requires service intervention 6 ID indicator blue on Server has been highlighted using management blade web interface ServerView Operations Manager or the ID button on the front panel for easy identification 7 Power indic...

Page 286: ...ade from the system unit as described in section Removing a server blade on page 53 Ê Take the server blade down to its minimum startup configuration Ê Reinstall the server blade in the system unit as described in section Installing the server blade in the system unit on page 57 Ê Switch on the server blade as described in section Switching on the server blade on page 60 Ê Login to the management ...

Reviews:

No comments