manualshive.com logo in svg

Dell DR4000 Restore Manager, Owner'S Manual, Page: 119 / 156

Share
Download
Dell DR4000 Restore Manager Owner'S Manual Download Page 119

Installing System Components

119

13

If applicable, install the iDRAC6 Enterprise card. See "Installing an 

iDRAC6 Enterprise Card" on page 94.

14

If applicable, install the iDRAC6 Express card. See "Installing an iDRAC6 

Express Card" on page 91.

15

Replace the cooling shroud. See "Installing the Cooling Shroud" on 

page 63.

16

Close the system. See "Closing the System" on page 61.

17

Reconnect the system to its electrical outlet and turn the system on, 

including any attached peripherals.

18

If applicable, replace the front bezel. See "Removing the Front Bezel" on 

page 59.

book.book  Page 119  Friday, January 6, 2012  2:11 PM

Summary of Contents for DR4000 Restore Manager

Page 1: ...Dell DR4000 Systems Owner s Manual Regulatory Model E13S Series Regulatory Type E13S001 ...

Page 2: ... to change without notice 2012 Dell Inc All rights reserved Reproduction of these materials in any manner whatsoever without the written permission of Dell Inc is strictly forbidden Trademarks used in this text Dell the DELL logo and PowerVault are trademarks of Dell Inc Other trademarks and trade names may be used in this publication to refer to either the entities claiming the marks and names or...

Page 3: ...l Features and Indicators 15 NIC Indicator Codes 17 NVRAM Card Indicator Codes 17 Power Indicator Codes 18 Diagnostic Lights 19 System Messages 22 Warning Messages 37 Diagnostics Messages 37 Alert Messages 37 Other Information You May Need 38 2 Using the System Setup Program and UEFI Boot Manager 39 Entering the System Setup Program 39 Responding to Error Messages 39 ...

Page 4: ...nt Screen 48 System Security Screen 48 Exit Screen 50 System and Setup Password Features 51 Using the System Password 51 Using the Setup Password 53 Embedded System Management 54 Baseboard Management Controller Configuration 55 Entering the BMC Setup Module 55 iDRAC Configuration Utility 56 Entering the iDRAC Configuration Utility 56 3 Installing System Components 57 Recommended Tools 57 Inside th...

Page 5: ...g a Hot Swap Hard Drive 65 Removing a Hard Drive From a Hard Drive Carrier 66 Installing a Hard Drive Into a Hard Drive Carrier 67 Internal Hard Drives 67 Removing an Internal Hard Drive Bay 67 Installing an Internal Hard Drive Bay 69 Removing an Internal Hard Drive From the Internal Hard Drive Bay 69 Installing a Hard Drive Into a Hard Drive Bay 70 Cooling Fans 71 Removing a Cooling Fan 71 Instal...

Page 6: ...stalling an Expansion Card Riser 86 Integrated Storage Controller Card 86 Removing the Storage Controller Card 87 Installing the Storage Controller Card 89 iDRAC6 Express Card 90 Removing an iDRAC6 Express Card 90 Installing an iDRAC6 Express Card 91 iDRAC6 Enterprise Card 93 Removing an iDRAC6 Enterprise Card 93 Installing an iDRAC6 Enterprise Card 94 Processors 95 Removing a Processor 95 Install...

Page 7: ...wer Distribution Board 113 Removing the Power Distribution Board 113 Installing the Power Distribution Board 115 System Board 116 Removing the System Board 116 Installing the System Board 118 4 Troubleshooting Your System 121 Safety First For You and Your System 121 Troubleshooting System Startup Failure 121 Troubleshooting External Connections 121 Troubleshooting the Video Subsystem 122 Troublesh...

Page 8: ...eshooting an Internal Hard Drive 132 Troubleshooting a Storage Controller 133 Troubleshooting Expansion Cards 134 Troubleshooting Processors 136 5 Running the System Diagnostics 139 Embedded System Diagnostics Features 139 When to Use the Embedded System Diagnostics 139 Running the Embedded System Diagnostics 140 Embedded System Diagnostics Testing Options 140 Using the Custom Test Options 140 Sel...

Page 9: ...Contents 9 6 Jumpers and Connectors 143 System Board Jumpers 143 System Board Connectors 144 Disabling a Forgotten Password 146 7 Getting Help 149 Contacting Dell 149 Index 151 ...

Page 10: ...10 Contents ...

Page 11: ...Enters the BIOS Boot Manager depending on the system s boot configuration See Using the System Setup Program and UEFI Boot Manager on page 39 F12 Starts Preboot eXecution Environment PXE boot Ctrl E Enters the Baseboard Management Controller BMC or Integrated Dell Remote Access Controller iDRAC Configuration Utility which allows access to the system event log SEL and configuration of remote access...

Page 12: ...n The power button controls the DC power supply output to the system When the system bezel is installed the power button is not accessible NOTE When powering on the system the video monitor can take from several seconds to over 2 minutes to display an image depending on the amount of memory installed in the system NOTE On ACPI compliant operating systems turning off the system using the power butt...

Page 13: ...n be used to locate a particular system within a rack When one of these buttons is pushed the blue system status indicator on the back blinks until one of the buttons is pushed again 5 Hard drive Twelve 3 5 inch hot swappable SAS drives 6 System identification panel A slide out panel for system information including the Express Service tag embedded NIC MAC address and iDRAC6 Enterprise card MAC ad...

Page 14: ... Condition Blinks green two times per second Identify drive preparing for removal Off Drive ready for insertion or removal NOTE The drive status indicator remains off until all hard drives are initialized after system power is applied Drives are not ready for insertion or removal during this time Blinks green amber and off Drive predicted failure Blinks amber four times per second Drive failed 1 2...

Page 15: ...dicator Button or Connector Icon Description 1 Serial connector Connects a serial device to the system 2 Video connector Connects a VGA display to the system 3 iDRAC6 Enterprise port Dedicated management port for the iDRAC6 Enterprise card 4 VFlash media slot optional Connects an external SD memory card for the iDRAC6 Enterprise card 5 USB connectors 2 Connects USB devices to the system The ports ...

Page 16: ...lue during normal system operation Both the systems management software and the identification buttons located on the front and back of the system can cause the indicator to blink blue to identify a particular system Lights amber when the system needs attention due to a problem 10 System identification button The identification buttons on the front and back panels can be used to locate a particula...

Page 17: ...nt of power loss 1 link indicator 2 activity indicator Indicator Indicator Code Link and activity indicators are off The NIC is not connected to the network Link indicator is green The NIC is connected to a valid network link at 1000 Mbps Link indicator is amber The NIC is connected to a valid network link at 10 100 Mbps Activity indicator is green Network data is being sent or received Table 1 1 ...

Page 18: ...problem with the power supply Steady Off Card is activated and operational Off Steady Error due to one or more of the following Supercapacitors are not connected Flash module is either not connected or is not responding POST memory test has failed Blinking Off Steady Blinking Data activity from host Table 1 2 LED States during Power Loss Green Indicator Amber Indicator Backup Progress Blinking Off...

Page 19: ... that matches the capacity of the other installed power supply Figure 1 5 Power Supply Status Indicator Diagnostic Lights The four diagnostic indicator lights on the system front panel display error codes during system startup Table 1 3 lists the causes and possible corrective actions associated with these codes A highlighted circle indicates the light is on a non highlighted circle indicates the ...

Page 20: ... button Information only BIOS checksum failure detected system is in recovery mode See Getting Help on page 149 Possible processor failure See Troubleshooting Processors on page 136 Memory failure See Troubleshooting System Memory on page 129 Possible expansion card failure See Troubleshooting Expansion Cards on page 134 Possible video failure See Getting Help on page 149 Hard drive failure Ensure...

Page 21: ...esource and or system board hardware failure See Getting Help on page 149 Possible system resource configuration error See Contacting Dell on page 149 Other failure Ensure that the hard drives are properly connected See Troubleshooting a Hard Drive on page 131 for the appropriate drive installed in your system If the problem persists see Getting Help on page 149 Table 1 3 Diagnostic Indicator Code...

Page 22: ...sibly a faulty or removed memory module The Advanced ECC setting has been disabled Check other messages for a faulty memory module Reconfigure the memory modules for Advanced ECC mode See System Memory on page 76 Alert Advanced ECC Memory Mode disabled Memory configuration does not support Advanced ECC Memory Mode Advanced ECC Memory Mode was enabled in the system setup program but the current con...

Page 23: ...C power to the system for 10 seconds and restart the system Alert Node Interleaving disabled Memory configuration does not support Node Interleaving The memory configuration does not support node interleaving or the configuration has changed for example a memory module has failed so that node interleaving cannot be supported The system still runs but without node interleaving Ensure that the memor...

Page 24: ...ert Redundant memory disabled Memory configuration does not support redundant memory Memory sparing or memory mirroring was enabled in the system setup program but the current configuration does not support redundant memory A memory module may be faulty Check the memory modules for failure See Troubleshooting System Memory on page 129 Reset the memory setting if appropriate See Using the System Se...

Page 25: ... See Figure 6 1 for jumper location Restart the system and re enter the BIOS settings See Using the System Setup Program and UEFI Boot Manager on page 39 CPU set to minimum frequency The processor speed may be intentionally set lower for power conservation If not an intentional setting check any other system messages for possible causes CPU x installed with no memory Memory modules are required bu...

Page 26: ...ram to change the boot mode as needed The system failed to boot because UEFI boot mode is enabled in BIOS and the boot operating system is non UEFI Ensure that the boot mode is set correctly and that the proper bootable media is available See Using the System Setup Program and UEFI Boot Manager on page 39 Decreasing available memory Faulty or improperly installed memory modules Reseat the memory m...

Page 27: ...tional See Troubleshooting a USB Device on page 122 Gate A20 failure Faulty keyboard controller faulty system board See Getting Help on page 149 Invalid configuration information please run SETUP program An invalid system configuration caused a system to stop responding Run the System Setup program and review the current settings See Using the System Setup Program and UEFI Boot Manager on page 39 ...

Page 28: ...emory module disabled Ensure that the memory modules are installed in a valid configuration See General Memory Module Installation Guidelines on page 76 Memory Initialization Warning Memory size may be reduced Invalid memory configuration The system runs but with less memory than is physically available Ensure that the memory modules are installed in a valid configuration See General Memory Module...

Page 29: ...airs must be matched in size and geometry The memory configuration does not match the setting in BIOS The BIOS setting has been disabled Reconfigure the memory modules for Memory Mirroring mode See System Memory on page 76 No boot device available Faulty or missing hard drive or hard drive subsystem or no bootable USB key installed Use a bootable USB key or hard drive If the problem persists see T...

Page 30: ...y installed PCIe card in the specified slot Reseat the PCIe card in the specified slot number See Troubleshooting Expansion Cards on page 134 If the problem persists see Getting Help on page 149 Plug Play Configuration Error Error encountered in initializing PCIe device faulty system board Install the NVRAM_CLR jumper in the clear position pins 1 and 3 and reboot the system See Figure 6 1 for jump...

Page 31: ... Seek error Seek operation failed Faulty hard drive USB device or USB medium Replace the USB medium or device Ensure that the USB or SAS backplane cables are properly connected See Troubleshooting a USB Device on page 122 or Troubleshooting a Hard Drive on page 131 for the appropriate drive s installed in your system Shutdown failure General system error See Getting Help on page 149 Sparing mode d...

Page 32: ...oting System Memory on page 129 The following DIMMs should match in geometry x x Invalid memory configuration The specified memory modules do not match in size number of ranks or number of data lanes Ensure that the memory modules are installed in a valid configuration See General Memory Module Installation Guidelines on page 76 The following DIMMs should match in rank count x x The following DIMM...

Page 33: ...e the system battery See System Battery on page 100 Timer chip counter 2 failed Faulty system board See Getting Help on page 149 TPM configuration operation honored System will now reset A TPM configuration command has been entered The system reboots and executes the command Information only TPM configuration operation is pending Press I to Ignore OR M to Modify to allow this change and reset the ...

Page 34: ...uide for instructions on performing a field replacement of the flash memory Unexpected interrupt in protected mode Improperly seated memory modules or faulty keyboard mouse controller chip Reseat the memory modules See Troubleshooting System Memory on page 129 If the problem persists see Getting Help on page 149 Unsupported CPU combination Unsupported CPU stepping detected Processor s is not suppo...

Page 35: ...ory detected DIMM s installed in the following slot are not available when in 128 bit advanced ECC mode x x x The memory configuration is not optimal for Advanced ECC Memory Mode Modules in the specified slots are unused Reconfigure the memory for Advanced ECC Memory Mode or change the memory mode to Optimized or Sparing in the BIOS setup screen See System Memory on page 76 Warning A fatal error h...

Page 36: ...ower Supplies on page 74 Warning PSU mismatch PSU redundancy lost Check PSU A high output power supply and an Energy Smart power supply are installed in the system at the same time Install two high output or two Energy Smart power supplies in the system You can also run the system on one power supply until you can obtain two power supplies of the same type See Troubleshooting Power Supplies on pag...

Page 37: ...operating system Diagnostics Messages The system diagnostic utilities may issue messages if you run diagnostic tests on your system See Running the Embedded System Diagnostics on page 140 for more information about system diagnostics Alert Messages Systems management software generates alert messages for your system Alert messages include information status warning and failure messages for drive t...

Page 38: ...Getting Started Guide provides an overview of system features setting up your system and technical specifications Any media that ships with your system that provides documentation and tools for configuring and managing your system including those pertaining to the operating system system management software system updates and system components that you purchased with your system NOTE Always check ...

Page 39: ... restart your system 2 Press F2 after you see the following message F2 System Setup NOTE The system does not respond until the USB keyboard is active If your operating system begins to load before you press F2 allow the system to finish booting and then restart your system and try again Responding to Error Messages If an error message appears while the system is booting make a note of the message ...

Page 40: ...until you restart the system Keys Action Up arrow or Shift Tab Moves to the previous field Down arrow or Tab Moves to the next field Enter Spacebar left and right arrows Cycles through the settings in a field In many fields you can also type the appropriate value Esc Exits the System Setup program and restarts the system if any changes were made F1 Displays the System Setup program s help file ...

Page 41: ...lts are listed under their respective options in the following sections where applicable Option Description System Time Sets the time on the system s internal clock System Date Sets the date on the system s internal calendar Memory Settings Displays information related to installed memory See Memory Settings Screen on page 43 Processor Settings Displays information related to processors speed cach...

Page 42: ...3 for more information Keyboard NumLock On default Determines whether your system starts up with the NumLock mode activated on 101 or 102 key keyboards does not apply to 84 key keyboards Report Keyboard Errors Report default Enables or disables reporting of keyboard errors during the POST Select Report for host systems that have keyboards attached Select Do Not Report to suppress all error message...

Page 43: ...ndently of each other for improved memory performance When set to Mirror Mode memory mirroring is enabled When set to Advanced ECC Mode two controllers are joined in 128 bit mode running multi bit advanced ECC For information about the memory modes see System Memory on page 76 NOTE The Spare Mode option may not be present on all systems Node Interleaving Disabled default If this field is Enabled m...

Page 44: ...ory access NOTE Disable this option for applications that require high utilization of random memory access Hardware Prefetcher Enabled default Enables or disables the hardware prefetcher Execute Disable Enabled default Enables or disables Execute Disable Memory Protection Technology Number of Cores per Processor All default If set to All the maximum number of cores in each processor is enabled Tur...

Page 45: ...on type for a USB flash drive Hard disk allows the USB flash drive to act as a hard drive Auto automatically chooses an emulation type Boot Sequence Retry Disabled default If this field is enabled and the system has failed to boot the system reattempts to boot after 30 seconds Option Description Integrated SAS Controller Enabled default Enables or disables the integrated storage controller User Ac...

Page 46: ...llowed to initialize the timer When Disabled the timer is not initialized NOTE This feature is usable only with operating systems that support WDAT implementations of the Advanced Configuration and Power Interface ACPI 3 0b specification I OAT DMA Engine Disabled default Enables or disables the I O acceleration technology I OAT This feature should only be enabled if the hardware and software suppo...

Page 47: ...ion by SOL configure the same port address for console redirection and the serial device External Serial Connector Serial Device1 default Specifies whether Serial Device 1 Serial Device 2 or Remote Access Device has access to the external serial connector NOTE Only Serial Device 2 can be used for Serial Over LAN SOL To use console redirection by SOL configure the same port address for console redi...

Page 48: ...formance based on processor utilization Active Power Controller sets the CPU power to System DBPM the fan power to Minimum Power and the memory power to Maximum Performance The BIOS sets the processor performance based on processor utilization Maximum Performance sets all fields to Maximum Performance CPU Power and Performance Management Options are OS DBPM System DBPM Maximum Performance or Minim...

Page 49: ...es the pre boot measurements to the TPM during POST If On without Pre boot Measurements the system reports the TPM to the operating system and bypasses pre boot measurements TPM Activation No Change default When set to Activate the TPM is enabled to default settings When set to Deactivate the TPM is disabled The No Change state initiates no action The operational state of the TPM remains unchanged...

Page 50: ...nnel or by the operating system s documentation Pressing this button halts the operating system and displays a diagnostic screen Enables or disables the NMI feature AC Power Recovery Last default Determines how the system reacts when power is restored If set to Last the system returns to the last power state On turns on the system after power is restored Off allows the system to remain off after p...

Page 51: ...ord is assigned System Password is Enabled If Password Status is Unlocked you can change the system password If Locked you cannot change the system password Disabling the password jumper on the system board sets System Password to Disabled and you cannot change or enter a new system password When a system password is not assigned and the password jumper on the system board is in the enabled positi...

Page 52: ...ity enabled 1 Turn on or reboot your system by pressing Ctrl Alt Del 2 Type your password and press Enter To disable the password security 1 Turn on or reboot your system by pressing Ctrl Alt Del 2 Type your password and press Ctrl Enter When Password Status is Locked you must type the password and press Enter when prompted at reboot If an incorrect system password is entered the system displays a...

Page 53: ...p password only when the Setup Password is Not Enabled To assign a setup password highlight the Setup Password option and press the or key The system prompts you to enter and verify the password NOTE The setup password can be the same as the system password If the two passwords are different the setup password can be used as an alternate system password The system password cannot be used in place ...

Page 54: ... program and select the System Security 2 Highlight Setup Password press Enter to access the setup password window Press Enter twice to clear the existing setup password The setting changes to Not Enabled 3 If you want to assign a new setup password perform the steps in Assigning a Setup Password on page 53 Embedded System Management The Lifecycle Controller is an embedded utility that enables sys...

Page 55: ...gement Controller BMC utility is replaced by the iDRAC6 utility The BMC enables configuring monitoring and recovery of systems remotely BMC provides the following features Uses the system s integrated NIC Enables fault logging and SNMP alerting Provides access to system event log and sensor status Allows control of system functions including power on and off Functions independently of the system s...

Page 56: ...nable or disable the iDRAC6 local area network LAN through the dedicated iDRAC6 Enterprise card port or the embedded NICs Enable or disable IPMI over LAN Enable a LAN Platform Event Trap PET destination Attach or detach the Virtual Media devices Change the administrative username and password and manage user privileges View System Event Log SEL messages or clear messages from the log For additiona...

Page 57: ...m CAUTION Many repairs may only be done by a certified service technician You should only perform troubleshooting and simple repairs as authorized in your product documentation or as directed by the online or telephone service and support team Damage due to servicing that is not authorized by Dell is not covered by your warranty Read and follow the safety instructions that came with the product ...

Page 58: ...1 Inside the System 1 cooling fan 2 internal hard drives 2 3 expansion card riser 4 power supply bays 2 5 cooling shroud 6 heat sink processor 2 7 memory modules 8 8 system cooling fans 4 9 SAS backplane 10 hard drives 12 5 4 8 9 10 7 1 3 6 2 ...

Page 59: ...k the bezel 2 Lift the release latch next to the key lock 3 Rotate the left end of the bezel away from the front panel 4 Unhook the right end of the bezel and pull the bezel away from the system Figure 3 2 Removing and Installing the Front Bezel 1 release latch 2 keylock 3 bezel 4 hinge tab 3 2 1 4 ...

Page 60: ...ing and simple repairs as authorized in your product documentation or as directed by the online or telephone service and support team Damage due to servicing that is not authorized by Dell is not covered by your warranty Read and follow the safety instructions that came with the product Opening the System 1 Turn off the system and attached peripherals and disconnect the system from the electrical ...

Page 61: ... two hooks on the back edge of the cover fit the corresponding tabs on the back edge of the chassis See Figure 3 3 3 Slide the cover towards the front of the chassis and press down the latch 4 Rotate the latch release lock in the clockwise direction to secure the cover 5 Reconnect the system and peripherals to their electrical outlets and turn on the system 1 system cover latch 2 latch release loc...

Page 62: ...repairs as authorized in your product documentation or as directed by the online or telephone service and support team Damage due to servicing that is not authorized by Dell is not covered by your warranty Read and follow the safety instructions that came with the product 1 Open the system See Opening the System on page 60 2 Grasp and gently lift the shroud up and away from the system board See Fi...

Page 63: ... down into the chassis 3 Close the system See Closing the System on page 61 4 Reconnect the system to its electrical outlet and turn the system on including any attached peripherals Hard Drives Your system supports up to twelve 3 5 SAS hard drives in 3 5 hot swap hard drive carriers The hard drives are front mounted and connected to a SAS backplane through hard drive carriers See Figure 3 1 Removi...

Page 64: ...ators on the drive carrier signal that the drive can be removed safely See Hard Drive Indicator Patterns on page 14 If the drive has been online the green activity fault indicator flashes as the drive is powered down When the drive indicators are off the drive is ready for removal 3 Press the release button and open the drive carrier release handle to release the drive See Figure 3 6 4 Slide the h...

Page 65: ...to lock its handle next to a partially installed carrier can damage the partially installed carrier s shield spring and make it unusable CAUTION To prevent data loss ensure that your operating system supports hot swap drive installation See the documentation supplied with the operating system 1 If applicable remove the front bezel See Removing the Front Bezel on page 59 2 If present remove the dri...

Page 66: ...e Figure 3 6 5 Close the handle to lock the drive in place Removing a Hard Drive From a Hard Drive Carrier Remove the screws from the slide rails on the hard drive carrier and separate the hard drive from the carrier See Figure 3 7 Figure 3 7 Installing a Hot Swap Hard Drive Into a Hard Drive Carrier 1 hard drive 2 SAS indicator 3 drive carrier 4 screws 4 2 1 3 4 ...

Page 67: ...rnal hard drives in a RAID 1 configuration For information on RAID configuration see the RAID documentation at support dell com manuals Removing an Internal Hard Drive Bay CAUTION Many repairs may only be done by a certified service technician You should only perform troubleshooting and simple repairs as authorized in your product documentation or as directed by the online or telephone service and...

Page 68: ...68 Installing System Components Figure 3 8 Removing and Installing an Internal Hard Drive Bay 1 internal hard drives 2 2 release latch 3 internal hard drive bay 4 support bracket 2 3 1 4 ...

Page 69: ...ll is not covered by your warranty Read and follow the safety instructions that came with the product 1 Turn off the system and attached peripherals and disconnect the system from the electrical outlet and from the peripherals 2 Open the system See Opening the System on page 60 3 Lift the latch on your internal hard drive bay and align the bay on the support bracket 4 Rotate the latch to lock Remo...

Page 70: ... Dell Installing a Hard Drive Into a Hard Drive Bay 1 Insert the hard drive into the internal hard drive bay with the connector end of the drive at the back until it is flush with the back of the hard drive bay See Figure 3 9 2 Attach the four screws to secure the hard drive to the hard drive bracket 1 internal hard drive bay 2 release latch 3 screws 4 4 internal hard drive 3 1 2 4 ...

Page 71: ...e done by a certified service technician You should only perform troubleshooting and simple repairs as authorized in your product documentation or as directed by the online or telephone service and support team Damage due to servicing that is not authorized by Dell is not covered by your warranty Read and follow the safety instructions that came with the product 1 Turn off the system including any...

Page 72: ...72 Installing System Components Figure 3 10 Removing and Installing a Fan 1 fans 5 2 release tab 3 fan power cable 1 2 3 ...

Page 73: ... Align the fan module so that the side with the power cable faces toward the back of the system 2 Slide the fan module into the fan assembly until the fan is fully seated See Figure 3 10 3 Connect the fan s power cable to the power connector on the system board 4 Replace the internal hard drive carrier and bay See Installing an Internal Hard Drive Bay on page 69 5 Route the power cable through the...

Page 74: ... should only perform troubleshooting and simple repairs as authorized in your product documentation or as directed by the online or telephone service and support team Damage due to servicing that is not authorized by Dell is not covered by your warranty Read and follow the safety instructions that came with the product CAUTION The system requires one power supply to operate the system normally Rem...

Page 75: ...t For information about the cable management arm see the system s rack documentation 3 Connect the power cable to the power supply and plug the cable into a power outlet CAUTION When connecting the power cable secure the cable with the Velcro strap NOTE When installing hot swapping or hot adding a new power supply in a system with two power supplies allow several seconds for the system to recogniz...

Page 76: ...ory configurations that fail to observe these guidelines can prevent your system from starting and producing any video output RDIMMs and UDIMMs cannot be mixed Except for memory channels that are unused all populated memory channels must have identical configurations The memory configuration for each processor must be identical Memory modules of different sizes can be mixed in A1 A4 or B1 B4 for e...

Page 77: ... memory module s Mode Specific Guidelines Three memory channels are allocated to each processor The number of channels used and the allowable configurations depend on the memory mode selected Advanced ECC Lockstep Mode Support In this configuration the two channels closest to the processor are combined to form one 128 bit channel This mode supports Single Device Data Correction SDDC for both x4 an...

Page 78: ...ory module CAUTION Many repairs may only be done by a certified service technician You should only perform troubleshooting and simple repairs as authorized in your product documentation or as directed by the online or telephone service and support team Damage due to servicing that is not authorized by Dell is not covered by your warranty Read and follow the safety instructions that came with the p...

Page 79: ... the memory module CAUTION Many repairs may only be done by a certified service technician You should only perform troubleshooting and simple repairs as authorized in your product documentation or as directed by the online or telephone service and support team Damage due to servicing that is not authorized by Dell is not covered by your warranty Read and follow the safety instructions that came wi...

Page 80: ...our thumbs to lock the memory module into the socket When the memory module is properly seated in the socket the ejectors on the memory module socket snap in place 9 Repeat step 5 through step 8 of this procedure to install the remaining memory modules 10 Replace the cooling shroud See Installing the Cooling Shroud on page 63 11 Close the system See Closing the System on page 61 12 Start up the sy...

Page 81: ...on card slot CAUTION Expansion cards can only be installed in the slots on the expansion card riser Do not attempt to install expansion cards directly into the riser connector on the system board Expansion Card Installation Guidelines The expansion card slots support full height half length cards The expansion card slots are hot swappable PCI Express Generation 2 expansion cards are supported in a...

Page 82: ...the card 5 Open the expansion card latch See Figure 3 14 6 Grasp the expansion card by its edges and carefully remove it from the expansion card connector 7 If you are removing the card permanently install a metal filler bracket over the empty expansion slot opening and close the expansion card latch NOTE You must install a filler bracket over an empty expansion slot to maintain Federal Communicat...

Page 83: ...roduct 1 Turn off the system including any attached peripherals and disconnect the system from the electrical outlet 2 Open the system See Opening the System on page 60 3 Remove the cooling shroud See Removing the Cooling Shroud on page 62 4 If installed remove the expansion card from the expansion card slot See Removing an Expansion Card on page 82 5 If installed remove the storage controller car...

Page 84: ...rt team Damage due to servicing that is not authorized by Dell is not covered by your warranty Read and follow the safety instructions that came with the product 1 Unpack the expansion card and prepare it for installation For instructions see the documentation accompanying the card 2 Turn off the system including any attached peripherals and disconnect the system from the electrical outlet 1 integ...

Page 85: ...ntain FCC certification of the system The brackets also keep dust and dirt out of the system and aid in proper cooling and airflow inside the system 6 Holding the card by its edges position the card so that the card edge connector aligns with the expansion card connector on the expansion card riser 7 Insert the card edge connector firmly into the expansion card connector until the card is fully se...

Page 86: ...n the riser guides with the riser guide posts on the system board See Figure 3 13 2 Lower the expansion card riser into place until the expansion card riser connector is fully seated 3 If applicable reinstall the expansion card See Installing an Expansion Card on page 84 4 Reinstall the storage controller card See Installing the Storage Controller Card on page 89 5 Reconnect all cables 6 Close the...

Page 87: ...the safety instructions that came with the product 1 Turn off the system including any attached peripherals and disconnect the system from the electrical outlet 2 Open the system See Opening the System on page 60 3 If installed remove the expansion card See Removing an Expansion Card on page 82 4 Disconnect the SAS cables connected to the card 5 Disconnect the cable connecting the card to the expa...

Page 88: ...ents Figure 3 15 Installing and Removing the Storage Controller Card 1 storage connector 2 expansion card riser 3 storage controller card 4 storage controller card cable 5 SAS data cable connector 6 release lever blue 1 2 3 6 5 4 ...

Page 89: ... under the storage controller card slot on the expansion card riser 5 With the storage controller card edge facing the riser insert one side of the card into the black card edge guide NOTE For expansion card riser 1 the installation sequence must be slot 3 2 1 and 4 The storage controller card must be installed in Slot 4 after the other expansion cards have been installed 6 Bend outward on the blu...

Page 90: ...is not covered by your warranty Read and follow the safety instructions that came with the product 1 Turn off the system including any attached peripherals and disconnect the system from the electrical outlet 2 Open the system See Opening the System on page 60 3 Pull back slightly on the retention standoff tab at the front edge of the card and gently lift the card off the retention standoff See Fi...

Page 91: ...off the system including any attached peripherals and disconnect the system from the electrical outlet 2 Open the system See Opening the System on page 60 3 If installed remove all expansion cards from expansion card riser Removing an Expansion Card on page 82 4 Insert the notch on the Integrated Dell Remote Access Controller 6 iDRAC6 Express card into the clip on the system board 5 Align the fron...

Page 92: ...nstall all expansion cards in the expansion card riser See Installing an Expansion Card on page 84 8 Close the system See Closing the System on page 61 9 Reconnect the system and peripherals to their power sources and turn them on 1 iDRAC6 Express card 2 plastic standoff tab 3 notch 4 clip 4 2 1 3 ...

Page 93: ...ck panel See Figure 1 3 3 Open the system See Opening the System on page 60 4 Remove the cooling shroud See Removing the Cooling Shroud on page 62 5 Pull back slightly on the two tabs at the front edge of the card and gently lift the front edge of the card off of the retention standoffs As the card releases from the standoffs the connector under the card disengages from the system board connector ...

Page 94: ...system including any attached peripherals and disconnect the system from the electrical outlet 2 Open the system See Opening the System on page 60 3 Remove the cooling shroud See Removing the Cooling Shroud on page 62 4 Remove the plastic filler plug of the iDRAC6 Enterprise port from the system back panel 5 Angle the card so that the RJ 45 connector fits through the back panel opening See Figure ...

Page 95: ...on Processors Removing a Processor CAUTION Many repairs may only be done by a certified service technician You should only perform troubleshooting and simple repairs as authorized in your product documentation or as directed by the online or telephone service 1 VFlash SD card 2 VFlash media slot 3 iDRAC6 Enterprise card 4 retention standoff posts 2 5 retention standoff tabs 2 6 iDRAC6 Enterprise c...

Page 96: ...r to cool before handling them CAUTION Never remove the heat sink from a processor unless you intend to remove the processor The heat sink is necessary to maintain proper thermal conditions 5 Using a 2 Phillips screwdriver loosen one of the heat sink retention screws See Figure 3 18 6 Wait 30 seconds for the heat sink to loosen from the processor 7 Loosen the other three heat sink retention screws...

Page 97: ...t and leave the release lever up so that the socket is ready for the new processor CAUTION Be careful not to bend any of the pins on the ZIF socket when removing the processor Bending the pins can permanently damage the system board NOTE In single processor configurations the processor must be installed in socket CPU1 Install the blank in socket CPU2 only 1 heat sink 2 retention screws 4 2 1 ...

Page 98: ...98 Installing System Components Figure 3 19 Installing and Removing a Processor 1 processor 2 processor shield 3 notch in processor 4 socket key 5 ZIF socket 6 socket release lever 2 1 6 3 4 5 ...

Page 99: ...rease from the top of the processor using a lint free cloth 3 Align the processor with the socket keys on the ZIF socket See Figure 3 19 CAUTION Positioning the processor incorrectly can permanently damage the system board or the processor Be careful not to bend the pins in the ZIF socket 4 With the release lever on the processor socket in the open position align the processor with the socket keys...

Page 100: ...iagnostics System Battery Replacing the System Battery WARNING There is a danger of a new battery exploding if it is incorrectly installed Replace the battery only with the same or equivalent type recommended by the manufacturer See your safety information for additional info CAUTION Many repairs may only be done by a certified service technician You should only perform troubleshooting and simple ...

Page 101: ...n card riser press the blue tabs on the expansion card riser and lift it from the chassis See Removing an Expansion Card Riser on page 83 7 Locate the battery socket See System Board Connectors on page 144 CAUTION To avoid damage to the battery connector you must firmly support the connector while installing or removing a battery 8 Support the battery connector by firmly holding the positive side ...

Page 102: ... Expansion Card Riser on page 86 14 Replace the storage controller card See Installing the Storage Controller Card on page 89 15 Replace the expansion card from the expansion card slot now See Installing an Expansion Card on page 84 16 Replace the cooling shroud See Installing the Cooling Shroud on page 63 17 Close the system See Closing the System on page 61 18 Reconnect the system to the electri...

Page 103: ...nd follow the safety instructions that came with the product 1 Turn off the system including any attached peripherals and disconnect the system from the electrical outlet 2 Open the system See Opening the System on page 60 3 To disconnect the RAID battery cable by gently pulling the cable connector out of the connector See Figure 3 21 4 Pull back gently on the two tabs holding the RAID battery and...

Page 104: ...done by a certified service technician You should only perform troubleshooting and simple repairs as authorized in your product documentation or as directed by the online or telephone service and support team Damage due to servicing that is not authorized by Dell is not covered by your warranty Read and follow the safety instructions that came with the product 1 If installed remove the front bezel...

Page 105: ...Installing System Components 105 Figure 3 22 Removing and Installing the Control Panel LED 1 bracket 2 control panel cable 3 power button board 4 power button 5 control panel module 1 2 4 3 5 ...

Page 106: ...safety instructions that came with the product 1 Assemble the power button and power button board in the control panel module 2 Tighten the screw to secure the power button board to the control panel module See Figure 3 22 3 Connect the control panel cable to the control panel module 4 Replace the bracket and tighten the single screw to secure the bracket to the back of the control panel module 5 ...

Page 107: ...the safety instructions that came with the product 1 If installed remove the front bezel See Removing the Front Bezel on page 59 2 Turn off the system and attached peripherals and disconnect the system from the electrical outlet and peripherals 3 Using a Torx screwdriver remove the three screws that secure the front panel I O module to the system chassis and remove the module 4 Remove the single s...

Page 108: ...108 Installing System Components Figure 3 23 Removing and Installing the Front Panel I O Module 1 front panel I O cable 2 bracket 3 front panel I O module 4 mounting screws 3 4 2 3 1 ...

Page 109: ...nly be done by a certified service technician You should only perform troubleshooting and simple repairs as authorized in your product documentation or as directed by the online or telephone service and support team Damage due to servicing that is not authorized by Dell is not covered by your warranty Read and follow the safety instructions that came with the product 1 If applicable remove the bez...

Page 110: ...damage 8 Disconnect the internal hard drive cables 9 While pressing the two blue latches toward each other lift the backplane upward See Figure 3 24 10 When the backplane cannot slide upward any further pull the backplane toward the back of the system to remove it from the retention hooks 11 Lift the board out of the system being careful to avoid damaging components on the face of the board 12 Pla...

Page 111: ... a SAS Backplane 1 SAS backplane 2 backplane retention latches 2 3 SAS cables 4 control panel module cable 5 USB memory key connector 6 SAS backplane power cable 7 control panel cable 8 internal hard drive cables 2 9 USB cable 10 SAS backplane cable 1 3 7 2 8 9 6 4 10 5 ...

Page 112: ...he face of the board 2 Align the slots in the backplane with the retention hooks on the back of the drive bays then move the backplane forward until the retention hooks fit through the slots in the backplane See Figure 3 24 3 Slide the backplane downward until the two blue retention latches snap into place 4 Connect the SAS data and power cables to the SAS backplane 5 Install the hard drives in th...

Page 113: ...hard drive bay and support carrier See Removing an Internal Hard Drive Bay on page 67 2 If applicable remove the thumb screw on the support bracket and lift it up and out of the system 3 Remove the power supplies from the system See Removing a Power Supply on page 74 4 Remove the expansion cards See Removing an Expansion Card on page 82 5 Remove the expansion card riser See Removing an Expansion C...

Page 114: ...tem Components Figure 3 25 Removing and Installing the Power Distribution Board 1 screws 3 2 power supply connector 3 power interposer board 4 blue tab 5 power distribution board 6 fan module cable connector 6 2 1 5 4 3 ...

Page 115: ...e Figure 3 25 3 Install the three screws that secure the power distribution board to the chassis See Figure 3 25 4 Connect the power distribution cables to the system board see System Board on page 116 and fan cable to the power distribution board as shown in Figure 3 25 5 Align the support bracket over the Power Distribution board and secure in place with the screw 6 Install the internal hard dri...

Page 116: ...ur hard drives 1 Turn off the system and attached peripherals and disconnect the system from the electrical outlet 2 Open the system See Opening the System on page 60 3 Remove the cooling shroud See Removing the Cooling Shroud on page 62 4 Remove the power supplies See Removing a Power Supply on page 74 5 Remove all expansion cards and if applicable the storage controller card See Removing an Expa...

Page 117: ...mbly by grasping a memory module processor or other components 12 Grasp the system board assembly by the edges and lift the system board assembly away from the chassis See Figure 3 26 Figure 3 26 Removing and Installing the System Board 1 screws 9 2 system board assembly 1 2 ...

Page 118: ...and maneuver the system board to the bottom of the chassis until it lays completely flat 5 Push the system board toward the back of the chassis until the board is in place 6 Tighten the nine screws that secure the system board to the chassis See Figure 3 26 7 Transfer the processors to the new system board See Removing a Processor on page 95 and Installing a Processor on page 99 8 Remove the memor...

Page 119: ...press card See Installing an iDRAC6 Express Card on page 91 15 Replace the cooling shroud See Installing the Cooling Shroud on page 63 16 Close the system See Closing the System on page 61 17 Reconnect the system to its electrical outlet and turn the system on including any attached peripherals 18 If applicable replace the front bezel See Removing the Front Bezel on page 59 ...

Page 120: ...120 Installing System Components ...

Page 121: ...dware check for the following conditions If you boot the system to the BIOS boot mode after installing an operating system from the UEFI Boot Manager the system will hang The reverse is also true You must boot to the same boot mode in which you installed the operating system See Using the System Setup Program and UEFI Boot Manager on page 39 Invalid memory configurations could cause the system to ...

Page 122: ...osite side of the system 3 If the problem is resolved restart the system enter the System Setup program and check if the nonfunctioning USB ports are enabled 4 Replace the keyboard mouse with another working keyboard mouse If the problem is resolved replace the faulty keyboard mouse If the problem is not resolved proceed to the next step to begin troubleshooting the other USB devices attached to t...

Page 123: ...n off the system and the serial device and swap the device with a comparable device 4 Turn on the system and the serial device If the problem is resolved replace the serial device If the problem persists see Getting Help on page 149 Troubleshooting a NIC 1 Run the appropriate Embedded System diagnostic test See Running the Embedded System Diagnostics on page 140 2 Restart the system and check for ...

Page 124: ...ng Help on page 149 Troubleshooting an NVRAM card CAUTION Installing the NVRAM card in a legacy PCI slot reduces performance and might cause irreparable damage to the PCIe host interface Ensure that the NVRAM card is installed in a PCIe x4 or x8 slot Troubleshooting a Wet System CAUTION Many repairs may only be done by a certified service technician You should only perform troubleshooting and simp...

Page 125: ...ved in step 3 6 Close the system See Closing the System on page 61 7 Reconnect the system to the electrical outlet and turn on the system and attached peripherals If the system does not start properly see Getting Help on page 149 8 If the system starts properly shut down the system and reinstall all of the expansion cards that you removed See Installing an Expansion Card on page 84 9 Run the appro...

Page 126: ...that the following components are properly installed Expansion card riser Power supplies Fans Processors and heat sinks Memory modules Hard drive carriers Cooling shroud 4 Ensure that all cables are properly connected 5 Close the system See Closing the System on page 61 6 Run the system board tests in the system diagnostics See Running the System Diagnostics on page 139 If the tests fail see Getti...

Page 127: ...system time to speed up or slow down If the system seems to operate normally except for the time kept in the System Setup program the problem may be caused by software rather than by a defective battery Troubleshooting Power Supplies 1 Identify the faulty power supply by the power supply s fault indicator See Power Indicator Codes on page 18 CAUTION At least one power supply must be installed for ...

Page 128: ... is removed or has failed See Troubleshooting a Fan on page 128 The expansion card installation guidelines have not been followed See Expansion Card Installation Guidelines on page 81 Troubleshooting a Fan CAUTION Many repairs may only be done by a certified service technician You should only perform troubleshooting and simple repairs as authorized in your product documentation or as directed by t...

Page 129: ...ation Guidelines on page 76 and verify that your memory configuration complies with all applicable guidelines 1 If the system is operational run the appropriate Embedded System diagnostic test See Running the Embedded System Diagnostics on page 140 If diagnostics indicates a fault follow the corrective actions provided by the diagnostic program 2 If the system is not operational turn off the syste...

Page 130: ... system memory setting See Memory Settings Screen on page 43 If the problem is not resolved proceed with the next step 14 Turn off the system and attached peripherals and disconnect the system from the power source 15 Open the system See Opening the System on page 60 16 If a diagnostic test or error message indicates a specific memory module as faulty swap or replace the module 17 To troubleshoot ...

Page 131: ... on page 140 Depending on the results of the diagnostics test proceed as needed through the following steps 2 If installed remove the front bezel See Removing the Front Bezel on page 59 3 If your system has a RAID controller and your hard drives are configured in a RAID array perform the following steps a Restart the system and enter the host adapter configuration utility program by pressing Ctrl ...

Page 132: ... CAUTION This troubleshooting procedure can destroy data stored on the hard drive Before you proceed back up all files on the hard drive 1 Run the appropriate Embedded System diagnostics test See Running the Embedded System Diagnostics on page 140 Depending on the results of the diagnostics test proceed as needed through the following steps 2 If your system has a RAID controller and your hard driv...

Page 133: ...ll is not covered by your warranty Read and follow the safety instructions that came with the product NOTE When troubleshooting a SAS or SAS RAID controller also see the documentation for your operating system and the controller 1 Run the appropriate Embedded System diagnostic test See Running the Embedded System Diagnostics on page 140 2 Enter the System Setup program and ensure that the SAS or P...

Page 134: ...erals If the problem persists see Getting Help on page 149 13 Replace the front bezel See Installing the Front Bezel on page 60 Troubleshooting Expansion Cards CAUTION Many repairs may only be done by a certified service technician You should only perform troubleshooting and simple repairs as authorized in your product documentation or as directed by the online or telephone service and support tea...

Page 135: ... Remove all expansion cards installed in the system See Removing an Expansion Card on page 82 12 Close the system See Closing the System on page 61 13 Reconnect the system to the electrical outlet and turn on the system and attached peripherals 14 Run the appropriate Embedded System diagnostic test See Running the System Diagnostics on page 139 If the tests fail see Getting Help on page 149 15 For...

Page 136: ...d disconnect the system from the electrical outlet 3 Open the system See Opening the System on page 60 4 Ensure that each processor and heat sink are properly installed See Installing a Processor on page 99 5 Close the system See Closing the System on page 61 6 Reconnect the system to the electrical outlet and turn on the system and attached peripherals 7 Run the appropriate Embedded System diagno...

Page 137: ...n page 149 14 Turn off the system and attached peripherals and disconnect the system from the electrical outlet 15 Open the system See Opening the System on page 60 16 Replace processor 1 with processor 2 See Installing a Processor on page 99 17 Repeat step 11 through step 13 If you have tested both the processors and the problem persists the system board is faulty See Getting Help on page 149 ...

Page 138: ...138 Troubleshooting Your System ...

Page 139: ... or devices The system diagnostics menus and options allow you to Run tests individually or collectively Control the sequence of tests Repeat tests Display print or save test results Temporarily suspend testing if an error is detected or terminate testing when a user defined error limit is reached View help messages that briefly describe each test and its parameters View status messages that infor...

Page 140: ... to run all or specific diagnostics tests or to exit Embedded System Diagnostics Testing Options Click the testing option in the Main Menu window Using the Custom Test Options When you select Custom Test in the Main Menu window the Customize window allows you to select the device s to be tested select specific options for testing and view the test results Testing Option Function Express Test Perfo...

Page 141: ...s only tests that require no user intervention Quick Tests Only Runs only the quick tests on the device Show Ending Timestamp Time stamps the test log Test Iterations Selects the number of times the test is run Log output file pathname Enables you to specify the USB memory key where the test log file is saved You cannot save the file to a hard drive Viewing Information and Results The following ta...

Page 142: ...142 Running the System Diagnostics ...

Page 143: ...hows the location of the configuration jumpers on the system board Table 6 1 lists the jumper settings Table 6 1 System Board Jumper Settings Jumper Setting Description PWRD_EN default The password feature is enabled pins 2 4 The password feature is disabled and iDRAC6 local access is unlocked at the next AC power cycle pins 4 6 NVRAM_CLR default The configuration settings are retained at system b...

Page 144: ... Connectors System Board Connectors See Figure 6 1 and Table 6 2 for the location and description of the system board connectors Figure 6 1 System Board Connectors 3 4 14 16 15 1 2 5 6 7 8 10 11 12 13 17 18 19 20 21 9 ...

Page 145: ...ctor 7 A3 A2 A1 A4 Memory module slot A3 white release lever Memory module slot A2 white release lever Memory module slot A1 white release lever Memory module slot A4 8 FAN3 Cooling fan 3 connector 9 BATTERY System battery 10 FAN4 Cooling fan 4 connector 11 12V 8 pin power connector 12 FP_CONN control panel connector 13 BP_CONN Backplane power connector 14 PWR_CONN 24 pin power connector 15 PDB_ I...

Page 146: ...thorized by Dell is not covered by your warranty Read and follow the safety instructions that came with the product 1 Turn off the system including any attached peripherals and disconnect the system from the electrical outlet 2 Open the system See Opening the System on page 60 3 Move the password jumper to the disabled position to clear the password See Table 6 1 See Figure 6 1 to locate the passw...

Page 147: ... new password s the next time it boots 6 Turn off the system including any attached peripherals and disconnect the system from the electrical outlet 7 Open the system See Opening the System on page 60 8 Move the password jumper back to the enabled position to restore the password function See Table 6 1 9 Close the system See Closing the System on page 61 10 Reconnect your system and peripherals to...

Page 148: ...148 Jumpers and Connectors ...

Page 149: ... based support and service options Availability varies by country and product and some services may not be available in your area To contact Dell for sales technical support or customer service issues 1 Visit support dell com 2 Verify your country or region in the Choose A Country Region drop down menu at the bottom of the page 3 Click Contact Us on the left side of the page 4 Select the appropria...

Page 150: ...150 Getting Help ...

Page 151: ...stem board 144 USB 12 video 12 contacting Dell 149 control panel assembly features 12 installing 106 removing 104 cooling fan replacing 73 cooling fans troubleshooting 128 cooling shroud installing 63 removing 62 cover closing 61 opening 60 D damaged systems troubleshooting 126 Dell contacting 149 diagnostics advanced testing options 140 testing options 140 when to use 139 DIMMs See memory modules...

Page 152: ... H hard drive troubleshooting 131 132 hard drives hot swappable installing 65 heat sink 97 I iDRAC card installing 90 93 system port 15 iDRAC Configuration Utility 56 indicators back panel 15 front panel 12 NIC 17 power 12 18 installing control panel assembly 106 cooling shroud 63 expansion card 84 front panel IO module 109 hard drive hot swappable 65 hard drive blank 64 iDRAC card 90 93 memory mo...

Page 153: ... 17 NICs back panel connectors 15 troubleshooting 123 O Optimizer memory mode 78 P password setup 53 system 51 passwords disabling 146 phone numbers 149 POST accessing system features 11 power indicators 12 18 power supplies indicators 18 removing 74 replacing 75 processor installing 99 removing 95 See processor upgrades 95 processors troubleshooting 136 R remote access controller See iDRAC removi...

Page 154: ...ter card troubleshooting 133 securing your system 48 52 setup password 53 slots See expansion slots startup accessing system features 11 support contacting Dell 149 system closing 61 opening 60 system board connectors 144 installing 118 jumpers 143 removing 116 system cooling troubleshooting 128 system features accessing 11 system messages 22 system password 51 system setup program boot settings 4...

Page 155: ...ard 122 memory 129 NIC 123 processors 136 SAS RAID controller daughter card 133 system cooling 128 video 122 wet system 124 U Unified Server Configurator Lifecyle Controller 54 upgrades processor 95 USB front panel connectors 12 USB device back panel connectors 15 V video back panel connector 15 front panel connectors 12 troubleshooting 122 W warning messages 37 warranty 38 wet system troubleshoot...

Page 156: ...156 Index ...

Reviews:

No comments