Quantum DXi6902 Скачать руководство пользователя страница 32

Страница: 32 / 242

Chapter 1: DXi6902 Basic Operations
DXi6902 Node

Quantum DXi6902 User’s Guide

Node Hard Drive Carrier

Indicators

Each hard drive carrier has two LED indicators (see

Figure 5

• Drive activity indicator (green)

•

Flashing

- Indicates hard disk drive activity.

• Drive failure indicator (green and amber)

•

Off

- Drive ready for insertion or removal

Note:

The drive status indicator remains off until all hard

drives are initialized after system power is applied.

Drives are not ready for insertion or removal during this

time.

•

Blinks green two times per second

- Identify drive/preparing

for removal

•

Blinks green, amber, and off

- Drive predicted failure

•

Blinks amber four times per second

- Drive failed

•

Blinks green slowly

- Drive rebuilding

•

Steady green

- Drive online

•

Blinks green three seconds, amber three seconds, and off

three seconds

- Rebuild aborted

Note:

If a drive fails, you will be notified by a service ticket in the

remote management console.

Power supply (PSU2)

750 watt hot-swappable power supply.

Optional 2 x 10 GbE

(SFP+) Ethernet ports

Optional 4 x 1 GbE

Ethernet ports

For data ingest/read.

Item

Indicator, Button, or

Connector

Icon

Description

Содержание DXi6902

Страница 1: ...Field Service Manual Quantum DXi6902 with DXi 3 0 0_69 Software 6 68164 01 Rev A...

Страница 2: ...EMENT Quantum the Quantum Logo Backup Recovery Archive It s What We Do Be Certain Be Quantum Certain DLT the DLT Logo DLTSage DLTtape the DLTtape Logo DXi DXi Accent Dynamic Powerdown FastSense FlexLi...

Страница 3: ...Port Indicators 10 Node Power Supply Indicators 11 DXi6902 Array and Expansion Modules 12 Array and Expansion Module Front Panel Features and Indicators 13 Array and Expansion Module Hard Drive Carri...

Страница 4: ...47 Collect Log Service Menu 47 Blockpool Verify and Blockpool Corruption Actions 48 Determining the Status of a Blockpool Verify 49 Running the Blockpool Verification Script 51 Detecting Blob Corrupti...

Страница 5: ...placing Node Memory Modules 99 Obtaining a Replacement Node Memory Module 99 Replacing a Node Memory Module 99 Replacing Node Expansion Cards 106 Identifying Expansion Card Locations 107 Obtaining a R...

Страница 6: ...ard Drive 164 Replacing Array or Expansion Module Power Supplies 168 Obtaining a Replacement Array or Expansion Power Supply 168 Replacing an Array or Expansion Power Supply 169 Replacing an Array Mod...

Страница 7: ...Guide vii Removing and Installing the Information Tag 199 DXi6902 Replacement Cables 201 Chapter 4 DXi6902 BIOS Settings 203 Updating the DXi6902 BIOS 203 Required DXi6902 BIOS Settings 204 Appendix...

Страница 8: ...Contents viii Quantum DXi6902 User s Guide...

Страница 9: ...er With Dual Optical or Twinax Ports 11 Figure 8 Power Supply LED 12 Figure 9 Array and Expansion Module LED Indicators Front Panel 14 Figure 10 Hard Drive Sled LEDs 15 Figure 11 Array Module LED Indi...

Страница 10: ...ools Menu 48 Figure 32 Blockpool Corruption Detection and Fix Flow 54 Figure 33 Initial WinSCP Log in Screen 68 Figure 34 Root Directory 69 Figure 35 Progress Bar 69 Figure 36 Removing the DXi6902 Nod...

Страница 11: ...Expansion Card Riser 1 111 Figure 62 Removing and Installing an Expansion Card in Riser 1 112 Figure 63 Removing and Installing the Expansion Card Riser 2 114 Figure 64 Removing and Installing the Ex...

Страница 12: ...Backplane 141 Figure 87 Removing and Installing the Hard Drive Backplane 142 Figure 88 Hard Drive Backplane Cabling Diagram 143 Figure 89 DXi6902 Node System Board 145 Figure 90 Removing and Installi...

Страница 13: ...oving and Installing an Array Module Controller 183 Figure 115 Removing and Installing an Array Module BBU 184 Figure 116 DXi6902 Expansion Module ESM Canister 186 Figure 117 Checking Expansion Module...

Страница 14: ...DXi6902 SAS Cabling EBODs 2 1 through 2 6 215 Figure 131 DXi6902 Power Cabling 216 Figure 132 DXi6902 Fibre Channel Cabling 216 Figure 133 DXi6902 Ethernet Cabling 216 Figure 134 DXi6902 Ethernet Cabl...

Страница 15: ...EDs and Buttons 17 Table 6 Expansion Module LEDs and Buttons 20 Table 7 Maximum Allowed Replication Sources and Replication Streams 55 Table 8 Maximum Traditional and OST Replication Streams 58 Table...

Страница 16: ...Tables xvi Quantum DXi6902 User s Guide...

Страница 17: ...g Service menus and FRU replacement Audience This manual is written for DXi6902 operators and administrators Note It is useful for the audience to have a basic understanding of Windows Linux and UNIX...

Страница 18: ...for DXi6902 components Notational Conventions This manual uses the following conventions Note Note emphasizes important information related to the main topic Caution Caution indicates potential hazar...

Страница 19: ...mes all risk in this aspect This unit is engineered and manufactured to meet all safety and regulatory requirements Be aware that improper use may result in bodily injury damage to the equipment or in...

Страница 20: ...for the DXi6902 Document No Document Title Document Description 6 00618 System Safety and Regulatory Information Quantum Products Lists all safety and regulatory information for all Quantum products 6...

Страница 21: ...cal publications send e mail to doc comments quantum com 6 68161 DXi6902 Site Planning Guide Provides site planninginformation for the DXi6902 6 67079 DXi Series NetBackup and Backup Exec OST Configur...

Страница 22: ...antum com ServiceandSupport Index aspx eSupport Submit online service requests update contact information add attachments and receive status updates via e mail Online Service accounts are free from Qu...

Страница 23: ...t Warranty For more information on the Quantum Worldwide End User Standard Limited Product Warranty http www quantum com pdf QuantumWarranty pdf United States 1 800 284 5101 toll free 1 720 249 5700 E...

Страница 24: ...Preface xxiv Quantum DXi6902 User s Guide...

Страница 25: ...anagement Locating DXi6902 Serial Numbers DXi6902 Node The DXi6902 Node is a computer server that provides control for the DXi software host OS and software applications The Node also stores support d...

Страница 26: ...rrier Indicators Node 1 GbE Ethernet Port Indicators Node 10 GbE Ethernet Port Indicators Node Power Supply Indicators Node Front Panel Features and Indicators Figure 1 shows the controls indicators a...

Страница 27: ...on Turning off the power without properly shutting down the system may result in loss of data see Turning On and Shutting Down the System on page 21 Note To shut down the Node in the event of an emerg...

Страница 28: ...ystems support one of the following 10 GbE cable types 10 GbE optical cable lengths of up to 300 meters with OM3 cables and up to 100 meters with OM2 cables Quantum recommends using the two 10 meter L...

Страница 29: ...opper Twinax cable options that Quantum provides do not support all switches Please note the supported switches during the purchase configuration process and if the Twinax cables supplied by Quantum a...

Страница 30: ...Optional 2 x 10GbE Ports Figure 4 Rear View DXi6902 With Optional 4 x 1GbE Ports Table 2 Rear Panel Connectors DXi6902 6 7 16 6 16 7 8 9 Item Indicator Button or Connector Icon Description 1 2 x 10 G...

Страница 31: ...ns on the front and back panels can be used to locate a particular system within a rack When one of these buttons is pressed the LCD panel on the front and the system status indicator on the back flas...

Страница 32: ...or insertion or removal during this time Blinks green two times per second Identify drive preparing for removal Blinks green amber and off Drive predicted failure Blinks amber four times per second Dr...

Страница 33: ...the configuration Quantum DXi6902 systems include four or eight 1 GbE Ethernet ports Each 1 GbE port on the back panel has two LED indicators see Figure 6 Link indicator Off Indicates the port is not...

Страница 34: ...ports or dual Twinax ports The 10 GbE adapter with dual optical or Twinax ports has two LED indicators for each 10 GbE port see Figure 7 Link indicator Off Indicates the port is not connected to the n...

Страница 35: ...is operational Flashing amber Indicates a problem with the power supply Flashing green When hot adding a power supply this indicates that the power supply is mismatched with the other power supply Rep...

Страница 36: ...1 or 2 Array modules RBODs Array modules connect to the Node and contain dual RAID controllers which provide storage management logic The DXi6902 includes 0 to 13 Expansion modules EBODs which connec...

Страница 37: ...sis to help you determine the correct port connections Caution Do not disconnect any SAS cables from the Array or Expansion modules during normal system operation Unplugging a SAS cable while the syst...

Страница 38: ...er temperature Power Standby power Button LED Description Enclosure locate white Indicates the location of the enclosure within the system Service action required amber Indicates at least one of the c...

Страница 39: ...that has failed If you accidentally remove a good drive wait 30 seconds before reinserting the drive Drive service action allowed Drive active Drive fault LED Description LEDs off No power Drive serv...

Страница 40: ...Port 2 ID Diag Port 1 Port 2 ID Diag Standby power Service action allowed DC enabled Service action required AC enabled Power switch SAS host 1 fault SAS host 1 activity SAS host 2 activity SAS host...

Страница 41: ...ery fault amber On battery has failed Off battery operating as normal Battery charging green Blinking battery is charging On battery is fully charged Off battery fault or no battery Service action all...

Страница 42: ...able 6 Standby power green Solid main DC power is off and 5V standby power is on DC enabled green Solid DC power rails are within specified limits Service action allowed blue Solid safe for Quantum fi...

Страница 43: ...D Diag Port 1 ID Diag Port 1 Standby power Service action allowed DC enabled Service action required AC enabled Power switch SAS expansion activity SAS expansion fault Serial port Ethernet link rate E...

Страница 44: ...m field service to remove ESM from slot Service action required amber Solid ESM has failed and requires attention Note Fault light will be on during power up and will turn off after self test sequence...

Страница 45: ...ntil the seven segment display on the rear of the module displays 00 approximately 1 minute 2 Turn on both power switches on the back of each Array module RBOD see Figure 14 Wait until the seven segme...

Страница 46: ...hat all backup and replication jobs are finished and that space reclamation activity is complete 1 Shut down the system from the remote management console using the Shutdown option on the Utilities Re...

Страница 47: ...n the same network as the DXi6902 Note For complete information about using all features of the remote management console refer to the Quantum DXi6902 User s Guide 6 68159 Access the remote management...

Страница 48: ...rowser address box type the IP address of the DXi6902 and then press Enter The Login window displays see Figure 16 Figure 16 Login Window If the Login window does not display verify that the IP addres...

Страница 49: ...is password Note To log on as the service user select Administrator and enter the service password Contact Quantum ASPS for the service password Note Passwords are limited to 15 characters Alphanumer...

Страница 50: ...than one Administrator can log on to the DXi at a time If two Administrators attempt to make configuration changes on the same page at the same time it is possible that one Administrator s changes wi...

Страница 51: ...n Module EBOD Serial Number You need this number for an Array module or Expansion module replacement component You can locate the system serial number on the Home page of the remote management console...

Страница 52: ...Chapter 1 DXi6902 Basic Operations Locating DXi6902 Serial Numbers 28 Quantum DXi6902 User s Guide Figure 18 DXi6902 Information Tag Pull out information tag to view system serial number...

Страница 53: ...erminal emulation program Note The Service Menu cannot be accessed using the DXi6902 remote management console Instead you must connect to the service port This chapter contains the following sections...

Страница 54: ...e iDRAC Port Connecting to the System Node Using the Serial Port Note You can also directly connect a monitor and keyboard to the DXi6902 Node Figure 19 Connecting to the DXi6902 Node Rear View Connec...

Страница 55: ...to view the boot process and access the motherboard and RAID controller BIOS settings Note Before attempting to connect to the iDRAC port you can check the current iDRAC port settings to make sure yo...

Страница 56: ...ng to the DXi6902 Node 32 Quantum DXi6902 User s Guide Figure 20 iDRAC Login 4 Enter admin for username and Qa Ip00 for password 5 Press Enter The iDRAC Main Console displays see Figure 21 Figure 21 i...

Страница 57: ...command prompt displays Connecting to the System Node Using the Serial Port Connecting to the serial port allows access to the root command prompt and the Service Menu To connect to the DXi6902 using...

Страница 58: ...mpt type the following command and press Enter sh opt DXi scripts service sh The Service Menu displays see Figure 22 Figure 22 Service Menu Service Menu 0 Hardware Configuration Tools for Configuring...

Страница 59: ...ge Setup the NetApp RBOD EBODs storage 3 Factory Detect Hardware Create Factory List of Detected Hardware 4 Encryption Tools System wide Encryption for Disk Security The Hardware Configuration Menu in...

Страница 60: ...r The Setup IPMI Menu displays See Figure 24 on page 36 Figure 24 Setup IPMI Menu Service Setup IPMI Menu 0 Configure IPMI Configure IPMI network user password Information 1 Show IPMI SHow IPMI networ...

Страница 61: ...Internal RAID H710 Configuration Menu displays see Figure 25 Figure 25 Internal RAID H710 Configuration Menu Internal RAID H710 Configuration Menu 0 Clear RAID sets on H710 Delete all non boot RAIDset...

Страница 62: ...the day counter for a replaced array BBU 8 Re enable Array tray alarms Re enable tray alarms on all arrays 9 Show Premium Feature Info Show all installed PFK licenses 10 Upload Turbo PFK file Install...

Страница 63: ...Key PFK licenses installed on each Array module Upload Turbo PFK file Installs a Turbo Premium Feature Key PFK on the selected Array module controller When an Expansion module EBOD is added to the DXi...

Страница 64: ...te information Factory Detect Hardware should show all components to be up and no component in an error status If there are errors make sure the new hardware is installed correctly Note Hardware statu...

Страница 65: ...mium Feature in RBODs 4 Download Premium Feature file Sent Encryption PFK file to an RBOD 5 Backup Disk Security Keys Export Encryption keys info in a ZIP file 6 Enable Disk Security Turn on Encryptio...

Страница 66: ...ber The system serial number is in the format SSYYWWPPPXXXXX where SS Site code AV for Avnet in Chandler AZ YY Build year code two numeric characters 12 for 2012 WW Build week code two numeric charact...

Страница 67: ...play Node Serial Number Displays the serial number of the Node Note The Node serial number matches the Dell Service Tag Display OUI Number Displays the OUI Organizational Unique Identifier number for...

Страница 68: ...s the following options Reboot Node Reboots the DXi6902 system Collect Gathers diagnostic information on the DXi6902 system This information should be available prior to contacting customer support Pr...

Страница 69: ...eb Svc Allows you to enable disable start stop or check the status of the OpenManage Server Admintrator tool Pick to Order PTO Menu The Pick to Order PTO Menu allows you to perform Pick to Order PTO c...

Страница 70: ...og Command Line To gather a collect log using the command line interface 1 Connect to the DXi6902 Node see Connecting to the DXi6902 Node on page 30 2 At the command prompt type the following command...

Страница 71: ...ad dialog system Options download Download current diagnostic log to the current directory system Download the system diagnostic log Collect Log Service Menu To gather a collect log using the Service...

Страница 72: ...e the option for Collect and press Enter This option collects data from the Node and creates a compressed file 5 FTP the following file to another system scratch collect collect serial_number date tim...

Страница 73: ...tion Determining the Status of a Blockpool Verify To determine the status of a blockpool verify 1 Connect to the DXi6902 Node see Connecting to the DXi6902 Node on page 30 2 At the command prompt type...

Страница 74: ...g cluster tree 100 complete 20100828 08 42 54 073 N 990 Verify Verifying BLOB tree heap free lists 20100828 08 42 54 096 N 990 Verify Verifying BLOB tree heap free lists 33 complete 20100828 08 42 54...

Страница 75: ...rifying BLOBs blocklet lists 100 complete 20100828 20 20 02 602 N 990 Verify Verify complete 5 Note the time when the blockpool verify completed Running the Blockpool Verification Script The blockpool...

Страница 76: ...ut which BLOBs if any are affected by these corrupted blocklets 1 At the command prompt type hurricane blockpool bin blockpool verify Dblob_data_fast Nlocal localhost This operation takes some time to...

Страница 77: ...ckpool will display messages when it repairs blocklets However to prove that all of the corruption has been repaired a second blockpool verification script must be run If it shows no corruption messag...

Страница 78: ...Chapter 2 DXi6902 Service Menus Blockpool Verify and Blockpool Corruption Actions 54 Quantum DXi6902 User s Guide Figure 32 Blockpool Corruption Detection and Fix Flow...

Страница 79: ...f replication streams on the Source system see Table 7 Note Replication performance is directly affected by the number of replication streams If you decrease the configured number of streams on the so...

Страница 80: ...sources for a target a reboot of the target system is necessary Similarly to change the maximum number of replication streams that a source uses a reboot of each source system is necessary Configuring...

Страница 81: ...guring the Target System To change the maximum allowed sources that replicate to the target system 1 Connect to the DXi6902 Node see Connecting to the DXi6902 Node on page 30 2 Edit the DXi6800 conf f...

Страница 82: ...ase the number of OST Replication streams you must decrease the number of Traditional Replication streams on the source system see Table 8 Table 8 Maximum Traditional and OST Replication Streams Cauti...

Страница 83: ...see Connecting to the DXi6902 Node on page 30 2 Edit the ObjectManager conf file using VI editor in the data hurricane conf directory 3 In the ObjectManager conf file change the variable MaxReplicatio...

Страница 84: ...xecute the following commands on the target system 1 Connect to the DXi see Connecting to the DXi6902 Node on page 30 2 Edit the Common conf file using VI editor in the data hurricane conf directory 3...

Страница 85: ...trieving a Crash Dump File If a DXi6902 has crashed a crash dump file is created and stored on the system The purpose of this procedure is to retrieve the crash dump file and then transfer it to engin...

Страница 86: ...PS CRASH has occurred Jul 02 11 43 50 Flames KDUMP INFO begin stack traces Jul 02 11 44 26 Flames KDUMP PID 30096 TASK ffff8107a9d0e140 CPU 2 COMMAND bash Jul 02 11 44 27 Flames KDUMP 0 ffff8103dbb3bd...

Страница 87: ...mes KDUMP R10 0000000000000000 R11 0000000000000246 R12 0000000000000001 Jul 02 11 44 27 Flames KDUMP R13 0000000000000002 R14 00002b659e4478c0 R15 0000000000000002 Jul 02 11 44 27 Flames KDUMP ORIG_R...

Страница 88: ...ce you see Uncompressed kernel dump was saved in red above the crash dump file will be located in the snfs Kdump directory Note The snfs Kdumps kdump_ name folder contains multiple files Make sure to...

Страница 89: ...st compress the file so it can be more easily transferred 1 Open a shell prompt to the DXi using a program like PuTTY or winSCP 2 Locate the crash dump file using the location displayed in the message...

Страница 90: ...r x 1 root root 50367686 Jul 2 12 05 vmlinux 2 6 19 2smp_CX0916BVA00005_2009 07 02_11h43m48s 4 Retrieve the KDUMP trace from the message so it can be used with the problem description grep KDUMP kdump...

Страница 91: ...z2 file The option produces and output to and the option is a UNIX continuation marker that tells the command to continue on the next line Transferring the Crash Dump File Use WinSCP to transfer the c...

Страница 92: ...set to 22 Username root Password contact Quantum ASPS for the service password Leave all other fields as defaults Note You can also leave the Username and Password boxes blank and you will be prompte...

Страница 93: ...sing Norton Interface drag and drop the files as needed in order to copy them on to or off of the DXi system Before any process is completed you will be prompted with a dialog box confirming your acti...

Страница 94: ...ted all operations you can quit the WinSCP program Caution WinSCP is a very powerful tool so use extreme caution when using it Double check your steps before performing any action to ensure you are in...

Страница 95: ...ing the Node Inside the DXi6902 Node Replacing Node Hard Drives Replacing Node Power Supplies Replacing Node Cooling Fans Replacing Node Memory Modules Replacing Node Expansion Cards Replacing the Nod...

Страница 96: ...placeable Units and CRUs Customer Replaceable Units in the DXi6902 are capable of being removed and replaced during normal system operation hot swappable Refer to Table 9 to determine if a part is hot...

Страница 97: ...ted Storage Controller Battery FRU No Yes Yes Network Daughter Card FRU No Yes Yes Processor and Heatsink FRU No Yes Yes Hard Drive Backplane FRU No Yes Yes System Board FRU No Yes Yes Node FRU No Yes...

Страница 98: ...grounding strap when working inside the Node Some DXi6902 FRU replacement procedures require removing the Node from the rack and opening the top cover to access the inside of the Node Refer to the fol...

Страница 99: ...page 21 2 If installed remove the front bezel from the Node by lifting the latch on the left side of the bezel 3 Remove all power SAS Ethernet and Fibre Channel cables from the rear of the Node Make...

Страница 100: ...the J slots 7 Lift the Node up and away from the rack and place it on a level surface WARNING The DXi6902 Node including hard drives weighs 53 0 pounds 24 0 kg A minimum of two people are required to...

Страница 101: ...lockwise to the unlocked position see Figure 38 Figure 38 Removing the Node Cover 10 Lift the latch on top of the Node and slide the cover back 11 Grasp the cover on both sides and carefully lift the...

Страница 102: ...d if you are replacing a cooling fan Figure 39 Removing the Cooling Shroud Closing the DXi6902 Node To replace the DXi6902 Node cover and return the Node to the rack after completing the repair 1 Repl...

Страница 103: ...Guide 79 Figure 40 Replacing the Cooling Shroud b Lower the cooling shroud into the chassis until it is firmly seated Note For proper seating of the cooling shroud in the chassis ensure that the cabl...

Страница 104: ...Node chassis and offset the cover slightly back so that it clears the chassis hooks and lays flush on the chassis 4 Push down the latch to move the cover into the closed position 5 Rotate the latch r...

Страница 105: ...downward until all the rail standoffs are seated in the J slots 9 Push the Node inward until the lock levers click into place 10 Press the slide release lock buttons on both rails and slide the Node...

Страница 106: ...nt Procedures Opening and Closing the Node 82 Quantum DXi6902 User s Guide 13 If applicable replace the front bezel Insert the right side of the bezel into the slots on the right side of the Node then...

Страница 107: ...Node Quantum DXi6902 User s Guide 83 Inside the DXi6902 Node Figure 43 illustrates the interior of the DXi6902 Node with the cover removed Refer to Table 10 for a description of the numbered items Fig...

Страница 108: ...sections Obtaining a Replacement Node Hard Drive Replacing a Node Hard Drive Note DXi6902 hard disk drives are hot swappable You do not need to shut down the system to remove and replace a drive Item...

Страница 109: ...ake sure that you have the required replacement kit The appropriate replacement drive will be provided by Quantum Customer Support see Figure 44 Caution Always make sure to use Quantum supplied new dr...

Страница 110: ...use of a grounding strap when performing this procedure Removing and replacing a hard drive in the Node includes the following steps 1 Identifying a Failed Drive in the Node 2 Removing a Drive from t...

Страница 111: ...during this time Blinks green two times per second Identify drive preparing for removal Blinks green amber and off Drive predicted failure Blinks amber four times per second Drive failed Blinks green...

Страница 112: ...surface can damage the drive mechanism or connectors To avoid damage when removing or installing a drive always place your hand under the drive to support its weight 1 If installed remove the front b...

Страница 113: ...hard drive carrier and attempting to lock its handle next to a partially installed carrier can damage the partially installed carrier s shield spring and make it unusable 1 Press the release button o...

Страница 114: ...the following sections Obtaining a Replacement Node Power Supply Replacing a Node Power Supply Note DXi6902 uses redundant hot swappable power supplies You do not need to shut down the system to remo...

Страница 115: ...ent console Caution Use appropriate ESD precautions including the use of a grounding strap when performing this procedure Removing and replacing a power supply in the Node includes the following steps...

Страница 116: ...upply is operational Flashing amber Indicates a problem with the power supply Flashing green When hot adding a power supply this indicates that the power supply is mismatched with the other power supp...

Страница 117: ...ake sure to remove the failed power supply amber or dark LED handle and not the functioning power supply green LED handle If you remove the functioning power supply power will be lost to the Node and...

Страница 118: ...ower supply is fully seated and the release latch snaps into place see Figure 51 2 Connect the power cable to the power supply and plug the cable into a power outlet 3 Secure the power cable with the...

Страница 119: ...that you have the required replacement kit The appropriate replacement cooling fan will be provided by Quantum Customer Support see Figure 52 You will need the following items to perform this procedu...

Страница 120: ...ng Fan in the Node Removing a Failed Cooling Fan from the Node Installing a Replacement Cooling Fan in the Node Identifying a Failed Cooling Fan in the Node The DXi6902 Node contains six cooling fans...

Страница 121: ...the Node To remove the failed cooling fan from the DXi6902 Node 1 Shut down the system see Turning On and Shutting Down the System on page 21 2 Remove the Node cover see Opening and Closing the Node...

Страница 122: ...ign the plug at the base of the cooling fan with the connector on the system board see Figure 54 2 Slide the cooling fan into the securing slots until the tabs lock into place 3 Close the Node cover s...

Страница 123: ...inning the replacement procedure make sure that you have the required replacement kit The appropriate replacement memory module will be provided by Quantum Customer Support see Figure 55 You will need...

Страница 124: ...y modules DIMMs The Node has 24 memory sockets divided into 2 sets A and B of 12 slots each Each set A or B is dedicated to one CPU Only 8 or 16 of the 24 memory sockets are populated with memory modu...

Страница 125: ...Quantum DXi6902 User s Guide 101 119 510 TB 1 or 2 RBODs 3 or more EBODs 256 GB 8 memory modules are installed in slots A1 A4 and B1 B4 white sockets 8 memory modules are installed in slots A5 A8 and...

Страница 126: ...e Memory Modules 102 Quantum DXi6902 User s Guide Figure 56 DXi6902 Node Memory Module Locations White sockets 8 memory modules are installed for all configurations Black sockets 8 memory modules are...

Страница 127: ...art Then try accessing OpenManage again 3 Enter the following login credentials and click Submit Username root Password Contact Quantum ASPS for the service password The OpenManage Server Administrato...

Страница 128: ...he memory modules are hot to the touch for some time after the system has been powered down Allow time for the memory modules to cool before handling them 1 Shut down the system see Turning On and Shu...

Страница 129: ...s by the card edges and avoid touching the components on the memory module 1 Align the memory module s edge connector with the alignment key of the memory module socket and insert the memory module in...

Страница 130: ...nistrator window displays 9 In the left pane navigate to System Main System Chassis Memory see Figure 57 10 Verify that the value for Total Installed Capacity Available to the OS is the expected value...

Страница 131: ...riser and slot where each expansion card is installed Note Expansion card riser 1 accommodates low profile PCIe cards while expansion card riser 2 and 3 accommodate standard height PCIe cards Figure 5...

Страница 132: ...d will be provided by Quantum Customer Support see Figure 60 You will need the following items to perform this procedure Replacement expansion card kit provided by Quantum Customer Support ESD groundi...

Страница 133: ...Expansion Cards Quantum DXi6902 User s Guide 109 Replacement quad port 1 GbE card Intel i350 PN 9YD6K Replacement dual port 8 Gb Fibre Channel card Qlogic QLE2562 PN 6H20P Replacement quad port 8 Gb F...

Страница 134: ...No card Dual port 10 GbE card Intel X520 DA2 Quad port 1 GbE card Intel i350 Slot 3 bottom Dual port 8 Gb Fibre Channel card Qlogic QLE2562 Caution Use appropriate ESD precautions including the use of...

Страница 135: ...s connected to the expansion card 4 Holding the touch points lift the expansion card riser 1 from the riser connector on the system board see Figure 61 Figure 61 Removing and Installing the Expansion...

Страница 136: ...card connector see Figure 62 2 Insert the card edge connector firmly into the expansion card connector until the card is fully seated 3 Close the expansion card latches 4 If applicable connect any ca...

Страница 137: ...t 7 bottom Quad port 8 Gb Fibre Channel card Qlogic QLE2564 Caution Use appropriate ESD precautions including the use of a grounding strap when performing this procedure Removing and replacing an expa...

Страница 138: ...s Guide Figure 63 Removing and Installing the Expansion Card Riser 2 Figure 64 Removing and Installing the Expansion Card Riser 3 Expansion card riser 2 Back riser guide Front riser guide Expansion c...

Страница 139: ...nsion Card in Riser 2 or 3 Installing an Expansion Card in Riser 2 or 3 To install a replacement expansion card in riser 2 or 3 1 Holding the card by its edges position the card so that the connector...

Страница 140: ...sole delete any administrative alerts and close any service tickets Replacing the Node Integrated Storage Controller This section describes how to remove and replace the integrated storage controller...

Страница 141: ...torage controller in the Node includes the following steps Identifying a Failed Integrated Storage Controller Removing a Failed Integrated Storage Controller Installing a Replacement Integrated Storag...

Страница 142: ...d Storage Controller To remove the integrated storage controller from the DXi6902 Node 1 Shut down the system see Turning On and Shutting Down the System on page 21 2 Remove the Node cover see Opening...

Страница 143: ...f the integrated storage controller card with the card holder on the system board see Figure 68 2 Lower the other end of the card into the storage controller card holder on the system board 3 Press th...

Страница 144: ...at Rest Encryption on a Replacement Integrated Storage Controller on page 120 8 Run Factory Detect Hardware from the Service Menu to detect the new hardware see Factory Detect Hardware on page 40 9 I...

Страница 145: ...ined in the recovery file Note that accessing the recovery file requires the password used at the time the file was created 1 In the remote management console check the Data at Rest Encryption status...

Страница 146: ...ot the system 4 During the boot process press Ctrl R when prompted to enter the PERC H710 RAID controller BIOS 5 In the RAID BIOS navigate to the VD Mgmt screen and select PERC H710 Mini in the list s...

Страница 147: ...d to enable Data at Rest Encryption on the system in the Passphrase box Enter the passphrase again in the Confirm box and then select OK see Figure 72 Note You do not need to enter a Security Key Iden...

Страница 148: ...ntroller 124 Quantum DXi6902 User s Guide Figure 73 PERC RAID BIOS Key Import 9 on the VD Mgmt screen verify that the disk group is imported and that disks are no longer shaded gray see Figure 74 Figu...

Страница 149: ...ttery Obtaining a Replacement Integrated Storage Controller Battery Before beginning the replacement procedure make sure that you have the required replacement kit The appropriate replacement integrat...

Страница 150: ...Storage Controller Battery Removing a Failed Integrated Storage Controller Battery Installing a Replacement Integrated Storage Controller Battery Identifying a Failed Integrated Storage Controller Bat...

Страница 151: ...e Controller Battery To remove the integrated storage controller battery from the DXi6902 Node 1 Shut down the system see Turning On and Shutting Down the System on page 21 2 Remove the Node cover see...

Страница 152: ...ttery into the controller see Figure 77 2 Replace the integrated storage controller see Replacing the Integrated Storage Controller 3 Close the Node cover see Opening and Closing the Node 4 Run Factor...

Страница 153: ...rm this procedure Replacement network daughter card kit provided by Quantum Customer Support 2 Phillips screwdriver ESD grounding wrist strap included in the replacement kit Figure 78 DXi6902 Node Net...

Страница 154: ...etwork daughter card that provides three customer accessible 1 GbE ports ETH1 ETH2 and ETH3 in the Node Note The service port ETH0 is also located on the network daughter card To identify a failed net...

Страница 155: ...Riser 2 or 3 4 Using a 2 Phillips screwdriver loosen the two captive screws that secure the network daughter card to the system board see Figure 80 5 Hold the network daughter card by the edges on ei...

Страница 156: ...card riser 2 see Replacing an Expansion Card in Riser 2 or 3 6 Close the Node cover see Opening and Closing the Node 7 Turn on the system see Turning On and Shutting Down the System on page 21 8 Run...

Страница 157: ...2 Phillips screwdriver ESD grounding wrist strap included in the replacement kit Figure 81 DXi6902 Node Processor and Heatsink Replacing the Processor and Heatsink This section describes how to remove...

Страница 158: ...Removing a Failed Processor and Heatsink Installing a Replacement Processor and Heatsink Identifying a Failed Processor and Heatsink The DXi6902 Node contains two processors CPUs and each is covered b...

Страница 159: ...rocessor and Heatsink To remove the processor and heatsink from the DXi6902 Node WARNING The heatsink and processor are hot to the touch for some time after the system has been powered down Allow the...

Страница 160: ...socket 5 Lift the heatsink away from the processor and set the heatsink aside Figure 83 Removing and Installing the Processor Heatsink Caution The processor is held in its socket under strong pressure...

Страница 161: ...0 degrees upward Figure 84 Processor Shield Opening and Closing Lever Sequence 10 Rotate the processor shield upward and out of the way see Figure 85 Caution The socket pins are fragile and can be per...

Страница 162: ...bottom of the processor Touch only the side edges of the processor Figure 85 Removing and Installing a Processor Installing a Replacement Processor and Heatsink To install the replacement processor an...

Страница 163: ...l it is locked in position see Figure 84 5 Similarly rotate the socket release lever near the unlock icon until it is locked in position 6 Using a clean lint free cloth remove the thermal grease from...

Страница 164: ...ons Obtaining a Replacement Hard Drive Backplane Replacing the Hard Drive Backplane Obtaining a Replacement Hard Drive Backplane Before beginning the replacement procedure make sure that you have the...

Страница 165: ...s including the use of a grounding strap when performing this procedure Removing and replacing the hard drive backplane in the Node includes the following steps Removing a Failed Hard Drive Backplane...

Страница 166: ...e removing the backplane Caution You must note the number of each hard drive and temporarily label them before removal so that you can replace them in the same locations 5 Disconnect the SAS data sign...

Страница 167: ...to align the hard drive backplane see Figure 87 and Figure 88 2 Slide down the hard drive backplane until the release tabs snap into place 3 Connect the SAS data signal and power cables to the backpla...

Страница 168: ...ment console delete any administrative alerts and close any service tickets Replacing the Node System Board This section describes how to remove and replace the system board in the DXi6902 Node and co...

Страница 169: ...ion Use appropriate ESD precautions including the use of a grounding strap when performing this procedure Removing and replacing the hard drive backplane in the Node includes the following steps Remov...

Страница 170: ...ng and Installing the PCIe Card Holder on page 197 f Cable retention bracket see Removing and Installing the Cable Retention Bracket on page 198 g If present support bracket see Figure 91 Note The sup...

Страница 171: ...em board 7 Grasp the system board holder lift the blue release pin and slide the system board toward the front of the system see Figure 91 Caution Take care not to damage the system identification but...

Страница 172: ...or some time after the system has been powered down Allow the heatsink and processor to cool before handling them 9 Remove the memory modules see Replacing a Node Memory Module on page 99 10 Remove th...

Страница 173: ...n button while placing the system board into the chassis 3 Push the system board toward the back of the chassis until the board clicks into place 4 Replace the following components a Cable retention b...

Страница 174: ...the System Setup Main Menu c Follow the steps described in Required DXi6902 BIOS Settings on page 204 to verify that all BIOS settings are set to the Quantum required value and make any changes as ne...

Страница 175: ...Connecting to the System Node Using the iDRAC Port on page 31 2 Unzip the Asset_Tag zip file 3 Copy the unzipped asset com file to the bootable USB flash drive 4 Record the Dell service tag number fr...

Страница 176: ...erview iDRAC Settings iDRAC Firmware Update 3 Upload the 82DFV_R720_CustBSU pm file Wait for the iDRAC reboot to complete 4 Reboot the system Wait for the system to complete one more self reboot befor...

Страница 177: ...g the Node This section describes how to remove and replace the DXi6902 Node and contains the following sections Obtaining a Replacement Node Replacing the Node Obtaining a Replacement Node Before beg...

Страница 178: ...affixed to the information tag by Quantum manufacturing and contains the original system serial number Figure 93 DXi6902 Information Tag Replacing the Node This section describes how to remove and rep...

Страница 179: ...Removing and replacing the DXi6902 Node includes the following steps Removing a Failed Node Installing a Replacement Node Removing a Failed Node To remove the DXi6902 Node 1 Shut down the system see...

Страница 180: ...ll the optional dual port 10 GbE card along with any associated SFPs or optional quad port 1 GbE card in PCIe slot 2 in the replacement Node see Replacing an Expansion Card in Riser 1 on page 110 3 In...

Страница 181: ...9 In the remote management console delete any administrative alerts and close any service tickets Replacing the Node Chassis This section describes how to remove and replace the DXi6902 Node chassis...

Страница 182: ...on Use appropriate ESD precautions including the use of a grounding strap when performing this procedure Removing and replacing the DXi6902 Node chassis includes the following steps Removing a Failed...

Страница 183: ...placing an Expansion Card in Riser 2 or 3 on page 113 d Integrated storage controller card see Replacing the Integrated Storage Controller on page 117 e PCIe card holder see Removing and Installing th...

Страница 184: ...e Connector 6 Disconnect all cables from the system board 7 Grasp the system board holder lift the blue release pin and slide the system board toward the front of the system see Figure 96 Caution Take...

Страница 185: ...correct locations in the replacement chassis see Replacing a Node Hard Drive on page 86 Caution You must install each hard drive in its correct location If the hard drive locations are not correct the...

Страница 186: ...ble Retention Bracket on page 198 d PCIe card holder see Removing and Installing the PCIe Card Holder on page 197 e Integrated storage controller card see Replacing the Integrated Storage Controller o...

Страница 187: ...Hard Drive Replacing an Array or Expansion Hard Drive Note DXi6902 hard disk drives are hot swappable You do not need to shut down the system to remove and replace a drive Caution Powering down the s...

Страница 188: ...pluggable drive carriers that fit into the drive bays Caution Do not remove a hard drive that has not failed Instead only remove a drive that has failed If you accidently remove a good drive wait 30 s...

Страница 189: ...n Module Drive Slot Numbering Use the indicator LEDs on the hard drive carrier to identify the failed hard drive see Figure 99 LEDs off No power Drive service action allowed LED blue Solid It is safe...

Страница 190: ...possibility of overheating the equipment Caution Bumping drives against another surface can damage the drive mechanism or connectors To avoid damage when removing or installing a drive always place yo...

Страница 191: ...drive carrier and attempting to lock its handle next to a partially installed carrier can damage the partially installed carrier s shield spring and make it unusable 1 Open the hard drive carrier han...

Страница 192: ...e following sections Obtaining a Replacement Array or Expansion Power Supply Replacing an Array or Expansion Power Supply Note DXi6902 uses redundant hot swappable power supplies You do not need to sh...

Страница 193: ...D precautions including the use of a grounding strap when performing this procedure Removing and replacing a power supply in an Array module RBOD or Expansion module EBOD includes the following steps...

Страница 194: ...r is off and 5V standby power is on DC enabled solid green DC power rails are within specified limits Service action allowed solid blue Safe to remove power supply from slot Service action required so...

Страница 195: ...supply service action LED is amber and not the functioning power supply AC enabled LED is green If you remove the functioning power supply power will be lost to the Array or Expansion module and data...

Страница 196: ...ide the replacement power supply into the chassis until the power supply snaps into place and push the handles into the closed position 3 Connect the power cable to the power supply and plug the cable...

Страница 197: ...it The appropriate replacement Array module controller will be provided by Quantum Customer Support see Figure 105 You will need the following items to perform this procedure Replacement Array module...

Страница 198: ...Identifying a Failed Array Module Controller The DXi6902 contains one or two Array modules RBODs Each Array module contains two controllers A and B To identify the failed Array module controller 1 Ac...

Страница 199: ...Service Action Allowed This LED is lit when the component has failed and is ready to be replaced Figure 107 Array Module Controller Failure LEDs Removing a Failed Array Module Controller To remove the...

Страница 200: ...ntroller 176 Quantum DXi6902 User s Guide Figure 108 Controller Air Blocker 2 Disconnect the SAS cables from the failed controller 3 Pull the handles out and toward the sides of the controller caniste...

Страница 201: ...e the BBU from the failed Array module controller and install the BBU in the replacement controller see Replacing an Array Module BBU on page 179 2 Remove the controller air blocker from the controlle...

Страница 202: ...s Obtaining a Replacement Array Module BBU Replacing an Array Module BBU Obtaining a Replacement Array Module BBU Before beginning the replacement procedure make sure that you have the required replac...

Страница 203: ...ing a Failed Array Module BBU Installing a Replacement Array Module BBU Identifying a Failed Array Module BBU The DXi6902 contains one or two Array modules RBODs Each Array module contains two control...

Страница 204: ...le BBU Status 4 Locate the controller with the failed BBU on the back of the Array module by viewing the Service Action LEDs see Figure 112 Service Action Required This LED is lit when the component h...

Страница 205: ...iled Array Module BBU To remove the failed Array module BBU 1 Prepare the controller air blocker by removing it from the packaging and folding it inward at right angles so it is ready to insert into t...

Страница 206: ...82 Quantum DXi6902 User s Guide Figure 113 Controller Air Blocker 2 Disconnect the SAS cables from the controller with the failed BBU 3 Pull the handles out and toward the sides of the controller cani...

Страница 207: ...14 Removing and Installing an Array Module Controller 4 Insert the controller air blocker into the open controller slot to make sure that correct airflow is maintained 5 Press down on both of the top...

Страница 208: ...an Array Module BBU 6 Unscrew the thumbscrew that secures the failed battery to the controller canister and remove the battery Installing a Replacement Array Module BBU To install the replacement Arr...

Страница 209: ...Reset the BBU installation date a Access the Service Menu see Accessing the DXi6902 Service Menu on page 34 b Navigate to Hardware Configuration Setup external Storage Reset Array BBU date c Follow th...

Страница 210: ...ter provided by Quantum Customer Support ESD grounding wrist strap included in the replacement kit Figure 116 DXi6902 Expansion Module ESM Canister Replacing an Expansion Module ESM Canister This sect...

Страница 211: ...age 23 2 Navigate to the Status Hardware Details Storage Arrays page 3 Note the status of the volumes in each Array module see Figure 117 Tray_0 corresponds to the Expansion module EBOD connected to a...

Страница 212: ...xpansion Module ESM Canister Failure LEDs Removing a Failed Expansion Module ESM To remove the failed Expansion module ESM canister 1 Disconnect the SAS cables from the failed ESM canister 2 Pull the...

Страница 213: ...empty bay until the canister snaps into place and push the handles into the closed position see Figure 119 2 Connect the SAS cables to the new ESM canister see DXi6902 Cabling Diagrams on page 211 3 U...

Страница 214: ...an Array or Expansion Module This section describes how to remove and replace the DXi6902 Array module RBOD or Expansion module EBOD chassis and contains the following sections Obtaining a Replacement...

Страница 215: ...e of a grounding strap when performing this procedure Removing and replacing the DXi6902 Array module or Expansion module chassis includes the following steps Removing a Failed Array or Expansion Modu...

Страница 216: ...Expansion Module b Remove the 10 32 x 3 4 in SEM screw with washer on each side of the Array or Expansion Module s front panel Caution Do not use excessive force when pulling the chassis forward to fu...

Страница 217: ...oller as you face the rear of the module and Controller B right controller as you face the rear of the module Remove both Expansion module ESM canisters from the failed Expansion module chassis see Re...

Страница 218: ...ule chassis in the rack see Figure 121 a Align the Array or Expansion module with the rails and slide the module into the rack b Use a 10 32 x 3 4 in SEM screw with washer at the bottom hole on each s...

Страница 219: ...ng the Cooling Fan Assembly Removing and Installing the PCIe Card Holder Removing and Installing the Cable Retention Bracket Removing and Installing the Information Tag Removing and Installing the Coo...

Страница 220: ...are correctly installed and retained by the cable retention bracket before installing the cooling fan assembly Incorrectly installed cables may get damaged 1 Align the cooling fan assembly slots with...

Страница 221: ...he PCIe Card Holder Removing the PCIe Card Holder To remove the PCIe card holder 1 Press the release tab and the touch point to release the PCIe card holder from the chassis see Figure 123 2 Lift the...

Страница 222: ...the cable retention bracket 1 Remove all cables routed through the cable retention bracket see Figure 124 2 Press the tab and slide the cable retention bracket toward the front of the chassis to relea...

Страница 223: ...e information tag from both the failed Node chassis and the replacement Node chassis Removing the Information Tag Replacing the Information Tag Removing the Information Tag To remove the information t...

Страница 224: ...tag out of the slot to remove it from the control panel Figure 126 Removing and Installing the Information Tag Replacing the Information Tag To replace the information tag 1 Replace the information t...

Страница 225: ...es until you have shut down the DXi6902 see Turning On and Shutting Down the System on page 21 If a SAS cable is removed while the system is running data may be lost Table 13 DXi6902 Replacement Cable...

Страница 226: ...Chapter 3 DXi6902 FRU Removal and Replacement Procedures DXi6902 Replacement Cables 202 Quantum DXi6902 User s Guide 9 02558 01 Replacement power cable C13 to C14 Part Number Description...

Страница 227: ...ting the DXi6902 BIOS The DXI6902 includes a custom Quantum BIOS that is installed on the system board in the Node After replacing the system board the BIOS is automatically updated to the correct Qua...

Страница 228: ...Enter 3 Verify that all BIOS settings are set to the Quantum required value and make any changes as needed see Table 14 Note Required values that differ from the default values are shaded yellow Pay...

Страница 229: ...nsaction ID Settings Disabled Enabled Disabled Virtualization Technology Disabled Enabled Enabled Adjacent Cache Line Prefetch Enabled Disabled Enabled Hardware Prefetcher Enabled Disabled Enabled DCU...

Страница 230: ...er 1 PERC H710 Mini Integrated RAID Controller 1 PERC H710 Mini One Time Boot Device List Disabled BIOS Boot Sequence Device BIOS Hard Disk Drive Sequence Device UEFI Boot Sequence Device Disabled Int...

Страница 231: ...Device2 COM1 Serial Device1 COM1 Serial Device2 COM2 Serial Device1 COM1 Serial Device2 COM2 omconfig chassis biossetup attribute SerialPortAddress setting Serial1Com1Serial2Co m2 External Serial Con...

Страница 232: ...PM Clear No No Intel TXT On Off Off Power Button Enabled Disabled Enabled NMI Button Disabled Enabled Disabled AC Power Recovery Last On Off Off omconfig chassis biossetup attribute AcPwrRcvry setting...

Страница 233: ...209 Report Keyboard Errors Report Do Not Report Do Not Report omconfig chassis biossetup attribute ReportKbdErr setting NoReport F1 F2 Prompt on Error Enabled Disabled Disabled omconfig chassis biosse...

Страница 234: ...Chapter 4 DXi6902 BIOS Settings Required DXi6902 BIOS Settings 210 Quantum DXi6902 User s Guide...

Страница 235: ...ode to RBODs Figure 128 DXi6902 SAS Cabling RBODs to EBODs Figure 129 DXi6902 SAS Cabling EBODs 1 1 through 1 7 Figure 130 DXi6902 SAS Cabling EBODs 2 1 through 2 6 Figure 131 DXi6902 Power Cabling Fi...

Страница 236: ...212 Quantum DXi6902 User s Guide Figure 127 DXi6902 SAS Cabling Node to RBODs Port 1 Port 2 ID Diag O O Port 1 Port 2 ID Diag O O Port 1 Port 2 ID Diag O O Port 1 Port 2 ID Diag O O System Node Rear R...

Страница 237: ...3 Figure 128 DXi6902 SAS Cabling RBODs to EBODs ID Diag Port 1 ID Diag Port 1 Port 1 Port 2 ID Diag O O Port 1 Port 2 ID Diag O O RBOD 2 EBOD 2 1 1A 1B 1A 1B ID Diag Port 1 ID Diag Port 1 Port 1 Port...

Страница 238: ...t 1 ID Diag Port 1 EBOD 1 7 EBOD 1 6 ID Diag Port 1 ID Diag Port 1 EBOD 1 4 EBOD 1 3 ID Diag Port 1 ID Diag Port 1 ID Diag Port 1 ID Diag Port 1 ID Diag Port 1 ID Diag Port 1 EBOD 1 5 EBOD 1 2 EBOD 1...

Страница 239: ...hrough 2 6 ID Diag Port 1 ID Diag Port 1 EBOD 2 6 EBOD 2 5 ID Diag Port 1 ID Diag Port 1 EBOD 2 3 EBOD 2 2 ID Diag Port 1 ID Diag Port 1 ID Diag Port 1 ID Diag Port 1 EBOD 2 4 EBOD 2 1 ID Diag Port 1...

Страница 240: ...ng System Node Rear RBOD 1 or 2 ID Diag Port 1 ID Diag Port 1 EBOD 0 to 13 Port 1 Port 2 ID Diag O O Port 1 Port 2 ID Diag O O Primary AC Source Secondary AC Source Primary AC Source Secondary AC Sour...

Страница 241: ...Quantum DXi6902 User s Guide 217 Figure 134 DXi6902 Ethernet Cabling Optional Quad Port 1 GbE Card Figure 135 DXi6902 Ethernet Cabling Optional Dual Port 10 GbE Card System Node Rear To 1 GbE Network...

Страница 242: ...Appendix A DXi6902 Cabling Diagrams 218 Quantum DXi6902 User s Guide...

Результаты поиска

Содержание DXi6902

Отзывы:

Quantum DXi6902, Руководство по обслуживанию на месте, Страница: 32 / 242

Результаты поиска

Содержание DXi6902

Отзывы: