background image

Figure 64 shows an incorrect configuration of a single host with both iSCSI and
Fibre Channel connections.

Important:

Do not cable the storage subsystem and host systems as shown in
Figure 64. A storage subsystem with the incorrect storage topology
might appear to be functional, but incorrect cabling can potentially have
unpredictable results.

Figure 65 on page 94 shows an example of a multiple-host, multiple-port,
multiple-switch configuration.

Gigabit
Ethernet
switch

Host with two single-port iSCSI

HBAs and one dual-port Fibre

Channel HBA

DS5020

Fibre
Channel
switch

Controller A

Controller B

Fibre
Channel
ports

iSCSI
ports

Fibre
Channel
ports

iSCSI
ports

Fibre
Channel
ports

iSCSI
ports

Figure 64. Example of an incorrect configuration: Single-host, iSCSI and Fibre Channel

Chapter 3. Cabling the storage subsystem

93

Содержание System Storage DS5020

Страница 1: ...IBM System Storage DS5020 Storage Subsystem Installation User s and Maintenance Guide GA32 0954 03...

Страница 2: ......

Страница 3: ...IBM System Storage DS5020 Storage Subsystem Installation User s and Maintenance Guide GA32 0954 03...

Страница 4: ...o the IBM System Storage DS5020 Storage Subsystem with controller firmware version 7 60 and to all subsequent releases and modifications until otherwise indicated in new editions This edition replaces...

Страница 5: ...nnel defined 3 iSCSI defined 3 SAS defined 3 SATA defined 4 T10 PI defined 4 Inventory checklist 4 Product updates and support notifications 6 Best practices guidelines 6 Storage subsystem components...

Страница 6: ...nnecting an LC LC cable to an SFP module 54 Removing an LC LC Fibre Channel cable 56 Connecting storage expansion enclosures to the DS5020 56 Redundant drive channel pair 57 Overview of steps to conne...

Страница 7: ...on 101 Firmware updates 102 Troubleshooting the storage subsystem 102 Checking the LEDs 103 AC power supply and fan unit LEDs 103 Front LEDs 104 Battery unit LEDs 105 Controller LEDs 106 Seven segment...

Страница 8: ...ary 199 DS5000 and DS4000 storage expansion enclosure documents 200 Other DS5000 and DS4000 related documents 201 Appendix F Accessibility 203 Notices 205 Trademarks 206 Important notes 207 Particulat...

Страница 9: ...Glossary 213 Index 223 Contents vii...

Страница 10: ...viii IBM System Storage DS5020 Storage Subsystem Installation User s and Maintenance Guide...

Страница 11: ...3 29 Unlocking the SFP module latch wire variety 53 30 LC LC Fibre Channel cable 54 31 Removing fiber optic cable protective caps 55 32 Inserting an LC LC Fibre Channel cable into an SFP module 55 33...

Страница 12: ...with two standard and two optional Fibre Channel host expansion channels 107 71 LEDs controls and connectors of controllers with two standard Fibre Channel host expansion channels and two optional iSC...

Страница 13: ...tic codes 110 20 Drive LED activity 131 21 Data transfer rates for drive modules 140 22 Symptom to FRU index 158 23 Parts listing DS5020 storage subsystem 166 24 Storage subsystem and controller infor...

Страница 14: ...xii IBM System Storage DS5020 Storage Subsystem Installation User s and Maintenance Guide...

Страница 15: ...us to you A danger statement is placed just before the description of a potentially lethal or extremely hazardous procedure step or situation v Caution These statements indicate situations that can be...

Страница 16: ...water or structural damage v Disconnect the attached power cords telecommunications systems networks and modems before you open the device covers unless instructed otherwise in the installation and co...

Страница 17: ...t 3 CAUTION When laser products such as CD ROMs DVD drives fiber optic devices or transmitters are installed note the following v Do not remove the covers Removing the covers of the laser product coul...

Страница 18: ...Statement 5 CAUTION The power control button on the device and the power switch on the power supply do not turn off the electrical current supplied to the device The device also might have more than...

Страница 19: ...y the NEC and IEC 60950 1 First Edition The Standard for Safety of Information Technology Equipment v Connect the equipment to a reliably grounded safety extra low voltage SELV source An SELV source i...

Страница 20: ...accessories sold with this product will expose you to lead a chemical known to the State of California to cause cancer and birth defects or other reproductive harm Wash hands after handling xviii IBM...

Страница 21: ...and off recover from an overheated power supply and fan unit troubleshoot and interpret LEDs Chapter 5 Replacing components on page 123 contains step by step instructions about how to install or remov...

Страница 22: ...Host Support Guide for storage subsystems with controller firmware version 7 8x xx xx or later v DS4100 Storage Subsystem Installation User s and Maintenance Guide v DS4200 Express Storage Subsystem I...

Страница 23: ...ide v DS4100 Storage Subsystem Installation User s and Maintenance Guide v IBM System Storage DS5000 Installation User s and Maintenance Guide v IBM System Storage DS5020 Installation User s and Maint...

Страница 24: ...m 7 Configure DS5020 network settings v IBM System Storage DS Storage Manager Version 10 Installation and Host Support Guide for storage subsystems with controller firmware version earlier than 7 8x x...

Страница 25: ...ontroller firmware version earlier than 7 8x xx xx v IBM System Storage DS Storage Manager Version 10 8 Installation and Host Support Guide for storage subsystems with controller firmware version 7 8x...

Страница 26: ...storage subsystems with controller firmware version 7 8x xx xx or later v DS Storage Manager online help 17 Configure host partitions 18 Verify host access to DS5020 storage subsystem 19 Add storage c...

Страница 27: ...Web site www ibm com systems support storage disk 2 Click the link for your storage subsystem for example DS5020 3 When the subsystem support page opens click the Download tab 4 Under the Download ta...

Страница 28: ...System p servers Find the latest information supporting System p AIX and Linux servers http www ibm com systems support supportsite wss brandmain brandind 5000025 Support for System x servers Find the...

Страница 29: ...7 days a week In the U K these services are available Monday through Friday from 9 a m to 6 p m Fire suppression systems A fire suppression system is the responsibility of the customer The customer s...

Страница 30: ...xxviii IBM System Storage DS5020 Storage Subsystem Installation User s and Maintenance Guide...

Страница 31: ...s the attachment of up to six EXP520 and EXP810 storage expansion enclosures resulting in the capability to connect up to 112 E DDMs and enabling storage configurations of over 67 TB using 600 GB Fibr...

Страница 32: ...ze storage management help simplify partitioning of the DS5020 storage subsystem and strategically allocate storage capacity to maximize storage space Base DS5020 features The base DS5020 storage subs...

Страница 33: ...AS defined Fibre Channel defined Fibre Channel technology is outlined in the SCSI 3 Fibre Channel Protocol SCSI FCP standard Fibre Channel is a high speed data transport technology that is used for ma...

Страница 34: ...ity information The T10 PI model is an industry standard and is defined by the T10 Committee For more details about T10 PI and SCSI including technical specifications see the T10 Committee website htt...

Страница 35: ...t the additional kits might also be shipped in the DS5020 shipping box Note Depending on your DS5020 configuration you might need to purchase the applicable host software kit for your host server oper...

Страница 36: ...on units inside a rack cabinet or directly into external receptacles Similarly make sure that the right and left power supplies of the storage expansion enclosures attached to the DS5020 are plugged i...

Страница 37: ...ons occur The storage subsystem logical drives and subsystems are not displayed in the DS Storage Manager graphical user interface GUI The storage subsystem logical drives and subsystems do not come o...

Страница 38: ...an E DDM CRU If it fails it must be replaced in its entirety E DDM ATA translator card FC SAS interposer card bezel and tray The DS5020 drive tray is not interchangeable with the drive tray of other D...

Страница 39: ...lient program either to place it in a failed state or to place the array that is associated with the E DDM or E DDMs in an offline state before you remove it from the enclosure Controllers The DS5020...

Страница 40: ...2 168 129 101 255 255 255 0 Controller B management port 1 192 168 128 102 255 255 255 0 Controller B management port 2 192 168 129 102 255 255 255 0 Controller A iSCSI port 1 192 168 130 101 255 255...

Страница 41: ...tem configuration are not currently set to have unique single digit x1 values for the enclosures in the same redundant drive channel loop pair make the changes to the enclosure IDs in the next mainten...

Страница 42: ...ent station that is connected by an Ethernet cable to the storage subsystem controllers Controller B Controller A Fibre Channel host channels Ethernet ports Serial port Enclosure ID Dual ported drive...

Страница 43: ...hrough a Fibre Channel I O path Complete the following steps to assign static TCP IP addresses to the DS5020 storage subsystem controllers by way of the host that has Fibre Channel connectivity to the...

Страница 44: ...8 129 102 v Subnet mask 255 255 255 0 2 Start the DS Storage Manager client software The Enterprise Management window opens 3 Add the DS5020 subsystem to the Enterprise Management domain using the def...

Страница 45: ...supply maintains electrical power to the storage subsystem To preserve the optimal airflow do not remove a failed power supply and fan unit CRU from the DS5020 chassis until you are ready to replace i...

Страница 46: ...contains lithium and can explode if not properly used handled or disposed of Do not v Throw or immerse into water v Heat to more than 100 C 212 F v Repair or disassemble Dispose of the battery as requ...

Страница 47: ...ing starts after the battery is charged to the programmed level During the battery learn cycle cache will be active if the battery is in good condition If the battery fails the learn cycle it is marke...

Страница 48: ...e Note the following Laser radiation when open Do not stare into the beam do not view directly with optical instruments and avoid direct exposure to the beam Class 1 Laser statement Class 1 Laser Prod...

Страница 49: ...ller firmware and NVSRAM the storage expansion enclosure drive enclosure ESM firmware and Fibre Channel and SATA drive CRUs firmware must be installed to achieve optimal functionality manageability an...

Страница 50: ...s There are two different methods to determine the DS5020 storage subsystem and firmware versions Each method uses the DS Storage Manager client that manages the DS5020 storage subsystem Method One 1...

Страница 51: ...to the right The Drive Enclosure Component Information window opens 2 Click the ESM icon in the left pane The ESM information displays in the right pane of the Drive Enclosure Component Information wi...

Страница 52: ...ith front cage frame and midplane Table 4 DS5020 component weights Unit Weight Drive 0 75 kg 1 65 lb Power supply and fan unit 3 60 kg 7 95 lb Controller including cache battery backup unit 2 8 kg 6 1...

Страница 53: ...r lower Table 6 Temperature and humidity requirements for storage subsystem when in storage or in transit Condition Parameter Requirement Temperature Storage range 10 65 C 14 149 F Maximum rate of cha...

Страница 54: ...t above sea level Storage 30 5 m 100 ft below sea level to 3000 m 9842 ft above sea level Transit 30 5 m 100 ft below sea level to 12 000 m 40 000 ft above sea level Airflow and heat dissipation Figur...

Страница 55: ...stem Table 12 DS5020 sound levels Measurement Level Sound power normal operation 6 4 bels Sound pressure normal operation 68 dBA These levels are measured in controlled acoustical environments accordi...

Страница 56: ...ing currents for the DS5020 storage subsystem with ac power supply and fan unit CRUs AC power recovery After a total ac power failure after normal power is restored the storage subsystem performs powe...

Страница 57: ...tach Kits contact your marketing representative v Where racks are in rows front to front or back to back a gap of at least 1220 mm 48 in must separate the rows across the cold aisle v For proper airfl...

Страница 58: ...28 IBM System Storage DS5020 Storage Subsystem Installation User s and Maintenance Guide...

Страница 59: ...the shipping box in order to lighten the storage subsystem See Removing the CRUs on page 38 The following steps summarize the DS5020 installation process Each step below is detailed in this chapter o...

Страница 60: ...See Connecting storage expansion enclosures to the DS5020 on page 56 11 Perform one of the following cabling tasks to enable management of the DS5020 configuration v If you are using out of band mana...

Страница 61: ...scharge ESD observe the following precautions v Limit your movement Movement can cause static electricity to build up around you v Handle the device carefully holding it by its edges or its frame v Do...

Страница 62: ...ger DVDs Complete the following steps to prepare the DS5020 storage subsystem for installation into a rack cabinet 1 Prepare the site to meet all area environmental power and site requirements For mor...

Страница 63: ...rder more than one host software kit the additional kits might also be shipped in the DS5020 shipping box The DVD also includes the DS5020 storage subsystem controller firmware For the latest controll...

Страница 64: ...C 72 F See Environmental requirements and specifications on page 23 for details v For proper airflow do not block the air vents allow for at least 15 cm 6 in of air space v For optimal rack stability...

Страница 65: ...the 3U section where you will install the DS5020 U boundaries are shown as horizontal dashed lines in the rack mounting templates The highlighted squares on each side of the template mark the positio...

Страница 66: ...ximum This rack conforms to the EIA standard Where you place the support rails in the rack depends on where you intend to position the storage subsystem Complete the following steps to install the sup...

Страница 67: ...mounting templates on page 175 Use the preceding templates Figure 13 on page 35 and Figure 14 on page 36 to identify the proper locations for inserting M5 screws when mounting the support rails and DS...

Страница 68: ...ls are lined up evenly on the racks Failure to do so will cause the storage subsystem to fit unevenly in the rack 12 Continue with Removing the CRUs Installing the DS5020 This section provides instruc...

Страница 69: ...units v Hot swap E DDMs You do not need to remove the blank trays Statement 4 18 kg 39 7 lb 32 kg 70 5 lb 55 kg 121 2 lb CAUTION Use safe practices when lifting Attention The storage subsystem comes...

Страница 70: ...removing the power supply and fan unit Complete the following steps to remove a power supply and fan unit 1 Squeeze the latch and pull the lever open 90 so that the lever is horizontal to release the...

Страница 71: ...ion and identify your E DDMs You must record this information to be able to replace the E DDMs in their original order before they were removed 2 Release the latch on the E DDM CRU by pressing on the...

Страница 72: ...form all of the remaining procedures in this chapter 2 Using a No 2 Phillips screwdriver remove the four M4 screws from the rails two screws in each rail If you do not remove these screws the storage...

Страница 73: ...the front of the support rails b Using the M4 screws that you removed in step 2 on page 42 secure the sides in the back of the DS5020 to the rack rails c Secure the front of the DS5020 using the four...

Страница 74: ...n the DS5020 chassis in opposite orientations If you cannot fully insert the controller into the controller bay flip it 180 and reinsert it 1 Slide one controller into the empty slot in the storage su...

Страница 75: ...so that the lever on each power supply and fan unit opens toward the interior of the storage subsystem as shown in Figure 22 Do not force fit The power supply and fan unit is designed to prevent it fr...

Страница 76: ...e hot swap E DDMs that you removed before the installation Attention After you remove an E DDM CRU wait 70 seconds before replacing or reseating the E DDM CRU to allow the E DDM to properly spin down...

Страница 77: ...troller next to the indicator lights It provides a unique identifier for each enclosure in the DS5020 storage subsystem configuration The controller automatically sets the enclosure ID number You can...

Страница 78: ...sceiver SFP connects each host channel port and drive channel port to hosts and storage expansion enclosures Notes 1 IBM SFP options have been tested and approved for the IBM DS storage products For o...

Страница 79: ...e enough slack in the cables so they do not bend to a diameter of less than 76 mm 3 in or a radius less than 38 mm 1 5 in when extended or become pinched when retracted v Do not overtighten the cable...

Страница 80: ...ve SFPs on a single storage subsystem Use only short wave SFPs You can use the DS Storage Manager client to view the Storage Subsystem Profile to verify that you are not mixing long wave and short wav...

Страница 81: ...umber option number and FRU part number to identify its speed There are no physical features that distinguish a 4 Gbps SFP from an 8 Gbps SFP Statement 3 CAUTION When laser products such as CD ROMs DV...

Страница 82: ...n To avoid damage to the cable or to the SFP module make sure that you unplug the LC LC Fibre Channel cable before you remove the SFP module 1 Remove the LC LC Fibre Channel cable from the SFP module...

Страница 83: ...l cables The LC LC Fibre Channel cable is a fiber optic cable that you use to connect DS5020 Fibre Channel ports into one of the following devices v An SFP module installed in a Fibre Channel switch p...

Страница 84: ...result in hazardous radiation exposure DANGER Some laser products contain an embedded Class 3A or Class 3B laser diode Note the following Laser radiation when open Do not stare into the beam do not v...

Страница 85: ...er end of the LC LC cable Save the protective caps for future use 6 Connect this end of the LC LC cable to one of the following devices v An SFP module that is installed in a separate storage expansio...

Страница 86: ...on the connector to remove the cable from the SFP module as shown in Figure 34 3 Replace the protective caps on the cable ends 4 Replace the protective cap on the SFP module Connecting storage expansi...

Страница 87: ...20 configuration you must purchase the DS5020 Drive Attach 65 112 feature option in addition to the DS5020 Drive Attach 33 64 feature option These two feature options enable you to attach the addition...

Страница 88: ...in which the DS5020 is powered on when storage expansion enclosures are connected refer to the IBM System Storage DS4000 Hard Drive and Storage Expansion Enclosure Installation and Migration Guide Dri...

Страница 89: ...rage expansion enclosures after you turn on the power to the configuration Always connect power to the storage expansion enclosures first and then connect power to the DS5020 After you have powered on...

Страница 90: ...5020 hardware might differ slightly from the following illustration Notes 1 EXP810 and EXP520 storage expansion enclosures can be intermixed behind a DS5020 drive channel 2 To cable EXP810 storage exp...

Страница 91: ...storage expansion enclosures Do not use Figure 37 as the cabling diagram Follow the instructions in DS5020 storage subsystem and supported storage expansion enclosure drive cabling schemes on page 69...

Страница 92: ...shown in Figure 38 As Figure 38 shows when you connect additional storage expansion enclosures you can connect them by using the additional port of the drive channel When you connect additional stora...

Страница 93: ...ge topology might appear to be functional but incorrect cabling can potentially have unpredictable results DS5020 Port 2 drive channel 1 Port 1 drive channel 1 Port 1 drive channel 2 Port 2 drive chan...

Страница 94: ...gure 40 shows the correct cabling topology 1A 1B DS5020 Port 2 drive channel 1 1A 1B 1A 1B 1A 1B Port 1 drive channel 1 Port 1 drive channel 2 Port 2 drive channel 2 1A 1B 1A 1B Figure 40 Cabling for...

Страница 95: ...ion enclosures Figure 41 shows the correct cabling topology 1A 1B DS5020 Port 2 drive channel 1 1A 1B 1A 1B 1A 1B Port 1 drive channel 1 Port 1 drive channel 2 Port 2 drive channel 2 1A 1B 1A 1B 1A 1B...

Страница 96: ...e correct cabling topology 1A 1B DS5020 Port 2 drive channel 1 1A 1B 1A 1B 1A 1B Port 1 drive channel 1 Port 1 drive channel 2 Port 2 drive channel 2 1A 1B 1B 1A 1B 1A 1B 1A 1B 1B 1A 1B 1B 1A 1A 1A Fi...

Страница 97: ...shows a cabling topology that is incorrect because all six storage expansion enclosures are connected to a single drive port of the dual ported drive channel in each controller Chapter 3 Cabling the...

Страница 98: ...ion is a configuration in which you have more than one type of storage expansion enclosure Figure 44 on page 69 shows the cabling topology if you have three EXP520s and three EXP810s attached to the D...

Страница 99: ...to one or more EXP810 storage expansion enclosures The following sections show cabling schemes for supported storage expansion enclosures and the DS5020 storage subsystem v One DS5020 and one EXP520 s...

Страница 100: ...e drive channel from each of controllers A and B must be connected to the same set of storage expansion enclosures to provide drive channel redundancy Do not connect a combination of more than six sto...

Страница 101: ...orage expansion enclosure Last EXP810 storage expansion enclosure DS5020 ESM port 1A ESM port 1B ESM port 1A ESM port 1A ESM port 1B ESM port 1A ESM port 1A ESM port 1B Drive channel port Other storag...

Страница 102: ...annel 1 in controller A on the DS5020 to port 1B on the left ESM of the EXP520 2 Connect either port 1 or port 2 of the dual ported drive channel 2 in controller B on the DS5020 to port 1B on the righ...

Страница 103: ...XP520 one 2 Connect port 1B on the right ESM in EXP520 one to port 1 of drive channel 2 on the DS5020 3 Connect port 1 of drive channel 1 on the DS5020 to port 1B on the left ESM in EXP520 two 4 Conne...

Страница 104: ...Perform the following steps to create the cabling scheme shown in Figure 48 1 Connect port 1A on the left ESM in EXP520 one to port 1B on the left ESM in EXP520 two 2 Connect port 1A on the right ESM...

Страница 105: ...rt 1 of drive channel 1 on the DS5020 6 Connect port 1B on the right ESM in EXP520 three to port 2 of drive channel 2 on the DS5020 Figure 49 on page 76 shows the cabling scheme for one DS5020 storage...

Страница 106: ...2 of drive channel 1 on the DS5020 4 Connect port 1B on the right ESM in EXP520 two to port 1 of drive channel 2 on the DS5020 1A 1B DS5020 Port 2 drive channel 1 1A 1B 1A 1B 1A 1B Port 1 drive channe...

Страница 107: ...1B on the left ESM in EXP520 three to port 1 of drive channel 1 on the DS5020 8 Connect port 1B on the right ESM in EXP520 four to port 2 of drive channel 2 on the DS5020 Figure 50 on page 78 shows t...

Страница 108: ...drive channel 2 Port 2 drive channel 2 1A 1B 1B 1A 1B 1A 1B 1A 1B 1B 1A 1B 1B 1A 1A 1A EXP520 one EXP520 two EXP520 three EXP520 four EXP520 five EXP520 six Figure 50 One DS5020 and a maximum of six E...

Страница 109: ...on the right ESM in EXP520 three to port 1 of drive channel 2 on the DS5020 7 Connect port 1A on the left ESM in EXP520 four to port 1B on the left ESM in EXP520 five 8 Connect port 1A on the right E...

Страница 110: ...e of 00 77 However if your DS5020 enclosure ID was set to a value within the range of 00 77 it might be in conflict with the enclosure ID of one of the drive expansion enclosures that you are trying t...

Страница 111: ...ports for controller A located on the top and controller B located on the bottom on the back of the storage subsystem Figure 51 shows the location of the ports on the DS5020 storage subsystem Only on...

Страница 112: ...t server that the storage subsystem logical drives are mapped to is not Microsoft Windows 2000 or Server 2003 then you have to make a direct out of band management connection to the storage subsystem...

Страница 113: ...or change the default controller Ethernet IP addresses by using Dynamic Host Configuration Protocol DHCP or Storage Manager client Refer to the DS Storage Manager Installation User s and Maintenance...

Страница 114: ...20 RAID controllers perform the following steps 1 Install SFP modules in host ports on RAID controllers A and B Note The alignment of the single ported host channels on controller A are opposite those...

Страница 115: ...ation 3 Repeat steps 1 on page 84 and 2 on page 84 for each additional redundant host connection You can create two redundant host connections for the DS5020 storage subsystem as shown in Figure 55 on...

Страница 116: ...0 supports redundant drive channel loop configurations You can build a redundant drive channel loop by connecting one or more storage expansion enclosures to the redundant RAID controllers Then use du...

Страница 117: ...ions as shown in Figure 56 on page 88 v Single SAN fabric configuration as shown in Figure 57 on page 88 v Dual SAN fabric configuration as shown in Figure 58 on page 89 v Two storage subsystems in a...

Страница 118: ...t system with two host adapters DS5020 Host system with two host adapters Host system with two host adapters DS5020 Figure 56 Examples of redundant host direct attached Fibre Channel SAN configuration...

Страница 119: ...n DS Storage subsystem Host system with two host adapters Host system with two host adapters Switch Switch Figure 58 Example of a dual SAN fabric configuration Host system with two host adapters Host...

Страница 120: ...ultiple port configuration as shown in Figure 61 on page 91 v Multiple host multiple port configuration as shown in Figure 62 on page 91 v iSCSI and Fibre Channel hosts multiple port configuration as...

Страница 121: ...tiple host multiple port and redundant path and controller configuration You can replace the two gigabit Ethernet switches in the illustration with one large gigabit Ethernet switch that has the requi...

Страница 122: ...annel HBA DS5020 Fibre Channel switch Controller A Controller B Fibre Channel ports iSCSI ports Fibre Channel ports Host 2 with two single port iSCSI HBAs iSCSI ports Figure 63 Example of iSCSI and Fi...

Страница 123: ...ially have unpredictable results Figure 65 on page 94 shows an example of a multiple host multiple port multiple switch configuration Gigabit Ethernet switch Host with two single port iSCSI HBAs and o...

Страница 124: ...s to connect the ac power supply cable 1 Using the supplied strap tie the power supply cord to the rail to provide strain relief for the power cord 2 Connect the power cable to the power supply 3 Plug...

Страница 125: ...logs at once and zip them into a single file by clicking Advanced Troubleshooting Collect All Support Data in the DS Storage Manager Subsystem Management window v DS5020 storage subsystem management...

Страница 126: ...stem Storage DS5020 and DS4000 Interoperability Matrix at the following Web site http www ibm com systems support storage config ssic index jsp Hardware responsibilities In addition to the Health Chec...

Страница 127: ...power outage see Restoring power after an unexpected shutdown on page 114 Turning on the storage subsystem The following procedure covers two situations v The entire storage subsystem has been shut d...

Страница 128: ...e is started by the DS5020 controller 6 Turn on both power switches on the back of each controller in the storage subsystem Depending on the number of storage expansion enclosures in the configuration...

Страница 129: ...Guide for storage subsystems with controller firmware version earlier than 7 8x xx xx v IBM System Storage DS Storage Manager Version 10 8 Installation and Host Support Guide for storage subsystems wi...

Страница 130: ...Manager event monitoring service to enable continuous monitoring of the status of the storage subsystem For more information about the importance of this information see Monitoring status through sof...

Страница 131: ...ly the DS Storage Manager client can identify the risk level and provide the necessary recovery procedures Note For PFA flags the General system error LED and Drive Needs Attention LEDs do not turn on...

Страница 132: ...ile that is included in each firmware or DS Storage Manager software package for any limitations subsystem firmware prerequisites or download sequence information For example the controller firmware c...

Страница 133: ...h this storage subsystem create save and print a storage subsystem profile This information might be helpful when troubleshooting Contact IBM Customer and Technical Support for assistance When the rec...

Страница 134: ...LEDs and controls on the front of the DS5020 storage subsystem Power supply and fan unit power LED Service action allowed ok to remove LED Service action required fault LED Direct current enabled LED...

Страница 135: ...te the locate function in the DS Storage Manger Subsystem Management window menu The normal state is off N A Global Summary Fault Off On Global Power On Off Link Rate Switch Set to the 4 Gbps position...

Страница 136: ...mine the problem with the storage subsystem Also inspect the Needs Attention LEDs on the other components in the storage subsystem to help isolate the fault to a specific component Controller LEDs Thi...

Страница 137: ...controllers with two standard Fibre Channel host expansion channels Heartbeat LED Diagnostic LED Cache active LED Service action required LED Service action allowed LED Host channel speed L1 LED Driv...

Страница 138: ...eed On 1 Gbps Off 100 Mbps Not applicable Note The iSCSI ports have the same LEDs as the management Ethernet ports As with the Ethernet port LEDs the iSCSI port LEDs represent link speed and link acti...

Страница 139: ...e off this indicates one or more of the following conditions v The host or drive channel ports are bad or the SFPs are faulty v An SFP module is inserted with no Fibre Channel cable attached Note If t...

Страница 140: ...eneral these codes are displayed only when the storage subsystem is in a non operational state The storage subsystem might be non operational due to a configuration problem such as mismatched types or...

Страница 141: ...see Restoring power after an unexpected shutdown on page 114 Turning off the storage subsystem Use the following procedure to turn off the power for the DS5020 Attention Except in an emergency never t...

Страница 142: ...eview the following information before you continue with the power off procedure below Turn off the power to each device based on the following shutdown sequence 1 Turn off power to the host before th...

Страница 143: ...s the status Optimal on all configuration components v Yes Go to step 6 v No Go to step 5 5 To diagnose and correct the fault complete the following steps a Run the Recovery Guru by selecting the Reco...

Страница 144: ...re failure in the power system or in the midplane Complete the following steps to shut down the system during an emergency 1 If you have time stop all I O activity to the storage subsystem by shutting...

Страница 145: ...quired 3 Check the system documentation of the hardware devices that you want to turn on and determine the proper startup sequence Note Be sure to turn on all of the storage expansion enclosures and v...

Страница 146: ...r than 55 C do not assume the power supply and fan unit CRU is bad Take action to cool down the environment first When the operating environment temperature is within the specified operating environme...

Страница 147: ...power switch on each controller in the storage subsystem power off all connected storage expansion enclosures and flip the disconnected devices to the OFF position Wait 1 minute and then turn on power...

Страница 148: ...he battery Each RAID controller in the DS5020 storage subsystem contains either 1 GB or 2 GB of cache memory to store read and write operations Both RAID controllers in the DS5020 must have the same a...

Страница 149: ...ger client to check your cache memory settings before assuming a hardware failure Figure 73 shows the locations of the Cache Active LEDs on the front of the RAID controllers Subsystem cache battery Th...

Страница 150: ...the Storage Manager online help Replace the battery only when it is in a failed state as indicated by the Storage Manager Recovery Guru Each power supply and fan unit has green Battery LEDs on the ba...

Страница 151: ...new battery is installed to replace a failed battery or when the subsystem power is turned on after months of inactivity Attention Write caching is suspended while a battery pack is either charging o...

Страница 152: ...122 IBM System Storage DS5020 Storage Subsystem Installation User s and Maintenance Guide...

Страница 153: ...tall it directly into your storage subsystem without setting it down If it is necessary to set the device down place it in its static protective package Do not place the device on your storage subsyst...

Страница 154: ...pply and fan units is not optimal replace that component before you proceed with the controller replacement procedure v If you are replacing an optimal controller first make sure that the other contro...

Страница 155: ...nt 3 CAUTION When laser products such as CD ROMs DVD drives fiber optic devices or transmitters are installed note the following v Do not remove the covers Removing the covers of the laser product cou...

Страница 156: ...n Figure 76 c With the SFP latch in the unlocked position extract the SFP module v For SFP modules that contain plastic tabs slide the SFP module out of the port v For SFP modules that contain wire ta...

Страница 157: ...not have the cache battery backup unit already installed follow the instructions in Replacing a battery unit on page 147 to remove the cache backup battery from the existing controller unit and insert...

Страница 158: ...icates a problem status go to Troubleshooting the storage subsystem on page 102 v If the new controller is offline go to step 17 17 If the newly inserted controller is in an offline state see the DS S...

Страница 159: ...on the drive drawer board Working with hot swap E DDMs This section explains how you can increase the storage subsystem capacity by adding more E DDMs or by replacing existing E DDMs with ones that h...

Страница 160: ...refer to the IBM Storage Systems support Web site http www ibm com systems support storage disk Use of non supported drives in the drive modules can cause the storage subsystem to fail In configurati...

Страница 161: ...o indicate that the drive is properly installed and is spun up by the DS5020 controller Fault LED Flashing amber The amber LED flashes to indicate that a drive has been identified by the software Faul...

Страница 162: ...0 EV DDMs and FRUs 1 Read the documentation that comes with the E DDM 2 Check for Fault LEDs shown in Figure 81 If any amber LEDs are lit see Solving problems on page 157 3 Determine the bay into whic...

Страница 163: ...is associated with the E DDM or E DDMs in an offline state before you remove it from the storage subsystem Attention Failure to replace the E DDMs in their correct bays might result in loss of data If...

Страница 164: ...ll the E DDM again Attention After you remove an E DDM CRU wait 70 seconds before replacing or reseating the E DDM CRU to allow it to properly spin down Failure to do so might cause undesired events 8...

Страница 165: ...use depends on the following considerations v Which method most closely matches the drive upgrade procedure in the operating system or the storage management software documentation v Which RAID level...

Страница 166: ...replacing You need the backup to restore data on the E DDMs later in this procedure Attention When you handle static sensitive devices take precautions to avoid damage from static electricity For deta...

Страница 167: ...stem documentation for the hardware devices that you intend to power on and then determine the proper startup sequence Use the following power on sequence where applicable a Turn on the power to the s...

Страница 168: ...the paragraphs that discuss the differences between the two possible upgrade procedures v Your software documentation regarding E DDM upgrades and installation v The documentation that comes with the...

Страница 169: ...t copying to the new E DDM until the data is reconstructed on the hot spare This increases the time that is required to complete the procedure 7 Check the green Drive activity LED and the amber Drive...

Страница 170: ...the link rate setting for more information Do not mix 2 GB drives and 4 GB drives in the same storage subsystem If drives of mixed speeds reside in the storage subsystem the data transfer rate changes...

Страница 171: ...ator light is on the data transfer rate is set for 2 Gbps Important Make sure that the link rate for the DS5020 is always set to 4 Gbps The DS5020 supports only 4 Gbps FC drives 3 Is the data transfer...

Страница 172: ...nly the green indicator LEDs on v Yes The drive is working correctly v No If an amber Fault indicator LED is on go to 10 10 Diagnose and correct the fault a To run the Recovery Guru select the Recover...

Страница 173: ...ts fail or if the power supply and fan units cannot maintain an internal temperature below 68 C 154 F the power supply and fan units in the unit will automatically shut down an over temperature condit...

Страница 174: ...ep 3 3 Put on anti static protection 4 Unpack the new power supply and fan unit Save all packing material in case you need to return the failed power supply and fan unit CRU Note The new power supply...

Страница 175: ...o Disconnect 1 Turn everything OFF 2 First attach all cables to devices 3 Attach signal cables to connectors 4 Attach power cords to outlet 5 Turn device ON 1 Turn everything OFF 2 First remove power...

Страница 176: ...ac and power LED are off The new unit might be installed incorrectly The power supply and fan unit switch might not be turned on The power cord plug might not be fully inserted into the power outlet o...

Страница 177: ...Each RAID controller in the DS5020 storage subsystem contains a rechargeable battery unit that backs up data in cache to flash memory in the event of power failure Use the following procedure if the D...

Страница 178: ...o determine which of the two battery units has failed See Battery unit LEDs on page 105 Replace only the battery unit that is indicated as failed by the LEDs 4 Put on anti static protection 5 To remov...

Страница 179: ...the event you need to return the new battery unit 8 Insert the new battery unit into the controller chassis Make sure the new battery unit is inserted in the correct orientation in the battery unit ba...

Страница 180: ...ctionality Figure 87 on page 151 illustrates installing an SFP module Electrostatic discharge can damage sensitive components To prevent electrostatic discharge damage to the storage subsystem use pro...

Страница 181: ...perform path diagnostics to make sure that the Fibre Channel cable is good and the SFP on the other end of the Fibre Channel connection is working properly When finished go to step 12 v Bypass LED and...

Страница 182: ...ce make sure that all power cords are disconnected from the power supply and fan unit input connectors 2 Label and remove the power cords from the power supply and fan units of the DS5020 storage subs...

Страница 183: ...he E DDM CRUs and any blank drive CRUs if the storage subsystem has them See Removing an E DDM on page 41 for instructions Place E DDMs on ESD protected surface Attention Do not stack E DDM CRUs on to...

Страница 184: ...Grab the two sides of the front cage frame and remove it from the chassis Note The front cage frame might be tightly secured in the chassis Mount the DS5020 chassis back in the rack using the four M5...

Страница 185: ...on them before you removed them to guide you 21 Re insert the controllers and reconnect the Fibre Channel cables 22 Re insert the power supply and fan units and reconnect the power supply cords 23 Tu...

Страница 186: ...156 IBM System Storage DS5020 Storage Subsystem Installation User s and Maintenance Guide...

Страница 187: ...m Storage DS Storage Manager client program also provides you with additional diagnostic aids Solving problems This section contains information to help you solve some of the problems you might have w...

Страница 188: ...ED Note Controller Service Action Allowed LED will also be lit RAID controller failure Replace the RAID controller See your controller documentation for more information See Chapter 5 Replacing compon...

Страница 189: ...D is lit also replace the RAID controller Front panel Global Summary Fault LED General machine fault A Fault LED is lit somewhere on the storage subsystem check for Amber LEDs on CRUs General machine...

Страница 190: ...lashing All drive CRUs The DS5020 controllers are going through start of day process booting up Wait up to five minutes for the controllers to complete the booting process If the drive Activity LEDs a...

Страница 191: ...roller drive SFP port does not operate at 4 Gbps Use the Recovery Guru menu function in the Storage Manager Client Subsystem Management window to verify the problem and replace the SFP with a 4 Gbps S...

Страница 192: ...technical support representative Several CRUs Hardware failure Replace the affected CRUs If this does not correct the problem replace the RAID controllers followed by the midplane Contact an IBM techn...

Страница 193: ...removed the empty drive slots must be filled with other drives or blank drive trays to ensure optimal air flow Drives are incompatible Power to incompatible drives is turned off when they are inserte...

Страница 194: ...pport representative Hard disk drive is not visible in RAID management software Several CRUs Drive or drives failed Replace failed drive or drives Fibre Channel cable failed Replace Fibre Channel cabl...

Страница 195: ...ubsystem supports up to 32 drives which includes 16 drives in the DS5020 chassis and the 16 more drives if an EXP520 storage expansion enclosure is attached To attach more than 32 drives you much purc...

Страница 196: ...23 Parts listing DS5020 storage subsystem Index DS5020 storage subsystem CRU P N 1 Rail kit 41Y5152 2 Battery 59Y5260 1 8 2 3 4 5 6 7 9 10 Figure 90 DS5020 storage subsystem parts list 166 IBM System...

Страница 197: ...C drive channel ports 59Y5254 Controller 2 GB memory two standard 8 Gbps FC host ports two standard 4 Gbps FC drive channel ports 59Y5256 Controller 2 GB memory two standard 8 Gbps FC host ports one o...

Страница 198: ...15 000 RPM drive module 59Y5341 E DDM 3 Gbps SATA 750 GB HS 7 200 RPM drive module 43W9715 E DDM 3 Gbps SATA 1 TB 7 200 RPM drive module 44X3241 E DDM 3 Gbps SATA 2 TB HS 7 200 RPM drive module 59Y548...

Страница 199: ...n that of the drive FRUs they replace These new drives are programmed in manufacturing to give the same capacity as the drive FRUs being replaced The new drive FRUs have different model IDs than the o...

Страница 200: ...o the drive properties given here Refer to the drive properties given here dcsi0035 Figure 91 An IBM hologram label example 170 IBM System Storage DS5020 Storage Subsystem Installation User s and Main...

Страница 201: ...data whenever you contact your IBM technical support representative Identification numbers Record and retain the following information Product name IBM System Storage DS5020 Storage Subsystem Machine...

Страница 202: ...st or Domain Name System DNS table The information is also helpful if you add storage subsystems after initial installation Refer to your Storage Manager documentation for detailed instructions about...

Страница 203: ...tion record Storage subsystem name Management method Controllers Ethernet and IP addresses and host name Host IP address and host name Controller A Controller B Finance Direct Hardware Ethernet addres...

Страница 204: ...orresponding bay number Attention If you replace a hard disk drive in the wrong drive bay it might cause a loss of data Table 26 Hard disk drive record Bay number Hard disk drive serial number Replace...

Страница 205: ...gure 93 on page 177 to identify the proper locations for inserting M5 screws when mounting the support rails and DS5020 to a rack The locations for the M5 screws are highlighted in the templates The D...

Страница 206: ...Figure 92 Front rack mounting template 176 IBM System Storage DS5020 Storage Subsystem Installation User s and Maintenance Guide...

Страница 207: ...1U 1U 0U 0U Rail assembly Rear Left Rail assembly Rear right Printed in the U S A REAR Rack Mounting Template DS5020 Rail mounting guide Figure 93 Rear rack mounting template Appendix B Rack mounting...

Страница 208: ...178 IBM System Storage DS5020 Storage Subsystem Installation User s and Maintenance Guide...

Страница 209: ...r any other products installed in the rack or cabinet that connect to hazardous voltage Examples of OSHA approved NRTLs for the U S v UL v ETL v CSA with CSA NRTL or CSA US mark Examples of approved N...

Страница 210: ...example a power distribution unit with a 30 A connection will be rated for a total load of 24 A 30 A x 80 percent Therefore the sum of all equipment connected to the power distribution unit in this e...

Страница 211: ...for safe servicing of any of the product s installed in the racks Contact your Installation Planning Representative if in doubt Rack specifications When installing a DS5000 storage subsystem or DS5000...

Страница 212: ...he rack or cabinet must be 719 mm 28 3 in apart and the internal width bounded by the mounting flanges at least 494 mm 19 45 in for the IBM Storage System or eServer rails to fit in your rack or cabin...

Страница 213: ...r 6 75 mm min 15 9 mm 15 9 mm 12 7 mm 15 9 mm 15 9 mm 12 7 mm Figure 95 Rack specifications dimensions top front view Bottom Front of Rack EIA Hole Spacing 6 75 mm min 15 9 mm 15 9 mm 15 9 mm 15 9 mm...

Страница 214: ...ave stabilization feet or brackets installed both in the front and rear of the rack or have another means of preventing the rack cabinet from tipping while the drawer or device is pulled into its extr...

Страница 215: ...let temperature as specified in the server specifications The perforations should yield at least 34 percent minimum open area per square inch Appendix C Specifications for non IBM rack installation 18...

Страница 216: ...186 IBM System Storage DS5020 Storage Subsystem Installation User s and Maintenance Guide...

Страница 217: ...ree conductor cord a maximum of 15 feet in length and a tandem blade grounding type attachment plug rated 15 amperes 250 volts For units intended to be operated at 230 volts outside the U S Use a cord...

Страница 218: ...deloupe Guinea Guinea Bissau Hungary Iceland Indonesia Iran Kazakhstan Kyrgyzstan Laos Peoples Democratic Republic of Latvia Lebanon Lithuania Luxembourg Macedonia former Yugoslav Republic of Madagasc...

Страница 219: ...8m Israel 39Y7919 9828 Power Cord 250v 10A 2 8m Liechtenstein Switzerland 39Y7922 9829 Power Cord 250v 10A 2 8m Bangladesh Lesotho Maceo Maldives Namibia Nepal Pakistan Samoa South Africa Sri Lanka S...

Страница 220: ...190 IBM System Storage DS5020 Storage Subsystem Installation User s and Maintenance Guide...

Страница 221: ...order DS Storage Manager Version 10 library Table 28 associates each document in the DS Storage Manager Version 10 library with its related common user tasks Table 28 DS Storage Manager Version 10 tit...

Страница 222: ...e DS5020 storage subsystem library with its related common user tasks Table 29 DS5020 storage subsystem document titles by user tasks Title User Tasks Planning Hardware Installation Software Installat...

Страница 223: ...Tasks Planning Hardware Installation Software Installation Configuration Operation and Administration Diagnosis and Maintenance IBM System Storage DS4800 Storage Subsystem Installation User s and Main...

Страница 224: ...IBM System Storage DS4700 Storage Subsystem Installation User s and Maintenance Guide U U U U U IBM System Storage Quick Start Guide Quick Reference for the DS4700 and DS4200 Sections 2 3 and 4 also...

Страница 225: ...s Title User Tasks Planning Hardware Installation Software Installation Configuration Operation and Administration Diagnosis and Maintenance IBM TotalStorage DS4500 Storage Subsystem Installation User...

Страница 226: ...ardware Installation Software Installation Configuration Operation and Administration Diagnosis and Maintenance IBM TotalStorage DS4400 Fibre Channel Storage Server User s Guide U U U U U IBM TotalSto...

Страница 227: ...ation Operation and Administration Diagnosis and Maintenance IBM TotalStorage DS4300 Storage Subsystem Installation User s and Maintenance Guide U U U U U IBM TotalStorage DS4300 Rack Mounting Instruc...

Страница 228: ...g Hardware Installation Software Installation Configuration Operation and Administration Diagnosis and Maintenance IBM System Storage DS4200 Express Storage Subsystem Installation User s and Maintenan...

Страница 229: ...system document titles by user tasks Title User Tasks Planning Hardware Installation Software Installation Configuration Operation and Administration Diagnosis and Maintenance IBM TotalStorage DS4100...

Страница 230: ...ge Expansion Enclosure Installation User s and Maintenance Guide U U U U U IBM System Storage Quick Start Guide Quick Reference for the DS4700 and DS4200 Sections 2 3 and 4 also for installing the EXP...

Страница 231: ...rvice Information U U U U IBM TotalStorage DS4000 FC2 133 Host Bus Adapter Installation and User s Guide U U IBM TotalStorage DS4000 FC2 133 Dual Port Host Bus Adapter Installation and User s Guide U...

Страница 232: ...202 IBM System Storage DS5020 Storage Subsystem Installation User s and Maintenance Guide...

Страница 233: ...other required host software components Table 39 defines the keyboard operations that enable you to navigate select or activate user interface components The following terms are used in the table v N...

Страница 234: ...nt to select press Ctrl Spacebar to select the drive If you press Spacebar alone when multiple drives are selected then all selections are removed Use the Ctrl Spacebar combination to deselect a drive...

Страница 235: ...S OF NON INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not...

Страница 236: ...tion softcopy the photographs and color illustrations may not appear Trademarks IBM the IBM logo and ibm com are trademarks or registered trademarks of International Business Machines Corp registered...

Страница 237: ...y not IBM Some software might differ from its retail version if available and might not include user manuals or all program functionality Particulate contamination Attention Airborne particulates incl...

Страница 238: ...lina U S A Documentation format The publications for this product are in Adobe Portable Document Format PDF and should be compliant with accessibility standards If you experience difficulties when you...

Страница 239: ...conform la norme NMB 003 du Canada Australia and New Zealand Class A Statement Attention This is a Class A product In a domestic environment this product might cause radio interference in which case t...

Страница 240: ...den angemessene Ma nahmen zu ergreifen und daf r aufzukommen Deutschland Einhaltung des Gesetzes ber die elektromagnetische Vertr glichkeit von Ger ten Dieses Produkt entspricht dem Gesetz ber die ele...

Страница 241: ...chnology Industries Association JEITA Statement Japanese Electronics and Information Technology Industries Association JEITA Confirmed Harmonics Guideline products less than or equal to 20 A per phase...

Страница 242: ...ss A Electronic Emission Statement Taiwan Contact Information IBM Taiwan Product Service Contact Info IBM Taiwan Corporation 3F No 7 Song Ren Rd Taipei Taiwan Tel 0800 016 888 f2c00790 212 IBM System...

Страница 243: ...el adapter advanced technology AT bus architecture A bus standard for IBM compatibles It extends the XT bus architecture to 16 bits and also allows for bus mastering although only the first 16 MB of m...

Страница 244: ...stop I O to the controllers during the process CRC See cyclic redundancy check CRT See cathode ray tube CRU See customer replaceable unit customer replaceable unit CRU An assembly or part that a custo...

Страница 245: ...enclosure that monitors the environmental condition of the components in that enclosure Not all storage subsystems have ESM canisters E_port See expansion port error correction coding ECC A method fo...

Страница 246: ...de Global Copy remote mirroring Metro Mirroring graphical user interface GUI A type of computer interface that presents a visual metaphor of a real world scene often of a desktop by combining high res...

Страница 247: ...evice in the Physical and Data Path maps LAN See local area network LBA See logical block address local area network LAN A computer network located on a user s premises within a limited geographic are...

Страница 248: ...Network Management Protocol SNMP a station that runs management application programs that monitor and control network elements NMI See non maskable interrupt NMS See network management station non mas...

Страница 249: ...k RVSD A virtual shared disk on a server node configured to provide continuous access to data and file systems in a cluster redundant array of independent disks RAID A collection of disk drives array...

Страница 250: ...used to convert signals between optical fiber cables and switches An SFP is smaller than a gigabit interface converter GBIC See also gigabit interface converter SNMP See Simple Network Management Prot...

Страница 251: ...t particular TL_port Transmission Control Protocol TCP A communication protocol used in the Internet and in any network that follows the Internet Engineering Task Force IETF standards for internetwork...

Страница 252: ...222 IBM System Storage DS5020 Storage Subsystem Installation User s and Maintenance Guide...

Страница 253: ...20 and four storage expansion enclosures 65 one DS5020 and one storage expansion enclosure 61 one DS5020 and three storage expansion enclosures 64 one DS5020 and two or more storage expansion enclosur...

Страница 254: ...e subsystem library 198 DS4300 Storage subsystem library 197 DS4400 Storage subsystem library 196 DS4500 Storage subsystem library 195 DS4800 Storage subsystem library 193 194 DS5020 dimensions 21 env...

Страница 255: ...119 inventory checklist 4 iSCSI loop configurations 90 iSCSI technology definition of 3 L LC LC Fibre Channel cable 53 connecting to an SFP 54 removing 56 LEDs battery unit 105 LEDs continued cache ba...

Страница 256: ...ve devices 31 SAS technology definition of 3 SATA technology definition of 4 settings Fibre Channel loop and ID 47 48 storage expansion enclosure 80 SFP modules described 18 replacing 149 shock and vi...

Страница 257: ...s 206 troubleshooting 157 no power 116 overheating problem 116 procedure 102 turning off power for planned shutdown 111 turning off the storage subsystem 111 turning on power after power supply shutdo...

Страница 258: ...228 IBM System Storage DS5020 Storage Subsystem Installation User s and Maintenance Guide...

Страница 259: ......

Страница 260: ...Part Number 00Y5047 Printed in USA GA32 0954 03 1P P N 00Y5047...

Отзывы: