manualshive.com logo in svg

IBM TotalStorage DS4800, Инструкция по установке, эксплуатации и обслуживанию

Поделиться
Скачать
IBM TotalStorage DS4800 Скачать руководство пользователя страница 19

About

 

this

 

document

 

This

 

document

 

provides

 

instructions

 

for

 

installing

 

and

 

customizing

 

the

 

configuration

 

of

 

your

 

IBM

 

System

 

Storage

 

DS4800

 

Storage

 

Subsystem.

 

It

 

also

 

provides

 

hardware

 

maintenance

 

procedures

 

and

 

troubleshooting

 

information.

 

FAStT

 

product

 

renaming

 

IBM

 

has

 

renamed

 

some

 

FAStT

 

family

 

products.

 

Table

 

1

 

identifies

 

each

 

DS4000

 

product

 

name

 

with

 

its

 

corresponding

 

previous

 

FAStT

 

product

 

name.

 

Note

 

that

 

this

 

change

 

of

 

product

 

name

 

only

 

indicates

 

no

 

change

 

in

 

functionality

 

or

 

warranty.

 

All

 

products

 

listed

 

below

 

with

 

new

 

names

 

are

 

functionally

 

equivalent

 

and

 

fully

 

interoperable.

 

Each

 

DS4000

 

product

 

retains

 

full

 

IBM

 

service

 

as

 

outlined

 

in

 

service

 

contracts

 

issued

 

for

 

analogous

 

FAStT

 

products.

 

 

Table

 

1.

 

Mapping

 

of

 

FAStT

 

names

 

to

 

DS4000

 

series

 

names

 

Previous

 

FAStT

 

Product

 

Name

 

Current

 

DS4000

 

Product

 

Name

 

IBM

 

TotalStorage

 

FAStT

 

Storage

 

Server

 

IBM

 

TotalStorage

 

DS4000

 

FAStT

 

DS4000

 

FAStT

 

Family

 

DS4000

 

Mid-range

 

Disk

 

System

 

FAStT

 

Storage

 

Manager

 

vX.Y

 

(for

 

example

 

v9.10)

 

DS4000

 

Storage

 

Manager

 

vX.Y

 

(for

 

example

 

v9.10)

 

FAStT100

 

DS4100

 

FAStT600

 

DS4300

 

FAStT600

 

with

 

Turbo

 

Feature

 

DS4300

 

Turbo

 

FAStT700

 

DS4400

 

FAStT900

 

DS4500

 

EXP700

 

DS4000

 

EXP700

 

EXP710

 

DS4000

 

EXP710

 

EXP100

 

DS4000

 

EXP100

 

FAStT

 

FlashCopy

 

FlashCopy

 

for

 

DS4000

 

FAStT

 

VolumeCopy

 

VolumeCopy

 

for

 

DS4000

 

FAStT

 

Remote

 

Mirror

 

(RM)

 

Enhanced

 

Remote

 

Mirroring

 

for

 

DS4000

 

FAStT

 

Synchronous

 

Mirroring

 

Metro

 

Mirroring

 

for

 

DS4000

 

Global

 

Copy

 

for

 

DS4000

(New

 

Feature

 

=

 

Asynchronous

 

Mirroring

 

without

 

Consistency

 

Group)

 

Global

 

Mirroring

 

for

 

DS4000

(New

 

Feature

 

=

 

Asynchronous

 

Mirroring

 

with

 

Consistency

 

Group)

  

Who

 

should

 

read

 

this

 

document

 

This

 

document

 

is

 

intended

 

for

 

system

 

operators

 

and

 

service

 

technicians

 

who

 

have

 

extensive

 

knowledge

 

of

 

fibre

 

channel

 

and

 

network

 

technology.

 

 

©

 

Copyright

 

IBM

 

Corp.

 

2006,

 

2007

 

xvii

Содержание TotalStorage DS4800

Страница 1: ...IBM System Storage DS4800 Storage Subsystem Installation User s and Maintenance Guide GC26 7845 01...

Страница 2: ......

Страница 3: ...IBM System Storage DS4800 Storage Subsystem Installation User s and Maintenance Guide GC26 7845 01...

Страница 4: ...the general information in Notices on page 197 Second Edition March 2007 Copyright International Business Machines Corporation 2006 2007 All rights reserved US Government Users Restricted Rights Use d...

Страница 5: ...step or situation v Caution These statements indicate situations that can be potentially hazardous to you A caution statement is placed just before the description of a potentially hazardous procedur...

Страница 6: ...water or structural damage v Disconnect the attached power cords telecommunications systems networks and modems before you open the device covers unless instructed otherwise in the installation and c...

Страница 7: ...ining a lithium battery replace it only with the same module type made by the same manufacturer The battery contains lithium and can explode if not properly used handled or disposed of Do not v Throw...

Страница 8: ...f controls or adjustments or performance of procedures other than those specified herein might result in hazardous radiation exposure DANGER Some laser products contain an embedded Class 3A or Class 3...

Страница 9: ...ower control button on the device and the power switch on the power supply do not turn off the electrical current supplied to the device The device also might have more than one power cord To remove a...

Страница 10: ...these components If you suspect a problem with one of these parts contact a service technician Statement 7 DANGER Handling the cord on this product or cords associated with accessories sold with this...

Страница 11: ...SATA defined 2 Features at a glance 2 Clustering support 3 Inventory checklist 3 Product updates 5 Best practices guidelines 6 Storage server components 7 Intermixing storage expansion enclosures in t...

Страница 12: ...nect storage expansion enclosure to a storage subsystem 62 DS4800 drive cabling rules and recommendations 67 DS4800 Storage Subsystem drive cabling topologies 69 One DS4800 and one storage expansion e...

Страница 13: ...controller LEDs 120 Numeric display LEDs 123 Power supply fan LEDs 125 Interconnect battery unit LEDs 127 Recovering from an overheated power supply and fan unit 129 Resetting the controller circuit b...

Страница 14: ...endix E Accessibility 195 Notices 197 Trademarks 197 Important notes 198 Product recycling and disposal 199 Battery return program 199 Electronic emission notices 200 Federal Communications Commission...

Страница 15: ...om fiber optic cables 52 30 Connecting fiber optic cable to SFP 52 31 LC LC fibre channel cable 53 32 Removing fiber optic cable protective caps 54 33 Inserting an LC LC fibre channel cable into an SF...

Страница 16: ...RAID controller LEDs 120 76 Numeric display LEDs 124 77 Power supply fan LEDs 126 78 Interconnect battery unit LEDs 127 79 Power supply fan Needs Attention LEDs 133 80 RAID controller connections 134...

Страница 17: ...4 78 24 Description of Figure 55 80 25 Description of Figure 56 on page 82 82 26 Description of Figure 57 on page 83 83 27 Description of Figure 58 on page 84 84 28 Recommended enclosure ID settings s...

Страница 18: ...xvi IBM System Storage DS4800 Storage Subsystem Installation User s and Maintenance Guide...

Страница 19: ...Name Current DS4000 Product Name IBM TotalStorage FAStT Storage Server IBM TotalStorage DS4000 FAStT DS4000 FAStT Family DS4000 Mid range Disk System FAStT Storage Manager vX Y for example v9 10 DS40...

Страница 20: ...800 Storage Subsystem It also contains the parts listing for the DS4800 Storage Subsystem Appendix A Additional DS4000 documentation on page 177 provides references to other DS4000 publications Append...

Страница 21: ...n User s and Maintenance Guide v DS4800 Storage Subsystem Installation User s and Maintenance Guide 2 Mount the DS4000 storage subsystem in the rack v DS4800 Storage Subsystem Installation User s and...

Страница 22: ...ess Storage Subsystem Installation User s and Maintenance Guide v DS4300 Storage Subsystem Installation User s and Maintenance Guide v DS4400 Fibre Channel Cabling Instructions v DS4500 Storage Subsys...

Страница 23: ...0 Storage Manager Installation and Support Guide for AIX HP UX Solaris and Linux on POWER v DS4000 Storage Manager Installation and Support Guide for Windows 2000 Server 2003 NetWare ESX Server and Li...

Страница 24: ...and Support Guide for AIX HP UX Solaris and Linux on POWER v DS4000 Storage Manager Installation and Support Guide for Windows 2000 Server 2003 NetWare ESX Server and Linux 16 Configure arrays and log...

Страница 25: ...ng the diagnostic programs The troubleshooting information or the diagnostic programs might tell you that you need additional or updated device drivers or other software Web sites The most up to date...

Страница 26: ...c systems support supportsite wss brandmain brandind 5345868In the Product family drop down menu select Disk systems and in the Product drop down menu select your Storage Subsystem for example DS4800...

Страница 27: ...eseller if your reseller is authorized by IBM to provide warranty service Go to the following Web site for support telephone numbers www ibm com planetwide In the U S and Canada hardware service and s...

Страница 28: ...h your IBM DS4800 Each caution and danger statement is numbered for easy reference to the corresponding statements in the safety document The following types of notices and statements are used in this...

Страница 29: ...RAID controllers equipped with sixteen 4 Gbps fibre channel ports eight ports per controller for attachment of host servers and DS4000 storage expansion enclosures The DS4800 supports attachment of up...

Страница 30: ...r from the DS4800 to DS4000 storage expansion enclosure ports is a 4 Gbps fibre channel connection that supports a data transfer rate up to 400 MBps at half duplex and 800 MBps at full duplex SATA def...

Страница 31: ...EXP710 and 2 Gbps 4 Gbps EXP810 storage expansion enclosures Note DS4000 EXP700s are not supported by the DS4800 unless they have been upgraded with the DS4000 EXP700 Models 1RU 1RX Switched ESM Opti...

Страница 32: ...ppropriate IBM DS4000 Storage Manager host software The CD also contains includes firmware online help and publications in Adobe Acrobat Portable Document Format PDF For a list of available IBM DS4000...

Страница 33: ...tial installation and when product updates become available To be notified of important product updates you must first register at the IBM Support and Download Web site www 1 ibm com servers storage s...

Страница 34: ...will have power in the event that only one power circuit is available In addition having all the right or all the left power cables plug into the same power circuit enables the DS4000 devices in the...

Страница 35: ...s the I O activity between a host and the drives in a RAID array The storage subsystem has eight single ported host channels four per controller and can support up to 256 hosts with two FC connections...

Страница 36: ...nts in the DS4800 All DS4800 models support hot swap RAID controllers power supply and fan units cache protected batteries and interconnect battery units so you can remove and replace these components...

Страница 37: ...pport the following DS4000 storage expansion enclosures v EXP100 v EXP710 v EXP810 See Table 4 for the maximum number of storage expansion enclosures that can be attached to the DS4800 storage subsyst...

Страница 38: ...ported storage expansion enclosures combinations per redundant drive channel loop pair in a DS4800 configuration The table assumes that a DS4800 storage subsystem is generally capable of addressing th...

Страница 39: ...s Information about the condition of the controllers is conveyed by indicator LEDs on the back of each controller RAID controller LEDs on page 120 identifies the indicator LEDs on the RAID controller...

Страница 40: ...es See Table 4 on page 9 for more information The drive channel port speeds are auto negotiated between 2 Gbps and 4 Gbps fibre channel speeds Both ports of a dual ported drive channel must be connect...

Страница 41: ...or the fibre channel switch to determine the fastest compatible speed between the controllers and the host or switch The fastest compatible speed becomes the operating speed of the link If the host or...

Страница 42: ...and 192 168 129 102 respectively The default subnet mask for all four Ethernet ports is 255 255 255 0 See Setting up IP addresses for DS4000 storage controllers for more information RS 232 serial port...

Страница 43: ...torage subsystem controllers Complete the following steps to assign static TCP IP addresses to the DS4000 storage subsystem controllers using default TCP IP addresses that are assigned to the DS4000 s...

Страница 44: ...D on the controller turns on when the cache contains data that has not been written to hard disk drives The Cache Active LED turns off when the cache does not contain unwritten data Table 6 lists the...

Страница 45: ...n electrical power and cooling to the storage subsystem Note Contact IBM for information about possible future DC to DC power supply support Install the power supply and fan units from the front of th...

Страница 46: ...lways make sure that both power supply and fan units are operational Interconnect battery unit The interconnect battery unit is a removable midplane that provides cross coupled signal connection betwe...

Страница 47: ...data access Figure 7 shows how the interconnect battery unit slides into the front of the storage subsystem The levers securing the interconnect battery units to the storage subsystem are shown in the...

Страница 48: ...v Overall DS4800 Configuration Needs Attention v Locate Identify Note The displayed order of the Overall DS4800 Configuration Needs Attention and Locate Identify LEDs on the interconnect battery unit...

Страница 49: ...pment sufficient space to install operate and service the storage subsystem and sufficient ventilation to provide a free flow of air to the unit Dimensions Figure 9 shows the dimensions of the DS4800...

Страница 50: ...on dimensions Height Width Depth 44 4 cm 17 5 in 62 2 cm 24 5 in 78 7 cm 31 0 in Environmental requirements and specifications This section describes the environmental conditions that are prerequisite...

Страница 51: ...l expose the unit to greater risk of failure from external causes Altitude Table 11 lists the acceptable altitudes for operating storing and shipping the DS4800 Storage Subsystem Table 11 DS4800 Stora...

Страница 52: ...Operational shock and operational vibration requirements are shown in this section Operational Shock The DS4800 will be able to withstand the following shock Subjecting the equipment to a single shock...

Страница 53: ...ments and power cord routing instructions Consider the following information when preparing the DS4800 Storage Subsystem installation site v Protective ground Site wiring must include a protective gro...

Страница 54: ...on They use standard industrial wiring with line to neutral or line to line power connections The agency ratings for the DS4800 Storage Subsystem are 5 amps at 100 VAC and 2 25 amps at 240 VAC These a...

Страница 55: ...et The following steps summarize the DS4800 installation process Each step below is detailed in this chapter or in the next two chapters Chapter 3 Cabling the storage subsystem and Chapter 4 Operating...

Страница 56: ...www 912 ibm com PremiumFeatures All drives in the connected storage expansion enclosure or enclosures must contain no prior configuration data Additionally if at least two drives are not installed in...

Страница 57: ...evice where others can handle and possibly damage the device v While the device is still in its static protective package touch it to an unpainted metal part of the system unit for at least two second...

Страница 58: ...or installation Before installing the DS4800 Storage Subsystem create a detailed plan of how this unit will be used in your storage configuration The plan should include determining RAID levels failov...

Страница 59: ...e proceeding 4 Assemble the tools and equipment that you will need for installation These include the following v Region specific power cords required for the DS4800 v 5 16 8 mm hex nut driver v Mediu...

Страница 60: ...e install host bus adapters HBAs switches or other devices v Route interface cables from the hosts or switches to the installation area v Route main power cords to the installation area For informatio...

Страница 61: ...tes and instructions to identify the proper locations for inserting M5 screws when mounting the support rails and DS4800 to a rack cabinet The locations for the M5 screws are highlighted in the templa...

Страница 62: ...2 U 1 U 1 U 0 U 4 U 4 U Rail assembly Front left Rail assembly Front right Rail assembly Front left Rail assembly Front right Figure 12 DS4800 front rack mounting template 34 IBM System Storage DS4800...

Страница 63: ...cm 24 in minimum and 81 28 cm 32 in maximum This rack cabinet conforms to the EIA standard Where you place the support rails in the rack cabinet depends on where you intend to position the storage su...

Страница 64: ...rails mounting hardware and line jumper cords without first removing the DS4800 from the shipping box 4 Locate the two support rails and six of the eight M5 screws that are supplied with the DS4800 5...

Страница 65: ...e for installing this screw Attention Do not completely tighten the screw until you have installed the DS4800 in the rack cabinet 9 From the rear of the rack thread and loosely tighten two M5 screws u...

Страница 66: ...engages a pin on the DS4800 Storage Subsystem enclosure to pull the component securely into the enclosure Figure 16 on page 39 shows the lever the latch that locks the lever into position and the pin...

Страница 67: ...r a listing of the weights of each DS4800 Storage Subsystem component To replace a component and lock the latches complete the following steps 1 With the levers perpendicular to the face of the compon...

Страница 68: ...g the support rails on page 33 You will need an antistatic wrist strap and a cart or level surface to hold the components Observe the precautions listed in Handling static sensitive devices on page 29...

Страница 69: ...tistatic protection and keep it on while performing all of the remaining procedures in this section 3 Remove both power supply and fan units Figure 17 shows the locations of the power supply and fan u...

Страница 70: ...the DS4800 on the support rails This section provides instructions on how to install the DS4800 in a rack cabinet Attention If you have not yet done so record the serial number machine type and model...

Страница 71: ...unit from the shipping box and into the rack cabinet For safety reasons IBM does not recommend moving the DS4800 Storage Subsystem from the shipping box and installing the DS4800 on the support rails...

Страница 72: ...with Replacing the components Replacing the components Complete the following procedure to replace the components in the storage subsystem 1 Replace the power supply and fan units Figure 21 on page 45...

Страница 73: ...e component 3 Replace the controllers at the back of the DS4800 Figure 23 on page 46 shows the location of the controllers and the controller levers Ensure that controller A is returned to the upper s...

Страница 74: ...the pins snap into place Figure 24 shows how to install the front bezel in the storage subsystem 5 Continue with Chapter 3 Cabling the storage subsystem on page 47 RAID controller A Levers in release...

Страница 75: ...ns and two dual ported drive channel connections You use a small form factor pluggable transceiver SFP to connect each host channel port and drive channel port to hosts and storage expansion enclosure...

Страница 76: ...FPs in DS4800 host or drive ports In DS4800 configurations you may use long wave SFPs only in fibre channel switches that connect DS4800s over long distances v The SFP module housing has an integral g...

Страница 77: ...specified herein might result in hazardous radiation exposure DANGER Some laser products contain an embedded Class 3A or Class 3B laser diode Note the following Laser radiation when open Do not stare...

Страница 78: ...ports in controller B SFPs should be inserted with the exposed printed circuit facing down 5 Connect a fibre channel cable Removing SFP modules To remove the SFP module from the fibre channel port pe...

Страница 79: ...port 4 Replace the protective cap on the SFP module 5 Place the SFP module into a static protective package 6 Replace the protective cap on the port Installing fiber optic cables To connect the fiber...

Страница 80: ...he protective caps Using LC LC fibre channel cables The LC LC fibre channel cable is a fiber optic cable that you use to connect DS4800 fibre channel ports to one of the following devices v An SFP mod...

Страница 81: ...on page 55 Connecting an LC LC cable to an SFP module Complete the following steps to connect an LC LC fibre channel cable to an SFP module Statement 3 CAUTION When laser products such as CD ROMs DVD...

Страница 82: ...4 Carefully insert this end of the LC LC cable into an SFP module that is installed in the DS4800 The cable connector is keyed to ensure it is inserted into the SFP module correctly Holding the conne...

Страница 83: ...nd of the LC LC cable that connects into the SFP module or host bus adapter press down and hold the lever to release the latches as shown in Figure 34 2 Carefully pull on the connector to remove the c...

Страница 84: ...to connect an LC SC fibre channel cable adapter to a device Statement 3 CAUTION When laser products such as CD ROMs DVD drives fiber optic devices or transmitters are installed note the following v D...

Страница 85: ...s for future use 4 Carefully insert the other end of the LC LC cable into the LC connector end of the LC SC cable adapter as shown in Figure 38 Push in the connector until it clicks into place 5 If yo...

Страница 86: ...d of the cable that connects into the LC connector end of the LC SC cable adapter press down and hold the lever to release the latches Figure 39 shows the location of the lever and latches 2 Carefully...

Страница 87: ...r drive migration might cause loss of configuration and other storage subsystem problems Contact your IBM support representative for additional information Redundant drive channel pair For the DS4800...

Страница 88: ...or example in drive channel loop pair 1 both ports 4 and 3 of channel 1 and port 1 and 2 of channel 3 are used Note In the following drive cabling figures the DS4800 graphics are simplified to show on...

Страница 89: ...igure 42 on page 62 shows a second example of the drive channel cabling for redundant drive channel pairs Figure 41 Example 1 of redundant drive channel pairs Chapter 3 Cabling the storage subsystem 6...

Страница 90: ...s in the Installation User s and Maintenance Guide for your DS4000 storage expansion enclosure for setting up and mounting the storage expansion enclosures 2 Select a cabling topology appropriate for...

Страница 91: ...ng diagram provided in DS4800 Storage Subsystem drive cabling topologies on page 69 that most closely matches your configuration Figure 43 shows the name of the ESM ports in the EXP710 EXP100 storage...

Страница 92: ...to the DS4800 using the first ports of the dual ported drive channels 1 and 3 For EXP810 storage expansion enclosure connections Perform the following steps to cable EXP810 storage expansion enclosure...

Страница 93: ...On the first storage expansion enclosure from the group of drive expansion enclosures that will be cabled together connect the 1B port on the right ESM board to the 1A port on the right ESM board of...

Страница 94: ...rt on the left ESM board of the first storage expansion enclosure in the group of drive expansion enclosures See callout 1 in Figure 46 Figure 46 Connecting EXP810 enclosures to DS4800 66 IBM System S...

Страница 95: ...e expansion enclosure ID settings on page 86 and refer to your storage expansion enclosure installation manual for details on setting the enclosure ID The DS4800 Storage Subsystem locates the drives i...

Страница 96: ...specific cabling rules and restrictions when intermixing EXP100 EXP710 and EXP810 storage expansion enclosures see One DS4800 to two or more storage expansion enclosures in a mixed environment on page...

Страница 97: ...nections as described in the tables If you have hardware to include in your topology other than what is shown in these examples use the examples as a starting point for creating your specific topology...

Страница 98: ...ts Do not use these graphics for actual cabling diagrams instead use the relevant instructions found in Connecting storage expansion enclosures to the DS4800 on page 59 One DS4800 and one storage expa...

Страница 99: ...expansion enclosures Figure 49 on page 72 shows the recommended cabling schemes As Figure 49 on page 72 and Figure 50 on page 73 show when you connect additional storage expansion enclosures you can c...

Страница 100: ...rt 4 of controller A to enclosure 1 2 and drive port 3 to enclosure 2 4 2 To create drive channel 3 connect drive port 1 of controller B to enclosure 1 2 and drive port 2 to enclosure 2 4 Figure 49 Vi...

Страница 101: ...bling one DS4800 Storage Subsystem to four storage expansion enclosures Figure 51 on page 74 shows the recommended cabling scheme Figure 50 View 2 One DS4800 and two storage expansion enclosures cabli...

Страница 102: ...te drive channel 3 connect drive port 1 of controller B to enclosure 1 5 and drive port 2 to enclosure 2 6 3 To create drive channel 2 connect drive port 2 of controller A to enclosure 3 7 and drive p...

Страница 103: ...nnel pairs instead of connecting in groups of up to four storage expansion enclosures behind certain ports of the two redundant drive channel pairs Figure 52 illustrates the cabling topology for one D...

Страница 104: ...onnect drive port 2 of controller B to enclosures 4 8 and 3 7 3 To create drive channel 2 after connecting enclosures 5 and 6 together connect drive port 2 of controller A to enclosures 6 10 and 5 9 a...

Страница 105: ...1 controller A 4B Connection from drive port 4 controller B 21 Drive channel 1 23 Drive channel 3 22 Drive channel 2 24 Drive channel 4 5 EXP810 enclosure 1 6 EXP810 enclosure 2 7 EXP810 enclosure 3 8...

Страница 106: ...ive port 2 of controller A 2A to enclosures 8 12 9 13 10 14 and 11 15 then connect drive port 3 of controller B 3B to enclosures 11 15 10 14 9 13 and 8 12 5 To create drive channel 4 after connecting...

Страница 107: ...ESMs in the expansion enclosures as indicated by 21 23 22 and 24 2 To create drive channel 1 after connecting enclosures 1 2 3 and 4 together 21 connect drive port 4 of controller A 4A to EXP710 encl...

Страница 108: ...port 4 controller A 1B Connection from drive port 1 controller B 3A Connection from drive port 3 controller A 2B Connection from drive port 2 controller B 2A Connection from drive port 2 controller A...

Страница 109: ...12 5 To create drive channel 4 after connecting EXP710 enclosures 12 16 13 17 14 18 and 15 19 together 24 connect drive port 1 of controller A 1A to enclosures 12 16 13 17 14 18 and 15 19 then connec...

Страница 110: ...he OUT port to the IN port on subsequent storage expansion enclosures until you complete the chain Figure 56 shows an acceptable EXP710 EXP810 and EXP100 intermix configuration in a DS4800 environment...

Страница 111: ...use storage subsystem controller is cabled to the IN port on the left ESM of the first EXP100 Table 26 Description of Figure 57 Number Description 1 DS4800 Storage Subsystem 2 EXP100 ESM B ESM A ESM B...

Страница 112: ...ures to a DS4800 storage subsystem Complete the following steps to cable the storage expansion enclosures to a DS4800 Storage Subsystem 1 Install an SFP module in the drive mini hub port on the back o...

Страница 113: ...pansion enclosure settings This section provides information about storage expansion enclosure settings For additional detail refer to the Installation User s and Maintenance Guide for your storage ex...

Страница 114: ...pansion enclosures ship with a panel installed over the enclosure ID switch to prevent the enclosure ID from being changed accidentally If your DS4000 storage expansion enclosure has a protective pane...

Страница 115: ...re ID was set to a value within the range of 00 77 it may be in conflict with the enclosure ID of one of the EXP100 or EXP710 drive expansion enclosures that you are trying to attach to the DS4800 If...

Страница 116: ...pters Figure 61 shows the location on the RAID controllers where the host system cables connect 3 Repeat steps 1 and 2 for each additional redundant host connection You can create a maximum of four re...

Страница 117: ...result in loss of data access and in some cases in loss of data Do not make connections to the serial port unless you do so under the direct supervision and guidance of IBM support personnel 2 To mini...

Страница 118: ...nagement you want to use You can manage the storage subsystems in either of two ways host agent in band management or direct out of band management For more information on setting up in band or out of...

Страница 119: ...an Ethernet network Table 29 Description of Figure 61 Number Description 1 Two DS4800s 2 Fibre channel I O path 3 Host server 4 Ethernet network 5 Management station Direct out of band management met...

Страница 120: ...gement station Fibre channel connections The storage subsystem fibre channel connection consists of up to four host fibre channel loops and a redundant drive fibre channel loop The host fibre channel...

Страница 121: ...ions in the following sections show common host system configurations Note The minimum DS4800 configuration is eight partitions Sixteen partition and sixty four partition configuration upgrades are av...

Страница 122: ...st system with two host adapters Host system with two host adapters DS4800 Switch Switch Note Node level redundancy with cluster software Host system with two host adapters Host system with two host a...

Страница 123: ...o host systems and two fibre channel switches Host system with two host adapters Host system with two host adapters Switch Switch Interswitch link ds48003 DS4800 Figure 67 Example of a single SAN fabr...

Страница 124: ...can connect the power cords to a primary power unit inside the rack cabinet such as a properly grounded ac distribution unit or to an external source such as a properly grounded electrical outlet The...

Страница 125: ...the device The device also might have more than one power cord To remove all electrical current from the device ensure that all power cords are disconnected from the power source 1 2 1 Verify that the...

Страница 126: ...ords to two independent IBM approved power distribution units in the rack cabinet Connecting each RAID controller to independent power sources maintains power redundancy If applicable repeat this step...

Страница 127: ...subsystem is powered on each RAID controller unit will handle approximately one half of the current load To ensure maximum protection against power loss due to RAID controller unit failure or a power...

Страница 128: ...rtition key will be lost and will need to be regenerated using instructions on the IBM DS4000 Solutions and Premium Features Web site https www 912 ibm com PremiumFeatures Additionally if at least two...

Страница 129: ...storage expansion enclosures for information on power requirements Contact your IBM service representative for additional information if needed v Before any planned system shutdown or after any system...

Страница 130: ...In addition you should also collect event logs for the host servers that have mapped logical drives from the storage subsystem Attention Save these event log files to a server disk that will remain ac...

Страница 131: ...aintenance helps to support the optimal performance of your DS4000 storage configuration You should periodically inspect the Fibre Channel components of your DS4000 storage configuration IBM suggests...

Страница 132: ...es to the overall health of the system Removing and replacing the front bezel The storage subsystem is equipped with a front bezel which you must remove in order to access the power supply and fan uni...

Страница 133: ...nections are optimal by checking the indicator LEDs before you turn on the DS4800 The controllers might not recognize the correct configuration if the storage expansion enclosure hard disk drives are...

Страница 134: ...nd off intermittently Depending on your configuration it can take several minutes for each storage expansion enclosure to power up Check the LEDs on the front and back of all the storage expansion enc...

Страница 135: ...see Checking the LEDs on page 118 c Open the DS4000 Storage Manager Subsystem Management window and display the Physical View for the configuration The status for each component will be either Optimal...

Страница 136: ...xcept in the following situations v Instructions in a hardware or software procedure require you to turn off the power v A service technician tells you to turn off the power v A planned power outage o...

Страница 137: ...require you to perform other procedures before you turn off the power 1 Stop all I O activity to the storage subsystems 2 Determine the status of all storage subsystems and components in the configur...

Страница 138: ...ing on the storage subsystem on page 105 Restoring power after an unexpected shutdown The storage subsystem might shut down unexpectedly under any of the following conditions Attention There might be...

Страница 139: ...you can correct the problem when you turn on the power again 3 Turn off all power supply switches then unplug the power cords from the storage subsystem Restoring power after an emergency shutdown Us...

Страница 140: ...ler The storage subsystem might take several minutes to power up depending on the number of attached storage expansion enclosures The battery self test might take an additional 15 minutes The LEDs wil...

Страница 141: ...etects a subsystem failure that has been detected by the controllers prior to the enabling of the alarm The GUI indicates alarm status for the storage subsystem as follows v Alarm control enabled disa...

Страница 142: ...might include v HBA drivers v Controller firmware version listed in Table 32 v IP addresses for RAID controllers for in band management only v Additional documentation for switches and HBAs if needed...

Страница 143: ...rt of the storage management software installation the DS4000 Storage Manager Enterprise Management window must remain open If you close the window you will not receive any alert notifications from th...

Страница 144: ...DS4000 Storage Manager client gives these procedures Attention Not following the software recovery procedures can result in data loss Firmware updates DS4800 models 80 82 84 and 88 will be shipped wit...

Страница 145: ...ocedure to check the status of the storage subsystem If a problem has occurred use the DS4000 Storage Manager client software and the LEDs on the storage subsystem to help locate a failed component 1...

Страница 146: ...ems For information about printing a storage subsystem profile and saving the support data refer to the Subsystem Management window online help Checking the LEDs The LEDs display the status of the sto...

Страница 147: ...unit LEDs on page 127 Off On Storage subsystem locate Overall DS4800 Configuration Needs Attention Amber LED Off On There are one or more failures in the DS4800 storage subsystem configuration The cau...

Страница 148: ...Green LED See Table 35 on page 123 2 Host Channel Speed L2 Green LED Controller A Controller B ds48041 12 12 7 8 9 9 8 7 3 4 5 6 3 4 5 6 3 4 5 6 3 4 5 6 10 10 11 11 10 10 11 11 1 1 1 1 2 2 2 2 2 2 2...

Страница 149: ...her port Amber LED Off On Bypass problem v An SFP module is inserted in the port and the connected fibre channel is either absent or not properly connected v The storage expansion enclosure connected...

Страница 150: ...enclosure connected to this port is not powered on v There is a problem with the fibre channel connection between this port and the fibre channel port of the connected ESM in the storage expansion en...

Страница 151: ...eric display LEDs The L1 and L2 LEDs for each host and drive channel combine to indicate the status and the operating speed of each host or drive channel Table 35 Host and drive channel LED definition...

Страница 152: ...al point between the display numbers The diagnostic LED tuns off when the numeric display shows the storage subsystem enclosure ID The numeric display shows various diagnostic codes as the controllers...

Страница 153: ...the right and left power supply and fan units are in mirror positions The five LED positions for the left power supply and fan unit are aligned vertically on the right side of the unit The five LED p...

Страница 154: ...he left power supply and fan unit is linked to controller B v The connected PDUs are turned off or have been tripped v There is no power at the wall circuit v The power supply and fan unit is bad Batt...

Страница 155: ...rage subsystem on page 117 Service Action Allowed Blue LED Off On Safe to remove Resetting the controller circuit breakers on page 131 Replacing a power supply and fan unit on page 152 Interconnect ba...

Страница 156: ...configuration has developed a fault Inspect the Needs Attention LEDs on the other components in the DS4800 Storage Subsystem and in all connected DS4000 storage expansion enclosures to isolate the fa...

Страница 157: ...n including all attached storage expansion enclosures The Power Overall DS4800 Configuration Needs Attention and Locate LEDs shine through the front bezel Attention If the Overall DS4800 Configuration...

Страница 158: ...ls to help cool the cabinet air temperature and prevent damage to the storage subsystem Note If both power supply and fan units have shut down start with step 4 below 1 If applicable turn off the alar...

Страница 159: ...o locate the specific failed component c When the procedure is completed select Recheck in the Recovery Guru This will rerun the Recovery Guru to ensure that the problem has been fixed d If the proble...

Страница 160: ...t the circuit breaker Tools and Equipment v Antistatic protection v Nonconducting stylus Electrostatic discharge can damage sensitive components Touching the storage subsystem or its components withou...

Страница 161: ...less than a 2 inch radius Note You do not need to remove the interface cables from the controllers to reset the circuit breakers however IBM recommends that you remove the cables to make this procedu...

Страница 162: ...h the latches to release the levers and rotate the levers to disengage them from the locking pins b Use the levers to slide the controller from the storage subsystem chassis Figure 81 on page 135 show...

Страница 163: ...tive controller slots in the DS4800 the location of the circuit breaker access holes are as follows On controller A the circuit breaker access holes are located on the left side on the bottom of the c...

Страница 164: ...D status proceed to one of the following steps v All channel speed LEDs are on and the Needs Attention LED is off Go to step 16 v All channel speed LEDs are off or the Needs Attention LED is on Verify...

Страница 165: ...orrect FRU part number for the DS4800 controller In addition the DS4800 model 80A H has different replacement FRU part numbers than the DS4800 model 82A H controllers even though they both have the sa...

Страница 166: ...and can explode if not properly used handled or disposed of Do not v Throw or immerse into water v Heat to more than 100 C 212 F v Repair or disassemble Dispose of the battery as required by local ord...

Страница 167: ...e battery is fully charged v The LED flashes when the battery is charging or performing a self test v The LED is off if the battery or the battery charger has failed The interconnect battery unit also...

Страница 168: ...140 IBM System Storage DS4800 Storage Subsystem Installation User s and Maintenance Guide...

Страница 169: ...guration You can perform all replacement procedures in this chapter as hot swap procedures Hot swap refers to the ability to remove and replace a storage subsystem component while the storage subsyste...

Страница 170: ...instructions in the DS4000 Storage Manager Subsystem Management window or refer to the appropriate component replacement instructions in this chapter for the additional diagnostics required in this c...

Страница 171: ...components that need attention For example if both the right power supply and fan unit which is linked to controller A and controller B need attention then neither of the Service Action Allowed statu...

Страница 172: ...mounted to the bottom of the component Figure 85 shows a component lever and the latch that secures the lever The top view shows the lever in the closed and latched position The bottom view shows the...

Страница 173: ...model 80 DS4800 controller into a DS4800 model 80 subsystem will cause it to be held in lock down state v Both power supply and fan units must be connected and powered on with no lighted Needs Attent...

Страница 174: ...oller B Controller A is inserted in the top controller bay controller B is inserted in the bottom controller bay Then apply the controller labels for host channels drive channels power inputs and nume...

Страница 175: ...odules that contain wire tabs unlock the SFP module latch by pulling the wire latch outward 90 as shown in Figure 89 on page 148 Host channels RS 232 serial port Ethernet ports Dual ported drive chann...

Страница 176: ...en disconnect the power cord from the ac power connector on the controller 10 Release the controller levers and remove the failed controller Figure 90 shows the controller sliding from the storage sub...

Страница 177: ...ction end Leave some slack between the clamp and the controller end of the power cord Replace the strain relief nut and tighten it securely to secure the clamp to the controller unit 13 Connect the po...

Страница 178: ...of cache memory per controller v DS4800 Model 84A H Cache upgrade to 16 GB This option upgrades the DS4800 controllers with 4 GB of cache memory per controller to DS4800 controllers with 8 GB of cach...

Страница 179: ...pgrades 6 If the DS4800 host ports are connected to a FC switch use the switch monitor tool to make sure that the DS4800 host ports and the server FC HBA ports are operating optimally No unexpected po...

Страница 180: ...al outlets are working and so on For more information see Restoring power after an unexpected shutdown on page 110 The storage subsystem cooling system consists of two fans in each of the two power su...

Страница 181: ...ct a problem with one of these parts contact a service technician 1 Use the DS4000 Storage Manager client software to print a storage subsystem profile 2 Locate the storage subsystem that contains the...

Страница 182: ...nd might damage the equipment To avoid damage use proper antistatic protection while handling any components 6 Put on antistatic protection 7 Unpack the new power supply and fan unit Save all packing...

Страница 183: ...Verify that the power supply and fan unit does not snag any other DS4800 cables when you slide it into the bay d Press the power supply and fan unit fully into the slot engaging the pins on the enclos...

Страница 184: ...entative In addition do not replace the interconnect battery unit in any of the following situations v The DS4800 controller A is not in optimal state v The FC paths from the hosts to the host ports i...

Страница 185: ...sping the sides and pulling the cover toward you Attention Never remove a power supply and fan unit unless the Service Action Allowed status LED is turned on 4 Is the Service Action Allowed status LED...

Страница 186: ...levers straight out simultaneously to remove the failed interconnect battery unit Figure 96 on page 159 shows the interconnect battery unit sliding from the storage subsystem chassis Service Action A...

Страница 187: ...erconnect battery unit slide the back of the unit into the appropriate slot c Holding both levers slowly push the interconnect battery unit until the notches in both the left and right levers are alig...

Страница 188: ...heck the status of the battery Because write caching is disabled when either one of the backup battery packs fail IBM recommends that you replace the failed battery pack as soon as possible to minimiz...

Страница 189: ...g materials in the event you need to return the new battery pack 7 Remove the battery access cover from the front of the interconnect battery unit by removing the thumb screws that secure the cover to...

Страница 190: ...have the IBM part number listed on the battery available prior to your call For information on disposal of sealed Lithium Ion batteries outside the United States go to http www ibm com ibm environment...

Страница 191: ...er 24 hours check the Needs Attention and Battery LEDs to ensure that the battery is working properly If the replacement battery indicates a fault contact your IBM support representative Replacing an...

Страница 192: ...devices When working with fiber optic cables do not pinch them step on them or locate them in aisles or walkways Do not overtighten the cable straps or bend the cables to a radius smaller than 38 mm...

Страница 193: ...and Needs Attention LED are off Go to step 12 11 Is the problem corrected v Yes Go to step 12 v No Contact your IBM support representative 12 Remove the antistatic protection 13 Use the DS4000 Storage...

Страница 194: ...ported drive channels SFPs are inserted into the ports and then fibre channel cables are connected to the SFPs For information on installing SFPs and fiber optic cables see Working with SFPs and fiber...

Страница 195: ...ion Guide for more information The loopback test and the sendEcho test are described in the IBM System Storage DS4000 Problem Determination Guide In the IBM System Storage SAN Integration Server confi...

Страница 196: ...e channel cables Verify that the fibre channel cables are in good condition and that there are no cable management issues such as using plastic ties to wrap the cables or bending the cable loops to le...

Страница 197: ...plugged in and all power switches are on If applicable check that the main circuit breakers for the rack cabinet are turned on AC power failure Check the main circuit breaker and AC outlet Both power...

Страница 198: ...an unit is not lit Call IBM Support if needed Interconnect battery unit v Amber Battery Needs Attention LED on the interconnect battery unit is on v Green Battery LED on the power supply and fan unit...

Страница 199: ...re to place it online If one controller has been placed offline by the other controller contact IBM support This is the case if you use the DS4000 Storage Manager client software to make repeated atte...

Страница 200: ...All DS4800s are shipped with both cache battery packs installed Call IBM support if a cache battery pack is missing and you did not remove any of the cache battery packs from the DS4800 L1 and L2 Lin...

Страница 201: ...by both the blue Service Action Allowed LED and the amber Needs Attention LED has failed and can be replaced Unable to access drives Expansion drive channel Fibre channel cabling has failed Ensure tha...

Страница 202: ...stem controller as an independent storage subsystem or a partially managed device contact IBM support for instructions on how to force firmware synchronization in the replacement controller The DS4800...

Страница 203: ...te and without batteries 23R0532 4 DS4800 chassis 23R1980 Model 80H 23R1982 Model 80A Non Korea 23R1981 Model 80A Korea 22R4272 Model 82H 22R4491 Model 84H 22R4493 Model 82A Non Korea 22R4494 Model 82...

Страница 204: ...ory Model 88A H 23R0537 6 DS4800 mounting rail assembly 22R4839 7 DS4800 battery cover plate with 4 screws 22R4874 DS4800 battery 22R4875 LC wrap plug assembly 24P0950 LC SC fibre channel cable adapte...

Страница 205: ...cument in the DS4000 Storage Manager Version 9 library with its related common user tasks Table 41 DS4000 Storage Manager Version 9 titles by user tasks Title User tasks Planning Hardware installation...

Страница 206: ...Storage Subsystem library with its related common user tasks Table 42 DS4800 Storage Subsystem document titles by user tasks Title User Tasks Planning Hardware Installation Software Installation Conf...

Страница 207: ...ent titles by user tasks Title User Tasks Planning Hardware Installation Software Installation Configuration Operation and Administration Diagnosis and Maintenance IBM System Storage DS4700 Storage Su...

Страница 208: ...r Tasks Planning Hardware Installation Software Installation Configuration Operation and Administration Diagnosis and Maintenance IBM TotalStorage DS4500 Storage Subsystem Installation User s and Main...

Страница 209: ...asks Title User Tasks Planning Hardware Installation Software Installation Configuration Operation and Administration Diagnosis and Maintenance IBM TotalStorage DS4400 Fibre Channel Storage Server Use...

Страница 210: ...tion and Administration Diagnosis and Maintenance IBM TotalStorage DS4300 Storage Subsystem Installation User s and Maintenance Guide U U U U U IBM TotalStorage DS4300 Rack Mounting Instructions U U I...

Страница 211: ...ystem document titles by user tasks Title User Tasks Planning Hardware Installation Software Installation Configuration Operation and Administration Diagnosis and Maintenance IBM System Storage DS4200...

Страница 212: ...ment titles by user tasks Title User Tasks Planning Hardware Installation Software Installation Configuration Operation and Administration Diagnosis and Maintenance IBM TotalStorage DS4100 Storage Ser...

Страница 213: ...em Storage DS4000 EXP810 Storage Expansion Enclosure Installation User s and Maintenance Guide U U U U U IBM TotalStorage DS4000 EXP700 and EXP710 Storage Expansion Enclosures Installation User s and...

Страница 214: ...U U IBM TotalStorage DS4000 FC2 133 Host Bus Adapter Installation and User s Guide U U IBM TotalStorage DS4000 FC2 133 Dual Port Host Bus Adapter Installation and User s Guide U U IBM Netfinity Fibre...

Страница 215: ...ach RAID controller as shown in Figure 103 Product name IBM System Storage DS4800 Storage Subsystem Machine type 1815 Model number Serial number RAID controller MAC addresses Controller firmware versi...

Страница 216: ...188 IBM System Storage DS4800 Storage Subsystem Installation User s and Maintenance Guide...

Страница 217: ...gure 105 on page 191 to identify the proper locations for inserting M5 screws when mounting the support rails and DS4800 in a rack cabinet The locations for the M5 screws are highlighted in the templa...

Страница 218: ...U 1 U 1 U 0 U 4 U 4 U Rail assembly Front left Rail assembly Front right Rail assembly Front left Rail assembly Front right Figure 104 DS4800 front rack mounting template 190 IBM System Storage DS480...

Страница 219: ...Rack Mounting Template 0 U 2 U 3 U 3 U 2 U 1 U 1 U 0 U 4 U 4 U Rail assembly Rear left Rail assembly Rear right Printed in the U S A Figure 105 DS4800 rear rack mounting template Appendix C Rack moun...

Страница 220: ...192 IBM System Storage DS4800 Storage Subsystem Installation User s and Maintenance Guide...

Страница 221: ...nly in that country or region IBM power cord part number Cord specifics Used in these countries or regions 13F9940 250V 10A 2 8M Australia Fiji Kiribati Nauru New Zealand Papua New Guinea 39Y7917 250V...

Страница 222: ...riya 39Y7920 250V 10A 2 8M Israel 39Y7931 250V 10A 2 8M Antigua and Barbuda Aruba Bahamas Barbados Belize Bermuda Bolivia Brazil Caicos Islands Canada Cayman Islands Costa Rica Columbia Cuba Dominican...

Страница 223: ...e other required host software components Table 51 defines the keyboard operations that enable you to navigate select or activate user interface components The following terms are used in the table v...

Страница 224: ...want to select press Ctrl Spacebar to select the drive If you press Spacebar alone when multiple drives are selected then all selections are removed Use the Ctrl Spacebar combination to deselect a dri...

Страница 225: ...OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE Some jurisdictions do not allow discla...

Страница 226: ...ssor storage real and virtual storage or channel volume KB stands for approximately 1000 bytes MB stands for approximately 1 000 000 bytes and GB stands for approximately 1 000 000 000 bytes When refe...

Страница 227: ...way Appliances are labeled in accordance with European Directive 2002 96 EC concerning waste electrical and electronic equipment WEEE The Directive determines the framework for the return and recyclin...

Страница 228: ...lded and grounded cables and connectors must be used in order to meet FCC emission limits Properly shielded and grounded cables and connectors must be used in order to meet FCC emission limits IBM is...

Страница 229: ...ion of the laws of the Member States relating to electromagnetic compatibility IBM cannot accept responsibility for any failure to satisfy the protection requirements resulting from a nonrecommended m...

Страница 230: ...Japanese Voluntary Control Council for Interference VCCI statement 202 IBM System Storage DS4800 Storage Subsystem Installation User s and Maintenance Guide...

Страница 231: ...ndard for IBM compatibles It extends the XT bus architecture to 16 bits and also allows for bus mastering although only the first 16 MB of main memory are available for direct access agent A server pr...

Страница 232: ...mer replaceable unit customer replaceable unit CRU An assembly or part that a customer can replace in its entirety when any of its components fail Contrast with field replaceable unit FRU cyclic redun...

Страница 233: ...expansion port error correction coding ECC A method for encoding data so that transmission errors can be detected and corrected by examining the data on the receiving end Most ECCs are characterized b...

Страница 234: ...al metaphor of a real world scene often of a desktop by combining high resolution graphics pointing devices menu bars and other menus overlapping windows icons and the object action relationship GUI S...

Страница 235: ...local area network LBA See logical block address local area network LAN A computer network located on a user s premises within a limited geographic area logical block address LBA The address of a log...

Страница 236: ...Network Management Protocol SNMP a station that runs management application programs that monitor and control network elements NMI See non maskable interrupt NMS See network management station non mas...

Страница 237: ...sk RVSD A virtual shared disk on a server node configured to provide continuous access to data and file systems in a cluster redundant array of independent disks RAID A collection of disk drives array...

Страница 238: ...used to convert signals between optical fiber cables and switches An SFP is smaller than a gigabit interface converter GBIC See also gigabit interface converter SNMP See Simple Network Management Prot...

Страница 239: ...t particular TL_port Transmission Control Protocol TCP A communication protocol used in the Internet and in any network that follows the Internet Engineering Task Force IETF standards for internetwork...

Страница 240: ...212 IBM System Storage DS4800 Storage Subsystem Installation User s and Maintenance Guide...

Страница 241: ...em 90 connecting to the DS4800 different ways to connect four storage expansion enclosures 68 controller memory 16 controllers cable connections 11 drive channels 12 Ethernet ports 14 host channels 13...

Страница 242: ...t 201 Chinese 200 European Union EMC Directive conformance statement 201 Industry Canada Class A emission compliance statement 201 FCC Class A notice continued Japanese Voluntary Control Council for I...

Страница 243: ...operating 101 overheated power supply and fan unit recovery 129 parts list 174 power supply and fan unit LEDs 125 IBM System Storage DS4800 continued power supply and fan units 17 preparing for instal...

Страница 244: ...hut down what to do 130 installing 155 LEDs 125 recovery from overheating 129 replacement procedure 153 replacing 152 shutdown recovery 130 power supply and fan units overview 17 R rack cabinet EIA 31...

Страница 245: ...port 3 10 turning on before storage subsystem 105 storage subsystem configuration installing 96 storage subsystem management methods 90 storage subsystem profile 6 101 switch technical support Web sit...

Страница 246: ...218 IBM System Storage DS4800 Storage Subsystem Installation User s and Maintenance Guide...

Страница 247: ...prices please contact your IBM branch office your IBM business partner or your authorized remarketer When you send comments to IBM you grant IBM a nonexclusive right to use or distribute your comment...

Страница 248: ...Development Dept GZW 9000 South Rita Road Tuscon AZ U S A 85744 0001 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _...

Страница 249: ......

Страница 250: ...Part Number 22R6614 Printed in USA GC26 7845 01 1P P N 22R6614...

Отзывы:

Нет отзывов

Бренды по названию

Популярные бренды

Загрузить еще бренды