IBM System storage DS6000 Series Скачать руководство пользователя страница 371 | Manualshive

Страница: 371 / 406

background image

Appendix B. Using the DS6000 with iSeries

347

When you get to the point where you will select the volumes to be added, you will see a panel
similar to that shown in Figure B-27. Multipath volumes appear as DMPxxx. Highlight the
disks you want to add to the disk pool and click Add.

Figure B-27 Adding a multipath volume

The remaining steps are identical to those in “Adding volumes to an Independent Auxiliary
Storage Pool” on page 334.

Note:

For multipath volumes, only one path is shown. In order to see the additional paths,

see “Managing multipath volumes using iSeries Navigator” on page 349.

«
...
369
370
371
372
373
...
»

Содержание System storage DS6000 Series

Страница 1: ...Juan Antonio Garay Torsten Knobloch Donald C Laing Christine O Sullivan Stu S Preacher Torsten Rothenwaldt Tetsuroh Sano Jing Nan Tang Anthony Vandewerdt Alexander Warmuth Roland Wolf Enterprise class...

Страница 2: ......

Страница 3: ...International Technical Support Organization The IBM TotalStorage DS6000 Series Concepts and Architecture March 2005 SG24 6471 00...

Страница 4: ...t an early version of DS6000 microcode was used for the screen captures and command output so some details may vary from the currently available microcode Note Before using this information and the pr...

Страница 5: ...iii...

Страница 6: ...iv DS6000 Series Concepts and Architecture...

Страница 7: ...rted environment 9 1 2 5 Business continuance functions 10 1 2 6 Resiliency 13 1 2 7 Interoperability 13 1 2 8 Service and setup 13 1 2 9 Configuration flexibility 14 1 3 Positioning the IBM TotalStor...

Страница 8: ...ID 10 overview 53 3 3 3 Spare creation 54 3 3 4 Predictive Failure Analysis PFA 55 3 3 5 Disk scrubbing 55 3 3 6 Disk path redundancy 55 3 4 Power subsystem RAS 56 3 5 System service 57 3 5 1 Example...

Страница 9: ...preparation 116 7 2 1 Floor and space requirements 116 7 2 2 Power requirements 118 7 2 3 Environmental requirements 118 7 2 4 Preparing the rack 119 7 3 System management interfaces 119 7 3 1 IBM To...

Страница 10: ...The DS Storage Manager Logical configuration 151 9 1 Configuration hierarchy terminology and concepts 152 9 1 1 Storage configuration terminology 152 9 1 2 Summary of the DS Storage Manager logical co...

Страница 11: ...ON to FICON ports 222 11 2 4 PPRC over Fibre Channel links 222 11 2 5 Fixed LSS to RAID rank affinity and increasing DDM size 222 11 3 How does the DS6000 address the challenge 223 11 3 1 Fibre Channe...

Страница 12: ...d hardware based data migration 255 13 2 3 Hardware and microcode based migration 258 13 3 Data migration based on logical migration 263 13 3 1 Data Set Services Utility DFSMSdss 264 13 3 2 Data migra...

Страница 13: ...305 LVM configuration 308 AIX access methods for I O 308 Boot device support 309 AIX on IBM iSeries 309 Monitoring I O performance 310 Linux 312 Support issues that distinguish Linux from other operat...

Страница 14: ...hannel adapters 355 Size and number of LUNs 355 Recommended number of ranks 355 Sharing ranks between iSeries and other servers 356 Connecting via SAN switches 356 Migration 357 OS 400 mirroring 357 M...

Страница 15: ...in this publication at any time without notice Any references in this information to non IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web s...

Страница 16: ...em 38 Tivoli TotalStorage Proven TotalStorage VSE ESA The following terms are trademarks of other companies Solaris Java and all Java based trademarks and logos are trademarks or registered trademarks...

Страница 17: ...updated on December 13 2005 March 2005 First Edition This revision reflects the addition deletion or modification of new and changed information described below Changed information see change bars Fi...

Страница 18: ...xvi DS6000 Series Concepts and Architecture...

Страница 19: ...m Characteristics of the DS6000 series described in this redbook also include the DS6000 copy functions FlashCopy Metro Mirror Global Copy and Global Mirror The performance features particularly the s...

Страница 20: ...previous zSeries Copy Services redbook Juan Antonio Garay is a Storage Systems Field Technical Sales Specialist in Germany He has five years of experience in supporting and implementing z OS and Open...

Страница 21: ...of IBM China He has nine years of experience in the IT field His main job responsiblity is providing technical support and IBM storage solutions to IBM professionals Business Partners and customers Hi...

Страница 22: ...ons to this project Susan Barrett IBM Austin James Cammarata IBM Chicago Dave Heggen IBM Dallas John Amann Craig Gordon Rosemary McCutchen IBM Gaithersburg Hartmut Bohnacker Michael Eggloff Matthias G...

Страница 23: ...with leading edge technologies You ll team with IBM technical professionals Business Partners and or customers Your efforts will help increase product acceptance and customer satisfaction As a bonus y...

Страница 24: ...xxii DS6000 Series Concepts and Architecture...

Страница 25: ...BM Corp 2004 All rights reserved 1 Part 1 Introduction In this part we introduce the IBM TotalStorage DS6000 series and its key features The topics covered include Product overview Positioning Perform...

Страница 26: ...2 DS6000 Series Concepts and Architecture...

Страница 27: ...ter provides an overview of the features functions and benefits of the IBM TotalStorage DS6000 series of storage servers The topics covered include Overview of the DS6000 series and its benefits Posit...

Страница 28: ...investments Storage asset consolidation can be greatly assisted by virtualization Virtualization software solutions are designed to logically combine separate physical storage systems into a single vi...

Страница 29: ...Models 800 and 750 as well as with the new DS8000 series The DS6000 series offers an entirely new era in price performance and scalability Now for the first time zSeries and iSeries customers have th...

Страница 30: ...s for the storage server and the host adapters respectively and another PowerPC 750FX 500 MHz processor for the device adapter on each controller card The DS6800 is equipped with 2 GB memory in each c...

Страница 31: ...AL interface Instead of forming an FC AL loop each disk drive is connected to two Fibre Channel switches within each enclosure With this switching technology there is a point to point connection to e...

Страница 32: ...rding to your performance needs you can select from four different disk drive types fast 73 GB and 146 GB drives rotating at 15 000 RPM good performing and cost efficient 146 GB drives operating at 10...

Страница 33: ...ou to create and save a task using the GUI The following list highlights a few of the specific types of functions that you can perform with the DS CLI Check and verify your storage unit configuration...

Страница 34: ...a persistent FlashCopy The benefits of FlashCopy are that the point in time copy is immediately available for use for backups and the source volume is immediately released so that applications can be...

Страница 35: ...S6800 to hold off I O activity to a LUN or volume until all LUNs volumes within the group have established a FlashCopy relationship to their targets and the FlashCopy Consistency Group Created command...

Страница 36: ...nplanned outage The actual lag in data currency experienced will depend upon a number of factors including specific workload characteristics and bandwidth between the local and remote sites Efficient...

Страница 37: ...D 10 arrays will require that two global spares are defined in the DS6800 controller enclosure If you have expansion enclosures the first enclosure will have another two global spares More spares coul...

Страница 38: ...call home process Configuration changes like adding disk drives or expansion enclosures are a non disruptive process Most maintenance actions are non disruptive including downloading and activating ne...

Страница 39: ...ividual Host Bus Adapters HBAs and volumes but instead with groups Summary In summary the DS6000 series allows for Up to 32 logical subsystems Up to 8192 logical volumes Up to 1040 volume groups Up to...

Страница 40: ...gh end or midrange systems without the need for the storage administrators to learn about a new product This will reduce your management costs and the total cost of ownership 1 3 2 Common management f...

Страница 41: ...but their functions differ With respect to performance the DS4000 series range is below the DS6000 series You have the option to choose from different DS4000 models among them very low cost entry mode...

Страница 42: ...servers It is designed as a highly scalable solution supporting both very large files and very large numbers of files without the limitations normally associated with Network File System NFS or Commo...

Страница 43: ...rformance features available on the DS6000 series Parallel Access Volumes PAV PAV is an optional feature for zSeries environments which enables a single zSeries server to simultaneously process multip...

Страница 44: ...20 DS6000 Series Concepts and Architecture...

Страница 45: ...1 Part 2 Architecture In this part we describe various aspects of the DS6000 series architecture These include Hardware components RAS reliability availability and serviceability Virtualization concep...

Страница 46: ...22 DS6000 Series Concepts and Architecture...

Страница 47: ...6000 Here you can read about the DS6000 hardware platform and its components Server enclosure Expansion enclosure Controller architecture Disk subsystem Server enclosure RAID controller card Expansion...

Страница 48: ...is the front display panel that provides status indicators You can also see the disk drive modules or DDMs Each enclosure can hold up to 16 DDMs Figure 2 2 DS6800 rear view The rear view of the DS680...

Страница 49: ...that connects these components is pictured in Figure 2 4 on page 26 Effectively the DS6800 consists of two controller cards Each controller card contains an integrated four port host adapter to connec...

Страница 50: ...n case it is required again The controllers attempt to anticipate future reads by an algorithm known as SARC sequential prefetching in adaptive replacement cache Data is held in cache as long as possi...

Страница 51: ...he main memory 2 3 2 Cache management Most if not all high end disk systems have internal cache integrated into the system design and some amount of system cache is required for operation Over time ca...

Страница 52: ...running concurrently To decide which pages are evicted when the cache is full sequential and random non sequential data is separated into different lists see Figure 2 5 A page which has been brought i...

Страница 53: ...sure can contain 16 DDMs or dummy carriers A dummy carrier looks very similar to a DDM in appearance but contains no electronics As discussed earlier from the front the server enclosure and the expans...

Страница 54: ...ion is different The key features of switched FC AL technology are Standard FC AL communication protocol from DA to DDMs Direct point to point links are established between DA and DDM Isolation capabi...

Страница 55: ...on page 32 It depicts how the DS6000 is divided into two disk loops The server enclosure which contains the first 16 DDMs is on loop 0 The first expansion enclosure is placed on loop 1 This allows fo...

Страница 56: ...uration discussed in Chapter 10 DS CLI on page 195 you have the choice of creating a RAID 5 or RAID 10 array by choosing one or two array sites The first and the third array sites created on each loop...

Страница 57: ...t of an unexpected power failure If the DS6800 were to power off with un destaged writes in NVS then after reboot the controller would read this reserved area and destage the writes For more details o...

Страница 58: ...de cables respectively all with LC connectors The ports in each controller are effectively on a PCI X 64 Bit 133 MHz card the same card used in the DS8000 The chipset is driven by a new high function...

Страница 59: ...dicator is green and indicates that the controller card is powered on The lower indicator is amber and indicates that this controller requires service 2 6 Expansion enclosure SBOD controller card The...

Страница 60: ...is cabled to the lower SBOD card In each case cables run from the disk exp ports to the in ports of the SBOD card A second expansion enclosure has been added by running cables from the out ports on th...

Страница 61: ...ng to add more expansion enclosures to that loop Figure 2 15 Expansion enclosure cabling Disk Contrl ports on loop 1 2 7 Front panel The DS6000 front operator panel allows you to perform a health chec...

Страница 62: ...System Information amber Circled letter i This indicator is normally off If it is on solid then an error has occurred that cannot be fixed by light path diagnostics To turn this light off you need to...

Страница 63: ...llow a user to attach the enclosure to eServer rack identifier hardware This allows you to identify in which rack a particular DS6000 storage or expansion enclosure is located Power control switch Dat...

Страница 64: ...Each power supply has two integrated fans for cooling the entire enclosure If a power supply fails it should not be removed from the enclosure until a replacement one is available Both power supplies...

Страница 65: ...ht hand BBU supports the lower controller Figure 2 19 DS6000 battery backup unit indicators The DS6000 battery units have three indicators detailed in Table 2 4 Table 2 4 DS6800 battery backup unit in...

Страница 66: ...controllers to set IP addresses After this it is not used You do not use it to connect the two controllers together for normal operation or during initial power on this connectivity is supplied by an...

Страница 67: ...43 2 13 Summary This chapter has described the various components that make up a DS6000 For additional information there is documentation available on the Web at http www 1 ibm com servers storage sup...

Страница 68: ...44 DS6000 Series Concepts and Architecture...

Страница 69: ...chapter describes the RAS reliability availability and serviceability characteristics of the DS6000 series Specific topics covered are Controller RAS Host connection availability Disk subsystem RAS Po...

Страница 70: ...s which for open systems are called LUNs and for zSeries 3390s These logical volumes belong to a logical subsystem LSS For open systems the LSS membership is not that important unless you are using Co...

Страница 71: ...ers are actively processing I O requests This section describes the failover and failback procedures that occur between the DS6800 controllers when an abnormal condition has affected one of them Failo...

Страница 72: ...with RAID 10 storage certain RAID related recoveries may cause latency impacts in excess of 15 seconds If you have real time response requirements in this area contact IBM to determine the latest info...

Страница 73: ...nts to validate the integrity of the data as it moves throughout the disk system It is also checked by the DS6800 before the data is sent to the host in response to a read I O request Further the meta...

Страница 74: ...DS6800 each using multiple paths the eight host adapter ports that are available in the DS6800 may not be sufficient to accommodate all the connections The solution to this problem is the use of SAN...

Страница 75: ...00 or DS8000 A new version of SDD Version 1 6 will also allow SDD to manage pathing to the DS6800 and DS8000 SDD provides availability through automatic I O path failover If a failure occurs in the da...

Страница 76: ...ystem images with the appropriate level of software support will respond to such requests by varying off the affected paths and either notifying the DS6800 subsystem that the paths are offline or that...

Страница 77: ...the failed drive require data to be read from the other drives in the array to reconstruct the data The remaining requests are satisfied by reading the drive containing the data in the normal way Perf...

Страница 78: ...his normally means that two spares will be created in the server enclosure and two spares in the first expansion enclosure Spares are created as the array sites are created which occurs when the DDMs...

Страница 79: ...alert the user and can also use call home e mail notification 3 3 5 Disk scrubbing The DS6000 will periodically read all sectors on a disk This is designed to occur without any interference to applica...

Страница 80: ...nt of a BBU failure the RAID controller that relies on that BBU for data protection will remove itself from service and go offline until its BBU is fully charged If both BBUs were to fail then the ent...

Страница 81: ...e DS6000 and using the simple replacement instructions detailed there they remove and replace the failed DDM with a new one After replacing the DDM the System Alert Indicator will be turned off automa...

Страница 82: ...t removal instructions Figure 3 7 on page 59 shows an example of the screen the user may see On this screen users are given the ability to do things like a View an animation of the removal and replace...

Страница 83: ...alls the replacement power supply 3 Finally the user checks the component view to review system health after the repair An example of this is shown in Figure 3 8 In this example we can see that all th...

Страница 84: ...emind button is pressed the Alert indicator will go from solid to a 2 second blip short blip on for 250 msec every 2 seconds The System Alert Indicator and the System Information Indicator can be on a...

Страница 85: ...arts there are currently two tiers Tier 1 CRUs are relatively easy to replace while Tier 2 CRUs are generally more expensive parts or parts that require more skill to replace Tier 1 CRU parts Battery...

Страница 86: ...g the code level currently available at http www 1 ibm com servers storage support disk index html The currently available version of code can also be downloaded from this Web site Users can also regi...

Страница 87: ...he alternative to load code non concurrently This means that both controllers are unavailable for a short period of time This method can be performed in a smaller window of time 3 7 Summary This chapt...

Страница 88: ...64 DS6000 Series Concepts and Architecture...

Страница 89: ...ncepts This chapter describes the virtualization concepts for the DS6000 and the abstraction layers for disk virtualization The topics covered are Array sites Arrays Ranks Extent pools Logical volumes...

Страница 90: ...e set is a group of 4 drives of the same capacity and RPM The disk drives can be accessed by a pair of device adapters Each device adapter has four paths to the disk drives The four paths provide two...

Страница 91: ...So it is not really a loop but a switched FC AL loop with the FC AL addressing schema Arbitrated Loop Physical Addressing AL PA 4 2 1 Array sites An array site is a group of four DDMs What DDMs make u...

Страница 92: ...up to two spares may be taken from the array sites used to construct the array on each device interface loop See Chapter 5 IBM TotalStorage DS6000 model overview on page 83 for more details Figure 4...

Страница 93: ...Figure 4 4 on page 70 and indicated by the small squares in Figure 4 5 on page 71 The process of forming a rank does two things The array is defined for either fixed block open systems or CKD zSeries...

Страница 94: ...he squares in the rank just indicate that the extent is composed of several blocks from different DDMs Figure 4 4 Forming an FB rank with 1 GB extents 4 2 4 Extent pools An extent pool is a logical co...

Страница 95: ...gn it to one server and define a CKD extent pool and assign it to the other server Additional extent pools may be desirable to segregate ranks with different DDM types Ranks are organized in two rank...

Страница 96: ...ume is composed of one or more extents from one CKD extent pool CKD extents are of the size of 3390 Model 1 which has 1113 cylinders However when you define a zSeries CKD volume you do not specify the...

Страница 97: ...cation process for FB volumes is very similar and is shown in Figure 4 7 on page 74 1113 1113 1113 1113 free Extent Pool CKD0 Rank x Rank y 3390 Mod 3 1113 free 1113 free used used Allocate 3226 cylin...

Страница 98: ...the iSeries LUNs are in terms of the 512 byte block capacity and are expressed in GB 109 These capacities should be converted to GB 230 when considering effective utilization of extents which are 1 G...

Страница 99: ...nd device adapters and associated ranks The association of an 8 pack to a device adapter determined what LSS numbers could be chosen for a volume On an ESS up to 16 LSSs could be defined depending on...

Страница 100: ...of pairs to preserve data consistency across all pairs in case you have a problem with one of the pairs is done at the LSS level With the option now to put all or most of the volumes of a certain app...

Страница 101: ...the address group X g and the LSS number within the address group X a and the position of the LUN within the LSS X bb For example LUN X 1101 denotes the second X 01 LUN in LSS X 11 of address group 1...

Страница 102: ...s how the logical volume number is converted to a host addressable LUN_ID on the Fibre Channel SCSI interface A map volume group type is used in conjunction with FC SCSI host types that poll for LUNs...

Страница 103: ...B or CKD data Next the extents were added to an extent pool which determined which storage server would serve the ranks and aggregated the extents of all ranks in the extent pool for subsequent alloca...

Страница 104: ...tribute your data for optimal performance you should make sure that you spread it across the two servers and across the two loops and across several ranks If you use some kind of a logical volume mana...

Страница 105: ...sed Volumes for a particular application can be kept in a single LSS Smaller LSSs can be defined if required for systems applications requiring less storage Test systems can have their own LSSs with f...

Страница 106: ...82 DS6000 Series Concepts and Architecture Dynamically add remove volumes Virtualization reduces storage management requirements...

Страница 107: ...iew of the IBM TotalStorage DS6000 storage server which is from here on referred to as the DS6000 While the DS6000 is physically small it is a highly scalable and powerfully performing storage server...

Страница 108: ...er DDM As of this writing the maximum storage capacity with 300 GB DDMs is 4 8 TB with 16 DDMs in a 1750 511 model It measures 5 25 inches high and is available in a 19 inch rack mountable package Fig...

Страница 109: ...r an up to date and complete interoperability matrix refer to http www ibm com servers storage disk ds6000 interop html Figure 5 2 DS6800 Model 1750 511 rear view 5 1 2 DS6000 Model 1750 EX1 To config...

Страница 110: ...displays the rear view of the expansion enclosure which is a bit different compared to the rear view of the 1750 511 model Figure 5 4 shows a 1750 511 model with two expansion 1750 EX1 models Figure...

Страница 111: ...ne group can have a server enclosure and up to three expansion enclosures we call this group Loop 0 and the other group can have up to four expansion enclosures we call this group Loop 1 You can attac...

Страница 112: ...res with Fibre Channel cables 1 16 FC switch FC switch 1 16 FC switch FC switch 16 1 FC switch FC switch 16 1 FC switch FC switch Controller 0 16 1 FC switch FC switch 1 16 FC switch FC switch Control...

Страница 113: ...es run on the DS6000 server enclosure and they support open systems and zSeries environments Copy Services has four interfaces a Web based interface DS Storage Manager a command line interface DS CLI...

Страница 114: ...n detail in the next section You can manage the Copy Services functions through a command line interface DS CLI and a new Web based interface DS Storage Manager You also can manage the Copy Services f...

Страница 115: ...the target volume it is read from the target volume If the backup data is not copied yet it is read from the source volume Write to the source volume When you write some data to the source volume at...

Страница 116: ...me depending on the application It can be used for online backup testing of new applications or for creating a database for data mining purposes The copy looks exactly like the original source volume...

Страница 117: ...y the changed tracks to the target volumes If some tracks on the target volumes are updated these tracks are overwritten by the corresponding tracks from the source volume If you want you can also iss...

Страница 118: ...or extent can be FlashCopied to up to 12 target volumes or target extents as illustrated in Figure 6 4 Figure 6 4 Multiple Relationship FlashCopy Note If a FlashCopy source volume has more than one ta...

Страница 119: ...host I O level not the application level subsequent write operations to the source volumes that are part of the Consistency Group During the temporary queueing Establish FlashCopy is completed The te...

Страница 120: ...sistent copies The copies have power fail or crash level consistency This means that if you suddenly power off your server without stopping your applications and without destaging the data in the file...

Страница 121: ...e this feature for data backup and disaster recovery Remote Mirror and Copy is an optional function To use it you must purchase the Remote Mirror and Copy 2244 function authorization model which is 22...

Страница 122: ...Therefore the copy is normally fuzzy but you can make a consistent copy through synchronization called a go to sync operation After the synchronization you can issue FlashCopy at the secondary site t...

Страница 123: ...you to choose your remote site location based on business needs and enables site separation to add protection from localized disasters A consistent and restartable copy of the data at the remote site...

Страница 124: ...to reduce the time that is required to switch back to the local site after a planned or unplanned outage Figure 6 9 Global Mirror How Global Mirror works We explain how Global Mirror works in Figure...

Страница 125: ...the C volumes The C volumes now contain the consistent copy of data Because the B volumes usually contain a fuzzy copy of the data from the local site not when doing the FlashCopy the C volumes are u...

Страница 126: ...the B volumes the last consistent data in the B volume is moved to the C volumes Therefore at some time a part of consistent data is in the B volume and the other part of consistent data is in the C v...

Страница 127: ...tro Mirror Synchronous PPRC Description Metro Mirror is a function for synchronous data copy at a distance Advantages There is no data loss and it allows for rapid recovery for distances up to 300 km...

Страница 128: ...ions before assigning PPRC secondary volumes For example in an AIX environment AIX assigns specific IDs to each volume PVID PPRC secondary volumes have the same PVID as PPRC primary volumes AIX cannot...

Страница 129: ...ration involving a log volume and a data volume 1 Write to log volume Data Record 2 is being updated 2 Update Data Record 2 on data volume 3 Write to log volume Data Record 2 update complete If the co...

Страница 130: ...I O activity to a volume for a time period by putting the source volume into an extended long busy state This operation can be done across multiple LUNs or volumes and even across multiple storage uni...

Страница 131: ...also not completed because the 2nd operation is not completed In this case the 1st update is included in the backup copy and the 2nd and 3rd updates are not included Therefore this case is consistent...

Страница 132: ...y Service functions can be initiated over the following interfaces zSeries Host I O Interface DS Storage Manager Web based Interface DS Command Line Interface DS CLI DS open application programming in...

Страница 133: ...r in your computers DS Storage Manager issues its commands to the Management Console via the Ethernet network The DS Storage Manager can be used for almost all functions for Copy Services The followin...

Страница 134: ...e DS Open API through the IBM TotalStorage Common Information Model CIM agent a middleware application that provides a CIM compliant interface The DS Open API uses the CIM technology to manage proprie...

Страница 135: ...0 DS8000 and ESS 800 at the same time The DS Storage Manager does not support the ESS 800 6 5 Future Plan According to the announcement letter IBM has issued a Statement of General Direction IBM inten...

Страница 136: ...112 DS6000 Series Concepts and Architecture...

Страница 137: ...ning and configuration In this part we present an overview of the planning and configuration necessary before installing your DS6000 The topics include Installation planning Configuration planning Log...

Страница 138: ...114 DS6000 Series Concepts and Architecture...

Страница 139: ...ew DS6000 in your environment Refer to the latest version of the IBM TotalStorage DS6000 Introduction and Planning Guide GC26 7679 for further details In this chapter we cover the following topics Gen...

Страница 140: ...ordered will be assembled into an IBM TotalStorage 2101 200 System Rack by IBM manufacturing The feature code 0801 is used to indicate that the DS6000 series ordered will be shipped as an assembled e...

Страница 141: ...ck Table 7 2 DS6800 dimensions and weight within a 2101 200 rack 1 Find out the floor load rating of the location where you plan to install the storage units 2 Determine whether the floor load rating...

Страница 142: ...ary line cords to any storage or expansion enclosure in a DS6000 series The DS6000 series power inputs are single phase Table 7 4 DS6000 series input voltage requirements Two independent power outlets...

Страница 143: ...ack do not overload the power outlets Always connect the storage server to a properly grounded outlet Always connect the rack power to at least two different power circuits or sources This reduces the...

Страница 144: ...ation management through customer written applications Either way the DS Open API presents another option for managing storage units by complementing the use of the IBM TotalStorage DS Storage Manager...

Страница 145: ...where you run the DS Storage Manager Gateway Provide the dotted decimal or symbolic name address of the gateway for example 9 123 123 123 or sanjosegate Subnet mask Provide the dotted decimal address...

Страница 146: ...ady have a SAN infrastructure with enough unused ports to connect a DS6000 or you might need additional Fibre Channel switches If you have switches that operate at 1 Gbps you might consider replacing...

Страница 147: ...cards you should also configure a FICON port on the second controller card to allow failover from one server to the other to work A FICON port on the DS6000 can be connected directly to a zSeries hos...

Страница 148: ...IBM authorization acquired through the DS6800 feature numbers for example feature code 5001 OEL 5 TB unit must cover the physical capacity of the DS6800 system where system is defined as the base encl...

Страница 149: ...planning This chapter discusses configuration planning considerations when implementing the DS6000 series in your environment The topics covered are Configuration planning DS6000 Management Console DS...

Страница 150: ...e Mirror and Copy and Parallel Access Volumes See 8 3 DS6000 licensed functions on page 131 for a in depth discussion of the licensed functions Delivery requirements to receive delivery of the DS6000...

Страница 151: ...connects to the appropriate DS6000 series Online configuration which provides real time configuration management support The ability to execute Copy Services functions The capability for configuratio...

Страница 152: ...configuration actions on existing storage plexes and storage units at the time each action is initiated View and change communication settings for DS6000s Express configuration Express configuration p...

Страница 153: ...a service contract with IBM The customer does problem determination replacement parts ordering and the actual replacement of the component Most components are customer replaceable units CRUs In the ev...

Страница 154: ...e in Copy Services environment The DS CLI will communicate with the DS Management Console when performing copy services commands so the network connecting the management console to the DS6000 should b...

Страница 155: ...nce 8 2 7 Simple Network Management Protocol SNMP The storage unit generates SNMP traps and supports a read only management information base MIB to allow monitoring by your network You have to decide...

Страница 156: ...torage feature activation on page 137 Table 8 1 provides the feature codes for the operating environment licenses for the DS6000 series Table 8 1 Operating environment license feature codes Licensed f...

Страница 157: ...n you order Point in Time Copy functions you specify the feature code that represents the physical capacity to authorize for the function Table 8 3 provides the feature codes for the Point in Time Cop...

Страница 158: ...ysical capacity of the entire DS6000 series including the DS6800 and DS6000 expansion units For example if the total capacity of the DS6000 series is 30 TB then you will purchase an OEL feature for 30...

Страница 159: ...scope from FB to CKD or CKD to FB lateral change or reduction in license scope from ALL to FB or CKD will be disruptive and will require an IML In this example the user decides to FlashCopy 10 TB of...

Страница 160: ...Metro Mirror feature for both the primary DS6000 and secondary DS6000 user data plus FC data upto 20 TB FB DS6000 with 45 TB disk capacity in total 45 TB FlashCopy authorization 45 TB of OEL Licensed...

Страница 161: ...responsibility The client can manage and activate functions through the IBM Disk Storage Feature Activation DSFA Web site at http www ibm com storage dsfa Management refers to the use of the IBM Disk...

Страница 162: ...l 1750 511 you can install up to 16 DDMs in the server enclosure The minimum storage capability with 8 DDMs is 584 GB 73 GB x 8 DDMs The maximum storage capability with 16 DDMs for the DS6800 model is...

Страница 163: ...ce adapter chipset inside controller 1 Expansion enclosure 3 Additional enclosures to a maximum of four per loop 1 16 1 16 1 16 Additional enclosures to a maximum of four per loop Expansion enclosure...

Страница 164: ...the allocation unit is an extent of this size RAID type RAID type DDM capacity Array Format Extents Binary GB Decimal GB 3390 3 devices 3390 9 devices 32760 cyl devices 65520 cyl devices RAID10 73GB R...

Страница 165: ...drives and the DS6000 will automatically assign four spare disks two spare disks for each loop After that if you add more expansion enclosures you can configure RAID arrays and the DS6000 will not ass...

Страница 166: ...osure has two spare disks Expansion enclosures 2 and 3 don t have spare disks Figure 8 10 Sparing example 1 RAID 5 Sparing Example 1 RAID 5 All same capacity same rpm Assumes all devices same capacity...

Страница 167: ...t installation you also configure four spare disks in expansion enclosures 2 and 3 A 300 GB DDM cannot be spared onto a 146 GB DDM therefore 300 GB arrays need 300 GB spare disks for RAID configuratio...

Страница 168: ...pare disks in the array And even if the installation sequence is reversed 146 GB DDMs are installed into the first two enclosures and 73 GB DDMs are installed into the next two expansion enclosures yo...

Страница 169: ...uestions you will be in a position to choose the appropriate tools and utilities such as standard operating system mirroring basic commands software packages remote copy technologies and migration app...

Страница 170: ...ata migration backup or restore packages and database tools to migrate data An example of a data migration package is BRMS Backup Recovery and Media Services Tivoli Storage Manager TSM can also be use...

Страница 171: ...mooth risk averse data migration to your DS6000 The advantages of these migration appliances are Facilitated by custom migration tools Minimal customer involvement by IT staff except planning Minimal...

Страница 172: ...only Monitoring performance Hot spot avoidance Preferred paths 8 6 1 Disk Magic An IBM representative or an IBM Business Partner may model your workload using Disk Magic before migrating to the DS6000...

Страница 173: ...dle PAV devices are assigned to base devices that need more PAV devices to manage the workload WLM manages the PAV devices on an LSS group level Plan for the number of alias devices that your configur...

Страница 174: ...nment to a server or controller card In other words preferred paths to a server avoid having to cross the PCI X connection Therefore there is a performance penalty if data from a logical volume manage...

Страница 175: ...ager Logical configuration In this chapter the following topics are discussed Configuration hierarchy terminology and concepts Summary of the DS Storage Manager logical configuration steps Introducing...

Страница 176: ...upported by a single Management Console Storage unit A storage unit also known as a storage facility is a single physical storage subsystem DS6000 Host attachment A host attachment is one or more host...

Страница 177: ...e can access the same volume group The safest approach to this concept is to configure one host per volume group If shared access to the LUN is required for example for dual pathing or clustering then...

Страница 178: ...ent pool must be of the same data format CKD or FB Each extent pool is associated with server 0 or server 1 Although it is possible to create extent pools with ranks of different drive capacities spee...

Страница 179: ...eir server affinity at this time If you are using the custom configuration we recommend for user manageability reasons that the client associate the rank even numbers to server 0 and the rank odd numb...

Страница 180: ...For CKD the minimum size is a 3390 Mod1 For FB the minimum size is 1 GB The maximum volume or LUN size is equal to the size of the extent pool with the following limitation 56 GB for CKD with the appr...

Страница 181: ...volume group not two or more Assigning host attachments from multiple hosts systems even running different operating system types is allowed in the same volume group Figure 9 3 Diagram of the relatio...

Страница 182: ...e group The maximum number of volume groups for the DS6000 is 1040 Address groups An address group is a group of FB or CKD LSSs An address group has up to 16 LSSs The DS6000 supports two address group...

Страница 183: ...rouping of volumes LUNs for Copy Services and various other purposes Some of these purposes are explained here along with general considerations about LSSs in the DS6000 as follows The LSS or LCU dete...

Страница 184: ...256 logical volumes 4096 logical volumes or addresses This means that if you had two address groups defined in the DS6000 you would have 8192 2 X 4096 8192 addresses or logical volumes As an example W...

Страница 185: ...data format type 3 Determine the number and the size of logical volumes needed to fulfill the capacity requirements for zSeries and Open System hosts 4 Determine the number of rank types FB or CKD and...

Страница 186: ...AID format is a 6 P S If you add up the parity chunks it equals one disk s worth of capacity If you choose RAID 10 then you would have two spares with no parity and a 3 X 3 2 spares configuration This...

Страница 187: ...0 To connect to the DS6000 through the browser enter the URL of a PC or the MC you may have purchased The URL consists of the TCP IP address as shown in Figure 9 10 on page 164 or a fully qualified na...

Страница 188: ...the selected fully qualified name for example MC_0 for the default MC as shown in Figure 9 11 Then bookmark it for ease of use When assigning the number we recommend that you make the last field of t...

Страница 189: ...pre configure new configurations or modify existing configurations to be executed at a later time For example the Simulated Manager could be used to execute or modify changes at an off peak hour Real...

Страница 190: ...use of PAVs from the DSFA Web site Simulated manager configuration You can begin using the simulated mode immediately after logging on to the DS Storage Manager However if you want to make your confi...

Страница 191: ...he following users Novice users with little knowledge of storage concepts who want to quickly and easily set up and begin using storage Expert users who want to quickly configure a storage complex by...

Страница 192: ...ions Passwords are limited to a total of 16 characters The user name can not be part of the password This entry will appear as asterisks You can grant user access privileges from this screen as well F...

Страница 193: ...rmation center 9 2 3 Navigating the GUI Knowing what icons radio buttons and check boxes to click in the GUI will help you efficiently navigate your way through the configurator and successfully confi...

Страница 194: ...se icon 4 to access the Info Center You get a help menu screen that prompts you for input on help topics 5 Click icon 5 to close the panels that open when you click the configuration menus in the My W...

Страница 195: ...t all 3 Show filter row 4 Clear all filters 5 Edit sort 6 Clear all sorts The caret called out as item 7 is a simple ascending descending sort on a single column Clicking the pull down menu identified...

Страница 196: ...configure your storage environment in the following order This does not mean that you have to follow this guide exactly You can get the same results by following a different order so long as you defin...

Страница 197: ...n in Figure 9 24 Click Next and Finish in the verification step 9 3 2 Configuring the storage unit To create the storage unit expand the Manage Hardware section click Storage units 2 click Create from...

Страница 198: ...k the Machine Type Model from the pull down list 2 Fill in the Nickname 3 Type in the Description 4 Click the Select Storage Complex from the pull down and choose the storage complex on which you wish...

Страница 199: ...path as shown in Figure 9 27 Figure 9 27 View of the Defined licensed function panel Fill in the fields shown in Figure 9 27 as follows The number of licensed TB for the Operation environment The quan...

Страница 200: ...r the appropriate information and click OK 9 3 3 Configuring the logical host systems To create a logical host for the storage unit that you just created click Host Systems as shown in Figure 9 29 You...

Страница 201: ...ge Complex action pull down and highlight the storage complex on which you wish to configure click Create and Go The screen will advance to the General host information panel shown in Figure 9 31 Figu...

Страница 202: ...host ports panel will update the new information as shown in Figure 9 33 Figure 9 33 View of Define host ports panel with updated host information Note Selecting Group ports to share a common set of...

Страница 203: ...will advance to the Select storage units panel shown in Figure 9 34 Figure 9 34 View of the Select storage unit panel Highlight the Available storage units that you wish click Add and Next The screen...

Страница 204: ...lid storage unit I O ports under the This host attachment can login to field 3 Click Apply assignment and OK 4 Verify and click Finish 9 3 4 Creating arrays from array sites Under Configure Storage cl...

Страница 205: ...e an 8 disk array that two 4 disk array sites will be placed together to form an eight disk array Physical disks from any array site could be placed through a predetermined algorithm into the array It...

Страница 206: ...e on either storage type FB or CKD as shown in Figure 9 38 Figure 9 38 View of the Add arrays to ranks panel with FB selected Click Next to bring up the Verification panel then click Finish to configu...

Страница 207: ...ew of creating custom arrays from four disk array sites At this point you can select from the list of four disk array sites to put together to make an eight disk array If you click Next the second arr...

Страница 208: ...ting extent pools To create extent pools expand the Configure Storage section click Extent pools click Create from the Select Action pull down and click Go Follow the panel directions with each advanc...

Страница 209: ...w of Define properties panel Click Next and Finish 9 3 6 Creating FB volumes from extents Under Simulated Manager expand the Open systems section and click Volumes Click Create from the Select Action...

Страница 210: ...panel To determine the quantity and size of the volumes use the calculators to determine the max size versus quantity as shown in Figure 9 44 Figure 9 44 The Define volume properties panel It is here...

Страница 211: ...s of creating the volumes 9 3 7 Creating volume groups Under Simulated Manager Open Systems perform the following steps to configure the volume groups 1 Click Volume Groups 2 Click Create 3 Click Go F...

Страница 212: ...Architecture Figure 9 46 The Define volume group properties filled out Select the host attachment you wish to associate the volume group with See Figure 9 47 Figure 9 47 The Select host Attachments pa...

Страница 213: ...ck the check box next to the desired volume group and click the Apply button 6 Click OK You can verify that the volume is now assigned to the desired host s volume group by performing the following st...

Страница 214: ...anager zSeries perform the following steps 1 Click Volumes zSeries 2 Click the Select Action pull down and highlight Create 3 Click Go 4 In the Select Extent pool panel click the radio button next to...

Страница 215: ...ck Next 13 Under the Verification panel click Finish 9 3 12 Using the Express Configuration Wizard After you have created and defined the storage complex unit arrays ranks and extent pools you can use...

Страница 216: ...Simulated manager as shown in Figure 9 50 Figure 9 50 View of the Real time Manager panel 2 Click Storage units 3 Select the check box beside the storage unit name as shown in Figure 9 51 Figure 9 51...

Страница 217: ...is displayed as shown in Figure 9 52 9 4 Summary In this chapter we have discussed the configuration hierarchy terminology and concepts We have recommended an order and methodology for configuring th...

Страница 218: ...194 DS6000 Series Concepts and Architecture...

Страница 219: ...an be used to configure and maintain the DS6000 and DS8000 series It also describes how it can be used to manage Copy Services relationships In this chapter we describe Functionality Supported environ...

Страница 220: ...establish dynamic FlashCopy and remote copy relationships on ESS 800 storage servers without using saved tasks Prior to the DS CLI the ESS Copy Services CLI generally did not allow a script to directl...

Страница 221: ...torage disk ds6000 interop htm or http www ibm com servers storage disk ds8000 interop htm 10 4 Installation methods The DS CLI is supplied and installed via a CD that ships with the machine The insta...

Страница 222: ...n one cluster of the ESS Interaction with this server is via either a Web Copy Services GUI interface or via a CLI interface To use the CLI the open systems host needs to have ESS Copy Services CLI so...

Страница 223: ...ardware Management Console S HMC The S HMC then interprets the command and issues it to the relevant server in the relevant DS8000 using the redundant internal network that connects the S HMCs to the...

Страница 224: ...ork Clearly a major benefit of this setup is that the internal network within the DS8000 has no single points of failure By using a second S HMC it is possible to create a completely redundant communi...

Страница 225: ...SS CS CLI is required then both the DS CLI and the ESS CLI will have to be installed on the same open systems host as shown in Figure 10 4 on page 202 Each CLI installs into a separate directory Depen...

Страница 226: ...command flow Open systems host Cluster 1 Infoserver Cluster 2 ESS 800 Infoserver Server 0 Server 1 DS8000 CLI interface CLI interface Storage HMC Network interface ESS CLI software ESS CLI software D...

Страница 227: ...and the help screens If a userid is not a member of any group then it is automatically placed into the no_access group to prevent it from performing any functions The default userid supplied with an...

Страница 228: ...pt ibm dscli chmod x listusers sh opt ibm dscli listusers sh opt ibm dscli cat userlist txt Name Group admin admin opt ibm dscli Interactive mode In the interactive mode the user starts the DS CLI pro...

Страница 229: ...ese include the ability to input variables from a file and send output to a file The DS CLI commands are also designed to be case insensitive This means commands can be entered in either upper lower o...

Страница 230: ...correct password then a failure reason and a return code will be presented Standard techniques to collect and analyze return codes can be used The return codes used by the DS CLI are shown in Table 10...

Страница 231: ...CMUN00018E lsflash Unable to connect to the management console server exit status of dscli 3 A connection error occurred Try pinging 10 0 0 1 If this fails call network support on 555 1001 C Program F...

Страница 232: ...SS 800 They will not work with the ESS F20 The storage management commands within the DS CLI also will not work with ESS 800 This means that customers who currently have a mix of 800s and F20s will ha...

Страница 233: ...he contents of each saved task and write them to a file 2 Review server scripts that perform task set up and execute saved ESS tasks Phase two Perform Having performed the review the scripts need to b...

Страница 234: ...4 IBM ESSCLI 2 4 0 Task Name Type Status H_Epath_test16 PPRCEstablishPaths NotRunning H_Epath_test17 PPRCEstablishPaths NotRunning Brocade_pr_lss10 PPRCEstablishPair NotRunning Brocade_pr_lss11 PPRCEs...

Страница 235: ...esscli show task s 10 0 0 1 u csadmin p passw0rd d name Flash10041005 Wed Nov 24 10 37 17 EST 2004 IBM ESSCLI 2 4 0 Taskname Flash10041005 Tasktype FCEstablish Options NoBackgroundCopy SourceServer 21...

Страница 236: ...xact syntax TargetServer 2105 23953 N A We only need to use the dev once so this is redundant Source and Target vols 1004 1005 1004 1005 The volume numbers don t change We simply separate them with a...

Страница 237: ...serid If you have DS CLI access to a DS offline configuration tool S HMC or DS Storage Management console then you can create an encrypted password file This will allow you to avoid specifying the pas...

Страница 238: ...23953 hmc1 10 0 0 1 hmc2 127 0 0 1 Username must be specified if a password file is used username csadmin Password filename is the name of an encrypted password file This file is located at C Program...

Страница 239: ...interpreter An example is shown in Example 10 18 Example 10 18 Establishing a FlashCopy with a single command anthony aixsrv opt ibm dscli dscli mkflash nocp 1004 1005 CMUC00137I mkflash FlashCopy pai...

Страница 240: ...ided some important information about the DS CLI This new CLI allows considerable flexibility in how DS6000 and DS8000 series storage servers are configured and managed It also detailed how an existin...

Страница 241: ...ed 217 Part 4 Implementation and management in the z OS environment In this part we discuss considerations for the DS6000 series when used in the z OS environment The topics include z OS software Data...

Страница 242: ...218 DS6000 Series Concepts and Architecture...

Страница 243: ...ilable in early 2005 Contact your IBM sales representative for more information about this tool and the benchmark testing that was done by the Tucson performance measurement lab Note that Disk Magic i...

Страница 244: ...rated architecture based on standard building blocks as introduced by IBM with the Seascape architecture Over time all components became not only bigger in capacity and faster in speed but also more s...

Страница 245: ...t task for a storage server than for a host server Perhaps it became this way with the evolving complexity of storage servers 11 2 Where do we start The IBM Enterprise Storage Server 2105 ESS already...

Страница 246: ...al to exceed the throughput capabilities of other vendor s FICON ports When not properly configured for example with four FICON ports in the same HA these powerful FICON ports have the potential to sa...

Страница 247: ...AL shortcomings There are some shortcomings with plain FC AL The most obvious ones are As the term arbitration implies each individual disk within an FC AL loop competes with the other disks to get o...

Страница 248: ...Channel disk subsystem backend as displayed in Figure 11 3 on page 225 Note that each DA port can concurrently send and receive data Processor Adapter Adapter Memory Processor Adapter Adapter To host...

Страница 249: ...ations due to two simultaneous operations from each DA to allow for two concurrent read operations and two concurrent write operations at the same time Despite the superior performance don t forget th...

Страница 250: ...ssors from this task The actual throughput and performance of a DA is not only determined by the 2 Gbps ports and used hardware but also by the firmware efficiency 11 3 3 New four port host adapters B...

Страница 251: ...uster or rather a dual server design which is also found in the ESS and DS8000 series This offers an enterprise class level of availability and functionality in a space efficient modular design at a l...

Страница 252: ...Figure 11 8 on page 229 shows a DS6800 with one DS6000 expansion enclosure Processor Adapter Adapter Adapter Adapter To host servers Storage server 2 Gbps Fibre Channel ports Card 1 Controller chipset...

Страница 253: ...its local HA and its Fibre Channel ports This introduces the concept of a preferred path When a volume has an affinity for example to server 0 and is accessed through a port in the HA of server 0 then...

Страница 254: ...he disk space requirements of the application should be well understood 11 4 1 Workload characteristics The answers to questions like how many host connections do I need how much cache do I need and t...

Страница 255: ...ial files but there are also benefits for random access DS6000 logical volumes are composed of extents An extent pool is a logical construct to manage a set of extents One or more ranks with the same...

Страница 256: ...t striping wherever possible to distribute the read and write I O access patterns across the physical resources of the DS6000 Note The stripe size has to be large enough to keep sequential data relati...

Страница 257: ...outing for optimum performance Multipath load balancing of data flow prevents a single path from becoming overloaded causing input output congestion that occurs when many I O operations are directed t...

Страница 258: ...to connect to zSeries servers or as FCP ports to connect to Fibre Channel attached open systems hosts The example in Figure 11 11 shows only two FICON Express channels But just two FICON Express chan...

Страница 259: ...to follow could be to propose how many ESS 800s might be consolidated into a DS6000 model From that you can derive the number of ESS 750s ESS F20s and ESS E20s which can collapse into a DS6000 The old...

Страница 260: ...mean FICON ports in the disk storage servers Coming from FICON Express channels you should then keep a minimum of four FICON ports You might consider using 25 fewer FICON ports in the DS6000 than the...

Страница 261: ...st impression At GA the IBM internal tool Disk Magic helps to model configurations based on customer workload data An IBM representative can contact support personnel who will use Disk Magic to config...

Страница 262: ...ent pools to ease the management effort of such a configuration But you can still assign multiple storage groups when you are not concerned about the placement of less active volumes Figure 11 12 Exte...

Страница 263: ...umes there is no straightforward approach to place certain volumes into certain ranks For example when you create the first 20 DB2 logging volumes they would be allocated in a consecutive fashion in t...

Страница 264: ...aller high performance extent pools through discrete SMS storage groups for each DS6000 For example in a dual logging database environment allow assignment of extent pool0 to the first logging volume...

Страница 265: ...of up to 67 2 TB Just the base enclosure provides up to 4 8 TB of physical storage capacity with 16 DDMs and 300 GB per DDM The small and fast DS6000 with its rich functionality and compatibility wit...

Страница 266: ...242 DS6000 Series Concepts and Architecture...

Страница 267: ...z VM z VSE and Transaction Processing Facility TPF software enhancements that support the DS6000 series The enhancements include Scalability support Large volume support Hardware configuration defini...

Страница 268: ...enhanced by way of a Small Programming Enhancement SPE Hosts with operating system software levels prior to z OS 1 4 are not supported zLinux will only recognize the DS6000 as a 2105 device The PSP in...

Страница 269: ...stay within this limit Today with the IBM 2105 volumes 32 760 cylinders are supported This gives us the capability to remain within the 64K limit But as today s storage facilities tend to expand to e...

Страница 270: ...3390 3390A 3390B device types that can be defined to this control unit The definition of the 1750 control unit type will allow this storage facility to be uniquely reflected in the Hardware Configura...

Страница 271: ...specified Figure 12 1 SETCACHE options All other parameters should be accepted as they are today on the IBM 2105 For example setting device caching ON is accepted but has no affect on the subsystem 1...

Страница 272: ...will then identify preferred paths to the IOS I Os will be directed to preferred paths to avoid crossing the PCI X connection The only time this will not be honored is when there are no preferred pat...

Страница 273: ...g system that supplies a virtual machine to each logged on user The DS6000 will be supported on z VM 4 4 and higher The PSP information can be found at http www 1 ibm com servers resourcelink svc03100...

Страница 274: ...ne transaction processing It is used by industries demanding large transaction volumes such as airlines and banks The DS6000 will be supported on TPF 4 1 and higher The PSP information can be found at...

Страница 275: ...sk storage server The focus is on z OS environments The following topics are covered from a planning standpoint Data migration objectives in z OS environments Data migration based on physical migratio...

Страница 276: ...ls 8 192 logical volumes This is twice as much as what an ESS 2105 can configure for a z OS based configuration This suggests consolidation of multiple control units into a single DS6800 disk storage...

Страница 277: ...3390 3 Or to put it differently we see on a single volume as many concurrent I Os as we see on nine 3390 3 volumes Despite the PAV support it still might be necessary to balance disk storage activiti...

Страница 278: ...om 3380 to 3390 Utilizing physical full volume operations is possible through the following software microcode and hardware based functions Software based DFSMSdss TDMF FDRPAS Software and hardware ba...

Страница 279: ...pen in a controlled fashion This lengthens the migration period so if possible other approaches might be considered Both software products are usually associated with fees or service based fees except...

Страница 280: ...sure Agent tasks which need to be installed in the customer systems An FDRPAS master task runs on the Piper CPU which coordinates all activities Monitor tasks are required on the customer s systems to...

Страница 281: ...ther alternative is z OS Global Mirror XRC XRC is an asynchronous solution that has a mode for disaster recovery solutions as well as a particular migration mode of SESSIONTYPE MIGRATE This mode does...

Страница 282: ...Ms because there is no potential bottleneck when coming from ESCON This would take about one day to replicate all data from the two F20s to the DS6800 provided there is not too much application write...

Страница 283: ...n the TSO command support for cascaded volumes is available Otherwise use a combination of TSO commands and ICKDSF for just defining the cascaded bit when setting up Metro Mirror between the intermedi...

Страница 284: ...r side does not impact the application write I Os due to its asynchronous data replication but the drawback here is that it does not guarantee data consistency at the receiving site Before switching f...

Страница 285: ...RAGE CONTROL DESCRIPTOR E8 DEVICE DESCRIPTOR 0A ADDITIONAL DEVICE INFORMATION 4A000035 ICK04030I DEVICE IS A PEER TO PEER REMOTE COPY VOLUME QUERY REMOTE COPY VOLUME PRIMARY SECONDARY SSID CCA SSID CC...

Страница 286: ...y some REXX based procedure might have to scan through the ICKDSF SYSPRINT output which is directed to a data set ICKDSF asks for a JCL DD statement for each single volume to query when the volume is...

Страница 287: ...microcode based mirroring capabilities require the right hardware as source and target disk servers For completeness it is pointed out that Global Mirror is also an option to migrate data from an ESS...

Страница 288: ...YSIN DD COPY DS INC EXCLUDE SYS1 VTOCIX SYS1 VVDS LIDD IN001 IN002 IN003 DELETE CATALOG ALLDATA ALLX WAIT 0 0 ADMIN OPT 3 CANCELERROR Example 13 5 depicts how to migrate all data sets from certain vol...

Страница 289: ...ction This approach assumes that the old and new disk servers can be configured concurrently to the concerned host servers 13 3 2 Data migration within the system managed storage environment System ma...

Страница 290: ...gh the ISMF panel sequence to make the SMS volume status complete and permanent Example 13 6 is the first ISMF Storage Group Application panel where you choose the actual Source Control Data Set SCDS...

Страница 291: ...rial Number List Specify a Single Volume in Prefix or Range of Volumes Prefix From To Suffix Type ______ ______ ______ _____ _ XC 6510 6514 X Use ENTER to Perform Selection Use HELP Command for Help U...

Страница 292: ...isnew MZBCVS2 disnew DISALL DISNEW QUIALL QUINEW SYS GROUP sysplex minus systems in the sysplex explicitly defined in the SCDS After pressing Enter and PF3 to validate and perform the SMS volume statu...

Страница 293: ...Copy Pool Backup only 4 Delete Delete Volumes from Volume Serial Number List Specify a Single Volume in Prefix or Range of Volumes Prefix From To Suffix Type ______ ______ ______ _____ _ DB2 001 300 X...

Страница 294: ...based on software utilities Problems encountered when not using an allocation manager like system managed storage are less flexibility when using esoteric unit names or complex and time consuming tas...

Страница 295: ...e to a different device type Another logical migration approach is possible through the CP PTAPE command PTAPE dumps spool files to tape and then re loads these files back onto the new disk storage La...

Страница 296: ...MF or FDRPAS is available and the customer is used to managing volume migration using these software tools then it is a likely approach to use these tools for a larger scale migration as well In other...

Страница 297: ...Implementation and management in the open systems environment In this part we discuss considerations for the DS6000 series when used in an open systems environment The topics include Open systems sup...

Страница 298: ...274 DS6000 Series Concepts and Architecture...

Страница 299: ...escribe how the DS6000 fits into your open systems environment In particular we discuss The extent of the open systems support Where to find detailed and accurate information Major changes from the ES...

Страница 300: ...Where to look for updated and detailed information on page 277 14 1 1 Supported operating systems and servers Table 14 1 provides an overview of the open system platforms operating systems and high a...

Страница 301: ...e output of the tool will be ambiguous You will be shown a list of all supported HBAs together with the required firmware and device driver levels for your combination Furthermore you can select a det...

Страница 302: ...r their HBAs requires registration Platform and operating system vendors pages The platform and operating system vendors also provide lots of support information to their customers Go there for genera...

Страница 303: ...ation about boot support Refer to The DS6000 Interoperability Matrix on page 277 The DS6000 Host Systems Attachment Guide SC26 7680 helps you with the procedures necessary to set up your host in order...

Страница 304: ...is accessed from a port that is located on the other server The request for the logical volume and the data would have to be transferred across the bridge interface that connects both servers These t...

Страница 305: ...thing MPIO see IBM AIX on page 303 IBM OS 400 V5R3 multipath support see Appendix B Using the DS6000 with iSeries on page 329 In addition there are third party multipathing solutions such as Veritas D...

Страница 306: ...r all supported host operating systems The DS CLI can be used in two modes Single command mode You invoke the DS CLI program in an operating system shell or command prompt and pass the command it is t...

Страница 307: ...ise policies for managing host application database and filesystem data As a component of the IBM TotalStorage Productivity Center Multiple Device Manager is designed to reduce the complexity of manag...

Страница 308: ...bed in the following sections 14 5 1 Device Manager The Device Manager DM builds on the IBM Director technology It uses the Service Level Protocol SLP to discover supported storage systems on the SAN...

Страница 309: ...re 14 3 shows an example of a DM view Figure 14 3 Sample Device Manager view In summary the Device Manager is responsible for Discovery of supported devices SMI S compliant Data collection asset avail...

Страница 310: ...individual device You can view performance data from the performance manager database in both graphical and tabular forms This is shown in Figure 14 4 Figure 14 4 Example screenshot of TPC for Disk T...

Страница 311: ...nd line interface 14 6 Global Mirror Utility The DS6000 Global Mirror Utility GMU is a standalone tool to provide a management layer for IBM TotalStorage Global Mirror failover and failback FO FB It p...

Страница 312: ...ified volumes and issue commands to bring the volumes back into sync It offers a tool to execute eRCMF configuration checks This check is intended to verify that the eRCMF configuration matches the ph...

Страница 313: ...Chapter 15 Data migration in the open systems environment In this chapter we discuss important concepts for the migration of data to the new DS6000 Data migration considerations Data migration and con...

Страница 314: ...solidation Very often the goal of a consolidation effort is a combination of more than one of these types There are many different methods for data migration To decide on what is best in your case gat...

Страница 315: ...or ways to propagate configuration changes throughout the cluster 15 2 Comparison of migration methods There are numerous methods that can be used to migrate data from one storage system to another We...

Страница 316: ...annot preserve advanced metadata such as access control lists or permissions Copy raw devices For raw data there are tools that allow you to read and write disk devices directly such as dd for UNIX Th...

Страница 317: ...w storage cut over time Once the initial copy is complete you keep it up to date with incremental rsync runs for instance once a day during off peak hours At the cut over time you stop the application...

Страница 318: ...Restore Every serious IT operation will have ways to back up and restore data They can be used for data migration We list this method here because it shares the common advantages and disadvantages wit...

Страница 319: ...refore be used with only minimum system administrator attention They also do not add any additional CPU load to the host systems and they don t require the host system to be connected to both storage...

Страница 320: ...t be mirrored directly to a DS6000 Combination of Metro Mirror and Global Copy A cascading Metro Mirror and Global Copy solution is useful in two cases The source system is already mirrored for disast...

Страница 321: ...lidation of small volumes into larger ones because it is not aware of the data structure Since Piper has to be in the data path between the host and the source storage system it can only be used to mi...

Страница 322: ...t and accurate data migration IBM provides technical specialists at your location to plan and migrate your data to your DS6000 disk system This migration is accomplished using either native operating...

Страница 323: ...tems specifics In this appendix we describe the particular issues of some operating systems with respect to the attachment to a DS6000 The following subjects are covered Planning considerations Common...

Страница 324: ...n we raise the questions you will have to ask and the things you have to consider before you start We don t cover all the items in detail because this is not an implementation book For a more detailed...

Страница 325: ...to use These tools offer a quick way to tell whether a system is I O bound iostat sar System Activity Report vmstat Virtual Memory Statistics IOSTAT The base tool for evaluating I O performance of di...

Страница 326: ...epending on the application layout this report could lead to the conclusion that the I O load of this system is unbalanced Some disks get a lot more I O request than others A consequence of this could...

Страница 327: ...spent executing operating system calls idle Time the system was idle with no outstanding I O requests The implementation of the sar command is different for the various UNIX variants However the outp...

Страница 328: ...S under AIX there was a set of Host Attachment scripts which could be downloaded from the Web For the DS6000 using AIX SDD 1 6 0 0 these scripts have been consolidated into one called FCP Host Attachm...

Страница 329: ...ronments in IBM products SDD resides in the host system with the native disk device driver and manages redundant connections between the host server and the DS6000 SDD is available for AIX 5 1 5 2 and...

Страница 330: ...th The storage device paths are managed to provide high availability and load balancing of storage I O MPIO is part of the base kernel and is available for AIX 5 2 and AIX 5 3 The base functionality o...

Страница 331: ...pg Enabled hdisk0 scsi0 Enabled hdisk1 scsi0 Enabled hdisk2 scsi0 Enabled hdisk3 scsi7 Enabled hdisk4 scsi7 Missing hdisk9 fscsi0 Missing hdisk10 fscsi0 Missing hdisk11 fscsi0 Missing hdisk12 fscsi0 M...

Страница 332: ...stributed across separate disks With the introduction of SAN technology LVM mirroring can even provide protection against a site failure Using long wave Fibre Channel connections a mirror can be stret...

Страница 333: ...serialization of write accesses It improves performance for many environments particularly commercial relational databases In many cases the database performance achieved using Concurrent I O with JFS...

Страница 334: ...ols on page 301 for more information filemon The filemon command monitors the performance of the file system and reports the I O activity with regard to files virtual memory segments logical volumes a...

Страница 335: ...MPIO FC 1750 0 87 672 776 86 1 dev hdisk81 IBM MPIO FC 1750 0 86 392 960 80 4 dev hdisk63 IBM MPIO FC 1750 0 86 328 776 65 6 dev hdisk83 IBM MPIO FC 1750 0 86 528 624 68 5 dev hdisk69 IBM MPIO FC 1750...

Страница 336: ...pport Depending on the target group the distributions differ largely in the kind of support that is available Linux is available for almost all computer architectures Linux is rapidly changing All the...

Страница 337: ...o an IBM eServer BladeCenter running Linux It can be downloaded from http publib b boulder ibm com abstracts sg246261 html Linux with zSeries and ESS Essentials The redbook Linux with zSeries and ESS...

Страница 338: ...CSI disks The Linux SCSI I O subsystem has some peculiarities that are important enough to be described here even if they show up in some of the publications listed in the previous section Some Linux...

Страница 339: ...on of special device files for the 17th SCSI disk and its first three partitions Example A 10 Create new special device files for SCSI disks mknod dev sdq b 65 0 mknod dev sdq1 b 65 1 mknod dev sdq2 b...

Страница 340: ...contains a list of the supported kernels The version of the Linux Logical Volume Manager that comes with all current Linux distributions does not support its physical volumes being placed on SDD vpat...

Страница 341: ...before the built in parallel SCSI controller that drives the system disks This will lead to shifted special device file names of the system disk and can result in the system being unable to boot prop...

Страница 342: ...st be issued for each installed HBA with the adapter instance being the SCSI instance number of the HBA After the FC HBAs rescan the fabric you can make the new devices available to the system with th...

Страница 343: ...a device behind it When assigning volumes to a Linux host with QLogic FC HBAs make sure LUNs start at 0 and are in consecutive order Otherwise the LUNs after a gap will not be discovered by the host...

Страница 344: ...0 Vendor IBM Model 32P0032a S320 1 Rev 1 Type Processor ANSI SCSI revision 02 Host scsi2 Channel 00 Id 00 Lun 00 Vendor IBM Model 1750511 Rev 545 Type Direct Access ANSI SCSI revision 03 Host scsi2 Ch...

Страница 345: ...SCSI devices They are called the sg tools They talk to the SCSI devices through the generic SCSI layer which is represented by special device files dev sg0 dev sg0 and so on By default SLES 8 provide...

Страница 346: ...AL_MACHINE SYSTEM CurrentControlSet Services Disk TimeOutValue The value has the data type REG DWORD and should be set to 0x0000003c hexadecimal 60 decimal SDD for Windows An important task with a Win...

Страница 347: ...ID arrays Windows Server 2003 components that work with VDS include the Disk Management MMC snap in the DiskPart command line tool and the DiskRAID command line tool which is available in the Windows...

Страница 348: ...istent point in time copies of data known as shadow copies It integrates IBM TotalStorage FlashCopy to produce consistent shadow copies while also coordinating with business applications file system s...

Страница 349: ...the SCSI standard for vendor s private use OpenVMS does not identify a Fibre Channel disk by its path or SCSI target LUN like other operating systems It relies on the UDID Although OpenVMS uses the W...

Страница 350: ...e with a minimum size as a dummy volume for usage as the CCL Multiple OpenVMS hosts even in different clusters that access the same storage system can share the same volume as LUN 0 because there will...

Страница 351: ...tp research compaq com wrl DECarchives DTJ DTJ301 DTJ301SC TXT The DS6000 provides volumes as SCSI 3 devices and thus does not implement a forced error indicator It also does not support the READL and...

Страница 352: ...328 DS6000 Series Concepts and Architecture...

Страница 353: ...Using the DS6000 with iSeries In this appendix the following topics are discussed Supported environment Logical volume sizes Protected versus unprotected volumes Multipath Adding units to OS 400 confi...

Страница 354: ...age_hw html Software The iSeries must be running V5R2 or V5R3 i5 OS of OS 400 In addition at the time of writing the following PTFs are required V5R2 MF33327 MF33301 MF33469 MF33302 SI14711 and SI1475...

Страница 355: ...therwise when mirroring is started to mirror the LSU to the DS6000 LUN OS 400 will attempt to mirror all unprotected volumes Changing LUN protection It is not possible to simply change a volume from p...

Страница 356: ...be done either using the green screen interface with Dedicated Service Tools DST or System Service Tools SST or with the iSeries Navigator GUI The following example shows how to add a logical volume...

Страница 357: ...to panel Specify the ASP number next to the desired units Here we have specified ASP1 the System ASP Press Enter Work with Disk Units Select one of the following 1 Display disk configuration 2 Work wi...

Страница 358: ...5 shows the initial panel Specify ASPs to Add Units to Specify the ASP to add each unit to Specify Serial Resource ASP Number Type Model Capacity Name 21 662C5 4326 050 35165 DD124 21 54782 4326 050 3...

Страница 359: ...nitial panel 2 Expand the iSeries to which you wish to add the logical volume and sign on to that server as shown in Figure B 7 Figure B 7 iSeries Navigator Signon to iSeries panel 3 Expand Configurat...

Страница 360: ...8 iSeries Navigator Disk Units 4 You will be asked to sign on to SST as shown in Figure B 9 Enter your Service tools ID and password and press OK Figure B 9 SST Signon 5 Right click Disk Pools and sel...

Страница 361: ...Appendix B Using the DS6000 with iSeries 337 Figure B 10 Create a new disk pool 6 The New Disk Pool wizard appears as shown in Figure B 11 Click Next Figure B 11 New disk pool welcome...

Страница 362: ...es the type of logical volume you are adding If you leave it unchecked you will see all available disks Select OK to continue Figure B 12 Defining a new disk pool 8 A confirmation panel like that show...

Страница 363: ...st of non configured units similar to that shown in Figure B 15 will appear Highlight the disks you want to add to the disk pool and click Add Figure B 15 Choose the disks to add to the Disk Pool 11 A...

Страница 364: ...Click Finish to add the disks to the Disk Pool Figure B 17 New Disk Pool Summary 13 Take note of and respond to any message dialogs which appear After taking action on any messages the New Disk Pool...

Страница 365: ...gure B 19 Figure B 19 Disks added successfully to Disk Pool 15 The new Disk Pool can be seen on iSeries Navigator Disk Pools in Figure B 20 Figure B 20 New Disk Pool shown on iSeries Navigator 16 To s...

Страница 366: ...tical to OS 400 due to the single level storage architecture Multipath is not available for iSeries internal disk units but the likelihood of path failure is much less with internal drives This is bec...

Страница 367: ...re fifteen single points of failure excluding the iSeries itself and the DS6000 storage facility Failure points 9 12 will not be present if you do not use an Inter Switch Link ISL to extend your SAN A...

Страница 368: ...haring Configuring multipath iSeries has two I O adapters that support DS6000 2766 2 Gigabit Fibre Channel Disk Controller PCI 2787 2 Gigabit Fibre Channel Disk Controller PCI X Both can be used for m...

Страница 369: ...es is also assigned to a Fibre Channel I O adapter on a different BUS in the second iSeries I O tower or rack via a host adapter in the upper controller card Adding multipath volumes to iSeries using...

Страница 370: ...ce ASP Number Type Model Capacity Name 21 662C5 4326 050 35165 DD124 21 54782 4326 050 35165 DD136 1 75 1118707 1750 A85 35165 DMP135 F3 Exit F5 Refresh F11 Display disk configuration capacity F12 Can...

Страница 371: ...s appear as DMPxxx Highlight the disks you want to add to the disk pool and click Add Figure B 27 Adding a multipath volume The remaining steps are identical to those in Adding volumes to an Independe...

Страница 372: ...een on iSeries Navigator Disk Pools in Figure B 28 Figure B 28 New Disk Pool shown on iSeries Navigator To see the logical volume as shown in Figure B 29 expand Configuration and Service Hardware Disk...

Страница 373: ...using the standard disk panels in iSeries Navigator only a single the initial path is shown The following steps show how to see the additional paths To see the number of paths available for a logical...

Страница 374: ...s Concepts and Architecture To see the other connections to a logical unit right click the unit and select Properties as shown in Figure B 31 on page 350 Figure B 31 Selecting properties for a multipa...

Страница 375: ...DS6000 with iSeries 351 You will then see the General Properties tab for the selected unit as in Figure B 32 The first path is shown as Device 1 in the box labelled Storage Figure B 32 Multipath logi...

Страница 376: ...other node The system enforces the following rules when you use multipath disk units in a multiple system environment If you move an IOP with a multipath connection to a different logical partition yo...

Страница 377: ...may lead to some small performance degradation for large workloads This is because a longer path will be used when accessing logical volumes across the midplane in the DS6000 Preferred path would acce...

Страница 378: ...gularly or perhaps FlashCopy target volumes which could use this capacity but not impact on the I O per sec on those arrays For very high write environments you may also consider using RAID 10 which o...

Страница 379: ...reduce I O queues and wait times by allowing OS 400 to support more parallel I Os From the values in Table B 2 you can calculate the number of iSeries Fibre Channel adapters for your required iSeries...

Страница 380: ...rally iSeries has a relatively high I O rate while that of other servers may be lower often below one I O per GB per second As an example a Windows file server with a large data capacity may normally...

Страница 381: ...erted back to a protected model and therefore will be a candidate for all future OS 400 mirroring whether you want this or not See Table B 1 on page 330 for DS6000 logical volume models and types Metr...

Страница 382: ...to use native OS 400 functions to migrate data from existing disks to the DS6000 whether the existing disks are internal or external When you assign the new DS6000 logical volumes to the iSeries initi...

Страница 383: ...the balancing function completes or is ended If the balance function is run for a few hours and then stopped it will continue from where it left off when the balance function restarts This allows the...

Страница 384: ...for other functions such as test development and Lotus Notes When backups are to be done the FlashCopy target can be attached to the partition without affecting the rest of the users Or perhaps more...

Страница 385: ...Copy and Remote Mirror functions with iSeries for practical reasons many customers will choose not to use basic FlashCopy and Remote Mirror functions due to the LSU restrictions discussed previously I...

Страница 386: ...also supports earlier iSeries servers The DS6000 requires the following iSeries I O adapters to attach directly to an iSeries or i5 Linux partition 0612 Linux Direct Attach PCI 0626 Linux Direct Atta...

Страница 387: ...Services for TotalStorage disk systems IBM Implementation Services for TotalStorage Copy Functions IBM Implementation Services for TotalStorage Command Line Interface IBM Migration Services for eServ...

Страница 388: ...Copy Functions This service is designed to assist in the planning implementation and testing of the IBM TotalStorage advanced copy functions Point in Time Copy and remote mirroring solutions For more...

Страница 389: ...on page 297 For the latest information about this service visit the following Web site http www ibm com servers storage services featured hardware_assist html For more information about IBM Migration...

Страница 390: ...featured microsoft_application_environment html GDSSolution IBM eServer iSeries Copy Services For the iSeries environment IBM offers a special toolkit which allows you to use the advanced Copy Servic...

Страница 391: ...Appendix C Service and support offerings 367 Figure 15 9 Example of the Supported Product List SPL from the IBM Support Line...

Страница 392: ...368 DS6000 Series Concepts and Architecture...

Страница 393: ...olutions Guide SG24 6547 iSeries and IBM TotalStorage A Guide to Implementing External Disk on eServer i5 SG24 7120 IBM TotalStorage Productivity Center V2 3 Getting Started SG24 6490 Managing Disk Su...

Страница 394: ...for the DS6000 http www ibm com servers storage support disk 1750 html Supported servers for the DS6000 http www storage ibm com hardsoft products DS6000 supserver htm The interoperability matrix htt...

Страница 395: ...m How to get IBM Redbooks You can search for view or download Redbooks Redpapers Hints and Tips draft publications and Additional materials as well as order hardcopy Redbooks or CD ROMs at this Web si...

Страница 396: ...372 DS6000 Series Concepts and Architecture...

Страница 397: ...ler architecture 25 failover and failback 46 Copy Services 89 definitions 90 interfaces 108 iSeries 360 iSeries toolkit 361 management 130 CRU 61 CRU endpoint indicator 60 CUIR 52 D DA 33 ports indica...

Страница 398: ...37 Global Copy 90 Global Mirror 90 hardware components 23 hardware overview 6 health indicators 35 host attachment 78 Host Systems Attachment Guide 300 information lifecycle management 4 infrastructur...

Страница 399: ...data sets 10 inband commands 11 97 incremental 11 multiple relationship 10 options 92 persistent 97 floating spares 54 floor and space requirements 116 front panel 37 FRU 61 G GDPS 365 GDS for MSCS 3...

Страница 400: ...hannel adapters 355 OS 400 data migration 358 OS 400 mirroring 357 preferred path for DS6000 353 protected versus unprotected volumes 330 recommended number of ranks 355 Remote Mirror and Copy 360 sha...

Страница 401: ...150 UNIX tools 301 z OS 233 DS6000 size 235 potential 234 statistics 247 Persistent FlashCopy 97 PFA 13 55 physical migration zSeries 254 DFSMSdss 254 Piper 255 297 Point in time Copy see PTC port Eth...

Страница 402: ...tion indicator 60 system service 57 card 42 T TPC 282 for disk 285 for replication 287 TPF 250 U UNIX iostat 301 performance monitoring tools 301 SAR vmstat 303 V VDS 323 Virtual Disk Service see VDS...

Страница 403: ...0 5 spine 0 475 0 873 250 459 pages DS6000 Series Concepts and Architecture DS6000 Series Concepts and Architecture DS6000 Series Concepts and Architecture DS6000 Series Concepts and Architecture...

Страница 404: ...DS6000 Series Concepts and Architecture DS6000 Series Concepts and Architecture...

Страница 405: ......

Страница 406: ...6000 storage server series its architecture its logical design hardware design and components advanced functions performance features and specific characteristics The information contained in this boo...

Отзывы:

Нет отзывов

Похожие инструкции для System storage DS6000 Series

Бренд: Yamaha Страницы: 3

Home Garden Pub

Бренд: Fishers Woodscraft Страницы: 8

H-Post Shelving

Бренд: Hallowell Страницы: 7

Бренд: IBM Страницы: 2

Бренд: Yardmaster Страницы: 24

Бренд: I-O DATA Страницы: 2

Бренд: U.S. General Страницы: 2

G-RAID mini 1TB

Бренд: G-Technology Страницы: 13

Бренд: Synology Страницы: 31

Бренд: Philips Страницы: 2

Бренд: Philips Страницы: 2

Бренд: Philips Страницы: 2

Бренд: Philips Страницы: 2

Бренд: Philips Страницы: 2

Бренд: Philips Страницы: 2

Бренд: Philips Страницы: 2

Бренд: Philips Страницы: 2

Бренд: Philips Страницы: 10

Бренды по названию

0 1 2 3 4 5 6 7 8 9 A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Популярные бренды

Загрузить еще бренды