IBM System storage DS6000 Series Redbooks Download Page 22 | Manualshive

Page: 22 / 406

background image

xx

DS6000 Series: Concepts and Architecture

Figure 0-1 Front row - Cathy, Torsten R, Torsten K, Andre, Toni, Werner, Tetsuroh. Back row - Roland, Olivier, Anthony,
Tang, Christine, Alex, Stu, Heinz, Chuck.

We want to thank all the members of John Amann’s team at the Washington Systems Center
in Gaithersburg, MD for hosting us. Craig Gordon and Rosemary McCutchen were especially
helpful in getting us access to beta code and hardware.

Thanks to the following people for their contributions to this project:

Susan Barrett
IBM Austin

James Cammarata
IBM Chicago

Dave Heggen
IBM Dallas

John Amann, Craig Gordon, Rosemary McCutchen
IBM Gaithersburg

Hartmut Bohnacker, Michael Eggloff, Matthias Gubitz, Ulrich Rendels, Jens Wissenbach,
Dietmar Zeller
IBM Germany

Brian Sherman
IBM Markham

Ray Koehler
IBM Minneapolis

John Staubi
IBM Poughkeepsie

Steve Grillo, Duikaruna Soepangkat, David Vaughn
IBM Raleigh

Amit Dave, Selwyn Dickey, Chuck Grimm, Nick Harris, Andy Kulich, Jim Tuckwell, Joe Writz
IBM Rochester

Charlie Burger, Gene Cullum, Michael Factor, Brian Kraemer, Ling Pong, Jeff Steffan, Pete
Urbisci, Steve Van Gundy, Diane Williams
IBM San Jose

Jana Jamsek
IBM Slovenia

«
...
20
21
22
23
24
...
»

Summary of Contents for System storage DS6000 Series

Page 1: ...Juan Antonio Garay Torsten Knobloch Donald C Laing Christine O Sullivan Stu S Preacher Torsten Rothenwaldt Tetsuroh Sano Jing Nan Tang Anthony Vandewerdt Alexander Warmuth Roland Wolf Enterprise class...

Page 2: ......

Page 3: ...International Technical Support Organization The IBM TotalStorage DS6000 Series Concepts and Architecture March 2005 SG24 6471 00...

Page 4: ...t an early version of DS6000 microcode was used for the screen captures and command output so some details may vary from the currently available microcode Note Before using this information and the pr...

Page 5: ...iii...

Page 6: ...iv DS6000 Series Concepts and Architecture...

Page 7: ...rted environment 9 1 2 5 Business continuance functions 10 1 2 6 Resiliency 13 1 2 7 Interoperability 13 1 2 8 Service and setup 13 1 2 9 Configuration flexibility 14 1 3 Positioning the IBM TotalStor...

Page 8: ...ID 10 overview 53 3 3 3 Spare creation 54 3 3 4 Predictive Failure Analysis PFA 55 3 3 5 Disk scrubbing 55 3 3 6 Disk path redundancy 55 3 4 Power subsystem RAS 56 3 5 System service 57 3 5 1 Example...

Page 9: ...preparation 116 7 2 1 Floor and space requirements 116 7 2 2 Power requirements 118 7 2 3 Environmental requirements 118 7 2 4 Preparing the rack 119 7 3 System management interfaces 119 7 3 1 IBM To...

Page 10: ...The DS Storage Manager Logical configuration 151 9 1 Configuration hierarchy terminology and concepts 152 9 1 1 Storage configuration terminology 152 9 1 2 Summary of the DS Storage Manager logical co...

Page 11: ...ON to FICON ports 222 11 2 4 PPRC over Fibre Channel links 222 11 2 5 Fixed LSS to RAID rank affinity and increasing DDM size 222 11 3 How does the DS6000 address the challenge 223 11 3 1 Fibre Channe...

Page 12: ...d hardware based data migration 255 13 2 3 Hardware and microcode based migration 258 13 3 Data migration based on logical migration 263 13 3 1 Data Set Services Utility DFSMSdss 264 13 3 2 Data migra...

Page 13: ...305 LVM configuration 308 AIX access methods for I O 308 Boot device support 309 AIX on IBM iSeries 309 Monitoring I O performance 310 Linux 312 Support issues that distinguish Linux from other operat...

Page 14: ...hannel adapters 355 Size and number of LUNs 355 Recommended number of ranks 355 Sharing ranks between iSeries and other servers 356 Connecting via SAN switches 356 Migration 357 OS 400 mirroring 357 M...

Page 15: ...in this publication at any time without notice Any references in this information to non IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web s...

Page 16: ...em 38 Tivoli TotalStorage Proven TotalStorage VSE ESA The following terms are trademarks of other companies Solaris Java and all Java based trademarks and logos are trademarks or registered trademarks...

Page 17: ...updated on December 13 2005 March 2005 First Edition This revision reflects the addition deletion or modification of new and changed information described below Changed information see change bars Fi...

Page 18: ...xvi DS6000 Series Concepts and Architecture...

Page 19: ...m Characteristics of the DS6000 series described in this redbook also include the DS6000 copy functions FlashCopy Metro Mirror Global Copy and Global Mirror The performance features particularly the s...

Page 20: ...previous zSeries Copy Services redbook Juan Antonio Garay is a Storage Systems Field Technical Sales Specialist in Germany He has five years of experience in supporting and implementing z OS and Open...

Page 21: ...of IBM China He has nine years of experience in the IT field His main job responsiblity is providing technical support and IBM storage solutions to IBM professionals Business Partners and customers Hi...

Page 22: ...ons to this project Susan Barrett IBM Austin James Cammarata IBM Chicago Dave Heggen IBM Dallas John Amann Craig Gordon Rosemary McCutchen IBM Gaithersburg Hartmut Bohnacker Michael Eggloff Matthias G...

Page 23: ...with leading edge technologies You ll team with IBM technical professionals Business Partners and or customers Your efforts will help increase product acceptance and customer satisfaction As a bonus y...

Page 24: ...xxii DS6000 Series Concepts and Architecture...

Page 25: ...BM Corp 2004 All rights reserved 1 Part 1 Introduction In this part we introduce the IBM TotalStorage DS6000 series and its key features The topics covered include Product overview Positioning Perform...

Page 26: ...2 DS6000 Series Concepts and Architecture...

Page 27: ...ter provides an overview of the features functions and benefits of the IBM TotalStorage DS6000 series of storage servers The topics covered include Overview of the DS6000 series and its benefits Posit...

Page 28: ...investments Storage asset consolidation can be greatly assisted by virtualization Virtualization software solutions are designed to logically combine separate physical storage systems into a single vi...

Page 29: ...Models 800 and 750 as well as with the new DS8000 series The DS6000 series offers an entirely new era in price performance and scalability Now for the first time zSeries and iSeries customers have th...

Page 30: ...s for the storage server and the host adapters respectively and another PowerPC 750FX 500 MHz processor for the device adapter on each controller card The DS6800 is equipped with 2 GB memory in each c...

Page 31: ...AL interface Instead of forming an FC AL loop each disk drive is connected to two Fibre Channel switches within each enclosure With this switching technology there is a point to point connection to e...

Page 32: ...rding to your performance needs you can select from four different disk drive types fast 73 GB and 146 GB drives rotating at 15 000 RPM good performing and cost efficient 146 GB drives operating at 10...

Page 33: ...ou to create and save a task using the GUI The following list highlights a few of the specific types of functions that you can perform with the DS CLI Check and verify your storage unit configuration...

Page 34: ...a persistent FlashCopy The benefits of FlashCopy are that the point in time copy is immediately available for use for backups and the source volume is immediately released so that applications can be...

Page 35: ...S6800 to hold off I O activity to a LUN or volume until all LUNs volumes within the group have established a FlashCopy relationship to their targets and the FlashCopy Consistency Group Created command...

Page 36: ...nplanned outage The actual lag in data currency experienced will depend upon a number of factors including specific workload characteristics and bandwidth between the local and remote sites Efficient...

Page 37: ...D 10 arrays will require that two global spares are defined in the DS6800 controller enclosure If you have expansion enclosures the first enclosure will have another two global spares More spares coul...

Page 38: ...call home process Configuration changes like adding disk drives or expansion enclosures are a non disruptive process Most maintenance actions are non disruptive including downloading and activating ne...

Page 39: ...ividual Host Bus Adapters HBAs and volumes but instead with groups Summary In summary the DS6000 series allows for Up to 32 logical subsystems Up to 8192 logical volumes Up to 1040 volume groups Up to...

Page 40: ...gh end or midrange systems without the need for the storage administrators to learn about a new product This will reduce your management costs and the total cost of ownership 1 3 2 Common management f...

Page 41: ...but their functions differ With respect to performance the DS4000 series range is below the DS6000 series You have the option to choose from different DS4000 models among them very low cost entry mode...

Page 42: ...servers It is designed as a highly scalable solution supporting both very large files and very large numbers of files without the limitations normally associated with Network File System NFS or Commo...

Page 43: ...rformance features available on the DS6000 series Parallel Access Volumes PAV PAV is an optional feature for zSeries environments which enables a single zSeries server to simultaneously process multip...

Page 44: ...20 DS6000 Series Concepts and Architecture...

Page 45: ...1 Part 2 Architecture In this part we describe various aspects of the DS6000 series architecture These include Hardware components RAS reliability availability and serviceability Virtualization concep...

Page 46: ...22 DS6000 Series Concepts and Architecture...

Page 47: ...6000 Here you can read about the DS6000 hardware platform and its components Server enclosure Expansion enclosure Controller architecture Disk subsystem Server enclosure RAID controller card Expansion...

Page 48: ...is the front display panel that provides status indicators You can also see the disk drive modules or DDMs Each enclosure can hold up to 16 DDMs Figure 2 2 DS6800 rear view The rear view of the DS680...

Page 49: ...that connects these components is pictured in Figure 2 4 on page 26 Effectively the DS6800 consists of two controller cards Each controller card contains an integrated four port host adapter to connec...

Page 50: ...n case it is required again The controllers attempt to anticipate future reads by an algorithm known as SARC sequential prefetching in adaptive replacement cache Data is held in cache as long as possi...

Page 51: ...he main memory 2 3 2 Cache management Most if not all high end disk systems have internal cache integrated into the system design and some amount of system cache is required for operation Over time ca...

Page 52: ...running concurrently To decide which pages are evicted when the cache is full sequential and random non sequential data is separated into different lists see Figure 2 5 A page which has been brought i...

Page 53: ...sure can contain 16 DDMs or dummy carriers A dummy carrier looks very similar to a DDM in appearance but contains no electronics As discussed earlier from the front the server enclosure and the expans...

Page 54: ...ion is different The key features of switched FC AL technology are Standard FC AL communication protocol from DA to DDMs Direct point to point links are established between DA and DDM Isolation capabi...

Page 55: ...on page 32 It depicts how the DS6000 is divided into two disk loops The server enclosure which contains the first 16 DDMs is on loop 0 The first expansion enclosure is placed on loop 1 This allows fo...

Page 56: ...uration discussed in Chapter 10 DS CLI on page 195 you have the choice of creating a RAID 5 or RAID 10 array by choosing one or two array sites The first and the third array sites created on each loop...

Page 57: ...t of an unexpected power failure If the DS6800 were to power off with un destaged writes in NVS then after reboot the controller would read this reserved area and destage the writes For more details o...

Page 58: ...de cables respectively all with LC connectors The ports in each controller are effectively on a PCI X 64 Bit 133 MHz card the same card used in the DS8000 The chipset is driven by a new high function...

Page 59: ...dicator is green and indicates that the controller card is powered on The lower indicator is amber and indicates that this controller requires service 2 6 Expansion enclosure SBOD controller card The...

Page 60: ...is cabled to the lower SBOD card In each case cables run from the disk exp ports to the in ports of the SBOD card A second expansion enclosure has been added by running cables from the out ports on th...

Page 61: ...ng to add more expansion enclosures to that loop Figure 2 15 Expansion enclosure cabling Disk Contrl ports on loop 1 2 7 Front panel The DS6000 front operator panel allows you to perform a health chec...

Page 62: ...System Information amber Circled letter i This indicator is normally off If it is on solid then an error has occurred that cannot be fixed by light path diagnostics To turn this light off you need to...

Page 63: ...llow a user to attach the enclosure to eServer rack identifier hardware This allows you to identify in which rack a particular DS6000 storage or expansion enclosure is located Power control switch Dat...

Page 64: ...Each power supply has two integrated fans for cooling the entire enclosure If a power supply fails it should not be removed from the enclosure until a replacement one is available Both power supplies...

Page 65: ...ht hand BBU supports the lower controller Figure 2 19 DS6000 battery backup unit indicators The DS6000 battery units have three indicators detailed in Table 2 4 Table 2 4 DS6800 battery backup unit in...

Page 66: ...controllers to set IP addresses After this it is not used You do not use it to connect the two controllers together for normal operation or during initial power on this connectivity is supplied by an...

Page 67: ...43 2 13 Summary This chapter has described the various components that make up a DS6000 For additional information there is documentation available on the Web at http www 1 ibm com servers storage sup...

Page 68: ...44 DS6000 Series Concepts and Architecture...

Page 69: ...chapter describes the RAS reliability availability and serviceability characteristics of the DS6000 series Specific topics covered are Controller RAS Host connection availability Disk subsystem RAS Po...

Page 70: ...s which for open systems are called LUNs and for zSeries 3390s These logical volumes belong to a logical subsystem LSS For open systems the LSS membership is not that important unless you are using Co...

Page 71: ...ers are actively processing I O requests This section describes the failover and failback procedures that occur between the DS6800 controllers when an abnormal condition has affected one of them Failo...

Page 72: ...with RAID 10 storage certain RAID related recoveries may cause latency impacts in excess of 15 seconds If you have real time response requirements in this area contact IBM to determine the latest info...

Page 73: ...nts to validate the integrity of the data as it moves throughout the disk system It is also checked by the DS6800 before the data is sent to the host in response to a read I O request Further the meta...

Page 74: ...DS6800 each using multiple paths the eight host adapter ports that are available in the DS6800 may not be sufficient to accommodate all the connections The solution to this problem is the use of SAN...

Page 75: ...00 or DS8000 A new version of SDD Version 1 6 will also allow SDD to manage pathing to the DS6800 and DS8000 SDD provides availability through automatic I O path failover If a failure occurs in the da...

Page 76: ...ystem images with the appropriate level of software support will respond to such requests by varying off the affected paths and either notifying the DS6800 subsystem that the paths are offline or that...

Page 77: ...the failed drive require data to be read from the other drives in the array to reconstruct the data The remaining requests are satisfied by reading the drive containing the data in the normal way Perf...

Page 78: ...his normally means that two spares will be created in the server enclosure and two spares in the first expansion enclosure Spares are created as the array sites are created which occurs when the DDMs...

Page 79: ...alert the user and can also use call home e mail notification 3 3 5 Disk scrubbing The DS6000 will periodically read all sectors on a disk This is designed to occur without any interference to applica...

Page 80: ...nt of a BBU failure the RAID controller that relies on that BBU for data protection will remove itself from service and go offline until its BBU is fully charged If both BBUs were to fail then the ent...

Page 81: ...e DS6000 and using the simple replacement instructions detailed there they remove and replace the failed DDM with a new one After replacing the DDM the System Alert Indicator will be turned off automa...

Page 82: ...t removal instructions Figure 3 7 on page 59 shows an example of the screen the user may see On this screen users are given the ability to do things like a View an animation of the removal and replace...

Page 83: ...alls the replacement power supply 3 Finally the user checks the component view to review system health after the repair An example of this is shown in Figure 3 8 In this example we can see that all th...

Page 84: ...emind button is pressed the Alert indicator will go from solid to a 2 second blip short blip on for 250 msec every 2 seconds The System Alert Indicator and the System Information Indicator can be on a...

Page 85: ...arts there are currently two tiers Tier 1 CRUs are relatively easy to replace while Tier 2 CRUs are generally more expensive parts or parts that require more skill to replace Tier 1 CRU parts Battery...

Page 86: ...g the code level currently available at http www 1 ibm com servers storage support disk index html The currently available version of code can also be downloaded from this Web site Users can also regi...

Page 87: ...he alternative to load code non concurrently This means that both controllers are unavailable for a short period of time This method can be performed in a smaller window of time 3 7 Summary This chapt...

Page 88: ...64 DS6000 Series Concepts and Architecture...

Page 89: ...ncepts This chapter describes the virtualization concepts for the DS6000 and the abstraction layers for disk virtualization The topics covered are Array sites Arrays Ranks Extent pools Logical volumes...

Page 90: ...e set is a group of 4 drives of the same capacity and RPM The disk drives can be accessed by a pair of device adapters Each device adapter has four paths to the disk drives The four paths provide two...

Page 91: ...So it is not really a loop but a switched FC AL loop with the FC AL addressing schema Arbitrated Loop Physical Addressing AL PA 4 2 1 Array sites An array site is a group of four DDMs What DDMs make u...

Page 92: ...up to two spares may be taken from the array sites used to construct the array on each device interface loop See Chapter 5 IBM TotalStorage DS6000 model overview on page 83 for more details Figure 4...

Page 93: ...Figure 4 4 on page 70 and indicated by the small squares in Figure 4 5 on page 71 The process of forming a rank does two things The array is defined for either fixed block open systems or CKD zSeries...

Page 94: ...he squares in the rank just indicate that the extent is composed of several blocks from different DDMs Figure 4 4 Forming an FB rank with 1 GB extents 4 2 4 Extent pools An extent pool is a logical co...

Page 95: ...gn it to one server and define a CKD extent pool and assign it to the other server Additional extent pools may be desirable to segregate ranks with different DDM types Ranks are organized in two rank...

Page 96: ...ume is composed of one or more extents from one CKD extent pool CKD extents are of the size of 3390 Model 1 which has 1113 cylinders However when you define a zSeries CKD volume you do not specify the...

Page 97: ...cation process for FB volumes is very similar and is shown in Figure 4 7 on page 74 1113 1113 1113 1113 free Extent Pool CKD0 Rank x Rank y 3390 Mod 3 1113 free 1113 free used used Allocate 3226 cylin...

Page 98: ...the iSeries LUNs are in terms of the 512 byte block capacity and are expressed in GB 109 These capacities should be converted to GB 230 when considering effective utilization of extents which are 1 G...

Page 99: ...nd device adapters and associated ranks The association of an 8 pack to a device adapter determined what LSS numbers could be chosen for a volume On an ESS up to 16 LSSs could be defined depending on...

Page 100: ...of pairs to preserve data consistency across all pairs in case you have a problem with one of the pairs is done at the LSS level With the option now to put all or most of the volumes of a certain app...

Page 101: ...the address group X g and the LSS number within the address group X a and the position of the LUN within the LSS X bb For example LUN X 1101 denotes the second X 01 LUN in LSS X 11 of address group 1...

Page 102: ...s how the logical volume number is converted to a host addressable LUN_ID on the Fibre Channel SCSI interface A map volume group type is used in conjunction with FC SCSI host types that poll for LUNs...

Page 103: ...B or CKD data Next the extents were added to an extent pool which determined which storage server would serve the ranks and aggregated the extents of all ranks in the extent pool for subsequent alloca...

Page 104: ...tribute your data for optimal performance you should make sure that you spread it across the two servers and across the two loops and across several ranks If you use some kind of a logical volume mana...

Page 105: ...sed Volumes for a particular application can be kept in a single LSS Smaller LSSs can be defined if required for systems applications requiring less storage Test systems can have their own LSSs with f...

Page 106: ...82 DS6000 Series Concepts and Architecture Dynamically add remove volumes Virtualization reduces storage management requirements...

Page 107: ...iew of the IBM TotalStorage DS6000 storage server which is from here on referred to as the DS6000 While the DS6000 is physically small it is a highly scalable and powerfully performing storage server...

Page 108: ...er DDM As of this writing the maximum storage capacity with 300 GB DDMs is 4 8 TB with 16 DDMs in a 1750 511 model It measures 5 25 inches high and is available in a 19 inch rack mountable package Fig...

Page 109: ...r an up to date and complete interoperability matrix refer to http www ibm com servers storage disk ds6000 interop html Figure 5 2 DS6800 Model 1750 511 rear view 5 1 2 DS6000 Model 1750 EX1 To config...

Page 110: ...displays the rear view of the expansion enclosure which is a bit different compared to the rear view of the 1750 511 model Figure 5 4 shows a 1750 511 model with two expansion 1750 EX1 models Figure...

Page 111: ...ne group can have a server enclosure and up to three expansion enclosures we call this group Loop 0 and the other group can have up to four expansion enclosures we call this group Loop 1 You can attac...

Page 112: ...res with Fibre Channel cables 1 16 FC switch FC switch 1 16 FC switch FC switch 16 1 FC switch FC switch 16 1 FC switch FC switch Controller 0 16 1 FC switch FC switch 1 16 FC switch FC switch Control...

Page 113: ...es run on the DS6000 server enclosure and they support open systems and zSeries environments Copy Services has four interfaces a Web based interface DS Storage Manager a command line interface DS CLI...

Page 114: ...n detail in the next section You can manage the Copy Services functions through a command line interface DS CLI and a new Web based interface DS Storage Manager You also can manage the Copy Services f...

Page 115: ...the target volume it is read from the target volume If the backup data is not copied yet it is read from the source volume Write to the source volume When you write some data to the source volume at...

Page 116: ...me depending on the application It can be used for online backup testing of new applications or for creating a database for data mining purposes The copy looks exactly like the original source volume...

Page 117: ...y the changed tracks to the target volumes If some tracks on the target volumes are updated these tracks are overwritten by the corresponding tracks from the source volume If you want you can also iss...

Page 118: ...or extent can be FlashCopied to up to 12 target volumes or target extents as illustrated in Figure 6 4 Figure 6 4 Multiple Relationship FlashCopy Note If a FlashCopy source volume has more than one ta...

Page 119: ...host I O level not the application level subsequent write operations to the source volumes that are part of the Consistency Group During the temporary queueing Establish FlashCopy is completed The te...

Page 120: ...sistent copies The copies have power fail or crash level consistency This means that if you suddenly power off your server without stopping your applications and without destaging the data in the file...

Page 121: ...e this feature for data backup and disaster recovery Remote Mirror and Copy is an optional function To use it you must purchase the Remote Mirror and Copy 2244 function authorization model which is 22...

Page 122: ...Therefore the copy is normally fuzzy but you can make a consistent copy through synchronization called a go to sync operation After the synchronization you can issue FlashCopy at the secondary site t...

Page 123: ...you to choose your remote site location based on business needs and enables site separation to add protection from localized disasters A consistent and restartable copy of the data at the remote site...

Page 124: ...to reduce the time that is required to switch back to the local site after a planned or unplanned outage Figure 6 9 Global Mirror How Global Mirror works We explain how Global Mirror works in Figure...

Page 125: ...the C volumes The C volumes now contain the consistent copy of data Because the B volumes usually contain a fuzzy copy of the data from the local site not when doing the FlashCopy the C volumes are u...

Page 126: ...the B volumes the last consistent data in the B volume is moved to the C volumes Therefore at some time a part of consistent data is in the B volume and the other part of consistent data is in the C v...

Page 127: ...tro Mirror Synchronous PPRC Description Metro Mirror is a function for synchronous data copy at a distance Advantages There is no data loss and it allows for rapid recovery for distances up to 300 km...

Page 128: ...ions before assigning PPRC secondary volumes For example in an AIX environment AIX assigns specific IDs to each volume PVID PPRC secondary volumes have the same PVID as PPRC primary volumes AIX cannot...

Page 129: ...ration involving a log volume and a data volume 1 Write to log volume Data Record 2 is being updated 2 Update Data Record 2 on data volume 3 Write to log volume Data Record 2 update complete If the co...

Page 130: ...I O activity to a volume for a time period by putting the source volume into an extended long busy state This operation can be done across multiple LUNs or volumes and even across multiple storage uni...

Page 131: ...also not completed because the 2nd operation is not completed In this case the 1st update is included in the backup copy and the 2nd and 3rd updates are not included Therefore this case is consistent...

Page 132: ...y Service functions can be initiated over the following interfaces zSeries Host I O Interface DS Storage Manager Web based Interface DS Command Line Interface DS CLI DS open application programming in...

Page 133: ...r in your computers DS Storage Manager issues its commands to the Management Console via the Ethernet network The DS Storage Manager can be used for almost all functions for Copy Services The followin...

Page 134: ...e DS Open API through the IBM TotalStorage Common Information Model CIM agent a middleware application that provides a CIM compliant interface The DS Open API uses the CIM technology to manage proprie...

Page 135: ...0 DS8000 and ESS 800 at the same time The DS Storage Manager does not support the ESS 800 6 5 Future Plan According to the announcement letter IBM has issued a Statement of General Direction IBM inten...

Page 136: ...112 DS6000 Series Concepts and Architecture...

Page 137: ...ning and configuration In this part we present an overview of the planning and configuration necessary before installing your DS6000 The topics include Installation planning Configuration planning Log...

Page 138: ...114 DS6000 Series Concepts and Architecture...

Page 139: ...ew DS6000 in your environment Refer to the latest version of the IBM TotalStorage DS6000 Introduction and Planning Guide GC26 7679 for further details In this chapter we cover the following topics Gen...

Page 140: ...ordered will be assembled into an IBM TotalStorage 2101 200 System Rack by IBM manufacturing The feature code 0801 is used to indicate that the DS6000 series ordered will be shipped as an assembled e...

Page 141: ...ck Table 7 2 DS6800 dimensions and weight within a 2101 200 rack 1 Find out the floor load rating of the location where you plan to install the storage units 2 Determine whether the floor load rating...

Page 142: ...ary line cords to any storage or expansion enclosure in a DS6000 series The DS6000 series power inputs are single phase Table 7 4 DS6000 series input voltage requirements Two independent power outlets...

Page 143: ...ack do not overload the power outlets Always connect the storage server to a properly grounded outlet Always connect the rack power to at least two different power circuits or sources This reduces the...

Page 144: ...ation management through customer written applications Either way the DS Open API presents another option for managing storage units by complementing the use of the IBM TotalStorage DS Storage Manager...

Page 145: ...where you run the DS Storage Manager Gateway Provide the dotted decimal or symbolic name address of the gateway for example 9 123 123 123 or sanjosegate Subnet mask Provide the dotted decimal address...

Page 146: ...ady have a SAN infrastructure with enough unused ports to connect a DS6000 or you might need additional Fibre Channel switches If you have switches that operate at 1 Gbps you might consider replacing...

Page 147: ...cards you should also configure a FICON port on the second controller card to allow failover from one server to the other to work A FICON port on the DS6000 can be connected directly to a zSeries hos...

Page 148: ...IBM authorization acquired through the DS6800 feature numbers for example feature code 5001 OEL 5 TB unit must cover the physical capacity of the DS6800 system where system is defined as the base encl...

Page 149: ...planning This chapter discusses configuration planning considerations when implementing the DS6000 series in your environment The topics covered are Configuration planning DS6000 Management Console DS...

Page 150: ...e Mirror and Copy and Parallel Access Volumes See 8 3 DS6000 licensed functions on page 131 for a in depth discussion of the licensed functions Delivery requirements to receive delivery of the DS6000...

Page 151: ...connects to the appropriate DS6000 series Online configuration which provides real time configuration management support The ability to execute Copy Services functions The capability for configuratio...

Page 152: ...configuration actions on existing storage plexes and storage units at the time each action is initiated View and change communication settings for DS6000s Express configuration Express configuration p...

Page 153: ...a service contract with IBM The customer does problem determination replacement parts ordering and the actual replacement of the component Most components are customer replaceable units CRUs In the ev...

Page 154: ...e in Copy Services environment The DS CLI will communicate with the DS Management Console when performing copy services commands so the network connecting the management console to the DS6000 should b...

Page 155: ...nce 8 2 7 Simple Network Management Protocol SNMP The storage unit generates SNMP traps and supports a read only management information base MIB to allow monitoring by your network You have to decide...

Page 156: ...torage feature activation on page 137 Table 8 1 provides the feature codes for the operating environment licenses for the DS6000 series Table 8 1 Operating environment license feature codes Licensed f...

Page 157: ...n you order Point in Time Copy functions you specify the feature code that represents the physical capacity to authorize for the function Table 8 3 provides the feature codes for the Point in Time Cop...

Page 158: ...ysical capacity of the entire DS6000 series including the DS6800 and DS6000 expansion units For example if the total capacity of the DS6000 series is 30 TB then you will purchase an OEL feature for 30...

Page 159: ...scope from FB to CKD or CKD to FB lateral change or reduction in license scope from ALL to FB or CKD will be disruptive and will require an IML In this example the user decides to FlashCopy 10 TB of...

Page 160: ...Metro Mirror feature for both the primary DS6000 and secondary DS6000 user data plus FC data upto 20 TB FB DS6000 with 45 TB disk capacity in total 45 TB FlashCopy authorization 45 TB of OEL Licensed...

Page 161: ...responsibility The client can manage and activate functions through the IBM Disk Storage Feature Activation DSFA Web site at http www ibm com storage dsfa Management refers to the use of the IBM Disk...

Page 162: ...l 1750 511 you can install up to 16 DDMs in the server enclosure The minimum storage capability with 8 DDMs is 584 GB 73 GB x 8 DDMs The maximum storage capability with 16 DDMs for the DS6800 model is...

Page 163: ...ce adapter chipset inside controller 1 Expansion enclosure 3 Additional enclosures to a maximum of four per loop 1 16 1 16 1 16 Additional enclosures to a maximum of four per loop Expansion enclosure...

Page 164: ...the allocation unit is an extent of this size RAID type RAID type DDM capacity Array Format Extents Binary GB Decimal GB 3390 3 devices 3390 9 devices 32760 cyl devices 65520 cyl devices RAID10 73GB R...

Page 165: ...drives and the DS6000 will automatically assign four spare disks two spare disks for each loop After that if you add more expansion enclosures you can configure RAID arrays and the DS6000 will not ass...

Page 166: ...osure has two spare disks Expansion enclosures 2 and 3 don t have spare disks Figure 8 10 Sparing example 1 RAID 5 Sparing Example 1 RAID 5 All same capacity same rpm Assumes all devices same capacity...

Page 167: ...t installation you also configure four spare disks in expansion enclosures 2 and 3 A 300 GB DDM cannot be spared onto a 146 GB DDM therefore 300 GB arrays need 300 GB spare disks for RAID configuratio...

Page 168: ...pare disks in the array And even if the installation sequence is reversed 146 GB DDMs are installed into the first two enclosures and 73 GB DDMs are installed into the next two expansion enclosures yo...

Page 169: ...uestions you will be in a position to choose the appropriate tools and utilities such as standard operating system mirroring basic commands software packages remote copy technologies and migration app...

Page 170: ...ata migration backup or restore packages and database tools to migrate data An example of a data migration package is BRMS Backup Recovery and Media Services Tivoli Storage Manager TSM can also be use...

Page 171: ...mooth risk averse data migration to your DS6000 The advantages of these migration appliances are Facilitated by custom migration tools Minimal customer involvement by IT staff except planning Minimal...

Page 172: ...only Monitoring performance Hot spot avoidance Preferred paths 8 6 1 Disk Magic An IBM representative or an IBM Business Partner may model your workload using Disk Magic before migrating to the DS6000...

Page 173: ...dle PAV devices are assigned to base devices that need more PAV devices to manage the workload WLM manages the PAV devices on an LSS group level Plan for the number of alias devices that your configur...

Page 174: ...nment to a server or controller card In other words preferred paths to a server avoid having to cross the PCI X connection Therefore there is a performance penalty if data from a logical volume manage...

Page 175: ...ager Logical configuration In this chapter the following topics are discussed Configuration hierarchy terminology and concepts Summary of the DS Storage Manager logical configuration steps Introducing...

Page 176: ...upported by a single Management Console Storage unit A storage unit also known as a storage facility is a single physical storage subsystem DS6000 Host attachment A host attachment is one or more host...

Page 177: ...e can access the same volume group The safest approach to this concept is to configure one host per volume group If shared access to the LUN is required for example for dual pathing or clustering then...

Page 178: ...ent pool must be of the same data format CKD or FB Each extent pool is associated with server 0 or server 1 Although it is possible to create extent pools with ranks of different drive capacities spee...

Page 179: ...eir server affinity at this time If you are using the custom configuration we recommend for user manageability reasons that the client associate the rank even numbers to server 0 and the rank odd numb...

Page 180: ...For CKD the minimum size is a 3390 Mod1 For FB the minimum size is 1 GB The maximum volume or LUN size is equal to the size of the extent pool with the following limitation 56 GB for CKD with the appr...

Page 181: ...volume group not two or more Assigning host attachments from multiple hosts systems even running different operating system types is allowed in the same volume group Figure 9 3 Diagram of the relatio...

Page 182: ...e group The maximum number of volume groups for the DS6000 is 1040 Address groups An address group is a group of FB or CKD LSSs An address group has up to 16 LSSs The DS6000 supports two address group...

Page 183: ...rouping of volumes LUNs for Copy Services and various other purposes Some of these purposes are explained here along with general considerations about LSSs in the DS6000 as follows The LSS or LCU dete...

Page 184: ...256 logical volumes 4096 logical volumes or addresses This means that if you had two address groups defined in the DS6000 you would have 8192 2 X 4096 8192 addresses or logical volumes As an example W...

Page 185: ...data format type 3 Determine the number and the size of logical volumes needed to fulfill the capacity requirements for zSeries and Open System hosts 4 Determine the number of rank types FB or CKD and...

Page 186: ...AID format is a 6 P S If you add up the parity chunks it equals one disk s worth of capacity If you choose RAID 10 then you would have two spares with no parity and a 3 X 3 2 spares configuration This...

Page 187: ...0 To connect to the DS6000 through the browser enter the URL of a PC or the MC you may have purchased The URL consists of the TCP IP address as shown in Figure 9 10 on page 164 or a fully qualified na...

Page 188: ...the selected fully qualified name for example MC_0 for the default MC as shown in Figure 9 11 Then bookmark it for ease of use When assigning the number we recommend that you make the last field of t...

Page 189: ...pre configure new configurations or modify existing configurations to be executed at a later time For example the Simulated Manager could be used to execute or modify changes at an off peak hour Real...

Page 190: ...use of PAVs from the DSFA Web site Simulated manager configuration You can begin using the simulated mode immediately after logging on to the DS Storage Manager However if you want to make your confi...

Page 191: ...he following users Novice users with little knowledge of storage concepts who want to quickly and easily set up and begin using storage Expert users who want to quickly configure a storage complex by...

Page 192: ...ions Passwords are limited to a total of 16 characters The user name can not be part of the password This entry will appear as asterisks You can grant user access privileges from this screen as well F...

Page 193: ...rmation center 9 2 3 Navigating the GUI Knowing what icons radio buttons and check boxes to click in the GUI will help you efficiently navigate your way through the configurator and successfully confi...

Page 194: ...se icon 4 to access the Info Center You get a help menu screen that prompts you for input on help topics 5 Click icon 5 to close the panels that open when you click the configuration menus in the My W...

Page 195: ...t all 3 Show filter row 4 Clear all filters 5 Edit sort 6 Clear all sorts The caret called out as item 7 is a simple ascending descending sort on a single column Clicking the pull down menu identified...

Page 196: ...configure your storage environment in the following order This does not mean that you have to follow this guide exactly You can get the same results by following a different order so long as you defin...

Page 197: ...n in Figure 9 24 Click Next and Finish in the verification step 9 3 2 Configuring the storage unit To create the storage unit expand the Manage Hardware section click Storage units 2 click Create from...

Page 198: ...k the Machine Type Model from the pull down list 2 Fill in the Nickname 3 Type in the Description 4 Click the Select Storage Complex from the pull down and choose the storage complex on which you wish...

Page 199: ...path as shown in Figure 9 27 Figure 9 27 View of the Defined licensed function panel Fill in the fields shown in Figure 9 27 as follows The number of licensed TB for the Operation environment The quan...

Page 200: ...r the appropriate information and click OK 9 3 3 Configuring the logical host systems To create a logical host for the storage unit that you just created click Host Systems as shown in Figure 9 29 You...

Page 201: ...ge Complex action pull down and highlight the storage complex on which you wish to configure click Create and Go The screen will advance to the General host information panel shown in Figure 9 31 Figu...

Page 202: ...host ports panel will update the new information as shown in Figure 9 33 Figure 9 33 View of Define host ports panel with updated host information Note Selecting Group ports to share a common set of...

Page 203: ...will advance to the Select storage units panel shown in Figure 9 34 Figure 9 34 View of the Select storage unit panel Highlight the Available storage units that you wish click Add and Next The screen...

Page 204: ...lid storage unit I O ports under the This host attachment can login to field 3 Click Apply assignment and OK 4 Verify and click Finish 9 3 4 Creating arrays from array sites Under Configure Storage cl...

Page 205: ...e an 8 disk array that two 4 disk array sites will be placed together to form an eight disk array Physical disks from any array site could be placed through a predetermined algorithm into the array It...

Page 206: ...e on either storage type FB or CKD as shown in Figure 9 38 Figure 9 38 View of the Add arrays to ranks panel with FB selected Click Next to bring up the Verification panel then click Finish to configu...

Page 207: ...ew of creating custom arrays from four disk array sites At this point you can select from the list of four disk array sites to put together to make an eight disk array If you click Next the second arr...

Page 208: ...ting extent pools To create extent pools expand the Configure Storage section click Extent pools click Create from the Select Action pull down and click Go Follow the panel directions with each advanc...

Page 209: ...w of Define properties panel Click Next and Finish 9 3 6 Creating FB volumes from extents Under Simulated Manager expand the Open systems section and click Volumes Click Create from the Select Action...

Page 210: ...panel To determine the quantity and size of the volumes use the calculators to determine the max size versus quantity as shown in Figure 9 44 Figure 9 44 The Define volume properties panel It is here...

Page 211: ...s of creating the volumes 9 3 7 Creating volume groups Under Simulated Manager Open Systems perform the following steps to configure the volume groups 1 Click Volume Groups 2 Click Create 3 Click Go F...

Page 212: ...Architecture Figure 9 46 The Define volume group properties filled out Select the host attachment you wish to associate the volume group with See Figure 9 47 Figure 9 47 The Select host Attachments pa...

Page 213: ...ck the check box next to the desired volume group and click the Apply button 6 Click OK You can verify that the volume is now assigned to the desired host s volume group by performing the following st...

Page 214: ...anager zSeries perform the following steps 1 Click Volumes zSeries 2 Click the Select Action pull down and highlight Create 3 Click Go 4 In the Select Extent pool panel click the radio button next to...

Page 215: ...ck Next 13 Under the Verification panel click Finish 9 3 12 Using the Express Configuration Wizard After you have created and defined the storage complex unit arrays ranks and extent pools you can use...

Page 216: ...Simulated manager as shown in Figure 9 50 Figure 9 50 View of the Real time Manager panel 2 Click Storage units 3 Select the check box beside the storage unit name as shown in Figure 9 51 Figure 9 51...

Page 217: ...is displayed as shown in Figure 9 52 9 4 Summary In this chapter we have discussed the configuration hierarchy terminology and concepts We have recommended an order and methodology for configuring th...

Page 218: ...194 DS6000 Series Concepts and Architecture...

Page 219: ...an be used to configure and maintain the DS6000 and DS8000 series It also describes how it can be used to manage Copy Services relationships In this chapter we describe Functionality Supported environ...

Page 220: ...establish dynamic FlashCopy and remote copy relationships on ESS 800 storage servers without using saved tasks Prior to the DS CLI the ESS Copy Services CLI generally did not allow a script to directl...

Page 221: ...torage disk ds6000 interop htm or http www ibm com servers storage disk ds8000 interop htm 10 4 Installation methods The DS CLI is supplied and installed via a CD that ships with the machine The insta...

Page 222: ...n one cluster of the ESS Interaction with this server is via either a Web Copy Services GUI interface or via a CLI interface To use the CLI the open systems host needs to have ESS Copy Services CLI so...

Page 223: ...ardware Management Console S HMC The S HMC then interprets the command and issues it to the relevant server in the relevant DS8000 using the redundant internal network that connects the S HMCs to the...

Page 224: ...ork Clearly a major benefit of this setup is that the internal network within the DS8000 has no single points of failure By using a second S HMC it is possible to create a completely redundant communi...

Page 225: ...SS CS CLI is required then both the DS CLI and the ESS CLI will have to be installed on the same open systems host as shown in Figure 10 4 on page 202 Each CLI installs into a separate directory Depen...

Page 226: ...command flow Open systems host Cluster 1 Infoserver Cluster 2 ESS 800 Infoserver Server 0 Server 1 DS8000 CLI interface CLI interface Storage HMC Network interface ESS CLI software ESS CLI software D...

Page 227: ...and the help screens If a userid is not a member of any group then it is automatically placed into the no_access group to prevent it from performing any functions The default userid supplied with an...

Page 228: ...pt ibm dscli chmod x listusers sh opt ibm dscli listusers sh opt ibm dscli cat userlist txt Name Group admin admin opt ibm dscli Interactive mode In the interactive mode the user starts the DS CLI pro...

Page 229: ...ese include the ability to input variables from a file and send output to a file The DS CLI commands are also designed to be case insensitive This means commands can be entered in either upper lower o...

Page 230: ...correct password then a failure reason and a return code will be presented Standard techniques to collect and analyze return codes can be used The return codes used by the DS CLI are shown in Table 10...

Page 231: ...CMUN00018E lsflash Unable to connect to the management console server exit status of dscli 3 A connection error occurred Try pinging 10 0 0 1 If this fails call network support on 555 1001 C Program F...

Page 232: ...SS 800 They will not work with the ESS F20 The storage management commands within the DS CLI also will not work with ESS 800 This means that customers who currently have a mix of 800s and F20s will ha...

Page 233: ...he contents of each saved task and write them to a file 2 Review server scripts that perform task set up and execute saved ESS tasks Phase two Perform Having performed the review the scripts need to b...

Page 234: ...4 IBM ESSCLI 2 4 0 Task Name Type Status H_Epath_test16 PPRCEstablishPaths NotRunning H_Epath_test17 PPRCEstablishPaths NotRunning Brocade_pr_lss10 PPRCEstablishPair NotRunning Brocade_pr_lss11 PPRCEs...

Page 235: ...esscli show task s 10 0 0 1 u csadmin p passw0rd d name Flash10041005 Wed Nov 24 10 37 17 EST 2004 IBM ESSCLI 2 4 0 Taskname Flash10041005 Tasktype FCEstablish Options NoBackgroundCopy SourceServer 21...

Page 236: ...xact syntax TargetServer 2105 23953 N A We only need to use the dev once so this is redundant Source and Target vols 1004 1005 1004 1005 The volume numbers don t change We simply separate them with a...

Page 237: ...serid If you have DS CLI access to a DS offline configuration tool S HMC or DS Storage Management console then you can create an encrypted password file This will allow you to avoid specifying the pas...

Page 238: ...23953 hmc1 10 0 0 1 hmc2 127 0 0 1 Username must be specified if a password file is used username csadmin Password filename is the name of an encrypted password file This file is located at C Program...

Page 239: ...interpreter An example is shown in Example 10 18 Example 10 18 Establishing a FlashCopy with a single command anthony aixsrv opt ibm dscli dscli mkflash nocp 1004 1005 CMUC00137I mkflash FlashCopy pai...

Page 240: ...ided some important information about the DS CLI This new CLI allows considerable flexibility in how DS6000 and DS8000 series storage servers are configured and managed It also detailed how an existin...

Page 241: ...ed 217 Part 4 Implementation and management in the z OS environment In this part we discuss considerations for the DS6000 series when used in the z OS environment The topics include z OS software Data...

Page 242: ...218 DS6000 Series Concepts and Architecture...

Page 243: ...ilable in early 2005 Contact your IBM sales representative for more information about this tool and the benchmark testing that was done by the Tucson performance measurement lab Note that Disk Magic i...

Page 244: ...rated architecture based on standard building blocks as introduced by IBM with the Seascape architecture Over time all components became not only bigger in capacity and faster in speed but also more s...

Page 245: ...t task for a storage server than for a host server Perhaps it became this way with the evolving complexity of storage servers 11 2 Where do we start The IBM Enterprise Storage Server 2105 ESS already...

Page 246: ...al to exceed the throughput capabilities of other vendor s FICON ports When not properly configured for example with four FICON ports in the same HA these powerful FICON ports have the potential to sa...

Page 247: ...AL shortcomings There are some shortcomings with plain FC AL The most obvious ones are As the term arbitration implies each individual disk within an FC AL loop competes with the other disks to get o...

Page 248: ...Channel disk subsystem backend as displayed in Figure 11 3 on page 225 Note that each DA port can concurrently send and receive data Processor Adapter Adapter Memory Processor Adapter Adapter To host...

Page 249: ...ations due to two simultaneous operations from each DA to allow for two concurrent read operations and two concurrent write operations at the same time Despite the superior performance don t forget th...

Page 250: ...ssors from this task The actual throughput and performance of a DA is not only determined by the 2 Gbps ports and used hardware but also by the firmware efficiency 11 3 3 New four port host adapters B...

Page 251: ...uster or rather a dual server design which is also found in the ESS and DS8000 series This offers an enterprise class level of availability and functionality in a space efficient modular design at a l...

Page 252: ...Figure 11 8 on page 229 shows a DS6800 with one DS6000 expansion enclosure Processor Adapter Adapter Adapter Adapter To host servers Storage server 2 Gbps Fibre Channel ports Card 1 Controller chipset...

Page 253: ...its local HA and its Fibre Channel ports This introduces the concept of a preferred path When a volume has an affinity for example to server 0 and is accessed through a port in the HA of server 0 then...

Page 254: ...he disk space requirements of the application should be well understood 11 4 1 Workload characteristics The answers to questions like how many host connections do I need how much cache do I need and t...

Page 255: ...ial files but there are also benefits for random access DS6000 logical volumes are composed of extents An extent pool is a logical construct to manage a set of extents One or more ranks with the same...

Page 256: ...t striping wherever possible to distribute the read and write I O access patterns across the physical resources of the DS6000 Note The stripe size has to be large enough to keep sequential data relati...

Page 257: ...outing for optimum performance Multipath load balancing of data flow prevents a single path from becoming overloaded causing input output congestion that occurs when many I O operations are directed t...

Page 258: ...to connect to zSeries servers or as FCP ports to connect to Fibre Channel attached open systems hosts The example in Figure 11 11 shows only two FICON Express channels But just two FICON Express chan...

Page 259: ...to follow could be to propose how many ESS 800s might be consolidated into a DS6000 model From that you can derive the number of ESS 750s ESS F20s and ESS E20s which can collapse into a DS6000 The old...

Page 260: ...mean FICON ports in the disk storage servers Coming from FICON Express channels you should then keep a minimum of four FICON ports You might consider using 25 fewer FICON ports in the DS6000 than the...

Page 261: ...st impression At GA the IBM internal tool Disk Magic helps to model configurations based on customer workload data An IBM representative can contact support personnel who will use Disk Magic to config...

Page 262: ...ent pools to ease the management effort of such a configuration But you can still assign multiple storage groups when you are not concerned about the placement of less active volumes Figure 11 12 Exte...

Page 263: ...umes there is no straightforward approach to place certain volumes into certain ranks For example when you create the first 20 DB2 logging volumes they would be allocated in a consecutive fashion in t...

Page 264: ...aller high performance extent pools through discrete SMS storage groups for each DS6000 For example in a dual logging database environment allow assignment of extent pool0 to the first logging volume...

Page 265: ...of up to 67 2 TB Just the base enclosure provides up to 4 8 TB of physical storage capacity with 16 DDMs and 300 GB per DDM The small and fast DS6000 with its rich functionality and compatibility wit...

Page 266: ...242 DS6000 Series Concepts and Architecture...

Page 267: ...z VM z VSE and Transaction Processing Facility TPF software enhancements that support the DS6000 series The enhancements include Scalability support Large volume support Hardware configuration defini...

Page 268: ...enhanced by way of a Small Programming Enhancement SPE Hosts with operating system software levels prior to z OS 1 4 are not supported zLinux will only recognize the DS6000 as a 2105 device The PSP in...

Page 269: ...stay within this limit Today with the IBM 2105 volumes 32 760 cylinders are supported This gives us the capability to remain within the 64K limit But as today s storage facilities tend to expand to e...

Page 270: ...3390 3390A 3390B device types that can be defined to this control unit The definition of the 1750 control unit type will allow this storage facility to be uniquely reflected in the Hardware Configura...

Page 271: ...specified Figure 12 1 SETCACHE options All other parameters should be accepted as they are today on the IBM 2105 For example setting device caching ON is accepted but has no affect on the subsystem 1...

Page 272: ...will then identify preferred paths to the IOS I Os will be directed to preferred paths to avoid crossing the PCI X connection The only time this will not be honored is when there are no preferred pat...

Page 273: ...g system that supplies a virtual machine to each logged on user The DS6000 will be supported on z VM 4 4 and higher The PSP information can be found at http www 1 ibm com servers resourcelink svc03100...

Page 274: ...ne transaction processing It is used by industries demanding large transaction volumes such as airlines and banks The DS6000 will be supported on TPF 4 1 and higher The PSP information can be found at...

Page 275: ...sk storage server The focus is on z OS environments The following topics are covered from a planning standpoint Data migration objectives in z OS environments Data migration based on physical migratio...

Page 276: ...ls 8 192 logical volumes This is twice as much as what an ESS 2105 can configure for a z OS based configuration This suggests consolidation of multiple control units into a single DS6800 disk storage...

Page 277: ...3390 3 Or to put it differently we see on a single volume as many concurrent I Os as we see on nine 3390 3 volumes Despite the PAV support it still might be necessary to balance disk storage activiti...

Page 278: ...om 3380 to 3390 Utilizing physical full volume operations is possible through the following software microcode and hardware based functions Software based DFSMSdss TDMF FDRPAS Software and hardware ba...

Page 279: ...pen in a controlled fashion This lengthens the migration period so if possible other approaches might be considered Both software products are usually associated with fees or service based fees except...

Page 280: ...sure Agent tasks which need to be installed in the customer systems An FDRPAS master task runs on the Piper CPU which coordinates all activities Monitor tasks are required on the customer s systems to...

Page 281: ...ther alternative is z OS Global Mirror XRC XRC is an asynchronous solution that has a mode for disaster recovery solutions as well as a particular migration mode of SESSIONTYPE MIGRATE This mode does...

Page 282: ...Ms because there is no potential bottleneck when coming from ESCON This would take about one day to replicate all data from the two F20s to the DS6800 provided there is not too much application write...

Page 283: ...n the TSO command support for cascaded volumes is available Otherwise use a combination of TSO commands and ICKDSF for just defining the cascaded bit when setting up Metro Mirror between the intermedi...

Page 284: ...r side does not impact the application write I Os due to its asynchronous data replication but the drawback here is that it does not guarantee data consistency at the receiving site Before switching f...

Page 285: ...RAGE CONTROL DESCRIPTOR E8 DEVICE DESCRIPTOR 0A ADDITIONAL DEVICE INFORMATION 4A000035 ICK04030I DEVICE IS A PEER TO PEER REMOTE COPY VOLUME QUERY REMOTE COPY VOLUME PRIMARY SECONDARY SSID CCA SSID CC...

Page 286: ...y some REXX based procedure might have to scan through the ICKDSF SYSPRINT output which is directed to a data set ICKDSF asks for a JCL DD statement for each single volume to query when the volume is...

Page 287: ...microcode based mirroring capabilities require the right hardware as source and target disk servers For completeness it is pointed out that Global Mirror is also an option to migrate data from an ESS...

Page 288: ...YSIN DD COPY DS INC EXCLUDE SYS1 VTOCIX SYS1 VVDS LIDD IN001 IN002 IN003 DELETE CATALOG ALLDATA ALLX WAIT 0 0 ADMIN OPT 3 CANCELERROR Example 13 5 depicts how to migrate all data sets from certain vol...

Page 289: ...ction This approach assumes that the old and new disk servers can be configured concurrently to the concerned host servers 13 3 2 Data migration within the system managed storage environment System ma...

Page 290: ...gh the ISMF panel sequence to make the SMS volume status complete and permanent Example 13 6 is the first ISMF Storage Group Application panel where you choose the actual Source Control Data Set SCDS...

Page 291: ...rial Number List Specify a Single Volume in Prefix or Range of Volumes Prefix From To Suffix Type ______ ______ ______ _____ _ XC 6510 6514 X Use ENTER to Perform Selection Use HELP Command for Help U...

Page 292: ...isnew MZBCVS2 disnew DISALL DISNEW QUIALL QUINEW SYS GROUP sysplex minus systems in the sysplex explicitly defined in the SCDS After pressing Enter and PF3 to validate and perform the SMS volume statu...

Page 293: ...Copy Pool Backup only 4 Delete Delete Volumes from Volume Serial Number List Specify a Single Volume in Prefix or Range of Volumes Prefix From To Suffix Type ______ ______ ______ _____ _ DB2 001 300 X...

Page 294: ...based on software utilities Problems encountered when not using an allocation manager like system managed storage are less flexibility when using esoteric unit names or complex and time consuming tas...

Page 295: ...e to a different device type Another logical migration approach is possible through the CP PTAPE command PTAPE dumps spool files to tape and then re loads these files back onto the new disk storage La...

Page 296: ...MF or FDRPAS is available and the customer is used to managing volume migration using these software tools then it is a likely approach to use these tools for a larger scale migration as well In other...

Page 297: ...Implementation and management in the open systems environment In this part we discuss considerations for the DS6000 series when used in an open systems environment The topics include Open systems sup...

Page 298: ...274 DS6000 Series Concepts and Architecture...

Page 299: ...escribe how the DS6000 fits into your open systems environment In particular we discuss The extent of the open systems support Where to find detailed and accurate information Major changes from the ES...

Page 300: ...Where to look for updated and detailed information on page 277 14 1 1 Supported operating systems and servers Table 14 1 provides an overview of the open system platforms operating systems and high a...

Page 301: ...e output of the tool will be ambiguous You will be shown a list of all supported HBAs together with the required firmware and device driver levels for your combination Furthermore you can select a det...

Page 302: ...r their HBAs requires registration Platform and operating system vendors pages The platform and operating system vendors also provide lots of support information to their customers Go there for genera...

Page 303: ...ation about boot support Refer to The DS6000 Interoperability Matrix on page 277 The DS6000 Host Systems Attachment Guide SC26 7680 helps you with the procedures necessary to set up your host in order...

Page 304: ...is accessed from a port that is located on the other server The request for the logical volume and the data would have to be transferred across the bridge interface that connects both servers These t...

Page 305: ...thing MPIO see IBM AIX on page 303 IBM OS 400 V5R3 multipath support see Appendix B Using the DS6000 with iSeries on page 329 In addition there are third party multipathing solutions such as Veritas D...

Page 306: ...r all supported host operating systems The DS CLI can be used in two modes Single command mode You invoke the DS CLI program in an operating system shell or command prompt and pass the command it is t...

Page 307: ...ise policies for managing host application database and filesystem data As a component of the IBM TotalStorage Productivity Center Multiple Device Manager is designed to reduce the complexity of manag...

Page 308: ...bed in the following sections 14 5 1 Device Manager The Device Manager DM builds on the IBM Director technology It uses the Service Level Protocol SLP to discover supported storage systems on the SAN...

Page 309: ...re 14 3 shows an example of a DM view Figure 14 3 Sample Device Manager view In summary the Device Manager is responsible for Discovery of supported devices SMI S compliant Data collection asset avail...

Page 310: ...individual device You can view performance data from the performance manager database in both graphical and tabular forms This is shown in Figure 14 4 Figure 14 4 Example screenshot of TPC for Disk T...

Page 311: ...nd line interface 14 6 Global Mirror Utility The DS6000 Global Mirror Utility GMU is a standalone tool to provide a management layer for IBM TotalStorage Global Mirror failover and failback FO FB It p...

Page 312: ...ified volumes and issue commands to bring the volumes back into sync It offers a tool to execute eRCMF configuration checks This check is intended to verify that the eRCMF configuration matches the ph...

Page 313: ...Chapter 15 Data migration in the open systems environment In this chapter we discuss important concepts for the migration of data to the new DS6000 Data migration considerations Data migration and con...

Page 314: ...solidation Very often the goal of a consolidation effort is a combination of more than one of these types There are many different methods for data migration To decide on what is best in your case gat...

Page 315: ...or ways to propagate configuration changes throughout the cluster 15 2 Comparison of migration methods There are numerous methods that can be used to migrate data from one storage system to another We...

Page 316: ...annot preserve advanced metadata such as access control lists or permissions Copy raw devices For raw data there are tools that allow you to read and write disk devices directly such as dd for UNIX Th...

Page 317: ...w storage cut over time Once the initial copy is complete you keep it up to date with incremental rsync runs for instance once a day during off peak hours At the cut over time you stop the application...

Page 318: ...Restore Every serious IT operation will have ways to back up and restore data They can be used for data migration We list this method here because it shares the common advantages and disadvantages wit...

Page 319: ...refore be used with only minimum system administrator attention They also do not add any additional CPU load to the host systems and they don t require the host system to be connected to both storage...

Page 320: ...t be mirrored directly to a DS6000 Combination of Metro Mirror and Global Copy A cascading Metro Mirror and Global Copy solution is useful in two cases The source system is already mirrored for disast...

Page 321: ...lidation of small volumes into larger ones because it is not aware of the data structure Since Piper has to be in the data path between the host and the source storage system it can only be used to mi...

Page 322: ...t and accurate data migration IBM provides technical specialists at your location to plan and migrate your data to your DS6000 disk system This migration is accomplished using either native operating...

Page 323: ...tems specifics In this appendix we describe the particular issues of some operating systems with respect to the attachment to a DS6000 The following subjects are covered Planning considerations Common...

Page 324: ...n we raise the questions you will have to ask and the things you have to consider before you start We don t cover all the items in detail because this is not an implementation book For a more detailed...

Page 325: ...to use These tools offer a quick way to tell whether a system is I O bound iostat sar System Activity Report vmstat Virtual Memory Statistics IOSTAT The base tool for evaluating I O performance of di...

Page 326: ...epending on the application layout this report could lead to the conclusion that the I O load of this system is unbalanced Some disks get a lot more I O request than others A consequence of this could...

Page 327: ...spent executing operating system calls idle Time the system was idle with no outstanding I O requests The implementation of the sar command is different for the various UNIX variants However the outp...

Page 328: ...S under AIX there was a set of Host Attachment scripts which could be downloaded from the Web For the DS6000 using AIX SDD 1 6 0 0 these scripts have been consolidated into one called FCP Host Attachm...

Page 329: ...ronments in IBM products SDD resides in the host system with the native disk device driver and manages redundant connections between the host server and the DS6000 SDD is available for AIX 5 1 5 2 and...

Page 330: ...th The storage device paths are managed to provide high availability and load balancing of storage I O MPIO is part of the base kernel and is available for AIX 5 2 and AIX 5 3 The base functionality o...

Page 331: ...pg Enabled hdisk0 scsi0 Enabled hdisk1 scsi0 Enabled hdisk2 scsi0 Enabled hdisk3 scsi7 Enabled hdisk4 scsi7 Missing hdisk9 fscsi0 Missing hdisk10 fscsi0 Missing hdisk11 fscsi0 Missing hdisk12 fscsi0 M...

Page 332: ...stributed across separate disks With the introduction of SAN technology LVM mirroring can even provide protection against a site failure Using long wave Fibre Channel connections a mirror can be stret...

Page 333: ...serialization of write accesses It improves performance for many environments particularly commercial relational databases In many cases the database performance achieved using Concurrent I O with JFS...

Page 334: ...ols on page 301 for more information filemon The filemon command monitors the performance of the file system and reports the I O activity with regard to files virtual memory segments logical volumes a...

Page 335: ...MPIO FC 1750 0 87 672 776 86 1 dev hdisk81 IBM MPIO FC 1750 0 86 392 960 80 4 dev hdisk63 IBM MPIO FC 1750 0 86 328 776 65 6 dev hdisk83 IBM MPIO FC 1750 0 86 528 624 68 5 dev hdisk69 IBM MPIO FC 1750...

Page 336: ...pport Depending on the target group the distributions differ largely in the kind of support that is available Linux is available for almost all computer architectures Linux is rapidly changing All the...

Page 337: ...o an IBM eServer BladeCenter running Linux It can be downloaded from http publib b boulder ibm com abstracts sg246261 html Linux with zSeries and ESS Essentials The redbook Linux with zSeries and ESS...

Page 338: ...CSI disks The Linux SCSI I O subsystem has some peculiarities that are important enough to be described here even if they show up in some of the publications listed in the previous section Some Linux...

Page 339: ...on of special device files for the 17th SCSI disk and its first three partitions Example A 10 Create new special device files for SCSI disks mknod dev sdq b 65 0 mknod dev sdq1 b 65 1 mknod dev sdq2 b...

Page 340: ...contains a list of the supported kernels The version of the Linux Logical Volume Manager that comes with all current Linux distributions does not support its physical volumes being placed on SDD vpat...

Page 341: ...before the built in parallel SCSI controller that drives the system disks This will lead to shifted special device file names of the system disk and can result in the system being unable to boot prop...

Page 342: ...st be issued for each installed HBA with the adapter instance being the SCSI instance number of the HBA After the FC HBAs rescan the fabric you can make the new devices available to the system with th...

Page 343: ...a device behind it When assigning volumes to a Linux host with QLogic FC HBAs make sure LUNs start at 0 and are in consecutive order Otherwise the LUNs after a gap will not be discovered by the host...

Page 344: ...0 Vendor IBM Model 32P0032a S320 1 Rev 1 Type Processor ANSI SCSI revision 02 Host scsi2 Channel 00 Id 00 Lun 00 Vendor IBM Model 1750511 Rev 545 Type Direct Access ANSI SCSI revision 03 Host scsi2 Ch...

Page 345: ...SCSI devices They are called the sg tools They talk to the SCSI devices through the generic SCSI layer which is represented by special device files dev sg0 dev sg0 and so on By default SLES 8 provide...

Page 346: ...AL_MACHINE SYSTEM CurrentControlSet Services Disk TimeOutValue The value has the data type REG DWORD and should be set to 0x0000003c hexadecimal 60 decimal SDD for Windows An important task with a Win...

Page 347: ...ID arrays Windows Server 2003 components that work with VDS include the Disk Management MMC snap in the DiskPart command line tool and the DiskRAID command line tool which is available in the Windows...

Page 348: ...istent point in time copies of data known as shadow copies It integrates IBM TotalStorage FlashCopy to produce consistent shadow copies while also coordinating with business applications file system s...

Page 349: ...the SCSI standard for vendor s private use OpenVMS does not identify a Fibre Channel disk by its path or SCSI target LUN like other operating systems It relies on the UDID Although OpenVMS uses the W...

Page 350: ...e with a minimum size as a dummy volume for usage as the CCL Multiple OpenVMS hosts even in different clusters that access the same storage system can share the same volume as LUN 0 because there will...

Page 351: ...tp research compaq com wrl DECarchives DTJ DTJ301 DTJ301SC TXT The DS6000 provides volumes as SCSI 3 devices and thus does not implement a forced error indicator It also does not support the READL and...

Page 352: ...328 DS6000 Series Concepts and Architecture...

Page 353: ...Using the DS6000 with iSeries In this appendix the following topics are discussed Supported environment Logical volume sizes Protected versus unprotected volumes Multipath Adding units to OS 400 confi...

Page 354: ...age_hw html Software The iSeries must be running V5R2 or V5R3 i5 OS of OS 400 In addition at the time of writing the following PTFs are required V5R2 MF33327 MF33301 MF33469 MF33302 SI14711 and SI1475...

Page 355: ...therwise when mirroring is started to mirror the LSU to the DS6000 LUN OS 400 will attempt to mirror all unprotected volumes Changing LUN protection It is not possible to simply change a volume from p...

Page 356: ...be done either using the green screen interface with Dedicated Service Tools DST or System Service Tools SST or with the iSeries Navigator GUI The following example shows how to add a logical volume...

Page 357: ...to panel Specify the ASP number next to the desired units Here we have specified ASP1 the System ASP Press Enter Work with Disk Units Select one of the following 1 Display disk configuration 2 Work wi...

Page 358: ...5 shows the initial panel Specify ASPs to Add Units to Specify the ASP to add each unit to Specify Serial Resource ASP Number Type Model Capacity Name 21 662C5 4326 050 35165 DD124 21 54782 4326 050 3...

Page 359: ...nitial panel 2 Expand the iSeries to which you wish to add the logical volume and sign on to that server as shown in Figure B 7 Figure B 7 iSeries Navigator Signon to iSeries panel 3 Expand Configurat...

Page 360: ...8 iSeries Navigator Disk Units 4 You will be asked to sign on to SST as shown in Figure B 9 Enter your Service tools ID and password and press OK Figure B 9 SST Signon 5 Right click Disk Pools and sel...

Page 361: ...Appendix B Using the DS6000 with iSeries 337 Figure B 10 Create a new disk pool 6 The New Disk Pool wizard appears as shown in Figure B 11 Click Next Figure B 11 New disk pool welcome...

Page 362: ...es the type of logical volume you are adding If you leave it unchecked you will see all available disks Select OK to continue Figure B 12 Defining a new disk pool 8 A confirmation panel like that show...

Page 363: ...st of non configured units similar to that shown in Figure B 15 will appear Highlight the disks you want to add to the disk pool and click Add Figure B 15 Choose the disks to add to the Disk Pool 11 A...

Page 364: ...Click Finish to add the disks to the Disk Pool Figure B 17 New Disk Pool Summary 13 Take note of and respond to any message dialogs which appear After taking action on any messages the New Disk Pool...

Page 365: ...gure B 19 Figure B 19 Disks added successfully to Disk Pool 15 The new Disk Pool can be seen on iSeries Navigator Disk Pools in Figure B 20 Figure B 20 New Disk Pool shown on iSeries Navigator 16 To s...

Page 366: ...tical to OS 400 due to the single level storage architecture Multipath is not available for iSeries internal disk units but the likelihood of path failure is much less with internal drives This is bec...

Page 367: ...re fifteen single points of failure excluding the iSeries itself and the DS6000 storage facility Failure points 9 12 will not be present if you do not use an Inter Switch Link ISL to extend your SAN A...

Page 368: ...haring Configuring multipath iSeries has two I O adapters that support DS6000 2766 2 Gigabit Fibre Channel Disk Controller PCI 2787 2 Gigabit Fibre Channel Disk Controller PCI X Both can be used for m...

Page 369: ...es is also assigned to a Fibre Channel I O adapter on a different BUS in the second iSeries I O tower or rack via a host adapter in the upper controller card Adding multipath volumes to iSeries using...

Page 370: ...ce ASP Number Type Model Capacity Name 21 662C5 4326 050 35165 DD124 21 54782 4326 050 35165 DD136 1 75 1118707 1750 A85 35165 DMP135 F3 Exit F5 Refresh F11 Display disk configuration capacity F12 Can...

Page 371: ...s appear as DMPxxx Highlight the disks you want to add to the disk pool and click Add Figure B 27 Adding a multipath volume The remaining steps are identical to those in Adding volumes to an Independe...

Page 372: ...een on iSeries Navigator Disk Pools in Figure B 28 Figure B 28 New Disk Pool shown on iSeries Navigator To see the logical volume as shown in Figure B 29 expand Configuration and Service Hardware Disk...

Page 373: ...using the standard disk panels in iSeries Navigator only a single the initial path is shown The following steps show how to see the additional paths To see the number of paths available for a logical...

Page 374: ...s Concepts and Architecture To see the other connections to a logical unit right click the unit and select Properties as shown in Figure B 31 on page 350 Figure B 31 Selecting properties for a multipa...

Page 375: ...DS6000 with iSeries 351 You will then see the General Properties tab for the selected unit as in Figure B 32 The first path is shown as Device 1 in the box labelled Storage Figure B 32 Multipath logi...

Page 376: ...other node The system enforces the following rules when you use multipath disk units in a multiple system environment If you move an IOP with a multipath connection to a different logical partition yo...

Page 377: ...may lead to some small performance degradation for large workloads This is because a longer path will be used when accessing logical volumes across the midplane in the DS6000 Preferred path would acce...

Page 378: ...gularly or perhaps FlashCopy target volumes which could use this capacity but not impact on the I O per sec on those arrays For very high write environments you may also consider using RAID 10 which o...

Page 379: ...reduce I O queues and wait times by allowing OS 400 to support more parallel I Os From the values in Table B 2 you can calculate the number of iSeries Fibre Channel adapters for your required iSeries...

Page 380: ...rally iSeries has a relatively high I O rate while that of other servers may be lower often below one I O per GB per second As an example a Windows file server with a large data capacity may normally...

Page 381: ...erted back to a protected model and therefore will be a candidate for all future OS 400 mirroring whether you want this or not See Table B 1 on page 330 for DS6000 logical volume models and types Metr...

Page 382: ...to use native OS 400 functions to migrate data from existing disks to the DS6000 whether the existing disks are internal or external When you assign the new DS6000 logical volumes to the iSeries initi...

Page 383: ...the balancing function completes or is ended If the balance function is run for a few hours and then stopped it will continue from where it left off when the balance function restarts This allows the...

Page 384: ...for other functions such as test development and Lotus Notes When backups are to be done the FlashCopy target can be attached to the partition without affecting the rest of the users Or perhaps more...

Page 385: ...Copy and Remote Mirror functions with iSeries for practical reasons many customers will choose not to use basic FlashCopy and Remote Mirror functions due to the LSU restrictions discussed previously I...

Page 386: ...also supports earlier iSeries servers The DS6000 requires the following iSeries I O adapters to attach directly to an iSeries or i5 Linux partition 0612 Linux Direct Attach PCI 0626 Linux Direct Atta...

Page 387: ...Services for TotalStorage disk systems IBM Implementation Services for TotalStorage Copy Functions IBM Implementation Services for TotalStorage Command Line Interface IBM Migration Services for eServ...

Page 388: ...Copy Functions This service is designed to assist in the planning implementation and testing of the IBM TotalStorage advanced copy functions Point in Time Copy and remote mirroring solutions For more...

Page 389: ...on page 297 For the latest information about this service visit the following Web site http www ibm com servers storage services featured hardware_assist html For more information about IBM Migration...

Page 390: ...featured microsoft_application_environment html GDSSolution IBM eServer iSeries Copy Services For the iSeries environment IBM offers a special toolkit which allows you to use the advanced Copy Servic...

Page 391: ...Appendix C Service and support offerings 367 Figure 15 9 Example of the Supported Product List SPL from the IBM Support Line...

Page 392: ...368 DS6000 Series Concepts and Architecture...

Page 393: ...olutions Guide SG24 6547 iSeries and IBM TotalStorage A Guide to Implementing External Disk on eServer i5 SG24 7120 IBM TotalStorage Productivity Center V2 3 Getting Started SG24 6490 Managing Disk Su...

Page 394: ...for the DS6000 http www ibm com servers storage support disk 1750 html Supported servers for the DS6000 http www storage ibm com hardsoft products DS6000 supserver htm The interoperability matrix htt...

Page 395: ...m How to get IBM Redbooks You can search for view or download Redbooks Redpapers Hints and Tips draft publications and Additional materials as well as order hardcopy Redbooks or CD ROMs at this Web si...

Page 396: ...372 DS6000 Series Concepts and Architecture...

Page 397: ...ler architecture 25 failover and failback 46 Copy Services 89 definitions 90 interfaces 108 iSeries 360 iSeries toolkit 361 management 130 CRU 61 CRU endpoint indicator 60 CUIR 52 D DA 33 ports indica...

Page 398: ...37 Global Copy 90 Global Mirror 90 hardware components 23 hardware overview 6 health indicators 35 host attachment 78 Host Systems Attachment Guide 300 information lifecycle management 4 infrastructur...

Page 399: ...data sets 10 inband commands 11 97 incremental 11 multiple relationship 10 options 92 persistent 97 floating spares 54 floor and space requirements 116 front panel 37 FRU 61 G GDPS 365 GDS for MSCS 3...

Page 400: ...hannel adapters 355 OS 400 data migration 358 OS 400 mirroring 357 preferred path for DS6000 353 protected versus unprotected volumes 330 recommended number of ranks 355 Remote Mirror and Copy 360 sha...

Page 401: ...150 UNIX tools 301 z OS 233 DS6000 size 235 potential 234 statistics 247 Persistent FlashCopy 97 PFA 13 55 physical migration zSeries 254 DFSMSdss 254 Piper 255 297 Point in time Copy see PTC port Eth...

Page 402: ...tion indicator 60 system service 57 card 42 T TPC 282 for disk 285 for replication 287 TPF 250 U UNIX iostat 301 performance monitoring tools 301 SAR vmstat 303 V VDS 323 Virtual Disk Service see VDS...

Page 403: ...0 5 spine 0 475 0 873 250 459 pages DS6000 Series Concepts and Architecture DS6000 Series Concepts and Architecture DS6000 Series Concepts and Architecture DS6000 Series Concepts and Architecture...

Page 404: ...DS6000 Series Concepts and Architecture DS6000 Series Concepts and Architecture...

Page 405: ......

Page 406: ...6000 storage server series its architecture its logical design hardware design and components advanced functions performance features and specific characteristics The information contained in this boo...

Reviews:

No comments

Related manuals for System storage DS6000 Series

System Storage EXN2000

Brand: IBM Pages: 14

Brand: LaCie Pages: 22

Momentus ST320LM010

Brand: Seagate Pages: 33

OMNISTOR 4000f SERIES

Brand: Rackable Systems Pages: 112

Brand: Seagate Pages: 32

ST9160310AS - Momentus 5400.5 160 GB Hard Drive

Brand: Seagate Pages: 2

DLT-V4 EZ-Kit Series

Brand: Freecom Pages: 2

Tackle Caddy Console

Brand: RAILBLAZA Pages: 2

Brand: Facom Pages: 2

WD VelociRaptor WD1500BLFS

Brand: Western Digital Pages: 2

Brand: Clarke Pages: 6

Brand: XtendLan Pages: 24

Brand: Zte Pages: 46

Brand: ClearSpan Pages: 25

Brand: OLT Pages: 43

Brand: Festool Pages: 132

Brand: Black Box Pages: 20

Brand: HOLZMANN MASCHINEN Pages: 13

Brands by name

0 1 2 3 4 5 6 7 8 9 A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Popular brands

Load more brands