manualshive.com logo in svg

IBM Storwize V7000, Introduction And Implementation Manual, Page: 308 / 670

Share
Download
IBM Storwize V7000 Introduction And Implementation Manual Download Page 308

288

 

IBM Flex System V7000 Storage Node Introduction and Implementation Guide

At this point, we go back to our Microsoft Windows 2008 server and check that the disk has 
disappeared from the Server Manager Disk Administrator, as verified in Figure 7-5. We also 
use the Rescan Disks option to confirm it.

Figure 7-5   Windows server with disk from DS4300 unmapped - rescan disks

Now we are ready to start the importing of the disk to IBM Flex System V7000 Storage Node.

Summary of Contents for Storwize V7000

Page 1: ...ode Introduction and Implementation Guide John Sexton Tilak Buneti Eva Ho Massimo Rosati Introduction to IBM Flex System family features and functions IBM Flex System V7000 Storage Node hardware overv...

Page 2: ......

Page 3: ...International Technical Support Organization IBM Flex System V7000 Storage Node Introduction and Implementation Guide September 2013 SG24 8068 01...

Page 4: ...Restricted Rights Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp Second Edition September 2013 This edition applies to IBM Flex System V7000 Storage Node Version 7...

Page 5: ...ex System p260 and p24L Compute Nodes 20 1 4 5 IBM Flex System p460 Compute Node 22 1 5 I O modules 24 1 5 1 IBM Flex System Fabric CN4093 10 Gb Converged Scalable Switch 25 1 5 2 IBM Flex System Fabr...

Page 6: ...es 69 2 6 11 Mirrored volumes 70 2 6 12 Easy Tier 71 2 6 13 Real time Compression 72 2 6 14 iSCSI 72 2 6 15 Fibre Channel over Ethernet FCoE 74 2 7 Advanced copy services 75 2 7 1 FlashCopy 75 2 7 2 I...

Page 7: ...setup 163 4 2 2 Using CMM for initial setup 167 4 3 IBM Flex System V7000 Storage Node Setup Wizard 170 4 4 System management 181 4 4 1 Graphical User Interface GUI 181 4 4 2 Launching IBM Flex System...

Page 8: ...ion 269 6 2 Creating a new host 273 6 2 1 Creating a Fibre Channel attached host 273 6 2 2 Creating an iSCSI attached host 276 6 3 Mapping a volume to the host 279 6 3 1 Mapping newly created volumes...

Page 9: ...mirroring and migration options 448 10 2 Tunable timeout 448 10 3 Usage of mirroring for migration 448 10 4 Managing Volume Mirror and migration with the GUI 450 Chapter 11 SAN connections and config...

Page 10: ...ting and preparing DM MPIO volumes for use 567 Chapter 13 Maintenance and troubleshooting 571 13 1 Reliability availability and serviceability RAS 572 13 2 Hardware and LED descriptions 573 13 2 1 Und...

Page 11: ...ot included 618 13 12 2 Configuring Call Home if FSM is included 620 13 13 IBM Flex System V7000 Storage Node power on and off 625 13 13 1 Powering on the system 626 13 13 2 Powering off the system us...

Page 12: ...x IBM Flex System V7000 Storage Node Introduction and Implementation Guide...

Page 13: ...not in any manner serve as an endorsement of those websites The materials at those websites are not part of the materials for this IBM product and use of those websites is at your own risk IBM may us...

Page 14: ...er DS4000 DS6000 DS8000 Easy Tier Electronic Service Agent EnergyScale FlashCopy HACMP IBM Flex System IBM Flex System Manager IBM PureData IBM Power Systems POWER7 PowerVM POWER PureApplication PureD...

Page 15: ...leigh Center John Sexton is temporarily assigned at the International Technical Support Organization Raleigh Center as team leader for this project He is a Certified Consulting IT Specialist based in...

Page 16: ...recovery and business continuity solutions He has achieved Brocade and Cisco SAN Design Certifications and is supporting critical and complex client engagements in the SAN and storage areas Massimo ha...

Page 17: ...elcome Your comments are important to us We want our books to be as helpful as possible Send us your comments about this book or other IBM Redbooks publications in one of the following ways Use the on...

Page 18: ...xvi IBM Flex System V7000 Storage Node Introduction and Implementation Guide...

Page 19: ...1 2 TB 2 5 inch10K RPM SAS drives instead of 1 TB 7 2K RPM NL SAS drives which was formerly the maximum size drive available Changed information Scalability enhancements enable the Storwize Software f...

Page 20: ...xviii IBM Flex System V7000 Storage Node Introduction and Implementation Guide...

Page 21: ...ay and provide flexible expansion capabilities for the future The scalable hardware features and the unprecedented cooling capabilities of the IBM Flex System products help you optimize hardware utili...

Page 22: ...timal performance and efficiency Simplified experience from design to purchase to maintenance creates efficiencies quickly IBM PureSystems offerings are optimized for performance and virtualized for e...

Page 23: ...eed and lower cost Like IBM PureApplication System it offers built in expertise integration by design and a simplified experience throughout its life cycle IBM Flex System V7000 Storage Node The produ...

Page 24: ...n connect to either the IBM Storwize V7000 Control Enclosure or the IBM Flex System V7000 Control Enclosures Figure 1 2 shows the different IBM PureSystems and their building blocks Figure 1 1 IBM Pur...

Page 25: ...is built on decades of expertise enabling deep integration and central management of the comprehensive open choice infrastructure system and dramatically cutting down on the skills and training requi...

Page 26: ...Component IBM PureFlex System Express IBM PureFlex System Standard IBM PureFlex System Enterprise IBM PureFlex System 42U Rack 1 1 1 IBM Flex System Enterprise Chassis 1 1 1 IBM Flex System Fabric EN...

Page 27: ...tterns of expertise help deliver and manage business processes services and applications by encapsulating best practices and expertise into a repeatable and deployable form This best practice knowledg...

Page 28: ...two types which is known as a hybrid configuration The bundled on site services provide some initial compute node configuration and might differ for IBM System x nodes and Power Systems compute nodes...

Page 29: ...r early versions of the IBM Flex System the only integrated storage was the IBM Storwize V7000 that was external to the IBM Flex System Enterprise Chassis With the introduction of IBM Flex System V700...

Page 30: ...uirements You can select from these offerings which are designed for key client initiatives and help simplify ordering and configuration While we only provide a summary of the IBM Flex System componen...

Page 31: ...ures that provides air cooling for switches IBM Flex System Chassis Management Module and power supplies Upper airflow inlet apertures that provide cooling for power supplies For proper cooling each b...

Page 32: ...hassis management support for up to four chassis Two IBM Flex System Chassis Management Module CMMs The CMM provides single chassis management support The chassis can be configured with the following...

Page 33: ...ly to be preferred Using three or six with N N redundancy power supplies allows for a balanced 3 phase configuration All power supply modules are combined into a single power domain within the chassis...

Page 34: ...f four 80 mm fan modules and two 40 mm fan modules The fan modules plug into the chassis and connect to the fan distribution cards The 80 mm fan modules can be added as required to support chassis coo...

Page 35: ...includes those with Intel Xeon processors or with IBM POWER7 processors Depending on the compute node design it can come in one of two different form factors Half wide node Occupies one chassis bay h...

Page 36: ...IBM Flex System x240 Compute Node The IBM Flex System x240 Compute Node available as machine type 8737 is a half wide two socket server running the latest Intel Xeon processor E5 2600 family processo...

Page 37: ...this embedded controller The x240 with the Intel Xeon E5 2600 series processors can support up to 768 GB of memory in total when using 32 GB LRDIMMs and with both processors installed The x240 uses D...

Page 38: ...ide connectivity to the Enterprise Chassis midplane I O expansion The x240 has two PCIe 3 0 x16 I O expansion connectors for attaching I O adapter cards There is also another expansion connector desig...

Page 39: ...r six or eight cores per processor with up to 16 threads per socket The processors have up to 20 MB of shared L3 cache Hyper Threading Turbo Boost Technology 2 0 depending on processor model one Quick...

Page 40: ...in IBM Flex System Enterprise Chassis units to provide a high density high performance compute node environment using advanced processing technology The IBM Flex System p24L Compute Node shares severa...

Page 41: ...As with previous generations of systems based on POWER processors the design philosophy for POWER7 processor based systems is one of system wide balance in which the POWER7 processor plays an importa...

Page 42: ...e The IBM Flex System Chassis Management Module CLI provides access to the text console command prompt on each server through a SOL connection enabling the p260 and p24L to be managed from a remote lo...

Page 43: ...plays an important role Each POWER7 processor has two integrated memory controllers in the chip Industry standard DDR3 Registered DIMM RDIMM technology is used to increase reliability speed and densi...

Page 44: ...e I O is controlled by four P7 IOC I O controller hub chips This provides additional flexibility when assigning resources within Virtual I O Server VIOS to specific Virtual Machine LPARs 1 5 I O modul...

Page 45: ...on adapter installed in its place It is shown on the node in Bay 2 in Figure 1 14 1 5 1 IBM Flex System Fabric CN4093 10 Gb Converged Scalable Switch The IBM Flex System Fabric CN4093 10 Gb Converged...

Page 46: ...l internal 10 Gb Ethernet FCoE ports external 10 Gb 40 Gb Ethernet FCoE ports and external flexible ports usable for either 10 Gb Ethernet FCoE or 4 8 Gb Fibre Channel For switch management access can...

Page 47: ...dapter ports on the compute nodes Four port adapters The EN4091 10 Gb Ethernet Pass thru has only 14 internal ports As a result only two ports on each compute node are enabled one for each of two pass...

Page 48: ...ts After all licenses have been assigned you can manually move those licenses from one port to another As it is dynamic no defined ports are reserved except ports 0 and 29 The FC5022 16Gb ESB Switch h...

Page 49: ...ckTools Requires a current version of the JRE on your workstation before pointing a web browser to the modules IP address The IP Address of the module must be configured QuickTools does not require a...

Page 50: ...re virtualized storage systems designed to complement virtualized server environments They provide unmatched performance availability advanced functions and highly scalable capacity IBM Storwize V7000...

Page 51: ...dynamic migration Supports faster and more efficient data copies for online backup testing or data mining Offers flexible server and storage management with easy to use GUI for block and file storage...

Page 52: ...cy technologies are very significant and include reduced acquisition cost because less hardware is required reduced rack space and lower power and cooling costs throughout the lifetime of the system W...

Page 53: ...ors intuitive control of these systems and provides a single integrated approach for managing both block and file storage requirements in the same system For organizations looking to manage both physi...

Page 54: ...ng Easy Tier technology Dynamic migration helps decreasing migration times from weeks or months to days eliminate the cost of add on migration tools and provides continuous availability of application...

Page 55: ...er iSCSI based SANs provide all the benefits of centralized shared storage in terms of storage consolidation and adequate levels of performance but use traditional IP based Ethernet networks instead o...

Page 56: ...innovative storage offering that delivers essential storage efficiency technologies and exceptional ease of use and performance all integrated into a compact modular design IBM Flex System V7000 Stora...

Page 57: ...it is based We describe in detail the controller and the expansion enclosures that make up the hardware of IBM Flex System V7000 Storage Node and point out the differences between them We present the...

Page 58: ...Directed Maintenance Procedures DMP to assist in resolving any events that might occur IBM Flex System V7000 Storage Node is a clusterable scalable storage system and an external virtualization device...

Page 59: ...sures are attached which provides redundant access to the drives that are inside the enclosures Each IBM Flex System V7000 Storage Node control canister has one chain connection Clone A copy of a volu...

Page 60: ...e up each SAS2 chain Managed disk MDisk A component of a storage pool that is managed by a clustered system An MDisk is either a RAID array created using the internal storage or a Small Computer Syste...

Page 61: ...em I O modules and switches for host I O and replication features through this midplane The control enclosure also houses a pair of redundant control canisters along with their cache batteries for bac...

Page 62: ...Storage Node capabilities Various new scalability enhancements are included in IBM Flex System V7000 Storage Node releases when they become available These enhancements allow the Storwize Software fa...

Page 63: ...tem V7000 Storage Node license Provides a volume level point in time copy function for any storage being virtualized by IBM Flex System V7000 Storage Node This function is designed to create copies fo...

Page 64: ...he FlashCopy relationships into a consistency group commands can be issued against all of the volumes in the group This action enables a consistent point in time copy of all of the data even though it...

Page 65: ...e Node managed storage IBM System Storage Easy Tier license included on the base license for IBM Flex System V7000 Storage Node Provides a mechanism to seamlessly migrate hot spots to the most appropr...

Page 66: ...d to a storage pool it is available to be virtualized and used by any of the features and functions of IBM Flex System V7000 Storage Node 2 4 IBM Flex System V7000 Storage Node licensing IBM Flex Syst...

Page 67: ...Flex System V7000 Control Enclosure one Flex System V7000 Disk Expansion Enclosure and three Storwize V7000 expansion enclosures 2076 212 224 This configuration would require three IBM Storwize Family...

Page 68: ...secondary system for a total of 10 enclosure licenses For primary and secondary systems with differing numbers of enclosures using Remote Mirroring the number of licenses needed for each system is th...

Page 69: ...BM Storwize Family Software for Flex System V7000 licenses The Flex System V7000 Remote Mirroring licensing would be as follows a sum of seven licenses for the primary systems plus seven licenses for...

Page 70: ...zation of that external storage system to perform that replacement activity are granted a 45 day period for use of external virtualization without having to purchase a license to complete such a migra...

Page 71: ...available slots in each control canister for populating with host network adapters The adapters in each of the node canisters must be of the same type in each of the slots so the adapters are install...

Page 72: ...ections to the Flex System Enterprise Chassis through the midplane and the switch modules It is important to remember that both control canisters must be populated with the same type of host network a...

Page 73: ...the control canister has only one SAS2 port per canister for the connection of expansion enclosures to add capacity There are also two USB connections available for support use to perform maintenance...

Page 74: ...re Fault Indicates that a hardware fault has occurred Controller Indicators Power Slow Blink Power available but Processor shutdown Fast blink doing POST On solid Powered up Status Off not operational...

Page 75: ...its cover removed Figure 2 6 Components and board layout of the expansion canister As shown in Figure 2 6 the expansion canister does not contain a battery backup unit as the control canister does It...

Page 76: ...rive slots at the time of writing Table 2 6 IBM Flex System V7000 Storage Node internal supported drives Drive capacity Drive speed Drive physical size 146 GB 15K RPM SAS 2 5 inch 300 GB 15K RPM SAS 2...

Page 77: ...connector on the power supply unit The PSU has an IEC C14 socket and the mains connection cable has a C13 plug As shown in Figure 2 9 the PSU has one green status LED indicator A to show it is powere...

Page 78: ...00 Storage Node drives Both drives use a different carrier and contain a different product identifier in their code Therefore the IBM Storwize V7000 expansions support their own drive types in their e...

Page 79: ...ns 2 6 IBM Flex System V7000 Storage Node components IBM Flex System V7000 Storage Node is an integrated entry midrange virtualization RAID storage to support and interface with the IBM Flex System It...

Page 80: ...on through the switch modules of the IBM Flex System At the time of writing attachment of external hosts to IBM Flex System V7000 Storage Node is not supported Hosts are defined to IBM Flex System V70...

Page 81: ...p have access to them Up to 512 host server objects can be defined in two I O groups 2 6 4 Clustered system A clustered system consists of one to four control enclosures All configuration monitoring a...

Page 82: ...as a set of goals that describe the wanted location and performance of each array A sequence of drive failures and hot spare takeovers can leave an array unbalanced that is with members that do not ma...

Page 83: ...t be visible to a host system on the storage area network as it should only be zoned to IBM Flex System V7000 Storage Node system An MDisk has four possible modes Array Array mode MDisks are construct...

Page 84: ...ge systems The preferred internal drives are the ones in the control enclosure and hot spares However when an external virtualized storage system is being managed it is preferred to have the active qu...

Page 85: ...tem A system can manage 4 million 4 x 1024 x 1024 extents For example a system with a 16 MB extent size can manage up to 16 MB x 4 MB 64 TB of storage The effect of extent size on the maximum volume s...

Page 86: ...ype disk size and disk speeds revolutions per minute rpms They must have the same RAID type RAID array size and disk spindle count in the RAID grouping The disk subsystems providing the MDisks must ha...

Page 87: ...to a host system by the clustered system In our virtualized environment the host system has a volume mapped to it by IBM Flex System V7000 Storage Node IBM Flex System V7000 Storage Node translates th...

Page 88: ...volume is required to be expanded the sequential volume is converted to a striped volume by policy when the expansion occurs Figure 2 15 shows examples of the striped and sequential volume types Figur...

Page 89: ...the volume and its virtual capacity The real capacity determines the quantity of MDisk extents that are allocated for the volume The virtual capacity is the capacity of the volume reported to IBM Fle...

Page 90: ...nt storage pool and can be on different managed physical storage systems which can help provide a better level of high availability to a solution When a host system issues a write to a mirrored volume...

Page 91: ...ions to perform the move Likewise the downgrading of an extent will not take place until the space is needed for a hotter extent Figure 2 17 shows the basic structure and concepts of this function Fig...

Page 92: ...d environment Real time Compression can be purchased to run on one control enclosure I O Group of a cluster and not for another allowing for shared environments that exist depending on need and provid...

Page 93: ...age Node control canister represents an iSCSI node and provides statically allocated IP addresses Each iSCSI node that is an initiator or target has a unique iSCSI Qualified Name IQN which can have a...

Page 94: ...sing separate flows However separate transactions can be delivered through separate TCP connections within one session For further details about configuring iSCSI see Chapter 11 SAN connections and co...

Page 95: ...time PiT copy technology FlashCopy can be performed on multiple source and target volumes FlashCopy permits the management operations to be coordinated so that a common single point in time is chosen...

Page 96: ...m to the other With the wide variety of storage systems that can be managed under any of these systems the options available for replicating data between a number of storage systems are multiplied Rem...

Page 97: ...Remote Copy can be maintained in one of two modes synchronous or asynchronous With IBM Flex System V7000 Storage Node Metro Mirror and Global Mirror are the IBM branded terms for the functions that a...

Page 98: ...rror relationships per clustered system 4096 This configuration can be any mix of Metro Mirror and Global Mirror relationships Remote Copy relationships per consistency group 4096 No additional limit...

Page 99: ...to protect your applications and your computers You can also use security features to restrict access by the IBM service representative Your IBM service representative can provide you with more detail...

Page 100: ...Home function if you have a maintenance contract with IBM or if IBM Flex System V7000 Storage Node is within the warranty period To send email you must configure at least one SMTP server You can spec...

Page 101: ...hooting Recovery and Maintenance Guide GC27 4205 http publib boulder ibm com infocenter flexsys information topic com ibm acc 4939 doc site_bkmap_trblshoot pdf IBM Flex System V7000 Storage Node Suppo...

Page 102: ...ube com watch v I2rzt3m2gP0 IBM Storwize V7000 Interface tour http www youtube com watch v FPbNRs9HacQ IBM Storwize V7000 Volume management http www youtube com watch v YXeKqH8Sd9o IBM Storwize V7000...

Page 103: ...various components of the IBM Flex Systems a description of all these options is beyond the scope of this publication In this chapter we cover the management options related to IBM Flex System V7000 S...

Page 104: ...into an existing IT infrastructure Within an IBM Flex System Enterprise Chassis each node has a management controller that communicates by the CMM enabled 1 GbE private management network allowing cen...

Page 105: ...he Flexible Service Processor FSP provides a serial over LAN SOL interface for performing common management tasks on the node Table 3 1 gives an overview of system management capabilities of these har...

Page 106: ...visioning for greater responsiveness to business needs and lower overall cost IBM Flex System V7000 Storage Node is an Integrated storage system with advanced storage efficiency capabilities which inc...

Page 107: ...to four distinct levels based on their level of depth and detail in management capabilities Starting Level IBM Flex System Manager Chassis Map The IBM Flex System Manager offers a Chassis Map which is...

Page 108: ...ing the IBM Flex System Manager Storage Control feature It provides integrated management with a Systems and Storage perspective Using Storage Control you can manage storage across your data center co...

Page 109: ...e advanced disk features such as Easy Tier Mirroring Clustering and Virtualization can be configured using the IBM Flex System V7000 Storage Node User Interface Figure 3 3 shows the IBM Flex System V7...

Page 110: ...ssis One CMM comes standard with the chassis There are two CMM bays in the chassis giving the user an option to add a second CMM for redundancy Through an embedded firmware stack the CMM implements fu...

Page 111: ...o them Accessing and configuring I O modules Monitoring the status of the compute nodes and other components Selecting recipients for alert notification of specific events Using a remote console for t...

Page 112: ...fault the CMM is configured to respond to Dynamic Host Configuration Protocol DHCP first before using its static IPv4 address It has the following default settings IP address 192 168 70 100 Subnet 255...

Page 113: ...n in Figure 3 5 Figure 3 5 IBM Chassis Management Module Login panel An Initial Setup wizard starts automatically the first time you access the web interface of a new CMM or a CMM that has been reset...

Page 114: ...components in a single view and gives the ability to easily configure the networking options for a device In IBM Flex System Chassis Management Module web interface a dedicated option is given in the...

Page 115: ...ous chassis Components If you want to view the IP address of a certain component instantly you can just scroll the mouse pointer over the View option This action opens a pop up window that shows the I...

Page 116: ...odule access features in the next section due to its relevance with storage management For detailed information regarding IBM Flex System Chassis Management Module configuration capabilities visit the...

Page 117: ...es The next view shows the available I O modules in IBM Flex System It also gives options to Power and Restart the modules as well as the Actions that can be performed on them as shown in Figure 3 11...

Page 118: ...BM Flex System as shown in Figure 3 13 Figure 3 13 IO Connectivity status of an I O module To open the Module specific Management interface click the I O Module wanted then the Actions drop down butto...

Page 119: ...stem Status shows the view of Chassis front and back It also shows health status of the individual physical components Moving the mouse over the selected component and clicking opens a pop up window t...

Page 120: ...ending on the component one or more tabs show information regarding hardware firmware events and other information An example is shown in Figure 3 17 Figure 3 17 CMM Details of the selected Chassis co...

Page 121: ...ems management 101 The quick view includes part information serial number health and temperature and unique identity depending on the component as shown in Figure 3 18 Figure 3 18 CMM System Informati...

Page 122: ...ar view there is a list of Actions for the particular canister which includes options to power on off restart the service processor launch the detailed IBM Flex System V7000 Management application as...

Page 123: ...ll times The IBM Flex System Chassis Management Module web interface provides multiple views to facilitate monitoring and notification of environmental and temporal indices Figure 3 21 shows the Envir...

Page 124: ...ectivity Figure 3 24 shows the DC Power consumption for the component Figure 3 24 Power consumption of selected chassis component The IBM Flex System Chassis Management Module web interface has a menu...

Page 125: ...gy view from Chassis Management menu The Chassis Hardware Topology view provides very detailed information regarding all the hardware in the chassis It goes deep to the point where you can select indi...

Page 126: ...gement Storage Nodes The Storage Nodes view shows all the storage enclosures in the chassis and gives options to perform various actions on these including power and restart options as shown in Figure...

Page 127: ...tify LED The IBM Flex System V7000 Storage Node User Interface is used for detailed level management and configuration of IBM Flex System V7000 Storage Node Click Actions Launch Storage Node Controlle...

Page 128: ...3 32 IBM Flex System V7000 Storage Node Login window After logging in you can use the IBM Flex System V7000 Storage User Interface directly to perform the management tasks as shown in Figure 3 33 Figu...

Page 129: ...ll as retrieve service data files from the chassis components To acquire service data from the Storage Node you can perform the following steps From the top menu select Service and Support Download Se...

Page 130: ...node to initiate the generation of a new dump file on the compute node and overwrite any existing file The CMM will then collect the dump file by transferring it from the node to the CMM file system...

Page 131: ...system select Mgt Module Management File Management as shown in Figure 3 38 Figure 3 38 File Management The File Management page shows the status of CMM s local storage file system and the CMM File M...

Page 132: ...service folder The file containing a dump of service data is also copied in tftproot service for ease of transfer as shown in Figure 3 41 Figure 3 41 Viewing service data in tftp service path For mor...

Page 133: ...ized management tasks in connection with security performance resource allocation and application version control and so on Figure 3 42 shows the front view of an IBM Flex System Manager Node Figure 3...

Page 134: ...torage and network capacities in response to changing business and operational needs The built in network provides a reliable and secure communications path for management and inter rack data sharing...

Page 135: ...Flex System Manager overview The system infrastructure has the following salient features IBM Flex System V7000 Storage Node Integrated virtualized IBM Flex System Storage IBM Flex System Storage Vir...

Page 136: ...rage management with Flex System Manager offers simplified storage virtualization capabilities The following storage features are included in IBM Flex System Manager Storage device discovery Showing r...

Page 137: ...n management functions These components must be activated using the available IBM Feature on Demand FoD software entitlement licenses and they are licensed on a per chassis basis IBM Flex System Manag...

Page 138: ...ervention replaces a failed compute node from a designated pool of spare compute nodes Using an enhanced graphical user interface GUI creates addresses for compute nodes saves the address profiles and...

Page 139: ...ure and use IBM Fabric Manager IFM visit this website http publib boulder ibm com infocenter flexsys information topic com ibm acc iof m doc dw1li_advanced html 3 3 3 Logging in to the IBM Flex System...

Page 140: ...on how to navigate around in the IBM Flex System Manager The home page shown in Figure 3 46 has six tabs that help you with setting up and administering the system Home tab Initial setup of the system...

Page 141: ...manager window whick offers the following functions Clicking the chassis name opens IBM FSM Explorer covered later in this section The global search tab helps in finding tasks and hardware resources...

Page 142: ...information regarding the IBM Flex System Manager web interface and management options see the IBM Flex System Manager Systems Management Guide located at the following website http publib boulder ibm...

Page 143: ...will be opened in IBM FSM Explorer and other tasks will be opened in IBM Flex System Manager When choosing tasks in the horizontal menu it will inform you if the selected task is opened in IBM Flex S...

Page 144: ...es gives you shortcuts to Storage control Network control and the Service and Support pages in IBM Flex System Manager Figure 3 51 shows the Home menu Figure 3 51 IBM FSM Explorer Home menu The System...

Page 145: ...Compliance policy page Policies Automation gives you shortcuts to the Event Automation Plans page Event Actions page Event Filters page and Command Automation page Jobs gives you a shortcut to the Act...

Page 146: ...m to automatically report errors to IBM and Server Auditing which takes you to the Credentials page Administration gives you links to Administration Plug ins and Applications tab in the IBM Flex Syste...

Page 147: ...r 3 Systems management 127 Figure 3 56 IBM FSM Explorer Vertical menu The Chassis page gives you an overview of what is installed in one or more chassis as shown in Figure 3 57 Figure 3 57 IBM FSM Exp...

Page 148: ...tion and Implementation Guide To get to the chassis map click the chassis in the vertical menu in Figure 3 56 on page 127 then click the chassis under the chassis list as shown here in Figure 3 58 Fig...

Page 149: ...of the front and back with a graphical representation of the components as shown in Figure 3 59 Figure 3 59 IBM FSM Explorer chassis map To get detailed information about the chassis and chassis comp...

Page 150: ...ementation Guide As shown in Figure 3 60 Common Actions for chassis are shown below the Hardware map at the left side The detailed information regarding the chassis is shown below the Hardware map at...

Page 151: ...menu shows the actions that can be performed at the Chassis level as shown in Figure 3 61 Figure 3 61 IBM FSM Explorer Component Actions The Host page gives you an overview of Hosts as shown in Figure...

Page 152: ...Figure 3 63 IBM FSM Explorer Network page The Storage page gives you an overview of all storage systems fabrics and storage pools as shown in Figure 3 64 Figure 3 64 IBM FSM Explorer Storage page Rig...

Page 153: ...manager as seen in Figure 3 66 Figure 3 66 IBM FSM Explorer Storage page launch IBM Storwize V7000 Element manager Choosing the option to Launch the IBM Storwize V7000 Element Manager will open the IB...

Page 154: ...t 12 numbers specify the MAC address of the switch will show you all the switches as depicted in Figure 3 68 Figure 3 68 IBM FSM Explorer Storage page Switch overview Double clicking one of the switch...

Page 155: ...assis Network Control Network Control is a plug in that enables utilization of integrated network management capabilities of an FSM It has the following key features Logical network management allows...

Page 156: ...SM Explorer select Home menu Network Control as shown in Figure 3 72 Figure 3 72 IBM FSM Explorer Network control This action will open up the Network Control interface which shows management and conf...

Page 157: ...et Switches along with information regarding state access IP and type as shown in Figure 3 74 Figure 3 74 Viewing Ethernet Switches in FSM Select a switch by clicking its given name to go to the detai...

Page 158: ...to view and gather Inventory for the selected switch module as shown in Figure 3 76 Figure 3 76 Gathering Inventory information References For further details regarding the management and configuratio...

Page 159: ...IBM Support using the IBM Electronic Service Agent ESA tool The IBM Flex System Manager will automatically monitor for any serviceable problems on your managed hardware and automatically report probl...

Page 160: ...shown in Figure 3 78 Figure 3 78 SSM default view To collect logs from IBM Flex System V7000 Storage Node using IBM Flex System Manager click Manage support files as shown in Figure 3 78 The Manage su...

Page 161: ...f Electronic Service Agent is enabled Upload to ftp server Upload to an ftp server on your network Copy to media USB key inserted into IBM Flex System Manager Node Download Download the support file t...

Page 162: ...as shown in Figure 3 81 Figure 3 81 IBM FSM Collect support file choose device To select the type of support file that you want to collect click the Support Files tab then select the file and click C...

Page 163: ...tion with VMControl featuring the following additional functions Storage provisioning for image creation deployment and cloning Ability to manage storage system pool life cycle take group actions acro...

Page 164: ...o the device which simplifies configuration setup and improves device management reliability After being discovered these devices will be listed as being managed by Storage Manager in Flex System Mana...

Page 165: ...Chapter 3 Systems management 145 The Storage Management window opens showing all storage that is available in the managed IBM Flex Systems as shown in Figure 3 84 Figure 3 84 FSM Storage Control...

Page 166: ...System V7000 Storage Node Introduction and Implementation Guide You can also manage storage by going to Home tab Additional Setup Manage System Storage as shown in Figure 3 85 Figure 3 85 FSM Manage S...

Page 167: ...em Manager and see which actions there can be performed click Network Storage in the storage control window as shown in Figure 3 86 Figure 3 86 FSM Storage Control Network storage This will open a new...

Page 168: ...ng the storage system name opens the selected storage systems properties page as shown in Figure 3 88 Figure 3 88 FSM Storage Control Storage systems From this window you are able to see General infor...

Page 169: ...0 Storage Node management application as shown in Figure 3 89 Figure 3 89 FSM Storage Control Storage system action menu Managing external storage systems To manage external storage subsystems in IBM...

Page 170: ...urce Explorer shows a snapshot of all resources in the IBM Flex System Manager managed by the IBM Flex System Manager as shown in Figure 3 91 Figure 3 91 Resource Explorer 3 Navigate to manage discove...

Page 171: ...151 4 Select target storage for example IBM Storwize V7000 as shown in Figure 3 93 Figure 3 93 Selecting target storage 5 Select the tab Inventory to collect the inventory as shown in Figure 3 94 Figu...

Page 172: ...or the devices managed by Storage Control 1 Select Resource Explorer All Systems Select the managed device for which you want to get the latest health and alert status 2 Discover and then unlock the d...

Page 173: ...ms management 153 Discovering new storage systems To discover new storage systems from the storage management window click Discover Storage as shown in Figure 3 96 Figure 3 96 FSM Storage Management D...

Page 174: ...r the IP address of the storage you need to discover the ssh private key then click Discover as shown in Figure 3 98 Figure 3 98 Discovering new storage IP address and SSH private Key A new job will s...

Page 175: ...0 Storage Node from CMM and FSM For more information regarding CMM and FSM see Implementing Systems Management of IBM PureFlex System SG24 8060 and the IBM InfoCenter http publib boulder ibm com infoc...

Page 176: ...156 IBM Flex System V7000 Storage Node Introduction and Implementation Guide...

Page 177: ...n overview of the planning tasks and refer you to the appropriate documentation as needed to enable you to complete these tasks To help you get started and guide you to a successful installation and i...

Page 178: ...Node enclosures a control enclosure and up to two IBM Flex System V7000 Expansion Enclosures Any additional expansion enclosures would have to be IBM Storwize V7000 expansion enclosures attached exte...

Page 179: ...tem Single host zoning guarantees segregation After IBM Flex System V7000 Storage Node hosts and optional external storage systems are connected to the SAN fabrics zoning needs to be implemented See C...

Page 180: ...ode in the system If Ethernet port 2 1 Gbps is configured with a system IP address Ethernet port 2 on all nodes must also be connected to the same subnet However the subnet for Ethernet port 2 does no...

Page 181: ...Flex chassis and provides centralized system monitoring and management For more details see Chapter 3 Systems management on page 83 The IBM Flex System Manager Node FSM The Flex System Manager FSM is...

Page 182: ...ou are using a supported web browser from the following website http www ibm com support entry portal overview hardware puresystems pureflex_ system storage_node flex_system_v7000 Decide whether you a...

Page 183: ...cessary steps when using the FSM web user interface 1 Open a web browser and point it to the IP address of the FSM and log in The following menu panel Figure 4 1 displays giving you a number of select...

Page 184: ...ws you to select the applicable enclosure from the Chassis Map as shown in Figure 4 2 Figure 4 2 Select and launch the chassis in the Chassis Manager 2 In the Chassis Manager select the applicable cha...

Page 185: ...sis by performing the following steps a Select General Actions Add or Remove a chassis as shown in Figure 4 4 B b Select the chassis that you want to manage c Click Manage The Management Domain page d...

Page 186: ...ine the Service IP setting for the storage node For details see 4 5 1 Changing the Service IP address on page 184 Figure 4 5 Launch Storage Manager V7000 5 The next window is a welcome window from the...

Page 187: ...Storage Node Setup Wizard on page 170 4 2 2 Using CMM for initial setup When the first control enclosure for a new IBM Flex System V7000 is installed in the Flex System chassis you must create and con...

Page 188: ...eft canister in the graphical representation of the storage node as shown in Figure 4 8 Figure 4 8 CMM Select Launch Storage Node Console 3 A browser window displays the IBM Flex System V7000 Storage...

Page 189: ...he CD or in the IBM Flex System V7000 Storage Node Information Center After clicking Launch in Figure 4 9 you have an option to Create a new system or Add to an existing system as shown in Figure 4 10...

Page 190: ...tifications and storage configuration If you are not ready to complete further configuration steps use the configuration tasks in the management GUI to complete the configuration at a later time When...

Page 191: ...ement GUI Click Next to continue this initial setup process as shown in Figure 4 14 Figure 4 14 IBM Flex system v7000 s Welcome to System Setup window Note During the initial setup of the Flex System...

Page 192: ...nse Agreement 3 Optionally you can type in the number of System licenses needed for use of certain functions and click Apply and Next as shown in Figure 4 16 The System Licenses include External Virtu...

Page 193: ...on 173 4 Set up a system name and click Apply and Next as shown in Figure 4 17 Figure 4 17 Setup wizard Set a name for the system 5 Set up the date and time for your system and click Apply and Next as...

Page 194: ...ect Yes and click Next to continue as shown in Figure 4 19 Figure 4 19 Configure Call Home support notifications Enter the System Location information and click Next to continue as shown in Figure 4 2...

Page 195: ...problems on the system Click Apply and Next to continue as shown in Figure 4 21 Figure 4 21 Enter contact details Enter at least one Email Server that your company uses Click Apply and Next to contin...

Page 196: ...ents occur Click Apply and Next to continue as shown in Figure 4 23 Figure 4 23 Enter user Emails to receive notifications A summary of what changes have been applied to the system is displayed Click...

Page 197: ...matically configure the storage and click Next to continue as shown in Figure 4 25 Figure 4 25 Configure storage options Figure 4 26 Summary of automatically configured storage Note Select No if you w...

Page 198: ...s been detected by the system correctly and click Next as shown in Figure 4 27 Figure 4 27 Summary of non automatically configure storage 8 After you have set up your system configure settings for eac...

Page 199: ...type you want and click Create Host as shown in Figure 4 29 Figure 4 29 Select the Host Type Configure an optional Host Name Fibre Channel ports and other advanced settings if host type Fibre Channel...

Page 200: ...anced settings if host type iSCSI Host is selected as shown in Figure 4 31 Figure 4 31 Configure iSCSI Host Type 9 Click Finish to complete the Setup wizard task 10 IBM Flex System V7000 Storage Node...

Page 201: ...r GUI access to configure manage and troubleshoot IBM Flex System V7000 Storage Node It is also used primarily to configure RAID arrays and logical drives assign logical drives to hosts replace and re...

Page 202: ...GUI logon panel Figure 4 34 shows an overview of the IBM Flex System V7000 Storage Node GUI Figure 4 34 IBM Flex System V7000 Storage Node management GUI Welcome page For more details on how to use I...

Page 203: ...M after the management IP been assigned to your new Flex System V7000 storage node To start log in to the CMM GUI and navigate to the Chassis Map Right click the canister that you want to launch and s...

Page 204: ...e the software from another node Upgrade software on node canisters manually versus performing a standard upgrade procedure Configure a control enclosure chassis after replacement Change the Service I...

Page 205: ...he chassis is operating and reachable you can use the CMM to change the Service IP address of a node canister as shown in Figure 4 38 1 Open the CMM and navigate to the Chassis Management Component IP...

Page 206: ...ls Setup and maintenance of client public SSH keys on the clustered system Setup and maintenance of volumes Setup of logical host objects Mapping volumes to hosts Navigating from managed hosts to volu...

Page 207: ...he system knows how to connect to and log on to the system Pay attention to those times when the normal system administrators are not available because of vacation or illness Record the following info...

Page 208: ...188 IBM Flex System V7000 Storage Node Introduction and Implementation Guide...

Page 209: ...ights reserved 189 Chapter 5 IBM Flex System V7000 Storage Node GUI interface This chapter provides an overview of the graphical user interface GUI of IBM Flex System V7000 Storage Node and shows how...

Page 210: ...is loosely based on the IBM System Storage XIV software and has a similar look and feel For more information about managing IBM Flex System V7000 Storage Node through FSM see Chapter 3 Systems managem...

Page 211: ...Pools menu The Volumes menu The Hosts menu The Copy Services menu The Access menu The Settings menu In the middle of the window is a diagram illustrating the existing configuration Clicking the functi...

Page 212: ...GUI navigation 5 1 3 Navigation Navigating around the management tool is simple You can hover the cursor over one of the eight function icons on the left side of the window which highlights the functi...

Page 213: ...I interface 193 Figure 5 4 Navigation using the menu options Figure 5 5 shows a list of IBM Flex System V7000 Storage Node Software function icons and the associated menu options Figure 5 5 IBM Flex S...

Page 214: ...rage Node Introduction and Implementation Guide For example if you click the Volumes menu you can change the window s view Figure 5 6 This action also applies to any other menu options Figure 5 6 Navi...

Page 215: ...ou select multiple items by using a combination of the Shift or Ctrl keys To select multiple items in a display click the first item press and hold the Shift key and click the last item in the list yo...

Page 216: ...5 8 Multiple selections using the Ctrl key 5 1 5 Status Indicators menus Other useful tools are the Status Indicators that appear at the bottom of the window These indicators provide information about...

Page 217: ...running tasks by clicking the button on the status indicator bars at the bottom of the window Figure 5 10 Figure 5 10 Status Indicators Figure 5 11 shows additional information about Recently Complet...

Page 218: ...management software provides an efficient and quick mechanism for navigating between the various different functions Clicking one of the eight function icons on the left side of the window causes a m...

Page 219: ...describe the Monitoring menu and its options 5 3 1 Monitoring System Details menu Figure 5 13 shows the Monitoring System Details menu in which detailed information about firmware World Wide Names Ca...

Page 220: ...pper level of the Details section of IBM Flex System V7000 Storage Node From here the firmware level host name and capacity usage can be reviewed and changed Clicking the Actions button here gives a n...

Page 221: ...stem V7000 Storage Node Control Enclosure contains Array controllers called canisters as well as up to 24 disk drives Figure 5 15 shows properties for the Control Enclosure including the system Serial...

Page 222: ...section of the properties for the left canister including Node name SAS port status and the iSCSI Initiator name IQN Name Figure 5 16 System Details canisters Note In Figure 5 16 and Figure 5 17 the...

Page 223: ...m V7000 Storage Node For more information about SAN zoning for IBM Flex System V7000 Storage Node see Chapter 11 SAN connections and configuration on page 457 For more information about SAN zoning wit...

Page 224: ...evice Logins Reduced This error happens when an underlying disk array loses connectivity through one or more of its ports that are zoned to and connected to IBM Flex System V7000 Storage Node A few po...

Page 225: ...l exists and it will fix the issue if possible The fix procedure can bring the system out of a Degraded state and into a Healthy state Figure 5 19 shows that an error exists which must be fixed To fix...

Page 226: ...ent and click Run Fix Procedure as shown in Figure 5 20 if a DMP for that event is available Figure 5 20 Run Fix Procedure using right click menu option Note A DMP may not available for the selected e...

Page 227: ...ge Node GUI interface 207 To view the details of a specific event click Properties see Figure 5 20 on page 206 The Properties window opens as depicted in Figure 5 21 and shows the details of the error...

Page 228: ...edure Example In Figure 5 22 we demonstrate how an Error code 1625 is fixed Before starting the fix procedure we filter events to Recommended Actions so that only errors are displayed that require att...

Page 229: ...procedure for the error with Event Code1625 Incorrect Controller Configuration Figure 5 23 Error code 1625 Fix Procedure Step 1 Figure 5 24 shows the next step of the fix procedure The fix procedure i...

Page 230: ...5 25 shows the third step of the fix procedure where we mark the error as fixed Click Next to rescan the system to see if the error persists Figure 5 25 Error code 1625 Fix Procedure Step 3 Figure 5...

Page 231: ...terface 211 Figure 5 27 shows that the rescan has finished Figure 5 27 Error code 1625 Fix Procedure Step 5 Figure 5 28 shows that the rescan has finished and that the error condition has been resolve...

Page 232: ...nterfaces subdivided into FC Fibre Channel traffic towards hosts iSCSI shows iSCSI traffic towards hosts SAS traffic towards the disk back end Volumes traffic towards hosts that has mapped volumes sub...

Page 233: ...stem V7000 Storage Node Figure 5 30 Monitoring Performance If there is a need for additional performance monitors the most optimal tool is Tivoli Storage Productivity Center From there you can manage...

Page 234: ...ile server to form file systems The Pools menu contains different views of the IBM Flex System V7000 Storage Node Volumes MDisks Internal Storage and External Storage as well as the System Migration f...

Page 235: ...an create or delete storage pools as well as perform actions on volumes such as mapping to host or unmapping from host Also migration actions can be performed from the Volumes by Pool menu Figure 5 32...

Page 236: ...d manages these drives Clicking the Internal Storage option opens the window shown in Figure 5 33 From this window you can place internal disk drives into storage pools This window also provides the o...

Page 237: ...nal Storage systems menu By default the external storage controllers are named controller0 controller1 and so on It is good practice to rename these controllers to a more meaningful name for your stor...

Page 238: ...Figure 5 35 shows the MDisks that are available to the IBM Flex System V7000 Storage Node system The MDisks show whether they are managed in which case the storage pool is displayed or whether they ar...

Page 239: ...m V7000 Storage Node GUI interface 219 By right clicking the display bar Figure 5 36 you can choose to change the fields that are displayed Select the items that you want to be displayed Figure 5 36 D...

Page 240: ...de From this window you can choose to either use the options of the Actions button or you can choose to highlight the particular MDisks that you require right click and access the same options Figure...

Page 241: ...volumes are built from extents in the pools and are presented to hosts as external disks In this section we describe the Volumes menu and its options When you hover the cursor over the Volumes functio...

Page 242: ...ion Guide 5 5 1 The Volumes menu Clicking the Volumes menu opens the window shown in Figure 5 39 From here you can perform tasks on the volumes for example shrink or enlarge them map them to a host or...

Page 243: ...is menu you can perform various operations on the volumes You can use the Actions button to access these operations or you can right click the volume name which opens a list of operations that can be...

Page 244: ...BM Flex System V7000 Storage Node Introduction and Implementation Guide 5 5 2 Volumes by Pool menu Clicking the Volumes by Pool menu opens the window shown in Figure 5 41 Figure 5 41 Volumes by Pool w...

Page 245: ...ce 225 Similar to the previous window you can either use the Actions button to access the menu operations or you can right click the pool to display a list of valid commands Figure 5 42 Figure 5 42 Co...

Page 246: ...ntroduction and Implementation Guide 5 5 3 Volumes by Host menu Clicking the Volumes by Host option opens the window shown in Figure 5 43 This window shows the volumes that have been mapped to a given...

Page 247: ...ght click the pool to show a list of valid commands Figure 5 44 Figure 5 44 Commands for a single volume from the Volume by Host menu By using the Migrate to Another Pool option the Volume can be migr...

Page 248: ...Node Introduction and Implementation Guide 5 6 Hosts menu In this section we describe the Hosts menu and its options When you hover the cursor over the Host function icon a menu opens Figure 5 45 Figu...

Page 249: ...orage Node GUI interface 229 5 6 1 Hosts menu Clicking Hosts opens the window shown in Figure 5 46 From here you can modify host mappings unmap hosts rename hosts and create new hosts Figure 5 46 The...

Page 250: ...and right click it to access the commands Figure 5 47 Figure 5 47 Actions for a single host from the Hosts menu Figure 5 48 shows an example of Modify Mappings The selected Windows server has no mappi...

Page 251: ...ge Node GUI interface 231 5 6 2 Ports by Host menu Clicking Ports by Hosts opens the window shown in Figure 5 49 This window shows the Fibre Channel and iSCSI ports that are assigned to a particular h...

Page 252: ...ge Node Introduction and Implementation Guide Clicking the Actions button allows you to modify the mappings unmap volumes rename hosts and delete ports Figure 5 50 Figure 5 50 Actions for a single hos...

Page 253: ...I interface 233 5 6 3 Host Mappings menu Clicking Host Mappings opens the window shown in Figure 5 51 This window shows the Host name SCSI identifier Volume name and the Volume identifier for all the...

Page 254: ...n use the Actions buttons shown in Figure 5 52 or you can select a host and right click it to access the commands Figure 5 52 Actions for a single host from the Host Mapping window By unmapping the vo...

Page 255: ...ata FlashCopy is provided for instant copy of block volumes within the cluster Remote copy is used to copy block volumes to another location on another cluster and it can be done synchronously Metro M...

Page 256: ...py menu Clicking FlashCopy opens the window shown in Figure 5 54 This window shows the volumes that are available If you right click a volume a list of operations opens From here you can perform tasks...

Page 257: ...ck to Properties to open the Volume Details window shown in Figure 5 55 From here you can click the tabs at the top of the window to display additional information such as the hosts that the volume or...

Page 258: ...me consistency across volumes Clicking FlashCopy Consistency Group opens the window shown in Figure 5 56 This window enables FlashCopy relationships to be placed into a consistency group You can also...

Page 259: ...target volumes Clicking FlashCopy Mappings opens the window shown in Figure 5 57 From this window you can Start Stop Delete and Rename the FlashCopy mappings There is also an option to move the relat...

Page 260: ...In this section we describe the Access menu and its options If you hover the cursor over the Access function icon a menu opens Figure 5 58 Figure 5 58 Navigate to the Access menu The Access menu allo...

Page 261: ...m V7000 Storage Node GUI interface 241 5 8 1 Users menu Figure 5 59 shows the Users window This window enables you to create and delete new users change and remove passwords and add and remove SSH key...

Page 262: ...e SSH key has been generated Starting with SAN Volume Controller and IBM Storwize V7000 V6 3 SSH keys are not required for CLI access and you can choose either to use SSH or password for CLI authentic...

Page 263: ...that even if initiated from the GUI most actions cause a CLI command to be run so it will also be logged Clicking Audit Log opens the window shown in Figure 5 61 The cluster maintains an audit log of...

Page 264: ...n and Implementation Guide 5 9 Settings menu In the following section we describe the Settings menu and its options If you hover the cursor over the Settings function icon the Settings menus opens Fig...

Page 265: ...he levels of alerting as preferred Clicking Event Notification opens the window shown in Figure 5 63 From this window you can configure email alerts which is included in the Call Home function and SNM...

Page 266: ...the system User groups define roles that authorize the users within that group to a specific set of operations on the system A remote user is authenticated on a remote service with Lightweight Direct...

Page 267: ...System V7000 Storage Node controllers called canisters These controllers have their own IP addresses where several support actions can be performed Some of them are listed here Place a canister in Ser...

Page 268: ...el paths over the SAN and only relates to the storage nodes When you click Fibre Channel Figure 5 65 some useful information is displayed about Fibre Channel connectivity In this example we review con...

Page 269: ...pport case is opened by IBM Flex System V7000 Storage Node administrators The overall process is that the system administrator download the requested support package from the system and then upload it...

Page 270: ...files or delete them by selecting a single item Figure 5 67 and click either the Download or Delete options under the Actions button Figure 5 67 Download Delete options of the Actions button Delete op...

Page 271: ...tem V7000 Storage Node GUI interface 251 At the upper right of the window there is a Node option that can be used to show node canister 1 or 2 log files Figure 5 68 Figure 5 68 Change the log listing...

Page 272: ...pens the window shown in Figure 5 69 This window provides a number of different options that can be used to collect logs and save state information from the cluster Figure 5 69 Download a support pack...

Page 273: ...Save to save a copy of the package Figure 5 71 Figure 5 71 Save the Support Package The downloaded support package is now ready for analysis which is part of how IBM Support deals with troubleshootin...

Page 274: ...for the cluster enter licensing details if needed and perform software upgrades for the cluster Clicking the General option opens the window shown in Figure 5 72 This window provides options to set th...

Page 275: ...first part of this chapter describes how to create the different types of volumes on IBM Flex System V7000 Storage Node The second part of this chapter explains how to map volumes to the attached host...

Page 276: ...Create new volumes starting at section 6 1 1 Creating a generic volume on page 258 Create a new host starting at section 6 2 Creating a new host on page 273 Map volumes to the host at section 6 3 Mapp...

Page 277: ...striped volume that is space efficient There are choices under the Advanced button to help determine how much space is fully allocated initially and how large the volume is able to grow as described i...

Page 278: ...neric volume type This type of volume is fully provisioned with the entire size of the volume space dedicated to it The host and IBM Flex System V7000 Storage Node can see the fully allocated space To...

Page 279: ...tly done in one step If you want to create volumes with different size capacity you can do so by creating one volume at a time The result is shown in Figure 6 4 Figure 6 4 Create a generic volume 3 En...

Page 280: ...s as though they were fully allocated When creating a thin provisioned volume it is possible to specify two capacities the real physical capacity allocated to the volume from the storage pool and its...

Page 281: ...size 3 Under the Volume Name field is a summary showing that you are about to make a thin provisioned volume how much virtual space is will be available the space that will be allocated real size and...

Page 282: ...k OK to return to Figure 6 7 on page 261 5 In Figure 6 7 on page 261 click Create and the creation task completes as shown next in Figure 6 9 Figure 6 9 Create a thin provisioned volume CLI command di...

Page 283: ...ary Pool called External_Pool2 and is placed externally to IBM Flex System V7000 Storage Node on an IBM DS4000 disk array By using this configuration the volume is tolerant to either internal or exter...

Page 284: ...ool by clicking it and the view changes to the second pool Figure 6 11 Figure 6 11 Create a mirrored volume Select the secondary pool 3 Select the secondary pool by clicking it and enter a volume name...

Page 285: ...ced mirroring features 5 In the advanced mirroring settings you are able to specify a synchronization rate Enter a Mirror Sync Rate between 1 100 With this option you can set the importance of the cop...

Page 286: ...host on page 279 6 1 4 Creating a thin mirror volume A thin mirror volume allows you to allocate the required space on demand described in 6 1 2 Creating a thin provisioned volume on page 260 and have...

Page 287: ...Basic volume and host configuration 267 To create a thin mirror volume complete the following steps 1 From the New Volume window select Thin Mirror Figure 6 15 Figure 6 15 Thin Mirror select the prim...

Page 288: ...the primary pool by clicking it and the view changes to the second pool Figure 6 16 Figure 6 16 Thin Mirror select the secondary pool 3 Select the pool for the secondary copy and enter a name and a s...

Page 289: ...ntelligence to automatically detect and judge if a block is a good candidate for compression If the compression software detects that a block is unlikely to achieve a reasonable compression ratio it w...

Page 290: ...as designed with transparency in mind so that it can be implemented without changes to applications Creating a compressed volume To create a Compressed volume complete the following steps 1 Click New...

Page 291: ...reate Figure 6 20 Compressed Volume Enter name and size Note It is recommended to use separate storage pools for compressed and non compressed volumes To achieve the best results compressed and non co...

Page 292: ...ick Advanced and choose capacity management options as shown in Figure 6 21 Figure 6 21 Compressed Volume Advanced properties 4 The Compressed volume is created while the CLI command displays Figure 6...

Page 293: ...host Before you can map any volumes to a host the host has to be created first Here we demonstrate how to create new hosts in IBM Flex System V7000 Storage Node It is assumed that all correct zoning o...

Page 294: ...Figure 6 24 New Host creation select iSCSI or FC host 3 After choosing the host type give the host a name 4 Select the World Wide Names WWNs that represent the Host Bus Adaptor HBA ports in your serv...

Page 295: ...host configuration 275 5 Click Create Host after you finished adding WWNs Figure 6 26 shows that we have added our HBA ports and are now ready to create our new host Figure 6 26 New Host creation two...

Page 296: ...re 6 27 The newly created host appears 6 2 2 Creating an iSCSI attached host The process of creating a iSCSI attached host is almost similar to creating an FC attached host as shown in 6 2 1 Creating...

Page 297: ...iSCSI host 3 At the Create Host window type a name for the new host 4 Insert the iSCSI initiator name which you copied from the hosts iSCSI initiator software Figure 6 28 shows that we pasted in the...

Page 298: ...color red and Offline This indicates that no traffic has yet travelled over the connection To make it green and online you have to connect to the iSCSI attached host and from there discover the IBM Fl...

Page 299: ...various type of volumes available on IBM Flex System V7000 Storage Node The volume creation wizard gives the option to Create or Create and Map to Host If you clicked Create the next logical step is t...

Page 300: ...Map to host choose host 3 The Modify Mappings window opens and your host and the newly created volume are already selected Click OK and the volume is mapped to the host Figure 6 33 Figure 6 33 Map to...

Page 301: ...umes and store data on them Go to Chapter 12 Host configuration on page 511 to discover the volumes on the host and make some additional host settings if required 6 3 2 Additional features You can als...

Page 302: ...s been changed from 1024 to 2048 per cluster This increase in number of hosts can be used for any host type subject to limit restrictions for that host type Increased number of volumes or LUNs per hos...

Page 303: ...stem V7000 Storage Node Migrating the data allows applications to benefit from new features such as Easy Tier an intuitive management GUI and advanced storage replication functions that better support...

Page 304: ...V7000 Storage Node management GUI Settings General Licensing External Virtualization As part of the preparation work for data migration external storage systems must be configured on IBM Flex System...

Page 305: ...ifically for this scenario to guide you through the process and describe the steps along the way 7 2 1 Check the Windows 2008 host before the migration Before starting the actual migration using the I...

Page 306: ...ver Manager Disk Administrator take the disk offline so that the Microsoft Windows 2008 server does not lose connectivity to an online disk Shut down the Microsoft Windows 2008 server The server might...

Page 307: ...he Windows server and is mapped to IBM Flex System V7000 Storage Node instead Figure 7 3 DS4300 Storage Manager Disk Remapping Figure 7 4 shows that the disks are no longer mapped to the Windows serve...

Page 308: ...08 server and check that the disk has disappeared from the Server Manager Disk Administrator as verified in Figure 7 5 We also use the Rescan Disks option to confirm it Figure 7 5 Windows server with...

Page 309: ...Storage Node GUI Click Pools External Storage and click Detect MDisks as shown in Figure 7 6 Figure 7 6 Remapped disks are visible in the External Storage section There are two ways to access the men...

Page 310: ...System V7000 Storage Node Introduction and Implementation Guide Method 2 Click the Pools icon and click System Migration as shown in Figure 7 8 Figure 7 8 Migrate Storage from Pools menu System Migra...

Page 311: ...e Migration Wizard 291 Whichever method you choose the storage migration window opens Figure 7 9 Click Start New Migration to start the Storage Migration Wizard Figure 7 9 Storage Migration Wizard Sta...

Page 312: ...Next as shown in Figure 7 10 Figure 7 10 Storage Migration Wizard step 1 Before You Begin window Figure 7 10 shows there are some restrictions and prerequisites to using the Storage Migration Wizard...

Page 313: ...rage vMotion feature might be an alternative to migrating VMware clusters For more information about this topic see the following website http www vmware com products vmotion overview html Prerequisit...

Page 314: ...Migration Wizard step 3 Map Storage After you click Next IBM Flex System V7000 Storage Node starts to discover external devices if you have correctly zoned the external storage systems with IBM Flex...

Page 315: ...andidates for migration At this point you may need to reference the SCSI ID information that you recorded earlier to identify these MDisks individually In our example the MDisks have been selected to...

Page 316: ...host zones have been configured correctly If the host that needs to access the data on the volume after the migration is complete is in the list select the host and click Next If the host has not bee...

Page 317: ...during migration as shown in Figure 7 15 Figure 7 15 Storage Migration Wizard step 6 Map to host 7 Select the host you want to use for mapping the volume Click Next to continue with the Storage Migrat...

Page 318: ...map the volumes to the hosts select the volumes and click Map to Host A window opens with a drop down list of the hosts At this time IBM Flex System V7000 Storage Node will let you choose the host to...

Page 319: ...lect Storage Pool The destination storage pool of the data migration could be an external storage pool or an internal storage pool Make sure that there is enough space in the storage pool that you sel...

Page 320: ...ge Migration Wizard step 8 Migration has begun To check the status of migration you can go to the Pools System Migration window where you will see a progress bar indicating the percentage of migration...

Page 321: ...a is still being copied to the destination storage pool in the background the host can continue to access the disks and perform its operations normally since the migration is transparent to the host F...

Page 322: ...ou need to do is to Finalize the migration By Finalizing the migration the link between the two mirror copies is broken and the source disks at the IBM DS4300 return as unmanaged MDisks to IBM Flex Sy...

Page 323: ...ta migration in percentage completion on a progress bar in the migration window You can also find the target storage pool to which your volumes are being migrated to along with the status of the volum...

Page 324: ...Storage Migration Wizard Finalize migration step 1 After you select Finalize in the Action menu IBM Flex System V7000 Storage Node requires you to confirm the migration of the volumes Verify the volu...

Page 325: ...one and remove the older storage system from IBM Flex System V7000 Storage Node For more information about the IBM Flex System V7000 Storage Node migration function see Chapter 10 Volume mirroring and...

Page 326: ...ion Wizard you will need to complete this task now by following these few simple steps to map the disk to host 1 Figure 7 28 shows how to navigate to Pools Volumes By Pool from where we locate the vol...

Page 327: ...sts that are currently configured on IBM Flex System V7000 Storage Node and allow us to select the appropriate host we want to map the volume Figure 7 29 Modify Host Mappings Choose a host When the ho...

Page 328: ...ed in the first step now appears in the right side of the window and is ready to be mapped Figure 7 30 Modify Host Mappings You can change the SCSI ID of the new mappings through the Edit SCSI ID butt...

Page 329: ...iate or comply with any previous naming convention Figure 7 31 shows how we can review the volume name that has automatically been assigned to a migrated volume It can be done by selecting Volumes Vol...

Page 330: ...n Guide Figure 7 32 shows how we rename the volume by selecting Volumes Volumes by Host Actions Rename Figure 7 32 After Migration Rename volume step 1 Figure 7 33 shows how we can give the volume a m...

Page 331: ...Chapter 7 Storage Migration Wizard 311 Figure 7 34 shows that the volume has been renamed Figure 7 34 After Migration Volume Name after rename...

Page 332: ...312 IBM Flex System V7000 Storage Node Introduction and Implementation Guide...

Page 333: ...Storage Node manages physical storage resources All storage resources under IBM Flex System V7000 Storage Node control are managed using Storage pools Storage pools and the volumes that you build on t...

Page 334: ...ge Migration Wizard on page 283 we demonstrate how to discover and import LUNs from external disk array controllers and how to migrate data out of these and into IBM Flex System V7000 Storage Node You...

Page 335: ...orage pools 315 An alternative way to access the Internal Storage window is by clicking the Pools icon on the left side of the window Figure 8 2 Figure 8 2 Access the Internal Storage window using the...

Page 336: ...the drives by their type or class for example you can choose to show only SAS NL SAS or SSD drives Figure 8 3 Internal Storage window On the right side of the Internal Storage window the internal disk...

Page 337: ...nted in that column Figure 8 4 Internal Storage window details selection In addition you can find the current internal storage capacity allocation indicator at the upper right The Total Capacity shows...

Page 338: ...drives The Fix Error action starts the Directed Maintenance Procedure DMP for a defective drive The internal drives can be taken offline when there are problems on the drives by clicking Take Offline...

Page 339: ...hot spare if required Failed The drive was either intentionally taken offline or failed due to an error Click Mark as in the Actions drop down menu and select the role you want the drive to be assigne...

Page 340: ...n Figure 8 8 Drive identification LED feature Click Turn LED Off when finished working with it Clicking Show Dependent Volumes shows you the volumes that are dependent on the drives Clicking Propertie...

Page 341: ...re 8 10 Properties tab with details 8 1 2 Configuring internal storage To configure internal storage for use with hosts click Pools Internal Storage and click the Configure Storage button see Figure 8...

Page 342: ...specific goal for the number of drives per array the number of spare drives to maintain redundancy and whether the drives in the array are balanced across enclosure chains thus protecting the array fr...

Page 343: ...ent configuration on page 328 Preset Purpose RAID level Drives per array goal Spare goal Chain balance Basic RAID 5 Protects against a single drive failure Data and one stripe of parity are striped ac...

Page 344: ...12 Recommended configuration The following RAID presets are recommended for different drive classes SSD Easy Tier preset for solid state drives Basic RAID 5 for SAS drives Basic RAID 6 for Nearline SA...

Page 345: ...the recommended configuration the Wizard proposed to build three arrays using the Basic RAID 5 for the 146 GB drives discovered as 136 23GB and one basic RAID 5 for the 300 GB drives discovered as 278...

Page 346: ...duction and Implementation Guide The progress window opens Figure 8 14 The array is available for I O during this process The initialization does not affect the availability due to possible member dri...

Page 347: ...Figure 8 15 Figure 8 15 Capacity allocation after applying the recommended configuration If the proposed configuration does not meet your requirements click Select a different configuration on the ini...

Page 348: ...ers a more flexible way for the configuration of the internal storage as compared to the Use the recommended configuration preset in terms of drive selection RAID level and storage pool to be used Com...

Page 349: ...ne of the two optimization options Optimize for Performance The goal of this algorithm is to create arrays with identical capacity and performance characteristics to achieve the best possible performa...

Page 350: ...330 IBM Flex System V7000 Storage Node Introduction and Implementation Guide Figure 8 18 shows an example of the performance optimized setup Figure 8 18 Example of a performance optimized setup...

Page 351: ...ssign the capacity to Figure 8 19 You can either select an existing storage pool that does not contain MDisks or a pool containing MDisks with similar performance characteristics which is listed autom...

Page 352: ...2 IBM Flex System V7000 Storage Node Introduction and Implementation Guide Alternatively create a new storage pool and type in a name for the new storage pool Figure 8 20 Figure 8 20 Create storage po...

Page 353: ...Running Tasks status indicator and the capacity allocation indicator has been updated as well Optimize for Capacity The goal of this algorithm is to create a setup with the maximum usable capacity dep...

Page 354: ...e system provisions the available drives in even sized MDisks In our case we get two MDisks with six disks and one with only five In the Capacity Optimized setup all available Candidate disk drives ar...

Page 355: ...dow from the Overview window Another way to access the MDisks window is by using the Physical Storage function icon on the left Figure 8 24 Provisioned drives You can choose the number of drives to be...

Page 356: ...d Implementation Guide Figure 8 24 Access the MDisks by Pools window using the Physical Storage function icon The MDisks by Pools window Figure 8 25 lets you manage all MDisks made of both internal an...

Page 357: ...Groups Unmanaged The MDisk is not a member of any storage pools which means it is not being used by IBM Flex System V7000 Storage Node LUNs presented by external storage systems to IBM Flex System V70...

Page 358: ...e by storage subsystem capacity and so on Several MDisks can be selected by pressing the Ctrl key while clicking the MDisks listed with the mouse Also the Detect MDisks icon initiates a SAN discovery...

Page 359: ...V7000 Storage Node completes this action the MDisk shows up in the pool that it was added to Figure 8 30 Figure 8 30 MDisk added to pool Existing data If there is existing data on the unmanaged MDisks...

Page 360: ...ministrative purposes you may want to remove MDisks from storage pools to reorganize your storage allocation You can remove MDisks from storage pools by selecting the MDisks and clicking Remove from P...

Page 361: ...the MDisks from the pool select the Remove the MDisk from the storage pool even if it has data on it The system migrates the data to other MDisks in the pool check box Figure 8 32 Confirm the removal...

Page 362: ...continue to use their storage that is now presented by IBM Flex System V7000 Storage Node instead To achieve this configuration the existing external LUNs must be imported as an image mode volume usi...

Page 363: ...cide whether the volume is imported as a generic volume the default or as thin provisioned Caching for the volume can be disabled it is enabled by default Select Use thin provisioning only if the volu...

Page 364: ...8 36 Import Wizard step 2 of 2 You can check the migration progress in the Running Tasks status indicator as also shown in Figure 8 33 on page 342 or by clicking Pools System Migration After the migr...

Page 365: ...be used as a regular MDisk to host volumes Figure 8 38 Figure 8 38 MDisk mode changed to Managed after a destination pool was selected during Import If you do not select a destination pool during Ste...

Page 366: ...ed to Image when no destination pool was selected during Import If needed the image mode volume can be migrated manually into a different pool by selecting either Migration to Another Pool or Volume C...

Page 367: ...ture is normally used as a method to migrate data from existing external LUNs into storage pools either located internally or externally on IBM Flex System V7000 Storage Node You should not use image...

Page 368: ...d as a destination Figure 8 43 Migrate Image Mode Volume into a regular storage pool The migration itself internally uses the volume copy function which creates a second copy of the existing volume in...

Page 369: ...Chapter 8 Storage pools 349 Figure 8 44 Stop Migration When the migration progress reaches 100 click Finalize to complete the migration process Figure 8 45 Figure 8 45 Finalize volume migration...

Page 370: ...finalizes Figure 8 46 Volume Migration Finalizes At this point the migration of the Image mode disk has finished and the Migration Pool called MigrationPool_8192 will be empty although it has not disa...

Page 371: ...t the wanted number of spare drives that is required to protect the array from drive failures If the number of drives assigned as Spare does not meet the configured spare goal an error is logged in th...

Page 372: ...r message in the event log Select an MDisk containing the drive to be replaced and click RAID Actions Swap Drive In the Swap Drive window Figure 8 48 select the member drive to be replaced and click N...

Page 373: ...indow Data on MDisks is migrated to other MDisks in the pool assuming enough space is available on the remaining MDisks in the pool After deleting an MDisk from a pool its member drives return to cand...

Page 374: ...e changed manually by the user To assign a specific tier to an MDisk click Pools MDisks by Pool and click Select Tier from the Actions drop down menu Figure 8 51 Figure 8 51 Select Tier for an MDisk F...

Page 375: ...ny new MDisks that have been mapped to the IBM Flex System V7000 Storage Node storage system and rebalances MDisk access across the available controller device ports This action also detects any loss...

Page 376: ...n menu Input the new name of your MDisk and click Rename Figure 8 54 Figure 8 54 Rename MDisk Show Dependent Volumes The volumes dependent on a particular MDisk can be displayed by selecting the MDisk...

Page 377: ...elect the MDisk and click Properties from the Actions drop down menu A window opens as shown in Figure 8 56 Figure 8 56 MDisk overview with general information There are three tabs in this information...

Page 378: ...Drives tab you find all the member drives of this MDisk Figure 8 58 Also all actions described in 8 1 1 Actions on internal drives on page 318 can be performed on the drives listed here Figure 8 58 M...

Page 379: ...ls window opens It was shown previously in Figure 8 23 on page 335 The other way to access the Pools window is to click the icon to the left that shows a physical hard disk the Pools icon The MDisk by...

Page 380: ...d from here as well When you expand a pool s entry by clicking the plus sign to the left of the pool s icon you get access to the MDisks associated with this pool You can perform all actions on them a...

Page 381: ...olumes host mappings and MDisks that are associated with this pool check box and click Delete Figure 8 63 Confirm the deletion of the pool In this example we are deleting a migration pool that has an...

Page 382: ...ge mode MDisk returns as Not in a Pool All the managed or image mode MDisks in the pool return to a status of unmanaged after the pool is deleted If an MDisk pool is deleted all the array mode MDisks...

Page 383: ...m V7000 Storage Node are provided by the shared code of the IBM SAN Virtual Controller SVC and the IBM Storwize V7000 and all function in the same manner This commonality allows for the pairing of the...

Page 384: ...t its configuration and use You can use FlashCopy to solve critical and challenging business needs that require duplication of data on your source volume Volumes can remain online and active while you...

Page 385: ...ment and testing data must be restored from traditional backup media such as tape Depending the amount of data and the technology in use this process could easily take a day or more With FlashCopy a c...

Page 386: ...g completed in the background Also the handling of the redirection of the host I O which is being written to the source volume with respect to a T0 point in time while the target volume is held true t...

Page 387: ...mappings If a single source volume has multiple target FlashCopy volumes the write to the source volume does not cause its data to be copied to all of the targets Instead it is copied to the newest t...

Page 388: ...data copied per second as the background copy rate Table 9 1 The cleaning rates controls the rate at which the cleaning process operates The purpose of the cleaning process is to copy or flush data f...

Page 389: ...for the source volume is flushed from the cache Any read or write data for the target volume is discarded from the cache If the mapping is incremental and a previous mapping has completed the mapping...

Page 390: ...orms the operation on all FlashCopy mappings contained within the consistency group Figure 9 2 shows a consistency group consisting of two FlashCopy mappings Figure 9 2 FlashCopy consistency group Dep...

Page 391: ...ross all the volumes would be consistent A FlashCopy consistency group can contain up to 512 FlashCopy mappings The more mappings you have the more time it takes to prepare the consistency group Flash...

Page 392: ...mapping in the consistency group is in the Suspended state Empty The consistency group does not have any FlashCopy mappings Reverse FlashCopy Reverse FlashCopy enables FlashCopy targets to become res...

Page 393: ...y groups cannot contain more than one FlashCopy map with the same target volume 9 2 3 Planning for FlashCopy There are several items that must be considered before performing a FlashCopy which we desc...

Page 394: ...target volume with 0 of the capacity allocated at the time of creation The preset uses a FlashCopy mapping with 0 background copy so that only data written to the source or target is copied to the ta...

Page 395: ...windows for managing FlashCopy under the Copy Services function icon including FlashCopy Consistency Groups Figure 9 4 shows the Copy Services function icon Figure 9 4 Copy Services function icon Mos...

Page 396: ...ndow opens Figure 9 5 In the FlashCopy window the FlashCopy mappings are organized by volumes Figure 9 5 FlashCopy window Click FlashCopy Mappings in the Copy Services function icon menu and the Flash...

Page 397: ...and the Consistency Groups window opens Figure 9 7 Figure 9 7 Consistency Groups window Quick path to create FlashCopy presets It is easy to create FlashCopy using the presets in the FlashCopy window...

Page 398: ...ction and Implementation Guide Creating a clone In the FlashCopy window choose a volume and click New Clone from the Actions drop down menu Figure 9 9 Figure 9 9 Create a clone from the preset You now...

Page 399: ...de Copy Services 379 Creating a backup In the FlashCopy window choose a volume and click New Backup from the Actions drop down menu Figure 9 10 Figure 9 10 Create a backup from the preset You now have...

Page 400: ...e Figure 9 11 The progress bars behind the target volumes indicate the copy progress in percentage The copy progress remains 0 for snapshot there is no change happening at the moment because only the...

Page 401: ...us indicator Figure 9 12 The FlashCopy progress After the copy progresses complete you find the FlashCopy with the clone preset has been deleted automatically Figure 9 13 Figure 9 13 FlashCopy progres...

Page 402: ...shCopy You have two options Click Create new target volumes if you have not created the target volume yet a After you click Create new target volumes the wizard lets you choose the preset but no matte...

Page 403: ...vanced Settings which are Background Copy 0 Incremental No Auto Delete after completion No Cleaning Rate 0 It is the same with clone preset and the default settings of clone preset can be found when y...

Page 404: ...he backup preset Change the settings of the FlashCopy mapping according to your requirements and click Next b In the next step you could add your FlashCopy mapping to a consistency group Figure 9 19 I...

Page 405: ...ge pool for your new volume Figure 9 21 Click Next to continue Figure 9 21 Choose another storage pool to create the volume e You are prompted to select how you want to manage volume capacity Create a...

Page 406: ...e 9 23 Figure 9 23 The default choice when creating a FlashCopy mapping f Click Finish when you make your decision and the FlashCopy mapping is created on your volume with a new target Figure 9 24 The...

Page 407: ...umes select Use existing target volumes Figure 9 25 Figure 9 25 Create FlashCopy mapping using existing target volume a You need to choose the target volume for the source volume you selected Select t...

Page 408: ...ant click Next to continue Figure 9 27 Add FlashCopy mapping c Select the preset and adjust the settings Figure 9 28 Make sure that the settings meet your requirements and click Next Figure 9 28 Selec...

Page 409: ...g using existing target volumes in the FlashCopy window Starting a FlashCopy mapping Most of the actions of FlashCopy mapping can be taken both in the FlashCopy window or the FlashCopy Mapping window...

Page 410: ...ashCopy mapping The FlashCopy mapping can be stopped by selecting the FlashCopy target volume in the FlashCopy window and clicking the Stop option from the Actions drop down menu Figure 9 32 After the...

Page 411: ...a suffix containing numbers The name of the target volumes could be changed to be more meaningful in your environment To change the name of the target volume select the FlashCopy target volume in the...

Page 412: ...e changed to be more meaningful to you To change the name of a FlashCopy mapping select the FlashCopy mapping in the FlashCopy Mappings window and click the Rename Mapping option from the Actions drop...

Page 413: ...d to confirm your action to delete FlashCopy mappings in the window that opens Figure 9 38 Verify the number of FlashCopy mappings that you need to delete and if you want to delete the FlashCopy mappi...

Page 414: ...the Show Dependent Mappings option from the Actions drop down menu Figure 9 39 Figure 9 39 Show Dependent Mappings menu The FlashCopy mapping dependency tree opens Figure 9 40 Figure 9 40 FlashCopy m...

Page 415: ...n created by selecting the FlashCopy target volume in the FlashCopy window and clicking the Edit Properties option from the Actions drop down menu Figure 9 41 Figure 9 41 Edit Properties menu You can...

Page 416: ...he Consistency Groups window The Consistency Groups window Figure 9 44 is where you can manage both consistency groups and FlashCopy mappings Figure 9 44 Consistency Groups window In the left pane of...

Page 417: ...page 396 You are prompted to enter the name of the new consistency group Figure 9 45 Following your naming conventions enter the name of the new consistency group in the box and click Create Figure 9...

Page 418: ...ghting a mapping pressing and holding the Shift key and clicking the last item in the range Multiple selections can be made by pressing and holding the Ctrl key and clicking each mapping individually...

Page 419: ...click Start from the Actions drop down menu Figure 9 49 Figure 9 49 Start a consistency group After you start the consistency group all the FlashCopy mappings in the consistency group are started at...

Page 420: ...the stop process completes the FlashCopy mappings in the consistency group are in the Stopped state and a red X icon appears on the function icon of this consistency group to indicate an alert Figure...

Page 421: ...cking Remove from Consistency Group from the Actions drop down menu of the FlashCopy mapping Figure 9 53 Figure 9 53 Remove from consistency group The FlashCopy mappings are returned to the Not in a G...

Page 422: ...e separated by distance but distance is not a requirement After the partnership creation has been configured on both systems further communication between the node canisters in each of the storage sys...

Page 423: ...m an IBM Storwize V7000 as external storage for it to manage The layer for IBM Flex System V7000 Storage Node can be switched by running svctask chsystem layer replication Generally switch the layer w...

Page 424: ...uster where you are making changes you cannot change the layer If any host object is defined to an IBM Flex System V7000 Storage Node SAN Volume Controller or IBM Storwize V7000 system attached you ca...

Page 425: ...re 9 58 Star topology for IBM Flex System V7000 Storage Node Full meshed topology Figure 9 59 Figure 9 59 Full meshed IBM Flex System V7000 Storage Node Partnerships These partnerships are also valid...

Page 426: ...ote cluster is present but the remote cluster is stopped Fully Configured Local Excluded Indicates that a partnership is defined between a local and remote cluster however the local cluster has been e...

Page 427: ...in a relationship must be the same size The Remote Copy relationship can be established on the volumes within one IBM Flex System V7000 Storage Node which is called an intra cluster relationship The r...

Page 428: ...nd understand your workload In asynchronous Remote Copy which Global Mirror provides write operations are completed on the primary site and the write acknowledgement is sent to the host before it is r...

Page 429: ...and applying those sets of I Os in the assigned order at the secondary In a failover scenario where the secondary site needs to become the master source of data depending on the workload pattern and t...

Page 430: ...olumes are used at both the primary and secondary end of the replication link At the primary the change volume stores copy on write changes from primary that might need to be sent to secondary to pres...

Page 431: ...er then these updates are wasted network traffic and the I O can be completed much more efficiently Figure 9 65 Figure 9 64 Global Mirror I O replication without Change Volumes In Figure 9 65 the same...

Page 432: ...can be issued to a Remote Copy consistency group and therefore simultaneously for all Metro Mirror relationships defined within that consistency group or to a single Metro Mirror relationship that is...

Page 433: ...ite I O operations but the secondary volumes are not accessible for either one A copy process must be started to make the secondary volumes consistent InconsistentCopying The primary volumes are acces...

Page 434: ...o Mirroring is supported over either an FC or FCoE FC SAN infrastructure Global Mirroring function supports copy operations between volumes that are separated by distances over 300 km Global Mirroring...

Page 435: ...of enclosures that are authorized for the feature This topic is detailed in 2 4 IBM Flex System V7000 Storage Node licensing on page 46 Remote Copy configuration limits Table 9 4 lists the Remote Cop...

Page 436: ...trip latency requirement Bandwidth The bandwidth must satisfy the following requirements If you are not using Change Volumes Be able to sustain peak write load for all mirrored volumes and background...

Page 437: ...the data If the relationship does not reach the synchronized state you can use the FlashCopy target volume at the secondary site 9 4 Troubleshooting Remote Copy Remote Copy Global Mirror and Metro Mi...

Page 438: ...the devices add 200 s of delay each way The distance adds 9 5 ms each way for a total of 19 ms Combined with the device latency that is 19 4 ms of physical latency at a minimum This latency is under...

Page 439: ...8 there are orders of magnitude of difference between the different link bandwidths It is easy to see how 1920 errors can arise when your bandwidth is insufficient and why you should never use a TCP I...

Page 440: ...tential SAN congestion Anytime a zero buffer credit timer is above 2 of the total time of the sample interval it is likely to cause problems Next always ask your network provider to check the status o...

Page 441: ...nership is defined in our example Figure 9 70 so you must create a partnership between the IBM Flex System V7000 Storage Node system and the other mirroring member Click New Partnership in the Partner...

Page 442: ...System_V7000 and ITSO_SVC1 The bandwidth you need to input here is used by the background copy process between the clusters in the partnership To set the background copy bandwidth optimally make sure...

Page 443: ...ted between two IBM Flex System V7000 Storage Node systems and both systems are ready for further configuration of Remote Copy relationships Figure 9 74 Figure 9 74 Fully configured partnership You ca...

Page 444: ...tion Guide Stopping and starting a partnership You can stop the partnership by clicking Stop Partnership from the Actions drop down menu Figure 9 76 If you stop the partnership the relationship using...

Page 445: ...ted as Fully Configured Stopped Figure 9 77 Figure 9 77 Fully configured partnership in stopped state You can restart a stopped partnership by clicking Start Partnership from the Actions drop down men...

Page 446: ...ership Managing stand alone Remote Copy relationships A Remote Copy relationship can be defined between two volumes where one is the master source and the other one is the auxiliary target volume Usag...

Page 447: ...all the Remote Copy relationships that are not in any Remote Copy consistency groups are displayed on the right Creating stand alone Remote Copy relationships To create a Remote Copy relationship clic...

Page 448: ...th the same volume size as the selected master volume After you select the volumes based on your requirement click Add Figure 9 84 Select the master and auxiliary volume You can define multiple and in...

Page 449: ...9 86 Activate initial data copy If you select Yes the volumes are already synchronized in this step a warning message opens Figure 9 87 Make sure that the volumes are truly identical and then click OK...

Page 450: ...dependent Remote Copy relationships are defined and displayed in the Not in a Group list Figure 9 89 Figure 9 89 Remote Copy relationship create completes Optionally you can monitor the ongoing initia...

Page 451: ...menu Figure 9 91 Figure 9 91 Stop Remote Copy relationship Allow secondary read write access if required and click Stop Relationship Figure 9 92 Figure 9 92 Option to allow secondary read write access...

Page 452: ...m the Actions drop down menu Figure 9 94 Figure 9 94 Start a Remote Copy relationship When starting a Remote Copy relationship the most important item is selecting the copy direction Both master and a...

Page 453: ...9 97 If you switch the Remote Copy relationship the copy direction of the relationship becomes the opposite that is the current primary volume becomes the secondary while the current secondary volume...

Page 454: ...you that the primary volume in this relationship has been changed Figure 9 98 Note the switch icon on the state of the relationship Renaming a stand alone Remote Copy relationship The Remote Copy rela...

Page 455: ...ting a stand alone Remote Copy relationship The Remote Copy relationship can be deleted by selecting the relationship and clicking Delete Relationship from the Actions drop down menu Figure 9 101 Figu...

Page 456: ...enter a name for your new consistency group Figure 9 104 Figure 9 104 Enter a name for the new consistency group You can choose to create an empty consistency group or if you want add Remote Copy rel...

Page 457: ...group by clicking the name and entering a new one At the upper right of the Remote Copy window you can take actions on the Remote Copy consistency group Below the Relationship function icon you find a...

Page 458: ...Your Remote Copy relationships are now in the consistency group you selected Starting a consistency group The Remote Copy relationship could be started by clicking Start from the Actions drop down me...

Page 459: ...stopped by clicking Stop in the Actions drop down menu Figure 9 111 Figure 9 111 Stop the consistency group You can allow read write access to secondary volumes by selecting the check box Figure 9 112...

Page 460: ...y group click Switch from the Actions drop down menu Figure 9 113 Figure 9 113 Switch the copy direction of a consistency group A warning message opens Figure 9 114 After the switch the primary cluste...

Page 461: ...Remote Copy relationships from a consistency group You are prompted to confirm the Remote Copy relationships you want to delete from the consistency group Figure 9 116 Make sure the Remote Copy relati...

Page 462: ...up The consistency group is deleted Changing Remote Copy type of Relationship or Consistency Group Starting from software 7 1 0 it is possible to change the copy type MM GM or Cycling mode of a Remote...

Page 463: ...M to GM Cycling Mode As the first action you need to stop the relationship Click Stop from the Actions drop down menu as shown in Figure 9 119 Figure 9 119 Stopping a Standalone Relationship After sto...

Page 464: ...e 9 122 Optional Cycling mode setting Finally restart your relationship Click Start from the Actions drop down menu as shown in Figure 9 123 Figure 9 123 Start of Remote Copy Relationship Important At...

Page 465: ...t you can use the same procedure to change the Consistency Group from MM to GM or from GM to GM Cycling Mode As the first action you need to stop the Consistency Group Click Stop from the Actions drop...

Page 466: ...window appears as shown in Figure 9 127 then click OK to change the type of relationship from GM to MM Figure 9 127 Selecting Metro Mirror copy type Finally restart your relationship Click Start from...

Page 467: ...veloped as a part of the IBM SAN Volume Controller SVC and is shared with IBM Flex System V7000 Storage Node It provides an ability to mirror data between storage pools in the same I O group of the IB...

Page 468: ...two value settings to choose from either latency or redundancy The latency tuning option is the default and prioritizes host I O latency It is set by running the command chvdisk mirrowritepriority lat...

Page 469: ...ows you to migrate data from an already existing volume or LUN on another external storage subsystem into IBM Flex System V7000 Storage Node as an image mode mdisk With this function the volume can th...

Page 470: ...n the Volume Mirror process first select the Volume by Pool from the storage Pool icon drop down list as shown in Figure 10 1 Figure 10 1 Volumes by Pool from the Storage Pool icon To make a volume mi...

Page 471: ...ror that is generic thin provisioned or compressed and select the storage pool to use for the copy Figure 10 3 Make sure that you select a storage pool with sufficient space and similar performance ch...

Page 472: ...mirroring progress percentage by highlighting the volume and selecting the Volume Synchronization tab in the Running Tasks radial Figure 10 5 Figure 10 4 The distribution of extents for primary and m...

Page 473: ...Volume on either Copy0 or Copy1 you can specify a new volume name and confirm the split Figure 10 7 Figure 10 7 Confirmation of volume mirror split After providing a new volume name optional and confi...

Page 474: ...ing the Migrate to Another Pool option Volume Mirror advantages over Migrate to Another Pool When you move data Volume Mirror has always two copies of data With Volume Mirror you can add a copy on a l...

Page 475: ...s the current pool of each volume copy so you can better determine which storage pool to use If you need to move volumes between Storage Pools or you need to migrate a volume from Thin Provisioned to...

Page 476: ...456 IBM Flex System V7000 Storage Node Introduction and Implementation Guide...

Page 477: ...for storage provisioning It includes the three types of storage area network Fibre Channel FC iSCSI and FC over Ethernet FCoE To do this we look at the configuration of two of the I O module models th...

Page 478: ...and faster Ethernet communications to a host server 10 Gbps is becoming more widely available and used for host server connections Higher speeds of 40 Gbps Ethernet are becoming available and a 100 G...

Page 479: ...on IBM Flex System V7000 Storage Node IBM Flex System EN2024 4 port 1 Gb Ethernet Adapter IBM Flex System EN4054 4 port 10 Gb Ethernet Adapter IBM Flex System FC3172 2 port 8 Gb FC Adapter IBM Flex Sy...

Page 480: ...four serial links to an I O module switch The Network Adapter port uses only one of these four lanes per switch the unused lanes are provided for future options Each of these individual I O links or...

Page 481: ...that the iSCSI SAN uses Ethernet instead of FC transport technology The iSCSI protocol uses TCP to transport SCSI I O commands over a network It provides block level data access where it encapsulates...

Page 482: ...t Protocol network safely and use appropriate quality of service QoS procedures iSCSI security The iSCSI can be used in networks where unauthorized data can be accessed allowing for different security...

Page 483: ...hardware initiators offload TCP stack processing functions and iSCSI command processing functions The hardware adapter looks and functions similar to a SCSI disk interface just as an FC HBA does The...

Page 484: ...name plus the iSCSI initiator session ID ISID plus the target portal group tag target side ID The Flex System V7000 Storage Node multisession approach is to have relationships between multiple initia...

Page 485: ...1 3 4 iSCSI multipath connectivity You can choose from several technologies to improve the availability of iSCSI by using multiple network paths Multipathing requires a software layer above the iSCSI...

Page 486: ...ompute node has two Ethernet adapter ports that are already configured with IP addresses This example does not take into account of network planning required or configuration on the Ethernet switch or...

Page 487: ...2145 flexsystemv7000 node2 They are used by the host to connect to IBM Flex System V7000 Storage Node When the canister ports are configured as shown in Table 11 1 the link should be tested by the hos...

Page 488: ...ng a new set of Data Center Bridging DCB standards that enhance existing 802 1 bridge definitions The enhancements provide a converged network that allows multiple applications to run over a single ph...

Page 489: ...such as port types and lower layer initialization protocols must also be replaced by new constructs in FCoE Such mappings are defined by the FC BB 5 standard Figure 11 8 FCoE protocol mapping Figure 1...

Page 490: ...ter HBA for storage traffic With converged networks that might include FCoE these two adapters are replaced by a converged network adapter CNA that services both protocol stacks A cable from the CNA c...

Page 491: ...n the IBM Flex System Enterprise Chassis where the CN4093 provides the FCF switching functionality The CN4093 switch with its FCF function and FC ports can connect to external Fibre Channel SANs In th...

Page 492: ...which are the VN_Port to VF_Port virtual links as shown in Figure 11 11 FIP entails the following high level steps 1 The end device or compute node broadcasts a FIP VLAN request to the CN4093 and any...

Page 493: ...n FC switching element is that it joins the FC fabric as a domain It will give it the CN4093 capability of switching data between the compute node using FCoE and an external storage controller attache...

Page 494: ...For a compute node to connect to IBM Flex System V7000 Storage Node and access storage the VLAN must have FCF enabled Thus all storage data will remain within the IBM Flex System Enterprise Chassis an...

Page 495: ...SSH connection over the network 3 Using the Browser Based Interface BBI over the network 4 Using a serial connection by the serial port on the CN4093 The Telnet SSH connection accesses two types of C...

Page 496: ...ldp LLDP TX RX are disabled on port EXT11 Oct 26 14 22 12 fd8c 215d 178e c0de 3640 b5ff fe5c f5ef NOTICE lldp LLDP TX RX are disabled on port EXT12 Router config vlan 1002 VLAN 1002 is created VLAN Cr...

Page 497: ...ns are established between IBM Flex System V7000 Storage Node on ports INTA3 4 and the Compute Node 11 in bay 11 has FCoE configured and also established a connection from port INTA11 Example 11 4 Che...

Page 498: ...3 and Canister 2 uses INTA4 Figure 11 14 Active 10 Gb Ethernet on Canister 2 Creating FCoE zoning with the GUI Using CN4093 GUI by creating a zone which will include PWWNs of the host Compute Node 11...

Page 499: ...gement address of CN4093 as shown in Figure 11 15 the default user is USERID and the password is PASSW0RD with a zero instead of the letter O Figure 11 15 Login screen Then click the switch Dashboard...

Page 500: ...shown in Figure 11 17 Figure 11 17 Configure tab Then click the FC FCoE folder on the left side of your screen as shown in Figure 11 18 Figure 11 18 FC FCoE folder At this point you are redirected to...

Page 501: ...and configuration 481 Open the Zoning folder and then click FC Alias as shown in Figure 11 20 Figure 11 20 Zoning FC Alias folders Then click Add to create the new alias as shown in Figure 11 21 Figu...

Page 502: ...is A_FSV7K_N11 and the WWPN as shown in Figure 11 22 then click Submit Figure 11 22 Define the name and wwpn of the new Alias To create the Alias you can choose to use a manual entry of WWPN or pick...

Page 503: ...hapter 11 SAN connections and configuration 483 Figure 11 24 Create new Zone Add a new Zone in our example it is Z_FSV7K_N11 and then click Submit as shown in Figure 11 25 Figure 11 25 Define Zone nam...

Page 504: ...ew defined zone as shown in Figure 11 26 Figure 11 26 List of Zones By clicking the Zone name you get the Zone Configuration panel where you can use the Add to Zone button to add further aliases to th...

Page 505: ...create a new Zoneset as show in Figure 11 28 on page 485 Figure 11 28 Zoneset Configuration Write the name of the new Zoneset in our example CN4093_1 and click on Submit as shown in Figure 11 29 Figur...

Page 506: ...ction and Implementation Guide Figure 11 30 List of Zoneset A list of created zones is displayed highlight the zone you created and then click on Add as shown in Figure 11 31 to add to the newly defin...

Page 507: ...r 11 SAN connections and configuration 487 Select Activate to make the Zoneset active as shown in Figure 11 32 Figure 11 32 Select activate Zonset Configuration Then click on Submit as shown in Figure...

Page 508: ...d storage can be accessed by the O S platform residing on Compute Node 11 The zoning steps are exactly the same as when using the GUI and are very much the same as is used on FC zoning Create Zone Cre...

Page 509: ...Zoneset CN4093_1 has 1 zones zoneset name CN4093_1 zone name FSV7K_N11 pwwn 50 05 07 68 05 04 02 50 pwwn 50 05 07 68 05 04 02 51 pwwn 10 00 5c f3 fc 6e 30 01 Default Zone Deny Router config After thi...

Page 510: ...b Converged Scalable Switch in the IBM Flex System Enterprise Chassis are configured the second 10 Gb Ethernet port in each canister becomes active completing the configuration Figure 11 35 shows Cani...

Page 511: ...er high performance Fibre Channel uses a serial data transport scheme similar to other computer networks streaming packets frames of bits one behind the other in a single data line to achieve high dat...

Page 512: ...low control methodology Fibre Channel is able to deliver data as fast as the destination device buffer is able to receive it And low transmission overheads enable high sustained utilization rates with...

Page 513: ...Upper layers The upper layer includes two layers FC 3 and FC 4 Common services FC 3 FC 3 defines functions that span multiple ports on a single node or fabric The following functions are currently sup...

Page 514: ...ibre Channel topologies Fibre Channel provides three distinct and one hybrid interconnection topologies By having more than one interconnection option available a particular application can choose the...

Page 515: ...it Node_name assigned by the manufacturer The node name will be unique if the manufacturer has registered a range of addresses with the IEEE and so is normally referred to as a World Wide Name An N_Po...

Page 516: ...hannel switches in the IBM Flex System Enterprise Chassis I O Module slots In this section we will look at the connectivity between the FC switches in IBM Flex System Enterprise Chassis I O Module slo...

Page 517: ...1 If possible make the Alias using WWPNs soft zoning 2 When you connect internal switches to an external SAN to optimize bandwidth and for the best high availability use trunking also known as port ch...

Page 518: ...he next example we show how to connect to log in and start the zoning configuration process on one FC3171 8Gb SAN Switch The procedure must be repeated on the second switch in the IBM Flex System Ente...

Page 519: ...tion Figure 11 41 has the following settings indicated External port connection from external storage controller IBM DS4000 x240 compute node FC adapter in Bay 11 adapter slot 2 IBM Flex System V7000...

Page 520: ...rage Node Introduction and Implementation Guide Figure 11 42 shows from the main menu how to access and configure zoning by clicking Zoning and selecting the Edit Zoning option Figure 11 42 Access the...

Page 521: ...is displayed Proceed as follows 1 Select the option and click to create a zoneset 2 When the Zoneset creation box displays type the zoneset name When you have named the zoneset and clicked OK the new...

Page 522: ...ias name A WWN has to be added to the alias just created In Figure 11 45 with the mouse highlight the alias you want to use and the WWN that you want to add In this example we have created the alias R...

Page 523: ...k that the WWN has been added to the alias as shown in Figure 11 46 Figure 11 46 Check that WWN has been added to alias Creating a zone We now want to create a zone and add to it aliases that are alre...

Page 524: ...Guide Figure 11 47 Create and name a zone Adding an alias to a zone Figure 11 48 shows the highlighted alias where it is dragged using the mouse onto the required zone It is repeated for all aliases r...

Page 525: ...een created and placed in the zoneset all that is required now is to activate the zoneset To do so highlight the zone that you want to activate with the mouse and click Apply in the bottom right side...

Page 526: ...n the chassis 11 5 5 Multipathing The previous example has created a single zone which includes the initiator from the x240 compute node and two targets one from each of the storage node canisters Fro...

Page 527: ...also be configured into Transparent Mode where it will connect to an external FC SAN switch using F_Port N_Port instead of E Port E Port connectivity From the external SAN and switches all zoning con...

Page 528: ...ic has a theoretical limit of around 250 It also has practical limit generally in the range 50 75 The recommended limits vary by vendor In Transparent Mode on the edge of an FC SAN fabric NPIV is used...

Page 529: ...BA and Ethernet network interface card NIC in a CNA card FCoE is a relatively new storage protocol and is increasing in popularity with Ethernet speeds to exceed that of FC FCoE may well displace FC a...

Page 530: ...510 IBM Flex System V7000 Storage Node Introduction and Implementation Guide...

Page 531: ...opyright IBM Corp 2013 All rights reserved 511 Chapter 12 Host configuration This chapter describes how to configure hosts to access the volumes that are created on IBM Flex System V7000 Storage Node...

Page 532: ...re operating systems IBM Power Systems These compute nodes come with IBM Power7 microprocessors and provide the function reliability and performance of the Power7 systems in a small form factor design...

Page 533: ...System Storage Interoperability Center SSIC to check which code levels are supported to attach your host to your storage SSIC is a web tool that checks the interoperation of host storage switches and...

Page 534: ...equipment we have for demonstrating volume attachment for a Fibre Channel FC capable server is an IBM Flex System x240 Compute Node running Microsoft Windows 2012 Standard Version 6 2 9200 Build 9200...

Page 535: ...ogram Files IBM SDDDSM and enter the command gethba to determine the WWPN of the FC adapters as shown in Example 12 1 Example 12 1 Check WWPN from Windows PowerShell PS C Users Administrator cd C Prog...

Page 536: ...ers 2 Adpt Name State Mode Select Errors Paths Active 0 Scsi Port1 Bus0 NORMAL ACTIVE 3274 0 4 4 1 Scsi Port2 Bus0 NORMAL ACTIVE 3189 0 4 4 PS C Program Files IBM SDDDSM 5 Enter datapath query device...

Page 537: ...t2 Bus0 Disk3 Part0 OPEN NORMAL 23 0 PS C Program Files IBM SDDDSM 6 The output provides information about the connected volumes In our example there are two disks connected Disks 2 and 3 for the Wind...

Page 538: ...ct Online Figure 12 7 Figure 12 7 Place a disk online 8 Right click the disk again click Initialize Disk Figure 12 8 and click OK Figure 12 8 Initialize Disk option 9 Right click the right pane and cl...

Page 539: ...l need to be allocated and a New Simple Volume created by right clicking the volume as shown in Figure 12 10 Figure 12 10 Create Volume type 11 When the volume type has been selected the Simple Volume...

Page 540: ...r several paths and is able to store data on the storage subsystem 12 3 2 Windows 2008 R2 iSCSI volume attachment The equipment we have for demonstrating volume attachment for an iSCSI capable server...

Page 541: ...s view ISCSI host selected Figure 12 14 Mapped Volumes to an iSCSI host The host details show you which volumes are currently mapped to the host and you also see the volume UID and the SCSI ID In our...

Page 542: ...nter the IP address of one of the IBM Flex System V7000 Storage Node iSCSI ports and click Quick Connect Figure 12 16 Figure 12 16 iSCSI Quick Connect iSCSI IP addresses The iSCSI IP addresses are dif...

Page 543: ...ly using a single path at the moment To enable multipathing for iSCSI targets more actions are required Complete the following steps 1 Click Start Run and type cmd to open a command prompt Enter Serve...

Page 544: ...nd Implementation Guide 2 Click Start Administrative Tools MPIO click the Discover Multi Paths tab and select the Add support for iSCSI devices check box Figure 12 18 Figure 12 18 Enable iSCSI MPIO 3...

Page 545: ...nitiator to open the iSCSI Configuration tab and navigate to the Discovery tab Figure 12 19 Figure 12 19 iSCSI Properties Discovery tab 5 Click Discover Portal enter the IP address of another IBM Flex...

Page 546: ...Guide 6 Return to the Targets tab Figure 12 21 and you find the new connection there is listed as Inactive Figure 12 21 Inactive target ports 7 Highlight the inactive port and click Connect The Connec...

Page 547: ...ox and click OK The second port now appears as Connected Figure 12 23 Figure 12 23 Second target port connected Repeat this step for each IBM Flex System V7000 Storage Node port you want to use for iS...

Page 548: ...et the disk online initialize it create a file system on it and then it is ready to use The detailed steps of this process are similar to those described in 12 3 1 Windows 2012 R2 Fibre Channel volume...

Page 549: ...ws server iSCSI Initiator Applet menu Figure 12 26 shows the IBM Flex System V7000 Storage Node canister IQNs defined From this window click the Properties box with the mouse for node1 Figure 12 26 Ca...

Page 550: ...target IP addresses of the iSCSI configuration Figure 12 28 iSCSI target IP addresses and adapter portal IPs The Favorite Targets tab shows two entries for each canister as shown in Figure 12 29 wher...

Page 551: ...hted and Devices box clicked Figure 12 30 Two Disk 1 s identified on first target From here there are two disks of the same name identified as Target 0 and Target 1 They represent two paths one from e...

Page 552: ...ticator sends a challenge message to the specific server peer The server responds with a value that is checked by IBM Flex System V7000 Storage Node If there is a match IBM Flex System V7000 Storage N...

Page 553: ...Storage Node It is done on the FCoE capable switches and includes configuring VLANs For more information about how to configure VLANs on the CN4093 FCoE switches see Chapter 11 SAN connections and con...

Page 554: ...operties Mapped Volumes for the host This panel displays which Fibre Channel ports are defined for the host FCoE ports displays as if they were ordinary FC ports showing World Wide Names WWNs for the...

Page 555: ...hich drivers are needed for the Windows 2008 R2 server Figure 12 35 FCoE drivers are installed 5 On the Windows 2008 R2 server install appropriate multipathing software Multipathing software for FCoE...

Page 556: ...x System V7000 Storage Node volumes are mapped to the Windows 2008 R2 server Figure 12 36 Emulex OneCommand Manager 7 On the Windows 2008 R2 server click Start All Programs Subsystem Device Driver DSM...

Page 557: ...t note that the UID of the mapped IBM Flex System V7000 Storage Node volume reflects the SERIAL number in the output from datapath query device 8 Open the Windows Disk Management window by clicking St...

Page 558: ...ows 2008 R2 server is prepared to access the volumes over four paths and is able to store data on the storage subsystem For more information about FCoE see Chapter 11 SAN connections and configuration...

Page 559: ...Chapter 12 Host configuration 539 Figure 12 39 Example ESX FC host Navigate to the Mapped Volumes tab Figure 12 40 Figure 12 40 Mapped Volumes to ESX FC host...

Page 560: ...o the ESX FC host using SCSI ID 1 The UID of the volume is also displayed Connect to your VMware ESX Server using the vSphere client navigate to the Configuration tab and select Storage Adapters Figur...

Page 561: ...ck Add Storage Figure 12 43 Figure 12 43 vSphere Client Storage 5 The Add Storage wizard opens Click Select Disk LUN and click Next The IBM Flex System V7000 Storage Node disk appears Figure 12 44 Hig...

Page 562: ...he disk After you click Finish the wizard closes and you return to the storage view In Figure 12 45 you see that the new volume has been added to the configuration Figure 12 45 Add Storage task comple...

Page 563: ...Paths to customize the multipath settings Select Round Robin Figure 12 47 and click Change Figure 12 47 Select a data store multipath setting The storage disk is available and ready to use for your V...

Page 564: ...tion Guide 12 4 2 VMware ESX iSCSI attachment To perform a VMware ESX iSCSI attachment complete the following steps 1 Right click your VMware ESX Fibre iSCSI host in the All Hosts view Figure 12 48 an...

Page 565: ...2 In the Host Details window you see that there is one volume connected to the ESX iSCSI host using SCSI ID 0 The UID of the volume is also displayed Connect to your VMware ESX Server using the vSphe...

Page 566: ...P address is the IP address of a node in the I O group from which you are mapping the iSCSI volume Leave the IP port number at the default value of 3260 and click OK The connection between the initiat...

Page 567: ...I Initiator properties by clicking Close Figure 12 51 on page 546 You are prompted to rescan for new storage devices Confirm the scan by clicking Yes Figure 12 53 Figure 12 53 Confirm the rescan 6 Go...

Page 568: ...uction and Implementation Guide 7 The Add Storage wizard opens Figure 12 55 Select Disk LUN and click Next Figure 12 55 Select Disk LUN menu 8 The new iSCSI LUN displays Highlight it and click Next Fi...

Page 569: ...Host configuration 549 9 Review the disk layout and click Next Figure 12 57 Figure 12 57 Current Disk Layout 10 Enter a name for the data store and click Next Figure 12 58 Figure 12 58 Enter a data s...

Page 570: ...torage Node Introduction and Implementation Guide 11 Select the maximum file size and click Next Figure 12 59 Figure 12 59 Maximum file size 12 Review your selections and click Finish Figure 12 60 Fig...

Page 571: ...task can take a few minutes After the tasks complete the new data store appears in the storage view Figure 12 61 Figure 12 61 New data store available 13 Highlight the new data store and click Propert...

Page 572: ...t supported on AIX and we were unable to test iSCSI connectivity This section detail FC connectivity only which is probably the most popular method of external block level storage connectivity for AIX...

Page 573: ...tings are as they should be Example 12 7 Check HBA settings lsattr El fscsi0 attach switch How this adapter is CONNECTED False dyntrk yes Dynamic Tracking of FC Devices True fc_err_recov fast_fail FC...

Page 574: ...up we check on IBM Flex System V7000 Storage Node For each step we use the WWN of each HBA as a reference Example 12 8 shows how to display the WWN or Network Address of each adapter Example 12 8 Disp...

Page 575: ...orage Node Figure 12 66 shows that both HBAs WWNs are correctly displayed in a defined host Figure 12 66 WWNs displayed of defined Host 12 5 4 Installing the 2145 host attachment support package To co...

Page 576: ...s configures and makes available all storage device paths SDDPCM then manages these paths to provide High availability and load balancing of storage I O Automatic path failover protection Concurrent d...

Page 577: ...b objrepos devices sddpcm 71 rte 2 6 3 2 COMMITTED IBM SDD PCM for AIX V71 Path etc objrepos devices sddpcm 71 rte 2 6 3 2 COMMITTED IBM SDD PCM for AIX V71 The machine will require a reboot following...

Page 578: ...jfs2 a logname INLINE A y p rw d test2lv u v7kgrp m test2 File system created successfully 6527596 kilobytes total disk space New File System size is 13107200 mount t v7kgrp The lspv command output n...

Page 579: ...th State and Mode for each of the defined hdisks The status that both adapters are showing as optimal with State OPEN and Mode NORMAL Additionally an asterisk displayed next to paths indicates inactiv...

Page 580: ...for POWER with current maintenance updates available from Novell to enable all planned functionality Red Hat Enterprise Linux 5 7 for POWER or later Red Hat Enterprise Linux 6 2 for POWER or later For...

Page 581: ...2 16 cat proc scsi scsi command example root diomede sdd cat proc scsi scsi Attached devices Host scsi4 Channel 00 Id 00 Lun 00 Vendor IBM Model 2145 Rev 0000 Type Unknown ANSI SCSI revision 04 Host s...

Page 582: ...IBM pid 2145 serial 60050768018201bee000000000000035 lun_id 60050768018201bee000000000000035 ctlr_flag 1 ctlr_nbr 0 df_ctlr 0 RTPG command a3 0a 00 00 00 00 00 00 08 20 00 00 total datalen 52 datalen_...

Page 583: ...LICY Optimized Sequential SERIAL 60050768018201bee000000000000035 Path Adapter Hard Disk State Mode Select Errors 0 Host0Channel0 sda CLOSE NORMAL 1 0 1 Host0Channel0 sdb CLOSE NORMAL 0 0 2 Host1Chann...

Page 584: ...ount 2 state inactive WWPN 21000024FF2F2548 node_logged_in_count 2 state active IBM_4939 FlexSystem V7000 superuser lshostvdiskmap x240_slot11 id name SCSI_id vdisk_id vdisk_name vdisk_UID IO_group_id...

Page 585: ...1 1011 default 1011 Using default value 1011 Command m for help w The partition table has been altered Calling ioctl to re read partition table Syncing disks root Palau 2 Create a file system on the...

Page 586: ...VICE NAME vpatha TYPE 2145 POLICY Optimized Sequential SERIAL 60050768018201bee000000000000035 Path Adapter Hard Disk State Mode Select Errors 0 Host0Channel0 sda OPEN NORMAL 1 0 1 Host0Channel0 sdb O...

Page 587: ...dm multipath modprobe dm round robin service multipathd start chkconfig multipathd on Example 12 28 shows the commands that are issued on a Red Hat Enterprise Linux 5 1 operating system Example 12 28...

Page 588: ...ytes Device Boot Start End Blocks Id System dev hda1 1 13 104391 83 Linux dev hda2 14 9730 78051802 8e Linux LVM Disk dev sda 4244 MB 4244635648 bytes 131 heads 62 sectors track 1020 cylinders Units c...

Page 589: ...Disk dev dm 2 4244 MB 4244635648 bytes 255 heads 63 sectors track 516 cylinders Units cylinders of 16065 512 8225280 bytes Disk dev dm 2 doesn t contain a valid partition table Disk dev dm 3 4244 MB...

Page 590: ...em blocks 1061158912 32 block groups 32768 blocks per group 32768 fragments per group 16192 inodes per group Superblock backups stored on blocks 32768 98304 163840 229376 294912 819200 884736 Writing...

Page 591: ...nd restore process Software upgrade procedure Drive firmware upgrade procedure Troubleshooting Audit log navigation Support data collection Using event notification Configuring Call Home Event log nav...

Page 592: ...dition or deletion of the IBM Flex System V7000 nodes canister in a cluster Software recovery through a service panel push button Automatic software version correction when replacing a node Detailed s...

Page 593: ...an IBM Flex System V7000 Storage Node enclosure The LEDs provide a general idea of the state of the system You can obtain more detail from the IBM Flex System V7000 Storage Node management GUI and the...

Page 594: ...arious LED states that you might observe Figure 13 2 Node canister LED layout State Description Off The LED is continuously not lit or off Blinking slowly The LED turns on and off at a frequency of 1...

Page 595: ...ss of communicating with the CMM during initial insertion of the canister If the canister remains in this state for more than 10 minutes Try re seating the canister If the state persists follow the ha...

Page 596: ...ister has failed and should be replaced If it is available use the IBM Flex System V7000 Storage Node management GUI event log Fix Procedures to guide the replacement Internal fault Amber There is a f...

Page 597: ...Indicates the battery is charging but cannot yet support a save of cache and system state data The node canister cannot operate as an active member of the system in this state No action is required w...

Page 598: ...connects to are both powered on The cable might need replacing ON SOLID There is at least 1 active SAS link in the wide port established and there is no external port activity It is a normal state and...

Page 599: ...cedure shows the status for the expansion canisters It does not show the status for the drives IBM Flex System V7000 expansion enclosure includes two expansion canisters that are located in the rear o...

Page 600: ...ansion enclosure identify Blue Expansion enclosure fault Amber 1 3 2 4 5 6 7 8 9 State Description Off LED is continuously not lit or off Blinking slowly LED turns on and off at a frequency of 1 Hz It...

Page 601: ...management controller is in the process of communicating with the Chassis Management Module CMM during initial insertion of the expansion canister If the expansion canister remains in this state for m...

Page 602: ...ng correctly Expansion enclosure fault Amber This LED is not used and it is always off SAS port status Amber OFF There have been no faults or conditions detected by the expansion canister on the SAS p...

Page 603: ...The green power on LEDs are on solid when the enclosure is powered on successfully 3 Verify that no faults are detected on the control enclosure after powering on Use Table 13 2 on page 575 to verify...

Page 604: ...tasks and task categories such as Resource Explorer Availability Inventory System status and Health Summary Figure 13 4 shows you how to open the navigation area Click the tab with the arrow icon on t...

Page 605: ...s of systems that have been discovered You can display this information using one of the System Status and Health tasks by navigating to a specific resource in IBM Flex System Manager or by using the...

Page 606: ...mmary task displays information about the resources that IBM Flex System Manager has discovered You can customize the content that is displayed on the System Status and Health pages with information t...

Page 607: ...m searching for them all in the Monitors task Problems The Problems task lists only those elevated status set entries that are problem status set entries that have been reported to IBM Flex System Man...

Page 608: ...complete the Setup wizard the management software discovers IBM Flex System V7000 Storage Node and collects inventory automatically After IBM Flex System V7000 Storage Node is managed by the manageme...

Page 609: ...hassis that contains IBM Flex System V7000 Storage Node Figure 13 8 shows that a chassis is selected and a Map view opens Figure 13 8 Select a managed chassis from the Chassis Map view 3 From the Chas...

Page 610: ...igure 13 10 Accessing IBM Flex System V7000 Storage Node management options from the FSM Chassis Manager view Tip You can launch the IBM Flex System V7000 Storage Node management GUI from the Chassis...

Page 611: ...gement The following links are available under Storage Tasks Discover Storage Use discover storage to discover and automatically collect inventory for manageable storage resources including Fibre Chan...

Page 612: ...s shown in Figure 13 12 Figure 13 12 Using CMM to launch IBM Flex System V7000 Storage Node You can also click the Chassis Management tab and view properties and settings of the following components i...

Page 613: ...e configuration settings for each system are stored on each node If power fails on a system or if a node in a system is replaced the system configuration settings are automatically restored when the r...

Page 614: ...ng software on nodes However you can also upgrade each node manually Upgrading the IBM Flex System V7000 Storage Node using FSM Use the IBM FSM to update the firmware and software for IBM Flex System...

Page 615: ...cedure the majority of configuration commands are not available This automatic procedure provides a unified mechanism to upgrade the entire system in a coordinated process with no user intervention Ex...

Page 616: ...node and return the node to the system You repeat this process for the remaining nodes until the last node is removed from the system Every node must be upgraded to the same code level You cannot inte...

Page 617: ...e test utility This utility checks for known issues which might prevent the upgrade from completing successfully Click Download link to download the test utility directly from the web server If your w...

Page 618: ...ility to the system as shown in Figure 13 18 Figure 13 18 Step 1 of 5 Upload upgrade test utility to the system 6 Enter the correct version info that you want to upgrade to In this example our upgrade...

Page 619: ...did not match with the upgrade version that you want to upgrade to you will receive a message similar to the following message This version of svcupgradetest cannot test the currently running version...

Page 620: ...orming you that the first node then the other node has been upgraded When both nodes have been rebooted you have completed your IBM Flex System V7000 Storage Node software upgrade as shown in Figure 1...

Page 621: ...following website http www 01 ibm com support docview wss uid ssg1S1004250 13 7 1 Multi drive upgrade utility An enhanced Multi Drive Upgrade Utility is available with the release of IBM Storwize Fami...

Page 622: ...ndicates if any of your drives are not running at the latest level of firmware 3 Using scp or pscp copy the firmware upgrade file and the Software Upgrade Test Utility package to the home admin upgrad...

Page 623: ...iew shows compute nodes and storage control enclosures including canisters and drives The rear view shows power modules cooling devices fan mux I O module and Management module The front view of the C...

Page 624: ...on and Implementation Guide 3 Hover the mouse over a component to view information about it as shown in Figure 13 26 and Figure 13 27 Figure 13 26 CMM Chassis Map node canister status information Figu...

Page 625: ...g 605 4 Click a component to view detailed information which appears in tabs in the Details panel as shown in Figure 13 28 and Figure 13 29 Figure 13 28 CMM Chassis Map node canister detail tabs Figur...

Page 626: ...00 Storage Node as the starting point to diagnose and resolve a problem Use the management GUI to run Fix Procedures This topic explains how to run the recommended action for any unfixed alert and wha...

Page 627: ...e You might be directed to the IBM Flex System V7000 Service Assistant GUI or to Flex System chassis management devices It is highly recommended that you set up CLI access to the control enclosure aft...

Page 628: ...ement Protocol SNMP an SNMP trap is sent to an SNMP manager that is configured by the customer If enabled log messages can be forwarded from a sender to a receiver on an IP network by using the Syslog...

Page 629: ...ed size After it is full newer entries replace entries that are no longer required To avoid having a repeated event that fills the event log some records in the event log refer to multiple occurrences...

Page 630: ...he CD or IBM Flex System Information Center website for lists of the error event IDs and corresponding error codes Node error codes describe failure that relate to a specific node canister Because nod...

Page 631: ...aphs Figure 13 34 Recommended actions The Recommended Actions tab shows the highest priority maintenance procedure that needs to be run Use the troubleshooting wizard to allow the IBM Flex System V700...

Page 632: ...to review the physical FC cabling to determine the issue and then click Run Fix Procedure to fix the error Figure 13 36 A best practice is to review the event logs and recommended actions periodicall...

Page 633: ...so far on the IBM Flex System V7000 system information and service commands are not included Figure 13 37 Audit Log from Access window 13 10 Support data collection The following data collection step...

Page 634: ...ion and Implementation Guide Figure 13 38 Collect support package Figure 13 39 shows all log files Search for the file you want Select it right click it and select Download to download it to your work...

Page 635: ...ect System Management Server service data using the CMM To collect System Management Server service data using the CMM use the following steps 1 Open the CMM In the menu bar at the top click Service a...

Page 636: ...llowing view displays all support files on this system that have been collected by Service and Support Manager Support files can contain detailed system information used to help diagnose a serviceable...

Page 637: ...IBM Flex System V7000 to send each type of notification to specific recipients Correct any issues reported by your system as soon as possible To avoid monitoring for new events by constantly monitorin...

Page 638: ...ance contract with IBM or if IBM Flex System V7000 Storage Node is within the warranty period To send email you must configure at least one SMTP server You can specify as many as five additional SMTP...

Page 639: ...g a test email to check that all connections and infrastructure are set up correctly no further Call Home configuration is required Figure 13 43 Configure email event notification during initial setup...

Page 640: ...sks on IBM Flex System V7000 Storage Node FSM performs Call Home for problems that IBM Flex System V7000 Storage Node and CMM identify They appear in Service and Support Manager See Setting up IBM Ele...

Page 641: ...configure Call Home if the FSM is included 1 You can set up and enable the Electronic Service Agent from the Additional Setup tab of the IBM Flex System Manager home page Getting started with the Ele...

Page 642: ...Provide default information about the physical locations of your systems Information can be overridden for specific systems by clicking Resource Explorer selecting a system and clicking Location unde...

Page 643: ...k OK An Internet connection is required to use this function Specify how the Internet should be accessed Click Test Internet Connection to verify that your Internet access is working properly as shown...

Page 644: ...V7000 Storage Node Introduction and Implementation Guide 5 Click Next Provide an IBM ID to be associated with information sent by the Electronic Service Agent as shown in Figure 13 50 Figure 13 50 Pr...

Page 645: ...age Node management GUI When shutting down only one node canister all the activities remain active on the partner node however when you shut down the entire cluster your access to volumes and data wil...

Page 646: ...es power on to the Flex System V7000 Storage Node 2 If the Flex System V7000 Storage Node does not automatically power on use the CMM as shown in Figure 13 52 or FSM GUI CLI to request power on for th...

Page 647: ...V7000 Storage Node management GUI to shut down the system as shown in Figure 13 54 Figure 13 54 Shutting down the system from IBM Flex System V7000 GUI Figure 13 55 Confirmation to start the shutdown...

Page 648: ...er using the IBM Flex System V7000 Storage Node service GUI 1 Open the CMM and navigate to the Chassis Map The Chassis Map displayed by the CMM provides a graphical view of the IBM Flex System Enterpr...

Page 649: ...n This appendix describes the setup of the command line interface CLI and shows how to set up the host for a Secure Shell connection To demonstrate this capability we used PuTTY a freeware tool that i...

Page 650: ...e To access the CLI either a private and a public key pair or a user name and password are necessary The following steps are required to enable CLI access with an SSH session which would likely be use...

Page 651: ...s http www chiark greenend org uk Download the following tools PuTTY SSH client putty exe PuTTY key generator puttygen exe Generating a public and private key pair To generate a public and private key...

Page 652: ...s are generated save them for later use Click Save public key Figure A 3 Figure A 3 Save public key To generate keys The blank area indicated by the message is the large blank rectangle on the GUI ins...

Page 653: ...ed with a warning message Figure A 5 Click Yes to save the private key without a passphrase Figure A 5 Confirm the security warning 7 When prompted enter a name for example icat select a secure place...

Page 654: ...After you have created your SSH key pair you need to upload your SSH public key onto IBM Flex System V7000 Storage Node Complete the following steps 1 Open the user section Figure A 6 Figure A 6 Open...

Page 655: ...635 3 To upload the public key click Browse select your public key and click OK Figure A 8 Figure A 8 Select public key 4 Click OK and the key is uploaded Figure A 9 Figure A 9 Public key upload compl...

Page 656: ...To accomplish this task complete the following steps 1 Start PuTTY Figure A 10 Figure A 10 PuTTY Configuration In the right pane under the Specify the destination you want to connect to section selec...

Page 657: ...side of the PuTTY Configuration window click Connection SSH Auth As shown in Figure A 12 in the right pane in the Private key file for authentication field under the Authentication Parameters section...

Page 658: ...ne enter the host name or system IP address of the IBM Flex System V7000 Storage Node cluster in the Host Name field and enter a session name in the Saved Sessions field Figure A 13 Figure A 13 Enter...

Page 659: ...em V7000 Storage Node If you have scripts from those platforms that use these prefixes they run without problems If you type svcinfo or svctask and press the Tab key twice all the available subcommand...

Page 660: ...nter lsvdiskhostmap Example A 5 Example A 5 List all hosts mapped to a volume IBM_4939 FlexSystem_V7000 admin lshostvdiskmap ESXi 1 id name SCSI_id vdisk_id vdisk_name vdisk_UID 4 ESXi 1 0 2 ESXi Redb...

Page 661: ...d 2 vdisk_UID 600507680185853FF000000000000011 throttling 0 preferred_node_id 2 IBM_4939 FlexSystem_V7000 admin FlashCopy example commands IBM Flex System V7000 Storage Node offers powerful FlashCopy...

Page 662: ...startfcmap prep FCMAP_1 IBM_4939 FlexSystem_V7000 admin lsfcmap id name source_vdisk_id source_vdisk_name target_vdisk_id target_vdisk_name group_id group_name status progress copy_rate clean_progres...

Page 663: ...eric_hdd tier_capacity 5 45TB tier_free_capacity 2 10TB has_nas_key no layer storage rc_buffer_size 48 IBM_4939 FlexSystem_V7000 admin lssystem id 0000020061614FFC name FlexSystem_V7000 gm_inter_clust...

Page 664: ...644 IBM Flex System V7000 Storage Node Introduction and Implementation Guide...

Page 665: ...nts and other Redbooks publications Redpaper publications Web Docs draft and additional materials at the following website ibm com redbooks Other publications These publications are also relevant as f...

Page 666: ...oulder ibm com infocenter flexsys information topic com ibm acc 4939 learning site_learning_tutorials html IBM Flex System V7000 Storage Node Support home http www ibm com support entry portal overvie...

Page 667: ...orage Node Introduction and Implementation IBM Flex System V7000 Storage Node Introduction and Implementation Guide IBM Flex System V7000 Storage Node Introduction and Implementation Guide IBM Flex Sy...

Page 668: ...IBM Flex System V7000 Storage Node Introduction and Implementation Guide IBM Flex System V7000 Storage Node Introduction and Implementation Guide...

Page 669: ......

Page 670: ...M Flex Systems product family and is a modular storage system designed to fit into the IBM Flex System Enterprise chassis When purchased in the IBM PureFlex configurations IBM Flex System V7000 Storag...

Reviews:

No comments