Hitachi NVM Navigator v03-07 User Manual Download Page 294 | Manualshive

Page: 294 / 316

background image

6-28

Troubleshooting

HVM Navigator User's Guide - Migration

Migration has failed with "LPAR Migration was canceled. FC Login

timeout (Migration WWN)."

Event

LPAR migration has failed with “LPAR Migration was canceled. FC Login timeout

(Migration WWN).”

This error shows that a timeout has occurred while a command has been

logging into a port on a storage device.

Figure 6-29 Migration Progress window

Solution

Take the measure according to the following procedure.
1.

Set the timeout period to log into storage device which can be set in HVM

Navigator to the following value.
The timeout period to log into storage device
=(the time period set in “Login Delay Time” textbox in “Storage Login

Setting” in “Option” window) * 2 + 5 seconds
For changing the timeout period to log into storage device, see

Hitachi

Compute Blade HVM Navigator User's Guide - Getting Started.

2.

Execute the LPAR migration again.

«
...
292
293
294
295
296
...
»

Summary of Contents for NVM Navigator v03-07

Page 1: ...MK 99COM024 12 Hitachi Compute Blade HVM Navigator User s Guide Migration Document organization Product version Getting help Contents FA STFIN D LIN KS ...

Page 2: ...s and services can be ordered only under the terms and conditions of the applicable Hitachi Data Systems Corporation agreements The use of Hitachi Ltd products is governed by the terms of your agreements with Hitachi Data Systems Corporation Hitachi is a registered trademark of Hitachi Ltd in the United States and other countries Hitachi Data Systems is a registered trademark and service mark of H...

Page 3: ...1 Environment 2 2 Environment for Shutdown mode 2 2 Environment for Concurrent Maintenance mode 2 6 Time period for migration 2 10 Time period for Shutdown mode 2 10 Time period for Concurrent Maintenance mode 2 10 Application requirements 2 12 Application requirement for Shutdown mode 2 16 Application requirement for Concurrent Maintenance mode 2 42 Migration procedures 3 1 Shutdown mode 3 2 Proc...

Page 4: ... Giving migration WWN a nickname 3 100 Exporting CSV file 3 104 Adding storage device 3 106 Commands for Storage Navigator Modular 2 CLI executed from HVM Navigator3 113 Other functions and settings 4 1 Functions and settings for Shutdown and Concurrent Maintenance modes 4 2 Confirming applicable conditions before migration execution 4 2 Specifying LAN port IP address for management server 4 4 Tra...

Page 5: ...e 5 12 Migration of USB assignment state 5 13 Scheduling mode of assigned USB is dedicated 5 13 Scheduling mode of assigned USB is exclusive shared 5 14 Certain FC HBA port is changed from shared to dedicated 5 16 Duplication of displayed or retrieved WWN 5 16 Failure on remote shutdown of Windows 5 16 Shutting down or rebooting guest OS on the LPAR to migrate 5 17 Migrating LPAR where EFI Shell i...

Page 6: ... 18 Cannot access guest OS 6 19 Migration has failed with Transferring device data cannot be completed within an LPAR suspension period message 6 20 Migration has failed with Connection refused HVM HVM message 6 21 Migration has failed with Wait until another operation is completed and retry message 6 22 Migration has failed with HVM access error message 6 23 Migration has failed with LPAR Migrati...

Page 7: ...3 Detailed error information detected by Migration Management Server thread module 6 39 Detailed error information detected by LPAR Mover module 6 40 Detailed error information detected by Recovery module 6 41 Detailed error information detected by the source HVM 6 42 Collecting failure information 6 43 List of icons 7 1 ...

Page 8: ...viii Contents HVM Navigator User s Guide Migration ...

Page 9: ...for reference This preface includes the following information Intended audience Product version Release notes Document organization Referenced documents Document conventions Convention for storage capacity values Getting help Comments Notice The use of Compute Blade and all other Hitachi Data Systems products is governed by the terms of your agreement s with Hitachi Data Systems ...

Page 10: ...tware Firmware Driver for Hardware Operating System and Applications The reader is familiar with the location where the Compute Blade will be installed including knowledge of physical characteristics power systems and specifications and environmental specifications Product version This document revision applies to HVM Navigator V03 07 Release notes Read the release notes before installing and usin...

Page 11: ...cedures Describes the migration procedures Chapter 4 Other functions and settings Describes some of the other functions and settings of migration Chapter 5 Usage notes Describes the usage notes Chapter 6 Troubleshooting Describes the troubleshooting Chapter 7 List of icons Describes the icons which are displayed in the windows Referenced documents Compute Blade documents Hitachi Compute Blade 2000...

Page 12: ...ariable square bracket Optional values braces Required or expected value vertical bar Choice between two or more options or arguments _ underline Default value for example a b This document uses the following icons to draw attention to information Icon Meaning Description WARNING This indicates the presence of a potential risk that might cause death or severe injury CAUTION This indicates the pres...

Page 13: ... bytes 1 gigabyte GB 1 000 MB or 1 0003 bytes 1 terabyte TB 1 000 GB or 1 0004 bytes 1 petabyte PB 1 000 TB or 1 0005 bytes 1 exabyte EB 1 000 PB or 1 0006 bytes Logical storage capacity values for example logical device capacity are calculated based on the following values Logical capacity unit Value 1 block 512 bytes 1 KB 1 024 210 bytes 1 MB 1 024 KB or 1 0242 bytes 1 GB 1 024 MB or 1 0243 byte...

Page 14: ...log on to the Hitachi Data Systems Portal for contact information https portal hds com Comments Please send us your comments on this document doc comments hds com Include the document title and number including the revision level for example 07 and refer to specific sections and paragraphs whenever possible All comments become the property of Hitachi Data Systems Corporation Thank you ...

Page 15: ...1 Overview 1 1 HVM Navigator User s Guide Migration Overview This chapter describes the overview of LPAR migration ...

Page 16: ...an have an LPAR migrate with the guest OS on the LPAR down in Shutdown mode Therefore you must shut down the guest OS in advance or set the shutdown option for the guest OS so that the guest OS is shut down automatically prior to migration when you execute the LPAR migration When you execute LPAR migration in Concurrent Maintenance mode you can exchange or update the following kinds of hardware an...

Page 17: ...ation requirements and the kinds of configuration data to migrate are different between Shutdown mode and Concurrent Maintenance mode Application requirements for Shutdown mode are much less than those for Concurrent Maintenance mode When you have an LPAR migrate on a server blade on a system environment for the first time we recommend that you have the LPAR migrate in Shutdown mode Figure 1 2 Dif...

Page 18: ...ned Logical CPUs 1 Yes Shared mode No of assigned processors Yes Yes Dedicated mode No of assigned processors Yes Yes Processor number Yes No Whereas when the guest NUMA function is enabled Yes Service ratio Yes Yes Allocated memory size Yes Yes Idle Detection Yes Yes Auto Activation Order Yes Yes Auto Clear Yes Yes Processor Capping Yes Yes Virtual Console Enabled Y Disabled N Yes Yes Virtual Con...

Page 19: ...o No Yes Shared NIC Logical MAC address MAC seed Yes Exchanged between the source and destination Logical MAC address set manually 3 Virtual NIC assignment Yes Yes VLAN mode Tag Untag or Undef Yes Yes VLANID Yes Yes MMIO Register No No Yes VNIC Device Type Yes Yes VF NIC Logical MAC address MAC seed No Exchanged between the source and destination Logical MAC address set manually 3 Virtual NIC assi...

Page 20: ...e Zone Yes No Import Config Yes No TimeSync Yes No Local timer device HPET No No Yes PIT No No Yes PM Timer No No Yes The other local devices USB Assigned state A A or Yes Yes Used state R or R No No VGA Assigned state A or Yes Yes VRAM data No No Yes VGA Register No No Yes Super I O No No Yes Serial IO Port Register No No Output data of Serial Terminal No No Firmware SMBIOS No No UUID No Yes Logi...

Page 21: ...8 4x or higher 17 61 or earlier any HVM firmware version for CB500 or any HVM firmware versions for CB2500 2 The value of the item isn t transferred to the destination server blade and is set to A on destination server blade when the Guest NUMA feature is disabled whereas when the Guest NUMA feature is enabled it is transferred to the destination server blade 3 You can change the setting However d...

Page 22: ...1 8 Overview HVM Navigator User s Guide Migration ...

Page 23: ...2 Usage environments 2 1 HVM Navigator User s Guide Migration Usage environments This chapter describes the usage environments Environment Time period for migration Application requirements ...

Page 24: ...of HVM Navigator are not available For the available combinations of HVM and HVM Navigator see Table 2 2 and 2 3 12 Table 2 1 Supported server blade model Platform Supported model BS1000 CB2000 Standard Server Blade A1 A2 R3 S3 R4 S4 High Performance Server Blade A1 A2 CB320 P4 P5 CB500 CB520H Server Blade A1 B1 A2 B2 B3 CB520A Server Blade A1 CB540A Server Blade A1 B1 CB520X Server Blade B1 B2 CB...

Page 25: ...gher V03 02 or higher 59 6x or higher V03 04 or higher 59 71 or higher V03 04 A or higher 59 73 or higher V03 07 or higher High Performance Server Blade 78 70 or earlier V02 01 or earlier 78 73 or higher V02 02 or higher 79 0x or higher V03 00 A or higher 79 1x to 79 20 V03 00 E or higher 79 21 or higher V03 01 or higher 79 51 or higher V03 02 A or higher 79 6x or higher V03 04 or higher 79 71 or ...

Page 26: ...or higher 02 1x or higher V03 07 or higher CB520A Server Blade 01 1x V03 00 B or higher 01 2x or higher V03 00 D or higher 01 3x V03 00 E or higher 01 4x or higher V03 01or higher 01 6x or higher V03 02 or higher 01 7x or higher V03 03 or higher 01 8x or higher V03 04 or higher 02 00 or higher V03 05 or higher 02 05 or higher V03 06 or higher 02 1x or higher V03 07 or higher CB540A Server Blade 01...

Page 27: ...e B1 model 02 00 or higher V03 05 or higher 02 05 or higher V03 06 or higher 02 1x or higher V03 07 or higher CB520X Server Blade B2 model 02 1x or higher V03 07 or higher Be sure to confirm the application requirements described in Application requirements for Shutdown mode in Usage environments before executing LPAR migration in Shutdown mode You can execute LPAR migration in Shutdown mode throu...

Page 28: ...de on old versions of HVM For available combinations of HVM and HVM Navigator see Table 2 5 and 2 6 Table 2 4 Supported server blade model Platform Supported model BS1000 CB2000 Standard Server Blade A1 A2 R3 S3 R4 S4 High Performance Server Blade A1 A2 CB320 CB500 CB520H Server Blade A1 B1 A2 B2 B3 CB520A Server Blade A1 CB540A Server Blade A1 B1 CB520X Server Blade B1 B2 CB2500 CB520H Server Bla...

Page 29: ... or higher 59 2x or higher V03 01 or higher 59 51 or higher V03 02 or higher 59 6x or higher V03 04 or higher 59 71 or higher V03 04 A or higher 59 73 or higher V03 07 or higher High Performance Server Blade 79 1x or 79 20 V03 00 E or higher 79 2x or higher V03 01 or higher 79 51 or higher V03 02 or higher 79 52 or higher V03 03 or higher 79 6x or higher V03 04 or higher 79 71 or higher V03 04 A o...

Page 30: ...3 00 E or higher 01 4x or higher V03 01 or higher 01 6x or higher V03 02 or higher 01 7x or higher V03 03 or higher 01 8x or higher V03 04 or higher 01 9x or higher V03 04 A or higher 02 00 or higher V03 05 or higher 02 05 or higher V03 06 or higher 02 1x or higher V03 07 or higher CB540A Server Blade 01 3x or higher V03 00 E or higher 01 4x or higher V03 01 or higher 01 6x or higher V03 02 or hig...

Page 31: ...ent Maintenance mode in Usage environments before executing LPAR migration in Concurrent Maintenance mode You must fulfill the following requirements to execute LPAR migration in Concurrent Maintenance mode The HVM models both are Enterprise when you execute LPAR migration in Concurrent Maintenance mode between CB2000 s The HVM models both are Advanced or Enterprise when you execute LPAR migration...

Page 32: ...at type of guest OS running on the LPAR what application programs running on the guest OS and other factors Time period for Concurrent Maintenance mode The time required for an LPAR to migrate in Concurrent Maintenance mode the sum of the time periods for the following items Backing up HVM configurations and retrieving HVM dumps on the source and destination server HVMs Saving LPAR configurations ...

Page 33: ... rate is calculated as follows The following is an example when Memory transfer rate is 100 Mbps The time period varies depending on the memory size for the guest OS running on the source LPAR and load of a migration path between the source and destination LPARs The bigger the memory the longer the time For example when memory size is 4 GB 4096 MB the theoretical time period is 40 96 seconds There...

Page 34: ...Source Server Chassis CPU Memory NIC FC HBA Guest OS management server HVM Navigator Middleware Application Software storage device Data for LPAR Configuration Logical CPU Memory Logical NIC Logical FC HBA UEFI Item does not need to be confirmed Management module BMC HVM LPAR Server Blade EFI BIOS Destination Server Chassis CPU Memory NIC FC HBA Guest OS Middleware Application Software Logical CPU...

Page 35: ...ments of LPAR to migrate The requirements of the LPAR to migrate for both Shutdown and Concurrent Maintenance mode are shown in the following table When the following requirements are fulfilled you don t need to confirm Application requirements of the LPAR to migrate for Shutdown mode or Application requirements of the LPAR to migrate for Concurrent Maintenance mode in Usage environments Table 2 8...

Page 36: ... on the source server blade is installed in the same slot on the destination server blade 2 FC HBA A FC HBA card installed in a slot on the source server blade is installed in the same slot on the destination server blade 2 BMC No requirement EFI BIOS The major version and the setting of EFI BIOS are matched between the source and destination server blades 3 The version and the setting of EFI BIOS...

Page 37: ...er blade model in Application requirements for Concurrent Maintenance mode in Usage environments 2 The actual requirement is more relaxed than the above content For the detail see Requirement of configuration of NIC and FC HBA in Application requirements for Shutdown mode in Usage environments or Application requirements for Concurrent Maintenance mode in Usage environments 3 When the major versio...

Page 38: ...grates the processors with the same physical numbers may not be assigned to the LPAR on the destination server blade However the same number of processors assigned to the LPAR on the source server blade is assigned on destination server blade 2 The LPAR to migrate might not be activated on the destination server blade because of memory fragmentation In this case the message The MMS could not activ...

Page 39: ...t of source and destination server blades Management module version You can execute LPAR migration in Shutdown mode regardless of management module version Dictionary version You can execute LPAR migration in Shutdown mode regardless of Dictionary version Server blade Location You cannot execute LPAR migration in Shutdown mode from an LPAR number to another within a server blade For relation of th...

Page 40: ... the same between the source and destination server blades CB2000 standard server blade Table 2 13 Combination of server blades CB2000 standard server blade Destination Source A1 A2 R3 S3 R4 S4 A1 Yes No No No No No A2 No Yes No No No No R3 No No Yes No No No S3 No No No Yes No No R4 No No No No Yes No S4 No No No No No Yes See the following 1 CB2000 high performance server blade Table 2 14 Combin...

Page 41: ...rver blades CB500 CB520H server blade Destination Source A1 B1 A2 B2 B3 A1 Yes No No No No B1 No Yes No No No A2 No No Yes No No B2 No No No Yes No B3 No No No No Yes CB500 CB520A server blade Table 2 17 Combination of server blades CB500 CB520A server blade Destination Source A1 A1 Yes CB500 CB540A server blade Table 2 18 Combination of Server Blades CB500 CB540A server blade Destination Source A...

Page 42: ...r blade Table 2 20 Combination of server blades CB2500 CB520H server blade Destination Source B3 B3 Yes CB2500 CB520X server blade Table 2 21 Combination of server blades CB2500 CB520X server blade Destination Source B1 B2 B1 Yes No B2 No Yes When you execute LPAR migration in Shutdown mode between CB520X server blades see 4 When you execute LPAR migration in Shutdown mode between CB520X server bl...

Page 43: ...2 22 Combination of processor type of CB2000 standard server blade A2 model Destination Source E5503 Xeon the 5600s number E5503 Yes No Xeon the 5600s number No When you execute LPAR migration between CB2000 standard server blades where Intel Westmere EP processor Xeon the 5600s number s are installed The HVM firmware versions on the source and destination server blades are compatible as shown in ...

Page 44: ...h Intel Westmere EP Processors Destination Source 17 61 or earlier 17 70 or higher 17 62 or earlier Yes 17 70 or higher Yes When you execute LPAR migration on HVM firmware version 17 70 or higher you must set Main Advanced Processor Options AES Support to Disabled in the BIOS For the setting see Hitachi Compute Blade 320 USER S GUIDE 3 SMP configurations of CB2000 high performance server blade You...

Page 45: ...wn in the following table Table 2 27 Combination of configurations on source and destination server blades Destination Source 1 blade server partition 2 blade SMP server partition 4 blade SMP server partition 1 blade server partition Yes No No 2 blade SMP server partition No Yes No 4 blade SMP server partition No No Yes Number of physical processors The number of physical processors must be matche...

Page 46: ...Viewer window The kind of each FC HBA card on the source and destination server blades such as Mezzanine card or expansion card is matched The number of the ports with which each FC HBA card on the source and destination server blades is accompanied is matched The relative slot number where each FC HBA card on the source and destination server blades is located is matched Application software for ...

Page 47: ...tion Source 01 xx 02 xx 03 xx 04 xx 09 xx 10 xx 11 xx 12 xx 01 xx Yes Yes No No No No No No 02 xx Yes Yes No No No No No No 03 xx No No Yes Yes No No No No 04 xx No No Yes Yes No No No No 09 xx No No No No Yes Yes No No 10 xx No No No No Yes Yes No No 11 xx No No No No No No Yes Yes 12 xx No No No No No No Yes Yes Table 2 30 Combination of compatible EFI versions CB2000 high performance server bla...

Page 48: ...s Yes Yes Yes 05 xx Yes Yes Yes Yes Yes Yes Yes 06 xx Yes Yes Yes Yes Yes Yes Yes 07 xx Yes Yes Yes Yes Yes Yes Yes Table 2 32 Combination of compatible EFI versions CB500 CB520H server blade A2 B2 Destination Source 10 xx 10 xx Yes Table 2 33 Combination of compatible EFI versions CB500 CB520H server blade B3 Destination Source 08 xx 08 xx Yes Table 2 34 Combination of compatible EFI versions CB5...

Page 49: ... CB500 CB520X server blade B1 Destination Source 06 xx 07 xx 06 xx Yes Yes 07 xx Yes Yes Table 2 37 Combination of compatible EFI versions CB500 CB520X server blade B2 Destination Source 09 xx 09 xx Yes Table 2 38 Combination of compatible EFI versions CB2500 CB520H server blade B3 Destination Source 08 xx 08 xx Yes Table 2 39 Combination of compatible EFI versions CB2500 CB520X server blade B1 De...

Page 50: ...e Migration Table 2 40 Combination of compatible EFI versions CB2500 CB520X server blade B2 Destination Source 09 xx 09 xx Yes Setting of EFI BIOS The settings of EFI and BIOS between the source and destination server blades must be the same Shutdown ...

Page 51: ...n the source and destination server blades in the following table Table 2 41 Reference for each model Platform Model Reference CB2000 Standard server blade A1 1 1 A2 1 2 R3 1 3 S3 1 4 R4 1 5 S4 1 6 High performance server blade A1 1 7 A2 1 8 CB320 PCI expansion server blade P4 2 1 P5 2 2 CB500 CB520H Server Blade A1 3 1 B1 3 2 A2 3 3 B2 3 4 B3 3 5 CB520A Server Blade A1 3 6 CB540A Server Blade A1 ...

Page 52: ...e Note on expansion of VNIC System No 1 2 CB2000 standard server blade A2 model Table 2 43 Combination of HVM on CB2000 standard server blade A2 model Destination Source 58 50 or earlier Between 58 60 and 58 70 Between 58 71 and 58 72 Between 58 80 and 59 11 59 21 or higher 58 50 or earlier Yes Yes1 Yes1 No No Between 58 60 and 58 70 Yes1 Yes Yes No No Between 58 71 and 58 72 Yes1 2 Yes2 Yes2 No N...

Page 53: ... Table 2 45 Combination of HVM on CB2000 standard server blade S3 model Destination Source Between 59 00 and 59 11 59 21 or higher Between 59 00 and 59 11 Yes Yes 59 21 or higher Yes Yes See Note on expansion of VNIC System No 1 5 CB2000 standard server blade R4 model Table 2 46 Combination of HVM on CB2000 standard server blade R4 model Destination Source 59 51 or higher 59 51 or higher Yes 1 6 C...

Page 54: ...000 high performance server blade A2 model Table 2 49 Combination of HVM on CB2000 high performance server blade A2 model Destination Source 78 70 or earlier Between 78 71 and 78 72 Between 78 80 and 79 11 79 21 or higher 78 70 or earlier Yes Yes No No Between 78 71 and 78 72 Yes1 Yes1 No No Between 78 80 and 79 11 No No Yes1 Yes2 79 21 or higher No No Yes2 Yes Notes 1 See Note on assigning NIC po...

Page 55: ...el Table 2 52 Combination of HVM on CB500 CB520H server blade A1 model Destination Source Between 01 01 and 01 30 Between 01 40 and 01 90 02 01 or higher Between 01 01 and 01 30 Yes Yes No Between 01 40 and 01 90 Yes Yes No 02 01 or higher No No Yes See Note on assigning NIC ports in Application requirements for Shutdown mode in Usage environments 3 2 CB500 CB520H server blade B1 model Table 2 53 ...

Page 56: ...90 02 01 or higher Between 01 60 and 01 90 Yes No 02 01 or higher No Yes 3 5 CB500 CB520H server blade B3 model Table 2 56 Combination of HVM on CB500 CB520H server blade B3 model Destination Source 02 05 or higher 02 05 or higher Yes 3 6 CB500 CB520A server blade A1 model Table 2 57 Combination of HVM on CB500 CB520A server blade A1 model Destination Source Between 01 01 and 01 30 Between 01 40 a...

Page 57: ... HVM on CB500 CB540A Server Blade B1 Model Destination Source Between 01 02 and 01 30 Between 01 40 and 01 90 02 01 or higher Between 01 02 and 01 30 Yes Yes No Between 01 40 and 01 90 Yes Yes No 02 01 or higher No No Yes See Note on assigning NIC ports in Application requirements for Shutdown mode in Usage environments 3 9 CB500 CB520X server blade B1 model Table 2 60 Combination of HVM on CB500 ...

Page 58: ...s HVM system time zone The time zones for the source and destination HVMs must be the same In Japan set the time zones to 9 00 HVM system time synchronization with NTP server You must set the same server as the NTP server to synchronize HVM system times for the source and destination HVMs For CB2000 CB500 or CB2500 We recommended that you set the management module as the NTP server Security streng...

Page 59: ...LPAR migration on HVMs on CB 2000 high performance server blades Destination Source HVM firmware version 79 11 or earlier 79 21 or higher HVM firmware version VNIC System No 128 or smaller 128 or smaller 129 or greater 79 11 or earlier 128 or smaller Yes Yes No 79 21 or higher 128 or smaller Yes Yes Yes 129 or greater No Yes Yes CB500 When you execute LPAR migration in Shutdown mode between HVM fi...

Page 60: ... cannot migrate regardless a value of Hvm Operating Mode Case1 When segments with the same shared NIC number are assigned to nonconsecutive Virtual NIC Numbers Ex 2a and 2b are assigned to Virtual NIC Number 2 and 4 Figure 2 2 Segments with the same shared NIC number are assigned to nonconsecutive Virtual NIC Numbers Case2 When segments are assigned to any of Virtual NIC Number 8 through 15 Ex 1a ...

Page 61: ...0 through 6 Figure 2 4 A segment is assigned two or more times to an LPAR Case4 When only some of the segments belonging to a controller with multiple ports is assigned to an LPAR Ex Only 1a belonging to a controller with two ports is assigned to Virtual NIC Number 0 Figure 2 5 Some of the segments belonging to a controller with multiple ports is assigned to an LPAR Shutdown ...

Page 62: ...HVM For the combinations of migratable environments between the source and destination HVMs see the following tables Table 2 68 Combination of migratable environments CB2000 standard server blade Destination Source HVM firmware version HVM firmware version VNIC Device Type 58 8x or earlier 59 0x or higher Standard Expansion Standard Expansion 58 8x or earlier Yes Yes Yes 59 0x or earlier NIC1 Yes ...

Page 63: ... the number of times shown in the following table you must reboot the server blade or execute Force Recovery according to Hitachi Compute Blade 500 Series Logical partitioning manager User s Guide Table 2 70 Migratable number of times to HVM firmware version 01 01 HVM firmware version Number of assigned shared NIC ports 2 4 01 01 200 400 Shutdown ...

Page 64: ... in shared mode are assigned Under 8 ports are assigned Notes 1 When an LPAR to which processors with a specified physical processor number in dedicated mode are assigned migrates the processors with the same physical numbers may not be assigned to the LPAR on the destination server blade However the same number of processors assigned to the LPAR on the source server blade is assigned on destinati...

Page 65: ...1 or higher 6 1 59 31or higher 59 31or higher 6 2 59 31or higher 59 31or higher 59 31or higher 6 4 59 41 or higher 59 41 or higher 59 41 or higher 59 51 or higher 6 5 59 70 or higher 59 70 or higher 59 70 or higher 59 70 or higher 6 6 59 72 or higher 59 72 or higher 59 72 or higher 59 72 or higher Windows Server 2003 R2 SP2 2008 59 10 or higher 2008 SP2 59 10 or higher 59 10 or higher 59 10 or hig...

Page 66: ... 31 or higher 6 2 79 31 or higher 79 31 or higher 6 4 79 41 or higher 79 41 or higher 6 5 79 70 or higher 79 70 or higher 6 6 79 72 or higher 79 72 or higher Windows Server 2003 R2 SP2 2008 2008 SP2 79 10 or higher 79 10 or higher 2008 R2 79 10 or higher 79 10 or higher 2008 R2 SP1 79 10 or higher 79 10 or higher 2012 79 21 or higher 79 21 or higher 2012 R2 79 52 or higher Unsupported Unsupported ...

Page 67: ...r 01 90 or higher 02 05 or higher 01 90 or higher 01 90 or higher 01 81 or higher 6 6 02 06 or higher 02 06 or higher 02 06 or higher 02 06 or higher 02 06 or higher 02 06 or higher Windows Server 2003 R2 SP2 2008 2008 SP2 01 30 or higher 01 30 or higher 01 30 or higher 2008 R2 01 30 or higher 01 30 or higher 01 30 or higher 2008 R2 SP1 01 30 or higher 01 60 or higher 01 30 or higher 01 30 or high...

Page 68: ...rise Linux 6 5 02 05 or higher 02 01 or higher 6 6 02 06 or higher 02 06 or higher 02 10 or higher Windows Server 2008 R2 SP1 02 01 or higher 2012 02 05 or higher 02 01 or higher 02 10 or higher 2012 R2 02 05 or higher 02 01 or higher 02 10 or higher Unsupported Unsupported implies that the OS is unavailable as a guest OS or you cannot have the OS migrate in Concurrent Maintenance mode Concurrent ...

Page 69: ...ver blade For relation of the locations of the source and destination server blades between which you can execute LPAR migration in Shutdown mode confirm the following table Table 2 77 Requirements of location between source and destination server blades Location Migratable Within a server blade No From a server blade to another within a server chassis Yes From a server blade in a chassis to anoth...

Page 70: ...dard server blade Table 2 78 Combination of server blades CB2000 standard server blade Destination Source A1 A2 R3 S3 R4 S4 A1 Yes No No No No No A2 No Yes No No No No R3 No No Yes No No No S3 No No No Yes No No R4 No No No No Yes No S4 No No No No No Yes See 1 CB2000 high performance server blade Table 2 79 Combination of server blades CB2000 high performance server blade Destination Source A1 A2...

Page 71: ...B3 No No No No Yes Table 2 81 Combination of server blades CB520A server blade Destination Source A1 A1 Yes Table 2 82 Combination of server blades CB540A server blade Destination Source A1 B1 A1 Yes No B1 No Yes Table 2 83 Combination of server blades CB520X server blade Destination Source B1 B2 B1 Yes No B2 No Yes When you execute LPAR migration in Concurrent Maintenance mode between CB520X serv...

Page 72: ... of server blades CB520H server blade Destination Source A1 A1 Yes Table 2 85 Combination of server blades CB520X server blade Destination Source B1 B2 B1 Yes No B2 No Yes When you execute LPAR migration in Concurrent Maintenance mode between CB520X server blades see 3 Concurrent Maintenance ...

Page 73: ... Yes No Xeon the 5600s number No Yes 2 SMP configurations of CB2000 high performance server blade You can execute LPAR migration in Concurrent Maintenance mode only when the server blade configurations between the source and destination server blades are the same Confirm the combinations of the configurations of the source and destination server blades between which you can execute LPAR migration ...

Page 74: ...in the following table Table 2 89 Combination of configurations on source and destination server blades Destination Source 1 blade server partition 2 blade SMP server partition 4 blade SMP server partition 1 blade server partition Yes No No 2 blade SMP server partition No Yes No 4 blade SMP server partition No No Yes Number of physical processors The number of physical processors must be matched b...

Page 75: ...ame of each FC HBA card on the source and destination server blades is matched Note that you can confirm device names in Migration Config Viewer window The kind of each FC HBA card on the source and destination server blades such as Mezzanine card or expansion card is matched The number of the ports with which each FC HBA card on the source and destination server blades is accompanied is matched T...

Page 76: ... 17 2096 or higher x64 4 6 17 2096 or higher 6 4 x86 1 6 17 2114 or higher x64 4 6 17 2114 or higher 6 5 x86 1 6 17 2114 or higher x64 4 6 17 2114 or higher 6 6 x86 1 6 17 2116 or higher x64 4 6 17 2116 or higher Windows Server 2003 R2 SP2 x86 x64 Server 2008 x86 1 1 6 880 or higher x64 4 1 6 880 or higher Server 2008 SP2 x86 1 1 6 880 or higher x64 4 1 6 880 or higher Server 2008 R2 4 2 6 880 or ...

Page 77: ...her ServerConductor Advanced Agent 09 50 or higher Be sure to see Migrating LPAR from the standby server blade after Cold standby switching in Usage Notes when you use ServerConductor Blade Server Manager 08 xx 09 53 or earlier BMC version You can execute LPAR migration in Concurrent Maintenance mode regardless of BMC version EFI BIOS EFI BIOS version The EFI version or the BIOS version must be ma...

Page 78: ...the source and destination server blades in the following table Table 2 93 Reference for each model Platform Model Reference CB2000 Standard server blade A1 1 1 A2 1 2 R3 1 3 S3 1 4 R4 1 5 S4 1 6 High performance server blade A1 1 7 A2 1 8 CB320 PCI expansion server blade P4 P5 CB500 CB520H server blade A1 3 1 B1 3 2 A2 3 3 B2 3 4 B3 3 5 CB520A server blade A1 3 6 CB540A server blade A1 3 7 B1 3 8...

Page 79: ...9 11 or earlier 59 21 or higher 59 11 or earlier Yes Yes 59 21 or higher Yes Yes See Note on expansion of VNIC System No 1 3 CB2000 standard server blade R3 model Table 2 96 Combination of HVM on CB2000 standard server blade R3 model Destination Source 59 11 or earlier 59 21 or higher 59 11 or earlier Yes Yes 59 21 or higher Yes Yes See Note on expansion of VNIC System No 1 4 CB2000 standard serve...

Page 80: ...51 or higher Yes 1 7 CB2000 high performance server blade A1 model Table 2 100 Combination of HVM on CB2000 high performance server blade A1 model Destination Source 79 11 or earlier 79 21 or higher 79 11 or earlier Yes Yes 79 21 or higher Yes Yes See Note on expansion of VNIC System No 1 8 CB2000 high performance server blade A2 model Table 2 101 Combination of HVM on CB2000 high performance serv...

Page 81: ...R on the destination server blade Otherwise you cannot have the LPAR migrate onto HVM firmware version 01 51 or earlier 3 2 CB500 CB520H server blade B1 model We don t support LPAR migrations to HVM firmware version 01 60 from another HVM firmware version Table 2 103 Combination of HVM on CB500 CB520H server blade B1 model Destination Source 01 30 01 40 to 01 51 01 60 Between 01 70 and 01 90 02 01...

Page 82: ...90 No Yes No 02 01 or higher No No Yes 3 4 CB500 CB520H server blade B2 model We don t support LPAR migrations to HVM firmware version 01 60 from another HVM firmware version Table 2 105 Combination of HVM on CB500 CB520H server blade B2 model Destination Source 01 60 Between 01 70 and 01 90 02 01 or higher 01 60 Yes Yes No Between 01 70 and 01 90 No Yes No 02 01 or higher No No Yes 3 5 CB500 CB52...

Page 83: ...R on the destination server blade Otherwise you cannot have the LPAR migrate onto HVM firmware version 01 51 or earlier 3 7 CB500 CB540A server blade A1 model We don t support LPAR migrations to HVM firmware version 01 60 from another HVM firmware version Table 2 108 Combination of HVM on CB500 CB540A server blade A1 model Destination Source 01 30 01 40 to 01 51 01 60 Between 01 70 and 01 90 02 01...

Page 84: ...No 02 01 or higher No No No No Yes 1 See Note on expansion of VNIC System No 2 When an LPAR has migrated from HVM firmware 01 60 to HVM firmware 01 70 we recommend that you reboot the LPAR on the destination server blade Otherwise you cannot have the LPAR migrate onto HVM firmware version 01 51 or earlier 3 9 CB500 CB520X server blade B1 model Table 2 110 Combination of HVM on CB500 CB520X server ...

Page 85: ...e 02 05 or higher 02 05 or higher Yes 4 2 CB2500 CB520X server blade B1 model Table 2 113 Combination of HVM on CB2500 CB520X server blade B1 model Destination Source 02 01 or higher 02 01 or higher Yes 4 3 CB2500 CB520X server blade B2 model Table 2 114 Combination of HVM on CB2500 CB520X server blade B2 model Destination Source 02 10 or higher 02 10 or higher Yes Concurrent Maintenance ...

Page 86: ... No Advanced Yes Enterprise Yes Table 2 117 HVM model CB2500 HVM model Supported Essential No Advanced Yes Enterprise Yes HVM system time zone The time zones for the source and destination HVMs must be the same In Japan set the time zones to 9 00 HVM system time synchronization with NTP server You must set the same server as the NTP server to synchronize HVM system times for the source and destina...

Page 87: ...ommunication settings cannot be applied in an LPAR migration in Concurrent Maintenance mode even if you have configured any TLS communication settings for ordinary communications For migration path see Migration path in Application requirement for Concurrent Maintenance mode in Application requirements in Usage environment Also for security strength see HVM Management Command HvmSh Operation Guide...

Page 88: ...e following table Table 2 118 LPAR migration on HVMs on CB 2000 standard server blades Destination Source HVM firmware version 59 11 or earlier 59 21 or higher HVM firmware version VNIC System No 128 or smaller 128 or smaller 129 or greater 59 11 or earlier 128 or smaller Yes Yes No 59 21 or higher 128 or smaller Yes Yes Yes 129 or greater No Yes Yes Table 2 119 LPAR migration on HVMs on CB 2000 h...

Page 89: ...tdown mode Confirm the combinations of the source and destination between which you can execute LPAR migration in Shutdown mode in the following table Table 2 120 LPAR migration on HVMs on CB 500 server blades Destination Source HVM firmware version 01 30 or earlier 01 40 or higher HVM firmware version VNIC System No 128 or smaller 128 or smaller 129 or greater 01 30 or earlier 128 or smaller Yes ...

Page 90: ...No No Connected directly to storage device P2P connection No No Loop connection No No Available when only 8Gbps Fibre Channel adapters are installed Connecting to storage device You must connect FC cables to the same ports on the same storage device from the source and destination server blades via FC switch modules SAN security You must register a Migration WWPN created with the same vfcID as a W...

Page 91: ...bes the setting of a migration path for CB 2000 CB 500 or CB 2500 Be sure to confirm the following table before configuring a migration path Table 2 122 Setting of migration path Migration path CB2000 CB500 CB2500 Path Adapter Management path No1 Yes Yes Not management path Emulex 10 Gbps NIC2 Yes Yes Yes Intel 1 Gbps NIC3 Yes Yes Yes Intel 10 Gbps NIC No No No Broadcom3 Yes Yes Yes Notes 1 On CB2...

Page 92: ...ing to the following network configurations The recommended network configuration for each server configuration varies depending on the kind of NIC set as a migration path to be used during a migration and other factors When you set a port of the onboard NIC or a mezzanine NIC in use as the migration path to be used Figure 2 6 Recommended network configuration Path configuration 1 Concurrent Maint...

Page 93: ...VM Navigator User s Guide Migration When you set a port of the onboard NIC or a mezzanine NIC exclusively as the migration path to be used Figure 2 7 Recommended network configuration Path configuration 2 Concurrent Maintenance ...

Page 94: ...environments HVM Navigator User s Guide Migration When you set a port of a PCI card in use as the migration path to be used Figure 2 8 Recommended network configuration Path configuration 3 Concurrent Maintenance ...

Page 95: ...ments 2 73 HVM Navigator User s Guide Migration When a port of a PCI card is set exclusively as the migration path to be used Figure 2 9 Recommended network configuration Path configuration 4 Concurrent Maintenance ...

Page 96: ...ess a value of Hvm Operating Mode Case1 When segments with the same shared NIC number are assigned to nonconsecutive Virtual NIC Numbers Ex 2a and 2b are assigned to Virtual NIC Number 2 and 4 Figure 2 10 Segments with the same shared NIC number are assigned to nonconsecutive Virtual NIC Numbers Case2 When segments are assigned to any of Virtual NIC Number 8 through 15 Ex 1a and 1b are assigned to...

Page 97: ...h 6 Figure 2 12 A segment is assigned two or more times to an LPAR Case4 When only some of the segments belonging to a controller with multiple ports is assigned to an LPAR Ex Only 1a belonging to a controller with two ports is assigned to Virtual NIC Number 0 Figure 2 13 Some of the segments belonging to a controller with multiple ports is assigned to an LPAR Concurrent Maintenance ...

Page 98: ...the combination of migratable environments between the source and destination HVMs see the following tables Table 2 123 Combination of migratable environments CB2000 standard server blade Destination Source HVM firmware version HVM firmware version VNIC Device Type 58 8x or earlier 59 0x or higher Standard Expansion Standard Expansion 58 8x or earlier Yes Yes Yes 59 0x or earlier NIC1 Yes Yes Yes ...

Page 99: ...ber of times shown in the following table you must reboot the server blade or execute Force Recovery according to Hitachi Compute Blade 500 Series Logical partitioning manager User s Guide Table 2 125 Migratable number of times to HVM firmware version 01 01 HVM firmware version Number of assigned shared NIC ports 2 4 01 01 200 400 Concurrent Maintenance ...

Page 100: ...2 78 Usage environments HVM Navigator User s Guide Migration ...

Page 101: ... Migration procedures This chapter describes the migration procedures Shutdown mode Concurrent Maintenance mode Policy migration Return migration Exchanging or updating hardware or software Saving and backing up HVM configuration Registering removing migration WWN ...

Page 102: ... have the LPAR migrate to another server blade is introduced in this section You can confirm a result of migration more easily by having only one LPAR migrate at a time Therefore when you have an LPAR which hasn t migrated in the current system configuration migrate we recommend that you have the LPAR migrate in Shutdown mode ...

Page 103: ...hutdown window Operation on HVM Navigator Operation on Web console for CB 500 Web console for CB 2500 SC BSM Execute LPAR migration in Shutdown mode Migration Shutdown window Check LPAR migration in progress Migration Progress window Check LPAR migration completion Migration Progress window Are there LPARs necessary to be recovered The migrated LPAR is running Recover LPARs See Recovering flow Yes...

Page 104: ...e HVMs and then back up the HVM configurations For saving and backing up HVM configuration see Saving and backing up HVM configuration in Migration procedures 2 Select Migration tab in Main window Save HVM configurations on the source and destination server blades when you have changed the HVM firmware versions or the HVM configurations Also back up the HVM configurations on the source and destina...

Page 105: ...Migration procedures 3 5 HVM Navigator User s Guide Migration 3 Click Update button 4 Select Shutdown radio button in Mode Selection group box and then click Migration button in Migration Menu group box ...

Page 106: ...igration Shutdown window is displayed and HVMs for CB2000 CB320 CB500 or CB2500 which are registered to HVM Navigator appear in Source group box and Destination group box 5 Select the source LPAR in Source group box and the destination LPAR in Destination group box ...

Page 107: ...oxes Uncheck Tree view check box in Target Selection group box in Migration Shutdown window Then the combo boxes to select source and destination chassis HVMs LPARs are displayed Besides in destination group box the combo box to select processor group So choose the way to select them according to the system environment ...

Page 108: ...box in Migration Shutdown window Then the function which selects the destination LPAR automatically is enabled When you check Auto check box and then select an HVM in Destination group box the LPAR with the youngest LPAR number is selected as the destination LPAR automatically However when you change the destination LPAR to another LPAR select the destination LPAR manually ...

Page 109: ...essor group for the destination LPAR a Check Group check box in Target Selection group box in Migration Shutdown window b Select an HVM in Destination group box c Select a processor group in Destination group box d Available LPARs are displayed in LPAR other Select the destination LPAR ...

Page 110: ...u check Shutdown Guest OS Source check box varies depending on the LPAR status of the LPAR to migrate When the LPAR status is Act Activated you must check Shutdown Guest OS Source check box and set the items in Guest OS Information for remote shutdown group box In addition Activate Destination LPAR check box is the setting to specify the LPAR status on the destination LPAR after LPAR migration ...

Page 111: ...ion 7 Confirmation window is displayed Confirm the content and then if the content has no problems click OK button 8 When executing LPAR migration in Shutdown mode Migration Progress window is displayed You can confirm the progress in Migration Progress window ...

Page 112: ...3 12 Migration procedures HVM Navigator User s Guide Migration 9 When LPAR migration finishes normally Migration completed message is displayed in Migration Progress Click Close button ...

Page 113: ...gator Recover an LPAR Are there the other LPARs necessary to be recovered The migrated LPAR is running Yes No Figure 3 2 Operation flow to recover LPAR When there are LPARs which have failed to be recovered on an HVM restore the HVM configuration with the backup file retrieved before migration However take note that the HVM configuration is returned to the configuration corresponding to the backup...

Page 114: ...ress window and recovering LPAR is often executed automatically However there are some cases which are recovered automatically depending on the kind of errors or the states of HVM or LPAR configurations in a momentary time which the error occurs In such a case recover all LPARs necessary to be recovered and execute LPAR migration again If the automatic recover explained above fails recover LPARs a...

Page 115: ...ng recovery x LPARs where x means the number of the LPARs If you have encountered a failure in Migration Policy window during policy migration click Migration button in Main window to open Migration window 3 In Source or Destination group box each LPAR necessary to be recovered is indicated by a pin icon Select the first one with a mouse click and then click Recovery Execute button If you check Sh...

Page 116: ...3 16 Migration procedures HVM Navigator User s Guide Migration 4 Confirmation window is displayed Confirm the content and then click OK button ...

Page 117: ...er s Guide Migration 5 Migration Progress window is displayed You can confirm the progress of recovering an LPAR 6 When recovering an LPAR finishes normally Recovery completed message is displayed in Recovery Progress window Click Close button ...

Page 118: ...s the number of the LPARs Recover all LPARs necessary to be recovered If the recovery process fails the error cause may not be removed Eliminate the error cause and retry Also confirm the HVM because HVM possible to has a hardware failure If the recovery process fails even though removed the error cause and retry reactivate the HVM using back up configuration information If executed LPAR migration...

Page 119: ...re already registered Set at least one migration path for each of the source and destination HVMs You don t need to set the above items whenever you execute LPAR migration in Concurrent Maintenance mode However you must set the above items only when you have changed the environment to which an LPAR to migrate belongs Operation on HVM Navigator Start HVM Navigator Register Storage Navigator Modular...

Page 120: ...Register the migration WWPNs assigned to the LPAR to migrate to the host group to which the WWPNs assigned to the LPAR to migrate are already registered You can register the migration WWPNs in the same way as the way to register the WWPNs You can register migration WWPNs to the host groups to which the WWPNs assigned to the LPAR to migrate are already registered through HVM Navigator For the opera...

Page 121: ...following figures Don t add a migration WWPN in a zone with the WWPN corresponding to the counterpart of the migration WWPN Correct configuration zone Zone OS WWN_OS WWN_D Zone with WWPN Zone OS Zone necessary to be configured for LPAR migration in Concurrent Maintenance mode Zone OS_Mig zone Zone OS_Mig WWN_OS_Mig WWN_D Guest OS FC WWN_OS WWN_OS_Mig FCSW Storage device LU WWN_D WWN_OS WWPN on an ...

Page 122: ...g WWN_D Zone with WWPN Zone OS Typical wrong configuration Guest OS Zone with WWPN Zone OS FC WWN_OS WWN_OS_Mig FCSW Zone OS WWN_D WWN_OS_Mig WWN_OS Storage device LU WWN_D WWN_OS WWPN on an FC port on a guest OS WWN_OS_Mig migration WWPN on an FC port on a guest OS Typical wrong configuration ...

Page 123: ...ath for each of the source and destination HVMs Be sure to confirm Migration path in Application requirement for Concurrent Maintenance mode in Usage environments before moving onto the following steps 5 Select Concurrent Maintenance radio button in Mode Selection group box and then click Path Setting button in Migration Menu group box ...

Page 124: ...3 24 Migration procedures HVM Navigator User s Guide Migration ...

Page 125: ...dow is displayed 6 Select the source or destination HVM in HVM Selection group box and then click Add Row button in Migration Path List group box Next check the check box in Primary Use column and input a name for the migration path in Name column Finally select a segment in Segment column ...

Page 126: ...bnet mask in the Subnet Mask column Note that any IPv6 addresses do not work as migration paths 8 Also set at least one migration path for the source or destination HVM in the same way as the way explained above Then click Save button We recommend that you set a VLANID for each migration path to isolate packets from those in another network As a result the packets are not referred to through anoth...

Page 127: ...Migration procedures 3 27 HVM Navigator User s Guide Migration 9 Click OK button 10 Save window is displayed Click OK button 11 Click Close button ...

Page 128: ...destination server blades Back up HVM configurations on source and destination server blades Start HVM Navigator Click Update button Main window Select source and destination LPARs Migration Shutdown window Operation on HVM Navigator Operation on Web console for CB 500 Web console for CB 2500 SC BSM It s necessary that you execute the above two when you haven t done them Check CPU load and network...

Page 129: ...Maintenance mode Migration Config Viewer window Are there no error Yes No Take measures for errors Are there LPARs with WWNs which need to be rolled back Roll back WWNs See Rolling back flow No Check if there are no errors HVM Console window Yes Fig 3 38 Reconfigure a clustering configuration with the destination server blade Figure 3 4 Operation flow for Concurrent Maintenance mode For the proced...

Page 130: ...PAR migration in Concurrent Maintenance mode may occur when you execute LPAR migration in Concurrent Maintenance mode under the condition that one or more of the above items are fulfilled Save HVM configurations on the source and destination server blades when you have changed HVM firmware versions or the HVM configurations Also back up HVM configurations on the source and destination server blade...

Page 131: ...Migration procedures 3 31 HVM Navigator User s Guide Migration 4 Select Migration tab 5 Click Update button ...

Page 132: ...ance radio button in Mode Selection group box and then select Migration button in Migration Menu group box Migration Concurrent Maintenance window is displayed and HVMs for CB2000 CB320 CB500 or CB2500 which are registered to HVM Navigator appear in Source group box and Destination group box ...

Page 133: ...Migration procedures 3 33 HVM Navigator User s Guide Migration 7 Select the source LPAR in Source group box and select the destination LPAR in Destination group box ...

Page 134: ...eck Tree view check box in Target Selection group box in Migration Concurrent Maintenance window Then the combo boxes to select the source and destination chassis HVMs LPARs are displayed Besides in the destination group box the combo box to select a processor group So choose the way to select them according to the system environment ...

Page 135: ...Migration Concurrent Maintenance window Then the function which selects the destination LPAR automatically is enabled When you check Auto check box and then select an HVM in Destination group box the LPAR with the youngest LPAR number is selected as the destination LPAR automatically However when you change the destination LPAR to another LPAR select the destination LPAR manually ...

Page 136: ...roup for the destination LPAR a Check Group check box in Target Selection group box in Migration Concurrent Maintenance window b Select an HVM in Destination group box c Select a processor group in Destination group box d Available LPARs are displayed in LPAR other Select the destination LPAR ...

Page 137: ...n HVMs FC HBA ports installed on the source and destination server blades are stable There are no errors for the source and destination HVMs The FC HBA ports installed on the source and destination server blades are stable When a log showing that the state of an FC HBA port has changed hasn t displayed for 5 or more minutes you can see that all FC HBA ports installed on the source and destination ...

Page 138: ...3 38 Migration procedures HVM Navigator User s Guide Migration 10 Click Show Config button ...

Page 139: ...on Config Viewer window the configuration requirements aren t fulfilled Fulfill all configuration requirements and then execute LPAR migration in Concurrent Maintenance mode For items shown in Migration Config Viewer see Appendix A Item in Migration Config Viewer window Also the measures which you should take when the Error icon is shown in each row are shown in Appendix A Item in Migration Config...

Page 140: ...3 40 Migration procedures HVM Navigator User s Guide Migration 12 Click Migration Execute button 13 Click OK button ...

Page 141: ...or User s Guide Migration LPAR migration in Concurrent Maintenance mode starts 14 When executing LPAR migration in Concurrent Maintenance mode Migration Progress window is displayed You can confirm the progress in Migration Progress window ...

Page 142: ...3 42 Migration procedures HVM Navigator User s Guide Migration 15 When LPAR migration finishes normally Migration completed message is displayed in Migration Progress Click Close button ...

Page 143: ... System Logs screen in HVM Console window If no errors have occurred you can see that LPAR migration in Concurrent Maintenance mode finished normally Operate HVM Navigator according to the following procedure to display HVM System Logs screen 1 Select the source and destination HVMs and then click Show System Logs button in Migration Concurrent Maintenance window ...

Page 144: ...3 44 Migration procedures HVM Navigator User s Guide Migration 2 Confirm if no errors have occurred on the source and destination HVMs ...

Page 145: ...igator Recover an LPAR Are there the other LPARs which need to be recovered A migrated LPAR is running Yes No Figure 3 5 Operation flow to recover LPAR When there are LPARs which have failed to be recovered on an HVM restore the HVM configuration with the backup file retrieved before migration However take note that the HVM configuration is returned to the configuration corresponding to the backup...

Page 146: ...ss window and recovering LPAR is often executed automatically However there are some cases which are recovered automatically depending on the kind of errors or the states of HVM or LPAR configurations in a momentary time which the error occurs In such a case recover all LPARs necessary to be recovered and execute LPAR migration again If the automatic recover explained the above fails recover LPARs...

Page 147: ...covered exist the number of the LPARs is shown at Target Selection group box in Migration Shutdown window in the form of LPARs requiring recovery x LPARs where x means the number of the LPARs If you have encountered a failure in Migration Policy window during policy migration click Migration button in Main window to open Migration window ...

Page 148: ...ch LPAR necessary to be recovered is indicated by a pin icon Select the first one with a mouse click and then click Recovery Execute button If you check Show the LPARs check box only the LPARs necessary to be recovered are shown 4 Confirmation window is displayed Confirm the content and then click OK button ...

Page 149: ...ing an LPAR finishes normally Recovery completed message is displayed in Recovery Progress window Click Close button As mentioned before the number of LPARs necessary to be recovered is shown in Target Selection group box in Migration window in the form of LPARs requiring recovery x LPARs where x means the number of the LPARs Recover all LPARs necessary to be recovered ...

Page 150: ...emoved Eliminate the error cause and retry Also confirm the HVM because HVM possible to has a hardware failure If the recovery process fails even though removed the error cause and retry reactivate the HVM using back up configuration information If executed LPAR migration after back up the LPAR revert to the former state ...

Page 151: ...rate HVM Navigator according to the following procedure if there are LPARs with WWNs necessary to be rolled back Operation on HVM Navigator Recover an LPAR Are there the other LPARs which need to be recovered A migrated LPAR is running Yes No Figure 3 6 Operation flow to roll back WWNs ...

Page 152: ...ARs You cannot have all LPARs migrate on an HVM where LPARs with WWNs necessary to be rolled back are located without rolling back WWNs for the LPARs Operate HVM Navigator according to the following procedure to roll back WWNs 1 Select an HVM where LPARs with WWNs necessary to be rolled back are located in Migration Shutdown window or Migration Concurrent Maintenance window After selecting the HVM...

Page 153: ...Migration procedures 3 53 HVM Navigator User s Guide Migration Rollback WWN window is displayed 2 Select an LPAR with WWNs necessary to be rolled back and then click Rollback Execute button ...

Page 154: ...3 54 Migration procedures HVM Navigator User s Guide Migration 3 Click OK button 4 Click OK button again after rolling back has completed ...

Page 155: ...us List group box Roll back the other LPARs with WWNs necessary to be rolled back in the same way Roll back WWNs for all LPARs with WWNs necessary to be rolled back If you don t roll back WWNs for any or all of the LPARs you cannot have all LPARs migrate on an HVM where LPARs with WWNs necessary to be rolled back are located ...

Page 156: ... cannot register entries for both Shutdown mode and Concurrent Maintenance mode together in a migration policy Save HVM configurations on the source and destination server blades when you have changed HVM firmware versions or the HVM configurations Also back up HVM configurations on the source and destination server blades together when you have changed any or both of the HVM configurations MAC ad...

Page 157: ...1 When HVMs to be the source and destination HVMs fulfill the first of the above notes save HVM configurations on the HVMs and then back up the HVM configuration For saving and backing up HVM configuration see Saving and backing up HVM configuration in Migration procedures 2 Click Migration Policy button in Migration Menu group box 3 Migration Policy window is displayed Click New button ...

Page 158: ...click Add Shutdown button or Add Concurrent Maintenance button When you add entries for Shutdown mode click Add Shutdown button On the other hand when you add entries for Concurrent Maintenance mode click Add Concurrent Maintenance button The following procedure is explained with the sample procedure to add entries for Concurrent Maintenance mode However when you add entries for Shutdown mode you ...

Page 159: ...ration 6 Add Concurrent Maintenance window is displayed Select the source LPAR in Source group box and select the destination LPAR in Destination group box Confirmation window is displayed Confirm the contents in Confirmation window and then Click OK button ...

Page 160: ...cedures HVM Navigator User s Guide Migration 7 Add entries to a migration policy by repeating the above 5 operation 8 Confirm entries in the created migration policy in Migration Policy window and then click Save button ...

Page 161: ...edures 3 61 HVM Navigator User s Guide Migration 9 Click OK button You can create a migration policy according to the above procedure When you need another migration policy create it according to the above procedure ...

Page 162: ...cute Policy migration Operate HVM Navigator according to the following procedure to execute Policy migration 1 Select a migration policy in the combo box in Migration Policy window When you execute Policy migration after creating or editing a migration policy click Release button ...

Page 163: ...Navigator User s Guide Migration 2 The entries registered in selected migration policy are displayed Confirm the entries and then click Execute button 3 Confirmation window is displayed Confirm the contents and then click OK button ...

Page 164: ...dures HVM Navigator User s Guide Migration Migration Progress window is displayed When a migration policy has completed Migration completed is displayed and is displayed in each row for an entry in Migration Policy window ...

Page 165: ...Migration procedures 3 65 HVM Navigator User s Guide Migration 4 Click Close button ...

Page 166: ...tton in Migration Policy migration is interrupted after the entry which is in progress when you click Stop button finishes Policy migration isn t interrupted while an entry is in progress 2 After interrupting Policy migration Confirmation window is displayed When you cancel the execution of the following entries click End button On the other hand when you execute the following entries click Contin...

Page 167: ... new migration policy with the same entries by changing the name for a migration policy Procedure to add entry Operate HVM Navigator according to the following procedure to add an entry to a migration policy 1 Select a migration policy in the combo box in Migration Policy window and then click Modify button 2 Click Add Concurrent Maintenance button ...

Page 168: ...gration 3 Add Concurrent Maintenance window is displayed Select the source LPAR in Source group box and select the destination LPAR in Destination group box Then click Add button Confirmation window is displayed Confirm the contents and then click OK button ...

Page 169: ...ocedure to create migration policy in Policy migration in Migration procedures 4 Confirm the entries in the changed migration policy in Migration Policy window and then click Save button or Save as button When you click Save as button Save as Policy window is displayed Input the name for a migration policy and then click OK button 5 Confirmation window is displayed Click OK button ...

Page 170: ...dure to delete entry Operate HVM Navigator according to the following procedure to delete an entry from a migration policy 1 Select a migration policy in the combo box in Migration Policy window and then click Modify button 2 Select an entry and then click Delete button ...

Page 171: ...s in the changed migration policy in Migration Policy window and then click Save button or Save as button When you click Save as button Save as Policy window is displayed Input the name for a migration policy and then click OK button 4 Confirmation window is displayed Click OK button ...

Page 172: ...cording to the following procedure to change the order to execute entries 1 Select a migration policy in the combo box in Migration Policy window and then click Modify button 2 Select an entry and then click Up button or Down button to change the order to execute the entry The example to click Up button is shown in the following figure ...

Page 173: ... User s Guide Migration 3 Click Save button or Save as button When you click Save as button Save as Policy window is displayed Input the name for a migration policy and then click OK button 4 Confirmation window is displayed Click OK button ...

Page 174: ...ware follow the procedure shown in Table 3 1 As an example the procedure to update HVM firmware is shown in Table 3 1 Table 3 1 Procedure to update HVM firmware with Concurrent Maintenance mode Process No Process Reference 1 Have all LPARs on HVM 1 migrate to HVM 2 Concurrent Maintenance mode in Migration procedures 2 Update HVM firmware Hitachi Compute Blade HVM Version Upgrade Revision Update Gu...

Page 175: ...s or the HVM configurations Also back up HVM configurations on the source and destination server blades together when you have changed any or both of the HVM configurations The expression in BSM SC may vary depending on the version Saving HVM configuration The procedure to save HVM configuration is as follows 1 Select HVM Management M HVM Configuration Management H in Server Conductor Blade Server...

Page 176: ...assis where the source or the destination HVM is located and then right click on the HVM Next click Save to HVM S 3 Click Yes button The following window is being displayed while saving HVM configuration When this window closes it tells that saving HVM configuration is completed ...

Page 177: ...n The procedure to back up HVM configuration is as follows 1 Select HVM Management M HVM Configuration Management H in Server Conductor Blade Server Manager window 2 Select the Chassis where the source or the destination HVM is located and then right click on the HVM Next click Save to HVM S ...

Page 178: ...3 78 Migration procedures HVM Navigator User s Guide Migration 3 Click Browse button and then set the folder to output a backup file and the file name Next click OK button 4 Click OK button ...

Page 179: ...restore HVM configurations on the source and destination server blades together You can find out that restoring HVM configurations have succeeded by confirming the following conditions are fulfilled There is an LPAR which has failed to be recovered on the source HVM The configurations on the above LPAR and the source HVM are the same as the previous backed up configuration There isn t an LPAR whic...

Page 180: ...only migration WWPNs to host groups through HVM Navigator Also you cannot register migration WWPNs to host groups when WWPNs aren t registered So confirm if WWPNs are registered in advance The way is supported in the following combination of storage devices and Storage Navigator Modular 2 CLI Table 3 2 Supported storage device and Storage Navigator Modular 2 CLI Supported storage device Supported ...

Page 181: ...ting window Remove migration WWPN Register migration WWN Setting window Check the result Register migration WWN Setting window Give migration WWPN a nickname Register migration WWN Setting window Check the result Register migration WWN Setting window Export CSV file Register migration WWN Setting window Complete Figure 3 7 Flow to register remove migration WWPN For the actions in see Prerequisite ...

Page 182: ...Navigator Modular 2 CLI If so remember the password On the other hand when Storage Navigator Modular 2 CLI isn t installed on the management server install Storage Navigator Modular 2 CLI version 11 50 or higher on the management server The other versions aren t supported For installing Storage Navigator Modular 2 CLI see Hitachi Unified Storage Command Line Interface Reference Guide Confirm if th...

Page 183: ...res 3 83 HVM Navigator User s Guide Migration 1 Start HVM Navigator For starting HVM Navigator see Hitachi Compute Blade HVM Navigator User s Guide Getting Started 2 Select Setting Option in the menu bar in Main window ...

Page 184: ...3 84 Migration procedures HVM Navigator User s Guide Migration 3 Select Migration Option in the left pane 4 Click Open button in Storage Navigator Setting ...

Page 185: ...Migration procedures 3 85 HVM Navigator User s Guide Migration 5 Select Storage Navigator Modular 2 CLI folder and then click OK button 6 Click Apply button or OK button ...

Page 186: ...3 86 Migration procedures HVM Navigator User s Guide Migration 7 Select Migration tab 8 Click Update button ...

Page 187: ...Migration procedures 3 87 HVM Navigator User s Guide Migration 9 Check Concurrent Maintenance radio button in Mode Selection group box ...

Page 188: ...gration 10 Click WWN Setting button in Migration Menu group box 11 Input the IP address for a port on a storage device You should input the IP address for the port to access the host group to which the WWPN created with the same vfcID is already registered ...

Page 189: ...Migration procedures 3 89 HVM Navigator User s Guide Migration 12 Click Search button 13 Select the IP address for a port on the storage device to register to HVM Navigator and then click Add button ...

Page 190: ...ng window is displayed Set the following parameters for security in this window Administration Mode The security parameter on the Storage Navigator Modular 2 CLI Password Protection One of the security parameters on a storage device Account Authentication One of the security parameters on a storage device ...

Page 191: ... as well In addition you can check Save password setting check box When you check the check box the above parameters are saved after clicking OK button So use the check box as needed Table 3 4 Setting for Administration mode Setting for Administration Mode on Storage Navigator Modular 2 CLI Value which you should set in Storage Password Setting window Administration Mode check box Password text bo...

Page 192: ...ained When Storage Password Setting window is displayed not only rows for the storage devices which you have registered in the previous steps but also those for the storage devices which have been registered in advance are displayed However you should input values for only the storage devices which you are managing you may not input those for the other storage devices 15 Check Administration Mode ...

Page 193: ...Migration procedures 3 93 HVM Navigator User s Guide Migration 17 Select Account Authentication in Security Type column 18 Input a user ID in User ID column and a password in Password column ...

Page 194: ...ssword Protection or Account Authentication the window showing that an execution file in Storage Navigator Modular 2 CLI folder is being executed after you click OK button on the following Confirm window While the window is being displayed don t click or type anything on the window If you do so you may not be able to register or remove migration WWPNs or give migration WWPNs nicknames ...

Page 195: ...onfirm if the storage devices which you have registered to HVM Navigator in the previous steps are displayed 22 Confirm registration status of WWPNs such as what WWPNs are registered what host groups the WWPNs are registered to and so on For the icons displayed in State Vfc column and State Mig column see List of icons ...

Page 196: ...host group Register migration WWPNs to host groups according to the following procedure 1 Check the check boxes for the migration WWPNs to register to host groups you can check the check boxes for the migration WWPNs which the icon in State Vfc is and the one in State Mig is 2 Click Add button ...

Page 197: ...Migration procedures 3 97 HVM Navigator User s Guide Migration 3 Click OK button 4 Click OK button 5 Confirm that the icons in both Last Status column and Status Mig column are ...

Page 198: ...ost group Remove migration WWPNs to host groups according to the following procedure 1 Check the check boxes for the migration WWPNs to remove from host groups you can check the check boxes for the migration WWPNs which the icon in State Vfc is and the one in State Mig is 2 Click Delete button ...

Page 199: ...Migration procedures 3 99 HVM Navigator User s Guide Migration 3 Click OK button 4 Click OK button 5 Confirm that the icon is in Last Status column and the one is in Status Mig column ...

Page 200: ...name Give migration WWPNs a nickname according to the following procedure 1 Check the check boxes for the migration WWPNs to give migration WWPNs a nickname you can check the check boxes for the migration WWPNs which the icon in State Vfc is and the one in State Mig is 2 Click Nickname button ...

Page 201: ...Migration procedures 3 101 HVM Navigator User s Guide Migration 3 Click Default button 4 Click Execute button ...

Page 202: ...3 102 Migration procedures HVM Navigator User s Guide Migration 5 Click OK button 6 Click OK button 7 Select Multiple Rows radio button ...

Page 203: ...Migration procedures 3 103 HVM Navigator User s Guide Migration 8 Confirm that the icon is in Last Status column and the nickname is displayed in Migration WWPN Nickname column ...

Page 204: ...de Migration Exporting CSV file Export a CSV file where the registration status of WWPNs and migration WWPNs is recorded according to the following procedure 1 Click Export CSV button 2 Select the folder to export the CSV file and then click Save button ...

Page 205: ...Migration procedures 3 105 HVM Navigator User s Guide Migration 3 Click OK button ...

Page 206: ...rocedures HVM Navigator User s Guide Migration Adding storage device Add storage devices according to the following procedure 1 Click Update button 2 Check Concurrent Maintenance radio button in Mode Selection group box ...

Page 207: ...splayed 4 All storage devices which are registered to HVM Navigator are displayed in Storage Password Setting window Of the storage devices you should input values for only the storage devices which you are managing you may not input those for the other storage devices Then click OK button For the parameters in Storage Password Setting window see Prerequisite Setting in Registering removing migrat...

Page 208: ...3 108 Migration procedures HVM Navigator User s Guide Migration 5 Click OK button 6 Click Add Storage button ...

Page 209: ...or User s Guide Migration 7 Input the IP address for a port on a storage device You should input the IP address for the port to access the host group to which the WWPN created with the same vfcID is already registered 8 Click Search button ...

Page 210: ...3 110 Migration procedures HVM Navigator User s Guide Migration 9 Select the IP address for the port on the storage device to register to HVM Navigator and then click Add button 10 Click OK button ...

Page 211: ...Migration procedures 3 111 HVM Navigator User s Guide Migration 11 Confirm that the storage devices have been added to HVM Navigator in Storage Selection group box 12 Click Password Setting button ...

Page 212: ...w see Prerequisite Setting in Registering removing migration WWN 14 Click OK button 15 Register Migration WWN Setting window is displayed Confirm that the added storage devices are displayed 16 Confirm registration status of WWPNs such as what WWPNs are registered what host groups the WWPNs are registered to and so on For the icons displayed in State Vfc column and State Mig column see List of ico...

Page 213: ...ign permhg 0 A migration WWPN gno 001 Register a migration WWPN to a host group This command is used when any command with the migration WWPN hasn t been executed in the past 9500V unit array unit name rm permhg 0 A migration WWPN migration WWNN gno 001 the other machine type unit array unit name rm permhg 0 A migration WWPN gno 001 Remove a migration WWPN from a host group 9500V auhgwwn unit arra...

Page 214: ...es HVM Navigator User s Guide Migration Command Use on HVM Navigator auaccountenv set uid 001 authentication unit array unit name Set account data onto a storage device auaccountenv rm Remove account data from a storage device ...

Page 215: ...n Other functions and settings This chapter describes some of the other functions and settings of migration Functions and settings for Shutdown and Concurrent Maintenance modes Functions and settings for Shutdown mode Functions and settings for Concurrent Maintenance mode ...

Page 216: ...nditions on your selections before executing LPAR migration in Shutdown mode or in Concurrent Maintenance mode Follow the procedure below 1 When you are to execute LPAR migration in Shutdown mode or in Concurrent Maintenance mode select the source LPAR and the destination LPAR in the Migration Shutdown or Migration Concurrent Maintenance window and then click Show Config button in Target Selection...

Page 217: ...ation policy and the entry in the target migration policy in the Migration Policy window and then click Show Config button in the same window 2 In the Migration Configuration Viewer window confirm if the source and the destination values are identical in the group boxes of Server Configuration HBA Configuration and NIC Configuration ...

Page 218: ...Error icon is shown in each row are shown in Appendix A Item in Migration Config Viewer window Specifying LAN port IP address for management server LAN port IP address of HVM Navigator Management server is registered as a BSM IP address on HVM for security reason This registered IP address can be the only accessible address to HVM If there are multiple HVM accessible LAN port on HVM Navigator Mana...

Page 219: ...ist available on Migration tab in the Main window By tracing back this list you can find the history and results of migration on the target server By checking Show LPAR History check box in the same window a classified list of migration history by LPARs is displayed in the newly opened Migration History window From this list you can find the original HVM server blade and LPAR number for the target...

Page 220: ...ation by LPAR name per migration instance represented as follows Now Information on the target LPAR on the HVM where the target LPAR is running old1 Information on the target LPAR on the HVM where the target LPAR was running before the last one migration instance oldxx Information on the target LPAR on the HVM where the target LPAR was running before the last xx migration instances ...

Page 221: ...plays set Allow inbound remote administration exception to Enable in the window The subsequent operating procedure depends on OS version Follow the procedure described on website Setup2 Setting up local accounts Sharing and security model for local accounts must be set to Classic Select Start Run and then type secpol msc After a window displays set Sharing and security model for local accounts to ...

Page 222: ...ing your log in information to the server With this option enabled a failed migration attempt due to a power on state of the source LPAR is retried after performing automatic shutdown of the source server This capability is called remote shutdown in this manual and related manuals To enable remote shutdown follow the procedure below 1 In the Migration window or Add Migration window specify the fol...

Page 223: ...ddress with the specified user ID and password Accordingly the HVM Navigator installed management server must be provided with a physical path to the source server always accessible by the specified user while the source server must be set to accept access from the specified user Also be noted that enabling of Forced shutdown may cause corruption of in process data because the OS is forcibly shut ...

Page 224: ...VM When you change the above settings in the latter way operate HVM Navigator according to the following procedure 1 In Migration Shutdown window or Add Migration window uncheck Activate Destination LPAR check box in migration option group box in Migration Setting group box 2 Execute LPAR migration in Shutdown mode 3 Ensure normal completion of the execution of migration 4 Change the above setting...

Page 225: ...l to confirm if the resources such as CPU memory NIC FC HBA and so on assigned to an LPAR are available on the destination HVM Operate HVM Navigator according to the following procedure to execute Rehearsal 1 Select the source LPAR in Source group box and select the destination LPAR in Destination group box and then click Rehearsal button in Rehearsal Execute group box ...

Page 226: ...4 12 Other functions and settings HVM Navigator User s Guide Migration 2 Confirmation window is displayed Click OK button Rehearsal starts ...

Page 227: ...ions and settings 4 13 HVM Navigator User s Guide Migration 3 Migration Progress window is displayed Confirm the progress of Rehearsal 4 When Rehearsal finishes Migration completed is displayed Click Close button ...

Page 228: ...igration paths for the source and destination HVMs is available Operate HVM Navigator according to the following procedure to execute Connectivity Test 1 Select the source LPAR in Source group box and select the destination LPAR in Destination group box and then click Connectivity Test button in Rehearsal Execute group box 2 Confirmation window is displayed Click OK button ...

Page 229: ...Other functions and settings 4 15 HVM Navigator User s Guide Migration Connectivity Test starts 3 Migration Progress window is displayed Confirm the progress of Connectivity Test ...

Page 230: ...4 16 Other functions and settings HVM Navigator User s Guide Migration 4 When Connectivity Test finishes Connectivity Test completed is displayed Click Close button ...

Page 231: ...ion in Concurrent Maintenance mode is set to 900 seconds as default You can extend a timeout period for LPAR migration in Concurrent Maintenance mode in Migration Progress window only while memory data on the source LPAR are being transferred Operate HVM Navigator according to the following procedure to extend a timeout period for LPAR migration in Concurrent Maintenance mode 1 Specify an extended...

Page 232: ...4 18 Other functions and settings HVM Navigator User s Guide Migration 2 Extend window is displayed Click OK button The extended timeout period is displayed in Progress Detail group box ...

Page 233: ... so on you can cancel the LPAR migration You can cancel LPAR migration in progress in Migration Progress window only while memory data on the source LPAR are being transferred Operate HVM Navigator according to the following procedure to cancel LPAR migration in progress 1 Click Cancel Migration button while executing migration process Stage 0x4050 SRC Transferring Memory Data is being displayed 2...

Page 234: ...4 20 Other functions and settings HVM Navigator User s Guide Migration When Canceling LPAR migration has completed Progress Migration completed is displayed in Migration Progress window ...

Page 235: ...e destination LPAR You can monitor them only while memory data on the source LPAR are being transferred Operate HVM Navigator according to the following procedure to monitor memory size remaining on the source LPAR and transfer rate of memory date to the destination LPAR 1 Click Show Monitor button while executing migration process Stage 0x4050 SRC Transferring Memory Data is being displayed ...

Page 236: ...nation LPAR 3 Take any of the measure to extend a timeout period for LPAR migration in Concurrent Maintenance mode and the one to cancel LPAR migration depending on the memory transfer state For extending a timeout period for LPAR migration in Concurrent Maintenance mode see Extending timeout period for LPAR migration in Concurrent Maintenance mode Also for canceling LPAR migration see Canceling L...

Page 237: ... storage device Changing VNIC System No on the system environment where LPAR migration has been executed Migration of parameters to apply remote shutdown to source LPAR Service ratio after migration Migration of server with dedicated PCI device allocation Migration of MAC addresses Migration of WWPNs Migration of LPAR enabling VC Virtual COM console Migration of LPAR with assigned virtual NICs Por...

Page 238: ...ting migrations by operating multiple HVM Navigators Migration timeout in Concurrent Maintenance mode Setting of scheduling operation for the LPAR to migrate Backing up HVM configuration by operating SC BSM SC BSM alerts for Shutdown mode SC BSM alerts for Concurrent Maintenance mode SC BSM Display after Migration with Windows Server 2008 R2 N M Failover during Migration Migrating LPAR form the st...

Page 239: ...System No on the system environment where LPAR migration has been executed LPAR Migration of parameters to apply remote shutdown to source LPAR Service ratio after migration Migration of server with dedicated PCI device allocation Migration of MAC addresses Migration of WWPNs Migration of LPAR enabling VC Virtual COM console Migration of LPAR with assigned virtual NICs Ports with Inter LPAR Packet...

Page 240: ...etting of scheduling operation for the LPAR to migrate Backing up HVM configuration by operating SC BSM SC BSM alerts for Shutdown mode SC BSM alerts for Concurrent Maintenance mode SC BSM Display after Migration with Windows Server 2008 R2 N M Failover during Migration Migrating LPAR form the standby server blade after Cold standby switching System Console Test of Configuring N M Cold Standby UPS...

Page 241: ...VM configuration is backed up save the HVM configuration or execute LPAR migration and then back up the HVM configuration again Backing up HVM configuration An HVM configuration can be backed up through an SC BSM the web console for CB500 or the web console for CB2500 Back up HVM configurations on the source and destination server blades and maintain the two HVM configurations together Restoring H...

Page 242: ...configurations are initialized in the environment which you execute LPAR migration in LPARs which aren t determined on the HVM do not exist LPARs to migrate from the HVM don t exist on one of the other HVMs See Section Tracing Back Migration History on Servers LPARs Tracing Back Migration History on Servers LPARs for inspect the conditions above The above notes are not applied when DC commands are...

Page 243: ...n the HVM which a target LPAR is running The HVM which a target LPAR is running Migration in progress 1 One or more LPARs needs to be recovered 2 3 Parameter you can operate Parameter you cannot operate 1 An operation which you deactivate or reactivate the LPAR to migrate through an SC BSM and an operation to deactivate an LPAR automatically through the scheduling operation in an SC BSM are availa...

Page 244: ...orts are down you cannot execute LPAR migration in Concurrent Maintenance mode In this case an Error icon is shown in Device Status row in either of the HBA configuration table or the NIC configuration table in the Migration Config Viewer window Then confirm the network environment on the destination HVM link NIC and FC HBA ports and execute LPAR migration in Concurrent Maintenance mode again High...

Page 245: ...ng of Boot Device List Table 5 3 Different point between source and destination and major measures Different point Measure Setting items on EFI driver Ports on a storage device which are connected to the server blade 3 Storage devices which are connected to the server blade 2 3 Way to connect to a storage device connection to storage devices through FC switch module connection directly to storage ...

Page 246: ...n to source LPAR Parameters inside Guest OS Information for remote shutdown subgroup of Migration Setting group box in the Migration or Add Migration window are transferred with the LPAR when the execution of migration has been successful and are not transferred when the said execution has failed Of the failed cases if an error has been detected after migrating the LPAR configuration information t...

Page 247: ...ersion Migration functions of HVM Navigator do not support migration of a server to which PCI devices are allocated in dedicated scheduling mode As a workaround the device scheduling mode at the source should be temporarily changed to shared mode prior to migration and further changed to dedicated mode at the destination after the migration Be noted that a change in the device scheduling mode requ...

Page 248: ...le VCs a VC is allocated to the LPAR Migration of LPAR with assigned virtual NICs An LPAR with assigned virtual NICs Va Vd cannot communicate with the other LPARs on the source HVM which has communicated with after the LPAR migrates When such an LPAR migrates confirm the network settings For virtual NICs see Hitachi Compute Blade CB2000 User s Guide Hitachi Compute Blade CB320 User s Guide Hitachi...

Page 249: ...ou have an LPAR to which a dedicated USB device is assigned migrate in Shutdown mode operate the LPAR as follows Release the dedicated USB device from the LPAR migrate the LPAR and then assign the dedicated USB device to the LPAR When you have an LPAR migrate in Concurrent Maintenance mode An LPAR to which a dedicated USB device is assigned cannot migrate in Concurrent Maintenance mode ...

Page 250: ...gher 79 00 or higher 17 82 or higher 01 00 or higher or 02 01 or higher migrates in Shutdown mode is not transferred For the state on the destination HVM depending on HVM firmware version and the state on the source serer blade and the LPAR status on the destination server blade see the following table Table 5 4 Assigned state of USB on destination HVM HVM firmware version Assigned state on source...

Page 251: ...in HVM Options is set to Enable the assigned state depends on the assigned states of the other LPARs on the destination server blade When an LPAR which assigned state is R running exists the assigned state of the LPAR to migrate becomes A assigned However When an LPAR which assigned state is R running does not exist the assigned state of the LPAR to migrate becomes R running For USB Auto Allocatio...

Page 252: ...igration are displayed or retrieved However WWNs exchanged by migration are assigned to the FC HBA port when the scheduling mode for the FC HBA is changed to dedicated mode The avoidances for the above problems are shown below 1 Confirm WWNs of FC HBA port in dedicated mode on the display by drvcfg command in EFI driver 2 Do not change FC HBA ports assigned to the LPAR to migrate to dedicated mode...

Page 253: ...ows Server 2003 Shutdown process on Windows Server 2003 may interrupt on the conditions shown on Microsoft Support In such a case solve the problem by stopping the screen saver at first Then it may be required to stop the screen saver and logon screen saver of the guest OS shut down depending on the usage environment of guest OS For the detail information see Microsoft Support Microsoft Support at...

Page 254: ...5 18 Usage notes HVM Navigator User s Guide Migration Figure 5 1 Migration Progress window ...

Page 255: ...e LPAR is migrating in Concurrent Maintenance mode packets sent to the LPAR with UDP communication are lost So send the packets to the destination LPAR after finishing LPAR migration However packets sent to the LPAR with TCP communication are sent automatically again after finishing LPAR migration Memory high load on guest OS on the LPAR to migrate When memory writing rate is faster than memory tr...

Page 256: ...ng Adjust LPAR Time in Date and Time screen in HVM screen instead For Adjust LPAR Time see Hitachi Compute Blade 2000 USER S GUIDE Hitachi Compute Blade 500 Series Logical partitioning manager User s Guide or Hitachi Compute Blade 2500 Series Logical partitioning manager User Guide Network setting on Linux OS on the LPAR to migrate When the guest OS running on the LPAR to migrate is Linux and mult...

Page 257: ... mode or roll back WWNs The kind of the states of FC HBA ports and the messages to be displayed in HVM System Logs screen when the states of FC HBA ports are changed are shown in the following table Table 5 5 Kind of states of FC HBA ports and messages to be displayed Port state Event column Detail column Linked HVM detected available Shared FC Link Shared FC Link is Available Slot x Port x Down V...

Page 258: ...5 22 Usage notes HVM Navigator User s Guide Migration Figure 5 3 HVM Console window ...

Page 259: ...Navigator from registration in particular those objects which are continuously powered off or not existing Values of Processor Node and Memory Node When you execute LPAR migration values which are set on the source HVM aren t migrated and the values are changed to A on the destination HVM Table 5 6 Processor Node and Memory Node after LPAR migration Item Value Processor Node A Memory Node A Displa...

Page 260: ...rent Maintenance mode passes For the way to calculate a time period for Concurrent Maintenance mode see a subsection Time period for Concurrent Maintenance mode in the chapter Usage environment Also for timeout period for LPAR migration in Concurrent Maintenance mode see a subsection Extending timeout period for LPAR migration in Concurrent Maintenance mode in the chapter Other functions and setti...

Page 261: ...ioned the remote shutdown function detects a power on state of the migration source server LPAR and terminates the migration execution to fail at first and then executes the server shutdown and retries the migration Be aware that the above messages indicate the said initial failure of migration before retry Final determination between success and fail for the migration should be made from the resu...

Page 262: ...elayed Start N M Failover during Migration In an environment where N M cold standby is configured N M failover may take place at a failure on the destination server during the execution of Migration functions of HVM Navigator On such occasions inconsistency of the LPAR configuration information between the source and the destination could result depending on the timing of failure To resolve such i...

Page 263: ...erConductor Blade Server Manager Plus 09 00 You operate HVMs according to the following procedure a Cold standby switching occurs between the active server blade A to the standby server blade B b Migrate an LPAR from the standby server blade B to the server blade C c Execute Recover to active system for the active server blade A and the standby server blade B To prevent the above event occurring f...

Page 264: ... executed then take the following actions 1 Remove the target server blade from the N M cold standby group 2 Boot the HVM on the target server blade 3 Press F9 on HVM menu screen and then save the HVM configuration 4 Turn off the server blade and then turn on the server blade again There are some cases that the same WWNs as another active server blade are displayed on the HVM screen Combined use w...

Page 265: ...the HVM is active In order to avoid inadvertent shutdown of HVM during migration of LPARs HVM Navigator suppresses the auto shutdown until the migration executed by Migration functions is complete By doing so the HVM remains active even during any scheduled migration where all LPARs on the HVM become deactivated power off state to save power or to execute scheduled maintenance Similarly migration ...

Page 266: ...5 30 Usage notes HVM Navigator User s Guide Migration ...

Page 267: ...gration Troubleshooting This chapter describes the troubleshooting Troubleshooting flow Taking action when LPAR activation has been disabled Taking action when Policy migration failed Troubleshooting FAQs Error information Collecting failure information ...

Page 268: ...cHVM or Connection refused MMS DstHVM is displayed Another message is displayed Check the setting of the migration paths used for the source and destination HVM Execute Create error log file in DL commands in System console and retrieve HVM dump files in Options screen in HVM screen Retrieve HVM dump files and Technical Information files for HVM Navigator in Main window Yes No Are there any errors...

Page 269: ...M screen shows The specified LPAR has corrupted in an LPAR migration please try to recover the LPAR Solution Execute manual recovery of the activation disabled LPAR in accordance with Executing manual recovery when automatic recovery is not done in this chapter Taking action when Policy migration has failed Event When Policy migration has failed the following window pops up In such a case follow t...

Page 270: ...he error content in the error stopped Migration Progress window Figure 6 3 Migration Progress window 2 Inspect the error stopped migration entry in the Migration Policy window Click Show Config button to open the Migration Config Viewer window Figure 6 4 Migration Policy window ...

Page 271: ...Troubleshooting 6 5 HVM Navigator User s Guide Migration 3 Inspect the configuration items of the source and destination LPARs in Migration Config Viewer window Figure 6 5 Migration Config Viewer window ...

Page 272: ... If the error cause has not been identified click Skip button to skip this migration entry and resume from the next entry or End button to terminate the Policy migration here In the event that you skipped or ended with an error stopped migration entry After the completion of Policy migration execute single migration inspect the error cause and take appropriate actions according to your finding Eit...

Page 273: ...opens Figure 6 7 Migration window When this event occurs there are three suspect causes JRE Java Runtime Environment is not installed or the installed JRE is not in an applicable version Environment variable to the installed JRE is not set or not properly set Communication between HVM Navigator and HVM is lost Take actions according to Solution 1 Solution 2 or Solution 3 corresponding to the said ...

Page 274: ...When JRE is not installed or the installed JRE is not in an applicable version 1 From the menu bar in the Main window click Setting and select Option Figure 6 8 Main window 2 In the Option window opened select Migration Service in the left pane Figure 6 9 Option window ...

Page 275: ...allation Manual Solution 3 When communication between HVM Navigator and HVM is lost 1 Confirm if the HVM registered to be managed by HVM Navigator is up and normally running through HVM screen host management window of SC BSM or HVM configuration management window of SC BSM 2 If you see no problem in the step above then inspect the link between the HVM and HVM Navigator by issuing diagnostic comma...

Page 276: ...on Remote shutdown of server has failed Event An LPAR migration has failed with a message including Process Guest OS Shutdown or Shutdown waiting Figure 6 11 Migration Progress window Solution 1 Click Update button in the Main window Figure 6 12 Main window ...

Page 277: ...opriate according to Section Executing Migration with Automatic Shutdown of Source Server of Chapter 4 b Ensure that the access from the management server to the target source server is not restricted by such means as personal firewall c For Windows 2008 Server confirm if the usage note in Section Failure on Remote Shutdown of Windows Server 2008 of Chapter 6 d For Linux confirm if SSH Component N...

Page 278: ...e shutdown processing in this particular OS environment usually requires more than five minutes change the wait time value by the following operation Select Migration Option from Setting menu in the Main window of HVM Navigator select Migration Option menu in the left pane of the Option window opened type a new value in Waiting TimeOut Limit text box and click Apply or OK button Figure 6 14 Option...

Page 279: ... LPAR has started migrating with the icon showing the LPAR status of the LPAR indicating deactivated in Migration window Figure 6 15 Migration Progress window This event occurs when the actual state of an LPAR is different from the one recognized by HVM Navigator e g Update operation was not executed or the state was externally changed by HVM screen or SC BSM Take actions according to the procedur...

Page 280: ...e Migration Solution 1 Click Update button in the Main window Figure 6 16 Main window 2 In the Migration window confirm the state of the error stopped LPAR If the source server turns out to be in the Activate state shut down it and retry migration ...

Page 281: ... that the subject HVM is not being operated from SC BSM and that HVM screen is not in use If a sub screen opened from HVM screen remains open the process of HVM screen is recognized busy Take actions according to the procedure in Solution below Solution 1 Confirm if HVM screen including sub screens is open at the source or at the destination If it is wait until the ongoing operation ends or termin...

Page 282: ... Close button in the currently open Migration window 2 Click Migration button on Migration tab in the Main window of HVM Navigator This event occurs when many objects are registered to HVM Navigator or when the management server running HVM Navigator is operating under a heavy load To make the operating environment more comfortable we recommend that you review the content of System Configuration t...

Page 283: ...onse Timeout Figure 6 19 Migration Progress window Solution Reset the time out period for connection with HVMs longer according to Hitachi Compute Blade HVM Navigator User s Guide Getting Started However if LPAR migration fails with this message in Refresh HVM Information process you could clear this error by clicking Update button in Main window ...

Page 284: ...lization of VM java lang Class Not Found Exception error in opening JAR file message Figure 6 20 Migration Progress window Solution Terminate unnecessary application to reserve enough physical memory and retry migration If this message is displayed again even though enough memory is reserved for migration inspect the link between the system and java Confirm the java path is set properly and retry ...

Page 285: ...the guest OS on migrated LPARs because Windows Firewall Internet Connection Sharing ICS service does not start on the guest OS on destination server blade after migration This issue is introduced on Microsoft support page http support microsoft com kb 922918 Solution See the web page as shown above and take measures as needed ...

Page 286: ...vice data cannot be completed within an LPAR suspension period This error occurs because a migrated LPAR cannot boot on the destination server blade within LPAR suspension time Figure 6 21 Migration Progress window Solution Execute the LPAR migration again When the LPAR migration fails though you attempt to the LPAR migration several times reduce the memory load as much as possible on the guest OS...

Page 287: ... the source and the destination server blade or the management path connected with the destination server blade Figure 6 22 Migration Progress window Solution Confirm if errors for management paths and or migration paths on the source and destination server blades occur in HVM System Logs screen in HVM Console window Then clear the errors and execute the LPAR migration again For confirming HVM Sys...

Page 288: ...message Event LPAR migration has failed with Wait until another operation is completed and retry This error occurs because you execute the LPAR migration while a popped up display in HVM screen is being displayed Figure 6 23 Migration Progress window Solution Close the popped up display in HVM screen and execute the LPAR migration again ...

Page 289: ...d Figure 6 24 Migration Progress window Solution Extend a timeout period for connection with HVMs We recommend that you extend a timeout period by about 10 seconds However it s not certain that a timeout doesn t occur by this measure though the measure makes the timeout be more difficult to occur So be sure that the timeout may occur again though you take this measure For setting a timeout period ...

Page 290: ...ination server blade hasn t completed within a timeout period for LPAR migration in Concurrent Maintenance mode Figure 6 25 Migration Progress window Solution Take either or both of the following measures When you take both measures it s easier for LPAR migration to succeed 1 Reduce memory load on the guest OS on the LPAR to migrate and then execute the LPAR migration again 2 Execute the LPAR migr...

Page 291: ...LPAR has migrated Figure 6 26 Migration Progress window Solution Take the measure according to the following procedure 1 Confirm if there are errors in HVM System Logs screen in HVM screen 2 When you find out one or more errors in HVM System Logs screen in HVM screen retrieve HVM dump files and Technical Information files for HVM Navigator in Main window For retrieving Technical Information files ...

Page 292: ...s that the source HVM has been requested to write some data on the area where the HVM isn t allowed to write by the guest OS Figure 6 27 Migration Progress window Solution Take the measure according to the following procedure 1 Retrieve HVM dump files and Technical Information files for HVM Navigator in Main window For retrieving Technical Information files see Hitachi Compute Blade HVM Navigator ...

Page 293: ...or Event LPAR migration has failed with EMG000000002ab4006 000000000000003e 0000000000000030 0 HVM SR C Internal error The HVM detected an FC error This error shows that I O load on the HVM has been too high to execute LPAR migration in Concurrent Maintenance mode because of the I O requests from the guest OS Figure 6 28 Migration Progress window Solution Execute the LPAR migration again ...

Page 294: ...storage device Figure 6 29 Migration Progress window Solution Take the measure according to the following procedure 1 Set the timeout period to log into storage device which can be set in HVM Navigator to the following value The timeout period to log into storage device the time period set in Login Delay Time textbox in Storage Login Setting in Option window 2 5 seconds For changing the timeout pe...

Page 295: ...HVM detected an FC error This error shows that LPAR migration has failed due to the zoning configuration for the guest OS to migrate Confirm the error code shown in Migration Progress window not to identify the error as another one Figure 6 30 Migration Progress window Solution Confirm the zoning configuration according to the explanation shown in Prerequisite setting in Concurrent Maintenance mod...

Page 296: ...where the NIC controllers assigned to the source LPARs are mounted on the source server blade may be different Figure 6 31 Migration Progress window Solution When VF NICs are assigned to the source LPAR confirm the scheduling modes of the controllers on the NIC cards mounted on the same slot numbers on the source and destination server blades are set to the same mode You can confirm if VF NICs are...

Page 297: ...es error message error code in red large characters and suggested actions in black normal characters Basically you are expected to follow such actions suggested For errors detected by the execution modules of Migration functions also see the detailed error information in detail field explained in the subsections below Figure 6 30 Migration Progress window ...

Page 298: ...S Migration Console module MMC cannot connect to Migration Management module MMS In the Option window opened from the Main window check the following make corrections if necessary and then retry migration The communication between management server HVM Navigator and the specified server blade is normal Select Migration Service in the left pane and see if Migration Service Status in right pane is R...

Page 299: ...ion The specified server blade HVM is up and running Communication to the IP address of the specified server blade is established 0x00 00000003 0020 02 0x0000 0000 0000 0000 0x01 The specified blade is busy Please wait until another migration is completed Check if the same server blade is specified as source and destination Wait for completion of preceding migration and then retry migration 0x00 0...

Page 300: ...nd then retry migration 0x00 00000001 0070 02 0x0000 0000 0000 0000 0x01 The specified LPAR is not found Communication was lost while acquiring information from server blade Check for recovery of connection and then retry migration 0x00 00000001 00A0 02 0x0000 0000 0000 0000 0x01 Invalid Migration Path was Specified Src Blade Check the setting of a migration path for the source HVM 0x00 00000001 0...

Page 301: ...HVM firmware to the version which is supporting migration of Processor Group 0x00 0000100E 0080 02 0x0000 0000 0000 0000 0x01 CPUIDs are different Check the server blade model and retry 0x00 0000100F 0080 02 0x0000 0000 0000 0000 0x01 CPUIDs AES NI PCLMULQDQ are different Check server blade model of source and destination server blades and retry 0x00 00001010 0080 02 0x0000 0000 0000 0000 0x01 CPU...

Page 302: ...pter at the destination reconfigure it to meet that of the source and then retry migration 0x00 3 0030 02 0x 4 byte_VnicID 0002 2 byte_VnicNum 0x01 The MMS could not define a VNIC 3 Virtual NIC 2a on an LPAR because the specified VNIC has VLAN IDs which are not acceptable in the destination blade Unable to define an NIC at the destination to have the same VLAN configuration settings as the source ...

Page 303: ...R has an unexpected condition Check the setting of Virtual COM consoles on source and destination HVMs and retry 0x00 60000004 0030 02 0x0000 0000 0000 0001 0x01 The MMS could not define an LPAR because the Virtual Console on the specified blade not empty Check the setting of VC number on the destination HVM and retry 0x00 70000001 0030 02 0x0000 0000 0000 0001 0x01 The MMS could not define an LPA...

Page 304: ...t the destination due to memory fragmentation Resolve the fragmentation in the destination HVM and then retry migration 0x00 80000004 0030 02 0x0000 0000 0020 000b 0x01 The MMS could not activate LPAR The number of activated LPAR reaches the maximum Deactivate one or more LPARs on the destination HVM and retry 0x00 0040 02 0x0000 0000 0000 0000 0x01 Connection refused MMS Blade Migration terminate...

Page 305: ...ment path as the migration path to be used 0x00 F003 03 0x101 The specified LPAR is busy please wait until another migration is completed Wait until the ongoing migration is complete and then retry migration 0x00 F009 03 0x01 The MMS could not accept a migration with activated LPAR Please shutdown the specified LPAR and try again Deactivate the specified LPAR and retry migration 0x00 F00a 03 0x01 ...

Page 306: ...SB device and retry 0x00 00000000 0030 05 to 0x00 00000000 0070 05 0x20 Terminated by user None 0x00 0030 04 to 0x00 0070 04 0x02 Connection refused LPAR Mover LPAR Mover Migration terminated by a network failure Check for recovery of connection between the source HVM and the destination HVM and then retry migration 0x00 0030 05 to 0x00 0070 05 0x02 Connection refused LPAR Mover LPAR Mover Migrati...

Page 307: ...ery operation terminated by a network failure Check for recovery of connection between the management server and the specified server blade and then retry recovery operation 0x00 00000001 0011 10 0x00 00000001 0031 10 0x01 The specified LPAR is busy please wait until another migration is completed Wait until the ongoing migration is complete and then retry recovery operation 0x00 00000016 0011 00 ...

Page 308: ...igible op1 1st operand that identifies error detail in hexadecimal code See Description op2 2nd operand that identifies supplementary information in hexadecimal code See Description op3 3rd operand that identifies reason in hexadecimal code See Description Message Shows error detail in text See Description Table 6 6 Error message detected by the source HVM op1 op2 op3 Message interpretation Sugges...

Page 309: ...ng 6 43 HVM Navigator User s Guide Migration Collecting failure information When you cannot resolve a failure which has occurred on HVM Navigator see Hitachi Compute Blade HVM Navigator User s Guide Getting Started ...

Page 310: ...6 44 Troubleshooting HVM Navigator User s Guide Migration ...

Page 311: ...7 List of icons 7 1 HVM Navigator User s Guide Migration List of icons This chapter describes the icons which are displayed in the windows ...

Page 312: ...r Migration functions A failing LPAR blocked from use An LPAR that has failed in the last status update An LPAR being migrated This icon is displayed on the applicable LPAR after the completion of status update process An unrecoverable LPAR An LPAR that WWNs are being rolled back An LPAR that accesses to LUs with a migration WWPN An LPAR that the state of assigned WWNs is Unknown Icon Showing An H...

Page 313: ...on Showing The WWPN is registered to any of the storage devices registered to HVM Navigator The WWPN isn t registered to any of the storage devices registered to HVM Navigator The icons showing if a migration WWPN is registered are shown in Table 7 5 Table 7 5 Icon showing if migration WWPN is registered Icon Showing The migration WWPN is registered to any of the storage devices registered to HVM ...

Page 314: ...7 4 List of icons HVM Navigator User s Guide Migration ...

Page 315: ...HVM Navigator User s Guide Migration ...

Page 316: ... 2845 Lafayette Street Santa Clara California 95050 2639 U S A www hds com Regional Contact Information Americas 1 408 970 1000 info hds com Europe Middle East and Africa 44 0 1753 618000 info emea hds com Asia Pacific 852 3189 7900 hds marketing apac hds com ...

Reviews:

No comments

Related manuals for NVM Navigator v03-07

6219 - IntelliStation M - Pro

Brand: IBM Pages: 317

Brand: LaCie Pages: 4

Brand: Avaya Pages: 19

TJA670 domovea Basic

Brand: hager Pages: 36

Brand: 3onedata Pages: 3

Brand: LOOQS Pages: 69

Brand: 7th Sense Pages: 156

Brand: PROXMOX Pages: 2

Xtreme PC NG6452

Brand: Chip PC Pages: 4

Proline Promass 300 EtherNet/IP

Brand: TechnipFMC Pages: 36

NAS TS-877XU-RP-3600-8G-US

Brand: QNAP Pages: 12

Brand: QNAP Pages: 15

Brand: Inspur Pages: 196

TimeProvider 5000

Brand: Symmetricom Pages: 534

Brand: AirLive Pages: 113

Brand: Uniwide Pages: 96

Brand: Plustek Pages: 86

Brand: GRASS VALLEY Pages: 106

Brands by name

0 1 2 3 4 5 6 7 8 9 A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Popular brands

Load more brands