manualshive.com logo in svg

Hitachi Virtual Storage Platform, User And Reference Manual

Share
Download
Hitachi Virtual Storage Platform User And Reference Manual Download Page 135

C

A

K

D E F G H I J

L M N O P Q R S T U V W X Y

#

B

Z

A B C D

F

G

J

K L

M

P

R S T

V

Glossary–1

Hitachi VSP User and Reference Guide

Glossary

This glossary defines the special terms used in this document. 

A

ATA

Advanced Technology Attachment. This is a disk drive implementation 

that integrates the controller on the disk drive.

array

See disk array.

B

back-end director (BED)

The hardware component that controls the transfer of data between the 

drives and cache. A BED feature consists of a pair of boards. A BED is 

also referred to as a disk adapter (DKA).

BED

See 

back-end director

.

BS

basic (power) supply

C

CCI

Command Control Interface

CHA

See channel adapter.

Summary of Contents for Virtual Storage Platform

Page 1: ...Product Version Document Organization Getting Help FASTFIND LINKS Contents Hitachi Virtual Storage Platform User and Reference Guide MK 90RD7042 11 ...

Page 2: ...ument may not be currently available Refer to the most recent product announcement or contact your local Hitachi Data Systems sales office for information about feature and product availability Notice Hitachi Data Systems products and services can be ordered only under the terms and conditions of Hitachi Data Systems applicable agreements The use of Hitachi Data Systems products is governed by the...

Page 3: ... Accessing product documentation xi Getting help xi Comments xi 1 Introduction 1 1 Hitachi Virtual Storage Platform overview 1 2 Hardware overview 1 2 Controller chassis 1 4 Drive Chassis 1 5 Flash chassis 1 5 Features 1 6 Scalability 1 6 High performance 1 7 High capacity 1 8 Connectivity 1 8 Virtual Storage Platform 1 8 Storage Navigator 1 8 High reliability 1 8 Non disruptive service and upgrad...

Page 4: ... host modes and host mode options 2 8 System option modes 2 8 Host Modes and host mode options 2 55 Open systems operations 2 55 Open systems compatibility and functionality 2 55 Open systems host platform support 2 55 Open systems configuration 2 56 Storage Navigator 2 56 3 System Components 3 1 Controller chassis 3 2 System control panel 3 5 Drive chassis 3 5 Flash components 3 8 Flash module 3 ...

Page 5: ...pecifications A 1 Mechanical specifications A 2 Electrical specifications A 2 Environmental specifications A 6 Equipment noise A 7 Heat output and air flow A 7 PDU specifications A 8 B Comparing Universal Storage Platform V VM and Virtual Storage Platform B 1 Comparison of USP V VM and VSP B 2 C Regulatory Compliance C 1 Regulatory compliance C 2 US FCC Notice C 4 Electronic emissions testing C 4 ...

Page 6: ...Hitachi VSP User and Reference Guide vi Contents ...

Page 7: ...y to understand how to use this product and keep a copy for reference This preface includes the following information Safety and environmental information Intended audience Product version Document revision level Changes in this revision Referenced documents Document organization Document conventions Convention for storage capacity values Accessing product documentation Getting help Comments ...

Page 8: ...tual Storage Platform firmware version 70 06 0x and later Document revision level Caution Before operating or working on the Virtual Storage Platform read the safety and environmental information in Regulatory Compliance on page C 1 Revision Date Description MK 90RD7042 00 August 2010 Initial release MK 90RD7042 01 November 2010 Supersedes and replaces MK 90RD7042 00 MK 90RD7042 02 December 2010 S...

Page 9: ...apter 2 Functional and Operational Characteristics This chapter provides a brief description of the Virtual Storage Platform architecture Chapter 3 System Components This chapter provides a more detailed description of the hardware than chapter 1 Chapter 4 Power On Off Procedures This chapter describes the procedures to power the Virtual Storage Platform storage system on and off and also provides...

Page 10: ...ed to indicate variables square brackets Indicates optional values Example a b indicates that you can choose a b or nothing braces Indicates required or expected values Example a b indicates that you must choose either a or b vertical bar Indicates that you have a choice between two or more options or arguments Examples a b indicates that you can choose a b or nothing a b indicates that you must c...

Page 11: ... Systems customer support staff is available 24 hours a day seven days a week If you need technical support log on to the Hitachi Data Systems support portal for contact information https Portal HDS com Comments Please send us your comments on this document doc comments hds com Include the document title number and revision Please refer to specific sections and paragraphs whenever possible Thank y...

Page 12: ...Hitachi VSP User and Reference Guide xii Preface ...

Page 13: ...d software used in the Hitachi Virtual Storage Platform storage system Detailed information is located in Chapter 2 Functional and Operational Characteristics on page 2 1 and Chapter 3 System Components on page 3 1 Hitachi Virtual Storage Platform overview Hardware overview Features Specifications Software features and functions ...

Page 14: ...ability serviceability and access to disk drives and other components when maintenance is needed Each component contains a set of LEDs that indicate the operational status of the component The system includes new and upgraded software features including Dynamic Tiering and a significantly improved task oriented version of Storage Navigator that is designed for ease of use and includes context sens...

Page 15: ...ontain one or two drive chassis in the same rack Each drive chassis can contain one of three drive types SFF HDDs or SSDs LFF HDDs or SSDs or Flash Module drives The controller chassis contains the control logic processors memory and interfaces to the drive chassis and the host servers A drive chassis consist of disk or SSD drives power supplies and the interface circuitry connecting it to the con...

Page 16: ...es eight front end directors four back end directors and ten dual fan assemblies It is mounted at the bottom of the rack because it is the heavier of the two units If a system has two SVPs both SVPs are mounted in controller chassis 0 The following illustration shows the locations of the components in the controller chassis The controller chassis is described in more detail in System Components on...

Page 17: ... inch HDD disk drives or SSD flash drives The second type can contain up to 128 2 1 2 inch HDD disk drives or SSD flash drives The maximum number of 3 1 2 inch drives in a Virtual Storage Platform is 1280 The maximum number of 2 1 2 drives is 2048 The maximum number of Flash Module drives is 96 Details are located in System Components on page 3 1 Flash chassis In addition to the chassis for disk a...

Page 18: ... be configured in several ways as needed to meet customer requirements The minimum configuration is a single rack containing one controller chassis in a diskless configuration A small VSP system could include a single rack containing one controller chassis and one or two disk drive or flash chassis A mid sized VSP system could include one to three racks containing one controller chassis and up to ...

Page 19: ... FEDs and BEDs and varying cache capacities as follows Two to six FEDs each is a pair of boards This provides a total of 12 when all of the FED slots are used and there are no BEDs installed as in a diskless system The maximum total number of FEDs and BEDs is 12 Two to four BEDs each is a pair of boards This provides a total of 8 when all of the BED slots are used In this case only two FEDs can be...

Page 20: ...imum physical disk capacity of approximately 2 521 PB per full storage system Connectivity Virtual Storage Platform The Virtual Storage Platform storage system supports most major IBM Mainframe operating systems and Open System operating systems such as Microsoft Windows Oracle Solaris IBM AIX Linux HP UX and VMware For more complete information on the supported operating systems go to http www hd...

Page 21: ...s turned off and moved out of the way A Service Processor mounted on the controller chassis monitors the running condition of the storage system Connecting the SVP with a service center enables remote maintenance The firmware microcode can be upgraded without disrupting the operation of the storage system The firmware is stored in shared memory part of the cache memory module and transferred in a ...

Page 22: ...6 64 2WL 32 Number of ports per installation unit FC 2 4 8 GB 80 96 1 16 8 176 192 1 16 8 FICON 2 4 8 GB 80 16 176 16 Device I F Controller chassis drive chassis Interface SAS Dual Port Data transfer rate Max 6 GBps Maximum number of HDD per SAS I F 256 2 5 inch HDD 160 3 5 inch HDD Maximum number of FEDs 4 if drives installed 6 if diskless 8 if drives installed 12 if diskless Channel I F Mainfram...

Page 23: ...oxes in the flash chassis Spares are additional Maximum Number of Drives Drive Type inches Drive Chassis Single Module 3 rack system Dual Module 6 rack system HDD 3 1 2 80 640 1280 HDD 2 1 2 128 1024 2048 SSD 3 1 2 802 1283 2563 SSD 2 1 2 1282 1283 2563 Flash module 48 964 192 Notes 1 Each drive size requires its own chassis Drive sizes cannot be mixed in a single chassis 2 SSD drives can be mount...

Page 24: ...hassis pair max capacity 1 6 GB Raw 147 2 140 8 147 2 140 8 140 8 128 0 Usable 73 6 70 4 110 4 123 2 105 6 112 0 3 2 GB Raw 254 4 281 6 254 4 281 6 281 6 256 0 Usable 147 2 140 8 220 8 246 4 211 2 224 0 Total P9500 max capacity 1 6 GB Raw 294 4 281 6 294 4 281 6 281 6 256 0 Usable 147 2 140 8 220 8 246 4 211 2 224 0 3 2 GB Raw 588 8 563 2 588 8 563 2 563 2 512 0 Usable 294 4 281 6 441 6 492 8 422 ...

Page 25: ...orkload Manager WLM host software function and supports both static and dynamic PAV functionality Flash Acceleration Flash acceleration increases the performance of SSD and HDD drives by altering the IO and executing directly to the disks Typically the IO execution involves interaction between BED back end director and VSP s cores Therefore it releases the busy processing cycles and allows them to...

Page 26: ...mode for open systems TrueCopy for Mainframe provides synchronous copy for mainframe systems Hitachi ShadowImage In System Replication software and Hitachi ShadowImage In System Replication software for Mainframe Creates internal copies of volumes for purposes such as application testing and offline backup Can be used in conjunction with True Copy or Universal Replicator to maintain multiple copie...

Page 27: ... Universal Storage Platform V VM Storage Logical Partition SLPR It allows for additional granularity and flexibility of the management of storage resources Feature Description Audit Log Function The Audit Log function monitors all operations performed using Storage Navigator and the SVP generates a syslog and outputs the syslog to the Storage Navigator computer Hitachi SNMP Agent Provides support ...

Page 28: ...Hitachi VSP User and Reference Guide 1 16 Introduction ...

Page 29: ...unctional and operational characteristics of the Hitachi Virtual Storage Platform storage system including the system architecture System architecture overview Hardware architecture RAID implementation overview CU images LVIs and logical units Mainframe operations System option modes host modes and host mode options Open systems operations Storage Navigator ...

Page 30: ...emblies A controller chassis that contains the logic and processing components A drive chassis that contains the disk drives and solid state drives This chassis is not included in diskless systems These assemblies are explained briefly in Chapter 1 Introduction on page 1 1 and in detail in Chapter 3 System Components on page 3 1 RAID implementation overview This section provides an overview of the...

Page 31: ...le RAID 10 2D 2D Layout on page 2 3 illustrates a sample RAID 10 2D 2D layout A RAID 10 2D 2D array group consists of two pairs of data drives in a mirrored configuration regardless of data drive capacity A RAID 10 4D 4D group combines two RAID 10 2D 2D groups Data is striped to two drives and mirrored to the other two drives The stripe consists of two data chunks The primary and secondary stripes...

Page 32: ...gical devices LDEVs are evenly dispersed in the array group so that the performance of each LDEV within the array group is the same This figure also shows the parity chunks that are the Exclusive OR EOR of the data chunks The parity and data chunks rotate after each stripe The total data in each stripe is either 24 logical tracks eight tracks per chunk for mainframe data or 2304 blocks 768 blocks ...

Page 33: ...onventional concatenation of RAID 10 array groups 4D 4D the Virtual Storage Platform supports LDEV striping across multiple RAID 5 array groups for improved logical unit performance in open system environments The advantages of LDEV striping are Improved performance especially of an individual logical unit due to an increase in the number of data drives that constitute an array group Better worklo...

Page 34: ... in the primary logical disk controller LDKC The Virtual Storage Platform supports the following control unit CU emulation types 3990 6 3990 6E 2105 2107 The mainframe data management features of the Virtual Storage Platform may have restrictions on CU image compatibility For further information on CU image support refer to the Mainframe Host Attachment and Operations Guide or contact your Hitachi...

Page 35: ...N Expansion LUSE features Using VLL users can configure multiple logical units under a single LDEV Using Virtual LVI or LUSE users can concatenate multiple logical units into large volumes For further information on VLL and Virtual LVI see the Performance Guide and the Provisioning Guide for Open Systems Mainframe operations This section provides high level descriptions of mainframe compatibility ...

Page 36: ...e contact Hitachi Data Systems support The Hitachi Storage Navigator User Guide provides instructions for installing configuring and using Storage Navigator to perform resource and data management operations on the Virtual Storage Platform storage systems The Provisioning Guide for Mainframe Systems and Hitachi Volume Shredder User Guide provides instructions for converting single volumes LVIs int...

Page 37: ...ates whether the mode applies to the main control unit MCU and or the remote control unit RCU Table 2 1 System option modes Mode Category Description Default MCU RCU 20 Public Optional R VOL read only function OFF MCU 22 Common Regarding the correction copy or the drive copy in case ECCs LRC PINs are set on the track of copy source HDD mode 22 can be used to interrupt the copy processing default o...

Page 38: ...e TrueCopy Sync M VOL Mainframe Volume Pair status Duplex Pending Mode 64 OFF default When receiving the Freeze command pairs that fulfill the conditions below are suspended and the SCP is set In the case of CU emulation type 2105 2017 the path between MCU and RCU is deleted while the path is not deleted but unusable with Query displayed only in the case of CU emulation type 3990 When receiving th...

Page 39: ...ompletion of Freeze command and RUN command gets longer estimate of time to report completion 1 second per 1000 pairs and MIH may occur MCU RCU 80 ShadowImage for Mainframe For RAID 300 400 450 SI for OPEN or Mainframe In response to the Restore instruction from the host or Storage Navigator the following operation is performed regardless of specifying Quick or Normal For RAID 500 600 700 SI for O...

Page 40: ...OFF default By specifying Split or Resync Steady Quick Split or Normal Quick Resync is respectively executed in accordance with Normal Quick setting For details see SOM 122 sheet Note 1 For RAID500 and later models this mode is applied to use scripts etc that are used on RAID400 and 450 2 In the case of RAID500 and later models executing the pairresync command from RAID Manager may be related to t...

Page 41: ...when performing CVS operations on Storage Navigator In addition the LDEV specifying format on the Maintenance window of the SVP is in low speed as well Notes 1 For more details about mode 269 see worksheet Mode269 detail for RAID700 2 Mode 269 is effective only when using the SVP to format the CVS OFF MCU RCU 278 Open Tru64 Host Mode 07 and OpenVMS Host Mode 05 Caution Host offline Required OFF 29...

Page 42: ...C 2180 option Description SIM RC 2180 RIO path failure between MCU and RCU was not reported to host DKC reports SSB with F M F5 instead of reporting SIM RC 2180 in the case Micro program has been modified to report SIM RC 2180 with newly assigned system option Mode as individual function for specific customer Usage Mode 308 ON SIM RC 2180 is reported which is compatible with older Hitachi specific...

Page 43: ...he entire destage processing is periodically determined by using the highest workload of all CLPRs a The larger the workload is the larger the amount of the entire destage processing becomes a Write Pending capacity of CLPR x Cache capacity of CLPR x x 0 to 31 CLPR whose value above is the highest of all CLPRs Because the destage processing would be accelerated depending on CLPR with high workload...

Page 44: ...Storage Navigator GUI Notes 1 If the LDEV is not written with data 0 before performing the function the LDEV format may fail 2 After the format processing make sure to set system option mode 457 to OFF OFF MCU RCU 459 ShadowImage for Mainframe ShadowImage When the secondary volume of an SI SIz pair is an external volume the transaction to change the status from SP PEND to SPLIT is as follows 1 Mod...

Page 45: ...OFF 467 ShadowImage ShadowImage for Mainframe Co mpatible FlashCopy V2 Snapshot Volume Migration Universal Volume Manager For the following features the current copy processing slows down when the percentage of dirty data is 60 or higher and it stops when the percentage is 75 or higher Mode 467 is provided to prevent the percentage from exceeding 60 so that the host performance is not affected Sha...

Page 46: ...SIMs must be reported to them SIMs reported by setting the mode to ON are SIM RC 601xxx Pool utilization threshold excess Earlier than 70 05 0x 00 00 SIM RC 601xxx Pool utilization threshold excess 603000 SM Space Warning 70 05 0x 00 00 or higher Mode 471 ON This kind of SIMs is reported to maintenance personnel Mode 471 OFF default This kind of SIMs is not reported to maintenance personnel Note S...

Page 47: ...icated script Initial copy operation is prioritized over update I O Therefore the processing speed of the update I O slows down by about 15 s per command 3 If this mode is set to ON the processing speed of update I O slows down by about 15 s per command version downgrade is disabled and Take Over is not available 4 If the mode is not set to ON for both or either sides the behavior is as follows Wi...

Page 48: ... 484 ON PPRC path QUERY is displayed with the New Spec Mode 484 OFF default PPRC path QUERY is displayed with the Previous Spec ESCON interface Note 1 Set this mode to ON when you want to maintain compatibility with the Previous Spec for PPRC path QUERY display under the environment where IBM host function such as PPRC and GDPS is used 2 When an old model or a RAID500 that doesn t support this mod...

Page 49: ...ure not to apply the mode when the host I O performance is considered to be important 3 The mode will be noneffective if 3 or more pairs of DKAs are not mounted 4 Make sure to set mode 467 to OFF when using mode 491 since the performance may not improve 5 The mode is noneffective for the NSC model OFF 495 NAS Function That the secondary volume where S VOL Disable is set means the NAS file system i...

Page 50: ...niversal Replicator for Mainframe When a Universal Replicator for Mainframe pair is in the Duplex state this option switches the display of Consistency Time C T between the values at JNL restore completion and at JNL copy completion Mode 530 ON C T displays the value of when JNL copy is completed Mode 530 OFF default C T displays the value of when JNL restore is completed Note At the time of Purge...

Page 51: ...f this option is set to ON As described in the BCM user s guide the script must be performed with Command Device offline 2 This option is applied to operations from BCM that is operated on MVS 556 Open Prevents an error code from being set in the 8 11th bytes in the standard 16 byte sense byte Mode 556 ON An error code is not set in the 8 11th bytes in the standard 16 byte sense byte Mode 556 OFF ...

Page 52: ... check for any influence on GDPS PPRC 5 The SOM must be enabled only when the CASCADE option is specified for the ESTPAIR command for the DKC emulation type 2105 2107 OFF MCU RCU The unit where TCz and SIz in a cascading configuration use the same volume 589 Universal Volume Manager Turning this option ON changes the frequency of progress updates when disconnecting an external volume of disconnect...

Page 53: ...to set whether an audit log is to be stored onto the system disk or not Mode 676 ON An audit log is stored onto the system disk Mode 676 OFF default An audit log is not stored onto the system disk This mode is also enabled disabled by enabling disabling Audit Log Buffer on the Audit Log Setting window which can be opened by selecting Settings Security Audit Log Setting on Storage Navigator Notes 1...

Page 54: ...ration is slowed down Mode 689 OFF default The initial copy and resync copy operations are not slowed down when the Write Pending rate on RCU exceeds 60 the same as before Note 1 1 This mode can be set online 2 2 The micro programs on both MCU and RCU must support this mode 3 3 This mode should be set per customer s requests 4 4 If the Write Pending status long keeps 60 or more on RCU it takes ext...

Page 55: ... can be set online 2 This mode should be set per customer s requests 3 If the Write Pending status long keeps 60 or more on RCU it takes extra time for the initial copy to be completed by making up for the prevented copy operation 4 If the Write Pending status long keeps 60 or more on RCU the pair status may become Suspend due to the JNL Vol being full OFF 696 Open This mode is available to enable...

Page 56: ... 3 With this system option OFF if searching for external devices with VMA set the VMA information cannot be read 4 When the mode is set to ON while the following conditions are met the external volume is blocked a RAID700 70 03 3x 00 00 or higher version is used on the storage system b An external volume to which Nondisruptive Migration NDM attribute is set exists c The external volume is reserved...

Page 57: ... Default Copy processing requested is not registered into a newly created queue Only the existing queue is used Note If the PDEV is highly loaded the priority of Read Write processing made by SI VM Snapshot Compatible FlashCopy V2 or Resync may become lower As a consequence the copy speed may be slower OFF 720 Universal Volume Manager Mainframe and Open Supports the Active Path Load Balancing APLB...

Page 58: ...re unavailable to select Mode 721 OFF default When a parity group is uninstalled or installed the operation is as before the LED of the drive is illuminated and the drive must be unmounted and remounted Notes 1 When the RAID level or emulation type is changed for the existing parity group this option should be applied only if the drive mounted position remains the same at the time of the parity gr...

Page 59: ...ing this SOM is recommended when the above condition 1 is not met and SUN storage is used as an external storage Applying this SOM is recommended if the above condition 1 is not met and a maintenance operation such as firmware update causing controller reboot is executed on the external storage side while a storage system other than Hitachi product is used as an external subsystem 2 For R700 70 02...

Page 60: ...Protect attribute for the target DP VOL using DRU when any write operation is requested to the area where the page allocation is not provided at a time when the HDP pool is full Not to set in the case of Read request Mode 729 OFF default Not to set the Protect attribute for the target DP VOL using DRU when any write operation is requested to the area where the page allocation is not provided at a ...

Page 61: ... applied when Volume Migration or Quick Restore operation can be suspended during LDEV related maintenance 2 Set mode 733 to ON if you want to perform any LDEV related maintenance activities and you do not want these operations to fail when Volume Migration or Quick Restore is active 3 This option is recommended as functional improvement to avoid maintenance failures In SOM e cases of a failure in...

Page 62: ... occurs frequently other SIMs may not be reported 3 Though turning on this option can increase the warning effect if measures such as adding a pool fail to be done in time so that the pool becomes full MODE 729 can be used to prevent file systems from being destroyed 4 Turning on MODE 741 can provide the SIM report to not only the users but also the service personnel OFF 741 Microcode version V02 ...

Page 63: ...IX is connected using UVM 2 Enable the setting of EMC SCSI Flag SC3 for the port of the SYMMETRIX connected with Virtual Storage Platform If the setting of EMC SCSI Flag SC3 is not enabled the effect of this mode may not be achieved 3 If you want to enable this mode immediately after setting perform Check Paths on each path one by one for all the external ports connected to the SYMMETRIX Without d...

Page 64: ...vated Notes 1 This option is applied when no change in performance characteristic is desired 2 All HDP pools are subject to the HDP Rebalance function 3 When a pool is newly installed the load may be concentrated on the installed pool volumes 4 When 0 data discarding is executed load may be unbalanced among pool volumes OFF 757 Open and Mainframe Enables disables output of in band audit logs Mode ...

Page 65: ...pension with an EXCTG specified is executed 4 BCM is installed on the host where the time stamping function is available In the case of multiple host configuration SYSPLEX timer is available on the system 2 If this mode is set to ON while the BCM does not exist in the environment where the time stamping function is available In the case of multiple host configuration SYSPLEX timer is available on ...

Page 66: ...uted Mode 769 OFF default The retry operation is enabled when the path creation operation is executed retry operation is executed Notes 1 This mode is applied when the three conditions below are met SOM 114 is set to OFF operation of automatically switching the port is disabled HMO 49 and HMO 50 are set to OFF 70 02 31 00 00 and higher TPC R is used it is not applied in normal operation 2 When SOM...

Page 67: ...VOL pair suspend or deletion operation from BCM the F M FB message is output to the host Notes 1 Set this mode to ON in the environment where TC TCA for z OS is used from BCM and the MCU host does not need the F M FB message output during an S VOL pair suspend or deletion operation from BCM 2 If this mode is set to ON the F M FB message is not output to the host when the status of P VOL changes to...

Page 68: ...is set to 15 seconds or less 3 The mode is applied to reduce RI O MIH time to 5 seconds 4 The mode is effective for the entire system Notes 1 This function is available for all the TC Sync for z OS and TC Sync pairs on the subsystem unable to specify the pairs that are using this function or not 2 RAID700 To apply the mode to TC Sync both MCU and RCU must be RAID700 and micro program must be the s...

Page 69: ...ccur due to a path failure between DKCs 9 Operation commands are not available for promptly switching to an alternate path 10 The mode works for the pair for which initial pair creation or Resync operation is executed 11 Micro program downgrade to an unsupported version cannot be executed unless all the TC Sync for z OS or TC Sync pairs are suspended or deleted 12 See the appendix of the SOM for o...

Page 70: ... 803 OFF default While a HDP pool VOL is blocked if a read or write I O is issued to the blocked pool VOL the DRU attribute is not set to Protect Notes 1 1 This mode is applied when a file system using HDP pool VOLs is used Data Retention Utility is installed 2 2 Because the DRU attribute is set to Protect for the V VOL a read I O is also disabled 3 3 If Data Retention Utility is not installed the...

Page 71: ...SOM 467 set to ON Write pending rate of an MP blade that has LDEV ownership of the copy target is high Usage rate of a parity group to which the copy target LDEV belongs is low ShadowImage copy progress is delayed 2 This mode is available only when SOM 467 is set to ON 3 If the workload of the copy target parity group is high the copy processing may not be improved even if this mode is set to ON 8...

Page 72: ... DP VOL format the change is not reflected to the format of the DP VOL being executed but the format continues in the same method OFF 872 Universal Volume Manager When the mode is applied the order of data transfer slots is guaranteed at the destaging from P9500 to an external storage Mode 872 ON The order of data transfer slots from VSP to an external storage is guaranteed Mode 872 OFF default Th...

Page 73: ...rovement of I O response performance in low workload is required 2 Because the processing on the Mainframe target port is prioritized other processing may take longer time compared to that when the mode is set to OFF OFF 895 TrueCopy for Mainframe Setting the mode to ON or OFF the link type with transfer speed of 8 GBps or 4 GBps is reported respectively Mode 895 ON When the FICON FC link up speed...

Page 74: ...und format for a DP HDT pool in the environment where new page allocation for example when system files are created from a host for newly created multiple HDP VOLs frequently occurs and the write performance degrades because of an increase in write pending rate 2 When the mode is set to ON because up to 42MB s of ECCG performance is used local copy performance may degrade by about 10 Therefore con...

Page 75: ...iers becomes narrow and the frequency of page allocation increases Notes 1 Apply the mode when the usage of upper tier is low and that of lower tier is high 2 The mode must be used with SOM 898 3 Narrowing the gray zone increases the number of pages to migrate between tiers per relocation 4 When Tier1 is SSD while SOM 901 is set to ON the effect of SOM 897 and 898 to the gray zone of Tire1 and Tie...

Page 76: ...ead In addition the settings of SOM 897 and 898 are effective for Tire2 and Tier3 Please also see spreadsheet SOM 897_898_901 for more details about the relations between SOM 897 898 and 901 OFF 899 Volume Migration In combination with the SOM 900 setting whether to execute and when to start the I O synchronous copy change as follows Mode 899 ON SOM 900 is ON I O synchronous copy starts without re...

Page 77: ... the mode to ON or OFF the page allocation method of Tier Level ALL when the drive type of tier1 is SSD changes as follows Mode 901 ON For tier1 drive type is SSD pages are allocated until the capacity reaches the limit Without consideration of performance limitation exceedance allocation is done from highly loaded pages until reaching the capacity limit When the capacity of the tier1 reaches the ...

Page 78: ...ng multi platforms configuration the requirement for response time is severe 2 The number of pages to be migrated per unit time at tier relocation decreases OFF 908 Universal Replicator Universal Replicator for Mainframe The mode can change CM capacity allocated to MPBs with different workloads Mode 908 ON The difference in CM allocation capacity among MPBs with different workload is large Mode 90...

Page 79: ... restrictions are applied to the HDP pool 1 When execution mode is Auto monitoring the target HDP pool is disabled 2 When execution mode is Manual a request to start monitoring the target HDP pool is not accepted 3 Monitoring information weighted average information of the target HDP pool is discarded Mode 912 OFF default HDT monitoring information of a HDP pool containing a HDP VOL to which the p...

Page 80: ... is applied when multiple LDEVs are created in a parity group or external volume group 2 If the mode setting is changed during pool shrink the shrink processing may fail 3 When the mode is set to OFF the processing to average page usage rate among pool volumes in a parity group or external volume group works therefore the drive workload becomes high because the migration source and target are in t...

Page 81: ... started Mode 930 OFF default The zero data page reclamation is performed See sheet SOM 930 for relationship with SOM 755 and SOM 859 Notes 1 The mode is applied when stopping or disabling zero data page reclamation by user request is required 2 When the mode is set to ON the zero data page reclamation does not work at all Zero data page reclamation by Write Same and UNMAP and IO synchronous page ...

Page 82: ...formed without shared memory for HDT installed an error is reported and the monitoring data collection fails 5 Before removing the shared memory for HDT set the mode to OFF and wait for 30 minutes 6 Tier relocation with monitoring data collected when the mode is set to ON is disabled 7 When HDP is converted into HDT after purchase of PP license the collected monitoring data is discarded Operation ...

Page 83: ...tions for the open systems environment including Multi initiator I O configurations in which multiple host systems are attached to the same fibre channel interface Fibre channel arbitrated loop FC AL and fabric topologies Command tag queuing Industry standard failover and logical volume management software SNMP remote storage system management The Virtual Storage Platform s global cache enables an...

Page 84: ... Guide for Open Systems describes and provides instructions for configuring the Virtual Storage Platform for host operations including FC port configuration LUN mapping host groups host modes and host mode options and LUN Security Each fibre channel port on the Virtual Storage Platform provides addressing capabilities for up to 2 048 LUNs across as many as 255 host groups each with its own LUN 0 h...

Page 85: ...ed on any machine that supports a Java Virtual Machine JVM A PC hosting the Storage Navigator software is called a remote console Each time a remote console accesses and logs into the SVP of the desired storage system the Storage Navigator applet is downloaded from the SVP to the remote console Figure 2 6 Storage Navigator and SVP Configuration on page 2 57illustrates remote console and SVP config...

Page 86: ...Hitachi VSP User and Reference Guide 2 58 Functional and Operational Characteristics ...

Page 87: ...chi VSP User and Reference Guide System Components This chapter describes the hardware used in the Hitachi Virtual Storage Platform Controller chassis System control panel Drive chassis Flash components Cache memory Shared memory ...

Page 88: ...nternal processor and 128 bytes of edit buffer memory back end director BED 0 if diskless 2 with drives 4 A BED is an interface board that provides connection to the disk drives and SSDs Provides the control functions for data transfer between drives and cache The BED contains DRR Data Recover and Reconstruct a parity generator circuit It supports eight FIBRE path and offers 32 KB of buffer for ea...

Page 89: ...h the minimum number of components The system control panel 1 in the front view is described in the next section Figure 3 1 Controller chassis front view minimum configuration Item Description Item Description Control Panel Fan 10 total Cache optional Cache Memory Adapter Space for additional Virtual Storage Director Virtual Storage Director microprocessor board ...

Page 90: ...configuration Item Description Item Description Power Supply 2 min 4 max Power Supply optional Hub optional back end director optional BED FED optional Express Switch optional Express Switch Service Processor back end director Fan 10 total Sub SVP Provides ethernet connection to the SVP back end director BED ...

Page 91: ...ther of the clusters Applied to both storage clusters Blinking Indicates that a SVP failure has occurred ALARM Red LED Indicates DC under voltage of any DKC part DC over current abnormally high temperature or that an unrecoverable failure occurred READY Green LED Indicates that input output operation on the channel interface is enabled PS ON Green LED Indicates that the system is powered on that t...

Page 92: ...F DKU disk unit chassis which contains up to 80 LFF 3 1 2 inch HDD or SSD drives An SFF DKU disk unit chassis which contains up to 128 SFF 2 1 2 inch HDD or SSD drives The following illustration shows the rear view of a DKU drive chassis The front view is exactly like the rear view The table following the illustration describes the drive chassis components Figure 3 4 Drive Chassis Item Description...

Page 93: ...sets of fans work together to create a large airflow through the unit Either fan assembly is sufficient to cool the unit Therefore there is no time limit when changing disk drives as long as either the front or the rear fan assembly is in place WARNING To prevent the unit from overheating both the front and rear fan assemblies should never be opened at the same time while the system is running Fig...

Page 94: ... on the Hitachi Accelerated Flash storage components including the flash module flash box and flash chassis Flash module The flash module factory designation FMD is a Hitachi custom designed and manufactured enterprise class solid state storage module It uses a high performance custom ASIC flash controller and standard flash memory chips in an implementation that exceeds the performance of expensi...

Page 95: ...le has an error and should be replaced SAS ENC Module Power LED SAS ENC Module Alarm LED indicates fatal error condition SAS ENC standard IN connector SAS ENC high performance IN connector ENC adapter connects the flash modules to the BEDs in the controller via ENC cables SAS ENC standard OUT connector SAS ENC high performance OUT connector Power cord receptacle Power Supply 220 VAC input draws ap...

Page 96: ...ing the four flash boxes but since it takes the place of a drive chassis the cluster is referred to as a chassis for consistency The maximum number of flash modules in a flash chassis is 48 The minimum number of flash modules is four one in each flash box Flash modules can be added to a flash chassis in increments of four eight or 16 depending on the desired RAID configuration as explained in the ...

Page 97: ... chassis holds only 48 drives while the drive chassis holds either 80 LFF or 128 SFF drives Each flash box is divided into two logical units The LUs are numbered vertically in the FBX starting with the lower logical units and then the upper logical units The FMDs are numbered 0 through 5 in each logical unit The following figure shows the numbering sequence Figure 3 9 Flash configurations ...

Page 98: ...e FBX as a set Just as the RAID group number is determined according to the location of a disk drive the RAID group number is also determined by the location of a flash module in a flash chassis even though the number of FMDs is smaller than the number of drives in a drive chassis This is because the VSP is capable of variable numbers of drives in a drive chassis The following illustration shows t...

Page 99: ...e cache and shared memory for the system Each pair is called a cluster From one to four cache clusters can be installed in a controller chassis In addition to the memory on the cache boards 4 GB of cache memory is also located on each Virtual Storage Director microprocessor board See the following illustration Figure 3 11 Cache memory Item Description Item Description Virtual Storage Director VSD ...

Page 100: ...y The Virtual Storage Platform shared memory is not on a separate memory module as it was in the previous hardware systems Shared memory resides by default on the first pair of cache boards in controller chassis 0 When you install software features such as Copy on Write Snapshot or Universal Replicator the shared memory usage increases as software features are installed Shared memory can use up to...

Page 101: ...ribes the prerequisites and procedures to turn the Hitachi Virtual Storage Platform power on and off It also describes the procedures to follow in case of a power failure Safety and environmental information Standby mode Normal Power On Off procedures Emergency power off on procedure Battery backup operations ...

Page 102: ...ifications single phase current on page A 3 and Table A 3 VSP Electrical specifications three phase current on page A 4 for power consumption specifications To put the storage system into standby mode from the OFF condition 1 Ensure that power is available to the AC input boxes and PDUs in all racks in which the VSP storage system is installed 2 Turn all PDU power switches breakers ON To put the s...

Page 103: ... system to complete its power on self test and boot up processes Depending on the storage system configuration this may take several minutes 4 When the READY LED is ON the storage system boot up operations are complete and the storage system is ready for use Note The control panel includes a safety feature to prevent the storage system power from accidentally being turned on or off The PS power ON...

Page 104: ... system in case of emergency and how to bring the system back up after an emergency shutdown Emergency power off procedure The VSP storage system does not have an emergency power off switch as the USP V VM storage system does To shut the system down in an emergency power off the controllers first then the drive units as follows 1 Open the back doors of both racks that contain control units a Turn ...

Page 105: ...n write through mode This synchronously writes to HDDs to prevent slow data throughput in the cache When the battery charge is 50 or more the cache write protect mode operates normally When a power failure occurs and continues for 20 milliseconds or less the storage system continues normal operation If the power failure exceeds 20 milliseconds the storage system uses power from the batteries to ba...

Page 106: ...ies slowly discharge They will have a charge of less than 50 after two weeks without power When fully discharged the batteries must be connected to power for three hours to fully recharge Long Term Array Storage While connected the cache destage batteries will completely discharge in two to three weeks without power applied If you do not use a Virtual Storage Platform for two weeks or more ask tec...

Page 107: ... 1 Hitachi VSP User and Reference Guide Troubleshooting This chapter provides troubleshooting information for the Hitachi Virtual Storage Platform storage system Getting help Solving problems Service Information Messages ...

Page 108: ... for use by Hitachi Data Systems personnel logged in the SYS1 LOGREC dataset of the mainframe host system displayed by the Storage Navigator software and reported over SNMP to the open system host The SIM Error Condition Recommended Action Error message displayed Determine the type of error refer to the SIM codes section If possible remove the cause of the error If you cannot correct the error con...

Page 109: ...ot require immediate attention and are addressed during routine maintenance The serious and acute SIMs highest severity are reported to the mainframe hosts once every eight hours The following figure illustrates a typical 32 byte SIM from the VSP storage system SIMs are displayed by reference code RC and severity The six digit RC which is composed of bytes 22 23 and 13 identifies the possible erro...

Page 110: ...Hitachi VSP User and Reference Guide 5 4 Troubleshooting ...

Page 111: ...is appendix provides the mechanical electrical and environmental specifications for the Hitachi Virtual Storage Platform storage system Mechanical specifications Electrical specifications Environmental specifications Equipment noise Heat output and air flow PDU specifications ...

Page 112: ...s1 on page A 5 lists the PDU specifications for both single phase and three phase power Note The weight and electrical specifications in the following tables were taken from a VSP storage system in a controlled environment To calculate the weight and current draw of a specific system use the Power and Weight calculator at http www hds com go weight and power calculator Contact technical support if...

Page 113: ...2 5 SAS 5 7KVA 3 5 SATA Maximum 41 4 KVA Controller chassis power supply Input Current 7 18 A Steady Current 3 59 A Leakage Current 0 28 mA Inrush current 1st 0 p 20 A 2nd 0 p 15A 1st 0 p Time 25 80 ms Drive chassis power supply Input Current 6 52 A Steady Current 3 26A Leakage Current 0 28 mA Inrush current 1st 0 p 20 A 2nd 0 p 15A 1st 0 p Time 25 80 ms Power consumption system in standby Maximum...

Page 114: ...80 ms Configuration1 Idle2 Max Power Consumption3 5 Current per rack Current per rack R00 module 0 controller4 17 3 19 6 R10 module 1 controller4 18 2 17 9 R01 module 0 drive 1 4 14 8 18 3 R11 module 1 drive 1 4 16 1 19 6 R02 module 0 drive 2 4 17 2 17 1 R12 module 1 drive 2 4 14 4 18 6 Total Power 98 111 1 Notes 1 The numbers in this table were provided by measuring system power consumption in a ...

Page 115: ... B C gnd 208 V 30 A per phase 3 UL489 15 A 2 pole 2 pole 3 wire 220 VAC A B or B C gnd APAC EMEA IEC 309 red 4 pole 5 wire A B C Neut gnd 400 V 32 A per phase 6 16 A 2 pole 1 pole 3 wire 200 VAC A N or B N or C N gnd Required number of plugs in each PDU per chassis Controller chassis Drive chassis Same power cord and plug for all power supplies 4 4 Required number of PDUs per rack Controller rack ...

Page 116: ...ge 0 15 m9 Altitude 60 m to 3 000 m Notes 1 Environmental specification for operating condition should be satisfied before the storage system is powered on Maximum temperature of 32 C should be strictly satisfied at air inlet portion 2 Recommended temperature range is 21 to 24 C 3 Non operating condition includes both packing and unpacking conditions unless otherwise specified 4 On shipping storag...

Page 117: ...assis by themselves The racks do not contain fans Airflow is from front to back Table A 7 VSP Heat Power and airflow Model Number Heat Output kW Power Consumption kVA Air Flow cubic ft cubic meters min DKC710I CBX 0 4751 0 500 1 247 2 7 0 DKC F710I SBX 0 5701 0 6001 317 8 9 0 DKC F710I UBX 0 5701 0 6001 317 8 9 0 DKC F710I SVP 0 052 0 055 DKC F710I HUB 0 010 0 010 DKC F710I 16MFS 0 118 0 124 DKC F...

Page 118: ...M 0 010 0 011 DKC F710I 400S1M 0 010 0 011 DKC F710I 1R6FM3 0 017 18 watts DKC F710I 3R2FM 0 018 19 watts Notes 1 Maximum values when all fans rotate at maximum speed 2 Power is consumed during the battery back up time only 3 Flash module drive Estimated power consumption for a 12 drive FMU is 531 watts including the ENCs and power supplies Model Number Heat Output kW Power Consumption kVA Air Flo...

Page 119: ...HERWISE NOTED ANGULAR 1 XXXX 010 XXX 015 XX 030 X FOR REF ONLY UNLESS OTHERWISE SPECIFIED ALL DIMENSIONS ARE IN INCHES DIMENSIONAL LIMITS APPLY AFTER APPLICATION OF FINISH INTERPRET DRAWING PER ASME Y14 5M 1994 THIRD ANGLE PROJECTION DATE DATE 1 1 PS1465B PDU WITH L6 30P Great Lakes Case Cabinet Co Inc Part PS1465B 1 Shipping please ensure 1 units per box 2 Box Labels Each box to have one label th...

Page 120: ... 1 XXXX 010 XXX 015 XX 030 X FOR REF ONLY UNLESS OTHERWISE SPECIFIED ALL DIMENSIONS ARE IN INCHES DIMENSIONAL LIMITS APPLY AFTER APPLICATION OF FINISH INTERPRET DRAWING PER ASME Y14 5M 1994 THIRD ANGLE PROJECTION DATE DATE 1 1 PS1474B PDU WITH L15 30P Great Lakes Case Cabinets Inc Part PS1474B 1 Shipping please ensure 1 units per box 2 Box Labels Each box to have one label that displays the Serial...

Page 121: ... User and Reference Guide Comparing Universal Storage Platform V VM and Virtual Storage Platform This appendix provides a comparison of the main features of Universal Storage Platform V VM storage system and the Hitachi Virtual Storage Platform storage system Comparison of USP V VM and VSP ...

Page 122: ...ocessing across the FEDs and BEDs as needed Faster access to system control information through the use of onboard memory on the virtual storage directors microprocessors Storage management usability improvements The new version includes a user friendly task based GUI that reduces the number of operations needed to complete a task and includes wizards to assist users in new or repetitive tasks thi...

Page 123: ...ICKDSF DFSMSdss ANTRQS TSO ICKDSF DFSMSdss ANTRQS Feature Virtual Storage Platform Universal Storage Platform V VM Basic Functions DKC Emulation Type 2105 2107 3990 2105 2107 Drive Emulation Type 3380 3 3390 1 2 3 3R 9 L M 3380 3 3390 1 2 3 3R 9 L M HOST I F FICON FICON The biggest size of pair creatable volume 3390 M 3390 M Maximum number of pairs in system 16k 16k Maximum number of CTGs in syste...

Page 124: ... Open 9 Open E Open L Open V Host I F Fibre Fibre Maximum size of pair creatable volume Open V 4 TB Open V 4 TB Maximum number of pairs in system 16k Pair 16k Pair Maximum number of CTG in system 256CTG 256CTG Maximum number of pair in one CTG 8192 Pair 8192 Pair Saving differential bitmap Save to SSD Save to SVPSave to SYSTEM DISK Operation interface Storage Navigator Raidmanager Inband Raidmanag...

Page 125: ...his manual The Regulatory Compliance section includes a table of regulatory specifications that are met by the VSP storage system Following the tables are individual regulatory statements that explain the requirements of various countries Regulatory compliance US FCC Notice European Declaration of Conformity Notice of export controls Hazardous and toxic substances Disposal Recycling Electronic emi...

Page 126: ...h America Electronic emission certifications TUV Safety Report EMC Report TUV GS License EMC Certificate CE Mark Yes CE Mark European Union Electronic emission control CB Report and Certificate Yes TUV Worldwide Electronic emission control Test Report for C Tick Approval Yes Australia and New Zealand VCCI Registration for Product and Accessories VCCI Class A Yes VCCI Japan Safety certification GOS...

Page 127: ...R 60950 1 2005GOST R 51318 22 99 24 99 GOST R 51317 3 2 2006 3 3 2008 Certificate No POCC JP Aß46 B81290 North America FCC FCC Part15 SubpartA 2010 Test Report No 29HE0209 YK B 30HE0216 SH B 30KE0010 SH B European Union EN EN55022 2006 A1EN 61000 3 2 2006EN61000 3 3 2008EN55024 1998 A1 A2 Test Report No 12606815 001 EMC Australia and New Zealand C Tick AS NZS CISPR22 2009 Test Report No 29HE0209 Y...

Page 128: ... comply with the FCC regulations you must use shielded cables with your installation The EMI tests were done in the following configurations DKC710I CBX DKC F710I SBX DKC710I CBX DKC F710I SBX DKC F710I UBX Copies of the Underwriters Laboratories EMI compliance certificates are located at the end of this chapter If trouble occurs in another configuration a user may be requested to take appropriate...

Page 129: ...ng pollution caused by electronic information products Toxic and Hazardous Substances and Elements Unit Lead PB Mercury Hg Cadmium Cd Hexavalent Chronium Cr VI Polybrominated biphenyls PBB Polybrominated diphenyl ethers PBDE Controlle r chassis X O O O O O Drive chassis X O O O O O The Symbol O indicates that this toxic or hazardous substances contained in all of the homogeneous materials for this...

Page 130: ...pages Contact Hitachi Data Systems support for detailed information if needed The cache backup battery unit includes a nickel hydride battery A nickel hydride battery should be recycled when it is no longer usable When you replace the battery unit do not dispose of the old one in the trash recycle the battery instead The mark posted on the battery unit is a three arrow mark that means a recyclable...

Page 131: ...Regulatory Compliance C 7 Hitachi VSP User and Reference Guide Figure C 1 UL EMI compliance certificate 1 of 3 ...

Page 132: ...Hitachi VSP User and Reference Guide C 8 Regulatory Compliance Figure C 2 UL EMI compliance certificate 2 of 3 ...

Page 133: ...Regulatory Compliance C 9 Hitachi VSP User and Reference Guide Figure C 3 UL IMI compliance certificate 1 of 3 ...

Page 134: ...Hitachi VSP User and Reference Guide C 10 Regulatory Compliance ...

Page 135: ...gy Attachment This is a disk drive implementation that integrates the controller on the disk drive array See disk array B back end director BED The hardware component that controls the transfer of data between the drives and cache A BED feature consists of a pair of boards A BED is also referred to as a disk adapter DKA BED See back end director BS basic power supply C CCI Command Control Interfac...

Page 136: ...DD SSD connecting cables and racks drive chassis The hardware component of the Virtual Storage Platform that houses disk drives and or flash drives The Virtual Storage Platform can be configured with up to 16 drive chassis dynamic provisioning An approach to managing storage Instead of reserving a fixed amount of storage it removes capacity from the available pool when data is actually written to ...

Page 137: ...lementation that exceeds the performance of expensive SLC SSDs but costs less than less expensive MLC SSDs free capacity The amount of storage space in bytes that is available for use by the host systems front end director FED The hardware component that processes channel commands from hosts and manages host access to cache G GLPR global logical partition J JRE Java Runtime Environment JVM Java Vi...

Page 138: ...d a logical unit LU logical volume See volume logical volume image LVI A logical volume that is configured for use by mainframe hosts for example 3390 9 LUN logical unit number LUSE LUN Size Expansion LVI See logical volume image LVI M mirror In Universal Replicator each pair relationship in and between journal groups is called a mirror Each pair is assigned a mirror ID when it is created The mirr...

Page 139: ... duplicate set of user data and parity information This redundant configuration prevents data loss in case a disk drive within the RAID configuration fails and enables regeneration of user data in the event that one of the array s member disks or the access path to it fails RAID group A set of RAID disks that have the same capacity and are treated as one group for data storage and recovery A RAID ...

Page 140: ... parameters for the RAID storage systems that enable the storage system to be tailored to unique customer operating requirements SOMs are set on the service processor T tiered storage A layered structure of performance levels or tiers that matches data access requirements with the appropriate performance tiers The tiers are Tier 1 Static content Tier 1 is fully supported computing expected to be p...

Page 141: ...storage directors of the Virtual Storage Platform storage system control the front end directors and manage front end access to cache memory volume A logical device LDEV or a set of concatenated LDEVs in the case of LUSE that has been defined to one or more hosts as a single data storage unit A mainframe volume is called a logical volume image LVI and an open systems volume is called a logical uni...

Page 142: ...C A K D E F G H I J L M N O P Q R S T U V W X Y B Z A B C D F G J K L M P R S T V Hitachi VSP User and Reference Guide Glossary 8 ...

Page 143: ...export controls C 5 fans controller chassis 3 2 cooling 3 7 drive chassis 3 8 features hardware 1 2 software 1 12 flash box 3 9 flash chassis 3 10 flash module 3 8 front end director 3 2 hardware description 1 2 host modes 2 55 hub 3 2 logical units 2 6 2 7 mainframe 2 7 2 8 memory cache 3 13 shared 3 14 microprocessor 3 2 new features 1 2 operations battery backup 4 5 option modes system 2 8 powe...

Page 144: ...Hitachi VSP User and Reference Guide Index 2 ...

Page 145: ...Hitachi VSP User and Reference Guide ...

Page 146: ... 2845 Lafayette Street Santa Clara California 95050 2639 U S A www hds com Regional Contact Information Americas 1 408 970 1000 info hds com Europe Middle East and Africa 44 0 1753 618000 info emea hds com Asia Pacific 852 3189 7900 hds marketing apac hds com ...

Reviews:

No comments

Brands by name

Popular brands

Load more brands