manualshive.com logo in svg

HP HPE 3PAR StoreServ 8000, Installation Manual

Share
Download
HP HPE 3PAR StoreServ 8000 Installation Manual Download Page 58

5.

Tighten the captive screws (see

Figure 56 (page 58)

).

NOTE:

Verify that the cable restraint shipping brackets are aligned and leveled with the

link connectors before tightening the captive screws. Hewlett Packard Enterprise recommends
tightening the screws to 19 in-lbs.

Figure 56 Tightening cable restraint shipping bracket captive screws

Removing Cable Restraint Shipping Brackets

Remove the cable restraint shipping brackets only when the cabinet is in its final location.
1.

Remove the data cables from the hook and loop straps.

2.

Loosen the captive screws (see

Figure 57 (page 58)

).

Figure 57 Loosening cable restraint shipping captive screws

3.

Remove the brackets. Be careful not to damage the attached data cables.

Continue on to

“System Setup Verification” (page 83)

, before powering on the system

and initializing the SP and system software.

58

Hewlett Packard Enterprise Factory-Integrated System Setup

Summary of Contents for HPE 3PAR StoreServ 8000

Page 1: ...nstallation Guide Abstract This guide is designed to instruct qualified technicians who are authorized to install the HPE 3PAR StoreServ 8000 Storage system and associated hardware components Part Number QL226 99104 Published May 2016 ...

Page 2: ...11 and 12 212 Commercial Computer Software Computer Software Documentation and Technical Data for Commercial Items are licensed to the U S Government under vendor s standard commercial license Links to third party websites take you outside the Hewlett Packard Enterprise website Hewlett Packard Enterprise has no control over and is not responsible for information outside the Hewlett Packard Enterpr...

Page 3: ...nterconnect Link Ports LEDs 21 PCI HBA Adapters Ports 22 PCI HBA adapters LEDs 25 4 port 16 Gb FC adapter LEDs 25 2 port 10 Gb iSCSI FCoE Converged Network Adapter CNA LEDs 25 2 port 10 Gb Ethernet NIC adapter LEDs 26 4 port 1 Gb NIC adapter LEDs 26 Node LEDs 27 Installing Enclosures 28 Drive Enclosure Numbering 28 Drives 29 Drive LEDs 30 I O Module 31 I O Module LEDs 31 I O Module Numbering 32 SA...

Page 4: ...5 inch drive Allocating and Loading Guidelines 68 Best Practices LFF Drive Allocating and Loading Guidelines 69 Avoid Practices LFF Drive Allocating and Loading Guidelines 70 Installing LFF 3 5 inch Drives 71 Service Processor Installation 72 Installing SP Rail Kit and Service Processor 74 7 System Cabling 79 Cabling Controller Nodes 4 Node systems 79 Host Ethernet Cables 80 Cabling Expansion Cabi...

Page 5: ...ummary 107 SP Wizard Step 9 Applying Settings 107 SP Wizard Step 10 Finish 107 HPE 3PAR Storage System Setup Wizard 108 System Wizard Step 1 Welcome 109 System Wizard Step 2 Enter Serial Number 110 System Wizard Step 3 Verify Storage System 111 System Wizard Step 4 Configure Networking 111 System Wizard Step 5 Configuring Time 111 System Wizard Step 6 Change Password 111 System Wizard Step 7 Verif...

Page 6: ... IP Setup Wizard Fails or Hangs 131 G System Deinstallation 133 System Inventory 133 Uninstalling the System 134 H Warranty and regulatory information 138 Warranty information 138 Regulatory Information 138 Regulatory Model Numbers 138 Safety Precautions 138 General Precautions 138 Symbols on Equipment 139 Precautions for Maintaining and Servicing Products 139 Power Cords 140 Batteries 141 Power S...

Page 7: ...upport strategy NOTE Electronic equipment has special packing for shipping and receives special handling during transportation Hewlett Packard Enterprise is responsible for the manufacturing environment and packaging for shipping Environment For optimal performance at a specific location systems require controlled environmental conditions that can best be facilitated through raised flooring and un...

Page 8: ...oduct specifications Verify that any additional support equipment is properly installed and operational Prior to installation tasks Prior to installation review the packaging to verify that no tampering of the goods has occurred When unpacking the equipment verify that the delivered shipment of all the equipment is correct Refer to the packing slip and the SKUs with the shipment Complete the Syste...

Page 9: ...ill need to unpack the system is based on the dimensions of the container the ramp and the room required to access the system so that it can be moved to its placement destination NOTE For more information about placement of the systems and reserving room for service access see the HPE 3PAR StoreServ 8000 Storage Site Planning Manual at http www hpe com info storage docs Acclimatization Systems are...

Page 10: ...void injury data loss and damage observe the following general precautions when installing or servicing the system Using improper tools can result in damage to the system Prepare an ESD work surface by placing an antistatic mat on the floor or on a table near the system Attach the ground lead of the mat to an unpainted surface of the rack Always use the wrist grounding strap provided with the syst...

Page 11: ...r containers until they arrive at static free workstations Cover workstations with approved static dissipating material Use a wrist strap connected to the work surface and properly grounded earthed tools and equipment Keep the work area free of nonconductive materials such as ordinary plastic assembly aids and foam packing Ensure that you are always properly grounded earthed when touching a static...

Page 12: ...ge to the equipment The total rack load should not exceed 80 of the branch circuit rating Consult the electrical authority with jurisdiction over your facility wiring and safety electrical requirements before performing the installation Remove all pluggable power supplies and modules to reduce the weight of the product Always load the heaviest item first from the bottom of the rack and up This mak...

Page 13: ...nstallation in Existing Rack page 59 For configuration specifications and installation requirements see the HPE 3PAR StoreServ 8000 Storage Site Planning Manual at http www hpe com info storage docs For information about supported hardware and software platforms see the HPE Single Point of Connectivity Knowledge HPE SPOCK website HPE SPOCK http www hpe com storage spock For supported versions of m...

Page 14: ...ed in this chapter and are numbered according to their order and location in the cabinet Bezel Bezel LEDs Figure 1 Bezel LEDs location Table 2 Bezel LEDs description Indicates State Appearance Function LED Icon Power On Green System Power 1 No power Off Enclosure powered by the battery On Amber System Standby Fault On Amber Module Fault 2 No fault Off Fault An issue exists with one or more drives ...

Page 15: ... PCMs in the rear of the enclosure Contains two controller nodes in the rear of the enclosure Controller Nodes The controller node is a system component that caches and manages data in a system and provides hosts with a coherent virtualized view of the system The controller nodes are located in the rear of the controller node enclosure The 3PAR StoreServ 8000 Storage servers contain either two con...

Page 16: ...Node Numbering Figure 2 2 node system numbering Figure 3 4 node system numbering 16 Identifying System Components and Numbering ...

Page 17: ...thernet port used to connect to the storage array management interfaces Ethernet 3 RC 1 Onboard Gb Ethernet port used for remote copy or file persona Ethernet Slot 3 4 Two onboard 12 Gb SAS ports used with SAS cables for connecting to the drive enclosures and I O modules SAS Slot 1 5 DP 1 DP 2 Two Cluster Interconnect Link ports used with 4 directional interconnect link cables that connect the con...

Page 18: ...emote copy functionality Figure 5 Ethernet LEDs location Table 4 Ethernet LEDs description Indicates State Appearance LED Function 1 GbE Link On Green Link Up Speed 1 100 Mb Link On Amber No link established or 10 Mb Link Off No link activity On Green Activity 2 Link activity Flashing No link established Off 18 Identifying System Components and Numbering ...

Page 19: ... P in the SSMC Slot Port Port N 0 1 FC 1 N Node number N 0 2 FC 2 N Node number Figure 6 FC ports LEDs location Table 5 FC Ports LEDs description Indicates State Appearance LED Function Wake up failure dead device or power not applied Off No light All ports Not connected Off Amber Port Speed 1 Connected at 8 Gb s 3 Fast Flashes Connected at 16 Gb s 4 Fast Flashes Normal Connected link up On Green ...

Page 20: ...7 SAS ports LEDs location Table 6 SAS ports LEDs description Indicates State Appearance Function Icon No link activity On Green Activity Link activity Flashing No link established Off Fault On Amber Fault Locate command issued Flashing Normal operation Off 20 Identifying System Components and Numbering ...

Page 21: ...k ports LEDs description Indicates State Appearance LED Function Fault No link established On Amber Fault 1 1 Interconnect link cabling error Flashing 2 Controller node in wrong slot 3 Serial number mismatch between controller nodes No fault Off Link established On Green Status 2 No link established Off LED Troubleshooting Incorrectly configured interconnect link cables result in the port amber LE...

Page 22: ...FC adapter 2 port 10 Gb iSCSI FCoE Converged Network Adapter CNA 4 port 1 Gb Ethernet NIC adapter 2 port 10 Gb Ethernet adapter Figure 9 4 port 16 Gb FC adapter ports location Table 8 4 port 16 Gb FC adapter ports description Slot Port Port 2 1 1 2 2 2 2 3 3 2 4 4 22 Identifying System Components and Numbering ...

Page 23: ... port 1 on the left Table 9 2 port 10 Gb iSCI FCoE CNA ports description Slot Port Port 2 1 1 2 2 2 Figure 11 2 port 10 Gb Ethernet adapter ports location NOTE On some 2 port 10 Gb Ethernet adapters the faceplate labeling is reversed with the port on the left identified as port 2 when oriented as in the illustration above This illustration shows the corrected label with port 1 on the left Table 10...

Page 24: ... labeling is incorrect indicating that the ports are numbered 0 through 3 The illustration above shows the corrected label indicating that the ports are numbered 1 through 4 Table 11 4 port 1 Gb Ethernet NIC adapter ports description Slot Port Port 2 1 1 2 2 2 2 3 3 2 4 4 24 Identifying System Components and Numbering ...

Page 25: ...2 port 10 Gb iSCSI FCoE Converged Network Adapter CNA LEDs Figure 14 2 port 10 Gb s iSCSI FCoE CNA LEDs location NOTE On some 2 port 10 Gb iSCSI FCoE CNAs the faceplate labeling is reversed with the port on the left identified as port 2 when oriented as in the illustration above This illustration shows the corrected label with port 1 on the left Table 13 2 port 10 Gb iSCSI FCoE CNA LEDs descriptio...

Page 26: ... speed 10 Gb s On Amber Amb 10 G 1 Link speed 1 Gb s Off Link up On Green ACT Port 2 Link activity Flashing 4 port 1 Gb NIC adapter LEDs Figure 16 4 port 1 Gb NIC adapter LEDs location NOTE On some 4 port 1 Gb Ethernet NIC Adapters the faceplate labeling is incorrect indicating that the ports are numbered 0 through 3 The illustration above shows the corrected label indicating that the ports are nu...

Page 27: ... On Green Status Boot Rapid Flashing Cluster member Slow Flashing Locate active and safe to remove Solid Blue Blue UID Service Locate active but not safe to remove Flashing Blue The locatenode command used to flash the UID LED Locate not active Off Fault On Amber Fault No fault Off Node in cluster and fault Flashing Controller Nodes 27 ...

Page 28: ... Holds up to 24 LFF 3 5 inch drives in four vertical columns at the front of the drive enclosure Contains two 580 W gold series PCMs in the rear of the enclosure Contains two I O modules in the rear of the enclosure Figure 19 page 28 shows the 3 5 inch drive enclosure and only applies to the LFF drive enclosure Drive Enclosure Numbering There are two types of drive enclosures The maximum number of...

Page 29: ... node enclosure Drives Drive types FC SFF 10 K or 15 K NL SFF and LFF 7 2 K SSD SFF 480 GB 920 GB 1 92 TB 3 84 TB SSD LFF 480 GB Drive sizes SFF 2 5 inch drives Can be installed only in SFF 2 5 inch drive enclosure and the controller node enclosures LFF 3 5 inch drives Can be installed only in the LFF 3 5 inch drive enclosure For more information see Drives Installation page 65 Drives 29 ...

Page 30: ...placement On Amber Fault 1 The locatecage command is issued which flashes all drive fault LEDs for up Flashing to 15 minutes The I O module Fault LEDs at the rear of the enclosure also flash Fault LEDs for failed drives do not flash Normal operation On Green Status 2 Activity Flashing 30 Identifying System Components and Numbering ...

Page 31: ...op The I O Module base 0 is identified by its red label and the base 1 is identified by its green label See Figure 22 page 32 and Figure 23 page 32 I O Module LEDs Figure 21 I O Module LEDs location Table 18 I O Module LEDs description Indicates State Appearance Function LED Icon Power On Green Power No power Off Locate active and safe to remove On Blue UID Service Locate active but not safe to re...

Page 32: ...I O Module Numbering Figure 22 I O modules numbering SFF 2 5 inch drive enclosure Figure 23 I O modules numbering LFF 3 5 inch drive enclosure 32 Identifying System Components and Numbering ...

Page 33: ...ocation Table 19 I O module SAS ports LEDs description Indicates State Appearance Function LED Icon Links up but no activity On Green Activity No cable bad cable not ready or no power Off Activity Flashing Fault On Amber Fault No fault Off I O Module 33 ...

Page 34: ...the two PCMs are located diagonally from one another and the remaining PCM slots are filled with blank panels see Figure 30 page 38 CAUTION Do not proceed without first correcting all fault indications except for PCM batteries NOTE The batteries are fully charged during shipment The batteries might lose some charge and show a degraded status immediately when power is applied This is a temporary co...

Page 35: ...irmware download Flashing AC present and PCM On OK On Green PCM OK Standby mode Flashing PCM fail or PCM fault On Amber Fan Fail Firmware download Flashing No AC power or fault or out of tolerance On Amber DC Output Fail Firmware download Flashing Hard fault not recoverable On Amber Battery Fail Soft fault recoverable Flashing Present and charged On Green Battery Good Charging or disarmed Flashing...

Page 36: ...764 W PCM Numbering Node Enclosure Figure 26 PCMs numbering 2 node 2U node enclosure Figure 27 PCMs numbering 4 node 4U node enclosure 36 Identifying System Components and Numbering ...

Page 37: ...nction LED Icon No AC power or PCM fault On Amber AC input fail Partner PCM Faulty Off or Firmware Download Flashing AC Present and PCM On OK On Green PCM OK Standby mode Flashing PCM fail or PCM fault On Amber Fan Fail Firmware download Flashing No AC power or fault or out of tolerance On Amber DC Output Fail Firmware download Flashing Power Cooling Modules 37 ...

Page 38: ...580 W PCM LEDs Drive Enclosure Figure 29 PCMs numbering SFF 2 5 inch drive enclosure Figure 30 PCMs numbering LFF 3 5 inch drive enclosure 38 Identifying System Components and Numbering ...

Page 39: ...remove controller node and drive chassis power supples The PDUs mounted vertically at the bottom of the rack so to provide a front mounting unit space Service Processor There are two types of service processors Physical service processor SP SPS single power supply RPS redundant power supply Virtual service processor VSP If your configuration includes a physical SP it is located at the bottom of th...

Page 40: ...g power on sequence Flashing System in standby power still on On Amber Power cord not attached or power failure Off System on and system health normal On Green Health 2 System health degraded Flashing Amber System health critical Flashing Red System power off Off Link to network On Green NIC Status 3 Network activity Flashing No network link activity Off Active On Blue UID Service 4 System managed...

Page 41: ...n Power Supply 4 NOTE Might not be applicable to your system for hot plug HPE CS power supplies ONLY Off represents one or more of the following conditions Off Power unavailable Power supply failure Power supply in standby mode Power supply error Designated port for servicing iLO Port Virtual Serial Port 5 For more information about LED status see Understanding LED Indicator Status in the HPE 3PAR...

Page 42: ...on for Server Storage Power Networking and Rack Products available at the Hewlett Packard Enterprise website http www hpe com support Safety Compliance EnterpriseProducts Important Power Safety Precautions Follow these safety precautions when connecting multiple hardware components to power sources WARNING To reduce the risk of personal injury from high leakage current verify earth connection befo...

Page 43: ...for or otherwise capable of operation from the same line voltage supplying the PDU Failure to verify the voltage can lead to severe equipment damage Do not overload the PDU The total input current rating of all equipment connected to each output cannot exceed the total output rating marked on the PDU Use only the hardware provided to install the PDU Required Tools The following tools are required ...

Page 44: ...Single PDU configuration with the outlets facing the center of the rack Figure 34 Single PDU configuration outlets facing center Single PDU configuration with the outlets facing the rear of the rack Figure 35 Single PDU configuration outlets facing rear 44 Power Distribution Unit Installation ...

Page 45: ...the outlets facing the center of the rack Figure 36 Double PDU outlets facing center Inverted PDU with the input power cable exiting the top of the rack Figure 37 Inverted PDU power cable exiting top Recommended PDU Configurations 45 ...

Page 46: ... Figure 38 1U and 0U configurations Installing PDUs 1 Align the power cord retention bracket with the screw holes marked C and install Figure 39 Power cord retention bracket installation 46 Power Distribution Unit Installation ...

Page 47: ...ting pins with the screw holes marked M and install Figure 40 Align mounting pins 3 If installing the PDU inverted install a mounting pin on the bottom of the PDU Figure 41 Inverted PDU mounting pin Installing PDUs 47 ...

Page 48: ...install the input power cable retention band Figure 42 Input power cable retention band 5 If needed insert and secure the power cable Figure 43 Insert and secure power cable 48 Power Distribution Unit Installation ...

Page 49: ...igure 44 Engage mounting pins CAUTION When moving a rack with installed components always use the shipping bracket to secure the PDU 7 If the PDU is not inverted install the shipping bracket above the PDU Figure 45 Shipping bracket installation PDU not inverted Installing PDUs 49 ...

Page 50: ...racket over the mounting pin below the PDU Figure 46 Shipping bracket installation PDU inverted Installing 1U and 0U PDUs 1 Attach the power cord retention bracket Figure 47 Attach power cord retention bracket 50 Power Distribution Unit Installation ...

Page 51: ...2 Attach the appropriate mounting brackets Figure 48 Attach 1U mounting brackets Figure 49 Attach 0U mounting brackets Installing 1U and 0U PDUs 51 ...

Page 52: ...3 Install the cage nuts Figure 50 Cage nut installation 52 Power Distribution Unit Installation ...

Page 53: ...4 Install the PDU Figure 51 1U PDU installation Figure 52 0U PDU installation Installing 1U and 0U PDUs 53 ...

Page 54: ...Securing Cable Retention Bracket Secure the cable weight to the retention bracket Figure 53 Cable retention bracket The installation has been completed 54 Power Distribution Unit Installation ...

Page 55: ...ackaging for damages Report any noticeable damages to the customer Hewlett Packard Enterprise and carrier company 2 Refer to the diagrams on the outside of the cardboard shipping container and follow the instructions about how to remove the packaging CAUTION Use extreme caution and avoid tipping the rack Use more than two people to safely guide the oversized rack down the ramp Positioning the Cabi...

Page 56: ...support the connected data cables and connectors during transport Hewlett Packard Enterprise recommends installing the brackets before transporting the system to another location to prevent damage to the connectors The brackets are not required if the system is in a stationary position so remove and store the brackets for later use Installing Cable Restraint Shipping Brackets 1 Connect the data ca...

Page 57: ...4 Attach the brackets to the side rails Figure 55 Attaching cable restraint shipping brackets to the side rails Cable Restraint Shipping Brackets installation and removal 57 ...

Page 58: ...able Restraint Shipping Brackets Remove the cable restraint shipping brackets only when the cabinet is in its final location 1 Remove the data cables from the hook and loop straps 2 Loosen the captive screws see Figure 57 page 58 Figure 57 Loosening cable restraint shipping captive screws 3 Remove the brackets Be careful not to damage the attached data cables Continue on to System Setup Verificati...

Page 59: ...n enclosure in the rack first mount the two rail shelves to the rack NOTE To mount a one unit 1U rail shelf used for installing an SP to the rack follow the installation instructions included with that SP 1U mounting kit Table 24 page 59 lists the rail kit part numbers for each type of installation Table 24 Rail kit part numbers Rail Kit Assembly 4U Enclosure Rail Kit Assembly 2U Enclosure PN 7939...

Page 60: ...e support bracket before transporting 1 Align the middle support bracket holes with the top holes of the rails The orientation of the middle support bracket is neutral 2 Insert and tighten screws Figure 58 Middle support bracket installation NOTE Two kinds of T25 Torx screws are provided The shoulder screws that fasten the rail channels to the inside of the rack are longer and have flat not rounde...

Page 61: ... for the other rail 6 Check both sides at the front and rear of the rack to ensure that all screws are installed properly Installing a 4U Rail Shelf 1 Verify that each directional specific rail matches with the side of a rack post Labels denote the FRONT RIGHT and FRONT LEFT of the rails 2 Align one end of the rail channel with the holes of the rack post and then push to seat the locating pins in ...

Page 62: ...Figure 60 Cage nut installation 7 Check both sides at the front and rear of the rack to ensure that all screws are installed properly 62 System Component Installation in Existing Rack ...

Page 63: ...vailable to reserve this space Installing Enclosures To install an enclosure in the rack 1 Determine that the enclosure is oriented correctly by looking at the rear of the enclosure Verify the controller node numbering by reviewing the controller node label located on both edges of the enclosure slot Figure 61 Verify controller node numbering 2 At the front of the enclosure remove the yellow panel...

Page 64: ...torque level of 13 in lbs See Figure 62 page 64 Figure 62 Tighten the enclosure hold down screws NOTE Due to limited access use a short length or right angle Torx bit screwdriver to tighten the hold down screws 6 Reinsert the yellow panels on each side of the enclosure 7 Connect the power cables to the power supplies connect the data cables and then secure them with cable straps WARNING Do not pow...

Page 65: ...ity with Data Encryption page 121 NOTE The drives added to the drive enclosure on each controller node pair should be balanced across controller node pairs then across device ports on each controller node pair and finally by enclosures on the same port To avoid any cabling errors all enclosures must have drive pairs installed before powering on the enclosure Each drive includes a green and amber L...

Page 66: ...SFF drive allocation one enclosure Example 2 Figure 65 page 66 shows two SFF enclosures loaded with three pairs of FC drives five pairs of NL drives and two pairs of SSDs Figure 65 SFF drive allocation two enclosures Avoid Practices SFF Drive Allocating and Loading Guidelines When allocating and loading drives in enclosures the following are the Hewlett Packard Enterprise recommended avoid practic...

Page 67: ...F drive in the following manner because this example demonstrates an unbalanced load due to the NL drives not being installed in even pairs Avoid having an odd number of NL drives in both cages Figure 66 SFF unbalanced load Drives Installation 67 ...

Page 68: ...the drive into the bay until the handle begins to engage and then push firmly until it clicks 3 Close the handle to fully seat the drive into the drive bay Figure 67 SFF 2 5 inch drive installation LFF 3 5 inch drive Allocating and Loading Guidelines Figure 68 LFF3 5 inch drive enclosure drive loading order 68 System Component Installation in Existing Rack ...

Page 69: ...s moving upward in the column before moving to the bottom of the next column to the right Load all of the same drive types in adjacent slots before loading the next drive type Allocate the same number of drive type pairs to every enclosure in the system If there is more than one expansion enclosure in the system load the drives in pairs so they are balanced across the enclosure node connections DP...

Page 70: ...e drive enclosure when additional enclosures are available Leaving empty slots in each enclosure allows for future upgrade flexibility to add a new drive type and license features Avoid Practices Example Avoid loading the LFF drives in the following manner because this example demonstrates an unbalanced load due to the NL drives being installed across all four columns Instead the drives should hav...

Page 71: ...rive fully extended position the drive so the handle opens from the left align and slide the drive into the bay until the handle begins to engage and then push firmly until it clicks 3 Close the handle to fully seat the drive into the drive bay Figure 71 LFF 3 5 inch drive installation Drives Installation 71 ...

Page 72: ... a 10 character serial number for configuration This requires up to date versions of HPE 3PAR SP and VSP software and of HPE 3PAR SmartStart The 3PAR StoreServ Storage system serial number can be obtained from the following Located at the rear of the system next to the power switch for the controller node enclosure PCM Located in the product documentation provided with your system The following co...

Page 73: ...or threaded hole racks The following table identifies any tools required for each type of rack Table 25 Rack types Picture Tools Required Rack Type None Round hole None Square hole Screwdriver Threaded hole Figure 72 Cable management arm Service Processor Installation 73 ...

Page 74: ...ge Cabling Configuration Guide at http www hpe com info storage docs Use these procedures when installing the SP into an existing rack Before you begin verify that you have the proper SP 1U Mounting Kit PN 734806 B21 1 Adjust the length of the mounting rails 2 Install the threaded hole cage nuts and rail mounting pins into the rack hole positions where you want to install the component Use two cag...

Page 75: ...h the proper screws Figure 74 Mount and fasten mounting rails 4 Repeat step 1 through step 3 for the other mounting rail WARNING To prevent the risk of injury or equipment damage inspect the rack to ensure that it is adequately stabilized before installing the SP Service Processor Installation 75 ...

Page 76: ...nent and then snapping it into place Figure 75 Installing SP NOTE Align the rail pin guides with the slots on the mounting rails before setting the SP Figure 76 Installing side rails 6 Slide the SP onto the mounting rails and into the rack 76 System Component Installation in Existing Rack ...

Page 77: ...Figure 77 Sliding SP into rack Figure 78 Sliding SP into rack 7 Fasten the SP to the rack Service Processor Installation 77 ...

Page 78: ...uipment Do not disable the power cord grounding plug The grounding plug is an important safety feature Plug the power cord into a grounded earthed electrical outlet where it is accessible at all times Unplug the power cord from the power supply to disconnect power to the equipment Do not route the power cord through a walkway area or adjacent to other hardware components that might pinch or damage...

Page 79: ...red 0 to 3 from bottom to top Connections between controller nodes 0 and 1 and controller nodes 2 and 3 use four large interconnect link cables These cables are directional and must be installed correctly for the system to function properly When inserting the cables you should hear a click to confirm the cable is correctly installed If you have to force the connector the connector is upside down a...

Page 80: ...m the Ethernet switch or hub to two controller nodes Ethernet Separate connections from host computers to each controller node through a switch Connection from a host computer to one controller node Fibre Channel with connections distributed evenly across all controller nodes Example SAN Switch A connected to even ports and SAN Switch B connected to odd ports NOTE To provide redundancy and to perm...

Page 81: ...ack and the right side cabling ID 1 is gray WARNING The PCM latch can damage any cables that are routed in such a way that when the PCM latch handle is closed it will cut into any cable that becomes wedged between the latch and the rack post Keep the cables clear from the PCM latch mechanism NOTE When installing cabling ensure clear access to all system components All power cords need to be tied a...

Page 82: ...ws 2 Pull down on the PDU to lower NOTE If necessary loosen the two bottom screws to unhinge the bracket before pulling down the PDU 3 Ensure that the PDUs are in a fully lowered position before accessing Figure 81 Disengage PDU Pivot brackets Cabling Power Strips Power strips can be located on the side of the rack to supply power to the PCMs Power strips are connected to PDUs Do not exceed the ca...

Page 83: ...roviders Each of the PDUs must connect to a power socket with power being separately supplied by an independent electrical power source CAUTION Ensure that the system does not exceed the ratings of the power sources and PDUs and adheres to the guidelines described in HPE 3PAR StoreServ 8000 Storage Site Planning Manual Power Cord Connections Each system arrives with all internal power cords config...

Page 84: ...rienced environmental changes during the transit allow enough time for the system to acclimatize before proceeding with the power on sequence CAUTION To prevent potential damage to system hardware do not power on the system until it is fully acclimatized If condensation is present after the 24 hour acclimatization period wait for all condensation to fully evaporate before completing the power on s...

Page 85: ... the LEDs as described in Verifying LED Status page 85 Verifying LED Status 1 At the front of the system verify that the bezel System Power green LED is On and the Drive Status amber LED is Off See Bezel LEDs page 14 and Drive LEDs page 30 NOTE If either the bezel Module Fault amber LED or Drive Status amber LED are On do not proceed until the problem is resolved 2 At the rear of the system verify...

Page 86: ...cklist Verify the following before you begin the installation process All components are installed The system is properly set up and powered on All LEDs are properly functioning All LEDs have a status of green The physical SP is installed powered on and assigned an IP address or the VSP is installed connected and the temporary IP address is determined 86 System Installation Checklists ...

Page 87: ... HPE 3PAR SmartStart Static IP address IPv4 only Subnet mask Gateway IP address CAUTION Do not enable DNS settings while you are initializing the system software the system will fail See 3PAR StoreServ Storage Detection Issues page 130 You may need DNS settings later if you want to verify remote support connectivity To configure DNS settings manually after you have completed software initializatio...

Page 88: ...acters in length and can consist of alphanumeric characters and the following special characters period plus dash equal and forward slash New password to assign 3parcust to access HPE 3PAR Service Processor Onsite Customer Care SPOCC NOTE All passwords for the 3parcust username must be between 7 and 32 characters in length and can consist of alphanumeric characters and the following special charac...

Page 89: ... name IP address IPv4 only Subnet mask Gateway IP address Date and time Copy from the SP Enter manually Use Network Time Protocol NTP server information Time Zone New password to assign 3paradm to access the system NOTE Passwords for the 3paradm username can include all printable characters and be between 6 8 characters in length HPE 3PAR SmartStart Performing host configurations after using 3PAR ...

Page 90: ... supported application Do not use vMotion to migrate the VSP from one physical server to another Using vMotion might cause communication failure and interrupt system service Prerequisites for Using a Virtual Service Processor Before you connect to a VSP review and complete the following tasks dev comment This item corresponds to this customer advisory 3PAR Virtual Service Processor for Hyper V Ins...

Page 91: ...f purchase to your IT administrator product manager or purchasing agent If you cannot locate the DVD contact Hewlett Packard Enterprise Support through http www hpe com support hpesc and request assistance with the 3PAR StoreServ 8000 Storage product NOTE Use the installation media that was shipped with your order If you do not have installation media download the latest Hewlett Packard Enterprise...

Page 92: ...Selecting this option powers on the VSP after the installation is complete c Click Finish A Deployment Completed Successfully message should appear after a few minutes 11 Click Close 12 On the left navigation pane verify that the system is on and the green icon is displaying on the new VSP If necessary expanding navigation tree to find the deployed VSP a Right click the VSP in the virtual machine ...

Page 93: ... the VSP is rebooted you will need to repeat this procedure before continuing to the SP configuration Importing the HPE 3PAR VSP into Hyper V This section describes the prerequisites and procedures for importing the VSP into Hyper V WARNING Do not install the VSP on the same 3PAR StoreServ 8000 Storage system that the VSP manages because it might lead to the inability of properly managing the arra...

Page 94: ... available before deploying the VSP OVF file Access your ESXi server to download the VMware vSphere client or see the VMware website https my vmware com web vmware downloads Verify that the 3PAR VSP software DVD is available NOTE Installation DVDs are not typically shipped with the system If you selected the License to Use LTU delivery method of physical delivery when ordering your system use the ...

Page 95: ...nager dialog box click New virtual network select External and then click Add 3 In the Virtual Network Properties area enter a name for the virtual network in the Name field Then click to select External and enter the name of the adapter you want to connect the network to Make sure that the Allow management operating system to share this network adapter check box is selected 4 Click Apply Importin...

Page 96: ...guration procedures might vary depending on the version of Microsoft Windows Apple Mac or Linux OS in use a Select Control Panel Network and Internet Network and Sharing Center and click Change adapter settings b Right click the connection name for the port and select Properties c In the Properties dialog box on the Networking tab double click Internet Protocol Version 4 TCP IPv4 in the list d In ...

Page 97: ...layed until the adapter settings have been changed in the SP and verified No action is necessary the dialog box closes within three minutes NOTE If the Service Processor IP Setup wizard fails see The Service Processor IP Setup Wizard Fails or Hangs page 131 for resolution 8 Select Finish when completed 9 Repeat Step 3 and re configure the original LAN settings on the laptop 10 Disconnect the netwo...

Page 98: ... 2012 is currently running in the system 1 0 gigahertz or higher processor 1 GB of installed RAM 2 GB recommended CD ROM or DVD ROM drive 1280 x 1024 or better screen resolution 3PAR SmartStart Online Help is supported in the following browser versions Microsoft Internet Explorer 8 9 and 10 Mozilla Firefox 14 through 21 After completing the checklist continue on to HPE 3PAR SmartStart System Setup...

Page 99: ...ate your CD go to the following website http www hpe com support softwaredepot 3PAR To launch 3PAR SmartStart 1 Insert the 3PAR SmartStart CD 2 Follow the onscreen instructions to launch the application If the auto run menu does not appear navigate to DVD drive and double click the smartstart exe file NOTE The 3PAR SmartStart application might take up to 3 minutes to launch Continue on to HPE 3PAR...

Page 100: ...rocessor 3 Enter the SP IP address and click OK NOTE When configuring in a DHCP network environment enter the VSP temporary IP address NOTE A security certificate error might occur when there is a problem with a web server s use of the certificate and a blocking page will appear You can ignore the warning and continue to go to the website 4 On the browser window enter setupusr in the User Name box...

Page 101: ...e assigned to the SP IP address IPv4 only Subnet mask and gateway if you are using custom settings DNS settings CAUTION Do not enable DNS settings while you are initializing the system software the system will fail See 3PAR StoreServ Storage Detection Issues page 130 You may need DNS settings later if you want to verify remote support connectivity To configure DNS settings manually after you have ...

Page 102: ...l number Figure 83 Serial number label locations for 4 node and 2 node systems NOTE The 3PAR StoreServ Storage system serial number can be obtained from the following locations At the rear of the system next to the power switch for the controller node enclosure PCM In the product documentation provided with your system 102 Service Processor and System Setup ...

Page 103: ...t SP IP address If you continue to use the temporary SP IP address you might prevent additional system setups from being completed successfully Subnet Mask Gateway To adjust the network speed expand the Advanced option to change the settings If you are setting up a VSP enter the permanent IPv4 address that you want to assign to it 2 Do NOT check the Enable DNS Support box or enter values for Domai...

Page 104: ...arrays An inbound connection enables remote access to the SP and 3PAR StoreServ Storage for authorized 3PAR support personnel An outbound connection enables diagnostic data transfer such as alerts 3PAR StoreServ Storage SP log files and configuration files for remote diagnosis Secure Network Mode is a mode used for communication between the SP and HPE 3PAR Connection portal for diagnostic data tra...

Page 105: ...er the required credentials 3 To mask identifying information in all SP log files select the Make contents of Service Processor log files anonymous box in the Advanced pane When you make the log files anonymous the Remote Support process replaces object names in log files such as TopSecretVirtualVolume with meaningless sequential labels such as VVnnn 4 Click Next proceed to the next wizard step Th...

Page 106: ... usernames must be between 7 and 32 characters in length and consist of alphanumeric characters and the following special characters period plus dash equal and forward slash For more information on assigning these passwords see your completed System Software Installation checklist NOTE To reset a password for the SP log into SPOCC If you lose all your SP passwords and are unable to log into SPOCC ...

Page 107: ...peration status The Status column indicates an operation is in progress with an arrow icon and indicates an operation is complete with a check mark icon 1 Verify that the Setup Progress operations have all completed 2 Click OK to finish the SP setup SP Wizard Step 10 Finish The Finish page describes the status of the SP setup Click Finish The SP setup wizard has been completed Continue on to HPE 3...

Page 108: ...he wizard continues to display incorrectly try refreshing the browser page 1 On the 3 Setup storage system page click the Setup storage system link 2 Enter the SP IP address Click OK NOTE When configuring in a DHCP network environment enter the VSP temporary IP address NOTE The 3PAR Storage System Setup Wizard might not display correctly when using Microsoft Internet Explorer 10 with default secur...

Page 109: ...d configure your 3PAR StoreServ 8000 Storage use the new 10 digit serial number This requires up to date versions of 3PAR SP and VSP software and of 3PAR SmartStart System name System networking information IP address IPv4 only Subnet mask and gateway Time zone information Date and time or Network Time Protocol NTP server address if using automatic date time configuration Time zone region 2 Click ...

Page 110: ...ed to the next wizard step NOTE The 3PAR StoreServ Storage system serial number can be obtained from the following locations At the rear of the system next to the power switch for the controller node enclosure PCM In the product documentation provided with your system Figure 84 Serial number label locations 110 Service Processor and System Setup ...

Page 111: ...ice Processor Setup Wizard 1 Select the Enter data and time manually option if you want to manually enter the date and time If you select Automatic complete the NTP Server field and click Test to get the specified NTP time 2 In the Time Zone section select the appropriate region 3 Click Next to proceed to the next wizard step System Wizard Step 6 Change Password This login profile is used to add t...

Page 112: ...d for HPE 3PAR service user accounts For additional information about strong passwords see the Using Strong Passwords section in the HPE 3PAR StoreServ Storage Concepts Guide Post Installation System Tasks The system setup is now complete Perform the recommended tasks before using the system Validating Remote Support For more information see Validating Remote Support page 129 Identifying Physical ...

Page 113: ...hysical location of the drive cage is identified make a note of the drive cage number on a separate paper for reference during servicing NOTE Table below provides example of drive cage numbering 4U 4U 4U 4U 4U 4U 4U 4U 4U 4U 4U 4U 1U 1U 1U 4U 4U 4U 4U 4U 4U 4U 4U 4U 4U 4U 4U Cage rack 2 Cage rack 1 Node rack 5 Enter setcage position Rack xx Rack Unit yy cage n where xx is the rack designator R0 is...

Page 114: ...roduct documentation to identify the recommended software update method To download product updates go to either of the following Hewlett Packard Enterprise Support Center Get connected with updates page www hpe com support e updates Software Depot website www hpe com support softwaredepot To view and update your entitlements and to link your contracts and warranties with your profile go to the He...

Page 115: ...e information about CSR contact your local service provider or go to the CSR website www hpe com support selfrepair Remote support Remote support is available with supported devices as part of your warranty or contractual support agreement It provides intelligent event diagnosis and automatic secure submission of hardware event notifications to Hewlett Packard Enterprise which will initiate a fast...

Page 116: ...nding on system configuration Appearance of components Quantity of components Whether or not the component is included with the system System Components LFF 3 5 in Drive Enclosure 4U SFF 2 5 in Drive Enclosure 2U Controller Node I O module 116 System Illustrated Contents List ...

Page 117: ...Battery for 764 W PCM 764 W PCM without Battery 580 W PCM 3 5 inch LFF Drive 2 5 inch SFF Drive System Components 117 ...

Page 118: ...3 5 inch LFF Drive Assembly Blank 2 5 inch SFF Drive Assembly Blank w Yellow Touch Point HPE 3PAR Service Processor 3PAR Service Processor 118 System Illustrated Contents List ...

Page 119: ...il kits Rail Kits Right and left 2U rail subassembly right subassembly shown Right and left 4U rail subassembly right subassembly shown M5 screw for middle support bracket M5 shoulder screw for rack mount M5 cage nut Middle support bracket Rail Kits 119 ...

Page 120: ...ps Product documentation Various enclosure labels RJ45 to serial crossover cable RJ45 to serial cable CAT5 cable Ethernet crossover cable Enclosure rack mounting fasteners Software kits Entitlement certificate Controller node CSR and HBA CSU limitations 1U filler panels Link cables Shipping bracket kit 120 System Illustrated Contents List ...

Page 121: ... encrypted state NOTE To review the encryption status of individual drives within the system use the SSMC This feature allows you to perform the following encryption related tasks Check encryption status Enable encryption Back up an authentication key Restore an authentication key Generate a new key Recover a key For more information about enabling the feature see the HPE 3PAR StoreServ Management...

Page 122: ...erv Storage LFF 3 5 inch Drive Enclosure For more information about drive enclosures see Installing Enclosures page 28 Information About Drive Enclosure Upgrades The number of drive enclosures attached to a specific controller node pair should be determined by the desired RAID set size and HA Cage protection requirements Drive enclosures should be added and configured to meet the HA Cage protectio...

Page 123: ...rades You can install additional drives to upgrade partially populated drive enclosures WARNING If the 3PAR StoreServ Storage is enabled with the Data at Rest DAR encryption feature only use the self encrypting drives SED Using a non self encrypting drive might cause errors during the upgrade process NOTE SSDs have a limited number of writes that can occur before reaching the SSD s write endurance...

Page 124: ...AR StoreServ 8000 Storage 2 5 inch Drive Enclosure Drive Placement Drive pairs should be placed in the lowest available slot numbers Figure 85 3PAR StoreServ 8000 Storage 2 5 inch Drive Enclosure drive placement order 3PAR StoreServ 8000 Storage 3 5 inch Drive Enclosure Drive Placement Drive pairs should be populated in columns and in the lowest available vertical slots in that column Figure 86 3P...

Page 125: ...pane 6 Install the drives See Drives Installation page 65 The display refreshes periodically and you should see the inserted drives as New in the State column They are ready to be admitted into the system which occurs automatically Within six minutes depending on the system load and the size of the upgrade the State of the new drives changes to Normal and the system starts to initialize the chunkl...

Page 126: ...ables to the power supplies and secure them with cable straps 10 On the PCMs press the power switch to ON NOTE It might take up to five minutes for the drive enclosure to fully power up with the storage drives available to the cluster 11 Verify that the drive enclosure fan module I O modules drive enclosure power supplies and drive status LEDs are lit green and operating normally 12 To verify that...

Page 127: ...fy the 3PAR StoreServ Storage Manually Launching the HPE 3PAR Service Processor Setup Wizard To manually launch the 3PAR Service Processor Setup Wizard 1 Launch a web browser 2 Enter the SP URL address For example https 10 0 120 227 sp SpSetupWizard html 3 In the User Name text box enter setupusr and leave the password blank You can change the password in the 3PAR Service Processor Setup Wizard 4 ...

Page 128: ...ou must have superuser access 2 Install the downloaded software package and follow the setup instructions NOTE The first time you open SSMC after installation you must set up the user name and password for the administrator account Click Set credential in the dialog that appears After setting the administrator credential you must log in to the Administrator Console and add an 3PAR StoreServ Storag...

Page 129: ...transferring files successfully by doing the following steps a Click Home to return to the SPOCC home page The Transfer Status entry indicates the overall status of SP file transfer b To access the SP File Transfer Monitor click Transfer Status c Verify that the SP file transfer is successful The Last transfer status entry should include information about the last SP transfer including the date an...

Page 130: ...d set up a new valid IP address For more information on determining and assigning the SP IP address see Configuring Physical Service Processor IP Address page 96 After you have set up the new IP address use that new permanent IP address to restart the 3PAR Service Processor Setup Wizard Enter the URL https permanent SP IP address sp SpSetupWizard html If you are using a VSP use the temporary SP IP...

Page 131: ...ervice Processor is connected to a network that can access the internet 2 If a Proxy Server is required to access the internet verify proxy configuration 3 Verify Firewall and or Proxy Server rules allow connections to the HP Collector and Global Access Servers 4 For more information on configuring Remote Support see the HP 3PAR StoreServ 8000 Storage Installation Guide NOTE Until remote connectiv...

Page 132: ...Workaround Close the browser and relaunch the Service Processor IP Setup wizard 132 Troubleshooting ...

Page 133: ...erify with a system administrator that the system is prepared for shutdown Complete the System Inventory page 133 System Inventory To complete the system inventory record the following information for each system to be uninstalled Customer name Site information System serial numbers Issue the showinventory CLI command Software currently running on the system Issue the following CLI commands to obt...

Page 134: ...ction provides instructions for uninstalling a system 1 Connect the laptop to the highest numbered controller node with a serial cable 2 Using a terminal emulator program such as Putty set your maintenance laptop console to the following settings Value Setting 57600 Baud rate 8 Data bits None Parity 1 Stop bits Xon Xoff Flow control 3 Log into the node as the console user 3PAR TM OS version sernum...

Page 135: ...s during deinstallation depends on the type size and number of disks The estimate of 71 minutes below is an example NOTE If you do not wait for the chunklets to be initialized data still resides on the disks but cannot be easily accessed When the chunklets are initialized zeros are written over the existing data If you require additional assistance contact 3PAR Technical Support At this point all ...

Page 136: ...ss ENTER 1 SP Main HP 3PAR Service Processor Menu Transfer media ethernet Transfer status Ok Enter Control C at any time to abort this process 1 SP Control Status 2 Network Configuration 3 StoreServ Configuration Management 4 StoreServ Product Maintenance 5 Local Notification Configuration 6 Site Authentication Key Manipulation 7 Interactive CLI for a StoreServ X Exit 13 From the SP Control menu e...

Page 137: ...ower off the SP Are you certain you want to halt the SP now y or n 15 Set all power breakers on the PDUs to the OFF position CAUTION For safety precaution all drive magazines must be properly grounded before removing from the drive chassis 16 Unplug the system main power cords 17 For system and drive expansion cabinets coil all main powers cords and strap the cable along the mounting rails on the ...

Page 138: ...the marketing name or model number of the product Products covered by this manual Regulatory Model Number RMN 3PARA SV1009 NOTE All options and models of the 3PAR StoreServ 8000 Storage family are covered by this manual Safety Precautions Retain and follow all product safety and operating instructions Always refer to the documentation printed or electronic supplied with your product If there is a ...

Page 139: ...utions for Maintaining and Servicing Products To reduce the risk of electric shock or damage to the equipment when installing maintaining or servicing products observe the following precautions Some products contain power supplies that are capable of producing hazardous energy levels Refer to the documentation included with your product to determine whether it contains these power supplies The ins...

Page 140: ...tal rack load should not exceed 80 percent of the branch circuit rating Consult the electrical authority having jurisdiction over your facility wiring and installation requirements Power Cords To reduce the risk of electric shock or damage to the equipment Use an approved AC OR DC or DC power cord as appropriate If you have questions about the type of power cord to use contact your Hewlett Packard...

Page 141: ...f the system has multiple sources of power you must disconnect all AC or DC power cords from the power supplies to completely disconnect power from the system Verify that the external power source connected to your product matches the type of power source indicated on the electrical ratings label If you are not sure of the type of power source required consult your Hewlett Packard Enterprise autho...

Page 142: ...142 Warranty and regulatory information ...

Page 143: ...Regulatory Information 143 ...

Page 144: ...evices do not have an FCC logo or ID on the label This equipment has been tested and found to comply with the limits for a Class A digital device pursuant to Part 15 of the FCC rules These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment This equipment generates uses and can radiate radio frequency energy a...

Page 145: ...ur du Canada Such compliance is denoted on the product rating label by CAN ICES 3 A NMB 3 A European Union Notices This compliance is indicated by the following conformity marking placed on the product This product complies with the following EU directives Low Voltage Directive 2006 95 EC EMC Directive 2004 108 EC RoHS Directive 2011 65 EU Compliance with these directives implies conformity to app...

Page 146: ...anufacturer and Local Representative Information Manufacturer information Hewlett Packard Enterprise Company 3000 Hanover Street Palo Alto CA 94304 U S Local representative information Russian Russia Belarus Kazakhstan Local representative information Kazakh Russia 146 Warranty and regulatory information ...

Page 147: ... Y indicates the year counting from within each new decade with 2000 as the starting point for example 238 2 for 2002 and 38 for the week of September 9 In addition 2010 is indicated by 0 2011 by 1 2012 by 2 2013 by 3 and so forth YYWW where YY indicates the year using a base year of 2000 for example 0238 02 for 2002 and 38 for the week of September 9 Regulatory Information 147 ...

Page 148: ...erial Content Declaration This product complies with the India E waste Management and Handling Rule 2011 and prohibits use of lead mercury hexavalent chromium polybrominated biphenyls or polybrominated diphenyl ethers in concentrations exceeding 0 1 weight and 0 01 weight for cadmium except for the exemptions set in Schedule 2 of the Rule Turkey RoHS material content declaration Ukraine RoHS mater...

Page 149: ... per second H HBA host bus adapter K kg kilogram L LFF large form factor N NL near line NTP Network Time Protocol P PCM power cooling module PDU power distribution unit PN part number S SAS Serial Attached SCSI SFF small form factor SKU stock keeping unit SP service processor SSH secure shell SSMC HPE 3PAR StoreServ Management Console V VSP virtual service processor 149 ...

Page 150: ...ly blank 118 SPs 39 118 VSPs 39 contacting Hewlett Packard Enterprise 114 controller nodes see nodes customer self repair 115 D deinstallation system 133 documentation providing feedback on 115 drive enclosures 28 adding 122 numbering 28 numbering LFF enclosure 28 numbering SFF enclosure 28 drives 29 adding 122 allocating and loading LFF drives 68 allocating and loading SFF drives 65 encryption 12...

Page 151: ...nodes 27 SPs 40 verifying status 85 LFF large form factor 3 5 inch drive 14 licenses redeeming and registering 12 N nodes 15 controller nodes 14 enclosure 15 illustration 116 LEDs 27 numbering 2 node system 16 numbering 4 node system 16 ports 17 O open virtualization format see OVF OVF open virtualization format 90 P PCMs 580 W 34 580 W LEDs 37 764 W 34 764 W LEDs 35 illustrations 117 numbering 2 ...

Page 152: ...s 113 installation in existing rack 59 inventory 133 post install tasks 112 powering on 85 pre installation planning 7 repositioning 83 setup 98 setup wizard 108 shipping containers 9 SmartStart setup 99 uninstalling 134 unpacking enclosures 59 verifying LED status 85 verifying power connections 83 verifying setup 83 T troubleshooting 130 duplicate IP address issues 130 HPE 3PAR SmartStart 130 IP ...

Reviews:

No comments

Brands by name

Popular brands

Load more brands