manualshive.com logo in svg

NetApp E2600, Installation Manual

Share
Download
NetApp E2600 Installation Manual Download Page 88

80

E2600 Controller-Drive Tray Installation Guide

5.

Lower the drive straight down, and then rotate the drive handle down until the drive snaps into place under the 
drive release lever.

Figure 74  Drive Release Lever Locked by the Drive Handle

1.

Drive Release Lever

2.

Drive Handle

6.

Install the other drives in rows from left to right until the drive drawer is fully populated.

Figure 75  Fully-Populated Drive Drawer

7.

Push the drive drawer all the way back into the drive tray, closing the levers on each side of the drive drawer. 

ATTENTION Risk of equipment malfunction

 – Make sure you push both levers to each side so that the drive 

drawer is completely closed. The drive drawer must be completely closed to prevent excess airflow, which has the 
potential to damage the drives.

8.

Continue onto the next drive drawer, repeating step 

1

 through step 

7

 for each drive drawer in the configuration.

Summary of Contents for E2600

Page 1: ...rive Trays NetApp Inc 495 East Java Drive Sunnyvale CA 94089 U S A Telephone 1 408 822 6000 Fax 1 408 822 4501 Support telephone 1 888 4 NETAPP Document comments doccomments netapp com Information Web http www netapp com Part number 53014 00 Rev A March 2012 ...

Page 2: ...UDING BUT NOT LIMITED TO PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE NetApp reserves the right to change any products des...

Page 3: ... logo SyncMirror Tech OnTap The evolution of storage Topio vFiler VFM Virtual File Manager VPolicy WAFL Web Filer and XBB are trademarks or registered trademarks of NetApp Inc in the United States other countries or both IBM the IBM logo and ibm com are trademarks or registered trademarks of International Business Machines Corporation in the United States other countries or both A complete and cur...

Page 4: ...iv ...

Page 5: ...es and SAS Cables 9 Things to Know Taking a Quick Glance at the Hardware in a E2612 Controller Drive Tray or a E2624 Controller Drive Tray Configuration 11 Things to Know Taking a Quick Glance at the Hardware in a E2660 Controller Drive Tray Configuration 18 For Additional Information on the E2600 Controller Drive Tray Configuration 23 Step 2 Installing and Configuring the Switches 25 Things to Kn...

Page 6: ... Controller Drive Tray Configurations 55 Things to Know General Installation of Drive Trays with the E2600 Controller Drive Tray 55 Procedure Installing the DE1600 Drive Trays and the DE5600 Drive Trays 55 Procedure Installing Drives in the DE1600 Drive Trays and the DE5600 Drive Trays 64 Things to Know General Installation of Drive Trays with the E2660 Controller Drive Tray 66 Steps to Install DE...

Page 7: ...Cords 93 Procedure Connecting AC Power Cords 94 Procedure Connecting DC Power Cords 94 Step 10 Turning on the Power and Checking for Problems in a E2600 Controller Drive Tray Configuration 95 Procedure Turning On the Power to the Storage Array and Checking for Problems in a E2600 Controller Drive Tray Configuration 95 Things to Know LEDs on the E2600 Controller Drive Tray 96 Things to Know LEDs on...

Page 8: ...troller Drive Tray 123 Things to Know Seven Segment Display Sequence Code Definitions 123 Things to Know Lock Down Codes for the Controller Drive Tray 125 Things to Know Seven Segment Display Use Cases 126 Things to Know Seven Segment Display for the Drive Trays 128 Regulatory Compliance Statements FCC 1 ...

Page 9: ...pgrade to 192 The upgrade is a Premium feature Any combination of E2600 controller drive trays attached to DE1600 drive trays or DE5600 drive trays not to exceed a maximum of 96 or 192 drive slots or 8 total trays in the storage array Two 6 Gb s host connectors Options of 1 GB 2 GB or 4 GB memory for cache offload to flash memory battery backup Simplex E2612 controller drive tray or E2624 controll...

Page 10: ...192 The upgrade is a Premium feature Any combination of E2600 controller drive trays attached to DE1600 drive trays or DE5600 drive trays not to exceed a maximum of 96 or 192 drive slots or 8 total trays in the storage array Two 6 Gb s host connectors in addition to one of the following host interface cards Two 6 Gb s SAS connectors Four 1 Gb s iSCSI connectors Two 10 Gb s iSCI connectors Four 8 G...

Page 11: ...oller manages the input output I O between the host system and data volumes drive tray One tray with drives one or two environmental services modules ESMs power supplies and fans A drive tray does not contain controllers environmental services module ESM A canister in the drive tray that monitors the status of the components An ESM also serves as the connection point to transfer data between the d...

Page 12: ...rdware Item Included with the Controller Drive Tray Cabinet Make sure that your cabinet meets the installation site specifications of the various E2600 storage array components Refer to the Storage System Site Preparation Guide for more information Depending on the power supply limitations of your cabinet you might need to install more than one cabinet to accommodate the different components of th...

Page 13: ...ng to an external power source such as a wall plug Your cabinet might have special power cords that you use instead of the power cords that ship with the controller drive tray and the drive trays Optional Two DC power connector cables are provided with each drive tray for connection to centralized DC power plant equipment Four DC power connector cables are provided if additional redundancy is requ...

Page 14: ...rs two SFP transceivers for each fiber optic cable Depending on the configuration of your storage array you might need to use 8 Gb s Fibre Channel SFP transceivers You must purchase only Restriction of Hazardous Substances RoHS compliant SFP transceivers Copper Fibre Channel cables optional Use these cables for connections within the storage array For the differences between the fiber optic cables...

Page 15: ...cables to connect the controller drive tray to the drive tray Serial cable This cable is used for support only You do not need to connect it during initial installation DB9 to PS2 adapter cable This cable adapts the DB9 connector on commercially available serial cables to the PS2 connector on the controller Item Included with the Controller Drive Tray or Drive Trays ...

Page 16: ...are backup copies SANtricity ES Storage Manager Installation DVD SANtricity ES Storage Manager software and documentation To access product documentation use the documentation map file doc_launcher html which is located in the docs directory Item Included with the Tray Labels Help you to identify cable connections and lets you more easily trace cables from one tray to another A cart Holds the tray...

Page 17: ...lower data rate Ports for 8 Gb s Fibre Channel host connections require SFP transceivers designed for this data rate These SFP transceivers look similar to other SFP transceivers but are not compatible with other types of connections SFP transceivers for 1 Gb s iSCSI and 10 Gb s iSCSI connections have a different physical interface for the cable and are not compatible with other types of connectio...

Page 18: ...FP Transceiver 2 Fiber Optic Cable Figure 2 1 Gb s iSCSI Cable Connection 1 Active SFP Transceiver 2 Copper Cable with RJ 45 Connector Figure 3 Copper Fibre Channel Cable Connection 1 Copper Fibre Channel Cable 2 Passive SFP Transceiver Figure 4 SAS Cable Connection 1 SAS Cable 2 SFF 8088 Connector ...

Page 19: ...ce DA premium feature ATTENTION Possible data loss If you remove all Fibre Channel Host Interface Cards HICs from a storage array and replace them with HICs of a different network protocol such as iSCSI or SAS you must first deactivate the Remote Volume Mirroring RVM premium feature if it is activated If you do not deactivate RVM before removing the Fibre Channel HICs your system loses data access...

Page 20: ...e LED 6 Drive Canister Figure 7 E2612 or E2624 Duplex Configuration Rear View 1 Controller A Canister 2 Seven Segment Display 3 Host Interface Card Connector 1 4 Host Interface Card Connector 2 5 Serial Connector 6 Ethernet Connector 1 7 Ethernet Link Active LED 8 Ethernet Link Rate LED 9 Ethernet Connector 2 10 Host SFF 8088 Connector 2 Native 11 Host Link 2 Fault LED 12 Host Link 2 Active LED 13...

Page 21: ... Connector and Switch Figure 8 E2612 or E2624 Right Rear Subplate with No Host Interface Card 1 ESM Expansion Fault LED 2 ESM Expansion Active LED 3 Expansion SFF 8088 Port Connector Figure 9 E2612 or E2624 Right Rear Subplate with a SAS Host Interface Card 1 Host Interface Card Link 3 Up LED 2 Host Interface Card Link 3 Active LED 3 SFF 8088 Host Interface Card Connector 3 4 Host Interface Card L...

Page 22: ...t Interface Card Link 5 Active LED 9 FC Host Interface Card Connector 5 10 Host Interface Card Link 6 Up LED 11 Host Interface Card Link 6 Active LED 12 FC Host Interface Card Connector 6 13 ESM Expansion Fault LED 14 ESM Expansion Active LED 15 Expansion SFF 8088 Port Connector Figure 11 E2612 or E2624 Right Rear Subplate with a 1 Gb iSCSI Host Interface Card 1 Host Interface Card Link 3 Up LED 2...

Page 23: ...ower Fan A Canister optional 10 Standby Power LED 11 Power Fan DC Power LED 12 Power Fan Service Action Allowed LED 13 Power Fan Service Action Required LED 14 Power Fan AC Power LED ATTENTION Possible equipment damage You must use the supported drives in the drive tray to ensure proper performance For information about supported drives contact a Technical Support representative ATTENTION Risk of ...

Page 24: ... View 1 Left End Cap Has the Drive Tray LEDs 2 Drives 3 Right End Cap Figure 15 DE1600 Drive Tray or DE5600 Drive Tray with AC Power Option Rear View 1 ESM A Canister 2 Host Connector 1 3 Host Connector 2 4 Seven Segment Display Indicators 5 Serial Connector 6 Ethernet Connector 7 Expansion Port SFF 8088 Connector 8 Power Fan Canister 9 Power Connector 10 Power Switch 11 ESM B Canister ...

Page 25: ... service person is required to make the DC power connection per NEC and CEC guidelines A two pole 30 amp circuit breaker is required between the DC power source and the drive tray for over current and short circuit protection Before turning off any power switches on a DC powered drive tray you must disconnect the two pole 30 amp circuit breaker WARNING W12 Risk of electrical shock This unit has mo...

Page 26: ...ocations NOTE iSCSI host ports do not support the Data Assurance DA premium feature ATTENTION Possible data loss If you remove all Fibre Channel Host Interface Cards HICs from a storage array and replace them with HICs of a different network protocol such as iSCSI or SAS you must first deactivate the Remote Volume Mirroring RVM premium feature if it is activated If you do not deactivate RVM before...

Page 27: ...nk 2 Rate LED 12 Host Link 2 Fault LED 13 Base Host SFF 8088 Connector 2 14 Host Link 2 Active LED 15 Host Link 1 Fault LED 16 Host Link 1 Active LED 17 Base Host SFF 8088 Connector 1 18 Controller A Canister 19 ESM Expansion Fault LED 20 ESM Expansion Active LED 21 Expansion SFF 8088 Port Connector 22 Second Seven Segment Display Field 23 First Seven Segment Display Field 24 Cache Active LED 25 C...

Page 28: ... ESM Expansion Active LED 3 Expansion SFF 8088 Port Connector Figure 20 E2660 Right Rear Subplate with a SAS Host Interface Card 1 Host Interface Card Link 3 Up LED 2 Host Interface Card Link 3 Active LED 3 SFF 8088 Host Interface Card Connector 3 4 Host Interface Card Link 4 Up LED 5 Host Interface Card Link 4 Active LED 6 SFF 8088 Host Interface Card Connector 4 7 ESM Expansion Fault LED 8 ESM E...

Page 29: ... Host Interface Card Link 4 Up LED 5 Host Interface Card Link 4 Active LED 6 FC Host Interface Card Connector 4 7 Host Interface Card Link 5 Up LED 8 Host Interface Card Link 5 Active LED 9 FC Host Interface Card Connector 5 10 Host Interface Card Link 6 Up LED 11 Host Interface Card Link 6 Active LED 12 FC Host Interface Card Connector 6 13 ESM Expansion Fault LED 14 ESM Expansion Active LED 15 E...

Page 30: ...4 Host Interface Card Link 4 Up LED 5 Host Interface Card Link 4 Active LED 6 iSCSI Host Interface Card Connector 4 7 Host Interface Card Link 5 Up LED 8 Host Interface Card Link 5 Active LED 9 iSCSI Host Interface Card Connector 5 10 Host Interface Card Link 6 Up LED 11 Host Interface Card Link 6 Active LED 12 iSCSI Host Interface Card Connector 6 13 ESM Expansion Fault LED 14 ESM Expansion Activ...

Page 31: ...re proper performance For information on supported drives contact a Technical Support representative ATTENTION Risk of equipment malfunction To avoid exceeding the functional and environmental limits install only drives that have been provided or approved by the original manufacturer Not all controller drive trays are shipped with prepopulated drives System integrators resellers system administrat...

Page 32: ...24 E2600 Controller Drive Tray Installation Guide ...

Page 33: ...or SAS switches The SFPs in Table 10 on page 27 apply only if you are using Fibre Channel network protocols The switches in the following table are certified for use with an E2600 storage array and an E2660 storage array which both use SANtricity ES Storage Manager Version 10 83 Table 7 Supported Fibre Channel Switches and Routers Vendor Model Data Rate Brocade 200E 4 Gb 4100 4 Gb 48000 4 Gb 5000 ...

Page 34: ...on utilities that are supplied with the switch QLogic SANbox5200 4 Gb SANbox3600 4 Gb SANbox3800 8 Gb SANbox5600 8 Gb SANbox5800 8 Gb SANbox9000 8 Gb 6142 8 Gb Vendor Model Data Rate Connection Brocade 8000 10 GB Optical Turbiron 10 GB Optical Cisco Catalyst 3560 1 GB RJ45 Catalyst 3750G 24TS 1 GB RJ45 Nexus 5000 10 GB Optical Dell 8024 10 GB Copper RJ45 8024F 10 GB Ethernet SFP Plus 24 port Power...

Page 35: ... by accessing it from the applicable switch vendor s website This update might require that you cycle power to the switch 4 Find your switch in the following table to see whether you need to make further configuration changes Use your switch s configuration utility to make the changes Table 11 Supported Switch Vendors and Required Configuration Changes Vendor Version Data Rate Avago 8G 10G Finisar...

Page 36: ...28 E2600 Controller Drive Tray Installation Guide ...

Page 37: ...rds NICs for iSCSI of the HICs to which you connect them For the best performance HBAs for SAS and FC connections should support the highest data rate supported by the HICs to which they connect For maximum hardware redundancy you must install a minimum of two HBAs for either SAS or FC host connections or two NICs or iSCSI HBAs for iSCSI host connections in each host Using both ports of a dual por...

Page 38: ...to obtain the world wide name for the port instead of using the BIOS utility If a configuration uses a Fibre Channel switch you can find the World Wide Port Name of each port of the HBA from the name server of the switch 4 Reboot or start your host 5 While your host is booting look for the prompt to access the HBA BIOS utility 6 Select each HBA to view its HBA host port world wide name 7 Record th...

Page 39: ...mestic and worldwide operation NOTE Make sure that the combined power requirements of your trays do not exceed the power capacity of your cabinet Procedure Installing the E2600 Controller Drive Tray Figure 24 Airflow Direction Through and Clearance Requirements for the E2600 Controller Drive Tray with 12 Drives 1 76 cm 30 in clearance in front of the cabinet 2 61 cm 24 in clearance behind the cabi...

Page 40: ... from moving 3 Install the mounting rails in the cabinet For more information refer to the installation instructions that are included with your mounting rails If you are installing the mounting rails above an existing tray position the mounting rails directly above the existing tray If you are installing the mounting rails below an existing tray allow 17 8 cm 7 00 in clearance below the existing ...

Page 41: ...ray alternating so that the cabinet does not become top heavy 4 With the help of one other person slide the rear of the controller drive tray onto the mounting rails Make sure that the top mounting holes on the controller drive tray align with the mounting rail holes of the cabinet see Figure 26 and Figure 27 on page 34 The rear of the controller drive tray slides into the slots on the mounting ra...

Page 42: ...r drive tray to the mounting rails 7 Install the bezel on the front of the controller drive tray 8 Install the drive trays Refer to Connecting the E2600 Controller Drive Tray to the Drive Trays on page 81 Procedure Installing the E2660 Controller Drive Tray You can install the high density 6 Gb SAS SBB 2 0 compliant E2660 controller drive tray into an Industry standard cabinet provided it has a de...

Page 43: ... in clearance behind the cabinet NOTE Fans pull air through the controller drive tray from front to back across the drives 2 Lower the feet on the cabinet to keep the cabinet from moving WARNING W09 Risk of bodily injury Three persons are required to safely lift the component WARNING W15 Risk of bodily injury An empty tray weighs approximately 56 7 kg 125 lb Three persons are required to safely mo...

Page 44: ... Controller Drive Tray with Controller Drive Tray Handles Two on Each Side Set the drive tray aside Figure 29 E2660 Controller Drive Tray with Controller Drive Tray Handles Two on Each Side 4 Position the mounting rails in the cabinet Figure 30 Positioning the Mounting Rails in the Cabinet 1 Screws for Securing the Mounting Rail to the Cabinet Front 2 Screws for Securing the Mounting Rail to the C...

Page 45: ...1 Front of the Mounting Rail 2 Two M5 Screws for the Rear EIA Support Rail 3 Front of the Cabinet 4 Two M5 Screws for the Front EIA Support Rail 5 Adjustable Rail Tightening Screws 6 Rear Hold Down Screw 7 Cabinet Mounting Holes on the Front EIA Support Rail 8 Cabinet Mounting Holes on the Rear EIA Support Rail 9 Mounting Rail Lip b Remove the rear hold down screw It protrudes from the inside of t...

Page 46: ...ws on the mounting rail are loose so that the mounting rail can extend or contract as needed see the figure Short Adjustable Mounting Rail Left Side b Place the mounting rail inside the cabinet and extend the mounting rail until the flanges on the mounting rail touch the inside of the cabinet see the figure Short Adjustable Mounting Rail Attached to the Cabinet on page 39 c Insert one M5 screw thr...

Page 47: ...ree persons are required to safely move an empty tray If the tray is populated with components a mechanized lift is required to safely move the tray 9 With the help of at least two other persons slide the rear of the controller drive tray onto the mounting rails The controller drive tray is correctly aligned when the mounting holes on the front flanges of the controller drive tray align with the m...

Page 48: ...dles two to a side b Use the front enclosure lift handles to slide the drive tray all the way into the cabinet c After the drive tray is securely in the cabinet use your thumb to unlatch and remove the front enclosure lift handles two to a side Figure 34 Removing an Enclosure Lift Handle from the Controller Drive Tray 1 Pull the thumb latch away from the controller drive tray to detach the hook 2 ...

Page 49: ...ray to the mounting rails a Insert two M5 screws through the bottom holes of a flange on the controller drive tray so that the screws go through the EIA support rail and engage the bottom captured nuts in the mounting rail Tighten the screws b Repeat substep a for the second flange Figure 35 Attaching the Front of the Controller Drive Tray 1 Four Screws for Securing the Front of the Controller Dri...

Page 50: ... drive tray by pressing on the tab holding the fan canister handle in place and then pulling the fan canister toward you Figure 36 Removing the Fan Canister from the Controller Drive Tray 1 Fan Canister Handle 13 Use the fan canister handle to pull the fan canister out of the controller drive tray ...

Page 51: ... 10 32 screw through the side sheet metal of the controller drive tray into the captured nut on the side of the mounting rail Tighten the screws b Repeat substep a for the other side NOTE After the controller drive tray is installed there should be seven screws on each side right and left of the cabinet NOTE Make sure that each drive drawer in the controller drive tray is securely fastened to ensu...

Page 52: ...15 Slide the fan canister all the way back into the drive tray until the tab on the fan canister latches 16 Attach the bezel onto the front of the controller drive tray NOTE If the E2660 controller drive tray needs to have drives installed refer to Steps to Install Drives on the DE6600 Drive Tray on page 77 ...

Page 53: ...y Things to Know Host Channels ATTENTION Possible hardware damage To prevent electrostatic discharge damage to the tray use proper antistatic protection when you handle tray components Each controller has from two to six host ports Two of the host ports are standard and support 6 Gb s SAS data rates Two to four of the host ports are optional and if present are located on a host interface card HIC ...

Page 54: ...xamples in the following section for example cabling patterns 1 Perform one of these actions You are using an FC HIC Go to step 2 You are using either a SAS or an iSCSI HIC Go to step 4 Connections for iSCSI use copper cables with RJ 45 connectors and do not require SFP transceivers Connections for SAS use SFF 8088 connectors 2 Make sure that the appropriate type of SFP transceiver is inserted int...

Page 55: ...ions are required do not connect a host to the highest numbered host channel because this channel is used for the mirrored communications Figure 41 Direct Topology One Host Connected to a Single Controller 1 Host 2 HBA 1 or NIC 1 3 HBA 2 or NIC 2 4 Host Port 1 5 Host Port 2 6 Controller A ...

Page 56: ...48 E2600 Controller Drive Tray Installation Guide Figure 42 Direct Topology Two Hosts Connected to a Single Controller 1 Host 2 HBA 1 or NIC 1 3 HBA 2 or NIC 2 4 Host Port 1 5 Host Port 2 6 Controller A ...

Page 57: ...troller Drive Tray Installation Guide 49 Figure 43 Switch Topology Two Hosts Connected to a Single Controller Through a Switch 1 Host 2 HBA 1 or NIC 1 3 HBA 2 or NIC 2 4 Host Port 1 5 Host Port 2 6 Controller A ...

Page 58: ...0 Controller Drive Tray Installation Guide Figure 44 Direct Topology One Host and a Dual Controller Drive Tray 1 Host 2 HBA 1 or NIC 1 3 HBA 2 or NIC 2 4 Host Port 1 5 Host Port 2 6 Controller A 7 Controller B ...

Page 59: ...ve Tray Installation Guide 51 Figure 45 Direct Topology Two Hosts and a Dual Controller Drive Tray for Maximum Redundancy 1 Hosts 2 HBA 1 or NIC 1 3 HBA 2 or NIC 2 4 Host Port 1 5 Host Port 2 6 Controller A 7 Controller B ...

Page 60: ... Controller Drive Tray Installation Guide Figure 46 Mixed Topology Two Hosts and a Dual Controller Drive Tray 1 Hosts 2 HBA 1 or NIC 1 3 HBA 2 or NIC 2 4 Host Port 1 5 Host Port 2 6 Controller A 7 Controller B ...

Page 61: ...the cable either into an HBA in the host direct topology or into a switch fabric topology 6 Affix a label to each end of the cable with this information A label is very important if you need to disconnect cables to service a controller Include this information on the labels The host name and the HBA port for direct topology The switch name and the port for fabric topology The controller ID for exa...

Page 62: ...on Assume that a cable is connected between port 1 in HBA 1 of a host named Engineering and host channel 1 of controller A A label abbreviation could be as follows 7 Repeat step 2 on page 46 through step 6 on page 53 for each controller and host channel that you intend to use ...

Page 63: ...rays around the controller drive tray Keep as much weight as possible in the bottom half of the cabinet NOTE Refer to the Storage System Site Preparation Guide on the SANtricity ES Storage Manager Installation DVD for important considerations about cabinet installation Procedure Installing the DE1600 Drive Trays and the DE5600 Drive Trays WARNING W08 Risk of bodily injury Two persons are required ...

Page 64: ...nts 1 76 cm 30 in clearance in front of the cabinet 2 61 cm 24 in clearance behind the cabinet Figure 49 DE5600 Drive Tray Airflow and Clearance Requirements 1 76 cm 30 in clearance in front of the cabinet 2 61 cm 24 in clearance behind the cabinet NOTE Fans pull air through the tray from front to back across the drives 2 Lower the feet on the cabinet to keep the cabinet from moving 3 Remove the d...

Page 65: ...y 3 Clearance Above and Below the Existing Tray 4 Screws for Securing the Mounting Rail to the Cabinet Front and Rear 5 Industry Standard Cabinet If you are installing the mounting rails above an existing tray position the mounting rails directly above the tray If you are installing the mounting rails below an existing tray allow 8 8 cm 3 5 in vertical clearance for the drive tray ...

Page 66: ...at the mounting rail can extend or contract as needed Figure 51 Attaching the Mounting Rails to the Cabinet 1 Cabinet Mounting Holes 2 Adjustment Screws for Locking the Mounting Rail Length 3 Mounting Rails 4 Clip for Securing the Rear of the Drive Tray b Place the mounting rail inside the cabinet and extend the mounting rail until the flanges on the mounting rail touch the inside of the cabinet ...

Page 67: ...he alignment spacers help position and hold the mounting rail Figure 52 Alignment Spacers on the Mounting Rail 1 Alignment Spacers d Insert one M5 screw through the front of the cabinet and into the top captured nut in the mounting rail Tighten the screw e Insert two M5 screws through the rear of the cabinet and into the captured nuts in the rear flange in the mounting rail Tighten the screws f Ti...

Page 68: ... front flanges of the drive tray align with the mounting holes on the front of the mounting rails Refer to item 4 in the following figure The rear edge of the drive tray sheet metal fits into the clip on the mounting rail The holes in the drive tray sheet metal for the rear hold down screws align with the captured nuts in the side of the mounting rails Figure 53 Sliding the Drive Tray into the Cli...

Page 69: ...drive tray to the mounting rails a Insert one M5 screw through the bottom hole of a flange on the drive tray so that the screw goes through the cabinet rail and engages the bottom captured nut in the mounting rail Tighten the screw b Repeat substep a for the second flange Figure 54 Attaching the Front of the DE1600 Drive Tray 1 Screws for Securing the Front of the Drive Tray ...

Page 70: ...ews for Securing the Front of the Drive Tray 8 Secure the side of the drive tray to the mounting rails by performing these substeps a Insert one M5 screw through the side sheet metal of the drive tray into the captured nut on the side of the mounting rail Tighten the screw b Repeat substep a for the other side ...

Page 71: ...ay a Put the top of the end cap on the hinge tab that is part of the drive tray mounting flange b Gently press on the bottom of the end cap until it snaps into place over the retainer on the bottom of the drive tray mounting flange Figure 56 Attaching the End Caps to the DE1600 Drive Tray 1 Hinge Tab 2 Retainer ...

Page 72: ...his procedure to install the drives If your drive tray already has drives installed you can skip this step and go to either Things to Know AC Power Cords on page 93 or Things to Know DC Power Cords on page 93 ATTENTION Risk of equipment malfunction To avoid exceeding the functional and environmental limits install only drives that have been provided or approved by the original manufacturer Drives ...

Page 73: ...ormation that depends upon the correct sequence of the drives in the tray 1 Beginning with the first drive slot in the upper left side of the drive tray place the drive on the slot guides and slide the drive all the way into the slot 2 Push the drive handle to the right DE1600 drive tray or down DE5600 drive tray to lock the drive securely in place Figure 58 Installing a Drive in the DE1600 Drive ...

Page 74: ...net For more information refer to the SANtricity ES Storage Manager Installation DVD Special site preparation is not required for any of these drive trays beyond what is normally found in a computer lab environment The power supplies meet standard voltage requirements for both domestic and worldwide operation Take these precautions Install the drive trays in locations within the cabinet that let y...

Page 75: ...ou are mounting the DE6600 drive tray in a cabinet with square holes use the eight shoulder washers in the rail kit to align the screws in the holes see step 4 on page 69 through step 7 on page 71 1 Make sure that the cabinet is in the final location Make sure that you meet the clearance requirements shown in the following figure Figure 60 Drive Tray Airflow and Clearance Requirements 1 81 cm 32 i...

Page 76: ...ximately 56 7 kg 125 lb Three persons are required to safely move an empty tray If the tray is populated with components a mechanized lift is required to safely move the tray 3 With the help of at least two other persons remove the drive tray and all of the contents from the shipping carton using the four drive tray handles two to a side as shown in the following figure Set the drive tray aside Fi...

Page 77: ...osition the mounting rails directly above the tray If you are installing the mounting rails below an existing tray allow 17 8 cm 7 in vertical clearance for a DE6600 drive tray 5 To attach the mounting rails to the cabinet do one of the following If you are using the long fixed size mounting rails go to step 6 If you are using the shorter adjustable mounting rails go to step 7 6 To attach the long...

Page 78: ... the rail and prevents you from sliding the drive tray onto the rails c Place the mounting rail inside the cabinet and extend the mounting rail until the flanges on the mounting rail touch the inside of the cabinet d Insert one M5 screw through the front of the cabinet and screw it into the top captured nut in the mounting rail e Insert two M5 screws through the rear of the cabinet and screw them ...

Page 79: ...tend or contract as needed see the figure Short Adjustable Mounting Rail Left Side b Place the mounting rail inside the cabinet and extend the mounting rail until the flanges on the mounting rail touch the inside of the cabinet see the figure Short Adjustable Mounting Rail Attached to the Cabinet on page 72 c Insert one M5 screw through the front of the cabinet and screw it into the top captured n...

Page 80: ...tion Guide g Insert one M5 screw through the front of the mounting rail This screw attaches the drive tray to the cabinet Figure 65 Short Adjustable Mounting Rail Attached to the Cabinet 1 Cabinet Mounting Holes on the Front EIA Support Rail ...

Page 81: ...f the tray is populated with components a mechanized lift is required to safely move the tray 10 After the controller drive tray is correctly aligned remove the enclosure lift handles as shown in the figure Removing an Enclosure Lift Handle from the Drive Tray a Use your thumb to unlatch and remove the rear enclosure lift handles two to a side b Use the front enclosure lift handles to slide the dr...

Page 82: ...rive tray to the mounting rails a Insert two M5 screws through the bottom holes of a flange on the drive tray so that the screws go through the EIA support rail and engage the bottom captured nuts in the mounting rail Tighten the screws b Repeat substep a for the second flange Figure 67 Attaching the Front of the Drive Tray 1 Four Screws for Securing the Front of the Drive Tray ...

Page 83: ...m the drive tray by pressing on the tab holding the fan canister handle in place and then pulling the fan canister toward you Figure 68 Removing the Fan Canister from the Drive Tray 1 Fan Canister Handle 13 Use the fan canister handle to pull the fan canister out of the drive tray ...

Page 84: ...sert a 10 32 screw through the side sheet metal of the drive tray into the captured nut on the side of the mounting rail Tighten the screws b Repeat substep a for the other side NOTE After the drive tray is installed there should be seven screws on each side right and left of the cabinet NOTE Make sure that each drive drawer in the drive tray is securely fastened to ensure proper air flow to the d...

Page 85: ...ve been provided or approved by the original manufacturer Drives might be shipped but not installed System integrators resellers system administrators or users can install the drives NOTE The installation order within each drawer is from left to right in rows Slots 1 4 7 and 10 must have a drive installed in these locations to make sure there is sufficient air flow to the drives To verify these sl...

Page 86: ...ve tray instructions on the Software and Documentation DVD and make sure you have the replacement drive in hand before starting the task 1 Beginning with the top drawer in the drive tray release the levers on each side of the drawer by pulling both towards the center Figure 71 Levers on the Drive Drawer 1 Levers to Release the Drive Drawer 2 Pull on the extended levers to pull the drive drawer out...

Page 87: ...ith the first drive raise the drive handle to the vertical position Figure 72 Raised Drive Handle 4 Align the two raised buttons on each side over the matching gap in the drive channel on the drawer Figure 73 Side View of Drive with Raised Handle 1 Raised Buttons ...

Page 88: ... drive drawer is fully populated Figure 75 Fully Populated Drive Drawer 7 Push the drive drawer all the way back into the drive tray closing the levers on each side of the drive drawer ATTENTION Risk of equipment malfunction Make sure you push both levers to each side so that the drive drawer is completely closed The drive drawer must be completely closed to prevent excess airflow which has the po...

Page 89: ...on such as creating new volumes Each controller has one dual ported SAS expansion connector to connect to the drive trays Figure 76 Drive Channel Ports on the E2600 Controller Drive Tray Rear View 1 Controller Canister 2 SAS Expansion Connector NOTE To maintain data access in the event of the failure of a controller an ESM or a drive channel you must connect a drive tray or a string of drive trays...

Page 90: ...3 SAS Connector 2 In 4 Expansion Connector Out 5 ESM B Things to Know Drive Tray Cabling Configurations Simplex System The following figure shows an example of cable configurations from the simplex E2600 controller drive tray to either a DE1600 drive tray or a DE5600 drive tray Use this example as a guide to connect cables in your storage array NOTE You can use either SAS IN connector on a drive t...

Page 91: ...he drive trays Use these examples as guides to connect cables in your storage array NOTE The configuration shown in the fourth image in this topic provides an example of tray loss protection With tray loss protection if one drive tray cannot be accessed you still have access to the remaining drive trays Figure 79 Controller Drive Tray Above the Drive Tray Figure 80 Controller Drive Tray Between Tw...

Page 92: ...84 E2600 Controller Drive Tray Installation Guide Figure 81 Controller Drive Tray with Three Drive Trays ...

Page 93: ... SAS EXP SAS EXP SAS IN SAS IN SAS IN SAS IN SAS EXP SAS EXP SAS IN SAS IN SAS IN SAS IN SAS EXP SAS EXP SAS IN SAS IN SAS IN SAS IN SAS EXP SAS EXP SAS IN SAS IN SAS IN SAS IN SAS EXP SAS EXP SAS IN SAS IN SAS IN SAS IN SAS EXP SAS EXP SAS IN SAS IN SAS IN SAS IN You can use up to eight trays in a storage array Procedure Connecting the DE1600 Drive Trays and the DE5600 Drive Trays 1 Use the follo...

Page 94: ...re shows these arrows on an ESM If the cable is connected either between the two left center ESM connectors or between two upper right ESM connectors communication between the two drive trays is lost NOTE It does not matter which of the two left center ESM connectors you use to connect to the expansion connector on the far right side Figure 83 Connecting a Cable from One ESM to a Second ESM Yes Go...

Page 95: ...ese handles on each controller is located next to a host connector The close spacing between the handle and the host connector might make it difficult to remove a cable that is attached to the host connector If this problem occurs use a flat blade screwdriver to push in the release component on the cable connector The E2660 controller drive tray supports the DE6600 drive tray for expansion The max...

Page 96: ...he exposed OSP cabling The addition of Primary Protectors is not sufficient protection in order to connect these interfaces metallically to OSP wiring The cable shall be shielded Twisted Pair STP and must be grounded at both ends to meet the intra building lightning requirements from section 4 6 9 2 of GR 1089 CORE Issue 5 Things to Know Drive Trays with the E2660 Controller Drive Tray Each DE6600...

Page 97: ...here is a black plastic plug in the SAS expansion connector of the controller remove it 3 Insert one end of the cable into the SAS expansion connector on the controller in slot A in the controller drive tray 4 Insert the other end of the cable into the connector with an up arrow on the ESM in slot A in the drive tray 5 Are you adding more drive trays Number of Drive Trays that You Plan to Connect ...

Page 98: ... in the next drive tray insert the other end of the cable into one of the connectors in the left center of the ESM 8 Repeat step 6 through step 7 for each drive tray that you intend to add to the storage array 9 To each end of the cables attach a label with this information The controller ID for example controller A The ESM ID for example ESM A The ESM connector In or Out The drive tray ID For exa...

Page 99: ...ht be at risk of being accessed from outside of your network These Ethernet connections are intended for out of band management and have nothing to do with the iSCSI host interface cards HICs whether 1Gb s or 10Gb s Ethernet port 2 on each controller is reserved for access by your Technical Support representative In limited situations in which the storage management station is connected directly t...

Page 100: ...E2600 Controller Drive Tray Installation Guide 92 ...

Page 101: ... a drive tray with one of its power supplies removed You can use the power cords shipped with the controller drive tray with typical outlets used in the destination country such as a wall receptacle or an uninterruptible power supply UPS These power cords however are not intended for use in most EIA compliant cabinets Things to Know DC Power Cords If your controller drive trays have the DC power o...

Page 102: ...ore turning off any power switches on a DC powered controller drive tray you must disconnect the two pole 20 amp circuit breaker 1 Disconnect the two pole 20 amp circuit breaker for the storage array 2 Make sure that all of the DC power switches on the DC powered controller drive tray are turned off 3 Make sure that all of the DC power switches on the DC powered drive trays are turned off 4 Connec...

Page 103: ...ve tray NOTE While the power is being applied to the trays the LEDs on the front and the rear of the trays come on and go off intermittently 1 Turn on both Power switches on each drive tray that is attached to the controller drive tray Depending on your configuration it can take several minutes for each drive tray to complete the power on process NOTE Before you go to step 2 check the LEDs on the ...

Page 104: ...ED Table 15 LEDs on the Left End Cap Location LED Color On Off 1 Controller Drive Tray Locate White Identifies a controller drive tray that you are trying to find Normal status 2 Service Action Required Amber A component within the controller drive tray needs attention Normal status 3 Controller Drive Tray Over Temperature Amber The temperature of the controller drive tray has reached an unsafe le...

Page 105: ... LED Color On Blinking Off 1 Drive Power Green The power is turned on and the drive is operating normally Drive I O activity is taking place The power is turned off 2 Drive Service Action Required Amber An error has occurred Normal status 3 Drive Service Action Allowed Blue The drive canister can be removed safely from the controller drive tray The drive canister cannot be removed safely from the ...

Page 106: ...wed LED 12 Controller Service Action Required LED 13 Cache Active LED 14 Seven Segment Tray ID Table 18 LEDs on the Controller Canister Main Faceplate Drive State Drive Power LED Green Drive Service Action Required LED Amber Power is not applied Off Off Normal operation The power is turned on but drive I O activity is not occurring On Off Normal operation Drive I O activity is occurring Blinking O...

Page 107: ... Action Allowed LED Green At least one of the four PHYs in the Host IN port is working and a link exists to the device connected to the IN port connector A link error has occurred 9 Battery Service Action Required LED Amber The battery in the controller canister has failed Normal status 10 Battery Charging LED Green The battery is fully charged The LED blinks when the battery is charging The contr...

Page 108: ...rd Link 6 Active LED 9 Expansion Fault LED 10 Expansion Active LED Table 19 LEDs on the Controller Canister Host Interface Card Subplates Location LED Color On Off 1 Host Interface Card Link 3 Up LED Green The Ethernet link has auto negotiated to 1 Gb s The Ethernet link is down or does not auto negotiate to 1 Gb s 2 Host Interface Card Link 3 Active LED Green The link is up LED blinks when there ...

Page 109: ...o negotiate to 1 Gb s 8 Host Interface Card Link 6 Active LED Green The link is up LED blinks when there is activity The link is not active 9 Expansion Fault LED Amber At least one of the four PHY is working but another PHY cannot establish the same link to the device connected to the Expansion OUT connector Normal status 10 Expansion Active LED Green At least one of the four PHYs in the OUT conne...

Page 110: ...ntroller drive tray is in Standby mode and DC power is not available The controller drive tray is not in Standby mode and DC power is available 2 Power Fan DC Power Green DC power from the power fan canister is available DC power from the power fan canister is not available 3 Power Fan Service Action Allowed Blue The power fan canister can be removed safely from the controller drive tray The power...

Page 111: ...ion Required LED 10 Battery Charging LED 11 Controller Service Action Allowed LED 12 Controller Service Action Required LED 13 Cache Active LED 14 Seven Segment Tray ID Location LED Color On Off 1 Controller Drive Tray Locate White Identifies a controller drive tray that you are trying to find Normal status 2 Service Action Required Amber A component within the controller drive tray needs attentio...

Page 112: ...Ys in the Host IN port is working and a link exists to the device connected to the IN port connector A link error has occurred 7 Host Link 2 Service Action Required LED Amber At least one of the four PHYs is working but another PHY cannot establish the same link to the device connected to the Host IN port connector No link error has occurred 8 Host Link 2 Service Action Allowed LED Green At least ...

Page 113: ...face Card Link 6 Up LED 8 Host Interface Card Link 6 Active LED 9 Expansion Fault LED 10 Expansion Active LED Table 23 LEDs on the Controller Canister Host Interface Card Subplates 12 Controller Service Action Required LED Amber A fault exists within the controller canister Normal status 13 Cache Active LED Green Cache is active Cache is inactive or the controller canister has been removed from th...

Page 114: ...p LED Green The Ethernet link has auto negotiated to 1 Gb s The Ethernet link is down or does not auto negotiate to 1 Gb s 8 Host Interface Card Link 6 Active LED Green The link is up LED blinks when there is activity The link is not active 9 Expansion Fault LED Amber At least one of the four PHY is working but another PHY cannot establish the same link to the device connected to the Expansion OUT...

Page 115: ...lable The controller drive tray is not in Standby mode and DC power is available 2 Power Fan DC Power Green DC power from the power fan canister is available DC power from the power fan canister is not available 3 Power Fan Service Action Allowed Blue The power fan canister can be removed safely from the controller drive tray The power fan canister cannot be removed safely from the controller driv...

Page 116: ...ally Blinking The battery is charging Service Action Allowed Drive left LED no symbol Power fan Controller Battery On You can remove the canister safely See Things to Know Service Action Allowed LEDs Service Action Required Fault Drive On When the drive tray LED is on the cable is attached and at least one lane has a link up status but at least one lane has a link down status Off One of the follow...

Page 117: ... Off The temperature of the drive tray is within operational range Standby Power Front bezel on the controller drive tray On The controller tray is in standby mode and the main DC power is off Off The controller drive tray is not in standby mode and the main DC power is on Seven Segment ID Diagnostic Display Controller The tray ID or a diagnostic code is indicated see Things to Know Dynamic Displa...

Page 118: ...ns that dictate a canister should not be removed the Service Action Allowed LED stays off The Service Action Allowed LED automatically comes on or goes off as conditions change In most cases the Service Action Allowed LED comes on when the Service Action Required Fault LED comes on for a canister AC power Power fan NOTE The LED is directly above or below the AC power switch and the AC power connec...

Page 119: ... Action Required LED 3 Drive Tray Over Temperature LED 4 Drive Tray Power LED 5 Drive Tray Standby Power LED Table 26 LEDs on the Left End Cap Location LED Color On Off 1 Drive Tray Locate White Identifies a drive tray that you are trying to find Normal status 2 Drive Tray Service Action Required Amber A component within the drive tray needs attention Normal status 3 Drive Tray Over Temperature Am...

Page 120: ...ly Drive I O activity is taking place The power is turned off 2 Drive Service Action Required Amber An error has occurred Normal status 3 Drive Service Action Allowed Blue The drive canister can be removed safely from the drive tray The drive canister cannot be removed safely from the drive tray Drive State Drive Power LED Green Drive Service Action Required LED Amber Power is not applied Off Off ...

Page 121: ... Tray ID Table 29 LEDs on the ESM Canister Service action required A fault condition exists and the drive is offline On On Location LED Color On Off 1 Host Link 1 Fault Amber At least one PHY of the four connectors is working but another PHY cannot establish the same link to the device connected to the Host IN port connector No link error has occurred 2 Host Link 1 Active Green At least one of the...

Page 122: ...establish the same link to the Expansion Out connector Normal status 8 ESM Expansion Link Active Green At least one of the four PHYs in the Out port is working and a link exists to the device connected to the Expansion Out connector A link error has occurred 9 ESM Service Action Allowed Blue The ESM can be removed safely from the drive tray The ESM cannot be removed safely from the drive tray 10 E...

Page 123: ...r Location LED Color On Off 1 Standby Power Green The drive tray is in Standby mode and DC power is not available The drive tray is not in Standby mode and DC power is available 2 Power Fan DC Power Green DC power from the power fan canister is available DC power from the power fan canister is not available 3 Power Fan Service Action Allowed Blue The power fan canister can be removed safely from t...

Page 124: ...p Location LED Color On Off 1 Drive Tray Locate White Identifies a drive tray that you are trying to find Normal status 2 Drive Tray Service Action Required Amber A component within the drive tray needs attention Normal status 3 Drive Tray Over Temperature Amber The temperature of the drive tray has reached an unsafe level Normal status 4 Drive Tray Power Green Power is present Power is not presen...

Page 125: ... has occurred 2 ESM Link Port 1A Green The link is up A link error has occurred 3 ESM Link Port 1B Bypass Green The link is up A link error has occurred 4 ESM Link Fault Port 1B Amber A link error has occurred No link error has occurred 5 ESM Service Action Allowed Blue The ESM can be removed safely from the drive tray The ESM cannot be removed safely from the drive tray 6 ESM Service Action Requi...

Page 126: ...y Power Green The drive tray is in Standby mode and DC power is not available The drive tray is not in Standby mode and DC power is available 2 Power DC Power Green DC power from the power canister is available DC power from the power canister is not available 3 Power Service Action Allowed Blue The power canister can be removed safely from the drive tray The power canister cannot be removed safel...

Page 127: ...LEDs on the Drawer Location LED Color On Off 1 Fan Power Green Power from the fan canister is available Power to the fan customer replaceable unit CRU is available 2 Fan Service Action Required Amber A fault exists within the fan canister Normal status 3 Fan Service Action Allowed Blue The fan canister can be removed safely from the drive tray The fan canister cannot be removed safely from the dri...

Page 128: ...or drives 1 through 12 in the drive drawer Green The power is turned on and the drive is operating normally Drive I O activity is taking place The power is turned off Drive State Drive Activity LED Green Drive Service Action Required LED Amber Power is not applied Off Off Normal operation The power is turned on but drive I O activity is not occurring On Off Normal operation Drive I O activity is o...

Page 129: ... Action Required LED Amber Power is not applied Off Off Normal operation The power is turned on but drive I O activity is not occurring On Off Normal operation Drive I O activity is occurring Blinking Off Service action required A fault condition exists and the drive is offline On On LED Symbol Location General Behavior Power Drive tray ESM canister Power fan canister On Power is applied to the dr...

Page 130: ...the power fan canister and the drive are operating normally The drive has an LED but no symbol AC Power ESM canister Power fan canister On AC power is present Off AC power is not present DC Power Power fan canister On Regulated DC power from the power canister and the fan canister is present Off Regulated DC power from the power fan canister is not present Link Service Action Required Fault ESM ca...

Page 131: ...d in this Controller State Code Power on Diagnostics Suspended Operational Processor DIMM Px yes yes no Cache DIMM CX yes yes no Host Card HX yes yes yes Flash Drive FX no no yes Battery no no no Category delimiter The separator between category detail code pairs is used when more than one category detail code pair exists in the sequence End of sequence delimiter The end of sequence delimiter is a...

Page 132: ... the Controller Drive Tray on page 125 and Things to Know Seven Segment Display Use Cases on page 126 Category delimiter dash The separator between category detail code pairs is used when more than one category detail code pair exists in the sequence End of sequence delimiter blank SeeNote 4 The end of sequence delimiter is automatically inserted by the hardware at the end of a code sequence Notes...

Page 133: ...ed which results in a suspended controller state L3 A persistent hardware error has occurred which results in a suspended controller state L4 A persistent data protection error has occurred which results in a suspended controller state L5 An auto code synchronization ACS failure has been detected which results in a suspended controller state L6 An unsupported host interface card has been detected ...

Page 134: ...oller state Lr A non field replaceable unit has been replaced which results in a suspended controller state LU The start of day SOD reboot limit has been exceeded which results in a suspended controller state Use Cases Repeating Sequence Use Case Controller power on Normal power on controller insertion Controller inserted while held in reset SE 88 blank Use Cases Repeating Sequence Use Case Operat...

Page 135: ...devices have been detected SE LC dash CF Fx blank Use Cases Repeating Sequence Use Case Controller is suspended and there are no other errors to report All lockdown conditions OH Lx blank Use Cases Repeating Sequence Use Case The controller is suspended because of component errors Persistent processor DIMM error correcting code ECC errors have been detected OE L2 dash CF CX blank Persistent cache ...

Page 136: ...gnostic errors A cache memory diagnostic failure has been detected dE L2 dash CF Cx blank A base controller diagnostic failure has been detected dE L3 dash CF b1 blank A base controller IOC diagnostic failure has been detected dE L3 dash CF b2 blank Diagnostic Code Description The firmware is booting 8 8 or 88 This ESM is being held in reset by another ESM AA ESM A firmware is in the process of bo...

Page 137: ...ttempts has been exceeded H4 This ESM cannot communicate with the alternate ESM H5 A midplane harness failure has been detected in the drive tray H6 An ESM firmware failure has been detected H8 SFP transceivers are present in currently unsupported ESM slots either 2A or 2B Secondary trunking SFP transceiver slots 2A and 2B are not supported Look for the SFP transceiver with the blinking amber LED ...

Page 138: ...130 E2600 Controller Drive Tray Installation Guide ...

Page 139: ...correct interference caused by such unauthorized modification substitution or attachment Laser Products Statement This equipment uses Small Form factor Pluggable SFP optical transceivers which are unmodified Class 1 laser products pursuant to 21 CFR Subchapter J Section 1040 10 All optical transceivers used with this product are required to be 21 CFR certified Class 1 laser products For outside th...

Page 140: ...ent This Class A digital apparatus meets all requirements of the Canadian Interference Causing Equipment Regulations Cet appareil numérique de la classé A respecte toutes les exigences du Règlement sure le matèriel brouilleur du Canada ...

Page 141: ......

Page 142: ...Copyright 2012 NetApp Inc All rights reserved ...

Reviews:

No comments

Brands by name

Popular brands

Load more brands