background image

144

template, port numbering

 

133

terminal emulator, configuring

 

42

text symbols

 

10

V

verifying IP connectivity

 

117

vibration, requirement

 

107

viewing available LUNs

 

118

voltage compliance rating

 

131

W

warning

rack stability

 

11

warnings

lasers, radiation

 

123

waste electrical and electronic equipment directive

 

123

web sites

HP storage

 

11

HP Subscriber’s choice

 

11

weight, DC Director

 

102

WWN bezel

 

81

LEDs

 

82

replacing

 

81

WWN card

 

81

error messages

 

60

LEDs

 

82

replacing

 

81

status

 

60

82

Summary of Contents for A7990A - StorageWorks SAN Director 4/16 Blade Switch

Page 1: ...HP StorageWorks DC SAN Backbone Director hardware reference guide Part number 5697 7346 First edition September 2008 ...

Page 2: ...opyright No part of this document may be photocopied reproduced or translated into another language without the prior written consent of Hewlett Packard The information is provided as is without warranty of any kind and is subject to change without notice The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services Nothin...

Page 3: ...me and items required 23 Site preparation unpacking the DC Director 24 Items included with the DC Director 25 Installing the DC Director in the 14U Rack Mount Kit 26 14U Rack Mount Kit parts list 26 Attaching the shelf brackets 28 Removing the chassis door 30 Installing the chassis in the cabinet 30 Replacing the chassis door 32 Powering on the DC Director 34 Port numbering 34 Managing cables 35 U...

Page 4: ...tch blade CR8 76 How to determine whether or not to replace a CP blade 76 Removing a core switch blade CR8 77 Replacing a core switch blade CR8 78 Power supply 78 Removing a power supply 79 Replacing a power supply 79 Blower assembly 80 Removing a blower assembly 80 Replacing a blower assembly 80 WWN bezel logo plate and WWN card 81 How to determine whether or not to replace a WWN card 81 Removing...

Page 5: ... the Director 113 Cabling the FC4 16IP Blade 113 Recommendations for cable management 114 Configuring the FC4 16IP blade 115 Activating iSCSI 115 Enabling the persistently disabled ports 115 Configuring the IP interface 116 Creating the IP interface of the GbE port 116 Adding IP routes on a GbE port Optional 116 Verifying IP connectivity 117 Adding ARP entries 117 Configuring an iSCSI interface 11...

Page 6: ...rack rails 32 8 DC Director door aligned with chassis 33 9 Inserting DC Director door on chassis ball studs 33 10 ICL connectors on CR8 blade 36 11 Inter chassis link ICL connections configuration 1 37 12 Inter chassis link ICL connections configuration 2 38 13 Inter chassis link ICL connections configuration 3 39 14 Inter chassis link ICL connections configuration 4 40 15 FC8 16 Director blade 48...

Page 7: ...descriptions 59 12 Blower assembly LED descriptions 60 13 Messages that may indicate WWN card failure 60 14 WWN bezel LED descriptions 61 15 WWN LED patterns 82 16 Commands identifying the WWN card status 82 17 WWN card related system log messages 82 18 Critical information checklist 87 19 Cable routing table for DC Director 48 ports shown 98 20 System architecture 101 21 System size and weight 10...

Page 8: ...8 ...

Page 9: ...nce This guide is intended for system administrators and technicians with knowledge of Storage area networks SANs HP StorageWorks Fibre Channel enterprise class switches Directors Related documentation The following documents provide related information HP StorageWorks Fabric OS 6 x release notes HP StorageWorks Fabbric OS 6 x administrator guide You can find these and other documents from the Man...

Page 10: ...le 1 Document conventions Convention Element Blue text Table 1 Cross reference links and e mail addresses Blue underlined text http www hp com Website addresses Bold text Key that are pressed Text typed into a GUI element such as into a box GUI elements that are clicked or selected such as menu and list items buttons tabs and check boxes Italics text Text emphasis Monospace text File and directory...

Page 11: ... repair HP customer self repair CSR programs allow you to repair your StorageWorks product If a CSR part needs replacing HP ships the part directly to you so that you can install it at your convenience Some parts do not qualify for CSR Your HP authorized service provider will determine whether a repair can be accomplished by CSR For more information about CSR contact your local service provider Fo...

Page 12: ...ttp www hp com support downloads Documentation feedback HP welcomes your feedback To make comments and suggestions about product documentation please send a message to storagedocs feedback hp com All submissions become the property of HP ...

Page 13: ...ering investment protection interoperability and fabric based intelligence advantages NOTE Throughout this document the HP StorageWorks DC SAN Backbone Director is referenced using its abbreviated name DC Director Features Key features of the DC Director include Up to 384 ports in a single chassis providing high port density for a scalable solution to drive high port count SAN configurations Suppo...

Page 14: ...4 18i 18 port 16 FC 2 GbE up to four blades per chassis supporting FCR Services and FCIP FC4 16IP 16 port 8 FC 8 GbE up to four blades per chassis supporting iSCSI bridging Modular hot swappable field replaceable units FRUs Three blower assemblies Up to four power supplies At 240 VAC two power supplies are required to provide redundancy although the DC Director can be fully powered with a single p...

Page 15: ...e non port non cable side to the port cable side and out the exhaust vent Figure 1 shows a sample configuration of the port side of the DC Director Figure 1 Port side of the DC Director sample configuration 1 Exhaust Vent 4 FC8 48 Director blade 2 Core switch blade CR8 5 Cable management comb 3 CP blade CP8 25369a 1 2 3 4 5 ...

Page 16: ...he DC Director Figure 2 displays a sample configuration of the non port side view of the DC Director Figure 2 Non port side of the DC Director sample configuration 1 WWN bezel logo plate 3 Blower assembly 2 Power supply 25370a 2 3 1 ...

Page 17: ...ane connections for the storage server blade This core switch blade is compatible only with the DC Director 16 port 8 Gbps Director blade FC8 16 A 16 port Director blade supporting 1 2 4 and 8Gbps port speeds This Director blade is compatible only with the 4 256 SAN Director or DC Director CP blades 32 port 8 Gbps Director blade FC8 32 A 32 port Director blade supporting 1 2 4 and 8Gbps port speed...

Page 18: ...s simplify troubleshooting The standby CP Blade monitors diagnostics to ensure it is operational in case a failover is required Bus monitoring and control of blades and other FRUs Serviceability The DC Director provides the following features to enhance and ensure serviceability Modular design with hot swappable components Flash memory that stores two firmware images per CP Blade USB port on CP bl...

Page 19: ...Mv3 When SNMP devices send SNMP messages to a management console running SAN management software the information is stored in a Management Table 3 Security features Security Features DH CHAP Login banner SSHv2 using AES 3DES RSA Monitoring of attempted security breaches via audit logging HTTPS using AES Monitoring of attempted security breaches via Fabric Watch Security Class SNPMv3 FC security po...

Page 20: ...ion monitoring management and statistics collection Supported in XP Storage array environments only Note Contact your local HP representative for information on exact Fabric OS versions supported T4401A HP StorageWorks MP Blade Performance Extension LTU Activates the high performance extension services for either IP or FC connectivity in the B Series MP Router Blade FR4 18i IP and FC extension ser...

Page 21: ... blade FC8 48 AK860A HP StorageWorks SAN Director 6 Port 10Gb FC blade FC10 6 AK861A HP StorageWorks B Series Multi protocol MP Router blade FR4 18i AG461A Optical transceivers HP 4Gb Short Wave B Series FC SFP 1 Pack AJ715A HP 4Gb Long Wave B Series FC SFP 1 Pack 10km AK870A HP 8Gb Short Wave B Series FC SFP 1 Pack AJ716A HP BLc 10Gb SR XFP Opt Kit 443756 B21 HP BLc 10Gb LR XFP Opt Kit 443757 B21...

Page 22: ...P 2 m Multi mode OM2 LC SC Fiber Channel Cable 221691 B21 HP 5 m Multi mode OM2 LC SC Fiber Channel Cable 221691 B22 HP 15 m Multi mode OM2 LC SC Fibre Channel Cable 221691 B23 HP 30 m Multi mode OM2 LC SC Fibre Channel Cable 221691 B26 HP 50 m Multi mode OM2 LC SC Fibre Channel Cable 221691 B27 Table 5 DC Director orderable hardware continued Accessory Part number ...

Page 23: ...ewer ports require less time These time estimates assume a prepared installation site and appropriate power and network connectivity Table 6 Installation tasks time and items required Installation task Time estimate Items required Site preparation and unpacking DC Director 30 minutes 1 2 in socket wrench to remove pallet bolts 2 Phillips screwdriver for cable management comb Pallet jack Hydraulic ...

Page 24: ...ical rating on the chassis nameplate Location close to the chassis and easily accessible Grounded outlets installed by a licensed electrician compatible with the power cords TIP To maximize fault tolerance connect each power cord to a separate power source 5 Ensure that the air intake and exhaust vents have a minimum of 2 in of airspace 6 Ensure that the air temperature on the air intake side is l...

Page 25: ...ts holding the DC Director to the pallet and remove the brackets 15 Gently slide the chassis onto the final installation surface ensuring that it remains supported during the transfer 16 Ensure that the chassis is oriented so that the non port side has access to intake air cool 17 Reinstall the cable management comb See Replacing a cable management comb on page 65 18 Reinstall the chassis door See...

Page 26: ...he hardware provided in the 4U rack mount kit that ships with your DC Director Table 7 Items supplied with the 14U rack mount kit DC Director Item Description Quantity For all types of installations A Left rack mount shelf bracket rail brackets may differ from the illustration 1 B Right rack mount shelf bracket rail brackets may differ from the illustration 1 C 10 32 x 5 8 inch 1 58 cm panhead Phi...

Page 27: ...DC SAN Backbone Director hardware reference guide 27 I Figure 3 14U Rack Mount Kit contents 26419a ...

Page 28: ...cabinet rails adjustable ends installed on the side of the rack on the intake air aisle 1 Locate the shelf brackets in the 14U Rack Mount Kit 2 Locate and loosen the adjusting screws on the brackets Figure 4 Items A and B to allow for adjustment to cabinet depth Figure 4 Left and right shelf brackets installed on rails ...

Page 29: ... cm kgs For rails with square holes Position the left and right rack mount shelf brackets Items A and B and attach to the cabinet rails Use eight screws with and alignment washers per bracket four on each end Tighten the screws to a torque of 80 in lbs 92 cm kgs Figure 5 Shelf bracket and clip or retainer nut placement on cabinet rails NOTE Standard EIA rails have holes in sets of three spaces bet...

Page 30: ...ing longer screws can damage the chassis Removing the chassis door See Removing the chassis door on page 30 to uninstall the chassis door Installing the chassis in the cabinet To install the DC Director in the cabinet CAUTION A fully populated DC Director eight FC8 48 port cards 384 ports weighs approximately 104 kg 228 lbs and requires a hydraulic or assisted lift to install it 1 Use a lift to ra...

Page 31: ...DC SAN Backbone Director hardware reference guide 31 Figure 6 Positioning the DC Director for installation in a cabinet 26422 1 2 1 Rack cabinet shown without sheet metal 2 Air instake side of chassis ...

Page 32: ...hinged It installs on the chassis by snapping the four ball studs located on the chassis into the receiving hardware on the door 1 Align the eight ball studs on the chassis with mounting hardware locations on the chassis door while pushing the door straight into the chassis 2 Snap the door into place over the ball studs Figure 8 and Figure 9 26423a ...

Page 33: ...DC Director door aligned with chassis Figure 9 Inserting DC Director door on chassis ball studs 26424a 26425a 1 2 3 Inside view of door Outside view of door 1 Ball stud on chassis outside view of door 2 Mounting hardware 3 Ball stud on chassis inside view of door ...

Page 34: ...ive reboots on the DC Director using the diagDisablePost command NOTE To prevent a potential IP address conflict do not connect the DC Director to the network until the IP addresses are configured see Log In and Configuration on page 21 Port numbering The DC Director uses the following port numbering method see Port numbering templates on page 133 IMPORTANT Slots are numbered 1 through 12 from lef...

Page 35: ...cate which ports can be used in the same ISL Trunking group eight ports marked with solid black ovals alternate with eight ports marked with oval outlines See Table 26 on page 108 for a listing of supported cable speeds and distances Using the optional HP StorageWorks DC SAN Backbone Director Inter Chassis Link ICL cable kit You can purchase the optional HP StorageWorks DC SAN Backbone Director In...

Page 36: ...36 Installation Figure 10 ICL connectors on CR8 blade 1 Status LED 4 ATTN LED 2 Power LED 5 ICL connector 3 LINK LED ICL 0 LINK ATTN ICL 0 LINK ATTN ICL 1 LINK ATTN 26378a 1 2 3 4 5 ...

Page 37: ...e guide 37 Figure 1 1 Inter chassis link ICL connections configuration 1 1 Chassis 1 5 ICL connector ICL 1 2 Core switch blades CR8 6 ICL connector ICL 0 3 Core Processor blades CP8 7 ICL cables 4 Port blades 8 Chassis 2 26372a 1 2 3 4 5 6 7 8 ...

Page 38: ...12 Inter chassis link ICL connections configuration 2 1 Chassis 1 5 ICL connector ICL 1 2 Core switch blades CR8 6 ICL connector ICL 0 3 Core processor blades CP8 7 ICL cables 4 Port blades 8 Chassis 2 26373a 1 2 3 4 5 6 7 8 ...

Page 39: ...ce guide 39 Figure 13 Inter chassis link ICL connections configuration 3 1 Chassis 1 5 ICL connector ICL 1 2 Core switch blades CR8 6 ICL connector ICL 0 3 Core processor blades CP8 7 ICL cables 4 Port blades 8 Chassis 2 26374a 1 2 3 4 5 6 7 8 ...

Page 40: ...14 Inter chassis link ICL connections configuration 4 1 Chassis 1 5 ICL connector ICL 1 2 Core switch blades CR8 6 ICL connector ICL 0 3 Core processor blades CP8 7 ICL cables 4 Port blades 8 Chassis 2 26375a 1 2 3 4 5 6 7 8 ...

Page 41: ...e configuration information is mirrored to the standby CP blade which allows the current configuration to remain available even if the active CP blade fails The configuration information for the DC Director is stored in the WWN cards and the flash memory of the CP blades The configuration can be backed up to a workstation uploaded and then downloaded to the active CP blade if necessary The basic s...

Page 42: ...n the user is prompted to enter new admin and user passwords Make sure to write down the new passwords and keep this information in a secure location Fabric OS swDir swDir login admin Password Please change your passwords now Use Control C to exit or press Enter key to proceed Password was not changed Will prompt again at next login until password is changed swDir admin 7 Modify passwords optional...

Page 43: ... Director External IPs must not use these addresses 3 Set up the CP0 IP address by entering the ipaddrset cp 0 command swDir admin ipAddrSet cp 0 Enter the information at the prompts 4 Set up the CP1 IP address by entering the ipaddrset cp 1 command swDir admin ipAddrSet cp 1 Enter the information at the prompts This is a sample IP configuration swDir admin ipaddrset sw 0 Ethernet IP Address 0 0 0...

Page 44: ...Tools or Fabric Manager 4 To complete any additional DC Director configuration procedures through a Telnet session log in to the DC Director by Telnet using the admin login The default password is password Customize a switch name The switch name of the DC Director can be up to 15 characters long can include alpha numeric and underscore characters and must begin with an alpha character NOTE Changin...

Page 45: ...lide in easily ensure that it is correctly oriented 3 Position a cable so that the key the ridge on one side of the cable connector is aligned with the slot in the transceiver Insert the cable into the transceiver until the latching mechanism clicks Cables are keyed so that they can be inserted in only one way If a cable does not slide in easily ensure that it is correctly oriented 4 Repeat step 1...

Page 46: ... configUpload at the prompt This command uploads the DC Director configuration to the server so that it is available for downloading to a replacement DC Director An easy way to back up configuration information is to enable logging on your Telnet session then running the following commands and saving the output in a file on a secure host configShow ipaddrShow licenseShow switchShow Keep copies of ...

Page 47: ...l steps or maintenance This chapter provides information about determining the status of each component using LEDs and CLI commands Refer to theand the for additional information Director blade To determine the status of a Director blade 1 Check the LEDs on the blade Figure 15 illustrates the FC8 16 Figure 16 illustrates the FC8 32 Figure 17 illustrates the FC8 48 Figure 18 illustrates the FC10 6 ...

Page 48: ...48 Monitor system components Figure 15 FC8 16 Director blade 1 Power LED 3 FC port 2 Status LED 4 Port status LED 1 2 3 4 ...

Page 49: ...DC SAN Backbone Director hardware reference guide 49 Figure 16 FC8 32 Director blade 1 Power LED 3 FC port 2 Status LED 4 Port status LED 25367a 1 2 3 4 ...

Page 50: ...48 Director blade 1 Power LED 3 FC port 2 Status LED 4 Port status LED 32 8 33 9 34 10 35 11 24 0 25 1 26 2 27 3 28 4 29 5 30 6 31 7 32 8 33 9 34 10 35 11 36 12 37 13 38 14 39 15 40 16 41 17 42 18 43 19 44 20 45 21 46 22 47 23 47 25366a 1 2 3 4 ...

Page 51: ...DC SAN Backbone Director hardware reference guide 51 Figure 18 FC10 6 Director blade 1 Power LED 3 FC port 2 Status LED 4 Port status LED 25368a 1 2 3 4 ...

Page 52: ...52 Monitor system components Figure 19 FC4 16IP Director blade 1 Power LED 3 FC port 2 Status LED 4 Port status LED 7 6 5 4 3 2 1 0 FC4 16IP 7 6 5 4 3 2 1 0 GE FC 7 6 5 4 FC 25358a 1 2 3 4 ...

Page 53: ...Backbone Director hardware reference guide 53 Figure 20 FR4 18i Director blade 1 Power LED 3 FC port 2 Status LED 4 Port status LED 7 6 5 4 3 2 1 0 15 14 13 12 1 1 10 9 8 FR4 18i GE1 GE0 7 6 5 4 25359a 1 2 3 4 ...

Page 54: ...ogress Allow 60 seconds for polling to complete Connected device is configured in an offline state Verify the status of the connected device Steady green Port is online connected to an external device but has no traffic No action required Slow flashing green on 1 second then off 1 second Port is online but segmented indicating a loopback plug or cable or an incompatible switch Verify that the corr...

Page 55: ...porarily change during POST and other diagnostic tests For information about how to interpret the LED patterns see Table 9 2 Check port blade status by typing slotShow and haShow Figure 21 identifies the CP8 blade Figure 21 Control Processor blade CP8 1 Status LED 5 Console port 10101 2 Power LED 6 Ethernet port MGMT 3 USB LED 7 Ethernet port SERVICE 4 USB port 8 Active CP LED 26401a 1 2 3 4 5 6 7...

Page 56: ...ng Ensure blade is firmly seated and switch has completed booting If LED remains yellow contact HP Slow flashing amber on 2 seconds then off 2 seconds CP blade is not seated correctly or is faulty Pull blade out and reseat it If LED continues to flash replace blade Fast flashing amber on 1 2 second then off 1 2 second Environmental range exceeded Check for out of bounds environmental condition and...

Page 57: ...N Backbone Director hardware reference guide 57 Figure 22 Core switch blade CR8 1 Status LED 4 ATTN LED 2 Power LED 5 ICL connector 3 LINK LED ICL 0 LINK ATTN ICL 0 LINK ATTN ICL 1 LINK ATTN 26397a 1 2 3 4 5 ...

Page 58: ... off CP blade does not have incoming power Ensure blade is firmly seated and has power Status No light LED is off CP blade is either healthy or does not have power Verify that the power LED is on Steady amber CP blade is faulty or the switch is still booting Ensure blade is firmly seated and switch has completed booting If LED remains yellow contact HP Slow flashing amber on 2 seconds then off 2 s...

Page 59: ...lays OK Absent or Faulty The RPM of each fan in the assembly is also provided If a blower assembly is absent or faulty contact HP Figure 24 displays the blower assembly Figure 24 Blower assembly Table 1 1 Power supply LED descriptions LED purpose Color Status Recommended action Power No light LED is off Power supply does not have incoming power and is not providing power to the DC Director Ensure ...

Page 60: ... has a failure full or partial Replace blower assembly Slow flashing amber on 2 sec then off 2 sec Blower assembly is not seated correctly or is faulty Pull unit out and reseat If LED continues to flash replace unit Flashing amber on 1 2 sec then off 3 5 sec Fan is disabled Run fanEnable to enable the fan Fast flashing amber on 1 2 sec then off 1 2 sec Environmental range exceeded Check for out of...

Page 61: ...WN blade do not light up 1 CP blade Status above and Power below LEDs 3 Port blade Status above and Power below LEDs 2 CR blade Status above and Power below LEDs 4 Slot numbers Table 14 WWN bezel LED descriptions LED purpose Color Status Recommended action Port blade CP blade CR blade Power Steady green Power is OK No action required Port blade CP blade CR blade Status Steady amber Blade is faulty...

Page 62: ...62 Monitor system components ...

Page 63: ... Director can continue operating during many of the FRU replacements if the conditions specified in the procedure are followed Chassis door page 64 Cable management comb page 65 Director blade page 66 Director blade filler panel page 69 Control processor blade CP8 page 70 Core switch blade CR8 page 76 Power supply page 78 Blower assembly page 80 WWN bezel logo plate and WWN card page 81 SFPs and X...

Page 64: ...OTE The door is not hinged It installs straight on the chassis by snapping four ball studs located on the chassis into receiving hardware on the door in the cabinet 1 Grip the side of the door 2 Gently pull the door straight out from the chassis until it snaps out of the ball studs on the chassis Replacing the chassis door To replace the chassis door see Figure 26 1 Align the holes in the door wit...

Page 65: ...e ESD sensitive components in antistatic packaging To remove the cable management comb 1 Rearrange the cables around the cable management comb 2 Unscrew and save the four screws holding the comb to the chassis see Figure 27 Support the comb to prevent it from falling 3 Remove the cable management comb Replacing a cable management comb To replace a cable management comb see Figure 27 1 Position and...

Page 66: ...Remove the chassis door See Removing the chassis door on page 64 2 A failed port or Director blade can be identified by inspecting the LEDs on the front panel of each blade Check the power LED status LED and port status LED to identify any possible problems See Chapter 4 Monitor system components on page 33 for LED locations 3 Establish a Telnet or console session Before replacing a Director blade...

Page 67: ...orted in a future version of Fabric OS 10 Wait for the power LED to turn off in response to the hot swap request before removing the Director blade 1 1 FC8 16 FC8 32 and FC8 48 Director blades Open the ejectors Pull the Director blade out of the chassis using the ejectors FC10 6 Director blade FC4 16IP and FR4 18i Director blades Unscrew the two thumb screws from the top and bottom ejectors on the...

Page 68: ... 48 Director blades Adjust the ejectors to the closed position by pulling them away from the center of the blade FC10 6 Director blade FC4 16IP Director blades Close the ejectors by pushing the handles toward the center of the Director blade until the ejectors lock The levering action of the handles seats the Director blade in the slot IMPORTANT At the time of this document s release the FC4 16IP ...

Page 69: ...ct cooling of the chassis and protection from dust To remove a filler panel 1 Remove the chassis door See Removing the chassis door on page 64 2 Unscrew the thumb screw at the bottom of the panel using the Phillips screwdriver 3 Pull the filler panel out of the chassis see Figure 29 Replacing a filler panel To replace a filler panel 1 Orient the filler panel see Figure 29 2 Slide the filler panel ...

Page 70: ...al console is not available The slotShow command does not show that the CP blade is enabled The haShow command indicates an error The clock is inaccurate or the CP blade does not boot up or shut down normally Any of the following messages display in the error log Slot unknown message relating to a CP slot CP blade errors or I2 C timeouts FRU FRU_FAULTY messages for a CP blade Configuration loader ...

Page 71: ...he functioning CP blade the new active CP blade to verify the failover This is an example of failing over a CP blade Fabric OS cp1 cp1 login admin Password swDir admin hashow Local CP Slot 7 CP1 Active Remote CP Slot 6 CP0 Standby Healthy HA enabled Heartbeat Up HA State synchronized swDir admin hafailover Local CP Slot 7 CP1 Active Remote CP Slot 6 CP0 Standby Healthy HA enabled Heartbeat Up HA S...

Page 72: ...ove the power connectors Also store ESD sensitive components in antistatic packaging To remove a CP blade IMPORTANT The CP8 blade is compatible with the DC Director only 1 Remove the chassis door See Removing the chassis door on page 64 2 Login to the active CP as the admin user You can use a serial cable or Telnet Web Tools or Fabric Manger Determine which CP is active using the haShow command or...

Page 73: ...ach handle using the Phillips screwdriver 4 Turn the CP blade on by sliding the ON OFF switch in the top handle up to cover the thumb screw 5 Verify that the power LED is green If not ensure that the CP blade has power and is firmly seated and that the ejectors are in the locked position 6 Connect the cables to the new CP blade 7 Verify the installation See Verifying operation of the new CP blade ...

Page 74: ... 17 ENABLED 9 SW BLADE 16 ENABLED 10 SW BLADE 16 ENABLED 11 SW BLADE 16 ENABLED 12 SW BLADE 16 ENABLED swDir admin 5 Type firmwareShow to determine the Fabric OS version This example shows sample output for the firmwareShow command swDir admin firmwareShow Local CP Slot 7 CP1 Active Primary partition v6 0 0 Secondary Partition v6 0 0 Remote CP Slot 6 CP0 Standby Primary partition v6 0 0 Secondary ...

Page 75: ...ges have been downloaded successfully Firmwaredownload has completed successfully 2007 07 03 15 05 56 SULB 1002 924 INFO DC Director Firmwaredownload command has completed successfully 8 Type firmwareDownloadStatus to verify that the firmware has been updated 9 Type haEnable to reenable HA on the active CP blade 10 Type reboot 1 1 Verify that the reboot is complete and the CP blades have achieved ...

Page 76: ...tinuing The following events might indicate that a core switch blade is faulty The status LED on the core switch blade is lit amber or the power LED is not lit The slotShow command does not show that the core switch blade is enabled The haShow command indicates an error Any of the following messages display in the error log Slot unknown message relating to a core switch blade slot core switch blad...

Page 77: ...tions above the power connectors on the chassis 1 Remove the chassis door See Removing the chassis door on page 64 2 Power off the blade by sliding the slider switch in the top ejector down to the off position see Figure 31 3 Disconnect cables from the faulty core switch blade 4 Unscrew the thumb screw from both ejectors using the Phillips screwdriver 5 Lever open both ejector handles simultaneous...

Page 78: ... screwdriver 4 Turn the core switch blade on by sliding the ON OFF switch in the top handle up to cover the thumb screw 5 Verify that the power LED is green might require a few seconds If not ensure that the core switch blade has power and is firmly seated and that the ejectors are in the locked position 6 Connect to the new core switch blade 7 Pack the faulty core switch blade in the packaging pr...

Page 79: ...s is functioning A fully populated DC Director requires a minimum of one power supply slot at all times to ensure power to the DC Director 2 Loosen the thumb screw 3 Pull the handle out and down see Figure 32 4 Support the power supply from beneath Use the handle to remove the power supply out of the chassis 5 If you are not replacing the power supply insert a filler panel into the slot Replacing ...

Page 80: ...tor chassis provides a grounding connection above the power connectors Also store ESD sensitive components in antistatic packaging 1 Before removing a blower assembly verify that the other blower assemblies are functioning correctly The power LEDs should be steady green 2 Use the screwdriver to loosen the captive screws at the top and bottom of blower assembly 3 Support the blower assembly from un...

Page 81: ...rd If a serial console session is used serial cable and a workstation computer with a terminal emulator application such as HyperTerminal for Windows systems or TIP for Solaris systems How to determine whether or not to replace a WWN card Before replacing a WWN card verify that the replacement is necessary Any of the following events can indicate that the card requires replacement Status LEDs on t...

Page 82: ...e 15 WWN LED patterns LED location purpose Color Status Recommended action Director blade CP CR blade power Steady green Power is okay No action required Director blade CP CR blade status Steady amber Card is faulty Check card No light LED is off Card is okay No action required Table 16 Commands identifying the WWN card status Data Related commands WWN values wwn chassisShow Data about the chassis...

Page 83: ...hole process to be completed If this process is not complete due to a power cycle or CP failover please follow the recovery procedure in Core Switch WWN Card Removal and Replacement document Do you wish to continue y n Backing up WWN card data please wait about 25 seconds for further instruction Please install the new FRU now If this session lost for any reason please re enter the frureplace comma...

Page 84: ...he bottom Align the connector on the card with the connector on the chassis the connectors are keyed to ensure correct installation and use the push pad to press the card onto the connector until it is seated To install a WWN card that does not have handles Hold the card by the edges and press the card onto the connector until it is fully seated Use the Phillips screwdriver and the two screws to a...

Page 85: ...provided with the new card and return it to the DC Director supplier SFPs and XFPs To remove an SFP or XFP FC10 6 blade only transceiver pull the bale down and out sliding the transceiver out of the Director blade To install an SFP or XFP position one of the transceivers so that the key is oriented correctly to the port Insert the transceiver into the port until it is firmly seated and the latchin...

Page 86: ... grounding pads for protecting all blades and WWN cards Serial cable and workstation computer with a terminal emulator application such as HyperTerminal for Windows systems or TIP for Solaris systems required only if serial console session used Pallet jack or hydraulic or assisted lift that raises a minimum of 55 in 140 cm and carries a minimum of 250 lb 1 13 kg A surface to place the old chassis ...

Page 87: ...omer defined FTP server making it available for downloading For more information about this command refer to the Fabric OS Command Reference rsl8 st03 dcx 1 admin configupload Protocol scp or ftp ftp ftp Server Name or IP Address host 123 123 123 123 User Name user Admin24 File Name config txt config switch0 txt Password Upload complete rsl8 st03 dcx 1 admin 3 Record the DC Director values on a wo...

Page 88: ...bre Channel Subnetmask 255 255 255 0 CP0 Ethernet IP Address 10 32 50 10 Ethernet Subnetmask 255 55 0 0 HostName cp0 Gateway Address 10 32 40 1 CP1 Ethernet IP Address 10 32 50 11 Ethernet Subnetmask 255 55 0 0 HostName cp1 Gateway Address 10 32 40 1 Backplane IP address of CP0 10 0 0 6 Backplane IP address of CP1 10 0 0 7 rsl8 st03 dcx 1 admin rsl8 st03 dcx 1 admin 6 Display and record the manufa...

Page 89: ... 03 Factory Serial Num KP000000195 Manufacture Day 1 Month 1 Year 2007 Update Day 14 Month 3 Year 2012 Time Alive 187 days Time Awake 3 days output truncated CHASSIS WWN Unit 1 in same assembly as WWN Unit 2 Header Version 2 Power Consume Factor 3 Factory Part Num 60 0001501 07 Factory Serial Num FT02X805BE2 Manufacture Day 26 Month 3 Year 2007 Update Day 14 Month 3 Year 2009 Time Alive 207 days T...

Page 90: ...N This file provides a backup of all the information that might be required by Technical Support The information can be used after the DC Director is restored to the fabric to verify that no unintentional changes have occurred to the fabric rsl8 st03 dcx 1 admin supportshow version 6 0 0 output truncated rsl8 st03 dcx 1 admin 9 Record the cable connections between the DC Director and the target de...

Page 91: ...ee Removing a power supply on page 79 6 Remove the blower assemblies see Removing a blower assembly on page 80 7 Remove the WWN bezel logo plate and WWN cards see Removing the WWN bezel logo plate and WWN card on page 83 Install replacement chassis To install the replacement chassis CAUTION A fully populated DC Director eight FC8 48 port cards 384 ports weighs approximately 104 kg 228 lbs and requ...

Page 92: ...on page 73 5 Replace the core switch blades CR8 see Replacing a core switch blade CR8 on page 78 6 Replace the port blades or filler panels see Replacing a Director blade on page 68 7 Replace the cable management comb see Replacing a cable management comb on page 65 8 Connect the power cords to the power supplies and the power outlets 9 Power on the DC Director 10 The DC Director performs a power ...

Page 93: ...losses or damages arising from or relating to the use of the Root or Factory account and the functionality enabled thereby by anyone other than the Equipment vendor s authorized engineers Proceeding with the usage of this switch as the Root or Factory user explicitly indicates your agreement to the terms of this disclaimer rsl8 st03 dcx 1 root 2 Change the factory serial number to the serial numbe...

Page 94: ... the installed cards are detected and that their status is operational enabled rsl8 st03 dcx 1 admin slotShow Slot Blade Type ID Status 1 UNKNOWN VACANT 2 UNKNOWN VACANT 3 SW BLADE 55 ENABLED 4 UNKNOWN VACANT 5 CORE BLADE 52 ENABLED 6 CP BLADE 50 ENABLED 7 CP BLADE 50 ENABLED 8 CORE BLADE 52 ENABLED 9 UNKNOWN VACANT 10 SW BLADE 51 ENABLED 11 UNKNOWN VACANT 12 UNKNOWN VACANT rsl8 st03 dcx 1 admin 3...

Page 95: ...Module 36 3 4 0b2400idN4OnlineF Port 10 00 00 00 c9 51 00 2d 37 3 5 0b2500__N8No_Module rsl8 st03 dcx 1 admin 4 Verify that all the IP address information is correct by typing ipAddrShow and checking the results against the IP information recorded in the config miscinfo txt file rsl8 st03 dcx 1 admin ipaddrshow SWITCH Ethernet IP Address 10 32 50 12 Ethernet Subnetmask 255 55 0 0 Fibre Channel IP ...

Page 96: ...nd cables to the port blades NOTE The ports and cables used in trunking groups must meet specific requirements For a list of these requirements refer to the HP StorageWorks Fabric OS 6 x administrator guide a Position one of the transceivers so that the key is oriented correctly to the port and insert the transceiver into the port until it is firmly seated and the latching mechanism clicks b Selec...

Page 97: ...3 00 29 46 10 00 00 01 73 00 29 46 na Fabric Port Name 20 0f 00 60 69 90 03 f0 output truncated rsl8 st03 dcx 1 admin nsallshow 020f00 021fda 021fdc 021fe0 021fe1 5 Nx_Ports in the Fabric rsl8 st03 dcx 1 admin switchshow switchName rsl8 st03 dcx 01 output truncated rsl8 st03 dcx 1 admin fabricshow Switch ID Worldwide Name Enet IP Addr FC IP Addr Name output truncated rsl8 st03 dcx 1 admin 2 Determ...

Page 98: ... name or IP of the host for the download process to complete For more information refer to the help configdownload command or the Fabric OS Command Reference rsl8 st03 dcx 1 admin configdownload Server Name or IP Address host 123 123 123 123 User Name None Admin24 File Name config txt config switch txt Password xxxxxxxx download complete rsl8 st03 dcx 1 admin If other issues exist contact HP Cable...

Page 99: ...e 99 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 Table 19 Cable routing table for DC Director 48 ports shown continued Slot port Cable labels Connected device Slot port of device Slot Port Switch end Device end ...

Page 100: ...100 Installing Field replaceable units FRUs ...

Page 101: ...ull duplex 4 25 Gbps line speed full duplex 8 50 Gbps line speed full duplex 10 51875 Gbps line speed full duplex with FC10 6 blade Autosensing of 1 2 4 and 8 Gbps port speeds optionally programmable to fixed port speed speed matching between 1 2 4 and 8 Gbps port speeds ISL Trunking Can use up to 8 ports in a trunk group to form a 64 Gbps trunk Aggregate bandwidth 6144 Gbps end to end when using ...

Page 102: ...extended form factor pluggable XFP FC10 6 only LC connector short wave laser SWL up to 500 m 1 640 ft long wave laser LWL up to 10 km 6 2 mi extended long wave laser ELWL up to 80 km 49 6 mi distance depends on fiber optic cable and port speed Copper 1GbE connections on the FC4 16IP blade Fabric services Simple Name Server Registered State Change Notification RSN Advanced Zoning Web Tools Fabric W...

Page 103: ... of an earthquake Additional equipment does not exceed the rack s weight limits Power specifications The power subsystem is a redundant 48V DC power distribution system with a provision for up to four 1000 watt 48V DC bulk power supplies Four bulk power supplies produce the intermediate distribution voltage in the distributed power system The power specifications in Table 23 are calculated for ful...

Page 104: ...d switch requires a maximum of 750 Volt Amps This results in a main current of 3 2 amps at 240V AC line voltage or 3 6 amps at 208V AC line voltage The rated AC input range is 180 to 264V AC or 1 10 to 120 VAC Supported power range Nominal 200 to 240 VAC single phase or 1 10 to 120 VAC single phase Input frequency range 47 63 Hz Power supplies each Output voltages 48V at 20 amps 12V at 4 amps Maxi...

Page 105: ...an d BS 1363A United Kingdom Hong Kong AS 31 12 Australia Ne w Zealand IEC 60309 32A 6h 230 V Argentina X Australia X Austria X Bahrain X Belgium X Brazil X Chile X China People s Rep X Czech Rep of X Denmark X Egypt X England X Finland X X France X Germany X Greece X Hong Kong X Hungary X India X Indonesia X Ireland North X Ireland South X Israel X Italy X Japan X Korea South X Malaysia Alternate...

Page 106: ...weden X Switzerland X Taiwan X Turkey X United Arab Emirate X United Kingdom Ireland X United States X X Venezuela X Yugoslavia X Table 24 Power cord types international continued Country Plug style NEMA L6 20 USA Canada Mexico other North American locations CEE 7 7 Schuko Continental Europe Irelan d BS 1363A United Kingdom Hong Kong AS 31 12 Australia Ne w Zealand IEC 60309 32A 6h 230 V ...

Page 107: ...imately 850 nm optical LWL long wavelength approximately 1310 nm and Extended Long Wavelength ELWL laser transmitter SFPs and XFPs FC10 6 only approximately 1550 nm The strength of the signal is determined by the type of SFP or XFP in use The ports are universal and self configuring capable of becoming F_Ports fabric ports FL_Ports fabric loop enabled or E_Ports expansion ports The ports meet all ...

Page 108: ... m OM2 150 m OM3 62 5 microns 2 Gbps 0 5 m 150 m 4 Gbps 0 5 m 70 m 8 Gbps 0 5 m 21 m Table 27 General specifications Specification Description Configurable port types F_Port FL_Port E_Port EX_Port VEX_Port VE_Port and M_Port System architecture Nonblocking shared memory System processor FreeScale 8548 1 2 GHz ANSI fibre channel protocol FC PH Fibre Channel Physical and Signalling Interface standar...

Page 109: ...blade provides the following features Sixteen Fibre Channel SFP ports supporting the Fibre Channel Routing Services with link speeds up to 1 2 or 4Gbps 2 GbE ports supporting the FCIP and Fibre Channel Routing Services with link speeds up to 1Gbps Each GbE port can support up to eight FCIP tunnels Each FCIP tunnel is represented and managed as a virtual Fibre Channel E_Port Fibre Channel Routing S...

Page 110: ... using the power switch at the top of the blade The FR4 18i blade begins POST After POST is complete the firmware version on the blade will autolevel with the firmware version on the active CP 5 Verify that the blade power and blade status LEDs are green after the blade has completed booting Figure 36 B Series MP Router blade FR4 18i components Table 28 B Series MP Router blade FR4 18i components ...

Page 111: ...entation If a transceiver does not install easily ensure that it is correctly oriented c Repeat Steps a and b for the remaining ports as required 2 Connect the cables to the transceivers CAUTION A 50 micron cable should not be bent to a radius less than 2 inches under full tensile load and 1 2 inches with no tensile load Tie wraps are not recommended for optical cables because they are easily over...

Page 112: ... and eight Gigabit Ethernet GbE copper RJ 45 ports NOTE The number of initiators per port blade or chassis depends on the version of Fabric OS and is subject to change from release to release The FC4 16IP blade provides the following features Eight FC SFP ports supporting link speeds of 1 2 or 4Gbps Eight GbE ports for bridging iSCSI hosts Support distances up to 100 m over Category 5e and Categor...

Page 113: ...s completed booting up Cabling the FC4 16IP Blade To complete the switch configuration 1 Install the SFP transceivers in the Fibre Channel as required a Remove the rubber plugs from the ports to be used b Position a transceiver so that it is oriented correctly and insert it into a port until it is firmly seated and the latching mechanism clicks For instructions specific to the type of transceiver ...

Page 114: ... switchRole Subordinate switchDomain 15 switchId fffc0f switchWwn 10 00 00 60 69 80 47 74 AD Feature Active Current AD 0 zoning ON SIMPLE_CFG switchBeacon OFF blade1 Beacon OFF blade2 Beacon OFF blade3 Beacon OFF blade7 Beacon OFF blade9 Beacon OFF blade10 Beacon OFF Index Slot Port Address Media Speed State Proto 112 10 0 0f7000 id N4 No_Light Disabled Persistent 113 10 1 0f7100 N4 No_Module Disa...

Page 115: ... enable feature disable feature show args The following example shows iSCSI service being enabled Enabling the persistently disabled ports Depending on the previous usage of the selected chassis slot the FC4 16IP ports may be persistently disabled Before you can successfully configure the iSCSI Target Gateway you must persistently enable the ports 1 Enter the portCfgShow command to view ports that...

Page 116: ...estination routes on an IP interface You can add IP routes when you want to reach the destination IP of the remote site through a preferred gateway IP You can specify a maximum of 32 routes per GbE port The syntax for this command is portcfg iproute slot ge port args The gateway address must be on the same IP subnet as the port IP address The following example shows two routes being added to an in...

Page 117: ... on page 1 18 Defining CHAP entries for iSCSI device authentication on page 120 Finalizing the iSCSI configuration on page 120 Configuring iSCSI protocol for each iSCSI port Optional on page 121 Creating iSCSI virtual targets An iSCSI target must be created before exposing it to the iSCSI host You must provide the number of Logical Unit Numbers LUNs and the actual LUN values that are part of the c...

Page 118: ...layed You can specify either port or node WWNs fclunquery w wwn To view the available LUNs use the fcLunQuery command Once you have the port WWN of the LUN you want to add to the iSCSI target you can add the LUN map to the target with the specified target name FC WWN provides the port WWN of the physical FC target and the LUN map maps the physical FC LUNs to virtual iSCSI LUNs The LUN mapping is s...

Page 119: ...t to enable the configuration The discovery domain set contains discovery domains More than one discovery domain set can be created defined but only one can be enabled at any point in time The active enabled discovery domain set will be enforced across all the switches in the fabric When you execute commit the switch on which the command is executed will become the master and will merge the discov...

Page 120: ...mit all iscsicfg commit all To commit the configurations that have been created and modified use the iscsiCfg command as follows The iscsicfg commit all command commits local configuration changes and also pushes iSCSI configuration from a local switch to remote switches in the fabric However during the commit process other configuration changes not yet commited on remote switches need to be abort...

Page 121: ...o be enabled iscsiportcfg modify show clearstats default slot_number port_number The following example shows how to configure the parameters described above for slot 7 and port ge0 The following example shows how to display the configured port parameters switch admin iscsiportcfg modify 7 ge0 Please specify parameters to modify Usage iscsiportcfg modify show clearstats default SlotNumber PortNumbe...

Page 122: ...122 Intelligent blades ...

Page 123: ...quires the user to be notified that any changes or modifications made to this device that are not expressly approved by Hewlett Packard Company may void the user s authority to operate the equipment Cables Connections to this device must be made with shielded cables with metallic RFI EMI connector hoods in order to maintain compliance with FCC Rules and Regulations Regulatory compliance identifica...

Page 124: ...Subchapter J A label on the plastic FO transceiver housing indicates the certification Outside the USA the FO transceiver is certified as a Class 1 laser product conforming to the requirements contained in IEC 825 1 1993 and EN 60825 1 1994 including Amendment 1 1 1996 and Amendment 2 2001 Laser product label The optional label in Figure 37 or equivalent may be located on the surface of the HP sup...

Page 125: ... avis Canadien Class A equipment This Class A digital apparatus meets all requirements of the Canadian Interference Causing Equipment Regulations Cet appareil numérique de la classe A respecte toutes les exigences du Règlement sur le matériel brouilleur du Canada European Union regulatory notice BSMI notice ...

Page 126: ...126 Regulatory compliance and safety notices Japanese notice Korean notices MIC statement Republic of Korea Korean Notice A and B ...

Page 127: ...e EPUP logos shown on the product and FRUs alter or expand that warranty that Brocade provides with respect to its products as set forth in the applicable contract between Brocade and its customer Brocade hereby disclaims all other warranties and representations with respect to the information contained on this CD including the implied warranties of merchantability fitness for a particular purpose...

Page 128: ...dicates that no such substances are used or that the concentration is within the aforementioned limits China ROHS Hazardous Substances Toxic Substances HS TS Concentration Chart Name of the Component Hazardous Toxic Substance Elements Lead PB Mercury Hg Cadmiu m Cd Hexavalent Chromium CR6 Polybrominat ed Biphenyl PBB Polybrominat ed Diphenyl Ether PBDE Fibre Channel Switch X O O O O O Fan Blower a...

Page 129: ...their containers until they arrive at static free workstations Place parts on a grounded surface before removing them from their containers Avoid touching pins leads or circuitry Always make sure you are properly grounded when touching a static sensitive component or assembly Grounding methods There are several methods for grounding Use one or more of the following methods when handling or install...

Page 130: ...ion about battery replacement or proper disposal contact an HP authorized reseller or HP authorized service provider WARNING Your switch contains an internal lithium manganese dioxide a vanadium pentoxide or an alkaline battery pack There is risk of fire and burns if the battery pack is not properly handled To reduce the risk of personal injury Do not attempt to recharge the battery Do not expose ...

Page 131: ...ltage and current rating of the cord should be greater than the voltage and current rating marked on the product In addition the diameter of the wire must be a minimum of 1 00 mm2 or 18 AWG and the length of the cord must be between 1 8 m 6 ft and 3 6 m 12 ft If you have questions about the type of power cord to use contact an HP authorized service provider NOTE Route power cords so that they will...

Page 132: ...132 Regulatory compliance and safety notices ...

Page 133: ...ng pattern for the DC Director These templates show Port side populated with eight FC8 48 port blades two CR8 blades and two CP8 blades Port side populated with eight FC8 32 port blades two CR8 blades and two CP8 blades Port side populated with eight FC8 16 port blades two CR8 blades and two CP8 blades FC10 6 port blade FC4 16IP application blade FR4 18i application blade ...

Page 134: ...134 Port numbering templates 26390a ...

Page 135: ...DC SAN Backbone Director hardware reference guide 135 26389a ...

Page 136: ...136 Port numbering templates 26388a ...

Page 137: ...DC SAN Backbone Director hardware reference guide 137 FC 10 6 0 2 3 4 5 1 2 3 4 5 FC 10 6 2 3 4 5 1 1 0 0 26391a ...

Page 138: ...0590 01 Rev A FC4 16IP 7 6 5 4 3 2 1 0 GE FC 7 6 5 4 3 2 1 0 56 0000590 01 Rev A FC4 16IP 7 6 5 4 3 2 1 0 GE FC 7 6 5 4 3 2 1 0 56 0000590 01 Rev A FC4 16IP 7 6 5 4 3 2 1 0 GE FC 7 6 5 4 3 2 1 0 56 0000590 01 Rev A FC4 16IP 7 6 5 4 3 2 1 0 GE FC 26393a ...

Page 139: ...SAN Backbone Director hardware reference guide 139 7 6 5 4 3 2 1 0 GE1 GE0 15 14 13 12 11 10 9 8 56 0000590 01 Rev A FR4 18i 7 6 5 4 3 2 1 0 GE1 GE0 15 14 13 12 11 10 9 8 56 0000590 01 Rev A FR4 18i 26394a ...

Page 140: ...140 Port numbering templates ...

Page 141: ...pace requirement 24 reliability 18 serviceability 18 software features 18 specifications general 108 BSMI regulatory compliance notice 125 buffers frame 101 C cable management 114 cable management comb replacing 65 cables FCC compliance statement 123 shielded 123 cables managing 35 cabling 113 Canada regulatory compliance notice 125 certification and classification information laser 124 certificat...

Page 142: ...1 FRUs blades weights 103 blower assembly 59 LEDs 60 replacing 80 cable management comb replacing 65 chassis door replacing 64 CP blade LEDs 56 replacing 70 CR blade LEDs 58 replacing 76 Director blade replacing 66 filler panel replacing 69 power supply 58 LEDs 59 replacing 78 removing and replacing 63 weights 103 WWN bezel LEDs 82 replacing 81 WWN card LEDs 82 replacing 81 G GbE see Italic gigabi...

Page 143: ...lity warning 11 recycling battery 130 recycling Taiwan EPA battery 130 regulatory compliance information number 123 notices BSMI 125 Canada 125 European Union 125 HP series number 123 IEC EMC statement worldwide 125 Japan 126 Korean 126 lasers 123 modifications 123 shielded cables 123 related documentation 9 reliability features 18 removal and replacement procedures Removing a filler panel Directo...

Page 144: ...viewing available LUNs 118 voltage compliance rating 131 W warning rack stability 11 warnings lasers radiation 123 waste electrical and electronic equipment directive 123 web sites HP storage 11 HP Subscriber s choice 11 weight DC Director 102 WWN bezel 81 LEDs 82 replacing 81 WWN card 81 error messages 60 LEDs 82 replacing 81 status 60 82 ...

Reviews: