IBM SAN768B-2 Скачать руководство пользователя | Manualshive

Страница: 81 / 228

background image

Chapter 4. Monitoring system components

The SAN768B-2 is engineered for reliability and requires no routine operational
steps or maintenance. This chapter provides information about determining the
status of the following components using LEDs and CLI commands. Refer to the

Fabric OS Administrator’s Guide

and the

Web Tools Administrator’s Guide

for

additional information.

There are two commands that can be especially helpful in monitoring the health of
the SAN768B-2. These commands are

switchShow

and

chassisShow

. Examples of

these commands are shown below. Note in the

switchShow

command that

"switchType" for the SAN768B-2 as well as the 16-Gbps speed identification for
capable ports are displayed. The output has been truncated to reduce information
duplication.

SAN768B-2_8 :root> switchshow
switchName: SAN768B-2_8
switchType: 120.3
switchState: Online
switchMode: Native
switchRole: Subordinate
switchDomain: 120
switchId: fffc78
switchWwn: 10:00:00:05:1e:d2:64:00
zoning: ON (BB)
switchBeacon: OFF
FC Router: ON
FC Router BB Fabric ID: 10
Address Mode: 0
Index

Slot

Port Address Media Speed State

Proto

=======================================================

0

1

0

780000

id

N8

No_Light

FC Disabled

(Switch not ready for EX_Ports)

1

1

1

780100

id

N8

No_Light

FC Disabled

(Switch not ready for EX_Ports)

2

1

2

780200

--

N8

No_Module FC Disabled

(Switch not ready for EX_Ports)

3

1

3

780300

--

N8

No_Module FC Disabled

(Switch not ready for EX_Ports)

4

1

4

780400

id

N8

No_Light

FC Disabled

(Switch not ready for EX_Ports)

5

1

5

780500

id

N8

No_Light

FC Disabled

(Switch not ready for EX_Ports)

6

1

6

780600

id

N8

No_Light

FC

7

1

7

780700

id

4G

No_Light

FC

8

1

8

780800

--

N8

No_Module FC

9

1

9

780900

--

N8

No_Module FC

10

1

10

780a00

--

N8

No_Module FC

11

1

11

780b00

--

N8

No_Module FC

12

1

12

780c00

--

--

Offline

VE Disabled (Persistent)

13

1

13

780d00

--

--

Offline

VE

14

1

14

780e00

--

--

Offline

VE

15

1

15

780f00

--

--

Offline

VE

128

1

16

788000

--

--

Offline

VE

129

1

17

788100

--

--

Offline

VE

130

1

18

788200

--

--

Offline

VE

131

1

19

788300

--

--

Offline

VE

132

1

20

788400

--

--

Offline

VE

133

1

21

788500

--

--

Offline

VE

134

1

22

788600

--

--

Offline

VE Disabled (Persistent)

135

1

23

788700

--

--

Offline VE

© Copyright IBM Corp. 2011, 2014

53

«
...
79
80
81
82
83
...
»

Содержание SAN768B-2

Страница 1: ...IBM System Storage SAN768B 2 Installation Service and User Guide Service information 2499 816 GA32 0893 06...

Страница 2: ......

Страница 3: ...IBM System Storage SAN768B 2 Installation Service and User Guide Service information 2499 816 GA32 0893 06...

Страница 4: ...NESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A...

Страница 5: ...vers and attaching cables 44 Managing cables 46 Planning for encryption optional 47 Configuring for encryption optional 49 Verifying correct operation and backing up the configuration 50 Fabric OS fir...

Страница 6: ...ng POST and boot results 160 POST 160 Boot 161 Diagnostics 161 General troubleshooting 161 Link troubleshooting 163 Appendix D Tuning the system 169 Identifying whether you need to tune your system 16...

Страница 7: ...is door 82 30 Cable management comb 83 31 Port and application removal and installation FC16 48 port blade shown as example 86 32 Filler panel removal and installation 88 33 Removing the control proce...

Страница 8: ...vi SAN768B 2 Installation Service and User Guide...

Страница 9: ...ric OS 94 15 Transceiver types and speeds by blade model 108 16 ICL connector port LEDs 114 17 Critical information checklist example 120 18 Sample cable routing table for SAN768B 2 129 19 Cable routi...

Страница 10: ...viii SAN768B 2 Installation Service and User Guide...

Страница 11: ...ixth edition The following changes were made in this edition v Revisions to replacement procedures for WWN card v Updates to the WWN diagnostics Fifth edition The following changes were made in this e...

Страница 12: ...Fabric OS firmware downloads go to the IBM Support Portal http www ibm com supportportal Search for the product Machine type or product name and then follow links for Downloads More detailed instruct...

Страница 13: ...oration 3F No 7 Song Ren Rd Taipei Taiwan Tel 0800 016 888 How to send your comments Your feedback is important in helping us provide the most accurate and high quality information If you have comment...

Страница 14: ...xii SAN768B 2 Installation Service and User Guide...

Страница 15: ...ns that follow v Note These notices provide important tips guidance or advice v Attention notices on page xix These notices indicate potential damage to programs devices or data v Caution notices on p...

Страница 16: ...d place dangerous voltage on the metal shell If any of the conditions are not as described STOP Ensure the improper voltage or impedance conditions are corrected before proceeding D003 DANGER An elect...

Страница 17: ...ossible use one hand only to connect or disconnect signal cables v Never turn on any equipment when there is evidence of fire water or structural damage v Disconnect the attached power cords telecommu...

Страница 18: ...relocation or disposal 500 lbs 227 kg a69i0333 DANGER Heavy equipment personal injury or equipment damage might result if mishandled D006 Caution notices A caution notice calls attention to a situatio...

Страница 19: ...d electrical outlet to avoid electrical shock C018 CAUTION This product might contain one or more of the following devices CD ROM drive DVD ROM drive DVD RAM drive or laser module which are Class 1 la...

Страница 20: ...ER Hazardous voltage current or energy levels are present inside any component that has this label attached Do not open any cover or barrier that contains this label L001 DANGER Rack mounted devices a...

Страница 21: ...Do not bend a fiber cable to a radius less than 5 cm 2 in you can damage the cable Do not use tie wraps for optical cables they can be easily overtightened and damage the cable ESD precautions Attent...

Страница 22: ...sure that the outlet is correctly wired and grounded to prevent an electrical shock R001 part 1 of 2 CAUTION v Do not install a unit in a rack where the internal rack ambient temperatures will exceed...

Страница 23: ...t all door openings are at least 760 x 2030 mm 30 x 80 in Ensure that all devices shelves drawers doors and cables are secure Ensure that the four leveling pads are raised to their highest position En...

Страница 24: ...changes that might have been made to the machine If any are present obtain the Non IBM Alteration Attachment Survey form number R009 from the IBM branch office Complete the form and return it to the...

Страница 25: ...ions Center v IBM System Storage SAN768B 2 Installation Service and User Guide GA32 0893 this document which is also available in accessible HTML format on the documentation CD v Safety Notices v IBM...

Страница 26: ...r Brocade 7840 SAN42B R 2498 Model R42 Brocade 6520 SAN96B 5 2498 Models F96 and N96 Brocade 6505 SAN24B 5 2498 Model F24 249824G X24 Brocade 6510 SAN48B 5 2498 Model F48 Brocade DCX 8510 4 SAN384B 2...

Страница 27: ...eader software Help for a list of keyboard shortcuts that it supports Vendor software The SAN768B 2 includes certain vendor software that is not covered under the IBM license agreement IBM makes no re...

Страница 28: ...xxvi SAN768B 2 Installation Service and User Guide...

Страница 29: ...ally configured on any port of the 32 and 48 port 16 Gbps port blades v Support for most of the application port blade and control processor CP blades that are supported in the SAN384B 2 with the exce...

Страница 30: ...can be connected with FOS v7 0 0 Both CR16 8 blades are active v Modular hot swappable port blades 64 port 8 Gbps blades FC8 64 32 port 8 Gbps blades FC8 32E 48 port 8 Gbps blades FC8 48E 32 port 16...

Страница 31: ...e v World Wide Name WWN cards on the nonport side to maintain chassis specific information such as WWNs IP addresses and summary status information of each port blade and power supply through LEDs Not...

Страница 32: ...port side of the SAN768B 2 with 4 power supplies 1 WWN bezel WWN cards are behind bezel 3 Blower assembly 2 Power supply b768ig014 Figure 2 Blower non port side of the SAN768B 2 sample configuration...

Страница 33: ...he SAN768B 2 SAN384B 2 SAN768B and SAN384B Core switch blade N A CR16 8 The CR16 8 blade contains the ASICs for switching between port blades Every port blade connects to every core switch blade There...

Страница 34: ...s Fibre Channel port speeds The blade also supports port based in flight encryption decryption and compression decompression This port blade is compatible with the SAN384B 2 and SAN768B 2 64 port 16 G...

Страница 35: ...e Channel port speeds This port blade is compatible with the SAN384B 2 and SAN768B 2 This blade requires Fabric OS v7 0 1 or later to run in this chassis 64 port 8 Gbps port blade FC3864 FC8 64 A 64 p...

Страница 36: ...o the Fabric OS Release Notes for limitations in using this blade Encryption blade FC3895 FS8 18 The FS8 18 blade enables data cryptographic encryption decryption and data compression capabilities for...

Страница 37: ...cking v Power on self test POST v Dual control processors that enable hot nondisruptive fast firmware upgrades v One serial port and two Ethernet ports on each control processor for management and for...

Страница 38: ...FICON with CUP Activation optional feature providing in band management of the supported SAN b type switch and director products by system automation for z OS from IBM System z10 EC z10 BC z9 EC z9 BC...

Страница 39: ...alling this feature on this and other models The publication is available through the IBM Support Portal http www ibm com supportportal or the IBM Publications Center http www ibm com shop publication...

Страница 40: ...12 SAN768B 2 Installation Service and User Guide...

Страница 41: ...to Chapter 3 Starting and configuring the SAN768B 2 on page 35 Note Some illustrations in this chapter display different models of IBM SAN directors Details such as the cable management comb and blad...

Страница 42: ...tion You should direct questions about these procedures to your regional representative Note The bridge tool PN 18P5855 which is required for installation is shipped with a new or a replacement SAN768...

Страница 43: ...ickup You must return the lift tool at the time that was scheduled with UPS Logistics If you need to change the scheduled return time or date contact UPS Logistics You are responsible for ensuring tha...

Страница 44: ...ctices when moving the product C015 Note A fully populated SAN768B 2 eight FC8 64 or FC16 64 port cards 512 ports weighs approximately 159 2 kg 351 lb and requires the specified lift tool to install i...

Страница 45: ...minal server port and a terminal emulator application such as HyperTerminal Ethernet IP addresses for the chassis and for both control processor blades total of 3 IP addreses Installing an Ethernet ca...

Страница 46: ...f a plastic shipping tray 5 Use the lift tool to transport the new chassis to the installation area Doorways must be wider than 91 cm 36 in to accommodate the chassis on the shipping tray Attaching th...

Страница 47: ...rack mount kit These brackets are adjustable to allow for different cabinet depths 2 Loosen the adjusting screws on both brackets enough to allow the brackets to slide easily to adjust the length E 16...

Страница 48: ...crews on the shelf brackets to a torque of 37 cm kg 32 in lb 5 Attach the clip nuts D for rails with round holes or retainer nuts F for rails with square holes to the vertical rails on the exhaust ais...

Страница 49: ...Set the door aside for later reinstallation Note Reinstall the chassis door once the chassis is installed and all cabling and configuration is complete The chassis door must be installed to ensure th...

Страница 50: ...SAN768B 2 This procedure uses parts from the 14U chassis mount kit See Rack mount kit parts list on page 18 for a list of these parts svc00169 55 kg 121 2 lbs CAUTION A fully populated SAN768B 2 weigh...

Страница 51: ...s can reduce the weight by up to 27 kg 60 lb 4 Move the lift tool next to the pallet that contains the SAN768B 2 a Adjust the load platform to the same height b Slide the SAN768B 2 from the pallet ont...

Страница 52: ...the bridge tool by completing the following steps a Remove the bridge tool from the box and adjust the two supports 45 b Hold the bridge tool between the two vertical supports near EIA unit 3 and mov...

Страница 53: ...tom hole and the middle hole in EIA unit 20 d Align the one pin section of the support on the front of the vertical rail with the middle hole in EIA unit 18 13 Turn the winch crank counterclockwise 1...

Страница 54: ...r 21 Reinstall the cable management comb if it was removed see Removing and installing a cable management comb on page 82 22 If ICL cables will not be used ensure that the ICL sockets in the core swit...

Страница 55: ...b768ig032 Figure 10 Aligning the chassis door with the ball studs on the chassis Chapter 2 Installing a SAN768B 2 in a cabinet 27...

Страница 56: ...itch Blade 3648 FC8 48E 48 port 8 Gbps Enhanced Port Blade 3649 FC16 64 64 port 16 Gbps Port Blade 3664 FC8 64 64 port 8 Gbps Switch Blade 3864 FCOE10 24 24 port FCoE 10 GbE Blade 3880 FX8 24 FCIP 8 G...

Страница 57: ...left set of ports v 16 through 31 from bottom to top on the right set of ports 0 7 8 15 16 23 24 31 FC16 48 port blade v 0 through 23 from bottom to top on the left set of ports v 24 through 47 from...

Страница 58: ...ups are permitted The three groups are defined as follows v Trunk group 0 FC ports 0 1 v Trunk group 1 FC ports 6 7 v Trunk group 2 FC ports 2 3 4 5 8 9 10 11 Chassis slots Slots are numbered 1 12 fro...

Страница 59: ...d the devices to which they are connected v Keep LEDs visible by routing port cables and other cables away from the LEDs v Use hook and loop type straps to secure and organize fiber optic cables Atten...

Страница 60: ...page xiii before starting any service procedure Due to the weight of the product this procedure requires a lift tool See Ordering the lift tool on page 14 for information on ordering the lift tool Pe...

Страница 61: ...m to the lift tool If anti tip bars were previously attached skip to step 10 a Pull up on the leg lock pins and insert the anti tip bars inside the legs b Pull on the legs and anti tip bars to ensure...

Страница 62: ...17 Lower the load platform to its lowest position 18 Remove the bridge tool 34 SAN768B 2 Installation Service and User Guide...

Страница 63: ...ess and subnet mask for the chassis v IP addresses host names subnet masks and gateway addresses for both CP blades v Switch name v Domain ID for the SAN768B 2 optional v WWN for the SAN768B 2 Note Th...

Страница 64: ...tomize a switch name on page 40 3 Specify a domain ID for the SAN768B 2 See Setting the Domain ID on page 41 4 Set the date and time See Setting the date and time on page 41 5 Verify the PID mode and...

Страница 65: ...ts on page 53 You can bypass POST by using the fastBoot command You can also disable POST for successive reboots on the SAN768B 2 using the diagDisablePost command Attention To prevent a potential IP...

Страница 66: ...press Enter key to proceed Password was not changed Will prompt again at next login until password is changed swDir admin 2 Optional Modify passwords To skip modifying the password press Ctrl C Passwo...

Страница 67: ...up the CP1 blade IP address by entering the ipaddrset cp 1 command at the prompt swDir admin ipAddrSet cp 1 Enter the requested information at the prompts The following is a sample IP configuration f...

Страница 68: ...nd to an Ethernet 10 100 1000 Base T LAN The SAN768B 2 can now be accessed by remote connection using the command line via Telnet or any of the management tools such as Web Tools or Network Advisor 4...

Страница 69: ...e and time settings are used for logging events Switch operation does not depend on the date and time a switch with an incorrect date and time value still functions properly However because the date a...

Страница 70: ...ase identify a location so that time zone rules can be set correctly 3 Enter the appropriate number or Ctrl D to quit 4 At the prompt select a country location 5 At the prompt enter the appropriate nu...

Страница 71: ...o Based Area Assignment 2 Port Based Area Assignment 0 2 0 1 WWN Based persistent PID yes y no n no yes command output truncated You can check the PID setting using the configshow command as in the fo...

Страница 72: ...4 port blade While they will fit in other blades this configuration is unsupported and will generate an error Attention With increasing speeds of optical connections the cleanliness of the connections...

Страница 73: ...atible with the QSFP transceivers designed for the core blades They are not interchangeable In the core blades these transceivers and cables are used to form the inter chassis links ICL with neighbori...

Страница 74: ...G No_Module FC 267 3 11 2 16G No_Module FC 268 3 12 3 16G No_Module FC 269 3 13 3 16G No_Module FC 270 3 14 3 16G No_Module FC 271 3 15 3 16G No_Module FC 736 3 16 4 16G No_Module FC 737 3 17 4 16G No...

Страница 75: ...ce planning is essential to achieve a successful implementation of encryption Attention Setup of the FS8 18 blade for encryption requires the use of this document for the physical installation of the...

Страница 76: ...to a TKLM appliance section of the Fabric OS Encryption Administrator s Guide Supporting Tivoli Key Lifecycle Manager TKLM Environments for detailed information on initial setup That section includes...

Страница 77: ...the encryption group v Generating and backing up the master key in RSA environments v Handling key vault high availability v Configuring cluster interconnect After completing the pre initialization ta...

Страница 78: ...pload vf command This command uploads the system virtual fabric data b Enter the configupload command This command uploads the system configuration 6 Run the following commands to see more configurati...

Страница 79: ...page The FOS documents on the Brocade page may be newer versions than the ones shipped with your product Downloading and installing firmware See Downloading firmware from an FTP server on page 94 and...

Страница 80: ...23 14 23 06 2011 The system is going down for system halt NOW 2 Power off the chassis by flipping the AC power switches on the power supplies to O The LEDs inside AC power switches should turn off To...

Страница 81: ...0 05 1e d2 64 00 zoning ON BB switchBeacon OFF FC Router ON FC Router BB Fabric ID 10 Address Mode 0 Index Slot Port Address Media Speed State Proto 0 1 0 780000 id N8 No_Light FC Disabled Switch not...

Страница 82: ...FC 43 3 11 782b00 N8 No_Module FC 44 3 12 782c00 id N8 No_Light FC 45 3 13 782d00 id N8 No_Light FC 46 3 14 782e00 N8 No_Module FC 47 3 15 782f00 N8 No_Light FC 165 3 21 78a500 N8 No_Module FC 166 3...

Страница 83: ...78b300 id N8 No_Light FC 180 4 20 78b400 id N8 No_Light FC 181 4 21 78b500 id N8 No_Light FC 182 4 22 78b600 id N8 No_Light FC 183 4 23 78b700 id AN No_Sync FC 184 4 24 78b800 N16 No_Module FC 185 4...

Страница 84: ...G No_Module FC 1155 5 35 16G No_Module FC 1156 5 36 16G No_Module FC 1157 5 37 16G No_Module FC 1158 5 38 16G No_Module FC 1159 5 39 16G No_Module FC 1160 5 40 16G No_Module FC 1161 5 41 16G No_Module...

Страница 85: ...AHJ0449E042 Manufacture Day 14 Month 12 Year 2010 Update Day 28 Month 3 Year 2011 Time Alive 299 days Time Awake 0 days CP BLADE Slot 7 Header Version 2 Power Consume Factor 40 Factory Part Num 60 100...

Страница 86: ...28 Month 3 Year 2011 Time Alive 319 days Time Awake 0 days POWER SUPPLY Unit 3 Header Version 2 Power Consume Factor 2000 Factory Part Num 23 0000067 01 Factory Serial Num AGC2M44EKPL Manufacture Day...

Страница 87: ...ication or encryption blade status v Control processor blade CP8 status on page 70 v Core switch blade CR16 8 status on page 72 v Power supply status on page 74 v Blower assembly status on page 75 v W...

Страница 88: ...mation about how to interpret the LED patterns see Table 7 on page 68 2 Check the blade status by entering the slotShow command 1 Power LED 3 Fibre Channel port 2 Status LED 4 Port status LED 2 1 3 4...

Страница 89: ...1 Power LED 3 Fibre Channel port 2 Status LED 4 Port status LED 2 1 3 4 b768ig065 Figure 16 FC16 48 port blade Chapter 4 Monitoring system components 61...

Страница 90: ...lade power LED 3 Port status LED for FC port 36 2 Blade status LED 4 QSFP port 9 FC ports 36 39 bottom to up b768ig079 1 2 3 4 Figure 17 FC16 64 port blade 62 SAN768B 2 Installation Service and User G...

Страница 91: ...1 Power LED 3 Fibre Channel port 2 Status LED 4 Port status LED 2 1 3 4 b768ig074 Figure 18 FC8 32E port blade Chapter 4 Monitoring system components 63...

Страница 92: ...1 Power LED 3 Fibre Channel port 2 Status LED 4 Port status LED 2 1 3 4 b768ig075 Figure 19 FC18 48E port blade 64 SAN768B 2 Installation Service and User Guide...

Страница 93: ...tus LED 3 Fibre Channel port 2 Power LED 4 Port status LED Note The FC8 64 port blade requires narrower OM 3 LC cables 3 4 1 2 B768ig046 Figure 20 FC8 64 port blade Chapter 4 Monitoring system compone...

Страница 94: ...1 Power LED 3 Fibre Channel port 2 Status LED 4 Port status LED 3 4 b768ig060 1 2 Figure 21 FS8 18 encryption blade 66 SAN768B 2 Installation Service and User Guide...

Страница 95: ...ap 2 Status LED 5 10 GbE port 0 3 GbE port 6 6 10 GbE port 0 LED See Table 6 on page 29 for information about trunking groups on this blade 4 3 5 6 7 2 1 b768ig047 Figure 22 FX8 24 extension blade Cha...

Страница 96: ...pplication and encryption blade LED descriptions LED purpose Color Status Recommended action Blade power LED Steady green Blade is enabled No action required No light LED is off Blade is not powered o...

Страница 97: ...ce Steady green Port is online connected to an external device but has no traffic No action required Slow flashing green on 1 second then off 1 second Port is online but segmented indicating a loopbac...

Страница 98: ...s online with traffic flowing through port No action required Fast flashing amber on 1 4 second then off 1 4 second Transceiver or port is faulty Change the transceiver or reset the switch from the wo...

Страница 99: ...CP LED Table 8 CP8 blade LED descriptions LED purpose Color Status Recommended Action Power Steady green CP blade is on No action required No light LED is off CP blade is not on Ensure blade is firml...

Страница 100: ...nd connected device is functioning Flickering green amber Ethernet link is healthy and traffic is flowing through port No action required Ethernet link speed No light LED is off Ethernet link speed is...

Страница 101: ...tatus LEDs 3 QSFP port map and trunking diagram Table 9 CR16 8 blade LED descriptions LED purpose Color Status Recommended Action Power Steady green CR16 8 blade is on No action required No light LED...

Страница 102: ...r QSFP module is in all four ports have no signal no sync Ensure that the cable is properly connected If the LED remains amber contact IBM Blinking amber Port is disabled or faulted FC link activity s...

Страница 103: ...s of a blower assembly 1 Check the LED indicators on the blower assembly see Figure 27 on page 76 The LED patterns may temporarily change during POST and other diagnostic tests for information about h...

Страница 104: ...commended action Power No Light LED is off Blower assembly does not have power Ensure that the blower assembly is firmly seated and has power Steady green Blower assembly has power No action required...

Страница 105: ...ds environmental condition resolve any problems and reseat the unit If the LED continues to flash replace the unit WWN bezel and WWN card status Complete the following steps to determine the status of...

Страница 106: ...49 sequence number INFO system name FRU WWN insertion detected WWN unit removal was detected timestamp EM 1050 sequence number INFO system name FRU WWN removal detected 2 Check the LED indicators on t...

Страница 107: ...d Power below LEDs 3 Port blade Status above and Power below LEDs 2 CR blade Status above and Power below LEDs 4 Slot numbers 2 3 4 1 b768ig004 Figure 28 LEDs on the WWN card and bezel Chapter 4 Monit...

Страница 108: ...80 SAN768B 2 Installation Service and User Guide...

Страница 109: ...nd installing a power supply on page 101 v Removing and installing a blower assembly on page 103 v Removing and installing a WWN bezel and WWN card on page 104 v Removing and installing transceivers o...

Страница 110: ...SAN768B 2 can continue to operate during the replacement of the cable management comb Time and items required The replacement procedure for the cable management comb takes less than 5 minutes A 1 Phil...

Страница 111: ...nagement comb Installing a cable management comb Complete the following steps to install a cable management comb 1 Align the comb with the four screw holes on the chassis 2 Position insert and tighten...

Страница 112: ...lacement blade or filler panel v 2 Phillips screwdriver v Small form factor pluggable SFP mSFP or QSFP transceivers as needed v Optical and copper cables as needed Removing a port or application blade...

Страница 113: ...disconnecting cables 7 Disconnect all cables and transceivers from the blade For mSFP transceivers FC8 64 only it is recommended that you use the pull tab to remove the transceiver from the blade bef...

Страница 114: ...lade inside the upper and lower rail guides in the slot and slide the blade into the slot until it is firmly seated 3 Adjust the ejectors to the closed position by pulling them away from the center of...

Страница 115: ...yption optional on page 49 Removing and installing blade filler panels This section describes how to remove and install port blade filler panels Note Some filler panels have two thumbscrews and some h...

Страница 116: ...cessor blade This section describes how to remove and install a control processor CP8 blade Each SAN768B 2 has two CP8 blades located in slots 6 and 7 Attention The firmware upgrade policy for CP8 bla...

Страница 117: ...processor CP8 blade Attention If you are upgrading through multiple versions of the Fabric OS refer to Table 14 on page 94 to see which intermediate versions of the Fabric OS you will need Plan caref...

Страница 118: ...e CP blade and run the following steps a Run the haFailover command to make the standby CP blade the active blade The currently active CP blade becomes the standby blade Wait until the status LED on t...

Страница 119: ...aFailover command Wait until the failover has completed Use the haShow command to verify the CPs are synchronized and the failover is complete Depending on the nature of the CP failure it is possible...

Страница 120: ...information about determining the version of firmware on the replacement CP blade and upgrading it if necessary Attention Follow ESD precautions see ESD precautions on page xix when handling any blad...

Страница 121: ...de which must be at least 7 0 0 IBM recommends upgrading to the latest version available 3 If the firmware versions for both CP blades are the same skip to Completing the CP8 blade replacement on page...

Страница 122: ...firmwaredownload s v7 0 0a Server IP 10 31 2 25 Protocol IPv4 Checking system settings for firmwaredownload System settings check passed You are running firmwaredownload with auto reboot disabled Aft...

Страница 123: ...ully Use firmwareshow to verify the firmware versions 5 On the standby CP blade the blade for which you just changed the firmware level run reboot The reboot of the standby CP will initiate a firmware...

Страница 124: ...blade as admin if you are not still logged in and enter usbStorage e to enable the USB device 4 Remove the serial cable from the active CP blade and attach it to the standby CP blade and log in as ad...

Страница 125: ...sions 7 Ensure that you are still logged in to the standby CP blade the blade for which you just changed the firmware level and type reboot The reboot of the standby CP will initiate a firmwarecommit...

Страница 126: ...lthy HA Enabled Heartbeat Up HA State Synchronized 3 Enter firmwareShow to verify that the firmware version has been updated and that the versions are the same on the two CP blades Not all slot number...

Страница 127: ...eric system driver messages FABSYS Platform system driver messages Platform Error messages that indicate a problem with a core switch blade Function fail messages for the core switch blade master For...

Страница 128: ...flat side of the core switch blade inside the lower and upper blade guides in the slot Slide the core switch blade into the slot until it is firmly seated 3 Close the ejectors by rotating them away fr...

Страница 129: ...cedures Removing and installing a power supply Use this procedure to remove and install a power supply Attention Depending on the blade configuration of the chassis and the number of power supplies in...

Страница 130: ...the SAN768B 2 will continue operating during the replacement check the power LEDs to verify that the minimum number of power supplies is functioning See Power specifications on page 150 to determine y...

Страница 131: ...s are operating To ensure continuous adequate cooling maintain all three operating blower assemblies at all times except for the brief period when replacing a blower assembly Time and items required T...

Страница 132: ...e are two WWN cards located beneath the WWN bezel logo plate Use this procedure to remove and replace the WWN cards If IBM Support has determined that a WWN card needs to be replaced you need to power...

Страница 133: ...precautions see ESD precautions on page xix 1 Open a Telnet session to the chassis and log in to the active CP as admin The default password is password 2 Verify you are logged into the active CP Run...

Страница 134: ...front indicating the logical switch number if virtual fabrics is enabled switch FID128 root chassisshow output truncated WWN Unit 1 Header Version 2 Power Consume Factor 1 Factory Part Num 60 1000491...

Страница 135: ...ly pull them out from the chassis 11 Set the WWN cards on a static free surface such as a grounding pad Installing the WWN cards and WWN bezel logo plate Complete these steps to replace the WWN cards...

Страница 136: ...to IBM for failure analysis following local procedures Removing and installing transceivers Use the instructions in this section to remove and replace transceivers Attention Use only transceivers tha...

Страница 137: ...nect components v Ensure that cables are properly seated within the connector Kits are commercially available for cleaning and inspecting these connections IBM also offers services that will ensure op...

Страница 138: ...once you have disconnected the unit from the port Complete these steps to remove an SFP or SFP with a wire bail 1 Remove the chassis door if you have not already done so see Removing and installing t...

Страница 139: ...is aligned with the slot in the transceiver Insert the cable into the transceiver until the latching mechanism clicks Cables are keyed so that they can be inserted in only one way If a cable does not...

Страница 140: ...ace the chassis door Removing a QSFP transceiver Use this procedure to remove QSFP transceivers supported on the FC16 64 port blades and core switch blades Follow these guidelines while removing and r...

Страница 141: ...rocedure 6 Replace the chassis door Removing and installing the chassis door on page 81 Removing and replacing inter chassis link QSFP cables Use this procedure to remove and install ICL QSFP cables F...

Страница 142: ...in and all ports are online Full link is established No action required Complete this procedure to remove and install the ICL cables Refer to the Fabric OS Administrator s Guide for the configuration...

Страница 143: ...push it into the QSFP transceiver until it is firmly seated The cable housing is keyed to fit into the QSFP in one way The status LED displays steady amber until both ends of the cable are inserted an...

Страница 144: ...ly SAN768B 2 chassis are shown in the figure the chassis can be any combination of SAN384B 2 and SAN768B 2 The cabling scheme should follow the parallel example shown in the Figure 45 Each line in the...

Страница 145: ...gure 47 on page 118 The SAN768B 2 chassis are shown in the figure but either chassis can be connected In this example the three trunk groups are shown in different shades 384b089 Figure 46 Core edge I...

Страница 146: ...her the SAN768B 2 chassis requires replacement This procedure is to be completed only by a qualified IBM service representative See the following topics for specific information v Verifying necessity...

Страница 147: ...N768B 2 components such as a power supply blower assembly port blade CP blade or WWN card do not function properly even after the component is replaced v Intermittent FAULTY codes for blades Re seat t...

Страница 148: ...output Output from supportShow command Location of spptshow txt file Notes regarding supportShow output Information about the new chassis New factory serial number New serial number if available Comp...

Страница 149: ...6 Backplane IP address of CP1 10 0 0 7 switch admin switch admin 6 Display and record the manufacturer serial numbers Enter chassisShow then copy the command output into the config miscinfo txt file...

Страница 150: ...r depending on the size of the SAN This file provides a backup of all the information that might be required by Technical Support The information can be used after the SAN768B 2 is restored to the fab...

Страница 151: ...re that any component removals and installations do not affect the stability of the cabinet Attention Follow ESD precautions see ESD precautions on page xix You must remove the field replaceable units...

Страница 152: ...he order listed below Refer to Chapter 5 Removing and installing components on page 81 for more specific installation information Attention Do not force the installation of any of the components If th...

Страница 153: ...name or IP of the host for the download process to complete For more information refer to the help configdownload command or the Fabric OS Command Reference Complete the following steps to download t...

Страница 154: ...ttributes FID 128 Base Switch No Default Switch Yes Address Mode 0 Index Slot Port Address Media Speed State Proto 0 1 0 500000 4G No_Module FC 1 1 1 500100 4G No_Module FC 2 1 2 500200 4G No_Module F...

Страница 155: ...transceiver into each port if not already installed see Removing and installing transceivers on page 108 b Select the cable that corresponds to the port and position it so that the key the ridge on o...

Страница 156: ...the SANafter txt file and the information in the SANbefor txt file created earlier In particular look for differences in the following items v Device types v Number of devices v ISL and port states v...

Страница 157: ...t 5 Serial North 24 5 Serial Work 5 2 Workstation 5 Com 2 Slot 8 Port 15 North 24 8 15 RAID 16 4 RAID 16 4 Table 19 Cable routing table for SAN768B 2 64 ports shown Slot Port Cable labels Connected de...

Страница 158: ...2 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 Removing the batteries Local regulations might require removal of the battery before recycling or disposing of the product Batteries ar...

Страница 159: ...68 C66 C69 R158 R377 R381 R376 R378 R380 R375 R170 R373 R374 R365 R372 R171 R172 R366 R368 R173 R367 R175 R174 R176 R178 R177 C26 C27 R383 R382 R379 C83 C81 C82 C80 R623 R624 R516 C12 C15 C16 C19 C7 C...

Страница 160: ...See the Environmental Notices and User Guide that is shipped with the product for more information on battery recycling and disposal 1 b32e4016 Figure 49 Encryption blade battery location 132 SAN768B...

Страница 161: ...ling a QSFP and Enterprise inter chassis license on page 144 FC3632 FC3648 FC3664 Installing a 16 Gb port blade 32 48 or 64 ports These features can be installed while the SAN768B 2 is operational Att...

Страница 162: ...n the Fabric OS Administrator s Guide 4 Remove the filler panel or port blade Refer to Removing a port or application blade on page 84 or Removing a filler panel on page 87 Attention A filler panel sh...

Страница 163: ...ceable parts inside the port blade These instructions cover installing a new FC3633 or FC3649 in the SAN768B 2 chassis Time required 20 minutes or less per port blade Items required v ESD electrostati...

Страница 164: ...Verify the installation For information about how to check the status of hardware components using the CLI see the Fabric OS Administrator s Guide which is located on the product documentation CD FC3...

Страница 165: ...o Removing a port or application blade on page 84 or Removing a filler panel on page 87 6 Remove any mSFPs that are pre installed in the new FC8 64 blade and set them aside for later installation See...

Страница 166: ...Installing a port blade with incorrect alignment damages both the chassis and the replacement part v Hold the port blade by the edges of the metal pan Do not use the ejectors to hold a port blade v D...

Страница 167: ...S Administrator s Guide 5 Remove the filler panel or port blade Refer to Removing a port or application blade on page 84 or Removing a filler panel on page 87 Attention A filler panel should be remove...

Страница 168: ...uter v Port blade v Phillips screwdriver v Transceivers as needed v Optical cables as needed Procedures Follow the links in the steps below to complete this installation 1 Remove the chassis door See...

Страница 169: ...cryption Blade FS8 18 Attention Thoroughly review Planning for encryption optional on page 47 prior to installing the FS8 18 encryption blade You must also refer to the Fabric OS Encryption Administra...

Страница 170: ...ing a filler panel on page 87 Attention A filler panel should be removed only when being replaced with a blade or new filler panel Any slot that is not occupied by a blade should be occupied by a fill...

Страница 171: ...nstalled while the SAN768B 2 is operational Time required Less than 10 minutes not including installation of the ICL license or any needed configuration Items required v QSFPs and four ICL cables v FC...

Страница 172: ...rocedures Attention If ICL cables are not in use rubber plugs must be inserted in the QSFPs Follow the links in the steps below to complete this installation 1 Verify that the QSFP is the correct part...

Страница 173: ...shared memory System processor FreeScale 8548 1 2 GHz ANSI Fibre Channel protocol FC PH Fibre Channel Physical and Signalling Interface standard Classes of service Class 2 Class 3 Class F interswitch...

Страница 174: ...g and 2 1 sec blade to blade at 16 Gbps cutthrough routing FX8 24 blade FC to FC 2 1 sec any port to any port at 8 Gbps cut through routing FCOE10 24 blade CEE to CEE different ASIC and FCoE to FC 5 s...

Страница 175: ...g Extended Fabrics Fabric Watch FDMI FICON CUP Frame Redirection FSPF Integrated Routing IPFC ISL Trunking Management Server N_Port Trunking NPIV NTP v3 Port Fencing Registered State Change Notificati...

Страница 176: ...2 lb Power supply 2 45 kg 5 4 lb Blower assembly 5 73 kg 12 6 lb WWN bezel 0 3 kg 0 6 lb Cable management comb 0 45 kg 1 0 lb Chassis door 2 09 kg 4 6 lb Facility requirements The facility where the S...

Страница 177: ...half sine wave 33G 11ms duration half sine wave Vibration 0 5G p p 5 500 Hz at 1 0 octave minute 2 0G p p 5 500 Hz at 1 0 octave minute Air flow 595 cu m hr 350 cu ft min None required Heat dissipatio...

Страница 178: ...be equipped with multiple power cords To remove all hazardous voltages disconnect all power cords L003 The power subsystem is a redundant 48 V dc power distribution system with a provision for up to...

Страница 179: ...128 ports FC8 32E 8 Gb 140 Port blade 32 8 Gbps SFP up to 8 256 FC8 48E 8 Gb 160 4 Port blade 48 8 Gbps SFP up to 8 384 FC8 64 8 Gb 126 4 Port blade 64 8 Gbps SFP up to 8 512 FC16 32 16 Gb 140 Port b...

Страница 180: ...312 ft N A mSFP SFP 8 Gbps 21 m 68 ft 50 m 164 ft 150 m 492 ft 190 m 623 ft N A SFP 10 Gbps 33 m 108 ft 82 m 269 ft 300 m 984 ft 550 m 1804 ft N A mSFP SFP 16 Gbps 15 m 49 ft 35 m 115 ft 100 m 328 ft...

Страница 181: ...etermined fixed lengths The FC16 64 port blade supports the following types of cables 1 QSFP MTP connector 1 QSFP MTP connector 2 SFP LC connectors With the support for breakout cables each port can b...

Страница 182: ...154 SAN768B 2 Installation Service and User Guide...

Страница 183: ...orts supporting the Fibre Channel Routing Services and 10 external 1 Gigabit Ethernet 1 GbE or GE SFP ports supporting the Fibre Channel Over IP FCIP feature There are also 2 licensable external 10 Gi...

Страница 184: ...RTT on a limited number of GbE ports v Adaptive Rate Limiting licensable v TCP performance graphing in Web Tools v FCIP tunnels A maximum of ten FCIP tunnels for all GbE ports Four tunnels maximum pe...

Страница 185: ...itching v Standard Ethernet encapsulation v End of row deployment Note With the switch in online state if all the FCOE ports are shut down offline diagnostics continue to run The blade will be in inop...

Страница 186: ...158 SAN768B 2 Installation Service and User Guide...

Страница 187: ...ore information v Review RASlog entries v Enter the sensorShow command to determine the status of the hardware components v Run diagnostic tests see Diagnostics on page 161 v Reboot the SAN768B 2 or p...

Страница 188: ...hassis can be rebooted using the reboot to reboot each CP individually or fastBoot commands The fastBoot command reboots the switches without running POST If the active CP blade is rebooted it fails o...

Страница 189: ...nel Note Diagnostic error messages do not necessarily indicate that the system requires maintenance Each diagnostic test can be implemented by entering the related command through a Telnet or serial s...

Страница 190: ...or a link cannot be established There might be a conflict with the CP Ethernet link speed negotiation set up by the network Specify the CP Ethernet link speed by entering the ifModeSet command Configu...

Страница 191: ...tatus For more information enter the diagDisablePost command then the slotPowerOff slot number and slotPowerOn slot number commands Resolve the source of the problem or replace the blade as required P...

Страница 192: ...d on a particular link it is easy to jump to the conclusion that the link is causing the network issue This might not be the case Since CRC errors are just symptoms of a link issue we need to trace th...

Страница 193: ...f the transceiver is abnormally low it is recommended that you v Re seat the transceivers for the failing link v Clean the connector and optical fiber Most link issues are solved by completing these s...

Страница 194: ...s the transmit power of 8 Gbps 4 Gbps LWL transceivers may be observed to overdrive the 3dB maximum receive power of 2 Gbps LWL transceivers If the maximum receive power of the 2 Gbps LWL transceiver...

Страница 195: ...Receive Power dB Avago AFCT 57M5ATPZ 2 Gbps 10 km SFP 3 dB Finisar FTLF1319P1xTL 2 Gbps 10 km SFP 0 dB Finisar FTRJ1319P1xTL 2 Gbps 10 km SFP 0 dB JDSU JSH 12L1DD1 2 Gbps 10 km SFP 1 dB Hitachi Cable...

Страница 196: ...168 SAN768B 2 Installation Service and User Guide...

Страница 197: ...rrors are detected on a port over a certain period of time This counter includes CRC and other bit errors for internal ports The Cx 1010 event message has the following format Message timestamp Cx 101...

Страница 198: ...ber is S3 meaning that the port blade in slot 3 potentially has the issue and blade port number is 1 The error message includes a timestamp with the format year month date hh mm ss ms The error messag...

Страница 199: ...sure that the power LED on the core blade has gone off for one minute and then re slide the black slider switch back up to the on position If physical access to the box is not possible the core blade...

Страница 200: ...D128 admin slotshow Slot Blade Type ID Status 1 AP BLADE 75 ENABLED 2 UNKNOWN VACANT 3 CORE BLADE 99 ENABLED 4 CP BLADE 50 ENABLED 5 CP BLADE 50 ENABLED 6 CORE BLADE 99 ENABLED 7 UNKNOWN VACANT 8 SW B...

Страница 201: ...N VACANT 3 CORE BLADE 99 ENABLED 4 CP BLADE 50 ENABLED 5 CP BLADE 50 ENABLED 6 CORE BLADE 99 ENABLED 7 UNKNOWN VACANT 8 SW BLADE 96 POWERING UP SAN384B 2 FID128 admin slotshow SAN384B 2 FID128 admin s...

Страница 202: ...174 SAN768B 2 Installation Service and User Guide...

Страница 203: ...8 shows an FC16 32 port blade v Figure 55 on page 179 shows an FC16 48 port blade v Figure 56 on page 180 shows an FC16 64 port blade v Figure 57 on page 181 shows an FC8 32E port blade v Figure 58 on...

Страница 204: ...orts 8 11 bottom to top Table 33 on page 177 shows the mappings from the numbered ports on the face of the core blade to the port mappings as shown by the slotShow command Each external port maps to f...

Страница 205: ...re blades External port number slotShow port numbers External port number slotShow port numbers 0 0 3 8 32 35 1 4 7 9 36 39 2 8 11 10 40 43 3 12 15 11 44 47 4 16 19 12 48 51 5 20 23 13 52 55 6 24 27 1...

Страница 206: ...1 Power LED 3 FC ports 16 31 bottom to top 2 Status LED 4 FC ports 0 15 bottom to top 2 1 5 3 4 6 7 b768ig071 Figure 54 FC16 32 port blade 178 SAN768B 2 Installation Service and User Guide...

Страница 207: ...1 Power LED 3 FC ports 24 47 bottom to top 2 Status LED 4 FC ports 0 23 bottom to top 2 3 1 4 b768ig072 Figure 55 FC16 48 port blade Appendix E Blade port numbering 179...

Страница 208: ...1 QSFP ports 8 15 FC ports 32 63 bottom to up 2 QSFP ports 0 7 FC ports 0 31 bottom to up 1 2 63 0 32 31 b768ig080 Figure 56 FC16 64 port blade 180 SAN768B 2 Installation Service and User Guide...

Страница 209: ...wer LED 5 FC ports 16 23 2 Status LED 6 FC ports 0 7 3 FC ports 24 31 7 FC ports 8 15 4 Port and trunking group map 2 1 5 3 4 6 7 b768ig077 Figure 57 FC8 32E port blade Appendix E Blade port numbering...

Страница 210: ...1 Power LED 3 FC ports 24 47 2 Status LED 4 FC ports 0 23 2 3 1 4 b768ig078 Figure 58 FC8 48E port blade 182 SAN768B 2 Installation Service and User Guide...

Страница 211: ...5 Port 61 LED 2 Status LED 6 Port 29 LED 3 Port 63 7 FC ports 32 63 bottom to top 4 Port 30 8 FC ports 0 31 bottom to top 8 7 3 2 1 5 6 4 b768ig057 Figure 59 FC8 64 port blade Appendix E Blade port n...

Страница 212: ...1 Gigabit Ethernet ports GE0 GE1 3 FC ports 0 15 bottom to top 2 Key management smart card interface b768ig061 3 2 1 Figure 60 FS8 18 encryption blade 184 SAN768B 2 Installation Service and User Guide...

Страница 213: ...LED 6 FC ports 6 11 numbered bottom to top 3 1 GbE ports 0 3 numbered bottom to top 7 1 GbE ports 4 9 numbered bottom to top 4 10 GbE ports 0 1 numbered bottom to top 3 4 5 6 7 1 2 b768ig034 Figure 61...

Страница 214: ...1 10 GbE CEE ports 12 23 3 Power LED 2 10 GbE CEE ports 0 11 4 Status LED 2 1 4 3 b768ig082 Figure 62 FCOE10 24 FCoE blade 186 SAN768B 2 Installation Service and User Guide...

Страница 215: ...CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON INFRINGEMENT MERCHANTABILITY OR FITNESS F...

Страница 216: ...ademarks and logos are trademarks of Sun Microsystems Inc in the United States other countries or both Linux is a registered trademark of Linus Torvalds in the United States other countries or both Mi...

Страница 217: ...tion requirements resulting from a non recommended modification of the product including the fitting of non IBM option cards Attention This is an EN55022 Class A product In a domestic environment this...

Страница 218: ...gt in bereinstimmung mit dem Deutschen EMVG das EG Konformit tszeichen CE zu f hren Verantwortlich f r die Einhaltung der EMV Vorschriften ist der Hersteller International Business Machines Corp New O...

Страница 219: ...t has obtained EMC registration for commercial use In the event that it has been mistakenly sold or purchased please exchange it for equipment certified for home use Russia Electromagnetic Interferenc...

Страница 220: ...192 SAN768B 2 Installation Service and User Guide...

Страница 221: ...cable routing tables 128 cables attaching 44 installing 44 managing 46 caution notices xvi definition xvi examples xvi chassis installing components 124 removing 123 replacing 123 status 159 chassis...

Страница 222: ...nhanced 48 port 8 Gbps blade 135 FC3664 installing 64 port blade 133 FC3864 installing 64 port blade 136 FC3880 installing FCOE10 24 blade 138 FC3890 FX8 24 blade 140 FC3895 FS8 18 encryption blade 14...

Страница 223: ...POST results 160 introduction 1 IP addresses SAN768B 2 38 IP addresses SAN768B 2 38 L labels safety xviii LEDs blower assembly 75 core switch blade 72 CR16 8 blade 72 encryption blades 68 port blades...

Страница 224: ...ade 84 FC8 32E 84 FC8 32E blade 84 FC8 48E 84 FC8 48E blade 84 FC8 64 84 FCOE10 24 84 filler panel 87 FS8 18 84 FS8 18 blade 84 FX8 24 84 FX8 24 blade 84 inter chassis link QSFP cable 113 115 mSFP 108...

Страница 225: ...e routing 128 Taiwan contact information xi time setting 41 time required installing a SAN768B 2 16 time server NTP 42 time zone setting 42 tools required installing a SAN768B 2 17 trademarks 188 tran...

Страница 226: ...198 SAN768B 2 Installation Service and User Guide...

Страница 227: ......

Страница 228: ...Part Number 00MA746 Printed in USA GA32 0893 06 1P P N 00MA746...

Отзывы:

Нет отзывов

Похожие инструкции для SAN768B-2

TotalStorage DS4800

Бренд: IBM Страницы: 250

TotalStorage DS4800

Бренд: IBM Страницы: 234

Бренд: IBM Страницы: 136

System Storage DS5020

Бренд: IBM Страницы: 31

Netfinity EXP300

Бренд: IBM Страницы: 93

Бренд: IBM Страницы: 270

Ultrastar 36LZX

Бренд: IBM Страницы: 2

DCAS-32160 - Ultrastar 2.1 GB Hard Drive

Бренд: IBM Страницы: 232

System Storage TS3310

Бренд: IBM Страницы: 531

Azure StorSimple 8100

Бренд: Microsoft Страницы: 12

Бренд: trueCall Страницы: 16

Бренд: PNY Страницы: 17

Бренд: Thecus Страницы: 120

TeraStation WSS WS5020N6

Бренд: Buffalo Страницы: 115

Бренд: Spectra Logic Страницы: 15

Бренд: Netstor Страницы: 17

Barracuda 5400.1 ST340015A

Бренд: Seagate Страницы: 2

Бренд: AEG Страницы: 16

Бренды по названию

0 1 2 3 4 5 6 7 8 9 A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Популярные бренды

Загрузить еще бренды