manualshive.com logo in svg

Bull Escala Power7 Series, Руководство пользователя

Поделиться
Скачать
Bull Escala Power7 Series Скачать руководство пользователя страница 31

5.

To add expansion units to the end of an existing SPCN loop, do the following:

a.

Move the SPCN cable from connector SPCN 1 on the system unit to connector SPCN 0 on one of
the expansion units to be added.

b.

Connect another SPCN cable to connector SPCN 1 on the expansion unit added in step 5a.

c.

If there is another expansion unit to add, do the following:

1)

Find the other end of the cable attached to connector SPCN 1 on the expansion unit just
added. Connect this end to connector SPCN 0 on the next expansion unit to be added.

2)

Connect another SPCN cable to connector SPCN 1 on the expansion unit added in step 5c1.

d.

Repeat step 5c for each additional expansion unit.

e.

Connect the other end of the cable from connector SPCN 1 on the last expansion unit added to
connector SPCN 1 on the system unit.

f.

Return to the procedure that sent you here.

Examples: SPCN connections

Review examples that show how SPCN cables are connected to expansion units and system units for
form SPCN loops.

Note:

The physical connectors on the system units and expansion units might not have labels, or might

have labels that are different from what is shown in the examples. In all cases, the top or left SPCN
connector corresponds to SPCN 0 in the examples, and the bottom or right SPCN connector corresponds
to SPCN 1 in the examples.

Figure 3. Example of SPCN connections

I/O expansion units

19

Содержание Escala Power7 Series

Страница 1: ...Enclosures and expansion units ESCALA Power7 REFERENCE 86 A1 23FF 07 ...

Страница 2: ......

Страница 3: ...8231 E2C Bull Escala E4 700 E4 700T Power 740 8205 E6B Bull Escala E4 705 Power 740 8205 E6C Bull Escala E5 700 Power 750 8233 E8B Bull Escala M6 700 Power 770 9117 MMB Bull Escala M6 705 Power 770 9117 MMC Bull Escala M6 715 Power 770 9117 MMD Bull Escala M7 700 Power 780 9179 MHB Bull Escala M7 705 Power 780 9179 MHC Bull Escala M7 715 Power 780 9179 MHD References to Power 755 8236 E8C models a...

Страница 4: ...f the proprietors of the trademarks mentioned in this manual All brand names and software and hardware product names are subject to trademark and or patent protection Quoting of brand and product names is for information purposes only and does not represent trademark misuse The information in this document is subject to change without notice Bull will not be liable for errors contained herein or f...

Страница 5: ... unit nonconcurrently 38 Disk drive enclosures 41 Overview for disk drive enclosures 41 31D 24 or 31T 24 SCSI disk drive enclosure 41 Connecting the 31D 24 or 31T 24 to a system running AIX 41 Connecting and configuring the 31D 24 or 31T 24 SCSI disk drive enclosure in an AIX clustered environment 45 Connecting and configuring the disk drive enclosure in a system running Linux 47 Connecting and co...

Страница 6: ...em that is not managed by an HMC or an SDMC 109 Stopping a system by using the HMC 111 Removing an enclosure from the rack 112 Connector locations 115 Server connectors 115 Model 02E 4B connector locations 115 Model 02E 4C connector locations 116 Model 05F 6B connector locations 117 Model 05E 6C connector locations 118 Model 31E 1C connector locations 118 Model 31E 2C connector locations 119 Model...

Страница 7: ... this requirement applies to your country a safety information booklet is included in the publications package shipped with the product The booklet contains the safety information in your national language with references to the U S English source Before using a U S English publication to install operate or service this product you must first become familiar with the related safety information in ...

Страница 8: ...quipment that will be attached to this product to properly wired outlets v When possible use one hand only to connect or disconnect signal cables v Never turn on any equipment when there is evidence of fire water or structural damage v Disconnect the attached power cords telecommunications systems networks and modems before you open the device covers unless instructed otherwise in the installation...

Страница 9: ...he customer to ensure that the outlet is correctly wired and grounded to prevent an electrical shock CAUTION v Do not install a unit in a rack where the internal rack ambient temperatures will exceed the manufacturer s recommended ambient temperature for all your rack mounted devices v Do not install a unit in a rack where the air flow is compromised Ensure that air flow is not blocked or reduced ...

Страница 10: ...ou choose can support the weight of the loaded rack cabinet Refer to the documentation that comes with your rack cabinet for the weight of a loaded rack cabinet v Verify that all door openings are at least 760 x 230 mm 30 x 80 in v Ensure that all devices shelves drawers doors and cables are secure v Ensure that the four leveling pads are raised to their highest position v Ensure that there is no ...

Страница 11: ... This product might contain one or more of the following devices CD ROM drive DVD ROM drive DVD RAM drive or laser module which are Class 1 laser products Note the following information v Do not remove the covers Removing the covers of the laser product could result in exposure to hazardous laser radiation There are no serviceable parts inside the device v Use of the controls or adjustments or per...

Страница 12: ...The following comments apply to the servers that have been designated as conforming to NEBS Network Equipment Building System GR 1089 CORE The equipment is suitable for installation in the following v Network telecommunications facilities v Locations where the NEC National Electrical Code applies The intrabuilding ports of this equipment are suitable for connection to intrabuilding or unexposed wi...

Страница 13: ...on unit must all be DDR capable While components that are strictly SDR capable cannot run at DDR components that are DDR capable can be configured by the system firmware to run at SDR if one or more of the other components in the interconnection are strictly SDR capable For example a DDR capable I O expansion unit that is connected to a GX adapter that has a strictly SDR capable I O hub chip will ...

Страница 14: ...xpansion units can result in unintended I O configuration changes including a change to the bus numbers assigned to existing expansion units When bus numbers change partition profiles cannot find existing I O resources If this procedure completes with unexpected results contact your next level of support Your next level of support can attempt to restore bus numbers provided that you obtained a rec...

Страница 15: ...for the expansion unit The system uses 12X SDR or 12X DDR cables to send and receive customer data and auxiliary control information to and from the expansion unit and to download expansion unit firmware The SPCN is used to control power to the expansion units and is used as a backup to the 12X cables for auxiliary control and firmware download 12X SDR cables Observe the following guidelines when ...

Страница 16: ... connecting expansion units to system units When you set up an expansion unit observe the configuration rules in the following list v Expansion units must be connected to system units with 12X cables in a loop topology Doing so produces redundancy in the 12X connections so that a single cable failure does not result in loss of communication between the system unit and one or more expansion units v...

Страница 17: ...number of loops 2 loops 14G 30 4 58 02 2 58 77 2 14G 30 8 58 02 4 58 77 4 31E 1C Note The related system 31E 2B does not support expansion units 58 02 58 77 Maximum number of loops 1 loop 58 02 2 58 77 2 58 02 2 58 77 2 31E 2C Note The related system 31E 2B does not support expansion units 58 02 58 77 Maximum number of loops 1 loop 58 02 2 58 77 2 58 02 2 58 77 2 33E 8B 14G 30 58 02 58 77 The maxi...

Страница 18: ... 4 58 02 2 58 77 2 14G 30 32 58 02 16 58 77 16 17M MC 14G 30 58 02 58 77 The maximum number of loops is equal to two loops per processor drawer in the system configuration 1 processor drawer 2 loops 2 processor drawers 4 loops 3 processor drawers 6 loops 4 processor drawers 8 loops 14G 30 4 58 02 2 58 77 2 14G 30 32 58 02 16 58 77 16 17M MD 14G 30 58 02 58 77 The maximum number of loops is equal t...

Страница 19: ...ssor books 28 loops 8 processor books 32 loops 57 98 1 58 03 1 58 73 1 57 98 32 58 03 32 58 73 32 79M HB 14G 30 58 02 58 77 The maximum number of loops is equal to two loops per processor drawer in the system configuration 1 processor drawer 2 loops 2 processor drawers 4 loops 3 processor drawers 6 loops 4 processor drawers 8 loops 14G 30 4 58 02 2 58 77 2 14G 30 32 58 02 16 58 77 16 79M HC 14G 30...

Страница 20: ... To add expansion units follow these steps 1 Based on your system configuration and power requirements choose one of the following options for installing your expansion unit Important For more information about the system power requirements for your configuration see Overview for I O expansion units http publib boulder ibm com infocenter systems scope hw topic p7hamconcepts htm v If you are connec...

Страница 21: ...ib boulder ibm com infocenter systems scope hw topic p7hby asmi htm b In the ASMI navigation area expand System Configuration and select Configure I O Enclosures v If the SPCN Firmware Update Policy shows Expanded record the current setting so you can restore it later Then change the setting to Enabled v If the SPCN Firmware Update Policy shows Enabled or Disabled continue without changing the set...

Страница 22: ...the expansion unit that just powered on After the green power indicators are lit on the control panels of all the expansion units that were just powered on wait 10 additional minutes for hardware initialization to complete before proceeding to step 11 10 To start the system or logical partition see Starting the system or logical partition http publib boulder ibm com infocenter systems scope hw top...

Страница 23: ...y that existed before the expansion units were added v If your system is not managed by an HMC or SDMC see Verifying the 12X loops without an HMC or SDMC http publib boulder ibm com infocenter systems scope hw topic p7ham ipham_verifyriohslloopwohmc htm Note While concurrently adding an expansion unit it is normal for error logs and event logs to be created with the following 12X related reference...

Страница 24: ... mm dd is the day on which the procedure occurred Press Enter 3 Continue with step 14b b Specify event criteria that includes events that were generated during the procedure c Close any serviceable events generated during the procedure that include one of the reference codes listed in step 11 on page 10 or step 13 on page 11 d Perform problem analysis on any remaining open serviceable events 15 St...

Страница 25: ...isting 12X loop see Adding expansion units to an existing 12X loop on page 14 Adding expansion units to a new 12X loop Use this procedure when adding new expansion units creates a new 12X loop as opposed to being added to an existing 12X loop Before you begin Unused 12X connectors might be covered with metal clips Remove the clips before you install the 12X cables About this task Complete the foll...

Страница 26: ...I O resources might require additional configuration changes If your system is currently powered off you can prevent bus number changes by powering on the system to the firmware standby state before completing the steps to add the expansion units At firmware standby state system firmware is active but all partitions remain powered off Use one of the following procedures to power on the system to f...

Страница 27: ...by state a In the ASMI navigation area expand Power Restart Control b Click Power On Off System c Record the current setting for the Server firmware start policy field so that you can restore the setting later in this procedure d Set the Server firmware start policy field to Standby User Initiated 3 Click Save settings and power on 4 Periodically click Power On Off System again to refresh the info...

Страница 28: ...ector 1 on the expansion unit just added Connect that end to connector 0 on the next expansion unit to be added 2 Connect another cable to connector 1 on the expansion unit added in step 6c1 d Repeat step 6c for each additional expansion unit e Find the other end of the cable from connector 1 on the last expansion unit added Connect that end to connector 0 on the expansion unit that was disconnect...

Страница 29: ...pansion units into the system power control network SPCN Before you begin v Find the SPCN cables that were included with your expansion units v Review Examples SPCN connections on page 19 and then return here and continue with the instructions Figure 1 Example of a server that has one GX adapter connected to two expansion units Figure 2 Example of a server that has two GX adapters each connected t...

Страница 30: ...nnect this cable end to connector SPCN 0 on one of the expansion units to be added b Connect another SPCN cable to connector SPCN 1 on the expansion unit added in step 3a c If there is another expansion unit to add do the following 1 Find the other end of the cable attached to connector SPCN 1 on the expansion unit just added Connect this end to connector SPCN 0 on the next expansion unit to be ad...

Страница 31: ...xpansion unit added in step 5c1 d Repeat step 5c for each additional expansion unit e Connect the other end of the cable from connector SPCN 1 on the last expansion unit added to connector SPCN 1 on the system unit f Return to the procedure that sent you here Examples SPCN connections Review examples that show how SPCN cables are connected to expansion units and system units for form SPCN loops No...

Страница 32: ...lick Servers c Select the system with which you want to work Information about the system is shown under the Tasks section in the content area v If your system is managed by an SDMC complete the following steps a From the navigation area expand Hosts b Click the server with which you want to work Information about the system is shown under the Actions menu 3 To verify the new expansion units compl...

Страница 33: ...ayed contact your next level of support for assistance 4 Return to the procedure that sent you here Verifying the 12X loops with an HMC or SDMC If your system is managed by a Hardware Management Console HMC or Systems Director Management Console SDMC learn how to verify that the 12X loops have no failed links About this task To verify that the 12X loops do not have failed links on a system that is...

Страница 34: ...d A row must be repaired if either the Leading Port Status column or Trailing Port Status column contains a value other than Operational Exception Exclude any rows that represent nodes belonging to a loop with no I O expansion units connected The Hardware Topology data displays a loop with no I O expansion units as a bus with only one associated row That row contains a Node Type of Local NIC a Lea...

Страница 35: ... the Resources tab b Expand Hosts c Right click the host and clickService and Support Reference Code History d Scan for any serviceable events that include 10009xxx reference codes and perform one of the following actions v If there are no serviceable events that include 10009xxx reference codes close the Reference Code History window v If serviceable events include 10009xxx reference codes there ...

Страница 36: ...x reference codes press Enter to return to the command line v If there are problems with 10009xxx reference codes there might be problems with the SPCN loop Perform problem analysis on the problems to correct them before continuing If you need assistance contact your service provider f Go to step 8 7 Use the Linux error log to verify that there are no 10009xxx errors where x is any character 0 9 o...

Страница 37: ...ill not automatically recognize the MTMS value corrections that you are about to make For these partitions to recognize the new values you will be required to take additional action that is potentially disruptive Before proceeding with this step review step 10e to determine whether it is possible to take the required actions at this time If it is not continue to step 11 on page 26 without performi...

Страница 38: ...nit If these values are not set correctly the system might not manage the expansion unit properly To set the configuration ID use the Advanced System Management Interface ASMI However if the ASMI is not available you can use the physical control panel to set the configuration ID To set the MTMS value you must use the ASMI not the control panel However if you do not have access to the ASMI the syst...

Страница 39: ...or 14G 30 and 57 96 expansion units v 0x8E for 58 02 and 58 77 expansion units b Compare the Type Model and Serial number column values for the expansion unit you are working with to the type model and serial values on the expansion unit label The expansion unit location code displayed in the Location code column appears in one of the following formats v UTMPx xxx xxxxxxx where x is any character ...

Страница 40: ...d in the Power Control Network Identifier Type Model Serial number and Location code columns for the expansion unit you are working with by expanding System Configuration Configure I O Enclosures Do not use the Back button to do this 16 If the system is powered on to firmware standby state and you entered a new power control network identifier configuration ID in step 12 the expansion unit will po...

Страница 41: ...loaded to the expansion units that require an update 6 Change the SPCN Firmware Update Policy setting back to the setting that was recorded in step 3 and click Save Policy Setting Notes v To monitor the progress of the SPCN firmware update click Configure I O Enclosures The window is refreshed Do not use the browser Back or Refresh button to monitor the update progress The Power Control Network Fi...

Страница 42: ...3Gb SAS RAID Adapters Four FC 5901 PCIe Dual x4 SAS Adapters Linux Either a FC 5901 PCIe Dual x4 SAS Adapter or two FC 5903 PCIe Dual x4 3Gb SAS RAID Adapters Either two FC 5901 PCIe Dual x4 SAS Adapters or two FC 5903 PCIe Dual x4 3Gb SAS RAID Adapters Four FC 5901 PCIe Dual x4 SAS Adapters Note FC means feature code Tip If you are planning to configure RAID arrays ensure that you have the minimu...

Страница 43: ...ms this must be a RAID configuration v You must have a minimum of one disk drive per hard disk drive partition One hard disk drive partition with 18 disks and a single SAS adapter For related physical cabling see Figure 6 on page 33 Operating system AIX or Linux SAS adapter One FC 5901 PCIe Dual x4 SAS Adapter Hard disk drive partitions switch position 1 Cables One FC 3688 SAS cable AT 0 6 meter T...

Страница 44: ...al SAS connectivity to all drives For related physical cabling see Figure 10 on page 34 Operating system AIX or Linux SAS adapter Either two FC 5901 PCIe Dual x4 SAS Adapters or two FC 5903 PCIe Dual x4 3Gb SAS RAID Adapters Hard disk drive partitions switch position 1 Cables Two FC 3688 SAS cables AT 0 6 meter Note For AIX and Linux operating systems this must be a RAID configuration Four hard di...

Страница 45: ...ch position 2 Figure 6 Physical cabling example One hard disk drive partition with 18 disks a single SAS adapter and hard disk drive partitions switch position 1 Figure 7 Physical cabling example Two hard disk drive partitions with nine disks per hard disk drive partition a single SAS adapter and hard disk drive partitions switch position 2 I O expansion units 33 ...

Страница 46: ...position 2 Figure 9 Physical cabling example Two hard disk drive partitions with nine disks per hard disk drive partition four SAS adapters and hard disk drive partitions switch position 2 Figure 10 Physical cabling example One hard disk drive partition with 18 disks dual SAS adapters and hard disk drive partitions switch position 1 34 Enclosures and expansion units ...

Страница 47: ...out removing an I O expansion unit from a system unit concurrently Before you begin Perform the following tasks before beginning this procedure v Record the time that you start the procedure The time is referenced later during error log analysis v Before performing any procedure that involves changes to the 12X cabling or configuration for a system that is managed by an HMC obtain a record that id...

Страница 48: ...to activate and deactivate identify indicators 4 Select from the following options v If the system is not in manufacturing default configuration MDC continue with step 5 v If the system is in manufacturing default configuration MDC continue with step 7 5 If the system is not in manufacturing default configuration MDC use one of the following methods to ensure that none of the I O resources in the ...

Страница 49: ...moved and attach it to connector T1 on the system unit Important This step creates a 12X loop with no expansion units on it The 12X cable runs between connectors T1 and T2 on the system unit Do not skip this step It is necessary for the system firmware to complete the removal process c Wait 2 minutes d Remove the 12X cable between connectors T1 and T2 on the system unit e Continue with step 15 14 ...

Страница 50: ...mestamps indicate that they occurred during the removal procedure and if the resources that they reference were involved in the procedure v 100090F0 This permanent reference code indicates that an expansion unit was dropped from the power control network v 10009135 This informational or permanent reference code indicates an open SPCN loop v 10009137 This permanent reference code indicates disrupti...

Страница 51: ... or a system unit b Disconnect the SPCN cable from the SPCN 1 connector on the expansion unit that is to be removed Attach that end to the connector on the preceding unit on the SPCN loop that was opened in the previous step 5 Familiarize yourself with the 12X cabling rules For 12X examples see Examples 12X connections on page 16 6 Choose from the following options for removing an expansion unit f...

Страница 52: ...Serviceability Manage Serviceable Events b In the Manage Serviceable Events window click OK to accept the default criteria A table is shown with the serviceable events that match the criteria c Continue with step 12 v If the system is managed by an SDMC complete the following steps to check for new serviceable events a On the Welcome page click the Manage tab b Select Service and Support Manager S...

Страница 53: ...connect your SCSI cables to either a single initiator repeater card or to a dual initiator repeater card on your SCSI disk drive enclosure The dual initiator repeater card allows for high availability and combining groups of disk drive slots Notes Before you connect the enclosure read the following information v You must install the SCSI adapter that is used to connect the enclosure to your system...

Страница 54: ...eater cards install your SCSI cable A to the initiator repeater card B v Single SCSI adapter connected to single repeater card If in this case you are not sharing resources disk drives simply attach each SCSI cable to each repeater card v Two SCSI adapters connected to dual repeater cards If in this case you are sharing resources disk drives between systems or logical partitions ensure that you un...

Страница 55: ...31D 24 or 31T 24 SCSI disk drive enclosure in an AIX clustered environment on page 45 v Single SCSI adapter connected to a combination of single and dual repeater cards If in this case you are sharing resources disk drives between systems or logical partitions ensure that you Figure 13 SCSI cable to a dual repeater card that is connected with two SCSI cards Figure 14 SCSI cable to a dual repeater ...

Страница 56: ...d to the attached system The AIX CRON job must be set up during the initial installation Note Errors on disks in the enclosure are found without this CRON job script through normal automatic error log analysis However other errors such as fan and power supply problems in the enclosure are found only when you run diagnostics manually You can run diagnostics manually by using the following scripts T...

Страница 57: ... the usr lpp diagnostics bin directory with the following contents bin ksh Name run_ses_healthcheck Location usr lpp diagnostics bin Function SCSI SES hourly heathcheck for i in lsdev Cc container t ses s scsi F name S available do diag cd i dev null if ne 0 then usr lpp diagnostics bin diagrpt o tmp ses health output you might want to process the output prior to placing it in a file somhow notify...

Страница 58: ...gical name of the adapter to record the I O slot physical slot that each SCSI adapter uses for example lscfg vpl scsi0 where scsi0is the logical name of the adapter v Use the lsattr command as in the following example to find the ID of the adapter scsi0 lsattr E l scsi0 grep id Note Do not use wildcard characters or full path names on the command line for the device name designation In the resulti...

Страница 59: ... port of the adapter on Host A SCSI address 6 to the interior port on the repeater card in slot C4 on the enclosure This connection creates a set of shared disks and a shared SCSI bus such as scsi1 between Host A and Host B The important thing to consider at this point is that this is an independent SCSI bus or set of shared disks from the first set Ensuring that you do not mistakenly connect one ...

Страница 60: ...ter cardSCSI repeater card 2 Connect the SCSI cable to your repeater card Choose one of the following options based on your situation v If you are using only single repeater cards or only dual repeater cards connect your SCSI cable A to the initiator repeater card B For more details see Figure 17 Figure 18 on page 49 or Figure 19 on page 49 depending on your situation Figure 17 SCSI cable to a sin...

Страница 61: ...connect the dual repeater card A to the single repeater card B by using a different SCSI cable For more information see Figure 20 on page 50 Figure 18 SCSI cable to a dual repeater card that is connected with two SCSI cards Figure 19 SCSI cable to a dual repeater card that is connected with four SCSI cards Disk drive enclosures 49 ...

Страница 62: ...the SCSI adapter D on the server For more details see Figure 21 4 Add the disk drives to the operating system that you are using See Disk drives and solid state drives http publib boulder ibm com infocenter systems scope hw topic p7hal arebykickoff htm Figure 20 SCSI cable to a dual repeater card that is connected to a single repeater card Figure 21 SCSI cable to a SCSI interface card 50 Enclosure...

Страница 63: ... the configuration of the system to which the enclosure is attached This CRON runs the diagnostic application diag_encl daily at 3 a m The application reads the contents of the etc diagela diagela config file to determine which applications have been registered to be notified of errors To add additional notification methods or for more information see that file By default the following notificatio...

Страница 64: ...sure are long enough to allow concurrent maintenance of disk drives v If you are using only single repeater cards or only dual repeater cards connect your SCSI cable A to the initiator repeater card B v If you are using a combination of single and dual repeater cards connect a SCSI cable to the dual initiator repeater card A Then connect the dual repeater card A to the single repeater card B with ...

Страница 65: ...this step Connect the other end of the SCSI cable C to the SCSI adapter D on the server For more details see Figure 24 SCSI repeater card Learn about where to place and how to install the SCSI repeater card Figure 23 SCSI cable to a dual repeater card that is connected to a single repeater card Figure 24 SCSI cable to a SCSI interface card Disk drive enclosures 53 ...

Страница 66: ...Use the following figures and instructions to place your repeater cards Figure 25 Repeater card location of the and 31D 24 SCSI disk drive enclosure drawer models 54 Enclosures and expansion units ...

Страница 67: ...ard and 0 dual repeater cards C5 2 single repeater cards and 0 dual repeater cards Place in order C5 and C2 3 single repeater cards and 0 dual repeater cards Place in order C5 C2 and C4 4 single repeater cards and 0 dual repeater cards Place in order C5 C2 C4 and C3 1 single repeater card and 1 dual repeater card Place in C5 first and then place dual repeater card C4 1 single repeater card and 2 d...

Страница 68: ...der C4 and C3 0 single repeater cards and 1 dual repeater card C4 0 single repeater cards and 2 dual repeater cards Place in order C4 and C5 0 single repeater cards and 3 dual repeater cards Place in order C4 C5 and C3 0 single repeater cards and 4 dual repeater cards Restriction This configuration is for servers models only Place in order C4 C5 C3 and C2 The following figures show the locations o...

Страница 69: ...w repeater card 58 86 SAS disk drive enclosure Learn about the SAS disk drive enclosure and how to assemble it 1 Review the following information The SAS disk drive enclosure can hold up to 12 disk drives The enclosure cannot be split into two independent groups The SAS disk drive enclosure supports the following operating systems v AIX v Linux v VIOS To determine the level of software that you ne...

Страница 70: ...8231 E2B 8231 E1C or 8231 E2C http publib boulder ibm com infocenter systems scope hw topic p7hal installsasport_72x_74x htm v For installing the external SAS port on the 7 20 02E 4B and 02E 4C or 7 40 05F 6B and 05E 6C see Installing the external SAS port on an 8202 E4B 8202 E4C 8205 E6B or 8205 E6C system http publib boulder ibm com infocenter systems scope hw topic p7hal installsasport_72x_74x ...

Страница 71: ...o a system with support for a single SAS drawer go to step 8 on page 61 If your SAS configuration requirements are not supported by any of these options go to step 9 on page 62 5 Perform the connection of one 58 86 disk drive enclosure A to a single SAS adapter C by using a YO cable B as shown in Figure 29 Go to step 9 on page 62 6 Perform the connection of two 58 86 disk drive enclosures A and C ...

Страница 72: ...n be attached to an additional 58 86 disk drive enclosure B and D by using two EE cables E and F as shown in Figure 31 on page 61 Figure 30 Connection of two 58 86 disk drive enclosures to a single SAS adapter by using YO cables where each enclosure can be attached to an additional 58 86 disk drive enclosure by using two EE cables 60 Enclosures and expansion units ...

Страница 73: ...to a system with support for a single SAS drawer C as shown in Figure 32 on page 62 Figure 31 Connection of two 58 86 disk drive enclosures to a SAS adapter pair by using X cables where each enclosure can be attached to an additional 58 86 disk drive enclosure by using two EE cables Disk drive enclosures 61 ...

Страница 74: ...ttp publib boulder ibm com infocenter systems scope hw topic p7haj p7hajhsmverify htm 58 87 SAS disk drive enclosure Learn about the SAS disk drive enclosure and how to assemble it 1 Review the following information The SAS disk drive enclosure can hold up to 24 disk drives The enclosure can be split logically into one two or four independent groups The SAS disk drive enclosure supports the follow...

Страница 75: ...osure v Power off the logical partitions or systems that own the adapters to which you are connecting the 58 87 disk drive enclosure To do one of these required actions choose one of the following options Important If you are connecting to two SAS adapters or to the two internal RAID controllers of a PCIe storage enclosure both of the SAS adapters or RAID controllers must be powered off or unconfi...

Страница 76: ... using information printed on stickers at the rear of the enclosure Stickers are attached to the lower left shelf of the chassis A and the center support between the enclosure services manager ESM modules B The stickers indicate whether the enclosure is set to mode 1 mode 2 or mode 4 as shown in Figure 33 3 Confirm that all adapters that you need to connect to the 58 87 disk drive enclosure have b...

Страница 77: ...dapter cards to represent the external server connections The adapter card can represent one of the following connection types v The external port of an adapter that you confirmed in step 3 on page 64 v The external port of an internal adapter cable that you confirmed in step 4 on page 64 7 Choose one of the following options to cable the SAS adapter Note Adapters are cabled to the 58 87 by using ...

Страница 78: ...with support for a single SAS drawer C as shown in Figure 34 Note v The system with support for a single SAS drawer C has access to all 24 drive bays Go to step 18 on page 75 9 Perform the mode 1 connection of one 58 87 disk drive enclosure A by using a YO cable B to a single SAS adapter C as shown in Figure 35 on page 67 Note v The single SAS adapter C has access to all 24 drive bays Figure 34 Mo...

Страница 79: ...es A and B by using YO cables C and D to a single SAS adapter E as shown in Figure 36 on page 68 Note v The single SAS adapter E has access to all 48 drive bays Figure 35 Mode 1 connection of one 58 87 disk drive enclosure by using a YO cable to a single SAS adapter Disk drive enclosures 67 ...

Страница 80: ...ter pair C has access to the other adapter and to all 24 drive bays v If 6 Gb SAS adapters are being used they must be cabled by using 6 Gb SAS cables v For SAS adapter pairs you can attach the cables to any adapter port as long as you use the same port in both adapters Figure 36 Mode 1 connection of two 58 87 disk drive enclosures by using YO cables to a single SAS adapter 68 Enclosures and expan...

Страница 81: ...the SAS adapter pair C has access to the other adapter and to all 48 drive bays v If 6 Gb SAS adapters are being used they must be cabled by using 6 Gb SAS cables v For SAS adapter pairs you can attach the cables to any adapter port as long as you use the same port in both adapters Figure 37 Mode 1 connection of one 58 87 disk drive enclosure by using YO cables to a SAS adapter pair Disk drive enc...

Страница 82: ...A has access to the other adapter and to all disk drive bays v Both EX cables C from the same 58 87 disk drive enclosure must be attached to the same numbered port on the PCIe storage enclosure as shown in Figure 39 Figure 38 Mode 1 connection of two 58 87 disk drive enclosures by using YO cables to a SAS adapter pair Figure 39 Mode 1 connection of one 58 87 disk drive enclosure by using EX cables...

Страница 83: ...t on the PCIe storage enclosure as shown in Figure 40 Go to step 18 on page 75 15 Perform the mode 2 connection of one 58 87 disk drive enclosure A by using YO cables B to two independent SAS adapters as shown in Figure 41 on page 72 Notes v Independent SAS adapter 1 C does not have access to the other independent adapter and has access only to drive bays D1 D12 v Independent SAS adapter 2 D does ...

Страница 84: ...ve bays D1 D12 v Each adapter in the SAS adapter pair 2 D has access to the other adapter and to drive bays D13 D24 v If 6 Gb SAS adapters are being used they must be cabled by using 6 Gb SAS cables v For SAS adapter pairs you can attach the cables to any adapter port as long as you use the same port in both adapters Figure 41 Mode 2 connection of one 58 87 disk drive enclosure by using YO cables ...

Страница 85: ...igure 44 on page 74 This adapter does not have access to any other independent adapter and has access only to drive bays D7 D12 v The cable that plugs into independent SAS adapter 3 E contains a label with the P1 identifier G as shown in Figure 44 on page 74 This adapter does not have access to any other independent adapter and has access only to drive bays D13 D18 v The cable that plugs into inde...

Страница 86: ...e 75 Figure 43 Mode 4 connection of one 58 87 disk drive enclosure by using X cables to four independent SAS adapters Figure 44 Labels for SAS adapter cables showing P1 and P2 identifiers 74 Enclosures and expansion units ...

Страница 87: ...he beginning of this procedure v Add the disk drives to the operating system you are using v Verify the new configuration To verify that the system or logical partition recognizes the disk drive enclosure see Verifying the installed part http publib boulder ibm com infocenter systems scope hw topic p7haj p7hajhsmverify htm Removing your disk drive enclosures Learn about removing a disk drive enclo...

Страница 88: ...76 Enclosures and expansion units ...

Страница 89: ...e event of a single PCIe link failure because of the dual controllers and fully redundant SAS fabric within the PCIe storage enclosure Furthermore there are multiple ways to connect the two PCIe cables from a PCIe storage enclosure to the system some of which result in a better high availability characteristics than others For example connecting the two PCIe cables from a PCIe storage enclosure to...

Страница 90: ...the cables follow your site plan and keep the following items in mind v Avoid creating a safety hazard v Avoid damaging the cables v Avoid placing cables parallel to high voltage lines 5 Continue with Connecting a PCIe storage enclosure on page 80 Where to install a PCIe storage enclosure Before you begin the installation process plan where you want to install the PCIe storage enclosures When deci...

Страница 91: ...s are connected to system units in a point to point topology v Only one PCIe storage enclosure can be connected to a system side PCIe connector Daisy chain cabling of PCIe storage enclosures is not supported v PCIe storage enclosures can be connected to two PCIe connectors on the same PCIe GX adapter or split across two different PCIe GX adapters Connecting the two PCIe cables to different PCIe GX...

Страница 92: ...he logical partitions active v If you are adding a 58 88 PCIe storage enclosure the system must be powered off to complete the addition v If you are adding any other PCIe storage enclosure you can add it either with the system powered on and partitions active or with the system powered off Adding a PCIe storage enclosure with the system powered on Learn how to add a PCIe storage enclosure to the s...

Страница 93: ...rs to install them After the PCIe GX adapters are installed continue with the steps in this procedure to connect the PCI storage enclosure 5 Using PCIe cables connect the PCIe storage enclosure to the GX adapters that were identified in step 4 Note If you are connecting a PCIe cable to an unused PCIe connector that is covered with a plastic or metal clip remove the clip before installing the PCIe ...

Страница 94: ...ndicator that you recorded in step 6c4 that is not active complete the following steps Note Identify indicators are not active if the PCIe cables are connected to the wrong PCIe connectors or the PCIe connector location codes were calculated incorrectly 1 On the Identify Indicators display select the entry with a Location column value that matches the Host Port column value from the PCIe Hardware ...

Страница 95: ...igned to a partition that is powered on continue with step 14 on page 84 Note If your system is powered on to firmware standby state choose this option The system is in firmware standby state if there are no active partitions 11 Complete the following steps a Log in to the operating system with admin or service level authority If you need assistance contact the system administrator b Choose one of...

Страница 96: ...alue that when appended to the Unit ID value in the upper right corner forms the complete I O slot identifier string j Type 3 in the Opt column for the entry that you located in step 12i and press Enter Note Specifying option 3 starts the Hardware Resource Concurrent Maintenance menu for the selected packaging hardware resource k On the Hardware Resource Concurrent Maintenance display start the Po...

Страница 97: ...mpleting the verification steps of that procedure confirm that the PCIe links that connect the PCIe storage enclosure being added have Link Status column values of Operational and valid I O Enclosure Port column values The I O Enclosure Port column value looks similar to Uxxxx 001 yyyyyyy P1 Cz T3 where xxxx is the feature code of the PCIe storage enclosure connected by the PCIe link yyyyyyy is th...

Страница 98: ...necting PCIe storage enclosures with PCIe cables on page 87 v Steps in this topic require you to have connector information for the hardware in your configuration For details about PCIe storage enclosure or system connector locations see Connector locations on page 115 About this task To add a PCIe storage enclosure to the system with the system powered off complete the following steps 1 If the sy...

Страница 99: ...lete this step for each system Requirement All PCIe hardware topology problems that are discovered in this step must be corrected before continuing with this procedure 10 Check for and handle new serviceable events that were generated during the procedure by completing the steps in Checking for new serviceable events http publib boulder ibm com infocenter systems scope hw topic p7ham p7ham_genfile...

Страница 100: ...age enclosure as shown in the following figure Figure 45 Example of a server that has one PCIe GX adapter connected to a single storage enclosure Figure 46 Example of a server that has two PCIe GX adapters connected to two storage enclosures Figure 47 Example of a server that has two PCIe GX adapters connected to a single storage enclosure 88 Enclosures and expansion units ...

Страница 101: ...wn in the following figure 6 Connect two processor drawers each with a PCIe GX adapter to a single storage enclosure for high availability as shown in the following figure Figure 48 Example of a server that has two PCIe GX adapters connected to two storage enclosures for high availability Figure 49 Example of two servers each with a PCIe GX adapter connected to a single storage enclosure for high ...

Страница 102: ...ence codes where x is any character 0 9 or A F by completing the following steps a In the navigation area of the HMC select Service Management b In the content area select Manage Serviceable Events c In the Manage Serviceable Events window perform the following steps 1 Click Open for the Serviceable Event Status field 2 Select the machine type model and serial number MTMS of the server you are wor...

Страница 103: ...ns might apply to your configuration Choose only the first option that applies v If your system has a partition running the VIOS operating system continue with step 2 v If your system has a partition running the AIX operating system continue with step 3 on page 92 v If your system has a partition running the Linux operating system continue with step 4 on page 92 2 Use the VIOS error log to verify ...

Страница 104: ... Perform problem analysis to correct the problems before continuing If you need assistance contact your service provider f Go to step 5 4 Use the Linux error log to verify that there are no B7006Axx errors where x is any character 0 9 or A F by completing the following steps a Log in as the root user If you need assistance contact the system administrator b At the Linux command prompt type service...

Страница 105: ...lpful in identifying these entries v When an entry corresponds to a PCIe storage enclosure connection the Slot column contains an identifier that resembles Uwwww uuu xxxxxxx P1 Cy Tz L1 where wwww is the feature code of the system uuu is a set of three numbers 0 9 or letters A F xxxxxxx is the serial number of the system y is a number greater than 0 and z is 1 or 2 An example of an actual Slot col...

Страница 106: ...ons might apply to your configuration Choose only the first option that applies v If your system is managed by a Hardware Management Console HMC complete the following steps a In the navigation area click Serviceability Manage Serviceable Events b Continue with step 2 on page 95 v If your system is managed by a Systems Director Management Console SDMC complete the following steps a On the Welcome ...

Страница 107: ...his task To activate an identify indicator to locate a hardware component complete the following steps Procedure 1 Access the Advanced System Management Interface ASMI by using an authority level of administrator or authorized service provider For details about using the ASMI see Managing the Advanced System Management Interface http publib boulder ibm com infocenter systems scope hw topic p7hby a...

Страница 108: ...ite Tip If you are planning to configure RAID arrays ensure that you have the minimum number of available disks for each RAID level RAID 0 Minimum of one drive per array RAID 5 Minimum of three drives per array RAID 6 Minimum of four drives per array RAID 10 Minimum of two drives per array For additional SAS RAID controller information for the AIX operating system see SAS RAID controller for AIX F...

Страница 109: ...RAID array The ED R1 PCIe storage enclosure is supported by the following operating systems v AIX v Linux v VIOS Note VIOS does not support the ED R1 PCIe storage enclosure for an client To determine the level of software that you need to support the PCIe storage enclosure feature see IBM Prerequisite Tip If you are planning to configure RAID arrays ensure that you have the minimum number of avail...

Страница 110: ...ck vary by model this procedure provides only conceptual removal steps Removing a PCIe storage enclosure with the system powered on Learn about removing a PCIe storage enclosure from a system with the system powered on Before you begin Important To use this procedure to remove a PCIe storage enclosure from the system the enclosure must be a model that can be removed with the system powered on Revi...

Страница 111: ...rdware Topology data to identify all PCIe links that are associated with the PCIe storage enclosure that you are removing To make this identification scan the I O Enclosure Port column for values that contain the feature code and serial number that you recorded in step 1 on page 98 Note The I O Enclosure Port column value looks similar to Uxxxx 001 yyyyyyy P1 Cz T3 where xxxx is the feature code o...

Страница 112: ...iguration 6 Click Manage Profiles 7 For each profile for the selected logical partition that includes the I O slot complete the following steps a On the Manage Profiles display click the name of the profile that includes the I O slot b On the Logical Partition Profile Properties display click the I O tab c Select the entry that has the identifier string of the I O slot in the Location Code column ...

Страница 113: ...physically remove the adapter r If there is another I O slot to process return to step 5 on page 99 Otherwise continue with step 10 on page 102 8 Complete the following steps a Log in to the operating system with admin or service level authority If you need assistance contact the system administrator b At the command prompt type strsst and press Enter c Type the service tools user ID and service t...

Страница 114: ...he following steps 1 Choose one of the following options v If your system is managed by an HMC complete the following steps a From the navigation bar expand Systems Management b Click Servers c Select the server you are working with d In the Tasks area expand Hardware Information e Click PCIe Hardware Topology v If your system is not managed by an HMC complete the following steps a Access the ASMI...

Страница 115: ...service provider For details about using the ASMI see Managing the Advanced System Management Interface http publib boulder ibm com infocenter systems scope hw topic p7hby asmi htm 2 In the ASMI navigation area expand System Configuration 3 Click PCIe Hardware Topology b Select the PCIe link Note The link status for each link that is associated with the PCIe storage enclosure that you are removing...

Страница 116: ...l of the PCIe storage enclosure 2 Optional You can use identify indicators to assist in locating the PCIe cables and connectors that are associated with the PCIe storage enclosure you are removing To illuminate the identify indicators for the two connections to the PCIe storage enclosure being removed complete the following steps Important v To perform this step your system must be running firmwar...

Страница 117: ...tem 5 Disconnect the PCIe cables from the PCIe storage enclosure Note If you chose to activate identify indicators in step 2 on page 104 they can be used now to assist in locating the PCIe connectors from which to disconnect the cables 6 Disconnect the PCIe cables from the PCIe connectors on the system Note If you chose to activate identify indicators in step 2 on page 104 they can be used now to ...

Страница 118: ...ed during the procedure by completing the steps in Checking for new serviceable events on page 94 Use the date and time information that you recorded in step 10 on page 105 as the start time for searching for new serviceable events Important If the PCIe storage enclosure is connected to two systems complete this step for each system 106 Enclosures and expansion units ...

Страница 119: ...MI to start a system that is not managed by a Hardware Management Console HMC or an Systems Director Management Console SDMC About this task To start a system that is not managed by a HMC or SDMC follow these steps Procedure 1 Open the front rack door if necessary 2 Before you press the power button on the control panel ensure that power is connected to the system unit as follows v All system powe...

Страница 120: ... the enclosure No light indicates that no resources in the enclosure are being identified v D Attention light No light indicates that the system is operating normally A solid light indicates that the system requires attention v E USB port v F Enclosure fault roll up light A constant light indicates a fault indicator active in the system No light indicates that the system is operating normally v G ...

Страница 121: ...lay while the system is being started When the power on light on the control panel stops blinking and remains on the system power is on Stopping a system or logical partition Learn how to stop a system or logical partition as a part of a system upgrade or service action About this task Attention Using either the power on button on the control panel or entering commands at the Hardware Management C...

Страница 122: ...r turns off the power on light begins to slowly flash and the system goes into a standby state 3 At the Linux command line type shutdown h now The command stops the operating system The system power turns off the power on light begins to slowly flash and the system goes into a standby state 4 Record the IPL type and the IPL mode from the control panel display to help you return the system to this ...

Страница 123: ...he managed system without shutting down the logical partitions first causes the logical partitions to shut down abnormally and can cause data loss If you use a Virtual I O Server VIOS logical partition ensure that all clients are shut down or that the clients have access to their devices using an alternate method To power off a managed system you must be a member of one of the following roles v Su...

Страница 124: ...ou are currently searching You might need to search earlier Power Systems hardware levels to locate installation documentation for the enclosure that you are removing About this task To remove an enclosure from the rack complete the following steps Procedure 1 If the enclosure is not in a powered off state power off the enclosure Important If the enclosure that you are removing is an expansion uni...

Страница 125: ... screws These models were installed in a two step process that used rail mounting guides In the first step the guides were temporarily attached to the sides of the enclosure In the second step the guides were removed after the enclosure was lifted onto the rails and securing screws were installed For the purposes of this procedure identify these screws as the weight bearing screws that you are dir...

Страница 126: ...ep 6a on page 113 as securing screws When this step is complete the enclosure should no longer be secured to the rack or to any mounting hardware CAUTION Be careful in this step to remove only the screws that prevent the enclosure from being detached from the rack or mounting hardware If you believe that a particular screw might be weight bearing do not remove it 10 Perform a test lift of the encl...

Страница 127: ... system power control network SPCN and PCIe connector locations Server connectors Learn about connector locations for servers Model 02E 4B connector locations Learn about connector locations on rack mounted and stand alone models Figure 53 Model 02E 4B connector locations on a rack mounted model Reference information 115 ...

Страница 128: ...ure is as follows The card installed in the C1 slot can contain two 12X double data rate DDR cable ports However the C1 slot cannot provide both 12X DDR and PCIe cable ports The server contains a serial attached SCSI SAS port Figure 54 Model 02E 4B connector locations on a stand alone model Figure 55 Model 02E 4C connections for expansion units disk drive enclosures and PCIe storage enclosures 116...

Страница 129: ...lled in the C1 slot can contain two PCIe cable ports However the C1 slot cannot provide both PCIe and 12X DDR cable ports The server contains a SAS port Model 05F 6B connector locations Learn about connector locations on rack mounted and stand alone models Figure 56 Model 05F 6B connector locations on a rack mounted model Figure 57 Model 05F 6B connector locations on a stand alone model Reference ...

Страница 130: ...gure is as follows The server contains a SAS port v Support for PCIe storage enclosures as shown in the preceding figure is as follows The card installed in the C1 slot can contain two PCIe cable ports However the C1 slot cannot provide both PCIe and 12X DDR cable ports The card installed in the C8 slot can contain two PCIe cable ports However the C8 slot cannot provide both PCIe and 12X DDR cable...

Страница 131: ...he C8 and C6 slot space to be occupied The server cannot support both 12X DDR cable ports and PCIe cable ports at the same time The server contains a serial attached SCSI SAS port v Support for disk drive enclosures as shown in the preceding figure is as follows The server contains a SAS port v Support for PCIe storage enclosures as shown in the following figure is as follows The card installed in...

Страница 132: ...nnector locations Learn about connector locations on rack mounted models Figure 61 Model 31E 2C connections for PCIe storage enclosures Figure 62 Model 31E 2B connector locations on a rack mounted model Figure 63 Model 33E 8B connector locations on a rack mounted model 120 Enclosures and expansion units ...

Страница 133: ...M MD 79M HC or 79M HD connector locations Learn about connector locations Enclosure connectors Learn about connector locations for I O enclosures Figure 64 Model 17M MB or 79M HB connector locations Figure 65 Model 17M MC 17M MD 79M HC or 79M HD connector locations Reference information 121 ...

Страница 134: ...Learn about connector locations Model 58 02 and 58 77 connector locations Learn about connector locations Figure 66 Model 57 96 or 14G 30 connector locations Figure 67 Model 58 02 and 58 77 connector locations 122 Enclosures and expansion units ...

Страница 135: ...8 87 connector locations Learn about connector locations Model 58 88 connector locations Learn about connector locations Figure 68 Model 58 86 connector locations Figure 69 Model 58 87 connector locations Figure 70 Model 58 88 connector locations Reference information 123 ...

Страница 136: ...Model ED R1 connector locations Learn about connector locations Figure 71 Model ED R1 connector locations 124 Enclosures and expansion units ...

Страница 137: ...mprovements and or changes in the product s and or the program s described in this publication at any time without notice Any references in this information to websites not owned by the manufacturer are provided for convenience only and do not in any manner serve as an endorsement of those websites The materials at those websites are not part of the materials for this product and use of those webs...

Страница 138: ... the manufacturer s support websites for updated information and fixes applicable to the system and related software Homologation statement This product may not be certified in your country for connection by any means whatsoever to interfaces of public telecommunications networks Further certification may be required by law prior to making any such connection Contact an IBM representative or resel...

Страница 139: ...tection requirements of EU Council Directive 2004 108 EC on the approximation of the laws of the Member States relating to electromagnetic compatibility IBM cannot accept responsibility for any failure to satisfy the protection requirements resulting from a non recommended modification of the product including the fitting of non IBM option cards This product has been tested and found to comply wit...

Страница 140: ...Interference EMI Statement People s Republic of China Declaration This is a Class A product In a domestic environment this product may cause radio interference in which case the user may need to perform practical action Electromagnetic Interference EMI Statement Taiwan The following is a summary of the EMI Taiwan statement above Warning This is a Class A product In a domestic environment this prod...

Страница 141: ...ist eine Einrichtung der Klasse A Diese Einrichtung kann im Wohnbereich Funk Störungen verursachen in diesem Fall kann vom Betreiber verlangt werden angemessene Maßnahmen zu ergreifen und dafür aufzukommen Deutschland Einhaltung des Gesetzes über die elektromagnetische Verträglichkeit von Geräten Dieses Produkt entspricht dem Gesetz über die elektromagnetische Verträglichkeit von Geräten EMVG Dies...

Страница 142: ... the interference by one or more of the following measures v Reorient or relocate the receiving antenna v Increase the separation between the equipment and receiver v Connect the equipment into an outlet on a circuit different from that to which the receiver is connected v Consult an IBM authorized dealer or service representative for help Properly shielded and grounded cables and connectors must ...

Страница 143: ...pean Standard EN 55022 The limits for Class B equipment were derived for typical residential environments to provide reasonable protection against interference with licensed communication equipment European Community contact IBM Deutschland GmbH Technical Regulations Department M372 IBM Allee 1 71139 Ehningen Germany Tele 49 7032 15 2941 email lugi de ibm com VCCI Statement Japan Japanese Electron...

Страница 144: ... bzw der EMC EG Richtlinie 2004 108 EG für Geräte der Klasse B Dieses Gerät ist berechtigt in Übereinstimmung mit dem Deutschen EMVG das EG Konformitätszeichen CE zu führen Verantwortlich für die Einhaltung der EMV Vorschriften ist der Hersteller International Business Machines Corp New Orchard Road Armonk New York 10504 Tel 914 499 1900 Der verantwortliche Ansprechpartner des Herstellers in der E...

Страница 145: ...d herein whenever in its discretion the use of the publications is detrimental to its interest or as determined by the manufacturer the above instructions are not being properly followed You may not download export or re export this information except in full compliance with all applicable laws and regulations including all United States export laws and regulations THE MANUFACTURER MAKES NO GUARAN...

Страница 146: ...134 Enclosures and expansion units ...

Страница 147: ......

Страница 148: ...Printed in USA ...

Отзывы:

Нет отзывов

Бренды по названию

Популярные бренды

Загрузить еще бренды