background image

3ware

® 

Serial ATA RAID Controller

Supports the 9650SE, 9590SE, and 9550SX Models

PN 720-0146-00   
October 2006 

User Guide

Содержание 9650SE-2LPB-10 - Sataii Raid Hardware 10Pk

Страница 1: ...3ware Serial ATA RAID Controller Supports the 9650SE 9590SE and 9550SX Models PN 720 0146 00 October 2006 User Guide...

Страница 2: ...o 3BM StorSwitch and R5 Fusion are all trademarks of AMCC PowerPC and the PowerPC logo are trademarks of International Business Machines Corporation Linux is a registered trademark of Linus Torvalds i...

Страница 3: ...Motherboard Boot Sequence 29 What Next 29 Chapter 4 Driver Installation 30 Driver Installation Under Windows 31 Materials Required 31 Creating a 3ware Driver Diskette 32 Installing the 3ware Driver an...

Страница 4: ...g Your Controller 85 Viewing Information About a Controller 85 About Controller Policies 87 Viewing Controller Policies 89 Setting the Auto Rebuild Policy 90 Using Auto Carving for Multi LUN Support 9...

Страница 5: ...146 Background Tasks 147 About Initialization 148 About Verification 151 Starting a Verify Manually 153 Rebuilding Units 155 Cancelling a Rebuild and Restarting It with a Different Drive 159 Setting...

Страница 6: ...age 217 Chapter 12 Troubleshooting 220 Web Resources 220 Before Contacting Customer Support 221 Command Logging 221 Problems and Solutions 222 Hardware Installation 222 Software Installation 222 Probl...

Страница 7: ...or performing tasks using two tools one at the BIOS level 3ware BIOS Manager 2 referred to as 3BM 2 and one that runs in a browser 3ware Disk Manager 2 referred to as 3DM 2 You can also perform many t...

Страница 8: ...change existing configurations and set unit policies 9 Maintaining Units Describes how to check unit and drive status review alarms and errors schedule background maintenance tasks and manually start...

Страница 9: ...different operating systems including Mac OS X Microsoft Windows Linux and FreeBSD and runs in a number of different browsers In addition the version shown in screenshots may not match your version F...

Страница 10: ...elf Window Click the Show Navigation button to display the Table of Contents Note The 3ware HTML Bookshelf is created as a set of HTML documents that are often displayed from a website When installed...

Страница 11: ...topics What s New with 9650SE Models System Requirements Understanding RAID Concepts and Levels 3ware Tools for Configuration and Management Monitoring Maintenance and Troubleshooting Features What s...

Страница 12: ...y_sata2 asp Drives may be of any capacity or physical form factor The length of shielded and unshielded interface cables may not exceed 1M 39 for Serial ATA controllers Table 2 Required Slots for 3war...

Страница 13: ...te power supply for drives 3DM 2 3ware Disk Manager displays information in a browser It requires one of the following browsers Internet Explorer 5 5 and later Mozilla Firefox 1 2 and later Netscape 7...

Страница 14: ...a 3ware RAID controller provides improved performance because 3ware s TwinStor technology reads from both drives simultaneously Striping Striping across disks allows data to be written and accessed o...

Страница 15: ...ed in a striped disk array see Figure 2 large files are distributed across the multiple disks using RAID 0 techniques Striped disk arrays give exceptional performance particularly for data intensive a...

Страница 16: ...neously reading alternating sections of the file For the smaller random transactions the data is read from a single optimal drive head Figure 3 RAID 1 Configuration Example RAID 5 RAID 5 provides perf...

Страница 17: ...uilding or one sub unit may be initializing while another is verifying RAID 6 requires a minimum of five drives Performance and storage efficiency also increase as the number of drives increase Figure...

Страница 18: ...AID 50 is a combination of RAID 5 with RAID 0 This array type provides fault tolerance and high performance RAID 50 requires a minimum of six drives Several combinations are available with RAID 50 For...

Страница 19: ...taken to prevent system hangs and data loss JBOD A JBOD acronym for Just a Bunch of Disks is an unconfigured disk attached to your 3ware RAID controller JBOD configuration is no longer supported in t...

Страница 20: ...fault tolerance and a read speed advantage over non RAID disks RAID 5 This type of unit provides performance fault tolerance and high storage efficiency RAID 5 units can tolerate one drive failing bef...

Страница 21: ...10 with hot spare Combination of RAID 0 RAID 1 hot spare single disk 6 or more RAID 6 RAID 6 with hot spare RAID 50 Combination of RAID 0 1 5 6 10 hot spare single disk Table 5 Drive Capacity RAID Lev...

Страница 22: ...ult prior to unit creation If a unit over 2 TB was created prior to enabling the auto carve option its capacity visible to the operating system will still be 2TB no additional capacity will be registe...

Страница 23: ...specify the type of events that should generate these notifications For details see Using the Alert Utility Under Windows on page 144 3ware CLI Command Line Interface The 3ware CLI provides the functi...

Страница 24: ...nitialize rebuild verify and self test tasks can all be run in the background at scheduled times This lets you choose a time for these tasks to be run when it will be least disruptive to your system Y...

Страница 25: ...rive needs to be checked or replaced For more information see Locating a Drive by Blinking Its LED on page 166 Auto Rebuild For times when you do not have a spare available setting the Auto Rebuild po...

Страница 26: ...the installation guide that came with your 3ware controller If you do not have a hardcopy of the installation manual it is available in the 3ware Documentation folder on your 3ware CD and you can down...

Страница 27: ...nagement and Maintenance Features 3ware RAID controllers include a number of features to help you manage and maintain the controller and your configured units The default settings for these features a...

Страница 28: ...or Factor 2048 GB 3BM 3DM CLI Drives Per Spinup 1 3BM CLI Delay Between Spinup 6 seconds 3BM CLI Export Unconfigured JBOD Disks Disabled 3BM CLI Staggered Method ATA 6 3BM Staggered Spinup Enabled 3BM...

Страница 29: ...CLI a Although the default Self test Task Schedule is for 24 hours self test tasks are run only at the beginning of that time period and take just a few minutes For more information about task schedul...

Страница 30: ...rts and the number of drives attached For more information see Determining What RAID Level to Use on page 10 Basic Steps for Creating a Unit The process of configuring your RAID units includes these m...

Страница 31: ...your system a screen lists the available boards See Figure 9 In this case highlight the board with which you want to work and press Enter Figure 9 3ware Controller Board Selection Screen You see a sc...

Страница 32: ...the drives and create a unit 1 Select the drives to be included by highlighting each one and pressing Enter to select it When you select a drive an asterisk appears next to it in the left most column...

Страница 33: ...ou selected Figure 13 List of Configuration Choices for Four Drives 3 Use the arrow keys to highlight the desired RAID configuration and press Enter For information about the different RAID levels and...

Страница 34: ...s or press Tab to move to the field you want to change 2 Press Enter to see the available options 3 Use the arrow keys to select the option you want and press Enter to choose it For details about thes...

Страница 35: ...ve finished making all your changes and press F8 3 If the volume summary screen appears review the information and press any key to continue Multiple volumes will be created if you entered a Boot Volu...

Страница 36: ...l the operating system on one of them make that unit be the first unit Unit 0 in the list of Exportable Units To move a unit up in the list highlight it and press the Page Up key You will also want to...

Страница 37: ...to the drives is complete Once you have booted to the operating system background initialization of the unit will begin after a delay of up to ten minutes For complete information about initializatio...

Страница 38: ...the words Hot Spare appear next to the drive in the Available Drives list Figure 17 Hot Spare Indicated If a hot spare is already enabled you can disable it by following the same process Leaving Indiv...

Страница 39: ...you have other disks installed on the motherboard the 3ware RAID controller should precede them in boot order If you will install your OS on a disk or unit attached to the 3ware RAID controller specif...

Страница 40: ...nit connected to another controller or to the motherboard you start the operating system and then install the driver Details for both situations are described in this chapter Driver information is org...

Страница 41: ...ows on a new drive or unit Install the 3ware driver when Windows is already installed on a different device Materials Required 3ware software CD ROM Microsoft Windows 2000 XP or Server 2003 installati...

Страница 42: ...d Windows on a New Unit If you want to install Windows on a new drive configuration managed by the 3ware RAID controller follow the instructions in this section If Windows is already installed on anot...

Страница 43: ...to Windows on page 38 Installing the Driver on a System that Boots from a Different Device If you are installing the 3ware RAID controller on a system that already has the operating system installed o...

Страница 44: ...he Wizard prompts you to select a device driver click Have Disk and then indicate that it is on the CD or floppy 3 If the Digital Signature Not Found message appears click Yes to continue the installa...

Страница 45: ...Found New Hardware Wizard After the driver has been installed continue with the instructions under Making Units Managed by a 3ware Controller Available to Windows on page 38 To install the 3ware driv...

Страница 46: ...CD or floppy 3 If the Digital Signature Not Found message appears click Yes to continue the installation 4 When the Completing the Found New Hardware Wizard screen appears Figure 23 click Finish Figu...

Страница 47: ...are Wizard 7 When the second Completing the Found New Hardware screen appears click Finish Figure 25 Completing Found New Hardware Wizard After the driver has been installed continue with the instruct...

Страница 48: ...in as the system administrator 2 Use Disk Administrator to partition and format the new units or disks From the Start menu choose Administrative Tools Computer Management In the Computer Management w...

Страница 49: ...it on page 42 Installing the 3ware Kernel Driver Module on a Red Hat or Fedora Core Linux System that Boots From a Different Device on page 44 SuSE Linux Installation Installing the 3ware Kernel Drive...

Страница 50: ...uSE and other similar distributions running similar kernel strings Be Sure to Use the Correct Drivers The Linux 3w 9xxx drivers included on the 3ware CD are for 3ware 9650SE 9590SE 9550SX and 9500S se...

Страница 51: ...i_host hostid stats where hostid is usually host0 unless other SCSI devices are available in which case it may be host1 or higher If you have a 2 6 kernel without sysfs type the following command dmes...

Страница 52: ...o create a driver install diskette from the menu Extract the kernel driver module from the modules cgz file using this command zcat modules cgz cpio idvm Installing the 3ware Kernel Driver Module whil...

Страница 53: ...ding 3w 9xxx 5 When prompted select the proper language and keyboard types for your locality 6 If asked for what type of media select Local CD ROM since you are installing from the Red Hat CD ROM For...

Страница 54: ...ng refers to the kernel version The kernel string will have different endings depending on the kernel you are using For an SMP kernel multi processor the kernel string will end in smp For example 2 6...

Страница 55: ...kernel drivers scsi 3w 9xxx ko If prompted to overwrite type y For Redhat workstation on AMD Opteron and Intel EM64T type cp mnt cdrom packages drivers linux redhat ws4_u4 x86_64 3w 9xxx ko or smp li...

Страница 56: ...pe cp mnt cdrom packages drivers linux redhat ws4_u4 x86 3w 9xxx ko smp or hug lib modules put kernel string here kernel drivers scsi 3w 9xxx ko For Fedora Core on Intel x86 and AMD x86 type cp mnt cd...

Страница 57: ...create a 3ware driver install diskette If Linux is already installed on another device you may install the 3ware kernel driver module from the 3ware software CD ROM To create a driver install diskette...

Страница 58: ...talling SuSE Linux on a New Unit 1 Boot directly from the SuSE installation CD 1 or DVD 2 When installing SuSE press either the F6 key or the F3 key depending on the version You will then be prompted...

Страница 59: ...e sles10 x86 3w 9xxx ko smp or big lib modules kernel string kernel drivers scsi 3w 9xxx ko For SuSE Linux Enterprise Server 10 0 64 bit x86_64 type cp cdrom packages drivers linux suse sles10 x86_64...

Страница 60: ...e information see 3ware knowledgebase article 14546 http www 3ware com KB article aspx id 14546 Driver Installation Under FreeBSD A drive or unit managed by the 3ware RAID controller may be configured...

Страница 61: ...ble at packages drivers freebsd src 6 x You can also download the FreeBSD kernel driver module from the 3ware web site http www 3ware com support index asp FreeBSD installation CD ROM or diskette Inst...

Страница 62: ...9590SE 9650SE kernel driver module into the floppy drive 5 Run sysinstall In the sysinstall menu choose Configure Load KLD Load twa_9ksx ko from the floppy drive 6 Install the FreeBSD OS remove the fl...

Страница 63: ...kernel driver module for FreeBSD 6 1 can be found on both the 3ware CD ROM and the 3ware website For step by step instructions see To update the kernel with new driver source on page 53 11 Build the...

Страница 64: ...make install 5 Reboot your system When the system reboots the new kernel driver module will load automatically Installing the 3ware Kernel Driver Module on a FreeBSD System that Boots from a Device o...

Страница 65: ...OS version being used The sources for the new kernel driver module for FreeBSD 5 4 can be found on both the 3ware CD ROM and the 3ware website For step by step instructions see To update the kernel w...

Страница 66: ...roller User Guide 4 cd sys modules twa make 5 Comment out the entry Device twa 3ware 9000 series in usr src sys i386 conf or usr src sys amd64 conf 6 Rebuild the kernel and reboot your system 7 Load t...

Страница 67: ...ut doing particular tasks in 3BM refer to the later sections in this guide Starting 3BM 2 You access 3BM during the start up process for your computer 1 Power up or reboot your system 2 While the syst...

Страница 68: ...28 In this case highlight the board with which you want to work and press Enter Figure 28 AMCC Boards Selection Screen You see a screen similar to the warning message below warning you that changing y...

Страница 69: ...es controllers in your system the 7000 8000 series controllers are not listed on the selection screen shown in Figure 28 Instead an additional BIOS summary will appear for the 7000 8000 series control...

Страница 70: ...its have been configured If you have more than one unit the boot unit is the one at the top of the list You can change the order by highlighting a unit and pressing the PgUp or PgDn key Unusable Array...

Страница 71: ...Keys OR Left and Right Arrow Keys OR Tab and Shift Tab Select or unselect what is currently highlighted A selection may be a drive in a list of drives a button at the bottom of the screen or a field...

Страница 72: ...to the main 3ware BIOS Manager screen from the Advanced Details screen Any key Move a highlighted unit up or down in the list of exportable units The top most unit will become the bootable unit if yo...

Страница 73: ...o show about configured units on the start up screen Expanded lists each unit and shows the specific drives that make up the unit Collapsed lists only the configured units Pause Time Sets the number o...

Страница 74: ...r The BIOS Options Settings screen appears Figure 31 BIOS Option Settings Screen 3 Tab through the options and make the desired changes 4 Tab to OK and press Enter to return to the main screen Note Du...

Страница 75: ...Tab to Information and press Enter 2 On the pop up menu select Controller and press Enter Note that the particular versions shown in the screen shot below may not be current Figure 32 Advanced Detail...

Страница 76: ...nistrative and maintenance tasks locally or remotely Two levels of access to 3DM are provided user and administrator Users have view only access and can check the status of drives and units Administra...

Страница 77: ...RAID controller You can also download the current version from the website at http www 3ware com support download asp 3DM must be installed on the system in which the controller is installed 3DM does...

Страница 78: ...release To install 3DM on a Windows system 1 Insert the 3DM CD ROM and click Install 3DM 2 when the menu appears Or if you downloaded 3DM from the website find the file setup exe and double click it t...

Страница 79: ...email server You can get the IP address of your mail server by typing nslookup file server name To find the mail server name ask your system administrator or check the e mail preferences or setup opt...

Страница 80: ...tion gives you the option of starting 3DM and opening the browser window 8 If you elect to launch the web interface a Security Alert may appear as shown in Figure 35 Click Yes continue You may install...

Страница 81: ...3DM2 from the Start Programs 3ware 3DM2 menu If you change add or remove an IP address or change the machine name of the machine on which you have installed 3DM 2 you will need to recreate the securit...

Страница 82: ...onal information the name of the mail server the user name for the person who will send the E mail notification typically the local host name and the user name for the person who will receive the e ma...

Страница 83: ...r Windows and as a daemon under Linux and FreeBSD Normally after installation the 3DM process starts automatically when you start your system It is a good idea to leave the 3DM process running on the...

Страница 84: ...calhost 888 You can also replace localhost with the IP address of the computer that contains the 3ware controller For example https IP address 888 2 The first time you start 3DM when the security cert...

Страница 85: ...ow to start it To start the 3DM daemon manually 1 Login as root on the machine on which 3DM is installed 2 Afterwards type etc init d tdm2 start 3 To start the 3DM web application open your browser an...

Страница 86: ...rom a browser on any computer over an internet connection You do not need to install the 3DM software on the remote computer To connect to 3DM2 through your web browser In the address line of your bro...

Страница 87: ...rovides summary or detail information about your 3ware RAID controller and the resources connected to it As you work in 3DM the Messages area just below the menu bar displays information about the res...

Страница 88: ...Management Maintenance page The Monitor menu gives you access to the Alarms page the BBU page and the Enclosure Summary page The Alarms page shows a list of alarms including the specific alarm message...

Страница 89: ...can also be set For details see Setting the Frequency of Page Refreshes on page 84 3DM Screens and What They re Used For The table below shows a list of the pages you work with in 3DM and describes w...

Страница 90: ...rrent controller and provides summary information about each drive To see this page choose Information Drive Information from the menu bar Drive Details window Shows the SMART data for a specific driv...

Страница 91: ...o view this page choose Monitor Alarms on the menu bar Battery Backup page Shows the status of a Battery Backup Unit BBU if one is installed and allows you to test the battery To view this page choose...

Страница 92: ...u want to change User or Administrator 3 Type the current password in the Current Password field If you are changing the password for the first time the factory set default password is 3ware 4 Enter t...

Страница 93: ...e e mail address es to which notifications are sent Separate multiple addresses with a comma or a semicolon Enter the SMTP server name or IP of the mail server for the computer where the 3ware control...

Страница 94: ...ge Port The page refreshes and a message at the top of the screen confirms that the listening port has been changed Setting the Frequency of Page Refreshes Since the status of the drives attached to y...

Страница 95: ...a Controller You can check the controller model serial number firmware and driver versions and the status of the 3ware RAID controller in your computer If you have more than one controller in your sys...

Страница 96: ...s If you have more than one controller in the system you can switch between them by selecting the one you want from the Select Controller drop down list at the right of the menu bar This drop down is...

Страница 97: ...en in 3BM Figure 43 Auto Rebuild Determines whether the Auto Rebuild policy is enabled or disabled When disabled degraded units can only be rebuilt with designated spares When enabled the controller f...

Страница 98: ...whether JBOD disks should be exported to the operating system JBOD configuration is strongly discouraged for newly added disks on a 9000 series controller in favor of Single Disk so by default this s...

Страница 99: ...nd Carve Size policies can be changed on this page The other policies can be changed in 3BM or through the CLI For a summary of the initial default settings see Table 6 Default Settings for Policies a...

Страница 100: ...o Rebuild is disabled only spares will be automatically used to rebuild degraded units When Auto Rebuild is enabled the firmware will select drives to use for automatically rebuilding a degraded unit...

Страница 101: ...ing system as separate volumes These chunks are sometimes known as multiple LUNs logical units However throughout the 3ware documentation they are referred to as volumes For example using the default...

Страница 102: ...ividual volumes existing data is not affected To use auto carving 1 Enable the auto carving feature You can do so using 3DM or 3BM In 3DM enable Auto Carving at the bottom of the Management Controller...

Страница 103: ...the future Existing units will not be affected To set the carve size in 3DM 1 Choose Management Controller Settings from the menu bar in 3DM 2 In the Other Controller Settings section at the bottom o...

Страница 104: ...ugh these fields making the choices you want to use Staggered Spinup Choose enabled or disabled Number of Drives Per Spinup Select the number of drives from 1 to 4 8 12 or 16 depending on the number o...

Страница 105: ...port of unconfigured disks 1 At the main 3BM screen Tab to Policy and press Enter 2 On the Policy Control screen Tab to Export Unconfigured Disks press Enter to display the choices use the arrow keys...

Страница 106: ...Level Migration or Online Capacity Expansion Deleting a Unit Removing a Unit Moving a Unit from One Controller to Another Adding a Drive Removing a Drive Rescanning the Controller Configuring a New U...

Страница 107: ...lude from one to sixteen drives in the unit depending on the number of drives available and the model of the controller you have For information about how many drives to select for a given RAID level...

Страница 108: ...fault Auto Verify disabled by default Continue on Source Error During Rebuild disabled by default StorSave Profile Protection by default The particular policies that you can adjust when you create the...

Страница 109: ...or greater and have enabled the Auto Carving policy the boot volume will be created in addition to any volumes created through auto carving For more information about auto carving see Using Auto Carvi...

Страница 110: ...the RAID configuration you want 5 If stripe size applies to the RAID type you select select a Stripe Size Stripe size does not apply to RAID 1 6 Optional In the Name box enter a name for the unit up...

Страница 111: ...s the carve size you may see multiple unit volumes in your operating system For details see Using Auto Carving for Multi LUN Support on page 91 In Linux a device node will now be associated with each...

Страница 112: ...8 Figure 48 3BM Main Display To create a unit through 3BM 1 At the main 3BM screen select the drives to be included by highlighting each one and pressing Enter or Space to select it When you select a...

Страница 113: ...for the unit then press Enter again to set the name 5 Tab to the RAID Configuration field and press Enter to display a list of available configurations for the number of drives you selected Figure 51...

Страница 114: ...summary screen appears review the information and press any key to continue Multiple volumes will be created if you entered a Boot Volume Size of greater than zero 0 or if auto carving is enabled and...

Страница 115: ...bed under Partitioning Formatting and Mounting Units on page 106 Ordering Units in 3BM If you configure multiple units in 3BM and you want to install the OS on one of them so that you can boot from th...

Страница 116: ...dows Linux or FreeBSD For the appropriate steps see To partition format and mount under Windows on page 106 To partition format and mount under Linux on page 107 To partition format and mount under Fr...

Страница 117: ...Partition type primary File system type ext2 Start 0 End 1074gb parted p Disk geometry for dev sda 0kB 1074GB Disk label type gpt Number Start End Size File system Name Flags1 17kB 1074MB 1074MB 4 Cr...

Страница 118: ...degraded disk or larger is available the hot spare will automatically replace the failed drive in the unit without user intervention It is a good idea to create a hot spare after you create a redundan...

Страница 119: ...words Hot Spare appear next to the drive in the Available Drives list Note 3ware s 9000 series RAID controllers use drive coercion so that drives from differing manufacturers and with slightly differe...

Страница 120: ...name or rename a unit through 3DM 1 Make sure the appropriate controller is selected in the drop down list at the right of the menu bar 2 Choose Management Controller Settings from the menu bar in 3D...

Страница 121: ...3BM Details about adjusting each policy are described on the following pages Unit Write Cache Determines whether write cache is enabled for the unit When the write cache is enabled data is stored loca...

Страница 122: ...ntrollers only ECC errors are an indication of errors that have occurred on a particular drive since it was last read When not enabled a rebuild will abort upon encountering an ECC error and the unit...

Страница 123: ...t s StorSave profile can also determine whether the write cache can be enabled or disabled A warning message will be given if the change is not permitted due to the StorSave setting and the state of t...

Страница 124: ...own The initial default setting is for write cache to be enabled 5 Press Enter to display the choices use the arrow keys to select the setting you want and press Enter again to choose it 6 Tab to the...

Страница 125: ...een When this policy is set ECC Error Correcting Code errors are ignored when they are encountered during a rebuild ECC errors are typically defects that have been detected in the drive since initiali...

Страница 126: ...choose it 6 Tab to the OK button and press Enter to select it You return to the main 3BM screen 7 When you are finished making changes press F8 to save them and exit 3BM Enabling and Disabling Queuing...

Страница 127: ...he Configure Disk Array screen Tab to the field Drive Queuing Mode 5 Press Enter to display the choices use the arrow keys to select the setting you want Enabled or Disabled and press Enter again to c...

Страница 128: ...s the writing of data to disk and preserves a copy of data that has not yet been written to the disk media Following a power failure or in the event of accidental drive removal and reinsertion the fir...

Страница 129: ...rough 3DM In 3DM the StorSave Profile is a unit policy that can be set on the Controller Settings page To set the StorSave profile through 3DM 1 Choose Management Controller Settings from the menu bar...

Страница 130: ...for example a RAID 1 to a RAID 5 Unit Capacity Expansion for example adding a 4th drive to a 3 drive RAID 5 You can also use RLM to change the stripe size of a unit This section includes the following...

Страница 131: ...rom a 3 drive RAID 0 to a 4 drive RAID 5 with both having the same stripe size As can be seen every piece of user data is moved from its original physical location Figure 57 RAID Level Migration Examp...

Страница 132: ...al drives Once the unit is split any pending writes cannot be written to the second drive In addition the file system on the drive will not be clean Instead shut down the system replace one of the dri...

Страница 133: ...on Expanding Unit Capacity You can expand a unit s capacity by adding one or more drives to it without changing the RAID level except for singles and RAID 1 units Since a single can only have one driv...

Страница 134: ...iguration After you change the configuration of a unit you must inform the operating system of the change and you may need to re partition the unit Resizing the file system to match the new partition...

Страница 135: ...own as a dynamic disk right click on it and choose Extend Volume to launch the Extend Volume wizard In Linux and FreeBSD the tools to expand the partition and file system can vary depending on the ker...

Страница 136: ...u want to keep 3 Unmount the unit This step is very important If a unit is not unmounted and you delete it it is the equivalent of physically yanking a hard drive out from under the operating system Y...

Страница 137: ...rough 3BM In 3BM the command for deleting a unit is on the main 3BM screen To delete a unit through 3BM 1 At the main 3BM screen select the unit in the list of Exportable Units by highlighting it and...

Страница 138: ...re the drives for safekeeping purposes This process is sometimes referred to as array roaming Hot swap carriers are required unless you power down first When you remove a unit in contrast to deleting...

Страница 139: ...Management and select Disk Management Remove the logical drive letter for the unit Under Linux and FreeBSD you can unmount the unit with this command umount mount location 3 In 3DM choose Management M...

Страница 140: ...If your drives are not in hot swap carriers you do not need to remove a unit via 3BM Simply power down and remove the applicable drives Refer to your system s user guide for details on removing fixed...

Страница 141: ...3ware 9000 series controller You will then have the advantages of the 9000 series controller The drive units must be in normal mode before moving the units Incomplete degraded rebuilding or initializ...

Страница 142: ...0 board as described earlier in this section Moving Legacy JBOD Units to a 9000 Controller The steps for moving legacy JBOD units vary depending on the operating system you are using To move legacy JB...

Страница 143: ...n 5 Press the F8 or Esc key to save and exit Moving Units from a 9500S to a 9550SX 9590SE or 9650SE Controller It is fairly straightforward to move units from a 9500S controller to a 9550SX 9590SE or...

Страница 144: ...9650SE 5 Boot to the operating system 6 Install the driver for the 9550SX 9590SE or 9650SE If you are using Linux you may also need to update the initial RAM disk with the mkinitrd command 7 Reboot if...

Страница 145: ...To remove a drive 1 In 3DM choose Management Maintenance Note When you add a drive to your system and connect it to the controller it is automatically detected and listed in 3DM If it does not immedi...

Страница 146: ...part of a unit the unit may become degraded as shown in Figure 60 Figure 60 Result of Removing Drive from Unit in 3DM Rescanning the Controller When you make a change by physically adding or removing...

Страница 147: ...DM 1 In 3DM choose Management Maintenance 2 On the Maintenance page click Rescan Controller 3DM scans the controller for information about units and drives and updates the information shown on the Mai...

Страница 148: ...are described in this section which is organized into the following topics Checking Unit and Drive Status through 3DM Enclosure LED Status Indicators Unit Statuses Drive Statuses About Degraded Units...

Страница 149: ...building while another part is degraded or initializing When this is the case you will see both statuses listed at the top unit level When you drill in to see details you will be able to see which the...

Страница 150: ...ng table is an example only Check with your chassis documentation for more details Not all features may be supported or implemented in the same way Table 11 Meaning of LED Colors and Behavior Color Dr...

Страница 151: ...unit is valid For more information see About Verification on page 151 Verify Paused The unit is set to verify however scheduling is enabled and the present time is not during a scheduled timeslot Veri...

Страница 152: ...t drive failure RAID 5 RAID 6 and RAID 50 units achieve fault tolerance by using a simple exclusive OR function to generate the parity data that is distributed on all drives RAID 6 adds an extra level...

Страница 153: ...n also be used to notify you of events The next few pages describe these capabilities Viewing Alarms Errors and Other Events on page 143 Using the Alert Utility Under Windows on page 144 Downloading a...

Страница 154: ...page 144 A list of the possible error and other event messages is provided under Error and Notification Messages on page 224 To view alarms errors and other events in 3DM 1 Choose Monitor Alarms The...

Страница 155: ...Windows System Tray 2 In the Windows Audible Visual Alarm window select the types of alerts you want to be notified of If you want to turn off the sound alarm and only have a pop up message appear che...

Страница 156: ...og link 4 When the Save or Open dialog box appears navigate to where you want to save the log and click OK Viewing SMART Data About a Drive You can view SMART Self Monitoring Analysis and Reporting Te...

Страница 157: ...erformance so using a schedule can minimize the impact This section includes the following topics related to background tasks About Initialization About Verification Starting a Verify Manually Rebuild...

Страница 158: ...overwriting any existing data in the process In contrast background initialization uses an algorithm to resynch the parity information on the drives and does not rewrite existing data A foreground in...

Страница 159: ...with 5 or more disks and RAID 50 units with 2 subunits of 5 or 6 disks will improve after the unit has been initialized For these configurations initialization begins automatically after you create t...

Страница 160: ...er drive or set of drives Notes For RAID 5 with more 5 or more drives it is strongly recommended that you initialize the unit before using it Initializing such a unit is critical to insuring data inte...

Страница 161: ...to replace drives before they fail You can manually request a verify or you can enable the Auto Verify policy and the controller will automatically start verification once every 24 hours See Starting...

Страница 162: ...e the controller writes to that location the drive reallocates the data to a different sector Verification of Redundant Units Verification of redundant units also reads each sector working from lowest...

Страница 163: ...otification of repair is posted to the alarms list The result is a restoration of drive and data integrity the primary and redundant data are again both valid If the unit is not redundant it is recomm...

Страница 164: ...rt actively verifying until the next scheduled time A Stop Verify link appears next to the unit on the Maintenance page If you need to stop the verify process use this link If initialization starts be...

Страница 165: ...the rebuild itself only happens when the operating system is running Rebuilds on multiple units can take place simultaneously If multiple drives are faulted in a RAID 10 configuration the drives are r...

Страница 166: ...g box displays available drives select the drive you want to replace the failed drive and click OK Figure 65 Selecting a Drive when Rebuilding 5 If the degraded unit has more than one failed drive for...

Страница 167: ...You can start the rebuild of a degraded unit manually in 3BM or you can let the system boot and allow the rebuild to take place automatically To start the rebuild of a unit through 3BM 1 Reboot the s...

Страница 168: ...the operating system finishes loading If the rebuild fails and you have no available drives do one of the following If your system has hot swap carriers you can replace the failed or Not In Use drive...

Страница 169: ...ave your changes and exit 3BM The unit will begin rebuilding about ten minutes after the operating system finishes loading and the 3ware driver has loaded Cancelling a Rebuild and Restarting It with a...

Страница 170: ...e Rate and Verify Rate Initialization occurs at the Rebuild rate To change the background task rate 1 Choose Management Controller Settings from the menu bar 2 In the Background Task Rate section of t...

Страница 171: ...unit will occur at the same time Scheduling Background Tasks You can set up scheduling windows for when background tasks occur so that routine maintenance of storage media occurs when it will be leas...

Страница 172: ...it left off at the next scheduled time block Viewing Current Task Schedules You can see the current schedules for background tasks on the Scheduling page To view the current task schedule 1 Choose Man...

Страница 173: ...build migrate or verify a unit right away without waiting for the next scheduled time In this case you can disable the schedule as described below You can easily disable a current Verify or Rebuild Mi...

Страница 174: ...asks or Self test Tasks select it from the drop down list at the top of the page 3 Select the checkbox next to the schedule s you want to remove 4 Click the Remove Checked button The page refreshes an...

Страница 175: ...been exceeded For more information about these self tests see the 3DM Reference section Scheduling page on page 201 Initially these tests are set to run every 24 hours You can change the schedule for...

Страница 176: ...ividual drives by blinking their associated LED You can issue the command to blink the LED through 3DM For details about what the different LED patterns on the enclosure mean see Enclosure LED Status...

Страница 177: ...e and no longer need to see the LED return to this page and uncheck the Identify box To blink the LEDs for all drives in a unit 1 Choose from the main menu in 3DM 2 On the list of units locate the uni...

Страница 178: ...sions see the Controller Summary page on page 190 Using 3BM you can see the firmware version see Displaying Controller Information on page 65 Using the CLI you can see both the driver and firmware ver...

Страница 179: ...download them individually from the website To download the driver or firmware 1 On the 3ware website www 3ware com navigate to Service and Support Software Downloads 2 Click Download Released Softwar...

Страница 180: ...driver diskette extract the driver files to a floppy in step 9 above Refer to the user manual for your operating system for more details Updating the 3ware Driver and Firmware Under Windows You can up...

Страница 181: ...the folder that contains the driver files you extracted in step 1 5 Double click on the file 3wUpdate exe You see a progress dialog box while the update utility checks your system for the driver and f...

Страница 182: ...ure 72 Update Utility Showing Available Drivers If either your driver or firmware is current the screen reflects that Figure 73 Update Utility Showing Drivers Already Current The updater tells you whe...

Страница 183: ...as available Figure 74 Update Utility Showing Only Driver Available If the driver is not found the 3ware Driver Firmware Update dialog box will show only the firmware as available Figure 75 Update Ut...

Страница 184: ...your computer click Yes to do so If you were working from a driver diskette be sure to remove the floppy diskette before Windows reboots 8 If the driver you are installing is unsigned you will receive...

Страница 185: ...box appears it lists the number of controllers in your system shows available updates for the device driver and available firmware updates for the first controller Figure 76 Show Update Utility Showi...

Страница 186: ...s as described under Updating the Driver and Firmware on page 169 2 Backup any critical data prior to updating the 3ware driver 3 Log in to your system as system administrator 4 From the Start menu op...

Страница 187: ...screen asks whether you want to connect to Windows Update Select No not this time and click Next Figure 78 Windows XP Service Pack 2 Windows Update Query 12 When you reach the Hardware Update Wizard...

Страница 188: ...er Guide 13 At the next screen Figure 80 choose Don t search I will choose the driver to install and click Next Figure 80 Search for Driver Screen 14 When the Select the Device Driver screen appears F...

Страница 189: ...Disk Dialog Box 16 When the Select the Device Driver dialog box reappears select the appropriate driver Figure 83 and click Next Be sure to select the correct driver name otherwise the driver upgrade...

Страница 190: ...of the Hardware Update Wizard appears again informing you that Windows has found new hardware If you are using Windows XP Service Pack 2 the screen asks whether you want to connect to Windows Update F...

Страница 191: ...ng the 3ware driver 3 Change the directory to the location with the extracted driver 4 Copy the files to the appropriate directory In the commands below replace version with the applicable Red Hat or...

Страница 192: ...s add the following line to etc modules conf alias scsi_hostadapter 3w 9xxx o For 2 6 Kernels add the following line to etc modprobe conf alias scsi_hostadapter 3w 9xxx ko 6 Complete the upgrade by up...

Страница 193: ...xx ko For SuSE high memory systems 2 6 kernels cp version 3w 9xxx big lib modules kernel kernel drivers scsi 3w 9xxx ko 5 Run sbin depmod a 6 Make sure the file lib modules kernel string modules dep c...

Страница 194: ...wledgebase article 14546 http www 3ware com KB article aspx id 14546 Updating the 3ware Driver Under FreeBSD The following steps describe how to update the 3ware driver under FreeBSD To update the 3wa...

Страница 195: ...the utility the versions currently installed on your controller and recommends whether to upgrade the firmware or not Warning We strongly recommend backing up your data before updating the firmware Up...

Страница 196: ...tery backup unit attached to your controller in both 3DM 2 and 3BM To view information about a BBU in 3DM 2 On the menu bar choose Monitor Battery Backup The Battery Backup page appears on which you c...

Страница 197: ...s While running the battery test and until charging is completed write cache is temporarily disabled For how to replace the battery see the installation guide that came with your controller To test th...

Страница 198: ...ller User Guide Figure 87 BBU Information Screen While Battery is Testing To test the battery in a BBU in 3BM 1 At the main 3BM screen tab to the BBU button and press Enter 2 Tab to Test Battery Capac...

Страница 199: ...organized by 3DM page as the pages are organized on the 3DM menu bar Controller Summary page Controller Details page Unit Information page Unit Details page Drive Information page Drive Details window...

Страница 200: ...erial The serial number of the controller The serial number is also printed on a sticker on the outside bracket of the controller Firmware The firmware version running on the controller Driver The dri...

Страница 201: ...are version running on the controller Driver The driver version being used to interact with the controller BIOS The BIOS version on the controller Boot Loader Boot Loader version on the controller Mem...

Страница 202: ...lick the link in the Unit column Unit The unit number assigned to the unit by the firmware For 9000 series controllers unit numbers are in sequential order Name If a name has been given to this unit i...

Страница 203: ...Details page appears when you click an ID number on the Unit Information page Because it is a sub page of Unit Information the page title in the menu bar continues to display Unit Information even wh...

Страница 204: ...gle Disk Spare JBOD or Disk Volumes Displays the number of volumes in a unit This is usually 1 If you created a boot volume on this unit or if you have a unit on which you have enabled the auto carvin...

Страница 205: ...the Port column Port The port to which the drive is connected Model The model of the drive Capacity The physical capacity of the drive Note that the capacity as shown on 3DM screen is calculated as 1K...

Страница 206: ...dow appears when you click a Port on the Drive Information page This Drive Details window shows some Extra Drive Information including NCQ and SATA Link Speed support and the SMART data for the drive...

Страница 207: ...ields show the fastest link speed that the disk drive supports and the current speed that the drive is running SMART Data SMART data is displayed as hex values Consult your disk drive manufacturer for...

Страница 208: ...s each unit on the current controller and shows you whether the policies are currently enabled or disabled for each unit Write Cache When write cache is enabled data is stored locally in memory on the...

Страница 209: ...ables Native Command Queuing NCQ for drives in the unit By default queuing is disabled You can enable it if desired NCQ only operates when the feature is enabled at both the drive and the controller I...

Страница 210: ...y unit that is over a specified size known as the carve size will be broken down into multiple volumes of that size plus a remainder volume The default carve size is 2048 GB 2 TB For example using the...

Страница 211: ...r when background tasks rebuild migrate initialize verify and self test should occur Background tasks can have impact on the performance of your system so you may prefer to schedule them at times when...

Страница 212: ...one or more of the existing scheduled slots and then add new slots For step by step instructions for adding and removing schedules and setting schedules to be followed or ignored see Scheduling Backg...

Страница 213: ...he system is booted Check SMART Thresholds This test checks to see whether SMART thresholds have been exceeded The SMART thresholds indicate when a drive is likely to fail based on the number of error...

Страница 214: ...nagement Maintenance from the menu bar The Maintenance page lets you perform maintenance tasks on existing units on the current controller and lets you create new units by configuring available drives...

Страница 215: ...ch unit created In Windows the device manager will reflect the changes under the disk drives icon This process is known as importing drives If new drives do not have any data indicating they were prev...

Страница 216: ...rrently active or paused click on the unit number to display the Unit Details page which has that information For an explanation of the statuses see Unit Statuses on page 140 Drive Information Port Th...

Страница 217: ...specify whether more processing time should be given to verifying or to I O For more information see About Verification on page 151 and Setting Background Task Rate on page 160 While a unit is verifyi...

Страница 218: ...nal available drives In the dialog box are two drop down menus one for choosing the RAID level and one for choosing stripe size You can only migrate a unit to a RAID level that will be larger than the...

Страница 219: ...ounted and the system is not accessing it For example make sure you are not copying files to the unit and make sure that there are no applications with open files on that unit If a unit is not unmount...

Страница 220: ...ype Name Stripe size if applicable and unit policy settings A window like the one below shows the drives you selected and lets you specify configuration settings Caution Before you click Delete Unit m...

Страница 221: ...e configuring a RAID 50 with twelve drives an additional field appears in which you select the number of drives per subunit 3 4 or 6 Figure 99 Configuring a RAID 50 with 12 Drives Name You can enter a...

Страница 222: ...ilable Protection Balanced and Performance For more information see Setting the StorSave Profile for a Unit on page 117 Alarms page Figure 100 Alarms Page The Alarms page appears when you click Monito...

Страница 223: ...of the event Three levels are provided Errors are shown next to a red box Warnings are shown next to a yellow box Information is shown next to a blue box Time The time shown for alarms is the time th...

Страница 224: ...ered on This indicates the BBU is not able to backup the 3ware RAID controller When the BBU is in the charging state write caching is disabled automatically on all units attached to the controller Fau...

Страница 225: ...e a higher number in this field As the battery ages the backup capacity diminishes Last Capacity Test Indicates the date when the last battery test was completed To test the battery click the Test Bat...

Страница 226: ...e ID of the enclosure and the status of the fan either OK or Unknown Temp Sensor Summary Shows the ID of the enclosure and the temperature in the enclosure The maximum temperature for successful use o...

Страница 227: ...ge refresh frequency whether remote access is permitted and the incoming port for 3DM to listen for requests The initial settings for most of these preferences are specified during installation of 3DM...

Страница 228: ...dress Save E mail Settings button Saves the e mail notification settings Send Test Message button Sends a test message using the saved e mail settings Password Use the fields in this section to set th...

Страница 229: ...ote computer HTTP Settings Listening Port This field specifies the HTTP port to be used by 3DM when listening for communications The default port setting is 888 If you change this port make sure the p...

Страница 230: ...frequently asked questions software releases and compatibility information related to 3ware RAID controllers refer to 3ware support page at http www 3ware com support 3ware knowledgebase http www 3war...

Страница 231: ...want to take a snapshot of these pages so that you can respond to questions about your system configuration to the Customer Support representative Command Logging All changes that are made to RAID co...

Страница 232: ...you are using a PCI riser card try installing it directly Some riser cards are not compatible with certain motherboards Q2 Need to boot from a non 3ware device without changing the boot order If the...

Страница 233: ...BIOS updates Q5 A unit shows up as unusable similar to the following 4 drive 64K RAID 5 missing drive s Port 1 Maxtor 6B300S0 279 86 GB Port 2 Maxtor 6B300S0 279 86 GB The unit has some but not all o...

Страница 234: ...parentheses at the beginning of the message description For example in the string 0x04 0x002B 002B is the message number To find additional information about the message 2B you would look up 002B in t...

Страница 235: ...0027 DCB checksum error detected 0028 DCB version unsupported 0029 Verify started 002A Verify failed 002B Verify completed 002C Source drive ECC error overwritten 002D Source drive error occurred 002...

Страница 236: ...voltage is low 0046 Battery voltage is high 0047 Battery voltage is too low 0048 Battery voltage is too high 0049 Battery temperature is normal 004A Battery temperature is low 004B Battery temperatur...

Страница 237: ...ild Removing a drive from a redundant unit will cause this message to display immediately For more information see Locating a Drive by Blinking Its LED on page 166 0003 Controller error occurred The 3...

Страница 238: ...ternal algorithm to logically connect drives that belong to the same unit If after rollcall a member of a unit is not found the INCOMPLETE UNIT message is sent Examples of incomplete units are as foll...

Страница 239: ...mittent disk drive power cable or interface cable For links to drive manufacturer diagnostic utilities and troubleshooting advice see http www 3ware com KB article aspx id 14924 000A Drive error detec...

Страница 240: ...If this unit was a redundant unit and the initialize failed happened because of a problem on a particular disk drive then the unit will be degraded on that disk drive s port 000F SMART threshold exce...

Страница 241: ...ve insertion caused a unit that was inoperable to become operational again Any data that was on that unit will still be there This message is sent only after the offline timer expires 20 seconds 0021...

Страница 242: ...al attention should be paid to any drive in a unit that begins to indicate sector repair messages This may be an indication of a drive that is beginning to fail You may wish to replace the drive espec...

Страница 243: ...led the Drive Configuration Block DCB As part of power on initialization the 3ware RAID controller performs a checksum of the DCB area to ensure consistency If this error occurs please contact 3ware t...

Страница 244: ...automatically initialized The initialize will not erase user data but will recalculate and rewrite user parity data For more information on the Initialize function see About Initialization on page 14...

Страница 245: ...e correctly jumpered to correspond to UDMA 100 or 133 drives For a list of compatible drives see http www 3ware com products compatibility_sata asp 0032 Spare capacity too small for some units This me...

Страница 246: ...to the 3ware 9500S controller which has removable memory This message will be sent if there is no SODIMM memory connected to the controller In this case the controller is inoperable and cannot be use...

Страница 247: ...s message is sent when migration is paused Migration follows the rebuild schedule For more information see Scheduling Background Tasks on page 161 003F Flash file system error detected The 3ware RAID...

Страница 248: ...latent error that needs to be addressed before any future errors so this DCB is re written with the primary copy to rectify the errors found If this error occurs we recommend verifying the unit See St...

Страница 249: ...ll be posted to the host 004A Battery temperature is low The Battery Backup Unit measures and evaluates the battery pack temperature on a continuous basis If the temperature is below a warning thresho...

Страница 250: ...e charge cycle and may take up to 20 hours to complete During this test the Battery Backup Unit cannot backup the 3ware RAID controller all units have their write cache disabled until the test complet...

Страница 251: ...is posted when the Battery Backup Unit completes a battery charge cycle 0057 Battery charging fault This message is posted when the charger of the Battery Backup Unit has detected a battery fault duri...

Страница 252: ...to backup the 3ware RAID controller in case of a power failure This message is posted when the result of the health test is below the fault threshold This indicates that the battery pack must be repla...

Страница 253: ...pendices The following information is available in the appendices Appendix A Glossary on page 244 Appendix B Compliance and Conformity Statements on page 250 Appendix C Warranty Technical Support and...

Страница 254: ...city Serial ATA SATA RAID storage solutions A Chip AccerATA chip Automated data port to handle asynchronous ATA disk drive interface AMCC Applied Micro Circuits Corporation provides the essential buil...

Страница 255: ...unit configuration data on the drives Note that in 3ware software tools arrays are referred to as units DCB Disk configuration block This is 3ware proprietary RAID table information that is written t...

Страница 256: ...rive In 3BM BIOS exportable units and drives are those that will be available to the operating system when you boot your computer Fault tolerant A RAID unit which provides the ability to recover from...

Страница 257: ...three tier units and as such will have sub sub units JBOD An unconfigured single drive The acronym is derived from just a bunch of disks Note that earlier versions of the 3ware RAID controller exporte...

Страница 258: ...ge system s performance and provide fault tolerance protection against data loss Rebuild task schedule The specification for when rebuilding may occur including start time and duration Rebuild a unit...

Страница 259: ...ts in an increase in performance Striping occurs in RAID 0 5 10 and 50 Subunit A logical unit of storage that is part of another unit For example the mirrored pairs RAID 1 in a RAID 10 unit are subuni...

Страница 260: ...f not installed and used in accordance with the instructions may cause harmful interference to radio communications However there is no guarantee that interference will not occur in a particular insta...

Страница 261: ...on Technical Regulations and or normative documents EN 55022 Limits and methods of measurements of radio interference character istics of information technology equipment EN 61000 4 2Electromagnetic c...

Страница 262: ...r new or refurbished to be functionally equivalent to new Products or parts replaced under this provision shall become the property of AMCC Battery Backup Unit BBU Hardware 1 Year Hardware Warranty AM...

Страница 263: ...ts which vary from state to state Some states do not allow the exclusion of incidental or consequential damages or allow limitation of implied warranties or their duration so that the above exclusions...

Страница 264: ...s 3ware software library is accessible at http www 3ware com support download asp Web based software downloads feature upgrading multiple switches simultaneously For specific answers to questions or t...

Страница 265: ...y is below warning level 0058 241 Battery capacity test completed 0050 240 Battery capacity test is overdue 0053 241 Battery capacity test started 004E 240 Battery charging completed 0056 241 Battery...

Страница 266: ...t 193 194 Auto Rebuild policy 15 87 setting 90 200 Auto Verify policy for units 198 setting 3DM 114 auto carving 91 auto carving policy 87 auto carving policy setting 200 available drives 3DM 210 B ba...

Страница 267: ...red 0003 227 Controller Information screen 65 Controller reset occurred 0001 227 Controller Settings page 3DM 197 Controller Summary page 3DM 190 conventions in the user guide ix creating a unit 3BM 1...

Страница 268: ...ault tolerant definition 246 FCC Radio Frequency Interference Statement 250 firmware definition 246 showing version 190 191 updating 169 updating in Windows 170 Flash file system error detected 003F 2...

Страница 269: ...ted 0035 235 Migration failed 0034 235 Migration paused 003E 237 Migration started 0033 235 mirrored disk array definition 4 247 RAID 1 5 motherboard boot sequence 29 Mozilla setting up 67 Multi LUN s...

Страница 270: ...task duration 162 Scheduling page 3DM 201 Sector repair completed 0023 232 security certificate when starting 3DM 74 security message related to active content in the 3ware HTML Bookshelf x self test...

Страница 271: ...mber definition 249 unit statuses 140 verifying a unit 153 in 3BM 154 in 3DM 154 volumes 194 write cache 3DM 198 write cache enabling and disabling 113 Unit deleted 000D 230 Unit Details page 3DM 193...

Страница 272: ...volumes in a unit 194 multiple from one unit 91 resulting from auto carvings 194 W Warranty 252 WHQL Windows Hardware Quality Labs Microsoft 251 WinAVAlarm 144 Windows alert utility WinAVAlarm 144 wo...

Отзывы: