manualshive.com logo in svg

3Ware 9650SE-2LPB-10 - Sataii Raid Hardware 10Pk, User Manual, Page: 198 / 272

Share
Download
3Ware 9650SE-2LPB-10 - Sataii Raid Hardware 10Pk User Manual Download Page 198

Chapter 10. Maintaining Your Controller

188

3ware Serial ATA RAID Controller User Guide

Figure 87.  BBU Information Screen While Battery is Testing

To test the battery in a BBU in 3BM

1

At the main 3BM screen, tab to the 

BBU

 button and press 

Enter

.

2

Tab  to 

Test Battery Capacity

 and press 

Enter

.

Figure 88.  Battery Backup Information Screen in 3BM

Summary of Contents for 9650SE-2LPB-10 - Sataii Raid Hardware 10Pk

Page 1: ...3ware Serial ATA RAID Controller Supports the 9650SE 9590SE and 9550SX Models PN 720 0146 00 October 2006 User Guide...

Page 2: ...o 3BM StorSwitch and R5 Fusion are all trademarks of AMCC PowerPC and the PowerPC logo are trademarks of International Business Machines Corporation Linux is a registered trademark of Linus Torvalds i...

Page 3: ...Motherboard Boot Sequence 29 What Next 29 Chapter 4 Driver Installation 30 Driver Installation Under Windows 31 Materials Required 31 Creating a 3ware Driver Diskette 32 Installing the 3ware Driver an...

Page 4: ...g Your Controller 85 Viewing Information About a Controller 85 About Controller Policies 87 Viewing Controller Policies 89 Setting the Auto Rebuild Policy 90 Using Auto Carving for Multi LUN Support 9...

Page 5: ...146 Background Tasks 147 About Initialization 148 About Verification 151 Starting a Verify Manually 153 Rebuilding Units 155 Cancelling a Rebuild and Restarting It with a Different Drive 159 Setting...

Page 6: ...age 217 Chapter 12 Troubleshooting 220 Web Resources 220 Before Contacting Customer Support 221 Command Logging 221 Problems and Solutions 222 Hardware Installation 222 Software Installation 222 Probl...

Page 7: ...or performing tasks using two tools one at the BIOS level 3ware BIOS Manager 2 referred to as 3BM 2 and one that runs in a browser 3ware Disk Manager 2 referred to as 3DM 2 You can also perform many t...

Page 8: ...change existing configurations and set unit policies 9 Maintaining Units Describes how to check unit and drive status review alarms and errors schedule background maintenance tasks and manually start...

Page 9: ...different operating systems including Mac OS X Microsoft Windows Linux and FreeBSD and runs in a number of different browsers In addition the version shown in screenshots may not match your version F...

Page 10: ...elf Window Click the Show Navigation button to display the Table of Contents Note The 3ware HTML Bookshelf is created as a set of HTML documents that are often displayed from a website When installed...

Page 11: ...topics What s New with 9650SE Models System Requirements Understanding RAID Concepts and Levels 3ware Tools for Configuration and Management Monitoring Maintenance and Troubleshooting Features What s...

Page 12: ...y_sata2 asp Drives may be of any capacity or physical form factor The length of shielded and unshielded interface cables may not exceed 1M 39 for Serial ATA controllers Table 2 Required Slots for 3war...

Page 13: ...te power supply for drives 3DM 2 3ware Disk Manager displays information in a browser It requires one of the following browsers Internet Explorer 5 5 and later Mozilla Firefox 1 2 and later Netscape 7...

Page 14: ...a 3ware RAID controller provides improved performance because 3ware s TwinStor technology reads from both drives simultaneously Striping Striping across disks allows data to be written and accessed o...

Page 15: ...ed in a striped disk array see Figure 2 large files are distributed across the multiple disks using RAID 0 techniques Striped disk arrays give exceptional performance particularly for data intensive a...

Page 16: ...neously reading alternating sections of the file For the smaller random transactions the data is read from a single optimal drive head Figure 3 RAID 1 Configuration Example RAID 5 RAID 5 provides perf...

Page 17: ...uilding or one sub unit may be initializing while another is verifying RAID 6 requires a minimum of five drives Performance and storage efficiency also increase as the number of drives increase Figure...

Page 18: ...AID 50 is a combination of RAID 5 with RAID 0 This array type provides fault tolerance and high performance RAID 50 requires a minimum of six drives Several combinations are available with RAID 50 For...

Page 19: ...taken to prevent system hangs and data loss JBOD A JBOD acronym for Just a Bunch of Disks is an unconfigured disk attached to your 3ware RAID controller JBOD configuration is no longer supported in t...

Page 20: ...fault tolerance and a read speed advantage over non RAID disks RAID 5 This type of unit provides performance fault tolerance and high storage efficiency RAID 5 units can tolerate one drive failing bef...

Page 21: ...10 with hot spare Combination of RAID 0 RAID 1 hot spare single disk 6 or more RAID 6 RAID 6 with hot spare RAID 50 Combination of RAID 0 1 5 6 10 hot spare single disk Table 5 Drive Capacity RAID Lev...

Page 22: ...ult prior to unit creation If a unit over 2 TB was created prior to enabling the auto carve option its capacity visible to the operating system will still be 2TB no additional capacity will be registe...

Page 23: ...specify the type of events that should generate these notifications For details see Using the Alert Utility Under Windows on page 144 3ware CLI Command Line Interface The 3ware CLI provides the functi...

Page 24: ...nitialize rebuild verify and self test tasks can all be run in the background at scheduled times This lets you choose a time for these tasks to be run when it will be least disruptive to your system Y...

Page 25: ...rive needs to be checked or replaced For more information see Locating a Drive by Blinking Its LED on page 166 Auto Rebuild For times when you do not have a spare available setting the Auto Rebuild po...

Page 26: ...the installation guide that came with your 3ware controller If you do not have a hardcopy of the installation manual it is available in the 3ware Documentation folder on your 3ware CD and you can down...

Page 27: ...nagement and Maintenance Features 3ware RAID controllers include a number of features to help you manage and maintain the controller and your configured units The default settings for these features a...

Page 28: ...or Factor 2048 GB 3BM 3DM CLI Drives Per Spinup 1 3BM CLI Delay Between Spinup 6 seconds 3BM CLI Export Unconfigured JBOD Disks Disabled 3BM CLI Staggered Method ATA 6 3BM Staggered Spinup Enabled 3BM...

Page 29: ...CLI a Although the default Self test Task Schedule is for 24 hours self test tasks are run only at the beginning of that time period and take just a few minutes For more information about task schedul...

Page 30: ...rts and the number of drives attached For more information see Determining What RAID Level to Use on page 10 Basic Steps for Creating a Unit The process of configuring your RAID units includes these m...

Page 31: ...your system a screen lists the available boards See Figure 9 In this case highlight the board with which you want to work and press Enter Figure 9 3ware Controller Board Selection Screen You see a sc...

Page 32: ...the drives and create a unit 1 Select the drives to be included by highlighting each one and pressing Enter to select it When you select a drive an asterisk appears next to it in the left most column...

Page 33: ...ou selected Figure 13 List of Configuration Choices for Four Drives 3 Use the arrow keys to highlight the desired RAID configuration and press Enter For information about the different RAID levels and...

Page 34: ...s or press Tab to move to the field you want to change 2 Press Enter to see the available options 3 Use the arrow keys to select the option you want and press Enter to choose it For details about thes...

Page 35: ...ve finished making all your changes and press F8 3 If the volume summary screen appears review the information and press any key to continue Multiple volumes will be created if you entered a Boot Volu...

Page 36: ...l the operating system on one of them make that unit be the first unit Unit 0 in the list of Exportable Units To move a unit up in the list highlight it and press the Page Up key You will also want to...

Page 37: ...to the drives is complete Once you have booted to the operating system background initialization of the unit will begin after a delay of up to ten minutes For complete information about initializatio...

Page 38: ...the words Hot Spare appear next to the drive in the Available Drives list Figure 17 Hot Spare Indicated If a hot spare is already enabled you can disable it by following the same process Leaving Indiv...

Page 39: ...you have other disks installed on the motherboard the 3ware RAID controller should precede them in boot order If you will install your OS on a disk or unit attached to the 3ware RAID controller specif...

Page 40: ...nit connected to another controller or to the motherboard you start the operating system and then install the driver Details for both situations are described in this chapter Driver information is org...

Page 41: ...ows on a new drive or unit Install the 3ware driver when Windows is already installed on a different device Materials Required 3ware software CD ROM Microsoft Windows 2000 XP or Server 2003 installati...

Page 42: ...d Windows on a New Unit If you want to install Windows on a new drive configuration managed by the 3ware RAID controller follow the instructions in this section If Windows is already installed on anot...

Page 43: ...to Windows on page 38 Installing the Driver on a System that Boots from a Different Device If you are installing the 3ware RAID controller on a system that already has the operating system installed o...

Page 44: ...he Wizard prompts you to select a device driver click Have Disk and then indicate that it is on the CD or floppy 3 If the Digital Signature Not Found message appears click Yes to continue the installa...

Page 45: ...Found New Hardware Wizard After the driver has been installed continue with the instructions under Making Units Managed by a 3ware Controller Available to Windows on page 38 To install the 3ware driv...

Page 46: ...CD or floppy 3 If the Digital Signature Not Found message appears click Yes to continue the installation 4 When the Completing the Found New Hardware Wizard screen appears Figure 23 click Finish Figu...

Page 47: ...are Wizard 7 When the second Completing the Found New Hardware screen appears click Finish Figure 25 Completing Found New Hardware Wizard After the driver has been installed continue with the instruct...

Page 48: ...in as the system administrator 2 Use Disk Administrator to partition and format the new units or disks From the Start menu choose Administrative Tools Computer Management In the Computer Management w...

Page 49: ...it on page 42 Installing the 3ware Kernel Driver Module on a Red Hat or Fedora Core Linux System that Boots From a Different Device on page 44 SuSE Linux Installation Installing the 3ware Kernel Drive...

Page 50: ...uSE and other similar distributions running similar kernel strings Be Sure to Use the Correct Drivers The Linux 3w 9xxx drivers included on the 3ware CD are for 3ware 9650SE 9590SE 9550SX and 9500S se...

Page 51: ...i_host hostid stats where hostid is usually host0 unless other SCSI devices are available in which case it may be host1 or higher If you have a 2 6 kernel without sysfs type the following command dmes...

Page 52: ...o create a driver install diskette from the menu Extract the kernel driver module from the modules cgz file using this command zcat modules cgz cpio idvm Installing the 3ware Kernel Driver Module whil...

Page 53: ...ding 3w 9xxx 5 When prompted select the proper language and keyboard types for your locality 6 If asked for what type of media select Local CD ROM since you are installing from the Red Hat CD ROM For...

Page 54: ...ng refers to the kernel version The kernel string will have different endings depending on the kernel you are using For an SMP kernel multi processor the kernel string will end in smp For example 2 6...

Page 55: ...kernel drivers scsi 3w 9xxx ko If prompted to overwrite type y For Redhat workstation on AMD Opteron and Intel EM64T type cp mnt cdrom packages drivers linux redhat ws4_u4 x86_64 3w 9xxx ko or smp li...

Page 56: ...pe cp mnt cdrom packages drivers linux redhat ws4_u4 x86 3w 9xxx ko smp or hug lib modules put kernel string here kernel drivers scsi 3w 9xxx ko For Fedora Core on Intel x86 and AMD x86 type cp mnt cd...

Page 57: ...create a 3ware driver install diskette If Linux is already installed on another device you may install the 3ware kernel driver module from the 3ware software CD ROM To create a driver install diskette...

Page 58: ...talling SuSE Linux on a New Unit 1 Boot directly from the SuSE installation CD 1 or DVD 2 When installing SuSE press either the F6 key or the F3 key depending on the version You will then be prompted...

Page 59: ...e sles10 x86 3w 9xxx ko smp or big lib modules kernel string kernel drivers scsi 3w 9xxx ko For SuSE Linux Enterprise Server 10 0 64 bit x86_64 type cp cdrom packages drivers linux suse sles10 x86_64...

Page 60: ...e information see 3ware knowledgebase article 14546 http www 3ware com KB article aspx id 14546 Driver Installation Under FreeBSD A drive or unit managed by the 3ware RAID controller may be configured...

Page 61: ...ble at packages drivers freebsd src 6 x You can also download the FreeBSD kernel driver module from the 3ware web site http www 3ware com support index asp FreeBSD installation CD ROM or diskette Inst...

Page 62: ...9590SE 9650SE kernel driver module into the floppy drive 5 Run sysinstall In the sysinstall menu choose Configure Load KLD Load twa_9ksx ko from the floppy drive 6 Install the FreeBSD OS remove the fl...

Page 63: ...kernel driver module for FreeBSD 6 1 can be found on both the 3ware CD ROM and the 3ware website For step by step instructions see To update the kernel with new driver source on page 53 11 Build the...

Page 64: ...make install 5 Reboot your system When the system reboots the new kernel driver module will load automatically Installing the 3ware Kernel Driver Module on a FreeBSD System that Boots from a Device o...

Page 65: ...OS version being used The sources for the new kernel driver module for FreeBSD 5 4 can be found on both the 3ware CD ROM and the 3ware website For step by step instructions see To update the kernel w...

Page 66: ...roller User Guide 4 cd sys modules twa make 5 Comment out the entry Device twa 3ware 9000 series in usr src sys i386 conf or usr src sys amd64 conf 6 Rebuild the kernel and reboot your system 7 Load t...

Page 67: ...ut doing particular tasks in 3BM refer to the later sections in this guide Starting 3BM 2 You access 3BM during the start up process for your computer 1 Power up or reboot your system 2 While the syst...

Page 68: ...28 In this case highlight the board with which you want to work and press Enter Figure 28 AMCC Boards Selection Screen You see a screen similar to the warning message below warning you that changing y...

Page 69: ...es controllers in your system the 7000 8000 series controllers are not listed on the selection screen shown in Figure 28 Instead an additional BIOS summary will appear for the 7000 8000 series control...

Page 70: ...its have been configured If you have more than one unit the boot unit is the one at the top of the list You can change the order by highlighting a unit and pressing the PgUp or PgDn key Unusable Array...

Page 71: ...Keys OR Left and Right Arrow Keys OR Tab and Shift Tab Select or unselect what is currently highlighted A selection may be a drive in a list of drives a button at the bottom of the screen or a field...

Page 72: ...to the main 3ware BIOS Manager screen from the Advanced Details screen Any key Move a highlighted unit up or down in the list of exportable units The top most unit will become the bootable unit if yo...

Page 73: ...o show about configured units on the start up screen Expanded lists each unit and shows the specific drives that make up the unit Collapsed lists only the configured units Pause Time Sets the number o...

Page 74: ...r The BIOS Options Settings screen appears Figure 31 BIOS Option Settings Screen 3 Tab through the options and make the desired changes 4 Tab to OK and press Enter to return to the main screen Note Du...

Page 75: ...Tab to Information and press Enter 2 On the pop up menu select Controller and press Enter Note that the particular versions shown in the screen shot below may not be current Figure 32 Advanced Detail...

Page 76: ...nistrative and maintenance tasks locally or remotely Two levels of access to 3DM are provided user and administrator Users have view only access and can check the status of drives and units Administra...

Page 77: ...RAID controller You can also download the current version from the website at http www 3ware com support download asp 3DM must be installed on the system in which the controller is installed 3DM does...

Page 78: ...release To install 3DM on a Windows system 1 Insert the 3DM CD ROM and click Install 3DM 2 when the menu appears Or if you downloaded 3DM from the website find the file setup exe and double click it t...

Page 79: ...email server You can get the IP address of your mail server by typing nslookup file server name To find the mail server name ask your system administrator or check the e mail preferences or setup opt...

Page 80: ...tion gives you the option of starting 3DM and opening the browser window 8 If you elect to launch the web interface a Security Alert may appear as shown in Figure 35 Click Yes continue You may install...

Page 81: ...3DM2 from the Start Programs 3ware 3DM2 menu If you change add or remove an IP address or change the machine name of the machine on which you have installed 3DM 2 you will need to recreate the securit...

Page 82: ...onal information the name of the mail server the user name for the person who will send the E mail notification typically the local host name and the user name for the person who will receive the e ma...

Page 83: ...r Windows and as a daemon under Linux and FreeBSD Normally after installation the 3DM process starts automatically when you start your system It is a good idea to leave the 3DM process running on the...

Page 84: ...calhost 888 You can also replace localhost with the IP address of the computer that contains the 3ware controller For example https IP address 888 2 The first time you start 3DM when the security cert...

Page 85: ...ow to start it To start the 3DM daemon manually 1 Login as root on the machine on which 3DM is installed 2 Afterwards type etc init d tdm2 start 3 To start the 3DM web application open your browser an...

Page 86: ...rom a browser on any computer over an internet connection You do not need to install the 3DM software on the remote computer To connect to 3DM2 through your web browser In the address line of your bro...

Page 87: ...rovides summary or detail information about your 3ware RAID controller and the resources connected to it As you work in 3DM the Messages area just below the menu bar displays information about the res...

Page 88: ...Management Maintenance page The Monitor menu gives you access to the Alarms page the BBU page and the Enclosure Summary page The Alarms page shows a list of alarms including the specific alarm message...

Page 89: ...can also be set For details see Setting the Frequency of Page Refreshes on page 84 3DM Screens and What They re Used For The table below shows a list of the pages you work with in 3DM and describes w...

Page 90: ...rrent controller and provides summary information about each drive To see this page choose Information Drive Information from the menu bar Drive Details window Shows the SMART data for a specific driv...

Page 91: ...o view this page choose Monitor Alarms on the menu bar Battery Backup page Shows the status of a Battery Backup Unit BBU if one is installed and allows you to test the battery To view this page choose...

Page 92: ...u want to change User or Administrator 3 Type the current password in the Current Password field If you are changing the password for the first time the factory set default password is 3ware 4 Enter t...

Page 93: ...e e mail address es to which notifications are sent Separate multiple addresses with a comma or a semicolon Enter the SMTP server name or IP of the mail server for the computer where the 3ware control...

Page 94: ...ge Port The page refreshes and a message at the top of the screen confirms that the listening port has been changed Setting the Frequency of Page Refreshes Since the status of the drives attached to y...

Page 95: ...a Controller You can check the controller model serial number firmware and driver versions and the status of the 3ware RAID controller in your computer If you have more than one controller in your sys...

Page 96: ...s If you have more than one controller in the system you can switch between them by selecting the one you want from the Select Controller drop down list at the right of the menu bar This drop down is...

Page 97: ...en in 3BM Figure 43 Auto Rebuild Determines whether the Auto Rebuild policy is enabled or disabled When disabled degraded units can only be rebuilt with designated spares When enabled the controller f...

Page 98: ...whether JBOD disks should be exported to the operating system JBOD configuration is strongly discouraged for newly added disks on a 9000 series controller in favor of Single Disk so by default this s...

Page 99: ...nd Carve Size policies can be changed on this page The other policies can be changed in 3BM or through the CLI For a summary of the initial default settings see Table 6 Default Settings for Policies a...

Page 100: ...o Rebuild is disabled only spares will be automatically used to rebuild degraded units When Auto Rebuild is enabled the firmware will select drives to use for automatically rebuilding a degraded unit...

Page 101: ...ing system as separate volumes These chunks are sometimes known as multiple LUNs logical units However throughout the 3ware documentation they are referred to as volumes For example using the default...

Page 102: ...ividual volumes existing data is not affected To use auto carving 1 Enable the auto carving feature You can do so using 3DM or 3BM In 3DM enable Auto Carving at the bottom of the Management Controller...

Page 103: ...the future Existing units will not be affected To set the carve size in 3DM 1 Choose Management Controller Settings from the menu bar in 3DM 2 In the Other Controller Settings section at the bottom o...

Page 104: ...ugh these fields making the choices you want to use Staggered Spinup Choose enabled or disabled Number of Drives Per Spinup Select the number of drives from 1 to 4 8 12 or 16 depending on the number o...

Page 105: ...port of unconfigured disks 1 At the main 3BM screen Tab to Policy and press Enter 2 On the Policy Control screen Tab to Export Unconfigured Disks press Enter to display the choices use the arrow keys...

Page 106: ...Level Migration or Online Capacity Expansion Deleting a Unit Removing a Unit Moving a Unit from One Controller to Another Adding a Drive Removing a Drive Rescanning the Controller Configuring a New U...

Page 107: ...lude from one to sixteen drives in the unit depending on the number of drives available and the model of the controller you have For information about how many drives to select for a given RAID level...

Page 108: ...fault Auto Verify disabled by default Continue on Source Error During Rebuild disabled by default StorSave Profile Protection by default The particular policies that you can adjust when you create the...

Page 109: ...or greater and have enabled the Auto Carving policy the boot volume will be created in addition to any volumes created through auto carving For more information about auto carving see Using Auto Carvi...

Page 110: ...the RAID configuration you want 5 If stripe size applies to the RAID type you select select a Stripe Size Stripe size does not apply to RAID 1 6 Optional In the Name box enter a name for the unit up...

Page 111: ...s the carve size you may see multiple unit volumes in your operating system For details see Using Auto Carving for Multi LUN Support on page 91 In Linux a device node will now be associated with each...

Page 112: ...8 Figure 48 3BM Main Display To create a unit through 3BM 1 At the main 3BM screen select the drives to be included by highlighting each one and pressing Enter or Space to select it When you select a...

Page 113: ...for the unit then press Enter again to set the name 5 Tab to the RAID Configuration field and press Enter to display a list of available configurations for the number of drives you selected Figure 51...

Page 114: ...summary screen appears review the information and press any key to continue Multiple volumes will be created if you entered a Boot Volume Size of greater than zero 0 or if auto carving is enabled and...

Page 115: ...bed under Partitioning Formatting and Mounting Units on page 106 Ordering Units in 3BM If you configure multiple units in 3BM and you want to install the OS on one of them so that you can boot from th...

Page 116: ...dows Linux or FreeBSD For the appropriate steps see To partition format and mount under Windows on page 106 To partition format and mount under Linux on page 107 To partition format and mount under Fr...

Page 117: ...Partition type primary File system type ext2 Start 0 End 1074gb parted p Disk geometry for dev sda 0kB 1074GB Disk label type gpt Number Start End Size File system Name Flags1 17kB 1074MB 1074MB 4 Cr...

Page 118: ...degraded disk or larger is available the hot spare will automatically replace the failed drive in the unit without user intervention It is a good idea to create a hot spare after you create a redundan...

Page 119: ...words Hot Spare appear next to the drive in the Available Drives list Note 3ware s 9000 series RAID controllers use drive coercion so that drives from differing manufacturers and with slightly differe...

Page 120: ...name or rename a unit through 3DM 1 Make sure the appropriate controller is selected in the drop down list at the right of the menu bar 2 Choose Management Controller Settings from the menu bar in 3D...

Page 121: ...3BM Details about adjusting each policy are described on the following pages Unit Write Cache Determines whether write cache is enabled for the unit When the write cache is enabled data is stored loca...

Page 122: ...ntrollers only ECC errors are an indication of errors that have occurred on a particular drive since it was last read When not enabled a rebuild will abort upon encountering an ECC error and the unit...

Page 123: ...t s StorSave profile can also determine whether the write cache can be enabled or disabled A warning message will be given if the change is not permitted due to the StorSave setting and the state of t...

Page 124: ...own The initial default setting is for write cache to be enabled 5 Press Enter to display the choices use the arrow keys to select the setting you want and press Enter again to choose it 6 Tab to the...

Page 125: ...een When this policy is set ECC Error Correcting Code errors are ignored when they are encountered during a rebuild ECC errors are typically defects that have been detected in the drive since initiali...

Page 126: ...choose it 6 Tab to the OK button and press Enter to select it You return to the main 3BM screen 7 When you are finished making changes press F8 to save them and exit 3BM Enabling and Disabling Queuing...

Page 127: ...he Configure Disk Array screen Tab to the field Drive Queuing Mode 5 Press Enter to display the choices use the arrow keys to select the setting you want Enabled or Disabled and press Enter again to c...

Page 128: ...s the writing of data to disk and preserves a copy of data that has not yet been written to the disk media Following a power failure or in the event of accidental drive removal and reinsertion the fir...

Page 129: ...rough 3DM In 3DM the StorSave Profile is a unit policy that can be set on the Controller Settings page To set the StorSave profile through 3DM 1 Choose Management Controller Settings from the menu bar...

Page 130: ...for example a RAID 1 to a RAID 5 Unit Capacity Expansion for example adding a 4th drive to a 3 drive RAID 5 You can also use RLM to change the stripe size of a unit This section includes the following...

Page 131: ...rom a 3 drive RAID 0 to a 4 drive RAID 5 with both having the same stripe size As can be seen every piece of user data is moved from its original physical location Figure 57 RAID Level Migration Examp...

Page 132: ...al drives Once the unit is split any pending writes cannot be written to the second drive In addition the file system on the drive will not be clean Instead shut down the system replace one of the dri...

Page 133: ...on Expanding Unit Capacity You can expand a unit s capacity by adding one or more drives to it without changing the RAID level except for singles and RAID 1 units Since a single can only have one driv...

Page 134: ...iguration After you change the configuration of a unit you must inform the operating system of the change and you may need to re partition the unit Resizing the file system to match the new partition...

Page 135: ...own as a dynamic disk right click on it and choose Extend Volume to launch the Extend Volume wizard In Linux and FreeBSD the tools to expand the partition and file system can vary depending on the ker...

Page 136: ...u want to keep 3 Unmount the unit This step is very important If a unit is not unmounted and you delete it it is the equivalent of physically yanking a hard drive out from under the operating system Y...

Page 137: ...rough 3BM In 3BM the command for deleting a unit is on the main 3BM screen To delete a unit through 3BM 1 At the main 3BM screen select the unit in the list of Exportable Units by highlighting it and...

Page 138: ...re the drives for safekeeping purposes This process is sometimes referred to as array roaming Hot swap carriers are required unless you power down first When you remove a unit in contrast to deleting...

Page 139: ...Management and select Disk Management Remove the logical drive letter for the unit Under Linux and FreeBSD you can unmount the unit with this command umount mount location 3 In 3DM choose Management M...

Page 140: ...If your drives are not in hot swap carriers you do not need to remove a unit via 3BM Simply power down and remove the applicable drives Refer to your system s user guide for details on removing fixed...

Page 141: ...3ware 9000 series controller You will then have the advantages of the 9000 series controller The drive units must be in normal mode before moving the units Incomplete degraded rebuilding or initializ...

Page 142: ...0 board as described earlier in this section Moving Legacy JBOD Units to a 9000 Controller The steps for moving legacy JBOD units vary depending on the operating system you are using To move legacy JB...

Page 143: ...n 5 Press the F8 or Esc key to save and exit Moving Units from a 9500S to a 9550SX 9590SE or 9650SE Controller It is fairly straightforward to move units from a 9500S controller to a 9550SX 9590SE or...

Page 144: ...9650SE 5 Boot to the operating system 6 Install the driver for the 9550SX 9590SE or 9650SE If you are using Linux you may also need to update the initial RAM disk with the mkinitrd command 7 Reboot if...

Page 145: ...To remove a drive 1 In 3DM choose Management Maintenance Note When you add a drive to your system and connect it to the controller it is automatically detected and listed in 3DM If it does not immedi...

Page 146: ...part of a unit the unit may become degraded as shown in Figure 60 Figure 60 Result of Removing Drive from Unit in 3DM Rescanning the Controller When you make a change by physically adding or removing...

Page 147: ...DM 1 In 3DM choose Management Maintenance 2 On the Maintenance page click Rescan Controller 3DM scans the controller for information about units and drives and updates the information shown on the Mai...

Page 148: ...are described in this section which is organized into the following topics Checking Unit and Drive Status through 3DM Enclosure LED Status Indicators Unit Statuses Drive Statuses About Degraded Units...

Page 149: ...building while another part is degraded or initializing When this is the case you will see both statuses listed at the top unit level When you drill in to see details you will be able to see which the...

Page 150: ...ng table is an example only Check with your chassis documentation for more details Not all features may be supported or implemented in the same way Table 11 Meaning of LED Colors and Behavior Color Dr...

Page 151: ...unit is valid For more information see About Verification on page 151 Verify Paused The unit is set to verify however scheduling is enabled and the present time is not during a scheduled timeslot Veri...

Page 152: ...t drive failure RAID 5 RAID 6 and RAID 50 units achieve fault tolerance by using a simple exclusive OR function to generate the parity data that is distributed on all drives RAID 6 adds an extra level...

Page 153: ...n also be used to notify you of events The next few pages describe these capabilities Viewing Alarms Errors and Other Events on page 143 Using the Alert Utility Under Windows on page 144 Downloading a...

Page 154: ...page 144 A list of the possible error and other event messages is provided under Error and Notification Messages on page 224 To view alarms errors and other events in 3DM 1 Choose Monitor Alarms The...

Page 155: ...Windows System Tray 2 In the Windows Audible Visual Alarm window select the types of alerts you want to be notified of If you want to turn off the sound alarm and only have a pop up message appear che...

Page 156: ...og link 4 When the Save or Open dialog box appears navigate to where you want to save the log and click OK Viewing SMART Data About a Drive You can view SMART Self Monitoring Analysis and Reporting Te...

Page 157: ...erformance so using a schedule can minimize the impact This section includes the following topics related to background tasks About Initialization About Verification Starting a Verify Manually Rebuild...

Page 158: ...overwriting any existing data in the process In contrast background initialization uses an algorithm to resynch the parity information on the drives and does not rewrite existing data A foreground in...

Page 159: ...with 5 or more disks and RAID 50 units with 2 subunits of 5 or 6 disks will improve after the unit has been initialized For these configurations initialization begins automatically after you create t...

Page 160: ...er drive or set of drives Notes For RAID 5 with more 5 or more drives it is strongly recommended that you initialize the unit before using it Initializing such a unit is critical to insuring data inte...

Page 161: ...to replace drives before they fail You can manually request a verify or you can enable the Auto Verify policy and the controller will automatically start verification once every 24 hours See Starting...

Page 162: ...e the controller writes to that location the drive reallocates the data to a different sector Verification of Redundant Units Verification of redundant units also reads each sector working from lowest...

Page 163: ...otification of repair is posted to the alarms list The result is a restoration of drive and data integrity the primary and redundant data are again both valid If the unit is not redundant it is recomm...

Page 164: ...rt actively verifying until the next scheduled time A Stop Verify link appears next to the unit on the Maintenance page If you need to stop the verify process use this link If initialization starts be...

Page 165: ...the rebuild itself only happens when the operating system is running Rebuilds on multiple units can take place simultaneously If multiple drives are faulted in a RAID 10 configuration the drives are r...

Page 166: ...g box displays available drives select the drive you want to replace the failed drive and click OK Figure 65 Selecting a Drive when Rebuilding 5 If the degraded unit has more than one failed drive for...

Page 167: ...You can start the rebuild of a degraded unit manually in 3BM or you can let the system boot and allow the rebuild to take place automatically To start the rebuild of a unit through 3BM 1 Reboot the s...

Page 168: ...the operating system finishes loading If the rebuild fails and you have no available drives do one of the following If your system has hot swap carriers you can replace the failed or Not In Use drive...

Page 169: ...ave your changes and exit 3BM The unit will begin rebuilding about ten minutes after the operating system finishes loading and the 3ware driver has loaded Cancelling a Rebuild and Restarting It with a...

Page 170: ...e Rate and Verify Rate Initialization occurs at the Rebuild rate To change the background task rate 1 Choose Management Controller Settings from the menu bar 2 In the Background Task Rate section of t...

Page 171: ...unit will occur at the same time Scheduling Background Tasks You can set up scheduling windows for when background tasks occur so that routine maintenance of storage media occurs when it will be leas...

Page 172: ...it left off at the next scheduled time block Viewing Current Task Schedules You can see the current schedules for background tasks on the Scheduling page To view the current task schedule 1 Choose Man...

Page 173: ...build migrate or verify a unit right away without waiting for the next scheduled time In this case you can disable the schedule as described below You can easily disable a current Verify or Rebuild Mi...

Page 174: ...asks or Self test Tasks select it from the drop down list at the top of the page 3 Select the checkbox next to the schedule s you want to remove 4 Click the Remove Checked button The page refreshes an...

Page 175: ...been exceeded For more information about these self tests see the 3DM Reference section Scheduling page on page 201 Initially these tests are set to run every 24 hours You can change the schedule for...

Page 176: ...ividual drives by blinking their associated LED You can issue the command to blink the LED through 3DM For details about what the different LED patterns on the enclosure mean see Enclosure LED Status...

Page 177: ...e and no longer need to see the LED return to this page and uncheck the Identify box To blink the LEDs for all drives in a unit 1 Choose from the main menu in 3DM 2 On the list of units locate the uni...

Page 178: ...sions see the Controller Summary page on page 190 Using 3BM you can see the firmware version see Displaying Controller Information on page 65 Using the CLI you can see both the driver and firmware ver...

Page 179: ...download them individually from the website To download the driver or firmware 1 On the 3ware website www 3ware com navigate to Service and Support Software Downloads 2 Click Download Released Softwar...

Page 180: ...driver diskette extract the driver files to a floppy in step 9 above Refer to the user manual for your operating system for more details Updating the 3ware Driver and Firmware Under Windows You can up...

Page 181: ...the folder that contains the driver files you extracted in step 1 5 Double click on the file 3wUpdate exe You see a progress dialog box while the update utility checks your system for the driver and f...

Page 182: ...ure 72 Update Utility Showing Available Drivers If either your driver or firmware is current the screen reflects that Figure 73 Update Utility Showing Drivers Already Current The updater tells you whe...

Page 183: ...as available Figure 74 Update Utility Showing Only Driver Available If the driver is not found the 3ware Driver Firmware Update dialog box will show only the firmware as available Figure 75 Update Ut...

Page 184: ...your computer click Yes to do so If you were working from a driver diskette be sure to remove the floppy diskette before Windows reboots 8 If the driver you are installing is unsigned you will receive...

Page 185: ...box appears it lists the number of controllers in your system shows available updates for the device driver and available firmware updates for the first controller Figure 76 Show Update Utility Showi...

Page 186: ...s as described under Updating the Driver and Firmware on page 169 2 Backup any critical data prior to updating the 3ware driver 3 Log in to your system as system administrator 4 From the Start menu op...

Page 187: ...screen asks whether you want to connect to Windows Update Select No not this time and click Next Figure 78 Windows XP Service Pack 2 Windows Update Query 12 When you reach the Hardware Update Wizard...

Page 188: ...er Guide 13 At the next screen Figure 80 choose Don t search I will choose the driver to install and click Next Figure 80 Search for Driver Screen 14 When the Select the Device Driver screen appears F...

Page 189: ...Disk Dialog Box 16 When the Select the Device Driver dialog box reappears select the appropriate driver Figure 83 and click Next Be sure to select the correct driver name otherwise the driver upgrade...

Page 190: ...of the Hardware Update Wizard appears again informing you that Windows has found new hardware If you are using Windows XP Service Pack 2 the screen asks whether you want to connect to Windows Update F...

Page 191: ...ng the 3ware driver 3 Change the directory to the location with the extracted driver 4 Copy the files to the appropriate directory In the commands below replace version with the applicable Red Hat or...

Page 192: ...s add the following line to etc modules conf alias scsi_hostadapter 3w 9xxx o For 2 6 Kernels add the following line to etc modprobe conf alias scsi_hostadapter 3w 9xxx ko 6 Complete the upgrade by up...

Page 193: ...xx ko For SuSE high memory systems 2 6 kernels cp version 3w 9xxx big lib modules kernel kernel drivers scsi 3w 9xxx ko 5 Run sbin depmod a 6 Make sure the file lib modules kernel string modules dep c...

Page 194: ...wledgebase article 14546 http www 3ware com KB article aspx id 14546 Updating the 3ware Driver Under FreeBSD The following steps describe how to update the 3ware driver under FreeBSD To update the 3wa...

Page 195: ...the utility the versions currently installed on your controller and recommends whether to upgrade the firmware or not Warning We strongly recommend backing up your data before updating the firmware Up...

Page 196: ...tery backup unit attached to your controller in both 3DM 2 and 3BM To view information about a BBU in 3DM 2 On the menu bar choose Monitor Battery Backup The Battery Backup page appears on which you c...

Page 197: ...s While running the battery test and until charging is completed write cache is temporarily disabled For how to replace the battery see the installation guide that came with your controller To test th...

Page 198: ...ller User Guide Figure 87 BBU Information Screen While Battery is Testing To test the battery in a BBU in 3BM 1 At the main 3BM screen tab to the BBU button and press Enter 2 Tab to Test Battery Capac...

Page 199: ...organized by 3DM page as the pages are organized on the 3DM menu bar Controller Summary page Controller Details page Unit Information page Unit Details page Drive Information page Drive Details window...

Page 200: ...erial The serial number of the controller The serial number is also printed on a sticker on the outside bracket of the controller Firmware The firmware version running on the controller Driver The dri...

Page 201: ...are version running on the controller Driver The driver version being used to interact with the controller BIOS The BIOS version on the controller Boot Loader Boot Loader version on the controller Mem...

Page 202: ...lick the link in the Unit column Unit The unit number assigned to the unit by the firmware For 9000 series controllers unit numbers are in sequential order Name If a name has been given to this unit i...

Page 203: ...Details page appears when you click an ID number on the Unit Information page Because it is a sub page of Unit Information the page title in the menu bar continues to display Unit Information even wh...

Page 204: ...gle Disk Spare JBOD or Disk Volumes Displays the number of volumes in a unit This is usually 1 If you created a boot volume on this unit or if you have a unit on which you have enabled the auto carvin...

Page 205: ...the Port column Port The port to which the drive is connected Model The model of the drive Capacity The physical capacity of the drive Note that the capacity as shown on 3DM screen is calculated as 1K...

Page 206: ...dow appears when you click a Port on the Drive Information page This Drive Details window shows some Extra Drive Information including NCQ and SATA Link Speed support and the SMART data for the drive...

Page 207: ...ields show the fastest link speed that the disk drive supports and the current speed that the drive is running SMART Data SMART data is displayed as hex values Consult your disk drive manufacturer for...

Page 208: ...s each unit on the current controller and shows you whether the policies are currently enabled or disabled for each unit Write Cache When write cache is enabled data is stored locally in memory on the...

Page 209: ...ables Native Command Queuing NCQ for drives in the unit By default queuing is disabled You can enable it if desired NCQ only operates when the feature is enabled at both the drive and the controller I...

Page 210: ...y unit that is over a specified size known as the carve size will be broken down into multiple volumes of that size plus a remainder volume The default carve size is 2048 GB 2 TB For example using the...

Page 211: ...r when background tasks rebuild migrate initialize verify and self test should occur Background tasks can have impact on the performance of your system so you may prefer to schedule them at times when...

Page 212: ...one or more of the existing scheduled slots and then add new slots For step by step instructions for adding and removing schedules and setting schedules to be followed or ignored see Scheduling Backg...

Page 213: ...he system is booted Check SMART Thresholds This test checks to see whether SMART thresholds have been exceeded The SMART thresholds indicate when a drive is likely to fail based on the number of error...

Page 214: ...nagement Maintenance from the menu bar The Maintenance page lets you perform maintenance tasks on existing units on the current controller and lets you create new units by configuring available drives...

Page 215: ...ch unit created In Windows the device manager will reflect the changes under the disk drives icon This process is known as importing drives If new drives do not have any data indicating they were prev...

Page 216: ...rrently active or paused click on the unit number to display the Unit Details page which has that information For an explanation of the statuses see Unit Statuses on page 140 Drive Information Port Th...

Page 217: ...specify whether more processing time should be given to verifying or to I O For more information see About Verification on page 151 and Setting Background Task Rate on page 160 While a unit is verifyi...

Page 218: ...nal available drives In the dialog box are two drop down menus one for choosing the RAID level and one for choosing stripe size You can only migrate a unit to a RAID level that will be larger than the...

Page 219: ...ounted and the system is not accessing it For example make sure you are not copying files to the unit and make sure that there are no applications with open files on that unit If a unit is not unmount...

Page 220: ...ype Name Stripe size if applicable and unit policy settings A window like the one below shows the drives you selected and lets you specify configuration settings Caution Before you click Delete Unit m...

Page 221: ...e configuring a RAID 50 with twelve drives an additional field appears in which you select the number of drives per subunit 3 4 or 6 Figure 99 Configuring a RAID 50 with 12 Drives Name You can enter a...

Page 222: ...ilable Protection Balanced and Performance For more information see Setting the StorSave Profile for a Unit on page 117 Alarms page Figure 100 Alarms Page The Alarms page appears when you click Monito...

Page 223: ...of the event Three levels are provided Errors are shown next to a red box Warnings are shown next to a yellow box Information is shown next to a blue box Time The time shown for alarms is the time th...

Page 224: ...ered on This indicates the BBU is not able to backup the 3ware RAID controller When the BBU is in the charging state write caching is disabled automatically on all units attached to the controller Fau...

Page 225: ...e a higher number in this field As the battery ages the backup capacity diminishes Last Capacity Test Indicates the date when the last battery test was completed To test the battery click the Test Bat...

Page 226: ...e ID of the enclosure and the status of the fan either OK or Unknown Temp Sensor Summary Shows the ID of the enclosure and the temperature in the enclosure The maximum temperature for successful use o...

Page 227: ...ge refresh frequency whether remote access is permitted and the incoming port for 3DM to listen for requests The initial settings for most of these preferences are specified during installation of 3DM...

Page 228: ...dress Save E mail Settings button Saves the e mail notification settings Send Test Message button Sends a test message using the saved e mail settings Password Use the fields in this section to set th...

Page 229: ...ote computer HTTP Settings Listening Port This field specifies the HTTP port to be used by 3DM when listening for communications The default port setting is 888 If you change this port make sure the p...

Page 230: ...frequently asked questions software releases and compatibility information related to 3ware RAID controllers refer to 3ware support page at http www 3ware com support 3ware knowledgebase http www 3war...

Page 231: ...want to take a snapshot of these pages so that you can respond to questions about your system configuration to the Customer Support representative Command Logging All changes that are made to RAID co...

Page 232: ...you are using a PCI riser card try installing it directly Some riser cards are not compatible with certain motherboards Q2 Need to boot from a non 3ware device without changing the boot order If the...

Page 233: ...BIOS updates Q5 A unit shows up as unusable similar to the following 4 drive 64K RAID 5 missing drive s Port 1 Maxtor 6B300S0 279 86 GB Port 2 Maxtor 6B300S0 279 86 GB The unit has some but not all o...

Page 234: ...parentheses at the beginning of the message description For example in the string 0x04 0x002B 002B is the message number To find additional information about the message 2B you would look up 002B in t...

Page 235: ...0027 DCB checksum error detected 0028 DCB version unsupported 0029 Verify started 002A Verify failed 002B Verify completed 002C Source drive ECC error overwritten 002D Source drive error occurred 002...

Page 236: ...voltage is low 0046 Battery voltage is high 0047 Battery voltage is too low 0048 Battery voltage is too high 0049 Battery temperature is normal 004A Battery temperature is low 004B Battery temperatur...

Page 237: ...ild Removing a drive from a redundant unit will cause this message to display immediately For more information see Locating a Drive by Blinking Its LED on page 166 0003 Controller error occurred The 3...

Page 238: ...ternal algorithm to logically connect drives that belong to the same unit If after rollcall a member of a unit is not found the INCOMPLETE UNIT message is sent Examples of incomplete units are as foll...

Page 239: ...mittent disk drive power cable or interface cable For links to drive manufacturer diagnostic utilities and troubleshooting advice see http www 3ware com KB article aspx id 14924 000A Drive error detec...

Page 240: ...If this unit was a redundant unit and the initialize failed happened because of a problem on a particular disk drive then the unit will be degraded on that disk drive s port 000F SMART threshold exce...

Page 241: ...ve insertion caused a unit that was inoperable to become operational again Any data that was on that unit will still be there This message is sent only after the offline timer expires 20 seconds 0021...

Page 242: ...al attention should be paid to any drive in a unit that begins to indicate sector repair messages This may be an indication of a drive that is beginning to fail You may wish to replace the drive espec...

Page 243: ...led the Drive Configuration Block DCB As part of power on initialization the 3ware RAID controller performs a checksum of the DCB area to ensure consistency If this error occurs please contact 3ware t...

Page 244: ...automatically initialized The initialize will not erase user data but will recalculate and rewrite user parity data For more information on the Initialize function see About Initialization on page 14...

Page 245: ...e correctly jumpered to correspond to UDMA 100 or 133 drives For a list of compatible drives see http www 3ware com products compatibility_sata asp 0032 Spare capacity too small for some units This me...

Page 246: ...to the 3ware 9500S controller which has removable memory This message will be sent if there is no SODIMM memory connected to the controller In this case the controller is inoperable and cannot be use...

Page 247: ...s message is sent when migration is paused Migration follows the rebuild schedule For more information see Scheduling Background Tasks on page 161 003F Flash file system error detected The 3ware RAID...

Page 248: ...latent error that needs to be addressed before any future errors so this DCB is re written with the primary copy to rectify the errors found If this error occurs we recommend verifying the unit See St...

Page 249: ...ll be posted to the host 004A Battery temperature is low The Battery Backup Unit measures and evaluates the battery pack temperature on a continuous basis If the temperature is below a warning thresho...

Page 250: ...e charge cycle and may take up to 20 hours to complete During this test the Battery Backup Unit cannot backup the 3ware RAID controller all units have their write cache disabled until the test complet...

Page 251: ...is posted when the Battery Backup Unit completes a battery charge cycle 0057 Battery charging fault This message is posted when the charger of the Battery Backup Unit has detected a battery fault duri...

Page 252: ...to backup the 3ware RAID controller in case of a power failure This message is posted when the result of the health test is below the fault threshold This indicates that the battery pack must be repla...

Page 253: ...pendices The following information is available in the appendices Appendix A Glossary on page 244 Appendix B Compliance and Conformity Statements on page 250 Appendix C Warranty Technical Support and...

Page 254: ...city Serial ATA SATA RAID storage solutions A Chip AccerATA chip Automated data port to handle asynchronous ATA disk drive interface AMCC Applied Micro Circuits Corporation provides the essential buil...

Page 255: ...unit configuration data on the drives Note that in 3ware software tools arrays are referred to as units DCB Disk configuration block This is 3ware proprietary RAID table information that is written t...

Page 256: ...rive In 3BM BIOS exportable units and drives are those that will be available to the operating system when you boot your computer Fault tolerant A RAID unit which provides the ability to recover from...

Page 257: ...three tier units and as such will have sub sub units JBOD An unconfigured single drive The acronym is derived from just a bunch of disks Note that earlier versions of the 3ware RAID controller exporte...

Page 258: ...ge system s performance and provide fault tolerance protection against data loss Rebuild task schedule The specification for when rebuilding may occur including start time and duration Rebuild a unit...

Page 259: ...ts in an increase in performance Striping occurs in RAID 0 5 10 and 50 Subunit A logical unit of storage that is part of another unit For example the mirrored pairs RAID 1 in a RAID 10 unit are subuni...

Page 260: ...f not installed and used in accordance with the instructions may cause harmful interference to radio communications However there is no guarantee that interference will not occur in a particular insta...

Page 261: ...on Technical Regulations and or normative documents EN 55022 Limits and methods of measurements of radio interference character istics of information technology equipment EN 61000 4 2Electromagnetic c...

Page 262: ...r new or refurbished to be functionally equivalent to new Products or parts replaced under this provision shall become the property of AMCC Battery Backup Unit BBU Hardware 1 Year Hardware Warranty AM...

Page 263: ...ts which vary from state to state Some states do not allow the exclusion of incidental or consequential damages or allow limitation of implied warranties or their duration so that the above exclusions...

Page 264: ...s 3ware software library is accessible at http www 3ware com support download asp Web based software downloads feature upgrading multiple switches simultaneously For specific answers to questions or t...

Page 265: ...y is below warning level 0058 241 Battery capacity test completed 0050 240 Battery capacity test is overdue 0053 241 Battery capacity test started 004E 240 Battery charging completed 0056 241 Battery...

Page 266: ...t 193 194 Auto Rebuild policy 15 87 setting 90 200 Auto Verify policy for units 198 setting 3DM 114 auto carving 91 auto carving policy 87 auto carving policy setting 200 available drives 3DM 210 B ba...

Page 267: ...red 0003 227 Controller Information screen 65 Controller reset occurred 0001 227 Controller Settings page 3DM 197 Controller Summary page 3DM 190 conventions in the user guide ix creating a unit 3BM 1...

Page 268: ...ault tolerant definition 246 FCC Radio Frequency Interference Statement 250 firmware definition 246 showing version 190 191 updating 169 updating in Windows 170 Flash file system error detected 003F 2...

Page 269: ...ted 0035 235 Migration failed 0034 235 Migration paused 003E 237 Migration started 0033 235 mirrored disk array definition 4 247 RAID 1 5 motherboard boot sequence 29 Mozilla setting up 67 Multi LUN s...

Page 270: ...task duration 162 Scheduling page 3DM 201 Sector repair completed 0023 232 security certificate when starting 3DM 74 security message related to active content in the 3ware HTML Bookshelf x self test...

Page 271: ...mber definition 249 unit statuses 140 verifying a unit 153 in 3BM 154 in 3DM 154 volumes 194 write cache 3DM 198 write cache enabling and disabling 113 Unit deleted 000D 230 Unit Details page 3DM 193...

Page 272: ...volumes in a unit 194 multiple from one unit 91 resulting from auto carvings 194 W Warranty 252 WHQL Windows Hardware Quality Labs Microsoft 251 WinAVAlarm 144 Windows alert utility WinAVAlarm 144 wo...

Reviews:

No comments