Qsan Technology P300H61 Скачать руководство пользователя

Страница: 6 / 154

Chapter 1

Overview

1.1

Features

QSAN

subsystem can provide non-stop service with a high degree of fault tolerance by

using

QSAN

RAID technology and advanced array management features.

P300H61 / P300H71

subsystem connects to the host system by iSCSI interface. It can

be configured to numerous RAID level. The subsystem provides reliable data protection for
servers by using

RAID 6. The RAID 6 allows two HDD failures without any impact on the

existing data. Data can be recovered from the existing data and parity drives. (Data can
be recovered from the rest drives.)

Figure 1.1.1 (P300H61)

Figure 1.1.2 (P300H71)

Содержание P300H61

Страница 1: ...1 QSAN iSCSI subsystem P300H61 P300H71 GbE iSCSI to SATA II SAS RAID subsystem User Manual Version 7 79 MAR 2011 QSAN Technology Inc http www QsanTechnology com User Manual QUM201106 P300H61_P300H71 ...

Страница 2: ...al has been reviewed for accuracy but not for product warranty because of the various environments OS settings Information and specification will be changed without further notice For any update information please visit www QsanTechnology com and your contact windows Thank you for using QSAN Technology Inc products if you have any question please e mail to support qsan com tw We will answer your q...

Страница 3: ... 4 Rear view 22 2 4 Install battery backup module optional 24 2 5 Deployment 25 Chapter 3 Quick setup 29 3 1 Management interfaces 29 3 1 1 Serial console 29 3 1 2 Remote control 29 3 1 3 LCM 29 3 1 4 Web UI 32 3 2 How to use the system quickly 34 3 2 1 Quick installation 34 3 2 2 Volume creation wizard 37 Chapter 4 Configuration 40 4 1 Web UI management interface hierarchy 40 4 2 System configura...

Страница 4: ...7 Reboot and shutdown 88 4 7 Home Logout Mute 89 4 7 1 Home 89 4 7 2 Logout 89 4 7 3 Mute 89 Chapter 5 Advanced operations 90 5 1 Volume rebuild 90 5 2 RG migration and moving 92 5 3 VD extension 94 5 4 QSnap 95 5 4 1 Create snapshot volume 95 5 4 2 Auto snapshot 96 5 4 3 Rollback 97 5 4 4 QSnap constraint 98 5 5 Disk roaming 100 5 6 VD clone 101 5 7 SAS JBOD expansion 108 5 7 1 Connecting JBOD 10...

Страница 5: ...5 6 2 Event notifications 135 6 3 How to get support 142 Appendix 146 A Compatibility list 146 B Microsoft iSCSI initiator 147 ...

Страница 6: ...1 subsystem connects to the host system by iSCSI interface It can be configured to numerous RAID level The subsystem provides reliable data protection for servers by using RAID 6 The RAID 6 allows two HDD failures without any impact on the existing data Data can be recovered from the existing data and parity drives Data can be recovered from the rest drives Figure 1 1 1 P300H61 Figure 1 1 2 P300H7...

Страница 7: ...onsume any host CPU time thus the server is dedicated to the specific or other application The snapshot copies can be taken manually or by schedule every hour or every day depends on the modification QSAN subsystem is the most cost effective disk array system with completely integrated high performance and data protection capabilities which meet or exceed the highest industry standards and the bes...

Страница 8: ...o combine multiple drives together to form one large logical drive This RAID drive obtains performance capacity and reliability than a single drive The operating system detects the RAID drive as a single storage device 1 2 1 Terminology The document uses the following terms Part 1 Common RAID Redundant Array of Independent Disks There are different RAID levels with different degree of data protect...

Страница 9: ... non volatile media Each data is synchronized in both data cache and accessed physical disks WB Write Back cache write policy A caching technique in which the completion of a write request is signaled as soon as the data is in cache and actual writing to non volatile media occurs at a later time It speeds up system write performance but needs to bear the risk where data may be inconsistent between...

Страница 10: ...anism to control access to an iSCSI storage system over the iSCSI data ports iSNS Internet Storage Name Service Part 3 Dual controller SBB Storage Bridge Bay The objective of the Storage Bridge Bay Working Group SBB is to create a specification that defines mechanical electrical and low level enclosure management requirements for an enclosure controller slot that will support a variety of storage ...

Страница 11: ...rsed parity over the member disks RAID 3 needs at least three hard drives RAID 6 2 dimensional parity protection over the member disks RAID 6 needs at least four hard drives RAID 0 1 Mirroring of the member RAID 0 volumes RAID 0 1 needs at least four hard drives RAID 10 Striping over the member RAID 1 volumes RAID 10 needs at least four hard drives RAID 30 Striping over the member RAID 3 volumes R...

Страница 12: ... or the Internet IP SANs are true SANs Storage Area Networks which allow several servers to attach to an infinite number of storage volumes by using iSCSI over TCP IP networks IP SANs can scale the storage capacity with any type and brand of storage system In addition it can be used by any type of network Ethernet Fast Ethernet Gigabit Ethernet and 10 Gigabit Ethernet and combination of operating ...

Страница 13: ...tion list Below are the available links 1 Link to download the Microsoft iSCSI software initiator http www microsoft com downloads details aspx FamilyID 12cb3c1a 15d6 4585 b385 befd1319f825 DisplayLang en 2 In current Linux distributions OS built in iSCSI initiators are usually available For different kernels there are different iSCSI drivers Please check Appendix A for iSCSI initiator certificati...

Страница 14: ... from both controllers 5 Management port seamless take over The management port can be transferred smoothly to the other controller with the same IP address 6 Online firmware upgrade no system down time 7 Backward compatible to P210V61 volume configurations 8 Multiple target iSCSI nodes per controller support Each LUN can be attached to one of 32 nodes from each controller 9 Front end 4 x GbE iSCS...

Страница 15: ... Instant RAID volume availability 10 Auto volume rebuilding 11 On line volume migration with no system down time Advanced data protection 1 QSnap writeable snapshot Built in snapshot with rollback enabled Snapshot enabled up to 16 volumes each logical volume supports up to 32 snapshot volumes total 512 snapshot volumes per system 2 Microsoft Windows Volume Shadow Copy Services VSS 3 Configurable N...

Страница 16: ...g and failover through MPIO MC S Trunking and LACP 5 Up to 32 multiple nodes support Host connection 1 4 x iSCSI GbE ports per controller 2 Host access control Read Write and Read Only 3 Up to 128 sessions per controller 4 One logic volume can be shared by as many as 16 hosts OS support Windows Linux Solaris Mac Drive support 1 SAS 2 SATA II optional 3 SCSI 3 compliant 4 Multiple IO transaction pr...

Страница 17: ...s been shown to be in compliance with and was tested in accordance with the measurement procedures specified in the Standards and Specifications listed below and as indicated in the measurement report number xxxxxxxx E Technical Standard EMC DIRECTIVE 2004 108 EC EN55022 EN55024 UL statement Rack Mount Instructions The following or similar rack mount instructions are included with the installation...

Страница 18: ... the handles is for rack mount use only Do not use the handles to carry or transport the systems The ITE is not intended to be installed and used in a home school or public area accessible to the general population and the thumbscrews should be tightened with a tool after both initial installation and subsequent access to the panel Warning Remove all power supply cords before service This equipmen...

Страница 19: ...2 2 Before installation Before starting prepare the following items 1 A host with a Gigabit Ethernet NIC or iSCSI HBA 2 CAT 5e or CAT 6 network cables for management port and iSCSI data ports 3 Prepare storage system configuration plan 4 Prepare management port and iSCSI data ports network information When using static IP please prepare static IP addresses subnet mask and default gateway 5 Gigabit...

Страница 20: ...16 Slot 22 Slot 5 Slot 11 Slot 17 Slot 23 Slot 6 Slot 12 Slot 18 Slot 24 The drives can be installed into any slot in the enclosure Slot numbering will be reflected in web UI Tips It s better to install at least one drive in slot 1 4 System event logs are saved in these drives Otherwise event logs will be gone after reboot 2 3 2 Control panel There are five buttons to control QSAN LCM LCD Control ...

Страница 21: ...n ESC button 2 3 3 Install drives Remove a drive tray Then install a HDD To install SAS drives align the edge of the SAS drive to the back end of tray the backplane can directly connect to SAS drives To install SATA II drives with QSATA boards align the QSATA board edge to the back end of tray the backplane can connect SATA II drives through QSATA boards Figure 2 3 3 1 SAS drives Figure 2 3 3 2 SA...

Страница 22: ...y description HDD fault LED Red HDD is failure Off HDD is good HDD activity LED Blue HDD is active Violet blinking HDD is accessing Off No HDD Latch for tray kit removal HDD tray handhold 2 3 4 Rear view Figure 2 3 4 1 P300H61 ...

Страница 23: ...23 Figure 2 3 4 2 P300H71 PSU and Fan module description Power supply unit PSU3 Fan module FAN2 Power supply unit PSU2 Power supply unit PSU1 Fan module FAN1 Controller 1 Controller 2 Figure 2 3 4 3 ...

Страница 24: ...Off No data on the cache BBM LED Green BBM installed and powered Off No BBM BBM Status Button When the system power is off press the BBM status button if the BBM LED is Green then the BBM still has power to keep data on the cache If not then the BBM power is ran out and cannot keep the data on the cache anymore Management port Console port RS 232 port for UPS SAS JBOD expansion port 2 4 Install ba...

Страница 25: ...ot pluggable Regardless of the subsystem is turned on or off 2 Remove the cover of BBM 3 Insert the BBM 4 Tighten the BBM and use screws to lock the both sides 5 Done 2 5 Deployment Please refer to the following topology and have all the connections ready ...

Страница 26: ...ower on P300H61 P300H71 and J300H61 J300H71 optional first and then power on hosts and iSNS server 4 Host server is suggested to logon the target twice both controller 1 and controller 2 and then MPIO should be setup automatically Tips iSNS server is recommended for dual controller system For better data service availability all the connections among host servers GbE switches and the dual controll...

Страница 27: ...27 Figure 2 5 2 The following topology is the connections for console and UPS optional Figure 2 5 3 ...

Страница 28: ...irectly 2 Using RS 232 cable for UPS gray color phone jack to DB9 male to connect from controller to APC Smart UPS serial cable DB9 female side and then connect the serial cable to APC Smart UPS Caution It may not work when connecting the RS 232 cable for UPS gray color phone jack to DB9 male to APC Smart UPS directly ...

Страница 29: ... control Terminal type vt100 Login name admin Default password 1234 3 1 2 Remote control SSH secure shell software is required for remote login The SSH client software is available at the following web site SSH Tectia Client http www ssh com PuTTY http www chiark greenend org uk Host name 192 168 10 50 Please check the DHCP address first on LCM Login name admin Default password 1234 Tips QSAN prod...

Страница 30: ...n error occurs Reset Shutdown Reset or shutdown controller Quick Install Quick steps to create a volume Please refer to next chapter for detailed operation steps in web UI Volume Wizard Smart steps to create a volume Please refer to next chapter for detailed operation steps in web UI View IP Setting Display current IP address subnet mask and gateway Change IP config Set IP address subnet mask and ...

Страница 31: ...1 Use default algorithm Volume Size xxx GB Apply The Config Yes No Volume Wizard JBOD x RAID 0 RAID 1 RAID 3 RAID 5 RAID 6 RAID 0 1 new x disk xxx BG Adjust Volume Size Apply The Config Yes No IP Config Static IP IP Address 192 168 010 050 IP Subnet Mask 255 255 255 0 View IP Setting IP Gateway 192 168 010 254 DHCP Yes No IP Address Adjust IP address IP Subnet Mask Adjust Submask IP IP Gateway Adj...

Страница 32: ...GUI to operate Be sure to connect the LAN cable The default IP setting is DHCP open the browser and enter http 192 168 10 50 Please check the DHCP address first on LCM And then it will pop up a dialog for authentication Figure 3 1 4 1 User name admin Default password 1234 After login choose the functions which lists on the left side of window to make any configuration ...

Страница 33: ...reen Temperature is normal Red Temperature is abnormal Voltage light Green voltage is normal Red voltage is abnormal UPS light Green UPS works well Red UPS fails Fan light Green Fan works well Red Fan fails Power light Green Power works well Red Power fails Dual controller light Green Both controller 1 and controller 2 are present and well Orange The system is degraded and there is only 1 ...

Страница 34: ...default value is enabled but some applications will disable it 3 2 How to use the system quickly The following methods will describe the quick guide to use this subsystem 3 2 1 Quick installation Please make sure that there are some free drives installed in this system SAS drivers are recommended Please check the hard drive details in Volume configuration Physical disk Figure 3 2 1 1 Step1 Click t...

Страница 35: ...35 Figure 3 2 1 2 Step2 Confirm the management port IP address and DNS and then click Next Figure 3 2 1 3 Step 3 Set up the data port IP and click Next ...

Страница 36: ...36 Figure 3 2 1 4 Step 4 Set up the RAID level and volume size and click Next Figure 3 2 1 5 Step 5 Check all items and click Finish ...

Страница 37: ... find that some HDDs are available free status The result is using smarter policy designed by QSAN It gives user 1 Biggest capacity of RAID level for user to choose and 2 The fewest disk number for RAID level volume size E g user chooses RAID 5 and the controller has 12 200G 4 80G HDDs inserted If we use all 16 HDDs for a RAID 5 and then the maximum size of volume is 1200G 80G 15 By the wizard we ...

Страница 38: ...38 Figure 3 2 2 1 Step 2 Please select the combination of the RG capacity or Use default algorithm for maximum RG capacity After RG size is chosen click Next Figure 3 2 2 2 ...

Страница 39: ...default number Then click Next Figure 3 2 2 3 Step 4 Confirmation page Click Finish if all setups are correct Then a VD will be created Step 5 Done The system is available now Figure 3 2 2 4 Figure 3 2 2 4 A virtual disk of RAID 0 is created and is named by system itself ...

Страница 40: ...nt Create Modify user information Delete Volume configuration Physical disk Set Free disk Set Global spare Set Dedicated spare Upgrade Disk Scrub Turn on off the indication LED More information RAID group Create Migrate Move Activate Deactivate Parity check Delete Set preferred owner Set disk property More information Virtual disk Create Extend Parity check Delete Set property Attach LUN Detach LU...

Страница 41: ...port Export Import file Reboot and shutdown Reboot Shutdown Quick installation Step 1 Step 2 Step 3 Step 4 Confirm Volume creation wizard Step 1 Step 2 Step 3 Confirm 4 2 System configuration System configuration is designed for setting up the System setting Network setting Login setting Mail setting and Notification setting Figure 4 2 1 4 2 1 System setting System setting can setup system name an...

Страница 42: ...turn on the system indication LED Click again to turn off 4 2 2 Network setting Network setting is for changing IP address for remote administration usage There are 3 options DHCP Get IP address from DHCP server BOOTP Get IP address from BOOTP server and static IP The default setting is DHCP User can change the HTTP HTTPS and SSH port number when the default port number is not allowed on host serv...

Страница 43: ...nt multiple users access the same system in the same time 1 Auto logout The options are 1 Disabled 2 5 minutes 3 30 minutes 4 1 hour The system will log out automatically when user is inactive for a period of time 2 Login lock Disabled or Enabled When the login lock is enabled the system allows only one user to login or modify system settings ...

Страница 44: ...ation Some mail servers would check Mail from address and need authentication for anti spam Please fill the necessary fields and click Send test mail to test whether email functions are available User can also select which levels of event logs are needed to be sent via Mail Default setting only enables ERROR and WARNING event logs Please also make sure the DNS server IP is well setup so the event ...

Страница 45: ...1 4 2 5 Notification setting Notification setting can set up SNMP trap for alerting via SNMP pop up message via Windows messenger not MSN alert via syslog protocol and event log filter for web UI and LCM notifications ...

Страница 46: ...NING event log in SNMP There are many SNMP tools The following web sites are for your reference SNMPc http www snmpc com Net SNMP http net snmp sourceforge net If necessary click Download to get MIB file and import to SNMP To use Messenger user must enable the service Messenger in Windows Start Control Panel Administrative Tools Services Messenger and then event logs can ...

Страница 47: ... kiwisyslog com Most UNIX systems build in syslog daemon Event log filter setting can enable event log display on Pop up events and LCM 4 3 iSCSI configuration iSCSI configuration is designed for setting up the Entity Property NIC Node Session and CHAP account Figure 4 3 1 4 3 1 NIC NIC can change IP addresses of iSCSI data ports P300H61 P300H71 has four gigabit ports on each controller to transmi...

Страница 48: ...P Figure 4 3 1 2 Default gateway Default gateway can be changed by checking the gray button of LAN port click Become default gateway There can be only one default gateway MTU Jumbo frame MTU Maximum Transmission Unit size can be enabled by checking the gray button of LAN port click Enable jumbo frame Maximum jumbo frame size is 3900 bytes Caution The MTU size of switching hub and HBA on host must ...

Страница 49: ...nd the limits of any single port 3 LACP The Link Aggregation Control Protocol LACP is part of IEEE specification 802 3ad that allows bundling several physical ports together to form a single logical channel LACP allows a network switch to negotiate an automatic bundle by sending LACP packets to the peer The advantages of LACP are 1 increases the bandwidth 2 failover when link status fails on a por...

Страница 50: ...gure 4 3 1 5 shows a user can ping host from the target to make sure the data port connection is well 4 3 2 Entity property Entity property can view the entity name of the system and setup iSNS IP for iSNS Internet Storage Name Service iSNS protocol allows automated discovery management and configuration of iSCSI devices on a TCP IP network Using iSNS it needs to install an iSNS server in SAN Add ...

Страница 51: ...4 3 3 1 CHAP CHAP is the abbreviation of Challenge Handshake Authentication Protocol CHAP is a strong authentication method used in point to point for user login It s a type of authentication in which the authentication server sends the client a key to be used for encrypting the username and password CHAP enables the username and password to transmit in an encrypted form for protection ...

Страница 52: ...lick Authenticate 3 Select CHAP Figure 4 3 3 2 4 Click OK Figure 4 3 3 3 5 Go to iSCSI configuration CHAP account page to create CHAP account Please refer to next section for more detail 6 Check the gray button of OP column click User 7 Select CHAP user s which will be used It s a multi option it can be one or more If choosing none CHAP can not work ...

Страница 53: ...tal Users can change the portals belonging to the device node of each controller 1 Check the gray button of OP column next to one device node 2 Select Change portal 3 Choose the portals for the controller 4 Click OK to confirm Figure 4 3 3 5 Rename alias User can create an alias to one device node ...

Страница 54: ... user cannot login 4 3 4 Session Session can display current iSCSI session and connection information including the following items 1 TSIH target session identifying handle 2 Host Initiator Name 3 Controller Target Name 4 InitialR2T Initial Ready to Transfer 5 Immed data Immediate data 6 MaxDataOutR2T Maximum Data Outstanding Ready to Transfer 7 MaxDataBurstLen Maximum Data Burst Length 8 DataSegi...

Страница 55: ...e 4 3 4 2 iSCSI Connection 4 3 5 CHAP account CHAP account can manage a CHAP account for authentication P300H61 P300H71 can create multiple CHAP accounts To setup CHAP account please follow the procedures 1 Click Create 2 Enter User Secret and Confirm secret again Node can be selected here or later If selecting none it can be enabled later in iSCSI configuration Node User ...

Страница 56: ... 3 5 2 4 Click Delete to delete CHAP account 4 4 Volume configuration Volume configuration is designed for setting up the volume configuration which includes Physical disk RAID group Virtual disk Snapshot Logical unit and QReplica optional ...

Страница 57: ...e function can be selected and inactive functions show up in gray color and cannot be selected For example set PD slot number 4 to dedicated spare disk Step 1 Check to the gray button of PD 4 select Set Dedicated spare it will link to next page Figure 4 4 1 1 Step 2 If there is any RG which is in protected RAID level and can be set with dedicate spare disk select one RG and then click Submit Figur...

Страница 58: ...s are free disks Step 4 The unit of size can be changed from GB to MB It will display the capacity of hard drive in MB Figure 4 4 1 4 PD column description Slot The position of a hard drive The button next to the number of slot shows the functions which can be executed Size GB MB Capacity of hard drive The unit can be displayed in GB or MB RG Name RAID group name ...

Страница 59: ...Usage The usage of hard drive RAID disk This hard drive has been set to a RAID group Free disk This hard drive is free for use Dedicated spare This hard drive has been set as dedicated spare of a RG Global spare This hard drive has been set as global spare of all RGs Vendor Hard drive vendor Serial Hard drive serial number Type Hard drive type SATA SATA disk SATA2 SATA II disk SAS SAS disk Write c...

Страница 60: ...d drive detail information 4 4 2 RAID group RAID group can view the status of each RAID group create and modify RAID groups The following is an example to create a RG Step 1 Click Create enter Name choose RAID level click Select PD to select PD assign the RG s Preferred owner Then click OK The Write Cache option is to enable or disable the write cache option of hard drives The Standby option is to...

Страница 61: ...ure 4 4 2 2 Figure 4 4 2 2 There is a RAID 0 with 4 physical disks named RG R0 The second RAID group is a RAID 5 with 3 physical disks named RG R5 Step 3 Done View RAID group page RG column description The button includes the functions which can be executed Name RAID group name ...

Страница 62: ...g the RAID group is being scrubbed Health The health of RAID group Good the RAID group is good Failed the RAID group fails Degraded the RAID group is not healthy and not completed The reason could be lack of disk s or have failed disk RAID The RAID level of the RAID group Current owner The owner of the RAID group The default owner is controller 1 Preferred owner The preferred owner of the RAID gro...

Страница 63: ...ge the disk property of write cache and standby options Write cache Enabled Enable disk write cache Default Disabled Disable disk write cache Standby Disabled Disable auto spindown Default 30 sec 1 min 5 min 30 min Enable hard drive auto spindown to save power when no access after certain period of time Read ahead Enabled Enable disk read ahead Default Disabled Disable disk read ahead Command queu...

Страница 64: ...ty and change Readahead option if necessary Erase option will wipe out old data in VD to prevent that OS recognizes the old partition There are three options in Erase None default erase first 1GB or full disk Last select Type mode for normal or clone usage Then click OK Figure 4 4 3 1 Caution If shutdown or reboot the system when creating VD the erase process will stop Step 2 Confirm page Click OK...

Страница 65: ... Size GB MB Total capacity of the virtual disk The unit can be displayed in GB or MB Write The right of virtual disk WT Write Through WB Write Back RO Read Only Priority The priority of virtual disk HI HIgh priority MD MiDdle priority LO LOw priority Bg rate Background task priority 4 3 2 1 0 Default value is 4 The higher number the background priority of a VD is the more background I O will be sc...

Страница 66: ...Optimal the virtual disk is working well and there is no failed disk in the RG Degraded At least one disk from the RG of the Virtual disk is failed or plugged out Failed the RAID group disk of the VD has single or multiple failed disks than its RAID level can recover from data loss Partially optimal the virtual disk has experienced recoverable read errors After passing parity check the health will...

Страница 67: ...mismatches count to 1 10 20 100 Delete Delete the virtual disk Set property Change the VD name right priority bg rate and read ahead Right WT Write Through WB Write Back Default RO Read Only Priority HI HIgh priority Default MD MiDdle priority LO LOw priority Bg rate 4 3 2 1 0 Default value is 4 The higher number the background priority of a VD is the more background I O will be scheduled to execu...

Страница 68: ... schedule Set snapshot space Set snapshot space for taking snapshot Please refer to next chapter for more detail Cleanup snapshot Clean all snapshots of a VD and release the snapshot space Take snapshot Take a snapshot on the virtual disk Auto snapshot Set auto snapshot on the virtual disk List snapshot List all snapshots of the virtual disk More information Show virtual disk detail information 4 ...

Страница 69: ...napshot space is 15GB and used 1GB for saving snapshot index Step 3 Take a snapshot In Volume configuration Snapshot click Take snapshot It will link to next page Enter a snapshot name Figure 4 4 4 3 Step 4 Expose the snapshot VD Check to the gray button next to the Snapshot VD number click Expose Enter a capacity for snapshot VD If size is zero the exposed snapshot VD will be read only Otherwise ...

Страница 70: ...er to the next section for attaching a LUN Step 6 Done Snapshot VD can be used Snapshot column description The button includes the functions which can be executed Name Snapshot VD name Used GB MB The amount of snapshot space that has been used The unit can be displayed in GB or MB Status The status of snapshot N A The snapshot is normal Replicated The snapshot is for clone or QReplica usage Abort ...

Страница 71: ...peration description Expose Unexpose Expose unexpose the snapshot VD Rollback Rollback the snapshot VD Delete Delete the snapshot VD Attach Attach a LUN Detach Detach a LUN List LUN List attached LUN s 4 4 5 Logical unit Logical unit can view create and modify the status of attached logical unit number s of each VD User can attach LUN by clicking the Attach Host must enter with an iSCSI node name ...

Страница 72: ...s of access control are followed from the LUN created time the earlier created LUN is prior to the matching rules For example there are 2 LUN rules for the same VD one is LUN 0 and the other is iqn host1 LUN 1 The host iqn host2 can login successfully because it matches the rule 1 Wildcard and are allowed in this field can replace any word can replace only one character For example iqn host iqn ho...

Страница 73: ...fter system boots up automatically Then set a global spare disk Last delete all of them Step 1 Create a RG RAID group To create a RAID group please follow the procedures Figure 4 4 6 1 1 Select Volume configuration RAID group 2 Click Create 3 Input a RG Name choose a RAID level from the list click Select PD to choose the RAID physical disks then click OK 4 Check the setting Click OK if all setups ...

Страница 74: ...olume please follow the procedures Figure 4 4 6 3 1 Select Volume configuration Virtual disk 2 Click Create 3 Input a VD name choose a RG Name and enter a size for this VD decide the stripe height block size read write mode bg rate and set priority finally click OK 4 Done A VD has been created 5 Follow the above steps to create another VD ...

Страница 75: ...figuration Virtual disk check the gray button next to the VD number click Attach LUN 2 In Volume configuration Logical unit click Attach The procedures are as follows Figure 4 4 6 5 1 Select a VD 2 Input Host IQN which is an iSCSI node name for access control or fill in wildcard which means every host can access to this volume Choose LUN and permission and then click OK 3 Done Figure 4 4 6 6 Figur...

Страница 76: ... 2 Check the gray button next to the PD slot click Set global space 3 Global spare status is shown in Usage column Figure 4 4 6 7 Figure 4 4 6 7 Slot 4 is set as a global spare disk Step 5 Done Delete VDs RG please follow the below steps Step 6 Detach a LUN from the VD In Volume configuration Logical unit Figure 4 4 6 8 1 Check the gray button next to the LUN click Detach There will pop up a confi...

Страница 77: ... been deleted Tips The action of deleting one RG will succeed only when all of the related VD s are deleted in this RG Otherwise user cannot delete this RG Step 9 Free a global spare disk To free a global spare disk please follow the procedures 1 Select Volume configuration Physical disk 2 Check the gray button next to the PD slot click Set Free disk Step 10 Done all volumes have been deleted 4 5 ...

Страница 78: ...0 seconds When there are 3 errors consecutively system sends ERROR event log 5 Power sensors 10 seconds when there are 3 errors consecutively system sends ERROR event log 6 LED status 10 seconds Figure 4 5 1 4 5 1 Hardware monitor Hardware monitor can view the information of current voltages and temperatures ...

Страница 79: ...79 ...

Страница 80: ...hen one of these sensor is over high temperature threshold for continuous 3 minutes auto shutdown will be triggered immediately 2 The core processor temperature limit is 80 The iSCSI NIC temperature limit is 65 The SAS expandor and SAS controller temperature limit is 65 3 If the high temperature situation doesn t last for 3 minutes system will not trigger auto shutdown 4 5 2 UPS UPS can set up UPS...

Страница 81: ...attery level When below the setting level the system will shutdown Setting level to 0 will disable UPS Shutdown delay s If power failure occurs and system power can not recover after the time setting the system will shutdown Setting delay to 0 will disable the function Shutdown UPS Select ON when power is gone UPS will shutdown by itself after the system shutdown successfully After power comes bac...

Страница 82: ...ure 4 5 1 1 Enable SES in LUN 0 and can be accessed from every host The SES client software is available at the following web site SANtools http www santools com 4 5 4 Hard drive S M A R T S M A R T Self Monitoring Analysis and Reporting Technology is a diagnostic tool for hard drives to deliver warning of drive failures in advance S M A R T provides users chances to take actions before possible d...

Страница 83: ...t value the number in parenthesis is the threshold value The threshold values from different hard drive vendors are different please refer to hard drive vendors specification for details S M A R T only supports SATA drives SAS drives do not have this function now It will show N A in the web page for SAS drives Figure 4 5 4 1 SAS drives Figure 4 5 4 2 SATA drives 4 6 System maintenance ...

Страница 84: ...import and export all controller configuration to a file and Reboot and shutdown to reboot or shutdown the system Figure 4 6 1 4 6 1 System information System information can display system information including CPU type installed system memory firmware version serial numbers of dual controllers backplane ID and system status Figure 4 6 1 1 Status description Normal Dual controllers are in normal ...

Страница 85: ...txt Click Clear button to clear all event logs Click Mute button to stop alarm if system alerts Figure 4 6 2 1 The event log is displayed in reverse order which means the latest event log is on the first top page The event logs are actually saved in the first four hard drives each hard drive has one copy of event log For one system there are four copies of event logs to make sure users can check e...

Страница 86: ...xt system boot up Otherwise the event logs cannot be saved and would be disappeared 4 6 3 Upgrade Upgrade can upgrade controller firmware JBOD firmware change operation mode and activate QReplica license Before upgrade it s better to use Export function to backup all configurations to a file Figure 4 6 3 1 ...

Страница 87: ...only switch this mode to Single This mode indicates singel upgradable Enter the MAC address displayed in System configuration Network setting such as 001378xxxxxx case insensitive and then click Confirm At last it can activate QReplica function if there is a license here Select the license file and then click Confirm 4 6 4 Firmware synchronization Firmware synchronization can synchronize the firmw...

Страница 88: ...available in export and not available in import which can avoid confliction date deleting between two subsystems which mean if one system already has valuable volumes in the disks and user may forget and overwrite it Use import could return to original configuration If the volume setting was also imported user s current volumes will be overwritten with different configuration Figure 4 6 6 1 1 Impo...

Страница 89: ...a protection Figure 4 6 7 1 4 7 Home Logout Mute In the right upper corner of web UI there are 3 individual icons Home Logout and Mute Figure 4 7 1 4 7 1 Home Click Home to return to home page 4 7 2 Logout For security reason please use Logout to exit the web UI To re login the system please enter username and password again 4 7 3 Mute Click Mute to stop the alarm when error occurs ...

Страница 90: ...y existing RG it would be FR Free disk and the system will start Auto Rebuild If user only removes the failed disk and plugs the same failed disk in the same slot again the auto rebuild will start running But rebuilding in the same failed disk may impact customer data if the status of disk is unstable QSAN suggests all customers not to rebuild in the failed disk for better data protection 2 When t...

Страница 91: ... 2 dimensional parity protection over the member disks RAID 6 allows two hard drives failure or unplugging If it needs to rebuild two hard drives at the same time it will rebuild the first one then the other in sequence RAID 0 1 Mirroring of RAID 0 volumes RAID 0 1 allows two hard drive failures or unplugging but at the same array RAID 10 Striping over the member of RAID 1 volumes RAID 10 allows t...

Страница 92: ...sical disks Take examples as following Figure 5 2 1 The below operations are not allowed when a RG is being migrated or moved System would reject these operations 1 Add dedicated spare 2 Remove a dedicated spare 3 Create a new VD 4 Delete a VD 5 Extend a VD 6 Scrub a VD 7 Perform another migration operation 8 Scrub entire RG 9 Take a snapshot 10 Delete a snapshot 11 Expose a snapshot 12 Rollback t...

Страница 93: ...e original RAID level is RAID 6 and user wants to migrate to RAID 0 system will evaluate whether this operation is safe or not and appear a warning message of Sure to migrate to a lower protection array Figure 5 2 2 4 Double check the setting of RAID level and RAID PD slot If there is no problem click OK 5 Finally a confirmation page shows the detail of RAID information If there is no problem clic...

Страница 94: ... Change the size The size must be larger than the original and then click OK to start extension Figure 5 3 1 4 Extension starts If VD needs initialization it will display an Initiating in Status and complete percentage of initialization in R Figure 5 3 2 Tips The size of VD extension must be larger than original Caution VD Extension cannot be executed during rebuilding or migration ...

Страница 95: ...an errors and so on Snap VD is allocated within the same RG in which the snapshot is taken we suggest to reserve 20 of RG size or more for snapshot space Please refer to the following figure for snapshot concept Figure 5 4 1 5 4 1 Create snapshot volume To take a snapshot of the data please follow the procedures 1 Select Volume configuration Virtual disk 2 Check the gray button next to the VD numb...

Страница 96: ...ead written and the size is the maximum capacity for writing 8 Attach a LUN to the snapshot VD Please refer to the previous chapter for attaching a LUN 9 Done It can be used as a disk Figure 5 4 1 2 Figure 5 4 1 2 This is the snapshot list of VD 01 There are two snapshots Snapshot VD SnapVD 01 is exposed as read only SnapVD 02 is exposed as read write 1 There are two methods to clean all snapshots...

Страница 97: ...to snapshot 2 The auto snapshot can be set monthly weekly daily or hourly 3 Done It will take snapshots automatically Figure 5 4 2 1 Figure 5 4 2 1 It will take snapshots every month and keep the last 32 snapshot copies Tips Daily snapshot will be taken at every 00 00 Weekly snapshot will be taken every Sunday 00 00 Monthly snapshot will be taken every first day of month 00 00 5 4 3 Rollback ...

Страница 98: ...h this technique snapshot only copies the changed data instead of copying whole data It will save a lot of disk space Create a data consistent snapshot Before using snapshot user has to know why sometimes the data corrupts after rollback of snapshot Please refer to the following diagram When user modifies the data from host the data will pass through file system and memory of the host write cachin...

Страница 99: ...lume shadow copy service to prevent this issue VSS is a mechanism for creating consistent point in time copies of data known as shadow copies It is a coordinator between backup software application SQL or Exchange and storages to make sure the snapshot without the problem of data inconsistent For more detail about the VSS please refer to http technet microsoft com en us library cc785914 aspx QSAN ...

Страница 100: ...nt situations 1 If the snapshot is configured as auto snapshot the latest one the 33rd snapshot will replace the oldest one the first snapshot and so on 2 If the snapshot is taken manually when taking the 33rd snapshot will fail and a warning message will be showed on Web UI Rollback Delete snapshot When a snapshot has been rollbacked the other snapshots which are earlier than it will also be remo...

Страница 101: ...hedule and deploy the clone rules The procedures of VD clone are on the following 1 Copy all data from source VD to target VD at the beginning full copy 2 Use QSnap technology to perform the incremental copy afterwards Please be fully aware that the incremental copy needs to use snapshot to compare the data difference Therefore the enough snapshot space for VD clone is very important The following...

Страница 102: ...re the objects a Source VD and a Target VD Before starting clone process it needs to deploy the VD Clone rule first Click Configuration Figure 5 6 3 4 There are three clone configurations describe on the following Figure 5 6 4 ...

Страница 103: ... threshold system will automatically take a clone snapshot and start VD clone process The purpose of threshold could prevent the incremental copy fail immediately when running out of snapshot space For example the default threshold is 50 The system will check the snapshot space every hour When the snapshot space is used over 50 the system will synchronize the source VD and target VD automatically ...

Страница 104: ...tart a full copy Caution The default snapshot space allocated by the subsystem is two times the size of source virtual disk That is the best value of QSAN s suggestion If user sets snapshot space by manually and lower than the default value user should take the risk if the snapshot space is not enough and VD clone job will fail 5 After deploying the VD clone rule the VD clone process can be starte...

Страница 105: ...en set Figure 5 6 10 8 Click Start clone the clone process will start Figure 5 6 11 9 The default setting will create a snapshot space automatically which the capacity is double size of the VD space Before starting clone system will initiate the snapshot space ...

Страница 106: ...11 Click Schedule clone to set up the clone by schedule Figure 5 6 14 12 There are Set Clone schedule and Clear Clone schedule in this page Please remember that Threshold and Restart the task an hour later if failed options in VD configuration will take effect after clone schedule has been set ...

Страница 107: ... it will get a warning message This is not enough of snapshot space for the operation At this time the user needs to clean up the snapshot space in order to operate the clone process Each time the clone snapshot failed it means that the system loses the reference value of incremental data So it will start a full copy at next clone process When running out of snapshot space the flow diagram of VD c...

Страница 108: ...t extra SAS dual JBOD controller When connecting to a dual JBOD which can be detected it will be displayed in Show PD for of Volume configuration Physical disk For example Local JBOD 1 QSAN J300H JBOD 2 QSAN J300H etc Local means disks in local controller and so on The hard drives in JBOD can be used as local disks ...

Страница 109: ...109 Figure 5 7 1 1 Figure 5 7 1 1 Display all PDs in JBOD 1 Enclosure management Hardware monitor can display the hardware status of SAS JBODs ...

Страница 110: ...110 Figure 5 7 1 2 ...

Страница 111: ...111 Figure 5 7 1 3 Enclosure management S M A R T can display S M A R T information of all PDs including Local and all SAS JBODs ...

Страница 112: ... chassis 3 When support SATA drives for the redundant JBOD model the QJSATA bridge board is required The QSATA multiplexer board does not apply to this model 4 The following table is the maximum JBOD numbers and maximum HDD numbers with different chassises can be cascaded Dual controller Dual JBOD Single controller Single JBOD RAID Storage System D424 D224 D316 D212 C424 C224 C316 C212 J300Q D424 ...

Страница 113: ...ng the JBOD system must reboot manually to make the new firmware took effect 5 8 MPIO and MC S These features come from iSCSi initiator They can be setup from iSCSI initiator to establish redundant paths for sending I O from the initiator to the target 1 MPIO In Microsoft Windows server base system Microsoft MPIO driver allows initiators to login multiple sessions to the same target and aggregate ...

Страница 114: ...tocol which allows combining several connections inside a single session for performance and failover purposes In this way I O can be sent on any TCP IP connection to the target If one connection fails another connection can continue processing I O without interruption to the application ...

Страница 115: ...to manage load balance and failover Notice that iSCSI connections and sessions are different than TCP IP connections and sessions The above figures describe the difference between MPIO and MC S There are some considerations when user chooses MC S or MPIO for multipathing 1 If user uses hardware iSCSI off load HBA then MPIO is the only one choice 2 If user needs to specify different load balance po...

Страница 116: ...P port can be defined as active or passive QSAN controller implements it as active mode which means that LACP port sends LACP protocol packets automatically Please notice that using the same configurations between QSAN controller and gigabit switch The usage occasion of LACP A It s necessary to use LACP in a network environment of multiple switches When adding new devices LACP will separate the tr...

Страница 117: ...e host can not connect the link with storage device 5 10 Dual controllers 5 10 1 Perform I O Please refer to the following topology and have all the connections ready To perform I O on dual controllers server host should setup MPIO MPIO policy will keep I O running and prevent fail connection with single controller failure ...

Страница 118: ... owner the default owner is controller 1 To change the RG ownership please follow the procedures 1 Select Volume configuration RAID group 2 Check the gray button next to the RG name click Set preferred owner 3 The ownership of the RG will be switched to the other controller Figure 5 10 2 1 ...

Страница 119: ...controller mode The firmware of two controllers is different or the size of memory of two controllers is different In this stage only master controller can work and I O will force to write through for protecting data 4 Single Single controller mode In the stage the controller must stay in slot A QSATA boards for SATA drives are not necessary The differences between single and degraded are describe...

Страница 120: ...llowing step Activate the license key and start to use it Activate the license key User needs to obtain a license key and download it to the RAID subsystem to activate the QReplica function Each license key is unique and dedicated to a specific subsystem It means that the license key for subsystem A can not be used on another subsystem To obtain the license key please contact sales for assistance ...

Страница 121: ...le QReplica in the operation menu Figure 5 11 2 Create backup virtual disk on the target subsystem 1 Before creating the replication job on the source subsystem user has to create a virtual disk on the target subsystem and set the type of the virtual disk as BACKUP ...

Страница 122: ...ackup virtual disk needs to be attached to a LUN ID before creating replication job And the virtual disk of BACKUP type can only be attached with Read only permission to prevent it from being modified incautiously Figure 5 11 4 ...

Страница 123: ...eplication job Figure 5 11 5 2 Select the source virtual disk which will be replicated to the target subsystem and click Next Figure 5 11 6 Tips If the message displays that there is not enough space for creation please refer to the section of Configure the snapshot space below for solution 3 Enter the IP address of iSCSI port on controller 1 of the target subsystem Click Next to continue ...

Страница 124: ...ata replication User has to log on the iSCSI node to create the iSCSI connection for the data transmission Enter the CHAP information if necessary and select the target node to log no Click Next to continue Figure 5 11 8 5 Choose the backup virtual disk and click Next ...

Страница 125: ... listed on the QReplica page Figure 5 11 10 Run the replication job 1 Click the OP button on the replication job to open operation menu Click Start to run the replication job Figure 5 11 11 2 Click Start again to confirm the execution of the replication job ...

Страница 126: ...us information and the progress is expressed by percentage Figure 5 11 13 Create multi path on the replication job 1 Click the Create multi path in the operation menu of the replication job Figure 5 11 14 2 Enter the IP of iSCSI port on controller 2 of the target subsystem ...

Страница 127: ...127 Figure 5 11 15 3 Select the iSCSI node to log on and click Next Figure 5 11 16 4 Choose the same target virtual disk and click Next ...

Страница 128: ...re 5 11 17 5 A new target will be added in this replication job as a redundancy path Figure 5 11 18 Configure the replication job to run by schedule 1 Click Schedule in the operation menu of the replication job ...

Страница 129: ...our by day by week or by month The execution time can be configurable per user s need If the scheduled time of execution is arrived but the pervious replication job is stilling going the scheduled execution will be ignored once Figure 5 11 20 Configure the snapshot space ...

Страница 130: ... job is created 1 To configure the snapshot space settings of QReplica click the configuration button Figure 5 11 21 There are three settings in the QReplica configuration menu Figure 5 11 22 Snapshot space specifies the ratio of snapshot space allocated to the source virtual disk automatically when the snapshot space is not configured in advance The default ratio is 2 to 1 It means when the repli...

Страница 131: ...he size of source virtual disk That is the best value of QSAN s suggestion If user sets snapshot space by manually and lower than the default value user should take the risk if the snapshot space is not enough and QReplica job will fail Best practice for the first time It is always being a problem that to do full copy over LAN or WAN when the replication job is executed at the first time It may ta...

Страница 132: ...nish full copy at the first time For a new replication job created on an existing virtual disk with data stored A full copy is necessary to synchronize the volume image between the source and target virtual disks It usually takes a long time to synchronize the data especially when the replication job is running over WAN with limited bandwidth Now QSAN provides an efficient way to let user execute ...

Страница 133: ...anged from the name of target virtual disk into QRep Figure 5 11 25 Caution Changing a cloning job to a replication job is only available when the cloning job has been finished This change is irreversible 5 Deactivate the RAID group which the target virtual disk resides in and move all physical disks of the RAID group to the target subsystem Then activate the RAID group in the target subsystem and...

Страница 134: ...11 27 8 Follow the steps as creating a new replication job 9 If a wrong target virtual disk is selected when rebuilding the replication job an error message of The rebuilding source and target VDs are mismatched will be pop up and stops user from finishing the creation ...

Страница 135: ...sk slot read block error ERROR HDD write error Disk slot write block error ERROR HDD error Disk slot is disabled ERROR HDD IO timeout Disk slot gets no response INFO PD upgrade started PD string starts upgrading firmware process INFO PD upgrade finished PD string finished upgrading firmware process WARNING PD upgrade failed PD string upgrade firmware failed HW events Level Type Description WARNING...

Страница 136: ...tring is not present ERROR Fan over speed Fan string is over speed WARNING Thermal level 1 System temperature string is higher ERROR Thermal level 2 System Overheated string ERROR Thermal level 2 shutdown System Overheated string The system will auto shutdown immediately ERROR Thermal level 2 CTR shutdown The controller will auto shutdown immediately reason Overheated string WARNING Thermal ignore...

Страница 137: ...ame has been LUN attached INFO VD attach LUN fail Failed to attach LUN to VD name INFO VD detach LUN OK VD name has been detached INFO VD detach LUN fail Failed to attach LUN from bus number SCSI ID number lun number INFO VD init started VD name starts initialization INFO VD init finished VD name completes initialization WARNING VD init failed Failed to complete initialization of VD name INFO VD r...

Страница 138: ...t error create global Failed to create the global cache INFO PD dedicated spare Assign PD slot to be the dedicated spare disk of RG name INFO PD global spare Assign PD slot to Global Spare Disks WARNING PD read error Read error occurred at LBA address address of PD slot WARNING PD write error Write error occurred at LBA address address of PD slot WARNING Scrub wrong parity The parity data inconsis...

Страница 139: ...a assigned to snapshot name is reached INFO Snap clear space The snapshot space of VD name is cleared iSCSI events Level Type Description INFO iSCSI login accepted iSCSI login from IP succeeds INFO iSCSI login rejected iSCSI login from IP was rejected reason string INFO iSCSI logout recvd iSCSI logout from IP was received reason string Battery backup events Level Type Description INFO BBM start sy...

Страница 140: ...write error occurred at LBA address address of JBOD name PD slot WARNING PD read error Read error occurred at LBA address address of JBOD name PD slot WARNING PD write error Write error occurred at LBA address address of JBOD name PD slot INFO PD scrub started JBOD name PD slot starts disk scrubbing process INFO PD scrub completed JBOD name PD slot completed disk scrubbing process WARNING PS fail ...

Страница 141: ...evel Type Description INFO RG owner changed The preferred owner of RG name has been changed to controller number INFO Force CTR write through Controller number forced to adopt write through mode on failover INFO Restore CTR cache mode Controller number restored to previous caching mode on failback INFO Failover complete All volumes in controller number completed failover process INFO Failback comp...

Страница 142: ... VD replicate aborted The replication in VD name was aborted INFO VD set as replica VD name has been configured as a replica INFO VD set as RAID VD name has been configured as a RAID volume INFO VD replica set The replica of VD name has been designated INFO VD replica reset The replica of VD name is no longer designated WARNING Auto qrep not enable Auto QReplica task QReplica is not enabled for VD...

Страница 143: ...r board Serial number 1 Firmware version 2 Backplane Chassis model Backplane version Target configuration RAID configuration 3 Attached file name Event log 4 Attached file name Target HDD HDD amount HDD vendor model HDD firmware version Host information Server vendor model OS HBA HBA BIOS Driver version SW initiator version Others LAN Switch vendor model ...

Страница 144: ...he following descriptions 1 In Maintenance Info Controller serial no or In Maintenance System information Controller serial no 2 In Maintenance Info System information or In Maintenance System information System information 3 In Maintenance Config import export Export or In Maintenance Import and export Export 4 In System config Event log Download or In Maintenance Event log Download Other informa...

Страница 145: ...145 This form can be got from QSAN ftp ftp ftp qsan com tw QSAN_Support_Form doc ...

Страница 146: ...4 24 x GbE HP ProCurve 1800 24G 24 x GbE Netgear GS724T 24 x GbE ZxXEL GS2200 24 x GbE SAS hard drive FF Vendor Family Model Capa city RPM Inter face Ca che FW Comment 3 5 Hitachi Ultrastar 15K600 HUS156045VLS600 450GB 15000 6Gb s 64 A510 3 5 Seagate Cheetah 15K 7 ST3600057SS 600GB 15000 6Gb s 16 0004 3 5 Seagate Cheetah NS 2 ST3600002SS 600GB 10000 6Gb s 16 0004 3 5 Seagate Constellation ES ST320...

Страница 147: ...Gb s 64 01 01V01 3 5 Western Digital RE4 WD5003ABYX 500GB 7200 3Gb s 64 01 01S01 3 5 Western Digital RE4 GP WD2002FYPS 2TB Intelli Power 3Gb s 64 04 01G01 3 5 Western Digital RE3 WD1002FBYS 1TB 7200 3Gb s 32 03 00C05 3 5 Western Digital Caviar Blue WD5000AAKS 500GB 7200 3Gb s 16 Please contact support qsan com tw to get the newest compatibility list B Microsoft iSCSI initiator Here is the step by ...

Страница 148: ...148 Figure B 1 3 Click Done Figure B 2 Figure B 3 ...

Страница 149: ...onnect to an iSCSI disk now MPIO 5 If running MPIO please continue 6 Click Discovery tab to connect the second path 7 Click Discover Portal Input IP address or DNS name of the target Figure B 4 Figure B 5 8 Click OK ...

Страница 150: ...econd path and then click Connect 10 Enable Enable multi path checkbox Then click OK 11 Done it can connect to an iSCSI disk with MPIO MC S 12 If running MC S please continue 13 Select one target name click Properties 14 Click MCS to add additional connections ...

Страница 151: ...151 Figure B 8 Figure B 9 15 Click Add 16 Click Advanced ...

Страница 152: ...B 10 Figure B 11 17 Select Initiator IP and Target portal IP and then click OK 18 Click Connect 19 Click OK Figure B 12 Figure B 13 20 Done Disconnect 21 Select the target name click Disconnect and then click Yes ...

Страница 153: ...153 Figure B 14 22 Done the iSCSI device disconnect successfully ...

Страница 154: ...154 System information P300H61 P300H71 SW version 2 0 1 ...

Результаты поиска

Содержание P300H61

Отзывы:

Похожие инструкции для P300H61

Бренды по названию

Популярные бренды

Qsan Technology P300H61, Руководство пользователя

Результаты поиска

Содержание P300H61

Отзывы:

Похожие инструкции для P300H61

Бренды по названию

Популярные бренды