manualshive.com logo in svg

HP IBRIX X9000, Installation Manual, Page: 140 / 151

Share
Download
HP IBRIX X9000 Installation Manual Download Page 140

When you select

Ok

, the Configure Network dialog box reappears. Select

bond1

, and then

select

Configure

. The Edit Bonded Interface dialog box is displayed. Enter the IP address and

netmask for

bond1

and select

Ok

.

4.

The Configuration Summary lists the configuration you have specified. Select

Commit

to

continue.

The server is registered in the cluster and the wizard configures a passive management console
(Fusion Manager) on the server.

Repeat this procedure to install the second server.

140 Expanding an X9320 cluster with an X9320 starter kit

Summary of Contents for IBRIX X9000

Page 1: ...figure X9000 series Network Storage systems at customer sites For upgrade information see the administration guide for your system For the latest X9000 guides browse to http www hp com support manuals In the storage section select NAS Systems and then select HP X9000 Network Storage Systems from the IBRIX Storage Systems section HP Part Number TA768 96058 Published June 2012 Edition 9 ...

Page 2: ...ory Description Software Version Date Edition Initial release of HP StorageWorks X9000 File Serving Software 5 3 1 November 2009 1 Updated license and quotas information 5 3 2 or later December 2009 2 Major revision of installation and configuration information 5 4 or later April 2010 3 Added X9720 on site commissioning information revise system restore information removed installation blueprints ...

Page 3: ... 45 Installation checklist 46 Configuring OA1 IP addresses for Onboard Administrator 46 Installing the latest IBRIX X9000 software release 48 Starting the installation and configuring the chassis 49 Creating the cluster on blade 1 59 Installing additional X9730 blades 63 Firmware updates 72 Troubleshooting 74 4 Post installation tasks 77 Updating license keys 77 Configuring and Enabling High Avail...

Page 4: ...the Active Directory server 90 Using the Windows X9000 client GUI 90 Preferring a user network interface for a Windows client 91 Enabling file system access 91 Managing Access Control Lists 91 Uninstalling X9000 clients 94 Uninstalling Linux X9000 clients 94 Uninstalling Windows X9000 clients 95 7 Completing the X9730 Performance Module installation 96 Prerequisites 96 Installing the latest IBRIX ...

Page 5: ...iBand network 145 Enabling client access 146 Setting up Voltaire InfiniBand 146 12 Support and other resources 148 Contacting HP 148 Related information 148 HP websites 148 13 Documentation feedback 149 Glossary 150 Contents 5 ...

Page 6: ...fied network creates bond0 on specific interfaces For 10GigE systems bond0 uses the eth4 eth5 interfaces For Quad GigE systems bond0 uses interfaces eth4 eth7 If necessary you can create separate user and cluster networks with bond0 as the cluster network and bond1 as the user network The initial installation configures bond0 on the default interfaces Later you can customize bond0 selecting the co...

Page 7: ...onal 8 Installing the latest IBRIX X9000 software release Obtain the latest 6 1 release from the IBRIX X9000 software dropbox Download the Quick Restore ISO image and then use either a DVD or a USB key to install the image Use a DVD 1 Burn the ISO image to a DVD 2 Insert the Quick Restore DVD into the server s DVD ROM drive 3 Restart the server to boot from the DVD ROM 4 When the HP Network Storag...

Page 8: ...join an existing cluster or create a new cluster If there is not an existing cluster or you chose to create a new cluster the setup process asks for information about the server you are using On the System Date and Time dialog box enter the system date day month year and time in 24 hour format Tab to the Time Zone field and press Enter to display a list of time zones Select your time zone from the...

Page 9: ...ical networks over the same physical networking hardware IBRIX supports the ability to associate a VLAN tag with a FSN interface For more information see the HP IBRIX X9000 Network Storage System Network Best Practices Guide NOTE If you see a message reporting that link down networking issues were detected select Proceed 4 The Configuration Summary lists your configuration Select Commit to continu...

Page 10: ...tered a gateway IP address on the Server Setup screen If your cluster does not meet these conditions select text mode to complete the installation If you would like to proceed with the Getting Started Wizard from some other laptop or desktop connected to the network select Install in graphical mode remotely To proceed with the Getting Started Wizard on this console select Install in graphical mode...

Page 11: ...t sessions The VIF remains highly available Also enter the IP addresses and domain for your DNS servers and the IP addresses for your NTP servers 2 The Network Configuration dialog box lists the Ethernet devices included in bond0 The bond0 configuration is complete select Continue and go to the next step NOTE If required the configuration of bond0 can be modified at this step by selecting bond0 in...

Page 12: ...iguration Summary lists the configuration you have specified Select Commit to continue 4 The wizard now configures the active management console Fusion Manager on the server 12 Installing X9300 and X9320 systems ...

Page 13: ...ress dotted quad format third ip quad ip3 the third section of an IP address dotted quad format second ip quad ip2 the second section of an IP address dotted quad format first ip quad ip1 the first section of an IP address dotted quad format address ip the IP address with dots replaced by dashes reverse address rip The IP addresses reversed by quads with dots replaced by dashes uuid A Universally ...

Page 14: ...without the template Complete the following steps 1 Using a console connection log into the server you are installing The Set IP or Discover FMs dialog box appears Select Discover Existing Clusters to join them from this console 2 The installation wizard scans the network for existing clusters and lists the clusters on the Join Cluster dialog box Select the appropriate cluster 3 The Verify Hostnam...

Page 15: ...sses for your NTP servers 6 The Network Configuration dialog box lists the Ethernet devices included in bond0 The bond0 configuration is complete select Continue and go to the next step NOTE If required the configuration of bond0 can be modified at this step by selecting bond0 in the list of devices and then selecting Configure This should be necessary only if you have a non standard configuration...

Page 16: ...lists the configuration you specified Select Commit to apply the configuration 8 The wizard registers a passive Fusion Manager on the server and then configures and starts it The installation is complete 16 Installing X9300 and X9320 systems ...

Page 17: ...this console 2 The installation wizard scans the network for existing clusters and lists the clusters on the Join Cluster dialog box Select the appropriate Fusion Manager management console 3 The server you are installing is assigned the appropriate name from the template plus the last octet IP for a hostname If necessary you can change the hostname on the Verify Hostname dialog box Completing the...

Page 18: ...and go to step 4 NOTE To change the configuration select Reject and the following screen appears Choose Select FM Again to reselect the Fusion Manager and use the template again To configure the server manually select Enter FM IP and use the procedure corresponding to your network type 18 Installing X9300 and X9320 systems ...

Page 19: ...e entire cluster for management use Think of it as the IP address of the cluster You should connect to this VIF in future GUI management sessions The VIF remains highly available Also enter the IP addresses and domain for your DNS servers and the IP addresses for your NTP servers 2 The Network Configuration dialog box lists the Ethernet devices included in bond0 If the devices are correct go to st...

Page 20: ...al port PCI add on card 1G quad port PCI add on card Embedded server network ports Change the slave devices as necessary for your configuration When you select Ok the Configuration Summary dialog box appears Select Back and return to the Network Configuration dialog box 3 Set up bond1 On the Network Configuration dialog box select Add Device 20 Installing X9300 and X9320 systems ...

Page 21: ...e Add Bonded Interface dialog box enter bond1 as the name for the interface Also specify the appropriate options and slave devices Use mode 6 bonding for a 1GbE network and mode 1 bonding for a 10GbE network Completing the installation in text mode separate cluster and user networks 21 ...

Page 22: ...IP address and netmask for bond1 and select Ok 4 The Configuration Summary lists the configuration you have specified Select Commit to continue 5 The wizard now configures the active management console Fusion Manager on the server 6 The template does not apply when configuring separate cluster and user networks Select Exit 22 Installing X9300 and X9320 systems ...

Page 23: ...10 125 1 8 If you want to be able to access the FM through the bond1 user network create a management VIF for bond1 ibrix_fm c VIF IP address d VIF device n VIF netmask V VIF type For example ibrix_fm c 10 10 125 101 d bond1 1 n 255 255 255 0 v user The installation is complete on the first server Installing additional servers Complete the following steps 1 Using a console connection log into the ...

Page 24: ...ing clusters and lists the clusters on the Join Cluster dialog box Select the appropriate cluster 4 The Verify Hostname dialog box lists a hostname generated from the management console Accept or change this hostname 24 Installing X9300 and X9320 systems ...

Page 25: ... the following screen appears Select Enter FM IP 6 On the System Date and Time dialog box enter the system date day month year and time 24 hour format Tab to the Time Zone field and press Enter to display a list of time zones Select your time zone from the list Completing the installation in text mode separate cluster and user networks 25 ...

Page 26: ...ks If your default gateway is on a different subnet than bond0 skip this field Later in this procedure you can select either Web UI or ASCII text mode to complete the installation A gateway address is required to use the Web UI VLAN capabilities provide hardware support for running multiple logical networks over the same physical networking hardware IBRIX supports the ability to associate a VLAN t...

Page 27: ...the IP you are registering for this server 10 The Network Configuration dialog box lists the Ethernet devices in bond0 If the devices are correct go to the next step If the devices are not correct select bond0 and then select Configure to customize the interface Completing the installation in text mode separate cluster and user networks 27 ...

Page 28: ...hange the slave devices as necessary for your configuration When you select Ok the Configuration Summary dialog box appears Select Back and return to the Network Configuration dialog box 1 1 Configure bond1 Select Add Device from the Network Configuration dialog box 28 Installing X9300 and X9320 systems ...

Page 29: ...e Add Bonded Interface dialog box enter bond1 as the name for the interface Also specify the appropriate options and slave devices Use mode 6 bonding for a 1GbE network and mode 1 bonding for a 10GbE network Completing the installation in text mode separate cluster and user networks 29 ...

Page 30: ...lect Commit to apply the configuration 13 The wizard registers a passive Fusion Manager on the server and then configures and starts it 14 If necessary complete the bond1 configuration on the server If the bond0 cluster network is not routed and the bond1 user network is routed set the default gateway in etc sysconfig network On the active FM set the default route for bond1 ibrix_nic r n bond h ho...

Page 31: ...uster in a few steps Be sure to have the necessary IP addresses available NOTE This wizard can be used only for X9300 and X9320 systems Running the wizard The Cluster Settings page asks for information to identify your cluster Enter a name for the cluster and specify the Cluster Management IP address Running the wizard 31 ...

Page 32: ... and place them in a location that can be accessed by this server Use Browse to locate a license key and then click Add Repeat this step for each license key Enter the DNS server addresses and search domain for your cluster Also enter your NTP server addresses 32 Configuring the cluster with the Getting Started Wizard X9300 X9320 systems ...

Page 33: ...y DNS Server Unreachable will be displayed on the screen The File Servers page lists all servers the wizard found on your network If the list includes servers that do not belong in the cluster select those servers and click Remove If a server is not defined select the server and click Configure Running the wizard 33 ...

Page 34: ...local console setup screens Then click Add on the File Servers page to add the servers to the cluster You can enter a single IP address on the Add Servers dialog box or you can enter a range of addresses and the wizard will locate all servers with IP addresses in the range HP recommends that you create a single file system using all of the available storage You can create this file system on the C...

Page 35: ...rameters such as user permissions Other configuration options are also available for NFS shares See the HP IBRIX X9000 Network Storage System File System User Guide for more information The Summary lists any warnings or other items noted during the post installation checks Click the items to determine whether further action is needed Running the wizard 35 ...

Page 36: ...age lasts only a few minutes Check the network settings and retry the operation You can also use the following command to set the cluster name ibrix_fm_tune S o clusterName clusterName fusionPrimaryAddress Vif add The cluster name and IP address cannot be retrieved Probable cause The command timed out Troubleshooting steps There may be network inconsistencies or an outage Usually a network outage ...

Page 37: ... to set the VIF If that fails QR the system DNS FTP page DNS and NTP cannot be set Probable cause The specified DNS NTP IP addresses are not valid The specified domain name is not valid A network outage or connectivity situation occurred Troubleshooting steps Determine whether the specified IP addresses are valid and can be pinged Run the following command dig short DNS server IP If the output is ...

Page 38: ...servers If the expected servers do not have an IP address use the ILO to check the status service ibrix_discovery status If the discovery service is stopped restart the service service ibrix_discovery start Then relaunch the wizard and go to the File Servers page NOTE If the discovery does not start files may be missing or deleted QR the system Check the X9000 software version on the servers If th...

Page 39: ...already configured and it can be used to form the cluster 2 Check the status of the Fusion Manager registration on the primary server ibrix_fm i If the output is FusionServer fusion manager name not set active quorum is not configured and the output of ibrix_fm l is empty use the following command to perform the Fusion Manager registration ibrix_fm R hostname I ipaddress 3 Run the ibrix_server l c...

Page 40: ...ossibly storage was not detected on either or both of the servers or the storage was detected but it did not match with the expected HA pair If so run the appropriate command on both servers X9730 systems ccu show controllers all X 9300 X9320 systems multipath ll If the output matches on both servers restart the discovery service service ibrix_discovery restart Then relaunch the wizard and go to t...

Page 41: ...port remove the network configuration and then restart the discovery service Check the value of the LISTENER_BONDED_INTERFACE entry in the usr local ibrix bin listener conf file If it is not bond0 edit the file setting the value for LISTENER_BONDED_INTERFACE to bond0 and then restart the discovery service service ibrix_discovery restart If the previous step fails to create the bond0 interface eith...

Page 42: ...rmine the storage exposed to the server run the following command to discover devices ibrix_pv a List the discovered physical devices ibrix_pv l If the output of ibrix_pv l is not empty relaunch the wizard go to the Create a Default File System page and create the file system If the output is empty no storage is available to create the file system Add storage to the server and run the previous com...

Page 43: ...UNTPOINT After mounting the file system on the servers use the GUI share creation wizard or a CLI command to create the export or share Create a NFS export ibrix_exportfs f FSNAME h HOSTLIST p CLIENT1 PATHNAME1 CLIENT2 PATHNAME2 o OPTIONS b Create a CIFS share ibrix_cifs a f FSNAME s SHARENAME p SHAREPATH D SHAREDESCRIPTION S SETTINGLIST h HOSTLIST Check the output of the appropriate command to ve...

Page 44: ...e other layouts provide a dedicated management network a user cluster network and optionally additional user networks To use a network layout other than the default you need to specify the number of networks needed when you run the X9730 Setup Wizard NOTE Use layout 2 when adding an X9730 system to an existing 2 network IBRIX cluster See the HP IBRIX X9000 Network Storage System Network Best Pract...

Page 45: ...ICMP ping the OA to verify the route between the user cluster and management subnets If it is unsuccessful the installer software prompts the user to correct the problem before proceeding IP address requirements You will need the following IP addresses when you perform the installation for the unified network The HP IBRIX X9000 Network Storage System Network Best Practices Guide describes the comp...

Page 46: ...sks 5 Update license keys if needed Enable High Availability Configure Ibrix Collect Configure HP Insight Remote Support Create file systems if not already configured Optionally also configure the following features NFS CIFS HTTP HTTPS FTP FTPS shares Remote replication Data retention and validation Antivirus support Software snapshots Data tiering NDMP Backup Protocol Support Adding Linux and Win...

Page 47: ...A Network Properties screen set Enclosure IP to Enable and press OK to Accept 5 On the Network Settings OA1 Active screen navigate to Active IPv4 and press OK 6 On the Change OA1 Network Mode screen change DHCP to Static and press OK to Accept Configuring OA1 IP addresses for Onboard Administrator 47 ...

Page 48: ...t 6 1 release from the IBRIX X9000 software dropbox Download the Quick Restore ISO image and transfer it to a DVD or USB key Use a DVD 1 Burn the ISO image to a DVD 2 Insert the Quick Restore DVD into a USB DVD drive cabled to the Onboard Administrator or to the Dongle connecting the drive to the front of the blade IMPORTANT Use an external USB drive that has external power do not rely on the USB ...

Page 49: ...talled 6 Restart the server to boot from the USB key Press F1 1 and use option 3 7 When the HP Network Storage System screen appears enter qr to install the software Repeat steps 5 8 on each server Starting the installation and configuring the chassis To start the installation complete the following steps 1 Boot the blades in the cluster 2 Log into blade1 The X9730 Setup dialog box is displayed 3 ...

Page 50: ...for an existing cluster If a cluster is found you will be asked if you want to join an existing cluster or create a new cluster If there is not an existing cluster or you chose to a new cluster the setup wizard asks for information about the blade you are using On the System Date and Time dialog box enter the system date day month year and time 24 hour format Tab to the Time Zone field and press E...

Page 51: ... default gateway is on a different subnet than bond0 skip this field Later in this procedure you can select either Web UI or ASCII text mode to complete the installation A gateway address is required to use the Web UI VLAN capabilities provide hardware support for running multiple logical networks over the same physical networking hardware IBRIX supports the ability to associate a VLAN tag with a ...

Page 52: ...the system is at factory defaults the administrator passwords for OA and Virtual Connect are on the labels affixed to the back of the chassis If the passwords have been reset enter the new passwords IMPORTANT If you are using the default unified network layout select OK on the Confirming Onboard Administrator and Virtual Connect Settings dialog box and go to step 8 To use a different network layou...

Page 53: ... manager fails the GUI displays a message asking you to initialize the credential manager manually See Credential Manager initialization failed page 76 for more information 9 The wizard now verifies the OA firmware 10 Set the iLO IP addresses On the Get iLO IP Addresses dialog box select the method you want to use to set up iLO IP addresses Use the space bar to select deselect the check boxes To c...

Page 54: ...ified on the Confirm iLO IP Addresses dialog box Select Ok to continue 12 Configure the chassis interconnect bays VCs and SAS switches On the Get Interconnect IP Addresses dialog box specify whether you want to configure the Interconnect IC IP addresses in sequence or manually Use the space bar to select deselect the check boxes 54 Installing X9730 systems ...

Page 55: ...or example if 172 16 3 21 is the starting Interconnect IC IP Address the installer sets the Interconnect IC IP Addresses in the range 172 16 3 21 172 16 3 28 To configure the Interconnect IP addresses manually enter each address on the Set Interconnect IP Addresses dialog box 13 The wizard lists the IP addresses you specified on the Confirm IC IP Addresses dialog box Select Ok to continue Starting...

Page 56: ...ails you can retry the attempt or re enter the password Retry is helpful only if a timeout caused the VC password check to fail When the wizard can log into the VC manager successfully it verifies the VC firmware and asks you to update it if necessary When the firmware is at the correct level the wizard verifies that the VC is in a default state 17 The wizard configures the VC The setup process fi...

Page 57: ... The wizard verifies the VC configuration and then creates an hpspAdmin user account on each iLO 20 The wizard validates the VC configuration and verifies the SAS firmware If necessary the SAS switches are flashed with the correct firmware 21 The wizard verifies the SAS configuration After determining the correct layout of the storage hardware the wizard configures the SAS switch zoning so that co...

Page 58: ...pears 23 The wizard takes the following actions Verifies the SAS configuration to ensure that SAS zoning is set up correctly Powers on blades 2 16 Verifies storage firmware to ensure that is set up correctly Validates the LUN layout and configures it if necessary 24 The wizard now forms bond0 from eth0 and eth3 58 Installing X9730 systems ...

Page 59: ... the IP address and netmask for the Management Console IP also called the Cluster Management IP This IP address runs on a virtual interface VIF assigned to the entire cluster for management use Think of it as the IP address of the cluster You should connect to this VIF in future GUI management sessions The VIF remains highly available NOTE If you are using a 2 3 or 4 network layout install this IP...

Page 60: ...ut the necessary bonds appear on the Network Configuration screen with the correct eth devices assigned but you need to configure the IP addresses for those bonds Select a bond and then select Configure On the Edit Bonded Interface dialog box enter the IP address and netmask and specify any bond options The Configuration Summary lists the configuration you have specified Select Commit to continue ...

Page 61: ...for all other layouts 3 The wizard now configures the active management console Fusion Manager on this blade 4 Optionally create a template to configure the remaining nodes The template is useful only if all of the blades have identical network configurations such as in the unified network Creating the cluster on blade 1 61 ...

Page 62: ...dress dotted quad format address ip the IP address with dots replaced by dashes reverse address rip The IP addresses reversed by quads with dots replaced by dashes uuid A Universally Unique Identifier Following are some examples template ib74s fourth ip quad ip hostname 192 168 74 3 ib74s3 template ib74s first ip quad ip hostname 192 168 74 3 ib74s192 template Noname address ip hostname 192 168 74...

Page 63: ...yed 3 The wizard verifies the firmware on the system and notifies you if a firmware update is needed IMPORTANT HP recommends that you update the firmware before continuing with the installation X9730 systems have been tested with specific firmware recipes Continuing the installation without upgrading to a supported firmware recipe can result in a defective system Installing additional X9730 blades...

Page 64: ...re the blade Select Ok and go to Configure the blade with the template page 64 Configure the blade manually If you did not create a template when configuring the first blade the following screen appears when you select Ok on the Join Cluster dialog box Select Enter FM IP and go to Configure the blade manually page 68 Configure the blade with the template Complete the following steps 64 Installing ...

Page 65: ...ation dialog box shows the configuration for this blade If the configuration is correct select Accept and go to step 3 NOTE To change the configuration select Reject and the following screen appears Choose Select FM Again to reselect the Fusion Manager and use the template again To configure the node manually select Enter FM IP and and go to Configure the blade manually page 68 Installing addition...

Page 66: ...tion Creates the hpspAdmin user accounts on the iLOs Verifies the chassis configuration Checks the firmware on the SAS switches and notifies you if an update is needed Verifies the SAS configuration Checks the storage firmware and notifies you if an update is needed Verifies the storage configuration Sets up networking on the blade 5 On the Join a Cluster Step 2 dialog box enter the requested info...

Page 67: ...ists the interfaces configured on bond0 The configuration is complete Select Continue 7 The Configuration Summary dialog box lists the configuration you specified Select Commit to apply the configuration Installing additional X9730 blades 67 ...

Page 68: ...le on the blade The installation is complete Configure the blade manually Complete the following steps 1 On the System Date and Time dialog box enter the system date day month year and time 24 hour format Tab to the Time Zone field and press Enter to display a list of time zones Then select your time zone from the list 68 Installing X9730 systems ...

Page 69: ...ween networks If your default gateway is on a different subnet than bond0 skip this field VLAN capabilities provide hardware support for running multiple logical networks over the same physical networking hardware IBRIX supports the ability to associate a VLAN tag with a FSN interface For more information see the HP IBRIX X9000 Network Storage System Network Best Practices Guide 3 The Configuratio...

Page 70: ...update is needed Verifies the SAS configuration Checks the storage firmware and notifies you if an update is needed Verifies the storage configuration Sets up networking on the blade 6 On the Join a Cluster Step 2 dialog box enter the requested information NOTE On the dialog box Register IP is the Fusion Manager management console IP not the IP you are registering for this blade 7 The Network Conf...

Page 71: ... screen with the correct eth devices assigned but you need to configure the IP addresses for those bonds Select a bond and then select Configure On the Edit Bonded Interface dialog box enter the IP address and netmask and specify any bond options 8 The Configuration Summary dialog box lists the configuration you specified Select Commit to apply the configuration Installing additional X9730 blades ...

Page 72: ...1 for all other layouts 9 The wizard registers and starts a passive management console on the blade The installation is complete Firmware updates If a firmware check determines that firmware needs to be updated you will see a dialog box such as the following Select Update to perform the update 72 Installing X9730 systems ...

Page 73: ...the firmware during the installation Select Skip and then downgrade the firmware when the installation is complete Chassis component firmware can be flashed only from blade 1 When installing blade 2 and any remaining blades you may see the following dialog box if you are running the GUI from a blade other than blade 1 Firmware updates 73 ...

Page 74: ...gure the OA as follows to resolve this condition 1 From the Main Menu of the Insight Display navigate to Enclosure Settings and press OK 2 On the Enclosure Settings screen select Active OA and press OK 3 On the Network Settings OA1 Active screen select Network Properties and press OK 74 Installing X9730 systems ...

Page 75: ...ions can also make the install software unable to ping the OA VC is uplinked from X1 rather than X6 be sure that X6 is used on X9730 systems A gateway is not defined for OA or blade No L3 route from VC to OA Duplicate IP is used for blade or OA VC is already configured but blade 1 does not have a VC profile assigned An X9370 blade boots but the Installer cannot ping the VC After the blade boots th...

Page 76: ...e iLO with its credentials opt hp platform bin hpsp_credmgmt update cred cred selector chassis global ilo cred type upwpair cred username ilo username cred password ilo password OA and VC have different usernames and passwords 1 Update the OA with its credentials opt hp platform bin hpsp_credmgmt update cred cred selector chassis chassis oa cred type upwpair cred username username cred password pa...

Page 77: ... each VIF ibrix_nic b H node1 bond0 1 node2 bond0 2 nl ibrix_nic b H node2 bond0 1 node1 bond0 2 To enable HA on the cluster run the following command on the server hosting the active Fusion Manager ibrix_server m NOTE The example shows bond0 which is correct for the unified network If you are using separate cluster and user networks specify bond1 Using the management console GUI The GUI is a brow...

Page 78: ...e default value for path is the root directory of the file system The other default values on the dialog box should be adequate for most sites NFS client implementation tuning NFS clients perform best when the file system policy on the X9000 file servers is tuned to prefer LOCAL segments and ROUNDROBIN This tuning ensures that NFS client writes will not suffer a network hop penalty on writes to re...

Page 79: ...stem File System User Guide Managing X9000 CIFS Shares in an Active Directory Environment Quick Start Guide Configuring other X9000 software features See the HP IBRIX X9000 Network Storage System File System User Guide for information about configuring the following features HTTP HTTPS FTP FTPS Remote replication Data retention and validation Antivirus support Software snapshots Block snapshots no...

Page 80: ...uld be configured in backup pairs and the NICs on those servers should also be configured for failover For Linux and Windows X9000 clients the servers hosting the VIF should be configured in backup pairs However X9000 clients do not support backup NICs Instead X9000 clients should connect to the parent bond of the user VIF or to a different VIF Creating a bonded VIF NOTE The examples in this chapt...

Page 81: ...ailover for your file serving nodes execute the following command ibrix_server m h SERVERNAME Example configuration This example uses two nodes ib50 81 and ib50 82 These nodes are backups for each other forming a backup pair root ib50 80 ibrix_server l Segment Servers SERVER_NAME BACKUP STATE HA ID GROUP ib50 81 ib50 82 Up on 132cf61a d25b 40f8 890e e97363ae0d0b servers ib50 82 ib50 81 Up on 7d258...

Page 82: ... the command once for each destination host that the client should contact using the specified interface ibrix_client n h SRCHOST A DESTNOST IFNAME For example ibrix_client n h client12 mycompany com A ib50 81 mycompany com bond1 NOTE Because the backup NIC cannot be used as a preferred network interface for X9000 clients add one or more user network interfaces to ensure that HA and client communi...

Page 83: ... that will be used for cluster network communications Set up only one cluster interface NOTE By default communication flows through the cluster network If that network is not available to the client you will need to prefer another communication route for the client See Preferring a network interface for a Linux X9000 client page 84 Verify that the client machine can communicate with the active man...

Page 84: ...ew management console and clusterIF is the interface used for cluster communication The new management console entry is added to the client iadconf xml file and can be listed by executing the ibrix_client command on the new management console 2 Restart X9000 client services etc init d ibrix_client restart 3 On the new cluster s active management console create a mountpoint for the new client and s...

Page 85: ...rs are supported Windows Server 2003 SP2 This server requires that you install the SFU 3 5 software which is part of the server distribution but is not installed by default Windows Server 2003 R2 Windows Server 2008 Windows Server 2008 R2 User Mapping Services are built into these operating systems and do not require installation Note the following requirements DHCP must be disabled X9000 software...

Page 86: ...software does not require installation of the entire SFU 3 5 package What you install depends on what your site supports but you must install the Server for NIS component When you run the SFU Setup Wizard you must select at least the Server for NIS component Services for UNIX on Windows 2003 R2 and later 1 On the Active Directory server open the Control Panel select Add or Remove Programs and clic...

Page 87: ...r and delegate control folder The proxy user queries the Active Directory server on behalf of the client to find mappings from Linux UIDs GIDs to Windows SSIDs This user is required It must be defined in the management console using the ibrix_activedirectory command and it must be created in Active Directory 1 Log in to the Active Directory s Main Catalog server and open the Active Directory Users...

Page 88: ... user name and unknown Windows user name with passwords as created in Active Directory On Windows Server 2003 R2 the E and F arguments are required Use the field names gidNumber and uidNumber as values installdirectory bin ibrix_activedirectory S d DOMAIN_NAME i DOMAIN_CONTROLLER_IP u PROXY_USER p PROXY_PASSWORD E UID_FIELD_NAME F GID_FIELD_NAME W WIN_USER NOTE Specify the proxy user name in the f...

Page 89: ...y and enter the file system name To mount the file system to a Windows drive click Mount NOTE If you are using Remote Desktop to access the client and the drive letter is not displayed log out and log back in This is a known limitation of Windows Terminal Services when exposing new drives If you change the Active Directory settings on the management console copy the new settings to each client Als...

Page 90: ...ll window select Password and Group file and enter c temp passwd for the password file and c temp group for the group file 8 On Windows Server 2003 SP2 only create the file SFU program directory nis ldif log Windows does not create this file when SFU is installed 9 Open a DOS command window and change directory to c temp 10 Import user UIDs and GIDs to the Active Directory server nis2ad y ibrix a ...

Page 91: ...s clients need to contact Enabling file system access Only the user who has mounted the filesystem initially has write permission or full privilege for the file system To give other users write permission edit the properties of the root file system Managing Access Control Lists Because the Windows X9000 client might be operating in a mixed mode environment in which the same file objects are access...

Page 92: ... Owner owning user Group special ACE Group owning group Everyone special ACE Other The permissions for each special ACE are set according to the bits in each category If all bits in some categories are cleared no corresponding special ACE is added to the file ACL and no explicit deny ACE is generated User mapping Owner mapping Each file and directory in Linux has a UID that defines its owner and s...

Page 93: ...y explicitly set rights override inherited restrictions To view a Windows ACL right click any file on a Windows 2003 server and select Properties Click the Security tab The Security Properties tab for your file will look like the one shown below If you right click a directory the additional permission List Directory Contents is included This is the ACL for a file named postinfo html The windows sh...

Page 94: ...ion is set on directories for Group and Other A Linux like permission schema can be implemented by setting a umask The umask defines the bits to clear in the target file mode mask For example a umask of 022 allows for an initial file mode mask permission of 755 rwx for owner and rx for group and other An administrator can restrict it by setting the umask to 077 which clears all bits in the file mo...

Page 95: ... a client complete the following steps 1 On the active management console delete the Windows X9000 clients from the configuration database installdirectory bin ibrix_client d h CLIENTLIST 2 Locally uninstall the Windows X9000 client software from each X9000 client via the Add or Remove Programs utility in the Control Panel Uninstalling X9000 clients 95 ...

Page 96: ...strator or to the Dongle connecting the drive to the front of the blade IMPORTANT Use an external USB drive that has external power do not rely on the USB bus for power to drive the device 3 Restart the server to boot from the DVD ROM 4 When the HP Network Storage System screen appears enter qr to install the software Repeat steps 2 4 on each server Use a USB key 1 Copy the ISO to a Linux system 2...

Page 97: ...me manner The following screen shows the cluster before the expansion blades are added To install the first blade in the expansion module complete these steps 1 Log into the blade in the first expansion slot The X9730 Setup dialog box is displayed 2 The setup wizard verifies the firmware on the system and notifies you if a firmware update is needed See Firmware updates page 72 for more information...

Page 98: ...pe can result in a defective system 3 The setup wizard checks the network for an existing active Management Console When the Set IP or Discover FMs dialog box appears select Discover Existing Clusters to join them from this console 4 The wizard scans again for active management consoles and lists them on the Join Cluster dialog log Select the appropriate management console 98 Completing the X9730 ...

Page 99: ... the IP address is incorrect If all of the information on the Verify Configuration dialog box is correct select Accept and go to the next step If you rejected the configuration the following screen appears Select Enter FM IP to continue On the System Date and Time dialog box enter the system date day month year and time 24 hour format Tab to the Time Zone field and press Enter to display a list of...

Page 100: ...up the hpspAdmin user account 8 The wizard verifies the SAS configuration After determining the correct layout of the storage hardware the wizard configures the SAS switch zoning so that couplets see the same storage 9 The wizard powers off blade 4 applies the SAS configuration and then reboots blade 3 Log into blade 3 when the Linux login prompt appears 100 Completing the X9730 Performance Module...

Page 101: ... dialog box appears enter the requested information NOTE Register IP is the Fusion Manager management console IP not the IP you are registering for this server 13 The Network Configuration dialog box lists the configuration for bond0 If your cluster uses the unified network the configuration is correct Select Continue 14 The Configuration Summary dialog box lists the configuration of the blade If ...

Page 102: ...dure is similar to the first node except the firmware chassis SAS and storage checks are already in place 1 Log into the second expansion node slot 4 in our example 2 Log into the blade in the first expansion slot The X9730 Setup dialog box is displayed 3 The setup wizard verifies the firmware on the system and notifies you if a firmware update is needed See Firmware updates page 72 for more infor...

Page 103: ...re recipe can result in a defective system 4 The setup wizard checks the network for an existing active Management Console When the Set IP or Discover FMs dialog box appears select Discover Existing Clusters to join them from this console 5 The wizard scans again for active management consoles and lists them on the Join Cluster dialog log Select the appropriate management console Installing the se...

Page 104: ... hostname in the previous step select Reject because the IP address is incorrect If all of the information on the Verify Configuration dialog box is correct select Accept and go to the next step If you rejected the configuration enter the information for the blade on the Server Setup dialog box Review the information on the Configuration Summary that appears next and select Commit 104 Completing t...

Page 105: ...tion Validates the storage RAID LUN configuration 9 When the Join a Cluster Step 2 dialog box appears enter the requested information NOTE Register IP is the Fusion Manager management console IP not the IP you are registering for this server 10 The Network Configuration dialog box lists the configuration for bond0 If your cluster uses the unified network the configuration is correct Select Continu...

Page 106: ...ole is installed and registered on the blade 13 The GUI now shows the fourth blade in the cluster Using the new storage To make the new storage available to the cluster take these steps Verify the vendor storage Import the new physical volumes into the IBRIX database Extend an existing file system to include the new physical volumes or create a new file system 106 Completing the X9730 Performance ...

Page 107: ...the following command root ib121 121 ibrix_chassis l NAME HARDWARE_UNIT_TYPE SERIAL_NUMBER x9730_ch_09USE127C72Y x9730 09USE127C72Y Import the new physical volumes into the IBRIX database The new physical volumes must be discovered by the X9000 software to make them available to the cluster On the GUI select Storage from the Navigator The Storage panel lists the physical volumes included on the or...

Page 108: ...x_mount f ibfs1 h ib121 123 ib121 124 m ibfs1 Extend the file system On the CLI use the following command ibrix_fs e f FSNAME p PVLIST t TIERNAME The following command extends file system ibfs1 with physical volumes d39 d68 and assigns them to data tier SAS ibrix_fs e f ibfs1 p d39 d40 d41 d42 d43 d44 d63 d64 d65 d66 d67 d68 t SAS To expand a file system from the GUI select the file system on the ...

Page 109: ... ibrix_chassis i nl ibrix_chassis i s nl ibrix_vs i 5 Run the following command on each node to confirm that the Fusion Manager started hpspmon with the appropriate categories ps ef grep hpspmon The categories in the process list output should be STORAGE SERVER CHASSIS for one designated Vendor Storage server an X9720 node All other X9720 nodes should show monitoring of the SERVER category only X9...

Page 110: ...d to make USB the QR installer dd if ISO file name with path of dev sdi oflag direct bs 1M For example dd if X9000 QRDVD 6 2 96 1 x86_64 iso of dev sdi oflag direct bs 1M 4491 0 records in 4491 0 records out 4709154816 bytes 4 7 GB copied 957 784 seconds 4 9 MB s 5 Insert the USB key into the first expansion blade 6 Restart the blade to boot from the USB key Press F1 1 and use option 3 7 When the ...

Page 111: ...allation X9730 systems have been tested with specific firmware recipes Continuing the installation without upgrading to a supported firmware recipe can result in a defective system 3 Select Join an existing cluster from the System Deployment Menu 4 The wizard scans the network for active management consoles and lists them on the Join Cluster dialog log Select the appropriate management console Ins...

Page 112: ...e correct hostname and select Ok 6 The Verify Configuration dialog box shows the configuration of the blade Select Reject it is necessary to specify the setup information for the server The following screen appears Select Enter FM IP 1 12 Expanding an X9720 or X9320 10GbE cluster by an X9730 module ...

Page 113: ...aracters in hostnames uppercase characters can cause issues with IBRIX software Do not use an underscore _ in the hostname The IP address is the address of the server on bond0 The default gateway provides a route between networks If your default gateway is on a different subnet than bond0 skip this field VLAN capabilities provide hardware support for running multiple logical networks over the same...

Page 114: ...luster uses separate user and cluster networks you need to configure those networks on this blade If the existing cluster uses a single network go to step 8 for information about the Confirming Onboard Administrator and Virtual Connect Settings dialog box If the existing cluster uses a different network layout do not make any entries on the Confirming Onboard Administrator and Virtual Connect Sett...

Page 115: ...networks used by the existing cluster this will typically be 2 8 The Confirming Onboard Administrator and Virtual Connect Settings dialog box is displayed again and the setup wizard configures the chassis on the X9730 system This step will fail if the OA IP address has not been set up or if the blade cannot communicate with the OA Installing the first expansion blade 1 15 ...

Page 116: ...ter information select Back to return to the Confirming Onboard Administrator and Virtual Connect Settings dialog box where you can make your changes NOTE If the initialization of the credential manager fails the wizard displays a message asking you to initialize the credential manager manually See Credential Manager initialization failed page 76 for more information 10 The wizard now verifies the...

Page 117: ...og box 12 The wizard lists the IP addresses you specified on the Confirm iLO IP Addresses dialog box Select Ok to continue 13 Configure the chassis interconnect bays VCs and SAS switches On the Get Interconnect IP Addresses dialog box specify whether you want to configure the Interconnect IC IP addresses in sequence or manually Use the space bar to select deselect the check boxes Installing the fi...

Page 118: ...ample if 172 16 3 21 is the starting Interconnect IC IP Address the installer sets the Interconnect IC IP Addresses in the range 172 16 3 21 172 16 3 28 To configure the Interconnect IP addresses manually enter each address on the Set Interconnect IP Addresses dialog box 14 The wizard lists the IP addresses you specified on the Confirm IC IP Addresses dialog box Select Ok to continue 1 18 Expandin...

Page 119: ...u can retry the attempt or re enter the password Retry is helpful only if a timeout caused the VC password check to fail When the wizard can log into the VC manager successfully it verifies the VC firmware and asks you to update it if necessary When the firmware is at the correct level the wizard verifies that the VC is in a default state 18 The wizard configures the VC The setup process first pow...

Page 120: ... wizard verifies the VC configuration and then creates an hpspAdmin user account on each iLO 21 The wizard validates the VC configuration and verifies the SAS firmware If necessary the SAS switches are flashed with the correct firmware 22 The wizard verifies the SAS configuration After determining the correct layout of the storage hardware the wizard configures the SAS switch zoning so that couple...

Page 121: ...g is set up correctly Powers on blades 2 16 Verifies storage firmware to ensure that is set up correctly Validates the LUN layout and configures it if necessary 25 The wizard now forms bond0 from eth0 and eth3 26 When the Join a Cluster Step 2 dialog box appears enter the requested information NOTE Register IP is the Fusion Manager management console IP not the IP you are registering for this serv...

Page 122: ...t eth devices should be assigned but you need to configure the IP addresses for the bonds Select a bond and then select Configure On the Edit Bonded Interface dialog box enter the IP address and netmask and specify any bond options 122 Expanding an X9720 or X9320 10GbE cluster by an X9730 module ...

Page 123: ...xpansion blade The installation procedure is similar to the first node except the firmware chassis SAS and storage checks are already in place 1 Log into the second expansion node slot 4 in our example 2 Log into the blade in the first expansion slot The X9730 Setup dialog box is displayed 3 The setup wizard verifies the firmware on the system and notifies you if a firmware update is needed See Fi...

Page 124: ...n result in a defective system 4 The setup wizard checks the network for an existing active Management Console When the Set IP or Discover FMs dialog box appears select Discover Existing Clusters to join them from this console 5 The wizard scans again for active management consoles and lists them on the Join Cluster dialog log Select the appropriate management console 124 Expanding an X9720 or X93...

Page 125: ...hostname later in this procedure 7 The Verify Configuration dialog box shows the configuration of the blade Select Reject as it is necessary to customize the configuration The following screen appears Select Enter FM IP Enter the information for the blade on the Server Setup dialog box Installing the second expansion blade 125 ...

Page 126: ...rifies the VC firmware Validates the chassis configuration Verifies VC authentication Sets up the hpspAdmin iLO user account Verifies SAS firmware Verifies SAS configuration Validates the storage RAID LUN configuration During the checks you will be asked for authentication information for blade1 126 Expanding an X9720 or X9320 10GbE cluster by an X9730 module ...

Page 127: ...ation NOTE Register IP is the Fusion Manager management console IP not the IP you are registering for this server 10 The Network Configuration dialog box lists the interfaces you configured earlier The correct eth devices should be assigned but you need to configure the IP addresses for those bonds Select a bond and then select Configure Installing the second expansion blade 127 ...

Page 128: ...er the IP address and netmask and specify any bond options 1 1 The Configuration Summary dialog box lists the configuration of the blade If the information is correct select Commit 128 Expanding an X9720 or X9320 10GbE cluster by an X9730 module ...

Page 129: ...ree To verify the vendor storage from IBRIX run the following command to list the VS storage modules root ib121 121 ibrix_vs l NAME HARDWARE_UNIT_TYPE x9730_ch_09USE127C72Y_vs1 x9730 x9730_ch_09USE127C72Y_vs2 x9730 The first entry is the original X9730 system The second entry is the new expansion module To verify the chassis registration run the following command root ib121 121 ibrix_chassis l NAM...

Page 130: ...n Manager specifying just the new blades with the h option ibrix_pv a h ib121 123 ib121 124 To see the LUNs associated with the physical volumes select Vendor Storage from the Navigator and select the new storage expansion module from the Vendor Storage Panel In the lower Navigator expand the Summary completely and select LUN 130 Expanding an X9720 or X9320 10GbE cluster by an X9730 module ...

Page 131: ...nd ibrix_fs e f FSNAME p PVLIST t TIERNAME The following command extends file system ibfs1 with physical volumes d39 d68 and assigns them to data tier SAS ibrix_fs e f ibfs1 p d39 d40 d41 d42 d43 d44 d63 d64 d65 d66 d67 d68 t SAS To expand a file system from the GUI select the file system on the Filesystems panel and then select Extend on the Summary panel The Extend Filesystem dialog box allows y...

Page 132: ...screen appears enter qr to install the software Repeat steps 2 4 on the second expansion server Use a USB key 1 Copy the ISO to a Linux system 2 Insert a USB key into the Linux system 3 Execute cat proc partitions to find the USB device partition which is displayed as dev sdX For example cat proc partitions major minor blocks name 8 128 15633408 sdi 4 Execute the following dd command to make USB t...

Page 133: ...4 If you selected a cluster on the Join Cluster dialog box the Verify Hostname dialog box displays a hostname generated by the management console If the name is not correct enter the correct hostname 5 The Verify Configuration dialog box shows the configuration of the blade If you did not change the hostname on the previous screen select Accept and the wizard will set up the server Installing the ...

Page 134: ... server on bond0 Note the following The hostname can include alphanumeric characters and the hyphen special character It is a best practice to use only lowercase characters in hostnames uppercase characters can cause issues with IBRIX software Do not use an underscore _ in the hostname The IP address is the address of the server on bond0 The default gateway provides a route between networks If you...

Page 135: ...ied network ASCII text mode This is a continuation of the menus you have been using This method must be used for all other expansions as it allows the network on the new server to be configured to match the existing cluster See Completing the installation in text mode page 136 Completing the installation with the eWizard Exit the ASCII text wizard and complete the following steps 1 Open the GUI se...

Page 136: ...ew server in the same manner When you click OK the new server is updated on the GUI IBRIX HA is also configured on the servers 6 Click Next on the File Servers screen to save the configuration and the servers will be automatically registered in the cluster 7 Exit the wizard Completing the installation in text mode Continue the installation of the first expansion server 136 Expanding an X9320 clust...

Page 137: ...ion on the new server either add interface eth0 or create a bond1 composed of onboard eth0 3 as slave interfaces X9320 with 1GbE having the cluster network on bond0 eth0 3 and the user network on bond1 eth4 7 To match this configuration on the new server edit the bond0 configuration to use use eth0 3 as the slave interfaces and create bond1 with eth 4 7 as slave interfaces The Network Configuratio...

Page 138: ...ange the slave devices as necessary When you select Ok the Configuration Summary dialog box appears Select Back and return to the Network Configuration dialog box 3 Set up bond1 if necessary On the Network Configuration dialog box select Add Device 138 Expanding an X9320 cluster with an X9320 starter kit ...

Page 139: ...d interface On the Add Bonded Interface dialog box enter bond1 as the name for the interface Also specify the appropriate options and slave devices Use mode 6 bonding for a 1GbE network and mode 1 bonding for a 10GbE network Completing the installation in text mode 139 ...

Page 140: ...ress and netmask for bond1 and select Ok 4 The Configuration Summary lists the configuration you have specified Select Commit to continue The server is registered in the cluster and the wizard configures a passive management console Fusion Manager on the server Repeat this procedure to install the second server 140 Expanding an X9320 cluster with an X9320 starter kit ...

Page 141: ...inux client ibrixinit tc C CLUSTER_IF i CLUSTER_NAME VIF_IP For example ibrixinit tc C eth4 i 192 168 49 54 Uninstall the X9000 client package from a Linux client including the RPMs ibrixinit tc u Uninstall the X9000 client package from a Linux client retaining the RPMs ibrixinit tc U Options Description Option Specifies the cluster interface C CLUSTER_IF Specifies the User virtual interface devic...

Page 142: ...c Uninstalls Fusion Manager file serving node software and statstool components including the RPMs on this file serving node u Specifies the Cluster virtual interface IP address for Fusion Manager v VIF_IP Specifies a port for Fusion Manager to listen on for webserver communication w PORT_NO Display a help message for this command h 142 Using ibrixinit ...

Page 143: ...iniBand couplets InfiniBand is supported for X9300 and X9320 systems The following logical network diagram shows an InfiniBand configuration The following diagram shows network cabling for an InfiniBand configuration 143 ...

Page 144: ...he software NOTE Review the release notes for general limitations and known issues with OFED 1 5 3 Installing the driver To install the driver complete the following steps 1 Install the IB OFFED stack without 32 bit and without ib bonding Run the following command mlnxofedinstall without 32bit without ib bonding 2 Check the affected kernel files lib modules 2 6 18 194 el5 updates kernel net sunrpc...

Page 145: ...rmine whether opensmd is running on the IB network If opensmd is not running issue the following commands chkconfig opensmd on service opensmd start 7 Verify the status of the HCA NOTE If you are using Host Based SM by default it is tied to Port1 of the HCA Run the following checks ofed_info ibstat ibclearcounters ibdiagnet lw 4x ls 10 r 8 Verify that the link is up and the state is active If the ...

Page 146: ...g up Voltaire InfiniBand NOTE The installation procedure of the third party InfiniBand drivers could be different than what is shown here Refer to the third party documentation for the most accurate information or contact your vendor s support 1 Install the Voltaire OFED drivers on each couplet and client root ib VoltaireOFED 1 4 2_2 k2 6 18 128 el5 x86_64 install sh installation will replace your...

Page 147: ...ng ufm client utils package usr src redhat RPMS x86_64 ufm client utils 1 0 0 2 x86_64 rpm Preparing 100 1 ufm client utils 100 Service ufmdiscoverd installed succesfully Service ib gvd installed succesfully DHCP client installed succesfully Start ufmagentd UFM discover proc started UFM gvd proc started 3 Install the Voltaire UFM server software on the UFM server This is a Management Server that i...

Page 148: ...se Notes HP IBRIX X9000 Network Storage System File System User Guide HP IBRIX X9000 Network Storage System CLI Reference HP IBRIX X9300 X9320 Network Storage System Administrator Guide HP IBRIX X9720 X9730 Network Storage System Administrator Guide Related documents are available on the Manuals page at http www hp com support manuals On the Manuals page select storage NAS Systems Ibrix Storage Sy...

Page 149: ...cumentation that meets your needs To help us improve the documentation send any errors suggestions or comments to Documentation Feedback docsfeedback hp com Include the document title and part number version number or the URL when submitting your feedback 149 ...

Page 150: ...unch of disks KVM Keyboard video and mouse LUN Logical unit number A LUN results from mapping a logical unit number port ID and LDEV ID to a RAID group The size of the LUN is determined by the emulation mode of the LDEV and the number of LDEVs associated with the LUN MTU Maximum Transmission Unit NAS Network attached storage NFS Network file system The protocol used in most UNIX environments to sh...

Page 151: ...on USM SNMP User Security Model VACM SNMP View Access Control Model VC HP Virtual Connect VIF Virtual interface WINS Windows Internet Naming Service WWN World Wide Name A unique identifier assigned to a Fibre Channel device WWNN World wide node name A globally unique 64 bit identifier assigned to each Fibre Channel node process WWPN World wide port name A unique 64 bit address used in a FC storage...

Reviews:

No comments