Red Hat GFS 6.0 - Administrator'S Manual Download Page 18

Page: 18 / 192

Chapter 1. Red Hat GFS Overview

LAN

Clients

GNBD

servers

SAN

Fabric

GFS

Applications

Shared Files

Figure 1-2. GFS and GNBD with a SAN

1.2.3. Economy and Performance

Figure 1-3 shows how Linux client applications can take advantage of an existing Ethernet topology

to gain shared access to all block storage devices. Client data files and file systems can be shared with

GFS on each client. Application failover can be fully automated with Red Hat Cluster Suite.

Summary of Contents for GFS 6.0 -

Page 1: ...Red Hat GFS 6 0 Administrator s Guide...

Page 2: ...ailable at http www opencontent org openpub Distribution of substantively modified versions of this document is prohibited without the explicit permission of the copyright holder Distribution of the w...

Page 3: ...3 4 Cluster Configuration Management 6 1 4 GFS Software Subsystems 7 1 5 Before Configuring GFS 9 2 System Requirements 11 2 1 Platform Requirements 11 2 2 TCP IP Network 11 2 3 Fibre Channel Storage...

Page 4: ...age 30 5 8 2 Example procedure 31 5 9 Erasing a Pool Volume 31 5 9 1 Usage 32 5 9 2 Example 32 5 9 3 Comments 32 5 10 Renaming a Pool Volume 32 5 10 1 Usage 32 5 10 2 Example 33 5 11 Changing a Pool V...

Page 5: ...5 8 2 1 Selection of LOCK_GULM Servers 85 8 2 2 Number of LOCK_GULM Servers 86 8 2 3 Starting LOCK_GULM Servers 86 8 2 4 Fencing and LOCK_GULM 86 8 2 5 Shutting Down a LOCK_GULM Server 86 8 3 LOCK_NOL...

Page 6: ...itch 116 10 2 2 WTI Network Power Switch 117 10 2 3 Brocade FC Switch 117 10 2 4 Vixel FC Switch 117 10 2 5 HP RILOE Card 118 10 2 6 GNBD 118 10 2 7 Manual 118 11 Using GNBD 121 11 1 GNBD Driver and C...

Page 7: ...OCK_GULM SLM Embedded 148 C 3 1 Key Characteristics 148 C 3 2 Kernel Modules Loaded 150 C 3 3 Setup Process 150 C 4 LOCK_GULM SLM External 153 C 4 1 Key Characteristics 154 C 4 2 Kernel Modules Loaded...

Page 8: ......

Page 9: ...presented this way include the following command Linux commands and other operating system commands when used are represented this way This style should indicate to you that you can type the word or p...

Page 10: ...n menu If you click on the word on the GUI screen the rest of the menu should appear For example Under File on a GNOME terminal the New Tab option allows you to open multiple shell prompts in the same...

Page 11: ...ionally we use several different strategies to draw your attention to certain pieces of information In order of how critical the information is to your system these items are marked as a note tip impo...

Page 12: ...n and the support documenta tion included in your subscription you must activate your subscription by registering with Red Hat Registration includes these simple steps Provide a Red Hat login Provide...

Page 13: ...Enter the following command from the command line as the root user usr bin up2date register 5 Recommended References For additional references about related topics refer to the following table Topic...

Page 14: ...Applications and High Availability Blueprints for High Availability Designing Resilient Distributed Systems by E Marcus and H Stern Wiley 2000 Provides a summary of best practices in high availabilit...

Page 15: ...d Changed Features with the Initial Release of Red Hat GFS 6 0 File access control lists ACLs and extended file attributes in GFS file systems This release adds the capability of setting and getting f...

Page 16: ...r Suite For GFS upgrade instructions refer to Appendix B Upgrading GFS 1 2 Performance Scalability and Economy You can deploy GFS in a variety of configurations to suit your needs for performance scal...

Page 17: ...ients as block storage devices by GNBD servers From the perspective of a client application storage is accessed as if it were directly attached to the server in which the application is running Stored...

Page 18: ...SAN 1 2 3 Economy and Performance Figure 1 3 shows how Linux client applications can take advantage of an existing Ethernet topology to gain shared access to all block storage devices Client data fil...

Page 19: ...configuration management 1 3 1 Cluster Volume Management Cluster volume management provides simplified management of volumes and the ability to dynam ically extend file system capacity without interru...

Page 20: ...s for more information on cluster management Fencing is the ability to isolate or fence off a cluster node when that node loses its heartbeat no tification with the rest of the cluster nodes Fencing e...

Page 21: ...re ccs_servd CCS server daemon that distributes CCS data from a single server to ccsd daemons when a shared device is not used for storing CCS data Fence fence_node Command used by lock_gulmd when a f...

Page 22: ...LOCK_GULM lock_gulm o Kernel module that is installed on GFS nodes using the LOCK_GULM lock module lock_gulmd Server daemon that runs on each node and communicates with all nodes in GFS cluster gulm_...

Page 23: ...ame later in this book The cluster name can be 1 to 16 characters long For example this book uses a cluster name alpha in some example configuration procedures Number of file systems Determine how man...

Page 24: ...ook for the CCS files fence ccs and nodes ccs Refer to Section 6 6 Creating the fence ccs File and Section 6 7 Creating the nodes ccs File for more information To help determine the type of fencing me...

Page 25: ...e Architecture RAM Red Hat Enterprise Linux AS ES or WS Version 3 Update 2 or later ia64 x86 64 x86 SMP supported 256 MB minimum Table 2 1 Platform Requirements 2 2 TCP IP Network All GFS nodes must b...

Page 26: ...array or JBOD Note Make sure that the devices can operate reliably when heavily accessed simultaneously from multiple initiators Note Make sure that your GFS configuration does not exceed the number...

Page 27: ...he key characteristics of your GFS configuration refer to Section 1 5 Before Configuring GFS and have completed the following tasks Installed prerequisite software Specified a persistent major number...

Page 28: ...ith the HP RILOE card refer to HP RILOE Card on page 147 Verify that the utility is installed on each of those nodes by looking for usr sbin stunnel The Stunnel utility is available via up2date 3 1 2...

Page 29: ...modules RPM is for an SMP or hugemem kernel GFS modules smp 6 0 2 20 1 i686 rpm The GFS tools RPM is common to all kernels 2 At each node install the GFS tools RPM using the rpm Uvh command For examp...

Page 30: ...ULM If you are using LOCK_NOLOCK refer to Appendix C Basic GFS Examples for information about which GFS kernel modules you should load To load the GFS kernel modules follow these steps 1 Run depmod a...

Page 31: ...Using Red Hat GFS with Red Hat Cluster Suite 4 1 Prerequisite Tasks Before setting up the GFS software make sure that you have noted the key characteristics of your GFS configuration refer to Section...

Page 32: ...e for a New Volume b Create a pool for each file system Refer to Section 5 5 Creating a Pool Volume Command usage pool_tool c ConfigFile 2 Create a CCS pool a Create pool configuration file Refer to S...

Page 33: ...age ccsd d CCADevice Note You can use GFS init d scripts included with GFS to automate starting and stopping the Cluster Configuration System For more information about GFS init d scripts refer to Cha...

Page 34: ...mount t gfs o acl BlockDevice MountPoint The o acl mount option allows manipulating file ACLs If a file system is mounted without the o acl mount option users are allowed to view ACLs with getfacl bu...

Page 35: ...t presents physical storage devices such as disks or RAID ar rays as logical volumes to GFS cluster nodes Pool can aggregate storage devices either by concate nating the underlying storage or by strip...

Page 36: ...umber and printing a pool configuration act on existing pools and require one or more pool names to be specified on the command line Flag Function Section Page Reference c Create a pool Section 5 5 Cr...

Page 37: ...olume Table 5 3 pool_assemble Command Functions Flag Option D Enable debugging output h Help Show usage information q Be quiet Do not display output from the command V Display command version informat...

Page 38: ...ommand version information then exit v Verbose operation Table 5 6 pool_info Command Options 5 2 4 pool_mp The pool_mp command is for managing multipathing on running pools refer to Table 5 7 and Tabl...

Page 39: ...ool label or an unknown label on them Note The pool_tool s command does not detect ondisk labels other than those mentioned in the pre ceding paragraph 5 3 1 Usage pool_tool s 5 3 2 Example In this ex...

Page 40: ...extension poolname cfg For example the pool configuration file for a pool named pool0 would be defined in configuration file pool0 cfg Before creating a configuration file you can check to see what d...

Page 41: ...he device is to be added id is the device identifier Number the devices in order beginning with 0 device specifies the device node to be used for example dev sda1 Table 5 9 Pool Configuration File Key...

Page 42: ...at defines the pool 5 5 2 Example In this example the pool0 cfg file describes the new pool pool0 created by the command pool_tool c pool0 cfg 5 5 3 Comments Multiple pools can be created with one poo...

Page 43: ...ames are specified all pools visible to the system are activated Deactivating a Pool Volume pool_assemble r PoolName PoolName Specifies the pool to deactivate More than one name can be listed If no po...

Page 44: ...Usage pool_tool p PoolName PoolName Specifies the pool name s for which to display information If no pool names are specified all active pools are displayed 5 7 2 Example In this example the pool_tool...

Page 45: ...ce 0 1 dev sdc1 pooldevice 0 2 dev sdd1 pooldevice 0 3 dev sde1 2 Edit the new file pool0 new cfg by adding one or more subpools that contain the devices or partitions as indicated in this example poo...

Page 46: ...pool names are specified all pools are erased 5 9 2 Example This example erases all disk labels for pool0 pool_tool e pool0 5 9 3 Comments The O override flag bypasses the confirmation step 5 10 Renam...

Page 47: ...ber of a pool 5 11 1 Usage pool_tool m Number PoolName Number Specifies the new minor number to be used PoolName Specifies the name of the pool to be changed The minor number must have a value between...

Page 48: ...displays the following basic information about the named pool s the pool name the minor number the device node alias the capacity whether or not the pool is being used and the multipathing type Using...

Page 49: ...l_info command with the c option clears statistics from the named pools 5 13 1 Usage Display the Number of Reads and Writes pool_info s PoolName PoolName Specifies the pool name for which to display i...

Page 50: ...d robin stripe value PoolName Specifies the pool on which to adjust multipathing If no pool names are specified this action is attempted on all active pools Reintegrate Failed Paths pool_mp r PoolName...

Page 51: ...GFS with Red Hat Cluster you can create CCS files with GFS Druid For information about configuring and using GFS with Red Hat Cluster Suite refer to Appendix A Using Red Hat GFS with Red Hat Cluster S...

Page 52: ...y For each node having multiple paths to FC storage include in the fencing method section a fencing device for each path to FC storage GFS supports dual power supply fencing with the APC MasterSwitch...

Page 53: ...10 Using the Fencing System 6 5 Creating the cluster ccs File Creating the cluster ccs file consists of specifying the following parameters Cluster name Each node that runs LOCK_GULM server Optional...

Page 54: ...lt value of Seconds is 15 To ensure that nodes respond within the Seconds value the interval for heartbeats sent by all nodes is automatically set to two thirds of the Seconds parameter value The Seco...

Page 55: ...evices in the fence ccs file APC MasterSwitch WTI NPS Network Power Switch Brocade FC Fibre Channel switch McData FC switch Vixel FC switch GNBD HP RILOE card xCAT Egenera BladeFrame system Manual War...

Page 56: ...le 6 2 for a fence ccs file that specifies an APC MasterSwitch fencing device b For each WTI NPS fencing device specify the following parameters DeviceName the fencing agent agent as fence_wti IPAddre...

Page 57: ...ers DeviceName the fencing agent agent as fence_xcat and RpowerBinaryPath Refer to Example 6 10 for a fence ccs file that specifies an xCAT fencing device i For each Egenera BladeFrame fencing device...

Page 58: ...ile Structure fence_devices fence_wti fence_devices DeviceName agent fence_brocade ipaddr IPAddress login LoginName passwd LoginPassword DeviceName Figure 6 4 File Structure fence_devices fence_brocad...

Page 59: ...ices fence_vixel fence_devices DeviceName agent fence_gnbd server ServerName server ServerName DeviceName Figure 6 7 File Structure fence_devices fence_gnbd without GNBD Multipath fence_devices Device...

Page 60: ...fence_devices fence_rib fence_devices DeviceName agent fence_xcat rpower RpowerBinaryPath DeviceName Figure 6 10 File Structure fence_devices fence_xcat fence_devices DeviceName agent fence_egenera c...

Page 61: ...n to a power switch an FC switch or a RILOE card multipath Selects GNBD multipath style fencing CAUTION When multipath style fencing is used if the gnbd_servd process of a GNBD server node cannot be c...

Page 62: ...Named apc1 and apc2 fence_devices wti1 agent fence_wti ipaddr 10 0 3 3 passwd password wti2 agent fence_wti ipaddr 10 0 3 4 passwd password Example 6 3 WTI NPS Fencing Devices Named wti1 and wti2 fen...

Page 63: ...Fencing Devices Named mdfc1 and mdfc2 fence_devices vixel1 agent fence_vixel ipaddr 10 0 3 3 passwd password vixel2 agent fence_vixel ipaddr 10 0 3 4 passwd password Example 6 6 Vixel FC Switch Fenci...

Page 64: ...number of retries to 5 with retrys 5 and sets the wait time between retries to 3 with wait_time 3 Example 6 8 GNBD Fencing Device Named gnbdmp with GNBD Multipath fence_devices riloe1 agent fence_rib...

Page 65: ...ode must specify at least two fencing devices one fencing device for each power supply Similarly if a node has multiple paths to FC storage then the fencing method for the node must specify one fencin...

Page 66: ...ode needs recovery Cluster operation is halted during the intervention b If you are configuring a node for cascaded fencing use the file format in Figure 6 22 Refer to Table 6 3 for syntax description...

Page 67: ...d PortNumber Refer to Example 6 16 for a nodes ccs file that specifies Brocade FC switch fencing Refer to Example 6 17 for a nodes ccs file that specifies McData FC switch fencing Refer to Example 6 1...

Page 68: ...es IFNAME IPAddress fence MethodName DeviceName port PortNumber switch SwitchNumber NodeName File format for node identification same format for all nodes File format for APC MasterSwitch fencing meth...

Page 69: ...fencing method for node with dual power supplies Fencing a node with dual power supplies requires that both power supplies be powered off before rebooting the node To accomplish that the nodes ccs fi...

Page 70: ...NodeName NodeName ip_interfaces IFNAME IPAddress fence MethodName DeviceName port PortNumber NodeName File format for node identification same format for all nodes File format for WTI NPS fencing meth...

Page 71: ...ame port PortNumber DeviceName port PortNumber NodeName File format for node identification same format for all nodes File format for Brocade McData or Vixel FC Switch fencing method Additional fencin...

Page 72: ...es NodeName NodeName ip_interfaces IFNAME IPAddress fence MethodName DeviceName ipaddr IPAddress NodeName File format for node identification same format for all nodes File format for GNBD fencing met...

Page 73: ...eName NodeName ip_interfaces IFNAME IPAddress fence MethodName DeviceName localport PortNumber NodeName File format for node identification same format for all nodes File format for HP RILOE fencing m...

Page 74: ...des NodeName NodeName ip_interfaces IFNAME IPAddress fence MethodName DeviceName nodename NodelistName NodeName File format for node identification same format for all nodes File format for xCAT fenci...

Page 75: ...e ip_interfaces IFNAME IPAddress fence MethodName DeviceName lpan LPANName pserver PserverName NodeName File format for node identification same format for all nodes File format for Egenera BladeFrame...

Page 76: ...odeName File format for node identification same format for all nodes File format for manual fencing method Figure 6 21 File Format nodes ccs Manual Fencing Method Warning Manual fencing should not be...

Page 77: ...odeName ip_interfaces IFName IPAddress fence MethodName DeviceName Device specific parameter s MethodName DeviceName Device specific parameter s MethodName DeviceName Device specific parameter s NodeN...

Page 78: ...le only with Red Hat GFS 6 0 for Red Hat Enterprise Linux 3 Update 5 and later For the fence section If GNBD fencing The IP address of this node the node to be fenced If manual fencing IP address of t...

Page 79: ...verify the IP address associated with the hostname you can issue a ping command to the hostname PortNumber For power and FC switches The port number on the switch to which this node is connected For H...

Page 80: ...nce power apc1 Fencing device for power supply 1 port 6 switch 1 option off Power down power supply 1 apc2 Fencing device for power supply 2 port 7 switch 2 option off Power down power supply 2 apc1 F...

Page 81: ...i0 10 0 0 1 fence power wti1 port 1 n02 Example 6 15 Node Defined for WTI NPS Fencing nodes n01 ip_interfaces hsi0 10 0 0 1 fence san silkworm1 port 3 silkworm2 Additional fencing device for additiona...

Page 82: ...fc2 Additional fencing device for additional port 4 path to FC storage n02 Example 6 17 Node Defined for McData FC Switch Fencing nodes n01 ip_interfaces hsi0 10 0 0 1 fence san vixel1 port 3 vixel2 A...

Page 83: ...Files 69 nodes n01 ip_interfaces hsi0 10 0 0 1 fence server gnbd ipaddr 10 0 1 1 n02 Example 6 19 Node Defined for GNBD Fencing nodes n01 ip_interfaces hsi0 10 0 0 1 fence riloe riloe1 localport 2345...

Page 84: ...er Configuration System Files nodes n01 ip_interfaces hsi0 10 0 0 1 fence blade center xcat nodename blade 01 n02 ip_interfaces hsi0 10 0 0 2 fence blade center xcat nodename blade 02 n03 Example 6 21...

Page 85: ...enter egenera lpan opsgroup pserver ops 1 n02 ip_interfaces hsi0 10 0 0 2 fence blade center egenera lpan opsgroup pserver ops 2 n03 Example 6 22 Nodes Defined for Egenera BladeFrame Fencing nodes n01...

Page 86: ..._interfaces eth0 10 0 1 21 fence san Fencing with Brocade FC switch brocade1 port 1 power Fencing with APC MasterSwitch apc port 1 switch 1 n02 This example shows a node that can be fenced using a Bro...

Page 87: ...wer APC MasterSwitch fencing device apc1 port 6 switch 2 n02 Example 6 25 GNBD Server Node Defined for APC Fencing Single Power Supply nodes n01 ip_interfaces wizzy 10 0 0 1 Must be an IP address not...

Page 88: ...74 Chapter 6 Creating the Cluster Configuration System Files...

Page 89: ...are some considerations you must take into account for the location of CCS files Refer to Section 11 2 Considerations for Using GNBD Multipath 7 1 Creating a CCS Archive A CCS archive is a collection...

Page 90: ...on the size of the device it may take several seconds to create a CCA device for the first time due to initialization of the device The ccs_tool command uses the Linux raw device interface to update...

Page 91: ...odes 7 3 Using Other CCS Administrative Options The following sections detail other administrative functions that can be performed by the ccs_tool command 7 3 1 Extracting Files from a CCS Archive Whe...

Page 92: ...to a CCS Archive The ccs_tool diff command can be used to compare a directory of CCS configuration files with the configuration files in a CCS archive The output from the ccs_tool diff command is disp...

Page 93: ...ration files in root alpha new 3 Create a new CCS archive on the CCA device by using the O override flag to forcibly over write the existing CCS archive ccs_tool O create root alpha new dev pool alpha...

Page 94: ...er supplied variable that specifies the name of the cluster The CCA file must be located on the node that runs ccs_servd 7 5 1 1 1 Usage ccs_tool create Directory CCAFile Directory The relative path t...

Page 95: ...e network and requires two parameters on the ccsd command line the IP address and optional port number of the node running ccs_servd and the name of the cluster 7 5 1 3 1 Usage ccsd s IPAddress PortNu...

Page 96: ...ing a CCS Archive and Section 7 2 Starting CCS in the Cluster The advantages are that there is no server point of failure and that updates to the CCS archive happen atomically However not every cluste...

Page 97: ...Chapter 7 Using the Cluster Configuration System 83 Note When you update a CCS archive update the shared device archive first then update the local archives Be sure to keep the archives synchronized...

Page 98: ...84 Chapter 7 Using the Cluster Configuration System...

Page 99: ...GFS to be used as a local file system 8 2 LOCK_GULM RLM and SLM are both implemented by the LOCK_GULM system LOCK_GULM is based on a central server daemon that manages lock and cluster state for all G...

Page 100: ...m number of nodes required is three the maximum number of nodes is five 8 2 3 Starting LOCK_GULM Servers If no lock_gulmd servers are running in the cluster take caution before restarting them you mus...

Page 101: ...the IP address or hostname of the node running the instance of lock_gulmd to be terminated 8 3 LOCK_NOLOCK The LOCK_NOLOCK system allows GFS to be used as a local file system on a single node The kern...

Page 102: ...88 Chapter 8 Using Clustering and Locking Systems...

Page 103: ...on 9 14 Shutting Down a GFS Cluster Section 9 15 Starting a GFS Cluster 9 1 Making a File System Making a GFS file system is one of the final tasks in the process of configuring and setting up a GFS c...

Page 104: ...be created by the gfs_mkfs command One journal is re quired for each node that mounts the file system More journals can be specified to allow for easier future expansion BlockDevice Usually specifies...

Page 105: ...he standard GFS locking module LOCK_NOLOCK May be used when GFS is acting as a local file system one node only O Prevents the gfs_mkfs command from asking for confirmation before writing the file syst...

Page 106: ...le system resides MountPoint Specifies the directory where the GFS file system should be mounted 9 2 2 Example In this example the GFS file system on the pool0 block device is mounted on the gfs1 dire...

Page 107: ...ion should not be used when GFS file systems are shared Forces GFS to treat the file system as a multihost file system By default using LOCK_NOLOCK automatically turns on the localcaching and localflo...

Page 108: ...Other Linux quota facilities cannot be used with GFS 9 4 1 Setting Quotas Two quota settings are available for each user ID UID or group ID GID a hard limit and a warn limit A hard limit is the amoun...

Page 109: ...fs_quota warn g 21 l 50 k f gfs 9 4 2 Displaying Quota Limits and Usage Quota limits and current usage can be displayed for a specific user or group using the gfs_quota get command The entire contents...

Page 110: ...rd limit set for the user or group This value is zero if no limit has been set Value The actual amount of disk space used by the user or group 9 4 2 3 Comments When displaying quota information the gf...

Page 111: ...ges to the quota_quantum parameter are not persistent across unmounts You can use the gfs_quota sync command to synchronize the quota information from a node to the on disk quota file between the auto...

Page 112: ...with a value of 1 instead of 0 as the final command line parameter Even when GFS is not enforcing quotas it still keeps track of the file system usage for all users and groups so that quota usage info...

Page 113: ...tune gfs quota_account 1 gfs_quota init f gfs 9 5 Growing a File System The gfs_grow command is used to expand a GFS file system after the device where the file system resides has been expanded Runnin...

Page 114: ...Pool Volume for additional information After running the gfs_grow command run a df command to check that the new space is now avail able in the file system 9 5 3 Examples In this example the file sys...

Page 115: ...un on a mounted file system but it only needs to be run on one node in the cluster All the other nodes sense that the expansion has occurred To verify that the changes were successful you can use the...

Page 116: ...fies the size of the new journals in MBytes Default journal size is 128 MBytes The minimum size is 32 MBytes To add journals of different sizes to the file system the gfs_jadd command must be run for...

Page 117: ...r written to must also be 512 byte aligned One of the following methods can be used to enable direct I O on a file O_DIRECT GFS file attribute GFS directory attribute 9 7 1 O_DIRECT If an application...

Page 118: ...erit_directio flag is set 9 7 3 2 Example In this example the command sets the inherit_directio flag on the directory named gfs1 data gfs_tool setflag inherit_directio gfs1 data 9 8 Data Journaling Or...

Page 119: ...ory Specifies the directory where the flag is set or cleared File Specifies the zero length file where the flag is set or cleared 9 8 2 Examples This example shows setting the inherit_jdata flag on a...

Page 120: ...ce MountPoint o noatime BlockDevice Specifies the block device where the GFS file system resides MountPoint Specifies the directory where the GFS file system should be mounted 9 9 1 2 Example In this...

Page 121: ...ntPoint atime_quantum Seconds MountPoint Specifies the directory where the GFS file system is mounted Seconds Specifies the update period in seconds 9 9 2 2 Examples In this example all GFS tunable pa...

Page 122: ...tool command Typical usage of the gfs_tool command is described here 9 11 1 Usage Displaying Statistics gfs_tool counters MountPoint The counters action flag displays statistics about a file system If...

Page 123: ...ng the gfs_fsck command The gfs_fsck command must only be run on a file system that is unmounted from all nodes Note On nodes running Red Hat GFS 6 0 for Red Hat Enterprise Linux 3 Update 5 and later...

Page 124: ...he link depends on the node or user following the link CDPN variables can be used in any path name not just with symbolic links However the CDPN variable name cannot be combined with other characters...

Page 125: ...duced by the following command entry echo uname m _ uname s uid This variable resolves to a real file or directory named with the user ID string produced by the following command entry echo id u gid T...

Page 126: ...wn a LOCK_GULM Server for more information 3 Kill the CCS daemon on all nodes 4 Deactivate all pools on all nodes Refer to Section 5 6 Activating Deactivating a Pool Volume for more information Note Y...

Page 127: ...le 2 At each node start the CCS daemon specifying the CCA device on at the command line Refer to Section 7 2 Starting CCS in the Cluster for more information Command usage ccsd d CCADevice 3 Start the...

Page 128: ...114 Chapter 9 Managing GFS...

Page 129: ...f the nodes in the cluster Eventually that condition may bring the entire cluster to a halt as other nodes require access to those parts of the file system If a node fails it cannot be permitted to re...

Page 130: ...orized Red Hat support representative if there is a device you wish to use for fencing that is not described in the following sections The following sections describe the fencing methods available wit...

Page 131: ...itch and disables the specific port associated with the node Nodes with multiple FC paths can have each path disabled Refer to Section 6 6 Creating the fence ccs File and Section 6 7 Creating the node...

Page 132: ...al hardware The fence_gnbd fencing agent instructs all GNBD servers to disallow all I O from a fenced node When a fenced node is reset and re imports the GNBD devices the GNBD servers again allow the...

Page 133: ...istrator must run the command fence_ack_manual to indicate to the system that the failed node has been reset Recovery of the reset node will then proceed Refer to Section 6 6 Creating the fence ccs Fi...

Page 134: ...120 Chapter 10 Using the Fencing System...

Page 135: ...le clients forms a shared storage configuration through which GFS can be used The GNBD driver is implemented through the following client and server kernel modules gnbd o Implements the GNBD device dr...

Page 136: ...re stored locally or on FC attached storage c Enable caching Reads from the exported GNBD and takes advantage of the Linux page cache By default the gnbd_export command does not enable caching Caution...

Page 137: ...BD server by hostname or IP address from which to import GNBDs All GNBDs exported from the server are imported on the client running this command 11 1 2 2 Example This example imports all GNBDs from t...

Page 138: ...side on local storage or FC attached storage Because lock servers need to start before GNBD servers can start a lock server cannot access CCS files through a GNBD If a lock server is running on a GFS...

Page 139: ...he following fencing devices APC MasterSwitch fence_apc fence agent WTI NPS fence_wti fence agent Brocade FC switch fence_brocade fence agent McData FC switch fence_mcdata fence agent Vixel FC switch...

Page 140: ...d to mount a GFS file system The GNBD server node must not import gnbd_import command other GNBD devices to run the file system 2 The GNBD server must export all the GNBDs in uncached mode and it must...

Page 141: ...ovides the following init d scripts that are installed automatically when GFS is installed pool ccsd lock_gulmd gfs The scripts automatically start and stop GFS modules during startup and shutdown of...

Page 142: ...s all the devices and loads any pool that it finds To specify the pools on which to operate the variable POOLS must be set in etc sysconfig gfs You can define multiple pools by separating the pool nam...

Page 143: ...e that includes the GFS file system trin1 gfs dev pool trin1 gfs gfs gfs defaults 0 0 If you do not want a GFS file system to automatically mount on startup add noauto to the options in the ect fstab...

Page 144: ...130 Chapter 12 Using GFS init d Scripts...

Page 145: ...llowing at the command line redhat config gfscluster gulm bridge This is a fence method available for Red Hat Cluster nodes if and only if the Red Hat GFS RPM is installed on the node that the Cluster...

Page 146: ...des for configuring an initial instance of Red Hat GFS by writing the three Red Hat GFS configuration files cluster ccs nodes ccs and fence ccs The GFS Setup Druid requires an etc cluster xml file whe...

Page 147: ...our deployment you may want to consider using multiple Red Hat Cluster Manager clusters A 3 3 Upgrading Red Hat GFS 5 2 1 to Red Hat GFS 6 0 To upgrade Red Hat GFS 5 2 1 to Red Hat GFS 6 0 follow the...

Page 148: ...134 Appendix A Using Red Hat GFS with Red Hat Cluster Suite...

Page 149: ...wing actions a Unmount GFS file systems from all nodes b Stop lock servers c Stop ccsd on all nodes d Deactivate pools e Unload kernel modules 2 Install new software This step consists of the followin...

Page 150: ...described in Step 5 Command usage ccs_tool create Directory Device Reference Section 7 1 Creating a CCS Archive Example ccs_tool create root alpha dev pool alpha_cca 7 Start ccsd on all nodes This in...

Page 151: ...ULM servers This section provides the following information about the example Section C 1 1 Key Characteristics Section C 1 2 Kernel Modules Loaded Section C 1 3 Setup Process C 1 1 Key Characteristic...

Page 152: ...device driver is loaded and that it loaded without errors The small partition dev sda1 is used to store the cluster configuration information The two re maining partitions dev sda2 sdb1 are used for t...

Page 153: ...ing poolname alpha_cca subpools 1 subpool 0 0 1 pooldevice 0 0 dev sda1 3 Use the pool_tool command to create all the pools as follows n01 pool_tool c pool_gfs01 cf pool_gfs02 cf alpha_cca cf Pool lab...

Page 154: ...ip_interfaces eth0 10 0 1 2 fence power apc port 2 n03 ip_interfaces eth0 10 0 1 3 fence power apc port 3 Note If your cluster is running Red Hat GFS 6 0 for Red Hat Enterprise Linux 3 Update 5 and la...

Page 155: ...is rebooted The CCA device must be specified when starting ccsd n01 ccsd d dev pool alpha_cca n02 ccsd d dev pool alpha_cca n03 ccsd d dev pool alpha_cca 8 At each node start the LOCK_GULM server n01...

Page 156: ...ning LOCK_GULM only This section provides the following information about the example Section C 2 1 Key Characteristics Section C 2 2 Kernel Modules Loaded Section C 2 3 Setup Process C 2 1 Key Charac...

Page 157: ...ary to load an appropriate device driver If the shared storage devices are not visible on each node confirm that the device driver is loaded and that it loaded without errors The small partition dev s...

Page 158: ...the pool that will be used for CCS data The pool does not need to be very large The name of the pool will be alpha_cca The name of the cluster alpha followed by _cca The file should look like the foll...

Page 159: ...assembled pool_gfs02 assembled 5 Create CCS files a Create a directory called root alpha on node n01 as follows n01 mkdir root alpha n01 cd root alpha b Create the cluster ccs file This file contains...

Page 160: ...les n03 ip_interfaces eth0 10 0 1 3 fence power apc port 3 lck01 ip_interfaces eth0 10 0 1 4 fence power apc port 4 lck02 ip_interfaces eth0 10 0 1 5 fence power apc port 5 lck03 ip_interfaces eth0 10...

Page 161: ...gent fence_apc ipaddr 10 0 1 10 login apc passwd apc 6 Create the CCS Archive on the CCA Device Note This step only needs to be done once and from a single node It should not be performed every time t...

Page 162: ...Protocol lock_gulm Lock Table alpha gfs02 Syncing All Done 10 Mount the GFS file systems on all the nodes Mount points gfs01 and gfs02 are used on each node n01 mount t gfs dev pool pool_gfs01 gfs01...

Page 163: ...r to Table C 10 for storage device information Number of file systems 2 File system names gfs01 and gfs02 File system mounting Each GFS node mounts the two file systems Cluster name alpha Host Name IP...

Page 164: ...example consists of the following steps 1 Create pool configurations for the two file systems Create pool configuration files for each file system s pool pool_gfs01 for the first file system and pool_...

Page 165: ...assembled pool_gfs01 assembled pool_gfs02 assembled 5 Create CCS files a Create a directory called root alpha on node n01 as follows n01 mkdir root alpha n01 cd root alpha b Create the cluster ccs fil...

Page 166: ...for syntax description of the usedev parameter d Create the fence ccs file This file contains information required for the fencing method s used by the GFS cluster The file should look like the follow...

Page 167: ...ool pool_gfs01 Device dev pool pool_gfs01 Blocksize 4096 Filesystem Size 1963216 Journals 3 Resource Groups 30 Locking Protocol lock_gulm Lock Table alpha gfs01 Syncing All Done n01 gfs_mkfs p lock_gu...

Page 168: ...3 Refer to Table C 12 for node information Number of lock server nodes 1 The lock server is run on one of the GFS nodes embedded Refer to Table C 13 for node information Locking protocol LOCK_GULM Th...

Page 169: ...mmand cat proc partitions Depending on the hardware configuration of the GFS nodes the names of the devices may be different on each node If the output of the cat proc partitions command shows only en...

Page 170: ...nodes Note This step must be performed every time a node is rebooted If it is not the pool devices will not be accessible Activate the pools using the pool_assemble a command for each node as follows...

Page 171: ...ess and node specific I O fencing parameters The file should look like the following nodes n01 ip_interfaces eth0 10 0 1 1 fence power apc port 1 n02 ip_interfaces eth0 10 0 1 2 fence power apc port 2...

Page 172: ...CCS Archive on the CCA Device Note This step only needs to be done once and from a single node It should not be performed every time the cluster is restarted Use the ccs_tool command to create the arc...

Page 173: ...mount t gfs dev pool pool_gfs01 gfs01 n02 mount t gfs dev pool pool_gfs02 gfs02 n03 mount t gfs dev pool pool_gfs01 gfs01 n03 mount t gfs dev pool pool_gfs02 gfs02 C 5 LOCK_GULM SLM External and GNBD...

Page 174: ...transport layer for the storage devices Refer to Table C 19 for storage device information Number of file systems 2 File system names gfs01 and gfs02 File system mounting Each GFS node mounts the two...

Page 175: ...n at each node in your GFS cluster by running the follow ing command cat proc partitions Depending on the hardware configuration of the GFS nodes the names of the devices may be different on each node...

Page 176: ...files should look like the following poolname pool_gfs01 subpools 1 subpool 0 0 1 pooldevice 0 0 dev gnbd gfs01 poolname pool_gfs02 subpools 1 subpool 0 0 1 pooldevice 0 0 dev gnbd gfs02 4 Create a p...

Page 177: ...ssembled lcksrv pool_assemble a Activate pools alpha_cca assembled pool_gfs01 assembled pool_gfs02 assembled 7 Create CCS files a Create a directory called root alpha on node n01 as follows n01 mkdir...

Page 178: ...Update 5 and later you can use the optional usedev parameter to explicitly specify an IP address rather than relying on an IP address from libresolv For more information about the optional usedev par...

Page 179: ...CA device must be specified when starting ccsd n01 ccsd d dev pool alpha_cca n02 ccsd d dev pool alpha_cca n03 ccsd d dev pool alpha_cca lcksrv ccsd d dev pool alpha_cca 10 At each node start the LOCK...

Page 180: ...ng two GFS file systems Only a single node is required because the file system will not be mounted in cluster mode This section provides the following information about the example Section C 6 1 Key C...

Page 181: ...on each node If the output of the cat proc partitions command shows only entire disk devices for example dev sda instead of dev sda1 then the storage devices have not been partitioned To partition a...

Page 182: ...y needs to be done once It should not be performed every time the cluster is restarted Use the ccs_tool command to create the archive from the CCS configuration files n01 ccs_tool create root alpha ro...

Page 183: ...pool pool_gfs02 Blocksize 4096 Filesystem Size 1963416 Journals 1 Resource Groups 30 Locking Protocol lock_nolock Lock Table Syncing All Done 7 Mount the GFS file systems on the nodes Mount points gfs...

Page 184: ...170 Appendix C Basic GFS Examples...

Page 185: ...and server 79 local CCA files 82 changing CCS configuration files 79 combining CCS methods 82 creating a CCS archive 75 starting CCS in the cluster 76 using 75 using clustering and locking systems 85...

Page 186: ...rectory attribute 104 file attribute 103 O_DIRECT 103 growing 99 making 89 mounting 91 quota management 94 disabling enabling quota accounting 98 disabling enabling quota enforcement 98 displaying quo...

Page 187: ...tware 13 installation tasks 14 GFS RPM installation 15 loading GFS kernel modules 15 prerequisite tasks 13 clock synchronization software 14 perl Net Telnet module 13 persistent major number utility 1...

Page 188: ...troduction prerequisite tasks configuration initial 17 installing system software 13 clock synchronization software 14 perl Net Telnet module 13 persistent major number 14 stunnel utility 14 Q quota m...

Page 189: ...on examples 160 lock server node information examples 143 154 160 mount options 93 nodes css variables 64 platform requirements 11 pool configuration file keyword and variable de scriptions 26 pool_as...

Page 190: ......

Page 191: ...d Hat Enterprise Linux Installation Guide for x86 Itanium AMD64 and Intel Extended Memory 64 Technology Intel EM64T Karsten Wade Primary Writer Maintainer of the Red Hat SELinux Application Developmen...

Page 192: ...8 Nadine Richter German translations Audrey Simons French translations Francesco Valente Italian translations Sarah Wang Simplified Chinese translations Ben Hung Pin Wu Traditional Chinese translation...

Search results

Summary of Contents for GFS 6.0 -

Reviews:

Related manuals for GFS 6.0 -

Brands by name

Popular brands

Red Hat GFS 6.0 -, Administrator'S Manual

Search results

Summary of Contents for GFS 6.0 -

Reviews:

Related manuals for GFS 6.0 -

Brands by name

Popular brands