manualshive.com logo in svg

HP Fabric OS 7.0.0, Troubleshooting Manual, Page: 88 / 138

Share
Download
HP Fabric OS 7.0.0 Troubleshooting Manual Download Page 88

68

Fabric OS Troubleshooting and Diagnostics Guide

53-1002150-02

Fabric identification issues

7

•

VE_Ports cannot exist in a logical switch that has XISL use turned on. Although VE_Ports are 
allowed in a base switch, Fabric OS v6.2.0 and later do not support the use of VE_Ports to 
carry traffic for logical fabrics using XISLs. They can be used to carry FCR traffic through EX_ 
and VEX_Ports. You should make sure your configuration does not result in the use of VE_Ports 
in a base switch for logical fabric traffic.

•

Admin Domains are mutually exclusive with Virtual Fabrics. When Virtual Fabrics is enabled, all 
access control is based on the Virtual Fabric context.

•

Traffic Isolation zones with no-failover option are not supported in logical fabrics. TI zones 
defined in the base fabric for logical fabric traffic must allow failover. 

NOTE

A new option “Disable FID check” has been added to configure fabric parameter options. This can 
be used to disable FID check for FICON logical switches.

Fabric identification issues

Symptom

E_Ports directly connecting two logical switches do not form, or are disabled.

Probable cause and recommended action

The FIDs on each of the logical switches must be the same. 

Use the lsCfg 

--

show command to view the current FIDs on the chassis and then the lsCfg 

--

change 

FID

 -newfid 

FID 

command

 

to change the FID.

Symptom

Invalid FID. 

Probable cause and recommended action

FIDs for switches may be from 1 through 128 as long as they are not already in use, except 
EX_Ports which are only assigned FIDs from 1 through 127.

Use the lsCfg 

--

show command to verify if the FID is in use. If it is, use another FID.

Symptom

The FID is currently in use. 

Probable cause and recommended action

You may not create two (2) or more logical switches with the same FID. 

Use the lsCfg 

--

show and fcrFabricShow commands to view FIDs in use.

Logical Fabric issues

Symptom

Logical port <port_number> disabled.

Probable cause and recommended action

This message indicates an LISL was disabled due to some protocol conflict or security or policy 
violation. This can result in possible traffic issues. You should resolve the cause of the conflict and 
re-enable the LISL using the lfCfg 

--

lislenable command.

 

Summary of Contents for Fabric OS 7.0.0

Page 1: ...53 1002150 02 03 June 2011 Fabric OS Troubleshooting and Diagnostics Guide Supporting Fabric OS v7 0 0 ...

Page 2: ...s Inc shall have no liability or responsibility to any person or entity with respect to any loss cost liability or damages arising from the information contained in this book or the computer programs that accompany it The product described by this document may contain open source software covered by the GNU General Public License or other open source license agreements To find out which open sourc...

Page 3: ...pport for checking physical connections updated commands removed obsolete information and moved the FCIP and FICON chapters into their respective books July 2009 Fabric OS Troubleshooting and Diagnostics Guide 53 1001769 01 Added support for the Rolling Reboot Detection feature and the Superping tool added enhancements for supportSave and spinFab updated commands transferred the iSCSI chapter into...

Page 4: ...iv Fabric OS Troubleshooting and Diagnostics Guide 53 1002150 02 ...

Page 5: ...sources xviii Other industry resources xix Getting technical help xix Document feedback xx Chapter 1 Introduction In this chapter 1 Troubleshooting overview 1 Network time protocol 1 Most common problem areas 2 Questions for common symptoms 2 Gathering information for your switch support provider 5 Setting up your switch for FTP 5 Capturing a supportSave 5 Capturing output from a console 6 Capturi...

Page 6: ... a point to point initialization failure 29 Correcting a port that has come up in the wrong mode 30 Marginal links 30 Troubleshooting a marginal link 31 Device login issues 33 Pinpointing problems with device logins 33 Media related issues 35 Testing a port s external transmit and receive path 36 Testing a switch s internal components 36 Testing components to and from the HBA 36 Segmented fabrics ...

Page 7: ...55 Considerations for downgrading firmware 56 Preinstallation messages 56 Blade types 58 Firmware versions 59 Platform 59 Routing 60 Chapter 6 Security In this chapter 61 Passwords 61 Password recovery options 62 Device authentication 62 Protocol and certificate management 63 Gathering additional information 63 SNMP 64 Gathering additional information 64 FIPS 64 Chapter 7 Virtual Fabrics In this c...

Page 8: ...Channel connectivity problems 80 Checking for zoning problems 81 Gathering additional information 82 Chapter 10 Diagnostic Features In this chapter 83 About Fabric OS diagnostics 83 Diagnostic information 83 Power on self test 84 Disabling POST 86 Enabling POST 86 Switch status 86 Viewing the overall status of the switch 86 Displaying switch information 87 Displaying the uptime for a switch 88 Usi...

Page 9: ...sage log one message at a time 101 Clearing the system message log 101 Port log 101 Viewing the port log 101 Syslogd configuration 103 Configuring the host 103 Configuring the switch 104 Automatic trace dump transfers 105 Specifying a remote server 105 Enabling the automatic transfer of trace dumps 105 Setting up periodic checking of the remote server 105 Saving comprehensive diagnostic files to t...

Page 10: ...x Fabric OS Troubleshooting and Diagnostics Guide 53 1002150 02 ...

Page 11: ...nverged Enhanced Ethernet Administrator s Guide xi 53 1002163 01 Figures Figure 1 Superping and fcPing paths 18 Figure 2 Simple port initialization process 24 Figure 3 Example of a D_Port configuration 92 ...

Page 12: ...xii Converged Enhanced Ethernet Administrator s Guide 53 1002163 01 ...

Page 13: ...nections 42 Table 10 Brocade configuration and connection 48 Table 11 USB error handling 56 Table 12 Password recovery options 62 Table 13 SSL messages and actions 63 Table 14 Commands for debugging zoning 77 Table 15 Switch summary information 87 Table 16 VF output values 87 Table 17 switchShow command output 88 Table 18 Port type support 89 Table 19 Error summary description 98 Table 20 Commands...

Page 14: ...xiv Converged Enhanced Ethernet Administrator s Guide 53 1002163 01 ...

Page 15: ...es Chapter 4 Configuration provides troubleshooting information and procedures for configuration file issues Chapter 5 Firmware Download Errors provides procedures for troubleshooting firmware download issues Chapter 6 Security provides procedures for user account and security issues Chapter 7 Virtual Fabrics provides procedures for troubleshooting Virtual Fabrics Chapter 8 ISL Trunking provides p...

Page 16: ...re platforms are supported by this release of Fabric OS Brocade 300 switch Brocade 5100 switch Brocade 5300 switch Brocade 5410 embedded switch Brocade 5424 embedded switch Brocade 5450 embedded switch Brocade 5460 embedded switch Brocade 5470 embedded switch Brocade 5480 embedded switch Brocade 6510 switch Brocade 7800 extension switch Brocade 8000 FCoE switch Brocade VA 40FC Brocade Encryption S...

Page 17: ... switchShow In actual examples command lettercase is often all lowercase Otherwise this manual specifically notes those cases in which a command is case sensitive Command syntax in this manual follows these conventions Command examples This book describes how to perform configuration tasks using the Fabric OS command line interface but does not describe the commands in detail For complete descript...

Page 18: ...ons or situations that can be potentially lethal or extremely hazardous to you Safety labels are also attached directly to products to warn of these conditions or situations Key terms For definitions specific to Brocade and Fibre Channel refer to the Brocade Glossary For definitions of SAN specific terms visit the Storage Networking Industry Association online dictionary at http www snia org educa...

Page 19: ...t the Fibre Channel Industry Association website http www fibrechannel org Getting technical help Contact your switch support supplier for hardware firmware and software support including product repairs and part ordering To expedite your call have the following information available 1 General Information Switch model Switch operating system version Error numbers and messages received supportSave ...

Page 20: ...omb 3 World Wide Name WWN Use the licenseIdShow command to display the chassis WWN If you cannot use the licenseIdShow command because the switch is inoperable you can get the WWN from the same place as the serial number except for the Brocade DCX For the Brocade DCX access the numbers on the WWN cards by removing the Brocade logo plate at the top of the nonport side of the chassis Document feedba...

Page 21: ...minating the possibility of a fault within the fabric see if the problem is on the storage side or the host side and continue a more detailed diagnosis from there Using this approach can quickly pinpoint and isolate problems For example if a host cannot detect a storage device run the switchShow command to determine if the storage device is logically connected to the switch If not focus first on t...

Page 22: ...ng devices Marginal links unstable connections Incorrect zoning configurations Incorrect switch configurations Switch LEDs Switch commands for example switchShow or nsAllShow for diagnostics Web or GUI based monitoring and management software tools Storage Devices Physical issues between switch and devices Incorrect storage software configurations Device LEDs Storage diagnostic tools Switch comman...

Page 23: ... FCIP tunnel including the network between FCIP tunnel endpoints Fibre Channel over IP Administrator s Guide FCIP tunnel does not come online FCIP tunnel including the network between FCIP tunnel endpoints Fibre Channel over IP Administrator s Guide FCIP tunnel does not form Licensing Fabric parameters Chapter 2 General Fibre Channel over IP Administrator s Guide FCIP tunnel is sluggish FCIP tunne...

Page 24: ...ics SCSI retry errors Buffer credits FCIP tunnel bandwidth Fibre Channel over IP Administrator s Guide SCSI timeout errors Links HBA Buffer credits FCIP tunnel bandwidth Chapter 3 Connectivity Chapter 8 ISL Trunking Fibre Channel over IP Administrator s Guide Switch constantly reboots Rolling reboot detection FIPS Chapter 6 Security Switch is unable to join fabric Security policies Zoning Fabric p...

Page 25: ... account with admin permissions 2 Type the supportFtp command and respond to the prompts Example of supportFTP command switch admin supportftp s Host IP Addr 1080 8 800 200C 417A User Name njoe userFoo Password hidden Remote Dir support supportftp parameters changed Capturing a supportSave The supportSave command uses the default switch name to replace the chassis name regardless if the chassis na...

Page 26: ...support directory must be used While running the supportSave command you may encounter a timeout A timeout occurs if the system is in busy state due to CPU or I O bound from a lot of port traffic or file access If this occurs an SS 1004 is generated to both the console and the RASlog to report the error You must rerun the supportSave command with the t option Example of SS 1004 message SS 1004 One...

Page 27: ...tirety and be ready to send to your switch support provider when you contact them Having this information immediately available expedites the information gathering process that is necessary to begin determining the problem and finding a solution Basic information 1 What is the switch s current Fabric OS level To determine the switch s Fabric OS level type the firmwareShow command and write down th...

Page 28: ... List vendor and version 5 If you have a redundant fabric did a failover occur 6 Was POST enabled on the switch 7 Which CP blade was active Only applicable to Brocade DCX DCX 8510 family and DCX 4S enterprise class platforms Detailed problem information Obtain as much of the following informational items as possible prior to contacting the SAN technical support vendor Document the sequence of even...

Page 29: ...actions or changes made to the switch the fabric and the SAN or metaSAN In Table 3 list the environmental changes added to the network Gathering additional information Below are features that require you to gather additional information The additional information is necessary in order for your switch support provider to effectively and efficiently troubleshoot your issue Refer to the chapter or do...

Page 30: ...10 Fabric OS Troubleshooting and Diagnostics Guide 53 1002150 02 Building a case for your switch support provider 1 ...

Page 31: ...rect licenses installed by using the licenseShow command Symptom A feature is not working Probable cause and recommended action Refer to the Fabric OS Administrator s Guide to determine if the appropriate licenses are installed on the local switch and any connecting switches Determining installed licenses 1 Connect to the switch and log in using an account with admin permissions 2 Type the license...

Page 32: ...RAS logs is necessary to prevent minor problems from becoming larger issues or in catching problems at an early stage Below are some common problems that can occur with or in your system message log Symptom Inaccurate information in the system message log Probable cause and recommended action In rare instances events gathered by the track change feature can report inaccurate information to the sys...

Page 33: ...message Refer to the Fabric OS Message Reference for information specific to the error that was received Some Chip Port CPT and Environmental Monitor EM messages contain I2 C related information If the I2 C message does not indicate the specific hardware that may be failing begin debugging the hardware as this is the most likely cause The next sections provide procedures for debugging the hardware...

Page 34: ...ands these types of reboots are ignored by the Rolling Reboot Detection RRD feature They include the following reboot haFailover fastBoot firmwareDownload The RRD feature is activated and halts rebooting when an unexpected reboot reason is shown continuously in the reboot history within a certain period of time The period of time is switch dependent The following are considered unexpected reboots ...

Page 35: ...activated admin level permission is required to log in Enter the supportShow or supportSave command to collect a limited amount of data to resolve the issue ATTENTION The limited supportSave used with the RRD feature does not support USB Restrictions The following restrictions are applicable on the RRD feature The RRD works only on CFOS based systems and is not available on AP blades If FIPS mode ...

Page 36: ...e domain controller to the destination port identifiers The ELS ECHO request elicits an ELS ECHO response from a port identifier in the fabric and validates link connectivity Use the fcPing command to validate link connectivity to a single device or between a pair of devices ATTENTION There are some devices that do not support the ELS ECHO request In these cases the device either does not respond ...

Page 37: ...0 37 25 ad 05 Zone Check Not Zoned Pinging 10 00 00 00 c9 29 0e c4 0x20800 with 12 bytes of data received reply from 10 00 00 00 c9 29 0e c4 12 bytes time 1162 usec received reply from 10 00 00 00 c9 29 0e c4 12 bytes time 1013 usec received reply from 10 00 00 00 c9 29 0e c4 12 bytes time 1442 usec received reply from 10 00 00 00 c9 29 0e c4 12 bytes time 1052 usec received reply from 10 00 00 00...

Page 38: ...onnected It works by sending ECHO frames to a destination device and outputs the status of each ISL it traverses whether or not the response from the destination device is received Each ECHO frame may choose any path from multiple available paths in the fabric to reach the destination device This utility allows you to do the following Run a sanity test that exercises all the ISLs and internal link...

Page 39: ...links giving the errors NOTE Superping provides an indication if all ISLs are covered If all the ISLs are not covered you can increase the coverage count and maximum retries to transmit so that complete coverage of all ISLs is achieved Consider the following example in which a few errors are recorded on ISLs 3 205 2 25 3 204 2 27 2 42 101 3 and 2 1 101 8 But a maximum of 100 percentage errors are ...

Page 40: ...al information from a source port index on the local switch to a destination port index on another switch This routing information describes the full path that a data stream travels between these ports including all intermediate switches ATTENTION Using the pathInfo command when exchange based routing is turned on can provide different paths with each attempt The routing and statistics information...

Page 41: ...yes y no n no y Trace reverse path yes y no n no Source route yes y no n no Timeout 1 30 5 Target port is Embedded Hop In Port Domain ID Name Out Port BW Cost 0 E 9 web226 2 1G 1000 Port E 2 Tx Rx Tx Rx B s 1s 0 0 B s 64s 1 1 Txcrdz 1s 0 Txcrdz 64s 0 F s 1s 0 0 F s 64s 2743 0 Words 2752748 2822763 Frames 219849 50881 Errors 0 Hop In Port Domain ID Name Out Port BW Cost 1 3 10 web229 12 1G 1000 Por...

Page 42: ... the enforce tag feature to prevent a backbone switch from accepting unwanted LSAN zone databases into its local database Symptom Host application times out Probable cause and recommended action The FCR tends to take a long time more than 5 seconds to present and setup paths for the proxy devices Certain hosts are able to do discovery much faster as a result they end up timing out Use the speed ta...

Page 43: ... for non loop fabric capable devices E_Port Expansion port Assigned to ISL links F_Port Fabric port Assigned to fabric capable devices EX_Port A type of E_Port It connects a Fibre Channel router to an edge fabric From the point of view of a switch in an edge fabric an EX_Port appears as a normal E_Port It follows applicable Fibre Channel standards as other E_Ports However the router terminates EX_...

Page 44: ...ion Verify that the correct licensing is installed or that the port is not configured as a loop port a G_Port or the port speed is not set FIGURE 2 Simple port initialization process The FCP auto discovery process enables private storage devices that accept the process login PRLI to communicate in a fabric If device probing is enabled the embedded port logs in PLOGI and attempts a PRLI into the de...

Page 45: ...s Symptom No light from the port LEDs Probable cause and recommended action If there is no light coming from the port LED then no signal is being detected Check your cable and SFP to determine the physical fault Connection problems Determine if the problem is the target or the host then continue to divide the suspected problem path in half until you can pinpoint the problem One of the most common ...

Page 46: ...nfigured correctly The third party vendor multi pathing input output MPIO software if it is being used is configured correctly The HBA and storage device and the driver and firmware are compatible with switch based on the compatibility matrix The driver settings and binaries are up to date The device Basic Input Output System BIOS settings are correct The HBA configuration is correct according to ...

Page 47: ...C SYMMETRIX 5267 Fabric Port Name 20 0f 00 60 69 10 9b 5b 2 Look for the device in the NS list which lists the nodes connected to that switch This allows you to determine if a particular node is accessible on the network If the device is not present in the NS list the problem is between the device and the switch There may be a time out communication problem between edge devices and the name server...

Page 48: ...speed settings of all the ports 2 Enter the switchShow command to determine if the port has module light 3 Enter the portCfgSpeed command to change the port speed to 1 2 4 or 8 Gbps depending on what speed can be used by both devices This should correct the negotiation by setting to one speed 4 Enter the portLogShow or portLogDump command 5 Check the events area of the output time task event port ...

Page 49: ...ck argument number four for the loop initialization soft assigned LISA frame 0x11050100 switch admin portlogdumpport 4 time task event port cmd args 11 40 02 078 PORT Rx3 23 20 22000000 00000000 ffffffff 11050100 Received LISA frame The LISA frame indicates that the loop initialization is complete 3 Skip point to point initialization by using the portCfgLport command The switch changes to point to...

Page 50: ...y the functional operation of a path on a switch This test sends frames from a given port s transmitter and loops them back into the same port s receiver The loopback is done at the parallel loopback path The path traversed in this test does not include the media or the fiber cable TABLE 4 SwitchShow output and suggested action Output Suggested action Disabled If the port is disabled because persi...

Page 51: ...g a CRC error marks the frame with a bad EOF and passes the frame on to its destination Subsequent ports in the path also detect the CRC error and the crc_err counter increments on these ports However since the first port marked the frame with a bad EOF the good EOF counter on the subsequent ports does not increment The marginal link associated with the port with an increasing good EOF counter is ...

Page 52: ... marginal port problems If the problem stops or goes away the switch port or the SFP is marginal proceed to step 6 If the problem does not stop or go away see step 7 4 Run portLoopbackTest on the marginal port You need an adapter to run the loopback test for the SFP Otherwise run the test on the marginal port using the loopback mode lb 5 Use the different modes shown in Table 5 to test the port Re...

Page 53: ...0 00 00 05 1e 8f c1 31 2 2 N8 No_Module 3 3 N8 No_Module output truncated 61 61 N8 No_Module 62 62 N8 No_Module 63 63 N8 No_Module 64 64 id N2 Online E Port 10 00 00 05 1e 34 d0 05 1_d1 Trunk master 65 65 N8 No_Module 66 66 N8 No_Module 67 67 id AN No_Sync 68 68 id N2 Online L Port 13 public 69 69 N8 No_Module 70 70 N8 No_Module 71 71 id N2 Online L Port 13 public 72 72 N8 No_Module 73 73 N8 No_Mo...

Page 54: ...problem Track errors by sampling the port errors every five or ten minutes until you see the problem occur again 5 Enter the portFlagsShow command then check to see how a port has logged in and where a login failed if a failure occurred switch admin portflagsshow Port SNMP Physical Flags 0 Offline In_Sync PRESENT U_PORT LED 1 Online In_Sync PRESENT ACTIVE F_PORT G_PORT U_PORT LOGICAL_ONLINE LOGIN ...

Page 55: ... 11030000 16 52 42 572 PORT Rx3 8 20 22000000 00000000 ffffffff 11030000 NOTE See Port log on page 101 for overview information about portLogDump Media related issues This section provides procedures that help pinpoint any media related issues such as bad cables and SFPs in the fabric The tests listed in Table 6 are a combination of structural and functional tests that can be used to provide an ov...

Page 56: ... from the HBA 1 Connect to the switch and log in as admin 2 Enter the portTest command refer to the Fabric OS Command Reference for information on the command options See Table 7 on page 36 for a list of additional tests that can be used to determine the switch components that are not functioning properly Refer to the Fabric OS Command Reference for additional command information The HBA s bcu fcD...

Page 57: ...e priority Long distance fabric not necessary on Brocade DCX DCX4 S 6510 and 8510 fabrics For more information regarding these ASIC types refer to Appendix A Switch Type and Blade ID Reconciling fabric parameters individually 1 Log in to one of the segmented switches as admin 2 Enter the configShow pattern fabric ops command 3 Log in to another switch in the same fabric as admin 4 Enter the config...

Page 58: ...ll domain ID conflicts are resolved 1 Enter the fabricShow command on a switch from one of the fabrics 2 In a separate Telnet window enter the fabricShow command on a switch from the second fabric 3 Compare the fabricShow output from the two fabrics Note the number of domain ID conflicts there may be several duplicate domain IDs that must be changed Determine which switches have domain overlap and...

Page 59: ...f you suspect that you are trying to introduce a switch into a fabric that has an older version of code check the release notes to verify that any features on that switch are supported in the fabric with the newer code When the Management Server MS Platform services are enabled on a switch running Fabric OS v7 0 0 and you try to merge this switch into a fabric that does not have this feature enabl...

Page 60: ...r traffic takes an extra route to the mirror port When the extra route is removed the frames between the two ports go directly to the destination port The frames at the mirror port could be queued at the destination port behind those frames that went directly to the destination port To prevent this IOD issue port mirroring drops those frames from the mirror port when a connection is disabled If IO...

Page 61: ...s must share the same mirror port Supported platforms Port mirroring is supported only in FC ports In general a platform or blade supporting port mirroring supports both the mirror ports and the mirror connections Exemptions are listed in Table 8 TABLE 8 Port mirroring platform supportability Brocade Model Fabric OS v6 1 1_enc Fabric OS v6 34 0 Fabric OS v6 4 0 Brocade Encryptiop Switch FS8 18 DCX...

Page 62: ...or port The disable command disables the mirror port configuration Adding a port mirror connection 1 Connect to the switch and log in using an account with admin permissions 2 Enter the portMirror add slotnumber portnumber SourceID DestID command TABLE 9 Maximum number of mirror connections Model Maximum Number of Mirror Connections chassis wide Brocade 300 1 Brocade 5100 3 Brocade 5300 1 Brocade ...

Page 63: ...ion 1 Connect to the switch and log in using an account with admin permissions 2 Enter the portMirror del SourceID DestID command For example to delete the port mirror connection on mirror port 2 you might type switch admin portMirror del 0x011400 0x240400 Displaying port mirror connections 1 Connect to the switch and log in using an account with admin permissions 2 Enter the portMirror show comma...

Page 64: ...44 Fabric OS Troubleshooting and Diagnostics Guide 53 1002150 02 Port mirroring 3 ...

Page 65: ...trator s Guide For information about Virtual Fabrics using Fabric OS v6 3 0 or later refer to the Fabric OS Administrator s Guide Symptom The configuration upload fails Probable cause and recommended action If the configuration upload fails It may be because of one or more of the following reasons The FTP or SCP server s host name is not known to the switch Verify with your network administrator t...

Page 66: ...DP and TCP traffic Example of a failed ping C ping 192 163 163 50 Pinging 192 163 163 50 with 32 bytes of data Request timed out Request timed out Request timed out Request timed out Ping statistics for 192 163 163 50 Packets Sent 4 Received 0 Lost 4 100 loss If your ping has failed then you should verify the following The ports are open on the firewall The FTP server is up and running You do not ...

Page 67: ...ch Note however that you must disable the switch for some configuration downloads For more information on how to perform a configuration download without disabling a switch refer to the Fabric OS Administrator s Guide You do not have permission on the host to perform configuration download There may be some restrictions if you are using Admin Domains or Role Based Access Control For more informati...

Page 68: ...load generates both RASLog and Audit log messages resulting from execution of the configDownload command The following messages are written to the logs configDownload completed successfully RASLog and Audit log configUpload completed successfully RASLog configDownload not permitted Audit log configUpload not permitted RASLog Warning Downloading configuration without disabling the switch was unsucc...

Page 69: ...on form 4 Switch name Ethernet IP address Ethernet subnet mask Total number of local devices nsShow Total number of devices in fabric nsAllShow Total number of switches in the fabric fabricShow TABLE 10 Brocade configuration and connection Continued Brocade configuration settings Value ...

Page 70: ...50 Fabric OS Troubleshooting and Diagnostics Guide 53 1002150 02 Brocade configuration form 4 ...

Page 71: ...des can be inserted only into slots 5 and 8 on the Brocade DCX CR4S 8 blades can be inserted only into slots 3 and 6 on the Brocade DCX 4S Typically issues detected during firmware download to AP blades do not require recovery actions on your part If you experience frequent failovers between CPs that have different versions of firmware then you may notice multiple blade firmware downloads and a lo...

Page 72: ...e process The entire firmware download and commit process takes approximately 17 minutes If there is a problem wait for the time out 30 minutes for network problems before issuing the firmwareDownload command again Disrupting the process can render the switch inoperable and require you to seek help from your switch service provider Do not disconnect the switch from power during the process because...

Page 73: ...ptom Cannot upgrade directly to v6 3 0 Probable cause and recommended action If the switch is running Fabric OS v6 1 0 or earlier you are not allowed to upgrade directly to v6 3 0 because of the one version rule Upgrade your switch to Fabric OS version v6 2 0 before upgrading to v6 3 0 The one version rule also applies to downgrading Symptom Server is inaccessible or firmware path is invalid Proba...

Page 74: ...tch with Interop mode turned on Probable cause and recommended action On single CP Interop fabric does not support Coordinated HotCode Load Perform a firmwareDownload o command The operand bypasses the checking of Coordinated HotCode Load HCL On single CP systems in interop fabrics the HCL protocol is used to ensure data traffic is not disrupted during firmware upgrades This option allows a firmwa...

Page 75: ...n running on the active CP Then re issue the firmwareDownload command to download the desired firmware version to both CPs For example if CP0 is running v6 3 0 on the primary and secondary partitions and CP1 is running v7 0 0 on the primary and secondary partitions then synchronize them by issuing the firmwareDownload command NOTE Some of the messages include error codes as shown in the following ...

Page 76: ...m v6 4 0 to v7 0 0 Cannot upgrade directly to 7 0 Please upgrade to 6 3 first and then upgrade to 7 0 Upgrade is not allowed because one of the Zones or configs defined on this switch is using a reserved naming prefix msfr_zn or msfr_cfg please rename them before upgrading Upgrade is not allowed because Secondary RKM KV is registered Please use cryptocfg dereg keyvault label to deregister the seco...

Page 77: ... move all vacant ports to the default Downgrade is not allowed because there are ports configued in CSCTL mode Please use portcfgqos CLI to disable the CSCTL mode Downgrade is not allowed because dynamic area mode is enabled on default switch Please use configure CLI to turn this mode off Downgrade is not allowed because zoning configuration has overlapping enhanced TI zones in one or more context...

Page 78: ...ountered during firmware downgrade The following example shows blocking messages that you may see if you were downgrading from v7 0 0 to v6 4 0 with any v6 4 1 software features enabled Downgrade is not allowed because AG is enabled Please run ag modedisable command to disable AG mode before proceeding IPSec on FCIP tunnels requires version v6 3 1 or higher for 7800 and v6 4 0 or higher for FX8 24...

Page 79: ...3 0 Upgrade your switch to v6 2 0 first before upgrading to the requested version Probable cause and recommended action If the switch is running v6 1 0 or earlier you are not allowed to upgrade directly to v6 3 0 because of the two version rule Upgrade your switch to Fabric OS version v6 2 0 before upgrading to v6 3 0 Message Non disruptive firmwaredownload is not supported when downgrading to 6 1...

Page 80: ...delete the base switch and disable Virtual Fabrics prior to downgrading the firmware Routing This message refers to any route settings that must be changed prior to downgrading the switch s firmware Message Downgrade is not allowed because IOD Delay value is configured for one or more domains Please use ioddelayshow and ioddelayreset to disable them before downgrading Probable cause and recommende...

Page 81: ... this procedure to recover the password for the default accounts of user admin and factory If you do not know the root password you must contact your service support provider to recover admin passwords Recovering passwords 1 Open a CLI connection serial or Telnet to the switch 2 Log in as root 3 Enter the command for the type of password that was lost passwd user passwd admin passwd factory 4 Ente...

Page 82: ...cation Symptom Switch is unable to authenticate device Probable cause and recommended action When the device authentication policy is set to ON the switch expects a FLOGI with the FC SP bit set If this bit is not set the switch rejects the FLOGI with reason LS_LOGICAL_ERROR 0x03 in the switch log with the explanation of Authentication Required 0x48 and disables the port Set the device authenticati...

Page 83: ...ctions you can take to correct the problem Gathering additional information For security related issues use the following guidelines to gather additional data for your switch support provider Perform a supportSave n command If not sure about the problem area collect a supportSave n from all switches in the fabric TABLE 13 SSL messages and actions Message Action The page cannot be displayed The SSL...

Page 84: ...uses related to this generic issue You must verify the following There are no port filters in the firewalls between the fabric and the SNMP management station If your SNMP management station is a dual homed server check that the routing tables are set up correctly for your network If you continue to have problems collect the data in the next section and contact your switch support provider Gatheri...

Page 85: ...onal are recorded in the system log or are output to the local console This includes logging both passing and failing results If the tests fail on your switch it constantly reboots Because boot PROM access is disabled you are not able to exit out of the reboot You must send the switch back to your switch service provider for repair ...

Page 86: ...66 Fabric OS Troubleshooting and Diagnostics Guide 53 1002150 02 FIPS 6 ...

Page 87: ...fault switch If XISL is turned off you can connect ICLs to other logical switches A default switch can be configured as a base switch in the Brocade 5100 and 5300 switches but not in a Brocade DCX or DCX 4S Fabric IDs of default switches cannot be manually changed The default switch is able to participate in a logical fabric using extended ISLs XISLs In the Brocade DCX and DCX 4S the default switc...

Page 88: ...c identification issues Symptom E_Ports directly connecting two logical switches do not form or are disabled Probable cause and recommended action The FIDs on each of the logical switches must be the same Use the lsCfg show command to view the current FIDs on the chassis and then the lsCfg change FID newfid FID command to change the FID Symptom Invalid FID Probable cause and recommended action FID...

Page 89: ...the base switch If the port is not intended to be used as an EX_Port use the portCfgDefault command to reset the port to its default configuration Symptom An EX_ or VEX_Port is disabled with reason Conflict XISL capability domain Probable cause and recommended action Use the configure command to set the value on the Allow XISL use to OFF on all logical switches of the connecting edge fabric Sympto...

Page 90: ...determine if you have a Brocade FR4 18i blade refer to the Fabric OS Administrator s Guide Symptom Validation of switch configuration changes is not supported on this platform Probable cause and recommended action This platform is unknown to the logical switch subsystem Symptom Given slot number is not valid on this platform Probable cause and recommended action You are specifying a slot number th...

Page 91: ...ror while creating the switch Symptom A port or ports cannot be moved to the requested switch because it would exceed the 256 area limit for this switch Probable cause and recommended action The area limit would be exceeded if the lsCfg command were allowed Symptom A port or ports cannot be moved to the requested switch because it may only exist in a base or default switch Probable cause and recom...

Page 92: ...lowing guidelines to gather additional data for your switch support provider Perform the supportSave command If not sure about the problem area perform the supportSave command on all chassis and logical switches in the fabric If you think it may be related to E_Port authentication then perform the supportSave n command on both switches or logical switches of the affected E_Port ...

Page 93: ... Example of an unformed E_Port This example shows that port 3 is not configured as an E_Port ecp admin trunkdebug 126 127 port 126 is not E EX port port 127 is not E EX port Example of a formed E_Port ecp admin trunkdebug 100 101 port 100 and 101 connect to the switch 10 00 00 05 1e 34 02 45 The trunkDebug command displays the possible reason that two ports cannot be trunked Possible reasons are T...

Page 94: ...le to talk to a storage device Symptom Trunk goes offline and online bounces Probable cause and recommended action A port disabled at one end because of buffer underallocation causes all the disabled ports at the other end to become enabled Some of these enabled ports become disabled because of a lack of buffers which in turn triggers ports to be enabled once again at the other end While the syste...

Page 95: ...rder the zone member list if necessary The newly changed zone configuration are not effective until you issue the cfgEnable command This should be done during a maintenance window because this may cause disruption in large fabrics Verifying a fabric merge problem 1 Enter the switchShow command to validate that the segmentation is because of a zone issue 2 Review Segmented fabrics to view the diffe...

Page 96: ...iguration switch admin zone show Defined TI zone configuration TI Zone Name greenzone Port List 2 2 3 3 5 3 4 11 Configured Status Activated Failover Enabled Enabled Status Activated Failover Enabled TI Zone Name purplezone Port List 1 2 1 3 3 3 4 5 Configured Status Activated Failover Enabled Enabled Status Deactivated Failover Enabled TI Zone Name bluezone Port List 9 2 9 3 8 3 8 5 Configured St...

Page 97: ...ion in one fabric is different from the definition of a zone object with the same name in the other fabric Zone conflicts Zone conflicts can be resolved by saving a configuration file with the configUpload command examining the zoning information in the file and performing a cut and paste operation so that the configuration information matches in the fabrics being merged After examining the config...

Page 98: ...en cleared This forces the zones to merge and populates the switches with the correct zoning database The fabrics then merge cfgEnable Use to enable and activate make effective the specified configuration cfgSave Use to save changes to the zone configuration database cfgTransAbort Use to abort the current zoning transaction without committing it cfgTransShow Use to display the ID of the current zo...

Page 99: ... admin 2 Enter the cfgShow command and print the output 3 Start another Telnet session and connect to the next fabric as an admin 4 Enter the cfgShow command and print the output 5 Compare the two fabric zone configurations line by line and look for an incompatible configuration 6 Connect to one of the fabrics 7 Run zone configure edit commands to edit the fabric zone configuration for the segment...

Page 100: ...s the request and another device rejects the request switch admin fcping 10 00 00 00 c9 29 0e c4 21 00 00 20 37 25 ad 05 Source 10 00 00 00 c9 29 0e c4 Destination 21 00 00 20 37 25 ad 05 Zone Check Not Zoned Pinging 10 00 00 00 c9 29 0e c4 0x20800 with 12 bytes of date received reply from 10 00 00 00 c9 29 0e c4 12 bytes time 1162 usec received reply from 10 00 00 00 c9 29 0e c4 12 bytes time 101...

Page 101: ... default zone mode by entering the defZone show command If it is no access change it to all access To modify default zone mode from no access to all access enter the defZone all command and then the cfgSave command 2 Confirm that the specific edge devices that must communicate with each other are in the same zone If they are not in the same zone and zoning is enabled proceed to step 3 If they are ...

Page 102: ...2150 02 Gathering additional information 9 Gathering additional information Collect the data from a supportSave n command Then collect the data from the cfgTransShow command For the port having the problem collect the data from the filterPortShow port command ...

Page 103: ...ically during the power on self test POST Automatically on an individual blade whenever it is installed into a director chassis Manually using Fabric OS CLI commands The error messages generated during these test activities are sent to the serial console and system message logs whose output formats may differ slightly Use the diagHelp command to receive a list of all available diagnostic commands ...

Page 104: ...d support minimal validation because of the restriction on test duration Their purpose is to give a basic health check before a new switch joins a fabric These tests are divided into two groups POST1 and POST2 POST1 validates the hardware interconnect of the device and POST2 validates the ability of the device to pass data frames between the ports The specific set of diagnostic and test commands r...

Page 105: ...SW4100_P45 Services starting a COLD recovery mdev c line 638 2005 03 31 20 12 49 MFIC 1002 220 INFO Paulsa45 Chassis FRU header not programmed for switch NID using defaults applies only to FICON environments sysctrld all services Active 2005 03 31 20 12 50 DGD 5001 0 INFO SW4100_P45 Slot 0 has started POST main c line 1189 POST1 Started running Thu Mar 31 20 12 51 GMT 2005 POST1 Test 1 Running tur...

Page 106: ...f the switch including its power supplies fans and temperature If the status of any one of these components is either marginal or down the overall status of the switch is also displayed as marginal or down If all components have a healthy status the switch displays a healthy status To modify the rules used to classify the health of each component use the switchStatusPolicySet command To view the r...

Page 107: ... Online Offline Testing or Faulty switchMode Switch operation mode Native Interop or Access Gateway switchRole Principal Subordinate or Disabled switchDomain ID 0 31 or 1 23 switchId Switch embedded port D_ID switchWwn Switch World Wide Name WWN switchBeacon Switch beaconing state On or Off zoning When Access Gateway mode disabled the name of the active zone displays in parenthesis FC Router FC Ro...

Page 108: ...ration the total cumulative amount of uptime since the system was first powered on and the load average over the past one minute 1 29 in the preceding example five minutes 1 31 in the example and 15 minutes 1 27 in the example The reason for the last switch reboot is also recorded in the system message log TABLE 17 switchShow command output Variable Definition Index Index follows Area up to 255 Th...

Page 109: ...ver some performance degradation occurs due to the ISL links being saturated with the test frames This test should be run with caution on a live fabric Table 18 lists the supported ports for the specified version of Fabric OS when using the spinFab command Debugging spinFab errors Link errors and tx rx errors are seen when the spinFab test fails TABLE 18 Port type support Port type Supported in v6...

Page 110: ... run the spinFab command Verify that the link comes up by enabling and disabling the local or remote ports Verify that the source of the error is either local port or remote port This can be done by monitoring the port statistics on both the ends simultaneously Refer to Displaying the port statistics on page 96 for more information on how to display the statistics for a port Verify that the cables...

Page 111: ...found in Fabric OS v6 2 0 and earlier versions only Clearing the error counters This procedure clears the port hardware statistics including ALPA based CRC monitor End to End monitor and filter based performance monitor statistics 1 Connect to the switch and log in as admin 2 Enter the portStatsClear slot port command Enabling a port 1 Connect to the switch and log in using an account with admin p...

Page 112: ...gnostics when the ports come online and includes the following tests Electrical loopback 16G SFPs only Optical loopback 16G SFPs only Link traffic 16G SFPs and 10G SFPs Link latency and distance measurement 16G SFPs and 10G SFPs FIGURE 3 Example of a D_Port configuration Use the D_Port tests for the following situations Testing a new ISL before adding it to the fabric Testing a trunk member before...

Page 113: ... and 2 for Port 2 on Switch B shown in Figure 3 switchB admin portdisable 2 switchB admin portcfgdport enable 2 4 Enable Port 1 on Switch A using the portenable command The tests start automatically when either port is enabled switchA admin portenable 1 5 Enable Port 2 on Switch B using the portenable command switchB admin portenable 2 6 While the test is running enter the portdporttest command wi...

Page 114: ...es Test Result 24 ONLINE E O PASSED 26 ONLINE E O RESPONDER 33 OFFLINE RESPONDER 8 Optional Restart the test on both switches if one of the switches reboots or if the test does not complete on one of the switches Perform a portdporttest stop command and restart the test with the portdporttest start command on both switches Disabling D_Port To disable D_Port perform the following tasks 1 Disable Po...

Page 115: ... Host Bus Adapters R_RDY mode Encryption mode Compression mode Port information Use the following instructions to view information about ports and to help diagnose if your switch is experiencing port problems Viewing the status of a port 1 Connect to the switch and log in as admin 2 Enter the portShow slot port command specifying the number that corresponds to the port you are troubleshooting In t...

Page 116: ...coding errors received and number of class 2 and class 3 frames received Refer to the Fabric OS Command Reference for additional portStatsShow command information such as the syntax for slot or port numbering switch admin portstatsshow 68 stat_wtx 113535 4 byte words transmitted stat_wrx 22813 4 byte words received stat_ftx 9259 Frames transmitted stat_frx 821 Frames received stat_c2_frx 0 Class 2...

Page 117: ...tional portErrShow command information switch admin porterrshow frames enc crc crc too too bad enc disc link loss loss frjt fbsy tx rx in err g_eof shrt long eof out c3 fail sync sig 0 665k 7 0k 0 0 0 0 0 0 6 0 0 1 2 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 3 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 5 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 6 0 0 0 0 0 0 0 0 0...

Page 118: ...tch type Checking the temperature fan and power supply 1 Log in to the switch as user 2 Enter the sensorShow command Refer to the Fabric OS Command Reference for details regarding the sensor numbers 3 Check the temperature output Look for indications of high or low temperatures TABLE 19 Error summary description Error type Description frames tx Frames transmitted frames rx Frames received enc in E...

Page 119: ...n unit installed FAULTY Fan has exceeded hardware tolerance The output from this command varies depending on switch type and number of fans present Refer to the appropriate hardware reference manual for details regarding the fan status You may first consider re seating the fan unplug it and plug it back in Checking the status of a power supply 1 Connect to the switch and log in as admin 2 Enter th...

Page 120: ...ables messages to be saved across power cycles and reboots The Brocade DCX DCX 8510 family and DCX 4S enterprise class platforms maintain the same RASlog for each of the two CP blades Since all RASlog messages are routed to the Active CP the message CPU ID is added as part of the RASlog message attribute RASlog message attribute SLOT is defined to identify the CPU that generated the message For ex...

Page 121: ...2 Enter the errShow command Clearing the system message log 1 Connect to the switch and log in as admin 2 Enter the errClear command 3 Repeat step 2 on the standby CP for a complete erasure of the message log All switch and chassis events are removed from both CPs Port log The Fabric OS maintains an internal log of all port activity The port log stores entries for each port as a circular buffer Th...

Page 122: ...ctions and communications within a fabric By understanding the processes that are taking place in the fabric issues can be identified and located The portLogDump command displays the port log showing a portion of the Fibre Channel payload and header FC PH The header contains control and addressing information associated with the frame The payload contains the information being transported by the f...

Page 123: ...d service for logging system messages made available by default on UNIX and Linux operating systems It is available as a third party application for Windows operating systems Fabric OS can be configured to use a UNIX style syslogd process to forward system events and error messages to log files on a remote host system The host system can be running UNIX Linux or any other operating system that sup...

Page 124: ... shown in this example switch admin syslogdipadd 1080 8 800 200C 417A switch admin syslogdipadd 1081 8 800 200C 417A switch admin syslogdipadd 1082 8 800 200C 417A switch admin syslogdipadd 10 1 2 4 switch admin syslogdipadd 10 1 2 5 switch admin syslogdipadd 10 1 2 6 switch admin syslogdipshow syslog IP address 1080 8 800 200C 417A syslog IP address 1081 8 800 200C 417A syslog IP address 1082 8 8...

Page 125: ...ng of the remote server so that you are alerted if the server becomes unavailable and you can correct the problem After the setup is complete you can run the supportSave c command to save RASLog TRACE supportShow core file FFDC data and other diagnostic support information to the server without specifying server details The following procedures describe the tasks for setting up automatic transfer ...

Page 126: ...the prompts switch admin supportsave c This command will collect RASLOG TRACE supportShow core file FFDC data and other support information and then transfer them to a FTP SCP server or a USB device This operation can take several minutes NOTE supportSave will transfer existing trace dump file first then automatically generate and transfer latest one There will be two trace dump files transfered a...

Page 127: ...board of the switch The revision number is not necessary when converting the number Convert the value using Table 22 TABLE 22 switchType to B Series model converter switchType B Series switch model ASIC Base switch speed 12 3900 BLOOM 2 Gb 32 port switch 16 3200 BLOOM 2 Gb 8 port value line switch 21 24000 BLOOMII 2 Gb 128 port core fabric switch 26 3850 BLOOMII 2 Gb 16 port switch with switch lim...

Page 128: ... embedded switch 75 M5424 GoldenEye2 8 Gb 24 port embedded switch 76 8000 Condor2 8 Gb 16 FC port 10 GbE 8 Ethernet port switch 77 Brocade DCX 4S Condor2 8 Gb 192 port core fabric backbone 83 7800 Condor2 8 Gb 16 FC ports 6 GbE ports extension switch TABLE 23 B series blade model descriptions Blade ID B series blade model ASIC Description 4 FC2 16 n a 2 Gb 16 FC ports blade 24 FR4 18i Condor 16 FC...

Page 129: ...nting 3 bytes is called a hex triplet Fibre Channel uses hexadecimal notation in hex triplets to specify well known addresses and port IDs Example of hexadecimal triplet conversion Notice the PID in the nsshow output is in hexadecimal shown in blue in the following example Switch admin nsshow Type Pid COS PortName NodeName TTL sec N 610600 2 3 10 00 00 00 c9 29 b3 84 20 00 00 00 c9 29 b3 84 na FC4...

Page 130: ...47 48 49 4a 4b 4c 4d 4e 4f 50 Decimal 81 82 83 84 85 86 87 88 89 90 Hex 51 52 53 54 55 56 57 58 59 5a Decimal 91 92 93 94 95 96 97 98 99 100 Hex 5b 5c 5d 5e 5f 60 61 62 63 64 Decimal 101 102 103 104 105 106 107 108 109 110 Hex 65 66 67 68 69 6a 6b 6c 6d 6e Decimal 111 112 113 114 115 116 117 118 119 120 Hex 6f 70 71 72 73 74 75 76 77 78 Decimal 121 122 123 124 125 126 127 128 129 130 Hex 79 7a 7b ...

Page 131: ...x c9 ca cb cc cd ce cf d0 d1 d2 Decimal 211 212 213 214 215 216 217 218 219 220 Hex d3 d4 d5 d6 d7 d8 d9 da db dc Decimal 221 222 223 224 225 226 227 228 229 230 Hex dd de df e0 e1 e2 e3 e4 e5 e6 Decimal 231 232 233 234 235 236 237 238 239 240 Hex e7 e8 e9 ea eb ec ed ef ee f0 Decimal 241 242 243 244 245 246 247 248 249 250 Hex f1 f2 f3 f4 f5 f6 f7 f8 f9 fa Decimal 251 252 253 254 255 Hex fb fc fd...

Page 132: ...112 Fabric OS Troubleshooting and Diagnostics Guide 53 1002150 02 Hexadecimal overview B ...

Page 133: ...reboots during the download 47 upload fails 45 connectivity no connectivity between host and storage 4 no connectivity between switches 4 contacting your switch support provider 5 core file warning messages 13 correcting device login issues 33 link failures 28 marginal links 30 crc_err counter 31 crc_g_eof counter 31 D D_Port 92 devices login issues 33 missing 2 Diagnostic Port 92 diagnostics 83 c...

Page 134: ... type 24 is inserted 71 blade application firmware failed 59 blade application firrmware failed 59 blade errors 59 blade is faulty 52 blade is stuck in the LOADING state 52 cannot download the requested firmware 54 cannot download with interop mode turned on 54 cannot upgrade directly to v6 0 59 command fails 3 firmware path is invalid 53 firmwaredownload is already in progress 54 gathering additi...

Page 135: ...ht on LEDs 4 NS 26 NTP 1 O output from a console 6 P passwords recovering forgotten 61 pathInfo 20 performance problems 4 PLOGI 24 POD enabling ports 91 point to point initialization failure 29 port 91 bypassed 30 disabled 30 enabling 91 in wrong mode 30 initialization 23 loopback 30 Port cannot be moved 4 port information viewing 93 port mirroring 23 adding port connections 42 considerations 40 d...

Page 136: ...atus viewing 86 switchType 107 symptoms 2 synchronize switches 1 syslog messages 4 T targets 2 temperature fan and power supply 98 temperature status of 100 test a port 36 a switch 36 TI zone problem 75 time is not in sync 11 troubleshooting certificates 63 corrupt certificate 63 invalid certificate 63 trunk bounces 4 failed to form 4 trunk goes offline and online bounces 74 U U_Port 23 user forgo...

Page 137: ...Fabric OS Troubleshooting and Diagnostics Guide 117 53 1002150 02 Z zone configuration 2 configuration mismatch 5 content mismatch 5 resolving conflicts 77 troubleshooting 77 type mismatch 5 ...

Page 138: ...118 Fabric OS Troubleshooting and Diagnostics Guide 53 1002150 02 ...

Reviews:

No comments