manualshive.com logo in svg

Sun Microsystems StorEdge, Troubleshooting Manual, Page: 69 / 168

Share
Download
Sun Microsystems StorEdge Troubleshooting Manual Download Page 69

Chapter 4

Troubleshooting Example

51

... (

continuation

)

DETAILS:

Sep 13 13:05:36 WWN:50020f23000003d5    diag226.Central.Sun.COM mpxio: [ID

779286 kern.info] /scsi_vhci/ssd@g60020f20000003d53d34940b0004e6b6 (ssd0)

multipath status: degraded, path /pci@1f,2000/SUNW,qlc@1/fp@0,0 (fp4) to target

address: 50020f23000003d5,3 is offline

Sep 13 13:05:36 WWN:50020f23000003d5    diag226.Central.Sun.COM mpxio: [ID

779286 kern.info] /scsi_vhci/ssd@g60020f20000003d53d3493d80003c090 (ssd1)

multipath status: degraded, path /pci@1f,2000/SUNW,qlc@1/fp@0,0 (fp4) to target

address: 50020f23000003d5,2 is offline

Sep 13 13:05:36 WWN:50020f23000003d5    diag226.Central.Sun.COM mpxio: [ID

779286 kern.info] /scsi_vhci/ssd@g60020f20000003d53d3493930006a222 (ssd2)

multipath status: degraded, path /pci@1f,2000/SUNW,qlc@1/fp@0,0 (fp4) to target

address: 50020f23000003d5,1 is offline

Sep 13 13:05:36 WWN:50020f23000003d5    diag226.Central.Sun.COM mpxio: [ID

779286 kern.info] /scsi_vhci/ssd@g60020f20000003d53d349365000c1691 (ssd3)

multipath status: degraded, path /pci@1f,2000/SUNW,qlc@1/fp@0,0 (fp4) to target

address: 50020f23000003d5,0 is offline

----------------------------------------------------------------------

Site     : FSDE LAB Broomfield CO

Source   : diag226.central.sun.com

Severity : Warning (Actionable)

Category : HOST   DeviceId : host:diag226.central.sun.com

EventType: AlarmEvent.M.hba

EventCode: 7.5.20

EventTime: 2002/09/13 13:06:49

DESCRIPTION: status of hba /devices/pci@1f,2000/SUNW,qlc@1/fp@0,0:devctl on

diag226.central.sun.com changed from CONNECTED to NOT CONNECTED

INFORMATION:

monitors changes in the output of luxadm -e port

 Found path to 20 HBA ports

 /devices/sbus@2,0/SUNW,socal@d,10000:0   NOT CONNECTED

 .....

RECOMMENDED-ACTION:

use ’luxadm -e port’ to confirm hba status, verify if all paths are working

... (

continued

)

Summary of Contents for StorEdge

Page 1: ...nc 4150 Network Circle Santa Clara CA 95054 U S A 650 960 1300 Send comments about this document to docfeedback sun com Sun StorEdge SAN 4 0 Release Field Troubleshooting Guide Part No 816 6580 11 October 2002 Revision A ...

Page 2: ...Sun Microsystems Inc 4150 Network Circle Santa Clara CA 95054 Etats Unis Tous droits réservés Ce produit ou document est protégé par un copyright et distribué avec des licences qui en restreignent l utilisation la copie la distribution et la décompilation Aucune partie de ce produit ou document ne peut être reproduite sous aucune forme par quelque moyen que ce soit sans l autorisation préalable et...

Page 3: ...s of the Sun StorEdge SAN 4 0 Release 3 Cascading Switches E_Ports 7 2 Configurations 9 Supported Hardware 10 Supported Configurations 12 Operating Environments 12 Hosts 13 Host Operating Environment Rules 14 Storage Arrays 14 Array Storage Rules 15 Host Bus Adapters 15 ...

Page 4: ...Port Types 19 New Sun StorEdge SAN 4 0 Release Port Types 19 Sun StorEdge and Brocade Communications Systems Port Descriptions and Differences 19 Zones 21 Name Server Zones 21 Overlapping Zones 21 Zoning Rules 22 Configuration Guidelines 22 Switches 22 Zones and Arrays 22 Zones and Storage 23 Cascading Rules 23 Rules for Adding and Removing Devices While the Hosts are Online 23 Configuration Examp...

Page 5: ...g the Sun Switch 41 Using Switch Counter Information 41 qlctest Test 42 4 Troubleshooting Example 43 Example Configuration 44 Example Assumptions 45 Troubleshooting Outline 45 Troubleshooting Example of a Host to Switch Error 47 Determine the Error 47 Determine the Extent of the Problem 53 Check the Array Status 55 Check the Switch Status 56 Test the FRUs 57 Storage Automated Diagnostics Environme...

Page 6: ... Brocade Web Site 72 To Install Firmware from UNIX Solaris 72 To Install Firmware using FTP 74 Upgrading the SAN 76 Downloading Patches and Packages 76 Verifying Upgrade Compliance 76 To Upgrade the Software 76 Volume Management 77 Sun StorEdge SAN 4 0 Release 77 cfgadm Plug in Library Packages 78 Software Installation 79 To Upgrade the Storage Automated Diagnostic Environment Version 2 1 Package ...

Page 7: ...ving Power 99 General Troubleshooting Procedures 101 Troubleshooting Case Study 103 Configuration 103 Storage Automated Diagnostic Environment Version 2 1 Topology 104 C Brocade Communications Systems Error Messages 121 Error Message Formats 122 Front Panel Message Formats 122 To Display Error Messages from the Front Panel 123 Diagnostic Error Message Formats 123 D Converting Sun FC Switches Fibre...

Page 8: ...viii Sun StorEdge SAN 4 0 Release Field Troubleshooting Guide October 2002 ...

Page 9: ...vironment Diagnostic Tests Window 36 FIGURE 3 4 Storage Automated Diagnostic Environment Test from Topology Window 37 FIGURE 3 5 Storage Automated Diagnostic Environment Test from Topology Window with Background Reduced to 66 38 FIGURE 3 6 Storage Automated Diagnostic Environment Test from Topology Window with Background Reduced to 66 and Components Arranged for Viewing 39 FIGURE 4 1 Troubleshooti...

Page 10: ...esult Details with Remedy Request 113 FIGURE B 6 Test Result Details Showing a Successful Test 114 FIGURE B 7 Continued Link Test Example Results 115 FIGURE B 8 Continued Link Test Example Results 116 FIGURE B 9 Storage Automated Diagnostic Environment Version 2 1 Test from Topology Window 119 ...

Page 11: ...kages Compatibility Matrix 18 TABLE 2 9 Sun StorEdge and Brocade Communications Systems Port Descriptions 19 TABLE 2 10 Differences Between Sun StorEdge and Brocade Port Communications Systems Port Nomenclature 20 TABLE 2 11 Arrays Zones and Initiators 23 TABLE A 1 Software Download Sites 67 TABLE A 2 Software Installation Sequence 69 TABLE B 1 SAN Supportability Matrix with Solaris 8 02 02 Update...

Page 12: ...r 2002 TABLE B 6 Differences Between Sun StorEdge and Brocade Port Communications Systems Port Nomenclature 95 TABLE C 1 Probable Failure Actions 123 TABLE C 2 Error Message Codes Defined 124 TABLE C 3 Diagnostic Error Messages 128 TABLE D 1 ASIC and Port Values 142 ...

Page 13: ...derstanding of the product The Appendices found in this guide explain how to diagnose and troubleshoot Brocade Communications Systems Inc Silkworm switches Using UNIX Commands This document may not contain information on basic UNIX commands and procedures such as shutting down the system booting the system and configuring devices See one or more of the following for this information Solaris Handbo...

Page 14: ...ve mail AaBbCc123 What you type when contrasted with on screen computer output su Password AaBbCc123 Book titles new words or terms words to be emphasized Command line variable replace with a real name or value Read Chapter 6 in the User s Guide These are called class options You must be superuser to do this To delete a file type rm filename Shell Prompt C shell machine_name C shell superuser mach...

Page 15: ...16 5246 Installer user information 1 Gbyte switch Sun StorEdge Network Switch 16 SANbox 16 with E_Ports Installer s User Manual N A Sun StorEdge Network Switch with E_Ports Management Manual N A Sun StorEdge Network FC Switch 8 and Switch 16 Release Notes 816 0842 Installer user information 2 Gbyte switch Sun StorEdge Network 2Gb Switch 8 16 SANbox2 Management Manual 875 3264 Sun StorEdge Network ...

Page 16: ...re Channel Host Adapter Installation Guide 806 4199 Sun StorEdge CompactPCI Dual Fibre Channel Network Adapter Installation and User s Guide 806 6991 Sun StorEdge SBus Dual Fibre Channel Host Adapter Release Notes 816 2490 Sun StorEdge 2G FC PCI Single Channel Network Adapter Installation Guide 816 4999 Sun StorEdge 2G FC PCI Double Channel Network Adapter Installation Guide 816 5001 Tools Sun Sto...

Page 17: ...interested in improving its documentation and welcomes your comments and suggestions You can email your comments to Sun at docfeedback sun com Please include the part number 816 6580 11 of your document in the subject line of your email man pages cfgadm utility cfgadm_fp 1M n a format utility format 1M n a luxadm utility luxadm 1M n a Find these documents at http www sun com products n solutions h...

Page 18: ...xvi Sun StorEdge SAN 4 0 Release Field Troubleshooting Guide October 2002 ...

Page 19: ... audience for this troubleshooting guide is Sun Service Representatives As such it is therefore assumed that you have been trained on all the components that comprise your particular storage and switch configuration This manual only addresses troubleshooting No repair or corrective action procedures are contained herein This chapter contains the following sections Document Scope on page 2 New Feat...

Page 20: ...e switch as shown in the following diagram FIGURE 1 1 Switch and Interconnections The Storage Automated Diagnostic Environment version 2 1 software package is required to support the configurations in this document Additional information and resources are available at http www sun com storage san or at http sunsolve Sun COM Product Patches PatchPro These websites contain information on software ve...

Page 21: ...nected switches or three ISL links between switches N A Cascaded configuration limit increased to eight linear connected switches or seven ISL links between switches Two of the ISL links can use long wave transceivers and cables SAN configurations limited to single switch or simple cascades Support for local host and storage device attachment with short or long wave cables and transceivers for dis...

Page 22: ... supported for fibre channel arbitrated loop and fabric configurations G_ and GL_ports supported for connections to arrays G_ and GL_ports automatically negotiate in inter switch connections to E_ports TL_ports should be manually configured for loop connections to storage devices ISLs N A Short and long wave cables and transceivers supported Same Long wave only 1 Gbit GBICs supported for connectiv...

Page 23: ...ement application manages the 1 Gbit switches with old firmware only N A New switch management tools are available See the vendor specific documentation for details N A Multipathing and load balancing supported with the Sun StorEdge Traffic Manager application Multipathing and load balancing through the Sun StorEdge Traffic Manager application with SunCluster 3 0 or VERITAS Cluster Server TABLE 1 ...

Page 24: ...orEdge 2G FC PCI Single Channel Network Adapter card Sun StorEdge 2G FC PCI Dual Channel Network Adapter card Supported Storage Devices Sun StorEdge A5200 and A3500FC arrays supported Sun StorEdge T3 and T3 arrays supported New Sun StorEdge T3 array firmware is supported The Sun StorEdge 39x0 69x0 and 99x0 series are also supported Third party Compatibility N A N A Interoperability capability with...

Page 25: ...iguration The use of longwave SFPs and long haul fiber optics allows users to reach geographically separated storage and servers perhaps for disaster recovery purposes The following limitations exist for cascading with the Sun STorEdge SAN 4 0 release If 1 and 2 gigabit switches are used together a maximum of 16 switches can be cascaded If only 2 gigabit switches are used a maximum of 64 switches ...

Page 26: ...8 Sun StorEdge SAN 4 0 Release Field Troubleshooting Guide October 2002 ...

Page 27: ...hapter contains the following sections Supported Hardware on page 10 Supported Configurations on page 12 Operating Environments on page 12 Hosts on page 13 Storage Arrays on page 14 Host Bus Adapters on page 15 Software Packages and Patches on page 16 Switches on page 18 Switch Port Types on page 19 Zones on page 21 Configuration Guidelines on page 22 Configuration Examples on page 24 ...

Page 28: ... 3960 Sun StorEdge 39x0 storage series 6910 6960 Sun StorEdge 69x0 storage series 9910 9960 Sun StorEdge 99x0 storage series X6799A Sun StorEdge PCI Single Fibre Channel Network Adapter X6727A Sun StorEdge PCI Dual Fibre Channel Network Adapter X6748A Sun StorEdge cPCI Dual Fibre Channel Network Adapter X6757A Sun StorEdge SBus Dual Fibre Channel Host Bus Adapter X6767A Sun StorEdge 2G FC PCI Sing...

Page 29: ...SC X9724A 15 meter fiber cable LC SC X9732a 2 meter fiber cable LC LC X9733a 5 meter fiber cable LC LC X9734a 15 meter fiber cable LC LC 1 You must use a long wave SFP and corresponding long wave fiber cable if you cascade more than 500 meters TABLE 2 1 Supported Hardware Continued Model Part Number or System Code Description ...

Page 30: ...nsure switch redundancy See the example diagrams in this chapter for more information on the supported configurations Operating Environments TABLE 2 2 Sun StorEdge SAN 4 0 Release Sun Operating Environment Compatibility Matrix Operating Environment Version Notes Sun Solaris 2 6 Not supported Sun Solaris 7 Not supported Sun Solaris 8 02 02 Update 7 or later Sun Solaris 9 ...

Page 31: ...art of the Sun StorEdge Network Foundation Software Storage Automated Diagnostic Environment 2 1 To find all required patches http sunsolve Sun COM Product Patches PatchPro Network Storage Products or Solaris Recommended Patch Cluster Describe your system then click Generate Patch List PCI X6799A2 X6727A3 2 Sun StorEdge PCI Single Fibre Channel Network Adapter Amber 3 Sun StorEdge PCI Dual Fibre C...

Page 32: ...zone is supported You must be using PCI dual Fibre Channel Network Adapter and PCI single Fibre Channel Network Adapter HBAs Storage Arrays TABLE 2 4 Sun StorEdge SAN 4 0 Release Storage Array Compatibility Matrix Firmware Levels for Storage Version Notes Sun StorEdge T3 array 1 17b and 1 18 controller firmware Translated loop TL switch mode Sun StorEdge T3 array 2 1 controller firmware TL fabric ...

Page 33: ...ts for simple arrays T3WG but 4 initiators 2 hosts for a partner pair T3ES Each host has one path to each of the Sun StorEdge T3 arrays in the partner pair TABLE 2 6 Sun StorEdge SAN 4 0 Release HBA Compatibility Matrix FW Code Levels for HBAs and I O Boards Version X6757A Sun StorEdge SBus Dual Fibre Channel Host Bus Adapter 1 13 06 or higher X6799A Sun StorEdge PCI Single Fibre Channel Network A...

Page 34: ...TCHPRO Interactive menu is displayed 5 Select all the appropriate features of your system in the following areas of the menu Operating System Release Platform 6 Click Generate Patch List To generate the most recent patch list for a specific Sun StorEdge SAN 4 0 Release Configuration 1 Access the SunSolve web site http sunsolve Sun COM The SUNSOLVE ONLINE menu is displayed 2 Under SunSolve Contents...

Page 35: ...LE 2 7 may not be present in all configurations TABLE 2 7 Unbundled Software Package Minimum Revision Minimum Patch if any JAVA SDK JDK 1 3 02 StorageTek 9840 1 28 126 Instant Image 3 0 SNDR 3 0 Alternate Pathing 2 3 1 110722 01 110432 04 Sun Enterprise 3x00 4x00 5x00 6x00 Flash Prom 3 2 28 103346 29 Sun Fire 3800 4800 4810 6800 Flash Prom 5 11 6 111346 02 E450 Flash Prom 3 22 0 106122 09 E250 Fla...

Page 36: ...skSuite 4 2 1 See SunSolve for the latest patches StorTools 4 2 Extra functionality for V880 Storage Automated Diagnostic Environment 2 1 See SunSolve for the latest patches Sun StorEdge Network Storage Agent 2 1 See SunSolve for the latest patches Sun StorEdge Network Data Replicator 3 0 See SunSolve for the latest patches Sun StorEdge Component Manager 2 2 See SunSolve for the latest patches VER...

Page 37: ...ystems Port Descriptions Port Nomenclature Function E_Port Expansion or inter switch port A type of switch port that can be connected to an E_Port of another switch to in effect create a cascading interswitch link ISL F_Port Fabric port A fabric port that is point to point only not loop capable and used to connect N_Ports to the switch FL_Port Fabric loop port A fabric port that is loop capable an...

Page 38: ...ted loop port Loop port This port enables private devices to communicate with fabric or public devices In the Brocade switch this address translation is automatic In Sun StorEdge switches the private device must be configured on a TL_Port N A U_Port Universal Port This port can operate as an E_Port F_Port or FL_Port A port is defined as a U_Port when it is not yet fully connected or has not yet as...

Page 39: ...ort based and WWN based zones can overlap When creating overlapping NS zones one or more switch ports is in at least two zones When a port is in multiple zones one host or storage device attached to a switch port to be a member of many zones and resources can be shared If a resource is shared in multiple zones it can be made available to multiple zones by using overlapping zones When connecting mu...

Page 40: ... have local and remote storage in the same zone so that storage can be mirrored at both locations Configuration Guidelines Switches For high availability applications configure two sets of switches in parallel Zones and Arrays Sun StorEdge T3 arrays support name server zones or zones in which a host has made a point to point Fabric connection to a switch and the Sun StorEdge T3 array is attached t...

Page 41: ...es While the Hosts are Online You can add all initial and additional storage devices while the host is online In high availability configurations where alternative methods to reconstruct the data exist you can remove a device or path Host volume management or multi pathing software handles this device removal For non available configurations you must ensure that no host application is configured t...

Page 42: ...d to One Storage Array FIGURE 2 1 shows one host connected through fiber optic cables to a Sun StorEdge T3 array enterprise configuration FIGURE 2 1 Single Host Connected to One Sun StorEdge T3 Array Enterprise Configuration Host Host Adapter Host Adapter Switches Sun StorEdge T3 array partner pair Fibre optic cables ...

Page 43: ...vices are on different zones Each controller that is connected to a switch must have a unique loop ID Whenever you add a second controller to a switch make sure that the loop ID of the controller being connected is different from the loop ID of any other controller currently connected to the same switch Caution Make sure that the controller module of the array is split between two switches For exa...

Page 44: ...4 0 Release Field Troubleshooting Guide October 2002 FIGURE 2 2 Single Host Connected to Multiple Sun StorEdge T3 Array Enterprise Configurations Sun StorEdge T3 array partner pairs Host Switches Host Adapter Host Adapter ...

Page 45: ... can attach different storage types to the same switch so long as the storage devices are on different zones Each controller that is connected to a switch must have a unique loop ID Whenever you add a second controller to a switch make sure that the loop ID of the controller being connected is different from the loop ID of any other controller currently connected to the same switch Caution Ensure ...

Page 46: ...se Field Troubleshooting Guide October 2002 FIGURE 2 3 Two Hosts Connected to Four Sun StorEdge T3 Array Enterprise Configurations Sun StorEdge T3 partner pairs Switches Host Host Host Adapter Host Adapter Host Adapter Host Adapter ...

Page 47: ...Group Each Host with Separate Non shared Storage Note You must enable Sun StorEdge Traffic Manager software for failover across multiple hosts to function The mp_support on the Sun StorEdge T3 array should be set to mpxio Sun StorEdge Traffic Manager Software Sun StorEdge L180 or L700 FC Tape Library Sun StorEdge T3 partner pairs Sun StorEdge A5200 Array Switch 0 Switch 1 Sun Enterprise 420 Sun En...

Page 48: ...30 Sun StorEdge SAN 4 0 Release Field Troubleshooting Guide October 2002 ...

Page 49: ... Network Fibre Channel Switch 16 Detailed installation and configuration information can be found in the respective documentation of the tools This chapter contains the following sections Diagnostic Tools on page 32 Storage Automated Diagnostic Environment Version 2 1 on page 32 Sun Explorer Data Collector SUNWexplo and T3Extractor on page 40 Diagnosing and Troubleshooting the Sun Switch on page 4...

Page 50: ...ent version 2 1 is a host based online health and diagnostic monitoring tool for a storage area network SAN and direct attached storage DAS devices It can be configured to monitor on a 24 hour basis collecting information that enhances the reliability availability and serviceability RAS of the storage devices FIGURE 3 1 Storage Automated Diagnostic Environment Version 2 1 Home Window ...

Page 51: ...ic Environment Version 2 1 Functions For each device the Storage Automated Diagnostic Environment version 2 1 performs the following functions 1 Sends the information by way of a discovery event to the system administrator through an interface with the transport mechanisms Note The first access to a device yields a discovery event that collects all the information about that device plus other even...

Page 52: ... Storage Automated Diagnostic Environment User s Guide Version 2 1 Sun StorEdge PCI FC 100 Host Adapter Board Test ifptest Sun StorEdge PCI Dual Fibre Channel Host Adapter Board Test qlctest Sun StorEdge SBus FC 100 Host Adapter Board Test socaltest Sun StorEdge Network FC Switch 16 Switch Test switchtest Sun StorEdge T3 and T3 array Tests t3ofdg t3test t3volverify Virtualization Engine Tests vedi...

Page 53: ...in the Storage Automated Diagnostic Environment home window Three links are then displayed below the tab as shown in FIGURE 3 2 FIGURE 3 2 Storage Automated Diagnostic Environment Diagnose Tab Selected 2 Click the Diagnostic Tests link Five tests are displayed as shown in FIGURE 3 3 ...

Page 54: ...operate on in band or out of band data paths The Storage Automated Diagnostic Environment causes the test to be run on the appropriate Host Storage Automated Diagnostic Environment s implementation of diagnostic tests verify the operation of all the user selected components Tests are selected from a graphical view of the system s topology The Storage Automated Diagnostic Environment version 2 1 Gr...

Page 55: ...Chapter 3 Diagnostics 37 FIGURE 3 4 Storage Automated Diagnostic Environment Test from Topology Window ...

Page 56: ...38 Sun StorEdge SAN 4 0 Release Field Troubleshooting Guide October 2002 FIGURE 3 5 Storage Automated Diagnostic Environment Test from Topology Window with Background Reduced to 66 ...

Page 57: ...Chapter 3 Diagnostics 39 FIGURE 3 6 Storage Automated Diagnostic Environment Test from Topology Window with Background Reduced to 66 and Components Arranged for Viewing ...

Page 58: ...ools that collect pertinent information you need to see the complete picture of the host Visit the following websites for more information and to download these tools Explorer http eservices central knowledge products explorer T3Extractor http hes west nws products T3 tools html Note You can gather the same information by querying the Storage Automated Diagnostic Environment version 2 1 that you c...

Page 59: ...n 4 Diagnostics Troubleshooting Using Switch Counter Information Switch counter information can be helpful in supporting troubleshooting the Sun StorEdge Network Fibre Channel Switch 16 Some general points to keep in mind when viewing switch counter information are Quickly increasing counter values or abnormally high counter values may indicate a problem A LIP that occurs on one port in a zone pro...

Page 60: ...s SAN index html The SAN Solutions menu is displayed 2 Click Other Documentation 3 Click Sun StorEdge tm Network 2Gb Switch 16 SANbox2 Management Manual See Section 4 Managing Ports qlctest Test If you are running the Storage Automated Diagnostic Environment version 2 1 application you can also run the Sun StorEdge PCI Dual Fibre Channel Host Adapter Board Test qlctest which might increase the fol...

Page 61: ...ration This chapter contains the following sections Example Configuration on page 44 Example Assumptions on page 45 Troubleshooting Outline on page 45 Troubleshooting Example of a Host to Switch Error on page 47 Determine the Error on page 47 Determine the Extent of the Problem on page 53 Check the Array Status on page 55 Check the Switch Status on page 56 Test the FRUs on page 57 Verify the Fix o...

Page 62: ...0 Release patches and packages Two Sun StorEdge T3 arrays in an enterprise configuration 1 LUN per array Two Sun StorEdge 2 Gbyte Fibre Channel switches One single port 2 Gbyte HBA and one dual port 2 Gbyte HBA Storage Automated Diagnostic Environment version 2 1 with patch 113230 01 The setup example high level topology is displayed in FIGURE 4 1 FIGURE 4 1 Troubleshooting Example Viewed with Sto...

Page 63: ...e This section lists the broad steps on how to approach a SAN problem It lays out a methodical approach and lists various tools and resources available at each step Using the Storage Automated Diagnostic Environment version 2 1 for monitoring vastly decreases the time consuming process of narrowing down the problem 1 Determine the error Storage Automated Diagnostic Environment version 2 1 alert em...

Page 64: ... can use the Storage Automated Diagnostic Environment version 2 1 to detect user configuration errors that may not show up as hard errors anywhere else For example a user might accidentally change a switch port to a different mode TL to F or rezone a switch 5 Test the FRUs Storage Automated Diagnostic Environment version 2 1 diagnostic tests switchtest and qlctest Sun StorEdge T3 array tests OFDG ...

Page 65: ...Troubleshooting Example 47 Troubleshooting Example of a Host to Switch Error Determine the Error The first indication of a problem can come from a Storage Automated Diagnostic Environment version 2 1 email alert ...

Page 66: ...n 0 failover Site FSDE LAB Broomfield CO Source diag226 central sun com Severity Warning Category MESSAGE DeviceId message diag226 central sun com EventType LogEvent driver SSD_WARN EventCode 9 20 330 EventTime 2002 09 13 13 06 26 DESCRIPTION Found 1 driver SSD_WARN warnings s in logfile var adm messages on diag226 central sun com id 80fee746 INFORMATION These warnings could indicate a faulty link...

Page 67: ...CO Source diag226 central sun com Severity Warning Category MESSAGE DeviceId message diag226 central sun com EventType LogEvent driver SCSI_TRAN_FAILED EventCode 9 20 318 EventTime 2002 09 13 13 06 26 DESCRIPTION Found 1 driver SCSI_TRAN_FAILED warnings s in logfile var adm messages on diag226 central sun com id 80fee746 INFORMATION The SCSI driver is posting warnings RECOMMENDED ACTION 1 Check fo...

Page 68: ...al sun com EventType LogEvent driver MPXIO_offline EventCode 9 20 313 EventTime 2002 09 13 13 06 27 DESCRIPTION Found 4 driver MPXIO_offline warnings s in logfile var adm messages on diag226 central sun com id 80fee746 INFORMATION The MPxIO multipathing software has noted the path to a storage device has gone offline RECOMMENDED ACTION 1 Check the Topology View to see what device s are affected 2 ...

Page 69: ... path pci 1f 2000 SUNW qlc 1 fp 0 0 fp4 to target address 50020f23000003d5 1 is offline Sep 13 13 05 36 WWN 50020f23000003d5 diag226 Central Sun COM mpxio ID 779286 kern info scsi_vhci ssd g60020f20000003d53d349365000c1691 ssd3 multipath status degraded path pci 1f 2000 SUNW qlc 1 fp 0 0 fp4 to target address 50020f23000003d5 0 is offline Site FSDE LAB Broomfield CO Source diag226 central sun com ...

Page 70: ...4 went offline continuation Site FSDE LAB Broomfield CO Source diag226 central sun com Severity Error Actionable Category SWITCH2 DeviceId switch2 100000c0dd00bfda EventType StateChangeEvent M port 0 EventCode 12 26 35 EventTime 2002 09 13 13 06 35 DESCRIPTION port 0 in SWITCH2 sw 67 84 ip 172 20 67 84 is now Not Available state changed from online to offline INFORMATION A port on the switch2 has ...

Page 71: ...y of the Storage Automated Diagnostic Environment version 2 1 to see if any problems are shown An example is shown in FIGURE 4 2 FIGURE 4 2 Troubleshooting Example View 2 From FIGURE 4 2 it can be seen that the error is only affecting a single path This can be confirmed by using the cfgadm command ...

Page 72: ...one switch cfgadm al Ap_Id Type Receptacle Occupant Condition c0 scsi bus connected configured unknown c0 dsk c0t0d0 disk connected configured unknown c0 dsk c0t1d0 disk connected configured unknown c1 scsi bus connected configured unknown c1 dsk c1t6d0 CD ROM connected configured unknown c2 fc connected unconfigured unknown c3 fc connected unconfigured unknown c4 fc private connected unconfigured...

Page 73: ...ere is probably an upstream path problem t3b2 2 fru stat CTLR STATUS STATE ROLE PARTNER TEMP u1ctr ready enabled master u2ctr 41 5 u2ctr ready enabled alt master u1ctr 39 0 t3b2 3 port list port targetid addr_type status host wwn u1p1 4 hard online sun 50020f23000003d5 u2p1 5 hard online sun 50020f23000003c5 t3b2 4 port listmap port targetid addr_type lun volume owner access u1p1 4 hard 0 vol1 u2 ...

Page 74: ...e display of the Storage Automated Diagnostic Environment as shown in FIGURE 4 3 FIGURE 4 3 Troubleshooting Example View 3 FIGURE 4 3 indicates that the problem is that the switch Port 0 has gone offline It also shows that the only other device that is affected is the host This indicates a host switch connection problem ...

Page 75: ...ironment switchtest and qlctest Tests 1 Remove one end of the cable of the HBA switch link 2 Insert loopback plug into the HBA 3 Run the qlctest If the test fails replace HBA and re run the qlctest If the test passes continue below 4 Insert loopback plug into Switch SFP Port 5 Run the switchtest If the test passes most likely problem is cable If the test fails continue below 6 Replace SFP and re r...

Page 76: ...AL ERROR Didn t detect loop as being online and user selected external loopback option Return code from checking path devices pci 1f 2000 SUNW qlc 1 fp 0 0 devctl was 131337 qlctest failed error code 256 Remove FC Cable from hba devices pci 1f 2000 SUNW qlc 1 fp 0 0 devctl Insert FC Loopback Cable into hba devices pci 1f 2000 SUNW qlc 1 fp 0 0 devctl Continue Isolation qlctest started on hba port ...

Page 77: ... successfully qlctest completed successfully error code 0 Remove FC Loopback Cable from hba devices pci 1f 2000 SUNW qlc 1 fp 0 0 devctl Restore ORIGINAL FC Cable into hba devices pci 1f 2000 SUNW qlc 1 fp 0 0 devctl ORIGINAL hba devices pci 1f 2000 SUNW qlc 1 fp 0 0 devctl is Functional Remove FC Cable from switch2 100000c0dd00bfda sw 67 84 port 0 Insert FC Loopback Cable into switch2 100000c0dd0...

Page 78: ...e7e7e switch2test Started Connected to 172 20 67 84 Switch Model type is SANbox2 16 Power and Fans are okay Detected a loopback plug inserted onto this port Get original port counters for port 0 Detected port type Not Initialized External loopback test passed Get port counters after testing for port 0 Compare of port counters passed Test Passed switch2test completed successfully error code 0 Remov...

Page 79: ...003c5 are connected and configured cfgadm al Ap_Id Type Receptacle Occupant Condition c0 scsi bus connected configured unknown c0 dsk c0t0d0 disk connected configured unknown c0 dsk c0t1d0 disk connected configured unknown c1 scsi bus connected configured unknown c1 dsk c1t6d0 CD ROM connected configured unknown c2 fc connected unconfigured unknown c3 fc connected unconfigured unknown c4 fc privat...

Page 80: ...DEVICE PROPERTIES for disk dev rdsk c9t60020F20000003D53D349365000C1691d0s2 Status Port A O K Status Port B O K Vendor SUN Product ID T300 WWN Node 50020f20000003c5 WWN Port A 50020f23000003d5 WWN Port B 50020f23000003c5 Revision 0201 Serial Num Unsupported Unformatted capacity 51203 250 MBytes Write Cache Enabled Read Cache Enabled Minimum prefetch 0x0 Maximum prefetch 0x0 Device Type Disk device...

Page 81: ...us Port A O K Status Port B O K Vendor SUN Product ID T300 WWN Node 50020f20000003d5 WWN Port A 50020f23000003d5 WWN Port B 50020f23000003c5 Revision 0201 Serial Num Unsupported Unformatted capacity 51203 250 MBytes Write Cache Enabled Read Cache Enabled Minimum prefetch 0x0 Maximum prefetch 0x0 Device Type Disk device Path s dev rdsk c9t60020F20000003D53D349365000C1691d0s2 devices scsi_vhci ssd g...

Page 82: ...64 Sun StorEdge SAN 4 0 Release Field Troubleshooting Guide October 2002 ...

Page 83: ...be how to install a new SAN system using Brocade Communications Systems Inc Silkworm switch Installing a New SAN on page 66 Downloading Patches and Packages on page 67 Installing the Software on page 69 Installing Firmware on Brocade Communications Systems Silkworm Switches on page 72 Upgrading the SAN on page 76 ...

Page 84: ...ager is a separately installed software product that provides host based storage management such a disk labeling mirroring striping and RAID 5 Brocade Webtools Brocade switches support Java enabled Webtools Brocade Webtools is a GUI that provides management capabilities such as maintaining zones setting port attributes and setting up cascaded switches cfgadm plug in for Fabric on demand node creat...

Page 85: ...onitoring and diagnostic testing Downloading Patches and Packages You can download the required software components from the following web sites listed in TABLE A 1 To Verify Successful Patch Downloads 1 Use one of the following three utilities to obtain the checksum value of the patch you downloaded CODE EXAMPLE A 1 Solaris usr bin sum Utility Note The sum utility can also be located in the usr u...

Page 86: ... or from http sunsolve Sun com md5 md5 tar z 2 Compare the checksum value that is displayed to the patch checksum value given at the checksum File link http sunsolve Sun com If the values are identical the patches were properly downloaded Note The checksum file at http sunsolve Sun com is approximately 614 Kbytes md5sum 108982 09 zip 1297fcd385f618c726489c6c7f57c900 108982 09 zip ...

Page 87: ...s 8 Recommended and Security patch cluster 3 SUNWsan Sun StorEdge SAN Foundation Kit 4 SUNWcfpl cfgadm plug in 32 bit package 5 SUNWcfplx cfgadm plug in 64 bit package 6 111412 07 Sun StorEdge Traffic Manager 7 111095 07 fcp fp fctl usoc drivers patch 8 111096 03 fcip driver patch 9 111097 07 qlc driver patch 10 111413 07 luxadm liba5k and libg_fc patch 11 111846 03 cfgadm plug in library patch 12...

Page 88: ...ch installation instructions and notes 3 Install the SUNWsan package 4 Install cfgadm plug in library packages SUNWcfpl and SUNWcfplx 5 Install the SAN Foundation Kit SUNWsan patch 111847 04 or higher if required 6 InstallSun StorEdge Traffic Manager patch 111412 07 7 Install fctl fp fcp usoc driver patch 111095 xx 8 Install fcip driver patch 9 Install qlc driver patch pkgadd d SUNWsan pkgadd d pk...

Page 89: ... Reboot the system For each of the storage devices upgrade the software firmware or configuration After the above steps you can leverage additional features provided by Brocade Silkworm 2400 8 port 2800 16 port 3800 16 port and 12000 32 64 128 port for Sun StorEdge Traffic Manager functionality additional fabric zones additional initiators per zone host fabric connectivity cascaded switch configur...

Page 90: ... Brocade Partner Network link 3 Enter the Sun internal login Enter the Sun internal password 4 Under Services and Support click Firmware 5 Click the appropriate firmware version see TABLE B 3 6 Download the appropriate firmware version see TABLE B 3 UNIX version and the Readme txt file to your local host To Install Firmware from UNIX Solaris Follow these steps 1 From the Brocade web site retrieve ...

Page 91: ...file Note If you are logged in as a normal user and not as root the rhosts file is referred to the user s home directory rhosts file For example if a normal user named nancy is logged in she would edit the file home nancy rhosts 4 If you are using a UNIX system with Solaris installed check the etc nsswitch conf file to make sure the hosts lookup table is appropriately set vi etc hosts IP_address s...

Page 92: ...shd If you invoke the command with three parameters rshd is used If you invoke the command with four parameters ftp is used 1 From a UNIX system telnet into the switch and download the firmware with the firmwareDownload command 2 To check the syntax type Note With version 2 1 and higher commands are not case sensitive 3 Check the syntax by typing firmwaredownload and following the screen prompts S...

Page 93: ...sion as shown in CODE EXAMPLE A 4 CODE EXAMPLE A 4 Verification of Firmware Version oem240 admin firmwareDownload Server Name or IP Address host 10 32 99 29 User Name user root File Name usr switch firmware var tmp v2 6 x Protocol RSHD or FTP rshd ftp Password 84776 3832 130980 csum 2ef6 loading to ram writing flash 0 writing flash 1 download complete oem240 admin fastboot sw5 admin firmwareDownlo...

Page 94: ... 1 on page 84 for the supportability matrix To Upgrade the Software If you have multiple hosts on your SAN you can upgrade them simultaneously or one at a time without affecting your SAN environment Hosts that are not being upgraded will not be affected during the upgrade You can upgrade the host software one host at a time or several hosts in parallel Caution Your system will be unavailable to us...

Page 95: ...8 02 02 Update 7 For information on how to upgrade your systems refer to Solaris 8 Installation Supplement part number 806 5182 available at http docs sun com Sun StorEdge SAN 4 0 Release The packages on your system that were previously used should be available To verify their availability use the pkginfo command pkg_name is the name of the package on which you need to obtain the information 1 Upg...

Page 96: ... 01 has not been installed install it using the patchadd command 2 If your system does not have the SUNWsan package installed install the new SUNWsan package from your Sun StorEdge SAN 4 0 Release software cfgadm Plug in Library Packages 1 Install cfgadm plug in library packages SUNWcfpl and SUNWcfplx system SUNWsan SAN Foundation Kit showrev p grep 111847 Patch 111847 01 Obsoletes Requires Incomp...

Page 97: ...g the SUNWstade package and the Brocade Communications Systems patch For detailed installation and usage instructions for the Storage Automated Diagnostic Environment version 2 1 refer to the Storage Automated Diagnostic Environment User s Guide Version 2 1 1 If your SAN Management host is not running the current version remove the existing package and install the latest version Remove the old pac...

Page 98: ...torEdge SAN 4 0 Release Field Troubleshooting Guide October 2002 5 Check your SAN Management host to verify the version of the Storage Automated Diagnostic Environment version 2 1 installed pkginfo l SUNWstade ...

Page 99: ... that of a configuration that contains the current Sun StorEdge Network Fibre Channel family of switches Current support is limited to diagnosing failures down to the FRU level In Sun s support model the entire Silkworm switch is considered a FRU Many of Brocade s internal diagnostics and messages while useful for depot or Root Cause Analysis situations are not ultimately relevant to a Sun Service...

Page 100: ...800 Hardware Reference Brocade Silkworm 12000 Hardware Reference Manual Brocade Fabric OS Reference Brocade Fabric OS Release Notes Brocade Fabric OS Procedures Guide Brocade WebTools User s Guide Brocade Zoning User s Guide Brocade QuickLoop User s Guide Sun Documentation The Sun StorEdge switch documents are referenced for overall configuration guidelines Sun StorEdge SAN 4 0 Release Installatio...

Page 101: ...configurations follow the same rules for maximum number of initiators supported number of arrays per zone and other hardware specific information Refer to Chapter 2 Configurations of this guide for supported hardware configurations Brocade Communications Systems Silkworm switch configurations and Sun switch configurations have the minimum software package requirements shown in TABLE B 1 ...

Page 102: ...raffic Manager Software as part of Sun StorEdge Network Foundation Software Storage Automated Diagnostic Environment 2 1 To find all required patches http sunsolve Sun COM Product Patches PatchPro Network Storage Products or Solaris Recommended Patch Cluster Describe your system then click Generate Patch List PCI X6799A2 X6727A3 2 Sun StorEdge PCI Single Fibre Channel Network Adapter Amber 3 Sun S...

Page 103: ...Switches Firmware Switch Software Licenses Brocade Silkworm 24001 1 Brocade Silkworm 2400 2800 and 3800 FC Switches may be intermixed Interoperability with other vendor switches is not supported at this time v2 6 0c Fabric OS v2 6 0c Zoning Quickloop Webtools Brocade Silkworm 2800 v2 6 0c Fabric OS v2 6 0c Zoning Quickloop Webtools Brocade Silkworm 38002 2 Brocade Silkworm 3800 FC Switches are sup...

Page 104: ...te features of your system in the following areas of the menu OS Release Platform Disk Array Tape Libraries Disk Drives Tape Drives Switches and HBAs SAN Products Brocade SAN Release Software 6 Click Generate Patch List TABLE B 4 Application Supportability Matrix with Solaris 8 02 02 Update 7 or Later Name Version Patches VERITAS Volume Manager 3 2 GA To find all VERITAS Volume Manager patches htt...

Page 105: ...rsion 2 1 and Brocade Switches The minimum Brocade Silkworm switch firmware to use with Storage Automated Diagnostic Environment version 2 1 is v2 6 0c Diagnostic Tools The tools available for troubleshooting differ from the original release of the Sun StorEdge SAN 4 0 Release Since then Sun StorEdge StorTools 4 x and Network Storage Agent 2 1 have had their functionality combined into a single di...

Page 106: ...172 20 67 167 passwd password iterations 1000 Called with options dev 5 172 20 67 167 passwd xxxxxxx iterations 1000 Connect to 172 20 67 167 Opened 172 20 67 167 Logged into 172 20 67 167 Clear port errors send diagClearError 5 Port errors cleared port is in loopback mode Running command CrossPortTest 1000 1 Note You should only have a loopback on port 5 If you have more than one loopback install...

Page 107: ...ts spinFab for testing E Port connections between switches and loopPortTest for testing L Ports supportShow switchShow qlShow diagShow crossPortTest loopPortTest spinFab nsShow supportShow supportShow runs nearly all commands and should be gathered when placing a service call or escalation The Explorer Data Collection utility SUNWexplo gathers the supportShow output if the Brocade Silkworm switch ...

Page 108: ... 167 Connected to 172 20 67 167 Escape character is Fabric OS tm Release v2 6 0 login admin Password diag167 admin supportshow Kernel 5 4 Fabric OS v2 6 0 Made on Tue Jan 15 15 10 28 PST 2002 Flash Tue Jan 15 15 12 04 PST 2002 BootProm Thu Jun 17 15 20 39 PDT 1999 26 25 26 25 27 Centigrade 78 77 78 77 80 Fahrenheit Power Supply 1 is absent ...

Page 109: ...oning ON Main port 0 sw Online E Port 10 00 00 60 69 10 71 25 diag164 upstream port 1 No_Module port 2 sw Online F Port 21 01 00 e0 8b 23 61 f9 port 3 No_Module port 4 No_Module port 5 No_Module port 6 sw Online E Port 10 00 00 60 69 10 71 25 diag164 port 7 sw Online F Port 21 00 00 e0 8b 03 61 f9 diag167 admin qlshow Self 10 00 00 60 69 20 1e fc domain 1 State Master Scope single AL_PA bitmap 300...

Page 110: ...ector plug inserted diag167 admin diagshow nTicks 0 Max 4473924 Diagnostics Status Tue Mar 19 14 04 30 2002 port 0 1 2 3 4 5 6 7 diags OK OK OK OK OK OK OK OK state UP DN UP DN DN DN UP UP pt0 4086880 frTx 64382622 frRx 0 LLI_errs pt2 38616950 frTx 300398 frRx 12 LLI_errs pt6 28852033 frTx 235091315 frRx 111 LLI_errs pt7 331090679 frTx 8930476 frRx 31 LLI_errs Central Memory OK Total Diag Frames T...

Page 111: ...ports with Loopback connectors use crossPortTest CODE EXAMPLE B 7 spinFab Example Output diag164 admin loopporttest 100 2 0x7e7e7e7e 4 Configuring L port 2 to Cable Loopback Port done Will use pattern 7e7e7e7e 7e7e7e7e 7e7e7e7e 7e7e7e7e Running Loop Port Test passed Configuring Loopback L port s back to normal L port s done diag167 admin spinfab 1 0 0 spinFab0 running spinFab0 Completed 1 megs sta...

Page 112: ...ted to the switch diag164 admin nsshow Type Pid COS PortName NodeName TTL sec NL 0312e4 3 50 02 0f 23 00 00 3d 2c 50 02 0f 20 00 00 3d 2c na FC4s FCP SUN T300 0118 Fabric Port Name 20 02 00 60 69 10 71 25 NL 031ee8 3 50 02 0f 23 00 00 3e e5 50 02 0f 20 00 00 3e e5 na FC4s FCP SUN T300 0118 Fabric Port Name 20 0e 00 60 69 10 71 25 The Local Name Server has 2 entries ...

Page 113: ...s defined as a G_Port when it is not yet fully connected or has not yet assumed a specific function in the fabric GL_Port Generic loop port This port can automatically configure as either an E_Port F_Port or an FL_Port A port is defined as a G_Port when it is not yet fully connected or has not yet assumed a specific function in the fabric TABLE B 6 Differences Between Sun StorEdge and Brocade Port...

Page 114: ...mmands that the switch supports However the screen is limited in size and messages are restricted to one or two lines of output Once the IP addressed is configured through the front panel further switch setup and diagnostics can be run via a telnet connection or the WebTools GUI See the Brocade Silkworm 2800 Hardware Reference Manual for more details on the front panel operation WebTools GUI The W...

Page 115: ...cade Communications Systems Switch Troubleshooting 97 FIGURE B 1 Brocade Webtools GUI See the Brocade Web Tools User s Guide for more information on WebTools usage Note The rest of this guide will assume telnet usage ...

Page 116: ...est POST execution per warm boot executes a shorter version of Memory Test Boot time with POST varies depending on boot method As the POST test successfully performs each test a message Passed is displayed via telnet on the front panel After the switch completes the POST the port module returns to a steady state from the flashing state shown during tests If a yellow port module light is displayed ...

Page 117: ...nition No light showing No light or signal carrier no module no cable for media interface LEDs Steady yellow Receiving light or signal but not yet online Slow1 yellow Disabled result of diagnostics or portDisable command Flashes every two seconds Fast2 yellow Error fault with port Flashes every 1 2 second Steady green Online connected with device Slow1 green Online but segmented loopback cable or ...

Page 118: ...on to bring connected ports online 7 Fabric analysis the switch checks for ports connected to other Fabric elements If there are other Fabric elements connected it identifies the master switch 8 Address assignment once the master switch has been identified port addresses may be assigned Each switch tries to keep the same addresses that were previously used These are stored in the switch s configur...

Page 119: ...output luxadm e port output Storage Automated Diagnostic Environment version 2 1 Topology error display Multipathing information Sun StorEdge Traffic Manager and VxDMP Note The information gathered here will determine which subsection to focus your attention Host to Switch Switch to Switch cascaded or Switch to Storage 3 Check Array Status Open a telnet session to the Sun StorEdge T3 array Refer t...

Page 120: ...rossPortTest Sun StorEdge T3 Array tests such as T3OFDG 1M Sun StorEdge A3500FC arrays Healthcheck Note The conclusion of these tests isolate the problem to a FRU to be replaced Follow the appropriate hardware manual for proper FRU replacement procedures 6 Verify the fix var adm messages path online multipath informational messages Storage Automated Diagnostic Environment version 2 1 status Sun St...

Page 121: ...ted using the dex disk exerciser to simulate customer load and the steps below allowed the I O to continue uninterrupted throughout the procedure Configuration Sun Fire V880 Solaris 8 02 02 Update 7 with all recommended and latest Sun StorEdge Network Foundation Software patches Sun StorEdge T3 array Partner Pair with FW 1 18 Brocade Silkworm 2400 and 2800 switches with v2 6 0 firmware Storage Aut...

Page 122: ... Version 2 1 Topology In FIGURE B 2 a Sun StorEdge T3 array enterprise configuration is connected to a cascaded switch In another possible configuration two separate switches can be used to eliminate a single point of failure FIGURE B 2 Storage Automated Diagnostic Environment Version 2 1 Test from Topology Window ...

Page 123: ...ology GUI to identify failing segment of the data path 3 Verify correct FC switch configuration 4 Verify port is enabled Site Lab Broom Source diag229 central sun com Severity Error Actionable Category MESSAGE DeviceId message diag229 central sun com EventType LogEvent driver QLC_LOOP_OFFLINE EventCode 9 20 315 EventTime 2002 07 11 10 32 45 Found 1 driver QLC_LOOP_OFFLINE errors s in logfile var a...

Page 124: ...ntral sun com changed from O K to Missing target t3 t3 67 166 172 20 67 166 INFORMATION luxadm display reported a change in the port status of one of it s paths The agent then tries to find which enclosure this path corresponds to by reviewing it s database of T3 s and VE s luxadm display 2a00006022004188 DEVICE PROPERTIES for disk 2a00006022004188 Status Port A O K monitoring this field Vendor SU...

Page 125: ...atusA on diag229 central sun com changed from O K to Missing target t3 t3 67 166 172 20 67 166 INFORMATION luxadm display reported a change in the port status of one of it s paths The agent then tries to find which enclosure this path corresponds to by reviewing it s database of T3 s and VE s luxadm display 2a00006022004188 DEVICE PROPERTIES for disk 2a00006022004188 Status Port A O K monitoring t...

Page 126: ...Edge T3 arrays shows the following cfgadm al c3 fc fabric connected configured unknown c3 50020f23000068cc unavailable connected configured unusable c4 fc private connected unconfigured unknown c5 fc fabric connected configured unknown c5 210000e08b05041c unknown connected unconfigured unknown c5 50020f2300004331 disk connected configured unknown luxadm display dev rdsk c6t60020F2000003EE53AAF7A09...

Page 127: ... Switch Troubleshooting 109 From the topology notice the HBA and port two of the first switch have errors Note From this Topology view concentrate on the link between the HBA and the switch port 2 Ports highlighted by the color red are circled ...

Page 128: ...ess u1p1 1 hard 0 diag169u1v1 u1 primary u1p1 1 hard 1 diag169u2v1 u1 failover u2p1 2 hard 0 diag169u1v1 u1 failover u2p1 2 hard 1 diag169u2v1 u1 primary diag167 admin switchshow switchName diag167 switchType 3 4 switchState Online switchMode Native switchRole Subordinate switchDomain 1 switchId fffc01 switchWwn 10 00 00 60 69 20 1e fc switchBeacon OFF Zoning ON Main port 0 sw Online E Port 10 00 ...

Page 129: ...s Note Before starting the Link Test you must enter the password for the Brocade switch in the configuration menu a Using the Storage Automated Diagnostic Environment version 2 1 right click on the box on the link that connects the HBA and switch port A pop up menu appears b From the menu click on Start Link Test The Link Test components are displayed on the right side of the window See FIGURE B 4...

Page 130: ... October 2002 FIGURE B 4 Storage Automated Diagnostic Environment version 2 1 Link Test Display The Link Test starts by running the HBA Test In this example the HBA Test fails The Link Test then requests you to insert a loopback cable into the HBA See FIGURE B 5 ...

Page 131: ...Appendix B Brocade Communications Systems Switch Troubleshooting 113 FIGURE B 5 Test Result Details with Remedy Request ...

Page 132: ...hooting Guide October 2002 The Link Test then runs the HBA Test again This time the HBA Test succeeds and you are requested to reconnect the loopback cable into the HBA as shown in FIGURE B 6 FIGURE B 6 Test Result Details Showing a Successful Test ...

Page 133: ... 115 The Link Test new runs the Switch Port Test In this example the Switch Port Test passes The Link Test then requests you to insert a new fiber cable between the HBA and the Brocade switch port as shown in FIGURE B 7 FIGURE B 7 Continued Link Test Example Results ...

Page 134: ...ield Troubleshooting Guide October 2002 The Link Test then reruns the HBA Test This time the HBA Test passes and the Link Test indicates that the fiber cable is the suspected failure cause FIGURE B 8 Continued Link Test Example Results ...

Page 135: ... c3 device is connected b Check the status of the device with the luxadm failover command cfgadm al c3 fc fabric connected configured unknown c3 50020f23000068cc disk connected configured unusable c4 fc private connected unconfigured unknown c5 fc fabric connected configured unknown c5 210000e08b05041c unknown connected unconfigured unknown c5 50020f2300004331 disk connected configured unknown ...

Page 136: ...0068cc WWN Port A 50020f23000068cc Revision 0200 Serial Num Unsupported Unformatted capacity 119514 500 MBytes Write Cache Enabled Read Cache Enabled Minimum prefetch 0x0 Maximum prefetch 0x0 Device Type Disk device Path s dev rdsk c3t50020F23000068CCd0s2 devices pci 9 600000 pci 2 SUNW qlc 5 fp 0 0 ssd w50020f23000068cc 0 c raw DEVICE PROPERTIES for disk 50020f23000068cc Status Port B O K Vendor ...

Page 137: ...Appendix B Brocade Communications Systems Switch Troubleshooting 119 FIGURE B 9 Storage Automated Diagnostic Environment Version 2 1 Test from Topology Window ...

Page 138: ...120 Sun StorEdge SAN 4 0 Release Field Troubleshooting Guide October 2002 ...

Page 139: ...de Communications Systems Error Messages This appendix explains the error message format and possible errors and contains the following topics Error Message Formats on page 122 Diagnostic Error Message Formats on page 123 ...

Page 140: ...RAM and are lost whenever power is removed from the switch Access the error message log to view error messages before removing power Front Panel Message Formats The Brocade Silkworm switch s front panel displays error messages The first line includes the error s date and time The beginning of each second line on the front panel display starts with the module name error name and the severity level ...

Page 141: ...earError port command should only be used during diagnostic procedures to reset a bad port for retest Some messages contain the following abbreviations sb Should Be er Bits in error Note If you run the portStateShow or the diagShow command prior to running a test errors may appear as a result of the normal synchronization process These errors should be addressed if the number of errors found incre...

Page 142: ...board assembly crossPortTest replace mainboard assembly SFP or fiber cable spinSilk replace mainboard assembly SFP or fibre cable TABLE C 2 Error Message Codes Defined Error Number Test Name Error Name 0001 n a DIAG CLEAR_ERR 0004 n a DIAG POST_SKIPPED 0B15 sramRetentionTest DIAG REGERR 0B16 DIAG REGERR_UNRST 0B0F DIAG BUS_TIMEOUT 1F25 cmemRetentionTest DIAG LCMRS 1F26 DIAG LCMTO 1F27 DIAG LCMEM 0...

Page 143: ...MEMTX 1029 DIAG CMNOBUF 102A DIAG CMERRTYPE 102B DIAG CMERRPTN 102C DIAG INTNOTCLR 103O DIAG BADINT 106F DIAG TIMEOUT 2030 cmiTest DIAG BADINT 2031 DIAG INTNIL 2032 DIAG CMISA1 2033 DIAG CMINOCAP 2034 DIAG CMIINVCAP 2035 DIAG CMIDATA 2036 DIAG CMICKSUM 223B camTest DIAG CAMINIT 223C DIAG CAMSID TABLE C 2 Error Message Codes Defined Continued Error Number Test Name Error Name ...

Page 144: ... DIAG ERRSTAT 2LONG 2644 DIAG ERRSTAT BADEOF 2645 DIAG ERRSTAT ENCOUT 2646 DIAG ERRSTAT BADORD 2647 DIAG ERRSTAT DISCC3 264F DIAG INIT 265F DIAG PORT_DIED 266E DIAG DATA 266F DIAG TIMEOUT 2660 DIAG STATS FTX 2661 DIAG STATS FRX 2662 DIAG STATS C3FRX 2670 DIAG PORTABSENT 2671 DIAG XMIT TABLE C 2 Error Message Codes Defined Continued Error Number Test Name Error Name ...

Page 145: ...NG 3044 DIAG ERRSTAT BADEOF 3045 DIAG ERRSTATENCOUT 3046 DIAG ERRSTAT BADORD 3047 DIAG ERRSTAT DISC3 304F DIAG INIT 305F DIAG PORTDIED 3060 DIAG STATS FTX 3061 DIAG STATS FRX 3062 DIAG STATS C3FRX 306E DIAG DATA 306F DIAG TIMEOUT 3070 DIAG PORTABSENT 3071 DIAG XMIT 3078 DIAG PORTWRONG TABLE C 2 Error Message Codes Defined Continued Error Number Test Name Error Name ...

Page 146: ...tralMemoryTest cmiTest Port received an interrupt when not expecting one ASIC failure Replace mainboard assembly DIAG BUS_TIMEOUT Err 0BOF 4040F portRegTest sramRetentionTest ASIC register or ASIC SRAM did not respond to an ASIC data access ASIC failure Replace mainboard assembly DIAG CAMINIT Err 223B camTest Port failed to initialize due to one of the following reasons Switch not disabled Diagnos...

Page 147: ... Port got the wrong CMEM error type ASIC failure Replace mainboard assembly DIAG CMICKSUM Err 2036 cmiTest CMI message received failed bad checksum test ASIC or mainboard failure Replace mainboard assembly DIAG CMIDATA Err 2035 cmiTest CMI data received but did not match data transmitted ASIC or mainboard failure Replace mainboard assembly DIAG CMIINVCAP Err 2034 cmiTest Unintended ASIC erroneousl...

Page 148: ...t Bad symbol on fiber optic cable DiscC3 Discarded Class 3 frames ASIC mainboard SFP module or fiber cable failure Replace mainboard assembly SFP module or fiber cable DIAG INIT Err 264F 304F 384F portLoopbackTest crossPortTest spinSilk Port failed to go active in the loopback mode requested ASIC mainboard SFP module or fiber cable failure Replace mainboard assembly SFP module or fiber cable DIAG ...

Page 149: ...d assembly DIAG MEMORY Err 0110 ramTest Data read from RAM location did not match previously written data into same location CPU RAM failure Replace mainboard assembly or DRAM module DIAG PORTABSENT Err 2670 3070 3870 portLoopbackTest crossPortTest spinSilk Port is not present ASIC or mainboard failure Replace mainboard assembly DIAG PORTDIED Err 265F 305F 385F portLoopbackTest crossPortTest spinS...

Page 150: ... transmitted FramesRx number of frames received CI3FrmRx number of Class 3 frames received ASIC SFP module or fiber cable failure Replace mainboard assembly SFP module or fiber cable DIAG TIMEOUT Err 266F 306F 386F portLoopbackTest crossPortTest centralMemoryTest For portLoopbackTest and crossPortTest Port failed to receive frame within timeout period For centralMemoryTest Port failed to detect an...

Page 151: ... system s flash memory has encountered an error OS error The system attempts to recover from its mirrored backup Contact customer support RPC SVC_EXIT An RPC service daemon has terminated prematurely or unexpectedly OS error Contact customer support RPC SVC_REG An RPC service daemon could not establish service for a particular protocol handler OS error Contact customer support TEMP 1_FAILED LOG_WA...

Page 152: ...S Invalid IU OS error Contact customer support FCIU IUCOUNT L S Total number of IUs Count 0 OS error Contact customer support FCPH EXCHBAD L S Bad exchange OS error Contact customer support FCPH EXCHFREE L S Unable to free an exchange OS error Contact customer support MQ QWRITE L M Message queue overflow Task blocked Contact customer support MQ QREAD L M Message queue unread OS error Contact custo...

Page 153: ...PANIC ZOMTIMSET LOG_PANIC Attempt to set a zombie timer OS error Contact customer support PANIC ZOMTIMKILL LOG_PANIC Zombie timer destroyed OS error Contact customer support PANIC FREETIMRLSD LOG_PANIC Free timer released OS error Contact customer support PANIC TIMEUSECNT LOG_PANIC Timer use count exceeded OS error Contact customer support PANIC LSDB_CKSUM LOG_PANIC Link State Database checksum fa...

Page 154: ...DB MAXINCARN LOG_WARNING Local link state record reached max incarnation OS error Contact customer support FLOOD INVLSU LOG_WARNING Discard received LSU OS error Contact customer support FLOOD INVLSR LOG_WARNING Unknown LSR type OS error Contact customer support FLOOD LSRLEN LOG_ERROR Excessive LSU length OS error Contact customer support HLO INVHLO LOG_ERROR Invalid Hello received from port OS er...

Page 155: ...stomer support MCAST REMBRANCH LOG_ERROR Remove branch failed OS error Contact customer support MCAST NOPARENT LOG_ERROR Null parent OS error Contact customer support MCAST NOPARENTLSR LOG_ERROR Null IsrP OS error Contact customer support UCAST ADDPATH LOG_CRITICAL Add path failed OS error Contact customer support UCAST ADDPORT LOG_WARNING Add port failed OS error Contact customer support UCAST RE...

Page 156: ...138 Sun StorEdge SAN 4 0 Release Field Troubleshooting Guide October 2002 ...

Page 157: ...endix explains how the Sun FC switch encodes Fibre Channel addresses Note This information only applies to the Sun FC switches This appendix contains the following topics Converting a Fabric Address into Fabric ID Chassis ID ASIC Port and AL_PA on page 140 Example on page 141 ...

Page 158: ...online Mar 7 10 06 18 vikings scsi ID 799468 kern info ssd7 at fp0 name w50020f230 0009697 0 bus address 1084e4 Mar 7 10 06 18 vikings genunix ID 936769 kern info ssd7 is pci 8 700000 pci 3 SUNW qlc 4 fp 0 0 ssd w50020f2300009697 0 Mar 7 10 06 18 vikings scsi ID 365881 kern info SUN T300 0117 cyl 34145 alt 2 hd 56 sec 128 Mar 7 10 06 18 vikings genunix ID 408114 kern info pci 8 700000 pci 3 SUNW q...

Page 159: ...y 000100001000010011100100 which is the number used in this example 000100001000010011100100 is the 24 bit binary representation of 1084e4 Qlogic encodes this number in the following way The AL_PA will be zero if the device is a full fabric device otherwise it will be the AL_PA of the loop device StorEdge Network Fibre Channel Switches have 2 or 4 ASICS 2 on the 8port switch 4 on the 16port switch...

Page 160: ...of the switch is 2 The port in question is port 0 of ASIC 1 Port 0 of ASIC 1 is Port 5 if you were to look at the switch faceplate Refer to TABLE D 1 to see the ASIC Port breakdown The AL_PA of the device is E4 Knowing this information you can easily determine where this device is located in the SAN See TABLE D 1 0001 Fabric ID 000010 Chassis ID 0001 ASIC ID 00 Port ID 11100100 AL_PA Fabric ID 1 C...

Page 161: ...Appendix D Converting Sun FC Switches Fibre Channel Addresses 143 5 1 0 6 1 1 7 1 2 8 1 3 9 2 0 10 2 1 11 2 2 12 2 3 13 3 0 14 3 1 15 3 2 16 3 3 TABLE D 1 ASIC and Port Values Continued ...

Page 162: ...144 Sun StorEdge SAN 4 0 Release Field Troubleshooting Guide October 2002 ...

Page 163: ...Fibre Channel switch a port that supports Arbitrated Loop devices G_Port A generic port that can automatically configure as either an E_Port or a F_Port GL_Port A generic loop port can automatically configure as an E_Port F_Port or a FL_Port L_Port A loop port that enables private devices to communicate with fabric or public devices NAS Network Attached Storage SNDR Sun StorEdge Network Data Repli...

Page 164: ...any as 16 Fabric wide zones that define the ports that can communicate with each other A particular port may be placed in only one Hard Zone no overlapping Hard Zones If Hard Zones are enabled Name Server Zones and SL Zones will not communicate across defined Hard Zone boundaries Name Server Zones allow the division of the Fabric one or more Switch chassis into as many as 256 Fabric wide zones tha...

Page 165: ...rays 25 single host connected to one storage array 24 configuration examples 24 configuration guidelines 22 configurations 9 conventions typographic xii count cascade limit 3 hop limit 3 ISL limit 3 ISL link limit 3 long wave tranceiver limit 3 maximum switches 3 D diagnostic tool T3Extractor 40 disaster tolerant configuration 3 document purpose 1 scope 2 documentation accessing online xv F fabric...

Page 166: ...long wave SFP 3 tranceiver 3 loop port 4 M maximum switch count 3 mesh configuration 3 multipathing support 5 N name server port based zones 4 WWN based zones 4 zoning 4 nested zoning 4 network adapters 6 NS zoning 4 O operating environment required Solaris release 14 overlapping zones 4 P packages supported 16 partial fabric support 3 patches downloading using Sunsolve 16 supported 16 port based ...

Page 167: ...uration guidelines 22 connection of 10 management tools 5 switch counters role in troubleshooting 41 rules when viewing 41 switch port types 19 T third party compatability 6 switch capability 6 TL port 4 tools diagnostic 31 topologies supported 3 tranceivers 3 translative loop 4 troubleshooting steps to use to approach a SAN problem 45 U UNIX commands use of xi V VERITAS Cluster Server 5 W website...

Page 168: ...150 Sun StorEdge SAN 4 0 Release Field Troubleshooting Guide October 2002 ...

Reviews:

No comments