Calix BLM 1500 R13
Trouble Shooting Guide
November 2015
#220-00687, Rev 10
Page 1: ...Calix BLM 1500 R13 Trouble Shooting Guide November 2015 220 00687 Rev 10...
Page 2: ......
Page 3: ...rocessing Troubles 14 Locating Troubles 15 Recovery Requirements 16 Chapter 2 Connection to BLM 1500 17 TAP301 On site Craft Connection Failure 19 TAP302 Remote Craft Connection Failure 20 Chapter 3 F...
Page 4: ...lures 45 TAP501 Voice Service Failure 47 TAP502 Slow Data Service 50 TAP503 Intermittent Data Service 53 TAP504 Loss of Data Service 56 TAP505 Video Freezing 58 TAP506 Video Pixelating 65 TAP507 Loss...
Page 5: ...by written agreement with Calix Calix All Rights Reserved Hot CSR 90 Severity Emergency 91 Severity High 92 Severity Medium 93 Severity Low 94 Data Collection for CSR 95 Mandatory Enclosures for all...
Page 6: ...6 Proprietary Information Not for use or disclosure except by written agreement with Calix Calix All Rights Reserved...
Page 7: ...lexer 1500 BLM 1500 gigabit passive optical network GPON optical line termination OLT BLM 1500 is a member of the BLM 1500 product line which includes the BLM 1500 GPON OLT the EntriView element manag...
Page 8: ...ietary Information Not for use or disclosure except by written agreement with Calix Calix All Rights Reserved Revision History Revision Summary of Changes 10 Initial document release for BLM 1500 Rele...
Page 9: ...as the result of an alarm or event message System awaiting input GUI Objects GUI objects such as menus fields and buttons are bold in text On the File menu click Exit Key Combinations Key combination...
Page 10: ...ause severe personal injury if not avoided Warnings alert you to the possibility of personal injury CAUTION Caution indicates the presence of a hazard that can cause minor to moderate personal injury...
Page 11: ...single subscriber applications The GPON standards use the term ONT by default unless a distinction is necessary This documentation uses both terms identifying any units that use pluggable line cards a...
Page 12: ...set of troubleshooting guides for BLM 1500 network solution BLM 1500 Trouble Shooting Guide EntriView Trouble Shooting Guide The BLM 1500 Trouble Shooting Guide covers the faults that occur on the BL...
Page 13: ...closure except by written agreement with Calix Calix All Rights Reserved Chapter 1 Getting Started with Troubleshooting This chapter introduces the structure of BLM 1500 Troubleshooting Guide and prov...
Page 14: ...ement with Calix Calix All Rights Reserved Roadmap of Processing Troubles Figure Roadmap of Processing Troubles displays the structure of this guide as well as each troubleshooting chapter It visualiz...
Page 15: ...tten agreement with Calix Calix All Rights Reserved Locating Troubles Figure How to Locate Troubles provides navigation to troubleshoot a specific type of problem How to Locate Troubles Note Contact y...
Page 16: ...Support Gateway RSG access must be provided to the engineers involved in the case An RSG installation makes it possible to reach all nodes in a customer network and to provide support services like up...
Page 17: ...or disclosure except by written agreement with Calix Calix All Rights Reserved Chapter 2 Connection to BLM 1500 Before any troubleshooting started enable communication with BLM 1500 to assess the dama...
Page 18: ...ix Calix All Rights Reserved Figure Troubleshooting Craft Connection to BLM 1500 navigates you to a specific BLM 1500 craft connection failure Troubleshooting Craft Connection to BLM 1500 TAP301 On si...
Page 19: ...ailed login attempts typo or incorrect password Contact the appropriate department to ensure your login is still valid Did this correct the problem If yes STOP You have completed this procedure If no...
Page 20: ...tion software modem port and management network 3 Ensure that the MODEM is configured as follows 9600 Baud 8 Data bits No parity One stop bit Flow control off Did this give you access If yes go to Ste...
Page 21: ...yes go to Step 8 If no contact your next level Support 8 At this point a prompt displays as follows BLM1500 Broadband Loop Multiplexer Copyright 2001 2007 Entrisphere Inc Fabric Control Processor ttyp...
Page 22: ...22 Proprietary Information Not for use or disclosure except by written agreement with Calix Calix All Rights Reserved...
Page 23: ...ooting for faults alerted by BLM 1500 indicators Alarms can be observed from indicators on Fabric Packs see Faults on FAB3 Packs on page 24 FAB3 in slot X FAB3 in slot Y Application Packs see Faults o...
Page 24: ...All Rights Reserved Faults on FAB3 Packs Figure Troubleshooting Faults on FAB3 navigates you to a specific trouble occurred on FAB3 packs Troubleshooting Faults on FAB3 TAP201 FAULT Led Lit on page 2...
Page 25: ...Step 5 5 Log in to BLM 1500 using CLI 6 From the enable prompt enter show release Does the FAB3 with the FAULT LED lit show a software release If yes go to Step 7 If no contact your next level Support...
Page 26: ...sh booting contact your next level of Support If no go to Step 3 3 Verify the state of FAB3 packs through the following From CLI console Through the enable prompt enter show pack state From EntriView...
Page 27: ...or MAJ LED on FAB3 lights These LEDs indicate a service affecting condition in the BLM 1500 system that requires immediate corrective action 1 Is the Critical CRIT or Major MAJ LED lit on the FAB3 pac...
Page 28: ...Troubleshooting Faults on Application Packs navigates you to troubleshoot a specific trouble occurred on application packs Troubleshooting Faults on Application Packs TAP208 ALL LEDs Lit on page 29 TA...
Page 29: ...e issue is a software mismatch between Application and FAB packs Enter show pack inventory If yes then go to TAP263 Pack or Port in Mismatch State MEA on page 40 If no go to Step 4 Is the assigned pac...
Page 30: ...o Step 4 4 See a list of BLM 1500 events Log in to the BLM 1500 using CLI From the enable prompt enter msg query send all From Map View double click a BLM icon and select Fault Current Events 5 Is the...
Page 31: ...ep 7 If no contact your next level Support 6 Enter show pack inventory fab x Note the software version indicated 7 Enter show pack inventory pack_aid Does the software on the application pack in quest...
Page 32: ...NT in active state on the ODN If no go to Step 4 A single defect may be affecting the entire ODN If yes go to Step 7 Parts of the ODN are operational A defect or configuration error is causing alarms...
Page 33: ...ommands show ont olt x y state and show ont olt x y pon where x is a pack number 0 to 15 and y is the port number 1 through 4 Is there an active power outage or for each ONT in a FLT state could the O...
Page 34: ...the FLT state and pull out and reseat the fiber connector Working with a network administrator use the commands from step 9 Did the Amber Alarm LED go out If yes STOP You have completed this procedure...
Page 35: ...D light If yes STOP You have completed this procedure If no go to Step 3 3 Log in to the BLM 1500 using CLI From the enable prompt enter show port state pack_aid for the G28 or E28 pack Is the G28 or...
Page 36: ...have completed this procedure If no go to Step 3 3 To check current alarms in CLI enter msg query send all Is there an LOS Loss of Signal alarm on the port If yes go to Step 4 If no go to Step 5 4 Is...
Page 37: ...designed for troubleshooting when suspecting a pack failure 1 To view the state of application packs in the BLM 1500 enter the CLI command show pack state pack_aid What is the state of the suspect pac...
Page 38: ...lking address test PASS G28 24 CPORT 0 XP test PASS G28 25 CPORT 1 XP test PASS G28 26 CPORT 0 BMU DRAM walking 1s data test PASS G28 27 CPORT 1 BMU DRAM walking 1s data test PASS G28 28 CPORT 0 BMU D...
Page 39: ...6 OOS AU UAS GSX 0 7 OOS AU UAS GSX 0 8 OOS AU UAS GSX 0 B1 IS NR ACT GSX 0 X1 OOS AU UAS GSX 0 X2 OOS AU UAS BLM1500 x BLM1500 x show port state ap 12 PORT PST Secondary States ALIAS DNALIAS NAME OLT...
Page 40: ...port TAP263 Pack or Port in Mismatch State MEA TAP263 is designed for troubleshooting when a pack or port is in MEA 1 Is the MEA on a pack or a port If the MEA is on a pack go to Step 2 If the MEA is...
Page 41: ...d this procedure If no go to TAP265 Pack Reset or Replacement on page 42 TAP264 Pack or Port in Maintenance State MT A pack or port in maintenance MT mode is in a service affecting state OOS A pack or...
Page 42: ...s to the pack slot assignment and y refers to the port number Did this correct the problem If yes STOP You have completed this procedure If no go to Step 4 4 If it is a GSX optical port 1Gb or 10Gb re...
Page 43: ...p Calix engineers complete a Root Cause Analysis RCA to ensure appropriate long term corrective action 1 Before resetting or replacing any pack capture the outputs of the following commands msg query...
Page 44: ...esetting the chip set Have the on site technician loosen the latch screw and raise the lower latch on the application pack for a couple of seconds then return it to its lower position Allow 1 minute f...
Page 45: ...mises However they can be located and identified at the ISP site for instance at the central office CO through On site craft connection to the BLM 1500 subtending the faulty ONTs Remote craft connecti...
Page 46: ...ependencies for providing these services verify any failed triple play services in order of dependencies voice data then video Troubleshooting Service Failures TAP501 Voice Service Failure on page 47...
Page 47: ...ctivating voice services on a newly provisioned ONU reboot the ONU Enter init sys ont x y z Caution This procedure can result in service interruptions or loss of data Note Wait 5 minutes after rebooti...
Page 48: ...1x is not configured Verify that the ONT ONU has the correct 802 1x provisioning Enter show ont ont x y z dot1x 12 Verify that the ONT ONU has the 802 1x feature enabled Enter show ont ont x y z mib...
Page 49: ...commands on the G6 CLI Enter run ip ping cpe ip address Confirm ONT ONU is reachable from the voice gateway Enter show cpe cpe name Confirm that the status shows that the correct CPE is active and op...
Page 50: ...cedure shows the CLI commands to enter but not sample results Refer to the BLM 1500 GPON CLI Reference Guide to view sample results for these commands To resolve slow Internet service problems 1 Open...
Page 51: ...y z vlan Have the NOC confirm the bandwidth allocation placed on this VLAN 8 Verify that the traffic container TCONT grant on the ONT is properly defined Enter show ont ont x y z grant Confirm the up...
Page 52: ...ct a PC to the residential gateway Ethernet port and verify the PC receives an IP address from residential gateway Send ping packets from the PC to a network side host Capture this traffic on G28 or E...
Page 53: ...L line Enter allow pm ont x y z For ONU Enter show ont ont x y z pm ont xdsl 3 1 ports are 3 1 through 5 4 For ONT Enter show ont ont x y z pm ont eth Confirm there are no errors on the ports Perform...
Page 54: ...d and that the queue profile is properly defined Enter show port conf gsx x y Enter show port conf gsx x b1 Enter show port conf olt x y Enter show pack conf ap x GP4 or GP8 pack Enter show profile qu...
Page 55: ...Note Broadcasting traffic observed at the mirrored source port is doubled at the mirroring target port 14 If this does not solve the problem contact Calix Support 15 Remove the mirror port on the G28...
Page 56: ...The following procedure instructs troubleshooting for loss of data service 1 Open a command console and enable logging of session activity to a file 2 Verify that the VLAN numbering and GEM cross conn...
Page 57: ...rk protocol analyzer running to the mirror port Enter config Enter interface ap x Enter mirror source both gsx x y Enter mirror target both gsx x y 12 Reboot the residential gateway to trigger an exch...
Page 58: ...ns for x minutes or seconds and stops Video only runs for x amount of time after channel change is performed Video starts and stops continuously with no user intervention Video runs normally but pictu...
Page 59: ...d traffic statistics associated with the multicast group Enter show probe ipmc vlan multicast IP address ap x Note The ap x can only be the AID of a GP8 pack Record the output and report it for furthe...
Page 60: ...signed to this customer on the network side has the correct bandwidth rate settings to support video Enter show crs gem Record the VLAN ID and flow ID associated with the ONT Enter show ont ont x y z...
Page 61: ...the connector has no dirt or scratches 4 Check the fiber connection for this same optical cable at the PON splitter ensuring the connector has no dirt or scratches 5 If that does not resolve the issue...
Page 62: ...tor has no dirt or scratches 4 Check the fiber connection for this same optical cable at the PON splitter ensuring the connector has no dirt or scratches 5 If that does not resolve the issue test the...
Page 63: ...e connector has no dirt or scratches 4 Check the fiber connection for this same optical cable at the PON splitter ensuring the connector has no dirt or scratches 5 If that does not resolve the issue t...
Page 64: ...e connector has no dirt or scratches 4 Check the fiber connection for this same optical cable at the PON splitter ensuring the connector has no dirt or scratches 5 If that does not resolve the issue t...
Page 65: ...e Enter show ont ont x y z software 3 Check the state of the BLM 1500 and ONT packs and ports Enter show pack state Enter show port state ap x Enter show ont ont x y z pack state Enter show ont ont x...
Page 66: ...replace the OLT pack 7 If the issue is occurring on multiple ONUs that span multiple OLT ports on the same OLT pack continue Otherwise go to Step 8 Verify the appropriate queue profile is applied to t...
Page 67: ...ical info This shows all the ONTs optical receive power for the OLT Verify that the Rx signal level at the ONT is within the specified operating range 10 to 27 dBm 3 Check the fiber connection to the...
Page 68: ...al receive power for the OLT Verify that the Rx signal level at the ONT is within the specified operating range 10 to 27 dBm 3 Check the fiber connection to the ONT ensuring there is no dirt or scratc...
Page 69: ...to 27 dBm 3 Check the fiber connection to the ONU ensuring there is no dirt or scratches on the connector 4 Check the fiber connection for this same optical cable at the PON splitter ensuring the conn...
Page 70: ...at the PON splitter ensuring the connector has no dirt or scratches 5 Ensure that all T550G cards are seated 6 If the issue is not resolved check the inside wiring and CPE 7 Connect test equipment to...
Page 71: ...w sample screens for these commands To resolve loss of video service 1 Video on Demand uses the same path as unicast Internet data Therefore confirming the steps outlined inTAP504 Loss of Data Service...
Page 72: ...Verify that the expected video channel is correctly defined Enter show ipmc multicast IP address 12 Verify that the subscriber STB is actually sending IGMP requests Enter show sdv subscriber ipmc ont...
Page 73: ...p wire Network interface device wiring Premises wiring Line connected to correct port on DSL modem 2 Verify DSL pack operation by viewing the LED status Normal status is Power LED on Fault LED off Ind...
Page 74: ...e following command while the port is temporarily up if possible Enter show ont ont x y z dsl data xdsl x y Use this information to verify that the downstream and upstream transmit power levels and Si...
Page 75: ...y tail drops Enter debug pack pack_id pss qcnt all Dump five packets on both ingress and egress Enter debug pack pack_id pss pktdmp 5 5 2 Identify whether the packets are dropped at C port or C 192 En...
Page 76: ...eve pack console log Enter show pack log pack_id Upstream Traffic 1 Check out if any errors on FAB3 processor Enter debug pack pack_id fperr cportnumber 2 Identify whether the packets are dropped at C...
Page 77: ...lation on page 89 for CSR procedure 6 Check out the last lookup miss at C port and confirm whether the packets are those dropped at C port Enter debug pack pack_id lut hdr 7 Check whether prestera chi...
Page 78: ...78 Proprietary Information Not for use or disclosure except by written agreement with Calix Calix All Rights Reserved...
Page 79: ...1500 This chapter covers the general knowledge and concepts on BLM 1500 and BLM 1500 network solution Overview of BLM 1500 Network on page 80 TNG301 Craft Connection to BLM 1500 on page 81 TNG401 Und...
Page 80: ...of BLM 1500 network and function of each node can assist identifying and locating troubles Figure Overview of BLM 1500 Network shows a sample BLM 1500 network solution Overview of EDA 1500 Network Tro...
Page 81: ...ion to BLM 1500 through the serial console RS 232 interface on the primary fabric pack in slot X can be established by Connecting to an on site terminal Connecting to remote terminal Figure Craft Conn...
Page 82: ...cks Getting familiar with indicators on each type of packs can be helpful in troubleshooting failures alerted by indicators BLM 1500 packs comprise Fabric Packs FAB3 in slot X FAB3 in slot Y Applicati...
Page 83: ...LEDs of Older Version GP8 and E28 Application Pack LEDs of Newer Version GP8 and E28 Application Pack The new version of GP8 and E28 application pack is introduced since R11A CP7 Both older and newer...
Page 84: ...nd pack is beginning Power On Self Test POST Off Off Flashing Green Pack is booting Steady Red Off Off The slide latch is open or pack did not boot successfully failed POST or has since experienced a...
Page 85: ...No major conditions exist on the BLM 1500 MIN Amber A condition may exist requiring corrective action Either non service affecting or affecting up to two lines Off No minor conditions exist on the BLM...
Page 86: ...nd locating can be completed by 1 Retrieve Alarm logs from MSG QUERY SEND ALL and MSG QUERY SEND LOG on CLI console RTRV ALM ALL on TL1 console Alarm View on EntriView EMS Indicators on BLM 1500 packs...
Page 87: ...w pack inventory RTRV EQPT INV View inventory of application packs including part number serial number running software version and more System Information show system names RTRV NE View system name S...
Page 88: ...switch Change boot partition allow swtowkg fab fab x y ALW SWTOWKG EQPT Restore FAB protection after a failed upgrade allow swtoprotn fab fab x y ALW SWTOPROTN EQPT Restore FAB protection after a fai...
Page 89: ...orted to the next level support in the form of Customer Service Request CSR when any troubleshooting lasts over 60 minutes A Root Cause Analysis RCA study of the fault will be required by the Working...
Page 90: ...Problem is affecting subscribers network management or other important functions but is not an emergency Network affecting 2 or more nodes in the customer network are affected Issue has high customer...
Page 91: ...y until the system is in service or the emergency situation has a preliminary solution acceptable to the customer Examples of emergency faults Complete system failure The BLM 1500 system does not hand...
Page 92: ...ce affecting a specific area of functionality For example billing for subscriber services or an EntriView malfunction BLM 1500 network and system maintenance routines are disturbed or inaccessible Non...
Page 93: ...ity of network and system maintenance routines Application pack failure Fabric E28 G28 GP4 or GP8 Traffic disturbance for up to X number of end users per the Service Level Agreement SLA Non vital alar...
Page 94: ...towards a specific destination is reduced with maximum 5 Unnecessary redundant alarms Minor PM counter is lost or faulty Erroneous trace Node restart at non supported handling of node Fault making the...
Page 95: ...of the specific issue observed Description of the hardware and software configuration at the time the issue was observed and any configuration changes made since Description of any events occurring wh...
Page 96: ...the service provider Corporate information Contact person Product information family release level hardware CLEI code software version Management system OS Windows Solaris EntriView software version...
Page 97: ...greement with Calix Calix All Rights Reserved Chapter 7 Glossary ANM Access to Network Module CSR Customer Service Request FLS First Line Local Support MEA Mismatch Equipment Attributes MT Maintenance...
Page 98: ...hts Reserved OID Object Identifier PBS Peak Burst Size PIR Peak Information Rate POST Power On Self Test PTM Packet Transfer Mode QoS Quality of Service RCA Root Cause Analysis RSG Remote Support Gate...
Page 99: ...y written agreement with Calix Calix All Rights Reserved SLS Second Line Local Support SNR Signal to Noise Ratio TPM Telephony Port Modules UAS Unassigned state VoD Video on Demand WLA Working Level A...
Page 100: ...100 Proprietary Information Not for use or disclosure except by written agreement with Calix Calix All Rights Reserved...
Page 101: ...nce List 1 BLM 1500 Alarms Reference Guide 2 BLM 1500 CLI Reference Guide Multimode Commands 3 BLM 1500 CLI Reference Guide Executive Mode Commands 4 BLM 1500 TL1 Reference Guide System Administration...
