Troubleshooting BGP [IP Routing] - Cisco Systems

Troubleshooting BGP  [IP Routing] - Cisco Systems: Troubleshooting BGP

awesome

FUD for troubleshooting

I have witnessed many companies and politicians making very good use of "FUD" in public relations. (see FUD)
In my line of work I see a correlary in action far too often in the area of troublehooting or worse, in root cause investigation. Since I love a good TLA (three letter acronym) I will manufacture one to describe it. SIR = Superstition, Ignorance, and Rumor.

SUPERSTITION

This problem is vaguely familiar (I can't be sure because I never put that much thought into problem definition. So, really, almost all of them are vaguely familiar.) So, I should immediately launch into "what fixed it last time."

Although I make enthusiastic use my good fortune on days when my superhero underpants are working in my favor, I can't really respect this as the basis of troubleshooting. When your job is to be the final point of escalation for major enterprise issues in an organization you really need to THINK BEFORE YOU ACT. Even greater a pox upon you if you are a support engineer for a worldwide company to whom enterprise customers pay $100K per year for support. In this case you must obey me when I tell you to ESCALATE ME NOW. (Or I will be forced to declare my help call a critical incident and waste everybody's resources because you are stupid.)

IGNORANCE

I am the subject matter expert and this is my answer. By definition I am right.

There is no shame in not knowing something. Indeed we all must accept that no single person can ever know everything that might be relevant to figuring out my problems and so I expect ignorance. However, I cannot abide someone without the common sense and humilty to admit that they are not the all-knowing god of [insert product/technology here.] When a solution is presented based on a 20-something's omniscience and I don't think it will help I will often ask questions which get a patronizing answer. When I get tired of being called "dude" and tell him to ESCALATE ME NOW. It never works. I have to call his boss. I am embarrassed for these people when even though the investigation goes another direction and it's pretty obvious they were on the wrong track, they will continue to interject excuses or reasoning why they thought the way they did long after it is relevant.
Even when it's not taken that far, IGNORANCE--leading to a lack of motivation to GATHER ALL THE FACTS--feeds a frenzy of SUPERSTITION and RUMOR in a dysfunctional troubleshooting environment.

RUMOR

They are having a network problem.

Lack of problem definition and detailed investigation will often lead IT Management (and, ugh, even sometimes USER Management) to pigeon hole problems/systems/people on a snap judgement of the cause of a problem based faulty or lack of investigation to define problems. This failure is manifested in management decisions for project direction, purchasing, and professional development are not based on fact. The root of this phenomina information gathered by the IT first contact person with the user who has the problem (Service Desk.)
RUMOR also feeds back into SUPERSTITION
When this phylosophy is deeply entrenched in the culture of the organization users don't even report their problems they tell the helpdesk that "the DNS server is having that problem again" and so the service desk opens a ticket to the same group that "fixed it last time." Too bad that the job of the service desk person is going to have to be done by an engineer who has to call the user back (and possibly play phone tag first) to get further details about a problem only to find out that (surprise!) the user was wrong about the diagnosis of their problem and it has to be requeued to the correct resources. The time wasted by this activity aggravates the user and often makes the real problem harder to diagnose. So when the lucky top tier engineer has to call the user for additional information about the problem the user is either to angry to be helpful or has forgotten any useful details.

Windows Server Black Logon Screen

VERY strange problem
Hit ctrl-alt-del to logon console or rdp to server. Just get a black screen with difficult to see underscores where the logon fields should be. We were able to tab around on the screen and enter logon name and password and get logged on. (or input logon credentials into RDP before connecting.) Once logged on, everything was normal.

We had two servers, both of which happened to be print servers, both experienced this same issue at the same time. Windows 2003 R2 SP2 running as ESX3.5 virtual machine.


Cause
As strange as the problem is, it's root cause is strangely simple.
The display colors for the default user have all been changed to black.
After a tedious internet search we learned about someone elses experience with this exact problem: http://www.flyhpr.com/blog/2005/02/windows-2003-black-login-screen.htm.
They said they called Microsoft and this is a known problem but supposedly they didn't get enough calls about it to make it a KB article. (???that is also a very strange answer, by the way.) The article goes on to say Microsoft told them this issue was caused by the system drive running out of space.


Solution

Change the colors back to normal. copy the clipped lines below
and save a .REG file. Run it on the server with the problem.
-------------------------------------------------------
Windows Registry Editor Version 5.00

[HKEY_USERS\.DEFAULT\Control Panel\Colors]
"ActiveBorder"="212 208 200"
"ActiveTitle"="10 36 106"
"AppWorkSpace"="128 128 128"
"Background"="102 111 116"
"ButtonAlternateFace"="181 181 181"
"ButtonDkShadow"="64 64 64"
"ButtonFace"="212 208 200"
"ButtonHilight"="255 255 255"
"ButtonLight"="212 208 200"
"ButtonShadow"="128 128 128"
"ButtonText"="0 0 0"
"GradientActiveTitle"="166 202 240"
"GradientInactiveTitle"="192 192 192"
"GrayText"="128 128 128"
"Hilight"="10 36 106"
"HilightText"="255 255 255"
"HotTrackingColor"="0 0 128"
"InactiveBorder"="212 208 200"
"InactiveTitle"="128 128 128"
"InactiveTitleText"="212 208 200"
"InfoText"="0 0 0"
"InfoWindow"="255 255 225"
"Menu"="212 208 200"
"MenuText"="0 0 0"
"Scrollbar"="212 208 200"
"TitleText"="255 255 255"
"Window"="255 255 255"
"WindowFrame"="0 0 0"
"WindowText"="0 0 0"
-------------------------------------------------------

Good article: Storms RIP the Net

This is an informative recounting by Laura Chappel of the investigation and repair of network traffic issue crippling a network. Nothing could stay connected even long enough to do a "normal" packet capture.
She had them setup a quick packet capture outside the GUI to allow for getting on and getting the capture before being bumped off.

tshark -c 100 -w gen1.pcap

The -c parameter indicates the number of packets to capture. The -w parameter is
used to define the name of the trace file to create.

Looking at the 100 packets the fact that the IP Identification field matched for every packet indicated that this was a looping condition rather than some kind of denial of service from a single host.

A switch loop is easy to create and often hard to troubleshoot, unless you are looking for this exact condition. And often the opportunity to create a loop is made available to the masses with proliferation of workgroup switches to avoid spending a couple hundred bucks on having another jack installed. ("Gee, here's an end of a cable coming out of a big tangle under my desk. It must need plugged in...")

Separating broadcast domains into several VLAN's, like one per floor or some other logical separation, can limit the scope of a problem due to a switch loop. At least only one VLAN will be down and you have a narrower search area for the loop -- check the log on one or two switches instead of 20-30.

CTX681954 - Troubleshooting Citrix ICA Printer Auto-creation: "Troubleshooting Citrix ICA Printer Auto-creation "

Wfshell.exe crashes - Brian Madden's Community Forum: "Grimace Not-So-Newbie


Posts: 25
Score: 0
Joined: 3/17/2005
Status: offline This issue has been addressed in a Citrix Hotfix Rollo-up:
PSE400W2KR01

http://support.citrix.com/article/CTX107504

_____________________________

La Grimace

(in reply to Joe McGaugh)
Post #: 8


RE: Wfshell.exe crashes - 1/24/2006 7:59:23 AM


Joe McGaugh
Master Yoda





Posts: 676
Score: 13
Joined: 4/15/2005
From: Florida
Status: offline Yes, this roll-up addresses the issue, but I have not seen it Resolve the issue in all instances. If you don't do the necessary maintenance as Jeff and I both have talked about, your putting a bandaid on the issue. Clean the servers and remove all the native drivers first, set printing policies for UDP and do not allow native drivers to install automatically, and if your still having an issue, then try the patch.

_____________________________

Joe "

wfshell.exe error

From one kind soul:

Duplicate Problem by terminating application by clicking on "X". Log back in and you should see the wfshell error.

Resolution -

In the "CCC" under advanced setting, change the configuration from "reset" on a broken connection to disconnect. After this configuration is made, log off and log back on and terminate application by clicking on the "X" again. After you log back on, the wfshell error should not be there nor in the eventlog anymore. You can modify the idle timeout settings for users to control the number of active idle session on your Citrix Servers. Hope this helps and it is a relief to finally find the cause of this error.

CDM Redirector Timeout Event ID 70 in MetaFrame 1.8 :: Tokeshi.com :: Citrix Support :: Thin Client Support :: Terminal Server Help

In the Events Viewer, one might find a varied number of Event ID 70 in Source: CDM Redirector. This occurs when the CDM Redirector timed out on a request to access a client device.

This can happen when an application is attempting to access a client device during a disconnect. Check Citrix Server Administration when you see this event appear. The best way to determine what client device is causing this issue is to turn off client device mapping (client drive, client COM ports, client printers). Enable them one at a time and monitor the client's logon process.

If you find that you are receiving these timeouts due to network latency, you can modify the timeout values in the registry key shown below:

WARNING: Using Registry Editor incorrectly can cause serious problems that may require you to reinstall your operating system. Microsoft cannot guarantee that problems resulting from the incorrect use of Registry Editor can be solved. Use Registry Editor at your own risk.

For information about how to edit the registry, view the "Changing Keys and Values" Help topic in Registry Editor (Regedit.exe) or the "Add and Delete Information in the Registry" and "Edit Registry Data" Help topics in Regedt32.exe. Make sure you back up the registry before you edit it. If you are running Windows NT, be sure to update your Emergency Repair Disk (ERD).

The registry key is:

HKLM\SYSTEM\CurrentControlSet\Services\Cdm\Parameters

Increase the settings for the following:
RequestTimeout: Default is 60 (in seconds)
DirCacheTimeout Default is 30 seconds
CacheTimeout is 60 seconds

Event ID 9646 is logged when you try to send many messages in Exchange Server 2003

Can't get to Delta.com

I had this problem recently and a search of the internet found several interesting instances of this happening to others.
One example:http://www.experts-exchange.com/Security/Q_21649945.html

Title: Access denied due to security policy violation

When I link to some web pages, I am getting this error message:

Access denied due to security policy violation

Reject ID: 438f612b-1000c-42852404-7b6

Does this look familiar to anyone? I can't tell if it is coming from something on our end (like Checkpoint NG) or on the other end. It does not happen all the time on even the same website, just on certain pages for unknown reason.

I see several reports of this exact message on Google, but no one has an answer to the question.


Comment from HarkinsIT
Date: 01/26/2006 01:15PM PST
Comment


I have received a few complaints from my users about not being able to access particular sites, particularly delta.com. I too am running a Checkpoint firewall (NGXR60) and have found the following in the SmartDefense log: WSE0020008 found both content-length and transfer-encoding headers in response.

It's not the same exact error you are getting, RobWalling, but I was wondering if you had found a solution. I have been pouring through the SmartDefense settings and so far I haven’t been able to find anything about this.

Just wondering.....Thanks.


Comment from RobWalling
Date: 01/27/2006 11:40AM PST
Comment


After playing around with it for awhile and talking to Check Point, I finally discovered that the "ASCII Only Response Headers" criteria in the Web Intelligence piece of NGX was what was causing the problem. Turned it off, and all is well. Putting it in "Monitor Only" mode will still block it though! That really made it hard to figure out. Of course what opened myself up to, I'm not sure yet.

BTW, user ID "mobot" is someone else. I latched onto this thread because it described the same problem.

Thanks.



Comment from m298908
Date: 01/27/2006 11:42AM PST
Comment


We recently upgraded our check point software also and started having this for delta.com. It is only happening to one user though. I can get to it fine. Very strange. I will try this setting and see if it works. thx

Outlook::Auto Completion Cache

Utility to export nickname (NK2) file
Thanks to a comment on my 3/16/05 article at: Outlook Name Completion Issues
Beta utility:
http://www.epute.com/nk2csv/

VOIP::Cisco::Echo

Troubleshooting Echo in VOIP calls

http://tools.cisco.com/cmn/jsp/index.jsp?id=14832&redir=YES

http://www.cisco.com/en/US/tech/tk652/tk698/technologies_tech_note09186a0080149a1f.shtml

Cisco::VOIP::Delay establishing call

PROBLEM:
Sometimes it take 5-10 seconds for both parties of a call to hear each other.
This is because the RTP channel comes up after the source router receives a connect message from the destination router. The connect message is sent after the person being called picks up the phone.
FIX:
At the router, input the following:

ROUTER(config)#voice rtp send-recv

This will make all calls establish the voice path in both directions before a connect messagee is received from the dest switch.

Mail::Internet::Test

E-Mail Testing with Autoresponders

Here is a list of Email Autoresponders. If you send an email to one or more of the addresses listed below you should receive an autoreply in a few minutes.

test@tznet.com Auburndale, WI
test@digiplay.com Chicago, IL
test@toltbbs.com Toledo, OH
test@cshore.com Madison, CT
test@sstar.com Metairie, LA

Windows::Citrix::Event Log

Event 10016 caused by COM component permissions or failure of service account:

http://support.microsoft.com/default.aspx?scid=kb;en-us;899965&sd=ee

Print jobs that you send to a Windows Server 2003 print queue when the printer is out of paper take a long time after you add paper

WORKAROUND
To work around this problem, turn off SNMP port status for the print queues. To do this, follow these steps:1. Click Start, and then click Printers and Faxes.
2. Right-click the printer, and then click Properties.
3. Click the Ports tab.
4. Click Configure Port.
5. Click to clear the SNMP Status Enabled check box, click OK, and then click Close.