Might be a Bone Head Simple Question But...

Oh man! Ad-blocking software has been detected! :'(

This website is run by the community, for the community... and it needs advertisements in order to keep running. Blocking our ads means your killing our stats!
Please disable your ad-block, or become a premium member to hide all advertisements and this notice.

Here is the scenario...I have a server that loses connectivity for a couple seconds about 20 times a night between the hours of 2000 to 2100. The switch log shows where the link comes back up. Never shows it goes down though. I feel it is a utilization issue.

How does a switch know that a device is connected? Does it send keep alives by broadcasting down that port? Its gotta be a broadcasted keep alive so that the switch knows there is something alive down that port. Gotta find an article on this...grrrrr

 

Oh man! Ad-blocking software has been detected! :'(

This website is run by the community, for the community... and it needs advertisements in order to keep running. Blocking our ads means your killing our stats!
Please disable your ad-block, or become a premium member to hide all advertisements and this notice.

What does it actualy show when the link comes back up? This might give you a clue as to which machine is at fault here.

What about the server logs?

Harry.

 

Oh I got the server. Nothing in the server logs actually. Well there was a dns issue, but that was corrected and the issue still is occuring. Only impacting 1 machine.

Regardless the dns issue was minor and didn't impact connectivity. The server logs don't show any connectivity issue that I can see.

May 29 21:01:57: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 29 21:02:39: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 29 21:03:05: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 29 21:03:33: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 29 21:04:21: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 29 21:04:41: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 29 21:05:05: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 30 20:35:57: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 30 20:36:27: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 30 20:36:55: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 30 20:38:11: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 30 20:44:23: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 30 20:45:09: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 30 20:46:01: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 30 20:46:39: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 30 20:47:59: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 30 20:48:41: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 30 20:49:05: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 30 20:49:47: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 30 20:50:07: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 30 20:54:47: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 30 20:55:15: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 30 20:55:41: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 30 20:56:25: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 30 20:57:45: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 30 20:58:31: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 30 20:58:55: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 30 20:59:15: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 30 21:00:07: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
May 30 21:01:09: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up

I found a suggestion from Cisco about making sure there is no power management on the PC/Server or NIC. Also you can set up a setting against jitter, but this is the only server in our network so far that I have ever found this issue with as of yet.

Current configuration : 192 bytes
!
interface FastEthernet0/11
description ENTRIISBMC01 (NADCWPWEBBMC02)
switchport access vlan 293
switchport mode access
no ip address
duplex full
speed 100
spanning-tree portfast
end

I already replace the cable and moved the port from f0/6 to f0/11. I should have moved it to a different switch fabric but I didn't think of it at the time and need to research more into that anyways...

Thanks for the input!!!

 

First I'll say that I am *definitely* no Cisco expert!

A possibility that comes to mind is that the NIC on the server is marginal with the switch, and at the times given something is running somewhere that just takes it further out.

One thing to try which I'm always being told about by Cisco people is to not let the switch and server NIC be 'auto' but nail-up the correct info such as speed and duplex.

Harry.

 

Auto-neg is definately not the best idea.

duplex full
speed 100

Takes care of that issue though.

 

Indeed - I saw that on the config you posted. But what about doing the same on the NIC on the server?

Harry.

 

Good point. I don't have access to check the nic config on theserver. I can see the logs, but not the nic config. I bet I could do it from the command line though. ipconfig /all should tell me if it is auto or full...will have to check.

 

Then again, why does it only loose connection between 2000 and 2100 hour? That is if I read correct.
The config for the NIC surely does not change for that specific hour.

 

I was assuming some system load caused a flaky NIC to dip out.

Perhaps the best thing to do is fit a known good card (like the Intel Pro series) and use that instead.

Harry.

 

Since it happens every day at the same time, it has to be an issue with something running on the server causing over-utilisation.

Its not something daft like a backup running across the network is it? Can you run a trace on the server and see whats happening at those times?

 

I do believe it is a Network Backup on our TSM system. The system runs a backup. However, I still wonder why the link drops. My conjecture is that the ethernet circuit uses some kind of broadcast keep alives to keep the circuit up. But I never read anything before on it. I know between Cisco Devices it sends a multicast keep alive (CDP). I would have to review the specifics so I might be a little off...please no flaming...I am on holiday tonight.

 

I don't know whether the problem has anything to do with multicast or not. But one way to check is to examine output from the following:

show ip mroute summary
show ip mroute
show ip mroute active
show ip mroute count

 

NetEyeBall,

Questions:

1. How many physical ethernet cables are there between the switch and the server?
2. What kind of server is connected to the switch?

 

Should be a single span which like I said, we replaced and switched ports. Not that a physical issue still might be there, but it is odd that it is only happening between 2000 to 2100 hours approximately.

It is a Windows 2k server of some variant.

I opened a TAC case to get more info as to why we aren’t seeing a down alert, just an up alert in the log. Also server people are putting some perf logs in place.

Will keep you updated. Thanks for the tips and suggestions thus far!

 

Cisco TAC response thus far about the Up statement showing in the log and not the down statement...

"Regarding the error message on the 3550 that says the link came up (and just up), it actually a bug in the older code. The link goes down for such a short period of time, that it only logs the 'link up' message. The fix / workaround for this bug is to use a different method of detecting link down when autonegotiation is enabled. The fix is to upgrade to 12.1(14)EA1 or above."

So that answers one question. TAC is still researching the rest and my server guys are looking at the server.

My gut is heavily leaning to a server side issue.

 

NetEyeBall,

Since I'm currently studying for the 640-861 DESGN exam and Cisco lists a prerequisite for the CCDA certificate of "CCNA level knowledge and BCMSN level knowledge is needed to prepare for the CCDA certification exam," I decided to get myself the Cisco Press Building Cisco Multilayer Switched Networks (BCMSN) (Authorized Self-Study Guide), 4th Edition book.

I read that portfast is associated with STP, in particular PVST+. Under default conditions, STP assumes that all switch ports are to be part of the STP process unless configured otherwise. An example of "configured otherwise" is portfast. Basically, if you configure a switch port for portfast, instead of the four STP states of blocking, listening, learning, forwarding there are only two, blocking and forwarding. The purpose of portfast is that the network engineer/administrator knows that a particular switch port will terminate in a non-STP networking device such as a server. Since the portfast switch port leads to a network endpoint, the chances of switching loops is 0. Therefore, the portfast switch port is ignored from the STP process.

As you should know as a CCNA certified individual, that STP runs the whole election for root bridge/switch, election for designated bridge/switches, then switch ports eventually settle down to either forwarding or blocking mode. So based on the "flip flop" state of the portfast switch port from down, to up, to down, to up, it seems to suggest that STP is being run that causes the porfast switch port to go from up to down, or in STP terminology forwarding to blocking. I'm guessing STP has converged permitting the portfast switch port to go from blocking directly to forwarding since it was configured for portfast.

So, in order to cover yourself from a networking standpoint, verify that all switch ports in the same VLAN as the server are configured to be "portfast" also. That way, any unknown/authorized network change, such as plugging servers into different switch ports, will clearly be the fault of the server guys and not the network guys.

Another thing to check is the STP and/or VTP structure. Maybe the STP flip-flop state may be due to an absence of a designed VTP Server, VTP Transparent, and VTP Client structure as well as configuration of the switch priority values to influence the STP election process so that there is a STP structure of one root bridge/switch, planned designated bridge/switches, and the rest of the switches are physically in an extended star configuration relative to the planned designated bridge/switches.

Also, based on the TAC's response of "...The fix / workaround for this bug is to use a different method of detecting link down when autonegotiation is enabled..." suggests that you should specifically configure both the duplex and speed settings for the switch port leading to a server instead of relying on autonegotiation. It's kinda like the hazard of configuring trunk ports on opposite sides of a trunk to "auto" because side 1's configuration depends on side 2 and side 2 is waiting for side 1 to make up their mind. So by explicitly configuring the switch port speed and duplex, if the server's interface is configured for autonegotiation, then the server's interface should autonegotiate quickly to the switch's configured duplex and speed settings.

I hope this helps.

Source:

1. CCDA-Career Certifications & Paths - Cisco Systems - http://www.cisco.com/web/learning/le3/le2/le0/le4/learning_certification_type_home.html
2. Building Cisco Multilayer Switched Networks (BCMSN) (Authorized Self-Study Guide), 4th Edition - http://www.ciscopress.com/bookstore/product.asp?isbn=1587052733&rl=1

 

Thanks for the suggestions!

The port is not set for auto-negotiation. Not sure why TAC thought that. The port is set for 100 full.

I will check out the other point though.

The server guys are swapping NICs. It is possible that the NIC is flaking when the traffic is reaching near maximum.

 

I was thinking on what you suggested. Wouldn't a spanning tree recalc be like 20 seconds at least to move from blocking to forwarding? I will have to look at it a bit more in depth.

 

NetEyeBall,

The purpose of portfast is to skip the listening and learning steps of STP. Portfast is designed to go to an end point, like a server or workstation. Since the server or workstation isn't going to send STP BPDUs to the switch port it is connected to, the switch is not likely to "hear" anything to "listen" to, therefore it would have no new data to "learn" from. That's why portfast is designed to go from blocking, skip listening, skip learning, then to forwarding. The obvious benefit here is that with less steps, the STP convergence process is quicker than the time required for a complete STP cycle of blocking, listening, learning, and forwarding.

 

OK. We swapped the NIC. Still occuring at the same time. I have been feeling it has been due to network activity. So I started looking at the keep alive command. Tonight I upped the default keep alive to 30 seconds (from 10). The port sends keep alives to detect a down circuit. See the following command explination:

http://www.cisco.com/univercd/cc/td...os122/122cgcr/finter_r/irfinter.htm#wp1017769

We shall see if this helps or not.