Might be a Bone Head Simple Question But...

Oh man! Ad-blocking software has been detected! :'(

This website is run by the community, for the community... and it needs advertisements in order to keep running. Blocking our ads means your killing our stats!
Please disable your ad-block, or become a premium member to hide all advertisements and this notice.

ok...changing the keep alive from 10 to 20 to 30 to 50 to 100 and then to no keepalive had no impact. I was able to recreate the issue when I ran a manual incremental backup.

 

Oh man! Ad-blocking software has been detected! :'(

This website is run by the community, for the community... and it needs advertisements in order to keep running. Blocking our ads means your killing our stats!
Please disable your ad-block, or become a premium member to hide all advertisements and this notice.

NetEyeBall,

Does the backup routine temporarily shuts down the NIC so that "no new updates" occur during the backup process?

 

No. It shouldn't since it has to back up the server to a SAN disk across the network.

 

Do you have the sh int output for the interface?

Spice_Weasel

 

ERP-TapeRoom-1#sh int f0/11
FastEthernet0/11 is up, line protocol is up
Hardware is Fast Ethernet, address is 000c.316a.9e8b (bia 000c.316a.9e8b)
Description: ENTRIISBMC01 (NADCWPWEBBMC02)
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s
input flow-control is off, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters 1d00h
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue :0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
20806785 packets input, 3607384365 bytes, 0 no buffer
Received 273 broadcasts, 0 runts, 2 giants, 0 throttles
2 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
3711805 packets output, 261950493 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

ERP-TapeRoom-1#sh run int f0/11
Building configuration...

Current configuration : 192 bytes
!
interface FastEthernet0/11
description ENTRIISBMC01 (NADCWPWEBBMC02)
switchport access vlan 293
switchport mode access
no ip address
duplex full
speed 100
spanning-tree portfast
end

ERP-TapeRoom-1# sh int f0/11 switchport
Name: Fa0/11
Switchport: Enabled
Administrative Mode: static access
Operational Mode: static access
Administrative Trunking Encapsulation: negotiate
Operational Trunking Encapsulation: native
Negotiation of Trunking: Off
Access Mode VLAN: 293 (ERP-VLAN)
Trunking Native Mode VLAN: 1 (default)
Administrative private-vlan host-association: none
Administrative private-vlan mapping: none
Operational private-vlan: none
Trunking VLANs Enabled: ALL
Pruning VLANs Enabled: 2-1001

Protected: false
Unknown unicast blocked: disabled
Unknown multicast blocked: disabled


Voice VLAN: none (Inactive)
Appliance trust: none

ERP-TapeRoom-1#sh int f0/11 trunk

Port Mode Encapsulation Status Native vlan
Fa0/11 off negotiate not-trunking 1

Port Vlans allowed on trunk
Fa0/11 293

Port Vlans allowed and active in management domain
Fa0/11 293

Port Vlans in spanning tree forwarding state and not pruned
Fa0/11 293

Jun 10 19:31:57: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 19:34:45: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 19:35:09: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 19:35:31: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 19:37:19: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 19:37:39: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 19:38:01: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 19:38:23: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 19:38:49: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 19:39:15: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 19:39:35: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 19:40:28: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 19:53:27: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 19:54:33: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 19:55:01: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 19:55:27: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 19:57:21: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 20:00:59: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 20:01:27: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 20:01:55: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 20:06:41: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 20:13:29: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 20:17:01: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 20:18:47: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 20:20:59: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 20:21:57: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 20:22:19: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 20:22:41: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up
Jun 10 20:23:59: %LINK-3-UPDOWN: Interface FastEthernet0/11, changed state to up

Here you go guys! Thanks for all your input!!!

 

Questions:

1. When you mention "a SAN disk across the network" is it one of those hard disks in a box with a power cord, lights, switches/buttons, and an ethernet port?
   1. If so, what make and model is it?
2. What does "#show spanning-tree vlan 293" say?

 

Interesting...

A few thoughts:
The total transmitted by the server over the last 24 hours is 3607384365 bytes. Even if all that was transmitted during the period where interface flaps are occurring (approx. 50 minutes) that is a tiny load. At 100 full duplex you can move 3.6 gig in under 8 minutes easily. Transmitting 3.6 gig over 50 minutes is a light load, there should be no problems with interface and switch capacity.

sh int reveals 0 interface resets so no problems with keepalives. 0 lost carrier indicates no problems with physical connectivity. Zero drops in the input and output queue show no problems with excessive load. The only slightly odd part of the output is the 2 giants, often caused by a malfunctioning or jabbering NIC, or possible interference on the cable.

The config is fine and you have changed ports and cables. BTW, do you use BPDUguard or BPDUfilter on portfast ports?

It would be useful to try autonegotiate. Autonegotiate is often preferred now, especially as a number of servers (e.g. Sun boxes) disable some NIC functions when forced to 100 full.

Spice_Weasel

 

Updated NIC Drivers on server. I think that might have gotten it!!!!

 

So the networking issue was a symptom of a system issue. So, in current hind sight, do you think this was a "bone head simple question?"

Keep us posted if the updated NIC drivers was the solution to this issue.

 

The server hasn't lost connectivity since the night of June 10th. The morning of the 11th they updated the NIC drivers. So haven't had a problem since.

I have verified that the TSM backups have run since then so it would seem the update to the drivers resolved the issue with the NIC.

Yes, the loss of connectivity was a symptom of a system issue rather then a network issue.

Thanks to everyone for their insight!

As a side note TAC still hasn't gotten back to me how a switch at layer 1 or 2 can tell if a circuit is alive and the port should be up or down. Perhaps it is a layer 1 voltage change. I am not sure.

Also it always surprises me even in very large IT companies, a cable tester is a rare thing. I would think it would be 100% nessasary, but then I don't work in the data center hardware team. Perhaps they think it is easier just to run a new cable?

 

Off the top of my head, a good Fluke Category 5/5e/6 certification cable tester costs around $5000ish or so. The cost of uncertified RJ-45 connectors, uncertified UTP cabling, and an RJ-45 connector crimper is mathematically a relatively small fraction of said $5000ish or so the cable tester would cost. Even if you repull new cable, that still doesn't mean that it's certified to the UTP category level and therefore performance is not guaranteed at all. But, from a "productivity" perspective, the cable pullers are "doing something" to [try] to fix the problem. Also from a productivity perspective, as long as the parts and labor to pull and repull cable comes in less than $5000ish, it'll keep the beancounters happy. Welcome to "Intro to Business Math."