Issue tonight...Server lost network connectivity

Oh man! Ad-blocking software has been detected! :'(

This website is run by the community, for the community... and it needs advertisements in order to keep running. Blocking our ads means your killing our stats!
Please disable your ad-block, or become a premium member to hide all advertisements and this notice.

So we had a server lose network connectivity. The RDC contacted the Unix group to troubleshoot as they should have. The Unix group contacted me to look at the network portion.

I log in and tracert the ip to the last layer 3 hop. Login. Show Arp and Mac to get the switch port. Look at the config of the switchport. The description is wrong.

I have Unix verify the mac address on the server side. It is correct. The port is up/up but the server wont return a ping. I see communication on the port. What could be wrong?

I have the Unix guy give me the gateway on the server. Bingo...turns out the port was configured for the wrong vlan. The gateway is to Vlan 518 versus 144. So I make the change and clean up the description. Server is communicating now and the application owners are happy.

I would say as a take away:

Know how to quickly track down a switch port from a IP address using tools such as Traceroute, nbtstat (for windows servers), ping, and sh arp, and sh mac with the | include statements.

Keep your descriptions accurate in your switches.

If possible document your network very well including which servers go to which switch and port. Good for the next guy coming into your job if you leave.

Oh...my conjecture is that the RDC was doing some cable moves and they plugged the server into the wrong port. The server was out of service for 2.5 hours. Doh! My side of the issue took 10 to 15 minutes.

 

Oh man! Ad-blocking software has been detected! :'(

This website is run by the community, for the community... and it needs advertisements in order to keep running. Blocking our ads means your killing our stats!
Please disable your ad-block, or become a premium member to hide all advertisements and this notice.

While you're in your job, it'll be good for you. Not only document the current network configuration but also document the changes.

Ah, the good ol' "this other port is the same as any others" logic from a power strip.

NetEyeBall,

What does "RDC" stand for?

 

RDC stands for Regional Data Center. We have 7 data centers currently.

RDC auditing is outside my scope. That is for the RDC ops to do. But it is a good thing that is for sure!

It is looking like the issue wasn't with the cable movers (which I still think it is) but with the network support or network engineering that made a block change to the ports for a project. Dunno. We shall see what the root cause analysis comes up with.

 

Maybe its just me but I would recommend keeping it confidentail too... nomatter how small you may think it is.

 

Of course your descriptions are part of your documentation (much like a programmer adding comments to their code).

Documenting your base network is a pain in the arse. I remember once being tasked with tracing each and every cable in the server room, labeling them at each end, and mapping the cabling setup. Of course, that's only one aspect of network documentation but it's an important one.

Once the full, base documentation is done, it's only as good (as R.H. Lee says) as keeping it up to date. Any little changes you go through should be documented and in fact, editing the documentation should be part of your official procedure (make change, document change). If you keep up on this, it won't be a big chore. If you only update your documentation every six-months to a year, you are causing your own headaches.

 

LOL at techs and vlans. The amount of times I've had to explain patiently that not 'all' ports on a switch are necessarily just plug & play...

Case in point - I once worked somewhere (quite similar to where I am today in fact) that had a mahoosive Foundry core switch with a huge backplane. Everything was plumbed into this beast (sixty odd workstations, twenty servers, network devices - everything except the phone system basically). A guy pulled a server out of a dedicated vlan and plugged it into a port I'd configured in the dmz vlan. it took me twenty minutes to explain what a vlan was to this bloke - despite him being a CCNA. For the first 15 minutes of the 'conversation' he just flat out refused to believe that there even was such a thing as a vlan.

As a side note, and by coincidence, guess what I've been doing today? Documenting the vlans on the core switch in our data centre and creating a schedule spreadsheet for them