Server disappearing off the net

[TimO]

We've got a server that keeps disappearing off of the network.  Luckily I didn't set it up, so It's Not My Fault [tm].

It's a Debian box, and it looks like something fairly fundamental is happening, since last time it happened, I pointed nmap on another machine at it, and as far as nmap was concerned, the machine wasn't there.

There are a few entries in the logs which to me just suggest that NFS and Exim had issues seeing the outside world, and these entries are other things being upset by the net going all squirrelly and not the cause of it.

Can anyone make any suggestions of likely things that I can look at?  I'm sitting here "tail -f"-ing the output from syslog and hoping that things die, so I can watch what happens, but so far it's working fine!

[Greenbank]

I take it you've put "*.debug" in syslogd.conf too...

It may also be outputting panic info to the console. So if it's headless then you might want to whack a monitor on it (or set it up to have console over serial). And if you're on the console note that the panic info may be being pushed off the screen by your tailing...

[rae]

Check physical first - cables, cards and duplex settings.  What does the connected switch have to say about it?

[pcolbeck]

I'm with Rae. Look at the switch stats.

[TimO]

It's connected to a dumb switch, but other machines connected to that switch do not have any problems with their network access, but equally cannot see this machine when it goes a bit nuts any more than any remote machines.

It doesn't permanently die, since it seems to come back eventually, and there seems to be nothing obvious in the logs.  There are entries for any debug events to be logged in syslogd's conf file.

I've checked the cabling and switch as best I can, and they all seem fine.

As usual in these situations it's doing it's best impersonation of a watched kettle at the moment...

[pcolbeck]

Try moving it to a different port on the switch. Try a new patch lead. Check the speed and duplex of the Ethernet port and see if its the same as that on a box that isn't having any issues. Buy a managed switch. Unmanaged switches are for home use only.

[TimO]

It's not a particularly heavily used machine, and I can't easily justify buying a managed switch to hang it off.

It's not my machine, but the guy who does run it doesn't really know what's happening, but is happy to give me root access so I can poke around and see if I can see if there is anything obviously wrong.

No one other than him or me has root access (as far as I can see...).

When this happens it's more of an annoyance, since it comes back under its own steam.  I'm just not sure why it's going a bit doolally.

[rae]

Assuming it is lightly used, fire up Wireshark and leave it running on the interface.   Next time it barfs, look at the results.    Hardware problems are generally pretty obvious if you have access to the logs, and are the first place to check - duplex settings are crackers.   

By "not seeing it" are you sure that the server is unhappy - i.e. that it not an external problem.   DNS perhaps? 

[Greenbank]

Duplicate IP addresses?

Before we had a DHCP range the usual routine was to ping an IP on that subnet and if there was no response they'd use that IP address.

Of course that relied on people not typoing either address and interpreting the ping results properly.

Snooping the network traffic for arp requests is interesting. Make a note of the machine's MAC to compare when it next goes screwy.

[TimO]

Actually, given that this is in academia somebody nicking the IP address isn't entirely implausible, so I will make sure that I've got a record of the MAC address.

I've also got a screen, keyboard and mouse on the machine now, so I can fiddle with it directly next time things go castors up, which the seem determined not to do at the moment.  Of course it's possible that keep a TCP connection open to the machine is stopping things from failing, if it is something weird with the network setup.

Thanks for suggestions, I've got Wireshark on this machine, so it may well be useful to have it sat watching that machine if necessary.

[TimO]

It looks like it's something to do with the networking is failing on the server.  When it happens, if I go to the machine and (i) ping a machine elsewhere or (ii) restart the networking or (iii) type in "route" to show the routing table, then after a short pause, all three of these cause the server to start responding again.

I still have no idea why the network is dying like this though...

[aglet]

It looks like it's something to do with the networking is failing on the server.  When it happens, if I go to the machine and (i) ping a machine elsewhere or (ii) restart the networking or (iii) type in "route" to show the routing table, then after a short pause, all three of these cause the server to start responding again.

I still have no idea why the network is dying like this though...

Might it be a driver bug?  See if there's a more recent version of whatever driver is required for the NIC (or bung another one in from a better different manufacturer and try that instead).  Does the driver have a debug parameter you can pass to it?  See Documentation/networking/<driver>.txt

[pcolbeck]

Could also be a duplicate IP address. If the server has been idle and the other machine with the same IP address has been active on the LAN then all the other machines will have the rogue machines MAC address in their arp caches (IP address to MAC address tables). If you do a ping from the server or restart one of the network services it's possible (certain with ping) that you are putting a packet onto the LAN; this will flush the rogue MAC address from the other machines arp caches and replace it with the correct one and off you go again.

[nuttycyclist]

From another machine, when the server is "dead", what does "nbtstat -A {ip.address}" respond with?   (might not work, as I'm windows based here, but we'll often see a different computer name returned from the netBIOS remote machine table thanks to a screwey network.

[pcolbeck]

Another thing to try when the machine has fallen off the net is:

On a windows box try to ping the server that has a problem then do:

arp -a

Check to make sure that the MAC address against the server IP address is actually the MAC address of the NIC in question on the server.

This of course assumes that server and the windows PC are on the same subnet as if not you would just get the MAC address of the routers Ethernet NIC.

[TimO]

We've solved it at the moment by the somewhat bodged approach of a cron job that causes ping to fire a packet off every ten seconds to another server.  This appears to keep things up.

Apparently the guy who runs it is planning to re-install the server in the next few weeks, so I think I've spent enough time on it, and hopefully after the re-install things will all work perfectly (as if...?!)

[TimO]

After a lot of faffing about, experimentation etc, the official reply from our networking people is that "It appears to be a problem due to the normal operation of the new switches."

Which is impressive.  If these switches don't see an outgoing packet from the server every 5 minutes or so, they just seem to give up on it.  The line is still there, it appears to be up and operating, but has no activity.  Once the machine sends a packet out, everything works again, but no incoming packets ever get to it when it's in this state.

I'm very impressed that we have networking support who can't actually come up with a better solution than we did ourselves, ie a cron job pinging every five minutes.  Even my £50 router at home doesn't loose machines when they don't do much, but apparently we buy and install expensive managed switches that do.

[Greenbank]

It's "features" like that which really piss me off.

[tiermat]

After a lot of faffing about, experimentation etc, the official reply from our networking people is that "It appears to be a problem due to the normal operation of the new switches."

Which is impressive.  If these switches don't see an outgoing packet from the server every 5 minutes or so, they just seem to give up on it.  The line is still there, it appears to be up and operating, but has no activity.  Once the machine sends a packet out, everything works again, but no incoming packets ever get to it when it's in this state.

I'm very impressed that we have networking support who can't actually come up with a better solution than we did ourselves, ie a cron job pinging every five minutes.  Even my £50 router at home doesn't loose machines when they don't do much, but apparently we buy and install expensive managed switches that do.

Hmmm, now let me guess, Cisco?  they have really really brilliant features such as this....

They call them features, I call them a PITA...

[rae]

Which is impressive.  If these switches don't see an outgoing packet from the server every 5 minutes or so, they just seem to give up on it.  The line is still there, it appears to be up and operating, but has no activity.  Once the machine sends a packet out, everything works again, but no incoming packets ever get to it when it's in this state.

I'm very impressed that we have networking support who can't actually come up with a better solution than we did ourselves, ie a cron job pinging every five minutes.  Even my £50 router at home doesn't loose machines when they don't do much, but apparently we buy and install expensive managed switches that do. 

Hmmm.  If they are Cisco, I can confirm that my managed switches (and I have a lot of 6509s....) do not exhibit this behaviour.

[ScumOfTheRoad]

Which is impressive.  If these switches don't see an outgoing packet from the server every 5 minutes or so, they just seem to give up on it.  The line is still there, it appears to be up and operating, but has no activity.  Once the machine sends a packet out, everything works again, but no incoming packets ever get to it when it's in this state.

Eh? I don't get this one.
I can understand a MAC address table in the switch being flushed every five minutes - that would effectively "forget" the port to which the server is connected.

But why would that stop incoming packets for that server getting to it?
A switch is a multi-port learning bridge. So packets to a MAC address which has no known port should surely go out on them all till it is recognised? How else would a new device ever be recognised on the LAN, or how else could you plug a server into a new port?

[pcolbeck]

I'm sorry but that sounds like bo***cks to me. My job is configuring switches and routers and has been for the last 15 years and I have never heard of anything like this.

Pat

Cisco CCIE #2305

[TimO]

Well, I don't know the exact cause, we don't have any access to the switch, but if we don't touch the server, for just over six minutes, and then ping it, nothing happens.  Watching the activity lights on the back of the server they flicker away like made for this six minutes, and then just stop flickering, the switch seems to stop sending anything (including broadcast packets) to the machine.

Some sort of packet leaving the machine every five minutes cures it.  Moving the machine onto one of the older switches cures it.

It's a bit of a pain.

[pcolbeck]

Your ping is refreshing a cache somewhere probably. Either an arp cache or a mac-address table. The switch guys are fobbing you off.

[TimO]

Unfortunately, we talk to the IT support bods, eventually they agree it's a networking issue, and go and talk to the networking bods, and they tell us it isn't there fault, it's the new switches...

It wouldn't surprise me if they haven't got a clue what they're doing, I remember taking about ten minutes explaining to one of them once that it was possible for one machine to actually have more than one IP address associated with it... (ignoring the fact that until he started buggering about with the DNS it had been working like that for several years).