Work stuff

It’s the Easter holidays at work so there are no students around. Not only can I now drive to work in a quarter of the time it usually takes, I can also muck around with the network without inconveniencing thousands of people. 😉

In an effort to track down the network problems we’ve been having for a couple of months, I installed a traffic analyser to see whether it would reveal anything unusual. I couldn’t believe how much crap I found! 😯

There were thousands of broadcasts taking place every second. 😕

Spanning Tree Protocol (which we don’t use and thought was turned off everywhere) was the main culprit. Ethereal revealed that it was still enabled on a few switches. As soon as I had tackled those, the occasional packet loss we had been experiencing stopped. 😎

I also discovered that there were dozens of IPX-enabled devices on the network – JetDirects and network enabled printers. We haven’t use IPX since ditching Novell 8 years ago, so these could safely be reconfigured. 😐

We stopped using WINS a couple of years ago too, but for some reason we didn’t disable NetBIOS on all our workstations. This meant that all name lookups were being done by broadcast rather than DNS, resulting in a great deal of unnecessary traffic. That was fixed easily via DHCP settings and Group Policy, but losing “NET VIEW” broke a few scripts. 🙁

I then fired up a few hundred workstations to simulate typical network load – it’s no good fixing problems when there are no students around and the network isn’t stretched. The packet loss came back almost immediately. 😳

A bit more sniffing with Ethereal discovered that our classroom viewer program, ABControl, was the culprit. Every 10-15 seconds, all classroom machines were sending out a UDP broadcast so that they could be picked up by the administrator program. With about 700 machines on this subnet, all doing this at precisely the same time, it was creating a broadcast storm and causing packet loss. 😡

It’s a shame to get rid of this program because the tutors love it, but there’s no way we can entertain that sort of traffic on the network. I’m pleased that we’ve finally got to the bottom of this problem though.