by Arlos » Mon Feb 19, 2007 2:06 am
This one wasn't me, but someone I worked with when I was at Mindspring several years ago. The guy was a junior tech, and was supposed to let one of the 2 senior techs (me and one other guy) do the actual programming changes to routers. Well, us 2 senior techs had gone to lunch, and he got it into his head to actually take one of our porjects and handle it himself. Anyway, we worked hand in hand with another ISP that actually owned the routers, and had to call them when we needed to make changes to get the root-equivalent-level password for the router we were going to. They were only supposed to give that out to me or the other senior tech, but the junior tech somehow got them to give it to them.
So, he goes in to the main router that connected all of Mindspring's customers in Atlanta to the rest of the world, makes a couple changes, gets disconnected, thinks nothing of it, considers it a job well done, and plans on being all self-important and flashign the ego when us senior techs got back from lunch. Suddenly, however, the tech support lines for Atlanta customers light up like a berserk christmas tree. People are connected, but can't see the world.
What the guy had done is change the IP information on the main backbone interface for the router, so that it was now unable to talk to anything else in the world. Period. He had also turned on its debugging levels to such a degree that it was overloading the router's memory and CPU. We couldn't even connet into it remotely to fix it, someone with a laptop had to race to the site and console in and fix it. Of course, with the debugging levels that high, it wouldn't even respond to console commands, because there was no memory left to process them. They had to compeltely disonnect every bit of cabling to this thing, so that there was no longer any input happening, before they were able to get in and shut down the debugging and then fix the addressing problem. Ultimately, every single Mindspring customer in the Atlanta area was down for several hours. Businesses, users, everything. Somehow the guy managed to avoid getting fired, HOW I don't know. He was never allowed to do ANYTHING with the routers ever again, of course, though.
No idea how much that debacle cost Mindspring, but it had to be significant.
-Arlos