This happened a few years ago, I was helping out a new department that sells network services to smaller customers. It was mainly going in, properly setting up their network for authentication, fixing cabling issues, explaining about planned IT expenditures, and a little tech support if they signed up for the long haul. At first it was just to help small cities get setup properly instead of fighting with what ever they could duck tape together, it's turned into the NS department actually remotely managing a lot of sites.
At this one site we had to do the whole shebang, they had 17 rooms, meeting rooms, file rooms, office spaces, lobbies, you get the picture it's just a big one floor office building. There was 1 big meeting room, it made me think of a church every time I was at this site. They wanted big things, they wanted all the computers networked, they were using serial to unix server to run software, and they wanted a new windows server, with megafast network to all the computers, they wanted internal email, they wanted to get to the internet and to get email from their residents. They all wanted to print to the large capacity copier, and they wanted to have all their city documents stored digitally for easy access.
We, me and two field reps, invaded their space, we crawled through the building, we did network wire maps, and we did electrical wiring maps, we drew pictures, and suggested servers and we had big presentation boards made up. And we got to the meeting to explain all of this to the city manager, office manager, and the department heads. I was the talking head, since I had the best grasp of all of the setup.
They agreed whole heartedly with the plan, and we got to work. They had the city engineer come in and do the electrical wiring, apparently he was a full time electrician, the city engineer job was part time, and charitable work. He was married to the cities arborist (tree girl, I called her Squirrely, her name was Shirley), so it was a good situation, he low balled the electrical work and I think he did it at cost to be honest.
We got them to build a small room new the electrical room, they insulated the heck out of it, and installed an air conditioner just for this one room, we even got them to put in a security door with a number pad lock for security. We set them up with all network connections, and email accounts, plus access to the internet at every desk. I've never seen so many AOL icons instantly appear before the day we gave them all access to the internet. We got them setup with a document management system, and they assigned a few people to scan documents half days a week. They were loving their megafast network and every time someone printed to the copier, you'd think it was Mardi Gras off in that room. I bid them good luck, left a card and headed back to the bear cave (at the time I was a grumpy network admin, so my office was the Bear Cave(don't poke the bear)).
A few weeks later there was a problem, they were having internet trouble, at the time they didn't have anything to manage internet traffic. A simple nat firewall built into the router from the ISP. So I work remotely to try to find the problem, it is indeed slow. The ISP claims they sit around 90% utilization every day all day, all day happened to mean 24 hours this time. So I figure there is someone downloading warez, or maybe one of the workstations has a headless ftp host on it. But nothing shows in the simple logs of the router. I let them know I'm working to find the problem, and decide i'll have to put something in between the lan and the internet that logs a little more than the router.
I get there and drop an linux proxy boss in and router all outgoing traffic through it, I wander around and visit with some people. Unlike the people I work with daily, these people seem to like me, no one is complaining to me about anything, it's not my fault that the file transfer isn't instant to a customer site. After an hour or so I go back to the proxy, and tear into the logs....nothing...nothing abnormal is going on from any machine going through the proxy. So it must be something not going through the proxy.
I check the router again, it's only showing the proxy and some pin hole traffic for the SMTP to the email server. Nothing much going on there, I write down the IP and remote to my office, I run a port scan against the IP address and i'm not seeing anything special open. I just happen to notice the subnet on my last look at the router, it's .248, that would be 5-6 IPs for this router. I go into the phone closet and get to prowling, it's a router but it's not a cable modem with router, there is a cable modem up near the ceiling with a 4 port switch built in to it, with 2 cables coming out. One cable is going to the router, the other cable is going....well, that's a good question.
I call the field rep for the area and he's going to come over, it was company policy at the time that you never topped a wall without at least one other company employee present. The rep showed up and we got to look around.
We could follow the mystery cable for about 10 ft and then it disappeared into a trunk of cables heading to the other side of the building. So we got to fox and hounding everything, couldn't find it on the remote side, we found every other connection. Everything was labeled properly and we didn't find any connection that wasn't labeled. I shrugged and unplugged the cable, and the internet got fast almost immediately. We checked around and no one lost their connection, everyone was tickled pink over how fast we made the internet. The rep and I went back to the network closet, and I clipped the cable off above the ceiling tile. I then told the rep to keep an eye on that cable end, if they complained about the internet being out or slow then check it first.
Week later, the internet was slow again, the rep got there and the cable had been spliced to make it long enough to plug back into the cable modem. He called me and I told him to clip it, and ask them for a list of who had access to that room. Maybe they gave access to a new guy we didn't know about.
Week or so later, the internet slowed again, this time the cable had a shroud on it, metal braided sleeve, that covered the cable from the modem to the cable trunk where we lost it. This time I told him to cut the cable at the trunk, and tell the office manager to change the code. I asked about having him setup a camera if they had any, they didn't and wouldn't, something about the city charter banning them.
Yet again, the internet go slow, the rep called and let me know he found a note this time.
Look whoever is cutting this cable is taking out the internet for the road maintenance staff, you need to stop cutting this cable. This is ridiculous, stop cutting the cable.
I didn't know they had a road maintenance staff, thought it was all county roads. Turned out to be the Waste Disposal department, they decided to rename themselves. They were in a tin metal building out back of the main building, they were mostly responsible for litter and tree disposal, but they did maintenance on city vehicles and occasionally filled potholes. In all of the changes being made, it was never brought up that they needed access to the network. They still did everything on paper for the most part, work orders, time sheets, the city manager and office manager didn't even know they had a computer back there. They had took it upon themselves to get the cable guys to run them a cable down and out a underground pipe to the outbuilding. They picked a computer out of the pile of replaced hardware and plugged it up. All without informing anyone, they claimed they expected to be denied a computer if they asked.
Then because we didn't know about the computer, it didn't have any access to anything. We had assigned IPs in DHCP statically using MAC address and reserved all the unused. So they plugged into the cable modem, got a internet LIVE ip address and slowly but surely acquired the nastiness that was eating all the internet bandwidth.
The rep found that someone had setup a headless ftp host, and there was quite a bit of german porn being shared with someone someplace. There were also like 5000 malware on the pc. We got them a new pc, then got them behind the simple router, and everything was great for a few months.
We later went back and installed a proxy server with daily reports because the city manager caught someone out there looking at NSFW stuff on the PC. And soon after the Road Maintenance department lost the PC access in their building because someones car didn't get fixed on time and they were caught playing video games.
We ended up using the cable run to that building to install a security webcam so they could be better managed.
**** i'm in a story writing mood today, having to reimage servers for next round texting and i'm stuck watching progress bars....it's always progress bars....