We have some family friends that we trade favors with. He is a general contractor and handyman. For my part, every time I pick up a tool, there is a pretty high probability that someone is going to land up in the ER. But I know my way around computers and networks.
I owed my friend (let's call him HM for Handyman) a favor, as he recently installed a new microwave for me. He needed a new router, possibly a range extender and an IP enabled surveillance camera configured. All relatively easy stuff, right?
So I arrived at his house, all hopeful and optimistic that I could get everything done in an hour two at the max. Let's start with the router. It's an ancient LinkSys 802.11g router, possible from the Eisenhower administration. I pull out my phone to look for SSIDs, and there I see my friend's SSID broadcasting as a very strong signal, secured with WPA2. (I'm right next to the router at this point). As I'm watching the site survey, I notice another SSID '#EvilCableCompany#wifi', also broadcasting as a very strong signal - but unsecured. No password needed.
'That's strange', I think to myself (this is almost always a prelude to trouble). So I set my phone to connect to the open SSID. It connects almost instantly. I'm able to browse the Web. Google, Wikipedia, CNN - just about every website I pull up loads almost instantly.
On a hunch, I look at the cable modem. It's one of the newer models from #EvilCableCompany#. This model happens to come with a cable modem and WiFi router in one box. I converse with HM:
Me: "When did you get this modem?"
HM: "About a year ago. It was installed by #EvilCableCompany#"
I power down the modem and the router. Suddenly, both SSIDs disappear - the secured one, and the open one. "Uh Oh."
After looking up the unsecured routers WiFi gateway address, I was able to connect to the admin page - with default username and password. Yup - the installers had connected HM's old router to the modem (with all defaults in place)...but had also left the inbuilt WiFi enabled and unsecured. I removed the old router and secured the builtin WiFi with WPA2. Then installed the IP camera and range extender.
This means that for about a year HM's cable modem was broadcasting an unsecured SSID. Which means that Joe Public (i.e. any of his neighbors) could have attached and done pretty much anything they wanted without consequences.
tl;dr - contractors hired by #EvilCableCompany# couldn't figure out how to disable the internal Wifi. So they decided the best course of action was to leave it open and unsecured.
This is my origin story, it's laid out in chapter format and I seek to share something different because there is nothing new on my cutover story yet
$Me - IT student, really really good with computers and networks already, couldn't take Senio4000 level classes as a freshman due to some CompSci rule apparently.
$Systems - Grouchy, didn't really fix anything to the public eye, somehow unstable DHCP using public IP's internally and 6 DNS entries per IP was a normal configuration.
$ITSec - Had some very interesting ideas about network security. $NetTeam - Smart, overloaded, at the mercy of someone else calling the shots. $CIO - Got problems somewhat sorted out, definitely better than the previous 2 years. $Bradford - Actual name of the NAC, it's terrible and very forgetful implementation.
This was the story, campus network generally was bad, when it worked, it was okay, but there were many days where no one could get online, sometimes it was the fault of IT, usually just DHCP dying, and lastly because of the magnetic attraction fiber has with heavy equipment - as it is SCP-3709 compliant
it's great read and it explains why fiber gets cut so much.
Problems would always manifest themselves in the Fall/Autumn semester which was the freshmen flood of new students.
I immediately recognized the problem even before I started - DHCP scope/pool exhaustion! IT was informed promptly. IT did not succeed in improving the situation. Off and on the network would go, round and round IT would go, never to stop or make anything much better.
One time, at a student banquet, we got to meet various upperclassmen and visit the job fair, one guy was talking in an IT group about the student who kept hammering on them to fix the internet. *I introduced myself as that student* He responded in a joking manner, so you're the one who kept telling me how to do my job! I have ideas, I know of ways that work, trying to help *shrug*
So, I applied and made it on a helpdesk position at the school, never did make it to Systems or NetOps, but I rocked that helpdesk and was assigned team lead to
teach interns what it's like to wrangle computers.
A year or so later, a wild IT Security officer appears! They host an open student forum about various concerns and making improvements for the campus. I ask about getting off of public addresses internally, she says due to various compliance reasons, they can't do that yet.
$Me - Ooookay... can you at least fix DHCP to not crash every week? $S - Uhhhh, work in progress...
A few months later due to $ITSec's decisions... $Me - Hey guys, do you have email? Jack - No... Steve from a previous story about a jar
Jack inquires in the weekly IT meeting, $ITSec had changed system passwords without telling anyone and broke about 7 different and very important services. Including a firewall rule change that denied all email connections in and out of the College.
That was a fun morning on the helpdesk, the phone never did stop ringing...
3 month's later it's midterm exam week and as the evening progresses, the connections are dropping off one by one, not good, not good!
I run over to the datacenter and inform them. $Systems- No it's not, what do you know kid? $Me *internally* Apparently more than you... $Me *externally* Watch, see my phone not getting an address? The network/DHCP is dying! $Systems- Bugger off!
You guessed it, the entire network floundered for the rest of the weekend.
A New CIO: His arrival is announced in a very formal and professional way, like an esteemed scholar would write a sonnet at sunset. I email him with the eloquence of a Renaissance poet describing how there is more downtime than uptime and DHCP must be fixed to not ever crash under extremely high demand every fall/spring. He acknowledges it and says it will be prioritized accordingly.
For a while, things did get alot better, then there was some political power struggle which drove off a good part of the IT team and that means all of the good people with great ideas left...
Suddenly, a new NAC arrives without warning or surveying the students on what they want out of an awesome network. Bradford was unleashed and stomped out all hope of having a reliable, fast, or efficient network access.
Me *thinking* - You know what, since all web requests are redirected the portal, let me change my DNS to 220.127.116.11, BOOM! Bypassed! Next semester, all DNS requests are now forwarded internally through Bradford, you're welcome said the IT guys. ;)
Well kids, it was fun while it lasted.
Bradford was notorious to forget previously registered staff computers, three computers a week was the average where a professor would submit a request that they're not getting email/internet anymore.Can IT reach the computer remotely? Nope! What's the IP? Bradford Isolation network, figures *rolls eyes*
We had to submit port and computer exemptions all the time just to get computers imaged, staff online, and new systems deployed, it was a huge waste of time and $DesktopSupport had to bug $Networking quite frequently to speed up the process.
They moved Bradford to wired only because of how much of a pain wireless was for the students, I would say that there was a 25% failure rate to get people online for the wifi. After $Net moved to a $HoneyComb PPSK, it dropped/dripped the failure rate down to 10% but for some reason, $ITsec did not believe in 802.1X/Radius even after multiple professional colleagues recommended it.
Once again, the PPSK method was not presented or held an open forum for wifi modernization, they just announced, hey we're doing a thing, call the helpdesk if it breaks, may the odds be ever in your favor. During this time, they finally NAT'ed the wifi network, it only took asking them 4000 times and 15 years of prep time /s.
Last in the whirlwind take, internet failures on the WAN side happened multiple times, and there was one outage that was $DeathStarISP's fault and was never disclosed what happened, because when you kill cellular + wired service for a large portion of the state, there's going to be a rather large and annoyed set of students needing to finish exams.
Randomly one fine spring afternoon, the internet just stopped, and it wasn't DNS or DHCP like before. $RandomProf - Well, there goes the online instructional tools, powerpoint time! We get an announcement later saying there was digging happening in a nearby city, combined with the SCP compliant fiber and assorted heavy construction that goes with the project, they predictably they cut the only connection serving the school.
Did they relocate the fiber away from the construction area? No....
Did it get cut more than once? Yep! 4 times during especially busy parts of the semesters.
Did they learn their lesson and get a second connection? Eventually...
Throughout all of it, I learned what not to do, I discovered some new things, aced senior level Networking without ever having to study, and sharpened my tech support skills tenfold while repairing friend and company computers/figuring out the kludgy network design/asking why on earth would you do it this way?