IT Horror Stories: The Preventable Disasters I’ve Seen in 25 Years
A Halloween list with a straight face: the five IT disasters I’ve watched do real damage in twenty-five years. Not one of them was a freak accident. Every one was a risk somebody already knew about and hadn’t gotten around to fixing.
Every IT person has a horror story. The outage that ate a weekend. The restore that wasn’t there. The one server nobody wanted to touch. Trade them over a beer and they sound like bad luck — the technology turning on you in the dark.
They aren’t bad luck. After a career that runs from wearable-computing startups and public companies through twenty-plus years of enterprise infrastructure, I can tell you the scariest stories in IT almost never start with something nobody saw coming. They start with something everybody saw coming and nobody owned. The call was coming from inside the house the entire time.
So, in the spirit of the season, here are five of them. Each one is real — a few of them are mine — and each one ends the same way: with the boring, unglamorous fix that would have kept it from ever becoming a story.
1. “We have backups.” The night the tapes were blank.
The most expensive four words in IT are “we have backups.” Not because they’re a lie, exactly. The backup job runs. The dashboard is green. Somebody checks a box on an audit once a year that says yes, we back up our data. Everyone sleeps fine.
Then the day comes — a failed array, a ransomware hit, a deleted database — and someone finally tries to restore. And the backups are there, technically, but the restore fails. The job had been silently skipping the one database that mattered for months. Or the tapes were being written but never verified, and half of them are unreadable. Or the restore works, but it takes eleven hours nobody budgeted for, and the business is down the whole time.
A backup you have never restored from is not a backup. It’s a rumor. It’s a hope with a schedule attached. The only thing that tells you whether you can recover is actually recovering — pulling a real file, a real database, a real server, back from your backups, on a regular basis, and timing how long it takes.
The fix is unglamorous and it works: keep more than one copy, in more than one place, with at least one copy you can’t overwrite or encrypt from the live network — the old 3-2-1 guideline, now usually extended with an immutable or offline copy. Then test the restore on a schedule, not after the fire. CIS Control 11 puts data recovery and recovery testing in the top tier of controls for exactly this reason: having the backup and being able to restore it are two different capabilities, and only one of them saves you. I’ve written up the full version of this — RTO, RPO, and why the restore you’ve never tested is a rumor — because it’s the single most common gap I find.
2. “Only Dave knows how that works.” Dave left in 2019.
Every organization has a Dave. Sometimes Dave is a person; sometimes Dave is you. Dave is the one who understands the server that runs billing, or the script that closes the month, or the firewall rules nobody has touched in six years. It all works, so nobody asks how. The knowledge lives in exactly one head, and the organization quietly bets the business on that head never leaving, never getting hit by the proverbial bus, never going on vacation during a crisis.
Then Dave leaves. And the system that was “fine” becomes a black box that everyone is afraid to breathe on, because the one person who knew why it was built that way is gone, and the documentation is a folder with three out-of-date screenshots in it.
I’ll own this one personally. When I rebuilt a twenty-year-old operations system that three companies run on every day, I wrote honestly that the old application had no documentation beyond what lived in my own head. That’s not a brag. That’s the exact risk I’m describing, and I was the single point of failure. The rebuild wasn’t just about newer technology — it was about getting that knowledge out of my head and into something another person could pick up.
The fix is documentation and cross-training treated as part of the job, not a someday project. Not a novel — a runbook. How it’s built, why it’s built that way, how to restart it, who to call, where the keys are. A good test: if you got hit by the bus tomorrow, could a competent stranger keep the lights on from what you’ve written down? If the honest answer is no, the bus-factor is one, and that’s a horror story waiting for a cast.
3. “It’s a quick change, no need to tell anyone.” Narrator: it was not quick.
This is the one that gets the people who are good at their jobs. The confident ones. A small change — a firewall rule, a DNS record, a config tweak, a “while I’m in here” cleanup — that is so obviously minor it doesn’t need a ticket, doesn’t need a window, doesn’t need a heads-up. Thirty seconds of work.
And then it isn’t. The quick change takes down email. Or it works perfectly and silently breaks something three systems away that nobody connects to the change until two days and a lot of panic later — because there’s no record that the change ever happened. The outage isn’t the disaster. The disaster is that nobody can answer the first question every incident asks: what changed?
The fix isn’t bureaucracy, and I say that as someone who has no patience for change-control theater. It’s a lightweight, actually-followed habit: write down what you changed, when, and how to undo it, before you change it. Make real changes in a window, not at 4:30 on a Friday. Have a rollback before you have a problem. When something breaks right after, the change log turns a two-day mystery into a two-minute answer. The best incident responders I know aren’t faster at fixing things — they’re faster at knowing what to do in the first hour, and that starts with knowing what moved.
4. “That’s not plugged into the UPS, is it?” It was the one thing that wasn’t.
Here’s a cheap one that has humbled smarter people than me. The power blips. Everything rides it out on battery like it’s supposed to — the servers, the switches, the SAN. Except one thing. The one thing you forgot, or someone unplugged to vacuum and plugged back into the wall instead of the UPS, or the one circuit that was “temporary” four years ago. And of course it’s the domain controller, or the core switch, or the firewall — the one piece that takes everything else down with it when it drops.
Resilience isn’t about how much redundancy you bought. It’s about the one gap in it you didn’t know was there. The expensive UPS, the generator, the dual power supplies — none of it matters if the critical path runs through the one outlet that isn’t protected. The monster is always the thing you stopped looking at because you assumed it was handled.
The fix is a walk, not a purchase: physically trace the critical path and verify it, don’t assume it. What’s actually plugged into protected power? What happens, in order, when the power drops for real — have you ever watched it, or do you just believe the diagram? Keep a current inventory of what you have and what it depends on, because the server you forgot exists is the one that will find you. You can’t protect what you’ve lost track of.
5. “The certificate expired at 2 a.m. on a Saturday.” They always do.
Nothing in IT has a crueler sense of timing than a TLS certificate. It doesn’t fail when you’re watching. It fails the instant it expires — a date set months or years ago by someone who has since moved on — and that date is always 2 a.m. on a holiday weekend. Suddenly the website throws security warnings, or the app won’t talk to the API, or email stops flowing, and the clock that’s been ticking quietly for two years goes off all at once.
Expired certificates have taken down payment systems, government services, and global platforms — not because the fix is hard (renew the cert) but because nobody was watching the clock. The knowledge that it was going to expire existed. It just didn’t live anywhere that would tell you before the deadline instead of after.
The fix is to stop trusting human memory for anything with an expiration date: inventory every certificate, domain, and license, and put real monitoring and alerting on the dates. Something that warns you thirty days out, loudly, to a place a human actually looks — not a calendar reminder the person who set it took with them when they left. Automate renewal where you can. The whole point of good monitoring is to know first, before your customers tell you. A cert expiry is the easiest outage in the world to prevent and one of the most common to suffer, and the only thing standing between those two facts is whether anyone is watching the date.
The twist ending: the monster was always inside
Line those five up and the pattern is impossible to miss. The untested backup. The knowledge in one head. The undocumented change. The gap in the power path. The clock nobody was watching. Not one of them was a sophisticated attacker or an act of God. Every single one was a known, boring, deferred risk — a thing somebody could have pointed to months earlier and said “we should really deal with that,” and then didn’t, because it was working fine right up until it wasn’t.
That’s the actual horror of IT. It isn’t the dramatic breach. It’s the slow accumulation of “we’ll get to it” — the risks everyone can see and nobody owns, quietly compounding in the dark until the night they all decide to introduce themselves at once. The scary part was never the technology. It was the backlog.
Where this usually goes wrong
- Green dashboards mistaken for proof. The job ran is not the same as the data came back. If you’ve never restored it, you don’t know you can.
- Documentation treated as a someday task. It never becomes urgent until the one person who didn’t need it is gone — and then it’s too late to ask them.
- “Quick” changes with no record. The speed you save skipping the log, you pay back tenfold the next time something breaks and nobody knows what moved.
- Resilience assumed, not verified. The redundancy you paid for is only as good as the one gap you never walked the floor to check.
- Expiration dates living in someone’s head. Certs, domains, licenses, warranties — if the only reminder is human memory, the outage is already scheduled.
How to exorcise your own haunted house
None of the fixes here are expensive or clever. That’s the point — these are the disasters you prevent with discipline, not budget. If you want to spend an afternoon making your environment less haunted, start here:
- Restore something real this week. Not check the backup ran — actually pull a file or a database back and time it. If you can’t, you just found your worst risk.
- Find your bus-factor-of-one systems. List the things only one person understands. Those are your ghosts. Write the runbook before the person walks out the door.
- Start a change log, however simple. Date, what changed, how to undo it. One shared note is infinitely better than nothing when the next “what changed?” lands.
- Walk the rack. Trace the critical path on real power, and while you’re at it, check who still has access that shouldn’t. Verify the things you’ve been assuming.
- Inventory your expiration dates. Every cert, domain, and license, with an alert that fires early and loud, to a place someone watches.
Do those five things and most of the horror stories in this post simply can’t happen to you. The disasters that make the best stories are, overwhelmingly, the ones that were the most preventable. For a federal buyer or a prime evaluating past performance, that distinction matters: the organizations worth trusting with a mission aren’t the ones that got lucky and never had a bad night — they’re the ones that did the boring work so the bad night never came. The scariest thing in your environment isn’t a hacker in a hoodie. It’s the risk you already know about and keep walking past.
Veteran Forge Strategies is an SBA-Certified Service-Disabled Veteran-Owned Small Business providing IT infrastructure, operations, and cybersecurity support to small businesses and federal clients from Northern Virginia. If you’ve got a system nobody wants to touch, a backup nobody has tested, or a clock nobody is watching, that is exactly the engagement — get in touch. You can also read about how a fractional IT engagement works, the path from SAM.gov to a first subcontract, or browse the rest of the Deck Log.