Deck Log featured image: Second Path. The outage wasn't yours. The downtime was. Tiles read Provider Down, Backup Link Up, Plan Practiced.
| |

The Outage Wasn’t Yours. The Downtime Was.

After 25 years of running infrastructure, I can tell you which outages are the hardest to sit through. They aren’t the ones in your own server room. They’re the ones where the status page belongs to somebody else.

Six Days in November

Between November 12 and November 17, the country got a tour of how much daily work rides on systems nobody in the building controls. It’s worth getting the sequence right before drawing lessons from it, because the details are where the lessons are.

On Thursday the 12th, Microsoft 365 went down and stayed down for most of a day. Outlook, Exchange Online, Teams and Copilot were all hit, and reporting on Microsoft’s own incident timeline puts the outage at nearly 24 hours. On Friday the 13th, the FAA ordered a nationwide ground stop that lasted more than four hours. The agency linked it to a newly installed router, and the root cause was still under investigation at last report. On Monday the 16th, JPMorgan Chase had a software problem that slowed transactions, and customers at other banks felt it whenever a payment touched Chase. United Airlines lost its systems for about an hour the same day. On Tuesday, Apple had trouble with iCloud sign-in and Apple Pay after an internal update, and Marriott’s sign-in was down for a stretch.

Two facts about that week matter more than the list itself.

First, none of it was an attack. Officials and the companies involved said the incidents were unrelated and that there was no evidence of a coordinated cyberattack. Every cause that has been reported so far is ordinary: a software defect, a new router, an internal update. That is the uncomfortable part. You can’t buy a security product that protects you from your provider’s change calendar.

Second, the largest outage of the week was spread by a failover. According to coverage of Microsoft’s preliminary incident report, the root problem was a defect introduced in a July update to the gateway servers that sit in front of Microsoft 365. It sat there for months. As the report itself put it, “Our normal load testing did not detect the latent issue.” Then three things lined up. Part of that gateway capacity was already offline for an unrelated reason. Teams had automatically failed over onto the same gateways that carry Exchange Online. The extra load triggered the defect, and the automatic failover that was supposed to protect the service moved the overload from one group of servers to the next. At the low point, most of those servers had stopped responding. Teams, OneDrive and SharePoint could move to other infrastructure and came back sooner. Exchange Online had nowhere else to go, which is why email was the last thing to return.

I’m not picking on Microsoft. That is a hard failure to foresee, and the company shared a detailed preliminary account within days. I’m pointing at it because it shows, at the largest scale there is, the mistake small organizations make all the time: a backup that shares a weakness with the thing it backs up is not a second path. It’s the same path with a second label on it.

Recovery Is for Your Failures. Resilience Is for Theirs.

Most continuity planning is about your own bad day. The server dies, the ransomware lands, the building floods, and you restore. I’ve written about that at length, including why a restore you have never tested is a rumor. Recovery time, recovery point, tested backups: all of it assumes there is something of yours to bring back.

A provider outage breaks that model. Nothing of yours failed. Your data is intact. Your backups are fine and completely beside the point, because there is nothing to restore and nowhere to restore it to. You can’t open a ticket that moves you up the line, and you can’t fix it by working harder. The only question that matters for the next several hours is whether your people can keep doing the work that pays the bills while somebody else repairs their system.

That’s a different discipline from disaster recovery, and it gets far less attention. Recovery asks how fast you can get back. Resilience asks what still works in the meantime. A business can be excellent at the first and helpless at the second.

Where a Small Business Is Quietly Single-Threaded

Large enterprises pay people to hunt for single points of failure. Small organizations usually find theirs the hard way. In my experience they sit in the same handful of places.

  • One internet connection. Every cloud service you pay for is exactly as available as the one line coming into the building.
  • One communications platform. Email, chat, calendars and video all live with one vendor. When it’s down, the tool you would use to tell everyone it’s down is also down.
  • Phones that ride the same line. A hosted phone system on the office internet connection is not a separate channel. It fails with the connection.
  • One sign-in for everything. Single sign-on is good security and I recommend it. It also means that when the identity provider has a bad morning, applications that are perfectly healthy become unreachable. This is one more reason to manage identity and access deliberately and not just switch it on.
  • One way to get paid. One card processor, one invoicing tool, one bank. The Chase incident showed that even people who banked somewhere else were slowed down when the other side of the transaction was affected.
  • One place the plan lives. The emergency contact list and the continuity plan are stored in the document system that is currently unavailable.
  • The plumbing nobody looks at. A single DNS host or a single domain registrar account sits underneath all of it. I covered that layer in It’s Always DNS.

None of these is a mistake by itself. Consolidating on one platform is usually the right call for a small team. It’s cheaper, simpler to secure and easier to support. The mistake is consolidating without ever writing down what happens on the day that one platform is unavailable.

A Second Path, Not a Second Copy

Backups are second copies. They protect data. What gets you through a provider outage is a second path: another way to do the same job that doesn’t depend on the thing that just failed. For a small business that doesn’t mean duplicating everything. It means a short list of inexpensive alternates for the functions you can’t go a day without.

Connectivity. A second internet connection from a different carrier, using a different technology, is the best-value redundancy most offices can buy. If the primary is fiber or cable, a cellular or fixed-wireless backup with automatic failover on the firewall covers the common failures. Ask where the lines physically enter the building. Two circuits in the same conduit fail to the same backhoe.

Communications. Decide ahead of time how the team talks when the main platform is down, and how customers reach you. A group text list, a conference bridge from a different provider, call forwarding from the office numbers to mobile phones. Pick one, write it down and make sure the contact list exists somewhere other than the system that’s down. A printed copy in a desk drawer still works on the worst day.

Getting paid and taking orders. Know what your point-of-sale and invoicing tools do offline. Know how you would take an order by phone and enter it later. If one processor or one portal is the only way money comes in, find out what the alternate is before you need it. That’s an operations question, and the time to ask your bank and your processor is on a quiet day.

The work itself. Ask each team what it needs in hand to keep working for one day without the cloud. Usually it’s a short list: this week’s schedule, the active job files, the price sheet, the customer contact list. Offline sync or a scheduled export of those few things is cheap. The goal isn’t a full shadow copy of the business. It’s enough to keep serving customers until the lights come back on.

Getting in. Keep a small number of emergency administrator accounts that don’t depend on the normal sign-in path, protected and monitored, so an identity outage doesn’t lock you out of your own systems. Check how your multi-factor prompts are delivered. If the codes arrive by email and email is the thing that’s down, you’ve built a loop.

Then apply the test the big November outage failed: does the second path share anything with the first? The same carrier under a reseller’s brand, the same sign-in, the same power strip, the same vendor on the other end. Redundancy that shares a dependency is decoration. Trace each alternate back until you find what it actually relies on.

Decide What Is Worth a Second Path

You can’t make everything redundant, and you shouldn’t try. Most of the value is in deciding deliberately. A simple version of the exercise fits on one page.

List the eight or ten things the business does that produce revenue or meet an obligation: answer the phone, quote a job, take a payment, ship an order, run payroll, submit a deliverable. Next to each, write the outside services it depends on. Then write what an hour, a day and three days without it would cost you in money, in commitments missed and in customers who go somewhere else.

The ranking that comes out usually surprises people. The function everyone worries about can often wait a day with no real harm. The one nobody thought about, such as the single shared mailbox that receives every purchase order, turns out to be the one that stops the business.

For each item there are only three honest answers: build a second path, write down a manual workaround, or accept the downtime on purpose. The third is a perfectly good answer for a lot of functions. “If the accounting system is down for a day, we wait” is a plan, as long as someone decided it in advance. What costs companies money is discovering the answer during the outage.

Practice the Bad Day

A continuity plan nobody has opened is a document, not a capability. The cheapest test there is takes an hour around a table: email and chat went down ten minutes ago and the provider says there’s no estimate. What does each person do next? Who calls the customer with the delivery due today? Where is the contact list? Who decides when to switch to the alternate, and who tells the staff?

You’ll find the gaps in the first ten minutes, and almost all of them are cheap to close. It’s the same reason I argue for writing the incident plan before the fire. The first hour goes to whoever already knows their job.

Two habits make the exercise stick. Test the failover for real at least twice a year by unplugging the primary connection and watching what happens. And add your critical providers’ status feeds to the things you watch, because you want to know before the phone rings that the problem isn’t yours. Half the lost time in a provider outage is spent troubleshooting your own network for a fault that isn’t there.

The Federal Angle: The Government’s Clock Does Not Stop

For contractors there’s a harder edge to this, and it’s often misunderstood in both directions.

Start with what the rules don’t say. Articles aimed at contractors often imply that CMMC or NIST SP 800-171 requires redundant systems. It doesn’t. As I noted in the disaster recovery piece, 800-171 leaves the contingency planning family out, because that family is about availability and the standard is about protecting the confidentiality of controlled unclassified information. No assessor will write you up for having one internet line.

Your contract may still hold you to the date, though, and that’s where the exposure is.

Proposals. Under the standard instructions to offerors in FAR 52.215-1, getting the proposal to the government on time is the offeror’s responsibility, and late is late. The exceptions are narrow. The one worth planning around covers an electronic submission that reached the government’s system by 5:00 p.m. one working day before the deadline. There is a separate provision that extends a deadline when an emergency keeps the government from receiving proposals. It protects you when the government’s system is down. It doesn’t cover an outage at a commercial service that only you depend on. During a week like that one, the practical moves are to submit a day early and, if you can’t, to call the contracting officer before the deadline, when an extension by amendment is still possible.

Performance. The excusable delay clauses, FAR 52.249-14 and its counterparts in the default and commercial clauses, excuse a late delivery when the cause was beyond your control and without your fault or negligence. A nationwide provider outage is plainly beyond a small contractor’s control. Fault is the open question, and I’m not aware of a published decision yet that applies these clauses to the November outages. Expect to be asked whether the work could have continued another way and whether you had a reasonable fallback for a dependency you knew was critical. When the failure belongs to a supplier, the test is stricter still. And even when a delay is excused, the relief is time on the schedule, not money. The commercial clause also requires written notice to the contracting officer as soon as reasonably possible.

Put plainly, a second path isn’t a compliance checkbox. It’s what lets you say, truthfully and in writing, that you did what a reasonable contractor would have done. None of this is legal advice. Read the clauses that are actually in your contract and ask counsel or your contracting officer when the stakes are real.

Where This Usually Goes Wrong

  • Treating backups as the continuity plan. Backups protect data. They don’t keep the doors open while a provider is down.
  • Buying a “redundant” connection that shares a carrier, a conduit or a power source with the primary.
  • Running the phones, the email and the chat on one platform over one line, and calling them three channels.
  • Keeping the continuity plan and the contact list only in the system most likely to be unavailable.
  • Installing failover and never pulling the plug to see whether it works.
  • Spending the first hour troubleshooting your own network because nobody checked the provider’s status page.
  • Assuming the provider’s service credit covers the loss. A credit refunds part of a subscription fee. It doesn’t replace a lost day of revenue or a missed deadline.
  • Waiting until 4:45 on the due date to submit the proposal.

You can’t prevent the giants from going down. November made that plain, and the causes were mundane enough that it’ll happen again. What you can control is smaller and more useful: which parts of your business stop when they do, for how long, and whether anyone in the building already knows what to do next. That work fits in a few afternoons, and the time to do it is while everything is up.


Veteran Forge Strategies is an SBA-Certified Service-Disabled Veteran-Owned Small Business providing IT infrastructure, operations, and cybersecurity support to small businesses and federal clients from Northern Virginia. If this month’s outages left you wondering how much of your operation depends on a single provider, mapping that out is a good place to start — get in touch. You can also read about how a fractional IT engagement works, or browse the rest of the Deck Log.

Similar Posts