After the Microsoft Outage: Inside the High-Stakes Reality of Fixing a System Crash

Office workers in a panic from a fire.
When this system is down, do not panic. Image Midjourney v8.2

 You learn a lot about a person when they are in a difficult situation. Adversity tends to reduce a person to their most fundamental level. Some develop focus, and others fall apart in ugly fashion. As a technology professional, you often wind up in these crisis situations because networks, email, and instant messaging keep the business world spinning. This week, Microsoft suffered a serious outage, and I wanted to give you an inside look at what it's like to fix something like that.

Microsoft Outlook, Teams, and Office 365 went down for about 19 hours this week. As of Saturday, November 14, Microsoft says that service is fully restored, but intermittent issues are still being reported. I have lived through enough of this to know it changes you. While we couldn't control Microsoft's servers this week, here is what happens inside the war room when our own systems go down.

A major system outage begins like a typical day. You might have a cup of coffee at your desk and a bagel nearby to munch on. Nothing is out of the ordinary, but you notice a few reports start to trickle in. The email is behaving funny, or a piece of software is acting strange. After a sip of coffee and a bite of the bagel, you do a quick triage, decide the steering committee meeting is more important, and prepare for it. The sunny situation darkens quickly as more people come into the office and reports of poor software behavior increase. At this point, you put the steering committee meeting on hold and begin routine system checks.

Something looks wrong, but you cannot put your finger on it. Warning messages appear in log files, and servers are running, but response times are slow. You are about to start making phone calls when the trickle of troubling signs becomes a flood. Servers are shutting down, people are offline, and products are stuck on the dock because they cannot be processed. The system is down.

If email or instant messages still work, a tsunami of alerts lands in your inbox, with the most serious coming from the vice president, who has no time to speak with you and wants to meet you in their office. They want answers: what is happening, and what you are going to do to fix it. Finally, they want a timeline and regular updates. Now you and everyone involved with the software are in a conference room, hashing out how to fix the problem.

The situation is frantic because money is on fire. Business has stopped, and people are standing around getting paid for nothing. The vice president expects updates each hour so the CIO can brief the CEO. You are processing a lot of information from technical professionals, trying to diagnose a problem and then fix it. As a team lead, you are counting on them to stay calm and focused.

As the minutes turn into hours, you exhaust likely scenarios and grasp wilder theories. Perhaps a software patch on a cloud server was incompatible with the enterprise software. Maybe a bad data field triggered an infinite loop that crashed the software. Some combination of random factors came together to create this mess, and you and your team must figure it out.

You start to call home to your family to tell them there is a crisis at the office and you might not be home until it is fixed. The vice president, who calls every hour, grows more impatient, and you start mentally packing your desk because, by hour eight, it sure feels like someone has to take the fall. It is a brutal emotional obstacle course.

By hour 12, you are a blurry mess of coffee, cheap takeout, and exhaustion. You have a working plan, and the team is slowly testing out ideas and fixes. Eventually, the system comes back online. You report to the vice president and give them a preliminary debrief. Despite everything, you are expected to come into the office early the next day. People have questions, and you seem to be the only one with answers.

That is what a typical software outage is like for a scrum master or project manager. After the executive inquisition, you might keep your job or clean out your desk. Either way, it was dangerous and stressful. Fortunately, they don't happen often, but when they do, they're the ultimate test of your leadership.

It has been a rough week for the technology business. Smart, hardworking people are dealing with increasingly fragile technology stacks, and they're keeping the planes flying and the email updating. Remember this during the next major outage, and keep people like me in mind as we try to fix the problems that crop up in our increasingly technological lives.

Until next time.

Edward J Wisniowski

Edward J Wisniowski

Ed Wisniowski is a software development veteran. He specializes in improving organization product ownership, helping developers become better artisans, and attempting to scale agile in organizations.
Sugar Grove, IL