November's outage cluster had no attacker: NZ change checks

By DataForge team23 November 20265 min read

US officials say the outages that hit Microsoft, AWS, banks and airlines in mid-November came from maintenance and changes. Here's what NZ IT teams should check.

What happened

Between 12 and 21 November, a run of large service failures hit organisations that NZ businesses use every day, or that work much like the ones they use. A running outage timeline puts the Microsoft 365 failure on 12 and 13 November at about 24 hours. Workday followed on 13 and 14 November, after a backup site connected to live databases during a test. The same timeline has AWS us-east-1 losing 11 services for about three hours on 15 November, with no public post-event summary yet. JPMorgan Chase blamed a software bug in on-premise hardware for its banking outage on 16 November, and Apple said "an internal update" caused its iCloud sign-in and App Store problems on 17 November. The NSA's classified network was down for almost four days, which the agency put down to scheduled maintenance. Ticketmaster went down in the US, Mexico and Australia on the evening of 20 November.

The timing led to talk of a coordinated attack. On 18 November, the US Director of National Intelligence, the Homeland Security Secretary and the FBI Deputy Director told a House committee they'd found nothing linking the incidents. Deseret News reported that Jay Clayton described them as "separate service outages associated with maintenance and technical changes" and said each one had a different cause. The timeline's summary of the hearing notes the testimony came before the PlayStation Network and Ticketmaster failures, so those two weren't part of the officials' assessment.

Why the cause matters more than the coincidence

Cyberattacks are what boards ask about after a week like this one. The causes on record point somewhere less dramatic. A gateway defect introduced months earlier, a disaster recovery test that touched production, a new router during a network upgrade and an internal software update are all change and configuration problems. Your own ITIL 4 change practice exists to catch failures of exactly this kind.

The pattern isn't new. In an interview IEEE published on 19 November, Robert Charette, who's documented system failures for about two decades, argues that clusters like November's are unusual in timing but ordinary in kind. He puts the root causes down to system complexity, poorly understood interfaces between systems, weak configuration management and cost decisions that leave safeguards out. BigPanda's weekly brief cites Control Plane's tracker of 62 major outages at AWS, Azure, Google Cloud and Cloudflare since 2020, and configuration or deployment changes caused 32% of them.

None of these incidents started in New Zealand. Even so, NZ organisations ran into the effects, whether through Microsoft 365 or Workday or through staff who couldn't sign in to Apple services. A vendor's change becomes your incident, and your change calendar won't show it.

What NZ businesses should take from it

The first lesson is about telling causes apart under pressure. When several suppliers fail in the same week, a service desk can easily assume one cause and either escalate everything to security or wave everything off as a vendor problem. The US officials took several days to say publicly that the incidents weren't linked. Your major incident process should treat attribution as an open question, with security involved early, until there's evidence either way. A failed sign-in service and a credential attack can look the same for the first hour.

The second lesson concerns communication with your own people when the supplier is slow. Fortune spoke to several CIOs after the Microsoft outage. Lockton's CIO said Microsoft was slow to warn customers about the possible global impact. Ropes & Gray had a secondary email system and an emergency staff notification tool already running when Outlook was down for about 18 hours. Pendo's CISO suggested that if a fix is estimated at two hours, you tell stakeholders four, because recovery usually runs longer than the first estimate. That advice is easy to adopt and costs nothing.

The third lesson is that a supplier's test can take you down. Workday's failure came during a test of its backup site. Inland Revenue is running planned disaster recovery testing with myIR, Gateway Services and its self-service phone line unavailable from 3pm Saturday 28 November to 9am Sunday 29 November, after a first window the previous weekend. IRD has published those windows weeks ahead, and most suppliers don't. Few NZ firms ask their SaaS providers when they run failover tests, or whether those tests can touch production data.

The last lesson is that redundancy only counts if it has been tested against the failure you're likely to face. Charette's view in the same IEEE interview is that organisations know how to protect their systems and often don't, because the safeguard costs money and the outage hasn't happened yet. For a mid-sized NZ business, the affordable answer is usually a short list of critical services, each with a documented fallback that someone has rehearsed.

Practical next steps

  • Map supplier changes into your change view. For your top ten services, record where each vendor publishes planned maintenance and change notices, and have someone check those sources during your weekly change review. Microsoft 365 message centre, AWS Health and your payroll or ERP vendor's status page are the usual starting points.
  • Add an attribution step to major incidents. When more than one external service fails within a short window, the incident manager should bring in security to confirm or rule out a common cause before anyone tells staff or customers what happened.
  • Pre-write your outage messages. Draft staff and customer templates for email, collaboration and identity outages, and send them through a channel that doesn't depend on Microsoft 365 or Google Workspace. Set a rule for padding recovery estimates.
  • Ask suppliers about their testing. Add questions to your next contract review about how often the vendor tests failover, whether tests can affect live data, and how much notice you get. Workday's incident gives you a concrete example to point to.
  • Rehearse one fallback this quarter. Pick the service that would hurt most if it went down, such as email, payroll or your ERP, and run a tabletop exercise in which it's gone for a full working day. Write down what broke in the exercise and fix the gaps before the summer shutdown.
  • Track your own change failure rate. If you can't say what share of your incidents follow a change, start tagging incident tickets with any linked change record. Automating that link in your ITSM tool is a small process automation job with an early payoff.

Microsoft and AWS haven't published full post-incident reports for their November failures yet. When those reports arrive, read them for the change that triggered each failure and the control that should have caught it, then check whether you have the same gap.

Want to know what this means for your organisation? Our team in Auckland can show you your top three opportunities in a free 30 minute assessment.

Get my free assessment How we help
Get new articles by RSS