AEMO's datacentre move hit forecasts its change notice didn't list

A routine change with a long tail

On 10 November the Australian Energy Market Operator moved its Electricity Market Management System (EMMS) production systems to its alternative datacentre. The move went through formal change control. AEMO published the change, CHG0114238, as a market notice on 30 October and reissued it on 2 November. The notice set a change window from 08:00 on 9 November to 12:00 on 11 November and a one-hour outage from 18:00 to 19:00 AEST on 10 November. AEMO classed the work as a Normal change. The notice listed three expected effects: delayed delivery of FPP calculations through the participant data model, an intermittently unavailable Market Portal, and, in the reissued version, an intermittently unavailable Gas Supply Hub.

AEMO marked the change complete in notices 146239 and 146240, issued late on 10 November and on the morning of 11 November. Neither completion notice mentions any problem.

Energy analysts at WattClarity saw something else in the dispatch data. Linton Corbet's analysis, published on 11 November, shows that at 18:25 AEST the share of wind and solar units dispatched on their own self-forecasts dropped from about 54% of the fleet to 7%. From 18:30, for three five-minute dispatch intervals, neither self-forecasts nor AEMO's own AWEFS and ASEFS forecasting systems were in use. Most units fell back to SCADA readings, so each unit's output at the start of an interval became its forecast for the end of that interval. Self-forecasts mostly recovered by 18:50. AEMO's own wind and solar forecasts didn't return for units without self-forecasts until 22:25, about four hours after the trouble began and more than three hours after the outage window closed.

Corbet is careful about cause. He writes that the datacentre transfer "appears to have coincided" with the disruption, so the connection rests on timing. AEMO hasn't published an explanation at the time of writing.

Why a forecasting glitch matters to people who don't run power grids

AEMO describes forecasts of intermittent generation as critical to power system security and an efficient market. For roughly four hours, a large part of the National Electricity Market's wind and solar fleet was dispatched on a cruder method than usual. The impact statement participants had read beforehand covered portal access, gas hub availability and a data delay, and it described one hour of disruption.

Most NZ organisations will see their own change process in that gap. Change records describe what the implementer expects to touch. Downstream consumers of the system notice everything else, often hours later and through their own monitoring. AEMO's process has more structure than most: a numbered change, advance notice to the market, a defined outage window and completion notices. Even so, nothing in the published record would have told a wind farm operator to watch its forecast source that evening.

Datacentre and site transfers carry this risk because a failover changes more than the application being moved. Scheduled jobs, message feeds, integration endpoints, certificates, DNS records and database replication all have to land cleanly on the other site. A feed that reconnects late or starts in a degraded mode won't necessarily raise an alarm, because the main system looks healthy. Once the outage window closes, the change team declares success and the gap stays open until someone downstream checks.

The NZ angle this month

NZ businesses will be on the receiving end of a similar exercise shortly. Inland Revenue has scheduled disaster recovery testing over two weekends, 21 to 22 November and 28 to 29 November, from 3pm Saturday to 9am Sunday. myIR, Gateway Services and the self-service phone line will be affected. Gateway Services is the channel that payroll and accounting software uses to file with IRD, so any organisation running weekend payroll or automated filing jobs depends on it, often without anyone on the finance team knowing it exists.

IRD's notice tells customers which services go down and when. It can't tell a business which of its own scheduled jobs will fail, retry, queue or quietly drop during those hours, or whether they'll recover once IRD's systems return. Only the business's own IT team or software provider can answer those questions, and the AEMO case shows the answers can differ from the window printed on the notice.

The same applies to internal changes. A mid-sized NZ firm moving its ERP to a new hosting provider, failing over a SQL cluster or swapping its firewall will usually write an impact statement from the implementer's side. Teams that consume the affected data (finance reporting, warehouse integrations, customer portals, overnight batch runs) usually aren't asked what they depend on until something breaks.

What to change in your own process

AEMO's records show a sound process at the notification stage. The weak points sit in impact analysis and in the check after the change. Both are fixable without new tooling, and both fit inside ITIL 4's change enablement and service validation practices, which we cover in our ITIL 4 framework services.

• Map consumers along with components. For any change to a shared system, list the downstream jobs, feeds and integrations that read from it, along with their owners. A configuration management database helps, but a spreadsheet maintained by service owners works for smaller firms.
• Ask consumers to review the impact statement. Send the draft to the owners of dependent services before approval and ask what they'd expect to see if their feed broke. Their answers become your post-change checks.
• Define completion as verified recovery. Close a change only after named downstream checks pass, such as a scheduled job completing, a feed timestamp advancing or an integration returning current data. Getting the main system back up doesn't count as completion on its own.
• Watch past the outage window. Keep monitoring dependent services for several hours after a site transfer or failover, and assign someone to do it. In the AEMO case the AWEFS and ASEFS feeds came back more than three hours after the window closed.
• Prepare for IRD's test weekends now. Ask your payroll and accounting software providers how their products handle Gateway Services being unavailable between 3pm Saturday and 9am Sunday on 21 to 22 and 28 to 29 November. Move any filing jobs scheduled in those windows, and check on the Monday that queued submissions went through.
• Review your own DR tests the same way. If your organisation runs failover tests, record what broke downstream as well as whether the primary system came up. Over time, those findings make the next impact statement accurate.

AEMO may yet show that the forecasting disruption had a different cause. The lesson for change managers holds either way: the people who depend on a system notice impacts the change record never listed, and a change process that doesn't ask those people beforehand will keep finding problems after the change has closed.