According to Ars Technica UniSuper, an Aussie pension fund that's got a hefty $135 billion in funds and a membership of 647,000, found its account at Google Cloud completely erased, backups and all that were stashed on the service.

Thankfully, UniSuper had the foresight to keep some backups elsewhere and managed to get its data back. But, as per UniSuper's own incident diary, things went dark on 2 May, and it wasn't until May 15 that they got everything up and running again.

UniSuper's website is chock-a-block with the kind of admin horror stories you've got to read to believe.

On 8 May, there was this joint announcement from UniSuper's head honcho Peter Chun, and Google Cloud's big boss, Thomas Kurian, which went like this:

“Google Cloud CEO, Thomas Kurian has confirmed that the disruption arose from an unprecedented sequence of events whereby an inadvertent misconfiguration during provisioning of UniSuper’s Private Cloud services ultimately resulted in the deletion of UniSuper’s Private Cloud subscription. This is an isolated, ‘one-of-a-kind occurrence’ that has never before occurred with any of Google Cloud’s clients globally. This should not have happened. Google Cloud has identified the events that led to this disruption and taken measures to ensure this does not happen again."

But the next bit showed just how dire the situation was and why it was taking an age to sort out.

"UniSuper had duplication in two geographies as a protection against outages and loss. However, when the deletion of UniSuper’s Private Cloud subscription occurred, it caused deletion across both of these geographies."

Every cloud service keeps full backups, which you'd think are there for the absolute worst. But it turns out the real worst-case scenario is "Google bins your account," and poof, all those backups vanish too.

Google Cloud is meant to have fail-safes that stop account deletion, but they all went on holiday, and the only way out was a restore from a different cloud provider.

UniSuper is what the Aussies call a "superannuation fund." It's a retirement pot that employers chip into from an employee's wages; down under, the law says a bit of super payment is a must for everyone with a job.

A press release in June 2023 was all about UniSuper's grand move to Google Cloud, with Sam Cooper, UniSuper's Head of Architecture, saying, “With Google Cloud VMware Engine, migrating to the cloud is streamlined and extremely easy. It’s all about efficiencies that help us deliver highly competitive fees for our members."

The whole palaver wasn't just about getting backups back online but also sorting all the requests and payments that were piling up during the fortnight the lights were out.

On 3 May, Google Cloud came out with UniSuper in a joint statement, making it clear the blackout wasn't down to hackers. The UniSuper CEO reassured everyone that "member accounts are safe," "no data was exposed to unauthorised third parties," and "pension payments have not been disrupted."

The CEO's update added:  "While a full root cause analysis is ongoing, Google Cloud has confirmed this is an isolated one-of-a-kind issue that has not previously arisen elsewhere. Google Cloud has confirmed that they are taking measures to ensure this issue does not happen again." Chun also mentioned UniSuper had a backup cloud provider and they'd be working hard to "minimise" any data loss.

On 7 May,  the CEO chipped in, "Google Cloud has issued a statement today which confirms again that the fault originated within their service as a ‘one of its kind,’ unprecedented occurrence" and that "Google Cloud sincerely apologizes for the inconvenience this has caused."

A week after everything went dark, on 9 May, UniSuper showed signs of life. Logins for "online UniSuper accounts" (pretty sure that's just the website) were back, but the outage notice warned that "account balances shown may not reflect transactions which have not yet been processed due to the outage."

An earlier update had marked "29 April " as the date they'd roll back to for balances. Over the next week, they gradually got various bits of the website and app back to normal. May 13 saw the first hint of the mobile app coming back to life, with a note that balances were still catching up and

"We are processing transactions as quickly as we can." The final update, on May 15, confirmed, "UniSuper can confirm that all member-facing services have been fully restored, with our retirement calculators now available again."