event:
Recently, the Google Cloud 1 service suffered a major and unprecedented outage that led to the accidental deletion of a private cloud account for financial services provider UniSuper, sparking widespread concern and discussion. Below is a detailed review of the incident.
The mega-fund boss and Google Cloud's global CEO issued a joint statement apologizing to affected users for the "extremely frustrating and disappointing" outage.
More than half a million members of UniSuper Funds were unable to access their respective pension accounts for a whole week. Services are starting to be restored one by one after more than a week of system downtime, and investment account balances will show last week's amount data.
UniSuper CEO Peter Chun assured members that the outage was not the result of a cyber attack and that no personal data was compromised during the outage. The cause of the outage was an accidental misconfiguration of Google Cloud during the provisioning of UniSuper's private cloud service, which resulted in the deletion of UniSuper's private cloud subscription account.
Google Cloud CEO Thomas Kurian confirmed that the outage was caused by an unprecedented chain of events. Google Cloud has identified the events that led to the outage and has taken steps to ensure that a similar incident does not occur again.
Although UniSuper typically has copies in both regions to ensure service reliability, both regions' cloud instances were deleted due to the deletion of the cloud subscription account. Ultimately, UniSuper was able to restore service thanks to a backup from another provider. This incident highlights the importance of data backup in cloud services.
UniSuper worked collaboratively with the Google Cloud team to fully restore all core systems, including hundreds of virtual machines, databases and applications. This collaboration demonstrates the critical importance of close collaboration between cloud service providers and their customers in the face of a major outage.
With about $125 billion under management under UniSuper, this incident is certainly a tough test for a money management company of this magnitude.
Opinion:
If the AliCloud global service unavailability outage was "epic," then the Google Cloud outage was "unparalleled. While the former was mainly about service availability, this outage hit the very root of many organizations' lives -- data integrity.
As far as I know this should be a new record in the history of cloud computing -- the first time a database has been deleted on this scale. The last time there was a similar data integrity breach was between Tencent and Foreword NC.
But a small startup is completely indistinguishable from a large fund in charge of hundreds of billions of dollars; the scope and scale of the impact is completely indistinguishable -- everything under the entire cloud account is gone!
This incident once again demonstrated the importance of (off-site, multi-cloud, different vendors) backups -- UniSuper was lucky they had other backups.
But if you believe that a public cloud vendor's data backups in other regions/availability zones can help you "cover your bases", then remember this case - avoid Vendor Lock-in and Always has Plan B.
refer to:
The Guardian UK report on the incident:
https://www.unisuper.com.au/about-us/media-centre/2024/a-joint-statement-from-unisuper-and-google-cloud
UniSuper CEO and Google Cloud CEO Joint Statement:
https://www.unisuper.com.au/about-us/media-centre/2024/a-joint-statement-from-unisuper-and-google-cloud
Original article by Chief Security Officer, if reproduced, please credit https://cncso.com/en/google-cloud-configuration-error-causes-user-data-loss.html
