3 ms·
Potential GCP SSD Data Corruption Issue
We received this from our Scylla account rep:
Subject: Urgent: Action Required - Identified Issue with GCP SSD Disks Affecting ScyllaDB Clusters
Dear Customer,
We trust this message finds you well. We are reaching out to bring your attention to a critical update
concerning your ScyllaDB servers hosted on the Google Cloud Platform (GCP).
Google has recently identified an issue affecting certain SSD disks that may have implications on the data
reliability of your ScyllaDB clusters on GCP. Recognizing the urgency of the situation, we highly
recommend taking immediate action to minimize any potential disruption to your operations.
To address this promptly, we advise contacting Google Cloud Support for personalized assistance. Follow
these steps to assess and mitigate the impact:
Reach out to Google Support to verify whether any of the cluster nodes you are operating on GCP are
utilizing the affected SSDs.
If Google identifies a susceptible node, our recommendation is to perform scrub and removal of the
corrupted sstables, (contact our support team for details), then bootstrap a new node to replace the
potentially compromised one. Subsequently, decommission the affected node. During this process, any
existing corruption will be identified, and appropriate steps, such as using removenode or deleting
corrupted SSTables, can be taken with our technical support team.
While there is a slim chance that the replacement node may be susceptible, Google has assured us that
the probability is low. Unfortunately, there is no proactive way to determine susceptibility.
Google is actively deploying a fix, although an ETA for completion has not been provided. While the
probability of encountering a problematic node is expected to decrease over time, we advise addressing
this issue proactively.
Already reached out to Google support, but past experience would suggest they will not be helpful. Anyone have any additional information about this issue?