6 ms·
I can't speak to the specific incident. We've been running almost 400 servers (instances and k8s cluster nodes) for over a year on GCP and we've been quite happ
by markbnj 8y ago
I can't speak to the specific incident. We've been running almost 400 servers (instances and k8s cluster nodes) for over a year on GCP and we've been quite happy with the performance and reliability, as well as the support response when we have needed it. I did want to address this comment...
> What if the card holder is on leave and is unreachable for three days? We would have lost everything — years of work — millions of dollars in lost revenue.
You should never be in this position. If this were to happen to us we would be able to create a new project with a different payment instrument, and provision it from the ground up with terraform, puppet and helm scripts. The only thing we would have to fix up manually are some DNS records and we could probably have everything back up in a few hours. Eventually when we have moved all of our services to k8s I would expect to be able to do this even on a different cloud provider if that were necessary.
- deltateam 8y agoI wrote a similar comment, thats what the best practices are for
- jacquesm 8y agoRestarting the service from scratch is one thing, but what about all your data? Some of these services have 100's T's of data hanging of them and if Google would delete that because of some perceived violation of their terms then that is not something you can recover from in a couple of hours, if at all. This is one of the reasons I always implore people to have a backup of their data with another provider or at least under a different account. That protects against all kinds of accidents but also against malice.
- pavs 8y agoBackup is a thing. If your company is making millions of dollars off your business you should have a redundant backup of everything including (especially) your data.
- Endy 8y agoWhy does it matter whether you're making millions of dollars? If you have any information which you would like to not lose for any reason, back it up in as many formats and locations as is feasible.
- pavs 8y agoAgreed. I mentioned the money angle because I felt the person I replied to implied that 100s of terabytes of data are too expensive to backup. If you are making money or if the data is important for you to lose than you should have a backup, anything else is faulty planning.
- jacquesm 8y ago> I felt the person I replied to implied that 100s of terabytes of data are too expensive to backup. Well, you felt wrong. Of course you should back up those 100s of terabytes, in fact that it is that much information is an excellent reason on top of all the other ones to back it up, re-creating it is going to be next to impossible. It's just that the companies I look at - not all, but definitely some - seem to be under the impression that the cloud (or their cloud provider) can be trusted. Which is wrong for many reasons, not just this article.
- pavs 8y agoSorry, I stand corrected.
- jacquesm 8y agoNo problem, it's just that with 'This is one of the reasons I always implore people to have a backup of their data with another provider or at least under a different account.' that passage I thought I had the backup angle more than covered. What bugs me about it is that there are some companies that give serious pushback because their cloud providers keep on hammering in to them how reliable their cloud is and that any back-up will surely be less reliable than their cloud solution and oh by the way we also have a backup feature that you can use. They don't realize that even then they still have all their eggs in the one basket: their cloud account.
- ma2rten 8y agoHuh? You can provision new servers, but you can't just easily move over all the data, can you?