5 ms·
One thing that has irked me about everyone's flippant comments about moving to the cloud is that the "devops as a recovery mechanism" generally only works for s
by jiggawatts 5y ago
One thing that has irked me about everyone's flippant comments about moving to the cloud is that the "devops as a recovery mechanism" generally only works for single-app startups or small shops with only a few dozen simple VMs at most.
Some of my customers have thousands of VMs in their cloud, and they aren't cloned cattle! They're pets. Thousands upon thousands of named pets. Each with their individual, special recovery requirements. This then has a nice thick crust of PaaS and SaaS layered on top in a tangle of interdependencies that no human can unravel.
Some resources were built using ARM templates. Some with PowerShell scripts. Some with Terraform. A handful with Bicep. Most with click-ops. These are kept in any one of a dozen source control systems, and deployed mostly manually by some consultant that has quit his consulting company and can't be reached.
Most cloud vendors "solve" this by providing snapshots of virtual machines as a backup product.
Congratulations big vendors! We can now recover exactly one type of resource out out of hundreds of IaaS, PaaS, and SaaS offerings. Well done.
For everything else:
WARNING: THIS ACTION IS IRREVERSIBLE!
Fantastic. No worries though, I can... err... export the definition, right? Wrong. That doesn't work for something like 50% of all resource types. Even if it "works", good luck restoring inter-dependent resources in the right order.
Okay, you got things restored! Good job! Except now your DNS Zones have picked a different random pool of name servers and are inaccessible for days. Your PaaS systems are now on different random IP addresses and noone can access them because legacy firewall systems don't like the cloud. All your managed identities have reset their GUIDs and lost their role assignments. The dynamically assigned NIC IP addresses have been scrambled. Your certificates have evaporated.
"But, but, the cloud is redundant! And replicated!" you're just itching to say.
Repeat after me:
A synchronous replica is not a backup.
A synchronous replica is not a backup.
A synchronous replica is not a backup.
Repeat it.
Do you know what it takes to obliterate a cloud-only business, permanently and irreparably?
Two commands.
I won't repeat them here, because like Voldemort's name it simply invites trouble to speak them out loud.
- Drblessing 5y agoThis is a work of literature.
- 908B64B197 5y agoSome companies just do Cloud so their yearly report contains the word Cloud you know?
- jiggawatts 5y agoMostly they seem to do it to move CapEx into OpEx. I never quite understood why spending more money is better if it comes from a different bucket. I'm sure there's some explanation that only makes sense if you don't look too closely.
- mbreese 5y agoI always assumed it had to do with taxes... CapEx can't be written off in it's entirety, but rather has to be calculated as depreciation over time. OpEx however, is a business expense can be written off completely. I'm not sure how that really benefits an organization with a time horizon that is longer than the time it takes to depreciate a server. You still get to write off the full price of a server, it just takes a few years longer (and it was probably cheaper!). But then again, I'm not in finance...
- korethr 5y agoThis is a post written by a person who's been at this a while, and has spent at least a portion of that time as Cassandra.
- MandieD 5y ago“Your PaaS systems are now on different random IP addresses and noone can access them because legacy firewall systems don't like the cloud.” The whole post should be printed out and pinned to the Kanban/Scrum/whatever board of every infrastructure/DevOps team, but this sentence in particular. This property of Azure (and I imagine every other cloud provider) was one of the nastier fights we had with the guys who run the on-prem firewalls.
- spadros 5y agoYeah, this guy has worked in an enterprise. Reminds me of the experience in big non-tech corporations. Years and years of terribly executed legacy SaaS and PaaS building into a permalayer of crap mostly created by contractors that will be gone in 6 months. The top level management don't understand tech, so they pay for cheaper and cheaper contractors to "maintain" the permalayer of crap. It's a never ending spiral of pain, hiring turnover, and bad code.
- markus_zhang 5y agoI guess the only thing we can do is to pray. In those enterprises probably no one knows everything and if one of them breaks who knows what happens...