3 ms·

"A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable." -- Leslie Lamport. I feel t

by m0zg 8y ago

"A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable." -- Leslie Lamport. I feel this should be rephrased for cloud computing at this point. The more people rely on cloud, the more these global fuck-ups are going affect them. Makes me feel pretty good about that server rack in my garage that addresses most of my own (and my business') compute needs.

bdibs 8y ago
Because nothing could happen to your rack (or garage)?
- mrmondo 8y ago
  I can’t say for sure, but I’m assuming they meant because it is (in comparison) not a distributed (or complex) system.
- m0zg 8y ago
  Bad things can happen to it, but at least nobody futzes with it all the time. Most of e.g. Google's major outages (Cloud and Borg) are configuration error or human error. I run my own systems very conservatively. I use conservative technology choices, my kernels are years behind bleeding edge (although I do install security updates, of course), and I don't try to fix or "improve" what's not broken because my paycheck does not depend on that. And in the unlikely event something does get fucked up, I can fix it within less than an hour and I'm the only one affected. It's also extremely unlikely to have any kind of "global" outage. Don't get me wrong, I use cloud (GCP, if you must know) too, and if my business grew massively, I'd probably use it more. But frankly I'm more satisfied with my own "on prem" solution. Single rack which basically pays for itself every 3 months or so in cloud costs, what's not to like?
  - joncrane 8y ago
    What if you're on vacation when something happens, such as a power outage? What about a massive event like an earthquake/flood/wind storm/fire/etc?
    - m0zg 8y ago
      Far less likely events than an SRE pushing a bad config.
  - jds375 8y ago
    I totally understand what you’re saying, but at what point does it become “putting money under your mattress” versus using a bank? Yes a major outage that’s no fault of my own could take down my site - and that isn’t great. On the other hand, wouldn’t you trust them more than yourself? Just like you trust a bank more than yourself to keep your money safe? I’m sure the average major cloud service has far better uptime than the average on prem solution. At what point does it become hubris to spin up your own on prem solution?
    - m0zg 8y ago
      >> wouldn’t you trust them more than yourself Nope. >> versus using a bank If my bank charged 25% of my balance every month, I'd very quickly choose the mattress. :-) But I do recognize that my needs are pretty uncommon.
userbinator 8y ago
Another perhaps more relevant way to think about this is that security and availability are at odds with each other, and in this case a system designed for security made a secure choice. The more secure a system is designed to be, the more likely it is to treat unusual conditions as an attack and possibly perform some destructive action to thwart the assumed attacker. Think of phones configured to delete all data after X incorrect password attempts, HSMs with anti-tamper switches, etc.
- danillonunes 8y ago
  I don’t quite get what security is being accomplished in this case. You can’t access the key store, so you nuke the encrypted database?
outworlder 8y ago
If you already have a rack available with spare capacity, and your business is not expected to blow up overnight, and can tolerate failures(and the turnaround time for your or your employees to fix stuff, order spare parts, etc), sure, why not? The capacity is there, might as well use it. That said, if you didn't have said rack, I'm not so sure it would be worth it to even make a purchase order. Sure things outside of your control may break when you are using a cloud. But guess what, things outside your control will also break, on-prem. Particularly hardware, and network connectivity. There is no way your networking can be better than, say, GCP's own networking, or that you can deploy redundant workloads across availability zones (or even regions!) yourself. By the time a purchase order for a new server can arrive, we can have a production-ready system running, with redundancy across availability zones, automatic failover, CDNs, backups, the works. Basically, I don't care if someone knocks out power in my block, if someone cuts a network cable, or even if a machine goes up in flames. One thing I would say is: even if you are very happy with your current setup, if you have some time to automate a similar setup on the cloud (keyword: automate), then I would suggest doing just that, and offload backups to the cloud too. Even if only as a business continuity thing.
- m0zg 8y ago
  My networking is good enough for my needs: 10GbE copper. Not 100GbE, but 100GbE is hard to actually fully utilize without jumping through major hoops. My business is mainly deep learning R&D. Current cloud GPU, networking, and storage pricing gives me ulcers given my compute needs and the size of my datasets. I do run my website in cloud, with redundancy and all that. I also use cloud storage for backup, and for K8S registry. If I was selling e.g. inference services, I'd be running them in cloud too (passing the costs onto the clients). Most of my local workloads could easily be shifted right across to any decent K8S provider. But the fact is, my lone rack has been humming along with zero unscheduled downtime for 3 years now. I can count several global outages in each of the three major cloud services in this timeframe (most of them during US work hours, BTW), so I'm inarguably better off with the setup I have now than I would be if I moved it all to the cloud. Not to mention it already paid for itself several times over even though I burn through several hundred dollars a month in electricity.
boulos 8y ago
Disclosure: I work on Google Cloud. I’ve always enjoyed this quote, but my problem with [the description of] this outage is the third-party dependency. Packets can’t get from your cloud provider to downstream users of CenturyLink? That’s fair. Your cloud provider can’t send packets to/from CenturyLink, so they nuke your database? I literally don’t understand. Is the service described actually a third-party service that’s been white boxed? (I mean this in the most honest way possible. I do not understand the details, and I found the article surprising).
- m0zg 8y ago
  Yes, sounds like bad design to me. But the uber-problem here is with complexity. Google had its own share of screwups (most of which users never saw, but some of which they did), albeit not quite to the extent of destroying data at scale (and when GMail did destroy data at scale, it had tape backups, so it was never actually lost). The root cause of nearly all of these screwups is that large, complex systems can't be fully understood or observed, and that a good chunk of knowledge about such systems is institutional, rather than explicit. So from time to time people _will_ make assumptions that don't match reality, and reality will punish them for it. Which is what, I strongly suspect, happened in this case.
Waterluvian 8y ago
I feel that while valid for you, this sentiment is highly out of touch with the reality of the needs, resources, and capabilities of most people who need these kinds of systems. It reminds me of a friend who wonders why his parents don't just install Ubuntu because windows is so awful.

by m0zg 8y ago

bdibs 8y ago

Because nothing could happen to your rack (or garage)?

mrmondo 8y ago
I can’t say for sure, but I’m assuming they meant because it is (in comparison) not a distributed (or complex) system.
m0zg 8y ago
Bad things can happen to it, but at least nobody futzes with it all the time. Most of e.g. Google's major outages (Cloud and Borg) are configuration error or human error. I run my own systems very conservatively. I use conservative technology choices, my kernels are years behind bleeding edge (although I do install security updates, of course), and I don't try to fix or "improve" what's not broken because my paycheck does not depend on that. And in the unlikely event something does get fucked up, I can fix it within less than an hour and I'm the only one affected. It's also extremely unlikely to have any kind of "global" outage. Don't get me wrong, I use cloud (GCP, if you must know) too, and if my business grew massively, I'd probably use it more. But frankly I'm more satisfied with my own "on prem" solution. Single rack which basically pays for itself every 3 months or so in cloud costs, what's not to like?
- joncrane 8y ago
  What if you're on vacation when something happens, such as a power outage? What about a massive event like an earthquake/flood/wind storm/fire/etc?
  - m0zg 8y ago
    Far less likely events than an SRE pushing a bad config.
- jds375 8y ago
  I totally understand what you’re saying, but at what point does it become “putting money under your mattress” versus using a bank? Yes a major outage that’s no fault of my own could take down my site - and that isn’t great. On the other hand, wouldn’t you trust them more than yourself? Just like you trust a bank more than yourself to keep your money safe? I’m sure the average major cloud service has far better uptime than the average on prem solution. At what point does it become hubris to spin up your own on prem solution?
  - m0zg 8y ago
    >> wouldn’t you trust them more than yourself Nope. >> versus using a bank If my bank charged 25% of my balance every month, I'd very quickly choose the mattress. :-) But I do recognize that my needs are pretty uncommon.

userbinator 8y ago

Another perhaps more relevant way to think about this is that security and availability are at odds with each other, and in this case a system designed for security made a secure choice. The more secure a system is designed to be, the more likely it is to treat unusual conditions as an attack and possibly perform some destructive action to thwart the assumed attacker. Think of phones configured to delete all data after X incorrect password attempts, HSMs with anti-tamper switches, etc.

danillonunes 8y ago
I don’t quite get what security is being accomplished in this case. You can’t access the key store, so you nuke the encrypted database?

outworlder 8y ago

If you already have a rack available with spare capacity, and your business is not expected to blow up overnight, and can tolerate failures(and the turnaround time for your or your employees to fix stuff, order spare parts, etc), sure, why not? The capacity is there, might as well use it. That said, if you didn't have said rack, I'm not so sure it would be worth it to even make a purchase order. Sure things outside of your control may break when you are using a cloud. But guess what, things outside your control will also break, on-prem. Particularly hardware, and network connectivity. There is no way your networking can be better than, say, GCP's own networking, or that you can deploy redundant workloads across availability zones (or even regions!) yourself. By the time a purchase order for a new server can arrive, we can have a production-ready system running, with redundancy across availability zones, automatic failover, CDNs, backups, the works. Basically, I don't care if someone knocks out power in my block, if someone cuts a network cable, or even if a machine goes up in flames. One thing I would say is: even if you are very happy with your current setup, if you have some time to automate a similar setup on the cloud (keyword: automate), then I would suggest doing just that, and offload backups to the cloud too. Even if only as a business continuity thing.

m0zg 8y ago
My networking is good enough for my needs: 10GbE copper. Not 100GbE, but 100GbE is hard to actually fully utilize without jumping through major hoops. My business is mainly deep learning R&D. Current cloud GPU, networking, and storage pricing gives me ulcers given my compute needs and the size of my datasets. I do run my website in cloud, with redundancy and all that. I also use cloud storage for backup, and for K8S registry. If I was selling e.g. inference services, I'd be running them in cloud too (passing the costs onto the clients). Most of my local workloads could easily be shifted right across to any decent K8S provider. But the fact is, my lone rack has been humming along with zero unscheduled downtime for 3 years now. I can count several global outages in each of the three major cloud services in this timeframe (most of them during US work hours, BTW), so I'm inarguably better off with the setup I have now than I would be if I moved it all to the cloud. Not to mention it already paid for itself several times over even though I burn through several hundred dollars a month in electricity.

boulos 8y ago

Disclosure: I work on Google Cloud. I’ve always enjoyed this quote, but my problem with [the description of] this outage is the third-party dependency. Packets can’t get from your cloud provider to downstream users of CenturyLink? That’s fair. Your cloud provider can’t send packets to/from CenturyLink, so they nuke your database? I literally don’t understand. Is the service described actually a third-party service that’s been white boxed? (I mean this in the most honest way possible. I do not understand the details, and I found the article surprising).

m0zg 8y ago
Yes, sounds like bad design to me. But the uber-problem here is with complexity. Google had its own share of screwups (most of which users never saw, but some of which they did), albeit not quite to the extent of destroying data at scale (and when GMail did destroy data at scale, it had tape backups, so it was never actually lost). The root cause of nearly all of these screwups is that large, complex systems can't be fully understood or observed, and that a good chunk of knowledge about such systems is institutional, rather than explicit. So from time to time people _will_ make assumptions that don't match reality, and reality will punish them for it. Which is what, I strongly suspect, happened in this case.

Waterluvian 8y ago

I feel that while valid for you, this sentiment is highly out of touch with the reality of the needs, resources, and capabilities of most people who need these kinds of systems. It reminds me of a friend who wonders why his parents don't just install Ubuntu because windows is so awful.