5 ms·

Very valid points. Its hard to describe the complexities and weird error or latency or other conditions that happen in cloud environments. OPs people kinda shru

Very valid points. Its hard to describe the complexities and weird error or latency or other conditions that happen in cloud environments. OPs people kinda shrug and ask that you essentially turn it off and back on again and that usually resolves the issue at that time. There are so many abstraction layers when all the infrastructure is software run and I bet a lot of those introduce performance degradation. Another element of this is I would think its more fun for a systems person to be closer to the hardware. Specifying, ordering and unboxing servers, disks, switches and setting things up sounds satisfying to me.

senttoschool 4y ago
>Its hard to describe the complexities and weird error or latency or other conditions that happen in cloud environments. As oppose to complexities of self-host using self purchased hardware?
- Nextgrid 4y ago
  They are much less frequent, and at the very least you have full access to the hardware to debug if needed. Can you get me access to the AWS control plane's logs so I can understand why my RDS instance is stuck in "modifying" for 12+ hours?
  - senttoschool 4y ago
    AWS has over a decade of experience and a ton of full-time dev ops to make complicated cloud usage easy. You're not going to beat the reliability by going inhouse.
    - BackBlast 4y ago
      If something goes wrong at AWS, and it does. You have no power over recovery. The more of the stack you own, the more such ability you do have. If you run a competent organization, this can be a boon. If you have serious deficiencies, it may well be the opposite.
    - Nextgrid 4y ago
      I'm going to beat its reliability (and successfully do so) because the complexity of my stack is less than 1% of Amazon's. I have way less moving parts (and complex interactions between them) to worry about than Amazon. My only liability is hardware failure, and it turns out that stuff has gotten pretty reliable nowadays. Am I able to beat Amazon on their wide range of services? Absolutely not - if you're looking for FAANG-level scale, you're talking to the wrong person. But for the handful of services I do need (HTTP LB, app servers, relational database and cache), I beat them because I only need to deliver very specific services and can schedule the risky operations (maintenance, hardware upgrades, etc) around my business' needs, something AWS can't do because as a global provider any time they pick for maintenance would inevitably clash with someone's business-critical hours. Turns out most companies also only require a small chunk of what AWS offers, so the rest of the AWS control plane's complexity and its global scale is actually a major liability. > AWS has over a decade of experience and a ton of full-time dev ops to make complicated cloud usage easy. And yet that one time a client's RDS instance ran out of disk space, it got stuck in "modifying" for 12+ hours (I assume an AWS operator eventually manually logged in and fixed it) with no way for us to recover, forcing us to restore a backup and reconstruct missing data from logs to bring our service back online in a timely manner. Would it fail the same way on bare-metal? Yes, but at least I can SSH in and fix it immediately. Not to mention, the only reason it ran out of space is because AWS' pricing forced us to be very conservative with storage, where as the bare-metal machines I rent - for far cheaper than AWS' offerings - come with terabytes of storage as standard so this incident would've never happened to begin with.
      - truetraveller 4y ago
        This is a fantastic rebuttal to those who claim "AWS has way more resources/experience, thus they will be more reliable".
        CoolCold 4y ago
        well, for someone, it's definitely is - I can easily imagine someone who use things like https://news.ycombinator.com/item?id=33392886 https://news.ycombinator.com/item?id=33392886 > The thing that Soul or other similar tools follow is to automatically turn your database into you backend. You don't need to implement any endpoints, Soul already did it for you. may have no clue anyway how to be reliable - which is totally fine of course.
    - jjav 4y ago
      > You're not going to beat the reliability by going inhouse. OP was talking about observability and diagnosability. These will be infinitely better/easier with access to the hardware & OS. But on reliability, if you keep your stack simple and boring, you can pretty easily also beat them on reliability. Complexity is the top enemy of reliability (and security).
      - senttoschool 4y ago
        Postgres is a simple stack. On AWS, you can have a Postgres instance with multi-region availability. How do you plan to do this with your own hardware?
wazoox 4y ago
Ah yeah, I remember when a friend's company struggled for weeks with some terrible slowdowns on their Azure-based cloud infrastructure. After hours on hours of support calls and emails and weeks of hair-pulling, everything suddenly went better : after careful interviews with Azure support team, they understood that their VMs were running on hosts in two different racks, and that the link between the racks had just been upgraded from gigabit to 10-gig when the ToR switches were replaced.

senttoschool 4y ago

>Its hard to describe the complexities and weird error or latency or other conditions that happen in cloud environments. As oppose to complexities of self-host using self purchased hardware?

Nextgrid 4y ago
They are much less frequent, and at the very least you have full access to the hardware to debug if needed. Can you get me access to the AWS control plane's logs so I can understand why my RDS instance is stuck in "modifying" for 12+ hours?
- senttoschool 4y ago
  AWS has over a decade of experience and a ton of full-time dev ops to make complicated cloud usage easy. You're not going to beat the reliability by going inhouse.
  - BackBlast 4y ago
    If something goes wrong at AWS, and it does. You have no power over recovery. The more of the stack you own, the more such ability you do have. If you run a competent organization, this can be a boon. If you have serious deficiencies, it may well be the opposite.
  - Nextgrid 4y ago
    I'm going to beat its reliability (and successfully do so) because the complexity of my stack is less than 1% of Amazon's. I have way less moving parts (and complex interactions between them) to worry about than Amazon. My only liability is hardware failure, and it turns out that stuff has gotten pretty reliable nowadays. Am I able to beat Amazon on their wide range of services? Absolutely not - if you're looking for FAANG-level scale, you're talking to the wrong person. But for the handful of services I do need (HTTP LB, app servers, relational database and cache), I beat them because I only need to deliver very specific services and can schedule the risky operations (maintenance, hardware upgrades, etc) around my business' needs, something AWS can't do because as a global provider any time they pick for maintenance would inevitably clash with someone's business-critical hours. Turns out most companies also only require a small chunk of what AWS offers, so the rest of the AWS control plane's complexity and its global scale is actually a major liability. > AWS has over a decade of experience and a ton of full-time dev ops to make complicated cloud usage easy. And yet that one time a client's RDS instance ran out of disk space, it got stuck in "modifying" for 12+ hours (I assume an AWS operator eventually manually logged in and fixed it) with no way for us to recover, forcing us to restore a backup and reconstruct missing data from logs to bring our service back online in a timely manner. Would it fail the same way on bare-metal? Yes, but at least I can SSH in and fix it immediately. Not to mention, the only reason it ran out of space is because AWS' pricing forced us to be very conservative with storage, where as the bare-metal machines I rent - for far cheaper than AWS' offerings - come with terabytes of storage as standard so this incident would've never happened to begin with.
    - truetraveller 4y ago
      This is a fantastic rebuttal to those who claim "AWS has way more resources/experience, thus they will be more reliable".
      - CoolCold 4y ago
        well, for someone, it's definitely is - I can easily imagine someone who use things like https://news.ycombinator.com/item?id=33392886 https://news.ycombinator.com/item?id=33392886 > The thing that Soul or other similar tools follow is to automatically turn your database into you backend. You don't need to implement any endpoints, Soul already did it for you. may have no clue anyway how to be reliable - which is totally fine of course.
  - jjav 4y ago
    > You're not going to beat the reliability by going inhouse. OP was talking about observability and diagnosability. These will be infinitely better/easier with access to the hardware & OS. But on reliability, if you keep your stack simple and boring, you can pretty easily also beat them on reliability. Complexity is the top enemy of reliability (and security).
    - senttoschool 4y ago
      Postgres is a simple stack. On AWS, you can have a Postgres instance with multi-region availability. How do you plan to do this with your own hardware?

wazoox 4y ago

Ah yeah, I remember when a friend's company struggled for weeks with some terrible slowdowns on their Azure-based cloud infrastructure. After hours on hours of support calls and emails and weeks of hair-pulling, everything suddenly went better : after careful interviews with Azure support team, they understood that their VMs were running on hosts in two different racks, and that the link between the racks had just been upgraded from gigabit to 10-gig when the ToR switches were replaced.