6 ms·

Anecdotal evidence. I've also seen colleagues spend weeks arguing with AWS support while debugging a weird performance degradation issue, that would have been s

by lima 7y ago

Anecdotal evidence. I've also seen colleagues spend weeks arguing with AWS support while debugging a weird performance degradation issue, that would have been straight-forward to investigate in a bare metal deployment with full control over everything. It's not like the cloud is a magical place where no unexpected issues ever happen. Cloud providers can be surprisingly buggy, especially AWS, particularly at scale when you start hitting the implicit scaling limits their docs conveniently forgot to mention. Each technology has their unique challenges.

argd678 7y ago
I’ve had performance issues with GCP, however I can switch cloud providers in that case without too much effort. With physical hardware that’s much harder.
- lima 7y ago
  If you have a well-built and automated bare-metal deployment, it's no harder than switching cloud providers. At the scale where it makes sense to go bare metal, either would be a lot of effort. An extra abstraction layer like k8s makes it a lot easier, which is exciting. It's also precisely why IBM bought Red Hat - a well-built k8s distribution like OpenShift is one of the very few real alternative to public clouds for many companies.
  - windexh8er 7y ago
    > An extra abstraction layer like k8s makes it a lot easier, which is exciting. It's also precisely why IBM bought Red Hat - a well-built k8s distribution like OpenShift is one of the very few real alternative to public clouds for many companies. OpenShift is built on a prescriptive mentality. They choose everything for you from OS through pipeline and many deviations are unsupported. As you mentioned there are others, but depending on your environment OpenShift is either a very good fit or square-peg-round-hole.
    - lima 7y ago
      OpenShift is a superset of k8s, so you're free to customize it (you just won't get support if it breaks).
    - humbleMouse 7y ago
      Openshift doesn't choose the pipeline. You can deploy to it however you want. Openshift is literally the same exact thing as kubernetes, with a simpler api and a nice ui. Openshift is extremely flexible and you can use it however you want. Don't know why you think it's so restrictive.
      - lima 7y ago
        It has the same API as k8s. It's literally the same thing, OpenShift just adds some PaaS feature on top of k8s.
        snuxoll 7y ago
        Which is precisely why I've run it for over 2 years now. The devs on my team love throwing a git URL into OpenShift and have a project go live in mere minutes, previously a more senior developer would need to setup a CI pipeline and deployment automation and even with experience it was a chore. It's not like I'm personally afraid of k8s or building docker images myself either, but anything to help my team be more productive without needing to hold hands or do gruntwork on their behalf improves the lives of everyone.
        lima 7y ago
        Exactly right. I don't understand all those teams reinventing the wheel building custom k8s distributions, rather than using something like OpenShift and focusing on their core business. Running custom k8s in production is the 2019 equivalent of using Gentoo.
  - argd678 7y ago
    You’re forgetting the network needs to be replaced too, there’s lead times on getting hardware, there the whole racking and cabling, and even at that if the this is due to a more complex networking issue then I can’t swap out the network since there are too many components and they’re shared by other teams. In the cloud I can do this quickly tho.
alexnewman 7y ago
I've absurd bugs at AWS,GCP and azure. I love dynamic capacity but clouds are super flakey, especially AWS. Azure networks are the worst. GCP I haven't used in over a year so I wont talk ...
wazoox 7y ago
Yup, the famous "leaky abstractions"[0]. You can make some reliable assumptions of the data rate of a 40Gbps link in your DC between this and that physical server, and how to get the best out of it. On AWS? Not so much. 0: https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/ https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...
falcolas 7y ago
> spend weeks arguing with AWS support Yup. We've run into that repeatedly. The "we didn't notice anything on our side, please send more screenshots and logs" gets really old when working with "managed services". Network packet losses/truncations, EKS control plane failures, cloudformation stacks getting stuck in really wierd states, inconsistent cloudformation implementations for new and existing services, ENI weirdness in containers... Managed services just feel like they aren't - more and more every day. Amazon (or any other cloud) will never have the same investment in your availability and infrastructure as you will.
- rabi_penguin 7y ago
  Running into one of those exact issues (cloudformation stacks getting stuck in really weird states) right now. A real pain in the ass, because I have zero control or visibility into it.
  - toomuchtodo 7y ago
    Is switching to Terraform an option?
    - falcolas 7y ago
      Based on our own research - probably. But it requires tooling to integrate into a CI/CD pipeline, conversion of existing CF stack definitions to Terraform definitions, and a lossy/downtime laden migration. For a team with the goals of IAC, hands-off, and high-availability infrastructure, migrating is a pain.
      - ReidZB 7y ago
        With Terraform's import functionality, you might be able to migrate into Terraform without any downtime. I believe there are even utilities to translate CloudFormation code straight into Terraform, though I've never tried them. It would definitely be an annoying migration, even if import works for your use-case, but honestly, dealing with CloudFormation is so irritating on a day-to-day basis for me that I'd consider it worth it.
john_moscow 7y ago
You need to treat a commodity like a commodity. It's perfectly OK to run business-critical processes in someone's cloud, as long as you have a fully automated process that will deploy your entire environment from the latest backup to any set of cloud nodes at any cloud provider you point it to. Unless you're doing that, you're just shifting risks, not eliminating them.

by lima 7y ago

argd678 7y ago

I’ve had performance issues with GCP, however I can switch cloud providers in that case without too much effort. With physical hardware that’s much harder.

lima 7y ago
If you have a well-built and automated bare-metal deployment, it's no harder than switching cloud providers. At the scale where it makes sense to go bare metal, either would be a lot of effort. An extra abstraction layer like k8s makes it a lot easier, which is exciting. It's also precisely why IBM bought Red Hat - a well-built k8s distribution like OpenShift is one of the very few real alternative to public clouds for many companies.
- windexh8er 7y ago
  > An extra abstraction layer like k8s makes it a lot easier, which is exciting. It's also precisely why IBM bought Red Hat - a well-built k8s distribution like OpenShift is one of the very few real alternative to public clouds for many companies. OpenShift is built on a prescriptive mentality. They choose everything for you from OS through pipeline and many deviations are unsupported. As you mentioned there are others, but depending on your environment OpenShift is either a very good fit or square-peg-round-hole.
  - lima 7y ago
    OpenShift is a superset of k8s, so you're free to customize it (you just won't get support if it breaks).
  - humbleMouse 7y ago
    Openshift doesn't choose the pipeline. You can deploy to it however you want. Openshift is literally the same exact thing as kubernetes, with a simpler api and a nice ui. Openshift is extremely flexible and you can use it however you want. Don't know why you think it's so restrictive.
    - lima 7y ago
      It has the same API as k8s. It's literally the same thing, OpenShift just adds some PaaS feature on top of k8s.
      - snuxoll 7y ago
        Which is precisely why I've run it for over 2 years now. The devs on my team love throwing a git URL into OpenShift and have a project go live in mere minutes, previously a more senior developer would need to setup a CI pipeline and deployment automation and even with experience it was a chore. It's not like I'm personally afraid of k8s or building docker images myself either, but anything to help my team be more productive without needing to hold hands or do gruntwork on their behalf improves the lives of everyone.
        lima 7y ago
        Exactly right. I don't understand all those teams reinventing the wheel building custom k8s distributions, rather than using something like OpenShift and focusing on their core business. Running custom k8s in production is the 2019 equivalent of using Gentoo.
- argd678 7y ago
  You’re forgetting the network needs to be replaced too, there’s lead times on getting hardware, there the whole racking and cabling, and even at that if the this is due to a more complex networking issue then I can’t swap out the network since there are too many components and they’re shared by other teams. In the cloud I can do this quickly tho.

alexnewman 7y ago

I've absurd bugs at AWS,GCP and azure. I love dynamic capacity but clouds are super flakey, especially AWS. Azure networks are the worst. GCP I haven't used in over a year so I wont talk ...

wazoox 7y ago

Yup, the famous "leaky abstractions"[0]. You can make some reliable assumptions of the data rate of a 40Gbps link in your DC between this and that physical server, and how to get the best out of it. On AWS? Not so much. 0: https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/ https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...

falcolas 7y ago

> spend weeks arguing with AWS support Yup. We've run into that repeatedly. The "we didn't notice anything on our side, please send more screenshots and logs" gets really old when working with "managed services". Network packet losses/truncations, EKS control plane failures, cloudformation stacks getting stuck in really wierd states, inconsistent cloudformation implementations for new and existing services, ENI weirdness in containers... Managed services just feel like they aren't - more and more every day. Amazon (or any other cloud) will never have the same investment in your availability and infrastructure as you will.

rabi_penguin 7y ago
Running into one of those exact issues (cloudformation stacks getting stuck in really weird states) right now. A real pain in the ass, because I have zero control or visibility into it.
- toomuchtodo 7y ago
  Is switching to Terraform an option?
  - falcolas 7y ago
    Based on our own research - probably. But it requires tooling to integrate into a CI/CD pipeline, conversion of existing CF stack definitions to Terraform definitions, and a lossy/downtime laden migration. For a team with the goals of IAC, hands-off, and high-availability infrastructure, migrating is a pain.
    - ReidZB 7y ago
      With Terraform's import functionality, you might be able to migrate into Terraform without any downtime. I believe there are even utilities to translate CloudFormation code straight into Terraform, though I've never tried them. It would definitely be an annoying migration, even if import works for your use-case, but honestly, dealing with CloudFormation is so irritating on a day-to-day basis for me that I'd consider it worth it.

john_moscow 7y ago

You need to treat a commodity like a commodity. It's perfectly OK to run business-critical processes in someone's cloud, as long as you have a fully automated process that will deploy your entire environment from the latest backup to any set of cloud nodes at any cloud provider you point it to. Unless you're doing that, you're just shifting risks, not eliminating them.