5 ms·

Do you actually think Kubernetes infrastructure is built without hair-pulling and rage? If so, I've got some stories for you.

nickthemagicman 7y ago
Well it's contained the hair-pulling and rage to just the kubernetes administrators instead of the whole organization. :P
- falcolas 7y ago
  No, the rage extends to the developers too. Writing all of the yaml for a full service requires a fair bit of understanding of what's going on in k8s. And helm templating.
shadowgovt 7y ago
It's hair-pulling and rage that results in forward progress on a fungible toolchain (as opposed to hair-pulling and rage that results in a bespoke one-company ecosystem where you leave behind more than half the useful experience you gained when you transfer to a company based upon ecosystem fundamentals utterly different from the ones bought with the blood, sweat, and tears of solving the problem at the first place).
pm90 7y ago
Using managed kubernetes solutions alleviates most of the hair pulling.
- falcolas 7y ago
  Speaking as someone using EKS: No, no it doesn't. The only thing we really handed over to someone else was the challenge of running a control plane. We still have to roll our own node images (for compliance reasons, the default images are insufficiently secure), still have to work directly with AWS to get larger etcd clusters, still have to manage the cluster itself... The trouble has barely been below the threshold of pain to just run our own control plane.
  - otterley 7y ago
    Out of curiosity, have you considered using ECS instead? If not, why not? (Disclaimer: I work for AWS as a specialist solutions architect.)
    - blaser-waffle 7y ago
      Not that person, but I do a lot of docker at a big enterprise. Sell me.
      - otterley 7y ago
        Drop me a line - eff eye ess see emm eye at amazon dot com :)
- busterarm 7y ago
  Absolutely no it does not. I work with GKE and EKS almost exclusively and it compounds the headaches most of the time. There is varying support for Kubernetes resources in both and all kinds of vendor-specific ways you need to do things to keep you locked in. Every single managed Kubernetes solution available is a unique, special snowflake. Most folks in-house ones are the same. Saying that Kubernetes isn't some bespoke solution is a myth. Kubernetes is a collection of haphazardly-assembled components in varying states of maturity and that landscape changes dramatically with minor release versions (GKE's Stable release branch is still 1.13, for example). YMMV.
  - pm90 7y ago
    Give specific problems you see and I will give you a reasoned reply. I will not engage with hyperbole devoid of facts.
    - busterarm 7y ago
      It's help I don't need and didn't ask for. I'm managing several million dollars worth of infrastructure in both. I gave enough detail that anyone familiar enough with both platforms should know some of the issues I'm talking about already.
      - pm90 7y ago
        You did not.
        busterarm 7y ago
        > There is varying support for Kubernetes resources I dunno, EKS not having built-in support for Ingress objects is a pretty large and obvious elephant in the room. I'm not playing this game. I've stated my experiences and suggested the level of interaction I've had with the platforms in dollar figures enough to be comfortable with what I'm saying here.
        7y ago
LoSboccacc 7y ago
yeah but for fighting with all the details once upfront you get an infinitely replicable environment. not just scalability but you can have as much test environments, all identical, spinning up with an apply. and the next guy in line can do it too! possibly with a convenient front-end and little understanding, so you can spend your skills elsewhere not for everyone, sure, but then which tool is?
- busterarm 7y ago
  I had that with terraform before Kubernetes anyway.
  - LoSboccacc 7y ago
    ...aren't them from the same age?
    - busterarm 7y ago
      Two different paths that can get you to the same place (but usually don't and surely Kubernetes is the cheaper option month-to-month).
    - LoSboccacc 7y ago
      they're bot 2014ish, but hacker news and facts aren't a good match, better a long tirade and snarky comments against the grain, I guess.
      - busterarm 7y ago
        One was usable in production long before the other. Years even.
- yongjik 7y ago
  Maybe I'm particularly unfortunate, but that wasn't my experience at all. It's either bash scripts that make you want to pull your hair out, or yaml files that make you want to pull your hair out. The same people who wrote these bash scripts are writing yaml files, and with k8s there are infinitely more interesting ways to bork your jobs.

nickthemagicman 7y ago

Well it's contained the hair-pulling and rage to just the kubernetes administrators instead of the whole organization. :P

falcolas 7y ago
No, the rage extends to the developers too. Writing all of the yaml for a full service requires a fair bit of understanding of what's going on in k8s. And helm templating.

shadowgovt 7y ago

It's hair-pulling and rage that results in forward progress on a fungible toolchain (as opposed to hair-pulling and rage that results in a bespoke one-company ecosystem where you leave behind more than half the useful experience you gained when you transfer to a company based upon ecosystem fundamentals utterly different from the ones bought with the blood, sweat, and tears of solving the problem at the first place).

pm90 7y ago

Using managed kubernetes solutions alleviates most of the hair pulling.

falcolas 7y ago
Speaking as someone using EKS: No, no it doesn't. The only thing we really handed over to someone else was the challenge of running a control plane. We still have to roll our own node images (for compliance reasons, the default images are insufficiently secure), still have to work directly with AWS to get larger etcd clusters, still have to manage the cluster itself... The trouble has barely been below the threshold of pain to just run our own control plane.
- otterley 7y ago
  Out of curiosity, have you considered using ECS instead? If not, why not? (Disclaimer: I work for AWS as a specialist solutions architect.)
  - blaser-waffle 7y ago
    Not that person, but I do a lot of docker at a big enterprise. Sell me.
    - otterley 7y ago
      Drop me a line - eff eye ess see emm eye at amazon dot com :)
busterarm 7y ago
Absolutely no it does not. I work with GKE and EKS almost exclusively and it compounds the headaches most of the time. There is varying support for Kubernetes resources in both and all kinds of vendor-specific ways you need to do things to keep you locked in. Every single managed Kubernetes solution available is a unique, special snowflake. Most folks in-house ones are the same. Saying that Kubernetes isn't some bespoke solution is a myth. Kubernetes is a collection of haphazardly-assembled components in varying states of maturity and that landscape changes dramatically with minor release versions (GKE's Stable release branch is still 1.13, for example). YMMV.
- pm90 7y ago
  Give specific problems you see and I will give you a reasoned reply. I will not engage with hyperbole devoid of facts.
  - busterarm 7y ago
    It's help I don't need and didn't ask for. I'm managing several million dollars worth of infrastructure in both. I gave enough detail that anyone familiar enough with both platforms should know some of the issues I'm talking about already.
    - pm90 7y ago
      You did not.
      - busterarm 7y ago
        > There is varying support for Kubernetes resources I dunno, EKS not having built-in support for Ingress objects is a pretty large and obvious elephant in the room. I'm not playing this game. I've stated my experiences and suggested the level of interaction I've had with the platforms in dollar figures enough to be comfortable with what I'm saying here.
        7y ago

LoSboccacc 7y ago

yeah but for fighting with all the details once upfront you get an infinitely replicable environment. not just scalability but you can have as much test environments, all identical, spinning up with an apply. and the next guy in line can do it too! possibly with a convenient front-end and little understanding, so you can spend your skills elsewhere not for everyone, sure, but then which tool is?

busterarm 7y ago
I had that with terraform before Kubernetes anyway.
- LoSboccacc 7y ago
  ...aren't them from the same age?
  - busterarm 7y ago
    Two different paths that can get you to the same place (but usually don't and surely Kubernetes is the cheaper option month-to-month).
  - LoSboccacc 7y ago
    they're bot 2014ish, but hacker news and facts aren't a good match, better a long tirade and snarky comments against the grain, I guess.
    - busterarm 7y ago
      One was usable in production long before the other. Years even.
yongjik 7y ago
Maybe I'm particularly unfortunate, but that wasn't my experience at all. It's either bash scripts that make you want to pull your hair out, or yaml files that make you want to pull your hair out. The same people who wrote these bash scripts are writing yaml files, and with k8s there are infinitely more interesting ways to bork your jobs.