4 ms·

> Why Does Developing on Kubernetes Suck ? IMHO because we are in a phase of transition. Having worked for years in software industry , I'm convinced we are h

> Why Does Developing on Kubernetes Suck ? IMHO because we are in a phase of transition. Having worked for years in software industry , I'm convinced we are halfway to a much bigger transformation for Software Engineers / SRE , Developers etc... I work in a Neobank ( N26 , Revolut, etc...) , we are currently in the process of re-writing our entire Core Banking System with MicroServices on top of Kubernetes with Kafka. Not a single day pass without having engineers needing to have an exchange about defining basically all of the the terms that exist within the K8/Docker/Kafa world. - What's a Pod ? How does a pod behave if Kafa goes down ? Do we really need ZooKeeper etc.... Their workflows is insanely complex and requires hours if not a day to deploy a single change... obviously let's not even talk about the amount of work our SRE has in the pipe to "package" the entire stack of 150+ services in K8 through a single YAML file.... I'm sure this complexity is temporary. One tech will automate all of this away and building , deploying and running will be much simpler. In it's current shape , K8 remains me of mainframe ecosystem I used to deal with at brick & mortars banks. Powerful systems , but they requires a tremendous amount of work and experts to be properly managed and take full advantage of it. Just like mainframes , K8 leaves very little room for "mistakes" or "approximation".

jasonvorhe 7y ago
Something is very wrong at this company if developers need hours or even days to deploy changes in a microservice architecture. I suggest to find an outside consultant with verifiable Kubernetes experience in production to get this checked out. Maybe a bit of outside perspective will help your teams to fix this. Good luck.
- longcommonname 7y ago
  Legacy banking companies have very strict change management systems. These systems usually require many physical eyes on each change. This magnifests into a fear of fully automated change management.
rapsey 7y ago
> Their workflows is insanely complex and requires hours if not a day to deploy a single change... obviously let's not even talk about the amount of work our SRE has in the pipe to "package" the entire stack of 150+ services in K8 through a single YAML file.... One should always keep in mind the famous aphorism: > "All problems in computer science can be solved by another level of indirection, except for the problem of too many levels of indirection". You may have just hit the "except" part.
- meowface 7y ago
  That's a fantastic quote. Don't know how I haven't come across it before. It definitely sums up our profession in a nutshell.
VBprogrammer 7y ago
There is a lot of terminology for sure. We've gone from "is docker ready for production usage?" to "Hey Bob, look at this, a service which isn't running in a container yet" in about 5 years. In order to work effectively within kubernetes you've got to understand docker, with all of its nomenclature (images, containers, volumes, layers etc). Then you've got the things kubernetes layers on top (pods, ingress, stateful sets, persistent volumes etc). You then probably layer Helm (adding tiller, deployments, charts, templates) or something else on top of that. We've still got the complexity from before that, remember when ansible, puppet and chef were all the rage? You're probably still using that to initially configure your bare metal. You still have to understand all of the quirks of your data store, except now instead of throwing everything in a huge SQL DB you have some things in Redis, MongoDB, Kafka and maybe some legacy stuff in that big DB in the corner. Let's not forget your CI stack in Jenkins, Travis or maybe codefresh which builds and runs your tests. That probably does some magic with docker too. Oh, and if anything goes wrong in any of that you'd better be well grounded in general unixy debugging (permissions, networking etc). It's certainly getting to the point that I feel no one person can be an expert in all of it, especially because when it all works you can more or less let it do its thing. When it breaks though it feels like playing Jenga.
DonHopkins 7y ago
You need to upgrade to the new Zookeeper 2: Zookeepier! https://www.youtube.com/watch?v=_F-RyuDLR4o https://www.youtube.com/watch?v=_F-RyuDLR4o
imtringued 7y ago
>I'm sure this complexity is temporary. One tech will automate all of this away and building , deploying and running will be much simpler. Kubernetes was supposed to be one of those technologies.
- nikisweeting 7y ago
  > I'm convinced we are halfway to a much bigger transformation for Software Engineers / SRE , Developers etc... I think kubernetes is the intro that gave everyone a taste of whats possible. What's yet to come is an easy abstraction layer or OS integration that makes it possible to use without learning 100 bits of new terminology and config minutiae.
collyw 7y ago
>Their workflows is insanely complex and requires hours if not a day to deploy a single change... Isn't this what micro-services were supposed to be the solution to? I get the feeling the whole thing is oversold.
- arethuza 7y ago
  Isn't it kind of defeating the purpose of micro-services if you deploy every one of them every time?
  - js4ever 7y ago
    Exactly! I've seen this so many times in the past few months... It seems peoples don't understand the true nature of microservices
    - physicles 7y ago
      Yep, it’s common enough that it’s considered an anti-pattern. It’s called “distributed monolith.”

jasonvorhe 7y ago

Something is very wrong at this company if developers need hours or even days to deploy changes in a microservice architecture. I suggest to find an outside consultant with verifiable Kubernetes experience in production to get this checked out. Maybe a bit of outside perspective will help your teams to fix this. Good luck.

longcommonname 7y ago
Legacy banking companies have very strict change management systems. These systems usually require many physical eyes on each change. This magnifests into a fear of fully automated change management.

rapsey 7y ago

> Their workflows is insanely complex and requires hours if not a day to deploy a single change... obviously let's not even talk about the amount of work our SRE has in the pipe to "package" the entire stack of 150+ services in K8 through a single YAML file.... One should always keep in mind the famous aphorism: > "All problems in computer science can be solved by another level of indirection, except for the problem of too many levels of indirection". You may have just hit the "except" part.

meowface 7y ago
That's a fantastic quote. Don't know how I haven't come across it before. It definitely sums up our profession in a nutshell.

VBprogrammer 7y ago

There is a lot of terminology for sure. We've gone from "is docker ready for production usage?" to "Hey Bob, look at this, a service which isn't running in a container yet" in about 5 years. In order to work effectively within kubernetes you've got to understand docker, with all of its nomenclature (images, containers, volumes, layers etc). Then you've got the things kubernetes layers on top (pods, ingress, stateful sets, persistent volumes etc). You then probably layer Helm (adding tiller, deployments, charts, templates) or something else on top of that. We've still got the complexity from before that, remember when ansible, puppet and chef were all the rage? You're probably still using that to initially configure your bare metal. You still have to understand all of the quirks of your data store, except now instead of throwing everything in a huge SQL DB you have some things in Redis, MongoDB, Kafka and maybe some legacy stuff in that big DB in the corner. Let's not forget your CI stack in Jenkins, Travis or maybe codefresh which builds and runs your tests. That probably does some magic with docker too. Oh, and if anything goes wrong in any of that you'd better be well grounded in general unixy debugging (permissions, networking etc). It's certainly getting to the point that I feel no one person can be an expert in all of it, especially because when it all works you can more or less let it do its thing. When it breaks though it feels like playing Jenga.

DonHopkins 7y ago

You need to upgrade to the new Zookeeper 2: Zookeepier! https://www.youtube.com/watch?v=_F-RyuDLR4o https://www.youtube.com/watch?v=_F-RyuDLR4o

imtringued 7y ago

>I'm sure this complexity is temporary. One tech will automate all of this away and building , deploying and running will be much simpler. Kubernetes was supposed to be one of those technologies.

nikisweeting 7y ago
> I'm convinced we are halfway to a much bigger transformation for Software Engineers / SRE , Developers etc... I think kubernetes is the intro that gave everyone a taste of whats possible. What's yet to come is an easy abstraction layer or OS integration that makes it possible to use without learning 100 bits of new terminology and config minutiae.

collyw 7y ago

>Their workflows is insanely complex and requires hours if not a day to deploy a single change... Isn't this what micro-services were supposed to be the solution to? I get the feeling the whole thing is oversold.

arethuza 7y ago
Isn't it kind of defeating the purpose of micro-services if you deploy every one of them every time?
- js4ever 7y ago
  Exactly! I've seen this so many times in the past few months... It seems peoples don't understand the true nature of microservices
  - physicles 7y ago
    Yep, it’s common enough that it’s considered an anti-pattern. It’s called “distributed monolith.”