5 ms·

In my experience, the end result is having a team of kubernetes fanatics reinventing the wheels themselves and forcing everyone to use their custom ego filling

by midrus 5y ago

In my experience, the end result is having a team of kubernetes fanatics reinventing the wheels themselves and forcing everyone to use their custom ego filling tools instead of paying $100 per month to an already existing solution. Most developers just need something like heroku, tsuru, appengine, or openstack. But because kubefans want to have fun too and try to be as cool as Google or Facebook we end up having to deal with tons of custom Golang tooling with fancy names, half assed documentation and grumpy looks when you ask for help because you don't understand the 38 steps guide to deploy a service.

sascha_sl 5y ago
We're not here to compete for most cynical take. $100 is not really a price point for any of the things you mentioned, and the way PaaS charge is often restrictive of development model (want microservices? we charge by the service. want a monolith? sorry, we only have instances up to this size. need to scale in smaller increments than "one machine"? too bad.) When I helped build my first platform, it was a migration from Beanstalk to Kubernetes (1.8 at the time). Not only did it save 70%+ costs, it allowed developers to cut services as large as they wanted, and scale them as fast or slow as they wanted. Metrics and logging were nothing they had to worry about on the infra layer either. I wouldn't build these things again, OAM / KubeVela is a nice compromise if you want one. Also, I wouldn't give a dev direct access to Openstack the same way I woudn't give them access to Kubernetes itself. They have too many footguns and too much complexity, either you end up with half-baked deployments or half the teams time spent on doing things the right way.
- midrus 5y ago
  Of course it is not $100, but it is a lot less than the salaries of a team of infrastructure engineers most of the times
  - sascha_sl 5y ago
    Anecdotally, when my entire team started leaving over the span of 5 months or so after a new CEO attempted to force a move to every service provider he had previously worked with (GCP over AwS, Cloudflare over Akamai, MS Teams over Slack), the non-cloud costs for the platform jumped to about 6 times the teams combined salary when they decided to hire specialist contractors, and even they had a hard time matching the raw performance of that cluster. The typical case of feature teams not being allowed to refactor / reduce technical debt was extreme. Some frontend requests would result in hundreds of backend requests for no good reason. We've always been ignored by management, so cluster was on the verge of coming apart at the (mostly "spending half the cpu time in conntrack") seams for years. So close, it'd sometimes just die for a minute, probably because someone decided to use HTTP way too liberally (network is free, right?). I had a pcap file from a sample of 10 machines, and a 30 second sample was so large, it was impossible to analyze with any real-time tools. No managed Kubernetes could match this out of the box. Adding a mesh to do more directed requests for this to be fixed failed even when we tried it. As far as I know they're still using the platform as is (in maintenance mode, with the contractor assisting) and fired that CEO pretty soon after. Is this a bit of a special case? Sure! Does it still happen a lot, no matter your infrastructure? Count on it.
    - midrus 5y ago
      I don't know why would you need contractors, maybe that is what went wrong? A company I worked for years ago moved our custom built platform to AppEngine Flex and it was the best thing we could have done. We embaraced DevOps as it should be done (not with "devops people" but with developers understanding their production environment). No need for contractors. Just the documentations and an easy to use, well tested and battle proof platform. Similar experience many, many years ago with Tsuru, which is a kind of smaller self hosted heroku, although this was a company of about 30 enhineers
      - sascha_sl 5y ago
        Scale, in terms of diversity of use-cases rather than requests per second, seems to be a very important factor in choosing custom, semi-custom or ready-made. Kubernetes being "a platform for building platforms" still rings true to me, and it is extremely good at that. One major issue was that the culture wasn't great. SREs without teeth and Platform Engineers that fix the shortcomings of development teams with glue more than anything. As for why contractors, this was the IT division of a big retail business that refused to pay competitive salaries in solidarity with their other workers, but never fired anyone for underperforming (or not performing at all), so the people that actually get work done tend to leave pretty fast. They tried to compensate this with contractors. Some departments were fine, and I loved the team (high standards, created lots of value for teams), but it wasn't a great long term workplace.
halfmatthalfcat 5y ago
> Most developers just need something like heroku, tsuru, appengine, or openstack. A lot of companies don't operate at Google or Facebook scale but many companies operate at a scale which quickly outgrows the solutions you mentioned. Maybe you just haven't worked at places with larger scale to see the benefits things like Kubernetes provides?
- midrus 5y ago
  The key is in what that scale is. Everyone thinks they're already at that scale, or soon will be. Most are not and won't for a very long time. I've worked for large companies (> 500 engineers) where building your own custom thing might make sense. But I've also worked for companies with less than 60 engineers where about 10 of them where infrastructure engineers building all this craziness we're talking about. Put into the equation all these salaries and the opportunity cost and you have a pretty juicy budget for appengine or similar.

by midrus 5y ago

sascha_sl 5y ago

We're not here to compete for most cynical take. $100 is not really a price point for any of the things you mentioned, and the way PaaS charge is often restrictive of development model (want microservices? we charge by the service. want a monolith? sorry, we only have instances up to this size. need to scale in smaller increments than "one machine"? too bad.) When I helped build my first platform, it was a migration from Beanstalk to Kubernetes (1.8 at the time). Not only did it save 70%+ costs, it allowed developers to cut services as large as they wanted, and scale them as fast or slow as they wanted. Metrics and logging were nothing they had to worry about on the infra layer either. I wouldn't build these things again, OAM / KubeVela is a nice compromise if you want one. Also, I wouldn't give a dev direct access to Openstack the same way I woudn't give them access to Kubernetes itself. They have too many footguns and too much complexity, either you end up with half-baked deployments or half the teams time spent on doing things the right way.

midrus 5y ago
Of course it is not $100, but it is a lot less than the salaries of a team of infrastructure engineers most of the times
- sascha_sl 5y ago
  Anecdotally, when my entire team started leaving over the span of 5 months or so after a new CEO attempted to force a move to every service provider he had previously worked with (GCP over AwS, Cloudflare over Akamai, MS Teams over Slack), the non-cloud costs for the platform jumped to about 6 times the teams combined salary when they decided to hire specialist contractors, and even they had a hard time matching the raw performance of that cluster. The typical case of feature teams not being allowed to refactor / reduce technical debt was extreme. Some frontend requests would result in hundreds of backend requests for no good reason. We've always been ignored by management, so cluster was on the verge of coming apart at the (mostly "spending half the cpu time in conntrack") seams for years. So close, it'd sometimes just die for a minute, probably because someone decided to use HTTP way too liberally (network is free, right?). I had a pcap file from a sample of 10 machines, and a 30 second sample was so large, it was impossible to analyze with any real-time tools. No managed Kubernetes could match this out of the box. Adding a mesh to do more directed requests for this to be fixed failed even when we tried it. As far as I know they're still using the platform as is (in maintenance mode, with the contractor assisting) and fired that CEO pretty soon after. Is this a bit of a special case? Sure! Does it still happen a lot, no matter your infrastructure? Count on it.
  - midrus 5y ago
    I don't know why would you need contractors, maybe that is what went wrong? A company I worked for years ago moved our custom built platform to AppEngine Flex and it was the best thing we could have done. We embaraced DevOps as it should be done (not with "devops people" but with developers understanding their production environment). No need for contractors. Just the documentations and an easy to use, well tested and battle proof platform. Similar experience many, many years ago with Tsuru, which is a kind of smaller self hosted heroku, although this was a company of about 30 enhineers
    - sascha_sl 5y ago
      Scale, in terms of diversity of use-cases rather than requests per second, seems to be a very important factor in choosing custom, semi-custom or ready-made. Kubernetes being "a platform for building platforms" still rings true to me, and it is extremely good at that. One major issue was that the culture wasn't great. SREs without teeth and Platform Engineers that fix the shortcomings of development teams with glue more than anything. As for why contractors, this was the IT division of a big retail business that refused to pay competitive salaries in solidarity with their other workers, but never fired anyone for underperforming (or not performing at all), so the people that actually get work done tend to leave pretty fast. They tried to compensate this with contractors. Some departments were fine, and I loved the team (high standards, created lots of value for teams), but it wasn't a great long term workplace.

halfmatthalfcat 5y ago

> Most developers just need something like heroku, tsuru, appengine, or openstack. A lot of companies don't operate at Google or Facebook scale but many companies operate at a scale which quickly outgrows the solutions you mentioned. Maybe you just haven't worked at places with larger scale to see the benefits things like Kubernetes provides?

midrus 5y ago
The key is in what that scale is. Everyone thinks they're already at that scale, or soon will be. Most are not and won't for a very long time. I've worked for large companies (> 500 engineers) where building your own custom thing might make sense. But I've also worked for companies with less than 60 engineers where about 10 of them where infrastructure engineers building all this craziness we're talking about. Put into the equation all these salaries and the opportunity cost and you have a pretty juicy budget for appengine or similar.