5 ms·

I was once talking to an ex google site reliability engineer. He said there are maybe a handful of companies in the world that _need_ k8s. I tend to agree. A lo

I was once talking to an ex google site reliability engineer. He said there are maybe a handful of companies in the world that _need_ k8s. I tend to agree. A lot of people practice hype driven development.

x86x87 3y ago
I tend to agree. K8s makes a lot of sense if you are running your own bare metal servers at scale. If you are already using the cloud, maybe leverage abstraction already available in that context.
- candiddevmike 3y ago
  You either recreate a less reliable version of kubernetes for workload ops or you go all in on your cloud provider and hope they'll be responsible for your destiny. Vanilla Kubernetes is just enough abstraction to avoid both of those situations.
  - x86x87 3y ago
    You cannot really be cloud agnostic these days - even when using k8s. So learning to use the capabilities the cloud provides is key.
    - p_l 3y ago
      Doesn't really mesh with my experience, especially the longer k8s been out. It can be cheaper to depend on cloud provider to ship some features, but with tools like crossplane you can abstract that out so developers can just "order" a database service etc. for their application.
  - PUSH_AX 3y ago
    Is “hope” the new replacement for SLAs? Or am I missing something with that statement?
    - k8sToGo 3y ago
      SLA do not prevent something from breaking, unfortunately. It is just a blame construct.
    - p_l 3y ago
      "Hope" that your cloud provider matches as well your needs as you thought, that vendor lock-in doesn't let them milk you with high prices, etc. etc. None of that is prevented with SLA
      - PUSH_AX 3y ago
        This requires the same skill and experience as figuring out if k8s is going to be a good fit. Arguably if you can’t evaluate the raw cloud offerings and jump on a supposed silver bullet you need to stop immediately.
        p_l 3y ago
        At this point I found out that k8s knowledge is more portable, whereas your trove of $VENDOR_1 knowledge might suddenly have issues because for reasons outside of your capacity to control there's now big spending contract signed with $VENDOR_2 and a mandate to move. And with smaller companies I tend to find k8s way more cost effective. I pulled things I wouldn't be able to fit in a budget otherwise.
- k8sToGo 3y ago
  I joined a team that used AWS without kubernetes. Thousands of fragile weird python and bash scripts. Deployment was always such a headache. A few months later I transitioned the team to use containers with proper CI/CD and EKS with Terraform and Argo CD. The team and also the managers like it, since we could deploy quite quickly.
  - evantbyrne 3y ago
    This is an apples-to-oranges comparison. You would still have to write and maintain glue without the presence of a proper CD.
  - deleted 3y ago
    [deleted]
  - PUSH_AX 3y ago
    Thanks for the anecdote k8sToGo
misiti3780 3y ago
if not k8, what would other people be using? ECS?
- k8sToGo 3y ago
  From my experience, classical VMs with self written Bash scripts. The horror!
- kenhwang 3y ago
  If you're on AWS, yeah, I'd say just use ECS until you need more complexity. Our ECS deployments have been unproblematic for years now. Our K8s clusters never goes more than a couple days without some sort of strange issue popping up. Arguably it could be because my company outsourced maintenance of it to an army of idiots. But K8s is a tool that is only as good as the operator, and competence can be hard to come by at some companies.
  - p_l 3y ago
    K8s or no K8s, outsource to lowest bidder and you'll get unworkable platform :|
    - kenhwang 3y ago
      Agreed. But if you're already on AWS, I'd say the quality floor is already higher than the potential at 95%+ of other companies. So I say unless you're at a company that pays top salaries for the top 5% of engineering talent, you're probably better off just using the AWS provided service.
      - p_l 3y ago
        I used to have a saying back when Heroku was more favourable, is that you use Heroku because you want to go bankrupt. AWS is at times similar. Depending on your local market, AWS bills might be way worse than the cost of few bright ops people who will let you choose from offerings including running dev envs on random assortment of dedicated servers and local e-waste escapees
geodel 3y ago
And that hype is in large part created by Google and other cloud vendors. To be honest I hardly see any reasonable/actionable advice from Cloud/SAAS vendors. Either it is to sell their stuff or generic stuff like "One should be securing / monitoring their stuff running in prod". Oh wow, never thought or done any such thing before.
imiric 3y ago
That might be true, but unfortunately the state of the art infrastructure tooling is mostly centered around k8s. This means that companies choose k8s (or related technologies like k3s, Microk8s, etc.) not because they strictly _need_ k8s, but because it improves their workflows. Otherwise they would need to invest a disproportionate amount of time and effort adopting and maintaining alternative tooling, while getting an inferior experience. Choosing k8s is not just based on scaling requirements anymore. There are also benefits of being compatible with a rich ecosystem of software.
- PUSH_AX 3y ago
  Can you specify what state of the art infra tooling you mean?
  - imiric 3y ago
    Continuous deployment systems like ArgoCD and Flux, user friendly local development environments with tools like Tilt, novel networking, distributed storage, distributed tracing, etc. systems that are basically plug-and-play, etc. Search for "awesome k8s" and you'll get many lists of these. It's surely possible to cobble all of this together without k8s, but k8s' main advantage is exposing a standardized API that simplifies managing this entire ecosystem. It often makes it worth the additional overhead of adopting, understanding and managing k8s itself.
k8sToGo 3y ago
I push for k8s because I know it. Why not use something that I know how to use? I know how to quickly set up a cluster, what to deploy, and teach other team members about fundamentals. How many people out there really need C# or object oriented programming? The argument you present might be valid if you decide to use a tech stack prior having much experience with it.
- nprateem 3y ago
  Yeah that's the point. You know it and stuff everyone else.
  - p_l 3y ago
    Some custom bash/python/ansible monstrosity is only going to be known by few brains in the world. K8s is remarkably easier to retain institutional knowledge as well as spread it.
    - nprateem 3y ago
      If you're expecting app/FE devs to have to learn it you're putting a ton of barriers in their way in terms of deploying. Just chucking a container on a non-k8s managed platform (e.g. Cloud Run) would be much simpler, and no pile of bash scripts.
      - p_l 3y ago
        PaaSes are for companies with money to burn, most of the time. A good k8s team (even a single person, to be quite honest) is going to work towards providing your application teams with simple templates to let them deploy their software easily. Just let them do it. Also, in my experience, you either have to spend ridiculous amounts of money on SaaS/PaaS, or you find that you have to host a lot more than just your application and suddenly the deployment story becomes more complex. Depending on where you are and how much you're willing to burn money, you might find out that k8s experts are cheaper than the money saved by not going PaaS.
      - 3y ago
planetafro 3y ago
Just a thought as well in my corpo experience: Unfortunately, there are some spaces that distribute solutions as k8s-only... Which sucks. I've noticed this mostly in the data science/engineering world. These are solutions that could be easily served up in a small docker compose env. The complexity/upsell/devops BS is strong. To add insult to injury, I've seen more than one use IaC cloud tooling as an install script vs a maintainable and idempotent solution. It's all quite sad really.
p_l 3y ago
There's a difference between need or you don't survive and it improves our operations. The former is a very small set involving having huge amounts of bare metal systems. The latter is suprisingly large set of companies, sometimes even with one server.
Thaxll 3y ago
It's a dumb statement especially from an SRE, it's typically a comment from people that don't understand k8s and think that k8s is only there to have the SLA of Google. For most use case k8s is not there to give you HA but to give you a standard way of deploying a stack, that being on the cloud or on prem.
- PUSH_AX 3y ago
  He understood it fully, he was running a multi day course on it when I spoke to him. He was candid about the tech, most of us where there at the behest of our orgs.
  - p_l 3y ago
    In my personal experience, Google SREs as well as k8s devs sometimes didn't grok how wide k8s usability was - they also can be blind to financial aspects of companies living outside of Silly Valley.
throwawaaarrgh 3y ago
Most companies in the world don't need to develop software. Software development itself is hype. But there's lots of money in it, despite no actual value being created most of the time.
mrj 3y ago
Kubernetes scales down pretty well. I don't use network layers or crazy ingress setups. I keep it simple and Kubernetes works great. What's wonderful is that when I work on multiple clouds, my knowledge transfers just fine. I don't think of the AWS solution or the GCS solution, I use the same kubectl to check out both, view logs, inspect and fix. Even when I got tired of waiting for GKE to spin up a node, running Github actions on a self-hosted microk8s meant instant pod starts and very little fuss. But using Kubernetes meant I got to take advantage of the Github operator, which let me reuse the same machine for multiple builds without the headaches. When I want to run some open source, I often find a helm chart the helps me get set up. Nowadays running open source packages can involve all kinds of dependencies, but getting it running on a k8s cluster to check it out, or even in prod, is a relatively straight forward editing of some values files. I've recently ran Uptrace and Superset that way. They're not a bajillion requests per second setups, they don't have to be, and it was far easier to set up than most methods. I would say your friend is right. Few people _need_ k8s but it's one interface to a bunch of complicated proprietary stuff. I can know core small, core set of k8s tools really well and forget half of the junk that I ever knew about public clouds. It's all the same patterns, transferable and reliable.

x86x87 3y ago

I tend to agree. K8s makes a lot of sense if you are running your own bare metal servers at scale. If you are already using the cloud, maybe leverage abstraction already available in that context.

candiddevmike 3y ago
You either recreate a less reliable version of kubernetes for workload ops or you go all in on your cloud provider and hope they'll be responsible for your destiny. Vanilla Kubernetes is just enough abstraction to avoid both of those situations.
- x86x87 3y ago
  You cannot really be cloud agnostic these days - even when using k8s. So learning to use the capabilities the cloud provides is key.
  - p_l 3y ago
    Doesn't really mesh with my experience, especially the longer k8s been out. It can be cheaper to depend on cloud provider to ship some features, but with tools like crossplane you can abstract that out so developers can just "order" a database service etc. for their application.
- PUSH_AX 3y ago
  Is “hope” the new replacement for SLAs? Or am I missing something with that statement?
  - k8sToGo 3y ago
    SLA do not prevent something from breaking, unfortunately. It is just a blame construct.
  - p_l 3y ago
    "Hope" that your cloud provider matches as well your needs as you thought, that vendor lock-in doesn't let them milk you with high prices, etc. etc. None of that is prevented with SLA
    - PUSH_AX 3y ago
      This requires the same skill and experience as figuring out if k8s is going to be a good fit. Arguably if you can’t evaluate the raw cloud offerings and jump on a supposed silver bullet you need to stop immediately.
      - p_l 3y ago
        At this point I found out that k8s knowledge is more portable, whereas your trove of $VENDOR_1 knowledge might suddenly have issues because for reasons outside of your capacity to control there's now big spending contract signed with $VENDOR_2 and a mandate to move. And with smaller companies I tend to find k8s way more cost effective. I pulled things I wouldn't be able to fit in a budget otherwise.
k8sToGo 3y ago
I joined a team that used AWS without kubernetes. Thousands of fragile weird python and bash scripts. Deployment was always such a headache. A few months later I transitioned the team to use containers with proper CI/CD and EKS with Terraform and Argo CD. The team and also the managers like it, since we could deploy quite quickly.
- evantbyrne 3y ago
  This is an apples-to-oranges comparison. You would still have to write and maintain glue without the presence of a proper CD.
- deleted 3y ago
  [deleted]
- PUSH_AX 3y ago
  Thanks for the anecdote k8sToGo

misiti3780 3y ago

if not k8, what would other people be using? ECS?

k8sToGo 3y ago
From my experience, classical VMs with self written Bash scripts. The horror!
kenhwang 3y ago
If you're on AWS, yeah, I'd say just use ECS until you need more complexity. Our ECS deployments have been unproblematic for years now. Our K8s clusters never goes more than a couple days without some sort of strange issue popping up. Arguably it could be because my company outsourced maintenance of it to an army of idiots. But K8s is a tool that is only as good as the operator, and competence can be hard to come by at some companies.
- p_l 3y ago
  K8s or no K8s, outsource to lowest bidder and you'll get unworkable platform :|
  - kenhwang 3y ago
    Agreed. But if you're already on AWS, I'd say the quality floor is already higher than the potential at 95%+ of other companies. So I say unless you're at a company that pays top salaries for the top 5% of engineering talent, you're probably better off just using the AWS provided service.
    - p_l 3y ago
      I used to have a saying back when Heroku was more favourable, is that you use Heroku because you want to go bankrupt. AWS is at times similar. Depending on your local market, AWS bills might be way worse than the cost of few bright ops people who will let you choose from offerings including running dev envs on random assortment of dedicated servers and local e-waste escapees

geodel 3y ago

And that hype is in large part created by Google and other cloud vendors. To be honest I hardly see any reasonable/actionable advice from Cloud/SAAS vendors. Either it is to sell their stuff or generic stuff like "One should be securing / monitoring their stuff running in prod". Oh wow, never thought or done any such thing before.

imiric 3y ago

That might be true, but unfortunately the state of the art infrastructure tooling is mostly centered around k8s. This means that companies choose k8s (or related technologies like k3s, Microk8s, etc.) not because they strictly _need_ k8s, but because it improves their workflows. Otherwise they would need to invest a disproportionate amount of time and effort adopting and maintaining alternative tooling, while getting an inferior experience. Choosing k8s is not just based on scaling requirements anymore. There are also benefits of being compatible with a rich ecosystem of software.

PUSH_AX 3y ago
Can you specify what state of the art infra tooling you mean?
- imiric 3y ago
  Continuous deployment systems like ArgoCD and Flux, user friendly local development environments with tools like Tilt, novel networking, distributed storage, distributed tracing, etc. systems that are basically plug-and-play, etc. Search for "awesome k8s" and you'll get many lists of these. It's surely possible to cobble all of this together without k8s, but k8s' main advantage is exposing a standardized API that simplifies managing this entire ecosystem. It often makes it worth the additional overhead of adopting, understanding and managing k8s itself.

k8sToGo 3y ago

I push for k8s because I know it. Why not use something that I know how to use? I know how to quickly set up a cluster, what to deploy, and teach other team members about fundamentals. How many people out there really need C# or object oriented programming? The argument you present might be valid if you decide to use a tech stack prior having much experience with it.

nprateem 3y ago
Yeah that's the point. You know it and stuff everyone else.
- p_l 3y ago
  Some custom bash/python/ansible monstrosity is only going to be known by few brains in the world. K8s is remarkably easier to retain institutional knowledge as well as spread it.
  - nprateem 3y ago
    If you're expecting app/FE devs to have to learn it you're putting a ton of barriers in their way in terms of deploying. Just chucking a container on a non-k8s managed platform (e.g. Cloud Run) would be much simpler, and no pile of bash scripts.
    - p_l 3y ago
      PaaSes are for companies with money to burn, most of the time. A good k8s team (even a single person, to be quite honest) is going to work towards providing your application teams with simple templates to let them deploy their software easily. Just let them do it. Also, in my experience, you either have to spend ridiculous amounts of money on SaaS/PaaS, or you find that you have to host a lot more than just your application and suddenly the deployment story becomes more complex. Depending on where you are and how much you're willing to burn money, you might find out that k8s experts are cheaper than the money saved by not going PaaS.
    - 3y ago

planetafro 3y ago

Just a thought as well in my corpo experience: Unfortunately, there are some spaces that distribute solutions as k8s-only... Which sucks. I've noticed this mostly in the data science/engineering world. These are solutions that could be easily served up in a small docker compose env. The complexity/upsell/devops BS is strong. To add insult to injury, I've seen more than one use IaC cloud tooling as an install script vs a maintainable and idempotent solution. It's all quite sad really.

p_l 3y ago

There's a difference between need or you don't survive and it improves our operations. The former is a very small set involving having huge amounts of bare metal systems. The latter is suprisingly large set of companies, sometimes even with one server.

Thaxll 3y ago

It's a dumb statement especially from an SRE, it's typically a comment from people that don't understand k8s and think that k8s is only there to have the SLA of Google. For most use case k8s is not there to give you HA but to give you a standard way of deploying a stack, that being on the cloud or on prem.

PUSH_AX 3y ago
He understood it fully, he was running a multi day course on it when I spoke to him. He was candid about the tech, most of us where there at the behest of our orgs.
- p_l 3y ago
  In my personal experience, Google SREs as well as k8s devs sometimes didn't grok how wide k8s usability was - they also can be blind to financial aspects of companies living outside of Silly Valley.

throwawaaarrgh 3y ago

Most companies in the world don't need to develop software. Software development itself is hype. But there's lots of money in it, despite no actual value being created most of the time.

mrj 3y ago

Kubernetes scales down pretty well. I don't use network layers or crazy ingress setups. I keep it simple and Kubernetes works great. What's wonderful is that when I work on multiple clouds, my knowledge transfers just fine. I don't think of the AWS solution or the GCS solution, I use the same kubectl to check out both, view logs, inspect and fix. Even when I got tired of waiting for GKE to spin up a node, running Github actions on a self-hosted microk8s meant instant pod starts and very little fuss. But using Kubernetes meant I got to take advantage of the Github operator, which let me reuse the same machine for multiple builds without the headaches. When I want to run some open source, I often find a helm chart the helps me get set up. Nowadays running open source packages can involve all kinds of dependencies, but getting it running on a k8s cluster to check it out, or even in prod, is a relatively straight forward editing of some values files. I've recently ran Uptrace and Superset that way. They're not a bajillion requests per second setups, they don't have to be, and it was far easier to set up than most methods. I would say your friend is right. Few people _need_ k8s but it's one interface to a bunch of complicated proprietary stuff. I can know core small, core set of k8s tools really well and forget half of the junk that I ever knew about public clouds. It's all the same patterns, transferable and reliable.