5 ms·

Because they already had it running in k8s. And k8s scales very well to very low and high numbers. Because k8s provides battle tested Features like rollout lb

Because they already had it running in k8s. And k8s scales very well to very low and high numbers. Because k8s provides battle tested Features like rollout lb etc. And the ecosystem is great. Certmanager, argocd kube stack. I'm baffled tbh how they had such a difference experience with k8s than I do

aflag 4y ago
Deploying a fixed number of servers to a fixed number of hosts has been battle tested for the past 40+ years. It does work.
- Melingo 4y ago
  It definitely does not. The main issues we faced with over 700VMs were: outdated os, full disks, full inodes, broken hardware, missing backups or missing backup strategy, oom. K8s health itself, fixes out of memory by restarting a pod, solves storage by shipping logs out and killing a pod in case it still runs full, has a rollout startegy, health checks and readiness probes. It provides easy deployment mechanism out of the box, adding a domain is easy, certificates get renewed centrally and automatically. Scaling is just a replica number and you have node Autoupgrade features build in. K8s provides what people build manually out of the box, certified, open sourced and battle tested.
  - vidarh 4y ago
    Deploying a fixed number.of instances to a fixed number of servers does not imply doing it manually.
    - Melingo 4y ago
      And I didn't say that. We had all of these problems with self developed automatisation. It still was garbage. K8s just solves those issues out if the box.
      - prmoustache 4y ago
        Your own failures do not define a model.
        Melingo 4y ago
        And? My experience still counts for something and the example with those 700 VMS is something I didn't just saw once.
        tptacek 4y ago
        Having huge sprawling swarms of VMs is, for some teams, a problem to be solved, not a fact of life to be designed around.
        Melingo 4y ago
        Sry I'm not getting your point. If I understand it right: VMS were not there because people needed VMS they were there because people needed compute. We moved everything to k8s and we were able to do this because k8s can
        tptacek 4y ago
        The point is to deliver a small set of applications, not to come up with the most horizontally scalable possible deployment fabric.
        vidarh 4y ago
        I ran 1000+ VMs on a self developed orchestration mechanism for many years and it was trivial. This isn't a hard problem to solve, though many of the solutions will end up looking similar to some of the decisions made for K8S. E.g. pre-K8S we ran with an overlay network like K8S, and service discovery, like K8S, and an ingress based on Nginx like many K8S installs. There's certainly a reason why K8S looks the way it does, but K8S also has to be generic where you can often reasonably make other choices when you know your specific workload.
  - fxtentacle 4y ago
    Ansible is your friend. Btw, we're talking about the team that built Capistrano, so they certainly know how to automate deployments.
    - Melingo 4y ago
      Nope Ansible is horrible in comparison to k8s. Alone the Paradigma shift from doing things step by step vs describing what you need and than things happen on it is a game changer. K8s is probably 100x easier than Ansible. And Ansible also has it's bigger ecosystem like Ansible tower. Basically your k8s control plane but in bad
      - KronisLV 4y ago
        > Alone the Paradigma shift from doing things step by step vs describing what you need and than things happen on it is a game changer. I've actually used both in conjunction and it was decent: Ansible for managing accounts, directories, installed packages (the stuff you might actually need to run containers and/or an orchestrator), essentially taking care of the "infrastructure" part for on-prem nodes, so that the actual workloads can then be launched as containers. In that mode of work, there was very little imperative about Ansible, for example: - name: Ensure we have a group ansible.builtin.group: name: somegroup gid: 2000 state: present - name: Ensure that we have a user that belongs to the group ansible.builtin.user: name: someuser uid: 3000 shell: /bin/bash groups: somegroup append: yes state: present This can help you setup some monitoring for the nodes themselves, install updates, mess around with any PKI stuff you need to do and so on, everything that you could achieve either manually or by some Bash scripts running through SSH. Better yet, the people who just want to run the containers won't have to think about any of this, so it ensures separation of concerns as well. Deploying apps through Ansible directly can work, but most of the container orchestrators might admittedly be better suited for this, if you are okay with containerized workloads. There, they all shine: Docker Swarm, Hashicorp Nomad, Kubernetes (K3s is really great) and so on...
    - tapoxi 4y ago
      I'm on GKE. The hosts and control plane are managed for me. All I need to do is build/test/security scan images and then promote/deploy the image (via Helm) when it goes out to prod. Using config management and introducing config drift and management of the underlying operating system is a lot more to think about, and a lot more that can go wrong.
  - re-thc 4y ago
    The difference is it's likely possible to have 7 physical servers replace those 700VMs when you have your own hardware without all the overhead. It is much easier to maintain when you look at those numbers.
    - Melingo 4y ago
      Not in my case. Every VM had 4 cores and 20gig me. Run on quite big blades
      - re-thc 4y ago
        Your case is fine. AMD's 4th-Gen EPYC Genoa processors can do up to 192 cores and 384 threads in a single machine (2 socket) with TBs of RAM. In most cloud environments a "core" can be just a thread. Older machines have had 4-CPU socket based chassis with many cores as well. Definitely doable.
prmoustache 4y ago
The main issue is the ecosystem imho. Bare vanilla k8s or k3s is nice but it doesn't do much outside of your homelab. Once you want k8s on production in the cloud you have to start about thinking of: - loadbalancing and ingress controller - storage - network - iam and roles - security groups - centralized logging - registry management - vulnerability scanning - ci/cd - gitops And all this is no less complex with k8s than with nomad, bare docker or whatever they chose. And definitely no less complex because it is on a major cloud provider.
- Melingo 4y ago
  In all managed services all of that comes out of the box. Ingress, lb, storage, network... And I have my small setup running with all of it too. Took me a weekend to set it up. Rke2, nginx ingress, classic lb in front, cert manager and everything else in argocd.
  - 0x500x79 4y ago
    Hey Melingo, I noticed that you responded to a lot of different threads in this post. It seems like you are a bit dismissive of people's experiences using K8s. I have also run K8s at scale, and it is not easy, it is not out of the box in cloud providers. There are a ton of addons, knobs, and work that has to be doen to build a sustainable and "production ready" version of K8s (for my requirements) in AWS. K8s is NOT easy, and I do not believe that in it's current form it is the pinnacle of deployment/orchestration technologies. I am waiting for what is next, because the pain that I have personally experienced around K8s that I know others are feeling as well does not make it a perfect solution for everything, and definitely not usable for others. At the end of the day it's a tool, and it is sometimes difficult to work with.
    - prmoustache 4y ago
      Also when you do a mistake on a key part it can fail in a very spectacular way and it can be tricky to debug the issue immediately. It is usually a game of finding the correct spaghetti(log) in a full plate.
mr_ndrsn 4y ago
We did! And it did work. And there are def some great things that I (we) love about k8s. Personally, the declarative aspect of it was chef's kiss. "I want 2 of these and 3 of these, please", and it just happens. Which is the primary reason why we did investigate k8s on-prem. We had already done the work to k8s-ify the apps, let's not throw that away. But running k8s on-prem is different than running your own k8s in the cloud is different than running on managed k8s in the cloud. Providing all of the bits k8s needs to really work was going to really stretch our team, but we figured with the right support from a vendor, we could make it work. We worked up a spike of harvester + rancher + longhorn and had something that we could use as if it were a cloud. It was pretty slick. Then we got the pricing on support for all of that, and decided to spend that half million elsewhere. We own our hardware, we rent cabs and pay for power & network. We've got a pretty simple pxeboot setup to provision hardware with a bare OS that we can use with chef to provide the common bits needed. It's not 'ultimately flexible in every way', but it's 'flexible enough to meet the needs of our workloads'.
- Bagged2347 4y ago
  What is your position at 37Signals and how do you like it? I'm really impressed by the innovation that comes out of you guys and the workplace culture you folks have.
  - mr_ndrsn 4y ago
    I'm a Lead SRE on the Ops team. We've got a fantastic bunch of folks, they're amazing to work with!

aflag 4y ago

Deploying a fixed number of servers to a fixed number of hosts has been battle tested for the past 40+ years. It does work.

Melingo 4y ago
It definitely does not. The main issues we faced with over 700VMs were: outdated os, full disks, full inodes, broken hardware, missing backups or missing backup strategy, oom. K8s health itself, fixes out of memory by restarting a pod, solves storage by shipping logs out and killing a pod in case it still runs full, has a rollout startegy, health checks and readiness probes. It provides easy deployment mechanism out of the box, adding a domain is easy, certificates get renewed centrally and automatically. Scaling is just a replica number and you have node Autoupgrade features build in. K8s provides what people build manually out of the box, certified, open sourced and battle tested.
- vidarh 4y ago
  Deploying a fixed number.of instances to a fixed number of servers does not imply doing it manually.
  - Melingo 4y ago
    And I didn't say that. We had all of these problems with self developed automatisation. It still was garbage. K8s just solves those issues out if the box.
    - prmoustache 4y ago
      Your own failures do not define a model.
      - Melingo 4y ago
        And? My experience still counts for something and the example with those 700 VMS is something I didn't just saw once.
        tptacek 4y ago
        Having huge sprawling swarms of VMs is, for some teams, a problem to be solved, not a fact of life to be designed around.
        Melingo 4y ago
        Sry I'm not getting your point. If I understand it right: VMS were not there because people needed VMS they were there because people needed compute. We moved everything to k8s and we were able to do this because k8s can
        tptacek 4y ago
        The point is to deliver a small set of applications, not to come up with the most horizontally scalable possible deployment fabric.
        vidarh 4y ago
        I ran 1000+ VMs on a self developed orchestration mechanism for many years and it was trivial. This isn't a hard problem to solve, though many of the solutions will end up looking similar to some of the decisions made for K8S. E.g. pre-K8S we ran with an overlay network like K8S, and service discovery, like K8S, and an ingress based on Nginx like many K8S installs. There's certainly a reason why K8S looks the way it does, but K8S also has to be generic where you can often reasonably make other choices when you know your specific workload.
- fxtentacle 4y ago
  Ansible is your friend. Btw, we're talking about the team that built Capistrano, so they certainly know how to automate deployments.
  - Melingo 4y ago
    Nope Ansible is horrible in comparison to k8s. Alone the Paradigma shift from doing things step by step vs describing what you need and than things happen on it is a game changer. K8s is probably 100x easier than Ansible. And Ansible also has it's bigger ecosystem like Ansible tower. Basically your k8s control plane but in bad
    - KronisLV 4y ago
      > Alone the Paradigma shift from doing things step by step vs describing what you need and than things happen on it is a game changer. I've actually used both in conjunction and it was decent: Ansible for managing accounts, directories, installed packages (the stuff you might actually need to run containers and/or an orchestrator), essentially taking care of the "infrastructure" part for on-prem nodes, so that the actual workloads can then be launched as containers. In that mode of work, there was very little imperative about Ansible, for example: - name: Ensure we have a group ansible.builtin.group: name: somegroup gid: 2000 state: present - name: Ensure that we have a user that belongs to the group ansible.builtin.user: name: someuser uid: 3000 shell: /bin/bash groups: somegroup append: yes state: present This can help you setup some monitoring for the nodes themselves, install updates, mess around with any PKI stuff you need to do and so on, everything that you could achieve either manually or by some Bash scripts running through SSH. Better yet, the people who just want to run the containers won't have to think about any of this, so it ensures separation of concerns as well. Deploying apps through Ansible directly can work, but most of the container orchestrators might admittedly be better suited for this, if you are okay with containerized workloads. There, they all shine: Docker Swarm, Hashicorp Nomad, Kubernetes (K3s is really great) and so on...
  - tapoxi 4y ago
    I'm on GKE. The hosts and control plane are managed for me. All I need to do is build/test/security scan images and then promote/deploy the image (via Helm) when it goes out to prod. Using config management and introducing config drift and management of the underlying operating system is a lot more to think about, and a lot more that can go wrong.
- re-thc 4y ago
  The difference is it's likely possible to have 7 physical servers replace those 700VMs when you have your own hardware without all the overhead. It is much easier to maintain when you look at those numbers.
  - Melingo 4y ago
    Not in my case. Every VM had 4 cores and 20gig me. Run on quite big blades
    - re-thc 4y ago
      Your case is fine. AMD's 4th-Gen EPYC Genoa processors can do up to 192 cores and 384 threads in a single machine (2 socket) with TBs of RAM. In most cloud environments a "core" can be just a thread. Older machines have had 4-CPU socket based chassis with many cores as well. Definitely doable.

prmoustache 4y ago

The main issue is the ecosystem imho. Bare vanilla k8s or k3s is nice but it doesn't do much outside of your homelab. Once you want k8s on production in the cloud you have to start about thinking of: - loadbalancing and ingress controller - storage - network - iam and roles - security groups - centralized logging - registry management - vulnerability scanning - ci/cd - gitops And all this is no less complex with k8s than with nomad, bare docker or whatever they chose. And definitely no less complex because it is on a major cloud provider.

Melingo 4y ago
In all managed services all of that comes out of the box. Ingress, lb, storage, network... And I have my small setup running with all of it too. Took me a weekend to set it up. Rke2, nginx ingress, classic lb in front, cert manager and everything else in argocd.
- 0x500x79 4y ago
  Hey Melingo, I noticed that you responded to a lot of different threads in this post. It seems like you are a bit dismissive of people's experiences using K8s. I have also run K8s at scale, and it is not easy, it is not out of the box in cloud providers. There are a ton of addons, knobs, and work that has to be doen to build a sustainable and "production ready" version of K8s (for my requirements) in AWS. K8s is NOT easy, and I do not believe that in it's current form it is the pinnacle of deployment/orchestration technologies. I am waiting for what is next, because the pain that I have personally experienced around K8s that I know others are feeling as well does not make it a perfect solution for everything, and definitely not usable for others. At the end of the day it's a tool, and it is sometimes difficult to work with.
  - prmoustache 4y ago
    Also when you do a mistake on a key part it can fail in a very spectacular way and it can be tricky to debug the issue immediately. It is usually a game of finding the correct spaghetti(log) in a full plate.

mr_ndrsn 4y ago

We did! And it did work. And there are def some great things that I (we) love about k8s. Personally, the declarative aspect of it was chef's kiss. "I want 2 of these and 3 of these, please", and it just happens. Which is the primary reason why we did investigate k8s on-prem. We had already done the work to k8s-ify the apps, let's not throw that away. But running k8s on-prem is different than running your own k8s in the cloud is different than running on managed k8s in the cloud. Providing all of the bits k8s needs to really work was going to really stretch our team, but we figured with the right support from a vendor, we could make it work. We worked up a spike of harvester + rancher + longhorn and had something that we could use as if it were a cloud. It was pretty slick. Then we got the pricing on support for all of that, and decided to spend that half million elsewhere. We own our hardware, we rent cabs and pay for power & network. We've got a pretty simple pxeboot setup to provision hardware with a bare OS that we can use with chef to provide the common bits needed. It's not 'ultimately flexible in every way', but it's 'flexible enough to meet the needs of our workloads'.

Bagged2347 4y ago
What is your position at 37Signals and how do you like it? I'm really impressed by the innovation that comes out of you guys and the workplace culture you folks have.
- mr_ndrsn 4y ago
  I'm a Lead SRE on the Ops team. We've got a fantastic bunch of folks, they're amazing to work with!