7 ms·

Only one sentence about why they chose to abandon K8s: > It all sounded like a win-win situation, but [on-prem kubernetes] turned out to be a very expensive an

Only one sentence about why they chose to abandon K8s: > It all sounded like a win-win situation, but [on-prem kubernetes] turned out to be a very expensive and operationally complicated idea, so we had to get back to the drawing board pretty soon. It was very expensive and operationally complicated to self-host k8s, so they decided to build their own orchestration tooling? Sort of undercuts their main argument that this bit isn't even remotely fleshed out.

benatkin 4y ago
Well to be fair Kubernetes doesn't always pluralize the names of collections, since you can run "kubectl get deployment/myapp". You don't want to do the equivalent of "select * from user" do you? That doesn't make any sense!!! And don't translate that to "get all the records from the user table"! That's "get all the records from the users table". (Rails defaults to plural, Django to singular for table names. Not sure about the equivalent for Kubernetes but in the CLI surprisingly you can use either)
mike1o1 4y ago
To be fair, from the article it says that they built the bulk of the tool and did the first migration in a 6-week cycle. mrsk looks fairly straight forward, and feels like Capistrano but for containers. The first commit of mrsk is only on January 7th of this year. In less than a six-week cycle, we built those operational foundations, shaped mrsk to its functional form and had Tadalist running in production on our own hardware.
- bastawhiz 4y ago
  They spent a month and a half building tooling _capable of handling their smallest application_, representing an extremely tiny fraction of their cloud usage.
imiric 4y ago
Sometimes there's value in building bespoke solutions. If you don't need many of the features of the off-the-shelf solution, and find the complexity overwhelming and the knowledge and operational costs too high, then building a purpose-built solution to fit your use case exactly can be very beneficial. You do need lots of expertise and relatively simple applications to replace something like k8s, but 37signals seems up to the task, and judging by the article, they picked their least critical apps to start with. It sounds like a success story so far. Kudos to them for releasing MRSK, it definitely looks interesting. As a side note, I've become disgruntled at k8s becoming the defacto standard for deploying services at scale. We need different approaches to container orchestration, that do things differently (perhaps even rethinking containers!), and focus on simplicity and usability instead of just hyper scalability, which many projects don't need. I was a fan of Docker Swarm for a long time, and still use it at home, but I wouldn't dare recommend it professionally anymore. Especially with the current way Docker Inc. is managed.
- skor 4y ago
  > wouldn't dare recommend it professionally anymore please provide more insight, thanks
  - worldsayshi 4y ago
    It seems a bit confusing. Here's some context: https://github.com/docker/roadmap/issues/175 https://github.com/docker/roadmap/issues/175
    - jacooper 4y ago
      Doesn't seem to change anything though? Swarm is still alive and well?
      - worldsayshi 4y ago
        Yeah I'm a bit surprised to hear that. I had only heard a lot of teams giving up swarm when it was deprecated. Didn't know they just restructured the project.
        jacooper 4y ago
        Yup its crappy communication from docker, the ticket is asking for clarity to put an end to the swarm is dead meme. Even mirantis is back supporting swarmkit and adding new features.
- kobalsky 4y ago
  > and focus on simplicity and usability instead of just hyper scalability I don't get why people in IT need a boogeyman. Looks like "k8s is hard" is the new "pointers are hard" we use to scare other people now.
  - imiric 4y ago
    That's not a great comparison, but it works in a sense. Not all languages and applications benefit from pointers. The issue is not about k8s being hard. Yes, it has a steep learning curve, but many technologies do. The issue is that learning all of its intricacies, and maintaining it and the services that run on top of it, requires valuable resources many companies don't have, especially early on. And the benefits it provides are, for the most part, not needed early on in a project's lifecycle, and often never. In financial terms, it's very high risk, with low ROI. If there's a solution that lowers the investment and maintenance costs, while being valuable in the short and long term, then that's generally a more favorable solution for most projects that don't operate at Google's scale.
oxfordmale 4y ago
It seems they are migrating for migration's sake. AWS ECS -> Google GKE -> AWS EKS -> Self hosted Kubernetes -> bespoke solution. They are pretty much changing their tech stack every few years, never maturing their knowledge of their current tech stack. I am sure it makes sense to self host, but given their track record I am wondering if they wil be migrating to the cloud again in a few years.
- axlee 4y ago
  To be fair that served them well in the past: the reason why anyone knows about 37signals is because they reinvented the wheel back in 2004 with Rails, but what a great reinvention it was. Who knows what can come next.
  - atonse 4y ago
    Which wheel did they reinvent? Rails literally set a bunch of standards used by just about every framework today… app generators, conventions over configuration, asset pipelines, you name it.
    - pxc 4y ago
      I think what they wanted to convey was not the redundancy of 'reinventing the wheel', but the ambitious scope and from-scratch approach associated with the phrase. Maybe 'rolled their own' or 'first invented the universe' would have been slightly better.
      - dexterdog 4y ago
        But does doing something like that once 20 years ago mean they can do it again?
        pxc 4y ago
        I dunno. I was a kid when Rails took over the world, so I couldn't even begin to tell you why it succeeded in the way that it did. But I do feel like they probably know what they're doing enough to have a more modest version of success with this other project, i.e., meeting their own needs well without burning up too much money or time. They're still a really small, focused company, and they have a lot of relevant experience.
wankle 4y ago
k3s is lightweight and even I have clusters running, I can sync them too if I wish, easily, I agree, it seems odd they didn't go with some kube design on prem.
jameshart 4y ago
We are talking about 37Signals here. This is the company that, when faced with the problem of making a shared to-do list application, created Ruby on Rails. And when they decided to write up their remote working policy, published a New York Times bestselling business book. This is not a company that merely shaves its Yaks. It offers a full menu of Yak barber services, and then launches a line of successful Yak grooming products.
- madeofpalk 4y ago
  ...and don't forget the time where they wrote a blog post, kept posting through it, and a significant amount of the company quit.
  - ubercore 4y ago
    I do forget that time -- what's the context?
    - madeofpalk 4y ago
      https://www.theverge.com/2021/5/4/22419512/basecamp-political-speech-policy-fallout https://www.theverge.com/2021/5/4/22419512/basecamp-politica...
  - dangwhy 4y ago
    > significant amount of the company quit. no they didn't
    - mr_ndrsn 4y ago
      Yes, they did. This is not a debatable fact. IIRC, it was 30%+ of the company.
      - dangwhy 4y ago
        i am sure you will supply proof for your claims.
crabbone 4y ago
> It was very expensive and operationally complicated to self-host k8s, so they decided to build their own orchestration tooling? You are deeply misunderstanding Kubernetes if you think it's some sort of a turnkey solution that solves all your infrastructure problems. Virtually everything of value in Kubernetes isn't Kubernetes -- you have to add it on later, and manage it yourself. Container runtime? -- that's not Kubernetes. Database to store deployment info? -- that's not Kubernetes. Network setup and management? -- that's not Kubernetes. Storage setup and management? -- still not Kubernetes. When you start using Kubernetes for real, you will end up replacing almost every component it brings by default with something else. CoreDNS? -- sucks for big systems. Volumes? You aren't going to be using volumes from local filesystem... that's insane! You'll probably set up Ceph or something like that and add some operators to help you use it. Permission management? -- Well, you are out of luck in a major way here... you have, basically, Kyverno, but it really, really sucks (and it's still not Kubernetes!). Real-life Kubernetes deployments end up being very poorly stitched together piles of different components. So much so that you start wishing you'd never touched that thing because a huge fraction of the stuff you now need to manage is integration with Kubernetes on top of the functionality provided by these components.
- benatkin 4y ago
  > You are deeply misunderstanding Kubernetes if you think it's some sort of a turnkey solution that solves all your infrastructure problems. Virtually everything of value in Kubernetes isn't Kubernetes -- you have to add it on later, and manage it yourself. Container runtime? -- that's not Kubernetes. Database to store deployment info? -- that's not Kubernetes. Network setup and management? -- that's not Kubernetes. Storage setup and management? -- still not Kubernetes. When you install Kubernetes, you get a container runtime. That's a distribution I guess. Part of this seems like GNU/Linux. The other stuff you're listing isn't solved by MRSK either...
  - Jnr 4y ago
    But seems like with mrsk you do not need that other stuff. With Kubernetes you highly likely have to deal with most of that stuff.
tptacek 4y ago
The whole second half of the article is about why they decided to stop using K8s.
danmaz74 4y ago
Whatever their reasons, I'm very happy to be able to use a much simpler alternative to K8s to orchestrate my apps on bare metal servers.
Jnr 4y ago
k8s is an industry standard currently, but it is not great. The lack of available free/open tooling to set up and manage them (the cluster) properly seems to indicate that it is also a way of selling it (cloud). Meaning that if you want to use k8s you have to go with the large cloud providers, otherwise your life will be painful. I for one am patiently waiting for more innovation in this area and seeing that there are companies that try to disrupt/improve it makes me hopeful and I appreciate it.

benatkin 4y ago

Well to be fair Kubernetes doesn't always pluralize the names of collections, since you can run "kubectl get deployment/myapp". You don't want to do the equivalent of "select * from user" do you? That doesn't make any sense!!! And don't translate that to "get all the records from the user table"! That's "get all the records from the users table". (Rails defaults to plural, Django to singular for table names. Not sure about the equivalent for Kubernetes but in the CLI surprisingly you can use either)

mike1o1 4y ago

To be fair, from the article it says that they built the bulk of the tool and did the first migration in a 6-week cycle. mrsk looks fairly straight forward, and feels like Capistrano but for containers. The first commit of mrsk is only on January 7th of this year. In less than a six-week cycle, we built those operational foundations, shaped mrsk to its functional form and had Tadalist running in production on our own hardware.

bastawhiz 4y ago
They spent a month and a half building tooling _capable of handling their smallest application_, representing an extremely tiny fraction of their cloud usage.

imiric 4y ago

Sometimes there's value in building bespoke solutions. If you don't need many of the features of the off-the-shelf solution, and find the complexity overwhelming and the knowledge and operational costs too high, then building a purpose-built solution to fit your use case exactly can be very beneficial. You do need lots of expertise and relatively simple applications to replace something like k8s, but 37signals seems up to the task, and judging by the article, they picked their least critical apps to start with. It sounds like a success story so far. Kudos to them for releasing MRSK, it definitely looks interesting. As a side note, I've become disgruntled at k8s becoming the defacto standard for deploying services at scale. We need different approaches to container orchestration, that do things differently (perhaps even rethinking containers!), and focus on simplicity and usability instead of just hyper scalability, which many projects don't need. I was a fan of Docker Swarm for a long time, and still use it at home, but I wouldn't dare recommend it professionally anymore. Especially with the current way Docker Inc. is managed.

skor 4y ago
> wouldn't dare recommend it professionally anymore please provide more insight, thanks
- worldsayshi 4y ago
  It seems a bit confusing. Here's some context: https://github.com/docker/roadmap/issues/175 https://github.com/docker/roadmap/issues/175
  - jacooper 4y ago
    Doesn't seem to change anything though? Swarm is still alive and well?
    - worldsayshi 4y ago
      Yeah I'm a bit surprised to hear that. I had only heard a lot of teams giving up swarm when it was deprecated. Didn't know they just restructured the project.
      - jacooper 4y ago
        Yup its crappy communication from docker, the ticket is asking for clarity to put an end to the swarm is dead meme. Even mirantis is back supporting swarmkit and adding new features.
kobalsky 4y ago
> and focus on simplicity and usability instead of just hyper scalability I don't get why people in IT need a boogeyman. Looks like "k8s is hard" is the new "pointers are hard" we use to scare other people now.
- imiric 4y ago
  That's not a great comparison, but it works in a sense. Not all languages and applications benefit from pointers. The issue is not about k8s being hard. Yes, it has a steep learning curve, but many technologies do. The issue is that learning all of its intricacies, and maintaining it and the services that run on top of it, requires valuable resources many companies don't have, especially early on. And the benefits it provides are, for the most part, not needed early on in a project's lifecycle, and often never. In financial terms, it's very high risk, with low ROI. If there's a solution that lowers the investment and maintenance costs, while being valuable in the short and long term, then that's generally a more favorable solution for most projects that don't operate at Google's scale.

oxfordmale 4y ago

It seems they are migrating for migration's sake. AWS ECS -> Google GKE -> AWS EKS -> Self hosted Kubernetes -> bespoke solution. They are pretty much changing their tech stack every few years, never maturing their knowledge of their current tech stack. I am sure it makes sense to self host, but given their track record I am wondering if they wil be migrating to the cloud again in a few years.

axlee 4y ago
To be fair that served them well in the past: the reason why anyone knows about 37signals is because they reinvented the wheel back in 2004 with Rails, but what a great reinvention it was. Who knows what can come next.
- atonse 4y ago
  Which wheel did they reinvent? Rails literally set a bunch of standards used by just about every framework today… app generators, conventions over configuration, asset pipelines, you name it.
  - pxc 4y ago
    I think what they wanted to convey was not the redundancy of 'reinventing the wheel', but the ambitious scope and from-scratch approach associated with the phrase. Maybe 'rolled their own' or 'first invented the universe' would have been slightly better.
    - dexterdog 4y ago
      But does doing something like that once 20 years ago mean they can do it again?
      - pxc 4y ago
        I dunno. I was a kid when Rails took over the world, so I couldn't even begin to tell you why it succeeded in the way that it did. But I do feel like they probably know what they're doing enough to have a more modest version of success with this other project, i.e., meeting their own needs well without burning up too much money or time. They're still a really small, focused company, and they have a lot of relevant experience.

wankle 4y ago

k3s is lightweight and even I have clusters running, I can sync them too if I wish, easily, I agree, it seems odd they didn't go with some kube design on prem.

jameshart 4y ago

We are talking about 37Signals here. This is the company that, when faced with the problem of making a shared to-do list application, created Ruby on Rails. And when they decided to write up their remote working policy, published a New York Times bestselling business book. This is not a company that merely shaves its Yaks. It offers a full menu of Yak barber services, and then launches a line of successful Yak grooming products.

madeofpalk 4y ago
...and don't forget the time where they wrote a blog post, kept posting through it, and a significant amount of the company quit.
- ubercore 4y ago
  I do forget that time -- what's the context?
  - madeofpalk 4y ago
    https://www.theverge.com/2021/5/4/22419512/basecamp-political-speech-policy-fallout https://www.theverge.com/2021/5/4/22419512/basecamp-politica...
- dangwhy 4y ago
  > significant amount of the company quit. no they didn't
  - mr_ndrsn 4y ago
    Yes, they did. This is not a debatable fact. IIRC, it was 30%+ of the company.
    - dangwhy 4y ago
      i am sure you will supply proof for your claims.

crabbone 4y ago

> It was very expensive and operationally complicated to self-host k8s, so they decided to build their own orchestration tooling? You are deeply misunderstanding Kubernetes if you think it's some sort of a turnkey solution that solves all your infrastructure problems. Virtually everything of value in Kubernetes isn't Kubernetes -- you have to add it on later, and manage it yourself. Container runtime? -- that's not Kubernetes. Database to store deployment info? -- that's not Kubernetes. Network setup and management? -- that's not Kubernetes. Storage setup and management? -- still not Kubernetes. When you start using Kubernetes for real, you will end up replacing almost every component it brings by default with something else. CoreDNS? -- sucks for big systems. Volumes? You aren't going to be using volumes from local filesystem... that's insane! You'll probably set up Ceph or something like that and add some operators to help you use it. Permission management? -- Well, you are out of luck in a major way here... you have, basically, Kyverno, but it really, really sucks (and it's still not Kubernetes!). Real-life Kubernetes deployments end up being very poorly stitched together piles of different components. So much so that you start wishing you'd never touched that thing because a huge fraction of the stuff you now need to manage is integration with Kubernetes on top of the functionality provided by these components.

benatkin 4y ago
> You are deeply misunderstanding Kubernetes if you think it's some sort of a turnkey solution that solves all your infrastructure problems. Virtually everything of value in Kubernetes isn't Kubernetes -- you have to add it on later, and manage it yourself. Container runtime? -- that's not Kubernetes. Database to store deployment info? -- that's not Kubernetes. Network setup and management? -- that's not Kubernetes. Storage setup and management? -- still not Kubernetes. When you install Kubernetes, you get a container runtime. That's a distribution I guess. Part of this seems like GNU/Linux. The other stuff you're listing isn't solved by MRSK either...
- Jnr 4y ago
  But seems like with mrsk you do not need that other stuff. With Kubernetes you highly likely have to deal with most of that stuff.

tptacek 4y ago

The whole second half of the article is about why they decided to stop using K8s.

danmaz74 4y ago

Whatever their reasons, I'm very happy to be able to use a much simpler alternative to K8s to orchestrate my apps on bare metal servers.

Jnr 4y ago

k8s is an industry standard currently, but it is not great. The lack of available free/open tooling to set up and manage them (the cluster) properly seems to indicate that it is also a way of selling it (cloud). Meaning that if you want to use k8s you have to go with the large cloud providers, otherwise your life will be painful. I for one am patiently waiting for more innovation in this area and seeing that there are companies that try to disrupt/improve it makes me hopeful and I appreciate it.