4 ms·

I think this sort of project is in short supply. Kubernetes seems to have sucked all the air out of the room, and yet we need variety and different approaches

I think this sort of project is in short supply. Kubernetes seems to have sucked all the air out of the room, and yet we need variety and different approaches to grow. I look forward to (trying) to replicate something similar for my own understanding. Any architecture docs will be gratefully recveied

0xbadcafebee 5y ago
I think the architecture of something like K8s pretty well sums up all (or most) of the components needed for a distributed system. What's really needed today is differentiation on the actual end result of that design. We need simpler technology that works better, and that's hard; it's not enough to have a fancy idea and throw together some lines of code. That's been done, and it kinda sucks. What's needed now is to focus on making it easier and simpler to do specific things. For example: most people just want to receive infinite HTTP requests and route them to custom business logic, automatically scaling as needed. So why can't I just push a button and make that happen? AWS ECS has a command-line client now that attempts to do that, but it's clunky and unreliable. And K8s doesn't have anything like that; it's all hand-crafted YAML files and mountains of custom cluster configuration. I just want to run some code at scale. And we've implemented that (scaling web requests/business logic) a million times by now. But it's never really gotten significantly easier. Docker is a good example of the amount of work you need to pour into a technical solution to make it a game-changing user solution. Docker is like 50 different technical solutions rolled up into one user-friendly command-line tool and a backend daemon. You could do most of what Docker does in LXC, but it was painful. Docker made it easy. That's what we need for running services at scale in general. Docker Swarm was created basically to scratch that itch, but it still hasn't really taken off. I'm hopeful it does one day.
- sofixa 5y ago
  > Docker Swarm was created basically to scratch that itch, but it still hasn't really taken off. I'm hopeful it does one day. It never will, that ship sailed a long time ago. Docker the company is struggling to survive after making wrong bets multiple times. The "enterprise" part, including Swarm, was sold off to Mirantis after years of neglect, and got some much needed investment and big fixes. Nonetheless, Docker Swarm is basically abandonware with terrible reputation without a viable future. Hashicorp's Nomad is much better placed to compete with Kubernetes. Shameless plug of my article on the matter: https://atodorov.me/2021/02/27/why-you-should-take-a-look-at-nomad-before-jumping-on-kubernetes/ https://atodorov.me/2021/02/27/why-you-should-take-a-look-at...
  - scns 5y ago
    Thanks for your article, the first pic made me and my colleague laugh really hard.
  - jokethrowaway 5y ago
    I moved from nomad to swarm recently (this is running a proxy + dbs + a few replicas of a node app + tons of long running CPU intensive jobs spanning 10 servers). A few weird api decisions but I found the developer's workflow to be better. I got rid of some weird allocation behaviours I couldn't explain with nomad (the real motivator to get off nomad). I also don't have to install anything.
  - neatze 5y ago
    > Docker Swarm is basically abandonware with terrible reputation. Can you provide source for terrible reputation.
- samsquire 5y ago
  I agree with you. Great comment. I just want to write some code that can scale from zero to infinity. Cluster or deployment Autoscaling in Kubernetes is not exactly what I want. Kubernetes gets some abstractions right - such as pods, statefulsets, ingress, deployments What we now need is abstraction of endpoints, routes, request, response type/schema (to do automatic marshalling), id space sharding like vitess or logical server identifiers as part of the guid for what server the data belongs to..microservice becomes a bucket of routes/rpc calls not hardcoded to a microservice. Envoy, linkerd and hystrix look like a nightmare to configure right. I think the configuration behind them could be more elegant. The software is probably fine but it's a nightmare to get the right data structure that does what you want - hence the mountains of YAML configuration like you mentioned. I don't want to care if it's the storage layer or the application layer - it should all be scalable. I want to be able to set a maximum budget in financial terms and open ended autoscaling ceilling.

0xbadcafebee 5y ago

I think the architecture of something like K8s pretty well sums up all (or most) of the components needed for a distributed system. What's really needed today is differentiation on the actual end result of that design. We need simpler technology that works better, and that's hard; it's not enough to have a fancy idea and throw together some lines of code. That's been done, and it kinda sucks. What's needed now is to focus on making it easier and simpler to do specific things. For example: most people just want to receive infinite HTTP requests and route them to custom business logic, automatically scaling as needed. So why can't I just push a button and make that happen? AWS ECS has a command-line client now that attempts to do that, but it's clunky and unreliable. And K8s doesn't have anything like that; it's all hand-crafted YAML files and mountains of custom cluster configuration. I just want to run some code at scale. And we've implemented that (scaling web requests/business logic) a million times by now. But it's never really gotten significantly easier. Docker is a good example of the amount of work you need to pour into a technical solution to make it a game-changing user solution. Docker is like 50 different technical solutions rolled up into one user-friendly command-line tool and a backend daemon. You could do most of what Docker does in LXC, but it was painful. Docker made it easy. That's what we need for running services at scale in general. Docker Swarm was created basically to scratch that itch, but it still hasn't really taken off. I'm hopeful it does one day.

sofixa 5y ago
> Docker Swarm was created basically to scratch that itch, but it still hasn't really taken off. I'm hopeful it does one day. It never will, that ship sailed a long time ago. Docker the company is struggling to survive after making wrong bets multiple times. The "enterprise" part, including Swarm, was sold off to Mirantis after years of neglect, and got some much needed investment and big fixes. Nonetheless, Docker Swarm is basically abandonware with terrible reputation without a viable future. Hashicorp's Nomad is much better placed to compete with Kubernetes. Shameless plug of my article on the matter: https://atodorov.me/2021/02/27/why-you-should-take-a-look-at-nomad-before-jumping-on-kubernetes/ https://atodorov.me/2021/02/27/why-you-should-take-a-look-at...
- scns 5y ago
  Thanks for your article, the first pic made me and my colleague laugh really hard.
- jokethrowaway 5y ago
  I moved from nomad to swarm recently (this is running a proxy + dbs + a few replicas of a node app + tons of long running CPU intensive jobs spanning 10 servers). A few weird api decisions but I found the developer's workflow to be better. I got rid of some weird allocation behaviours I couldn't explain with nomad (the real motivator to get off nomad). I also don't have to install anything.
- neatze 5y ago
  > Docker Swarm is basically abandonware with terrible reputation. Can you provide source for terrible reputation.
samsquire 5y ago
I agree with you. Great comment. I just want to write some code that can scale from zero to infinity. Cluster or deployment Autoscaling in Kubernetes is not exactly what I want. Kubernetes gets some abstractions right - such as pods, statefulsets, ingress, deployments What we now need is abstraction of endpoints, routes, request, response type/schema (to do automatic marshalling), id space sharding like vitess or logical server identifiers as part of the guid for what server the data belongs to..microservice becomes a bucket of routes/rpc calls not hardcoded to a microservice. Envoy, linkerd and hystrix look like a nightmare to configure right. I think the configuration behind them could be more elegant. The software is probably fine but it's a nightmare to get the right data structure that does what you want - hence the mountains of YAML configuration like you mentioned. I don't want to care if it's the storage layer or the application layer - it should all be scalable. I want to be able to set a maximum budget in financial terms and open ended autoscaling ceilling.