3 ms·

I don't follow where you're trying to go here, or how you intend this to contradict anything I said. You're right that most people don't run 100k servers. In

by tene 5y ago

I don't follow where you're trying to go here, or how you intend this to contradict anything I said. You're right that most people don't run 100k servers. In fact, most people don't run any servers at all, and don't use kubernetes. Some people do, and some of those people use approaches like this. The comment I was responding to, by my reading, seems to express incredulity that anyone would do this, and seems to imply that this is a bad, counterproductive idea that nobody should ever implement. The most notable quote is "I feel like I landed in crazy-land". Do you read it differently from me? My intent in my reply was to describe some of the use-cases where this is helpful. It's not only FAANG that run more than a small handful of clusters; there are plenty of smaller businesses that need to run large numbers of clusters. I don't know anything about the company that wrote this article, but I found https://zitadel.ch/usecases https://zitadel.ch/usecases on their website, and given that description, it sounds quite plausible that they run many small clusters in many clouds and datacenters. They probably don't need a ton of compute, but I wouldn't be surprised if they had latency needs for being close to their customers, or other specialized requirements that motivate dedicated clusters. I also don't know anything about the insurance industry, so I'm curious to hear if I've guessed incorrectly, but it doesn't seem like the kind of business with especially high compute needs. I believe you when you say that your company has no use for this. Can you believe me when I say that there are genuine problems these systems are trying to address? I keep hearing about companies who fund their employees building unnecessary pointless over-engineered production automation, but I have yet to ever encounter one in real life. Every SRE job I've had so far has been on a team that really wanted to invest time in improving infrastructure and automation, but couldn't get time away from the toil for it. If anyone can recommend a company that over-invests in infrastructure automation and architecture instead of under-invests, I'd dearly love to try it and see if it's as bad as I've heard. Even without high compute needs or large numbers of clusters, there's still a lot of benefits you can get when you can move things to declarative configurations and away from global mutable state and human minds. I agree that this can go wrong, but that doesn't mean it's categorically worthless to try. On the other hand, if you're just looking to gripe about it having gone wrong for you, could you share some war stories? I may be feeling idealistic from frustration with low investment in tools support and production automation lately, so maybe I could use some horror stories of it going badly to scare me straight. :)

KronisLV 5y ago
I think that the contradiction here comes from the fact that these tools that are suited for large scale operations, like Kubernetes, end up getting standartized and adopted even by smaller corporations, which have neither the specialists, nor the resources to utilize them properly. Be it because of FOMO (fear of missing out), CV driven development or something else entirely, but i've seen this a number of times in the industry and it's always gone poorly. Instead of relatively quick deployments with Docker Swarm, Hashicorp Nomad, Docker Compose or anything of the sort, it suddenly becomes an uphill battle of trying to administer the darn cluster, as opposed to just being able to develop software, even with turnkey solutions like Rancher ( https://rancher.com/ https://rancher.com/ which is great, by the way), especially if the company has only recently adopted Kubernetes. In contrast, i think that Docker Swarm does a much better job at smaller scales, because: - it uses way less resources than Kubernetes (which matters on smaller nodes) - it is included in the install of Docker and therefore is easy to launch - it supports the Compose format, which is often used for development with Docker Compose - it is relatively simple, yet handles most of the concerns of smaller scale projects (i.e. excluding things like autoscaling or CRDs) - there are solutions to manage it with web interfaces as well, like Portainer, if necessary ( https://www.portainer.io/ ) Or, if you need Kubernetes in a non-managed format, i'd suggest that anyone take a look at projects like K3s ( https://k3s.io/ https://k3s.io/ ), which attempt to cut out all of the unnecessary components and features of Kubernetes, that most people won't use. In my opinion, going with Kubernetes as a whole or even full Kubernetes distros (or building your own clusters for prod) is akin to choosing Apache Kafka because you want to build an event driven system, and then needing to throw a team at the thing to actually keep it running smoothly, instead of just choosing something simpler, like RabbitMQ or ZeroMQ. I'd argue that most companies out there don't even have "SRE jobs", but instead it's a duty that gets thrown upon regular devs who also need to deliver features. It's probably easy to overestimate how capable most companies out there are in regards to throwing resources at problems to make them disappear, mostly due to a lack of said resources.
- marcus_holmes 5y ago
  Startups are insisting on K8s experience despite targeting a market that will never need more than a single server if they write the application properly. I look at "if you've got 100K servers..." and ask "why does anyone needs 100K servers?", rather than "how would I manage that?". Each layer adds complexity and reduces performance. Less layers are better. Write Unikernel applications

by tene 5y ago

KronisLV 5y ago

I think that the contradiction here comes from the fact that these tools that are suited for large scale operations, like Kubernetes, end up getting standartized and adopted even by smaller corporations, which have neither the specialists, nor the resources to utilize them properly. Be it because of FOMO (fear of missing out), CV driven development or something else entirely, but i've seen this a number of times in the industry and it's always gone poorly. Instead of relatively quick deployments with Docker Swarm, Hashicorp Nomad, Docker Compose or anything of the sort, it suddenly becomes an uphill battle of trying to administer the darn cluster, as opposed to just being able to develop software, even with turnkey solutions like Rancher ( https://rancher.com/ https://rancher.com/ which is great, by the way), especially if the company has only recently adopted Kubernetes. In contrast, i think that Docker Swarm does a much better job at smaller scales, because: - it uses way less resources than Kubernetes (which matters on smaller nodes) - it is included in the install of Docker and therefore is easy to launch - it supports the Compose format, which is often used for development with Docker Compose - it is relatively simple, yet handles most of the concerns of smaller scale projects (i.e. excluding things like autoscaling or CRDs) - there are solutions to manage it with web interfaces as well, like Portainer, if necessary ( https://www.portainer.io/ ) Or, if you need Kubernetes in a non-managed format, i'd suggest that anyone take a look at projects like K3s ( https://k3s.io/ https://k3s.io/ ), which attempt to cut out all of the unnecessary components and features of Kubernetes, that most people won't use. In my opinion, going with Kubernetes as a whole or even full Kubernetes distros (or building your own clusters for prod) is akin to choosing Apache Kafka because you want to build an event driven system, and then needing to throw a team at the thing to actually keep it running smoothly, instead of just choosing something simpler, like RabbitMQ or ZeroMQ. I'd argue that most companies out there don't even have "SRE jobs", but instead it's a duty that gets thrown upon regular devs who also need to deliver features. It's probably easy to overestimate how capable most companies out there are in regards to throwing resources at problems to make them disappear, mostly due to a lack of said resources.

marcus_holmes 5y ago
Startups are insisting on K8s experience despite targeting a market that will never need more than a single server if they write the application properly. I look at "if you've got 100K servers..." and ask "why does anyone needs 100K servers?", rather than "how would I manage that?". Each layer adds complexity and reduces performance. Less layers are better. Write Unikernel applications