6 ms·

Yes, they did, but did it really take less effort than running it on Kubernetes?

by 988747 4y ago

lowercased 4y ago
Yes.
morelisp 4y ago
Yes.
deterministic 4y ago
Yep. Still doing it today. Very large scale Enterprise systems with complex multi-country/multi-organisational operational rules running 24/7. Millions of lines of code. No Kubernetes. No micro-services. No BS. It’s simple. It works. And it has worked for 30+ years.
nosequel 4y ago
Yes. Much.
jamwt 4y ago
Yes, way less effort
nailer 4y ago
Yes.
athoscouto 4y ago
I'm going to diverge from sibling comments: it depends. As the article points out, k8s may really simplify deploys for devs, while giving the autonomy. But it isn't always worth it.
winrid 4y ago
Yeah... at small to medium scale anyway.
majormajor 4y ago
Definitely if they weren't running very complex services or their business tolerated an occasional maintenance window. The particular way Kubernetes can bite you is that it makes it much easy to start with far more complex setups - not necessarily much harder than to start with a simple setup! - but then you have to maintain and operate those complex setups forever, even if you don't need them that much. If you're growing your team and your services, having a much bigger, more complicated toolbox for people to reach into on day 1 gives you way more chance of building expensive painful tech debt.
- morelisp 4y ago
  The operational value of ~weekly ~30 min "maintenance windows" is wildly underestimated by teams today, and its business cost is wildly overestimated.
- suralind 4y ago
  I think it may appear so because Kubernetes promotes good practices. Do logging, do metrics, do traces. That list quickly grows and while these are good practices, there's a real cost to implement them. But I wouldn't agree that Kubernetes means building tech debt - on the contrary, if you see the tech debt, k8s makes it easier to get rid of it, but that of course takes time and if you don't do it regularly that tech debt is only gonna grow.
  - majormajor 4y ago
    I just rarely see people tackling greenfield problems with the discipline to choose to do Kubernetes without also choosing to do "distributed" in a broader, complexity-multiplying way. If not for Kubernetes (and particularly Kube + cloud offerings) I really doubt they'd do all the setup necessary to get a bunch of distributed systems/services running with such horizontal sprawl.
slyall 4y ago
No it didn't. You ended up with each site doing things differently. You'd go somewhere and they would have a magical program with a cute name written by a founder that distributed traffic, scheduled jobs and did autoscaling. It would have weird quirks and nobody understood it. Or you wouldn't have it at all. You'd have a nice simple infra and no autoscaling and deploys would be hard and involve manually copying files.
deathanatos 4y ago
People think it took less effort. Right up until you needed to do one of the very many things k8s implements. For example, in multiple previous employers, we had cronjobs: you just set up a cronjob on the server, I mean, really, how hard is that to do? And that server was a single point of failure: we can't just spin up a second server running crond, obviously, as then the job runs twice. Something would need to provide some sort of locking, then the job would need to take advantage of that, we'd need the job to be idempotent … all of which, except the last, k8s does out of the box. (And it mostly forces your hand on the last.) Need to reboot for security patches? We just didn't do that, unless it was something like Heartbleed where it was like "okay we have to". k8s permits me to evict workloads while obeying PDB — in previous orgs, "PDBs" (hell, we didn't even have a word to describe the concept) were just tribal knowledge known only by those of us who SRE'd enough stuff to know how each service worked, and what you needed to know to stop/restart it, and then do that times waaay too many VMs. With k8s, a daemonset can just handle things generically, and automatically. Need to deploy? Pre-k8s, that was just bespoke scripts, e.g., in something like Ansible. If a replica failed to start after deployment, did the script cease deployment? Not the first time it brought everything down, it didn't: it had to grow that feature by learning the hard way. (Although I suppose you can decide that you don't need that readiness check in k8s, but it's at least a hell of a lot easier to get off the ground with.) Need a new VM? What are the chances that the current one actually matches the Ansible, and wasn't snowflaked? (All it takes is one dev, and one point in time, doing one custom command!) The list of operational things that k8s supports that are common amongst "I need to serve this, in production" things goes on. The worse part of k8s thus far has been Azure's half-aaS'd version of it. I've been pretty satisfied with GKE, but I've only recently gotten to know it and I've not pushed it quite as hard as AKS yet. So we'll see.
- lowercased 4y ago
  "Pre-k8s, that was just bespoke scripts, e.g., in something like Ansible" How are these 'bespoke scripts' but helm charts are not 'bespoke'? Or do you consider them 'bespoke-but-better'?
  - deathanatos 4y ago
ranger207 4y ago
Yes, until you've scaled enough that it wasn't. If you're deploying a dev or staging server or even prod for your first few thousand users then you can get by with a handful of servers and stuff. But a lot of stuff that works well on one or three servers starts working less well on a dozen servers, and it's around that point that the up-front difficulty of k8s starts to pay off with the lower long-term difficulty
- morelisp 4y ago
  Whatever crossover point might exist for Kubernetes it's not at a dozen servers, at the low end it's maybe 50. The fair comparison isn't against "yolo scp my php to /var/www", but any of the disciplined orchestration/containerization tools other than Kubernetes. I ran ~40 servers across 3 DCs with about 1/3 of my time going to ops using salt and systemd. The next company, we ran about 80 in one DC with one dedicated ops/infra person also embedded in the dev team + "smart hands" contracts in the DC. Today that runs in large part on Kubernetes; it's now about 150 servers and takes basically two full ops people disconnected from our dev completely, plus some unspecified but large percentage of a ~10 person "platform team", with a constant trickle of unsatisfying issues around storage, load balancing, operator compatibility, etc. Our day-to-day dev workflow has not gotten notably simpler.

Yes, they did, but did it really take less effort than running it on Kubernetes?

by 988747 4y ago

Yes, they did, but did it really take less effort than running it on Kubernetes?

lowercased 4y ago

Yes.

morelisp 4y ago

Yes.

deterministic 4y ago

Yep. Still doing it today. Very large scale Enterprise systems with complex multi-country/multi-organisational operational rules running 24/7. Millions of lines of code. No Kubernetes. No micro-services. No BS. It’s simple. It works. And it has worked for 30+ years.

nosequel 4y ago

Yes. Much.

jamwt 4y ago

Yes, way less effort

nailer 4y ago

Yes.

athoscouto 4y ago

I'm going to diverge from sibling comments: it depends. As the article points out, k8s may really simplify deploys for devs, while giving the autonomy. But it isn't always worth it.

winrid 4y ago

Yeah... at small to medium scale anyway.

majormajor 4y ago

Definitely if they weren't running very complex services or their business tolerated an occasional maintenance window. The particular way Kubernetes can bite you is that it makes it much easy to start with far more complex setups - not necessarily much harder than to start with a simple setup! - but then you have to maintain and operate those complex setups forever, even if you don't need them that much. If you're growing your team and your services, having a much bigger, more complicated toolbox for people to reach into on day 1 gives you way more chance of building expensive painful tech debt.

morelisp 4y ago
The operational value of ~weekly ~30 min "maintenance windows" is wildly underestimated by teams today, and its business cost is wildly overestimated.
suralind 4y ago
I think it may appear so because Kubernetes promotes good practices. Do logging, do metrics, do traces. That list quickly grows and while these are good practices, there's a real cost to implement them. But I wouldn't agree that Kubernetes means building tech debt - on the contrary, if you see the tech debt, k8s makes it easier to get rid of it, but that of course takes time and if you don't do it regularly that tech debt is only gonna grow.
- majormajor 4y ago
  I just rarely see people tackling greenfield problems with the discipline to choose to do Kubernetes without also choosing to do "distributed" in a broader, complexity-multiplying way. If not for Kubernetes (and particularly Kube + cloud offerings) I really doubt they'd do all the setup necessary to get a bunch of distributed systems/services running with such horizontal sprawl.

slyall 4y ago

No it didn't. You ended up with each site doing things differently. You'd go somewhere and they would have a magical program with a cute name written by a founder that distributed traffic, scheduled jobs and did autoscaling. It would have weird quirks and nobody understood it. Or you wouldn't have it at all. You'd have a nice simple infra and no autoscaling and deploys would be hard and involve manually copying files.

deathanatos 4y ago

People think it took less effort. Right up until you needed to do one of the very many things k8s implements. For example, in multiple previous employers, we had cronjobs: you just set up a cronjob on the server, I mean, really, how hard is that to do? And that server was a single point of failure: we can't just spin up a second server running crond, obviously, as then the job runs twice. Something would need to provide some sort of locking, then the job would need to take advantage of that, we'd need the job to be idempotent … all of which, except the last, k8s does out of the box. (And it mostly forces your hand on the last.) Need to reboot for security patches? We just didn't do that, unless it was something like Heartbleed where it was like "okay we have to". k8s permits me to evict workloads while obeying PDB — in previous orgs, "PDBs" (hell, we didn't even have a word to describe the concept) were just tribal knowledge known only by those of us who SRE'd enough stuff to know how each service worked, and what you needed to know to stop/restart it, and then do that times waaay too many VMs. With k8s, a daemonset can just handle things generically, and automatically. Need to deploy? Pre-k8s, that was just bespoke scripts, e.g., in something like Ansible. If a replica failed to start after deployment, did the script cease deployment? Not the first time it brought everything down, it didn't: it had to grow that feature by learning the hard way. (Although I suppose you can decide that you don't need that readiness check in k8s, but it's at least a hell of a lot easier to get off the ground with.) Need a new VM? What are the chances that the current one actually matches the Ansible, and wasn't snowflaked? (All it takes is one dev, and one point in time, doing one custom command!) The list of operational things that k8s supports that are common amongst "I need to serve this, in production" things goes on. The worse part of k8s thus far has been Azure's half-aaS'd version of it. I've been pretty satisfied with GKE, but I've only recently gotten to know it and I've not pushed it quite as hard as AKS yet. So we'll see.

lowercased 4y ago
"Pre-k8s, that was just bespoke scripts, e.g., in something like Ansible" How are these 'bespoke scripts' but helm charts are not 'bespoke'? Or do you consider them 'bespoke-but-better'?
- deathanatos 4y ago

ranger207 4y ago

Yes, until you've scaled enough that it wasn't. If you're deploying a dev or staging server or even prod for your first few thousand users then you can get by with a handful of servers and stuff. But a lot of stuff that works well on one or three servers starts working less well on a dozen servers, and it's around that point that the up-front difficulty of k8s starts to pay off with the lower long-term difficulty

morelisp 4y ago
Whatever crossover point might exist for Kubernetes it's not at a dozen servers, at the low end it's maybe 50. The fair comparison isn't against "yolo scp my php to /var/www", but any of the disciplined orchestration/containerization tools other than Kubernetes. I ran ~40 servers across 3 DCs with about 1/3 of my time going to ops using salt and systemd. The next company, we ran about 80 in one DC with one dedicated ops/infra person also embedded in the dev team + "smart hands" contracts in the DC. Today that runs in large part on Kubernetes; it's now about 150 servers and takes basically two full ops people disconnected from our dev completely, plus some unspecified but large percentage of a ~10 person "platform team", with a constant trickle of unsatisfying issues around storage, load balancing, operator compatibility, etc. Our day-to-day dev workflow has not gotten notably simpler.