3 ms·

SO is always impressive - love that their redis servers with 256GB RAM peak at 2% CPU load :) SO is also my go-to argument when some smart "architect" proposes

SO is always impressive - love that their redis servers with 256GB RAM peak at 2% CPU load :) SO is also my go-to argument when some smart "architect" proposes redundant Kubernetes cluster instances for some company-local project. People seem to have lost the feeling what is needed to serve a couple of thousand concurrent users (For company internal usages which I specialize in, you hardly will get more users). Everyone thinks they are Google or Netflix. Meanwhile, SO runs on 1-2 Racks with an amount of server that would not even justify kubernetes or even docker.

bscphil 6y ago
The "1-2 racks" perspective is great too, really makes you think the old XKCD joke [1] about tripping over the power cable might not be that far wrong. ;-) [1] https://xkcd.com/908/ https://xkcd.com/908/
sofixa 6y ago
SO really isn't a great example, they have considerations most companies don't - Windows and SQL Server licensing. When shit like that is involved, scale out rarely seems like a better choice. It's not only about the amount of users, it's also a matter of availability. Even the most stupid low-use barely-does-anything internal apps at my company get deployed either to two machines or a Nomad cluster for redundancy ( across two DCs). Design for failure and all that. Failure is unlikely, but it's trivial to setup at least active-passive redundancy just in case, it will make failures much easier.
mschuster91 6y ago
> SO is also my go-to argument when some smart "architect" proposes redundant Kubernetes cluster instances for some company-local project. Technically you don't need Kubernetes, yes. But: There are advantages that Kubernetes gives you even for a small shop: - assuming you have a decent shared storage, it's a matter of about 30 minutes to replace a completely failed machine - plug the server in, install a bare-bones Ubuntu, kubeadm join, done. If you use Puppet and netboot install, you can go even faster (Source: been there, done that). And the best thing: assuming well written health checks users won't even notice you just had a node fail as k8s will take care of rescheduling. - no need to wrangle with systemd unit files (or, worse, classic init.d scripts) for your application. For most scenarios you will either find Docker-embedded healthchecks somewhere or you can easily write your own so that Kubernetes can automatically - no "hidden undocumented state" like wonky manual customizations somewhere in /etc that can mess up disaster recovery / horizontal scale, as everything relevant is included in either the Kubernetes spec or the Docker images. Side effect: this also massively reduces the ops load during upgrades, as all there is on a typical k8s node should be the base OS and Docker (or, in newest k8s versions, not even that anymore) - it's easy to set up new development instances in a CI/CD environment - generally, it's easier to get stuff done in corporate environments: just spin up a container on your cluster and that's it, no wrestling with finance and three levels of sign-off to get approval for a VM or, worse, bare metal. I won't deny that there are issues though, especially if you're selfhosting: - you will end up with issues with basic network tasks very quickly during setup, MetalLB is a nightmare, but smooth once you do have set it up. Most stuff is made with the assumption of every machine being in a fully Internet-reachable cluster (coughs in certbot), once you diverge from that (e.g. because of corp requiring you have to have dedicated "load balancer" nodes that only serve to direct traffic from outside to inside and "application" nodes not be directly internet-reachable) you're on your own. - most likely you'll end up with one or two sandwich layers of load balancing (k8s ingress for one, and if you have it an external LB/WAF), which makes stuff like XFF headers ... interesting to say the least - same if you're running anything with UDP, e.g. RTMP streaming - the various networking layers are extremely hard to debug as most of k8s networking (no matter the overlay you use) is a boatload of iptables black magic. Even if you have a decade of experience...
- littlecranky67 6y ago
  Your arguments are true, but you did not consider the complexity that you have now introduced in a small shop operation. You will need kubernetes knowledge and experienced engineers on that matter. I would argue that the SO setup with 9 webservers, 2x2 DB servers and 2 redis servers could easily be administered with 20 year old knowledge about networks and linux/windows itself. And I also argue, lack of experience of fiddling with redundant kubernetes is a more likely source of downtime than hardware failure and keeping things simple.
  - mschuster91 6y ago
    > You will need kubernetes knowledge and experienced engineers on that matter. For a small shop you'll need one person knowing that stuff, or you bring in an external consultant for setting up and maintaining the cluster, or you move to some cloud provider (k8s is basically a commodity that everyone and their dog offers, not just the big 3!) so you don't have to worry about that at all. And a cluster for basic stuff is not even that expensive if you do want to run your own. Three worker machines and one (or, if you want HA, two) NAS systems... half a rack and you're set. The benefit you have is your engineers will waste a lot less time setting up, maintaining and tearing down development and QA environments. As for the SO setup: the day-to-day maintenance of them should be fairly simple - but AFAIK they had to do a lot of development effort to get the cluster to that efficiency, including writing their own "tag DB".
    - littlecranky67 6y ago
      You will always need 2-3 experts, because in case of an incident, your 1 engineer might be on sick/holiday leave. Well but lets walk one step back looking at SO, they are a windows shop (.NET, MS SQL Server) so I doubt k8s would be found in their setup.
    - toyg 6y ago
      Ah yes, I’ll make my critical infrastructure totally dependent on some outside consultant who may or may not be around when I really need him. That sounds like a great strategy. /s

bscphil 6y ago

The "1-2 racks" perspective is great too, really makes you think the old XKCD joke [1] about tripping over the power cable might not be that far wrong. ;-) [1] https://xkcd.com/908/ https://xkcd.com/908/

sofixa 6y ago

SO really isn't a great example, they have considerations most companies don't - Windows and SQL Server licensing. When shit like that is involved, scale out rarely seems like a better choice. It's not only about the amount of users, it's also a matter of availability. Even the most stupid low-use barely-does-anything internal apps at my company get deployed either to two machines or a Nomad cluster for redundancy ( across two DCs). Design for failure and all that. Failure is unlikely, but it's trivial to setup at least active-passive redundancy just in case, it will make failures much easier.

mschuster91 6y ago

> SO is also my go-to argument when some smart "architect" proposes redundant Kubernetes cluster instances for some company-local project. Technically you don't need Kubernetes, yes. But: There are advantages that Kubernetes gives you even for a small shop: - assuming you have a decent shared storage, it's a matter of about 30 minutes to replace a completely failed machine - plug the server in, install a bare-bones Ubuntu, kubeadm join, done. If you use Puppet and netboot install, you can go even faster (Source: been there, done that). And the best thing: assuming well written health checks users won't even notice you just had a node fail as k8s will take care of rescheduling. - no need to wrangle with systemd unit files (or, worse, classic init.d scripts) for your application. For most scenarios you will either find Docker-embedded healthchecks somewhere or you can easily write your own so that Kubernetes can automatically - no "hidden undocumented state" like wonky manual customizations somewhere in /etc that can mess up disaster recovery / horizontal scale, as everything relevant is included in either the Kubernetes spec or the Docker images. Side effect: this also massively reduces the ops load during upgrades, as all there is on a typical k8s node should be the base OS and Docker (or, in newest k8s versions, not even that anymore) - it's easy to set up new development instances in a CI/CD environment - generally, it's easier to get stuff done in corporate environments: just spin up a container on your cluster and that's it, no wrestling with finance and three levels of sign-off to get approval for a VM or, worse, bare metal. I won't deny that there are issues though, especially if you're selfhosting: - you will end up with issues with basic network tasks very quickly during setup, MetalLB is a nightmare, but smooth once you do have set it up. Most stuff is made with the assumption of every machine being in a fully Internet-reachable cluster (coughs in certbot), once you diverge from that (e.g. because of corp requiring you have to have dedicated "load balancer" nodes that only serve to direct traffic from outside to inside and "application" nodes not be directly internet-reachable) you're on your own. - most likely you'll end up with one or two sandwich layers of load balancing (k8s ingress for one, and if you have it an external LB/WAF), which makes stuff like XFF headers ... interesting to say the least - same if you're running anything with UDP, e.g. RTMP streaming - the various networking layers are extremely hard to debug as most of k8s networking (no matter the overlay you use) is a boatload of iptables black magic. Even if you have a decade of experience...

littlecranky67 6y ago
Your arguments are true, but you did not consider the complexity that you have now introduced in a small shop operation. You will need kubernetes knowledge and experienced engineers on that matter. I would argue that the SO setup with 9 webservers, 2x2 DB servers and 2 redis servers could easily be administered with 20 year old knowledge about networks and linux/windows itself. And I also argue, lack of experience of fiddling with redundant kubernetes is a more likely source of downtime than hardware failure and keeping things simple.
- mschuster91 6y ago
  > You will need kubernetes knowledge and experienced engineers on that matter. For a small shop you'll need one person knowing that stuff, or you bring in an external consultant for setting up and maintaining the cluster, or you move to some cloud provider (k8s is basically a commodity that everyone and their dog offers, not just the big 3!) so you don't have to worry about that at all. And a cluster for basic stuff is not even that expensive if you do want to run your own. Three worker machines and one (or, if you want HA, two) NAS systems... half a rack and you're set. The benefit you have is your engineers will waste a lot less time setting up, maintaining and tearing down development and QA environments. As for the SO setup: the day-to-day maintenance of them should be fairly simple - but AFAIK they had to do a lot of development effort to get the cluster to that efficiency, including writing their own "tag DB".
  - littlecranky67 6y ago
    You will always need 2-3 experts, because in case of an incident, your 1 engineer might be on sick/holiday leave. Well but lets walk one step back looking at SO, they are a windows shop (.NET, MS SQL Server) so I doubt k8s would be found in their setup.
  - toyg 6y ago
    Ah yes, I’ll make my critical infrastructure totally dependent on some outside consultant who may or may not be around when I really need him. That sounds like a great strategy. /s