3 ms·

Well, in non-competitive "markets" like government services, your failure will just manifest itself in different ways. You will still fall behind more agile org

Well, in non-competitive "markets" like government services, your failure will just manifest itself in different ways. You will still fall behind more agile organizations, but instead of your company going bankrupt, citizens will increasingly view the government as inept, loose trust, etc. A terrible enough digital infrastructure will eventually become a drag on your GDP and lower your standard of living relative to other places. What's funny about your quotes is that Kubernetes is now 8 years old, very mature, very capable, widely-used, well-understood, with lots of tooling and wide-spread know-how. It is "boring".

KronisLV 4y ago
(this got kind of long, might make it into a blog post later) > Well, in non-competitive "markets" like government services, your failure will just manifest itself in different ways. It is true that failure might manifest in different ways, but this possibly conflates not using the newest tech with failure, which isn't necessarily true. Back when the COVID tracing apps were being implemented, my country managed to mostly get it done with using a pretty old tech stack: some Ruby, some Node here and there, as well as OCI containers without anything like K8s operators or CRDs. It worked fine, it scaled decently, there were no loud snafus in the media or glaring deficiencies. I personally got to write the homepage with a pretty boring tech stack: React with no hooks, Bootstrap (instead of something like Tailwind), Nginx and Node for some read-only API components and S3 for files: https://apturicovid.lv/#en https://apturicovid.lv/#en Though maybe the difference here was that it was a volunteer project by people who actually cared about what they were developing, instead of just getting a fat paycheck and padding their CVs (more about that below). Sometimes you just work at a scale where you don't need millions of writes/reads per second, given that your country literally doesn't have enough citizens for that sort of a load. You also don't always need to explore something like CQRS, pub-sub and event driven design and so on. Dare I say it, it would have probably worked just fine with Apache, PHP, Varnish and jQuery as well. I've seen some of the largest companies in the entire country succeed whilst running on somewhat dated monoliths, any outages being inconsequential to getting their business processes done. I've seen some others fail with microservices due to underestimating how complicated getting them into prod would actually be, especially with plenty of dated red tape around it. That said, there have also been monoliths failing and microservices succeeding, it's just that the newness of any particular approach/technology doesn't always equate success or failure. This is situational, but that's my point - there are a lot of comparatively smaller projects and companies out there. Most people don't deal with the interesting sorts of problems that SV companies work with. > You will still fall behind more agile organizations, but instead of your company going bankrupt, citizens will increasingly view the government as inept, loose trust, etc. A terrible enough digital infrastructure will eventually become a drag on your GDP and lower your standard of living relative to other places. If we are talking about failure in particular, i mostly agree; that is actually a pretty good point! However, once again, there is some nuance about it. In my country, the tax declaration system (https://eds.vid.gov.lv https://eds.vid.gov.lv) has had a history of falling over on the day when you can submit your taxes or when under more severe load. And yet, everyone gets their stuff done regardless, even if they need to come back to it a day or two later, after seeing that it's overwhelmed with the load. There is little more impact apart from inconvenience. In contrast, the e-health system of the country has had like a dozen million euros invested in it and still doesn't work properly [1][2]: > In December, the availability of e-health was disrupted for seven days (with one exception - on working days), but for another five working days it did not actually work. For example, on December 7, none of the e-health services were available for more than four hours (4 h 13 min) during the working day. January looks a little better - the disruption is in six working days, but in February three days of e-health is disrupted, but for four days it can be said that the system does not work. [1] https://www-lsm-lv.translate.goog/raksts/zinas/zinu-analize/e-veselibu-radis-no-jauna-izmaksas-meramas-miljonos.a396672/?_x_tr_sl=lv&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp https://www-lsm-lv.translate.goog/raksts/zinas/zinu-analize/... [2] https://www-db-lv.translate.goog/zinas/jauna-eveseliba-plano-ieguldit-vismaz-sesus-miljonus-eiro-501618?_x_tr_sl=lv&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp https://www-db-lv.translate.goog/zinas/jauna-eveseliba-plano... And yet, in regards to the failures of it, there is a lot of talk not of technological factors, but rather bad planning, changing requirements, bad development, bad testing and lot of overpromising, underdelivering and lots of filling their own pockets when they should have been working [3]. > "We are very worried. The situation is that the Ministry of Health has not learned to say what it wants. Different performers have designed the unknown, which has been said by various experts. They have designed on different platforms that are not compatible with each other, ”admitted Belēvičs. However, it is possible to "combine these incompatible parts on one payment axis, creating an opportunity for patients to pay co-payments for medicines and reimbursement for examinations and other health care services with a bank card". According to Belevich, this project is still to be saved, but it would need additional funding. So far, more than ten million lats have been spent on the project, but the minister did not specify the exact amount. It has already been reported that e-health computer systems have been developed but are not compatible with each other, according to the National Health Service responsible for the "e-health" project. The exchange of information between systems is not possible. Errors are currently being sought and corrected, but the project should be completed by mid-summer. [3] https://www-lsm-lv.translate.goog/raksts/zinas/latvija/belevics-lidz-sim-e-veselibas-projektam-iztereti-vairak-neka-10-miljoni-precizets.a121240/?_x_tr_sl=lv&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp https://www-lsm-lv.translate.goog/raksts/zinas/latvija/belev... That last bit of text was written 7 years ago and the project never went anywhere. And no amount of ClickHouse, Go microservices or Kubernetes would have fixed that. It's important to recognize that some systems and domains matter more than others and that planning and the way something is developed might have a bigger impact than any choice of a particular set of technologies for it. > What's funny about your quotes is that Kubernetes is now 8 years old, very mature, very capable, widely-used, well-understood, with lots of tooling and wide-spread know-how. It is "boring". I fear that this isn't entirely true, just look at: https://k8s.af/ https://k8s.af/ People still make the same mistakes that they did years ago and for many out there, there is nothing that Kubernetes gives them (of the features that they actually use) that Docker Swarm or Nomad wouldn't provide, both being more simplistic solutions. Actually, the long Roblox outage a while ago [4][5] was caused by relying on Nomad and not expecting certain failure modes, so not even those are foolproof either. > The outage lasted for 72 hours and 45 minutes, or just over 3 days, making it the longest outage in Roblox history. [4] https://roblox.fandom.com/wiki/2021_Roblox_outage https://roblox.fandom.com/wiki/2021_Roblox_outage > The root cause was due to two issues. Enabling a relatively new streaming feature on Consul under unusually high read and write load led to excessive contention and poor performance. In addition, our particular load conditions triggered a pathological performance issue in BoltDB. The open source BoltDB system is used within Consul to manage write-ahead-logs for leader election and data replication. [5] https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/ https://blog.roblox.com/2022/01/roblox-return-to-service-10-... For Kubernetes specific issues, see the link above, but it's always the same story with complex pieces of software: especially if you just think that you're one of the large corps that have large Ops/DevOps departments but actually have just decided to use Kubernetes/Kafka/another solution without having the proper resources to keep it running successfully. It is capable, it is mature, it has a lot of tooling (though a lot of it is sadly SaaS/PaaS money grab attempts), but it is not nearly as well understood as it should be and thus is too widely used, to the point where i've heard people unironically make the suggestion of: "Yeah, you should never build your own clusters, just pay one of the large corporations do to that for you." It is getting more boring, thanks to things like K3s/k0s/MicroK8s, Rancher/Portainer, Lens/k9s, Helm and other lightweight distros and tools to make running it more feasible for a wide variety of deployments, but it's still too much for many. Just consider needing a separate node with >4 GB of RAM to act as the leader server for a cluster, when both Docker Swarm and Nomad both get by with a few hundred MB at most. While there are many benefits to orchestration (to the point of this being a game changer, as well as using containers and 12 Factor Apps for cloud native solutions), the resource requirements and usage are both examples of Wirth's law and have gone leaps backwards from where we were with systemd and Ansible. To me, the sweet spot is still something like: - local development: Docker (though Podman is cool, too) - local development of multiple services: Docker Compose - running containers on a single server: Docker Compose - orchestrating services across multiple servers: Docker Swarm (with ingress being a regular container) and maybe Portainer - more complex orchestration, if you really need it: K3s/RKE with Portainer/Rancher (which deal with the complexity nicely)

KronisLV 4y ago

(this got kind of long, might make it into a blog post later) > Well, in non-competitive "markets" like government services, your failure will just manifest itself in different ways. It is true that failure might manifest in different ways, but this possibly conflates not using the newest tech with failure, which isn't necessarily true. Back when the COVID tracing apps were being implemented, my country managed to mostly get it done with using a pretty old tech stack: some Ruby, some Node here and there, as well as OCI containers without anything like K8s operators or CRDs. It worked fine, it scaled decently, there were no loud snafus in the media or glaring deficiencies. I personally got to write the homepage with a pretty boring tech stack: React with no hooks, Bootstrap (instead of something like Tailwind), Nginx and Node for some read-only API components and S3 for files: https://apturicovid.lv/#en https://apturicovid.lv/#en Though maybe the difference here was that it was a volunteer project by people who actually cared about what they were developing, instead of just getting a fat paycheck and padding their CVs (more about that below). Sometimes you just work at a scale where you don't need millions of writes/reads per second, given that your country literally doesn't have enough citizens for that sort of a load. You also don't always need to explore something like CQRS, pub-sub and event driven design and so on. Dare I say it, it would have probably worked just fine with Apache, PHP, Varnish and jQuery as well. I've seen some of the largest companies in the entire country succeed whilst running on somewhat dated monoliths, any outages being inconsequential to getting their business processes done. I've seen some others fail with microservices due to underestimating how complicated getting them into prod would actually be, especially with plenty of dated red tape around it. That said, there have also been monoliths failing and microservices succeeding, it's just that the newness of any particular approach/technology doesn't always equate success or failure. This is situational, but that's my point - there are a lot of comparatively smaller projects and companies out there. Most people don't deal with the interesting sorts of problems that SV companies work with. > You will still fall behind more agile organizations, but instead of your company going bankrupt, citizens will increasingly view the government as inept, loose trust, etc. A terrible enough digital infrastructure will eventually become a drag on your GDP and lower your standard of living relative to other places. If we are talking about failure in particular, i mostly agree; that is actually a pretty good point! However, once again, there is some nuance about it. In my country, the tax declaration system (https://eds.vid.gov.lv https://eds.vid.gov.lv) has had a history of falling over on the day when you can submit your taxes or when under more severe load. And yet, everyone gets their stuff done regardless, even if they need to come back to it a day or two later, after seeing that it's overwhelmed with the load. There is little more impact apart from inconvenience. In contrast, the e-health system of the country has had like a dozen million euros invested in it and still doesn't work properly [1][2]: > In December, the availability of e-health was disrupted for seven days (with one exception - on working days), but for another five working days it did not actually work. For example, on December 7, none of the e-health services were available for more than four hours (4 h 13 min) during the working day. January looks a little better - the disruption is in six working days, but in February three days of e-health is disrupted, but for four days it can be said that the system does not work. [1] https://www-lsm-lv.translate.goog/raksts/zinas/zinu-analize/e-veselibu-radis-no-jauna-izmaksas-meramas-miljonos.a396672/?_x_tr_sl=lv&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp https://www-lsm-lv.translate.goog/raksts/zinas/zinu-analize/... [2] https://www-db-lv.translate.goog/zinas/jauna-eveseliba-plano-ieguldit-vismaz-sesus-miljonus-eiro-501618?_x_tr_sl=lv&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp https://www-db-lv.translate.goog/zinas/jauna-eveseliba-plano... And yet, in regards to the failures of it, there is a lot of talk not of technological factors, but rather bad planning, changing requirements, bad development, bad testing and lot of overpromising, underdelivering and lots of filling their own pockets when they should have been working [3]. > "We are very worried. The situation is that the Ministry of Health has not learned to say what it wants. Different performers have designed the unknown, which has been said by various experts. They have designed on different platforms that are not compatible with each other, ”admitted Belēvičs. However, it is possible to "combine these incompatible parts on one payment axis, creating an opportunity for patients to pay co-payments for medicines and reimbursement for examinations and other health care services with a bank card". According to Belevich, this project is still to be saved, but it would need additional funding. So far, more than ten million lats have been spent on the project, but the minister did not specify the exact amount. It has already been reported that e-health computer systems have been developed but are not compatible with each other, according to the National Health Service responsible for the "e-health" project. The exchange of information between systems is not possible. Errors are currently being sought and corrected, but the project should be completed by mid-summer. [3] https://www-lsm-lv.translate.goog/raksts/zinas/latvija/belevics-lidz-sim-e-veselibas-projektam-iztereti-vairak-neka-10-miljoni-precizets.a121240/?_x_tr_sl=lv&_x_tr_tl=en&_x_tr_hl=en&_x_tr_pto=wapp https://www-lsm-lv.translate.goog/raksts/zinas/latvija/belev... That last bit of text was written 7 years ago and the project never went anywhere. And no amount of ClickHouse, Go microservices or Kubernetes would have fixed that. It's important to recognize that some systems and domains matter more than others and that planning and the way something is developed might have a bigger impact than any choice of a particular set of technologies for it. > What's funny about your quotes is that Kubernetes is now 8 years old, very mature, very capable, widely-used, well-understood, with lots of tooling and wide-spread know-how. It is "boring". I fear that this isn't entirely true, just look at: https://k8s.af/ https://k8s.af/ People still make the same mistakes that they did years ago and for many out there, there is nothing that Kubernetes gives them (of the features that they actually use) that Docker Swarm or Nomad wouldn't provide, both being more simplistic solutions. Actually, the long Roblox outage a while ago [4][5] was caused by relying on Nomad and not expecting certain failure modes, so not even those are foolproof either. > The outage lasted for 72 hours and 45 minutes, or just over 3 days, making it the longest outage in Roblox history. [4] https://roblox.fandom.com/wiki/2021_Roblox_outage https://roblox.fandom.com/wiki/2021_Roblox_outage > The root cause was due to two issues. Enabling a relatively new streaming feature on Consul under unusually high read and write load led to excessive contention and poor performance. In addition, our particular load conditions triggered a pathological performance issue in BoltDB. The open source BoltDB system is used within Consul to manage write-ahead-logs for leader election and data replication. [5] https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/ https://blog.roblox.com/2022/01/roblox-return-to-service-10-... For Kubernetes specific issues, see the link above, but it's always the same story with complex pieces of software: especially if you just think that you're one of the large corps that have large Ops/DevOps departments but actually have just decided to use Kubernetes/Kafka/another solution without having the proper resources to keep it running successfully. It is capable, it is mature, it has a lot of tooling (though a lot of it is sadly SaaS/PaaS money grab attempts), but it is not nearly as well understood as it should be and thus is too widely used, to the point where i've heard people unironically make the suggestion of: "Yeah, you should never build your own clusters, just pay one of the large corporations do to that for you." It is getting more boring, thanks to things like K3s/k0s/MicroK8s, Rancher/Portainer, Lens/k9s, Helm and other lightweight distros and tools to make running it more feasible for a wide variety of deployments, but it's still too much for many. Just consider needing a separate node with >4 GB of RAM to act as the leader server for a cluster, when both Docker Swarm and Nomad both get by with a few hundred MB at most. While there are many benefits to orchestration (to the point of this being a game changer, as well as using containers and 12 Factor Apps for cloud native solutions), the resource requirements and usage are both examples of Wirth's law and have gone leaps backwards from where we were with systemd and Ansible. To me, the sweet spot is still something like: - local development: Docker (though Podman is cool, too) - local development of multiple services: Docker Compose - running containers on a single server: Docker Compose - orchestrating services across multiple servers: Docker Swarm (with ingress being a regular container) and maybe Portainer - more complex orchestration, if you really need it: K3s/RKE with Portainer/Rancher (which deal with the complexity nicely)