5 ms·

Exactly. I am consulting with a startup right now that chose to go everything docker/k8s. The CTO is half-shocked/half-depressed by the complexity of our archi

by nokya 4y ago

Exactly. I am consulting with a startup right now that chose to go everything docker/k8s. The CTO is half-shocked/half-depressed by the complexity of our architecture meetings, although he used to be a banking software architect in his previous assignments. Every question I ask ends up in a long 15 minutes monologue by the guy who architected all of it, even the most simple questions. They are soon launching a mobile app (only a mobile app and its corresponding API, not even a website) and they already have more than 60 containers running and talking to each other across three k8s clusters and half of them interact directly with third-parties outside. Even as I am being paid by the hour, I really feel sad for both the CTO and the developers attending the meeting. k8s is definitely not for everyone. Google has thousands hardware systems running the same hypervisor, same OS, same container engine and highly specialized stacks of micro-services that need to run by the thousands. And even, I am not sure that k8s would satisfy Google's actual needs tbh. Ironically, there are some companies that highly benefit from this and they are not necessarily "large" companies. In my case, k8s and devops in general made my life infinitely easier for on-site trainings: those who come with a poorly configured or decade-old laptop can actually enjoy the labs at the same pace than every other attendee.

rahkiin 4y ago
60 containers sounds like an architecture problem, not a Kubernetes problem. Kubernetes does not stop you from running 1 container in 1 pod receiving ingress and talking to a database.
- deathanatos 4y ago
  Presuming too (it's hard to tell) that they mean 60 different types of containers. One of my clusters currently has ~311 containers, but that's mostly due to replication. If I count actually different containers (like, unique PodSpecs, or so), that count drops to ≈30. Even that is "high", and from an architectural standpoint, it isn't really a number I'd use. E.g., we have a simple daemon, but it also has a cronjob associated with it. So it has "2" PodSpecs by that count. But architecturally I'd call it a single thing. How it implements itself, that's up to it. A lot of our "unique PodSpec" count, too, comes from utility type things that do one thing, and do it well. Logging (which comes from our vendor) is 3 PodSpecs. Metrics is another 3. We have a network latency measurement (literally ping shoved into a container…): PodSpec. A thing that checks certs to ensure they don't expire: PodSpec. HTTP proxy (for SSRF avoidance): PodSpec. A tool that rotates out nodes so that their OSes can be patched: PodSpec. Let's Encrypt automation (a third party too): 3 PodSpecs … but hey, it does its job, and it's a third party tool, so what do I care, so long as it works the API between me and it suffices (and honestly, its logs are pretty good. When it has had problems, I've usually been able to discern why from the logs). DB backup. But most of these don't really add much conceptual overhead; any one is maybe tied (conceptually) to our applications, but not really to all the other utilities. (E.g., there isn't really coupling between, say, the cert renewer and the logging tooling.) A confused/new dev might need to have it explained to them what any of those given tools do, ofc., but … many of them you can just Google. … in previous jobs where we didn't use Kubernetes, we mostly just ignored a lot of the tasks that these tools handle. E.g., reboot a VM for patches? It was a custom procedure, depending on VM, and what is running on that VM. You needed to understand that, determine what the implications were … etc. And the end result was that reboots just didn't happen. K8s abstracts that (in the form of PDBs, and readiness checks) and can thus automate it. (And ensure that new loads don't need TLC that … an app dev realistically isn't going to be given the time to give.) If we needed a common thing on every node? That would be rough. We did finally get to having a common base VM image, but even them, all of the per-app VM images would need to be rebased on the newer one, and then all rolled out, and who/how would one even track that? And … in practice, it didn't happen.
  - nokya 4y ago
    Thank you for your comment. It's a mix of both what you described: different types of containers + what I have flagged as "utility containers" (stuff they just installed because it serves a purpose very well). The problem I see with this approach is that it has become very difficult to evaluate system-wide topics such as accessibility (or security, or performance) as we constantly deal with a very diverse technological stack and increasingly complex attack surface. In my opinion, this makes finding competent people who can actually evaluate or assess work almost impossible, unless you hire a Lemming who will run some third-party scanner he found on GitHub: if the scanner doesn't say something is awful or critical, then almost everyone at the table is instantly convinced the system is perfectly robust. I try to warn my clients by asking them if they think that a judge will be satisfied if they answer "we ran the scanner the other day and the scanner said it was all good" after a customer sues them for failing to comply with a disabilities act.
- datavirtue 4y ago
  It's the gateway drug
- raffraffraff 4y ago
  Yep, that's microservices gone mad
Spivak 4y ago
Good god, there's a reason people worship monoliths. And here I think my company's app is over-engineered for using Sidekiq and Lambda for similar workloads.
siva7 4y ago
this is not the fault of kubernetes but of microservice architecture gone horribly wrong

by nokya 4y ago

rahkiin 4y ago

60 containers sounds like an architecture problem, not a Kubernetes problem. Kubernetes does not stop you from running 1 container in 1 pod receiving ingress and talking to a database.

deathanatos 4y ago
Presuming too (it's hard to tell) that they mean 60 different types of containers. One of my clusters currently has ~311 containers, but that's mostly due to replication. If I count actually different containers (like, unique PodSpecs, or so), that count drops to ≈30. Even that is "high", and from an architectural standpoint, it isn't really a number I'd use. E.g., we have a simple daemon, but it also has a cronjob associated with it. So it has "2" PodSpecs by that count. But architecturally I'd call it a single thing. How it implements itself, that's up to it. A lot of our "unique PodSpec" count, too, comes from utility type things that do one thing, and do it well. Logging (which comes from our vendor) is 3 PodSpecs. Metrics is another 3. We have a network latency measurement (literally ping shoved into a container…): PodSpec. A thing that checks certs to ensure they don't expire: PodSpec. HTTP proxy (for SSRF avoidance): PodSpec. A tool that rotates out nodes so that their OSes can be patched: PodSpec. Let's Encrypt automation (a third party too): 3 PodSpecs … but hey, it does its job, and it's a third party tool, so what do I care, so long as it works the API between me and it suffices (and honestly, its logs are pretty good. When it has had problems, I've usually been able to discern why from the logs). DB backup. But most of these don't really add much conceptual overhead; any one is maybe tied (conceptually) to our applications, but not really to all the other utilities. (E.g., there isn't really coupling between, say, the cert renewer and the logging tooling.) A confused/new dev might need to have it explained to them what any of those given tools do, ofc., but … many of them you can just Google. … in previous jobs where we didn't use Kubernetes, we mostly just ignored a lot of the tasks that these tools handle. E.g., reboot a VM for patches? It was a custom procedure, depending on VM, and what is running on that VM. You needed to understand that, determine what the implications were … etc. And the end result was that reboots just didn't happen. K8s abstracts that (in the form of PDBs, and readiness checks) and can thus automate it. (And ensure that new loads don't need TLC that … an app dev realistically isn't going to be given the time to give.) If we needed a common thing on every node? That would be rough. We did finally get to having a common base VM image, but even them, all of the per-app VM images would need to be rebased on the newer one, and then all rolled out, and who/how would one even track that? And … in practice, it didn't happen.
- nokya 4y ago
  Thank you for your comment. It's a mix of both what you described: different types of containers + what I have flagged as "utility containers" (stuff they just installed because it serves a purpose very well). The problem I see with this approach is that it has become very difficult to evaluate system-wide topics such as accessibility (or security, or performance) as we constantly deal with a very diverse technological stack and increasingly complex attack surface. In my opinion, this makes finding competent people who can actually evaluate or assess work almost impossible, unless you hire a Lemming who will run some third-party scanner he found on GitHub: if the scanner doesn't say something is awful or critical, then almost everyone at the table is instantly convinced the system is perfectly robust. I try to warn my clients by asking them if they think that a judge will be satisfied if they answer "we ran the scanner the other day and the scanner said it was all good" after a customer sues them for failing to comply with a disabilities act.
datavirtue 4y ago
It's the gateway drug
raffraffraff 4y ago
Yep, that's microservices gone mad

Spivak 4y ago

Good god, there's a reason people worship monoliths. And here I think my company's app is over-engineered for using Sidekiq and Lambda for similar workloads.

siva7 4y ago

this is not the fault of kubernetes but of microservice architecture gone horribly wrong