4 ms·

Another team's product at our company runs inside a Kubernetes cluster which consists of lots of microservices. CPU's are at 80% all the time. Redis, RabbitMQ,

by kgeist 2y ago

Another team's product at our company runs inside a Kubernetes cluster which consists of lots of microservices. CPU's are at 80% all the time. Redis, RabbitMQ, MySQL, Elastic Search etc. There's no number crunching, mostly CRUD with some business logic on top. It requires a separate operations team to manage it (otherwise this goldberg machine easily explodes). A few days ago I asked our ops team, what's our rps? ~500. Out of curiosity, I ran a few benchmarks to stress test my monolithic pet project in Go with a single DB for everything and it easily handled 15k/s on my home computer, too, without any special optimizations. I understand its load is totally different from our production server, and my benchmark is most likely flawed, but the order of magnitude of difference is interesting. I was told the production cluster also has additional 500 rps just for communication between the microservices. I suspect there's just too much overhead in their setup: communication between services, serialization of data between different DBs, etc. If it was a single modular monolith with a single DB, I suspect it could be faster and easier to maintain... Another issue is that they also sell it as an on-premise solution. And with the complexity growing, it now often fails to run on clients' infra.

rovr138 2y ago
At first I was "eh, if it runs okay and can just be scaled, it's fine as long as they know the limits, how it scales, and it's maintained". > And with the complexity growing, it now often fails to run on clients' infra. then I saw this and... yeah completely different beast
earnesti 2y ago
But imagine how many billable hours were created with this thing
- Nextgrid 2y ago
  Unless it's a consultancy project it's less about billable hours and more about internal busywork and resume fodder. In a ZIRP/startups environment this kind of BS was rewarded by the market and thus "engineers" would take any opportunity to build up such unproductive over-engineering skills to polish up their resume for the next opportunity.
ndriscoll 2y ago
We had pretty much the same deal at my last job. We had somewhere around 50 services that ought to have been one. Batch processes would take hours as they made 10s of millions of internal API calls (which could also fail of course) instead of running for a minute or two just doing the work with a couple SQL queries. Who knows how many hours wasted debating the boundaries and contracts between services when those layers didn't need to exist in the first place. Millions a year spent on hardware. All in the name of "scalability" when we also had a few hundred rps, and the thing obviously had scaling issues exactly because it was being done that way. Very frustrating.
Nextgrid 2y ago
I bet those CPUs are mostly busy just de/serializing JSON, processing HTTP and TLS for all the various intra-service calls.
Scarbutt 2y ago
Not to disagree, but if your company has around 300,000 LoC in business logic and your pet project has 1,000 LoC, the processing times are going to be significantly different.
- kgeist 2y ago
  It has more business logic, true, but the average request is just the usual "retrieve an object from the DB, modify a few columns, and save back". Just like in my project. They have a higher LoC because they have more use cases, but a single use case alone is not very complex and is comparable to my project's code in its complexity. So I think it's fair to compare the two when it comes to rps.

by kgeist 2y ago

rovr138 2y ago

At first I was "eh, if it runs okay and can just be scaled, it's fine as long as they know the limits, how it scales, and it's maintained". > And with the complexity growing, it now often fails to run on clients' infra. then I saw this and... yeah completely different beast

earnesti 2y ago

But imagine how many billable hours were created with this thing

Nextgrid 2y ago
Unless it's a consultancy project it's less about billable hours and more about internal busywork and resume fodder. In a ZIRP/startups environment this kind of BS was rewarded by the market and thus "engineers" would take any opportunity to build up such unproductive over-engineering skills to polish up their resume for the next opportunity.

ndriscoll 2y ago

We had pretty much the same deal at my last job. We had somewhere around 50 services that ought to have been one. Batch processes would take hours as they made 10s of millions of internal API calls (which could also fail of course) instead of running for a minute or two just doing the work with a couple SQL queries. Who knows how many hours wasted debating the boundaries and contracts between services when those layers didn't need to exist in the first place. Millions a year spent on hardware. All in the name of "scalability" when we also had a few hundred rps, and the thing obviously had scaling issues exactly because it was being done that way. Very frustrating.

Nextgrid 2y ago

I bet those CPUs are mostly busy just de/serializing JSON, processing HTTP and TLS for all the various intra-service calls.

Scarbutt 2y ago

Not to disagree, but if your company has around 300,000 LoC in business logic and your pet project has 1,000 LoC, the processing times are going to be significantly different.

kgeist 2y ago
It has more business logic, true, but the average request is just the usual "retrieve an object from the DB, modify a few columns, and save back". Just like in my project. They have a higher LoC because they have more use cases, but a single use case alone is not very complex and is comparable to my project's code in its complexity. So I think it's fair to compare the two when it comes to rps.