5 ms·

Once when starting a new gig I inherited a "microservices" architecture. They were having performance problems and "needed" to migrate to microservices. They d

Once when starting a new gig I inherited a "microservices" architecture. They were having performance problems and "needed" to migrate to microservices. They developed 12 seperate applications, all in the same repo, deployed independently it's own JVM. Of course if you were using microservices, you needed docker as well, so they had also developed a giant docker container containing all 12 microservices which they deployed to a single host (all managed by supervisord). Of course since they had 12 different JVM applications, the services needed a host with at least 9GiB of RAM so they used a larger instance. Everything was provisioned manually by the way because there was no service discovery or container orchestration - just a docker container running on a host (an upgrade from running the production processes in a tmux instance). What they really had was a giant monolithic application with a complicated deployment process and an insane JVM overhead. Moving to the larger instance likely solved the performance issues. In place they now had multiple over provisioned instances (for "HA"), and combined with other questionable decisions, were paying ~100k/year for a web backend that did no more than ~50 requests/minute at peak. But hey at least they were doing real devops like Netflix. For me, I've become a bit more aware of cargo cult development. I can't say I'm completely immune to cargo cult driven development either (I once rewrote an entire Angular application in React because "Angular is dead") so it really opened my eyes how I could also implement "solutions" without truly understanding why they are useful.

eadmund 9y ago
> They developed 12 seperate applications, all in the same repo, deployed independently it's own JVM. I've dealt with an even worse system, with a dozen separate applications, each in its own repo, then with various repos containing shared code. But the whole thing was really one interconnected system, such that a change to one component often required changes to the shared code, which required updates to all the other services. It was a nightmare. At least your folks had the good sense to use a single repository.
- paulddraper 9y ago
  Agreed. Multiple closely dependent repos by one organization is the real nightmare.
- crescentfresh 9y ago
  > then with various repos containing shared code What source control system? Also, from the article: > even though theoretically services can be deployed in isolation, you find that due to the inter-dependencies between services, you have to deploy sets of services as a group This is the situation we are in, like you were.
  - eadmund 9y ago
    > What source control system? Git in our case. And our direction was not to use submodules or anything like that to make life manageable. It was pretty unpleasant.
z3t4 9y ago
You don't always have to know why but it's somewhat frightening that so many "engineers" don't have a clue why they are doing something (because Google does it). And I'm of course guilty of it myself jumping on the hype-train or uncritically taking advice from domain experts only to find out years later that much of it was BS. Most of the time though, you will not reach enlightenment. I guess it's in our nature to follow authority, hype, trends and group think.
- walrus1066 9y ago
  They're only following their incentives. What's gonna look better on a devs CV: 'spent a year maintaining a CRUD monolith app' vs 'spent a year breaking monolith into microservices, with shiny language X to boot'. We can be a very fashion and buzzword driven industry sometimes. EDIT: this perverse incentive goes all they way to the top, through to CTO level. Sometimes I wonder if businesses understand just how much money and effort is wasted on pointless rewrites that make life harder for everyone.
  - therealdrag0 9y ago
    "Resume driven development" :)
- michaelcampbell 9y ago
  > it's somewhat frightening that so many "engineers" don't have a clue why they are doing something (because Google does it). This doesn't stop at engineering; open offices, teaser/trick based interviewing, OKR's, ... Even GOOGLE doesn't do some of those things anymore, but the follower sheep still do.
merb 9y ago
> Of course since they had 12 different JVM applications, the services needed a host with at least 9GiB of RAM so they used a larger instance. well experimentally oracle solved that problem, somewhat. you could now use CDS and *.so files for some parts of your application. it probably does not eliminate every problem, but yeah it helps a bit at least. but well it would've been easier to just use apache felix or so to start all the applications on a OSGI container. this would've probably saved like 5-7 GiB of RAM.
thebeardedone 9y ago
I recently had a similar experience; our product at work is a monolith not in the greatest shape as it has technical debt which we inherited and our product is usually used condescendingly when talking to other teams working on different products. To our surprise when we started testing it with cloud deployments, it was really lightweight compared to just one of the 25 java micro-services from the other teams. Their "microservices" suffered from the same JVM overhead and to remedy this they are joining their functionalities together (initially they had 30-40).
- virmundi 9y ago
  I'm switching to go partially due to the jvm. Hopefully I'll get better partitioning on a single small box as I start.
Clubber 9y ago
>They were having performance problems and "needed" to migrate to microservices. They developed 12 seperate applications, all in the same repo, deployed independently it's own JVM. 9 times out of 10 it's because developers don't know how to properly design and index the underlying RDBMS. I've noticed there is a severe lack of knowledge of that for the average developer.
giancarlostoro 9y ago
Sounds like they don't understand why it's called a microservice to begin with. They're not supposed to be solutions an entire piece of software, just dedicated bits at least that's what I'd figure with a name such as "micro". When we adopted microservices at my job (idk if Azure Functions count or not) we did it because we had 1 task we needed taken out of our main application for performance concerns and because we knew it would involve way more work to implement (.NET Framework codebase being ported to .NET Core which meant the dependencies from .NET Framework did not work anymore in .NET Core) but we eventually turned it into a WebAPI instead due to limitations of Azure Functions for what we wanted to do (process imagery of sorts).
- bonesss 9y ago
  > idk if Azure Functions count or not Azure Functions are technically a "serverless" product, but using them as y'all intended to is a textbook definition of a "microservice" :)
ataturk 9y ago
Microservices without using something like Spring cloud which offers service discovery, reverse-proxying, centralized configuration, etc. is not worth it. I only learned this the hard way myself--my previous role had what you described above and while it kinda sorta worked, I was of the opinion that microservices weren't working (they were using NetKernel for a lot of their services, too, kind of a weird, mixed environment spanning multiple generations of service architectures). At my current job, I learned what I didn't know! This place has a decent microservice infrastructure without all the lumps using Netflix's OSS suite. The only major drawback we're having now is that as we drive harder to move services into real cloud deployments, we are finding out that Kubernetes would be a huge help and that it conflicts with our existing service discovery and so we may have to re-architect some of it. The good news is that doing so isn't impossible, it just takes planning and effort. One thing I find very annoying here is that the developer setup doesn't really look like QA or PROD and that our use of Docker is in its infancy and that the developers can't agree on a lot of things. As an aside, several people won't let go of Eclipse. Overall, NetKernel was total hell for developers, total hell, and migrating off of it cost millions, but was ultimately successful. NetKernel was the only game in town for REST services at one time, however. That company had an architect who was a NetKernel fanboi and who pushed them in a bad direction years after they should have been migrating to something else--that was another HUGE lesson for me--that personality and political clout had far more to do with their decision making than anything else. The guy bullied people because he didn't want to move off of what he knew. Once he finally left the company, the deluge started and within one year, NetKernel was dead.

eadmund 9y ago

> They developed 12 seperate applications, all in the same repo, deployed independently it's own JVM. I've dealt with an even worse system, with a dozen separate applications, each in its own repo, then with various repos containing shared code. But the whole thing was really one interconnected system, such that a change to one component often required changes to the shared code, which required updates to all the other services. It was a nightmare. At least your folks had the good sense to use a single repository.

paulddraper 9y ago
Agreed. Multiple closely dependent repos by one organization is the real nightmare.
crescentfresh 9y ago
> then with various repos containing shared code What source control system? Also, from the article: > even though theoretically services can be deployed in isolation, you find that due to the inter-dependencies between services, you have to deploy sets of services as a group This is the situation we are in, like you were.
- eadmund 9y ago
  > What source control system? Git in our case. And our direction was not to use submodules or anything like that to make life manageable. It was pretty unpleasant.

z3t4 9y ago

You don't always have to know why but it's somewhat frightening that so many "engineers" don't have a clue why they are doing something (because Google does it). And I'm of course guilty of it myself jumping on the hype-train or uncritically taking advice from domain experts only to find out years later that much of it was BS. Most of the time though, you will not reach enlightenment. I guess it's in our nature to follow authority, hype, trends and group think.

walrus1066 9y ago
They're only following their incentives. What's gonna look better on a devs CV: 'spent a year maintaining a CRUD monolith app' vs 'spent a year breaking monolith into microservices, with shiny language X to boot'. We can be a very fashion and buzzword driven industry sometimes. EDIT: this perverse incentive goes all they way to the top, through to CTO level. Sometimes I wonder if businesses understand just how much money and effort is wasted on pointless rewrites that make life harder for everyone.
- therealdrag0 9y ago
  "Resume driven development" :)
michaelcampbell 9y ago
> it's somewhat frightening that so many "engineers" don't have a clue why they are doing something (because Google does it). This doesn't stop at engineering; open offices, teaser/trick based interviewing, OKR's, ... Even GOOGLE doesn't do some of those things anymore, but the follower sheep still do.

merb 9y ago

> Of course since they had 12 different JVM applications, the services needed a host with at least 9GiB of RAM so they used a larger instance. well experimentally oracle solved that problem, somewhat. you could now use CDS and *.so files for some parts of your application. it probably does not eliminate every problem, but yeah it helps a bit at least. but well it would've been easier to just use apache felix or so to start all the applications on a OSGI container. this would've probably saved like 5-7 GiB of RAM.

thebeardedone 9y ago

I recently had a similar experience; our product at work is a monolith not in the greatest shape as it has technical debt which we inherited and our product is usually used condescendingly when talking to other teams working on different products. To our surprise when we started testing it with cloud deployments, it was really lightweight compared to just one of the 25 java micro-services from the other teams. Their "microservices" suffered from the same JVM overhead and to remedy this they are joining their functionalities together (initially they had 30-40).

virmundi 9y ago
I'm switching to go partially due to the jvm. Hopefully I'll get better partitioning on a single small box as I start.

Clubber 9y ago

>They were having performance problems and "needed" to migrate to microservices. They developed 12 seperate applications, all in the same repo, deployed independently it's own JVM. 9 times out of 10 it's because developers don't know how to properly design and index the underlying RDBMS. I've noticed there is a severe lack of knowledge of that for the average developer.

giancarlostoro 9y ago

Sounds like they don't understand why it's called a microservice to begin with. They're not supposed to be solutions an entire piece of software, just dedicated bits at least that's what I'd figure with a name such as "micro". When we adopted microservices at my job (idk if Azure Functions count or not) we did it because we had 1 task we needed taken out of our main application for performance concerns and because we knew it would involve way more work to implement (.NET Framework codebase being ported to .NET Core which meant the dependencies from .NET Framework did not work anymore in .NET Core) but we eventually turned it into a WebAPI instead due to limitations of Azure Functions for what we wanted to do (process imagery of sorts).

bonesss 9y ago
> idk if Azure Functions count or not Azure Functions are technically a "serverless" product, but using them as y'all intended to is a textbook definition of a "microservice" :)

ataturk 9y ago

Microservices without using something like Spring cloud which offers service discovery, reverse-proxying, centralized configuration, etc. is not worth it. I only learned this the hard way myself--my previous role had what you described above and while it kinda sorta worked, I was of the opinion that microservices weren't working (they were using NetKernel for a lot of their services, too, kind of a weird, mixed environment spanning multiple generations of service architectures). At my current job, I learned what I didn't know! This place has a decent microservice infrastructure without all the lumps using Netflix's OSS suite. The only major drawback we're having now is that as we drive harder to move services into real cloud deployments, we are finding out that Kubernetes would be a huge help and that it conflicts with our existing service discovery and so we may have to re-architect some of it. The good news is that doing so isn't impossible, it just takes planning and effort. One thing I find very annoying here is that the developer setup doesn't really look like QA or PROD and that our use of Docker is in its infancy and that the developers can't agree on a lot of things. As an aside, several people won't let go of Eclipse. Overall, NetKernel was total hell for developers, total hell, and migrating off of it cost millions, but was ultimately successful. NetKernel was the only game in town for REST services at one time, however. That company had an architect who was a NetKernel fanboi and who pushed them in a bad direction years after they should have been migrating to something else--that was another HUGE lesson for me--that personality and political clout had far more to do with their decision making than anything else. The guy bullied people because he didn't want to move off of what he knew. Once he finally left the company, the deluge started and within one year, NetKernel was dead.