5 ms·

>You realize that it was so trivial that there was no need for a library in the first place Minimalism can bite you, too. We have around 30 microservices writt

by kgeist 4y ago

>You realize that it was so trivial that there was no need for a library in the first place Minimalism can bite you, too. We have around 30 microservices written in Go, each one one of them living in their own repository (separate codebases owned by different teams). Initially we followed Go's idiom "a little copying is better than a little dependency". So every team simply was borrowing code snippets from other microservices' codebases without creating a common shared library, because those snippets were "trivial". Then it hit production and we found that under load/diverse user input many of those trivial functions were quite buggy or inefficient with edge cases. Since those functions/classes were copypasted over multiple codebases, now you had to contact owners of 30 repositories and coordinate bugfixing in all of them, which is slow and painful. Now we're back to having a shared library of common functions, because it's well tested and it's easy to fix bugs by just upgrading. Often what seems trivial is not trivial at all.

withinboredom 4y ago
It sounds like what you described is not minimalism at all, except from the perception of individual teams. From the perspective of the product, your solution is minimalism.
mojuba 4y ago
I'd argue that's not minimalism. A lot of the times it relies on the engineer's intuition: is this trivial enough that it can be copied? If as you say those trivial functions were buggy and nobody saw the red flags while copying so many times by so many engineers, then you have an organizational problem.
- kgeist 4y ago
  >If as you say those trivial functions were buggy and nobody saw the red flags while copying so many times by so many engineers, then you have an organizational problem. If all potential problems with code could be noticed on spot, we wouldn't have the concept of "bugs". Yes, I mentioned minimalism in my post, but I was specifically referring to the idea of copying "trivial" code as opposed to using a dependency.
  - mojuba 4y ago
    I was talking about the kind of intuition that while looking at the code you should know whether it's safe to copy it or not, i.e. whether it should be examined closely before copying. From your story, the same class was copied 30 times by that many engineers (?) and it didn't raise any red flags, it means (and I'll be a bit blunt) you have a problem with your hiring process.
    - kgeist 4y ago
      Well, not every engineer is experienced enough to immediately notice all potential problems which can arise in production. It worked in microservice A, why won't it work in microservice B? It all comes down to expertise. The average engineer is probably better off relying on a versioned, upgradable dependency maintained by more experienced folks than on copying code snippets here and there.
    - watwut 4y ago
      When things like this happen, it is in 98% because adding dependency is comething organization discourages while copying is without punishment. Discouragement can be you having to defend it in too uncomfortable process or something similar.
heavyset_go 4y ago
This is a great example of dogmatism prevailing in this industry, and the resultant cargo cult programming. There's a long-standing trend of inflexible dogmatism that's pervasive in software development. If some thought leader, blog or Google/FB/Microsoft/etc said to do something in one way, then things must be done that way. I once had someone get red in the face and call me stupid because I suggested that hiring more employees to run things after-hours would be preferable to forcing developers to spend their free time working on-call, because apparently Google's SRE bible says that developers must support operations themselves, even if that means they're doing free work in the middle of the night after putting in their 40+ hours a week in. That was unacceptable because Google said so, who cares if employees are leaving in droves and those that remain are spread so thin that their contributions during actual work hours were suffering. Google knows best, we must do what Google says to do, even if that tanks productivity and retention. The company eventually hired teams in different timezones to keep things running 24/7.
- kgeist 4y ago
  >even if that means they're doing free work in the middle of the night after putting in their 40+ hours a week in We follow this too, but additional work is paid. It kind of makes sense, because developers who own a service can solve a problem faster and with a better solution than someone from a different team, because they know the full context. In practice, I had to work at night only a few times when production was down. If it's not supercritical, it can wait till the morning.
  - heavyset_go 4y ago
    I have no problem with voluntary on-call duty that is both paid handsomely with overtime rates, even for salaried employees, along with bonuses that make sacrificing free time worth it. That's how on-call duty works in many European countries, by law, and in unionized industries and companies here in the US. The company didn't push for on-call duty at all for our EU developers, because that would cost the company more money for higher rates and bonuses. Instead, on-call duty was the responsibility of US and Canadian workers, because the company didn't have to pay them for the extra burden and loss of free time. EU developers worked on the same products, and could have done on-call duty with us, but that was never on the table for reasons that had nothing to do with developers being familiar with the projects they're supporting. It was all about reducing costs and getting free work out of developers dressed up as "best practices" from Google.
yakshaving_jgt 4y ago
> Minimalism can bite you, too. We have around 30 microservices written in Go… I guess it depends on your idea of minimalism. To me, having 30 microservices sounds tremendously complex. The minimalist in me would want to see those 30 microservices converge into a single project.
- kgeist 4y ago
  We had a large, complex, spaghetti-driven monolith before the microservices. Aside from a few microservices which support our infrastructure, a microservice usually serves a single subproduct; several subproducts are united under one UI, they only share the user model and communicate via events. I didn't mean our microservice architecture is minimalist (although when viewed alone, a microservice itself is pretty simple), I was giving an example as to why copying code has disadvantages, too, such as once you decide to copy the code instead of using a dependency you are completely on your own, you will miss any new security, performance updates and bugfixes; even supposedly trivial code can be ridden with subtle bugs.
- mbrodersen 4y ago
  You are right. IMHO there is zero justification for making any application that complex.
jghn 4y ago
Perhaps this is an indication that there should be fewer than 30 microservices and/or less of a focus on "micro"
- ziml77 4y ago
  That’s an odd judgement to make when you don’t know all of the systems and use cases that those services support.
  - jghn 4y ago
    You might have missed the word "Perhaps" at the start?
8note 4y ago
From a microservices perspective, wouldn't you make a microservices to host that logic instead of copying the code for it everywhere?
- DeathArrow 4y ago
  And they'll end up with 300 microservices instead of 30.
mbrodersen 4y ago
30 microseervices? Seriously? What on earth is the justification for having that amount of complexity? I have worked with very complex, very large monolith and client/server applications for 25+ years. And I never had trouble maintaining the code or add functionality. My guess is that using microservices was basically an excuse to rewrite the application? And that rewriting it to a new cleanly designed monolith might have been a better choice. Using “advanced” technologies like Modules and Libraries. A friend of mine has taken over a legacy (20+ years) 100+ microservices application. And it is an absolute nightmare. The complexity of building and deploying it is the worst I have ever seen. 10x worse than then worst monolith I have ever worked with. Debugging is a nightmare. Synchronisation issues makes auto testing almost impossible. The only good thing is that all the microservices are written in the same programming language. Imagine if 5+ different languages and build systems were used instead? The complexity would have been out of this world.
- deleted 4y ago
  [deleted]

by kgeist 4y ago

withinboredom 4y ago

It sounds like what you described is not minimalism at all, except from the perception of individual teams. From the perspective of the product, your solution is minimalism.

mojuba 4y ago

I'd argue that's not minimalism. A lot of the times it relies on the engineer's intuition: is this trivial enough that it can be copied? If as you say those trivial functions were buggy and nobody saw the red flags while copying so many times by so many engineers, then you have an organizational problem.

kgeist 4y ago
>If as you say those trivial functions were buggy and nobody saw the red flags while copying so many times by so many engineers, then you have an organizational problem. If all potential problems with code could be noticed on spot, we wouldn't have the concept of "bugs". Yes, I mentioned minimalism in my post, but I was specifically referring to the idea of copying "trivial" code as opposed to using a dependency.
- mojuba 4y ago
  I was talking about the kind of intuition that while looking at the code you should know whether it's safe to copy it or not, i.e. whether it should be examined closely before copying. From your story, the same class was copied 30 times by that many engineers (?) and it didn't raise any red flags, it means (and I'll be a bit blunt) you have a problem with your hiring process.
  - kgeist 4y ago
    Well, not every engineer is experienced enough to immediately notice all potential problems which can arise in production. It worked in microservice A, why won't it work in microservice B? It all comes down to expertise. The average engineer is probably better off relying on a versioned, upgradable dependency maintained by more experienced folks than on copying code snippets here and there.
  - watwut 4y ago
    When things like this happen, it is in 98% because adding dependency is comething organization discourages while copying is without punishment. Discouragement can be you having to defend it in too uncomfortable process or something similar.

heavyset_go 4y ago

This is a great example of dogmatism prevailing in this industry, and the resultant cargo cult programming. There's a long-standing trend of inflexible dogmatism that's pervasive in software development. If some thought leader, blog or Google/FB/Microsoft/etc said to do something in one way, then things must be done that way. I once had someone get red in the face and call me stupid because I suggested that hiring more employees to run things after-hours would be preferable to forcing developers to spend their free time working on-call, because apparently Google's SRE bible says that developers must support operations themselves, even if that means they're doing free work in the middle of the night after putting in their 40+ hours a week in. That was unacceptable because Google said so, who cares if employees are leaving in droves and those that remain are spread so thin that their contributions during actual work hours were suffering. Google knows best, we must do what Google says to do, even if that tanks productivity and retention. The company eventually hired teams in different timezones to keep things running 24/7.

kgeist 4y ago
>even if that means they're doing free work in the middle of the night after putting in their 40+ hours a week in We follow this too, but additional work is paid. It kind of makes sense, because developers who own a service can solve a problem faster and with a better solution than someone from a different team, because they know the full context. In practice, I had to work at night only a few times when production was down. If it's not supercritical, it can wait till the morning.
- heavyset_go 4y ago
  I have no problem with voluntary on-call duty that is both paid handsomely with overtime rates, even for salaried employees, along with bonuses that make sacrificing free time worth it. That's how on-call duty works in many European countries, by law, and in unionized industries and companies here in the US. The company didn't push for on-call duty at all for our EU developers, because that would cost the company more money for higher rates and bonuses. Instead, on-call duty was the responsibility of US and Canadian workers, because the company didn't have to pay them for the extra burden and loss of free time. EU developers worked on the same products, and could have done on-call duty with us, but that was never on the table for reasons that had nothing to do with developers being familiar with the projects they're supporting. It was all about reducing costs and getting free work out of developers dressed up as "best practices" from Google.

yakshaving_jgt 4y ago

> Minimalism can bite you, too. We have around 30 microservices written in Go… I guess it depends on your idea of minimalism. To me, having 30 microservices sounds tremendously complex. The minimalist in me would want to see those 30 microservices converge into a single project.

kgeist 4y ago
We had a large, complex, spaghetti-driven monolith before the microservices. Aside from a few microservices which support our infrastructure, a microservice usually serves a single subproduct; several subproducts are united under one UI, they only share the user model and communicate via events. I didn't mean our microservice architecture is minimalist (although when viewed alone, a microservice itself is pretty simple), I was giving an example as to why copying code has disadvantages, too, such as once you decide to copy the code instead of using a dependency you are completely on your own, you will miss any new security, performance updates and bugfixes; even supposedly trivial code can be ridden with subtle bugs.
mbrodersen 4y ago
You are right. IMHO there is zero justification for making any application that complex.

jghn 4y ago

Perhaps this is an indication that there should be fewer than 30 microservices and/or less of a focus on "micro"

ziml77 4y ago
That’s an odd judgement to make when you don’t know all of the systems and use cases that those services support.
- jghn 4y ago
  You might have missed the word "Perhaps" at the start?

8note 4y ago

From a microservices perspective, wouldn't you make a microservices to host that logic instead of copying the code for it everywhere?

DeathArrow 4y ago
And they'll end up with 300 microservices instead of 30.

mbrodersen 4y ago

30 microseervices? Seriously? What on earth is the justification for having that amount of complexity? I have worked with very complex, very large monolith and client/server applications for 25+ years. And I never had trouble maintaining the code or add functionality. My guess is that using microservices was basically an excuse to rewrite the application? And that rewriting it to a new cleanly designed monolith might have been a better choice. Using “advanced” technologies like Modules and Libraries. A friend of mine has taken over a legacy (20+ years) 100+ microservices application. And it is an absolute nightmare. The complexity of building and deploying it is the worst I have ever seen. 10x worse than then worst monolith I have ever worked with. Debugging is a nightmare. Synchronisation issues makes auto testing almost impossible. The only good thing is that all the microservices are written in the same programming language. Imagine if 5+ different languages and build systems were used instead? The complexity would have been out of this world.

deleted 4y ago
[deleted]