4 ms·

I work on a massive monolith that has about 800 contributors and its just as complex to add something as simple as a user’s birthday, just not all the complexit

by bradhe 3y ago

I work on a massive monolith that has about 800 contributors and its just as complex to add something as simple as a user’s birthday, just not all the complexity is technological. It requires “organizational alignment” since you’re touching everyone’s code. There will be endless iterations on design and review. Sign off required by at least 2 architects. It will get added to multiple planning iterations. The actual code will take an afternoon or less. We’ll have to ensure we hit out 90% test coverage during code review but because of all the tests, it’ll be too big for one PR so it will need to be broken up into multiple PRs probably landing over multiple weekly releases. To facilitate that, we’ll put it behind a feature flag (of which there are currently 13,000). Once it hits production, and the dashboards/monitoring are put in place, it will get enabled and disabled over and over again as we’re not totally sure why our birthdate feature broke the metering service, but we think that’s the root cause—need to do a few weeks of analysis. Then, finally, in a year the engineer who was tasked with it will get a good performance rating, maybe be up for a promo! Just in time for him to jump into another project that’s failing horribly headed into its 3rd year in development: Allowing the user to set their timezone.

ramraj07 3y ago
This sounds like you work at a place that’ll bungle any tech stack paradigm.
deterministic 3y ago
That sounds crazy. Imagine how much worse it would be if it was a microservices system!
- scrlk 3y ago
  "Why is it so hard to display the birthday date on the settings page? Why can't we get it done this quarter?" "Look...I'm sorry, we've been over this, it's the design of our backend." https://www.youtube.com/watch?v=y8OnoxKotPQ https://www.youtube.com/watch?v=y8OnoxKotPQ
- ris58h 3y ago
  Totally agree. I don't see how microservices would solve this. Also monorepository doesn't mean that service is monolith.
  - bradhe 3y ago
    You (correctly) assumed that it's a monorepo, by the way. But yeah, you're being reductive overall. It's not a "microservices vs monolith" debate. I was using this example to show that people can fuck up any model.
    - deterministic 3y ago
      I agree that devs can and regularly do fuck up anything they touch.
theshrike79 3y ago
Some people who advocate for huge monorepos, like Google has, tend to forget that Google has whole teams building tools just to wrangle the huge singular codebase for refactoring and testing.
- bradhe 3y ago
  Bingo. Monorepos work because tooling makes it work. Just putting all your code in one place doesn't make it a monorepo--just makes it a mess.
  - gorgoiler 3y ago
    It was a mess before, but the mess wasn’t visible. Now the code is all visible to the IDE, it all compiles together, and tests together. You can see the mess and fix it. It’s possible to see enough of it to be able to change it and when you do the build stays green because you can make atomic changes. It’s less about why monorepos are good, and more about why on Earth would you split a product up over hard repo / tooling boundaries. Perhaps a Big Brain could get those boundaries right. I can’t, so when I need to move the boundary I would rather it was done by editing code inside a repo than coordinating changes between micro products. I’m obviously describing one scenario. What are the counter examples for where hard polyrepo boundaries are helpful? When your team are all IC1s?
    - jwestbury 3y ago
      > You can see the mess and fix it. The trouble is, you're often left unwinding a half-dozen layers of abstraction, and in many cases experiencing what looks like some sort of quantum code interaction, where changes in one part of the codebase seem to impact completely unrelated bits of code which now fail tests. The polyrepo + microservices approach helps enforce boundaries. It also makes rapid iteration easier with more limited tooling, because my code can no longer blow up someone else's code -- a problem which can be solved with dev toolchains, but those toolchains take a non-trivial amount of resources to support. I do agree, when you need to move boundaries, or need to refactor across multiple packages at once, a monorepo + monolith is great. (A monorepo + microservices can introduce many of the same problems as a polyrepo + microservices, though.)
aledalgrande 3y ago
OMG this sounds painful ;) both in processes and state of the codebase I can ship something in a couple of hours on a monolith where 3000+ engs work and deploy every day, but to be fair that's not what I've seen in any other company I worked at.
- PartiallyTyped 3y ago
  Where do you work at, and, are you hiring?
  - Sosh101 3y ago
    Ditto.
camgunz 3y ago
It sounds like this company doesn't have a competitive need to ship code changes, but when it does (i.e. a challenger appears or the moat disappears) it'll be trouble. I've tried--with varying levels of success--to really take to heart that software engineering is a never ending battle against complexity. You have to do it all the time and it has to be your paramount value, otherwise stuff like this happens. I don't think there's an architecture or ideology that ends the war; this is just the nature of the job.
andrewstuart 3y ago
Ugh that would make me hate software.
srvaroa 3y ago
This comment shows IMO that the real issue here is not really microservices or not microservices, but what the article calls "The apostles of the Church of Complexity". Neither microservices or monolith are a golden hammer, silver bullet or whatever (nor the opposite). They are tools each with their tradeoffs, which combine with the many context-dependent tradeoffs of each organization. They are not the problem. The problem is a) engineer's fascination with complexity, and confusing "simple" with "hack", b) how organizations keep cargo-culting tools, architectural patterns, and so forth. Applying architectures, design patterns, whatever naively, based on the belief that usage alone will deliver benefits. It doesn't.
- appplication 3y ago
  I don’t think complexity is usually intentional though. Really, writing complex code is easy. It’s writing simple code that’s hard. I’m in the process of wrapping up a reactor of one of our systems that I wrote a few years ago. Largely the refactor has been an exercise in just simplifying everything. The only reason I can effectively do this now is because we have a much better idea of our use case and how the entire system fits together. At the time, the complicity seemed necessary, though I’m not quite sure why. I think I just didn’t really understand what exactly this was really going to do when we were building it. I think this probably applies to companies growing quickly as well. Beyond core product, there’s a lot of things you can do, but what benefit does it bring? How does it fit into overall strategy? It’s easier to build something if you just carve off your own little area to ignore the rest of the system and assume you’ll eventually understand better how of make it integrate nicely or add value.
martypitt 3y ago
I think this is a great point, and shouldn't be just hand-waved away like you hit some bizarre edge case of Monoliths. Monoliths can have crippling downsides -- just different flavours of downsides from Microservices. What you gain in network latency and DRYness, you can lose in Autonomy and Breadth of codebase. Microservices vs Monoliths - just like everything else - is a question of tradeoffs, and making informed choices about when to apply them, and how to mitigate their downsides. The slightly more nuanced point in the OP's article is that adopting any engineering practice and blindly following as though your identity is linked to it, is a bad move.
danmaz74 3y ago
Having 13,000 feature flags is just insane. I'm so happy that at my previous job I insisted that we use "release" flags for every new feature were there wasn't an explicit requirement to be able to enable it for some customers and disable it for other ones. We made it so that you couldn't enable one of these release flags in production, and developers were required to remove the code for the flag before final tests and release. It requires a little more work for every feature which requires a flag, but it really pays off in terms of reduced code base complexity in just a few months.
- paledot 3y ago
  It is indeed insane, but there is some value to leaving coarse feature flags in production. I've seen outages resolved by flipping a feature flag that was accidentally left in production months after deployment, where without it we would've had to resort to more drastic or hackish measures to restore service. (Eg. If one query is overloading the database, you can disable the feature that runs the query and investigate the real problem at your leisure while the database stabilizes.) That only works in moderation, though - with 13k flags, there's no way the particular combination that you're about to release directly on production has been properly tested.
  - borplk 3y ago
    This is also known as a "kill switch" which is technically slightly different than feature flags that get used for release safety. You can bake kill switches into many features as a core part of the application for operational reasons. One similar thing I like is having global "maintenance" switches so you can gracefully put an application into maintenance mode.
    - paledot 3y ago
      Yeah, fair enough. At some point my ex-employer wised up and implemented kill switches on purpose instead of by accident.
crabbone 3y ago
Sounds like DB2?
- bradhe 3y ago
  Sounds like DB2.
deleted 3y ago
[deleted]

by bradhe 3y ago

ramraj07 3y ago

This sounds like you work at a place that’ll bungle any tech stack paradigm.

deterministic 3y ago

That sounds crazy. Imagine how much worse it would be if it was a microservices system!

scrlk 3y ago
"Why is it so hard to display the birthday date on the settings page? Why can't we get it done this quarter?" "Look...I'm sorry, we've been over this, it's the design of our backend." https://www.youtube.com/watch?v=y8OnoxKotPQ https://www.youtube.com/watch?v=y8OnoxKotPQ
ris58h 3y ago
Totally agree. I don't see how microservices would solve this. Also monorepository doesn't mean that service is monolith.
- bradhe 3y ago
  You (correctly) assumed that it's a monorepo, by the way. But yeah, you're being reductive overall. It's not a "microservices vs monolith" debate. I was using this example to show that people can fuck up any model.
  - deterministic 3y ago
    I agree that devs can and regularly do fuck up anything they touch.

theshrike79 3y ago

Some people who advocate for huge monorepos, like Google has, tend to forget that Google has whole teams building tools just to wrangle the huge singular codebase for refactoring and testing.

bradhe 3y ago
Bingo. Monorepos work because tooling makes it work. Just putting all your code in one place doesn't make it a monorepo--just makes it a mess.
- gorgoiler 3y ago
  It was a mess before, but the mess wasn’t visible. Now the code is all visible to the IDE, it all compiles together, and tests together. You can see the mess and fix it. It’s possible to see enough of it to be able to change it and when you do the build stays green because you can make atomic changes. It’s less about why monorepos are good, and more about why on Earth would you split a product up over hard repo / tooling boundaries. Perhaps a Big Brain could get those boundaries right. I can’t, so when I need to move the boundary I would rather it was done by editing code inside a repo than coordinating changes between micro products. I’m obviously describing one scenario. What are the counter examples for where hard polyrepo boundaries are helpful? When your team are all IC1s?
  - jwestbury 3y ago
    > You can see the mess and fix it. The trouble is, you're often left unwinding a half-dozen layers of abstraction, and in many cases experiencing what looks like some sort of quantum code interaction, where changes in one part of the codebase seem to impact completely unrelated bits of code which now fail tests. The polyrepo + microservices approach helps enforce boundaries. It also makes rapid iteration easier with more limited tooling, because my code can no longer blow up someone else's code -- a problem which can be solved with dev toolchains, but those toolchains take a non-trivial amount of resources to support. I do agree, when you need to move boundaries, or need to refactor across multiple packages at once, a monorepo + monolith is great. (A monorepo + microservices can introduce many of the same problems as a polyrepo + microservices, though.)

aledalgrande 3y ago

OMG this sounds painful ;) both in processes and state of the codebase I can ship something in a couple of hours on a monolith where 3000+ engs work and deploy every day, but to be fair that's not what I've seen in any other company I worked at.

PartiallyTyped 3y ago
Where do you work at, and, are you hiring?
- Sosh101 3y ago
  Ditto.

camgunz 3y ago

It sounds like this company doesn't have a competitive need to ship code changes, but when it does (i.e. a challenger appears or the moat disappears) it'll be trouble. I've tried--with varying levels of success--to really take to heart that software engineering is a never ending battle against complexity. You have to do it all the time and it has to be your paramount value, otherwise stuff like this happens. I don't think there's an architecture or ideology that ends the war; this is just the nature of the job.

andrewstuart 3y ago

Ugh that would make me hate software.

srvaroa 3y ago

This comment shows IMO that the real issue here is not really microservices or not microservices, but what the article calls "The apostles of the Church of Complexity". Neither microservices or monolith are a golden hammer, silver bullet or whatever (nor the opposite). They are tools each with their tradeoffs, which combine with the many context-dependent tradeoffs of each organization. They are not the problem. The problem is a) engineer's fascination with complexity, and confusing "simple" with "hack", b) how organizations keep cargo-culting tools, architectural patterns, and so forth. Applying architectures, design patterns, whatever naively, based on the belief that usage alone will deliver benefits. It doesn't.

appplication 3y ago
I don’t think complexity is usually intentional though. Really, writing complex code is easy. It’s writing simple code that’s hard. I’m in the process of wrapping up a reactor of one of our systems that I wrote a few years ago. Largely the refactor has been an exercise in just simplifying everything. The only reason I can effectively do this now is because we have a much better idea of our use case and how the entire system fits together. At the time, the complicity seemed necessary, though I’m not quite sure why. I think I just didn’t really understand what exactly this was really going to do when we were building it. I think this probably applies to companies growing quickly as well. Beyond core product, there’s a lot of things you can do, but what benefit does it bring? How does it fit into overall strategy? It’s easier to build something if you just carve off your own little area to ignore the rest of the system and assume you’ll eventually understand better how of make it integrate nicely or add value.

martypitt 3y ago

I think this is a great point, and shouldn't be just hand-waved away like you hit some bizarre edge case of Monoliths. Monoliths can have crippling downsides -- just different flavours of downsides from Microservices. What you gain in network latency and DRYness, you can lose in Autonomy and Breadth of codebase. Microservices vs Monoliths - just like everything else - is a question of tradeoffs, and making informed choices about when to apply them, and how to mitigate their downsides. The slightly more nuanced point in the OP's article is that adopting any engineering practice and blindly following as though your identity is linked to it, is a bad move.

danmaz74 3y ago

Having 13,000 feature flags is just insane. I'm so happy that at my previous job I insisted that we use "release" flags for every new feature were there wasn't an explicit requirement to be able to enable it for some customers and disable it for other ones. We made it so that you couldn't enable one of these release flags in production, and developers were required to remove the code for the flag before final tests and release. It requires a little more work for every feature which requires a flag, but it really pays off in terms of reduced code base complexity in just a few months.

paledot 3y ago
It is indeed insane, but there is some value to leaving coarse feature flags in production. I've seen outages resolved by flipping a feature flag that was accidentally left in production months after deployment, where without it we would've had to resort to more drastic or hackish measures to restore service. (Eg. If one query is overloading the database, you can disable the feature that runs the query and investigate the real problem at your leisure while the database stabilizes.) That only works in moderation, though - with 13k flags, there's no way the particular combination that you're about to release directly on production has been properly tested.
- borplk 3y ago
  This is also known as a "kill switch" which is technically slightly different than feature flags that get used for release safety. You can bake kill switches into many features as a core part of the application for operational reasons. One similar thing I like is having global "maintenance" switches so you can gracefully put an application into maintenance mode.
  - paledot 3y ago
    Yeah, fair enough. At some point my ex-employer wised up and implemented kill switches on purpose instead of by accident.

crabbone 3y ago

Sounds like DB2?

bradhe 3y ago
Sounds like DB2.

deleted 3y ago

[deleted]