4 ms·

I'd like to hear the logical conclusion of "Large Modules Considered Harmful." Is he implying infrastructure code (e.g., terraform or ansible or cloud formation

by Boxxed 8y ago

I'd like to hear the logical conclusion of "Large Modules Considered Harmful." Is he implying infrastructure code (e.g., terraform or ansible or cloud formation) should be split between the repositories for each component? Or just saying, "be smart about how you layout the infrastructure repo"?

brikis98 8y ago
More of the latter. Don't put all your eggs in one basket. Don't create a single "module" (i.e., single deployable thing) with 100,000 lines of code and all your infrastructure in it. Break things up into small, reusable, composable pieces. This is what you typically do in any general purpose programming language, and it turns out it's a very good idea with infrastructure-as-code languages too.
- btschaegg 8y ago
  So much this. I've gotten so tired of seeing the same story again and again: Someone sees something that's done by multiple pieces of code and goes "I´m gonna write a framework for that!". Two months later you've got an unmaintainable behemoth of a god class that does everything under the sun and drives everyone insane who has to look at it (a Cthulhu class, if you will). So, yes, I got to the same conclusion. Basically, solve every part of the problem: - in isolation/as focused as possible - as a library - with the easiest to use and most composable API you can come up with After that, if you have to, you can still glue them together into a framework-like thing. But at least, anyone can pick any feature out of it without succumbing to madness. Also, this is a great way to avoid the whole "Big Ball of Mud" problem that forces you to pull a whole ecosystem into your project although all you wanted to do was log something.
idunno246 8y ago
I think Segment’s terraform repos and blog show the logical conclusion of this pretty well. I’ve gone through the process of splitting up a single cfn stack for the entire environment, it was less about repo structure and more about deployment units for us https://github.com/segmentio/stack https://github.com/segmentio/stack
isodude 8y ago
Being struck by this a couple of times myself, if one component changes too much you end up with a large piece of code that has too much responsibility, never gets rewritten(because it takes too much time) and is bug prone because each commit is tough to know exactly how it affects everything. If you go with largish modules you need to be smart when you build it, if you build smaller modules you don't have to be that smart. Dumb is good, dumb is easier to explain and read for others (including yourself in 3 years). So, be dumb keep it simple, whatever that is in your case. If the code is easy maybe a giant repo is good, if the team is small. If the team is big maybe you have other means of validating access to your repo, but if not you need to split it up. But then commits need to be synced when pushed over several repos.. Edit: make sure it's possible to make clean commits and rewrite the whole code in small steps. Things Will Change(tm)
ams6110 8y ago
I would say it's be smart about it. I use Ansible quite a bit. When I first started, based on some examples I found, my play books were pretty complicated, with a lot of conditional steps, doing things (or not) based on results of previous steps, etc. What I found is that it's a lot easier to have a lot of roles that each do one little thing, and then use them in a play book as needed. A role might be as simple as installing a package, templating a configuration file, or maybe even just changing just one line in a config file. When roles are small and do just one thing, they are easy to combine in many different ways, and it's more obvious what's going on.

by Boxxed 8y ago

brikis98 8y ago

More of the latter. Don't put all your eggs in one basket. Don't create a single "module" (i.e., single deployable thing) with 100,000 lines of code and all your infrastructure in it. Break things up into small, reusable, composable pieces. This is what you typically do in any general purpose programming language, and it turns out it's a very good idea with infrastructure-as-code languages too.

btschaegg 8y ago
So much this. I've gotten so tired of seeing the same story again and again: Someone sees something that's done by multiple pieces of code and goes "I´m gonna write a framework for that!". Two months later you've got an unmaintainable behemoth of a god class that does everything under the sun and drives everyone insane who has to look at it (a Cthulhu class, if you will). So, yes, I got to the same conclusion. Basically, solve every part of the problem: - in isolation/as focused as possible - as a library - with the easiest to use and most composable API you can come up with After that, if you have to, you can still glue them together into a framework-like thing. But at least, anyone can pick any feature out of it without succumbing to madness. Also, this is a great way to avoid the whole "Big Ball of Mud" problem that forces you to pull a whole ecosystem into your project although all you wanted to do was log something.

idunno246 8y ago

I think Segment’s terraform repos and blog show the logical conclusion of this pretty well. I’ve gone through the process of splitting up a single cfn stack for the entire environment, it was less about repo structure and more about deployment units for us https://github.com/segmentio/stack https://github.com/segmentio/stack

isodude 8y ago

Being struck by this a couple of times myself, if one component changes too much you end up with a large piece of code that has too much responsibility, never gets rewritten(because it takes too much time) and is bug prone because each commit is tough to know exactly how it affects everything. If you go with largish modules you need to be smart when you build it, if you build smaller modules you don't have to be that smart. Dumb is good, dumb is easier to explain and read for others (including yourself in 3 years). So, be dumb keep it simple, whatever that is in your case. If the code is easy maybe a giant repo is good, if the team is small. If the team is big maybe you have other means of validating access to your repo, but if not you need to split it up. But then commits need to be synced when pushed over several repos.. Edit: make sure it's possible to make clean commits and rewrite the whole code in small steps. Things Will Change(tm)

ams6110 8y ago

I would say it's be smart about it. I use Ansible quite a bit. When I first started, based on some examples I found, my play books were pretty complicated, with a lot of conditional steps, doing things (or not) based on results of previous steps, etc. What I found is that it's a lot easier to have a lot of roles that each do one little thing, and then use them in a play book as needed. A role might be as simple as installing a package, templating a configuration file, or maybe even just changing just one line in a config file. When roles are small and do just one thing, they are easy to combine in many different ways, and it's more obvious what's going on.