5 ms·

If history is a guide, then you are correct. But I believe complexity is fractal (I think we all know this: "turtles all the way down..."), and that you can str

by daenz 4y ago

If history is a guide, then you are correct. But I believe complexity is fractal (I think we all know this: "turtles all the way down..."), and that you can structure and manage complexity with recursive rules, so that you only see the resolution that you care about, for the areas that you care about, and everything else is a low frequency representation. Frameworks approximately do this, but for specific domains: they let you organize complexity into well-defined areas and put a pin in them, easing the cognitive load. Then you can handle those abstractions more easily. But yes, to your point, because frameworks typically only tackle a few levels of complexity, you still get that complexity back when you use the framework to advance the problems to the edge of what your framework is designed to address. A recursive/fractal management of complexity will allow all levels of the hierarchy to feel similar, so you are never increasing complexity, only looking at a different resolution. I think the key to this is mapping out the fundamental organizational problems and how they relate to each other at different resolutions.

ItsMonkk 4y ago
> tackling more complex problems This is the 'induced demand' argument. As perfection seems impossible, and mistakes inevitable, this seems likely. However, the more I look into issues, the more I realize that a very small number of errors introduced early on is what ultimately causes a plethora of them. Just fixing a very small amount of these mistakes would have untold effects on computing over the long-term. That is, if we can get over the initial switching costs. > small team of engineers It took 2 men to over-take thousands creating Unix over Multix. It took Linus Torvalds - just about alone - a week to create git. We have already seen this prophecy come true. As teams get bigger, communication sales factorially, and the more people you have the more mistakes you make, which increases the size exponentially with each further mistake. What Unix and git showed is that when you put everything into a small team of engineers heads, they can work through the complexity enough until they can do it themselves. > Frameworks One of the things I realized a while ago is that pure/impure and library/framework have a decent mapping between the two. If you have a framework, you give it code and it acts for you, just like impure code. And so the problem we keep running into, is instead of inverting the flow like 'hexagonal architecture' says, we keep piling impure onto impure onto impure. Hexagonal says to not do anything of substance, keep the adapter clean, but each framework sure is doing something. Each layer on the stack we go up, the harder it is to get down. So now we have OS and applications and containers and k8s running micro-services that ends up being run in a web browser, when all we really needed was microkernals.
- pjscott 4y ago
  > As teams get bigger, communication sales factorially, That sounds a bit drastic. Surely at worst the scaling is quadratic?
  - thyrsus 4y ago
    You are correct, but quadratic is sufficiently bad.
  - retcon 4y ago
    Correct, sales does have to scale factorially to the development man years. That's the problem with the prevailing economic contradiction. See Jamie Dimon, JPM, three days ago.. I've never witnessed a bulge bracket chief concluding the most public statements with, roughly, "doh, I dunno, it's crazy" before. My father cut his banking teeth in the Great Depression. That was actually much more simple. (Whether because the tools of the trade were more simple or just thankfully..) "Hurricane" is the search term for Dimon's delivery...
  - ItsMonkk 4y ago
    The number of channels is exponential, that's just Metcalfe's law. The ability to suss-out meaning behind those channels and come to a shared understanding pushes it to factorial. If A is speaking to B and C, A needs to also think about what communication is happening between B and C, and this is different to what B knows about C and C knows about B. In a carefully balanced classroom you might manage to get everyone knowing the same things, and this will allow knowledge to scale. Think about how much better TV got once they knew that viewers would watch every episode. What happens in software is that specialization quickly comes into play such that this is impossible in any organization that doesn't use mob programming. Other people might as well be speaking a different language. Next time you are in a Sprint meeting, think about how much you don't understand. Even as the team lead or architect who designs the entire system and whose job it is to understand everything it will be a shocking amount - it's black boxes all the way down. You'll claim that you can't know everything and that anyone who tries will fail. And this is made worse as the only true way to learn something is to do it, and if you aren't actually challenging yourself on something, anything you learn without doing will just fade away - as Spaced Repetition shows. The bigger the team, the bigger the software, the more inevitable the collapse.
nradov 4y ago
The complexities at different levels of the hierarchy are so qualitatively different that I doubt whether a unified set of abstractions could ever be practically useful. We're going all the way from CPU microcode to loosely-coupled distributed systems with multiple dependencies on third-party services. If you want to make any progress you'll need to narrow down your focus.
lr1970 4y ago
> If history is a guide, then you are correct. But I believe complexity is fractal Actually, "combinatorial" or "exponential" would be better words to describe complexity explosion. The number of possible cases/scenarios/logic flows/etc grows exponentially as a system's complexity increases. It is a curse and a blessing at the same time keeping so many SWEs gainfully employed.
- yourapostasy 4y ago
  "Fractal" is used to describe the property of not resolving the complexity detail until you get up close ("zoom into" the complexity). The amount of use cases does expand alarmingly, but even combinatorial and exponential use cases imply they are inventoried and are thus amenable to prioritization. In a lot of human work (not just software, but any endeavor involving technology even as "simple" as building a tunnel or simply involving many people coordinating together), complexity emerges this "unknown unknowns" property. This has happened since time immemorial. If you don't believe me, go master how to live out a subsistence life in an agrarian area of $nation during the 1500's with their tools but modern knowledge, then try to convince yourself complexity did not exist back then.
qwertyzxcvmnbv 4y ago
I think Tudor Girba has a similar vision. E.g. check out https://gtoolkit.com/ https://gtoolkit.com/ and his talks on software environmentalism, such as https://m.youtube.com/watch?v=N3l3eB62oSw https://m.youtube.com/watch?v=N3l3eB62oSw
edmcnulty101 4y ago
This is a very cool comment. I agree with you. We've made operating systems extremely simple from a UI standpoint then we host web frameworks on top of those operating systems. And then we make clusters on top of those frameworks and then we make datacenters out of cluster, etc. All still manageable by humans but at continually higher and higher levels of complexity.
- 7thaccount 4y ago
  I sometimes wonder if we need to go back to the drawing board or model our problems differently lol.
  - edmcnulty101 4y ago
    yeah I would bet money there's better ways of doing things. you seem to have thought about this a bit did you have any ideas of how things could be rearranged to a more recursive solution?
coip 4y ago
You seem like a fun person to work with

by daenz 4y ago

ItsMonkk 4y ago

> tackling more complex problems This is the 'induced demand' argument. As perfection seems impossible, and mistakes inevitable, this seems likely. However, the more I look into issues, the more I realize that a very small number of errors introduced early on is what ultimately causes a plethora of them. Just fixing a very small amount of these mistakes would have untold effects on computing over the long-term. That is, if we can get over the initial switching costs. > small team of engineers It took 2 men to over-take thousands creating Unix over Multix. It took Linus Torvalds - just about alone - a week to create git. We have already seen this prophecy come true. As teams get bigger, communication sales factorially, and the more people you have the more mistakes you make, which increases the size exponentially with each further mistake. What Unix and git showed is that when you put everything into a small team of engineers heads, they can work through the complexity enough until they can do it themselves. > Frameworks One of the things I realized a while ago is that pure/impure and library/framework have a decent mapping between the two. If you have a framework, you give it code and it acts for you, just like impure code. And so the problem we keep running into, is instead of inverting the flow like 'hexagonal architecture' says, we keep piling impure onto impure onto impure. Hexagonal says to not do anything of substance, keep the adapter clean, but each framework sure is doing something. Each layer on the stack we go up, the harder it is to get down. So now we have OS and applications and containers and k8s running micro-services that ends up being run in a web browser, when all we really needed was microkernals.

pjscott 4y ago
> As teams get bigger, communication sales factorially, That sounds a bit drastic. Surely at worst the scaling is quadratic?
- thyrsus 4y ago
  You are correct, but quadratic is sufficiently bad.
- retcon 4y ago
  Correct, sales does have to scale factorially to the development man years. That's the problem with the prevailing economic contradiction. See Jamie Dimon, JPM, three days ago.. I've never witnessed a bulge bracket chief concluding the most public statements with, roughly, "doh, I dunno, it's crazy" before. My father cut his banking teeth in the Great Depression. That was actually much more simple. (Whether because the tools of the trade were more simple or just thankfully..) "Hurricane" is the search term for Dimon's delivery...
- ItsMonkk 4y ago
  The number of channels is exponential, that's just Metcalfe's law. The ability to suss-out meaning behind those channels and come to a shared understanding pushes it to factorial. If A is speaking to B and C, A needs to also think about what communication is happening between B and C, and this is different to what B knows about C and C knows about B. In a carefully balanced classroom you might manage to get everyone knowing the same things, and this will allow knowledge to scale. Think about how much better TV got once they knew that viewers would watch every episode. What happens in software is that specialization quickly comes into play such that this is impossible in any organization that doesn't use mob programming. Other people might as well be speaking a different language. Next time you are in a Sprint meeting, think about how much you don't understand. Even as the team lead or architect who designs the entire system and whose job it is to understand everything it will be a shocking amount - it's black boxes all the way down. You'll claim that you can't know everything and that anyone who tries will fail. And this is made worse as the only true way to learn something is to do it, and if you aren't actually challenging yourself on something, anything you learn without doing will just fade away - as Spaced Repetition shows. The bigger the team, the bigger the software, the more inevitable the collapse.

nradov 4y ago

The complexities at different levels of the hierarchy are so qualitatively different that I doubt whether a unified set of abstractions could ever be practically useful. We're going all the way from CPU microcode to loosely-coupled distributed systems with multiple dependencies on third-party services. If you want to make any progress you'll need to narrow down your focus.

lr1970 4y ago

> If history is a guide, then you are correct. But I believe complexity is fractal Actually, "combinatorial" or "exponential" would be better words to describe complexity explosion. The number of possible cases/scenarios/logic flows/etc grows exponentially as a system's complexity increases. It is a curse and a blessing at the same time keeping so many SWEs gainfully employed.

yourapostasy 4y ago
"Fractal" is used to describe the property of not resolving the complexity detail until you get up close ("zoom into" the complexity). The amount of use cases does expand alarmingly, but even combinatorial and exponential use cases imply they are inventoried and are thus amenable to prioritization. In a lot of human work (not just software, but any endeavor involving technology even as "simple" as building a tunnel or simply involving many people coordinating together), complexity emerges this "unknown unknowns" property. This has happened since time immemorial. If you don't believe me, go master how to live out a subsistence life in an agrarian area of $nation during the 1500's with their tools but modern knowledge, then try to convince yourself complexity did not exist back then.

qwertyzxcvmnbv 4y ago

I think Tudor Girba has a similar vision. E.g. check out https://gtoolkit.com/ https://gtoolkit.com/ and his talks on software environmentalism, such as https://m.youtube.com/watch?v=N3l3eB62oSw https://m.youtube.com/watch?v=N3l3eB62oSw

edmcnulty101 4y ago

This is a very cool comment. I agree with you. We've made operating systems extremely simple from a UI standpoint then we host web frameworks on top of those operating systems. And then we make clusters on top of those frameworks and then we make datacenters out of cluster, etc. All still manageable by humans but at continually higher and higher levels of complexity.

7thaccount 4y ago
I sometimes wonder if we need to go back to the drawing board or model our problems differently lol.
- edmcnulty101 4y ago
  yeah I would bet money there's better ways of doing things. you seem to have thought about this a bit did you have any ideas of how things could be rearranged to a more recursive solution?

coip 4y ago

You seem like a fun person to work with