4 ms·

What frustrates me about the software industry is the many failed lineages problem. There are many mountains of code that get the job done at various companies

What frustrates me about the software industry is the many failed lineages problem. There are many mountains of code that get the job done at various companies but isn't shared. The lineage of these in-house frameworks or effective solutions to problems kind of don't go anywhere, they simply end. The lineage ends and doesn't cross pollenate. So lessons aren't shared. We are in an era of explosive growth where there is a new framework or new library or new technology introduced. I learn from whitepapers and reading English descriptions of people's problems. I find this easier than reading someone else's code. Code is extremely powerful but it has a very high maintenance cost and change costs. Just getting up to speed on the Postgresql codebase or the Linux kernel or any other system tool is hard work and an investment. You kind of need to work 40 hours a week to get familiar with a codebase to be effective in it. But I can get familiar with a problem outside the context of a codebase by reading a whitepaper and writing some code. Reading a whitepaper feels more effective use of time than reading someone's codebase. You need to jump around a lot of code to understand the codebase. I think the effort for the ideal solution is underestimated. People abstract to their own understanding. And that abstraction might not be intuitive to other minds. I would prefer to work on a codebase that solves problems effectively than a codebase that is novel.

sshine 4y ago
I can't help but think that FOSS is the key to sharing the lineages. Every single place I worked, people just want to make code work, not readable.
bdg 4y ago
> I would prefer to work on a codebase that solves problems effectively than a codebase that is novel. I have a design for a very very boring code base that involves no frameworks or external libs. It works fantastic and has been used by some of the most productive teams I've managed. Those teams can manage "more services than we have people on the team by a scale of 2-4x" (quote from one of the engineers on one of those teams). But it's crazy boring. There's no frameworks to download, not dependencies to update, no big brand pages. It's just following some very simple rules about where to put different kinds of complexity, very strictly. It's a hard sell in any programming ecosystem because it has no brand power. I've debated create the "no framework" spec around this. I even have a tool that can enforce the standards in CI. I just don't know if anyone would actually care.
- conickal 4y ago
  This sounds interesting to me!
  - brobdingnagians 4y ago
    I second that. I really like the idea.
- AndreasHae 4y ago
  Would love to see this!
- hazz99 4y ago
  You should share the design principles!
  - tartoran 4y ago
    And make a generic framework /s
- porker 4y ago
  Is it open source or only available to clients?
  - bdg 4y ago
    I think I grossly oversold this thing because there's a lot of comments here asking for something. I don't really have this concept written down anywhere like a number of other ideas I have. But, I guess the short version is, if I had to make an elevator pitch or something: No framework is a configuration (maybe "distro" in the linux sense) of concepts (maybe "packages" in the software sense). A concept is either something you might use a framework or library for (and usually it exists somewhere), or it is something you would want a linter to find, and it might even be something that you want to ensure was done correctly at code review. I think this last one is the most accurate idea of what a "concept" is. Over time I have accumulated a small informal set of "packages" that can be implemented without a helper library in nearly the same amount of code as if you were to use that library anyway. The important part is that the running software doesn't depend on the third party code, but actually the developers depend on a rule book and anything that violates the rules should be treated the same as calling an third party package's API method that doesn't exist. In other words: the dependency remains entirely in concept-space, not disk space. This link below is not "no framework" but it is something I wrote where you can see the result of "no framework thinking". The concepts are stole from people who are probably smarter than me, have decades of experience and written books on these topics. The only difference is instead of turning it into a library to depend on, it's turned into rules for humans (which I guess is also what the book authors originally did anyway). I combined them and made them into a "distro" and I called it "modular provider architecture" (not very engaging or entertaining, but it does what's on the label). - https://github.com/Incognito/python-architecture-linter/tree/main/modular_provider_architecture_definition https://github.com/Incognito/python-architecture-linter/tree... That text document is meant to be an example of how developers should write an application. By the way, it has a demo application here which does basically nothing: - https://github.com/Incognito/python-architecture-linter-demo/tree/main/demo_project https://github.com/Incognito/python-architecture-linter-demo... It might be hard to see here because it's pretty silly example, but I managed a small/growing team of 3-5 developers who create over 15 different services following this pattern. They did end up using libraries to do things like send data to/from Kafka or a DB, but the Modular Provider Architecture's rules were always there. Oh, by the way, that repo I linked to, https://github.com/Incognito/python-architecture-linter/ https://github.com/Incognito/python-architecture-linter/ ... this is a proof of concept for a linter that could implement the "no framework" concept. It is a dev dependency of your project, meaning you have no production framework as a dependency. It is a tool that lets you configure "rules" for your project in the style of any linter you already know of. It's like a linter from hyperspace, you can "lint" rules like.... if a file is 3 levels deep, and depended on by methods anywhere in the project with the word "bob" in the method name name, but those methods don't have if-statements, and also the Afferent coupling of the module itself is less than 0.5 .... fail CI with an explanation why. It also has a feature for you to commit an exemption list. I used this in my teams once I started managing multiple large teams, and I could do things like generate entire reports across all projects of these really complex metrics that most linters and tools aren't really set up for. That code is in these files, sorry for the total mess, I was just hacking around and didn't really think of a nice way to structure the definition "API. My main goal was proving the concept. - https://github.com/Incognito/python-architecture-linter/blob/main/modular_provider_architecture_definition/definition.py https://github.com/Incognito/python-architecture-linter/blob... - https://github.com/Incognito/python-architecture-linter/blob/main/modular_provider_architecture_definition/validators.py https://github.com/Incognito/python-architecture-linter/blob... ==== So, to summarise: - "Modular Provider Architecture" is a "Distro" of "No Framework". Others can exist. - "Modular Provider Architecture" is composed of "concept packages". Many of those packages have actual software libraries as alternatives. Others do not because they are rules about how to write code. - Really complex ideas in "Modular Provider Architecture" can be enforced via CI. I think the "beautiful" thing (if you think sofware can also be art), is that there is a clear an obvious structure to the project, but the "framework" is entirely ephemeral, just like types in typescript. I guess the really important thing here is: enforce your architecture's rules and pick them very carefully in concert, not in isolation. ==== By the way. That's one idea I have for removing dependencies on frameworks. I also have a really crazy idea for eliminating the need for most software by replacing semver, but it takes me over 2 hours to explain it (I have really bad video recordings where I try).
jstimpfle 4y ago
For mildly interesting code that you can realistically write yourself, there isn't a strict need to share code and cross polinate. Every little shop has a certain budget to try out new things, play around, and learn. If we come up with more or less the same things, that's fine. I'm much more fond of sharing ideas than sharing code. IMO developers' understanding is at least as important as the economics of crafting code. Of course you can't rewrite Postgres for your CRUD project. It's fine to use third party projects but there might be less need than one would assume. A lot of projects don't need a full-blown database, and I'd find it great if there was more practical reasearch how to design minimal storage systems for example. I'd bet that the majority of all apps that depend on Postgres would easily fit in RAM, forever. That could mean one could get rid of SQL, or could apply other simplifications.
- paulryanrogers 4y ago
  > I'd bet that the majority of all apps that depend on Postgres would easily fit in RAM, forever. That could mean one could get rid of SQL, or could apply other simplifications. My guess is you're correct, yet RAM stored DBs cannot guarantee persistence when things go wrong. Or at least they may introduce data loss if changes aren't fsync'd to durable storage or 2PC with replicas.
  - jstimpfle 4y ago
    It already simplifies some things if data is available directly from RAM. And many stores are read-only or read-mostly.

sshine 4y ago

I can't help but think that FOSS is the key to sharing the lineages. Every single place I worked, people just want to make code work, not readable.

bdg 4y ago

> I would prefer to work on a codebase that solves problems effectively than a codebase that is novel. I have a design for a very very boring code base that involves no frameworks or external libs. It works fantastic and has been used by some of the most productive teams I've managed. Those teams can manage "more services than we have people on the team by a scale of 2-4x" (quote from one of the engineers on one of those teams). But it's crazy boring. There's no frameworks to download, not dependencies to update, no big brand pages. It's just following some very simple rules about where to put different kinds of complexity, very strictly. It's a hard sell in any programming ecosystem because it has no brand power. I've debated create the "no framework" spec around this. I even have a tool that can enforce the standards in CI. I just don't know if anyone would actually care.

conickal 4y ago
This sounds interesting to me!
- brobdingnagians 4y ago
  I second that. I really like the idea.
AndreasHae 4y ago
Would love to see this!
hazz99 4y ago
You should share the design principles!
- tartoran 4y ago
  And make a generic framework /s
porker 4y ago
Is it open source or only available to clients?
- bdg 4y ago
  I think I grossly oversold this thing because there's a lot of comments here asking for something. I don't really have this concept written down anywhere like a number of other ideas I have. But, I guess the short version is, if I had to make an elevator pitch or something: No framework is a configuration (maybe "distro" in the linux sense) of concepts (maybe "packages" in the software sense). A concept is either something you might use a framework or library for (and usually it exists somewhere), or it is something you would want a linter to find, and it might even be something that you want to ensure was done correctly at code review. I think this last one is the most accurate idea of what a "concept" is. Over time I have accumulated a small informal set of "packages" that can be implemented without a helper library in nearly the same amount of code as if you were to use that library anyway. The important part is that the running software doesn't depend on the third party code, but actually the developers depend on a rule book and anything that violates the rules should be treated the same as calling an third party package's API method that doesn't exist. In other words: the dependency remains entirely in concept-space, not disk space. This link below is not "no framework" but it is something I wrote where you can see the result of "no framework thinking". The concepts are stole from people who are probably smarter than me, have decades of experience and written books on these topics. The only difference is instead of turning it into a library to depend on, it's turned into rules for humans (which I guess is also what the book authors originally did anyway). I combined them and made them into a "distro" and I called it "modular provider architecture" (not very engaging or entertaining, but it does what's on the label). - https://github.com/Incognito/python-architecture-linter/tree/main/modular_provider_architecture_definition https://github.com/Incognito/python-architecture-linter/tree... That text document is meant to be an example of how developers should write an application. By the way, it has a demo application here which does basically nothing: - https://github.com/Incognito/python-architecture-linter-demo/tree/main/demo_project https://github.com/Incognito/python-architecture-linter-demo... It might be hard to see here because it's pretty silly example, but I managed a small/growing team of 3-5 developers who create over 15 different services following this pattern. They did end up using libraries to do things like send data to/from Kafka or a DB, but the Modular Provider Architecture's rules were always there. Oh, by the way, that repo I linked to, https://github.com/Incognito/python-architecture-linter/ https://github.com/Incognito/python-architecture-linter/ ... this is a proof of concept for a linter that could implement the "no framework" concept. It is a dev dependency of your project, meaning you have no production framework as a dependency. It is a tool that lets you configure "rules" for your project in the style of any linter you already know of. It's like a linter from hyperspace, you can "lint" rules like.... if a file is 3 levels deep, and depended on by methods anywhere in the project with the word "bob" in the method name name, but those methods don't have if-statements, and also the Afferent coupling of the module itself is less than 0.5 .... fail CI with an explanation why. It also has a feature for you to commit an exemption list. I used this in my teams once I started managing multiple large teams, and I could do things like generate entire reports across all projects of these really complex metrics that most linters and tools aren't really set up for. That code is in these files, sorry for the total mess, I was just hacking around and didn't really think of a nice way to structure the definition "API. My main goal was proving the concept. - https://github.com/Incognito/python-architecture-linter/blob/main/modular_provider_architecture_definition/definition.py https://github.com/Incognito/python-architecture-linter/blob... - https://github.com/Incognito/python-architecture-linter/blob/main/modular_provider_architecture_definition/validators.py https://github.com/Incognito/python-architecture-linter/blob... ==== So, to summarise: - "Modular Provider Architecture" is a "Distro" of "No Framework". Others can exist. - "Modular Provider Architecture" is composed of "concept packages". Many of those packages have actual software libraries as alternatives. Others do not because they are rules about how to write code. - Really complex ideas in "Modular Provider Architecture" can be enforced via CI. I think the "beautiful" thing (if you think sofware can also be art), is that there is a clear an obvious structure to the project, but the "framework" is entirely ephemeral, just like types in typescript. I guess the really important thing here is: enforce your architecture's rules and pick them very carefully in concert, not in isolation. ==== By the way. That's one idea I have for removing dependencies on frameworks. I also have a really crazy idea for eliminating the need for most software by replacing semver, but it takes me over 2 hours to explain it (I have really bad video recordings where I try).

jstimpfle 4y ago

For mildly interesting code that you can realistically write yourself, there isn't a strict need to share code and cross polinate. Every little shop has a certain budget to try out new things, play around, and learn. If we come up with more or less the same things, that's fine. I'm much more fond of sharing ideas than sharing code. IMO developers' understanding is at least as important as the economics of crafting code. Of course you can't rewrite Postgres for your CRUD project. It's fine to use third party projects but there might be less need than one would assume. A lot of projects don't need a full-blown database, and I'd find it great if there was more practical reasearch how to design minimal storage systems for example. I'd bet that the majority of all apps that depend on Postgres would easily fit in RAM, forever. That could mean one could get rid of SQL, or could apply other simplifications.

paulryanrogers 4y ago
> I'd bet that the majority of all apps that depend on Postgres would easily fit in RAM, forever. That could mean one could get rid of SQL, or could apply other simplifications. My guess is you're correct, yet RAM stored DBs cannot guarantee persistence when things go wrong. Or at least they may introduce data loss if changes aren't fsync'd to durable storage or 2PC with replicas.
- jstimpfle 4y ago
  It already simplifies some things if data is available directly from RAM. And many stores are read-only or read-mostly.