8 ms·

Layered Programming (2013)

82 points by sktrdie 7y ago

zwieback 7y ago
Sometimes I think this is a good idea but generally it seems like refactoring in a way that the code base always makes sense, even to a newcomer, is the correct goal. History can be kept in source control, if it's necessary to follow the history of code development it's probably a sign that the abstractions are not chosen correctly.
- akkartik 7y ago
  (Author here.) The analogy to `git log` is just that, an analogy. Here's a concrete thing that you get with layers that you don't get with `git log` or conventional abstractions and refactoring: alternatives. Often we see comments in codebases that some complex, hairy function is really a much simpler function with certain optimizations applied. There's been lots of work done to try to separate out functional spec from optimization so that the two can be managed separately. But this is all open research. Layers let you get the value of this separation today, with very little code (just a lot of reorientation in mindset). If you adopt layers you can use the simple version in an earlier layer, and then replace it in a later layer. Your CI system will exercise both, and flag if either stops passing tests. What used to be a code comment now stays up to date.
  - zwieback 7y ago
    that's a good point about continuing testing reference and optimized solutions - I could see a case being made for standardizing that kind of thing
- joe_the_user 7y ago
  Well, what if you could have approach described in the article but there would be a defined patch interface so that if you looked at it, each patch makes sense also?
jimbo1qaz 7y ago
Fascinating idea. I've definitely written and encountered programs where functions were more complicated than they could've been, in order to support features separate from the "main idea" of the program. Like FPS calculation, Ctrl+C handling, passing intermediate audio data to extra buffers for visualization... Could "splicing in" code it be implemented using aspect-oriented programming frameworks? https://en.wikipedia.org/wiki/Aspect-oriented_programming https://en.wikipedia.org/wiki/Aspect-oriented_programming I've never used them though. > There's a new constraint that has no counterpart in current programming practice — remove a feature, and everything before it should build and pass its tests: > build_and_test_until Looks like added tooling is needed, but seems worthwhile.
hacker_9 7y ago
Well this is interesting, I recently spent a weekend creating a 'splicing compiler', which could take a typescript project split up into features, where a feature was written in just one single typescript file. Each section of code could then specify new 'code hooks', for future code to insert itself at during compilation. My original inspiration was that English language is spoken sequentially, but ends up in a graph database (i.e. the brain's neural network). So you'd write code like a thesis, and the hooks you specify would allow the splicer to form the code graph. In theory it seemed like it would be useful, so I built Pacman with it and, well, it didn't really work in practice! Things ended up being more abstract in the end, and I'd be mentally juggling where hooks were created and what the final code would look like. Even showing the code output to the right wasn't that helpful, as the way the code was spliced together put things in not the most readable order. Additionally how to split up the code wasn't the most obvious, i.e. when should a hook be added? It often felt I'd only know after writing the code in full. At which point it just felt like extra work. I'd often find myself to be inserting code all over the place to get something working, and so the refactoring needed at the end became overwhelming. Ultimately these reasons lead me to abandon the idea.
- akkartik 7y ago
  I've written 35k LoC with this tool, and it does require some taste but not intractably so. A couple of rules of thumb: - Don't use layers in too fine-grained a manner. Often if I need to change two lines in a function but they're separated by 6 lines I err on the side of replacing the entire paragraph or even the entire function. Each layer should be readable in a self-contained manner as much as possible. - I never invalidate tests in an earlier layer. You never want to be in a situation where a layer lies to you about what desired behavior is.
bruth 7y ago
Posted in 2013
SkyBelow 7y ago
I think the core problem is that one of the root causes of the patchwork hard to understand code is lack of spending the effort to keep the code understandable. Refactor into new models, move around code so it makes more sense given changes in requirements, etc. If that level of maintenance isn't being performed (which is almost a given, assuming we are looking at a problem with such a patchwork problem), then this new tool won't be fully followed and the end result will be a patchwork code base that reaches nightmarish levels where some features are added in this layered approach, but other features are added directly to the code, and even other features are added directly to prior layers. If the tooling force people to use this in the recommended way then it may have interesting payoffs that are worthwhile. But lacking such an enforced requirement, I don't see applying being beneficial except in the code bases that least benefit from it.
- akkartik 7y ago
  I find this approach to be easier than conventional approaches, and my motivation and effort levels seem constant on both sides. You're right that there's a social component here. This tool tries to make things easier for authors under the assumption that people try to use it tastefully.
canadaduane 7y ago
Some other, related ideas & paradigms: Context-Oriented Programming: https://arxiv.org/pdf/1105.0069.pdf https://arxiv.org/pdf/1105.0069.pdf Feature-Oriented Software Development: https://en.wikipedia.org/wiki/Feature-oriented_programming https://en.wikipedia.org/wiki/Feature-oriented_programming
- akkartik 7y ago
  Yes, I was aware of both as well as AOP when I built this. Some comments on OP discuss them as well. The big difference between my approach and these is the weight-to-thrust ratio. These academic approaches tend to assume people don't want to change the pieces, just combine them. And they create huge tools to make that possible. Why can't people just change the pieces? I've never understood that. I don't need an algebra of program transformations intermediating between me and my code. I'd much rather hold the code directly in my mind. And that keeps the tooling needed really, really small. Just a hundred or so lines of C. Tooling is a double-edged thing. Adding tooling often encourages increases in codebase scale. Keeping tooling rudimentary can be a good thing by keeping codebases small.
crazypython 7y ago
This is just AOP.
- canadaduane 7y ago
  It's related, but I don't think AOP has a concept of an immutable past?
  - akkartik 7y ago
    Hmm, layers don't have an immutable past. Can you elaborate?
    - canadaduane 7y ago
      Ah, I misunderstood. I thought the layer stack was immutable.
- akkartik 7y ago
  https://news.ycombinator.com/item?id=21766557#21767531 https://news.ycombinator.com/item?id=21766557#21767531 It's like AOP with user-definable join points.
_bxg1 7y ago
Similar to Spring dependency-injection: https://www.journaldev.com/2410/spring-dependency-injection https://www.journaldev.com/2410/spring-dependency-injection It seems too powerful, though. I could see it quickly becoming intractable to find where the code lives that is causing a certain outcome. The root problem it's trying to solve is the fact that things often don't fit neatly in a single bucket, but our traditional idea of modules as mutually-exclusive and hierarchical categories forces you to choose just one place to "put" something. An idea I've had is to get rid of hierarchical modules altogether. Really they're a relic of directories and files. Instead, one could "tag" a function/class/constant as belonging to multiple modules, and one's editor could aggregate all members of a cross-cutting module as needed. Organization within actual files on disk becomes an implementation detail.
- akkartik 7y ago
  > I could see it quickly becoming intractable to find where the code lives that is causing a certain outcome. (Author here.) The tangler I describe inserts #line directives into the output. You can open up the generated C++ and see where a line comes from. Conventional C++ tools understand line directives as well, so the debugger experience is sane. I'm sure there are gaps in this tooling, but they're not intractable. Tooling around conventional modules has matured over decades. My approach hopefully shows some benefits of a road not taken.
  - _bxg1 7y ago
    It's certainly interesting, and I didn't mean to pre-judge too strongly. I definitely think we're at the point where we need some new ideas around this stuff, and probably any new ideas at this point will require some kind of special editor assistance, so yeah, I get what you mean in that regard.
- akkartik 7y ago
  > Instead, one could "tag" a function/class/constant as belonging to multiple modules, and one's editor could aggregate all members of a cross-cutting module as needed. Organization within actual files on disk becomes an implementation detail. Yes, this is interesting. Are you thinking of something like Ruby's modules and `include` directives? (https://ruby-doc.com/docs/ProgrammingRuby/html/tut_modules.html https://ruby-doc.com/docs/ProgrammingRuby/html/tut_modules.h...) Is there a reason the editor has to do this rather than the language?
veddox 7y ago
So you‘re trying to remove complexity by making it invisible? Sounds great while things work, but doesn‘t that turn debugging into a nightmarish exercise? Because I never get to see the code that‘s actually executing, the pieces of a function might be scattered over a dozen different layers. Strikes me as the ultimate recipe for self-modifying spaghetti code... Or am I missing something here?
- akkartik 7y ago
  I'm not sure what you mean by 'invisible'. Notice that I make zero claims of reducing complexity. Layers are only a way to manage and stage complexity. I responded to tooling questions at https://news.ycombinator.com/item?id=21766557#21767393 https://news.ycombinator.com/item?id=21766557#21767393
  - magicalhippo 7y ago
    Well studying 010vm.cc, I wouldn't know that 014literal_string.cc alters the to_string function. That information is hidden from me, until I stumble upon it in 014literal_string.cc.
    - akkartik 7y ago
      Most of the time you wouldn't care. The presence of a waypoint is a hint that _something_ is inserting code there. But as long as tests in 010vm.cc continue to pass, it shouldn't matter to it what future layers do. When debugging things, however, you often want to see the big picture of how the layers come together. In those situations I look at the generated C++. It's designed to be clean and easy to read, and it has #line directives so that you can tell what layer each line came from.
      - magicalhippo 7y ago
        But there's no "waypoint" there, it uses the function signature as a marker to insert the code. Guess I'd have to use it for a bit to get a feel for how it is in practice. My initial reaction is that it just moves the complexity around a bit, without really helping much.
spoondan 7y ago
About 15 years ago, I built a system like this for a web application. I defined a series of core extension points such as the request router and authentication and authorization providers. Modules provided extensions that connected to the core extension points, as well as define their own extension points that other modules could connect to. Dependency injection was used throughout so that implementations could be easily swapped (mostly used for the tests). It took quite a bit of time to develop this framework, but, as a result, everything was modularized, contained, and composed. It was a complete nightmare to reason about and work with, even for me, and especially for newcomers. A natural way of trying to understand a system is to look at its entry point. With a system like this, it looks like a skeleton: let di := DependencyContainer.make() let moduleSystem := ModuleSystem.with_container(di) di.register_singleton[IModuleSystem](moduleSystem) moduleSystem.find_and_register_modules() Where do we go from here? The next logical step is into `find_and_register_modules` or into the documentation for the module system (just kidding: I was a "senior engineer," I didn't write docs). We are left with a lot of questions. How does anything happen? It's somewhere inside the modules, but which one? Does the order that modules are loaded matter? The answer is maybe. Clearly, there are places where order of operations matter. For example, the main navigation should be ordered. How do we accomplish that? (The answer for my system was to have the interface for navigation extensions specify there was a `readonly weight: Float64` attribute. But then to actually understand why things are in the order they're in, and to get things ordered correctly, you need to look up the values for other navigation items. It's secret coupling.) More recently, I saw a team at my previous company build a system like this. There were dozens of interfaces and implementation classes and code for composing all of this, and the end result was you couldn't just go somewhere and see what was happening. What we really want to see is: match maybe_user Some(&user) => nav.add(UserProfileNavItem.for_user(user)) None => nav.add("Login", "/login") nav.add("Browse", "/browse") nav.add("Search", "/search") nav.add("Help", "/help") Do you have to make a change here whenever something gets added? Yes. You do. But the idea that your program needs infinite flexibility in all areas is simply wrong. And, in fact, everything ends up secretly coupled and inscrutable. For this reason, my advice has long been to limit extensibility to specific, narrow use cases (for example, filters in an image editing program). Do not build entire systems around extensibility. You are not building an abstract system, so don't waste your time with unnecessary and obscuring abstractions.
daralthus 7y ago
Interesting, although the features might leak into other codebases and services too, at which point compiler directives won't be enough. I believe this is a lot more tractable with Unision's[0] content addressable hashing. It's kinda made for that although more with an eye towards updates to distributed systems. [0] https://github.com/unisonweb/unison https://github.com/unisonweb/unison
c3534l 7y ago
It seems like a lot of programming ideas are just ways to manage spaghetti rather than reduce the spaghetti. I worry that a lot of programming is a sort of local minima, where people dig themselves into a hole that they can't iteratively dig themselves out of. Something like this, which allows you to write code full of cross-cutting concerns and lets the programmer fool themselves into thinking they wrote clean code because they swept everything under the rug - that kind of tool is just going to dig deeper holes.
- akkartik 7y ago
  > It seems like a lot of programming ideas are just ways to manage spaghetti rather than reduce the spaghetti. Agreed! This is why I built this approach in a ~1000 lines of code. The ability for tools to create new problems was very much on my mind. My current project using this approach is a bootstrapped computing stack from scratch: https://github.com/akkartik/mu#readme https://github.com/akkartik/mu#readme. It's at ~40k LoC (of machine code) and self-hosted, and it can create bootable images (when packaged with an OS kernel). I expect to get to approximately C's level of expressivity in another ~20k LoC or so. So I care very much about what I call the "thrust to weight ratio" of a codebase. The less code you have, the less spaghetti you can create. My implementation of layers will keel over and fail terribly for large projects with many team members. I consider that constraint to be a good thing.
cryptonector 7y ago
Once you get to codebase sizes like... an OS, a programming language's tools, a database... this sort of thing begins to look awfully simplistic. Instead you have to properly clean tech debt from time to time.
- akkartik 7y ago
  Or you have to keep the codebase small. I'm not convinced an OS or programming language or database needs millions of lines of code. I'm not alone in this scepticism. Check out Alan Kay's STEPS project from a few years ago. The goal was to create a complete computing stack in 20k LoC. My current project has a similar goal: https://github.com/akkartik/mu#readme https://github.com/akkartik/mu#readme You mentioned tech debt. I think I have a more aggressive definition of tech debt than most: any dependency you rely on that you don't understand the internals of is tech debt. It's fine to take on in the short term, but you should be planning to pay it off in the long term. The approach modern software takes, of continually adding dependencies (each of which is itself growing monotonically complex) faster than anyone can internalize them, this approach feels utterly insane. Under this definition, my tiny tool is the least of someone's problems if say they depend on 60 Ruby Gems. Look at my definition of x86 instructions: http://akkartik.github.io/mu/html/013direct_addressing.cc.html http://akkartik.github.io/mu/html/013direct_addressing.cc.ht... If this is tech debt, I'll stay in debtors' prison, because it's safer inside than outside.
  - jopsen 7y ago
    So if I use HTTPS to fetch resources from the internet, I must understand the internals of TCP, TLS, HTTP 1.1, HTTP/2, UDP, QUIC, HTTP/3 and the implementations of these protocols in the library I depend on for HTTP I/O. IMO, if you understand HTTP verbs, headers, response codes, body and the basics of REST. The beauty is that you don't need to understand the internals of every layer on the stack. In fact you can be unconcerned with whether libcurl does HTTP 1.1 or HTTP/2 on your behalf.
    - akkartik 7y ago
      Sure. Protocols are like air. And I'll even give you a pass on your OS kernel. Can you think of any examples where my idea may be more useful? How about we measure mean time between security incidents. Spend time understanding your dependencies in order of their security records. And you just have to learn enough to go noping away from it. Once you delete a dependency you don't have to waste your time on it anymore. Does that seem useful? If it just gets you thinking about the full cost of ownership of your dependencies, I think chatting with me has paid for itself :) (None of this applies if you aren't providing a service to others, of course. If you're just learning, use whatever you want!)
jillesvangurp 7y ago
Reminds me of Layom, which is something my phd supervisor worked on in the nineties. Later aspect oriented programming (which is similar) became a thing with languages like AspectJ around 1999 or so. These days you see a weaker form of this in the form of annotations in e.g. Python or Java or macros in Rust. Spring actually uses an AOP library that has its roots in AspectJ. I actually prefer a more recent trend in e.g. Kotlin to use internal DSLs instead of injected magic. The advantage is that it is simpler to debug and part of normal type checks, autocomplete, etc. since technically all you are doing is working with the language instead of augmenting it via generated code.
mdewing 7y ago
There seems to be some confusion between layers and AOP. Layers are a static mechanism - all the layers are composed to a final program before compilation. The layers have no effect at runtime (unlike AOP). If the all the layers are composed (tangled) without the '#line' directives, the code should look (and be) the same as a normally-written program. Modifying the program most likely will require adding a new layer. (If "layer" == "git commit" then this is only option under existing programming practice.) Programming with layers means the programmer has the additional option of modifying previous layers. Whether this can scale to large programs is an open (and interesting) question.
- akkartik 7y ago
  Thank you for the clear articulation!

82 points by sktrdie 7y ago

Sometimes I think this is a good idea but generally it seems like refactoring in a way that the code base always makes sense, even to a newcomer, is the correct goal. History can be kept in source control, if it's necessary to follow the history of code development it's probably a sign that the abstractions are not chosen correctly.

akkartik 7y ago
(Author here.) The analogy to `git log` is just that, an analogy. Here's a concrete thing that you get with layers that you don't get with `git log` or conventional abstractions and refactoring: alternatives. Often we see comments in codebases that some complex, hairy function is really a much simpler function with certain optimizations applied. There's been lots of work done to try to separate out functional spec from optimization so that the two can be managed separately. But this is all open research. Layers let you get the value of this separation today, with very little code (just a lot of reorientation in mindset). If you adopt layers you can use the simple version in an earlier layer, and then replace it in a later layer. Your CI system will exercise both, and flag if either stops passing tests. What used to be a code comment now stays up to date.
- zwieback 7y ago
  that's a good point about continuing testing reference and optimized solutions - I could see a case being made for standardizing that kind of thing
joe_the_user 7y ago
Well, what if you could have approach described in the article but there would be a defined patch interface so that if you looked at it, each patch makes sense also?

jimbo1qaz 7y ago

Fascinating idea. I've definitely written and encountered programs where functions were more complicated than they could've been, in order to support features separate from the "main idea" of the program. Like FPS calculation, Ctrl+C handling, passing intermediate audio data to extra buffers for visualization... Could "splicing in" code it be implemented using aspect-oriented programming frameworks? https://en.wikipedia.org/wiki/Aspect-oriented_programming https://en.wikipedia.org/wiki/Aspect-oriented_programming I've never used them though. > There's a new constraint that has no counterpart in current programming practice — remove a feature, and everything before it should build and pass its tests: > build_and_test_until Looks like added tooling is needed, but seems worthwhile.

hacker_9 7y ago

Well this is interesting, I recently spent a weekend creating a 'splicing compiler', which could take a typescript project split up into features, where a feature was written in just one single typescript file. Each section of code could then specify new 'code hooks', for future code to insert itself at during compilation. My original inspiration was that English language is spoken sequentially, but ends up in a graph database (i.e. the brain's neural network). So you'd write code like a thesis, and the hooks you specify would allow the splicer to form the code graph. In theory it seemed like it would be useful, so I built Pacman with it and, well, it didn't really work in practice! Things ended up being more abstract in the end, and I'd be mentally juggling where hooks were created and what the final code would look like. Even showing the code output to the right wasn't that helpful, as the way the code was spliced together put things in not the most readable order. Additionally how to split up the code wasn't the most obvious, i.e. when should a hook be added? It often felt I'd only know after writing the code in full. At which point it just felt like extra work. I'd often find myself to be inserting code all over the place to get something working, and so the refactoring needed at the end became overwhelming. Ultimately these reasons lead me to abandon the idea.

akkartik 7y ago
I've written 35k LoC with this tool, and it does require some taste but not intractably so. A couple of rules of thumb: - Don't use layers in too fine-grained a manner. Often if I need to change two lines in a function but they're separated by 6 lines I err on the side of replacing the entire paragraph or even the entire function. Each layer should be readable in a self-contained manner as much as possible. - I never invalidate tests in an earlier layer. You never want to be in a situation where a layer lies to you about what desired behavior is.

bruth 7y ago

Posted in 2013

SkyBelow 7y ago

I think the core problem is that one of the root causes of the patchwork hard to understand code is lack of spending the effort to keep the code understandable. Refactor into new models, move around code so it makes more sense given changes in requirements, etc. If that level of maintenance isn't being performed (which is almost a given, assuming we are looking at a problem with such a patchwork problem), then this new tool won't be fully followed and the end result will be a patchwork code base that reaches nightmarish levels where some features are added in this layered approach, but other features are added directly to the code, and even other features are added directly to prior layers. If the tooling force people to use this in the recommended way then it may have interesting payoffs that are worthwhile. But lacking such an enforced requirement, I don't see applying being beneficial except in the code bases that least benefit from it.

akkartik 7y ago
I find this approach to be easier than conventional approaches, and my motivation and effort levels seem constant on both sides. You're right that there's a social component here. This tool tries to make things easier for authors under the assumption that people try to use it tastefully.

canadaduane 7y ago

Some other, related ideas & paradigms: Context-Oriented Programming: https://arxiv.org/pdf/1105.0069.pdf https://arxiv.org/pdf/1105.0069.pdf Feature-Oriented Software Development: https://en.wikipedia.org/wiki/Feature-oriented_programming https://en.wikipedia.org/wiki/Feature-oriented_programming

akkartik 7y ago
Yes, I was aware of both as well as AOP when I built this. Some comments on OP discuss them as well. The big difference between my approach and these is the weight-to-thrust ratio. These academic approaches tend to assume people don't want to change the pieces, just combine them. And they create huge tools to make that possible. Why can't people just change the pieces? I've never understood that. I don't need an algebra of program transformations intermediating between me and my code. I'd much rather hold the code directly in my mind. And that keeps the tooling needed really, really small. Just a hundred or so lines of C. Tooling is a double-edged thing. Adding tooling often encourages increases in codebase scale. Keeping tooling rudimentary can be a good thing by keeping codebases small.

crazypython 7y ago

This is just AOP.

canadaduane 7y ago
It's related, but I don't think AOP has a concept of an immutable past?
- akkartik 7y ago
  Hmm, layers don't have an immutable past. Can you elaborate?
  - canadaduane 7y ago
    Ah, I misunderstood. I thought the layer stack was immutable.
akkartik 7y ago
https://news.ycombinator.com/item?id=21766557#21767531 https://news.ycombinator.com/item?id=21766557#21767531 It's like AOP with user-definable join points.

_bxg1 7y ago

Similar to Spring dependency-injection: https://www.journaldev.com/2410/spring-dependency-injection https://www.journaldev.com/2410/spring-dependency-injection It seems too powerful, though. I could see it quickly becoming intractable to find where the code lives that is causing a certain outcome. The root problem it's trying to solve is the fact that things often don't fit neatly in a single bucket, but our traditional idea of modules as mutually-exclusive and hierarchical categories forces you to choose just one place to "put" something. An idea I've had is to get rid of hierarchical modules altogether. Really they're a relic of directories and files. Instead, one could "tag" a function/class/constant as belonging to multiple modules, and one's editor could aggregate all members of a cross-cutting module as needed. Organization within actual files on disk becomes an implementation detail.

akkartik 7y ago
> I could see it quickly becoming intractable to find where the code lives that is causing a certain outcome. (Author here.) The tangler I describe inserts #line directives into the output. You can open up the generated C++ and see where a line comes from. Conventional C++ tools understand line directives as well, so the debugger experience is sane. I'm sure there are gaps in this tooling, but they're not intractable. Tooling around conventional modules has matured over decades. My approach hopefully shows some benefits of a road not taken.
- _bxg1 7y ago
  It's certainly interesting, and I didn't mean to pre-judge too strongly. I definitely think we're at the point where we need some new ideas around this stuff, and probably any new ideas at this point will require some kind of special editor assistance, so yeah, I get what you mean in that regard.
akkartik 7y ago
> Instead, one could "tag" a function/class/constant as belonging to multiple modules, and one's editor could aggregate all members of a cross-cutting module as needed. Organization within actual files on disk becomes an implementation detail. Yes, this is interesting. Are you thinking of something like Ruby's modules and `include` directives? (https://ruby-doc.com/docs/ProgrammingRuby/html/tut_modules.html https://ruby-doc.com/docs/ProgrammingRuby/html/tut_modules.h...) Is there a reason the editor has to do this rather than the language?

veddox 7y ago

So you‘re trying to remove complexity by making it invisible? Sounds great while things work, but doesn‘t that turn debugging into a nightmarish exercise? Because I never get to see the code that‘s actually executing, the pieces of a function might be scattered over a dozen different layers. Strikes me as the ultimate recipe for self-modifying spaghetti code... Or am I missing something here?

akkartik 7y ago
I'm not sure what you mean by 'invisible'. Notice that I make zero claims of reducing complexity. Layers are only a way to manage and stage complexity. I responded to tooling questions at https://news.ycombinator.com/item?id=21766557#21767393 https://news.ycombinator.com/item?id=21766557#21767393
- magicalhippo 7y ago
  Well studying 010vm.cc, I wouldn't know that 014literal_string.cc alters the to_string function. That information is hidden from me, until I stumble upon it in 014literal_string.cc.
  - akkartik 7y ago
    Most of the time you wouldn't care. The presence of a waypoint is a hint that _something_ is inserting code there. But as long as tests in 010vm.cc continue to pass, it shouldn't matter to it what future layers do. When debugging things, however, you often want to see the big picture of how the layers come together. In those situations I look at the generated C++. It's designed to be clean and easy to read, and it has #line directives so that you can tell what layer each line came from.
    - magicalhippo 7y ago
      But there's no "waypoint" there, it uses the function signature as a marker to insert the code. Guess I'd have to use it for a bit to get a feel for how it is in practice. My initial reaction is that it just moves the complexity around a bit, without really helping much.

spoondan 7y ago

About 15 years ago, I built a system like this for a web application. I defined a series of core extension points such as the request router and authentication and authorization providers. Modules provided extensions that connected to the core extension points, as well as define their own extension points that other modules could connect to. Dependency injection was used throughout so that implementations could be easily swapped (mostly used for the tests). It took quite a bit of time to develop this framework, but, as a result, everything was modularized, contained, and composed. It was a complete nightmare to reason about and work with, even for me, and especially for newcomers. A natural way of trying to understand a system is to look at its entry point. With a system like this, it looks like a skeleton: let di := DependencyContainer.make() let moduleSystem := ModuleSystem.with_container(di) di.register_singleton[IModuleSystem](moduleSystem) moduleSystem.find_and_register_modules() Where do we go from here? The next logical step is into `find_and_register_modules` or into the documentation for the module system (just kidding: I was a "senior engineer," I didn't write docs). We are left with a lot of questions. How does anything happen? It's somewhere inside the modules, but which one? Does the order that modules are loaded matter? The answer is maybe. Clearly, there are places where order of operations matter. For example, the main navigation should be ordered. How do we accomplish that? (The answer for my system was to have the interface for navigation extensions specify there was a `readonly weight: Float64` attribute. But then to actually understand why things are in the order they're in, and to get things ordered correctly, you need to look up the values for other navigation items. It's secret coupling.) More recently, I saw a team at my previous company build a system like this. There were dozens of interfaces and implementation classes and code for composing all of this, and the end result was you couldn't just go somewhere and see what was happening. What we really want to see is: match maybe_user Some(&user) => nav.add(UserProfileNavItem.for_user(user)) None => nav.add("Login", "/login") nav.add("Browse", "/browse") nav.add("Search", "/search") nav.add("Help", "/help") Do you have to make a change here whenever something gets added? Yes. You do. But the idea that your program needs infinite flexibility in all areas is simply wrong. And, in fact, everything ends up secretly coupled and inscrutable. For this reason, my advice has long been to limit extensibility to specific, narrow use cases (for example, filters in an image editing program). Do not build entire systems around extensibility. You are not building an abstract system, so don't waste your time with unnecessary and obscuring abstractions.

daralthus 7y ago

Interesting, although the features might leak into other codebases and services too, at which point compiler directives won't be enough. I believe this is a lot more tractable with Unision's[0] content addressable hashing. It's kinda made for that although more with an eye towards updates to distributed systems. [0] https://github.com/unisonweb/unison https://github.com/unisonweb/unison

c3534l 7y ago

It seems like a lot of programming ideas are just ways to manage spaghetti rather than reduce the spaghetti. I worry that a lot of programming is a sort of local minima, where people dig themselves into a hole that they can't iteratively dig themselves out of. Something like this, which allows you to write code full of cross-cutting concerns and lets the programmer fool themselves into thinking they wrote clean code because they swept everything under the rug - that kind of tool is just going to dig deeper holes.

akkartik 7y ago
> It seems like a lot of programming ideas are just ways to manage spaghetti rather than reduce the spaghetti. Agreed! This is why I built this approach in a ~1000 lines of code. The ability for tools to create new problems was very much on my mind. My current project using this approach is a bootstrapped computing stack from scratch: https://github.com/akkartik/mu#readme https://github.com/akkartik/mu#readme. It's at ~40k LoC (of machine code) and self-hosted, and it can create bootable images (when packaged with an OS kernel). I expect to get to approximately C's level of expressivity in another ~20k LoC or so. So I care very much about what I call the "thrust to weight ratio" of a codebase. The less code you have, the less spaghetti you can create. My implementation of layers will keel over and fail terribly for large projects with many team members. I consider that constraint to be a good thing.

cryptonector 7y ago

Once you get to codebase sizes like... an OS, a programming language's tools, a database... this sort of thing begins to look awfully simplistic. Instead you have to properly clean tech debt from time to time.

akkartik 7y ago
Or you have to keep the codebase small. I'm not convinced an OS or programming language or database needs millions of lines of code. I'm not alone in this scepticism. Check out Alan Kay's STEPS project from a few years ago. The goal was to create a complete computing stack in 20k LoC. My current project has a similar goal: https://github.com/akkartik/mu#readme https://github.com/akkartik/mu#readme You mentioned tech debt. I think I have a more aggressive definition of tech debt than most: any dependency you rely on that you don't understand the internals of is tech debt. It's fine to take on in the short term, but you should be planning to pay it off in the long term. The approach modern software takes, of continually adding dependencies (each of which is itself growing monotonically complex) faster than anyone can internalize them, this approach feels utterly insane. Under this definition, my tiny tool is the least of someone's problems if say they depend on 60 Ruby Gems. Look at my definition of x86 instructions: http://akkartik.github.io/mu/html/013direct_addressing.cc.html http://akkartik.github.io/mu/html/013direct_addressing.cc.ht... If this is tech debt, I'll stay in debtors' prison, because it's safer inside than outside.
- jopsen 7y ago
  So if I use HTTPS to fetch resources from the internet, I must understand the internals of TCP, TLS, HTTP 1.1, HTTP/2, UDP, QUIC, HTTP/3 and the implementations of these protocols in the library I depend on for HTTP I/O. IMO, if you understand HTTP verbs, headers, response codes, body and the basics of REST. The beauty is that you don't need to understand the internals of every layer on the stack. In fact you can be unconcerned with whether libcurl does HTTP 1.1 or HTTP/2 on your behalf.
  - akkartik 7y ago
    Sure. Protocols are like air. And I'll even give you a pass on your OS kernel. Can you think of any examples where my idea may be more useful? How about we measure mean time between security incidents. Spend time understanding your dependencies in order of their security records. And you just have to learn enough to go noping away from it. Once you delete a dependency you don't have to waste your time on it anymore. Does that seem useful? If it just gets you thinking about the full cost of ownership of your dependencies, I think chatting with me has paid for itself :) (None of this applies if you aren't providing a service to others, of course. If you're just learning, use whatever you want!)

jillesvangurp 7y ago

Reminds me of Layom, which is something my phd supervisor worked on in the nineties. Later aspect oriented programming (which is similar) became a thing with languages like AspectJ around 1999 or so. These days you see a weaker form of this in the form of annotations in e.g. Python or Java or macros in Rust. Spring actually uses an AOP library that has its roots in AspectJ. I actually prefer a more recent trend in e.g. Kotlin to use internal DSLs instead of injected magic. The advantage is that it is simpler to debug and part of normal type checks, autocomplete, etc. since technically all you are doing is working with the language instead of augmenting it via generated code.

mdewing 7y ago

There seems to be some confusion between layers and AOP. Layers are a static mechanism - all the layers are composed to a final program before compilation. The layers have no effect at runtime (unlike AOP). If the all the layers are composed (tangled) without the '#line' directives, the code should look (and be) the same as a normally-written program. Modifying the program most likely will require adding a new layer. (If "layer" == "git commit" then this is only option under existing programming practice.) Programming with layers means the programmer has the additional option of modifying previous layers. Whether this can scale to large programs is an open (and interesting) question.

akkartik 7y ago
Thank you for the clear articulation!