4 ms·

Glue: The Dark Matter of Software

79 points by mpweiher 5y ago

ducharmdev 5y ago
Interesting observations; although by definition glue code is amorphous and hard to pin down, I would like to see more concrete examples of the problem discussed here. Is this a general problem related to how we build interfaces for our implementations? Or is this more a problem with particular kinds of module systems? (Or both)
- dgb23 5y ago
  I think it’s interfaces and protocols at a granular level. If every interface is bespoke while also being explicitly described top down, then we end up with this quadratic complexity. Uniform, extensible interfaces and generic data structures are the solutions to this, as well as shifting complexity to data and away from code. Several modern languages, techniques and protocols address this in varying degrees and from different angles. An old language that does this very well is SQL.
  - kaycebasques 5y ago
    > Several modern languages, techniques and protocols address this in varying degrees and from different angles. Examples? (Asking as a curious beginner who doesn't know where to start.)
    - dgb23 5y ago
      Go‘s implicit interfaces, Rust‘s traits, Clojure‘s sequences, transducers, multimethods and spec, Julia‘s multiple dispatch, HTTP REST, JSON and json-schema. These are all attempts to enable generic, simple code over complex data structures, while respecting and incorporating extensibility and change. It’s an old idea but I think more importance is given to it in more recent years.
- ryanackley 5y ago
  I get frustrated with the layers between relational data and data structures you pass around in code. So for example, if you want a build a simple REST api for your data model, you have glue code to read/write the data and transfer it to/from DAO's (Data access objects). Then you have glue code to transfer the data into to/from object model you expose via the API. This feels like 90% of server side CRUD app programming these days. It's not quite boilerplate because it's a mix of ORM optimizations for your data model and data massaging to make it more digestable to end users of your API.
- sdenton4 5y ago
  It's all about compartmentalized code... Glue happens whenever compartments interact with one another. Examples, thinking of an app I used to work on: a) App frontend (UI logic) is compartmentalized from the app backend. Communication is handled by an async messaging system, requiring bundling up messages in both directions. b) The app backend talks to a few different servers, each with their on expected message set. glue glue glue. c) The app backend talks to the app database, inserting/altering info, and extracting database rows into objects which can be passed around. d) (We also had peer-to-peer communication over local wifi, which required its own massive pile of communication code...) e) Even different objects/methods in the backend code end up requiring small amounts of glue code to arrange+prepare argument sets to call one another. So, every time we create a division between two pieces of code, we introduce boundaries, which introduces glue. --- One way to mitigate the complexity is by making the passed messages first class. In our app, there was some 'AppThing' object which represented an instance of the fundamental thing the app was about. You then stuff that object with every piece of info about AppThings you could ever want, and pass it around freely between components. Invariably it gets kinda bloated and fractures, when someone doesn't want to bother building up the WHOLE thing with the necessary DB and server calls to ensure the data is all accurately filled in, when you just want to pass one or two fields to a neighboring UI element. And then you find yourself with lots of logic checking that certain fields are present amongst the million fields in your universal object... Another approach is to aggressively prune the graph of possible component interactions. Functionality should be well-encapsulated, with absolutely minimal APIs for external interactions. Actually draw the 'cell membranes' between different parts of the app, map out the interactions between cells, and find ways to cut it back.
mettamage 5y ago
I wrote a non-programmer summary to some of my friends, because my mind is blown. Here it is (warning: it's very informal). This is some next level 80/20 stuff, I'd argue it's even 99/1 (consultant speech for "being efficient" by seeing if you can exploit Pareto's principle [0]). A guy on YT [1] simulates in 7 minutes how the two biggest operating systems were built in the sixties (Multics by IBM and Unix by 2 guys in a garage). Nowadays we only have descendants of Unix, this simulation shows IMO two things: 1. How a chain of command screws up productivity (I doubt that anyone at IBM had the power/autonomy to change complete structure of the product. This was needed, but such change management is insane). 2. How the pipe command in Unix changed the mathematical nature of developing an operating system (in fact, it's even better than 99/1, when you think about it). In normal human language, the difference is: IBM programmed modular blocks. Unix programmed modular blocks. IBM then programmed how those modular blocks should interact. Unix created a pipe command and allowed computer users themselves to implement how these modular blocks should interact. At the time, computer users were all savvy enough like that. Result: when IBM created a feature, it needed to integrate it with all the other features that came before. When those 2 guys in a garage (Unix) created a feature, all they needed to do was create that feature. So mathematically you could say (where each term is the amount of work required for one new feature + integrating it with the operating system): IBM/Multics: (0 features_before + 1 new_feature) + (1 features_before + 1 new_feature) + (2 features_before + 1 new_feature) + (3 features_before + 1 new_feature) + (4 features_before + 1 new_feature) + (5 features_before + 1 new_feature) + (6 features_before + 1 new_feature) ... + (n_features_before + 1 new_feature) --> simplifies to my favorite formula which is: n(n−1)/2 --> you can also see it as --> 0.5n^2 - 0.5n --> So, roughly this is n^2. Just count the amount of numbers, it comes down to: 28 units at the 7th term/timestep. Unix/2 garage guys: 1 new_feature + 1 new_feature + 1 new_feature + 1 new_feature --> simplifies to n_new_features At term/timestep, you'd get 7. At the 100th feature, you'd literally 100^2 amount of coding work at IBM whereas at the Unix place you'd simply need 1 unit ... This isn't 80/20 or 99/1. This is n^2/n (meaning: what one needs n^2 time for, the other could do in n time). [0] https://en.wikipedia.org/wiki/Pareto_principle https://en.wikipedia.org/wiki/Pareto_principle [1] https://www.youtube.com/watch?v=3Ea3pkTCYx4&ab_channel=FOAM https://www.youtube.com/watch?v=3Ea3pkTCYx4&ab_channel=FOAM
- convolvatron 5y ago
p0nce 5y ago
Like gas, software naturally gets bigger up until it fills up any added value that might have been produced.
maltalex 5y ago
Very interesting post. It feels very applicable to the functional vs imperative programming debate, and I say that as someone who's mostly on the imperative side. The top row of the "perimeter" is very "functional". It's interesting to note that people from the same school of thought as those who built Unix (or even the same actual people) ended up creating Golang, which is anything but a functional language.
kgr 5y ago
https://www.youtube.com/watch?v=S4LbUv5FsGQ https://www.youtube.com/watch?v=S4LbUv5FsGQ covers this topic. As I say in the video, no point solution (ie. some new API) can solve the problem, when the need to integrate (glue) between so many point solutions, is the problem.
- kgr 5y ago
  Both the video from the article and the one I posted above are from this longer talk of mine: https://www.youtube.com/watch?v=PsFLlgrzn2E https://www.youtube.com/watch?v=PsFLlgrzn2E I think this the most important problem in software development to be given the least amount of attention, or even awareness. Another new MVC framework, build tool, or even programming language, isn't going to solve this problem. First we need to acknowledge the problem.
kuharich 5y ago
Past comments: https://news.ycombinator.com/item?id=27486706 https://news.ycombinator.com/item?id=27486706
PaulDavisThe1st 5y ago
I do a lot of stuff with my house. As a result I have on hand, at various points in time: * contact cement * superglue * mastic * wood glue * gorilla glue * liquid nails * various caulks Why so much glue? Because the right glue to use depends on the materials being joined, and the context in which the joint will need to survive (moisture, temperature, duration). As with construction and art, so with software. The Unix pipe is powerful because it enforced "the only thing you can connect together is line-oriented byte streams". But when you are building something where that's not the substrates in question, it's the wrong tool, and you need a different kind of glue. Expand that to all the possible different things you may wish/need to "glue" together in software, and it becomes fairly apparent why there's a huge diversity in the glue being used, and why dreaming of a world in which you get by with just the software analog of gorilla glue is the wrong thing to do.
- mpweiher 5y ago
  Unix pipes and filters were not presented as "the" solution, but as an example for the properties we want from our glue, i.e. we should be able to glue our pieces together without having to write much additional code. And yes, in order for this to work, we need lots of different kinds of glue: https://blog.metaobject.com/2019/02/why-architecture-oriented-programming.html https://blog.metaobject.com/2019/02/why-architecture-oriente... However, for most programming tasks, we have one kind of glue (with minor sub-variants): the procedure call. And so instead of having something that snaps together with a bit of glue, we have to write enormous amounts of boiler-plate in order to make the pieces fit. (The analogy only goes so far...)
  - PaulDavisThe1st 5y ago
    I'm been a compiled-language programmer for 35 or so years. I don't view procedure calls as glue, I perceive them as a fundamental mechanism for organizing code. At the level of code that I normally operate at, if you don't write everything in-line, then you've got procedure calls. Not glue, at least, not primarily. Glue is a way to move data (and perhaps other stuff too) across code boundaries that are imposed by design, language or other constraints. So, sure, stuffing a bunch of stuff on the stack, jumping to a new address and doing something with whatever is on the stack is, in some sense, a sort of glue. But your TFA seems to be talking about a much higher level of glue than this, I think. The sort of glue that's represented by things like (random list): protobufs, RPC, serialized JSON, even ye olde CORBA/OLE models. Questions like "I've got this customer data, I need to give it to some code that will munge it in some way... how do I do that?" In that sense, procedure calls are low level mechanism that will support a higher level solution. Again, if it's not all just inline execution of machine code, then at some point, there's going to be a procedure call no matter what abstraction level you're working at.
tobr 5y ago
You have two systems, so you need to write a layer of glue code to make them work together. The glue code turns out inelegant, spaghetti-like, hard to work with. So you spend a little bit of time cleaning it up, abstracting reusable things out, and generally systematizing things. Now you have three systems, so you need to write two layers of glue code to make them work together…

79 points by mpweiher 5y ago

Interesting observations; although by definition glue code is amorphous and hard to pin down, I would like to see more concrete examples of the problem discussed here. Is this a general problem related to how we build interfaces for our implementations? Or is this more a problem with particular kinds of module systems? (Or both)

dgb23 5y ago
I think it’s interfaces and protocols at a granular level. If every interface is bespoke while also being explicitly described top down, then we end up with this quadratic complexity. Uniform, extensible interfaces and generic data structures are the solutions to this, as well as shifting complexity to data and away from code. Several modern languages, techniques and protocols address this in varying degrees and from different angles. An old language that does this very well is SQL.
- kaycebasques 5y ago
  > Several modern languages, techniques and protocols address this in varying degrees and from different angles. Examples? (Asking as a curious beginner who doesn't know where to start.)
  - dgb23 5y ago
    Go‘s implicit interfaces, Rust‘s traits, Clojure‘s sequences, transducers, multimethods and spec, Julia‘s multiple dispatch, HTTP REST, JSON and json-schema. These are all attempts to enable generic, simple code over complex data structures, while respecting and incorporating extensibility and change. It’s an old idea but I think more importance is given to it in more recent years.
ryanackley 5y ago
I get frustrated with the layers between relational data and data structures you pass around in code. So for example, if you want a build a simple REST api for your data model, you have glue code to read/write the data and transfer it to/from DAO's (Data access objects). Then you have glue code to transfer the data into to/from object model you expose via the API. This feels like 90% of server side CRUD app programming these days. It's not quite boilerplate because it's a mix of ORM optimizations for your data model and data massaging to make it more digestable to end users of your API.
sdenton4 5y ago
It's all about compartmentalized code... Glue happens whenever compartments interact with one another. Examples, thinking of an app I used to work on: a) App frontend (UI logic) is compartmentalized from the app backend. Communication is handled by an async messaging system, requiring bundling up messages in both directions. b) The app backend talks to a few different servers, each with their on expected message set. glue glue glue. c) The app backend talks to the app database, inserting/altering info, and extracting database rows into objects which can be passed around. d) (We also had peer-to-peer communication over local wifi, which required its own massive pile of communication code...) e) Even different objects/methods in the backend code end up requiring small amounts of glue code to arrange+prepare argument sets to call one another. So, every time we create a division between two pieces of code, we introduce boundaries, which introduces glue. --- One way to mitigate the complexity is by making the passed messages first class. In our app, there was some 'AppThing' object which represented an instance of the fundamental thing the app was about. You then stuff that object with every piece of info about AppThings you could ever want, and pass it around freely between components. Invariably it gets kinda bloated and fractures, when someone doesn't want to bother building up the WHOLE thing with the necessary DB and server calls to ensure the data is all accurately filled in, when you just want to pass one or two fields to a neighboring UI element. And then you find yourself with lots of logic checking that certain fields are present amongst the million fields in your universal object... Another approach is to aggressively prune the graph of possible component interactions. Functionality should be well-encapsulated, with absolutely minimal APIs for external interactions. Actually draw the 'cell membranes' between different parts of the app, map out the interactions between cells, and find ways to cut it back.

mettamage 5y ago

I wrote a non-programmer summary to some of my friends, because my mind is blown. Here it is (warning: it's very informal). This is some next level 80/20 stuff, I'd argue it's even 99/1 (consultant speech for "being efficient" by seeing if you can exploit Pareto's principle [0]). A guy on YT [1] simulates in 7 minutes how the two biggest operating systems were built in the sixties (Multics by IBM and Unix by 2 guys in a garage). Nowadays we only have descendants of Unix, this simulation shows IMO two things: 1. How a chain of command screws up productivity (I doubt that anyone at IBM had the power/autonomy to change complete structure of the product. This was needed, but such change management is insane). 2. How the pipe command in Unix changed the mathematical nature of developing an operating system (in fact, it's even better than 99/1, when you think about it). In normal human language, the difference is: IBM programmed modular blocks. Unix programmed modular blocks. IBM then programmed how those modular blocks should interact. Unix created a pipe command and allowed computer users themselves to implement how these modular blocks should interact. At the time, computer users were all savvy enough like that. Result: when IBM created a feature, it needed to integrate it with all the other features that came before. When those 2 guys in a garage (Unix) created a feature, all they needed to do was create that feature. So mathematically you could say (where each term is the amount of work required for one new feature + integrating it with the operating system): IBM/Multics: (0 features_before + 1 new_feature) + (1 features_before + 1 new_feature) + (2 features_before + 1 new_feature) + (3 features_before + 1 new_feature) + (4 features_before + 1 new_feature) + (5 features_before + 1 new_feature) + (6 features_before + 1 new_feature) ... + (n_features_before + 1 new_feature) --> simplifies to my favorite formula which is: n(n−1)/2 --> you can also see it as --> 0.5n^2 - 0.5n --> So, roughly this is n^2. Just count the amount of numbers, it comes down to: 28 units at the 7th term/timestep. Unix/2 garage guys: 1 new_feature + 1 new_feature + 1 new_feature + 1 new_feature --> simplifies to n_new_features At term/timestep, you'd get 7. At the 100th feature, you'd literally 100^2 amount of coding work at IBM whereas at the Unix place you'd simply need 1 unit ... This isn't 80/20 or 99/1. This is n^2/n (meaning: what one needs n^2 time for, the other could do in n time). [0] https://en.wikipedia.org/wiki/Pareto_principle https://en.wikipedia.org/wiki/Pareto_principle [1] https://www.youtube.com/watch?v=3Ea3pkTCYx4&ab_channel=FOAM https://www.youtube.com/watch?v=3Ea3pkTCYx4&ab_channel=FOAM

convolvatron 5y ago

p0nce 5y ago

Like gas, software naturally gets bigger up until it fills up any added value that might have been produced.

maltalex 5y ago

Very interesting post. It feels very applicable to the functional vs imperative programming debate, and I say that as someone who's mostly on the imperative side. The top row of the "perimeter" is very "functional". It's interesting to note that people from the same school of thought as those who built Unix (or even the same actual people) ended up creating Golang, which is anything but a functional language.

kgr 5y ago

https://www.youtube.com/watch?v=S4LbUv5FsGQ https://www.youtube.com/watch?v=S4LbUv5FsGQ covers this topic. As I say in the video, no point solution (ie. some new API) can solve the problem, when the need to integrate (glue) between so many point solutions, is the problem.

kgr 5y ago
Both the video from the article and the one I posted above are from this longer talk of mine: https://www.youtube.com/watch?v=PsFLlgrzn2E https://www.youtube.com/watch?v=PsFLlgrzn2E I think this the most important problem in software development to be given the least amount of attention, or even awareness. Another new MVC framework, build tool, or even programming language, isn't going to solve this problem. First we need to acknowledge the problem.

kuharich 5y ago

Past comments: https://news.ycombinator.com/item?id=27486706 https://news.ycombinator.com/item?id=27486706

PaulDavisThe1st 5y ago

I do a lot of stuff with my house. As a result I have on hand, at various points in time: * contact cement * superglue * mastic * wood glue * gorilla glue * liquid nails * various caulks Why so much glue? Because the right glue to use depends on the materials being joined, and the context in which the joint will need to survive (moisture, temperature, duration). As with construction and art, so with software. The Unix pipe is powerful because it enforced "the only thing you can connect together is line-oriented byte streams". But when you are building something where that's not the substrates in question, it's the wrong tool, and you need a different kind of glue. Expand that to all the possible different things you may wish/need to "glue" together in software, and it becomes fairly apparent why there's a huge diversity in the glue being used, and why dreaming of a world in which you get by with just the software analog of gorilla glue is the wrong thing to do.

mpweiher 5y ago
Unix pipes and filters were not presented as "the" solution, but as an example for the properties we want from our glue, i.e. we should be able to glue our pieces together without having to write much additional code. And yes, in order for this to work, we need lots of different kinds of glue: https://blog.metaobject.com/2019/02/why-architecture-oriented-programming.html https://blog.metaobject.com/2019/02/why-architecture-oriente... However, for most programming tasks, we have one kind of glue (with minor sub-variants): the procedure call. And so instead of having something that snaps together with a bit of glue, we have to write enormous amounts of boiler-plate in order to make the pieces fit. (The analogy only goes so far...)
- PaulDavisThe1st 5y ago
  I'm been a compiled-language programmer for 35 or so years. I don't view procedure calls as glue, I perceive them as a fundamental mechanism for organizing code. At the level of code that I normally operate at, if you don't write everything in-line, then you've got procedure calls. Not glue, at least, not primarily. Glue is a way to move data (and perhaps other stuff too) across code boundaries that are imposed by design, language or other constraints. So, sure, stuffing a bunch of stuff on the stack, jumping to a new address and doing something with whatever is on the stack is, in some sense, a sort of glue. But your TFA seems to be talking about a much higher level of glue than this, I think. The sort of glue that's represented by things like (random list): protobufs, RPC, serialized JSON, even ye olde CORBA/OLE models. Questions like "I've got this customer data, I need to give it to some code that will munge it in some way... how do I do that?" In that sense, procedure calls are low level mechanism that will support a higher level solution. Again, if it's not all just inline execution of machine code, then at some point, there's going to be a procedure call no matter what abstraction level you're working at.

tobr 5y ago

You have two systems, so you need to write a layer of glue code to make them work together. The glue code turns out inelegant, spaghetti-like, hard to work with. So you spend a little bit of time cleaning it up, abstracting reusable things out, and generally systematizing things. Now you have three systems, so you need to write two layers of glue code to make them work together…