4 ms·

Lisp machines showed the way decades ago, computing took a different path. Here's the system's UDP receive-ip-packet method: https://twitter.com/RainerJoswig/st

by Jach 5y ago

Lisp machines showed the way decades ago, computing took a different path. Here's the system's UDP receive-ip-packet method: https://twitter.com/RainerJoswig/status/1215728406823886854 https://twitter.com/RainerJoswig/status/1215728406823886854 The copies vs dependencies thing doesn't seem like a useful framework to me -- in order to do anything with software that has dependencies, such dependencies need to be copied in some form that can be integrated into the final build. So the copies are there regardless. Different ecosystems differ on how easy it is to work with them. If all you get is a .so, then even if it's open source you're going to have to find and build your own to make changes. In Java you'll get jars, which may or may not contain java files and class files -- in the past I've unzipped jars, recompiled a single java file to replace a class file, rezipped it back up, and used that jar instead of the upstream jar. Not too bad. Some log4j 'fixes' involved stripping out some class files from jars, again not too bad, just don't accidentally clobber your changes later. Some JS projects will have additional instructions (or a bash script) that after npm downloads everything into the local node_modules, you need to go modify a particular JS file in there. It's easy for the changes to get clobbered later, but.. Lastly in Lisp land, we have quicklisp, so you point to a library dependency like normal and it downloads it locally elsewhere (so it only need be downloaded once and not for each project). When I'm writing a program using library X, and I'm curious how some function works, I can just jump-to-source and it'll take me to that local copy. If I want to change it, I can, I just edit it, and because Lisp has "compile", "compile-file", and "load" all as standard runtime functions, not separate programs, I can compile and load my changes into the running program and immediately test them. Maybe such changes are upstreamable, maybe not. You can maintain the changes in the original files, maybe going so far as a full source copy locally instead of using the ones quicklisp brought down, or just make your own local 'patch' file of redefined functions or extended classes that you load after loading the original and the patch simply recompiles and replaces things you've changed. It's also rather fun to do this to inspect and learn about the Lisp implementation you're using, what it's doing or how it's implementing things, changing stuff to see what happens/fix an edge case bug, whatever. Part of it I think is supported by a notion Thinking Forth calls a 'lexicon'. See its early section on 'Component Programming' but in short 'lexicon' refers to words (symbols) of a component that are used outside of a component. "A component may also contain definitions written solely to support the externally visible lexicon. We'll call the supporting definitions 'internal' words." Of course, without special effort, in Forth as in Lisp even those internal words are not really inaccessibly internal. In Lisp it's the difference of referring to an exported symbol by namespace:symbol and an un-exported one namespace::other-symbol. The other supporting notion here is a that of globally available "components", which are just a product of decomposition, not some formal thing like separate processes or libraries or what have you. "It's important to understand that a lexicon can be used by any and all of the components at higher levels. Each successive component does not bury its supporting components, as is often the case with layered approaches to design. Instead, each lexicon is free to use all of the commands beneath it. ... An important result of this approach is that the entire application employs a single syntax, which makes it easy to learn and maintain. This is why I use the term 'lexicon' and not 'language.' Languages have unique syntaxes." I'm not sure I agree about the 'easy to learn and maintain' bit but at least not making things painfully inaccessible can help, especially with re-use. There used to be OS concepts where theoretically if I wanted to play a music file, and knew I had a popular music player application installed, I could just reach in and call a function in that application, rather than writing my own or finding some "do one thing..." library. All programs on the OS were simultaneously programs and DLLs able to be used by other programs. I don't have many arguments in this space, it's just fun to think about, but good luck on your research. You might enjoy this quote: "When I see a top-down description of a system or language that has infinite libraries described by layers and layers, all I just see is a morass. I can't get a feel for it. I can't understand how the pieces fit; I can't understand something presented to me that's very complex." --Ken Thompson

akkartik 5y ago
Thank you! My ideas are definitely greatly influenced by my time in the Lisp world. Here's the first time I articulated them, back in 2011 (so before the first entry in OP): http://arclanguage.org/item?id=13263 http://arclanguage.org/item?id=13263. > in Lisp land, we have quicklisp, so you point to a library dependency like normal and it downloads it locally elsewhere (so it only need be downloaded once and not for each project). When I'm writing a program using library X, and I'm curious how some function works, I can just jump-to-source and it'll take me to that local copy. If I want to change it, I can, I just edit it, and because Lisp has "compile", "compile-file", and "load" all as standard runtime functions, not separate programs, I can compile and load my changes into the running program and immediately test them. Maybe such changes are upstreamable, maybe not. You can maintain the changes in the original files, maybe going so far as a full source copy locally instead of using the ones quicklisp brought down, or just make your own local 'patch' file of redefined functions or extended classes that you load after loading the original and the patch simply recompiles and replaces things you've changed. Is there an easy/idiomatic way to save your edits, or do they get blown away the next time you download the library? All my comments are in the context of not having tooling for this sort of thing. I wouldn't be surprised to learn the Gods at Common Lisp figured all this out years ago. (In which case, why is the whole asdf fracas happening?!)
- Jach 5y ago
  Heh, that thread brings to mind a bunch of things... One advantage of CL that helps with a 'borrowing' aspect that give people the jeebies is that the unit of compilation is much smaller than the file, so you can also borrow much less. Another is that methods are decoupled from classes so there's a lot of room for extensibility. (The LLGPL interestingly provides an incentive for the open-closed principle, i.e. you extend objects you're fine, but if you modify the library's then you are subject to the LGPL.) If you haven't read Gabriel's Patterns of Software book (free pdf on his site), I think you'd enjoy it. Your edits won't get blown away, at least by default, since quicklisp doesn't redownload a system that it knows about or check for 'corruption'. The way quicklisp does its own versioning also means if you update quicklisp's distributions lists and a newer version of the library has come out, it'll download that one into its own new folder and leave the old one alone. There's a cleanup function to clear out old things but I don't know of a case where that gets called hidden from you under the hood. Maybe there's some magic and interesting stuff related to this for emacs or mezanno OS or in the annals of ancient lisp machines but I'm a vim heretic ;) But in any case if you want to save stuff, you can just save it in a new or existing buffer... So options are basically as I described. To give a specific example, I have a script that fetches and parses some HTML to archive HN comments and I wanted to remove the HTML bits so I'd just have text, making it more markdown-y. There are lots of ways to do that and I'm pretty sure I chose one of the worst ones, but whatever. I was already using the Plump library, and after not achieving full success with its official extension mechanisms, one method I came across and stepped through was https://github.com/Shinmera/plump/blob/master/dom.lisp#L428 https://github.com/Shinmera/plump/blob/master/dom.lisp#L428 and I learned I could hijack it for my purposes. I started by editing and redefining it in place until I got what I wanted, but instead of saving my changes over the original, I simply copied it over to my script file, modifying slightly to account for namespaces e.g. it's "(defmethod plump-dom:text ((node plump:nesting-node))", thus redefining and overwriting the library implementation when my script is loaded and run. Some possible problems with this approach in general include later trying to integrate that script with other code that needs the default behavior (though CL support for :before/:after/:around auxiliary methods can help here, e.g. if I can't just subclass I could insert a seam with an :around method branching between my custom implementation over the library's without having to overwrite the library's; and in the long-term the SICL implementation will show the way to first-class environments that can allow multiple versions of stuff to co-exist nicely). Or the library could update and change the protocol, breaking my hack when I update to it. Or in other situations there may be more complex changes, like if you modify a macro but want the new modifications to apply to code using the old def, you need to redefine that old code, or if you redefine a class and want/need to specially migrate pre-existing instances you need to write an "update-instance-for-redefined-class" specializer, or if the changes just span across a lot of areas it may be infeasible to cherry-pick them into a 'patch' file/section of your script, so you're faced with choices on how much you want to fork the files of the lib and copy them into your own project to load. But anyway, all those possible problems are on me. The asdf noise isn't that big of a deal and I think is only a little related here technically since it's a rather special library situation. It's more 'interesting' socially and as a show of possible conflict from having a core piece of infrastructure provided by the implementation, but not owned/maintained by the implementation. An analogous situation would arise if gcc, clang, and visual studio all agreed to use and ship versions of musl for their libc with any other libcs being obsolete and long forgotten. A less analogous situation is the existing one that Linux distributions do -- sometimes they just distribute, sometimes they distribute-and-modify, whether they are the first place to go to report issues or whether they punt it off to upstream depends case-by-case.

by Jach 5y ago

akkartik 5y ago

Thank you! My ideas are definitely greatly influenced by my time in the Lisp world. Here's the first time I articulated them, back in 2011 (so before the first entry in OP): http://arclanguage.org/item?id=13263 http://arclanguage.org/item?id=13263. > in Lisp land, we have quicklisp, so you point to a library dependency like normal and it downloads it locally elsewhere (so it only need be downloaded once and not for each project). When I'm writing a program using library X, and I'm curious how some function works, I can just jump-to-source and it'll take me to that local copy. If I want to change it, I can, I just edit it, and because Lisp has "compile", "compile-file", and "load" all as standard runtime functions, not separate programs, I can compile and load my changes into the running program and immediately test them. Maybe such changes are upstreamable, maybe not. You can maintain the changes in the original files, maybe going so far as a full source copy locally instead of using the ones quicklisp brought down, or just make your own local 'patch' file of redefined functions or extended classes that you load after loading the original and the patch simply recompiles and replaces things you've changed. Is there an easy/idiomatic way to save your edits, or do they get blown away the next time you download the library? All my comments are in the context of not having tooling for this sort of thing. I wouldn't be surprised to learn the Gods at Common Lisp figured all this out years ago. (In which case, why is the whole asdf fracas happening?!)

Jach 5y ago
Heh, that thread brings to mind a bunch of things... One advantage of CL that helps with a 'borrowing' aspect that give people the jeebies is that the unit of compilation is much smaller than the file, so you can also borrow much less. Another is that methods are decoupled from classes so there's a lot of room for extensibility. (The LLGPL interestingly provides an incentive for the open-closed principle, i.e. you extend objects you're fine, but if you modify the library's then you are subject to the LGPL.) If you haven't read Gabriel's Patterns of Software book (free pdf on his site), I think you'd enjoy it. Your edits won't get blown away, at least by default, since quicklisp doesn't redownload a system that it knows about or check for 'corruption'. The way quicklisp does its own versioning also means if you update quicklisp's distributions lists and a newer version of the library has come out, it'll download that one into its own new folder and leave the old one alone. There's a cleanup function to clear out old things but I don't know of a case where that gets called hidden from you under the hood. Maybe there's some magic and interesting stuff related to this for emacs or mezanno OS or in the annals of ancient lisp machines but I'm a vim heretic ;) But in any case if you want to save stuff, you can just save it in a new or existing buffer... So options are basically as I described. To give a specific example, I have a script that fetches and parses some HTML to archive HN comments and I wanted to remove the HTML bits so I'd just have text, making it more markdown-y. There are lots of ways to do that and I'm pretty sure I chose one of the worst ones, but whatever. I was already using the Plump library, and after not achieving full success with its official extension mechanisms, one method I came across and stepped through was https://github.com/Shinmera/plump/blob/master/dom.lisp#L428 https://github.com/Shinmera/plump/blob/master/dom.lisp#L428 and I learned I could hijack it for my purposes. I started by editing and redefining it in place until I got what I wanted, but instead of saving my changes over the original, I simply copied it over to my script file, modifying slightly to account for namespaces e.g. it's "(defmethod plump-dom:text ((node plump:nesting-node))", thus redefining and overwriting the library implementation when my script is loaded and run. Some possible problems with this approach in general include later trying to integrate that script with other code that needs the default behavior (though CL support for :before/:after/:around auxiliary methods can help here, e.g. if I can't just subclass I could insert a seam with an :around method branching between my custom implementation over the library's without having to overwrite the library's; and in the long-term the SICL implementation will show the way to first-class environments that can allow multiple versions of stuff to co-exist nicely). Or the library could update and change the protocol, breaking my hack when I update to it. Or in other situations there may be more complex changes, like if you modify a macro but want the new modifications to apply to code using the old def, you need to redefine that old code, or if you redefine a class and want/need to specially migrate pre-existing instances you need to write an "update-instance-for-redefined-class" specializer, or if the changes just span across a lot of areas it may be infeasible to cherry-pick them into a 'patch' file/section of your script, so you're faced with choices on how much you want to fork the files of the lib and copy them into your own project to load. But anyway, all those possible problems are on me. The asdf noise isn't that big of a deal and I think is only a little related here technically since it's a rather special library situation. It's more 'interesting' socially and as a show of possible conflict from having a core piece of infrastructure provided by the implementation, but not owned/maintained by the implementation. An analogous situation would arise if gcc, clang, and visual studio all agreed to use and ship versions of musl for their libc with any other libcs being obsolete and long forgotten. A less analogous situation is the existing one that Linux distributions do -- sometimes they just distribute, sometimes they distribute-and-modify, whether they are the first place to go to report issues or whether they punt it off to upstream depends case-by-case.