5 ms·

Maybe this lisp syntax intro is a good place to ask veterans a related question. I find s-expressions quite beautiful and simplistic way to express both logic

Maybe this lisp syntax intro is a good place to ask veterans a related question. I find s-expressions quite beautiful and simplistic way to express both logic and data as a tree. On the other hand, top-level evaluation rules are often glossed over, and seem to mess up this elegance. I feel like there is a set of braces missing at top level, for s-expression concept to be complete. I don't really know enough about specific rules or reasoning why they need to be introduced. My mental model of top-level execution is an implicit `(do ...)` block. In rest of clojure code the `do` is a special form that is frowned upon because it is used only when side-effects are necessary. As it is, the 'module' concept is strictly tied to a file, which feels unnecessary. If top-level was enclosed in braces and evaluated with same rules as the rest of s-expressions, maybe we could move away from concept of files and, for example, store our code in a graph database. This would allow for IDEs that can collapse and expand code tree as needed. Rich Hickey uses expression 'place oriented programming' as derogatory term (PLOP) for relying on exact memory addresses as places for our data. It would seem to me that same term would apply to our coding practices - we tie code implementation to specific locations inside files on disk. This seems like accidental complexity that introduces mental burden. If location of implementation could be divorced from specific location inside specific source file, we could concentrate on implementation that is always in context where code is actually used. Is there any decent discussion of such concepts, or some other language that explores such direction? I'm lacking both terminology and in-depth knowledge of code evaluation to express this in more clear way.

deckiedan 7y ago
http://lighttable.com/ http://lighttable.com/ might be interesting for you if you've not seen it before... It's looking at some of those concepts. (I've not used it personally, but heard good things)
- bgorman 7y ago
  Definitely avoid lighttable. Unfortunately it is in a zombie like state with tons of open bugs, outdated dependencies and creators who abandoned the project. If you want a simple experience in an "electron" editor, I highly recommend VSCode + the Calva plugin.
qwerty456127 7y ago
I absolutely agree and think the same thing about Python - I dislike the module=file relation because I often want to split a module into multiple files and I also dream about storing code in a graph database often.
- sooheon 7y ago
  There is no such limitation in Clojure, though it is that way by convention. You can put all namespaces in one file, if that's how you want to roll. You can also split a module into multiple files.
- anentropic 7y ago
  There's nothing stopping you from splitting a module across multiple files in Python Yes, it will result in sub-modules If you hate addressing code in the sub-modules via the sub-module path you can import them in the __init__.py of the parent module Personally I dislike languages which add another layer of (IMHO worthless) complexity by decoupling the owning module of the code from the file path, so you have now have to resolve that before you can understand it.
  - qwerty456127 7y ago
    Well, that actually needs to be transparent to be useful for the way I want to use it. I can't even understand why would one need sub-modules so far. The actual reason is to have different members (i.e. every method) or sets of members of a same class to be implemented in separate files. I just hate to have long code files and to have code above and below a function declaration. Partial classes in C# are an example of how can that be implemented the way I like it. The only problem with the C# way is it adds an additional indentation level. I would prefer the module (or even the class) to be declared in the beginning of the file without contributing to indentation.
sooheon 7y ago
The file system is a storage medium that is ubiquitous and requires no install. It's an implementation detail, (and convenient for piggybacking on the Java classpath construction), but semantically Clojure can be represented as a graph where terminal nodes are forms and internal nodes are namespaces. I bet you could save all that to a db if you want, and load it all into a running repl again as well. You can put all your `ns` forms in one file, one after the other, and if you really wanted that top level s-expression, you could make a macro for that as well (check out the `ns` definition). Still you're stuck with the single file where everything lives, unless you want to move into a graph-db, but why? You'll have to bootstrap from code as text stored somewhere (even in your memory) at some point. > IDEs that can collapse and expand code tree as needed. This is "only" a representation problem, because the semantics are already there, you don't have to change the storage medium to buy this behavior.
- jmiskovic 7y ago
  Thanks for shedding some light on technical side. I'm still conflicted about the mental model. We should organize our code into deeply nested tree (with exception of cyclical references for recursions), but at top level we use linear and procedural model. In each s-expression there is typically 3 to 5 elements, but at top level there's usually more than dozen definitions. It seems jarring especially when contrasted to elegance of s-expressions. > This is "only" a representation problem What if I want to write new function? I still have to redirect it to specific location in specific file.
  - sooheon 7y ago
    > We should organize our code into deeply nested tree (with exception of cyclical references for recursions), but at top level we use linear and procedural model A filesystem with nested directories and files is a tree. The contents of a file are read in linear fashion, for the same reason you'd read an s-expression character by character--because it's simple and consistent. > In each s-expression there is typically 3 to 5 elements, but at top level there's usually more than dozen definitions. It seems jarring especially when contrasted to elegance of s-expressions There is no law for the "branching factor" of code. It depends on your style, the problem, etc. It seems like a somewhat superficial elegance you're chasing. > What if I want to write new function? I still have to redirect it to specific location in specific file. I'm trying to picture what your ideal alternative is, and it doesn't seem much different. Codebase backed by a graph representation that explicitly encodes dependencies in code loading order? When you add a new sexp you add a new edge and node? That is essentially what we have now, `require`s `refer`s and `declare`s as needed, and the convention that you may depend on everything introduced above your current location in the file.
roenxi 7y ago
> As it is, the 'module' concept is strictly tied to a file, which feels unnecessary. In the case of Clojure specifically; I'm not sure I've ever seen a concept of a 'module'. Clojure has a concept of a 'lib' [0] which is an artifact left over from being implemented on Java. The internal concept to Clojure for dividing up code is the namespace [1] which has no concept or link to files at all. There are two things going on here. First is the text files of source code on your computer that the build systems or what have you are interacting with. They are organised into files because storing code on a filesystem demands it. The second is once the code is parsed and moves into RAM those concepts fall away fairly quickly and you should be left mainly manipulating namespaces which are only linked to files if programmers want them to be. If you develop a build system that loads code from a database of some sort then it probably wouldn't have a concept of files, but I don't think anyone has come up with an impressive model for what the advantage is to that. You'd have to reimplement Git and create an Emacs extension to interface with the database; so there would need to be a big benefit to justify the cost. > My mental model of top-level execution is an implicit `(do ...)` block. In rest of clojure code the `do` is a special form that is frowned upon because it is used only when side-effects are necessary. Clojure isn't about ideological purity. If it seems that way it is only because Rich Hickey believes strongly that ideologically disciplined code is easier to debug. (do [...]) clauses are only a problem if they are being used in preference to an alternate way that is easier to test and debug. [0] https://clojure.org/reference/libs https://clojure.org/reference/libs [1] https://clojure.org/reference/namespaces https://clojure.org/reference/namespaces
- jmiskovic 7y ago
  Thank you for this wonderful explanation. Here are benefits that I think might justify the cost. - Easier code browsing: What is the body of this function? What are some optional arguments not used here? Where else is this function called from? - When teaching, ability to chose depth of visible code means more focused learning. For example, start from overview algorithm with function code collapsed, and then expand one by one to dive into each. - Reduced mental overhead due by eliminating concept of 'physical' location of code. No refactoring to extract common code into separate module, just reference same subtree from multiple locations. - Persistent code structures would allow for specializing a function to work differently in one place, while other callers still point to older implementation (until you decide that new implementation is superior). - A more strict tree-based approach to code would bring in many more visualization and manipulation techniques. It would be sort of seamless blend between text-based and visual programming. Regarding Git capabilities, I feel that Datomic model would provide much of same benefits. It would even allow for complete history of development without manual effort of checking in. Of course, talk is cheap and these are just random ideas. I'm not really complaining about Clojure. Its success is mostly due to pragmatic choices. I'm more curious has anyone put serious thought into purely tree-based lisp, and decided that it is somehow worse than top-level + s-expressions?
mark_l_watson 7y ago
re: ""As it is, the 'module' concept is strictly tied to a file, which feels unnecessary."" In Common Lisp you get to decide which package each source file contributes to. For utilities that are (or will be) used for multiple projects, I use a different package for each library and for small libraries it is likely that all the code will be in one file (plus a project and 'asd' file, details for which aren't important here). For non-library projects, like a complete application, I usually use many source files that all contribute to the same package. The point is that a Lisp like Common Lisp is almost infinitely flexible, but it really helps if you have a standard way that you like to set up your projects. I used to, back in the 1980s, use a hardware Lisp Machine where all my work was in one environment. I feel like I have something similar now by maintaining a large mono git repo for all of my work and research in Common Lisp. The Quicklisp package manager keeps my own packages and all 3rd party packages available to me in a transparent way. I love my own working environment, but if you go down the Lisp path, you should get an environment set up that is right for you.
- kazinator 7y ago
  In Common Lisp, you get to decide which package each individual token of source code contributes to. If a package abc exists, we can write abc::def. Now the abc::def symbol exists, if it did not before: the symbol named "DEF" in the package named "ABC". If we write (defun abc::def ...) we are binding that symbol to a function. Informally speaking, we are defining a function def in pakage abc.
dandanua 7y ago
Unison language has a beautiful idea to solve this https://www.unisonweb.org/docs/tour/#%F0%9F%A7%A0-the-big-technical-idea https://www.unisonweb.org/docs/tour/#%F0%9F%A7%A0-the-big-te... I didn't dive deep and still a bit skeptical, but it looks very promising.
- jmiskovic 7y ago
  I agree, hashing content of syntax sub-tree is great idea. Also I like the idea of personalizing the interpreter, like output "I found and typechecked these definitions". Now when things go wrong you have somebody to be angry at.
kazinator 7y ago
In Common Lisp, and similar dialects, the module concept is certainly not tied to a file. In fact, there is no module concept. Packages work at the symbol level. Any file can use symbols from any package. The concept of a compilation unit is closely tied to that of a file. That's not even remotely the same thing as a module. Even so, Common Lisp has with-compilation-unit which allows multiple files to be compiled together. They still emit separate files, but for the purpose of diagnosis (warnings) they are considered one unit. In concrete terms, complaining about undefined variables and functions is deferred until the entire compilation unit is processed. Top-level form processing is deliberately contrived in order to make certain practical situations conform to obvious expectations. For instance, suppose that we have a top-level form: (progn (defmacro foo ...) (foo)) this cannot work if the entire form is macro-expanded first, and then evaluated, because then (foo) is undefined. The eval-when control over the top-level is also very important in the face of compilation.
fulafel 7y ago
Files become namespaces (clojure term for modules) in the Clojure default behaviour, but namespaces can also be manipulated from code and don't need to have associated files. See https://clojure.org/reference/namespaces https://clojure.org/reference/namespaces - you can create and remove namespaces, introspect them, work with ns aliases, etc. So namespaces are not strictly tied to files in Clojure.
jonahbenton 7y ago
Yes, much of this space is fairly well trod, though completely agree the threads are hard to find. I'll try to point some out: 1. Clojure, top level forms, trees, and evaluation Agree in part about top level evaluation rules in Clojure. It does seem, for instance, like the ns form should enclose the rest of the forms that comprise that namespace, rather than essentially doing something that is unusual for Clojure- silently changing what seems to be a global context. When one digs a little deeper, however, there is a logic to those semantics. The main reason comes from the problem that the compiler faces in having to reconcile the use of a program thing- a symbol or name or variable or whatever-one-calls-the-named-elements that are used in a program- with the defining of that thing. There are basically two approaches to this problem. The compiler can read all the source code, find all the definitions, then reread all the source code, and match the uses to those definitions- and only then inform the programmer if there is some problem where a use doesn't match or doesn't have a definition. Even with modern computers, for large programs, this is too expensive and time consuming. What many compilers do instead is to require that any used names are defined "first". Clojure does this- it reads files from top to bottom, and it requires that any used names are defined earlier in the file. This notion of earlier- this notion that things defined in a program have an ordering to them, not just in their execution but also in their composition- this is deep and pervasive, and puts the lie in the idea that a program is just a big tree. One branch off this tree, so to speak, where the ordering in a file doesn't correspond to the compositional or execution ordering, is in the functional programming concept of monads. 2. Alternatives to file storage for program modules It is a pretty old idea that files are a poor way of storing source code (sorry). There is a long train of work that Wikipedia summarizes poorly with almost no references under Source Code In Database: https://en.wikipedia.org/wiki/Source_Code_in_Database https://en.wikipedia.org/wiki/Source_Code_in_Database. The idea here is that persisting code in a data structure and providing more "intuitive" tools for editing is better than requiring humans to work in files. (Microsoft even tried in the 1990s to roll out a version of Windows that used a database for pervasive structured storage, rather than a file system. This was a failure, and a lot has been written about it- google WinFS). Plain text files have a lot of underappreciated ergonomic properties. Their use doesn't keep tools from utilizing clever data structures to assist in the management and authoring of code in files. The SCID work has ultimately found its way into the cool incremental helpers and structural editors that most IDEs use now (Cursive/Paredit for Clojure: https://cursive-ide.com/userguide/paredit.html https://cursive-ide.com/userguide/paredit.html) Forgoing the text editing paradigm altogether takes you into the world of visual programming editors, which also have a long history. An influential player in the space from the early rise of personal computers was a product called ProGraph. This technique is also now pervasive in tools like Scratch, but also in big data where flow graphs for processing immense streams of data are often constructed using visual tools, for instance, Nifi. 3. Literate programming Another thread is Literate Programming, originally invented by Don Knuth. The idea is that the most important consumers of a program are other humans, not the computer, so one should author using tools that create both an artifact that a human can read, with both prose and code interspersed- as well as the code itself for a compiler to consume. But the combined prose/code artifact is a better way for communicating to other humans about the semantics of a program, than just the code. This is a particular endearing thread, and the tool called Marginalia in the Clojure world provides something of the experience that Knuth intended. 4. Version control and program semantics Yet another relevant thread is in version control, where the inability of files to keep the history of a programming authoring process is addressed. Early in Clojure's life Rich Hickey created a tool called Codeq: https://github.com/Datomic/codeq https://github.com/Datomic/codeq that loaded a git repo- essentially a graph of changes to program files- into a Datomic database- where Datomic can be seen as a graph db. There has been some more recent work to be able to run semantic queries on those graphs. This is immensely useful for looking for patterns of code that may have security problems. A company doing a lot of work in this space is called Source(d). Another set of tools for mining version control comes from a company called Empear, started by a programmer named Adam Tornhill. His work- originally in Clojure- looks at things like patterns of paired changes across files. Cases where the same sections of code in the same files are changed in the same commits demonstrate high "coupling" and are a "code smell." == All of this is really about building and maintaining a semantic model from the syntactic artifacts, which is what I read you as being ultimately interested in. There's a lot more, but that's all I have time for now. Hope that's helpful.

deckiedan 7y ago

http://lighttable.com/ http://lighttable.com/ might be interesting for you if you've not seen it before... It's looking at some of those concepts. (I've not used it personally, but heard good things)

bgorman 7y ago
Definitely avoid lighttable. Unfortunately it is in a zombie like state with tons of open bugs, outdated dependencies and creators who abandoned the project. If you want a simple experience in an "electron" editor, I highly recommend VSCode + the Calva plugin.

qwerty456127 7y ago

I absolutely agree and think the same thing about Python - I dislike the module=file relation because I often want to split a module into multiple files and I also dream about storing code in a graph database often.

sooheon 7y ago
There is no such limitation in Clojure, though it is that way by convention. You can put all namespaces in one file, if that's how you want to roll. You can also split a module into multiple files.
anentropic 7y ago
There's nothing stopping you from splitting a module across multiple files in Python Yes, it will result in sub-modules If you hate addressing code in the sub-modules via the sub-module path you can import them in the __init__.py of the parent module Personally I dislike languages which add another layer of (IMHO worthless) complexity by decoupling the owning module of the code from the file path, so you have now have to resolve that before you can understand it.
- qwerty456127 7y ago
  Well, that actually needs to be transparent to be useful for the way I want to use it. I can't even understand why would one need sub-modules so far. The actual reason is to have different members (i.e. every method) or sets of members of a same class to be implemented in separate files. I just hate to have long code files and to have code above and below a function declaration. Partial classes in C# are an example of how can that be implemented the way I like it. The only problem with the C# way is it adds an additional indentation level. I would prefer the module (or even the class) to be declared in the beginning of the file without contributing to indentation.

sooheon 7y ago

The file system is a storage medium that is ubiquitous and requires no install. It's an implementation detail, (and convenient for piggybacking on the Java classpath construction), but semantically Clojure can be represented as a graph where terminal nodes are forms and internal nodes are namespaces. I bet you could save all that to a db if you want, and load it all into a running repl again as well. You can put all your `ns` forms in one file, one after the other, and if you really wanted that top level s-expression, you could make a macro for that as well (check out the `ns` definition). Still you're stuck with the single file where everything lives, unless you want to move into a graph-db, but why? You'll have to bootstrap from code as text stored somewhere (even in your memory) at some point. > IDEs that can collapse and expand code tree as needed. This is "only" a representation problem, because the semantics are already there, you don't have to change the storage medium to buy this behavior.

jmiskovic 7y ago
Thanks for shedding some light on technical side. I'm still conflicted about the mental model. We should organize our code into deeply nested tree (with exception of cyclical references for recursions), but at top level we use linear and procedural model. In each s-expression there is typically 3 to 5 elements, but at top level there's usually more than dozen definitions. It seems jarring especially when contrasted to elegance of s-expressions. > This is "only" a representation problem What if I want to write new function? I still have to redirect it to specific location in specific file.
- sooheon 7y ago
  > We should organize our code into deeply nested tree (with exception of cyclical references for recursions), but at top level we use linear and procedural model A filesystem with nested directories and files is a tree. The contents of a file are read in linear fashion, for the same reason you'd read an s-expression character by character--because it's simple and consistent. > In each s-expression there is typically 3 to 5 elements, but at top level there's usually more than dozen definitions. It seems jarring especially when contrasted to elegance of s-expressions There is no law for the "branching factor" of code. It depends on your style, the problem, etc. It seems like a somewhat superficial elegance you're chasing. > What if I want to write new function? I still have to redirect it to specific location in specific file. I'm trying to picture what your ideal alternative is, and it doesn't seem much different. Codebase backed by a graph representation that explicitly encodes dependencies in code loading order? When you add a new sexp you add a new edge and node? That is essentially what we have now, `require`s `refer`s and `declare`s as needed, and the convention that you may depend on everything introduced above your current location in the file.

roenxi 7y ago

> As it is, the 'module' concept is strictly tied to a file, which feels unnecessary. In the case of Clojure specifically; I'm not sure I've ever seen a concept of a 'module'. Clojure has a concept of a 'lib' [0] which is an artifact left over from being implemented on Java. The internal concept to Clojure for dividing up code is the namespace [1] which has no concept or link to files at all. There are two things going on here. First is the text files of source code on your computer that the build systems or what have you are interacting with. They are organised into files because storing code on a filesystem demands it. The second is once the code is parsed and moves into RAM those concepts fall away fairly quickly and you should be left mainly manipulating namespaces which are only linked to files if programmers want them to be. If you develop a build system that loads code from a database of some sort then it probably wouldn't have a concept of files, but I don't think anyone has come up with an impressive model for what the advantage is to that. You'd have to reimplement Git and create an Emacs extension to interface with the database; so there would need to be a big benefit to justify the cost. > My mental model of top-level execution is an implicit `(do ...)` block. In rest of clojure code the `do` is a special form that is frowned upon because it is used only when side-effects are necessary. Clojure isn't about ideological purity. If it seems that way it is only because Rich Hickey believes strongly that ideologically disciplined code is easier to debug. (do [...]) clauses are only a problem if they are being used in preference to an alternate way that is easier to test and debug. [0] https://clojure.org/reference/libs https://clojure.org/reference/libs [1] https://clojure.org/reference/namespaces https://clojure.org/reference/namespaces

jmiskovic 7y ago
Thank you for this wonderful explanation. Here are benefits that I think might justify the cost. - Easier code browsing: What is the body of this function? What are some optional arguments not used here? Where else is this function called from? - When teaching, ability to chose depth of visible code means more focused learning. For example, start from overview algorithm with function code collapsed, and then expand one by one to dive into each. - Reduced mental overhead due by eliminating concept of 'physical' location of code. No refactoring to extract common code into separate module, just reference same subtree from multiple locations. - Persistent code structures would allow for specializing a function to work differently in one place, while other callers still point to older implementation (until you decide that new implementation is superior). - A more strict tree-based approach to code would bring in many more visualization and manipulation techniques. It would be sort of seamless blend between text-based and visual programming. Regarding Git capabilities, I feel that Datomic model would provide much of same benefits. It would even allow for complete history of development without manual effort of checking in. Of course, talk is cheap and these are just random ideas. I'm not really complaining about Clojure. Its success is mostly due to pragmatic choices. I'm more curious has anyone put serious thought into purely tree-based lisp, and decided that it is somehow worse than top-level + s-expressions?

mark_l_watson 7y ago

re: ""As it is, the 'module' concept is strictly tied to a file, which feels unnecessary."" In Common Lisp you get to decide which package each source file contributes to. For utilities that are (or will be) used for multiple projects, I use a different package for each library and for small libraries it is likely that all the code will be in one file (plus a project and 'asd' file, details for which aren't important here). For non-library projects, like a complete application, I usually use many source files that all contribute to the same package. The point is that a Lisp like Common Lisp is almost infinitely flexible, but it really helps if you have a standard way that you like to set up your projects. I used to, back in the 1980s, use a hardware Lisp Machine where all my work was in one environment. I feel like I have something similar now by maintaining a large mono git repo for all of my work and research in Common Lisp. The Quicklisp package manager keeps my own packages and all 3rd party packages available to me in a transparent way. I love my own working environment, but if you go down the Lisp path, you should get an environment set up that is right for you.

kazinator 7y ago
In Common Lisp, you get to decide which package each individual token of source code contributes to. If a package abc exists, we can write abc::def. Now the abc::def symbol exists, if it did not before: the symbol named "DEF" in the package named "ABC". If we write (defun abc::def ...) we are binding that symbol to a function. Informally speaking, we are defining a function def in pakage abc.

dandanua 7y ago

Unison language has a beautiful idea to solve this https://www.unisonweb.org/docs/tour/#%F0%9F%A7%A0-the-big-technical-idea https://www.unisonweb.org/docs/tour/#%F0%9F%A7%A0-the-big-te... I didn't dive deep and still a bit skeptical, but it looks very promising.

jmiskovic 7y ago
I agree, hashing content of syntax sub-tree is great idea. Also I like the idea of personalizing the interpreter, like output "I found and typechecked these definitions". Now when things go wrong you have somebody to be angry at.

kazinator 7y ago

In Common Lisp, and similar dialects, the module concept is certainly not tied to a file. In fact, there is no module concept. Packages work at the symbol level. Any file can use symbols from any package. The concept of a compilation unit is closely tied to that of a file. That's not even remotely the same thing as a module. Even so, Common Lisp has with-compilation-unit which allows multiple files to be compiled together. They still emit separate files, but for the purpose of diagnosis (warnings) they are considered one unit. In concrete terms, complaining about undefined variables and functions is deferred until the entire compilation unit is processed. Top-level form processing is deliberately contrived in order to make certain practical situations conform to obvious expectations. For instance, suppose that we have a top-level form: (progn (defmacro foo ...) (foo)) this cannot work if the entire form is macro-expanded first, and then evaluated, because then (foo) is undefined. The eval-when control over the top-level is also very important in the face of compilation.

fulafel 7y ago

Files become namespaces (clojure term for modules) in the Clojure default behaviour, but namespaces can also be manipulated from code and don't need to have associated files. See https://clojure.org/reference/namespaces https://clojure.org/reference/namespaces - you can create and remove namespaces, introspect them, work with ns aliases, etc. So namespaces are not strictly tied to files in Clojure.

jonahbenton 7y ago

Yes, much of this space is fairly well trod, though completely agree the threads are hard to find. I'll try to point some out: 1. Clojure, top level forms, trees, and evaluation Agree in part about top level evaluation rules in Clojure. It does seem, for instance, like the ns form should enclose the rest of the forms that comprise that namespace, rather than essentially doing something that is unusual for Clojure- silently changing what seems to be a global context. When one digs a little deeper, however, there is a logic to those semantics. The main reason comes from the problem that the compiler faces in having to reconcile the use of a program thing- a symbol or name or variable or whatever-one-calls-the-named-elements that are used in a program- with the defining of that thing. There are basically two approaches to this problem. The compiler can read all the source code, find all the definitions, then reread all the source code, and match the uses to those definitions- and only then inform the programmer if there is some problem where a use doesn't match or doesn't have a definition. Even with modern computers, for large programs, this is too expensive and time consuming. What many compilers do instead is to require that any used names are defined "first". Clojure does this- it reads files from top to bottom, and it requires that any used names are defined earlier in the file. This notion of earlier- this notion that things defined in a program have an ordering to them, not just in their execution but also in their composition- this is deep and pervasive, and puts the lie in the idea that a program is just a big tree. One branch off this tree, so to speak, where the ordering in a file doesn't correspond to the compositional or execution ordering, is in the functional programming concept of monads. 2. Alternatives to file storage for program modules It is a pretty old idea that files are a poor way of storing source code (sorry). There is a long train of work that Wikipedia summarizes poorly with almost no references under Source Code In Database: https://en.wikipedia.org/wiki/Source_Code_in_Database https://en.wikipedia.org/wiki/Source_Code_in_Database. The idea here is that persisting code in a data structure and providing more "intuitive" tools for editing is better than requiring humans to work in files. (Microsoft even tried in the 1990s to roll out a version of Windows that used a database for pervasive structured storage, rather than a file system. This was a failure, and a lot has been written about it- google WinFS). Plain text files have a lot of underappreciated ergonomic properties. Their use doesn't keep tools from utilizing clever data structures to assist in the management and authoring of code in files. The SCID work has ultimately found its way into the cool incremental helpers and structural editors that most IDEs use now (Cursive/Paredit for Clojure: https://cursive-ide.com/userguide/paredit.html https://cursive-ide.com/userguide/paredit.html) Forgoing the text editing paradigm altogether takes you into the world of visual programming editors, which also have a long history. An influential player in the space from the early rise of personal computers was a product called ProGraph. This technique is also now pervasive in tools like Scratch, but also in big data where flow graphs for processing immense streams of data are often constructed using visual tools, for instance, Nifi. 3. Literate programming Another thread is Literate Programming, originally invented by Don Knuth. The idea is that the most important consumers of a program are other humans, not the computer, so one should author using tools that create both an artifact that a human can read, with both prose and code interspersed- as well as the code itself for a compiler to consume. But the combined prose/code artifact is a better way for communicating to other humans about the semantics of a program, than just the code. This is a particular endearing thread, and the tool called Marginalia in the Clojure world provides something of the experience that Knuth intended. 4. Version control and program semantics Yet another relevant thread is in version control, where the inability of files to keep the history of a programming authoring process is addressed. Early in Clojure's life Rich Hickey created a tool called Codeq: https://github.com/Datomic/codeq https://github.com/Datomic/codeq that loaded a git repo- essentially a graph of changes to program files- into a Datomic database- where Datomic can be seen as a graph db. There has been some more recent work to be able to run semantic queries on those graphs. This is immensely useful for looking for patterns of code that may have security problems. A company doing a lot of work in this space is called Source(d). Another set of tools for mining version control comes from a company called Empear, started by a programmer named Adam Tornhill. His work- originally in Clojure- looks at things like patterns of paired changes across files. Cases where the same sections of code in the same files are changed in the same commits demonstrate high "coupling" and are a "code smell." == All of this is really about building and maintaining a semantic model from the syntactic artifacts, which is what I read you as being ultimately interested in. There's a lot more, but that's all I have time for now. Hope that's helpful.