5 ms·

Any results from your research? That seems interesting.

matheusmoreira 3y ago
Research is a bit too strong a word for it but I'll summarize my conclusions. Most people I talked with seemed used to and very happy with the Python model: modules are distinct namespaces for symbols and their referenced objects. Pretty much everyone I talked to had no difficulty understanding modules, it's always the package management that makes things complex no matter which language it is. Something I spent quite a bit of time thinking about was whether it was worth reifying that model into the language as first class objects. In other words, should import be a special keyword that makes the interpreter magically bind symbols or a function which returns a regular everyday normal object? At the end of the day, Python's modules are just dictionaries, just like Javascript modules. Should I make that fact apparent or hide it behind language keywords? Interestingly, both languages went in the opposite directions: Python went from special import syntax to allowing you to access modules as a dictionary, while Javascript went from a require function that returns an object to an import statement. I ultimately chose a somewhat weird mix of both, powered by lisp's flexibility. I also tried to figure out how compiled languages approached modules. So I dug up literature on Modula, Modula-2 and Oberon and tried to figure out how they represented modules. This ended up having a significant influence in my design in the form of isolated modules with export control. Each module contains its own table of symbols and their references. Importing is just setting a local symbol to the value of the other module's symbol, and only symbols in the exports list can be imported. I also liked the qualified and unqualified names: added the option to prefix the module name to the local symbol. I also thought a lot about how to map modules to the file system. The idea of program folders from Windows has been an inspiration for a long time now. The idea is if the module itself is reachable then all of its submodules are also reachable by the loader. I also think it's important that no file escapes the package directory. Python and Ruby frequently have thing.{rb,py} scripts and thing/ directories side-by-side, I sought to eliminate that for the module's root directory only which results in a main file like thing/thing.{rb,py}. To enable module-oriented development, I've found the most important feature of the modules and packaging system is editable libraries. Like pip's editable installs and npm link. When I develop a project, I often end up with several supporting libraries. Languages should support linking these local versions to the main project so they can be developed simultaneously.
- giraffe_lady 3y ago
  Have you looked into the "first class modules" system used by ocaml? AFAIK it's the only language of its maturity to use something like it. It's very powerful without adding mandatory complexity. But because basically no one comes to it with prior experience in a similar approach, it's hard to gauge how powerful, or when the added complexity of using it fully is actually worth it.
  - matheusmoreira 3y ago
    I have! I didn't reflect much on it though because I'm focused on creating a dynamic language. Perhaps I did not understand it? Another interesting concept I ran into while researching modules is parameterized modules. Essentially, modules that take their dependencies as arguments. The modular equivalent to dependency injection I suppose. Instead of a module importing by symbol a specific library as a dependency and some package manager resolving it to actual files in the load path later on, the programmer explicitly loads the library and passes it to the module as an argument. Instead of the symbolic imports we're all used to: (import lib); lib imports lib2 internally Module importing becomes analogous to function calls which construct an instance of the module given its dependencies: (import (lib2)) (import (lib lib2)) This is really elegant and more or less reifies package management into the language. However, it presents serious ergonomics issues because it forces the programmer to deal with all these package management and library loading details. The truth is we want to sweep all that ugly stuff under the rug, not deal with it every single time we import a module. The main benefit, the loose coupling that stems from the ability to substitute dependencies without having to change the importing module, can be accomplished in a declarative manner via package managers. Arch Linux packages for example may have a "provides" variable which allows multiple packages to implement an interface of sorts and be used interchangeably to satisfy dependencies. So I think parameterized modules imposed significant costs for little benefit.
- urxvtcd 3y ago
  Thanks for response. Some of these thoughts floated around in my head as well, though less ordered and well articulated. One thought regarding the quantification element. I’ve grown to appreciate the uniformity of Go’s solution, where you always import an entire namespace, optionally aliasing it to avoid conflicts. This small restriction makes some naming decisions simpler (for example sticking to config.Parse, and not the stuttering config.ParseConfig). Additionally it sprinkles the namespaces over the code so you get at least _some_ feel for them. On another note, on some occasions I have thought to myself that it’s easy for the import section to get to little scrutiny on reviews, and wondered if there’s something that could make us wonder „should x depend on y?” more often.

Research is a bit too strong a word for it but I'll summarize my conclusions. Most people I talked with seemed used to and very happy with the Python model: modules are distinct namespaces for symbols and their referenced objects. Pretty much everyone I talked to had no difficulty understanding modules, it's always the package management that makes things complex no matter which language it is. Something I spent quite a bit of time thinking about was whether it was worth reifying that model into the language as first class objects. In other words, should import be a special keyword that makes the interpreter magically bind symbols or a function which returns a regular everyday normal object? At the end of the day, Python's modules are just dictionaries, just like Javascript modules. Should I make that fact apparent or hide it behind language keywords? Interestingly, both languages went in the opposite directions: Python went from special import syntax to allowing you to access modules as a dictionary, while Javascript went from a require function that returns an object to an import statement. I ultimately chose a somewhat weird mix of both, powered by lisp's flexibility. I also tried to figure out how compiled languages approached modules. So I dug up literature on Modula, Modula-2 and Oberon and tried to figure out how they represented modules. This ended up having a significant influence in my design in the form of isolated modules with export control. Each module contains its own table of symbols and their references. Importing is just setting a local symbol to the value of the other module's symbol, and only symbols in the exports list can be imported. I also liked the qualified and unqualified names: added the option to prefix the module name to the local symbol. I also thought a lot about how to map modules to the file system. The idea of program folders from Windows has been an inspiration for a long time now. The idea is if the module itself is reachable then all of its submodules are also reachable by the loader. I also think it's important that no file escapes the package directory. Python and Ruby frequently have thing.{rb,py} scripts and thing/ directories side-by-side, I sought to eliminate that for the module's root directory only which results in a main file like thing/thing.{rb,py}. To enable module-oriented development, I've found the most important feature of the modules and packaging system is editable libraries. Like pip's editable installs and npm link. When I develop a project, I often end up with several supporting libraries. Languages should support linking these local versions to the main project so they can be developed simultaneously.

giraffe_lady 3y ago
Have you looked into the "first class modules" system used by ocaml? AFAIK it's the only language of its maturity to use something like it. It's very powerful without adding mandatory complexity. But because basically no one comes to it with prior experience in a similar approach, it's hard to gauge how powerful, or when the added complexity of using it fully is actually worth it.
- matheusmoreira 3y ago
  I have! I didn't reflect much on it though because I'm focused on creating a dynamic language. Perhaps I did not understand it? Another interesting concept I ran into while researching modules is parameterized modules. Essentially, modules that take their dependencies as arguments. The modular equivalent to dependency injection I suppose. Instead of a module importing by symbol a specific library as a dependency and some package manager resolving it to actual files in the load path later on, the programmer explicitly loads the library and passes it to the module as an argument. Instead of the symbolic imports we're all used to: (import lib); lib imports lib2 internally Module importing becomes analogous to function calls which construct an instance of the module given its dependencies: (import (lib2)) (import (lib lib2)) This is really elegant and more or less reifies package management into the language. However, it presents serious ergonomics issues because it forces the programmer to deal with all these package management and library loading details. The truth is we want to sweep all that ugly stuff under the rug, not deal with it every single time we import a module. The main benefit, the loose coupling that stems from the ability to substitute dependencies without having to change the importing module, can be accomplished in a declarative manner via package managers. Arch Linux packages for example may have a "provides" variable which allows multiple packages to implement an interface of sorts and be used interchangeably to satisfy dependencies. So I think parameterized modules imposed significant costs for little benefit.
urxvtcd 3y ago
Thanks for response. Some of these thoughts floated around in my head as well, though less ordered and well articulated. One thought regarding the quantification element. I’ve grown to appreciate the uniformity of Go’s solution, where you always import an entire namespace, optionally aliasing it to avoid conflicts. This small restriction makes some naming decisions simpler (for example sticking to config.Parse, and not the stuttering config.ParseConfig). Additionally it sprinkles the namespaces over the code so you get at least _some_ feel for them. On another note, on some occasions I have thought to myself that it’s easy for the import section to get to little scrutiny on reviews, and wondered if there’s something that could make us wonder „should x depend on y?” more often.