4 ms·

(author here) Hm very interesting, I hadn't seen this article! I would say it's highly related, but doesn't contradict anything I'm saying. His examples are a

by chubot 5y ago

(author here) Hm very interesting, I hadn't seen this article! I would say it's highly related, but doesn't contradict anything I'm saying. His examples are at a higher application level, in the domains of images and animation. I'm more focused the system level, like the basics of building code, distributing it, dynamically composing it, monitoring it, etc. There's definitely less need for types and schemas at that level, and more need for computation in his application domains. (I guess in networking terms this is a "smart middlebox" problem, as opposed to passive ones.) The commonality is that they are trying to avoid combinatorial explosions of code by having a shared representation, e.g. avoiding writing the same transformations for both PDF and JPG. I'd also say that he is overly negative on a few fronts -- I agree with a lot of it, but I'd frame the evolution as a success, not a failure. I guess my point the narrow waist is basically the only way we know how to build (really evolve) large scale systems over large time frames. We can do it well or do it badly, explicitly or implicitly... From the code and systems I've worked with, we could easily do a lot better. But I appreciate this article because identifying these kinds of problems is the first step. ---- Some random responses: Putting data inside a free-form key/value map doesn't change things much. It's barely an improvement over having a unknownData byte[] mix-in on each native type. It only pays off if you actually adopt a decomposable model and stick with it. That way the data is not unknown, but always provides a serviceable view on its own. Arguably this is the killer feature of a dynamic language. The benefit of "extensible data" is mainly "fully introspectable without recompilation." This part feels overly negative ... I would just frame this as a tradeoff between static and dynamic. There could be reasons that dynamic doesn't work in his domain, but that doesn't mean it doesn't work elsewhere. I have a bunch of material in drafts about types and distributed systems, sort of related to this "Maybe Not" discussion: https://lobste.rs/s/zdvg9y/maybe_not_rich_hickey https://lobste.rs/s/zdvg9y/maybe_not_rich_hickey (i.e. schemas/shapes should be decoupled from field presence; one is reusable and global and the other is local) From what I know Clojure has some good framings of the versioning problem, with Spec and RDF inspired schemas, but I haven't used it. Rather than the brittle versioning model, Hickey frames software evolution as "strengthen a promise" and "relax a guarantee". I don't know if it would be useful in those domains but it's worth addressing. I would be interested in unpacking the problem differing PDF and JPG metadata a little more... Off hand I'm not sure why it's so difficult; I feel like it's mostly a problem in certain statically-typed systems. I guess to put a big stake in the ground, you could say extensibility is inherently dynamic. Simply because you don't know what code is going to operate on your data 10 years from now. It hasn't been written yet, and the people who need it haven't been born yet. It's impossible in reality, not just in the type system :) I think the difference is his domains are living with the confines of a single machine and single application, so there is the expectation that you should be able to do better (have more type safety). While nobody expects to be able to reboot the Internet all at once and upgrade it. And even say at Google nobody wants to recompile all the code in a cluster at once and reboot it.

skybrian 5y ago
Yes, I don’t agree with everything in the article, but it points out some sticky problems that can make things complicated. One example that’s related to programming languages is the evolution of an abstract syntax tree. We often expect new compilers to work with old code when a new kind of AST node is added, but to what extent should existing tools work with new code that has a node type that it doesn’t understand? Also, if your AST datatype is a library, does adding a new node type immediately cause compiler errors in every tool that uses the AST, or do those tools compile fine and break at runtime when given new code? Sometimes as a workaround, “metadata” gets put in specially formatted comments, as a way of saying that most language tools can ignore it. Alternately there can be general-purpose annotations (as Java has). Also, often an intermediate representation gets used as a “narrow waist” but they are tricky to design because “lowering” code will often discard high-level constructs.
- chubot 5y ago
  Yes, the AST problem is one I've had a lot! It's hard. (The whole Oil language is defined as a big ML-style statically typed data structure, but it's not exposed: https://www.oilshell.org/release/0.9.8/source-code.wwz/frontend/syntax.asdl https://www.oilshell.org/release/0.9.8/source-code.wwz/front...) What I'd say is that we already have a narrow waist for every language -- text. That is the one that's stable! Having an additional stable AST format seems to be an insurmountable problem, especially in most statically typed languages. The basic reason is that it's easier to evolve text in a backward compatible way than types. This is probably obvious to some people, but controversial to others... It has a lot to do with the "ignoring new data" problem discussed in the blog post. It would be nice to unpack this a bit and find some references. ---- Related: I recently circulated this language implementation, which has a front end in OCaml, a backend in C++, and the IR is transmitted via protobuf. https://old.reddit.com/r/ProgrammingLanguages/comments/ss3w6n/an_accessible_introduction_to_type_theory_and/ https://old.reddit.com/r/ProgrammingLanguages/comments/ss3w6... Protobuf has the "runtime field presence" feature, which would be useful for making a backward compatible AST/ IR. That isn't done here, but it would be something to explore. ----- Also, thinking about the blog post a bit more, I'd say a huge number of decisions hinge on whether you can recompile the entire program or not. If you can, you should. But if you can't, a lot of programmers seem to have a hard time with runtime composition. Sometimes they even deny that it exists or should exist! In contrast, whereas admins/operators/SREs deal with runtime composition almost exclusively. Programmers are more familiar with a static view and build time composition. If I wanted to be ambitious, what I am getting at with this series is a theory of and guidelines for runtime composition, and versionless evolution. e.g. even in Windows, you never recompile your COM components together, even though they are written in statically typed languages. Global builds and atomic upgrades don't scale, even on a single computer.

by chubot 5y ago

skybrian 5y ago

Yes, I don’t agree with everything in the article, but it points out some sticky problems that can make things complicated. One example that’s related to programming languages is the evolution of an abstract syntax tree. We often expect new compilers to work with old code when a new kind of AST node is added, but to what extent should existing tools work with new code that has a node type that it doesn’t understand? Also, if your AST datatype is a library, does adding a new node type immediately cause compiler errors in every tool that uses the AST, or do those tools compile fine and break at runtime when given new code? Sometimes as a workaround, “metadata” gets put in specially formatted comments, as a way of saying that most language tools can ignore it. Alternately there can be general-purpose annotations (as Java has). Also, often an intermediate representation gets used as a “narrow waist” but they are tricky to design because “lowering” code will often discard high-level constructs.

chubot 5y ago
Yes, the AST problem is one I've had a lot! It's hard. (The whole Oil language is defined as a big ML-style statically typed data structure, but it's not exposed: https://www.oilshell.org/release/0.9.8/source-code.wwz/frontend/syntax.asdl https://www.oilshell.org/release/0.9.8/source-code.wwz/front...) What I'd say is that we already have a narrow waist for every language -- text. That is the one that's stable! Having an additional stable AST format seems to be an insurmountable problem, especially in most statically typed languages. The basic reason is that it's easier to evolve text in a backward compatible way than types. This is probably obvious to some people, but controversial to others... It has a lot to do with the "ignoring new data" problem discussed in the blog post. It would be nice to unpack this a bit and find some references. ---- Related: I recently circulated this language implementation, which has a front end in OCaml, a backend in C++, and the IR is transmitted via protobuf. https://old.reddit.com/r/ProgrammingLanguages/comments/ss3w6n/an_accessible_introduction_to_type_theory_and/ https://old.reddit.com/r/ProgrammingLanguages/comments/ss3w6... Protobuf has the "runtime field presence" feature, which would be useful for making a backward compatible AST/ IR. That isn't done here, but it would be something to explore. ----- Also, thinking about the blog post a bit more, I'd say a huge number of decisions hinge on whether you can recompile the entire program or not. If you can, you should. But if you can't, a lot of programmers seem to have a hard time with runtime composition. Sometimes they even deny that it exists or should exist! In contrast, whereas admins/operators/SREs deal with runtime composition almost exclusively. Programmers are more familiar with a static view and build time composition. If I wanted to be ambitious, what I am getting at with this series is a theory of and guidelines for runtime composition, and versionless evolution. e.g. even in Windows, you never recompile your COM components together, even though they are written in statically typed languages. Global builds and atomic upgrades don't scale, even on a single computer.