4 ms·

Literate programs are essays. They might be easy to read and reason about, but not modify or extend. A computer program is not best understood and managed as a

Literate programs are essays. They might be easy to read and reason about, but not modify or extend. A computer program is not best understood and managed as a linear artifact. Much of it's power is in graph nature. Documentation comments are great, but that's not the same as literate programming.

gmfawcett 7y ago
I don't think a literate program is more or less linear than the source code that is extracted/tangled from it. Both artifacts have a sequence: for a C program, the tangled version would put the #includes before declarations before definitions, for example. In that sense, the LP program is an alternate linearization of the program, in that the authors can choose the order in which to introduce the program. But few LP programs are naively linear -- they typically impose a tree structure on the code, made up of labelled sections and subsections. Readers don't have to start at line 1 of the program/essay, they can navigate from the table of contents to the section of interest. A compelling argument for LP is that it's an additive technology. If you don't want to read the essay, that's fine -- just tangle the code, and read the source-code artifact instead. With the right tooling (which admittedly may not exist!) an IDE could let you edit the tangled version directly, and put your edits back into the "essay" at the right places, so round-trip editing would be feasible.
- samatman 7y ago
  Exactly. It's not fair to compare 40(!) years of advances in tooling surrounding the pile-of-files approach to software, to the somewhat withered on the vine approach embodied in literate programming. It's a road not taken, and I think that's a pity, so I'm doing something about it <shrug>
  - gmfawcett 7y ago
    I hope you'll post something about your progress. It's not an easy problem to tackle, but hopefully a fun one. :)
    - samatman 7y ago
      I hope to have a Show HN ready by summer!
- Jtsummers 7y ago
  I think I understand part of the problem. Many "literate programs" aren't literate in Knuth's sense. they are merely inversions of the conventional model. Where text is the default and code is the special case that has to be demarcated. Things like literate markdown I've seen which typically read like a regular program with extra text: # A Literate Program This is a literate program, the language is C. We'll begin with the includes because that's what C has at the start of every C file, and not because it makes any sense for the presentation: ``` #include <stdio.h> ... ``` Here are the declarations, you can ignore these for now. ``` int main(); double square(double x); ``` Now that that's out of the way, ... If that's all most people see then they haven't actually seen the benefit of LP. Where you can push that boilerplate stuff to an appendix so no one has to see it unless they're changing the libraries used by the system or some other thing that's important, but less essential to the understanding that LP tries to promote.
  - gmfawcett 7y ago
    That's an excellent point. I said "most literate programs aren't linear" in my comment... But I wasn't considering the low-effort linear style that many people actually use, so I'm probably wrong on that. :) "Low-effort linear literate" is a useful style, but I think it falls quite short of what Knuth had in mind.
    - samatman 7y ago
      I refer to this as semiliterate programming. I've found it to be a useful bootstrap toward a properly literate environment, which will require considerable tooling support to provide a reasonably modern experience. Happily, we have the Language Server Protocol now, so many of the key components are already in place...
- jimbokun 7y ago
  > they typically impose a tree structure on the code, made up of labelled sections and subsections. But most non-trivial programs don't have a tree structure, but are a full directed graph. Methods calling other methods, classes inheriting from other classes or implementing interfaces, etc. Programming and debugging could involve traversing and modifying this graph in almost any order, and does not lend itself to one preferred linearization.
  - gmfawcett 7y ago
    An LP style that somehow reflected the various graphs of a program (control flow, inheritance, etc.) might be very interesting! I'm not sure what it would look like, but it sounds like a starting point for experimentation. A big program could be broken into modules -- as we already do -- and each module (or its sub-modules) could be documented in a literate style, independently from the other modules. Maybe the graphs of the program (or at least, the graphs of its highly-connected modules) could be presented as alternate trails, indices, tables of contents, etc., each with its own accompanying narrative overview. It sounds gnarly, but not impossible! On the other hand, losing linearity altogether seems to be an anti-goal for an inherently narrative programming style like LP. If you're telling a story (about code, or anything else), eventually you have to put one sentence before the next. At some level of granularity, you have to commit to straight lines. When he wrote his book, it's clear that Knuth was very much aware that LP was an unusual, and possibly crazy, idea. The book is written in a humble style, like an invitation to explore a design space with him, and not as a prescriptive text. For example, I think the fact that he included McIlroy's full critique in the book speaks to his intentions. I guess my point is that Knuth would probably love that you're challenging his ideas and exploring the design space with him, rather than dismissing LP outright.
  - dragonwriter 7y ago
    > But most non-trivial programs don't have a tree structure, Pretty much all programs have at least one tree structure that covers the entire program (the AST), though that may or may not be the most interesting view of the program structure.
- nyberg 7y ago
  One such program that I know of and use in conjunction with pandoc is enTangleD[1] which supports editing on both at the cost of comments embedded in the source to keep track of blocks. All that's really required is folding of the comments to get half way to the desired IDE. [1]: https://entangled.github.io https://entangled.github.io
Jtsummers 7y ago
> Literate programs are essays. They might be easy to read and reason about, but not modify or extend. What's your rationale behind that last part? Have you ever used any notebook-styled interface like with Mathematica or Jupyter? It's perfectly feasible to tear out a chunk of material and replace it with something else, if you've organized it well. This is no different than the same constraint for conventionally written software. You can't easily refactor shitty code. You can't easily refactor shitty literate code. If you organize your code well and write quality code, whether in the literate or conventional style, you can refactor it with relative ease. > A computer program is not best understood and managed as a linear artifact. Much of it's power is in graph nature. And how does that conflict with literate programming? Literate programming permits the reorganization of code into any arbitrary structure. Which means that the graphical nature of the code can be made even more obvious than in most conventional languages. You're no longer bound by single-module files (see Java) or other arbitrary textual constraints. You can place the code in the place that makes the most sense for explication. Or put it adjacent to where it's used, even if it ends up tangled in a different file.

gmfawcett 7y ago

I don't think a literate program is more or less linear than the source code that is extracted/tangled from it. Both artifacts have a sequence: for a C program, the tangled version would put the #includes before declarations before definitions, for example. In that sense, the LP program is an alternate linearization of the program, in that the authors can choose the order in which to introduce the program. But few LP programs are naively linear -- they typically impose a tree structure on the code, made up of labelled sections and subsections. Readers don't have to start at line 1 of the program/essay, they can navigate from the table of contents to the section of interest. A compelling argument for LP is that it's an additive technology. If you don't want to read the essay, that's fine -- just tangle the code, and read the source-code artifact instead. With the right tooling (which admittedly may not exist!) an IDE could let you edit the tangled version directly, and put your edits back into the "essay" at the right places, so round-trip editing would be feasible.

samatman 7y ago
Exactly. It's not fair to compare 40(!) years of advances in tooling surrounding the pile-of-files approach to software, to the somewhat withered on the vine approach embodied in literate programming. It's a road not taken, and I think that's a pity, so I'm doing something about it <shrug>
- gmfawcett 7y ago
  I hope you'll post something about your progress. It's not an easy problem to tackle, but hopefully a fun one. :)
  - samatman 7y ago
    I hope to have a Show HN ready by summer!
Jtsummers 7y ago
I think I understand part of the problem. Many "literate programs" aren't literate in Knuth's sense. they are merely inversions of the conventional model. Where text is the default and code is the special case that has to be demarcated. Things like literate markdown I've seen which typically read like a regular program with extra text: # A Literate Program This is a literate program, the language is C. We'll begin with the includes because that's what C has at the start of every C file, and not because it makes any sense for the presentation: ``` #include <stdio.h> ... ``` Here are the declarations, you can ignore these for now. ``` int main(); double square(double x); ``` Now that that's out of the way, ... If that's all most people see then they haven't actually seen the benefit of LP. Where you can push that boilerplate stuff to an appendix so no one has to see it unless they're changing the libraries used by the system or some other thing that's important, but less essential to the understanding that LP tries to promote.
- gmfawcett 7y ago
  That's an excellent point. I said "most literate programs aren't linear" in my comment... But I wasn't considering the low-effort linear style that many people actually use, so I'm probably wrong on that. :) "Low-effort linear literate" is a useful style, but I think it falls quite short of what Knuth had in mind.
  - samatman 7y ago
    I refer to this as semiliterate programming. I've found it to be a useful bootstrap toward a properly literate environment, which will require considerable tooling support to provide a reasonably modern experience. Happily, we have the Language Server Protocol now, so many of the key components are already in place...
jimbokun 7y ago
> they typically impose a tree structure on the code, made up of labelled sections and subsections. But most non-trivial programs don't have a tree structure, but are a full directed graph. Methods calling other methods, classes inheriting from other classes or implementing interfaces, etc. Programming and debugging could involve traversing and modifying this graph in almost any order, and does not lend itself to one preferred linearization.
- gmfawcett 7y ago
  An LP style that somehow reflected the various graphs of a program (control flow, inheritance, etc.) might be very interesting! I'm not sure what it would look like, but it sounds like a starting point for experimentation. A big program could be broken into modules -- as we already do -- and each module (or its sub-modules) could be documented in a literate style, independently from the other modules. Maybe the graphs of the program (or at least, the graphs of its highly-connected modules) could be presented as alternate trails, indices, tables of contents, etc., each with its own accompanying narrative overview. It sounds gnarly, but not impossible! On the other hand, losing linearity altogether seems to be an anti-goal for an inherently narrative programming style like LP. If you're telling a story (about code, or anything else), eventually you have to put one sentence before the next. At some level of granularity, you have to commit to straight lines. When he wrote his book, it's clear that Knuth was very much aware that LP was an unusual, and possibly crazy, idea. The book is written in a humble style, like an invitation to explore a design space with him, and not as a prescriptive text. For example, I think the fact that he included McIlroy's full critique in the book speaks to his intentions. I guess my point is that Knuth would probably love that you're challenging his ideas and exploring the design space with him, rather than dismissing LP outright.
- dragonwriter 7y ago
  > But most non-trivial programs don't have a tree structure, Pretty much all programs have at least one tree structure that covers the entire program (the AST), though that may or may not be the most interesting view of the program structure.
nyberg 7y ago
One such program that I know of and use in conjunction with pandoc is enTangleD[1] which supports editing on both at the cost of comments embedded in the source to keep track of blocks. All that's really required is folding of the comments to get half way to the desired IDE. [1]: https://entangled.github.io https://entangled.github.io

Jtsummers 7y ago

> Literate programs are essays. They might be easy to read and reason about, but not modify or extend. What's your rationale behind that last part? Have you ever used any notebook-styled interface like with Mathematica or Jupyter? It's perfectly feasible to tear out a chunk of material and replace it with something else, if you've organized it well. This is no different than the same constraint for conventionally written software. You can't easily refactor shitty code. You can't easily refactor shitty literate code. If you organize your code well and write quality code, whether in the literate or conventional style, you can refactor it with relative ease. > A computer program is not best understood and managed as a linear artifact. Much of it's power is in graph nature. And how does that conflict with literate programming? Literate programming permits the reorganization of code into any arbitrary structure. Which means that the graphical nature of the code can be made even more obvious than in most conventional languages. You're no longer bound by single-module files (see Java) or other arbitrary textual constraints. You can place the code in the place that makes the most sense for explication. Or put it adjacent to where it's used, even if it ends up tangled in a different file.