7 ms·

Are there any large (> 5 people teams) projects written with literate programming? Also are there any IDE plugins or error stack trace/debuggers for literate p

Are there any large (> 5 people teams) projects written with literate programming? Also are there any IDE plugins or error stack trace/debuggers for literate programming? I haven't really paid attention to literate programming in a long long time and I'm curious if the field has advanced. (Also I don't understand this: "A typical literate file produces many source files." Why? Why would you care about having multiple source files? Isn't the literate file the source at this point?)

k__ 5y ago
"Also are there any IDE plugins or error stack trace/debuggers for literate programming?" Good question! I always think it would be nice to just write Markdown sprinkled with code, but without IDE/editor support, it's dead in the water :(
- Yaina 5y ago
  I mean rust supports markdown comments and can compile them into documentation.[1] That's pretty good in my book in terms of documentation. [1]https://doc.rust-lang.org/rust-by-example/meta/doc.html https://doc.rust-lang.org/rust-by-example/meta/doc.html
  - k__ 5y ago
    I wouldn't call it good. But better than nothing, I guess.
    - Yaina 5y ago
      I think it's as good as it gets. The rough principal is there: You have one file, and you can derive the documentation and the program from it with different compilers. The idea of reading code like a book seems extremely flawed to me, and any attempts to create such languages either look like regular source code with slightly different syntax or are barely comprehensible (e.g. TeX)
      - deleted 5y ago
        [deleted]
      - db48x 5y ago
        Have you actually read TeX: The Book? It is far from “extremely flawed”.
        wrycoder 5y ago
        Or another Knuth book, The Stanford Graph Base? Early C compilers, prior to prototypes, were rigid about the order of the various functions and includes, and that interfered with the exposition of the design. Literate programming was created by Knuth to address that.
        housel 5y ago
        TeX (and Metafont, and the original Tangle and Weave) were written in Pascal, which is much more rigid with respect to ordering than early C was. The CWEB used for the Stanford Graphbase came later. (Silvio Levy did the initial adaptation of Pascal WEB to C, and Knuth took it over when he switched to programming in C.)
        Yaina 5y ago
        Well, I have neither read through the entire 500 pages of tex.pdf[1] nor the 25k lines of tex.web[2], but I certainly have an impression of both. I think it is very hard to comprehend what one single part of a given algorithm does and it's imho almost impossible to get a good picture of how all these pieces fit together. [1] https://texdoc.org/serve/tex.pdf/0 https://texdoc.org/serve/tex.pdf/0 [2] https://mirror.las.iastate.edu/tex-archive/systems/knuth/dist/tex/tex.web https://mirror.las.iastate.edu/tex-archive/systems/knuth/dis...
        db48x 5y ago
        Ok, tex.pdf is the same as TeX: The Book, but obviously rendered into a PDF instead of a hardcover book. The chapters of the book each cover one aspect of the whole system. Sometimes it is a data type (such as boxes, hash tables, token lists, stacks, etc), other times it is functions that operate on them (creating/printing/copying/destroying boxes, for instance), other times it is an algorithm. Giving each chapter a coherent topic like this allows the human reader to comprehend them in isolation, without distractions from unrelated code. Modern languages have more facilities for high–level abstractions than Pascal did, so you could argue that this is no longer necessary, but I disagree. Chapter 38 Breaking paragraphs into lines is a good example; it is all about the algorithm for breaking paragraphs into lines that fit into whatever width is currently available. The first thing he does is give a very high–level overview of the purpose of this code (line breaking of paragraphs), what primitives it operates on (boxes and vertical lists), the source of the algorithm (a paper he coauthored a few years previously), plus the improvements made to that algorithm (less memory usage, less likely to encounter numeric overflow). He then discusses how this algorithm interacts with the rest of the system. It is called with one explicit argument, and relies on this and that global state. It makes certain changes to that global state. It also adds a single global variable not previously mentioned, and here is where we start to see the benefit of Web over plain Pascal. Pascal requires that all global variables be declared ahead of time, before any of the functions are declared and before the block of code that forms the body of the program. If we were reading the Pascal source code directly, we would see dozens or hundreds of these declarations before we even know what part of the program they are for, and before we even know what parts the program has. Instead, we see here that a single declaration is added to section 13, Global Variables. All the other globals are hidden from view, because they aren’t relevant to the task of line breaking a paragraph: 〈 Global variables 13 〉 +≡ just_box : pointer ; { the hlist node for the last line of the new paragraph } Next, we are given an outline of the whole line_break procedure, where we can see that it has four basic steps: 〈 Declare subprocedures for line break 826 〉 procedure line_break (final_widow_penalty : integer ); label done , done1 , done2 , done3 , done4 , done5 , continue ; var 〈 Local variables for line breaking 862 〉 begin pack_begin_line ← mode_line ; { this is for over/underfull box messages } 〈 Get ready to start line breaking 816 〉; 〈 Find optimal breakpoints 863 〉; 〈 Break the paragraph at the chosen breakpoints, justify the resulting lines to the correct widths, and append them to the current vertical list 876 〉; 〈 Clean up the memory by removing the break nodes 865 〉; pack_begin_line ← 0; end; The third of those steps is rather more complex than the others, which is perhaps a little unusual. Also, the second step was the subject of a whole paper. Regardless, this is a pretty good overview of the process: Identify the optimal breakpoints, then rebuild the data structures so that instead of one line of text that doesn’t fit we have a vertical list of lines of text that do fit. If we immediately saw a few hundred lines of code here, we would have no idea what any of them were doing, or why. Of course, modern programming languages are much more expressive than Pascal. If this program had been written in Rust, for example, then the code would have started out much more readable. The boxes would be defined in a crate of their own, so that no global definitions would need to be inserted into the main crate. The main crate would have a single `use` line for the box crate, and would probably refer to box–related types and functions using prefixed names like `box::short_display` or `box::fast_delete_glue_ref`. The `line_box` function would only be as long as this overview here, because all of the small tasks that it does would be packaged up into iterators and such. That third subtask in the overview might literally be `current_vlist = break_paragraph(para, breakpoints).iter().map(justify_line).collect()`. The overview would almost not be necessary; the code is approaching the expressiveness of the overview. I think that this hows that that the literate programming style that Knuth uses does not make it “very hard to comprehend what one single part of a given algorithm does”, nor is it “almost impossible to get a good picture of how all these pieces fit together”. I think it enables Knuth to explain everything in a much clearer way than he ever could have done with comments in the source code. I think that many programs could benefit from a similar treatment, even when they are written in better languages than Pascal.
- goosedragons 5y ago
  R-studio supports this for R although you can sprinkle in other languages although they are 2nd class citizens unsurprisingly. Emacs has org-mode with babel that combined with poly-mode gets you every feature you'd expect in the major mode of the language you're writing in. Emacs also supports R-markdown/Sweave quite well.
  - k__ 5y ago
    Ah, yes I played around with R-studio last year. Pretty nice experience.
- ParetoOptimal 5y ago
  We need a polished debugger using robust detangling and source maps for literate programming.
goosedragons 5y ago
You might still want multiple tangled files to follow a familiar app structure, or for languages that need to be compiled or if you don't want users to have to add the extra step of tangling the source before they build your program.
guitarbill 5y ago
I worked on a larger project where some of the code was "literate" programming. It was an absolute pain to modify anything. Debugging, not so much, since tangle produces raw source code. This you can work with. The problem is working with the original files. Syntax highlighting? Good luck! But possibly you could work around this, e.g. via custom highlighting syntax. Same with any auto-complete, contextual IDE help, etc. Refactoring was painful. Also, the text absolutely destroys being able to scan and reason about the control flow quickly. Especially bad when a dev decides something needs "a lot of documentation" and writes a small novel. Needless to say, it was truly awful.
flukus 5y ago
IMHO the two biggest problems I find with existing tools are that they assume the documentation is the source of truth and that the tangling will leave artifacts in code. Both of these make them poorly suited to the sort of projects that most of us work on, I'm not Knuth writing a dead tree tome but I would like some better ways to add documentation to existing projects and have to integrate with others. I wrote a PoC of a tangling tool that worked through "virtual" files and had a syntax aware handler, there's enough information that it could possible work with language servers too, but sadly I haven't had time to take things further: https://gitlab.com/lusher/tanglemd https://gitlab.com/lusher/tanglemd
- ParetoOptimal 5y ago
  > they assume the documentation is the source of truth and that the tangling will leave artifacts in code. Leo avoids this by keeping a "shadow copy" of the artifacts or annotations and doing a diff on detangling between all 3 versions.

k__ 5y ago

"Also are there any IDE plugins or error stack trace/debuggers for literate programming?" Good question! I always think it would be nice to just write Markdown sprinkled with code, but without IDE/editor support, it's dead in the water :(

Yaina 5y ago
I mean rust supports markdown comments and can compile them into documentation.[1] That's pretty good in my book in terms of documentation. [1]https://doc.rust-lang.org/rust-by-example/meta/doc.html https://doc.rust-lang.org/rust-by-example/meta/doc.html
- k__ 5y ago
  I wouldn't call it good. But better than nothing, I guess.
  - Yaina 5y ago
    I think it's as good as it gets. The rough principal is there: You have one file, and you can derive the documentation and the program from it with different compilers. The idea of reading code like a book seems extremely flawed to me, and any attempts to create such languages either look like regular source code with slightly different syntax or are barely comprehensible (e.g. TeX)
    - deleted 5y ago
      [deleted]
    - db48x 5y ago
      Have you actually read TeX: The Book? It is far from “extremely flawed”.
      - wrycoder 5y ago
        Or another Knuth book, The Stanford Graph Base? Early C compilers, prior to prototypes, were rigid about the order of the various functions and includes, and that interfered with the exposition of the design. Literate programming was created by Knuth to address that.
        housel 5y ago
        TeX (and Metafont, and the original Tangle and Weave) were written in Pascal, which is much more rigid with respect to ordering than early C was. The CWEB used for the Stanford Graphbase came later. (Silvio Levy did the initial adaptation of Pascal WEB to C, and Knuth took it over when he switched to programming in C.)
      - Yaina 5y ago
        Well, I have neither read through the entire 500 pages of tex.pdf[1] nor the 25k lines of tex.web[2], but I certainly have an impression of both. I think it is very hard to comprehend what one single part of a given algorithm does and it's imho almost impossible to get a good picture of how all these pieces fit together. [1] https://texdoc.org/serve/tex.pdf/0 https://texdoc.org/serve/tex.pdf/0 [2] https://mirror.las.iastate.edu/tex-archive/systems/knuth/dist/tex/tex.web https://mirror.las.iastate.edu/tex-archive/systems/knuth/dis...
        db48x 5y ago
        Ok, tex.pdf is the same as TeX: The Book, but obviously rendered into a PDF instead of a hardcover book. The chapters of the book each cover one aspect of the whole system. Sometimes it is a data type (such as boxes, hash tables, token lists, stacks, etc), other times it is functions that operate on them (creating/printing/copying/destroying boxes, for instance), other times it is an algorithm. Giving each chapter a coherent topic like this allows the human reader to comprehend them in isolation, without distractions from unrelated code. Modern languages have more facilities for high–level abstractions than Pascal did, so you could argue that this is no longer necessary, but I disagree. Chapter 38 Breaking paragraphs into lines is a good example; it is all about the algorithm for breaking paragraphs into lines that fit into whatever width is currently available. The first thing he does is give a very high–level overview of the purpose of this code (line breaking of paragraphs), what primitives it operates on (boxes and vertical lists), the source of the algorithm (a paper he coauthored a few years previously), plus the improvements made to that algorithm (less memory usage, less likely to encounter numeric overflow). He then discusses how this algorithm interacts with the rest of the system. It is called with one explicit argument, and relies on this and that global state. It makes certain changes to that global state. It also adds a single global variable not previously mentioned, and here is where we start to see the benefit of Web over plain Pascal. Pascal requires that all global variables be declared ahead of time, before any of the functions are declared and before the block of code that forms the body of the program. If we were reading the Pascal source code directly, we would see dozens or hundreds of these declarations before we even know what part of the program they are for, and before we even know what parts the program has. Instead, we see here that a single declaration is added to section 13, Global Variables. All the other globals are hidden from view, because they aren’t relevant to the task of line breaking a paragraph: 〈 Global variables 13 〉 +≡ just_box : pointer ; { the hlist node for the last line of the new paragraph } Next, we are given an outline of the whole line_break procedure, where we can see that it has four basic steps: 〈 Declare subprocedures for line break 826 〉 procedure line_break (final_widow_penalty : integer ); label done , done1 , done2 , done3 , done4 , done5 , continue ; var 〈 Local variables for line breaking 862 〉 begin pack_begin_line ← mode_line ; { this is for over/underfull box messages } 〈 Get ready to start line breaking 816 〉; 〈 Find optimal breakpoints 863 〉; 〈 Break the paragraph at the chosen breakpoints, justify the resulting lines to the correct widths, and append them to the current vertical list 876 〉; 〈 Clean up the memory by removing the break nodes 865 〉; pack_begin_line ← 0; end; The third of those steps is rather more complex than the others, which is perhaps a little unusual. Also, the second step was the subject of a whole paper. Regardless, this is a pretty good overview of the process: Identify the optimal breakpoints, then rebuild the data structures so that instead of one line of text that doesn’t fit we have a vertical list of lines of text that do fit. If we immediately saw a few hundred lines of code here, we would have no idea what any of them were doing, or why. Of course, modern programming languages are much more expressive than Pascal. If this program had been written in Rust, for example, then the code would have started out much more readable. The boxes would be defined in a crate of their own, so that no global definitions would need to be inserted into the main crate. The main crate would have a single `use` line for the box crate, and would probably refer to box–related types and functions using prefixed names like `box::short_display` or `box::fast_delete_glue_ref`. The `line_box` function would only be as long as this overview here, because all of the small tasks that it does would be packaged up into iterators and such. That third subtask in the overview might literally be `current_vlist = break_paragraph(para, breakpoints).iter().map(justify_line).collect()`. The overview would almost not be necessary; the code is approaching the expressiveness of the overview. I think that this hows that that the literate programming style that Knuth uses does not make it “very hard to comprehend what one single part of a given algorithm does”, nor is it “almost impossible to get a good picture of how all these pieces fit together”. I think it enables Knuth to explain everything in a much clearer way than he ever could have done with comments in the source code. I think that many programs could benefit from a similar treatment, even when they are written in better languages than Pascal.
goosedragons 5y ago
R-studio supports this for R although you can sprinkle in other languages although they are 2nd class citizens unsurprisingly. Emacs has org-mode with babel that combined with poly-mode gets you every feature you'd expect in the major mode of the language you're writing in. Emacs also supports R-markdown/Sweave quite well.
- k__ 5y ago
  Ah, yes I played around with R-studio last year. Pretty nice experience.
ParetoOptimal 5y ago
We need a polished debugger using robust detangling and source maps for literate programming.

goosedragons 5y ago

You might still want multiple tangled files to follow a familiar app structure, or for languages that need to be compiled or if you don't want users to have to add the extra step of tangling the source before they build your program.

guitarbill 5y ago

I worked on a larger project where some of the code was "literate" programming. It was an absolute pain to modify anything. Debugging, not so much, since tangle produces raw source code. This you can work with. The problem is working with the original files. Syntax highlighting? Good luck! But possibly you could work around this, e.g. via custom highlighting syntax. Same with any auto-complete, contextual IDE help, etc. Refactoring was painful. Also, the text absolutely destroys being able to scan and reason about the control flow quickly. Especially bad when a dev decides something needs "a lot of documentation" and writes a small novel. Needless to say, it was truly awful.

flukus 5y ago

IMHO the two biggest problems I find with existing tools are that they assume the documentation is the source of truth and that the tangling will leave artifacts in code. Both of these make them poorly suited to the sort of projects that most of us work on, I'm not Knuth writing a dead tree tome but I would like some better ways to add documentation to existing projects and have to integrate with others. I wrote a PoC of a tangling tool that worked through "virtual" files and had a syntax aware handler, there's enough information that it could possible work with language servers too, but sadly I haven't had time to take things further: https://gitlab.com/lusher/tanglemd https://gitlab.com/lusher/tanglemd

ParetoOptimal 5y ago
> they assume the documentation is the source of truth and that the tangling will leave artifacts in code. Leo avoids this by keeping a "shadow copy" of the artifacts or annotations and doing a diff on detangling between all 3 versions.