7 ms·

Literate Programming: Articles

50 points by ddougj 3y ago

smcl 3y ago
That's ... a lot of articles. Is there any article in particular worth checking out?
- alex_muscar 3y ago
  Knuth's original article is a good starting point: http://literateprogramming.com/knuthweb.pdf http://literateprogramming.com/knuthweb.pdf. Knuth's Programming Pearls entries are good reading as well.
  - smcl 3y ago
    Yeah I have read the original Literate Programming text and played with tangle/weave a bit before realising jupyter notebooks were more suitable for what I was doing. I'm just wondering if there is something in particular the submitter wanted to highlight, or whether they just wanted to share the concept of "Literate Programming" in general
- everybodyknows 3y ago
  Clicking through the links I find this: > In general, literate programs combine source and documentation in a single file. Literate programming tools then parse the file to produce either readable documentation or compilable source. http://literateprogramming.com/lpfaq.pdf#page11 http://literateprogramming.com/lpfaq.pdf#page11 By this litmus test, the Go standard library qualifies as literate programming. Document production tools are go doc or godoc. The .go files are themselves already compilable source.
  - smcl 3y ago
    I don't think this is quite the same thing. It's nice that godoc autogenerates documentation on the functions + types each module in the stdlib provides (like here: https://pkg.go.dev/compress/bzip2@go1.20.4 https://pkg.go.dev/compress/bzip2@go1.20.4) but it doesn't look like it goes any further. In LP you'd get explanation of the implementation details interspersed with little chunks of code, it wouldn't necessarily be in the same order as the final file would be (just the order it makes sense to see it in). It might be worth checking through the PDF the sister comment to yours linked - http://literateprogramming.com/knuthweb.pdf http://literateprogramming.com/knuthweb.pdf - you'll see what I mean.
Davidnkt89 3y ago
[flagged]
- jmillikin 3y ago
  I wrote a literate programming preprocessor in Haskell[0], and used it to weave/tangle a D-Bus implementation (also in Haskell). The woven PDF[1] weighs in around 90 pages. I'll quote from the linked page regarding my conclusions: ------------>8---------- But in the end, I was never able to realize the promised benefits of Literate Programming. The typeset PDF was not easier to read than hypertext documentation generated by Haddock, and both were obviously worse than the very nice docs being created by the Python community with Sphinx. So in mid 2012 (~3 years after starting the project) I removed all the literate annotations and styles, dropped the fancy build scripts, and released haskell-dbus 0.10 as standard Haskell. ------------>8---------- I'm sure an expert in LaTeX could do a better job with the formatting, and with the benefit of 10 additional years of experience there are some design decisions I would do differently in the code, but even so I think it was a reasonably effortful attempt. I've never felt any urge to try Literate Programming again. [0] https://john-millikin.com/software/anansi#history https://john-millikin.com/software/anansi#history [1] https://github.com/jmillikin/haskell-dbus/releases/download/dbus-core_0.9.2.1/dbus-core_0.9.2.1.pdf https://github.com/jmillikin/haskell-dbus/releases/download/...
  - cnity 3y ago
    Cute fleurons.
  - smcl 3y ago
    I really hope that "--->8---" catches on here as a way to quote an extract from something nicely :)
    - yuchi 3y ago
      The actual size may change a lot depending on the font, but there’s a nice Unicode symbol for that: ——— <redacted> ——— Update: apparently HN doesn’t like some characters… Anyway these are the characters: https://unicode.org/charts/nameslist/n_2700.html https://unicode.org/charts/nameslist/n_2700.html - 2700 Black Safety Scissors - 2701 Upper Blade Scissors - 2702 Black Scissors - 2703 Lower Blade Scissors - 2704 White Scissors
pseudonymcoward 3y ago
I recently tried Literate Programming and I've found that it has some downsides not commonly discussed by its advocates. (It also has upsides which are valid, this post may come across as overly negative because I'm only covering negatives.) * It messes with tooling. If you're lucky then your editor will be smart enough to syntax highlight inside code blocks, or can be taught to do so easily. It's unlikely that more advanced IDE-style features will work. Obviously tools could be adapted with custom plugins but literate programming isn't popular enough for these to already exist as far as I'm aware. * It adds overhead to refactoring. Refactoring code now means editing and rewriting a book, doing a good job requires significantly more effort. If you have designed the program thoroughly ahead of time you can avoid this, but that's not generally the way most programmers work. * The "includes" problem. Most language require you to include/import/require packages used in a file. These are usually all placed at the start. This means a lot of chapters will start with "here are all the includes we'll need" if you're using a linear format. More complex formats that rearrange the code to generate outputs can do a better job but it's still a little clunky. Overall my conclusion was that literate programming works best if you write code like Donald Knuth: work solo, have a good idea of the entire design and a fixed scope which you can complete to declare both book and program "done forever". Unfortunately that doesn't cover most real world programming. I don't think it works well in the most common commercial/programming team style settings.
- bradrn 3y ago
  > Obviously tools could be adapted with custom plugins but literate programming isn't popular enough for these to already exist as far as I'm aware. I’ve heard that org-babel [https://orgmode.org/worg/org-contrib/babel/intro.html https://orgmode.org/worg/org-contrib/babel/intro.html] works pretty well for this, though I’ve never used it myself.
  - tmtvl 3y ago
    Org Babel is really nice, but it doesn't really solve the problem, rather it sidesteps it my giving you the option to edit a block of code in a dedicated buffer. I don't know how well that works with LSP (I don't use LSP) but it does allow you to make full use of SLIME.
mjw1007 3y ago
It might be nice if modern languages defined an official way to "flip" the interpretation of a source file, so that by default the content is treated as markdown and the code goes in fenced blocks (rather than defaulting to code and providing a way to mark comment blocks). Then we might hope for support in most tooling to appear quite quickly. I think one reason why literate programming is less attractive nowadays (compared to the days of Pascal) is that programming languages have got better at letting you arrange your code in whatever way you like. So maybe something like the above is enough to get 90% of the advantages of a full literate programming system.
- jmillikin 3y ago
  Every language with block comments supports this, by starting the file with a comment and wrapping code in end-comment/start-comment pairs. It's not especially useful. For literate programming you really need the ability to move blocks of code around without the ceremony of function signatures.
  - mjw1007 3y ago
    If I use block comments like that in most contemporary IDEs, I find that by default they show the main text in flat "this is a comment" styling, even though they're in principle capable of using different kinds of highlighting for different parts of the file. Similarly, if I preview such a file on github I get syntax highlighting for the code but I don't get rendering for the markdown. So I think the advantage of an 'official' way of saying you're using this arrangement would be communicating intent to more-or-less generic tools.
    - jmillikin 3y ago
      That's an IDE configuration problem. Most of them support mixed-mode syntax highlighting nowadays, so you just need to configure your theme to treat block comments as Markdown (or LaTeX, HTML, whatever). You could probably even do it with Markdown directly, using fenced code blocks as part of your begin/end lines. I think you'll find that this approach offers very little benefit over either traditional commentary or linear notebook-style inline code blocks.
rosebay 3y ago
[dead]
daly 3y ago
A literate computer algebra system https://en.wikipedia.org/wiki/Axiom_(computer_algebra_system) https://en.wikipedia.org/wiki/Axiom_(computer_algebra_system...
lincpa 3y ago
[dead]
musingsole 3y ago
I've been experimenting a lot with using Jupyter Notebooks (http://jupyter.org/ http://jupyter.org/) to get many features of literate programming. The tooling can be a bit clunky -- though JupyterLab and some plugins can go a long way. Importing notebooks into other notebooks or modules is still kind of weird even with the `ipynb` library. However, on the otherside of it all, you have a collection of documents that contain documentation, code, and tests, all side by side and storable with proof of past execution. I'll take this arrangement over someone's unit test and CI suite every day of the week.
crispyambulance 3y ago
I think there's some distinctions that need to be made. IMHO, Literate programming can be an umbrella for a bunch of different things. Is it "notebook" style programming, like Mathematica, matlab, Jupyter, and markdown-based stuff like R/python + Quarto? In this mode, I think the focus is more on creating a document or communicating stuff typically with data analysis. And actually, "literate" is a very apt adjective for something that's ultimately intended to be a document of some kind. Or is it more general intended for writing arbitrary software applications and having everything in the source code? In other words, to understand the source, you just read the source and there are no other documents. The latter mode, I think, is still very much not around as far as I know (academic proof-of-concepts for papers don't count :-) ). Will it ever really exist?
- taeric 3y ago
  I have tried a few blog posts with literate style code. Some in a notebook style, like https://taeric.github.io/cube-permutations-1.html https://taeric.github.io/cube-permutations-1.html. in general, that style seems easier nowadays. There is Stanford GraphBase, and a really good MP3 book that are full programs. Not to mention TeX.
  - mananaysiempre 3y ago
    Also the PCC compiler published A Retargetable C Compiler by Fraser & Hanson, as well as the PBRT renderer as described in the Academy Award winner Physically Based Rendering[1] by Pharr, Jacob, & Humphreys. As for instances where the point is more the book than the program, there’s C Interfaces and Implementations by Hanson, and while the “literate diffs” style used by Nystrom’s Crafting Interpreters[2] comes at the idea from a somewhat different angle, I’d say the effect is largely the same. Ulix[3] is arguably an academic exercise, but at the same time a full Unix-like kernel should count for something, shouldn’t it? The overwhelming popularity of TeX all too often makes people forget about METAFONT. The type of computer fonts advocated by the latter has unfortunately failed to gain popularity, and its descendant MetaPost has a lot of competition, but I think the language itself is much nicer (while remaining true to its macroexpander roots), and it’s another literate program of about the same size. [1] Gratis at https://pbr-book.org/ https://pbr-book.org/ [2] Gratis at http://craftinginterpreters.com/contents.html http://craftinginterpreters.com/contents.html [3] https://www.ulixos.org/ https://www.ulixos.org/
Vox_Leone 3y ago
I could not help but notice the conspicuous absence of notebooks. That is an excellent resource. Thanks very much for posting [and thanks to everyone involved in the original compilation].
mkovach 3y ago
Actually, I have been doing this for some stuff now. I wrote my copy of tangle (based on noweb's notangle). I'm starting to write markdown files to that a literate programs for Dockerfile images (as well as the support scripts used in the docker images and support scripts to build/test/etc.). Since all of this work is in a git repo, it is nice to pass around documents so folks understand the thinking behind things. Also, (and the main reason I went towards it) is that when I pull what I need from the markdown files, I have steps built around to lint, test, and do additional checks to ensure the required naming, labeling, etc. It has worked well and reduced many `silly' errors when other folks use the images. Is it for everybody? Nope. But it works here. I also did LP many years ago when I worked at a place that produces custom software for embedded systems. Providing them with a friggin' novel about the program was also fun. But, we never .. once .. were told we needed to provide more documentation. LP has good points, but it isn't a one-size-fits-all. When asked to send some coding samples, it is fun to have a PDF file that starts 'Chapter 3: And Now, Security! Encryption and Other Pain Points!'
- metaed 3y ago
  [dead]
mncharity 3y ago
FWIW, I note emacs polymode's[1] poly-markdown, and pandoc's ast json dumps[2]. For decades, I searched for literate tooling for personal projects, but nothing stuck. Something about the design space seems to make it easy to hit design choices that are uncomfortable fits for one's use case, or personal taste, or preferences for tooling extension and customization. It reminds me of the personal information manager space. My most recent foray seemed more promising. poly-markdown permits editing polylingual md files, and is permissive towards markdown dialects. So I could use a custom literate markdown, normalize it for rich-but-restrictive pandoc, then get the pandoc ast and emit custom html. With Julia's MLStyle[3] for pattern matching, and a bit of front-end pretty, it was unexpectedly easy to reach something comfortable. Perhaps for literate tooling to gain greater traction, it helps that software dev progress permits folks to more easily cook their own? [1] https://github.com/polymode/polymode https://github.com/polymode/polymode [2] https://pandoc.org/filters.html https://pandoc.org/filters.html [3] https://thautwarm.github.io/MLStyle.jl/latest/ https://thautwarm.github.io/MLStyle.jl/latest/
- ParetoOptimal 3y ago
  Do you mind sharing your comparison of polymode versus org-mode for literate programming?

50 points by ddougj 3y ago

smcl 3y ago

That's ... a lot of articles. Is there any article in particular worth checking out?

alex_muscar 3y ago
Knuth's original article is a good starting point: http://literateprogramming.com/knuthweb.pdf http://literateprogramming.com/knuthweb.pdf. Knuth's Programming Pearls entries are good reading as well.
- smcl 3y ago
  Yeah I have read the original Literate Programming text and played with tangle/weave a bit before realising jupyter notebooks were more suitable for what I was doing. I'm just wondering if there is something in particular the submitter wanted to highlight, or whether they just wanted to share the concept of "Literate Programming" in general
everybodyknows 3y ago
Clicking through the links I find this: > In general, literate programs combine source and documentation in a single file. Literate programming tools then parse the file to produce either readable documentation or compilable source. http://literateprogramming.com/lpfaq.pdf#page11 http://literateprogramming.com/lpfaq.pdf#page11 By this litmus test, the Go standard library qualifies as literate programming. Document production tools are go doc or godoc. The .go files are themselves already compilable source.
- smcl 3y ago
  I don't think this is quite the same thing. It's nice that godoc autogenerates documentation on the functions + types each module in the stdlib provides (like here: https://pkg.go.dev/compress/bzip2@go1.20.4 https://pkg.go.dev/compress/bzip2@go1.20.4) but it doesn't look like it goes any further. In LP you'd get explanation of the implementation details interspersed with little chunks of code, it wouldn't necessarily be in the same order as the final file would be (just the order it makes sense to see it in). It might be worth checking through the PDF the sister comment to yours linked - http://literateprogramming.com/knuthweb.pdf http://literateprogramming.com/knuthweb.pdf - you'll see what I mean.

Davidnkt89 3y ago

[flagged]

jmillikin 3y ago
I wrote a literate programming preprocessor in Haskell[0], and used it to weave/tangle a D-Bus implementation (also in Haskell). The woven PDF[1] weighs in around 90 pages. I'll quote from the linked page regarding my conclusions: ------------>8---------- But in the end, I was never able to realize the promised benefits of Literate Programming. The typeset PDF was not easier to read than hypertext documentation generated by Haddock, and both were obviously worse than the very nice docs being created by the Python community with Sphinx. So in mid 2012 (~3 years after starting the project) I removed all the literate annotations and styles, dropped the fancy build scripts, and released haskell-dbus 0.10 as standard Haskell. ------------>8---------- I'm sure an expert in LaTeX could do a better job with the formatting, and with the benefit of 10 additional years of experience there are some design decisions I would do differently in the code, but even so I think it was a reasonably effortful attempt. I've never felt any urge to try Literate Programming again. [0] https://john-millikin.com/software/anansi#history https://john-millikin.com/software/anansi#history [1] https://github.com/jmillikin/haskell-dbus/releases/download/dbus-core_0.9.2.1/dbus-core_0.9.2.1.pdf https://github.com/jmillikin/haskell-dbus/releases/download/...
- cnity 3y ago
  Cute fleurons.
- smcl 3y ago
  I really hope that "--->8---" catches on here as a way to quote an extract from something nicely :)
  - yuchi 3y ago
    The actual size may change a lot depending on the font, but there’s a nice Unicode symbol for that: ——— <redacted> ——— Update: apparently HN doesn’t like some characters… Anyway these are the characters: https://unicode.org/charts/nameslist/n_2700.html https://unicode.org/charts/nameslist/n_2700.html - 2700 Black Safety Scissors - 2701 Upper Blade Scissors - 2702 Black Scissors - 2703 Lower Blade Scissors - 2704 White Scissors

pseudonymcoward 3y ago

I recently tried Literate Programming and I've found that it has some downsides not commonly discussed by its advocates. (It also has upsides which are valid, this post may come across as overly negative because I'm only covering negatives.) * It messes with tooling. If you're lucky then your editor will be smart enough to syntax highlight inside code blocks, or can be taught to do so easily. It's unlikely that more advanced IDE-style features will work. Obviously tools could be adapted with custom plugins but literate programming isn't popular enough for these to already exist as far as I'm aware. * It adds overhead to refactoring. Refactoring code now means editing and rewriting a book, doing a good job requires significantly more effort. If you have designed the program thoroughly ahead of time you can avoid this, but that's not generally the way most programmers work. * The "includes" problem. Most language require you to include/import/require packages used in a file. These are usually all placed at the start. This means a lot of chapters will start with "here are all the includes we'll need" if you're using a linear format. More complex formats that rearrange the code to generate outputs can do a better job but it's still a little clunky. Overall my conclusion was that literate programming works best if you write code like Donald Knuth: work solo, have a good idea of the entire design and a fixed scope which you can complete to declare both book and program "done forever". Unfortunately that doesn't cover most real world programming. I don't think it works well in the most common commercial/programming team style settings.

bradrn 3y ago
> Obviously tools could be adapted with custom plugins but literate programming isn't popular enough for these to already exist as far as I'm aware. I’ve heard that org-babel [https://orgmode.org/worg/org-contrib/babel/intro.html https://orgmode.org/worg/org-contrib/babel/intro.html] works pretty well for this, though I’ve never used it myself.
- tmtvl 3y ago
  Org Babel is really nice, but it doesn't really solve the problem, rather it sidesteps it my giving you the option to edit a block of code in a dedicated buffer. I don't know how well that works with LSP (I don't use LSP) but it does allow you to make full use of SLIME.

mjw1007 3y ago

It might be nice if modern languages defined an official way to "flip" the interpretation of a source file, so that by default the content is treated as markdown and the code goes in fenced blocks (rather than defaulting to code and providing a way to mark comment blocks). Then we might hope for support in most tooling to appear quite quickly. I think one reason why literate programming is less attractive nowadays (compared to the days of Pascal) is that programming languages have got better at letting you arrange your code in whatever way you like. So maybe something like the above is enough to get 90% of the advantages of a full literate programming system.

jmillikin 3y ago
Every language with block comments supports this, by starting the file with a comment and wrapping code in end-comment/start-comment pairs. It's not especially useful. For literate programming you really need the ability to move blocks of code around without the ceremony of function signatures.
- mjw1007 3y ago
  If I use block comments like that in most contemporary IDEs, I find that by default they show the main text in flat "this is a comment" styling, even though they're in principle capable of using different kinds of highlighting for different parts of the file. Similarly, if I preview such a file on github I get syntax highlighting for the code but I don't get rendering for the markdown. So I think the advantage of an 'official' way of saying you're using this arrangement would be communicating intent to more-or-less generic tools.
  - jmillikin 3y ago
    That's an IDE configuration problem. Most of them support mixed-mode syntax highlighting nowadays, so you just need to configure your theme to treat block comments as Markdown (or LaTeX, HTML, whatever). You could probably even do it with Markdown directly, using fenced code blocks as part of your begin/end lines. I think you'll find that this approach offers very little benefit over either traditional commentary or linear notebook-style inline code blocks.

rosebay 3y ago

[dead]

daly 3y ago

A literate computer algebra system https://en.wikipedia.org/wiki/Axiom_(computer_algebra_system) https://en.wikipedia.org/wiki/Axiom_(computer_algebra_system...

lincpa 3y ago

[dead]

musingsole 3y ago

I've been experimenting a lot with using Jupyter Notebooks (http://jupyter.org/ http://jupyter.org/) to get many features of literate programming. The tooling can be a bit clunky -- though JupyterLab and some plugins can go a long way. Importing notebooks into other notebooks or modules is still kind of weird even with the `ipynb` library. However, on the otherside of it all, you have a collection of documents that contain documentation, code, and tests, all side by side and storable with proof of past execution. I'll take this arrangement over someone's unit test and CI suite every day of the week.

crispyambulance 3y ago

I think there's some distinctions that need to be made. IMHO, Literate programming can be an umbrella for a bunch of different things. Is it "notebook" style programming, like Mathematica, matlab, Jupyter, and markdown-based stuff like R/python + Quarto? In this mode, I think the focus is more on creating a document or communicating stuff typically with data analysis. And actually, "literate" is a very apt adjective for something that's ultimately intended to be a document of some kind. Or is it more general intended for writing arbitrary software applications and having everything in the source code? In other words, to understand the source, you just read the source and there are no other documents. The latter mode, I think, is still very much not around as far as I know (academic proof-of-concepts for papers don't count :-) ). Will it ever really exist?

taeric 3y ago
I have tried a few blog posts with literate style code. Some in a notebook style, like https://taeric.github.io/cube-permutations-1.html https://taeric.github.io/cube-permutations-1.html. in general, that style seems easier nowadays. There is Stanford GraphBase, and a really good MP3 book that are full programs. Not to mention TeX.
- mananaysiempre 3y ago
  Also the PCC compiler published A Retargetable C Compiler by Fraser & Hanson, as well as the PBRT renderer as described in the Academy Award winner Physically Based Rendering[1] by Pharr, Jacob, & Humphreys. As for instances where the point is more the book than the program, there’s C Interfaces and Implementations by Hanson, and while the “literate diffs” style used by Nystrom’s Crafting Interpreters[2] comes at the idea from a somewhat different angle, I’d say the effect is largely the same. Ulix[3] is arguably an academic exercise, but at the same time a full Unix-like kernel should count for something, shouldn’t it? The overwhelming popularity of TeX all too often makes people forget about METAFONT. The type of computer fonts advocated by the latter has unfortunately failed to gain popularity, and its descendant MetaPost has a lot of competition, but I think the language itself is much nicer (while remaining true to its macroexpander roots), and it’s another literate program of about the same size. [1] Gratis at https://pbr-book.org/ https://pbr-book.org/ [2] Gratis at http://craftinginterpreters.com/contents.html http://craftinginterpreters.com/contents.html [3] https://www.ulixos.org/ https://www.ulixos.org/

Vox_Leone 3y ago

I could not help but notice the conspicuous absence of notebooks. That is an excellent resource. Thanks very much for posting [and thanks to everyone involved in the original compilation].

mkovach 3y ago

Actually, I have been doing this for some stuff now. I wrote my copy of tangle (based on noweb's notangle). I'm starting to write markdown files to that a literate programs for Dockerfile images (as well as the support scripts used in the docker images and support scripts to build/test/etc.). Since all of this work is in a git repo, it is nice to pass around documents so folks understand the thinking behind things. Also, (and the main reason I went towards it) is that when I pull what I need from the markdown files, I have steps built around to lint, test, and do additional checks to ensure the required naming, labeling, etc. It has worked well and reduced many `silly' errors when other folks use the images. Is it for everybody? Nope. But it works here. I also did LP many years ago when I worked at a place that produces custom software for embedded systems. Providing them with a friggin' novel about the program was also fun. But, we never .. once .. were told we needed to provide more documentation. LP has good points, but it isn't a one-size-fits-all. When asked to send some coding samples, it is fun to have a PDF file that starts 'Chapter 3: And Now, Security! Encryption and Other Pain Points!'

metaed 3y ago
[dead]

mncharity 3y ago

FWIW, I note emacs polymode's[1] poly-markdown, and pandoc's ast json dumps[2]. For decades, I searched for literate tooling for personal projects, but nothing stuck. Something about the design space seems to make it easy to hit design choices that are uncomfortable fits for one's use case, or personal taste, or preferences for tooling extension and customization. It reminds me of the personal information manager space. My most recent foray seemed more promising. poly-markdown permits editing polylingual md files, and is permissive towards markdown dialects. So I could use a custom literate markdown, normalize it for rich-but-restrictive pandoc, then get the pandoc ast and emit custom html. With Julia's MLStyle[3] for pattern matching, and a bit of front-end pretty, it was unexpectedly easy to reach something comfortable. Perhaps for literate tooling to gain greater traction, it helps that software dev progress permits folks to more easily cook their own? [1] https://github.com/polymode/polymode https://github.com/polymode/polymode [2] https://pandoc.org/filters.html https://pandoc.org/filters.html [3] https://thautwarm.github.io/MLStyle.jl/latest/ https://thautwarm.github.io/MLStyle.jl/latest/

ParetoOptimal 3y ago
Do you mind sharing your comparison of polymode versus org-mode for literate programming?