5 ms·

I tried to do it seriously a number of times. Perhaps I am doing it wrong, but my productivity drops like a stone. Writing down your thoughts as you go and mai

I tried to do it seriously a number of times. Perhaps I am doing it wrong, but my productivity drops like a stone. Writing down your thoughts as you go and maintaining them takes serious time. The romantic notion of writing (and reading) code like a book is appealing, but writing books is hard and arduous, it cannot be underestimated as a craft on its own right, and there is coding to be done. There is also the question of structuring the literate code. Telling a story of how you are building it or explaining how it works has a very different flow and order to how code is usually structured for good maintainability. Please correct me if I’m wrong, because I would love to dive into it, but I don’t think there has ever been any major piece of software developed following literate programming (at least as Knuth envisioned it). I also don’t think there is any significant book that contains a sizeable working program embedded in it throughout, that can be compiled and executed as-is. In practice Knuth was most concerned with embedding short code snippets in his papers and books. Having the whole thing be an actual compilable program was secondary, and it was mostly short academic proof-of-concept prototypes and algorithms. Don’t get me wrong, I love the concept, that’s why I have given it multiple serious tries over the years, likely I will again, and why I think I have some insight of what happens when you use it for “real-world” work.

llm_trw 2y ago
NoWeb can support multi-thousand page documents which can compile to tens of thousands of lines of code. I used it at a deep tech startup I worked a number of years ago to document the theory behind why the code was doing what it was. Doubly useful since I could just use a regular bibtex citation system for papers which had done some part of what we were trying to do. My code became the defacto technical onboarding document, still in use today, despite the fact that none of the code in it has been updated since I left. For examples you can read: http://brokestream.com/tex-web.html http://brokestream.com/tex-web.html TeX is written entirely in WEB.
- oersted 2y ago
  Thanks for the insight. Just to be clear, writing documentation after the fact with lots of code snippets is obviously good and is standard practice. You can take the next step and ensure that the entirety of the code is in your documentation as snippets. This usually doesn’t make much sense, there’s lots of code that it is not worth explaining in a literate style. And what’s the point of the documentation containing the whole program if it is written after the fact and you already have a standard more maintainable codebase as the ground truth? The fact that your literate code didn’t get touched says a lot. To me the name Literate Programming implies that you write the code in a literate style from the outset. If you make it literate after the fact it is just normal documentation with snippets isn’t it?
  - llm_trw 2y ago
    >The fact that your literate code didn’t get touched says a lot. The fact it's still used 5 years later without any of the code still being in production says more. >To me the name Literate Programming implies that you write the code in a literate style from the outset. This seems like a fundamental misunderstanding of what it means to write. You should perhaps look at how people who do it for a living write books or articles. The final document has little to do with what you spend most your time editing. I absolutely hacked on the tangled source code of the program when trying to fix bugs or extend capabilities. Once I knew what I wanted to do I put it back in the literate program, usually finding a lot more bugs in the process.
    - oersted 2y ago
      Perhaps I came across as too critical, I have a lot of respect for what you did and for the craft of technical writing. And I definitely understand that writing is not done linearly and is very iterative. Correct me if I'm wrong, but it sounds like you were documenting existing code and that the result was very valuable as documentation, but not necessarily as code. You were acting as a technical writer not a programmer, it's a bit of a disconnect to call it Literate Programming, even if you were using Literate Programming tooling (NoWeb). This kind of documentation is common practice all over industry and it is valued, but I don't think Literate Programming is considered to be widely adopted because of that.
      - llm_trw 2y ago
        I'm having a hard time even understanding what the question is here. You seem to be confusing the tools with the work being done. You can write a prototype of a C function in Python to see if you understand the requirements before you commit to the much harder task of writing it in C. That doesn't mean you're not writing a C program. The same is true for literate programming. I can write code outside the main literate program when I'm not sure it's meant to do before I put it back in.
        oersted 2y ago
- oersted 2y ago
  It seems like the site got the HN hug-of-death. Mirror: https://web.archive.org/web/20221130150047/http://brokestream.com/tex.pdf https://web.archive.org/web/20221130150047/http://brokestrea... I have skimmed the TeX literate PDF (I did a number of times in the past too). Frankly isn’t it just like normal code with verbose comments? I have seen lots of code like that and it is not referred to as literate. The only difference is that this is a PDF, which makes it less practical and it is still not particularly readable as a book. It might have great book-like typography but not the "narrative" structure that helps you properly understand how it works without getting bogged down in details first. There's no coherent outline, no chapters or sections for major systems or design decisions, no overarching overviews, no relating different parts and giving context. There's also no story of how it was built or a log of his thoughts throughout problem-solving process, that would have been another good angle. Instead it's just the code from top to bottom with embedded very local commentary. The code itself is actually rather hard to parse visually by modern typographic standards. The issue is that probably I am misinterpreting what Knuth intended. The Literate Programming concept was a product of its time, and it has evolved into more practical modern documentation standards that are not so tightly linked to the code and don't exhaustively cover every line. The only problematic thing about it might just be the grandiose name Literate Programming, without that it's mainly good common-sense advice for quality documentation, but not necessarily a practical programming paradigm like the name implies.
  - llm_trw 2y ago
    Again, I'm having a hard time understanding what the issue is. It seems like you are deeply confused about what literate programming is and how it works. Have you read the original paper here in full: http://www.literateprogramming.com/knuthweb.pdf http://www.literateprogramming.com/knuthweb.pdf ? All of the navigation issues are taken care of by using <<chunks of code>> in a nested structure. You follow the numbers in those, like a follow your own adventure game, to find out whatever you need. The index has a listing of everything used in the program along with where it was defined and where it was used in case you want to find something specific. More modern tools, like NoWeb, turn all of this into hyperlinks so you can jump around the pretty printed version without having to loop up page numbers.
WillAdams 2y ago
I've found that Literate Programming suits how I think/approach projects, and it has worked for some quite large projects in the past for me. I've been maintaining a list of programs published as books and resources for Literate Programming at: https://www.goodreads.com/review/list/21394355-william-adams?ref=nav_mybooks&shelf=literateprograms https://www.goodreads.com/review/list/21394355-william-adams... esp. see: https://www.pbrt.org/ https://www.pbrt.org/ and https://mruckert.userweb.mwn.de/understanding_mp3/index.html https://mruckert.userweb.mwn.de/understanding_mp3/index.html My current project is: https://github.com/WillAdams/gcodepreview https://github.com/WillAdams/gcodepreview which uses a LaTeX package for this which I put together with a bit of help from tex.stackexchange --- the big advantage to it is that it allows editing the documentation/code with "normal" syntax highlighting, the disadvantage is that the .sty file has to be edited/updated to match the files which are being output and I still don't have a good setup for the readme.md I find having the typeset PDF w/ its hyperlinked ToC and marginalia and indices helps a lot in having a "nice" version which I can look through to remind myself of what was intended at a given point, and most importantly, to find _where_ that was written down. Working on a re-write now --- we'll see if this holds up for that.
- oersted 2y ago
  Awesome links, thank you. I did come across "Physically Based Rendering" at some point, I forgot about it. This is definitely an excellent example of Literate Programming.
jerf 2y ago
Literate programming, as originally described by Knuth, is a good essential idea embodied as a bunch of accidental instantiations of the idea that have gone badly out of date. Knuth's ideas at the time add a layer on top of programming languages to allow you to rearrange the code in a lot of ways that the languages at the time didn't support well or at all. It essentially adds an independent concept of "function", and adds on top of any ability the language had to have documentation its own documentation overlay on top. Problem is, in the meantime, languages got a lot better at functions, got more flexible in their organization, built in better capabilities for documentation and comments, and it all goes a different direction than the languages did. The result in the modern era is a rather bizarre multi-headed hydra of conflicting ideas about how things should be documented and tested. If someone wanted to resurrect the ideas, they need to not just try to get people to do what Knuth laid out decades ago super harder... they really need to sit down from the very beginning and work out how to update it in the modern era to be less redundant to what we already have. It could be as simple as taking modern doc strings and upgrading them a bit to allow highly-formatted comments to be embedded into code. Or, instead of trying to "weave" the code into a static book, allow the user to specify an entry point and then follow through everything that happens in the functions that it calls and turn that into a book, e.g., say "I'm going to enter this web framework through this path, tell me everything that happens". Or some other idea I don't have yet. Something that harmonizes with modern languages instead of fighting them.

llm_trw 2y ago

NoWeb can support multi-thousand page documents which can compile to tens of thousands of lines of code. I used it at a deep tech startup I worked a number of years ago to document the theory behind why the code was doing what it was. Doubly useful since I could just use a regular bibtex citation system for papers which had done some part of what we were trying to do. My code became the defacto technical onboarding document, still in use today, despite the fact that none of the code in it has been updated since I left. For examples you can read: http://brokestream.com/tex-web.html http://brokestream.com/tex-web.html TeX is written entirely in WEB.

oersted 2y ago
Thanks for the insight. Just to be clear, writing documentation after the fact with lots of code snippets is obviously good and is standard practice. You can take the next step and ensure that the entirety of the code is in your documentation as snippets. This usually doesn’t make much sense, there’s lots of code that it is not worth explaining in a literate style. And what’s the point of the documentation containing the whole program if it is written after the fact and you already have a standard more maintainable codebase as the ground truth? The fact that your literate code didn’t get touched says a lot. To me the name Literate Programming implies that you write the code in a literate style from the outset. If you make it literate after the fact it is just normal documentation with snippets isn’t it?
- llm_trw 2y ago
  >The fact that your literate code didn’t get touched says a lot. The fact it's still used 5 years later without any of the code still being in production says more. >To me the name Literate Programming implies that you write the code in a literate style from the outset. This seems like a fundamental misunderstanding of what it means to write. You should perhaps look at how people who do it for a living write books or articles. The final document has little to do with what you spend most your time editing. I absolutely hacked on the tangled source code of the program when trying to fix bugs or extend capabilities. Once I knew what I wanted to do I put it back in the literate program, usually finding a lot more bugs in the process.
  - oersted 2y ago
    Perhaps I came across as too critical, I have a lot of respect for what you did and for the craft of technical writing. And I definitely understand that writing is not done linearly and is very iterative. Correct me if I'm wrong, but it sounds like you were documenting existing code and that the result was very valuable as documentation, but not necessarily as code. You were acting as a technical writer not a programmer, it's a bit of a disconnect to call it Literate Programming, even if you were using Literate Programming tooling (NoWeb). This kind of documentation is common practice all over industry and it is valued, but I don't think Literate Programming is considered to be widely adopted because of that.
    - llm_trw 2y ago
      I'm having a hard time even understanding what the question is here. You seem to be confusing the tools with the work being done. You can write a prototype of a C function in Python to see if you understand the requirements before you commit to the much harder task of writing it in C. That doesn't mean you're not writing a C program. The same is true for literate programming. I can write code outside the main literate program when I'm not sure it's meant to do before I put it back in.
      - oersted 2y ago
oersted 2y ago
It seems like the site got the HN hug-of-death. Mirror: https://web.archive.org/web/20221130150047/http://brokestream.com/tex.pdf https://web.archive.org/web/20221130150047/http://brokestrea... I have skimmed the TeX literate PDF (I did a number of times in the past too). Frankly isn’t it just like normal code with verbose comments? I have seen lots of code like that and it is not referred to as literate. The only difference is that this is a PDF, which makes it less practical and it is still not particularly readable as a book. It might have great book-like typography but not the "narrative" structure that helps you properly understand how it works without getting bogged down in details first. There's no coherent outline, no chapters or sections for major systems or design decisions, no overarching overviews, no relating different parts and giving context. There's also no story of how it was built or a log of his thoughts throughout problem-solving process, that would have been another good angle. Instead it's just the code from top to bottom with embedded very local commentary. The code itself is actually rather hard to parse visually by modern typographic standards. The issue is that probably I am misinterpreting what Knuth intended. The Literate Programming concept was a product of its time, and it has evolved into more practical modern documentation standards that are not so tightly linked to the code and don't exhaustively cover every line. The only problematic thing about it might just be the grandiose name Literate Programming, without that it's mainly good common-sense advice for quality documentation, but not necessarily a practical programming paradigm like the name implies.
- llm_trw 2y ago
  Again, I'm having a hard time understanding what the issue is. It seems like you are deeply confused about what literate programming is and how it works. Have you read the original paper here in full: http://www.literateprogramming.com/knuthweb.pdf http://www.literateprogramming.com/knuthweb.pdf ? All of the navigation issues are taken care of by using <<chunks of code>> in a nested structure. You follow the numbers in those, like a follow your own adventure game, to find out whatever you need. The index has a listing of everything used in the program along with where it was defined and where it was used in case you want to find something specific. More modern tools, like NoWeb, turn all of this into hyperlinks so you can jump around the pretty printed version without having to loop up page numbers.

WillAdams 2y ago

I've found that Literate Programming suits how I think/approach projects, and it has worked for some quite large projects in the past for me. I've been maintaining a list of programs published as books and resources for Literate Programming at: https://www.goodreads.com/review/list/21394355-william-adams?ref=nav_mybooks&shelf=literateprograms https://www.goodreads.com/review/list/21394355-william-adams... esp. see: https://www.pbrt.org/ https://www.pbrt.org/ and https://mruckert.userweb.mwn.de/understanding_mp3/index.html https://mruckert.userweb.mwn.de/understanding_mp3/index.html My current project is: https://github.com/WillAdams/gcodepreview https://github.com/WillAdams/gcodepreview which uses a LaTeX package for this which I put together with a bit of help from tex.stackexchange --- the big advantage to it is that it allows editing the documentation/code with "normal" syntax highlighting, the disadvantage is that the .sty file has to be edited/updated to match the files which are being output and I still don't have a good setup for the readme.md I find having the typeset PDF w/ its hyperlinked ToC and marginalia and indices helps a lot in having a "nice" version which I can look through to remind myself of what was intended at a given point, and most importantly, to find _where_ that was written down. Working on a re-write now --- we'll see if this holds up for that.

oersted 2y ago
Awesome links, thank you. I did come across "Physically Based Rendering" at some point, I forgot about it. This is definitely an excellent example of Literate Programming.

jerf 2y ago

Literate programming, as originally described by Knuth, is a good essential idea embodied as a bunch of accidental instantiations of the idea that have gone badly out of date. Knuth's ideas at the time add a layer on top of programming languages to allow you to rearrange the code in a lot of ways that the languages at the time didn't support well or at all. It essentially adds an independent concept of "function", and adds on top of any ability the language had to have documentation its own documentation overlay on top. Problem is, in the meantime, languages got a lot better at functions, got more flexible in their organization, built in better capabilities for documentation and comments, and it all goes a different direction than the languages did. The result in the modern era is a rather bizarre multi-headed hydra of conflicting ideas about how things should be documented and tested. If someone wanted to resurrect the ideas, they need to not just try to get people to do what Knuth laid out decades ago super harder... they really need to sit down from the very beginning and work out how to update it in the modern era to be less redundant to what we already have. It could be as simple as taking modern doc strings and upgrading them a bit to allow highly-formatted comments to be embedded into code. Or, instead of trying to "weave" the code into a static book, allow the user to specify an entry point and then follow through everything that happens in the functions that it calls and turn that into a book, e.g., say "I'm going to enter this web framework through this path, tell me everything that happens". Or some other idea I don't have yet. Something that harmonizes with modern languages instead of fighting them.