4 ms·

Peter Norvig's comment hits the nail on the head for me: " Peter Norvig 6 July 2016 at 11:47 I think the problem with Literate Programming is that assumes ther

Peter Norvig's comment hits the nail on the head for me: " Peter Norvig 6 July 2016 at 11:47 I think the problem with Literate Programming is that assumes there is a single best order of presentation of the explanation. I agree that the order imposed by the compiler is not always best, but different readers have different purposes. You don’t read documentation like a novel, cover to cover. You read the parts that you need for the task(s) you want to do now. What would be ideal is a tool to help construct such paths for each reader, just-in-time; not a tool that makes the author choose a single pth for all readers." Has anyone attempted something like this? I've heard you can do transclusion in org-mode which might be a starting point Edit: Some initial ideas: - Code can be deconstructed into blocks, and where the code is not self-documenting, prose can be added. Or even visualisations etc if a block is conceptually tricky to grok. You could even have MOOC-style validations to verify reader understanding for each block. - Some kind of topology of the blocks should be generated based on how they interact and how they are conceptually related - The author creates a few 'starting points' for different audiences, e.g. 'if you've used X before, start at Y' - From there next blocks to read are auto-suggested to the user. A map or network diagram of all blocks is also provided so the reader can chart their progress and see where the 'big ideas' lie. Edit 2: This also really reminds me of Bret Victor's 'Humane Representation of Thought' lecture (https://vimeo.com/115154289 https://vimeo.com/115154289, 45:40) where he says there is a conflict between code being an engineering specification and an authored work meant to be read by humans.

EdwardCoffin 10y ago
I don't understand why there not being a single best order of presentation for explanation is grounds to dismiss the whole endeavour. I bet that there are many pretty good orders for humans that are an improvement over the order the compiler wants the stuff in. Besides, the compiler doesn't demand a single best order, most programs could be re-ordered in a number of ways acceptable to the compiler (my point being that this criticism seems to hold literate programming to a standard that the incumbent of compiler-required ordering is not held to.) Edit: added the parenthetical comment
- deleted 10y ago
  [deleted]
- jarmitage 10y ago
  Yeah I agree. What I would imagine a tool like this would do is help to find and navigate the pretty good order(s) whether they have been hand crafted at first by the author or not. A comparison would be how desire paths form and are often turned into real paths. I like your parenthetical observation too!
  - EdwardCoffin 10y ago
    There was an interview with Guy Steele in the book Coders at Work, and in one bit he had something to say about actually referring to the (literate) source code for TeX [1]: Sometimes I've got a specific goal because I'm trying to solve a problem. There have been exactly two times, I think, that I was not able to fix a bug in my TeX macros by reading The TeXbook and it was necessary to go ahead and read TeX: The Program to find out exactly how a feature worked. In each case I was able to find my answer in 15 minutes because TeX: The Program is so well documented and cross-referenced. That, in itself, is an eye-opener--the fact that a program can be so organized and so documented, so indexed, that you can find something quickly. The other thing I learned from it is how a master programmer organizes data structures, how he organizes the code so as to make it easier to read. Knuth carefully laid out TeX: The Program so you could almost read it as a novel or something. You could read it in a linear pass. You'd probably want to do some jumping back and forth as you encountered various things. Of course, it was an enormous amount of work on his part, which is why very few programs have been done that way. [1] link to the page (337) in Google Books https://books.google.ca/books?id=2kMIqdfyT8kC&pg=PA337 https://books.google.ca/books?id=2kMIqdfyT8kC&pg=PA337
    - jarmitage 10y ago
      That's a great anecdote and makes we want to read TeX: The Program. I really think there's something in the 'jumping back and forth' of the reader that could be leveraged, like if there was a way of recording, distributing, aggregating, analysing it etc.
h8liu 10y ago
My hobby project, with dependency graph visualized: https://e8vm.io/e8vm https://e8vm.io/e8vm Source code: https://github.com/e8vm/e8vm https://github.com/e8vm/e8vm
- jarmitage 10y ago
  That is so cool, thanks for sharing. Is it possible to use the vis tool on other codebases?
  - h8liu 10y ago
    The layout engine is in this package: https://godoc.org/e8vm.io/e8vm/dagvis https://godoc.org/e8vm.io/e8vm/dagvis it is possible to layout any DAG (directed graph with no circles). The dependency graph of golang programs are generated by this package: https://godoc.org/e8vm.io/tools/godep https://godoc.org/e8vm.io/tools/godep I have not very well packaged the drawing part (sorry), but the essentials are here: https://github.com/e8vm/shanhu/blob/master/web/js/dag.js https://github.com/e8vm/shanhu/blob/master/web/js/dag.js it is using d3.
    - jarmitage 10y ago
      Super stuff, look forward to diving in deeper! Are you planning to extend this aspect of the project? What are your other plans for the project?
anonymousguy 10y ago
> Has anyone attempted something like this? Yes, I write all my personal projects in human order. To me human order is when the flow control of human consumption and computer execution are most closely aligned. I attempt to achieve this with depth and order. 1) Have a giant library that defines your library or application. 2) Inside the giant function nest child functions for the primary tasks in the order with which they will execute. In the case of the following example the options evaluation is first, followed by the lexer/parser is the second child task, followed by various code presentation tasks, and finally by an analysis task. http://prettydiff.com/lib/jspretty.js http://prettydiff.com/lib/jspretty.js 3) Break the child tasks down, as necessary, into reusable components. 4) I also believe a reference should always be declared before it used, which can dictate the order of reference declaration. My opinion is that when the flow control of the application is unclear you are wasting time during maintenance. You may not know where a problem is occurring, but if the flow of the application is immediately clear from reading the code you immediately know where to start, where to step to next, and where to stop. There is minimal guessing and it doesn't require breakpoints to figure it out.
akkartik 10y ago
I use a layer-based approach described at http://akkartik.name/post/wart-layers http://akkartik.name/post/wart-layers in https://github.com/akkartik/mu https://github.com/akkartik/mu. It's 20kLoC of literate code, and I was super concerned about optimizing it for non-linear reading, because that's how I need to browse the code for myself. It's not a system to generate multiple views like you want; instead, it's a single view that I find easy to skim and easy to reorganize at will: a) Each layer puts more important stuff up top. So you can start skimming from the first layer, but you don't have to read all the way down each layer, just get a sense of what it provides and why. b) There's not much emphasis on lengthy comments (which would get linear). Instead just the order in which code is presented does a lot of heavy lifting. c) The tangled code is intended to be readable (unlike Knuth's original tools), so I often jump down into it when I feel the need. Error messages and debugger support do show lines in the original literate sources, however.
- jarmitage 10y ago
  Fascinating, thanks for sharing. Tangential, but reading your website ("I'm working on ways to better convey the global structure of programs"), I've been talking a lot with a visually impaired researcher who works in Auditory Display. We've been talking about designing interactive audio displays to offer overviews of things like webpages and code. I'll definitely keep your work as a reference if that turns into a project!
  - akkartik 10y ago
    I'd love that! I couldn't find an email for you but if you want to ever get in touch my address is on my profile.

EdwardCoffin 10y ago

I don't understand why there not being a single best order of presentation for explanation is grounds to dismiss the whole endeavour. I bet that there are many pretty good orders for humans that are an improvement over the order the compiler wants the stuff in. Besides, the compiler doesn't demand a single best order, most programs could be re-ordered in a number of ways acceptable to the compiler (my point being that this criticism seems to hold literate programming to a standard that the incumbent of compiler-required ordering is not held to.) Edit: added the parenthetical comment

deleted 10y ago
[deleted]
jarmitage 10y ago
Yeah I agree. What I would imagine a tool like this would do is help to find and navigate the pretty good order(s) whether they have been hand crafted at first by the author or not. A comparison would be how desire paths form and are often turned into real paths. I like your parenthetical observation too!
- EdwardCoffin 10y ago
  There was an interview with Guy Steele in the book Coders at Work, and in one bit he had something to say about actually referring to the (literate) source code for TeX [1]: Sometimes I've got a specific goal because I'm trying to solve a problem. There have been exactly two times, I think, that I was not able to fix a bug in my TeX macros by reading The TeXbook and it was necessary to go ahead and read TeX: The Program to find out exactly how a feature worked. In each case I was able to find my answer in 15 minutes because TeX: The Program is so well documented and cross-referenced. That, in itself, is an eye-opener--the fact that a program can be so organized and so documented, so indexed, that you can find something quickly. The other thing I learned from it is how a master programmer organizes data structures, how he organizes the code so as to make it easier to read. Knuth carefully laid out TeX: The Program so you could almost read it as a novel or something. You could read it in a linear pass. You'd probably want to do some jumping back and forth as you encountered various things. Of course, it was an enormous amount of work on his part, which is why very few programs have been done that way. [1] link to the page (337) in Google Books https://books.google.ca/books?id=2kMIqdfyT8kC&pg=PA337 https://books.google.ca/books?id=2kMIqdfyT8kC&pg=PA337
  - jarmitage 10y ago
    That's a great anecdote and makes we want to read TeX: The Program. I really think there's something in the 'jumping back and forth' of the reader that could be leveraged, like if there was a way of recording, distributing, aggregating, analysing it etc.

h8liu 10y ago

My hobby project, with dependency graph visualized: https://e8vm.io/e8vm https://e8vm.io/e8vm Source code: https://github.com/e8vm/e8vm https://github.com/e8vm/e8vm

jarmitage 10y ago
That is so cool, thanks for sharing. Is it possible to use the vis tool on other codebases?
- h8liu 10y ago
  The layout engine is in this package: https://godoc.org/e8vm.io/e8vm/dagvis https://godoc.org/e8vm.io/e8vm/dagvis it is possible to layout any DAG (directed graph with no circles). The dependency graph of golang programs are generated by this package: https://godoc.org/e8vm.io/tools/godep https://godoc.org/e8vm.io/tools/godep I have not very well packaged the drawing part (sorry), but the essentials are here: https://github.com/e8vm/shanhu/blob/master/web/js/dag.js https://github.com/e8vm/shanhu/blob/master/web/js/dag.js it is using d3.
  - jarmitage 10y ago
    Super stuff, look forward to diving in deeper! Are you planning to extend this aspect of the project? What are your other plans for the project?

anonymousguy 10y ago

> Has anyone attempted something like this? Yes, I write all my personal projects in human order. To me human order is when the flow control of human consumption and computer execution are most closely aligned. I attempt to achieve this with depth and order. 1) Have a giant library that defines your library or application. 2) Inside the giant function nest child functions for the primary tasks in the order with which they will execute. In the case of the following example the options evaluation is first, followed by the lexer/parser is the second child task, followed by various code presentation tasks, and finally by an analysis task. http://prettydiff.com/lib/jspretty.js http://prettydiff.com/lib/jspretty.js 3) Break the child tasks down, as necessary, into reusable components. 4) I also believe a reference should always be declared before it used, which can dictate the order of reference declaration. My opinion is that when the flow control of the application is unclear you are wasting time during maintenance. You may not know where a problem is occurring, but if the flow of the application is immediately clear from reading the code you immediately know where to start, where to step to next, and where to stop. There is minimal guessing and it doesn't require breakpoints to figure it out.

akkartik 10y ago

I use a layer-based approach described at http://akkartik.name/post/wart-layers http://akkartik.name/post/wart-layers in https://github.com/akkartik/mu https://github.com/akkartik/mu. It's 20kLoC of literate code, and I was super concerned about optimizing it for non-linear reading, because that's how I need to browse the code for myself. It's not a system to generate multiple views like you want; instead, it's a single view that I find easy to skim and easy to reorganize at will: a) Each layer puts more important stuff up top. So you can start skimming from the first layer, but you don't have to read all the way down each layer, just get a sense of what it provides and why. b) There's not much emphasis on lengthy comments (which would get linear). Instead just the order in which code is presented does a lot of heavy lifting. c) The tangled code is intended to be readable (unlike Knuth's original tools), so I often jump down into it when I feel the need. Error messages and debugger support do show lines in the original literate sources, however.

jarmitage 10y ago
Fascinating, thanks for sharing. Tangential, but reading your website ("I'm working on ways to better convey the global structure of programs"), I've been talking a lot with a visually impaired researcher who works in Auditory Display. We've been talking about designing interactive audio displays to offer overviews of things like webpages and code. I'll definitely keep your work as a reference if that turns into a project!
- akkartik 10y ago
  I'd love that! I couldn't find an email for you but if you want to ever get in touch my address is on my profile.