6 ms·

Igneous Linearizer: semi-structured source code

75 points by seagreen 2y ago

zacgarby 2y ago
this is great! i’ve been thinking about exactly this (though styled after Logseq rather than Obsidian) but not gotten as far as implementing anything. that being said, the thing i haven’t been able to convince myself of yet is why these are different to just normal (in-line) functions? as in, why should i have to write [[foo]]: would it not be better to have all identifiers automatically linked?
- fwip 2y ago
  As a language-agnostic thing, I suppose you need to prevent the machinery from pulling in keywords and variable names accidentally (or the insides of strings or comments). I like the idea (also a fan of Unison's approach to code-in-the-db), but I worry about the potential issues that come from effectively having a single global namespace. Could be that I just don't have the discipline for it, though.
  - seagreen 2y ago
    > As a language-agnostic thing, I suppose you need to prevent the machinery from pulling in keywords and variable names accidentally (or the insides of strings or comments). Exactly. But zacgarby's right that you would want some auto-linking, so this is where language-specific plugins come in. The difference from today's world would be that those plugins would leave their results explicitly serialized in the source medium, so they wouldn't have to keep being reconstructed by every other tool. > I like the idea (also a fan of Unison's approach to code-in-the-db), but I worry about the potential issues that come from effectively having a single global namespace. Could be that I just don't have the discipline for it, though. I have lots of thoughts on this. I was initially disappointed that Unison kept a unique hierarchy to organize their code-- that seems so filesystem-ey and 1990s. However, I'm now a convert. The result of combining a unique hierarchy with explicit links between nodes is a 'compound graph' (or a 'cluster graph', depending, getting the language from https://rtsys.informatik.uni-kiel.de/~biblio/downloads/theses/ima-dt.pdf https://rtsys.informatik.uni-kiel.de/~biblio/downloads/these...). These are very respectable data structures! One thing they're good for is being able to always give a canonical title to a node, but varying what that title is depending on the situation. I think that for serious work the linearizer would want to copy this strategy as well. Right now it's flat because that's all I need for my website, but if you were doing big projects in it you'd want to follow Unison and have a hierarchy. In the `HashMap` folder you'd display `HashMap.get` with a link alias that shows plain `get`, but if that function is being called from some other folder it would appear as the full `HashMap.get`. You could still do all the other cool stuff like organize by tags and attributes using frontmatter, but for the particular purpose of display names having a global hierarchy is useful. EDIT: What matters more than what the linearizer does is what Obsidian displays, so it's there that the "take relative hierarchical position into account when showing links" logic would have to occur. That could be a plugin or maybe Obsidian's relative link feature, I haven't used the latter.
- seagreen 2y ago
  > this is great! i’ve been thinking about exactly this (though styled after Logseq rather than Obsidian) but not gotten as far as implementing anything. Thank you! I think [[links]] will work out of the box with Logseq since they're the same as Obsidian. Transclusions will be in the wrong format since Obsidian transclusions look like `![[this]]`, but it would be quick to modify the linearizer to handle them. You may not want transclusions though since transcluding code into other code is... very weird. I'm curious what use cases people come up with for it though.
arnsholt 2y ago
This is neat, but it does seem like a lot of work to get part of the way to what a Smalltalk already gives you.
- skulk 2y ago
  or emacs/org-mode
  - deleted 2y ago
    [deleted]
- seagreen 2y ago
  I love Smalltalk, and have done a reasonable amount of messing around with Cuis (which is awesome and everyone should try it). However this gives you two things that Smalltalk doesn't: 1. It's language agnostic (boring I know) 2. It promotes keeping your code and written texts in the same system where they're both first class. That way they can link between each other, transclude each other, be published together, be organized the same way, etc. I really think this is the most interesting thing about the project, it really feels important to me. Caveat: right now my written documents can link to/transclude code, but it doesn't work the other way yet. This is because the linearizer will see a link from code to documents as another definition and try to jam it in the source file. This would be an interesting use case for typed links, but Obsidian doesn't a have them AFAIK. Kind of cool since I haven't seen many other use cases for typed links in the wild. EDIT: It occurs to me that I've never used a Smalltalk notetaking or word processing program. Are there any that are integrated with the System Browser, so that they can link to (or even better embed) code? If anyone has more info please let me know!
  - couchbed 2y ago
    Lepiter is a Pharo-based notetaking app within the Glamorous Toolkit. I'm not sure it's mature enough to compete with Obsidian/etc., but it does allow linked and embedded code like you were thinking. https://lepiter.io/feenk/introducing-lepiter--knowledge-management--e2p6apqsz5npq7m4xte0kkywn/ https://lepiter.io/feenk/introducing-lepiter--knowledge-mana...
JonChesterfield 2y ago
This is an interesting direction. One thought is that obsidian can execute web assembly and a parser / sema checker written in something that turns into wasm can therefore be run on the source files. Can probably tie that to a syntax highlighter style thing for in-ide feedback. The other is that markdown is a tempting format for literate programming. I do have some notes in obsidian that are fed to cmark to product html. With some conventions, splitting a literate program into executable code embedded in a html document is probably doable as an XML pipeline. In a much simpler vein, I'm experimenting with machine configuration from within obsidian. The local DNS server sets itself up using a markdown file so editing an IP or adding a new machine can be done by changing that markdown. I hope the author continues down this path and writes more about the experience.
- nohat 2y ago
  Yeah, I definitely see using this for literate programming. Not quite sure the best way to organize it. Maybe use a static site compiler to auto host documentation version.
  - seagreen 2y ago
    I wish you could just use Obsidian Publish to host sites, but due to the indentation issue you have to control the rendering, which is a bummer. Obsidian Digital Garden[1] is FOSS, so it might be modifiable parse and output the code pages correctly. [1] https://github.com/oleeskild/obsidian-digital-garden https://github.com/oleeskild/obsidian-digital-garden
    - WorldMaker 2y ago
      The typical Markdown answer to needing indentation preserved is the "code fence" (triple backquotes ```), though I imagine the problem with that is that Obsidian by default stops dealing with Wikilinks inside fenced code. I don't know Obsidian that well, but maybe there's a way to use a code fence and have it support Wikilinks inside? A different direction to explore might be to explore proportional font coding techniques that rely less on whitespace. Lisp can be a good language to play with those ideas given whitespace isn't syntactic. Though idiomatic Lisp has certainly relied on semantic whitespace in coding styles for a very long time.
nbbaier 2y ago
> The solution I've been waiting for is source-code-in-the-database. I'm cheering on multiple projects attempting this. What are the projects you're especially bullish on?
- DannyBee 2y ago
  VisualAge C++, Java, and Smalltalk.
- seagreen 2y ago
  Hazel and Unison are two of the big ones. I'm friends with some of the Unison folks so I'm biased, but I really like how few features there are in the language. In general I'm just a huge sucker for subtractive improvement: if you can have a small number of awesome things (eg abilities) instead of a bunch of special case things (exception handling, monad trickery, dependency injection machinery) sign me up. I know less about Hazel, my understanding is that it's source-code-in-CRDTs, which is definitely structured source code though may not technically be in a database.
  - thyrsus 2y ago
    Is this the unison you mean? https://www.unison-lang.org/ https://www.unison-lang.org/ Unrelated: what has your experience been using igneous-linearizer to help understand other people's code?
    - seagreen 2y ago
      That's it. And the linearizer is only one way-- you write text with [[links]] and turn that into plaintext.
      - thyrsus 2y ago
        You may have encountered leo [ https://leo-editor.github.io/leo-editor/ https://leo-editor.github.io/leo-editor/ ]. I use it to pull in or write code and break it up into comprehensible pieces. It works, but it feels overcomplex. I'm open to something simpler, and will give this a try.
binary132 2y ago
I really appreciate the unordinary direction you went with these articles and your site in general. An enjoyable read!
- seagreen 2y ago
  All credit to Conor White-Sullivan for making links cool again. > unordinary direction Hahaha, this is a very polite way to put that=) But I do appreciate it.
  - binary132 2y ago
    Don’t take it as a backhanded compliment! We should all strive to surpass and excel the ordinary.

75 points by seagreen 2y ago

this is great! i’ve been thinking about exactly this (though styled after Logseq rather than Obsidian) but not gotten as far as implementing anything. that being said, the thing i haven’t been able to convince myself of yet is why these are different to just normal (in-line) functions? as in, why should i have to write [[foo]]: would it not be better to have all identifiers automatically linked?

fwip 2y ago
As a language-agnostic thing, I suppose you need to prevent the machinery from pulling in keywords and variable names accidentally (or the insides of strings or comments). I like the idea (also a fan of Unison's approach to code-in-the-db), but I worry about the potential issues that come from effectively having a single global namespace. Could be that I just don't have the discipline for it, though.
- seagreen 2y ago
  > As a language-agnostic thing, I suppose you need to prevent the machinery from pulling in keywords and variable names accidentally (or the insides of strings or comments). Exactly. But zacgarby's right that you would want some auto-linking, so this is where language-specific plugins come in. The difference from today's world would be that those plugins would leave their results explicitly serialized in the source medium, so they wouldn't have to keep being reconstructed by every other tool. > I like the idea (also a fan of Unison's approach to code-in-the-db), but I worry about the potential issues that come from effectively having a single global namespace. Could be that I just don't have the discipline for it, though. I have lots of thoughts on this. I was initially disappointed that Unison kept a unique hierarchy to organize their code-- that seems so filesystem-ey and 1990s. However, I'm now a convert. The result of combining a unique hierarchy with explicit links between nodes is a 'compound graph' (or a 'cluster graph', depending, getting the language from https://rtsys.informatik.uni-kiel.de/~biblio/downloads/theses/ima-dt.pdf https://rtsys.informatik.uni-kiel.de/~biblio/downloads/these...). These are very respectable data structures! One thing they're good for is being able to always give a canonical title to a node, but varying what that title is depending on the situation. I think that for serious work the linearizer would want to copy this strategy as well. Right now it's flat because that's all I need for my website, but if you were doing big projects in it you'd want to follow Unison and have a hierarchy. In the `HashMap` folder you'd display `HashMap.get` with a link alias that shows plain `get`, but if that function is being called from some other folder it would appear as the full `HashMap.get`. You could still do all the other cool stuff like organize by tags and attributes using frontmatter, but for the particular purpose of display names having a global hierarchy is useful. EDIT: What matters more than what the linearizer does is what Obsidian displays, so it's there that the "take relative hierarchical position into account when showing links" logic would have to occur. That could be a plugin or maybe Obsidian's relative link feature, I haven't used the latter.
seagreen 2y ago
> this is great! i’ve been thinking about exactly this (though styled after Logseq rather than Obsidian) but not gotten as far as implementing anything. Thank you! I think [[links]] will work out of the box with Logseq since they're the same as Obsidian. Transclusions will be in the wrong format since Obsidian transclusions look like `![[this]]`, but it would be quick to modify the linearizer to handle them. You may not want transclusions though since transcluding code into other code is... very weird. I'm curious what use cases people come up with for it though.

arnsholt 2y ago

This is neat, but it does seem like a lot of work to get part of the way to what a Smalltalk already gives you.

skulk 2y ago
or emacs/org-mode
- deleted 2y ago
  [deleted]
seagreen 2y ago
I love Smalltalk, and have done a reasonable amount of messing around with Cuis (which is awesome and everyone should try it). However this gives you two things that Smalltalk doesn't: 1. It's language agnostic (boring I know) 2. It promotes keeping your code and written texts in the same system where they're both first class. That way they can link between each other, transclude each other, be published together, be organized the same way, etc. I really think this is the most interesting thing about the project, it really feels important to me. Caveat: right now my written documents can link to/transclude code, but it doesn't work the other way yet. This is because the linearizer will see a link from code to documents as another definition and try to jam it in the source file. This would be an interesting use case for typed links, but Obsidian doesn't a have them AFAIK. Kind of cool since I haven't seen many other use cases for typed links in the wild. EDIT: It occurs to me that I've never used a Smalltalk notetaking or word processing program. Are there any that are integrated with the System Browser, so that they can link to (or even better embed) code? If anyone has more info please let me know!
- couchbed 2y ago
  Lepiter is a Pharo-based notetaking app within the Glamorous Toolkit. I'm not sure it's mature enough to compete with Obsidian/etc., but it does allow linked and embedded code like you were thinking. https://lepiter.io/feenk/introducing-lepiter--knowledge-management--e2p6apqsz5npq7m4xte0kkywn/ https://lepiter.io/feenk/introducing-lepiter--knowledge-mana...

JonChesterfield 2y ago

This is an interesting direction. One thought is that obsidian can execute web assembly and a parser / sema checker written in something that turns into wasm can therefore be run on the source files. Can probably tie that to a syntax highlighter style thing for in-ide feedback. The other is that markdown is a tempting format for literate programming. I do have some notes in obsidian that are fed to cmark to product html. With some conventions, splitting a literate program into executable code embedded in a html document is probably doable as an XML pipeline. In a much simpler vein, I'm experimenting with machine configuration from within obsidian. The local DNS server sets itself up using a markdown file so editing an IP or adding a new machine can be done by changing that markdown. I hope the author continues down this path and writes more about the experience.

nohat 2y ago
Yeah, I definitely see using this for literate programming. Not quite sure the best way to organize it. Maybe use a static site compiler to auto host documentation version.
- seagreen 2y ago
  I wish you could just use Obsidian Publish to host sites, but due to the indentation issue you have to control the rendering, which is a bummer. Obsidian Digital Garden[1] is FOSS, so it might be modifiable parse and output the code pages correctly. [1] https://github.com/oleeskild/obsidian-digital-garden https://github.com/oleeskild/obsidian-digital-garden
  - WorldMaker 2y ago
    The typical Markdown answer to needing indentation preserved is the "code fence" (triple backquotes ```), though I imagine the problem with that is that Obsidian by default stops dealing with Wikilinks inside fenced code. I don't know Obsidian that well, but maybe there's a way to use a code fence and have it support Wikilinks inside? A different direction to explore might be to explore proportional font coding techniques that rely less on whitespace. Lisp can be a good language to play with those ideas given whitespace isn't syntactic. Though idiomatic Lisp has certainly relied on semantic whitespace in coding styles for a very long time.

nbbaier 2y ago

> The solution I've been waiting for is source-code-in-the-database. I'm cheering on multiple projects attempting this. What are the projects you're especially bullish on?

DannyBee 2y ago
VisualAge C++, Java, and Smalltalk.
seagreen 2y ago
Hazel and Unison are two of the big ones. I'm friends with some of the Unison folks so I'm biased, but I really like how few features there are in the language. In general I'm just a huge sucker for subtractive improvement: if you can have a small number of awesome things (eg abilities) instead of a bunch of special case things (exception handling, monad trickery, dependency injection machinery) sign me up. I know less about Hazel, my understanding is that it's source-code-in-CRDTs, which is definitely structured source code though may not technically be in a database.
- thyrsus 2y ago
  Is this the unison you mean? https://www.unison-lang.org/ https://www.unison-lang.org/ Unrelated: what has your experience been using igneous-linearizer to help understand other people's code?
  - seagreen 2y ago
    That's it. And the linearizer is only one way-- you write text with [[links]] and turn that into plaintext.
    - thyrsus 2y ago
      You may have encountered leo [ https://leo-editor.github.io/leo-editor/ https://leo-editor.github.io/leo-editor/ ]. I use it to pull in or write code and break it up into comprehensible pieces. It works, but it feels overcomplex. I'm open to something simpler, and will give this a try.

binary132 2y ago

I really appreciate the unordinary direction you went with these articles and your site in general. An enjoyable read!

seagreen 2y ago
All credit to Conor White-Sullivan for making links cool again. > unordinary direction Hahaha, this is a very polite way to put that=) But I do appreciate it.
- binary132 2y ago
  Don’t take it as a backhanded compliment! We should all strive to surpass and excel the ordinary.