9 ms·
Treefrog: A code editor that uses both AST and text editing commands
- rajandatta 5y agoIntriguing idea. This would be very interesting to try with a Lisp or Scheme like language. Does it have any support for S-exp based languages?
- gushogg-blake 5y agoI haven't added any yet, but all parsing is done with tree-sitter so languages can be added pretty quickly. One issue is that tree mode works on whole lines (I think the benefit of AST-style navigation vs moving a text cursor diminishes as the selections you want get smaller), so it's better suited to languages where the blocks are naturally separated by newlines as opposed to the ")))))" style.
- b3morales 5y agoAre you using the official tree-sitter grammars as-is? Or tweaking/writing your own to support your use case? From what I've seen not all of them are suited to structural editing, since the emphasis is on syntax highlighting.
- gushogg-blake 5y agoYeah using them as is and haven't had any issues so far. I think they are at a slightly lower level of abstraction than an AST, e.g. an async method in JS is a method_definition with an "async" node followed by a property_identifier, whereas an AST might wrap all this into an AsyncMethod node - but all the info you need is there.
- rajandatta 5y agoParsing on newlines is probably not a great fit for Lisp-syntax languages. Does the availability of LSPs make a difference? Asking as I don't know your tool.
- Jtsummers 5y agoYeah, s-exprs don't break down by lines very well, but they are easy-ish to work with in a structured format, similar to HTML/XML editors since the delineation between elements is usually very clear. The only problem with lisps would be when macros are involved where there is more fluidity in the structure. But something like this: (defun double-all-the-things (things) (mapcar (lambda (x) (* 2 x)) things)) would be straightforward to navigate through with an s-expr aware editor.
- gushogg-blake 5y agoIt's more that the selection operates on whole lines in Tree mode, so it would parse fine with the tree-sitter grammar (for syntax highlighting etc) but selecting the inner contents of a block would take the block's closing ) with it. This would only matter if you wanted to move the contents somewhere though - editing and refactoring commands (including with the help of LSPs) will work, with the selection being the outermost node on the first selected line.
- bitwize 5y agoParedit has entered the chat.
- buescher 5y agoA long long time ago on a toy lisp called Interlisp/65 I used something that was like the s-exp equivalent of a line editor. https://atariwiki.org/wiki/Wiki.jsp?page=Lisp https://atariwiki.org/wiki/Wiki.jsp?page=Lisp Source is here: https://atariwiki.org/wiki/Wiki.jsp?page=LispEditor https://atariwiki.org/wiki/Wiki.jsp?page=LispEditor If you really wanted a vi-like experience for editing lisp, you could figure out how to make an editor like that full-screen and interactive.
- moonchild 5y agoInterlisp, fructure
- vekelly09 5y agoSpeaking of... By 1978 InterLISP had grown up quite a bit and its tree-based editor was quite powerful. It supported several tools like Masterscope and DWIM (Do What I Mean), the likes of which I would not see in text-based editors for almost two decades. It was quite habitable, even though a CLI tool and tree-based. I wrote all the code for my dissertation using that editor on a DEC-20. So much better than even EMACS at the time. The simplicity of S-expressions was critical to its success, IMHO. Extending the editor was simple, too. LISP, you know.
- eatonphil 5y agoThe scroll is really hard to use on my macbook. I can't scroll slowly enough. It moves massively up and down with each scroll even when I am very careful.
- gushogg-blake 5y agoThanks, I haven't tried it on a touchpad yet, will have a look.
- gushogg-blake 5y agoShould be fixed now
- catskul2 5y agoThat was fast : )
- gushogg-blake 5y ago:) I was surprised touchpad did anything tbh - turns out it sends wheel events with different deltaY/deltaX values, so I just had to use the actual values instead of what it was doing before (checking if < or > 0 and scrolling by 3 rows/columns worth)
- gushogg-blake 5y agoBackstory - I started experimenting with AST editing ideas after getting frustrated with my editor's lack of understanding of the structure and meaning of code, ie. the editing primitives all dealt with text and simple changes to the code often required repetitive manual text edits that had nothing to do with the meaning of the change. I went through a couple of prototypes and landed on the idea of augmenting a standard editing interface with a new mode that works on a simplified representation of the structure, which basically follows indentation levels - blocks of code like functions, loops, etc, can be moved and manipulated as units, and there are specific editing commands for the different elements. The main idea is to design all of the editing and navigation commands around the thought processes (instead of designing around a data structure, which I think is where traditional editors have gone wrong), so that it feels natural and intuitive to use - like something that's been designed for what editing code actually involves, as opposed to a text editor that's had lots of code intelligence stuff added on top of it.
- TylerJewell 5y agoClever line of thinking. Is it fair to think of IDEs that include language-specific refactoring capabilities as a providing a developer abstraction over AST editing?
- gushogg-blake 5y agoThanks! Yeah, I think of refactorings as being a level of abstraction above text and AST editing - kind of like macros for AST edits.
- NeutralForest 5y agoInteresting, there's been an [ongoing discussion to use tree-sitter with Emacs](https://archive.casouri.cat/note/2021/emacs-tree-sitter/index.html https://archive.casouri.cat/note/2021/emacs-tree-sitter/inde...) because right now, syntax highlighting is a mix of regex and functions. I don't how language parsing should be exposed to make it easy to have parsing in editors. But I believe some efforts like tree-sitter are most welcome and also more inline with what we expect editors to be.
- gushogg-blake 5y agoYeah, tree-sitter and LSP are both steps in the right direction I think.
- aidos 5y agoNeovim has moved in this direction too. There’s a lot of potential there. I hacked up a plug-in yesterday to let me run tree sitter queries over my codebase. I’d upgraded a dependency and some patterns were no longer valid so I was able to track them down in a way that would never have been possible with regex etc (certain function called with more than one argument chained with one of several other functions).
- sesm 5y agoWhat are hypothetical benefits of using this versus Webstorm/Intellij with Vim mode?
- gushogg-blake 5y agoOne example would be turning a group of statements "xyz" into "if (some condition) then xyz, else (something else)" - with a text editing interface you have to think a bit about the exact text selection you want (or enter visual mode in vi to get the lines), cut them, write the if statement, paste them in, and usually some reformatting, whereas with Tree mode (the dedicated mode for expressing meaningful edits to the code) you can do Esc-w for "wrap", type the if statement, and insert the original statements from a dedicated clipboard. Other cases are similar, e.g. being able to just drag and drop a div into another div without thinking about the exact text selection - the benefit is having to think slightly less about the mechanics of your editor or the particular data structure (text or ASTs).
- sesm 5y agoThanks for explanation! I think for the first case there is a ‘wrap in if-else’ refactoring in Webstorm/Intellij, but I rarely use it. Second case is interesting, I would be excited to have this for React components.
- RyEgswuCsn 5y agoIsn't this what the Intellij/Eclispe IDEs have been doing since the very beginning? All the powerful refactoring features are based on AST manipulation.
- gushogg-blake 5y agoYes, sort of, but they've always been added on top of a standard text editing interface so tend to be behind context menus (which is OK as they're also often high-level refactorings that aren't used that often). Treefrog's structure-aware edits are lower-level and more deeply integrated into the editing interface - there's a whole mode dedicated to them where the mouse and keyboard behave differently. The aim is to design an intuitive editing interface; the AST itself is just one tool to help with that.
- iddan 5y agoThis is the future of programming. Code isn’t (just) text, it’s a tree and we can edit it in a much more elegant way utilising it’s structure
- luckystarr 5y agoThere has been a movement to try something like this under the buzzword "projectional editing". The idea was to edit a representation of the code and not the code itself. People and companies experimented with it but didn't get much adoption. Not sure why. Perhaps you need to allow the developers to reach invalid states, because the editors I tried just refused entries which would result in a syntax violation. This is harder to implement though. :)
- gushogg-blake 5y agoYeah, that's one of the main issues with ASTs and why I think they make the same mistake as text editors ultimately - designing around a particular data structure just constrains you into weird UX scenarios that don't have anything to do with the actual problem of making something that lets you write code.
- Gibbon1 5y agoFriend of mine said he worked with a system like that. Problem with it was it couldn't save code that didn't compile. That's a defect. Am reminded of someones comment that they would like a compiler that just stubs out code that doesn't compile. For a system to be workable you need to be able to handle that. I feel that because no one uses AST aware editors no one understands what that would bring to the table. A counter example I've used CAD programs. And all of them store the design files as databases internally. Which means you can and people do preform database operations on them. Generate reports and perform update operations just like you would with an SQL type database.
- radford-neal 5y agoI wrote a tree editor for programs for my Master's thesis, in 1980. You can get it at https://www.cs.toronto.edu/~radford/MSc-thesis.abstract.html https://www.cs.toronto.edu/~radford/MSc-thesis.abstract.html The editor operates on the AST down to the lowest level, with careful design of keyboard input and cursor feedback to make that not too painful for things like infix operators. The language is described by a grammar, and there is an AST macro language. My conclusion afterwards was that the shift at that time away from keyboard input towards editing with a mouse was negative for editing in terms of the AST (rather than text), since with AST editing there's a less direct connection between seeing and pointing and what you're editing. I got interested in other things. But perhaps that was a premature conclusion.
- amichail 5y agoWhat do you think of TeXmacs, which uses a WYSIWYG tree editor for writing scientific/math documents? (TeXmacs is not based on TeX nor emacs but is inspired by both.)
- konkin 5y agoMath notation is hopelessly stuck in the 19th century and needs an update to the 20th. Polish notation with operators of known arity is an excellent way to ease natural scientists/mathematicians into a modern tree based grammar for mathematics without a soup of parenthesis, e.g. / 1 / 2 3 vs / / 1 2 3 A full s-expression based grammar for mathematics is unfortunately not something that you can write on paper and requires white space delineation to be readable. On a screen with something like emacs paredit both of those are a breeze however. The fact that higher order functions don't need a special notation is really freeing.
- Jtsummers 5y agoWhat value would Polish notation add over (largely) infix for the typical mathematician? Why do mathematicians need to be told what notation to use by non-mathematicians when their notations (note the plural) are in wide, and apparently successful, use today? How would Polish notation help, for instance, with things like describing set membership or logical relations? How is this more readable (using ASCII): -> a b Than the current approach: a -> b Which provides a convention sense of the relationship that the former does not? The second one can be "read" in order: a implies b. The first cannot.
- dehugger 5y agoAn explanation of what AST actually means would be helpful at the top of the intro page. Quite of a few of the selling points were lost on me due to lack of familiarity with the term. I can (and will) google it, but if I am looking at a new product, especially one I have to buy, it'd be nice to have the resources available to understand what it offers without turning to a third party.
- ziggus 5y agoYeah, I had to dredge my memory for what an AST is - ran into it ages ago on a compiler design project, but I could see how it works in a text manipulation context as well. https://en.wikipedia.org/wiki/Abstract_syntax_tree https://en.wikipedia.org/wiki/Abstract_syntax_tree
- catskul2 5y agoAny chance of opensourcing this?
- gushogg-blake 5y agoI'm not necessarily against it in the long term, just keeping all options open for now as I want to be able to work on it full time and that obviously involves making money somehow.
- catskul2 5y agoUnderstandable. I wonder if a patreon model might work. I've not hear many success stories about building companies around single not-yet-established pieces of software. But I've had similar dreams and wish you luck.
- svilen_dobrev 5y agosome notes: * ctrl-home/end do not go to very top/bottom * python tree-mode does not do anything sensible
- sweetsocks21 5y agoThe S,D shortcuts work for me, but I can't get J or K to do anything? Some UX feedback on what is valid at the current time may be useful. One problem with structural/projection editors is it's not as clear what is a valid operation in the current state. There's some existing work in this area you might find interesting as well: https://twitter.com/dm_0ney/status/1414742742530498566 https://twitter.com/dm_0ney/status/1414742742530498566 https://hazel.org/ https://hazel.org/ https://hazel.org/build/dev/ https://hazel.org/build/dev/ (expand the left side pane by clicking the (?) to see the valid operations)
- gushogg-blake 5y agoThanks. Just checked and j/k are actually not implemented yet, so that explains that!
- ngvrnd 5y agoI vaguely remember someone telling me, a long time ago, that Thomas Reps' PhD thesis (https://ieeexplore.ieee.org/book/6267325 https://ieeexplore.ieee.org/book/6267325) showed that it wasn't practical to build this kind of editor. Perhaps that was not at all the point; it was a long time ago. I wonder, though, can anyone explain 1) if Reps' thesis does bear on the Treefrog work 2) what it was in Reps' thesis that bears on the difficulty of making syntax-tree based code development tools?
- gushogg-blake 5y agoI wonder if it was important that an actual abstract syntax tree was used? I'm realising the title might be a bit misleading - it should probably be "AST-inspired" commands, as the tree-sitter trees are technically concrete as opposed to abstract and the commands aren't tightly coupled to a particular representation.
- billconan 5y agothe project page doesn't anything about supported languages ?
- gushogg-blake 5y agoCurrent langs are javascript, html and (s)css - there's a dropdown on the toolbar to create a new file in a given lang (the others there are partially supported, no Tree mode commands). The goal is to support all popular languages, as they can be added fairly easily with tree-sitter grammars.
- winter_blue 5y agoA quick note: the editor looks blurry/low-res on a 4K monitor (and I would guess, probably on other high-res displays as well).
- gushogg-blake 5y agoThanks! Are you on a mac? Someone else noticed it on mac, just on the actual editor, so I thought it might be to do with rendering text on canvas (sometimes you have to use half-pixel coordinates to get things to align properly)
- winter_blue 5y agoNot on a Mac. Just a 27-inch 4K display (with a 2.5x scaling configuration).
- catskul2 5y agoThe features I've always dreamed about from an AST aware editor was shell-like functionality where operations to rename, move, copy, create, etc were akin to file manipulations done on the command line. At one point I was thinking about the possibility of FUSE-ifying(https://en.wikipedia.org/wiki/Filesystem_in_Userspace https://en.wikipedia.org/wiki/Filesystem_in_Userspace) some code so that a normal shell could be used. Any chance of shell-like functionality from treefrog?
- gushogg-blake 5y agoInteresting! Yes, one of the main things I'm doing at the moment is thinking of and prototyping new commands and ways to represent code. I like the shell idea.
- gushogg-blake 5y agoSo things like classes and functions would be represented as files within a dir representing the actual file?
- catskul2 5y agoYeah! I think the difficult part of this idea though is maintaining/moving the necessary dependencies, or at least keeping track of them to warn that they're no longer satisfied. In many cases it probably requires awareness of packaging in a way that normal AST manipulation does not, at least to be particularly useful it would.
- gushogg-blake 5y agoRight - this is where I would use an LSP server
- int_19h 5y agoThe text in the editor is blurry - looks like it's rendered at a lower resolution, and then upscaled with bilinear filtering. (This is on a 4K display running Win11 with UI scale set to 200%)
- gushogg-blake 5y agoAh yeah, it's canvas based so I suppose it will be being scaled like an image. Thanks!
- thomasikzelf 5y agoI think you need to scale your canvas by window.devicePixelRatio for higher density screens. Anyway I am receiving an error on page load: Uncaught TypeError: Cannot read properties of undefined (reading 'watch') under platform.jsonStore.watch
- gushogg-blake 5y agoThanks! Error should be fixed now
- deleted 5y ago[deleted]
- byteware 5y agomaybe related? https://www.jetbrains.com/mps/ https://www.jetbrains.com/mps/
- leecommamichael 5y agoIDE-features like “Refactor” are my idea of how this kind of editing already exists.
- hackinthebochs 5y agoOne pet peeve of mine that this kind of editor might be able to address is how live compilation is sketchy when you're actively editing a file. For example when you're in the middle of an edit and you end up with a huge number of compilation errors because the parse of the file is in flux. Having the editor and the syntax parser aware of each other can help fix such issues. Semantic merging would also be a nice feature.
- tekknolagi 5y agoThis looks broken for me -- nothing loads on the homepage and I get some JS errors. I am on Firefox on macOS.
- vinodkd 5y agosame. tried FF and Chrome on Windows, and FF on Ubuntu. Blank space where the editor should be.
- gushogg-blake 5y agoThanks - fixed
- danielvaughn 5y agoI remember watching a talk about this, the performance improvement was staggering. Would be cool to see a big vendor like VSCode use it.
- thomasikzelf 5y agoSome feedback on the actual implementation: - I cannot reach inside if and while statements using tree mode - It is not clear which lines are available for refactorings - dragging a line reveals "+ If" and "+ else if" under an if statement. Dropping the line onto it does not work. - tree based moving does not move the viewport About the business side of it: I think it will be hard to create a whole editor just for these features. I know that Jetbrains editors also have ast aware refactorings and I often use moveBySymbol in sublime text which acts a lot like your tree navigation. I think it makes more sense to create this as a plugin, but it will be difficult to monetize that. Anyway I applaud you for creating something that actually works!
- gushogg-blake 5y agoThanks for the feedback! - can you move into an if/while if it's not already selected? For already selected blocks I've had it both ways and found it better to be able to drag from anywhere in the block - you can select a child by clicking it. Both ways have their advantages and disadvantages though. "d" should also work to navigate to a block's first child. - yes maybe some kind of visual indicator would be good, but if required after the first few uses then the interaction would need to be redesigned I think, as there shouldn't be any unexpected or unintuitive behaviour - hmm, that's working for me, does anything at all happen when you drop it? - yeah, haven't implemented that yet I thought about doing it as a plugin as well, but I think trying to add something genuinely new to an existing editor would be possibly harder than just making one (I've been working on Treefrog for about 7 months full time and using it as my only editor for the last 2), and would end up either not fitting in well with the existing UI or having clunky UX to switch between modes etc. Designing new modes from the ground up (and being able to prototype them quickly with web tech and Electron) allows for much easier experimentation.
- thomasikzelf 5y ago> can you move into an if/while if it's not already selected? Sorry I meant the condition of while, for and if. > hmm, that's working for me, does anything at all happen when you drop it? Now that I try it again it seems to be working. Before it would just drop the line bellow or above the "+ if".