4 ms·
Storing the AST could also make source code branching and merging more accurate.
by rwmj 8y ago
Storing the AST could also make source code branching and merging more accurate.
- ben509 8y agoStoring the AST works for a class of problems, namely whitespace issues, but a refactoring as simple as renaming a token still makes merges a mess. (And if you store a whitespace / comment decorated AST, you can still have a messy merge.) That said, you could address that and other issues, you're just working with something several steps removed from an AST. Or you could call an AST the 90% solution.
- filmor 8y agoYou could just assign a token an id on its first use and keep that around, even if the name changes. Merges would then be trivial.
- setzer22 8y agoAnd how do you decide, given two versions of a source file, if the token was renamed or it was a new one? This looks "trivial" for trivial cases, but I can imagine lots of difficult corner cases there. Depending on how you define what constitutes a "rename", this may even be undecidable.
- Uberphallus 8y agoAlso making SCM language dependant.
- Ajedi32 8y agoIn git, you could do this with a custom merge driver: https://github.com/Praqma/git-merge-driver https://github.com/Praqma/git-merge-driver No need to put that sort of thing in core.
- TeMPOraL 8y agoActually, even basic AST-aware diffing algorithm would be a nice thing. I write a lot of Lisp code - Lisp source is, quite literally, AST serialized to text - and regular diffs in SCMs are really annoying. Like, I remove some AST node in the middle, and git diff will think most of the function changed, just because indentation shifted a bit.
- HankB99 8y agoAST? Asynchronous System Trap? :o
- WorldMaker 8y agoMost languages don't have ASTs that work well in "degenerate" cases like unfinished/work in progress/sample code. These are things you often want to have in source control. There are also very few AST standards between languages making for a lot more language-dependent work for an SCM. (For what it is worth, I toyed with what I think to be a useful compromise to the idea, which is to use syntax highlighting tokenizers, which perform well and interoperate well with character-based diffs: https://github.com/WorldMaker/tokdiff https://github.com/WorldMaker/tokdiff)
- ken 8y agoI agree that degenerate cases are one of the major problems with trying to move to higher-level editing abstractions (and it's one of the big issues I faced with my own application), but I don't see how it's a problem here. If it doesn't have a valid AST, then it's not a valid program, either. If you're using a formatting program (like prettier or gofmt) and pass it an invalid program, you're either going to get an error, or undefined behavior. And anyone else opening it in a different editor is going to have their language-mode interpret the invalid program differently than yours, too. Source code tools like structured search won't work predictably, either. This sounds to me like saying "An XML structure editor? But what if I want to put an invalid XML structure in a file called foo.xml and commit it to the repository?" Or my first boss complaining that visual text editors didn't let you see every byte. Or Mel needing to know drum addresses so he could use them for constants. As time goes by, we move to higher level abstractions, and (thanks to tools like gofmt) we're already at the point where you probably shouldn't be pushing source code which doesn't even have a valid AST, and expect all your tools to work perfectly. There's plenty of ways to write and commit WIP code without needing an invalid AST on disk.
- WorldMaker 8y agoI thought similarly for many years. What's the use of code in source control that doesn't compile anyway? Then I started paying attention to all the reasons why we naturally might want to check in "invalid" code. All the cases where I'm never going to finish an entire refactor in a single commit, and the whole refactor makes far more sense in documented steps where many of the intermediate steps will never compile. All the cases where sending broken code to a source control server at the end of the day is both the best way to back it up and the easiest way to get fresh eyeballs on it in the morning. Even with CI systems in place, sometimes the errors that the CI bot can tell you are as useful as the ones your own machine's build environment can tell you. (Especially in those weird cases where it turns out to be that maybe its the build environment on your own machine that's the problem and you've been beating yourself up over a bad install of something that should be unrelated, and the build errors on the remote machine lead you to the real problem.) All the cases where I might build tests first, and the compiler is a test, especially in a static typed language, before working backwards to make the tests pass/run/compile. A lot of people like to think of programming code as some purely logical construct, but good code is poetry, even in its mistakes. Sometimes we need drafts to tell our stories right. Sometimes we need bad poetry in source control as a warning to others to help them realize what good poetry can be. It's interesting to want code editors to keep us from ever writing bad poetry to disk, but it's also somewhat inhumane. People write bad poetry all the time, it's a natural skill. Saving bad poetry for posterity is sometimes the only way we get better poetry.