3 ms·

Bit of a tangent, but if we commonly stored source in a standard, fully attributed AST instead of caveman text, caching/indexing and deterministic search would

Bit of a tangent, but if we commonly stored source in a standard, fully attributed AST instead of caveman text, caching/indexing and deterministic search would be a breeze. So would a million other useful applications such as every other IDE feature and source control. As the decades go by it confounds me that we are still messing with plain text, and with virtually no resistance to it. Maybe it's to keep the tabs vs. spaces flame war alive. _shrug_

zokier 3y ago
tbh the on-disk storage format is pretty irrelevant, its all just bits and bytes on disk anyways. for many cases you can consider source text a serialization of ast; no matter how you store it you still need to parse the data somehow. Sure some formats are easier to parse than others (hello sexprs), but that is just minor perf aspect, conceptually its all the same. this closely relates to the missing type discussion, there is no universal "tree" type, and as such it is also difficult to construct any universal "ast" type; different uses need trees structured in different ways: https://news.ycombinator.com/item?id=39592444 https://news.ycombinator.com/item?id=39592444
- owlstuffing 2y ago
  > tbh the on-disk storage format is pretty irrelevant, it’s all just bits… Nonsense. A completely attributed AST is the result of a full parse. The information in the AST is vastly richer than source. > there is no universal tree type Obviously. It should be understood that the idea is to have a standard AST API/format _per language_. As such it would be standard practice to provide this information as part of the parser/compiler library.
flohofwoe 3y ago
This is such an obvious and old idea that it would have killed text as source code by now if it would be an actually *good* idea ;) (IIRC one attempt was an IDE from IBM in the 90s).
- owlstuffing 2y ago
  One reason it hasn’t gotten traction is due to a “herding cats” problem - the breadth of existing languages and SCMs in use is too big to manage. But mostly it’s the business model, who’s going to invest the vast amount of time for this when there is virtually nothing in it for them, it’s a giant onion. There’s little money in tooling, there’s far less in metatooling. There’s also a bit of “squeaky wheel” resistance. Still a significant minority clinging to text editors as opposed to modern IDEs. But given the demographics here I imagine this segment will begin to shrink dramatically within a decade or so. Shrug.
pif 3y ago
> As the decades go by it confounds me that we are still messing with plain text As the decades go by, it confounds me that people have yet to understand the concept of simplicity.
- owlstuffing 2y ago
  I understand your position that raw source is a simpler format. But in terms of professional IDEs is it simpler to throw out the parsed information and save only raw changes and force static tooling/IDEs to maintain separate caches and indexes of AST models? Not to mention the out of sync issues caused by this separation. Is it simpler for source control managers like git to only understand text and make everyday changes such as rename refactors cascade across entire codebases and cause unnecessary, costly merge conflicts? There are countless examples where AST-as-source _minimizes_ the complexities currently foisted on developers both directly and indirectly through tooling inefficiencies and other issues rooted in having multiple sources of truth.
duped 3y ago
I see this comment all the time but it falls apart when you try and actually implement IDE features. Fundamentally, an abstract syntax tree is a canonical representation of a program in one grammar. You cannot standardize it, because different languages have different grammars, and therefore different abstract syntax trees. You also don't want an abstract syntax tree for IDE features at all, you want a concrete (or full) syntax tree - since ASTs are lossy and don't represent the text. Another barrier is that constructing an AST usually requires the input to be correct and complete, but most of the time, source code is incomplete and invalid. It doesn't make sense to cache the AST because it is invalidated nearly every key stroke - you want to be able to construct an AST for performing the queries you need as fast as possible in the context of invalid input. And finally, you still need to have the source text to display anything useful to the user. So you're not saving any work by using a cached AST, you're not saving memory or disk, and you still need the plaintext anyway. All that said, there is such a thing as the language server index format (LSIF) which LSPs can provide as a cache to clients that support LSP for performing queries on rarely invalidated source (eg dependencies) but it's not super useful for code in flight.
- owlstuffing 2y ago
  > different languages have different grammars Obviously. It should be understood that the idea is to have a standard AST API/format _per language_. As such it would be standard practice to provide this information as part of the parser/compiler library.

zokier 3y ago

tbh the on-disk storage format is pretty irrelevant, its all just bits and bytes on disk anyways. for many cases you can consider source text a serialization of ast; no matter how you store it you still need to parse the data somehow. Sure some formats are easier to parse than others (hello sexprs), but that is just minor perf aspect, conceptually its all the same. this closely relates to the missing type discussion, there is no universal "tree" type, and as such it is also difficult to construct any universal "ast" type; different uses need trees structured in different ways: https://news.ycombinator.com/item?id=39592444 https://news.ycombinator.com/item?id=39592444

owlstuffing 2y ago
> tbh the on-disk storage format is pretty irrelevant, it’s all just bits… Nonsense. A completely attributed AST is the result of a full parse. The information in the AST is vastly richer than source. > there is no universal tree type Obviously. It should be understood that the idea is to have a standard AST API/format _per language_. As such it would be standard practice to provide this information as part of the parser/compiler library.

flohofwoe 3y ago

This is such an obvious and old idea that it would have killed text as source code by now if it would be an actually *good* idea ;) (IIRC one attempt was an IDE from IBM in the 90s).

owlstuffing 2y ago
One reason it hasn’t gotten traction is due to a “herding cats” problem - the breadth of existing languages and SCMs in use is too big to manage. But mostly it’s the business model, who’s going to invest the vast amount of time for this when there is virtually nothing in it for them, it’s a giant onion. There’s little money in tooling, there’s far less in metatooling. There’s also a bit of “squeaky wheel” resistance. Still a significant minority clinging to text editors as opposed to modern IDEs. But given the demographics here I imagine this segment will begin to shrink dramatically within a decade or so. Shrug.

pif 3y ago

> As the decades go by it confounds me that we are still messing with plain text As the decades go by, it confounds me that people have yet to understand the concept of simplicity.

owlstuffing 2y ago
I understand your position that raw source is a simpler format. But in terms of professional IDEs is it simpler to throw out the parsed information and save only raw changes and force static tooling/IDEs to maintain separate caches and indexes of AST models? Not to mention the out of sync issues caused by this separation. Is it simpler for source control managers like git to only understand text and make everyday changes such as rename refactors cascade across entire codebases and cause unnecessary, costly merge conflicts? There are countless examples where AST-as-source _minimizes_ the complexities currently foisted on developers both directly and indirectly through tooling inefficiencies and other issues rooted in having multiple sources of truth.

duped 3y ago

I see this comment all the time but it falls apart when you try and actually implement IDE features. Fundamentally, an abstract syntax tree is a canonical representation of a program in one grammar. You cannot standardize it, because different languages have different grammars, and therefore different abstract syntax trees. You also don't want an abstract syntax tree for IDE features at all, you want a concrete (or full) syntax tree - since ASTs are lossy and don't represent the text. Another barrier is that constructing an AST usually requires the input to be correct and complete, but most of the time, source code is incomplete and invalid. It doesn't make sense to cache the AST because it is invalidated nearly every key stroke - you want to be able to construct an AST for performing the queries you need as fast as possible in the context of invalid input. And finally, you still need to have the source text to display anything useful to the user. So you're not saving any work by using a cached AST, you're not saving memory or disk, and you still need the plaintext anyway. All that said, there is such a thing as the language server index format (LSIF) which LSPs can provide as a cache to clients that support LSP for performing queries on rarely invalidated source (eg dependencies) but it's not super useful for code in flight.

owlstuffing 2y ago
> different languages have different grammars Obviously. It should be understood that the idea is to have a standard AST API/format _per language_. As such it would be standard practice to provide this information as part of the parser/compiler library.