4 ms·

This is imho a pure tooling problem. We still use a loosely representation (text) for a semantically very rich data format (code). When you look at this object

This is imho a pure tooling problem. We still use a loosely representation (text) for a semantically very rich data format (code). When you look at this objectively that's madness. We should finally use proper editors for our semantically rich data format instead of "dumb" text editors. (Which are by now actually not so dumb, and which do again completely crazy things to re-extract the semantic information from a representation that inherently erased this information while we're used a loosely text representation). We should imho eventually switch to "binary source code" and start using proper tools to edit this data. What we do currently is like editing Excel sheets in a hex editor… Binary source code (or at least something in the ballpark of XML tech) would also solve a lot of other problems: For example the rendering of the data on screen would be independent form the actual data on disc. This means bitching around "code style" and even partly syntax could finally end. The stuff on screen could be rendered with the help of some "style sheets". Everybody would get the preferred look & feel for their code. (We do this already partly with syntax highlighting; we "just" need to extend this concept). Also things like version control, dependencies, and such, could be build in. Proper indices for searching (and refactoring code) could be also added (things that our editors today compute anyway, but in some ad hoc and very language specific way). Compilation of such code would be faster as all the parsing wouldn't be needed (as it would happen already in the editor; actually, also what is already done today, only that we than again throw away this information for no reason, so the compiler "may" regenerate it over and over on every run, instead that this stuff ends up in a kind of database which could be even part of the "binary source code" format). I guess some innovative editor features nobody even considered until now would be also enable this way. It's time for progress! (Most likely, as almost always, someone will come up with some references to very old systems that did already this because they were ahead of their time; let's see what new stuff I'm going to learn this time :-D).

carapace 4y ago
> references to very old systems that did already this Here's one: Alice Pascal > In a syntax directed editor, you edit a program, not a piece of text. The editor works directly on the program as a tree -- matching the syntax trees by which the language is structured. The units you work with are not lines and characters but terms, expressions, statements and blocks. https://www.templetons.com/brad/alice.html https://www.templetons.com/brad/alice.html
duped 4y ago
Having done a fair bit of programming outside of text, text is not "loose" or less "rich" than the programs it is used to describe. It's actually far more explicit and easier to grok than alternative representations of programs. Problems like code style have been solved by linters and auto-formatters. A lot of the ideas you're pitching have been pitched before and died for the same reason: parsing text is easy for humans and computers. Although I'd quibble with some minor points, like "compilation of such code would be faster". The major bottlenecks in a compiler are codegen, optimization, and linking compilation units. Even bad parsers are extremely fast.
- still_grokking 4y ago
  The stuff on screen would be still (rich) text, of course. Only that it would be a rendering of the actually underlying data structure. Linters and auto-formatters didn't solve anything. They are just a constant PITA, and a source of arguments. And to quibble a little bit: I said "faster". I did not say how much… Just parsing is indeed very cheep by now. But a major performance issue was unmentioned: Type checking. In languages with advanced type systems that's the most costly part. Putting all that information into a database and carry it along "the code" would make things significantly faster. It would be like there would be only incremental compilation of the few line you're just editing.
  - duped 4y ago
    I've actually worked on this problem quite a bit and taken this approach, but it's not as obvious as you're making it out to be - I'm going to talk in generalizations about optimization, which is obviously bad so take this at face value. Type checking is usually dwarfed by macro expansion, import resolution, optimization, and code generation. There's very little I/O, not that much work to do, and it is pretty straightforward to cache (*). There's a trap to thinking that complex syntax and type systems are potential bottlenecks - it's true the compiler has to do more work, but it's still not that much compared to all the other time it has to waste doing mundane things. An example is import resolution. In order to type check a compilation unit you need to type check its imports. In order to do that, the compilation unit and its dependencies needs to be macro expanded. If the language supports some kind of scripted macro-programming then you need to evaluate the macros and possibly cannot cache the results (performance pitfall #1). If the language does not support explicit imports, then the entire dependency graph may need be scanned and checked to resolve the imports (performance pitfall #2). This is a case where the simplest and most expressive language designs can have the absolute worst performance regardless of the compiler's design - the language definition forces the compiler to do a ton of mundane work, in particular I/O and cache invalidation. A central in-memory database holding a binary representation of the AST does not solve this problem and is not necessarily faster than redoing all of the work up front. In my experience, something as mundane as "import x from y" has a much larger impact over the performance of a compiler than any type checker or parser, and incrementalism strategies are tightly coupled to the design of the language being compiled and not the storage layer. It has to be approached holistically for the language being compiled. Features like "style sheets" for code are just linter and auto formatter config files. It may be useful to reuse the compiler internals for this, but in practice every tooling author eventually learns not to do that. You will need to modify the IR generated by the compiler, and it probably doesn't have an API that's anything close to what you need for that problem (*) Of course I'm glossing over type systems that allow higher-kinded dependent types, but those are going to suck to check the same that macro heavy code sucks and currently isn't that widely used.
andsoitis 4y ago
I know what source code is and I know what binary code is. But I’ve never heard of binary source code, and my Google search doesn’t turn up anything either. Can you elucidate?
- torben-friis 4y ago
  Most code editors deal with abstract syntax trees for linting, etc. Think HTML DOM but to define the syntax relations of the code itself. I'm assuming OP talks about saving something like that format directly, instead of text. Then information like 'this method is being called with these two parameters' isn't mixed with presentational info like whether you separated the parameters with newlines, or whether you use tabs or spaces. The downside being that it would not necessarily be immediately readable, the text editors would have to parse it and present it in a readable format, kinda like what happens now with, say, excel sheets.
  - still_grokking 4y ago
    Even currently code isn't "immediately readable". It shows up, but until the IDE ingested the project it does not provide any features besides basic string editing, which is mostly useless. So no new issue here. But reading in a project would be much much faster if the IDE would just need to read a rich binary format!
    - palata 4y ago
      > Even currently code isn't "immediately readable" I read code in text editors (not full IDEs) all the time... > it does not provide any features besides basic string editing, which is mostly useless. I also do write code in text editors all the time. Not always - I do use IDEs sometimes. But at least half the time, I would say. I use vim a lot, and all I want there is syntax highlighting. I would hate it if I was forced to use a heavy IDE everytime I want to read/edit some source code. Also I like grepping into sources without having to run an IDE that can search through binary code.
- still_grokking 4y ago

carapace 4y ago

> references to very old systems that did already this Here's one: Alice Pascal > In a syntax directed editor, you edit a program, not a piece of text. The editor works directly on the program as a tree -- matching the syntax trees by which the language is structured. The units you work with are not lines and characters but terms, expressions, statements and blocks. https://www.templetons.com/brad/alice.html https://www.templetons.com/brad/alice.html

duped 4y ago

Having done a fair bit of programming outside of text, text is not "loose" or less "rich" than the programs it is used to describe. It's actually far more explicit and easier to grok than alternative representations of programs. Problems like code style have been solved by linters and auto-formatters. A lot of the ideas you're pitching have been pitched before and died for the same reason: parsing text is easy for humans and computers. Although I'd quibble with some minor points, like "compilation of such code would be faster". The major bottlenecks in a compiler are codegen, optimization, and linking compilation units. Even bad parsers are extremely fast.

still_grokking 4y ago
The stuff on screen would be still (rich) text, of course. Only that it would be a rendering of the actually underlying data structure. Linters and auto-formatters didn't solve anything. They are just a constant PITA, and a source of arguments. And to quibble a little bit: I said "faster". I did not say how much… Just parsing is indeed very cheep by now. But a major performance issue was unmentioned: Type checking. In languages with advanced type systems that's the most costly part. Putting all that information into a database and carry it along "the code" would make things significantly faster. It would be like there would be only incremental compilation of the few line you're just editing.
- duped 4y ago
  I've actually worked on this problem quite a bit and taken this approach, but it's not as obvious as you're making it out to be - I'm going to talk in generalizations about optimization, which is obviously bad so take this at face value. Type checking is usually dwarfed by macro expansion, import resolution, optimization, and code generation. There's very little I/O, not that much work to do, and it is pretty straightforward to cache (*). There's a trap to thinking that complex syntax and type systems are potential bottlenecks - it's true the compiler has to do more work, but it's still not that much compared to all the other time it has to waste doing mundane things. An example is import resolution. In order to type check a compilation unit you need to type check its imports. In order to do that, the compilation unit and its dependencies needs to be macro expanded. If the language supports some kind of scripted macro-programming then you need to evaluate the macros and possibly cannot cache the results (performance pitfall #1). If the language does not support explicit imports, then the entire dependency graph may need be scanned and checked to resolve the imports (performance pitfall #2). This is a case where the simplest and most expressive language designs can have the absolute worst performance regardless of the compiler's design - the language definition forces the compiler to do a ton of mundane work, in particular I/O and cache invalidation. A central in-memory database holding a binary representation of the AST does not solve this problem and is not necessarily faster than redoing all of the work up front. In my experience, something as mundane as "import x from y" has a much larger impact over the performance of a compiler than any type checker or parser, and incrementalism strategies are tightly coupled to the design of the language being compiled and not the storage layer. It has to be approached holistically for the language being compiled. Features like "style sheets" for code are just linter and auto formatter config files. It may be useful to reuse the compiler internals for this, but in practice every tooling author eventually learns not to do that. You will need to modify the IR generated by the compiler, and it probably doesn't have an API that's anything close to what you need for that problem (*) Of course I'm glossing over type systems that allow higher-kinded dependent types, but those are going to suck to check the same that macro heavy code sucks and currently isn't that widely used.

andsoitis 4y ago

I know what source code is and I know what binary code is. But I’ve never heard of binary source code, and my Google search doesn’t turn up anything either. Can you elucidate?

torben-friis 4y ago
Most code editors deal with abstract syntax trees for linting, etc. Think HTML DOM but to define the syntax relations of the code itself. I'm assuming OP talks about saving something like that format directly, instead of text. Then information like 'this method is being called with these two parameters' isn't mixed with presentational info like whether you separated the parameters with newlines, or whether you use tabs or spaces. The downside being that it would not necessarily be immediately readable, the text editors would have to parse it and present it in a readable format, kinda like what happens now with, say, excel sheets.
- still_grokking 4y ago
  Even currently code isn't "immediately readable". It shows up, but until the IDE ingested the project it does not provide any features besides basic string editing, which is mostly useless. So no new issue here. But reading in a project would be much much faster if the IDE would just need to read a rich binary format!
  - palata 4y ago
    > Even currently code isn't "immediately readable" I read code in text editors (not full IDEs) all the time... > it does not provide any features besides basic string editing, which is mostly useless. I also do write code in text editors all the time. Not always - I do use IDEs sometimes. But at least half the time, I would say. I use vim a lot, and all I want there is syntax highlighting. I would hate it if I was forced to use a heavy IDE everytime I want to read/edit some source code. Also I like grepping into sources without having to run an IDE that can search through binary code.
still_grokking 4y ago