7 ms·

Killing Off Wasabi – Part 2

90 points by GarethX 11y ago

andyjohnson0 11y ago
For those who missed it, discussion of part 1 is at [1]. [1] https://news.ycombinator.com/item?id=9777829 https://news.ycombinator.com/item?id=9777829
- unstabilo 11y ago
  [1] [1] Thank [2]! [2] you
  - savanaly 11y ago
    Honestly, I found the way he used [1] to be easier to parse even though it's the only link and at the end of the sentence. I appreciated it.
johnchristopher 11y ago
For a split second I thought it was going to be about Nullsoft's Wasabi https://en.wikipedia.org/wiki/Wasabi_%28software%29 https://en.wikipedia.org/wiki/Wasabi_%28software%29
- the-dude 11y ago
  I wonder why you have been downvoted. I thought of the same. Upvoted.
  - johnchristopher 11y ago
    Thanks. Someone might have `misclicked'.
anonymousDan 11y ago
Meta question, do transpilers actually work well in practice? My intuition is that they would have all sorts of limitations, but I've never had to use one. Could anyone point me to a resource describing what those limitations might be? In what circumstances would they work best?
- dagw 11y ago
  I use two transpilers in my day to day work, cython and coffeescript. In both cases they work really well and save me a bunch of time and effort. However in both cases the transpiler is written in parallel with the source language which always makes it a lot easier and the transpiler obviously supports 100% of the source language. I have also played with transpilers written for 'foreign' languages and then you do end up in a situation where you can only use a (often badly defined) subset of the source language, and debugging is a pain since the output is often very difficult to read. Still they're handy when you need them and can (sometimes) save you time compared to porting the code by hand.
- Arnt 11y ago
  Can't point to any resource... but my experience is that the main problems are a) parsing the input language and in the same way as other tools do and b) giving useful diagnostics and making the next-step compiler's and/or debugger's diagnostics useful. Far from trivial, but not insurmountable.
- chii 11y ago
  one of the most popular transpiler is http://haxe.org http://haxe.org - it compiles down to many different languages, and it works pretty well.
- tomjen3 11y ago
  Transpilers are just a new word for compilers - the major difference is that the target language is a high level one, but that just means that it is a lot easier to write the compiler (no optimizing register assignment!). I doubt a transpiler has any limitations that doesn't apply to compilers in general, though of course any particular one might.
- jerf 11y ago
  In the end, a "transpiler" is just a compiler and subject to the same limitations. There is no generic set of limitations that apply to all compilers. Some are very good because they do something relatively easy, like Coffeescript -> JavaScript. I don't mean that writing a great Coffeescript compiler [1] is necessarily "easy", but the Coffeescript language was tightly designed to map directly to Javascript constructs, so there's hardly any "limitations" at all; very nearly the entire target language is available to you, and to the extent that there may be JS you can't generate, it's probably JS the community generally agrees is a bad idea anyhow. Others, like gcc or LLVM, are actually bridging a fairly large gulf now between the source and target languages [2], but do a great job because of massive piles of effort by incredibly smart people. But if you did have to make generalizations, the two big ones are that generally you can have a performance problem if you're forced to compile something onto a target with a very different paradigm, and the target just doesn't let you have a low-enough level view of things to implement the source language's abstractions efficiently. For instance, consider the languages targeting the JVM before invokedynamic was added to the JVM... there was a fundamental paradigm mismatch that bled away a certain unavoidable amount of performance. The other problem you can get is that you may not get slick access to the low-level details of the target language; for instance, take a look at the piles of things that compile to JavaScript and compare how they allow you to use jQuery. It's anything from "What's the problem?" in CoffeeScript to being a full-on Foreign Function Interface-type call in GHCJS (even if you can get it pre-wrapped for you [3]), a Haskell->Javascript compiler. Both are really two aspects of the same "impedance mismatch" problem. [1]: Look, I'll be honest, I consider the term "transpiler" completely unnecessary. It's a compiler. Feel free to just keep using transpiler in your reply and I won't say anything again. I've had this argument, I bow to the trend, but this old fogey doesn't intend to change (and still considers it a serious impediment to understanding to believe they are two different things, despite using the exact same techniques, architecture, tools, mental models... grumble grumble). [2]: My recent previous comment goes into more detail on that claim: https://news.ycombinator.com/item?id=9800231 https://news.ycombinator.com/item?id=9800231 [3] https://github.com/ghcjs/ghcjs-jquery https://github.com/ghcjs/ghcjs-jquery
mwcampbell 11y ago
Why not open-source the whole compiler? Probably nobody else would use the whole thing as is, but there are surely other interesting bits, like the global type inference.
- taprun 11y ago
  Perhaps I misunderstand your question, but doesn't the article say the source is on github? https://github.com/FogCreek/RoslynGenerator https://github.com/FogCreek/RoslynGenerator
  - pweissbrod 11y ago
    In github it states "The CLR importer, lexer, parser, interpreter, type checker, language runtime, JavaScript generator, and other components of Wasabi are missing."
- krallja 11y ago
  The type inference code would make the world a worse place. Whole careers would be ruined by new developers closing one eye and peeking at the last close brace.
vog 11y ago
Although slightly off-topic, I find the mentioned idea of "syntax trivia" very compelling. This should not only be part of code parsers, but of any parsers. I thought about something similar for binary parsers. Here, too, there are often many different ways to express a certain structure. Most libraries (such as Exiv2 for PNG/JPEG/EPS/etc. metadata) perform the following procedure: First, they parse out all information they are interested in. Then the application modifies that structure. Then it has to be embedded back into to binary format (here the image). The latter step is a nasty type of merging. They go over the file, try to find places where to "safely" embed or replace parts, then serialize their stuff and try to put it there. However, if the binary parser was lossless, all changes to the parsed structure would preverse the "trivia", so serializing is straight forward. And any potential issues could be handled on application level in the parsed structure, rather than by guesswork and heuristics during the merge phase. The only problem could be large BLOBs, but those could easily be represented as position & offset in the original file, rather than as the actual binary data in memory. (assuming the file will not change in the meantime, but in that case there the "merging" approach is very dangerous, too.)
- Locke1689 11y ago
  Syntax trivia is a very important component and a pretty natural fit once you start thinking about the feature that we decided was absolutely necessary: full-fidelity round-tripping between parsed syntax trees and the original source. I think this is one of the most common mistakes compiler writers make -- they don't spend enough time thinking about incorrect code. Let me be clear: most code is wrong. If your code is right, you compile it once. If your code is wrong, you compile it many times until you get it right. More importantly, if you run your IDE off your compiler, almost every character you type is a wrong program. Having a full-fidelity syntax tree is essential for having great experiences with wrong code. In addition, it easily solves the problem of having to serialize your trees -- the source text is the serialization. This feeds into API considerations as well. A number of people have repeatedly talked about how cool it would be to have a lot of other tools understand your AST (like if Git could support checking in ASTs instead of source). This is the wrong way of looking at it. If you're dealing with a raw AST you often have to have domain-specific knowledge of the language itself. Instead, what you want is to take the thing with the most domain specific knowledge, the compiler, and allow it to answer questions, i.e. have an API. By having round-trippable source, all source is essentially given a transparent API that can be used just as if you were interacting with the source code itself. Anyway, this is going off the rails, but it's one of the numerous things I'd point to for many production compilers and say, "you are the past, this is the future."
ygra 11y ago
One minor correction: Trivia are part of tokens, not nodes, which allows to represent things like white space between the if keyword and the opening parenthesis. There is a convenience method to get leading and trailing trivia of nodes, though, but all it does is take the first token and grab trivia from there. Having worked with Roslyn for the past seven months I'm actually very impressed of the overall architecture. We're using it to convert C# to Java and JavaScript, so we essentially convert one AST into another. I took a few design decisions from Roslyn and applied them to our own AST.
- krallja 11y ago
  Thanks, I appreciate the correction. My experience with Roslyn was very ad-hoc: just enough to do the job, see the sights, and get out of the compiler business :) It's interesting how many people are out there trying to switch from one language to another!

90 points by GarethX 11y ago

For those who missed it, discussion of part 1 is at [1]. [1] https://news.ycombinator.com/item?id=9777829 https://news.ycombinator.com/item?id=9777829

unstabilo 11y ago
[1] [1] Thank [2]! [2] you
- savanaly 11y ago
  Honestly, I found the way he used [1] to be easier to parse even though it's the only link and at the end of the sentence. I appreciated it.

johnchristopher 11y ago

For a split second I thought it was going to be about Nullsoft's Wasabi https://en.wikipedia.org/wiki/Wasabi_%28software%29 https://en.wikipedia.org/wiki/Wasabi_%28software%29

the-dude 11y ago
I wonder why you have been downvoted. I thought of the same. Upvoted.
- johnchristopher 11y ago
  Thanks. Someone might have `misclicked'.

anonymousDan 11y ago

Meta question, do transpilers actually work well in practice? My intuition is that they would have all sorts of limitations, but I've never had to use one. Could anyone point me to a resource describing what those limitations might be? In what circumstances would they work best?

dagw 11y ago
I use two transpilers in my day to day work, cython and coffeescript. In both cases they work really well and save me a bunch of time and effort. However in both cases the transpiler is written in parallel with the source language which always makes it a lot easier and the transpiler obviously supports 100% of the source language. I have also played with transpilers written for 'foreign' languages and then you do end up in a situation where you can only use a (often badly defined) subset of the source language, and debugging is a pain since the output is often very difficult to read. Still they're handy when you need them and can (sometimes) save you time compared to porting the code by hand.
Arnt 11y ago
Can't point to any resource... but my experience is that the main problems are a) parsing the input language and in the same way as other tools do and b) giving useful diagnostics and making the next-step compiler's and/or debugger's diagnostics useful. Far from trivial, but not insurmountable.
chii 11y ago
one of the most popular transpiler is http://haxe.org http://haxe.org - it compiles down to many different languages, and it works pretty well.
tomjen3 11y ago
Transpilers are just a new word for compilers - the major difference is that the target language is a high level one, but that just means that it is a lot easier to write the compiler (no optimizing register assignment!). I doubt a transpiler has any limitations that doesn't apply to compilers in general, though of course any particular one might.
jerf 11y ago
In the end, a "transpiler" is just a compiler and subject to the same limitations. There is no generic set of limitations that apply to all compilers. Some are very good because they do something relatively easy, like Coffeescript -> JavaScript. I don't mean that writing a great Coffeescript compiler [1] is necessarily "easy", but the Coffeescript language was tightly designed to map directly to Javascript constructs, so there's hardly any "limitations" at all; very nearly the entire target language is available to you, and to the extent that there may be JS you can't generate, it's probably JS the community generally agrees is a bad idea anyhow. Others, like gcc or LLVM, are actually bridging a fairly large gulf now between the source and target languages [2], but do a great job because of massive piles of effort by incredibly smart people. But if you did have to make generalizations, the two big ones are that generally you can have a performance problem if you're forced to compile something onto a target with a very different paradigm, and the target just doesn't let you have a low-enough level view of things to implement the source language's abstractions efficiently. For instance, consider the languages targeting the JVM before invokedynamic was added to the JVM... there was a fundamental paradigm mismatch that bled away a certain unavoidable amount of performance. The other problem you can get is that you may not get slick access to the low-level details of the target language; for instance, take a look at the piles of things that compile to JavaScript and compare how they allow you to use jQuery. It's anything from "What's the problem?" in CoffeeScript to being a full-on Foreign Function Interface-type call in GHCJS (even if you can get it pre-wrapped for you [3]), a Haskell->Javascript compiler. Both are really two aspects of the same "impedance mismatch" problem. [1]: Look, I'll be honest, I consider the term "transpiler" completely unnecessary. It's a compiler. Feel free to just keep using transpiler in your reply and I won't say anything again. I've had this argument, I bow to the trend, but this old fogey doesn't intend to change (and still considers it a serious impediment to understanding to believe they are two different things, despite using the exact same techniques, architecture, tools, mental models... grumble grumble). [2]: My recent previous comment goes into more detail on that claim: https://news.ycombinator.com/item?id=9800231 https://news.ycombinator.com/item?id=9800231 [3] https://github.com/ghcjs/ghcjs-jquery https://github.com/ghcjs/ghcjs-jquery

mwcampbell 11y ago

Why not open-source the whole compiler? Probably nobody else would use the whole thing as is, but there are surely other interesting bits, like the global type inference.

taprun 11y ago
Perhaps I misunderstand your question, but doesn't the article say the source is on github? https://github.com/FogCreek/RoslynGenerator https://github.com/FogCreek/RoslynGenerator
- pweissbrod 11y ago
  In github it states "The CLR importer, lexer, parser, interpreter, type checker, language runtime, JavaScript generator, and other components of Wasabi are missing."
krallja 11y ago
The type inference code would make the world a worse place. Whole careers would be ruined by new developers closing one eye and peeking at the last close brace.

vog 11y ago

Although slightly off-topic, I find the mentioned idea of "syntax trivia" very compelling. This should not only be part of code parsers, but of any parsers. I thought about something similar for binary parsers. Here, too, there are often many different ways to express a certain structure. Most libraries (such as Exiv2 for PNG/JPEG/EPS/etc. metadata) perform the following procedure: First, they parse out all information they are interested in. Then the application modifies that structure. Then it has to be embedded back into to binary format (here the image). The latter step is a nasty type of merging. They go over the file, try to find places where to "safely" embed or replace parts, then serialize their stuff and try to put it there. However, if the binary parser was lossless, all changes to the parsed structure would preverse the "trivia", so serializing is straight forward. And any potential issues could be handled on application level in the parsed structure, rather than by guesswork and heuristics during the merge phase. The only problem could be large BLOBs, but those could easily be represented as position & offset in the original file, rather than as the actual binary data in memory. (assuming the file will not change in the meantime, but in that case there the "merging" approach is very dangerous, too.)

Locke1689 11y ago
Syntax trivia is a very important component and a pretty natural fit once you start thinking about the feature that we decided was absolutely necessary: full-fidelity round-tripping between parsed syntax trees and the original source. I think this is one of the most common mistakes compiler writers make -- they don't spend enough time thinking about incorrect code. Let me be clear: most code is wrong. If your code is right, you compile it once. If your code is wrong, you compile it many times until you get it right. More importantly, if you run your IDE off your compiler, almost every character you type is a wrong program. Having a full-fidelity syntax tree is essential for having great experiences with wrong code. In addition, it easily solves the problem of having to serialize your trees -- the source text is the serialization. This feeds into API considerations as well. A number of people have repeatedly talked about how cool it would be to have a lot of other tools understand your AST (like if Git could support checking in ASTs instead of source). This is the wrong way of looking at it. If you're dealing with a raw AST you often have to have domain-specific knowledge of the language itself. Instead, what you want is to take the thing with the most domain specific knowledge, the compiler, and allow it to answer questions, i.e. have an API. By having round-trippable source, all source is essentially given a transparent API that can be used just as if you were interacting with the source code itself. Anyway, this is going off the rails, but it's one of the numerous things I'd point to for many production compilers and say, "you are the past, this is the future."

ygra 11y ago

One minor correction: Trivia are part of tokens, not nodes, which allows to represent things like white space between the if keyword and the opening parenthesis. There is a convenience method to get leading and trailing trivia of nodes, though, but all it does is take the first token and grab trivia from there. Having worked with Roslyn for the past seven months I'm actually very impressed of the overall architecture. We're using it to convert C# to Java and JavaScript, so we essentially convert one AST into another. I took a few design decisions from Roslyn and applied them to our own AST.

krallja 11y ago
Thanks, I appreciate the correction. My experience with Roslyn was very ad-hoc: just enough to do the job, see the sights, and get out of the compiler business :) It's interesting how many people are out there trying to switch from one language to another!