11 ms·

Rust: The New LLVM

207 points by wcrichton 10y ago

nickpsecurity 10y ago
I'd rather see things compile to typed ASM or something. SPARK + Rust if we're talking intermediate language so we get SPARK's static benefits on some code with Rust's on other code.
- favorited 10y ago
  Then you'd probably want LLVM IR (or an equivalent). It was literally designed to be a hardware-agnostic strongly-typed SSA assembly. And/or you could do what Rust and Swift are doing (with MIR and SIL, respectively), and define your own ASM-like language between your parsed AST and LLVM IR. That way you can do high-level optimizations, static analysis, etc. before you drop too far down and lose much context.
  - nickpsecurity 10y ago
    LLVM folks told me it has issues that stem from fact it was created originally for C/C++ and x86. I'd think something more like TALC or, as you said, strongly-typed SSA that was language and HW neutral. Plenty of research on such stuff though.
programminggeek 10y ago
How long till someone writes "Javascript: The New LLVM" or even better "Javascript: The New Rust"?
- nhaliday 10y ago
  Speaking of which, https://www.destroyallsoftware.com/talks/the-birth-and-death-of-javascript https://www.destroyallsoftware.com/talks/the-birth-and-death...
  - programminggeek 10y ago
    Great talk, Gary is a cool dude.
- scriptproof 10y ago
  JavaScript is a target for many programming languages already. Maybe the title could be WebAssembly: The New LLVM.
dbcurtis 10y ago
Debug information, anyone? The author ignores the need for passing line numbers and storage symbols through some sort of debug dictionary. The kindest thing I can say about this article is "simplistic".
- erickt 10y ago
  Rust Core Team member here. I actually wrote up an RFC [1] that proposes adding JavaScript-like source map support to Rust, in order to improve our plugins-in-stable-code supports. Odds are we are going to reject it for now since I think we found a simpler way to achieve our goals, but we are certainly not opposed to adding support for it if there's demand. [1]: https://github.com/rust-lang/rfcs/pull/1573 https://github.com/rust-lang/rfcs/pull/1573
  - dbcurtis 10y ago
    I only had time to give your link a quick skim, but I'm impressed that the Rust team is actually thinking about how to pass debug information through from "Rust-front" tools. The #line directive in C is limited, at best. Debugging flex/lex/yacc/bison, for instance, is sketchy at best. But of course line numbers are only part of the problem. If someone really wants to create Spiffy New Language as a Rust-front, then SNL structures/objects/collections/whoozits will want to pass symbolic information to the debugger so that users can explore their data structures using meaningful names. And even so, this assumes that with debug symbols on, you can turn optimization off. Debugging optimized code gets wacky very fast because associating line numbers to source is not always possible any more, and gives non-intuitive results due to code motion and inlining even when it is possible. (You haven't lived until you've seen gdb's source line pointer jump around like a weasel on crack due to code re-arrangement.) At Rust's current level of maturity, I suspect there are bigger fish to fry than passing debug info from Rust-fronts, so rejecting ambitious debug symbol machinery is probably the right strategic decision for the time being.
trishume 10y ago
Counterpoints as someone considering writing a language that targets LLVM, and as someone who has used Rust a bunch: - Your compile times will be a lot slower than they would otherwise be. This is a big one for me. - If your type system is materially different than Rust's, you don't gain any benefits since you have to write your own type checker anyways. - LLVM is designed to be easy to machine generate, including having a library API for generating code. Rust doesn't yet. - You can't necessarily re-use Cargo if your language requires type info to link that wouldn't be in the Rust source files generated. - If you have semantic differences in memory model you'll need to give up the advantage of Rust's safety by having your code generate unsafe pointers, for example if you want to have shared mutability in a single-threaded context, which Rust doesn't handle nicely. - If you do re-use Rust's type checker the error messages you get out will be terrible and point into generated Rust code. - Debug info and debugging get even harder. - If you want Rust interoperability you have to replicate Rust's enormous type system in your language's FFI, which is hard if you also have your own type checker, see above. Basically it is a good idea, if your language has semantics and type system very close to Rust. A much better idea IMO is to target LLVM and then write an LLVM plugin that can link to Golang object files and generate Golang struct metadata. Golang's FFI and type system are much much simpler and more likely to be able to graft onto new languages. Also Go's library ecosystem is larger.
- Hitard 10y ago
  Very new to Rust ecosystem, but I wondered, regarding this last point about using an LLVM plugin to generate structs, etc... would this also be a better way to do clean FFI to functions that use C-style unions? I have been struggling to see how the Rust bindings for Vulkan can be improved.
- jdmichal 10y ago
  > You can't necessarily re-use Cargo if your language requires type info to link that wouldn't be in the Rust source files generated. As a supporting point, TypeScript requires definition files (.d.ts) for JavaScript libraries to properly type them. Unless the language has a 1:1 correspondence to Rust types, something similar will probably need to happen.
- 10y ago
__s 10y ago
llvm has alloca. llvm has computed goto. Saying safe-Rust can generate arbitrary "safe" LLVM is a bold statement As an aside it may be interesting if this could be "Target MIR", but that's a future prospect
- drdrey 10y ago
  computed goto, aka indirectbr in the LLVM IR: http://llvm.org/docs/LangRef.html#indirectbr-instruction http://llvm.org/docs/LangRef.html#indirectbr-instruction
- wcrichton 10y ago
  Rust doesn't generate arbitrary LLVM, but to my knowledge it does generate LLVM that is memory safe. That's the whole point of the borrow checker.
azakai 10y ago
> This is part of the inspiration for LLVM, or Low Level Virtual Machine, which is a kind of "abstract assembly" that looks vaguely like machine code but is platform-independent. LLVM is not platform-independent. LLVM IR has baked-in constants, ABI details, and other platform-specific things. The "LLVM IR is a compiler IR" post has a good summary, http://comments.gmane.org/gmane.comp.compilers.llvm.devel/43769 http://comments.gmane.org/gmane.comp.compilers.llvm.devel/43... One result of that is that the following quote is not true: > a compiler writer can target LLVM and then have his language work across platforms (e.g. on Linux, Mac, and Windows machines) Actually a compiler must generate different LLVM IR for each platform. It is true that LLVM does abstract some or even most things away - CPU architecture is (mostly) handled for you, for example, which is what the author mentions - but many important things aren't. > as well as with other languages that use LLVM You'll still need to make sure to use the right ABIs for those languages. In fact, LLVM has somewhat poor abstraction for the C ABI, in that it exists in clang and not in LLVM, as a result, Rust has had to duplicate a bunch of that code. This point actually supports the author's position - targeting Rust would actually be a more portable target than LLVM! Overall, it might make sense to compile to Rust. But probably to Rust MIR, the IR they are developing. MIR needs to mature and stabilize, but there are already signs of an ecosystem starting to form around it. It's interesting speculation, but it would be exciting to see Rust MIR be "the new LLVM."
- ZenoArrow 10y ago
  Reading the article, I get the impression that the VM part of the LLVM name is causing some confusion. For all those reading this that may have thought LLVM acts like a low-level version of something like the JVM (as the name suggests, I got caught out with this before too), it's not the case. LLVM is a compiler framework, the main output of which isn't bytecode for a VM but is rather machine code for the platform you're running on. There is no performance overhead from running LLVM-generated code because LLVM is not part of the final runtime.
  - ternaryoperator 10y ago
    That's very helpful. Having only read about LLVM without actually using it, your note clarified confusion I had about it.
Animats 10y ago
This guy wants to use Rust as a target language for a compiler. He likes that Rust doesn't have too much run-time machinery, such as a GC or a JVM. OK, fine. Compilers have been written to target C for similar reasons. Then he wants a "battle hardened GC", a JVM, access to the internal syntax tree of the compiler, and enough dynamism to allow an interpretive environment. What? Those are all features that Rust deliberately doesn't have. Adding them would drastically change the language and add considerable complexity. They would not contribute to Rust's main goal - provide something in which to write reliable production software. From the feature list this guy is asking for, what he really wants is Microsoft's ".NET" system. That has all the run-time stuff he wants, and multiple compilers compile to it.
- Manishearth 10y ago
  The point, which he outlines in the post, is to be able to mix high level programming with lower level languages. This is already a common pattern in the industry where you write your application in python/ruby and speed up the tight loops with C/C++/Rust. The lingua franca of the interop is C. Which means that the boundary between the two is very stark and unsafe. You can't use advanced typesystems or seamlessly share datatypes. I'm working on something similar these days (C++ and Rust though, no high level language). You lose all template/generics info at the C bindings boundary. Bindgen programs let you get it back to some degree, but forcing monomorphization/etc are all not easy problems to solve without making the compilers talk to each other (also not easy to do). Will's idea seems to be to use Rust as a target for a high level GCd language, which can then (mostly) seamlessly interop with "pure" Rust code, and the interop won't have an unsafe boundary.
  - pjmlp 10y ago
    You should check how COM and now WinRT (aka 2nd coming of COM+ Runtime) work. You get an OO ABI, with support for generics and OS level interoperability and reference counting as the GC algorithm. Yes, using COM from C is a pain, but it was never intended to be used as such, rather C++, VB, Delphi and any other Windows language able to speak COM.
ilaksh 10y ago
Web assembly is the new LLVM.
- irascible 10y ago
  Came here for this comment.
- ridiculous_fish 10y ago
  Doesn't web assembly put most responsibility for optimizations on the thing emitting it? That's a big difference from LLVM.
  - flamedoge 10y ago
    webassembly is just a binary IL spec
    - ridiculous_fish 10y ago
      I don't know much about it. Is it like Java bytecode, where the runtime is expected to do substantial optimization? Or is it more like traditional assembly where the optimizations are expected to have already been done?
      - btrask 10y ago
        If you're still curious, I dug this up: https://news.ycombinator.com/item?id=9734842 https://news.ycombinator.com/item?id=9734842 It sounds like WebAssembly is in a weird position, where it can be (relatively) directly translated to machine code, or more heavily optimized to support dynamic languages.
      - flamedoge 10y ago
        It is just binary IL. Doesn't matter what it is expected to do. You can do anything to it.
- wcrichton 10y ago
  I don't think that's at odds with the post. Rust could compile to WebAssembly through LLVM.
tathougies 10y ago
Haskell functions (pure or with side effects) simply cannot be made into anything resembling standard Rust types due to the presence of laziness. This post oversimplifies the problem of language interoperability. Interop is not limited because of lack of a common target. Every platform/OS combination has a well-defined ABI. Most languages have facilities to use functions that conform to this ABI and most can also export functions which use the ABI. The choice of LLVM has nothing to do with it -- LLVM simply uses the ABI. The fact that many compilers use LLVM is not because a hand-rolled solution couldn't implement this ABI, but rather because LLVM is easy to use.
- wcrichton 10y ago
  I don't believe interop is that simple, though. If you want, for example, PHP to interact with Python, you can't just go to the ABI level. You need code to define the translation between objects of one type in PHP to another type in Python. ABI-level compatibility is good for really low level stuff, but for everything else it just doesn't work. As for Haskell, I agree that it doesn't fit into this language vision. Both its lazy semantics and type system would be difficult to fit into the model I suggest.
ridiculous_fish 10y ago
Here's a weird bit of C++ code: std::vector<int> foo = ...; for (iter = foo.begin(); iter != foo.end(); ++iter) { *iter += 1; foo[0] += *iter; } A naive translation to Rust would run afoul of the borrow checker. If we are to compile C++ to Rust, how are we to address this? Should we simply generate unsafe Rust?
- hellofunk 10y ago
  What do you consider weird about this code? This is fairly straightforward C++, as far as syntax and language use. Do you mean the logic of what it is doing seems weird to you?
  - ridiculous_fish 10y ago
    Yes, it's just accomplishing something weird. And it's straightforward C++ (as you say) that uses multiple mutable references to a single object. Of course Rust forbids this. How would one transpile this code to Rust? Or is the idea that every language ought to have Rust-style semantics? If so, it's hard to reconcile that with "Rust is the new LLVM," since LLVM does not demand such strict semantics.
- Eerie 10y ago
  let mut foo = vec!(1, 2, 3); for mut iter in 0..foo.len() { iter += 1; foo[0] += iter; } No unsafe required, I think.
  - ridiculous_fish 10y ago
    Points for cleverness - you replaced the iterator with a call to len(). Array indices are sometimes thought of as back-door references, because they let you cheat in this manner. But you didn't quite get away with it. The C++ code increments the value in the array through the iterator, while the above Rust increments the iterator (err, index) itself. So it doesn't produce the same result. More generally, std::vector in C++ typically implements iterators with just raw pointers. This is quite unsafe, but doesn't impose borrow-checker restrictions on the user. How can such languages be compiled to Rust?
sid-kap 10y ago
Minor nit: GHC has a LLVM backend (in addition to an assembly backend and a C backend), but the assembly backend is the most commonly used. (The LLVM backend is known to produce better-optimized code, but the compile times are slower.)
kibwen 10y ago
As a long-time Rust guy, this article makes little sense to me. It is surely possible for any language to target Rust, but unless the semantics of that language are an unusually close match to Rust's then you're going to have to pull off some gymnastics. LLVM IR is actually designed to be a decent compilation target for a wider variety of languages (though I wish there was more documentation on what is and is not UB), but even LLVM IR is going to start looking like a poor choice of a target the farther your language is from C++.
- drdrey 10y ago
  UB as in undefined behavior?
  - throwaway40483 10y ago
    Yes
- wcrichton 10y ago
  Can you give an example of when language semantics are so different from Rust that it would be difficult to do this kind of thing? When I wrote this article, I was thinking major languages: Javascript, Java, Matlab, etc. and to my mind they all are suitable for compiling to Rust. Granted, I don't know the Java specification or its internals, I'm not claiming right now that everything specified by the JVM can definitely get ported to Rust, but moreso the general ideas behind the languages.
  - archgoon 10y ago
    Javascript has very different semantics from rust. It's not compiled for one thing, it's untyped, and it's executed by a dynamic interpreter that, in the case of v8, directly compiles hot code to native assembly.
    - wcrichton 10y ago
      Counterpoint: I'm writing a language to do just that [1]. A lot of the basis from this post comes from my experiment in writing Javascript that compiles to Rust! [1] https://github.com/willcrichton/lia https://github.com/willcrichton/lia
wilonth 10y ago
Such a bad idea, fanboyism is too strong here, Rust is one of the slowest languages to compile, and incremental compilation isn't even available yet. Your new language's compilation will take pretty much forever.
- wcrichton 10y ago
  Is your main objection just compilation times? I agree that's a trouble point, although I don't believe it will never get fixed. Incremental compilation is just around the corner! I assure you that this post is thought out past the point of just "fanboyism." Have some faith :-)
- steveklabnik 10y ago
  A PR for the first bit of incremental compilation was opened recently.
webkike 10y ago
This sounds like it is written by a person with little experience with compilers. The reason you compile to SSA is because it's easy to perform optimizations on. At some point you compile Rust to LLVM.
- wcrichton 10y ago
  Yes, and Rust does the hard work of turning your code into SSA and activating all those optimizations. Why should we have to recreate all the effort of the Rust compiler devs? And please refrain from ad hominem. It adds little to the conversation.
  - webkike 10y ago
    As others have pointed out, the difference is a sacrifice in compilation time to more quickly implement a langauge very semantically similar to rust. If it's not super similar to rust, you are simply jumping through more hoops than it is worth. Yes, LLVM IR is a pretty decent compilation target. And honestly, having costly abstractions provided to you would probably encourage you to generate poorer code. But I digress.
jeffdavis 10y ago
Really creative ideas in here. Others have raised a lot of good questions, but it seems like an interesting area to explore, anyway.
deleted 10y ago
[deleted]
zodiac 10y ago
> If you compile to Python, well, your program will probably run pretty slowly and you lose any static typing guarantees. What? The static typing guarantees should be checked (statically!) by the compiler, so the choice of language you emit doesn't need to affect what kind of type system you can design your input language to accept.
erlend_sh 10y ago
> To my knowledge, Lia is the first major language that actually compiles directly to Rust. You might be interested in checking out Dyon https://github.com/PistonDevelopers/dyon https://github.com/PistonDevelopers/dyon
andreaferretti 10y ago
In fact, as a compilation target, one wants a language with a lax semantics, not a strict one. This is (one of the reasons) why so many languages compile to C, instead of - say - to C++. Consider this fact for example: many languages make use of objects allocated on the heap in a way that prevents to know statically when they are to be disposed; then a GC takes care of that dinamically. How are they going to give Rust the necessary information to track lifetimes, when the information is not there to start with? Of course, one can just write a GC in Rust, but how is this an improvement with respect to a GC written in C? Also - what about typed languages - such as Go - that do not have generics? These are not going to map well to the Rust type system. Or what about lazy languages? Or languages that save the stack on the heap to implement call/cc? In short, whenever the semantics is different enough, you will not be able to map to idiomatic Rust constructs, and in fact if you don't want to pay the cost of interpretation, you will probably map to unsafe constructs most of the time.
- Manishearth 10y ago
  Rust most probably will be getting hooks that make it possible to safely integrate efficient GCs. Writing a safe nonconservative GC for rust has already been done, and once these hooks are in place it should be possible to write a good GC. Though for a codegenned language on top of rust you may not need these hooks. https://github.com/mystor/rust-callcc https://github.com/mystor/rust-callcc is callcc for Rust. Technically abuses a feature for a purpose it was explicitly not intended for, but meh :) callcc and gc are usually not useful operations in rust. That doesn't mean that its not possible to implement them. The implementation might be annoying to use, but of you are codegenning to rust this doesn't matter. Yes, some semantics won't copy over. I feel like most would though. Those that don't can often be emulated at the lower level with enough codegen.
  - andreaferretti 10y ago
    It's not that writing a GC is impossible - of course it is doable. The issue is that objects written in this hypothetical language will always make use of the GC because they do not have lifetime information in the first place. So the question arises naturally: why bother compiling to Rust if you have to avoid the borrow checker anyway?
vanderZwan 10y ago
Once static compilation gets off the ground, I wonder if Julia isn't an easier target for "high level LLVM". The whole language is built to be extensible, and people are already using it in comparable ways. APL implementation: https://www.youtube.com/watch?v=XVv1GipR5yU https://www.youtube.com/watch?v=XVv1GipR5yU As a code generator for the Go assembler: https://www.youtube.com/watch?v=02NkiDoRDCU https://www.youtube.com/watch?v=02NkiDoRDCU
bogomipz 10y ago
"CPUs are basically assembly interpreters, and that's the most fundamental unit of execution that we can target" I'm not trying to nitpick here but CPUs are interpreters of binary words. I think that sentence might be misleading to someone.
bogomipz 10y ago
Can someone explain this: "Second, LLVM is a simpler language to target than traditional assembly languages—it has nice features like infinite registers and recently support for stack frames." Aren't stack frames a basic component of any run time? How was support for those only recently added? Maybe I am not fully understanding the context. I looked at this docs but it didn't clear it up for me.
solidsnack9000 10y ago
Rust does offer some type features that could make language interop a lot better while not really imposing too much on the way other languages work. Traits in particular provide a way to model polymorphism that is compatible with subtypting but doesn't require it.
sitkack 10y ago
For all the corner case clarences out there, this is some forward thinking stuff that you should grok before you turn it down.

207 points by wcrichton 10y ago

I'd rather see things compile to typed ASM or something. SPARK + Rust if we're talking intermediate language so we get SPARK's static benefits on some code with Rust's on other code.

favorited 10y ago
Then you'd probably want LLVM IR (or an equivalent). It was literally designed to be a hardware-agnostic strongly-typed SSA assembly. And/or you could do what Rust and Swift are doing (with MIR and SIL, respectively), and define your own ASM-like language between your parsed AST and LLVM IR. That way you can do high-level optimizations, static analysis, etc. before you drop too far down and lose much context.
- nickpsecurity 10y ago
  LLVM folks told me it has issues that stem from fact it was created originally for C/C++ and x86. I'd think something more like TALC or, as you said, strongly-typed SSA that was language and HW neutral. Plenty of research on such stuff though.

programminggeek 10y ago

How long till someone writes "Javascript: The New LLVM" or even better "Javascript: The New Rust"?

nhaliday 10y ago
Speaking of which, https://www.destroyallsoftware.com/talks/the-birth-and-death-of-javascript https://www.destroyallsoftware.com/talks/the-birth-and-death...
- programminggeek 10y ago
  Great talk, Gary is a cool dude.
scriptproof 10y ago
JavaScript is a target for many programming languages already. Maybe the title could be WebAssembly: The New LLVM.

dbcurtis 10y ago

Debug information, anyone? The author ignores the need for passing line numbers and storage symbols through some sort of debug dictionary. The kindest thing I can say about this article is "simplistic".

erickt 10y ago
Rust Core Team member here. I actually wrote up an RFC [1] that proposes adding JavaScript-like source map support to Rust, in order to improve our plugins-in-stable-code supports. Odds are we are going to reject it for now since I think we found a simpler way to achieve our goals, but we are certainly not opposed to adding support for it if there's demand. [1]: https://github.com/rust-lang/rfcs/pull/1573 https://github.com/rust-lang/rfcs/pull/1573
- dbcurtis 10y ago
  I only had time to give your link a quick skim, but I'm impressed that the Rust team is actually thinking about how to pass debug information through from "Rust-front" tools. The #line directive in C is limited, at best. Debugging flex/lex/yacc/bison, for instance, is sketchy at best. But of course line numbers are only part of the problem. If someone really wants to create Spiffy New Language as a Rust-front, then SNL structures/objects/collections/whoozits will want to pass symbolic information to the debugger so that users can explore their data structures using meaningful names. And even so, this assumes that with debug symbols on, you can turn optimization off. Debugging optimized code gets wacky very fast because associating line numbers to source is not always possible any more, and gives non-intuitive results due to code motion and inlining even when it is possible. (You haven't lived until you've seen gdb's source line pointer jump around like a weasel on crack due to code re-arrangement.) At Rust's current level of maturity, I suspect there are bigger fish to fry than passing debug info from Rust-fronts, so rejecting ambitious debug symbol machinery is probably the right strategic decision for the time being.

trishume 10y ago

Counterpoints as someone considering writing a language that targets LLVM, and as someone who has used Rust a bunch: - Your compile times will be a lot slower than they would otherwise be. This is a big one for me. - If your type system is materially different than Rust's, you don't gain any benefits since you have to write your own type checker anyways. - LLVM is designed to be easy to machine generate, including having a library API for generating code. Rust doesn't yet. - You can't necessarily re-use Cargo if your language requires type info to link that wouldn't be in the Rust source files generated. - If you have semantic differences in memory model you'll need to give up the advantage of Rust's safety by having your code generate unsafe pointers, for example if you want to have shared mutability in a single-threaded context, which Rust doesn't handle nicely. - If you do re-use Rust's type checker the error messages you get out will be terrible and point into generated Rust code. - Debug info and debugging get even harder. - If you want Rust interoperability you have to replicate Rust's enormous type system in your language's FFI, which is hard if you also have your own type checker, see above. Basically it is a good idea, if your language has semantics and type system very close to Rust. A much better idea IMO is to target LLVM and then write an LLVM plugin that can link to Golang object files and generate Golang struct metadata. Golang's FFI and type system are much much simpler and more likely to be able to graft onto new languages. Also Go's library ecosystem is larger.

Hitard 10y ago
Very new to Rust ecosystem, but I wondered, regarding this last point about using an LLVM plugin to generate structs, etc... would this also be a better way to do clean FFI to functions that use C-style unions? I have been struggling to see how the Rust bindings for Vulkan can be improved.
jdmichal 10y ago
> You can't necessarily re-use Cargo if your language requires type info to link that wouldn't be in the Rust source files generated. As a supporting point, TypeScript requires definition files (.d.ts) for JavaScript libraries to properly type them. Unless the language has a 1:1 correspondence to Rust types, something similar will probably need to happen.
10y ago

__s 10y ago

llvm has alloca. llvm has computed goto. Saying safe-Rust can generate arbitrary "safe" LLVM is a bold statement As an aside it may be interesting if this could be "Target MIR", but that's a future prospect

drdrey 10y ago
computed goto, aka indirectbr in the LLVM IR: http://llvm.org/docs/LangRef.html#indirectbr-instruction http://llvm.org/docs/LangRef.html#indirectbr-instruction
wcrichton 10y ago
Rust doesn't generate arbitrary LLVM, but to my knowledge it does generate LLVM that is memory safe. That's the whole point of the borrow checker.

azakai 10y ago

> This is part of the inspiration for LLVM, or Low Level Virtual Machine, which is a kind of "abstract assembly" that looks vaguely like machine code but is platform-independent. LLVM is not platform-independent. LLVM IR has baked-in constants, ABI details, and other platform-specific things. The "LLVM IR is a compiler IR" post has a good summary, http://comments.gmane.org/gmane.comp.compilers.llvm.devel/43769 http://comments.gmane.org/gmane.comp.compilers.llvm.devel/43... One result of that is that the following quote is not true: > a compiler writer can target LLVM and then have his language work across platforms (e.g. on Linux, Mac, and Windows machines) Actually a compiler must generate different LLVM IR for each platform. It is true that LLVM does abstract some or even most things away - CPU architecture is (mostly) handled for you, for example, which is what the author mentions - but many important things aren't. > as well as with other languages that use LLVM You'll still need to make sure to use the right ABIs for those languages. In fact, LLVM has somewhat poor abstraction for the C ABI, in that it exists in clang and not in LLVM, as a result, Rust has had to duplicate a bunch of that code. This point actually supports the author's position - targeting Rust would actually be a more portable target than LLVM! Overall, it might make sense to compile to Rust. But probably to Rust MIR, the IR they are developing. MIR needs to mature and stabilize, but there are already signs of an ecosystem starting to form around it. It's interesting speculation, but it would be exciting to see Rust MIR be "the new LLVM."

ZenoArrow 10y ago
Reading the article, I get the impression that the VM part of the LLVM name is causing some confusion. For all those reading this that may have thought LLVM acts like a low-level version of something like the JVM (as the name suggests, I got caught out with this before too), it's not the case. LLVM is a compiler framework, the main output of which isn't bytecode for a VM but is rather machine code for the platform you're running on. There is no performance overhead from running LLVM-generated code because LLVM is not part of the final runtime.
- ternaryoperator 10y ago
  That's very helpful. Having only read about LLVM without actually using it, your note clarified confusion I had about it.

Animats 10y ago

This guy wants to use Rust as a target language for a compiler. He likes that Rust doesn't have too much run-time machinery, such as a GC or a JVM. OK, fine. Compilers have been written to target C for similar reasons. Then he wants a "battle hardened GC", a JVM, access to the internal syntax tree of the compiler, and enough dynamism to allow an interpretive environment. What? Those are all features that Rust deliberately doesn't have. Adding them would drastically change the language and add considerable complexity. They would not contribute to Rust's main goal - provide something in which to write reliable production software. From the feature list this guy is asking for, what he really wants is Microsoft's ".NET" system. That has all the run-time stuff he wants, and multiple compilers compile to it.

Manishearth 10y ago
The point, which he outlines in the post, is to be able to mix high level programming with lower level languages. This is already a common pattern in the industry where you write your application in python/ruby and speed up the tight loops with C/C++/Rust. The lingua franca of the interop is C. Which means that the boundary between the two is very stark and unsafe. You can't use advanced typesystems or seamlessly share datatypes. I'm working on something similar these days (C++ and Rust though, no high level language). You lose all template/generics info at the C bindings boundary. Bindgen programs let you get it back to some degree, but forcing monomorphization/etc are all not easy problems to solve without making the compilers talk to each other (also not easy to do). Will's idea seems to be to use Rust as a target for a high level GCd language, which can then (mostly) seamlessly interop with "pure" Rust code, and the interop won't have an unsafe boundary.
- pjmlp 10y ago
  You should check how COM and now WinRT (aka 2nd coming of COM+ Runtime) work. You get an OO ABI, with support for generics and OS level interoperability and reference counting as the GC algorithm. Yes, using COM from C is a pain, but it was never intended to be used as such, rather C++, VB, Delphi and any other Windows language able to speak COM.

ilaksh 10y ago

Web assembly is the new LLVM.

irascible 10y ago
Came here for this comment.
ridiculous_fish 10y ago
Doesn't web assembly put most responsibility for optimizations on the thing emitting it? That's a big difference from LLVM.
- flamedoge 10y ago
  webassembly is just a binary IL spec
  - ridiculous_fish 10y ago
    I don't know much about it. Is it like Java bytecode, where the runtime is expected to do substantial optimization? Or is it more like traditional assembly where the optimizations are expected to have already been done?
    - btrask 10y ago
      If you're still curious, I dug this up: https://news.ycombinator.com/item?id=9734842 https://news.ycombinator.com/item?id=9734842 It sounds like WebAssembly is in a weird position, where it can be (relatively) directly translated to machine code, or more heavily optimized to support dynamic languages.
    - flamedoge 10y ago
      It is just binary IL. Doesn't matter what it is expected to do. You can do anything to it.
wcrichton 10y ago
I don't think that's at odds with the post. Rust could compile to WebAssembly through LLVM.

tathougies 10y ago

Haskell functions (pure or with side effects) simply cannot be made into anything resembling standard Rust types due to the presence of laziness. This post oversimplifies the problem of language interoperability. Interop is not limited because of lack of a common target. Every platform/OS combination has a well-defined ABI. Most languages have facilities to use functions that conform to this ABI and most can also export functions which use the ABI. The choice of LLVM has nothing to do with it -- LLVM simply uses the ABI. The fact that many compilers use LLVM is not because a hand-rolled solution couldn't implement this ABI, but rather because LLVM is easy to use.

wcrichton 10y ago
I don't believe interop is that simple, though. If you want, for example, PHP to interact with Python, you can't just go to the ABI level. You need code to define the translation between objects of one type in PHP to another type in Python. ABI-level compatibility is good for really low level stuff, but for everything else it just doesn't work. As for Haskell, I agree that it doesn't fit into this language vision. Both its lazy semantics and type system would be difficult to fit into the model I suggest.

ridiculous_fish 10y ago

Here's a weird bit of C++ code: std::vector<int> foo = ...; for (iter = foo.begin(); iter != foo.end(); ++iter) { *iter += 1; foo[0] += *iter; } A naive translation to Rust would run afoul of the borrow checker. If we are to compile C++ to Rust, how are we to address this? Should we simply generate unsafe Rust?

hellofunk 10y ago
What do you consider weird about this code? This is fairly straightforward C++, as far as syntax and language use. Do you mean the logic of what it is doing seems weird to you?
- ridiculous_fish 10y ago
  Yes, it's just accomplishing something weird. And it's straightforward C++ (as you say) that uses multiple mutable references to a single object. Of course Rust forbids this. How would one transpile this code to Rust? Or is the idea that every language ought to have Rust-style semantics? If so, it's hard to reconcile that with "Rust is the new LLVM," since LLVM does not demand such strict semantics.
Eerie 10y ago
let mut foo = vec!(1, 2, 3); for mut iter in 0..foo.len() { iter += 1; foo[0] += iter; } No unsafe required, I think.
- ridiculous_fish 10y ago
  Points for cleverness - you replaced the iterator with a call to len(). Array indices are sometimes thought of as back-door references, because they let you cheat in this manner. But you didn't quite get away with it. The C++ code increments the value in the array through the iterator, while the above Rust increments the iterator (err, index) itself. So it doesn't produce the same result. More generally, std::vector in C++ typically implements iterators with just raw pointers. This is quite unsafe, but doesn't impose borrow-checker restrictions on the user. How can such languages be compiled to Rust?

sid-kap 10y ago

Minor nit: GHC has a LLVM backend (in addition to an assembly backend and a C backend), but the assembly backend is the most commonly used. (The LLVM backend is known to produce better-optimized code, but the compile times are slower.)

kibwen 10y ago

As a long-time Rust guy, this article makes little sense to me. It is surely possible for any language to target Rust, but unless the semantics of that language are an unusually close match to Rust's then you're going to have to pull off some gymnastics. LLVM IR is actually designed to be a decent compilation target for a wider variety of languages (though I wish there was more documentation on what is and is not UB), but even LLVM IR is going to start looking like a poor choice of a target the farther your language is from C++.

drdrey 10y ago
UB as in undefined behavior?
- throwaway40483 10y ago
  Yes
wcrichton 10y ago
Can you give an example of when language semantics are so different from Rust that it would be difficult to do this kind of thing? When I wrote this article, I was thinking major languages: Javascript, Java, Matlab, etc. and to my mind they all are suitable for compiling to Rust. Granted, I don't know the Java specification or its internals, I'm not claiming right now that everything specified by the JVM can definitely get ported to Rust, but moreso the general ideas behind the languages.
- archgoon 10y ago
  Javascript has very different semantics from rust. It's not compiled for one thing, it's untyped, and it's executed by a dynamic interpreter that, in the case of v8, directly compiles hot code to native assembly.
  - wcrichton 10y ago
    Counterpoint: I'm writing a language to do just that [1]. A lot of the basis from this post comes from my experiment in writing Javascript that compiles to Rust! [1] https://github.com/willcrichton/lia https://github.com/willcrichton/lia

wilonth 10y ago

Such a bad idea, fanboyism is too strong here, Rust is one of the slowest languages to compile, and incremental compilation isn't even available yet. Your new language's compilation will take pretty much forever.

wcrichton 10y ago
Is your main objection just compilation times? I agree that's a trouble point, although I don't believe it will never get fixed. Incremental compilation is just around the corner! I assure you that this post is thought out past the point of just "fanboyism." Have some faith :-)
steveklabnik 10y ago
A PR for the first bit of incremental compilation was opened recently.

webkike 10y ago

This sounds like it is written by a person with little experience with compilers. The reason you compile to SSA is because it's easy to perform optimizations on. At some point you compile Rust to LLVM.

wcrichton 10y ago
Yes, and Rust does the hard work of turning your code into SSA and activating all those optimizations. Why should we have to recreate all the effort of the Rust compiler devs? And please refrain from ad hominem. It adds little to the conversation.
- webkike 10y ago
  As others have pointed out, the difference is a sacrifice in compilation time to more quickly implement a langauge very semantically similar to rust. If it's not super similar to rust, you are simply jumping through more hoops than it is worth. Yes, LLVM IR is a pretty decent compilation target. And honestly, having costly abstractions provided to you would probably encourage you to generate poorer code. But I digress.

jeffdavis 10y ago

Really creative ideas in here. Others have raised a lot of good questions, but it seems like an interesting area to explore, anyway.

deleted 10y ago

[deleted]

zodiac 10y ago

> If you compile to Python, well, your program will probably run pretty slowly and you lose any static typing guarantees. What? The static typing guarantees should be checked (statically!) by the compiler, so the choice of language you emit doesn't need to affect what kind of type system you can design your input language to accept.

erlend_sh 10y ago

> To my knowledge, Lia is the first major language that actually compiles directly to Rust. You might be interested in checking out Dyon https://github.com/PistonDevelopers/dyon https://github.com/PistonDevelopers/dyon

andreaferretti 10y ago

In fact, as a compilation target, one wants a language with a lax semantics, not a strict one. This is (one of the reasons) why so many languages compile to C, instead of - say - to C++. Consider this fact for example: many languages make use of objects allocated on the heap in a way that prevents to know statically when they are to be disposed; then a GC takes care of that dinamically. How are they going to give Rust the necessary information to track lifetimes, when the information is not there to start with? Of course, one can just write a GC in Rust, but how is this an improvement with respect to a GC written in C? Also - what about typed languages - such as Go - that do not have generics? These are not going to map well to the Rust type system. Or what about lazy languages? Or languages that save the stack on the heap to implement call/cc? In short, whenever the semantics is different enough, you will not be able to map to idiomatic Rust constructs, and in fact if you don't want to pay the cost of interpretation, you will probably map to unsafe constructs most of the time.

Manishearth 10y ago
Rust most probably will be getting hooks that make it possible to safely integrate efficient GCs. Writing a safe nonconservative GC for rust has already been done, and once these hooks are in place it should be possible to write a good GC. Though for a codegenned language on top of rust you may not need these hooks. https://github.com/mystor/rust-callcc https://github.com/mystor/rust-callcc is callcc for Rust. Technically abuses a feature for a purpose it was explicitly not intended for, but meh :) callcc and gc are usually not useful operations in rust. That doesn't mean that its not possible to implement them. The implementation might be annoying to use, but of you are codegenning to rust this doesn't matter. Yes, some semantics won't copy over. I feel like most would though. Those that don't can often be emulated at the lower level with enough codegen.
- andreaferretti 10y ago
  It's not that writing a GC is impossible - of course it is doable. The issue is that objects written in this hypothetical language will always make use of the GC because they do not have lifetime information in the first place. So the question arises naturally: why bother compiling to Rust if you have to avoid the borrow checker anyway?

vanderZwan 10y ago

Once static compilation gets off the ground, I wonder if Julia isn't an easier target for "high level LLVM". The whole language is built to be extensible, and people are already using it in comparable ways. APL implementation: https://www.youtube.com/watch?v=XVv1GipR5yU https://www.youtube.com/watch?v=XVv1GipR5yU As a code generator for the Go assembler: https://www.youtube.com/watch?v=02NkiDoRDCU https://www.youtube.com/watch?v=02NkiDoRDCU

bogomipz 10y ago

"CPUs are basically assembly interpreters, and that's the most fundamental unit of execution that we can target" I'm not trying to nitpick here but CPUs are interpreters of binary words. I think that sentence might be misleading to someone.

bogomipz 10y ago

Can someone explain this: "Second, LLVM is a simpler language to target than traditional assembly languages—it has nice features like infinite registers and recently support for stack frames." Aren't stack frames a basic component of any run time? How was support for those only recently added? Maybe I am not fully understanding the context. I looked at this docs but it didn't clear it up for me.

solidsnack9000 10y ago

Rust does offer some type features that could make language interop a lot better while not really imposing too much on the way other languages work. Traits in particular provide a way to model polymorphism that is compatible with subtypting but doesn't require it.

sitkack 10y ago

For all the corner case clarences out there, this is some forward thinking stuff that you should grok before you turn it down.