3 ms·

This is actually a pretty good step in the right direction. Big question I have: can I just import (like I'd import anything else in a code base) the dependenc

This is actually a pretty good step in the right direction. Big question I have: can I just import (like I'd import anything else in a code base) the dependency and it'll "see" the exposed API surface of the module? This looks like it does some eval of the underlying code depending on what it is, but it must be explicitly mapped by the consumer (declaring the language and the code), which I respect wholly, but its a little different from the "ease of use" perspective. That said, its another promising approach, always interested to see how this matures. To answer the second half of this, its not about defining how Python Dict should map to a Java Hashmap. Its about how Java or Python or whatever language maps to WASM. As long as the produced output is valid WASM, it will work across any boundary that supports WASM. Inherently, it devoids itself from having to think about any specific language this way. Right now this also produces limitations (because features have to be specified uniformly so the target of compilation is the same for everyone) The inherent flaw of most multi-compat runtime systems, in my opinion, is making the consumer think about the underlying language. Rather, if you use Go to produce a WASM module, it can be used in Rust, or JavaScript, or C / C++ etc. without considering how it was written

mike_hearn 3y ago
Yes, depending on what you mean by "see". For a statically typed language using imports from a dynamically typed language, you'll need some interfaces and stuff to cast values to. But that's inherent. The C compiler doesn't have any concept of a dynamic type beyond void* so you'll always need to cast it to something more specific before using it. For the other way around that doesn't apply and you just use whatever you import. You don't technically have to specify the language. The file extension is usually enough to select the right language. I'm not sure how it gets easier, unless you are asking about a sort of polyglot package manager or build system. Indeed there isn't one of those, because they're usually language specific. It would be a good improvement to have though. Right now you'd need to use npm to get JS code, pip to get Python code etc. With respect to language mappings, I don't think WASM solves this. There is no mapping to WASM of a hashmap because the WASM type system doesn't have any such concept. Could you elaborate on what you're referring to here? As the name implies, it's a kind of assembly language, so it offers no more assistance in this task than ARM or x86 does. Moreover the approach of defining a universal "language" and mapping everything to it was tried and didn't work so great, that's sort of the classical JVM or CLR approach. Languages disagree on how core concepts work. For example a JS array is a very different thing to an array in C, a Java HashMap has a different API and behavior to a Python dict which in turn is different to the same concept in Scheme. You can make some progress by trying to compile everything through to the One True HashMap, but performance and edge cases at the boundaries become problems. The Truffle insight is to let each language be its own thing. Instead of having One True HashMap (or whatever), you let each language work with its own concepts and memory layouts then define a standard way to export basic operations like "read member", "invoke function", "access key" in the form of compiler IR snippets. So when you invoke polyglot_get_array_element on a Ruby array it's not really doing a function call, and there is no shared array type that every language has to compile to. Instead the function call is replaced with the actual machine code that the Ruby engine would use to look up an array element, and the result is stored in a C void* but it's not a pointer to some unified object concept or even a real pointer at all. That's why you can do something like casting it to a struct of function pointers, and get something that will then directly invoke Ruby methods. The cast operation is also being virtualized. This approach sidesteps the question of how to share stuff between languages without trying to compile it all to a generic meta-language. The operations are called the "interop protocol" which is a bit confusing because it's not actually a network protocol of any kind. It defines lots of high level mapping operations so you can do things like work with a time/date from a guest language using your host language's time/dat types. https://www.graalvm.org/truffle/javadoc/com/oracle/truffle/api/interop/InteropLibrary.html https://www.graalvm.org/truffle/javadoc/com/oracle/truffle/a... It's important to note here that the exported operations aren't target language specific. Once you implement the interop protocol for a Truffle language it works for every other language, there's no N:N binding work required. So it's a scalable approach.
- no_wizard 3y ago
  WASM is a specification. So when you compile to it, you compile to the specification. Any language runtime (be it JavaScript via V8, Go, or whatever) or language interop via compilation (Rust, C, C++) can interact with a WASM binary if their toolchain supports understanding it. Thats the key here. I can take my JS, compile it to WASM, and Rust can use it as a module because its compiler understands WASM (in simple terms). This all requires a WASM runtime of course, no doubting that. However, the mere fact I can take two languages, compile one to WASM and use that as the module in that other language, is what I'm talking about. And WASM compat is taking off pretty well so far. I admire Truffle's approach, it might prove to be long term better in solving this problem, because languages can move at the speed they want to move, and the interop story doesn't really change. Where as with WASM, you have to keep up with the WASM specification and runtimes, which could be a headache if not done in a community oriented, deliberate way. All that is to say, its up the language toolchain to deal with compiling that language to WASM, but once its WASM, its portable
  - mike_hearn 3y ago
    I'd be curious to see a demonstration codebase where JS or other dynamic languages are compiled to WASM and then loaded into something else with automatic FFI at the module level. Because I think you'd still need to work with v8::Value objects or an equivalent to use JS from C++, WASM or not. WASM isn't meant to be a compilation target for languages like JS, what dynamic languages need from a compiler is very different to what Rust needs.
    - no_wizard 3y ago
      To be clear, I'm talking about the following: I can write some code or use modules written in JavaScript that compile to WASM. I can use that compiled output with any runtime that can import WASM and expose its interface natively. To that end, I could use it in Rust, no problem. The same is in reverse. I can write a Rust program, compile to WASM, and use it in JS runtime (lots of projects do this already, actually. esbuild does this with Go as well). Right Now there are of course limitations, however if current trajectory holds, you'll be able to just import wasm directly into a compat runtime. So I can take the JS to WASM output and share it to any language to can understand it. This is, as I understand it, the advantage of WASM in the future tense (and if you use JavaScript, its kind of present tense, esp. with import assertions)

mike_hearn 3y ago

Yes, depending on what you mean by "see". For a statically typed language using imports from a dynamically typed language, you'll need some interfaces and stuff to cast values to. But that's inherent. The C compiler doesn't have any concept of a dynamic type beyond void* so you'll always need to cast it to something more specific before using it. For the other way around that doesn't apply and you just use whatever you import. You don't technically have to specify the language. The file extension is usually enough to select the right language. I'm not sure how it gets easier, unless you are asking about a sort of polyglot package manager or build system. Indeed there isn't one of those, because they're usually language specific. It would be a good improvement to have though. Right now you'd need to use npm to get JS code, pip to get Python code etc. With respect to language mappings, I don't think WASM solves this. There is no mapping to WASM of a hashmap because the WASM type system doesn't have any such concept. Could you elaborate on what you're referring to here? As the name implies, it's a kind of assembly language, so it offers no more assistance in this task than ARM or x86 does. Moreover the approach of defining a universal "language" and mapping everything to it was tried and didn't work so great, that's sort of the classical JVM or CLR approach. Languages disagree on how core concepts work. For example a JS array is a very different thing to an array in C, a Java HashMap has a different API and behavior to a Python dict which in turn is different to the same concept in Scheme. You can make some progress by trying to compile everything through to the One True HashMap, but performance and edge cases at the boundaries become problems. The Truffle insight is to let each language be its own thing. Instead of having One True HashMap (or whatever), you let each language work with its own concepts and memory layouts then define a standard way to export basic operations like "read member", "invoke function", "access key" in the form of compiler IR snippets. So when you invoke polyglot_get_array_element on a Ruby array it's not really doing a function call, and there is no shared array type that every language has to compile to. Instead the function call is replaced with the actual machine code that the Ruby engine would use to look up an array element, and the result is stored in a C void* but it's not a pointer to some unified object concept or even a real pointer at all. That's why you can do something like casting it to a struct of function pointers, and get something that will then directly invoke Ruby methods. The cast operation is also being virtualized. This approach sidesteps the question of how to share stuff between languages without trying to compile it all to a generic meta-language. The operations are called the "interop protocol" which is a bit confusing because it's not actually a network protocol of any kind. It defines lots of high level mapping operations so you can do things like work with a time/date from a guest language using your host language's time/dat types. https://www.graalvm.org/truffle/javadoc/com/oracle/truffle/api/interop/InteropLibrary.html https://www.graalvm.org/truffle/javadoc/com/oracle/truffle/a... It's important to note here that the exported operations aren't target language specific. Once you implement the interop protocol for a Truffle language it works for every other language, there's no N:N binding work required. So it's a scalable approach.

no_wizard 3y ago
WASM is a specification. So when you compile to it, you compile to the specification. Any language runtime (be it JavaScript via V8, Go, or whatever) or language interop via compilation (Rust, C, C++) can interact with a WASM binary if their toolchain supports understanding it. Thats the key here. I can take my JS, compile it to WASM, and Rust can use it as a module because its compiler understands WASM (in simple terms). This all requires a WASM runtime of course, no doubting that. However, the mere fact I can take two languages, compile one to WASM and use that as the module in that other language, is what I'm talking about. And WASM compat is taking off pretty well so far. I admire Truffle's approach, it might prove to be long term better in solving this problem, because languages can move at the speed they want to move, and the interop story doesn't really change. Where as with WASM, you have to keep up with the WASM specification and runtimes, which could be a headache if not done in a community oriented, deliberate way. All that is to say, its up the language toolchain to deal with compiling that language to WASM, but once its WASM, its portable
- mike_hearn 3y ago
  I'd be curious to see a demonstration codebase where JS or other dynamic languages are compiled to WASM and then loaded into something else with automatic FFI at the module level. Because I think you'd still need to work with v8::Value objects or an equivalent to use JS from C++, WASM or not. WASM isn't meant to be a compilation target for languages like JS, what dynamic languages need from a compiler is very different to what Rust needs.
  - no_wizard 3y ago
    To be clear, I'm talking about the following: I can write some code or use modules written in JavaScript that compile to WASM. I can use that compiled output with any runtime that can import WASM and expose its interface natively. To that end, I could use it in Rust, no problem. The same is in reverse. I can write a Rust program, compile to WASM, and use it in JS runtime (lots of projects do this already, actually. esbuild does this with Go as well). Right Now there are of course limitations, however if current trajectory holds, you'll be able to just import wasm directly into a compat runtime. So I can take the JS to WASM output and share it to any language to can understand it. This is, as I understand it, the advantage of WASM in the future tense (and if you use JavaScript, its kind of present tense, esp. with import assertions)