2 ms·

My experience with WASM so far, is that I can import WASM modules without caring at all about FFI. They're produced, and I can use them, as is, whatever their e

My experience with WASM so far, is that I can import WASM modules without caring at all about FFI. They're produced, and I can use them, as is, whatever their exposed interface is. I'm not even sure Truffle/Graal is doing that across language boundaries (yet). Of course, in practice, this is basically C, C++ and Rust being used in the web browser or with V8 / Deno, because its the biggest WASM runtime there is right now, but thats changing too, as language communities are starting to seriously consider first party support for WASM. To address the point of mapping types / data from one language to another: thats the whole point of WASM, is that those languages need to only bind their output to the WASM guidelines. Its all there is to it. Valid WASM will run in any valid WASM runtime

mike_hearn 3y ago
Truffle is definitely not only doing that across language boundaries but for far more languages. For example, using Ruby from JavaScript: var array = Polyglot.eval("ruby", "[1,2,42,4]") console.log(array[2]); Invoking a C/C++/Rust main method from Python: import polyglot cpart = polyglot.eval(language="llvm", path="polyglot") cpart.main() Reading a JS array from C: #include <stdio.h> #include <graalvm/llvm/polyglot.h> int main() { void *array = polyglot_eval("js", "[1,2,42,4]"); int element = polyglot_as_i32(polyglot_get_array_element(array, 2)); printf("%d\n", element); return element; } More examples here: https://www.graalvm.org/latest/reference-manual/polyglot-programming/ https://www.graalvm.org/latest/reference-manual/polyglot-pro... W.R.T. the "WASM guidelines", can you point us to these guidelines and where they define how to map a Python dict to a Java HashMap, for example? The Truffle Polyglot messaging spec does define how this is done in a way that avoids performance problems because the FFI is expressed in terms of compiler IR. WASM can't even run either of these languages natively so it can't define how they interop with each other.
- no_wizard 3y ago
  This is actually a pretty good step in the right direction. Big question I have: can I just import (like I'd import anything else in a code base) the dependency and it'll "see" the exposed API surface of the module? This looks like it does some eval of the underlying code depending on what it is, but it must be explicitly mapped by the consumer (declaring the language and the code), which I respect wholly, but its a little different from the "ease of use" perspective. That said, its another promising approach, always interested to see how this matures. To answer the second half of this, its not about defining how Python Dict should map to a Java Hashmap. Its about how Java or Python or whatever language maps to WASM. As long as the produced output is valid WASM, it will work across any boundary that supports WASM. Inherently, it devoids itself from having to think about any specific language this way. Right now this also produces limitations (because features have to be specified uniformly so the target of compilation is the same for everyone) The inherent flaw of most multi-compat runtime systems, in my opinion, is making the consumer think about the underlying language. Rather, if you use Go to produce a WASM module, it can be used in Rust, or JavaScript, or C / C++ etc. without considering how it was written
  - mike_hearn 3y ago
    Yes, depending on what you mean by "see". For a statically typed language using imports from a dynamically typed language, you'll need some interfaces and stuff to cast values to. But that's inherent. The C compiler doesn't have any concept of a dynamic type beyond void* so you'll always need to cast it to something more specific before using it. For the other way around that doesn't apply and you just use whatever you import. You don't technically have to specify the language. The file extension is usually enough to select the right language. I'm not sure how it gets easier, unless you are asking about a sort of polyglot package manager or build system. Indeed there isn't one of those, because they're usually language specific. It would be a good improvement to have though. Right now you'd need to use npm to get JS code, pip to get Python code etc. With respect to language mappings, I don't think WASM solves this. There is no mapping to WASM of a hashmap because the WASM type system doesn't have any such concept. Could you elaborate on what you're referring to here? As the name implies, it's a kind of assembly language, so it offers no more assistance in this task than ARM or x86 does. Moreover the approach of defining a universal "language" and mapping everything to it was tried and didn't work so great, that's sort of the classical JVM or CLR approach. Languages disagree on how core concepts work. For example a JS array is a very different thing to an array in C, a Java HashMap has a different API and behavior to a Python dict which in turn is different to the same concept in Scheme. You can make some progress by trying to compile everything through to the One True HashMap, but performance and edge cases at the boundaries become problems. The Truffle insight is to let each language be its own thing. Instead of having One True HashMap (or whatever), you let each language work with its own concepts and memory layouts then define a standard way to export basic operations like "read member", "invoke function", "access key" in the form of compiler IR snippets. So when you invoke polyglot_get_array_element on a Ruby array it's not really doing a function call, and there is no shared array type that every language has to compile to. Instead the function call is replaced with the actual machine code that the Ruby engine would use to look up an array element, and the result is stored in a C void* but it's not a pointer to some unified object concept or even a real pointer at all. That's why you can do something like casting it to a struct of function pointers, and get something that will then directly invoke Ruby methods. The cast operation is also being virtualized. This approach sidesteps the question of how to share stuff between languages without trying to compile it all to a generic meta-language. The operations are called the "interop protocol" which is a bit confusing because it's not actually a network protocol of any kind. It defines lots of high level mapping operations so you can do things like work with a time/date from a guest language using your host language's time/dat types. https://www.graalvm.org/truffle/javadoc/com/oracle/truffle/api/interop/InteropLibrary.html https://www.graalvm.org/truffle/javadoc/com/oracle/truffle/a... It's important to note here that the exported operations aren't target language specific. Once you implement the interop protocol for a Truffle language it works for every other language, there's no N:N binding work required. So it's a scalable approach.
- deleted 3y ago
  [deleted]

mike_hearn 3y ago

Truffle is definitely not only doing that across language boundaries but for far more languages. For example, using Ruby from JavaScript: var array = Polyglot.eval("ruby", "[1,2,42,4]") console.log(array[2]); Invoking a C/C++/Rust main method from Python: import polyglot cpart = polyglot.eval(language="llvm", path="polyglot") cpart.main() Reading a JS array from C: #include <stdio.h> #include <graalvm/llvm/polyglot.h> int main() { void *array = polyglot_eval("js", "[1,2,42,4]"); int element = polyglot_as_i32(polyglot_get_array_element(array, 2)); printf("%d\n", element); return element; } More examples here: https://www.graalvm.org/latest/reference-manual/polyglot-programming/ https://www.graalvm.org/latest/reference-manual/polyglot-pro... W.R.T. the "WASM guidelines", can you point us to these guidelines and where they define how to map a Python dict to a Java HashMap, for example? The Truffle Polyglot messaging spec does define how this is done in a way that avoids performance problems because the FFI is expressed in terms of compiler IR. WASM can't even run either of these languages natively so it can't define how they interop with each other.

no_wizard 3y ago
This is actually a pretty good step in the right direction. Big question I have: can I just import (like I'd import anything else in a code base) the dependency and it'll "see" the exposed API surface of the module? This looks like it does some eval of the underlying code depending on what it is, but it must be explicitly mapped by the consumer (declaring the language and the code), which I respect wholly, but its a little different from the "ease of use" perspective. That said, its another promising approach, always interested to see how this matures. To answer the second half of this, its not about defining how Python Dict should map to a Java Hashmap. Its about how Java or Python or whatever language maps to WASM. As long as the produced output is valid WASM, it will work across any boundary that supports WASM. Inherently, it devoids itself from having to think about any specific language this way. Right now this also produces limitations (because features have to be specified uniformly so the target of compilation is the same for everyone) The inherent flaw of most multi-compat runtime systems, in my opinion, is making the consumer think about the underlying language. Rather, if you use Go to produce a WASM module, it can be used in Rust, or JavaScript, or C / C++ etc. without considering how it was written
- mike_hearn 3y ago
  Yes, depending on what you mean by "see". For a statically typed language using imports from a dynamically typed language, you'll need some interfaces and stuff to cast values to. But that's inherent. The C compiler doesn't have any concept of a dynamic type beyond void* so you'll always need to cast it to something more specific before using it. For the other way around that doesn't apply and you just use whatever you import. You don't technically have to specify the language. The file extension is usually enough to select the right language. I'm not sure how it gets easier, unless you are asking about a sort of polyglot package manager or build system. Indeed there isn't one of those, because they're usually language specific. It would be a good improvement to have though. Right now you'd need to use npm to get JS code, pip to get Python code etc. With respect to language mappings, I don't think WASM solves this. There is no mapping to WASM of a hashmap because the WASM type system doesn't have any such concept. Could you elaborate on what you're referring to here? As the name implies, it's a kind of assembly language, so it offers no more assistance in this task than ARM or x86 does. Moreover the approach of defining a universal "language" and mapping everything to it was tried and didn't work so great, that's sort of the classical JVM or CLR approach. Languages disagree on how core concepts work. For example a JS array is a very different thing to an array in C, a Java HashMap has a different API and behavior to a Python dict which in turn is different to the same concept in Scheme. You can make some progress by trying to compile everything through to the One True HashMap, but performance and edge cases at the boundaries become problems. The Truffle insight is to let each language be its own thing. Instead of having One True HashMap (or whatever), you let each language work with its own concepts and memory layouts then define a standard way to export basic operations like "read member", "invoke function", "access key" in the form of compiler IR snippets. So when you invoke polyglot_get_array_element on a Ruby array it's not really doing a function call, and there is no shared array type that every language has to compile to. Instead the function call is replaced with the actual machine code that the Ruby engine would use to look up an array element, and the result is stored in a C void* but it's not a pointer to some unified object concept or even a real pointer at all. That's why you can do something like casting it to a struct of function pointers, and get something that will then directly invoke Ruby methods. The cast operation is also being virtualized. This approach sidesteps the question of how to share stuff between languages without trying to compile it all to a generic meta-language. The operations are called the "interop protocol" which is a bit confusing because it's not actually a network protocol of any kind. It defines lots of high level mapping operations so you can do things like work with a time/date from a guest language using your host language's time/dat types. https://www.graalvm.org/truffle/javadoc/com/oracle/truffle/api/interop/InteropLibrary.html https://www.graalvm.org/truffle/javadoc/com/oracle/truffle/a... It's important to note here that the exported operations aren't target language specific. Once you implement the interop protocol for a Truffle language it works for every other language, there's no N:N binding work required. So it's a scalable approach.
deleted 3y ago
[deleted]