3 ms·

Cross-language interactions suck. We need WebAssembly components and good code generators for a critical mass of languages before people actually start to use W

by chubot 5y ago

Cross-language interactions suck. We need WebAssembly components and good code generators for a critical mass of languages before people actually start to use Wasm across different languages. Unpopular opinion: Users will eventually realize that the lowest common denominator between languages is ... a BYTE STREAM (e.g. JSON/CSV/HTML), or what I think of as shell / Unix / Web -style composition. IDLs and code generators are useful in many limited domains (e.g. when you control both sides of the wire), but they bake in a lot of assumptions that people don't realize are language-specific. e.g. Protobufs are very good for C++ <-> C++ communication, but Java and Python users seem to dislike them equally, and even Go users do too. COM is probably better than what most people are proposing now -- it recognizes the problem is dynamic, rather than trying to create the leaky abstraction of "fake" static system. It's true that IDL is reinvented every 5 / 10 / 20 years. (Related recent story: https://news.ycombinator.com/item?id=30128048 https://news.ycombinator.com/item?id=30128048) I expect WASM will get some kind of component system (if it doesn't already exist), but many apps will still need to fall back to something more general. ----- I'm writing about byte streams as a narrow waist of interoperability now, and this is the "lead in" review post: http://www.oilshell.org/blog/2021/12/review-arch.html http://www.oilshell.org/blog/2021/12/review-arch.html Even though disparate WebAssembly components can run in the same process (with memory potentially shared by the host), for something as wide as the "Web", the lowest common denominator is still the common text-based interchange formats we already have. Related comment on WebAssembly: https://news.ycombinator.com/item?id=28581634 https://news.ycombinator.com/item?id=28581634 Programmers underestimate the degree to which languages and VMs are coupled i.e. I question whether WebAssembly is truly polyglot, i.e. GC requires a richer type system in the VM, and types create languages that are winners or losers. Losers are the language implementations that experience 2x-10x slowdowns.

krona 5y ago
The Interface Types Proposal is Wasm's answer to what you're describing: https://github.com/WebAssembly/interface-types/blob/main/proposals/interface-types/Explainer.md https://github.com/WebAssembly/interface-types/blob/main/pro...
deathanatos 5y ago
> e.g. Protobufs are very good for C++ <-> C++ communication, but Java and Python users seem to dislike them equally, and even Go users do too. Well, hilariously, when I looked at using it on a project, JSON outperformed protobuf in Python. JSON is implemented in C, Protobuf in Python, and the C decoder for a less efficient format won out. (Now, technically, Protobuf has a C implementation, but at the time I was testing, it segfaulted reliably. Which is the problem with C…) I also recall there were some severe problems with protobuf's ability to represent some types, but I don't remember what they are at this point. I thought it was something to do with sum types, but I'm looking at it now and it does have oneof, so IDK.
- chubot 5y ago
  Yup, it is crazy how much optimization these "narrow waist" formats get, and it goes even further with simdjson and so forth [1]. The Python protobuf implementation has a somewhat checkered history (I used protobuf v1 and v2 for a long time, and reviewed v3 a tiny bit). The type system issue is that protobufs to a large extent "replace" your language's types. It's essentially language-independent type. So that means you are limited to a lowest common denominator, and you have the issues of "winners" and "losers"... I would call Python somewhat of a "loser" in the protobuf world, i.e. it feels more second class and is more of a compromise. This doesn't mean that anybody did a bad job; it's just a fundamental issue with such IDLs. In contrast, JSON/XML/CSV are "data-first" and there are multiple ways of using them and parsing them. You can lazily parse all of them, DOM and SAX, for example, and you have push and pull parsers, etc. Protobufs have grown some of that but it wasn't the primary usage, and many people don't know about it. [1] https://github.com/simdjson/simdjson https://github.com/simdjson/simdjson
epolanski 5y ago
I don't think it is unpopular at all. I started writing some emacs plugin in the past, then ported the whole thing to the browser, and since then I have a hard time thinking in any other terms as byte stream, even though I like to call it buffer.
Kinrany 5y ago
It should be possible to create a lower-level IDL that starts with byte streams and makes no assumptions about performance requirements and the transport layer. Any language would be able to decode byte streams according to provided schema into intermediate formats (same semantics, different implementations) with a library, and then decode that significantly easier to understand data into application-specific formats. Rust's Serde is almost there, but uses dynamic structure instead of using a schema that allows custom types.
62951413 5y ago
I always thought Java was the original protobuf library. Of the big three (PB, Avro, Thrift) only protobufs have a great modern RPC library included. I suspect more people start using PB because of gRPC. Unfortunately many people are still stuck with JSON over HTTP. I believe gRPC is more popular among systems where mobile clients dominate. As a side note, if you expect to programmatically change your records (e.g. inject a new field on the server into a record received from a client) Avro is a much better choice. Avro also has a stronger schema migration story. But both are row-oriented formats with nearly identical IDLs anyway.

by chubot 5y ago

krona 5y ago

The Interface Types Proposal is Wasm's answer to what you're describing: https://github.com/WebAssembly/interface-types/blob/main/proposals/interface-types/Explainer.md https://github.com/WebAssembly/interface-types/blob/main/pro...

deathanatos 5y ago

> e.g. Protobufs are very good for C++ <-> C++ communication, but Java and Python users seem to dislike them equally, and even Go users do too. Well, hilariously, when I looked at using it on a project, JSON outperformed protobuf in Python. JSON is implemented in C, Protobuf in Python, and the C decoder for a less efficient format won out. (Now, technically, Protobuf has a C implementation, but at the time I was testing, it segfaulted reliably. Which is the problem with C…) I also recall there were some severe problems with protobuf's ability to represent some types, but I don't remember what they are at this point. I thought it was something to do with sum types, but I'm looking at it now and it does have oneof, so IDK.

chubot 5y ago
Yup, it is crazy how much optimization these "narrow waist" formats get, and it goes even further with simdjson and so forth [1]. The Python protobuf implementation has a somewhat checkered history (I used protobuf v1 and v2 for a long time, and reviewed v3 a tiny bit). The type system issue is that protobufs to a large extent "replace" your language's types. It's essentially language-independent type. So that means you are limited to a lowest common denominator, and you have the issues of "winners" and "losers"... I would call Python somewhat of a "loser" in the protobuf world, i.e. it feels more second class and is more of a compromise. This doesn't mean that anybody did a bad job; it's just a fundamental issue with such IDLs. In contrast, JSON/XML/CSV are "data-first" and there are multiple ways of using them and parsing them. You can lazily parse all of them, DOM and SAX, for example, and you have push and pull parsers, etc. Protobufs have grown some of that but it wasn't the primary usage, and many people don't know about it. [1] https://github.com/simdjson/simdjson https://github.com/simdjson/simdjson

epolanski 5y ago

I don't think it is unpopular at all. I started writing some emacs plugin in the past, then ported the whole thing to the browser, and since then I have a hard time thinking in any other terms as byte stream, even though I like to call it buffer.

Kinrany 5y ago

It should be possible to create a lower-level IDL that starts with byte streams and makes no assumptions about performance requirements and the transport layer. Any language would be able to decode byte streams according to provided schema into intermediate formats (same semantics, different implementations) with a library, and then decode that significantly easier to understand data into application-specific formats. Rust's Serde is almost there, but uses dynamic structure instead of using a schema that allows custom types.

62951413 5y ago

I always thought Java was the original protobuf library. Of the big three (PB, Avro, Thrift) only protobufs have a great modern RPC library included. I suspect more people start using PB because of gRPC. Unfortunately many people are still stuck with JSON over HTTP. I believe gRPC is more popular among systems where mobile clients dominate. As a side note, if you expect to programmatically change your records (e.g. inject a new field on the server into a record received from a client) Avro is a much better choice. Avro also has a stronger schema migration story. But both are row-oriented formats with nearly identical IDLs anyway.