4 ms·
> The article discovers an important tradeoff: speed vs. convenience. Generated code is faster but less convenient, because it adds steps to your compile. Rust
by pcwalton 13y ago
> The article discovers an important tradeoff: speed vs. convenience. Generated code is faster but less convenient, because it adds steps to your compile.
Rust's approach to serialization eliminates this unnecessary tradeoff, IMO: it uses the built-in macro system to parse the structure of your serializable data types and generates the specialized code at compile time. There is no special build step, you get the full speed of specialized code, and it's as simple as writing "#[deriving(Encodable)]" at the top of your data structures.
(As an added bonus, we're using the same infrastructure to generate trace hooks for precise GC in Servo, which eliminates a major pain point of integrating systems-level code with precise garbage collectors.)
- haberman 13y agoThat's cool, I'll have to look into this at some point. This approach would simplify compile-time code generation, but it doesn't help interpreted languages that have no compile step at all. > As an added bonus, we're using the same infrastructure to generate trace hooks for precise GC in Servo, which eliminates a major pain point of integrating systems-level code with precise garbage collectors. I don't know what this means, but it sounds really interesting, do you have a reference with more info?
- pcwalton 13y ago> This approach would simplify compile-time code generation, but it doesn't help interpreted languages that have no compile step at all. Well, it does if they have macro systems like Scheme :) Of course, mainstream interpreted languages don't. > I don't know what this means, but it sounds really interesting, do you have a reference with more info? There isn't an official writeup that I'm aware of, but I can briefly explain what it is. In precise GCs, you have to have a way for the GC to traverse all the GC'd objects that a particular object points to. This is a problem in languages like C++ that have no or insufficient reflection. Traditionally the solution has been for everyone to manually implement "trace hooks" or "visitor hooks" for all objects that participate in garbage collection, which are C++ methods that enumerate all of the objects that a given object points to; this is what Gecko (with the cycle collector) and Blink (with Oilpan) do. But this is tedious and error-prone, and is especially scary when you consider that errors can lead to hard-to-diagnose use-after-free vulnerabilities. We observed that this is a very similar problem to serialization; in serialization you want to call a method on every object that a given object points to (to serialize it), while in tracing you also want to call a method on every object that a given object points to (to trace it). So we decided to reuse the same compiler infrastructure. This has worked pretty well, despite the weirdness of saying `#[deriving(Encodable)]` for GC stuff. (We'll probably want to rename it.)
- haberman 13y ago> Well, it does if they have macro systems like Scheme :) I wouldn't generally count that because generated code in dynamic languages is (in my experience) an order of magnitude slower anyway. For example, generated protobuf-parsing code in Python is something crazy like 100x slower than "the same" generated code in C++. Python might not be the best example since it's a lot slower than other dynamic languages like JavaScript or Lua (don't know about Scheme). But in general my experience is that generated code in dynamic languages isn't in the same ballpark as generated code in a low-level language like C/C++ (and probably Rust). > So we decided to reuse the same compiler infrastructure. Very interesting. What is the function signature of the generated functions? Are you saying that the functions you generate for serialization are the same (and have the same signature) as the functions you generate for GC tracing?
- pcwalton 13y ago> Very interesting. What is the function signature of the generated functions? Are you saying that the functions you generate for serialization are the same (and have the same signature) as the functions you generate for GC tracing? Yes, they're the same. They take the actual serializer (JSON, or YAML, or the GC, etc) as a type parameter so that you can just write `#[deriving(Encodable)]` and have it be used with different serializers. Type parameters are expanded away at compile time, so this leads to zero overhead at runtime.
- haberman 13y agoGot it, so it looks like "Encoder" is this trait/interface: http://static.rust-lang.org/doc/master/serialize/trait.Encoder.html http://static.rust-lang.org/doc/master/serialize/trait.Encod... I think of what you call "Encoder" as a "Visitor" (just in case you're looking for renaming input :) So the function that you are generating at runtime is similar to a template function in C++, templated on the specific serializer (Encoder/Visitor/etc). One thing that this approach does not currently support (which admittedly is probably not required for most users, and is probably not in line with the overall design of Rust) is the ability to resume. Suppose you are serializing to a socket and the socket is not write-ready. You would need to block until the socket is write-ready (or allocate a buffer to hold the serialized data). This interface doesn't provide a way of suspending the visit/encode and resuming it later. This also doesn't seem to have a way of identifying the fields -- is this true? Like if you were trying to encode as JSON but wanted to know the name of each field?
- p0nce 13y agoWell, yeah, D does this routinely too, no macros needed. https://github.com/atilaneves/cerealed https://github.com/atilaneves/cerealed https://github.com/msgpack/msgpack-d https://github.com/msgpack/msgpack-d https://github.com/Orvid/JSONSerialization https://github.com/Orvid/JSONSerialization
- dwrensha 13y agoThe Encodable trait sounds convenient for lots of use cases. However, speed and convenience are not the only things you might want; multi-language support and protocol evolvability are often also important, and when they are, something like Cap'n Proto's schema language is very useful. Instead of "serialization frameworks", I like to call these libraries "type systems for distributed computing". I can then ask: where are your types defined?
- haberman 13y agoAgreed! I would also take it one step further and suggest that these libraries are "type systems for data interchange." Say you want to stuff your logs into a database like BigQuery (which I work on at Google) for later analysis. What is the schema of your logs? I dream of a world where the logfile->JSON parser can use the same schema definition as the database itself, so there is no glue/conversion required to turn one into the other. Likewise I think these schema languages are ideal for defining ASTs and other compiler infrastructure. Wouldn't it be cool if you could dump your AST into a database and run queries over it? Again, the goal is to do this without requiring a big schema mapping / transformation first. I believe in this vision, and I think Protocol Buffer schemas fit the bill as something that can be usefully applied to much more than just Protocol Buffers. The first step I'm taking towards this is making JSON a first-class citizen in my library.