5 ms·

As someone who has been working on parsing/serialization for many years, I can absolutely confirm that generating schema-specific code will beat generic code al

As someone who has been working on parsing/serialization for many years, I can absolutely confirm that generating schema-specific code will beat generic code almost every time. The article discovers an important tradeoff: speed vs. convenience. Generated code is faster but less convenient, because it adds steps to your compile. And you pay this compile-time overhead for every message type you want to manipulate. The pain of this generated code was one of Kenton Varda's motivations for creating Cap'n Proto after working on Protocol Buffers for years. Unlike Protocol Buffers, Cap'n Proto doesn't need to generate parsing/serialization code because its serialization format also works as an in-memory format. I have taken a somewhat different approach to the problem, with my Protocol Buffer / JSON / etc. serialization framework upb (https://github.com/haberman/upb https://github.com/haberman/upb). Instead of using static code generation, I use a JIT approach where I generate specialized code on-the-fly. This approach is particularly important when you are wrapping the library in an interpreted language like Python/Ruby/Lua/JavaScript, because users of these languages don't have a compile cycle at all, so adding one is a large inconvenience. My library isn't ready for prime time but I'm hoping to have something ready to use this year.

throwaway812 13y ago
The last time I looked at upb, it only supported parsing, which was a deal-breaker. Have you implemented serializing since then? It's hard to tell from the git log, which is mostly "dump from google internal sources XX-YY-ZZZZ". I'm very excited about upb! Thanks for your work on it over the years. Do you have any tasks that an outside contributor could help with? Thanks!
- haberman 13y ago
  Still no serialization, sorry. This is getting close to being my top priority though. It's a hard problem to design the kind of generic visitor I want while being speed-competitive with the fastest stuff that's out there. Probably can't use a lot of outside help, but thanks for asking. What's taking so long is a very intense process of refining the design. I am often reminded of this bit from "Worse is Better": "The right thing takes forever to design, but it is quite small at every point along the way." This is exactly the approach I am taking. It's taken forever to design, but over time it becomes more and more capable without the core growing in size very much. I hope that this intensive design work will pay off soon. Thanks for your interest and I hope to soon be able to reward it with a highly compelling parsing/serialization experience!
  - throwaway812 13y ago
    Awesome, looking forward to that day :). Keep up the great work.
pcwalton 13y ago
> The article discovers an important tradeoff: speed vs. convenience. Generated code is faster but less convenient, because it adds steps to your compile. Rust's approach to serialization eliminates this unnecessary tradeoff, IMO: it uses the built-in macro system to parse the structure of your serializable data types and generates the specialized code at compile time. There is no special build step, you get the full speed of specialized code, and it's as simple as writing "#[deriving(Encodable)]" at the top of your data structures. (As an added bonus, we're using the same infrastructure to generate trace hooks for precise GC in Servo, which eliminates a major pain point of integrating systems-level code with precise garbage collectors.)
- haberman 13y ago
  That's cool, I'll have to look into this at some point. This approach would simplify compile-time code generation, but it doesn't help interpreted languages that have no compile step at all. > As an added bonus, we're using the same infrastructure to generate trace hooks for precise GC in Servo, which eliminates a major pain point of integrating systems-level code with precise garbage collectors. I don't know what this means, but it sounds really interesting, do you have a reference with more info?
  - pcwalton 13y ago
    > This approach would simplify compile-time code generation, but it doesn't help interpreted languages that have no compile step at all. Well, it does if they have macro systems like Scheme :) Of course, mainstream interpreted languages don't. > I don't know what this means, but it sounds really interesting, do you have a reference with more info? There isn't an official writeup that I'm aware of, but I can briefly explain what it is. In precise GCs, you have to have a way for the GC to traverse all the GC'd objects that a particular object points to. This is a problem in languages like C++ that have no or insufficient reflection. Traditionally the solution has been for everyone to manually implement "trace hooks" or "visitor hooks" for all objects that participate in garbage collection, which are C++ methods that enumerate all of the objects that a given object points to; this is what Gecko (with the cycle collector) and Blink (with Oilpan) do. But this is tedious and error-prone, and is especially scary when you consider that errors can lead to hard-to-diagnose use-after-free vulnerabilities. We observed that this is a very similar problem to serialization; in serialization you want to call a method on every object that a given object points to (to serialize it), while in tracing you also want to call a method on every object that a given object points to (to trace it). So we decided to reuse the same compiler infrastructure. This has worked pretty well, despite the weirdness of saying `#[deriving(Encodable)]` for GC stuff. (We'll probably want to rename it.)
densh 13y ago
> The article discovers an important tradeoff: speed vs. convenience. Generated code is faster but less convenient, because it adds steps to your compile. Not if you use languages with built-in metaprogramming capabilities that are expressive enough to generate scheme-specific code at compile-time (e.g. Scheme, Scala, D, Rust, Nimrod etc) I'm personally a huge fun of typeclasses as implicits + macro materializers in Scala: http://docs.scala-lang.org/overviews/macros/implicits.html http://docs.scala-lang.org/overviews/macros/implicits.html
- thomasahle 13y ago
  Though that would require the implementation of those features being more advanced than the go json library's "runtime reflection".

throwaway812 13y ago

The last time I looked at upb, it only supported parsing, which was a deal-breaker. Have you implemented serializing since then? It's hard to tell from the git log, which is mostly "dump from google internal sources XX-YY-ZZZZ". I'm very excited about upb! Thanks for your work on it over the years. Do you have any tasks that an outside contributor could help with? Thanks!

haberman 13y ago
Still no serialization, sorry. This is getting close to being my top priority though. It's a hard problem to design the kind of generic visitor I want while being speed-competitive with the fastest stuff that's out there. Probably can't use a lot of outside help, but thanks for asking. What's taking so long is a very intense process of refining the design. I am often reminded of this bit from "Worse is Better": "The right thing takes forever to design, but it is quite small at every point along the way." This is exactly the approach I am taking. It's taken forever to design, but over time it becomes more and more capable without the core growing in size very much. I hope that this intensive design work will pay off soon. Thanks for your interest and I hope to soon be able to reward it with a highly compelling parsing/serialization experience!
- throwaway812 13y ago
  Awesome, looking forward to that day :). Keep up the great work.

pcwalton 13y ago

> The article discovers an important tradeoff: speed vs. convenience. Generated code is faster but less convenient, because it adds steps to your compile. Rust's approach to serialization eliminates this unnecessary tradeoff, IMO: it uses the built-in macro system to parse the structure of your serializable data types and generates the specialized code at compile time. There is no special build step, you get the full speed of specialized code, and it's as simple as writing "#[deriving(Encodable)]" at the top of your data structures. (As an added bonus, we're using the same infrastructure to generate trace hooks for precise GC in Servo, which eliminates a major pain point of integrating systems-level code with precise garbage collectors.)

haberman 13y ago
That's cool, I'll have to look into this at some point. This approach would simplify compile-time code generation, but it doesn't help interpreted languages that have no compile step at all. > As an added bonus, we're using the same infrastructure to generate trace hooks for precise GC in Servo, which eliminates a major pain point of integrating systems-level code with precise garbage collectors. I don't know what this means, but it sounds really interesting, do you have a reference with more info?
- pcwalton 13y ago
  > This approach would simplify compile-time code generation, but it doesn't help interpreted languages that have no compile step at all. Well, it does if they have macro systems like Scheme :) Of course, mainstream interpreted languages don't. > I don't know what this means, but it sounds really interesting, do you have a reference with more info? There isn't an official writeup that I'm aware of, but I can briefly explain what it is. In precise GCs, you have to have a way for the GC to traverse all the GC'd objects that a particular object points to. This is a problem in languages like C++ that have no or insufficient reflection. Traditionally the solution has been for everyone to manually implement "trace hooks" or "visitor hooks" for all objects that participate in garbage collection, which are C++ methods that enumerate all of the objects that a given object points to; this is what Gecko (with the cycle collector) and Blink (with Oilpan) do. But this is tedious and error-prone, and is especially scary when you consider that errors can lead to hard-to-diagnose use-after-free vulnerabilities. We observed that this is a very similar problem to serialization; in serialization you want to call a method on every object that a given object points to (to serialize it), while in tracing you also want to call a method on every object that a given object points to (to trace it). So we decided to reuse the same compiler infrastructure. This has worked pretty well, despite the weirdness of saying `#[deriving(Encodable)]` for GC stuff. (We'll probably want to rename it.)

densh 13y ago

> The article discovers an important tradeoff: speed vs. convenience. Generated code is faster but less convenient, because it adds steps to your compile. Not if you use languages with built-in metaprogramming capabilities that are expressive enough to generate scheme-specific code at compile-time (e.g. Scheme, Scala, D, Rust, Nimrod etc) I'm personally a huge fun of typeclasses as implicits + macro materializers in Scala: http://docs.scala-lang.org/overviews/macros/implicits.html http://docs.scala-lang.org/overviews/macros/implicits.html

thomasahle 13y ago
Though that would require the implementation of those features being more advanced than the go json library's "runtime reflection".