3 ms·

For a long time (much longer than I expected it would take) I've been working on a protobuf implementation in C that does not use Google's C++ implementation at

For a long time (much longer than I expected it would take) I've been working on a protobuf implementation in C that does not use Google's C++ implementation at all. I've been through about three rewrites and I finally have the interface right. I'm hoping it will be usable with Python soon (weeks). https://github.com/haberman/upb/wiki https://github.com/haberman/upb/wiki (if anyone's looking at the code, I'm working on the src-refactoring branch at the moment) The benefits of my approach are: * you can avoid depending on a 1MB C++ library. upb is more like 30k compiled. * you can avoid doing any code generation. instead you just load the .proto schema at runtime, so you don't have to get a C++ compiler involved. * Google's protobuf library does have a dynamic/reflection option that avoids my previous point, but it is ~10x slower than generating C++ code. My library, last time I benchmarked it, was 70-90% of the speed of generated C++.

jsarch 16y ago
Can you clarify what you mean by "70-90% of the speed of generated C++"? Suppose that the generated C++ takes 1.0 seconds. Does your implementation take 0.7-0.9s or 1.7-1.9s or something else?
- haberman 16y ago
  If the generated C++ can parse 1MB/s, I can parse at 700-900kB/s. 70-90% of the speed, not the time. So 1.1-1.4 seconds, in your example.
- deleted 16y ago
  [deleted]
sigil 16y ago
Here's a fast Python C extension for protobuf that's already usable: https://github.com/acg/lwpb https://github.com/acg/lwpb I read through your upb code about 3-4 months ago, was initially impressed, but couldn't get the Python extension to work. Certain abstractions really lost me, like pushing and pulling between sources and sinks. Why not just let a top-level event loop run the show in terms of buffered reads and size calculation for writes? But maybe you've refactored since.
- haberman 16y ago
  > but couldn't get the Python extension to work. Sorry, I should be clearer about the current state of the code, which for the Python extension is: currently completely broken. Since I was focusing on the core interfaces, the more peripheral pieces (like the language extensions) are totally broken at the moment. > Certain abstractions really lost me, like pushing and pulling between sources and sinks. Hopefully more documentation will make this clear. Making sources and sinks a general abstraction makes the event-based interface independent of any specific serialization (like protobuf binary format, protobuf text format, a JSON serialization of the same schema, etc). The key thing about protobufs is that a .proto file defines a typed tree structure, and the core interfaces of upb let you iterate over that tree structure, regardless of how exactly that tree structure was serialized. > Why not just let a top-level event loop run the show in terms of buffered reads and size calculation for writes? In my most recent interface, the upb_src does indeed run an event loop, and calls callbacks for every input value, or on a submessage start, or on a submessage end. For a while I wanted to make upb_src a pull-based interface instead, to give the application more control over the main loop, but this created more problems than it solved.
apotheon 16y ago
Looks interesting. I might need to dig in. I like the license, too.

jsarch 16y ago

Can you clarify what you mean by "70-90% of the speed of generated C++"? Suppose that the generated C++ takes 1.0 seconds. Does your implementation take 0.7-0.9s or 1.7-1.9s or something else?

haberman 16y ago
If the generated C++ can parse 1MB/s, I can parse at 700-900kB/s. 70-90% of the speed, not the time. So 1.1-1.4 seconds, in your example.
deleted 16y ago
[deleted]

sigil 16y ago

Here's a fast Python C extension for protobuf that's already usable: https://github.com/acg/lwpb https://github.com/acg/lwpb I read through your upb code about 3-4 months ago, was initially impressed, but couldn't get the Python extension to work. Certain abstractions really lost me, like pushing and pulling between sources and sinks. Why not just let a top-level event loop run the show in terms of buffered reads and size calculation for writes? But maybe you've refactored since.

haberman 16y ago
> but couldn't get the Python extension to work. Sorry, I should be clearer about the current state of the code, which for the Python extension is: currently completely broken. Since I was focusing on the core interfaces, the more peripheral pieces (like the language extensions) are totally broken at the moment. > Certain abstractions really lost me, like pushing and pulling between sources and sinks. Hopefully more documentation will make this clear. Making sources and sinks a general abstraction makes the event-based interface independent of any specific serialization (like protobuf binary format, protobuf text format, a JSON serialization of the same schema, etc). The key thing about protobufs is that a .proto file defines a typed tree structure, and the core interfaces of upb let you iterate over that tree structure, regardless of how exactly that tree structure was serialized. > Why not just let a top-level event loop run the show in terms of buffered reads and size calculation for writes? In my most recent interface, the upb_src does indeed run an event loop, and calls callbacks for every input value, or on a submessage start, or on a submessage end. For a while I wanted to make upb_src a pull-based interface instead, to give the application more control over the main loop, but this created more problems than it solved.

apotheon 16y ago

Looks interesting. I might need to dig in. I like the license, too.