5 ms·

More protobuf implementations is certainly good, Google's have been really really lacking for us. It would be really nice if there was a pure Python implementa

More protobuf implementations is certainly good, Google's have been really really lacking for us. It would be really nice if there was a pure Python implementation that didn't use tons of unnecessary metaprogramming.

haberman 11y ago
It's really interesting to me that the top two comments on this story both wish Protocol Buffers were different, but seem to be asking for opposite things. Your comment wishes that there was a pure-Python implementation that doesn't do metaprogramming. That would imply doing code generation, but having the generated code be more "concrete", so-to-speak (containing all of the actual implementation). However, a different comment (https://news.ycombinator.com/item?id=10762101 https://news.ycombinator.com/item?id=10762101) admires the thriftpy project which doesn't have a codegen step at all. This is in some sense the opposite of what you want: everything becomes metaprogramming. I work on Protocol Buffers at Google, and I frequently observe these kinds of opposing feelings about how protobufs ought to work. One thing I've learned is how incredibly difficult it is to please everybody. Another good example of this is whether to be pure-Python: some people (like you) want that. But it's nearly impossible to make it very fast, and lots of users are sensitive to speed (you can find various comments and blog articles complaining about the speed of Python protobuf).
- euyyn 11y ago
  Hey Josh! Jorge from gRPC here. What's your take on Pyrobuf?
- pixelmonkey 11y ago
  I think we learned a long time ago that though codegen is one path to speedy code, it is not the only path. The thriftpy module dynamically generates a Python module that is based on parsing the thrift schema. That happens once: at module import time. After that, Python has cached the module in sys.modules and it doesn't need to be evaluated again during the program's runtime. The module contains efficient Python classes that are Python bytecode just the same way "compiled" classes would be. But, thriftpy has Cython implementation of Thrift protocols and transports that compile down to C, not Python bytecode. Thus, the thriftpy impl's are not only more dynamic and less cumbersome than the codegen alternative, but they are also faster. In the Python community, we prefer to take other approaches to achieve execution speed. Dynamically generating a Python module is not "metaprogramming". It is not black magic. You can generate a module yourself simply by instantiating a Python module type. Hooking the import statement is an officially supported part of the language, and done by many libraries. This may all seem very worrying to a C++ or Java programmer, but in the Python community we have been doing dynamism with execution speed for about 10 years now, and we haven't regretted any of the results!
  - haberman 11y ago
    For what it's worth, I prefer the approach you describe. I have advocated for it for a long time. I've been writing C extensions to accelerate parsing in dynamic languages for 10 years, and I'm very familiar with this sort of dynamism. It's interesting though that you describe this approach as more Pythonic, because one of the most outspoken critics of this approach is a hard-core Python guy that I work with who has lots of experience with the Python ecosystem. He feels very strongly that it is more Pythonic to have very flat/concrete generated code that is transparent to the reader, and really does not like the idea of hooking import and generating everything at runtime. This is exactly what I am describing about how it is hard to please everybody. Different people appear to have fiercely different opinions about what is idiomatic. When we wrote the Ruby protobuf implementation this year, we took an approach much more along the lines of what I personally prefer. The extension is mostly implemented in C, and it's very easy to build types at runtime. It doesn't directly import .proto files (which I would have preferred) because there was still some desire from others to have some kind of code generation. So the approach we took was to use a Ruby-like DSL for describing the protobuf schema. But really this is nothing but a translation of the .proto file into the Ruby DSL. ie. take the .proto file: syntax = "proto3"; message Test { int32 foo = 1; double bar = 2; Test test = 3; } The "generated code" for this is simply: require 'google/protobuf' Google::Protobuf::DescriptorPool.generated_pool.build do add_message "Test" do optional :foo, :int32, 1 optional :bar, :double, 2 optional :test, :message, 3, "Test" end end Test = Google::Protobuf::DescriptorPool.generated_pool.lookup("Test").msgclass
- JulianWasTaken 11y ago
  I've certainly observed the same I think :) and yeah pleasing everyone is impossible -- I think though there's a somewhat coherent way to put at least part of those two things together. Not wanting code generation might come from people who don't like the additional build step and artifact deployment. I don't like that either, but what I mean by "less metaprogramming" is that the code that's generated currently makes copious use of descriptors, metaclasses, and other "complex" things in a way that makes it extremely JIT-unfriendly on PyPy. We had to completely abandon it in favor of using protoc-c wrapped via CFFI. So I care about speed (almost entirely -- protobuf powers systems for us that do ~350K messages per second). (I do though think it would be possible to satisfy both complaints simultaneously. I spent about 5 minutes trying to write a codegen-less, pure Python implementation: https://github.com/Julian/pb/tree/master/pb https://github.com/Julian/pb/tree/master/pb but I didn't find the Proto3 docs mature enough to figure out the binary protocol at the time. Not sure there's enough there to see what direction I was trying to go in, but I have given this a shot before). And thanks for the reply, I do agree that there is some conflicting concerns involved here.
  - haberman 11y ago
    Ah, that's very interesting -- your desire for more "pure" Python comes from a desire for speed. (Some people who wish the generated code was more concrete want this so the generated code is more readable). I am curious what led you to the conclusion that descriptors and metaclasses were making Python protobuf JIT-unfriendly. I ask because most of the "meta" stuff happens at startup, when you first import the module. It uses a metaclass to generate a bunch of Python methods, but after that they are just regular Python methods, and should be as easy for PyPy to optimize as anything else. There is a bit of reflection happening in some code-paths (like the __init__ method does loop over a list of fields to decide how to initialize them), but the metaclass at least is pretty much gone after import. For this reason, I suspect that even if we ditched all the metaclass stuff, you'd see PyPy performance pretty close to what you're seeing now. The only thing I could see potentially making a more significant difference is if there were generated parsing code that switches on field number, instead of looking up fields by number in a dictionary. But given that Python doesn't have an actual switch statement, this might not be an improvement at all.

haberman 11y ago

It's really interesting to me that the top two comments on this story both wish Protocol Buffers were different, but seem to be asking for opposite things. Your comment wishes that there was a pure-Python implementation that doesn't do metaprogramming. That would imply doing code generation, but having the generated code be more "concrete", so-to-speak (containing all of the actual implementation). However, a different comment (https://news.ycombinator.com/item?id=10762101 https://news.ycombinator.com/item?id=10762101) admires the thriftpy project which doesn't have a codegen step at all. This is in some sense the opposite of what you want: everything becomes metaprogramming. I work on Protocol Buffers at Google, and I frequently observe these kinds of opposing feelings about how protobufs ought to work. One thing I've learned is how incredibly difficult it is to please everybody. Another good example of this is whether to be pure-Python: some people (like you) want that. But it's nearly impossible to make it very fast, and lots of users are sensitive to speed (you can find various comments and blog articles complaining about the speed of Python protobuf).

euyyn 11y ago
Hey Josh! Jorge from gRPC here. What's your take on Pyrobuf?
pixelmonkey 11y ago
I think we learned a long time ago that though codegen is one path to speedy code, it is not the only path. The thriftpy module dynamically generates a Python module that is based on parsing the thrift schema. That happens once: at module import time. After that, Python has cached the module in sys.modules and it doesn't need to be evaluated again during the program's runtime. The module contains efficient Python classes that are Python bytecode just the same way "compiled" classes would be. But, thriftpy has Cython implementation of Thrift protocols and transports that compile down to C, not Python bytecode. Thus, the thriftpy impl's are not only more dynamic and less cumbersome than the codegen alternative, but they are also faster. In the Python community, we prefer to take other approaches to achieve execution speed. Dynamically generating a Python module is not "metaprogramming". It is not black magic. You can generate a module yourself simply by instantiating a Python module type. Hooking the import statement is an officially supported part of the language, and done by many libraries. This may all seem very worrying to a C++ or Java programmer, but in the Python community we have been doing dynamism with execution speed for about 10 years now, and we haven't regretted any of the results!
- haberman 11y ago
  For what it's worth, I prefer the approach you describe. I have advocated for it for a long time. I've been writing C extensions to accelerate parsing in dynamic languages for 10 years, and I'm very familiar with this sort of dynamism. It's interesting though that you describe this approach as more Pythonic, because one of the most outspoken critics of this approach is a hard-core Python guy that I work with who has lots of experience with the Python ecosystem. He feels very strongly that it is more Pythonic to have very flat/concrete generated code that is transparent to the reader, and really does not like the idea of hooking import and generating everything at runtime. This is exactly what I am describing about how it is hard to please everybody. Different people appear to have fiercely different opinions about what is idiomatic. When we wrote the Ruby protobuf implementation this year, we took an approach much more along the lines of what I personally prefer. The extension is mostly implemented in C, and it's very easy to build types at runtime. It doesn't directly import .proto files (which I would have preferred) because there was still some desire from others to have some kind of code generation. So the approach we took was to use a Ruby-like DSL for describing the protobuf schema. But really this is nothing but a translation of the .proto file into the Ruby DSL. ie. take the .proto file: syntax = "proto3"; message Test { int32 foo = 1; double bar = 2; Test test = 3; } The "generated code" for this is simply: require 'google/protobuf' Google::Protobuf::DescriptorPool.generated_pool.build do add_message "Test" do optional :foo, :int32, 1 optional :bar, :double, 2 optional :test, :message, 3, "Test" end end Test = Google::Protobuf::DescriptorPool.generated_pool.lookup("Test").msgclass
JulianWasTaken 11y ago
I've certainly observed the same I think :) and yeah pleasing everyone is impossible -- I think though there's a somewhat coherent way to put at least part of those two things together. Not wanting code generation might come from people who don't like the additional build step and artifact deployment. I don't like that either, but what I mean by "less metaprogramming" is that the code that's generated currently makes copious use of descriptors, metaclasses, and other "complex" things in a way that makes it extremely JIT-unfriendly on PyPy. We had to completely abandon it in favor of using protoc-c wrapped via CFFI. So I care about speed (almost entirely -- protobuf powers systems for us that do ~350K messages per second). (I do though think it would be possible to satisfy both complaints simultaneously. I spent about 5 minutes trying to write a codegen-less, pure Python implementation: https://github.com/Julian/pb/tree/master/pb https://github.com/Julian/pb/tree/master/pb but I didn't find the Proto3 docs mature enough to figure out the binary protocol at the time. Not sure there's enough there to see what direction I was trying to go in, but I have given this a shot before). And thanks for the reply, I do agree that there is some conflicting concerns involved here.
- haberman 11y ago
  Ah, that's very interesting -- your desire for more "pure" Python comes from a desire for speed. (Some people who wish the generated code was more concrete want this so the generated code is more readable). I am curious what led you to the conclusion that descriptors and metaclasses were making Python protobuf JIT-unfriendly. I ask because most of the "meta" stuff happens at startup, when you first import the module. It uses a metaclass to generate a bunch of Python methods, but after that they are just regular Python methods, and should be as easy for PyPy to optimize as anything else. There is a bit of reflection happening in some code-paths (like the __init__ method does loop over a list of fields to decide how to initialize them), but the metaclass at least is pretty much gone after import. For this reason, I suspect that even if we ditched all the metaclass stuff, you'd see PyPy performance pretty close to what you're seeing now. The only thing I could see potentially making a more significant difference is if there were generated parsing code that switches on field number, instead of looking up fields by number in a dictionary. But given that Python doesn't have an actual switch statement, this might not be an improvement at all.