6 ms·
Pyrobuf: A Cython Alternative to Google's Python Protobuf Library
- vvanders 11y agoIf your looking for a speedy alternative to protobuf then you owe it to yourself to try flatbuffers. https://google.github.io/flatbuffers/ https://google.github.io/flatbuffers/
- rkwasny 11y agoDon't bother, go directly to cap'n proto: https://capnproto.org https://capnproto.org
- vvanders 11y agoLast I looked cap'n proto didn't have any msvc support.
- eco 11y agoThe last release added it (partially). "The core serialization functionality sufficient for 90% of users is available, but reflection and RPC APIs are not."
- moonchrome 11y agoSo Python API still doesn't work on Windows ? (AFAIK it's using reflection bindings from C++ ?)
- cmdrfred 11y agoLets play Cap'n proto documentation or Disney film catchphrase! 1. "If the pattern holds, there will be an infinite number of releases before the end of this month." 2. "Infinity Faster!" 3. "To infinity and beyond!" ɹɐǝʎʇɥƃıן zznq s,ʎǝusıp sı Ɛ :ɹǝsʍu∀
- moonchrome 11y agoThere are a couple of reasons I chose FB over Can'n Proto, Python and full C++ doesn't work on MSVC/Windows (this might have changed recently especially given the Clang compiler integration into MS backend) Second problem is Capnproto doesn't have structs, ie. you want something like struct Vec3 {x: float, y: float, z: float} in CapnProto you have to either make Vec3 a ref type (+ pointer size and lookup indirection for each instance) or inline it manually (especially annoying given that you must manually number CapnProto fields, maybe this is a benefit in some use cases but it's of no use to me and only creates work). CapnProto is much bigger as well - it includes it's own RPC/distributed object protocol and library - the serialization part isn't separate they are in one library - that's taking in a lot of useless code that you would have to figure out if you decided to maintain it. Also worth noting FlatBuffers on Python are actually not that fast because they are pure-python - there are Cython versions in the PR (at least last time I checked) - this doesn't matter in my use case but worth pointing out. And the Python API is very very unpythonic (C style API with even the naming completely off from PEP8) compared to CapnProto which is excellent ! I needed a cross language typed serialization format for binary files and FlatBuffers is better for that than CapnProto IMO.
- dwrensha 11y ago> the serialization part isn't separate they are in one library Although Cap'n Proto's C++ implementation is hosted in a single git repository, it does compile to several distinct libraries. You can use just the core serialization/deserialization part, libcapnp, if that's all you need. There are separate components, libcapnpc and libcapnp-rpc, for dynamic reflection and the object-capability remote procedure call system.
- pixelmonkey 11y agoThrift is a very similar system for message serialization to Protobuf. The thriftpy project implements a pure Python parser for thrift files that dynamically generates Python modules without a build/codegen step. Also has Py2, Py3, and PyPy support. Awesome library for this sort of thing. Includes a Cython impl of Thrift parsing/serialization, too: https://github.com/eleme/thriftpy https://github.com/eleme/thriftpy
- Chris2048 11y agorelevant: https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html https://martin.kleppmann.com/2012/12/05/schema-evolution-in-...
- moonchrome 11y agoWhat is the benefit of using something like Thrift to something like CapNProto or FlatBuffers ?
- pixelmonkey 11y agoI suspect it comes down to the details of the serialization format. I know that Thrift and Protobuf were developed / publicly released around the same time (~2008). They both have serialization and RPC approaches. They both have an IDL format and a compiler for codegen in static languages like C++ and Java. Thrift was adopted by the Apache O/SS community and so it can be found in some Apache projects, e.g. Thrift is used as an interop format in Apache Storm and as a serialization backend supported by Apache Parquet. Thrift has had a lot of client libraries for different languages come out over the years, so it tends to work everywhere. IIRC, CapNProto and FlatBuffers were each built as fresh takes on protobuf. So I imagine they are similar, with IDLs and serialization approaches of their own. I took a quick look at CapNProto's Python module documentation and it looks very similar to thriftpy in philosophy, but I don't know its intricate details as I haven't used it. This guide to Thrift covers all the important details of its schema approach and is meant as a cross-language guide. It should be the official docs, but alas, it's not! https://diwakergupta.github.io/thrift-missing-guide/ https://diwakergupta.github.io/thrift-missing-guide/
- diffraction 11y agowhy do we keep reinventing the wheel and not use asn.1?
- asn1argh 11y agoThe spec is too big, and it's too hard to implement correctly (let alone efficiently).
- diffraction 11y agoyou are using it all the time without realizing it. http://www.marben-products.com/asn.1/market.html http://www.marben-products.com/asn.1/market.html protobuf, swift, etc happened when an advertising company thinks it is an engineering company
- slavik81 11y agoPerhaps they should do more marketing, because speaking as a C++ developer: I picked between protobufs, flatbuffers and capt'n proto because they were easy to use, had active communities, and they had websites which explained how and why I should use their compiler/library/protocol. When I search the web for information about asn.1, I find very little that is of practical use. What library should I use? Why should I use it? How does it benchmark in comparison to the other tools? I've seen a few asn.1 library webpages and they all seem to take it as a given that I'm just looking for some way to deal with asn.1 data. They don't bother to try to convince me that their tool is efficient, or even that asn.1 is the right choice for my data in the first place.
- diffraction 11y agoyou are dismissing an old and proven technology because you feel it is not marketing to you effectively, which is the backend to the global telecom system (ss7) for the past two decades? why is this a movement in computer science to throw out old things that work? this is confusing to me like nosql. but "good marketing" is something that is called cap'n proto, sounds like a joke? if you are curious here is a good blog post https://ttsiodras.github.io/asn1.html https://ttsiodras.github.io/asn1.html i am working with asn.1 the protocol MMS
- aeroevan 11y agoIf this gets python 3 support before google's official implementation I would be much more inclined to give it a shot, but it doesn't look like it's compatible yet either.
- tburmeister 11y agoTry now - I just merged in some changes to make it Python 3 compatible.
- JulianWasTaken 11y agoMore protobuf implementations is certainly good, Google's have been really really lacking for us. It would be really nice if there was a pure Python implementation that didn't use tons of unnecessary metaprogramming.
- haberman 11y agoIt's really interesting to me that the top two comments on this story both wish Protocol Buffers were different, but seem to be asking for opposite things. Your comment wishes that there was a pure-Python implementation that doesn't do metaprogramming. That would imply doing code generation, but having the generated code be more "concrete", so-to-speak (containing all of the actual implementation). However, a different comment (https://news.ycombinator.com/item?id=10762101 https://news.ycombinator.com/item?id=10762101) admires the thriftpy project which doesn't have a codegen step at all. This is in some sense the opposite of what you want: everything becomes metaprogramming. I work on Protocol Buffers at Google, and I frequently observe these kinds of opposing feelings about how protobufs ought to work. One thing I've learned is how incredibly difficult it is to please everybody. Another good example of this is whether to be pure-Python: some people (like you) want that. But it's nearly impossible to make it very fast, and lots of users are sensitive to speed (you can find various comments and blog articles complaining about the speed of Python protobuf).
- euyyn 11y agoHey Josh! Jorge from gRPC here. What's your take on Pyrobuf?
- pixelmonkey 11y agoI think we learned a long time ago that though codegen is one path to speedy code, it is not the only path. The thriftpy module dynamically generates a Python module that is based on parsing the thrift schema. That happens once: at module import time. After that, Python has cached the module in sys.modules and it doesn't need to be evaluated again during the program's runtime. The module contains efficient Python classes that are Python bytecode just the same way "compiled" classes would be. But, thriftpy has Cython implementation of Thrift protocols and transports that compile down to C, not Python bytecode. Thus, the thriftpy impl's are not only more dynamic and less cumbersome than the codegen alternative, but they are also faster. In the Python community, we prefer to take other approaches to achieve execution speed. Dynamically generating a Python module is not "metaprogramming". It is not black magic. You can generate a module yourself simply by instantiating a Python module type. Hooking the import statement is an officially supported part of the language, and done by many libraries. This may all seem very worrying to a C++ or Java programmer, but in the Python community we have been doing dynamism with execution speed for about 10 years now, and we haven't regretted any of the results!
- dacompton 11y agoIs support for python worse than go or clojure? It seems unimaginable.
- Dowwie 11y agoReduce your objects and validate using Marshmallow , de(en)code however you like.