7 ms·
However anything that requires some complex interprocess communication is essentially unworkable. You need to serialize and deserialize your data into wire-leve
by palish 15y ago
However anything that requires some complex interprocess communication is essentially unworkable. You need to serialize and deserialize your data into wire-level protocols, which is often prohibitively expensive, even when you can get it to work.
Would you post more info about previous attempts at this? I was about to create precisely that. Literally today.
But if it's fundamentally unworkable...
- chubot 15y agoWhat's fundamentally unworkable -- serialization?? There are plenty of options in Python -- pickle, JSON (no graphs), or protocol buffers/thrift if you want something language agnostic. Having a serialization layer in distributed applications is a pretty vanilla requirement... I am also confused about the "unworkable" comment. The "multiprocessing" module (which I have heard some not-so-good things about) automatically serializes Python objects over byte streams using pickle or some such thing.
- deleted 15y ago[deleted]
- DasIch 15y agoSerialization is very expansive compared to shared data structures. Even if not everybody is affected by that, it still makes serialization unsuitable as a general solution in this case.
- palish 15y agoI've been trying to tackle precisely that problem. Would someone please help me out and point me at a way of efficiently exposing a read-only block of memory to Python? EDIT: Looks like "MemoryView" introduced in 2.7 is the answer.
- veyron 15y ago2.7 introduced memoryview, and there are a slew of C API routines to interact with it ... as for shared memory, have you tried just mmap'ing the node from /dev/shm ?
- palish 15y agoAha! That sounds like just the ticket. I hope it's easy to use. A million thank-you's, and a Salami. EDIT: I just saw your edit. That's what I'm doing --- shm_open(), ftruncate(), then mmap(). A little annoying that it doesn't automatically kill the shared memory when the program terminates, but still...
- veyron 15y agoIt's pretty slick, and I've had good results using it in conjunction with a C program that generates data.
- deleted 15y ago[deleted]
- lubutu 15y agoI'd advise specifically against using Protobuf -- Google's support for Python seems to be pretty bad.
- ldng 15y agoMessagePack looks quite interesting (http://msgpack.org/ http://msgpack.org/) and seems to have quite a lot of binding (I don't the support level on each one though)
- mikeash 15y agoI've worked with protobufs in Python a bit, although not in production, and it seems fine. What trouble have you run into?
- lubutu 15y agoWell, the Protobuf PyPI package (which is two versions out of date) has not been able to be installed by Pip for over 3 years now. And yes, this is a known bug.