50 ms·
I think they mentioned pickling because that's what the multiprocessing queue uses by default. I think Pipe isn't necessarily a drop in replacement depending o
by nijave 4y ago
I think they mentioned pickling because that's what the multiprocessing queue uses by default.
I think Pipe isn't necessarily a drop in replacement depending on the complexity of object you want to share but I have found it significantly faster for simple things.
- while_true_ 4y agoExactly. My assumption is Pipe() is faster for short and simple data because I assume it doesn't do the serialization conversion that Pickle does.
- nijave 4y agoYup, if you need to serialize a complex object manually before piping then you end up paying the price again (like this the multiprocessing Queue) I'm guessing that's part of the reason the article didn't mention it (it looks like they're talking about a Pandas DataFrame which I would say is non-trivial--compared to a primitive type) I'd think Pipe + Parquet should beat filesystem though. That really depends on storage I guess Iirc I played a bit with msgpack and orjson to see if there was anything to gain over Pickle but I don't think it made much difference. You'd probably need to deal with structs Looking at CPython source (3.10), on Windows, you always get a NamedPipe. On other platforms, you get a OS pipe when duplex=False otherwise a socket (socket.socketpair)