4 ms·

You cannot share arbitrarily structured objects in the `ShareableList`, only atomic scalars and bytes / strings. If you want to share structured Python objects

You cannot share arbitrarily structured objects in the `ShareableList`, only atomic scalars and bytes / strings. If you want to share structured Python objects between instances, you have to pay the cost of `pickle.dump/pickle.dump` (CPU overhead for interprocess communication) + the memory cost of replicated objects in the processes.

tomrod 1y ago
I can fit a lot of json into bytes/strings though?
- cjbgkagh 1y ago
  Perhaps flatbuffers would be better?
  - tomrod 1y ago
    I love learning from folks on HN -- thanks! Will check it out.
    - notpushkin 1y ago
      Take a look at https://capnproto.org/ https://capnproto.org/ as well, while at it. Neither solve the copying problem, though.
      - deleted 1y ago
        [deleted]
      - frollogaston 1y ago
        Ah, I forgot capnproto doesn't let you edit a serialized proto in-memory, it's read-only. In theory this should be possible as long as you're not changing the length of anything, but I'm not surprised such trickery is unsupported. So this doesn't seem like a versatile solution for sharing data structs between two Python processes. You're gonna have to reserialize the whole thing if one side wants to edit, which is basically copying.
  - tinix 1y ago
    let me introduce you to quickle.
- vlovich123 1y ago
  That’s even worse than pickle.
  - tomrod 1y ago
    pickle pickles to pickle binary, yeah? So can stream that too with an io Buffer :D
- frollogaston 1y ago
  If all your state is already json-serializable, yeah. But that's just as expensive as copying if not more, hence what cjbgkagh said about flatbuffers.
  - frollogaston 1y ago
    oh nvm, that doesn't solve this either
- reliabilityguy 1y ago
  What’s the point? The whole idea is to share an object, and not to serialize them whether it’s json, pickle, or whatever.
  - tomrod 1y ago
    I mean, the answer to this is pretty straightforward -- because we can, not because we should :)
notpushkin 1y ago
We need a dataclass-like interface on top of a ShareableList.
- notpushkin 1y ago
  Actually, ShareableList feels like a tuple really (as it’s impossible to change its length). If we could mix ShareableList and collections.namedtuple together, it would get us 90% there (99.9% if we use typing.NamedTuple). Unfortunately, I can’t decipher either one [1, 2] from the first glance – maybe if I get some more sleep? [1]: https://github.com/python/cpython/blob/3.13/Lib/collections/__init__.py#L358 https://github.com/python/cpython/blob/3.13/Lib/collections/... [2]: https://github.com/python/cpython/blob/3.13/Lib/typing.py#L2979 https://github.com/python/cpython/blob/3.13/Lib/typing.py#L2...
sgarland 1y ago
So don’t do that? Send data to workers as primitives, and have a separate process that reads the results and serializes it into whatever form you want.

tomrod 1y ago

I can fit a lot of json into bytes/strings though?

cjbgkagh 1y ago
Perhaps flatbuffers would be better?
- tomrod 1y ago
  I love learning from folks on HN -- thanks! Will check it out.
  - notpushkin 1y ago
    Take a look at https://capnproto.org/ https://capnproto.org/ as well, while at it. Neither solve the copying problem, though.
    - deleted 1y ago
      [deleted]
    - frollogaston 1y ago
      Ah, I forgot capnproto doesn't let you edit a serialized proto in-memory, it's read-only. In theory this should be possible as long as you're not changing the length of anything, but I'm not surprised such trickery is unsupported. So this doesn't seem like a versatile solution for sharing data structs between two Python processes. You're gonna have to reserialize the whole thing if one side wants to edit, which is basically copying.
- tinix 1y ago
  let me introduce you to quickle.
vlovich123 1y ago
That’s even worse than pickle.
- tomrod 1y ago
  pickle pickles to pickle binary, yeah? So can stream that too with an io Buffer :D
frollogaston 1y ago
If all your state is already json-serializable, yeah. But that's just as expensive as copying if not more, hence what cjbgkagh said about flatbuffers.
- frollogaston 1y ago
  oh nvm, that doesn't solve this either
reliabilityguy 1y ago
What’s the point? The whole idea is to share an object, and not to serialize them whether it’s json, pickle, or whatever.
- tomrod 1y ago
  I mean, the answer to this is pretty straightforward -- because we can, not because we should :)

notpushkin 1y ago

We need a dataclass-like interface on top of a ShareableList.

notpushkin 1y ago
Actually, ShareableList feels like a tuple really (as it’s impossible to change its length). If we could mix ShareableList and collections.namedtuple together, it would get us 90% there (99.9% if we use typing.NamedTuple). Unfortunately, I can’t decipher either one [1, 2] from the first glance – maybe if I get some more sleep? [1]: https://github.com/python/cpython/blob/3.13/Lib/collections/__init__.py#L358 https://github.com/python/cpython/blob/3.13/Lib/collections/... [2]: https://github.com/python/cpython/blob/3.13/Lib/typing.py#L2979 https://github.com/python/cpython/blob/3.13/Lib/typing.py#L2...

sgarland 1y ago

So don’t do that? Send data to workers as primitives, and have a separate process that reads the results and serializes it into whatever form you want.