3 ms·

I got interested in Arrow recently after reading this blog post showing that Arrow (and Ray) are much faster than Pickle: https://rise.cs.berkeley.edu/blog/fast

I got interested in Arrow recently after reading this blog post showing that Arrow (and Ray) are much faster than Pickle: https://rise.cs.berkeley.edu/blog/fast-python-serialization-ray-apache-arrow/ https://rise.cs.berkeley.edu/blog/fast-python-serialization-... I have a question about whether it would fit this use-case: * I need a SUPER fast KV-store. * I'm on a single machine. * Keys are 10-bytes if you compress (or strings with 32 characters if you don't), unfortunately I can't store it as an 8-byte int. sqlite said it supports arbitrary precision numerics, but then I got burned finding out that casts integers to arbitrary precision floats and only keeps the first 14 digits of precision :\ * Values are 4-byte ints. Maybe 3 4-byte ints. * I have maybe 10B - 100B rows. * I need super fast lookup and depending upon my machine can't always cache this in memory, might need to work from disk. Would arrow be useful for this? Currently just using sqlite.

tyingq 6y ago
"much faster than Pickle" That isn't saying much, though. https://www.benfrederickson.com/images/python-serialization/speed.png https://www.benfrederickson.com/images/python-serialization/...
- jzer0cool 6y ago
  Unrelated: What is this *pickle* term origin? Python errors sometimes generate some "Could not pickle" errors and not sure what it tries to convey ...
  - tyingq 6y ago
    Pickling a cucumber stores and preserves it for later use, so the term is used for serialization. Still an odd analogy though, since you can't unpickle a pickle. I've heard freeze/thaw which is at least a little better.
    - jdub 6y ago
      It's Python. So it's Monty Python.
  - bobbylarrybobby 6y ago
    Pickle is the Python module that (de)serializes Python objects to some sort of binary format. Failure to pickle means the object couldn’t be serialized — the most common instance is probably trying to pass a lambda to a multiprocessing worker, because they don’t have names (you should use a def or functions.partial instead).
yeshengm 6y ago
Since you need fast lookup, traditional transactional row store like RocksDB seems a better fit.

tyingq 6y ago

"much faster than Pickle" That isn't saying much, though. https://www.benfrederickson.com/images/python-serialization/speed.png https://www.benfrederickson.com/images/python-serialization/...

jzer0cool 6y ago
Unrelated: What is this *pickle* term origin? Python errors sometimes generate some "Could not pickle" errors and not sure what it tries to convey ...
- tyingq 6y ago
  Pickling a cucumber stores and preserves it for later use, so the term is used for serialization. Still an odd analogy though, since you can't unpickle a pickle. I've heard freeze/thaw which is at least a little better.
  - jdub 6y ago
    It's Python. So it's Monty Python.
- bobbylarrybobby 6y ago
  Pickle is the Python module that (de)serializes Python objects to some sort of binary format. Failure to pickle means the object couldn’t be serialized — the most common instance is probably trying to pass a lambda to a multiprocessing worker, because they don’t have names (you should use a def or functions.partial instead).

yeshengm 6y ago

Since you need fast lookup, traditional transactional row store like RocksDB seems a better fit.