4 ms·

1) Conda helps with that quite a bit, Pandas is not a much heavier dependency than NumPy itself. 2) Depends a bit on your background, but to me this is not rea

by filmor 9y ago

1) Conda helps with that quite a bit, Pandas is not a much heavier dependency than NumPy itself. 2) Depends a bit on your background, but to me this is not really unexpected. Integers don't have a well-defined "missing" value while Floats do, so Pandas is trying to help you by not using python objects and instead converting to the "most useful" array type. It only does so if it can convert the integers without loss of precision. 3) This one I totally get, I wrote a custom, msgpack-based serialisation due to that for our usage (before Arrow was around, seriously considering that for data exchange now). 4) Apart from the changes to `resample` all of those breaking changes had a prior `DeprecationWarning`, IIRC.

RHSman2 9y ago
3) msgpack. Can you expand on that? I use this method and store/cache it for temporary passing between tasks in wf. Looking to arrow to be able to extend capability of this utility.
- filmor 9y ago
  I needed a binary, stable format to pass large dataframes efficiently to our JS frontend as well as to other Python processes. The builtin msgpack converter in Pandas was very pandas-centric in that it was basically just a msgpack representation of Pandas's internal structures, which didn't fit our purposes. In the end, I implemented a format that is essentially a list of columns, each having a specific type and some other metadata. The format for each column depends on the stored type, fallback is a msgpack sequence of strings/ints/dates mixed with nils, but array data is stored as-is and datetimes are stored either as an array of int64 with the unit attached or as begin, end, frequency.
  - RHSman2 9y ago
    Gotcha. Thanks. I see how arrow will help!
  - RHSman2 9y ago
    I just got a x3 speedup from initial tests using arrow. Then I found plasma and that has blown my mind. @wesm, how hard have you pushed plasma?
    - mycelium 9y ago
      Holy crap, thank you for the tip about Plasma. That project looks amazing. I've been in desperate need for something like this. For everyone else: http://arrow.apache.org/blog/2017/08/08/plasma-in-memory-object-store/ http://arrow.apache.org/blog/2017/08/08/plasma-in-memory-obj... Plasma, an in-memory object store that is being developed as part of Apache Arrow. Plasma holds immutable objects in shared memory so that they can be accessed efficiently by many clients across process boundaries. ... One of the goals of Apache Arrow is to serve as a common data layer enabling zero-copy data exchange between multiple frameworks. A key component of this vision is the use of off-heap memory management (via Plasma) for storing and sharing Arrow-serialized objects between applications.

by filmor 9y ago

RHSman2 9y ago

3) msgpack. Can you expand on that? I use this method and store/cache it for temporary passing between tasks in wf. Looking to arrow to be able to extend capability of this utility.

filmor 9y ago
I needed a binary, stable format to pass large dataframes efficiently to our JS frontend as well as to other Python processes. The builtin msgpack converter in Pandas was very pandas-centric in that it was basically just a msgpack representation of Pandas's internal structures, which didn't fit our purposes. In the end, I implemented a format that is essentially a list of columns, each having a specific type and some other metadata. The format for each column depends on the stored type, fallback is a msgpack sequence of strings/ints/dates mixed with nils, but array data is stored as-is and datetimes are stored either as an array of int64 with the unit attached or as begin, end, frequency.
- RHSman2 9y ago
  Gotcha. Thanks. I see how arrow will help!
- RHSman2 9y ago
  I just got a x3 speedup from initial tests using arrow. Then I found plasma and that has blown my mind. @wesm, how hard have you pushed plasma?
  - mycelium 9y ago
    Holy crap, thank you for the tip about Plasma. That project looks amazing. I've been in desperate need for something like this. For everyone else: http://arrow.apache.org/blog/2017/08/08/plasma-in-memory-object-store/ http://arrow.apache.org/blog/2017/08/08/plasma-in-memory-obj... Plasma, an in-memory object store that is being developed as part of Apache Arrow. Plasma holds immutable objects in shared memory so that they can be accessed efficiently by many clients across process boundaries. ... One of the goals of Apache Arrow is to serve as a common data layer enabling zero-copy data exchange between multiple frameworks. A key component of this vision is the use of off-heap memory management (via Plasma) for storing and sharing Arrow-serialized objects between applications.