3 ms·
Should be (2014). More interestingly, as much as numpy and everybody advises against it, I believe that pickling data into a zstd stream is one of the fastest
by jleahy 4y ago
Should be (2014).
More interestingly, as much as numpy and everybody advises against it, I believe that pickling data into a zstd stream is one of the fastest ways of storing sets of large matrices.
The 'recommended' alternatives include numpy.save (uncompressed, which is bad when lz4 is faster than memcpy and you're saving to disk), numpy.savez (uncompressed zip files, even worse), numpy.savez_compressed (zlib zip, awful), hdf5 (one of the worlds worst formats and also using zlib), etc. I wish it wasn't the case, but it certainly seems like a good argument for pickle.
- a-dub 4y agoeven though all the metadata is weird and overengineered, i would probably still use hdf5 as it provides for interop with other numerical computing environments (matlab, julia). also hdf5 is at least securable. pickle streams are not designed for that. it's good to be able to send your data to others. fwiw. matlab .mat files are hdf5 at their core. i should also note that json is pretty bad for numerical data. the specification says nothing about how much precision to retain and printf/scanf is ridiculously slow for storing floats.
- jleahy 4y agohdf5 is extremely slow however, pickle+zstd is faster and results in smaller files.
- a-dub 4y agohdf5+zstd would likely be comparable. good luck loading those pickle files 5y from now.
- jleahy 4y agohdf5+zstd is not a thing (or at least not a thing that's interoperable or usable 5y from now). I just wish there was a good off-the-shelf solution, this stuff is not difficult.
- a-dub 4y agoyeah, just pointing out that there's nothing inherently wrong with hdf5 and that the gains you speak of are likely from just from the use of modern compression standards. maybe there's room for a simplified standard... or maybe just the addition of better compression to hdf5. (although they move slow for very good reason)
- hyperbovine 4y agoWhy? I have successfully loaded pkl files that are much older than that.
- welterde 4y agoI kind of feel like you are doing something wrong with HDF5, since for my use cases it's the fastest solution by far.
- RhysU 4y agoWhy is mmaping out of the running?
- jleahy 4y agoNo compression, and its slower than just reading a file with ‘read’ if you plan to consume the whole thing.
- mr_toad 4y ago> Should be (2014). I was wondering why it didn’t mention Apache Arrow.