4 ms·
HDF5 is a hierarchical data storage format, it doesn't store or 'execute' code or retrieve language objects. In fact, it has nothing more to do with python or p
by Demiurge 13y ago
HDF5 is a hierarchical data storage format, it doesn't store or 'execute' code or retrieve language objects. In fact, it has nothing more to do with python or python data structures than NetCDF, geotiff or .mkv.
I guess this comparison is just saying "don't use pickle to efficiently store data"... Which is a bit of a 'dur'...
- tomrod 13y agoWhat's a better way to store, say, a dict in python? I've not used python for repeated input/output before, so this is an open question for me.
- reidrac 13y agoI would use JSON. I'm using python-memcached in a project and I was hitting memcache object size limit (python-memcached uses pickle by default). I wrote two simple functions to serialize/unserialize to/from JSON and compress the serialized data and the result was great and way below memcache limit. Sure, pickle is slightly faster, but in my experience JSON + compression is a good compromise between simplicity, size and speed.
- ericcholis 13y agoLets not forget portability to any number of languages
- stock_toaster 13y agoIf you use python-memcached, make sure to set pickle protocol to 2 (current highest protocol). The default is 0. You can see an improvement[1] in performance by using proto 2. [1]: https://read.twobitfifo.net/article/the-venerable-pickle/ https://read.twobitfifo.net/article/the-venerable-pickle/