4 ms·
Does anyone know of a data serialization framework that will do the usual structs and arbitrary size matrices? I'd love to use something like thrift for seriali
by chuckcode 12y ago
Does anyone know of a data serialization framework that will do the usual structs and arbitrary size matrices? I'd love to use something like thrift for serializing matrices or other larger data over a wire or onto disk but last time I looked thrift, protocol buffers and avro were all struct oriented and there wasn't a good way to parcel up a matrix or large vector. Currently I've just been using hdf5 as the default way to store reasonably large data matrices, cubes, vectors, etc.
- robrenaud 12y agoWhat's wrong with using a repeated float to represent a vector/matrix, perhaps with a repeated int for indices if you are using a sparse representation?
- chuckcode 12y agoYes, I could take some time and write it myself. Take a look at hdf5[1] and the features they have around meta information, data chunking, page level compression for efficient reading of sub areas of data spaces. It is a non-trivial amount of functionality but allows some pretty amazing performance, for example see what pytables [2] has built on top of it. Now if they had just made it easier to attach it to a wire as well as a file. [1] http://www.hdfgroup.org/HDF5/ http://www.hdfgroup.org/HDF5/ [2] http://www.pytables.org/moin http://www.pytables.org/moin
- jjtheblunt 12y agoLisp (Clojure, ...) reader and pr would do this, no?