4 ms·
Looks awesome! One data issue I've seen go relatively unaddressed in the Clojure community is the serialization of big matrices and arrays. There's a start on
by peatmoss 13y ago
Looks awesome! One data issue I've seen go relatively unaddressed in the Clojure community is the serialization of big matrices and arrays.
There's a start on a clojure hdf5 (hdf5 is a container format common in scientific circles) implementation, but it's a long ways from done. https://github.com/clojure-numerics/clj-hdf5 https://github.com/clojure-numerics/clj-hdf5 I'm not the author, but I am the negligent steward.
I'd love it if someone smarter / better at Clojure than me was interested in helping to think about useful, idiomatic high-level abstractions on top this high-performance data store.
PyTables does a great job of making gobs of hdf5 data easy to work with for analysts--I'm just too novice at Clojure/FP to know what is a reasonable analogue for Clojure.
- prospero 13y agoWithout knowing anything about hdf5 specifically, Vertigo [1] will let you treat a memory-mapped file (or a piece of one) as a normal Clojure data structure, as long as the element types are fixed-layout. [1] https://github.com/ztellman/vertigo https://github.com/ztellman/vertigo
- Mikera 13y agoI've been thinking about adding some IO features to core.matrix. Haven't got round to it yet, but linking this with hdf5 format could be very useful.