4 ms·
It might seem odd, but I achieve a similar thing by scripting my data processing tasks with GNU Make. You can incrementally add steps to the process, and it wi
by thaw13579 8y ago
It might seem odd, but I achieve a similar thing by scripting my data processing tasks with GNU Make. You can incrementally add steps to the process, and it will retain the existing results as you go along. My use case is scientific computing, where some of the steps may be the results of a costly optimization or simulation. I can update the workflow quite easily and not worry about redoing the expensive bits or making copies. Of course, all of the dependency management and parallel processing of Make are a bonus as well.
- dlcmh 8y agoThanks, I’d never thought of doing this before - found 2 resources to help me explore this approach: http://zmjones.com/make/ http://zmjones.com/make/ http://blog.kaggle.com/2012/10/15/make-for-data-scientists/ http://blog.kaggle.com/2012/10/15/make-for-data-scientists/
- leethargo 8y agoDoes not sound odd, but in your case you will store results on disk, rather than in memory? That means that there might be some overhead in (de-)serialization and also some restrictions on what kind of data can be stored as an intermediate result?
- jacobush 8y agoProbably but it's not a given. It depends on the tools. The tools might work with memory mapped data structures directly.
- AboutTheWhisles 8y agoYou can mitigate the reading from disk by memory mapping the file. I don't think serialization is a problem either. It is possible to make data structures that always exist as a single span of memory, so that they don't have a separate serialized representation. Most data structures are better served like this anyway, since prefetching and linear memory access is a huge speedup over pointer chasing. Building a data structure out of many separate heap allocations will also slow it down.
- gnulinux 8y agoYou can memory map and store binary, in-memory data (as long as you don't use pointers). So there'll be some overhead serializing (removing pointers) but ime it's not noticeable since most serialization tasks can be done in milliseconds (even if the dataset is Terabytes) whereas most algorithmic tasks are in the order of seconds, minutes or hours.
- gnulinux 8y agoSame! I use GNU Make for this. TBH I've never seen anyone using Make for this and my coworkers think it's a bit odd. But I think it's very practical.