3 ms·
If the interpreter state contains large variables checkpointing might not be viable (eg I have dataframes that are 100s of GB/large fractions of total available
by ivalm 7y ago
If the interpreter state contains large variables checkpointing might not be viable (eg I have dataframes that are 100s of GB/large fractions of total available memory, reading/writing from hard drive all the time would be relatively slow. If you can save deltas I guess it wouldn't be too space inefficient but I imagine still slow).
At the same time, I do like the idea of an append only notebook where you can:
1. Only run cells in sequential order
2. Only edit cells that are below the most recently ran cell.
Thankfully you can enforce it through code practice and the notebook is relatively guaranteed to be "run all"-able. You will need to refactor it after the initial dirty run, but at least it's easy to reason about.
- roblabla 7y agoAssuming the data isn't changed, thanks to CoW forking wouldn't cause any extra memory usage. If only a subset of data is changed, same thing - only the changed cells will take extra space. The problem only occurs when the whole variable changes - in which case yeah, you're SOL. I wonder what the usage patterns are for such datasets?
- ivalm 7y agoPersonal experience: when first looking at the data I often do lots of map /reduce style operations which might transform large portions of the dataframe. Question, if you use CoW then presumably your variable blocks are no longer contiguous, wouldn't this really slow down vector operations?
- roblabla 7y ago> Question, if you use CoW then presumably your variable blocks are no longer contiguous, wouldn't this really slow down vector operations? I don't think so. Vector operations require the data to be aligned to whatever the vector size is, no? E.G. 16-byte vector ops require the data to be aligned to 16-byte, etc... At least that's my understanding.
- mumblemumble 7y agoI want a notebook situation where the platform understand sampling, so that, while I'm doing my EDA and initial development and generally doing the kinds of work that are appropriate to do in notebook, I'm never working with 100GB data frames. I suspect that a big part of my annoyance about the current state of the data space is that parts of the ecosystem were designed with the needs of data scientists in mind, and other parts of the ecosystem were designed with the needs of data engineers in mind, and it's all been jammed together in a way that makes sure nobody can ever be happy.
- ivalm 7y agoYou can sample data if you want already (or sequentially load partial data, which is what I usually do if I just want to test basic transformations), but if you need to worry about rare occurrences (and don't know the rate) then sampling can be dangerous. For example, when validating data there are edge cases that are very rare (ie sometimes I catch issues that are less than one record per billion), it can be hard to catch them without looking at all of the data.