5 ms·
If I understand you, this is indeed possible and has been recently implemented in Python. Take a look at dask: http://dask.pydata.org/en/latest/ http://dask.pyd
by heydenberk 11y ago
If I understand you, this is indeed possible and has been recently implemented in Python. Take a look at dask: http://dask.pydata.org/en/latest/ http://dask.pydata.org/en/latest/
> Users interact with dask either by making graphs directly or through the dask collections which provide larger-than-memory counterparts to existing popular libraries:
- digitalzombie 11y agoJust to clarify, OP statement of "out of the core" means concurrent processes?
- chubot 11y agoI think he means operating on data that doesn't fit in memory, and as mentioned I think R and Python both have packages for that.
- IndianAstronaut 11y agoDask looks quite interesting. Though I am running into some issues with merges and such. My temporary fix is to use the chunking functionality of Pandas.