4 ms·
Have you thought about interfacing with DuckDB for out of core processing? https://github.com/kimmolinna/DuckDB.jl https://github.com/kimmolinna/DuckDB.jl
by 1egg0myegg0 5y ago
Have you thought about interfacing with DuckDB for out of core processing?
https://github.com/kimmolinna/DuckDB.jl https://github.com/kimmolinna/DuckDB.jl
- jpsamaroo 5y agoFirstly, I'll say that we already have work started to implement out-of-core directly in Dagger: https://github.com/JuliaParallel/Dagger.jl/pull/289 https://github.com/JuliaParallel/Dagger.jl/pull/289. With that PR in place, it should be possible to define a "storage device" which is backed by a database. I haven't had a chance to actually try this, since the PR still needs quite some work and testing, but it's definitely something on my radar!