4 ms·

One thing I would really like to see happen is out of core data and statistics in Julia just like SAS. Not possible in either R or Python.

heydenberk 11y ago
If I understand you, this is indeed possible and has been recently implemented in Python. Take a look at dask: http://dask.pydata.org/en/latest/ http://dask.pydata.org/en/latest/ > Users interact with dask either by making graphs directly or through the dask collections which provide larger-than-memory counterparts to existing popular libraries:
- digitalzombie 11y ago
  Just to clarify, OP statement of "out of the core" means concurrent processes?
  - chubot 11y ago
    I think he means operating on data that doesn't fit in memory, and as mentioned I think R and Python both have packages for that.
- IndianAstronaut 11y ago
  Dask looks quite interesting. Though I am running into some issues with merges and such. My temporary fix is to use the chunking functionality of Pandas.
johnmyleswhite 11y ago
There's work being done to wrap Dato's SFrames library, which would solve that problem.
- tadlan 11y ago
  That's just the data structure. Fitting models would be much tougher.
  - johnmyleswhite 11y ago
    That really depends on how you fit models. Maximum likelihood estimation (or MAP estimation) for models of IID database rows is often trivial to implement using an iterative algorithm that applies to both in-memory and out-of-core data stores.
    - Nrpf 11y ago
      You mean IRLS or SGD or something like that?
      - johnmyleswhite 11y ago
        I would use SGD when I can tolerate an inexact estimator. For problems when I need an exact MLE (because, for example, I need to invoke asympotic normality to get CI's), I can usually do exact computations using less memory than IRLS would require. In particular, what I had in mind is that most black-box optimization tools (e.g. L-BFGS) only need an implementation of a function and its gradient -- both of which can be accumulated iteratively using O(P) memory and O(N * P) time per iteration for any data store that can stream all N observations. Given code that can accumulate function values and gradients, an optimization method like L-BFGS can estimate parameters for many kinds of models. In my experience, there's often no need to materialize the kinds of dense matrices that are typically used in R's default approach to GLM model fitting.

heydenberk 11y ago

If I understand you, this is indeed possible and has been recently implemented in Python. Take a look at dask: http://dask.pydata.org/en/latest/ http://dask.pydata.org/en/latest/ > Users interact with dask either by making graphs directly or through the dask collections which provide larger-than-memory counterparts to existing popular libraries:

digitalzombie 11y ago
Just to clarify, OP statement of "out of the core" means concurrent processes?
- chubot 11y ago
  I think he means operating on data that doesn't fit in memory, and as mentioned I think R and Python both have packages for that.
IndianAstronaut 11y ago
Dask looks quite interesting. Though I am running into some issues with merges and such. My temporary fix is to use the chunking functionality of Pandas.

johnmyleswhite 11y ago

There's work being done to wrap Dato's SFrames library, which would solve that problem.

tadlan 11y ago
That's just the data structure. Fitting models would be much tougher.
- johnmyleswhite 11y ago
  That really depends on how you fit models. Maximum likelihood estimation (or MAP estimation) for models of IID database rows is often trivial to implement using an iterative algorithm that applies to both in-memory and out-of-core data stores.
  - Nrpf 11y ago
    You mean IRLS or SGD or something like that?
    - johnmyleswhite 11y ago
      I would use SGD when I can tolerate an inexact estimator. For problems when I need an exact MLE (because, for example, I need to invoke asympotic normality to get CI's), I can usually do exact computations using less memory than IRLS would require. In particular, what I had in mind is that most black-box optimization tools (e.g. L-BFGS) only need an implementation of a function and its gradient -- both of which can be accumulated iteratively using O(P) memory and O(N * P) time per iteration for any data store that can stream all N observations. Given code that can accumulate function values and gradients, an optimization method like L-BFGS can estimate parameters for many kinds of models. In my experience, there's often no need to materialize the kinds of dense matrices that are typically used in R's default approach to GLM model fitting.