5 ms·

Why Julia's DataFrames Are Still Slow

60 points by johnmyleswhite 11y ago

jbssm 11y ago
Is there any good alternative library to DataFrames?
IndianAstronaut 11y ago
One thing I would really like to see happen is out of core data and statistics in Julia just like SAS. Not possible in either R or Python.
- heydenberk 11y ago
  If I understand you, this is indeed possible and has been recently implemented in Python. Take a look at dask: http://dask.pydata.org/en/latest/ http://dask.pydata.org/en/latest/ > Users interact with dask either by making graphs directly or through the dask collections which provide larger-than-memory counterparts to existing popular libraries:
  - digitalzombie 11y ago
    Just to clarify, OP statement of "out of the core" means concurrent processes?
    - chubot 11y ago
      I think he means operating on data that doesn't fit in memory, and as mentioned I think R and Python both have packages for that.
  - IndianAstronaut 11y ago
    Dask looks quite interesting. Though I am running into some issues with merges and such. My temporary fix is to use the chunking functionality of Pandas.
- johnmyleswhite 11y ago
  There's work being done to wrap Dato's SFrames library, which would solve that problem.
  - tadlan 11y ago
    That's just the data structure. Fitting models would be much tougher.
    - johnmyleswhite 11y ago
      That really depends on how you fit models. Maximum likelihood estimation (or MAP estimation) for models of IID database rows is often trivial to implement using an iterative algorithm that applies to both in-memory and out-of-core data stores.
      - Nrpf 11y ago
        You mean IRLS or SGD or something like that?
        johnmyleswhite 11y ago
        I would use SGD when I can tolerate an inexact estimator. For problems when I need an exact MLE (because, for example, I need to invoke asympotic normality to get CI's), I can usually do exact computations using less memory than IRLS would require. In particular, what I had in mind is that most black-box optimization tools (e.g. L-BFGS) only need an implementation of a function and its gradient -- both of which can be accumulated iteratively using O(P) memory and O(N * P) time per iteration for any data store that can stream all N observations. Given code that can accumulate function values and gradients, an optimization method like L-BFGS can estimate parameters for many kinds of models. In my experience, there's often no need to materialize the kinds of dense matrices that are typically used in R's default approach to GLM model fitting.
sevensor 11y ago
Is there no way to take advantage of the fact that most columns, most of the time, are filled with doubles? This is both the expected case and the thing we want to go faster. I don't know compiler design, which is why I ask.
- johnmyleswhite 11y ago
  I don't see how this could be done without putting information about the internal representation of DataFrames into the compiler. Such an approach would seem to require breaking important abstraction barriers and would couple the language to a specific back-end for data representation that might need to be replaced in the future to handle things like out-of-core data frames.
jpfr 11y ago
Relevant video from juliacon on the type system core: https://www.youtube.com/watch?v=xUP3cSKb8sI https://www.youtube.com/watch?v=xUP3cSKb8sI Julia's type system is geared towards JIT compilation. Methods are compiled when they are first called. Then, of course, full type information for the arguments is available. That's quite enough for Matlab-style code with the occasional JITted method. But Julia has one glaring disadvantage for everything beyond that: The return type of methods cannot be specified / enforced. 1) With "black box" methods (where the return type cannot be inferred as in this DataFrames article) the code becomes horribly slow. And you have to dig into the internal method representations to become aware of the type inference results. 2) It hurts the ability of Julia to produce binary executables. When the types are not 100% inferrable, the entire JIT infrastructure needs to be dragged along. 3) Types are not only an aid for the compiler, but also an aid for the programmer. With SIUnits [1] and method return types, Julia could even tell when the physics represented in the code is flawed!! If Julia's type system were stronger, it could become a prime platform to develop Computer Algebra Systems (CAS). That could lead to a great unification of symbolic and numerical "computation platforms". However, current Julia is unable to represent the mathematics encoded in the type system of open source CAS like Axiom [2]. Also note the github issue on Julia and dependent typing [3]. Imho, there is still a great potential for the Julia type system without breaking existing code. [1] https://github.com/Keno/SIUnits.jl https://github.com/Keno/SIUnits.jl [2] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.27.2331&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.27.... [3] https://github.com/JuliaLang/julia/issues/6113 https://github.com/JuliaLang/julia/issues/6113
- unfamiliar 11y ago
  Personally I would like to see Julia go full static typing. The dynamicness of its current type system seems to offer me no advantages and is constantly causing me problems with performance. My code ends up covered in type declarations because I'm never sure when types are concrete or not,which kind of defeats the point of implicit typing. A lot of the time a seamingly innocuous change will result in Array{Float64,1} getting changed to Array{Any,1} which then gets propagated automatically though the whole program, since functions can't have return types. Julia claims to be the future of numerical computing, but they seem to put more effort into coming up with fancy abstractions that no engineer or scientist will ever understand than working on basic numerics. For example, the array implementation is very hit or miss and much much slower than and has much more complicated semantics than arrays in Fortran. I think this is due to the fact that arrays are not first class objects in the language, and probably because the authors don't do much multidimensional high performance work.

jbssm 11y ago

Is there any good alternative library to DataFrames?

One thing I would really like to see happen is out of core data and statistics in Julia just like SAS. Not possible in either R or Python.

heydenberk 11y ago
If I understand you, this is indeed possible and has been recently implemented in Python. Take a look at dask: http://dask.pydata.org/en/latest/ http://dask.pydata.org/en/latest/ > Users interact with dask either by making graphs directly or through the dask collections which provide larger-than-memory counterparts to existing popular libraries:
- digitalzombie 11y ago
  Just to clarify, OP statement of "out of the core" means concurrent processes?
  - chubot 11y ago
    I think he means operating on data that doesn't fit in memory, and as mentioned I think R and Python both have packages for that.
- IndianAstronaut 11y ago
  Dask looks quite interesting. Though I am running into some issues with merges and such. My temporary fix is to use the chunking functionality of Pandas.
johnmyleswhite 11y ago
There's work being done to wrap Dato's SFrames library, which would solve that problem.
- tadlan 11y ago
  That's just the data structure. Fitting models would be much tougher.
  - johnmyleswhite 11y ago
    That really depends on how you fit models. Maximum likelihood estimation (or MAP estimation) for models of IID database rows is often trivial to implement using an iterative algorithm that applies to both in-memory and out-of-core data stores.
    - Nrpf 11y ago
      You mean IRLS or SGD or something like that?
      - johnmyleswhite 11y ago
        I would use SGD when I can tolerate an inexact estimator. For problems when I need an exact MLE (because, for example, I need to invoke asympotic normality to get CI's), I can usually do exact computations using less memory than IRLS would require. In particular, what I had in mind is that most black-box optimization tools (e.g. L-BFGS) only need an implementation of a function and its gradient -- both of which can be accumulated iteratively using O(P) memory and O(N * P) time per iteration for any data store that can stream all N observations. Given code that can accumulate function values and gradients, an optimization method like L-BFGS can estimate parameters for many kinds of models. In my experience, there's often no need to materialize the kinds of dense matrices that are typically used in R's default approach to GLM model fitting.

sevensor 11y ago

Is there no way to take advantage of the fact that most columns, most of the time, are filled with doubles? This is both the expected case and the thing we want to go faster. I don't know compiler design, which is why I ask.

johnmyleswhite 11y ago
I don't see how this could be done without putting information about the internal representation of DataFrames into the compiler. Such an approach would seem to require breaking important abstraction barriers and would couple the language to a specific back-end for data representation that might need to be replaced in the future to handle things like out-of-core data frames.

jpfr 11y ago

Relevant video from juliacon on the type system core: https://www.youtube.com/watch?v=xUP3cSKb8sI https://www.youtube.com/watch?v=xUP3cSKb8sI Julia's type system is geared towards JIT compilation. Methods are compiled when they are first called. Then, of course, full type information for the arguments is available. That's quite enough for Matlab-style code with the occasional JITted method. But Julia has one glaring disadvantage for everything beyond that: The return type of methods cannot be specified / enforced. 1) With "black box" methods (where the return type cannot be inferred as in this DataFrames article) the code becomes horribly slow. And you have to dig into the internal method representations to become aware of the type inference results. 2) It hurts the ability of Julia to produce binary executables. When the types are not 100% inferrable, the entire JIT infrastructure needs to be dragged along. 3) Types are not only an aid for the compiler, but also an aid for the programmer. With SIUnits [1] and method return types, Julia could even tell when the physics represented in the code is flawed!! If Julia's type system were stronger, it could become a prime platform to develop Computer Algebra Systems (CAS). That could lead to a great unification of symbolic and numerical "computation platforms". However, current Julia is unable to represent the mathematics encoded in the type system of open source CAS like Axiom [2]. Also note the github issue on Julia and dependent typing [3]. Imho, there is still a great potential for the Julia type system without breaking existing code. [1] https://github.com/Keno/SIUnits.jl https://github.com/Keno/SIUnits.jl [2] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.27.2331&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.27.... [3] https://github.com/JuliaLang/julia/issues/6113 https://github.com/JuliaLang/julia/issues/6113

unfamiliar 11y ago
Personally I would like to see Julia go full static typing. The dynamicness of its current type system seems to offer me no advantages and is constantly causing me problems with performance. My code ends up covered in type declarations because I'm never sure when types are concrete or not,which kind of defeats the point of implicit typing. A lot of the time a seamingly innocuous change will result in Array{Float64,1} getting changed to Array{Any,1} which then gets propagated automatically though the whole program, since functions can't have return types. Julia claims to be the future of numerical computing, but they seem to put more effort into coming up with fancy abstractions that no engineer or scientist will ever understand than working on basic numerics. For example, the array implementation is very hit or miss and much much slower than and has much more complicated semantics than arrays in Fortran. I think this is due to the fact that arrays are not first class objects in the language, and probably because the authors don't do much multidimensional high performance work.