3 ms·

It seems like DataFrames.jl still has a ways to go before Julia can close the gap on R/data.table. I don't think these benchmarks include compilation time eithe

It seems like DataFrames.jl still has a ways to go before Julia can close the gap on R/data.table. I don't think these benchmarks include compilation time either?

ninjin 6y ago
Not that I am a heavy DataFrame user, but I have felt more at home with the comparatively light-weight TypeTables [1]. My understanding is that the rather complicated DataFrame ecosystem in Julia [2] mostly stems from whether tables should be immutable and/or typed. As far as I am aware there has not been any major push at the compiler level to speed up untyped code yet – although there should be plenty of room for improvements – which I suspect would benefit DataFrames greatly. [1]: https://github.com/JuliaData/TypedTables.jl https://github.com/JuliaData/TypedTables.jl [2]: https://typedtables.juliadata.org/stable/man/table/#dataframe https://typedtables.juliadata.org/stable/man/table/#datafram...
- pdeffebach 6y ago
  > As far as I am aware there has not been any major push at the compiler level to speed up untyped code yet – although there should be plenty of room for improvements – which I suspect would benefit DataFrames greatly. That's not quite correct. The major `source => fun => dest` API as part of DataFrames.jl was designed specifically to get around the non-typed container problem. And it definitely works. That's not the cause of slow performance. I think the reason is that, as you mentioned, DataFrames has a big API and a lot of development effort is put towards finalizing the API in preparation for 1.0. After that there will be much more focus on performance. In particular, some changes to optimize grouping may have recently been merged but didn't make it into the release by the time this test suite was run, as well as multi-threaded operations, which havent been finished yet, should speed things up a lot. That said, this new Polars library looks seriously impressive. Congrats to the developer.
SatvikBeri 6y ago
I started using Julia in December, DataFrames are in a sort of weird place because they're so much less necessary compared to e.g. Python. In Julia, you could just use a dict of arrays and get most of the benefits, thanks to libraries like Query.jl and Tables.jl. Thus the ecosystem is a lot more spread out. I actually use DataFrames much less than I used to in Python. This is mostly good, because you can apply the same operations on DataFrames, Streams, Time Series data, Differential Equations Results, etc., but it does mean that some of the specialized optimizations haven't made it into DataFrames.jl
- machinelabo 6y ago
  Those arguments apply to Python as well. There is nothing special about Julia that warrants your arguments.
  - SatvikBeri 6y ago
    I've been using Python a lot longer than I've been using Julia, and this isn't really true. Python tends towards much larger packages where everything is bundled together, and there are fairly deep language-level reasons for that. Python doesn't have major alternatives to pandas the way Julia has half a dozen alternatives to DataFrames. There is nothing like Query.jl that applies to all table-like structures in Python. In pandas, you'll see things like exponentially weighted moving averages, while DataFrames.jl is pretty much just the data structure. The centralization of the Python ecosystem and extra attention that pandas has gotten has made it much better in several ways – for example, pandas's indexing makes filtering significantly faster. These optimizations might make it to DataFrames.jl eventually, but I doubt you'll ever see the same level of centralization.
  - timClicks 6y ago
    I disagree. Python's data science community is strongly clustered around pandas, even though it's possible to use other approaches
superdimwit 6y ago
It does look like the benchmarks include compilation time.

It seems like DataFrames.jl still has a ways to go before Julia can close the gap on R/data.table. I don't think these benchmarks include compilation time either?

ninjin 6y ago

Not that I am a heavy DataFrame user, but I have felt more at home with the comparatively light-weight TypeTables [1]. My understanding is that the rather complicated DataFrame ecosystem in Julia [2] mostly stems from whether tables should be immutable and/or typed. As far as I am aware there has not been any major push at the compiler level to speed up untyped code yet – although there should be plenty of room for improvements – which I suspect would benefit DataFrames greatly. [1]: https://github.com/JuliaData/TypedTables.jl https://github.com/JuliaData/TypedTables.jl [2]: https://typedtables.juliadata.org/stable/man/table/#dataframe https://typedtables.juliadata.org/stable/man/table/#datafram...

pdeffebach 6y ago
> As far as I am aware there has not been any major push at the compiler level to speed up untyped code yet – although there should be plenty of room for improvements – which I suspect would benefit DataFrames greatly. That's not quite correct. The major `source => fun => dest` API as part of DataFrames.jl was designed specifically to get around the non-typed container problem. And it definitely works. That's not the cause of slow performance. I think the reason is that, as you mentioned, DataFrames has a big API and a lot of development effort is put towards finalizing the API in preparation for 1.0. After that there will be much more focus on performance. In particular, some changes to optimize grouping may have recently been merged but didn't make it into the release by the time this test suite was run, as well as multi-threaded operations, which havent been finished yet, should speed things up a lot. That said, this new Polars library looks seriously impressive. Congrats to the developer.

SatvikBeri 6y ago

I started using Julia in December, DataFrames are in a sort of weird place because they're so much less necessary compared to e.g. Python. In Julia, you could just use a dict of arrays and get most of the benefits, thanks to libraries like Query.jl and Tables.jl. Thus the ecosystem is a lot more spread out. I actually use DataFrames much less than I used to in Python. This is mostly good, because you can apply the same operations on DataFrames, Streams, Time Series data, Differential Equations Results, etc., but it does mean that some of the specialized optimizations haven't made it into DataFrames.jl

machinelabo 6y ago
Those arguments apply to Python as well. There is nothing special about Julia that warrants your arguments.
- SatvikBeri 6y ago
  I've been using Python a lot longer than I've been using Julia, and this isn't really true. Python tends towards much larger packages where everything is bundled together, and there are fairly deep language-level reasons for that. Python doesn't have major alternatives to pandas the way Julia has half a dozen alternatives to DataFrames. There is nothing like Query.jl that applies to all table-like structures in Python. In pandas, you'll see things like exponentially weighted moving averages, while DataFrames.jl is pretty much just the data structure. The centralization of the Python ecosystem and extra attention that pandas has gotten has made it much better in several ways – for example, pandas's indexing makes filtering significantly faster. These optimizations might make it to DataFrames.jl eventually, but I doubt you'll ever see the same level of centralization.
- timClicks 6y ago
  I disagree. Python's data science community is strongly clustered around pandas, even though it's possible to use other approaches

superdimwit 6y ago

It does look like the benchmarks include compilation time.