6 ms·

Just to set expectations while this package is decently stable and used by a number of people, both commercially and in the open source, it is basically in main

Just to set expectations while this package is decently stable and used by a number of people, both commercially and in the open source, it is basically in maintenance mode at this point in time. The intention a few years back was to build a complete analytical database on top of Julia, but have it be a mostly independent project. That vision didn't quite materialize, partly because not enough people wanted to do distributed analytical work (often a single big box is enough, particularly with a language as fast as Julia) and partly because the main developer left to pursue a PhD. I wouldn't be surprised if it got revived in the future, but for the moment, it's basically just the distributed tabular computing part (which is useful if you have that problem, but more limited than the name might imply).

StefanKarpinski 6y ago
I would add that while JuliaDB is really great for some applications, the data ecosystem is very much coalescing around DataFrames, which has recently caught up in performance with JuliaDB and has a broader more flexible API. (I'm one of the original authors of JuliaDB, so this is not a knock on the package at all.)
- electriccello 6y ago
  Any plans for DataFrames to support larger-than-memory data though? JuliaDB is super useful for this I've found, basically a solid Julia alternative to Dask.
  - StefanKarpinski 6y ago
    My current thinking is that it would make sense to implement the standard DataFrame abstraction (with all the generic functionality it offers) with a couple kinds of distributed data frame. One approach is to make a distributed stack of local data frames; another is to make a single data frame where each column is a distributed vector. Not sure which is better. Fortunately, Julia makes it really easy to experiment with these kinds of things. Of course, as soon as you have a system like that you need a scheduler for work at which point you want something like Dask or Dagger (https://github.com/JuliaParallel/Dagger.jl https://github.com/JuliaParallel/Dagger.jl).
    - oxinabox 6y ago
      We are so strong with the more general Tables.jl abstraction. Tables.jl took us a step away from having lots of types of `AbstractDataFrame`. Subtyping `AbstractDataFrame` is hard, I have to deal with two packages that do it and it is a big and not entirely documented interface. More natural is extending Tables.jl (like DataFrames and JuliaDB does). and we continue to build more tools that are table agnostic, and have APIs that DataFrames and a distributed table package special case when they can do it more efficienctly
  - g0wda 6y ago
    The basic Dask-like functionality is in Dagger.jl, we have plans to move out the distributed array functionality out of it and keep just the scheduler in there. I'm working on a set of functions to work with many DataFrames (or anything) in parallel and do it out of core if possible, it's basically like JuliaDB but with a FileTree abstraction rather than a table abstraction. http://shashi.biz/FileTrees.jl/lazy-parallel/ http://shashi.biz/FileTrees.jl/lazy-parallel/
  - phillc73 6y ago
    Would Query.jl[1] be an alternative to consider? Their README says: The package currently provides working implementations for in-memory data sources, but will eventually be able to translate queries into e.g. SQL. There is a prototype implementation of such a "query provider" for SQLite in the package, but it is experimental at this point and only works for a very small subset of queries. Still early days, but sounds like they're working on the same problem. [1] https://github.com/queryverse/Query.jl https://github.com/queryverse/Query.jl
mumblemumble 6y ago
> not enough people wanted to do distributed analytical work (often a single big box is enough, particularly with a language as fast as Julia) As someone who's in the midst of a plan to migrate off of Apache Spark, because we've discovered that, for our particular purposes, a single-machine implementation in a language with better crunching and multithreading chops than Java, can easily achieve comparable performance to a Spark cluster, while being good deal easier to develop and maintain, this comment strikes me as being very poignant. I have come to suspect that the big data ecosystem is a castle that was largely built on a nice, thick technical foundation that is largely composed of pointer chasing.
- solidasparagus 6y ago
  I think the problem is that people don't really appreciate how big your data has to be to be Big Data.
  - JulianMorrison 6y ago
    To be fair that threshold has risen substantially since the paper out of Google that kicked the whole thing off in 2003.
  - mumblemumble 6y ago
    Well, there is that. But, rewind 10 or 15 years to when Big Data was really taking off, and the world was also different in ways that meant that your data didn't need to be nearly so big, even after adjusting for hardware advances, to qualify as Big. Most the programming platforms and libraries that one might choose instead of Hadoop & friends didn't exist back then, or weren't or weren't particularly robust. So there weren't a lot of games other than Java in town. And Java (idiomatic Java, not benchmark-friendly Java) tends to reduce the volume of work you can squeeze out of a given hardware setup by introducing a hefty does of memory overhead.
- JulianMorrison 6y ago
  Big Data was initially a solution to a problem that basically nobody had except Google. It has remained such.
snicker7 6y ago
This is really sad. kdb+ is still really popular in the financial industry (with very few if any competitive alternatives, open source or otherwise). Julia has the potential to breakthrough in this area.

StefanKarpinski 6y ago

I would add that while JuliaDB is really great for some applications, the data ecosystem is very much coalescing around DataFrames, which has recently caught up in performance with JuliaDB and has a broader more flexible API. (I'm one of the original authors of JuliaDB, so this is not a knock on the package at all.)

electriccello 6y ago
Any plans for DataFrames to support larger-than-memory data though? JuliaDB is super useful for this I've found, basically a solid Julia alternative to Dask.
- StefanKarpinski 6y ago
  My current thinking is that it would make sense to implement the standard DataFrame abstraction (with all the generic functionality it offers) with a couple kinds of distributed data frame. One approach is to make a distributed stack of local data frames; another is to make a single data frame where each column is a distributed vector. Not sure which is better. Fortunately, Julia makes it really easy to experiment with these kinds of things. Of course, as soon as you have a system like that you need a scheduler for work at which point you want something like Dask or Dagger (https://github.com/JuliaParallel/Dagger.jl https://github.com/JuliaParallel/Dagger.jl).
  - oxinabox 6y ago
    We are so strong with the more general Tables.jl abstraction. Tables.jl took us a step away from having lots of types of `AbstractDataFrame`. Subtyping `AbstractDataFrame` is hard, I have to deal with two packages that do it and it is a big and not entirely documented interface. More natural is extending Tables.jl (like DataFrames and JuliaDB does). and we continue to build more tools that are table agnostic, and have APIs that DataFrames and a distributed table package special case when they can do it more efficienctly
- g0wda 6y ago
  The basic Dask-like functionality is in Dagger.jl, we have plans to move out the distributed array functionality out of it and keep just the scheduler in there. I'm working on a set of functions to work with many DataFrames (or anything) in parallel and do it out of core if possible, it's basically like JuliaDB but with a FileTree abstraction rather than a table abstraction. http://shashi.biz/FileTrees.jl/lazy-parallel/ http://shashi.biz/FileTrees.jl/lazy-parallel/
- phillc73 6y ago
  Would Query.jl[1] be an alternative to consider? Their README says: The package currently provides working implementations for in-memory data sources, but will eventually be able to translate queries into e.g. SQL. There is a prototype implementation of such a "query provider" for SQLite in the package, but it is experimental at this point and only works for a very small subset of queries. Still early days, but sounds like they're working on the same problem. [1] https://github.com/queryverse/Query.jl https://github.com/queryverse/Query.jl

mumblemumble 6y ago

> not enough people wanted to do distributed analytical work (often a single big box is enough, particularly with a language as fast as Julia) As someone who's in the midst of a plan to migrate off of Apache Spark, because we've discovered that, for our particular purposes, a single-machine implementation in a language with better crunching and multithreading chops than Java, can easily achieve comparable performance to a Spark cluster, while being good deal easier to develop and maintain, this comment strikes me as being very poignant. I have come to suspect that the big data ecosystem is a castle that was largely built on a nice, thick technical foundation that is largely composed of pointer chasing.

solidasparagus 6y ago
I think the problem is that people don't really appreciate how big your data has to be to be Big Data.
- JulianMorrison 6y ago
  To be fair that threshold has risen substantially since the paper out of Google that kicked the whole thing off in 2003.
- mumblemumble 6y ago
  Well, there is that. But, rewind 10 or 15 years to when Big Data was really taking off, and the world was also different in ways that meant that your data didn't need to be nearly so big, even after adjusting for hardware advances, to qualify as Big. Most the programming platforms and libraries that one might choose instead of Hadoop & friends didn't exist back then, or weren't or weren't particularly robust. So there weren't a lot of games other than Java in town. And Java (idiomatic Java, not benchmark-friendly Java) tends to reduce the volume of work you can squeeze out of a given hardware setup by introducing a hefty does of memory overhead.
JulianMorrison 6y ago
Big Data was initially a solution to a problem that basically nobody had except Google. It has remained such.

snicker7 6y ago

This is really sad. kdb+ is still really popular in the financial industry (with very few if any competitive alternatives, open source or otherwise). Julia has the potential to breakthrough in this area.