9 ms·
JuliaDB
- smabie 6y agoHow does JuliaDB compare to DataFrame.jl? I know JuliaDB has row indices, which is a very nice feature that DataFrame lacks. Is there any reason I shouldn't use JuliaDB even if I don't need persistence. I wish the Julia ecosystem was a little more integrated: there are a lot of different competing libraries that ostensibly do the same thing. Python has the advantage that it's obvious what you should use: numpy, Pandas, scipy, statsmodels, matplotlib, etc. With Julia, it's less clear. Though I think part of the reason is that actually releasing a new scientific computing Python library is incredibly difficult and requires a lot of expertise. Julia makes it pretty trivial for anyone to contribute a model that has excellent performance. This fragmentation is a common problem among expressive languages.
- elalaouifaris 6y agoI think that this is changing when you see how the community is organized on github. There is a Julia something (or specific name like sciml) for all major applications. I have rarely seen this level of integration in a language community.
- xiaodai 6y agoPartly, it is because Julia is newish and partly because of the Lisp curse that Julia suffers from. But nonsustsined efforts will fall by the way side and true gems will emerge as the clear front runner like DataFrames.jl
- akdor1154 6y agoI find the IndexedTables (the underlying dataframe-like structure) model to be nicer to work with DataFrames.jl . It's much narrower API but it's well designed enough that this is a good thing, imo. The codebase is also a fair bit smaller. It also uses strongly typed tables (e.g. Table<int, string> etc), whereas DataFrames is loosely typed. Again I think this is a good decision (though it does grate with the Julia JIT's property of "being slow" the first time you run a function on a new type) Finally its split into IndexedTables and NDSparse is again a good design decision that I have not seen replicated in any other dataframe library. It just seems all around better designed. On the other hand it is verging on being unmaintained.
- jabl 6y agoBefore numpy became the standard there was Numeric and Numarray, with painful rewrites for whoever had committed to those. And there was Scientific Python building upon those (I think it started using Numeric, then later on was ported on top of numpy), similar to scipy, but that is more or less dead nowadays. And before matplotlib became the standard, there was a bunch of older plotting libs; I recall using something that was a wrapper around gnuplot, IIRC. So I think it all mostly has to do with the fact that Julia is much newer and many even foundational areas are still in a state of flux. In a few decades the dust will have mostly settled. Of course, in the areas where people are pushing the envelope, it's good that a lot of approaches are tried out before the world settles on a winner.
- Tarrosion 6y agoI wish this website spent its banner space on "what it is" rather than "star us on GitHub!". I doubly wish this because it's pretty confusing what JuliaDB is! From the name, I expect it to be a database, but the website immediately compares it to Pandas. I think of Pandas as a library for in-memory manipulation and analysis of tabular data, not a database / persistence engine. So is JuliaDB not a database at all, but in fact an alternative to Pandas / DataFrames.jl? In fact, I've been using Julia for work and following the ecosystem since version 0.4 (we're at 1.5 now), and I'm still not sure what JuliaDB is. No doubt this is mostly due to me not having reason to look very deeply (and/or not being very perceptive), but certainly doesn't feel like the marketing copy is giving me any help...
- vosper 6y agoI think it is actually a database, so it seems they haven't committed the cardinal sin of putting "DB" in the name of something that's not a database. > JuliaDB is a pure Julia analytical database. It makes loading large datasets and playing with them easy and fast. JuliaDB needs to support a number of features: releational database operations, quickly parsing text files, parallel computing, data storage and compression. Got this from https://juliadb.org/talk/juliacon2018shashi/ https://juliadb.org/talk/juliacon2018shashi/ which appears to go into more details > This talk is a bottom-up look at the construction of JuliaDB. We will talk about the scope and implementation of underlying building block packages, namely IndexedTables, TextParse, Dagger, OnlineStats and PooledArrays.
- gopalv 6y ago> they haven't committed the cardinal sin of putting "DB" in the name of something that's not a database It doesn't seem to store data in any way - so it is definitely a data processing engine, but a database without an "INSERT" command feels a little off. From that point of view, this looks a lot like what original MapReduce did - the data lives outside it & is referenced as urls, but the engine itself does processing out-of-core and in-memory for very large datasets.
- 6y ago
- lma21 6y agowhy is it being compared to pandas ? is it a data analysis library?
- eigenspace 6y agoI always found it rather ironic that this package, which was developed by JuliaComputing, manages to violate two common conventions for package names in the julia ecosystem 1) don't put "julia" in the package name 2) don't use abbreviations in the package name Of course, JuliaDB is probably older than these conventions, but it's amusing nonetheless.
- boromi 6y agoVery true and confusing
- StefanKarpinski 6y agoIt was originally a working title and then we just never found a name we liked better, but yeah, that is pretty funny.
- elcomet 6y agoIt's not only a Julia package though, is it? If it's a full database engine, it might be usable from other languages. So the name makes sense.
- eigenspace 6y agoIt's 'just' a julia package.
- j88439h84 6y agoIt looks like barely faster than Pandas?
- nine_k 6y agoHey, Pandas wrap heavily optimized C and Fortran code, with a thin layer of Python interface on top. This thing is pure Julia. This being any faster than Pandas is a huge compliment to Julia the language in general and its JIT compiler in particular.
- srean 6y ago> This being any faster than Pandas is a huge compliment to Julia the language in general and its JIT compiler in particular. Indeed. This 'whole stack under the same language' is an important feature to have. You get to reap the advantages of an improved JIT. End to end autodiff is easier. Some of these things are a problem, for example, in PyPy because of the Python <-> C bridge.
- KenoFischer 6y agoJust to set expectations while this package is decently stable and used by a number of people, both commercially and in the open source, it is basically in maintenance mode at this point in time. The intention a few years back was to build a complete analytical database on top of Julia, but have it be a mostly independent project. That vision didn't quite materialize, partly because not enough people wanted to do distributed analytical work (often a single big box is enough, particularly with a language as fast as Julia) and partly because the main developer left to pursue a PhD. I wouldn't be surprised if it got revived in the future, but for the moment, it's basically just the distributed tabular computing part (which is useful if you have that problem, but more limited than the name might imply).
- StefanKarpinski 6y agoI would add that while JuliaDB is really great for some applications, the data ecosystem is very much coalescing around DataFrames, which has recently caught up in performance with JuliaDB and has a broader more flexible API. (I'm one of the original authors of JuliaDB, so this is not a knock on the package at all.)
- electriccello 6y agoAny plans for DataFrames to support larger-than-memory data though? JuliaDB is super useful for this I've found, basically a solid Julia alternative to Dask.
- StefanKarpinski 6y agoMy current thinking is that it would make sense to implement the standard DataFrame abstraction (with all the generic functionality it offers) with a couple kinds of distributed data frame. One approach is to make a distributed stack of local data frames; another is to make a single data frame where each column is a distributed vector. Not sure which is better. Fortunately, Julia makes it really easy to experiment with these kinds of things. Of course, as soon as you have a system like that you need a scheduler for work at which point you want something like Dask or Dagger (https://github.com/JuliaParallel/Dagger.jl https://github.com/JuliaParallel/Dagger.jl).
- nine_k 6y agoSo, it's a Julia implementation of something similar to Pandas / numpy. It's not a "DB" in the sense Postgres, SQLite, or dbm are "DBs", it's not persistent and data must fit in RAM. It's cool that it's pure Julia, so it's instantly portable everywhere Julia runs, and the code is safer than it could be were portions of it written in C.
- electriccello 6y agoDifference from DataFrames: it's possible to use on larger-than memory data, as long as you have it in separate csv files which can be super useful at times!