5 ms·

This point is interesting: So indeed, it does not matter whether the data is stored on disk or in memory --- column-stores are a win for these types of workloa

by chubot 9y ago

This point is interesting: So indeed, it does not matter whether the data is stored on disk or in memory --- column-stores are a win for these types of workloads. However, the reason is totally different. He says that for tables on disk, column layout is a win due to less data transferred from disk. But for tables in memory, the win is due to the fact that you can vectorize operations on adjacent vaules. I also think the convergence of these two approaches is interesting: one is from a database POV; the other is from a programming language POV. Column Databases came out of RDBMS research. They want to speed up SQL-like queries. Apache Arrow / Pandas came at it from the point of view of R, which descended from S [1]. It's basically an algebra for dealing with scientific data in an interactive programming language. In this view, rows are observations and columns are variables. It does seem to make sense for there to be two separate projects now, but I wonder if eventually they will converge. Probably one big difference is what write operations are supported, if any. R and I think Pandas do let you mutate a value in the middle of a column. ----- On another note, I have been looking at how to implement data frames. I believe the idea came from S, which dates back to 1976, but I only know of a few open source implementations: - R itself -- the code is known to be pretty old and slow, although rich with functionality. - Hadley Wickham's dplyr (the tbl type is an enhanced than data.frame). C++. - The data.table package in R. In C. - Pandas. In a mix of Python, Cython, and C. - Apache Arrow. In C++. - Julia DataFrames.jl. If anyone knows of others, I'm interested. [1] https://en.wikipedia.org/wiki/S_(programming_language) https://en.wikipedia.org/wiki/S_(programming_language)

drej 9y ago
Apache Spark. Scala
- chubot 9y ago
  Do you have any experience with data frames in Spark? Do they work well? My impression that it's more of an API to bolt the data frame abstraction onto whatever Apache Spark already did, rather than a native data frame representation. But I could be wrong -- I haven't used it. Is the query language Scala or something else? As far as I remember, native Spark is like MapReduce where you write the distributed functions in the same language that the framework is written in -- Scala. Also, I'm pretty sure Apache Spark is more immutable/functional, whereas in R and Python, data frames are mutable. But that model also has benefits, for redundancy and resilience to nodes being down, which I believe is the whole point of Spark. Although, I have a pet-theory that you don't really need distributed data frames. For 95% or 99% of data science, use stratified sampling to cut your data down to the size of a single machine. Then load it up in R or Python. You can make almost any decision with say 4 GiB of data. Often you can make it with 4 MB or 4 KB of data. It's not the amount of data that is the problem, but the quality.
  - joefkelley 9y ago
    Data frames do work well. You are correct that it is mainly an abstraction on top of the existing spark RDD representation. The language is Scala, though a slightly strange string/macro-based sql-ish API. Perhaps you can tell from that description I'm a bit luke-warm on it. There are a few big benefits to immutability. Redundancy and resilience is a big one, but you also get some improved performance because the compiler and planner can more aggressively optimize, and do things like predicate push-down since it doesn't have to materialize everything. It also gets you a lot of the more general programming benefits of immutability/FP in preventing bugs and such. I somewhat agree with your point about data volume. The situations that require Spark are relatively infrequent. But they are becoming more common. A lot of data is required when you start getting into building more complex machine learning models. But for a lot of your run-of-the-mill regression or basic understanding analysis, Spark is way overkill.
  - wenc 9y ago
    > Do you have any experience with data frames in Spark? Do they work well? Yes, for the most part. > Is the query language Scala or something else? You can use Spark SQL, which is essentially SQL; Spark parses it and pumps through a query planner and optimizer. You can also call methods in the Spark API (from Python, Scala, etc.) > I'm pretty sure Apache Spark is more immutable/functional. Yes, but that can be a good thing. It makes data transformations easier to reason about. > Although, I have a pet-theory that you don't really need distributed data frames. Yes, for most data science work on small datasets, you really don't. That said, there is some work being done in the Spark community to enable Spark to work as a computation engine for local datasets. That way one can standardize on one interface for both small data and big data. As of right now though, there is a fair bit of overhead if you want to use Spark on a smaller datasets -- it's way slower than R/Python dataframes. Hopefully some of those overhead issues will be addressed. This is why I think Arrow is such an exciting development. The promise is that you can use any compute engine on the same dataframe representation with no serialization overheads. So you can use nimbler tools like R and Python when the data is small, and Spark (or PySpark) on the same data when it needs to scale up. It abstracts your data away from your tooling while maintaining high performance.
wesm 9y ago
My objective, in fact, is to see the data science world unify data frame representations around Apache Arrow, so code written for Python, Julia, R, C++, etc. will all be portable across programming languages. See https://www.youtube.com/watch?v=wdmf1msbtVs https://www.youtube.com/watch?v=wdmf1msbtVs
Notre1 9y ago
For data frame implementations, you can also look at Spark's Dataset/DataFrame and Graphlab Create's SFrame. The company/team behind Graphlab Create was bought by Apple and the open sourced components haven't been updated since then. Because of that, I wouldn't use it in production, but if you are just looking for functioning implementations to compare, that gives you one more.
- chubot 9y ago
  Thanks, I never heard of Graphlab Create. It is a substantial piece of code! It says it's "out of core", which means it's probably more similar to Parquet/ORC than Arrow. But still interesting. https://github.com/turi-code/SFrame/tree/master/oss_src/sframe https://github.com/turi-code/SFrame/tree/master/oss_src/sfra... For comparison, dplyr and arrow: https://github.com/tidyverse/dplyr/tree/master/src https://github.com/tidyverse/dplyr/tree/master/src https://github.com/apache/arrow/tree/master/cpp/src/arrow https://github.com/apache/arrow/tree/master/cpp/src/arrow C++ does seem to be useful for stuff like this.
- suresk 9y ago
  I thought GraphLab Create had some nice functionality, it is too bad that the Apple acquisition appears to have effectively killed the publicly available product. It doesn't look like you can even pay for the commercial part anymore.
WaxProlix 9y ago
Do you have a working definition for what constitutes a 'data frame'? API/operations you can do to it, or maybe performance requirements of some sort? It seems like, for instance, a 2d array might qualify as a 'data frame' under some definitions.
- chubot 9y ago
  Data frame is fairly well defined, but it's a common source of confusion. In a data frame, the columns are of heterogeneous types (string, integer, bool, double, factor or "enum", etc.). In a vector/matrix, every cell has the same type. A short way to understand it is that R and Matlab aren't the same thing. They are not substitutes for one another. In R, the core type is the data frame. In Matlab, it is the matrix, of which a 2D array is a special case. (Both Matlab and R have grown some of the others' functionality, but it's not good at all. You can tell which one is primary.) Another way to say it: Julia at its core isn't an R replacement; it's a Matlab replacement. Apparently, the representation of missing values was a problem for DataFrames.jl. Whether or not individual entries in a vector are "nullable" has a large impact on how you write vectorized mathematical operations and how well it performs. And if you notice that data frames sound like SQL tables, well that's exactly why these Apache projects are converging/overlapping :) They are coming from different points of view, but they have the same underlying data model. One of my slogans is that "R is the only language without the ORM problem", because its native data model matches the data model of a relational database. I'm designing a shell which will feature data frames (eventually, probably not for a year or more): https://github.com/oilshell/oil/wiki/Oil-and-the-R-Language https://github.com/oilshell/oil/wiki/Oil-and-the-R-Language SQL is a very limited language, with essentially no capabilities for abstraction. Writing programs using the relational model in a Turing-complete language like R or Python is very powerful.
- wenc 9y ago
  A 2D array has a different algebra from a Dataframe. (though some operators are analogous, e.g. a transpose = a pivot) A dataframe is basically behaves like a database table, with all the attendant operations (aggregation, pivots, filters, etc.) The basic algebra for dataframes is relational algebra plus a few other useful database operations. The algebra for a 2D array (matrix) is linear algebra.
ACow_Adonis 9y ago
Its not publicly available, but I'm also working on an implementation of data-frame-esque structures. You've got most of the ones I'm familiar with there, barring arguably the SAS data set. Albeit that one is "row based", in terms of storage, like the database tables of yore, albeit the application of formats/types is done intellectually on a columnar level.
- chubot 9y ago
  Oh cool, what resources are you using to design the data structures? Is it just what you know from using data frames, existing code, etc.? Or any books or journal articles? The main thing I know of is that R's data.table package has the concept of indices on columns like SQL. R doesn't have that. I don't know if any of the others like Pandas, Arrow, or dplyr have it. I just Googled and found this article and a very old gist: http://appsilondatascience.com/blog/rstats/2017/03/02/r-fast-lookup.html http://appsilondatascience.com/blog/rstats/2017/03/02/r-fast... https://gist.github.com/arunsrinivasan/7997521 https://gist.github.com/arunsrinivasan/7997521 I would guess that most implementations don't have indices, and that might be an indication that they're not essential. But for someone with a computer science background it seems annoying not to use hash tables when they are so obviously applicable.
  - ACow_Adonis 9y ago
    The short answer is I'm using my own experience in the field and the limitations/frustrations I've had working with the existing ones :) As you've said, I think the inclusion of indices comes from people used to working with more of a database background speeding up frequent common queries. SAS data sets can be indexed, but I don't think there's anything intellectually to be gained there that isn't gotten just from general database indexes, since SAS data sets are designed to be on-disk data structures and operate very similarly to database tables. Talking completely off the cuff here, I imagine it has to do with the fact that these are mostly designed to be in-memory/columnar based structures, by people doing particular kinds of analysis on those structures. I'm not saying indices can't play a part, but that given the context, they're mainly in the realms of micro and query optimisations. I think one has to ask what kind of use case would an index support, and what does it gain in efficiency vs take away in complexity? For in memory columnar analysis on highly mutable data structures, its not quite so clear that it gains that much. First of all, if you're scanning an entire column, you're not gaining anything from indexes. Indeed, you're losing out because of the cost/space to store one. Second of all, if you're updating or mutating the data structure frequently in memory, then the cost of re-indexing the data structure needs to be paid each time. A cost even more questionable when in-memory access is already pretty fast. And in a lot of this field, that's a common use case. Thirdly, if you're just randomly accessing into points within the column, well usually columns are set out as collections of continuous arrays of memory. To offset into the array at a particular index i is likely faster than the double indirection of looking up an index (also in memory) and then offsetting into another point in the original array. R and the like already have the concept of indexing into a data frame using an array of integers, which you could argue not only covers the concept of an index, but possibly might even be more efficient than storing them in a hash. Lets say you want to index on a string or a complex object/combination of objects/values? Well there's a potential use case (since a string doesn't usually index straight into a low-level continuous array), but the likes of R does have the concept of row names (albeit, i can't remember how efficiently its implemented), which already lets you access a dataframe notionally based on the name of a row. Indeed, that's one of the places hashes have in my own implementation: figuring out how to index into the objects via strings labels. But I deliberately left the inclusion of row/column names optional (though column names are included by default since that seems to be the defacto norm). In practice, many programmers/programs in this space either legitimately don't need or don't use the concept. Indeed, I struggled quite intently whether to include rowname indexing as a concept at all, since I personally have barely used them in my day to day life for the last 10+ years. But I can't say there isn't a use for them. I suppose transition matrices on real world entities come to mind...
catawbasam 9y ago
Bcolz is interesting for its use of compression in both ram and disk. The ram compression is great when your data is just a little too big.

by chubot 9y ago

drej 9y ago

Apache Spark. Scala

chubot 9y ago
Do you have any experience with data frames in Spark? Do they work well? My impression that it's more of an API to bolt the data frame abstraction onto whatever Apache Spark already did, rather than a native data frame representation. But I could be wrong -- I haven't used it. Is the query language Scala or something else? As far as I remember, native Spark is like MapReduce where you write the distributed functions in the same language that the framework is written in -- Scala. Also, I'm pretty sure Apache Spark is more immutable/functional, whereas in R and Python, data frames are mutable. But that model also has benefits, for redundancy and resilience to nodes being down, which I believe is the whole point of Spark. Although, I have a pet-theory that you don't really need distributed data frames. For 95% or 99% of data science, use stratified sampling to cut your data down to the size of a single machine. Then load it up in R or Python. You can make almost any decision with say 4 GiB of data. Often you can make it with 4 MB or 4 KB of data. It's not the amount of data that is the problem, but the quality.
- joefkelley 9y ago
  Data frames do work well. You are correct that it is mainly an abstraction on top of the existing spark RDD representation. The language is Scala, though a slightly strange string/macro-based sql-ish API. Perhaps you can tell from that description I'm a bit luke-warm on it. There are a few big benefits to immutability. Redundancy and resilience is a big one, but you also get some improved performance because the compiler and planner can more aggressively optimize, and do things like predicate push-down since it doesn't have to materialize everything. It also gets you a lot of the more general programming benefits of immutability/FP in preventing bugs and such. I somewhat agree with your point about data volume. The situations that require Spark are relatively infrequent. But they are becoming more common. A lot of data is required when you start getting into building more complex machine learning models. But for a lot of your run-of-the-mill regression or basic understanding analysis, Spark is way overkill.
- wenc 9y ago
  > Do you have any experience with data frames in Spark? Do they work well? Yes, for the most part. > Is the query language Scala or something else? You can use Spark SQL, which is essentially SQL; Spark parses it and pumps through a query planner and optimizer. You can also call methods in the Spark API (from Python, Scala, etc.) > I'm pretty sure Apache Spark is more immutable/functional. Yes, but that can be a good thing. It makes data transformations easier to reason about. > Although, I have a pet-theory that you don't really need distributed data frames. Yes, for most data science work on small datasets, you really don't. That said, there is some work being done in the Spark community to enable Spark to work as a computation engine for local datasets. That way one can standardize on one interface for both small data and big data. As of right now though, there is a fair bit of overhead if you want to use Spark on a smaller datasets -- it's way slower than R/Python dataframes. Hopefully some of those overhead issues will be addressed. This is why I think Arrow is such an exciting development. The promise is that you can use any compute engine on the same dataframe representation with no serialization overheads. So you can use nimbler tools like R and Python when the data is small, and Spark (or PySpark) on the same data when it needs to scale up. It abstracts your data away from your tooling while maintaining high performance.

wesm 9y ago

My objective, in fact, is to see the data science world unify data frame representations around Apache Arrow, so code written for Python, Julia, R, C++, etc. will all be portable across programming languages. See https://www.youtube.com/watch?v=wdmf1msbtVs https://www.youtube.com/watch?v=wdmf1msbtVs

Notre1 9y ago

For data frame implementations, you can also look at Spark's Dataset/DataFrame and Graphlab Create's SFrame. The company/team behind Graphlab Create was bought by Apple and the open sourced components haven't been updated since then. Because of that, I wouldn't use it in production, but if you are just looking for functioning implementations to compare, that gives you one more.

chubot 9y ago
Thanks, I never heard of Graphlab Create. It is a substantial piece of code! It says it's "out of core", which means it's probably more similar to Parquet/ORC than Arrow. But still interesting. https://github.com/turi-code/SFrame/tree/master/oss_src/sframe https://github.com/turi-code/SFrame/tree/master/oss_src/sfra... For comparison, dplyr and arrow: https://github.com/tidyverse/dplyr/tree/master/src https://github.com/tidyverse/dplyr/tree/master/src https://github.com/apache/arrow/tree/master/cpp/src/arrow https://github.com/apache/arrow/tree/master/cpp/src/arrow C++ does seem to be useful for stuff like this.
suresk 9y ago
I thought GraphLab Create had some nice functionality, it is too bad that the Apple acquisition appears to have effectively killed the publicly available product. It doesn't look like you can even pay for the commercial part anymore.

WaxProlix 9y ago

Do you have a working definition for what constitutes a 'data frame'? API/operations you can do to it, or maybe performance requirements of some sort? It seems like, for instance, a 2d array might qualify as a 'data frame' under some definitions.

chubot 9y ago
Data frame is fairly well defined, but it's a common source of confusion. In a data frame, the columns are of heterogeneous types (string, integer, bool, double, factor or "enum", etc.). In a vector/matrix, every cell has the same type. A short way to understand it is that R and Matlab aren't the same thing. They are not substitutes for one another. In R, the core type is the data frame. In Matlab, it is the matrix, of which a 2D array is a special case. (Both Matlab and R have grown some of the others' functionality, but it's not good at all. You can tell which one is primary.) Another way to say it: Julia at its core isn't an R replacement; it's a Matlab replacement. Apparently, the representation of missing values was a problem for DataFrames.jl. Whether or not individual entries in a vector are "nullable" has a large impact on how you write vectorized mathematical operations and how well it performs. And if you notice that data frames sound like SQL tables, well that's exactly why these Apache projects are converging/overlapping :) They are coming from different points of view, but they have the same underlying data model. One of my slogans is that "R is the only language without the ORM problem", because its native data model matches the data model of a relational database. I'm designing a shell which will feature data frames (eventually, probably not for a year or more): https://github.com/oilshell/oil/wiki/Oil-and-the-R-Language https://github.com/oilshell/oil/wiki/Oil-and-the-R-Language SQL is a very limited language, with essentially no capabilities for abstraction. Writing programs using the relational model in a Turing-complete language like R or Python is very powerful.
wenc 9y ago
A 2D array has a different algebra from a Dataframe. (though some operators are analogous, e.g. a transpose = a pivot) A dataframe is basically behaves like a database table, with all the attendant operations (aggregation, pivots, filters, etc.) The basic algebra for dataframes is relational algebra plus a few other useful database operations. The algebra for a 2D array (matrix) is linear algebra.

ACow_Adonis 9y ago

Its not publicly available, but I'm also working on an implementation of data-frame-esque structures. You've got most of the ones I'm familiar with there, barring arguably the SAS data set. Albeit that one is "row based", in terms of storage, like the database tables of yore, albeit the application of formats/types is done intellectually on a columnar level.

chubot 9y ago
Oh cool, what resources are you using to design the data structures? Is it just what you know from using data frames, existing code, etc.? Or any books or journal articles? The main thing I know of is that R's data.table package has the concept of indices on columns like SQL. R doesn't have that. I don't know if any of the others like Pandas, Arrow, or dplyr have it. I just Googled and found this article and a very old gist: http://appsilondatascience.com/blog/rstats/2017/03/02/r-fast-lookup.html http://appsilondatascience.com/blog/rstats/2017/03/02/r-fast... https://gist.github.com/arunsrinivasan/7997521 https://gist.github.com/arunsrinivasan/7997521 I would guess that most implementations don't have indices, and that might be an indication that they're not essential. But for someone with a computer science background it seems annoying not to use hash tables when they are so obviously applicable.
- ACow_Adonis 9y ago
  The short answer is I'm using my own experience in the field and the limitations/frustrations I've had working with the existing ones :) As you've said, I think the inclusion of indices comes from people used to working with more of a database background speeding up frequent common queries. SAS data sets can be indexed, but I don't think there's anything intellectually to be gained there that isn't gotten just from general database indexes, since SAS data sets are designed to be on-disk data structures and operate very similarly to database tables. Talking completely off the cuff here, I imagine it has to do with the fact that these are mostly designed to be in-memory/columnar based structures, by people doing particular kinds of analysis on those structures. I'm not saying indices can't play a part, but that given the context, they're mainly in the realms of micro and query optimisations. I think one has to ask what kind of use case would an index support, and what does it gain in efficiency vs take away in complexity? For in memory columnar analysis on highly mutable data structures, its not quite so clear that it gains that much. First of all, if you're scanning an entire column, you're not gaining anything from indexes. Indeed, you're losing out because of the cost/space to store one. Second of all, if you're updating or mutating the data structure frequently in memory, then the cost of re-indexing the data structure needs to be paid each time. A cost even more questionable when in-memory access is already pretty fast. And in a lot of this field, that's a common use case. Thirdly, if you're just randomly accessing into points within the column, well usually columns are set out as collections of continuous arrays of memory. To offset into the array at a particular index i is likely faster than the double indirection of looking up an index (also in memory) and then offsetting into another point in the original array. R and the like already have the concept of indexing into a data frame using an array of integers, which you could argue not only covers the concept of an index, but possibly might even be more efficient than storing them in a hash. Lets say you want to index on a string or a complex object/combination of objects/values? Well there's a potential use case (since a string doesn't usually index straight into a low-level continuous array), but the likes of R does have the concept of row names (albeit, i can't remember how efficiently its implemented), which already lets you access a dataframe notionally based on the name of a row. Indeed, that's one of the places hashes have in my own implementation: figuring out how to index into the objects via strings labels. But I deliberately left the inclusion of row/column names optional (though column names are included by default since that seems to be the defacto norm). In practice, many programmers/programs in this space either legitimately don't need or don't use the concept. Indeed, I struggled quite intently whether to include rowname indexing as a concept at all, since I personally have barely used them in my day to day life for the last 10+ years. But I can't say there isn't a use for them. I suppose transition matrices on real world entities come to mind...

catawbasam 9y ago

Bcolz is interesting for its use of compression in both ram and disk. The ram compression is great when your data is just a little too big.