4 ms·

Is a "data frame" just a table with (edit: a lot of) storage optimization?

chubot 11y ago
As far a I know, the term "data frame" comes from R and its predecessors like S, where a data frame is the core data structure. Logically, it is indeed like an SQL table -- a column has a single type whereas a row has heterogeneous types. AFAIK, all implementations are column-oriented, which admits certain kind of implementation and optimization. SQL databases are mostly row-oriented, probably since updating a row at a time is a common operation. I would think of it as a table, but embedded in a programming language rather than a database (so you don't use SQL), with more operations, and which is very often used in a read-only fashion. The syntax in R is nicer than SQL in my opinion. It's more algebraic and composable. Instead of "SELECT name, address FROM foo WHERE age > 30", you can write foo[foo$age > 30, c('name', 'address')]. Some links: http://www.r-bloggers.com/select-operations-on-r-data-frames/ http://www.r-bloggers.com/select-operations-on-r-data-frames... Pandas is a data frame library for Python, based on R: http://pandas.pydata.org/pandas-docs/stable/basics.html http://pandas.pydata.org/pandas-docs/stable/basics.html This article explains the relevance of the relational model to data analysis / statistics (rows are observations, columns are variables): https://scholar.google.com/scholar?cluster=7796623832662932979&hl=en&as_sdt=0,5&sciodt=0,5 https://scholar.google.com/scholar?cluster=77966238326629329...
- stewbrew 11y ago
  "data frame is the core data structure" Actually, AFAIK a data.frame in R is actually a list of vectors (i.e. columns) with some constraints.
  - infinite8s 11y ago
    Same in pandas, although it's closer to a dictionary of column names to singly-typed vectors. Most simple columnar databases are structured that way as well (a more complicated design involves chunking the columns into pages).
  - chubot 11y ago
    That's not true: > d=data.frame(a=c(1,2,3),b=c(4,5,6)) > e=list(a=c(1,2,3),b=c(4,5,6)) > class(d) [1] "data.frame" > class(e) [1] "list" > d[c(TRUE,FALSE),] a b 1 1 4 3 3 6 > e[c(TRUE,FALSE),] Error in e[c(TRUE, FALSE), ] : incorrect number of dimensions They are represented similarly in R, but they are distinct data types. The data frame is the core data structure in the sense that many functions in R operate on data frames (but not lists of vectors).
    - stewbrew 11y ago
      1. You shouldn't use `=` for assignments in R but `<-`. `=` does late binding. 2. You shouldn't use `class()` here but `mode()` to check the actual underlying data structure. > mode(d) [1] "list" > mode(e) [1] "list" 3. The reason `[` works differently is because it a S3 method which invokes different functions for lists and data.frames -- that's why class(d) doesn't return "list". See `methods("[")`. See https://cran.r-project.org/doc/manuals/r-release/R-lang.html#Data-frame-objects https://cran.r-project.org/doc/manuals/r-release/R-lang.html... for details.
visarga 11y ago
It's a list of hashes, just like in MongoDB, where you can apply selection both of records and of keys inside a record and compute on them. The real advantage is that it works fast with data larger than memory. It can also be compared to the standard linux pipes. You can do head, tail, and cut-like operations on a data stream.

As far a I know, the term "data frame" comes from R and its predecessors like S, where a data frame is the core data structure. Logically, it is indeed like an SQL table -- a column has a single type whereas a row has heterogeneous types. AFAIK, all implementations are column-oriented, which admits certain kind of implementation and optimization. SQL databases are mostly row-oriented, probably since updating a row at a time is a common operation. I would think of it as a table, but embedded in a programming language rather than a database (so you don't use SQL), with more operations, and which is very often used in a read-only fashion. The syntax in R is nicer than SQL in my opinion. It's more algebraic and composable. Instead of "SELECT name, address FROM foo WHERE age > 30", you can write foo[foo$age > 30, c('name', 'address')]. Some links: http://www.r-bloggers.com/select-operations-on-r-data-frames/ http://www.r-bloggers.com/select-operations-on-r-data-frames... Pandas is a data frame library for Python, based on R: http://pandas.pydata.org/pandas-docs/stable/basics.html http://pandas.pydata.org/pandas-docs/stable/basics.html This article explains the relevance of the relational model to data analysis / statistics (rows are observations, columns are variables): https://scholar.google.com/scholar?cluster=7796623832662932979&hl=en&as_sdt=0,5&sciodt=0,5 https://scholar.google.com/scholar?cluster=77966238326629329...

stewbrew 11y ago
"data frame is the core data structure" Actually, AFAIK a data.frame in R is actually a list of vectors (i.e. columns) with some constraints.
- infinite8s 11y ago
  Same in pandas, although it's closer to a dictionary of column names to singly-typed vectors. Most simple columnar databases are structured that way as well (a more complicated design involves chunking the columns into pages).
- chubot 11y ago
  That's not true: > d=data.frame(a=c(1,2,3),b=c(4,5,6)) > e=list(a=c(1,2,3),b=c(4,5,6)) > class(d) [1] "data.frame" > class(e) [1] "list" > d[c(TRUE,FALSE),] a b 1 1 4 3 3 6 > e[c(TRUE,FALSE),] Error in e[c(TRUE, FALSE), ] : incorrect number of dimensions They are represented similarly in R, but they are distinct data types. The data frame is the core data structure in the sense that many functions in R operate on data frames (but not lists of vectors).
  - stewbrew 11y ago
    1. You shouldn't use `=` for assignments in R but `<-`. `=` does late binding. 2. You shouldn't use `class()` here but `mode()` to check the actual underlying data structure. > mode(d) [1] "list" > mode(e) [1] "list" 3. The reason `[` works differently is because it a S3 method which invokes different functions for lists and data.frames -- that's why class(d) doesn't return "list". See `methods("[")`. See https://cran.r-project.org/doc/manuals/r-release/R-lang.html#Data-frame-objects https://cran.r-project.org/doc/manuals/r-release/R-lang.html... for details.

visarga 11y ago

It's a list of hashes, just like in MongoDB, where you can apply selection both of records and of keys inside a record and compute on them. The real advantage is that it works fast with data larger than memory. It can also be compared to the standard linux pipes. You can do head, tail, and cut-like operations on a data stream.