3 ms·

OK thanks for the information. What I'm suggesting is that a two-phase workflow for analytics may actually make more sense than trying to unify things. (Here,

by chubot 9y ago

OK thanks for the information. What I'm suggesting is that a two-phase workflow for analytics may actually make more sense than trying to unify things. (Here, I'm considering analytics as distinct from big data in products, where you really need to compute on every user's data or every web page). Basically you have a disk-backed distributed column store with petabytes of data. At Google this is Dremel: "Dremel: interactive analysis of web-scale datasets" https://scholar.google.com/scholar?cluster=15217893017571539153&hl=en&as_sdt=0,5&sciodt=0,5 https://scholar.google.com/scholar?cluster=15217893017571539... And then you write SQL queries against it to load a sample of gigabytes onto a single machine. Stratified sampling is useful here, so you can representative samples of both big groups and small groups, which tends to happen a lot. And then you use something like R, Python, or Julia to analyze that subset. Despite the title of the Dremel Paper, it's not really good for interactive analysis. It's good for a quick "top K" query, which is sometimes all you need. But for anything beyond that, it's too slow. And SQL is a horrible language for writing analytics in. The data frame model is much better. SQL is awesome for loading data though. So I'm basically suggesting that the SQL/disk/distributed vs. the Programming language/memory/local distinction actually works well for most problems. You can cleanly divide your workflow into "what data do I need to answer this question?" vs. all the detailed and iterative analysis on a smaller data set. We do need a much better interchange format than CSV though.

wenc 9y ago
I'm not sure I understand exactly what you are proposing that is new. Could you help me see what I'm missing? (not being snarky, genuinely curious) 1) 2-phase workflow for analytics - Yes, this is being done today with different distributed column stores (Vertica, SQL Server columnstores etc.), queried into local dataframes. Spark can be used for that. 2) SQL good for data retrieval, dataframes better for analytics - OK. Ok, Spark can retrieve content via SQL into Spark Dataframes, and then apply RDD operations which are similar to dplyr type operations. 3) Better interchange format than CSV? OK, you can always write your SQL resultset as a Parquet file which can be natively accessed by Spark or Pandas (via PyArrow - there's Arrow again)
- chubot 9y ago
  It's not so much proposing something new, but calling into question the assumption that this is desirable: That way one can standardize on one interface for both small data and big data. For big data, I believe you want a set of operations that are efficient for data extraction: - Selecting and filtering. - Explicit and efficient joins. Disallow pathological joins not related to data extraction! - Sampling. Especially an efficient and accurate stratified sampling over groups. I believe this is missing from most databases, although I could be wrong since there are many of them I'm not familiar with. They typically have a fast inaccurate algorithm one, or a slow accurate one. Or they just allow you to sample an entire set of records from a table, not do "group by"-like semantics. I did what you would now call "data engineering" for many years and this was probably the #1 thing that came up over and over. That's pretty much it as far as I can tell. The goal is just to cut your data down to manageable size, not do anything overly smart. On the other hand, for small data, you want an expressive interface, not necessarily an efficient one. You want all the bells and whistles of R and dplyr (and I suppose Pandas, although I'm less familiar with it). A lot of those things are hard to implement efficiently, but you need them for scripting. Moreover sometimes it's just a lot easier to use an inefficient algorithm on small data, since most of the code is throwaway. I spent awhile looking at Apache Arrow a few weeks ago. I like the idea a lot. I feel like it may be trying to serve too many masters (but that's just a feeling). It definitely seems possible to underestimate the difference in requirements between small data and big data. I'm saying those differences are fundamental and not accidental, and you want different APIs so that programs are aware of the differences. Also, in most other domains there is a distinction between "interchange format" and "application format" (e.g. png/tiff files vs. photoshop files). It would be nice if that didn't exist, but it's there for a reason. This is a long argument, but some problems are more social than technical. Dumb, inefficient interchange formats solve a social problem.
  - wenc 9y ago
    Ah I wasn't sure which part of my reply was being referred to. I think there are two different phases of the data science process: (1) EDA and exploratory modeling. (2) productionizing models on the large. I believe Arrow can be part of (1) and (2). It is an in-memory format that aims to avoid serialization to/from different use points, so it is tooling and data-size agnostic. You can still sample from it and perform your exploration and analysis. Spark on the other hand and is mostly (2) -- it can be used for (1) but it is fairly weak at it due to overheads. It also doesn't support the full range of capabilities that dplyr (and other R packages) bring to the table. I would not use Spark for EDA or throwaway modeling. I think we can agree to disagree but I see a lot of advantages to having a common in-memory abstraction (such as Arrow) to small/big data whose difference is only that of cardinality. We don't have to mandate the use of such a format of course, but I think having such a format solves a large class of problems for a lot of people. At the same time, no one will stop using CSV (tabular)/JSON (hierarchical) even though these are inefficient, "dumb" formats. It is good enough for most applications and are what I consider local optimum solutions. Even Excel XLSX is a pretty good format for interchange. I am open to changing my mind but my experiences so far have not convinced me otherwise yet.

by chubot 9y ago

wenc 9y ago

I'm not sure I understand exactly what you are proposing that is new. Could you help me see what I'm missing? (not being snarky, genuinely curious) 1) 2-phase workflow for analytics - Yes, this is being done today with different distributed column stores (Vertica, SQL Server columnstores etc.), queried into local dataframes. Spark can be used for that. 2) SQL good for data retrieval, dataframes better for analytics - OK. Ok, Spark can retrieve content via SQL into Spark Dataframes, and then apply RDD operations which are similar to dplyr type operations. 3) Better interchange format than CSV? OK, you can always write your SQL resultset as a Parquet file which can be natively accessed by Spark or Pandas (via PyArrow - there's Arrow again)

chubot 9y ago
It's not so much proposing something new, but calling into question the assumption that this is desirable: That way one can standardize on one interface for both small data and big data. For big data, I believe you want a set of operations that are efficient for data extraction: - Selecting and filtering. - Explicit and efficient joins. Disallow pathological joins not related to data extraction! - Sampling. Especially an efficient and accurate stratified sampling over groups. I believe this is missing from most databases, although I could be wrong since there are many of them I'm not familiar with. They typically have a fast inaccurate algorithm one, or a slow accurate one. Or they just allow you to sample an entire set of records from a table, not do "group by"-like semantics. I did what you would now call "data engineering" for many years and this was probably the #1 thing that came up over and over. That's pretty much it as far as I can tell. The goal is just to cut your data down to manageable size, not do anything overly smart. On the other hand, for small data, you want an expressive interface, not necessarily an efficient one. You want all the bells and whistles of R and dplyr (and I suppose Pandas, although I'm less familiar with it). A lot of those things are hard to implement efficiently, but you need them for scripting. Moreover sometimes it's just a lot easier to use an inefficient algorithm on small data, since most of the code is throwaway. I spent awhile looking at Apache Arrow a few weeks ago. I like the idea a lot. I feel like it may be trying to serve too many masters (but that's just a feeling). It definitely seems possible to underestimate the difference in requirements between small data and big data. I'm saying those differences are fundamental and not accidental, and you want different APIs so that programs are aware of the differences. Also, in most other domains there is a distinction between "interchange format" and "application format" (e.g. png/tiff files vs. photoshop files). It would be nice if that didn't exist, but it's there for a reason. This is a long argument, but some problems are more social than technical. Dumb, inefficient interchange formats solve a social problem.
- wenc 9y ago
  Ah I wasn't sure which part of my reply was being referred to. I think there are two different phases of the data science process: (1) EDA and exploratory modeling. (2) productionizing models on the large. I believe Arrow can be part of (1) and (2). It is an in-memory format that aims to avoid serialization to/from different use points, so it is tooling and data-size agnostic. You can still sample from it and perform your exploration and analysis. Spark on the other hand and is mostly (2) -- it can be used for (1) but it is fairly weak at it due to overheads. It also doesn't support the full range of capabilities that dplyr (and other R packages) bring to the table. I would not use Spark for EDA or throwaway modeling. I think we can agree to disagree but I see a lot of advantages to having a common in-memory abstraction (such as Arrow) to small/big data whose difference is only that of cardinality. We don't have to mandate the use of such a format of course, but I think having such a format solves a large class of problems for a lot of people. At the same time, no one will stop using CSV (tabular)/JSON (hierarchical) even though these are inefficient, "dumb" formats. It is good enough for most applications and are what I consider local optimum solutions. Even Excel XLSX is a pretty good format for interchange. I am open to changing my mind but my experiences so far have not convinced me otherwise yet.