5 ms·

Show HN: Bistro – A light-weight column-oriented data processing engine

95 points by asavinov 9y ago

jitl 9y ago
An example would be great. Can you show how to do a given task with SQL, map/reduce, and your framework? Because right now I have no idea why I’d choose to learn this new stuff over using google-able tools I already know. Make your value proposition really clear.
- hoprocker 9y ago
  Agreed -- a motivating example in the Readme would really help people totally new to this concept grasp the how/why of the project.
  - sgolestane 9y ago
    On http://conceptoriented.org http://conceptoriented.org there is a link to http://conceptoriented.com http://conceptoriented.com which seem to be a demo.
    - asavinov 9y ago
      Source code for this web-app and previous projects: * UI (Angular 2): https://github.com/asavinov/sc-web https://github.com/asavinov/sc-web * REST server: https://github.com/asavinov/sc-rest https://github.com/asavinov/sc-rest * Core engine: https://github.com/asavinov/sc-core https://github.com/asavinov/sc-core - Bistro is a complete rewrite of this project There is an (old) implementation of this idea in C#: * UI (WPF): https://bitbucket.org/asavinov/dc-wpf https://bitbucket.org/asavinov/dc-wpf * Core engine (C#): https://bitbucket.org/asavinov/dce-csharp https://bitbucket.org/asavinov/dce-csharp And also some old Java implementation the engine in Java: * https://bitbucket.org/asavinov/dc-core https://bitbucket.org/asavinov/dc-core
krat0sprakhar 9y ago
Sorry for being that guy, but I just clicked into a random file in src to read the code, and found the code style (indentation etc.) to be quite weird https://github.com/asavinov/bistro/blob/master/core/src/main/java/org/conceptoriented/bistro/core/ColumnData.java https://github.com/asavinov/bistro/blob/master/core/src/main.... Might I suggest using https://github.com/google/google-java-format https://github.com/google/google-java-format for formatting?
- deleted 9y ago
  [deleted]
- deleted 9y ago
  [deleted]
- jahewson 9y ago
  This is what happens when tabs and spaces get mixed.
  - tomcam 9y ago
    So you’re a spacist?
    - jasonkostempski 9y ago
      If not, I run a space supremacist group out of my basement. Weekly meetings devoted entirely to tab-shaming.
KasianFranks 9y ago
This is neat. Vectorspace based AI calculations will benefit from this approach. Great work!
buremba 9y ago
Is it in-memory? Does it support replication or sharding? What's the main use-case? How does it differ from ORC, Parquet or Arrow? The repository doesn't have any information.
- asavinov 9y ago
  At the physical (storage) level it is in-memory and organized as a column store, that is, internally it stores a list of tables (with no data) and a list of columns (each being a Java array). It is definitely interesting and important to implement persistence (for example using Parquet or Arrow) as well as other mechanisms like sharding or replication (for big data processing, fault tolerance etc.) Yet, currently this direction has lower priority because the next task I want to focus on is in-stream analytics (an alternative to Kafka Streams). In general, the whole approach is focused on the logical level of data modeling and processing, that is, the goal is to increase performance and simplicity of development. The general idea (and hypothesis) is that defining how data is being processed using column operations is easier, more intuitive, less error-prone and easier to maintain than using purely set-operations. In other words, at logical level, it is an alternative to map-reduce, SQL, pandas and other models and frameworks where set-operations are used to process data.
  - jnordwick 9y ago
    > defining how data is being processed using column operations is easier, more intuitive, less error-prone and easier to maintain than using purely set-operations. And every APL programmer just nodded in agreement.
    - PeCaN 9y ago
      All 6 of us. :( Really though, I spend the time reading the readme thinking “This looks very cool, shame it doesn't have a nice language to go with it…”
  - heavenlyblue 9y ago
    Can you define the column operations by reducing them to a set of analogical steps performed after the query planner has run in an SQL database?
jnordwick 9y ago
Might new a cool idea, but not nearly fleshed out enough. I think a larger example instead of just individuals lines of code would be useful. Show a toy widget sales spreadsheet. What is the use case? Does it support time series? How works you do a moving average or pivot table?
- asavinov 9y ago
  > What is the use case? This is what I am trying to understand myself :) Previously I have implemented this approach as a web-app (self-service tool for working with tables where users can define columns as formulas - similar to spreadsheets). Discussed here: https://news.ycombinator.com/item?id=14351461 https://news.ycombinator.com/item?id=14351461 But it is too difficult to implement (much more resources are needed) so I switched to developing a library. Now I want to implement a server for in-stream analytics (an alternative to Kafka Streams). Hopefully, in the next version of Bistro. It will have quite significant portions of data processing logic in UDFs (retention policy, when to evaluate, how to add etc. when to do what). > Does it support time series? How works you do a moving average or pivot table? I am designing it now as a tool for stream processing (Bistro Streams) and hence it will support time series (each new row will get a time-stamp and the system will "know" how to deal with time). Current approach to user-defined functions (Evaluator interface) does not support moving average or other rolling aggregations. New API will be defined. This task has high priority since it is very important for stream analytics. Pivoting is conceptually more difficult because it is not an operation with data - it is an operation with schema (using data). Maybe some kind of ad-hoc solution will work.
  - jnordwick 9y ago
    Honestly, there is probably solid use cases for simplifying data exploration especially when dealing with large amounts of data. I have myself (as probably every developer in finance and many others) have tried to think of ways to build upon the intuitiveness and usefulness of the spreadsheet paradigm while making it more powerful. And if you can find a good way to do this along with a solid interface that non-programmers can use (Excel is the master at getting non-programmers to program), there is probably a lot of money to be made. Honestly, this looks like a query and analysis system that could be built on top of a column store so you don't need to deal with the storage part of the system yet. (I'm a big KBD+ fan). Good luck. I starred and watched the repo just to see how it progresses.
julienfr112 9y ago
How do that compare to SAS software (https://en.wikipedia.org/wiki/SAS_(software) https://en.wikipedia.org/wiki/SAS_(software)) ? Particularly the "DATA" steps.
- philkrylov 9y ago
  SAS DATA steps create new datasets (table) from one or more input datasets. As far as I understand the author, Bistro creates just a new column in an existing table.
  - asavinov 9y ago
    > Bistro creates just a new column in an existing table. What is new is that this column can use data from other tables - not necessarily this one. And eventually Bistro gets rid of joins and group-bys which are not difficult to understand but frequently their use is inappropriate, for example, in most cases, joins are used where we actually want to link two tables (new relation is not needed). Note also that we cannot avoid set operations and creating tables. Therefore, Bistro also supports table creation. Yet, it does it in a functional way, that is, by defining a new columns.
nickpeterson 9y ago
I skimmed the readme but didn't see the answer to what I regard as a basic question. How is this different from a view? I can easily make derived columns based on functions and reference those in other views (performance issues aside).
agibsonccc 9y ago
The core looks close enough to dataframes that I'd be curious to know how you compare to tablesaw: https://github.com/jtablesaw/tablesaw https://github.com/jtablesaw/tablesaw This looks neat but I'm not sure why I would care about this. There's a ton of solutions out there in the ecosystem out there already with a columnar like interface. Granted, we wrote our own as well[1] that uses the builder pattern that you then toss to an executor (our main backend is spark for this). One reason we wrote this is for persistence purposes. Being able to encode and persist a series of transforms that you can then load remotely has been very helpful for us in machine learning. We've since migrated this project to the eclipse foundation and intend on doing a rewrite of the interface as well as integrate our baked in tensor library[2] in to certain parts of the pipeline for speed purposes and handling things like computer vision workloads. In general, I always like seeing new takes on the columnar format processing approach but I'm just not seeing anything novel here. Clarification of intent would be great! [1]: https://github.com/deeplearning4j/DataVec https://github.com/deeplearning4j/DataVec [2]: https://github.com/deeplearning4j/nd4j https://github.com/deeplearning4j/nd4j
- asavinov 9y ago
  Bistro is not about a physical model and columnar (physical) representation although it relies on it. It is about logical level of representation and processing. There are at least three major questions when we want to introduce a new logical data model: * How we define columns within one table. Conceptually, it is easy, e.g., SELECT x, y, c = a+b FROM T. Yet, even in this simple case we see a controversy: this statement will create a table but our goal is not to create a table - we want to create a column (function). Bistro uses calc operation for that purpose. But it is of course not new. In pandas, for example, one can use df.apply. * How to connect several tables. RM, map-reduce and other set-oriented approaches use join which produces a new table. Here we have a similar controversy [1]: I do not want to produce a new table, it is not my goal. My goal is to link these two tables and this means creating a new column. Bistro introduces link operation for such columns. * How to aggregate data. Here a typical operation is group-by. It is an eclectic operation which combines several other operations and such an approach also has some problems [2]. Bistro changes the way data is aggregated by introducing accumulate functions which get one input (not a subset) and return one output value. An accumulate function is called for each element of the group by updating the current aggregate instead of computing the aggregate by getting the whole group. So linking and aggregation are what distinguishing Bistro from other approaches and frameworks to data processing including pandas, SQL and map-reduce. [1] https://www.researchgate.net/publication/301764816_Joins_vs_Links_or_Relational_Join_Considered_Harmful https://www.researchgate.net/publication/301764816_Joins_vs_... [2] https://www.researchgate.net/publication/316551218_From_Group-by_to_Accumulation_Data_Aggregation_Revisited https://www.researchgate.net/publication/316551218_From_Grou...
dgudkov 9y ago
Interesting idea. Columnar ETL can be quite efficient in some scenarios because frequently an ETL transformation (e.g. calculating a new column) effectively modifies an existing table, rather than creates a new one. This allows calculating only the delta, instead of re-building a new table from. This helps optimize performance and do calculations in-memory without slow disk I/O. Another advantage is that it allows performing many transformations (e.g. filtering) directly on dictionary compressed data, without decompressing it. This works well in Vertica [1] (based on C-Store DB [2]) which was our inspiration for building a light-weight ETL for business users that also uses a columnar in-memory data transformation engine [3]. [1] https://www.vertica.com/ https://www.vertica.com/ [2] http://db.csail.mit.edu/projects/cstore/ http://db.csail.mit.edu/projects/cstore/ [3] http://easymorph.com/in-memory-engine.html http://easymorph.com/in-memory-engine.html

95 points by asavinov 9y ago

jitl 9y ago

An example would be great. Can you show how to do a given task with SQL, map/reduce, and your framework? Because right now I have no idea why I’d choose to learn this new stuff over using google-able tools I already know. Make your value proposition really clear.

hoprocker 9y ago
Agreed -- a motivating example in the Readme would really help people totally new to this concept grasp the how/why of the project.
- sgolestane 9y ago
  On http://conceptoriented.org http://conceptoriented.org there is a link to http://conceptoriented.com http://conceptoriented.com which seem to be a demo.
  - asavinov 9y ago
    Source code for this web-app and previous projects: * UI (Angular 2): https://github.com/asavinov/sc-web https://github.com/asavinov/sc-web * REST server: https://github.com/asavinov/sc-rest https://github.com/asavinov/sc-rest * Core engine: https://github.com/asavinov/sc-core https://github.com/asavinov/sc-core - Bistro is a complete rewrite of this project There is an (old) implementation of this idea in C#: * UI (WPF): https://bitbucket.org/asavinov/dc-wpf https://bitbucket.org/asavinov/dc-wpf * Core engine (C#): https://bitbucket.org/asavinov/dce-csharp https://bitbucket.org/asavinov/dce-csharp And also some old Java implementation the engine in Java: * https://bitbucket.org/asavinov/dc-core https://bitbucket.org/asavinov/dc-core

krat0sprakhar 9y ago

Sorry for being that guy, but I just clicked into a random file in src to read the code, and found the code style (indentation etc.) to be quite weird https://github.com/asavinov/bistro/blob/master/core/src/main/java/org/conceptoriented/bistro/core/ColumnData.java https://github.com/asavinov/bistro/blob/master/core/src/main.... Might I suggest using https://github.com/google/google-java-format https://github.com/google/google-java-format for formatting?

deleted 9y ago
[deleted]
deleted 9y ago
[deleted]
jahewson 9y ago
This is what happens when tabs and spaces get mixed.
- tomcam 9y ago
  So you’re a spacist?
  - jasonkostempski 9y ago
    If not, I run a space supremacist group out of my basement. Weekly meetings devoted entirely to tab-shaming.

KasianFranks 9y ago

This is neat. Vectorspace based AI calculations will benefit from this approach. Great work!

buremba 9y ago

Is it in-memory? Does it support replication or sharding? What's the main use-case? How does it differ from ORC, Parquet or Arrow? The repository doesn't have any information.

asavinov 9y ago
At the physical (storage) level it is in-memory and organized as a column store, that is, internally it stores a list of tables (with no data) and a list of columns (each being a Java array). It is definitely interesting and important to implement persistence (for example using Parquet or Arrow) as well as other mechanisms like sharding or replication (for big data processing, fault tolerance etc.) Yet, currently this direction has lower priority because the next task I want to focus on is in-stream analytics (an alternative to Kafka Streams). In general, the whole approach is focused on the logical level of data modeling and processing, that is, the goal is to increase performance and simplicity of development. The general idea (and hypothesis) is that defining how data is being processed using column operations is easier, more intuitive, less error-prone and easier to maintain than using purely set-operations. In other words, at logical level, it is an alternative to map-reduce, SQL, pandas and other models and frameworks where set-operations are used to process data.
- jnordwick 9y ago
  > defining how data is being processed using column operations is easier, more intuitive, less error-prone and easier to maintain than using purely set-operations. And every APL programmer just nodded in agreement.
  - PeCaN 9y ago
    All 6 of us. :( Really though, I spend the time reading the readme thinking “This looks very cool, shame it doesn't have a nice language to go with it…”
- heavenlyblue 9y ago
  Can you define the column operations by reducing them to a set of analogical steps performed after the query planner has run in an SQL database?

jnordwick 9y ago

Might new a cool idea, but not nearly fleshed out enough. I think a larger example instead of just individuals lines of code would be useful. Show a toy widget sales spreadsheet. What is the use case? Does it support time series? How works you do a moving average or pivot table?

asavinov 9y ago
> What is the use case? This is what I am trying to understand myself :) Previously I have implemented this approach as a web-app (self-service tool for working with tables where users can define columns as formulas - similar to spreadsheets). Discussed here: https://news.ycombinator.com/item?id=14351461 https://news.ycombinator.com/item?id=14351461 But it is too difficult to implement (much more resources are needed) so I switched to developing a library. Now I want to implement a server for in-stream analytics (an alternative to Kafka Streams). Hopefully, in the next version of Bistro. It will have quite significant portions of data processing logic in UDFs (retention policy, when to evaluate, how to add etc. when to do what). > Does it support time series? How works you do a moving average or pivot table? I am designing it now as a tool for stream processing (Bistro Streams) and hence it will support time series (each new row will get a time-stamp and the system will "know" how to deal with time). Current approach to user-defined functions (Evaluator interface) does not support moving average or other rolling aggregations. New API will be defined. This task has high priority since it is very important for stream analytics. Pivoting is conceptually more difficult because it is not an operation with data - it is an operation with schema (using data). Maybe some kind of ad-hoc solution will work.
- jnordwick 9y ago
  Honestly, there is probably solid use cases for simplifying data exploration especially when dealing with large amounts of data. I have myself (as probably every developer in finance and many others) have tried to think of ways to build upon the intuitiveness and usefulness of the spreadsheet paradigm while making it more powerful. And if you can find a good way to do this along with a solid interface that non-programmers can use (Excel is the master at getting non-programmers to program), there is probably a lot of money to be made. Honestly, this looks like a query and analysis system that could be built on top of a column store so you don't need to deal with the storage part of the system yet. (I'm a big KBD+ fan). Good luck. I starred and watched the repo just to see how it progresses.

julienfr112 9y ago

How do that compare to SAS software (https://en.wikipedia.org/wiki/SAS_(software) https://en.wikipedia.org/wiki/SAS_(software)) ? Particularly the "DATA" steps.

philkrylov 9y ago
SAS DATA steps create new datasets (table) from one or more input datasets. As far as I understand the author, Bistro creates just a new column in an existing table.
- asavinov 9y ago
  > Bistro creates just a new column in an existing table. What is new is that this column can use data from other tables - not necessarily this one. And eventually Bistro gets rid of joins and group-bys which are not difficult to understand but frequently their use is inappropriate, for example, in most cases, joins are used where we actually want to link two tables (new relation is not needed). Note also that we cannot avoid set operations and creating tables. Therefore, Bistro also supports table creation. Yet, it does it in a functional way, that is, by defining a new columns.

nickpeterson 9y ago

I skimmed the readme but didn't see the answer to what I regard as a basic question. How is this different from a view? I can easily make derived columns based on functions and reference those in other views (performance issues aside).

agibsonccc 9y ago

The core looks close enough to dataframes that I'd be curious to know how you compare to tablesaw: https://github.com/jtablesaw/tablesaw https://github.com/jtablesaw/tablesaw This looks neat but I'm not sure why I would care about this. There's a ton of solutions out there in the ecosystem out there already with a columnar like interface. Granted, we wrote our own as well[1] that uses the builder pattern that you then toss to an executor (our main backend is spark for this). One reason we wrote this is for persistence purposes. Being able to encode and persist a series of transforms that you can then load remotely has been very helpful for us in machine learning. We've since migrated this project to the eclipse foundation and intend on doing a rewrite of the interface as well as integrate our baked in tensor library[2] in to certain parts of the pipeline for speed purposes and handling things like computer vision workloads. In general, I always like seeing new takes on the columnar format processing approach but I'm just not seeing anything novel here. Clarification of intent would be great! [1]: https://github.com/deeplearning4j/DataVec https://github.com/deeplearning4j/DataVec [2]: https://github.com/deeplearning4j/nd4j https://github.com/deeplearning4j/nd4j

asavinov 9y ago
Bistro is not about a physical model and columnar (physical) representation although it relies on it. It is about logical level of representation and processing. There are at least three major questions when we want to introduce a new logical data model: * How we define columns within one table. Conceptually, it is easy, e.g., SELECT x, y, c = a+b FROM T. Yet, even in this simple case we see a controversy: this statement will create a table but our goal is not to create a table - we want to create a column (function). Bistro uses calc operation for that purpose. But it is of course not new. In pandas, for example, one can use df.apply. * How to connect several tables. RM, map-reduce and other set-oriented approaches use join which produces a new table. Here we have a similar controversy [1]: I do not want to produce a new table, it is not my goal. My goal is to link these two tables and this means creating a new column. Bistro introduces link operation for such columns. * How to aggregate data. Here a typical operation is group-by. It is an eclectic operation which combines several other operations and such an approach also has some problems [2]. Bistro changes the way data is aggregated by introducing accumulate functions which get one input (not a subset) and return one output value. An accumulate function is called for each element of the group by updating the current aggregate instead of computing the aggregate by getting the whole group. So linking and aggregation are what distinguishing Bistro from other approaches and frameworks to data processing including pandas, SQL and map-reduce. [1] https://www.researchgate.net/publication/301764816_Joins_vs_Links_or_Relational_Join_Considered_Harmful https://www.researchgate.net/publication/301764816_Joins_vs_... [2] https://www.researchgate.net/publication/316551218_From_Group-by_to_Accumulation_Data_Aggregation_Revisited https://www.researchgate.net/publication/316551218_From_Grou...

dgudkov 9y ago

Interesting idea. Columnar ETL can be quite efficient in some scenarios because frequently an ETL transformation (e.g. calculating a new column) effectively modifies an existing table, rather than creates a new one. This allows calculating only the delta, instead of re-building a new table from. This helps optimize performance and do calculations in-memory without slow disk I/O. Another advantage is that it allows performing many transformations (e.g. filtering) directly on dictionary compressed data, without decompressing it. This works well in Vertica [1] (based on C-Store DB [2]) which was our inspiration for building a light-weight ETL for business users that also uses a columnar in-memory data transformation engine [3]. [1] https://www.vertica.com/ https://www.vertica.com/ [2] http://db.csail.mit.edu/projects/cstore/ http://db.csail.mit.edu/projects/cstore/ [3] http://easymorph.com/in-memory-engine.html http://easymorph.com/in-memory-engine.html