4 ms·

> What is the use case? This is what I am trying to understand myself :) Previously I have implemented this approach as a web-app (self-service tool for workin

> What is the use case? This is what I am trying to understand myself :) Previously I have implemented this approach as a web-app (self-service tool for working with tables where users can define columns as formulas - similar to spreadsheets). Discussed here: https://news.ycombinator.com/item?id=14351461 https://news.ycombinator.com/item?id=14351461 But it is too difficult to implement (much more resources are needed) so I switched to developing a library. Now I want to implement a server for in-stream analytics (an alternative to Kafka Streams). Hopefully, in the next version of Bistro. It will have quite significant portions of data processing logic in UDFs (retention policy, when to evaluate, how to add etc. when to do what). > Does it support time series? How works you do a moving average or pivot table? I am designing it now as a tool for stream processing (Bistro Streams) and hence it will support time series (each new row will get a time-stamp and the system will "know" how to deal with time). Current approach to user-defined functions (Evaluator interface) does not support moving average or other rolling aggregations. New API will be defined. This task has high priority since it is very important for stream analytics. Pivoting is conceptually more difficult because it is not an operation with data - it is an operation with schema (using data). Maybe some kind of ad-hoc solution will work.

jnordwick 9y ago
Honestly, there is probably solid use cases for simplifying data exploration especially when dealing with large amounts of data. I have myself (as probably every developer in finance and many others) have tried to think of ways to build upon the intuitiveness and usefulness of the spreadsheet paradigm while making it more powerful. And if you can find a good way to do this along with a solid interface that non-programmers can use (Excel is the master at getting non-programmers to program), there is probably a lot of money to be made. Honestly, this looks like a query and analysis system that could be built on top of a column store so you don't need to deal with the storage part of the system yet. (I'm a big KBD+ fan). Good luck. I starred and watched the repo just to see how it progresses.
- asavinov 9y ago
  Honestly, this looks like a query and analysis system that could be built on top of a column store so you don't need to deal with the storage part of the system yet. (I'm a big KBD+ fan). I like this idea because it allows for reusing an existing engine and, what is important, integrate it with many different engines (which might have already quite sophisticated mechanisms of data managements). Yet, such an engine has to expose quite low-level operations with data. Also, there has to be support for user-defined functions (lambdas).

jnordwick 9y ago

Honestly, there is probably solid use cases for simplifying data exploration especially when dealing with large amounts of data. I have myself (as probably every developer in finance and many others) have tried to think of ways to build upon the intuitiveness and usefulness of the spreadsheet paradigm while making it more powerful. And if you can find a good way to do this along with a solid interface that non-programmers can use (Excel is the master at getting non-programmers to program), there is probably a lot of money to be made. Honestly, this looks like a query and analysis system that could be built on top of a column store so you don't need to deal with the storage part of the system yet. (I'm a big KBD+ fan). Good luck. I starred and watched the repo just to see how it progresses.

asavinov 9y ago
Honestly, this looks like a query and analysis system that could be built on top of a column store so you don't need to deal with the storage part of the system yet. (I'm a big KBD+ fan). I like this idea because it allows for reusing an existing engine and, what is important, integrate it with many different engines (which might have already quite sophisticated mechanisms of data managements). Yet, such an engine has to expose quite low-level operations with data. Also, there has to be support for user-defined functions (lambdas).