6 ms·

I don't know Riak, other than its a distributed NoSQL key-value data store. Time series has always been prevalent in the fintec and quantitative finance, and o

by eggy 10y ago

I don't know Riak, other than its a distributed NoSQL key-value data store. Time series has always been prevalent in the fintec and quantitative finance, and other disciplines for decades. I read a book in the early 1990s on music as time series data, financial tickers, and so on. How is Riak different, or more suited to use than Kdb + q, J with JDB (free), Jd (a commercial J database like Kdb/q)[2], or the new Kerf lang/db being developed by Kevin Lawler[3]? Kevin also wrote kona, an opensource version of the "K programming language"[4]. Kdb is very fast at time series analysis on large datasets, and has many years of proven value in the financial industry. [1] https://kx.com/ https://kx.com/ [2] http://www.jsoftware.com/jdhelp/overview.html http://www.jsoftware.com/jdhelp/overview.html [3] https://github.com/kevinlawler/kerf https://github.com/kevinlawler/kerf [4] https://github.com/kevinlawler/kona https://github.com/kevinlawler/kona

MasterScrat 10y ago
> How is Riak different, or more suited to use than Kdb + q, J with JDB (free), Jd (a commercial J database like Kdb/q)[2], or the new Kerf lang/db being developed by Kevin Lawler[3]? I would really be interested to see an informed answer to this question!
anthonybsd 10y ago
KDB is not distributed and K (APL) is not particularly pleasant to work with. While it has a proven track record in fintech no one I know of is particularly fond of working with this technology. Not to mention the cost of K developers (200K+). Riak is simply offering a free alternative to these systems that is very palatable.
- gricardo99 10y ago
  I’m not sure what you mean by "not distributed". With kdb+ you have a lot of flexibility in how to setup the database. You can organize the data to be stored in a distributed fashion (across multiple devices, multiple servers), you can setup query load balancers to distribute work-loads, and you can replicate to multiple servers/devices. You don’t have to use K, you code in q, which most people find far easier to read/write. There’s a wealth of information to help with all this[1], and a very responsive user group. But yes, you do need kdb+ expertise to get full use out of the tool. And yes, feelings seem to run strong towards kdb+, in both directions, love/hate it. And correct again, it’s not free and the licensing cost is definitely a hurdle to wider adoption. [1] - http://code.kx.com http://code.kx.com
  - anthonybsd 10y ago
    >I’m not sure what you mean by "not distributed". With kdb+ you have a lot of flexibility in how to setup the database. You can organize the data to be stored in a distributed fashion (across multiple devices, multiple servers. So I was under impression (feel free to correct me) that kdb horizontal scaling was something akin to Oracle RAC. I.e. horizontal in the name only. I.e. the data is only ever available from one physical instance at a time.
    - gd1 10y ago
      Having worked with both Oracle and KDB, the thing to understand is that KDB (or K or Q) is a fully fledged language... you can do whatever you like really. If you want something distributed you just design something yourself and build it. It's not like Oracle where it is all opaque and buried in query plans and the like.
yummyfajitas 10y ago
As far as I know, this and KDB have very different use cases. KDB is a single box timeseries database - the closest open source analogue would probably be using Pandas + a big folder of CSV files. (KDB performs a lot better than this, however.) The use case is storing tick data + economic data for all the symbols. I.e., one timeseries per publicly traded company. The primary use case is loading a significant chunk of that data and running some statistical analysis on it. Also KDB pricing starts at $100k or something like that. In contrast, Riak's timeseries product is distributed. It could store millions of timeseries if you throw enough boxes at it. You probably won't be loading all the data for analysis, probably you'll be processing some of the data one series at a time, and the rest is just for reference. The main use case here is sensor networks (aka "internet of things") more than financial data. I.e., one timeseries per rotor on a drone, or per sensor on your phone, sometihng like that.
- gricardo99 10y ago
  > KDB is a single box timeseries It’s not limited to a single box. “single box” is the easiest setup, but by no means the common use case. Kdb+ can be, and is, used across multiple servers and multiple storage devices. > the closest open source analogue would probably be using Pandas + a big folder of CSV files I’m not familiar with Pandas, but I know what “a big folder of CSV files” looks like and that’s pretty far from kdb+. Historical/disk data is typically organized as “splayed/parted” tables, which means it’s column oriented, sorted, grouped/hashed by key columns, can be distributed across multiple devices[1], and has built-in compression. So yes, operating over binary data, with a well-organized physical layout, and some optimized data structures is going to be much faster, not to mention the implied storage/server cost savings. > Also KDB pricing starts at $100k or something like that. I wish Kx were more open about this, but this is not very accurate. You pay per CPU core, or for query throughput. The size of your data, or number of users doesn’t matter, so you can start-out for 1/10th this cost. Of course your need for query throughput will correlate with things like number of users/clients and size of your data. But the licensing model is pretty simple, and seems to fit well with how user access scales. I know Kx makes the argument that the licensing costs are more than offset by the reduced server footprint. Personally, I’d love to see an actual case study that shows costs for a open-source database solution (with some performance metrics/benchmarks), while a kdb+ performance-equivalent solution costs less (and projected to scale for less). I know that’s a complicated comparison, where you’d really need to account for things like developer/admin time/salaries, not just equipment/bandwidth/storage/licensing costs. But it would be great to at least see a cost comparison on the latter dimensions. [1] - http://code.kx.com/wiki/JB:KdbplusForMortals/kdbplus_database#1.6.2_The_Physical_Layout_of_a_kdb.2B_Database http://code.kx.com/wiki/JB:KdbplusForMortals/kdbplus_databas...
athenot 10y ago
Minor quibble: the article is about RiakTS, their time-series enhanced version of riak_core. riak_core's main strength is that it does key-value in a distributed/resilient manner, spreading values in multiple copies (at least 3) all over a cluster of servers. Kill one server, no problem. Need more capacity, add servers and it will rebalance itself. The TS part is just an optimization built on top of that, to make values which are near each other in time to be near each other in storage, for faster range-base retrieval.
- siculars 10y ago
  Yes, exactly. (Author)
  - eggy 10y ago
    TS enhanced version of riak_core, got it. Most TS, or tick DBs, are columnar-based, memory-mapped, fast and light systems. Are there any benchmarks similar to STAC-M3, which is a year's worth of NYSE data run on different hardware to gauge kdb+'s effectiveness on different hardware configurations [1]? It's a great way to gauge performance and TCO. Does it do both memory (streaming data) and disk-based (historical) storage for big data set analytics in realtime? I'd be interested to see numbers there. A lot of people think kdb+ is only for finance. There is a conference coming up in May that will have talks on q (the language for the kdb+ database) about natural language processing and machine learning in q to name a few. Another is about using it at a power plant to most efficiently route power based upon realtime data [2]. I only got into kdb+ and q with the free, non-commercial 32-bit version. I usually use J and sometimes APL, which had MapReduce since at least the 80s for APL.Check out this post from 2009 [3]. I guess the 'new shiny' bit threw me in your chosen title. [1] https://stacresearch.com/news/2014/02/13/stac-reports-intel-ivy-bridge-ex-stac-m3-and-kdb https://stacresearch.com/news/2014/02/13/stac-reports-intel-... [2] https://kxcon2016.com/agenda/ https://kxcon2016.com/agenda/ [3] http://blog.data-miners.com/2009/04/mapreduce-hadoop-everything-old-is-new.html http://blog.data-miners.com/2009/04/mapreduce-hadoop-everyth...
    - srpeck 10y ago
metaobject 10y ago
Time series analysis is also an important component of weather and climate research.

by eggy 10y ago

MasterScrat 10y ago

> How is Riak different, or more suited to use than Kdb + q, J with JDB (free), Jd (a commercial J database like Kdb/q)[2], or the new Kerf lang/db being developed by Kevin Lawler[3]? I would really be interested to see an informed answer to this question!

anthonybsd 10y ago

KDB is not distributed and K (APL) is not particularly pleasant to work with. While it has a proven track record in fintech no one I know of is particularly fond of working with this technology. Not to mention the cost of K developers (200K+). Riak is simply offering a free alternative to these systems that is very palatable.

gricardo99 10y ago
I’m not sure what you mean by "not distributed". With kdb+ you have a lot of flexibility in how to setup the database. You can organize the data to be stored in a distributed fashion (across multiple devices, multiple servers), you can setup query load balancers to distribute work-loads, and you can replicate to multiple servers/devices. You don’t have to use K, you code in q, which most people find far easier to read/write. There’s a wealth of information to help with all this[1], and a very responsive user group. But yes, you do need kdb+ expertise to get full use out of the tool. And yes, feelings seem to run strong towards kdb+, in both directions, love/hate it. And correct again, it’s not free and the licensing cost is definitely a hurdle to wider adoption. [1] - http://code.kx.com http://code.kx.com
- anthonybsd 10y ago
  >I’m not sure what you mean by "not distributed". With kdb+ you have a lot of flexibility in how to setup the database. You can organize the data to be stored in a distributed fashion (across multiple devices, multiple servers. So I was under impression (feel free to correct me) that kdb horizontal scaling was something akin to Oracle RAC. I.e. horizontal in the name only. I.e. the data is only ever available from one physical instance at a time.
  - gd1 10y ago
    Having worked with both Oracle and KDB, the thing to understand is that KDB (or K or Q) is a fully fledged language... you can do whatever you like really. If you want something distributed you just design something yourself and build it. It's not like Oracle where it is all opaque and buried in query plans and the like.

yummyfajitas 10y ago

As far as I know, this and KDB have very different use cases. KDB is a single box timeseries database - the closest open source analogue would probably be using Pandas + a big folder of CSV files. (KDB performs a lot better than this, however.) The use case is storing tick data + economic data for all the symbols. I.e., one timeseries per publicly traded company. The primary use case is loading a significant chunk of that data and running some statistical analysis on it. Also KDB pricing starts at $100k or something like that. In contrast, Riak's timeseries product is distributed. It could store millions of timeseries if you throw enough boxes at it. You probably won't be loading all the data for analysis, probably you'll be processing some of the data one series at a time, and the rest is just for reference. The main use case here is sensor networks (aka "internet of things") more than financial data. I.e., one timeseries per rotor on a drone, or per sensor on your phone, sometihng like that.

gricardo99 10y ago
> KDB is a single box timeseries It’s not limited to a single box. “single box” is the easiest setup, but by no means the common use case. Kdb+ can be, and is, used across multiple servers and multiple storage devices. > the closest open source analogue would probably be using Pandas + a big folder of CSV files I’m not familiar with Pandas, but I know what “a big folder of CSV files” looks like and that’s pretty far from kdb+. Historical/disk data is typically organized as “splayed/parted” tables, which means it’s column oriented, sorted, grouped/hashed by key columns, can be distributed across multiple devices[1], and has built-in compression. So yes, operating over binary data, with a well-organized physical layout, and some optimized data structures is going to be much faster, not to mention the implied storage/server cost savings. > Also KDB pricing starts at $100k or something like that. I wish Kx were more open about this, but this is not very accurate. You pay per CPU core, or for query throughput. The size of your data, or number of users doesn’t matter, so you can start-out for 1/10th this cost. Of course your need for query throughput will correlate with things like number of users/clients and size of your data. But the licensing model is pretty simple, and seems to fit well with how user access scales. I know Kx makes the argument that the licensing costs are more than offset by the reduced server footprint. Personally, I’d love to see an actual case study that shows costs for a open-source database solution (with some performance metrics/benchmarks), while a kdb+ performance-equivalent solution costs less (and projected to scale for less). I know that’s a complicated comparison, where you’d really need to account for things like developer/admin time/salaries, not just equipment/bandwidth/storage/licensing costs. But it would be great to at least see a cost comparison on the latter dimensions. [1] - http://code.kx.com/wiki/JB:KdbplusForMortals/kdbplus_database#1.6.2_The_Physical_Layout_of_a_kdb.2B_Database http://code.kx.com/wiki/JB:KdbplusForMortals/kdbplus_databas...

athenot 10y ago

Minor quibble: the article is about RiakTS, their time-series enhanced version of riak_core. riak_core's main strength is that it does key-value in a distributed/resilient manner, spreading values in multiple copies (at least 3) all over a cluster of servers. Kill one server, no problem. Need more capacity, add servers and it will rebalance itself. The TS part is just an optimization built on top of that, to make values which are near each other in time to be near each other in storage, for faster range-base retrieval.

siculars 10y ago
Yes, exactly. (Author)
- eggy 10y ago
  TS enhanced version of riak_core, got it. Most TS, or tick DBs, are columnar-based, memory-mapped, fast and light systems. Are there any benchmarks similar to STAC-M3, which is a year's worth of NYSE data run on different hardware to gauge kdb+'s effectiveness on different hardware configurations [1]? It's a great way to gauge performance and TCO. Does it do both memory (streaming data) and disk-based (historical) storage for big data set analytics in realtime? I'd be interested to see numbers there. A lot of people think kdb+ is only for finance. There is a conference coming up in May that will have talks on q (the language for the kdb+ database) about natural language processing and machine learning in q to name a few. Another is about using it at a power plant to most efficiently route power based upon realtime data [2]. I only got into kdb+ and q with the free, non-commercial 32-bit version. I usually use J and sometimes APL, which had MapReduce since at least the 80s for APL.Check out this post from 2009 [3]. I guess the 'new shiny' bit threw me in your chosen title. [1] https://stacresearch.com/news/2014/02/13/stac-reports-intel-ivy-bridge-ex-stac-m3-and-kdb https://stacresearch.com/news/2014/02/13/stac-reports-intel-... [2] https://kxcon2016.com/agenda/ https://kxcon2016.com/agenda/ [3] http://blog.data-miners.com/2009/04/mapreduce-hadoop-everything-old-is-new.html http://blog.data-miners.com/2009/04/mapreduce-hadoop-everyth...
  - srpeck 10y ago

metaobject 10y ago

Time series analysis is also an important component of weather and climate research.