3 ms·

> KDB is a single box timeseries It’s not limited to a single box. “single box” is the easiest setup, but by no means the common use case. Kdb+ can be, and i

> KDB is a single box timeseries It’s not limited to a single box. “single box” is the easiest setup, but by no means the common use case. Kdb+ can be, and is, used across multiple servers and multiple storage devices. > the closest open source analogue would probably be using Pandas + a big folder of CSV files I’m not familiar with Pandas, but I know what “a big folder of CSV files” looks like and that’s pretty far from kdb+. Historical/disk data is typically organized as “splayed/parted” tables, which means it’s column oriented, sorted, grouped/hashed by key columns, can be distributed across multiple devices[1], and has built-in compression. So yes, operating over binary data, with a well-organized physical layout, and some optimized data structures is going to be much faster, not to mention the implied storage/server cost savings. > Also KDB pricing starts at $100k or something like that. I wish Kx were more open about this, but this is not very accurate. You pay per CPU core, or for query throughput. The size of your data, or number of users doesn’t matter, so you can start-out for 1/10th this cost. Of course your need for query throughput will correlate with things like number of users/clients and size of your data. But the licensing model is pretty simple, and seems to fit well with how user access scales. I know Kx makes the argument that the licensing costs are more than offset by the reduced server footprint. Personally, I’d love to see an actual case study that shows costs for a open-source database solution (with some performance metrics/benchmarks), while a kdb+ performance-equivalent solution costs less (and projected to scale for less). I know that’s a complicated comparison, where you’d really need to account for things like developer/admin time/salaries, not just equipment/bandwidth/storage/licensing costs. But it would be great to at least see a cost comparison on the latter dimensions. [1] - http://code.kx.com/wiki/JB:KdbplusForMortals/kdbplus_database#1.6.2_The_Physical_Layout_of_a_kdb.2B_Database http://code.kx.com/wiki/JB:KdbplusForMortals/kdbplus_databas...

yummyfajitas 10y ago
So the file format is a lot better than CSV files, but in principle it's basically just a bunch of files. Maybe a better analogy would have been a big folder of feather/hdf5/etc files. (Incidentally, I'm a big fan of the folder/s3 bucket/etc full of CSV/binary files and use it whenever possible.) I agree - it's absolutely better to use than that, but it's a lot closer to that model than to the Riak model of querying a distributed system to send you the data. I stand corrected on single box and pricing - it's been a while since I've used it.
- eggy 10y ago
  kdb+/q/k are used for IOT applications [1], not just fin tec. After all, it is all time series data. The benchmarks given in a response above by srpeck [2], shows spark/shark to be 230 times slower than a k4 query, and using 50GB or RAM vs. 0.2GB RAM for k4. If RiakTS is relying on spark/shark as the in-memory database engine, it is already at a big disadvantage compared to k in terms of speed, and all the RAM that is going to be required on those distributed servers. I will have to look at the DDL/math functions available in RiakTS too, since that is how you get your work done regardless of speed of access. [1] http://www.kdnuggets.com/2016/04/kxcon2016-kdb-conference-may-montauk-ny.html http://www.kdnuggets.com/2016/04/kxcon2016-kdb-conference-ma... [2] http://kparc.com/q4/readme.txt http://kparc.com/q4/readme.txt
  - yummyfajitas 10y ago
    Very cool, I stand corrected. I hope one day I have another opportunity to play with KDB. As for the speed advantage, you'll have a similar speed advantage with python/pandas/big folder of CSV files. For all of Spark's claims on "speed", it's really just reducing the speed penalty of Hadoop from 500x to 50x. (Here 500x and 50x refer to the performance of loading flat files from a disk.)
    - gricardo99 10y ago
      Do you really mean flat CSV text files? I get the simplicity of that, but it seems really expensive (speed and size). But I'm used to tables with more than a dozen columns, and with kdb+ you only be pull in the columns of interest, and the rows of interest (due to on-disk sorting and grouping), which is a smaller subset, often much smaller.
      - yummyfajitas 10y ago
        By number, my data sets are usually in CSV. I could probably get some additional advantage via HDF5, but a gzipped CSV is usually good enough and simpler. By volume (i.e. on my 2 or 3 biggest data sets) I'll probably be mostly HDF5. I haven't tried feather yet but it looks pretty nice. KDB would probably be better, but don't underestimate what you can do with just a bunch of files.

yummyfajitas 10y ago

So the file format is a lot better than CSV files, but in principle it's basically just a bunch of files. Maybe a better analogy would have been a big folder of feather/hdf5/etc files. (Incidentally, I'm a big fan of the folder/s3 bucket/etc full of CSV/binary files and use it whenever possible.) I agree - it's absolutely better to use than that, but it's a lot closer to that model than to the Riak model of querying a distributed system to send you the data. I stand corrected on single box and pricing - it's been a while since I've used it.

eggy 10y ago
kdb+/q/k are used for IOT applications [1], not just fin tec. After all, it is all time series data. The benchmarks given in a response above by srpeck [2], shows spark/shark to be 230 times slower than a k4 query, and using 50GB or RAM vs. 0.2GB RAM for k4. If RiakTS is relying on spark/shark as the in-memory database engine, it is already at a big disadvantage compared to k in terms of speed, and all the RAM that is going to be required on those distributed servers. I will have to look at the DDL/math functions available in RiakTS too, since that is how you get your work done regardless of speed of access. [1] http://www.kdnuggets.com/2016/04/kxcon2016-kdb-conference-may-montauk-ny.html http://www.kdnuggets.com/2016/04/kxcon2016-kdb-conference-ma... [2] http://kparc.com/q4/readme.txt http://kparc.com/q4/readme.txt
- yummyfajitas 10y ago
  Very cool, I stand corrected. I hope one day I have another opportunity to play with KDB. As for the speed advantage, you'll have a similar speed advantage with python/pandas/big folder of CSV files. For all of Spark's claims on "speed", it's really just reducing the speed penalty of Hadoop from 500x to 50x. (Here 500x and 50x refer to the performance of loading flat files from a disk.)
  - gricardo99 10y ago
    Do you really mean flat CSV text files? I get the simplicity of that, but it seems really expensive (speed and size). But I'm used to tables with more than a dozen columns, and with kdb+ you only be pull in the columns of interest, and the rows of interest (due to on-disk sorting and grouping), which is a smaller subset, often much smaller.
    - yummyfajitas 10y ago
      By number, my data sets are usually in CSV. I could probably get some additional advantage via HDF5, but a gzipped CSV is usually good enough and simpler. By volume (i.e. on my 2 or 3 biggest data sets) I'll probably be mostly HDF5. I haven't tried feather yet but it looks pretty nice. KDB would probably be better, but don't underestimate what you can do with just a bunch of files.