3 ms·

Sure, but there are 12 people in the world who know how to use k and q efficiently, and there is no fail recovery or any way to deal with data that actually doe

Sure, but there are 12 people in the world who know how to use k and q efficiently, and there is no fail recovery or any way to deal with data that actually does not fit in ram

snicker7 6y ago
That is not true. RAM is cheap (terabytes per node is not uncommon). Moreover, a common pattern for kdb+ databases is to have two parts: a real-time database (rdb) that is typically memory backed and a historical database (hdb) that is serialized to disk (mmap'd for near-RAM speed). In practice, most data fetching is sequential (cache friendly), so you get good performance regardless. kdb+ is commonly used in the financial industry, where data volumes are large. The language itself is extremely minimal, so the learning curve is actually much lower than most languages (it forces you to write in a "vectorized" style, which is not too different than how numpy/R/matlab works). I was able to get up to speed in a couple of days. Source: Used to work with kdb+ (still do sometimes). Not a shill (I don't even like k/j/q/APL).
- AzzieElbab 6y ago
  I am kdb user, I am not arguing against it. More power to you if you can afford to endlessly scale kdb vertically. Rest of the world have to deal with distributed systems, because people do not want to pay for kdb or the machines to run it on
smabie 6y ago
kdb+/q mmaps everything, so the difference between in ram and on disk is immaterial. You're right that not that many people know k or q, but they're pretty easy to learn and a lot as fun as well! Also you can set up clusters using a very nice IPC system. You can seamlessly just send over code and data with very little fuss.
maest 6y ago
Maybe the solution then is to train more people to use kdb, not to build new languages/frameworks.
hpcjoe 6y ago
I gave a talk at some kx/kdb usergroups[2], and definitely saw more than 12 people there. There were hundreds. Its not a trivial language to learn, you have to be ok with very terse languages in general, and ok using the language as a way to mentally model the algorithm. Somewhat "mathematically". I'm not using/working with it any more, haven't in a few years. Most of my recent work is in Julia, by choice. Two very different philosophies at work here. Julia provides a sane, more or less classical language to work with, inclusive of numerous macros and modules to help drive productivity. k/q provides a fairly modern apl like experience, leveraging Iverson's idea of a programming language being a "tool for thought". FWIW, the forces behind k/q (Arthur Whitney, et al) are now working on Shakti[1]. Looks like an interesting project, more of a platform than kdb was. Bringing this back to JuliaDB, I had been hoping for a very julia centric/idiomatic persistence engine as well as an analytical engine. I'm no fan of SQL or its variations. In reading this, I've learned DataFrames is supposed to be this. However, my impression working with DataFrames has been that it is woefully slow on small data sets (100-ish rows). It cannot handle one of the slightly larger 50k line tabular data sets that I am importing via excel sheets. For that work, I may need to resort to Perl (my fall back programming/data munging language). This is important, as Perl has a capability of tieing (binding) a variable to an underlying data structure/file/... . This enables trivial persistence, in a completely idiomatic manner (update/insert into the tied variable by setting a value). It would be interesting if this capability existed within julia as well. [1] https://shakti.com/ https://shakti.com/ [2] https://kx.com/media/2018/03/joe_preview.jpeg https://kx.com/media/2018/03/joe_preview.jpeg (actual image of me talking)
- smabie 6y ago
  My impression hasn't been that DataFrame are slow at all. What do you mean it can't handle 50k rows? Like it takes a long time to load or what? Because right now I'm using a DataFrame that has 7m rows and 50 columns and it works very well.
  - hpcjoe 6y ago
    Hmm ... could be some of the other tooling around it. Right now I can't get XLSX to talk reliably to DataFrame. I rigged up some tests for 10, 100, 1000, ... points and performance above about 500 points fell off a cliff. I should look into this more. Thanks for letting me know it can handle things that size.
scottlocklin 6y ago
I dunno man, I think Kx deals with out of memory data in the most sane way imaginable if you assume time oriented problems and understand how computers work (aka latencies in squirting the data across different layers of hardware into the CPU). I have no idea what they're giving people for manuals for setting up for different query patterns these days, but the old K3 manuals were great; classics of database engineering and practical computer science. Petabytes before it was routine. The APL family takes some getting used to, as it's a significantly different paradigm from Julia and Python looking languages: arguably as different a paradigm as Lisp. It's also fairly natural if you've done any work with Matlab, NumPy or other array oriented packages, but you do have to put in the work. It doesn't get as much attention here, but Jd is pretty good also. It's also much lower barrier to entry than Kx if you're building a personal project with it (aka its free for personal use). J is a more complex language than K, but I like it better; more oriented towards mathematics. https://code.jsoftware.com/wiki/Jd/Overview https://code.jsoftware.com/wiki/Jd/Overview

Sure, but there are 12 people in the world who know how to use k and q efficiently, and there is no fail recovery or any way to deal with data that actually does not fit in ram

snicker7 6y ago

That is not true. RAM is cheap (terabytes per node is not uncommon). Moreover, a common pattern for kdb+ databases is to have two parts: a real-time database (rdb) that is typically memory backed and a historical database (hdb) that is serialized to disk (mmap'd for near-RAM speed). In practice, most data fetching is sequential (cache friendly), so you get good performance regardless. kdb+ is commonly used in the financial industry, where data volumes are large. The language itself is extremely minimal, so the learning curve is actually much lower than most languages (it forces you to write in a "vectorized" style, which is not too different than how numpy/R/matlab works). I was able to get up to speed in a couple of days. Source: Used to work with kdb+ (still do sometimes). Not a shill (I don't even like k/j/q/APL).

AzzieElbab 6y ago
I am kdb user, I am not arguing against it. More power to you if you can afford to endlessly scale kdb vertically. Rest of the world have to deal with distributed systems, because people do not want to pay for kdb or the machines to run it on

smabie 6y ago

kdb+/q mmaps everything, so the difference between in ram and on disk is immaterial. You're right that not that many people know k or q, but they're pretty easy to learn and a lot as fun as well! Also you can set up clusters using a very nice IPC system. You can seamlessly just send over code and data with very little fuss.

maest 6y ago

Maybe the solution then is to train more people to use kdb, not to build new languages/frameworks.

hpcjoe 6y ago

I gave a talk at some kx/kdb usergroups[2], and definitely saw more than 12 people there. There were hundreds. Its not a trivial language to learn, you have to be ok with very terse languages in general, and ok using the language as a way to mentally model the algorithm. Somewhat "mathematically". I'm not using/working with it any more, haven't in a few years. Most of my recent work is in Julia, by choice. Two very different philosophies at work here. Julia provides a sane, more or less classical language to work with, inclusive of numerous macros and modules to help drive productivity. k/q provides a fairly modern apl like experience, leveraging Iverson's idea of a programming language being a "tool for thought". FWIW, the forces behind k/q (Arthur Whitney, et al) are now working on Shakti[1]. Looks like an interesting project, more of a platform than kdb was. Bringing this back to JuliaDB, I had been hoping for a very julia centric/idiomatic persistence engine as well as an analytical engine. I'm no fan of SQL or its variations. In reading this, I've learned DataFrames is supposed to be this. However, my impression working with DataFrames has been that it is woefully slow on small data sets (100-ish rows). It cannot handle one of the slightly larger 50k line tabular data sets that I am importing via excel sheets. For that work, I may need to resort to Perl (my fall back programming/data munging language). This is important, as Perl has a capability of tieing (binding) a variable to an underlying data structure/file/... . This enables trivial persistence, in a completely idiomatic manner (update/insert into the tied variable by setting a value). It would be interesting if this capability existed within julia as well. [1] https://shakti.com/ https://shakti.com/ [2] https://kx.com/media/2018/03/joe_preview.jpeg https://kx.com/media/2018/03/joe_preview.jpeg (actual image of me talking)

smabie 6y ago
My impression hasn't been that DataFrame are slow at all. What do you mean it can't handle 50k rows? Like it takes a long time to load or what? Because right now I'm using a DataFrame that has 7m rows and 50 columns and it works very well.
- hpcjoe 6y ago
  Hmm ... could be some of the other tooling around it. Right now I can't get XLSX to talk reliably to DataFrame. I rigged up some tests for 10, 100, 1000, ... points and performance above about 500 points fell off a cliff. I should look into this more. Thanks for letting me know it can handle things that size.

scottlocklin 6y ago

I dunno man, I think Kx deals with out of memory data in the most sane way imaginable if you assume time oriented problems and understand how computers work (aka latencies in squirting the data across different layers of hardware into the CPU). I have no idea what they're giving people for manuals for setting up for different query patterns these days, but the old K3 manuals were great; classics of database engineering and practical computer science. Petabytes before it was routine. The APL family takes some getting used to, as it's a significantly different paradigm from Julia and Python looking languages: arguably as different a paradigm as Lisp. It's also fairly natural if you've done any work with Matlab, NumPy or other array oriented packages, but you do have to put in the work. It doesn't get as much attention here, but Jd is pretty good also. It's also much lower barrier to entry than Kx if you're building a personal project with it (aka its free for personal use). J is a more complex language than K, but I like it better; more oriented towards mathematics. https://code.jsoftware.com/wiki/Jd/Overview https://code.jsoftware.com/wiki/Jd/Overview