7 ms·

Notes from my KX/kdb experience: 1.) The in-memory DB .exe was around 500 KB. Imagine that. 2.) The Q language syntax, while consistent, is fairly arcane and

Notes from my KX/kdb experience: 1.) The in-memory DB .exe was around 500 KB. Imagine that. 2.) The Q language syntax, while consistent, is fairly arcane and throwback to decades past. 3.) The documentation and driver support is abysmal. 4.) It's supposedly extremely fast, but I can't help but wonder if this is a lot of successful PR and hype (like hedge fund bosses insisting on Oracle because it's the only db that 'scales')

_acme 9y ago
Hedge funds would be the ones to use kdb over Oracle, wouldn't they?
geocar 9y ago
> 1) The in-memory DB .exe was around 500 KB. Imagine that It still is, but the hot path is much smaller than that. > 2) The Q language syntax, while consistent, is fairly arcane and throwback to decades past. I can't comment about this. I don't mind the syntax. I prefer k syntax though. > 3) The documentation and driver support is abysmal. This is getting a lot better. The fusion API[1] goes a long way towards better "drivers", and the new documentation site[2] shows a lot of energy being put into organization. There's also Q for mortals[3] which is linearized for people who like that. [1]: http://code.kx.com/q/interfaces/fusion/ http://code.kx.com/q/interfaces/fusion/ [2]: http://code.kx.com/q/ http://code.kx.com/q/ [3]: http://code.kx.com/q4m3/preface/ http://code.kx.com/q4m3/preface/ > 4) It's supposedly extremely fast, but I can't help but wonder if this is a lot of successful PR and hype There are a lot of good benchmarks of kdb[4] unlike Oracle :) [4]: https://stacresearch.com/m3 https://stacresearch.com/m3
amypinka 9y ago
Seems plenty fast to me: http://tech.marksblogg.com/benchmarks.html http://tech.marksblogg.com/benchmarks.html
tluyben2 9y ago
2) I like K better (fitting the ideals of APL); I feel Q was done by Arthur to please some big client as it doesn't feel like he would choose that kind of thing (that's from reading interviews, seeing the iterations and his basic code philosophy)
- eggy 9y ago
  I like K better than Q too, but J[1] clicks with me more. J has JDB[2] and Jd[3] for things somewhat similar to qdb with Jd being the commercial offering similar to qdb rather than JDB. I would probably choose APL over Q if that were a choice. In J you can always make your definitions (verbs, nouns, etc...) plain words if you like the way Q reads. [1] jsoftware.com [2] http://code.jsoftware.com/wiki/JDB [3] http://www.jsoftware.com/jdhelp/overview.html
  - fusiongyro 9y ago
    Have you used Dyalog APL recently? They now have a rank operator and fork/hook from J. I am learning J but haven't made up my mind about this.
    - throwaway7645 9y ago
      Dyalog appears to be more popular with conferences and more products, but it costs $1k ish for a commercial license and nobody else can run your code without a license and server licenses aren't cheap. It also pretty much needs a special keyboard and a key mapping. J is free for pretty much everything and uses standard characters (although I really like the APL characters). I think they're both nice.
      - fusiongyro 9y ago
        Dyalog comes with a keyboard layout (on a Mac it just replaces the alt keys). It's quite easy to use. GNU APL's Emacs mode does the same thing, although mapped to super rather than alt (meta, in Emacs) by default. I'm aware of the licensing costs, I'm more curious about whether Dyalog is obtaining popularity versus J, and if so, why. Of course, three new people going to the Dyalog conference would be a 10% increase in popularity, it looks like… so maybe this far out on the long tail it doesn't matter.
        throwaway7645 9y ago
        Yea, I was just saying the key mappings can be a pain and your favorite keyboard probably doesn't have the APL symbols on it. The Dyalog IDE has a virtual keyboard, but I don't like those too much. If none of that bothers you, than no biggie. I'm guessing Dyalog has more production users and a bit more users than you see at the conference as they are typically held in the UK. J is free, so I bet a lot more people try it even though Dyalog has a free hobby license. J has a nice built in plotting library"viewmat" while Dyalog has sharpleaf. Both are nice, but sharpleaf has a GUI like doing charts in Excel. Dyalog can easily hook-in to .NET, so that is pretty helpful on Windows in the real-world. I'd agree it's a wash right now. What is your background and needs?
ktamura 9y ago
I used to use KX/kdb/Q/K daily for several years. I wrote a full implementation of reinforcement learning (15 lines), a lightweight MVC framework (to show reports and tables in an internal webapp) and even a Q syntax checker (abusing table as a data structure to hold parse trees). Good or bad, for the longest time, Q was my "go-to" programming language. Based on that experience... 1) Yes, but that's not huge by modern standard. 2) Q is a DSL version of K. As others have commented, K is a pretty clean implementation of APL, and Q makes K more approachable. 3) I have to agree here, but Q for Mortals makes up for it. 4) It is really fast. As we all know, a vast majority of us actually don't have terabytes and terabytes of data, especially after a reasonably cleanup / ETL / applying common sense. I suppose it helped that I worked in finance, which meant my desktop had 16GB of memory in 2009 and 128GB of memory on a server shared by 4-5 traders. Finally, Q was never intended for general-purpose computing nor a widespread adoption. At least when I was an active user, the mailing list had the same 20-30 people asking questions and 3-4 people answering them, including a@kx.com (= Arthur Whitney, the creator). Back then, I'd say there were at most 2-3k active users of Q/K in the world. Now that Kx Systems is part of First Derivative and has been working on expanding their customer base, perhaps they have more...?
- nkurz 9y ago
  1) Yes, but that's not huge by modern standard. OP could have phrased it better, but I presume his point was that 500KB is extremely small by modern standards. The whole executable fits comfortably in L3, so you'll probably never have a full cache miss for instructions. On the other hand, while it's cool that it's small, I'm not sure that binary size is a good proxy for performance. Instruction cache misses are rarely going to be a limiting factor.
  - beagle3 9y ago
    > Instruction cache misses are rarely going to be a limiting factor. k's performance is a combination of a lot of small things, each one independently doesn't seem to be that meaningful. And yet, the combination screams. The main interpreter core, for example, used to be <16K code and fit entirely within the I-cache; that means bytecode dispatch was essentially never re-fetched or re-decoded to micro instructions, and all the speculative execution predictors have a super high hit rate. When Python switched the interpreter loop from a switch to a threaded one, for example, they got ~20% speedup[0]; I wouldn't be surprised if the fitting entirely within the I-cache (which K did and Python didn't at the time) gives another 20% speedup. [0] https://bugs.python.org/issue4753 https://bugs.python.org/issue4753
flukus 9y ago
Any other downsides? Management has been convinced and we're apparently switching to it at work soon but there is so little information and most of that is marketing, so it's hard to get an idea of what we're walking into.
- LfLxfxxLxfxx 9y ago
  price

_acme 9y ago

Hedge funds would be the ones to use kdb over Oracle, wouldn't they?

geocar 9y ago

> 1) The in-memory DB .exe was around 500 KB. Imagine that It still is, but the hot path is much smaller than that. > 2) The Q language syntax, while consistent, is fairly arcane and throwback to decades past. I can't comment about this. I don't mind the syntax. I prefer k syntax though. > 3) The documentation and driver support is abysmal. This is getting a lot better. The fusion API[1] goes a long way towards better "drivers", and the new documentation site[2] shows a lot of energy being put into organization. There's also Q for mortals[3] which is linearized for people who like that. [1]: http://code.kx.com/q/interfaces/fusion/ http://code.kx.com/q/interfaces/fusion/ [2]: http://code.kx.com/q/ http://code.kx.com/q/ [3]: http://code.kx.com/q4m3/preface/ http://code.kx.com/q4m3/preface/ > 4) It's supposedly extremely fast, but I can't help but wonder if this is a lot of successful PR and hype There are a lot of good benchmarks of kdb[4] unlike Oracle :) [4]: https://stacresearch.com/m3 https://stacresearch.com/m3

amypinka 9y ago

Seems plenty fast to me: http://tech.marksblogg.com/benchmarks.html http://tech.marksblogg.com/benchmarks.html

tluyben2 9y ago

2) I like K better (fitting the ideals of APL); I feel Q was done by Arthur to please some big client as it doesn't feel like he would choose that kind of thing (that's from reading interviews, seeing the iterations and his basic code philosophy)

eggy 9y ago
I like K better than Q too, but J[1] clicks with me more. J has JDB[2] and Jd[3] for things somewhat similar to qdb with Jd being the commercial offering similar to qdb rather than JDB. I would probably choose APL over Q if that were a choice. In J you can always make your definitions (verbs, nouns, etc...) plain words if you like the way Q reads. [1] jsoftware.com [2] http://code.jsoftware.com/wiki/JDB [3] http://www.jsoftware.com/jdhelp/overview.html
- fusiongyro 9y ago
  Have you used Dyalog APL recently? They now have a rank operator and fork/hook from J. I am learning J but haven't made up my mind about this.
  - throwaway7645 9y ago
    Dyalog appears to be more popular with conferences and more products, but it costs $1k ish for a commercial license and nobody else can run your code without a license and server licenses aren't cheap. It also pretty much needs a special keyboard and a key mapping. J is free for pretty much everything and uses standard characters (although I really like the APL characters). I think they're both nice.
    - fusiongyro 9y ago
      Dyalog comes with a keyboard layout (on a Mac it just replaces the alt keys). It's quite easy to use. GNU APL's Emacs mode does the same thing, although mapped to super rather than alt (meta, in Emacs) by default. I'm aware of the licensing costs, I'm more curious about whether Dyalog is obtaining popularity versus J, and if so, why. Of course, three new people going to the Dyalog conference would be a 10% increase in popularity, it looks like… so maybe this far out on the long tail it doesn't matter.
      - throwaway7645 9y ago
        Yea, I was just saying the key mappings can be a pain and your favorite keyboard probably doesn't have the APL symbols on it. The Dyalog IDE has a virtual keyboard, but I don't like those too much. If none of that bothers you, than no biggie. I'm guessing Dyalog has more production users and a bit more users than you see at the conference as they are typically held in the UK. J is free, so I bet a lot more people try it even though Dyalog has a free hobby license. J has a nice built in plotting library"viewmat" while Dyalog has sharpleaf. Both are nice, but sharpleaf has a GUI like doing charts in Excel. Dyalog can easily hook-in to .NET, so that is pretty helpful on Windows in the real-world. I'd agree it's a wash right now. What is your background and needs?

ktamura 9y ago

I used to use KX/kdb/Q/K daily for several years. I wrote a full implementation of reinforcement learning (15 lines), a lightweight MVC framework (to show reports and tables in an internal webapp) and even a Q syntax checker (abusing table as a data structure to hold parse trees). Good or bad, for the longest time, Q was my "go-to" programming language. Based on that experience... 1) Yes, but that's not huge by modern standard. 2) Q is a DSL version of K. As others have commented, K is a pretty clean implementation of APL, and Q makes K more approachable. 3) I have to agree here, but Q for Mortals makes up for it. 4) It is really fast. As we all know, a vast majority of us actually don't have terabytes and terabytes of data, especially after a reasonably cleanup / ETL / applying common sense. I suppose it helped that I worked in finance, which meant my desktop had 16GB of memory in 2009 and 128GB of memory on a server shared by 4-5 traders. Finally, Q was never intended for general-purpose computing nor a widespread adoption. At least when I was an active user, the mailing list had the same 20-30 people asking questions and 3-4 people answering them, including a@kx.com (= Arthur Whitney, the creator). Back then, I'd say there were at most 2-3k active users of Q/K in the world. Now that Kx Systems is part of First Derivative and has been working on expanding their customer base, perhaps they have more...?

nkurz 9y ago
1) Yes, but that's not huge by modern standard. OP could have phrased it better, but I presume his point was that 500KB is extremely small by modern standards. The whole executable fits comfortably in L3, so you'll probably never have a full cache miss for instructions. On the other hand, while it's cool that it's small, I'm not sure that binary size is a good proxy for performance. Instruction cache misses are rarely going to be a limiting factor.
- beagle3 9y ago
  > Instruction cache misses are rarely going to be a limiting factor. k's performance is a combination of a lot of small things, each one independently doesn't seem to be that meaningful. And yet, the combination screams. The main interpreter core, for example, used to be <16K code and fit entirely within the I-cache; that means bytecode dispatch was essentially never re-fetched or re-decoded to micro instructions, and all the speculative execution predictors have a super high hit rate. When Python switched the interpreter loop from a switch to a threaded one, for example, they got ~20% speedup[0]; I wouldn't be surprised if the fitting entirely within the I-cache (which K did and Python didn't at the time) gives another 20% speedup. [0] https://bugs.python.org/issue4753 https://bugs.python.org/issue4753

flukus 9y ago

Any other downsides? Management has been convinced and we're apparently switching to it at work soon but there is so little information and most of that is marketing, so it's hard to get an idea of what we're walking into.

LfLxfxxLxfxx 9y ago
price