4 ms·

I'm having a blast playing with the free 32bit version of Q/K/Kdb+ these days in the context of a medium sized website which requires a good deal of data analys

by tlack 11y ago

I'm having a blast playing with the free 32bit version of Q/K/Kdb+ these days in the context of a medium sized website which requires a good deal of data analysis. As you probably know, Q/K are in the lineage of APL languages, with strong inherent vector-oriented capabilities, but Q uses an SQL-like dialect of words to replace APL's symbols. I've always found the idea of APL exciting, but it seemed to be a very isolated platform. "How do I use this on my site?" Q speaks SQL natively (using s.k) so it's easy to work into my MySQL-based flow, though not exactly the same as MySQL's syntax. It's got great built in facilities for bulk data loading and storing, which saves you a lot of time with the boring "getting my data set up" step when banging out little helper scripts. I've begun using Q as a cache in place of memcache/redis because I much prefer being able to ask flexible questions with a dynamic query language. Example: A MySQL query on a 1m row table that was often showing up in my slow query log at 1sec+ took less than 50ms in Q. Obviously Q and MySQL are apples to oranges, but it is still a handy adjunct when you need serious speed without full loss of flexibility. And writing Q directly is a really interesting exercise. My PHP and Node code has gotten better as a result.

wrp 11y ago
I have wondered about K/Q, because I gather that K is actually a simplification relative to APL, to make it more focused and easier to use. I thought it might be missing useful facilities for data munging. Like, I don't think it has regexes, does it?
- tlack 11y ago
  It has a limited version of regular expressions available in the "like" and "ss" functions: http://code.kx.com/wiki/Reference/like http://code.kx.com/wiki/Reference/like There's also a port of re2 available: http://code.kx.com/wiki/Cookbook/regex http://code.kx.com/wiki/Cookbook/regex The C API seems pretty simple once you get over the insane function (and macro!) names, so I think a pcre port could be done. In practice though I think Q gods prefer a split-based vector oriented approach when it comes to string chomping. Q's built in verbs and adverbs function much like regex operators do, but at an abstraction level tied fundamentally to your data.
fa 11y ago
Your experience with Q is really a satisfying read! Should I abandon my tepid attempts to get into J and throw myself on Q, despite its hefty pricetag for 64-bit version? Would you be able to do as much with Kona instead of Q?
- tlack 11y ago
  Glad to hear my useless observations benefited someone! Getting into J: I like J a lot. I like how professional the software is, their excellent docs, and their amazing community on IRC. But for some darn reason, the syntax just doesn't make sense to me. I know that sounds silly when we're talking about Q or K which are pretty unreadable, but I kinda-sorta grok the latter a lot more easily (the difficult part is adverbs, etc). I get why they chose the ascii chars they chose; with J, I feel totally lost. "NB."?! Cmonnnn.. :) Kona is based on an earlier version of K, the language that underlies Q, and though I am sure you could learn a ton from it, I need something that's been vetted a bit more because I intend to put it into production (crazy as that sounds) and Kona is still developing much of their HTTP and database support. I also wanted to learn the latest iteration, Q, which Kona hasn't implemented a dialect of just yet. Still a cool project that I follow actively on Github. I'm struggling with the 32bit vs 64bit thing now. I know I couldn't ever justify the 64bit version with my pathetic budget. The great thing about the 32bit version is that you can easily spawn multiple copies of the interpreter, and IPC is incredibly fast (1mil sync messages to localhost in 52sec on my SurfacePro3, 607msec async). So theoretically you could easily build "microservices" that each live in 4gb and pass messages back in forth[1]. At least in theory. I'll be documenting my experiments soon if all goes as planned. Already seeing big benefits. [1] That sounds hacky as hell coming from most platforms, but IPC in K/Q is baked in as a fundamental, powerful primitive, a bit like how Erlang's messages underlie the entire system. For instance, their Tick system (http://code.kx.com/wiki/Startingkdbplus/tick http://code.kx.com/wiki/Startingkdbplus/tick) used for insane volume is an ingenious way to feed out tons of data to tons of clients (firehose) and let them pick their own views of it.
  - avmich 11y ago
    Had a kind of opposite experience with K. J seems quite reasonable - intuitively - after reading very good "J for C programmers" (http://jsoftware.com/help/jforc/contents.htm http://jsoftware.com/help/jforc/contents.htm) . And symbols in J chosen for most operations also seem good - at least after some small use. "NB." is immediately "nota bene" (I guess knowing some Latin doesn't hurt - it's "note well", but I always understand it as "note that") - don't some other languages use this acronym for comments as well? What is IMO particularly good in K is the small size of the interpreter. J is "for the whole math", while K is "for practical 99% of cases" - at least it feels like that. Having small interpreter which fits in CPU cache allows potentially to run faster than with assembler, because K code is packed densely - so less time to wait for slow memory.
kanche 11y ago
I have just started working in kdb+/q and, it's my first hands on with any APL, it was a bit overwhelming at first to see an alien language like script written q but now, I am quite comfortable to understand what the script is doing in general though frequently I get stuck deep. The q gods' inclination to shorten and obfuscate everything(even the C api) makes it harder to grasp things at first, but later you also find guilty pleasure in writing one liner codes that can't be done without writing multiple loc in sql. It's a very interesting language.

by tlack 11y ago

wrp 11y ago

I have wondered about K/Q, because I gather that K is actually a simplification relative to APL, to make it more focused and easier to use. I thought it might be missing useful facilities for data munging. Like, I don't think it has regexes, does it?

tlack 11y ago
It has a limited version of regular expressions available in the "like" and "ss" functions: http://code.kx.com/wiki/Reference/like http://code.kx.com/wiki/Reference/like There's also a port of re2 available: http://code.kx.com/wiki/Cookbook/regex http://code.kx.com/wiki/Cookbook/regex The C API seems pretty simple once you get over the insane function (and macro!) names, so I think a pcre port could be done. In practice though I think Q gods prefer a split-based vector oriented approach when it comes to string chomping. Q's built in verbs and adverbs function much like regex operators do, but at an abstraction level tied fundamentally to your data.

fa 11y ago

Your experience with Q is really a satisfying read! Should I abandon my tepid attempts to get into J and throw myself on Q, despite its hefty pricetag for 64-bit version? Would you be able to do as much with Kona instead of Q?

tlack 11y ago
Glad to hear my useless observations benefited someone! Getting into J: I like J a lot. I like how professional the software is, their excellent docs, and their amazing community on IRC. But for some darn reason, the syntax just doesn't make sense to me. I know that sounds silly when we're talking about Q or K which are pretty unreadable, but I kinda-sorta grok the latter a lot more easily (the difficult part is adverbs, etc). I get why they chose the ascii chars they chose; with J, I feel totally lost. "NB."?! Cmonnnn.. :) Kona is based on an earlier version of K, the language that underlies Q, and though I am sure you could learn a ton from it, I need something that's been vetted a bit more because I intend to put it into production (crazy as that sounds) and Kona is still developing much of their HTTP and database support. I also wanted to learn the latest iteration, Q, which Kona hasn't implemented a dialect of just yet. Still a cool project that I follow actively on Github. I'm struggling with the 32bit vs 64bit thing now. I know I couldn't ever justify the 64bit version with my pathetic budget. The great thing about the 32bit version is that you can easily spawn multiple copies of the interpreter, and IPC is incredibly fast (1mil sync messages to localhost in 52sec on my SurfacePro3, 607msec async). So theoretically you could easily build "microservices" that each live in 4gb and pass messages back in forth[1]. At least in theory. I'll be documenting my experiments soon if all goes as planned. Already seeing big benefits. [1] That sounds hacky as hell coming from most platforms, but IPC in K/Q is baked in as a fundamental, powerful primitive, a bit like how Erlang's messages underlie the entire system. For instance, their Tick system (http://code.kx.com/wiki/Startingkdbplus/tick http://code.kx.com/wiki/Startingkdbplus/tick) used for insane volume is an ingenious way to feed out tons of data to tons of clients (firehose) and let them pick their own views of it.
- avmich 11y ago
  Had a kind of opposite experience with K. J seems quite reasonable - intuitively - after reading very good "J for C programmers" (http://jsoftware.com/help/jforc/contents.htm http://jsoftware.com/help/jforc/contents.htm) . And symbols in J chosen for most operations also seem good - at least after some small use. "NB." is immediately "nota bene" (I guess knowing some Latin doesn't hurt - it's "note well", but I always understand it as "note that") - don't some other languages use this acronym for comments as well? What is IMO particularly good in K is the small size of the interpreter. J is "for the whole math", while K is "for practical 99% of cases" - at least it feels like that. Having small interpreter which fits in CPU cache allows potentially to run faster than with assembler, because K code is packed densely - so less time to wait for slow memory.

kanche 11y ago

I have just started working in kdb+/q and, it's my first hands on with any APL, it was a bit overwhelming at first to see an alien language like script written q but now, I am quite comfortable to understand what the script is doing in general though frequently I get stuck deep. The q gods' inclination to shorten and obfuscate everything(even the C api) makes it harder to grasp things at first, but later you also find guilty pleasure in writing one liner codes that can't be done without writing multiple loc in sql. It's a very interesting language.