14 ms·

The Q Language

249 points by callinyouin 9y ago

noblethrasher 9y ago
Q was the subject of an awesome presentation by Tim Thornton at YOW! Lambda Jam 2016. [1] https://www.youtube.com/watch?v=ZGIPmC6wi7E https://www.youtube.com/watch?v=ZGIPmC6wi7E
- geocar 9y ago
  His description of primality testing in that video is really quite good[1]. If it starts too slow, maybe that's a good place to start? [1]: https://youtu.be/ZGIPmC6wi7E?t=9m56s https://youtu.be/ZGIPmC6wi7E?t=9m56s
dasmoth 9y ago
Aside: has anyone heard any recent news about the K5/K6 rewrites of the underlying interpreter? There was a fair amount of chatter about these a couple of years ago, but all gone rather quiet... (Some context: http://archive.vector.org.uk/art10501320 http://archive.vector.org.uk/art10501320)
- RodgerTheGreat 9y ago
  Currently it looks like k7 is going to be an actual commercial product in the near-ish future. Arthur is still hashing the details out, but the language design is substantially similar to k6. If anyone ideas or strong opinions about how to make k7 better, now would be a pretty good time to email Arthur. For those not already in the loop with respect to k6, the reference card (http://kparc.com/k.txt http://kparc.com/k.txt) provides a good overview. Note that it is neither exhaustive nor completely representative of the current state of the language.
amypinka 9y ago
The next time you buy some stock, thank the Q language for making it happen.
jonathanapp 9y ago
Notes from my KX/kdb experience: 1.) The in-memory DB .exe was around 500 KB. Imagine that. 2.) The Q language syntax, while consistent, is fairly arcane and throwback to decades past. 3.) The documentation and driver support is abysmal. 4.) It's supposedly extremely fast, but I can't help but wonder if this is a lot of successful PR and hype (like hedge fund bosses insisting on Oracle because it's the only db that 'scales')
- _acme 9y ago
  Hedge funds would be the ones to use kdb over Oracle, wouldn't they?
- geocar 9y ago
  > 1) The in-memory DB .exe was around 500 KB. Imagine that It still is, but the hot path is much smaller than that. > 2) The Q language syntax, while consistent, is fairly arcane and throwback to decades past. I can't comment about this. I don't mind the syntax. I prefer k syntax though. > 3) The documentation and driver support is abysmal. This is getting a lot better. The fusion API[1] goes a long way towards better "drivers", and the new documentation site[2] shows a lot of energy being put into organization. There's also Q for mortals[3] which is linearized for people who like that. [1]: http://code.kx.com/q/interfaces/fusion/ http://code.kx.com/q/interfaces/fusion/ [2]: http://code.kx.com/q/ http://code.kx.com/q/ [3]: http://code.kx.com/q4m3/preface/ http://code.kx.com/q4m3/preface/ > 4) It's supposedly extremely fast, but I can't help but wonder if this is a lot of successful PR and hype There are a lot of good benchmarks of kdb[4] unlike Oracle :) [4]: https://stacresearch.com/m3 https://stacresearch.com/m3
- amypinka 9y ago
  Seems plenty fast to me: http://tech.marksblogg.com/benchmarks.html http://tech.marksblogg.com/benchmarks.html
- tluyben2 9y ago
  2) I like K better (fitting the ideals of APL); I feel Q was done by Arthur to please some big client as it doesn't feel like he would choose that kind of thing (that's from reading interviews, seeing the iterations and his basic code philosophy)
http-teapot 9y ago
Imagine being French and having to explain you code in Q. (I am not certain if my comment follows HN's guidelines, apologies if I offended anyone)
- atrudeau 9y ago
  Or the query execution plan: Plan Q
- xamuel 9y ago
  The French got their revenge by giving us Coq
fiatjaf 9y ago
What is this?
- amypinka 9y ago
  Documentation for kdb+/q. MS had a posting for their new Q# language today on HN so that might have inspired someone to post this.
  - deleted 9y ago
    [deleted]
SifJar 9y ago
https://learnxinyminutes.com/docs/kdb+/ https://learnxinyminutes.com/docs/kdb+/ Another useful intro to the language
scottyelich 9y ago
... and can you use the 32bit version in commercial use? (Last I checked, no...) The license already changed once (for the worse, IMnsHO), so what's to say that won't happen again. Too risky to touch. No, can't use it (32bit version) for commercial, so this is a non-starter. Sorry. kOS? really? https://news.ycombinator.com/item?id=8475809 https://news.ycombinator.com/item?id=8475809
- beagle3 9y ago
  You can purchase a license if you want to be unaffected by future license changes. It might not be the right price for you, but it is for some. They are, in fact, courting only rich customers. Some languages are worth learning to expand your horizons. Lisp is one of them, even if you never use it, and the APL family (of which K/Q are members) is another. My C code has become faster, simpler, shorter and less buggy after I dabbled in K. YMMV, but using it in a commercial setting is not the only reason to look at it. > kOS? really? https://news.ycombinator.com/item?id=8475809 https://news.ycombinator.com/item?id=8475809 Really. What exactly is your "really?" question?
  - vram22 9y ago
    >My C code has become faster, simpler, shorter and less buggy after I dabbled in K. Interesting. Can you explain why you think that has happened?
    - beagle3 9y ago
      Several things: 1. K (and the entire APL family) eschew many of the layers upon layers of abstraction that modern software engineering uses, whether they are justified or not. It turns out, that they are mostly not justified. K gently pushes you toward thinking in a lower level of "what's really happening here?"; I'm not sure I can give a good example here - but the world looks different after taking the red pill. e.g., it is not uncommon in K to represent a tree as an two arrays, one of data, and one of parent pointers. Once you shake the "but I must abstract this!" feeling, you realize it works better. 2. K gently encourages doing work on batches of data. That is, it is idiomatic (and easier) to write functions that operate on arrays, and return arrays of processed data, then writing functions that operate on one element at a time. In turn, this means that the resulting program more often than not works in stages where each stage processes its entire input before going to the next stage (which uses the output from this stage as its input). In the "old" C/C++/C#/Java/Python world, it is idiomatic to push each element through a pipeline before going to the next one. 3. K encourages building a set of orthogonal, non trivial operations and combining them in various ways, rather then building layer upon layer of abstraction. It gives an example by giving an extremely useful basis of functions. E.g. http://nsl.com/k/t.k http://nsl.com/k/t.k implements a very fast, reasonably capable in memory database with aggregation, joins, projects and more. It takes all of 14 lines, all quite short. While you can't implement it in 14 lines of C, using the same principles you can probably do that in less than a hundred. Achieve the same in idiomatic C is going to be much harder and longer. 4. Finally, K gently encourages solutions which work well with modern memory and storage hierarchies. E.g., it encourages linear scan operations that touch all elements of an array over random access ones that touch only 1/10 of the elements. Instinct and idiom in other languages will guide you towards the latter, but the former is often much faster.
pyvpx 9y ago
what makes kdb so special and why isn't there an open/libre alternative?
- JadeNB 9y ago
  > what makes kdb so special and why isn't there an open/libre alternative? Maybe I'm reading too much into this, but it seems like you expect the answer "nothing much" to the first part of your question. I have done nothing more than read lots of articles on KDB's lineage and play around a bit with J, so take my answer with a considerable lump of salt, but my impression is that the answer to the second part of your question is "because there's something 'so special' about KDB"; my understanding is that it provides blazing-fast access to memory-compact databases, from a tiny codebase, building on the APL/J legacy. Why can't this be done in an open sourced way? Well, surely there's no inherent reason, but the fact that it hasn't been is probably evidence that it's not just a problem of trivially cloning existing work (or else someone would have done it).
  - throwaway7645 9y ago
    The real answer is Arthur Whitney is a god-like being with coding powers beyond the mortal realm. Seriously though they're a lot of stories of his excellent work if you search for them. He's supposedly working on kOS so you can run kdb+ on bare metal. Tiny, fast code is what he does. Sure it might look incomprehensible, but it is a few pages of code he can store in his brain at once. Aaron Hsu (someone has some links to his HN posts on here) explains this with his compiler which converts APL to GPU code. His entire compiler (which he has worked on for a long time) is a handful of pages. He says something like "there's no need for an abstraction if I can see everything at once". Most programming languages have implementations that are very long...which do you think has less bugs? Steve McConnals Code Complete has a statistic somewhere on bugs per 100 loc. That doesn't leave a lot of room in Arthur's code base (although you could say his code being so terse makes it an apples to oranges comparison I guess).
- dasmoth 9y ago
  There’s an open source impl of an earlier version of K: https://kona.github.io/#/ https://kona.github.io/#/ There’s also a JS implementation that aims to implement K6: http://johnearnest.github.io/ok/index.html http://johnearnest.github.io/ok/index.html I don’t believe either implement all the “database” side of kdb+ though. Edit: The J language is similar in some respects, and is GPLed. They've also got a columnar database... http://www.jsoftware.com/jdhelp/overview.html http://www.jsoftware.com/jdhelp/overview.html
kuwze 9y ago
On github kiyoto worked on this project[0] to help others understand Q code. [0]: https://github.com/kiyoto/ungod https://github.com/kiyoto/ungod
feelin_googley 9y ago
This is only software I have ever seen that actually got smaller over time. Not only that, it is the only software I have ever seen that used a GUI and then ditched it in a subsequent version. Few programs are so aligned with my own software sensibilities. Only complaint is that they used to have a FreeBSD port and now only have Linux and macOS but no BSD. Unfortunately Linux compat in BSD is being perceived by some as a potential security issue these days.
- oskarth 9y ago
  > This is only software I have ever seen that actually got smaller over time. I agree, this is admirable! Even if it is under different names (which I think is a far better approach), Sustrik does this too: http://250bpm.com/blog:50 http://250bpm.com/blog:50 He went from AMPQ (not his own creation) -> ZeroMQ -> nanomsg -> Libmill (essentially Go in C/UNIX style) Also, OpenBSD comes to mind (LibreSSL for example).
  - beagle3 9y ago
    Libmill/Libdill (go coroutines) are not on the AMPQ->ZeroMQ->Crossroads->Nanomsg (socket abstraction) path ; in fact, they have essentially nothing in common except sustrik.
    - oskarth 9y ago
      I wish you'd actually read the article before shooting your mouth off with a Well Actually. FTA: > Next one: nanomsg. An alternative to ZeroMQ. In comments: > How would you split nanomsg, then? > [...] 2. coroutine library, e.g. libtask, libmill [...] Not to mention that this is obvious if you understand what the core of each thing is in terms of capabilities. If you are gonna be the "technically..." guy, at least do it right. It's about decomposing things, which is the essence of good design, and - supposedly - the UNIX philosophy at its heart.
      - beagle3 9y ago
        As an actual user of ZeroMQ, nanomsg and dabbler of libdill, I assure you I know what I am talking about. Libdill/libmill is NOT on the same arch. It could be used to implement the next generation of zeromq/nanomsg, but it does not, in fact, provide any functionality that these provide today - it would be an internal implementation detail if sustrik followed through on that comment. But it would not be a nanomsg rewrite, likely something completely new - nanomsg has been rewritten, several times in several ways, by Garret D'amore who has taken over. D'amore explicitly rejected using e.g. libuv (mentioned in that comment) as a basis for a nanomsg rewrite. There's an "uncertainty" principle at work here - separating of concerns generates dependencies which in the grand scheme of things are not necessarily good. In fact, one of D'amore's rewrites used native OS threads in lieu of a co-routine library, which worked perfectly well on FreeBSD and Solaris, and abysmally on all other OSses. The main reason to use a coroutine library in a messaging library is actually that the underlying OS threads implementations scale so badly on most modern systems.
nz 9y ago
Found this illuminating interview with Arthur in the ACM queue: http://queue.acm.org/detail.cfm?id=1531242 http://queue.acm.org/detail.cfm?id=1531242
johnhenry 9y ago
Not directly related, but Microsoft just released a new language for developing quantum languages, Q#, which doesn't seem to be related at all. I hope this doesn't cause too much confusion.
aamederen 9y ago
As an outsider, my initial reaction is "does every new database system have to come up with their own SQL-like language?"
- beagle3 9y ago
  SQL's only virtue is that it is well known; It is otherwise not very good compared to other query languages, and every SQL engine extends it somewhat differently, extensions which you can't avoid cause the standard is too limited. I agree most query languages would be better off as minor extensions to SQL - but kdb+/Q is different. Unlike SQL which pays lip service to Codd's relational model but breaks it with things like TOP, ORDER BY, LIMIT and others, the Q language embraces the order between tuples to great effect, making e.g. "as-of" queries which are quite common trivial; whereas in SQL and the relational model, as-of queries are inefficient in either execution time or storage space (usually both), and reasonable execution speed schemas cannot, in fact, guarantee their integrity (which is often quoted as one of the the main advantages of the relational model). As another example, Q implements "foreign key chasing" also called "reference chasing", which is also implemented in the web2py DAL and surely others; compare[0] the equivalent tpc-h query: in q: select revenue avg supplier.nation=`BRAZIL by order.date.year from lineitem where order.customer.nation.region=`AMERICA, order.date.year in 1995 1996, part.type=`STEEL in sql: select o_year,sum(case when nation = 'BRAZIL' then revenue else 0 end)/sum(revenue) as mkt_share from(select year(o_orderdate) as o_year,revenue,n2.n_name as nation from part,supplier,lineitem,orders,customer,nation n1,nation n2,region where p_partkey = l_partkey and s_suppkey = l_suppkey and l_orderkey = o_orderkey and o_custkey = c_custkey and c_nationkey = n1.n_nationkey and n1.n_regionkey = r_regionkey and r_name = 'AMERICA' and s_nationkey = n2.n_nationkey and o_orderdate between date('1995-01-01') and date('1996-12-31') and p_type = 'STEEL') as all_nations group by o_year order by o_year; [0] bottom of http://kparc.com/d.txt http://kparc.com/d.txt
modeless 9y ago
The KDB code is terse to an unhealthy extreme. Check this gem: // remove more clutter #define O printf #define R return #define Z static #define P(x,y) {if(x)R(y);} #define U(x) P(!(x),0) #define SW switch #define CS(n,x) case n:x;break; https://github.com/KxSystems/kdb/blob/master/c/c/k.h#L96 https://github.com/KxSystems/kdb/blob/master/c/c/k.h#L96 Or this wall of code: https://github.com/KxSystems/kdb/blob/master/c/c.cs https://github.com/KxSystems/kdb/blob/master/c/c.cs
- deleted 9y ago
  [deleted]
- alexeiz 9y ago
  It's so terse, it's gotta be fast!
- kolme 9y ago
  Welp, you were not kidding: https://github.com/KxSystems/kdb/blob/master/c/c/curl.c https://github.com/KxSystems/kdb/blob/master/c/c/curl.c This code is basically obfuscated by hand. Absolutely unapproachable. Only the original author(s) can understand it. Judging by other comments, it seems to work well. So they seem to be good programmers producing working code. It's just not intelligible by other human beings, which is a pretty bad thing, but not the only factor in software quality/health. The Vim code base is at some parts straight batshit insane, but it's one of the most polished programs I've ever used. All that being said, I would find infuriating to work with such code. Nope!
  - beagle3 9y ago
    It takes time getting used to, yes. And it is not for all people. But it's just a foreign language; You could look at Japanese text[0] and make similar statements, and would be just as valid (or rather, invalid) as your statement. You expect to be able to read it because you're used to a class of languages which are all similar enough at the surface level -- perhaps you are even familiar with more than one fundamentally different classes, say, "lispish and algolish" or "germanic and latin". But that doesn't make APLish or Japanese[0] horrible. This is just APL using C syntax. [0] assuming, the proverbial you does not know Japanese
human_afterall 9y ago
For a second I thought you meant Q# - got pretty excited there :)
tree_of_item 9y ago
More APL threads for anyone interested: Smaller Code, Better Code: https://news.ycombinator.com/item?id=13565743 https://news.ycombinator.com/item?id=13565743 AMA: Explaining my 750 line compiler+runtime designed to GPU self-host APL: https://news.ycombinator.com/item?id=13797797 https://news.ycombinator.com/item?id=13797797
anonfunction 9y ago
Conway's Game of Life in Q: life:{3=a-x*4=a:2(sum -1 0 1 rotate\:,’/)/x}
- abraae 9y ago
  By comparison in APL: life←{↑1 ⍵∨.∧3 4=+/,¯1 0 1∘.⊖¯1 0 1∘.⌽⊂⍵}
  - LfLxfxxLxfxx 9y ago
    life←{≢⍸⍵}⌺3 3∊¨3+0,¨⊢ https://youtu.be/3FjYly2G_QI?t=565 https://youtu.be/3FjYly2G_QI?t=565
    - RodgerTheGreat 9y ago
      I think I've nearly convinced Arthur to include an equivalent to "stencil" in k7. Fingers crossed!
      - Avshalom 9y ago
        K's approach of arrays of arrays would preclude an efficient 2d stencil operation wouldn't it?
is0tope 9y ago
I have been an on and off q user for a few years. KDB+/Q is a great system, but in my opinion it is best used for its correct purpose;and that purpose is primarily time series and data streaming. The columnar structure of the DB as well as its IPC layer make it very good at creating chains of processes that can be used to stream row updates and branch them out to different processes with different responsibilities. Likewise it's on disk database is great for running complex (time series) queries. This speed and terseness comes at a cost of being fairly "old school" in its approach. It's only recently for instance the we got stack traces, and readability is definitely not for the faint of heart though this depends on how it is written. In my view the biggest thing that is needed right now is better tooling and libraries. Some attempts have been made to do this, and I am hearing that the new initiatives by kx will be addressing this in the coming months. The lack of standardized testing library/framework also can be problematic, as every team that I have seen does it slightly differently, and a "best practice" would beneficial.
beaumayns 9y ago
I mostly love K/Q, and recently rearranged my career to do more of it. There are some things to be aware of, though: * If your code doesn't spend most of its time in primitive verbs operating on large vectors, it's gonna be more or less as slow as any other interpreted language. Q and kdb+ can be fast and beautiful if you can arrange your problem in the right way, but it's not magic. * The internals are locked away. If you don't like the way something fundamental works, tough. I've known some folks to go to heroic lengths with debuggers and hacked up shared objects to get Q to do what they want. You could also get Kx to add the stuff you need (they're pretty reasonable and responsive). But, you can't really take it apart and put it back together again like you can with, say, Lua, Ruby, or Python. * Relating to the above point, one of the weaknesses of the language is that there are a lot of useful (even necessary) features packed into weird corners. There's little room for abstractions beyond the basics, so you get stuff like CSV parsing controlled by the structure of lists passed to a function called "0:". It's getting better documented lately, but it's still not pretty. * Various annoyances (no real module system, no lexical scoping, etc...) In many of those cases, I'm not even sure what could be done without compromising some other aspect of the language. Most of the time (at least for me), it's really a joy to use.
- Bootvis 9y ago
  Curious to know what you did to rearrange your career.
petermora 9y ago
I worked in Pascal, C/C++, Java, Python, R, Nim, Clojure, Rust, Go, Js, and created various projects in these languages. However, I couldn't get used to Q. I understand that it is fast, and I also started to like the functional programming aspect of it. But oh boy, there is no proper error messages (no, "`type" is not helpful). The short versions of various map and apply were handy, but there are no equivalent versions with longer names, therefore there is no way to write a code which is readable to a non Q-expert. The strange evaluation order made it impossible for me to read other people's code, I often started to add parentheses and checked whether the behavior changed. There is no debugger (I used KdbStudio). Once I used more than 16 local variables, which gave a runtime error with some strange unhelpful error message. I'm not questioning its usefulness, but I think that it could be much more developer friendly without compromising speed.
- easytiger 9y ago
  > therefore there is no way to write a code which is readable to a non Q-expert. If you could FD wouldn't be able to justify charging £1.5k/day per head consultancy fees for someone they just sent on a 1 week training course.
- geocar 9y ago
  I have worked in C (twenty-five years), C++, Python, Perl (twenty years), JavaScript, Common Lisp (for ten years), OCaml, forth, postscript, and also q/Kdb. I think a lot can be done to improve teaching q/Kdb. The current “best practice” is to change you until it makes sense but this takes time- anywhere from 6 months to a couple years based on your other experiences. And you still have to want to “get it”. I want that to be better. However this is alien technology: the terseness is a feature. Limited locals are a feature. The evaluation order is a feature. These things actually help it go fast (as surprising as that is!) We only just recently got “a debugger” and good error messages because of repeated complaints and wishes for it from newbies, but there is a reason experienced Kdb/q programmers never wanted it. Why doesn’t that reason prompt those newbies to figure out why? Yes, ideally you would learn the way and the why we do things, but there are a number of serious hurdles to overcome: To you q/Kdb seems merely unpolished even if maybe “fast”. Is it worth it? “Maybe”, you suggest, but then there are crazy people like me telling you something preposterous; that q/Kdb is actually incredibly well crafted, highly readable and an absolute pleasure to use. I'm also trying to say that everything is worth learning: Nim and Python and JavaScript are all basically the same thing. You learned one of them, you kindof learned them all, so adding another one feels like these things are easy to learn. Alien technology is alien though, you haven't learned any of it. How can we even talk to each other? I'm hopeful: Lambda was tricky, but it snuck in to things. Can we get tables, views, and high code density? We need to find something better- some better way to talk about it, but the peanut gallery is loud.
cousin_it 9y ago
I used to play with K long ago, it was tons of fun. Sometimes I think I should've been a quant and made crazy money :-)
anonu 9y ago
Q is great. I think wider adoption would come from two places: 1. adding more syntactic sugar to the language to improve readability 2. open-sourcing the thing or building an open-source compatible Q interpreter with all the nice features of KDB (functional programming, vector-based data structures, "scripting language within a database" approach, web features: HTTP server and websockets, etc.. etc.. )

249 points by callinyouin 9y ago

Q was the subject of an awesome presentation by Tim Thornton at YOW! Lambda Jam 2016. [1] https://www.youtube.com/watch?v=ZGIPmC6wi7E https://www.youtube.com/watch?v=ZGIPmC6wi7E

geocar 9y ago
His description of primality testing in that video is really quite good[1]. If it starts too slow, maybe that's a good place to start? [1]: https://youtu.be/ZGIPmC6wi7E?t=9m56s https://youtu.be/ZGIPmC6wi7E?t=9m56s

dasmoth 9y ago

Aside: has anyone heard any recent news about the K5/K6 rewrites of the underlying interpreter? There was a fair amount of chatter about these a couple of years ago, but all gone rather quiet... (Some context: http://archive.vector.org.uk/art10501320 http://archive.vector.org.uk/art10501320)

RodgerTheGreat 9y ago
Currently it looks like k7 is going to be an actual commercial product in the near-ish future. Arthur is still hashing the details out, but the language design is substantially similar to k6. If anyone ideas or strong opinions about how to make k7 better, now would be a pretty good time to email Arthur. For those not already in the loop with respect to k6, the reference card (http://kparc.com/k.txt http://kparc.com/k.txt) provides a good overview. Note that it is neither exhaustive nor completely representative of the current state of the language.

amypinka 9y ago

The next time you buy some stock, thank the Q language for making it happen.

jonathanapp 9y ago

Notes from my KX/kdb experience: 1.) The in-memory DB .exe was around 500 KB. Imagine that. 2.) The Q language syntax, while consistent, is fairly arcane and throwback to decades past. 3.) The documentation and driver support is abysmal. 4.) It's supposedly extremely fast, but I can't help but wonder if this is a lot of successful PR and hype (like hedge fund bosses insisting on Oracle because it's the only db that 'scales')

_acme 9y ago
Hedge funds would be the ones to use kdb over Oracle, wouldn't they?
geocar 9y ago
> 1) The in-memory DB .exe was around 500 KB. Imagine that It still is, but the hot path is much smaller than that. > 2) The Q language syntax, while consistent, is fairly arcane and throwback to decades past. I can't comment about this. I don't mind the syntax. I prefer k syntax though. > 3) The documentation and driver support is abysmal. This is getting a lot better. The fusion API[1] goes a long way towards better "drivers", and the new documentation site[2] shows a lot of energy being put into organization. There's also Q for mortals[3] which is linearized for people who like that. [1]: http://code.kx.com/q/interfaces/fusion/ http://code.kx.com/q/interfaces/fusion/ [2]: http://code.kx.com/q/ http://code.kx.com/q/ [3]: http://code.kx.com/q4m3/preface/ http://code.kx.com/q4m3/preface/ > 4) It's supposedly extremely fast, but I can't help but wonder if this is a lot of successful PR and hype There are a lot of good benchmarks of kdb[4] unlike Oracle :) [4]: https://stacresearch.com/m3 https://stacresearch.com/m3
amypinka 9y ago
Seems plenty fast to me: http://tech.marksblogg.com/benchmarks.html http://tech.marksblogg.com/benchmarks.html
tluyben2 9y ago
2) I like K better (fitting the ideals of APL); I feel Q was done by Arthur to please some big client as it doesn't feel like he would choose that kind of thing (that's from reading interviews, seeing the iterations and his basic code philosophy)

http-teapot 9y ago

Imagine being French and having to explain you code in Q. (I am not certain if my comment follows HN's guidelines, apologies if I offended anyone)

atrudeau 9y ago
Or the query execution plan: Plan Q
xamuel 9y ago
The French got their revenge by giving us Coq

fiatjaf 9y ago

What is this?

amypinka 9y ago
Documentation for kdb+/q. MS had a posting for their new Q# language today on HN so that might have inspired someone to post this.
- deleted 9y ago
  [deleted]

SifJar 9y ago

https://learnxinyminutes.com/docs/kdb+/ https://learnxinyminutes.com/docs/kdb+/ Another useful intro to the language

scottyelich 9y ago

... and can you use the 32bit version in commercial use? (Last I checked, no...) The license already changed once (for the worse, IMnsHO), so what's to say that won't happen again. Too risky to touch. No, can't use it (32bit version) for commercial, so this is a non-starter. Sorry. kOS? really? https://news.ycombinator.com/item?id=8475809 https://news.ycombinator.com/item?id=8475809

beagle3 9y ago
You can purchase a license if you want to be unaffected by future license changes. It might not be the right price for you, but it is for some. They are, in fact, courting only rich customers. Some languages are worth learning to expand your horizons. Lisp is one of them, even if you never use it, and the APL family (of which K/Q are members) is another. My C code has become faster, simpler, shorter and less buggy after I dabbled in K. YMMV, but using it in a commercial setting is not the only reason to look at it. > kOS? really? https://news.ycombinator.com/item?id=8475809 https://news.ycombinator.com/item?id=8475809 Really. What exactly is your "really?" question?
- vram22 9y ago
  >My C code has become faster, simpler, shorter and less buggy after I dabbled in K. Interesting. Can you explain why you think that has happened?
  - beagle3 9y ago
    Several things: 1. K (and the entire APL family) eschew many of the layers upon layers of abstraction that modern software engineering uses, whether they are justified or not. It turns out, that they are mostly not justified. K gently pushes you toward thinking in a lower level of "what's really happening here?"; I'm not sure I can give a good example here - but the world looks different after taking the red pill. e.g., it is not uncommon in K to represent a tree as an two arrays, one of data, and one of parent pointers. Once you shake the "but I must abstract this!" feeling, you realize it works better. 2. K gently encourages doing work on batches of data. That is, it is idiomatic (and easier) to write functions that operate on arrays, and return arrays of processed data, then writing functions that operate on one element at a time. In turn, this means that the resulting program more often than not works in stages where each stage processes its entire input before going to the next stage (which uses the output from this stage as its input). In the "old" C/C++/C#/Java/Python world, it is idiomatic to push each element through a pipeline before going to the next one. 3. K encourages building a set of orthogonal, non trivial operations and combining them in various ways, rather then building layer upon layer of abstraction. It gives an example by giving an extremely useful basis of functions. E.g. http://nsl.com/k/t.k http://nsl.com/k/t.k implements a very fast, reasonably capable in memory database with aggregation, joins, projects and more. It takes all of 14 lines, all quite short. While you can't implement it in 14 lines of C, using the same principles you can probably do that in less than a hundred. Achieve the same in idiomatic C is going to be much harder and longer. 4. Finally, K gently encourages solutions which work well with modern memory and storage hierarchies. E.g., it encourages linear scan operations that touch all elements of an array over random access ones that touch only 1/10 of the elements. Instinct and idiom in other languages will guide you towards the latter, but the former is often much faster.

pyvpx 9y ago

what makes kdb so special and why isn't there an open/libre alternative?

JadeNB 9y ago
> what makes kdb so special and why isn't there an open/libre alternative? Maybe I'm reading too much into this, but it seems like you expect the answer "nothing much" to the first part of your question. I have done nothing more than read lots of articles on KDB's lineage and play around a bit with J, so take my answer with a considerable lump of salt, but my impression is that the answer to the second part of your question is "because there's something 'so special' about KDB"; my understanding is that it provides blazing-fast access to memory-compact databases, from a tiny codebase, building on the APL/J legacy. Why can't this be done in an open sourced way? Well, surely there's no inherent reason, but the fact that it hasn't been is probably evidence that it's not just a problem of trivially cloning existing work (or else someone would have done it).
- throwaway7645 9y ago
  The real answer is Arthur Whitney is a god-like being with coding powers beyond the mortal realm. Seriously though they're a lot of stories of his excellent work if you search for them. He's supposedly working on kOS so you can run kdb+ on bare metal. Tiny, fast code is what he does. Sure it might look incomprehensible, but it is a few pages of code he can store in his brain at once. Aaron Hsu (someone has some links to his HN posts on here) explains this with his compiler which converts APL to GPU code. His entire compiler (which he has worked on for a long time) is a handful of pages. He says something like "there's no need for an abstraction if I can see everything at once". Most programming languages have implementations that are very long...which do you think has less bugs? Steve McConnals Code Complete has a statistic somewhere on bugs per 100 loc. That doesn't leave a lot of room in Arthur's code base (although you could say his code being so terse makes it an apples to oranges comparison I guess).
dasmoth 9y ago
There’s an open source impl of an earlier version of K: https://kona.github.io/#/ https://kona.github.io/#/ There’s also a JS implementation that aims to implement K6: http://johnearnest.github.io/ok/index.html http://johnearnest.github.io/ok/index.html I don’t believe either implement all the “database” side of kdb+ though. Edit: The J language is similar in some respects, and is GPLed. They've also got a columnar database... http://www.jsoftware.com/jdhelp/overview.html http://www.jsoftware.com/jdhelp/overview.html

kuwze 9y ago

On github kiyoto worked on this project[0] to help others understand Q code. [0]: https://github.com/kiyoto/ungod https://github.com/kiyoto/ungod

feelin_googley 9y ago

This is only software I have ever seen that actually got smaller over time. Not only that, it is the only software I have ever seen that used a GUI and then ditched it in a subsequent version. Few programs are so aligned with my own software sensibilities. Only complaint is that they used to have a FreeBSD port and now only have Linux and macOS but no BSD. Unfortunately Linux compat in BSD is being perceived by some as a potential security issue these days.

oskarth 9y ago
> This is only software I have ever seen that actually got smaller over time. I agree, this is admirable! Even if it is under different names (which I think is a far better approach), Sustrik does this too: http://250bpm.com/blog:50 http://250bpm.com/blog:50 He went from AMPQ (not his own creation) -> ZeroMQ -> nanomsg -> Libmill (essentially Go in C/UNIX style) Also, OpenBSD comes to mind (LibreSSL for example).
- beagle3 9y ago
  Libmill/Libdill (go coroutines) are not on the AMPQ->ZeroMQ->Crossroads->Nanomsg (socket abstraction) path ; in fact, they have essentially nothing in common except sustrik.
  - oskarth 9y ago
    I wish you'd actually read the article before shooting your mouth off with a Well Actually. FTA: > Next one: nanomsg. An alternative to ZeroMQ. In comments: > How would you split nanomsg, then? > [...] 2. coroutine library, e.g. libtask, libmill [...] Not to mention that this is obvious if you understand what the core of each thing is in terms of capabilities. If you are gonna be the "technically..." guy, at least do it right. It's about decomposing things, which is the essence of good design, and - supposedly - the UNIX philosophy at its heart.
    - beagle3 9y ago
      As an actual user of ZeroMQ, nanomsg and dabbler of libdill, I assure you I know what I am talking about. Libdill/libmill is NOT on the same arch. It could be used to implement the next generation of zeromq/nanomsg, but it does not, in fact, provide any functionality that these provide today - it would be an internal implementation detail if sustrik followed through on that comment. But it would not be a nanomsg rewrite, likely something completely new - nanomsg has been rewritten, several times in several ways, by Garret D'amore who has taken over. D'amore explicitly rejected using e.g. libuv (mentioned in that comment) as a basis for a nanomsg rewrite. There's an "uncertainty" principle at work here - separating of concerns generates dependencies which in the grand scheme of things are not necessarily good. In fact, one of D'amore's rewrites used native OS threads in lieu of a co-routine library, which worked perfectly well on FreeBSD and Solaris, and abysmally on all other OSses. The main reason to use a coroutine library in a messaging library is actually that the underlying OS threads implementations scale so badly on most modern systems.

nz 9y ago

Found this illuminating interview with Arthur in the ACM queue: http://queue.acm.org/detail.cfm?id=1531242 http://queue.acm.org/detail.cfm?id=1531242

johnhenry 9y ago

Not directly related, but Microsoft just released a new language for developing quantum languages, Q#, which doesn't seem to be related at all. I hope this doesn't cause too much confusion.

aamederen 9y ago

As an outsider, my initial reaction is "does every new database system have to come up with their own SQL-like language?"

beagle3 9y ago
SQL's only virtue is that it is well known; It is otherwise not very good compared to other query languages, and every SQL engine extends it somewhat differently, extensions which you can't avoid cause the standard is too limited. I agree most query languages would be better off as minor extensions to SQL - but kdb+/Q is different. Unlike SQL which pays lip service to Codd's relational model but breaks it with things like TOP, ORDER BY, LIMIT and others, the Q language embraces the order between tuples to great effect, making e.g. "as-of" queries which are quite common trivial; whereas in SQL and the relational model, as-of queries are inefficient in either execution time or storage space (usually both), and reasonable execution speed schemas cannot, in fact, guarantee their integrity (which is often quoted as one of the the main advantages of the relational model). As another example, Q implements "foreign key chasing" also called "reference chasing", which is also implemented in the web2py DAL and surely others; compare[0] the equivalent tpc-h query: in q: select revenue avg supplier.nation=`BRAZIL by order.date.year from lineitem where order.customer.nation.region=`AMERICA, order.date.year in 1995 1996, part.type=`STEEL in sql: select o_year,sum(case when nation = 'BRAZIL' then revenue else 0 end)/sum(revenue) as mkt_share from(select year(o_orderdate) as o_year,revenue,n2.n_name as nation from part,supplier,lineitem,orders,customer,nation n1,nation n2,region where p_partkey = l_partkey and s_suppkey = l_suppkey and l_orderkey = o_orderkey and o_custkey = c_custkey and c_nationkey = n1.n_nationkey and n1.n_regionkey = r_regionkey and r_name = 'AMERICA' and s_nationkey = n2.n_nationkey and o_orderdate between date('1995-01-01') and date('1996-12-31') and p_type = 'STEEL') as all_nations group by o_year order by o_year; [0] bottom of http://kparc.com/d.txt http://kparc.com/d.txt

modeless 9y ago

The KDB code is terse to an unhealthy extreme. Check this gem: // remove more clutter #define O printf #define R return #define Z static #define P(x,y) {if(x)R(y);} #define U(x) P(!(x),0) #define SW switch #define CS(n,x) case n:x;break; https://github.com/KxSystems/kdb/blob/master/c/c/k.h#L96 https://github.com/KxSystems/kdb/blob/master/c/c/k.h#L96 Or this wall of code: https://github.com/KxSystems/kdb/blob/master/c/c.cs https://github.com/KxSystems/kdb/blob/master/c/c.cs

deleted 9y ago
[deleted]
alexeiz 9y ago
It's so terse, it's gotta be fast!
kolme 9y ago
Welp, you were not kidding: https://github.com/KxSystems/kdb/blob/master/c/c/curl.c https://github.com/KxSystems/kdb/blob/master/c/c/curl.c This code is basically obfuscated by hand. Absolutely unapproachable. Only the original author(s) can understand it. Judging by other comments, it seems to work well. So they seem to be good programmers producing working code. It's just not intelligible by other human beings, which is a pretty bad thing, but not the only factor in software quality/health. The Vim code base is at some parts straight batshit insane, but it's one of the most polished programs I've ever used. All that being said, I would find infuriating to work with such code. Nope!
- beagle3 9y ago
  It takes time getting used to, yes. And it is not for all people. But it's just a foreign language; You could look at Japanese text[0] and make similar statements, and would be just as valid (or rather, invalid) as your statement. You expect to be able to read it because you're used to a class of languages which are all similar enough at the surface level -- perhaps you are even familiar with more than one fundamentally different classes, say, "lispish and algolish" or "germanic and latin". But that doesn't make APLish or Japanese[0] horrible. This is just APL using C syntax. [0] assuming, the proverbial you does not know Japanese

human_afterall 9y ago

For a second I thought you meant Q# - got pretty excited there :)

tree_of_item 9y ago

More APL threads for anyone interested: Smaller Code, Better Code: https://news.ycombinator.com/item?id=13565743 https://news.ycombinator.com/item?id=13565743 AMA: Explaining my 750 line compiler+runtime designed to GPU self-host APL: https://news.ycombinator.com/item?id=13797797 https://news.ycombinator.com/item?id=13797797

anonfunction 9y ago

Conway's Game of Life in Q: life:{3=a-x*4=a:2(sum -1 0 1 rotate\:,’/)/x}

abraae 9y ago
By comparison in APL: life←{↑1 ⍵∨.∧3 4=+/,¯1 0 1∘.⊖¯1 0 1∘.⌽⊂⍵}
- LfLxfxxLxfxx 9y ago
  life←{≢⍸⍵}⌺3 3∊¨3+0,¨⊢ https://youtu.be/3FjYly2G_QI?t=565 https://youtu.be/3FjYly2G_QI?t=565
  - RodgerTheGreat 9y ago
    I think I've nearly convinced Arthur to include an equivalent to "stencil" in k7. Fingers crossed!
    - Avshalom 9y ago
      K's approach of arrays of arrays would preclude an efficient 2d stencil operation wouldn't it?

is0tope 9y ago

I have been an on and off q user for a few years. KDB+/Q is a great system, but in my opinion it is best used for its correct purpose;and that purpose is primarily time series and data streaming. The columnar structure of the DB as well as its IPC layer make it very good at creating chains of processes that can be used to stream row updates and branch them out to different processes with different responsibilities. Likewise it's on disk database is great for running complex (time series) queries. This speed and terseness comes at a cost of being fairly "old school" in its approach. It's only recently for instance the we got stack traces, and readability is definitely not for the faint of heart though this depends on how it is written. In my view the biggest thing that is needed right now is better tooling and libraries. Some attempts have been made to do this, and I am hearing that the new initiatives by kx will be addressing this in the coming months. The lack of standardized testing library/framework also can be problematic, as every team that I have seen does it slightly differently, and a "best practice" would beneficial.

beaumayns 9y ago

I mostly love K/Q, and recently rearranged my career to do more of it. There are some things to be aware of, though: * If your code doesn't spend most of its time in primitive verbs operating on large vectors, it's gonna be more or less as slow as any other interpreted language. Q and kdb+ can be fast and beautiful if you can arrange your problem in the right way, but it's not magic. * The internals are locked away. If you don't like the way something fundamental works, tough. I've known some folks to go to heroic lengths with debuggers and hacked up shared objects to get Q to do what they want. You could also get Kx to add the stuff you need (they're pretty reasonable and responsive). But, you can't really take it apart and put it back together again like you can with, say, Lua, Ruby, or Python. * Relating to the above point, one of the weaknesses of the language is that there are a lot of useful (even necessary) features packed into weird corners. There's little room for abstractions beyond the basics, so you get stuff like CSV parsing controlled by the structure of lists passed to a function called "0:". It's getting better documented lately, but it's still not pretty. * Various annoyances (no real module system, no lexical scoping, etc...) In many of those cases, I'm not even sure what could be done without compromising some other aspect of the language. Most of the time (at least for me), it's really a joy to use.

Bootvis 9y ago
Curious to know what you did to rearrange your career.

petermora 9y ago

I worked in Pascal, C/C++, Java, Python, R, Nim, Clojure, Rust, Go, Js, and created various projects in these languages. However, I couldn't get used to Q. I understand that it is fast, and I also started to like the functional programming aspect of it. But oh boy, there is no proper error messages (no, "`type" is not helpful). The short versions of various map and apply were handy, but there are no equivalent versions with longer names, therefore there is no way to write a code which is readable to a non Q-expert. The strange evaluation order made it impossible for me to read other people's code, I often started to add parentheses and checked whether the behavior changed. There is no debugger (I used KdbStudio). Once I used more than 16 local variables, which gave a runtime error with some strange unhelpful error message. I'm not questioning its usefulness, but I think that it could be much more developer friendly without compromising speed.

easytiger 9y ago
> therefore there is no way to write a code which is readable to a non Q-expert. If you could FD wouldn't be able to justify charging £1.5k/day per head consultancy fees for someone they just sent on a 1 week training course.
geocar 9y ago
I have worked in C (twenty-five years), C++, Python, Perl (twenty years), JavaScript, Common Lisp (for ten years), OCaml, forth, postscript, and also q/Kdb. I think a lot can be done to improve teaching q/Kdb. The current “best practice” is to change you until it makes sense but this takes time- anywhere from 6 months to a couple years based on your other experiences. And you still have to want to “get it”. I want that to be better. However this is alien technology: the terseness is a feature. Limited locals are a feature. The evaluation order is a feature. These things actually help it go fast (as surprising as that is!) We only just recently got “a debugger” and good error messages because of repeated complaints and wishes for it from newbies, but there is a reason experienced Kdb/q programmers never wanted it. Why doesn’t that reason prompt those newbies to figure out why? Yes, ideally you would learn the way and the why we do things, but there are a number of serious hurdles to overcome: To you q/Kdb seems merely unpolished even if maybe “fast”. Is it worth it? “Maybe”, you suggest, but then there are crazy people like me telling you something preposterous; that q/Kdb is actually incredibly well crafted, highly readable and an absolute pleasure to use. I'm also trying to say that everything is worth learning: Nim and Python and JavaScript are all basically the same thing. You learned one of them, you kindof learned them all, so adding another one feels like these things are easy to learn. Alien technology is alien though, you haven't learned any of it. How can we even talk to each other? I'm hopeful: Lambda was tricky, but it snuck in to things. Can we get tables, views, and high code density? We need to find something better- some better way to talk about it, but the peanut gallery is loud.

cousin_it 9y ago

I used to play with K long ago, it was tons of fun. Sometimes I think I should've been a quant and made crazy money :-)

anonu 9y ago

Q is great. I think wider adoption would come from two places: 1. adding more syntactic sugar to the language to improve readability 2. open-sourcing the thing or building an open-source compatible Q interpreter with all the nice features of KDB (functional programming, vector-based data structures, "scripting language within a database" approach, web features: HTTP server and websockets, etc.. etc.. )