15 ms·

While I really like K/Q/J/APL (some of this is Stockholm Syndrome: I spent the first 3 years of my career writing K/Q at a trading firm, eventually becoming the

by ktamura 10y ago

While I really like K/Q/J/APL (some of this is Stockholm Syndrome: I spent the first 3 years of my career writing K/Q at a trading firm, eventually becoming the human debugger for my team's K/Q "investment". It got so annoying that I worked on a little parser library to help me debug: https://github.com/kiyoto/ungod https://github.com/kiyoto/ungod), they are doomed from a adoption/future viability point of view. 1. Steep learning curve without benefits: APLs like K/Q/J optimize for concision to the point of unreadability. Sure, it might be fun to decipher clever K/Q one-liners, but if you have to do it everyday for work in production, it's just exhausting. This lack of readability has limited adoption and the growth of its user community. For any technology is to have a viable community, it needs to be usable by a critical mass. K/Q in specific but APLs in general never had that, especially with more user-friendly alternatives like R/Matlab for scientific computing. 2. No open source community. There's Kona (https://github.com/kevinlawler/kona https://github.com/kevinlawler/kona) by Kevin Lawler, but again, its community has been small because the underlying language is not design to be usable by enough people. 3. Domain specificity of array languages: These days, array languages are just too limited in its scope. Scientific computing, the bread and butter of any array programming language, has been made possible in other languages (ex: Python), making array languages not powerful nor compelling enough to use because most of its strengths can be found in other languages and its weaknesses are too crippling. I heard that Arthur Whitney found his successors (a pair of Russian programmers from St. Petersburg, iirc), so K/Q will likely to be around beyond his retirement. That said, I think its user base will continue to shrink for above reasons.

f3lix133 10y ago
'3 years of my career writing K/Q at a trading firm, eventually becoming the human debugger for my team's K/Q "investment"' - that sounds so familiar to me lol, except it's my second year of using kdb in a trading firm, but still cannot run away from becoming a human debugger, and all I can use as a reference is code.kx.com.
- jzwinck 10y ago
  Is code.kx.com your only resource because there's not much else written publicly in these languages, or is it because your employer blocks you from using other resources which are publicly available?
- branchless 10y ago
  surely the error code makes it clear?!? 'type
jsmthrowaway 10y ago
I got exposed to kdb+ while interviewing in quant and would kill to use it in a non-financial metrics setting, but the language and ecosystem just can't let me get there. kdb+ is so fast and powerful you'd think it impossible. They're not directly comparable, but kdb+ would crush InfluxDB head to head in an apple/orange comparison if used for the same thing. I actually can't think of any time series-like store that remotely comes close to the feats I have seen kdb+ with Q accomplish. Too bad the productivity and knowledge overhead is so high. Seriously, I'm working in realtime metrics right now and I spent a week playing with kdb+ as the central store. I can tell there's a gold mine there, but I'm just not swinging a big enough pickaxe. If there was a dumb Python/Go/etc typical ops hacker frontend to that whole system I would be throwing every dollar I have to my name at it; my kingdom for burying that system under a few layers of abstraction and paying some quant Q hacker a Ferrari or two a year to hide the sausagemaking from everyone else. (Related aside: Like building furniture from stacks of cash? Learn K/Q/kdb+ and head for NYC.)
- gd1 10y ago
  It is very sad that they won't open up, and very short-sighted as well. I'm sure Whitney and KX are perfectly happy with their millions, but they are sitting on one of the most impressive software achievements around and refusing to share it. It's not even like it would cost them. First Derivatives could grow a consulting empire. I mean, look at open-source MongoDB. It is a piece of shit, and it is valued at $1.4B. KDB is finest thing I have ever used, and First Derivatives purchased a controlling stake in KX for £36.0M. It makes no sense. Maybe kOS will be open-source. Maybe it will be the big one, Arthur's chance to change the course of software development history.
  - bladecatcher 10y ago
    It is really sad indeed. But I suggest you take a look at the J language. It's free for commercial use, unless you want to integrate the JDB, the equivalent of KDB+ column store. It also provides a rich feature set: http://code.jsoftware.com/wiki/JDB/Announcement http://code.jsoftware.com/wiki/JDB/Announcement http://www.jsoftware.com/jdhelp/overview.html http://www.jsoftware.com/jdhelp/overview.html https://scottlocklin.wordpress.com/2012/09/18/a-look-at-the-j-language-the-fine-line-between-genius-and-insanity/ https://scottlocklin.wordpress.com/2012/09/18/a-look-at-the-... comes very close to capturing the ingenuity of J language. After going through the J primer, KDB+/Q feels like a distillation of J - it basically took the most accessible ideas of J and packaged them for mainstream production use - some of their features such as these make it easy to setup a computing harness for electronic trading in no time: http://code.kx.com/wiki/Cookbook/LoadingFromLargeFiles http://code.kx.com/wiki/Cookbook/LoadingFromLargeFiles http://code.kx.com/wiki/Startingkdbplus/tick http://code.kx.com/wiki/Startingkdbplus/tick Also, the footprint of the kdb runtime and installation is unparalleled. I've run systems 24 x 7 x 365
rtpg 10y ago
RE the learning curve: Do you think the APL family can exist without the terse syntax? Are there any aspects of the language that just wouldn't work well without it? A comparison I can think of is that Lisp's whole code/data duality works well because the code is s-exprs, and that it would be pretty awkward to transform that if the syntax were more python-y On the other hand, semicolons and braces have nothing to do with any Java features....
- beagle3 10y ago
  There's Kevin Lawler's new language, Kerf[0], which is a more verbose APL. [0] http://www.kerfsoftware.com/ http://www.kerfsoftware.com/
- avmich 10y ago
  > Do you think the APL family can exist without the terse syntax? Are there any aspects of the language that just wouldn't work well without it? Considering the evolution of APL, yes. Ken Iverson was teaching math in Harvard, when he was growing the - mathematical - notation to describe the subject formally. It was happening in late 1950's, and the computers were entering our life. At some point it became reasonable to use that evolved notation as A Programming Language - or APL, for short. However, the idea was to make understanding easier - all along! Same idea which Alan Kay is pursuing with his works, same idea with many good technologies, when they are young - Java, JavaScript... Importance of notation - be it in math, or in formal-and-executable-math is the ideology of early APL. Now they say "it's hard to beat expressiveness of a whiteboard", but still attempt to use, say, an integral sign (and, similarly, one-letter variables), as one would do when showing the idea on a whiteboard. The history of hand-written math preferred terse notation - and that is carried on in APL as executable math. So... yes, terseness is important, and APLers will also say that choice of symbols is important too. It's only to beginners that they feel strange - pretty soon programmer learns them, gets used to them and refuses to substitute them with more readable longer variants - even though that would make certain things better. You don't name variables on a whiteboard with long_names or camelCase - at best you use sub- and superscripts. Granted, you can have cups and power towers and other things, and they were really impractical with 1960's level technologies... yet at least you have a singular text direction for expressions, in editors or in print. J makes certain things more logical, and switches to ASCII. May be we'll invent even better notation (to me, this - http://matt.might.net/articles/discrete-math-and-code/ http://matt.might.net/articles/discrete-math-and-code/ - looks like a good start to think about basic building blocks), but so far, we have APL family languages as arguably closest thing to math notation. To some, it's the closest way they can imagine between making a solution in the head and explaining that to computer.
coliveira 10y ago
J language is open source, this makes it viable without a company to support it. Also, terseness may be a problem, but only when you're not used to it. Normal languages use a vertical structure, while APL languages use a horizontal structure. It is just a matter of looking for meaning in each line, instead of trying to combine lines to get the meaning of code, like we do in standard languages.
- pyrale 10y ago
  Ook is open-sourced too, and terseness is always a problem when you intend to work in a group of people or over long durations. Maybe you're one of the happy few that spends more time writing code than reading it, but for an overwhelming majority of devs, the opposite is true. That's why even small readability improvements matter much more than a couple cool tricks a language may have.
  - klibertp 10y ago
    Very long time ago I heard that C is completely unreadable[1]. I went and learnt it some time later and discovered that it was bullshit. Then I heard that C++ is unreadable[2] (I mean the source, not error messages...). I went and learned it and realized this, too, was bullshit. Another couple of years forward and I heard that it's impossible to read PHP code[3]. So I learned PHP (and started earning money thanks to it) and realised that "impossible is just a word". I then heard that indentation delimited blocks in Python make the code impossible to edit (and unreadable, btw)[4]. I learned it and realized that it's rather easy to configure your editor and make this non-issue. Around that time I heard that PERL is a write-only language[5]. I wanted to check, so I went and learned PERL. It took some time, but eventually I realized that, indeed, it's possible to read PERL. To make the rest of the story short: around that time, I decided I want to go polyglot and started researching various niche languages, like Lisp, Forth, Prolog, J, Factor, Avail, Haskell, Erlang and so on. I heard (mostly) that I'm insane and that it's impossible to either read or write any such language[1][2][3][4][5]. As you probably guessed already I went and learned most of them and realized that - shocking! - it is possible to both read and write all of them! Now, I don't know either K or Q, but I have a very, very, very hard time believing that they are too terse and unreadable because of that. From my experience, the "improvements in readability" are of course good, but they only shorten the time to master the language. Once you learn the language well enough, it becomes readable. And the very few syntactic issues which are objectively hard to distinguish/are confusing the eye, etc. tend to be worked around with formatting and font settings. Written Japanese is totally impossible to read if you don't know it - and it takes reasonably longer to learn it. However, once you know it, it's readable. A Japanese person reading a newspaper in Japanese uses as much effort as me reading a newspaper in my native language. [1] From Pascal and later Delphi users. [2] From C users. [3] From PERL users. [4] From PHP users. [5] From Python users.
bladecatcher 10y ago
I've been writing q code for more than 6 years now. It's definitely possible to write code which doesn't look like line noise. It's easy to build an automated testing layer on top of your q codebase too (via the c or Java bridges). I feel it's a perfectly legitimate language for high performance computing Edit: I recently started learning the J language. While it may not necessarily be a language you want to write your production code in, it teaches powerful concepts - it let's you write functions which can deal with radically different shapes of data with ease, via use of the operators to change the behavior of a verb to monad or dyad. It provides great insight into higher level descriptions of your domain
pmarreck 10y ago
I've done vastly less debugging after adopting TDD; does K/Q have a not so good testing facility? (It's of course not very hard to erect your own.)

by ktamura 10y ago

f3lix133 10y ago

'3 years of my career writing K/Q at a trading firm, eventually becoming the human debugger for my team's K/Q "investment"' - that sounds so familiar to me lol, except it's my second year of using kdb in a trading firm, but still cannot run away from becoming a human debugger, and all I can use as a reference is code.kx.com.

jzwinck 10y ago
Is code.kx.com your only resource because there's not much else written publicly in these languages, or is it because your employer blocks you from using other resources which are publicly available?
branchless 10y ago
surely the error code makes it clear?!? 'type

jsmthrowaway 10y ago

I got exposed to kdb+ while interviewing in quant and would kill to use it in a non-financial metrics setting, but the language and ecosystem just can't let me get there. kdb+ is so fast and powerful you'd think it impossible. They're not directly comparable, but kdb+ would crush InfluxDB head to head in an apple/orange comparison if used for the same thing. I actually can't think of any time series-like store that remotely comes close to the feats I have seen kdb+ with Q accomplish. Too bad the productivity and knowledge overhead is so high. Seriously, I'm working in realtime metrics right now and I spent a week playing with kdb+ as the central store. I can tell there's a gold mine there, but I'm just not swinging a big enough pickaxe. If there was a dumb Python/Go/etc typical ops hacker frontend to that whole system I would be throwing every dollar I have to my name at it; my kingdom for burying that system under a few layers of abstraction and paying some quant Q hacker a Ferrari or two a year to hide the sausagemaking from everyone else. (Related aside: Like building furniture from stacks of cash? Learn K/Q/kdb+ and head for NYC.)

gd1 10y ago
It is very sad that they won't open up, and very short-sighted as well. I'm sure Whitney and KX are perfectly happy with their millions, but they are sitting on one of the most impressive software achievements around and refusing to share it. It's not even like it would cost them. First Derivatives could grow a consulting empire. I mean, look at open-source MongoDB. It is a piece of shit, and it is valued at $1.4B. KDB is finest thing I have ever used, and First Derivatives purchased a controlling stake in KX for £36.0M. It makes no sense. Maybe kOS will be open-source. Maybe it will be the big one, Arthur's chance to change the course of software development history.
- bladecatcher 10y ago
  It is really sad indeed. But I suggest you take a look at the J language. It's free for commercial use, unless you want to integrate the JDB, the equivalent of KDB+ column store. It also provides a rich feature set: http://code.jsoftware.com/wiki/JDB/Announcement http://code.jsoftware.com/wiki/JDB/Announcement http://www.jsoftware.com/jdhelp/overview.html http://www.jsoftware.com/jdhelp/overview.html https://scottlocklin.wordpress.com/2012/09/18/a-look-at-the-j-language-the-fine-line-between-genius-and-insanity/ https://scottlocklin.wordpress.com/2012/09/18/a-look-at-the-... comes very close to capturing the ingenuity of J language. After going through the J primer, KDB+/Q feels like a distillation of J - it basically took the most accessible ideas of J and packaged them for mainstream production use - some of their features such as these make it easy to setup a computing harness for electronic trading in no time: http://code.kx.com/wiki/Cookbook/LoadingFromLargeFiles http://code.kx.com/wiki/Cookbook/LoadingFromLargeFiles http://code.kx.com/wiki/Startingkdbplus/tick http://code.kx.com/wiki/Startingkdbplus/tick Also, the footprint of the kdb runtime and installation is unparalleled. I've run systems 24 x 7 x 365

rtpg 10y ago

RE the learning curve: Do you think the APL family can exist without the terse syntax? Are there any aspects of the language that just wouldn't work well without it? A comparison I can think of is that Lisp's whole code/data duality works well because the code is s-exprs, and that it would be pretty awkward to transform that if the syntax were more python-y On the other hand, semicolons and braces have nothing to do with any Java features....

beagle3 10y ago
There's Kevin Lawler's new language, Kerf[0], which is a more verbose APL. [0] http://www.kerfsoftware.com/ http://www.kerfsoftware.com/
avmich 10y ago
> Do you think the APL family can exist without the terse syntax? Are there any aspects of the language that just wouldn't work well without it? Considering the evolution of APL, yes. Ken Iverson was teaching math in Harvard, when he was growing the - mathematical - notation to describe the subject formally. It was happening in late 1950's, and the computers were entering our life. At some point it became reasonable to use that evolved notation as A Programming Language - or APL, for short. However, the idea was to make understanding easier - all along! Same idea which Alan Kay is pursuing with his works, same idea with many good technologies, when they are young - Java, JavaScript... Importance of notation - be it in math, or in formal-and-executable-math is the ideology of early APL. Now they say "it's hard to beat expressiveness of a whiteboard", but still attempt to use, say, an integral sign (and, similarly, one-letter variables), as one would do when showing the idea on a whiteboard. The history of hand-written math preferred terse notation - and that is carried on in APL as executable math. So... yes, terseness is important, and APLers will also say that choice of symbols is important too. It's only to beginners that they feel strange - pretty soon programmer learns them, gets used to them and refuses to substitute them with more readable longer variants - even though that would make certain things better. You don't name variables on a whiteboard with long_names or camelCase - at best you use sub- and superscripts. Granted, you can have cups and power towers and other things, and they were really impractical with 1960's level technologies... yet at least you have a singular text direction for expressions, in editors or in print. J makes certain things more logical, and switches to ASCII. May be we'll invent even better notation (to me, this - http://matt.might.net/articles/discrete-math-and-code/ http://matt.might.net/articles/discrete-math-and-code/ - looks like a good start to think about basic building blocks), but so far, we have APL family languages as arguably closest thing to math notation. To some, it's the closest way they can imagine between making a solution in the head and explaining that to computer.

coliveira 10y ago

J language is open source, this makes it viable without a company to support it. Also, terseness may be a problem, but only when you're not used to it. Normal languages use a vertical structure, while APL languages use a horizontal structure. It is just a matter of looking for meaning in each line, instead of trying to combine lines to get the meaning of code, like we do in standard languages.

pyrale 10y ago
Ook is open-sourced too, and terseness is always a problem when you intend to work in a group of people or over long durations. Maybe you're one of the happy few that spends more time writing code than reading it, but for an overwhelming majority of devs, the opposite is true. That's why even small readability improvements matter much more than a couple cool tricks a language may have.
- klibertp 10y ago
  Very long time ago I heard that C is completely unreadable[1]. I went and learnt it some time later and discovered that it was bullshit. Then I heard that C++ is unreadable[2] (I mean the source, not error messages...). I went and learned it and realized this, too, was bullshit. Another couple of years forward and I heard that it's impossible to read PHP code[3]. So I learned PHP (and started earning money thanks to it) and realised that "impossible is just a word". I then heard that indentation delimited blocks in Python make the code impossible to edit (and unreadable, btw)[4]. I learned it and realized that it's rather easy to configure your editor and make this non-issue. Around that time I heard that PERL is a write-only language[5]. I wanted to check, so I went and learned PERL. It took some time, but eventually I realized that, indeed, it's possible to read PERL. To make the rest of the story short: around that time, I decided I want to go polyglot and started researching various niche languages, like Lisp, Forth, Prolog, J, Factor, Avail, Haskell, Erlang and so on. I heard (mostly) that I'm insane and that it's impossible to either read or write any such language[1][2][3][4][5]. As you probably guessed already I went and learned most of them and realized that - shocking! - it is possible to both read and write all of them! Now, I don't know either K or Q, but I have a very, very, very hard time believing that they are too terse and unreadable because of that. From my experience, the "improvements in readability" are of course good, but they only shorten the time to master the language. Once you learn the language well enough, it becomes readable. And the very few syntactic issues which are objectively hard to distinguish/are confusing the eye, etc. tend to be worked around with formatting and font settings. Written Japanese is totally impossible to read if you don't know it - and it takes reasonably longer to learn it. However, once you know it, it's readable. A Japanese person reading a newspaper in Japanese uses as much effort as me reading a newspaper in my native language. [1] From Pascal and later Delphi users. [2] From C users. [3] From PERL users. [4] From PHP users. [5] From Python users.

bladecatcher 10y ago

I've been writing q code for more than 6 years now. It's definitely possible to write code which doesn't look like line noise. It's easy to build an automated testing layer on top of your q codebase too (via the c or Java bridges). I feel it's a perfectly legitimate language for high performance computing Edit: I recently started learning the J language. While it may not necessarily be a language you want to write your production code in, it teaches powerful concepts - it let's you write functions which can deal with radically different shapes of data with ease, via use of the operators to change the behavior of a verb to monad or dyad. It provides great insight into higher level descriptions of your domain

pmarreck 10y ago

I've done vastly less debugging after adopting TDD; does K/Q have a not so good testing facility? (It's of course not very hard to erect your own.)