9 ms·

Show HN: Persistent Group Chat in kdb+/k/q

45 points by srpeck 12y ago

vizzah 12y ago
Looks to me like the app itself (chat.htm) is missing from the repository?
- srpeck 12y ago
  Thanks, added.
ketralnis 12y ago
f:<:`:index.txt / Open message log append-only h:("\n"/:0::`:index.txt),"\n" / Load message history from log .z.po:{(-:.z.w)h} / On join, send full message history .z.ws:{s:(" "/:($"du"$.z.Z))," ",x; / On message, add server timestamp {(-:x)y}\:[!:.z.W;s]; / Fan out to other connections h::(h,s,"\n"); / Update message history in memory (-:f)s} / Append to message log file .z.ph:{"\n"/:0::`:chat.htm} / Serve static client app That is really quite inscrutable
- srpeck 12y ago
  Is q more readable to you? file:hopen`:index.txt / Open message log append-only history:("\n"sv read0`:index.txt),"\n" / Load message history from log .z.po:{neg[.z.w]history} / On join, send full message history .z.ws:{s:(" "sv(string"du"$.z.Z))," ",x; / On message, add server timestamp {neg[x]y}\:[key .z.W;s]; / Fan out to other connections history::(history,s,"\n"); / Update message history in memory (neg file)s} / Append to message log file .z.ph:{"\n"sv read0`:chat.htm} / Serve static client app
- RodgerTheGreat 12y ago
  Have you ever read any documentation, tutorials or explanations of K? It seems rather odd to expect to be able to understand a programming language without learning about it first.
  - jnbiche 12y ago
    > It seems rather odd to expect to be able to understand a programming language without learning about it first. Really? You can't understand a new language that's in the same general paradigm as one you know already? Because I can, and I'm no PL genius. I've been able to very quickly understand code written in Swift, Ceylon, Pyret, Rust, Nimrod, and various other new-to-me languages in the past few years. But I can't make heads or tails of K. And that's clearly the way it's meant to be. It's the epitome of an esoteric language.
    - RodgerTheGreat 12y ago
      That's exactly it- K isn't in the same general paradigm as those languages. K resembles APL and other array programming languages. Syntax among those languages differs in some ways, but the most basic elements are the same. If you knew APL or J, K wouldn't look so alien.
    - beagle3 12y ago
      That's because Swift, Ceylon, Pyret, Rust, Nimrod make (relatively) small changes in syntax and semantics compared to Python, Java, C and friends - this is the Algol family of languages if you are old enough (and the "C" family if not). Can you read Prolog (assuming you're not fluent in it already)? It is a little farther along from Algol - it is generally verbose, but control flow is implicit (by failure, and negation IS failure), except where you make it explicit by saying '!' (read: "cut"), which means "never backtrack through this point". And there's no assignment, but unification plays a similar part - except it is much weirder (to the uninitiated). K is of the APL / J mold, which is much farther away from Algol and Prolog. It's mostly functional. It takes orthogonality beyond where Haskell does. Examples that might make you look again: "flatten" (convert recursively nested list/tree to one-level list with the same order as a DFS scan) is written ",//" "decode run length decoding" is written ",/#/'" (courtesy of https://github.com/JohnEarnest/ok/blob/gh-pages/examples/runlength.k https://github.com/JohnEarnest/ok/blob/gh-pages/examples/run... Now, these 3-char and 5-char examples are not predefined in any way; They emerge from composing the semantics of the individual characters that make them. Let that sink. Are you aware of any other language in which a straight concatenation of 3 primitives generates a "flatten" function? 5 primitives generates a "run length decoding" function? Something about K/APL/J is distinctly different. And if you believe that it looks different becuase it's "the way it's meant to be", you're missing out on a lot.
- Avshalom 12y ago
  All it's doing is opening a file, connecting to other clients and appending the messages between clients to the file. What you're missing is that K automatically sets up an environment with the necessary pieces in place. A server, the sockets, a type of reactive/eventloopy environment. Everything else is just function call/index/assignment and some mappy/foldy stuff In some ways its the same reason Mathematica seems so magic sometimes.
jnbiche 12y ago
Forget Perl, this is serious write-only software engineering. I mean, I'm a language polyglot, and I think these languages are probably great for multidimensional array processing. But for general purpose programming, not so much. But cool hack.
- mmagin 12y ago
  Every time these k folks show up on HN, it feels like Chuck Moore and Color Forth.
- geocar 12y ago
  Don't dismiss things you don't understand: I think array programming is extremely useful for parsing, and I also think that with some untraining and some practice you would find it easy to read and write. I have a new programmer- strong maths background, but no programming experience prior to K which she started about two weeks ago. Today, she's working on taking strings that look like this: y:"http://host/path/to.a.swf,http://host/path/to/b.swf" and finding out if the url-fragments are "claimed" by any any other user in a database, AND if the current user claims any url-fragments. So she has a database like this: d:`f`u!(("adap.tv";"adap.tv";"btrll.com";"btrll.com";"cbs.com";"cbsinter… l:?d`f and in about thirty minutes of puzzling came up with this: |/#:'y ss/:l[&~|/d.f[&x=d.u]~/:\:l] Since you don't know K I'll walk you through it: d.f[&x=d.u]~/:\:l This returns a list of bitmaps, such as: (10000000000000000b;00000100000000000b) What it's doing is taking all of the urls and indexing it by the where (&) address of each x (parameter to function: user id) is equal to the array of all of the users. That is, x=d.u returned: 01000001000000000000000000b and &x=d.u returned 1 7 (because the second and eighth element of that bitmap are ones). d.f[1 7] returns the second and eighth fragment. Then, she &~|/ which ors- the bitmaps (|/) together, finds their complement (~), and takes the where of (&) them, so in this case 0 5. l[0 5] then returns the strings in the master list. Note this is the list of all of the url fragments owned by other users: l[&~|/d.f[&x=d.u]~/:\:l] In PHP you might try to write: $mine = array(); $theirs = array(); foreach($db as $row) { if($row->u === $x) $mine[$row->f]=0; else $thiers[$row->f]=0; } $list = array_diff(array_keys($thiers),array_keys($mine)); but there's a lot of syntax here, and a number of PHP function calls that a beginner might not be aware of. She's just using this: http://www.timestored.com/kdb-guides/files/kdb-cheat-sheet.pdf So then on to y ss/: which basically does a `strpos()` of each of our urls against the list we're scanning (i.e. the thing that looks like "http://url,http://otherurl" http://url,http://otherurl". She then counts each (#:') result, takes the max again (|/). This means if ANY of those match, return `1b` otherwise return `0b`.
doublerebel 12y ago
Fantastic, I've been really interested to see a nonfinancial application of k that I could compare to implementations in languages I'm familiar with. If anyone else knows of other such k applications please share them!
- RodgerTheGreat 12y ago
  I was just tinkering with a tic tac toe game: https://github.com/JohnEarnest/ok/blob/gh-pages/examples/tictac.k https://github.com/JohnEarnest/ok/blob/gh-pages/examples/tic... I think I can still make a number of simplifications and generalizations, and I'd like to have a stab at an alpha-beta AI opponent. There's also a great deal of interesting code at No Stinking Loops: http://www.nsl.com http://www.nsl.com
- srpeck 12y ago
  Have a look at Arthur's text editor in k5: http://kparc.com/edit.k http://kparc.com/edit.k And of course the impending kOS operating system: http://kparc.com/ http://kparc.com/ with HN discussion: https://news.ycombinator.com/item?id=8475809 https://news.ycombinator.com/item?id=8475809 There are also some good links here, though you may have to dig: http://www.reddit.com/r/apljk/ http://www.reddit.com/r/apljk/
geocar 12y ago
Looks really good! F:"\n"/:0::;C:,[;"\n"];I:`:index.txt;f:-<I;h:C F I;H::F`:chat.htm .z.ws:{s:(" "/:($"du"$.z.Z),,x);{x y}\:[-!.z.W;s];h,::C s;f s} .z.po:{(-.z.w)h};.z.ph:{H} 1. "\n"/:0:: occurs twice and can be made a function (F) 2. You can project the `y` argument by itself (C) 3. Shorten symbols (I) 4. No need to (-f) all over the place when -< will do. (f) 5. You can load chat.htm once (H; .z.ph), but unfortunately K4 won't notice changes to the file, so am a bit divided about this one. 6. Save brackets! Join-enlist (,,), negate til (-!), and amend (,::) (.z.ws)
- srpeck 12y ago
  Thanks for the feedback; I pushed your cleaned up version to Github (https://github.com/srpeck/kchat/commit/5669fcca67fa609a9128b0ba65e34aea7adf5486 https://github.com/srpeck/kchat/commit/5669fcca67fa609a9128b...). As for your comment in #5, I noticed that I can hot load new versions of the server and client without impacting the existing clients' web socket connections. A production change to chat.htm only requires me to \l chat.k; any existing connections still have the older client running without interruption until they refresh, but all new connections get the new client. However, I am not all that concerned about hot loading new versions, as this is only deployed for a small team on the intranet.
  - geocar 12y ago
    Indeed! I sometimes use emacs and use an after-save-hook to send a command (e.g. to reload a file).
anonu 12y ago
ive been using kdb+/q for over 5 years now - in a financial setting. I'd be happy to see the language go more "mainstream". There are definitely a lot of complaints about the terseness. There's no doubt, this adds to the steepness of the learning curve. But once you "see the light" you start realizing that you can do the same thing you do in perl or python in just 1 line in KDB. It makes more sense.
- kbody 12y ago
  I'm working on a algo-trading system and thinking of moving my tick data to kdb+, however I see no price on their site, care to enlight? Also, do you know what are the limits of the free evaluation/32-bit version? Thanks
  - beagle3 12y ago
    32 bit version is 32 bit (databases limited to about 2GB per process, as are memory mappings). If that's good for you, use it. The 64-bit was, last I inquired (8 years ago or so) was in the "if you have to ask you can't afford it" oracle territory.

45 points by srpeck 12y ago

vizzah 12y ago

Looks to me like the app itself (chat.htm) is missing from the repository?

srpeck 12y ago
Thanks, added.

ketralnis 12y ago

f:<:`:index.txt / Open message log append-only h:("\n"/:0::`:index.txt),"\n" / Load message history from log .z.po:{(-:.z.w)h} / On join, send full message history .z.ws:{s:(" "/:($"du"$.z.Z))," ",x; / On message, add server timestamp {(-:x)y}\:[!:.z.W;s]; / Fan out to other connections h::(h,s,"\n"); / Update message history in memory (-:f)s} / Append to message log file .z.ph:{"\n"/:0::`:chat.htm} / Serve static client app That is really quite inscrutable

srpeck 12y ago
Is q more readable to you? file:hopen`:index.txt / Open message log append-only history:("\n"sv read0`:index.txt),"\n" / Load message history from log .z.po:{neg[.z.w]history} / On join, send full message history .z.ws:{s:(" "sv(string"du"$.z.Z))," ",x; / On message, add server timestamp {neg[x]y}\:[key .z.W;s]; / Fan out to other connections history::(history,s,"\n"); / Update message history in memory (neg file)s} / Append to message log file .z.ph:{"\n"sv read0`:chat.htm} / Serve static client app
RodgerTheGreat 12y ago
Have you ever read any documentation, tutorials or explanations of K? It seems rather odd to expect to be able to understand a programming language without learning about it first.
- jnbiche 12y ago
  > It seems rather odd to expect to be able to understand a programming language without learning about it first. Really? You can't understand a new language that's in the same general paradigm as one you know already? Because I can, and I'm no PL genius. I've been able to very quickly understand code written in Swift, Ceylon, Pyret, Rust, Nimrod, and various other new-to-me languages in the past few years. But I can't make heads or tails of K. And that's clearly the way it's meant to be. It's the epitome of an esoteric language.
  - RodgerTheGreat 12y ago
    That's exactly it- K isn't in the same general paradigm as those languages. K resembles APL and other array programming languages. Syntax among those languages differs in some ways, but the most basic elements are the same. If you knew APL or J, K wouldn't look so alien.
  - beagle3 12y ago
    That's because Swift, Ceylon, Pyret, Rust, Nimrod make (relatively) small changes in syntax and semantics compared to Python, Java, C and friends - this is the Algol family of languages if you are old enough (and the "C" family if not). Can you read Prolog (assuming you're not fluent in it already)? It is a little farther along from Algol - it is generally verbose, but control flow is implicit (by failure, and negation IS failure), except where you make it explicit by saying '!' (read: "cut"), which means "never backtrack through this point". And there's no assignment, but unification plays a similar part - except it is much weirder (to the uninitiated). K is of the APL / J mold, which is much farther away from Algol and Prolog. It's mostly functional. It takes orthogonality beyond where Haskell does. Examples that might make you look again: "flatten" (convert recursively nested list/tree to one-level list with the same order as a DFS scan) is written ",//" "decode run length decoding" is written ",/#/'" (courtesy of https://github.com/JohnEarnest/ok/blob/gh-pages/examples/runlength.k https://github.com/JohnEarnest/ok/blob/gh-pages/examples/run... Now, these 3-char and 5-char examples are not predefined in any way; They emerge from composing the semantics of the individual characters that make them. Let that sink. Are you aware of any other language in which a straight concatenation of 3 primitives generates a "flatten" function? 5 primitives generates a "run length decoding" function? Something about K/APL/J is distinctly different. And if you believe that it looks different becuase it's "the way it's meant to be", you're missing out on a lot.
Avshalom 12y ago
All it's doing is opening a file, connecting to other clients and appending the messages between clients to the file. What you're missing is that K automatically sets up an environment with the necessary pieces in place. A server, the sockets, a type of reactive/eventloopy environment. Everything else is just function call/index/assignment and some mappy/foldy stuff In some ways its the same reason Mathematica seems so magic sometimes.

jnbiche 12y ago

Forget Perl, this is serious write-only software engineering. I mean, I'm a language polyglot, and I think these languages are probably great for multidimensional array processing. But for general purpose programming, not so much. But cool hack.

mmagin 12y ago
Every time these k folks show up on HN, it feels like Chuck Moore and Color Forth.
geocar 12y ago
Don't dismiss things you don't understand: I think array programming is extremely useful for parsing, and I also think that with some untraining and some practice you would find it easy to read and write. I have a new programmer- strong maths background, but no programming experience prior to K which she started about two weeks ago. Today, she's working on taking strings that look like this: y:"http://host/path/to.a.swf,http://host/path/to/b.swf" and finding out if the url-fragments are "claimed" by any any other user in a database, AND if the current user claims any url-fragments. So she has a database like this: d:`f`u!(("adap.tv";"adap.tv";"btrll.com";"btrll.com";"cbs.com";"cbsinter… l:?d`f and in about thirty minutes of puzzling came up with this: |/#:'y ss/:l[&~|/d.f[&x=d.u]~/:\:l] Since you don't know K I'll walk you through it: d.f[&x=d.u]~/:\:l This returns a list of bitmaps, such as: (10000000000000000b;00000100000000000b) What it's doing is taking all of the urls and indexing it by the where (&) address of each x (parameter to function: user id) is equal to the array of all of the users. That is, x=d.u returned: 01000001000000000000000000b and &x=d.u returned 1 7 (because the second and eighth element of that bitmap are ones). d.f[1 7] returns the second and eighth fragment. Then, she &~|/ which ors- the bitmaps (|/) together, finds their complement (~), and takes the where of (&) them, so in this case 0 5. l[0 5] then returns the strings in the master list. Note this is the list of all of the url fragments owned by other users: l[&~|/d.f[&x=d.u]~/:\:l] In PHP you might try to write: $mine = array(); $theirs = array(); foreach($db as $row) { if($row->u === $x) $mine[$row->f]=0; else $thiers[$row->f]=0; } $list = array_diff(array_keys($thiers),array_keys($mine)); but there's a lot of syntax here, and a number of PHP function calls that a beginner might not be aware of. She's just using this: http://www.timestored.com/kdb-guides/files/kdb-cheat-sheet.pdf So then on to y ss/: which basically does a `strpos()` of each of our urls against the list we're scanning (i.e. the thing that looks like "http://url,http://otherurl" http://url,http://otherurl". She then counts each (#:') result, takes the max again (|/). This means if ANY of those match, return `1b` otherwise return `0b`.

doublerebel 12y ago

Fantastic, I've been really interested to see a nonfinancial application of k that I could compare to implementations in languages I'm familiar with. If anyone else knows of other such k applications please share them!

RodgerTheGreat 12y ago
I was just tinkering with a tic tac toe game: https://github.com/JohnEarnest/ok/blob/gh-pages/examples/tictac.k https://github.com/JohnEarnest/ok/blob/gh-pages/examples/tic... I think I can still make a number of simplifications and generalizations, and I'd like to have a stab at an alpha-beta AI opponent. There's also a great deal of interesting code at No Stinking Loops: http://www.nsl.com http://www.nsl.com
srpeck 12y ago
Have a look at Arthur's text editor in k5: http://kparc.com/edit.k http://kparc.com/edit.k And of course the impending kOS operating system: http://kparc.com/ http://kparc.com/ with HN discussion: https://news.ycombinator.com/item?id=8475809 https://news.ycombinator.com/item?id=8475809 There are also some good links here, though you may have to dig: http://www.reddit.com/r/apljk/ http://www.reddit.com/r/apljk/

geocar 12y ago

Looks really good! F:"\n"/:0::;C:,[;"\n"];I:`:index.txt;f:-<I;h:C F I;H::F`:chat.htm .z.ws:{s:(" "/:($"du"$.z.Z),,x);{x y}\:[-!.z.W;s];h,::C s;f s} .z.po:{(-.z.w)h};.z.ph:{H} 1. "\n"/:0:: occurs twice and can be made a function (F) 2. You can project the `y` argument by itself (C) 3. Shorten symbols (I) 4. No need to (-f) all over the place when -< will do. (f) 5. You can load chat.htm once (H; .z.ph), but unfortunately K4 won't notice changes to the file, so am a bit divided about this one. 6. Save brackets! Join-enlist (,,), negate til (-!), and amend (,::) (.z.ws)

srpeck 12y ago
Thanks for the feedback; I pushed your cleaned up version to Github (https://github.com/srpeck/kchat/commit/5669fcca67fa609a9128b0ba65e34aea7adf5486 https://github.com/srpeck/kchat/commit/5669fcca67fa609a9128b...). As for your comment in #5, I noticed that I can hot load new versions of the server and client without impacting the existing clients' web socket connections. A production change to chat.htm only requires me to \l chat.k; any existing connections still have the older client running without interruption until they refresh, but all new connections get the new client. However, I am not all that concerned about hot loading new versions, as this is only deployed for a small team on the intranet.
- geocar 12y ago
  Indeed! I sometimes use emacs and use an after-save-hook to send a command (e.g. to reload a file).

anonu 12y ago

ive been using kdb+/q for over 5 years now - in a financial setting. I'd be happy to see the language go more "mainstream". There are definitely a lot of complaints about the terseness. There's no doubt, this adds to the steepness of the learning curve. But once you "see the light" you start realizing that you can do the same thing you do in perl or python in just 1 line in KDB. It makes more sense.

kbody 12y ago
I'm working on a algo-trading system and thinking of moving my tick data to kdb+, however I see no price on their site, care to enlight? Also, do you know what are the limits of the free evaluation/32-bit version? Thanks
- beagle3 12y ago
  32 bit version is 32 bit (databases limited to about 2GB per process, as are memory mappings). If that's good for you, use it. The 64-bit was, last I inquired (8 years ago or so) was in the "if you have to ask you can't afford it" oracle territory.