14 ms·

K Language (2011)

178 points by lelf 7y ago

kick 7y ago
The RPN calculator example makes me really happy. As for the content of this page, there's a line within that now applies to itself: Introductory articles – somewhat dated but nevertheless rather informative: k4, what it lists as the most recent version, is about 20 years old now, and it talks about it like it's new. Still informative, though! EDIT: Giving it another look, it looks like the links have been updated over time (for example, the reference to oK), though the main content of the post is about ten years old and the initial list of links is over ten: https://web.archive.org/web/20110927003800/http://www.math.bas.bg/bantchev/place/k.html https://web.archive.org/web/20110927003800/http://www.math.b...
invalidOrTaken 7y ago
A story, since in retrospect I think it's worth telling. Some years ago I was at the SF Clojure meetup. Arthur Whitney's daughter worked at the sponsoring company, and he agreed to come tell us about K. In retrospect, I don't think we gave him the welcome he deserved. No one was rude or anything, but it seemed there was a disconnect: Arthur was keen to show off how fast K (and Kdb) was, over zillions and zillions of rows. But the thing that the Clojurists were all searching for (that got them into Clojure in the first place) was expressivity. Is Clojure fast? I don't know, generally the problems I face come down to avoiding balls of mud rather than performance bottlenecks. And I think that was true of most there. So Arthur got something of an underwhelming reception. I remember someone asking "Does K have the ability to self-modify, a la Lisp macros?" When Arthur said no, you could see most people in the room just sort of mentally shrug and move on. And this was too bad. Because recently I've been playing around with J (another APL descendant) and been very impressed by some expressivity/readability benefits. Some small things that have very big effects on the codebase you actually end up with. The first thing is the avoidance of abstraction. To use a Twitterism: Broke: Just write your code and don't divide it into functions, creating one long main method Woke: Divide your code up, naming parts that get reused Bespoke: If your code is made up of really really short things, it ends up being shorter than the names you would use, so you can just write the thing rather than your name for it. An analogy would be: there is no human-comprehensible way to communicate the idea of "picosecond" in less time than an actual picosecond. The other thing I didn't expect was the benefit of multiple dispatch being baked into e v e r y t h i n g. In Clojure I might write (map + a b) to add each index together; in J I could just write a+b. This is neat stuff! Best practices for keeping complexity down in APL's tend to be the opposite of what they are in other languages. Aaron Hsu gave a talk about this: https://www.youtube.com/watch?v=v7Mt0GYHU9A https://www.youtube.com/watch?v=v7Mt0GYHU9A It's too bad! Arthur came to tell us about speed---there's a reason it's used on giant datasets in finance, where performance translates directly into cash---but I wish we'd had the presence of mind to ask more about experience of writing K. So, Arthur, if you're reading this: Sorry everyone seemed kinda bored in SF a few years ago when you kindly came to present. We missed out!
- kazinator 7y ago
  > so you can just write the thing rather than your name for it I do not find this convincing. Names, especially good names, can convey additional meaning, which is helpful even if the names are longer than what they stand for in terms of raw character count. One obvious example is that I would often rather see a seven-letter manifest constant than the two-digit decimal literal it represents. In a similar vein, I would rather have a structure with named fields, than array indices or offsets. employee.department is objectively better than e[13]; a compiler can generate the equivalent of the latter from the former. In calculations that have many nodes in the syntax tree, it's helpful to break up the tree and give intermediate values names, which remains true even if the names are longer than the intermediate calculations. A name may longer than what it denotes in raw character count, but in spite of that it may less complex due to its flat structure which only indicates "same or not". If a 13 character identifier stands for some 9 character syntax that denotes a tree with 7 nodes, the identifier is certainly simpler. Lastly, different expressions that occur in a program can sometimes be very similar to each other. If we do not use names, then these differences can be hard to find. It might seem that identifiers which are longer than what they denote make the problem worse, but that is not necessarily so. Identifiers which are clearly different from each other can be selected for denoting forms that are very similar to each other. If instead of ((<]):) and ((<[):) I write foolyfoo and barlybar, that will be less error prone.
  - alimw 7y ago
    You might be right that good code needs names. But since in a terse language like J or K it may take longer to think up a good name than to write the code, it's handy that one doesn't have to have names for everything when prototyping.
    - rgoulter 7y ago
      What's the cost of reading it? Just use comments to roughly convey what a line of code does, and then for more precise understanding, take the time to read through it?
  - philosopher1234 7y ago
aasasd 7y ago
Not sure I actually understand why K is pushed here almost daily. Is it just the esoterics? Alright, APL and K are expressive for array computation. Do many people here deal with array computation? K is also supposedly fast—not once have I seen an explanation of what makes it fast and why those techniques aren't looted for open-source languages. I'm told that the interpreter is so small that it fits in the cache. Yeah, so how much time does a non-K program normally spend loading the code? How does this free K from loading and writing data? Is K in fact a thin layer on top of assembler (specifically SIMD and other extensions), for number crunching in the vein of shaders/CUDA?
- Avshalom 7y ago
  >> Is it just the esoterics? Kinda, yeah. It's more interesting--along some axes--than 'how we went from 100 customers to 101 customers' (which some how required a re-write in Java?) But also I feel like a lot of people look at their C/Python/SQL/JS/HTML/CSS stacks and their thousand person company and hundred thousand lines of code that kind a sorta just directs people at a Stripe widget and think: We Clearly Are Doing Something Wrong. Not that K/APL or Prolog or Smalltalk or whatever would actually magically make people do-the-right-thing or do-a-thing-better but they're sort of different worlds so they inspire a bit more wonder and reflection than 'how we moved from python to go to rust'
- iateanapple 7y ago
  > Not sure I actually understand why K is pushed here almost daily. Is it just the esoterics? Kind of - they have best practices that are almost the opposite of the standard software dev best practices - and they still produce high quality maintainable software. It’s interesting to see that a lot of “truths” in software development are at the very least context dependent if not outright pseudoscience. I don’t think you can simply “loot” their best practices - although some devs do write c code the way they would write k code. Edit: To expand on this for example even if you want to use short identifiers, multiple statements on one line, and avoid abstraction in the Java world you will be fighting the entire ecosystem to do it.
  - userbinator 7y ago
aparashk 7y ago
Good to see a link from the Bulgarian Academy of Sciences! I did my Masters thesis there.
aw1621107 7y ago
I posted something similar to this on the last k-related thread that came up, but new responses and perspectives are always nice: Lots of the discussion about the disadvantages of k/q/other APLs seem to revolve around its syntax and mental model. If said barriers were removed (e.g., if the relative obscurity of the syntax and semantics were not an issue and you could assume that you/your team/anyone who looks at your code is familiar), when would I not want to use k/q/APL/etc.? GC'd languages may not be good for latency/memory-sensitive programs. C/C++ may not be good for reliability/security-sensitive programs. Ruby/Python/Javascript may have performance issues. Haskell may have issues around predicting resource usage. Lisp may have issues around being too expressive. What issues may I run into that would make k/q/etc. not a good choice?
- beagle3 7y ago
  “Too expressive?” All APl variants I know are GC’s so what you wrote applies (though refcounted and cycle free so deterministic, and well defined resource consumption) They are mostly functional, so as usual care must be taken with memory usage when updating data structures. If your workload does not lend itself to APLization, your program will not enjoy any of the benefits, of course. But in my experience most projects have very significant parts that do.
  - aw1621107 7y ago
    "Too expressive" is suboptimal wording on my part; the intent was to cover the criticism that Lisp is flexible enough that Lisp programs tend to evolve their own potentially programmer-specific DSLs for the given problem domain. I guess a potentially more interesting question would be how I could identify workloads that APL-alikes would or would not work well for. In particular, are there resources where I can learn more about ways to translate problems to a format that lends themselves to APLization? One example is the substring search problem; I know about naive string search using loops, Bayer-Moore, etc., but I don't think I would have been able to come up with the APLish version of match-char-of-needle-in-haystack, rotate, and AND. Is this just something that gets picked up as one does more work with APL-alikes?
aw1621107 7y ago
Are there any papers or such that have any details about what allows k/q to be so fast? Or would those be considered implementation secrets? It's always nice to be able to read about the technology and design underpinning something, but in this case at least it'd be understandable if that were not available.
ngcc_hk 7y ago
Given such small footprint can it run in microprocessor level or a bit larger. Or fpga implementations?
- tluyben2 7y ago
  At Shakti they are working on that; there are quite a lot experimentations with embedded boards in the (small) community of people who work on the core. They might provide ARM compiles (they already work) so it should work on quite a range of devices; however you do need somewhat more memory than for instance I am used to work with (MCU's with 23 kb usable memory); I think it needs at least a few 100kb to do anything and (probably) a few meg to do actual work with it. It'll improve.
alexeiz 7y ago
K/KDB is a pile of shite. It's loved by quants and managers in financial firms. But it's hated by programmers who actually have to write code in it and deal with shitty code written by quants. K is full of arbitrary restrictions (two many global/local variables - what is it, 1970s?) and crazy quirks (function closing brace has to indented, otherwise you'll get a weird error). All in the name of performance, of course. The error reporting is insanely cryptic. It doesn't tell you what caused an error or even where it happened. It just tells you something like "'rank" - and that's it. Once your code grows beyond 10 lines, you can kiss your sanity goodbye. And let's talk about performance. K is inherently single threaded. If you have a powerful 96-core machine, it's of no use to you because your K code will be using only a single core. I've seen numpy easily beating K when numpy uses MKL library with automatic parallelization. There is a reason K/KDB is not very popular outside of a certain domain. It's not because it's proprietary and it costs a fortune. It's just bad.
- smabie 7y ago
  > K is full of arbitrary restrictions The restrictions, lack of lexical scoping, and weird error messages are indeed frustrating, especially when starting out. > And let's talk about performance. K is inherently single threaded. If you have a powerful 96-core machine, it's of no use to you because your K code will be using only a single core. I've seen numpy easily beating K when numpy uses MKL library with automatic parallelization. I'm not sure where you got that from, but it's not true. You can use peach to quite great affect. For example, to do a matrix multiply that'll utilize all slave threads (which you can set at runtime): mmu[;x] peach y For more info, check out https://code.kx.com/q/basics/peach/ https://code.kx.com/q/basics/peach/. And while the parallelization isn't automatic, it's pretty damn easy to just s/each/peach/g (you should profile first, though, of course). Moreover, you can just run multiple interpreters and communicate through IPC. It works really well and is pretty easy to do. Give me some numerical python code and I guarantee k/q can do it at least as fast. If your pipeline is something like do some number crunching and save a bunch of rows to MySQL, kdb+/q is always going to be faster, hands down. The performance benefits of having the db and language in one process cannot be underestimated. > There is a reason K/KDB is not very popular outside of a certain domain. It's not because it's proprietary and it costs a fortune. It's just bad. I strongly disagree. If it was up to me, I would throw out all the crap numerical Python code I write everyday and switch to kdb+/q. kdb+ is a pleasure to work with and the query language is so much better than SQL it's not even funny. The only real problem with kdb+/q is that there's not many libraries for it, in particular, neural net and ML libs would be on my wishlist. Though, I'm sure if you're a paying customer First Derivative will whip up some super high quality shit for you. Also, you're argument doesn't really make sense: if k just sucks, why is it used at all? Because quants and their managers are stupid?
smabie 7y ago
I love K/q, but there's some trade-offs (probably for speed) that I find extremely aggravating. The biggest sin of all is the lack of lexical scoping. If k/q had lexical scoping and macros, it would probably be my ideal programming language. Regardless, the philosophy behind the language is so different, and so incredible, I had the most fun I've had in a long time programming when I learned it. kdb+/q is a work of art and probably the best engineered piece of software I've ever come across.
- beagle3 7y ago
  All Ks have basic module level lexical scoping, which does not deliver all the benefits of lexical scoping, but has none of the pitfalls of dynamic scoping either. K2/K3 did have (a version of nested) lexical scoping - an inner function would close over the values at the time definition was encountered ; you can still simulate this in later Ks by passing what you need to close over and immediately projecting over existing values, e.g. make_adder:{[n] {[n,x] n+x}[n;]} Which defines an inner two-parameter function, and then projects it over the outer n parameter (this is how K2 implemented it IIUC). But that doesn't let you modify names in an upper scope, so is not really comparable to Lisp's lexical scoping or Lua's upvars and Python's nonlocals or whatever they are called these days. Andrey Zholos' kuc[0] dialect has a JIT and real closures, you might find that interesting, though it hasn't seen an update in a while. > probably the best engineered piece of software I've ever come across. As you can probably tell, I'm a K fan and evangelist myself. Yet personally I cringe at some of the kludges. I love K. It's awesome, it's amazing, but I wouldn't use the "best engineered" label myself; "Best set of tradeoffs" would probably be my description (which is a distinction without difference to some, I recognize - those who define engineering as the art and craft of the tradeoff). [0] http://althenia.net/kuc http://althenia.net/kuc
leggomylibro 7y ago
These languages come up here every now and then; they look very interesting, but it also seems like they must be very heavily optimized to be as fast as they are. So, sorry if this is a dumb question, but does K run on architectures other than x86/64? I checked https://kx.com/ https://kx.com/ but I couldn't find a definite answer. They mention IoT a lot, which makes me think it might run on embedded devices too?
- SifJar 7y ago
  kx also have a version for ARM (available here https://kx.com/download https://kx.com/download ). Note this is the non-commercial version, not sure if they have a commercial version for ARM. Their frequent mention of IoT is (I think) more to do with storing data from IoT devices, not directly on the devices themselves. Shakti (Arthur Whitney's current company) has also mentioned a wider support of architectures I think, but at the moment their k (k9) is in pretty early stages, still x86/64 only I think. There are various other open source implementations of k (posted elsewhere on this HN thread), some of them might also support other architectures (although they won't have the performance of Kx/Shakti versions).
  - leggomylibro 7y ago
    Ah, that makes sense - thanks. I guess that's just for Cortex-A ARM cores? I was hoping to try a lean language like this on a fast micro like the new Teensy boards, but I suppose those kinds of chips probably aren't used very much in their target applications like HFT. I do wish that people would stop using "IoT" to mean "sending tons of data to the cloud", but I guess that's how the buzzword crumbles.
- kick 7y ago
  It's not really that heavily optimized. It's just really small.
  - aw1621107 7y ago
    That's really interesting! What allows k to be so small compared to interpreters for other languages? Does it come down to the semantics of the language itself?
deleted 7y ago
[deleted]
lokedhs 7y ago
I'm currently working on my own APL dialect. Like the author of K, I decided to not be bound to the restrictions of APL, and want it to be more functional. As I am also doing a lot of Lisp, a lot of ideas come from there as well. The biggest difference in my language (which as of yet does not have a name) is that it is immutable, and function results are lazy. This means that it can be highly parallel, and the laziness means that there is much less synchronisation needed when parallelising computations. I'm also making some of the syntax more traditional, in order to make large programs more manageable. It's implemented as a Kotlin multiplatform project, which means that I can build the interpreter for the JVM, native and Javascript. Why am I talking about this? Well, it's because my biggest issue with APL is that it's not really suited for large programs (where large means on the order of 100 lines of code or more). Dyalog provides some extensions that make imperative programming easier, but the syntax is really ugly. Thus, I'm trying to merge the idea of concise APL syntax together with typical functional programming. So you'll be able to write code like this: if (foo = bar) { a←⍉b } else { print "debug message" a←x∘.+y } Even though the core syntax is still APL, the structure of the program is much easier to understand, compared to the way you'd write it in APL. I still don't know if experiment will actually lead to something good, but at least it's a fun project where I can play with Kotlin multiplatform support. The code is on my Github, but I'm not linking it here since it's quite early, and it's not really usable yet (there are a lot of functions missing). I'm expecting anyone who actually looks it up anyway to understand that it's not ready yet.
- Athas 7y ago
  > The biggest difference in my language (which as of yet does not have a name) is that it is immutable, and function results are lazy. This means that it can be highly parallel, and the laziness means that there is much less synchronisation needed when parallelising computations. In my experience, it is the other way around. Laziness when implemented as normally is essentially state plus control flow, which is death to parallelism. Parallel Haskell is obscure for this reason, and typical programming patterns use some form of "deepseq" construct whenever thread boundaries are crossed (explicitly or implicitly). Actually taking advantage of laziness in a parallel setting is very tricky. Offhand, I cannot recall anyone who has done so convincingly.
deleted 7y ago
[deleted]
Russelfuture 7y ago
This is an interesting thread - thx for posting. I used to know a bunch of folks that used both J and K. I still use APL - the old APLWin 3.5 from APL2000 (Manugistics/STSC). APL was the first language I learned as a tiny child. I write my pseudo code in it. I live in a ranch house out on the tundra, with a 50ft transponder antenna behind the place which links me to the 'net. I've used C, c++, Python with all the numpy stuff, etc and a few flavours of TensorFlow. But it's APL that gets it done. I've heard of K, and I knew a guy who worked for Arthur Whitney. Languages like K and J and APL can be fast, if you use them right. And it is almost tragic-comical, that the genius of the APL glyphic character set was given up, just as wonderful devices were created which could paint any character at all on the screen. My girlfriend uses Japanese Kangi and Hirogana on her iPad, no problem. In a GUI environment, the original APL characters are easy and clear - no need for silly ASCII overloading. See, I have these simple GUI apps, written many years ago in Windows, using APLWin 3.5. They still run, and I use them daily. They keep track of financial time series data, and I can use them in a variety of ways. The APL stuff can convert tables of various formats into conformable vectors, which can be transformed into input for GNUplot or TensorFlow or even (my favourite), Hinton's original Xerion product, which I have running on some Linux boxes. APL is simply a wonderful, easy, useful language that can allow one single person to support some mission-critical production-grade software. Example: This AM, I discovered that all the data tables that my data-table downloader had got last nite, had a new format, with a zero-row of data, and today's date. This meant when I updated the database of time series data, I got tables where every series had a bogus, zero-valued most recent date. It was 7:30 AM, and markets would open at 9:30. A bit of code inspection, and a simple fix: Update the MERGETABS function with two lines of code: ZEROROWS <- 0 = +/PNEW[;(1 downarrow iota 1 downarrow rho PNEW)] PNEW <- ( tilde ZEROROWS)/[1] PNEW And this solved the problem. Update the runtime version of the workspace by changing QuadLX latent expression to start the GUI when the W/S is loaded, and roll out the new W/S to a few different boxes. All done before Mkt open. The code above just takes the new table PNEW, selects the data columns, finds those that are zero valued and removes them using a boolean vector reduction along the highest dimension (the table's rows). Easy peasy. Done quick, and done right. With my data updated and correct, I run some stuff against it, and get the courage to hold my positions - concluding it will be an up day, as the stuff I run helps me make that decision. I close the day up, just under $40,000 to the good. (Crazy day...) And oh, btw, the APL stuff runs on both Linux and Windows boxes - and the Linux is both 32-bit and modern CentOS 64-bit. I still run the old 32-bit boxes because they are very fast and very reliable. The APL stuff runs on the Linux boxes under WINE, which works amazingly well. It's old school stuff, but it just works. APL is the magic that lets me do the other magic, that pays the bills. I gather that K can provide something similar for big corporate environments, when the datasets and timeseries get crazy large. You can write anything in any language - but one must not lose sight of the objective, which is to make the business side of things work well. One can write APL - or any language - in a way that is unclear and confusing. But you don't have to do so. If you are careful, you can build APL solutions that are fast, easy to maintain, and smoothly flexible. Maybe this is possible in K as well?

K Language (2011)

178 points by lelf 7y ago

kick 7y ago

The RPN calculator example makes me really happy. As for the content of this page, there's a line within that now applies to itself: Introductory articles – somewhat dated but nevertheless rather informative: k4, what it lists as the most recent version, is about 20 years old now, and it talks about it like it's new. Still informative, though! EDIT: Giving it another look, it looks like the links have been updated over time (for example, the reference to oK), though the main content of the post is about ten years old and the initial list of links is over ten: https://web.archive.org/web/20110927003800/http://www.math.bas.bg/bantchev/place/k.html https://web.archive.org/web/20110927003800/http://www.math.b...

invalidOrTaken 7y ago

A story, since in retrospect I think it's worth telling. Some years ago I was at the SF Clojure meetup. Arthur Whitney's daughter worked at the sponsoring company, and he agreed to come tell us about K. In retrospect, I don't think we gave him the welcome he deserved. No one was rude or anything, but it seemed there was a disconnect: Arthur was keen to show off how fast K (and Kdb) was, over zillions and zillions of rows. But the thing that the Clojurists were all searching for (that got them into Clojure in the first place) was expressivity. Is Clojure fast? I don't know, generally the problems I face come down to avoiding balls of mud rather than performance bottlenecks. And I think that was true of most there. So Arthur got something of an underwhelming reception. I remember someone asking "Does K have the ability to self-modify, a la Lisp macros?" When Arthur said no, you could see most people in the room just sort of mentally shrug and move on. And this was too bad. Because recently I've been playing around with J (another APL descendant) and been very impressed by some expressivity/readability benefits. Some small things that have very big effects on the codebase you actually end up with. The first thing is the avoidance of abstraction. To use a Twitterism: Broke: Just write your code and don't divide it into functions, creating one long main method Woke: Divide your code up, naming parts that get reused Bespoke: If your code is made up of really really short things, it ends up being shorter than the names you would use, so you can just write the thing rather than your name for it. An analogy would be: there is no human-comprehensible way to communicate the idea of "picosecond" in less time than an actual picosecond. The other thing I didn't expect was the benefit of multiple dispatch being baked into e v e r y t h i n g. In Clojure I might write (map + a b) to add each index together; in J I could just write a+b. This is neat stuff! Best practices for keeping complexity down in APL's tend to be the opposite of what they are in other languages. Aaron Hsu gave a talk about this: https://www.youtube.com/watch?v=v7Mt0GYHU9A https://www.youtube.com/watch?v=v7Mt0GYHU9A It's too bad! Arthur came to tell us about speed---there's a reason it's used on giant datasets in finance, where performance translates directly into cash---but I wish we'd had the presence of mind to ask more about experience of writing K. So, Arthur, if you're reading this: Sorry everyone seemed kinda bored in SF a few years ago when you kindly came to present. We missed out!

kazinator 7y ago
> so you can just write the thing rather than your name for it I do not find this convincing. Names, especially good names, can convey additional meaning, which is helpful even if the names are longer than what they stand for in terms of raw character count. One obvious example is that I would often rather see a seven-letter manifest constant than the two-digit decimal literal it represents. In a similar vein, I would rather have a structure with named fields, than array indices or offsets. employee.department is objectively better than e[13]; a compiler can generate the equivalent of the latter from the former. In calculations that have many nodes in the syntax tree, it's helpful to break up the tree and give intermediate values names, which remains true even if the names are longer than the intermediate calculations. A name may longer than what it denotes in raw character count, but in spite of that it may less complex due to its flat structure which only indicates "same or not". If a 13 character identifier stands for some 9 character syntax that denotes a tree with 7 nodes, the identifier is certainly simpler. Lastly, different expressions that occur in a program can sometimes be very similar to each other. If we do not use names, then these differences can be hard to find. It might seem that identifiers which are longer than what they denote make the problem worse, but that is not necessarily so. Identifiers which are clearly different from each other can be selected for denoting forms that are very similar to each other. If instead of ((<]):) and ((<[):) I write foolyfoo and barlybar, that will be less error prone.
- alimw 7y ago
  You might be right that good code needs names. But since in a terse language like J or K it may take longer to think up a good name than to write the code, it's handy that one doesn't have to have names for everything when prototyping.
  - rgoulter 7y ago
    What's the cost of reading it? Just use comments to roughly convey what a line of code does, and then for more precise understanding, take the time to read through it?
- philosopher1234 7y ago

aasasd 7y ago

Not sure I actually understand why K is pushed here almost daily. Is it just the esoterics? Alright, APL and K are expressive for array computation. Do many people here deal with array computation? K is also supposedly fast—not once have I seen an explanation of what makes it fast and why those techniques aren't looted for open-source languages. I'm told that the interpreter is so small that it fits in the cache. Yeah, so how much time does a non-K program normally spend loading the code? How does this free K from loading and writing data? Is K in fact a thin layer on top of assembler (specifically SIMD and other extensions), for number crunching in the vein of shaders/CUDA?

Avshalom 7y ago
>> Is it just the esoterics? Kinda, yeah. It's more interesting--along some axes--than 'how we went from 100 customers to 101 customers' (which some how required a re-write in Java?) But also I feel like a lot of people look at their C/Python/SQL/JS/HTML/CSS stacks and their thousand person company and hundred thousand lines of code that kind a sorta just directs people at a Stripe widget and think: We Clearly Are Doing Something Wrong. Not that K/APL or Prolog or Smalltalk or whatever would actually magically make people do-the-right-thing or do-a-thing-better but they're sort of different worlds so they inspire a bit more wonder and reflection than 'how we moved from python to go to rust'
iateanapple 7y ago
> Not sure I actually understand why K is pushed here almost daily. Is it just the esoterics? Kind of - they have best practices that are almost the opposite of the standard software dev best practices - and they still produce high quality maintainable software. It’s interesting to see that a lot of “truths” in software development are at the very least context dependent if not outright pseudoscience. I don’t think you can simply “loot” their best practices - although some devs do write c code the way they would write k code. Edit: To expand on this for example even if you want to use short identifiers, multiple statements on one line, and avoid abstraction in the Java world you will be fighting the entire ecosystem to do it.
- userbinator 7y ago

aparashk 7y ago

Good to see a link from the Bulgarian Academy of Sciences! I did my Masters thesis there.

aw1621107 7y ago

I posted something similar to this on the last k-related thread that came up, but new responses and perspectives are always nice: Lots of the discussion about the disadvantages of k/q/other APLs seem to revolve around its syntax and mental model. If said barriers were removed (e.g., if the relative obscurity of the syntax and semantics were not an issue and you could assume that you/your team/anyone who looks at your code is familiar), when would I not want to use k/q/APL/etc.? GC'd languages may not be good for latency/memory-sensitive programs. C/C++ may not be good for reliability/security-sensitive programs. Ruby/Python/Javascript may have performance issues. Haskell may have issues around predicting resource usage. Lisp may have issues around being too expressive. What issues may I run into that would make k/q/etc. not a good choice?

beagle3 7y ago
“Too expressive?” All APl variants I know are GC’s so what you wrote applies (though refcounted and cycle free so deterministic, and well defined resource consumption) They are mostly functional, so as usual care must be taken with memory usage when updating data structures. If your workload does not lend itself to APLization, your program will not enjoy any of the benefits, of course. But in my experience most projects have very significant parts that do.
- aw1621107 7y ago
  "Too expressive" is suboptimal wording on my part; the intent was to cover the criticism that Lisp is flexible enough that Lisp programs tend to evolve their own potentially programmer-specific DSLs for the given problem domain. I guess a potentially more interesting question would be how I could identify workloads that APL-alikes would or would not work well for. In particular, are there resources where I can learn more about ways to translate problems to a format that lends themselves to APLization? One example is the substring search problem; I know about naive string search using loops, Bayer-Moore, etc., but I don't think I would have been able to come up with the APLish version of match-char-of-needle-in-haystack, rotate, and AND. Is this just something that gets picked up as one does more work with APL-alikes?

aw1621107 7y ago

Are there any papers or such that have any details about what allows k/q to be so fast? Or would those be considered implementation secrets? It's always nice to be able to read about the technology and design underpinning something, but in this case at least it'd be understandable if that were not available.

ngcc_hk 7y ago

Given such small footprint can it run in microprocessor level or a bit larger. Or fpga implementations?

tluyben2 7y ago
At Shakti they are working on that; there are quite a lot experimentations with embedded boards in the (small) community of people who work on the core. They might provide ARM compiles (they already work) so it should work on quite a range of devices; however you do need somewhat more memory than for instance I am used to work with (MCU's with 23 kb usable memory); I think it needs at least a few 100kb to do anything and (probably) a few meg to do actual work with it. It'll improve.

alexeiz 7y ago

K/KDB is a pile of shite. It's loved by quants and managers in financial firms. But it's hated by programmers who actually have to write code in it and deal with shitty code written by quants. K is full of arbitrary restrictions (two many global/local variables - what is it, 1970s?) and crazy quirks (function closing brace has to indented, otherwise you'll get a weird error). All in the name of performance, of course. The error reporting is insanely cryptic. It doesn't tell you what caused an error or even where it happened. It just tells you something like "'rank" - and that's it. Once your code grows beyond 10 lines, you can kiss your sanity goodbye. And let's talk about performance. K is inherently single threaded. If you have a powerful 96-core machine, it's of no use to you because your K code will be using only a single core. I've seen numpy easily beating K when numpy uses MKL library with automatic parallelization. There is a reason K/KDB is not very popular outside of a certain domain. It's not because it's proprietary and it costs a fortune. It's just bad.

smabie 7y ago
> K is full of arbitrary restrictions The restrictions, lack of lexical scoping, and weird error messages are indeed frustrating, especially when starting out. > And let's talk about performance. K is inherently single threaded. If you have a powerful 96-core machine, it's of no use to you because your K code will be using only a single core. I've seen numpy easily beating K when numpy uses MKL library with automatic parallelization. I'm not sure where you got that from, but it's not true. You can use peach to quite great affect. For example, to do a matrix multiply that'll utilize all slave threads (which you can set at runtime): mmu[;x] peach y For more info, check out https://code.kx.com/q/basics/peach/ https://code.kx.com/q/basics/peach/. And while the parallelization isn't automatic, it's pretty damn easy to just s/each/peach/g (you should profile first, though, of course). Moreover, you can just run multiple interpreters and communicate through IPC. It works really well and is pretty easy to do. Give me some numerical python code and I guarantee k/q can do it at least as fast. If your pipeline is something like do some number crunching and save a bunch of rows to MySQL, kdb+/q is always going to be faster, hands down. The performance benefits of having the db and language in one process cannot be underestimated. > There is a reason K/KDB is not very popular outside of a certain domain. It's not because it's proprietary and it costs a fortune. It's just bad. I strongly disagree. If it was up to me, I would throw out all the crap numerical Python code I write everyday and switch to kdb+/q. kdb+ is a pleasure to work with and the query language is so much better than SQL it's not even funny. The only real problem with kdb+/q is that there's not many libraries for it, in particular, neural net and ML libs would be on my wishlist. Though, I'm sure if you're a paying customer First Derivative will whip up some super high quality shit for you. Also, you're argument doesn't really make sense: if k just sucks, why is it used at all? Because quants and their managers are stupid?

smabie 7y ago

I love K/q, but there's some trade-offs (probably for speed) that I find extremely aggravating. The biggest sin of all is the lack of lexical scoping. If k/q had lexical scoping and macros, it would probably be my ideal programming language. Regardless, the philosophy behind the language is so different, and so incredible, I had the most fun I've had in a long time programming when I learned it. kdb+/q is a work of art and probably the best engineered piece of software I've ever come across.

beagle3 7y ago
All Ks have basic module level lexical scoping, which does not deliver all the benefits of lexical scoping, but has none of the pitfalls of dynamic scoping either. K2/K3 did have (a version of nested) lexical scoping - an inner function would close over the values at the time definition was encountered ; you can still simulate this in later Ks by passing what you need to close over and immediately projecting over existing values, e.g. make_adder:{[n] {[n,x] n+x}[n;]} Which defines an inner two-parameter function, and then projects it over the outer n parameter (this is how K2 implemented it IIUC). But that doesn't let you modify names in an upper scope, so is not really comparable to Lisp's lexical scoping or Lua's upvars and Python's nonlocals or whatever they are called these days. Andrey Zholos' kuc[0] dialect has a JIT and real closures, you might find that interesting, though it hasn't seen an update in a while. > probably the best engineered piece of software I've ever come across. As you can probably tell, I'm a K fan and evangelist myself. Yet personally I cringe at some of the kludges. I love K. It's awesome, it's amazing, but I wouldn't use the "best engineered" label myself; "Best set of tradeoffs" would probably be my description (which is a distinction without difference to some, I recognize - those who define engineering as the art and craft of the tradeoff). [0] http://althenia.net/kuc http://althenia.net/kuc

leggomylibro 7y ago

These languages come up here every now and then; they look very interesting, but it also seems like they must be very heavily optimized to be as fast as they are. So, sorry if this is a dumb question, but does K run on architectures other than x86/64? I checked https://kx.com/ https://kx.com/ but I couldn't find a definite answer. They mention IoT a lot, which makes me think it might run on embedded devices too?

SifJar 7y ago
kx also have a version for ARM (available here https://kx.com/download https://kx.com/download ). Note this is the non-commercial version, not sure if they have a commercial version for ARM. Their frequent mention of IoT is (I think) more to do with storing data from IoT devices, not directly on the devices themselves. Shakti (Arthur Whitney's current company) has also mentioned a wider support of architectures I think, but at the moment their k (k9) is in pretty early stages, still x86/64 only I think. There are various other open source implementations of k (posted elsewhere on this HN thread), some of them might also support other architectures (although they won't have the performance of Kx/Shakti versions).
- leggomylibro 7y ago
  Ah, that makes sense - thanks. I guess that's just for Cortex-A ARM cores? I was hoping to try a lean language like this on a fast micro like the new Teensy boards, but I suppose those kinds of chips probably aren't used very much in their target applications like HFT. I do wish that people would stop using "IoT" to mean "sending tons of data to the cloud", but I guess that's how the buzzword crumbles.
kick 7y ago
It's not really that heavily optimized. It's just really small.
- aw1621107 7y ago
  That's really interesting! What allows k to be so small compared to interpreters for other languages? Does it come down to the semantics of the language itself?

deleted 7y ago

[deleted]

lokedhs 7y ago

I'm currently working on my own APL dialect. Like the author of K, I decided to not be bound to the restrictions of APL, and want it to be more functional. As I am also doing a lot of Lisp, a lot of ideas come from there as well. The biggest difference in my language (which as of yet does not have a name) is that it is immutable, and function results are lazy. This means that it can be highly parallel, and the laziness means that there is much less synchronisation needed when parallelising computations. I'm also making some of the syntax more traditional, in order to make large programs more manageable. It's implemented as a Kotlin multiplatform project, which means that I can build the interpreter for the JVM, native and Javascript. Why am I talking about this? Well, it's because my biggest issue with APL is that it's not really suited for large programs (where large means on the order of 100 lines of code or more). Dyalog provides some extensions that make imperative programming easier, but the syntax is really ugly. Thus, I'm trying to merge the idea of concise APL syntax together with typical functional programming. So you'll be able to write code like this: if (foo = bar) { a←⍉b } else { print "debug message" a←x∘.+y } Even though the core syntax is still APL, the structure of the program is much easier to understand, compared to the way you'd write it in APL. I still don't know if experiment will actually lead to something good, but at least it's a fun project where I can play with Kotlin multiplatform support. The code is on my Github, but I'm not linking it here since it's quite early, and it's not really usable yet (there are a lot of functions missing). I'm expecting anyone who actually looks it up anyway to understand that it's not ready yet.

Athas 7y ago
> The biggest difference in my language (which as of yet does not have a name) is that it is immutable, and function results are lazy. This means that it can be highly parallel, and the laziness means that there is much less synchronisation needed when parallelising computations. In my experience, it is the other way around. Laziness when implemented as normally is essentially state plus control flow, which is death to parallelism. Parallel Haskell is obscure for this reason, and typical programming patterns use some form of "deepseq" construct whenever thread boundaries are crossed (explicitly or implicitly). Actually taking advantage of laziness in a parallel setting is very tricky. Offhand, I cannot recall anyone who has done so convincingly.

deleted 7y ago

[deleted]

Russelfuture 7y ago

This is an interesting thread - thx for posting. I used to know a bunch of folks that used both J and K. I still use APL - the old APLWin 3.5 from APL2000 (Manugistics/STSC). APL was the first language I learned as a tiny child. I write my pseudo code in it. I live in a ranch house out on the tundra, with a 50ft transponder antenna behind the place which links me to the 'net. I've used C, c++, Python with all the numpy stuff, etc and a few flavours of TensorFlow. But it's APL that gets it done. I've heard of K, and I knew a guy who worked for Arthur Whitney. Languages like K and J and APL can be fast, if you use them right. And it is almost tragic-comical, that the genius of the APL glyphic character set was given up, just as wonderful devices were created which could paint any character at all on the screen. My girlfriend uses Japanese Kangi and Hirogana on her iPad, no problem. In a GUI environment, the original APL characters are easy and clear - no need for silly ASCII overloading. See, I have these simple GUI apps, written many years ago in Windows, using APLWin 3.5. They still run, and I use them daily. They keep track of financial time series data, and I can use them in a variety of ways. The APL stuff can convert tables of various formats into conformable vectors, which can be transformed into input for GNUplot or TensorFlow or even (my favourite), Hinton's original Xerion product, which I have running on some Linux boxes. APL is simply a wonderful, easy, useful language that can allow one single person to support some mission-critical production-grade software. Example: This AM, I discovered that all the data tables that my data-table downloader had got last nite, had a new format, with a zero-row of data, and today's date. This meant when I updated the database of time series data, I got tables where every series had a bogus, zero-valued most recent date. It was 7:30 AM, and markets would open at 9:30. A bit of code inspection, and a simple fix: Update the MERGETABS function with two lines of code: ZEROROWS <- 0 = +/PNEW[;(1 downarrow iota 1 downarrow rho PNEW)] PNEW <- ( tilde ZEROROWS)/[1] PNEW And this solved the problem. Update the runtime version of the workspace by changing QuadLX latent expression to start the GUI when the W/S is loaded, and roll out the new W/S to a few different boxes. All done before Mkt open. The code above just takes the new table PNEW, selects the data columns, finds those that are zero valued and removes them using a boolean vector reduction along the highest dimension (the table's rows). Easy peasy. Done quick, and done right. With my data updated and correct, I run some stuff against it, and get the courage to hold my positions - concluding it will be an up day, as the stuff I run helps me make that decision. I close the day up, just under $40,000 to the good. (Crazy day...) And oh, btw, the APL stuff runs on both Linux and Windows boxes - and the Linux is both 32-bit and modern CentOS 64-bit. I still run the old 32-bit boxes because they are very fast and very reliable. The APL stuff runs on the Linux boxes under WINE, which works amazingly well. It's old school stuff, but it just works. APL is the magic that lets me do the other magic, that pays the bills. I gather that K can provide something similar for big corporate environments, when the datasets and timeseries get crazy large. You can write anything in any language - but one must not lose sight of the objective, which is to make the business side of things work well. One can write APL - or any language - in a way that is unclear and confusing. But you don't have to do so. If you are careful, you can build APL solutions that are fast, easy to maintain, and smoothly flexible. Maybe this is possible in K as well?