4 ms·

I briefly worked with K and Q while doing research on high-level numerical computing. I found their claimed efficiency to be severely exaggerated. K is a very n

I briefly worked with K and Q while doing research on high-level numerical computing. I found their claimed efficiency to be severely exaggerated. K is a very naively implemented array-oriented language and I found it to be slower than Matlab or NumPy for many tasks.

tlack 11y ago
You should consider writing up something with examples. Given how it's implemented (unboxed directly typed mmap'd arrays) it's hard to see how much slack there could be in it. I and many others are beginning to experiment with Q and Kdb and your learnings might spark valuable debate - and perhaps saved time. :)
- iskander 11y ago
  I wouldn't voluntarily touch K/Q ever again, but if someone else is interested in doing a blog post: try implementing any iterative machine learning algorithms. The separate compilation of primitive operators requires creating many array temporaries that on the one hand, Matlab's JIT can fuse and, on the other hand, NumPy provides a richer set of compiled functions to work with. K/Q lets you (tersely) express complex array computations using just its core operators, but all those array constructions add up to comparatively bad performance.
  - beagle3 11y ago
    Do you remember which versions and wordlength you use of each software, in case I get a chance to do comparisons?
    - iskander 11y ago
      It was ~4 years ago on a quad core xeon with ~8GB of memory, using whatever was the most recent version of Q, Python/Numpy and Matlab at the time.
beagle3 11y ago
K is indeed "naively" implemented. In my experience, its magic is having no magic - everything is straightforwardly simple, the entire interpreter core fits inside the L1 cache, every single operator is doing sequential read/writes and (essentially) blasting through memory at the highest speed. I've used Matlab ages ago (before they used to JIT), so I can't compare, but I did have a chance to compare K and "straight" Numpy, and for my uses K won by a nontrivial margin (30-40% IIRC) despite using essentially the same solution, and Numpy having comparable implementation -- I guess that's where the cache and memory speed effects come in. I did venture into using numexpr, which got Numpy a little faster than K, and I would assume Numba and PyPy (neither of which was in existence or usable at the time) might have also helped -- but plain Numpy vs. K was a clear win for K. It obviously depends on the workload - but I specifically commented about Gorilla - which is a write-a-lot, read-a-little kind of workload, in which K excels and Numpy (as I remember it from 6 years ago) not so much.

tlack 11y ago

You should consider writing up something with examples. Given how it's implemented (unboxed directly typed mmap'd arrays) it's hard to see how much slack there could be in it. I and many others are beginning to experiment with Q and Kdb and your learnings might spark valuable debate - and perhaps saved time. :)

iskander 11y ago
I wouldn't voluntarily touch K/Q ever again, but if someone else is interested in doing a blog post: try implementing any iterative machine learning algorithms. The separate compilation of primitive operators requires creating many array temporaries that on the one hand, Matlab's JIT can fuse and, on the other hand, NumPy provides a richer set of compiled functions to work with. K/Q lets you (tersely) express complex array computations using just its core operators, but all those array constructions add up to comparatively bad performance.
- beagle3 11y ago
  Do you remember which versions and wordlength you use of each software, in case I get a chance to do comparisons?
  - iskander 11y ago
    It was ~4 years ago on a quad core xeon with ~8GB of memory, using whatever was the most recent version of Q, Python/Numpy and Matlab at the time.

beagle3 11y ago

K is indeed "naively" implemented. In my experience, its magic is having no magic - everything is straightforwardly simple, the entire interpreter core fits inside the L1 cache, every single operator is doing sequential read/writes and (essentially) blasting through memory at the highest speed. I've used Matlab ages ago (before they used to JIT), so I can't compare, but I did have a chance to compare K and "straight" Numpy, and for my uses K won by a nontrivial margin (30-40% IIRC) despite using essentially the same solution, and Numpy having comparable implementation -- I guess that's where the cache and memory speed effects come in. I did venture into using numexpr, which got Numpy a little faster than K, and I would assume Numba and PyPy (neither of which was in existence or usable at the time) might have also helped -- but plain Numpy vs. K was a clear win for K. It obviously depends on the workload - but I specifically commented about Gorilla - which is a write-a-lot, read-a-little kind of workload, in which K excels and Numpy (as I remember it from 6 years ago) not so much.