3 ms·

> So the author is incredulous but provides no proof or stats, just a long rant. I thought the same thing but on the other hand, what can you do? If you're not

> So the author is incredulous but provides no proof or stats, just a long rant. I thought the same thing but on the other hand, what can you do? If you're not allowed to publish benchmarks you only can rant like this even if it sounds petty. By getting attention here on HN it helps put pressure on commercial vendors to stop sharing unconfirmable benchmarks otherwise they will lose credibility since they've been called out. The author does make some good points about the fundamental nature of interpreters too. Although it's hard to confirm there too what these languages do without seeing their source. I mean that if K or some language is actually interpreting without a virtual machine at some stage, that is indeed going to be slower than any language with code running on a virtual machine. And also that any code that runs on a virtual machine must be slower than an equivalently intelligent compiled code since it's running on another layer of abstraction that compiled code isn't. ("equivalently intelligent" because yes a virtual machine could be faster than terrible compiled code.)

sz4kerto 5y ago
"And also that any language with a virtual machine must be slower than an equivalently intelligent compiled code since it's running on another layer of abstraction that compiled code isn't. " I think this is not correct. A VM has access to the data that the code is operating on, while an AOT compiler doesn't. It's possible that a VM (eg. JVM) produces faster code than a normal compiler.
- icen 5y ago
  Sadly it also has to consider the time it takes to compile code like that - we grumble, but not too loudly, at the time LLVM takes to compile things. If the JVM took that same time whilst running live that wouldn't be acceptable.
  - WJW 5y ago
    Surely this depends entirely on the total runtime of the program. A server-style program which will run for a couple of weeks can accommodate quite a bit of JIT compilation overhead.
- eatonphil 5y ago
  I'm talking about where the code is run. If it's run on a VM it's run on a VM. If it's run on the cpu by JIT-ed code it's not running on a VM. Edit: But fair I did say "language with a VM" which would include Java JIT-ed or not. Edit, edit: Modified the original comment to say "code on a VM" not "language with a VM".
theelous3 5y ago
I'm not familiar with k, and I'm not understanding this whole "no benchmarks" thing. So it's a proprietary lang? What stops benchmarks being run? Just how ridiculous are the licenses? Why has seemingly nobody run them anyway?
- eatonphil 5y ago
  I think the article answers all your questions.
- tsimionescu 5y ago
  Not sure about K, but I know for example that MS didn't allow publishing performance benchmarks of .NET (not sure of this changed with .NET core).
  - eatonphil 5y ago
    Both Oracle and MS SQL disallow this without approval. https://stackoverflow.com/a/12116865/1507139 https://stackoverflow.com/a/12116865/1507139
- geocar 5y ago
  > I'm not understanding this whole "no benchmarks" thing. Back in the 1980s, there was a paper published[1] that compared a number of database systems, and Oracle felt this paper harmed them. Perhaps if the authors knew Oracle better, they could have made more favourable benchmarks, they would argue, and whilst perhaps DeWitt et al could agree to that, they may point out they had not intended to harm Oracle, and being researchers and not marketers were given no notice or expectation that such a review could harm Oracle. And so for the avoidance of doubt, Oracle, Microsoft, and many others (probably including KX, although I've never checked) insist in their licensing agreements that they be able to approve benchmarks, ostensibly so that they have an opportunity to correct any misunderstandings the author might have about the software before. [1]: http://pages.cs.wisc.edu/~dewitt/includes/benchmarking/vldb83.pdf http://pages.cs.wisc.edu/~dewitt/includes/benchmarking/vldb8...
  - 5y ago
dagw 5y ago
Here are some (micro) benchmarks of q vs python that they did publish: https://kx.com/blog/a-comparison-of-python-and-q-for-data-problem-solving/ https://kx.com/blog/a-comparison-of-python-and-q-for-data-pr... tl;dr, while q is orders of magnitude faster that naive python, it is, at best, 5 times faster than optimal python using numpy. Based on those numbers I would be very surprised if an optimal C program wouldn't be significantly faster than both.
- mlochbaum 5y ago
  Almost! I can't quite replicate these tests because it doesn't give the type (integer or float, and width) of the database columns. There are many possibilities[0] so I tested a few. Fortunately the testing CPU (Intel Xeon E5-2650 v4 @ 2.20 GHz) is comparable to mine (Intel Core i5-6200U CPU @ 2.30GHz). Turbo (one-core speed) is 3.00 GHz versus 2.80 GHz with advantage to the Xeon, but my laptop CPU is a year newer and on a smaller process so could have better IPC. Both have AVX2, and the server CPU has a much better cache, but I don't think Dyalog is bandwidth limited. I adapted the vectorized Dőtsch solution, which gave the fastest times and is easy to write, to Dyalog APL. Pre-release 18.0 from when I worked there because I can't be bothered to do a real install, but I doubt the arithmetic code's been changed since 17.1. Here are results from a Dyalog session giving times in seconds for 8-byte floats, 4-byte integers, and 1-byte integers. From the article, Q takes 13E¯3 seconds. )copy dfns cmpx (sA sB pA pB)←?{1e6⍴ 0}¨⍳4 ⋄ cmpx '(sA×pA≥pB) + sB×pA≤pB' 9.3E¯3 (sA sB pA pB)←?{1e6⍴1e9}¨⍳4 ⋄ cmpx '(sA×pA≥pB) + sB×pA≤pB' 4.6E¯3 (sA sB pA pB)←?{1e6⍴120}¨⍳4 ⋄ cmpx '(sA×pA≥pB) + sB×pA≤pB' 1.3E¯3 If the table consists of floats then Dyalog appears substantially faster, although this could plausibly be due to better IPC on my CPU. It could also be a real increase. Dyalog uses bit booleans for the comparison results which allows it to make smaller reads and writes, and code for packing results to bit booleans and multiplying floats by booleans does have to be written with vector intrinsics ([1] indicates Q doesn't pack booleans). If the benchmark uses 8-byte ints then they should be comparable to floats, and if it's using smaller ints then Dyalog is clearly much better. Will see if I can get numpy benchmarks running. [0] https://code.kx.com/q/basics/datatypes/ https://code.kx.com/q/basics/datatypes/ [1] https://code.kx.com/q4m3/3_Lists/#323-simple-binary-lists https://code.kx.com/q4m3/3_Lists/#323-simple-binary-lists
geocar 5y ago
> If you're not allowed to publish benchmarks you only can rant like this even if it sounds petty. You can repeat whole-system benchmarks like STAC[1] or that some bloggers do[2], which specify the hardware and software, inputs and outputs for a problem. You can always do that. [1]: https://stacresearch.com/m3 https://stacresearch.com/m3 [2]: https://tech.marksblogg.com/billion-nyc-taxi-kdb.html https://tech.marksblogg.com/billion-nyc-taxi-kdb.html
- eatonphil 5y ago
  Interesting, but I don't understand how this is allowed under the license? Edit: it's not that these licenses says you cannot ever publish benchmarks it's that they must be approved. So I assume these benchmarks were approved.
lmm 5y ago
> I mean that if K or some language is actually interpreting without a virtual machine at some stage, that is indeed going to be slower than any language with code running on a virtual machine. Not necessarily. The VM itself will introduce overhead and potentially blow out cache etc.. You can argue, as the article does, that this is a small proportion of execution time if you're executing a lot of loops, but that's inherently workload-dependent. If the language being interpreted is denser than the VM bytecode - something that's believable for a language like K - then interpreting directly makes more efficient use of memory bandwidth than interpreting via bytecode.
- eatonphil 5y ago
  That's an interesting idea, I missed that. Has anyone published a PoC of this (not necessarily using any of these languages) that would give some quantitative credence?

sz4kerto 5y ago

"And also that any language with a virtual machine must be slower than an equivalently intelligent compiled code since it's running on another layer of abstraction that compiled code isn't. " I think this is not correct. A VM has access to the data that the code is operating on, while an AOT compiler doesn't. It's possible that a VM (eg. JVM) produces faster code than a normal compiler.

icen 5y ago
Sadly it also has to consider the time it takes to compile code like that - we grumble, but not too loudly, at the time LLVM takes to compile things. If the JVM took that same time whilst running live that wouldn't be acceptable.
- WJW 5y ago
  Surely this depends entirely on the total runtime of the program. A server-style program which will run for a couple of weeks can accommodate quite a bit of JIT compilation overhead.
eatonphil 5y ago
I'm talking about where the code is run. If it's run on a VM it's run on a VM. If it's run on the cpu by JIT-ed code it's not running on a VM. Edit: But fair I did say "language with a VM" which would include Java JIT-ed or not. Edit, edit: Modified the original comment to say "code on a VM" not "language with a VM".

theelous3 5y ago

I'm not familiar with k, and I'm not understanding this whole "no benchmarks" thing. So it's a proprietary lang? What stops benchmarks being run? Just how ridiculous are the licenses? Why has seemingly nobody run them anyway?

eatonphil 5y ago
I think the article answers all your questions.
tsimionescu 5y ago
Not sure about K, but I know for example that MS didn't allow publishing performance benchmarks of .NET (not sure of this changed with .NET core).
- eatonphil 5y ago
  Both Oracle and MS SQL disallow this without approval. https://stackoverflow.com/a/12116865/1507139 https://stackoverflow.com/a/12116865/1507139
geocar 5y ago
> I'm not understanding this whole "no benchmarks" thing. Back in the 1980s, there was a paper published[1] that compared a number of database systems, and Oracle felt this paper harmed them. Perhaps if the authors knew Oracle better, they could have made more favourable benchmarks, they would argue, and whilst perhaps DeWitt et al could agree to that, they may point out they had not intended to harm Oracle, and being researchers and not marketers were given no notice or expectation that such a review could harm Oracle. And so for the avoidance of doubt, Oracle, Microsoft, and many others (probably including KX, although I've never checked) insist in their licensing agreements that they be able to approve benchmarks, ostensibly so that they have an opportunity to correct any misunderstandings the author might have about the software before. [1]: http://pages.cs.wisc.edu/~dewitt/includes/benchmarking/vldb83.pdf http://pages.cs.wisc.edu/~dewitt/includes/benchmarking/vldb8...
- 5y ago

dagw 5y ago

Here are some (micro) benchmarks of q vs python that they did publish: https://kx.com/blog/a-comparison-of-python-and-q-for-data-problem-solving/ https://kx.com/blog/a-comparison-of-python-and-q-for-data-pr... tl;dr, while q is orders of magnitude faster that naive python, it is, at best, 5 times faster than optimal python using numpy. Based on those numbers I would be very surprised if an optimal C program wouldn't be significantly faster than both.

mlochbaum 5y ago
Almost! I can't quite replicate these tests because it doesn't give the type (integer or float, and width) of the database columns. There are many possibilities[0] so I tested a few. Fortunately the testing CPU (Intel Xeon E5-2650 v4 @ 2.20 GHz) is comparable to mine (Intel Core i5-6200U CPU @ 2.30GHz). Turbo (one-core speed) is 3.00 GHz versus 2.80 GHz with advantage to the Xeon, but my laptop CPU is a year newer and on a smaller process so could have better IPC. Both have AVX2, and the server CPU has a much better cache, but I don't think Dyalog is bandwidth limited. I adapted the vectorized Dőtsch solution, which gave the fastest times and is easy to write, to Dyalog APL. Pre-release 18.0 from when I worked there because I can't be bothered to do a real install, but I doubt the arithmetic code's been changed since 17.1. Here are results from a Dyalog session giving times in seconds for 8-byte floats, 4-byte integers, and 1-byte integers. From the article, Q takes 13E¯3 seconds. )copy dfns cmpx (sA sB pA pB)←?{1e6⍴ 0}¨⍳4 ⋄ cmpx '(sA×pA≥pB) + sB×pA≤pB' 9.3E¯3 (sA sB pA pB)←?{1e6⍴1e9}¨⍳4 ⋄ cmpx '(sA×pA≥pB) + sB×pA≤pB' 4.6E¯3 (sA sB pA pB)←?{1e6⍴120}¨⍳4 ⋄ cmpx '(sA×pA≥pB) + sB×pA≤pB' 1.3E¯3 If the table consists of floats then Dyalog appears substantially faster, although this could plausibly be due to better IPC on my CPU. It could also be a real increase. Dyalog uses bit booleans for the comparison results which allows it to make smaller reads and writes, and code for packing results to bit booleans and multiplying floats by booleans does have to be written with vector intrinsics ([1] indicates Q doesn't pack booleans). If the benchmark uses 8-byte ints then they should be comparable to floats, and if it's using smaller ints then Dyalog is clearly much better. Will see if I can get numpy benchmarks running. [0] https://code.kx.com/q/basics/datatypes/ https://code.kx.com/q/basics/datatypes/ [1] https://code.kx.com/q4m3/3_Lists/#323-simple-binary-lists https://code.kx.com/q4m3/3_Lists/#323-simple-binary-lists

geocar 5y ago

> If you're not allowed to publish benchmarks you only can rant like this even if it sounds petty. You can repeat whole-system benchmarks like STAC[1] or that some bloggers do[2], which specify the hardware and software, inputs and outputs for a problem. You can always do that. [1]: https://stacresearch.com/m3 https://stacresearch.com/m3 [2]: https://tech.marksblogg.com/billion-nyc-taxi-kdb.html https://tech.marksblogg.com/billion-nyc-taxi-kdb.html

eatonphil 5y ago
Interesting, but I don't understand how this is allowed under the license? Edit: it's not that these licenses says you cannot ever publish benchmarks it's that they must be approved. So I assume these benchmarks were approved.

lmm 5y ago

> I mean that if K or some language is actually interpreting without a virtual machine at some stage, that is indeed going to be slower than any language with code running on a virtual machine. Not necessarily. The VM itself will introduce overhead and potentially blow out cache etc.. You can argue, as the article does, that this is a small proportion of execution time if you're executing a lot of loops, but that's inherently workload-dependent. If the language being interpreted is denser than the VM bytecode - something that's believable for a language like K - then interpreting directly makes more efficient use of memory bandwidth than interpreting via bytecode.

eatonphil 5y ago
That's an interesting idea, I missed that. Has anyone published a PoC of this (not necessarily using any of these languages) that would give some quantitative credence?