6 ms·

So the author is incredulous but provides no proof or stats, just a long rant. Another overlooked "performance" feature of terse languages is that of the devel

by anonu 5y ago

So the author is incredulous but provides no proof or stats, just a long rant. Another overlooked "performance" feature of terse languages is that of the developer. Vector based, concise languages give you more in one line of code. And I don't find that readability suffers. Yes you need to stare at that one line longer, but there's more going on.

eatonphil 5y ago
> So the author is incredulous but provides no proof or stats, just a long rant. I thought the same thing but on the other hand, what can you do? If you're not allowed to publish benchmarks you only can rant like this even if it sounds petty. By getting attention here on HN it helps put pressure on commercial vendors to stop sharing unconfirmable benchmarks otherwise they will lose credibility since they've been called out. The author does make some good points about the fundamental nature of interpreters too. Although it's hard to confirm there too what these languages do without seeing their source. I mean that if K or some language is actually interpreting without a virtual machine at some stage, that is indeed going to be slower than any language with code running on a virtual machine. And also that any code that runs on a virtual machine must be slower than an equivalently intelligent compiled code since it's running on another layer of abstraction that compiled code isn't. ("equivalently intelligent" because yes a virtual machine could be faster than terrible compiled code.)
- sz4kerto 5y ago
  "And also that any language with a virtual machine must be slower than an equivalently intelligent compiled code since it's running on another layer of abstraction that compiled code isn't. " I think this is not correct. A VM has access to the data that the code is operating on, while an AOT compiler doesn't. It's possible that a VM (eg. JVM) produces faster code than a normal compiler.
  - icen 5y ago
    Sadly it also has to consider the time it takes to compile code like that - we grumble, but not too loudly, at the time LLVM takes to compile things. If the JVM took that same time whilst running live that wouldn't be acceptable.
    - WJW 5y ago
      Surely this depends entirely on the total runtime of the program. A server-style program which will run for a couple of weeks can accommodate quite a bit of JIT compilation overhead.
  - eatonphil 5y ago
    I'm talking about where the code is run. If it's run on a VM it's run on a VM. If it's run on the cpu by JIT-ed code it's not running on a VM. Edit: But fair I did say "language with a VM" which would include Java JIT-ed or not. Edit, edit: Modified the original comment to say "code on a VM" not "language with a VM".
- theelous3 5y ago
  I'm not familiar with k, and I'm not understanding this whole "no benchmarks" thing. So it's a proprietary lang? What stops benchmarks being run? Just how ridiculous are the licenses? Why has seemingly nobody run them anyway?
  - eatonphil 5y ago
    I think the article answers all your questions.
  - tsimionescu 5y ago
    Not sure about K, but I know for example that MS didn't allow publishing performance benchmarks of .NET (not sure of this changed with .NET core).
    - eatonphil 5y ago
      Both Oracle and MS SQL disallow this without approval. https://stackoverflow.com/a/12116865/1507139 https://stackoverflow.com/a/12116865/1507139
  - geocar 5y ago
    > I'm not understanding this whole "no benchmarks" thing. Back in the 1980s, there was a paper published[1] that compared a number of database systems, and Oracle felt this paper harmed them. Perhaps if the authors knew Oracle better, they could have made more favourable benchmarks, they would argue, and whilst perhaps DeWitt et al could agree to that, they may point out they had not intended to harm Oracle, and being researchers and not marketers were given no notice or expectation that such a review could harm Oracle. And so for the avoidance of doubt, Oracle, Microsoft, and many others (probably including KX, although I've never checked) insist in their licensing agreements that they be able to approve benchmarks, ostensibly so that they have an opportunity to correct any misunderstandings the author might have about the software before. [1]: http://pages.cs.wisc.edu/~dewitt/includes/benchmarking/vldb83.pdf http://pages.cs.wisc.edu/~dewitt/includes/benchmarking/vldb8...
    - 5y ago
- dagw 5y ago
  Here are some (micro) benchmarks of q vs python that they did publish: https://kx.com/blog/a-comparison-of-python-and-q-for-data-problem-solving/ https://kx.com/blog/a-comparison-of-python-and-q-for-data-pr... tl;dr, while q is orders of magnitude faster that naive python, it is, at best, 5 times faster than optimal python using numpy. Based on those numbers I would be very surprised if an optimal C program wouldn't be significantly faster than both.
  - mlochbaum 5y ago
    Almost! I can't quite replicate these tests because it doesn't give the type (integer or float, and width) of the database columns. There are many possibilities[0] so I tested a few. Fortunately the testing CPU (Intel Xeon E5-2650 v4 @ 2.20 GHz) is comparable to mine (Intel Core i5-6200U CPU @ 2.30GHz). Turbo (one-core speed) is 3.00 GHz versus 2.80 GHz with advantage to the Xeon, but my laptop CPU is a year newer and on a smaller process so could have better IPC. Both have AVX2, and the server CPU has a much better cache, but I don't think Dyalog is bandwidth limited. I adapted the vectorized Dőtsch solution, which gave the fastest times and is easy to write, to Dyalog APL. Pre-release 18.0 from when I worked there because I can't be bothered to do a real install, but I doubt the arithmetic code's been changed since 17.1. Here are results from a Dyalog session giving times in seconds for 8-byte floats, 4-byte integers, and 1-byte integers. From the article, Q takes 13E¯3 seconds. )copy dfns cmpx (sA sB pA pB)←?{1e6⍴ 0}¨⍳4 ⋄ cmpx '(sA×pA≥pB) + sB×pA≤pB' 9.3E¯3 (sA sB pA pB)←?{1e6⍴1e9}¨⍳4 ⋄ cmpx '(sA×pA≥pB) + sB×pA≤pB' 4.6E¯3 (sA sB pA pB)←?{1e6⍴120}¨⍳4 ⋄ cmpx '(sA×pA≥pB) + sB×pA≤pB' 1.3E¯3 If the table consists of floats then Dyalog appears substantially faster, although this could plausibly be due to better IPC on my CPU. It could also be a real increase. Dyalog uses bit booleans for the comparison results which allows it to make smaller reads and writes, and code for packing results to bit booleans and multiplying floats by booleans does have to be written with vector intrinsics ([1] indicates Q doesn't pack booleans). If the benchmark uses 8-byte ints then they should be comparable to floats, and if it's using smaller ints then Dyalog is clearly much better. Will see if I can get numpy benchmarks running. [0] https://code.kx.com/q/basics/datatypes/ https://code.kx.com/q/basics/datatypes/ [1] https://code.kx.com/q4m3/3_Lists/#323-simple-binary-lists https://code.kx.com/q4m3/3_Lists/#323-simple-binary-lists
- geocar 5y ago
  > If you're not allowed to publish benchmarks you only can rant like this even if it sounds petty. You can repeat whole-system benchmarks like STAC[1] or that some bloggers do[2], which specify the hardware and software, inputs and outputs for a problem. You can always do that. [1]: https://stacresearch.com/m3 https://stacresearch.com/m3 [2]: https://tech.marksblogg.com/billion-nyc-taxi-kdb.html https://tech.marksblogg.com/billion-nyc-taxi-kdb.html
  - eatonphil 5y ago
    Interesting, but I don't understand how this is allowed under the license? Edit: it's not that these licenses says you cannot ever publish benchmarks it's that they must be approved. So I assume these benchmarks were approved.
- lmm 5y ago
  > I mean that if K or some language is actually interpreting without a virtual machine at some stage, that is indeed going to be slower than any language with code running on a virtual machine. Not necessarily. The VM itself will introduce overhead and potentially blow out cache etc.. You can argue, as the article does, that this is a small proportion of execution time if you're executing a lot of loops, but that's inherently workload-dependent. If the language being interpreted is denser than the VM bytecode - something that's believable for a language like K - then interpreting directly makes more efficient use of memory bandwidth than interpreting via bytecode.
  - eatonphil 5y ago
    That's an interesting idea, I missed that. Has anyone published a PoC of this (not necessarily using any of these languages) that would give some quantitative credence?
tosh 5y ago
Concise languages are also more fun to use in a REPL (another aspect of "performance")
icen 5y ago
You might enjoy the language created by the author of the article, then. BQN is a language that favours terseness, even above and beyond k. It is in many ways an improvement on APL and J, importantly through a more convenient set of operators than found in either.
- rak1507 5y ago
  Personally from what I've seen I wouldn't say BQN values terseness more than k, probably less than most other array langs. (Edit: if you downvoted this: why?)
  - icen 5y ago
    I agree; I don't think BQN values terseness more than k. I think k values terseness a lot. I think it may be more convenient to write terse code in BQN over k, because there is greater facility to control function calls in a terse way. Things like under and over are quite common patterns that BQN finds easier (and shorter) than k. I've not spent a great deal of time yet with BQN, so this could be wrong.
    - rak1507 5y ago
      Yeah, under and over are great (wish k had more 'combinators' in general), but k does manage to cram a lot of stuff in with overloads. I feel like k is terser in general but I don't really have much evidence to back it up, just from playing a bit with both of them.
      - mlochbaum 5y ago
        The different character sets make it impossible to say what this would mean. My limited code golf experience is that most often BQN solutions use fewer characters than K ones, but BQN has 63 primitive glyphs to K's 25 or so (depending on version). I do think K design values using its limited resources to enable terse code more than BQN, but if screen space is what you care about then BQN will tend to use less of it.
tomp 5y ago
The author explicitly states that K licence prevents him from providing stats.
- gd1 5y ago
  There seem to be benchmarks available in other places: https://tech.marksblogg.com/benchmarks.html https://tech.marksblogg.com/benchmarks.html
- linspace 5y ago
  I'm surprised nevertheless that this is enforceable. Is there a precedent of someone being sued and losing?
  - eatonphil 5y ago
    If you're reliant on a vendor for a service why risk getting on their bad side?
    - linspace 5y ago
      What makes you think everyone doing benchmarks is reliant? it could be research, it could be curiosity, you actually could be competing... Edit: and nevertheless is quite depressing that as a paying customer you should be afraid
  - 1vuio0pswjnm7 5y ago
    No.
mlochbaum 5y ago
I wrote this article to link in response to evidence-free claims about K, and have in fact used it in this way twice in the two months since I wrote it. Didn't expect HN in general to take an interest. So I need to stress again that K is a really cool language and I like it a lot! If the article can provoke clear benchmarks against current Dyalog or J then that would be an ideal outcome, even if it proves me wrong. I already see some benchmarks linked here and will try to analyze these.
coldtea 5y ago
>So the author is incredulous but provides no proof or stats, just a long rant. Isn't the burden on proof on the ones making the claim for the speed of their product? Incredulity should be the default scientific stance here...

by anonu 5y ago

eatonphil 5y ago

> So the author is incredulous but provides no proof or stats, just a long rant. I thought the same thing but on the other hand, what can you do? If you're not allowed to publish benchmarks you only can rant like this even if it sounds petty. By getting attention here on HN it helps put pressure on commercial vendors to stop sharing unconfirmable benchmarks otherwise they will lose credibility since they've been called out. The author does make some good points about the fundamental nature of interpreters too. Although it's hard to confirm there too what these languages do without seeing their source. I mean that if K or some language is actually interpreting without a virtual machine at some stage, that is indeed going to be slower than any language with code running on a virtual machine. And also that any code that runs on a virtual machine must be slower than an equivalently intelligent compiled code since it's running on another layer of abstraction that compiled code isn't. ("equivalently intelligent" because yes a virtual machine could be faster than terrible compiled code.)

sz4kerto 5y ago
"And also that any language with a virtual machine must be slower than an equivalently intelligent compiled code since it's running on another layer of abstraction that compiled code isn't. " I think this is not correct. A VM has access to the data that the code is operating on, while an AOT compiler doesn't. It's possible that a VM (eg. JVM) produces faster code than a normal compiler.
- icen 5y ago
  Sadly it also has to consider the time it takes to compile code like that - we grumble, but not too loudly, at the time LLVM takes to compile things. If the JVM took that same time whilst running live that wouldn't be acceptable.
  - WJW 5y ago
    Surely this depends entirely on the total runtime of the program. A server-style program which will run for a couple of weeks can accommodate quite a bit of JIT compilation overhead.
- eatonphil 5y ago
  I'm talking about where the code is run. If it's run on a VM it's run on a VM. If it's run on the cpu by JIT-ed code it's not running on a VM. Edit: But fair I did say "language with a VM" which would include Java JIT-ed or not. Edit, edit: Modified the original comment to say "code on a VM" not "language with a VM".
theelous3 5y ago
I'm not familiar with k, and I'm not understanding this whole "no benchmarks" thing. So it's a proprietary lang? What stops benchmarks being run? Just how ridiculous are the licenses? Why has seemingly nobody run them anyway?
- eatonphil 5y ago
  I think the article answers all your questions.
- tsimionescu 5y ago
  Not sure about K, but I know for example that MS didn't allow publishing performance benchmarks of .NET (not sure of this changed with .NET core).
  - eatonphil 5y ago
    Both Oracle and MS SQL disallow this without approval. https://stackoverflow.com/a/12116865/1507139 https://stackoverflow.com/a/12116865/1507139
- geocar 5y ago
  > I'm not understanding this whole "no benchmarks" thing. Back in the 1980s, there was a paper published[1] that compared a number of database systems, and Oracle felt this paper harmed them. Perhaps if the authors knew Oracle better, they could have made more favourable benchmarks, they would argue, and whilst perhaps DeWitt et al could agree to that, they may point out they had not intended to harm Oracle, and being researchers and not marketers were given no notice or expectation that such a review could harm Oracle. And so for the avoidance of doubt, Oracle, Microsoft, and many others (probably including KX, although I've never checked) insist in their licensing agreements that they be able to approve benchmarks, ostensibly so that they have an opportunity to correct any misunderstandings the author might have about the software before. [1]: http://pages.cs.wisc.edu/~dewitt/includes/benchmarking/vldb83.pdf http://pages.cs.wisc.edu/~dewitt/includes/benchmarking/vldb8...
  - 5y ago
dagw 5y ago
Here are some (micro) benchmarks of q vs python that they did publish: https://kx.com/blog/a-comparison-of-python-and-q-for-data-problem-solving/ https://kx.com/blog/a-comparison-of-python-and-q-for-data-pr... tl;dr, while q is orders of magnitude faster that naive python, it is, at best, 5 times faster than optimal python using numpy. Based on those numbers I would be very surprised if an optimal C program wouldn't be significantly faster than both.
- mlochbaum 5y ago
  Almost! I can't quite replicate these tests because it doesn't give the type (integer or float, and width) of the database columns. There are many possibilities[0] so I tested a few. Fortunately the testing CPU (Intel Xeon E5-2650 v4 @ 2.20 GHz) is comparable to mine (Intel Core i5-6200U CPU @ 2.30GHz). Turbo (one-core speed) is 3.00 GHz versus 2.80 GHz with advantage to the Xeon, but my laptop CPU is a year newer and on a smaller process so could have better IPC. Both have AVX2, and the server CPU has a much better cache, but I don't think Dyalog is bandwidth limited. I adapted the vectorized Dőtsch solution, which gave the fastest times and is easy to write, to Dyalog APL. Pre-release 18.0 from when I worked there because I can't be bothered to do a real install, but I doubt the arithmetic code's been changed since 17.1. Here are results from a Dyalog session giving times in seconds for 8-byte floats, 4-byte integers, and 1-byte integers. From the article, Q takes 13E¯3 seconds. )copy dfns cmpx (sA sB pA pB)←?{1e6⍴ 0}¨⍳4 ⋄ cmpx '(sA×pA≥pB) + sB×pA≤pB' 9.3E¯3 (sA sB pA pB)←?{1e6⍴1e9}¨⍳4 ⋄ cmpx '(sA×pA≥pB) + sB×pA≤pB' 4.6E¯3 (sA sB pA pB)←?{1e6⍴120}¨⍳4 ⋄ cmpx '(sA×pA≥pB) + sB×pA≤pB' 1.3E¯3 If the table consists of floats then Dyalog appears substantially faster, although this could plausibly be due to better IPC on my CPU. It could also be a real increase. Dyalog uses bit booleans for the comparison results which allows it to make smaller reads and writes, and code for packing results to bit booleans and multiplying floats by booleans does have to be written with vector intrinsics ([1] indicates Q doesn't pack booleans). If the benchmark uses 8-byte ints then they should be comparable to floats, and if it's using smaller ints then Dyalog is clearly much better. Will see if I can get numpy benchmarks running. [0] https://code.kx.com/q/basics/datatypes/ https://code.kx.com/q/basics/datatypes/ [1] https://code.kx.com/q4m3/3_Lists/#323-simple-binary-lists https://code.kx.com/q4m3/3_Lists/#323-simple-binary-lists
geocar 5y ago
> If you're not allowed to publish benchmarks you only can rant like this even if it sounds petty. You can repeat whole-system benchmarks like STAC[1] or that some bloggers do[2], which specify the hardware and software, inputs and outputs for a problem. You can always do that. [1]: https://stacresearch.com/m3 https://stacresearch.com/m3 [2]: https://tech.marksblogg.com/billion-nyc-taxi-kdb.html https://tech.marksblogg.com/billion-nyc-taxi-kdb.html
- eatonphil 5y ago
  Interesting, but I don't understand how this is allowed under the license? Edit: it's not that these licenses says you cannot ever publish benchmarks it's that they must be approved. So I assume these benchmarks were approved.
lmm 5y ago
> I mean that if K or some language is actually interpreting without a virtual machine at some stage, that is indeed going to be slower than any language with code running on a virtual machine. Not necessarily. The VM itself will introduce overhead and potentially blow out cache etc.. You can argue, as the article does, that this is a small proportion of execution time if you're executing a lot of loops, but that's inherently workload-dependent. If the language being interpreted is denser than the VM bytecode - something that's believable for a language like K - then interpreting directly makes more efficient use of memory bandwidth than interpreting via bytecode.
- eatonphil 5y ago
  That's an interesting idea, I missed that. Has anyone published a PoC of this (not necessarily using any of these languages) that would give some quantitative credence?

tosh 5y ago

Concise languages are also more fun to use in a REPL (another aspect of "performance")

icen 5y ago

You might enjoy the language created by the author of the article, then. BQN is a language that favours terseness, even above and beyond k. It is in many ways an improvement on APL and J, importantly through a more convenient set of operators than found in either.

rak1507 5y ago
Personally from what I've seen I wouldn't say BQN values terseness more than k, probably less than most other array langs. (Edit: if you downvoted this: why?)
- icen 5y ago
  I agree; I don't think BQN values terseness more than k. I think k values terseness a lot. I think it may be more convenient to write terse code in BQN over k, because there is greater facility to control function calls in a terse way. Things like under and over are quite common patterns that BQN finds easier (and shorter) than k. I've not spent a great deal of time yet with BQN, so this could be wrong.
  - rak1507 5y ago
    Yeah, under and over are great (wish k had more 'combinators' in general), but k does manage to cram a lot of stuff in with overloads. I feel like k is terser in general but I don't really have much evidence to back it up, just from playing a bit with both of them.
    - mlochbaum 5y ago
      The different character sets make it impossible to say what this would mean. My limited code golf experience is that most often BQN solutions use fewer characters than K ones, but BQN has 63 primitive glyphs to K's 25 or so (depending on version). I do think K design values using its limited resources to enable terse code more than BQN, but if screen space is what you care about then BQN will tend to use less of it.

tomp 5y ago

The author explicitly states that K licence prevents him from providing stats.

gd1 5y ago
There seem to be benchmarks available in other places: https://tech.marksblogg.com/benchmarks.html https://tech.marksblogg.com/benchmarks.html
linspace 5y ago
I'm surprised nevertheless that this is enforceable. Is there a precedent of someone being sued and losing?
- eatonphil 5y ago
  If you're reliant on a vendor for a service why risk getting on their bad side?
  - linspace 5y ago
    What makes you think everyone doing benchmarks is reliant? it could be research, it could be curiosity, you actually could be competing... Edit: and nevertheless is quite depressing that as a paying customer you should be afraid
- 1vuio0pswjnm7 5y ago
  No.

mlochbaum 5y ago

I wrote this article to link in response to evidence-free claims about K, and have in fact used it in this way twice in the two months since I wrote it. Didn't expect HN in general to take an interest. So I need to stress again that K is a really cool language and I like it a lot! If the article can provoke clear benchmarks against current Dyalog or J then that would be an ideal outcome, even if it proves me wrong. I already see some benchmarks linked here and will try to analyze these.

coldtea 5y ago

>So the author is incredulous but provides no proof or stats, just a long rant. Isn't the burden on proof on the ones making the claim for the speed of their product? Incredulity should be the default scientific stance here...