3 ms·

I tried to keep this as faithful as possible to the original JS code I was copying from. In the actual SunSpider benchmark (in Javascript), a new Array object

I tried to keep this as faithful as possible to the original JS code I was copying from. In the actual SunSpider benchmark (in Javascript), a new Array object is allocated within that same loop, so I wrote the C++ and Java code to mimic that behaviour. I could have pulled the Array allocation out across all of the implementations, but I tried to avoid making any changes to the benchmarks unless there was a correctness-issue involved (e.g. moving makeCumulative in the fasta benchmark out to the prelude was a correctness issue.. since it's wrong to run it more than once on the same array).

duaneb 13y ago
Perhaps, but that's still something you just wouldn't do in native code. If you were to write code that way you wouldn't be writing C++ in the first place (I would hope). I understand the reasoning, but I also think it's misleading in terms of results.
- Skinney 13y ago
  Wouldn't it be more misleading if the C++ version did something different?
- kannanvijayan 13y ago
  It's not how you would write it in Javascript either, or Java. Actually, in general you wouldn't be implementing a sieve algorithm at all. That's a general pitfall of benchmarks like these - the micros don't test real programs so much as they test a limited set of implementation mechanisms. In this case, what we're measuring is: "allocate an array, fill it, and then scan it with various stride lengths and mutate it, and then free it.. what does that cost on average, given this spread of array sizes?" The useful thing with these benchmarks isn't what the final numbers are, but why they are what they are, and what that suggests about the underlying implementation. To put it another way, I think these sorts of comparisons are more useful for being able to confirm that some set of mechanisms work roughly equivalently in one vs the other system.. rather than useful for saying one is "better" than the other in any objective way. For example, the nsieve result suggested an issue with ARM codegeneration with asm.js. But the fact that scores between asm.js and native are pretty close on x86 desktops suggests that outside of codegeneration, the cost of allocating arrays, scanning them, and mutating them like this is roughly equivalent on asm.js and native. Similarly, the nbodies result might be suggesting that double-indirection in hot code is a weak spot for asm.js compared to native. The fasta result suggests that there are high overheads associated with using Java collections for small lookup tables of primitive values. With benchmarks, my opinion is that the scores themselves are less important than how you interpret them. Thus I'm not as concerned about how one would optimally implement a looped sieve algorithm in C++ vs Java vs JS, since that's not what I'm trying to get at.
  - duaneb 13y ago
    Ok, I'll give you this. I think you're entirely correct and I realize I'm not seeing the forest for the trees.

duaneb 13y ago

Perhaps, but that's still something you just wouldn't do in native code. If you were to write code that way you wouldn't be writing C++ in the first place (I would hope). I understand the reasoning, but I also think it's misleading in terms of results.

Skinney 13y ago
Wouldn't it be more misleading if the C++ version did something different?
kannanvijayan 13y ago
It's not how you would write it in Javascript either, or Java. Actually, in general you wouldn't be implementing a sieve algorithm at all. That's a general pitfall of benchmarks like these - the micros don't test real programs so much as they test a limited set of implementation mechanisms. In this case, what we're measuring is: "allocate an array, fill it, and then scan it with various stride lengths and mutate it, and then free it.. what does that cost on average, given this spread of array sizes?" The useful thing with these benchmarks isn't what the final numbers are, but why they are what they are, and what that suggests about the underlying implementation. To put it another way, I think these sorts of comparisons are more useful for being able to confirm that some set of mechanisms work roughly equivalently in one vs the other system.. rather than useful for saying one is "better" than the other in any objective way. For example, the nsieve result suggested an issue with ARM codegeneration with asm.js. But the fact that scores between asm.js and native are pretty close on x86 desktops suggests that outside of codegeneration, the cost of allocating arrays, scanning them, and mutating them like this is roughly equivalent on asm.js and native. Similarly, the nbodies result might be suggesting that double-indirection in hot code is a weak spot for asm.js compared to native. The fasta result suggests that there are high overheads associated with using Java collections for small lookup tables of primitive values. With benchmarks, my opinion is that the scores themselves are less important than how you interpret them. Thus I'm not as concerned about how one would optimally implement a looped sieve algorithm in C++ vs Java vs JS, since that's not what I'm trying to get at.
- duaneb 13y ago
  Ok, I'll give you this. I think you're entirely correct and I realize I'm not seeing the forest for the trees.