4 ms·

>But apparently bringing data is a waste of time. Frankly, bringing apple production data to an orange framers' convention is a waste of time... I keep seeing

>But apparently bringing data is a waste of time. Frankly, bringing apple production data to an orange framers' convention is a waste of time... I keep seeing you pop up in comments what feels like every time there is something about C# or dotnet or CoreCLR, again and again with your questionable comparisons and "data": - You're compiling Oberon+ to CLR/IL code using your own compiler. Oberon+ is a language you yourself maintain it seems. It's not C# or F#, which are by far the dominant languages for dotnet. To me at least, Oberon+ seems like a rather esoteric niche language without wide adoption, but I could be missing something. - The quality of the codegen your Oberon+-to-CLR/IL compiler manages to achieve compared to the dotnet compiler that people would actually use most of the time, that's a big unknown. But I see a real chance that the dotnet compiler, that saw countless hours of development time from a large team of some of the brightest domain experts that MS could find, is outperforming your compiler's codegen. - How well Oberon+ translates to CLR/IL (or CIL, basically bytecode for compatible .NET runtimes) is another unknown (is it straight forward, does it require the compiler to add a lot of "glue" code or structures to accommodate that language?) - You're not running your benchmarks on the dotnet CoreCLR, but on mono. And you're trying to handwave this important fact away by linking one of your own quora posts that again uses your own Oberon+-to-CLR/IL-with-my-own-compiler benchmarks that allege to show that there isn't much of a difference between mono and the CoreCLR runtimes. Maybe there isn't, or maybe there isn't just for the very narrow specific IL your Oberon+ compiler produces for the Oberon+ benchmark code you use. But even looking at your own results in your quora post, there seems to be a significant difference in performance even with your narrow use case when comparing mono to CoreCLR 5/6 performance, in favor of the CoreCLR. - You're running the result of what your own Oberon+-to-CLR/IL compiler spits out on mono-3 (last release 2015, EOL) and mono-5 (last release 2019, EOL) runtimes. Current stable mono-6 was released in 2019. So you're not even running on a current mono release, let alone a current CoreCLR release. The old (mono) runtimes and standard libraries lack a lot of performance-related features that modern CoreCLR versions have (especially when it comes to value types, Span<> and stackalloc, etc, that can be used explicitly in code or in a lot of cases even implicitly by an optimizing compiler) - You're running on rather old hardware, and a laptop hardware no less. The Intel Core Duo L9400 you mention was released in Q1'06, but according to my google-fu the HP EliteBook 2530p actually uses the SL9400 (is that a typo in your benchmark results pdf?), which was released in Q3'08. It still only comes with DDR2. Would be interesting to know what storage option yours has (as cold-start times of your benchmarks might be dominated e.g. by a slow 5400 rpm laptop grade HDD that is one of the 2530p's storage options). And you run benchmarks on a 32 bit OS. All of which is fine, if you're interested in results on such hardware, but I'd think most people would be more interested in results on slightly more modern hardware. - You managed this time to comment on a Rust-related post, agreeing that Rust is probably faster, trying to justify this assertion with a set of benchmarks that do not include Rust whatsoever, just saying that Rust performance might be close to C. Which is not a bad hypothesis to start with, but when you go on linking your data and nonchalantly proclaiming that "bringing data is a waste of time" when you didn't bring any data regarding Rust, then I don't get your point.

Rochus 5y ago
At least you had a look at the links, but obviously not good enough; your reproaches are unnecessary and unjustified. It is irrelevant that the benchmark is written in Oberon. Much more important is the fact that it is not just one of those non-representative micro benchmarks, and that the same code translates fairly directly into both CLR/IL and C. So we can directly compare the ability of CLR to optimize this code with the optimizations done by GCC or CLANG. The fact that Mono is only a factor of two behind is pretty good. But at least it's a factor of two. That Rust's performance is very close to C's is well known (from CLBG and other benchmarks); you're invited to bring your own version of the Are-we-fast-yet suite in Rust otherwise. You scatter doubts that my data is unrepresentative by simply making some rather adventurous claims whose credibility is not supported by any measurements or data at all. It is irrelevant that C# is the most commonly used language on .NET; it is by definition and design a "common language infrastructure", and the article comments on .NET in general (and makes claims at least as adventurous as yours, the ones discussed here also without supporting evidence). Regarding your nonsensical claims about Span<> and stackalloc, I recommend that you take a look at ECMA-335 and its development over time (and at the generated code). Also your assertion about the relevance of the HDD and the other mentioned hardware aspects makes me doubt whether you have any idea at all about what you are writing here; but in this day and age, that seems to be enough; so why should people take the effort of making measurements and publishing data?
- rndgermandude 5y ago
  > It is irrelevant that the benchmark is written in Oberon. It is not irrelevant from my perspective. It might be mostly irrelevant if Oberon, as you say, "fairly directly" translates to CRL/IL, that is, there is no need to introduce any (inefficient) glue constructs to support Oberon features. I didn't/do not know this, but if you say this to be the case now, good, that answers my doubts in that regard. However, this still does not speak to how good the codegen of your Oberon-to-CLR/IL compiler is. Yes, the mono or CoreCLR JIT will apply further optimizations, but in a significant part it relies on the quality of the codegen of the IL it gets passed to do so. >The fact that Mono is only a factor of two behind is pretty good. According to your benchmarks. I'd still be a lot more interested in how well the CoreCRL would fare when handed CLR/IL generated from idiomatic C# or F# by the dotnet (Roslyn) compiler. On recent hardware. Or at least a recent hardware architecture. Right now, it seems to me you're doing something analogous to compiling C with tcc and then making general performance statements about C performance based on tcc generated machine code. Even your own mono-3 and mono-5 results differ, so choice of runtime is clearly a factor. >It is irrelevant that C# is the most commonly used language on .NET; it is by definition and design a "common language infrastructure", and the article comments on .NET in general (and makes claims at least as adventurous as yours, the ones discussed here also without supporting evidence). It is not irrelevant. The claims unsupported by any evidence are yours by the way. You made statements about performance, and all you presented is what I consider at best highly questionable methodology. If you make such claims, and try to back them up by "data", you'll have to demonstrate that your Oberon+-benchmark-code-compiled-to-CLR/IR-as-run-on-old-mono-on-outdated-hardware benchmark methodology is indeed something that can be used to make such claims about ".NET" in a sound manner. As I said, I doubt that, and that's all I did, express doubts about your methodology, and why I have these doubts. > Regarding your nonsensical claims about Span<> and stackalloc, I recommend that you take a look at ECMA-335 and its development over time (and at the generated code). I have been following standard library and CLR/IR development, and Rosyln compiler development, and therefore know that my statements are not nonsensical. E.g. here[0] is the dotnet "Augments" document that lists, aside from many clarifications, outright additions not present in the latest ECMA-335 spec. See e.g. the changes/additions regarding Byref-like types, which in practice allow for all kinds of optimizations, a lot of them implemented in CoreCLR 5 or 6, but of course only enabled if the compiler's codegen emits these new byref-like type variants. Furthermore, a lot of the optimizations that current-generation to-CLR/IL compilers and runtimes/JITs perform do not rely on changes to ECMA-335 at all. E.g. as I mentioned above, your mono-3 runs the CLR/IL your compiler generated significantly faster than your mono-5. >Also your assertion about the relevance of the HDD and the other mentioned hardware aspects makes me doubt whether you have any idea at all about what you are writing here You're running on a somewhat memory-constrained old machine, so even if you try to run your benchmarks "hot", i.e. have everything required loaded into OS-level caches before the timing starts, I wouldn't be convinced that this is necessarily the case (disk cache pages might have been evicted, and pages in general might have been swapped out and need swapping back in again during a run, and if that happens the HDD performance would play a major role). Additionally, laptops (with relatively early versions SpeedStep-capable CPUs, as your is) are particularly susceptible to additional issues that might adversely affect the reliability of benchmark results, e.g. thermal throttling. If any of this happens, it might be (but is not necessarily) detectable in e.g. stddev (your benchmarks pdf lacks any mention of stddev, so I cannot know). I remember trying to benchmark something compute-heavy on my HP Elitebook 8440p[2] back in the day, and gave up after I couldn't get the thing to produce any sane results for different runs of the same code. The choice of your hardware might as well matter and create vastly different proportional results between the different runtimes you test when the same benchmarks are run on different hardware, even if there aren't any of the problems I described. As I wrote, your choice of hardware and software (and my anecdote) does not necessarily invalidate your results, but it might render your results rather meaningless for other, more commonly used current generation hardware. And that's something to be aware of. >so why should people take the effort of making measurements and publishing data? Sorry, but the effort you obviously put into generating your benchmarks and data does not preclude your work from being poked-and-prodded, questioned and criticized. [0] https://github.com/dotnet/runtime/blob/main/docs/design/specs/Ecma-335-Augments.md https://github.com/dotnet/runtime/blob/main/docs/design/spec... [1] I am furthermore not sure if you run the same CLR/IL; the mono-3 vs mono-5 seem to be using different versions of your OBXMC as indicated by the different dates that seem to correspond to version numbers for the other things benchmarked? [2] The variant with the Core i7-720QM if I remember correctly, slightly newer than yours, but probably even more susceptible to SpeedStep and thermal throttling.

Rochus 5y ago

At least you had a look at the links, but obviously not good enough; your reproaches are unnecessary and unjustified. It is irrelevant that the benchmark is written in Oberon. Much more important is the fact that it is not just one of those non-representative micro benchmarks, and that the same code translates fairly directly into both CLR/IL and C. So we can directly compare the ability of CLR to optimize this code with the optimizations done by GCC or CLANG. The fact that Mono is only a factor of two behind is pretty good. But at least it's a factor of two. That Rust's performance is very close to C's is well known (from CLBG and other benchmarks); you're invited to bring your own version of the Are-we-fast-yet suite in Rust otherwise. You scatter doubts that my data is unrepresentative by simply making some rather adventurous claims whose credibility is not supported by any measurements or data at all. It is irrelevant that C# is the most commonly used language on .NET; it is by definition and design a "common language infrastructure", and the article comments on .NET in general (and makes claims at least as adventurous as yours, the ones discussed here also without supporting evidence). Regarding your nonsensical claims about Span<> and stackalloc, I recommend that you take a look at ECMA-335 and its development over time (and at the generated code). Also your assertion about the relevance of the HDD and the other mentioned hardware aspects makes me doubt whether you have any idea at all about what you are writing here; but in this day and age, that seems to be enough; so why should people take the effort of making measurements and publishing data?

rndgermandude 5y ago
> It is irrelevant that the benchmark is written in Oberon. It is not irrelevant from my perspective. It might be mostly irrelevant if Oberon, as you say, "fairly directly" translates to CRL/IL, that is, there is no need to introduce any (inefficient) glue constructs to support Oberon features. I didn't/do not know this, but if you say this to be the case now, good, that answers my doubts in that regard. However, this still does not speak to how good the codegen of your Oberon-to-CLR/IL compiler is. Yes, the mono or CoreCLR JIT will apply further optimizations, but in a significant part it relies on the quality of the codegen of the IL it gets passed to do so. >The fact that Mono is only a factor of two behind is pretty good. According to your benchmarks. I'd still be a lot more interested in how well the CoreCRL would fare when handed CLR/IL generated from idiomatic C# or F# by the dotnet (Roslyn) compiler. On recent hardware. Or at least a recent hardware architecture. Right now, it seems to me you're doing something analogous to compiling C with tcc and then making general performance statements about C performance based on tcc generated machine code. Even your own mono-3 and mono-5 results differ, so choice of runtime is clearly a factor. >It is irrelevant that C# is the most commonly used language on .NET; it is by definition and design a "common language infrastructure", and the article comments on .NET in general (and makes claims at least as adventurous as yours, the ones discussed here also without supporting evidence). It is not irrelevant. The claims unsupported by any evidence are yours by the way. You made statements about performance, and all you presented is what I consider at best highly questionable methodology. If you make such claims, and try to back them up by "data", you'll have to demonstrate that your Oberon+-benchmark-code-compiled-to-CLR/IR-as-run-on-old-mono-on-outdated-hardware benchmark methodology is indeed something that can be used to make such claims about ".NET" in a sound manner. As I said, I doubt that, and that's all I did, express doubts about your methodology, and why I have these doubts. > Regarding your nonsensical claims about Span<> and stackalloc, I recommend that you take a look at ECMA-335 and its development over time (and at the generated code). I have been following standard library and CLR/IR development, and Rosyln compiler development, and therefore know that my statements are not nonsensical. E.g. here[0] is the dotnet "Augments" document that lists, aside from many clarifications, outright additions not present in the latest ECMA-335 spec. See e.g. the changes/additions regarding Byref-like types, which in practice allow for all kinds of optimizations, a lot of them implemented in CoreCLR 5 or 6, but of course only enabled if the compiler's codegen emits these new byref-like type variants. Furthermore, a lot of the optimizations that current-generation to-CLR/IL compilers and runtimes/JITs perform do not rely on changes to ECMA-335 at all. E.g. as I mentioned above, your mono-3 runs the CLR/IL your compiler generated significantly faster than your mono-5. >Also your assertion about the relevance of the HDD and the other mentioned hardware aspects makes me doubt whether you have any idea at all about what you are writing here You're running on a somewhat memory-constrained old machine, so even if you try to run your benchmarks "hot", i.e. have everything required loaded into OS-level caches before the timing starts, I wouldn't be convinced that this is necessarily the case (disk cache pages might have been evicted, and pages in general might have been swapped out and need swapping back in again during a run, and if that happens the HDD performance would play a major role). Additionally, laptops (with relatively early versions SpeedStep-capable CPUs, as your is) are particularly susceptible to additional issues that might adversely affect the reliability of benchmark results, e.g. thermal throttling. If any of this happens, it might be (but is not necessarily) detectable in e.g. stddev (your benchmarks pdf lacks any mention of stddev, so I cannot know). I remember trying to benchmark something compute-heavy on my HP Elitebook 8440p[2] back in the day, and gave up after I couldn't get the thing to produce any sane results for different runs of the same code. The choice of your hardware might as well matter and create vastly different proportional results between the different runtimes you test when the same benchmarks are run on different hardware, even if there aren't any of the problems I described. As I wrote, your choice of hardware and software (and my anecdote) does not necessarily invalidate your results, but it might render your results rather meaningless for other, more commonly used current generation hardware. And that's something to be aware of. >so why should people take the effort of making measurements and publishing data? Sorry, but the effort you obviously put into generating your benchmarks and data does not preclude your work from being poked-and-prodded, questioned and criticized. [0] https://github.com/dotnet/runtime/blob/main/docs/design/specs/Ecma-335-Augments.md https://github.com/dotnet/runtime/blob/main/docs/design/spec... [1] I am furthermore not sure if you run the same CLR/IL; the mono-3 vs mono-5 seem to be using different versions of your OBXMC as indicated by the different dates that seem to correspond to version numbers for the other things benchmarked? [2] The variant with the Core i7-720QM if I remember correctly, slightly newer than yours, but probably even more susceptible to SpeedStep and thermal throttling.