6 ms·

This is only because the Rust implementations are using particularly slow code paths, either because SIMD/AVX optimizations requires a nightly compiler, some op

by mmstick 10y ago

This is only because the Rust implementations are using particularly slow code paths, either because SIMD/AVX optimizations requires a nightly compiler, some optimizations would require unsafe code, or that other languages are using particularly hacky code that would never fly in real world software. For example, many of the Java/C/C++ benchmarks are using custom optimizations that should be illegal for the benchmarking. Case in point, some are featuring custom hash maps that feature hashing algorithms that, while fast, would never be useful as they provide no protection against collisions. You'll see a hashing algorithm in a C preprocessor, for example, that just fakes having an actual algorithm whereas Rust examples are sticking to the tried and tested production-grade algorithms shipping in the standard library.

pvg 10y ago
Which ones do you have in mind? Preprocessor sounds a little cheaty but picking a hash function that's a better fit for the data is a pretty basic, practical sort of optimization.
- simcop2387 10y ago
  I think he's talking about the C code here, right at the top http://benchmarksgame.alioth.debian.org/u64q/program.php?test=knucleotide&lang=gcc&id=1 http://benchmarksgame.alioth.debian.org/u64q/program.php?tes...
  - pvg 10y ago
    If that's the one it's a custom hash function which is then inlined as a macro. That still seems like a pretty vanilla C optimization that one might write in real C code.
    - eridius 10y ago
      The problem isn't that it's a macro. It's that it's a horrifically bad hash function. It's blazing fast, but completely unsuitable for use in any real code.
      - pvg 10y ago
        If it's a hash function that only works for this exact data set, I can see the argument. If it's a hash function that works for this kind of data (these particularly formatted, particularly constrained strings), it's fair play. Which one is it? 'It's not a good general purpose hash function' alone doesn't seem like a valid criticism, especially along with 'the Rust version uses a general purpose hash function'. Nobody said you had to use a general purpose hash function. Imagine you got a million strings of variable length such that the last 4 bytes are guaranteed to be unique value between 0 and 999999. Lopping off the last four bytes is a perfectly good hash function for that kind of data.
        eridius 10y ago
        Sure, but it's not a good benchmark. If you can demonstrate that you have blazing fast performance, but only for the exact input data that the benchmark uses, then you haven't demonstrated anything at all beyond an ability to tailor your code to a very specific input. But in the real world, we don't get to write our code for a very specific input (after all, if we knew what the input was ahead of time, we could just pre-calculate the answers instead of publishing the program). So yeah, if you can come up with a tailored hash algorithm that works correctly and is DoS-immune for all possible input for the program, go ahead and use that tailored hash algorithm. But if your hash algorithm is only correct for the specific input data the benchmark game uses, and would not be collision-resistant under real world conditions, then you really shouldn't use that hash as it does not demonstrate anything useful.
        pvg 10y ago
        Well, here's the thing. I don't think it's for the exact input, it's for a type of inputs. Custom hash functions for specific types of data are a basic optimization technique and I find it odd you'd even suggest every hash function should be 'DoS-immune'. There's absolutely nothing 'real world' about this unless you think the world consists entirely of hostile inputs. In the real world, people absolutely optimize. Your argument seems to be that that's not the intent of the benchmark which may be true but it's not clear from the rules provided at all. To me, it looks like the opposite is true - they talk about using a standard hash table and most of those allow user-specified hash functions.
lossolo 10y ago
> Case in point, some are featuring custom hash maps that feature hashing algorithms that, while fast, would never be useful as they provide no protection against collisions. They are useful in this case, aren't they? Custom hash map implementation can be also useful in other scenarios. This is feature of the language that allow you to do that so it's not "illegal" to use it. If Rust doesn't allow you to do something while other language does, it doesn't mean it should be illegal. I don't see any tricks in Go and it's faster than Rust and more memory efficient in almost half of the benchmarks. So what's your excuse for those cases ?
- steveklabnik 10y ago
  Rust can do custom hashes as well, to be clear. There's been some arguments over what hashes are allowed in the game in the past.
  - fungos 10y ago
    If it compiles and run it is allowed. There is no rule to benchmark other than having the same end result with best performance. I still can't get all the moral over this. The best thing here for Rust would be to achieve the best performance on this game and nothing else. To me this game is important and winning on it is more yet. So, if Rust don't have better result by "morals" this is so wrong that hurts. But each time I read these responses I ask myself if there is not a real problem there. I like Rust, but performance is decisive. It is really sad to see that Rust keeps going down on this game and that makes me question myself when trying to invest time on Rust.
    - sanderjd 10y ago
      It isn't a "moral" issue. It is about the usefulness of the benchmarks. If you're using them to make a recommendation about which programming language to choose, it is important to have a sense for how well the results generalize to the kind of programs you will be writing. Using unrealistic hacks targeted specifically to each benchmark is contra that goal. If, on the other hand, you're just treating the benchmarks as a fun competition akin to code golf, then by all means, hacks ahoy!

by mmstick 10y ago

pvg 10y ago

Which ones do you have in mind? Preprocessor sounds a little cheaty but picking a hash function that's a better fit for the data is a pretty basic, practical sort of optimization.

simcop2387 10y ago
I think he's talking about the C code here, right at the top http://benchmarksgame.alioth.debian.org/u64q/program.php?test=knucleotide&lang=gcc&id=1 http://benchmarksgame.alioth.debian.org/u64q/program.php?tes...
- pvg 10y ago
  If that's the one it's a custom hash function which is then inlined as a macro. That still seems like a pretty vanilla C optimization that one might write in real C code.
  - eridius 10y ago
    The problem isn't that it's a macro. It's that it's a horrifically bad hash function. It's blazing fast, but completely unsuitable for use in any real code.
    - pvg 10y ago
      If it's a hash function that only works for this exact data set, I can see the argument. If it's a hash function that works for this kind of data (these particularly formatted, particularly constrained strings), it's fair play. Which one is it? 'It's not a good general purpose hash function' alone doesn't seem like a valid criticism, especially along with 'the Rust version uses a general purpose hash function'. Nobody said you had to use a general purpose hash function. Imagine you got a million strings of variable length such that the last 4 bytes are guaranteed to be unique value between 0 and 999999. Lopping off the last four bytes is a perfectly good hash function for that kind of data.
      - eridius 10y ago
        Sure, but it's not a good benchmark. If you can demonstrate that you have blazing fast performance, but only for the exact input data that the benchmark uses, then you haven't demonstrated anything at all beyond an ability to tailor your code to a very specific input. But in the real world, we don't get to write our code for a very specific input (after all, if we knew what the input was ahead of time, we could just pre-calculate the answers instead of publishing the program). So yeah, if you can come up with a tailored hash algorithm that works correctly and is DoS-immune for all possible input for the program, go ahead and use that tailored hash algorithm. But if your hash algorithm is only correct for the specific input data the benchmark game uses, and would not be collision-resistant under real world conditions, then you really shouldn't use that hash as it does not demonstrate anything useful.
        pvg 10y ago
        Well, here's the thing. I don't think it's for the exact input, it's for a type of inputs. Custom hash functions for specific types of data are a basic optimization technique and I find it odd you'd even suggest every hash function should be 'DoS-immune'. There's absolutely nothing 'real world' about this unless you think the world consists entirely of hostile inputs. In the real world, people absolutely optimize. Your argument seems to be that that's not the intent of the benchmark which may be true but it's not clear from the rules provided at all. To me, it looks like the opposite is true - they talk about using a standard hash table and most of those allow user-specified hash functions.

lossolo 10y ago

> Case in point, some are featuring custom hash maps that feature hashing algorithms that, while fast, would never be useful as they provide no protection against collisions. They are useful in this case, aren't they? Custom hash map implementation can be also useful in other scenarios. This is feature of the language that allow you to do that so it's not "illegal" to use it. If Rust doesn't allow you to do something while other language does, it doesn't mean it should be illegal. I don't see any tricks in Go and it's faster than Rust and more memory efficient in almost half of the benchmarks. So what's your excuse for those cases ?

steveklabnik 10y ago
Rust can do custom hashes as well, to be clear. There's been some arguments over what hashes are allowed in the game in the past.
- fungos 10y ago
  If it compiles and run it is allowed. There is no rule to benchmark other than having the same end result with best performance. I still can't get all the moral over this. The best thing here for Rust would be to achieve the best performance on this game and nothing else. To me this game is important and winning on it is more yet. So, if Rust don't have better result by "morals" this is so wrong that hurts. But each time I read these responses I ask myself if there is not a real problem there. I like Rust, but performance is decisive. It is really sad to see that Rust keeps going down on this game and that makes me question myself when trying to invest time on Rust.
  - sanderjd 10y ago
    It isn't a "moral" issue. It is about the usefulness of the benchmarks. If you're using them to make a recommendation about which programming language to choose, it is important to have a sense for how well the results generalize to the kind of programs you will be writing. Using unrealistic hacks targeted specifically to each benchmark is contra that goal. If, on the other hand, you're just treating the benchmarks as a fun competition akin to code golf, then by all means, hacks ahoy!