5 ms·

How often does benchmark code have a function that takes two pointers that could potentially alias each other? If it's as rare as I think it is, it might not ha

How often does benchmark code have a function that takes two pointers that could potentially alias each other? If it's as rare as I think it is, it might not have that much of an impact on Rust's position in the benchmarks game. Still, real world performance will probably benefit from this fix so it's a positive change regardless.

jerf 6y ago
I've been fairly convinced for a while that once Rust matures (which is probably fairly close to "now", but I've held this opinion for years) that it's going to have a performance advantage in real code that's going to be hard to capture in benchmarks, because it's easy in a small benchmark to be very careful and ensure that you don't have aliasing, avoid extra copies, etc. Where I expect Rust to really shine performance-wise is at the larger scale of real code, where it affords code that copies less often because the programmer isn't sure in this particular function whether or not they own this so they just have to take a copy, or the compiler can't work out aliasing, etc. Ensuring at scale that you don't take extra copies, or have an aliasing problem, or that you don't have to take copies of things just to "be sure" in multithreading situations, is hard, and drains a lot of performance.
- throwaway894345 6y ago
  I agree with this. Benchmark code differs from real code in in that it approximates the performance ceiling for a language implementation; it's not "ordinary code" or even "somewhat optimized" but usually the most optimal code one can conceive of with little respect paid to competing concerns, like maintainability. Rust aspires to make idiomatic, maintainable code almost as performant as benchmark code by way of zero-cost abstractions; probably more so than any other language, including C and C++, and all the while keeping the ceiling high relative to other languages. Unfortunately, I'm still of the opinion that "idiomatic Rust" is quite a lot harder to write than idiomatic Go or C# or etc, and many applications absolutely index on developer velocity and performance and quality are "nice-to-haves". Many companies are making money hand over fist with Python and JavaScript, and Go and C# are quite a lot more performant and typesafe than those languages already; Rust is better still in these areas, but returns diminish. If C# and Go preclude 95% of the errors found in Python and JS, it's not worth trading tens or hundreds of percents of developer velocity for that extra 4% or so improvement in quality (a Go developer could recoup a bit more of that 4% by writing tests in the time they save over Rust). Of course, this is all subjective, and I've met smart people who argue with a straight face that Rust is better for developer velocity than languages like Go, so YMMV.
  - littlestymaar 6y ago
    > If C# and Go preclude 95% of the errors found in Python and JS Well Go and C#[1] still suffer from the billion dollar mistake (null pointers), which represents at least 1/3 of errors I've witnessed in JavaScript code, so I'd say they at best removes 70% of errors. And there's also logic errors, for which neither Go's or C#'s type system helps either, so maybe we're at 50% error reductions with Go and C# compared to JS and Python. I don't even think that Rust, with its really helpfull type system (ADT + Affine types) achieves 95% error reductions though. [1]: for C#, at least last time I used it, which was 2011
    - roca 6y ago
      Newtypes are also pretty great for detecting logic errors. (You can use them in Go and C# but they're not idiomatic and usually not zero-cost so people usually don't.)
      - throwaway894345 6y ago
        Newtypes are pretty idiomatic in Go (e.g., https://golang.org/src/os/types.go#L35 https://golang.org/src/os/types.go#L35), and they're so inexpensive that cost isn't a reason to forego them 99% of the time.
        roca 6y ago
        Sorry, my mistake, thanks for the correction.
      - pjmlp 6y ago
        Besides the sibling answer about Go, when you are in .NET world there are also F# and C++/CLI to chose from, so C# doesn't stand alone (leaving VB.NET out as it appeals other kind of developers).
    - Hawxy 6y ago
      > for C#, at least last time I used it, which was 2011 Since then, C# added nullable reference types: https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/builtin-types/nullable-reference-types https://docs.microsoft.com/en-us/dotnet/csharp/language-refe... In projects with complete NRT coverage, it's extremely rare I ever get a null ref, as the compiler will warn/error me if I'm using a possibly null value in an unsafe way.
- roca 6y ago
  I agree that the aliasing control and other features of Rust enable all kinds of interesting optimizations that currently aren't being done, and 'restrict' is only part of it. For example we also know that the data behind a shared reference is really immutable (apart from UnsafeCell etc). Unfortunately we may need an entirely new generation of compiler infrastructure to exploit these opportunities because LLVM is really a C/C++ compiler at heart and may not be happy about taking big changes that can't benefit C/C++.
- slaymaker1907 6y ago
  One big advantage when writing average code is that while Rust emphasizes generics and makes them easy to write, C++ makes it so difficult that you try and avoid it at all costs. Big, complex projects are going to probably use polymorphism all over the place in my experience which is not really captured by benchmarks. However, one advantage for C and C++ for such projects is that they make it far more pleasant to do bulk allocations. With C, you just need to group your mallocs and cast the memory and C++ offers placement new where you pass in the memory explicitly. With Rust, you need to reach into unsafe Rust (rightfully so), but unsafe Rust is very unpleasant to write.
  - millstone 6y ago
    I don't agree with the first paragraph at all. C++ templates are more powerful than Rust generics, and also easier to write, because they are not inherently type-checked. In practice templates are embraced and used extensively, not "avoided at all costs" (like C++ exceptions.)
    - fnord123 6y ago
      >C++ templates are more powerful than Rust generics, and also easier to write They are definitely not easier to write. The last time I checked, the error reporting were still so intractable that would be hard to take this argument in good faith. Maybe it improved? >In practice templates are embraced and used extensively, not "avoided at all costs" (like C++ exceptions.) I think many would agree that this is largely because the package management system for C++ is so awful that header-only libraries are popular since you just need to copy paste a single file into your project. C++ is a large language and there are some wildly different coding standards. Some do avoid templates (but you're right that GP overstated this). Why? Templates introduce a lot of code bloat because there is little opportunity for eliding copies of the same code at link time. Aside from bloating the global offset table in position independent executables (if the GOT doesn't fit in cache, perf drops), the compile time hit is also a non trivial issue. But this is a really exceptional scenario where you are shaving the monolith and need to get the ELF executable down (e.g. server side ball-of-mud, small embedded systems).
- millstone 6y ago
  Maybe but there's reasons to doubt this. For starters, it's not obvious that Rust copies less. Sometimes you copy in Rust because it's easier than threading lifetimes throughout; sometimes you copy because you are defending against an imaginary threat (consider std::env::var or std::env::args). One potentially important difference is integer indexes vs pointers. Compared to C, Rust uses int indexes less often within a function (iterators, not for loops) but more often between functions: in C you might store a pointer somewhere, while in Rust it's easier to store an index into a collection. But a ton of work has been poured into LLVM's induction variable analysis so this is swimming upstream. The aliasing could be really important; there's a ton of pessimizations for e.g. uint8 pointers. Part of the maturing is fully specifying Rust's UB and aliasing model.
- ashtonkem 6y ago
  Honestly, the biggest thing that attracts me to Rust is memory consumption. Most of my work is IO bound, usually on IO latency and not throughput, so the ability to pack more instances per container/vm/server/whatever would have a significant effect on my final bill per measurable unit.
kolbe 6y ago
The problem is that it’s “rare” and not “impossible.” Compilers have to be logically sound, not probabilistic when it comes to defined behaviors.
loeg 6y ago
Benchmark hackers can just sprinkle `restrict` all over the place, too. Real world C code doesn't get restrict by default and often isn't fully annotated, so the benchmarks may artificially hide the real-world difference.
volta87 6y ago
> How often does benchmark code have a function that takes two pointers that could potentially alias each other? If it's as rare as I think it is, it might not have that much of an impact on Rust's position in the benchmarks game. Most C++ STL algorithms take iterators, that in many cases are just pointers in benchmarks because these often only deal with arrays. Hell most C standard APIs (e.g. memcpy) take multiple pointers.

jerf 6y ago

I've been fairly convinced for a while that once Rust matures (which is probably fairly close to "now", but I've held this opinion for years) that it's going to have a performance advantage in real code that's going to be hard to capture in benchmarks, because it's easy in a small benchmark to be very careful and ensure that you don't have aliasing, avoid extra copies, etc. Where I expect Rust to really shine performance-wise is at the larger scale of real code, where it affords code that copies less often because the programmer isn't sure in this particular function whether or not they own this so they just have to take a copy, or the compiler can't work out aliasing, etc. Ensuring at scale that you don't take extra copies, or have an aliasing problem, or that you don't have to take copies of things just to "be sure" in multithreading situations, is hard, and drains a lot of performance.

throwaway894345 6y ago
I agree with this. Benchmark code differs from real code in in that it approximates the performance ceiling for a language implementation; it's not "ordinary code" or even "somewhat optimized" but usually the most optimal code one can conceive of with little respect paid to competing concerns, like maintainability. Rust aspires to make idiomatic, maintainable code almost as performant as benchmark code by way of zero-cost abstractions; probably more so than any other language, including C and C++, and all the while keeping the ceiling high relative to other languages. Unfortunately, I'm still of the opinion that "idiomatic Rust" is quite a lot harder to write than idiomatic Go or C# or etc, and many applications absolutely index on developer velocity and performance and quality are "nice-to-haves". Many companies are making money hand over fist with Python and JavaScript, and Go and C# are quite a lot more performant and typesafe than those languages already; Rust is better still in these areas, but returns diminish. If C# and Go preclude 95% of the errors found in Python and JS, it's not worth trading tens or hundreds of percents of developer velocity for that extra 4% or so improvement in quality (a Go developer could recoup a bit more of that 4% by writing tests in the time they save over Rust). Of course, this is all subjective, and I've met smart people who argue with a straight face that Rust is better for developer velocity than languages like Go, so YMMV.
- littlestymaar 6y ago
  > If C# and Go preclude 95% of the errors found in Python and JS Well Go and C#[1] still suffer from the billion dollar mistake (null pointers), which represents at least 1/3 of errors I've witnessed in JavaScript code, so I'd say they at best removes 70% of errors. And there's also logic errors, for which neither Go's or C#'s type system helps either, so maybe we're at 50% error reductions with Go and C# compared to JS and Python. I don't even think that Rust, with its really helpfull type system (ADT + Affine types) achieves 95% error reductions though. [1]: for C#, at least last time I used it, which was 2011
  - roca 6y ago
    Newtypes are also pretty great for detecting logic errors. (You can use them in Go and C# but they're not idiomatic and usually not zero-cost so people usually don't.)
    - throwaway894345 6y ago
      Newtypes are pretty idiomatic in Go (e.g., https://golang.org/src/os/types.go#L35 https://golang.org/src/os/types.go#L35), and they're so inexpensive that cost isn't a reason to forego them 99% of the time.
      - roca 6y ago
        Sorry, my mistake, thanks for the correction.
    - pjmlp 6y ago
      Besides the sibling answer about Go, when you are in .NET world there are also F# and C++/CLI to chose from, so C# doesn't stand alone (leaving VB.NET out as it appeals other kind of developers).
  - Hawxy 6y ago
    > for C#, at least last time I used it, which was 2011 Since then, C# added nullable reference types: https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/builtin-types/nullable-reference-types https://docs.microsoft.com/en-us/dotnet/csharp/language-refe... In projects with complete NRT coverage, it's extremely rare I ever get a null ref, as the compiler will warn/error me if I'm using a possibly null value in an unsafe way.
roca 6y ago
I agree that the aliasing control and other features of Rust enable all kinds of interesting optimizations that currently aren't being done, and 'restrict' is only part of it. For example we also know that the data behind a shared reference is really immutable (apart from UnsafeCell etc). Unfortunately we may need an entirely new generation of compiler infrastructure to exploit these opportunities because LLVM is really a C/C++ compiler at heart and may not be happy about taking big changes that can't benefit C/C++.
slaymaker1907 6y ago
One big advantage when writing average code is that while Rust emphasizes generics and makes them easy to write, C++ makes it so difficult that you try and avoid it at all costs. Big, complex projects are going to probably use polymorphism all over the place in my experience which is not really captured by benchmarks. However, one advantage for C and C++ for such projects is that they make it far more pleasant to do bulk allocations. With C, you just need to group your mallocs and cast the memory and C++ offers placement new where you pass in the memory explicitly. With Rust, you need to reach into unsafe Rust (rightfully so), but unsafe Rust is very unpleasant to write.
- millstone 6y ago
  I don't agree with the first paragraph at all. C++ templates are more powerful than Rust generics, and also easier to write, because they are not inherently type-checked. In practice templates are embraced and used extensively, not "avoided at all costs" (like C++ exceptions.)
  - fnord123 6y ago
    >C++ templates are more powerful than Rust generics, and also easier to write They are definitely not easier to write. The last time I checked, the error reporting were still so intractable that would be hard to take this argument in good faith. Maybe it improved? >In practice templates are embraced and used extensively, not "avoided at all costs" (like C++ exceptions.) I think many would agree that this is largely because the package management system for C++ is so awful that header-only libraries are popular since you just need to copy paste a single file into your project. C++ is a large language and there are some wildly different coding standards. Some do avoid templates (but you're right that GP overstated this). Why? Templates introduce a lot of code bloat because there is little opportunity for eliding copies of the same code at link time. Aside from bloating the global offset table in position independent executables (if the GOT doesn't fit in cache, perf drops), the compile time hit is also a non trivial issue. But this is a really exceptional scenario where you are shaving the monolith and need to get the ELF executable down (e.g. server side ball-of-mud, small embedded systems).
millstone 6y ago
Maybe but there's reasons to doubt this. For starters, it's not obvious that Rust copies less. Sometimes you copy in Rust because it's easier than threading lifetimes throughout; sometimes you copy because you are defending against an imaginary threat (consider std::env::var or std::env::args). One potentially important difference is integer indexes vs pointers. Compared to C, Rust uses int indexes less often within a function (iterators, not for loops) but more often between functions: in C you might store a pointer somewhere, while in Rust it's easier to store an index into a collection. But a ton of work has been poured into LLVM's induction variable analysis so this is swimming upstream. The aliasing could be really important; there's a ton of pessimizations for e.g. uint8 pointers. Part of the maturing is fully specifying Rust's UB and aliasing model.
ashtonkem 6y ago
Honestly, the biggest thing that attracts me to Rust is memory consumption. Most of my work is IO bound, usually on IO latency and not throughput, so the ability to pack more instances per container/vm/server/whatever would have a significant effect on my final bill per measurable unit.

kolbe 6y ago

The problem is that it’s “rare” and not “impossible.” Compilers have to be logically sound, not probabilistic when it comes to defined behaviors.

loeg 6y ago

Benchmark hackers can just sprinkle `restrict` all over the place, too. Real world C code doesn't get restrict by default and often isn't fully annotated, so the benchmarks may artificially hide the real-world difference.

volta87 6y ago

> How often does benchmark code have a function that takes two pointers that could potentially alias each other? If it's as rare as I think it is, it might not have that much of an impact on Rust's position in the benchmarks game. Most C++ STL algorithms take iterators, that in many cases are just pointers in benchmarks because these often only deal with arrays. Hell most C standard APIs (e.g. memcpy) take multiple pointers.