3 ms·

Reproducibility is at odds with Profile-Guided-Optimization. Especially on anything that involves networking and other IO that isn't consistent.

• michaelt 2y ago
  Why should it be? Does the profiler not output a hprof file or whatever, which is the input to the compiler making the release binary? Why not just store that?
• gnulinux 2y ago
  It's not at odds at all but it'll be "Monadic" in the sense that the output of system A will be part of the input to system A+1 which is complicated to organize in a systems setting, especially if you don't have access to a language that can verify. But it's absolutely achievable if you do have such a tool, e.g. you can do this in nix.
• zbobet2012 2y ago
  That's only the case if you did PGO with "live" data instead of replays from captured runs, which is best practice afaik.
• nrvn 2y ago
  from Go documentation[0]: > Committing profiles directly in the source repository is recommended as profiles are an input to the build important for reproducible (and performant!) builds. Storing alongside the source simplifies the build experience as there are no additional steps to get the profile beyond fetching the source. I very much hope other languages/frameworks can do the same. [0]: https://go.dev/doc/pgo#building https://go.dev/doc/pgo#building
  • nyrikki 2y ago
    The Performant claim there is counter to research I have heard. Plus as the PGO profile data is non-deterministic in most cases, even when compiled on the same hardware as the end machine "Committing profiles directly in the source repository" is the reason why they are deleted or at least excluded from the comparison. A quote from the paper that I remember on the subject[1] as these profiles are just about as machine dependent as you can get. > Unfortunately, most code improvements are not machine independent, and the few that truly are machine independent interact with those that are machine dependent causing phase-ordering problems. Hence, effectively there are no machine-independent code improvements. There were some differences between various Xeon chip's implementations of the same or neighboring generations that I personally ran into when we tried to copy profiles to avoid the cost of the profile runs that may make me a bit more sensitive to this, but I personally saw huge drops in performance well into the double digits that threw off our regression testing. IMHO this is exactly why your link suggested the following: > Your production environment is the best source of representative profiles for your application, as described in Collecting profiles. That is very different from Fedora using some random or generic profile for x86_64, which may or may not match the end users specific profile. [1] https://dl.acm.org/doi/10.5555/184716.184723 https://dl.acm.org/doi/10.5555/184716.184723
    • clhodapp 2y ago
      If those differences matter so much for your workloads, treat your different machine types as different different architectures, commit profiling data for all of them and (deterministically) compile individual builds for all of them. Fedora upstream was never going to do that for you anyway (way too many possible hardware configurations), so you were already going be in the business of setting that up for yourself.
• nyrikki 2y ago
  This is one of the "costs" of reproducible builds, just like the requirement to use pre-configured seeds for pseudo random number generators etc. It does hit real projects and may be part of the reason that "99%" is called out but Fedora also mentions that they can't match the official reproducible-builds.org meaning in the above just due to how RPMs work, so we will see what other constraints they have to loosen. Here is one example of where suse had to re-enable it for gzip. https://build.opensuse.org/request/show/499887 https://build.opensuse.org/request/show/499887 Here is a thread on PGO from the reproducible-builds mail list. https://lists.reproducible-builds.org/pipermail/rb-general/2022-June/002633.html https://lists.reproducible-builds.org/pipermail/rb-general/2... There are other costs like needing to get rid of parallel builds for some projects that make many people loosen the official constraints. The value of PGO+LTO being one. gcda profiles are unreproducible, but the code they produce is typically the same. If you look into the pipeline of some projects, they just delete the gcda output and then often try a rebuild if the code is different or other methods. While there are no ideal solutions, one that seems to work fairly well, assuming the upstream is doing reproducible builds, is to vendor the code, build a reproducible build to validate that vendored code, then enable optimizations. But I get that not everyone agrees that the value of reproducibility is primarily avoiding attacks on build infrastructure. However reproducible builds as nothing to do with MSO model checking etc... like some have claimed. Much of it is just deleting non-deterministic data as you can see here with debian, which fedora copied. https://salsa.debian.org/reproducible-builds/strip-nondeterminism https://salsa.debian.org/reproducible-builds/strip-nondeterm... As increasing the granularity of address-space randomization at compile and link time is easier than at the start of program execution, obviously there will be a cost (that is more than paid for by reducing supply chain risks IMHO) of reduced entropy for address randomization and thus does increase the risk of ROP style attacks. Regaining that entropy at compile and link time, if it is practical to recompile packages or vendor, may be worth the effort in some situations, probably best to do real PGO at that time too IMHO.
  • goodpoint 2y ago
    Yo, the attacker has access to the same binaries, so only runtime address randomization is useful.