4 ms·

> No problem, we thought and instructed the Rust compiler to disable its own ThinLTO pass when compiling for the cross-language case and indeed everything was f

by bla3 7y ago

> No problem, we thought and instructed the Rust compiler to disable its own ThinLTO pass when compiling for the cross-language case and indeed everything was fine -- until the segmentation faults mysteriously returned a few weeks later even though ThinLTO was still disabled. [...] Since then ThinLTO is turned off for libstd by default. Instead of fixing the crash, they landed a workaround. > We learned that all LLVM versions involved really have to be a close match in order for things to work out. The Rust compiler's documentation now offers a compatibility table for the various versions of Rust and Clang. It's cool they got it working, but it sounds like this is currently proof-of-concept quality and not very productionized yet. To me, the overall tone of the article sounds like they ran into a bunch of issues and opted for duct tape instead of systemic fixes. Which is fine to get things off the ground, of course! But I hope they take the time to go back and fix the underlying issues they ran into too.

pornel 7y ago
The long-term plan for Rust is to treat its standard library (almost) like any other crate, so it could be recompiled with your custom compiler settings if necessary. However, libstd is by necessity tied quite closely to the compiler, it's one of the oldest and most fundamental parts of the stack, and there are tons of little complications around making it "just" a crate, so changes to libstd/rustc/Cargo necessary to make that possible will take longer. In the meantime, changing one problematic compiler flag seems like a sensible solution.
dblohm7 7y ago
The low-level tools team consistently reports bugs against the upstream projects. I have no doubt that they did so during the course of this project.
pcwalton 7y ago
This isn't anything new. Rust has had to land workarounds for lots of LLVM issues in its history. For example, Rust had to stop using noalias on function parameters because LLVM miscompiled too many functions with it, as Rust can use it way more than C/C++ do and therefore it didn't receive much upstream test coverage.
- bla3 7y ago
  Rust could fix upstream issues it runs into, no?
  - throwupaway123 7y ago
    They actually do sometimes AFAIK
  - bzbarsky 7y ago
    They could, and they do. That said, they don't have infinite time, and if, as in this case, the upstream fix would: (1) be pretty involved and (2) be very likely to get regressed because upstream doesn't have the capability to run tests that would prevent that (e.g. because upstream only runs C++ compilation tests and there is no way to exercise the relevant bugs via C++ code), then investing in fixing upstream may not be the right tradeoff. In theory, one could first change upstream's test harness to allow Rust code, but that involves upstream tests depending on the Rust compiler frontend, which apart from being a technical problem is probably a political one. Maybe it would have been possible to do upstream tests via bitcode source instead of Rust or C++; I don't know about LLVM to say offhand. But in either case this is not as easy as just "fix a simple upstream bug"...
    - pcwalton 7y ago
      Upstream tests are generally done at the LLVM IR level actually. It's mostly just a question of (1) time; (2) worries about ongoing maintenance work upstream; (3) a general feeling that perhaps such optimizations are best done on MIR anyway, because they'll be more effective there than they would be in LLVM.
      - kzrdude 7y ago
        Wouldn't MIR be more portable to the future as well? Building on Rust's own equity and all that, because future Rust will probably still use MIR but could replace LLVM(?)
aseipp 7y ago
It's just how it works out in practice, in my experience. LLVM is a large, fast moving target that's incredibly complex to understand, because it has a complex job. It resolves many issues for you when you're developing a compiler, but you are gifted issues in return. One of those is that understanding, diagnosing, and properly fixing problems can take a very large amount of work. It has bugs! I mean, LTO has historically been fragile for a single source language when enough code gets thrown at it, much less two languages! Another is that compiler developers often don't have infinite amounts of time to sort out shenanigans like this when they come up. Users generally prefer the compiler to work, even if suboptimally, when compared to "not working", so there is some tension between things and how long they take. So landing workarounds in various ways -- sending patches upstream, using custom builds with bespoke patches, code generation workarounds -- all have to happen on a case by case basis. Many LLVM clients do this to varying degrees, and a lot of features like LTO start off fragile due to these kinds of things. Over time things will get tightened up, hopefully. When I worked on GHC (admittedly several years ago now), the #1 class of problems for the LLVM backend were toolchain compatibility issues above all else, because we relied on the users to provision it. At the time it was nothing short of a nightmare -- various incompatible versions between various platforms causing compilation failures or outright miscompilation, requiring custom patches or backports at times (complicated by distros backporting -- or not backporting -- their own fixes), some platforms needed version X while others were better served by Y, flat-out code generation buts in GHC and LLVM, etc. It's all much better these days, and many features/fixes got landed upstream to make it all happen, but that's just how it works. Rust made several design choices with their packaging/LLVM use that we didn't that I think were the right ones, but I'm not surprised they've had a host of challenges of their own to address. TINSTAAFL.

by bla3 7y ago

pornel 7y ago

The long-term plan for Rust is to treat its standard library (almost) like any other crate, so it could be recompiled with your custom compiler settings if necessary. However, libstd is by necessity tied quite closely to the compiler, it's one of the oldest and most fundamental parts of the stack, and there are tons of little complications around making it "just" a crate, so changes to libstd/rustc/Cargo necessary to make that possible will take longer. In the meantime, changing one problematic compiler flag seems like a sensible solution.

dblohm7 7y ago

The low-level tools team consistently reports bugs against the upstream projects. I have no doubt that they did so during the course of this project.

pcwalton 7y ago

This isn't anything new. Rust has had to land workarounds for lots of LLVM issues in its history. For example, Rust had to stop using noalias on function parameters because LLVM miscompiled too many functions with it, as Rust can use it way more than C/C++ do and therefore it didn't receive much upstream test coverage.

bla3 7y ago
Rust could fix upstream issues it runs into, no?
- throwupaway123 7y ago
  They actually do sometimes AFAIK
- bzbarsky 7y ago
  They could, and they do. That said, they don't have infinite time, and if, as in this case, the upstream fix would: (1) be pretty involved and (2) be very likely to get regressed because upstream doesn't have the capability to run tests that would prevent that (e.g. because upstream only runs C++ compilation tests and there is no way to exercise the relevant bugs via C++ code), then investing in fixing upstream may not be the right tradeoff. In theory, one could first change upstream's test harness to allow Rust code, but that involves upstream tests depending on the Rust compiler frontend, which apart from being a technical problem is probably a political one. Maybe it would have been possible to do upstream tests via bitcode source instead of Rust or C++; I don't know about LLVM to say offhand. But in either case this is not as easy as just "fix a simple upstream bug"...
  - pcwalton 7y ago
    Upstream tests are generally done at the LLVM IR level actually. It's mostly just a question of (1) time; (2) worries about ongoing maintenance work upstream; (3) a general feeling that perhaps such optimizations are best done on MIR anyway, because they'll be more effective there than they would be in LLVM.
    - kzrdude 7y ago
      Wouldn't MIR be more portable to the future as well? Building on Rust's own equity and all that, because future Rust will probably still use MIR but could replace LLVM(?)

aseipp 7y ago

It's just how it works out in practice, in my experience. LLVM is a large, fast moving target that's incredibly complex to understand, because it has a complex job. It resolves many issues for you when you're developing a compiler, but you are gifted issues in return. One of those is that understanding, diagnosing, and properly fixing problems can take a very large amount of work. It has bugs! I mean, LTO has historically been fragile for a single source language when enough code gets thrown at it, much less two languages! Another is that compiler developers often don't have infinite amounts of time to sort out shenanigans like this when they come up. Users generally prefer the compiler to work, even if suboptimally, when compared to "not working", so there is some tension between things and how long they take. So landing workarounds in various ways -- sending patches upstream, using custom builds with bespoke patches, code generation workarounds -- all have to happen on a case by case basis. Many LLVM clients do this to varying degrees, and a lot of features like LTO start off fragile due to these kinds of things. Over time things will get tightened up, hopefully. When I worked on GHC (admittedly several years ago now), the #1 class of problems for the LLVM backend were toolchain compatibility issues above all else, because we relied on the users to provision it. At the time it was nothing short of a nightmare -- various incompatible versions between various platforms causing compilation failures or outright miscompilation, requiring custom patches or backports at times (complicated by distros backporting -- or not backporting -- their own fixes), some platforms needed version X while others were better served by Y, flat-out code generation buts in GHC and LLVM, etc. It's all much better these days, and many features/fixes got landed upstream to make it all happen, but that's just how it works. Rust made several design choices with their packaging/LLVM use that we didn't that I think were the right ones, but I'm not surprised they've had a host of challenges of their own to address. TINSTAAFL.