6 ms·

> Mold is already very fast, however it doesn't do incremental linking and the author has stated that they don't intend to. Wild doesn't do incremental linking

> Mold is already very fast, however it doesn't do incremental linking and the author has stated that they don't intend to. Wild doesn't do incremental linking yet, but that is the end-goal. By writing Wild in Rust, it's hoped that the complexity of incremental linking will be achievable. Can someone explain what is so special about Rust for this?

ComputerGuru 2y ago
[flagged]
- nicce 2y ago
  I guess they meant that why write in Rust.
  - ComputerGuru 2y ago
    You caught me as I was trying to delete/move the comment because I missed that.
dralley 2y ago
I assume they're referring to thread-safety and the ability to more aggressively parallelize.
- compiler-guy 2y ago
  Mold and lld are already very heavily parallelized. It’s one of the things that makes them very fast already.
senkora 2y ago
I assume that he is referring to "fearless concurrency", the idea that Rust makes it possible to write more complex concurrent programs than other languages because of the safety guarantees: https://doc.rust-lang.org/book/ch16-00-concurrency.html https://doc.rust-lang.org/book/ch16-00-concurrency.html So the logic would go: 1. mold doesn't do incremental linking because it is too complex to do it while still being fast (concurrent). 2. Rust makes it possible to write very complex fast (concurrent) programs. 3. A new linker written in Rust can do incremental linking while still being fast (concurrent). EDIT: I meant this originally, but comments were posted before I added it so I want to be clear that this part is new: (Any of those three could be false; I take no strong position on that. But I believe that this is the motivating logic.)
- compiler-guy 2y ago
  Both mold and lld are already very heavily concurrent. There is no fear at all there.
- ComputerGuru 2y ago
  Actually a lot of the hacks that mold uses to be the fastest linker would be, ironically, harder to reproduce with rust because they’re antithetical to its approach. Eg Mold intentionally eschews used resource collection to speed up execution (it’ll be cleaned up by the os when the process exits) while rust has a strong RAII approach here that would introduce slowdowns.
  - Philpax 2y ago
    I mean, that's pretty easy to do in Rust: https://doc.rust-lang.org/std/mem/struct.ManuallyDrop.html https://doc.rust-lang.org/std/mem/struct.ManuallyDrop.html Also see various arena allocator crates, etc.
    - ComputerGuru 2y ago
      Not really. You would have to either wrap any standard library types in newtypes with ManuallyDrop implemented or (for some) use a custom allocator. And if you want to free some things in one go but not others that gets much harder, especially when you look at how easy a language like zig makes it. And if you intentionally leak everything it is onerous to get the borrow checker to realize that unless you use a leaked box for all declaration/allocations, which introduces both friction and performance regressions (due to memory access patterns) because the use of custom allocators doesn’t factor into lifetime analysis. (Spoken as a die-hard rust dev that still thinks it’s the better language than zig for most everything.)
      - Aurornis 2y ago
        > You would have to either wrap any standard library types in newtypes with ManuallyDrop implemented ManuallyDrop would presumably be implemented on large data structures where it matters, not on every single type involved in the program.
devit 2y ago
It's feasible to write complex correct programs with optimal performance in Rust, unlike any other programming language (complex+correct is not feasible in C/C++/assembly/Zig/etc., optimal performance not possible in any other language).
compiler-guy 2y ago
That’s puzzling to me too. Rust is a great language, and probably makes developing Wild faster. But the complexity of incremental linking doesn’t stem from the linker’s implementation language. It stems from all the tracking, reserved spacing, and other issues required to link a previously linked binary (or at least parts of it) a second time.
- tialaramex 2y ago
  I would guess the idea is that in Rust the complexity is cheaper on a "per unit" basis so you can afford more complexity. So yes, it is a more complicated problem than the previous linkers, but, in Rust maybe you can get that done anyway.
- IshKebab 2y ago
  Rust allows your to enforce more invariants at compile time, so implementing a complex system where you are likely to make a mistake and violate those invariants is easier.
IshKebab 2y ago
There are two main factors: 1. Rust's well designed type system and borrow checker makes writing code that works just easier. It has the "if it compiles it works" property (not unique to Rust; people say this about e.g. Haskell too). 2. Rust's type system - especially its trait system can be used to enforce safety constraints statically. The obvious one is the Send and Sync traits for thread safety, but there are others, e.g. the Fuchsia network code statically guarantees deadlocks are impossible. Mold is written in C++ which is extremely error prone in comparison.
manoweb 2y ago
That is baffling. Maybe the author assumes that a language with many safeguards will lead to keeping complexity under control for a difficult task. By the way I had to lookup what incremental linking is, in practice I think it means that code from libraries and modules that have not changed won’t need to be re-packed each time which ch will save time for frequent development builds, it’s actually ingenious
deleted 2y ago
[deleted]
the_duke 2y ago
Rust has a pretty good incremental caching compiler that makes debug builds relatively fast. Linking is often a very notable bottleneck for debug binaries and mold can make a big difference. So interest in speeding up linking for Rust is expected.
wffurr 2y ago
I went looking for some writing by the author about how he made wild fast, but couldn't find much: https://davidlattimore.github.io/ https://davidlattimore.github.io/
panstromek 2y ago
Apart from what others said, maybe he plans to use Salsa or something like that. Rust has a few popular libraries for doing this.

ComputerGuru 2y ago

[flagged]

nicce 2y ago
I guess they meant that why write in Rust.
- ComputerGuru 2y ago
  You caught me as I was trying to delete/move the comment because I missed that.

dralley 2y ago

I assume they're referring to thread-safety and the ability to more aggressively parallelize.

compiler-guy 2y ago
Mold and lld are already very heavily parallelized. It’s one of the things that makes them very fast already.

senkora 2y ago

I assume that he is referring to "fearless concurrency", the idea that Rust makes it possible to write more complex concurrent programs than other languages because of the safety guarantees: https://doc.rust-lang.org/book/ch16-00-concurrency.html https://doc.rust-lang.org/book/ch16-00-concurrency.html So the logic would go: 1. mold doesn't do incremental linking because it is too complex to do it while still being fast (concurrent). 2. Rust makes it possible to write very complex fast (concurrent) programs. 3. A new linker written in Rust can do incremental linking while still being fast (concurrent). EDIT: I meant this originally, but comments were posted before I added it so I want to be clear that this part is new: (Any of those three could be false; I take no strong position on that. But I believe that this is the motivating logic.)

compiler-guy 2y ago
Both mold and lld are already very heavily concurrent. There is no fear at all there.
ComputerGuru 2y ago
Actually a lot of the hacks that mold uses to be the fastest linker would be, ironically, harder to reproduce with rust because they’re antithetical to its approach. Eg Mold intentionally eschews used resource collection to speed up execution (it’ll be cleaned up by the os when the process exits) while rust has a strong RAII approach here that would introduce slowdowns.
- Philpax 2y ago
  I mean, that's pretty easy to do in Rust: https://doc.rust-lang.org/std/mem/struct.ManuallyDrop.html https://doc.rust-lang.org/std/mem/struct.ManuallyDrop.html Also see various arena allocator crates, etc.
  - ComputerGuru 2y ago
    Not really. You would have to either wrap any standard library types in newtypes with ManuallyDrop implemented or (for some) use a custom allocator. And if you want to free some things in one go but not others that gets much harder, especially when you look at how easy a language like zig makes it. And if you intentionally leak everything it is onerous to get the borrow checker to realize that unless you use a leaked box for all declaration/allocations, which introduces both friction and performance regressions (due to memory access patterns) because the use of custom allocators doesn’t factor into lifetime analysis. (Spoken as a die-hard rust dev that still thinks it’s the better language than zig for most everything.)
    - Aurornis 2y ago
      > You would have to either wrap any standard library types in newtypes with ManuallyDrop implemented ManuallyDrop would presumably be implemented on large data structures where it matters, not on every single type involved in the program.

devit 2y ago

It's feasible to write complex correct programs with optimal performance in Rust, unlike any other programming language (complex+correct is not feasible in C/C++/assembly/Zig/etc., optimal performance not possible in any other language).

compiler-guy 2y ago

That’s puzzling to me too. Rust is a great language, and probably makes developing Wild faster. But the complexity of incremental linking doesn’t stem from the linker’s implementation language. It stems from all the tracking, reserved spacing, and other issues required to link a previously linked binary (or at least parts of it) a second time.

tialaramex 2y ago
I would guess the idea is that in Rust the complexity is cheaper on a "per unit" basis so you can afford more complexity. So yes, it is a more complicated problem than the previous linkers, but, in Rust maybe you can get that done anyway.
IshKebab 2y ago
Rust allows your to enforce more invariants at compile time, so implementing a complex system where you are likely to make a mistake and violate those invariants is easier.

IshKebab 2y ago

There are two main factors: 1. Rust's well designed type system and borrow checker makes writing code that works just easier. It has the "if it compiles it works" property (not unique to Rust; people say this about e.g. Haskell too). 2. Rust's type system - especially its trait system can be used to enforce safety constraints statically. The obvious one is the Send and Sync traits for thread safety, but there are others, e.g. the Fuchsia network code statically guarantees deadlocks are impossible. Mold is written in C++ which is extremely error prone in comparison.

manoweb 2y ago

That is baffling. Maybe the author assumes that a language with many safeguards will lead to keeping complexity under control for a difficult task. By the way I had to lookup what incremental linking is, in practice I think it means that code from libraries and modules that have not changed won’t need to be re-packed each time which ch will save time for frequent development builds, it’s actually ingenious

deleted 2y ago

[deleted]

the_duke 2y ago

Rust has a pretty good incremental caching compiler that makes debug builds relatively fast. Linking is often a very notable bottleneck for debug binaries and mold can make a big difference. So interest in speeding up linking for Rust is expected.

wffurr 2y ago

I went looking for some writing by the author about how he made wild fast, but couldn't find much: https://davidlattimore.github.io/ https://davidlattimore.github.io/

panstromek 2y ago

Apart from what others said, maybe he plans to use Salsa or something like that. Rust has a few popular libraries for doing this.