15 ms·

Rust in Linux will be fantastic except for compile time. Rust (and the world) needs a Manhattan Project to build a fast Rust compiler (where by "fast" I mean bo

by roca 6y ago

Rust in Linux will be fantastic except for compile time. Rust (and the world) needs a Manhattan Project to build a fast Rust compiler (where by "fast" I mean both efficient and scalably parallel when compiling a workspace with many crates and long dependency chains). To put this in perspective, though: increasing the number of people paid to work on the Rust compiler by 10x would only mean hiring about 25 people. Compared to the size of the projects that are starting to depend on Rust, that's a rounding error.

crb002 6y ago
https://github.com/StanfordSNR/gg https://github.com/StanfordSNR/gg - should be almost the same as the GCC/LLVM thunk extractor. You have to pay for the borrow checker, but LLVM IR optimization passes should be the same complexity.
Ericson2314 6y ago
Replace "faster" with "more incremental". Trying to get a program to quickly do the exact same thing over and over again is a colossal waste of resource.
- hinkley 6y ago
  When you’re trying to balance a bad budget, you can’t dismiss many of the cost centers and still succeed. You can prioritize, but that makes the rest a matter of “when” not if. It can be hard to tell from the outside if someone is avoiding a problem because they don’t want to solve it, or because they are hoping for a more inspired solution. Tackling a problem when all the options are bad may block another alternative when it does finally surface. I’d lump this in with irreversible decisions and the advice to delay them to the last responsible moment. Who know what internal plumbing has to be moved around to make the compiler more incremental, parallel, or both.
  - Ericson2314 6y ago
    Yes, let's keep on all options on the table. But trying to constant-factor optomize O(code I didn't change) is a ridiculous approach that needs justification, and the burden of proof is on those that advocate it. Yet, this seem to be the default thing people beg for. Switching to incremental is hard programming, and that's why it hasn't happened overnight, but at least it is an algorithmically sane approach.
darksaints 6y ago
I know a lot of people will disagree with me, and I know that the rust compiler has room for improvement, but compile time isn't really that bad for rust when you consider what it actually does for you. Assuming no unsafe blocks, rust's type system eliminates entire classes of memory safety errors. It fundamentally eliminates data races. It eliminates large swaths of logical errors, due to its use of algebraic data types, and strong type enforcement. No null pointers, explicit error handling, and no undefined behavior. It boggles my mind that people will balk at compile times for rust, but then not even bat an eye at their C++ testkit that runs for 8+ hours, primarily checking for errors that rust would never let you make in the first place.
- benesch 6y ago
  Part of what makes rustc's slow compile times so frustrating is that the slow compile times are mostly unrelated to all of the fancy compile-time analysis you listed. You can see this in action if you run `cargo check`. That will run all the special static analysis (typeck, borrowck, dropck, etc.) and complete in a very reasonable amount of time. To a first approximation, rustc is slow because it generates terrible LLVM IR, and relies on a number of slow LLVM optimization passes to smooth that IR into something reasonable. As far as I can tell this is entirely a result of not having the necessary engineering resources to throw at the problem; Mozilla has a limited budget for Rust, and needs to spread that budget across not just the compiler toolchain, but the language and ecosystem. Brian Anderson's been working on a blog post series that covers Rust's slow compile times in more detail, if you're curious: https://pingcap.com/blog/tag/Rust https://pingcap.com/blog/tag/Rust
  - darksaints 6y ago
    Yeah, I realize that Rust has a lot of room for improvement. I'm just saying that compared to the alternative, you get a relatively inefficient compiler that, despite its inefficiency, still gives you a more efficient development process. You get instant feedback, which you can't get from a testkit, and you can eliminate massive amounts of testing. If the focus is entirely on reducing compile times, it can very quickly lead down a path that limits runtime performance. There is so much potential for increased runtime performance in strongly typed languages, due to the guarantees made by the type system. It allows for analysis and reasoning that not just enables micro-optimizations but also macro-optimizations across entire codebases. Entire theses have been written describing optimizations that rely on immutability alone, and we haven't even touched so many other possibilities that rust's design enables (such as the ownership system). I'm fairly familiar with the project to work on the LLVM-IR code emitter. If the focus is on reducing the complexity and cruft of the LLVM-IR emitted, that's a win all around (because you get faster code and faster compile times). I just hope that reducing compile times doesn't become the core objective, because that eliminates a lot of future potential.
zoul 6y ago
How much of the needed speedup is engineering and how much is research?
- roca 6y ago
  Depends on what you mean by "research". If you mean "problems with a high degree of novelty" then quite a lot of research, I think. Rust is unusual: it requires a lot of tricky type inference and static analysis from the compiler, and doesn't make developers structure their code with header files etc to make separate compilation easier; BUT unlike most languages with those properties, people are writing large real-world projects in it.
  - pornel 6y ago
    My understanding is that Rust's slowness is mainly from code generation and optimization. `cargo check` performs full parsing and type-checking, and runs reasonably quickly. The problem starts when Rust throws a ton of IR at LLVM. The current focus is on adding optimizations in Rust at higher level (MIR) so that it would cause less work for LLVM.
    - acqq 6y ago
      Imagine you have 16 cores available. Allowing parallel compilation where it is sequential at the moment, you get 16 times speedup on such a machine. 32 cores machine -- 32 times speedup etc. Or even 10 such machines -- 320 times speedup. On the other side, imagine you can even reduce the "ton of IR" tenfold but if the compilation is sequential, the tenfold speedup is fixed even if you have 32 or 320 cores. In general, allowing for parallel compilation is much more important topic, and it's worth even redesigning language to make that more achievable. Which also doesn't mean that producing "ton of IR" is a good thing. But working on allowing maximum parallelism is more fundamental. Also, discovering algorithms to allow all compiler passes as effective as possible is important for something to reach wider usability. But Rust must also get to achieve its promises of being "safer" in the scenarios where C is otherwise used, and I time and again see a lot of "unsafe" keywords in the sources wherever I see some implementation of anything fundamental.
the8472 6y ago
I assume dependency trees in the kernel would be much more shallow.
- roca 6y ago
  That might be true for drivers. Outside the kernel, I don't think it's going to be true for Android or Windows.
  - stjohnswarts 6y ago
    But this discussion is mostly about the kernel, isn't it? I'm not sure why throwing out rust weaknesses in general is appropriate when we're talking about a specific use case such as kernel drivers and other modules which do tend to be quite a bit more shallow than most large c/c++ projects I've been on.
    - roca 6y ago
      That's a fair point. I do think it's appropriate to point out that build times are a problem for a lot of projects and may affect the kernel so careful attention needs to be paid to that issue.
cxr 6y ago
> a Manhattan Project to build a fast Rust compiler Or work on getting what Rust gives you in a different language that isn't Rust (maybe entirely new, maybe not[1]). One underdiscussed thing about Rust is that it's so clearly a byproduct of the C++ world. And the most disappointing thing about Rust is that it probably won't ever be seen as a prototype that should be thrown away. 1. http://mbeddr.com/ http://mbeddr.com/
- nilkn 6y ago
  Rust is actually a byproduct of OCaml and to a lesser extent Haskell. This becomes obvious the more you use it, especially if you’ve written substantial code in either of those languages before. To really drive this home, I’d wager that you have a better chance of translating OCaml or non-fancy Haskell straight into Rust than you do C++.
  - gnuvince 6y ago
    > I’d wager that you have a better chance of translating OCaml or non-fancy Haskell straight into Rust than you do C++. I found that translating OCaml code to Rust is frustrating, because you cannot easily and cheaply replicate the functional approach. It's much more Rustic to take an object by mutable reference and modify it in-situ than it is to take an immutable reference and return a new object. The reason is that in OCaml, you have structural sharing, i.e., multiple pointers pointing into the same data structure, which is a big no-no in Rust, so if you want to replicate the functional OCaml code, you begin allocating a new, complete data structure every time, and that's costly.
    - nilkn 6y ago
      Well, of course. Rust is not designed around immutable data structures. There are going to be differences because it's simply not the same language. This [0] example, written by a notable Haskeller, illustrates what I mean pretty well and also touches on what you just pointed out. It's also very consistent with my own experience. [0]: https://donsbot.wordpress.com/2020/07/04/back-to-old-tricks-or-baby-steps-in-rust/ https://donsbot.wordpress.com/2020/07/04/back-to-old-tricks-...
mekster 6y ago
> increasing the number of people paid to work on the Rust compiler by 10x would only mean hiring about 25 people. Compared to the size of the projects that are starting to depend on Rust, that's a rounding error. And are you going to pay for the "rounding error" or just expecting someone to pay for millions a year for your idea?
- roca 6y ago
  It's not my idea. This is well-known. As for who pays for it: that's a tricky issue! But if we get into a situation where (for example) 10,000 developers spend an hour a day waiting for builds, we know 50 developers could fix the issue in a year, but that doesn't happen because we can't figure out how to structure it economically --- that would be a failure of organizational imagination.
- pchanda 6y ago
  I think OP meant that companies, like Amazon and MS, whose projects are starting to depend on rust should be putting some resources. Although to be fair, Microsoft and Amazon do pay for their infrastructure if I am not wrong.
  - roca 6y ago
    Right on both counts.
- deleted 6y ago
  [deleted]
- jodrellblank 6y ago
  This is such a low effort sneer-comment, it's a dismissive way to attack and putdown any idea that includes any amount of payment. It's the discussion equivalent of "whoever smelt it, dealt it" - if someone mentions an idea which costs money, demand as a first response if they are going to pay for it (assuming they aren't, instant dismissal), or accuse that the alternative is a kind of entitled and unfair expectation held of others, when neither need be the case. "That volcano looks like it's becoming active, maybe some sensors could give us an early warning of problems" - Are YOU going to pay for your little "idea"??? Then WHO IS? "That tree is getting dangeoursly high, it's at risk of falling down in a big storm this winter, maybe we could get it cut down before that" - Are YOU going to pay for it?! "Dumping raw sewage in the river is making people sick, treating it first would take a small amount of space and a small fraction of the council's existing budget" - and you want to TAX ME for YOUR clean river, I suppose?! As if OP is the only person who would benefit, as if automatically assuming the worst possible intent for who would pay for it, and as if "who would pay" is the first and only thing worth demanding an answer to, before even having a discussion about whether it's worth doing at all - and which of many ways it might be done.
Kenji 6y ago
That Rust compiles slowly is a meme at this point. C++ isn't faster. Nobody's complaining that C++ isn't viable until compilers get faster.
coldpie 6y ago
> increasing the number of people paid to work on the Rust compiler by 10x would only mean hiring about 25 people I don't think there are 25 people qualified to do that work looking for jobs. It's not something you can just throw money at, you need really qualified people to do that kind of work. Those people are in very high demand.
- roca 6y ago
  There are a lot of people out there who can do this kind of work. I know a lot of such people. Many of them don't have traditional CVs. Yes, most of them are employed. Some are under-employed. Some are well employed but might be keen to work on a project this important.
- steveklabnik 6y ago
  Note that that would be increasing the number for pay, you already have many people who have the relevant qualifications because they’re already doing the work, just only in their spare time. Second, the compiler team has put a ton of effort into mentorship infrastructure, and you can go from not knowing a ton to being a productive member of the team with their help. It "just" takes time and desire to do it.
  - hinkley 6y ago
    Or on some other project. Sure, there are probably some projects for which if the primary author leaves, their coworkers will drop it or rewrite it in another language, but for others the management will realize what a liability it was to have all of their eggs in that one basket and they’ll have to hire 2 to replace them. If you steal 25 to work on the kernel, you will likely see 25 open reqs to replace them. And more people will work with languages that they have seen reqs for.
  - darksaints 6y ago
    Interesting. What sort of mentorship infrastructure? What sort of people are they targeting for mentorship? I'd love to learn about compilers by working on an actual real world compiler, but I really only know high level details about how different components fit together. I also don't have a formal CS education, although I am currently working as a senior level data engineer. Not sure if that is the sort of person they are looking to mentor.
crazypython 6y ago
GCC 10 supports Dlang directly. (The support is still immature and it's being updated to the newer version of Dlang.) Dlang compiles quickly not because the language is simple, but because the compiler authors care about compilation speed: Lexer skips four spaces at once https://github.com/dlang/dmd/pull/11095 https://github.com/dlang/dmd/pull/11095 Optimize core logic to 7x compilation speed of language feature https://github.com/dlang/dmd/pull/11303 https://github.com/dlang/dmd/pull/11303 Cache rarely used types to improve memory locality https://github.com/dlang/dmd/pull/11363 https://github.com/dlang/dmd/pull/11363
- PudgePacket 6y ago
  Slightly off topic, I see D brought up incredibly consistently when Rust is mentioned on HN.. Is it just me?? (it's usually by walter themself but I digress..)
  - Keyframe 6y ago
    A number of people (including myself) migrated from what was known as D1 onto Rust. Steve (Klabnik) was also one of the refugees. I can't speak for others, but Rust is kind of what I was expecting for D1 to go to.
    - aberba 6y ago
      Wow, didn't know Steve was rocking D.
  - pjmlp 6y ago
    Some of us believe in systems programming languages with some form of GC, so it usually pops up as alternative. For me D is what .NET (thus C#) should have been if the Ext-VOS project (COM Runtime) decided to go native, instead of embracing J++ and then turning into .NET due to the lawsuit. But unfortunately D is a tiny community that gets by with small contributions and herding cats is hard.
- VWWHFSfQ 6y ago
  > but because the compiler authors care about compilation speed Is this meant to imply that the Rust compiler authors don't care about compilation speed?
rumanator 6y ago
> Rust in Linux will be fantastic except for compile time. Rust (and the world) needs a Manhattan Project to build a fast Rust compiler (where by "fast" I mean both efficient and scalably parallel when compiling a workspace with many crates and long dependency chains). A basic technique to parallelize sections is to split a project into modules. Is this not an option for the linux kernel?
- roca 6y ago
  Rust has great support for modules (crates in Rust lingo). It can compile modules in parallel when they don't depend on each other. The problem is that when you have module A depending on module B, rustc doesn't do a good job of compiling A and B in parallel. In contrast, in C/C++ you can handcraft your header files to ensure that A and B compile in parallel.
  - pjmlp 6y ago
    In C and C++ it is also quite common to use binary dependencies, something that cargo still doesn't do. However C++ is also now looking at parallelization opportunities for modules, which is a similar situation as crates.
    - fluffything 6y ago
      Every time you compile a rust project after an initial compile, cargo does use "binary" dependencies (bitcode dependencies). It is much better at this than C++. For example, change a trait in your Rust library, and recompile the tests, and compare that with changing a type in, say, range-v3 (or any other C++ header-only library). In C++, recompiles take pretty much the same time as clean builds (for range-v3, exactly the same time). In Rust, they are infinitely faster (100000x faster, depending on the project, but essentially almost instant, in contrast with initial compiles). So for development at least I'll pick Rust over C++ every day. I only do an initial compile once, but I edit and recompile a lot, and that's the bottleneck that I really care about. (Same with C btw, add a new API to a commonly use C header and re-compile, and you end up re-compiling the world, while from Rust, you just re-compile the part of the crate where the new API was added, and since no downstream crates use it, that's pretty much it - instant change).
sebazzz 6y ago
Typescript allows to compile without type checking at the sacrifice of correctness. If you would have a slow computer or huge code-base, you could consider using the type checker only for tests, pre-checkin check, or CI. (Not that Typescript is slow - but as an conceptual example) Would something similar be possible for Rust, so you still have the ability for correctness for CI and release builds, but allow for fast compilation when necessary or desired?
- deleted 6y ago
  [deleted]
- justwalt 6y ago
  I’ve heard someone say about the rust compiler that it will never allow compilation without its safety checks in place because it would decrease faith in any rust binary.
  - estebank 6y ago
    You could always replace the fn body of large parts of your codebase with unimplemented!() (or use the nightly only everybody-loops flag), but I don't think this will have as big a benefit as people would expect. cargo check though is probably something more people should know about.
- MinusGix 6y ago
  Typescript would have a far easier time in that scenario because much of it's syntax is simply Javascript with extra types attached. So, it could (I don't know the implementation details at all) 'just' strip away all the typescript-syntax, and give you your Javascript files. Rust has the issue that there's a lot of type-checking done, even in basic scenarios: such as setting a variable `let alpha = some_function(abc);`, you have to do some form of basic typechecking here to get the type of `alpha` (since that affects how it's stored on the stack). The simple case of `fn some_function(abc: u32) -> i16;` would be 'easy', but more complicated scenarios like generics would make your life harder. Though, I'm sure there's parts of the checking that could be eventually written so that they can be turned off, but I don't think it would provide nearly as much benefit as Typescript's non-correct compilation. Personally, I think it would be better to spend that time on just making the compiler faster. This is perhaps easier than I made it sound, but this is my assumptions as to why it wouldn't work that well.
alfiedotwtf 6y ago
I’m not sure why everyone cares so much about compile times compared to the only thing that matters - runtime performance. For the lifetime of a binary, compilation time is like 0.000001% of its existence. I’m more than happy to wait longer compared to other languages (even zero time at all with scripting languages) while cargo does it’s thing.
- mhluongo 6y ago
  Business reason: When you're in the business of building software, people are your biggest expense. You want to waste their time as little as possible. Dev reason: tight feedback loops are encouraging, and make it easier to build momentum. I hate being deep in a problem then having to "wait" for a compile.
- hellofunk 6y ago
  But compile times have a massive effect on developer time, which has a massive effect on the productivity and the amount of work that can actually be done. Comparing compile time to run time is rather strange, but compared as a percentage of a developer’s time, it’s very obvious why people are worried about it.
  - teraku 6y ago
    I think it's a bit of a shadow argument. So for business, the logic goes: Goal := save(Money). time(Developer) := costs(Money). Therefore, to achieve our Goal, we optimise developer time. But more often than not, businesses don't know how their developers spend their time. So as such, they optimise for results. The business doesn't care how you spend your time, as long as by the end of Sprint/Deadline/Whatever you have a result and/or a reason why not. And I'd bet some money that the reason will very rarely be "my compiler took to long". From a developer standpoint, I understand better (mainly because I'm a developer). But it still lacks a bit: So I started using Rust and I have a long history with C. The first compile with Rust takes very long (because of loading all dependencies), but afterwards it is slower than C, but it's okay. In return you get a lot of extra safety for your program, it's basically time you spent on your C program to fix all those nitty gritty memory safe optimisations and making sure there's no dangling pointers and free(NULL) errors. Of course with experience, these get less and less, but there's always risk. Nobody is perfect. For me as a developer, it's more of a shift in my thinking - which is, to be fair, hard to enforce on people and can only come from oneself. But now that I have a compiler which supports me handling memory and pointers, my mind is more at ease. It does these things I spend time debugging on for me (most of them, not all) and in return for longer compile time, I save development/debugging time. At first it feels a bit less productive, since you are just sitting around watching your compilation (there's an XKCD for that https://xkcd.com/303/ https://xkcd.com/303/), but you could also spend this time writing documentation, reading up on something or just grabbing your coffee (or leave the office and check in tomorrow). Things you would do anyways, it's just a matter of how you mentally approach it. I think the downside only comes when you want it to be negative, or when you're still transitioning.
zelly 6y ago
Yes, we should have a national politburo assign funds to Rust development. Although most of private industry (proles) disagrees, us in the Administration believe that Rust is superior. The proles--ahem I mean programmers with jobs--keep causing problems by using Unsafe technologies and being too dumb to use them. Rust should be funded by The People. It is a Public Good after all because using anything else would be an act of defiance against the will of The People.
fluffything 6y ago
> increasing the number of people paid to work on the Rust compiler by 10x would only mean hiring about 25 people. Citation needed ? I know at least 20 people being paid to work on the Rust compiler, which crank's your 10x number from 25 to 200. And this is only the people I can come up with from the top of my head.

by roca 6y ago

crb002 6y ago

https://github.com/StanfordSNR/gg https://github.com/StanfordSNR/gg - should be almost the same as the GCC/LLVM thunk extractor. You have to pay for the borrow checker, but LLVM IR optimization passes should be the same complexity.

Ericson2314 6y ago

Replace "faster" with "more incremental". Trying to get a program to quickly do the exact same thing over and over again is a colossal waste of resource.

hinkley 6y ago
When you’re trying to balance a bad budget, you can’t dismiss many of the cost centers and still succeed. You can prioritize, but that makes the rest a matter of “when” not if. It can be hard to tell from the outside if someone is avoiding a problem because they don’t want to solve it, or because they are hoping for a more inspired solution. Tackling a problem when all the options are bad may block another alternative when it does finally surface. I’d lump this in with irreversible decisions and the advice to delay them to the last responsible moment. Who know what internal plumbing has to be moved around to make the compiler more incremental, parallel, or both.
- Ericson2314 6y ago
  Yes, let's keep on all options on the table. But trying to constant-factor optomize O(code I didn't change) is a ridiculous approach that needs justification, and the burden of proof is on those that advocate it. Yet, this seem to be the default thing people beg for. Switching to incremental is hard programming, and that's why it hasn't happened overnight, but at least it is an algorithmically sane approach.

darksaints 6y ago

I know a lot of people will disagree with me, and I know that the rust compiler has room for improvement, but compile time isn't really that bad for rust when you consider what it actually does for you. Assuming no unsafe blocks, rust's type system eliminates entire classes of memory safety errors. It fundamentally eliminates data races. It eliminates large swaths of logical errors, due to its use of algebraic data types, and strong type enforcement. No null pointers, explicit error handling, and no undefined behavior. It boggles my mind that people will balk at compile times for rust, but then not even bat an eye at their C++ testkit that runs for 8+ hours, primarily checking for errors that rust would never let you make in the first place.

benesch 6y ago
Part of what makes rustc's slow compile times so frustrating is that the slow compile times are mostly unrelated to all of the fancy compile-time analysis you listed. You can see this in action if you run `cargo check`. That will run all the special static analysis (typeck, borrowck, dropck, etc.) and complete in a very reasonable amount of time. To a first approximation, rustc is slow because it generates terrible LLVM IR, and relies on a number of slow LLVM optimization passes to smooth that IR into something reasonable. As far as I can tell this is entirely a result of not having the necessary engineering resources to throw at the problem; Mozilla has a limited budget for Rust, and needs to spread that budget across not just the compiler toolchain, but the language and ecosystem. Brian Anderson's been working on a blog post series that covers Rust's slow compile times in more detail, if you're curious: https://pingcap.com/blog/tag/Rust https://pingcap.com/blog/tag/Rust
- darksaints 6y ago
  Yeah, I realize that Rust has a lot of room for improvement. I'm just saying that compared to the alternative, you get a relatively inefficient compiler that, despite its inefficiency, still gives you a more efficient development process. You get instant feedback, which you can't get from a testkit, and you can eliminate massive amounts of testing. If the focus is entirely on reducing compile times, it can very quickly lead down a path that limits runtime performance. There is so much potential for increased runtime performance in strongly typed languages, due to the guarantees made by the type system. It allows for analysis and reasoning that not just enables micro-optimizations but also macro-optimizations across entire codebases. Entire theses have been written describing optimizations that rely on immutability alone, and we haven't even touched so many other possibilities that rust's design enables (such as the ownership system). I'm fairly familiar with the project to work on the LLVM-IR code emitter. If the focus is on reducing the complexity and cruft of the LLVM-IR emitted, that's a win all around (because you get faster code and faster compile times). I just hope that reducing compile times doesn't become the core objective, because that eliminates a lot of future potential.

zoul 6y ago

How much of the needed speedup is engineering and how much is research?

roca 6y ago
Depends on what you mean by "research". If you mean "problems with a high degree of novelty" then quite a lot of research, I think. Rust is unusual: it requires a lot of tricky type inference and static analysis from the compiler, and doesn't make developers structure their code with header files etc to make separate compilation easier; BUT unlike most languages with those properties, people are writing large real-world projects in it.
- pornel 6y ago
  My understanding is that Rust's slowness is mainly from code generation and optimization. `cargo check` performs full parsing and type-checking, and runs reasonably quickly. The problem starts when Rust throws a ton of IR at LLVM. The current focus is on adding optimizations in Rust at higher level (MIR) so that it would cause less work for LLVM.
  - acqq 6y ago
    Imagine you have 16 cores available. Allowing parallel compilation where it is sequential at the moment, you get 16 times speedup on such a machine. 32 cores machine -- 32 times speedup etc. Or even 10 such machines -- 320 times speedup. On the other side, imagine you can even reduce the "ton of IR" tenfold but if the compilation is sequential, the tenfold speedup is fixed even if you have 32 or 320 cores. In general, allowing for parallel compilation is much more important topic, and it's worth even redesigning language to make that more achievable. Which also doesn't mean that producing "ton of IR" is a good thing. But working on allowing maximum parallelism is more fundamental. Also, discovering algorithms to allow all compiler passes as effective as possible is important for something to reach wider usability. But Rust must also get to achieve its promises of being "safer" in the scenarios where C is otherwise used, and I time and again see a lot of "unsafe" keywords in the sources wherever I see some implementation of anything fundamental.

the8472 6y ago

I assume dependency trees in the kernel would be much more shallow.

roca 6y ago
That might be true for drivers. Outside the kernel, I don't think it's going to be true for Android or Windows.
- stjohnswarts 6y ago
  But this discussion is mostly about the kernel, isn't it? I'm not sure why throwing out rust weaknesses in general is appropriate when we're talking about a specific use case such as kernel drivers and other modules which do tend to be quite a bit more shallow than most large c/c++ projects I've been on.
  - roca 6y ago
    That's a fair point. I do think it's appropriate to point out that build times are a problem for a lot of projects and may affect the kernel so careful attention needs to be paid to that issue.

cxr 6y ago

> a Manhattan Project to build a fast Rust compiler Or work on getting what Rust gives you in a different language that isn't Rust (maybe entirely new, maybe not[1]). One underdiscussed thing about Rust is that it's so clearly a byproduct of the C++ world. And the most disappointing thing about Rust is that it probably won't ever be seen as a prototype that should be thrown away. 1. http://mbeddr.com/ http://mbeddr.com/

nilkn 6y ago
Rust is actually a byproduct of OCaml and to a lesser extent Haskell. This becomes obvious the more you use it, especially if you’ve written substantial code in either of those languages before. To really drive this home, I’d wager that you have a better chance of translating OCaml or non-fancy Haskell straight into Rust than you do C++.
- gnuvince 6y ago
  > I’d wager that you have a better chance of translating OCaml or non-fancy Haskell straight into Rust than you do C++. I found that translating OCaml code to Rust is frustrating, because you cannot easily and cheaply replicate the functional approach. It's much more Rustic to take an object by mutable reference and modify it in-situ than it is to take an immutable reference and return a new object. The reason is that in OCaml, you have structural sharing, i.e., multiple pointers pointing into the same data structure, which is a big no-no in Rust, so if you want to replicate the functional OCaml code, you begin allocating a new, complete data structure every time, and that's costly.
  - nilkn 6y ago
    Well, of course. Rust is not designed around immutable data structures. There are going to be differences because it's simply not the same language. This [0] example, written by a notable Haskeller, illustrates what I mean pretty well and also touches on what you just pointed out. It's also very consistent with my own experience. [0]: https://donsbot.wordpress.com/2020/07/04/back-to-old-tricks-or-baby-steps-in-rust/ https://donsbot.wordpress.com/2020/07/04/back-to-old-tricks-...

mekster 6y ago

> increasing the number of people paid to work on the Rust compiler by 10x would only mean hiring about 25 people. Compared to the size of the projects that are starting to depend on Rust, that's a rounding error. And are you going to pay for the "rounding error" or just expecting someone to pay for millions a year for your idea?

roca 6y ago
It's not my idea. This is well-known. As for who pays for it: that's a tricky issue! But if we get into a situation where (for example) 10,000 developers spend an hour a day waiting for builds, we know 50 developers could fix the issue in a year, but that doesn't happen because we can't figure out how to structure it economically --- that would be a failure of organizational imagination.
pchanda 6y ago
I think OP meant that companies, like Amazon and MS, whose projects are starting to depend on rust should be putting some resources. Although to be fair, Microsoft and Amazon do pay for their infrastructure if I am not wrong.
- roca 6y ago
  Right on both counts.
deleted 6y ago
[deleted]
jodrellblank 6y ago
This is such a low effort sneer-comment, it's a dismissive way to attack and putdown any idea that includes any amount of payment. It's the discussion equivalent of "whoever smelt it, dealt it" - if someone mentions an idea which costs money, demand as a first response if they are going to pay for it (assuming they aren't, instant dismissal), or accuse that the alternative is a kind of entitled and unfair expectation held of others, when neither need be the case. "That volcano looks like it's becoming active, maybe some sensors could give us an early warning of problems" - Are YOU going to pay for your little "idea"??? Then WHO IS? "That tree is getting dangeoursly high, it's at risk of falling down in a big storm this winter, maybe we could get it cut down before that" - Are YOU going to pay for it?! "Dumping raw sewage in the river is making people sick, treating it first would take a small amount of space and a small fraction of the council's existing budget" - and you want to TAX ME for YOUR clean river, I suppose?! As if OP is the only person who would benefit, as if automatically assuming the worst possible intent for who would pay for it, and as if "who would pay" is the first and only thing worth demanding an answer to, before even having a discussion about whether it's worth doing at all - and which of many ways it might be done.

Kenji 6y ago

That Rust compiles slowly is a meme at this point. C++ isn't faster. Nobody's complaining that C++ isn't viable until compilers get faster.

coldpie 6y ago

> increasing the number of people paid to work on the Rust compiler by 10x would only mean hiring about 25 people I don't think there are 25 people qualified to do that work looking for jobs. It's not something you can just throw money at, you need really qualified people to do that kind of work. Those people are in very high demand.

roca 6y ago
There are a lot of people out there who can do this kind of work. I know a lot of such people. Many of them don't have traditional CVs. Yes, most of them are employed. Some are under-employed. Some are well employed but might be keen to work on a project this important.
steveklabnik 6y ago
Note that that would be increasing the number for pay, you already have many people who have the relevant qualifications because they’re already doing the work, just only in their spare time. Second, the compiler team has put a ton of effort into mentorship infrastructure, and you can go from not knowing a ton to being a productive member of the team with their help. It "just" takes time and desire to do it.
- hinkley 6y ago
  Or on some other project. Sure, there are probably some projects for which if the primary author leaves, their coworkers will drop it or rewrite it in another language, but for others the management will realize what a liability it was to have all of their eggs in that one basket and they’ll have to hire 2 to replace them. If you steal 25 to work on the kernel, you will likely see 25 open reqs to replace them. And more people will work with languages that they have seen reqs for.
- darksaints 6y ago
  Interesting. What sort of mentorship infrastructure? What sort of people are they targeting for mentorship? I'd love to learn about compilers by working on an actual real world compiler, but I really only know high level details about how different components fit together. I also don't have a formal CS education, although I am currently working as a senior level data engineer. Not sure if that is the sort of person they are looking to mentor.

crazypython 6y ago

GCC 10 supports Dlang directly. (The support is still immature and it's being updated to the newer version of Dlang.) Dlang compiles quickly not because the language is simple, but because the compiler authors care about compilation speed: Lexer skips four spaces at once https://github.com/dlang/dmd/pull/11095 https://github.com/dlang/dmd/pull/11095 Optimize core logic to 7x compilation speed of language feature https://github.com/dlang/dmd/pull/11303 https://github.com/dlang/dmd/pull/11303 Cache rarely used types to improve memory locality https://github.com/dlang/dmd/pull/11363 https://github.com/dlang/dmd/pull/11363

PudgePacket 6y ago
Slightly off topic, I see D brought up incredibly consistently when Rust is mentioned on HN.. Is it just me?? (it's usually by walter themself but I digress..)
- Keyframe 6y ago
  A number of people (including myself) migrated from what was known as D1 onto Rust. Steve (Klabnik) was also one of the refugees. I can't speak for others, but Rust is kind of what I was expecting for D1 to go to.
  - aberba 6y ago
    Wow, didn't know Steve was rocking D.
- pjmlp 6y ago
  Some of us believe in systems programming languages with some form of GC, so it usually pops up as alternative. For me D is what .NET (thus C#) should have been if the Ext-VOS project (COM Runtime) decided to go native, instead of embracing J++ and then turning into .NET due to the lawsuit. But unfortunately D is a tiny community that gets by with small contributions and herding cats is hard.
VWWHFSfQ 6y ago
> but because the compiler authors care about compilation speed Is this meant to imply that the Rust compiler authors don't care about compilation speed?

rumanator 6y ago

> Rust in Linux will be fantastic except for compile time. Rust (and the world) needs a Manhattan Project to build a fast Rust compiler (where by "fast" I mean both efficient and scalably parallel when compiling a workspace with many crates and long dependency chains). A basic technique to parallelize sections is to split a project into modules. Is this not an option for the linux kernel?

roca 6y ago
Rust has great support for modules (crates in Rust lingo). It can compile modules in parallel when they don't depend on each other. The problem is that when you have module A depending on module B, rustc doesn't do a good job of compiling A and B in parallel. In contrast, in C/C++ you can handcraft your header files to ensure that A and B compile in parallel.
- pjmlp 6y ago
  In C and C++ it is also quite common to use binary dependencies, something that cargo still doesn't do. However C++ is also now looking at parallelization opportunities for modules, which is a similar situation as crates.
  - fluffything 6y ago
    Every time you compile a rust project after an initial compile, cargo does use "binary" dependencies (bitcode dependencies). It is much better at this than C++. For example, change a trait in your Rust library, and recompile the tests, and compare that with changing a type in, say, range-v3 (or any other C++ header-only library). In C++, recompiles take pretty much the same time as clean builds (for range-v3, exactly the same time). In Rust, they are infinitely faster (100000x faster, depending on the project, but essentially almost instant, in contrast with initial compiles). So for development at least I'll pick Rust over C++ every day. I only do an initial compile once, but I edit and recompile a lot, and that's the bottleneck that I really care about. (Same with C btw, add a new API to a commonly use C header and re-compile, and you end up re-compiling the world, while from Rust, you just re-compile the part of the crate where the new API was added, and since no downstream crates use it, that's pretty much it - instant change).

sebazzz 6y ago

Typescript allows to compile without type checking at the sacrifice of correctness. If you would have a slow computer or huge code-base, you could consider using the type checker only for tests, pre-checkin check, or CI. (Not that Typescript is slow - but as an conceptual example) Would something similar be possible for Rust, so you still have the ability for correctness for CI and release builds, but allow for fast compilation when necessary or desired?

deleted 6y ago
[deleted]
justwalt 6y ago
I’ve heard someone say about the rust compiler that it will never allow compilation without its safety checks in place because it would decrease faith in any rust binary.
- estebank 6y ago
  You could always replace the fn body of large parts of your codebase with unimplemented!() (or use the nightly only everybody-loops flag), but I don't think this will have as big a benefit as people would expect. cargo check though is probably something more people should know about.
MinusGix 6y ago
Typescript would have a far easier time in that scenario because much of it's syntax is simply Javascript with extra types attached. So, it could (I don't know the implementation details at all) 'just' strip away all the typescript-syntax, and give you your Javascript files. Rust has the issue that there's a lot of type-checking done, even in basic scenarios: such as setting a variable `let alpha = some_function(abc);`, you have to do some form of basic typechecking here to get the type of `alpha` (since that affects how it's stored on the stack). The simple case of `fn some_function(abc: u32) -> i16;` would be 'easy', but more complicated scenarios like generics would make your life harder. Though, I'm sure there's parts of the checking that could be eventually written so that they can be turned off, but I don't think it would provide nearly as much benefit as Typescript's non-correct compilation. Personally, I think it would be better to spend that time on just making the compiler faster. This is perhaps easier than I made it sound, but this is my assumptions as to why it wouldn't work that well.

alfiedotwtf 6y ago

I’m not sure why everyone cares so much about compile times compared to the only thing that matters - runtime performance. For the lifetime of a binary, compilation time is like 0.000001% of its existence. I’m more than happy to wait longer compared to other languages (even zero time at all with scripting languages) while cargo does it’s thing.

mhluongo 6y ago
Business reason: When you're in the business of building software, people are your biggest expense. You want to waste their time as little as possible. Dev reason: tight feedback loops are encouraging, and make it easier to build momentum. I hate being deep in a problem then having to "wait" for a compile.
hellofunk 6y ago
But compile times have a massive effect on developer time, which has a massive effect on the productivity and the amount of work that can actually be done. Comparing compile time to run time is rather strange, but compared as a percentage of a developer’s time, it’s very obvious why people are worried about it.
- teraku 6y ago
  I think it's a bit of a shadow argument. So for business, the logic goes: Goal := save(Money). time(Developer) := costs(Money). Therefore, to achieve our Goal, we optimise developer time. But more often than not, businesses don't know how their developers spend their time. So as such, they optimise for results. The business doesn't care how you spend your time, as long as by the end of Sprint/Deadline/Whatever you have a result and/or a reason why not. And I'd bet some money that the reason will very rarely be "my compiler took to long". From a developer standpoint, I understand better (mainly because I'm a developer). But it still lacks a bit: So I started using Rust and I have a long history with C. The first compile with Rust takes very long (because of loading all dependencies), but afterwards it is slower than C, but it's okay. In return you get a lot of extra safety for your program, it's basically time you spent on your C program to fix all those nitty gritty memory safe optimisations and making sure there's no dangling pointers and free(NULL) errors. Of course with experience, these get less and less, but there's always risk. Nobody is perfect. For me as a developer, it's more of a shift in my thinking - which is, to be fair, hard to enforce on people and can only come from oneself. But now that I have a compiler which supports me handling memory and pointers, my mind is more at ease. It does these things I spend time debugging on for me (most of them, not all) and in return for longer compile time, I save development/debugging time. At first it feels a bit less productive, since you are just sitting around watching your compilation (there's an XKCD for that https://xkcd.com/303/ https://xkcd.com/303/), but you could also spend this time writing documentation, reading up on something or just grabbing your coffee (or leave the office and check in tomorrow). Things you would do anyways, it's just a matter of how you mentally approach it. I think the downside only comes when you want it to be negative, or when you're still transitioning.

zelly 6y ago

Yes, we should have a national politburo assign funds to Rust development. Although most of private industry (proles) disagrees, us in the Administration believe that Rust is superior. The proles--ahem I mean programmers with jobs--keep causing problems by using Unsafe technologies and being too dumb to use them. Rust should be funded by The People. It is a Public Good after all because using anything else would be an act of defiance against the will of The People.

fluffything 6y ago

> increasing the number of people paid to work on the Rust compiler by 10x would only mean hiring about 25 people. Citation needed ? I know at least 20 people being paid to work on the Rust compiler, which crank's your 10x number from 25 to 200. And this is only the people I can come up with from the top of my head.