4 ms·

A pet peeve of mine is the almost complete lack of empiricism in Computer Science. "Nevertheless, we believe that, even today, the advantages of using Rust [in

A pet peeve of mine is the almost complete lack of empiricism in Computer Science. "Nevertheless, we believe that, even today, the advantages of using Rust [in the Linux kernel] outweighs the cost." Where is the evidence? "Rust is obviously a better choice for Linux than C." "Ok, if it is obvious, it should be easy to demonstrate?" This is not only about Rust - 99% or so of all new tech can't demonstrate any benefits.

freeopinion 5y ago
While human preference is probably not the most important consideration and is debatable, it does count as one possible benefit.
nullc 5y ago
It's not just being able to demonstrate the benefits empirically, but the combination of that with constantly loudly touting them. When someone thinks their hashmap is faster in theory they get asked to benchmark it... it often isn't in practice. But when someone decides that some norm in whitespace, use of parenthesis, or use favored control-flow concepts will result in a lower defect rate or reduced maintenance costs we somehow are expected to accept these conclusions without clear evidence. Even though everywhere is in computer science where we can easily measure results there are countless examples of theoretically better stuff that works less well in practice. Why should we expect our intuition on human interactions to be any better than our intuitions over which data structures are faster?
emmericp 5y ago
Most "new tech" is engineering, not science. If you want empiricism try academic papers instead of mailing lists ;) Some numbers: most critical bugs in the Linux kernel are due to memory safety: 40 out of 65 bugs allowing for code execution found in Linux in 2017 could have been prevented by using a memory-safe language [0]. The paper is about Go, but the same logic applies for Rust. Fun fact: 39 out of these 40 bugs were in drivers [1], so I think starting with drivers in Rust is a good idea. [0] https://www.usenix.org/conference/osdi18/presentation/cutler https://www.usenix.org/conference/osdi18/presentation/cutler (really good read!) [1] https://arxiv.org/pdf/1909.06344.pdf https://arxiv.org/pdf/1909.06344.pdf (disclaimer: I'm a co-author of that paper)
- juped 5y ago
  Do you think engineering is somehow less empirical and more abstract than science?
- sk1459 5y ago
  Calling programming engineering in the absence of licensure or any standard of performance for the entire industry is kind of silly.
  - danhor 5y ago
    This is a popular argument, but I'd argue isn't really true. Looking at electrical engineering, there is no standard license and no standard of performance. Like programming, it's just too broad across different areas with vastly different requirements. The equivalent of very sloppy web development might be a blinky (non-kids) toy, that fails after a few uses with no particular requirements (other than "make it as cheap as possible"). Or look at how often smartphones and laptops fail electrically (or compound failures) for really obvious reasons, that just weren't important enough to fix. To address the particular point being made here, lots of things in EE are experience driven, for example which decoupling capacitors to use or to not use 90 degree angels on pcbs. I think many aspects of typical engineering are way more driven by experience and vague rules of thumb than we expect looking in from the outside. The reason why this is more obvious in software development may be because it's pretty hard to have good measurements and understanding failures is pretty hard and working around them is pretty easy.
  - mlindner 5y ago
    Very few countries in the world have "licensure" for engineers. It's not a protected term of any sort in the US nor in most of Europe. Canada is one of the few places that's strict about it. Occupational licensing is a form of job market restriction that's generally frowned upon.
    - _dps 5y ago
      In the three European countries I've lived in (2 in EU + UK), licensed engineers must sign off on new building construction plans, and take responsibility for the suitability of the design.
ben0x539 5y ago
Do you think it would be preferable for the Rust-in-the-kernel advocates to not engage with the kernel community at all until they have a prepared a large array of demonstrations?
trappist 5y ago
Maybe Linux would be better written in BASIC. We can't know until someone does it.
- scrubs 5y ago
  You're in to something ... Go op go!
kaba0 5y ago
It can be a pet peeve of mine as well, but at the same time, there are statements that require no empirical analysis to “prove”. Like, would you disagree that writing a program in brainfuck is more error-prone than Java? But even in medicine, probably the area with the most extensive empiricism (we have insane amounts of data on mouses and humans), there are claims that won’t ever get to the point of doing a study on. There are stupid things that get tested, eg homeopathy (it has plenty of studies showing it is quack), but I think noone would say it is irresponsible for a doctor to claim that hand healing over a gun shot is not effective, even without direct evidence. Of course there are surprising findings everywhere and as other commenters wrote, especially in low level optimizations the “obviously slower” solution can easily win. But I would wager that C is so terrible as a language with so little protection from even trivial mistypings, no real abstraction power requiring text-based macros that simply should have never existed to begin with, and the worst of all, not even goddamn namespaces making everything’s name indecipherable. On the other hand, I’m not necessarily bought that Rust’s borrow checker is the best way for handling memory-related errors, for example a syntactically better-than-C language like C++, or Zig might reduce memory bugs just as much, we don’t know.
ojeda 5y ago
> the almost complete lack of empiricism in Computer Science Claiming that an entire science field "lacks empiricism" is quite brave, and definitely false in the case of CS. Perhaps you are talking about software engineering in practice instead, but that is false too (for any serious project). > "Nevertheless, we believe that, even today, the advantages of using Rust [in the Linux kernel] outweighs the cost." Where is the evidence? Three sections of the RFC are spent on characterizing the sentence you quote. The LWN article is not the RFC. Furthermore, many of the claims I wrote in the RFC are simple facts (e.g. features that Rust has) that you can easily corroborate. For other claims, you can look up plenty of articles, scholarly and otherwise, about Rust benefits. But none of that really matters, because the only way to gather the evidence you are actually requesting (Rust in the Linux kernel) is getting into the kernel and evaluating Rust after a while. > "Rust is obviously a better choice for Linux than C." The RFC does not claim it is "obviously" a better choice. Quite the opposite, in fact: see the "Why not?" section.
- bjourne 5y ago
  > Claiming that an entire science field "lacks empiricism" is quite brave, and definitely false in the case of CS. > Perhaps you are talking about software engineering in practice instead, but that is false too (for any serious project). I'm not the first one lamenting the lack of experimentation in CS. See https://www.cs.princeton.edu/~jrex/teaching/spring2005/fft/moreexperiments.pdf https://www.cs.princeton.edu/~jrex/teaching/spring2005/fft/m..., http://www.inf.fu-berlin.de/inst/ag-se/teaching/S-Komponenten-2005/SendE05-Empirical.pdf http://www.inf.fu-berlin.de/inst/ag-se/teaching/S-Komponente..., https://ieeexplore.ieee.org/abstract/document/1027796 https://ieeexplore.ieee.org/abstract/document/1027796, https://ieeexplore.ieee.org/abstract/document/4221632 https://ieeexplore.ieee.org/abstract/document/4221632 > Three sections of the RFC are spent on characterizing the sentence you quote. The LWN article is not the RFC. I know, but it doesn't contain (empirical) evidence. > But none of that really matters, because the only way to gather the evidence you are actually requesting (Rust in the Linux kernel) is getting into the kernel and evaluating Rust after a while. That's the medical equivalent of giving the patient an unknown drug to see what happens. Did the patient get better? Maybe it was the drug! Did the patient get worse? Maybe it was not the drug! A better way could be to fork a less-visible (but still well-maintained) C project and rewrite it in Rust. If the defect rate over time is significantly lower than for the upstream C project then that is empirical evidence in support of a Linux rewrite in Rust. Running such an experiment would take a lot of time and effort, but that is no excuse for not doing it.

freeopinion 5y ago

While human preference is probably not the most important consideration and is debatable, it does count as one possible benefit.

nullc 5y ago

It's not just being able to demonstrate the benefits empirically, but the combination of that with constantly loudly touting them. When someone thinks their hashmap is faster in theory they get asked to benchmark it... it often isn't in practice. But when someone decides that some norm in whitespace, use of parenthesis, or use favored control-flow concepts will result in a lower defect rate or reduced maintenance costs we somehow are expected to accept these conclusions without clear evidence. Even though everywhere is in computer science where we can easily measure results there are countless examples of theoretically better stuff that works less well in practice. Why should we expect our intuition on human interactions to be any better than our intuitions over which data structures are faster?

emmericp 5y ago

Most "new tech" is engineering, not science. If you want empiricism try academic papers instead of mailing lists ;) Some numbers: most critical bugs in the Linux kernel are due to memory safety: 40 out of 65 bugs allowing for code execution found in Linux in 2017 could have been prevented by using a memory-safe language [0]. The paper is about Go, but the same logic applies for Rust. Fun fact: 39 out of these 40 bugs were in drivers [1], so I think starting with drivers in Rust is a good idea. [0] https://www.usenix.org/conference/osdi18/presentation/cutler https://www.usenix.org/conference/osdi18/presentation/cutler (really good read!) [1] https://arxiv.org/pdf/1909.06344.pdf https://arxiv.org/pdf/1909.06344.pdf (disclaimer: I'm a co-author of that paper)

juped 5y ago
Do you think engineering is somehow less empirical and more abstract than science?
sk1459 5y ago
Calling programming engineering in the absence of licensure or any standard of performance for the entire industry is kind of silly.
- danhor 5y ago
  This is a popular argument, but I'd argue isn't really true. Looking at electrical engineering, there is no standard license and no standard of performance. Like programming, it's just too broad across different areas with vastly different requirements. The equivalent of very sloppy web development might be a blinky (non-kids) toy, that fails after a few uses with no particular requirements (other than "make it as cheap as possible"). Or look at how often smartphones and laptops fail electrically (or compound failures) for really obvious reasons, that just weren't important enough to fix. To address the particular point being made here, lots of things in EE are experience driven, for example which decoupling capacitors to use or to not use 90 degree angels on pcbs. I think many aspects of typical engineering are way more driven by experience and vague rules of thumb than we expect looking in from the outside. The reason why this is more obvious in software development may be because it's pretty hard to have good measurements and understanding failures is pretty hard and working around them is pretty easy.
- mlindner 5y ago
  Very few countries in the world have "licensure" for engineers. It's not a protected term of any sort in the US nor in most of Europe. Canada is one of the few places that's strict about it. Occupational licensing is a form of job market restriction that's generally frowned upon.
  - _dps 5y ago
    In the three European countries I've lived in (2 in EU + UK), licensed engineers must sign off on new building construction plans, and take responsibility for the suitability of the design.

ben0x539 5y ago

Do you think it would be preferable for the Rust-in-the-kernel advocates to not engage with the kernel community at all until they have a prepared a large array of demonstrations?

trappist 5y ago

Maybe Linux would be better written in BASIC. We can't know until someone does it.

scrubs 5y ago
You're in to something ... Go op go!

kaba0 5y ago

It can be a pet peeve of mine as well, but at the same time, there are statements that require no empirical analysis to “prove”. Like, would you disagree that writing a program in brainfuck is more error-prone than Java? But even in medicine, probably the area with the most extensive empiricism (we have insane amounts of data on mouses and humans), there are claims that won’t ever get to the point of doing a study on. There are stupid things that get tested, eg homeopathy (it has plenty of studies showing it is quack), but I think noone would say it is irresponsible for a doctor to claim that hand healing over a gun shot is not effective, even without direct evidence. Of course there are surprising findings everywhere and as other commenters wrote, especially in low level optimizations the “obviously slower” solution can easily win. But I would wager that C is so terrible as a language with so little protection from even trivial mistypings, no real abstraction power requiring text-based macros that simply should have never existed to begin with, and the worst of all, not even goddamn namespaces making everything’s name indecipherable. On the other hand, I’m not necessarily bought that Rust’s borrow checker is the best way for handling memory-related errors, for example a syntactically better-than-C language like C++, or Zig might reduce memory bugs just as much, we don’t know.

ojeda 5y ago

> the almost complete lack of empiricism in Computer Science Claiming that an entire science field "lacks empiricism" is quite brave, and definitely false in the case of CS. Perhaps you are talking about software engineering in practice instead, but that is false too (for any serious project). > "Nevertheless, we believe that, even today, the advantages of using Rust [in the Linux kernel] outweighs the cost." Where is the evidence? Three sections of the RFC are spent on characterizing the sentence you quote. The LWN article is not the RFC. Furthermore, many of the claims I wrote in the RFC are simple facts (e.g. features that Rust has) that you can easily corroborate. For other claims, you can look up plenty of articles, scholarly and otherwise, about Rust benefits. But none of that really matters, because the only way to gather the evidence you are actually requesting (Rust in the Linux kernel) is getting into the kernel and evaluating Rust after a while. > "Rust is obviously a better choice for Linux than C." The RFC does not claim it is "obviously" a better choice. Quite the opposite, in fact: see the "Why not?" section.

bjourne 5y ago
> Claiming that an entire science field "lacks empiricism" is quite brave, and definitely false in the case of CS. > Perhaps you are talking about software engineering in practice instead, but that is false too (for any serious project). I'm not the first one lamenting the lack of experimentation in CS. See https://www.cs.princeton.edu/~jrex/teaching/spring2005/fft/moreexperiments.pdf https://www.cs.princeton.edu/~jrex/teaching/spring2005/fft/m..., http://www.inf.fu-berlin.de/inst/ag-se/teaching/S-Komponenten-2005/SendE05-Empirical.pdf http://www.inf.fu-berlin.de/inst/ag-se/teaching/S-Komponente..., https://ieeexplore.ieee.org/abstract/document/1027796 https://ieeexplore.ieee.org/abstract/document/1027796, https://ieeexplore.ieee.org/abstract/document/4221632 https://ieeexplore.ieee.org/abstract/document/4221632 > Three sections of the RFC are spent on characterizing the sentence you quote. The LWN article is not the RFC. I know, but it doesn't contain (empirical) evidence. > But none of that really matters, because the only way to gather the evidence you are actually requesting (Rust in the Linux kernel) is getting into the kernel and evaluating Rust after a while. That's the medical equivalent of giving the patient an unknown drug to see what happens. Did the patient get better? Maybe it was the drug! Did the patient get worse? Maybe it was not the drug! A better way could be to fork a less-visible (but still well-maintained) C project and rewrite it in Rust. If the defect rate over time is significantly lower than for the upstream C project then that is empirical evidence in support of a Linux rewrite in Rust. Running such an experiment would take a lot of time and effort, but that is no excuse for not doing it.