4 ms·

Between this and "beacown" I came to a realization. I work on codebases worth way less than these amd we do lots of different static analyzers two compilers, lo

Between this and "beacown" I came to a realization. I work on codebases worth way less than these amd we do lots of different static analyzers two compilers, lots of unit tests with coverage checks and asan/ubsan. The linux kernel and sha are worth tons of money. Thus, someone wrote all the pipelines and unit tests for that software too. The problem is, the people who paid for all that work aren't the people that rely on thos software's security. It's paid for by people that rely on it's insecurity. So we don't know about this stuff until they or the people they sold it too use these exploits. Edit: in case the solution I propose isn't obvious, organizations like red hatand google need to pay for these pipelines and unit tests. Getting good unit test coverage is expensive and ubsan and asan don't work without that coverage. So since no one has mentioned it yet, you could rewrite this stuff in rust as a poor man's substitute. It will catch some of the aame things, but ultimately there is no substitute for test coverage with sanitizers. Well, maybe formal analysis?

lmm 4y ago
> So since no one has mentioned it yet, you could rewrite this stuff in rust as a poor man's substitute. It will catch some of the aame things, but ultimately there is no substitute for test coverage with sanitizers. That's backwards. You'll catch more cases with a Rust-style type system that naturally checks everything, than with sanitisers that can only check the paths that get executed in tests.
- galangalalgol 4y ago
  Rust isn't perfect. UB is a bug in rust, but it occasionally has bugs. Ideally you'd do rust and asan with good unit tests. And yes, if you only picked one, it should be rust, but don't just pick one. And just because you are using rust is no excuse to skip static analysis like prusti, coverage with gcov, llvm, or tarpaulin, and certainly not unit tests.
  - deleted 4y ago
    [deleted]
  - tinglymintyfrsh 4y ago
    There is unlikely ever to be a perfect, but a getting better by constraining behavior that breaks things. Throw all of the compile-time, profiling, checked builds, and binary-level tools at projects for defense-in-depth and checklists, no matter the platform or the application. Fuzzing, valgrind, gperftools, dtrace, {[amt],ub}san, etc. and formal methods like seL4 if you can afford the investment. :>
    - galangalalgol 4y ago
      Does valgrind add anything to asan given the same unit tests? I haven't used in years because it is so slow.
      - saagarjha 4y ago
        Address sanitizer is not precise and can miss certain classes of bugs that Valgrind would catch.
        tinglymintyfrsh 4y ago
        Yeap. Although Google killed off afl, exercise code with fuzzing too that unit and integration tests didn't catch.
  - lmm 4y ago
    > And just because you are using rust is no excuse to skip static analysis like prusti, coverage with gcov, llvm, or tarpaulin, and certainly not unit tests. It absolutely is, and this kind of absolutism is what holds back the adoption of things like rust. In most real world software development, the target defect rate is not zero. The value proposition of something like rust for most businesses isn't that it lets you lower your defect rate; it's that it lets you maintain your existing (acceptable) defect rate at a much lower cost, by letting you drop your static analysis and most of your tests (reducing maintenance burdens) and still have a better bottom-line defect rate.
    - consp 4y ago
      So your assumption is that most to all defects are language related and not programmer error? I will bet the farm your defect rate will remain the same without any actual validation of what people write. Plenty of errors due to language quirks but lots of times I see missing statements, fixed values which should be variable, input checking failures due to unknown input, incorrect assumptions and lots of non language gelated bugs which will not go away.
      - green_on_black 4y ago
        You're not accounting for the fact that Rust ships with a mandatory static analyzer called rustc. Btw Google recently gave a talk about switching from Java to Kotlin, which significantly reduced bugs, despite less mature tooling. So...
      - lmm 4y ago
        > So your assumption is that most to all defects are language related What does that even mean? > I will bet the farm your defect rate will remain the same without any actual validation of what people write. In what sense is a typechecker (or the rust borrow checker) not "actual validation", but a static analyser is? > Plenty of errors due to language quirks but lots of times I see missing statements, fixed values which should be variable, input checking failures due to unknown input, incorrect assumptions and lots of non language gelated bugs Most of those sound like type errors to me. Errors where the program isn't doing what the programmer thought it should can generally be avoided by using more precise types. (The more insidious case is where the program is doing exactly what the programmer thought it should, but they've misunderstood the specification or not thought through the implications - but no kind of testing can catch that kind of bug).

lmm 4y ago

> So since no one has mentioned it yet, you could rewrite this stuff in rust as a poor man's substitute. It will catch some of the aame things, but ultimately there is no substitute for test coverage with sanitizers. That's backwards. You'll catch more cases with a Rust-style type system that naturally checks everything, than with sanitisers that can only check the paths that get executed in tests.

galangalalgol 4y ago
Rust isn't perfect. UB is a bug in rust, but it occasionally has bugs. Ideally you'd do rust and asan with good unit tests. And yes, if you only picked one, it should be rust, but don't just pick one. And just because you are using rust is no excuse to skip static analysis like prusti, coverage with gcov, llvm, or tarpaulin, and certainly not unit tests.
- deleted 4y ago
  [deleted]
- tinglymintyfrsh 4y ago
  There is unlikely ever to be a perfect, but a getting better by constraining behavior that breaks things. Throw all of the compile-time, profiling, checked builds, and binary-level tools at projects for defense-in-depth and checklists, no matter the platform or the application. Fuzzing, valgrind, gperftools, dtrace, {[amt],ub}san, etc. and formal methods like seL4 if you can afford the investment. :>
  - galangalalgol 4y ago
    Does valgrind add anything to asan given the same unit tests? I haven't used in years because it is so slow.
    - saagarjha 4y ago
      Address sanitizer is not precise and can miss certain classes of bugs that Valgrind would catch.
      - tinglymintyfrsh 4y ago
        Yeap. Although Google killed off afl, exercise code with fuzzing too that unit and integration tests didn't catch.
- lmm 4y ago
  > And just because you are using rust is no excuse to skip static analysis like prusti, coverage with gcov, llvm, or tarpaulin, and certainly not unit tests. It absolutely is, and this kind of absolutism is what holds back the adoption of things like rust. In most real world software development, the target defect rate is not zero. The value proposition of something like rust for most businesses isn't that it lets you lower your defect rate; it's that it lets you maintain your existing (acceptable) defect rate at a much lower cost, by letting you drop your static analysis and most of your tests (reducing maintenance burdens) and still have a better bottom-line defect rate.
  - consp 4y ago
    So your assumption is that most to all defects are language related and not programmer error? I will bet the farm your defect rate will remain the same without any actual validation of what people write. Plenty of errors due to language quirks but lots of times I see missing statements, fixed values which should be variable, input checking failures due to unknown input, incorrect assumptions and lots of non language gelated bugs which will not go away.
    - green_on_black 4y ago
      You're not accounting for the fact that Rust ships with a mandatory static analyzer called rustc. Btw Google recently gave a talk about switching from Java to Kotlin, which significantly reduced bugs, despite less mature tooling. So...
    - lmm 4y ago
      > So your assumption is that most to all defects are language related What does that even mean? > I will bet the farm your defect rate will remain the same without any actual validation of what people write. In what sense is a typechecker (or the rust borrow checker) not "actual validation", but a static analyser is? > Plenty of errors due to language quirks but lots of times I see missing statements, fixed values which should be variable, input checking failures due to unknown input, incorrect assumptions and lots of non language gelated bugs Most of those sound like type errors to me. Errors where the program isn't doing what the programmer thought it should can generally be avoided by using more precise types. (The more insidious case is where the program is doing exactly what the programmer thought it should, but they've misunderstood the specification or not thought through the implications - but no kind of testing can catch that kind of bug).