6 ms·

Evaluating the Effectiveness of Memory Safety Sanitizers

48 points by signa11 1y ago

unpaydijk 1y ago
Looks interesting, but unfortunately the research paper is behind a paywall
- signa11 1y ago
  really sorry about that gaffe ! i had access, and the content was too interesting to not share. as @osivertsson has kindly pointed out, may you please access that, and share your insights here ? thanks !
- ev234 1y ago
  Here is the author's version: https://publica-rest.fraunhofer.de/server/api/core/bitstreams/2db39f21-69f6-45c1-b117-0aa5cb096eac/content https://publica-rest.fraunhofer.de/server/api/core/bitstream...
osivertsson 1y ago
Download possible without paywall from https://publica.fraunhofer.de/entities/publication/9d7783f8-4158-4cae-9304-1bf5e7036935 https://publica.fraunhofer.de/entities/publication/9d7783f8-...
- rwmj 1y ago
  Unfortunately that gives a 500 error when attempting to download the PDF (maybe the server is overloaded now?)
  - rurban 1y ago
    https://publica-rest.fraunhofer.de/server/api/core/bitstreams/2db39f21-69f6-45c1-b117-0aa5cb096eac/content https://publica-rest.fraunhofer.de/server/api/core/bitstream...
Ygg2 1y ago
Conclusion is scathing: > The findings highlight significant variations in the theoretical detection capabilities of these techniques and reveal that, in practice, the implementations of most available sanitizers fall short of their conceptual potential. Furthermore, the evaluation demonstrates the complexities and diversity of memory bugs in C/C++, as well as the challenges associated with detecting them. For instance, our results show that SoftBound+CETS, a conceptually complete sanitizer, misses nearly a quarter of spatial memory bugs in its original implementation, while ASan, likely the most widely used memory sanitizer, cannot detect 50% of use-after-* bugs and any non-linear overflows and under- flows. Ultimately, our evaluation concludes that no sanitizer currently provides complete temporal or spatial memory safety
bgwalter 1y ago
If SoftBound+CETS has the best results, why does Fraunhofer not sponsor the creation of a Debian package? It is unmaintained: https://github.com/Fraunhofer-AISEC/softboundcets https://github.com/Fraunhofer-AISEC/softboundcets
- rurban 1y ago
  With this commit being the actual llvm patch for Softbounds+CETS: https://github.com/Fraunhofer-AISEC/softboundcets-llvm-project/commit/c014965edf13a3745e17c175408fa1ce9082f74c https://github.com/Fraunhofer-AISEC/softboundcets-llvm-proje... x86, x64 only. LTO only.
lou1306 1y ago
Weird that Infer [1] was not included in the evaluation. It supports C/C++ and its underlying reasoning framework (Separation Logic [2]) is exactly geared towards checking memory safety. [1] https://fbinfer.com/ https://fbinfer.com/ [2] https://en.wikipedia.org/wiki/Separation_logic https://en.wikipedia.org/wiki/Separation_logic
- UncleMeat 1y ago
  Sanitizers are runtime tools, not static analysis tools. Static analyzers are also virtually never sound as sound tools produce an outrageous number of false positives, especially when languages that so easily permit nonlocal mutation.
rwmj 1y ago
I wonder how true the assertion "This performance is partly achieved by sacrificing memory safety" is today. I suspect a sufficiently advanced compiler can remove bounds checks where they are provably unnecessary, and a sufficiently advanced CPU can run the remaining checks in parallel with the array accesses. But it'd be interesting if there's been any research on that. (A sufficiently advanced programming language can avoid the entire issue by writing loops as map, fold, etc. but we're talking about C here.)
- bluGill 1y ago
  There are a few things that cannot be done as fast in rust, but those are rare to need in the real world. Most of the things rust cannot do are around sharing memory between threads with locks - Humans have a very hard time getting code that does this to work correctly and usually have race conditions because they analysed the problem wrong.
  - rurban 1y ago
    That's actually easy to do, if you care about concurrency safety at first when designing a language and it's stdlib. Just forbid blocking IO and no locks needed.
- Someone 1y ago
  > I suspect a sufficiently advanced compiler can remove bounds checks where they are provably unnecessary, That’s true by definition, isn’t it? > and a sufficiently advanced CPU can run the remaining checks in parallel with the array accesses. But it still would slow down the program, as the CPU would have to commit resources to that bound checking that it then cannot use for doing other things.
- MindSpunk 1y ago
  > A sufficiently advanced compiler Much like how Intel Itanium would've taken over the world with the VLIW revolution if only we had one of these 'sufficiently advanced compilers'. It's a very vague statement that doesn't really mesh with reality. Decades of research in compilers have yet to yield one. I think you're right, and it's up to the programming language to provide optimization friendly primitives that fit into the mold of what we can prove as in-bounds. Speculative execution will never be free because those bounds checks will take up space in the execution pipeline that could've been used for other instructions.
ape4 1y ago
Seems like it should have "C/C++" in the title. Or maybe that's understood?
- ben-schaaf 1y ago
  Sanitizers aren't just for C/C++. Rust, go, D all have at least asan support.
  - vlovich123 1y ago
    Rust supports all the same sanitizers as c/C++ last I checked (at least UBSAN, TSAN, MSAN and ASAN which are the only ones I’ve ever seen used in practice on c/c++).
pizlonator 1y ago
I wonder if their test cases are available. Fil-C is specifically engineered to catch everything so it would be interesting to check it against their tests
- ev234 1y ago
  [dead]
pizlonator 1y ago
Wait hold up the same authors did MESH: https://arxiv.org/pdf/2108.08683 https://arxiv.org/pdf/2108.08683 So why isn't MESH part of the evaluation? And why isn't it mentioned even once in the paper?
- rurban 1y ago
  Weird indeed. But I also miss valgrind, which is precise, just very slow. I wonder if valgrind detects intra-object OOBAs.
  - ev234 1y ago
    Valgrind's Memcheck is included. It does not detect intra-object OOBAs.
    - rurban 1y ago
      Ah thanks, missed that
  - pizlonator 1y ago
    No, it's not that precise. Valgrind is based on tagging whether a memory location is live or not. So, it would not just fail the intra-object OOB test, it would also fail the "out of bounds but in bounds" case: https://github.com/pizlonator/llvm-project-deluge/blob/deluge/invisicaps_by_example.md#out-of-bounds-but-in-bounds https://github.com/pizlonator/llvm-project-deluge/blob/delug...

48 points by signa11 1y ago

unpaydijk 1y ago

Looks interesting, but unfortunately the research paper is behind a paywall

signa11 1y ago
really sorry about that gaffe ! i had access, and the content was too interesting to not share. as @osivertsson has kindly pointed out, may you please access that, and share your insights here ? thanks !
ev234 1y ago
Here is the author's version: https://publica-rest.fraunhofer.de/server/api/core/bitstreams/2db39f21-69f6-45c1-b117-0aa5cb096eac/content https://publica-rest.fraunhofer.de/server/api/core/bitstream...

osivertsson 1y ago

Download possible without paywall from https://publica.fraunhofer.de/entities/publication/9d7783f8-4158-4cae-9304-1bf5e7036935 https://publica.fraunhofer.de/entities/publication/9d7783f8-...

rwmj 1y ago
Unfortunately that gives a 500 error when attempting to download the PDF (maybe the server is overloaded now?)
- rurban 1y ago
  https://publica-rest.fraunhofer.de/server/api/core/bitstreams/2db39f21-69f6-45c1-b117-0aa5cb096eac/content https://publica-rest.fraunhofer.de/server/api/core/bitstream...

Ygg2 1y ago

Conclusion is scathing: > The findings highlight significant variations in the theoretical detection capabilities of these techniques and reveal that, in practice, the implementations of most available sanitizers fall short of their conceptual potential. Furthermore, the evaluation demonstrates the complexities and diversity of memory bugs in C/C++, as well as the challenges associated with detecting them. For instance, our results show that SoftBound+CETS, a conceptually complete sanitizer, misses nearly a quarter of spatial memory bugs in its original implementation, while ASan, likely the most widely used memory sanitizer, cannot detect 50% of use-after-* bugs and any non-linear overflows and under- flows. Ultimately, our evaluation concludes that no sanitizer currently provides complete temporal or spatial memory safety

bgwalter 1y ago

If SoftBound+CETS has the best results, why does Fraunhofer not sponsor the creation of a Debian package? It is unmaintained: https://github.com/Fraunhofer-AISEC/softboundcets https://github.com/Fraunhofer-AISEC/softboundcets

rurban 1y ago
With this commit being the actual llvm patch for Softbounds+CETS: https://github.com/Fraunhofer-AISEC/softboundcets-llvm-project/commit/c014965edf13a3745e17c175408fa1ce9082f74c https://github.com/Fraunhofer-AISEC/softboundcets-llvm-proje... x86, x64 only. LTO only.

lou1306 1y ago

Weird that Infer [1] was not included in the evaluation. It supports C/C++ and its underlying reasoning framework (Separation Logic [2]) is exactly geared towards checking memory safety. [1] https://fbinfer.com/ https://fbinfer.com/ [2] https://en.wikipedia.org/wiki/Separation_logic https://en.wikipedia.org/wiki/Separation_logic

UncleMeat 1y ago
Sanitizers are runtime tools, not static analysis tools. Static analyzers are also virtually never sound as sound tools produce an outrageous number of false positives, especially when languages that so easily permit nonlocal mutation.

rwmj 1y ago

I wonder how true the assertion "This performance is partly achieved by sacrificing memory safety" is today. I suspect a sufficiently advanced compiler can remove bounds checks where they are provably unnecessary, and a sufficiently advanced CPU can run the remaining checks in parallel with the array accesses. But it'd be interesting if there's been any research on that. (A sufficiently advanced programming language can avoid the entire issue by writing loops as map, fold, etc. but we're talking about C here.)

bluGill 1y ago
There are a few things that cannot be done as fast in rust, but those are rare to need in the real world. Most of the things rust cannot do are around sharing memory between threads with locks - Humans have a very hard time getting code that does this to work correctly and usually have race conditions because they analysed the problem wrong.
- rurban 1y ago
  That's actually easy to do, if you care about concurrency safety at first when designing a language and it's stdlib. Just forbid blocking IO and no locks needed.
Someone 1y ago
> I suspect a sufficiently advanced compiler can remove bounds checks where they are provably unnecessary, That’s true by definition, isn’t it? > and a sufficiently advanced CPU can run the remaining checks in parallel with the array accesses. But it still would slow down the program, as the CPU would have to commit resources to that bound checking that it then cannot use for doing other things.
MindSpunk 1y ago
> A sufficiently advanced compiler Much like how Intel Itanium would've taken over the world with the VLIW revolution if only we had one of these 'sufficiently advanced compilers'. It's a very vague statement that doesn't really mesh with reality. Decades of research in compilers have yet to yield one. I think you're right, and it's up to the programming language to provide optimization friendly primitives that fit into the mold of what we can prove as in-bounds. Speculative execution will never be free because those bounds checks will take up space in the execution pipeline that could've been used for other instructions.

ape4 1y ago

Seems like it should have "C/C++" in the title. Or maybe that's understood?

ben-schaaf 1y ago
Sanitizers aren't just for C/C++. Rust, go, D all have at least asan support.
- vlovich123 1y ago
  Rust supports all the same sanitizers as c/C++ last I checked (at least UBSAN, TSAN, MSAN and ASAN which are the only ones I’ve ever seen used in practice on c/c++).

pizlonator 1y ago

I wonder if their test cases are available. Fil-C is specifically engineered to catch everything so it would be interesting to check it against their tests

ev234 1y ago
[dead]

pizlonator 1y ago

Wait hold up the same authors did MESH: https://arxiv.org/pdf/2108.08683 https://arxiv.org/pdf/2108.08683 So why isn't MESH part of the evaluation? And why isn't it mentioned even once in the paper?

rurban 1y ago
Weird indeed. But I also miss valgrind, which is precise, just very slow. I wonder if valgrind detects intra-object OOBAs.
- ev234 1y ago
  Valgrind's Memcheck is included. It does not detect intra-object OOBAs.
  - rurban 1y ago
    Ah thanks, missed that
- pizlonator 1y ago
  No, it's not that precise. Valgrind is based on tagging whether a memory location is live or not. So, it would not just fail the intra-object OOB test, it would also fail the "out of bounds but in bounds" case: https://github.com/pizlonator/llvm-project-deluge/blob/deluge/invisicaps_by_example.md#out-of-bounds-but-in-bounds https://github.com/pizlonator/llvm-project-deluge/blob/delug...