9 ms·

Imposing memory security in C [video]

64 points by transpute 2y ago

pizlonator 2y ago
To me, "memory safety" really means: - There are clearly stated guarantees, like "a pointer cannot access outside the bounds of its allocation" and "if you free an object while there are still pointers to it then those pointers cannot be dereferenced". These guarantees should be something you can reason about formally, and they should be falsifiable. Not sure this presentation really has that. It's not clear what they prevent, and what they don't prevent. - There is no way to break out of the clearly stated guarantees. Totally unclear that whatever guarantees they have are actually guarded against in all cases. For example, what if a tmp_alloc'd object pointer escapes into another tmp_alloc'd object with different lifetime - I get that they wouldn't write code that does that intentionally, but "memory safety" to me means that if you did write that code, you'd either get a compile error or a runtime error. It's possible to ascribe clearly stated guarantees to C and to make it impossible to break out of them (Fil-C and CHERI both achieve that).
- johnnyjeans 2y ago
  > There is no way to break out of the clearly stated guarantees. I disagree on this, and having escape hatches is critically important. Are we really going to call Rust or Haskell memory unsafe because they offer ways to break their safety guarantees?
  - pizlonator 2y ago
    I think that Rust's clearly stated guarantee holds if you never say "unsafe". That's still a clear statement, because it's trivial to tell if you used "unsafe" or not.
    - jeffrallen 2y ago
      But the Rust ecosystem is littered with unsafe, so good luck getting the actual benefits of Rust. :(
      - pizlonator 2y ago
        Great reason to use Fil-C
      - woodruffw 2y ago
        The actual benefits seem manifest: there aren't nearly as many public reports of memory corruption (much less exploitable corruption) in Rust components, even when those components make extensive use of unsafety directly or transitively. (This seems like one of those "throw the baby out with the bathwater" cases that people relitigate around Rust -- there's ample empirical evidence that building safe abstractions around unsafe primitives works well.)
        pizlonator 2y ago
        > The actual benefits seem manifest: there aren't nearly as many public reports of memory corruption (much less exploitable corruption) in Rust components, even when those components make extensive use of unsafety directly or transitively. Not sure the data is clean enough to draw meaningful conclusions because of confounding factors. The biggest confounding factor is that Rust is relatively new, code written in it is even newer, and folks who research vulns may not have applied the same level of anger to Rust as to C. That said, your point about "throwing the baby out with the bathwater" is well taken. I would expect that Rust has much fewer vulns than C/C++. My point is only that it's an unproven expectation.
        woodruffw 2y ago
        > Not sure the data is clean enough to draw meaningful conclusions because of confounding factors. I'm thinking of things like the Windows user- and kernel-mode font parsers; these have a pretty long and steady public history of exploitation that seems to have mostly stopped with the Rust rewrite 1-2 years ago. I don't think that's because vuln researches have stopped looking at them! But yeah, I would like it if Google and Microsoft (among others) would put more hard data out there. I don't think of the Windows kernel teams as typically suffering from hype-driven development, so my abductive conclusion is that they have strong supporting data internally. Edit: here's a hard data source from Google, showing that Rust has contributed to a marked decline in memory unsafety in Android[1]. [1]: https://security.googleblog.com/2022/12/memory-safe-languages-in-android-13.html https://security.googleblog.com/2022/12/memory-safe-language...
    - the__alchemist 2y ago
      Trivial is not a word I would use here! Rust's `unsafe` gets fuzzy as you transverse an operation's dependencies! There are many applications where marking a function as `unsafe` is subjective.
      - pizlonator 2y ago
        True. I do think the ideal kind of memory safe language either has no "unsafe", or has an "unsafe" feature that only needs to be used in super rare an obscure cases (Java is like that, sort of). Fil-C has no "unsafe", so in that sense Fil-C is safer than Rust. You don't need an escape hatch if the memory safety guarantees are dialed in just right.
        titzer 2y ago
        In Virgil, native targets have a couple of unsafe operations available, one of which is to be able to forge a closure from a pair of a code pointer and an object reference. This is used, e.g. to implement Wizard's JIT and fast interpreter, which generate new machine code at runtime. I can't imagine the level of proof necessary to make that safe--and not just the safety of the generated machine code, but it often lacks bounds checks because it relies on running verified Wasm bytecode, which cannot go out of bounds. So the proof would have to include a complete proof of correctness for the code validation algorithm (i.e. part of Wizard itself).
        pizlonator 2y ago
        I dream of a JIT API that lets you propose machine code that is checked using an abstract interpreter to ensure that you have adequate checks to stay within the host language's type system. Someday, man
        porridgeraisin 2y ago
        I don't get this, can you explain?
    - johnnyjeans 2y ago
      Maybe I just misinterpret what you mean. When you say "no way" and "all cases", I take your meaning literally. The existence of pointers to bypass the borrow checker, disabling runtime bounds checks and unsafe blocks are exactly that: escape hatches to break Rust's safety, in the same way type-casting is an escape hatch to break C's (anemic) type safety, and unsafePerformIO in Haskell is an escape hatch to break every bone in your body.
      - pizlonator 2y ago
        That’s fair. FWIW, Fil-C’s guarantees are literally what you want. There’s no escape.
- kstrauser 2y ago
  Thinking aloud, and this is probably a bad idea for reasons I haven’t thought of. What if pointers were a combination of values, like a 32 bit “zone” plus a 32 bit “offset” (where 32/32 is probably really 28/36 or something that allows >4GB allocations, but let’s figure that out later). Then each malloc() could increment the zone number, or pick an unused one randomly, so that there’s enormous space between consecutive allocs and an address wouldn’t be reissued quickly. A dangling pointer would the point at an address that isn’t mapped at all until possibly 2^32 malloc()s later. It wouldn’t help with long-lived dangling pointers, but would catch accessing a pointer right after it was freed. I guess, more generally, why are addresses reused before they absolutely must be?
  - pizlonator 2y ago
    That’s a way of achieving safety that has so many costs: - physical fragmentation (you won’t be able to put two live objects into the same page) - virtual fragmentation (there’s kernel memory cost to having huge reservations) - 32 bit size limit Fil-C achieves safety without any of those compromises.
    - kstrauser 2y ago
      For sure. I'm under no illusion that it wouldn't be costly. What I'm trying to suss out is whether libc could hypothetically change to give better safety to existing compiled binaries.
      - pizlonator 2y ago
        Two things: - The costs of your solution really are prohibitive. Lots of stuff just won't run. - "Better" isn't good enough because attackers are good at finding the loopholes. You need a guarantee.
  - throwawaymaths 2y ago
    you can do this easily with virtual memory, and IIRC Zig's general purpose allocator does under some circumstances (don't remember if its default or if it needs a flag).
- PaulDavisThe1st 2y ago
  > There are clearly stated guarantees, like "a pointer cannot access outside the bounds of its allocation" But that's not a pointer in anything like the sense of a C pointer. You'd need to reword that (as I know you've been doing with FiL-C) to be something more like: no reference to a (variable|allocation|object) may ever be used to access memory that is not a part of the object. Pointers are not that, and the work you've done in FiL-C to make them closer to that makes them also be "not pointers" in a classic sense. I'm OK with that, it just needs to be more clear.
  - pizlonator 2y ago
    Semantics. You can call Fil-C’s pointers whatever you like. You can call them capabilities if that works better for you. The point of my post is to enumerate the set of things you’d need to do to pointers to make them safe. If that then means we’ve created something that you wouldn’t call a pointer then like whatever
    - tredre3 2y ago
      > Semantics. If your goal is just to redefine the word then by all means, continue. But semantics are very important if your goal is to drive adoption of your ideas. You can't misuse a term and then get pissy when people don't understand you.
      - pizlonator 2y ago
        Seems like bro understood me just fine. In Fil-C, pointers are called “pointers”.
    - PaulDavisThe1st 2y ago
      And my point is that you cannot make C pointers safe. You can make something else that is safe, and you're clearly hard at work on that, which is great.
      - pizlonator 2y ago
- blacksqr 2y ago
  There are and have been many techniques and projects for making C more memory-safe. The crucial question it always comes down to is what performance hit do you take using them? That's why C has been on top for so long. Seat-of-the-pants hand-crafted C has always been the fastest high-level language.
  - WalterBright 2y ago
    I found that C programs rarely evolve beyond their initial design. The trouble is, it's hard to refactor C programs. For example, struct S { int a; }; struct S s; s.a = 3; struct S *p; p->a = 3; I.e. a . is for direct access, -> for indirect access. Let's say you want to change passing S by value to passing S by pointer. Now you have to update every use, instead of just the declaration. This is how it would work in D: struct S { int a; } S s; s.a = 3; S* p; p.a = 3; ref S q; q.a = 3; And so refactoring becomes much easier, and so happens more often.
    - WalterBright 2y ago
      > C has always been the fastest high-level language. C has another big speed problem. Strings are 0 terminated, rather than length terminated. This means constant scanning of strings to find their length. Even worse, the scanning of the string reloads the cache with the string contents, which is pretty bad for performance. Of course, you could use `struct String { char *p; size_t length; };` but since every library you want to connect to uses 0 terminated strings, you're out on your island all alone, so pragmatically it does not work. Another speed-destroying problem with C strings is you cannot take a substring that does not require allocating a new string and then copying the data. (Unless the substring is right-justified.) This is not fast in any universe. D uses length-denoted strings as a basic data type, and with string processing code, it is much faster than C. Substrings are quick and easy. You can still interface with C because D string literals implicitly convert to C string literals, as the literals are 0 terminated. So this works in D: printf("hello world!\n"); (People sometimes rag on me for still using printf, but printf is the most optimized and debugged library function in the world, so I take advantage!)
SV_BubbleTime 2y ago
I am in no way at all better than that guy. Not even sort of. I appreciate his talk. However, if I were to make a presentation based on my superior C practices, it would have to be implementation and example heavy. All of his rules sound great, except for when you have to break them or you don’t know how to do the things he’s talking about in your code, because you need to get something done today. It reads a little like “I’ve learned a lot of lessons over my career, you should learn my lessons. You’re welcome.”
- rrrix1 2y ago
  The talk was obviously extremely time-limited, as demonstrated when they basically skipped the last handful of slides and then it abruptly ended. I think for the time allocated, it was just right, and they did include a couple of examples where it made sense.
beardyw 2y ago
I remember chasing down a memory leak in my first commercial C code. Took me a long while to discover that if you allocate zero bytes you still have to free it! After that I took nothing for granted.
- weinzierl 2y ago
  It's not even guaranteed that it doesn't allocate, so a malloc(0) could cause an out of memory.
  - ignoramous 2y ago
    > malloc(0) could cause an out of memory tbh, 640K RAM ought to be enough for anybody.
    - weinzierl 2y ago
      For the last drop to make the cup run over it doesn't matter how big the cup is.
debatem1 2y ago
I don't think anyone ever doubted that a C program could be memory safe. The problem is knowing without exhaustive work whether yours is one of them. These aren't bad practices, but I don't think they satisfy that desire either.
cryptonector 2y ago
Nah. I use C a lot, but none of this is enough to make C safe. You really need the language and the tools to enforce discipline. Oh, and things like the cleanup attribute are not standard C either, so this is not portable code.
- throwawaymaths 2y ago
  usually portability in C includes the provision that you can drop in whatever #includes you want?
  - cryptonector 2y ago
    No, it's really not that simple at all.
    - throwawaymaths 2y ago
      Probably depends on the macro, but ok.
      - pjmlp 2y ago
        It starts by discovering how little most folks know from ISO C legalese versus what their compiler does, and it goes from there when adding anything else not part of the standard library.
- imtringued 2y ago
  I didn't see anything new either. What I would expect from C developers is this: * Run all CI with UBSAN. Create versions of popular distributions that build every single package with UBSAN just to catch the bugs. * Use design by contract patterns for pre and postconditions (library induced UB is popular in C++) * Use model checking software like CBMC to statically guarantee the absence of UB and validity of the contracts * Build a fuzzer for every method that cannot be formally verified This is the bare minimum needed to keep C/C++ safe. The same applies to unsafe Rust by the way.

64 points by transpute 2y ago

To me, "memory safety" really means: - There are clearly stated guarantees, like "a pointer cannot access outside the bounds of its allocation" and "if you free an object while there are still pointers to it then those pointers cannot be dereferenced". These guarantees should be something you can reason about formally, and they should be falsifiable. Not sure this presentation really has that. It's not clear what they prevent, and what they don't prevent. - There is no way to break out of the clearly stated guarantees. Totally unclear that whatever guarantees they have are actually guarded against in all cases. For example, what if a tmp_alloc'd object pointer escapes into another tmp_alloc'd object with different lifetime - I get that they wouldn't write code that does that intentionally, but "memory safety" to me means that if you did write that code, you'd either get a compile error or a runtime error. It's possible to ascribe clearly stated guarantees to C and to make it impossible to break out of them (Fil-C and CHERI both achieve that).

johnnyjeans 2y ago
> There is no way to break out of the clearly stated guarantees. I disagree on this, and having escape hatches is critically important. Are we really going to call Rust or Haskell memory unsafe because they offer ways to break their safety guarantees?
- pizlonator 2y ago
  I think that Rust's clearly stated guarantee holds if you never say "unsafe". That's still a clear statement, because it's trivial to tell if you used "unsafe" or not.
  - jeffrallen 2y ago
    But the Rust ecosystem is littered with unsafe, so good luck getting the actual benefits of Rust. :(
    - pizlonator 2y ago
      Great reason to use Fil-C
    - woodruffw 2y ago
      The actual benefits seem manifest: there aren't nearly as many public reports of memory corruption (much less exploitable corruption) in Rust components, even when those components make extensive use of unsafety directly or transitively. (This seems like one of those "throw the baby out with the bathwater" cases that people relitigate around Rust -- there's ample empirical evidence that building safe abstractions around unsafe primitives works well.)
      - pizlonator 2y ago
        > The actual benefits seem manifest: there aren't nearly as many public reports of memory corruption (much less exploitable corruption) in Rust components, even when those components make extensive use of unsafety directly or transitively. Not sure the data is clean enough to draw meaningful conclusions because of confounding factors. The biggest confounding factor is that Rust is relatively new, code written in it is even newer, and folks who research vulns may not have applied the same level of anger to Rust as to C. That said, your point about "throwing the baby out with the bathwater" is well taken. I would expect that Rust has much fewer vulns than C/C++. My point is only that it's an unproven expectation.
        woodruffw 2y ago
        > Not sure the data is clean enough to draw meaningful conclusions because of confounding factors. I'm thinking of things like the Windows user- and kernel-mode font parsers; these have a pretty long and steady public history of exploitation that seems to have mostly stopped with the Rust rewrite 1-2 years ago. I don't think that's because vuln researches have stopped looking at them! But yeah, I would like it if Google and Microsoft (among others) would put more hard data out there. I don't think of the Windows kernel teams as typically suffering from hype-driven development, so my abductive conclusion is that they have strong supporting data internally. Edit: here's a hard data source from Google, showing that Rust has contributed to a marked decline in memory unsafety in Android[1]. [1]: https://security.googleblog.com/2022/12/memory-safe-languages-in-android-13.html https://security.googleblog.com/2022/12/memory-safe-language...
  - the__alchemist 2y ago
    Trivial is not a word I would use here! Rust's `unsafe` gets fuzzy as you transverse an operation's dependencies! There are many applications where marking a function as `unsafe` is subjective.
    - pizlonator 2y ago
      True. I do think the ideal kind of memory safe language either has no "unsafe", or has an "unsafe" feature that only needs to be used in super rare an obscure cases (Java is like that, sort of). Fil-C has no "unsafe", so in that sense Fil-C is safer than Rust. You don't need an escape hatch if the memory safety guarantees are dialed in just right.
      - titzer 2y ago
        In Virgil, native targets have a couple of unsafe operations available, one of which is to be able to forge a closure from a pair of a code pointer and an object reference. This is used, e.g. to implement Wizard's JIT and fast interpreter, which generate new machine code at runtime. I can't imagine the level of proof necessary to make that safe--and not just the safety of the generated machine code, but it often lacks bounds checks because it relies on running verified Wasm bytecode, which cannot go out of bounds. So the proof would have to include a complete proof of correctness for the code validation algorithm (i.e. part of Wizard itself).
        pizlonator 2y ago
        I dream of a JIT API that lets you propose machine code that is checked using an abstract interpreter to ensure that you have adequate checks to stay within the host language's type system. Someday, man
        porridgeraisin 2y ago
        I don't get this, can you explain?
  - johnnyjeans 2y ago
    Maybe I just misinterpret what you mean. When you say "no way" and "all cases", I take your meaning literally. The existence of pointers to bypass the borrow checker, disabling runtime bounds checks and unsafe blocks are exactly that: escape hatches to break Rust's safety, in the same way type-casting is an escape hatch to break C's (anemic) type safety, and unsafePerformIO in Haskell is an escape hatch to break every bone in your body.
    - pizlonator 2y ago
      That’s fair. FWIW, Fil-C’s guarantees are literally what you want. There’s no escape.
kstrauser 2y ago
Thinking aloud, and this is probably a bad idea for reasons I haven’t thought of. What if pointers were a combination of values, like a 32 bit “zone” plus a 32 bit “offset” (where 32/32 is probably really 28/36 or something that allows >4GB allocations, but let’s figure that out later). Then each malloc() could increment the zone number, or pick an unused one randomly, so that there’s enormous space between consecutive allocs and an address wouldn’t be reissued quickly. A dangling pointer would the point at an address that isn’t mapped at all until possibly 2^32 malloc()s later. It wouldn’t help with long-lived dangling pointers, but would catch accessing a pointer right after it was freed. I guess, more generally, why are addresses reused before they absolutely must be?
- pizlonator 2y ago
  That’s a way of achieving safety that has so many costs: - physical fragmentation (you won’t be able to put two live objects into the same page) - virtual fragmentation (there’s kernel memory cost to having huge reservations) - 32 bit size limit Fil-C achieves safety without any of those compromises.
  - kstrauser 2y ago
    For sure. I'm under no illusion that it wouldn't be costly. What I'm trying to suss out is whether libc could hypothetically change to give better safety to existing compiled binaries.
    - pizlonator 2y ago
      Two things: - The costs of your solution really are prohibitive. Lots of stuff just won't run. - "Better" isn't good enough because attackers are good at finding the loopholes. You need a guarantee.
- throwawaymaths 2y ago
  you can do this easily with virtual memory, and IIRC Zig's general purpose allocator does under some circumstances (don't remember if its default or if it needs a flag).
PaulDavisThe1st 2y ago
> There are clearly stated guarantees, like "a pointer cannot access outside the bounds of its allocation" But that's not a pointer in anything like the sense of a C pointer. You'd need to reword that (as I know you've been doing with FiL-C) to be something more like: no reference to a (variable|allocation|object) may ever be used to access memory that is not a part of the object. Pointers are not that, and the work you've done in FiL-C to make them closer to that makes them also be "not pointers" in a classic sense. I'm OK with that, it just needs to be more clear.
- pizlonator 2y ago
  Semantics. You can call Fil-C’s pointers whatever you like. You can call them capabilities if that works better for you. The point of my post is to enumerate the set of things you’d need to do to pointers to make them safe. If that then means we’ve created something that you wouldn’t call a pointer then like whatever
  - tredre3 2y ago
    > Semantics. If your goal is just to redefine the word then by all means, continue. But semantics are very important if your goal is to drive adoption of your ideas. You can't misuse a term and then get pissy when people don't understand you.
    - pizlonator 2y ago
      Seems like bro understood me just fine. In Fil-C, pointers are called “pointers”.
  - PaulDavisThe1st 2y ago
    And my point is that you cannot make C pointers safe. You can make something else that is safe, and you're clearly hard at work on that, which is great.
    - pizlonator 2y ago
blacksqr 2y ago
There are and have been many techniques and projects for making C more memory-safe. The crucial question it always comes down to is what performance hit do you take using them? That's why C has been on top for so long. Seat-of-the-pants hand-crafted C has always been the fastest high-level language.
- WalterBright 2y ago
  I found that C programs rarely evolve beyond their initial design. The trouble is, it's hard to refactor C programs. For example, struct S { int a; }; struct S s; s.a = 3; struct S *p; p->a = 3; I.e. a . is for direct access, -> for indirect access. Let's say you want to change passing S by value to passing S by pointer. Now you have to update every use, instead of just the declaration. This is how it would work in D: struct S { int a; } S s; s.a = 3; S* p; p.a = 3; ref S q; q.a = 3; And so refactoring becomes much easier, and so happens more often.
  - WalterBright 2y ago
    > C has always been the fastest high-level language. C has another big speed problem. Strings are 0 terminated, rather than length terminated. This means constant scanning of strings to find their length. Even worse, the scanning of the string reloads the cache with the string contents, which is pretty bad for performance. Of course, you could use `struct String { char *p; size_t length; };` but since every library you want to connect to uses 0 terminated strings, you're out on your island all alone, so pragmatically it does not work. Another speed-destroying problem with C strings is you cannot take a substring that does not require allocating a new string and then copying the data. (Unless the substring is right-justified.) This is not fast in any universe. D uses length-denoted strings as a basic data type, and with string processing code, it is much faster than C. Substrings are quick and easy. You can still interface with C because D string literals implicitly convert to C string literals, as the literals are 0 terminated. So this works in D: printf("hello world!\n"); (People sometimes rag on me for still using printf, but printf is the most optimized and debugged library function in the world, so I take advantage!)

SV_BubbleTime 2y ago

I am in no way at all better than that guy. Not even sort of. I appreciate his talk. However, if I were to make a presentation based on my superior C practices, it would have to be implementation and example heavy. All of his rules sound great, except for when you have to break them or you don’t know how to do the things he’s talking about in your code, because you need to get something done today. It reads a little like “I’ve learned a lot of lessons over my career, you should learn my lessons. You’re welcome.”

rrrix1 2y ago
The talk was obviously extremely time-limited, as demonstrated when they basically skipped the last handful of slides and then it abruptly ended. I think for the time allocated, it was just right, and they did include a couple of examples where it made sense.

beardyw 2y ago

I remember chasing down a memory leak in my first commercial C code. Took me a long while to discover that if you allocate zero bytes you still have to free it! After that I took nothing for granted.

weinzierl 2y ago
It's not even guaranteed that it doesn't allocate, so a malloc(0) could cause an out of memory.
- ignoramous 2y ago
  > malloc(0) could cause an out of memory tbh, 640K RAM ought to be enough for anybody.
  - weinzierl 2y ago
    For the last drop to make the cup run over it doesn't matter how big the cup is.

debatem1 2y ago

I don't think anyone ever doubted that a C program could be memory safe. The problem is knowing without exhaustive work whether yours is one of them. These aren't bad practices, but I don't think they satisfy that desire either.

cryptonector 2y ago

Nah. I use C a lot, but none of this is enough to make C safe. You really need the language and the tools to enforce discipline. Oh, and things like the cleanup attribute are not standard C either, so this is not portable code.

throwawaymaths 2y ago
usually portability in C includes the provision that you can drop in whatever #includes you want?
- cryptonector 2y ago
  No, it's really not that simple at all.
  - throwawaymaths 2y ago
    Probably depends on the macro, but ok.
    - pjmlp 2y ago
      It starts by discovering how little most folks know from ISO C legalese versus what their compiler does, and it goes from there when adding anything else not part of the standard library.
imtringued 2y ago
I didn't see anything new either. What I would expect from C developers is this: * Run all CI with UBSAN. Create versions of popular distributions that build every single package with UBSAN just to catch the bugs. * Use design by contract patterns for pre and postconditions (library induced UB is popular in C++) * Use model checking software like CBMC to statically guarantee the absence of UB and validity of the contracts * Build a fuzzer for every method that cannot be formally verified This is the bare minimum needed to keep C/C++ safe. The same applies to unsafe Rust by the way.