7 ms·

...Except, of course, the the compiler can and will optimize away any such memset. Do not attempt to write secure software with C / C++. It is a Bad Idea(TM).

...Except, of course, the the compiler can and will optimize away any such memset. Do not attempt to write secure software with C / C++. It is a Bad Idea(TM). Because what you wrote is not what gets run. For instance: there is no way to write a portable secure memset in portable C / C++. (You can write a "secure" memset that works in all current compilers, but that is not the same thing. What doesn't get optimized now can and will be optimized tomorrow.)

deleted 11y ago
[deleted]
kosmic_k 11y ago
So what is someone working with embedded devices for IoT supposed to do? Much of that work is based on ARM C/C++ Compilers.
- pjc50 11y ago
  In practice, of course, memset actually works, because it's a function and the compiler's usage tracing is nowhere near able to spot that you don't reference those zeros that you write. (IoT security is doom for other reasons though, mostly UI, updatability and cloud services)
  - vardump 11y ago
    Really? In my experience most compilers treat it pretty much like an intrinsic and generate specialized set of instructions if they can.
  - TheLoneWolfling 11y ago
    For now. And then that code you wrote now silently becomes deadly a few years down the line.
    - pjc50 11y ago
      I would be very interested to see the compiler that can optimise away memset across a shared library boundary.
      - TheLoneWolfling 11y ago
        A JITter would be able to do that. And the JVM can (and will) do the equivalent for Java code. Remember: there is nothing that specifies that C / C++ needs to be compiled. Also: you could have said the same twenty years ago about many of the optimizations that currently compilers do.
      - pcwalton 11y ago
        LLVM can and will do it. It will assume it knows what a function named "memcpy" (for example) does and optimizes accordingly. (Look at TargetLibraryInfo.cpp and grep for LibFunc::memset in, for example, SimplifyLibCalls.cpp.) (That said, I think TheLoneWolfling is being too strong with his/her claims. You can get modern compilers to avoid dangerous optimizations; it's just not for the faint of heart.)
        TheLoneWolfling 11y ago
        I never said you couldn't get a particular compiler to. Or indeed, all current compilers. I am saying that it's impossible to do so and remain in the realm of portable C / C++. There is a distinction.
        TheLoneWolfling 11y ago
        Also: isn't that a bug? Is there something in a C / C++ standard that states that a function named "memcpy" (for example) is necessarily the normal function?
        pcwalton 11y ago
        Compilers have been doing this for a long time. The optimizations that this enables are essential for performance. They shouldn't stop; if the spec prohibits it, the spec should change (and if it doesn't, the compilers should ignore the spec).
- TheLoneWolfling 11y ago
  Either: * Use a specific compiler and verify. * Don't use C / C++. * Panic.
  - moyix 11y ago
    C11 actually adds the memset_s function, which is guaranteed by the language spec not to be optimized away: > memset may be optimized away (under the as-if rules) if the object modified by this function is not accessed again for the rest of its lifetime. For that reason, this function cannot be used to scrub memory (e.g. to fill an array that stored a password with zeroes). This optimization is prohibited for memset_s: it is guaranteed to perform the memory write. http://en.cppreference.com/w/c/string/byte/memset http://en.cppreference.com/w/c/string/byte/memset
    - TheLoneWolfling 11y ago
      Except, of course, that memset_s is still not enough. The compiler can and will copy things around, and it is not required to memset_s said cop(y)/(ies) away.
- coryrc 11y ago
  1. Compile without optimization (maybe just the crypto parts, if that's possible). 2. Write all crypto stuff in assembly, link in as static binary.
  - TheLoneWolfling 11y ago
    Neither of those work in general. W.r.t. 1, the compiler's definition of no optimization today is not the same thing as it was last version, or will be next version. For instance, on IA-64 there are things the compiler has to do that are typically considered optimizations. W.r.t. 2, you have to make sure there is no link-time optimization happening.
    - epsylon 11y ago
      You could also dynamically link.
      - TheLoneWolfling 11y ago
        That does not solve the problem. That only hides it and means it will be deadly later. For instance, when someone runs it in an emulator for backwards compatibility purposes. Or when someone runs it in a JITter. Or even just if the compiler decides to special-case for the existing link target.
    - mpu 11y ago
      Lto does not work on assembly, it only works if some IR is stored in the .o files (like gimple for gcc) iirc.
      - TheLoneWolfling 11y ago
        Currently. However, that is not an inherent restriction - that is only a restriction on current compilers. It is entirely possible for a compiler to read the assembly of things being linked and optimize based on that.
- Someone 11y ago
  Use a compiler that has extensions that _do_ guarantee that memory gets erased. That's what the gcc function attributes are for. Alternatively, use a library that the C compiler doesn't know so much about that it will attempt to remove calling functions in them. If you copy your standard library's memset in a separate DLL that is not the standard C library, the compiler will not even see the code during compilation, so it has to compile a function call. The linker (or a JIT in your C runtime) is allowed to remove calls to the function, if it can prove that it doesn't have side effects. However, to prove that, it has to look at the assembly of the function; it cannot use the way simpler heuristic "it came from <memory.h> and is called memset"
  - nkurz 11y ago
    If you copy your standard library's memset in a separate DLL that is not the standard C library, the compiler will not even see the code during compilation, so it has to compile a function call. Although I'd like things to behave this way, I don't think this is true. The C standard library was incorporated into the language spec for C89. The behaviors of the named functions within it are specified, and the compiler is allowed to inline it's own version (ignoring your custom code) and then optimize out the inlined portion. So while it's possible that the external linkage approach still works with certain compilers, it's not portable. I believe you are OK with the external approach if you use a non-standard name (my_secure_memset_pretty_please()), but that just shifts the problem to forcing the compiler to generate your external function without making the same dangerous optimizations. In the end, I fear you are left with three options: blind faith, non-standard language extensions, or switching to a more secure language (likely assembly). If there are other options, I love to hear about them.
davtbaum 11y ago
> ...Except, of course, the the compiler can and will optimize away any such memset. It's dependent on the function attributes of memset (e.g. __attribute__((pure))) - it won't always be optimized out. https://www.cs.auckland.ac.nz/references/c/gcc4.7/Function-Attributes.html https://www.cs.auckland.ac.nz/references/c/gcc4.7/Function-A...
- TheLoneWolfling 11y ago
  I am not talking about if it does currently optimize away a function call to an alternative to memset, I am talking about if it is allowed to optimize it away. The compiler is allowed to optimize away that function call regardless of if it is memset or your own alternative. Note that that is not the same thing as saying if it does currently optimize away that function call.
Joky 11y ago
I don't get it, the compiler operates within the memory model specified by the language. If it "optimizes" a memset it does not change the behavior of the program (or it is a bug in the compiler which is a different topic).
- TheLoneWolfling 11y ago
  You're correct - the compiler operates within the memory model of the language. But C / C++'s memory model is broken w.r.t. security. There is no way to ensure that something is actually overwritten, because under the memory models of C and C++ you cannot ever read that memory again, even though in actuality you can.
  - ikeboy 11y ago
    Why not generate random data then read it? Or even a constant, then read it?
    - TheLoneWolfling 11y ago
      Because the compiler will optimize it out. Even if you return the random data / constant the compiler will optimize out the store to the variable and just pass it through directly.
  - Joky 11y ago
    I believe volatile force the compiler to issue a real write to memory and can't be optimized away.
    - TheLoneWolfling 11y ago
      Except, of course, that the compiler can perfectly legally copy the variable behind the scenes and not overwrite the copy. I mean it when I say you cannot.
      - Joky 11y ago
kllrnohj 11y ago
> For instance: there is no way to write a portable secure memset in portable C / C++. Of course there is, you just use the volatile keyword. volatile guarantees that all read/writes have corresponding memory accesses and cannot be optimized away. It's not going to be as fast as memset but it's definitely portable and it won't be THAT slow. Then for platforms that have memset_s defer to that instead, otherwise fallback to the totally portable volatile + for loop.
- vvanders 11y ago
  Volatile should only be used with hardware registers. It doesn't do exactly what you want here. It will guarantee that memory will be accessed but it doesn't guarantee the ordering which can lead to some really nasty behaviour. The only place that keywork should be used is a qualifier for member functions or when used in an embedded sense. It's not well defined outside of that scope.
  - kllrnohj 11y ago
    volatile's designed purpose is for memory-mapped things, not hardware registers.
- dezgeg 11y ago
  > Of course there is, you just use the volatile keyword. Colin Percival of FreeBSD disagrees: http://www.daemonology.net/blog/2014-09-04-how-to-zero-a-buffer.html http://www.daemonology.net/blog/2014-09-04-how-to-zero-a-buf...
  - TheLoneWolfling 11y ago
    And that doesn't even get into the other aspect of it: Namely that C / C++ allows temporary copies of variables that are not cleared afterwards. The most obvious case of this being things being temporarily copied into registers / stack, but there are other examples as well.
  - kllrnohj 11y ago
    Not really. He just says if the compiler can optimize away prior to it being declared volatile.
trentnelson 11y ago
SecureZeroMemory: https://msdn.microsoft.com/en-us/library/windows/desktop/aa366877(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...
- TheLoneWolfling 11y ago
  ...I do not think the word "portable" means what you think it means.
overgard 11y ago
You could just compile with optimizations off.
- TheLoneWolfling 11y ago
  There is no such thing in portable C / C++. There are some ISAs that pretty much require optimizations, for instance.
jimjimjim 11y ago
cheat. after resetting the memory, copy the first and last bytes and log them out somewhere. that'll prevent any optimizations
- TheLoneWolfling 11y ago
  If only it were so simple. You underestimate the evilness of compilers. The compiler can (and will!) just propagate the values through directly and skip the memset.
lmm 11y ago
I'm not aware of any portable language that offers semantics that guarantee data is never rendered accessible to the outside world (I could well believe that current implementations don't, but I'm not aware of anything that specifies it). It always comes down to platform-specific APIs. But that's not a reason to not use C - unless you can suggest a better alternative?
- TheLoneWolfling 11y ago
  Quite frankly, assembly currently is about it. And that's not the most portable thing out there, I know. I think that C or C++ could, without too much effort, support semantics that would allow for this sort of thing. Something as simple as a "secure" keyword that could be applied to variables (where it means "leak as little as possible about this variable when it goes out of scope") or functions (where it means the same, but for the function itself and all locals of the function).

deleted 11y ago

[deleted]

kosmic_k 11y ago

So what is someone working with embedded devices for IoT supposed to do? Much of that work is based on ARM C/C++ Compilers.

pjc50 11y ago
In practice, of course, memset actually works, because it's a function and the compiler's usage tracing is nowhere near able to spot that you don't reference those zeros that you write. (IoT security is doom for other reasons though, mostly UI, updatability and cloud services)
- vardump 11y ago
  Really? In my experience most compilers treat it pretty much like an intrinsic and generate specialized set of instructions if they can.
- TheLoneWolfling 11y ago
  For now. And then that code you wrote now silently becomes deadly a few years down the line.
  - pjc50 11y ago
    I would be very interested to see the compiler that can optimise away memset across a shared library boundary.
    - TheLoneWolfling 11y ago
      A JITter would be able to do that. And the JVM can (and will) do the equivalent for Java code. Remember: there is nothing that specifies that C / C++ needs to be compiled. Also: you could have said the same twenty years ago about many of the optimizations that currently compilers do.
    - pcwalton 11y ago
      LLVM can and will do it. It will assume it knows what a function named "memcpy" (for example) does and optimizes accordingly. (Look at TargetLibraryInfo.cpp and grep for LibFunc::memset in, for example, SimplifyLibCalls.cpp.) (That said, I think TheLoneWolfling is being too strong with his/her claims. You can get modern compilers to avoid dangerous optimizations; it's just not for the faint of heart.)
      - TheLoneWolfling 11y ago
        I never said you couldn't get a particular compiler to. Or indeed, all current compilers. I am saying that it's impossible to do so and remain in the realm of portable C / C++. There is a distinction.
      - TheLoneWolfling 11y ago
        Also: isn't that a bug? Is there something in a C / C++ standard that states that a function named "memcpy" (for example) is necessarily the normal function?
        pcwalton 11y ago
        Compilers have been doing this for a long time. The optimizations that this enables are essential for performance. They shouldn't stop; if the spec prohibits it, the spec should change (and if it doesn't, the compilers should ignore the spec).
TheLoneWolfling 11y ago
Either: * Use a specific compiler and verify. * Don't use C / C++. * Panic.
- moyix 11y ago
  C11 actually adds the memset_s function, which is guaranteed by the language spec not to be optimized away: > memset may be optimized away (under the as-if rules) if the object modified by this function is not accessed again for the rest of its lifetime. For that reason, this function cannot be used to scrub memory (e.g. to fill an array that stored a password with zeroes). This optimization is prohibited for memset_s: it is guaranteed to perform the memory write. http://en.cppreference.com/w/c/string/byte/memset http://en.cppreference.com/w/c/string/byte/memset
  - TheLoneWolfling 11y ago
    Except, of course, that memset_s is still not enough. The compiler can and will copy things around, and it is not required to memset_s said cop(y)/(ies) away.
coryrc 11y ago
1. Compile without optimization (maybe just the crypto parts, if that's possible). 2. Write all crypto stuff in assembly, link in as static binary.
- TheLoneWolfling 11y ago
  Neither of those work in general. W.r.t. 1, the compiler's definition of no optimization today is not the same thing as it was last version, or will be next version. For instance, on IA-64 there are things the compiler has to do that are typically considered optimizations. W.r.t. 2, you have to make sure there is no link-time optimization happening.
  - epsylon 11y ago
    You could also dynamically link.
    - TheLoneWolfling 11y ago
      That does not solve the problem. That only hides it and means it will be deadly later. For instance, when someone runs it in an emulator for backwards compatibility purposes. Or when someone runs it in a JITter. Or even just if the compiler decides to special-case for the existing link target.
  - mpu 11y ago
    Lto does not work on assembly, it only works if some IR is stored in the .o files (like gimple for gcc) iirc.
    - TheLoneWolfling 11y ago
      Currently. However, that is not an inherent restriction - that is only a restriction on current compilers. It is entirely possible for a compiler to read the assembly of things being linked and optimize based on that.
Someone 11y ago
Use a compiler that has extensions that _do_ guarantee that memory gets erased. That's what the gcc function attributes are for. Alternatively, use a library that the C compiler doesn't know so much about that it will attempt to remove calling functions in them. If you copy your standard library's memset in a separate DLL that is not the standard C library, the compiler will not even see the code during compilation, so it has to compile a function call. The linker (or a JIT in your C runtime) is allowed to remove calls to the function, if it can prove that it doesn't have side effects. However, to prove that, it has to look at the assembly of the function; it cannot use the way simpler heuristic "it came from <memory.h> and is called memset"
- nkurz 11y ago
  If you copy your standard library's memset in a separate DLL that is not the standard C library, the compiler will not even see the code during compilation, so it has to compile a function call. Although I'd like things to behave this way, I don't think this is true. The C standard library was incorporated into the language spec for C89. The behaviors of the named functions within it are specified, and the compiler is allowed to inline it's own version (ignoring your custom code) and then optimize out the inlined portion. So while it's possible that the external linkage approach still works with certain compilers, it's not portable. I believe you are OK with the external approach if you use a non-standard name (my_secure_memset_pretty_please()), but that just shifts the problem to forcing the compiler to generate your external function without making the same dangerous optimizations. In the end, I fear you are left with three options: blind faith, non-standard language extensions, or switching to a more secure language (likely assembly). If there are other options, I love to hear about them.

davtbaum 11y ago

> ...Except, of course, the the compiler can and will optimize away any such memset. It's dependent on the function attributes of memset (e.g. __attribute__((pure))) - it won't always be optimized out. https://www.cs.auckland.ac.nz/references/c/gcc4.7/Function-Attributes.html https://www.cs.auckland.ac.nz/references/c/gcc4.7/Function-A...

TheLoneWolfling 11y ago
I am not talking about if it does currently optimize away a function call to an alternative to memset, I am talking about if it is allowed to optimize it away. The compiler is allowed to optimize away that function call regardless of if it is memset or your own alternative. Note that that is not the same thing as saying if it does currently optimize away that function call.

Joky 11y ago

I don't get it, the compiler operates within the memory model specified by the language. If it "optimizes" a memset it does not change the behavior of the program (or it is a bug in the compiler which is a different topic).

TheLoneWolfling 11y ago
You're correct - the compiler operates within the memory model of the language. But C / C++'s memory model is broken w.r.t. security. There is no way to ensure that something is actually overwritten, because under the memory models of C and C++ you cannot ever read that memory again, even though in actuality you can.
- ikeboy 11y ago
  Why not generate random data then read it? Or even a constant, then read it?
  - TheLoneWolfling 11y ago
    Because the compiler will optimize it out. Even if you return the random data / constant the compiler will optimize out the store to the variable and just pass it through directly.
- Joky 11y ago
  I believe volatile force the compiler to issue a real write to memory and can't be optimized away.
  - TheLoneWolfling 11y ago
    Except, of course, that the compiler can perfectly legally copy the variable behind the scenes and not overwrite the copy. I mean it when I say you cannot.
    - Joky 11y ago

kllrnohj 11y ago

> For instance: there is no way to write a portable secure memset in portable C / C++. Of course there is, you just use the volatile keyword. volatile guarantees that all read/writes have corresponding memory accesses and cannot be optimized away. It's not going to be as fast as memset but it's definitely portable and it won't be THAT slow. Then for platforms that have memset_s defer to that instead, otherwise fallback to the totally portable volatile + for loop.

vvanders 11y ago
Volatile should only be used with hardware registers. It doesn't do exactly what you want here. It will guarantee that memory will be accessed but it doesn't guarantee the ordering which can lead to some really nasty behaviour. The only place that keywork should be used is a qualifier for member functions or when used in an embedded sense. It's not well defined outside of that scope.
- kllrnohj 11y ago
  volatile's designed purpose is for memory-mapped things, not hardware registers.
dezgeg 11y ago
> Of course there is, you just use the volatile keyword. Colin Percival of FreeBSD disagrees: http://www.daemonology.net/blog/2014-09-04-how-to-zero-a-buffer.html http://www.daemonology.net/blog/2014-09-04-how-to-zero-a-buf...
- TheLoneWolfling 11y ago
  And that doesn't even get into the other aspect of it: Namely that C / C++ allows temporary copies of variables that are not cleared afterwards. The most obvious case of this being things being temporarily copied into registers / stack, but there are other examples as well.
- kllrnohj 11y ago
  Not really. He just says if the compiler can optimize away prior to it being declared volatile.

trentnelson 11y ago

SecureZeroMemory: https://msdn.microsoft.com/en-us/library/windows/desktop/aa366877(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...

TheLoneWolfling 11y ago
...I do not think the word "portable" means what you think it means.

overgard 11y ago

You could just compile with optimizations off.

TheLoneWolfling 11y ago
There is no such thing in portable C / C++. There are some ISAs that pretty much require optimizations, for instance.

jimjimjim 11y ago

cheat. after resetting the memory, copy the first and last bytes and log them out somewhere. that'll prevent any optimizations

TheLoneWolfling 11y ago
If only it were so simple. You underestimate the evilness of compilers. The compiler can (and will!) just propagate the values through directly and skip the memset.

lmm 11y ago

I'm not aware of any portable language that offers semantics that guarantee data is never rendered accessible to the outside world (I could well believe that current implementations don't, but I'm not aware of anything that specifies it). It always comes down to platform-specific APIs. But that's not a reason to not use C - unless you can suggest a better alternative?

TheLoneWolfling 11y ago
Quite frankly, assembly currently is about it. And that's not the most portable thing out there, I know. I think that C or C++ could, without too much effort, support semantics that would allow for this sort of thing. Something as simple as a "secure" keyword that could be applied to variables (where it means "leak as little as possible about this variable when it goes out of scope") or functions (where it means the same, but for the function itself and all locals of the function).