6 ms·

Compilers are sane and written by very smart people. They do have bugs just like any other software. In my 20+ yrs of coding, I can count the number of bugs I'v

Compilers are sane and written by very smart people. They do have bugs just like any other software. In my 20+ yrs of coding, I can count the number of bugs I've encountered on one hand. memset tend to be optimized out in the dead code elimination pass and highly relies on SSA. Compilers adhere to the abstract machine specs. If you want memset not to be optimized out then use memset_s because the spec explicitly doesn't allow that. if an optimization relies on a UB and surprised the programmer then the programmer was also relying on a UB in the first place.

secondcoming 5y ago
Isn't memset different in that it's an external function that compiler writers have chosen to intercept and perform themselves? Why is saying 'use memset_s' acceptable when they could have stopped treating memset as a special case?
- chrisseaton 5y ago
  > Isn't memset different in that it's an external function that compiler writers have chosen to intercept and perform themselves? The semantics of memset and memset_s are defined by a standard. They're not unknown functions that could do anything, and they're breaking a rule by treating them differently, like you're suggesting they are.
- Someone 5y ago
  If you include a standard header (using <> brackets) and call a function that the standard says get declared when doing that, the compiler is allowed to assume you want the standard behavior, and inline that/replace it by a specialized version/etc. Optimizing memset, in particular, can reap huge benefits, for example for sizes of 4 or 8 bytes in tight loops or for zeroing cache line sized blocks.
- halayli 5y ago
  memset is not a special case. Compiler designers are very much against special casing, and rightfully so. Compilers do have an intrinsic equivalent but that has nothing to do with whether it gets optimized out or kept. Following SSA rules, if you are writing to a non-volatile memory and not reading from it for the remaining duration of its lifetime then it's dead code. use memset_s was a suggestion if your goal is to scrub memory because it guarantees the write will happen. The guarantee comes from the C standard: > Unlike memset, any call to the memset_s function shall be evaluated strictly according to the rules of the abstract machine as described in (5.1.2.3). That is, any call to the memset_s function shall assume that the memory indicated by s and n may be accessible in the future and thus must contain the values indicated by c Which essentially means treat the memory as-if it's volatile.
  - 1ris 5y ago
    Are you allowed to cast the void * to volatile void *, pass that to memset and except the same result, or is the void *, volatile void*, void * conversion eliminated aswell?
    - ncmncm 5y ago
      You can wave whichever dead chickens you like over the keyboard, all to the same effect. The compiler ensures only semantics that are observable. If you pass a pointer to a volatile or atomic object to another function the compiler cannot see into, the compiler knows that any specified operations might be observable, so must act accordingly. Then, operations on that object will not be optimized away. Or, anyway, not until someone builds the program using Link Time Optimization ("-flto") and the called function body becomes visible.
      - 1ris 5y ago
        I'm confused. Is the memset to the temporary volatile void* observable or not, and is the temporary possibly eliminated?
        ncmncm 5y ago
        Casting the pointer passed to memset has no effect. The compiler knows what the storage pointed-to is: a temporary object. The compiler knows that memset does not allow a pointer to the object to escape from view. So, it can eliminate the call. Calling C11's memset_s, or MS's SecureZeroMemory, or FreeBSD's explicit_bzero, has the effect, to the compiler, of making the object observable, so that operations on it will not be optimized away. There is no way to express this locally within the language; you must rely on facilities provided by a Standard or your platform, or on the dodgy linkage trick cited elsewhere.
    - halayli 5y ago
      volatile is a type qualifier and can be casted. However, memset's declaration takes a void* which prevents you from passing a volatile void*. casting or not won't change anything in this case because what matters is what memset takes and not what the pointer's fully qualified type is prior to calling memset. From the compiler's perspective, you're passing a non-volatile pointer to memset which makes it a candidate for elimination if the pointer's lifetime ends after memset call.
userbinator 5y ago
If anything I think they are written by people who are too smart --- theoreticians who care only about the standard and can't be bothered exercising common sense when the standard doesn't specify what to do. http://blog.metaobject.com/2014/04/cc-osmartass.html http://blog.metaobject.com/2014/04/cc-osmartass.html
- beached_whale 5y ago
  The example about signed overflow misses the point, on good code it results in a performance improvement. Similar for dereference of null pointers; it allows the compiler to remove visible successive checks(e.g if you call *ptr, then all further calls to ptr will not be null until it is written to). This makes it faster to express preconditions but let the compiler make safer code faster.
- halayli 5y ago
  Your definition of what's considered common sense is based on your sole understanding of computers. It's more like your intuition. Just because you don't understand something fully doesn't mean it's not common sense. It's very common that programmers don't bother reading the standard or attempt to understand the concept of an abstract machine. Programmers that know what they are doing tend not to be surprised by such nuances.
  - userbinator 5y ago
    No, I've read the standard. The "abstract machine" is, as its name implies, abstract. In reality, there are other concerns that are equally if not more important. That is the problem with compiler writers, ignoring those practical concerns. They prefer to downvote rather than face the reality of going outside their imaginary abstract machine world. "In theory, theory and practice are the same. In practice, they are not."
    - josephcsible 5y ago
      The C standard can be thought of as a bunch of minimum requirements that program authors and compiler authors have to follow. What you seem to be arguing is for compiler authors to go above and beyond what's required of them, so that you as a program author can get away with doing less than the minimum that's required for you.
1ris 5y ago
The problem with programming is almost never the lack of smartness. It's almost always the opposite: People being too clever. The sane thing in a realistic scenario is to ban all usage of memset and add a diagnostic to use memset_s, like any responsible programmer should do with strcpy. Then however, was it really wise to make this machinery, that the people who need it the most are most likely to not have, necessary? I really, really don't think so. \Edit: Here is some code written by a very smart person, Niels Fugerson. I think (not sure) it tries to wipe sensive material from memory and gets it wrong. Really not sure, tho. https://github.com/MacPass/KeePassKit/blob/master/TwoFish/twofish.c https://github.com/MacPass/KeePassKit/blob/master/TwoFish/tw...
- ncmncm 5y ago
  Yes, this code gets it wrong. It demonstrates that there are many different meanings for "smart", and even for "smart programmer". Nobody qualifies for all of them.
  - 1ris 5y ago
    If Niels Furgerson does not qualify as "smart" for programming I don't and I don't think i have ever met anyone who does.
    - halayli 5y ago
      being smart doesn't eliminate the possibility of making mistakes. Are you expecting that a smart human being isn't going to make mistakes? Go through the revision history and see how many bugs got fixed.
- halayli 5y ago
  There's absolutely nothing wrong with using memset. I don't understand why you would suggest such a ban. If your goal is to scrub a memory region after you're completely done using it then you shouldn't use memset. Compilers won't optimize out memset if it changes the program's behavior within the boundaries of abstract machine's specification. A call to memset to clear a memory region after you're done with it because a password was stored there can be optimized out because it doesn't change the behavior of the program. The code you shared is incorrect if your definition of correctness here includes K being wiped out.

secondcoming 5y ago

Isn't memset different in that it's an external function that compiler writers have chosen to intercept and perform themselves? Why is saying 'use memset_s' acceptable when they could have stopped treating memset as a special case?

chrisseaton 5y ago
> Isn't memset different in that it's an external function that compiler writers have chosen to intercept and perform themselves? The semantics of memset and memset_s are defined by a standard. They're not unknown functions that could do anything, and they're breaking a rule by treating them differently, like you're suggesting they are.
Someone 5y ago
If you include a standard header (using <> brackets) and call a function that the standard says get declared when doing that, the compiler is allowed to assume you want the standard behavior, and inline that/replace it by a specialized version/etc. Optimizing memset, in particular, can reap huge benefits, for example for sizes of 4 or 8 bytes in tight loops or for zeroing cache line sized blocks.
halayli 5y ago
memset is not a special case. Compiler designers are very much against special casing, and rightfully so. Compilers do have an intrinsic equivalent but that has nothing to do with whether it gets optimized out or kept. Following SSA rules, if you are writing to a non-volatile memory and not reading from it for the remaining duration of its lifetime then it's dead code. use memset_s was a suggestion if your goal is to scrub memory because it guarantees the write will happen. The guarantee comes from the C standard: > Unlike memset, any call to the memset_s function shall be evaluated strictly according to the rules of the abstract machine as described in (5.1.2.3). That is, any call to the memset_s function shall assume that the memory indicated by s and n may be accessible in the future and thus must contain the values indicated by c Which essentially means treat the memory as-if it's volatile.
- 1ris 5y ago
  Are you allowed to cast the void * to volatile void *, pass that to memset and except the same result, or is the void *, volatile void*, void * conversion eliminated aswell?
  - ncmncm 5y ago
    You can wave whichever dead chickens you like over the keyboard, all to the same effect. The compiler ensures only semantics that are observable. If you pass a pointer to a volatile or atomic object to another function the compiler cannot see into, the compiler knows that any specified operations might be observable, so must act accordingly. Then, operations on that object will not be optimized away. Or, anyway, not until someone builds the program using Link Time Optimization ("-flto") and the called function body becomes visible.
    - 1ris 5y ago
      I'm confused. Is the memset to the temporary volatile void* observable or not, and is the temporary possibly eliminated?
      - ncmncm 5y ago
        Casting the pointer passed to memset has no effect. The compiler knows what the storage pointed-to is: a temporary object. The compiler knows that memset does not allow a pointer to the object to escape from view. So, it can eliminate the call. Calling C11's memset_s, or MS's SecureZeroMemory, or FreeBSD's explicit_bzero, has the effect, to the compiler, of making the object observable, so that operations on it will not be optimized away. There is no way to express this locally within the language; you must rely on facilities provided by a Standard or your platform, or on the dodgy linkage trick cited elsewhere.
  - halayli 5y ago
    volatile is a type qualifier and can be casted. However, memset's declaration takes a void* which prevents you from passing a volatile void*. casting or not won't change anything in this case because what matters is what memset takes and not what the pointer's fully qualified type is prior to calling memset. From the compiler's perspective, you're passing a non-volatile pointer to memset which makes it a candidate for elimination if the pointer's lifetime ends after memset call.

userbinator 5y ago

If anything I think they are written by people who are too smart --- theoreticians who care only about the standard and can't be bothered exercising common sense when the standard doesn't specify what to do. http://blog.metaobject.com/2014/04/cc-osmartass.html http://blog.metaobject.com/2014/04/cc-osmartass.html

beached_whale 5y ago
The example about signed overflow misses the point, on good code it results in a performance improvement. Similar for dereference of null pointers; it allows the compiler to remove visible successive checks(e.g if you call *ptr, then all further calls to ptr will not be null until it is written to). This makes it faster to express preconditions but let the compiler make safer code faster.
halayli 5y ago
Your definition of what's considered common sense is based on your sole understanding of computers. It's more like your intuition. Just because you don't understand something fully doesn't mean it's not common sense. It's very common that programmers don't bother reading the standard or attempt to understand the concept of an abstract machine. Programmers that know what they are doing tend not to be surprised by such nuances.
- userbinator 5y ago
  No, I've read the standard. The "abstract machine" is, as its name implies, abstract. In reality, there are other concerns that are equally if not more important. That is the problem with compiler writers, ignoring those practical concerns. They prefer to downvote rather than face the reality of going outside their imaginary abstract machine world. "In theory, theory and practice are the same. In practice, they are not."
  - josephcsible 5y ago
    The C standard can be thought of as a bunch of minimum requirements that program authors and compiler authors have to follow. What you seem to be arguing is for compiler authors to go above and beyond what's required of them, so that you as a program author can get away with doing less than the minimum that's required for you.

1ris 5y ago

The problem with programming is almost never the lack of smartness. It's almost always the opposite: People being too clever. The sane thing in a realistic scenario is to ban all usage of memset and add a diagnostic to use memset_s, like any responsible programmer should do with strcpy. Then however, was it really wise to make this machinery, that the people who need it the most are most likely to not have, necessary? I really, really don't think so. \Edit: Here is some code written by a very smart person, Niels Fugerson. I think (not sure) it tries to wipe sensive material from memory and gets it wrong. Really not sure, tho. https://github.com/MacPass/KeePassKit/blob/master/TwoFish/twofish.c https://github.com/MacPass/KeePassKit/blob/master/TwoFish/tw...

ncmncm 5y ago
Yes, this code gets it wrong. It demonstrates that there are many different meanings for "smart", and even for "smart programmer". Nobody qualifies for all of them.
- 1ris 5y ago
  If Niels Furgerson does not qualify as "smart" for programming I don't and I don't think i have ever met anyone who does.
  - halayli 5y ago
    being smart doesn't eliminate the possibility of making mistakes. Are you expecting that a smart human being isn't going to make mistakes? Go through the revision history and see how many bugs got fixed.
halayli 5y ago
There's absolutely nothing wrong with using memset. I don't understand why you would suggest such a ban. If your goal is to scrub a memory region after you're completely done using it then you shouldn't use memset. Compilers won't optimize out memset if it changes the program's behavior within the boundaries of abstract machine's specification. A call to memset to clear a memory region after you're done with it because a password was stored there can be optimized out because it doesn't change the behavior of the program. The code you shared is incorrect if your definition of correctness here includes K being wiped out.