5 ms·

No one "always writes correct large-scale code". I have worked with many who write large-scale C and C++ code with no undefined behavior. Is that what you mean

No one "always writes correct large-scale code". I have worked with many who write large-scale C and C++ code with no undefined behavior. Is that what you meant?

pcwalton 10y ago
Yes. I've never seen it. All large-scale C and C++ codebases I've ever seen, from large companies to small ones, have had memory safety/undefined behavior problems [1]. If "just hire better programmers" were a workable solution, surely one company out of {Google, Apple, Microsoft, Facebook} would have succeeded at that strategy by now. It's very easy to write C and C++ code that looks like it's free of undefined behavior, but in every case I've seen they end up falling when attackers actively try to look for problems. [1]: Maybe qmail is the one exception, though even that had a famous debate related to overflow.
- jjnoakes 10y ago
  Have you ever worked with teams who use aggressive static analysis tools to detect and catch undefined behavior? Because I have, and it works incredibly well. Of course, the caveat is that once you turn up the static analysis aggressiveness (assuming you use a good static analysis tool), you will need to put in plenty of assert(index < len), ownership annotations, and other such items in your code to satisfy the checks. But once you do, it's really hard (probably not impossible, but really hard) to trigger undefined behavior without the static analysis tool catching it. Because if the tool decides it can't prove there's no undefined behavior, it is configured to complain, and you adjust the code until it stops. I'm not saying many teams do this, but I am disputing that no teams do this.
  - vardump 10y ago
    What tools do you recommend?
    - jjnoakes 10y ago
      Unfortunately I have never used one available commercially which meets my needs. We author one in-house.
      - nickpsecurity 10y ago
        Your tool is better than Astree Analyzer and Polyspace? I think you should consider GPLing or licensing it with commercial support. Judging from their prices & effectiveness, you'll make a killing doing better while undercutting their licensing.
        jjnoakes 10y ago
        The problem is usability and licensing costs. Our in-house tool is amazingly powerful and fits our needs but the usability would need improvement (no surprise) and we license some of the utility code we use (which is cheaper if we don't resell our tool).
        nickpsecurity 10y ago
        You couldn't replace the utility code with something in FOSS? And that's why your company is holding out on a static analyzer better than anything out there? The case for doing something to open that tool is just getting stronger. Btw, what utility tool are you licensing? What makes it irreplaceable? I know of only one in this field that I truly couldn't replace for compilers or static analysis. Even then, there's quite a few that handle the job well enough to not need it. So, I don't use it.
        jjnoakes 10y ago
        The utility tool has OSS equivalents but we've evaluated them all. None of them have the features we need, and adding them would be person-years of work we can't afford. My hope is to some day open source our internal tool, but we have to wait until the OSS features we need catch up. Until then, we aren't willing to pay the high license fee to redistribute this utility freely. I'm speaking a little cryptically because I'd rather not have this tied directly to any particular company (neither mine nor the one we license this code from). I'm not speaking on their behalves officially; only my own.
  - pcwalton 10y ago
    > Have you ever worked with teams who use aggressive static analysis tools to detect and catch undefined behavior? Coverity is actually used on browser engines, and it has not been able to stem the tide of exploitable security vulnerabilities. Sound static analysis is just too difficult on idiomatic C and C++: it's effectively impossible, as the language was just not designed for it.
    - jjnoakes 10y ago
      I don't agree. I work with static analysis daily and idiomatic code is the easiest to analyze. If coverity isn't catching the bugs and they are undefined behavior bugs then coverity needs to be more aggressive. But it probably doesn't support things like ownership annotations, and it probably gives up strict checking in order to avoid false positives. The problem is developers. Developers are lazy. Most developers won't even turn on compiler warnings, never mind using the sanitizers. So tool makers create sub-par tools to cater to them. Fewer false warnings means more customers, even if it also means fewer security holes. It's why we use an in house tool and crank up the checks to 11.
  - pjmlp 10y ago
    > I'm not saying many teams do this, but I am disputing that no teams do this. If you watch Herb Sutter's talk at CppCon 2015, at a given point he asks the audience how many know and use such tools. It is one of the most important C++ conferences, usually attended by the most savvy C++ developers in the world, the amount of the audience saying that they do use such tools was ruffly 1%! 1% shows how much the majority of C and C++ developers, or their employers, care about writing proper safe code in those languages. The only way out is moving to programming languages whose safety must be explicitly turned off, and not having to be explicitly turned on, because most won't bother taking the effort to turning it on.
    - jjnoakes 10y ago
      No one is arguing that the industry moving to a safer language will reduce the level of nasty bugs in the world. That would be silly to argue against, because that's basically a tautology anyway. What I'm disputing is that no teams write good C or C++ code. In fact, the 1% who raised their hands proves my point. It isn't 0%. That's all I'm saying. Don't smear our names with the "all C and C++ developers write code with exploitable undefined behavior bugs in it" when you mean "most C and C++ developers write code with exploitable undefined behavior bugs in it".
      - pjmlp 10y ago
        C++ is my language to go, every time I need to go out of JVM and .NET ecosystems and I do take all efforts to be part of that 1%. Once upon a time I gave C++ class to first year students at the university where I took my degree. Worked with C++ at some well known companies and research institutions. Still, I won't say I never write exploitable undefined behaviour free code in C++, let alone C. Because I cannot control the code that gets used by third party libraries, written by team members or even having the whole UB use cases on my head.
        jjnoakes 10y ago
        If you use the right tools (static analysis and sanitizers and fuzzers) your own code (and any code you rely on which you have the source for) will be handled. If you rely on binary dependencies which have quality problems, then u can't help you there. But that could be an attack vector in any language. And if you are worried about the operating system itself, then there really is no easy way around it unless you want to run a unikernel for everything. And even then you may hit a CPU microcode bug it hardware bug...

No one "always writes correct large-scale code". I have worked with many who write large-scale C and C++ code with no undefined behavior. Is that what you meant?

pcwalton 10y ago

Yes. I've never seen it. All large-scale C and C++ codebases I've ever seen, from large companies to small ones, have had memory safety/undefined behavior problems [1]. If "just hire better programmers" were a workable solution, surely one company out of {Google, Apple, Microsoft, Facebook} would have succeeded at that strategy by now. It's very easy to write C and C++ code that looks like it's free of undefined behavior, but in every case I've seen they end up falling when attackers actively try to look for problems. [1]: Maybe qmail is the one exception, though even that had a famous debate related to overflow.

jjnoakes 10y ago
Have you ever worked with teams who use aggressive static analysis tools to detect and catch undefined behavior? Because I have, and it works incredibly well. Of course, the caveat is that once you turn up the static analysis aggressiveness (assuming you use a good static analysis tool), you will need to put in plenty of assert(index < len), ownership annotations, and other such items in your code to satisfy the checks. But once you do, it's really hard (probably not impossible, but really hard) to trigger undefined behavior without the static analysis tool catching it. Because if the tool decides it can't prove there's no undefined behavior, it is configured to complain, and you adjust the code until it stops. I'm not saying many teams do this, but I am disputing that no teams do this.
- vardump 10y ago
  What tools do you recommend?
  - jjnoakes 10y ago
    Unfortunately I have never used one available commercially which meets my needs. We author one in-house.
    - nickpsecurity 10y ago
      Your tool is better than Astree Analyzer and Polyspace? I think you should consider GPLing or licensing it with commercial support. Judging from their prices & effectiveness, you'll make a killing doing better while undercutting their licensing.
      - jjnoakes 10y ago
        The problem is usability and licensing costs. Our in-house tool is amazingly powerful and fits our needs but the usability would need improvement (no surprise) and we license some of the utility code we use (which is cheaper if we don't resell our tool).
        nickpsecurity 10y ago
        You couldn't replace the utility code with something in FOSS? And that's why your company is holding out on a static analyzer better than anything out there? The case for doing something to open that tool is just getting stronger. Btw, what utility tool are you licensing? What makes it irreplaceable? I know of only one in this field that I truly couldn't replace for compilers or static analysis. Even then, there's quite a few that handle the job well enough to not need it. So, I don't use it.
        jjnoakes 10y ago
        The utility tool has OSS equivalents but we've evaluated them all. None of them have the features we need, and adding them would be person-years of work we can't afford. My hope is to some day open source our internal tool, but we have to wait until the OSS features we need catch up. Until then, we aren't willing to pay the high license fee to redistribute this utility freely. I'm speaking a little cryptically because I'd rather not have this tied directly to any particular company (neither mine nor the one we license this code from). I'm not speaking on their behalves officially; only my own.
- pcwalton 10y ago
  > Have you ever worked with teams who use aggressive static analysis tools to detect and catch undefined behavior? Coverity is actually used on browser engines, and it has not been able to stem the tide of exploitable security vulnerabilities. Sound static analysis is just too difficult on idiomatic C and C++: it's effectively impossible, as the language was just not designed for it.
  - jjnoakes 10y ago
    I don't agree. I work with static analysis daily and idiomatic code is the easiest to analyze. If coverity isn't catching the bugs and they are undefined behavior bugs then coverity needs to be more aggressive. But it probably doesn't support things like ownership annotations, and it probably gives up strict checking in order to avoid false positives. The problem is developers. Developers are lazy. Most developers won't even turn on compiler warnings, never mind using the sanitizers. So tool makers create sub-par tools to cater to them. Fewer false warnings means more customers, even if it also means fewer security holes. It's why we use an in house tool and crank up the checks to 11.
- pjmlp 10y ago
  > I'm not saying many teams do this, but I am disputing that no teams do this. If you watch Herb Sutter's talk at CppCon 2015, at a given point he asks the audience how many know and use such tools. It is one of the most important C++ conferences, usually attended by the most savvy C++ developers in the world, the amount of the audience saying that they do use such tools was ruffly 1%! 1% shows how much the majority of C and C++ developers, or their employers, care about writing proper safe code in those languages. The only way out is moving to programming languages whose safety must be explicitly turned off, and not having to be explicitly turned on, because most won't bother taking the effort to turning it on.
  - jjnoakes 10y ago
    No one is arguing that the industry moving to a safer language will reduce the level of nasty bugs in the world. That would be silly to argue against, because that's basically a tautology anyway. What I'm disputing is that no teams write good C or C++ code. In fact, the 1% who raised their hands proves my point. It isn't 0%. That's all I'm saying. Don't smear our names with the "all C and C++ developers write code with exploitable undefined behavior bugs in it" when you mean "most C and C++ developers write code with exploitable undefined behavior bugs in it".
    - pjmlp 10y ago
      C++ is my language to go, every time I need to go out of JVM and .NET ecosystems and I do take all efforts to be part of that 1%. Once upon a time I gave C++ class to first year students at the university where I took my degree. Worked with C++ at some well known companies and research institutions. Still, I won't say I never write exploitable undefined behaviour free code in C++, let alone C. Because I cannot control the code that gets used by third party libraries, written by team members or even having the whole UB use cases on my head.
      - jjnoakes 10y ago
        If you use the right tools (static analysis and sanitizers and fuzzers) your own code (and any code you rely on which you have the source for) will be handled. If you rely on binary dependencies which have quality problems, then u can't help you there. But that could be an attack vector in any language. And if you are worried about the operating system itself, then there really is no easy way around it unless you want to run a unikernel for everything. And even then you may hit a CPU microcode bug it hardware bug...